No cordination among usage service running on multiple management servers

[kiranchavala]

problem

If there are multiple management server and usage service is running on both the management servers then there is no coordination

versions

ACS 4.20

The steps to reproduce the bug

Steps to reproduce the issue

1. Have a CloudStack environment (4.20.2) with 2 mgmt servers

2. Have the CloudStack usage service running on both management servers

3. On mgmt 1 , stop the usage service , Check the usage job in the UI

4. On mgmt 2 , stop the usage service, check the usage job in th UI

It becomes Red N/A

5. On mgmt 2 , start the usage service

6. It remains in alert state until the usage service is started on mgmt 1

7. Stop the usage service on mgmt2 and check the status

Expected behaviour

If the multiple management servers and usage service is running on all the servers then according to the docs, they should coordinate

https://docs.cloudstack.apache.org/en/4.22.0.0/installguide/optional_installation.html#installing-the-usage-server-optional

The Usage Servers will coordinate usage processing. A site that is concerned about availability should install Usage Servers on at least two Management Servers.

What to do about it?

Usage service running on multiple management server should coordinate

[DaanHoogland]

This is how usage works:

• if one starts it claims the job
• if the second starts it refuses to claim the job
• if the first stops and reliquishes the job, nothing happens until
• a usage server wakes up and checks if there is a claimed job.

@kiranchavala, how did you stop the usage server?
Did you check the job records in the DB?

[kiranchavala]

@DaanHoogland Thanks for the usage service logic

Please find the steps that I followed

1. Have a CloudStack environment (4.20.2) with 2 mgmt servers (mgmt1 and mgmt2)

2. Have the CloudStack usage service running on both management servers

3. Change the global setting

usage.stats.job.aggregation.range to 5
usage.stats.job.exec.time to an upcoming timestamp

Restart the cloudstack usage service

Check the usage log on both the management servers , the following message should be present

2026-01-09 08:56:44,369 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (main:[]) (logid:) Usage is configured to execute in time zone [UTC], at [09:00], each [5] minutes; the current time in that timezone is [2026-01-09T08:56:44+0000] and the next job is scheduled to execute at [2026-01-09T09:00:00+0000]. During its execution, Usage will aggregate stats according to the time zone [GMT] defined in global setting [usage.aggregation.timezone].

4. Stop the cloudstack-usage service on mgmt1

service cloudstack-usage stop


[root@ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 ~]# service cloudstack-usage status
Redirecting to /bin/systemctl status cloudstack-usage.service
● cloudstack-usage.service - CloudStack Usage Server
   Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2026-01-09 08:58:14 UTC; 2h 51min ago
     Docs: http://www.cloudstack.org/
  Process: 3822 ExecStart=/bin/sh -ec /usr/bin/java -Dpid=${JAVA_PID} $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=143)
 Main PID: 3822 (code=exited, status=143)

Jan 09 08:56:30 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: cloudstack-usage.service: Main process exited, code=exited, status=143/n/a
Jan 09 08:56:30 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: cloudstack-usage.service: Failed with result 'exit-code'.
Jan 09 08:56:30 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: Stopped CloudStack Usage Server.
Jan 09 08:56:30 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: Started CloudStack Usage Server.
Jan 09 08:58:14 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: Stopping CloudStack Usage Server...
Jan 09 08:58:14 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: cloudstack-usage.service: Main process exited, code=exited, status=143/n/a
Jan 09 08:58:14 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: cloudstack-usage.service: Failed with result 'exit-code'.
Jan 09 08:58:14 ref-trl-10556-k-Mol8-kiran-chavala-mgmt1 systemd[1]: Stopped CloudStack Usage Server.

Check the usage status on the UI

5. Usage log on mgmt 1

2026-01-09 08:58:14,188 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Thread-0:[]) (logid:) exiting Usage Manager
2026-01-09 08:58:14,195 INFO  [usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_ae7858fa] (Thread-0:[]) (logid:) Found 1 opens job, to remove
2026-01-09 08:58:14,195 DEBUG [usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_ae7858fa] (Thread-0:[]) (logid:) Removing job - id: 5, pid: 3822, job type: 0, scheduled: 0, heartbeat: 2026-01-09T08:57:59.000+0000

6. Usage log om mgm2

2026-01-09 09:00:00,002 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) starting usage job...
2026-01-09 09:00:00,006 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) Not owner of usage job, skipping...
2026-01-09 09:00:00,007 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) usage job complete
2026-01-09 09:05:00,001 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) starting usage job...
2026-01-09 09:05:00,004 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) Not owner of usage job, skipping...
2026-01-09 09:05:00,004 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) usage job complete
2026-01-09 09:10:00,001 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) starting usage job...
2026-01-09 09:10:00,004 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) Not owner of usage job, skipping...
2026-01-09 09:10:00,005 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) usage job complete
2026-01-09 09:15:00,001 INFO  [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) starting usage job...

[DaanHoogland]

Sorry @kiranchavala , but you have not convinced me yet, I will replay your scenario.

[Praba-11]

Hi, I’d like to work on this issue. I’m planning to look into how the usage service schedules and executes usage processing when multiple management servers are present, and propose a coordination mechanism so jobs don’t fail when one usage service instance is stopped.

Before I start, are there any expected patterns the project prefers (e.g., DB locks, leader election) for coordinating across MS nodes?

[DaanHoogland]

@Praba-11 , first of all thank you for chipping in. The usage is an (kind of) off-line batch job that runs daily. Ideally It would be able to run and create usage data for specific intervals and usage types. The most important part is that the “owner” of the job will run the usage job for the latest period, and that would hopefully mean the last 24hrs. Taking over a job should never need to happen, but if you think some kind of leader election could benefit the processing, feel free to propose.

If you want to make use of DB locks you are in for a big job. This would only make sense when processing would not be done on the basis of the whole event set, but on a per account or even a per entity basis.

please report back whether you finish the job or not, so we know the state of the issue? thanks.

[DaanHoogland]

@kiranchavala , I ran your scenario and the job is coordinated between the server that is running and the one that is stopped, the red bullit point is there indeed but that does not mean the job is not taken over. The red (or green) bullit only says whether the MS you are connected to has a usage server running locally. I think this is not very useful functionality.

[Praba-11]

@DaanHoogland, Thanks for the clarification and for running the scenario.

Based on your explanation, it looks like the usage job coordination is already functioning correctly, and the red/green indicator only reflects whether the local MS has a usage service running, not the global job ownership or execution state.

So the confusion seems to be around UI feedback rather than backend coordination. I’ll step back from proposing coordination changes here.