Track volume usage data at a vm granularity as well #11531

abh1sar · 2025-08-29T05:57:51Z

Description

This PR fixes #10985

Some users have asked to show the virtualmachineid also in volume usage, to track it per instance.

There will be two kinds of records now.

Without the virtualmachineid - tracking the cumulative volume usage in the usage window. This is to be consistent with the current behaviour.
With virtualmachineid - tracking the volume usage for the time it was attached to the vm in the usage window.

If the volume was attached to multiple instances in the same usage window, we will have separate entries for each VM.

Schema changes

Add column vm_id to cloud.usage_event and cloud_usage.usage_event
Add column vm_id to cloud_usage.usage_volume

I have also done some code refactoring in createVolumeHelperEvent() to make it easier to follow.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

listUsageRecords for a volume in a 2 minute (0.033333 hours) window.
The volume was created, then attached to VM-00a42c28-74ae-4а0c-b38a-4e3b88f11241. It was then detached and attached to another VM QA-d6df4657-2fb4-4e26-b7de-21743c5463cd. So we have 3 records.

How Has This Been Tested?

How did you try to break this feature and the system with this change?

abh1sar · 2025-08-29T06:00:36Z

@blueorangutan package

blueorangutan · 2025-08-29T06:02:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-08-29T06:07:04Z

Codecov Report

❌ Patch coverage is 2.89017% with 168 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.50%. Comparing base (cd12fa5) to head (acef620).

Files with missing lines	Patch %	Lines
...rc/main/java/com/cloud/usage/UsageManagerImpl.java	0.00%	82 Missing ⚠️
...n/java/com/cloud/usage/dao/UsageVolumeDaoImpl.java	0.00%	20 Missing ⚠️
...ma/src/main/java/com/cloud/event/UsageEventVO.java	0.00%	17 Missing ⚠️
...java/com/cloud/usage/parser/VolumeUsageParser.java	0.00%	13 Missing ⚠️
...src/main/java/com/cloud/event/UsageEventUtils.java	0.00%	8 Missing ⚠️
...a/src/main/java/com/cloud/usage/UsageVolumeVO.java	0.00%	8 Missing ⚠️
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	22.22%	7 Missing ⚠️
...er/src/main/java/com/cloud/hypervisor/KVMGuru.java	0.00%	4 Missing ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	40.00%	3 Missing ⚠️
...stack/engine/orchestration/VolumeOrchestrator.java	0.00%	2 Missing ⚠️
... and 3 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11531      +/-   ##
============================================
- Coverage     17.50%   17.50%   -0.01%     
  Complexity    15425    15425              
============================================
  Files          5894     5894              
  Lines        526877   526937      +60     
  Branches      64337    64341       +4     
============================================
+ Hits          92232    92233       +1     
- Misses       424269   424327      +58     
- Partials      10376    10377       +1

Flag	Coverage Δ
uitests	`3.61% <ø> (ø)`
unittests	`18.56% <2.89%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

abh1sar · 2025-08-29T06:13:12Z

engine/schema/src/main/java/com/cloud/usage/dao/UsageVolumeDaoImpl.java

    public UsageVolumeDaoImpl() {
    }

-    @Override


This kind of update method is not required since #5785 added auto-increment primary keys. We can directly use the GenericDao.update()

abh1sar · 2025-08-29T06:49:46Z

usage/src/main/java/com/cloud/usage/UsageManagerImpl.java

-            if (volumesVOs.size() > 0) {
-                //This is a safeguard to avoid double counting of volumes.
-                s_logger.error("Found duplicate usage entry for volume: " + volId + " assigned to account: " + event.getAccountId() + "; marking as deleted...");
+            if (event.getVmId() != null) {


VOLUME_CREATE event will contain the vmId also when it is created during Instance deployment. Need to add a vm specific entry also in that case.

blueorangutan · 2025-08-29T07:03:43Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14760

DaanHoogland

looks good, except for the upgrade path (and needs testing)

engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade41930to41940.java

engine/schema/src/main/resources/META-INF/db/schema-41930to41940.sql

sudo87 · 2025-08-29T10:55:47Z

clgtm

DaanHoogland · 2025-09-01T11:19:19Z

code looks good in isolation @abh1sar , but I see some bears on the road: what if someone upgrades from 20.1 to 21 and then on to 22; How do we guarantee they get the DB updates?
It may be wiser to put this in version 22.

Sorry, I overlooked this when I asked you to rebase earlier.

…nto vol-usage

abh1sar · 2025-09-30T13:27:15Z

@blueorangutan package

blueorangutan · 2025-09-30T13:28:03Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-30T14:41:25Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 15238

abh1sar · 2025-09-30T15:04:03Z

@blueorangutan package

blueorangutan · 2025-09-30T15:06:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-30T16:27:58Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15241

vishesh92 · 2025-10-01T05:32:17Z

@blueorangutan test keepEnv

blueorangutan · 2025-10-01T05:34:04Z

@vishesh92 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

usage/src/main/java/com/cloud/usage/UsageManagerImpl.java

vishesh92

code lgtm

Co-authored-by: Vishesh <[email protected]>

abh1sar · 2025-10-01T08:08:07Z

@blueorangutan package

blueorangutan · 2025-10-01T08:10:03Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-10-01T09:44:17Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15249

github-actions · 2025-10-01T14:05:49Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…nto vol-usage

abh1sar · 2025-10-01T14:26:05Z

@blueorangutan package

blueorangutan · 2025-10-01T14:28:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-10-01T16:04:20Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15257

blueorangutan · 2025-10-01T22:39:06Z

[SF] Trillian test result (tid-14492)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 58325 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11531-t14492-kvm-ol8.zip
Smoke tests completed. 146 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	334.74	test_kubernetes_clusters.py

boring-cyborg bot added component:orchestration component:storage component:usage-server labels Aug 29, 2025

abh1sar changed the base branch from main to 4.19 August 29, 2025 05:58

abh1sar closed this Aug 29, 2025

abh1sar reopened this Aug 29, 2025

abh1sar commented Aug 29, 2025

View reviewed changes

DaanHoogland reviewed Aug 29, 2025

View reviewed changes

engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade41930to41940.java Show resolved Hide resolved

engine/schema/src/main/resources/META-INF/db/schema-41930to41940.sql Show resolved Hide resolved

abh1sar force-pushed the vol-usage branch from cfe9746 to 49abb29 Compare September 1, 2025 08:24

boring-cyborg bot added component:agent component:build component:integration labels Sep 1, 2025

abh1sar changed the base branch from 4.19 to 4.20 September 1, 2025 08:25

abh1sar requested a review from DaanHoogland September 1, 2025 08:27

weizhouapache added this to the 4.22.0 milestone Sep 1, 2025

abh1sar added 3 commits September 9, 2025 10:53

Track volume usage data at vm granularity as well.

36c4f3d

fix UT and schema

3ba0d96

Schema change and UT fix

025ae71

abh1sar force-pushed the vol-usage branch from 49abb29 to 025ae71 Compare September 9, 2025 11:05

boring-cyborg bot added the component:api label Sep 9, 2025

abh1sar changed the base branch from 4.20 to main September 9, 2025 11:05

abh1sar closed this Sep 9, 2025

abh1sar reopened this Sep 9, 2025

abh1sar added 4 commits September 30, 2025 18:19

address review comments in UsageManagerImpl

736d9b4

Merge branch 'vol-usage' of https://github.com/shapeblue/cloudstack i…

114aff9

…nto vol-usage

Merge remote-tracking branch 'upstream/main' into vol-usage

2d52ad6

fix String.format in UsageManagerImpl

072645b

apache deleted a comment from blueorangutan Oct 1, 2025

vishesh92 reviewed Oct 1, 2025

View reviewed changes

usage/src/main/java/com/cloud/usage/UsageManagerImpl.java Outdated Show resolved Hide resolved

vishesh92 approved these changes Oct 1, 2025

View reviewed changes

Update usage/src/main/java/com/cloud/usage/UsageManagerImpl.java

76706f5

Co-authored-by: Vishesh <[email protected]>

vishesh92 added the status:needs-testing label Oct 1, 2025

github-actions bot added the status:has-conflicts label Oct 1, 2025

abh1sar added 2 commits October 1, 2025 19:54

Merge remote-tracking branch 'upstream/main' into vol-usage

2b381c7

Merge branch 'vol-usage' of https://github.com/shapeblue/cloudstack i…

acef620

…nto vol-usage

github-actions bot removed the status:has-conflicts label Oct 1, 2025

Track volume usage data at a vm granularity as well #11531

Are you sure you want to change the base?

Track volume usage data at a vm granularity as well #11531

Uh oh!

Conversation

abh1sar commented Aug 29, 2025

Description

Schema changes

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

abh1sar commented Aug 29, 2025

Uh oh!

blueorangutan commented Aug 29, 2025

Uh oh!

codecov bot commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

abh1sar Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

abh1sar Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Aug 29, 2025

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sudo87 commented Aug 29, 2025

Uh oh!

DaanHoogland commented Sep 1, 2025

Uh oh!

abh1sar commented Sep 30, 2025

Uh oh!

blueorangutan commented Sep 30, 2025

Uh oh!

blueorangutan commented Sep 30, 2025

Uh oh!

abh1sar commented Sep 30, 2025

Uh oh!

blueorangutan commented Sep 30, 2025

Uh oh!

blueorangutan commented Sep 30, 2025

Uh oh!

vishesh92 commented Oct 1, 2025

Uh oh!

blueorangutan commented Oct 1, 2025

Uh oh!

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar commented Oct 1, 2025

Uh oh!

blueorangutan commented Oct 1, 2025

Uh oh!

blueorangutan commented Oct 1, 2025

Uh oh!

github-actions bot commented Oct 1, 2025

Uh oh!

abh1sar commented Oct 1, 2025

Uh oh!

blueorangutan commented Oct 1, 2025

Uh oh!

blueorangutan commented Oct 1, 2025

Uh oh!

blueorangutan commented Oct 1, 2025

Uh oh!

Uh oh!

codecov bot commented Aug 29, 2025 •

edited

Loading