Migrate volume improvements, to bypass secondary storage when copy volume between pools is allowed directly #11625

sureshanaparti · 2025-09-12T10:11:17Z

Description

This PR improves migrate volume, to bypass secondary storage when copy volume between pools is allowed directly.

Also, includes the following improvements:

Updates the pool type of the volume wherever required.
Updates suitable disk offering(s) for volume(s) after migrate VM with volumes when there is a change in pool type (shared or local) and suitable disk offering is available. Migrate VM with volume(s) bypasses the service and disk offerings of the volumes, as the target pools for migration are specified, this is the expected behavior currently. But, there seems to be problem when change in pool type (shared or local).
Updates error message when volume migration between local to shared & shared to local without respective disk offering.
Updates Endpoint selection to consider host scope first while copying between primary storages.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested (offline) volume migration between:

NFS pools in the cluster
Local Storage to NFS and vice versa
Ceph to NFS / Local Storage

How did you try to break this feature and the system with this change?

sureshanaparti · 2025-09-12T10:12:18Z

@blueorangutan package

blueorangutan · 2025-09-12T10:14:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-09-12T10:19:36Z

Codecov Report

❌ Patch coverage is 26.66667% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.43%. Comparing base (0108ffd) to head (6ae97f6).
⚠️ Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java	0.00%	26 Missing ⚠️
...tack/storage/motion/AncientDataMotionStrategy.java	52.83%	11 Missing and 14 partials ⚠️
...torage/motion/StorageSystemDataMotionStrategy.java	0.00%	23 Missing ⚠️
...tack/storage/endpoint/DefaultEndPointSelector.java	0.00%	7 Missing ⚠️
...ava/com/cloud/storage/dao/DiskOfferingDaoImpl.java	40.00%	6 Missing ⚠️
...stack/engine/orchestration/VolumeOrchestrator.java	33.33%	4 Missing ⚠️
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	60.00%	3 Missing and 1 partial ⚠️
...ack/engine/subsystem/api/storage/ClusterScope.java	0.00%	3 Missing ⚠️
...dstack/engine/subsystem/api/storage/HostScope.java	40.00%	3 Missing ⚠️
...dstack/engine/subsystem/api/storage/ZoneScope.java	0.00%	3 Missing ⚠️
... and 13 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11625      +/-   ##
============================================
+ Coverage     17.39%   17.43%   +0.03%     
- Complexity    15283    15351      +68     
============================================
  Files          5889     5892       +3     
  Lines        526183   526655     +472     
  Branches      64242    64322      +80     
============================================
+ Hits          91541    91818     +277     
- Misses       424298   424471     +173     
- Partials      10344    10366      +22

Flag	Coverage Δ
uitests	`3.61% <ø> (-0.01%)`	⬇️
unittests	`18.48% <26.66%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2025-09-12T11:40:06Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 14980

sureshanaparti · 2025-09-12T12:41:28Z

@blueorangutan package

blueorangutan · 2025-09-12T12:42:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Copilot

Pull Request Overview

This PR enhances the volume migration functionality to bypass secondary storage when direct copy between pools is possible, improving performance for supported pool types.

Key changes:

Added poolType parameter to volume import/update operations to track storage pool types
Implemented logic to determine when secondary storage can be bypassed during volume migration
Updated volume database records to include pool type information when pools change

Reviewed Changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
AncientDataMotionStrategy.java	Adds bypass logic for supported pool types (NFS, Filesystem) with scope validation
VolumeOrchestrator.java	Updates import/update methods to include poolType parameter
Multiple volume management files	Adds setPoolType() calls when updating volume pool assignments
KVMStorageProcessor.java	Fixes path handling logic for volume copying between pools
VMSnapshotManagerImpl.java	Corrects spelling error in exception messages
Scope classes	Adds toString() methods for better debugging

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java

blueorangutan · 2025-09-12T14:52:35Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 14981

weizhouapache · 2025-09-12T18:11:38Z

@blueorangutan test

blueorangutan · 2025-09-12T18:12:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-09-13T10:05:22Z

[SF] Trillian test result (tid-14312)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53901 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11625-t14312-kvm-ol8.zip
Smoke tests completed. 146 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	1.16	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	28.84	test_kubernetes_clusters.py

weizhouapache

overall lgtm

not tested yet

...datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java

...chestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

sureshanaparti · 2025-09-16T08:14:11Z

@blueorangutan package

blueorangutan · 2025-09-16T08:16:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-17T07:51:25Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 15051

sureshanaparti · 2025-09-17T11:19:38Z

@blueorangutan package

blueorangutan · 2025-09-17T11:20:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-17T13:51:03Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15058

github-actions · 2025-09-17T20:46:17Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2025-09-17T21:04:05Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…lume between pools is allowed directly

- local storage on host in the same zone - cluser-wide pools in the same zone

…permits

… some code improvements

sureshanaparti · 2025-09-18T05:25:01Z

@blueorangutan package

blueorangutan · 2025-09-18T05:26:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-18T06:38:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15073

sureshanaparti · 2025-09-23T09:13:59Z

@blueorangutan test

blueorangutan · 2025-09-23T09:16:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-09-24T01:05:22Z

[SF] Trillian test result (tid-14425)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53661 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11625-t14425-kvm-ol8.zip
Smoke tests completed. 146 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	1.18	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	34.99	test_kubernetes_clusters.py

…volumes when change in pool type (shared or local) Currently, Migrate VM with volume(s) bypasses the service and disk offerings of the volumes, as the target pools for migration are specified, which ignores the offerings. Offering change is required when pool type (shared or local) is changed, mainly - when volume on shared pool is migrated to local pool - when volume on local pool is migrated to shared pool

…offering type mismatches (both are not shared/local)

…ween primary storages

sureshanaparti · 2025-09-26T05:28:23Z

@blueorangutan package

blueorangutan · 2025-09-26T05:30:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-26T06:59:25Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15177

DaanHoogland · 2025-09-26T13:20:22Z

@blueorangutan test

blueorangutan · 2025-09-26T13:22:06Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-09-27T05:17:03Z

[SF] Trillian test result (tid-14452)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53643 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11625-t14452-kvm-ol8.zip
Smoke tests completed. 147 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

boring-cyborg bot added component:compute component:hyperv component:kvm component:orchestration component:primary-storage component:storage component:vmware labels Sep 12, 2025

sureshanaparti requested review from weizhouapache and Copilot September 12, 2025 13:47

sureshanaparti added the status:needs-testing label Sep 12, 2025

sureshanaparti requested a review from kiranchavala September 12, 2025 13:47

sureshanaparti added this to the 4.22.0 milestone Sep 12, 2025

Copilot AI reviewed Sep 12, 2025

View reviewed changes

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java Show resolved Hide resolved

sureshanaparti added this to Apache CloudStack 4.22.0 Sep 12, 2025

sureshanaparti moved this to In Progress in Apache CloudStack 4.22.0 Sep 12, 2025

weizhouapache approved these changes Sep 15, 2025

View reviewed changes

...datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java Outdated Show resolved Hide resolved

...chestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java Show resolved Hide resolved

sureshanaparti force-pushed the migrate-volume-improvements-to-bypass-sec-store branch from d646d67 to 89813ff Compare September 16, 2025 08:13

sureshanaparti marked this pull request as ready for review September 16, 2025 08:13

github-actions bot added the status:has-conflicts label Sep 17, 2025

sureshanaparti added 5 commits September 18, 2025 10:54

Migrate volume improvements, to bypass secondary storage when copy vo…

ef60abd

…lume between pools is allowed directly

Bypass secondary storage for copy volume between zone-wide pools and

faf94a6

- local storage on host in the same zone - cluser-wide pools in the same zone

Bypass secondary storage for volumes on ceph/rdb pool when the scope …

72617d1

…permits

Fix dest disk format while migrating volume from ceph/rbd to nfs, and…

0eeed55

… some code improvements

unit tests

b134782

sureshanaparti force-pushed the migrate-volume-improvements-to-bypass-sec-store branch from ad217ed to b134782 Compare September 18, 2025 05:24

github-actions bot removed the status:has-conflicts label Sep 18, 2025

sureshanaparti added 3 commits September 25, 2025 14:10

Update with proper message while migrate volume when target pool and …

11c2776

…offering type mismatches (both are not shared/local)

Consider host scope first during endpoint selection while copying bet…

6ae97f6

…ween primary storages

boring-cyborg bot added the component:api label Sep 26, 2025

Migrate volume improvements, to bypass secondary storage when copy volume between pools is allowed directly #11625

Are you sure you want to change the base?

Migrate volume improvements, to bypass secondary storage when copy volume between pools is allowed directly #11625

Uh oh!

Conversation

sureshanaparti commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Sep 12, 2025

Uh oh!

blueorangutan commented Sep 12, 2025

Uh oh!

codecov bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Sep 12, 2025

Uh oh!

sureshanaparti commented Sep 12, 2025

Uh oh!

blueorangutan commented Sep 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

blueorangutan commented Sep 12, 2025

Uh oh!

weizhouapache commented Sep 12, 2025

Uh oh!

blueorangutan commented Sep 12, 2025

Uh oh!

blueorangutan commented Sep 13, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Sep 16, 2025

Uh oh!

blueorangutan commented Sep 16, 2025

Uh oh!

blueorangutan commented Sep 17, 2025

Uh oh!

sureshanaparti commented Sep 17, 2025

Uh oh!

blueorangutan commented Sep 17, 2025

Uh oh!

blueorangutan commented Sep 17, 2025

Uh oh!

github-actions bot commented Sep 17, 2025

Uh oh!

github-actions bot commented Sep 17, 2025

Uh oh!

sureshanaparti commented Sep 18, 2025

Uh oh!

blueorangutan commented Sep 18, 2025

Uh oh!

blueorangutan commented Sep 18, 2025

Uh oh!

sureshanaparti commented Sep 23, 2025

Uh oh!

blueorangutan commented Sep 23, 2025

Uh oh!

blueorangutan commented Sep 24, 2025

Uh oh!

sureshanaparti commented Sep 26, 2025

Uh oh!

blueorangutan commented Sep 26, 2025

sureshanaparti commented Sep 12, 2025 •

edited

Loading

codecov bot commented Sep 12, 2025 •

edited

Loading