This PR enhances the existing CLVM implementation which was based on the deprecated CLVM technology which was based on corosync/pacemaker. With RHEL 7 having reached EOL, CLVM seems to be broken. CLVM supports RAW volumes on LVM , where as CLVM_NG support QCOW2 on LVM.
Further details: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modernized+CLVM%3A+Enhancements+and+CLVM_NG+support
NOTE: On testing - it was identified that incremental snapshots for clvm-ng do not work as expected. As of now it's been removed from scope. So, CLVM and CLVM_NG would only support full snapshots.
* add support for proper cleanup of snapshots and prevent vol snapshot of running vm
* remove snap vol restriction for sunning vms
* refactor clvm code
* add support for live migration
* add support for migrating lvm lock
* clvm deletion called explicitly
* made necessary changes to allow migration of lock and deletion of detached volumes
* fix create vol from snap and attach
* add support to revert snapshot for clvm
* add support to revert snapshot for clvm
* make zero fill configurable
* make setting non-dynamic & fix test
* fix locking at vol/vm creation
* fix revert snapshot format type and handle revert snapshot functionality for clvm
* 1. Create clvmlockmanager and move common code \n
2. handle attaching volumes to stopped VMs \n
3. Handle lock transfer when VM is started on another host
* add license
* remove command/answer classes from sonar coverage check
* add support for new gen clvm with template (qcow2) backing
* Add support for clvm_ng - which allows qcow2 on block storage , linked clones, etc
* fix test and use physical size + 50% of virtual size for backing file, while virtual size + pe for disk
* migrate clvm volumes as full clone and allow migration from clvm to nfs
* fix clvm_ng to nfs migration, and handle overhead calc
* support live migration from clvm_ng to nfs and vice-versa
* add support to migrate to and from clvm to nfs
* fix creation of volume on destination host during migration to clvm/clvm-ng
* support live vm migration between clvm -> clvm-ng (vice-versa), nfs -> clvm (vice-versa) and nfs->clvm-ng (vice-versa)
* add unit tests for clvm/clvm_ng operations
* Add support for incremental volume snapshots for clvm_ng
* prevent snapshot backup for incremental clvm_ng snaps, fix build failure, add unit tests
* fix lockhost on creation of volumes from snap and fix bitmap issue when migrating a vol with incremental snap
* restrict pre and post migration commands to only kvm hosts where vm has CLVM/CLVM-NG volumes
* evist lock tracking - use lvs command to get lock host than DB
* add test for pre/post migration
* Create a CLVM storage adaptor
* update existing clvm get stats method
* fix precommit check failure
* Apply suggestions from code review
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
* Apply suggestions from code review
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
* improve lock host retrieval logic and quicker retrival using db host as first check point and then fanning out
* add proper support for resizing of clvm_ng which calculated PE correctly for qcow2 metadata
* fallback to full snapshots for clvm-ng - incremental not supported in 4.23
* expunge volume detail of lock host on vm expunge
* if vmmigration with volume is done to the same clvm volume group, then dont do data transfer, just lock transfer and vm
* add clvm pools with deterministic uuid , so as to prevent adding the same pool twic
* added a small improvement to factor in a senario when lv is inactive on all hosts, could happen in storage outage issue
* address comment - extract common code for endpoint identification if clvm pool type
* Address comments - add early return guard to reduce indentation
* minor improvement - when migrating vm with volumes, if there's a failures, change the clvm vols to exclusive on source from shared, and on success, change dest vol to exclusive only for cross-pool migration
* cleanup unused code and tests for incremental snaps for clvmng and other cleanups
* allow storage browser to list lv in clvm, fix clvm shrink, overprovisioning factor isnt used for clvm pools - so set it to 1 and prevented display of provisioning type for clvm
* no need to have locktransfercommand to execute in sequence
* increase lv cmd timeouts to consider cluster load
---------
Co-authored-by: Pearl Dsilva <pearl1954@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
* 4.22:
VM Deployment using snapshot in new zone (#13178)
Change exception treatment on incremental snapshot wait (#12665)
Move checkRoleEscalation outside DB transaction in createAccount (#13044)
Fix/flasharray delete rename destroy patch conflict (#13049)
Fix VPC network offerings listing in isolated network creation form (#12645)
systemvm: accept ipv6 established/related return traffic (#13173)
update debian change log
Updating pom.xml version numbers for release 4.22.2.0-SNAPSHOT
Updating pom.xml version numbers for release 4.22.1.0
Update suse15 packaging spec, use qemu-ovmf-x86_64 package instead of edk2-ovmf for agent (#13133)
Change disk-only VM snapshot removal message (#11182)
Update mysql java connector version to 8.4.0 (matching version for MySQL 8.4) (#12640)
adaptive: honor user-provided capacityBytes when provider stats are unavailable (#13059)
Flexibilize public IP selection (#11076)
* 4.20:
Fix/flasharray delete rename destroy patch conflict (#13049)
Fix VPC network offerings listing in isolated network creation form (#12645)
Update mysql java connector version to 8.4.0 (matching version for MySQL 8.4) (#12640)
adaptive: honor user-provided capacityBytes when provider stats are unavailable (#13059)
Flexibilize public IP selection (#11076)
* Host HA code improvements
* Fix to not cancel VM HA items when Host HA is enabled & inspection in progress, and some code improvements
- When Host HA inspection in progress, the investigor returns the Host Status as Up which cancels the VM HA items
- Don't cancel the VM HA items, instead reschedule them to try again later
* Changes to consider Recovered/Available Host HA state along with the agent connection status to determine the Host HA inspection in progress or not, and some code improvements
* Linstor: fix create volume from snapshot on primary storage
When creating a volume from a snapshot on Linstor primary storage
(with lin.backup.snapshots=false), the operation fails with:
"Only the following image types are currently supported: VHD, OVA,
QCOW2, RAW (for PowerFlex and FiberChannel)"
Root cause: the Linstor driver does not handle SNAPSHOT -> VOLUME in
its canCopy()/copyAsync() methods. This causes DataMotionServiceImpl
to fall through to StorageSystemDataMotionStrategy (selected because
Linstor advertises STORAGE_SYSTEM_SNAPSHOT=true). That strategy's
verifyFormatWithPoolType() rejects RAW format for Linstor pools,
since RAW is only allowed for PowerFlex and FiberChannel.
Additionally, VolumeOrchestrator.createVolumeFromSnapshot() attempts
to back up the snapshot to secondary storage when the storage plugin
does not advertise CAN_CREATE_TEMPLATE_FROM_SNAPSHOT. This backup
fails because the snapshot only exists on Linstor primary storage.
Fix:
- Add CAN_CREATE_TEMPLATE_FROM_SNAPSHOT capability so the
orchestrator skips the backup-to-secondary path
- Add canCopySnapshotToVolumeCond() to match SNAPSHOT -> VOLUME
when both are on the same Linstor primary store
- Wire it into canCopy() to intercept at DataMotionServiceImpl
before strategy selection, bypassing StorageSystemDataMotionStrategy
- Implement copySnapshotToVolume() which delegates to the existing
createResourceFromSnapshot() for native Linstor snapshot restore
This follows the same pattern used by the StorPool plugin, which
handles SNAPSHOT -> VOLUME directly in its driver rather than going
through StorageSystemDataMotionStrategy.
Tested on CloudStack 4.22 with Linstor LVM_THIN storage, creating
a volume from a 1TB CNPG Postgres database snapshot. Volume creates
successfully with correct path and deletes cleanly.
* Let CloudRuntimeException propagate from copySnapshotToVolume
Remove try/catch in copySnapshotToVolume so that CloudRuntimeException
from createResourceFromSnapshot propagates to the caller, ensuring
CloudStack properly notices and reports the failure.
* Fix CAN_CREATE_TEMPLATE_FROM_SNAPSHOT breaking template creation
Setting CAN_CREATE_TEMPLATE_FROM_SNAPSHOT unconditionally to true
caused createTemplate from snapshot to take the StorPool-specific
code path in TemplateManagerImpl, which sends a CopyCommand to a
system VM that Linstor cannot handle.
Fix: make CAN_CREATE_TEMPLATE_FROM_SNAPSHOT conditional on the same
flag as STORAGE_SYSTEM_SNAPSHOT (!BackupSnapshots). When snapshots
are backed up to secondary (the default), the old template creation
flow works. When snapshots stay on primary, the direct path is used.
Also fix checkstyle: remove unused DataObject import in test.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* PowerFlex/ScaleIO client initialization, authentication and command execution improvements
* Migrate VM with volume not supported yet for PowerFlex/ScaleIO
* review changes
* Linstor: Refactor resource creation methods to LinstorUtil
Move reusable methods from LinstorPrimaryDataStoreDriverImpl to LinstorUtil
to enable sharing with other components:
- logLinstorAnswer, logLinstorAnswers, checkLinstorAnswersThrow
- getRscGrp, getEncryptedLayerList, applyQoSSettings
- createResourceBase, createResource, spawnResource
- canShareTemplateForResourceGroup, foundShareableTemplate
Add LIN_PROP_DRBDOPT_EXACT_SIZE constant and exactSize parameter support
for DRBD exact-size property handling during resource creation.
* Linstor: Add LinstorDataMotionStrategy for VM live migration
Implement DataMotionStrategy for live migration of VMs with volumes on
Linstor or other primary storage.
Key features:
- Support live migration with storage from other primary storages
- Preserve DRBD exact-size property during migration
* 4.22:
fix install path for systemvm templates when introducing new sec storage (#11605)
fix Sensitive Data Exposure Through Exception Logging in OVM Hypervis… (#12032)
Fix snapshot physical size after migration (#12166)
ConfigDrive: use file absolute path instead of canonical path to create ISO (#11623)
Add log for null templateVO (#12406)
snapshot: fix listSnapshots for volume which got delete and whose storage pool got deleted (#12433)
Notify user if template upgrade is not required (#12483)
Fix: proper permissions for systemvm template registrations on hardened systems (#12098)
Allow modification of user vm details if user.vm.readonly.details is empty (#10456)
NPE fix while deleting storage pool when pool has detached volumes (#12451)
* NPE fix while deleting storage pool when pool has detached volumes
* review
* unit tests
* Added log for volumes not attached to any VMs
* update filter, log and test
* updated volume dao method names returning non destroyed volumes
* build fix
---------
Co-authored-by: dahn <daan@onecht.net>
This PR aligns the use of terminology, renaming VM / virtual machine references to 'Instance' and also capitalising the terms Templates, Network, Snapshot, User, Account in CloudStack APIs, error and log messages, events, tooltips, etc. Many typos, grammar and spelling mistakes were fixed, also terms like IPv4, VPN, VPC, etc. were properly capitalised. Some error messages were cleaned for better readability. The test cases, expecting some exception strings were adjusted accordingly.
Here is the wiki page, describing the changes in details:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Object+Naming+and+Title+Case+Convention
---------
Co-authored-by: Manoj Kumar <manojkr.itbhu@gmail.com>
Co-authored-by: Harikrishna <harikrishna.patnala@gmail.com>
Pearl Dsilva
James Peru Mmbono
Fabricio Duarte
Eugenio Grosso
Suresh Kumar Anaparti
Rene Peinthor
Sergiy Kukunin
Daan Hoogland
sandeeplocharla
Abhisar Sinha
poddm
Wei Zhou
owsferraro
Abhishek Kumar
vladimirpetrov
dahn
Harikrishna Patnala
John Bampton