This PR enhances the existing CLVM implementation which was based on the deprecated CLVM technology which was based on corosync/pacemaker. With RHEL 7 having reached EOL, CLVM seems to be broken. CLVM supports RAW volumes on LVM , where as CLVM_NG support QCOW2 on LVM.
Further details: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modernized+CLVM%3A+Enhancements+and+CLVM_NG+support
NOTE: On testing - it was identified that incremental snapshots for clvm-ng do not work as expected. As of now it's been removed from scope. So, CLVM and CLVM_NG would only support full snapshots.
* add support for proper cleanup of snapshots and prevent vol snapshot of running vm
* remove snap vol restriction for sunning vms
* refactor clvm code
* add support for live migration
* add support for migrating lvm lock
* clvm deletion called explicitly
* made necessary changes to allow migration of lock and deletion of detached volumes
* fix create vol from snap and attach
* add support to revert snapshot for clvm
* add support to revert snapshot for clvm
* make zero fill configurable
* make setting non-dynamic & fix test
* fix locking at vol/vm creation
* fix revert snapshot format type and handle revert snapshot functionality for clvm
* 1. Create clvmlockmanager and move common code \n
2. handle attaching volumes to stopped VMs \n
3. Handle lock transfer when VM is started on another host
* add license
* remove command/answer classes from sonar coverage check
* add support for new gen clvm with template (qcow2) backing
* Add support for clvm_ng - which allows qcow2 on block storage , linked clones, etc
* fix test and use physical size + 50% of virtual size for backing file, while virtual size + pe for disk
* migrate clvm volumes as full clone and allow migration from clvm to nfs
* fix clvm_ng to nfs migration, and handle overhead calc
* support live migration from clvm_ng to nfs and vice-versa
* add support to migrate to and from clvm to nfs
* fix creation of volume on destination host during migration to clvm/clvm-ng
* support live vm migration between clvm -> clvm-ng (vice-versa), nfs -> clvm (vice-versa) and nfs->clvm-ng (vice-versa)
* add unit tests for clvm/clvm_ng operations
* Add support for incremental volume snapshots for clvm_ng
* prevent snapshot backup for incremental clvm_ng snaps, fix build failure, add unit tests
* fix lockhost on creation of volumes from snap and fix bitmap issue when migrating a vol with incremental snap
* restrict pre and post migration commands to only kvm hosts where vm has CLVM/CLVM-NG volumes
* evist lock tracking - use lvs command to get lock host than DB
* add test for pre/post migration
* Create a CLVM storage adaptor
* update existing clvm get stats method
* fix precommit check failure
* Apply suggestions from code review
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
* Apply suggestions from code review
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
* improve lock host retrieval logic and quicker retrival using db host as first check point and then fanning out
* add proper support for resizing of clvm_ng which calculated PE correctly for qcow2 metadata
* fallback to full snapshots for clvm-ng - incremental not supported in 4.23
* expunge volume detail of lock host on vm expunge
* if vmmigration with volume is done to the same clvm volume group, then dont do data transfer, just lock transfer and vm
* add clvm pools with deterministic uuid , so as to prevent adding the same pool twic
* added a small improvement to factor in a senario when lv is inactive on all hosts, could happen in storage outage issue
* address comment - extract common code for endpoint identification if clvm pool type
* Address comments - add early return guard to reduce indentation
* minor improvement - when migrating vm with volumes, if there's a failures, change the clvm vols to exclusive on source from shared, and on success, change dest vol to exclusive only for cross-pool migration
* cleanup unused code and tests for incremental snaps for clvmng and other cleanups
* allow storage browser to list lv in clvm, fix clvm shrink, overprovisioning factor isnt used for clvm pools - so set it to 1 and prevented display of provisioning type for clvm
* no need to have locktransfercommand to execute in sequence
* increase lv cmd timeouts to consider cluster load
---------
Co-authored-by: Pearl Dsilva <pearl1954@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
This PR introduces the initial implementation of Veeam integration support for KVM in CloudStack by adding a UHAPI-compatible server and image server components.
Veeam Backup & Replication interacts with virtualization platforms using its Universal Hypervisor API (UHAPI). To enable backup and restore workflows for CloudStack-managed KVM environments, this change introduces a UHAPI server that exposes CloudStack resources through a UHAPI-compatible interface.
In addition to the control plane APIs, an image server component is introduced to handle the data transfer operations required during backup and restore workflows.
The integration consists of two main components:
1. UHAPI Server (Control Plane) named CloudStack Veeam Control Service
A lightweight UHAPI server runs inside the CloudStack management server and exposes endpoints under:
/ovirt-engine
- /api - For APIs
- /sso - For authentication
- /services/pki-resource - For certificates
This server provides inventory discovery APIs required by Veeam and translates CloudStack resources into the structures expected by UHAPI.
The server:
- exposes infrastructure inventory
- handles authentication and session tokens
- maps CloudStack resources to UHAPI-compatible representations
2. Image Server (Data Plane) named CloudStack Image Service
A separate image server component is introduced to handle backup and restore data transfer operations.
This component:
- serves disk image data during backup
- receives image data during restore operations
- exposes endpoints used by Veeam worker components
- integrates with CloudStack storage to read and write VM disk data
The separation between both these components server ensures that:
- metadata APIs and control operations remain lightweight
- bulk image transfer operations are handled independently
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Abhisar Sinha <63767682+abh1sar@users.noreply.github.com>
Co-authored-by: abh1sar <abhisar.sinha@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
* Host HA code improvements
* Fix to not cancel VM HA items when Host HA is enabled & inspection in progress, and some code improvements
- When Host HA inspection in progress, the investigor returns the Host Status as Up which cancels the VM HA items
- Don't cancel the VM HA items, instead reschedule them to try again later
* Changes to consider Recovered/Available Host HA state along with the agent connection status to determine the Host HA inspection in progress or not, and some code improvements
* 4.22:
Fix issue when restoring backup after migration of volume (#12549)
Usage: Heartbeat should not schedule usage job when a job is already running (#12616)
Allow limit queries without random ordering (#12598)
engine/schema: fix cluster/zone settings with encrypted values (#12626)
Fix injection of preset variables into the JS interpreter (#12515)
Fix issue with multiple KVM Host entries in host table (#12589)
Add a Prometheus metric to track host certificate expiry (#12613)
ssvm: delete temp directory while deleting entity download url (#12562)
* fix that log sensitive infomation in cmd of script
* Remove unnecessary line break in Script.java
* Update utils/src/main/java/com/cloud/utils/script/Script.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor logging in Script class to simplify handling of sensitive arguments
* Improve command logging in Script class to include full command line when debugging
* Remove unused _passwordCommand flag from Script class to simplify code
* Update utils/src/main/java/com/cloud/utils/script/Script.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Remove unused import for KeyStoreUtils
* Update utils/src/main/java/com/cloud/utils/script/Script.java
---------
Co-authored-by: chenyoulong20g@ict.ac.cn <chenyoulong20g@ict.ac.cn>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: dahn <daan@onecht.net>
Co-authored-by: dahn <daan.hoogland@gmail.com>
* Sensitive information logged in SshHelper.sshExecute method
* Fix that Sensitive information logged in SshHelper.sshExecute method2
* Fix sensitive information handling in SshHelper and its tests
---------
Co-authored-by: chenyoulong20g@ict.ac.cn <chenyoulong20g@ict.ac.cn>
Bundling all hypervisor SystemVM templates in release packages simplifies installs but inflates build time and artifact size. This change enables downloading templates on demand when they’re not found after package installation. The download path is wired into both cloud-setup-management and the existing SystemVM template registration flow.
For connected or mirrored environments, a repository URL prefix can be provided to support air-gapped setups: pass --systemvm-templates-repository <URL-prefix> to cloud-setup-management, or set system.vm.templates.download.repository=<URL-prefix> in server.properties for post-setup registration.
If templates are already present (bundled or preseeded), behavior is unchanged and no download is attempted.
---------
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Pearl Dsilva
Bryan Lima
Vishesh
Abhishek Kumar
Suresh Kumar Anaparti
dahn
John Bampton
Manoj Kumar
Nicolas Vazquez
Daan Hoogland
Fabricio Duarte
Wei Zhou
Edward-x
Abhisar Sinha