Commit Graph

228 Commits

Author SHA1 Message Date
Abhishek Kumar adec5f439d
kvm: add ssvm storage nic null uri check during plug (#11557)
* kvm: add ssvm storage nic null uri check during plug

Fixes #11552

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-09-06 13:34:23 +02:00
Suresh Kumar Anaparti 5a90da3276
Fix for PowerFlex MDM configuration on host while preparing the SDC connection (#11458) 2025-08-18 11:57:54 +05:30
Bernardo De Marco Gonçalves ba2d70ab21
[KVM] CPU Features for System VMs (#10964)
* CPU features for System VMs

* Apply guest.cpu.features for System VMs
2025-08-15 20:02:50 +05:30
Vishesh bcd738caa6
Fix GPU discovery script to make it run with mdev for SR-IOV enabled devices (#11340) 2025-07-31 18:29:35 +05:30
Vishesh f6ad184ea2
Feature: Add support for GPU with KVM hosts (#11143)
This PR allows attaching of GPU devices via PCI, mdev or VF to an Instance for KVM.

It allows the operator to discover the GPU devices on the KVM host and create a Compute Offering with GPU support based on the available GPU devices on the host. Once the operator has created the Compute offering, it can be used by users to launch Instances with GPU devices.
2025-07-29 13:46:24 +05:30
Daan Hoogland b57994eeb0 Merge branch '4.20' 2025-07-16 14:05:19 +02:00
Suresh Kumar Anaparti c94f75c7ea
PowerFlex/ScaleIO - Wait after SDC service start/restart/stop, and retry to fetch SDC id/guid (#11099)
* [PowerFlex/ScaleIO] Added wait time after SDC service start/restart/stop, and retries to fetch SDC id/guid

* Added agent property 'powerflex.sdc.service.wait' for the time (in secs) to wait after SDC service start/restart/stop

* code improvements
2025-07-16 12:32:09 +05:30
João Jandre 53eb2c5b9b
File-based disk-only VM snapshot with KVM as hypervisor (#10632)
Co-authored-by: João Jandre <joao@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2025-07-16 08:54:07 +02:00
Suresh Kumar Anaparti 3220eb442a
PowerFlex/ScaleIO - MDM and host SDC connection enhancements (#11047)
* Cumulative enhancements fix for ScaleIO: MDM add/remove, Host prepare/unprepare, validate Storage Pool can be created in Agent.

- Implemented validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted
- Implemented Storage Pool validation by checking whether MDM addresses from configuration file and from memory (using CLI) matches, otherwise file ModifyStoragePool command.
- Introduced configuration key to apply timeout after making MDM changes for ScaleIO: powerflex.mdm.change.apply.timeout.ms (default 1000ms)
- Implemented logic to apply timeout after making MDM changes for ScaleIO in prepare and unprepare logic
- Added detection of MDM removal support via CLI
- If MDM removal support via CLI supported then use CLI, fall back to edit drv_cfg.txt and restart scini instead

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
2025-07-16 08:25:28 +02:00
Daan Hoogland 0dbd761fbb Merge branch '4.20' 2025-07-14 11:17:08 +02:00
Suresh Kumar Anaparti 80f46ad55d
[VMware to KVM Migration] Fix for converted instance npe issue when source vmware instance ovf is exported from management server (#11003) 2025-07-04 13:54:54 +05:30
Suresh Kumar Anaparti 572fc11a64
[PowerFlex] Add & Remove PowerFlex/ScaleIO MDMs for the storage SDC connections (#9903)
* Add & Remove PowerFlex/ScaleIO MDMs while preparing & unpreparing the storage SDC connections (instead of start & stop scini)

* Add/Remove MDM IP addresses during Host connection/disconnection to/from storage pool when powerflex.connect.on.demand is false

* unit test fixes

* Don't remove MDM IPs from SDC when any volumes mapped to SDC

* Don't remove MDM IPs when other pools of same ScaleIO/PowerFlex cluster are connected

* rebase fixes

* update changes, to not remove/disconnect MDMs on maintenance

* import fixes after rebase
2025-05-15 12:42:13 +05:30
João Jandre 6fdaf51ddc
KVM incremental snapshot feature (#9270)
* KVM incremental snapshot feature

* fix log

* fix merge issues

* fix creation of folder

* fix snapshot update

* Check for hypervisor type during parent search

* fix some small bugs

* fix tests

* Address reviews

* do not remove storPool snapshots

* add support for downloading diff snaps

* Add multiple zones support

* make copied snapshots have normal names

* address reviews

* Fix in progress

* continue fix

* Fix bulk delete

* change log to trace

* Start fix on multiple secondary storages for a single zone

* Fix multiple secondary storages for a single zone

* Fix tests

* fix log

* remove bitmaps when deleting snapshots

* minor fixes

* update sql to new file

* Fix merge issues

* Create new snap chain when changing configuration

* add verification

* Fix snapshot operation selector

* fix bitmap removal

* fix chain on different storages

* address reviews

* fix small issue

* fix test

---------

Co-authored-by: João Jandre <joao@scclouds.com.br>
2025-05-12 10:50:30 -03:00
Pearl Dsilva 1e5d133033 Merge branch '4.20' of https://github.com/apache/cloudstack 2025-05-12 13:12:09 +05:30
Pearl Dsilva a21f912be3 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-05-12 12:41:34 +05:30
Wei Zhou ea32a1a71a
server: fetch IP of VMs on L2 networks (#10431) 2025-05-09 10:19:30 +02:00
Wei Zhou fd74895ad0
New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) (#10514) 2025-05-02 09:15:03 +02:00
Pearl Dsilva 2df1ac5106 Merge branch '4.20' of https://github.com/apache/cloudstack 2025-04-28 12:15:48 +05:30
Wei Zhou 5d28e66b52
vTPM: support KVM and VMware (#10543)
* KVM: add Virtual TPM model and version

* KVM: add admin-only VM setting GUEST.CPU.MODE and GUEST.CPU.MODEL

* VMware: add vTPM

* vTPM: do not set Key due to 'Cannot add multiple devices using the same device key..'

* vTPM: add unit test testTpmModel

* engine/schema: remove user vm details for guest CPU mode/model

* vTPM: extra methods as Daan's requests

* vTPM: add unit tests in VmwareResourceTest

* vTPM: update unit tests in VmwareResourceTest

* vTPM: add unit test in LibvirtComputingResourceTest

* vTPM: use the default TPM version if an invalid version is passed

* vTPM: requires UEFI on vmware and do nothing if it is not enabled/disabled

* vTPM: let uses to add UEFI on vmware

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* vTPM: remove template details for guest CPU mode/model

* UI: boot vm from ISO into UEFI/SECURE mode

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-04-26 14:35:36 +05:30
Eric Kalendra 55c811547a
Update Mockito to 5.16.1 (#10686)
Dependency name change mockito-inline to mockito-core. Inline is now the default and the last version of mockito-inline released is 5.2.0.

assertj-core in user-authenticators/saml2 pulls in an incompatible version of byte-buddy and required an exclusion. Updating the version of assertj is left for a future PR.
The upgrade requires Java 11+, dropping support for Java 8. CloudStack documentation already says to use Java 11 and does not indicate that java 8 is supported.

Test classes using @RunWith(MockitoJUnitRunner.class) now get run in strict mode. Changes were made to tests where the stubbing intention was clear. In ManagementServerMaintenanceManagerImplTest there are 5 tests where the intention of the test is unclear. Each of the statements now use Mockito.lenient() to avoid the exception. Other cases in the tests follow a similar pattern.
Minor clean up.

Both @Spy and Mockito.spy( should not be used. Favored the annotation.
Both @RunWith(MockitoJUnitRunner.class) and MockitoAnnotations.openMocks(this); should not be used. Favored the annotation.
Unnecessary extends TestCase removed.
@InjectMocks and new in statement unnecessary. Removed new when issue presented.
Some of the Cmd classes like UpdateNetworkCmd have a type tree that includes fields of type Object. This appears to cause issues with injection, requiring that @Mock fields be available. This is where the following fields were added in multiple places:
Object job;
ResponseGenerator _responseGenerator;
Wrong number of parameters for Mockito.when in LibvirtRevertSnapshotCommandWrapperTest.java
2025-04-16 18:10:28 +05:30
Bryan Lima cb4848bc1a
Add support to RBD erasure code pools (#9808)
* Readd filename string on qemuimg create

* Remove empty object on the data pool details of storage pools with no data pool

* Only use the method createPhysicalDiskByLibVirt with RBD when the pool is of erasure code type. Also added javadoc for createPhysicalDisk method

* Change literal '/' string to File.separator

* Add support for erasure code pools

* Fix null on putAll
2025-04-02 08:19:00 -03:00
Abhishek Kumar 1c1dad977e Merge remote-tracking branch 'apache/4.20' 2025-03-06 09:55:27 +05:30
Pearl Dsilva 48ed5e2417 Merge branch '4.19' into 4.20 2025-03-04 09:17:16 -05:00
Rene Glover f0179855f5
add use of virsh domifaddr to get VM external DHCP IP (#10376)
* add use of virsh domifaddr to get VM external DHCP IP

* updates to modularize LibvirtGetVmIpAddressCommandWrapper per comments; added test cases to cover 90%+ scenarios

* updates to modularize LibvirtGetVmIpAddressCommandWrapper per comments; added test cases to cover 90%+ scenarios

* updates to modularize LibvirtGetVmIpAddressCommandWrapper per comments; added test cases to cover 90%+ scenarios
2025-02-28 10:13:53 +05:30
Daan Hoogland 0dcb8da03a Merge branch '4.20' 2025-02-12 16:54:05 +01:00
Daan Hoogland 4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
Nicolas Vazquez 4f604c00b6
Support virtio-blk root disk controller (#10229) 2025-02-10 15:03:07 +01:00
niyamsw 5df15a7aa6
KVM/s390x Support: Add support for KVM on s390x architecture (#10038)
Signed-off-by: Niyam Siwach <niyam@ibm.com>
Signed-off-by: Himanshu Mishra <Himanshu.Mishra2@ibm.com>
2025-01-29 17:34:16 +01:00
Daan Hoogland 34d2a3bc86 Merge branch '4.19' into 4.20 2025-01-24 17:01:42 +01:00
Nicolas Vazquez 7e295ec4e1
[KVM] Add watchdog model none to disable use of watchdogs on KVM agent (#10203) 2025-01-23 13:15:02 +01:00
Wei Zhou b186272f68
kvm: add SCSI controllers based on the number of virtio-SCSI disks (#9823)
According to libvirt code, the units per scsi controller is set to 7
therefore, we need to create scsi controller every 7 disks (including CDROM).

50cc7a0d9d/src/conf/domain_conf.h (L3007-L3008)

50cc7a0d9d/src/conf/domain_conf.c (L6701-L6704)
2025-01-22 14:00:02 +01:00
Abhishek Kumar bd488c4bba
server, plugin: enhance storage stats for IOPS (#10034)
Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned.
For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool.
For local storage pool, implementation has been added using iostat to return currently used IOPS.
StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-07 17:17:12 +05:30
Daan Hoogland b7f0aac519 Merge branch '4.19' into 4.20 2024-12-20 14:34:39 +01:00
Rene Peinthor a9587bfd2e
kvm-storage: provide isVMMigrate information to storage plugins (#10093)
Particular Linstor needs can use this information to only allow
dual volume access for live migration and not enable it in general,
which can and will lead to data corruption if for some reason
2 VMs get started on 2 different hosts.
2024-12-18 09:13:41 +01:00
Daan Hoogland da54234585 Merge branch '4.19' into 4.20.merge 2024-12-03 16:32:15 +01:00
Vishesh 37eec06166
Allow VMWare import via another host (#9787)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-11-28 17:50:08 +01:00
João Jandre b38ee63c48 Merge branch '4.19' 2024-11-13 10:47:24 -03:00
Rene Peinthor dfe4a67859
kvm: ref-count secondary storage pool usage (#9498)
If a secondary storage pool is used by e.g.
2 concurrent snapshot->template actions,
if the first action finished it removed the netfs mount
point for the other action.
Now the storage pools are usage ref-counted and will only
deleted if there are no more users.
2024-11-13 10:32:46 -03:00
João Jandre 8af08ddafb Merge branch '4.19' 2024-11-04 08:58:48 -03:00
Rene Peinthor a6cef7a78d
linstor/kvm: Workaround a qemu bug and IDE bus discard enabled. (#9859)
qemu has a bug versions prior 7.0 with discard enabled and using the IDE bus.
It would crash the qemu process and kill the virtual machine,
this is most noticeable on installing a windows guest from the
Windows ISO installer.
2024-11-04 08:46:40 -03:00
Fabricio Duarte b11897cdfa
Add logs to `LibvirtComputingResource`'s metrics collection process (#8511)
* Add logs to LibvirtComputingResource's metrics collecting process

* Apply Joao's suggestions

Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>

* Adjust some logs

* Print memory statistics log in one line

---------

Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-09-06 10:49:22 -03:00
Vishesh 35fd17c62e
Merge branch '4.19' 2024-07-17 17:43:40 +05:30
Suresh Kumar Anaparti 7f8f113709
Merge branch '4.18' into 4.19 2024-07-15 00:59:06 +05:30
Suresh Kumar Anaparti 877c4d9a64
LibvirtOvsFetchInterfaceCommandWrapperTest fix (test fails in mac) - skip it if no interfaces with eth and wl (#9382)
* test fix (fails in mac) - skip it if no interfaces with eth and wl

* review comments
2024-07-15 00:56:23 +05:30
Abhishek Kumar 3c9bc07d06 Merge remote-tracking branch 'apache/4.19' 2024-07-06 13:01:50 +05:30
Abhishek Kumar 2cf838cc1d framework/cluster: improve cluster service, integration API server
- mTLS implementation for cluster service communication
- Listen only on the specified cluster node IP address instead of all interfaces
- Validate incoming cluster service requests are from peer management servers based on the server's certificate dns name which can be through global config - ca.framework.cert.management.custom.san
- Hardening of KVM command wrapper script exeicution
- Improve API server integration port check
- cloudstack-management.default: don't have JMX configuration if not needed. JMX is used for instrumentation; users who need to use it should enable it explicitly

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-06 11:29:26 +05:30
Abhishek Kumar beebeed5e2 framework/cluster: improve cluster service and integration API service
- mTLS implementation for cluster service communication
- Listen only on the specified cluster node IP address instead of all interfaces
- Validate incoming cluster service requests are from peer management servers based on the server's certificate dns name which can be through global config - ca.framework.cert.management.custom.san
- Hardening of KVM command wrapper script execution
- Improve API server integration port check
- cloudstack-management.default: don't have JMX configuration if not needed. JMX is used for instrumentation; users who need to use it should enable it explicitly

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-05 19:05:06 +05:30
Vishesh 0ec7c72875
Merge branch '4.19' 2024-07-01 12:41:45 +05:30
Suresh Kumar Anaparti 2ca1b474bd
PowerFlex/ScaleIO SDC client connection improvements (#9268)
* Mitigation for non-scalable Powerflex/ScaleIO clients
- Added ScaleIOSDCManager to manage SDC connections, checks clients limit, prepare and unprepare SDC on the hosts.
- Added commands for prepare and unprepare storage clients to prepare/start and stop SDC service respectively on the hosts.
- Introduced config 'storage.pool.connected.clients.limit' at storage level for client limits, currently support for Powerflex only.

* tests issue fixed

* refactor / improvements

* lock with powerflex systemid while checking connections limit

* updated powerflex systemid lock to hold till sdc preparation

* Added custom stats support for storage pool, through listStoragePools API

* code improvements, and unit tests

* unit tests fixes

* Update config 'storage.pool.connected.clients.limit' to dynamic, and some improvements

* Stop SDC on host after migration if no volumes mapped to host

* Wait for SDC to connect after scini service start, and some log improvements

* Do not throw exception (log it) when SDC is not connected while revoking access for the powerflex volume

* some log improvements
2024-06-29 10:01:50 +05:30
Vishesh 90fe1d5fdc
Merge branch '4.19' 2024-06-29 03:35:24 +05:30
Suresh Kumar Anaparti 46f672563e
Improve migration of external VMware VMs into KVM cluster (#8815)
* Create/Export OVA file of the VM on external vCenter host, to temporary conversion location (NFS)

* Fixed ova issue on untar/extract ovf from ova file
"tar -xf" cmd on ova fails with "ovf: Not found in archive" while extracting ovf file

* Updated VMware to KVM instance migration using OVA

* Refactoring and cleanup

* test fixes

* Consider zone wide pools in the destination cluster for instance conversion

* Remove local storage pool support as temporary conversion location
- OVA export not possible as the pool is not accessible outside host, NFS pools are supported.

* cleanup unused code

* some improvements, and refactoring

* import nic unit tests

* vmware guru unit tests

* Separate clone VM and create template file for VMware migration
- Export OVA (of the cloned VM) to the conversion location takes time.
- Do any validations with cloned VM before creating the template (and fail early).
- Updated unit tests.

* Check conversion support on host before clone vm / create template on vmware (and fail early)

* minor code improvements

* Auto select the host with instance conversion capability

* Skip instance conversion supported response param for non-KVM hosts

* Show supported conversion hosts in the UI

* Skip persistence map update if network doesn't exist

* Added support to export OVA from KVM host, through ovftool (when installed in KVM host)

* Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadvmfiles', to be generic

* Updated hardcoded UI messages with message labels

* Updated UI to support importvm api param - forcemstodownloadvmfiles

* Improved instance conversion support checks on ubuntu hosts, and for windows guest vms

* Use OVF template (VM disks and spec files) for instance conversion from VMware, instead of OVA file
 - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

* OVF export tool parallel threads code improvements

* Updated 'convert.vmware.instance.to.kvm.timeout' config default value to 3 hrs

* Config values check & code improvements

* Updated import log, with time taken and vm details

* Support for parallel downloads of VMware VM disk files while exporting OVF from MS, and other changes below.
- Skip clone for powered off VMs
- Fixes to support standalone host (with its default datacenter)
- Some code improvements

* rebase fixes

* rebase fixes

* minor improvement

* code improvements - threads configuration, and api parameter changes to import vm files

* typo fix in error msg
2024-06-27 21:14:13 +05:30
Abhisar Sinha 4eb43651e2
Ability to specify NFS mount options while adding a primary storage and modify them on a pre-existing primary storage (#8947)
* Ability to specify NFS mount options while adding a primary storage and modify it later

* Pull 8947: Rename all occurrence of nfsopt to nfsMountOpt and added nfsMountOpts to ApiConstants

* Pull 8947: Refactor code - move into separate methods

* Pull 8947: CollectionsUtils.isNotEmpty and switch statement in LibvirtStoragePoolDef.java

* Pull 8947: UI - cancel maintainenace will remount the storage pool and apply the options

* Pull 8947: UI - moved edit NFS mount options to edit Primary Storage form

* Pull 8947: UI - moved 'NFS Mount Options' to below 'Type' in dataview

* Pull 8947: Fixed message in AddPrimaryStorage.vue

* Pull 8947: Convert _nfsmountOpts to Set in libvirtStoragePoolDef

* Pull 8947: Throw exception and log error if mount fails due to incorrect mount option

* Pull 8947: Added UT and moved integration test to component/maint

* Pull 8947: Review comments

* Pull 8947: Removed password from integration test

* Pull 8947: move details allocation to inside the if loop in getStoragePoolNFSMountOpts

* Pull 8947: Fixed a bug in AddPrimaryStorage.vue

* Pull 8947: Pool should remain in maintenance mode if mount fails

* Pull 8947: Removed password from integration test

* Pull 8947: Added UT

* Pull 8875: Fixed a bug in CloudStackPrimaryDataStoreLifeCycleImplTest

* Pull 8875: Fixed a bug in LibvirtStoragePoolDefTest

* Pull 8947: minor code restructuring

* Pull 8947 : added some ut for coverage

* Fix LibvirtStorageAdapterTest UT
2024-06-25 23:45:35 +05:30
Vishesh 3923f80c22
Merge branch '4.19' 2024-06-25 18:53:57 +05:30
slavkap 8b07b66f14
Fix volume snapshot of encrypted NFS/StorPool volume (#8873)
* Fix volume snapshot of encrypted NFS/StorPool volume

* remove comments

* removed invoking the real qemu convert command

* fix UnsatisfiedLink error in unit tests

* addressed comments

extracted method
2024-06-24 13:09:21 +05:30
Daan Hoogland cb9b3134f7 Merge branch '4.19' 2024-06-14 10:30:10 +02:00
Vishesh 74f5e52e6e
Fix unit test failure (#9238) 2024-06-13 16:06:35 +05:30
Vishesh 6fe835e118
Merge branch '4.19' 2024-06-13 11:57:55 +05:30
Wei Zhou b2ef53b8a2
kvm: replace ISO path in vm XML configuration during vm migration (#9212)
* kvm: replace ISO path in vm XML configuration during vm migration

* Update 9212: address comments

* kvm: fix vm migration if there are multiple image stores
2024-06-12 16:01:23 +02:00
Wei Zhou 5433e775e5
New feature: Implicit host tags (#8929)
* Merge two HostTagVO and HostTagDaoImpl

* Implicit host tags

* PR8929: add since

* Update variable names

* Update 8929: add unit test in LibvirtComputingResourceTest

* Update 8929: add explicithosttags in response

* Update 8929 UI: Update explicit host tags

* Update 8929: remove host tags and change labels on UI

* Update 8929: update host_view to use explicit_host_tags.is_tag_a_rule

* Update: ui polish for host tags

* Update 8929: fix UI error if no host tags
2024-05-30 17:21:13 +05:30
Daan Hoogland e61f3bae4d Merge branch '4.19' 2024-04-29 11:37:40 +02:00
Rohit Yadav 3de1f8b4ba Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-29 13:44:34 +05:30
João Jandre cec6ade257
change live migration API used on kvm (#8952) 2024-04-25 09:35:25 +02:00
Marcus Sorensen 27124c1031
Add ability to set cpu.threadspercore similar to existing cpu.corespersocket (#8850)
* Add ability to set cpu.threadspercore similar to existing cpu.corespersocket

* add cpu.threadspercore to VM and template detail options

* Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* add vm detail for KVM

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-04-24 18:01:21 +05:30
Daan Hoogland a358c9a410 Merge release branch 4.19 to main
* 4.19:
  New feature: Import/Unamange DATA volume from storage pool (#8808)
2024-04-23 16:13:16 +02:00
Wei Zhou 0b857def68
New feature: Import/Unamange DATA volume from storage pool (#8808) 2024-04-23 16:05:59 +02:00
Suresh Kumar Anaparti dfd5158d67
Allow kvm storage plugin to customize diskdef, add geometry (#8839)
* Allow kvm storage plugin to customize diskdef, add geometry

* formatting update

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2024-04-18 18:21:17 +05:30
Vishesh 19f79b1d94
Merge branch '4.19' 2024-04-12 16:31:40 +05:30
Suresh Kumar Anaparti d3e020a545
Mark libvirt events experimental, add properties flag (#8825)
* Mark libvirt events experimental, add properties flag

* unit test fixes

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2024-04-11 17:06:33 +05:30
Marcus Sorensen 92bfb4dcaa
Support KVM storage implementations controlling logical/physical block io size (#8724)
* Support KVM storage implementations controlling logical/physical block io size

* Support custom block size during disk attach

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2024-03-15 14:06:25 +05:30
nvazquez db564b12b6
Merge branch '4.19' 2024-03-14 22:46:48 -03:00
Wei Zhou a7ec8738a2
kvm: fix NPE while import KVM VMs from other hosts (#8720) 2024-03-04 09:46:28 +01:00
Abhishek Kumar b29ec2bf12 Merge remote-tracking branch 'apache/4.19' 2024-03-01 17:40:58 +05:30
Harikrishna c462be1412
New API "checkVolume" to check and repair any leaks or issues reported by qemu-img check (#8577)
* Introduced a new API checkVolumeAndRepair that allows users or admins to check and repair if any leaks observed.
Currently this is supported only for KVM

* some fixes

* Added unit tests

* addressed review comments

* add repair volume while granting access

* Changed repair parameter to accept both leaks/all

* Introduced new global setting volume.check.and.repair.before.use to do volume check and repair before VM start or volume attach operations

* Added volume check and repair changes only during VM start and volume attach operations

* Refactored the names to look similar across the code

* Some code fixes

* remove unused code

* Renamed repair values

* Fixed unit tests

* changed version

* Address review comments

* Code refactored

* used volume name in logs

* Changed the API to Async and the setting scope to storage pool

* Fixed exit value handling with check volume command

* Fixed storage scope to the setting

* Fix volume format issues

* Refactored the log messages

* Fix formatting
2024-02-29 14:41:49 +05:30
João Jandre 49cecaed06
Normalize loggers and upgrade log4j 1.2 to log4j 2.19 (#7131)
* Normalize logs

All classes that could have their loggers inherited from their fathers had their own loggers deleted;
Most loggers didn't have to be static, so most of them were normalized so that they wouldn't be;
All loggers are protected now;
Static logger's name are now 'LOGGER';
Non-static logger's name are now 'logger';
New class DbUpgradeAbstractImpl created so that all Upgraders extend it and inherit its logger

* Upgrade log4j

* fix errors caused by the merge

* Refactor cglibThrowableRenderer functionality to log4j2 and upgrade the last configuration files

* fix sonarcloud bug

* Fix errors caused by merge, remove some unused loggers, and rename a variable that was mistakenly renamed on the normalization commit

* Readd snmpTrapAppender, remove TestAppender

* Regenerate changes

* regenerate changes

* refactor last custom appender

* fix systemvm configuration xml

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* Fix utils pom

* fix some tests

* regenerate changes

* Fix jar being printed on exception

* fix logging in system VMs, fix commands not having log4j2 classpath.

* regenerate changes

* Fix some unwanted renomeations

* fix end of file

* regenerate changes

* regenerate changes

* fix merge error

* regenerate changes

* fix tests

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* readd reload4j to tungsten as juniper depends on it

* Regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* re-add reload4j dependency to network-contrail, as juniper depends on it

* regenerate changes

* regenerate changes

* regenerate changes

* fix typo

* regenerate changes

* regenerate changes

* Fix end of files

* regenerate changes

* add logj42 to cloud-utils-SHADED.jar

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* fix some tests

* Regenerate changes

* Regenerate changes

* fix test

* Regenerate changes

* Regenerate changes
2024-02-08 09:55:41 -03:00
Vishesh 399bd0a067
Upgrade to mockito 4 and handle Mockito deprecations (#8427) 2024-02-06 14:20:37 +01:00
Suresh Kumar Anaparti 8ea9fc911d
StoragePoolType as class (#8544)
* StoragePoolType as a class

* Fix agent side StoragePoolType enum to class

* Handle StoragePoolType for StoragePoolJoinVO

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fix UserVMJoinVO for StoragePoolType

* fixed missing imports

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fixed equals for the enum.

* removed not needed try/catch for prepareAttribute

* Added license to the file.

* Implemented "supportsPhysicalDiskCopy" for storage adaptor.

Co-authored-by: mprokopchuk <mprokopchuk@apple.com>

* Add javadoc to StoragePoolType class

* Add unit test for StoragePoolType comparisons

* StoragePoolType "==" and ".equals()" fix.

* Fix StoragePoolType for FiberChannelAdapter

* Fix for abstract storage adaptor set up issue

* review comments

* Pass StoragePoolType object for poolType dao attribute

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@gmail.com>
2024-02-05 13:27:15 +05:30
Nicolas Vazquez 59e78cbc45
Fix KVM unmanage disks path (#8483)
This PR fixes the volumes path on KVM import unmanaged instances

Fixes: #8479
2024-01-11 14:45:57 +05:30
Abhishek Kumar 82f7abddb3 Merge remote-tracking branch 'apache/4.18' 2023-12-13 11:24:15 +05:30
Bryan Lima 3bb318bab9
kvm: Add support for cgroupv2 (#8252)
1. Problem description

In Apache CloudStack (ACS), when a VM is deployed in a host with the KVM hypervisor, an XML file is created in the assigned host, which has a property shares that defines the weight of the VM to access the host CPU. The value of this property has no unit, and it is a relative measure to calculate how much CPU a given VM will have in the host. However, this value has a limit, which depends on the version of cgroup utilized by the host's kernel. The problem lies at the range value of shares that varies between both versions: [2, 264144] for cgroups version 1; and [1, 10000] for cgroups version 2. Currently, ACS calculates the value of shares using Equation 1, presented below, where CPU is the number of cores and speed is the CPU frequency; both specified in the VM's compute offering. Therefore, if a compute offering has, for example, 6 cores at 2 GHz, the shares value will be 12000 and an exception will be thrown by libvirt if the host utilizes cgroup v2. The second version is becoming the default one in current Linux distributions; thus, it is necessary to address this limitation.

    Equation 1
    shares = CPU * speed

Fixes: #6744
2. Proposed changes

To address the problem described, we propose to apply a scale conversion considering the max shares of the host. Using the same formula currently utilized by ACS, it is possible to calculate the maximum shares of a VM for a given host. In other words, using the number of cores and the nominal speed of the host's CPU as the upper limit of shares allowed to a VM. Then, this value will be scaled to the allowed interval of [1, 10000] of cgroup v2 by using a linear scale conversion.

The VM shares would be calculated as Equation 2, presented below, where VM requested shares is the requested shares value calculated using Equation 1, cgroup upper limit is fixed with a value of 10000 (cgroups v2 upper limit), and host max shares is the maximum shares value of the host, calculated using Equation 1. Using Equation 2, the only case where a VM passes the cgroup v2 limit is when the user requests more resources than the host has, which is not possible with the current implementation of ACS.

    Equation 2
    shares = (VM requested shares * cgroup upper limit)/host max shares

To implement the proposal, the following APIs will be updated: deployVirtualMachine, migrateVirtualMachine and scaleVirtualMachine. When a VM is being deployed, a new verification will be added to find a suitable host. The max shares of each host will be calculated, and the VM calculated shares will be verified if it does not surpass the host's value. Likewise, the migration of VMs will have a similar new verification. Lastly, the scale of VMs will also have the same verification for the VM's host.

To determine the max shares of a given host, we will use the same equation currently used in ACS for calculating the shares of VMs, presented in Section 1. When Equation 1 is used to determine the maximum shares of a host, CPU is the number of cores of the host, and speed is the nominal CPU speed, i.e., considering the CPU's base frequency.

It is important to note that these changes are only for hosts with the KVM hypervisor using cgroup v2 for now.
2023-12-13 10:51:24 +05:30
Nicolas Vazquez 371ad9f55b
New Feature: Import VMware VMs into KVM (#7881)
This PR adds the capability in CloudStack to convert VMware Instances disk(s) to KVM using virt-v2v and import them as CloudStack instances. It enables CloudStack operators to import VMware instances from vSphere into a KVM cluster managed by CloudStack. vSphere/VMware setup might be managed by CloudStack or be a standalone setup.

    CloudStack will let the administrator select a VM from an existing VMware vCenter in the CloudStack environment or external vCenter requesting vCenter IP, Datacenter name and credentials.
    The migrated VM will be imported as a KVM instance
    The migration is done through virt-v2v: https://access.redhat.com/articles/1351473, https://www.ovirt.org/develop/release-management/features/virt/virt-v2v-integration.html
    The migration process timeout can be set by the setting convert.instance.process.timeout
    Before attempting the virt-v2v migration, CloudStack will create a clone of the source VM on VMware. The clone VM will be removed after the registration process finishes.
    CloudStack will delegate the migration action to a KVM host and the host will attempt to migrate the VM invoking virt-v2v. In case the guest OS is not supported then CloudStack will handle the error operation as a failure
    The migration process using virt-v2v may not be a fast process
    CloudStack will not perform any check about the guest OS compatibility for the virt-v2v library as indicated on: https://access.redhat.com/articles/1351473.
2023-12-07 12:59:56 +05:30
slavkap 2bb182c3e1
KVM Host HA enhancement for StorPool storage (#8045)
Extending the current functionality of KVM Host HA for the StorPool storage plugin and the option for easy integration for the rest of the storage plugins to support Host HA

This extension works like the current NFS storage implementation. It allows it to be used simultaneously with NFS and StorPool storage or only with StorPool primary storage.

If it is used with different primary storages like NFS and StorPool, and one of the health checks fails for storage, there is an option to report the failure to the management with the global config kvm.ha.fence.on.storage.heartbeat.failure. By default this option is disabled when enabled the Host HA service will continue with the checks on the host and eventually will fence the host
2023-11-04 12:35:37 +05:30
Daan Hoogland 72cf9740f9 Merge branch '4.18' 2023-10-06 13:50:29 +02:00
Daniel Augusto Veronezi Salvador 9b8eaeea78
Fix: Convert volume to another directory instead of copying it while taking volume snapshots on KVM (#8041) 2023-10-06 09:47:34 +02:00
Marcus Sorensen 82b981854b
KVM Agent config to reserve dom0 CPUs (#7987)
This PR allows an admin to reserve some hypervisor host CPUs for system use. Another way to think of it is limiting the number of CPUs allocatable to VMs. This can be useful if the admin wants to do other things with the hypervisor's CPU, for example reserve some cores for running hyperconverged storage processes.

Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-10-06 10:50:18 +05:30
Marcus Sorensen 28c4be1cf2
Fix style for LibvirtComputingResource variable names and its dependencies (#7991)
* Fix style for LibvirtComputingResource variable names and its dependencies

* More variable name fixes

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-09-27 12:38:25 +05:30
Vishesh 84e770bf95
Remove powermock from pom.xml (#8000) 2023-09-26 13:31:56 +02:00
Wei Zhou 45616aaf61 Merge remote-tracking branch 'origin/4.18' 2023-09-14 14:00:01 +02:00
Marcus Sorensen f049d4d189
Increase reserve on ScaleIO disk formatting for fragmentation (#7955)
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-09-14 16:43:16 +05:30
Wei Zhou c683de4a55
kvm: fix unit test LibvirtReplugNicCommandWrapperTest (#7908) 2023-08-24 21:41:04 +05:30
Vishesh 82f9106633
Remove powermock from kvm hypervisor plugin (#7818)
* Remove powermock from kvm hypervisor plugin

* Add @RunWith

* Remove unnecessary mocks

* Remove commented code
2023-08-10 09:04:36 +02:00
Daan Hoogland eb31e3d795 Merge release branch 4.18 to main
* 4.18:
  Allow KVM overcommit to work without reducing minimum VM memory when vm ballooning is disabled (#7810)
2023-08-06 10:41:00 +02:00
Rohit Yadav dc5e4f3ec6
Allow KVM overcommit to work without reducing minimum VM memory when vm ballooning is disabled (#7810)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-08-06 10:39:14 +02:00
Daan Hoogland d51d8a4a13 Merge release branch 4.18 to main
* 4.18:
  UI: Filter templates by zone and hypervisor type when reinstall a VM (#7739)
  KVM: fix SSVM starting when overprovisioning memory (#7663)
  pom.xml: add property project.systemvm.template.location (#7706)
  cloudutils: fix adding rocky9 host failure due to missing /etc/sysconfig/libvirtd (#7779)
  server: get id from persisted object ReservationVO (#7785)
  search in (too) large result sets (#7766)
  ui: fix 404 error when list volumes of system vms (#7772)
  packaging: install tzdata-java on centos7/centos8 (#7768)
2023-07-31 09:04:44 +02:00
dahn d127d7939d
KVM: fix SSVM starting when overprovisioning memory (#7663) 2023-07-28 11:23:30 +02:00
Vishesh 594c70dde0
Sync precommit config from main (#7732)
Co-authored-by: John Bampton <jbampton@users.noreply.github.com>
Co-authored-by: dahn <daan@onecht.net>
2023-07-07 11:18:16 +02:00
Wei Zhou 09a4a252d7 Merge remote-tracking branch 'apache/4.18' into HEAD 2023-06-21 15:08:56 +02:00
Harikrishna 40cc10a73d
Allow volume migrations in ScaleIO within and across ScaleIO storage clusters (#7408)
* Live storage migration of volume in scaleIO within same storage scaleio cluster

* Added migrate command

* Recent changes of migration across clusters

* Fixed uuid

* recent changes

* Pivot changes

* working blockcopy api in libvirt

* Checking block copy status

* Formatting code

* Fixed failures

* code refactoring and some changes

* Removed unused methods

* removed unused imports

* Unit tests to check if volume belongs to same or different storage scaleio cluster

* Unit tests for volume livemigration in ScaleIOPrimaryDataStoreDriver

* Fixed offline volume migration case and allowed encrypted volume migration

* Added more integration tests

* Support for migration of encrypted volumes across different scaleio clusters

* Fix UI notifications for migrate volume

* Data volume offline migration: save encryption details to destination volume entry

* Offline storage migration for scaleio encrypted volumes

* Allow multiple Volumes to be migrated with migrateVirtualMachineWithVolume API

* Removed unused unittests

* Removed duplicate keys in migrate volume vue file

* Fix Unit tests

* Add volume secrets if does not exists during volume migrations. secrets are getting cleared on package upgrades.

* Fix secret UUID for encrypted volume migration

* Added a null check for secret before removing

* Added more unit tests

* Fixed passphrase check

* Add image options to the encypted volume conversion
2023-06-21 11:57:05 +05:30
Wei Zhou 9d46df57f2
kvm: add vm setting for nic multiqueue number and packed virtqueues (#7333)
This PR adds two vm setting for user vms on KVM

- nic multiqueue number
- packed virtqueues enabled . optional are true and false (false by default). It requires qemu>=4.2.0 and libvirt >=6.3.0

Tested ok on ubuntu 22 and rocky 8.4
2023-05-09 15:19:26 +05:30
Rohit Yadav a2561df25b Merge remote-tracking branch 'origin/4.18' 2023-05-08 12:57:38 +05:30
Marcus Sorensen ec0f8bddf6
Support local storage live migration for direct download templates (#7453)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-05-04 17:37:58 -03:00