Commit Graph

2625 Commits

Author SHA1 Message Date
DK101010 1d17d7cfbc
Add internal cs name to vm during the ingest (#5242)
* add internal cs name to vm during the ingest

* Update server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* remove lang.StringUtils

* Update vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/HostMO.java

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

* Update vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/HostMO.java

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-08-13 16:38:52 -03:00
DK101010 1bfb2f9fb1
disable hot add memory and cpu via vm settings (#4630)
* disable hot add memory and cpu via vm settings

* add alternative implementation for hot add memory and cpu

* add log entry

* Modify and add log entry for hotadd

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2021-08-13 11:54:06 -03:00
Abhishek Kumar 6581a1bccd
test/vmware: add live migratevmwithvolume test and fix (#5289)
Co-authored-by: Gabriel Brascher <gabriel@apache.org>
2021-08-11 13:52:05 +02:00
davidjumani cb1078aa20
Merge remote-tracking branch 'apache/4.15' into main 2021-08-09 15:10:51 +05:30
Abhishek Kumar 1ccb42017f
server: skip zone check for PERHOST iso during attachIso (#5270)
* server: skip zone check for PERHOST iso during attachIso

Hypervisor tools ISO - vmware-toools.iso, xs-tools.iso are marked as PERHOST in DB. They are active but not downloaded to the secondary storages and hence no template-zone entry.
Skips the template-zone check for such templates.

Fixes #5265

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* inverted check

* use constants in TemplateManager

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-08-09 14:02:25 +05:30
Daniel Augusto Veronezi Salvador 82df04ecc8
Improve HA logs (#5241)
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-07-30 21:13:16 +02:00
Daniel Augusto Veronezi Salvador 7b752c3077
Externalize KVM Agent storage's timeout configuration (#5239)
* Externalize KVM Agent storage's timeout configuration

* Address @nvazquez review

* Add empty line at the end of the agent.properties file

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-07-28 15:45:27 +02:00
SadiJr eff2da2518
Refactor and improvements for method com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVMFromSpec() (#5149)
* Refactor method createVMFromSpec

* Add unit tests

* Fix test

* Extract if block to method for add extra configs to VM Domain XML

* Split travis tests trying to isolate which test is causing an error

* Override toString() method

* Update documentation

* Fix checkstyle error (line with trailing spaces)

* Change VirtualMachineTO print of object

* Add try except to find message error. Remove after test

* Fix indent

* Trying to understanding why is happening in this code

* Refactor method createVMFromSpec

* Add unit tests

* Fix test

* Extract if block to method for add extra configs to VM Domain XML

* Split travis tests trying to isolate which test is causing an error

* Override toString() method

* Update documentation

* Fix checkstyle error (line with trailing spaces)

* Remove unnecessary comment

* Revert travis tests

Co-authored-by: SadiJr <17a0db2854@firemailbox.club>
2021-07-21 15:07:25 -03:00
dahn 6f93e5cd08
Revert "Externalize kvm agent storage timeout configuration (#4585)" (#5218)
This reverts commit 05a978c249.
2021-07-20 09:16:43 +02:00
Daniel Augusto Veronezi Salvador 05a978c249
Externalize kvm agent storage timeout configuration (#4585)
* Externalize KVM Agent storage's timeout configuration

Created a class of constant agent's properties available to configure on "agent.properties".
Created a class to provides a facility to read the agent's properties file and get its properties.

* Refactored KVHAMonitor nested thread and changed some logs

* It has been added the timeout's config in the agent.properties file

* Rename classes

* Rename var and remove comment

* Fix typo with word "heartbeat"

* Extract multiple methods call to variables

* Add unit tests to file handler

* Increase info about the property

* Create inner class Property

* Rename method getProperty to getPropertyValue

* Remove copyright

* Remove copyright

* Extract code to createHeartBeatCommand

* Change method access from protected to private

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-07-19 09:41:50 +02:00
Spaceman1984 96c9c5a5e2
Added disk provisioning type support for VMWare (#4640)
* Added disk provisioning type support for VMWare

* Review changes

* Fixed unit test

* Review changes

* Added missing licenses

* Review changes

* Update StoragePoolInfo.java

Removed white space

* Review change - Getting disk provisioning strictness setting using the zone id and not the pool id

* Delete __init__.py

* Merge fix

* Fixed failing test

* Added comment about parameters

* Added error log when update fails

* Added exception when using API

* Ordering storage pool selection to prefer thick disk capable pools if available

* Removed unused parameter

* Reordering changes

* Returning storage pool details after update

* Removed multiple pool update, updated marvin test, removed duplicate enum

* Removed comment

* Removed unused import

* Removed for loop

* Added missing return statements for failed checks

* Class name change

* Null pointer

* Added more info when a deployment fails

* Null pointer

* Update api/src/main/java/org/apache/cloudstack/api/BaseListCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Small bug fix on API response and added missing bracket

* Removed datastore cluster code

* Removed unused imports, added missing signature

* Removed duplicate config key

* Revert "Added more info when a deployment fails"

This reverts commit 2486db78dc.

Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-07-16 22:37:42 -03:00
Abhishek Kumar 5ac184edd6 Merge remote-tracking branch 'apache/4.15' into main 2021-07-15 14:02:49 +05:30
Rohit Yadav 5228fae7b8
kvm: pre-add 32 PCI controller for hot-plug issue on ARM64 (#5193)
On newer libvirt/qemu it seems PCI hot-plugging could be an issue as
seen in:

https://www.suse.com/support/kb/doc/?id=000019383
https://bugs.launchpad.net/nova/+bug/1836065

This was found to be true on ARM64/aarch64 platform (tested on
RaspberryPi4). As per the default machine doc, it advises to
pre-allocate PCI controllers on the machine and pcie-to-pci-bridge based
controller for legacy PCI models:
https://libvirt.org/pci-hotplug.html#x86_64-q35

This patch introduces the concept as a workaround until a proper fix is
done (ideally in the upstream libvirt/qemu projects). Until then client
code can add 32 PCI controllers and a pcie-to-pci-bridge controller for
aarch64 platforms.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-07-15 13:15:10 +05:30
Gabriel Beims Bräscher 1d831a32a9
kvmk: KVM NFS disk IO driver supporting IO_URING (#5012)
Currently there is no disk IO driver configuration for VMs running on KVM. That's OK for most the cases; however, recently there have been added some quite interesting optimizations with the IO driver io_uring.

Note that IO URING requires:

Qemu >= 5.0, and
Libvirt >= 6.3.0.
By using io_uring we can see a massive I/O performance improvement within Virtual Machines running from Local and/or NFS storage.

This implementation enhances the KVM disk configuration by adding workflow for setting the disk IO drivers. Additionally, if the Qemu and Libvirt versions matches with the required for having io_uring we are going to set it on the VM. If there is no support for such driver we keep it as it is nowadays, without any IO driver configured.

Fixes: #4883
2021-07-15 13:02:44 +05:30
Abhishek Kumar 757bc2d917
vmware: fix migrate vm with volume (#5170)
* vmware: fix migrate vm with volume

Recent forward merge of 4.15 branch accidentally brought a bug in VM relocation method for VMware while trying to find datastore for the migrated volume.
This PR fixes it by using either of available target or source host.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2021-07-15 12:56:26 +05:30
Wei Zhou cf0f1feb5e
configdrive: fix some failures in tests/component/test_configdrive.py (#5144)
* server: fix failed to apply userdata when enable static nat

* server: fix cannot expunge vm as applyUserdata fails

* configdrive: fix ISO is not recognized when plug a new nic

* configdrive: detach and attach configdrive ISO as it is changed when plug a new nic or migrate vm

* configdrive test: (1) password file does not exists in recreated ISO; (2) vm hostname should be changed after migration

* configdrive: use centos55 template with sshkey and configdrive support

* configdrive: disklabel is 'config-2' for configdrive ISO

* configdrive: use copy for configdrive ISO and move for other template/volume/iso

* configdrive: use public-keys.txt

* configdrive test: fix (1) update_template ; (2) ssh into vm by keypair
2021-07-15 12:49:37 +05:30
Rohit Yadav 16d0795130 Merge remote-tracking branch 'origin/4.15' into main 2021-07-05 11:55:58 +05:30
Wei Zhou f286f01516
kvm: fix VM HA on zone-wide storage pools (#5164) 2021-07-05 11:51:07 +05:30
Rohit Yadav d916e416ec Updating pom.xml version numbers for release 4.15.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-07-02 22:59:07 +05:30
Rohit Yadav 379454caae Updating pom.xml version numbers for release 4.15.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-28 15:27:27 +05:30
Pearl Dsilva 5edcf33ff0
vxlan: Fix failure to delete vxlan (#5079) 2021-06-25 09:03:54 +02:00
davidjumani 29109b4332
db: Cleanup obsolete tables (#5002)
* db: Cleanup unused tables

* Removing volume_host_ref references

* Removing template_host_ref references

* fix space issue

* Fix fk constraint

* Removing certificate table

* Revert "Removing certificate table"

This reverts commit fa24e6483f.

* Addressing comments
2021-06-24 16:50:31 -03:00
slavkap 8e33cf7159
kvm: fix of detach volume while OS is in boot state (#4572)
* fix of detach volume while OS is in boot state

* check that the VM could be accessed trough ssh
2021-06-18 06:51:49 +05:30
Daniel Augusto Veronezi Salvador 2ececbf994
kvm: Improve logs on agent start (#4958)
This PR intends to improve logging on agent start to facilitate troubleshooting.

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2021-06-17 13:21:30 +05:30
nicolas d2ab350a40 Merge branch '4.15' into main 2021-06-16 15:38:18 -03:00
Wei Zhou 67182dc49b
vxlan: arp does not work between hosts as multicast group is communicated over physical nic instead of linux bridge (#5078)
* vxlan: arp does not work between hosts as multicast group is communicated over physical nic instead of linux bridge

when linux bridge is setup (refer to http://docs.cloudstack.apache.org/projects/archived-cloudstack-getting-started/en/latest/networking/vxlan.html#configure-product-to-use-vxlan-plugin) and used as the kvm traffic label of physical networks, the vms on different hosts cannot reach each other.

(1) does not work:
```
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvxlan.sh -v 1001 -p eth1 -b brvx-1001 -o add
```
"bridge fdb" shows
```
00:00:00:00:00:00 dev vxlan1001 dst 239.0.3.233 via eth1 self permanent
```

(2) this works:
```
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvxlan.sh -v 1001 -p cloudbr1 -b brvx-1001 -o add
```
"bridge fdb" shows
```
00:00:00:00:00:00 dev vxlan1001 dst 239.0.3.233 via cloudbr1 self permanent
```

* vxlan: fix issue if kvm network label is not set
2021-06-16 08:52:31 +05:30
Harikrishna 12b2e80d82
vmware: Fix fetching chain_info of the volumes. It is used to assume datastore names are in the form of UUIDs but it can be any name. So fetch chain_info based on the datastore name. (#5097)
his PR fixes the problem of not updating the chain info or setting chain info to null after volume migrations.

Problem: While fetching the volume chain info, management server assumes datastore name to be a UUID (this is true only for NFS storages added by CloudStack) but datastore name can be with any name.
Solution: To fetch the volume chain info, use datastore name instead of UUID.

The fix is made in the flow of following API operations

migrateVirtualMachine
migrateVirtualMachineWithVolume
migrateVolume
2021-06-11 20:06:06 +05:30
Rohit Yadav a0788f5816 Merge remote-tracking branch 'origin/4.15' 2021-06-10 12:39:44 +05:30
slavkap b0f2112dc2
kvm: Fix of some UEFI related issues (#5072)
* Fix of some UEFI related issues

1 - fix of attach/detach ISO of VM with UEFI boot type
2 - if OS type of an ISO is categorized as "Other" the bus type of the disk
will be set to "sata"

* Simplify the validation of OS types
2021-06-09 11:15:07 +05:30
Suresh Kumar Anaparti 958182481e cloudstack: make code more inclusive
Inclusivity changes for CloudStack

- Change default git branch name from 'master' to 'main' (post renaming/changing default git branch to 'main' in git repo)
- Rename some offensive words/terms as appropriate for inclusiveness.

This PR updates the default git branch to 'main', as part of #4887.

Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-08 15:47:20 +05:30
Rohit Yadav 5aefd3e204 Merge remote-tracking branch 'origin/4.15' 2021-06-02 14:03:54 +05:30
Rohit Yadav 8c7e39eb89
xenserver: attempt eject and destroying patch VBD separately (#5053)
Loop through all VMs and try to eject and remove systemvm.iso VBD upon host connect.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-01 20:55:57 +05:30
Harikrishna 5a841159c5
vmware: Create fcd folder on local storage in VMware vSphere (#5057)
This PR fixes the issue of missing fcd folder in local storage in case of VMware vSphere.
with this fix, a folder with name fcd is created whenever local storage is initiated.
2021-06-01 20:55:23 +05:30
Abhishek Kumar 4b7fa4e630 Merge remote-tracking branch 'apache/4.15'
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-24 16:43:53 +05:30
Spaceman1984 5f734f718e
vmware: Disk controller vmware deploy as is (#5006)
Fixes #4344
2021-05-19 14:33:16 +05:30
Daniel Augusto Veronezi Salvador 6b9f3fb501
Improve logs on KVMHAVMActivityChecker (#4705)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-05-11 09:12:59 +02:00
Rohit Yadav cb167072a1 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-05-07 16:37:42 +05:30
Harikrishna 32e3bbdcc5
VMware Datastore Cluster primary storage pool synchronisation (#4871)
Datastore cluster as a primary storage support is already there. But if any changes at vCenter to datastore cluster like addition/removal of datastore is not synchronised with CloudStack directly. It needs removal of primary storage from CloudStack and add it again to CloudStack.

Here synchronisation of datastore cluster is fixed without need to remove or add the datastore cluster.
1. A new API is introduced syncStoragePool which takes datastore cluster storage pool UUID as the parameter. This API checks if there any changes in the datastore cluster and updates management server accordingly.
2. During synchronisation if a new child datastore is found in datastore cluster, then management server will create a new child storage pool in database under the datastore cluster. If the new child storage pool is already added as an individual storage pool then the existing storage pool entry will be converted to child storage pool (instead of creating a new storage pool entry)
3. During synchronisaton if the existing child datastore in CloudStack is found to be removed on vCenter then management server removes that child datastore from datastore cluster and makes it an individual storage pool.
The above behaviour is on par with the vCenter behaviour when adding and removing child datastore.
2021-05-07 16:30:54 +05:30
Rohit Yadav 4742ac15f7 Merge remote-tracking branch 'origin/4.15' 2021-04-29 21:50:40 +05:30
Pearl Dsilva 98e2ed3c4f
vmware: Add force parameter to iso attach/detach operations (#4907)
Fixes: #4808, #4941

This PR adds a force flag to the attachIso / detachIso commands, especially for VMware where it is noticed that when trying to either detach an iso or attach an iso when there already exists another present it fails to do the necessary operation as from ACS end we either answer the question returned by Esxi for CDRom disconnect operation as No (for detach operation) or do not answer the question at all (for Attach operation).

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-04-28 13:46:03 +05:30
Rohit Yadav 1e859a0e2c Merge remote-tracking branch 'origin/4.15' 2021-04-27 22:20:30 +05:30
Pearl Dsilva d4db1015b8
marvin: fix test_scale_vm for xenserver/Xcp-ng (#4929)
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-04-27 10:21:48 +05:30
Abhishek Kumar 42c83b08f5 Merge remote-tracking branch 'apache/4.15' 2021-04-26 14:33:58 +05:30
Abhishek Kumar a30d518e8a
vmware: fix stopped VM volume migration (#4758)
* prevent other vm disks getting deleted

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* vmware: fix inter-cluster stopped vm migration

Fixes #4838

For inter-cluster migration without shared storage, VMware needs a host to be specified. Fix is to specify an appropriate host in the target cluster.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix detached volume inter-cluster migration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* cleanup unused method

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* review changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* vmware: allow attached volume migration using VmwareStorageMotionStrategy

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* find vm clusterid with multiple ROOT volumes

VM can have multiple ROOT volumes and some can be on zone-wide store therefore iterate over all of them till a cluster ID is found.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix successive storage migration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix intercluster check

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor vm cluster, host method

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove inter-pod check

Added by mistake, VMware won't have pods

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address review comment

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-24 18:55:25 +05:30
Rohit Yadav f42024714c Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-04-15 16:17:13 +05:30
Rohit Yadav 0302750aac
vmware: Add support for VMware 7 (#4300) 2021-04-15 16:10:14 +05:30
Daniel Augusto Veronezi Salvador 911376e495
Improve logs on KVMHostActivityChecker (#4706)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-04-13 08:27:25 +02:00
Rohit Yadav 8fd9150a89 Merge remote-tracking branch 'origin/4.15' 2021-04-13 00:10:17 +05:30
Nicolas Vazquez 9cf1e0e869
vmware: Fix VMware OVF properties copy from template (#4738)
* Fix VMware OVF properties copy from template

* Fix vapp marvin test

* Remove unused code

* Fix check for deploy as is details

* Access class fields
2021-04-12 18:04:58 +05:30
Abhishek Kumar cce736709e Merge remote-tracking branch 'apache/4.15' 2021-04-12 11:43:57 +05:30
Abhishek Kumar fdefee75ff
vmware: fix inter-cluster stopped vm and volume migration (#4895)
Fixes #4838

For inter-cluster migration without shared storage, VMware needs a host to be specified. Fix is to specify an appropriate host in the target cluster during a stopped VM migration. Also, find target datastore using the host in the target cluster.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-10 13:22:28 +05:30
Rohit Yadav 22f6c19248 Merge remote-tracking branch 'origin/4.15' 2021-04-09 13:21:07 +05:30
Abhishek Kumar d8c6e00498
hypervisor: XCP-ng 8.2 support (#4672)
Adds new/missing guest os mappings for XCP-ng/Xenserver 8.1
Copy guest OS mappings from XCP-ng/Xenserver 8.1 for XCP-ng/Xenserver 8.2
Adds Ubuntu 20.04 guest os mapping for XCP-ng/Xenserver 8.2

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-09 13:12:06 +05:30
Rohit Yadav 76e9c2772b Merge remote-tracking branch 'origin/4.15' 2021-04-06 16:50:31 +05:30
Nicolas Vazquez 35b20b2367
vmware: Fix worker VM hardware version format (#4851)
This PR fixes a small bug when explicitly setting VM hardware versions lower than version 10.

Vmware expects the hardware version in format: vmx-DD where DD is a two-digit representation of the virtual hardware version. For hardware version lower than 10, CloudStack was not using to digits for the hardware version number, which ended up on an error while creating worker VMs. (vmx-8 for example instead of vmx-08)
2021-04-06 16:28:52 +05:30
Rohit Yadav 49acd1dec7 Merge remote-tracking branch 'origin/4.15' 2021-04-06 15:09:48 +05:30
slavkap 52c36cadf0
kvm: Fix deploy VM from ISOs with UEFI (#4773)
This PR fixes #4244
deploying of VMs from ISOs and from templates with UEFI boot type
deploying of VMs from ISOs and from templates with UEFI boot type with
volumes in RAW format
2021-04-06 15:00:21 +05:30
Rohit Yadav 3066c95548 Merge remote-tracking branch 'origin/4.15' 2021-04-05 14:59:15 +05:30
Rohit Yadav 6c3f4b43aa Merge remote-tracking branch 'origin/4.14' into 4.15 2021-04-05 14:58:41 +05:30
Pearl Dsilva 0dbeb262e4
server: Support for persistence mode in L2 networks (#4561)
This PR aims at introducing persistence mode in L2 networks and enhancing the behavior in Isolated networks
Doc PR apache/cloudstack-documentation#183

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-04-05 14:37:11 +05:30
Abhishek Kumar 6048afb464
xenserver: retrieve correct name-label for presetup store (#4816)
Fixes #4729

As reported in the issue ACP 4.7 used a normal UUID in db for a presetup primary store on Xenserver.
Later the value has been changed to store's path with '/' removed.
Current changes try to retrieve SR's name-lable from store's path if UUID doesn't match path field for a pre-setup store.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-05 14:28:27 +05:30
Rohit Yadav def65ec873 Merge remote-tracking branch 'origin/4.15' 2021-04-04 13:09:41 +05:30
Wei Zhou 09428380f7
kvm: remove unnecessary new String (#4870)
Thanks @rubieHess to point it out.
see #4800 (comment)
2021-04-04 13:08:29 +05:30
Rohit Yadav c1a02e1697 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-03-29 16:34:22 +05:30
Abhishek Kumar d763169b1c
Restore VMware VM naming convention option (#4581)
* initial chanes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* make check explicit for instance name flag

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* allow hiding vm details (in ui)

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* condition based on name instead of displayname

Signed-off-by: Abhishek Kumar <abhishek.kumar@shapeblue.com>
2021-03-29 16:13:14 +05:30
Pearl Dsilva 97176690b8
server: Fix issue with volume resize on VMWare (deploy as-is templates) (#4829)
This PR fixes the issue pertaining to volume resize on VMWare for deploy as-is templates. VMware deploy as-is templates are those that are deployed as per the specification in the imported OVF. Hence override root disk size will not be adhered to for such templates. Moreover, when we deploy VMs in stopped state and resize the volume, the root disk doesn't get resized but the volume size is merely updated in the DB.
This PR also includes the following (for deploy as-is templates):
- Disables overriding root disk size during VM deployment on the UI
- Disables selection of compute offerings with root disk size specified, at the time of deployment
- Provided users with the option to deploy VM is stopped state via UI (so as to give an option to users to resize the volumes before starting the VM)

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-03-29 12:54:47 +05:30
Daan Hoogland 8903647187 Merge branch '4.15' 2021-03-25 17:48:45 +01:00
Wei Zhou 952b242722
kvm: Do not set backing file format of DATADISK in vm start/migration (#4800)
* kvm: Do not rebase DATADISK in vm start/migration

* #4800: add comment and use isBlank
2021-03-24 20:22:51 +01:00
Daniel Augusto Veronezi Salvador 4e90a8c454
Qemu 2.10 requires `-U` flag to read volume metadata (#4567)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-03-18 11:54:01 +01:00
sureshanaparti 81dfcbb5f5
server: Use ACPI event to reboot VM on KVM, and Use 'forced' reboot option to stop and start the VM(s) (#4681)
* Updated libvirt's native reboot operation for VM on KVM using ACPI event, and Added 'forced' reboot option to stop and start the VM (using rebootVirtualMachine API)

* Added 'forced' reboot option for System VM and Router

- New parameter 'forced' in rebootSystemVm API, to stop and then start System VM
- New parameter 'forced' in rebootRouter API, to force stop and then start Router

* Added force reboot tests for User VM, System VM and Router
2021-03-06 14:58:56 +05:30
Rohit Yadav fa067e02a7 Updating pom.xml version numbers for release 4.14.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-03-02 12:32:27 +05:30
Rohit Yadav 77290df0d5 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-26 12:09:11 +05:30
Greg Goodrich af0f6422ec
kvm: Suspending the VM prior to deleting snapshots to avoid corruption, th… (#4032)
These changes are related to PR #3194, but include suspending/resuming the VM when doing a VM snapshot as well, when deleting a VM snapshot, as it is performing the same operations via Libvirt. Also, there was an issue with the UI/localization changes in the prior PR, as that PR was altering the Volume snapshot behavior, but was altering the VM snapshot wording. Both have been altered in this PR.

Issuing this in response to the work happening in PR #4029.
2021-02-25 22:11:00 +05:30
sureshanaparti eba186aa40
storage: New Dell EMC PowerFlex Plugin (formerly ScaleIO, VxFlexOS) (#4304)
Added support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack (for KVM hypervisor) and enabled VM/Volume operations on that pool (using pool tag).
Please find more details in the FS here:
https://cwiki.apache.org/confluence/x/cDl4CQ

Documentation PR: apache/cloudstack-documentation#169

This enables support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack

Other improvements addressed in addition to PowerFlex/ScaleIO support:

- Added support for config drives in host cache for KVM
	=> Changed configuration "vm.configdrive.primarypool.enabled" scope from Global to Zone level
	=> Introduced new zone level configuration "vm.configdrive.force.host.cache.use" (default: false) to force host cache for config drives
	=> Introduced new zone level configuration "vm.configdrive.use.host.cache.on.unsupported.pool" (default: true) to use host cache for config drives when storage pool doesn't support config drive
	=> Added new parameter "host.cache.location" (default: /var/cache/cloud) in KVM agent.properties for specifying the host cache path and create config drives on the "/config" directory on the host cache path
	=> Maintain the config drive location and use it when required on any config drive operation (migrate, delete)

- Detect virtual size from the template URL while registering direct download qcow2 (of KVM hypervisor) templates

- Updated full deployment destination for preparing the network(s) on VM start

- Propagate the direct download certificates uploaded to the newly added KVM hosts

- Discover the template size for direct download templates using any available host from the zones specified on template registration
	=> When zones are not specified while registering template, template size discovery is performed using any available host, which is picked up randomly from one of the available zones

- Release the VM resources when VM is sync-ed to Stopped state on PowerReportMissing (after graceful period)

- Retry VM deployment/start when the host cannot grant access to volume/template

- Mark never-used or downloaded templates as Destroyed on deletion, without sending any DeleteCommand
	=> Do not trigger any DeleteCommand for never-used or downloaded templates as these doesn't exist and cannot be deleted from the datastore

- Check the router filesystem is writable or not, before performing health checks
	=> Introduce a new test "filesystem.writable.test" to check the filesystem is writable or not
	=> The router health checks keeps the config info at "/var/cache/cloud" and updates the monitor results at "/root" for health checks, both are different partitions. So, test at both the locations.
	=> Added new script: "filesystem_writable_check.py" at /opt/cloud/bin/ to check the filesystem is writable or not

- Fixed NPE issue, template is null for DATA disks. Copy template to target storage for ROOT disk (with template id), skip DATA disk(s)

* Addressed some issues for few operations on PowerFlex storage pool.

- Updated migration volume operation to sync the status and wait for migration to complete.

- Updated VM Snapshot naming, for uniqueness in ScaleIO volume name when more than one volume exists in the VM.

- Added sync lock while spooling managed storage template before volume creation from the template (non-direct download).

- Updated resize volume error message string.

- Blocked the below operations on PowerFlex storage pool:
  -> Extract Volume
  -> Create Snapshot for VMSnapshot

* Added the PowerFlex/ScaleIO client connection pool to manage the ScaleIO gateway clients, which uses a single gateway client per Powerflex/ScaleIO storage pool and renews it when the session token expires.

- The token is valid for 8 hours from the time it was created, unless there has been no activity for 10 minutes.
  Reference: https://cpsdocs.dellemc.com/bundle/PF_REST_API_RG/page/GUID-92430F19-9F44-42B6-B898-87D5307AE59B.html

Other fixes included:

- Fail the VM deployment when the host specified in the deployVirtualMachine cmd is not in the right state (i.e. either Resource State is not Enabled or Status is not Up)

- Use the physical file size of the template to check the free space availability on the host, while downloading the direct download templates.

- Perform basic tests (for connectivity and file system) on router before updating the health check config data
	=> Validate the basic tests (connectivity and file system check) on router
	=> Cleanup the health check results when router is destroyed

* Updated PowerFlex/ScaleIO storage plugin version to 4.16.0.0

* UI Changes to support storage plugin for PowerFlex/ScaleIO storage pool.
- PowerFlex pool URL generated from the UI inputs(Gateway, Username, Password, Storage Pool) when adding "PowerFlex" Primary Storage
- Updated protocol to "custom" for PowerFlex provider
- Allow VM Snapshot for stopped VM on KVM hypervisor and PowerFlex/ScaleIO storage pool

and Minor improvements in PowerFlex/ScaleIO storage plugin code

* Added support for PowerFlex/ScaleIO volume migration across different PowerFlex storage instances.

- findStoragePoolsForMigration API returns PowerFlex pool(s) of different instance as suitable pool(s), for volume(s) on PowerFlex storage pool.
- Volume(s) with snapshots are not allowed to migrate to different PowerFlex instance.
- Volume(s) of running VM are not allowed to migrate to other PowerFlex storage pools.
- Volume migration from PowerFlex pool to Non-PowerFlex pool, and vice versa are not supported.

* Fixed change service offering smoke tests in test_service_offerings.py, test_vm_snapshots.py

* Added the PowerFlex/ScaleIO volume/snapshot name to the paths of respective CloudStack resources (Templates, Volumes, Snapshots and VM Snapshots)

* Added new response parameter “supportsStorageSnapshot” (true/false) to volume response, and Updated UI to hide the async backup option while taking snapshot for volume(s) with storage snapshot support.

* Fix to remove the duplicate zone wide pools listed while finding storage pools for migration

* Updated PowerFlex/ScaleIO volume migration checks and rollback migration on failure

* Fixed the PowerFlex/ScaleIO volume name inconsistency issue in the volume path after migration, due to rename failure
2021-02-24 14:58:33 +05:30
Rohit Yadav 6634382f69 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-19 15:30:21 +05:30
Wei Zhou db1e0f49dc
kvm: set Backing Format of qcow2 images in vm start and migration (#4656)
In previous cloudstack versions, qcow2 image does not have a backing file format.
however, it is required in newer qemu versions, for example qemu 4.2 on ubuntu 20.04.

steps to reproduce the issue
(1) install cloudstack 4.14 or previous version, and ubuntu 19.04 or 18.04/16.04 LTS.
(2) create vms.
(3) upgrade to 4.15, upgrade os to ubuntu 20.04 , or install a new server with ubuntu 20.04.
(4) migrate vm from old ubuntu version to ubuntu 20.04, failed with exception below
```
2021-02-04 13:43:07,397 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (agentRequest-Handler-1:null) (logid:93da9385) ExecutionException : org.libvirt.LibvirtException: Requested operation is not valid: format of backing image '/mnt/03b6f487-9eaf-38bf-ad2d-d985423b832f/66990fcc-fd98-4932-9649-989bf6583d59' of image '/mnt/03b6f487-9eaf-38bf-ad2d-d985423b832f/a3dd1f0f-2557-4e07-951c-e4eb7b3f38b2' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)
```
(5)stop vm, and start it on ubuntu 20.04 server. failed with exception below
```
2021-02-04 13:46:29,766 WARN  [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:null) (logid:b54745a7) LibvirtException
org.libvirt.LibvirtException: Requested operation is not valid: format of backing image '/mnt/03b6f487-9eaf-38bf-ad2d-d985423b832f/66990fcc-fd98-4932-9649-989bf6583d59' of image '/mnt/03b6f487-9eaf-38bf-ad2d-d985423b832f/a3dd1f0f-2557-4e07-951c-e4eb7b3f38b2' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)
```

To make testing easier, step 1 and 2 can be replaced by
```
qemu-img create -f qcow2 -b <backing file> <qcow2 image>
```
so qcow2 image does not have a backing file format.
2021-02-19 14:06:47 +05:30
Abhishek Kumar d6e8b53736
vmware: vm migration improvements (#4385)
- Fixes inter-cluster migration of VMs
- Allows migration of stopped VM with disks attached to different and suitable pools
- Improves inter-cluster detached volume migration
- Allows inter-cluster migration (clusters of same Pod) for system VMs, VRs on VMware
- Allows storage migration for stopped system VMs, VRs on VMware within same Pod if StoragePool cluster scopetype

Linked Primate PR: https://github.com/apache/cloudstack-primate/pull/789 [Changes merged in this PR after new UI merge]
Documentation PR: https://github.com/apache/cloudstack-documentation/pull/170

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-02-12 12:41:41 +05:30
Rohit Yadav 66f0beda5f Updating pom.xml version numbers for release 4.14.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-08 16:24:09 +05:30
Daan Hoogland ff376d8187 Merge release branch 4.15 to master
* 4.15:
  server: select root disk based on user input during vm import (#4591)
  kvm: Use Q35 chipset for UEFI x86_64 (#4576)
  server: fix wrong error message when create isolated network without SourceNat (#4624)
  server: add possibility to scale vm to current customer offerings (#4622)
  server: keep networks order and ips while move a vm with multiple networks (#4602)
  server: throw exception when update vm nic on L2 network (#4625)
  doc: fix typo in install notes (#4633)
2021-02-01 09:58:52 +00:00
Daan Hoogland b6b778f003 Merge release branch 4.14 to 4.15
* 4.14:
  server: select root disk based on user input during vm import (#4591)
  kvm: Use Q35 chipset for UEFI x86_64 (#4576)
  server: fix wrong error message when create isolated network without SourceNat (#4624)
  server: add possibility to scale vm to current customer offerings (#4622)
  server: keep networks order and ips while move a vm with multiple networks (#4602)
  server: throw exception when update vm nic on L2 network (#4625)
  doc: fix typo in install notes (#4633)
2021-02-01 09:57:35 +00:00
jairov4 e9dda98a87
kvm: Use Q35 chipset for UEFI x86_64 (#4576)
Fix #4245

This PR uses Q35 chipset for UEFI in x86_64.
Currently this mistakenly only enabled for secure boot
2021-02-01 14:22:29 +05:30
Daniel Augusto Veronezi Salvador 3a4a82df69
kvm: Replicate `rootDiskController` behavior to `dataDiskController`. (#4569)
The bus type to `data disk` volumes is hardcoded to `virtio` or `scsi`, when using virtio-scsi (or, based on the template type). Therefore, there is no way to specify the bus type to data disk volumes (as we have for root disks).
This PR intends to replicate the `rootDiskController` behavior to `dataDiskController`, allowing the definition of the controller.

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-01-28 15:58:06 +05:30
Rohit Yadav 4aa7a96bc4 Merge remote-tracking branch 'origin/4.15'
This merges apache/cloudstack-primate under ui and removes the legacy UI
from ui/legacy in master/4.16 as voted on dev ML.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-20 13:50:25 +05:30
div8cn c9a1d300cb
kvm: Fix RBD primary storage host port null error (#4565)
Add RBD main storage through UI, it will fail when there is no host port parameter;
Because when we created the pool, we did not add the port target in the xml
2021-01-14 06:20:16 +05:30
Rohit Yadav b482da8c91 Updating pom.xml version numbers for release 4.15.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-11 13:58:30 +05:30
Daan Hoogland 280c13a4bb Updating pom.xml version numbers for release 4.15.0.0
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2021-01-05 15:51:02 +00:00
Rohit Yadav 015bb20caa Merge remote-tracking branch 'origin/4.15' 2021-01-05 15:25:39 +05:30
Rohit Yadav c82688a355
kvm: Fix double-escape issue while creating rbd disk options (#4568)
This fixes issue introduced in c3554ec31d
which enable block of code that will double escape rados host/monitor
port.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-05 15:23:38 +05:30
Daan Hoogland 81e9e6809b Updating pom.xml version numbers for release 4.15.1.0-SNAPSHOT
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2021-01-04 11:34:46 +00:00
Daan Hoogland e26202f23e Updating pom.xml version numbers for release 4.16.0.0-SNAPSHOT
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2021-01-04 11:32:10 +00:00
Daan Hoogland 01b3e361c7 Updating pom.xml version numbers for release 4.15.0.0
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2020-12-23 16:32:25 +00:00
Rohit Yadav e5b7733498 Merge remote-tracking branch 'origin/4.14'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-12-10 14:58:23 +05:30
Wei Zhou 1a47719588
kvm: set cpu topology only if cpucore per socket is positive value (#4527)
This PR fixes a regression issue in #4497

In cloudstack 4.14 or before, the cpu topology is set only when cpucore per socket is set (to 4 or 6).
in other conditions, there is no cpu topology in vm xml definition.

with #4497, vm will have cpu topology in its xml definition, if cpucore per socket is not set.

    <topology sockets='<vm cpu cores>' cores='1' threads='1'/>
Not sure if it causes any issue. I think it would be better not to add this part in vm xml definition if cpucore per socket is not set.
2020-12-10 14:29:28 +05:30
Rohit Yadav e0242d5793
xenserver: check and eject patch vbd for systemvms (#4525)
XenServer 7.1 has an file descriptor/tapdisk iso-caching issue where new systemvm.iso are not recognised and inside the VR/ssvm/cpvm file IO error is seen. This was only reproducible with XS7.1 (intermittently), the fix was to check and eject the systemvm.iso (old/stale/cached), then insert the new systemvm.iso and then eject it.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-12-10 00:42:41 +05:30
Wei Zhou 93f3d35207
kvm: FIX cpucorespersocket is not working on KVM (#4497) 2020-12-09 15:07:51 +01:00
Alexandru Bagu cd356c0513
change debug to warn for unknown exceptions (#4521)
Co-authored-by: Alexandru Bagu <abagu@bsss-MacBook-Pro.local>
2020-12-09 12:24:54 +01:00
Harikrishna 51034be270
Fix migrateVMwithVolumes API in case of multiple volumes on VMware (#4480)
Problem:
When migrateVMwithVolumes API is tried on a VM with two volumes to migrate to a different host and tried to migrate only one volume, Cloudstack migrates both the Volumes but then marks only one of them migrated. This makes volume inaccessible due to inconsitency in path of volume in cloudstack and vsphere

Solution:
Set the target datastore in relocate spec properly for each volume
2020-11-20 12:38:06 +00:00
Spaceman1984 87e1fa9e02
Removed sensitive info from UI/API when volume attach/detach fails (#4476)
* Removed sensitive info from UI when attach/detach fails

* minor formatting change
2020-11-19 04:39:37 +05:30
nvazquez 7d7e7f7941 Check there is at least a host to verify pool compatibility for a storage policy 2020-10-26 09:07:23 -03:00
nvazquez 85f5512ec3 Fix unit test failure 2020-10-25 02:53:43 -03:00