Commit Graph

2888 Commits

Author SHA1 Message Date
Gabriel Beims Bräscher ca78f5b386
Enhance log messages with host name (#4575)
* Enhance log messages with hostName

* Use host.toString() on most of host logs.

* Remove redundant "Host" in logs and enhance logs

* duplicated "for"

* Adopt String.format, and enhance code

* Address reviews enhancing log messages

Update server/src/main/java/com/cloud/resource/ResourceManagerImpl.java
    -- server/src/main/java/com/cloud/vm/UserVmManagerImpl.java
    -- server/src/main/java/com/cloud/resource/RollingMaintenanceManagerImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Fix String.format issue and change log message from debug to warn

* Fix checkstyle issue

* Fix string.format log

* Address review: enhance logs

* Enhance log of hosts in maintenance avoid list

* Remove "VM" on logs as vm.toString() already appends VM-<details>

* Add more details of the VM when postStateTransitionEvent

* Address reviewer and enhance VMInstanceVO.toString()

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2021-07-13 17:35:59 -03:00
Pearl Dsilva 3fd92502bb
Prevent starting a VM in destroyed state (or any state but Stopped) (#5165)
* Prevent starting a VM in destroyed state (or any state but Stopped)
2021-07-09 15:11:30 +05:30
Rohit Yadav d916e416ec Updating pom.xml version numbers for release 4.15.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-07-02 22:59:07 +05:30
Abhishek Kumar 5fd970d88f
db, server: refactor host_view to prevent duplicate entries (#4796)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-07-01 11:00:51 -03:00
Rohit Yadav 379454caae Updating pom.xml version numbers for release 4.15.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-28 15:27:27 +05:30
Rohit Yadav f58b72f6f7 Merge remote-tracking branch 'origin/4.15' 2021-06-27 18:25:46 +05:30
DK101010 53963256d8
server: Bug/false positive success message vm start (#5148)
* add throws statement during the retry process

* Update engine/orchestration/src/main/java/org/apache/cloudstack/engine/cloud/entity/api/VMEntityManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-06-27 06:40:30 +05:30
slavkap d82909318f
server: Fix of delete of Ceph's snapshots from secondary storage (#5130)
This PR fixes the deletion will be handled by DefaultSnapshotStrategy::deleteSnapshot #4797
2021-06-25 12:04:36 +05:30
davidjumani 29109b4332
db: Cleanup obsolete tables (#5002)
* db: Cleanup unused tables

* Removing volume_host_ref references

* Removing template_host_ref references

* fix space issue

* Fix fk constraint

* Removing certificate table

* Revert "Removing certificate table"

This reverts commit fa24e6483f.

* Addressing comments
2021-06-24 16:50:31 -03:00
davidjumani 1a7bfa98d8
Follow consistent vo table name convention (#5004) 2021-06-24 16:46:50 -03:00
nicolas d2ab350a40 Merge branch '4.15' into main 2021-06-16 15:38:18 -03:00
sureshanaparti 07cabbe7ac
scaleio: Updated PowerFlex/ScaleIO gateway client with some improvements. (#5037)
- Added connection manager to the gateway client.
 - Renew the client session on '401 Unauthorized' response.
 - Refactored the gateway client calls, for GET and POST methods.
 - Consume the http entity content after login/(re)authentication and close the content stream if exists.
 - Updated storage pool client connection timeout configuration 'storage.pool.client.timeout' to non-dynamic.
 - Added storage pool client max connections configuration 'storage.pool.client.max.connections' (default: 100) to specify the maximum connections for the ScaleIO storage pool client.
 - Updated unit tests.
and blocked the attach volume operation for uploaded volume on ScaleIO/PowerFlex storage pool
2021-06-16 12:45:27 +05:30
Harikrishna 12b2e80d82
vmware: Fix fetching chain_info of the volumes. It is used to assume datastore names are in the form of UUIDs but it can be any name. So fetch chain_info based on the datastore name. (#5097)
his PR fixes the problem of not updating the chain info or setting chain info to null after volume migrations.

Problem: While fetching the volume chain info, management server assumes datastore name to be a UUID (this is true only for NFS storages added by CloudStack) but datastore name can be with any name.
Solution: To fetch the volume chain info, use datastore name instead of UUID.

The fix is made in the flow of following API operations

migrateVirtualMachine
migrateVirtualMachineWithVolume
migrateVolume
2021-06-11 20:06:06 +05:30
Daniel Augusto Veronezi Salvador bf6266188c
Improve logs on RemoteAccessVpnManagerImpl (#4960)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-06-10 16:51:45 +02:00
Harikrishna 083646b3c3
server: VM dynamic scaling option granularity (#4643)
This PR introduces new granularity levels to configure VM dynamic scalability. Previously VM is configured to be dynamically scalable based on the template and global setting. Now we bringing this option to configure at service offering and VM level also.

VM can dynamically scale only when all flags are ON at VM level, template, service offering and global setting. If any of the flags is set to false then VM cannot be scalable. This result will be persisted in DB for each VM and will be honoured for that VM till it is updated.

We are introducing 'dynamicscalingallowed' parameter with permitted values of true or false for deployVM API and createServiceOffering API.

Following are the API parameter changes:
createServiceOffering API: 
          dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.
deployVirtualMachine API: 
          dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.

Following are the UI changes:
Service offering creation has ON/OFF switch for dynamic scaling enabled with default value true
2021-06-08 17:10:00 +05:30
Suresh Kumar Anaparti 958182481e cloudstack: make code more inclusive
Inclusivity changes for CloudStack

- Change default git branch name from 'master' to 'main' (post renaming/changing default git branch to 'main' in git repo)
- Rename some offensive words/terms as appropriate for inclusiveness.

This PR updates the default git branch to 'main', as part of #4887.

Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-08 15:47:20 +05:30
Rohit Yadav d10cdb495f Merge remote-tracking branch 'origin/4.15' 2021-06-07 15:34:52 +05:30
Gabriel Beims Bräscher 3ee563905d
kvm: Check for VLAN or VXLAN in NetworkDaoImpl.listByPhysicalNetworkPvlan (#5074)
This PR fixes #5071; where it was reported an issue when creating a network with VXLAN.
2021-06-05 22:25:01 +05:30
Rohit Yadav d184e282b2 Merge remote-tracking branch 'origin/4.15' 2021-05-31 20:01:09 +05:30
Pearl Dsilva d04fa0201d
server: usage generated for destroyed VMs with no backups (#5017)
Fixes: #4990
When a VM associated with a backup offering is destroyed/expunged, the backup offering isn't unassigned, and despite the VM having no backups present, backup usage is generated. This PR prevent usage record generation when there are no backups present for a VM with a backup offering associated to it. This is done by ensuring that usage event for backups is generated only when a the backup size > 0
2021-05-31 18:59:48 +05:30
Rohit Yadav 57e4e3b6b1 Merge remote-tracking branch 'origin/4.15' 2021-05-31 16:04:42 +05:30
Rohit Yadav fbc8610f6e Merge remote-tracking branch 'origin/4.14' into 4.15 2021-05-31 15:54:56 +05:30
Gabriel Beims Bräscher a78f676037
engine: fix network with SG disabled still has security group script adding rules on KVM (#5049)
This PR fixes #5047 which can be reproduced on Zones with _(I) Advanced Networks, (II) Security Groups enabled for the Zone, (III)  network offering without Security Groups_; for instance, `DefaultSharedNetworkOffering` which does not list Security Group as supported service.

The issue is due to the following code inside the method `VirtualMachineManagerImpl.orchestrateReboot`:
[VirtualMachineManagerImpl.java#L3340](280c13a4bb/engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java (L3340)).

```
  final Answer rebootAnswer = cmds.getAnswer(RebootAnswer.class);
  if (rebootAnswer != null && rebootAnswer.getResult()) {
      if (dc.isSecurityGroupEnabled() && vm.getType() == VirtualMachine.Type.User) {
          List<Long> affectedVms = new ArrayList<Long>();
          affectedVms.add(vm.getId());
          _securityGroupManager.scheduleRulesetUpdateToHosts(affectedVms, true, null);
      }
      return;
  }
```
2021-05-31 15:52:26 +05:30
Gabriel Beims Bräscher a3cdd1f836
Allow deploy Admin VMs and VRs in disabled zones/pods/clusters (#3600) 2021-05-28 10:45:30 +02:00
Daniel Augusto Veronezi Salvador 6531ee5871
Externalize config to set min memory/cpu with division by overprovisi… (#4722)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-05-28 10:15:51 +02:00
Abhishek Kumar 4b7fa4e630 Merge remote-tracking branch 'apache/4.15'
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-24 16:43:53 +05:30
Pearl Dsilva 2eae0f5385
SystemVM: Set agent state to disconnected on Stopping the systemVM (#5010)
Fixes: #4972
This PR sets systevms' agent state to disconnected when it is stopped. Currently, when a systemVM (Console Proxy VM / Secondary storage VM) is stopped, the agent state still appears to be 'Up'
2021-05-19 13:00:17 +05:30
Abhishek Kumar 426f14b6ed Merge remote-tracking branch 'apache/4.15'
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-18 15:19:20 +05:30
Rohit Yadav 2286c8d2bf Merge remote-tracking branch 'origin/4.14' into 4.15 2021-05-14 23:19:06 +05:30
Abhishek Kumar dc91a1fd4d
server: destroy ssvm, cpvm on last host maintenance (#4644)
* server: destroy ssvm, cpvm on last host maintenance

When a single or last UP host enters into maintenance just stopping SSVM and CPVM will leave behind VMs on hypervisor side. As these system vms will be recreated they can be destroyed.
Fixes #3719

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix methods

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* immediately destroy systemvms

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix destroy

Added bypassHostMaintenance flag in Comma.java class to allow command to be handled by host agent even when host is in maintenace.
Flag is set true only for delete commands for ssvm and cpvm.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unit test fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing return statement

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

VM should be stopped with cleanup before calling expunge else it server may through error with host in PrepareForMaintenance state.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* rename

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-14 23:16:15 +05:30
Wei Zhou e2183ed666
forceha: fix two issues when (1)stop vm from inside (2) force remove host (#4647)
* forceha: fix vm is not started if it is poweroff from inside

steps to reproduce the issue
(1) make sure force.ha is true in global setting. if not, change it to true, and restart mgt server
(2) create a service offering , ha is not enabled
(3) create a vm
(4) log into the vm, and power off via cli.

expected result: vm is started again by cloudstack
actual result: vm is not started.

* forceha: fix vms are still running if host is force-removed

when host can be force removed, however vms are stopped in cloudstack, but not stopped on host
```
(localcloud) 🐱 > delete host id="a5625393-444d-4d0a-b31d-62baf88a8be1" forced=true
{
  "success": true
}```

after some minutes, vms are still runnning on host
```
root@mgt01:~# ssh node63 virsh list
 Id   Name        State
---------------------------
 1    i-2-19-VM   running
 2    i-2-11-VM   running
```

error message are
```
Cannot transmit host 2 to Enabled state
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state = Enabled event = DeleteHost
        at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1216)
        at com.cloud.resource.ResourceManagerImpl$1.doInTransactionWithoutResult(ResourceManagerImpl.java:907)
```

* forceha: Make ForceHA dynamic
2021-05-14 23:14:39 +05:30
Wei Zhou 1b28ea1ebb
network: fix dhcp/password/metadata issues on shared networks with multiple subnets (#5013)
* #4943: apply iptables for password and metadata

* #4943: fix wrong ip alias

* #4943: revert previous change and add ip_aliases

Co-authored-by: Wei Zhou <weizhouapache@gmail.com>
2021-05-13 14:31:47 +05:30
Daniel Augusto Veronezi Salvador 3674e80219
Improve logs on SecondaryStorageManagerImpl and few refactors (#4955)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-05-10 09:47:42 +02:00
Rohit Yadav cb167072a1 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-05-07 16:37:42 +05:30
Harikrishna 32e3bbdcc5
VMware Datastore Cluster primary storage pool synchronisation (#4871)
Datastore cluster as a primary storage support is already there. But if any changes at vCenter to datastore cluster like addition/removal of datastore is not synchronised with CloudStack directly. It needs removal of primary storage from CloudStack and add it again to CloudStack.

Here synchronisation of datastore cluster is fixed without need to remove or add the datastore cluster.
1. A new API is introduced syncStoragePool which takes datastore cluster storage pool UUID as the parameter. This API checks if there any changes in the datastore cluster and updates management server accordingly.
2. During synchronisation if a new child datastore is found in datastore cluster, then management server will create a new child storage pool in database under the datastore cluster. If the new child storage pool is already added as an individual storage pool then the existing storage pool entry will be converted to child storage pool (instead of creating a new storage pool entry)
3. During synchronisaton if the existing child datastore in CloudStack is found to be removed on vCenter then management server removes that child datastore from datastore cluster and makes it an individual storage pool.
The above behaviour is on par with the vCenter behaviour when adding and removing child datastore.
2021-05-07 16:30:54 +05:30
Rohit Yadav fce84e62ab Merge remote-tracking branch 'origin/4.15' 2021-05-06 00:27:43 +05:30
Pearl Dsilva bc80815cf5
server: Adding VPN options for IKE version and IKE split connections (#4953)
IKE version allows selecting ike (autoselect), ikev1, or ikev2.
Split connections gives an option of separating the first right subnet from the rest, and kicking out individual statements for each right subnet for better cross-compatibility.

Backported from PR: #4137
update per PR suggestion

Fixes #3138

Co-authored-by: Greg Goodrich <ggoodrich@ippathways.com>
Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-05-05 12:54:23 +05:30
Rohit Yadav 1abd10199c Merge remote-tracking branch 'origin/4.15' 2021-05-04 19:37:45 +05:30
Gabriel Beims Bräscher ab790c11d5
server: Allow to upgrade service offerings from local <> shared storage pools (#4915)
This PR addresses the issue raised at #4545 (Fail to change Service offering from local <> shared storage).

When upgrading a VM service offering it is validated if the new offering has the same storage scope (local or shared) as the current offering. I think that the validation makes sense in a way of preventing running Root disks with an offering that does not match the current storage pool. However, the validation only compares both offerings and does not consider that it is possible to migrate Volumes between local <> shared storage pools.

The idea behind this implementation is that CloudStack should check the scope of the current storage pool which the ROOT volume is allocated; this, it is possible to migrate the volume between storage pools and list/upgrade according to the offerings that are supported for such pool.

This PR also fixes an issue where the API command that lists offerings for a VM should follow the same idea and list based on the storage pool that the volume is allocated and not the previous offering.

Fixes: #4545
2021-04-30 11:59:50 +05:30
Olivier Lemasle 72f6612971
server: Increase max length for VMInstanceVO.backupVolumes (#4967)
The default length is 255, which caused a truncation of data if
the JSON object representing the backup volumes is too big.
It caused errors when backups were made on VMs with 3 volumes
or more.

`vm_instance.backup_volumes` has the type TEXT, which has a
maximal length of 65535 characters.

Fixes #4965
2021-04-30 11:57:56 +05:30
Rohit Yadav 4742ac15f7 Merge remote-tracking branch 'origin/4.15' 2021-04-29 21:50:40 +05:30
dahn be255e4203
server: protect against stray snapshot-details without snapshot (#4924)
This PR makes sure no orphaned snapshot details are considered in the cleanup at startup job.
a real solution would be to implement some kind of cascading delete, but as the parent record is "only" marked as removed this would be a bit com

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-04-29 20:40:29 +05:30
Abhishek Kumar 42c83b08f5 Merge remote-tracking branch 'apache/4.15' 2021-04-26 14:33:58 +05:30
Nicolas Vazquez f728287aa2
server: Fix template garbage collection cleanup (#4944) 2021-04-24 18:57:47 +05:30
Abhishek Kumar a30d518e8a
vmware: fix stopped VM volume migration (#4758)
* prevent other vm disks getting deleted

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* vmware: fix inter-cluster stopped vm migration

Fixes #4838

For inter-cluster migration without shared storage, VMware needs a host to be specified. Fix is to specify an appropriate host in the target cluster.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix detached volume inter-cluster migration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* cleanup unused method

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* review changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* vmware: allow attached volume migration using VmwareStorageMotionStrategy

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* find vm clusterid with multiple ROOT volumes

VM can have multiple ROOT volumes and some can be on zone-wide store therefore iterate over all of them till a cluster ID is found.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix successive storage migration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix intercluster check

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor vm cluster, host method

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove inter-pod check

Added by mistake, VMware won't have pods

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address review comment

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-24 18:55:25 +05:30
slavkap b4ee4acaf3
server: Fix volume state on migrate with migrateVirtualMachineWithVolume API call (#4934)
When invoking migrateVirtualMachineWithVolume API call and a strategy isn't found the volumes are left in Migrating state

This PR puts back the volumes to Ready state.
2021-04-22 14:30:18 +05:30
Gabriel Beims Bräscher de557663ec
Migrate/Stop VMs with local storage when preparing host for maintenance (#4212) 2021-04-19 09:41:42 +02:00
Rohit Yadav f42024714c Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-04-15 16:17:13 +05:30
Rohit Yadav 0302750aac
vmware: Add support for VMware 7 (#4300) 2021-04-15 16:10:14 +05:30
Harikrishna f00b5fc7ac
server: Fix for the issue of recover VM not able to attach the data disks which are there before destroy in case of VMware (#4493)
This PR fixes: #4462

Problem Statement:
In case of VMware, when a VM having multiple data disk is destroyed (without expunge) and tried to recover the VM then the previous data disks are not attached to the VM like before destroy. Only root disk is attached to the VM.

Root cause:
All data disks were removed as part of VM destroy. Only the volumes which are selected to delete (while destroying VM) are supposed to be detached and destroyed.

Solution:
During VM destroy, detach and destroy only volumes which are selected during VM destroy. Detach the other volumes during expunge of VM.
2021-04-15 12:50:53 +05:30
Daniel Augusto Veronezi Salvador 911376e495
Improve logs on KVMHostActivityChecker (#4706)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-04-13 08:27:25 +02:00
Abhishek Kumar cce736709e Merge remote-tracking branch 'apache/4.15' 2021-04-12 11:43:57 +05:30
Pearl Dsilva a64ad9d9b7
server: Prevent vm snapshots being indefinitely stuck in Expunging state on deletion failure (#4898)
Fixes #4201

This PR addresses the issue of a vm snapshot being indefinitely stuck is Expunging state in case deletion fails. 

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-04-12 08:09:37 +05:30
Abhishek Kumar fdefee75ff
vmware: fix inter-cluster stopped vm and volume migration (#4895)
Fixes #4838

For inter-cluster migration without shared storage, VMware needs a host to be specified. Fix is to specify an appropriate host in the target cluster during a stopped VM migration. Also, find target datastore using the host in the target cluster.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-10 13:22:28 +05:30
Rohit Yadav 22f6c19248 Merge remote-tracking branch 'origin/4.15' 2021-04-09 13:21:07 +05:30
Abhishek Kumar 6d0d7012f1
api,engine/schema,server: vm readonly details rename (#4671)
* api,engine/schema,server: vm readonly details rename

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: update api response key

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-09 13:20:11 +05:30
Rohit Yadav ca8920dd36 Merge remote-tracking branch 'origin/4.14' into 4.15 2021-04-09 13:17:39 +05:30
Abhishek Kumar d8c6e00498
hypervisor: XCP-ng 8.2 support (#4672)
Adds new/missing guest os mappings for XCP-ng/Xenserver 8.1
Copy guest OS mappings from XCP-ng/Xenserver 8.1 for XCP-ng/Xenserver 8.2
Adds Ubuntu 20.04 guest os mapping for XCP-ng/Xenserver 8.2

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-09 13:12:06 +05:30
Abhishek Kumar cd60b8d97d
host-allocator: check capacity for suitable hosts (#4884)
Fixes #4517

Adds capacity checks for RandomAllocator (host allocator)

Factors out host cpu capability and capacity check wrt serviceoffering code into CapacityManager.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-09 12:35:58 +05:30
Rohit Yadav 7270ca7e25 Merge remote-tracking branch 'origin/4.14' into 4.15 2021-04-06 12:51:26 +05:30
Gabriel Beims Bräscher cb91a769d3
Fix npe when migrating vm with volume (#4698) (#4775)
Cherry-pick commit 59fba4916b and fix conflict.

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2021-04-06 11:54:29 +05:30
Pearl Dsilva 0dbeb262e4
server: Support for persistence mode in L2 networks (#4561)
This PR aims at introducing persistence mode in L2 networks and enhancing the behavior in Isolated networks
Doc PR apache/cloudstack-documentation#183

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-04-05 14:37:11 +05:30
Rohit Yadav d4635e3442 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-04-01 14:35:01 +05:30
Wei Zhou 63c91c1458
server: Fix network statistics for vpc (#3944)
This contains 3 main changes
(1) add NETWORK_STATS_ethX for all nics with public ips in VPC VRs (current: NETWORK_STATS_eth1)
(2) DO NOT create records in user_statistics for each VPC tier (only one record per public nic per VPC VR)
(3) send NetworkUsageCommand before unplugging a NIC with public IPs from VPC VR
2021-04-01 12:43:06 +05:30
Rakesh 76ba5c62d9
server: Fix displaying public IP address of shared networks (#4675)
Public IP addresses dedicated to one domain should not be accessed
by other domains. Also, root admin should be able to display all
public ip addresses in system.

Currently following issues exist

1. Public IP address assigned to one domain can be accessed by
other sibling domains

If use.system.public.ip is false then child domains should not
see public ip of ROOT domain

Before fix
```
(test1) mgt01 > list publicipaddresses listall=true fordisplay=true allocatedonly=false forvirtualnetwork=true filter=ipaddress,
{
  "count": 59,
  "publicipaddress": [
```

After fix

```
(test) mgt01 > list publicipaddresses listall=true fordisplay=true allocatedonly=false forvirtualnetwork=true filter=ipaddress,
{
  "count": 10,
```
2021-04-01 12:39:01 +05:30
Rohit Yadav c1a02e1697 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-03-29 16:34:22 +05:30
Wei Zhou b8884efa7f
server: create DB entry for storage pool capacity when create storage pool (#4805)
* server: create DB entry for storage pool capacity when create storage pool

* Revert "server: create DB entry for storage pool capacity when create storage pool"

This reverts commit e790167bfe.

* server: create DB entry for storage pool capacity when create zone-wide storage pools
2021-03-29 16:21:24 +05:30
Abhishek Kumar d763169b1c
Restore VMware VM naming convention option (#4581)
* initial chanes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* make check explicit for instance name flag

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* allow hiding vm details (in ui)

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* condition based on name instead of displayname

Signed-off-by: Abhishek Kumar <abhishek.kumar@shapeblue.com>
2021-03-29 16:13:14 +05:30
Daan Hoogland 8903647187 Merge branch '4.15' 2021-03-25 17:48:45 +01:00
Rohit Yadav 9b1d1e6de3
systemvmtemplate: new template for 4.15.1 (#4793)
Update new systemvmtemplate for 4.15.1.0; synced:
http://download.cloudstack.org/systemvm/4.15/

A new template is necessary due to many security fixes over the last year, the 4.15.0 systemvmtemplate was created about a year ago.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-03-24 17:08:46 +05:30
Rohit Yadav 9f730eabfa Merge remote-tracking branch 'origin/4.15' 2021-03-24 12:46:24 +05:30
Pearl Dsilva 546bf3d5a2
server: Update vm_template table to set template as removed on deletion (#4748)
* Update vm_template table removed field when template is deleted

* Update method name

* address comment

* Extracted code to separate methods

* Address test failure

* refactor test cleanup

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-03-24 12:41:03 +05:30
Rohit Yadav 775de36688 Merge remote-tracking branch 'origin/4.15' 2021-03-17 17:46:16 +05:30
Pearl Dsilva 136252d65d
server: Maintain order or project owners added to account (#4822)
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-03-17 17:36:46 +05:30
Rohit Yadav c8f7c0d823 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-03-11 13:33:44 +05:30
Michael 1cfb44994f
db: add schema upgrade from 4.15.0.0 to 4.15.1.0 (#4574) 2021-03-11 13:24:29 +05:30
Daniel Augusto Veronezi Salvador 59fba4916b
Fix npe when migrating vm with volume (#4698)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-03-08 17:56:56 +01:00
sureshanaparti 81dfcbb5f5
server: Use ACPI event to reboot VM on KVM, and Use 'forced' reboot option to stop and start the VM(s) (#4681)
* Updated libvirt's native reboot operation for VM on KVM using ACPI event, and Added 'forced' reboot option to stop and start the VM (using rebootVirtualMachine API)

* Added 'forced' reboot option for System VM and Router

- New parameter 'forced' in rebootSystemVm API, to stop and then start System VM
- New parameter 'forced' in rebootRouter API, to force stop and then start Router

* Added force reboot tests for User VM, System VM and Router
2021-03-06 14:58:56 +05:30
sureshanaparti 45e77dd6f0
server: Clean up the duplicate volume when the destination managed volume creation failed on migrate volume operation (#4730)
Duplicated volumes after failed migration in Allocated state

Fix: Clean up the duplicate volume when the destination managed volume creation failed on migrate volume operation
2021-03-03 13:30:08 +05:30
Rohit Yadav fa067e02a7 Updating pom.xml version numbers for release 4.14.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-03-02 12:32:27 +05:30
Rohit Yadav 77290df0d5 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-26 12:09:11 +05:30
Abhishek Kumar 88337bdea4
server: fix finding pools for volume migration (#4693)
While finding pools for volume migration list following compatible storages:
- all zone-wide storages of the same hypervisor.
- when the volume is attached to a VM, then all storages from the same cluster as that of VM.
- for detached volume, all storages that belong to clusters of the same hypervisor. 

Fixes #4692 
Fixes #4400
2021-02-25 22:13:50 +05:30
Rakesh 787491871a
server: Look for active templates for VR deployment (#4047)
If the template from which VR is created got deleted, the state
is set to inactive and removed to null.
Since the template is already deleted, the VR can't be created
using this template again.

If someone restarts network with cleanup then it will try to
deploy the vr from the old non existing template again.
So search only for active template which are not yet deleted.
2021-02-25 22:05:31 +05:30
Rohit Yadav 1d8a4978e9 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-24 17:39:12 +05:30
Wei Zhou 5a3ae159ca
upgrade: check systemvm template before db changes (#4582)
* Upgrade: check systemvm template before db changes

* Upgrade: move some codes to a separated method

* #4582 add txn.commit()
2021-02-24 16:26:31 +05:30
sureshanaparti eba186aa40
storage: New Dell EMC PowerFlex Plugin (formerly ScaleIO, VxFlexOS) (#4304)
Added support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack (for KVM hypervisor) and enabled VM/Volume operations on that pool (using pool tag).
Please find more details in the FS here:
https://cwiki.apache.org/confluence/x/cDl4CQ

Documentation PR: apache/cloudstack-documentation#169

This enables support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack

Other improvements addressed in addition to PowerFlex/ScaleIO support:

- Added support for config drives in host cache for KVM
	=> Changed configuration "vm.configdrive.primarypool.enabled" scope from Global to Zone level
	=> Introduced new zone level configuration "vm.configdrive.force.host.cache.use" (default: false) to force host cache for config drives
	=> Introduced new zone level configuration "vm.configdrive.use.host.cache.on.unsupported.pool" (default: true) to use host cache for config drives when storage pool doesn't support config drive
	=> Added new parameter "host.cache.location" (default: /var/cache/cloud) in KVM agent.properties for specifying the host cache path and create config drives on the "/config" directory on the host cache path
	=> Maintain the config drive location and use it when required on any config drive operation (migrate, delete)

- Detect virtual size from the template URL while registering direct download qcow2 (of KVM hypervisor) templates

- Updated full deployment destination for preparing the network(s) on VM start

- Propagate the direct download certificates uploaded to the newly added KVM hosts

- Discover the template size for direct download templates using any available host from the zones specified on template registration
	=> When zones are not specified while registering template, template size discovery is performed using any available host, which is picked up randomly from one of the available zones

- Release the VM resources when VM is sync-ed to Stopped state on PowerReportMissing (after graceful period)

- Retry VM deployment/start when the host cannot grant access to volume/template

- Mark never-used or downloaded templates as Destroyed on deletion, without sending any DeleteCommand
	=> Do not trigger any DeleteCommand for never-used or downloaded templates as these doesn't exist and cannot be deleted from the datastore

- Check the router filesystem is writable or not, before performing health checks
	=> Introduce a new test "filesystem.writable.test" to check the filesystem is writable or not
	=> The router health checks keeps the config info at "/var/cache/cloud" and updates the monitor results at "/root" for health checks, both are different partitions. So, test at both the locations.
	=> Added new script: "filesystem_writable_check.py" at /opt/cloud/bin/ to check the filesystem is writable or not

- Fixed NPE issue, template is null for DATA disks. Copy template to target storage for ROOT disk (with template id), skip DATA disk(s)

* Addressed some issues for few operations on PowerFlex storage pool.

- Updated migration volume operation to sync the status and wait for migration to complete.

- Updated VM Snapshot naming, for uniqueness in ScaleIO volume name when more than one volume exists in the VM.

- Added sync lock while spooling managed storage template before volume creation from the template (non-direct download).

- Updated resize volume error message string.

- Blocked the below operations on PowerFlex storage pool:
  -> Extract Volume
  -> Create Snapshot for VMSnapshot

* Added the PowerFlex/ScaleIO client connection pool to manage the ScaleIO gateway clients, which uses a single gateway client per Powerflex/ScaleIO storage pool and renews it when the session token expires.

- The token is valid for 8 hours from the time it was created, unless there has been no activity for 10 minutes.
  Reference: https://cpsdocs.dellemc.com/bundle/PF_REST_API_RG/page/GUID-92430F19-9F44-42B6-B898-87D5307AE59B.html

Other fixes included:

- Fail the VM deployment when the host specified in the deployVirtualMachine cmd is not in the right state (i.e. either Resource State is not Enabled or Status is not Up)

- Use the physical file size of the template to check the free space availability on the host, while downloading the direct download templates.

- Perform basic tests (for connectivity and file system) on router before updating the health check config data
	=> Validate the basic tests (connectivity and file system check) on router
	=> Cleanup the health check results when router is destroyed

* Updated PowerFlex/ScaleIO storage plugin version to 4.16.0.0

* UI Changes to support storage plugin for PowerFlex/ScaleIO storage pool.
- PowerFlex pool URL generated from the UI inputs(Gateway, Username, Password, Storage Pool) when adding "PowerFlex" Primary Storage
- Updated protocol to "custom" for PowerFlex provider
- Allow VM Snapshot for stopped VM on KVM hypervisor and PowerFlex/ScaleIO storage pool

and Minor improvements in PowerFlex/ScaleIO storage plugin code

* Added support for PowerFlex/ScaleIO volume migration across different PowerFlex storage instances.

- findStoragePoolsForMigration API returns PowerFlex pool(s) of different instance as suitable pool(s), for volume(s) on PowerFlex storage pool.
- Volume(s) with snapshots are not allowed to migrate to different PowerFlex instance.
- Volume(s) of running VM are not allowed to migrate to other PowerFlex storage pools.
- Volume migration from PowerFlex pool to Non-PowerFlex pool, and vice versa are not supported.

* Fixed change service offering smoke tests in test_service_offerings.py, test_vm_snapshots.py

* Added the PowerFlex/ScaleIO volume/snapshot name to the paths of respective CloudStack resources (Templates, Volumes, Snapshots and VM Snapshots)

* Added new response parameter “supportsStorageSnapshot” (true/false) to volume response, and Updated UI to hide the async backup option while taking snapshot for volume(s) with storage snapshot support.

* Fix to remove the duplicate zone wide pools listed while finding storage pools for migration

* Updated PowerFlex/ScaleIO volume migration checks and rollback migration on failure

* Fixed the PowerFlex/ScaleIO volume name inconsistency issue in the volume path after migration, due to rename failure
2021-02-24 14:58:33 +05:30
Pearl Dsilva aa01580381
network: Specify IP for VR in shared networks (#4503)
This PR enables admins to specify IP for a VR in a shared network.
2021-02-18 13:54:09 +05:30
Pearl Dsilva b6fe9f99eb
Network Offering: Allow enabling network and vpc offering during creation (#4564)
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-02-13 10:19:06 +00:00
Sina Kashipazha 543f9827ff
add creation date as a value for domains and accounts. (#4649) 2021-02-13 10:07:53 +00:00
Abhishek Kumar d6e8b53736
vmware: vm migration improvements (#4385)
- Fixes inter-cluster migration of VMs
- Allows migration of stopped VM with disks attached to different and suitable pools
- Improves inter-cluster detached volume migration
- Allows inter-cluster migration (clusters of same Pod) for system VMs, VRs on VMware
- Allows storage migration for stopped system VMs, VRs on VMware within same Pod if StoragePool cluster scopetype

Linked Primate PR: https://github.com/apache/cloudstack-primate/pull/789 [Changes merged in this PR after new UI merge]
Documentation PR: https://github.com/apache/cloudstack-documentation/pull/170

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-02-12 12:41:41 +05:30
Rohit Yadav 66f0beda5f Updating pom.xml version numbers for release 4.14.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-08 16:24:09 +05:30
Rohit Yadav ba127dab3e Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-05 16:02:26 +05:30
Rohit Yadav 6bde1384ff Merge remote-tracking branch 'origin/4.14' into 4.15
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-05 16:01:01 +05:30
Wei Zhou 78f73c1bc6
server: Fix update capacity for hosts take long time if there are many service offerings (#4623)
Steps to reproduce the issue:

(1)Create 10000 service offerings (by db changes below or cloudmonkey).

```
DROP PROCEDURE IF EXISTS cloud.insert_service_offering;

DELIMITER $$
CREATE PROCEDURE cloud.insert_service_offering()
BEGIN
  DECLARE count INT DEFAULT 10000;
  SET @offeringid = (select max(id)+1 from disk_offering);

  WHILE count > 0 DO
    INSERT INTO disk_offering (id,name,uuid,display_text,disk_size,type,created) values (@offeringid,'test-offering-wei',uuid(), 'test-offering-wei',0,'Service',now());
    INSERT INTO service_offering (id,cpu,speed,ram_size) values (@offeringid, 1, 500,256);
    SET @offeringid = @offeringid + 1;
    SET count = count - 1;
  END WHILE;
END $$
DELIMITER ;

CALL cloud.insert_service_offering();

mysql> CALL cloud.insert_service_offering();
Query OK, 0 rows affected (2 min 30.85 sec)
```

(2) Check the total time of periodical capacity check in cloudstack.

Without this patch, it spend 2.5 seconds (2 hosts)
```
2021-01-15 16:10:12,793 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-5d5f3b3b) (logid:f5eb68ba) Running Capacity Checker ...
2021-01-15 16:10:15,287 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-5d5f3b3b) (logid:f5eb68ba) Done running Capacity Checker ...
```

With this patch ,it spend 1.3 seconds (2 hosts)
```
2021-01-15 16:12:43,604 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-a2a7f3f1) (logid:f7e0a4c5) Running Capacity Checker ...
2021-01-15 16:12:44,927 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-a2a7f3f1) (logid:f7e0a4c5) Done running Capacity Checker ...
```

If there are 100 hosts, the total time will be reduced from 100+ seconds to around 10 seconds.
2021-02-04 14:43:57 +05:30
Daan Hoogland ff376d8187 Merge release branch 4.15 to master
* 4.15:
  server: select root disk based on user input during vm import (#4591)
  kvm: Use Q35 chipset for UEFI x86_64 (#4576)
  server: fix wrong error message when create isolated network without SourceNat (#4624)
  server: add possibility to scale vm to current customer offerings (#4622)
  server: keep networks order and ips while move a vm with multiple networks (#4602)
  server: throw exception when update vm nic on L2 network (#4625)
  doc: fix typo in install notes (#4633)
2021-02-01 09:58:52 +00:00
Daan Hoogland b6b778f003 Merge release branch 4.14 to 4.15
* 4.14:
  server: select root disk based on user input during vm import (#4591)
  kvm: Use Q35 chipset for UEFI x86_64 (#4576)
  server: fix wrong error message when create isolated network without SourceNat (#4624)
  server: add possibility to scale vm to current customer offerings (#4622)
  server: keep networks order and ips while move a vm with multiple networks (#4602)
  server: throw exception when update vm nic on L2 network (#4625)
  doc: fix typo in install notes (#4633)
2021-02-01 09:57:35 +00:00
Wei Zhou 313ae1f449
server: fix wrong error message when create isolated network without SourceNat (#4624)
This PR fixes wrong message when create isolated network without SourceNat.
2021-02-01 14:15:47 +05:30
Wei Zhou 1913c6854e
server: keep networks order and ips while move a vm with multiple networks (#4602)
This PR fixes an issue when move a vm from an account to another account.

Steps to reproduce the issue
(1) create a vm with multiple shared networks (in advanced zone, or advanced zone with security groups)
(2) create another account (in same domain who can also access the shared networks)
(3) move vm to new account, with a list of networkid

expected result: the vm has nics on the networks in same order as specified in API request, and nics have the same ips as before actual result: network order is not same as specified, ips are changed.
2021-02-01 14:14:20 +05:30
Rohit Yadav e6fef05f30 Merge remote-tracking branch 'origin/4.15' 2021-01-28 14:25:40 +05:30
Rohit Yadav 74bae56642 Merge remote-tracking branch 'origin/4.14' into 4.15
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-28 14:24:25 +05:30