Commit Graph

3275 Commits

Author SHA1 Message Date
Abhishek Kumar 1510b44f03
backport: add more unit tests and fix related to #327 (#378)
Adds:

- Fix for volume limit checks for disk offerings with multiple tags - When a VM is deployed with multiple disks having offerings with multiple tags the resource limit check may falter as currently it tries to check based on individual diskoffering. With this, change if offering d1 and d2 for volumes v1 and v2 both have tag1, server will check volume limits for tag tag1 using the combined size of v1 and v2.
- Fix for template tag hosts in random host allocator - May affect use of template tag, service offering tags and random host allocator together. The current code for the random host allocator falters while trying to find the host allocation. This was found and fixed during the addition of the unit test here, https://github.com/shapeblue/cloudstack-apple/pull/378/files#diff-bbf9baea014e5cc1dfe9e7d13467c9857208cfe65e93883721d88a6f0452f912
- Unit tests for changes in api,server,ui: tagged resource limits #327
2024-03-01 17:22:14 +05:30
Harikrishna 747d1101c1
New API "checkVolume" to check and repair any leaks or repair all issues (#362)
* Introduced a new API "checkVolumeAndRepair" that allows users or admins to check and repair if any leaks observed.
Currently this is supported only for KVM

* some fixes

* Added unit tests

* addressed review comments

* add repair volume while granting access

* Changed repair parameter to accept both leaks/all values

* Introduced new global setting volume.check.and.repair.before.use to do volume check and repair before VM start or volume attach operations

* Added volume check and repair changes only during VM start and volume attach operations

* Refactored the names to look similar  across the code

* Some code fixes

* remove unused code

* Renamed repair values

* Addressed review comments

* code refactored

* used volume name in logs

* Changed the API to Async and the setting scope to storage pool

* Fixed exit value handling with check volume command

* Fixed storage scope to the setting

* Fixed volume format issues

* Refactored the log messages

* Fix formatting
2024-02-29 14:40:40 +05:30
anniejili 2df750c2f4
Fixed query param for getDomainReservation. (#388)
Co-authored-by: Annie Li <ji_li@apple.com>
2024-02-28 08:10:16 -07:00
Vishesh f30e07b312
Fix host stuck in connecting state (#375)
* Fix host stuck in connecting state (#8502)

There are a lot of test failures due to test_vm_life_cycle.py in multiple PRs due to host not available for migration of VMs.
#8438 (comment)
#8433 (comment)
#7344 (comment)

While debugging I noticed that the hosts get stuck in Connecting state because MS is waiting for a response of the ReadyCommand from the agent. Since we take a lock on connection and disconnection, restarting the agent doesn't work. To fix this, we have to restart the MS or wait for ~1 hour (default timeout).

On the agent side, it gets stuck waiting for a response from the Script execution.

To reproduce, run smoke/test_vm_life_cycle.py (TestSecuredVmMigration test class to be specific). Once the tests are complete, you will notice that some hosts are stuck in Connecting state. And restarting the agent fails due to the named lock. Locks on DB can be checked using the below query.

SELECT *
FROM performance_schema.metadata_locks
INNER JOIN performance_schema.threads ON THREAD_ID = OWNER_THREAD_ID
WHERE PROCESSLIST_ID <> CONNECTION_ID() \G;

This PR adds a wait for the ready command and a timeout to the Script execution to ensure that the thread doesn't get stuck and the named lock from database is released.

* Externalise a few timeouts & fix timeout for hostSupportsUefi in libvirt ready command wrapper (#8547)

This PR fixes bug introduced in #8502. Timeout for script execution was set to 60 ms instead of 60s which resulted in host not getting UEFI enabled. This is a blocker for 4.19 release.

We do this by introducing a new agent parameter `agent.script.timeout` (default - 60 seconds) to use as a timeout for the script checking host's UEFI status.

We also externalize the timeout for the ReadyCommand by introducing a new global setting `ready.command.wait` (default - 60 seconds).

For ModifyStoragePoolCommand, we don't externalize the timeout to avoid confusion for the user. Since, the required timeout can vary depending on the provider in use and we are only setting the wait for default host listener for now. Instead, we reuse the global `wait` setting by dividing it by `5` making the default value of 6 minutes (1800/5 = 360s) for ModifyStoragePoolCommand.

Note: the actual time, the MS waits is twice the wait set for a Command. Check reference code below.
19250403e6/engine/orchestration/src/main/java/com/cloud/agent/manager/AgentAttache.java (L406-L442)

* fixup
2024-02-21 13:44:53 +05:30
Suresh Kumar Anaparti 89f93746ac
Storage plugin support to check if volume on datastore requires access for migration (#380)
* Check if volume on datastore requires access for migration, and grant/revoke volume access if requires

* Updated default implementation for requiresAccessForMigration method in PrimaryDataStoreDriver
2024-02-20 11:32:32 -07:00
Vishesh 8b01c0aa62
Update VM's state if powerstate & state are not in sync (#368)
* Update VM's state if powerstate & state are not in sync

* Add unit tests

* some code improvements for instacne / power state check

* Update power state after vm stop confirmation (as power state 'PowerOn' is kept after vm stop and not updated later)

* Reset the power state update counter before migrate, to allow power state sync to proper state / host

* Do not consider transitional states (Starting, Stopping) to check power state sync

* set powerstate to off for all vm types

---------

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-02-20 14:44:04 +05:30
Abhishek Kumar 6a9cdedda4
api,server,ui: tagged resource limits (#327)
Introduces the concept of tagged resource limits. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types,

Host limits

    user_vm
    cpu
    memory

Storage limits

    volume
    primary_storage

Following global settings can used to specify tags for which limit needs to be enforced,

    Host: resource.limit.host.tags
    Storage: resource.limit.storage.tags

Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI.

Enhances use of templatetag for VM deployment and template creation

Adds option to list disk offering with suitability flag for a virtualmachine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the reponse.
2024-02-07 17:35:15 +05:30
Suresh Kumar Anaparti b44710c8a9
Pass StoragePoolType object for poolType dao attribute - fixes conversion to DB column (#371) 2024-02-02 14:10:02 +05:30
Marcus Sorensen 40dd867198
Apple base418 storagepooltype as class (#351)
* StoragePoolType as a class

* Fix agent side StoragePoolType enum to class

* Handle StoragePoolType for StoragePoolJoinVO

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fix UserVMJoinVO for StoragePoolType

* fixed missing imports

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fixed equals for the enum.

* removed not needed try/catch for prepareAttribute

* Added license to the file.

* Implemented "supportsPhysicalDiskCopy" for storage adaptor. (#352)

Co-authored-by: mprokopchuk <mprokopchuk@apple.com>

* Add javadoc to StoragePoolType class

* Add unit test for StoragePoolType comparisons

* StoragePoolType "==" and ".equals()" fix.

* Fix for abstract storage adaptor set up issue

* review comments

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-01-25 14:58:44 +05:30
Suresh Kumar Anaparti 7fef155621
Remove sensitive params (VmPassword, etc) from VMWork log (#369)
* Remove sensitive params (VmPassword, etc) from VMWork log

* Added unit tests

* review comments
2024-01-24 17:49:20 +05:30
Suresh Kumar Anaparti e704b6e492
Fix reorder/list pools when cluster details are not set (#358)
* Fix reorder/list pools when cluster details are not set

* minor code improvements

* added unit tests
2024-01-18 15:29:00 +05:30
kishankavala 99939d22a7
CleanUp Async Jobs after mgmt server maintenance (#356)
* Cleanup Volume AsyncJob after mgmt server stop

* Clean Up Vm async job resources during mggmt server stop

* Use State.isTransitional method to identify trnsition states

* Add cleanup for Network Async Job

* Add license

* Added RevertSnapshotting to volume transition state. Fixed spacing code style

* Added transitional flag in Volume state

* Updated network event for failed job, (re)added cleanup for volumes created from snapshots, and some code improvements

* Added java doc for volume state constructor

* Fixed cleanup SNAPSHOT_ID entry in volume details for failed volumes created from snapshots

---------

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-01-09 17:54:26 +05:30
Harikrishna 28be74e0b9
Add lock mechanism considering template id, pool id, host id (#345)
* Add lock mechanism considering template id, pool id, host id

* Added missing lock
2023-12-06 13:51:25 +05:30
Vishesh b9c3752ce0
Fix: Select another pod if all hosts in the pod becomes unavailable (#339) 2023-11-07 15:11:21 +01:00
Vishesh a7c7a33131
Apple base418 agent lock during reconnect (#340)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-11-03 16:56:15 +01:00
Harikrishna 198e48c7c5 Fix VM snapshot size during storage capacity check (#8101)
(cherry picked from commit 0183e25279)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-26 12:20:37 +05:30
slavkap a768c96a6d Create snapshot from VM snapshot without memory for NFS/Local storage (#8117)
(cherry picked from commit 6ae3b73ca2)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-26 12:18:56 +05:30
Rohit Yadav 18e7276df0 storage: allow VM snapshots without memory for KVM when global setting allows (#8062)
This removes the conditional logic where comment notest to remove it
after PR #5297 is merged that is applicable for ACS 4.18+. Only when the
global setting is enabled and memory isn't selected, VM snapshot could
be allowed for VMs on KVM that have qemu-guest-agent running.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 8350ce5aa4)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-11 21:01:31 +05:30
Pearl Dsilva 73c86b8a30 VR live patching: Allow live patch of VPC VRs even if networks are in allocated / shutdown state (#7958)
(cherry picked from commit 951ba04cf0)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-09 19:15:32 +05:30
Rohit Yadav 1b35a5aa1c Fix build, missing imports
Note: these are however redundant, both on main branch and the private
branch across two classes.

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-28 14:31:24 +05:30
Abhishek Kumar 160a13a029 userdata: fix append scenarios (#7741)
Fixes case of appending userdata when both template and vm data are either shellscript or cloudconfig
Fixes error when appending gzip userdata
Fixes case when userdata manual text from VM is not getting decoded-encoded correctly.
Fixes case of appending multipart data when both template and vm data contain same format types.
Refactor - moved validateUserData method to UserDataManager class
Refactor userdata test to check resultant multipart userdata thoroughly

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit 729e6d1446)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-28 12:48:46 +05:30
Marcus Sorensen c01ed90569 Trigger out of band VM state update via libvirt event when VM stops (#7963)
* Trigger out of band VM state update via libvirt event when VM stops

* Add License headers, refactor nested try

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 3694667f50)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-28 12:22:15 +05:30
Nicolas Vazquez 62fce86234 Generate cloud-init multipart user data for template append policy (#7643)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit b1fc279872)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 16:50:41 +05:30
Nicolas Vazquez 4dfa38aae9 plugins: Add Custom hypervisor minimal changes (#7692)
Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.

This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts

(cherry picked from commit 8b5ba13b81)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 16:19:42 +05:30
Marcus Sorensen fe50018fba Allow configkey to set 'cloud-name' cloud-init metadata (#7964)
* Allow configkey to set 'cloud-name' cloud-init metadata

* Update engine/api/src/main/java/com/cloud/vm/VirtualMachineManager.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/network/NetworkModelImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/network/router/CommandSetupHelper.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Revert "Update server/src/main/java/com/cloud/network/router/CommandSetupHelper.java"

This reverts commit 8abc3e38c4.

* Revert "Update server/src/main/java/com/cloud/network/NetworkModelImpl.java"

This reverts commit 7f239be919.

* Rework/Fix review code suggestions

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
(cherry picked from commit 155a30748c)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:47:58 +05:30
Oscar Sandoval be2bea64a9 fix units label (#325)
(cherry picked from commit d708603b6346033f0d023edfebe85113fcc928eb)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:45:45 +05:30
Marcus Sorensen d8db09ac58 Publish event for VM.STOP when out of band stop is detected
Signed-off-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 72a3d54db0a0bb0afbcbee35abc1e98ef102aa49)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-27 15:41:47 +05:30
Nicolas Vazquez 20952b4842 Auto Enable/Disable KVM hosts (#7170)
* Auto Enable Disable KVM hosts

* Improve health check result

* Fix corner cases

* Script path refactor

* Fix sonar cloud reports

* Fix last code smells

* Add marvin tests

* Fix new line on agent.properties to prevent host add failures

* Send alert on auto-enable-disable and add annotations when the setting is enabled

* Address reviews

* Add a reason for enabling or disabling a host when the automatic feature is enabled

* Fix comment on the marvin test description

* Fix for disabling the feature if the admin has manually updated the host resource state before any health check result

(cherry picked from commit be66eb2a35)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-15 13:15:59 +05:30
Rohit Yadav c8f02a8b34 engine/orchestartion: fix/refactor to Account.Type.ADMIN
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-15 13:15:59 +05:30
Marcus Sorensen 0c192f3609 When VM start fails at host for admin, report error (#208)
* When VM start fails at host for admin, report error

Signed-off-by: Marcus Sorensen <mls@apple.com>

* Report ResourceUnavailableExceptions that result in InsufficientCapacityException to admin

* Update error message to be more straightforward

Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
(cherry picked from commit 8885d252f0)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-09-14 20:00:46 +05:30
Wei Zhou 4bdff06acd Updating pom.xml version numbers for release 4.18.1.0
Signed-off-by: Wei Zhou <weizhou@apache.org>
2023-09-07 08:50:50 +02:00
Marcus Sorensen 2cccd8f754
db: Add index on cluster_details.name for FirstFitPlanner speedup (#7922) 2023-08-31 21:43:23 +02:00
Wei Zhou 0dd6bb7c67
systemvm: update size and state of systemvm template during registration (#7874) 2023-08-25 16:46:32 +02:00
Rohit Yadav e964395bd4
vmware: improve solidfire storage plugin integration and fix cases (#3) (#7761)
This fixes the following cases in which Solidfire storage integration
caused issues when using Solidfire datadisks with VMware:

1. Take Volume Snapshot of Solidfire data disk
2. Delete an active Instance with Solidfire data disk attached
3. Attach used existing Solidfire data disk to a running/stopped VM
4. Stop and Start an instance with Solidfire data disks attached
5. Expand disk by resizing Solidfire data disk by providing size
6. Expand disk by changing disk offering for the Solidfire data disk

Additional changes:
- Use VMFS6 as managed datastore type if the host supports
- Refactor detection and splitting of managed storage ds name in storage
  processor
- Restrict storage rescanning for managed datastore when resizing

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-25 11:36:03 +05:30
Wei Zhou 3c38ed7a65
server: allow user to list available IPs on shared networks (#7898)
This fixes #7817
2023-08-24 21:42:01 +05:30
Wei Zhou e4117e68d2
engine/schema: fix duplicated guest OSes in 4.18.0.0 (#7799)
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-08-23 09:44:34 +02:00
Wei Zhou db0e7a65af
server: check hostId when attach disk to a Stopped vm with local storage (#7886)
This fixes #7834
2023-08-22 14:52:20 +05:30
Wei Zhou 78bdde9e98
AutoScaling: support Managed User Data (#7769) 2023-08-22 11:07:16 +02:00
Wei Zhou c8d6e50539
VMware: add support for 8.0b (8.0.0.2), 8.0c (8.0.0.3) (#7380)
* VMware: add support for 8.0b (8.0.0.2)

* VMware 8: add new guest os mappings in VirtualMachineGuestOsIdentifier

The full list can be found at https://developer.vmware.com/apis/1355/vsphere

* VMware: get guest os mappings of parent version

* VMware8: remove guest os mappings for 8.0.0.2

* VMware8: fix code smells

* vmware: remove annotations in VmwareVmImplementerTest which caused 0.0% code coverage

* VMware8: add a unit test case

* VMware: add support for 8.0c (8.0.0.3)

* VMware8: move to CloudStackVersion.getVMwareParentVersion

* VMware: add support for 8.0u1 (8.0.1.0)

* Copy engine/schema/src/main/java/com/cloud/upgrade/GuestOsMapper.java from PR 6979

* Copy engine/schema/src/main/java/com/cloud/storage/dao/GuestOSHypervisorDao.java from PR 6979

* VMware: ignore the last number in VMware versions

* VMware: copy guest os mapping from 8.0 to 8.0.1

* VMware: add unit tests in VmwareVmImplementerTest.java

* Copy engine/schema/src/test/java/com/cloud/upgrade/GuestOsMapperTest.java from PR 6979

* VMware8: retry vm poweron if fails due to exception "File system specific implementation of Ioctl[file] failed"

This fixes a weird issue on vmware8. When power on a vm, sometimes it fails due to error

2023-04-27 07:04:43,207 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-442:ctx-cdd42b03 10.0.32.133, job-105/job-106, cmd: StartCommand) (logid:8a24a607) StartCommand failed due to [Exception: java.lang.RuntimeException
Message: File system specific implementation of Ioctl[file] failed
].
java.lang.RuntimeException: File system specific implementation of Ioctl[file] failed
        at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:426)
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.powerOn(VirtualMachineMO.java:288)

in vmware.log on ESXi host, it shows

2023-04-27T09:20:41.713Z In(05)+ vmx - Power on failure messages: File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to lock the file
2023-04-27T09:20:41.713Z In(05)+ vmx - Cannot open the disk '/vmfs/volumes/7b29c876-ac102328/i-2-167-VM/ROOT-167.vmdk' or one of the snapshot disks it depends on.
2023-04-27T09:20:41.713Z In(05)+ vmx - Module 'Disk' power on failed.
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to start the virtual machine.

There is a KB article for it, but I still do not know why and how to fix it.
https://kb.vmware.com/s/article/1004232

* VMware: extract to method powerOnVM

* vmware: fix mistake in logs

* vmware8: use curl instead of wget to fix test failures

Traceback (most recent call last):
  File "/root/test_internal_lb.py", line 555, in test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
    self.execute_internallb_roundrobin_tests(vpc_offering)
  File "/root/test_internal_lb.py", line 641, in execute_internallb_roundrobin_tests
    client_vm, applb.sourceipaddress, max_http_requests)
  File "/root/test_internal_lb.py", line 497, in run_ssh_test_accross_hosts
    (e, clienthost.public_ip))
AssertionError: list index out of range: SSH failed for VM with IP Address: 10.0.52.187

and

sshClient: DEBUG: {Cmd: /usr/bin/wget -T3 -qO- --user=admin --password=password http://10.1.2.253:8081/admin?stats via Host: 10.0.52.188} {returns: ["/usr/bin/wget: '/usr/lib/libpcre.so.1' is not an ELF file", "/usr/bin/wget: can't load library 'libpcre.so.1'"]}

* VMware: correct guest OS names in hypervisor mappings for VMware 8.0

el9 and variants were introduced by https://github.com/apache/cloudstack/pull/7059
they are supported with guest os identifiers since VMware 8.0

see https://vdc-repo.vmware.com/vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/vim.vm.GuestOsDescriptor.GuestOsIdentifier.html

* VMware: add Ubuntu 20.04 and 22.04 support for vmware 7.0+

* PR7380: only add guest os mappings for Ubuntu 20.04

* PR7380: Correct RHEL9 guest os names and others for VMware 8.0

* PR7380: correct guest os names on 8.0.0.1 as well

* PR7380: remove Windows 12 and Windows Server 2025 which are not released yet
2023-08-17 10:42:42 +02:00
Wei Zhou 1605b2f0db
server: remove registered userdata when cleanup an account (#7777) 2023-08-10 14:33:59 +02:00
Rohit Yadav 4f85c5568f
engine/schema: fix CPU cap limitation for upgraded environment (#7808)
CPU cap limitation was enabled as part of
https://github.com/apache/cloudstack/pull/6420 that changes behaviour
for existing environments. The CPU cap limitation on KVM causes
systemvms to not start or be really slow in nested and virtualised
environments.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-07 10:19:33 +05:30
mprokopchuk ab0297ea9b
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted (#7760)
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted

* Update server/src/main/java/com/cloud/vm/UserVmManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

---------

Co-authored-by: Maxim Prokopchuk <mprokopchuk@apple.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-07 10:18:17 +05:30
kishankavala 57779a8586
Remove nic allocation txn during Vm deploy. During Vm deploy failure, this will allow cleanup of any nics that are successfully provisioned (#7809) 2023-08-04 08:42:38 +02:00
dahn 2bb079189c
make NoopDbUpgrade do a systemvm template check (#7564) 2023-07-31 16:08:14 +02:00
Wei Zhou f4a4417e4c
pom.xml: add property project.systemvm.template.location (#7706)
* pom.xml: add property project.systemvm.template.location

* pom.xml: update project.systemvm.template.location
2023-07-28 11:05:44 +02:00
dahn 73a269e3b3
guarantee MAC uniqueness (#7634)
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
2023-07-19 10:25:01 +02:00
Ben 981d74825a
Add L2 networks to Zones with SG (#7719) 2023-07-11 09:11:23 +02:00
Vishesh 594c70dde0
Sync precommit config from main (#7732)
Co-authored-by: John Bampton <jbampton@users.noreply.github.com>
Co-authored-by: dahn <daan@onecht.net>
2023-07-07 11:18:16 +02:00
Abhishek Kumar 939ee9e153
server,engine-orchestration: allocate vm without transaction (#7695)
When deploying a VM is failed during the allocation process it may leave the resources that have been already allocated before the failure. They will get removed from the database as the whole code block is wrapped inside a transaction twice but the server would not inform the network or storage plugins to clean up the allocated resources.
This PR removes Transactions during VM allocation which results in the allocated VM and its resource records being persisted in DB even during failures. When failure is encountered VM is moved to Error state. This helps VM and its resources to be properly deallocated when it is expunged either by a server task such as ExpungeTask or during manual expunge.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-07-06 14:04:38 +05:30
dahn acc6f4e725
NPE guard (#7691) 2023-07-06 09:29:42 +02:00
Harikrishna 0cbe77024a
Fix foreign key constraints and the mysql procedure that is used (#7381) 2023-07-06 08:43:49 +02:00
Abhishek Kumar 31dbdd0f5c
engine-orchestration: fix volume size resource count mismatch (#7666)
Fixes case of account/domain having negative storage count when there is a volume size difference while deploying volumes on certain stroages. In case of Powerflex volume size results in a multiple of 8. If user deploys a volume of 12GB it will result in 16GB in size. But currently CloudStack will not update resource count after deployment.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-07-03 13:00:32 +05:30
Wei Zhou 985f0ecb53
Tungsten: change conserve_mode of default network offering to 0 (#7511) 2023-06-26 13:36:36 +02:00
slavkap faaf72b1a4
Volume encryption support for StorPool plug-in (#7539)
Supported Virtual machine operations:
- live migration of VM to another host
- virtual machine snapshots (group snapshot without memory)
- revert VM snapshot
- delete VM snapshot
Supported Volume operations:
- attach/detach volume
- live migrate volume between two StorPool primary storages
- volume snapshot
- delete snapshot
- revert snapshot
2023-06-26 11:24:51 +02:00
Nicolas Vazquez c809201247
Fix: Volumes on lost local storage cannot be removed (#7594) 2023-06-23 12:22:15 +02:00
Harikrishna 40cc10a73d
Allow volume migrations in ScaleIO within and across ScaleIO storage clusters (#7408)
* Live storage migration of volume in scaleIO within same storage scaleio cluster

* Added migrate command

* Recent changes of migration across clusters

* Fixed uuid

* recent changes

* Pivot changes

* working blockcopy api in libvirt

* Checking block copy status

* Formatting code

* Fixed failures

* code refactoring and some changes

* Removed unused methods

* removed unused imports

* Unit tests to check if volume belongs to same or different storage scaleio cluster

* Unit tests for volume livemigration in ScaleIOPrimaryDataStoreDriver

* Fixed offline volume migration case and allowed encrypted volume migration

* Added more integration tests

* Support for migration of encrypted volumes across different scaleio clusters

* Fix UI notifications for migrate volume

* Data volume offline migration: save encryption details to destination volume entry

* Offline storage migration for scaleio encrypted volumes

* Allow multiple Volumes to be migrated with migrateVirtualMachineWithVolume API

* Removed unused unittests

* Removed duplicate keys in migrate volume vue file

* Fix Unit tests

* Add volume secrets if does not exists during volume migrations. secrets are getting cleared on package upgrades.

* Fix secret UUID for encrypted volume migration

* Added a null check for secret before removing

* Added more unit tests

* Fixed passphrase check

* Add image options to the encypted volume conversion
2023-06-21 11:57:05 +05:30
Abhishek Kumar 3748f32bc7
engine-orchestration,vmware: hypervisor migration during start vm migration (#7444)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-06-20 17:13:08 +02:00
Abhishek Kumar ef9dec11e7
guestos: add Windows Server 2022 (#7508)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-05-31 16:02:48 +02:00
Abhishek Kumar 32caf9057e
engine-storage: fix errored template becomes active (#7485)
* engine-storage: fix errored template becomes active

Fixes #7342

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-05-12 15:23:16 +02:00
Oscar Sandoval b6443a2b1f
increase log detail for limit checking, fix getDomainReservation() (#7506)
In troubleshooting ops issues we see logs like:

Maximum domain resource limits of Type 'user_vm' for Domain Id = 763 is exceeded: Domain Resource Limit = (1 bytes) 1, Current Domain Resource Amount = (0 bytes) 0, Requested Resource Amount = (1 bytes) 1."

However there is one missing value (currentResourceReservation) that is used in the calculation of limit check but it is not logged, which leads to confusion. Above we see we are using “0” and requested 1, with our limit being 1, but was rejected. Without logging all the values used in the calculation we don’t understand why it failed.

Additionally, if we had this log above it would be clearer that a second bug is occurring. When we query for domain level resource reservations in “getDomainReservation” the actual SearchBuilder is the listAccountAndTypeSearch, not the listDomainAndTypeSearch. As a result, when we call getDomainReservation the query returns any outstanding domain reservation for any account, as domain ID is not a valid filter for the account search.

This PR:

Increases detailed information in log for checking resource limit to include reservations information for functions: checkDomainResourceLimit() and checkAccountResourceLimit

Fixes getDomainReservation() to use listDomainAndTypeSearch instead of listAccountAndTypeSearch

Co-authored-by: Oscar Sandoval <osandovalocana@apple.com>
2023-05-12 12:53:18 +05:30
Marcus Sorensen ec0f8bddf6
Support local storage live migration for direct download templates (#7453)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-05-04 17:37:58 -03:00
dahn 8bbe2f7cb2
engine/schema: use junit 4 (same as the rest of the project tests) 2023-04-27 11:36:27 +02:00
Abhishek Kumar afccde5cfc
server,engine-schema: fix listing service offering for vm scale (#7398)
Fixes #7389

Fixes listing of service offerings for VM scale when the current offering has  `disk_offering_strictness=true`

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-04-17 14:19:20 +05:30
Harikrishna Patnala 2a68259d0d Merge remote-tracking branch 'origin/4.17' into 4.18 2023-04-17 11:46:17 +05:30
Harikrishna 96897f07f2
Set hypervisor hostname via userdata only when the global settings are allowed (#7360)
* Set hypervisor hostname via userdata only when the corresponding global settings are enabled

* reformat code
2023-04-17 11:25:56 +05:30
Nicolas Vazquez 2dc016adde
Fix for direct download templates with multiple bypassed references (#7400)
This PR fixes an issue observed on multiple zones and direct download templates on KVM, in which a template gets multiple records on the template_store_ref table. When this happens, the template cannot be used as direct download. In case of a system VM template using direct download, system VM deployments fail
2023-04-13 12:48:29 +05:30
Harikrishna 65512af779
server: Roles search by keyword (#7384)
This PR fixes #7362 and also other search criteria to use the name as an exact search where keyword is also there.

Made UI changes for roles search to make use of keyword instead of name.
2023-04-11 22:24:29 +05:30
Harikrishna b774ee5d11
vmware: Datastore cluster synchronization should check if the child datastores are in UP state or not (#7385)
This fix ensures when datastore cluster in VMware is added as a primary storage pool in CloudStack then all the child datastores (which already exists in CS) should be in Up state.

For example:

1. Datastore Cluster DS has two child datastores A and B in vCenter. (B is already added as a storage pool in CloudStack)
2. Now try to add datastore cluster DS into CloudStack as a primary storage pool
3. CloudStack tries to add child datastores A and B in CloudStack, since B is already there in CloudStack, it will reuse the existing storagepool entry and will keep under parent Storage pool DS.

During Step 3 we are now checking if B is Up state or not.
2023-04-11 22:23:12 +05:30
Abhishek Kumar ce981574c3 Merge remote-tracking branch 'apache/4.17' into 4.18 2023-03-31 17:00:00 +05:30
Abhishek Kumar 9aee625ab4
orchestration: fix error on deleted template vm start (#7327)
* server: fix error on deleted template vm start

When a VM is deployed with start flag as false and the template is deleted before the VM start, NPE is obeserved it is started for the first time.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-03-31 15:14:09 +05:30
Rohit Yadav 74a414e76f Fix build failure and 4.18.0.0->4.18.1.0 upgrade path
Unblocks smoketests/health check runs on 4.18 branch.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-03-16 13:18:26 +05:30
Daan Hoogland 05cda2729f Updating pom.xml version numbers for release 4.18.1.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-15 19:38:14 +01:00
Daan Hoogland 0574087284 Updating pom.xml version numbers for release 4.18.0.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2023-03-11 09:35:41 +01:00
SadiJr ad64ee3090
[Usage] Fix wrong usage_type (#7229) 2023-03-03 13:10:38 +01:00
Wei Zhou 8592de95fa
Move PassphraseVO to use String instead of byte[] to support Encrypt annotation (#7302)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-03-03 13:08:17 +01:00
dahn aa881f6bf3
avoid hard coded guest OS IDs (#7279) 2023-02-23 15:19:45 +01:00
Wei Zhou e61051e4fc
SQL: Remove snapshot references if primary storage pool has been removed (#7205) 2023-02-21 14:48:36 +01:00
Harikrishna 7484636997
Userdata sql changes using procedures (#7255) 2023-02-20 12:07:52 +01:00
Wei Zhou 80c1b6b2a2
engine/schema: fix IDEMPOTENT_ADD_FOREIGN_KEY procedure does not work on mysql8 (#7252) 2023-02-17 11:46:09 +01:00
Wei Zhou 8ef35466de
Tungsten: fix functional issues (#7173)
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-02-13 09:15:28 +01:00
Harikrishna a3feccf70c
User two factor authentication (#6924)
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-02-13 09:14:17 +01:00
Daan Hoogland 1b89a22ab0 fix merge conflicts of merge branch '4.17' 2023-02-08 16:52:35 +01:00
Daan Hoogland 2149e82856 Merge release branch 4.17 to main 2023-02-08 15:50:11 +01:00
Wei Zhou 66f351543a
server: do not deploy or upgrade vm with inactive service offering (#7063)
Co-authored-by: Stephan Krug <stekrug@icloud.com>
2023-02-08 09:40:30 +01:00
Abhishek Kumar 6d287793a5
guestos: add el9 distros (#7059)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-02-03 12:46:53 +01:00
Bryan Lima 5c20e7844c
Quota VM_DISK tariff calculation (#7152)
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2023-02-03 08:43:05 +01:00
Wei Zhou 62e342c1bc
utils,framework/db: Introduce new database encryption cipher based on AesGcmJce (#7003) 2023-02-02 16:25:49 +01:00
Nicolas Vazquez 89bf4750ab
Add console session cleanup task (#7132) 2023-02-01 16:53:54 +01:00
David Jumani c774b865c9
Tungsten integration (#7065)
Co-authored-by: rtodirica <rtodirica@ena.com>
Co-authored-by: Huy Le <huylm@unitech.vn>
Co-authored-by: radu-todirica <Radu.Todirica@ness.com>
Co-authored-by: Huy Le <minh.le@ext.ewerk.com>
Co-authored-by: Simon Weller <siweller77@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-02-01 09:19:53 +01:00
Suresh Kumar Anaparti d8c7e34b38
Improve global settings UI to be more intuitive/logical (#5797)
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: nvazquez <nicovazquez90@gmail.com>
Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-01-31 11:23:43 +01:00
Abhishek Kumar 3b6ce97097
infra: edge zones (#6840)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-01-31 09:36:45 +01:00
nvazquez 83c2bfacd8
Merge branch '4.17' 2023-01-30 07:53:58 -03:00
Abhishek Kumar 028ca74fb6
ui,server,api: resource metrics improvements (#6803)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-01-30 09:48:03 +01:00
Nicolas Vazquez c78a777d3a
Fix: memory leak on volume allocation (#7136) 2023-01-30 09:44:50 +01:00
João Jandre 61a722548f
Create API to reassign volume (#6938) 2023-01-27 11:10:56 +01:00
slavkap d288bb0c78
KVM support of iothreads and IO driver policy (#6909) 2023-01-25 12:34:05 +01:00
Rohit Yadav ea14c59210
vmware: support vsphere 8.0 and 8.0.0.1 versions (#7119) 2023-01-25 09:20:21 +01:00
Wei Zhou 42f0d758c1
vpc/network: set mtu of networks/vpcs/nics during 4.18 upgrade (#7124) 2023-01-24 09:51:08 +01:00
Abhishek Kumar 2dd29558fe
server: fix listnetworkofferings with domain, refactor listvpofferings (#6748) 2023-01-24 09:30:12 +01:00
John Bampton d74f64a2e1
Use lowercase HTTP header field names so we are compatible with HTTP/2 (#7006) 2023-01-23 11:17:54 +01:00