Don't use LibvirtServerDiscoverer's processHostAdded() in CustomServerDiscoverer
(cherry picked from commit ba0dfea6fd469063d6a154d2056f250feed84f10)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* UI: Fix user role login due to missing API access on custom hypervisor name
* Refactor to include the custom HW display name as part of the response of listCapabilities API
* Add since parameter
(cherry picked from commit 940733cedf)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor
This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack
The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.
CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately
This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)
Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin
- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
- In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
- Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
- Local storage (use hypervisor hosts local storage as primary storage)
- Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.
This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.
- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts
(cherry picked from commit 8b5ba13b81)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Allow configkey to set 'cloud-name' cloud-init metadata
* Update engine/api/src/main/java/com/cloud/vm/VirtualMachineManager.java
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
* Update server/src/main/java/com/cloud/network/NetworkModelImpl.java
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
* Update server/src/main/java/com/cloud/network/router/CommandSetupHelper.java
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
* Revert "Update server/src/main/java/com/cloud/network/router/CommandSetupHelper.java"
This reverts commit 8abc3e38c4.
* Revert "Update server/src/main/java/com/cloud/network/NetworkModelImpl.java"
This reverts commit 7f239be919.
* Rework/Fix review code suggestions
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
(cherry picked from commit 155a30748c)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Auto Enable Disable KVM hosts
* Improve health check result
* Fix corner cases
* Script path refactor
* Fix sonar cloud reports
* Fix last code smells
* Add marvin tests
* Fix new line on agent.properties to prevent host add failures
* Send alert on auto-enable-disable and add annotations when the setting is enabled
* Address reviews
* Add a reason for enabling or disabling a host when the automatic feature is enabled
* Fix comment on the marvin test description
* Fix for disabling the feature if the admin has manually updated the host resource state before any health check result
(cherry picked from commit be66eb2a35)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
If the original value is `false`, and search build is configured without the condition. Now change the value to `true`, it will not get effective due to missing condition.
* Don't allow inadvertent deletion of hidden details via API
* Update VM details unit test ensuring system/hidden details not removed
* Update test/integration/component/test_update_vm.py
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
* Replace Hashtable with LinkedHashMap in createIsoResponse
This change replaces the use of Hashtable with LinkedHashMap in the `createIsoResponse` method of `ViewResponseHelper`.
The reason for this modification is to maintain the insertion order of entries, which isn't the case with Hashtable.
This could lead to more predictable results and behaviors in calling methods.
* Replace Hashtable with LinkedHashMap in view response creation methods
Changed Hashtable to LinkedHashMap in various response creation methods within ViewResponseHelper class.
This modification ensures an ordered iteration which is beneficial for scenarios where the insertion order of responses needs to be maintained consistently.
---------
Co-authored-by: Sina Kashipazha <soreana@users.noreply.github.com>
This PR addresses rare case of potential overlap of resource reservation and resource count.
For different resource types there could be some delay between incrementing of the resource count and clearing of the earlier done reservation. This may result in failures when there are parallel deployments happening.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
When deploying a VM is failed during the allocation process it may leave the resources that have been already allocated before the failure. They will get removed from the database as the whole code block is wrapped inside a transaction twice but the server would not inform the network or storage plugins to clean up the allocated resources.
This PR removes Transactions during VM allocation which results in the allocated VM and its resource records being persisted in DB even during failures. When failure is encountered VM is moved to Error state. This helps VM and its resources to be properly deallocated when it is expunged either by a server task such as ExpungeTask or during manual expunge.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
This PR resource throws exception with the correct error code and logs the error message when a resource allocation failure is encountered during resize volume operation.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Supported Virtual machine operations:
- live migration of VM to another host
- virtual machine snapshots (group snapshot without memory)
- revert VM snapshot
- delete VM snapshot
Supported Volume operations:
- attach/detach volume
- live migrate volume between two StorPool primary storages
- volume snapshot
- delete snapshot
- revert snapshot
This PR adds a check on host's status. Without this if the agent is not in Up or Connecting state, expunging of a VM fails.
Steps to reproduce:
- Enable vm.configdrive.force.host.cache.use in Global Configuration.
- Create a L2 network with config drive
- Deploy a vm with the L2 network created in previous step
- Stop the vm and destroy vm (not expunge it)
- Stop the cloudstack-agent on the VM's host
- Expunge the vm
Fixes: #7428
* Live storage migration of volume in scaleIO within same storage scaleio cluster
* Added migrate command
* Recent changes of migration across clusters
* Fixed uuid
* recent changes
* Pivot changes
* working blockcopy api in libvirt
* Checking block copy status
* Formatting code
* Fixed failures
* code refactoring and some changes
* Removed unused methods
* removed unused imports
* Unit tests to check if volume belongs to same or different storage scaleio cluster
* Unit tests for volume livemigration in ScaleIOPrimaryDataStoreDriver
* Fixed offline volume migration case and allowed encrypted volume migration
* Added more integration tests
* Support for migration of encrypted volumes across different scaleio clusters
* Fix UI notifications for migrate volume
* Data volume offline migration: save encryption details to destination volume entry
* Offline storage migration for scaleio encrypted volumes
* Allow multiple Volumes to be migrated with migrateVirtualMachineWithVolume API
* Removed unused unittests
* Removed duplicate keys in migrate volume vue file
* Fix Unit tests
* Add volume secrets if does not exists during volume migrations. secrets are getting cleared on package upgrades.
* Fix secret UUID for encrypted volume migration
* Added a null check for secret before removing
* Added more unit tests
* Fixed passphrase check
* Add image options to the encypted volume conversion
In troubleshooting ops issues we see logs like:
Maximum domain resource limits of Type 'user_vm' for Domain Id = 763 is exceeded: Domain Resource Limit = (1 bytes) 1, Current Domain Resource Amount = (0 bytes) 0, Requested Resource Amount = (1 bytes) 1."
However there is one missing value (currentResourceReservation) that is used in the calculation of limit check but it is not logged, which leads to confusion. Above we see we are using “0” and requested 1, with our limit being 1, but was rejected. Without logging all the values used in the calculation we don’t understand why it failed.
Additionally, if we had this log above it would be clearer that a second bug is occurring. When we query for domain level resource reservations in “getDomainReservation” the actual SearchBuilder is the listAccountAndTypeSearch, not the listDomainAndTypeSearch. As a result, when we call getDomainReservation the query returns any outstanding domain reservation for any account, as domain ID is not a valid filter for the account search.
This PR:
Increases detailed information in log for checking resource limit to include reservations information for functions: checkDomainResourceLimit() and checkAccountResourceLimit
Fixes getDomainReservation() to use listDomainAndTypeSearch instead of listAccountAndTypeSearch
Co-authored-by: Oscar Sandoval <osandovalocana@apple.com>
Fixes#6697
Allows the server to generate started and completed events for VM.CREATE event type when VM is deployed with startvm=false.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
This PR adjusts the DirectDownload certificate check initial delay. Since the time unit is in hours, I think it was a mistake to schedule the initial check to be in 60 hours after management servers start - the intention was likely 60 seconds. We had turned this feature on to run hourly, not realizing we would have to wait 2.5 days to see it first run!
Co-authored-by: Marcus Sorensen <mls@apple.com>
This PR fixes a null pointer edge case where a PowerFlex volume is attached to a VM.
In this edge case, a VM has been created, started, stopped, and then reimaged. VM has a last host ID but the newly attached volume does not yet have a storage pool assigned, it will be assigned on the VM start. However, we assume that if the VM's host is not null, we need to try to revoke access to the volume for the host. Since there is no storage pool yet for the volume, we hit a null pointer when checking to see if the volume's pool is PowerFlex.
This was affecting all storage types, I could reproduce it with local storage, since the null pointer is at the check for pool's type.
Co-authored-by: Marcus Sorensen <mls@apple.com>
This PR allows admin to filter resources by state for systemvms, router & storagepool. This is part of #7366 .
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This PR fixes#7362 and also other search criteria to use the name as an exact search where keyword is also there.
Made UI changes for roles search to make use of keyword instead of name.
This fix ensures when datastore cluster in VMware is added as a primary storage pool in CloudStack then all the child datastores (which already exists in CS) should be in Up state.
For example:
1. Datastore Cluster DS has two child datastores A and B in vCenter. (B is already added as a storage pool in CloudStack)
2. Now try to add datastore cluster DS into CloudStack as a primary storage pool
3. CloudStack tries to add child datastores A and B in CloudStack, since B is already there in CloudStack, it will reuse the existing storagepool entry and will keep under parent Storage pool DS.
During Step 3 we are now checking if B is Up state or not.
Fixes the case when userdata variable value contains '=' sign. This PR considers everything before occurrence of first '=' sign as key and remaining string as value.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* 4.17:
server: add/update user vm details for new root disk size (issue 7139) (#7145)
UI: fix missing networkid and zoneid when add instance from vpc network (#7151)
This PR allows securing the console access through CloudStack to the virtual machines running on KVM. The secure access is achieved through the generated certificates for the CA Framework in CloudStack, that provides mutual TLS connections between agents. These certificates are used to also secure the connection between the console proxies and the VNC ports for VM console access.
This feature is only supported on the KVM hypervisor
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+KVM+VNC+connection+using+the+CA+framework
Marcus Sorensen
Nicolas Vazquez
Abhishek Kumar
Wei Zhou
Fabricio Duarte
Sina Kashipazha
Charles Queiroz
dahn
João Jandre
mprokopchuk
Rohit Yadav
Harikrishna
Ben
Vishesh
slavkap
Oscar Sandoval
kishankavala
David Jumani
Suresh Kumar Anaparti
Stephan Krug