* UserData as first clas resource
* Few fixes
* Added userdata id in deploy VM flow
* Fixed userdata append scenario between template userdata and user provided userdata
* UI: added a new section for userdata
* Added userdata details to the deployVM cmd flow
* Write userdata details into metadata VR
* Updated template response and views
* UI: added userdata id to deloyvm wizard
* Added userdata list to deploy VM form
* Added userdata params to registeruserdata UI form
* Small fixes and added userdata to updateVM flow
* Fixed unit tests and imports
* Userdata Navigation in template view
* Added userdata denyoverride flag and userdata params section
* Added ToolTips and fixes deploy VM, register userdata form
* added userdata policy list to register template form
* Allow override append of userdata in deploy VM flow
* update userdata linking to template
* Few UI fixes in deploy VM and edit template form
* fixes in deploy VM form to support deny userdata policy
* Added unit tests for userdata
* Added unit tests for linking userdata to template
* Remove unused imports
* Move test file to proper files
* Fix unused imports
* Fix Userdata delete flow
* Few improvements in the code
* Adding marvin tests for userdata
* Fixed marvin tests for registerd userdata
* Added few more marvin tests for userdata
* Few code fixes
* Few more code fixes
* Added userdata details to register and upload iso forms
* Added userdata selection in deploy VM form for ISOs
* Add comments section to userdata
* Added new API to reset UserData of a VM along with UpdateVM API
* Added new UnitTests for resetVMUserData
* Added resetVMUserdata in UI for stopped VM
* Added blank values for userdata in edit template form
* Added ISO id to the linkuserdatatotemplate API
* Added validation to userdata params so that it wont contain any VR metadafile names
* Removed required param for iso id in linkuserdatatotemplate cmd
* Added length to userdata param
* remove delete cascade on user_vm and vm_template tables for userdata id foreign key reference.
* Fix custom userdata params for config drive by adding userdata file name and value metadata json
* Fix marvin test case
* added comments to marvin test cases
* Fix document link in UI
* Added a check while deleting the userdata, to see if no VMs are using the userdata
* Added unit tests
* Removed labels added during merge
* added success message for userdata registration
* Added Schema changes to 4160 to 4161 upgrade path
* Fixed imports and some errors
* Fix service offering uuid in mysql view
* UI changes wrt to 4.16 branch mostly related to vue2
* Some UI fixes
* Register userdata and update template form fixes
* Fixed compute.js related to userdata
* UI fixes and user_vm_view wrt sshkey
* Fixed update template form
* Fix deploy VM and userdata reset forms
* Fixed Register and upload template and ISO forms
* Fixed getting params for userdata from template in instance creation form
* Removed CloudZonesNetworkElement.java as part of rebase, which is actually removed in https://github.com/shapeblue/cloudstack-apple/pull/191
* Fixed userdata selection in deploy vm and reset userdata vm forms
* Fixed method calls after rebase
This PR introduces volume encryption option to service offerings and disk offerings. Fixes#136
There is a hypervisor component and a storage pool component. Hypervisors are responsible for being capable of running/using the encrypted volumes. Storage pools are responsible for being able to create, copy, resize, etc. Hypervisors will report encryption support in their details, storage pools are marked for encryption support by pool type.
The initial offering for experimental release of this feature will have support for encryption on Local, NFS, SharedMountPoint, and ScaleIO storage types.
When volumes choosing an encrypted offering are allocated to a pool, the pool type must be capable of supporting encryption and this is enforced.
When VMs are started and they have an encrypted volume, the hypervisor must be capable of supporting encryption. Also, if volumes are attached to running VMs, the attach will only work if the hypervisor supports encryption.
This change includes a few other minor changes - for example the ability to force the KVM hypervisor private IP. This was necessary in my testing of ScaleIO, where the KVM hypervisors had multiple IPs and the ScaleIO storage only functions if the hypervisor as a ScaleIO client matches IPs with what CloudStack sees as the hypervisor IP.
For experimental release of this feature, some volume workflows like extract volume and migrate volume aren't supported for encrypted volumes. In the future we could support these, as well as migrating from unencrypted to encrypted offerings, and vice versa.
It may also be possible to configure encryption specifics in the future, perhaps at the pool level or the offering level. Currently, there is only one workable encryption offering for KVM that is supported by Libvirt and Qemu for raw and qcow2 disk files, LUKS version 1. This PR ensures we at least store this encryption format associated with each volume, with the expectation that later we may have LUKS v2 volumes or something else. Thus we will have the information necessary to use each volume with Libvirt if/when other formats are introduced.
I think the most disruptive change here is probably a refactoring of the QemuImg utility to support newer flags like --object. I've tested the change against the basic Qemu 1.5.3 that comes with EL7 and I believe it is good, but it will be nice to see the results of some functional tests. Most of the other changes are limited to changing behavior only if volume encryption is requested.
Working on documentation for the CloudStack docs. One thing to note is that hypervisors that run the stock EL7 version of Qemu will not support encryption. This is tested to be detected and report properly via the CloudStack API/UI. I intend to like to have a support matrix in the CloudStack docs.
I may add a few more unit tests. I'd also like some guidance on having functional tests. I'm not sure if there's a separate framework, or if Marvin is still used, or what the current thing is.
* Add Qemu object flag to QemuImg create
* Add apache license header to new files
* Add Qemu object flag to QemuImg convert
* Set host details if hypervisor supports LUKS
* Add disk encrypt flag to APIs, diskoffering
* Schema upgrade 4.16.0.0 to 4.16.1.0 to support vol encryption
* Add Libvirt secret on disk attach, and refer to it in disk XML
* Add implementation of luks volume encryption to QCOW2 and RAW disk prep
* Start VMs that have encrypted volumes
* Add encrypt option to service offering and root volume provisioning
* Refactor volume passphrase into its own table and object
* CryptSetup, use key files to pass keys instead of command line
* Update storage types and allocators to select encryption support
* Allow agent.properties to define the hypervisor's private IP
* Implement createPhysicalDisk for ScaleIOStorageAdaptor
* UI: Add encrypt options to offerings
* UI module security updates
* Revert "UI module security updates" - belongs in base
This reverts commit a7cb7cf7f57aad38f0b5e5d67389c187b88ffd94.
* Add --target-is-zero support for QemuImg
* Allow qemu image options to be passed, API support convert encrypted
* Switch hypervisor encryption support detection to use KeyFiles
* Fixes for ScaleIO root disk encryption
* Resize root disk if it won't fit encryption header
* Use cryptsetup to prep raw root disks, when supported
* Create qcow2 formatting if necessary during initial template copy to ScaleIO
* Allow setting no cache for qemu-img during disk convert
* Use 1M sparse on qemu-img convert for zero target disks
* UI: Add volume encryption support to hypervisor details
* QemuImg use --image-opts and --object depending on version
* Only send storage commands that require encryption to hosts that support encryption
* Move host encryption detail to a static constant
* Update host selection to account for volume encryption support
Only attach volumes if encryption requirements are met
* Ensure resizeVolume won't allow changing encryption
* Catch edge cases for clearing passphrase when volume is removed
* Disable volume migration and extraction for encrypted volumes
* Register volume secret on destination host during live migration
* Fix configdrive path editing during live migration
* Ensure configdrive path is edited properly during live migration
* Pass along and store volume encryption format during creation
* Fixes for rebase
* Fix tests after rebase
* Add unit tests for DeploymentPlanningManagerImpl to support encryption
* Deployment planner tests for encryption support on last host
* Add deployment tests for encryption when calling planner
* Added Libvirt DiskDef test for encryption details
* Add test for KeyFile utility
* Add CryptSetup tests
* Add QemuImageOptionsTest
* add smoke tests for API level changes on create/list offerings
* Fix schema upgrade, do disk_offering_view first
* Fix UI to show hypervisor encryption support
* Load details into hostVO before trying to query them for encryption
* Remove whitespace in CreateNetworkOfferingTest
* Move QemuImageOptions to use constants for flag keys
* Set physical disk encrypt format during createDiskFromTemplate in KVM Agent
* Whitespace in AbstractStoragePoolAllocator
* Fix whitespace in VolumeDaoImpl
* Support old Qemu in convert
* Log how long it takes to generate a passphrase during volume creation
* Move passphrase generation to async portion of createVolume
* Revert "Allow agent.properties to define the hypervisor's private IP"
This reverts commit 6ea9377505f0e5ff9839156771a241aaa1925e70.
* Updated ScaleIO/PowerFlex storage plugin to support separate (storage) network for Host(KVM) SDC connection. (#144)
* Added smoke tests for volume encryption (in KVM). (#149)
* Updated ScaleIO pool unit tests.
* Some improvements/fixes for code smells (in ScaleIO storage plugin).
* Updated review changes for ScaleIO improvements.
* Updated host response parameter 'encryptionsupported' in the UI.
* Move passphrase generation for the volume to async portion, while deploying VM (#158)
* Move passphrase generation for the volume to async portion, while deploying VM.
* Updated logs, to include volume details.
* Fix schema upgrade, create passphrase table first
* Fixed the DB upgrade issue (as noticed in the logs below.)
DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLException: Failed to open the referenced table 'passphrase'
ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
* Fixes for snapshots with encrypted qcow2
Fixes#159#160#163
* Support create/delete encrypted snapshots of encrypted qcow2 volumes
* Select endpoints that support encryption when snapshotting encrypted volumes
* Update revert snapshot to be compatible with encrypted snapshots
* Disallow volume and template create from encrypted vols/snapshots
* Disallow VM memory snapshots on encrypted vols. Fixes#157
* Fix for TemplateManagerImpl unit test failure
* Support offline resize of encrypted volumes. Fixes#168
* Fix for resize volume unit tests
* Updated libvirt resize volume unit tests
* Support volume encryption on kvm only, and passphrase generation refactor (#169)
* Fail deploy VM when ROOT/DATA volume's offering has encryption enabled, on non-KVM hypervisors
* Fail attach volume when volume's offering has encryption enabled, on non-KVM hypervisors
* Refactor passphrase generation for volume
* Apply encryption to dest volume for live local storage migration
fixes#161
* Apply encryption to data volumes during live storage migration
Fixes#161
* Use the same encryption passphrase id for migrating volumes
* Pass secret consumer during storage migration prepare
Fix for #161
* Fixes create / delete volume snapshot issue, for stopped VMs
* Block volume snapshot if encrypted and VM is running
Fixes#159
* Block snap schedules on encrypted volumes
Fix for #159
* Support cryptsetup where luks type defaults to 2
Fixes#170
* Modify domain XML secret UUID when storage migrating VM
Fix for #172
* Remove any libvirt secrets on VM stop and post migration
Fix for #172
* Update disk profile with encryption requirement from the disk offering (#176)
Update disk profile with encryption requirement from the disk offering
and some code improvements
* Updated review changes / javadoc in ScaleIOUtil
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Add resource ID and resource type to event.
In UI, adds Events tab in resource view for the supporting resources.
Following SQL changes needed to support events with resource details in DB,
```
-- Alter event table to add resource_id and resource_type
ALTER TABLE `cloud`.`event`
ADD COLUMN `resource_id` bigint unsigned COMMENT 'ID of the resource associated with the even' AFTER `domain_id`,
ADD COLUMN `resource_type` varchar(32) COMMENT 'Account role in the project (Owner or Regular)' AFTER `resource_id`;
DROP VIEW IF EXISTS `cloud`.`event_view`;
CREATE VIEW `cloud`.`event_view` AS
SELECT
event.id,
event.uuid,
event.type,
event.state,
event.description,
event.resource_id,
event.resource_type,
event.created,
event.level,
event.parameters,
event.start_id,
eve.uuid start_uuid,
event.user_id,
event.archived,
event.display,
user.username user_name,
account.id account_id,
account.uuid account_uuid,
account.account_name account_name,
account.type account_type,
domain.id domain_id,
domain.uuid domain_uuid,
domain.name domain_name,
domain.path domain_path,
projects.id project_id,
projects.uuid project_uuid,
projects.name project_name
FROM
`cloud`.`event`
INNER JOIN
`cloud`.`account` ON event.account_id = account.id
INNER JOIN
`cloud`.`domain` ON event.domain_id = domain.id
INNER JOIN
`cloud`.`user` ON event.user_id = user.id
LEFT JOIN
`cloud`.`projects` ON projects.project_account_id = event.account_id
LEFT JOIN
`cloud`.`event` eve ON event.start_id = eve.id;
```
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* network: update ip in lb/pf/dnat tables when update vm nic ip
* Update: create 3 separated methods for DNAT/LB/PF update
* Update: invert if condition to reduce code indentation
* trace nics additions
* work queue patch for network to add
* add secondary key to job
* logging improvements and naming of field(s)
* several naming corrections
* extra check if net already exists for vm
* placeholder job with secondary object
* constraint on entering the same job multiple times
* error handling/warning message
* review comments applied
Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Wei Zhou <wei.zhou@shapeblue.com>
* Allow storage plugins to use storage/volume stats
Allow the rest of the storage plugins, not only ScaleIO to implement and
use the storage/volume statistics directly instead of sending cmd to
hosts
* Get the store driver from pool instead of passing it as argument
required change from code review to pick the store driver from pool not
to pass it as argument
* resource limit: Fix resource limit check on VM start
* add check to validate if cpu/memory are within limits for custom offering + exception handling
* unit tests
Co-authored-by: utchoang <hoangnm@unitech.vn>
* api: Update DNS on changing VM name
* refactor backend code + UI - add warning messaging to prompt users to restart VM in case particular fields are updated
* address comments
* Extend addAnnotation and listAnnotations APIs
* Allow users to add, list and remove comments
* Add adminsonly UI and allow admins or owners to remove comments
* New annotations tab
* In progress: new comments section
* Address review comments
* Fix
* Fix annotationfilter and comments section
* Add keyword and delete action
* Fix and rename annotations tab
* Update annotation visibility API and update comments table accordingly
* Allow users seeing all the comments for their owned resources
* Extend comments for volumes and snapshots
* Extend comments to multiple entities
* Add uuid to ssh keypairs
* SSH keypair UI refactor
* Extend comments to the infrastructure entities
* Add missing entities
* Fix upgrade version for ssh keypairs
* Fix typo on DB upgrade schema
* Fix annotations table columns when there is no data
* Extend the list view of items showing they if they have comments
* Remove extra test
* Add annotation permissions
* Address review comments
* Extend marvin tests for annotations
* updating ui stuff
* addition to toggle visibility
* Fix pagination on comments section
* Extend to kubernetes clusters
* Fixes after last review
* Change default value for adminsonly column
* Remove the required field for the annotationfilter parameter
* Small fixes on visibility and other fixes
* Cleanup to reduce files changed
* Rollback extra line
* Address review comments
* Fix cleanup error on smoke test
* Fix sending incorrect parameter to checkPermissions method
* Add check domain access for the calling account for domain networks
* Fix only display annotations icon if there are comments the user can see
* Simply change the Save button label to Submit
* Change order of the Tools menu to provent users getting 404 error on clicking the text instead of expanding
* Remove comments when removing entities
* Address review comments on marvin tests
* Allow users to list annotations for an entity ID
* Allow users to see all comments for allowed entities
* Fix search filters
* Remove username from search filter
* Add pagination to the annotations tab
* Display username for user comments
* Fix add permissions for domain and resource admins
* Fix for domain admins
* Trivial but important UI fix
* Replace pagination for annotations tab
* Add confirmation for delete comment
* Lint warnings
* Fix reduced list as domain admin
* Fix display remove comment button for non admins
* Improve display remove action button
* Remove unused parameter on groupShow
* Include a clock icon to the all comments filter except for root admin
* Move cleanup SQL to the correct file after rebasing main
Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
* server: fix reset sshkey is broken in master/4.16
* Revert "server: fix reset sshkey is broken in master/4.16"
This reverts commit db278cf412.
* update #5390
* server: fix another regression of #4819 and #5274
* update #5390
* server: Optional destination host when migrate a vm
* #4378: migrate systemvms/routers with optional host
* Migrate vms across clusters
After enabling maintenance mode on host, if no suitable hosts
are found in the same cluster then search for hosts in
different clusters having the same hypervisor type
set global setting migrate.vm.across.clusters to true
* search all clusters in zone when migrate vm across clusters if applicable
* Honor migrate.vm.across.clusters when migrate vm without destination
* Check MIGRATE_VM_ACROSS_CLUSTERS in zone setting
* #4534 Fix Vms are migrated to same clusters in CloudStack caused by dedicated resources.
* #4534 extract some codes to methods
* fix#4534: an error in 'git merge'
* fix#4534: remove useless methods in FirstFitPlanner.java
* fix#4534: vms are stopped in host maintenance
* fix#4534: across-cluster migration of vms with cluster-scoped pools is supported by vmware vmotion
* fix#4534: migrate systemvms is only possible across clusters in same pod to avoid potential network errors.
* fix#4534: code optimization
Co-authored-by: Rakesh Venkatesh <r.venkatesh@global.leaseweb.com>
Co-authored-by: Sina Kashipazha <s.kashipazha@global.leaseweb.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Sina Kashipazha <soreana@users.noreply.github.com>
* Create utility to centralize byte convertions
* Add/change toString definitions
* Create Libvirt handler to ScaleVmCommand
* Enable dynamic scalling VM with KVM
* Move config from interface to class and rename it
As every variable declared in interfaces are already final,
this moving will be needed to mock tests in nexts commits
* Configure VM max memory and cpu cores
The values are according to service offering or global configs
* Extract dpdk configuration to a method and test it
* Extract OS desc config to a method and test it
* Extract guest resource def to a method and test it
Improve libvirt def
* Refactor LibvirtVMDef.GuestResourceDef
* Refactor ScaleVmCommand
* Improve VMInstaVO toString()
* Refactor upgradeRunningVirtualMachine method
* Turn int variables into long on utility
* Verify if VM is scalable on KVMGuru
* Rename some KVMGuruTest's methods
* Change vm's xml to work with max memory
* Verify if service offering is dynamic before scale
* Create methods to retrieve data from domain
* Create def to hotplug memory
* Adjust the way command was scaling the VM
* Fix database persistence before executing command
* Send more info to host to improve log
* Fix var name
* Fix missing "}"
* Undo unnecessary changes
* Address review
* Fix scale validation
* Add VM prepared for dynamic scaling validation
* Refactor LibvirtScaleVmCommandWrapper and improve unit tests
* Remove duplicated method
* Add RuntimeException check
* Remove copyright from header
* Remove copyright from header
* Remove copyright from header
* Remove copyright from header
* Remove copyright from header
* Update ByteScaleUtilsTest.java
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Currently we can send a default value of 4K/32K for GET/POST request of
user data field. Most new browsers and also nginx support till 1MB of
post data.
Added a new global setting `vm.userdata.max.length` with default value of
32KB which can be increased till 1MB.
I've upgraded a stage environment from an older 4.16-SNAPSHOT to the current one and found a regression bug at the VM migration.
When calling the migrateVirtualMachineWithVolume, the following InvalidParameterValueException is launched: Unsupported hypervisor: KVM for VM migration, we support XenServer/VMware/KVM only.
* Enhance log messages with hostName
* Use host.toString() on most of host logs.
* Remove redundant "Host" in logs and enhance logs
* duplicated "for"
* Adopt String.format, and enhance code
* Address reviews enhancing log messages
Update server/src/main/java/com/cloud/resource/ResourceManagerImpl.java
-- server/src/main/java/com/cloud/vm/UserVmManagerImpl.java
-- server/src/main/java/com/cloud/resource/RollingMaintenanceManagerImpl.java
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
* Fix String.format issue and change log message from debug to warn
* Fix checkstyle issue
* Fix string.format log
* Address review: enhance logs
* Enhance log of hosts in maintenance avoid list
* Remove "VM" on logs as vm.toString() already appends VM-<details>
* Add more details of the VM when postStateTransitionEvent
* Address reviewer and enhance VMInstanceVO.toString()
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
On API `resetSSHKeyForVirtualMachine`, ACS also regenerates VM password when it uses a template with `Password Enabled` as true; there is already anAPI to reset VM password, therefore, the reset SSH keys API should not reset the VM SSH password as well.
Besides running a meaningless process, the VM's password regeneration slows down the main process and may cause a confusion in operations due to password change in the VM without being explicity requested.
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
This PR fixes the issue that nic has wrong gateway after updating vm nic.
Steps to reproduce the issue
(1) create shared network (in advanced zone or advanced zone with sg)
(2) create new shared network (with same startip/endip/netmask, but different gateway).
(3) create a vm in new network
(4) stop vm and update vm nic ip address
Expected result:
The vm has correct gateway and netmask (of second network)
Actual result:
The vm has wrong gateway and netmask (of first network)
This PR introduces new granularity levels to configure VM dynamic scalability. Previously VM is configured to be dynamically scalable based on the template and global setting. Now we bringing this option to configure at service offering and VM level also.
VM can dynamically scale only when all flags are ON at VM level, template, service offering and global setting. If any of the flags is set to false then VM cannot be scalable. This result will be persisted in DB for each VM and will be honoured for that VM till it is updated.
We are introducing 'dynamicscalingallowed' parameter with permitted values of true or false for deployVM API and createServiceOffering API.
Following are the API parameter changes:
createServiceOffering API:
dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.
deployVirtualMachine API:
dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.
Following are the UI changes:
Service offering creation has ON/OFF switch for dynamic scaling enabled with default value true
Inclusivity changes for CloudStack
- Change default git branch name from 'master' to 'main' (post renaming/changing default git branch to 'main' in git repo)
- Rename some offensive words/terms as appropriate for inclusiveness.
This PR updates the default git branch to 'main', as part of #4887.
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixes: #4990
When a VM associated with a backup offering is destroyed/expunged, the backup offering isn't unassigned, and despite the VM having no backups present, backup usage is generated. This PR prevent usage record generation when there are no backups present for a VM with a backup offering associated to it. This is done by ensuring that usage event for backups is generated only when a the backup size > 0
Harikrishna
Marcus Sorensen
mprokopchuk
dahn
Suresh Kumar Anaparti
Abhishek Kumar
Abhishek Kumar
dahn
Nicolas Vazquez
Wei Zhou
sureshanaparti
davidjumani
Pearl Dsilva
slavkap
Rohit Yadav
Wei Zhou
Wei Zhou
Daniel Augusto Veronezi Salvador
Rakesh
Gabriel Beims Bräscher
Suresh Kumar Anaparti