* Support RFB 3.8 and VNC auth working
* IN PROGRESS: Add TigerVNC classes and authenticate, work left on the messaging
* Fix console display
* Cleanup
* Last in-progress work
* Don't block reads in case stream empty, use Link to init client certs (#215)
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Unused files and methods cleanup
* Rewrite finished
* More cleanup
* Rename console server session to VM display name (#216)
* Rename console server session to VM display name
* Fix after rebasing
* Add encryption bar when available
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
* Add missing license header
* Remove unused variable from the TLS security
* Address review comments and sonar cloud reports
* Address more review comments
* Last sonarcloud code smell fixes
* Automate VNC TLS provisioning and improve UI
* Add missing cases for sonarcloud report
* Fix certs renewal issue on configuring TLS
* Address review comments
* Refactor serviceConfig script
* Fix certs propagation TLS conf
* Fix unsecure host tests
Co-authored-by: Marcus Sorensen <marcus_sorensen@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Apple FR66 - Host Control Plane Status
* Apple FR66 - use Offline instead of Disconnected
* Apple FR66: fix smoke test test_router_host_control_state
* Apple FR66: reorder import
* Apple FR66: revert DetailsTab.vue and apply new changes
* Update PR: update en.json
* Update PR: add hostcontrolstate to routers/systemvms
* Update PR: test stop/start cloudstack-agent in smoke test
* Update PR: fix UI build error (The template root requires exactly one element vue/valid-template-root)
* Update PR: update message on ui
* Update PR: Disable rebootVM and create vm/volume snapshot for KVM VMs
* Update PR: add more unit tests
* When VM is created and ROOT volume is created it should emit a VOLUME.CREATE event
* added space between methods
* Updated volume context comments.
Co-authored-by: Maxim Prokopchuk <mprokopchuk@apple.com>
* Improve logs
* Remove unnecessary comments
* Use diamond inference
* Fix some logs
* Remove unnecessary unboxing
* Create method to handle job result
* Remove unused vars and fix some logics
* Extract code to method and few adjusts
* Use CollectionUtils
* Extract pending work job validation to method
* Create new constructors
* Extract work job and info creation to a method
* Extract submit async job to a method
* Extract find vm by id to a method
* Change log level from trace to debug
* Remove unnused methods and add logs
* Undo code remotion
* Remove asserts and fix conditionals
* Address @GabrielBrascher reviews
* Remove double quotes from keys in manual json
* Undo code remotion
* Add object to log
* Remove statement from try/catch
* Implement toString with ReflectionToStringBuilderUtils
* Fix errors related to merge main
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
* When VM start fails at host for admin, report error
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Report ResourceUnavailableExceptions that result in InsufficientCapacityException to admin
* Update error message to be more straightforward
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
There are some networking setups where system VM communications are proxied off of the hypervisor host on which the system VM is running. For example, if the KVM management network is a NAT bridge, or the network plugin employs user mode network for system VM management interfaces, then system VM agent comms look as though they come form the hypervisor host.
In such a setup, the certificate authentication for agents fails because the source IP is that of the host of the system VM, rather than the system VM itself, and this IP is not in the connecting certificate presented. This PR adds a configuration value that allows the system VM cert to contain the host IP that the system VM is scheduled on. This allows such setups to maintain auth strictness on agent auth.
Co-authored-by: Marcus Sorensen <mls@apple.com>
* UserData as first clas resource
* Few fixes
* Added userdata id in deploy VM flow
* Fixed userdata append scenario between template userdata and user provided userdata
* UI: added a new section for userdata
* Added userdata details to the deployVM cmd flow
* Write userdata details into metadata VR
* Updated template response and views
* UI: added userdata id to deloyvm wizard
* Added userdata list to deploy VM form
* Added userdata params to registeruserdata UI form
* Small fixes and added userdata to updateVM flow
* Fixed unit tests and imports
* Userdata Navigation in template view
* Added userdata denyoverride flag and userdata params section
* Added ToolTips and fixes deploy VM, register userdata form
* added userdata policy list to register template form
* Allow override append of userdata in deploy VM flow
* update userdata linking to template
* Few UI fixes in deploy VM and edit template form
* fixes in deploy VM form to support deny userdata policy
* Added unit tests for userdata
* Added unit tests for linking userdata to template
* Remove unused imports
* Move test file to proper files
* Fix unused imports
* Fix Userdata delete flow
* Few improvements in the code
* Adding marvin tests for userdata
* Fixed marvin tests for registerd userdata
* Added few more marvin tests for userdata
* Few code fixes
* Few more code fixes
* Added userdata details to register and upload iso forms
* Added userdata selection in deploy VM form for ISOs
* Add comments section to userdata
* Added new API to reset UserData of a VM along with UpdateVM API
* Added new UnitTests for resetVMUserData
* Added resetVMUserdata in UI for stopped VM
* Added blank values for userdata in edit template form
* Added ISO id to the linkuserdatatotemplate API
* Added validation to userdata params so that it wont contain any VR metadafile names
* Removed required param for iso id in linkuserdatatotemplate cmd
* Added length to userdata param
* remove delete cascade on user_vm and vm_template tables for userdata id foreign key reference.
* Fix custom userdata params for config drive by adding userdata file name and value metadata json
* Fix marvin test case
* added comments to marvin test cases
* Fix document link in UI
* Added a check while deleting the userdata, to see if no VMs are using the userdata
* Added unit tests
* Removed labels added during merge
* added success message for userdata registration
* Added Schema changes to 4160 to 4161 upgrade path
* Fixed imports and some errors
* Fix service offering uuid in mysql view
* UI changes wrt to 4.16 branch mostly related to vue2
* Some UI fixes
* Register userdata and update template form fixes
* Fixed compute.js related to userdata
* UI fixes and user_vm_view wrt sshkey
* Fixed update template form
* Fix deploy VM and userdata reset forms
* Fixed Register and upload template and ISO forms
* Fixed getting params for userdata from template in instance creation form
* Removed CloudZonesNetworkElement.java as part of rebase, which is actually removed in https://github.com/shapeblue/cloudstack-apple/pull/191
* Fixed userdata selection in deploy vm and reset userdata vm forms
* Fixed method calls after rebase
Install haveged on mgmt server, which could require random generation for VM/volume passphrase
rng-tools and cryptsetup for this feature on kvm hosts. Automatically start haveged on mgmt server
and rngd on kvm hosts.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com
* Console access enhancements
* Remove extra logging
* Fix security hotspot
* Fix sonar cloud code smells
* Refactor API response
* Minor fix
* Refactor and increase timeout on ssh to cpvm
* Add marvin tests and extend permissions
* Fix account type
* Add unit tests
* Check vncport file exits on CPVM before attempting to add rules
* Change how vncport is read on cpvm
* Extra validation refactor
* Fix wrong token API param on UI
* Refactor vnc port selection to 8080 or 8443
* Do not display the input token modal and improve error message on console
* Improve error message and prevent opening blank popup when errors
* Fix logging exception due to algorithm
Fixes#194
Commit f27de63 introduced a new version of opensaml. That version brought jcl-over-slf4j-1.7.5.jar, jul-to-slf4j-1.7.5.jar, and log4j-over-slf4j-1.7.5.jar as dependencies, which causes Agents and Usages to not generate logs.
In order to make the logs to work again, this PR intends to exclude these dependencies while building the packages.
(cherry picked from commit e76df16d9f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Use cryptsetup w/o zeroing for encrypted scaleio - faster
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Pass storage scope during KVM volume migration to avoid remotely mounting local storage
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Add method to choose template pool based on scope
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Clean up null check when creating migration options
Signed-off-by: Marcus Sorensen <mls@apple.com>
* ScaleIO enhancements - thin/thick encrypted, online resize
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Updated resource counter to include correct size after volume creation/resize and other improvements
- Recalculate resource counters for root domain in the periodic task
- Update correct size in the primary_storage resource counter after volume creation/resize
- Some code improvements
* Removed extra white space
* review and sonarcloud issues
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
This PR introduces volume encryption option to service offerings and disk offerings. Fixes#136
There is a hypervisor component and a storage pool component. Hypervisors are responsible for being capable of running/using the encrypted volumes. Storage pools are responsible for being able to create, copy, resize, etc. Hypervisors will report encryption support in their details, storage pools are marked for encryption support by pool type.
The initial offering for experimental release of this feature will have support for encryption on Local, NFS, SharedMountPoint, and ScaleIO storage types.
When volumes choosing an encrypted offering are allocated to a pool, the pool type must be capable of supporting encryption and this is enforced.
When VMs are started and they have an encrypted volume, the hypervisor must be capable of supporting encryption. Also, if volumes are attached to running VMs, the attach will only work if the hypervisor supports encryption.
This change includes a few other minor changes - for example the ability to force the KVM hypervisor private IP. This was necessary in my testing of ScaleIO, where the KVM hypervisors had multiple IPs and the ScaleIO storage only functions if the hypervisor as a ScaleIO client matches IPs with what CloudStack sees as the hypervisor IP.
For experimental release of this feature, some volume workflows like extract volume and migrate volume aren't supported for encrypted volumes. In the future we could support these, as well as migrating from unencrypted to encrypted offerings, and vice versa.
It may also be possible to configure encryption specifics in the future, perhaps at the pool level or the offering level. Currently, there is only one workable encryption offering for KVM that is supported by Libvirt and Qemu for raw and qcow2 disk files, LUKS version 1. This PR ensures we at least store this encryption format associated with each volume, with the expectation that later we may have LUKS v2 volumes or something else. Thus we will have the information necessary to use each volume with Libvirt if/when other formats are introduced.
I think the most disruptive change here is probably a refactoring of the QemuImg utility to support newer flags like --object. I've tested the change against the basic Qemu 1.5.3 that comes with EL7 and I believe it is good, but it will be nice to see the results of some functional tests. Most of the other changes are limited to changing behavior only if volume encryption is requested.
Working on documentation for the CloudStack docs. One thing to note is that hypervisors that run the stock EL7 version of Qemu will not support encryption. This is tested to be detected and report properly via the CloudStack API/UI. I intend to like to have a support matrix in the CloudStack docs.
I may add a few more unit tests. I'd also like some guidance on having functional tests. I'm not sure if there's a separate framework, or if Marvin is still used, or what the current thing is.
* Add Qemu object flag to QemuImg create
* Add apache license header to new files
* Add Qemu object flag to QemuImg convert
* Set host details if hypervisor supports LUKS
* Add disk encrypt flag to APIs, diskoffering
* Schema upgrade 4.16.0.0 to 4.16.1.0 to support vol encryption
* Add Libvirt secret on disk attach, and refer to it in disk XML
* Add implementation of luks volume encryption to QCOW2 and RAW disk prep
* Start VMs that have encrypted volumes
* Add encrypt option to service offering and root volume provisioning
* Refactor volume passphrase into its own table and object
* CryptSetup, use key files to pass keys instead of command line
* Update storage types and allocators to select encryption support
* Allow agent.properties to define the hypervisor's private IP
* Implement createPhysicalDisk for ScaleIOStorageAdaptor
* UI: Add encrypt options to offerings
* UI module security updates
* Revert "UI module security updates" - belongs in base
This reverts commit a7cb7cf7f57aad38f0b5e5d67389c187b88ffd94.
* Add --target-is-zero support for QemuImg
* Allow qemu image options to be passed, API support convert encrypted
* Switch hypervisor encryption support detection to use KeyFiles
* Fixes for ScaleIO root disk encryption
* Resize root disk if it won't fit encryption header
* Use cryptsetup to prep raw root disks, when supported
* Create qcow2 formatting if necessary during initial template copy to ScaleIO
* Allow setting no cache for qemu-img during disk convert
* Use 1M sparse on qemu-img convert for zero target disks
* UI: Add volume encryption support to hypervisor details
* QemuImg use --image-opts and --object depending on version
* Only send storage commands that require encryption to hosts that support encryption
* Move host encryption detail to a static constant
* Update host selection to account for volume encryption support
Only attach volumes if encryption requirements are met
* Ensure resizeVolume won't allow changing encryption
* Catch edge cases for clearing passphrase when volume is removed
* Disable volume migration and extraction for encrypted volumes
* Register volume secret on destination host during live migration
* Fix configdrive path editing during live migration
* Ensure configdrive path is edited properly during live migration
* Pass along and store volume encryption format during creation
* Fixes for rebase
* Fix tests after rebase
* Add unit tests for DeploymentPlanningManagerImpl to support encryption
* Deployment planner tests for encryption support on last host
* Add deployment tests for encryption when calling planner
* Added Libvirt DiskDef test for encryption details
* Add test for KeyFile utility
* Add CryptSetup tests
* Add QemuImageOptionsTest
* add smoke tests for API level changes on create/list offerings
* Fix schema upgrade, do disk_offering_view first
* Fix UI to show hypervisor encryption support
* Load details into hostVO before trying to query them for encryption
* Remove whitespace in CreateNetworkOfferingTest
* Move QemuImageOptions to use constants for flag keys
* Set physical disk encrypt format during createDiskFromTemplate in KVM Agent
* Whitespace in AbstractStoragePoolAllocator
* Fix whitespace in VolumeDaoImpl
* Support old Qemu in convert
* Log how long it takes to generate a passphrase during volume creation
* Move passphrase generation to async portion of createVolume
* Revert "Allow agent.properties to define the hypervisor's private IP"
This reverts commit 6ea9377505f0e5ff9839156771a241aaa1925e70.
* Updated ScaleIO/PowerFlex storage plugin to support separate (storage) network for Host(KVM) SDC connection. (#144)
* Added smoke tests for volume encryption (in KVM). (#149)
* Updated ScaleIO pool unit tests.
* Some improvements/fixes for code smells (in ScaleIO storage plugin).
* Updated review changes for ScaleIO improvements.
* Updated host response parameter 'encryptionsupported' in the UI.
* Move passphrase generation for the volume to async portion, while deploying VM (#158)
* Move passphrase generation for the volume to async portion, while deploying VM.
* Updated logs, to include volume details.
* Fix schema upgrade, create passphrase table first
* Fixed the DB upgrade issue (as noticed in the logs below.)
DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLException: Failed to open the referenced table 'passphrase'
ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script
* Fixes for snapshots with encrypted qcow2
Fixes#159#160#163
* Support create/delete encrypted snapshots of encrypted qcow2 volumes
* Select endpoints that support encryption when snapshotting encrypted volumes
* Update revert snapshot to be compatible with encrypted snapshots
* Disallow volume and template create from encrypted vols/snapshots
* Disallow VM memory snapshots on encrypted vols. Fixes#157
* Fix for TemplateManagerImpl unit test failure
* Support offline resize of encrypted volumes. Fixes#168
* Fix for resize volume unit tests
* Updated libvirt resize volume unit tests
* Support volume encryption on kvm only, and passphrase generation refactor (#169)
* Fail deploy VM when ROOT/DATA volume's offering has encryption enabled, on non-KVM hypervisors
* Fail attach volume when volume's offering has encryption enabled, on non-KVM hypervisors
* Refactor passphrase generation for volume
* Apply encryption to dest volume for live local storage migration
fixes#161
* Apply encryption to data volumes during live storage migration
Fixes#161
* Use the same encryption passphrase id for migrating volumes
* Pass secret consumer during storage migration prepare
Fix for #161
* Fixes create / delete volume snapshot issue, for stopped VMs
* Block volume snapshot if encrypted and VM is running
Fixes#159
* Block snap schedules on encrypted volumes
Fix for #159
* Support cryptsetup where luks type defaults to 2
Fixes#170
* Modify domain XML secret UUID when storage migrating VM
Fix for #172
* Remove any libvirt secrets on VM stop and post migration
Fix for #172
* Update disk profile with encryption requirement from the disk offering (#176)
Update disk profile with encryption requirement from the disk offering
and some code improvements
* Updated review changes / javadoc in ScaleIOUtil
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
* Allow for arbitrary disk offering details to be saved/displayed
* Update api/src/main/java/org/apache/cloudstack/api/response/DiskOfferingResponse.java
* Extract the IO_URING configuration into the agent.properties (#6253)
When using advanced virtualization the IO Driver is not supported. The
admin will decide if want to enable/disable this configuration from
agent.properties file. The default value is true
* kvm: truncate vnc password to 8 chars (#6244)
This PR truncates the vnc password of kvm vms to 8 chars to support latest versions of libvirt.
* merge fix
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* [KVM] Enable IOURING only when it is available on the host (#6399)
* [KVM] Disable IOURING by default on agents
* Refactor
* Remove agent property for iouring
* Restore property
* Refactor suse check and enable on ubuntu by default
* Refactor irrespective of guest OS
* Improvement
* Logs and new path
* Refactor condition to enable iouring
* Improve condition
* Refactor property check
* Improvement
* Doc comment
* Extend comment
* Move method
* Add log
* [KVM] Fix VM migration error due to VNC password on libvirt limiting versions (#6404)
* [KVM] Fix VM migration error due to VNC password on libvirt limiting versions
* Fix passwd value
* Simplify implementation
Co-authored-by: slavkap <51903378+slavkap@users.noreply.github.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
* agent: enable ssl only for kvm agent (not in system vms)
* Revert "agent: enable ssl only for kvm agent (not in system vms)"
This reverts commit b2d76bad2e.
* Revert "KVM: Enable SSL if keystore exists (#6200)"
This reverts commit 4525f8c8e7.
* KVM: Enable SSL if keystore exists in LibvirtComputingResource.java
Co-authored-by: Wei Zhou <weizhou@apache.org>
Schema changes needed for Event Improvements functionality to work are being added in 4.16.0 to 4.16.1.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Add resource ID and resource type to event.
In UI, adds Events tab in resource view for the supporting resources.
Following SQL changes needed to support events with resource details in DB,
```
-- Alter event table to add resource_id and resource_type
ALTER TABLE `cloud`.`event`
ADD COLUMN `resource_id` bigint unsigned COMMENT 'ID of the resource associated with the even' AFTER `domain_id`,
ADD COLUMN `resource_type` varchar(32) COMMENT 'Account role in the project (Owner or Regular)' AFTER `resource_id`;
DROP VIEW IF EXISTS `cloud`.`event_view`;
CREATE VIEW `cloud`.`event_view` AS
SELECT
event.id,
event.uuid,
event.type,
event.state,
event.description,
event.resource_id,
event.resource_type,
event.created,
event.level,
event.parameters,
event.start_id,
eve.uuid start_uuid,
event.user_id,
event.archived,
event.display,
user.username user_name,
account.id account_id,
account.uuid account_uuid,
account.account_name account_name,
account.type account_type,
domain.id domain_id,
domain.uuid domain_uuid,
domain.name domain_name,
domain.path domain_path,
projects.id project_id,
projects.uuid project_uuid,
projects.name project_name
FROM
`cloud`.`event`
INNER JOIN
`cloud`.`account` ON event.account_id = account.id
INNER JOIN
`cloud`.`domain` ON event.domain_id = domain.id
INNER JOIN
`cloud`.`user` ON event.user_id = user.id
LEFT JOIN
`cloud`.`projects` ON projects.project_account_id = event.account_id
LEFT JOIN
`cloud`.`event` eve ON event.start_id = eve.id;
```
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* Use base clock when detecting host CPU speed from file, to match lscpu
Allow for manually setting the CPU speed via agent.properties if all else fails
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Update agent/conf/agent.properties
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
* prevent <ctrl>-<enter> handler from <space> from toggling checkboxes
* enter vs ctrl-enter issue remaining: toggles active switch on submit
* some fixes by hook on containing div
* other identified forms
Co-authored-by: Daan Hoogland <dahn@onecht.net>
Nicolas Vazquez
Wei Zhou
mprokopchuk
Marcus Sorensen
Harikrishna
Rohit Yadav
dahn
Daniel Augusto Veronezi Salvador
Marcus Sorensen
Suresh Kumar Anaparti
Maxim Prokopchuk
Abhishek Kumar
Pearl Dsilva
Suresh Kumar Anaparti
dahn