* Live storage migration of volume in scaleIO within same storage scaleio cluster
* Added migrate command
* Recent changes of migration across clusters
* Fixed uuid
* recent changes
* Pivot changes
* working blockcopy api in libvirt
* Checking block copy status
* Formatting code
* Fixed unit tests
* Upgrade libvirt java version to 0.5.3
* code refactoring and some changes
* Removed unused methods
* removed unused imports
* Unit tests to check if volume belongs to same or different storage scaleio cluster
* Unit tests for volume livemigration in ScaleIOPrimaryDataStoreDriver
* Fixed offline volume migration case and allowed encrypted volume migration
* Added more integration tests
* Support for migration of encrypted volumes across different scaleio clusters
* Fix UI notifications for migrate volume
* Data volume offline migration: save encryption details to destination volume entry
* Offline storage migration for scaleio encrypted volumes
* Allow multiple Volumes to be migrated with migrateVirtualMachineWithVolume API
* Add volume secrets if does not exists during volume migrations. secrets are getting cleared on package upgrades.
* Fix Unit tests
* Fix secret UUID for encrypted volume migration
* Added a null check for secret before removing
* Added new tests
* Auto Enable Disable KVM hosts
* Improve health check result
* Fix corner cases
* Script path refactor
* Fix sonar cloud reports
* Fix last code smells
* Add marvin tests
* Fix new line on agent.properties to prevent host add failures
* Send alert on auto-enable-disable and add annotations when the setting is enabled
* Address reviews
* Add a reason for enabling or disabling a host when the automatic feature is enabled
* Fix comment on the marvin test description
* Fix for disabling the feature if the admin has manually updated the host resource state before any health check result
* Support Jetty's live cert reload
* Update ServerDaemon.java
---------
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Fix TLS backend VNC for non root qemu
* Add CA directory to the qemu group
* Fix qemu group permissions
* Final fix for users on the qemu group
* Fix dynamic qemu group search
* Retrieve group from qemu.conf file
* Address review comments
* Move PassphraseVO to use String instead of byte[] to support Encrypt annotation
* Check for unencrypted passphrases before migrating passphrase table
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Add EncryptedElementType key resolver to SAML plugin
* saml: Fix SAML SSO plugin redirect URL (#6457)
This PR fixes the issue #6427 -> SAML request must be appended to an IdP URL as a query param with an ampersand, if the URL already contains a question mark, as opposed to always assume that IdP URLs don't have any query params.
Google's IdP URL for instance looks like this: https://accounts.google.com/o/saml2/idp?idpid=<ID>, therefore the expected redirect URL would be https://accounts.google.com/o/saml2/idp?idpid=<ID>&SAMLRequest=<SAMLRequest>
This code change is backwards compatible with the current behaviour.
* Apply backport for SAML session cookie path
https://github.com/apache/cloudstack/pull/6149
* ui: Logout before login (#6193)
This PR calls the logout API before login, to cleanup any duplicate sessionkey, as it was done on the legacy UI: #4326Fixes: #6127
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Luis Moreira <Luis-3M@users.noreply.github.com>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
* Rough start swapping DB Encryption, add CLI PoC
* Enhance EncryptionCLI to have command line parsing
* Refactor new encryption behind AeadBase64Encryptor for every use
* Add comment about encryption passwords
* EncryptionSecretKeyChanger - use reflection to find all encrypted tables
Over the years this hasn't been updated properly. Use reflection to find
the tables with encrypted fields. This will also ensure any plugins in
the classpath that add tables will get their encrypted fields updated as well.
Table vpn_users has encrypted columns [password]
Table sslcerts has encrypted columns [password, key]
Table user_view has encrypted columns [secret_key]
Table account_details has encrypted columns [value]
Table domain_details has encrypted columns [value]
Table s2s_customer_gateway has encrypted columns [ipsec_psk]
Table ucs_manager has encrypted columns [password]
Table vm_instance has encrypted columns [vnc_password]
Table passphrase has encrypted columns [passphrase]
Table keystore has encrypted columns [key]
Table external_stratosphere_ssp_credentials has encrypted columns [password]
Table storage_pool has encrypted columns [user_info]
Table remote_access_vpn has encrypted columns [ipsec_psk]
Table user has encrypted columns [secret_key]
Table oobm has encrypted columns [password]
* Apple FR68: add new class CloudStackEncryptor
* Apple FR68: add interface com.cloud.utils.crypt.Encryptor
* Apple FR68: update com.cloud.utils.EncryptionUtil
* Apple FR68: add cloudstack-utils.jar to cloudstack-common package
* Apple FR68: use cloudstack-utils.jar in scripts
* Apple FR68: revert replace.properties to original version
* Apple FR68: update EncryptionSecretKeyChanger
* Apple FR68: Add EncryptorVersion to CloudStackEncryptor
* Apple FR68: Update com.cloud.utils.crypt.EncryptionCLI
* Apple FR68: Remove check on EncryptionSecretKeyChecker.useEncryption in CloudStackEncryptor
* Apple FR68: update EncryptionSecretKeyChanger part2
* Apple FR68: update EncryptionSecretKeyChanger part3 (force update)
* Apple FR68: move cloud-migrate-databases.in to deprecated and recreate it with java command
* Apple FR68: update EncryptionSecretKeyChanger part4 (add skip-database-migration)
* Apple FR68: set encryptor in first encryption in CloudStackEncryptor
* Apple FR68: save db.cloud.encryptor.version in db.properties
* Apple FR68: update EncryptionSecretKeyChanger part4 (clear db.cloud.encryptor.version)
* Apple FR68: load and save db.cloud.encryptor.version in db.properties
* Apple FR68: Add caller class name in debug messages
* Apple FR68: consider non-exist tables and columns
* Apple FR68: skip tables if no data exists
* Apple FR68: remove GeneralSecurityException from code
* Apple FR68: hide value with Asterisks in CloudStackEncryptor
* Apple FR68: log an error message when fail to load 'init'
* Apple FR68: remove setup/bindir/cloud-migrate-databases.deprecated.in which I think it is not needed
* Apple FR68: add new encryptor version to EncryptionSecretKeyChanger
* Apple FR68: use System.exit(1) in EncryptionSecretKeyChanger
* Apple FR68: check arguments in cloudstack-migrate-databases
* Apple FR68: remove all org.jasypt.* in code
* Apple FR68: initilize database encryptors by getting 'init'
* Apple FR68: migrate server.properties
* Apple FR68: load new management key from environment variable CLOUD_SECRET_KEY_NEW
* Apple FR68: fix unable to load 'init' in fresh installation
* Apple FR68: fix 'Rolling back the transaction' in txn.close
* Apple FR68: improve logging in cloudstack-migrate-databases
* Apple FR68: hide value with Asterisks in other encryptors
* Apple FR68: System.exit(1) if fail to migrate server.properties
* Apple FR68: migrate values from cluster_details,user_vm_details,etc
* Apple FR68: refactor EncryptionSecretKeyChanger
* Apple FR68: update user_vm_deploy_as_is_details values
* Apple FR68: update image_store.url (if protocol is cifs) and storage_pool.path (if pool_type is SMB)
* Apple FR68: minor improvement EncryptionSecretKeyChanger
* Apple FR68: add unit test EncryptionSecretKeyChangerTest
* Apple FR68: support encryption type 'env' in cloudstack-setup-databases to get env "CLOUD_SECRET_KEY" before passed value
* Apple FR68: rename Encryptor to Base64Encryptor
* Apple FR68: Backport community PR 6542
* Apple FR68: code optimization
* Apple FR68: use Options and StringUtils
* Apple FR68: add license headers
* Apple FR68: refactor CloudStackEncryptor as per Daan's review
* Apple FR68: refactor DatabaseUpgradeChecker as per Daan's review
* Apple FR68: show error message in usage.log if fail to get encrypted configurations
* Apple FR68: load new MS key from env before migration
* Apple FR68: return 1 if fail to parse arguments of EncryptionCLI
* Apple FR68: fix code smells
* Apple FR68: fix code smells (part2)
* Apple FR68: revert FOOTER of cloudstack-migrate-databases to use \n
* Apple FR68: update help message of cloudstack-setup-databases
* Apple FR68: fix code smells (part3)
* Apple FR68: make changes as per suggestions
* Apple FR68: migrate database if new encryptor version is set to different
Testing result: (assume db.cloud.encryptor.version=V1)
(1) migrate only db.properties (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(2) migrate only db.properties (same db key, new db encryptorversion)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(3) migrate only db.properties (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(4) migrate only db.properties (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(5) migrate only db.properties (different db key, new db version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(6) migrate only db.properties (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(7) migrate db.properties and database (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1 --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(8) migrate db.properties and database (same db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(9) migrate db.properties and database (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(10) migrate db.properties and database (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(11) migrate db.properties and database (different db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(12) migrate db.properties and database (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
* smoke test: fix test_primary_storage.py
* smoke test: Do NOT run tests in test_primary_storage.py in parallel
This also fixes an issue in detachvolume
'Failed to detach volume Test Volume-yyyyyy from VM VM-zzzzzz; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment'
* Update PR7003: rename method
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Create table to store available console sessions
* Handle the new table in Java (VO and DAO)
* Manage console sessions via database
* Fix cherry-pick: verify in database if the session exist
* Address reviews: rename table to console_session and rename java objects according to the table name
* Redesign console_session to store more data
* Remove unnecessary constructor
* Use create table syntax previous to mariadb 10.5
* Add console session cleanup task
* Address review
* Add missing config keys
* Fix sonar cloud reports
* In progress fix console load report
* Fix remove console when session ends
* Improve setting description
---------
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
* Do not allow UEFI deployments on non UEFI enabled hosts
* Fix UEFI detection on KVM
* Refactor
* Improvement
agent: Detect existing hosts with UEFI support (#6139)
* agent: Pass uefi enabled status as part of ready command
* Cleanup
* Fix checkstyle
* Save uefi status if different
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Support RFB 3.8 and VNC auth working
* IN PROGRESS: Add TigerVNC classes and authenticate, work left on the messaging
* Fix console display
* Cleanup
* Last in-progress work
* Don't block reads in case stream empty, use Link to init client certs (#215)
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Unused files and methods cleanup
* Rewrite finished
* More cleanup
* Rename console server session to VM display name (#216)
* Rename console server session to VM display name
* Fix after rebasing
* Add encryption bar when available
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
* Add missing license header
* Remove unused variable from the TLS security
* Address review comments and sonar cloud reports
* Address more review comments
* Last sonarcloud code smell fixes
* Automate VNC TLS provisioning and improve UI
* Add missing cases for sonarcloud report
* Fix certs renewal issue on configuring TLS
* Address review comments
* Refactor serviceConfig script
* Fix certs propagation TLS conf
* Fix unsecure host tests
Co-authored-by: Marcus Sorensen <marcus_sorensen@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Apple FR66 - Host Control Plane Status
* Apple FR66 - use Offline instead of Disconnected
* Apple FR66: fix smoke test test_router_host_control_state
* Apple FR66: reorder import
* Apple FR66: revert DetailsTab.vue and apply new changes
* Update PR: update en.json
* Update PR: add hostcontrolstate to routers/systemvms
* Update PR: test stop/start cloudstack-agent in smoke test
* Update PR: fix UI build error (The template root requires exactly one element vue/valid-template-root)
* Update PR: update message on ui
* Update PR: Disable rebootVM and create vm/volume snapshot for KVM VMs
* Update PR: add more unit tests
* When VM is created and ROOT volume is created it should emit a VOLUME.CREATE event
* added space between methods
* Updated volume context comments.
Co-authored-by: Maxim Prokopchuk <mprokopchuk@apple.com>
* Improve logs
* Remove unnecessary comments
* Use diamond inference
* Fix some logs
* Remove unnecessary unboxing
* Create method to handle job result
* Remove unused vars and fix some logics
* Extract code to method and few adjusts
* Use CollectionUtils
* Extract pending work job validation to method
* Create new constructors
* Extract work job and info creation to a method
* Extract submit async job to a method
* Extract find vm by id to a method
* Change log level from trace to debug
* Remove unnused methods and add logs
* Undo code remotion
* Remove asserts and fix conditionals
* Address @GabrielBrascher reviews
* Remove double quotes from keys in manual json
* Undo code remotion
* Add object to log
* Remove statement from try/catch
* Implement toString with ReflectionToStringBuilderUtils
* Fix errors related to merge main
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
* When VM start fails at host for admin, report error
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Report ResourceUnavailableExceptions that result in InsufficientCapacityException to admin
* Update error message to be more straightforward
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
There are some networking setups where system VM communications are proxied off of the hypervisor host on which the system VM is running. For example, if the KVM management network is a NAT bridge, or the network plugin employs user mode network for system VM management interfaces, then system VM agent comms look as though they come form the hypervisor host.
In such a setup, the certificate authentication for agents fails because the source IP is that of the host of the system VM, rather than the system VM itself, and this IP is not in the connecting certificate presented. This PR adds a configuration value that allows the system VM cert to contain the host IP that the system VM is scheduled on. This allows such setups to maintain auth strictness on agent auth.
Co-authored-by: Marcus Sorensen <mls@apple.com>
* UserData as first clas resource
* Few fixes
* Added userdata id in deploy VM flow
* Fixed userdata append scenario between template userdata and user provided userdata
* UI: added a new section for userdata
* Added userdata details to the deployVM cmd flow
* Write userdata details into metadata VR
* Updated template response and views
* UI: added userdata id to deloyvm wizard
* Added userdata list to deploy VM form
* Added userdata params to registeruserdata UI form
* Small fixes and added userdata to updateVM flow
* Fixed unit tests and imports
* Userdata Navigation in template view
* Added userdata denyoverride flag and userdata params section
* Added ToolTips and fixes deploy VM, register userdata form
* added userdata policy list to register template form
* Allow override append of userdata in deploy VM flow
* update userdata linking to template
* Few UI fixes in deploy VM and edit template form
* fixes in deploy VM form to support deny userdata policy
* Added unit tests for userdata
* Added unit tests for linking userdata to template
* Remove unused imports
* Move test file to proper files
* Fix unused imports
* Fix Userdata delete flow
* Few improvements in the code
* Adding marvin tests for userdata
* Fixed marvin tests for registerd userdata
* Added few more marvin tests for userdata
* Few code fixes
* Few more code fixes
* Added userdata details to register and upload iso forms
* Added userdata selection in deploy VM form for ISOs
* Add comments section to userdata
* Added new API to reset UserData of a VM along with UpdateVM API
* Added new UnitTests for resetVMUserData
* Added resetVMUserdata in UI for stopped VM
* Added blank values for userdata in edit template form
* Added ISO id to the linkuserdatatotemplate API
* Added validation to userdata params so that it wont contain any VR metadafile names
* Removed required param for iso id in linkuserdatatotemplate cmd
* Added length to userdata param
* remove delete cascade on user_vm and vm_template tables for userdata id foreign key reference.
* Fix custom userdata params for config drive by adding userdata file name and value metadata json
* Fix marvin test case
* added comments to marvin test cases
* Fix document link in UI
* Added a check while deleting the userdata, to see if no VMs are using the userdata
* Added unit tests
* Removed labels added during merge
* added success message for userdata registration
* Added Schema changes to 4160 to 4161 upgrade path
* Fixed imports and some errors
* Fix service offering uuid in mysql view
* UI changes wrt to 4.16 branch mostly related to vue2
* Some UI fixes
* Register userdata and update template form fixes
* Fixed compute.js related to userdata
* UI fixes and user_vm_view wrt sshkey
* Fixed update template form
* Fix deploy VM and userdata reset forms
* Fixed Register and upload template and ISO forms
* Fixed getting params for userdata from template in instance creation form
* Removed CloudZonesNetworkElement.java as part of rebase, which is actually removed in https://github.com/shapeblue/cloudstack-apple/pull/191
* Fixed userdata selection in deploy vm and reset userdata vm forms
* Fixed method calls after rebase
Install haveged on mgmt server, which could require random generation for VM/volume passphrase
rng-tools and cryptsetup for this feature on kvm hosts. Automatically start haveged on mgmt server
and rngd on kvm hosts.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com
* Console access enhancements
* Remove extra logging
* Fix security hotspot
* Fix sonar cloud code smells
* Refactor API response
* Minor fix
* Refactor and increase timeout on ssh to cpvm
* Add marvin tests and extend permissions
* Fix account type
* Add unit tests
* Check vncport file exits on CPVM before attempting to add rules
* Change how vncport is read on cpvm
* Extra validation refactor
* Fix wrong token API param on UI
* Refactor vnc port selection to 8080 or 8443
* Do not display the input token modal and improve error message on console
* Improve error message and prevent opening blank popup when errors
* Fix logging exception due to algorithm
Fixes#194
Commit f27de63 introduced a new version of opensaml. That version brought jcl-over-slf4j-1.7.5.jar, jul-to-slf4j-1.7.5.jar, and log4j-over-slf4j-1.7.5.jar as dependencies, which causes Agents and Usages to not generate logs.
In order to make the logs to work again, this PR intends to exclude these dependencies while building the packages.
(cherry picked from commit e76df16d9f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Use cryptsetup w/o zeroing for encrypted scaleio - faster
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Pass storage scope during KVM volume migration to avoid remotely mounting local storage
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Add method to choose template pool based on scope
Signed-off-by: Marcus Sorensen <mls@apple.com>
* Clean up null check when creating migration options
Signed-off-by: Marcus Sorensen <mls@apple.com>
* ScaleIO enhancements - thin/thick encrypted, online resize
Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Updated resource counter to include correct size after volume creation/resize and other improvements
- Recalculate resource counters for root domain in the periodic task
- Update correct size in the primary_storage resource counter after volume creation/resize
- Some code improvements
* Removed extra white space
* review and sonarcloud issues
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
Harikrishna
Marcus Sorensen
Oscar Sandoval
Rohit Yadav
Oscar Sandoval
Nicolas Vazquez
Wei Zhou
Abhishek Kumar
Marcus Sorensen
mprokopchuk
dahn
Daniel Augusto Veronezi Salvador
Suresh Kumar Anaparti