* Rough start swapping DB Encryption, add CLI PoC
* Enhance EncryptionCLI to have command line parsing
* Refactor new encryption behind AeadBase64Encryptor for every use
* Add comment about encryption passwords
* EncryptionSecretKeyChanger - use reflection to find all encrypted tables
Over the years this hasn't been updated properly. Use reflection to find
the tables with encrypted fields. This will also ensure any plugins in
the classpath that add tables will get their encrypted fields updated as well.
Table vpn_users has encrypted columns [password]
Table sslcerts has encrypted columns [password, key]
Table user_view has encrypted columns [secret_key]
Table account_details has encrypted columns [value]
Table domain_details has encrypted columns [value]
Table s2s_customer_gateway has encrypted columns [ipsec_psk]
Table ucs_manager has encrypted columns [password]
Table vm_instance has encrypted columns [vnc_password]
Table passphrase has encrypted columns [passphrase]
Table keystore has encrypted columns [key]
Table external_stratosphere_ssp_credentials has encrypted columns [password]
Table storage_pool has encrypted columns [user_info]
Table remote_access_vpn has encrypted columns [ipsec_psk]
Table user has encrypted columns [secret_key]
Table oobm has encrypted columns [password]
* Apple FR68: add new class CloudStackEncryptor
* Apple FR68: add interface com.cloud.utils.crypt.Encryptor
* Apple FR68: update com.cloud.utils.EncryptionUtil
* Apple FR68: add cloudstack-utils.jar to cloudstack-common package
* Apple FR68: use cloudstack-utils.jar in scripts
* Apple FR68: revert replace.properties to original version
* Apple FR68: update EncryptionSecretKeyChanger
* Apple FR68: Add EncryptorVersion to CloudStackEncryptor
* Apple FR68: Update com.cloud.utils.crypt.EncryptionCLI
* Apple FR68: Remove check on EncryptionSecretKeyChecker.useEncryption in CloudStackEncryptor
* Apple FR68: update EncryptionSecretKeyChanger part2
* Apple FR68: update EncryptionSecretKeyChanger part3 (force update)
* Apple FR68: move cloud-migrate-databases.in to deprecated and recreate it with java command
* Apple FR68: update EncryptionSecretKeyChanger part4 (add skip-database-migration)
* Apple FR68: set encryptor in first encryption in CloudStackEncryptor
* Apple FR68: save db.cloud.encryptor.version in db.properties
* Apple FR68: update EncryptionSecretKeyChanger part4 (clear db.cloud.encryptor.version)
* Apple FR68: load and save db.cloud.encryptor.version in db.properties
* Apple FR68: Add caller class name in debug messages
* Apple FR68: consider non-exist tables and columns
* Apple FR68: skip tables if no data exists
* Apple FR68: remove GeneralSecurityException from code
* Apple FR68: hide value with Asterisks in CloudStackEncryptor
* Apple FR68: log an error message when fail to load 'init'
* Apple FR68: remove setup/bindir/cloud-migrate-databases.deprecated.in which I think it is not needed
* Apple FR68: add new encryptor version to EncryptionSecretKeyChanger
* Apple FR68: use System.exit(1) in EncryptionSecretKeyChanger
* Apple FR68: check arguments in cloudstack-migrate-databases
* Apple FR68: remove all org.jasypt.* in code
* Apple FR68: initilize database encryptors by getting 'init'
* Apple FR68: migrate server.properties
* Apple FR68: load new management key from environment variable CLOUD_SECRET_KEY_NEW
* Apple FR68: fix unable to load 'init' in fresh installation
* Apple FR68: fix 'Rolling back the transaction' in txn.close
* Apple FR68: improve logging in cloudstack-migrate-databases
* Apple FR68: hide value with Asterisks in other encryptors
* Apple FR68: System.exit(1) if fail to migrate server.properties
* Apple FR68: migrate values from cluster_details,user_vm_details,etc
* Apple FR68: refactor EncryptionSecretKeyChanger
* Apple FR68: update user_vm_deploy_as_is_details values
* Apple FR68: update image_store.url (if protocol is cifs) and storage_pool.path (if pool_type is SMB)
* Apple FR68: minor improvement EncryptionSecretKeyChanger
* Apple FR68: add unit test EncryptionSecretKeyChangerTest
* Apple FR68: support encryption type 'env' in cloudstack-setup-databases to get env "CLOUD_SECRET_KEY" before passed value
* Apple FR68: rename Encryptor to Base64Encryptor
* Apple FR68: Backport community PR 6542
* Apple FR68: code optimization
* Apple FR68: use Options and StringUtils
* Apple FR68: add license headers
* Apple FR68: refactor CloudStackEncryptor as per Daan's review
* Apple FR68: refactor DatabaseUpgradeChecker as per Daan's review
* Apple FR68: show error message in usage.log if fail to get encrypted configurations
* Apple FR68: load new MS key from env before migration
* Apple FR68: return 1 if fail to parse arguments of EncryptionCLI
* Apple FR68: fix code smells
* Apple FR68: fix code smells (part2)
* Apple FR68: revert FOOTER of cloudstack-migrate-databases to use \n
* Apple FR68: update help message of cloudstack-setup-databases
* Apple FR68: fix code smells (part3)
* Apple FR68: make changes as per suggestions
* Apple FR68: migrate database if new encryptor version is set to different
Testing result: (assume db.cloud.encryptor.version=V1)
(1) migrate only db.properties (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(2) migrate only db.properties (same db key, new db encryptorversion)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(3) migrate only db.properties (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(4) migrate only db.properties (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(5) migrate only db.properties (different db key, new db version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(6) migrate only db.properties (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(7) migrate db.properties and database (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1 --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(8) migrate db.properties and database (same db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(9) migrate db.properties and database (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(10) migrate db.properties and database (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(11) migrate db.properties and database (different db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(12) migrate db.properties and database (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
* smoke test: fix test_primary_storage.py
* smoke test: Do NOT run tests in test_primary_storage.py in parallel
This also fixes an issue in detachvolume
'Failed to detach volume Test Volume-yyyyyy from VM VM-zzzzzz; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment'
* Update PR7003: rename method
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Added ability to create schemas only when using cloudstack-setup-databases
* Renamed var name
* Added a check for passing --schema-only and --deploy-as together/.
* Moved validation to appropriate method
* Moved description
* fixed whitespace
* DB : Add support for MySQL 8
- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters
For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:
server_id = 1
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
binlog-format = 'ROW'
default-authentication-plugin=mysql_native_password
Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.
Developers can set empty password as follows:
> sudo mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY '';
In libvirt repository, there are two related commits
2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation
In libvirt.spec.in
/bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
deprecates the need to create/store cloud.jks in conf folder and
the db.cloud.keyStorePassphrase in db.properties file. This also
remove the need of the --keystore-passphrase in the
cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Removes awsapi db properties usage across codebase
- Removes references from spring xmls, test cases and TransactionLegacy
- Adds sql command to drop database cloudbridge in schema-451to460-cleanup.sql
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Summary: EC2 REST API : cloudbridge database is missing on the CloudStack Installation
After this change, cloudstack-setup-databases is launching cloud-bridge related databases properly.
- Fix developer prefill to use 4.0's schema
- Fix developer/pom.xml and cloud-setup-databases to not run create-schema-view,
the upgrade path is configured to do a rolling update and set it up
Signed-off-by: Rohit Yadav <bhaisaab@apache.org>
Make cloud-setup-databases compatible to python 2.4 and before.
Add code Prasanna Santhanam <tsp@apache.org>
Partially revert a6dcd7af49 which removed
the fix for CLOUDSTACK-199: Fix how cloud-setup-databases parses
Patch splits by right most @ in supplied argument to get
user:password and host substrings.
Less than python <2.4 the following is unsupported and produces a
SyntaxError.
try:
...code ...
except ValueError:
...code ...
finally:
...code ...
Workaround is the following
try:
try:
...code ...
except ValueError:
...code ...
finally:
Credits to Prasanna Santhanam <tsp@apache.org>
Signed-off-by: Rohit Yadav <bhaisaab@apache.org>
Wei Zhou
Spaceman1984
Rohit Yadav
Andrija Panic
davidjumani
Jayapal
Damodar Reddy
Rajani Karuturi
Hugo Trippaers
Pradeep Soundararajan
Rohit Yadav
Hugo Trippaers
Min Chen
Alex Huang
frank
Rohit Yadav
Edison Su
David Nalley
Edison Su
Frank