* Rough start swapping DB Encryption, add CLI PoC
* Enhance EncryptionCLI to have command line parsing
* Refactor new encryption behind AeadBase64Encryptor for every use
* Add comment about encryption passwords
* EncryptionSecretKeyChanger - use reflection to find all encrypted tables
Over the years this hasn't been updated properly. Use reflection to find
the tables with encrypted fields. This will also ensure any plugins in
the classpath that add tables will get their encrypted fields updated as well.
Table vpn_users has encrypted columns [password]
Table sslcerts has encrypted columns [password, key]
Table user_view has encrypted columns [secret_key]
Table account_details has encrypted columns [value]
Table domain_details has encrypted columns [value]
Table s2s_customer_gateway has encrypted columns [ipsec_psk]
Table ucs_manager has encrypted columns [password]
Table vm_instance has encrypted columns [vnc_password]
Table passphrase has encrypted columns [passphrase]
Table keystore has encrypted columns [key]
Table external_stratosphere_ssp_credentials has encrypted columns [password]
Table storage_pool has encrypted columns [user_info]
Table remote_access_vpn has encrypted columns [ipsec_psk]
Table user has encrypted columns [secret_key]
Table oobm has encrypted columns [password]
* Apple FR68: add new class CloudStackEncryptor
* Apple FR68: add interface com.cloud.utils.crypt.Encryptor
* Apple FR68: update com.cloud.utils.EncryptionUtil
* Apple FR68: add cloudstack-utils.jar to cloudstack-common package
* Apple FR68: use cloudstack-utils.jar in scripts
* Apple FR68: revert replace.properties to original version
* Apple FR68: update EncryptionSecretKeyChanger
* Apple FR68: Add EncryptorVersion to CloudStackEncryptor
* Apple FR68: Update com.cloud.utils.crypt.EncryptionCLI
* Apple FR68: Remove check on EncryptionSecretKeyChecker.useEncryption in CloudStackEncryptor
* Apple FR68: update EncryptionSecretKeyChanger part2
* Apple FR68: update EncryptionSecretKeyChanger part3 (force update)
* Apple FR68: move cloud-migrate-databases.in to deprecated and recreate it with java command
* Apple FR68: update EncryptionSecretKeyChanger part4 (add skip-database-migration)
* Apple FR68: set encryptor in first encryption in CloudStackEncryptor
* Apple FR68: save db.cloud.encryptor.version in db.properties
* Apple FR68: update EncryptionSecretKeyChanger part4 (clear db.cloud.encryptor.version)
* Apple FR68: load and save db.cloud.encryptor.version in db.properties
* Apple FR68: Add caller class name in debug messages
* Apple FR68: consider non-exist tables and columns
* Apple FR68: skip tables if no data exists
* Apple FR68: remove GeneralSecurityException from code
* Apple FR68: hide value with Asterisks in CloudStackEncryptor
* Apple FR68: log an error message when fail to load 'init'
* Apple FR68: remove setup/bindir/cloud-migrate-databases.deprecated.in which I think it is not needed
* Apple FR68: add new encryptor version to EncryptionSecretKeyChanger
* Apple FR68: use System.exit(1) in EncryptionSecretKeyChanger
* Apple FR68: check arguments in cloudstack-migrate-databases
* Apple FR68: remove all org.jasypt.* in code
* Apple FR68: initilize database encryptors by getting 'init'
* Apple FR68: migrate server.properties
* Apple FR68: load new management key from environment variable CLOUD_SECRET_KEY_NEW
* Apple FR68: fix unable to load 'init' in fresh installation
* Apple FR68: fix 'Rolling back the transaction' in txn.close
* Apple FR68: improve logging in cloudstack-migrate-databases
* Apple FR68: hide value with Asterisks in other encryptors
* Apple FR68: System.exit(1) if fail to migrate server.properties
* Apple FR68: migrate values from cluster_details,user_vm_details,etc
* Apple FR68: refactor EncryptionSecretKeyChanger
* Apple FR68: update user_vm_deploy_as_is_details values
* Apple FR68: update image_store.url (if protocol is cifs) and storage_pool.path (if pool_type is SMB)
* Apple FR68: minor improvement EncryptionSecretKeyChanger
* Apple FR68: add unit test EncryptionSecretKeyChangerTest
* Apple FR68: support encryption type 'env' in cloudstack-setup-databases to get env "CLOUD_SECRET_KEY" before passed value
* Apple FR68: rename Encryptor to Base64Encryptor
* Apple FR68: Backport community PR 6542
* Apple FR68: code optimization
* Apple FR68: use Options and StringUtils
* Apple FR68: add license headers
* Apple FR68: refactor CloudStackEncryptor as per Daan's review
* Apple FR68: refactor DatabaseUpgradeChecker as per Daan's review
* Apple FR68: show error message in usage.log if fail to get encrypted configurations
* Apple FR68: load new MS key from env before migration
* Apple FR68: return 1 if fail to parse arguments of EncryptionCLI
* Apple FR68: fix code smells
* Apple FR68: fix code smells (part2)
* Apple FR68: revert FOOTER of cloudstack-migrate-databases to use \n
* Apple FR68: update help message of cloudstack-setup-databases
* Apple FR68: fix code smells (part3)
* Apple FR68: make changes as per suggestions
* Apple FR68: migrate database if new encryptor version is set to different
Testing result: (assume db.cloud.encryptor.version=V1)
(1) migrate only db.properties (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(2) migrate only db.properties (same db key, new db encryptorversion)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(3) migrate only db.properties (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(4) migrate only db.properties (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(5) migrate only db.properties (different db key, new db version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(6) migrate only db.properties (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(7) migrate db.properties and database (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1 --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(8) migrate db.properties and database (same db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(9) migrate db.properties and database (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(10) migrate db.properties and database (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(11) migrate db.properties and database (different db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(12) migrate db.properties and database (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
* smoke test: fix test_primary_storage.py
* smoke test: Do NOT run tests in test_primary_storage.py in parallel
This also fixes an issue in detachvolume
'Failed to detach volume Test Volume-yyyyyy from VM VM-zzzzzz; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment'
* Update PR7003: rename method
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
* debian: remove duplicate agent jar copy
The cloud-agent is dependency of the KVM hypervisor plugin, don't
explicitly install/copy it again.
Fixes#4906
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Update rules
This changes deb and rpm packaging to build the UI using npm and bundle
it in the `cloudstack-management` package and a new `cloudstack-ui`
package. The `cloudstack-ui` package will install the UI under
`/usr/share/cloudstack-ui/`. For both packages the config.json will not
be overridden on upgrade and hosted at /etc/cloudstack/management
for the cloudstack-mangement package, and at /etc/cloudstack/ui for the
cloudstack-ui package. The cloudstack-ui package is for advanced users
who only want the UI want to setup reverse proxy (separate hosting of UI).
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* DB : Add support for MySQL 8
- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters
For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:
server_id = 1
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
binlog-format = 'ROW'
default-authentication-plugin=mysql_native_password
Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.
Developers can set empty password as follows:
> sudo mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY '';
In libvirt repository, there are two related commits
2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation
In libvirt.spec.in
/bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Using this tool on a hypervisor admins can query KVM Instances running
on that hypervisor if they have the Qemu Guest Agent installed.
All System VMs have this and they can be queried.
For example:
$ cloudstack-guest-tool i-2-25-VM
This will print some information about network and filesystem status.
root@hv-138-a05-23:~# ./cloudstack-guest-tool s-11-VM --command info|jq
{
"network": [
{
"ip-addresses": [
{
"prefix": 8,
"ip-address": "127.0.0.1",
"ip-address-type": "ipv4"
}
],
"name": "lo",
"hardware-address": "00:00:00:00:00:00"
},
{
"ip-addresses": [
{
"prefix": 16,
"ip-address": "169.254.242.169",
"ip-address-type": "ipv4"
}
],
"name": "eth0",
"hardware-address": "0e:00:a9:fe:f2:a9"
},
...
...
"filesystem": [
{
"mountpoint": "/var",
"disk": [
{
"bus": 0,
"bus-type": "virtio",
"target": 0,
"unit": 0,
"pci-controller": {
"slot": 7,
"bus": 0,
"domain": 0,
"function": 0
}
}
],
"type": "ext4",
"name": "vda6"
},
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Since we support only Ubuntu 16.04+ on master/4.12+, we can now use
the libvirt service name `libvirtd` for all distributions. This also
fixes an optional package name for libvirtd installation on Debian 9+.
Fixes#2909
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Ubuntu 14.04 will go EOL in April 2019. With a new CloudStack
release close to that date we can drop support for this Ubuntu
version and the master branch of CloudStack.
Ubuntu 16.04 (Xenial) and 18.04 (Bionic) both have systemd and
more recent Java versions which make it easier to run the CloudStack
KVM Agent on them.
In addition libvirt and Qemu are more up to date with features
which allow VMs to run better.
Yet to be implemented features in KVM can also leverage the newer
version of Qemu and libvirt without the need of taking older
version of them into account.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* Cleaup and code-formatting POM files
* Remove obsolete mycila license-maven-plugin
* Remove obsolete console-proxy/plugin project
* Move console-proxy-rdbconsole under console-proxy parent
* Use correct parent path for rdpconsole
* Order alphabetally items in setnextversion.sh
* Unifiy License header in POMs
* Alphabetic order of modules definition
* Extract all defined versions into parent pom
* Remove obsolete files: version-info.in, configure-info.in
* Remove redundant defaultGoal
* Remove useless checkstyle plugin from checkstyle project
* Order alphabetally items in pom.xml
* Add aditional SPACEs to fix debian build
* Don't execute checkstyle on parent projects
* Use UTF-8 encoding in building checkstyle project
* Extract plugin versions into properties
* Execute PMD plugin on all the projects with -Penablefindbugs
* Upgrade maven plugins to latest version
* Make sure to always look for apache parent pom from repository
* Fix incorrect version grep in debian packaging
* Fix rebase conflicts
* Fix rebase conflicts
* Remove PMD for now to be fixed on another PR
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
deprecates the need to create/store cloud.jks in conf folder and
the db.cloud.keyStorePassphrase in db.properties file. This also
remove the need of the --keystore-passphrase in the
cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Refactors and unifies usage of systemd script and default files across
CentOS and Ubuntu/Debian packaging system.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- systemd: Add a /etc/sysconfig/cloudstack-* file
This allows users to easily override variables passed to Java when
starting up.
It also creates a foundation for sharing the systemd service profile
between CentOS and Ubuntu since it only requires the environment file
to be changed.
- deb: Add Ubuntu 16.04 support
Ubuntu 16.04 differs from Ubuntu 14.04 in a few ways:
- systemd instead of sysvinit / upstart
- Java 8 support
The packaging now detects on which distribution it is being
build and based on that it installs different files in the
packages, but it also changes the Dependencies.
Packages for Ubuntu 16.04 will require Java 8 as a JRE
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This introduces two new cloudstack packages: marvin and integration-tests.
The two packages will make it easier for CI systems to install Marvin for a
specific cloudstack release/build and run integration tests that are specific
for that version/build.
- maven: add explicit juniper-contrail-api maven repository
- marvin: build source distribution for both install and package mvn phases
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This allows non-root users to add KVM hosts, the user should be an admin or
added to sudoers to execute sudo cloudstack-setup-agent.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: Remi Bergsma <apache@remi.nl>
This closes#288
(cherry picked from commit d2b0c1a32b)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Removes awsapi packaging rules for debian, centos63, centos7, fedora 20/21
- Removes catalina port 7080 service configs
- Fixes build replace properties for AWSAPILOG
- Removes maven profile for building awsapi and deploying db in developer profile
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
A better solution in future could be to pass the custom maven repo in local dir
from ACS_BUILD_OPTS instead of putting it here
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
By default it chooses a location based on $HOME which is not guranteed to
have a proper value on buildds.
We also need to take care of cleaning it up after the build.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Including following steps:
b. Run "cloudstack-agent-upgrade". This script will upgrade all the existing bridge name to new bridge name, and update related firewall rules.
c. install a libvirt hook:
c1. mkdir /etc/libvirt/hooks
c2. cp /usr/share/cloudstack-agent/lib/libvirtqemuhook /etc/libvirt/hooks/qemu
c3. chmod +x /etc/libvirt/hooks/qemu
c4. service libvirtd restart
(cherry picked from commit a0988780ad)
Signed-off-by: Wei Zhou <w.zhou@leaseweb.com>
(1) Replacing db.properties with management server db.properties
(2) Rename log4j-cloud_usage.xml to log4j-cloud.xml
(cherry picked from commit fb97e8e617)
Wei Zhou
davidjumani
Rohit Yadav
Nicolas Vazquez
Wido den Hollander
Rohit Yadav
Khosrow Moossavi
Marc-Aurèle Brothier
Ronald van Zantvoort
Wei Zhou
Pierre-Luc Dion
Rafael da Fonseca
Daan Hoogland
Felix Geyer
Marcus Sorensen
Hugo Trippaers
Wei Zhou
ynojima
Hiroaki KAWAI
Wido den Hollander