* Fix TLS backend VNC for non root qemu
* Add CA directory to the qemu group
* Fix qemu group permissions
* Final fix for users on the qemu group
* Fix dynamic qemu group search
* Retrieve group from qemu.conf file
* Address review comments
* Rough start swapping DB Encryption, add CLI PoC
* Enhance EncryptionCLI to have command line parsing
* Refactor new encryption behind AeadBase64Encryptor for every use
* Add comment about encryption passwords
* EncryptionSecretKeyChanger - use reflection to find all encrypted tables
Over the years this hasn't been updated properly. Use reflection to find
the tables with encrypted fields. This will also ensure any plugins in
the classpath that add tables will get their encrypted fields updated as well.
Table vpn_users has encrypted columns [password]
Table sslcerts has encrypted columns [password, key]
Table user_view has encrypted columns [secret_key]
Table account_details has encrypted columns [value]
Table domain_details has encrypted columns [value]
Table s2s_customer_gateway has encrypted columns [ipsec_psk]
Table ucs_manager has encrypted columns [password]
Table vm_instance has encrypted columns [vnc_password]
Table passphrase has encrypted columns [passphrase]
Table keystore has encrypted columns [key]
Table external_stratosphere_ssp_credentials has encrypted columns [password]
Table storage_pool has encrypted columns [user_info]
Table remote_access_vpn has encrypted columns [ipsec_psk]
Table user has encrypted columns [secret_key]
Table oobm has encrypted columns [password]
* Apple FR68: add new class CloudStackEncryptor
* Apple FR68: add interface com.cloud.utils.crypt.Encryptor
* Apple FR68: update com.cloud.utils.EncryptionUtil
* Apple FR68: add cloudstack-utils.jar to cloudstack-common package
* Apple FR68: use cloudstack-utils.jar in scripts
* Apple FR68: revert replace.properties to original version
* Apple FR68: update EncryptionSecretKeyChanger
* Apple FR68: Add EncryptorVersion to CloudStackEncryptor
* Apple FR68: Update com.cloud.utils.crypt.EncryptionCLI
* Apple FR68: Remove check on EncryptionSecretKeyChecker.useEncryption in CloudStackEncryptor
* Apple FR68: update EncryptionSecretKeyChanger part2
* Apple FR68: update EncryptionSecretKeyChanger part3 (force update)
* Apple FR68: move cloud-migrate-databases.in to deprecated and recreate it with java command
* Apple FR68: update EncryptionSecretKeyChanger part4 (add skip-database-migration)
* Apple FR68: set encryptor in first encryption in CloudStackEncryptor
* Apple FR68: save db.cloud.encryptor.version in db.properties
* Apple FR68: update EncryptionSecretKeyChanger part4 (clear db.cloud.encryptor.version)
* Apple FR68: load and save db.cloud.encryptor.version in db.properties
* Apple FR68: Add caller class name in debug messages
* Apple FR68: consider non-exist tables and columns
* Apple FR68: skip tables if no data exists
* Apple FR68: remove GeneralSecurityException from code
* Apple FR68: hide value with Asterisks in CloudStackEncryptor
* Apple FR68: log an error message when fail to load 'init'
* Apple FR68: remove setup/bindir/cloud-migrate-databases.deprecated.in which I think it is not needed
* Apple FR68: add new encryptor version to EncryptionSecretKeyChanger
* Apple FR68: use System.exit(1) in EncryptionSecretKeyChanger
* Apple FR68: check arguments in cloudstack-migrate-databases
* Apple FR68: remove all org.jasypt.* in code
* Apple FR68: initilize database encryptors by getting 'init'
* Apple FR68: migrate server.properties
* Apple FR68: load new management key from environment variable CLOUD_SECRET_KEY_NEW
* Apple FR68: fix unable to load 'init' in fresh installation
* Apple FR68: fix 'Rolling back the transaction' in txn.close
* Apple FR68: improve logging in cloudstack-migrate-databases
* Apple FR68: hide value with Asterisks in other encryptors
* Apple FR68: System.exit(1) if fail to migrate server.properties
* Apple FR68: migrate values from cluster_details,user_vm_details,etc
* Apple FR68: refactor EncryptionSecretKeyChanger
* Apple FR68: update user_vm_deploy_as_is_details values
* Apple FR68: update image_store.url (if protocol is cifs) and storage_pool.path (if pool_type is SMB)
* Apple FR68: minor improvement EncryptionSecretKeyChanger
* Apple FR68: add unit test EncryptionSecretKeyChangerTest
* Apple FR68: support encryption type 'env' in cloudstack-setup-databases to get env "CLOUD_SECRET_KEY" before passed value
* Apple FR68: rename Encryptor to Base64Encryptor
* Apple FR68: Backport community PR 6542
* Apple FR68: code optimization
* Apple FR68: use Options and StringUtils
* Apple FR68: add license headers
* Apple FR68: refactor CloudStackEncryptor as per Daan's review
* Apple FR68: refactor DatabaseUpgradeChecker as per Daan's review
* Apple FR68: show error message in usage.log if fail to get encrypted configurations
* Apple FR68: load new MS key from env before migration
* Apple FR68: return 1 if fail to parse arguments of EncryptionCLI
* Apple FR68: fix code smells
* Apple FR68: fix code smells (part2)
* Apple FR68: revert FOOTER of cloudstack-migrate-databases to use \n
* Apple FR68: update help message of cloudstack-setup-databases
* Apple FR68: fix code smells (part3)
* Apple FR68: make changes as per suggestions
* Apple FR68: migrate database if new encryptor version is set to different
Testing result: (assume db.cloud.encryptor.version=V1)
(1) migrate only db.properties (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(2) migrate only db.properties (same db key, new db encryptorversion)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(3) migrate only db.properties (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated
(4) migrate only db.properties (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(5) migrate only db.properties (different db key, new db version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is not migrated (mostly on secondary management servers)
(6) migrate only db.properties (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is not migrated (mostly on secondary management servers)
(7) migrate db.properties and database (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1 --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(8) migrate db.properties and database (same db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(9) migrate db.properties and database (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(10) migrate db.properties and database (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
(11) migrate db.properties and database (different db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V2
cloudstack database is migrated using encryptor V2
(12) migrate db.properties and database (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
db.cloud.encryptor.version=V1
cloudstack database is migrated using encryptor V1
* smoke test: fix test_primary_storage.py
* smoke test: Do NOT run tests in test_primary_storage.py in parallel
This also fixes an issue in detachvolume
'Failed to detach volume Test Volume-yyyyyy from VM VM-zzzzzz; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment'
* Update PR7003: rename method
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Support RFB 3.8 and VNC auth working
* IN PROGRESS: Add TigerVNC classes and authenticate, work left on the messaging
* Fix console display
* Cleanup
* Last in-progress work
* Don't block reads in case stream empty, use Link to init client certs (#215)
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Unused files and methods cleanup
* Rewrite finished
* More cleanup
* Rename console server session to VM display name (#216)
* Rename console server session to VM display name
* Fix after rebasing
* Add encryption bar when available
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
* Add missing license header
* Remove unused variable from the TLS security
* Address review comments and sonar cloud reports
* Address more review comments
* Last sonarcloud code smell fixes
* Automate VNC TLS provisioning and improve UI
* Add missing cases for sonarcloud report
* Fix certs renewal issue on configuring TLS
* Address review comments
* Refactor serviceConfig script
* Fix certs propagation TLS conf
* Fix unsecure host tests
Co-authored-by: Marcus Sorensen <marcus_sorensen@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
* This PR/commit comprises of the following:
- Support to fallback on the older systemVM template in case of no change in template across ACS versions
- Update core user to cloud in CKS
- Display details of accessing CKS nodes in the UI - K8s Access tab
- Update systemvm template from debian 11 to debian 11.2
- Update letsencrypt cert
- Remove docker dependency as from ACS 4.16 onward k8s has deprecated support for docker - use containerd as container runtime
* support for private registry - containerd
* Enable updating template type (only) for system owned templates via UI
* edit indents
* Address comments and move cmd from patch file to cloud-init runcmd
* temporary change
* update k8s test to use k8s version 1.21.5 (instead of 1.21.3 - due to https://github.com/kubernetes/kubernetes/pull/104530)
* support for private registry - containerd
* Enable updating template type (only) for system owned templates via UI
* smooth upgrade of cks clusters
* update pom file with temp download.cloudstack.org testing links
* fix pom
* add cgroup config for containerd
* add systemd config for kubelet
* add additional info during image registry config
* update to official links
* packaging: display First Install and Onboarding Message
* Update #5851: Update as per Rohit's comments
* Update #5851: display package name in help message
* Update #5851: display links of installed cloudstack version on UI
* Update #5851: fix vue warnings
* Add NFS version to mount command
* Remove extra line
* Extend NFS version to mount secondary storage
* Unused import
* Refactor NFS version to be granular
* Make use of the ConfigKey on the NFS version setting value
This PR fixes component test failure with test_multiple_nic_support.py, in advanced zone with security groups.
Co-authored-by: dahn <daan.hoogland@gmail.com>
Sometimes the chain reference count is 0 and in that case
proper iptables rules are not applied. Because of this, ping fails.
So check the reference count for the main chain and as well as
-IN and -OUT chain as well
Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
This PR adds support for SUSE Leap 15.2+ as a KVM hypervisor and Management / Usage server on Cloudstack
Related docs link apache/cloudstack-documentation#224
Co-authored-by: Michael <35783820+mib1185@users.noreply.github.com>
In ubuntu20 the interface name contains @ synbol and
because of that even the iptable rules for brdige name
contains this symbol which causes ping issues.
Remove the @ symbol from iptable rule to fix the issue
Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
* server: fix failed to apply userdata when enable static nat
* server: fix cannot expunge vm as applyUserdata fails
* configdrive: fix ISO is not recognized when plug a new nic
* configdrive: detach and attach configdrive ISO as it is changed when plug a new nic or migrate vm
* configdrive test: (1) password file does not exists in recreated ISO; (2) vm hostname should be changed after migration
* configdrive: use centos55 template with sshkey and configdrive support
* configdrive: disklabel is 'config-2' for configdrive ISO
* configdrive: use copy for configdrive ISO and move for other template/volume/iso
* configdrive: use public-keys.txt
* configdrive test: fix (1) update_template ; (2) ssh into vm by keypair
This PR aims at introducing persistence mode in L2 networks and enhancing the behavior in Isolated networks
Doc PR apache/cloudstack-documentation#183
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
CNI plugin release naming has changed, https://github.com/containernetworking/plugins/releases
Release are named for host OS from 0.8.0 onwards.
This change adds check for 404 response code and attempts retry
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* security_group.py: fix SyntaxWarning: "is" with a literal.
2020-04-27 09:43:54,172 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-2:null) (logid:c33ba330) /usr/share/cloudstack-common/scripts/vm/network/security_group.py:513: SyntaxWarning: "is" with a literal. Did you mean "=="?
if rules is None or rules is "":
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:522: SyntaxWarning: "is" with a literal. Did you mean "=="?
if rules is None or rules is "":
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:823: SyntaxWarning: "is" with a literal. Did you mean "=="?
if brName is None or brName is "":
* Ubuntu 20.04: Fix systemvm cannot start up
in Ubuntu 16.04:
root@node13:~# bridge -o link show
2: eth0 state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
5: vnet0 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
6: vnet1 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
7: vnet2 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
root@node13:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1
eth0
root@node13:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 |cut -d ":" -f1
eth0
in Ubuntu 20.04:
root@node62:~# bridge -o link show
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
10: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
12: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
root@node62:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1
ens3:
root@node62:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 |cut -d ':' -f1
ens3
* security_group.py: use 'if not' instead
This is an extention of #3732 for kvm.
This is restricted to ovs > 2.9.2
Since Xen uses ovs 2.6, pvlan is unsupported.
This also fixes issues of vms on the same pvlan unable to communicate if they're on the same host
* DB : Add support for MySQL 8
- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters
For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:
server_id = 1
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
binlog-format = 'ROW'
default-authentication-plugin=mysql_native_password
Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.
Developers can set empty password as follows:
> sudo mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY '';
In libvirt repository, there are two related commits
2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation
In libvirt.spec.in
/bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
/opt/cloud/bin/setup_heartbeat_sr.sh line 75 tests for “XenServer release 7.” in /etc/redhat-release which contains “XCP-ng release 8.0.0 (xenenterprise)” in my case so the lvcreate statement doesn’t contain the required options
Fixes: #3281
Co-authored-by: Ian Service <ian@service.to>
When scripts/vm/hypervisor/kvm/kvmvmactivity.sh is called with an incorrect file name, an error is printed which is then interpreted as output from the script.
When an incorrect file name is passed the script prints out:
stat: cannot stat ‘b51d7336-d964-44ee-be60-bf62783dabc’: No such file or directory
=====> DEAD <======
The KVMHAVMActivityChecker.java checkingHB() process is expecting just
=====> DEAD <======
but gets the unexpected error message and interprets the file as alive.
This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The VM ingestion feature allows CloudStack to discover, on-board, import existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future.
* Update message when keys are NOT being injected
* Correct the message after injectkeys.ssh is done
* Update message to a more meaningful one, since sometimes nothing is injected
* Update other 2
* typo
* * Complete API implementation
* Complete UI integration
* Complete marvin test
* Complete Secondary storage GC background task
* improve UI labels
* slight reword and add another missing description
* improve download message clarity
* Address comments
* multiple fixes and cleanups
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* fix more bugs, let it return ip rule list in another log file
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* fix missing iprule bug
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* add support for ARCHIVE type of object to be linked/setup on secstorage
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Fix retrieving files for Xenserver
* Update get_diagnostics_files.py
* Fix bug where executable scripts weren't handled
* Fixed error on script cmd generation
* Do not filter name for log files as it would override similar prefix script names
* Addressed code review comments
* log error instead of printstacktrace
* Treat script as executable and shell script
* Check missing script name case and write to output instead of catching exception
* Use shell = true instead of shlex to support any executable
* fix xenserver bug
* don't set dir permission for vmware
* Code review comments - refactoring
* Add check for possible NPE
* Remove unused imoprt after rebase
* Add better description for configs
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Rohit Yadav <rohit@apache.org>
Co-authored-by: Anurag Awasthi <anurag.awasthi@shapeblue.com>
Currently while creating ingress/egress rule for a security group,
we can specify only TCP/UDP/ICMP. Sometimes we need to add rules
for different protocol number or rules for all the above three
mentioned protocols.
In this new feature users can specify the protocol number or select
"ALL" option which will apply rules for TCP/UDP/ICMP
* Extract systemvm.iso using bsdtar if available.
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
* New dependency for CentOS 7 and Debian: bsdtar
bsdtar can extract iso images without mounting.
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
* Remove all 'mount' and 'umount' command call(s).
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
Nicolas Vazquez
Wei Zhou
Pearl Dsilva
davidjumani
Rohit Yadav
Wei Zhou
Rakesh
Daniel Augusto Veronezi Salvador
Abhishek Kumar
Daan Hoogland
Wei Zhou
ServiceTo
Spaceman1984
Abhishek Kumar
Andrija Panic
Paul Angus
Anurag Awasthi
Kai Takahashi