* Fix extract snapshot from VM snapshot on KVM
* Fix validation expression - does not need to escape the slash
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
this fixes the error below when create K8S ISO using scripts/util/create-kubernetes-binaries-iso.sh
```
+ echo 'Downloading image weaveworks/weave-kube:latest ---'
Downloading image weaveworks/weave-kube:latest ---
+ [[ weaveworks/weave-kube:latest == kubernetesui* ]]
+ [[ weaveworks/weave-kube:latest == apache* ]]
+ sudo ctr image pull weaveworks/weave-kube:latest
INFO[0000] trying next host error="failed to do request: Head https://weaveworks/v2/weave-kube/manifests/latest: dial tcp: lookup weaveworks: no such host" host=weaveworks
ctr: failed to resolve reference "weaveworks/weave-kube:latest": failed to do request: Head https://weaveworks/v2/weave-kube/manifests/latest: dial tcp: lookup weaveworks: no such host
```
* Improve log when live patching fails
* change patching path from /tmp to /var/cache/clou
* add iptable rule for console proxy (novnc)
* temporary template paths
* revert pom xml to original paths
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency
* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp
* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup
* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes
* Add ssh to k8s nodes details in the Access tab on the UI
* test
* Refactor ca/cert patching logic
* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script
* remove all references of systemvm.iso
* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs
* fix script timeout
* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand
* remove commented code + change core user to cloud for cks nodes
* Update ownership of ssh directory
* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)
* Add UI changes + move changes from patch file to runcmd
* test: validate performance for template modification during seeding
* create vms folder in cloudstack-commons directory - debian rules
* remove logic for on the fly template convert + update k8s test
* fix syntax issue - causing issue with shared network tests
* Code cleanup
* refactor patching logic - certs
* move logic of fixing rootdiskcontroller from upgrade to kubernetes service
* add livepatch option to restart network & vpc
* smooth upgrade of cks clusters
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency
* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp
* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup
* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes
* Add ssh to k8s nodes details in the Access tab on the UI
* test
* Refactor ca/cert patching logic
* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script
* remove all references of systemvm.iso
* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs
* fix script timeout
* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand
* remove commented code + change core user to cloud for cks nodes
* Update ownership of ssh directory
* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)
* Add UI changes + move changes from patch file to runcmd
* test: validate performance for template modification during seeding
* create vms folder in cloudstack-commons directory - debian rules
* remove logic for on the fly template convert + update k8s test
* fix syntax issue - causing issue with shared network tests
* Code cleanup
* add cgroup config for containerd
* add systemd config for kubelet
* add additional info during image registry config
* address comments
* add temp links of download.cloudstack.org
* address part of the comments
* address comments
* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0
* address comments - simplify
* fix vue3 related icon changes
* allow network commands when router template version is lower but is patched
* add internal LB to the list of routers to be patched on network restart with live patch
* add unit tests for API param validations and new helper utilities - file scp & checksum validations
* perform patching only for non-user i.e., system VMs
* add test to validate params
* remove unused import
* add column to domain_router to display software version and support networkrestart with livePatch from router view
* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false
* use router software version instead of checksum
* show N/A if no software version reported i.e., in upgraded envs
* fix deb failure
* update pom to official links of systemVM template
* Refactor create volume snapshot with running VM
* Refactor create volume snapshot with stopped VM
* Refactor create volume from snapshot
* Refactor create template from snapshot
* Refactor volume migration (migrateVolume/ migrateVirtualMachineWithVolume)
* Refactor snapshot deletion
* Refactor snapshot revertion
* Adjusts and fix cherry-pick conflicts
* Remove diffuse tests
* Add validation to add flag '--delete' on command 'virsh blockcommand' only if libvirt version is equal or higher 6.0.0
* Expunge temporary snapshot only if template creation is from snapshot
* Extract strings to constant
* Remove unused imports
* Fix error on revert backed up snapshot
* Turn method's return to void as it is not used
* Rename method in SnapshotHelper
* Fix folder creation when using SharedMountPoint pool
* Remove static import
* Remove unnused method
* Cover take snapshot in centos 7
* Handle right snapshot flag according to qemu version
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
* This PR/commit comprises of the following:
- Support to fallback on the older systemVM template in case of no change in template across ACS versions
- Update core user to cloud in CKS
- Display details of accessing CKS nodes in the UI - K8s Access tab
- Update systemvm template from debian 11 to debian 11.2
- Update letsencrypt cert
- Remove docker dependency as from ACS 4.16 onward k8s has deprecated support for docker - use containerd as container runtime
* support for private registry - containerd
* Enable updating template type (only) for system owned templates via UI
* edit indents
* Address comments and move cmd from patch file to cloud-init runcmd
* temporary change
* update k8s test to use k8s version 1.21.5 (instead of 1.21.3 - due to https://github.com/kubernetes/kubernetes/pull/104530)
* support for private registry - containerd
* Enable updating template type (only) for system owned templates via UI
* smooth upgrade of cks clusters
* update pom file with temp download.cloudstack.org testing links
* fix pom
* add cgroup config for containerd
* add systemd config for kubelet
* add additional info during image registry config
* update to official links
* packaging: display First Install and Onboarding Message
* Update #5851: Update as per Rohit's comments
* Update #5851: display package name in help message
* Update #5851: display links of installed cloudstack version on UI
* Update #5851: fix vue warnings
* Add NFS version to mount command
* Remove extra line
* Extend NFS version to mount secondary storage
* Unused import
* Refactor NFS version to be granular
* Make use of the ConfigKey on the NFS version setting value
This PR fixes component test failure with test_multiple_nic_support.py, in advanced zone with security groups.
Co-authored-by: dahn <daan.hoogland@gmail.com>
Sometimes the chain reference count is 0 and in that case
proper iptables rules are not applied. Because of this, ping fails.
So check the reference count for the main chain and as well as
-IN and -OUT chain as well
Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
This PR adds support for SUSE Leap 15.2+ as a KVM hypervisor and Management / Usage server on Cloudstack
Related docs link apache/cloudstack-documentation#224
Co-authored-by: Michael <35783820+mib1185@users.noreply.github.com>
In ubuntu20 the interface name contains @ synbol and
because of that even the iptable rules for brdige name
contains this symbol which causes ping issues.
Remove the @ symbol from iptable rule to fix the issue
Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
* server: fix failed to apply userdata when enable static nat
* server: fix cannot expunge vm as applyUserdata fails
* configdrive: fix ISO is not recognized when plug a new nic
* configdrive: detach and attach configdrive ISO as it is changed when plug a new nic or migrate vm
* configdrive test: (1) password file does not exists in recreated ISO; (2) vm hostname should be changed after migration
* configdrive: use centos55 template with sshkey and configdrive support
* configdrive: disklabel is 'config-2' for configdrive ISO
* configdrive: use copy for configdrive ISO and move for other template/volume/iso
* configdrive: use public-keys.txt
* configdrive test: fix (1) update_template ; (2) ssh into vm by keypair
This PR aims at introducing persistence mode in L2 networks and enhancing the behavior in Isolated networks
Doc PR apache/cloudstack-documentation#183
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
CNI plugin release naming has changed, https://github.com/containernetworking/plugins/releases
Release are named for host OS from 0.8.0 onwards.
This change adds check for 404 response code and attempts retry
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* security_group.py: fix SyntaxWarning: "is" with a literal.
2020-04-27 09:43:54,172 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-2:null) (logid:c33ba330) /usr/share/cloudstack-common/scripts/vm/network/security_group.py:513: SyntaxWarning: "is" with a literal. Did you mean "=="?
if rules is None or rules is "":
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:522: SyntaxWarning: "is" with a literal. Did you mean "=="?
if rules is None or rules is "":
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:823: SyntaxWarning: "is" with a literal. Did you mean "=="?
if brName is None or brName is "":
* Ubuntu 20.04: Fix systemvm cannot start up
in Ubuntu 16.04:
root@node13:~# bridge -o link show
2: eth0 state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
5: vnet0 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
6: vnet1 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
7: vnet2 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
root@node13:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1
eth0
root@node13:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 |cut -d ":" -f1
eth0
in Ubuntu 20.04:
root@node62:~# bridge -o link show
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
10: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
12: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
root@node62:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1
ens3:
root@node62:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 |cut -d ':' -f1
ens3
* security_group.py: use 'if not' instead
This is an extention of #3732 for kvm.
This is restricted to ovs > 2.9.2
Since Xen uses ovs 2.6, pvlan is unsupported.
This also fixes issues of vms on the same pvlan unable to communicate if they're on the same host
* DB : Add support for MySQL 8
- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters
For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:
server_id = 1
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
binlog-format = 'ROW'
default-authentication-plugin=mysql_native_password
Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.
Developers can set empty password as follows:
> sudo mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY '';
In libvirt repository, there are two related commits
2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation
In libvirt.spec.in
/bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
/bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
/opt/cloud/bin/setup_heartbeat_sr.sh line 75 tests for “XenServer release 7.” in /etc/redhat-release which contains “XCP-ng release 8.0.0 (xenenterprise)” in my case so the lvcreate statement doesn’t contain the required options
Fixes: #3281
Co-authored-by: Ian Service <ian@service.to>
When scripts/vm/hypervisor/kvm/kvmvmactivity.sh is called with an incorrect file name, an error is printed which is then interpreted as output from the script.
When an incorrect file name is passed the script prints out:
stat: cannot stat ‘b51d7336-d964-44ee-be60-bf62783dabc’: No such file or directory
=====> DEAD <======
The KVMHAVMActivityChecker.java checkingHB() process is expecting just
=====> DEAD <======
but gets the unexpected error message and interprets the file as alive.
This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The VM ingestion feature allows CloudStack to discover, on-board, import existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future.
* Update message when keys are NOT being injected
* Correct the message after injectkeys.ssh is done
* Update message to a more meaningful one, since sometimes nothing is injected
* Update other 2
* typo
* * Complete API implementation
* Complete UI integration
* Complete marvin test
* Complete Secondary storage GC background task
* improve UI labels
* slight reword and add another missing description
* improve download message clarity
* Address comments
* multiple fixes and cleanups
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* fix more bugs, let it return ip rule list in another log file
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* fix missing iprule bug
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* add support for ARCHIVE type of object to be linked/setup on secstorage
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Fix retrieving files for Xenserver
* Update get_diagnostics_files.py
* Fix bug where executable scripts weren't handled
* Fixed error on script cmd generation
* Do not filter name for log files as it would override similar prefix script names
* Addressed code review comments
* log error instead of printstacktrace
* Treat script as executable and shell script
* Check missing script name case and write to output instead of catching exception
* Use shell = true instead of shlex to support any executable
* fix xenserver bug
* don't set dir permission for vmware
* Code review comments - refactoring
* Add check for possible NPE
* Remove unused imoprt after rebase
* Add better description for configs
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Rohit Yadav <rohit@apache.org>
Co-authored-by: Anurag Awasthi <anurag.awasthi@shapeblue.com>
Currently while creating ingress/egress rule for a security group,
we can specify only TCP/UDP/ICMP. Sometimes we need to add rules
for different protocol number or rules for all the above three
mentioned protocols.
In this new feature users can specify the protocol number or select
"ALL" option which will apply rules for TCP/UDP/ICMP
* Extract systemvm.iso using bsdtar if available.
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
* New dependency for CentOS 7 and Debian: bsdtar
bsdtar can extract iso images without mounting.
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
* Remove all 'mount' and 'umount' command call(s).
Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
When I add a security group rule with cidr like 1.1.1.1/24, the rule is not applied on kvm hypervisor.
Ths issue does not exist in 4.13.0.0 and previous versions.
* kvm: Use 'ip' instead of 'brctl'
The command 'brctl' is deprecated and should no longer be used.
iproute2 supports all the features we need and therefor we should use
this instead of the old commands.
Feature wise this does not change anything. It just makes the code more
robust towards the future.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* kvm/modifyvlan: Use 'ip' instead of 'brctl'
brctl is deprecated and by using iproute2 we are future-proof
Signed-off-by: Wido den Hollander <wido@widodh.nl>
in agent.log on a host running with cloudstack 4.14.0.0-SNAPSHOT, it gives error
root@node212:~# /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 1272, in <module>
get_rule_logs_for_vms()
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 758, in get_rule_logs_for_vms
vms = virshlist(state)
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 104, in virshlist
alldomains = [d for domain in map(conn.lookupByID, conn.listDomainsID())]
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 104, in <listcomp>
alldomains = [d for domain in map(conn.lookupByID, conn.listDomainsID())]
NameError: name 'd' is not defined
We've encountered a corner case where bridge -o link show returned two lines per bridge instead of one. get_bridge_physdev in security_group.py returned bond0.701\nbond0.701.
Although this may very well be something on the hypervisor, we should limit the lines returned.
I therefore added a mere | head -1 to the function.
* kvm/security_group: Make Security Group Python 3 compatible
This script only runs on the KVM Hypervisors and these all support
Python 3.
As Python 2 is deprecated at the end of 2019 we need to fix these
scripts to work under Python 3.
CentOS 7, 8 and Ubuntu 16.04 and 18.04 all have Python 3 installed
by default.
Ubuntu 20.04 will no longer have Python 2 installed and therefor
this script needs to be modified to work with Python 3.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* Add dependency of python3 in packaging/centos7/cloud.spec
This PR partially fixes the logic around port forwarding rules on the Juniper SRX plugin. The code in the plugin is based on JunOS 10, which is very old. The changes here should not break compatibility, but should enable the plugin to be used on newer devices. Note that an additional change to a script file is required to be able to add port forwarding rules, but as this PR was targetted for 4.11.3, I thought it best not to include this change as it might break compatibility for anyone still using JunOS 10.
I've made the logic better and consistent for adding/removing static nat and port forwarding rules - these were multi-step processes which did not check each individual step. This would aid in manually fixing rules in case of further problems.
I've also improved the logging for communication with the SRX by stripping out the Apache header before sending it, and indicating the name of the template filename in use.
To be able to add port forwarding rules, the <dst-port> tags in dest-nat-rule-add.xml must be changed to <low>.
Fixes: #3379
This introduces a new patching script for patching systemvms on KVM
using qemu-guest-agent that runs inside the systemvm on startup. This
also removes the vport device which was previously used by the legacy
patching script and instead uses the modern and new uniform guest
agent vport for host-guest communication.
Also updates the sytemvmtemplate build config to use the latest Debian
9.9.0 iso.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Fix XenServer Security Groups 'vmops' script
- fix tokens = line.split(':') to tokens = line.split(';')
- fix expected tokens size from 5 to 4
- enhance logs
- remove unused vmops script. The XCP patch points to the vmops script
on the parent folder [1]. Thus, all XenServer versions are considering
the vmops script located at [2].
- fix UI ipv4/ipv6 cidr validator to allow a list of cidirs.
Fixing issue: #3192 Security Group rules not applied at all for
XenServer 6.5 / Advanced Zone
https://github.com/apache/cloudstack/issues/3192
* Update security group rules after VM migration
Add security group rules on target host
Cause: vmops script expected secondary IPs as "0;" but received "0:"
Remove security group network rules on source host.
Cause: destroy_network_rules_for_vm function on vmops script was not
called when migrating VM
* Add unit tests and address reviewers
This ensures that the systemvm agent (cloud.service) is not restarted
on certificate import. The agent has an inbuilt logic to attempt reconnection.
If the old certificates/keystore is invalid agent will attempt reconnection.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* security group: Replace deprecated optparse by argparse
Starting with Python 2.7 the library optparse has been replaced by
argpase.
This commit replaces the use of optparse by argparse
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* security group: Remove LXC support from security_group.py
LXC does not work and has been partially removed from CloudStack already
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* security group: Refactor libvirt code
Use a single function which properly throws an Exception when the
connection to libvirt fails.
Also simplify some logic, make it PEP-8 compatible and remove a unused
function from the code.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* security group: Raise Exception on execute() failure
If the executed command exists with a non-zero exit status we should
still return the output to the command, but also raise an Exception.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* security group: Use a function to determin the physical device of a bridge
We can not safely assume that the first device listed under a bridge is the
physical device.
With VXLAN isolation a vnet device can be attached to a bridge prior to the
vxlanXXXX device being attached.
We need to filter out those devices and then fetch the physical device attached
to the bridge.
In addition use the 'bridge' command instead of 'brctl'. 'bridge' is part of the
iproute2 utils just like 'ip' and should be considered as the new default.
This command is also available on EL6 and does not break any backwards compat.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* security group: --set is deprecated, use --match-set
These messages are seen in the KVM Agent log:
--set option deprecated, please use --match-set
Functionality does not change
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* security group: PEP-8 and indentation fixes
There were a lot of styling problems in the code:
- Missing whitespace or exess whitespace
- CaMelCaSe function names and variables
- 2-space indentation instead of 4 spaces
This commit addresses those issues.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* vxlan: Code indentation and styling fixes
This script was using TAB instead of 4 spaces and had many blank
lines containing whitespace.
This commit also fixes some Bash styling, but it does not touch the
functionality of the script.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* vxlan: Improve Bash if-statement logic
Bash suggest using double brackets instead of single brackets in
if-statement test logic
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* vxlan: Disable IPv6 on bridge and VXLAN devices
They are only transport devices and should not be interacting
in the IPv6 traffic.
If IPv6 is enabled Instances can connect to the Hypervisor over
Link-Local IPv6 which is a potential security issue.
By disabling IPv6 on the Bridge and VXLAN device they still forward
Layer 2 packets as intended, but they do not respond on anything.
IPv4 and IPv6 traffic towards the Instances is untouched and works
as before.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* vxlan: Refactor modifyvxlan.sh for KVM by using only iproute2
This commit refactors the modifyvxlan.sh script by using only iproute2,
the 'ip' command for all functions.
brctl is deprecated and most bridge functionality can be performed with
the 'ip' command.
This commit also fixes various Bash coding fixes and removes a lot of exit
status checking which was redundant.
In addition it add IPv6 underlay for VXLAN transport. If the caller (KVM Agent)
adds the '-6' flag it will generate IPv6 multicast groups and routes which will
transport the VXLAN encapsulated packaes over IPv6 multicast groups.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Currently secondary ipv6 addresses are added to the ipv4 ipset in security_group.py.
This doesn't work, so this patch adds a function to split a set of ips in ipv4 and ipv6 addresses.
Both the default_network_rules and network_rules_vmSecondaryIp functions now utilise this function and add the ips to the appropriate ipsets.
On actual testing, I could see that kvmheartbeat.sh script fails on NFS
server failure and stops the agent only. Any HA VMs could be launched
in different hosts, and recovery of NFS server could lead to a state
where a HA enabled VM runs on two hosts and can potentially cause
disk corruptions. In most cases, VM disk corruption will be worse than
VM downtime. I've kept the sleep interval between check/rounds but
reduced it to 10s. The change in behaviour was introduced in #2722.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Improved performance on creating VM for KVM virtualization.
On a huge hosts every "ifconfig | grep" takes a lot of time (about 2.5-3 minutes on hosts with 500 machines). For example: ip link show dev $vlanDev > /dev/null is faster than ifconfig |grep -w $vlanDev > /dev/null. But using ip command is much better. Using this patch you can create 500s machine in 10 seconds. You don't need slow ifconfig prints anymore.
Incorrect diskpath information was being sent to virsh blockresize, so the block device size was never refreshed to reflect the new disk size.
Fixes#2868
In 6233a77d15 as a part of PR #2432 the
bash() function was replaced by the execute() function.
Somehow this last calling of the bash() function was not caught by testing
and is still in there.
This causes Exceptions to be thrown by the Security Group script.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* [CLOUDSTACK-9261] Upgrate jQuery-UI to 1.11 (JQuery UI 1.8.4 prone to XSS)
* fix problems in the UI for lbCertificatePolicy and StaticNAT
* force jenkins build
* Fix about dialog
* Fix position of network service offering
This ensures that certificate setup includes all the IP addresses (v4
and v6) when a (KVM) host is added to CloudStack. This fixes#2530.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
On RHEL/CentOS/Fedora the bridge related sysctl rules are enabled
in kernel by default but can only be disabled. Enabling those keys
will fail, causing iptables/ebtables tables to not be created
and fails SG on CentOS.
This also fixes an integration test case, which assumes first few
tests complete within 3 minutes. In nested env the value may be large,
this increases the value to 20 minutes.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This extends securing of KVM hosts to securing of libvirt on KVM
host as well for TLS enabled live VM migration. To simplify implementation
securing of host implies that both host and libvirtd processes are
secured with management server's CA plugin issued certificates.
Based on whether keystore and certificates files are available at
/etc/cloudstack/agent, the KVM agent determines whether to use TLS or
TCP based uris for live VM migration. It is also enforced that a secured
host will allow live VM migration to/from other secured host, and an
unsecured hosts will allow live VM migration to/from other unsecured
host only.
Post upgrade the KVM agent on startup will expose its security state
(secured detail is sent as true or false) to the managements server that
gets saved in host_details for the host. This host detail can be accesed
via the listHosts response, and in the UI unsecured KVM hosts will show
up with the host state of ‘unsecured’. Further, a button has been added
that allows admins to provision/renew certificates to KVM hosts and can
be used to secure any unsecured KVM host.
The `cloudstack-setup-agent` was modified to accept a new flag `-s`
which will reconfigure libvirtd with following settings:
listen_tcp=0
listen_tls=1
tcp_port="16509"
tls_port="16514"
auth_tcp="none"
auth_tls="none"
key_file = "/etc/pki/libvirt/private/serverkey.pem"
cert_file = "/etc/pki/libvirt/servercert.pem"
ca_file = "/etc/pki/CA/cacert.pem"
For a connected KVM host agent, when the certificate are
renewed/provisioned a background task is scheduled that waits until all
of the agent tasks finish after which libvirt process is restarted and
finally the agent is restarted via AgentShell.
There are no API or DB changes.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- We should return a boolean and not a String 'true' or 'false'. Although this output is never checked by the calling function(s).
- Do not use == False or == None as that is not according to the Python specs.
- Calling just print 'hello' is deprecated and won't work in newer Python versions. We should use the print() function.
- Remove unused and commented function.
- Use logging.warning() instead of logging.warn()
- Use subprocess.check_output() for execution. This is the Python way of executing commands.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Automate dynamic roles migration for missing props file
- In case commands.properties file is missing, enables dynamic roles.
- Adds a new -D or --default flag to migrate-dynamicroles.py script
to simply update the global setting and use the default role-rule
permissions.
- Add warning message, ask admins to move to dynamic roles during upgrade
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
With merge of PR #2028 the separator for lines to the Security Group
Python script changed from : to ; to support IPv6 addresses.
This broke certain situations where rules were parsed improperly. This
commit fixes the issue.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Daniel Augusto Veronezi Salvador
Wei Zhou
Pearl Dsilva
Abhishek Kumar
John Bampton
Gabriel Beims Bräscher
Nicolas Vazquez
davidjumani
Rohit Yadav
Wei Zhou
Rakesh
Abhishek Kumar
Daan Hoogland
Wei Zhou
ServiceTo
Spaceman1984
Andrija Panic
Paul Angus
Anurag Awasthi
Kai Takahashi
Sven Vogel
Wido den Hollander
Rene Diepstraten
Richard Lawley
Gabriel Beims Bräscher
Rohit Yadav
Rene Diepstraten
alexanderbazhenoff
Simon Weller
Khosrow Moossavi
Slair1
Rafael Weingärtner
Olivier Lemasle