Commit Graph

1138 Commits

Author SHA1 Message Date
Wido den Hollander 4d7e95ec04 kvm: Use 'ip' instead of 'brctl' (#3623)
* kvm: Use 'ip' instead of 'brctl'

The command 'brctl' is deprecated and should no longer be used.

iproute2 supports all the features we need and therefor we should use
this instead of the old commands.

Feature wise this does not change anything. It just makes the code more
robust towards the future.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* kvm/modifyvlan: Use 'ip' instead of 'brctl'

brctl is deprecated and by using iproute2 we are future-proof

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-11-11 21:51:46 +05:30
Paul Angus 50fc045f36 Updating pom.xml version numbers for release 4.14.0.0-SNAPSHOT
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
2019-09-07 09:57:46 +01:00
Paul Angus 61b8b77913 Updating pom.xml version numbers for release 4.13.1.0-SNAPSHOT
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
2019-09-01 13:36:50 +01:00
Paul Angus 8e08b47cc9 Updating pom.xml version numbers for release 4.13.0.0
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
2019-08-20 15:35:49 +01:00
Gabriel Beims Bräscher 5dc982d8ba KVM local migration issue #3521 (#3533)
Fix regression bug that affects KVM local storage migration. Some of the desired execution flows for KVM local storage migration had been altered to allow only managed storage to execute. Fixed allowing managed and non managed storages to execute.

Fixes #3521
2019-08-07 15:41:30 +05:30
Wido den Hollander e894658f8c kvm: Allow Link Local Cidr (cloud0 interface) to be configured (#3500)
There are certain scenarios where the 169.254.0.0/16 subnet is used for different
purposes then CloudStack on a hypervisor.

Once of such scenarios is a BGP+EVPN+VXLAN setup using BGP Unnumbered where the
169.254.0.1 address is used by Frr/Zebra BGP routing to send traffic to the
neighboring router.

The following settings can be changed in the agent.properties (default values added):

control.cidr=169.254.0.0/16

Make sure the global setting 'control.cidr' matches the values defined in the agent.propeties!

In the future the mgmt server can send this parameter to a KVM Agent on startup, but at the moment
this framework is not in place and thus these values can't be send to the Agent in a proper manner.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-07-25 12:13:39 +05:30
Sid Kattoju c1d3f98775 server: filter volumes by host when refreshing stats (#3486)
Currently when refreshing disk usage stats all kvm agents are asked to collect stats for all volumes. In setups with multiple kvm hosts where managed storage is used, not all volumes are attached to all kvm hosts, this results in a large number of warnings in the kvm agent logs. This change introduces a filter step in case managed storage is used so that the management server only requests kvm agents for stats about volumes that are connected to each kvm host.
2019-07-24 11:59:51 +05:30
Gabriel Beims Bräscher 6a511fce40 kvm: Add ceph RBD snapshot rollback (#3502)
Add CephSnapshotStrategy to handle RBD revert (rollback) snapshot. In order to support RBD revert (rbd_rollback), this PR adds a CephSnapshotStrategy class to handle Ceph/RBD snapshot actions.
2019-07-23 19:40:56 +05:30
Nicolas Vazquez a1a9fb8977
KVM: Enhancements for direct download feature (#3374)
* Add revoke certificates API

* Add background task to sync certificates

* Fix marvin test and revoke certificate

* Fix certificate sent to hypervisor was missing headers

* Fix background task for uploading certificates to hosts
2019-07-18 02:39:00 -03:00
Sid Kattoju 1bf4dd39ca kvm: use IDE as the bus type for root disks and VIRTIO for data disks on platforms without support for para virtualization when using managed storage (#3319)
This change addresses #3089. There was an issue when disks were being added with bus type IDE when creating windows VMs from ISOs. It is not possible to select bus type when creating a VM with an ISO. The bus type is inferred based on the platform emulator string provided to the KVM agent. Currently when creating a VM with managed storage (ex: Solidfire) and OS type string Windows*, all disks are added as IDE. Qemu currently does not support multiple IDE controllers and this configuration results in VMs that cannot be started. This issue does not occur when using NFS as the storage provider due to logic in that KVM agent that makes all data volumes (non root) use a virtio controller for file based disk. Similar logic was added for raw physical disks so that managed storage has the same behavior as NFS. In addition specific versions were removed from the code that guesses the disk controller to be used based on the platform emulator string since most modern operating systems support virtio.

Fixes #3089
2019-07-12 12:35:23 +05:30
Nicolas Vazquez a75444a585
KVM: DPDK live migrations (#3365)
* DPDK live migrations

* Remove DPDK created ports if VM migration fails or prepare migration fails

* Rename DPDK classes lowercase
2019-06-25 12:23:09 -03:00
Nicolas Vazquez 0fbf5006b8 kvm: live storage migration intra cluster from NFS source and destination (#2983)
Feature Specification: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95653548

Live storage migration on KVM under these conditions:

From source and destination hosts within the same cluster
From NFS primary storage to NFS cluster-wide primary storage
Source NFS and destination NFS storage mounted on hosts
In order to enable this functionality, database should be updated in order to enable live storage capacibilty for KVM, if previous conditions are met. This is due to existing conflicts between qemu and libvirt versions. This has been tested on CentOS 6 hosts.

Additional notes:

To use this feature set the storage_motion_supported=1 in the hypervisor_capability table for KVM. This is done by default as the feature may not work in some environments, read below.
This feature of online storage+VM migration for KVM will only work with CentOS6 and possible Ubuntu as KVM hosts but not with CentOS7 due to:
https://bugs.centos.org/view.php?id=14026
https://bugzilla.redhat.com/show_bug.cgi?id=1219541
On CentOS7 the error we see is: " error: unable to execute QEMU command 'migrate': this feature or command is not currently supported" (reference https://ask.openstack.org/en/question/94186/live-migration-unable-to-execute-qemu-command-migrate/). Reading through various lists looks like the migrate feature with qemu may be available with paid versions of RHEL-EV but not centos7 however this works with CentOS6.
Fix for CentOS 7:

Create repo file on /etc/yum.repos.d/:
[qemu-kvm-rhev]
name=oVirt rebuilds of qemu-kvm-rhev
baseurl=http://resources.ovirt.org/pub/ovirt-3.5/rpm/el7Server/
mirrorlist=http://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.5-el7Server
enabled=1
skip_if_unavailable=1
gpgcheck=0
yum install qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-ev-2.3.0-29.1.el7.x86_64 qemu-img-ev-2.3.0-29.1.el7.x86_64
Reboot host

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-10 15:35:26 +05:30
nvazquez 50869fec92 Merge branch '4.12' 2019-06-04 09:49:39 -03:00
nvazquez bea627a52e Merge branch '4.11' into 4.12 2019-06-04 09:06:09 -03:00
Vladimir Melnik c94ee1454d kvm: suspend a VM before snapshot deletion (see PR #3193) (#3194)
To make sure that a qemu2-image won't be corrupted by the snapshot deletion procedure which is being performed after copying the snapshot to a secondary store, I'd propose to put a VM in to suspended state.

Additional reference: https://bugzilla.redhat.com/show_bug.cgi?id=920020#c5

Fixes #3193
2019-06-04 16:04:45 +05:30
Nicolas Vazquez 12c850ed2f
KVM: Improvements on upload direct download certificates (#2995)
* Improvements on upload direct download certificates

* Move upload direct download certificate logic to KVM plugin

* Extend unit test certificate expiration days

* Add marvin tests and command to revoke certificates

* Review comments

* Do not include revoke certificates API
2019-06-04 03:08:31 -03:00
Nicolas Vazquez 501aa7cd91
DPDK vHost User mode selection (#3153)
* DPDK vHost User mode selection

* SQL text field and DPDK classes refactor

* Fix NullPointerException after refactor

* Fix unit test

* Refactor details type
2019-05-29 08:36:33 -03:00
ustcweizhou 798b79fa5b kvm: disable cpu features if feature starts with '-' (#3335)
When I use SandyBridge as custom cpu in my testing, vm failed to start due to following error:
```
org.libvirt.LibvirtException: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: avx, xsave, aes, tsc-deadline, x2apic, pclmuldq
```

With this patch, it works with the following setting in agent.properties:
```
  guest.cpu.mode=custom
  guest.cpu.model=SandyBridge
  guest.cpu.features=-avx -xsave -aes -tsc-deadline -x2apic -pclmuldq
```

vm cpu is defined as below:
```
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='aes'/>
    <feature policy='disable' name='tsc-deadline'/>
    <feature policy='disable' name='x2apic'/>
    <feature policy='disable' name='pclmuldq'/>
  </cpu>
```
2019-05-27 18:43:38 +05:30
Rohit Yadav 2aa4842a94 Merge remote-tracking branch 'origin/4.12' 2019-05-23 18:17:42 +05:30
Rohit Yadav d77e69a2f2 Merge remote-tracking branch 'origin/4.11' into 4.12
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-05-23 18:16:45 +05:30
Rohit Yadav 0929866956
server: ssh-keygen in PEM format and reduce main systemvm patching script (#3333)
On first startup, the management server creates and saves a random
ssh keypair using ssh-keygen in the database. The command does
not specify keys in PEM format which is not the default as generated
by latest ssh-keygen tool.

The systemvmtemplate always needs re-building whenever there is a change
in the cloud-early-config file. This also tries to fix that by introducing a
stage 2 bootstrap.sh where the changes specific to hypervisor detection
etc are refactored/moved. The initial cloud-early-config only patches
before the other scripts are called.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-05-23 18:08:00 +05:30
Rohit Yadav 0700d91a68 Merge branch '4.12'
- Fixes PR #3146 db cleanup to the correct 4.12->4.13 upgrade path
- Fixes failing unit test due to jdk specific changes after forward
  merging

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-05-14 15:15:17 +05:30
Frank Maximus e11f7ee1ba RIP Nuage Cloudstack Plugin (#3146)
may it rest in peaces
2019-05-14 10:58:24 +02:00
Rohit Yadav 00ff536f81 Merge remote-tracking branch 'origin/4.11' into 4.12
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-05-14 14:26:11 +05:30
Rohit Yadav 9ff819da2c
systemvm: new qemu-guest-agent based patching for KVM (#3278)
This introduces a new patching script for patching systemvms on KVM
using qemu-guest-agent that runs inside the systemvm on startup. This
also removes the vport device which was previously used by the legacy
patching script and instead uses the modern and new uniform guest
agent vport for host-guest communication.

Also updates the sytemvmtemplate build config to use the latest Debian
9.9.0 iso.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-05-10 23:42:19 +05:30
ustcweizhou 3729511c37 kvm: Fix three issues with Ubuntu 16.04 hosts (#3227)
* ubuntu16: fix unable to add host if cloudbrX is not configured

while add a ubuntu16.04 host with native eth0 (cloudbrX is not configured),
the operation failed and I got the following error in /var/log/cloudstack/agent/setup.log

```
DEBUG:root:execute:ifconfig eth0
DEBUG:root:[Errno 2] No such file or directory
  File "/usr/lib/python2.7/dist-packages/cloudutils/serviceConfig.py", line 38, in configration
    result = self.config()
  File "/usr/lib/python2.7/dist-packages/cloudutils/serviceConfig.py", line 211, in config
    super(networkConfigUbuntu, self).cfgNetwork()
  File "/usr/lib/python2.7/dist-packages/cloudutils/serviceConfig.py", line 108, in cfgNetwork
    device = self.netcfg.getDefaultNetwork()
  File "/usr/lib/python2.7/dist-packages/cloudutils/networkConfig.py", line 53, in getDefaultNetwork
    pdi = networkConfig.getDevInfo(dev)
  File "/usr/lib/python2.7/dist-packages/cloudutils/networkConfig.py", line 157, in getDevInfo
    elif networkConfig.isBridge(dev) or networkConfig.isOvsBridge(dev):
```

The issue is caused by commit 9c7cd8c248
2017-09-19 16:45 Sigert Goeminne ● CLOUDSTACK-10081: CloudUtils getDevInfo function will now return "bridge" instead o

* ubuntu16: Stop service libvirt-bin.socket while add a host

service libvirt-bin.socket will be started when add a ubuntu 16.04 host
DEBUG:root:execute:sudo /usr/sbin/service libvirt-bin start

However, libvirt-bin service will be broken by it after restarting
Stopping service libvirt-bin.socket will fix the issue.

An example is given as below.

```
root@node32:~# /etc/init.d/libvirt-bin restart
[ ok ] Restarting libvirt-bin (via systemctl): libvirt-bin.service.
root@node32:~# virsh list
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

root@node32:~# systemctl stop libvirt-bin.socket

root@node32:~# /etc/init.d/libvirt-bin restart
[ ok ] Restarting libvirt-bin (via systemctl): libvirt-bin.service.
root@node32:~# virsh list
 Id    Name                           State
----------------------------------------------------

```

* ubuntu16: Diable libvirt default network

By default, libvirt will create default network virbr0 on kvm hypervisors.
If vm uses the same ip range 192.168.122.0/24, there will be some issues.

In some cases, if we run tcpdump inside vm, we will see the ip of kvm hypervisor as source ip.
2019-05-05 13:57:58 +05:30
Gabriel Beims Bräscher 8f7b27bbdc Mock Scanner, instead of scan the computer running the test. (#3173)
* Mock Scanner, instead of scan the computer running the test.

This allows non linux machines to run the tests without scanning for a
non existing /proc/meminfo.

* test fixes on 'other' platforms libvirt wrapper unit tests (#3)
2019-04-24 13:33:06 +02:00
dahn f88de7b863 kvm: NotImplemented, with message to signify what is not implemented, using commons-lang version (#3170)
NotImplementedException from sun.reflect.generics.reflectiveObjects is not standard/guarateed by the jvm so a custom implementation is in order.
2019-03-25 13:58:08 +05:30
GabrielBrascher 8d3feb100a Updating pom.xml version numbers for release 4.13.0.0-SNAPSHOT
Signed-off-by: GabrielBrascher <gabriel@pcextreme.nl>
2019-03-20 18:47:35 -03:00
GabrielBrascher a137398bf1 Updating pom.xml version numbers for release 4.12.0.0
Signed-off-by: GabrielBrascher <gabriel@pcextreme.nl>
2019-03-14 10:11:46 -03:00
Gabriel Beims Bräscher 709845f4a3
Keep iotune section in the VM's XML after live migration (#3171)
* Keep iotune section in the VM's XML after live migration

When live migrating a KVM VM among local storages, the VM loses the
<iotune> section on its XML, therefore, having no IO limitations.

This commit removes the piece of code that deletes the <iotune> section
in the XML.

* Add test for replaceStorage in LibvirtMigrateCommandWrapper

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* Fix Javadoc for method replaceIpForVNCInDescFile
2019-02-12 22:07:03 -02:00
Nathan Johnson 637cc6ec4e feature: add libvirt / qemu io bursting (#3133)
* feature: add libvirt / qemu io bursting

Adds the ability to set bursting features from libvirt / qemu

This allows you to utilize the iops and bytes temporary "burst" mode
introduced with libvirt 2.4 and improved upon with libvirt 2.6.

https://blogs.igalia.com/berto/2016/05/24/io-bursts-with-qemu-2-6/

* updates per rafael et al
2019-02-04 19:47:44 -02:00
Wido den Hollander c496c84c6c kvm: Properly report available memory to Management Server (#2795)
The KVM Agent had two mechanisms for reporting its capabilities
and memory to the Management Server.

On startup it would ask libvirt the amount of Memory the Host has
and subtract and add the reserved and overcommit memory.

When the HostStats were however reported to the Management Server
these two configured values on the Agent were no longer reported
in the statistics thus showing all the available memory in the
Agent/Host to the Management Server.

This commit unifies this by using the same logic on Agent Startup
and during statistics reporting.

  memory=3069636608, reservedMemory=1073741824

This was reported by a 4GB Hypervisor with this setting:

  host.reserved.mem.mb=1024

The GUI (thus API) would then show:

  Memory Total	2.86 GB

This way the Agent properly 'lies' to the Management Server about its
capabilities in terms of Memory.

This is very helpful if you want to overprovision or undercommit machines
for various reasons.

Overcommitting can be done when KSM or ZSwap or a fast SWAP device is
installed in the machine.

Underprovisioning is done when the Host might run other tasks then a KVM
hypervisor, for example when it runs in a hyperconverged setup with Ceph.

In addition internally many values have been changed from a Double to a Long
and also store the amount of bytes instead of Kilobytes.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-24 20:18:04 -02:00
Wido den Hollander ac73e7e671 kvm: Security Group enhancements and refactor old code (#3113)
* security group: Replace deprecated optparse by argparse

Starting with Python 2.7 the library optparse has been replaced by
argpase.

This commit replaces the use of optparse by argparse

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Remove LXC support from security_group.py

LXC does not work and has been partially removed from CloudStack already

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Refactor libvirt code

Use a single function which properly throws an Exception when the
connection to libvirt fails.

Also simplify some logic, make it PEP-8 compatible and remove a unused
function from the code.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Raise Exception on execute() failure

If the executed command exists with a non-zero exit status we should
still return the output to the command, but also raise an Exception.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Use a function to determin the physical device of a bridge

We can not safely assume that the first device listed under a bridge is the
physical device.

With VXLAN isolation a vnet device can be attached to a bridge prior to the
vxlanXXXX device being attached.

We need to filter out those devices and then fetch the physical device attached
to the bridge.

In addition use the 'bridge' command instead of 'brctl'. 'bridge' is part of the
iproute2 utils just like 'ip' and should be considered as the new default.

This command is also available on EL6 and does not break any backwards compat.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: --set is deprecated, use --match-set

These messages are seen in the KVM Agent log:

  --set option deprecated, please use --match-set

Functionality does not change

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: PEP-8 and indentation fixes

There were a lot of styling problems in the code:

- Missing whitespace or exess whitespace
- CaMelCaSe function names and variables
- 2-space indentation instead of 4 spaces

This commit addresses those issues.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-16 13:35:18 -02:00
Wido den Hollander c565db2cf2 kvm: Set amount of queues for Virtio SCSI driver to vCPU of Instance (#3101)
The additional queues can enhance the performance of the VirtIO SCSI disk
and it is recommended to set this to the amount of vCPUs a Instance is assigned.

  The optional queues attribute specifies the number of queues for the
  controller. For best performance, it's recommended to specify a value matching
  the number of vCPUs. Since 1.0.5 (QEMU and KVM only)

Source: https://libvirt.org/formatdomain.html#elementsVirtio

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-08 10:39:21 +01:00
Gabriel Beims Bräscher bf209405e7 Allow KVM VM live migration with ROOT volume on file storage type (#2997)
* Allow KVM VM live migration with ROOT volume on file

* Allow KVM VM live migration with ROOT volume on file
- Add JUnit tests

* Address reviewers and change some variable names to ease future
implementation (developers can easily guess the name and use
autocomplete)
2018-12-14 09:01:28 -02:00
Bitworks LLC 9dce8a5dea kvm: Added two more device name patterns to valid bridge slaves (lo* and dummy*) (#3000)
Added dummy and lo devices to be treated as a normal bridge slave devs.
Fixes #2998  
Added two more device names (lo* and dummy*). Implemented tests. Code was refactored.
Improved paths concatenation code from "+" to Paths.get.
2018-12-07 01:59:00 +05:30
Wido den Hollander d96bc05d10 kvm: Use 'ip route show default 0.0.0.0/0' to find the default gateway (#3080)
If a host has many routes this can be a magnitude faster then printing
all the routes and grepping for the default.

In some situations the host might have a large amount of routes due to
dynamic routing being used like OSPF or BGP.

In addition fix a couple of loglines which were throwing messages on
DEBUG while WARN and ERROR should be used there.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-12-06 14:04:47 +05:30
Rohit Yadav ac9562a4a1 Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-11-29 15:06:06 +05:30
Rohit Yadav 29b8a9da48
kvm: when untagged vxlan is used, use the default guest/public bridge (#3037)
When vxlan://untagged is used for public (or guest) network, use the
default public/guest bridge device same as how vlan://untagged works.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-11-28 22:22:30 +05:30
Paul Angus fb80e51307 Updating pom.xml version numbers for release 4.11.3.0-SNAPSHOT
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
2018-11-20 13:11:52 +00:00
Wido den Hollander 4809fe7cb4 kvm/ceph: When create a new RBD image use modern flags/features for the image (#2985)
These additional RBD features allow for faster lookups of how much space a RBD
image is using, but with the exclusive locking we prevent two VMs from writing
to the same RBD image at the same time.

These are the default features used by Ceph for any new RBD image.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-11-12 10:08:54 -02:00
Nicolas Vazquez 4de4eabd18
Enable DPDK support on KVM (#2839)
* Enable DPDK support on KVM

* Allow DPDK deployments on user VMs only

* Fix port name ordering
2018-11-07 09:29:01 -03:00
Rohit Yadav 7479e2877f Merge remote-tracking branch 'origin/4.11' 2018-10-30 15:15:35 +05:30
Rohit Yadav c6e53f6cc6
kvm: reset KVM host on heartbeat failure (#2984)
On actual testing, I could see that kvmheartbeat.sh script fails on NFS
server failure and stops the agent only. Any HA VMs could be launched
in different hosts, and recovery of NFS server could lead to a state
where a HA enabled VM runs on two hosts and can potentially cause
disk corruptions. In most cases, VM disk corruption will be worse than
VM downtime. I've kept the sleep interval between check/rounds but
reduced it to 10s. The change in behaviour was introduced in #2722.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-30 15:13:59 +05:30
Rohit Yadav 323d381767 Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:27:08 +05:30
Nicolas Vazquez dffb430975 kvm: Fix migrating VM from ISO failures (#2928)
Prevents errors while migrating VM from ISO:

Test 1: Deploy VM from ISO -> Live migrate VM to another host -> ERROR
Test 2: Register ISO using Direct Download on KVM -> Deploy VM from ISO -> Live migrate VM to another host -> ERROR

- Prevent NullPointerException migrating VM from ISO
- Prevent mount secondary storage on ISO direct downloads on KVM
2018-10-29 16:14:20 +05:30
Rohit Yadav 8738ca75b1
kvm: use libvirtd as service name for all distributions (#2925)
Since we support only Ubuntu 16.04+ on master/4.12+, we can now use
the libvirt service name `libvirtd` for all distributions. This also
fixes an optional package name for libvirtd installation on Debian 9+.
Fixes #2909

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 11:55:28 +05:30
Simon Weller c4b621a418 kvm: HyperV Enlightment for Improved Windows Server 2008+ Performance (#2870)
Windows has support for several paravirt features that it will use when running on Hyper-V, Microsoft's hypervisor. These features are called enlightenments. Many of the features are similar to paravirt functionality that exists with Linux on KVM (virtio, kvmclock, PV EOI, etc.)

Nowadays QEMU/KVM can also enable support for several Hyper-V enlightenments. When enabled, Windows VMs running on KVM will use many of the same paravirt optimizations they would use when running on Hyper-V.

A number of years ago, a PR was introduced that added a good portion of the code to enable this feature set, but it was never completed. This PR enables the existing features. The previous patch set detailed in #1013 also included the tests.

By selecting Windows PV, the enlightenment additions will be applied to the libvirt configuration. This is support on Windows Server 2008 and beyond, so all currently supported versions of Windows Server.

In our testing, we've seen benchmark improvements of around 20-25% running on Centos 7 hosts and it is also supported on Centos/RHEL 6.5 and later. Testing on Ubuntu would be appreciated.
2018-10-25 06:54:13 +05:30
Abhishek f5d52421b9 kvm: Remove libvirt-bin check for Debian (#2909) (#2912)
Debian does not have libvirt-bin package. Therefore, only for Ubuntu host distro process name libvirt-bin is used, otherwise libvirtd will be used
2018-10-22 17:48:11 +05:30
Rohit Yadav 3614f5dd8b Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-08-22 15:55:21 +05:30
Slair1 023dcec5ef CLOUDSTACK-10310 Fix KVM reboot on storage issue (#2722) 2018-08-20 10:28:03 +02:00
Wido den Hollander 65f31f1a9f kvm: Agent should not check if remaining memory on host is sufficient (#2766)
When a Instance is (attempted to be) started in KVM Host the Agent
should not worry about the allocated memory on this host.

To make a proper judgement we need to take more into account:

- Memory Overcommit ratio
- Host reserved memory
- Host overcommit memory

The Management Server has all the information and the DeploymentPlanner
has to make the decision if a Instance should and can be started on a
Host, not the host itself.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-08-08 12:14:26 +05:30
Rohit Yadav 7667846bfa
kvm: Fixes #2763 move post-renewal libvirt restart class suitably (#2764)
This fixes #2763 by moving a post cert-renewal class for kvm
plugin/hypervisor to src/main/java. The regression is due to change
in file-system layout due to maven standard refactoring on master and
issue was not caught during forward-merging of a PR from 4.11 branch.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-07-26 12:51:24 +05:30
Khosrow Moossavi 7c6630bca7 Cleanup POMs (#2613)
* Cleaup and code-formatting POM files

* Remove obsolete mycila license-maven-plugin

* Remove obsolete console-proxy/plugin project

* Move console-proxy-rdbconsole under console-proxy parent

* Use correct parent path for rdpconsole

* Order alphabetally items in setnextversion.sh

* Unifiy License header in POMs

* Alphabetic order of modules definition

* Extract all defined versions into parent pom

* Remove obsolete files: version-info.in, configure-info.in

* Remove redundant defaultGoal

* Remove useless checkstyle plugin from checkstyle project

* Order alphabetally items in pom.xml

* Add aditional SPACEs to fix debian build

* Don't execute checkstyle on parent projects

* Use UTF-8 encoding in building checkstyle project

* Extract plugin versions into properties

* Execute PMD plugin on all the projects with -Penablefindbugs

* Upgrade maven plugins to latest version

* Make sure to always look for apache parent pom from repository

* Fix incorrect version grep in debian packaging

* Fix rebase conflicts

* Fix rebase conflicts

* Remove PMD for now to be fixed on another PR
2018-07-25 14:39:37 -03:00
Wido den Hollander 443490179c cleanup: Remove unused variables (#2765)
Found some variables throughout the code which are not being used.
They can be removed.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-07-24 21:01:25 -03:00
Khosrow Moossavi 67860d9f46 maven: Updating pom.xml version numbers for release 4.11.2.0-SNAPSHOT (#2728)
Fixes the version in pom etc. to be consistent with versioning pattern as X.Y.Z.0-SNAPSHOT after a minor release.

Signed-off-by: Khosrow Moossavi <khos2ow@gmail.com>
2018-07-06 17:27:12 +05:30
Paul Angus 8ba318da19 Updating pom.xml version numbers for release 4.11.2-SNAPSHOT
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
2018-06-26 17:53:54 +01:00
Paul Angus 2cb2dacbe7 Updating pom.xml version numbers for release 4.11.1.0
Signed-off-by: Paul Angus <paulangus@PA-Ansible-GUI.sblab.local>
2018-06-21 15:52:43 +01:00
Daan Hoogland 1d05fead49 Merge branch '4.11' 2018-06-21 13:08:55 +02:00
dahn f02e402ebb kvm: send unsupported answer only when applicable (#2714)
Throw specific NPE child when command is known not to be known. Add unit tests.
2018-06-21 11:03:43 +05:30
Rohit Yadav 85750f918b Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-06-20 12:31:52 +05:30
Rohit Yadav 39471c8c00
configdrive: make fewer mountpoints on hosts (#2716)
This ensure that fewer mount points are made on hosts for either
primary storagepools or secondary storagepools.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-06-20 12:25:16 +05:30
Rohit Yadav 5c28a2a5b8 Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-06-07 16:07:52 +05:30
cl-k-takahashi d67af8661b kvm: check if storage pool is mounted before creating pool xml (#2696)
Now the KVM agent checks whether a storage pool is mounted or not mounted before calling storagePoolCreateXML().

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
2018-06-07 12:23:21 +05:30
Rohit Yadav 7c6777b8d3 Merge branch '4.11': allow config drives on primary storage for KVM (#2651)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 14:50:55 +05:30
Rohit Yadav acc5fdcdbd
CLOUDSTACK-10290: allow config drives on primary storage for KVM (#2651)
This introduces a new global setting `vm.configdrive.primarypool.enabled` to toggle creation/hosting of config drive iso files on primary storage, the default will be false causing them to be hosted on secondary storage. The current support is limited from hypervisor resource side and in current implementation limited to `KVM` only. The next big change is that config drive is created at a temporary location by management server and shipped to either KVM or SSVM agent via cmd-answer pattern, the data of which is not logged in logs. This saves us from adding genisoimage dependency on cloudstack-agent pkg.

The APIs to reset ssh public key, password and user-data (via update VM API) requires that VM should be shutdown. Therefore, in the refactoring I removed the case of updation of existing ISO. If there are objections I'll re-put the strategy to detach+attach new config iso as a way of updation. In the refactored implementation, the folder name is changed to lower-cased configdrive. And during VM start, migration or shutdown/removal if primary storage is enable for use, the KVM agent will handle cleanup tasks otherwise SSVM agent will handle them.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 14:27:23 +05:30
Rohit Yadav 93e374599a Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 13:23:13 +05:30
Mike Tutkowski 7e6fddb7ab managed-storage: Handle Ceph (#2655)
In 4.11.0, I added the ability to online migrate volumes from NFS to managed storage. This actually works for Ceph to managed storage in a private 4.8 branch, as well. I thought I had brought along all of the necessary code from that private 4.8 branch to make Ceph to managed storage functional in 4.11.0, but missed one piece (which is fixed by this PR).
2018-05-21 12:54:42 +05:30
Rohit Yadav ffe86e1c17 Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-08 16:02:17 +05:30
Rohit Yadav bd0959517b
hypervisor: allow Ubuntu 18.04 to be added as KVM host (#2626)
This adds and allows Ubuntu 18.04 to be used as KVM host. In addition,
on the UI when hypervisor version key is missing, this adds and display
the host os and version detail which is useful to show the KVM host
os and version.

When cache mode 'none' is used for empty cdrom drives, systemvms
and guest VMs fail to start on newer libvirtd such as Ubuntu bionic.
The fix is ensure that cachemode is not declared when drives are empty
upon starting of the VM. Similar issue logged at redhat here:
https://bugzilla.redhat.com/show_bug.cgi?id=1342999

The workaround is to ensure that we don't configure cachemode for
cdrom devices at all. This also fixes live VM migration issue.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-08 15:56:49 +05:30
Nathan Johnson a53dcd6aa9 ceph: Fixes #2611 use raw disk type for rdb (#2623)
Fix issue where kvm / ceph cannot create volumes. Fixes #2611
2018-05-08 15:00:44 +05:30
Kui LIU 503b2c649c CLOUDSTACK-10364: Change the "setXXX" method names to "generateXXX" (#2601)
The three methods are named as "setXXX", actually, they are not simple setter or getter.
They are further renamed as "generateXXX" with dahn's comments.
2018-05-01 20:47:54 +05:30
brett 4c42aafae0 [CLOUDSTACK-10356] Fix NPE in Cloudstack found with NPEDetector (#2573)
* fix https://issues.apache.org/jira/browse/CLOUDSTACK-10356

* del patch file

* Update ResourceCountDaoImpl.java

* fix some format

* fix code

* fix error message in VolumeOrchestrator

* add check null stmt

* del import unuse class

* use BooleanUtils to check Boolean

* fix error message

* delete unuse function

* delete the deprecated function  updateDomainCount

* add error log and throw exception in ProjectManagerImpl.java
2018-04-26 10:03:46 -03:00
Rohit Yadav 644b0910cd Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-20 00:46:43 +05:30
Rohit Yadav 8da2462469
CLOUDSTACK-10333: Secure Live VM Migration for KVM (#2505)
This extends securing of KVM hosts to securing of libvirt on KVM
host as well for TLS enabled live VM migration. To simplify implementation
securing of host implies that both host and libvirtd processes are
secured with management server's CA plugin issued certificates.

Based on whether keystore and certificates files are available at
/etc/cloudstack/agent, the KVM agent determines whether to use TLS or
TCP based uris for live VM migration. It is also enforced that a secured
host will allow live VM migration to/from other secured host, and an
unsecured hosts will allow live VM migration to/from other unsecured
host only.

Post upgrade the KVM agent on startup will expose its security state
(secured detail is sent as true or false) to the managements server that
gets saved in host_details for the host. This host detail can be accesed
via the listHosts response, and in the UI unsecured KVM hosts will show
up with the host state of ‘unsecured’. Further, a button has been added
that allows admins to provision/renew certificates to KVM hosts and can
be used to secure any unsecured KVM host.

The `cloudstack-setup-agent` was modified to accept a new flag `-s`
which will reconfigure libvirtd with following settings:

    listen_tcp=0
    listen_tls=1
    tcp_port="16509"
    tls_port="16514"
    auth_tcp="none"
    auth_tls="none"
    key_file = "/etc/pki/libvirt/private/serverkey.pem"
    cert_file = "/etc/pki/libvirt/servercert.pem"
    ca_file = "/etc/pki/CA/cacert.pem"

For a connected KVM host agent, when the certificate are
renewed/provisioned a background task is scheduled that waits until all
of the agent tasks finish after which libvirt process is restarted and
finally the agent is restarted via AgentShell.

There are no API or DB changes.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-20 00:36:18 +05:30
lzh3636 fed3492b57 CLOUDSTACK-10357: Improve log messages in methods (#2580)
Fix several logs that mismatch method.
Add stacktraces for throw new statements.
2018-04-20 00:33:27 +05:30
nvazquez 1c99fd7388 Merge branch '4.11' 2018-03-21 08:12:59 -03:00
Nicolas Vazquez 6a75423779 CLOUDSTACK-10231: Asserted fixes for Direct Download on KVM (#2408)
Several fixes addressed:

- Dettach ISO fails when trying to detach a direct download ISO
- Fix for metalink support on SSVM agents (this closes CLOUDSTACK-10238)
- Reinstall VM from bypassed registered template (this closes CLOUDSTACK-10250)
- Fix upload certificate error message even though operation was successful
- Fix metalink download, checksum retry logic and metalink SSVM downloader
2018-03-20 19:24:46 +05:30
Rohit Yadav 8ef131745a Merge branch '4.11' 2018-03-15 16:46:50 +05:30
Nicolas Vazquez 74db647dbb CLOUDSTACK-10321: CPU Cap for KVM (#2482) 2018-03-14 18:21:24 +00:00
Rohit Yadav 0ece15f86e Updating pom.xml version numbers for release 4.11.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-02-26 16:57:48 +01:00
Rohit Yadav 3db1746c03 Merge branch '4.11' 2018-02-26 15:32:20 +01:00
Nicolas Vazquez 703054964a CLOUDSTACK-10305: Rare race condition in KVM migration (#2466)
There is a race condition in the monitoring of the migration process on KVM. If the monitor wakes up in the tight window after the migration succeeds, but before the migration thread terminates, the monitor will get a LibvirtException “Domain not found: no domain with matching uuid” when checking on the migration status. This in turn causes CloudStack to sync the VM state to stop, in which it issues a defensive StopCommand to ensure it is correctly synced.

Fix: Prevent LibvirtException: "Domain not found" caused by the call to dm.getInfo()
2018-02-26 15:31:51 +01:00
Rohit Yadav f96398c127 Merge branch '4.11' 2018-02-14 11:56:00 +01:00
Frank Maximus dee5bb527e CLOUDSTACK-10261: Libvirt metadata create only one nuage-extension xml tag (#2441) 2018-02-12 12:06:44 +01:00
Rohit Yadav 6ffbce6159 Updating pom.xml version numbers for release 4.11.0.1-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-02-05 11:13:50 +01:00
Rohit Yadav 5dada1f7ed Updating pom.xml version numbers for release 4.11.0.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-26 13:13:37 +01:00
Rafael Weingärtner c591c5ad3e CLOUDSTACK-10248: Fix errors that appeared after #2283 (#2417)
This fixes move refactoring error introduced in #2283 
For instance, the class DatadiskTO is supposed to be in com.cloud.agent.api.to package. However, the folder structure it was placed in is com.cloud.agent.api.api.to.

Skip tests for cloud-plugin-hypervisor-ovm3:
For some unknown reason, there are quite a lot of broken test cases for cloud-plugin-hypervisor-ovm3. They might have appeared after some dependency upgrade and was overlooked by the person updating them. I checked them to see if they could be fixed, but these tests are not developed in a clear and clean manner. On top of that, we do not see (at least I) people using OVM3-hypervisor with ACS. Therefore, I decided to skip them.

Identention corrected to use spaces instead of tabs in XML files
2018-01-23 12:19:36 +01:00
Rohit Yadav 7e39961f22 Merge branch '4.11'
- CLOUDSTACK-10244: Cache whether to migrate storage

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-22 09:05:26 +01:00
Mike Tutkowski 2ca264ca4d CLOUDSTACK-10244: Cache whether to migrate storage (#2416)
KVM online storage migration fails, this fixes use of whether to migrate storage using a cached variable.
2018-01-21 15:49:54 +05:30
Marc-Aurèle Brothier 893a88d225 CLOUDSTACK-10105: Use maven standard project structure in all projects (#2283)
Remove maven standard module (which only a few were using) and get ride of maven customization for the projects structure.

- moved all directories to src/main/java, src/main/resources, src/main/scripts, src/test/java, src/test/resources
- grep scan to search for src/com and src/org left over
- grep for <project>/scripts to fix pom.xml configuration
- remove custom <build> configuration in pom.xml

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2018-01-20 03:19:27 +05:30
Rohit Yadav f7f14bfcc5 Merge branch '4.11' 2018-01-19 13:46:58 +05:30
Frank Maximus cfff78d9a1 CLOUDSTACK-10233: Use namespace in Libvirt domain metadata. (#2410)
The documentation of Libvirt specifies the requirement of using an XML namespace,
when having metadata in the Domain XML. The Nuage extenstion metadata was not
adhering to this specification, and the lastest Libvirt version ignores it in that case.
2018-01-19 13:44:16 +05:30
Rohit Yadav 072dbc0720 Updating pom.xml version numbers for master to 4.12.0.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-15 17:43:45 +05:30
Mike Tutkowski a30a31c9b7 CLOUDSTACK-9620: Enhancements for managed storage (#2298)
Allowed zone-wide primary storage based on a custom plug-in to be added via the GUI in a KVM-only environment (previously this only worked for XenServer and VMware)

Added support for root disks on managed storage with KVM

Added support for volume snapshots with managed storage on KVM

Enable creating a template directly from a volume (i.e. without having to go through a volume snapshot) on KVM with managed storage

Only allow the resizing of a volume for managed storage on KVM if the volume in question is either not attached to a VM or is attached to a VM in the Stopped state.

Included support for Reinstall VM on KVM with managed storage

Enabled offline migration on KVM from non-managed storage to managed storage and vice versa

Included support for online storage migration on KVM with managed storage (NFS and Ceph to managed storage)

Added support to download (extract) a managed-storage volume to a QCOW2 file

When uploading a file from outside of CloudStack to CloudStack, set the min and max IOPS, if applicable.

Included support for the KVM auto-convergence feature

The compression flag was actually added in version 1.0.3 (1000003) as opposed to version 1.3.0 (1003000) (changed this to reflect the correct version)

On KVM when using iSCSI-based managed storage, if the user shuts a VM down from the guest OS (as opposed to doing so from CloudStack), we need to pass to the KVM agent a list of applicable iSCSI volumes that need to be disconnected.

Added a new Global Setting: kvm.storage.live.migration.wait

For XenServer, added a check to enforce that only volumes from zone-wide managed storage can be storage motioned from a host in one cluster to a host in another cluster (cannot do so at the time being with volumes from cluster-scoped managed storage)

Don’t allow Storage XenMotion on a VM that has any managed-storage volume with one or more snapshots.

Enabled for managed storage with VMware: Template caching, create snapshot, delete snapshot, create volume from snapshot, and create template from snapshot

Added an SIOC API plug-in to support VMware SIOC

When starting a VM that uses managed storage in a cluster other than the one it last was running in, we need to remove the reference to the iSCSI volume from the original cluster.

Added the ability to revert a volume to a snapshot

Enabled cluster-scoped managed storage

Added support for VMware dynamic discovery
2018-01-15 00:05:52 +05:30
Frank Maximus b176648f90 CLOUDSTACK-9813: Extending Config Drive support (#2097)
Extending Config Drive support

* Added support for VMware
* Build configdrive.iso on ssvm
* Added support for VPC and Isolated Networks
* Moved implementation to new Service Provider
* UI fix: add support for urlencoded userdata
* Add support for building systemvm behind a proxy

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
2018-01-12 15:14:40 +05:30
Nicolas Vazquez e86bb41e0e CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates (#2379)
This feature allows using templates and ISOs avoiding secondary storage as intermediate cache on KVM. The virtual machine deployment process is enhanced to supported bypassed registered templates and ISOs, delegating the work of downloading them to primary storage to the KVM agent instead of the SSVM agent.

Template and ISO registration:
- When hypervisor is KVM, a checkbox is displayed with 'Direct Download' label.
- API methods registerTemplate and registerISO are both extended with this new parameter directdownload.
- On template or ISO registration, no download job is sent to SSVM agent, CloudStack would only persist an entry on template_store_ref indicating that template or ISO has been marked as 'Direct Download' (bypassing Secondary Storage). These entries are persisted as:
template_id = Template or ISO id on vm_template table
store_id NULL
download_state = BYPASSED
state = Ready
(Note: these entries allow users to deploy virtual machine from registered templates or ISOs)
- An URL validation command is sent to a random KVM host to check if template/ISO location can be reached. Metalink are also supported by this feature. In case of a metalink, it is fetched and URL check is performed on each of its URLs.
- Checksum should be provided as indicated on #2246: {ALGORITHM}CHKSUMHASH
- After template or ISO is registered, it would be displayed in the UI

Virtual machine deployment:
When a 'Direct Download' template is selected for deployment, CloudStack would delegate template downloading to destination storage pool via destination host by a new pluggable download manager.
Download manager would handle template downloading depending on URL protocol. In case of HTTP, request headers can be set by the user via vm_template_details. Those details should be persisted as:
Key: HTTP_HEADER
Value: HEADERNAME:HEADERVALUE

In case of HTTPS, a new API method is added uploadTemplateDirectDownloadCertificate to allow user importing a client certificate into all KVM hosts' keystore before deployment.
After template or ISO is downloaded to primary storage, usual entry would be persisted on template_spool_ref indicating the mapping between template/ISO and storage pool.
2018-01-09 12:22:18 +05:30
Wido den Hollander 9988c269b2 CLOUDSTACK-7958: Add configuration for limit to CIDRs for Admin API calls (#2046)
* Cleanup and Improve NetUtils

This class had many unused methods, inconsistent names and redundant code.

This commit cleans up code, renames a few methods and constants.

The global/account setting 'api.allowed.source.cidr.list' is set
to 0.0.0.0/0,::/0 by default preserve the current behavior and thus
allow API calls for accounts from all IPv4 and IPv6 subnets.

Users can set it to a comma-separated list of IPv4/IPv6 subnets to
restrict API calls for Admin accounts to certain parts of their network(s).

This is to improve Security. Should an attacker steal the Access/Secret key
of an account he/she still needs to be in a subnet from where accounts are
allowed to perform API calls.

This is a good security measure for APIs which are connected to the public internet.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-01-04 10:56:30 +05:30
Frank Maximus c436bc3ef9 CLOUDSTACK-10024: Network Migration (#2374)
Handle PR comments

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Kris Sterckx <kris.sterckx@nuagenetworks.net>
2018-01-03 13:50:19 +05:30
Rohit Yadav 85aee8d18d CLOUDSTACK-10013: SystemVM codebase refactorings and improvements
- Refactors and simplifies systemvm codebase file structures keeping
  the same resultant systemvm.iso packaging
- Password server systemd script and new postinit script that runs
  before sshd starts
- Fixes to keepalived and conntrackd config to make rVRs work again
- New /etc/issue featuring ascii based cloudmonkey logo/message and
  systemvmtemplate version
- SystemVM python codebase linted and tested. Added pylint/pep to
  Travis.
- iptables re-application fixes for non-VR systemvms.
- SystemVM template build fixes.
- Default secondary storage vm service offering boosted to have 2vCPUs
  and RAM equal to console proxy.
- Fixes to several marvin based smoke tests, especially rVR related
  tests. rVR tests to consider 3*advert_int+skew timeout before status
  is checked.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Remi Bergsma 551e11cf3e CLOUDSTACK-10013: Make the generated VR/json files unique (ports #1470)
This ports PR #1470 by @remibergsma.

Make the generated json files unique to prevent concurrency issues:
The json files now have UUIDs to prevent them from getting overwritten
before they've been executed. Prevents config to be pushed to the wrong
router.

2016-02-25 18:32:23,797 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) (logid:) Seq 2-4684025087442026584: Processing:  { Ans: , MgmtId: 90520732674657, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupA
nswer":{"results":["null - success: null","null - success: [INFO] update_config.py :: Processing incoming file => vm_dhcp_entry.json.4ea45061-2efb-4467-8eaa-db3d77fb0a7b\n[INFO] Processing JSON file vm_dhcp_entry.json.4ea4506
1-2efb-4467-8eaa-db3d77fb0a7b\n"],"result":true,"wait":0}}] }

On the router:
2016-02-25 18:32:23,416  merge.py __moveFile:298 Processed file written to /var/cache/cloud/processed/vm_dhcp_entry.json.4ea45061-2efb-4467-8eaa-db3d77fb0a7b.gz

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav a7b233c5dc Merge branch '4.10' 2017-12-23 09:20:24 +05:30
Rohit Yadav bb7d206dab Merge branch '4.9' into 4.10 2017-12-23 09:19:01 +05:30
Frank Maximus c12e1386d3 CLOUDSTACK-10127: For OVS to work use dev/nic id based on mac address for KVM (#2304)
This uses mac address based referencing for nic/dev id for KVM. This fixes openvswitch (ovs) regression.
2017-12-23 09:17:45 +05:30
Sigert Goeminne d49765619d CLOUDSTACK-10024: Network migration support
Co-Authored-By: Frank Maximus frank.maximus@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net

New API’s:

* migrateNetwork
* migrateVpc
2017-12-21 11:25:17 +01:00
Rafael Weingärtner 3c6df7c970 CLOUDSTACK-10196: Remove ejb-api 3.0 dependency (#2348)
Fixed QuotaAlertManagerImplTest, which was injecting mock objects manually.
2017-12-18 23:59:06 +05:30
Wido den Hollander be3a39ea4f CLOUDSTACK-10177: Only pass IPv6 address to Security Group Python script if present (#2355)
Otherwise we send down a 'null' to a ProcessBuilder in Java instead of a String and this
causes a NPE.

We should check first if the Instance has a IPv6 address before sending it there.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-12-11 21:47:50 +05:30
Wido den Hollander fdf2509060 CLOUDSTACK-10160: Fix typo in Libvirt XML definition for Virtio-SCSI (#2341)
* CLOUDSTACK-10160: Fix typo in Libvirt XML definition for Virtio-SCSI

The attribute for the XML element 'controller' should be 'model' and
not 'mode'.

Source: https://libvirt.org/formatdomain.html#elementsControllers

  A scsi controller has an optional attribute model, which is one of
  'auto', 'buslogic', 'ibmvscsi', 'lsilogic', 'lsisas1068', 'lsisas1078',
  'virtio-scsi' or 'vmpvscsi'.

In the current state a regular SCSI device is attached and not a Virtio-SCSI
device.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* CLOUDSTACK-10160: Add UnitTest for LibvirtVMDef.SCSIDef

To make sure the XML output string is correct

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-28 09:42:15 +05:30
Wido den Hollander 632479d8f8 CLOUDSTACK-9853: Add support for Secondary IPv6 Addresses and Subnets (#2028)
This commit adds support for passing IPv6 Addresses and/or Subnets as
Secondary IPs.

This is groundwork for CLOUDSTACK-9853 where IPv6 Subnets have to be
allowed in the Security Groups of Instances to we can add DHCPv6
Prefix Delegation.

Use ; instead of : for separating addresses, otherwise it would cause
problems with IPv6 Addresses.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-22 17:30:33 +05:30
Abhinandan Prateek 4627fb2cd7 CLOUDSTACK-9972: Enhance listVolume API to include physical size and … (#2158)
* CLOUDSTACK-9972: Enhance listVolume API to include physical size and utilization.
Also fixed pool, cluster and pod info

* CLOUDSTACK-9972: Fix volume_view and duplicate API constant

* CLOUDSTACK-9972: Backport Do not allow vms to be deployed on hosts that are in disabled pod

* CLOUDSTACK-9972: Fix localization missing keys

* CLOUDSTACK-9972: Fix sql path
2017-11-05 21:44:43 +05:30
Bitworks Software, Ltd 3381c38cc7 CLOUDSTACK-10073: KVM host RAM overprovisioning (#2266)
Commit enables a new feature for KVM hypervisor which purpose is to increase virtually amount of RAM available beyond the actual limit.
There is a new parameter in agent.properties: host.overcommit.mem.mb which enables adding specified amount of RAM to actually available. It is necessary to utilize KSM and ZSwap features which extend RAM with deduplication and compression.
2017-09-29 11:46:09 +05:30
Wido den Hollander b130e55088 CLOUDSTACK-9397: Add Watchdog timer to KVM Instance (#1707)
The watchdog timer adds functionality where the Hypervisor can detect if an
instance has crashed or stopped functioning.
The watchdog timer adds functionality where the Hypervisor can detect if an
instance has crashed or stopped functioning.

When the Instance has the 'watchdog' daemon running it will send heartbeats
to the /dev/watchdog device.

If these heartbeats are no longer received by the HV it will reset the Instance.

If the Instance never sends the heartbeats the HV does not take action. It only
takes action if it stops sending heartbeats.

This is supported since Libvirt 0.7.3 and can be defined in the XML format as
described in the docs: https://libvirt.org/formatdomain.html#elementsWatchdog

To the 'devices' section this will be added:

In the agent.properties the action to be taken can be defined:

vm.watchdog.action=reset

The same goes for the model. The Intel i6300esb is however the most commonly used.

vm.watchdog.model=i6300esb

When the Instance has the 'watchdog' daemon running it will send heartbeats
to the /dev/watchdog device.

If these heartbeats are no longer received by the HV it will reset the Instance.

If the Instance never sends the heartbeats the HV does not take action. It only
takes action if it stops sending heartbeats.

This is supported since Libvirt 0.7.3 and can be defined in the XML format as
described in the docs: https://libvirt.org/formatdomain.html#elementsWatchdog

To the 'devices' section this will be added:

  <watchdog model='i6300esb' action='reset'/>

In the agent.properties the action to be taken can be defined:

  vm.watchdog.action=reset

The same goes for the model. The Intel i6300esb is however the most commonly used.

  vm.watchdog.model=i6300esb

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-09-28 13:56:15 +05:30
Rohit Yadav 774998d76b Updating pom.xml version numbers for release 4.9.4.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-07 20:21:43 +05:30
Rohit Yadav d2c3408da7 CLOUDSTACK-9782: Improve scheduling of jobs
- Removed three bg thread tasks, uses FSM event-trigger based scheduling
- On successful recovery, kicks VM HA
- Improves overall HA scheduling and task submission, lower DB access

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav d145944be0 Updating pom.xml version numbers for release 4.9.3.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 14:52:37 +02:00
Wido den Hollander 2867080979 CLOUDSTACK-10034: Use libvirt to create new volumes and not rados-java (#2039)
Since libvirt 1.2.2 libvirt will properly create volumes
using RBD format 2.

We can use libvirt to creates the volumes which strips a bit of
code from the CloudStack Agent's responsbility.

RBD format 2 is already used by all volumes created by CloudStack.

This format is the most recent format of RBD and is still actively
being developed.

This removes the support for Ubuntu 12.04 as that does not have the
proper libvirt version available.

Signed-off-by: Wido den Hollander wido@widodh.nl

We can use libvirt to creates the volumes which strips a bit of
code from the CloudStack Agent's responsbility.

RBD format 2 is already used by all volumes created by CloudStack.

This format is the most recent format of RBD and is still actively
being developed.

This removes the support for Ubuntu 12.04 as that does not have the
proper libvirt version available.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-08-06 00:04:21 +02:00
Suresh Kumar Anaparti e61815a255 CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM
Updated hardcoded value with max data volumes limit from hypervisor capabilities.

(cherry picked from commit 93f5b6e8a3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Wei Zhou c03923c7e7 CLOUDSTACK-9113: skip vm with inconsistent state when getVmNetworkStats 2017-07-23 17:15:23 +02:00
Wei Zhou 960cb84083 CLOUDSTACK-7984: Collect network statistics for VMs on shared network (KVM implementation) 2017-07-23 17:15:23 +02:00
Rohit Yadav 445dbe7c53 Merge remote-tracking branch 'origin/4.10' 2017-07-14 15:00:11 +05:30
Rohit Yadav 32e96abea9 Merge remote-tracking branch 'origin/4.9' into 4.10 2017-07-14 14:59:17 +05:30
Wido den Hollander ca415e7436 CLOUDSTACK-9929: Do not gather statistics for CDROM or FLOPPY devices
Libvirt / Qemu (KVM) does not collect statistics about these either.

On some systems it might even yield a 'internal error' from libvirt
when attempting to gather block statistics from such devices.

For example Ubuntu 16.04 (Xenial) has a issue with this.

Skip them when looping through all devices.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-07-14 14:57:11 +05:30
Rajani Karuturi 4bc7c270fa Updating pom.xml version numbers for release 4.11.0.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-12 12:09:38 +05:30
Rajani Karuturi 4f229d19ad Updating pom.xml version numbers for release 4.10.1.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-10 10:36:24 +05:30
Rajani Karuturi 9d2893d44a Updating pom.xml version numbers for release 4.10.0.0
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-03 10:06:43 +05:30
Rohit Yadav ed376fcad6 Merge remote-tracking branch 'origin/4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-07 11:21:27 +05:30
Rohit Yadav e197652a28 CLOUDSTACK-9860: Fix stackoverflow issue
Fixes issue caused to a PR-refactoring from #2108, reported by
@borisstoyanov

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-06 16:14:03 +05:30
Rohit Yadav 8323a175f1 CLOUDSTACK-9860: Power off VMs when stopVM is called with forced=true
The 'force' option provided with the stopVirtualMachine API command is
often assumed to be a hard shutdown sent to the hypervisor, when in fact
it is for CloudStacks' internal use. CloudStack should be able to send
the 'hard' power-off request to the hosts.

When forced parameter on the stopVM API is true, power off (hard shutdown)
a VM. This uses initial changes from #1635 to pass the forced parameter
to hypervisor plugin via the StopCommand, and fixes force stop (poweroff)
handling for KVM, VMware and XenServer.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-25 17:25:22 +05:30
Rajani Karuturi e25a444a0c Merge pull request #2121 from bvbharatk/CLOUDSTACK-9641
CLOUDSTACK-9641 In KVM SSVM and CPVM may use the old cmdline data, if…
2017-05-23 10:07:55 +05:30
Bharat Kumar 3c80f00550 CLOUDSTACK-9641 In KVM SSVM and CPVM may use the old cmdline data, if we fail to fetch the new cmdline in the first pass. 2017-05-19 16:50:19 +05:30
Jayapal d04a3e842c CLOUDSTACK-9317: Update review comments and rule state column 2017-05-17 11:08:13 +05:30
Jayapal c20e0ef88f CLOUDSTACK-9317: Fixed disable static nat on leaving ips on interface 2017-05-17 11:03:50 +05:30
Rajani Karuturi 7434d91614 Merge pull request #1873 from Accelerite/dhcpOffloadFix
CLOUDSTACK-9709: Updated the vm ip fetch task to use the correct the …
2017-05-17 10:43:51 +05:30
Rajani Karuturi a4dd6bdeeb Merge pull request #1955 from myENA/virtio-scsi
CLOUDSTACK-8239 Add VirtIO SCSI support for KVM hosts
2017-04-20 15:36:34 +05:30
Rajani Karuturi ec2d4dd422 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number>
2017-03-23 15:19:31 +05:30
Wei Zhou bf93b6313e CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number> 2017-03-20 07:32:31 +01:00
Rajani Karuturi ad7ed7a178 Merge pull request #847 from kishankavala/CLOUDSTACK-8880
Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm.  free memory = total memory - (all vm memory)With memory over-provisioning set to 1, when mgmt server starts VMs in parallel on one host, then the memory allocated on that kvm can be larger than the actual physcial memory of the kvm host.

Fixed by checking free memory on host before starting Vm.
Added test case to check memory usage on Host.
Verified Vm deploy on Host with enough capacity and also without capacity

* pr/847:
  Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm.  free memory = total memory - (all vm memory)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:20:58 +05:30
Rajani Karuturi 3f0fbf251c Merge pull request #1953 from Accelerite/CLOUDSTACK-9794
CLOUDSTACK-9794: Unable to attach more than 14 devices to a VMUpdated hardcoded value with max data volumes limit from hypervisor capabilities.

* pr/1953:
  CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:19:04 +05:30
Rajani Karuturi 56e851ca46 Merge release branch 4.9 to master
* 4.9:
  moved logrotate from cron.daily to cron.hourly for vpcrouter in cloud-early-config
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent
2017-03-13 22:09:27 +05:30
Rajani Karuturi 7b719c71fc Merge pull request #1856 from ustcweizhou/set-kvm-host-params
[4.9] CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agentThe router.aggregation.command.each.timeout in global configuration is only applied on new created KVM host.
For existing KVM host, changing the value will not be effective.
We need to propagate the configuration to existing host when cloudstack-agent is connected.

* pr/1856:
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:08:47 +05:30
Suresh Kumar Anaparti 93f5b6e8a3 CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM
Updated hardcoded value with max data volumes limit from hypervisor capabilities.
2017-03-13 16:14:12 +05:30
Nathan Johnson 5c476492b1 CLOUDSTACK-8239 - Adding support for virtio-scsi on KVM hosts
This adds support for virtio-scsi on KVM hosts, either
for guests that are associated with a new os_type of 'Other PV Virtio-SCSI (64-bit)',
or when a VM or template is regstered with a detail parameter rootDiskController=scsi.

Update cloudstack add template dialog to allow for selecting rootDiskController with KVM

Update cloudstack kvm virtio-scsi to enable discard=unmap
2017-03-12 10:54:43 -05:00
Jayapal e3ae08b3ee CLOUDSTACK-9709: Updated the vm ip fetch task to use the correct the thread 2017-03-07 09:50:18 +05:30
Kishan Kavala 9a021904af Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm. free memory = total memory - (all vm memory) 2017-02-20 11:32:48 +05:30
Rajani Karuturi 7233ac37cd Merge pull request #977 from ustcweizhou/vm-snapshot
[4.10] CLOUDSTACK-8746: VM Snapshotting implementation for KVM

* pr/977:
  Fixes for testing VM Snapshots on KVM. Related to PR 977
  CLOUDSTACK-8746: vm snapshot implementation for KVM

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-01-31 05:58:56 +05:30
Wido den Hollander 84e496b4f9
CLOUDSTACK-676: IPv6 Basic Security Grouping for KVM
This commit implements basic Security Grouping for KVM in
Basic Networking.

It does not implement full Security Grouping yet, but it does:
- Prevent IP-Address source spoofing
- Allow DHCPv6 clients, but disallow DHCPv6 servers
- Disallow Instances to send out Router Advertisements

The Security Grouping allows ICMPv6 packets as described by RFC4890
as they are essential for IPv6 connectivity.

Following RFC4890 it allows:
- Router Solicitations
- Router Advertisements (incoming only)
- Neighbor Advertisements
- Neighbor Solicitations
- Packet Too Big
- Time Exceeded
- Destination Unreachable
- Parameter Problem
- Echo Request

ICMPv6 is a essential part of IPv6, without it connectivity will break or be very
unreliable.

For now it allows any UDP and TCP packet to be send in to the Instance which
effectively opens up the firewall completely.

Future commits will implement Security Grouping further which allows controlling UDP and TCP
ports for IPv6 like can be done with IPv4.

Regardless of the egress filtering (which can't be done yet) it will always allow outbound DNS
to port 53 over UDP or TCP.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-01-26 15:36:08 +01:00
Wei Zhou a2428508e2 CLOUDSTACK-8746: vm snapshot implementation for KVM
(1) add support to create/delete/revert vm snapshots on running vms with QCOW2 format
(2) add new API to create volume snapshot from vm snapshot
(3) delete metadata of vm snapshots before stopping/migrating and recover vm snapshots after starting/migrating
(4) enable deleting of VM snapshot on stopped vm or vm snapshot is not listed in qcow2 image.
(5) enable smoke tests for vmsnaphsots on KVM
2017-01-24 21:47:30 +01:00
Rohit Yadav 8b6e96bca9 Updating pom.xml version numbers for release 4.9.3.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-06 10:40:15 +05:30
Rohit Yadav dfc39c1f08 Updating pom.xml version numbers for release 4.9.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-03 12:28:47 +05:30
Wei Zhou 714221234d CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent 2016-12-22 12:00:10 +01:00
Rohit Yadav 0dce1c50c1 CLOUDSTACK-9456: Update Spring version in maven poms
- Bump spring-framework version to 4.x and Jetty to version that runs with JDK8
- Bump servet dependency version
- Migrate spring xmls to version 4, fixes schema locations that are 3.0
  dependent in various xmls.
- Fix failing tests due to spring upgrade
  (Thanks @marcaurele Marc-Aurèle Brothier for fixing them)
    * Fix test DeploymentPlanningManagerImplTest
    * Fix GloboDNS test

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-16 21:21:20 +05:30
Rohit Yadav 5e19e64f2f Updating pom.xml version numbers for release 4.9.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-16 20:48:16 +05:30
Rohit Yadav af2679959b Updating pom.xml version numbers for release 4.9.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-10 08:38:03 +05:30
Wido den Hollander 2a5f37c1b1
CLOUDSTACK-8715: Add channel to Instances for Qemu Guest Agent
This commit adds a additional VirtIO channel with the name
'org.qemu.guest_agent.0' to all Instances.

With the Qemu Guest Agent the Hypervisor gains more control over the Instance if
these tools are present inside the Instance, for example:

* Power control
* Flushing filesystems
* Fetching Network information

In the future this should allow safer snapshots on KVM since we can instruct the
Instance to flush the filesystems prior to snapshotting the disk.

More information: http://wiki.qemu.org/Features/QAPI/GuestAgent

Keep in mind that on Ubuntu AppArmor still needs to be disabled since the default
AppArmor profile doesn't allow libvirt to write into /var/lib/libvirt/qemu

This commit does not add any communication methods through API-calls, it merely
adds the channel to the Instances and installs the Guest Agent in the SSVMs.

With the addition of the Qemu Guest Agent channel a second channel appears in /dev
on a SSVM as a VirtIO port.

The order in which the ports are defined in the XML matters for the naming inside
the SSVM VM and by not relying on /dev/vportXX but looking for a static name the
SSVM still boots properly if the order in the XML definition is changed.

A SSVM with both ports attached will have something like this:

  root@v-215-VM:~# ls -l /dev/virtio-ports
  total 0
  lrwxrwxrwx 1 root root 11 May 13 21:41 org.qemu.guest_agent.0 -> ../vport0p2
  lrwxrwxrwx 1 root root 11 May 13 21:41 v-215-VM.vport -> ../vport0p1
  root@v-215-VM:~# ls -l /dev/vport*
  crw------- 1 root root 251, 1 May 13 21:41 /dev/vport0p1
  crw------- 1 root root 251, 2 May 13 21:41 /dev/vport0p2
  root@v-215-VM:~#

In this case the SSVM port points to /dev/vport0p1, but if the order in the XML
is different it might point to /dev/vport0p2

By looking for a portname with a pre-defined pattern in /dev/virtio-ports we
do not rely on the order in the XML definition.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2016-11-23 16:01:08 +01:00
Rohit Yadav 0642a6982f
Merge branch '4.9' 2016-11-23 14:22:15 +05:30
Rohit Yadav 55b918076f
Merge branch '4.8' into 4.9 2016-11-23 13:50:15 +05:30
Rohit Yadav ff616e700b Merge pull request #1745 from shapeblue/CLOUDSTACK-9503
CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.

* pr/1745:
  CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-23 13:41:52 +05:30
Rohit Yadav b59db0dc06 Merge pull request #1705 from nemo9cby/CLOUDSTACK-9465
Made the changes to improve logging.CLOUSTACK-9465 Several log refactoring/improvement suggestions.

There are two scenarios of logging which needs refactoring/improvement:

Method invocation replaced by variable

This means that in the logging code, the method invocation is pre-defined as a variable. for simplicity,          the method invocation should be replaced by the variable.

Delete variable which must be null

The variable in the logging code is null, there is no need to put the variable there.

* pr/1705:
  Made the changes to improve logging.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-03 16:48:21 +05:30
Abhinandan Prateek 83b5a8b2b2 CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration. 2016-11-01 16:14:23 +05:30
Nemo cd9c7737d1 Made the changes to improve logging. 2016-10-11 12:58:02 -04:00
Wido den Hollander 0beb41b6e7 CLOUDSTACK-9395: Add Virtio RNG device to Instances when configured
By adding a Random Number Generator device to Instances we can prevent
entropy starvation inside guest.

The default source is /dev/random on the host, but this can be configured
to another source when present, for example a hardware RNG.

When enabled it will add the following to the Instance's XML definition:

  <rng model='virtio'>
    <rate period='1000' bytes='2048' />
    <backend model='random'>/dev/random</backend>
  </rng>

If the Instance has the proper support, which most modern distributions have,
it will have a /dev/hwrng device which it can use for gathering entropy.

More information: https://libvirt.org/formatdomain.html#elementsRng
2016-10-04 12:44:55 +02:00
John Burwell 8d11511b1f Adds support for four position versions and optional db upgrades
Often, patch and security releases do not require schema migrations or
data migrations.  However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break.  With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path.  This approach allows the upgrade to proceed and
for the database to properly reflect the installed version.  This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.

Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes.  The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers.  This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers.   It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.

  * Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
  a version that was not explicitly specified.  It determines the
  releases the first release before it with database migrations and uses
  that list as the basis for the list for version being calculated.  A
  noop upgrade is then added to the list which causes no schema changes
  or data migrations, but will update the database to the version.
  * Adds unit tests for the upgrade path calculation logic in
  DatabaseUpgradeChecker
  * Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
  versions of this patch
  * Introduces the CloudStackVersion value object which parses and
  compares three (3) and four (4) position version numbers.  This class
  is intended to replace com.cloud.maint.Version.
  * Adds the junit-dataprovider dependency -- allowing test data to be
  concisely generated separately from the execution of a test case.
  Used extensively in the CloudStackVersionTest.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 13:32:32 +05:30
Nathan Johnson 46df85c5bf CLOUDSTACK-9461
This converts the rbd raw format on disk to qcow2 for compression.
2016-08-26 09:52:24 -05:00
Rohit Yadav 3a81a4498f Merge branch '4.9' 2016-08-24 12:15:24 +05:30
Rohit Yadav fa3fe7bb05 Merge pull request #1634 from shapeblue/patchviasocket-49-py26fix
[blocker] CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpmsThe patchviasocket script was rewritten in Python from PR #1533 and made
assumptions that Python 2.7 would be available. In case of CentOS, python 2.7
may not be available or installed. This change ensures that python-argparse
is installed which is used by this script.

/cc @wido @sverrirab @karuturi @jburwell

@blueorangutan package

* pr/1634:
  CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpms

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-24 12:14:02 +05:30
Rohit Yadav 9555492b4d Merge branch '4.9' 2016-08-23 14:16:53 +05:30
Rohit Yadav f13c224da1 Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:39 +05:30
Will Stevens 740bd45be6 Updating pom.xml version numbers for release 4.8.2-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-15 10:31:09 -04:00
Rohit Yadav 7b8ba24c64 CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpms
The patchviasocket script was rewritten in Python from PR #1533 and made
assumptions that Python 2.7 would be available. In case of CentOS, python 2.7
may not be available or installed. This change ensures that python-argparse
is installed which is used by this script.

Expose cmd error in the logs when patch command fails.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-10 16:25:17 +05:30
Will Stevens a63db21d16 Updating pom.xml version numbers for release 4.8.1
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-08 10:06:21 -04:00
Will Stevens 62aa3b2bfa Updating pom.xml version numbers for release 4.10.0-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-29 10:11:34 -04:00
Will Stevens 227ff3884d Updating pom.xml version numbers for release 4.9.0
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-25 16:56:04 -04:00
Aaron Hurt c8fce3ff31 improve logging readability 2016-07-11 12:05:06 -05:00
Aaron Hurt 44491448e3 Cleanup rbd contexts and improve exception logging
We noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up.  This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.

In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
2016-07-08 23:13:33 -05:00
Will Stevens b03a629c6a Merge pull request #1533 from greenqloud/pr-patchviasocket-convert-to-python
Convert patchviasocket to python (removes perl dependency for KVM agent)As requested here: https://github.com/apache/cloudstack/pull/1495

No scripts are using perl so that install requirement can be removed.
The new scripts are using standard python packages only.
Includes extensive unit test.
Note: perl-modules requirement is missing (fixed in mentioned PR) so do not merge that onto master.

* pr/1533:
  Revert "Add perl-modules as install dependency for cloudstack-agent"
  patchviasocket improve error handling
  Convert patchviasocket to python (removes perl dependency for KVM agent)

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:57:08 -04:00
Will Stevens 678b28f273 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:54:23 -04:00
Will Stevens b80696cbc0 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:53:42 -04:00
Sverrir A. Berg 0acd3c12a2 Convert patchviasocket to python (removes perl dependency for KVM agent)
As requested here: https://github.com/apache/cloudstack/pull/1495

No scripts are using perl so that install requirement can be removed.
The new scripts are using standard python packages only.
Includes extensive unit test.
2016-05-20 15:42:34 +00:00
Will Stevens 82b702dc9a Merge pull request #1403 from mike-tutkowski/xs-snapshots
Taking fast and efficient volume snapshots with XenServer (and your storage provider)A XenServer storage repository (SR) and virtual disk image (VDI) each have UUIDs that are immutable.

This poses a problem for SAN snapshots, if you intend on mounting the underlying snapshot SR alongside the source SR (duplicate UUIDs).

VMware has a solution for this called re-signaturing (so, in other words, the snapshot UUIDs can be changed).

This PR only deals with the CloudStack side of things, but it works in concert with a new XenServer storage manager created by CloudOps (this storage manager enables re-signaturing of XenServer SR and VDI UUIDs).

I have written Marvin integration tests to go along with this, but cannot yet check those into the CloudStack repo as they rely on SolidFire hardware.

If anyone would like to see these integration tests, please let me know.

JIRA ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-9281

Here's a video I made that shows this feature in action:

https://www.youtube.com/watch?v=YQ3pBeL-WaA&list=PLqOXKM0Bt13DFnQnwUx8ZtJzoyDV0Uuye&index=13

* pr/1403:
  Faster logic to see if a cluster supports resigning
  Support for backend snapshots with XenServer

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-20 08:33:07 -04:00
Wei Zhou 976b3b7df4 CLOUDSTACK-6928: fix issue disk I/O throttling not applied 2016-05-19 11:26:43 +02:00
Will Stevens 7be8585787 Merge pull request #1520 from NuxRo/kvm-numa-cpu-reporting
CPU socket count reporting correctionCPU socket count reporting correction
From https://github.com/MissionCriticalCloud/cosmic-plugin-hypervisor-kvm/pull/16

* pr/1520:
  Remove empty spaces causing the build to fail
  CPU socket count reporting correction

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-13 17:02:10 -04:00
Mike Tutkowski 2bd035d199 Support for backend snapshots with XenServer 2016-05-13 01:02:04 -06:00
Will Stevens 8c3722d953 Merge pull request #1444 from rafaelweingartner/workAroundPR780
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMThis PR introduces the changes proposed in PR #780 with some work to make the code null safe.

During this PR, I have also removed some unused code.

* pr/1444:
  Removed unnecessary check when creating the “userVmResponse” object.
  Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
  CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:49:14 -04:00
Will Stevens fa3bce5a83 Merge pull request #1496 from shapeblue/kvm-ha
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage- KVM-HA- Fix CheckOnHost for Local storage
 - Also skip HA on VMs that are using local storage

* pr/1496:
  CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage     - Also skip HA on VMs that are using local storage

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:49:04 -04:00
Will Stevens c9319e21fb Merge pull request #1230 from SafeSwissCloud/CLOUDSTACK-8302
CLOUDSTACK-8302: Removing snapshots on RBDSnapshot removing implemented if primary datastore is RBD
https://issues.apache.org/jira/browse/CLOUDSTACK-8302

* pr/1230:
  CLOUDSTACK-8302 - Cleanup snapshot on KVM with RBD Snapshot removing implemented on RBD. 1. On management side: when created new shanpshot we checking if our primary storage is RBD, then do not remove record from cloud.snapshot_store_ref with link to Ceph image via 'install_path' field. 2. On management side: when removing snapshot, also send command to agent 'DeleteCommand'. 3. On agent side: method implemented 'public Answer deleteSnapshot(final DeleteCommand cmd)'

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-04 10:26:01 -04:00
weingartner 866ac7b6e1 Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
It was worked around some possible runtime exceptions introduced by the
changes that were added by the PR 780. Basically, the points in which a
null pointer exception could happen, we added safety checks to avoid
them. It was create a specific method do that, all together test cases
were created for this newly method that was added.
2016-05-03 14:05:18 -03:00
Maneesha.P 7ad3c5e834 CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware. 2016-04-29 14:06:49 -03:00
Abhinandan Prateek 3b89cbe733 CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage
- Also skip HA on VMs that are using local storage
2016-04-29 12:38:33 +05:30
Will Stevens ad61d49d87 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe way
2016-04-28 15:57:43 -04:00
Will Stevens 43ab44dd58 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe way
2016-04-28 15:56:48 -04:00
Will Stevens c03e359824 Merge pull request #1348 from DaanHoogland/CLOUDSTACK-9142
CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe wayThe problem arises when the origin hypervisor has an ip addres that ends with 1, like '10.10.10.1' and the qemu VM description is containing an address that has that as part of its address, '10.10.10.100' for instance.
now migrating to '10.10.10.10' will change both addresses in the xml description file for qemu. It is fixed and unit tests are added. I am not sure yet how to integration test this. Regression will probably work so creating a PR now.

* pr/1348:
  CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe way

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-28 15:55:09 -04:00
NuxRo 108b3517af Remove empty spaces causing the build to fail 2016-04-27 19:15:02 +01:00
NuxRo 42de1f81ed CPU socket count reporting correction
CPU socket count reporting correction
From https://github.com/MissionCriticalCloud/cosmic-plugin-hypervisor-kvm/pull/16
2016-04-27 10:49:53 +01:00
Dmytro Shevchenko 10ae2aff28 CLOUDSTACK-8302 - Cleanup snapshot on KVM with RBD
Snapshot removing implemented on RBD.
1. On management side: when created new shanpshot we checking if our primary storage is RBD,
then do not remove record from cloud.snapshot_store_ref with link to Ceph
image via 'install_path' field.
2. On management side: when removing snapshot, also send command to agent 'DeleteCommand'.
3. On agent side: method implemented 'public Answer deleteSnapshot(final DeleteCommand cmd)'
2016-04-25 22:14:48 +03:00
Will Stevens f530a4c63a Merge pull request #1200 from koushik-das/CLOUDSTACK-9130
CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag

RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.

* pr/1200:
  CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-21 16:35:27 -04:00
Koushik Das d1def0a730 Merge pull request #1287 from DaanHoogland/securityrules-cleanup
SecurityGroupRulesCmd code cleanupWrote a test and cleaned some duplicate code with the objective to evaluate the jenkins pull request process at builds.a.o
worthwhile to keep, IMHO.

* pr/1287:
  SecurityGroupRulesCmd code cleanup review comments handled
  deal with PMD warnings
  code cleanup
  security rules test
  remove autogenerated pydev files

Signed-off-by: Koushik Das <koushik@apache.org>
2016-04-11 21:49:57 +05:30
Will Stevens c48b6c33c0 Revert "Merge pull request #780 from maneesha-p/pull-19" due code quality issues and lack of CI results.
This reverts commit dc0ba6bd1a, reversing
changes made to 63f58dd578.
2016-03-21 17:38:04 -04:00
Maneesha.P 732a85295d CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware. 2016-03-16 16:13:10 +05:30
Remi Bergsma 43ab98d823 Updating pom.xml version numbers for release 4.9.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 15:12:20 +01:00
Remi Bergsma 32fcc47117 Updating pom.xml version numbers for release 4.8.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 09:39:00 +01:00
Remi Bergsma a4ac9d3fff Updating pom.xml version numbers for release 4.7.2-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 09:33:56 +01:00
Daan Hoogland 8a6afa6d9f CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe way 2016-01-21 21:32:35 +01:00
Remi Bergsma 62f218b7bd Updating pom.xml version numbers for release 4.8.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-20 23:43:35 +01:00
Remi Bergsma 26936cdb2b Merge pull request #1242 from wido/fix-kvm-libvirt-test
test: Fix Libvirt test so that it works on WindowsThis test failed on Windows, using the File.separator it should run fine on Windows.

* pr/1242:
  test: Fix Libvirt test so that it works on Windows

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-19 17:54:44 +01:00
Remi Bergsma fb658f575d Merge pull request #1226 from borisroman/CLOUDSTACK-9148
Removed .pydevproject from plugin kvm hypervisor.Ping @wido @wilderrodrigues @remibergsma @miguelaferreira

It's there for no apparent reason...

Running integration tests now.

```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6:12.189s
[INFO] Finished at: Sat Dec 12 02:13:01 CET 2015
[INFO] Final Memory: 102M/808M
[INFO] ------------------------------------------------------------------------
```

* pr/1226:
  Removed .pydevproject from plugin kvm hypervisor.

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-17 13:04:58 +01:00
Daan Hoogland d39182f9d4 deal with PMD warnings 2015-12-26 09:02:30 +01:00
Daan Hoogland ffce2d601e remove autogenerated pydev files 2015-12-26 09:02:27 +01:00
Remi Bergsma 8f5a2920e8 Updating pom.xml version numbers for release 4.8.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-21 22:09:31 +01:00
Remi Bergsma 5b112408c7 Updating pom.xml version numbers for release 4.7.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-21 19:55:04 +01:00
Wido den Hollander 268b9b31e7 test: Fix Libvirt test so that it works on Windows 2015-12-14 11:33:47 +01:00
Remi Bergsma 2f26a859a9 Updating pom.xml version numbers for release 4.7.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-13 21:09:53 +01:00
Boris Schrijver 223e3d5466 Removed .pydevproject from plugin kvm hypervisor. 2015-12-13 14:53:08 +01:00
Remi Bergsma bdaa60fd6c Merge release branch 4.6 to master
* 4.6:
  Show actual diff in commits after merge
  CLOUDSTACK-9113: skip vm with inconsistent state when getVmStats/getVmDiskStats
2015-12-12 14:03:43 +01:00
Koushik Das 5b3ffeb6cd CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag
RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.
2015-12-09 15:19:10 +05:30
Wei Zhou 24edf1c1e4 CLOUDSTACK-9113: skip vm with inconsistent state when getVmStats/getVmDiskStats 2015-12-07 12:15:24 +01:00
Nick Livens 0957268e92 CLOUDSTACK-9095 : Hypervisor changes to support UserData for Nuage VSP 2015-12-07 10:26:47 +01:00
Daan Hoogland 638f1cf091 Merge pull request #1049 from DaanHoogland/CLOUDSTACK-9047
CLOUDSTACK-9047 rename enumsmake enums adhere to best practice naming conventions

* pr/1049:
  CLOUDSTACK-9046 rename enums to adhere to naming conventions
  CLOUDSTACK-9046 renamed enums in kvm plugin
  CLOUDSTACK-9047 use 'State's only with context   there are more types called 'State'   (or to be called so but now 'state')   So remove imports and prepend their enclosing class/context to them.

Signed-off-by: Daan Hoogland <daan@onecht.net>
2015-12-06 20:17:04 +01:00
Josh Harshman b6c900fdde CLOUDSTACK-9004: Add features to HyperVEnlightenmentFeatureDef
Add function to set vapic, spinlock and retries
Add function to get retry value
Modify toString to output appropriate XML for spinlock value if set

CLOUDSTACK-9004: Add features to HyperVEnlightenmentFeatureDef

Refactored set methods to get rid of code duplication.
Modified unit tests accordingly
2015-12-03 16:49:00 -08:00
Remi Bergsma 4ecfc29267 Merge release branch 4.6 to master
* 4.6:
  Use version for RC branch name instead of branch
  make sure all files are updates with new version
  Update L10N resource files with 4.6 strings from Transifex (20151129)
  Fix secondary storage not working with swift
  CLOUDSTACK-9083: Add disk serial to kvm virt xml
2015-11-30 22:30:52 +01:00
Rohit Yadav 12c395b560 CLOUDSTACK-9083: Add disk serial to kvm virt xml
Adds disk serial ids based on volume uuids to the virt xml. This may be useful
for appliances/software that needs some serial ids on the VM disks. This does not
impact existing/running VMs, the vm virt xmls will be updates for running VMs
the next time they are stopped/started.

For testing, disk serial (of debian based systemvm) in the virt xml matched that
in /sys/devices/pci0000:00:0000:00:07.0/virtio4/block/vda/serial.

We currently don't support scsi-blcok devices for which serial is not supported,
for this we've added a DeviceType (LUN) which may be used in future and a check
to not add the serial to the xml if disk type is LUN.
Refer: https://libvirt.org/formatdomain.html#elementsDisks

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-11-25 14:58:42 +05:30
Daan Hoogland 08eb96776a CLOUDSTACK-9037 patterns can be more elaborate then prefixes. 2015-11-23 11:48:44 +01:00
cirstofolini 1a64c247ad Removed unnecessary @Local annotations and their respective imports from the ComponentLifecycleBase class and its subclasses. 2015-11-21 18:31:11 -02:00
Remi Bergsma 7665bdc815 Merge pull request #1083 from borisroman/CLOUDSTACK-9062
CLOUDSTACK-9062: Improve S3 implementation.The S3 implementation is far from finished, this commit focuses on the bases.

 - Upgrade AWS SDK to latest version.
 - Rewrite S3 Template downloader.
 - Rewrite S3Utils utility class.
 - Improve addImageStoreS3 API command.
 - Split various classes for convenience.
 - Various minor improvements and code optimizations.

A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.

Please review thoroughly, both code inspection and (automated) integration tests. Currently no integration tests are available specifically for S3. Therefore the implementation is needed to be tested manually, for now...

What I tested:
 - Greenfield install -> will download latest systemvm template automatically to S3.
 - Upload a template/iso
 - Download a template/iso
 - Restart of management server -> list available templates -> doesn't download them again if available.

* pr/1083:
  CLOUDSTACK-9062: Improve S3 implementation.

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-20 22:27:17 +01:00
Boris Schrijver 5c0366c99e CLOUDSTACK-9062: Improve S3 implementation.
The S3 implementation is far from finished, this commit focusses on the bases.

 - Upgrade AWS SDK to latest version.
 - Rewrite S3 Template downloader.
 - Rewrite S3Utils utility class.
 - Improve addImageStoreS3 API command.
 - Split various classes for convenience.
 - Various minor improvements and code optimalisations.

A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
2015-11-19 15:29:10 +01:00
Wilder Rodrigues 942aa17d0c CLOUDSTACK-9067 - Replaces hardcoded paths with the VRScripts constants. 2015-11-18 13:12:45 +01:00
Daan Hoogland 962d4b4afe CLOUDSTACK-9046 renamed enums in kvm plugin 2015-11-16 13:41:14 +00:00
Remi Bergsma 1f53f2a93e Updating pom.xml version numbers for release 4.7.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-15 18:54:13 +01:00
Remi Bergsma b38c3bed0c Updating pom.xml version numbers for release 4.6.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-13 21:27:57 +01:00
Remi Bergsma e31ade03c6 Updating pom.xml version numbers for release 4.6.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-10 15:45:34 +01:00
Remi Bergsma 0b4cc5de87 Merge pull request #986 from wido/kvm-libvirt-unittests
Add Unit Tests for Libvirt/KVM storage codeThese classes were not covered by Unit Tests and this commit
adds some tests for their basic functionality.

* pr/986:
  Add Unit Tests for Libvirt/KVM storage code

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-02 17:53:31 +01:00
Wido den Hollander 602231132d kvm: Add UnitTests for LibvirtUtilitiesHelper
These were lacking, but this helper is used in various places
inside the KVM code.

Some simple tests to verify the helper is doing what we expect it
to do.
2015-10-30 12:34:00 +01:00
Wido den Hollander 7568f2123c Add Unit Tests for Libvirt/KVM storage code
These classes were not covered by Unit Tests and this commit
adds some tests for their basic functionality.
2015-10-27 13:21:28 +01:00
Daan Hoogland 5a134a6fe6 unit test for interface patterns in libvirt compute resource 2015-10-24 00:57:01 +02:00
Daan Hoogland f1ea27ed37 Merge branch 'pr/966' into interfacePatternCheck 2015-10-23 17:10:03 +02:00
David Mabry e1a401c023 Added support for KVM teamd devices to LibvirtComputingResource.java. This will allow users to utilze teamd nic teaming devices named team*. 2015-10-22 10:19:50 -05:00
Wido den Hollander d413b2c375 CLOUDSTACK-8879: Depend in rados-java 0.2.0
This should fix the CloudStack Agent from crashing when it has to
handle more then 16 RBD snapshots on one Volume.
2015-09-25 11:01:03 +02:00
Wido den Hollander 4f409a7c17 CLOUDSTACK-8645: Improve logging of RBD functionality in KVM
A simple commit which changes a couple of log lines.
2015-09-15 10:10:04 +02:00
Satoru Nakaya e8c5ed4e3d CLOUDSTACK-8838: Allow ensX enoX enpX enxX format for nics in CentOS 7 2015-09-13 22:13:30 +09:00
Kevin McCormick 99d20234d4 CLOUDSTACK-8678: Reserve RAM for KVM host OS
Use host.reserved.ram.mb agent property to modify total system RAM
before reporting to management server.

Remove dom0ram variable and its min/max calculation. Just reserve a
default of 1GB, unless overridden by host.reserved.mem.mb property.
2015-09-03 10:07:22 -07:00
Rohit Yadav dd9ba48efa Merge pull request #753 from shapeblue/master-CLOUDSTACK-8762
CLOUDSTACK-8762: Check to confirm disk activity before starting a VMImplements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* pr/753:
  CLOUDSTACK-8762: Check to confirm disk activity before starting a VM

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-09-02 14:55:54 +05:30
Wei Zhou c0a0aec0f9 Merge pull request #732 from ustcweizhou/revert-volume-snapshot-master
Guys, can you review it? things need to be discussed:
(1) this supports KVM/QCOW2 only. Anyone want to implement for other Hypervisor/format ?
(2) The original data volume (on primary storage) will be removed.
(3) The script uses the default timeout in libvirtComputingResource. Do we need to add one in global configuration (like copy.volume.wait or backup.snapshot.wait, create.volume.from.snapshot.wait)
(4) In scripts/storage/qcow2/managesnapshot.sh, I use "qemu-img convert -f qcow2 -O qcow2" to copy the snapshot from secondary to primary (hence there is no base image file), instead of "cp -f", this is because convert is faster than cp in my testing.

* pr/732:
  CLOUDSTACK-5863: revert volume snapshot for KVM/QCOW2

Signed-off-by: Wei Zhou <w.zhou@tech.leaseweb.com>
2015-09-01 16:18:40 +02:00
Mike Tutkowski c5a0d5e01c Support live migration on older version of Libvirt 2015-08-31 22:06:51 -06:00
Rajani Karuturi 8bc0294014 Revert "Merge pull request #714 from rafaelweingartner/master-lrg-cs-hackday-003"
This reverts commit cd7218e241, reversing
changes made to f5a7395cc2.

Reason for Revert:

noredist build failed with the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloud-plugin-hypervisor-vmware: Compilation failure
[ERROR] /home/jenkins/acs/workspace/build-master-noredist/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[484,12] error: non-static variable logger cannot be referenced from a static context
[ERROR] -> [Help 1]

even the normal build is broken as reported by @koushik-das on dev list
http://markmail.org/message/nngimssuzkj5gpbz
2015-08-31 11:27:57 +05:30
Rafael Weingartner 3818257a68 Solved jira ticket: CLOUDSTACK-8750 2015-08-28 22:35:08 -03:00
Rohit Yadav 52a98fa6cf CLOUDSTACK-8762: Check to confirm disk activity before starting a VM
Implements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-28 16:05:30 +05:30
Koushik Das e401d39d0b Merge pull request #741 from koushik-das/code_cleanup
messages: Removed unused agent command AttachVolumeCommand and corresponding answer AttachVolumeAnswer

These types are not used anywhere in the orchestration layer.

Signed-off-by: Koushik Das <koushik@apache.org>
2015-08-27 17:13:25 +05:30
Rohit Yadav e2a0d18a84 Merge pull request #696 from iwebhosting/rbd-live-resize
Default to notify only script to handle non-CLVM/QCOW cases.This relates to [CLOUDSTACK-8732](https://issues.apache.org/jira/browse/CLOUDSTACK-8732)

Before this commit the call to `getResizeScriptType` would throw an exception (earlier versions returned `null`, which was fine) - this caused the RBD case to fail. By changing the default to notify only we fix the case for any non-CLVM and non-QCOW cases, too.

This is RBD for now, but this should extend to new storage types supported by Libvirt natively in future.

This is my first attempted contribution: I can see a case for adding RBD logic to the actual getResizeScriptType call, too, but I felt that putting it `LibvirtResizeVolumeCommandWrapper.java` kept the special-casing of RBD (and comments about that) in one place.

### Caveat:

With Libvirt 1.2.2 this actually doesn't do the right thing - but it does do what the documentation *says* should be the right thing, so I'm going to test if this is a Libvirt bug which is fixed in a later version.

(To make it work I need to execute something like:

    virsh blockresize --path vda --size 100G i-7-44-VM

where vda is the path as far as the *guest* is concerned, and not an `rbd/` path - which *should* work, but doesn't.)

* pr/696:
  Default to notify only script to handle non-CLVM/QCOW cases.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-26 16:00:01 +05:30
Rohit Yadav 5d232ea3d9 Merge pull request #735 from shapeblue/kvm-linkbr-checks-master
CLOUDSTACK-8749: Add checks to prevent malformed/unexpected inputBased on @jburwell's comment on PR #718

* pr/735:
  CLOUDSTACK-8749: Add checks to prevent malformed/unexpected input

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-26 14:18:40 +05:30
Rohit Yadav 7e455fa2b7 CLOUDSTACK-8749: Add checks to prevent malformed/unexpected input
Based on @jburwell's comment on PR #718

This closes #735

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-26 13:34:21 +05:30
Koushik Das e1db6efc74 Removed unused agent command AttachVolumeCommand and corresponding answer AttachVolumeAnswer 2015-08-26 09:50:23 +05:30
Remi Bergsma 87fdb521f0 CLOUDSTACK-8443: don't try to fix co-mounted cgroups
This setting works on CentOS 6 / RHEL 6 but does nothing, as
"cpu" cgroup is not mounted. On CentOS 7 / RHEL 7 systemd does
mount cgroups and "cpu" is co-mounted with "cpuacc". Hence, if
we specify "cpu" then this results in an error because it can
only use them both, or none.

By removing the setting, we rely on the default of qemu, which
is:
cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]

Only if they are really mounted, they will be used. So, this will
work on both version 6 and 7.

The 'fix script' didn't work well, as after a reboot you'd still have qemu
throwing errors. Now we can handle the co-mountedcgroups.
2015-08-24 15:49:40 +02:00
Wei Zhou 92344c006d CLOUDSTACK-5863: revert volume snapshot for KVM/QCOW2 2015-08-24 11:01:50 +02:00
Rohit Yadav 3fd35dd4a7 CLOUDSTACK-8749: KVM - cleanup linklocal interface based on names
This tries to avoid cleaning by a device name.

This closes #718

(cherry picked from commit 74f697a2dd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-19 21:43:24 +05:30
Remi Bergsma caa524b01b Merge pull request #630 from manuiiit/pull-2
Fixed Coverity issue "Dereference null return value"Fixed Coverity issue "Dereference null return value"

* pr/630:
  Fixed Coverity issue "Dereference null return value"

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-08-17 08:38:38 +02:00
manuiiit 3cc0fecccc Fixed Coverity issue "Dereference null return value" 2015-08-15 01:34:24 +05:30
Aaron Brady 77a96ed1f7 Default to notify only script to handle non-CLVM/QCOW cases.
Before this commit the call to `getResizeScriptType` would throw
an exception (earlier versions returned `null`, which was fine) -
this caused the RBD case to fail. By changing the default to
notify only we fix the case for any non-CLVM and non-QCOW cases.

This is RBD for now, but this should extend to new storage
types suppored by Libvirt natively in future.
2015-08-14 12:15:07 +01:00
Rohit Yadav 869a83f8f8 plugins: fix test case, interface usage and checkstyle import issue
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-14 12:23:50 +05:30
Rohit Yadav 5d29b63cfa CLOUDSTACK-8613, CLOUDSTACK-6301: Dump KVM domain XML with secure flag
When dumping XML use appropriate flags:

1, VIR_DOMAIN_XML_SECURE (dump security sensitive information too)
8, VIR_DOMAIN_XML_MIGRATABLE (dump XML suitable for migration)

Source:
https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainXMLFlags

This fixes CVE 2015-3252: VNC password lost during VM migration across KVM
hosts. The issue is also seen when a VM is rebooted.

(cherry picked from commit cb2aca7516)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-08-14 10:57:36 +05:30
Daan Hoogland e2b6237464 CLOUDSTACK-8656: debug messages on interupted exceptions 2015-07-30 16:03:10 +02:00
Maneesha P 1f09b8c3bf Fixed Coverity issue "Dereference null return value"
added a not null check and a CloudRuntimeException in case of null

Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>

This closes #617
2015-07-23 15:46:17 +05:30
Wido den Hollander d1f76a2a84 CLOUDSTACK-8628: kvm: Disable Fencing when no NFS storage pools are present
On NFS we write a heartbeat, but without those we can not safely
fence off a host.

If we fence without knowing about a heartbeat we can cause a split-brain situation.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2015-07-17 12:33:11 +02:00
Daan Hoogland c53959513e findbugs: create wrapper to deal with static field
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>

This closes #573
2015-07-14 11:50:39 +02:00
Daan Hoogland 10853bfa8a adhere to naming conventions for enums
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
2015-07-14 11:50:39 +02:00
Wido den Hollander 63688e70a0 kvm: More verbose logging around storage pools in the Agent 2015-07-13 16:53:26 +02:00
Wido den Hollander 24f8e66e93 kvm: Write Libvirt instead of LibVirt 2015-07-13 16:22:33 +02:00
Wido den Hollander 7c37ba73a3 kvm: Add more verbose logging about what is happening with Libvirt connections 2015-07-13 16:21:41 +02:00
Daan Hoogland bcc8182f2f findbugs: write to static from instance these are the trivial cases of ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD as reported by findbugs
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>

This closes #539
2015-07-09 15:01:58 +02:00
Daan Hoogland 31c6b14dd4 Merge branch 'improvement/CLOUDSTACK-8607' change the host password on the host itself via the updateHostPassword API
- Make sure the new password replaces the old one in the queue
    - Updated the patch files for XenServer
    - Updated the script path on LibvirtComputing class
    - Adding update_host_passwd to VRScripts
    - Add implementation to CitrixUpdateHostPasswordCommandWrapper
    - Improve testUpdateHostPasswordCommand() unit test on CitrixRequestWrapperTest
    - Adding update_host_passwd.sh script
    - Adding the host IP address as an instance variable on UpdateHostPasswordCommand
    - Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
    - Make sure doUpdateHostPassword() doesn't get called if flag is set to false
    - Do not update XenServer hosts if the cluster ID is not informed

    This closes #559
2015-07-06 16:15:22 +02:00
wilderrodrigues e348e76eee CLOUDSTACK-8607 - Improving unit tests
- Adding more unit tests as suggested by Daan and Rajani.
   - Mocking SshHelper and the Script classes, used by Citrix and Libvirt resources respectively.
2015-07-06 14:58:17 +02:00
wilderrodrigues efa34361df CLOUDSTACK-8607 - Changed update script to return exit code based on the result
- Changed location of the update_host_passwd script
   - Updated the patch files for XenServer
   - Updated the script path on LibvirtComputing class
   - Removed the hostIP from the LibvirtUpdateHostPasswordCommandWrapper execute() method
2015-07-03 14:29:57 +02:00
Kishan Kavala 3a19a8916e BUG-ID:CLOUDSTACK-7472: kvmclock fix for LXC is not required after fixing CLOUDSTACK-8177
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #557
2015-07-03 15:21:18 +05:30
wilderrodrigues 0dd02ce043 CLOUDSTACK-8607 - Adding support to update host passwd on XenServer hypervisors
- Adding update_host_passwd to VRScripts
   - Add accessor method to host password on CitrixResourceBase
   - Add implementation to CitrixUpdateHostPasswordCommandWrapper
   - Improve testUpdateHostPasswordCommand() unit test on CitrixRequestWrapperTest
   - Add line to patch files on xenserver directory

Concerning the LibVirt change:

   - I forgot to assing the return of the getDefaultHypervisorScriptsDir() method to the hypervisorScriptsDir variable
2015-07-03 10:24:44 +02:00
wilderrodrigues 47c7a1083f CLOUDSTACK-8607 - Adding update_host_passwd.sh script
- Modifying the LibvirtUpdateHostPasswordCommandWrapper in order to execute the script on the host
   - Adding the script path to LibvirtComputingResource
   - Adding the host IP address as an instance variable on UpdateHostPasswordCommand
   - Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
2015-07-02 14:54:51 +02:00
wilderrodrigues 527d6ee77b Making possible to update KVM hosts password in the database via the API
- Make sure the doUpdateHostPassword() method returns the result from the answer, not always true
   - Added the LibvirtUpdateHostPasswordCommandWrapper class, which will handle the changes in the KVM agent
     That's will be further implemented in the future. The command will avoid the agent to complain about Unsupported command
   - Added a test to make sure the current implementation of the comment works
     - If changes in the future, it will also require changes. In that way, we make sure nobody will break it

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>

This closes #527
2015-06-29 09:59:12 +02:00
Daan Hoogland bbb165a037 findbugs: use preferred encoding explicitly 2015-06-20 10:36:15 +02:00
Daan Hoogland e3407a3939 findbugs: encoding using preferred Charset 2015-06-19 23:03:05 +02:00
Wido den Hollander 4b4c52ea77 CLOUDSTACK-8560: Stat the resulting image after copying from template and return the size
This way we update the DB with the actual size of the disk after deployment from template
2015-06-15 17:55:56 +02:00
Rafael da Fonseca ccd7d41ee3 Resolve findbugs encoding warning
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #392
2015-06-15 12:07:55 +03:00
Rafael da Fonseca f8b364b7c7 Was safe either way as this piece of code should only run in linux, this just gets rid of the findbugs warning
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #402
2015-06-15 12:04:47 +03:00
Rafael da Fonseca 3299f23ea2 Fix findbugs encoding warning in LibvirtCreatePrivateTemplateFromVolumeCommandWrapper Libvirt templates should be written in UTF-8, default was already doing that so this just gets rid of the findbugs warning
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #401
2015-06-15 12:04:24 +03:00
Rafael da Fonseca 949e97dec6 Fix findbugs warning in LibvirtConsoleProxyLoadCommandWrapper Encoding is now specified in both server and clients for the console proxy getstatus command For some reason, findbugs did not detect unsafe encoding issue in ConsoleProxyResource.java, is properly specified now though... CitrixConsoleProxyLoadCommandWrapper was specifying system default encoding for operation, should be ok because default for xenserver is UTF-8 since it's linux based, but it's best to specify exact encoding set on the server end to be consistent
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-06-15 12:04:18 +03:00
wilderrodrigues 93845afd53 Using a try-wioth resrouce block as suggested in @DaanHoogland review.
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>

This closes #355
2015-06-04 12:34:05 +02:00
wilderrodrigues 7f4e2c7cf0 Renaming the variable from "s" to "script"
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
2015-06-04 12:34:05 +02:00
wilderrodrigues 6138d9a690 Coverity issue 1116812 - Replacing concatenation with optionsBuffer.append(option.getKey()).append('=').append(option.getValue()).append(',');
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
2015-06-04 12:34:05 +02:00
wilderrodrigues 9ff38486a1 Coverity issue 1116677 - Avoiding catching only Exception. Makes the code too britle. - Catching the QemuImgException and throwing it to be caught further in the code - Surrounding the output stream with try/catch and throwing it to be further handled in the code. Closing the output stream quietly.
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
2015-06-04 12:34:05 +02:00
wilderrodrigues 6271663682 Formatting the code - Adding final modifier and indenting the code
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
2015-06-04 12:34:04 +02:00
Daan Hoogland 3d4d152753 CID 1302976 Scanner in try-with-resource
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #352
2015-06-04 00:09:40 +02:00
Daan Hoogland 109b6e94d3 CID 1302974: Scanner in try-with-resource
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #353
2015-06-04 00:09:16 +02:00
Rohit Yadav f341246888 CLOUDSTACK-8530: KVM hosts without active agents should be in Disconnected state
KVM hosts which are actuall up, but if their agents are shutdown should be put
in disconnected state. This would avoid getting the VMs HA'd and other commands
such as deploying a VM will exclude that host and save us from errors.

The improvement is that, we first try to contact the KVM host itself. If it fails
we assume that it's disconnected, and then ask its KVM neighbours if they can
check its status. If all of the KVM neighbours tell us that it's Down and we're
unable to reach the KVM host, then the host is possibly down. In case any of the
KVM neighbours tell us that it's Up but we're unable to reach the KVM host then
we can be sure that the agent is offline but the host is running.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>

This closes #340
2015-06-02 13:27:16 +02:00
wilderrodrigues 607a63b12d Mocking the Connect object used in the LibvirtMigrateCommandWrapper
When executing the tests in an environment where Libvirt is also installed, it
caused errors.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #342
2015-06-01 21:11:25 +02:00
wilderrodrigues b267c5fc5a Replacing real IPs by 127.0.0.1. - It was causing problems in some environments - The Ips should have been removed in a previous commit, but some of them were missed
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>

This closes #325
2015-05-29 11:25:40 +02:00
wilderrodrigues ce9014d2ec Fix the NPE tht was being caught by the executeRequest() method. - The LibvirtUtilitiesHelper should have been injected, but it did not work on the Agent side. Due to that, when sending a StartCommand we were experiencing NPE, which made impossible to get SSVM/CPVM started. - The LibvirtUtilitiesHelper class is now being instantiated withing the LibvirtComputingResource
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>

This closes #318
2015-05-27 13:19:57 +02:00
Rohit Yadav cde6ef94b8 CLOUDSTACK-8247: Pull average Cpu util report between polling
Pull average Cpu util report between polling intervals instead of since boot
instead of using values since uptime

(cherry picked from commit 04176eaf17)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Conflicts:
	plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

This closes #297
2015-05-25 15:52:00 +02:00
Rohit Yadav 9dcfbceae7 kvm: for disabling pxe, pass empty file
Passing the file argument to the xml break for EL 7.1, the fix removes
the argument as just passing rombar='off' with its file arg to be empty string.

This closes #290

(cherry picked from commit aafa0c80b3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-05-25 00:31:32 +02:00
Rohit Yadav d7f4498f76 kvm: Strip trailing comma for qemu-img convert options
Fix trailing comma for qemu-img convert options, Qemu 2.0+ not tolerant to it

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 90ac1aba13)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-05-22 13:33:54 +01:00
Rohit Yadav c37060a1ff CLOUDSTACK-8338: Fix hypervisor stats reporting for KVM on EL7
EL7 has a different output to 'free', use /proc/meminfo instead of a tool to be
more consistent across distros

(cherry picked from commit 212a05a345)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Conflicts:
	plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
2015-05-22 12:10:54 +01:00
wilderrodrigues 3efdc6c356 CLOUDSTACK-8502
Using Annotations on the Libvirt wrappers in order to increase maintainability
  - All wrappers modified
  - All 143 unit tests are green
2015-05-21 13:37:27 +02:00
wilderrodrigues 14c123e082 Merge branch 'fix/libvirt_unittests_timeout' of https://github.com/schubergphilis/cloudstack
This closes #270
2015-05-20 15:56:13 +02:00
wilderrodrigues 982a0235a0 CLOUDSTACK-8486
Removing real IPs from the tests because they cause a long running time for LibvirtComputingResourceTest
    - In a local machine it takes 1.977s, but in a KVM test environment it's taking 257.879 sec
2015-05-20 14:52:08 +02:00
Wido den Hollander 3894d34b2c CLOUDSTACK-8489: Set smbios information for guest
This allows a guest to easily detect that it is running inside CloudStack
and see which UUID it has.
2015-05-20 13:28:56 +02:00
wilderrodrigues ab7473eea0 Changing the JUnit runner in order to avoid problems with Java 8 - All tests passed using different approaches - Maven with parameters: -Dmaven.compiler.source=1.8 -Dmaven.compiler.target=1.8 - Maven without parameters
All builds were executed using javac 1.8.0_31

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #266
2015-05-20 09:53:46 +01:00
wilderrodrigues 0cdb4b6108 Fixing the testGetHostStatsCommand test under LibvirtComputingResourceTest. - Removed the expected value from the Test annotation - Mocking the bash path in order to avoid environment/OS issues
Fixing typo on LibvirtRequestWrapper
  - Replace linbvirtCommands by libvirtCommands on LibvirtRequestWrapper

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #255
2015-05-18 20:07:02 +02:00
wilderrodrigues c7d2e444d2 Fixing license header in a file added during the LibVirt refactor
- New class is MigrateKVMAsync
2015-05-13 11:56:43 +02:00
wilderrodrigues f575206ad4 Fixing testModifySshKeysCommand in the LibvirtComputingResourceTest class
- The test was okay, but when running in an environment where a /root/.ssh/id_rsa existed, it would return true then fail
  - We now mock the calls to methods that return the key paths, instead of relying in the static variables
2015-05-08 19:53:03 +02:00
wilderrodrigues 74ad48db55 Refactoring the LibvirtComputingResource
- Adding LibvirtStartCommandWrapper
  - 8 unit tests added
  - KVM hypervisor plugin with 23.2% coverage
2015-05-06 19:24:16 +02:00
wilderrodrigues 09656ca84e Refactoring the LibvirtComputingResource
- Adding LibvirtNetworkElementCommandWrapper and LibvirtStorageSubSystemCommandWrapper
  - 2 unit tests added
  - KVM hypervisor plugin with 22.2% coverage

I also refactored the StorageSubSystemCommand interface into an abstract class
  - Remove the pseudo-multiple-inheritance implementation
    - The StorageSubSystemCommand was an interface, not related to the Command class
      and its implementation were extending the Command class anyway. The whole structure is better now.
2015-05-06 19:24:16 +02:00
wilderrodrigues 08106e34d0 Refactoring the LibvirtComputingResource
- Adding LibvirtResizeVolumeCommandWrapper
  - 5 unit tests added
  - KVM hypervisor plugin with 22.1% coverage
2015-05-06 19:24:15 +02:00
wilderrodrigues ae505e7bef Refactoring the LibvirtComputingResource
- Addin LibvirtPvlanSetupCommandWrapper
  - 6 unit tests added
  - KVM hypervisor plugin with 21% coverage

From the 6 tests added, 2 were extra tests to increase the coverage of the LibvirtStopCommandWrapper
  - Increased from 35% to 78.7%
2015-05-06 19:24:15 +02:00
wilderrodrigues 7fd43f33d3 Refactoring the LibvirtComputingResource
- 5 unit tests added
  - KVM hypervisor plugin with 20.1% coverage
2015-05-06 19:24:15 +02:00
wilderrodrigues 0b1b2b6d92 Refactoring the LibvirtComputingResource
- Adding LibvirtCopyVolumeCommandWrapper

Refactoring the LibvirtUtilitiesHelper
  - Changing method name

Did not add any test to this commit due to the refactor mentioned abot.
Will proceed and add the tests

i# Please enter the commit message for your changes. Lines starting
2015-05-06 19:24:15 +02:00
wilderrodrigues 885b9e45d7 Renaming LibvirtConnectionWrapper to LibvirtUtilitiesHelper
- Gave it a better, more suggestive, name since I now added other methods to the class.
  - It makes easier to mock objects and get a better coverage of the classes
2015-05-06 19:24:14 +02:00
wilderrodrigues b3913ca1fb Refactoring the LibvirtComputingResource
- Adding LibvirtCreatePrivateTemplateFromSnapshotCommandWrapper
  - 5 unit tests added
  - KVM hypervisor plugin with 19.5% coverage
2015-05-06 19:24:14 +02:00
wilderrodrigues bcf78d3b43 Refactoring the LibvirtComputingResource
- Adding LibvirtBackupSnapshotCommandWrapper, LibvirtCreatePrivateTemplateFromVolumeCommandWrapper and LibvirtManageSnapshotCommandWrapper
  - 3 unit tests added
  - KVM hypervisor plugin with 18.3% coverage

Less tests added to those classes because the code is quite complex and way too long.
The tests added are just covering the new flow, to make sure it works fine. I will come back to those classes later.
2015-05-06 19:24:13 +02:00
wilderrodrigues 52d9f0c206 Refactoring the LibvirtComputingResource
- Adding LibvirtNetworkUsageCommandWrapper, LibvirtPlugNicCommandWrapper and LibvirtUnPlugNicCommandWrapper
  - 9 unit tests added
  - KVM hypervisor plugin with 18.3% coverage
2015-05-06 19:24:13 +02:00
wilderrodrigues ff7ae9ca0c Refactoring the LibvirtComputingResource
- Adding LibvirtCreateVolumeFromSnapshotCommandWrapper, LibvirtFenceCommandWrapper and LibvirtSecurityGroupRulesCommandWrapper
  - 6 unit tests added
  - KVM hypervisor plugin with 17.2% coverage
2015-05-06 19:24:13 +02:00
wilderrodrigues 3c8b217262 Refactoring the LibvirtComputingResource
- Adding LibvirtCheckOnHostCommandWrapper and LibvirtOvsCreateTunnelCommandWrapper
  - 4 unit tests added
  - KVM hypervisor plugin with 16.2% coverage
2015-05-06 19:24:12 +02:00
wilderrodrigues 6748a73b82 Refactoring the LibvirtComputingResource
- Adding 4 new command wrappers
  - 12 unit tests added
  - KVM hypervisor plugin with 15.5% coverage
2015-05-06 19:24:12 +02:00
wilderrodrigues 5499eecd33 Refactoring the LibvirtComputingResource
- Adding 7 new command wrappers
  - 10 unit tests added
  - KVM hypervisor plugin with 14.8% coverage
2015-05-06 19:24:12 +02:00
wilderrodrigues 4887ce7254 Refactoring the LibvirtComputingResource
- Adding LibvirtOvsDestroyBridgeCommandWrapper, LibvirtOvsSetupBridgeCommandWrapper
  - 4 unit tests added
  - KVM hypervisor plugin with 13.9% coverage

More tests added to cover LibvirtPrepareForMigrationCommandWrapper
  - Coverage of this wrapper broght from 37% to 90.6%
  - 4 new tests added
2015-05-06 19:24:11 +02:00
wilderrodrigues 0d860af659 Refactoring the LibvirtComputingResource
- Adding LibvirtDeleteStoragePoolCommandWrapper, LibvirtGetStorageStatsCommandWrapper and LibvirtUpgradeSnapshotCommandWrapper
  - 7 unit tests added
  - KVM hypervisor plugin with 13.6% coverage
2015-05-06 19:24:11 +02:00
wilderrodrigues 8268d353a2 Refactoring the LibvirtComputingResource
- Adding LibvirtPrimaryStorageDownloadCommandWrapper
  - 4 unit tests added
  - KVM hypervisor plugin with 13.3% coverage
2015-05-06 19:24:11 +02:00
wilderrodrigues 4472cade2f Refactoring the LibvirtComputingResource
- Adding LibvirtDestroyCommandWrapper
  - 2 unit tests added
  - KVM hypervisor plugin with 12.9% coverage
2015-05-06 19:24:11 +02:00
wilderrodrigues cb4670279f Refactoring the LibvirtComputingResource
- Adding LibvirtCreateCommandWrapper and LibvirtMaintainCommandWrapper
  - 4 unit tests added
  - KVM hypervisor plugin with 12.7% coverage
2015-05-06 19:24:10 +02:00
wilderrodrigues 6f757c6bf0 Refactoring the LibvirtComputingResource
- Adding LibvirtModifySshKeysCommandWrapper
  - 1 unit test added
  - KVM hypervisor plugin with 12.3% coverage
2015-05-06 19:24:10 +02:00
wilderrodrigues f14c7c2074 Refactoring the LibvirtComputingResource
- Adding LibvirtGetVncPortCommandWrapper
  - 2 unit tests added
  - KVM hypervisor plugin with 12.1% coverage
2015-05-06 19:20:43 +02:00
wilderrodrigues 7319a12600 Refactoring the LibvirtComputingResource
- Adding LibvirtCheckConsoleProxyLoadCommandWrapper, LibvirtConsoleProxyLoadCommandWrapper, LibvirtWatchConsoleProxyLoadCommandWrapperand CitrixConsoleProxyLoadCommandWrapper
  - 2 unit tests added
  - KVM hypervisor plugin with 12% coverage

Refactored the CommandWrapper interface in order to remove the esecuteProxyLoadScan, which is now
implemented bu subclasses.
2015-05-06 19:20:43 +02:00
wilderrodrigues 08a9523dcd Fixing assertion on the tests that are execting exceptions 2015-05-06 19:20:43 +02:00
wilderrodrigues 40886b337c Refactoring the LibvirtComputingResource
- Adding LibvirtAttachVolumeCommandWrapper
  - 3 unit tests added
  - KVM hypervisor plugin with 11.7% coverage
2015-05-06 19:20:42 +02:00
wilderrodrigues 4d216f1a63 Refactoring the LibvirtComputingResource
- Adding LibvirtAttachIsoCommandWrapper, LibvirtAttachIsoCommandWrapper and LibvirtAttachIsoCommandWrapper
  - 7 unit tests added
  - KVM hypervisor plugin with 11.5% coverage
2015-05-06 19:20:42 +02:00
wilderrodrigues be1e517615 Refactoring the LibvirtComputingResource
- Adding LibvirtPingTestCommandWrapper
  - 3 unit tests added
  - KVM hypervisor plugin with 11.2% coverage
2015-05-06 19:20:42 +02:00
wilderrodrigues 28e55462f1 Refactoring the LibvirtComputingResource
- Adding LibvirtMigrateCommandWrapper
  - 1 unit tests added
  - KVM hypervisor plugin with 10.9% coverage
2015-05-06 19:20:41 +02:00
wilderrodrigues d730efb866 Refactoring the LibvirtComputingResource
- Adding LibvirtCheckHealthCommandWrapper and LibvirtPrepareForMigrationCommandWrapper
  - 2 unit tests added
  - KVM hypervisor plugin with 10.8% coverage
2015-05-06 19:20:41 +02:00
wilderrodrigues 54f68f519f Removing the CheckStateCommand form the IF statement on LibvirtComputingResource
- It's not used and if so would cause an infinite loop.
2015-05-06 19:20:41 +02:00
wilderrodrigues 6e568125fc Refactoring the LibvirtComputingResource
- Adding LibvirtGetHosStatsCommandWrapper
  - 1 unit test added
  - KVM hypervisor with 10.5% coverage

Tests are a bit limited on this one becuause of the current implementation. Would clean it up later in a separate branch
2015-05-06 19:20:41 +02:00
wilderrodrigues 7a90c018fb Refactoring the LibvirtComputingResource
- Adding LibvirtRebootCommandWrapper and LibVirtRebootRouterCommandWrapper
  - 2 unit tests added
  - KVM hypervisor with 10.5% coverage
2015-05-06 19:20:40 +02:00
wilderrodrigues 51093afcf1 Refactoring the LibvirtComputingResource
- Adding LibvirtGetVmDiskStatsCommandWrapper
  - 1 unit test

Added getConnection() to LibvirtConnectionWrapper
2015-05-06 19:20:40 +02:00
wilderrodrigues 7086d64387 Refactoring the LibvirtComputingResource
- Adding LibvirtGetVmStatsCommandWrapper
  - 3 unit tests

Refactored the LibvirtConnectiobn by surrounding it with an wrapper.
  - Make it easier to cover the static/native calls
  - Added better coverage to StopCommand tests
2015-05-06 19:20:40 +02:00
wilderrodrigues 508f10527f Refactoring the LibvirtComputingResource
- Adding LibvirtStopCommandWrapper
  - LibvirtRequestWrapper
  - 1 unit tests

Refactored the RequestWrapper to make it better.
  - Changes also applied to the CitrixRequestWrapper
2015-05-06 19:20:39 +02:00
Laszlo Hornyak 58e5dca8ef CLOUDSTACK-8335: removed ceph repository - artifact resolved from maven central
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2015-05-01 11:10:06 +05:30
Rohit Yadav 1d99a38096 CLOUDSTACK-8424: enable cpu flags only for guest VMs
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-30 04:13:07 +02:00
Daan Hoogland 7b10591758 findbugs: String.split() will not return nulls by contract 2015-04-29 16:07:38 +02:00
Rajani Karuturi 0b8355920e Merge branch 'volume-upload' into master
This closes #206
2015-04-29 11:12:53 +05:30
Rohit Yadav 29b8800660 kvm: add VMXNET3 nic adapter to nic models
Linux kernel supports vmxnet3, allowing it in KVM plugin would allow us to
run ESX hosts on KVM hosts using CloudStack with vmxnet3 nic which can be
passed as VM's nicAdapter detail

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit e02d787f30)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-28 18:23:55 +02:00
Rohit Yadav 58cc569273 CLOUDSTACK-8424: Add cpu features if guest.cpu.features is set
This improvements checks for "guest.cpu.features" property which is a space
separated list of cpu features that is specific for a host. When added, it
will add  <feature policy='require' name='{{feature-you-listed}}'/> in the
<cpu> section of the generated vm spec xml.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit ea7fd37783)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-28 13:19:49 +02:00
Laszlo Hornyak 9cf31b0714 CLOUDSTACK-8335: removed libvirt.org repository
The only artifact resolved from libvirt.org was org.libvirt:libvirt:0.5.1
this artifact is now available from maven's default central repository

This closes #180

Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-24 16:58:59 +02:00
Rohit Yadav 05e4ba7350 kvm: fallback to eu.ceph.com mvn repo as ceph.com is down
As suggested by Wido on the dev ML changing the repo to eu.ceph.com to avoid
build failures. Will revert if ceph.com is up again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit c9fd57fff3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-16 17:37:11 +05:30
Remi Bergsma ac06ec02eb kvm: improve CentOS 7 support
CentOS 7 does not ship with ifconfig anymore. We should use ip commands instead.
This also works on older versions, like CentOS 6 and Ubuntu 12.x/14.x, that we
support.

This closes #165

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-14 16:51:14 +05:30
Rohit Yadav ccf13ec04e kvm: xml escape password
(cherry picked from commit 13de2691fd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Conflicts:
	plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
2015-04-13 16:10:42 +05:30