Commit Graph

2180 Commits

Author SHA1 Message Date
Wido den Hollander b130e55088 CLOUDSTACK-9397: Add Watchdog timer to KVM Instance (#1707)
The watchdog timer adds functionality where the Hypervisor can detect if an
instance has crashed or stopped functioning.
The watchdog timer adds functionality where the Hypervisor can detect if an
instance has crashed or stopped functioning.

When the Instance has the 'watchdog' daemon running it will send heartbeats
to the /dev/watchdog device.

If these heartbeats are no longer received by the HV it will reset the Instance.

If the Instance never sends the heartbeats the HV does not take action. It only
takes action if it stops sending heartbeats.

This is supported since Libvirt 0.7.3 and can be defined in the XML format as
described in the docs: https://libvirt.org/formatdomain.html#elementsWatchdog

To the 'devices' section this will be added:

In the agent.properties the action to be taken can be defined:

vm.watchdog.action=reset

The same goes for the model. The Intel i6300esb is however the most commonly used.

vm.watchdog.model=i6300esb

When the Instance has the 'watchdog' daemon running it will send heartbeats
to the /dev/watchdog device.

If these heartbeats are no longer received by the HV it will reset the Instance.

If the Instance never sends the heartbeats the HV does not take action. It only
takes action if it stops sending heartbeats.

This is supported since Libvirt 0.7.3 and can be defined in the XML format as
described in the docs: https://libvirt.org/formatdomain.html#elementsWatchdog

To the 'devices' section this will be added:

  <watchdog model='i6300esb' action='reset'/>

In the agent.properties the action to be taken can be defined:

  vm.watchdog.action=reset

The same goes for the model. The Intel i6300esb is however the most commonly used.

  vm.watchdog.model=i6300esb

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-09-28 13:56:15 +05:30
SudharmaJain 3f69c83f96 CLOUDSTACK-9956: File search on the vmware datastore may select wrong file if there are multiple files with same name (#2153)
If there are multiple files with the same name on vmware datastore, search operation may select any one file during volume related operations. This involves volume attach/detach, volume download, volume snapshot etc.

While using NetApp as the backup solution. This has .snapshot folder on the datastore and sometimes files from this folder gets selected during volume operations and the operation fails. Because of wrong selection of file following exception can be observed while volume deletion.

2017-02-23 19:39:05,750 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-304:ctx-a1dbf5d8 ac.local) delete volume failed due to Exception: java.lang.RuntimeException
Message: Cannot delete file [4cbcd46d44c53f5c8244c0aad26a97e1] .snapshot/hourly.2017-02-23_1605/r-97-VM/ROOT-97.vmdk

To fix this behavior I have added a global configuration by name vmware.search.exclude.folders which can be comma separated list of folder paths.

I have also added a unit test to test the new method.
2017-09-19 15:12:17 +05:30
vedulasantosh 48fa20698e CLOUDSTACK-9963 Root Disk controller value is changing while migrating VM 2017-09-11 16:19:10 +05:30
Rohit Yadav 774998d76b Updating pom.xml version numbers for release 4.9.4.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-07 20:21:43 +05:30
subhash yedugundla 0d81e88601 CLOUDSTACK-8939: VM Snapshot size with memory correctly calculated in cloud.usage_event (XenServer) (#914)
Fixing the error  with snapshot size calculation  in a vm with memory
2017-09-01 11:00:55 +02:00
Gabriel Beims Bräscher 01b040d19e CLOUDSTACK-9314: Remove unused code from XenServerStorageProcessor (#1443)
This commit removes unused code from
"com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor"
class and modifies some methods access level.

Remove unused methods:
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getSRByNameLabel(Connection,
String)
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getNfsSR(Connection,
StorageFilerTO)
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.directDownloadHttpTemplate(CopyCommand,
DecodedDataObject, DecodedDataObject)
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.execute(AttachPrimaryDataStoreCmd)


access level:
- from public to private:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.swiftUpload(Connection,
SwiftTO, String, String, String, Boolean, int)
- from public to protected:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.swiftBackupSnapshot(Connection,
SwiftTO, String, String, String, Boolean, int)
- from protected to private:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getSnapshotSize(Connection,
String, String, Boolean, int)
- from protected to private:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.backupSnapshot(Connection,
String, String, String, String, String, String, Boolean, int)
2017-08-31 09:34:24 +02:00
Boris Stoyanov f917ab660e CLOUDSTACK-9782: Improve host HA tests
- All tests should pass on KVM, Simulator
- Add test cases covering FSM state transitions and actions

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav d2c3408da7 CLOUDSTACK-9782: Improve scheduling of jobs
- Removed three bg thread tasks, uses FSM event-trigger based scheduling
- On successful recovery, kicks VM HA
- Improves overall HA scheduling and task submission, lower DB access

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav d145944be0 Updating pom.xml version numbers for release 4.9.3.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 14:52:37 +02:00
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
Daan Hoogland 94718c0bd3 CE-110 markedForGC is ignored so delete the templates in the task that finds them 2017-08-25 08:57:52 +02:00
Daan Hoogland 16b34c9a11 CE-110 default interval of 0 minutes means do not mark for cleaning 2017-08-25 08:57:51 +02:00
Daan Hoogland c10c342d9e CE-110 task for marking cleaning fully cloned templates
implemented by marking them for GC
2017-08-25 08:57:51 +02:00
sureshanaparti 0c6cf69eee CLOUDSTACK-9717: [VMware] RVRs have mismatching MAC addresses for extra public NICs. (#1878)
Fix: When RVR is enabled and Peer Router is available, get the MAC addresses of the extra public NICs from the Peer Router and set them to the router.
2017-08-21 11:04:20 +02:00
Wido den Hollander 2867080979 CLOUDSTACK-10034: Use libvirt to create new volumes and not rados-java (#2039)
Since libvirt 1.2.2 libvirt will properly create volumes
using RBD format 2.

We can use libvirt to creates the volumes which strips a bit of
code from the CloudStack Agent's responsbility.

RBD format 2 is already used by all volumes created by CloudStack.

This format is the most recent format of RBD and is still actively
being developed.

This removes the support for Ubuntu 12.04 as that does not have the
proper libvirt version available.

Signed-off-by: Wido den Hollander wido@widodh.nl

We can use libvirt to creates the volumes which strips a bit of
code from the CloudStack Agent's responsbility.

RBD format 2 is already used by all volumes created by CloudStack.

This format is the most recent format of RBD and is still actively
being developed.

This removes the support for Ubuntu 12.04 as that does not have the
proper libvirt version available.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-08-06 00:04:21 +02:00
Sateesh Chodapuneedi 6977cb3841 CLOUDSTACK-9684 Invalid zone id error while listing vmware zone
Issue
=====
While listing datacenters associated with a zone, only zone Id validation is required.
There is no need to have additional checks like zone is a legacy zone or not.

Fix
===
Removed unnecessary checks over zone ID and just checking if zone with specified ID exists or not.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
(cherry picked from commit 0ef1c17541)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
subhash yedugundla 1b26a486d1 BUG-ID:CLOUDSTACK-8921
Summary: CLOUDSTACK-8921
     snapshot_store_ref table should store actual size of back snapshot in secondary storage
Calling SR scan to make sure size is updated correctly

(cherry picked from commit 4e4b67cd96)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Rajani Karuturi 9ce2dde030 CLOUDSTACK-9834: prepareTemplate API call doesn't work well
with XenServer & Local SR (Db_exn.Uniqueness_constraint_violation)

removed the host uuid from SR label so that any host which has access to
the SR(all the hosts in the same pool) can reuse the same SR

(cherry picked from commit 1aa6a72bc7)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Rene Moser 05f94b8ef7 Bug fix for CLOUDSTACK-9762 Management Server UI (VM statistics page) CPU Utilized value is incorrect.
(cherry picked from commit b676a8a7c3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Anshul Gangwar e6855db78a CLOUDSTACK-9685: delete snapshot on primary associated with a volume when that volume is deleted
as that snapshot will never be going to use again and also it will fill up primary storage

(cherry picked from commit 336df84f17)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
nvazquez 1d25582ed6 CLOUDSTACK-9752: [Vmware] Optimization of volume attachness to vm
(cherry picked from commit 49dadc5505)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Suresh Kumar Anaparti 0fbbbdb83a CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.
(cherry picked from commit 8676b20276)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Suresh Kumar Anaparti e61815a255 CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM
Updated hardcoded value with max data volumes limit from hypervisor capabilities.

(cherry picked from commit 93f5b6e8a3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Wei Zhou c03923c7e7 CLOUDSTACK-9113: skip vm with inconsistent state when getVmNetworkStats 2017-07-23 17:15:23 +02:00
Wei Zhou 960cb84083 CLOUDSTACK-7984: Collect network statistics for VMs on shared network (KVM implementation) 2017-07-23 17:15:23 +02:00
Nitin Kumar Maharana b2b422c7d7 CLOUDSTACK-8672 : NCC Integration with CloudStack. 2017-07-20 12:42:43 +05:30
Rohit Yadav 445dbe7c53 Merge remote-tracking branch 'origin/4.10' 2017-07-14 15:00:11 +05:30
Rohit Yadav 32e96abea9 Merge remote-tracking branch 'origin/4.9' into 4.10 2017-07-14 14:59:17 +05:30
Wido den Hollander ca415e7436 CLOUDSTACK-9929: Do not gather statistics for CDROM or FLOPPY devices
Libvirt / Qemu (KVM) does not collect statistics about these either.

On some systems it might even yield a 'internal error' from libvirt
when attempting to gather block statistics from such devices.

For example Ubuntu 16.04 (Xenial) has a issue with this.

Skip them when looping through all devices.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-07-14 14:57:11 +05:30
Rajani Karuturi 4bc7c270fa Updating pom.xml version numbers for release 4.11.0.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-12 12:09:38 +05:30
Rajani Karuturi 4f229d19ad Updating pom.xml version numbers for release 4.10.1.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-10 10:36:24 +05:30
Rajani Karuturi 9d2893d44a Updating pom.xml version numbers for release 4.10.0.0
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-03 10:06:43 +05:30
Mike Tutkowski 330089a801
Only add the “.vmdk” suffix if need be
(cherry picked from commit b156d5de11)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-06-15 16:29:05 +05:30
Rohit Yadav ed376fcad6 Merge remote-tracking branch 'origin/4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-07 11:21:27 +05:30
Rohit Yadav e197652a28 CLOUDSTACK-9860: Fix stackoverflow issue
Fixes issue caused to a PR-refactoring from #2108, reported by
@borisstoyanov

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-06 16:14:03 +05:30
Rajani Karuturi 321712bcaf Merge pull request #1841 from sateesh-chodapuneedi/pr-cloudstack-9684
CLOUDSTACK-9684 Invalid zone id error while listing vmware zone
2017-06-06 16:04:24 +05:30
Rajani Karuturi c4c692afa5 Merge pull request #1862 from sateesh-chodapuneedi/pr-cloudstack-9704
CLOUDSTACK-9704 Remove dependency on VmwareContext object to fetch systemVM keyfile
2017-06-06 15:57:18 +05:30
Rajani Karuturi 7ee751e747 Merge pull request #1894 from Accelerite/CLOUDSTACK-9700
CLOUDSTACK-9700 Allow user to Register/Copy templates to multiple zones at the same time
2017-06-06 15:41:28 +05:30
Rajani Karuturi 51974cd568 Merge pull request #899 from yvsubhash/CLOUDSTACK-8921
BUG-ID:CLOUDSTACK-8921 snapshot_store_ref table should store actual size of backup snapshot in secondary storage
2017-06-06 15:38:42 +05:30
Rohit Yadav 8323a175f1 CLOUDSTACK-9860: Power off VMs when stopVM is called with forced=true
The 'force' option provided with the stopVirtualMachine API command is
often assumed to be a hard shutdown sent to the hypervisor, when in fact
it is for CloudStacks' internal use. CloudStack should be able to send
the 'hard' power-off request to the hosts.

When forced parameter on the stopVM API is true, power off (hard shutdown)
a VM. This uses initial changes from #1635 to pass the forced parameter
to hypervisor plugin via the StopCommand, and fixes force stop (poweroff)
handling for KVM, VMware and XenServer.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-25 17:25:22 +05:30
Rajani Karuturi e25a444a0c Merge pull request #2121 from bvbharatk/CLOUDSTACK-9641
CLOUDSTACK-9641 In KVM SSVM and CPVM may use the old cmdline data, if…
2017-05-23 10:07:55 +05:30
Bharat Kumar 3c80f00550 CLOUDSTACK-9641 In KVM SSVM and CPVM may use the old cmdline data, if we fail to fetch the new cmdline in the first pass. 2017-05-19 16:50:19 +05:30
Rajani Karuturi 701146ff28 Merge pull request #1908 from Accelerite/staticnat
CLOUDSTACK-9317: Fixed disable static nat on leaving ips on interface
2017-05-18 15:48:09 +05:30
Rajani Karuturi 8b3cadb55e Merge release branch 4.9 to master
* 4.9:
  Do not set gateway to 0.0.0.0 for windows clients
  CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
  ignore bogus default gateway   when a shared network is secondary the default gateway gets overwritten by a bogus one   dnsmasq does the right thing and replaces it with its own default which is not good for us   so check for '0.0.0.0'
  Activate NioTest following changes in CLOUDSTACK-9348 PR #1549
  CLOUDSTACK-9828: GetDomRVersionCommand fails to get the correct version as output Fix tries to return the output as a single command, instead of appending output from two commands
  CLOUDSTACK-3223 Exception observed while creating CPVM in VMware Setup with DVS
  CLOUDSTACK-9787: Fix wrong return value in NetUtils.isNetworkAWithinNetworkB
2017-05-17 11:36:52 +05:30
Jayapal d04a3e842c CLOUDSTACK-9317: Update review comments and rule state column 2017-05-17 11:08:13 +05:30
Jayapal c20e0ef88f CLOUDSTACK-9317: Fixed disable static nat on leaving ips on interface 2017-05-17 11:03:50 +05:30
Rajani Karuturi 7434d91614 Merge pull request #1873 from Accelerite/dhcpOffloadFix
CLOUDSTACK-9709: Updated the vm ip fetch task to use the correct the …
2017-05-17 10:43:51 +05:30
Rajani Karuturi 86eb5683dd Merge pull request #1840 from anshul1886/CLOUDSTACK-9685
CLOUDSTACK-9685: delete snapshot on primary associated with a volume …
2017-05-16 12:42:48 +05:30
Rajani Karuturi a4f3c71744 Merge pull request #1849 from sudhansu7/CLOUDSTACK-9690
CLOUDSTACK-9690: Scale CentOS7 VM fails with error
2017-05-16 11:24:50 +05:30
rashmidixit 49a862b223 CLOUDSTACK-9700: Allow user to Register/Copy templates to multiple zones at the same time 2017-05-15 14:07:22 +05:30
Rajani Karuturi 1aa6a72bc7 CLOUDSTACK-9834: prepareTemplate API call doesn't work well
with XenServer & Local SR (Db_exn.Uniqueness_constraint_violation)

removed the host uuid from SR label so that any host which has access to
the SR(all the hosts in the same pool) can reuse the same SR
2017-05-15 11:17:20 +05:30
Rajani Karuturi 45f62c3483 Merge pull request #1813 from priyankparihar/CLOUDSTACK-9604
CLOUDSTACK-9604: Root disk resize support for VMware and XenServer.
2017-05-08 10:15:57 +05:30
Sudhansu e7a7db3ef3 CLOUDSTACK-9690: Scale CentOS7 VM fails with error
1. Removed XenServerGuestOsMemoryMap from CitrixHelper.java
This java file was holding a static in memory map named XenServerGuestOsMemoryMap. This was the source for xenserver dynamic memory values(max and min). These values were moved to guest_os_details table.

2. DAO layer was modified to access these values.
3. VirtualMachineTo object was modified to populate the dynamic memory values.
4. addGuestOs and UpdateGuestOS api has been modified to update memory values.
2017-05-05 10:58:12 +05:30
Rohit Yadav c4326b09c6 CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
This fixes log4j xml to have @AGENTLOG@ replaced with values defined
in build/replace.properties.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-04 13:47:23 +05:30
Rajani Karuturi 57628b2dd0 Merge pull request #2030 from shapeblue/snapshot-housekeeping
CLOUDSTACK-9864 cleanup stale worker VMs after job expiry time
2017-05-02 11:08:50 +05:30
Rajani Karuturi 4ad29b474a Merge pull request #1918 from jayakarteek/cloudstack-9762
Management Server UI (VM statistics page) CPU Utilized value is incorrect.
2017-04-27 11:12:00 +05:30
Mowgli c2c1f01d2e Merge branch 'master' into CLOUDSTACK-9604 2017-04-26 11:48:56 +05:30
Rajani Karuturi fc8ed22abe Merge pull request #2060 from apache/4.9
Merge release branch 4.9 to master
2017-04-23 09:34:54 +05:30
Rajani Karuturi a4dd6bdeeb Merge pull request #1955 from myENA/virtio-scsi
CLOUDSTACK-8239 Add VirtIO SCSI support for KVM hosts
2017-04-20 15:36:34 +05:30
Daan Hoogland 084febb02b CE-113 use of duration (instead of the old clock-tick-based code 2017-04-18 18:11:00 +02:00
Daan Hoogland 8065aedf68 CE-113 extra trace log of worker VMs 2017-04-18 18:11:00 +02:00
Daan Hoogland 61189db1d9 CE-113 configure workervm gc based on job expiry 2017-04-18 18:11:00 +02:00
Suresh Kumar Anaparti bf77c07797 CLOUDSTACK-9719: [VMware] VR loses DHCP settings and VMs cannot obtain IP after HA recovery - Set high restart priority for the VR.
Test scenarios:
- Enable cluster HA after VR is created. Now stop and start VR and check its restart priority, should be High.
- Enable cluster HA before VR is created. Now create some VM and verify that VR created must have High restart priority.
2017-04-11 10:48:42 +05:30
Rajani Karuturi 4db186ef6f Merge pull request #1880 from Accelerite/CLOUDSTACK-9720
CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.Updated the template_spool_ref table with the correct template (VMware - OVA file) size.

* pr/1880:
  CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-04-06 12:11:22 +05:30
Rene Moser b676a8a7c3 Bug fix for CLOUDSTACK-9762 Management Server UI (VM statistics page) CPU Utilized value is incorrect. 2017-04-05 15:37:40 +05:30
subhash yedugundla 4e4b67cd96 BUG-ID:CLOUDSTACK-8921
Summary: CLOUDSTACK-8921
     snapshot_store_ref table should store actual size of back snapshot in secondary storage
Calling SR scan to make sure size is updated correctly
2017-04-04 15:02:17 +05:30
Rajani Karuturi ec2d4dd422 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number>
2017-03-23 15:19:31 +05:30
Wei Zhou bf93b6313e CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number> 2017-03-20 07:32:31 +01:00
Rajani Karuturi ad7ed7a178 Merge pull request #847 from kishankavala/CLOUDSTACK-8880
Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm.  free memory = total memory - (all vm memory)With memory over-provisioning set to 1, when mgmt server starts VMs in parallel on one host, then the memory allocated on that kvm can be larger than the actual physcial memory of the kvm host.

Fixed by checking free memory on host before starting Vm.
Added test case to check memory usage on Host.
Verified Vm deploy on Host with enough capacity and also without capacity

* pr/847:
  Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm.  free memory = total memory - (all vm memory)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:20:58 +05:30
Rajani Karuturi 3f0fbf251c Merge pull request #1953 from Accelerite/CLOUDSTACK-9794
CLOUDSTACK-9794: Unable to attach more than 14 devices to a VMUpdated hardcoded value with max data volumes limit from hypervisor capabilities.

* pr/1953:
  CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:19:04 +05:30
Rajani Karuturi 09802e0f3c Merge pull request #1861 from sateesh-chodapuneedi/pr-cloudstack-9698
CLOUDSTACK-9698 [VMware] Make hardcorded wait timeout for NIC adapter hotplug as configurableJira
===
CLOUDSTACK-9698 [VMware] Make hardcoded wait timeout for NIC adapter hotplug as configurable

Description
=========
Currently ACS waits for 15 seconds (hard coded) for hot-plugged NIC in VR running on VMware to get detected by guest OS.
The time taken to detect hot plugged NIC in guest OS depends on type of VMware NIC adapter like (E1000, VMXNET3, E1000e etc.)
and guest OS itself. In uncommon scenarios the NIC detection may take longer time than 15 seconds,
in such cases NIC hotplug would be treated as failure which results in VPC tier configuration failure.
Alternatively making the wait timeout for NIC adapter hotplug as configurable will be helpful for admins in such scenarios. This is specific to VR running over VMware hypervisor.

Also in future if VMware introduces new NIC adapter types which may take time to get detected by guest OS, it is good to have flexibility of
configuring the wait timeout as fallback mechanism in such scenarios.

Fix
===
Introduce new configuration parameter (via ConfigKey) "vmware.nic.hotplug.wait.timeout" which is "Wait timeout (milli seconds) for hot plugged NIC of VM to be detected by guest OS." as fallback instead of hard coded timeout, to ensure flexibility for admins given the listed scenarios above.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>

* pr/1861:
  CLOUDSTACK-9698 Make the wait timeout for NIC adapter hotplug as configurable

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:12:54 +05:30
Rajani Karuturi 56e851ca46 Merge release branch 4.9 to master
* 4.9:
  moved logrotate from cron.daily to cron.hourly for vpcrouter in cloud-early-config
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent
2017-03-13 22:09:27 +05:30
Rajani Karuturi 7b719c71fc Merge pull request #1856 from ustcweizhou/set-kvm-host-params
[4.9] CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agentThe router.aggregation.command.each.timeout in global configuration is only applied on new created KVM host.
For existing KVM host, changing the value will not be effective.
We need to propagate the configuration to existing host when cloudstack-agent is connected.

* pr/1856:
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:08:47 +05:30
Suresh Kumar Anaparti 93f5b6e8a3 CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM
Updated hardcoded value with max data volumes limit from hypervisor capabilities.
2017-03-13 16:14:12 +05:30
Suresh Kumar Anaparti 8676b20276 CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column. 2017-03-13 00:48:49 +05:30
Nathan Johnson 5c476492b1 CLOUDSTACK-8239 - Adding support for virtio-scsi on KVM hosts
This adds support for virtio-scsi on KVM hosts, either
for guests that are associated with a new os_type of 'Other PV Virtio-SCSI (64-bit)',
or when a VM or template is regstered with a detail parameter rootDiskController=scsi.

Update cloudstack add template dialog to allow for selecting rootDiskController with KVM

Update cloudstack kvm virtio-scsi to enable discard=unmap
2017-03-12 10:54:43 -05:00
Anshul And Priyank ec66256149 CLOUDSTACK-9604: Root disk resize support for VMware and XenServer. 2017-03-09 19:20:38 +05:30
Jayapal e3ae08b3ee CLOUDSTACK-9709: Updated the vm ip fetch task to use the correct the thread 2017-03-07 09:50:18 +05:30
Sateesh Chodapuneedi d171bb7857 CLOUDSTACK-9698 Make the wait timeout for NIC adapter hotplug as configurable
Currently ACS waits for 15 seconds (hard coded) for hot-plugged NIC in VR to get detected by guest OS.
The time taken to detect hot plugged NIC in guest OS depends on type of NIC adapter like (E1000, VMXNET3, E1000e etc.)
and guest OS itself. In uncommon scenarios the NIC detection may take longer time than 15 seconds,
in such cases NIC hotplug would be treated as failure which results in VPC tier configuration failure.
Alternatively making the wait timeout for NIC adapter hotplug as configurable will be helpful for admins in such scenarios.

Also in future if VMware introduces new NIC adapter types which may take time to get detected by guest OS, it is good to have flexibility of
configuring the wait timeout as fallback mechanism in such scenarios.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
2017-03-02 03:04:53 +05:30
Rajani Karuturi fa85151be9 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
  CLOUDSTACK-9788: Fix exception listNetworks with pagesize=0
  CLOUDSTACK-8663: Fixed various issues to allow VM snapshots and volume snapshots to exist together
  Fix HVM VM restart bug in XenServer
2017-02-28 05:47:06 +05:30
Rajani Karuturi d9bd01266f Merge pull request #1829 from syed/hvm-volume-attach-restart-fix
CLOUDSTACK-9363: Fix HVM VM restart bug in XenServerHere is the longer description of the problem:

By default XenServer limits HVM guests to only 4 disks. Two of those are reserved for the ROOT disk (deviceId=0) and CD ROM (device ID=3) which means that we can only attach 2 data disks. This limit however is removed when Xentools is installed on the guest. The information that a guest has Xentools installed and can handle more than 4 disks is stored in the VM metadata on XenServer. When a VM is shut down, Cloudstack removes the VM and all the metadata associated with the VM from XenServer. Now, when you start the VM again, even if it has Xentools installed, it will default to only 4 attachable disks.

Now this problem manifests itself when you have a HVM VM and you stop and start it with more than 2 data disks attached. The VM fails to start and the only way to start the VM is to detach the extra disks and then reattach them after the VM start.

In this fix, I am removing the check which is done before creating a `VBD` which enforces this limit. This will not affect current workflow and will fix the HVM issue.

@koushik-das this is related to the "autodetect" feature that you introduced a while back (https://issues.apache.org/jira/browse/CLOUDSTACK-8826). I would love your review on this fix.

* pr/1829:
  Fix HVM VM restart bug in XenServer

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-28 05:42:59 +05:30
Kishan Kavala 9a021904af Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm. free memory = total memory - (all vm memory) 2017-02-20 11:32:48 +05:30
Anshul Gangwar ca84fd4ffd CLOUDSTACK-8663: Fixed various issues to allow VM snapshots and volume
snapshots to exist together

Reverting VM to disk only snapshot in Xenserver corrupts VM

Stale NFS secondary storage on XS leads to volume creation failure from snapshot
2017-02-15 12:56:39 +05:30
nvazquez 49dadc5505 CLOUDSTACK-9752: [Vmware] Optimization of volume attachness to vm 2017-02-10 11:02:44 -03:00
nvazquez 6ce6cf67f0 CLOUDSTACK-9738: [Vmware] Optimize vm expunge process for instances with vm snapshots 2017-02-06 23:39:01 -03:00
Rajani Karuturi 7233ac37cd Merge pull request #977 from ustcweizhou/vm-snapshot
[4.10] CLOUDSTACK-8746: VM Snapshotting implementation for KVM

* pr/977:
  Fixes for testing VM Snapshots on KVM. Related to PR 977
  CLOUDSTACK-8746: vm snapshot implementation for KVM

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-01-31 05:58:56 +05:30
Rajani Karuturi f10c8bfe0c Merge pull request #1700 from wido/ipv6-basic-networking
CLOUDSTACK-9359: IPv6 for Basic NetworkingThis PR is a proposal for adding very basic IPv6 to Basic Networking. The main goal of this PR is that the API returns a valid IPv6 address over which the Instance is reachable.

The GUI will show the IPv6 address after deployment of the Instance.

![screenshot from 2016-10-03 16 34 56](https://cloud.githubusercontent.com/assets/326786/19070024/b06d2de6-8a29-11e6-8fe7-4902e2801ada.png)

If the table VLAN has a proper IPv6 CIDR configured the DirectPodBasedNetworkGuru will calculate the IPv6 Address the Instance will obtain using EUI-64 and SLAAC: https://tools.ietf.org/search/rfc4862

In this case the _vlan_ table contained:

<pre>mysql> select * from vlan \G
*************************** 1. row ***************************
                 id: 1
               uuid: 90e0716c-5261-4992-bb9d-0afd3006f476
            vlan_id: vlan://untagged
       vlan_gateway: 172.16.0.1
       vlan_netmask: 255.255.255.0
        description: 172.16.0.10-172.16.0.250
          vlan_type: DirectAttached
     data_center_id: 1
         network_id: 204
physical_network_id: 200
        ip6_gateway: 2001:980:7936:112::1
           ip6_cidr: 2001:980:7936:112::/64
          ip6_range: NULL
            removed: NULL
            created: 2016-07-19 20:39:41
1 row in set (0.00 sec)

mysql></pre>

It will then log:

<pre>2016-10-04 11:42:44,998 DEBUG [c.c.n.g.DirectPodBasedNetworkGuru] (Work-Job-Executor-1:ctx-1975ec54 job-186/job-187 ctx-0d967d88) (logid:275c4961) Found IPv6 CIDR 2001:980:7936:112::/64 for VLAN 1
2016-10-04 11:42:45,009 INFO  [c.c.n.g.DirectPodBasedNetworkGuru] (Work-Job-Executor-1:ctx-1975ec54 job-186/job-187 ctx-0d967d88) (logid:275c4961) Calculated IPv6 address 2001:980:7936:112:4ba:80ff:fe00:e9 using EUI-64 for NIC 6a05deab-b5d9-4116-80da-c94b48333e5e</pre>

The template has to be configured accordingly:
- No IPv6 Privacy Extensions
- Use SLAAC
- Follow RFC4862

This is also described in: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+in+Basic+Networking

The next steps after this will be:
- Security Grouping to prevent IPv6 Address Spoofing
- Security Grouping to filter ICMP, UDP and TCP traffic

* pr/1700:
  CLOUDSTACK-676: IPv6 In -and Egress filtering for Basic Networking
  CLOUDSTACK-676: IPv6 Basic Security Grouping for KVM
  CLOUDSTACK-9359: IPv6 for Basic Networking with KVM

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-01-27 05:42:44 +05:30
Rajani Karuturi 4721c53ea0 Merge pull request #1749 from mike-tutkowski/archived_snapshots
CLOUDSTACK-9619: Updates for SAN-assisted snapshotsThis PR is to address a few issues in #1600 (which was recently merged to master for 4.10).

In StorageSystemDataMotionStrategy.performCopyOfVdi we call getSnapshotDetails. In one such scenario, the source snapshot in question is coming from secondary storage (when we are creating a new volume on managed storage from a snapshot of ours thats on secondary storage).

This usually worked in the regression tests due to a bit of "luck": We retrieve the ID of the snapshot (which is on secondary storage) and then try to pull out its StorageVO object (which is for primary storage). If you happen to have a primary storage that matches the ID (which is the ID of a secondary storage), then getSnapshotDetails populates its Map<String, String> with inapplicable data (that is later ignored) and you dont easily see a problem. However, if you dont have a primary storage that matches that ID (which I didnt today because I had removed that primary storage), then a NullPointerException is thrown.

I have fixed that issue by skipping getSnapshotDetails if the source is coming from secondary storage.

While fixing that, I noticed a couple more problems:

1)       We can invoke grantAccess on a snapshot thats actually on secondary storage (this doesnt amount to much because the VolumeServiceImpl ignores the call when its not for a primary-storage driver).
2)       We can invoke revokeAccess on a snapshot thats actually on secondary storage (this doesnt amount to much because the VolumeServiceImpl ignores the call when its not for a primary-storage driver).

I have corrected those issues, as well.

I then came across one more problem:
         When using a SAN snapshot and copying it to secondary storage or creating a new managed-storage volume from a snapshot of ours on secondary storage, we attach to the SR in the XenServer code, but detach from it in the StorageSystemDataMotionStrategy code (by sending a message to the XenServer code to perform an SR detach). Since we know to detach from the SR after the copy is done, we should detach from the SR in the XenServer code (without that code having to be explicitly called from outside of the XenServer logic).

I went ahead and changed that, as well.

JIRA Ticket:
https://issues.apache.org/jira/browse/CLOUDSTACK-9619

* pr/1749:
  CLOUDSTACK-9619: Updates for SAN-assisted snapshots

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-01-27 05:35:06 +05:30
Wido den Hollander 84e496b4f9
CLOUDSTACK-676: IPv6 Basic Security Grouping for KVM
This commit implements basic Security Grouping for KVM in
Basic Networking.

It does not implement full Security Grouping yet, but it does:
- Prevent IP-Address source spoofing
- Allow DHCPv6 clients, but disallow DHCPv6 servers
- Disallow Instances to send out Router Advertisements

The Security Grouping allows ICMPv6 packets as described by RFC4890
as they are essential for IPv6 connectivity.

Following RFC4890 it allows:
- Router Solicitations
- Router Advertisements (incoming only)
- Neighbor Advertisements
- Neighbor Solicitations
- Packet Too Big
- Time Exceeded
- Destination Unreachable
- Parameter Problem
- Echo Request

ICMPv6 is a essential part of IPv6, without it connectivity will break or be very
unreliable.

For now it allows any UDP and TCP packet to be send in to the Instance which
effectively opens up the firewall completely.

Future commits will implement Security Grouping further which allows controlling UDP and TCP
ports for IPv6 like can be done with IPv4.

Regardless of the egress filtering (which can't be done yet) it will always allow outbound DNS
to port 53 over UDP or TCP.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-01-26 15:36:08 +01:00
Wei Zhou a2428508e2 CLOUDSTACK-8746: vm snapshot implementation for KVM
(1) add support to create/delete/revert vm snapshots on running vms with QCOW2 format
(2) add new API to create volume snapshot from vm snapshot
(3) delete metadata of vm snapshots before stopping/migrating and recover vm snapshots after starting/migrating
(4) enable deleting of VM snapshot on stopped vm or vm snapshot is not listed in qcow2 image.
(5) enable smoke tests for vmsnaphsots on KVM
2017-01-24 21:47:30 +01:00
Rohit Yadav 8b6e96bca9 Updating pom.xml version numbers for release 4.9.3.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-06 10:40:15 +05:30
Rohit Yadav dfc39c1f08 Updating pom.xml version numbers for release 4.9.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-03 12:28:47 +05:30
Sateesh Chodapuneedi 7ea41a5641 CLOUDSTACK-9704 Remove dependency on VmwareContext object to fetch system VM key file
While remote executing commands/scripts through SSH in VR, ACS uses system vm keyfile.
ACS is fetching this key file using VMwareContext object which encapsulates vCenter connection handle.
This is inefficient because of dependency on getServiceContext() which means a vCenter connection
handle which is not required just to fetch a file in name space in management server.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
2016-12-30 02:25:54 +05:30
Sateesh Chodapuneedi 0ef1c17541 CLOUDSTACK-9684 Invalid zone id error while listing vmware zone
Issue
=====
While listing datacenters associated with a zone, only zone Id validation is required.
There is no need to have additional checks like zone is a legacy zone or not.

Fix
===
Removed unnecessary checks over zone ID and just checking if zone with specified ID exists or not.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
2016-12-30 02:14:33 +05:30
Rohit Yadav ec847a890e Merge pull request #1638 from shapeblue/spring4-java8-only
CLOUDSTACK-9456: Migrate master to Spring 4.xThis changes makes CloudStack use spring 4:

```
- Bump spring-framework version to 4.x and Jetty to version that runs with JDK7
- Bump servet dependency version
- Migrates various xmls to use version independent schema uris
```

Outstanding issue:
    - Testing of various non-standard plugins such as network and storage plugins etc.

Since, this is a big change pinging for review -- @jburwell @karuturi @wido @murali-reddy @abhinandanprateek @DaanHoogland @GaborApatiNagy @JayapalUradi @kishankavala @K0zka @nvazquez @rafaelweingartner @pyr and others

@blueorangutan package

* pr/1638:
  CLOUDSTACK-9456: Update Spring version in maven poms

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-24 09:39:18 +05:30
Rohit Yadav a9f45dfc5f
Merge branch '4.9' 2016-12-23 17:50:42 +05:30
Wei Zhou 714221234d CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent 2016-12-22 12:00:10 +01:00
Rohit Yadav 342162bad7 Merge branch '4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:59:02 +05:30
Rohit Yadav a0e36b73ae Merge pull request #1711 from syed/xenserver7
XenServer 7 SupportThis PR adds support for XenServer 7. I have manually done the following tests
- Create a new cluster with XenServer7
- Add Primary storage: Should create an SR on XS7
- Add another XS7 host to the Pool
- Add host2 to Cloudstack
- Create VM1 from template
- Create VM2 from template
- Ping/SSH VM1 to VM2 and vice-versa
- Stop/Delete/Expunge VM2
- Create Data disk
- Attach it to VM1
- Create VM snaphsot of VM1
- Restore VM snapshot of VM1
- Delete VM snapshot of VM1
- Create Volume snapshot of Datadisk
- Create volume snapshot of Root disk
- Create new template from snapshot of root disk
- Create volume from snapshot of datadisk
- Detach datadisk volume
- Delete datadisk volume
- Aquire a public IP
- Create a static nat to VM1
- Live migrate VM1 while traffic on VM
- Delete VM1

* pr/1711:
  [CLOUDSTACK-9662] Add support for XenServer 7

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:18:57 +05:30