Commit Graph

3198 Commits

Author SHA1 Message Date
Rohit Yadav db7697216b cloudian: wip sso integration
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-13 12:57:16 +05:30
Rohit Yadav 170ad669e6 cloudian: fix NPE
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-12 17:40:56 +05:30
Rohit Yadav e1983c9650 cloudian: add message bus listeners
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-12 17:35:38 +05:30
Rohit Yadav f9b3b4af67 cloudian: add configs from connector props file to db
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-12 15:35:10 +05:30
Rohit Yadav df1a802cd5 cloudian: add connector plugin stubs
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-12 13:11:53 +05:30
Rohit Yadav 774998d76b Updating pom.xml version numbers for release 4.9.4.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-07 20:21:43 +05:30
Rohit Yadav d145944be0 Updating pom.xml version numbers for release 4.9.3.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 14:52:37 +02:00
Sateesh Chodapuneedi 6977cb3841 CLOUDSTACK-9684 Invalid zone id error while listing vmware zone
Issue
=====
While listing datacenters associated with a zone, only zone Id validation is required.
There is no need to have additional checks like zone is a legacy zone or not.

Fix
===
Removed unnecessary checks over zone ID and just checking if zone with specified ID exists or not.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
(cherry picked from commit 0ef1c17541)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
subhash yedugundla 1b26a486d1 BUG-ID:CLOUDSTACK-8921
Summary: CLOUDSTACK-8921
     snapshot_store_ref table should store actual size of back snapshot in secondary storage
Calling SR scan to make sure size is updated correctly

(cherry picked from commit 4e4b67cd96)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Nitesh Sarda 9334e091c0 CLOUDSTACK-9937 : dedicateCluster API response does not return correct detail in response
(cherry picked from commit 7364616aa4)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Rajani Karuturi 9ce2dde030 CLOUDSTACK-9834: prepareTemplate API call doesn't work well
with XenServer & Local SR (Db_exn.Uniqueness_constraint_violation)

removed the host uuid from SR label so that any host which has access to
the SR(all the hosts in the same pool) can reuse the same SR

(cherry picked from commit 1aa6a72bc7)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Rene Moser 05f94b8ef7 Bug fix for CLOUDSTACK-9762 Management Server UI (VM statistics page) CPU Utilized value is incorrect.
(cherry picked from commit b676a8a7c3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Anshul Gangwar e6855db78a CLOUDSTACK-9685: delete snapshot on primary associated with a volume when that volume is deleted
as that snapshot will never be going to use again and also it will fill up primary storage

(cherry picked from commit 336df84f17)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Nick Livens 1f44e884ad CLOUDSTACK-9321 : Multiple Internal LB rules (more than one Internal LB rule with same source IP address) are not getting resolved in the corresponding InternalLbVm instance's haproxy.cfg file
CLOUDSTACK-9321 : Adding component tests for VPC Network functionality - Internal LB rules

CLOUDSTACK-9321 : Extending Nuage VSP Internal LB Marvin tests

Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>, Frank Maximus <frank.maximus@nuagenetworks.net>

(cherry picked from commit 62e858131f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
nvazquez 1d25582ed6 CLOUDSTACK-9752: [Vmware] Optimization of volume attachness to vm
(cherry picked from commit 49dadc5505)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Suresh Kumar Anaparti 0fbbbdb83a CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.
(cherry picked from commit 8676b20276)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Suresh Kumar Anaparti e61815a255 CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM
Updated hardcoded value with max data volumes limit from hypervisor capabilities.

(cherry picked from commit 93f5b6e8a3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Frank Maximus 0ead11e8f1 CLOUDSTACK-9751: Fix public ip not applied, when added while VR is starting. (#1925)
Public IP state wasn't passed to the vsp client, making it ignore the apply public ip.

Reported-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-07-28 10:30:43 +02:00
Rohit Yadav 90e7cabf4e FIX3: Consider overcommit ratios with total/threshold values for host metrics
Consider the CPU and memory overcommit ratios with total cpu/ram values
or thresholds for host metrics. This will fix incorrect notification
(cells turning yellow/red) in the metrics view.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-24 12:42:02 +02:00
Wido den Hollander ca415e7436 CLOUDSTACK-9929: Do not gather statistics for CDROM or FLOPPY devices
Libvirt / Qemu (KVM) does not collect statistics about these either.

On some systems it might even yield a 'internal error' from libvirt
when attempting to gather block statistics from such devices.

For example Ubuntu 16.04 (Xenial) has a issue with this.

Skip them when looping through all devices.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-07-14 14:57:11 +05:30
Rohit Yadav e197652a28 CLOUDSTACK-9860: Fix stackoverflow issue
Fixes issue caused to a PR-refactoring from #2108, reported by
@borisstoyanov

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-06 16:14:03 +05:30
Rohit Yadav 8323a175f1 CLOUDSTACK-9860: Power off VMs when stopVM is called with forced=true
The 'force' option provided with the stopVirtualMachine API command is
often assumed to be a hard shutdown sent to the hypervisor, when in fact
it is for CloudStacks' internal use. CloudStack should be able to send
the 'hard' power-off request to the hosts.

When forced parameter on the stopVM API is true, power off (hard shutdown)
a VM. This uses initial changes from #1635 to pass the forced parameter
to hypervisor plugin via the StopCommand, and fixes force stop (poweroff)
handling for KVM, VMware and XenServer.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-25 17:25:22 +05:30
Rajani Karuturi 9c4eb959c3 Merge pull request #2070 from shapeblue/hyperv-log-fix
CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
2017-05-17 09:49:33 +05:30
Rohit Yadav c4326b09c6 CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
This fixes log4j xml to have @AGENTLOG@ replaced with values defined
in build/replace.properties.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-04 13:47:23 +05:30
Rohit Yadav efe7646977 CLOUDSTACK-9900: Fix high CPU deviation issues seen in metrics view
HostStats returns cpu usage in percentage while memory usage in bytes.
This fixes a regression in maximum CPU usage deviation that did not
assume the values to be in percentage and multiple the final ratios
with 100 which leads to 100x the actual deviation value.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-03 15:02:47 +05:30
Suresh Kumar Anaparti bf77c07797 CLOUDSTACK-9719: [VMware] VR loses DHCP settings and VMs cannot obtain IP after HA recovery - Set high restart priority for the VR.
Test scenarios:
- Enable cluster HA after VR is created. Now stop and start VR and check its restart priority, should be High.
- Enable cluster HA before VR is created. Now create some VM and verify that VR created must have High restart priority.
2017-04-11 10:48:42 +05:30
Rajani Karuturi 6548839417 Merge pull request #1944 from shapeblue/4.9-metrics-enhancement
CLOUDSTACK-9783: Improve metrics view performanceThis improves the metrics view feature by improving the rendering performance
of metrics view tables, by re-implementing the logic at the backend and data
served via APIs. In large environments, the older implementation would
make several API calls that increases both network and database load.

List of APIs introduced for improving the performance that re-implement the frontend logic at backend:

    listClustersMetrics
    listHostsMetrics
    listInfrastructure
    listStoragePoolsMetrics
    listVMsMetrics
    listVolumesMetrics
    listZonesMetrics

Pinging for review - @abhinandanprateek @DaanHoogland @borisstoyanov @karuturi @rashmidixit

Marvin test results:

=== TestName: test_list_clusters_metrics | Status : SUCCESS ===

=== TestName: test_list_hosts_metrics | Status : SUCCESS ===

=== TestName: test_list_infrastructure_metrics | Status : SUCCESS ===

=== TestName: test_list_pstorage_metrics | Status : SUCCESS ===

=== TestName: test_list_vms_metrics | Status : SUCCESS ===

=== TestName: test_list_volumes_metrics | Status : SUCCESS ===

=== TestName: test_list_zones_metrics | Status : SUCCESS ===

* pr/1944:
  CLOUDSTACK-9783: Improve metrics view performance

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-04-06 12:04:29 +05:30
Wei Zhou bf93b6313e CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number> 2017-03-20 07:32:31 +01:00
Rajani Karuturi 7b719c71fc Merge pull request #1856 from ustcweizhou/set-kvm-host-params
[4.9] CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agentThe router.aggregation.command.each.timeout in global configuration is only applied on new created KVM host.
For existing KVM host, changing the value will not be effective.
We need to propagate the configuration to existing host when cloudstack-agent is connected.

* pr/1856:
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:08:47 +05:30
Rohit Yadav 402253504e CLOUDSTACK-9783: Improve metrics view performance
This improves the metrics view feature by improving the rendering performance
of metrics view tables, by reimplementing the logic at the backend and data
served via APIs. In large environments, the older implementation would
make several API calls that increases both network and database load.

List of APIs introduced for improving the performance:

    listClustersMetrics
    listHostsMetrics
    listInfrastructure
    listStoragePoolsMetrics
    listVMsMetrics
    listVolumesMetrics
    listZonesMetrics

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-02-28 13:45:03 +05:30
Rajani Karuturi d9bd01266f Merge pull request #1829 from syed/hvm-volume-attach-restart-fix
CLOUDSTACK-9363: Fix HVM VM restart bug in XenServerHere is the longer description of the problem:

By default XenServer limits HVM guests to only 4 disks. Two of those are reserved for the ROOT disk (deviceId=0) and CD ROM (device ID=3) which means that we can only attach 2 data disks. This limit however is removed when Xentools is installed on the guest. The information that a guest has Xentools installed and can handle more than 4 disks is stored in the VM metadata on XenServer. When a VM is shut down, Cloudstack removes the VM and all the metadata associated with the VM from XenServer. Now, when you start the VM again, even if it has Xentools installed, it will default to only 4 attachable disks.

Now this problem manifests itself when you have a HVM VM and you stop and start it with more than 2 data disks attached. The VM fails to start and the only way to start the VM is to detach the extra disks and then reattach them after the VM start.

In this fix, I am removing the check which is done before creating a `VBD` which enforces this limit. This will not affect current workflow and will fix the HVM issue.

@koushik-das this is related to the "autodetect" feature that you introduced a while back (https://issues.apache.org/jira/browse/CLOUDSTACK-8826). I would love your review on this fix.

* pr/1829:
  Fix HVM VM restart bug in XenServer

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-28 05:42:59 +05:30
Anshul Gangwar ca84fd4ffd CLOUDSTACK-8663: Fixed various issues to allow VM snapshots and volume
snapshots to exist together

Reverting VM to disk only snapshot in Xenserver corrupts VM

Stale NFS secondary storage on XS leads to volume creation failure from snapshot
2017-02-15 12:56:39 +05:30
Rohit Yadav 8b6e96bca9 Updating pom.xml version numbers for release 4.9.3.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-06 10:40:15 +05:30
Rohit Yadav dfc39c1f08 Updating pom.xml version numbers for release 4.9.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-03 12:28:47 +05:30
Wei Zhou 714221234d CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent 2016-12-22 12:00:10 +01:00
Rohit Yadav a0e36b73ae Merge pull request #1711 from syed/xenserver7
XenServer 7 SupportThis PR adds support for XenServer 7. I have manually done the following tests
- Create a new cluster with XenServer7
- Add Primary storage: Should create an SR on XS7
- Add another XS7 host to the Pool
- Add host2 to Cloudstack
- Create VM1 from template
- Create VM2 from template
- Ping/SSH VM1 to VM2 and vice-versa
- Stop/Delete/Expunge VM2
- Create Data disk
- Attach it to VM1
- Create VM snaphsot of VM1
- Restore VM snapshot of VM1
- Delete VM snapshot of VM1
- Create Volume snapshot of Datadisk
- Create volume snapshot of Root disk
- Create new template from snapshot of root disk
- Create volume from snapshot of datadisk
- Detach datadisk volume
- Delete datadisk volume
- Aquire a public IP
- Create a static nat to VM1
- Live migrate VM1 while traffic on VM
- Delete VM1

* pr/1711:
  [CLOUDSTACK-9662] Add support for XenServer 7

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:18:57 +05:30
Syed eabf862ba9 [CLOUDSTACK-9662] Add support for XenServer 7 2016-12-21 16:58:10 -05:00
Sateesh Chodapuneedi 32e9e29a17 CLOUDSTACK-9673 Exception occured while creating the CPVM in the VmWare Setup over standard vSwitches
Issue
====
Exception occured while creating the CPVM in the VmWare Setup using standard vswitches.
StartCommand failed due to Exception: com.vmware.vim25.AlreadyExists
message: [] com.vmware.vim25.AlreadyExistsFaultMsg: The specified key, name, or identifier already exists

Fix
===
Ensure synchronization while attempting to create port group such that simultaneous attempts are not made with same port group name on same ESXi host.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
2016-12-22 01:35:10 +05:30
Rohit Yadav 5e19e64f2f Updating pom.xml version numbers for release 4.9.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-16 20:48:16 +05:30
Syed 966814b07d Fix HVM VM restart bug in XenServer 2016-12-15 14:12:59 -05:00
Rohit Yadav af2679959b Updating pom.xml version numbers for release 4.9.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-10 08:38:03 +05:30
Murali Reddy 6749785cab CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly
-when processing static nat rule, add a mangle table rule, to mark the traffic
   from the guest vm when it has associated static nat rule so that traffic gets
   routed using the route tabe of the device which has public ip associated

  -fix the case where nic_device_id is empty when ip is getting disassociated
   resulting in empty deviceid in ips.json

  -add utility methods in CsRule, and CsRoute to add 'ip rule' and 'ip route' rules respectivley

  -ensure traffic from all public interfaces are connection marked with device number, and restored
   for the reverse traffic. use the connection marked number to do device specific routing table lookup
   fill the device specific routing table with default route

  -component tests for testing multiple public interfaces of VR
2016-12-07 14:33:24 +05:30
Rohit Yadav abfcd5b95f CLOUDSTACK-9632: Upgrade bouncy castle to version 1.55
- Upgrades Maven dependency version to v1.55
- Fixes bountycastle usages and issues
- Adds timeout to jetty/annotation scanning
- Fixes servlet issue, uses servlet 3.1.0
- Downgrade javassist used by reflections to fix annotation process errors
- Make console-proxy-rdp bc dependency same as rest of the codebase
- Picks up PR #1510 by Daan

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-06 17:24:24 +05:30
Rohit Yadav 48b28f7d6e
Merge branch '4.8' into 4.9 2016-12-05 15:32:03 +05:30
Rohit Yadav 20aea27dc0
Merge pull request #1765 from shapeblue/CLOUDSTACK-9586
Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary storeThe race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously.

Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.

* pr/1765:
  Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary store The race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously. Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-05 15:31:25 +05:30
Rohit Yadav 8d14e8e8b5 Merge pull request #1729 from shapeblue/vmware-memleak-fix
CLOUDSTACK-9564: Fix memory leaks in VmwareContextPoolIn a recent management server crash, it was found that the largest contributor
to memory leak was in VmwareContextPool where a registry is held (arraylist)
that grows indefinitely. The list itself is not used anywhere or consumed. There
exists a hashmap (pool) that returns a list of contexts for existing poolkey
(address/username) that is used instead.

This fixes the issue by removing the arraylist registry, and limiting the
length of the context list for a given poolkey.

@blueorangutan package

* pr/1729:
  CLOUDSTACK-9564: Fix memory leaks in VmwareContextPool

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-05 15:15:58 +05:30
Abhinandan Prateek ba32ac1a7c Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary store
The race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously.
Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
2016-12-02 13:37:47 +05:30
Rohit Yadav 90a3d97c5e CLOUDSTACK-9564: Fix memory leaks in VmwareContextPool
In a recent management server crash, it was found that the largest contributor
to memory leak was in VmwareContextPool where a registry is held (arraylist)
that grows indefinitely. The list itself is not used anywhere or consumed. There
exists a hashmap (pool) that returns a list of contexts for existing poolkey
(address/username) that is used instead.

This fixes the issue by removing the arraylist registry, and limiting the
length of the context list for a given poolkey.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-01 15:27:29 +05:30
Suresh Kumar Anaparti 309da6a57f CLOUDSTACK-8854: Apple Mac OS/X VM get created without USB controller in ESXi hypervisors 2016-12-01 14:32:18 +05:30
Rohit Yadav 50f80cc2a0
Merge branch '4.8' into 4.9 2016-11-25 13:03:04 +05:30
Rohit Yadav 020606ec31 Merge pull request #1738 from SudharmaJain/cs-9566
CLOUDSTACK-9566 instance-id metadata for baremetal VM returns IDThere is difference in instance-id metadata across baremetal and other hypervisors.

On Baremetal
[root@ip-172-17-0-144 ~]# curl http://8.37.203.221/latest/meta-data/instance-id
6021

on Xen
[root@ip-172-17-2-103 ~]# curl http://172.17.0.252/latest/meta-data/instance-id
cbeb517a-e833-4a0c-b1e8-9ed70200fbbf

In both cases it should be vm's uuid.

* pr/1738:
  CLOUDSTACK-9566 instance-id metadata for baremetal VM returns ID

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-25 12:52:10 +05:30
Rohit Yadav 1f2184800b Merge pull request #1681 from murali-reddy/router_eth_device_index
CLOUDSTACK-9491: incorrect parsing of device list to find ethernet index of plugged NICIn VmwareResource, findRouterEthDeviceIndex() method find ethernet interface index given
the mac address. This method is used, once a nic is plugged to determine ethernet interface.
"/proc/sys/net/ipv4/conf" from the VR and looped through the devices to find the right
ethernet interface. Howver current logic read it once, and loops through the device list.
Its observerd device may not show up '/proc/sys/net/ipv4/conf' immediatly once NIC is plugged
in the VM from vCenter.

Fix ensured, while waiting for 15 sec in the loop, read the latest content from /proc/sys/net/ipv4/conf
, so that right device list is processed.

Manual tested VPC scenarios of adding new tiers which uses findRouterEthDeviceIndex, to find the guest/public network ethernet index.

* pr/1681:
  CLOUDSTACK-9491: incorrect parsing of device list to find ethernet index of plugged NIC

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-23 14:28:12 +05:30
Rohit Yadav 55b918076f
Merge branch '4.8' into 4.9 2016-11-23 13:50:15 +05:30
Rohit Yadav ff616e700b Merge pull request #1745 from shapeblue/CLOUDSTACK-9503
CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.

* pr/1745:
  CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-23 13:41:52 +05:30
Koushik Das bdc806e315 Merge release branch 4.8 to 4.9
* 4.8:
  CLOUDSTACK-9410: Data Disk shown as detached in XS
2016-11-21 11:12:09 +05:30
Koushik Das 7f051bd790 Signed-off-by: Koushik Das <koushik@apache.org> 2016-11-21 10:45:04 +05:30
John Burwell 20b43767d7 Merge pull request #1676 from nvazquez/dstemplates49
CLOUDSTACK-9502: DS template copies dont get deleted in VMware ESXi with multiple clusters and zone wide storage (include CLOUDSTACK-9386 into 4.9 release branch)Include #1560 into 4.9 release branch

* pr/1676:
  CLOUDSTACK-9502: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storage

Signed-off-by: John Burwell <meaux@cockamamy.net>
2016-11-16 22:15:50 -05:00
John Burwell 74639b305f Merge pull request #1677 from nvazquez/vmsnapshot12min
CLOUDSTACK-8830 - [Vmware] VM snapshot fails for 12 min after instance creation (Targeted for 4.9)Continuing work by @maneesha-p in #798

This closes #798

* pr/1677:
  CLOUDSTACK-8830: Fix for vm snapshots in Vmware, could not create vm snapshot until 12 minutes after vm creation due to vCenter sent null name on snpashot recent task

Signed-off-by: John Burwell <meaux@cockamamy.net>
2016-11-16 09:43:54 -05:00
Rohit Yadav e2e5cdb74f
Merge branch '4.8' into 4.9 2016-11-03 16:52:08 +05:30
Abhinandan Prateek 83b5a8b2b2 CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration. 2016-11-01 16:14:23 +05:30
Murali Reddy b449351a9f CLOUDSTACK-9491: incorrect parsing of device list to find ethernet index of plugged NIC
In VmwareResource, findRouterEthDeviceIndex() method find ethernet interface index given
  the mac address. This method is used, once a nic is plugged to determine ethernet interface.
  "/proc/sys/net/ipv4/conf" from the VR and looped through the devices to find the right
  ethernet interface. However current logic read it once, and loops through the device list.
  Its observerd device may not show up '/proc/sys/net/ipv4/conf' immediatly once NIC is plugged
  in the VM from vCenter.Fix ensured, while waiting for 15 sec in the loop, read the latest
  content from /proc/sys/net/ipv4/conf, so that right device list is processed.
2016-10-28 17:50:36 +05:30
Sudharma Jain 759a92d251 CLOUDSTACK-9566 instance-id metadata for baremetal VM returns ID 2016-10-27 13:50:39 +05:30
subhash yedugundla 38c56bdf44 CLOUDSTACK-9410: Data Disk shown as detached in XS 2016-10-25 14:57:33 +05:30
nvazquez 94222b1356 CLOUDSTACK-8830: Fix for vm snapshots in Vmware, could not create vm snapshot until 12 minutes after vm creation due to vCenter sent null name on snpashot recent task 2016-10-24 13:26:45 -03:00
subhash yedugundla 725ddb82a1 CLOUDSTACK-9554 Juniper Contrail plug-in is publishing events to wrong message bus 2016-10-21 12:14:21 +05:30
nvazquez 4104cea300 CLOUDSTACK-9502: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storage 2016-09-19 07:49:31 -07:00
John Burwell 8d11511b1f Adds support for four position versions and optional db upgrades
Often, patch and security releases do not require schema migrations or
data migrations.  However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break.  With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path.  This approach allows the upgrade to proceed and
for the database to properly reflect the installed version.  This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.

Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes.  The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers.  This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers.   It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.

  * Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
  a version that was not explicitly specified.  It determines the
  releases the first release before it with database migrations and uses
  that list as the basis for the list for version being calculated.  A
  noop upgrade is then added to the list which causes no schema changes
  or data migrations, but will update the database to the version.
  * Adds unit tests for the upgrade path calculation logic in
  DatabaseUpgradeChecker
  * Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
  versions of this patch
  * Introduces the CloudStackVersion value object which parses and
  compares three (3) and four (4) position version numbers.  This class
  is intended to replace com.cloud.maint.Version.
  * Adds the junit-dataprovider dependency -- allowing test data to be
  concisely generated separately from the execution of a test case.
  Used extensively in the CloudStackVersionTest.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 13:32:32 +05:30
Rohit Yadav fa3fe7bb05 Merge pull request #1634 from shapeblue/patchviasocket-49-py26fix
[blocker] CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpmsThe patchviasocket script was rewritten in Python from PR #1533 and made
assumptions that Python 2.7 would be available. In case of CentOS, python 2.7
may not be available or installed. This change ensures that python-argparse
is installed which is used by this script.

/cc @wido @sverrirab @karuturi @jburwell

@blueorangutan package

* pr/1634:
  CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpms

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-24 12:14:02 +05:30
Rohit Yadav f13c224da1 Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:39 +05:30
Will Stevens 740bd45be6 Updating pom.xml version numbers for release 4.8.2-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-15 10:31:09 -04:00
Rohit Yadav 7b8ba24c64 CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpms
The patchviasocket script was rewritten in Python from PR #1533 and made
assumptions that Python 2.7 would be available. In case of CentOS, python 2.7
may not be available or installed. This change ensures that python-argparse
is installed which is used by this script.

Expose cmd error in the logs when patch command fails.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-10 16:25:17 +05:30
Will Stevens a63db21d16 Updating pom.xml version numbers for release 4.8.1
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-08 10:06:21 -04:00
Will Stevens 227ff3884d Updating pom.xml version numbers for release 4.9.0
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-25 16:56:04 -04:00
Aaron Hurt c8fce3ff31 improve logging readability 2016-07-11 12:05:06 -05:00
Aaron Hurt 44491448e3 Cleanup rbd contexts and improve exception logging
We noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up.  This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.

In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
2016-07-08 23:13:33 -05:00
Will Stevens f7f23ec720 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
  Added ASF license to unit test file
  Added unit test to verify ordering
  Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
2016-06-28 11:21:04 -04:00
Will Stevens 142f07d77f Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
  Added ASF license to unit test file
  Added unit test to verify ordering
  Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
2016-06-28 11:20:16 -04:00
Anshul Gangwar 458264aac8 CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage 2016-06-24 09:31:20 +05:30
Will Stevens 3c800b4bfe Merge pull request #1518 from nvazquez/testnfs
CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts## Description
JIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9368
This pull request address a problem introduced in #1361 in which NFS version couldn't be changed after hosts resources were configured on startup (for hosts using `VmwareResource`), and as host parameters didn't include `nfs.version` key, it was set `null`.

## Proposed solution
In this proposed solution `nfsVersion` would be passed in `NfsTO` through `CopyCommand` to `VmwareResource`, who will check if NFS version is still configured or not. If not, it will use the one sent in the command and will set it to its storage processor and storage handler. After those setups, it will proceed executing command.

* pr/1518:
  CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:05:07 -04:00
Will Stevens b5606dcce8 Merge pull request #1553 from nlivens/mysql_driver_issue
Dynamically load drivers before creating our DB connectionsSolution to the mailing thread titled "MySQL : No suitable driver found for jdbc:mysql".
It doesn't harm that we explicitely load the MySQL driver, and for those which would use a commons-dbcp version < 1.4 this would fix it as well. Since JDBC 4.0, the JDBC driver can auto-register itself, but for some weird cases (like ours), it's not working. Therefore we need to explicitly load the JDBC driver.

* pr/1553:
  Dynamic loading of DB driver + support for other DB providers

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:00:42 -04:00
Will Stevens 05a7beccfe Merge pull request #1494 from nlivens/master_nuage
Remodeling of Nuage VSP Plugin + CLOUDSTACK-9294Hi all,

We've remodeled the Nuage VSP plugin to use the same model as VMWare is using (non-OSS). Before, we had a runtime dependency to the Nuage Client, this has been changed to a compile-time dependency instead because of multiple reasons (build management, readability, maintainability, ...)

We've adapted the code so it now uses model objects defined in the Nuage client instead of passing a list of parameters to the Nuage client. This is a lot more readable, and a lot more maintainable.

I've had a chat with @DaanHoogland about this approach, and he told me that ACS is trying to move away from the whole non-OSS approach. We're looking into the Juniper approach, we would set up a custom maven repository which would host the required dependencies for the Nuage VSP plugin.

Any remarks or suggestions are always welcome :)

* pr/1494:
  Nuage VSP : Extending Marvin test coverage
  Nuage VSP : Fix for NPE while cleaning up account when there are still resources belonging to that account
  CLOUDSTACK-9294 : Make sure to remove VR from VSD when removing the VPC
  CLOUDSTACK-9242 : Remodel Nuage VSP plugin

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:58:10 -04:00
Will Stevens b03a629c6a Merge pull request #1533 from greenqloud/pr-patchviasocket-convert-to-python
Convert patchviasocket to python (removes perl dependency for KVM agent)As requested here: https://github.com/apache/cloudstack/pull/1495

No scripts are using perl so that install requirement can be removed.
The new scripts are using standard python packages only.
Includes extensive unit test.
Note: perl-modules requirement is missing (fixed in mentioned PR) so do not merge that onto master.

* pr/1533:
  Revert "Add perl-modules as install dependency for cloudstack-agent"
  patchviasocket improve error handling
  Convert patchviasocket to python (removes perl dependency for KVM agent)

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:57:08 -04:00
Will Stevens 678b28f273 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:54:23 -04:00
Will Stevens b80696cbc0 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:53:42 -04:00
Nick Livens d862a09771 Nuage VSP : Fix for NPE while cleaning up account when there are still resources belonging to that account 2016-05-24 11:12:19 +02:00
Nick Livens 4b6179940d CLOUDSTACK-9294 : Make sure to remove VR from VSD when removing the VPC 2016-05-24 11:12:19 +02:00
Nick Livens b0c27e48d7 CLOUDSTACK-9242 : Remodel Nuage VSP plugin 2016-05-24 11:12:19 +02:00
Nick Livens 02c5d44f0b Dynamic loading of DB driver + support for other DB providers 2016-05-23 15:50:28 +02:00
Sverrir A. Berg 0acd3c12a2 Convert patchviasocket to python (removes perl dependency for KVM agent)
As requested here: https://github.com/apache/cloudstack/pull/1495

No scripts are using perl so that install requirement can be removed.
The new scripts are using standard python packages only.
Includes extensive unit test.
2016-05-20 15:42:34 +00:00
nvazquez 2d2819974e CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts 2016-05-20 07:31:16 -07:00
Will Stevens 82b702dc9a Merge pull request #1403 from mike-tutkowski/xs-snapshots
Taking fast and efficient volume snapshots with XenServer (and your storage provider)A XenServer storage repository (SR) and virtual disk image (VDI) each have UUIDs that are immutable.

This poses a problem for SAN snapshots, if you intend on mounting the underlying snapshot SR alongside the source SR (duplicate UUIDs).

VMware has a solution for this called re-signaturing (so, in other words, the snapshot UUIDs can be changed).

This PR only deals with the CloudStack side of things, but it works in concert with a new XenServer storage manager created by CloudOps (this storage manager enables re-signaturing of XenServer SR and VDI UUIDs).

I have written Marvin integration tests to go along with this, but cannot yet check those into the CloudStack repo as they rely on SolidFire hardware.

If anyone would like to see these integration tests, please let me know.

JIRA ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-9281

Here's a video I made that shows this feature in action:

https://www.youtube.com/watch?v=YQ3pBeL-WaA&list=PLqOXKM0Bt13DFnQnwUx8ZtJzoyDV0Uuye&index=13

* pr/1403:
  Faster logic to see if a cluster supports resigning
  Support for backend snapshots with XenServer

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-20 08:33:07 -04:00
Wei Zhou 976b3b7df4 CLOUDSTACK-6928: fix issue disk I/O throttling not applied 2016-05-19 11:26:43 +02:00
Will Stevens 7be8585787 Merge pull request #1520 from NuxRo/kvm-numa-cpu-reporting
CPU socket count reporting correctionCPU socket count reporting correction
From https://github.com/MissionCriticalCloud/cosmic-plugin-hypervisor-kvm/pull/16

* pr/1520:
  Remove empty spaces causing the build to fail
  CPU socket count reporting correction

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-13 17:02:10 -04:00
Mike Tutkowski 2bd035d199 Support for backend snapshots with XenServer 2016-05-13 01:02:04 -06:00
Rohit Yadav ae0f169123 CLOUDSTACK-9299: Fix test failures on CI
- Fixes oobm integration test to skip if known ipmitool bug is hit
- Fixes ProcessTest unit test case to use sleep
- Removes redundant unit test that covers code in ProcessTest

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-13 09:47:32 +05:30
Will Stevens 8c3722d953 Merge pull request #1444 from rafaelweingartner/workAroundPR780
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMThis PR introduces the changes proposed in PR #780 with some work to make the code null safe.

During this PR, I have also removed some unused code.

* pr/1444:
  Removed unnecessary check when creating the “userVmResponse” object.
  Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
  CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:49:14 -04:00
Will Stevens 20cf8b23c0 Merge pull request #1453 from GabrielBrascher/brascher-removeUnusedClasses2
Remove classes with no referencesI used UCDetector (http://www.ucdetector.org/) as a plugin for Eclipse.  With this tool, I discovered a lot of code without any reference (variables, methods and classes).

Following the work that was done at [https://github.com/apache/cloudstack/pull/1448]; this pull request had the goal of removing some of these classes. To check if I wasn't missing anything I searched for any file that could reference some of those classes. As I haven't found any way of these classes being used, they were removed. Note that some of them I found other references, but references such as commented lines or tests, nothing that could indicate their use (as XML files configuring beans or another class instantiating an object with "new").

Waiting for tests. Please tell me if I am missing something.

Removed Classes:
- org.apache.cloudstack.framework.jobs.JobCancellationException (**Note:** removed
variable JobCancellationException in com.cloud.utils.SerialVersionUID)
- org.apache.cloudstack.ldap.NoSuchLdapUserException (**Note:** removed test file
/cloud-plugin-user-authenticator-ldap/test/groovy/org/apache/cloudstack/ldap/NoSuchLdapUserExceptionSpec.groovy)
- com.cloud.agent.api.storage.CreateVolumeOVAAnswer
- com.cloud.exception.MissingParameterValueException
- org.apache.cloudstack.api.response.StatusResponse
- org.apache.cloudstack.api.response.VolumeDetailResponse
- org.apache.cloudstack.api.response.UpgradeVmResponse
- org.apache.cloudstack.api.response.AddIpToVmNicResponse
- org.apache.cloudstack.api.response.TemplateZoneResponse (**Note:** at
org.apache.cloudstack.api.response.TemplateResponse, there is this
comment "To avoid breaking backwards compatibility, we still treat a
template at different zones as different templates, so not embedding
template_zone information in this TemplateZoneResponse set. `private
Set<TemplateZoneResponse> zones;`" but right now it is not used)
- org.apache.cloudstack.api.response.NicDetailResponse

* pr/1453:
  Removed classes with no reference

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:48:03 -04:00
Will Stevens 688522ecd4 Merge pull request #1385 from DaanHoogland/CLOUDSTACK-9265
CLOUDSTACK-9265 cleanup around httpclient versionssome cleanup done
- replaced HttpStatus from org.apache.commons.httpclient with that from org.apache.http
- removed unthrown HttpException
- left auto reformat in place

* pr/1385:
  CLOUDSTACK-9265 cleanup around httpclient versions

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:11:28 -04:00
Will Stevens 7e41747c33 Merge pull request #816 from mike-tutkowski/addremovehosts2
Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

This PR addresses the following JIRA ticket:

https://issues.apache.org/jira/browse/CLOUDSTACK-8813

The problem is that there needs to be notifications sent when a host is added to, about to be removed from, and removed from a cluster.

Such notifications can be used for many purposes. For example, it can allow storage plug-ins to update ACLs on their storage systems. Also, it can allow us to clean up IQNs from ESXi hosts that are no longer needed.

* pr/816:
  CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:07:36 -04:00
Rohit Yadav 07564469e9 CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30