Commit Graph

9623 Commits

Author SHA1 Message Date
Will Stevens 88cd182272 Forward merged 4.8 to master and fixed conflicts in #1331 2016-05-19 12:08:08 -04:00
Will Stevens 8f330b0b92 Merge release branch 4.7 to 4.8
* 4.7:
  Fix Sync of template.properties in Swift
  Configure rVPC for router.redundant.vrrp.interval advert_int setting
  Have rVPCs use the router.redundant.vrrp.interval setting
  Resolve conflict as forceencap is already in master
  Split the cidr lists so we won't hit the iptables-resture limits
  Check the existence of 'forceencap' parameter before use
  Do not load previous firewall rules as we replace everyhing anyway
  Wait for dnsmasq to finish restart
  Remove duplicate spaces, and thus duplicate rules.
  Restore iptables at once using iptables-restore instead of calling iptables numerous times
  Add iptables copnversion script.
2016-05-18 15:54:32 -04:00
Will Stevens 06e52e4f34 Merge pull request #1331 from syed/swift-restart-fix
Fix Sync of template.properties in SwiftWhen using Swift as a secondary storage, we create a template.properties file which stores the metadata about the template. Currently the data which is present in the file is incorrect which leads to templates becoming unavailable after they are downloaded. This fix makes sure that the template.properties has the correct "path" set so that templates are available.

I've also done a bit of cleanup and made the code bit more clean.

* pr/1331:
  Fix Sync of template.properties in Swift

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-18 15:53:33 -04:00
Syed f5ac8ddded Fix Sync of template.properties in Swift 2016-05-13 22:08:16 +00:00
Will Stevens f2f895a5c2 Merge pull request #1537 from PaulAngus/master
Remove extraneous log directory and add catalina.out log rotationrebased and squashed
replacement for PR: https://github.com/apache/cloudstack/pull/1087

Remove extraneous log directory (/var/log/cloudstack-management)
and adds catalina.out log rotation

* pr/1537:
  Remove extraneous log directory and add catalina.out log rotation

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-13 17:04:18 -04:00
Remi Bergsma 2bfb12cba9 Have rVPCs use the router.redundant.vrrp.interval setting
It defaults to 1, which is hardcoded in the template:
./cosmic/cosmic-core/systemvm/patches/debian/config/opt/cloud/templates/keepalived.conf.templ

As non-VPC redundant routers use this setting, I think it makes sense to use it for rVPCs as well.

We also need a change to pickup the cmd_line parameter and use it in the Python code that configures the router.
2016-05-13 14:36:23 +02:00
Will Stevens 8c3722d953 Merge pull request #1444 from rafaelweingartner/workAroundPR780
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMThis PR introduces the changes proposed in PR #780 with some work to make the code null safe.

During this PR, I have also removed some unused code.

* pr/1444:
  Removed unnecessary check when creating the “userVmResponse” object.
  Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
  CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:49:14 -04:00
Will Stevens 7e41747c33 Merge pull request #816 from mike-tutkowski/addremovehosts2
Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

This PR addresses the following JIRA ticket:

https://issues.apache.org/jira/browse/CLOUDSTACK-8813

The problem is that there needs to be notifications sent when a host is added to, about to be removed from, and removed from a cluster.

Such notifications can be used for many purposes. For example, it can allow storage plug-ins to update ACLs on their storage systems. Also, it can allow us to clean up IQNs from ESXi hosts that are no longer needed.

* pr/816:
  CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:07:36 -04:00
Will Stevens 143265c49f Merge pull request #1502 from shapeblue/outofband-master
CLOUDSTACK-9299: Out-of-band Management for CloudStackSupport access to a hosts out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature, please install `ipmitool` (using yum/apt/brew) and `ipmisim`:
https://pypi.python.org/pypi/ipmisim

The default ipmitool location is assumed in /usr/bin, if this is different in your env please fix the global setting, see FS for details on various global settings.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

/cc @jburwell @swill @abhinandanprateek @murali-reddy @borisstoyanov

* pr/1502:
  maven: ignore utils/testsmallfileinactive for rat checking
  CLOUDSTACK-9378: Fix for #1497
  HypervisorUtilsTest: increate timeout to 8seconds
  travis: Use patched version of ipmitool for tests
  CLOUDSTACK-9299: Out-of-band Management for CloudStack

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:06:13 -04:00
Will Stevens 103d62ee02 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:04:53 -04:00
Will Stevens dbbc37ecc7 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:03:57 -04:00
Will Stevens 3fab75772f Merge pull request #1474 from remibergsma/47_private_gw_initial_config
Handle private gateways more reliablyWhen initialising a VPC router we need to know which IP/device corresponds to a private gateway. This is to solve a problem when stop/starting a VPC router (which gets the private gateway config as a guest network and as a result breaks the functionality). You read it right, the private gateway is sent as type=guest after reboot and type=public initially.

Before this change, you could add a private gw to a running router but you couldn't restart it (it would mix up the tiers). Now the private gateway is detected properly and it works just fine.

Booting without private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.0.42",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "None",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

Booting with private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.2.227",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "10.201.10.1",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

And:
```
cat cmdline
vpccidr=10.0.0.0/24 domain=cs2cloud dns1=8.8.8.8 privategateway=10.201.10.1 template=domP name=r-167-VM eth0ip=169.254.2.227 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w baremetalnotificationapikey=V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw host=192.168.22.61 port=8080
```

Logs:
```
2016-02-24 20:08:45,723 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-458d4c52 job-1402/job-1403 ctx-d5355fca) (logid:5772906c) Set privategateway field in cmd_line.json to 10.201.10.1
```

* pr/1474:
  Handle private gateways more reliably
  Add private gateway IP to router initialization config

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:02:55 -04:00
Will Stevens 309a60ea71 Merge pull request #1483 from remibergsma/pr1413-wilder-47
CLOUDSTACK-9287 - Fix unique mac address per rVPC routerThis is work by @wilderrodrigues, see PR #1413 It contains important fixes and I think it needs to be included so I send the PR again.

* pr/1483:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:00:40 -04:00
Paul Angus 11e3582fdf Remove extraneous log directory and add catalina.out log rotation 2016-05-12 08:41:39 +01:00
Rohit Yadav 07564469e9 CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Mike Tutkowski dad9e5d868 CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster 2016-05-11 08:02:46 -06:00
Rohit Yadav 003b97bcff maven: Fix jstl version usage
This would fix regression from recent mvn version changes. Without this
patch users get redirected to error.jsp as jstl-1.2 jar is not installed

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:58:27 +05:30
Rohit Yadav afcbd270b2 CLOUDSTACK-8562: Deprecate commands.properties
- Removes commands.properties file
- Fixes apidocs and marvin to be independent of commands.properties usage
- Removes bundling of commands.properties in deb/rpm packaging
- Removes file references across codebase

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:45:19 +05:30
Rohit Yadav 4347776ac6 CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
This feature allows root administrators to define new roles and associate API
permissions to them.

A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack

DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
  current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
  Admin, Domain Admin and User) which maintains this association by requiring
  all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
  removal of roles and/or modifications of permissions, without the need
  of restarting management server(s)

Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
  deployments will continue to use the older static role based api access checker
  with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
  roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
  to add existing set of permissions to the default roles. cloud.account
  will have a new role_id column which will be populated based on default roles
  as well

Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:45:19 +05:30
Rohit Yadav 95abb6efc3 CLOUDSTACK-9361: Centrally handle API validations
Validate API arguments based on annotations. Introduces:
- NotNullOrEmpty: for doing null and empty string checks
- PositiveNumber: number > 0 (natural number)

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:40:22 +05:30
Will Stevens 456680dbb2 Merge pull request #880 from SudharmaJain/cs-8901
CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads The thread pool was hardcoded to use 8 threads,
com.cloud.template.TemplateManagerImpl.configure(String, Map<String, Object>):
_preloadExecutor = Executors.newFixedThreadPool(8, new NamedThreadFactory("Template-Preloader"));

Added the change to pick threadpool size from configuration.

* pr/880:
  CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-11 00:08:45 -04:00
Will Stevens de582a41c0 Merge pull request #1497 from nvazquez/addidsparam
CLOUDSTACK-9351: Add ids parameter to resource listing API calls## General behaviour
A new parameter is added in each method, its type a list of IDs of the entity, it will be mutually exclusive with id. (Similar to <code>id</code> and <code>ids</code> parameters in <code>listVirtualMachines</code> method)

### API Methods affected
* <code>listTemplates</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVolumes</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listSnapshots</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVMSnapshots</code>: new parameter **<code>vmsnapshotids</code>**, mutually exclusive with <code>vmsnapshotid</code>

* pr/1497:
  CLOUDSTACK-9351: Add marvin test and add it to travis file
  CLOUDSTACK-9351: Add ids parameter to resource listing API calls

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:49:55 -04:00
Will Stevens fa3bce5a83 Merge pull request #1496 from shapeblue/kvm-ha
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage- KVM-HA- Fix CheckOnHost for Local storage
 - Also skip HA on VMs that are using local storage

* pr/1496:
  CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage     - Also skip HA on VMs that are using local storage

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:49:04 -04:00
Will Stevens 8a3fd10615 Merge pull request #1124 from rafaelweingartner/CID-1338387
CID-1338387: Deletion of method endPointSelector.selectHypervisorHostFollowing the discussions and analysis presented on PR #1056 create by @DaanHoogland
This PR is intended to push those changes that were discussed there regarding the of endPointSelector.selectHypervisorHost method.

* pr/1124:
  Deletion of method endPointSelector.selectHypervisorHost

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:47:50 -04:00
weingartner 417d9a5d80 Removed unnecessary check when creating the “userVmResponse” object. 2016-05-06 15:16:37 -03:00
nvazquez baa8610061 CLOUDSTACK-9351: Add ids parameter to resource listing API calls 2016-05-06 14:42:42 -03:00
Rohit Yadav 4d57ec04ac Merge branch '4.8'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-03 08:22:37 +05:30
Rohit Yadav 126a039eb8 Merge branch '4.7' into 4.8 2016-05-03 08:20:27 +05:30
Will Stevens 62d9f444e3 Merge pull request #1321 from nitin-maharana/CloudStack-Nitin5_4.7
CLOUDSTACK-8847: ListServiceOfferings is returning incompatible tagged offerings when called with VM idWhen calling listServiceOfferings with VM id as parameter. It is returning incompatible tagged offerings. It should only list all compatible tagged offerings. Compatible means the new service offering should contain all the tags of the existing service offering(Existing offering SUBSET of new offering). If that is the case It should list in the result and can be upgraded to that offering.

* pr/1321:
  CLOUDSTACK-8847: ListServiceOfferings is returning incompatible tagged offerings when called with VM id

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-02 16:59:57 -04:00
Will Stevens 9654c990c8 Merge pull request #1365 from shapeblue/4.7-vmware-diskchain
[4.7] vmware: improve support for disks- Improve disk chain usage while attaching, migrating disks
- Gets root disk controller based diskDeviceBusName from volume's chain info

* pr/1365:
  vmware: improve support for disks

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-02 16:58:41 -04:00
Maneesha.P 7ad3c5e834 CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware. 2016-04-29 14:06:49 -03:00
Abhinandan Prateek 3b89cbe733 CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage
- Also skip HA on VMs that are using local storage
2016-04-29 12:38:33 +05:30
Will Stevens 12d8e53dd5 Merge pull request #1454 from shapeblue/host-maint
CLOUDSTACK-9323: Fix cancel host maintenance canFix cancel host maintenance so that if maintenance is cancelled the host come back to normal state gracefully.

Added marvin tests for host maintennace.

* pr/1454:
  CLOUDSTACK-9323: Fix Cancel maintenance so that if maintenance is cancelled the host come back to normal state gracefully. Added marvin tests for host maintennace.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-28 16:05:10 -04:00
Will Stevens a5ee4432e7 Merge release branch 4.8 to master
* 4.8:
  Removed sleeps and used validateList as requested.
  Added required_hardware="false" attr above test_02_root_volume_attach_detach
  Modified test_volumes.py to include a hypervisor test for root attach/detach testing
  Let hypervisor type KVM and Simulator detach root volumes. Updated test_volumes.py to include a test for detaching and reattaching a root volume from a vm. I also had to update base.py to allow attach_volume to have the parameter deviceid to be passed as needed.
2016-04-28 16:04:20 -04:00
Will Stevens d11194abec Merge release branch 4.7 to 4.8
* 4.7:
  Removed sleeps and used validateList as requested.
  Added required_hardware="false" attr above test_02_root_volume_attach_detach
  Modified test_volumes.py to include a hypervisor test for root attach/detach testing
  Let hypervisor type KVM and Simulator detach root volumes. Updated test_volumes.py to include a test for detaching and reattaching a root volume from a vm. I also had to update base.py to allow attach_volume to have the parameter deviceid to be passed as needed.
2016-04-28 16:03:46 -04:00
Will Stevens 1979f42b2d Merge pull request #1500 from myENA/KVM_root_detach
CLOUDSTACK-9349: Enable root disk detach for KVM with new Marvin testsThis PR addresses the KVM detach/attach ROOT disks from VMs (CLOUDSTACK-9349).  In short, this allows the KVM Hypervisor, and I added the Simulator as a valid hypervisor for ease of development and testing of marvin, to detach a root volume and the reattach a root volume using the deviceid=0 flag to the attachVolume API.  I have also written a marvin integration test that verifies this feature works for both KVM and the Simulator.

Below is the marvin results files of the full marvin test_volumes.py.  All tests pass, including the new root detach/attach, on our KVM lab running with the patches in this PR.

[test_volumes_KIR4G3.zip](https://github.com/apache/cloudstack/files/223799/test_volumes_KIR4G3.zip)

* pr/1500:
  Removed sleeps and used validateList as requested.
  Added required_hardware="false" attr above test_02_root_volume_attach_detach
  Modified test_volumes.py to include a hypervisor test for root attach/detach testing
  Let hypervisor type KVM and Simulator detach root volumes. Updated test_volumes.py to include a test for detaching and reattaching a root volume from a vm. I also had to update base.py to allow attach_volume to have the parameter deviceid to be passed as needed.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-28 16:02:54 -04:00
Will Stevens 91ce3b9ffb Merge pull request #1522 from myENA/logjobid_master
Log asynchronous responses in the api logCurrently API responses for synchronous calls are logged, but asynchronous call responses are not.  This pull request makes a minor modification that logs the response including the JobId of all asynchronous requests.

As an example, here is what a stopVirtualMachine request looked like in the logs:

    2016-04-27 10:43:11,084 INFO  [a.c.c.a.ApiServer] (catalina-exec-3:ctx-37d9f693 ctx-d2368de3) (logid:3a0fad97) (userId=2 accountId=2 sessionId=AF8B1F726ACB5C3A637B8B300AA218A7) 10.103.0.207 -- GET command=stopVirtualMachine&id=f63b6fcc-e0b0-480f-8f7a-cba329634ba1&forced=false&response=json&_=1461771791036 200

After this modification, here is what the logs look like:

    2016-04-27 13:37:11,338 INFO  [a.c.c.a.ApiServer] (catalina-exec-6:ctx-915b5c84 ctx-a03152fa) (logid:66249df0) (userId=2 accountId=2 sessionId=9EF127EED5CA6E74797DFE487D980FAF) 10.103.0.207 -- GET command=stopVirtualMachine&id=f63b6fcc-e0b0-480f-8f7a-cba329634ba1&forced=false&response=json&_=1461782231194 200 {"stopvirtualmachineresponse":{"jobid":"5b9f4a9b-eabe-4fa4-849d-3d004bb65634"}}

* pr/1522:
  Log responses from asynchronous api commands

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-28 15:59:57 -04:00
Will Stevens 4db5e48f80 Merge pull request #1510 from shapeblue/4.9-mvn-version-safeupgradeonly
4.9 mvn version safeupgradeonlyUpgrades maven dependencies versions that can be safely upgraded without breaking console-proxy/crypto usage.

Bisected changes from: https://github.com/apache/cloudstack/pull/1397

cc @swill @DaanHoogland

* pr/1510:
  maven: fix dependency version support by JDK7
  further maven dependency updates from Daan
  framework/quota: fix checkstyle issue
  maven: Upgrade dependency versions

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-28 15:59:17 -04:00
Nathan Johnson 7778e24bf7 Log responses from asynchronous api commands
This is mainly useful so that we can see the job ids in the logs
whenever the job is created via the API.
2016-04-27 19:43:57 -05:00
Will Stevens 5c1ad900a2 Merge pull request #826 from kansal/CLOUDSTACK-8853
Fixed: Error given when creating VPN user in one network if VR for another network is stopped.

Problem: Adding a VPN user to an IP on an isolated network fails with an exception if the end-user has two isolated networks and one of them is not in use (i.e. VR is stopped).

Fix: Presently in the code, all the VR which are not in Running state are forced to throw error. Added a check for the Stopped and Stopping state routers. Configurations will be applied to them when they get restarted.

* pr/826:
  Fixed Coverity Issue: Unintentional Integer FLow
  Fixed: Error given when creating VPN user in one network if VR for another network is stopped

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-26 13:10:30 -04:00
Will Stevens 9b525f51a2 Merge pull request #1452 from prashanthvarma/master
CLOUDSTACK-9322: Support for Internal LB fuctionality with Nuage VSP SDN Plugin including Marvin test coverageTask: https://issues.apache.org/jira/browse/CLOUDSTACK-9322

PR contents:
1) UI changes to support LB provider InternalLbVm during VPC offering creation.
2) Bug fix for CLOUDSTACK-9320.
3) Nuage VSP SDN Plugin related enhancements for VPC network functionality.
4) Marvin test coverage for Internal LB support on master with Nuage VSP SDN Plugin.
5) Enhancements on our exiting Marvin test code (nuagevsp plugins directory).
6) PEP8 & PyFlakes compliance with our Marvin test code.

Test run:
CloudStack$ nosetests --with-marvin --marvin-config=nuage_ant.cfg test/integration/plugins/nuagevsp/ -a tags=nuagevsp

Test results:
Test user data and password reset functionality with Nuage VSP SDN plugin ... === TestName: test_nuage_UserDataPasswordReset | Status : SUCCESS ===
ok
Test Nuage VSP VPC Offering with different combinations of LB service providers ... === TestName: test_01_nuage_internallb_vpc_Offering | Status : SUCCESS ===
ok
Test Nuage VSP VPC Network Offering with and without Internal LB service ... === TestName: test_02_nuage_internallb_vpc_network_offering | Status : SUCCESS ===
ok
Test Nuage VSP VPC Networks with and without Internal LB service ... === TestName: test_03_nuage_internallb_vpc_networks | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with different combinations of Internal LB rules ... === TestName: test_04_nuage_internallb_rules | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality by performing (wget) traffic tests within a VPC ... === TestName: test_05_nuage_internallb_traffic | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with different LB algorithms by performing (wget) traffic tests ... === TestName: test_06_nuage_internallb_algorithms_traffic | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with restarts of VPC network components by performing (wget) ... === TestName: test_07_nuage_internallb_vpc_network_restarts_traffic | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with InternalLbVm appliance operations by performing (wget) ... === TestName: test_08_nuage_internallb_appliance_operations_traffic | Status : SUCCESS ===
ok
Test Basic VPC Network Functionality with Nuage VSP SDN plugin ... === TestName: test_nuage_vpc_network | Status : SUCCESS ===
ok
Test Nuage VSP SDN plugin with basic Isolated Network functionality ... === TestName: test_nuage_vsp | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 11 tests in 12094.705s

OK

Test run logs:
[results.txt](https://github.com/apache/cloudstack/files/187587/results.txt)
[runinfo.txt](https://github.com/apache/cloudstack/files/187588/runinfo.txt)

Test config file:
[nuage_ant.txt](https://github.com/apache/cloudstack/files/222711/nuage_ant.txt)

Note: Attached the Marvin config file as .txt instead of .cfg.

PEP8 & PyFlakes Compliance:
CloudStack$ pep8 --max-line-length=150 test/integration/plugins/nuagevsp/*.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/nuageTestCase.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_password_reset.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_vpc_internal_lb.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_vpc_network.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_vsp.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/*.py

#CLOUDSTACK-9322

* pr/1452:
  CLOUDSTACK-9322 : Marvin tests for Internal Lb with Nuage VSP
  CLOUDSTACK-9320 : InternalLBVM is not getting destroyed when the last Internal Load Balancer rule is removed for the corresponding source IP address
  CLOUDSTACK-9322 : Changes to support InternalLbVm with Nuage VSP plugin

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-25 15:53:58 -04:00
Abhinandan Prateek 182ab648b5 CLOUDSTACK-9323: Fix Cancel maintenance so that if maintenance is cancelled the host come back to normal state gracefully.
Added marvin tests for host maintennace.
2016-04-25 15:28:32 +05:30
Daan Hoogland 101668994d further maven dependency updates from Daan
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-04-22 16:36:37 +05:30
Will Stevens f530a4c63a Merge pull request #1200 from koushik-das/CLOUDSTACK-9130
CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag

RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.

* pr/1200:
  CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-21 16:35:27 -04:00
Kshitij Kansal fae20570c2 Fixed Coverity Issue: Unintentional Integer FLow 2016-04-21 12:06:36 +05:30
Kshitij Kansal e5e226e549 Fixed: Error given when creating VPN user in one network if VR for another network is stopped 2016-04-21 12:06:36 +05:30
SudharmaJain 93ce108e0c CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads 2016-04-21 09:13:06 +05:30
David Mabry 0b7cc087f8 Let hypervisor type KVM and Simulator detach root volumes.
Updated test_volumes.py to include a test for detaching and reattaching a root volume from a vm. I also had to update base.py to allow attach_volume to have the parameter deviceid to be passed as needed.
2016-04-18 15:40:36 -05:00
Rohit Yadav 8e4644e413 vmware: improve support for disks
- Improve disk chain usage while attaching, migrating disks
- Gets root disk controller based diskDeviceBusName from volume's chain info
- Refactor and move VirtualMachineDiskInfo to cloud-utils
- Allows mixing of scsi controller types
- Fixes a NPE case with map passed as null, for example in case of detach volume
  command
- Use a osdefault translator that allow use of recent os types added (enums of
  which) are not available in the sdk

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-04-18 22:44:18 +05:30
Will Stevens 08f2ec5044 Merge pull request #1263 from rafaelweingartner/lrg-cs-hackday-018
Removed unused code from com.cloud.api.ApiServer**Removed \_ from variables names**: private variables with \_ at the beginning is common in C++ but not in Java.

**Removed unused code from ApiServer:**
- com.cloud.api.ApiServer.getPluggableServices(): unused method;
- com.cloud.api.ApiServer.getApiAccessCheckers(): unused method;

**Methods and variables access level reviewed:**
- com.cloud.api.ApiServer.handleAsyncJobPublishEvent(String, String ,Object): this method was private but the annotation @MessageHandler requests public methods, as can be seen in org.apache.cloudstack.framework.messagebus.MessageDispatcher.buildHandlerMethodCache(Class\<?\>), which searches methods with the @MessageHandler annotation and changes
it to be accessible (setAccessible(true)). Thus, there is no reason for handleAsyncJobPublishEvent be a private method and lead some other dev to wrong conclusions about the use of the method;
- Global variables and methods called just by this class (ApiServer) were changed to private.

**Changed variables and methods from static to non-static (if possible):** as some variables/methods are used just by one object of this class, instantiated by Spring, they were changed to non-static.

With that, calls from com.cloud.api.ApiServlet.ApiServlet() that used static methods from ApiServer, were changed from ApiServer.\<staticMethodName\> to \_apiServer.\<methodName\> that refers to the org.apache.cloudstack.api.ApiServerService interface. Thus, methods com.cloud.api.ApiServer.getJSONContentType() and com.cloud.api.ApiServer.isSecureSessionCookieEnabled() had to be added in the interface (org.apache.cloudstack.api.ApiServerService, interface implemented by class ApiServer).

* pr/1263:
  The goal of this PR is to review com.cloud.api.ApiServer class, with the following actions:

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-18 08:34:53 -04:00