Commit Graph

9761 Commits

Author SHA1 Message Date
Sudhansu 542a2112ba CLOUDSTACK-9626: Instance fails to start after unsuccesful compute
offering  upgrade.
2016-12-09 17:06:13 +05:30
Nitin Kumar Maharana 25efc1acd6 CLOUDSTACK-9639: Unable to create shared network with vLan isolation 2016-12-09 12:55:31 +05:30
Rohit Yadav 7c27bf3c90
Merge branch '4.8' into 4.9 2016-12-09 00:33:46 +05:30
Rohit Yadav 8769597523 Merge pull request #1763 from sudhansu7/CLOUDSTACK-9594
CLOUDSTACK-9594: API "list templates templatefilter=all" reveals allAPI "list templates templatefilter=all" reveals all templates.
Using a "list templates templatefilter=all" API call any domain admin can see all templates of all domains in ACS. Information returned includes the account and domain of the template's owner.

The template data shows what that VM is using and any hints from the label. This would give an advantage in what attack vectors to use. The account and domain can possibly be used in brute force attack to guess the password and login information.

Test Scenario:

created two accounts in different domain.

```
mysql> select account_id,username,api_key from user where id in (4,5);
+------------+-----------+----------------------------------------------------------------------------------------+
| account_id | username  | api_key                                                                                |
+------------+-----------+----------------------------------------------------------------------------------------+
|          4 | sudadmin1 | 3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg |
|          5 | sudadmin  | N5uHVOrg1Ek1F1a_5OXTz4WpLG3ewHqcbPUSBjQ-2CTJdxmUe2go0S8fyqH4Np0scYiehYg2KqthZXCWEyKx1A |
+------------+-----------+----------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

mysql> select account_name,domain_id from account where id in (4,5);
+--------------+-----------+
| account_name | domain_id |
+--------------+-----------+
| sudadmin     |         2 |
| sudadmin1    |         3 |
+--------------+-----------+
2 rows in set (0.00 sec)
```

User sudadmin registered a private template named 'Debian'.

http://10.147.59.107:8080/client/api?apikey=N5uHVOrg1Ek1F1a_5OXTz4WpLG3ewHqcbPUSBjQ-2CTJdxmUe2go0S8fyqH4Np0scYiehYg2KqthZXCWEyKx1A&command=listTemplates&templatefilter=self&signature=ODt7zEWCLL20z1FT%2FIkd1molRaM%3D

listTemplate with "templatefilter=self", lists the newly registered template.

```
<listtemplatesresponse cloud-stack-version="4.8.0">
<count>1</count>
	<template>
		<id>51026d32-60ee-4e25-8ffd-3fa3c57fc14c</id>
		<name>Debian</name>
		<displaytext>Debian</displaytext>
		<ispublic>false</ispublic>
		<created>2016-11-10T17:18:00-0500</created>
		<isready>true</isready>
		<passwordenabled>false</passwordenabled>
		<format>VHD</format>
		<isfeatured>false</isfeatured>
		<crossZones>false</crossZones>
		<ostypeid>38c1fc84-a687-11e6-a8c8-06f654000053</ostypeid>
		<ostypename>Debian GNU/Linux 7(64-bit)</ostypename>
		<account>sudadmin</account>
		<zoneid>25fa5b74-d4c2-4bad-8e3a-ceffcd10985e</zoneid>
		<zonename>z1</zonename>
		<status>Download Complete</status>
		<size>2621440000</size>
		<templatetype>USER</templatetype>
		<hypervisor>XenServer</hypervisor>
		<domain>SUDDOMAIN</domain>
		<domainid>a350c00d-4048-4876-ae09-74ad4b7bb28c</domainid>
		<isextractable>false</isextractable>
		<checksum>e87a6d7291b999c92baa9623c9c3c207</checksum>
		<details>{hypervisortoolsversion=xenserver61}</details>
		<sshkeyenabled>false</sshkeyenabled>
		<isdynamicallyscalable>false</isdynamicallyscalable>
	</template>
</listtemplatesresponse>

```

User: sudadmin1
listTemplate with "templatefilter=self" does not list any template.

http://10.147.59.107:8080/client/api?apikey=3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg&command=listTemplates&templatefilter=self&signature=RfKsdg3RxDkqJotbTlHU2RdbdPA%3D

`<listtemplatesresponse cloud-stack-version="4.8.0"/>
`

NO TEMPLATES

**listTemplate with "templatefilter=all" lists all templates**

http://10.147.59.107:8080/client/api?apikey=3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg&command=listTemplates&templatefilter=all&signature=l5tubfyABT67d1jY702dvtZODbc%3D

Result:

```
<listtemplatesresponse cloud-stack-version="4.8.0">
<count>3</count>
	<template>
		<id>38451a02-a687-11e6-a8c8-06f654000053</id>
		<name>CentOS 5.6(64-bit) no GUI (XenServer)</name>
		<displaytext>CentOS 5.6(64-bit) no GUI (XenServer)</displaytext>
		<ispublic>true</ispublic>
		....
	</template>
	<template>
		<id>51026d32-60ee-4e25-8ffd-3fa3c57fc14c</id>
		<name>Debian</name>
		<displaytext>Debian</displaytext>
		<ispublic>false</ispublic>
		<created>2016-11-10T17:18:00-0500</created>
		<isready>true</isready>
		<passwordenabled>false</passwordenabled>
		<format>VHD</format>
		<isfeatured>false</isfeatured>
		<crossZones>false</crossZones>
		<ostypeid>38c1fc84-a687-11e6-a8c8-06f654000053</ostypeid>
		<ostypename>Debian GNU/Linux 7(64-bit)</ostypename>
		**<account>sudadmin</account>**
		<zoneid>25fa5b74-d4c2-4bad-8e3a-ceffcd10985e</zoneid>
		<zonename>z1</zonename>
		<size>2621440000</size>
		<templatetype>USER</templatetype>
		<hypervisor>XenServer</hypervisor>
		<domain>SUDDOMAIN</domain>
		<domainid>a350c00d-4048-4876-ae09-74ad4b7bb28c</domainid>
		<isextractable>false</isextractable>
		<checksum>e87a6d7291b999c92baa9623c9c3c207</checksum>
		<details>{hypervisortoolsversion=xenserver61}</details>
		<sshkeyenabled>false</sshkeyenabled>
		<isdynamicallyscalable>false</isdynamicallyscalable>
	</template>
	<template>
		<id>5f6af7bb-d965-4b9b-ab45-6d455b0d6bbe</id>
		<name>SystemVM Template (XenServer)</name>
		<displaytext>SystemVM Template (XenServer)</displaytext>
		<ispublic>false</ispublic>
		.....
	</template>
</listtemplatesresponse>

```

**After Fix:**

http://10.147.59.107:8080/client/api?apikey=3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg&command=listTemplates&templatefilter=all&signature=l5tubfyABT67d1jY702dvtZODbc%3D

```
<listtemplatesresponse cloud-stack-version="4.8.0">
<count>1</count>
	<template>
		<id>38451a02-a687-11e6-a8c8-06f654000053</id>
		<name>CentOS 5.6(64-bit) no GUI (XenServer)</name>
		<displaytext>CentOS 5.6(64-bit) no GUI (XenServer)</displaytext>
		<ispublic>true</ispublic>
		<created>2016-11-10T09:32:44-0500</created>
		<isready>true</isready>
		<passwordenabled>false</passwordenabled>
		<format>VHD</format>
		<isfeatured>true</isfeatured>
		<crossZones>true</crossZones>
		<ostypeid>38a2bfd6-a687-11e6-a8c8-06f654000053</ostypeid>
		<ostypename>CentOS 5.6 (64-bit)</ostypename>
		<account>system</account>
		<zoneid>25fa5b74-d4c2-4bad-8e3a-ceffcd10985e</zoneid>
		<zonename>z1</zonename>
		<size>21474836480</size>
		<templatetype>BUILTIN</templatetype>
		<hypervisor>XenServer</hypervisor>
		<domain>ROOT</domain>
		<domainid>383e0ea6-a687-11e6-a8c8-06f654000053</domainid>
		<isextractable>true</isextractable>
		<checksum>905cec879afd9c9d22ecc8036131a180</checksum>
		<sshkeyenabled>false</sshkeyenabled>
		<isdynamicallyscalable>true</isdynamicallyscalable>
	</template>
</listtemplatesresponse>

```

Bug has been fixed considering below points
1. templatefilter=all or isofilter=all is applicable only to admin and domain admin.
2. With templatefilter=all or isofilter=all below are the visiblity of templates in system.
- admin should be able to see all templates/iso in system.
- domain admin should be able to see all public template and templates under its domain tree (including sub domain).
- domain admin in a project context should be able to see all public templates and templates registered
as project account and templates which are shared(using updateTemplatePermission api) with project account.

Also Modified "test/integration/component/test_escalation_listTemplateDomainAdmin.py"
This marvin test was written for this scenario but for the second account "templatefilter=all" is not used.

* pr/1763:
  CLOUDSTACK-9594: reverted changes  introduced in  CLOUDSTACK-9376
  CLOUDSTACK-9594: API "list templates templatefilter=all" reveals all templates of all domains

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-09 00:31:09 +05:30
Rohit Yadav f70428a232 Merge pull request #1805 from sudhansu7/CLOUDSTACK-9637
CLOUDSTACK-9637: Template create from snapshot does not populate vm_t**ISSUE**
============
Template create from snapshot does not populate vm_template_details

**REPRO STEPS**
==================
1. Register a template A and specify property:
Root disk controller: scsi
NIC adapter type: E1000
Keyboard type: us

2. Create a vm instance from template A

3. Take volume snapshot for vm instance

4. Delete VM instance

5. Switch to "Storage->Snapshots", convert snapshot to a template B

6. Observe template B does not inherit property from template A, the table vm_template_details is empty

**SOLUTION**: Retrieve and add source template details to VMTemplateVO.

Before Fix:

```

mysql> select id,name,source_template_id from vm_template where id=202;
+-----+--------+--------------------+
| id  | name   | source_template_id |
+-----+--------+--------------------+
| 202 | Debian |               NULL |
+-----+--------+--------------------+
1 row in set (0.00 sec)

mysql> select * from vm_template_details where template_id=202;
+----+-------------+--------------------+-------+---------+
| id | template_id | name               | value | display |
+----+-------------+--------------------+-------+---------+
|  1 |         202 | keyboard           | us    |       1 |
|  2 |         202 | nicAdapter         | E1000 |       1 |
|  3 |         202 | rootDiskController | scsi  |       1 |
+----+-------------+--------------------+-------+---------+
3 rows in set (0.00 sec)

mysql> select id,name,source_template_id from vm_template where source_template_id=202;
+-----+----------------+--------------------+
| id  | name           | source_template_id |
+-----+----------------+--------------------+
| 203 | derived-debian |                202 |
+-----+----------------+--------------------+
1 row in set (0.00 sec)

mysql> select * from vm_template_details where template_id=203;
Empty set (0.00 sec)

After Fix:

mysql> select id,name,source_template_id from vm_template where source_template_id=202;
+-----+--------------------------+--------------------+
| id  | name                     | source_template_id |
+-----+--------------------------+--------------------+
| 203 | derived-debian           |                202 |
| 204 | debian-derived-after-fix |                202 |
+-----+--------------------------+--------------------+
2 rows in set (0.00 sec)

mysql> select * from vm_template_details where template_id=204;
+----+-------------+--------------------+-------+---------+
| id | template_id | name               | value | display |
+----+-------------+--------------------+-------+---------+
|  4 |         204 | keyboard           | us    |       1 |
|  5 |         204 | nicAdapter         | E1000 |       1 |
|  6 |         204 | rootDiskController | scsi  |       1 |
+----+-------------+--------------------+-------+---------+
3 rows in set (0.00 sec)

```

**Marvin Test :** test_template_from_snapshot_with_template_details.py

**Result:**
```
test_01_create_template_snampshot (integration.component.test_template_from_snapshot_with_template_details.TestCreateTemplate) ... === TestName: test_01_create_template_snampshot | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 1 test in 864.523s

OK
```

* pr/1805:
  CLOUDSTACK-9637: Template create from snapshot does not populate vm_template_details

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-08 17:14:52 +05:30
Rohit Yadav de7a06efec Merge pull request #1809 from Accelerite/CLOUDSTACK-9646
CLOUDSTACK-9646: No usage is generated for uploaded templates/volumes from localpublished usage events on successful upload of template or volume.

* pr/1809:
  CLOUDSTACK-9646: No usage is generated for uploaded templates/volumes from local

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-08 00:17:40 +05:30
Sudhansu 7059f9e12d CLOUDSTACK-9594: reverted changes introduced in CLOUDSTACK-9376 2016-12-06 23:17:37 +05:30
Rohit Yadav abfcd5b95f CLOUDSTACK-9632: Upgrade bouncy castle to version 1.55
- Upgrades Maven dependency version to v1.55
- Fixes bountycastle usages and issues
- Adds timeout to jetty/annotation scanning
- Fixes servlet issue, uses servlet 3.1.0
- Downgrade javassist used by reflections to fix annotation process errors
- Make console-proxy-rdp bc dependency same as rest of the codebase
- Picks up PR #1510 by Daan

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-06 17:24:24 +05:30
Rajani Karuturi aed59433a2 CLOUDSTACK-9646: No usage is generated for uploaded templates/volumes from local
published usage events on successful upload of template or volume.
2016-12-05 10:51:23 +05:30
Sudhansu 545a214a42 CLOUDSTACK-9637: Template create from snapshot does not populate vm_template_details
Summary: Retrieve and add source template details to VMTemplateVO.
2016-12-03 18:45:08 +05:30
Sudhansu 15eab829c7 CLOUDSTACK-9649: In the management server log there is an error
related to 0.0.0.0 IP address

Added guest ip of VR as control ip for a Basic zone with VMware.
2016-12-03 10:25:53 +05:30
Sudhansu 6314367eba CLOUDSTACK-9647: NIC adapter type becomes e1000 , even after changing the global parameter
"vmware.systemvm.nic.device.type" to vmxnet3 for VPC VR

Added NicAdapter details in PlugNicCommmand.
2016-12-02 13:49:37 +05:30
Rohit Yadav 50f80cc2a0
Merge branch '4.8' into 4.9 2016-11-25 13:03:04 +05:30
Rohit Yadav 5811d33658 Merge pull request #1635 from myENA/feature/honor_force_stop_vm
CLOUDSTACK-9451https://issues.apache.org/jira/browse/CLOUDSTACK-9451

Re-doing against 4.8 since this is a bug, not a feature.

* pr/1635:
  CLOUDSTACK-9451

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-25 12:56:23 +05:30
John Burwell dc1a7b4338 Merge release branch 4.8 to 4.9
* 4.8:
  CLOUDSTACK-9071: Properly parse stats.output.uri in StatsCollector
2016-11-17 00:46:12 -05:00
John Burwell 293ec4f3fc Merge pull request #1673 from wido/CLOUDSTACK-9071
CLOUDSTACK-9071: Properly parse stats.output.uri in StatsCollectorBoth host and path could have been NULL which causes the StatsCollector
no to start properly.

By checking if the Strings are not Empty or Null we make sure the StatsCollector
always runs and does not prevent the Management Server from starting.

* pr/1673:
  CLOUDSTACK-9071: Properly parse stats.output.uri in StatsCollector

Signed-off-by: John Burwell <meaux@cockamamy.net>
2016-11-17 00:45:21 -05:00
Sudhansu a13231226c CLOUDSTACK-9594: API "list templates templatefilter=all" reveals all
templates of all domains

Bug has been fixed considering below points
1. templatefilter=all or isofilter=all is applicable only to admin
and domain admin.
2. With templatefilter=all or isofilter=all below are the visiblity
of templates in system.
a. admin should be able to see all templates/iso in system.
b. domain admin should be able to see all public template and
templates under its domain tree (including sub domain).
c. domain admin in a project context should be able to see all public
 templates and templates registered as project account and templates
 which are shared(using updateTemplatePermission api) with project account.

Modified
"test/integration/component/test_escalation_listTemplateDomainAdmin.py"
This marvin test is written for this scenario but for the second account
 "templatefilter=all" is not used.
2016-11-11 19:56:35 +05:30
Rohit Yadav 240f44392c
Merge branch '4.8' into 4.9 2016-11-03 16:55:47 +05:30
Rohit Yadav c0a6fc2bfb
Merge pull request #1714 from yvsubhash/CLOUDSTACK-9553
CLOUDSTACK-9553 Usage event is not getting recorded for snapshots in a specific scenario

* pr/1714:
  CLOUDSTACK-9553 Usage event is not getting recorded for snapshots in a specific scenario

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-03 16:55:21 +05:30
Rohit Yadav 25b6990e22
server: Fix build regression caused by 32a39
A constructor signature has changed between 4.8 and 4.9+ branches which caused
failure in a unit test introduced by PR #1694. This fixes the unit test by
passing null as the additional parameter (the test does not need instantiated
object).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-03 09:14:35 +05:30
Rohit Yadav 19951346ad
Merge branch '4.8' into 4.9 2016-11-02 23:03:31 +05:30
Rohit Yadav 68f22e2a43
Merge pull request #1694 from shapeblue/kvm-no-storage-failfast
CLOUDSTACK-9509: Host Connects Without StorageKVM hosts on shared storage failure was accepted by mgmt server with the
host state as Up, even though there was no primary/shared storage available on
it. This patch offers a quick fix by throwing an exception in the storage monitor
which connects storage pool on host. The failure is trapped by agent manager
that disconnects the agent without any investigation.

Based on Lab tests, KVM agent may take upto 2 minutes to attempt NFS mount when
the storage is inaccessible (firewalled, or shutdown) before returning back with
an error. It is safe to assume that this won't add pressure on mgmt server due to
several reconnection attempts, and KVM agent would retry reconnection every 2
minutes.

For such KVM hosts, where failure happens due to storage issues; they will be
briefly put in Alert state but will be mostly be in Connecting state during which
the KVM host attempts to mount/reconfigure NFS storage pool.

/cc @jburwell @karuturi
@blueorangutan package

* pr/1694:
  CLOUDSTACK-9509: Host Connects Without Storage

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-02 22:57:51 +05:30
Rohit Yadav fcee71f35b Merge branch '4.8' into 4.9 2016-10-28 11:43:16 +05:30
Marc-Aurèle Brothier 158497d68a CLOUDSTACK-9544: Check access on account trying to generate user API keys
This fixes CVE-2016-6813

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-10-27 22:15:49 +05:30
subhash_y 3987b8c6b4 CLOUDSTACK-9553 Usage event is not getting recorded for snapshots in a specific scenario 2016-10-24 11:47:53 +05:30
Rajani Karuturi 72612f79cd Merge release branch 4.8 to 4.9
* 4.8:
  Emit a VOLUME_DELETE usage event when account deletion destroys an instance.
2016-10-24 11:13:06 +05:30
nnesic d989c5d8be Emit a VOLUME_DELETE usage event when account deletion destroys an instance.
Currently the logic about volume deletion seems to be that an event
should be emitted when the volume delete is requested, not when the
deletion completes.

The VolumeStateListener specifically ignores destroy events for ROOT
volumes, assuming that the ROOT volume only gets deleted when the
instance is destroyed and the UserVmManager should take care of it.

When deleting an account, all of its resources get destroyed, but the
instance expunging circumvents the UserVmManager, and thus we miss the
VOLUME_DESTROY usage event. The account manager now attempts to
propperly destroy the vm before expunging it. This way the destroy
logic is respected, including the event emission.
2016-10-21 09:26:59 +00:00
Rohit Yadav 32a397aa93 CLOUDSTACK-9509: Host Connects Without Storage
KVM hosts on shared storage failure was accepted by mgmt server with the
host state as Up, even though there was no primary/shared storage available on
it. This patch offers a quick fix by throwing an exception in the storage monitor
which connects storage pool on host. The failure is trapped by agent manager
that disconnects the agent without any investigation.

Based on Lab tests, KVM agent may take upto 2 minutes to attempt NFS mount when
the storage is inaccessible (firewalled, or shutdown) before returning back with
an error. It is safe to assume that this won't add pressure on mgmt server due to
several reconnection attempts, and KVM agent would retry reconnection every 2
minutes.

For such KVM hosts, where failure happens due to storage issues; they will be
briefly put in Alert state but will be mostly be in Connecting state during which
the KVM host attempts to mount/reconfigure NFS storage pool.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-10-21 10:22:32 +05:30
nvazquez f4a42ab0d7 CLOUDSTACK-9535: [API] listVMSnapshots improvement 2016-10-05 17:42:14 -03:00
Nathan Johnson 1a55a93f04 CLOUDSTACK-4858 Honors the snapshot.backup.rightafter configuration variable
Unhides snapshot.backup.rightafter from global configuration

If snapshot.backup.rightafter is set to false (defaults to true), snapshots are
not backed up to secondary storage
2016-09-30 09:39:27 -05:00
Wido den Hollander c1997a1705
CLOUDSTACK-9071: Properly parse stats.output.uri in StatsCollector
Both host and path could have been NULL which causes the StatsCollector
no to start properly.

By checking if the Strings are not Empty or Null we make sure the StatsCollector
always runs and does not prevent the Management Server from starting.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

Conflicts:
	server/src/com/cloud/server/StatsCollector.java
2016-09-29 16:00:38 +02:00
Nathan Johnson 53fd4a7997 CLOUDSTACK-9451
Honor the forced parameter to stop virtual machine api call.
2016-09-09 13:00:17 -05:00
John Burwell 8d11511b1f Adds support for four position versions and optional db upgrades
Often, patch and security releases do not require schema migrations or
data migrations.  However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break.  With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path.  This approach allows the upgrade to proceed and
for the database to properly reflect the installed version.  This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.

Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes.  The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers.  This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers.   It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.

  * Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
  a version that was not explicitly specified.  It determines the
  releases the first release before it with database migrations and uses
  that list as the basis for the list for version being calculated.  A
  noop upgrade is then added to the list which causes no schema changes
  or data migrations, but will update the database to the version.
  * Adds unit tests for the upgrade path calculation logic in
  DatabaseUpgradeChecker
  * Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
  versions of this patch
  * Introduces the CloudStackVersion value object which parses and
  compares three (3) and four (4) position version numbers.  This class
  is intended to replace com.cloud.maint.Version.
  * Adds the junit-dataprovider dependency -- allowing test data to be
  concisely generated separately from the execution of a test case.
  Used extensively in the CloudStackVersionTest.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 13:32:32 +05:30
Rohit Yadav e49ca1ecd2 Merge pull request #1646 from shapeblue/4.9-491upgradepath
[4.9/LTS] Add upgrade path from 4.9.0 to 4.9.1, change version to 4.9.1.0-SNAPSHOTThis adds db upgrade path from 4.9.0 to 4.9.1 and fixes a typo in default user role description (CLOUDSTACK-9449)

/cc @karuturi @jburwell  -- this will cause issues when fwd-merged to master, I can do the fwd-merging if you would like to avoid fixing the conflicts yourself

@blueorangutan package

* pr/1646:
  Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
  cloudstack: upgrade path from 4.9.0 to 4.9.1

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-23 13:48:30 +05:30
Rohit Yadav 1060dc47e5 Merge pull request #1620 from shapeblue/4.9-oobm-password-fix
oobm: simply change password transactional logic- Simplifies change password transactional logic without using pessmistic locks
- Adds a re-enter password field in the UI to valid ipmi/oobm password

* pr/1620:
  oobm: simply change password transactional logic

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-22 15:52:14 +05:30
Rohit Yadav f13c224da1 Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:39 +05:30
Will Stevens 740bd45be6 Updating pom.xml version numbers for release 4.8.2-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-15 10:31:09 -04:00
Will Stevens 007c9f0d3a Merge branch '4.8.1-RC20160808T1006' into 4.8 2016-08-15 10:28:33 -04:00
Will Stevens a63db21d16 Updating pom.xml version numbers for release 4.8.1
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-08 10:06:21 -04:00
Rajani Karuturi c03d035c8b Merge release branch 4.8 to 4.9
* 4.8:
  server: give more memory to tests
  packaging: Marvin and integration-tests packages
2016-08-05 14:30:45 +05:30
Rajani Karuturi a54a3b5cd5 Merge release branch 4.7 to 4.8
* 4.7:
  server: give more memory to tests
  packaging: Marvin and integration-tests packages
2016-08-05 14:29:16 +05:30
Rohit Yadav e5750b3331 server: give more memory to tests
Increases allowed max and permgen memory flags to maven-surefire plugins.
This fixes unit test failures in cloud-server.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit fd7273b446)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-04 15:52:29 +05:30
Rohit Yadav 8ac3c883cd oobm: simply change password transactional logic
- Simplifies change password transactional logic without using pessmistic locks
- Adds a re-enter password field in the UI to valid ipmi/oobm password

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-02 12:47:15 +05:30
Will Stevens 227ff3884d Updating pom.xml version numbers for release 4.9.0
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-25 16:56:04 -04:00
nvazquez 148e974482 CLOUDSTACK-9436: Release network resources on expunge command 2016-07-21 12:25:22 -03:00
Will Stevens 46a6530e70 Revert "Merge pull request #1594 from nvazquez/vmnetworkmapissue"
This reverts commit 9be93c6e90, reversing
changes made to 8d45d711bf.
2016-07-21 11:04:10 -04:00
nvazquez c754a0cf30 CLOUDSTACK-9407: Refactor 2016-06-21 13:49:55 -03:00
nvazquez d3f3fb0590 CLOUDSTACK-9407: Release network resources on expunge command 2016-06-16 12:35:42 -03:00
Nick Livens ffe72ca227 CLOUDSTACK-9399 : NPE during deletion of host when clusterId is null 2016-06-08 08:18:14 +02:00
Will Stevens 123e9f81ab fixed a variable name broken by merge conflicts 2016-05-27 16:32:20 -04:00
Will Stevens 75c8a55014 Merge pull request #1251 from koushik-das/CLOUDSTACK-9180
CLOUDSTACK-9180: Optimize concurrent VM deployment operation on same network

Check if VR needs to be allocated for a given network and only acquire lock if required

Refer to the bug for details.

* pr/1251:
  CLOUDSTACK-9180: Optimize concurrent VM deployment operation on same network Check if VR needs to be allocated for a given network and only acquire lock if required

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-27 15:20:49 -04:00
Will Stevens 121b3d6403 Merge pull request #1567 from exoscale/CLOUDSTACK-9238
CLOUDSTACK-9238: Fix URL length to 2048 for all url fields in VOI will update the PR to add max field length in the API commands too

* pr/1567:
  API: update url field max length
  not needed on host table
  Fix URL length to 2048 for all url fields in VO

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-27 15:20:22 -04:00
Will Stevens 20f779298d merging 4.8 -> master 2016-05-27 15:19:33 -04:00
Will Stevens 744f9d5615 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9376: Restrict listTemplates API with filter=all for root admin
  CLOUDSTACK-9369: Restrict default login to ldap/native users
  Add lsb-release dependency to mgmt server and agent on Debian/Ubuntu.
  Emit template UUID and class type over event bus when deleting templates.
2016-05-27 15:03:42 -04:00
Will Stevens cd5f3320c8 Merge pull request #1564 from greenqloud/pr-emit-template-uuid-on-delete-4.7
Emit template UUID and class type over event bus when deleting templatesNew version of #1378 for the 4.7b branch instead of 4.6.

* pr/1564:
  Emit template UUID and class type over event bus when deleting templates.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-27 15:02:17 -04:00
Rohit Yadav 0cb60a72fe CLOUDSTACK-9376: Restrict listTemplates API with filter=all for root admin
Restricts use of listemplates API with templatefilter=all for root admin only.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-27 15:01:15 -04:00
Rohit Yadav 566e7d9fac CLOUDSTACK-9369: Restrict default login to ldap/native users
- Restricts default login auth handler to ldap and native-cloudstack users
- Refactors and create re-usable method to find domain by id/path
- Adds unit test for refactored method in DomainManagerImpl
- Adds smoke test for login handler

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-27 15:00:05 -04:00
Marc-Aurèle Brothier a59ee03fd7 Fix URL length to 2048 for all url fields in VO 2016-05-27 08:16:05 +02:00
jeff 7aec943ef8 Emit template UUID and class type over event bus when deleting templates.
The behavior is now consistent with template creation. This commit
also adds a unit test for this functionality to make sure that it will
always happen.
2016-05-26 11:20:39 +00:00
Will Stevens 3c800b4bfe Merge pull request #1518 from nvazquez/testnfs
CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts## Description
JIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9368
This pull request address a problem introduced in #1361 in which NFS version couldn't be changed after hosts resources were configured on startup (for hosts using `VmwareResource`), and as host parameters didn't include `nfs.version` key, it was set `null`.

## Proposed solution
In this proposed solution `nfsVersion` would be passed in `NfsTO` through `CopyCommand` to `VmwareResource`, who will check if NFS version is still configured or not. If not, it will use the one sent in the command and will set it to its storage processor and storage handler. After those setups, it will proceed executing command.

* pr/1518:
  CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:05:07 -04:00
Will Stevens 978184bccb Merge pull request #1424 from syed/create-template-api-bug
[CLOUDSTACK-8973] Fix create template from snapshot returning null in case of region storeThis PR fixes a case where when we create a template from a snpashot in a region wide store, we don't set the cross zone flag which causes a null response to be returned
Tests:

Before fix
```
(local)  > create template snapshotid=33aa3f3b-5a47-4d2a-8d27-12952c01ebed displaytext=t2 ostypeid=20c8ead6-d750-11e5-9f8c-06524200007c name=t9

accountid = 1b13d7c2-d750-11e5-9f8c-06524200007c
cmd = org.apache.cloudstack.api.command.admin.template.CreateTemplateCmdByAdmin
created = 2016-02-23T16:09:24+0000
jobid = 4f9f5ff9-e7f0-4af6-999c-799431fd47de
jobinstanceid = a08a9711-bd31-43bb-80a2-49cf9d722a19
jobinstancetype = Template
jobprocstatus = 0
jobresult:
null:
crossZones = False
isfeatured = False
ispublic = False
isready = False
tags:
jobresultcode = 0
jobresulttype = object
jobstatus = 1
userid = 1b140f08-d750-11e5-9f8c-06524200007c
```

See the *null* in response

After fix:

```
(local)  > create template snapshotid=33aa3f3b-5a47-4d2a-8d27-12952c01ebed displaytext=t2 ostypeid=20c8ead6-d750-11e5-9f8c-06524200007c name=t11

accountid = 1b13d7c2-d750-11e5-9f8c-06524200007c
cmd = org.apache.cloudstack.api.command.admin.template.CreateTemplateCmdByAdmin
created = 2016-02-25T21:47:03+0000
jobid = 1b74209b-b3c1-4168-a243-f559aa0c081b
jobinstanceid = 06ecee5a-b1f2-4e67-80fb-f0f44b0aa198
jobinstancetype = Template
jobprocstatus = 0
jobresult:
template:
id = 06ecee5a-b1f2-4e67-80fb-f0f44b0aa198
name = t11
account = admin
created = 2016-02-25T21:47:03+0000
crossZones = True
displaytext = t2
domain = ROOT
domainid = 1b13ab80-d750-11e5-9f8c-06524200007c
format = VHD
hypervisor = XenServer
isdynamicallyscalable = False
isextractable = True
isfeatured = False
ispublic = False
isready = True
ostypeid = 20c8ead6-d750-11e5-9f8c-06524200007c
ostypename = CentOS 5 (64-bit)
passwordenabled = False
size = 21474836480
sourcetemplateid = 1af0f0cc-d750-11e5-9f8c-06524200007c
sshkeyenabled = False
status = Download Complete
tags:
templatetype = USER
jobresultcode = 0
jobresulttype = object
jobstatus = 1
userid = 1b140f08-d750-11e5-9f8c-06524200007c
```
Works correctly

* pr/1424:
  Fix create template from snapshot returning null in case of region store

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:04:31 -04:00
Will Stevens 6a6e164448 Merge pull request #1360 from greenqloud/pr-refactor-system-vm-network-creation
Refactor system VM default network creationTwo small commits which moves the retrieval of the default network for the console proxy and the SSVM into a separate protected method. It's a small change that makes the code more readable/maintainable and also makes the class more suitable for overriding should one want to do this. It's forward-ported from our 4.2 branch.

No new tests since this should not change any functionality, and thus should be covered by the existing unit tests.

Now on the master branch (#1359 was on the wrong branch).

* pr/1360:
  Refactor ssvm default network retrieval.
  Refactor console proxy default network retrieval.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:03:32 -04:00
Will Stevens b5606dcce8 Merge pull request #1553 from nlivens/mysql_driver_issue
Dynamically load drivers before creating our DB connectionsSolution to the mailing thread titled "MySQL : No suitable driver found for jdbc:mysql".
It doesn't harm that we explicitely load the MySQL driver, and for those which would use a commons-dbcp version < 1.4 this would fix it as well. Since JDBC 4.0, the JDBC driver can auto-register itself, but for some weird cases (like ours), it's not working. Therefore we need to explicitly load the JDBC driver.

* pr/1553:
  Dynamic loading of DB driver + support for other DB providers

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:00:42 -04:00
Will Stevens de205c5805 Merge pull request #1297 from DaanHoogland/CLOUDSTACK-9203
CLOUDSTACK-9203 Implement security group move on updateVM API call  cherry-picked from a exoscale internal fix

Conflicts:
	api/src/org/apache/cloudstack/api/command/user/vm/UpdateVMCmd.java
	server/src/com/cloud/vm/UserVmManager.java
	server/src/com/cloud/vm/UserVmManagerImpl.java

* pr/1297:
  CLOUDSTACK-9203 refactorred DeployVM code to be used by UpdateVM as well
  CLOUDSTACK-9203 security group update on running instance

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:59:35 -04:00
Will Stevens 550aabb308 Merge pull request #1550 from ustcweizhou/CLOUDSTACK-9380-listDomains-NPE
CLOUDSTACK-9380: fix NPE in listDomains API for a mistakeThe issue happens if volumeTotal is NULL in database.
This is caused by commit 0407fb334f for CLOUDSTACK-7847.

* pr/1550:
  CLOUDSTACK-9380: fix NPE in listDomains API for a mistake

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:58:46 -04:00
Will Stevens 9456aa0e38 Merge pull request #1503 from Amysta/fix-CLOUDSTACK-9358
CLOUDSTACK-9358: StringIndexOutOfBoundsException on eventsFixes JSON deserialization of `cmdInfo` (current process fails with `StringIndexOutOfBoundsException` when `cmdEventType` is the last parameter in the JSON string).

A `StringIndexOutOfBoundsException` is thrown in some cases during event publication.

Example: a stopVirtualMachine API request is executed, and fails with:

```
2016-04-15 09:24:43,080 ERROR [o.a.c.f.m.MessageDispatcher] (catalina-exec-1:ctx-840cbaa7 ctx-8daf0e9c ctx-f63af073) Unexpected exception when calling com.cloud.api.ApiServer.handleAsyncJobPublishEvent
java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor307.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.cloudstack.framework.messagebus.MessageDispatcher.dispatch(MessageDispatcher.java:75)
        at org.apache.cloudstack.framework.messagebus.MessageDispatcher.onPublishMessage(MessageDispatcher.java:45)
        at org.apache.cloudstack.framework.messagebus.MessageBusBase$SubscriptionNode.notifySubscribers(MessageBusBase.java:441)
        at org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageBusBase.java:178)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.publishOnEventBus(AsyncJobManagerImpl.java:1052)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:180)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:168)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:687)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:528)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:296)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:127)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:124)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:86)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1720)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -468
        at java.lang.String.substring(String.java:1967)
        at com.cloud.api.ApiServer.handleAsyncJobPublishEvent(ApiServer.java:270)
        at sun.reflect.GeneratedMethodAccessor307.invoke(Unknown Source)
        ... 41 more
```

The VM is not stopped, and the API returns the following error:

      HTTP/1.1 530 : InvocationTargetException when invoking event handler for subject: job.eventpublish

The error comes from JSON parsing in 339355594c/server/src/com/cloud/api/ApiServer.java (L270)

The `substring()` method throws a StringIndexOutOfBoundsException when `cmdEventType` is the last field in the JSON object.

The JSON object being the JSON serialization of a `HashMap` object, the order of the fields is not guaranteed. In my tests, I observed that there is no error with Java 7, but it fails with Java 8, which has a different Hashmap implementation.

For example, with request http://127.0.0.1:8096/client/api?command=stopVirtualMachine&id=734ef752-fe99-4d46-8f31-18130496473c, cmdInfo is:

```
{"ctxUserId":"1","httpmethod":"GET","ctxStartEventId":"105","id":"734ef752-fe99-4d46-8f31-18130496473c","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"734ef752-fe99-4d46-8f31-18130496473c\"}","ctxAccountId":"1","uuid":"734ef752-fe99-4d46-8f31-18130496473c","cmdEventType":"VM.STOP"}
```

and it fails.

Please note that the error is not present for all API requests, depending on the parameters fields.

* pr/1503:
  CLOUDSTACK-9358: StringIndexOutOfBoundsException on events

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:56:18 -04:00
Will Stevens 87f6ad3cc9 Merge pull request #846 from kishankavala/CLOUDSTACK-8870
Bug-ID: CLOUDSTACK-8870: Skip external device usage collection if no external devices existexternal network device usage monitor thread that runs every 5mins by default (based on global config external.network.stats.interval) and runs coalesce query to acquire a lock. When there are no external devices exist, there is no need to run usage collection.
Added test case to verify that usage collection task is not run when there are no External LB or External FW

* pr/846:
  Bug-ID: CLOUDSTACK-8870: Skip external device usage collection if no external devices exist

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:55:11 -04:00
Will Stevens 678b28f273 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:54:23 -04:00
Will Stevens b80696cbc0 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:53:42 -04:00
Will Stevens 82f1198ee6 Merge pull request #1410 from ustcweizhou/CLOUDSTACK-6928-io-throttling
CLOUDSTACK-6928: fix issue disk I/O throttling not appliedDisk IO throttling (for KVM) is not applied in the merge of 4.2.

Tests passed:
(1) start vm
(2) attach volume
(3) start vm with volume
(4) migrate vm (with volume)

* pr/1410:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:51:38 -04:00
Daan Hoogland 78cd64a6c2 CLOUDSTACK-9203 refactorred DeployVM code to be used by UpdateVM as well
Conflicts:
	api/src/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java
	api/src/org/apache/cloudstack/api/command/user/vm/UpdateVMCmd.java
2016-05-23 22:40:31 +02:00
Loic Lambiel 5877293434 CLOUDSTACK-9203 security group update on running instance
cherry-picked from a exoscale internal fix:

      Implement security group move on updateVM API call
      Prevent security group update while instance is running

Conflicts:
	api/src/org/apache/cloudstack/api/command/user/vm/UpdateVMCmd.java
	server/src/com/cloud/vm/UserVmManager.java
	server/src/com/cloud/vm/UserVmManagerImpl.java
2016-05-23 22:33:42 +02:00
Nick Livens 02c5d44f0b Dynamic loading of DB driver + support for other DB providers 2016-05-23 15:50:28 +02:00
Kishan Kavala c12d83601d Bug-ID: CLOUDSTACK-8870: Skip external device usage collection if no external devices exist 2016-05-23 15:23:23 +05:30
nvazquez 2d2819974e CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts 2016-05-20 07:31:16 -07:00
Will Stevens cb55624af8 Merge pull request #1523 from nlivens/bug/CLOUDSTACK-9365
CLOUDSTACK-9365 : updateVirtualMachine with userdata should not error when a VM is attached to multiple networks from which one or more doesn't support userdata

* pr/1523:
  Marvin script for cloudstack-9365
  CLOUDSTACK-9365 : updateVirtualMachine with userdata should not error when a VM is attached to multiple networks from which one or more doesn't support userdata

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-20 08:34:55 -04:00
Will Stevens 82b702dc9a Merge pull request #1403 from mike-tutkowski/xs-snapshots
Taking fast and efficient volume snapshots with XenServer (and your storage provider)A XenServer storage repository (SR) and virtual disk image (VDI) each have UUIDs that are immutable.

This poses a problem for SAN snapshots, if you intend on mounting the underlying snapshot SR alongside the source SR (duplicate UUIDs).

VMware has a solution for this called re-signaturing (so, in other words, the snapshot UUIDs can be changed).

This PR only deals with the CloudStack side of things, but it works in concert with a new XenServer storage manager created by CloudOps (this storage manager enables re-signaturing of XenServer SR and VDI UUIDs).

I have written Marvin integration tests to go along with this, but cannot yet check those into the CloudStack repo as they rely on SolidFire hardware.

If anyone would like to see these integration tests, please let me know.

JIRA ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-9281

Here's a video I made that shows this feature in action:

https://www.youtube.com/watch?v=YQ3pBeL-WaA&list=PLqOXKM0Bt13DFnQnwUx8ZtJzoyDV0Uuye&index=13

* pr/1403:
  Faster logic to see if a cluster supports resigning
  Support for backend snapshots with XenServer

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-20 08:33:07 -04:00
Will Stevens 3f5b3a16dd Merge pull request #1516 from sudhansu7/CLOUDSTACK-9366
CLOUDSTACK-9366: Capacity of one zone-wide primary storage ignoredDisable and Remove Host operation disables the primary storage capacity.

Steps to replicate:
Base Condition: There exists a host and storage pool with same id
Steps:
1. Find a host and storage pool having same id
2. Disable the host
3. CPU(1) and MEMORY(0) capacity in op_host_capacity for above host is disabled
4. STORAGE(3) capacity in op_host_capacity for storage pool with id same as above host is also disabled

RCA:
'host_id' column in 'op_host_capacity' table used for storing both storage pool id (for STORAGE capacity) and host id (MEMORY and CPU). While disabling a HOST we also disable the capacity associated with host.

Ideally while disabling capacity we should only disable MEMORY and CPU capacity, but we are not doing so.

Code Path:
ResourceManagerImpl.doDeleteHost() -> ResourceManagerImpl.resourceStateTransitTo() -> CapacityDaoImpl.updateCapacityState(null, null, null, host.getId(), capacityState.toString())

updateCapacityState is updating disabling all entries which matches the host_id. This will also disable a entry having storage pool id same as that of host id.

Changes:
introduced new capacityType parameter in updateCapacityState method and necessary changes to add capacity_type clause in sql
also fixed incorrect sql builder logic (unused code path for which it is never surfaced )
Added marvin test to  check host and storagepool capacity when host is disabled

Test Result:
```
Before Fix:
mysql> select ohc.host_id, ohc.`capacity_state`,  case capacity_type  when 0 then  'MEMORY'  when 1 then  'CPU'  ELSE  'STORAGE'  END as 'capacity_type' ,  total_capacity, case capacity_type  when 0 then  'HOST'  when 1 then  'HOST' ELSE  'STORAGE POOL' END as 'HOST/STORAGE POOL'  from op_host_capacity ohc where host_id=3;
+---------+----------------+---------------+----------------+-------------------+
| host_id | capacity_state | capacity_type | total_capacity | HOST/STORAGE POOL |
+---------+----------------+---------------+----------------+-------------------+
|       3 | Enabled        | MEMORY        |     8589934592 | HOST              |
|       3 | Enabled        | CPU           |          32000 | HOST              |
|       3 | Enabled        | STORAGE       |  2199023255552 | STORAGE POOL      |
+---------+----------------+---------------+----------------+-------------------+

9 rows in set (0.00 sec)

Disable Host 3 from UI.

mysql> select ohc.host_id, ohc.`capacity_state`,  case capacity_type  when 0 then  'MEMORY'  when 1 then  'CPU'  ELSE  'STORAGE'  END as 'capacity_type' ,  total_capacity, case capacity_type  when 0 then  'HOST'  when 1 then  'HOST' ELSE  'STORAGE POOL' END as 'HOST/STORAGE POOL'  from op_host_capacity ohc where host_id=3;
+---------+----------------+---------------+----------------+-------------------+
| host_id | capacity_state | capacity_type | total_capacity | HOST/STORAGE POOL |
+---------+----------------+---------------+----------------+-------------------+
|       3 | Disabled       | MEMORY        |     8589934592 | HOST              |
|       3 | Disabled       | CPU           |          32000 | HOST              |
|       3 | Disabled       | STORAGE       |  2199023255552 | STORAGE POOL      |
+---------+----------------+---------------+----------------+-------------------+

After Fix:

mysql> select ohc.host_id, ohc.`capacity_state`,  case capacity_type  when 0 then  'MEMORY'  when 1 then  'CPU'  ELSE  'STORAGE'  END as 'capacity_type' ,  total_capacity, case capacity_type  when 0 then  'HOST'  when 1 then  'HOST' ELSE  'STORAGE POOL' END as 'HOST/STORAGE POOL'  from op_host_capacity ohc where host_id=3;
+---------+----------------+---------------+----------------+-------------------+
| host_id | capacity_state | capacity_type | total_capacity | HOST/STORAGE POOL |
+---------+----------------+---------------+----------------+-------------------+
|       3 | Enabled        | MEMORY        |     8589934592 | HOST              |
|       3 | Enabled        | CPU           |          32000 | HOST              |
|       3 | Enabled        | STORAGE       |  2199023255552 | STORAGE POOL      |
+---------+----------------+---------------+----------------+-------------------+
3 rows in set (0.01 sec)

Disable Host 3 from UI.

mysql> select ohc.host_id, ohc.`capacity_state`,  case capacity_type  when 0 then  'MEMORY'  when 1 then  'CPU'  ELSE  'STORAGE'  END as 'capacity_type' ,  total_capacity, case capacity_type  when 0 then  'HOST'  when 1 then  'HOST' ELSE  'STORAGE POOL' END as 'HOST/STORAGE POOL'  from op_host_capacity ohc where host_id=3;
+---------+----------------+---------------+----------------+-------------------+
| host_id | capacity_state | capacity_type | total_capacity | HOST/STORAGE POOL |
+---------+----------------+---------------+----------------+-------------------+
|       3 | Disabled       | MEMORY        |     8589934592 | HOST              |
|       3 | Disabled       | CPU           |          32000 | HOST              |
|       3 | Enabled        | STORAGE       |  2199023255552 | STORAGE POOL      |
+---------+----------------+---------------+----------------+-------------------+
3 rows in set (0.00 sec)

Sudhansus-MAC:cloudstack sudhansu$  nosetests-2.7 --with-marvin --marvin-config=setup/dev/advanced.cfg test/integration/component/maint/test_capacity_host_delete.py

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /tmp//MarvinLogs//Apr_22_2016_22_42_27_X4VBWD. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
===final results are now copied to: /tmp//MarvinLogs/test_capacity_host_delete_9RHSNB===
Sudhansus-MAC:cloudstack sudhansu$ cat /tmp//MarvinLogs/test_capacity_host_delete_9RHSNB/results.txt
test_01_op_host_capacity_disable_host (integration.component.maint.test_capacity_host_delete.TestHosts) ... === TestName: test_01_op_host_capacity_disable_host | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 1 test in 0.168s

OK
```

* pr/1516:
  CLOUDSTACK-9366: Capacity of one zone-wide primary storage ignored

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-20 08:31:46 -04:00
Will Stevens 88cd182272 Forward merged 4.8 to master and fixed conflicts in #1331 2016-05-19 12:08:08 -04:00
Sudhansu 18a6aa89be CLOUDSTACK-9366: Capacity of one zone-wide primary storage ignored
introduced new capacityType parameter in updateCapacityState method and necessary changes to add capacity_type clause in sql
also fixed incorrect sql builder logic (unused code path for which it is never surfaced )
Added marvin test to  check host and storagepool capacity when host is disabled
Added conditions to ensure the capacity_type is added only when capacity_type length is greater than 0.
Added checks in marvin test to ensure the capacity exists for a host before disabling it.
Added  checks to avoid index out of range exception
2016-05-19 20:24:04 +05:30
Wei Zhou 976b3b7df4 CLOUDSTACK-6928: fix issue disk I/O throttling not applied 2016-05-19 11:26:43 +02:00
Will Stevens 8f330b0b92 Merge release branch 4.7 to 4.8
* 4.7:
  Fix Sync of template.properties in Swift
  Configure rVPC for router.redundant.vrrp.interval advert_int setting
  Have rVPCs use the router.redundant.vrrp.interval setting
  Resolve conflict as forceencap is already in master
  Split the cidr lists so we won't hit the iptables-resture limits
  Check the existence of 'forceencap' parameter before use
  Do not load previous firewall rules as we replace everyhing anyway
  Wait for dnsmasq to finish restart
  Remove duplicate spaces, and thus duplicate rules.
  Restore iptables at once using iptables-restore instead of calling iptables numerous times
  Add iptables copnversion script.
2016-05-18 15:54:32 -04:00
Will Stevens 06e52e4f34 Merge pull request #1331 from syed/swift-restart-fix
Fix Sync of template.properties in SwiftWhen using Swift as a secondary storage, we create a template.properties file which stores the metadata about the template. Currently the data which is present in the file is incorrect which leads to templates becoming unavailable after they are downloaded. This fix makes sure that the template.properties has the correct "path" set so that templates are available.

I've also done a bit of cleanup and made the code bit more clean.

* pr/1331:
  Fix Sync of template.properties in Swift

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-18 15:53:33 -04:00
Wei Zhou 59edbb5337 CLOUDSTACK-9380: fix NPE in listDomains API for a mistake 2016-05-17 14:45:50 +02:00
Mike Tutkowski 9d215562eb Faster logic to see if a cluster supports resigning 2016-05-16 07:18:39 -06:00
Syed f5ac8ddded Fix Sync of template.properties in Swift 2016-05-13 22:08:16 +00:00
Will Stevens f2f895a5c2 Merge pull request #1537 from PaulAngus/master
Remove extraneous log directory and add catalina.out log rotationrebased and squashed
replacement for PR: https://github.com/apache/cloudstack/pull/1087

Remove extraneous log directory (/var/log/cloudstack-management)
and adds catalina.out log rotation

* pr/1537:
  Remove extraneous log directory and add catalina.out log rotation

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-13 17:04:18 -04:00
Remi Bergsma 2bfb12cba9 Have rVPCs use the router.redundant.vrrp.interval setting
It defaults to 1, which is hardcoded in the template:
./cosmic/cosmic-core/systemvm/patches/debian/config/opt/cloud/templates/keepalived.conf.templ

As non-VPC redundant routers use this setting, I think it makes sense to use it for rVPCs as well.

We also need a change to pickup the cmd_line parameter and use it in the Python code that configures the router.
2016-05-13 14:36:23 +02:00
Mike Tutkowski 2bd035d199 Support for backend snapshots with XenServer 2016-05-13 01:02:04 -06:00
Will Stevens 8c3722d953 Merge pull request #1444 from rafaelweingartner/workAroundPR780
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMThis PR introduces the changes proposed in PR #780 with some work to make the code null safe.

During this PR, I have also removed some unused code.

* pr/1444:
  Removed unnecessary check when creating the “userVmResponse” object.
  Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
  CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:49:14 -04:00
Will Stevens 7e41747c33 Merge pull request #816 from mike-tutkowski/addremovehosts2
Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

This PR addresses the following JIRA ticket:

https://issues.apache.org/jira/browse/CLOUDSTACK-8813

The problem is that there needs to be notifications sent when a host is added to, about to be removed from, and removed from a cluster.

Such notifications can be used for many purposes. For example, it can allow storage plug-ins to update ACLs on their storage systems. Also, it can allow us to clean up IQNs from ESXi hosts that are no longer needed.

* pr/816:
  CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:07:36 -04:00
Will Stevens 143265c49f Merge pull request #1502 from shapeblue/outofband-master
CLOUDSTACK-9299: Out-of-band Management for CloudStackSupport access to a hosts out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature, please install `ipmitool` (using yum/apt/brew) and `ipmisim`:
https://pypi.python.org/pypi/ipmisim

The default ipmitool location is assumed in /usr/bin, if this is different in your env please fix the global setting, see FS for details on various global settings.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

/cc @jburwell @swill @abhinandanprateek @murali-reddy @borisstoyanov

* pr/1502:
  maven: ignore utils/testsmallfileinactive for rat checking
  CLOUDSTACK-9378: Fix for #1497
  HypervisorUtilsTest: increate timeout to 8seconds
  travis: Use patched version of ipmitool for tests
  CLOUDSTACK-9299: Out-of-band Management for CloudStack

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:06:13 -04:00
Will Stevens 103d62ee02 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:04:53 -04:00
Will Stevens dbbc37ecc7 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:03:57 -04:00
Will Stevens 3fab75772f Merge pull request #1474 from remibergsma/47_private_gw_initial_config
Handle private gateways more reliablyWhen initialising a VPC router we need to know which IP/device corresponds to a private gateway. This is to solve a problem when stop/starting a VPC router (which gets the private gateway config as a guest network and as a result breaks the functionality). You read it right, the private gateway is sent as type=guest after reboot and type=public initially.

Before this change, you could add a private gw to a running router but you couldn't restart it (it would mix up the tiers). Now the private gateway is detected properly and it works just fine.

Booting without private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.0.42",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "None",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

Booting with private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.2.227",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "10.201.10.1",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

And:
```
cat cmdline
vpccidr=10.0.0.0/24 domain=cs2cloud dns1=8.8.8.8 privategateway=10.201.10.1 template=domP name=r-167-VM eth0ip=169.254.2.227 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w baremetalnotificationapikey=V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw host=192.168.22.61 port=8080
```

Logs:
```
2016-02-24 20:08:45,723 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-458d4c52 job-1402/job-1403 ctx-d5355fca) (logid:5772906c) Set privategateway field in cmd_line.json to 10.201.10.1
```

* pr/1474:
  Handle private gateways more reliably
  Add private gateway IP to router initialization config

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:02:55 -04:00
Will Stevens 309a60ea71 Merge pull request #1483 from remibergsma/pr1413-wilder-47
CLOUDSTACK-9287 - Fix unique mac address per rVPC routerThis is work by @wilderrodrigues, see PR #1413 It contains important fixes and I think it needs to be included so I send the PR again.

* pr/1483:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:00:40 -04:00
Paul Angus 11e3582fdf Remove extraneous log directory and add catalina.out log rotation 2016-05-12 08:41:39 +01:00
Rohit Yadav 07564469e9 CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Mike Tutkowski dad9e5d868 CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster 2016-05-11 08:02:46 -06:00
Nick Livens 71c9c90e0c CLOUDSTACK-9365 : updateVirtualMachine with userdata should not error when a VM is attached to multiple networks from which one or more doesn't support userdata 2016-05-11 07:59:38 +02:00