[4.9/LTS] Add upgrade path from 4.9.0 to 4.9.1, change version to 4.9.1.0-SNAPSHOTThis adds db upgrade path from 4.9.0 to 4.9.1 and fixes a typo in default user role description (CLOUDSTACK-9449)
/cc @karuturi @jburwell -- this will cause issues when fwd-merged to master, I can do the fwd-merging if you would like to avoid fixing the conflicts yourself
@blueorangutan package
* pr/1646:
Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
cloudstack: upgrade path from 4.9.0 to 4.9.1
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9463: Fix dynamic-roles migrate script for old formatThe old commands.properties format included the full class name such as:
createAccount=com.cloud.api.commands.CreateAccountCmd;1
The migration script did not consider this format and fails. With this fix
the migration script will process both the formats, including processing a
commands.properties file with mixed format, for example:
$ cat commands.properties
### Account commands
createAccount=1
deleteAccount=2
markDefaultZoneForAccount=com.cloud.api.commands.MarkDefaultZoneForAccountCmd;3
$ python scripts/util/migrate-dynamicroles.py -d -f commands.properties
Apache CloudStack Role Permission Migration Tool
(c) Apache CloudStack Authors and the ASF, under the Apache License, Version 2.0
Running this migration tool will remove any default-role permissions from cloud.role_permissions. Do you want to continue? [y/N]y
The commands.properties file has been deprecated and moved at: commands.properties.deprecated
Running SQL query: DELETE FROM `cloud`.`role_permissions` WHERE `role_id` in (1,2,3,4);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 1, '*', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'deleteAccount', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'markDefaultZoneForAccount', 'ALLOW', 1);
Static role permissions from commands.properties have been migrated into the db
Running SQL query: UPDATE `cloud`.`configuration` SET value='true' where name='dynamic.apichecker.enabled'
Dynamic role based API checker has been enabled!
/cc @jburwell @karuturi @PaulAngus
Since we don't have any upgrade/marvin tests for this, the changes have been verified with above test as the script works against a commands.properties. A manual verification by anyone else would be required to validate the changes.
* pr/1649:
CLOUDSTACK-9463: Fix dynamic-roles migrate script for old format
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9459: the try's catch block was shortening the preparedstatement's lifethe try's catch block was shortening the preparedstatement's life resulting in bad resultset when used outside of try catch.
* pr/1641:
CLOUDSTACK-9459: the try's catch block was shortening the preparedstatement life resulting in bad resultset when used outside of try catch.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The old commands.properties format included the full class name such as:
createAccount=com.cloud.api.commands.CreateAccountCmd;1
The migration script did not consider this format and fails. With this fix
the migration script will process both the formats, including processing a
commands.properties file with mixed format, for example:
$ cat commands.properties
### Account commands
createAccount=1
deleteAccount=2
markDefaultZoneForAccount=com.cloud.api.commands.MarkDefaultZoneForAccountCmd;3
$ python scripts/util/migrate-dynamicroles.py -d -f commands.properties
Apache CloudStack Role Permission Migration Tool
(c) Apache CloudStack Authors and the ASF, under the Apache License, Version 2.0
Running this migration tool will remove any default-role permissions from cloud.role_permissions. Do you want to continue? [y/N]y
The commands.properties file has been deprecated and moved at: commands.properties.deprecated
Running SQL query: DELETE FROM `cloud`.`role_permissions` WHERE `role_id` in (1,2,3,4);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 1, '*', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'deleteAccount', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'markDefaultZoneForAccount', 'ALLOW', 1);
Static role permissions from commands.properties have been migrated into the db
Running SQL query: UPDATE `cloud`.`configuration` SET value='true' where name='dynamic.apichecker.enabled'
Dynamic role based API checker has been enabled!
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Adds db upgrade path from 4.9.0 to 4.9.1
- CLOUDSTACK-9449: Fix typo in default user role description
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Update base debian iso to version 7.11
- Upgrade ruby version to 2.3.0 (latest/stable)
- Fix Gemfile
- Update README
- Fix openswan pkg name with the same version
- Remove cloud-cleanup it's not available
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
packaging: Marvin and integration-tests packagesThis introduces two new cloudstack packages: marvin and integration-tests.
The two packages will make it easier for CI systems to install Marvin for a
specific cloudstack release/build and run integration tests that are specific
for that version/build.
Since Marvin may have its own dependencies, we're bundling the Marvin source tarball and installing it with pip in the post-installation script/phase.
This also fixed a build issue with using juniper-api maven repo.
/cc @swill @PaulAngus @jburwell @wido @bvbharat
* pr/1612:
server: give more memory to tests
packaging: Marvin and integration-tests packages
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Increases allowed max and permgen memory flags to maven-surefire plugins.
This fixes unit test failures in cloud-server.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit fd7273b446)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Simplifies change password transactional logic without using pessmistic locks
- Adds a re-enter password field in the UI to valid ipmi/oobm password
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Ensure that FW_EGRESS_RULE chain exists after upgrading the router
- Flush allow all egress rule on 0.0.0.0/0, if such a rule exists in the config
it will be added later (CLOUDSTACK-9437)
CLOUDSTACK-9436: vm_network_map table cleanup, release network resources on expunge commandJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9436
This PR replaces #1594
Due to error on `test/integration/smoke/test_vpc_redundant.py` it was found out that `vm_network_map` table should be less aggresive on vm stop
* pr/1613:
CLOUDSTACK-9436: Release network resources on expunge command
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This introduces two new cloudstack packages: marvin and integration-tests.
The two packages will make it easier for CI systems to install Marvin for a
specific cloudstack release/build and run integration tests that are specific
for that version/build.
- maven: add explicit juniper-contrail-api maven repository
- marvin: build source distribution for both install and package mvn phases
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9407: vm_network_map table doesnt get cleaned up properlyJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9407
### Introduction
It was found out that in production environments `vm_network_map` table entries were slowly growing. It was investigated how this entries were cleaned up.
### Behaviour
On vm creation, vm mappings are inserted on `vm_network_map`.
On vm stop, mappings are deleted from `vm_network_map` for vm, as a result of the release of its nics.
### Problem
If created vm is stopped from hypervisor side (at least on vSphere in which we tested it), when CloudStack realizes vm is stopped it doesn't clean up `vm_network_table,` and, as cleanup is made during vm stop, when vm is eventually destroyed and expunged it won't clean up their entries in that table.
### Proposed solution
We propose to move `vm_network_map` table cleanup to expunge command instead of stop command.
* pr/1594:
CLOUDSTACK-9407: Refactor
CLOUDSTACK-9407: Release network resources on expunge command
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Update L10N resource files with 4.9 strings from Transifex (20160607)cc @swill before the 4.9 release. Just only the latest FR translation.
* pr/1583:
Update L10N resource files with 4.9 strings from Transifex (20160709)
Signed-off-by: Will Stevens <williamstevens@gmail.com>
packaging: Add db.X.driver=jdbc:mysql to db.properties on upgradeThis is required afther the upgrade to 4.9.0 and for convience we
add this to the configuration so our users do not have to.
* pr/1610:
packaging: Add db.X.driver=jdbc:mysql to db.properties on upgrade
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9430] Added fix for adding/editing Network ACL rule orderingBUG: https://issues.apache.org/jira/browse/CLOUDSTACK-9430
The issue occurred because all of the ACL rules get inserted before the old ones. Then, the cleanup deletes the duplicate rows, and leaves any new rule in front of the old ones.
Here is an example with a simplified iptables view for ACL
Ex: adding a rule 4
before add:
1,2,3
during add:
1',2',3',4',1,2,3
after add:
4',1,2,3
After fix:
before add:
1,2,3
during add:
1,2,3,1',2',3',4'
after add:
1',2',3',4'
* pr/1609:
Added fix for adding/editing Network ACL rule ordering
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Cleanup RBD contexts after exceptions to prevent potential agent crashWe noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
* pr/1608:
improve logging readability
Cleanup rbd contexts and improve exception logging
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9348: Reduce Nio selector wait timeThis reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
/cc @swill @kiwiflyer guys can you please apply this fix in your env and test if you're still able to produce any Nio related error b/w mgmt server(s) and kvm agent(s) not being able to connect quickly. Please also watch out for any increased CPU usage (there should not be any significant change), in which case we may increase the timeout from 100ms to 200-400ms.
* pr/1601:
CLOUDSTACK-9348: Reduce Nio selector wait time
Signed-off-by: Will Stevens <williamstevens@gmail.com>
ui: show resize volume button to all users
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this bug fix all users would see resize volume button in the UI.
/cc @swill
* pr/1595:
ui: show resize volume button to all users
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVTPlease refer to CLOUDSTACK-9328 for the details.
Test Results:
==========
test_01_vpc_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_01_vpc_privategw_acl | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 108.273s
OK
* pr/1455:
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVT Bug-Id:## CLOUDSTACK-9328
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctlyBug in code set PFS to the same value (yes/no) as DPD.
file.addeq(" pfs=%s" % CsHelper.bool_to_yn(obj['dpd']))
* pr/1480:
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
Signed-off-by: Will Stevens <williamstevens@gmail.com>
We noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDsWith object store like Swift as secondary storage, if a compressed VHD is uploaded as a template, the `VHDProcessor` incorrectly calculates the virutal size leading to the template being useless. This fix tries to guess the virtual size by partially decompressing it and falls back to a sensible default which is the size of the file.
Before the fix: template.properties on Swift
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=2894447637315205059
```
After the fix
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=3145728000
```
Look at the `virutalsize` in both cases
* pr/1598:
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixes for VirtualRouters in Basic Networking, especially with mutliple ranges in VLANsDuring the last few modifications on the SystemVM scripts, it turns out quite a lot of stuff broke in our setups.
This PR fixes a number of things:
* Multiple IP's per VLAN interface are now supported & working again, including DNS, DHCP ranges, password and metadata services
* `useextdns` fixed (I had a small merge conflict with an attempt to fix this at 4.7, but these fixes are more comprehensive)
* CLOUDSTACK-8303
* Apache configs better in line with best-practices and distro-expected locations
* Added a few more helper functions & getters & setters for utility
* some minor cleanup & fixes
* pr/1547:
Remove /etc/apache2/sites-enabled/000-default in cloud-early-config
SysVM various fixes to previous refactorings * make CORS include a regular glob-matched one * fix NameVirtualHost in CsApp.py as well * even moar cleanups
SysVM: Cleanup and removal of old (and dangerous) config files * ports.conf * default & default-ssl sites * SSL config in httpd.conf * deprecated & dead setup_redundant_router in cloud-early-config
SysVM cloud-early-config: Intermediate fix for SecStore & CORS * Take setup from vhost.template rather than default(-ssl) * should move into Python CS code as well * Move CORS setup to separate conf * Modify vhost template to Optionally include the cors file * Add NameVirtualHost to vhost template for feature parity with ports.conf * Take setup from vhost.template rather than default(-ssl)
VR cloud-early-config: Commonize Apache2 common setup
VR cloud-early-config: Fix Apache2 alias cleanup
VR: consistent SSL setup, vhost is not an example, but a template
VR CsConfig: reintroduce old get_dns() behaviour for redundant non-VPC's
VR CsAddress fixes: * cleanup imports, * fix to_str(), * improve & fix service post_config logic * don't arpPing when there's no gateway
VR CsApp: Expose config to classes, move vhost confs to proper location, allow for multiple IP's per intf, sanitize servername, don't open port 53 if no DNS is foreseen
VR CsConfig: Add is_router(), is_dns(), has_dns(), has_metadata(), use_extdns(), fix get_dns() with use_extdns()
VR CsDhcp: allow multiple ranges & finite lease time (fixes CLOUDSTACK-8303)
VR CsGuestNetwork obey useextdns
VR merge.py ipalias fix & dhcpconfig stub notification
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
* 4.7:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storageIn turn this also fixes VM migration with local storage
This PR is created against 4.7 and can be forward merged to future branches also.
* pr/1596:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9404 Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted.
Issue: https://issues.apache.org/jira/browse/CLOUDSTACK-9404
In this example, I created rules with the port numbers the same as the rule numbers.
Chain ACL_INBOUND_eth2 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 225.0.0.50
ACCEPT all -- anywhere vrrp.mcast.net
DROP tcp -- anywhere anywhere tcp dpt:netstat
DROP tcp -- anywhere anywhere tcp dpt:10
DROP tcp -- anywhere anywhere tcp dpt:5
DROP tcp -- anywhere anywhere tcp dpt:3
DROP tcp -- anywhere anywhere tcp dpt:2
DROP all -- anywhere anywhere
We can see above that the rules are inverted.
After the fix:
Chain ACL_INBOUND_eth2 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 225.0.0.50
ACCEPT all -- anywhere vrrp.mcast.net
DROP tcp -- anywhere anywhere tcp dpt:2
DROP tcp -- anywhere anywhere tcp dpt:3
DROP tcp -- anywhere anywhere tcp dpt:5
DROP tcp -- anywhere anywhere tcp dpt:10
DROP tcp -- anywhere anywhere tcp dpt:netstat
DROP all -- anywhere anywhere
* pr/1581:
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
Signed-off-by: Will Stevens <williamstevens@gmail.com>
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this changes all users would see resize volume button in the UI.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Rohit Yadav
Abhinandan Prateek
Rajani Karuturi
Will Stevens
Patrick Dube
nvazquez
Wido den Hollander
Aaron Hurt
Milamber
Aaron Hurt
Syed
Anshul Gangwar
