- systemd: Add a /etc/sysconfig/cloudstack-* file
This allows users to easily override variables passed to Java when
starting up.
It also creates a foundation for sharing the systemd service profile
between CentOS and Ubuntu since it only requires the environment file
to be changed.
- deb: Add Ubuntu 16.04 support
Ubuntu 16.04 differs from Ubuntu 14.04 in a few ways:
- systemd instead of sysvinit / upstart
- Java 8 support
The packaging now detects on which distribution it is being
build and based on that it installs different files in the
packages, but it also changes the Dependencies.
Packages for Ubuntu 16.04 will require Java 8 as a JRE
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
[lts/blocker] CLOUDSTACK-9467: Add symlink to key file for usage serverOn fresh installation, the usage server fails to start if the `key` file does
not exist in its classpath. The issue is reproducible in environments (such as Trillian)
where the usage server is installed before cloudstack-setup-databases has been called.
Before the cloudstack db has been setup, the key file does not exist at its
default location and installation of usage-server fails to add a symlink to the
key file.
This fix adds a default symlink to `/etc/cloudstack/management/key` if a
symlink/file does not already exist in the /etc/cloudstack/usage directory.
On new installation, in the post-installation steps it checks if the symlink
or file exists, and adds a symlink if it does not exist. On existing
installations, if symlink or file exists then it will skip adding symlink.
/cc @jburwell @PaulAngus @karuturi
@blueorangutan package
* pr/1657:
CLOUDSTACK-9467: Add symlink to key file for usage server
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9466: Fix fk constraint failure in upgrade pathIn the 4.1.0-4.2.0 db upgrade path, it creates new tables to store secondary
(nfs) storage in image_store table and volumes in volume_store_ref table. In
the upgrade path, it first tries to migrate NFS storage pool where it excludes
storage pools which have been removed, but it migrates all the volumes without
checking if their storage pools have been removed. This causes fk constraint
failure as the volume/row being inserted refers to a storage pool which does
not exist in the image_store table.
The fix migrates all the nfs storage pools to image_store including removed
storage pools and in doing so migrates with the 'removed' field. This fixes
db upgrade for old pre-4.0 and 4.0/4.1 CloudStack clouds.
/cc @jburwell @PaulAngus @karuturi @abhinandanprateek @murali-reddy
* pr/1656:
CLOUDSTACK-9466: Fix fk constraint failure in upgrade path
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
[blocker] CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpmsThe patchviasocket script was rewritten in Python from PR #1533 and made
assumptions that Python 2.7 would be available. In case of CentOS, python 2.7
may not be available or installed. This change ensures that python-argparse
is installed which is used by this script.
/cc @wido @sverrirab @karuturi @jburwell
@blueorangutan package
* pr/1634:
CLOUDSTACK-9452: add python-argparse dependency on el6,7 rpms
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
In the 4.1.0-4.2.0 db upgrade path, it creates new tables to store secondary
(nfs) storage in image_store table and volumes in volume_store_ref table. In
the upgrade path, it first tries to migrate NFS storage pool where it excludes
storage pools which have been removed, but it migrates all the volumes without
checking if their storage pools have been removed. This causes fk constraint
failure as the volume/row being inserted refers to a storage pool which does
not exist in the image_store table.
The fix migrates all the nfs storage pools to image_store including removed
storage pools and in doing so migrates with the 'removed' field. This fixes
db upgrade for old pre-4.0 and 4.0/4.1 CloudStack clouds.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
On fresh installation, the usage server fails to start if the `key` file does
not exist in its classpath. The issue is reproducible in environments where
the usage server is installed before cloudstack-setup-databases has been called.
Before the cloudstack db has been setup, the key file does not exist at its
default location and installation of usage-server fails to add a symlink to the
key file.
This fix adds a default symlink to `/etc/cloudstack/management/key` if a
symlink/file does not already exist in the /etc/cloudstack/usage directory.
On new installation, in the post-installation steps it checks if the symlink
or file exists, and adds a symlink if it does not exist. On existing
installations, if symlink or file exists then it will skip adding symlink.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
[4.9/LTS] Add upgrade path from 4.9.0 to 4.9.1, change version to 4.9.1.0-SNAPSHOTThis adds db upgrade path from 4.9.0 to 4.9.1 and fixes a typo in default user role description (CLOUDSTACK-9449)
/cc @karuturi @jburwell -- this will cause issues when fwd-merged to master, I can do the fwd-merging if you would like to avoid fixing the conflicts yourself
@blueorangutan package
* pr/1646:
Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
cloudstack: upgrade path from 4.9.0 to 4.9.1
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9463: Fix dynamic-roles migrate script for old formatThe old commands.properties format included the full class name such as:
createAccount=com.cloud.api.commands.CreateAccountCmd;1
The migration script did not consider this format and fails. With this fix
the migration script will process both the formats, including processing a
commands.properties file with mixed format, for example:
$ cat commands.properties
### Account commands
createAccount=1
deleteAccount=2
markDefaultZoneForAccount=com.cloud.api.commands.MarkDefaultZoneForAccountCmd;3
$ python scripts/util/migrate-dynamicroles.py -d -f commands.properties
Apache CloudStack Role Permission Migration Tool
(c) Apache CloudStack Authors and the ASF, under the Apache License, Version 2.0
Running this migration tool will remove any default-role permissions from cloud.role_permissions. Do you want to continue? [y/N]y
The commands.properties file has been deprecated and moved at: commands.properties.deprecated
Running SQL query: DELETE FROM `cloud`.`role_permissions` WHERE `role_id` in (1,2,3,4);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 1, '*', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'deleteAccount', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'markDefaultZoneForAccount', 'ALLOW', 1);
Static role permissions from commands.properties have been migrated into the db
Running SQL query: UPDATE `cloud`.`configuration` SET value='true' where name='dynamic.apichecker.enabled'
Dynamic role based API checker has been enabled!
/cc @jburwell @karuturi @PaulAngus
Since we don't have any upgrade/marvin tests for this, the changes have been verified with above test as the script works against a commands.properties. A manual verification by anyone else would be required to validate the changes.
* pr/1649:
CLOUDSTACK-9463: Fix dynamic-roles migrate script for old format
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9459: the try's catch block was shortening the preparedstatement's lifethe try's catch block was shortening the preparedstatement's life resulting in bad resultset when used outside of try catch.
* pr/1641:
CLOUDSTACK-9459: the try's catch block was shortening the preparedstatement life resulting in bad resultset when used outside of try catch.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The old commands.properties format included the full class name such as:
createAccount=com.cloud.api.commands.CreateAccountCmd;1
The migration script did not consider this format and fails. With this fix
the migration script will process both the formats, including processing a
commands.properties file with mixed format, for example:
$ cat commands.properties
### Account commands
createAccount=1
deleteAccount=2
markDefaultZoneForAccount=com.cloud.api.commands.MarkDefaultZoneForAccountCmd;3
$ python scripts/util/migrate-dynamicroles.py -d -f commands.properties
Apache CloudStack Role Permission Migration Tool
(c) Apache CloudStack Authors and the ASF, under the Apache License, Version 2.0
Running this migration tool will remove any default-role permissions from cloud.role_permissions. Do you want to continue? [y/N]y
The commands.properties file has been deprecated and moved at: commands.properties.deprecated
Running SQL query: DELETE FROM `cloud`.`role_permissions` WHERE `role_id` in (1,2,3,4);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 1, '*', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'deleteAccount', 'ALLOW', 0);
Running SQL query: INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 2, 'markDefaultZoneForAccount', 'ALLOW', 1);
Static role permissions from commands.properties have been migrated into the db
Running SQL query: UPDATE `cloud`.`configuration` SET value='true' where name='dynamic.apichecker.enabled'
Dynamic role based API checker has been enabled!
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Adds db upgrade path from 4.9.0 to 4.9.1
- CLOUDSTACK-9449: Fix typo in default user role description
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The patchviasocket script was rewritten in Python from PR #1533 and made
assumptions that Python 2.7 would be available. In case of CentOS, python 2.7
may not be available or installed. This change ensures that python-argparse
is installed which is used by this script.
Expose cmd error in the logs when patch command fails.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Update base debian iso to version 7.11
- Upgrade ruby version to 2.3.0 (latest/stable)
- Fix Gemfile
- Update README
- Fix openswan pkg name with the same version
- Remove cloud-cleanup it's not available
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
packaging: Marvin and integration-tests packagesThis introduces two new cloudstack packages: marvin and integration-tests.
The two packages will make it easier for CI systems to install Marvin for a
specific cloudstack release/build and run integration tests that are specific
for that version/build.
Since Marvin may have its own dependencies, we're bundling the Marvin source tarball and installing it with pip in the post-installation script/phase.
This also fixed a build issue with using juniper-api maven repo.
/cc @swill @PaulAngus @jburwell @wido @bvbharat
* pr/1612:
server: give more memory to tests
packaging: Marvin and integration-tests packages
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Increases allowed max and permgen memory flags to maven-surefire plugins.
This fixes unit test failures in cloud-server.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit fd7273b446)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Simplifies change password transactional logic without using pessmistic locks
- Adds a re-enter password field in the UI to valid ipmi/oobm password
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Ensure that FW_EGRESS_RULE chain exists after upgrading the router
- Flush allow all egress rule on 0.0.0.0/0, if such a rule exists in the config
it will be added later (CLOUDSTACK-9437)
CLOUDSTACK-9436: vm_network_map table cleanup, release network resources on expunge commandJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9436
This PR replaces #1594
Due to error on `test/integration/smoke/test_vpc_redundant.py` it was found out that `vm_network_map` table should be less aggresive on vm stop
* pr/1613:
CLOUDSTACK-9436: Release network resources on expunge command
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This introduces two new cloudstack packages: marvin and integration-tests.
The two packages will make it easier for CI systems to install Marvin for a
specific cloudstack release/build and run integration tests that are specific
for that version/build.
- maven: add explicit juniper-contrail-api maven repository
- marvin: build source distribution for both install and package mvn phases
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9407: vm_network_map table doesnt get cleaned up properlyJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9407
### Introduction
It was found out that in production environments `vm_network_map` table entries were slowly growing. It was investigated how this entries were cleaned up.
### Behaviour
On vm creation, vm mappings are inserted on `vm_network_map`.
On vm stop, mappings are deleted from `vm_network_map` for vm, as a result of the release of its nics.
### Problem
If created vm is stopped from hypervisor side (at least on vSphere in which we tested it), when CloudStack realizes vm is stopped it doesn't clean up `vm_network_table,` and, as cleanup is made during vm stop, when vm is eventually destroyed and expunged it won't clean up their entries in that table.
### Proposed solution
We propose to move `vm_network_map` table cleanup to expunge command instead of stop command.
* pr/1594:
CLOUDSTACK-9407: Refactor
CLOUDSTACK-9407: Release network resources on expunge command
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Update L10N resource files with 4.9 strings from Transifex (20160607)cc @swill before the 4.9 release. Just only the latest FR translation.
* pr/1583:
Update L10N resource files with 4.9 strings from Transifex (20160709)
Signed-off-by: Will Stevens <williamstevens@gmail.com>
packaging: Add db.X.driver=jdbc:mysql to db.properties on upgradeThis is required afther the upgrade to 4.9.0 and for convience we
add this to the configuration so our users do not have to.
* pr/1610:
packaging: Add db.X.driver=jdbc:mysql to db.properties on upgrade
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9430] Added fix for adding/editing Network ACL rule orderingBUG: https://issues.apache.org/jira/browse/CLOUDSTACK-9430
The issue occurred because all of the ACL rules get inserted before the old ones. Then, the cleanup deletes the duplicate rows, and leaves any new rule in front of the old ones.
Here is an example with a simplified iptables view for ACL
Ex: adding a rule 4
before add:
1,2,3
during add:
1',2',3',4',1,2,3
after add:
4',1,2,3
After fix:
before add:
1,2,3
during add:
1,2,3,1',2',3',4'
after add:
1',2',3',4'
* pr/1609:
Added fix for adding/editing Network ACL rule ordering
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Cleanup RBD contexts after exceptions to prevent potential agent crashWe noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
* pr/1608:
improve logging readability
Cleanup rbd contexts and improve exception logging
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9348: Reduce Nio selector wait timeThis reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
/cc @swill @kiwiflyer guys can you please apply this fix in your env and test if you're still able to produce any Nio related error b/w mgmt server(s) and kvm agent(s) not being able to connect quickly. Please also watch out for any increased CPU usage (there should not be any significant change), in which case we may increase the timeout from 100ms to 200-400ms.
* pr/1601:
CLOUDSTACK-9348: Reduce Nio selector wait time
Signed-off-by: Will Stevens <williamstevens@gmail.com>
ui: show resize volume button to all users
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this bug fix all users would see resize volume button in the UI.
/cc @swill
* pr/1595:
ui: show resize volume button to all users
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVTPlease refer to CLOUDSTACK-9328 for the details.
Test Results:
==========
test_01_vpc_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_01_vpc_privategw_acl | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 108.273s
OK
* pr/1455:
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVT Bug-Id:## CLOUDSTACK-9328
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctlyBug in code set PFS to the same value (yes/no) as DPD.
file.addeq(" pfs=%s" % CsHelper.bool_to_yn(obj['dpd']))
* pr/1480:
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Wido den Hollander
Rohit Yadav
Abhinandan Prateek
Will Stevens
Rajani Karuturi
Patrick Dube
nvazquez
Aaron Hurt
Milamber
