* 4.7:
CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
Handle private gateways more reliably
CLOUDSTACK-9287 - Fix RVR public interface
CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
CLOUDSTACK-9287 - Refactor the interface state configuration
CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
CLOUDSTACK-9287 - Put private gateway interface down on backup router
CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
Add private gateway IP to router initialization config
apply static routes on change to master state
Handle private gateways more reliablyWhen initialising a VPC router we need to know which IP/device corresponds to a private gateway. This is to solve a problem when stop/starting a VPC router (which gets the private gateway config as a guest network and as a result breaks the functionality). You read it right, the private gateway is sent as type=guest after reboot and type=public initially.
Before this change, you could add a private gw to a running router but you couldn't restart it (it would mix up the tiers). Now the private gateway is detected properly and it works just fine.
Booting without private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
"config": {
"baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
"baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
"disable_rp_filter": "true",
"dns1": "8.8.8.8",
"domain": "cs2cloud",
"eth0ip": "169.254.0.42",
"eth0mask": "255.255.0.0",
"host": "192.168.22.61",
"name": "r-167-VM",
"port": "8080",
"privategateway": "None",
"redundant_router": "false",
"template": "domP",
"type": "vpcrouter",
"vpccidr": "10.0.0.0/24"
},
"id": "cmdline"
```
Booting with private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
"config": {
"baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
"baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
"disable_rp_filter": "true",
"dns1": "8.8.8.8",
"domain": "cs2cloud",
"eth0ip": "169.254.2.227",
"eth0mask": "255.255.0.0",
"host": "192.168.22.61",
"name": "r-167-VM",
"port": "8080",
"privategateway": "10.201.10.1",
"redundant_router": "false",
"template": "domP",
"type": "vpcrouter",
"vpccidr": "10.0.0.0/24"
},
"id": "cmdline"
```
And:
```
cat cmdline
vpccidr=10.0.0.0/24 domain=cs2cloud dns1=8.8.8.8 privategateway=10.201.10.1 template=domP name=r-167-VM eth0ip=169.254.2.227 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w baremetalnotificationapikey=V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw host=192.168.22.61 port=8080
```
Logs:
```
2016-02-24 20:08:45,723 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-458d4c52 job-1402/job-1403 ctx-d5355fca) (logid:5772906c) Set privategateway field in cmd_line.json to 10.201.10.1
```
* pr/1474:
Handle private gateways more reliably
Add private gateway IP to router initialization config
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Apply static routes on change to master stateRefactored static routes for private gateways so they also get loaded when the router switches to master state. Otherwise they're lost and connections drop after fail over.
* pr/1472:
apply static routes on change to master state
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9287 - Fix unique mac address per rVPC routerThis is work by @wilderrodrigues, see PR #1413 It contains important fixes and I think it needs to be included so I send the PR again.
* pr/1483:
CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
CLOUDSTACK-9287 - Fix RVR public interface
CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
CLOUDSTACK-9287 - Refactor the interface state configuration
CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
CLOUDSTACK-9287 - Put private gateway interface down on backup router
CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Disable the NioTest, it is far to dependent on the network configuration of the
machine running the test. Cherry-picked change from a historic commit where
this test was disabled.
(cherry picked from commit 881a6e164e)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This fixes several Jenkins failures as previous runs don't cleanup this
file created by one of the unit tests.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- For out-of-band management feature (CLOUDSTACK-9299) use patched version of
ipmitool that would work on trusty travis machines
- The ipmitool used is from xenial/16.04 release with patch from RedHat
https://bugzilla.redhat.com/show_bug.cgi?id=1286035
- Installs ipmitool from xenial repositories to get all the dependencies
and then install patched deb version
- Skip test if the known failure occurs
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-8562: Dynamic Role-Based API Checker for CloudStack### CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
This feature allows root administrators to define new roles and associate API
permissions to them.
A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
Admin, Domain Admin and User) which maintains this association by requiring
all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
removal of roles and/or modifications of permissions, without the need
of restarting management server(s)
Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
deployments will continue to use the older static role based api access checker
with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
to add existing set of permissions to the default roles. cloud.account
will have a new role_id column which will be populated based on default roles
as well
Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature
* pr/1489:
maven: Fix jstl version usage
CLOUDSTACK-8562: Deprecate commands.properties
CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
CLOUDSTACK-9361: Centrally handle API validations
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This would fix regression from recent mvn version changes. Without this
patch users get redirected to error.jsp as jstl-1.2 jar is not installed
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Removes commands.properties file
- Fixes apidocs and marvin to be independent of commands.properties usage
- Removes bundling of commands.properties in deb/rpm packaging
- Removes file references across codebase
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This feature allows root administrators to define new roles and associate API
permissions to them.
A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
Admin, Domain Admin and User) which maintains this association by requiring
all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
removal of roles and/or modifications of permissions, without the need
of restarting management server(s)
Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
deployments will continue to use the older static role based api access checker
with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
to add existing set of permissions to the default roles. cloud.account
will have a new role_id column which will be populated based on default roles
as well
Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Validate API arguments based on annotations. Introduces:
- NotNullOrEmpty: for doing null and empty string checks
- PositiveNumber: number > 0 (natural number)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads The thread pool was hardcoded to use 8 threads,
com.cloud.template.TemplateManagerImpl.configure(String, Map<String, Object>):
_preloadExecutor = Executors.newFixedThreadPool(8, new NamedThreadFactory("Template-Preloader"));
Added the change to pick threadpool size from configuration.
* pr/880:
CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Marvin: Replace a timer.sleep(30) with pulling logichttps://issues.apache.org/jira/browse/CLOUDSTACK-9374
From the ticket:
In the base.py file, there is a Host class with a delete instance method.
This method first attempts to transition the host into the maintenance resource state.
The first step in this process is to transition the host into the prepare-for-maintenance resource state.
A while later, the host can be transitioned completely into the maintenance resource state.
In an attempt to wait for this transition to occur, the delete method has a timer.sleep(30) call.
The hope is that the host will have transitioned from the prepare-for-maintenance resource state to the maintenance resource state within 30 seconds, but this does not always happen.
We should correct this problem by putting in logic to query the management server for the resource state of the host. If it's in the expected state, move on; else, sleep for a bit and try again (up to a certain limit).
* pr/1529:
Replace a timer.sleep(30) with pulling logic
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9340: General DB Optimization## Description
In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken and some deficiencies were discovered, we can group them in 4 groups:
* Incorrect PRIMARY key
* Duplicate PRIMARY KEY
* Missing indexes (Add indexes to avoid full table scans)
* Some views query (Change view to improve account retrieval speed)
* pr/1466:
CLOUDSTACK-9340: General DB Optimization
Signed-off-by: Will Stevens <williamstevens@gmail.com>
agent: Enable IPv6 connectivity for KVM Agent to Management ServerIPv4 is still preferred, so if the hostname of the Management Server
returns a A and AAAA-record the Agent will still connect to the
server over IPv4.
This situation will however allow to use a hostname which only has
a AAAA-record. In that case the Agent will connect to the Management
Server over IPv6.
* pr/1488:
agent: Enable IPv6 connectivity for KVM Agent to Management Server
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8818: Use MySQL native connector with PythonMySQLdb has been deprecated and is also not supported in Python 3.
mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.
https://dev.mysql.com/doc/connector-python/en/
* pr/1054:
CLOUDSTACK-8818: Use MySQL native connector with Python
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fai
* pr/1428:
Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fails to load successfully
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Removed Unused Void ClassRemoved the class _Void.java_ from _/cloud-framework-ipc/src/org/apache/cloudstack/framework/async/Void.java_ since the method _public void AsyncMethod(Object realParam,
AsyncCompletionCallback<Void> callback)_ does not exist and it does not seem to be used in another place.
Also, removed an unnecessary comment in
_/cloudstack-framework/ipc/test/org/apache/cloudstack/framework/codestyle/AsyncSampleListenerStyleCaller.java_
* pr/1440:
Removed Unused Void Class
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9351: Add ids parameter to resource listing API calls## General behaviour
A new parameter is added in each method, its type a list of IDs of the entity, it will be mutually exclusive with id. (Similar to <code>id</code> and <code>ids</code> parameters in <code>listVirtualMachines</code> method)
### API Methods affected
* <code>listTemplates</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVolumes</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listSnapshots</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVMSnapshots</code>: new parameter **<code>vmsnapshotids</code>**, mutually exclusive with <code>vmsnapshotid</code>
* pr/1497:
CLOUDSTACK-9351: Add marvin test and add it to travis file
CLOUDSTACK-9351: Add ids parameter to resource listing API calls
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage- KVM-HA- Fix CheckOnHost for Local storage
- Also skip HA on VMs that are using local storage
* pr/1496:
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage - Also skip HA on VMs that are using local storage
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CID-1338387: Deletion of method endPointSelector.selectHypervisorHostFollowing the discussions and analysis presented on PR #1056 create by @DaanHoogland
This PR is intended to push those changes that were discussed there regarding the of endPointSelector.selectHypervisorHost method.
* pr/1124:
Deletion of method endPointSelector.selectHypervisorHost
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
Update L10N resource files with 4.8 strings from Transifex (20160504) Force "translator" mode with the transifex client. Update Transifex client config file for 4.8 resources/L10N ref. (generated by Tx client)
L10n update 4.8 20160422@swill the good PR for 4.8 branch.
* pr/1515:
Update L10N resource files with 4.8 strings from Transifex (20160504) Force "translator" mode with the transifex client. Update Transifex client config file for 4.8 resources/L10N ref. (generated by Tx client)
Signed-off-by: Will Stevens <williamstevens@gmail.com>
On redundant VR setups, the primary resolver being handed out to instances is the guest_ip (primary IP for the VR). This might lead to problems upon failover, at least while the DHCP lease doesn't update (because the primary resolver will be checked first until times out, however it'll be gone upon failover).
If Global Setting use_ext_dns is true, we don't want the VR to be the primary resolver at all.
CLOUDSTACK-9348: Optimize NioTest and NioConnection main loop- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef78
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
/cc @swill please review and merge.
Sorry about the previous values, they were not optimized for virtualized env. The aggressive selector.wakeup will ensure main IO loop does not get blocked even by malicious users, even for any timeout (ssl handshake etc).
* pr/1534:
CLOUDSTACK-9348: Optimize NioTest and NioConnection main loop
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef78
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
MySQLdb has been deprecated and is also not supported in Python 3.
mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.
https://dev.mysql.com/doc/connector-python/en/
Update L10N resource files with 4.7 strings from Transifex (20160502)Force "translator" mode with the transifex client.
cc @swill the new PR.
* pr/1527:
Update L10N resource files with 4.7 strings from Transifex (20160502) Force "translator" mode with the transifex client.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Will Stevens
Paul Angus
Rohit Yadav
Mike Tutkowski
rahul singal
Nick Livens
nvazquez
Syed
dean.close
Nuno Tavares
weingartner
Wido den Hollander
Milamber
