Cleanup RBD contexts after exceptions to prevent potential agent crashWe noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
* pr/1608:
improve logging readability
Cleanup rbd contexts and improve exception logging
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9348: Reduce Nio selector wait timeThis reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
/cc @swill @kiwiflyer guys can you please apply this fix in your env and test if you're still able to produce any Nio related error b/w mgmt server(s) and kvm agent(s) not being able to connect quickly. Please also watch out for any increased CPU usage (there should not be any significant change), in which case we may increase the timeout from 100ms to 200-400ms.
* pr/1601:
CLOUDSTACK-9348: Reduce Nio selector wait time
Signed-off-by: Will Stevens <williamstevens@gmail.com>
ui: show resize volume button to all users
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this bug fix all users would see resize volume button in the UI.
/cc @swill
* pr/1595:
ui: show resize volume button to all users
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVTPlease refer to CLOUDSTACK-9328 for the details.
Test Results:
==========
test_01_vpc_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_01_vpc_privategw_acl | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 108.273s
OK
* pr/1455:
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVT Bug-Id:## CLOUDSTACK-9328
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctlyBug in code set PFS to the same value (yes/no) as DPD.
file.addeq(" pfs=%s" % CsHelper.bool_to_yn(obj['dpd']))
* pr/1480:
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
Signed-off-by: Will Stevens <williamstevens@gmail.com>
We noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDsWith object store like Swift as secondary storage, if a compressed VHD is uploaded as a template, the `VHDProcessor` incorrectly calculates the virutal size leading to the template being useless. This fix tries to guess the virtual size by partially decompressing it and falls back to a sensible default which is the size of the file.
Before the fix: template.properties on Swift
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=2894447637315205059
```
After the fix
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=3145728000
```
Look at the `virutalsize` in both cases
* pr/1598:
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixes for VirtualRouters in Basic Networking, especially with mutliple ranges in VLANsDuring the last few modifications on the SystemVM scripts, it turns out quite a lot of stuff broke in our setups.
This PR fixes a number of things:
* Multiple IP's per VLAN interface are now supported & working again, including DNS, DHCP ranges, password and metadata services
* `useextdns` fixed (I had a small merge conflict with an attempt to fix this at 4.7, but these fixes are more comprehensive)
* CLOUDSTACK-8303
* Apache configs better in line with best-practices and distro-expected locations
* Added a few more helper functions & getters & setters for utility
* some minor cleanup & fixes
* pr/1547:
Remove /etc/apache2/sites-enabled/000-default in cloud-early-config
SysVM various fixes to previous refactorings * make CORS include a regular glob-matched one * fix NameVirtualHost in CsApp.py as well * even moar cleanups
SysVM: Cleanup and removal of old (and dangerous) config files * ports.conf * default & default-ssl sites * SSL config in httpd.conf * deprecated & dead setup_redundant_router in cloud-early-config
SysVM cloud-early-config: Intermediate fix for SecStore & CORS * Take setup from vhost.template rather than default(-ssl) * should move into Python CS code as well * Move CORS setup to separate conf * Modify vhost template to Optionally include the cors file * Add NameVirtualHost to vhost template for feature parity with ports.conf * Take setup from vhost.template rather than default(-ssl)
VR cloud-early-config: Commonize Apache2 common setup
VR cloud-early-config: Fix Apache2 alias cleanup
VR: consistent SSL setup, vhost is not an example, but a template
VR CsConfig: reintroduce old get_dns() behaviour for redundant non-VPC's
VR CsAddress fixes: * cleanup imports, * fix to_str(), * improve & fix service post_config logic * don't arpPing when there's no gateway
VR CsApp: Expose config to classes, move vhost confs to proper location, allow for multiple IP's per intf, sanitize servername, don't open port 53 if no DNS is foreseen
VR CsConfig: Add is_router(), is_dns(), has_dns(), has_metadata(), use_extdns(), fix get_dns() with use_extdns()
VR CsDhcp: allow multiple ranges & finite lease time (fixes CLOUDSTACK-8303)
VR CsGuestNetwork obey useextdns
VR merge.py ipalias fix & dhcpconfig stub notification
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
* 4.7:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storageIn turn this also fixes VM migration with local storage
This PR is created against 4.7 and can be forward merged to future branches also.
* pr/1596:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9404 Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted.
Issue: https://issues.apache.org/jira/browse/CLOUDSTACK-9404
In this example, I created rules with the port numbers the same as the rule numbers.
Chain ACL_INBOUND_eth2 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 225.0.0.50
ACCEPT all -- anywhere vrrp.mcast.net
DROP tcp -- anywhere anywhere tcp dpt:netstat
DROP tcp -- anywhere anywhere tcp dpt:10
DROP tcp -- anywhere anywhere tcp dpt:5
DROP tcp -- anywhere anywhere tcp dpt:3
DROP tcp -- anywhere anywhere tcp dpt:2
DROP all -- anywhere anywhere
We can see above that the rules are inverted.
After the fix:
Chain ACL_INBOUND_eth2 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 225.0.0.50
ACCEPT all -- anywhere vrrp.mcast.net
DROP tcp -- anywhere anywhere tcp dpt:2
DROP tcp -- anywhere anywhere tcp dpt:3
DROP tcp -- anywhere anywhere tcp dpt:5
DROP tcp -- anywhere anywhere tcp dpt:10
DROP tcp -- anywhere anywhere tcp dpt:netstat
DROP all -- anywhere anywhere
* pr/1581:
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
Signed-off-by: Will Stevens <williamstevens@gmail.com>
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this changes all users would see resize volume button in the UI.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
in BVT
Bug-Id:## CLOUDSTACK-9328
[CLOUDSTACK-9328]: Made changes as per the review comment from Shwetaag
[CLOUDSTACK-9328]: Made changes based on the CI results
CLOUDSTACK-9399 : NPE during deletion of host when clusterId is nullIn most network plugins, there's a Resource class which will handle the communication with the actual device / underlaying client / ... They're configured as a host, so ACS is able to send commands towards it.
When they're configured as a host, the clusterId is not filled in since it's not relevant. Hence, the NPE while deleting this host because of ```long clusterId = host.getClusterId();```
* pr/1585:
Nuage VSP : Enhancing Marvin test coverage
CLOUDSTACK-9399 : Marvin test coverage for Nuage VSP device CRUD operations
CLOUDSTACK-9399 : NPE during deletion of host when clusterId is null
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[BLOCKER][FIX] CLOUDSTACK-9409: Add role_id to cloud_usage.accountAdds role_id column to cloud_usage.account, fixes UsageDaoImpl to insert
Accounts with role_id from account table. Without the fix, usage server fails to work.
This fixes a *blocker* for 4.9.0 reported by @nvazquez (thanks!)
/cc @swill @nvazquez for review and merge thanks.
* pr/1584:
CLOUDSTACK-9409: Add role_id to cloud_usage.account
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Adds role_id column to cloud_usage.account, fixes UsageDaoImpl to insert
Accounts with role_id from account table.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Take setup from vhost.template rather than default(-ssl)
* should move into Python CS code as well
* Move CORS setup to separate conf
* Modify vhost template to Optionally include the cors file
* Add NameVirtualHost to vhost template for feature parity with ports.conf
* Take setup from vhost.template rather than default(-ssl)
Make sure that the DB drivers are loaded before creating connectionsI've digged deeper, and found out that Tomcat is really specific in how it loads the JDBC drivers apparently.
If we would be using the standard JDBC connection pooling of Tomcat (tomcat-jdbc) instead of commons-dbcp, we would have the option to specify a "driverClassName" when creating our connection.
This is not the case for commons-dbcp, which we are using within ACS.
If you check an official example of Tomcat :
https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Plain_Ol'_Java
or
https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#As_a_Resource
As you can see in the above examples, both of them specify the driverClassName.
In the underlying implementation of Tomcat, Tomcat will do ```Class.forName(driverClassName)``` which will trigger the auto-registration of the Driver.
Tomcat code :
```java
if (driver==null) {
if (log.isDebugEnabled()) {
log.debug("Instantiating driver using class: "+poolProperties.getDriverClassName()+" [url="+poolProperties.getUrl()+"]");
}
driver = (java.sql.Driver) Class.forName(poolProperties.getDriverClassName(),
true, PooledConnection.class.getClassLoader()
).newInstance();
}
```
* pr/1574:
Make sure that the DB drivers are loaded before initiating connections
Signed-off-by: Will Stevens <williamstevens@gmail.com>
travis: use ipmitool from ubuntu repository@swill @pdion891 please review, merge this once Travis is green. The packages.shapeblue.com server is unreachable which is causing a wget command to fail. This fixes this by installing `ipmitool` from ubuntu repositories.
* pr/1570:
travis: add one more smoke test
travis: use ipmitool from ubuntu repository
Signed-off-by: Will Stevens <williamstevens@gmail.com>
fix noredist build because of missing maven dependency of vmware 6.0 libfix noredist build for new vmware lib
* pr/1569:
fix noredist build because of missing maven dependency of vmware 6.0 lib
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Will Stevens
Aaron Hurt
Aaron Hurt
Rohit Yadav
Syed
Anshul Gangwar
sanjeev
Prashanth Manthena
Nick Livens
Wido den Hollander
Ronald van Zantvoort
Patrick Dube
Pierre-Luc Dion
