Current SSL protocol and ciphers used in SystemVMs are not the
recommended. To analyze it is possible to use tests such as from SSL
Labs (https://www.ssllabs.com/ssltest/). This commit changes the grade
from C to -A
* 4.9:
Do not set gateway to 0.0.0.0 for windows clients
CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
ignore bogus default gateway when a shared network is secondary the default gateway gets overwritten by a bogus one dnsmasq does the right thing and replaces it with its own default which is not good for us so check for '0.0.0.0'
Activate NioTest following changes in CLOUDSTACK-9348 PR #1549
CLOUDSTACK-9828: GetDomRVersionCommand fails to get the correct version as output Fix tries to return the output as a single command, instead of appending output from two commands
CLOUDSTACK-3223 Exception observed while creating CPVM in VMware Setup with DVS
CLOUDSTACK-9787: Fix wrong return value in NetUtils.isNetworkAWithinNetworkB
MySQL 5.7 has a more strict SQL mode by default with which CloudStack
is not compatible.
By setting the SQL Mode to a more relaxed mode on run-time we can
run without changing any SQL server settings.
Admins could also apply this to the [mysqld] section of their my.cnf:
sql_mode = 'STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* 4.9:
CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
CLOUDSTACK-9788: Fix exception listNetworks with pagesize=0
CLOUDSTACK-8663: Fixed various issues to allow VM snapshots and volume snapshots to exist together
Fix HVM VM restart bug in XenServer
This commit implements Ingress and Egress filtering for IPv6 in
Basic Networking.
It allows for opening and closing ports just as can be done with IPv4.
Rules have to be specified twice, once for IPv4 and once for IPv6, for
example:
- 22 until 22: 0.0.0.0/0
- 22 until 22: ::/0
Egress filtering works the same as with IPv4. When no rule is applied all
traffic is allowed. Otherwise only the specified traffic (with DNS being
the exception) is allowed.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
This commit adds the initial functionality for IPv6 in Basic Networking.
When a valid IPv6 CIDR is configured for the POD/VLAN the DirectPodBasedNetworkGuru
will use the EUI-64 calculation to calculate the IPv6 Address the Instance will obtain.
For this it is required that the physical routers in the Layer 2 network (POD/VLAN) send out
Router Advertisements with the same subnet as configured in CloudStack.
A example subnet could be 2001:db8::/64
Using radvd a Linux Router could send out Router Advertisements using this configuration:
interface eth0
{
MinRtrAdvInterval 5;
MaxRtrAdvInterval 60;
AdvSendAdvert on;
AdvOtherConfigFlag off;
IgnoreIfMissing off;
prefix 2001:db8::/64 {
};
RDNSS 2001:db8:ffff::53 {
};
};
A Instance with MAC Address 06:7a:88:00:00:8b will obtain IPv6 address 2001:db8:100::47a:88ff:fe00:8b
Both Windows, Linux and FreeBSD use the same calculation for their IPv6 Addresses, this is specified
in RFC4862 (IPv6 Stateless Address Autoconfiguration).
Under Linux it is mandatory that IPv6 Privacy Extensions are disabled:
$ sysctl -w net.ipv6.conf.all.use_tempaddr=0
Windows should be configured to use the MAC Address as the identifier for the EUI-64/SLAAC calculation.
$ netsh interface ipv6 set privacy state=disabled store=persistent
$ netsh interface ipv6 set global randomizeidentifiers=disabled store=persistent
The IPv6 address is stored in the 'nics' table and is then returned by the API and will be shown in the UI.
Searching for a conflicting IPv6 Address it NOT required as each IPv6 address is based on the MAC Address
of the Instance and therefor unique.
Security Grouping has not been implemented yet and will follow in a upcoming commit.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
- Bump spring-framework version to 4.x and Jetty to version that runs with JDK8
- Bump servet dependency version
- Migrate spring xmls to version 4, fixes schema locations that are 3.0
dependent in various xmls.
- Fix failing tests due to spring upgrade
(Thanks @marcaurele Marc-Aurèle Brothier for fixing them)
* Fix test DeploymentPlanningManagerImplTest
* Fix GloboDNS test
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Upgrades Maven dependency version to v1.55
- Fixes bountycastle usages and issues
- Adds timeout to jetty/annotation scanning
- Fixes servlet issue, uses servlet 3.1.0
- Downgrade javassist used by reflections to fix annotation process errors
- Make console-proxy-rdp bc dependency same as rest of the codebase
- Picks up PR #1510 by Daan
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.
* pr/1745:
CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* 4.9:
CLOUDSTACK-9470: Fix for SshHelper - test_network_acl was failing on Vmware due to a bug in sshExecute, in which value returned was null and there was still stdout to consume. This fix addresses this problem, consuming stdout peoperly to return expected value in sshExecute
Often, patch and security releases do not require schema migrations or
data migrations. However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break. With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path. This approach allows the upgrade to proceed and
for the database to properly reflect the installed version. This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.
Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes. The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers. This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers. It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.
* Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
a version that was not explicitly specified. It determines the
releases the first release before it with database migrations and uses
that list as the basis for the list for version being calculated. A
noop upgrade is then added to the list which causes no schema changes
or data migrations, but will update the database to the version.
* Adds unit tests for the upgrade path calculation logic in
DatabaseUpgradeChecker
* Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
versions of this patch
* Introduces the CloudStackVersion value object which parses and
compares three (3) and four (4) position version numbers. This class
is intended to replace com.cloud.maint.Version.
* Adds the junit-dataprovider dependency -- allowing test data to be
concisely generated separately from the execution of a test case.
Used extensively in the CloudStackVersionTest.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9348: NioConnection improvementsReopened PR with squashed changes for a re-review and testing after https://github.com/apache/cloudstack/pull/1493 and sub-sequent PRs got reverted
* pr/1549:
CLOUDSTACK-9348: NioConnection improvements
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Taking fast and efficient volume snapshots with XenServer (and your storage provider)A XenServer storage repository (SR) and virtual disk image (VDI) each have UUIDs that are immutable.
This poses a problem for SAN snapshots, if you intend on mounting the underlying snapshot SR alongside the source SR (duplicate UUIDs).
VMware has a solution for this called re-signaturing (so, in other words, the snapshot UUIDs can be changed).
This PR only deals with the CloudStack side of things, but it works in concert with a new XenServer storage manager created by CloudOps (this storage manager enables re-signaturing of XenServer SR and VDI UUIDs).
I have written Marvin integration tests to go along with this, but cannot yet check those into the CloudStack repo as they rely on SolidFire hardware.
If anyone would like to see these integration tests, please let me know.
JIRA ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-9281
Here's a video I made that shows this feature in action:
https://www.youtube.com/watch?v=YQ3pBeL-WaA&list=PLqOXKM0Bt13DFnQnwUx8ZtJzoyDV0Uuye&index=13
* pr/1403:
Faster logic to see if a cluster supports resigning
Support for backend snapshots with XenServer
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- Unit test to demonstrate denial of service attack
The NioConnection uses blocking handlers for various events such as connect,
accept, read, write. In case a client connects NioServer (used by
agent mgr to service agents on port 8250) but fails to participate in SSL
handshake or just sits idle, this would block the main IO/selector loop in
NioConnection. Such a client could be either malicious or aggresive.
This unit test demonstrates such a malicious client that can perform a
denial-of-service attack on NioServer that blocks it to serve any other client.
- Use non-blocking SSL handshake
- Uses non-blocking socket config in NioClient and NioServer/NioConnection
- Scalable connectivity from agents and peer clustered-management server
- Removes blocking ssl handshake code with a non-blocking code
- Protects from denial-of-service issues that can degrade mgmt server responsiveness
due to an aggressive/malicious client
- Uses separate executor services for handling ssl handshakes
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This reverts commit 7ce0e10fbc, reversing
changes made to 29ba71f2db.
This was reverted because it seemed to be related to an issue
when doing a DeployDC, causing an `addHost` error.
This reverts commit 9f970f28b1, reversing
changes made to 6d0c92be72.
This was reverted because it seemed to be related to an issue
when doing a DeployDC, causing an `addHost` error.
This reverts commit f88cb88097, reversing
changes made to 688522ecd4.
This was reverted because it seemed to be related to an issue
when doing a DeployDC, causing an `addHost` error.
- Fixes oobm integration test to skip if known ipmitool bug is hit
- Fixes ProcessTest unit test case to use sleep
- Removes redundant unit test that covers code in ProcessTest
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Remove classes with no referencesI used UCDetector (http://www.ucdetector.org/) as a plugin for Eclipse. With this tool, I discovered a lot of code without any reference (variables, methods and classes).
Following the work that was done at [https://github.com/apache/cloudstack/pull/1448]; this pull request had the goal of removing some of these classes. To check if I wasn't missing anything I searched for any file that could reference some of those classes. As I haven't found any way of these classes being used, they were removed. Note that some of them I found other references, but references such as commented lines or tests, nothing that could indicate their use (as XML files configuring beans or another class instantiating an object with "new").
Waiting for tests. Please tell me if I am missing something.
Removed Classes:
- org.apache.cloudstack.framework.jobs.JobCancellationException (**Note:** removed
variable JobCancellationException in com.cloud.utils.SerialVersionUID)
- org.apache.cloudstack.ldap.NoSuchLdapUserException (**Note:** removed test file
/cloud-plugin-user-authenticator-ldap/test/groovy/org/apache/cloudstack/ldap/NoSuchLdapUserExceptionSpec.groovy)
- com.cloud.agent.api.storage.CreateVolumeOVAAnswer
- com.cloud.exception.MissingParameterValueException
- org.apache.cloudstack.api.response.StatusResponse
- org.apache.cloudstack.api.response.VolumeDetailResponse
- org.apache.cloudstack.api.response.UpgradeVmResponse
- org.apache.cloudstack.api.response.AddIpToVmNicResponse
- org.apache.cloudstack.api.response.TemplateZoneResponse (**Note:** at
org.apache.cloudstack.api.response.TemplateResponse, there is this
comment "To avoid breaking backwards compatibility, we still treat a
template at different zones as different templates, so not embedding
template_zone information in this TemplateZoneResponse set. `private
Set<TemplateZoneResponse> zones;`" but right now it is not used)
- org.apache.cloudstack.api.response.NicDetailResponse
* pr/1453:
Removed classes with no reference
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Disable NioTestHistorically NioTest has caused issue in CI environment and several developer machines due to network requirements which could be disabled by firewall or security enforcers such as selinux. This disables the test once again using a historic commit 881a6e1
Signed-off-by: Rohit Yadav <rohit.yada
To build and just run this test: mvn clean install -pl utils -Dtest=NioTest
* pr/1538:
utils: Disable NioTest
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Disable the NioTest, it is far to dependent on the network configuration of the
machine running the test. Cherry-picked change from a historic commit where
this test was disabled.
(cherry picked from commit 881a6e164e)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This feature allows root administrators to define new roles and associate API
permissions to them.
A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
Admin, Domain Admin and User) which maintains this association by requiring
all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
removal of roles and/or modifications of permissions, without the need
of restarting management server(s)
Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
deployments will continue to use the older static role based api access checker
with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
to add existing set of permissions to the default roles. cloud.account
will have a new role_id column which will be populated based on default roles
as well
Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef78
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9348: Use non-blocking SSL handshake in NioConnection/Link- Uses non-blocking socket config in NioClient and NioServer/NioConnection
- Scalable connectivity from agents and peer clustered-management server
- Removes blocking ssl handshake code with a non-blocking code
- Protects from denial-of-service issues that can degrade mgmt server responsiveness
due to an aggressive/malicious client
- Uses separate executor services for handling connect/accept events
Changes are covered the NioTest so I did not write a new test, advise how we can improve this. Further, I tried to invest time on writing a benchmark test to reproduce a degraded server but could not write it deterministic-ally (sometimes fails/passes but not always). Review, CI testing and feedback requested /cc @swill @jburwell @DaanHoogland @wido @remibergsma @rafaelweingartner @GabrielBrascher
* pr/1493:
CLOUDSTACK-9348: Use non-blocking SSL handshake
CLOUDSTACK-9348: Unit test to demonstrate denial of service attack
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- Uses non-blocking socket config in NioClient and NioServer/NioConnection
- Scalable connectivity from agents and peer clustered-management server
- Removes blocking ssl handshake code with a non-blocking code
- Protects from denial-of-service issues that can degrade mgmt server responsiveness
due to an aggressive/malicious client
- Uses separate executor services for handling ssl handshakes
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The NioConnection uses blocking handlers for various events such as connect,
accept, read, write. In case a client connects NioServer (used by
agent mgr to service agents on port 8250) but fails to participate in SSL
handshake or just sits idle, this would block the main IO/selector loop in
NioConnection. Such a client could be either malicious or aggresive.
This unit test demonstrates such a malicious client that can perform a
denial-of-service attack on NioServer that blocks it to serve any other client.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
4.9 mvn version safeupgradeonlyUpgrades maven dependencies versions that can be safely upgraded without breaking console-proxy/crypto usage.
Bisected changes from: https://github.com/apache/cloudstack/pull/1397
cc @swill @DaanHoogland
* pr/1510:
maven: fix dependency version support by JDK7
further maven dependency updates from Daan
framework/quota: fix checkstyle issue
maven: Upgrade dependency versions
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9352: Test fails in Widows as the file separator "/" is different from "\"**Problem:**
File separator in windows ("\") is different from the expected in the test ("/"); thus, the test *com.cloud.utils.SwiftUtilTest.testSplitSwiftPath()* will fail in Windows systems.
The problem is that the input of the test is "*container/object*" but the tested method uses the *File.separator* (that depends from the OS), in windows systems the tested method (*com.cloud.utils.SwiftUtil.splitSwiftPath(String)*) looks for a "\", as the string does not contain "\" it returns an empty string and consequently results in a test failure.
**Solution:**
Create a string `String input = "container" + File.separator + "object";`, with that the test will validate the tested method verifying if the method splits the string around matches of the given regular expression (in this case *File.separator*).
*JIRA link: https://issues.apache.org/jira/browse/CLOUDSTACK-9352*
* pr/1498:
Test fails in Widows as the file separator "/" is different from "\"
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Updated most dependencies to latest minor releases, EXCEPT:
- Gson 2.x
- Major spring framework version
- Servlet version
- Embedded jetty version
- Mockito version (beta)
- Mysql lib minor version upgrade (breaks mysql-ha plugin)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This closes#561
CLOUDSTACK-8611:Handle SSH if server "forget" to send exit statusContinuing the work started by @likitha, I did not cherry-picked the
commit (b9181c689e) from PR <https://github.com/apache/cloudstack/pull/561> due to the fact that the path of that SshHelper class was different of the current SshHelper; that is because the fact that by cherry-picking it would seem that I had changed all the class as the code is from another file.
I made some changes from the cherry-picked commit adding @wilderrodrigues suggestions (create simple methods to have reusable code, make unit tests and create the `WAITING_OPEN_SSH_SESSION` variable to manipulate with the delay of 1000 milliseconds).
Also, I tried to simplify the logic by assuming that ....
if ((conditions & ChannelCondition.EXIT_STATUS) != 0) {
if ((conditions & (ChannelCondition.STDOUT_DATA | ChannelCondition.STDERR_DATA)) == 0) {
break;
}
}
... is the same as `((conditions & ChannelCondition.EXIT_STATUS) != 0) && ((conditions & (ChannelCondition.STDOUT_DATA | ChannelCondition.STDERR_DATA)) == 0)`. This expression has the following results according to each possible condition.
|Condition|Value|result
|-----------------|-------|------|
TIMEOUT | 0000001|false
CLOSED | 0000010 |false
STDERR_DATA | 0000100 | false
STDERR_DATA | 0001000 | false
EOF | 0010000 | false
EXIT_STATUS | 0100000 | **true**
EXIT_SIGNAL | 1000000 | false
After testing all the possibilities we can note that the condition of `(conditions & ChannelCondition.EXIT_STATUS) != 0` is sufficient; thus, the simplified "if" conditional can be:
`if ((conditions & ChannelCondition.EXIT_STATUS) != 0) {
break;
}`
This proposed work can be explained by quoting @likitha:
>CheckS2SVpnConnectionsCommand execution involves executing a script (checkbatchs2svpn.sh) in the virtual router. Once CS has opened a session to a virtual router and executed a script in the router, it waits indefinitely till the session either times out or the exit status of the remote process is available. But it is possible that an EOF is reached by the process in the router and the router never set the exit status.
>References -
>1. Some servers never send the exit status, or occasionally "forget" to do so (http://grepcode.com/file/repo1.maven.org/maven2/org.jvnet.hudson/trilead-ssh2/build212-hudson-1/com/trilead/ssh2/ChannelCondition.java).
>2. Get the exit code/status from the remote command - if available. Be careful - not all server implementations return this value - (http://grepcode.com/file/repo1.maven.org/maven2/org.jvnet.hudson/trilead-ssh2/build212-hudson-1/com/trilead/ssh2/Session.java#Session.waitForCondition%28int%2Clong%29).
* pr/1459:
Handle SSH if server "forget" to send exit status
Signed-off-by: Will Stevens <williamstevens@gmail.com>
File separator in windows is different from linux (the expected in the
test); thus, the test
*com.cloud.utils.SwiftUtilTest.testSplitSwiftPath()* will fail in
windows. The problem is that the input of the test is
*"container/object"* but the tested method uses the *File.separator*
(that depends from the OS), in the windows the tested method
(*com.cloud.utils.SwiftUtil.splitSwiftPath(String)*) looks for a "\" in
windows systems, resulting in an empty string and consequently a failure
in the test.
Some solutions:
- the simple way is to create a string `String input = "container" +
File.separator + "object";`, thus independent of the OS, the test will
succeed.
- a tricky solution is to mock the final static variable
*File.separator* and return "/".
I picked the easy way.
Continued the work started by https://github.com/likitha
commit (b9181c6) from PR #561.
CS waits indefinitely for CheckS2SVpnConnectionsComm and to return.
While remote executing commands through ssh, handle channel condition of
EOF because we wait for the the condition.
The SshHelper of the PR #561 is of another path from the
current master, its path was
https://github.com/likitha/cloudstack/commits/CLOUDSTACK-8611/utils/src/com/cloud/utils/ssh/SshHelper.java;
thus, although this commit brings changes from PR #561, I did not
cherry-picked to keep the master file, otherwise it would look that I
had changed all the file.
by me.
- Improve disk chain usage while attaching, migrating disks
- Gets root disk controller based diskDeviceBusName from volume's chain info
- Refactor and move VirtualMachineDiskInfo to cloud-utils
- Allows mixing of scsi controller types
- Fixes a NPE case with map passed as null, for example in case of detach volume
command
- Use a osdefault translator that allow use of recent os types added (enums of
which) are not available in the sdk
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixed Profiler's unit tests bugs.### **Problem:**
The TestProfiler class was using Java Thread methods to test the
Profiler's functionality. That was causing the tests to fail sometimes
since the JVM's thread priority could be low on some OS.
### **Fix:**
Using PowerMockito to mock the System calls, the threads could be
removed. This makes the tests considerably faster, OS independent and
still guarantees the correct implementation of the Profiler class.
The changes on the Profiler's class was only to shorten the class's line
size by not assigning the return value to a variable returning it
straight out.
* pr/1445:
Fixed changes to match code conventions
Fixed Profiler's unit tests bugs.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- org.apache.cloudstack.framework.jobs.JobCancellationException (removed
variable JobCancellationException in com.cloud.utils.SerialVersionUID)
- org.apache.cloudstack.ldap.NoSuchLdapUserException (removed test file
/cloud-plugin-user-authenticator-ldap/test/groovy/org/apache/cloudstack/ldap/NoSuchLdapUserExceptionSpec.groovy)
- com.cloud.agent.api.storage.CreateVolumeOVAAnswer
- com.cloud.exception.MissingParameterValueException
- org.apache.cloudstack.api.response.StatusResponse
- org.apache.cloudstack.api.response.VolumeDetailResponse
- org.apache.cloudstack.api.response.UpgradeVmResponse
- org.apache.cloudstack.api.response.AddIpToVmNicResponse
- org.apache.cloudstack.api.response.TemplateZoneResponse (at
org.apache.cloudstack.api.response.TemplateResponse, there is this
comment "To avoid breaking backwards compatibility, we still treat a
template at different zones as different templates, so not embedding
template_zone information in this TemplateZoneResponse set. `private
Set<TemplateZoneResponse> zones;`" but right now it is not used)
- org.apache.cloudstack.api.response.NicDetailResponse
Add ability to download templates in SwiftThis PR adds the ability to download templates when using Swift as a secondary storage. Uses the "temp_url" feature of Swift so that tempates can be downloaded without authenticaiton.
* pr/1332:
Add ability to download templates in Swift
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Problem:
The TestProfiler class was using Java Thread methods to test the
Profiler's functionality. That was causing the tests to fail sometimes
since the JVM's thread priority could be low on some OS.
Fix:
Using PowerMockito to mock the System calls, the threads could be
removed. This makes the tests considerably faster, OS independent and
still guarantees the correct implementation of the Profiler class.
The changes on the Profiler's class was only to shorten the class's line
size by not assigning the return value to a variable returning it
straight out.
* 4.7:
Fix unable to setup more than one Site2Site VPN Connection
FIX S2S VPN rVPC: Check only redundant routers in state MASTER
PEP8 of integration/smoke/test_vpc_vpn
Add S2S VPN test for Redundant VPC
Make integration/smoke/test_vpc_vpn Hypervisor independant
FIX VPN: non-working ipsec commands
[UI] MADNESS
[DB] Add force_encap field to s2s_customer_gateway table
[ROUTER] Add forceencaps field to python router ipsec config method
[TEST] unittest needs rework
[MARVIN] Add forceencap field to VpnCustomerGateway class in marvin base
[CORE] Add Force UDP Encapsulation option to Site2Site VPN
CLOUDSTACK-9186: Root admin cannot see VPC created by Domain admin user
CLOUDSTACK-9192: UpdateVpnCustomerGateway is failing
CLOUDSTACK-6485 prevent ip asignment of private gw iface
CLOUDSTACK-9204 Do not error when staticroute is already gone
make both check lines consistent
CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh
CLOUDSTACK-9202 Bump ssh timeout
Factor in inaccurancy of System.nanoTime in class ProfilerThe time diff measurement by the class Profiler, is done by System.nanoTime().
System.nanoTime() is not 100% accurate (lots of info on this on the web). The inaccurance seems to vary/depend with different dependencies (multi-core/OS/kernels).
Problem:
In my case the code which measures `Thread.sleep(ONE_SECOND);` ~~takes~~ measures 999 milliseconds, which breaks the test expecting the sleep of 1000 milliseconds to ~~take~~ measure at least 1000 milliseconds.
Solution:
A `MARGIN` (of 100 milliseconds) is already present to factor in some time for overhead/surrounding code which adds time. Another margin could be added to factor in the possible time measurement error.
To limit additional lines of code, the constant `ONE_SECOND`, which is used by the testcase (sleep), is increased from 1000 milliseconds to `1000l + (MARGIN / 10)` (effectively 1010 milliseconds).
Thereby the inaccurancy on my system of ~1% ("faster") is caught as the `sleep(1010)` is measured as '1009' milliseconds.
* pr/1324:
Factor in inaccurancy of System.nanoTime
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9086: ACS allows to create isolated networks with invalide gateway IP address - Fixed and Test cases added
Problem: There was no check for the network and broadcast IP addresses in the case where we provide the gateway and netmask while creating the isolated network. As a result the provided IP gets assigned to the eth0 interface of the VR.
Note: This is in continuation of PR #1125 which I closed.
* pr/1188:
CLOUDSTACK-9086: ACS allows to create isolated networks with invalide gateway IP address - Fixed and Test cases added
Signed-off-by: Remi Bergsma <github@remi.nl>
Cwe 190coverity warnings of this type adressed. Some where dismissed and maybe with reason but it seemed possible to remove them and hence obligatory ;p
* pr/1057:
move back to original contract of isNetworksOverlap()
Changed the behavior of methods that use NetUtils.cidrToLong(String)
CWE-190 unit test for extremes of long netMaskFromCidr(long)
CWE-190 netmask as long form cidr-size as method
CID-1116482 cidrToLong cleanup of bitshift problem
CID-1116483 cidr to netmask bitshifts guarded with casts
CID-1116484 cast to long and use long as cidrsize type and simpel test
CID-1116485: cast cidr during bit shifting and simple test included
CID-1175714 casts before bit shift
Signed-off-by: Daan Hoogland <daan@onecht.net>
Removed unnecessary @Local annotations and their respective importsFollowing @rafaelweingartner 's findings in PR #714 that many spring beans contained an @Local annotation, we've decided to remove said annotations and their imports from the ComponentLifecycleBase class and its subclasses seeking a reduction of a few hundred lines of useless code.
I had already opened a pull request for this (#853) but at some point my commit disappeared from the PR entirely, showing no new changes in code, which caused it to be merged automatically (with no changes).
* pr/1102:
Removed unnecessary @Local annotations and their respective imports from the ComponentLifecycleBase class and its subclasses.
Signed-off-by: Remi Bergsma <github@remi.nl>
* 4.6:
CLOUDSTACK-9052 Shuffling the password to avoid having a subset of characters in fixed positions.
Refactor package.sh: * lint * adjust exit codes (1 for usage, 2 for maven, 3 for rpmbuild) * variable naming consistency * add option for package release version * revise synopsis and usage
Shuffling the password to avoid having a subset of characters in fixed positions.Related to CLOUDSTACK-9052.
I am shuffling the characters in the password, to avoid having a certain char type in fixed positions. I modified the tests accordingly to only check that the different character types are present.
I think it would be good to remove the hard requirement to have at least one of digits, upper-case, and lowercase chars, as it reduces the number of possible combinations passwords can take. What do you think?
* pr/1058:
CLOUDSTACK-9052 Shuffling the password to avoid having a subset of characters in fixed positions.
Signed-off-by: Remi Bergsma <github@remi.nl>
Given that the method com.cloud.utils.net.NetUtils.cidrToLong(String)
now throws an exception when receiving null or empty cidrs, there is the
need to change methods that use it. Those methods were changed and test
cases created.
The S3 implementation is far from finished, this commit focusses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimalisations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
These were lacking, but this helper is used in various places
inside the KVM code.
Some simple tests to verify the helper is doing what we expect it
to do.
CLOUDSTACK-8819: Added QCOW2 virtual size checking for S3.- Cleaned up S3TemplateDownloader
- Created static QCOW2 utils class.
- Reformatted some parts of DownloadManagerImpl
* pr/795:
Added Unit Tests for QCOW2Utils.
Fixed Findbugs issue introduced by 1c6378ec00 PR #795.
Added QCOW2 virtual size checking for S3.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
CLOUDSTACK-8762: Check to confirm disk activity before starting a VMImplements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* pr/753:
CLOUDSTACK-8762: Check to confirm disk activity before starting a VM
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Proposal for an improved CglibThrowableRendererFollowing our discussions on mailing list, here is a PR with a proposal for an improvement on CglibThrowableRenderer. We would still have to define a number to limit the stack traces of causes (I randomly chose 3, just to get the code running). The test case is also not good in my opinion, but I ran out of ideas (waiting for suggestions on that).
* pr/762:
Changed the test case to test each of the log traces in the array list.
Proposal for an improved CglibThrowableRenderer
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
This reverts commit cd7218e241, reversing
changes made to f5a7395cc2.
Reason for Revert:
noredist build failed with the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloud-plugin-hypervisor-vmware: Compilation failure
[ERROR] /home/jenkins/acs/workspace/build-master-noredist/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[484,12] error: non-static variable logger cannot be referenced from a static context
[ERROR] -> [Help 1]
even the normal build is broken as reported by @koushik-das on dev list
http://markmail.org/message/nngimssuzkj5gpbz
use fakelocalhost in BasicRestClientTestBasicRestClientTest will fail if the port 443 is open on localhost.
use fakelocalhost instead of localhost can fix it.
Please see the latest discussion in https://github.com/apache/cloudstack/pull/737
* pr/752:
use fakelocalhost in BasicRestClientTest
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
Implements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- The difference between 2 consecultive calls to System.nanoTime() returns a resolution that depends on the OS implementation. Due to that, the test was failing in an inttermmitent manner
- Today we decided to remove the test since it was not testing a direct feature of ACS
- Add -noverify JVM arg to surefire plugin, to allow Powermockito to
de-encapsulate private methods
- Add dependency on cloud-utils test-jar to use custom HttpRequest
matchers
- All HTTP protocol activities are now handled by RestClient
- This service is now only responsible for creating requests, and
dispatching them to the client
- Provides a Simple API for creating, updating, retrieving and deleting
objects
- Upgrade version of HTTP Components to 4.5
- Add helper class to create Http clients
- Add helper class to build http requests
- Add enum with the different Http Methods
- Add constants class for HTTP related values
CLOUDSTACK-8669: create volume failed due to null charsetAdded a new private method getCharSetFromConnection() which checks if
the connection charset is null and if it is null, returns
StringUtils.getPreferredCharset
regression caused by commit f03411ca04
* pr/695:
CLOUDSTACK-8669: create volume failed due to null charset
CLOUDSTACK-8669: making StringUtils.getDefaultCharset protected
Signed-off-by: Remi Bergsma <github@remi.nl>
Cloudstack 8656: do away with more silently ignoring exceptions.a lot of messages added.
some restructuring for test exception assertions and try-with-resource blocks
* pr/654: (29 commits)
CLOUDSTACK-8656: more logging instead of sysout
CLOUDSTACK-8656: use catch block for validation
CLOUDSTACK-8656: class in json specified not found
CLOUDSTACK-8656: removed unused classes
CLOUDSTACK-8656: restructure of tests
CLOUDSTACK-8656: reorganise sychronized block
CLOUDSTACK-8656: restructure tests to ensure exception throwing
CLOUDSTACK-8656: validate the throwing of ServerApiException
CLOUDSTACK-8656: logging ignored exceptions
CLOUDSTACK-8656: try-w-r removes need for empty catch block
CLOUDSTACK-8656: try-w-r instead of clunckey close-except
CLOUDSTACK-8656: deal with empty SQLException catch block by try-w-r
CLOUDSTACK-8656: unnecessary close construct removed
CLOUDSTACK-8656: message about timed buffer logging
CLOUDSTACK-8656: message about invalid number from store
CLOUDSTACK-8656: move cli test tool to separate file
CLOUDSTACK-8656: exception is the rule for some tests
CLOUDSTACK-8656: network related exception logging
CLOUDSTACK-8656: reporting ignored exceptions in server
CLOUDSTACK-8656: log in case we are on a platform not supporting UTF8
...
Signed-off-by: Remi Bergsma <github@remi.nl>
* pr/547:
CLOUDSTACK-8601. VMFS storage added as local storage can be re-added as shared storage. Fail addition of a VMFS shared storage pool in case it has already been added as local storage in CS.
Signed-off-by: Mike Tutkowski <mike.tutkowski@solidfire.com>
* pr/651:
CLOUDSTACK-8703: Fixed issue when listing directory on S3, it would only return objectSummaries when the anwser from the S3 System was truncated.
Signed-off-by: Remi Bergsma <github@remi.nl>
The newer SDKs API changed which causes our S3 Template Downloader to never complete.
Although we should fix the Template Downloader we can revert to the old AWS SDK for now.
The fix on the longer run will be rewriting the S3 Template Downloader.
Two methods had to be disabled for now since the old SDK does not support them. They can
be re-enabled when the Template Downloader has been fixed.
- This test was added in order to cover the new StringUtils.getDefaultCharset().
- One cannot be sure that StringUtils.getPreferredCharset() will always be equals to Charset.defaultCharset()
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
This closes#616
- Changing the test to call isUtf8Supported() before checking if the preferred charset is actually UTF-8
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
- Login is based on sessionkey HttpOnly Cookie
- ApiServlet does login verification using sessionKey from both the request cookies
and the API parameters. In both cases, if either or both are passed they should
match the sessionKey stored in the current session of the HttpRequest
- UI: it no longer needs to read or set sessionkey cookie
- UI: it no longer needs to return g_sessionKey value in the API requests, though
to support a sso mechanism g_sessionKey is still passed in the API is not null
- Secure jsessionid cookie is set to be HttpOnly and Secure
- SAML login should also set HttpOnly cookie before redirecting to UI
- SAML: listIdps & getSPMetadata APIs are readonly now, won't log out a logged in user
Performed tests (login, saml login if applicable, page refreshes, opening
multiple tabs, logout) with following combinations:
- SAML disabled, normal auth as admin, domain-admin and user
- SAML enabled, normal auth as admin, domain-admin and user; and saml sso as
admin, domain-admin and user
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This closes#574
This closes#308
* Move config options to SAML plugin
This moves all configuration options from Config.java to SAML auth manager. This
allows us to use the config framework.
* Make SAML2UserAuthenticator validate SAML token in httprequest
* Make logout API use ConfigKeys defined in saml auth manager
* Before doing SAML auth, cleanup local states and cookies
* Fix configurations in 4.5.1 to 4.5.2 upgrade path
* Fail if idp has no sso URL defined
* Add a default set of SAML SP cert for testing purposes
Now to enable and use saml, one needs to do a deploydb-saml after doing a deploydb
* UI remembers login selections, IDP server
- CLOUDSTACK-8458:
* On UI show dropdown list of discovered IdPs
* Support SAML Federation, where there may be more than one IdP
- New datastructure to hold metadata of SP or IdP
- Recursive processing of IdP metadata
- Fix login/logout APIs to get new interface and metadata data structure
- Add org/contact information to metadata
- Add new API: listIdps that returns list of all discovered IdPs
- Refactor and cleanup code and tests
- CLOUDSTACK-8459:
* Add HTTP-POST binding to SP metadata
* Authn requests must use either HTTP POST/Artifact binding
- CLOUDSTACK-8461:
* Use unspecified x509 cert as a fallback encryption/signing key
In case a IDP's metadata does not clearly say if their certificates need to be
used as signing or encryption and we don't find that, fallback to use the
unspecified key itself.
- CLOUDSTACK-8462:
* SAML Auth plugin should not do authorization
This removes logic to create user if they don't exist. This strictly now
assumes that users have been already created/imported/authorized by admins.
As per SAML v2.0 spec section 4.1.2, the SP provider should create authn requests using
either HTTP POST or HTTP Artifact binding to transfer the message through a
user agent (browser in our case). The use of HTTP Redirect was one of the reasons
why this plugin failed to work for some IdP servers that enforce this.
* Add new User Source
By reusing the source field, we can find if a user has been SAML enabled or not.
The limitation is that, once say a user is imported by LDAP and then SAML
enabled - they won't be able to use LDAP for authentication
* UI should allow users to pass in domain they want to log into, though it is
optional and needed only when a user has accounts across domains with same
username and authorized IDP server
* SAML users need to be authorized before they can authenticate
- New column entity to track saml entity id for a user
- Reusing source column to check if user is saml enabled or not
- Add new source types, saml2 and saml2disabled
- New table saml_token to solve the issue of multiple users across domains and
to enforce security by tracking authn token and checking the samlresponse for
the tokens
- Implement API: authorizeSamlSso to enable/disable saml authentication for a
user
- Stubs to implement saml token flushing/expiry
- CLOUDSTACK-8463:
* Use username attribute specified in global setting
Use username attribute defined by admin from a global setting
In case of encrypted assertion/attributes:
- Decrypt them
- Check signature if provided to check authenticity of message using IdP's
public key and SP's private key
- Loop through attributes to find the username
- CLOUDSTACK-8538:
* Add new global config for SAML request sig algorithm
- CLOUDSTACK-8539:
* Add metadata refresh timer task and token expiring
- Fix domain path and save it to saml_tokens
- Expire hour old saml tokens
- Refresh metadata based on timer task
- Fix unit tests
This closes#489
(cherry picked from commit 20ce346f3a)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Conflicts:
client/WEB-INF/classes/resources/messages_hu.properties
plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixCheckHealthCommandWrapper.java
plugins/user-authenticators/saml2/src/org/apache/cloudstack/api/command/SAML2LoginAPIAuthenticatorCmd.java
ui/scripts/ui-custom/login.js
- New implementation uses nanoseconds. Due to that, the places where the Profiler is used as a Monitor and/or
a stopwatch will suffer with the difference in the return
- Also added a getDuration(), which returns the time in nanoseconds in case someone wants to use it instead
- Added an extra test to check if the getDuration() works fine with nanoseconds
- Fixed the test that checks the time in milliseconds: I added an error margin to cover the test better
Signed-off-by: Daan Hoogland <daan@onecht.net>
- System.nanoTime() is the best way to measure elapsed time in Java.
- It gives a resolution on the order of microseconds
The System.currentTimeMillis() is used when calculating absolut time.
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
Sudhansu
gabrascher
Rajani Karuturi
Rajani Karuturi
Rajani Karuturi
Harikrishna Patnala
Aaron Hurt
Ross Hayden
Wido den Hollander
Marc-Aurèle Brothier
Jayapal
Wei Zhou
Will Stevens
Rohit Yadav
Abhinandan Prateek
John Burwell
nvazquez
Aaron Hurt
Nick Livens
Mike Tutkowski
Daan Hoogland
weingartner
Syed
Alexandre Limas Santana
Remi Bergsma
Bob
nvazquez
Kshitij Kansal
cirstofolini
Boris Schrijver
nnesic
wilderrodrigues
Rajani Karuturi
Miguel Ferreira
Daan Hoogland
Likitha Shetty
Rafael da Fonseca