This introduces two new cloudstack packages: marvin and integration-tests.
The two packages will make it easier for CI systems to install Marvin for a
specific cloudstack release/build and run integration tests that are specific
for that version/build.
- maven: add explicit juniper-contrail-api maven repository
- marvin: build source distribution for both install and package mvn phases
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9334: Support jenv and pyenv to manage Java and Python versions * Adds .java-version to specify Java 1.7 as the JDK version to use
for CloudStack development. jenv does not support comments in
this file. Therefore, no license header is present.
* Adds .python-vresion to specify Python 2.7.11 as the Python to
use for CloudStack development. pyenv does not support comments
in this file. Therefore, no license header is present.
* Adds requirements.txt to specify the default dependencies used
for CloudStack development. It does not include Marvin's
dependencies because it's bundle already defines the dependencies
which are installed with Marvin.
* Updates INSTALL.md to explain how to utilize jenv and pyenv for
CloudStack development
* pr/1460:
CLOUDSTACK-9334: Support jenv and pyenv to manage Java and Python versions
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This fixes several Jenkins failures as previous runs don't cleanup this
file created by one of the unit tests.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This would fix regression from recent mvn version changes. Without this
patch users get redirected to error.jsp as jstl-1.2 jar is not installed
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Fix jetty dependency that is compatible with Java7
- Upgrade minor revisions of dependencies
- Upgrade vmware mvn sdk dependency to 6.0
- Downgrade bounty castle version to 1.46 (same as before)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Updated most dependencies to latest minor releases, EXCEPT:
- Gson 2.x
- Major spring framework version
- Servlet version
- Embedded jetty version
- Mockito version (beta)
- Mysql lib minor version upgrade (breaks mysql-ha plugin)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Improve disk chain usage while attaching, migrating disks
- Gets root disk controller based diskDeviceBusName from volume's chain info
- Refactor and move VirtualMachineDiskInfo to cloud-utils
- Allows mixing of scsi controller types
- Fixes a NPE case with map passed as null, for example in case of detach volume
command
- Use a osdefault translator that allow use of recent os types added (enums of
which) are not available in the sdk
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Adds .java-version to specify Java 1.7 as the JDK version to use
for CloudStack development.
* Adds .python-vresion to specify Python 2.7.11 as the Python to
use for CloudStack development.
* Adds requirements.txt to specify the default dependencies used
for CloudStack development. It does not include Marvin's
dependencies because it's bundle already defines the dependencies
which are installed with Marvin.
* Adds .java-version and .python-version to the RAT excludes
because neither jenv or pyenv support comments in these files
* Updates INSTALL.md to explain how to utilize jenv and pyenv for
CloudStack development
Prevent live-lock in NSX API clientThe NSX api client relies on a sequence of responses to identify the need to authenticate and to follow redirects. In order to avoid live-locks, the NSX API client has a counter that will abort the execution after 5 consecutive requests that do not produce a Success (200) response.
When a NSX controller enters a faulty state it can allow authentication requests but deny any other API call. In such cases the NSX API client will consider the denied request a reason to follow a redirect and will enter into a live-lock (because the actual redirection is not happening).
This PR changes the NSX API client to no reset it's counter on a Success response from an authentication request. That is, the counter is only reset if another type of API call yields a Success response.
In addition, this PR also:
* changes the configuration of the license-maven-plugin to ignore files generated by pmd
* moves the NSX marvin test to a plugins folder
* refactors the NSX marvin test to reduce duplication
* adds an extra test case to the NSX marvin test that checks that NSX tunnels are properly created
* pr/1178:
Test NSX tunnel in guest network
Refactor test cases to reduce duplication
Use logger to print debug messages during test
Move NSX integrationt test to new plugins folder
Ignore pmd generated files during license check
Fix NSX rest client to not reset execution counter after a login
Add test for NSX plugin that simulates a live lock
Signed-off-by: Remi Bergsma <github@remi.nl>
Quota service while allowing for scalability will make sure that the cloud is
not exploited by attacks, careless use and program errors. To address this
problem, we propose to employ a quota-enforcement service that allows resource
usage within certain bounds as defined by policies and available quotas for
various entities. Quota service extends the functionality of usage server to
provide a measurement for the resources used by the accounts and domains using a
common unit referred to as cloud currency in this document. It can be configured
to ensure that your usage won’t exceed the budget allocated to accounts/domain
in cloud currency. It will let user know how much of the cloud resources he is
using. It will help the cloud admins, if they want, to ensure that a user does
not go beyond his allocated quota. Per usage cycle if a account is found to be
exceeding its quota then it is locked. Locking an account means that it will not
be able to initiat e a new resource allocation request, whether it is more
storage or an additional ip. Needless to say quota service as well as any action
on the account is configurable.
Changes from Github code review:
- Added marvin test for quota plugin API
- removed unused commented code
- debug messages in debug enabled check
- checks for nulls, fixed access to member variables and feature
- changes based on PR comments
- unit tests for UsageTypes
- unit tests for all Cmd classes
- unit tests for all service and manager impls
- try-catch-finally or try-with-resource in dao impls for failsafe db switching
- remove dead code
- add missing quota calculation case (regression fixed)
- replace tabs with spaces in pom.xmls
- quota: though default value for quota_calculated is 0, the usage server
makes it null while entering usage entries. Flipping the condition so
as to acocunt for that.
- quotatypes: fix NPE in quota type
- quota framework test fixes
- made statement period configurable
- changed default email templates to reflect the fact that exhausted quota may not result in a locked account
- added quotaUpdateCmd that refreshes quota balances and sends alerts and statements
- report quotaSummary command returns quota balance, quota usage and state for all account
- made UI framework changes to allow for text area input in edit views
- process usage entries that have greater than 0 usage
- orocess quota entries only if tariff is non zero
- if there are credit entries but no balance entry create a dummy balance entry
- remove any credit entries that are before the last balance entry
when displaying balance statement
- on a rerun the last balance is now getting added
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-+FS
PR: https://github.com/apache/cloudstack/pull/768
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This has been happening for the NSX and BigSwitch plugins.
The maven build prints something like:
[WARNING] Unknown file extension: .../plugins/network-elements/nicira-nvp/.pmd
[WARNING] Missing header in: .../plugins/network-elements/nicira-nvp/.pmdruleset.xml
CLOUDSTACK-9062: Improve S3 implementation.The S3 implementation is far from finished, this commit focuses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimizations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
Please review thoroughly, both code inspection and (automated) integration tests. Currently no integration tests are available specifically for S3. Therefore the implementation is needed to be tested manually, for now...
What I tested:
- Greenfield install -> will download latest systemvm template automatically to S3.
- Upload a template/iso
- Download a template/iso
- Restart of management server -> list available templates -> doesn't download them again if available.
* pr/1083:
CLOUDSTACK-9062: Improve S3 implementation.
Signed-off-by: Remi Bergsma <github@remi.nl>
* 4.6:
CLOUDSTACK-9053 security upgrade as per COLLECTIONS-580
CLOUDSTACK-9055: fix NPE in updating Redundant State of VPC networks
CLOUDSTACK-9057 remove old system vm upgrade code
The S3 implementation is far from finished, this commit focusses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimalisations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
- Add -noverify JVM arg to surefire plugin, to allow Powermockito to
de-encapsulate private methods
- Add dependency on cloud-utils test-jar to use custom HttpRequest
matchers
- Upgrade version of HTTP Components to 4.5
- Add helper class to create Http clients
- Add helper class to build http requests
- Add enum with the different Http Methods
- Add constants class for HTTP related values
- XML formatting
- Fix license header
- Reorder hamcrest, junit, mockito and powermock dependencies
* Since several libraries include a subset of hamcrest classes,
hamcrest should be declared first in the pom, inorder for its
classes to be loaded first by the JVM
- Depdend on hamcrest-all and exclude hamcrest classes from other
libraries
The newer SDKs API changed which causes our S3 Template Downloader to never complete.
Although we should fix the Template Downloader we can revert to the old AWS SDK for now.
The fix on the longer run will be rewriting the S3 Template Downloader.
Two methods had to be disabled for now since the old SDK does not support them. They can
be re-enabled when the Template Downloader has been fixed.
- Removes awsapi packaging rules for debian, centos63, centos7, fedora 20/21
- Removes catalina port 7080 service configs
- Fixes build replace properties for AWSAPILOG
- Removes maven profile for building awsapi and deploying db in developer profile
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
removing nio server as it is currently handling only https connections
and the parsing logic is also specific to agent communication.
current limitation of httpcomponents server is that the entire file is
read in memory. need figure out how to read it in chunks and send it
through a inputstreamreader to save on secondary storage.
Changes;
- Upgrades maven-war plugin to 4.5 (faster war packaging)
- Upgrade spring framework to latest minor release
- Upgrade ehcache, jasypt, httpclient, httpcore and other core dependencies
- Upgrade to latest ipv6 library, fix unit test NetUtilsTest
- httpcore and httpclient are sharing same version variable
- commons-httpclient is different that httpclient, the fix gives it a separate var
- Apidocs failed to generate and get stuck with new reflections version, for now
we will continue using 0.9.8
Newer dependencies can be listed using:
mvn versions:display-dependency-updates -Dnoredist -Dsimulator -P developer,systemvm
Testing;
- Tested using Maven 3.2.1
- Local noredist build with unit tests succeeds
- CloudStack mgmt server started, basic business layer tests work
- Observed 10-15% build time improvement using new maven-war plugin
Branch: bugfix/4.5-8011 (commits are squashed in favour of a linear history)
Pull request:
https://github.com/apache/cloudstack/pull/50
This closes#50
TravisCI build summary:
https://travis-ci.org/shapeblue/cloudstack/builds/42902172
- Build passes with unit tests
- Apidocs generates successfully
- Most integration tests pass, some fail due to timeout errors, second re-run
passes some of them
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Refactor tests:
- Upgrade tests to use jUnit4
- Add hamcrest dependency (contribution of by Laszio Hornyak)
- Break big tests in small unit tests
- Replace assertTrue/False with complex conditions by assertThat with
specific matchers
- Remove dead code:
- Private static method never called locally
- Add test for method that validates cidrs
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Findings:
- 32 int shifted by an amount not in range -31..31:
The shifts by 32 bits don't actually have any effect on the value
(as shown by the tests)
- possible null pointer dereference
- repeated conditional test
- field only ever set to null
All other uses of the field were to check if it was null,
which it was, so it was removed
Other actions:
- Upgrade jUnit to version 4
- Add PowerMock dependency
Signed-off-by: Hugo Trippaers <htrippaers@schubergphilis.com>
We now require at least Java 7 to build and run CloudStack.
Both the DEB and RPM packaging now also require Java 7 during installation
of the packages.
Activate this profile in your IDE if you're having problems with checkstyle when switching branches. Please ensure that you at least compile once with checkstyle enabled before committing.
Git ignores log files, so basically they shouldn't checked in. This
configures rat to ignore log files also. Specifically the juniper-contrail
code creates a log file during testing.
Fixing rebase issues after integrating with wmi v2 implementation.
Removing the executable attribute from some files.
Remove the unused wmi v1 interface file.
Unit test for DestroyCommand implementation in hyperv agent.
Fixed VM state changes w.r.t wmi version 2 changes
If a VM is already running, deploy virtual machine shouldn't fail and throw an exception.
Don't run vhd-util on templates which are present on CIFS. Hyperv uses cifs as secondary storage
Add a SCSI controller by default. This is needed so that data volumes can be added/removed
on a running vm.
Remove the hard coded path in the agent code.
Rat fixes for hyper agent. Added the missing headers in files where it was missing.
All of the code changes for the Spring Modularization will work in
a modularized context or a non-modularized context. This commit
is the final commit to turn modularization on. Revert this commit
to disable the modularization and go back to monolithic Spring
configuration.
The managed context framework provides a simple way to add logic
to ACS at the various entry points of the system. As threads are
launched and ran listeners can be registered for onEntry or onLeave
of the managed context. This framework will be used specifically
to handle DB transaction checking and setting up the CallContext.
This framework is need to transition away from ACS custom AOP to
Spring AOP.
All (almost) files belonging to the systemvm aer now centralize in the systemvm directory. The code for the separate functions is still in the services directory. This will make the code easier to understand and makes it clear that the systemvm is a separate item. It alos means that it can be excluded from the build entirely by not adding the systemvm profile, this will speed up the compiles somewhat.
The migrate method from libvirt supports passing down a different XML for running
the instance of the target hypervisor.
This enables the VNC to bind to the private IP Address of the hypervisor and during
migration this will be changed to the private IP address of the target host.
This way VNC doesn't listen world wide and is much safer.
Although libvirt supports resizing RBD volumes (and other formats) the
Java bindings (libvirt-java) don't.
Right now we use the Java bindings for librbd to handle the resizing for us,
but in the future this should be done by libvirt rather then these
Java bindings.
Rajani Karuturi
Will Stevens
Rohit Yadav
Ronald van Zantvoort
Abhinandan Prateek
Daan Hoogland
John Burwell
Remi Bergsma
Miguel Ferreira
Boris Schrijver
nvazquez
Wido den Hollander
Pierre-Luc Dion
Daan Hoogland
Rafael da Fonseca
Rajani Karuturi
Ian Duffy
Sebastien Goasguen
Hugo Trippaers
Mike Tutkowski
Koushik Das
Damodar Reddy
Anthony Xu
Anshul Gangwar
Alex Huang
edison
miguelaferreira
Antonio Fornie
Darren Shepherd
Devdeep Singh
Wido den Hollander
Chip Childers
Prasanna Santhanam
Chiradeep Vittal