This adds support for JDK11 in CloudStack 4.14+:
- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The implementation of GenericBaseDao#searchAndCount() can result in
incorrect count. This happens because the implementation of the
GenericBaseDao#getCount method excludes the groupBy components of the
search queries. This means the count returned will always be larger than
the actual count when not considering pagination.
The change was brought in b0ce8fd which also fails to explain the rationale.
Further investigation of the getCount usage reveal they are always
accompanied by search queries which include groupBy components via
GenericBaseDao#searchIncludingRemoved method.
```
Current code diff between search and count methods is as follows -
diff --git a/framework/db/src/main/java/com/cloud/uddtils/db/GenericDaoBase.java b/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
@@ -371,7 +371,7 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone
clause = null;
}
- final StringBuilder str = createPartialSelectSql(sc, clause != null, enableQueryCache);
+ final StringBuilder str = createCountSelect(sc, clause != null);
@@ -384,19 +384,12 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone
}
}
- List<Object> groupByValues = addGroupBy(str, sc);
- addFilter(str, filter);
-
+ // we have to disable group by in getting count, since count for groupBy clause will be different.
+ //List<Object> groupByValues = addGroupBy(str, sc);
final TransactionLegacy txn = TransactionLegacy.currentTxn();
- if (lock != null) {
- assert (txn.dbTxnStarted() == true) : "As nice as I can here now....how do you lock when there's no DB transaction? Review your db 101 course from college.";
- str.append(lock ? FOR_UPDATE_CLAUSE : SHARE_MODE_CLAUSE);
- }
-
@@ -410,20 +403,19 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone
+ /*
if (groupByValues != null) {
for (Object value : groupByValues) {
pstmt.setObject(i++, value);
}
}
+ */
- if (s_logger.isDebugEnabled() && lock != null) {
- txn.registerLock(pstmt.toString());
- }
final ResultSet rs = pstmt.executeQuery();
while (rs.next()) {
- result.add(toEntityBean(rs, cache));
+ return rs.getInt(1);
}
- return result;
+ return 0;
--
2.17.1
```
The fix is to update the way we setup query for counting with group by
params.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Cleaup and code-formatting POM files
* Remove obsolete mycila license-maven-plugin
* Remove obsolete console-proxy/plugin project
* Move console-proxy-rdbconsole under console-proxy parent
* Use correct parent path for rdpconsole
* Order alphabetally items in setnextversion.sh
* Unifiy License header in POMs
* Alphabetic order of modules definition
* Extract all defined versions into parent pom
* Remove obsolete files: version-info.in, configure-info.in
* Remove redundant defaultGoal
* Remove useless checkstyle plugin from checkstyle project
* Order alphabetally items in pom.xml
* Add aditional SPACEs to fix debian build
* Don't execute checkstyle on parent projects
* Use UTF-8 encoding in building checkstyle project
* Extract plugin versions into properties
* Execute PMD plugin on all the projects with -Penablefindbugs
* Upgrade maven plugins to latest version
* Make sure to always look for apache parent pom from repository
* Fix incorrect version grep in debian packaging
* Fix rebase conflicts
* Fix rebase conflicts
* Remove PMD for now to be fixed on another PR
Remove maven standard module (which only a few were using) and get ride of maven customization for the projects structure.
- moved all directories to src/main/java, src/main/resources, src/main/scripts, src/test/java, src/test/resources
- grep scan to search for src/com and src/org left over
- grep for <project>/scripts to fix pom.xml configuration
- remove custom <build> configuration in pom.xml
Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
The db queries in listTemplateAPI could be optimized to get unique results from the database which could help in reducing the listTemplate API response time.
When using CloudStack with database HA configuration user receives warnings constantly saying “No database selected”.
This problem happens at com.cloud.cluster.ClusterManagerImpl.getHeartbeatTask().new ManagedContextRunnable(){...}.runInContext(), line 550. The scheme of the database is not properly set in the transaction when database HA options are enabled.
The GroupBy + having isn't used currently in the code but was not clean.
It removes unused arguments and variables and adds a test based on a DAO
to show a full example on how to use it.
Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
- Bump spring-framework version to 4.x and Jetty to version that runs with JDK8
- Bump servet dependency version
- Migrate spring xmls to version 4, fixes schema locations that are 3.0
dependent in various xmls.
- Fix failing tests due to spring upgrade
(Thanks @marcaurele Marc-Aurèle Brothier for fixing them)
* Fix test DeploymentPlanningManagerImplTest
* Fix GloboDNS test
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Often, patch and security releases do not require schema migrations or
data migrations. However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break. With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path. This approach allows the upgrade to proceed and
for the database to properly reflect the installed version. This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.
Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes. The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers. This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers. It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.
* Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
a version that was not explicitly specified. It determines the
releases the first release before it with database migrations and uses
that list as the basis for the list for version being calculated. A
noop upgrade is then added to the list which causes no schema changes
or data migrations, but will update the database to the version.
* Adds unit tests for the upgrade path calculation logic in
DatabaseUpgradeChecker
* Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
versions of this patch
* Introduces the CloudStackVersion value object which parses and
compares three (3) and four (4) position version numbers. This class
is intended to replace com.cloud.maint.Version.
* Adds the junit-dataprovider dependency -- allowing test data to be
concisely generated separately from the execution of a test case.
Used extensively in the CloudStackVersionTest.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
DAO: Hit the cache for entity flagged as removed tooI came along this part of the code and I don't see any reason why the cache should not be used when fetching with the "removed" ones. It will help decrease the number of DB queries.
*It can be merged in many CS versions*
* pr/1532:
DAO: Rewrite change for method findByIdIncludingRemoved(ID id)
dao: Hit the cache for entity flagged as removed too since they are put in cache afterwards.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Quota service while allowing for scalability will make sure that the cloud is
not exploited by attacks, careless use and program errors. To address this
problem, we propose to employ a quota-enforcement service that allows resource
usage within certain bounds as defined by policies and available quotas for
various entities. Quota service extends the functionality of usage server to
provide a measurement for the resources used by the accounts and domains using a
common unit referred to as cloud currency in this document. It can be configured
to ensure that your usage won’t exceed the budget allocated to accounts/domain
in cloud currency. It will let user know how much of the cloud resources he is
using. It will help the cloud admins, if they want, to ensure that a user does
not go beyond his allocated quota. Per usage cycle if a account is found to be
exceeding its quota then it is locked. Locking an account means that it will not
be able to initiat e a new resource allocation request, whether it is more
storage or an additional ip. Needless to say quota service as well as any action
on the account is configurable.
Changes from Github code review:
- Added marvin test for quota plugin API
- removed unused commented code
- debug messages in debug enabled check
- checks for nulls, fixed access to member variables and feature
- changes based on PR comments
- unit tests for UsageTypes
- unit tests for all Cmd classes
- unit tests for all service and manager impls
- try-catch-finally or try-with-resource in dao impls for failsafe db switching
- remove dead code
- add missing quota calculation case (regression fixed)
- replace tabs with spaces in pom.xmls
- quota: though default value for quota_calculated is 0, the usage server
makes it null while entering usage entries. Flipping the condition so
as to acocunt for that.
- quotatypes: fix NPE in quota type
- quota framework test fixes
- made statement period configurable
- changed default email templates to reflect the fact that exhausted quota may not result in a locked account
- added quotaUpdateCmd that refreshes quota balances and sends alerts and statements
- report quotaSummary command returns quota balance, quota usage and state for all account
- made UI framework changes to allow for text area input in edit views
- process usage entries that have greater than 0 usage
- orocess quota entries only if tariff is non zero
- if there are credit entries but no balance entry create a dummy balance entry
- remove any credit entries that are before the last balance entry
when displaying balance statement
- on a rerun the last balance is now getting added
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-+FS
PR: https://github.com/apache/cloudstack/pull/768
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This reverts commit cd7218e241, reversing
changes made to f5a7395cc2.
Reason for Revert:
noredist build failed with the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloud-plugin-hypervisor-vmware: Compilation failure
[ERROR] /home/jenkins/acs/workspace/build-master-noredist/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[484,12] error: non-static variable logger cannot be referenced from a static context
[ERROR] -> [Help 1]
even the normal build is broken as reported by @koushik-das on dev list
http://markmail.org/message/nngimssuzkj5gpbz
Cloudstack 8656: do away with more silently ignoring exceptions.a lot of messages added.
some restructuring for test exception assertions and try-with-resource blocks
* pr/654: (29 commits)
CLOUDSTACK-8656: more logging instead of sysout
CLOUDSTACK-8656: use catch block for validation
CLOUDSTACK-8656: class in json specified not found
CLOUDSTACK-8656: removed unused classes
CLOUDSTACK-8656: restructure of tests
CLOUDSTACK-8656: reorganise sychronized block
CLOUDSTACK-8656: restructure tests to ensure exception throwing
CLOUDSTACK-8656: validate the throwing of ServerApiException
CLOUDSTACK-8656: logging ignored exceptions
CLOUDSTACK-8656: try-w-r removes need for empty catch block
CLOUDSTACK-8656: try-w-r instead of clunckey close-except
CLOUDSTACK-8656: deal with empty SQLException catch block by try-w-r
CLOUDSTACK-8656: unnecessary close construct removed
CLOUDSTACK-8656: message about timed buffer logging
CLOUDSTACK-8656: message about invalid number from store
CLOUDSTACK-8656: move cli test tool to separate file
CLOUDSTACK-8656: exception is the rule for some tests
CLOUDSTACK-8656: network related exception logging
CLOUDSTACK-8656: reporting ignored exceptions in server
CLOUDSTACK-8656: log in case we are on a platform not supporting UTF8
...
Signed-off-by: Remi Bergsma <github@remi.nl>
- New implementation uses nanoseconds. Due to that, the places where the Profiler is used as a Monitor and/or
a stopwatch will suffer with the difference in the return
- Also added a getDuration(), which returns the time in nanoseconds in case someone wants to use it instead
- Added an extra test to check if the getDuration() works fine with nanoseconds
- Fixed the test that checks the time in milliseconds: I added an error margin to cover the test better
Signed-off-by: Daan Hoogland <daan@onecht.net>
- Removes awsapi db properties usage across codebase
- Removes references from spring xmls, test cases and TransactionLegacy
- Adds sql command to drop database cloudbridge in schema-451to460-cleanup.sql
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Removing resource leaks from UsageSanityChecker and
refactoring it (encapsulation, removal of copy and paste, constants...)
Modularize static method for closing Statments in TransactionLegacy
and reusing this new method from other classes (Upgrade2214to30)
Create Unit and Integration Tests for UsageSanityChecker
Add DBUnit cases and integration profile for nitegration tests as
a base for future DB tests
Introduction of a new Transaction API that is more consistent with the style
of Spring's transaction managment. The existing Transaction class was renamed
to TransactionLegacy. All of the non-DAO code in the management server has been
updated to use the new Transaction API.
ACS is now comprised of a hierarchy of spring application contexts.
Each plugin can contribute configuration files to add to an existing
module or create it's own module.
Additionally, for the mgmt server, ACS custom AOP is no longer used
and instead we use Spring AOP to manage interceptors.
The managed context framework provides a simple way to add logic
to ACS at the various entry points of the system. As threads are
launched and ran listeners can be registered for onEntry or onLeave
of the managed context. This framework will be used specifically
to handle DB transaction checking and setting up the CallContext.
This framework is need to transition away from ACS custom AOP to
Spring AOP.
andrijapanicsb
Rohit Yadav
Daan Hoogland
Pearl Dsilva
Paul Angus
Abhishek Kumar
Anurag Awasthi
GabrielBrascher
Khosrow Moossavi
Nicolas Vazquez
Marc-Aurèle Brothier
PranaliM
Rafael Weingärtner
Wei Zhou
Rajani Karuturi
John Burwell
Abhinandan Prateek
Will Stevens
Nick Livens
nvazquez
Remi Bergsma
cirstofolini
ramamurtis
Sudhansu
Rajani Karuturi
Daan Hoogland
Daan Hoogland
wilderrodrigues
Laszlo Hornyak
Min Chen
Hugo Trippaers
Santhosh Edukulla
Anthony Xu
Alena Prokharchyk
Koushik Das
Antonio Fornie
Ian Southam
Prachi Damle
Alex Huang
Damodar Reddy
Darren Shepherd
Kishan Kavala
Darren Shepherd