* Commit for PR https://github.com/shapeblue/cloudstack-apple/pull/471
This PR adds a flag (customType) to HypervisorType to differentiate if a hypervisor plugin is built-in or customized.
With the customType flag, we can isolate conditional checks for allow the customized hypervisor plugin to proceed.
* Add customType field to HypervisorType to differentiate if the plugin is a customized type
* Address @shwstppr's comments
* Define a enum in HypervisorType to indicate specific functionalities of a hypervisor plugin type
* Refactoring to define if the hypervisor supports the operation during
initialization of the HypervisorType.
* Remove ROOT_DISK_SIZE_OVERRIDE_SUPPORTING_HYPERVISORS
---------
Co-authored-by: Annie Li <ji_li@apple.com>
During agent join and while changing configs - host and indirect.agent.lb.algorithm, optimize calling DB for zone's host list
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Added caching for ConfigKey value retrievals based on the Caffeine
in-memory caching library.
https://github.com/ben-manes/caffeine
Currently, expire time for a cache is 1 minute and each update of the
config key invalidates the cache.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
* Don't send sql exception/query from dao to upper layer, log it and send only the error message
* Updated charset to utf8mb4, for display_name column/user_vm table and job_result column/async_job table to support unicode chars & emojis
* Added API arg validator for RFC compliance domain name, to validate VM's host name
* Updated user resources name / display name column's charset to utf8mb4
* Check and update char set for affinity group name to utf8mb4, from the data migration in upgrade path
* Updated backup offering name column charset to utf8mb4
* Added unit tests for vm host/domain name validation
* Added smoke test to check resource name for vm, volume, service & disk offering, template, iso, account(first/lastname)
* Updated resource annotation charset to utf8mb4
* Updated some resources description charset to utf8mb4
- mTLS implementation for cluster service communication
- Listen only on the specified cluster node IP address instead of all interfaces
- Validate incoming cluster service requests are from peer management servers based on the server's certificate dns name which can be through global config - ca.framework.cert.management.custom.san
- Hardening of KVM command wrapper script execution
- Improve API server integration port check
- cloudstack-management.default: don't have JMX configuration if not needed. JMX is used for instrumentation; users who need to use it should enable it explicitly
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
(cherry picked from commit 4f5561937c)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Mitigation for non-scalable Powerflex/ScaleIO clients
- Added ScaleIOSDCManager to manage SDC connections, checks clients limit, prepare and unprepare SDC on the hosts.
- Added commands for prepare and unprepare storage clients to prepare/start and stop SDC service respectively on the hosts.
- Introduced config 'storage.pool.connected.clients.limit' at storage level for client limits, currently support for Powerflex only.
* tests issue fixed
* refactor / improvements
* lock with powerflex systemid while checking connections limit
* updated powerflex systemid lock to hold till sdc preparation
* Added custom stats support for storage pool, through listStoragePools API
* code improvements, and unit tests
* Update config 'storage.pool.connected.clients.limit' to dynamic, and some improvements
* Stop SDC on host after migration if no volumes mapped to host
* Wait for SDC to connect after scini service start, and some log improvements
* Do not throw exception (log it) when SDC is not connected while revoking access for the powerflex volume
* some log improvements
* Enforce strict host tag checking
* Add e2e tests
* Add more information to error log
* Fix e2e test
* Update global settings descrption
* fixup
* Fix e2e test teardown
This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine).
There would be three mechanisms for purging removed resources:
- Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
- API - New API `purgeExpungedResources`. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate
- Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.
Following new global settings have been added:
- `expunged.resources.purge.enabled`: Default: false. Whether to run a background task to purge the DB records of the expunged resources.
- `expunged.resources.purge.resources`: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the DB records of the expunged resources. Currently only VirtualMachine is supported. An empty value will result in considering all resource types for purging.
- `expunged.resources.purge.interval`: Default: 86400. Interval (in seconds) for the background task to purge the DB records of the expunged resources.
- `expunged.resources.purge.delay`: Default: 300. Initial delay (in seconds) to start the background task to purge the DB records of the expunged resources task.
- `expunged.resources.purge.batch.size`: Default: 50. Batch size to be used during purging of the DB records of the expunged resources.
- `expunged.resources.purge.start.time`: Default: (empty). Start time to be used by the background task to purge the DB records of the expunged resources. Use format `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`.
- `expunged.resources.purge.keep.past.days`: Default: 30. The number of days in the past from the execution time of the background task to purge the DB records of the expunged resources for which the expunged resources must not be purged. To enable purging DB records of the expunged resource till the execution of the background task, set the value to zero.
- `expunged.resource.purge.job.delay`: Default: 180. Delay (in seconds) to execute the purging of the DB records of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.
Upstream PRs:
https://github.com/apache/cloudstack/pull/8999https://github.com/apache/cloudstack-documentation/pull/397
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
* Validate user data with actual length, and some code improvements
* Ignore if user data is not set (don't fail)
* Validate user data after finalizing it
* Updated registerUserData API using POST call from UI, to support user data upto 1048576 bytes
* Apply suggestions from code review
* Added logs for user data
* Addressed review comments
* Check user data length with base64 encoded data, and some code improvements
* Merge two HostTagVO and HostTagDaoImpl
* Apple FR76: dynamic host tags
* Revert "Apple FR76: dynamic host tags"
This reverts commit 01b93a873f167018c4fafd0744c0de07ae4de4ed.
* Apple FR76: Implicit host tags
* Apple FR76: address Abhishek's comments
* Apple FR76: move updateImplicitTags
* Apple FR76: add since to other two responses
* Update 8929: add unit test in LibvirtComputingResourceTest
* Update variable names
* Update FR76: add explicithosttags in response
* Update FR76 UI: Update explicit host tags
* Update 8929: remove host tags and change labels on UI
* Update: ui polish for host tags
* fix since in responses
* Update 8929: fix UI error if no host tags
Found these CPU and DB hotspot that handle agent ping commands, this
adds idle load when there are high number of hosts. By design, there
isn't any quick win here. However, the power sync report/handling could
be improved, so it doesn't need to kick-in for every ping command
received.
Few more areas marked in the codebase.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This refactors a ResourceManager::listAvailHypervisorInZone method
that should return unique hypervisors for which existing hosts are Up
and processed. We can approximate this by assuming that those hosts
would have setup their hypervisor-specific systemvmtemplates. In a given
environment there wouldn't be thousands of systemvmtemplates, but can
have thousands of hosts. So, instead of scanning the entire cloud.host
table, we can make calculate guess by returning unique hypervisors of
systemvm templates which are ready. This method was used in
::processConnect() when an agent joins, to speed up its handling.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The type parameter isn't keyword, but a simple listHosts API call with
type=Routing, runs SELECT COUNT(*) FROM host WHERE host.type LIKE
'%Routing' AND host.removed IS NULL; ... which causes an unnecessary
full table scan.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
In this example commit, we look at:
- Adding missing indexes to speed up queries
- Reduce table scans by optimising sql query and using indexes
- Optimising sql queries to remove duplicate rows (use of distinct)
- Reduce CPU and DB load by using jprofiler to optimise both sql query
and CPU hotspots
server: reduce CPU and DB load caused by systemvm ::isZoneReady()
For this case, the sql query was fetching large number of table scans
only to determine if zone has any available pool+host to launch
systemvms. Accodingly the code and sql queries along with indexes
optimisations were used to lower both DB scans and mgmt server CPU load.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Abhishek Kumar
Cheng Cheng
Suresh Kumar Anaparti
mprokopchuk
mprokopchuk
Rohit Yadav
Vishesh
Suresh Kumar Anaparti
Wei Zhou