The type parameter isn't keyword, but a simple listHosts API call with
type=Routing, runs SELECT COUNT(*) FROM host WHERE host.type LIKE
'%Routing' AND host.removed IS NULL; ... which causes an unnecessary
full table scan.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
In this example commit, we look at:
- Adding missing indexes to speed up queries
- Reduce table scans by optimising sql query and using indexes
- Optimising sql queries to remove duplicate rows (use of distinct)
- Reduce CPU and DB load by using jprofiler to optimise both sql query
and CPU hotspots
server: reduce CPU and DB load caused by systemvm ::isZoneReady()
For this case, the sql query was fetching large number of table scans
only to determine if zone has any available pool+host to launch
systemvms. Accodingly the code and sql queries along with indexes
optimisations were used to lower both DB scans and mgmt server CPU load.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This refactors hotspot code to fetch just the count of hosts than
all the host VOs for a zone, during capacity scans for systemvms.
This reduces CPU and DB load, in really large (10k+ hosts) env.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Add ability to set cpu.threadspercore similar to existing cpu.corespersocket
* Add license to new test file
* Add tests to handle some edge cases
* Add some edge test cases to CPU topology
* Rework logic on KVM CPU topology, handle more cases
* Add more test cases
* Add more test cases
* Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
* Added cpu.threadspercore detail in listDetailOptions response (for KVM hypervisor)
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Add a global setting to control whether redirection is allowed while
downloading templates and volumes
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
This introduces new global settings to handle how client address checks
are handled by the API layer:
proxy.header.verify: enables/disables checking of ipaddresses from a
proxy set header
proxy.header.names: a list of names to check for allowed ipaddresses
from a proxy set header.
proxy.cidr: a list of cidrs for which \"proxy.header.names\" are
honoured if the \"Remote_Addr\" is in this list.
(cherry picked from commit b65546636d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Move allow.additional.vm.configuration.list.kvm from Global to Account setting
- Disallow VM details start with "extraconfig" when deploy VMs
- Skip changes on VM details start with "extraconfig" when update VM settings
- Allow only extraconfig for DPDK in service offering details
- Check if extraconfig values in vm details are supported when start VMs
- Check if extraconfig values in service offering details are supported when start VMs
- Disallow add/edit/update VM setting for extraconfig on UI
(cherry picked from commit e6e4fe16fb)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This PR changes the password.policy.regex default value to empty. With an empty value for the configuration, it is skipped during the password policy check, only when the configuration is set to something different than a blank string, the regex will get checked.
This way, when creating a user on org.apache.cloudstack.ldap.LdapAuthenticator#authenticate() we won't get an error by default, as an empty value for the password is passed.
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
* Use join instead of views for filtering volumes
* Use join instead of views for filtering events
* Use join instead of views for filtering accounts
* Use join instead of views for filtering domains
* Use join instead of views for filtering hosts
* Use join instead of views for filtering storage pools
* Use join instead of views for filtering service offerings
* Use join instead of views for filtering disk offerings
* Remove unused code
* Fix unit test
* Use disk_offering instead of disk_offering_view in service_offering_view
* Fixup
* Fix listing of diskoffering & serviceoffering
* Use constants instead of strings
* Make changes to prevent sql injection
* Remove commented code
* Prevent n+1 queries for template's response
* remove unused import
* refactor some code
* Add missing check for service offering's join with disk offering
* Fix n+1 queries for stoage pool metrics
* Remove n+1 queries from list accounts
* Remove unused imports
* remove todo
* Remove unused import
* Fixup query generation for nested joins
* Fixups
* Fix DB exception on ClientPreparedStatement
* events,alerts: Add missing indexes (#366)
* Fixup
* Fix resource count discrepancies
* Fixup while removing vm
* Fix discrepancies when starting VMs
* Fixup tests
* Fixups
* Don't take lock when amount is negative
Adds:
- Fix for volume limit checks for disk offerings with multiple tags - When a VM is deployed with multiple disks having offerings with multiple tags the resource limit check may falter as currently it tries to check based on individual diskoffering. With this, change if offering d1 and d2 for volumes v1 and v2 both have tag1, server will check volume limits for tag tag1 using the combined size of v1 and v2.
- Fix for template tag hosts in random host allocator - May affect use of template tag, service offering tags and random host allocator together. The current code for the random host allocator falters while trying to find the host allocation. This was found and fixed during the addition of the unit test here, https://github.com/shapeblue/cloudstack-apple/pull/378/files#diff-bbf9baea014e5cc1dfe9e7d13467c9857208cfe65e93883721d88a6f0452f912
- Unit tests for changes in api,server,ui: tagged resource limits #327
* Update PowerFlex storage stats on host connect (if any changes in capacity / used bytes)
* Sync the pool stats in DB with the actual stats from stats collector
* Updated capacityBytes check
* Revert "Updated capacityBytes check"
This reverts commit 3ffb17b2c4b3c794e5d0dbf4108d43255b4fbcca.
* Revert "Update PowerFlex storage stats on host connect (if any changes in capacity / used bytes)"
This reverts commit 9e473aed4c589b91f62cbe2fd135dc25e0adc1c3.
* Introduced a new API "checkVolumeAndRepair" that allows users or admins to check and repair if any leaks observed.
Currently this is supported only for KVM
* some fixes
* Added unit tests
* addressed review comments
* add repair volume while granting access
* Changed repair parameter to accept both leaks/all values
* Introduced new global setting volume.check.and.repair.before.use to do volume check and repair before VM start or volume attach operations
* Added volume check and repair changes only during VM start and volume attach operations
* Refactored the names to look similar across the code
* Some code fixes
* remove unused code
* Renamed repair values
* Addressed review comments
* code refactored
* used volume name in logs
* Changed the API to Async and the setting scope to storage pool
* Fixed exit value handling with check volume command
* Fixed storage scope to the setting
* Fixed volume format issues
* Refactored the log messages
* Fix formatting
* Added vm uuid as part of error response when vm create fails after vm entity is persisted
* Fixed styling issue
* Fixed styling issue.
* Fix unit tests
* Fixed merge conflicts.
* Fixed merge conflicts.
---------
Co-authored-by: Annie Li <ji_li@apple.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
Introduces the concept of tagged resource limits. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types,
Host limits
user_vm
cpu
memory
Storage limits
volume
primary_storage
Following global settings can used to specify tags for which limit needs to be enforced,
Host: resource.limit.host.tags
Storage: resource.limit.storage.tags
Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI.
Enhances use of templatetag for VM deployment and template creation
Adds option to list disk offering with suitability flag for a virtualmachine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the reponse.
* Allocate new volume on restore virtual machine operation when resource count increment succeeds
- keep them in transaction, and fail operation if resource count increment fails
* Added some (negative) unit tests for restore vm
* StoragePoolType as a class
* Fix agent side StoragePoolType enum to class
* Handle StoragePoolType for StoragePoolJoinVO
* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.
* Fix UserVMJoinVO for StoragePoolType
* fixed missing imports
* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.
* Fixed equals for the enum.
* removed not needed try/catch for prepareAttribute
* Added license to the file.
* Implemented "supportsPhysicalDiskCopy" for storage adaptor. (#352)
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
* Add javadoc to StoragePoolType class
* Add unit test for StoragePoolType comparisons
* StoragePoolType "==" and ".equals()" fix.
* Fix for abstract storage adaptor set up issue
* review comments
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
* Cleanup Volume AsyncJob after mgmt server stop
* Clean Up Vm async job resources during mggmt server stop
* Use State.isTransitional method to identify trnsition states
* Add cleanup for Network Async Job
* Add license
* Added RevertSnapshotting to volume transition state. Fixed spacing code style
* Added transitional flag in Volume state
* Updated network event for failed job, (re)added cleanup for volumes created from snapshots, and some code improvements
* Added java doc for volume state constructor
* Fixed cleanup SNAPSHOT_ID entry in volume details for failed volumes created from snapshots
---------
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
This adds the missing commit to the fix#335 from the upstream PR:
apache/cloudstack#7977
(cherry picked from commit b5f77f9c3b53af7e8b05730da9807a2c9eb017a5)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Run recalculation recurrent task only in the longest running management server
* Fix timeout and recalculation when no child domains
(cherry picked from commit 3f35e3a6ef24017d2f63613ca4362521b1ee78b6)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Rohit Yadav
Vishesh
Marcus Sorensen
Abhishek Kumar
dahn
Wei Zhou
Suresh Kumar Anaparti
Harikrishna
anniejili
kishankavala
slavkap
Nicolas Vazquez
Fabricio Duarte