Optimises DB query that seem to run against every Ping command, where
whole columns are fetched but only `id` column is used.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This refactors hotspot code to fetch just the count of hosts than
all the host VOs for a zone, during capacity scans for systemvms.
This reduces CPU and DB load, in really large (10k+ hosts) env.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This adds a NPE check on the s_depot.global() which can cause NPE in
case of unit tests, where s_depot is not null but the underlying config
dao is null (not mocked or initialised) via `s_depot.global()` becomes
null.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Add ability to set cpu.threadspercore similar to existing cpu.corespersocket
* Add license to new test file
* Add tests to handle some edge cases
* Add some edge test cases to CPU topology
* Rework logic on KVM CPU topology, handle more cases
* Add more test cases
* Add more test cases
* Update plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
* Added cpu.threadspercore detail in listDetailOptions response (for KVM hypervisor)
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
The uprgade from 4.18.1.0 to 4.18.2.0-SNAPSHOT failed with error
```
2023-09-12 16:12:19,003 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.18.1.0 Code Version = 4.18.2.0
2023-09-12 16:12:19,004 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.18.1.0 to 4.18.2.0
2023-09-12 16:12:19,036 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Running upgrade Upgrade41800to41810 to upgrade from 4.18.0.0-4.18.1.0 to 4.18.1.0
...
2023-09-12 16:12:19,041 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- Schema upgrade from 4.18.0.0 to 4.18.1.0
...
2023-09-12 16:12:21,602 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Statement: CREATE INDEX i_cluster_details__name on cluster_details (name)
2023-09-12 16:12:21,663 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Created index i_cluster_details__name
2023-09-12 16:12:21,673 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 2632 Name = Upgrade; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:97-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:215-$Proxy30.persist:-1-DatabaseUpgradeChecker.upgrade:319-DatabaseUpgradeChecker.check:403-CloudStackExtendedLifeCycle.checkIntegrity:64
```
It succeeded with this change.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit a88a47989369af204ea6ee8a5fd190311f43c74c)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Add a global setting to control whether redirection is allowed while
downloading templates and volumes
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
This introduces new global settings to handle how client address checks
are handled by the API layer:
proxy.header.verify: enables/disables checking of ipaddresses from a
proxy set header
proxy.header.names: a list of names to check for allowed ipaddresses
from a proxy set header.
proxy.cidr: a list of cidrs for which \"proxy.header.names\" are
honoured if the \"Remote_Addr\" is in this list.
(cherry picked from commit b65546636d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Move allow.additional.vm.configuration.list.kvm from Global to Account setting
- Disallow VM details start with "extraconfig" when deploy VMs
- Skip changes on VM details start with "extraconfig" when update VM settings
- Allow only extraconfig for DPDK in service offering details
- Check if extraconfig values in vm details are supported when start VMs
- Check if extraconfig values in service offering details are supported when start VMs
- Disallow add/edit/update VM setting for extraconfig on UI
(cherry picked from commit e6e4fe16fb)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This PR changes the password.policy.regex default value to empty. With an empty value for the configuration, it is skipped during the password policy check, only when the configuration is set to something different than a blank string, the regex will get checked.
This way, when creating a user on org.apache.cloudstack.ldap.LdapAuthenticator#authenticate() we won't get an error by default, as an empty value for the password is passed.
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
* Support KVM storage implementations controlling logical/physical block io size
* Support custom block size during disk attach
---------
Co-authored-by: Marcus Sorensen <mls@apple.com>
* Use join instead of views for filtering volumes
* Use join instead of views for filtering events
* Use join instead of views for filtering accounts
* Use join instead of views for filtering domains
* Use join instead of views for filtering hosts
* Use join instead of views for filtering storage pools
* Use join instead of views for filtering service offerings
* Use join instead of views for filtering disk offerings
* Remove unused code
* Fix unit test
* Use disk_offering instead of disk_offering_view in service_offering_view
* Fixup
* Fix listing of diskoffering & serviceoffering
* Use constants instead of strings
* Make changes to prevent sql injection
* Remove commented code
* Prevent n+1 queries for template's response
* remove unused import
* refactor some code
* Add missing check for service offering's join with disk offering
* Fix n+1 queries for stoage pool metrics
* Remove n+1 queries from list accounts
* Remove unused imports
* remove todo
* Remove unused import
* Fixup query generation for nested joins
* Fixups
* Fix DB exception on ClientPreparedStatement
* events,alerts: Add missing indexes (#366)
* Fixup
* Fix resource count discrepancies
* Fixup while removing vm
* Fix discrepancies when starting VMs
* Fixup tests
* Fixups
* Don't take lock when amount is negative
Adds:
- Fix for volume limit checks for disk offerings with multiple tags - When a VM is deployed with multiple disks having offerings with multiple tags the resource limit check may falter as currently it tries to check based on individual diskoffering. With this, change if offering d1 and d2 for volumes v1 and v2 both have tag1, server will check volume limits for tag tag1 using the combined size of v1 and v2.
- Fix for template tag hosts in random host allocator - May affect use of template tag, service offering tags and random host allocator together. The current code for the random host allocator falters while trying to find the host allocation. This was found and fixed during the addition of the unit test here, https://github.com/shapeblue/cloudstack-apple/pull/378/files#diff-bbf9baea014e5cc1dfe9e7d13467c9857208cfe65e93883721d88a6f0452f912
- Unit tests for changes in api,server,ui: tagged resource limits #327
* Update PowerFlex storage stats on host connect (if any changes in capacity / used bytes)
* Sync the pool stats in DB with the actual stats from stats collector
* Updated capacityBytes check
* Revert "Updated capacityBytes check"
This reverts commit 3ffb17b2c4b3c794e5d0dbf4108d43255b4fbcca.
* Revert "Update PowerFlex storage stats on host connect (if any changes in capacity / used bytes)"
This reverts commit 9e473aed4c589b91f62cbe2fd135dc25e0adc1c3.
* Introduced a new API "checkVolumeAndRepair" that allows users or admins to check and repair if any leaks observed.
Currently this is supported only for KVM
* some fixes
* Added unit tests
* addressed review comments
* add repair volume while granting access
* Changed repair parameter to accept both leaks/all values
* Introduced new global setting volume.check.and.repair.before.use to do volume check and repair before VM start or volume attach operations
* Added volume check and repair changes only during VM start and volume attach operations
* Refactored the names to look similar across the code
* Some code fixes
* remove unused code
* Renamed repair values
* Addressed review comments
* code refactored
* used volume name in logs
* Changed the API to Async and the setting scope to storage pool
* Fixed exit value handling with check volume command
* Fixed storage scope to the setting
* Fixed volume format issues
* Refactored the log messages
* Fix formatting
* Fix host stuck in connecting state (#8502)
There are a lot of test failures due to test_vm_life_cycle.py in multiple PRs due to host not available for migration of VMs.
#8438 (comment)
#8433 (comment)
#7344 (comment)
While debugging I noticed that the hosts get stuck in Connecting state because MS is waiting for a response of the ReadyCommand from the agent. Since we take a lock on connection and disconnection, restarting the agent doesn't work. To fix this, we have to restart the MS or wait for ~1 hour (default timeout).
On the agent side, it gets stuck waiting for a response from the Script execution.
To reproduce, run smoke/test_vm_life_cycle.py (TestSecuredVmMigration test class to be specific). Once the tests are complete, you will notice that some hosts are stuck in Connecting state. And restarting the agent fails due to the named lock. Locks on DB can be checked using the below query.
SELECT *
FROM performance_schema.metadata_locks
INNER JOIN performance_schema.threads ON THREAD_ID = OWNER_THREAD_ID
WHERE PROCESSLIST_ID <> CONNECTION_ID() \G;
This PR adds a wait for the ready command and a timeout to the Script execution to ensure that the thread doesn't get stuck and the named lock from database is released.
* Externalise a few timeouts & fix timeout for hostSupportsUefi in libvirt ready command wrapper (#8547)
This PR fixes bug introduced in #8502. Timeout for script execution was set to 60 ms instead of 60s which resulted in host not getting UEFI enabled. This is a blocker for 4.19 release.
We do this by introducing a new agent parameter `agent.script.timeout` (default - 60 seconds) to use as a timeout for the script checking host's UEFI status.
We also externalize the timeout for the ReadyCommand by introducing a new global setting `ready.command.wait` (default - 60 seconds).
For ModifyStoragePoolCommand, we don't externalize the timeout to avoid confusion for the user. Since, the required timeout can vary depending on the provider in use and we are only setting the wait for default host listener for now. Instead, we reuse the global `wait` setting by dividing it by `5` making the default value of 6 minutes (1800/5 = 360s) for ModifyStoragePoolCommand.
Note: the actual time, the MS waits is twice the wait set for a Command. Check reference code below.
19250403e6/engine/orchestration/src/main/java/com/cloud/agent/manager/AgentAttache.java (L406-L442)
* fixup
* Check if volume on datastore requires access for migration, and grant/revoke volume access if requires
* Updated default implementation for requiresAccessForMigration method in PrimaryDataStoreDriver
* Update VM's state if powerstate & state are not in sync
* Add unit tests
* some code improvements for instacne / power state check
* Update power state after vm stop confirmation (as power state 'PowerOn' is kept after vm stop and not updated later)
* Reset the power state update counter before migrate, to allow power state sync to proper state / host
* Do not consider transitional states (Starting, Stopping) to check power state sync
* set powerstate to off for all vm types
---------
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
* Added vm uuid as part of error response when vm create fails after vm entity is persisted
* Fixed styling issue
* Fixed styling issue.
* Fix unit tests
* Fixed merge conflicts.
* Fixed merge conflicts.
---------
Co-authored-by: Annie Li <ji_li@apple.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
Introduces the concept of tagged resource limits. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types,
Host limits
user_vm
cpu
memory
Storage limits
volume
primary_storage
Following global settings can used to specify tags for which limit needs to be enforced,
Host: resource.limit.host.tags
Storage: resource.limit.storage.tags
Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI.
Enhances use of templatetag for VM deployment and template creation
Adds option to list disk offering with suitability flag for a virtualmachine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the reponse.
Rohit Yadav
Vishesh
Marcus Sorensen
Wei Zhou
Abhishek Kumar
dahn
Suresh Kumar Anaparti
Harikrishna
anniejili