Commit Graph

3709 Commits

Author SHA1 Message Date
Suresh Kumar Anaparti 52d986081b
Updated Endpoint Selector to pick the Cluster in Enabled state (in addition to Host state) (#10757)
* Consider the clusters with allocation state 'Enabled' for EndPoint selection (in addition to Host state)

* Reset the pool id when create volume fails on the allocated pool

- the pool id is persisted while creating the volume, when it fails the pool id is not reverted. On next create volume attempt, CloudStack couldn't find any suitable primary storage even there are pools available with enough capacity as the pool is already assigned to volume which is in Allocated state (and storage pool compatibility check fails). Ensure volume is not assigned to any pool if create volume fails (so the next creation job would pick the suitable pool).

* endpoint check for resize

* update the resize error through callback result instead of exception

* logger fix
2025-05-13 17:48:49 +05:30
Suresh Kumar Anaparti 95489b8bdd
Direct agents rebalance improvements with multiple management server nodes (#10674)
Sometimes hypervisor hosts (direct agents) stuck with Disconnect state during agent rebalancing activity across multiple management server nodes. This issue was noticed during frequent restart of the management server nodes in the cluster.

When there are multiple management server nodes in a cluster, if one or more nodes are shutdown/start/restart, CloudStack will rebalance the hosts among the remaining nodes or move the nodes to the newly joined management server nodes. During the rebalancing period multiple operations could happen including:

- DirectAgentScan at interval of configured direct.agent.scan.interval
- AgentRebalanceScan to identify and schedule rebalance agents
- TransferAgentScan to transfer the host from original owner to future owner

**Current Rebalance behavior**

1. For hosts that have AgentAttache && not forForward but in Disconnect state, CloudStack simply ignore these hosts without trying to ping again or update the status of the host.
2. For hosts that have AgentAttache && forForward, CloudStack removes the agent but still try to loadDirectlyConnectedHost.

**Improved Rebalance behavior**
During DirectAgentScan: scanDirectAgentToLoad(),  identify hosts that for self-managed hosts that are in Disconnect state (disconnected after pingtimeout).

1. For hosts that have AgentAttache and is forForward, CloudStack should remove the agent
2. For hosts that have AgentAttache and is not forForward but in Disconnect state, CloudStack should try to investigate and update the status to Up if host is pingable.
3. For hosts that don't have AgentAttache, CloudStack should try to loadDirectlyConnectedHost.
2025-05-13 17:47:46 +05:30
Daan Hoogland 64828f66e8 Merge branch '4.20' 2025-05-13 13:34:23 +02:00
Pearl Dsilva 0648d000b2
Support XenServer 8.4 / XCP 8.3 - make scripts python3 compatible (#10684) 2025-05-13 12:35:04 +02:00
Phsm Qwerty 011fced91e
ehancement: add password to configdrive vendor_data.json (#10061) 2025-05-12 16:16:54 +02:00
João Jandre 6fdaf51ddc
KVM incremental snapshot feature (#9270)
* KVM incremental snapshot feature

* fix log

* fix merge issues

* fix creation of folder

* fix snapshot update

* Check for hypervisor type during parent search

* fix some small bugs

* fix tests

* Address reviews

* do not remove storPool snapshots

* add support for downloading diff snaps

* Add multiple zones support

* make copied snapshots have normal names

* address reviews

* Fix in progress

* continue fix

* Fix bulk delete

* change log to trace

* Start fix on multiple secondary storages for a single zone

* Fix multiple secondary storages for a single zone

* Fix tests

* fix log

* remove bitmaps when deleting snapshots

* minor fixes

* update sql to new file

* Fix merge issues

* Create new snap chain when changing configuration

* add verification

* Fix snapshot operation selector

* fix bitmap removal

* fix chain on different storages

* address reviews

* fix small issue

* fix test

---------

Co-authored-by: João Jandre <joao@scclouds.com.br>
2025-05-12 10:50:30 -03:00
Pearl Dsilva 1e5d133033 Merge branch '4.20' of https://github.com/apache/cloudstack 2025-05-12 13:12:09 +05:30
Pearl Dsilva a21f912be3 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-05-12 12:41:34 +05:30
Wei Zhou 7e2aa0efe4
engine/schema: create default network offering for vpc tier with conserve_mode=1 for fresh installation (#10744) 2025-05-09 13:51:43 +05:30
Fabricio Duarte 696bc50f3b
Backport #9888 to 4.19: Fix Usage inconsistencies (#10712) 2025-05-09 09:49:14 +02:00
Abhishek Kumar 919c9797cc
server: prevent duplicate HA works and alerts (#10624)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-05-06 10:42:30 +02:00
Pearl Dsilva 39c5641cbe Merge branch '4.20' of https://github.com/apache/cloudstack 2025-05-05 10:15:05 +05:30
Pearl Dsilva 91c1168861 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-05-02 14:59:05 +05:30
Suresh Kumar Anaparti 32cc45e840
[UI] Allow quiescevm and asyncbackup flags while taking volume snapshot from UI when these are supported for the volume (#10265) 2025-05-02 10:45:50 +02:00
Wei Zhou fd74895ad0
New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) (#10514) 2025-05-02 09:15:03 +02:00
Daan Hoogland d7d9d131b2 Merge branch '4.20' 2025-05-01 15:44:09 +02:00
Suresh Kumar Anaparti 9f229600e6
Add new config (non-dynamic) for agent connections monitor thread, and keep timeunit to secs (in sync with the earlier Wait config) (#10525) 2025-04-28 15:32:03 +02:00
Pearl Dsilva 2df1ac5106 Merge branch '4.20' of https://github.com/apache/cloudstack 2025-04-28 12:15:48 +05:30
Pearl Dsilva 0785ba046e Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-04-28 11:10:08 +05:30
Fabricio Duarte 9d263cd71b
Network Usage event model adjustments (#10755) 2025-04-26 17:35:28 +02:00
Wei Zhou 5d28e66b52
vTPM: support KVM and VMware (#10543)
* KVM: add Virtual TPM model and version

* KVM: add admin-only VM setting GUEST.CPU.MODE and GUEST.CPU.MODEL

* VMware: add vTPM

* vTPM: do not set Key due to 'Cannot add multiple devices using the same device key..'

* vTPM: add unit test testTpmModel

* engine/schema: remove user vm details for guest CPU mode/model

* vTPM: extra methods as Daan's requests

* vTPM: add unit tests in VmwareResourceTest

* vTPM: update unit tests in VmwareResourceTest

* vTPM: add unit test in LibvirtComputingResourceTest

* vTPM: use the default TPM version if an invalid version is passed

* vTPM: requires UEFI on vmware and do nothing if it is not enabled/disabled

* vTPM: let uses to add UEFI on vmware

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* vTPM: remove template details for guest CPU mode/model

* UI: boot vm from ISO into UEFI/SECURE mode

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-04-26 14:35:36 +05:30
Abhishek Kumar 12c077d704
api,ui: multi arch improvements (#10289) 2025-04-25 11:02:27 +02:00
Daan Hoogland 3c75d9363b Merge branch '4.20' 2025-04-17 15:59:41 +02:00
Daan Hoogland d7765343ef Merge branch '4.19' into 4.20 2025-04-17 15:40:10 +02:00
Wei Zhou 7b68615bd9
HA: set correct hostId of HA work for vm migration (#10591) 2025-04-17 10:02:46 +02:00
Bernardo De Marco Gonçalves 40d549b075
backport 10273 (#10702) 2025-04-16 08:45:31 +02:00
Fabricio Duarte ac6b1b382c
Migrate public templates that have URLs on data migration across secondary storages (#10364)
Co-authored-by: Fabricio Duarte <fabricio.duarte@scclouds.com.br>
2025-04-15 13:48:45 +02:00
Bryan Lima cb4848bc1a
Add support to RBD erasure code pools (#9808)
* Readd filename string on qemuimg create

* Remove empty object on the data pool details of storage pools with no data pool

* Only use the method createPhysicalDiskByLibVirt with RBD when the pool is of erasure code type. Also added javadoc for createPhysicalDisk method

* Change literal '/' string to File.separator

* Add support for erasure code pools

* Fix null on putAll
2025-04-02 08:19:00 -03:00
Daan Hoogland 8af021c6f6 Merge branch '4.20' 2025-03-27 17:03:13 +01:00
Daan Hoogland 5f93ce71bb Merge branch '4.19' into 4.20 2025-03-27 16:44:42 +01:00
Suresh Kumar Anaparti 9dceae4614
MS maintenance improvements (#10417)
* Update last agents during ms maintenance, and some code improvements

* Send 503 (Service Unavailable) response status when maintenance or shutdown is initiated
[Any load balancer in the clustered environment can avoid routing requests to this MS node]

* Migrate systemvm agents before routing host agents, and some code improvements

* Added events for ms maintenance and shutdown operations

* Added the following ms maintenance and shutdown improvements

- block new agent connections during prepare for maintenance of ms

- maintain avoids ms list

- propagate updated management servers list and lb algorithm in host and indirect.agent.lb.algorithm settings respectively, to systemvm (non-routing) agents

- updated setup ms list and migrate agent connections to executor service

- migrate agent connection through executor, and send the answer to the ms host that initiated the migration

- re-initialize ssl handshake executor if it is shutdown

- don't allow prepare for maintenance or shutdown when other management server nodes are in preparing states

- don't allow trigger shutdown when management server is up and other management server nodes are in preparing states

- stop agent connections monitor on ms maintenance

- update avoid ms list in ready command

- updated connected host from the client connection

- update last agents in ms metrics from the database

- updated some agent config descriptions

- update last management server in the hosts during shutdown

- added agents and lastagents in management server response

- updated management server maintenance & shutdown unit tests

- some code improvements

* refactored code / addressed comments

* removed shutdown testcase (maybe, calling System.exit)

* Revert "removed shutdown testcase (maybe, calling System.exit)"

This reverts commit e14b071715.

* avoid system.exit during shutdown test

* code improvements

* testcase fix

* Fix cutoff time in agent connections monitor thread
2025-03-19 14:18:05 +05:30
Rene Peinthor f4a7c8ab89
linstor: implement missing deleteDatastore (#10561)
Somehow deleteDatastore was never implemented, that meant:
templates haven't been cleaned up on datastore delete and
also agents have never been informed about storage pool removal.
2025-03-18 08:50:19 -04:00
Abhishek Kumar 1c1dad977e Merge remote-tracking branch 'apache/4.20' 2025-03-06 09:55:27 +05:30
Pearl Dsilva 3aabedd447
UI: Proper explanation for the global setting to avoid ambiguity (#10042) 2025-03-04 15:07:43 +01:00
Pearl Dsilva bdae23ed53
Fix listing disk offerings for newly created VMs that haven't yet been started (#10476) 2025-02-28 10:24:23 -05:00
Pearl Dsilva 3a28a87483 Merge branch '4.20' of https://github.com/apache/cloudstack 2025-02-27 11:20:25 -05:00
Daan Hoogland 88916dcf2b Merge branch '4.19' into 4.20 2025-02-27 17:10:54 +01:00
Wei Zhou f992ebb52a
fix volume migration across cluster-scope pools (#10266) 2025-02-27 08:50:22 -05:00
Abhishek Kumar e8ac477e9f
engine/orchestration: fix missing vm powerstate update vm state (#10407)
* engine/orchestration: fix missing vm powerstate update vm state

Fixes #10406
VMs were not moving to Stopped state when PowerReportMissing is processed.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add unit tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add license

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add lenient

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-02-25 15:50:27 +05:30
Daan Hoogland 4a3686297d Updating pom.xml version numbers for release 4.19.3.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-25 10:43:11 +01:00
Daan Hoogland 24b7c66251 Merge branch '4.20' 2025-02-24 14:33:12 +01:00
Nicole Schmidt c80b8860e4
Fix hostId verification on unsuccessful expunge operation (#10418) 2025-02-20 09:11:53 -05:00
Daan Hoogland 4e321d4356 Updating pom.xml version numbers for release 4.19.2.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-20 09:32:07 +01:00
Fabricio Duarte c0e05c4a6d
Fix Usage inconsistencies (#9888) 2025-02-19 08:44:26 -03:00
GaOrtiga 864751d5f9
Fix ordering of secondary storages with the algorithm `firstfitleastconsumed` (#8557)
* Fix ordering of secondary storages with the algorithm `firstfitleastconsumed`

* return store without checking all

* Add unit tests

---------

Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2025-02-14 08:33:04 -03:00
Abhisar Sinha 2a4a1f73d0
Support multi-scope configuration settings (#10300)
This PR introduces the concept of multi-scope configuration settings. In addition to the Global level, currently all configurations can be set at a single scope level.
It will be useful if a configuration can be set at multiple scopes. For example, a configuration set at the domain level
will apply for all accounts, but it can be set for an account as well. In which case the account level setting will override the domain level setting.

This is done by changing the column `scope` of table `configuration` from string (single scope) to bitmask (multiple scopes).

```
public enum Scope {
    Global(null, 1),
    Zone(Global, 1 << 1),
    Cluster(Zone, 1 << 2),
    StoragePool(Cluster, 1 << 3),
    ManagementServer(Global, 1 << 4),
    ImageStore(Zone, 1 << 5),
    Domain(Global, 1 << 6),
    Account(Domain, 1 << 7);
```
Each scope is also assigned a parent scope. When a configuration for a given scope is not defined but is available for multiple scope types, the value will be retrieved from the parent scope. If there is no parent scope or if the configuration is defined for a single scope only, the value will fall back to the global level.

Hierarchy for different scopes is defined as below :
- Global
    - Zone
        - Cluster
            - Storage Pool
        - Image Store
    - Management Server
    - Domain
        - Account

This PR also updates the scope of the following configurations (Storage Pool scope is added in addition to the existing Zone scope):
- pool.storage.allocated.capacity.disablethreshold
- pool.storage.allocated.resize.capacity.disablethreshold
- pool.storage.capacity.disablethreshold

Doc PR : https://github.com/apache/cloudstack-documentation/pull/476

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-02-14 11:25:01 +05:30
GaOrtiga f8563b86e7
Fix NPE during kubernetes cluster creation when network has rules with ports saved as null on DB (#9223)
Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2025-02-13 10:25:16 +01:00
Daan Hoogland 0dcb8da03a Merge branch '4.20' 2025-02-12 16:54:05 +01:00
Daan Hoogland 4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
Rene Glover 3337f425ff
Primera pure patches & various small fixes (#10132)
Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-02-07 13:19:34 +01:00
Rene Peinthor c09720a19a
systemvm-registration: update seeded template_store_ref sizes (#10317) 2025-02-07 13:19:05 +01:00
Abhisar Sinha a7beaaf73b
Add Resource Limits to Backups and Object Storage (#10017)
Doc PR : https://github.com/apache/cloudstack-documentation/pull/461
This PR fixes https://github.com/apache/cloudstack/issues/8638

== Description

Four new Resource Types have been added. Admin can configure corresponding resource limits for the tenants at different levels (domain, account, project) 
User dashboard's Storage section will show the new resources, their limits and current usage.

1. backup - No. of backups used by the account
2. backup_storage - Backup storage allocated for the account
3. bucket - No. of buckets used by the accounts
4. object_storage - Object storage allocated for the account.

Some other related changes done to BnR framework:

1. Maximum number of Backups to retain can be specified while creating Backup schedules, similar to Scheduled snapshots.
2. Oldest Scheduled backup of the same interval type will be deleted once the number reaches the configured max Backups value.
3. Code refactor: Moved syncBackups method from BackupProvider to the framework BackupManagerImpl, as it is a common functionality and all providers were using duplicated code.

Changes done to the Object Storage Framework

1. Quota parameter is made mandatory while creating a bucket. Bucket quota is considered to be the allocated space and will be used to enforce Resource limits.

== Schema Changes:

1. New Column `max_backups` added to `backup_schedule` table
4. New Column `backup_interval_type` added to `backups` table

== Api Changes:

1. createBackup: new Parameter `scheduleid`. It should be specified whenever a scheduled backup is created. This will translate to the `backup_interval_type` in the `backups` table.
3. createBackupScheduke: new Parameter `max_backups`. To specify maximum number of backups to retain for the given schedule.

== Configurations:

|Setting |Scope |Default Value |Description|
|-------|--------|--------------|-----------|
|backup.max.hourly |Global |8 |Maximum recurring hourly backups to be retained for an instance|
|backup.max.daily |Global |8 |Maximum recurring daily backups to be retained for an instance|
|backup.max.weekly |Global |8 |Maximum recurring weekly backups to be retained for an instance|
|backup.max.monthly |Global |8 |Maximum recurring monthly backups to be retained for an instance|
|max.account.backups| Global| 20 | The default maximum number of backups that can be created for an account|
|max.account.backup.storage| Global| 400 | The default maximum backup storage space (in GiB) that can be used for an account|
|max.domain.backups| Global| 40 | The default maximum number of backups that can be created for an domain|
|max.domain.backup.storage| Global| 800 | The default maximum backup storage space (in GiB) that can be used for an domain|
|max.project.backups| Global| 20 | The default maximum number of backups that can be created for an project|
|max.project.backup.storage| Global| 400 | The default maximum backup storage space (in GiB) that can be used for an project|

|Setting |Scope |Default Value |Description|
|-------|--------|--------------|-----------|
|max.account.buckets| Global| 20 | The default maximum number of buckets that can be created for an account|
|max.account.object.storage| Global| 400 | The default maximum object storage space (in GiB) that can be used for an account|
|max.domain.buckets| Global| 40 | The default maximum number of buckets that can be created for an domain|
|max.domain.object.storage| Global| 800 | The default maximum object storage space (in GiB) that can be used for an domain|
|max.project.buckets| Global| 20 | The default maximum number of buckets that can be created for an project|
|max.project.object.storage| Global| 400 | The default maximum object storage space (in GiB) that can be used for an project|


Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-07 16:56:20 +05:30
Manoj Kumar 648170cf9b
Fix lint error coming up in main (#10342) 2025-02-07 09:32:44 +01:00
Abhisar Sinha c5afee2101
UI improvements (#9773)
* Show Usage Server configuration in a separate pane

* UI: Option to attach volume to an instance during create volume

* Show service ip in management server details tab

* change Schedule Snapshots to Recurring Snapshots

* Change the hypervisor order so that kvm, vmware, xenserver show up first

* Remove extra space in hypervisor names in config.java

* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>

* Autofill vcenter details in add cluster form

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* Fix alignment on wrapping in global settings tabs

* rename Autofill vCenter credentials to Autofill vCenter credentials from Zone

* Rename Service Ip to Ip Address in management server response

* Change description of kvm.snapshot.enabled to say that it applies to volume snapshots

* Return error when kvm vm snapshot is taken withoutsnapshot memory

* Minor naming changes and grammar

* Fix tooltip for attach volume to instance button

* Show Usage Server configuration in a separate pane

* UI: Option to attach volume to an instance during create volume

* Show service ip in management server details tab

* change Schedule Snapshots to Recurring Snapshots

* Change the hypervisor order so that kvm, vmware, xenserver show up first

* Remove extra space in hypervisor names in config.java

* Autofill vcenter details in add cluster form

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* Fix alignment on wrapping in global settings tabs

* rename Autofill vCenter credentials to Autofill vCenter credentials from Zone

* Rename Service Ip to Ip Address in management server response

* Change description of kvm.snapshot.enabled to say that it applies to volume snapshots

* Return error when kvm vm snapshot is taken withoutsnapshot memory

* Minor naming changes and grammar

* Fix tooltip for attach volume to instance button

* Show Usage Server configuration in a separate pane

* UI: Option to attach volume to an instance during create volume

* Show service ip in management server details tab

* change Schedule Snapshots to Recurring Snapshots

* Change the hypervisor order so that kvm, vmware, xenserver show up first

* Remove extra space in hypervisor names in config.java

* Autofill vcenter details in add cluster form

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* Fix alignment on wrapping in global settings tabs

* rename Autofill vCenter credentials to Autofill vCenter credentials from Zone

* Rename Service Ip to Ip Address in management server response

* Change description of kvm.snapshot.enabled to say that it applies to volume snapshots

* Return error when kvm vm snapshot is taken withoutsnapshot memory

* Minor naming changes and grammar

* Fix tooltip for attach volume to instance button

* UI: Option to attach volume to an instance during create volume

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* moved db changes from 41900to42000 to 42000to42010

* Update group_id in already present usage configuration settings

* remove "schedule" from message in create Recurring Snapshots form

* Update server/src/main/java/com/cloud/vm/snapshot/VMSnapshotManagerImpl.java

---------

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Co-authored-by: Boris Stoyanov - a.k.a Bobby <bss.stoyanov@gmail.com>
Co-authored-by: Andrija Panic <45762285+andrijapanicsb@users.noreply.github.com>
2025-02-06 11:18:40 +05:30
Harikrishna 60af31c9c0
Decrypt zone, cluster, storage details for configuration values (#10237)
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
2025-02-03 13:30:57 +01:00
Daan Hoogland 2654890e86 Merge branch '4.20' 2025-02-01 21:20:08 +01:00
Daan Hoogland 085bd3bda5 Merge branch '4.19' into 4.20 2025-02-01 17:51:50 +01:00
Abhishek Kumar 0b5a5e8043
api,agent,server,engine-schema: scalability improvements (#9840)
* api,agent,server,engine-schema: scalability improvements

Following changes and improvements have been added:

- Improvements in handling of PingRoutingCommand

    1. Added global config - `vm.sync.power.state.transitioning`, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
    2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
    3. Optimized scanning stalled VMs

- Added option to set worker threads for capacity calculation using config - `capacity.calculate.workers`

- Added caching framework based on Caffeine in-memory caching library, https://github.com/ben-manes/caffeine

- Added caching for account/use role API access with expiration after write can be configured using config - `dynamic.apichecker.cache.period`. If set to zero then there will be no caching. Default is 0.

- Added caching for account/use role API access with expiration after write set to 60 seconds.

- Added caching for some recurring DB retrievals

    1. CapacityManager - listing service offerings - beneficial in host capacity calculation
    2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
    3. DownloadListener - hypervisors for zone - beneficial for host joins
    5. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands

- Optimized MS list retrieval for agent connect

- Optimize finding ready systemvm template for zone

- Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks

- Changes in agent-agentmanager connection with NIO client-server classes

    1. Optimized the use of the executor service
    2. Refactore Agent class to better handle connections.
    3. Do SSL handshakes within worker threads
    5. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - `agent.ssl.handshake.min.workers` and `agent.ssl.handshake.max.workers` can be used to control number of new connections management server handles at a time. `agent.ssl.handshake.timeout` can be used to set number of seconds after which SSL handshake times out at MS end.
    6. On agent side backoff and sslhandshake timeout can be controlled by agent properties. `backoff.seconds` and `ssl.handshake.timeout` properties can be used.

- Improvements in StatsCollection - minimize DB retrievals.

- Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.

- Improvements in hosts connection for a storage pool. Added config - `storage.pool.host.connect.workers` to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.

- Minor improvements in resource limit calculations wrt DB retrievals

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* test1, domaindetails, capacitymanager fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test2 - agent tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* capacitymanagertest fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert marvin/setup.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix indent

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* use space in sql

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address duplicate

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update host logs

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert e36c6a5d07

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix npe in capacity calculation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* move schema changes to 4.20.1 upgrade

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix build

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add some more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* checkstyle fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unnecessary mocks

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* replace statics

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* engine/orchestration,utils: limit number of concurrent new agent
connections

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor - remove unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unregister closed connections, monitor & cleanup

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add check for outdated vm filter in power sync

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* agent: synchronize sendRequest wait

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-01 12:28:41 +05:30
Suresh Kumar Anaparti 0d5047b8b7
Improve listing of HA and non-HA hosts when ha.tag setting is defined and hosts have multiple tags along with ha tag (#10240) 2025-01-31 14:07:00 +01:00
Abhishek Kumar 7abda3b963 Merge remote-tracking branch 'apache/4.20' 2025-01-31 16:55:38 +05:30
Abhishek Kumar ae2ffbe40b Merge remote-tracking branch 'apache/4.19' into 4.20 2025-01-31 16:54:22 +05:30
Wei Zhou fbb1ff78d6
Static Routes: fix check on wrong global configuration (#10066) 2025-01-31 11:04:13 +01:00
Fabricio Duarte 4201a2dc11
Fix Usage job not executing successfully (#10244)
* Add `api_key_access` to Usage account table

* use the procedure in cloud_usage instead
2025-01-30 22:07:50 +01:00
Bernardo De Marco Gonçalves a335feab6a
fix allocation of vmfs storage pools (#10201) 2025-01-30 18:44:48 +01:00
Bernardo De Marco Gonçalves d9af9bdb36
fix SQL syntax erros and target 419 (#10273) 2025-01-30 15:51:55 +01:00
Abhishek Kumar b93589b5bd
server: reset 2fa user configuration on incomplete setup (#10247)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-30 15:51:04 +01:00
Suresh Kumar Anaparti 3b108b968f
Support for Management Server Maintenance Mode (#9854)
* Support for Management Server Maintenance

- New APIs: prepareForMaintenance and cancelMaintenance, with required parameter - managementserverid.

- New management server states for maintenance: PreparingForMaintenance, Maintenance.

- listHosts API with optional parameter – managementserverid, to list the hosts connected to the management server.

- Support management server maintenance when more than one active management servers available.

- Triggers transfer agents to other available management servers for maintenance, new agent command MigrateAgentConnectionCommand to initiate transfer of indirect agents.

- New global config 'management.server.maintenance.timeout', to set the timeout (in mins) for the management server maintenance window, default: 60 mins.

- UI changes: Prepare and Cancel Maintenance in Management Server section, Connected Agents tab, New fields for hosts and management servers.

* Updated pending jobs check timer task with ScheduledExecutorService

* keep maintenance state on trigger shutdown call when ms is in maintenance

* add pending jobs count to ms response

* during ms heartbeat, update state to up only when it's down

* allow vm work jobs of async job created before prepare for maintenance

* Revert "keep maintenance state on trigger shutdown call when ms is in maintenance"

This reverts commit 607e13364679eac897f4d146bb3325ea7a61ba17.

* skip maintenance test when multiple management servers are not available, and not configured in host setting for kvm
2025-01-29 13:31:15 +05:30
Daan Hoogland 048649d351 Merge release branch 4.20 to main
* 4.20:
  server: investigate pending HA work when executing in new MS session (#10167)
  extra null guard (#10264)
2025-01-28 14:34:19 +01:00
Daan Hoogland 717ce981d4 Merge release branch 4.19 to 4.20
* 4.19:
  extra null guard (#10264)
2025-01-28 14:33:49 +01:00
Abhishek Kumar 33a37da9ec
server: investigate pending HA work when executing in new MS session (#10167)
For HA work items that are created for host state change, checks must be
done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track
the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA
work items in Investigating state. During execution of the HA work if it
is found in investigating state, checks are done to verify if the work
is still valid. If the jobs is found to be invalid it is cancelled.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-28 14:39:31 +05:30
dahn f652ad0d98
extra null guard (#10264) 2025-01-27 14:14:31 +01:00
Daan Hoogland 98f5663954 Merge branch '4.20' 2025-01-24 17:10:43 +01:00
Daan Hoogland 34d2a3bc86 Merge branch '4.19' into 4.20 2025-01-24 17:01:42 +01:00
dahn 0a77eb7f85
deal with NPE during host reconnect (#10158)
* log to see what command is being processed

* exception names
2025-01-24 15:39:56 +05:30
Daan Hoogland ecd60a4e46 Merge release branch 4.20 to main
* 4.20:
  Maintenance mode: Add host to deployment planner avoid list to fix local storage vm migration (#9892)
  Add project-user association normalization script to 4.20.1 upgrade (#10116)
  fix slider component for global settings of the range type (#10187)
  Clean up network permissions on account deletion (#10176)
2025-01-20 11:58:28 +01:00
Daan Hoogland 9f594c9699 Merge branch '4.19' into 4.20 2025-01-20 11:53:32 +01:00
Fabricio Duarte 2300c1caa3
Add project-user association normalization script to 4.20.1 upgrade (#10116) 2025-01-16 16:18:30 +01:00
Fabricio Duarte 449d3c7cb1
Create API to list Quota credits (#9590)
Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>
2025-01-16 15:19:32 +01:00
Bernardo De Marco Gonçalves 796bd4f72c
Clean up network permissions on account deletion (#10176) 2025-01-15 17:38:20 +01:00
Daan Hoogland bd874eaa44 Merge release branch 4.20 to main
* 4.20:
  systemvm: fix keystore is reset when patch a systemvm (#9900)
  no retrieval of null hosts (#10175)
  upgrade: consider multiple hypervisors and secondary storages (#10046)
  CheckOnHostCommand: add missing timeout setting (#9677)
2025-01-13 11:23:51 +01:00
Daan Hoogland e2cfddb1b6 Merge branch '4.19' into 4.20 2025-01-13 11:23:14 +01:00
Wei Zhou 7adc732992
upgrade: consider multiple hypervisors and secondary storages (#10046) 2025-01-13 09:28:18 +01:00
Daan Hoogland fadb39ece7 Merge release branch 4.20 to main
* 4.20:
  merge errors fixed
  Restrict the migration of volumes attached to VMs in Starting state (#9725)
  server, plugin: enhance storage stats for IOPS (#10034)
  Introducing granular command timeouts global setting (#9659)
  Improve logging to include more identifiable information (#9873)
2025-01-08 14:01:19 +01:00
Abhishek Kumar bd488c4bba
server, plugin: enhance storage stats for IOPS (#10034)
Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned.
For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool.
For local storage pool, implementation has been added using iostat to return currently used IOPS.
StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-07 17:17:12 +05:30
Harikrishna 9bc283e5c2
Introducing granular command timeouts global setting (#9659)
* Introducing granular command timeouts global setting

* fix marvin tests

* Fixed log messages

* some more log message fix

* Fix empty value setting

* Converted the global setting to non-dynamic

* set wait on command only when granular wait is defined. This is to keep the backward compatibility

* Improve error logging
2025-01-07 17:06:32 +05:30
Vishesh a4224e58cc
Improve logging to include more identifiable information (#9873)
* Improve logging to include more identifiable information for kvm plugin

* Update logging for scaleio plugin

* Improve logging to include more identifiable information for default volume storage plugin

* Improve logging to include more identifiable information for agent managers

* Improve logging to include more identifiable information for Listeners

* Replace ids with objects or uuids


* Improve logging to include more identifiable information for engine

* Improve logging to include more identifiable information for server

* Fixups in engine

* Improve logging to include more identifiable information for plugins

* Improve logging to include more identifiable information for Cmd classes

* Fix toString method for StorageFilterTO.java
2025-01-06 16:42:37 +05:30
John Bampton 5bae1188ff
pre-commit fix mixed line endings in XML files (#10148)
https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#mixed-line-ending
2025-01-03 09:42:09 +01:00
Daan Hoogland 2daffa34f2 Merge release branch 4.20 to main
* 4.20:
  VR: fix site-2-site VPN if split connections is enabled (#10067)
  UI: fix cannot open 'Edit tags' modal for static routes (#10065)
  Update ownership selection component to be language independent (#10052)
  Support to enable/disable VM High Availability manager and related alerts (#10118)
2024-12-30 13:35:30 +01:00
Suresh Kumar Anaparti 330ed25a6c
Support to enable/disable VM High Availability manager and related alerts (#10118)
- Adds new config 'vm.ha.enabled'  with Zone scope, to enable/disable VM High Availability manager. This is enable by default (for backward compatibilty).
  When enabled, the VM HA WorkItems (for VM Stop, Restart, Migration, Destroy) can be created and the scheduled items are executed.
  When disabled, new VM HA WorkItems are not allowed and the scheduled items are retried until max retries configured at 'vm.ha.migration.max.retries' (executed in case HA is re-enabled during retry attempts), and then purged after 'time.between.failures' by the cleanup thread that runs regularly at 'time.between.cleanup'.
- Adds new config 'vm.ha.alerts.enabled' with Zone scope, to enable/disable alerts for the VM HA operations. This is enabled by default.
2024-12-26 17:45:32 +05:30
Daan Hoogland 9295a1624d Merge release branch 4.20 to main
* 4.20:
  VR: apply iptables rules when add/remove static routes (#10064)
  Certificate and VM hostname validation improvements (#10051)
  set ulimit for server according to redhat spec (#10040)
  kvm-storage: provide isVMMigrate information to storage plugins (#10093)
  Allow config drive deletion of migrated VM, on host maintenance (#10045)
  linstor: improve heartbeat check with also asking linstor (#10105)
  server: simplify role change validation (#9173)
  UI: create VPC network offering with conserve mode (#10082)
  server: fix typo removeaccessvpn in VirtualRouterElement (#10086)
  UI: remove duplicated Instance Name in Public IP details page (#10087)
  UI: Fixes in the Usage UI (#10000)
  SAML2: add cookie with HttpOnly too #10013 (#10047)
  ui: Allow font-awesome icon usage and optimise icon size inconsistency (#9744)
2024-12-20 14:37:49 +01:00
Daan Hoogland b7f0aac519 Merge branch '4.19' into 4.20 2024-12-20 14:34:39 +01:00
Suresh Kumar Anaparti b4ad04badf
Allow config drive deletion of migrated VM, on host maintenance (#10045) 2024-12-18 09:12:28 +01:00
Bernardo De Marco Gonçalves f75a194c09
Persist IP addresses related to VM access via CPVM (#9534) 2024-12-10 11:43:17 +01:00
Rohit Yadav cb530c1749 Merge remote-tracking branch 'origin/4.20' 2024-12-09 11:36:40 +05:30
Felipe 810c4108c7
engine/schema: Add access modifier to `VolumeVO` (#9394)
* added private access modifiers to VolumeVO and fixed an occurrence where a field was being accessed without a getter

* renamed the field hostip to hostIp and removed duplicated methods
https://github.com/apache/cloudstack/pull/9394#issuecomment-2264816580
2024-12-06 08:51:42 +01:00
Daan Hoogland 27d2de180f Merge release branch 4.20 to main
* 4.20:
  move upgrade code to the right file (#10036)
2024-12-04 14:31:20 +01:00
dahn 0eb94a1396
move upgrade code to the right file (#10036)
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-12-04 14:30:39 +01:00
Daan Hoogland 97f9e8e816 Merge release branch 4.20 to main
* 4.20:
  move sql code to the right file (#10035)
2024-12-04 14:29:42 +01:00
Daan Hoogland 8b9db0869f Merge release branch 4.19 to 4.20
* 4.19:
  move sql code to the right file (#10035)
2024-12-04 14:29:03 +01:00
dahn 762d5493dd
move sql code to the right file (#10035)
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-12-04 14:28:04 +01:00