Commit Graph

3690 Commits

Author SHA1 Message Date
Pearl Dsilva 3a28a87483 Merge branch '4.20' of https://github.com/apache/cloudstack 2025-02-27 11:20:25 -05:00
Daan Hoogland 88916dcf2b Merge branch '4.19' into 4.20 2025-02-27 17:10:54 +01:00
Wei Zhou f992ebb52a
fix volume migration across cluster-scope pools (#10266) 2025-02-27 08:50:22 -05:00
nvazquez fbe0496b77
Merge branch 'main' into cks-enhancements-upstream 2025-02-26 10:23:02 -03:00
Abhishek Kumar e8ac477e9f
engine/orchestration: fix missing vm powerstate update vm state (#10407)
* engine/orchestration: fix missing vm powerstate update vm state

Fixes #10406
VMs were not moving to Stopped state when PowerReportMissing is processed.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add unit tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add license

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add lenient

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-02-25 15:50:27 +05:30
Daan Hoogland 4a3686297d Updating pom.xml version numbers for release 4.19.3.0-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-25 10:43:11 +01:00
Daan Hoogland 24b7c66251 Merge branch '4.20' 2025-02-24 14:33:12 +01:00
nvazquez 82e0bf534b
Fix unit test 2025-02-24 08:31:15 -03:00
nvazquez 0662117722
Merge branch 'main' into cks-enhancements-upstream 2025-02-23 23:13:36 -03:00
Nicole Schmidt c80b8860e4
Fix hostId verification on unsuccessful expunge operation (#10418) 2025-02-20 09:11:53 -05:00
Daan Hoogland 4e321d4356 Updating pom.xml version numbers for release 4.19.2.0
Signed-off-by: Daan Hoogland <daan@onecht.net>
2025-02-20 09:32:07 +01:00
Fabricio Duarte c0e05c4a6d
Fix Usage inconsistencies (#9888) 2025-02-19 08:44:26 -03:00
GaOrtiga 864751d5f9
Fix ordering of secondary storages with the algorithm `firstfitleastconsumed` (#8557)
* Fix ordering of secondary storages with the algorithm `firstfitleastconsumed`

* return store without checking all

* Add unit tests

---------

Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2025-02-14 08:33:04 -03:00
Abhisar Sinha 2a4a1f73d0
Support multi-scope configuration settings (#10300)
This PR introduces the concept of multi-scope configuration settings. In addition to the Global level, currently all configurations can be set at a single scope level.
It will be useful if a configuration can be set at multiple scopes. For example, a configuration set at the domain level
will apply for all accounts, but it can be set for an account as well. In which case the account level setting will override the domain level setting.

This is done by changing the column `scope` of table `configuration` from string (single scope) to bitmask (multiple scopes).

```
public enum Scope {
    Global(null, 1),
    Zone(Global, 1 << 1),
    Cluster(Zone, 1 << 2),
    StoragePool(Cluster, 1 << 3),
    ManagementServer(Global, 1 << 4),
    ImageStore(Zone, 1 << 5),
    Domain(Global, 1 << 6),
    Account(Domain, 1 << 7);
```
Each scope is also assigned a parent scope. When a configuration for a given scope is not defined but is available for multiple scope types, the value will be retrieved from the parent scope. If there is no parent scope or if the configuration is defined for a single scope only, the value will fall back to the global level.

Hierarchy for different scopes is defined as below :
- Global
    - Zone
        - Cluster
            - Storage Pool
        - Image Store
    - Management Server
    - Domain
        - Account

This PR also updates the scope of the following configurations (Storage Pool scope is added in addition to the existing Zone scope):
- pool.storage.allocated.capacity.disablethreshold
- pool.storage.allocated.resize.capacity.disablethreshold
- pool.storage.capacity.disablethreshold

Doc PR : https://github.com/apache/cloudstack-documentation/pull/476

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-02-14 11:25:01 +05:30
GaOrtiga f8563b86e7
Fix NPE during kubernetes cluster creation when network has rules with ports saved as null on DB (#9223)
Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2025-02-13 10:25:16 +01:00
Daan Hoogland 0dcb8da03a Merge branch '4.20' 2025-02-12 16:54:05 +01:00
Daan Hoogland 4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
Rene Glover 3337f425ff
Primera pure patches & various small fixes (#10132)
Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-02-07 13:19:34 +01:00
Rene Peinthor c09720a19a
systemvm-registration: update seeded template_store_ref sizes (#10317) 2025-02-07 13:19:05 +01:00
Abhisar Sinha a7beaaf73b
Add Resource Limits to Backups and Object Storage (#10017)
Doc PR : https://github.com/apache/cloudstack-documentation/pull/461
This PR fixes https://github.com/apache/cloudstack/issues/8638

== Description

Four new Resource Types have been added. Admin can configure corresponding resource limits for the tenants at different levels (domain, account, project) 
User dashboard's Storage section will show the new resources, their limits and current usage.

1. backup - No. of backups used by the account
2. backup_storage - Backup storage allocated for the account
3. bucket - No. of buckets used by the accounts
4. object_storage - Object storage allocated for the account.

Some other related changes done to BnR framework:

1. Maximum number of Backups to retain can be specified while creating Backup schedules, similar to Scheduled snapshots.
2. Oldest Scheduled backup of the same interval type will be deleted once the number reaches the configured max Backups value.
3. Code refactor: Moved syncBackups method from BackupProvider to the framework BackupManagerImpl, as it is a common functionality and all providers were using duplicated code.

Changes done to the Object Storage Framework

1. Quota parameter is made mandatory while creating a bucket. Bucket quota is considered to be the allocated space and will be used to enforce Resource limits.

== Schema Changes:

1. New Column `max_backups` added to `backup_schedule` table
4. New Column `backup_interval_type` added to `backups` table

== Api Changes:

1. createBackup: new Parameter `scheduleid`. It should be specified whenever a scheduled backup is created. This will translate to the `backup_interval_type` in the `backups` table.
3. createBackupScheduke: new Parameter `max_backups`. To specify maximum number of backups to retain for the given schedule.

== Configurations:

|Setting |Scope |Default Value |Description|
|-------|--------|--------------|-----------|
|backup.max.hourly |Global |8 |Maximum recurring hourly backups to be retained for an instance|
|backup.max.daily |Global |8 |Maximum recurring daily backups to be retained for an instance|
|backup.max.weekly |Global |8 |Maximum recurring weekly backups to be retained for an instance|
|backup.max.monthly |Global |8 |Maximum recurring monthly backups to be retained for an instance|
|max.account.backups| Global| 20 | The default maximum number of backups that can be created for an account|
|max.account.backup.storage| Global| 400 | The default maximum backup storage space (in GiB) that can be used for an account|
|max.domain.backups| Global| 40 | The default maximum number of backups that can be created for an domain|
|max.domain.backup.storage| Global| 800 | The default maximum backup storage space (in GiB) that can be used for an domain|
|max.project.backups| Global| 20 | The default maximum number of backups that can be created for an project|
|max.project.backup.storage| Global| 400 | The default maximum backup storage space (in GiB) that can be used for an project|

|Setting |Scope |Default Value |Description|
|-------|--------|--------------|-----------|
|max.account.buckets| Global| 20 | The default maximum number of buckets that can be created for an account|
|max.account.object.storage| Global| 400 | The default maximum object storage space (in GiB) that can be used for an account|
|max.domain.buckets| Global| 40 | The default maximum number of buckets that can be created for an domain|
|max.domain.object.storage| Global| 800 | The default maximum object storage space (in GiB) that can be used for an domain|
|max.project.buckets| Global| 20 | The default maximum number of buckets that can be created for an project|
|max.project.object.storage| Global| 400 | The default maximum object storage space (in GiB) that can be used for an project|


Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-07 16:56:20 +05:30
Manoj Kumar 648170cf9b
Fix lint error coming up in main (#10342) 2025-02-07 09:32:44 +01:00
Abhisar Sinha c5afee2101
UI improvements (#9773)
* Show Usage Server configuration in a separate pane

* UI: Option to attach volume to an instance during create volume

* Show service ip in management server details tab

* change Schedule Snapshots to Recurring Snapshots

* Change the hypervisor order so that kvm, vmware, xenserver show up first

* Remove extra space in hypervisor names in config.java

* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>

* Autofill vcenter details in add cluster form

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* Fix alignment on wrapping in global settings tabs

* rename Autofill vCenter credentials to Autofill vCenter credentials from Zone

* Rename Service Ip to Ip Address in management server response

* Change description of kvm.snapshot.enabled to say that it applies to volume snapshots

* Return error when kvm vm snapshot is taken withoutsnapshot memory

* Minor naming changes and grammar

* Fix tooltip for attach volume to instance button

* Show Usage Server configuration in a separate pane

* UI: Option to attach volume to an instance during create volume

* Show service ip in management server details tab

* change Schedule Snapshots to Recurring Snapshots

* Change the hypervisor order so that kvm, vmware, xenserver show up first

* Remove extra space in hypervisor names in config.java

* Autofill vcenter details in add cluster form

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* Fix alignment on wrapping in global settings tabs

* rename Autofill vCenter credentials to Autofill vCenter credentials from Zone

* Rename Service Ip to Ip Address in management server response

* Change description of kvm.snapshot.enabled to say that it applies to volume snapshots

* Return error when kvm vm snapshot is taken withoutsnapshot memory

* Minor naming changes and grammar

* Fix tooltip for attach volume to instance button

* Show Usage Server configuration in a separate pane

* UI: Option to attach volume to an instance during create volume

* Show service ip in management server details tab

* change Schedule Snapshots to Recurring Snapshots

* Change the hypervisor order so that kvm, vmware, xenserver show up first

* Remove extra space in hypervisor names in config.java

* Autofill vcenter details in add cluster form

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* Fix alignment on wrapping in global settings tabs

* rename Autofill vCenter credentials to Autofill vCenter credentials from Zone

* Rename Service Ip to Ip Address in management server response

* Change description of kvm.snapshot.enabled to say that it applies to volume snapshots

* Return error when kvm vm snapshot is taken withoutsnapshot memory

* Minor naming changes and grammar

* Fix tooltip for attach volume to instance button

* UI: Option to attach volume to an instance during create volume

* UI: condition to display create vm-vol-snapshots to same as create vol-snapshots

* moved db changes from 41900to42000 to 42000to42010

* Update group_id in already present usage configuration settings

* remove "schedule" from message in create Recurring Snapshots form

* Update server/src/main/java/com/cloud/vm/snapshot/VMSnapshotManagerImpl.java

---------

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Co-authored-by: Boris Stoyanov - a.k.a Bobby <bss.stoyanov@gmail.com>
Co-authored-by: Andrija Panic <45762285+andrijapanicsb@users.noreply.github.com>
2025-02-06 11:18:40 +05:30
Harikrishna 60af31c9c0
Decrypt zone, cluster, storage details for configuration values (#10237)
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
2025-02-03 13:30:57 +01:00
nvazquez 67a51beefe
Merge branch 'main' into cks-enhancements-upstream 2025-02-03 08:45:53 -03:00
Daan Hoogland 2654890e86 Merge branch '4.20' 2025-02-01 21:20:08 +01:00
Daan Hoogland 085bd3bda5 Merge branch '4.19' into 4.20 2025-02-01 17:51:50 +01:00
Abhishek Kumar 0b5a5e8043
api,agent,server,engine-schema: scalability improvements (#9840)
* api,agent,server,engine-schema: scalability improvements

Following changes and improvements have been added:

- Improvements in handling of PingRoutingCommand

    1. Added global config - `vm.sync.power.state.transitioning`, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
    2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
    3. Optimized scanning stalled VMs

- Added option to set worker threads for capacity calculation using config - `capacity.calculate.workers`

- Added caching framework based on Caffeine in-memory caching library, https://github.com/ben-manes/caffeine

- Added caching for account/use role API access with expiration after write can be configured using config - `dynamic.apichecker.cache.period`. If set to zero then there will be no caching. Default is 0.

- Added caching for account/use role API access with expiration after write set to 60 seconds.

- Added caching for some recurring DB retrievals

    1. CapacityManager - listing service offerings - beneficial in host capacity calculation
    2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
    3. DownloadListener - hypervisors for zone - beneficial for host joins
    5. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands

- Optimized MS list retrieval for agent connect

- Optimize finding ready systemvm template for zone

- Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks

- Changes in agent-agentmanager connection with NIO client-server classes

    1. Optimized the use of the executor service
    2. Refactore Agent class to better handle connections.
    3. Do SSL handshakes within worker threads
    5. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - `agent.ssl.handshake.min.workers` and `agent.ssl.handshake.max.workers` can be used to control number of new connections management server handles at a time. `agent.ssl.handshake.timeout` can be used to set number of seconds after which SSL handshake times out at MS end.
    6. On agent side backoff and sslhandshake timeout can be controlled by agent properties. `backoff.seconds` and `ssl.handshake.timeout` properties can be used.

- Improvements in StatsCollection - minimize DB retrievals.

- Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.

- Improvements in hosts connection for a storage pool. Added config - `storage.pool.host.connect.workers` to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.

- Minor improvements in resource limit calculations wrt DB retrievals

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* test1, domaindetails, capacitymanager fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test2 - agent tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* capacitymanagertest fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert marvin/setup.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix indent

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* use space in sql

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address duplicate

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update host logs

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert e36c6a5d07

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix npe in capacity calculation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* move schema changes to 4.20.1 upgrade

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix build

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add some more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* checkstyle fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unnecessary mocks

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* replace statics

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* engine/orchestration,utils: limit number of concurrent new agent
connections

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor - remove unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unregister closed connections, monitor & cleanup

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add check for outdated vm filter in power sync

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* agent: synchronize sendRequest wait

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-01 12:28:41 +05:30
Suresh Kumar Anaparti 0d5047b8b7
Improve listing of HA and non-HA hosts when ha.tag setting is defined and hosts have multiple tags along with ha tag (#10240) 2025-01-31 14:07:00 +01:00
Abhishek Kumar 7abda3b963 Merge remote-tracking branch 'apache/4.20' 2025-01-31 16:55:38 +05:30
Abhishek Kumar ae2ffbe40b Merge remote-tracking branch 'apache/4.19' into 4.20 2025-01-31 16:54:22 +05:30
Wei Zhou fbb1ff78d6
Static Routes: fix check on wrong global configuration (#10066) 2025-01-31 11:04:13 +01:00
Fabricio Duarte 4201a2dc11
Fix Usage job not executing successfully (#10244)
* Add `api_key_access` to Usage account table

* use the procedure in cloud_usage instead
2025-01-30 22:07:50 +01:00
Bernardo De Marco Gonçalves a335feab6a
fix allocation of vmfs storage pools (#10201) 2025-01-30 18:44:48 +01:00
Bernardo De Marco Gonçalves d9af9bdb36
fix SQL syntax erros and target 419 (#10273) 2025-01-30 15:51:55 +01:00
Abhishek Kumar b93589b5bd
server: reset 2fa user configuration on incomplete setup (#10247)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-30 15:51:04 +01:00
Suresh Kumar Anaparti 3b108b968f
Support for Management Server Maintenance Mode (#9854)
* Support for Management Server Maintenance

- New APIs: prepareForMaintenance and cancelMaintenance, with required parameter - managementserverid.

- New management server states for maintenance: PreparingForMaintenance, Maintenance.

- listHosts API with optional parameter – managementserverid, to list the hosts connected to the management server.

- Support management server maintenance when more than one active management servers available.

- Triggers transfer agents to other available management servers for maintenance, new agent command MigrateAgentConnectionCommand to initiate transfer of indirect agents.

- New global config 'management.server.maintenance.timeout', to set the timeout (in mins) for the management server maintenance window, default: 60 mins.

- UI changes: Prepare and Cancel Maintenance in Management Server section, Connected Agents tab, New fields for hosts and management servers.

* Updated pending jobs check timer task with ScheduledExecutorService

* keep maintenance state on trigger shutdown call when ms is in maintenance

* add pending jobs count to ms response

* during ms heartbeat, update state to up only when it's down

* allow vm work jobs of async job created before prepare for maintenance

* Revert "keep maintenance state on trigger shutdown call when ms is in maintenance"

This reverts commit 607e13364679eac897f4d146bb3325ea7a61ba17.

* skip maintenance test when multiple management servers are not available, and not configured in host setting for kvm
2025-01-29 13:31:15 +05:30
Daan Hoogland 048649d351 Merge release branch 4.20 to main
* 4.20:
  server: investigate pending HA work when executing in new MS session (#10167)
  extra null guard (#10264)
2025-01-28 14:34:19 +01:00
Daan Hoogland 717ce981d4 Merge release branch 4.19 to 4.20
* 4.19:
  extra null guard (#10264)
2025-01-28 14:33:49 +01:00
Abhishek Kumar 33a37da9ec
server: investigate pending HA work when executing in new MS session (#10167)
For HA work items that are created for host state change, checks must be
done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track
the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA
work items in Investigating state. During execution of the HA work if it
is found in investigating state, checks are done to verify if the work
is still valid. If the jobs is found to be invalid it is cancelled.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-28 14:39:31 +05:30
dahn f652ad0d98
extra null guard (#10264) 2025-01-27 14:14:31 +01:00
Daan Hoogland 98f5663954 Merge branch '4.20' 2025-01-24 17:10:43 +01:00
Daan Hoogland 34d2a3bc86 Merge branch '4.19' into 4.20 2025-01-24 17:01:42 +01:00
dahn 0a77eb7f85
deal with NPE during host reconnect (#10158)
* log to see what command is being processed

* exception names
2025-01-24 15:39:56 +05:30
Daan Hoogland ecd60a4e46 Merge release branch 4.20 to main
* 4.20:
  Maintenance mode: Add host to deployment planner avoid list to fix local storage vm migration (#9892)
  Add project-user association normalization script to 4.20.1 upgrade (#10116)
  fix slider component for global settings of the range type (#10187)
  Clean up network permissions on account deletion (#10176)
2025-01-20 11:58:28 +01:00
Daan Hoogland 9f594c9699 Merge branch '4.19' into 4.20 2025-01-20 11:53:32 +01:00
nvazquez b310852a91
Merge branch 'main' into cks-enhancements-upstream 2025-01-17 12:07:41 -03:00
Fabricio Duarte 2300c1caa3
Add project-user association normalization script to 4.20.1 upgrade (#10116) 2025-01-16 16:18:30 +01:00
Fabricio Duarte 449d3c7cb1
Create API to list Quota credits (#9590)
Co-authored-by: Bernardo De Marco Gonçalves <bernardomg2004@gmail.com>
2025-01-16 15:19:32 +01:00
Bernardo De Marco Gonçalves 796bd4f72c
Clean up network permissions on account deletion (#10176) 2025-01-15 17:38:20 +01:00
Daan Hoogland bd874eaa44 Merge release branch 4.20 to main
* 4.20:
  systemvm: fix keystore is reset when patch a systemvm (#9900)
  no retrieval of null hosts (#10175)
  upgrade: consider multiple hypervisors and secondary storages (#10046)
  CheckOnHostCommand: add missing timeout setting (#9677)
2025-01-13 11:23:51 +01:00
Daan Hoogland e2cfddb1b6 Merge branch '4.19' into 4.20 2025-01-13 11:23:14 +01:00
Wei Zhou 7adc732992
upgrade: consider multiple hypervisors and secondary storages (#10046) 2025-01-13 09:28:18 +01:00
nvazquez 4b6e0ecf2b
Merge branch 'main' into cks-enhancements-upstream 2025-01-08 12:07:49 -03:00
Daan Hoogland fadb39ece7 Merge release branch 4.20 to main
* 4.20:
  merge errors fixed
  Restrict the migration of volumes attached to VMs in Starting state (#9725)
  server, plugin: enhance storage stats for IOPS (#10034)
  Introducing granular command timeouts global setting (#9659)
  Improve logging to include more identifiable information (#9873)
2025-01-08 14:01:19 +01:00
Abhishek Kumar bd488c4bba
server, plugin: enhance storage stats for IOPS (#10034)
Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned.
For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool.
For local storage pool, implementation has been added using iostat to return currently used IOPS.
StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-01-07 17:17:12 +05:30
Harikrishna 9bc283e5c2
Introducing granular command timeouts global setting (#9659)
* Introducing granular command timeouts global setting

* fix marvin tests

* Fixed log messages

* some more log message fix

* Fix empty value setting

* Converted the global setting to non-dynamic

* set wait on command only when granular wait is defined. This is to keep the backward compatibility

* Improve error logging
2025-01-07 17:06:32 +05:30
Vishesh a4224e58cc
Improve logging to include more identifiable information (#9873)
* Improve logging to include more identifiable information for kvm plugin

* Update logging for scaleio plugin

* Improve logging to include more identifiable information for default volume storage plugin

* Improve logging to include more identifiable information for agent managers

* Improve logging to include more identifiable information for Listeners

* Replace ids with objects or uuids


* Improve logging to include more identifiable information for engine

* Improve logging to include more identifiable information for server

* Fixups in engine

* Improve logging to include more identifiable information for plugins

* Improve logging to include more identifiable information for Cmd classes

* Fix toString method for StorageFilterTO.java
2025-01-06 16:42:37 +05:30
John Bampton 5bae1188ff
pre-commit fix mixed line endings in XML files (#10148)
https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#mixed-line-ending
2025-01-03 09:42:09 +01:00
Pearl Dsilva 2a1de76517
Add support for using different CNI plugins with CKS
* Add support for using different CNI plugins with CKS

* remove unused import

* Add UI support and list cni config API

* necessary UI changes

* add license

* changes to support external cni

* UI changes

* Fix NPE on restarting VPC with additional public IPs

* fix merge conflict

* add asnumber to create k8s svc layer

* support cni framework to use as-numbers

* update code

* condition to ignore undefined jinja template variables
2025-01-02 10:42:29 -03:00
nvazquez b0eb65f31a
Merge branch 'main' into cks-enhancements-upstream 2025-01-02 08:27:10 -03:00
Daan Hoogland 2daffa34f2 Merge release branch 4.20 to main
* 4.20:
  VR: fix site-2-site VPN if split connections is enabled (#10067)
  UI: fix cannot open 'Edit tags' modal for static routes (#10065)
  Update ownership selection component to be language independent (#10052)
  Support to enable/disable VM High Availability manager and related alerts (#10118)
2024-12-30 13:35:30 +01:00
Suresh Kumar Anaparti 330ed25a6c
Support to enable/disable VM High Availability manager and related alerts (#10118)
- Adds new config 'vm.ha.enabled'  with Zone scope, to enable/disable VM High Availability manager. This is enable by default (for backward compatibilty).
  When enabled, the VM HA WorkItems (for VM Stop, Restart, Migration, Destroy) can be created and the scheduled items are executed.
  When disabled, new VM HA WorkItems are not allowed and the scheduled items are retried until max retries configured at 'vm.ha.migration.max.retries' (executed in case HA is re-enabled during retry attempts), and then purged after 'time.between.failures' by the cleanup thread that runs regularly at 'time.between.cleanup'.
- Adds new config 'vm.ha.alerts.enabled' with Zone scope, to enable/disable alerts for the VM HA operations. This is enabled by default.
2024-12-26 17:45:32 +05:30
Daan Hoogland 9295a1624d Merge release branch 4.20 to main
* 4.20:
  VR: apply iptables rules when add/remove static routes (#10064)
  Certificate and VM hostname validation improvements (#10051)
  set ulimit for server according to redhat spec (#10040)
  kvm-storage: provide isVMMigrate information to storage plugins (#10093)
  Allow config drive deletion of migrated VM, on host maintenance (#10045)
  linstor: improve heartbeat check with also asking linstor (#10105)
  server: simplify role change validation (#9173)
  UI: create VPC network offering with conserve mode (#10082)
  server: fix typo removeaccessvpn in VirtualRouterElement (#10086)
  UI: remove duplicated Instance Name in Public IP details page (#10087)
  UI: Fixes in the Usage UI (#10000)
  SAML2: add cookie with HttpOnly too #10013 (#10047)
  ui: Allow font-awesome icon usage and optimise icon size inconsistency (#9744)
2024-12-20 14:37:49 +01:00
Daan Hoogland b7f0aac519 Merge branch '4.19' into 4.20 2024-12-20 14:34:39 +01:00
Suresh Kumar Anaparti b4ad04badf
Allow config drive deletion of migrated VM, on host maintenance (#10045) 2024-12-18 09:12:28 +01:00
Bernardo De Marco Gonçalves f75a194c09
Persist IP addresses related to VM access via CPVM (#9534) 2024-12-10 11:43:17 +01:00
Rohit Yadav cb530c1749 Merge remote-tracking branch 'origin/4.20' 2024-12-09 11:36:40 +05:30
Felipe 810c4108c7
engine/schema: Add access modifier to `VolumeVO` (#9394)
* added private access modifiers to VolumeVO and fixed an occurrence where a field was being accessed without a getter

* renamed the field hostip to hostIp and removed duplicated methods
https://github.com/apache/cloudstack/pull/9394#issuecomment-2264816580
2024-12-06 08:51:42 +01:00
Daan Hoogland 27d2de180f Merge release branch 4.20 to main
* 4.20:
  move upgrade code to the right file (#10036)
2024-12-04 14:31:20 +01:00
dahn 0eb94a1396
move upgrade code to the right file (#10036)
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-12-04 14:30:39 +01:00
Daan Hoogland 97f9e8e816 Merge release branch 4.20 to main
* 4.20:
  move sql code to the right file (#10035)
2024-12-04 14:29:42 +01:00
Daan Hoogland 8b9db0869f Merge release branch 4.19 to 4.20
* 4.19:
  move sql code to the right file (#10035)
2024-12-04 14:29:03 +01:00
dahn 762d5493dd
move sql code to the right file (#10035)
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-12-04 14:28:04 +01:00
Daan Hoogland 205ebfb8b5 Merge release branch 4.20 to main
* 4.20:
  UI: Fix userdata and load balancer selection (#10016)
  Prevent password updates for SAML and LDAP users (#9999)
  cloudstack-migrate-databases: sql AND added (#10033)
  engine/schema: move SQLs to 4.20.0 to 4.20.1 upgrade (#10018)
  Remove user from project before deletion (#10008)
  Simplify validation for creating volume templates via UI (#9828)
2024-12-04 13:05:32 +01:00
Daan Hoogland 5ee61c226a Merge branch '4.19' into 4.20 2024-12-04 13:04:18 +01:00
Wei Zhou b3dc402aa8
engine/schema: move SQLs to 4.20.0 to 4.20.1 upgrade (#10018) 2024-12-04 08:11:28 +01:00
Fabricio Duarte ef1a58d837
Remove user from project before deletion (#10008)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-12-04 08:06:22 +01:00
Daan Hoogland 9b6f9b5f7d Merge release branch 4.20 to main
* 4.20:
  UI: Tooltip on the host information card to display the CPU speed in MHz and the memory value in MB (to 3 decimal places) (#9971)
  UI: Allow accounts of the `User` type to add other accounts or users to projects through UI (#9927)
  enable to create VPC portfowarding rules with source cidr (#7081)
  Add new column `last_id` to the table volumes (#9759)
  Allow VMWare import via another host (#9787)
  Linstor: add support for ISO block devices and direct download (#9792)
  get expunged VM data for job result (#9949)
  fix section divider display on auth page (#9966)
2024-12-03 16:33:51 +01:00
Daan Hoogland da54234585 Merge branch '4.19' into 4.20.merge 2024-12-03 16:32:15 +01:00
Wei Zhou da94ae2c1c
Merge remote-tracking branch 'apache/4.20' 2024-12-03 09:44:42 +01:00
Abhisar Sinha d17de834a5
Disable API Key Access for users, accounts and domains (#9741)
* cli changes to update user/account, list by apikeyaccess, domain level setting

* UI changes for updating user/account and searchfilter in listview

* make the api parameters and setting accessible only to root admin

* revert changes to ui/package-lock.json

* minor changes to description strings

* UT for ApiServer and AccountManagerImpl classes

* fix pre-commit failure

* Added a constant for the string System

* UT for searchForUsers and searchForAccounts

* Fix marvin test error

* Update schema to use idempotent add column

* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>

* Added user name uuid to logging

* Add events when api key access is changed via api or config setting

* fix the userid for api key access update event

* Fix ut failure after event logging

* Convert drop down to radio-button in edit user and account

* Add ApiKeyAccess status in User InfoCard for Users if Api key is generated

* Return apiKeyAccess in user and account response only for Root Admin

* fixed noredist build failure

* Show apikeyaccess on the left panel in the user view for root admins as well

* don't show divider if apiKeyAccess is not shown to user

* Fix events generated to set Username, Account and Domain of the caller correctly

* cli changes to update user/account, list by apikeyaccess, domain level setting

* UI changes for updating user/account and searchfilter in listview

* make the api parameters and setting accessible only to root admin

* revert changes to ui/package-lock.json

* minor changes to description strings

* UT for ApiServer and AccountManagerImpl classes

* fix pre-commit failure

* Added a constant for the string System

* UT for searchForUsers and searchForAccounts

* Fix marvin test error

* Update schema to use idempotent add column

* Added user name uuid to logging

* Add events when api key access is changed via api or config setting

* fix the userid for api key access update event

* Fix ut failure after event logging

* Convert drop down to radio-button in edit user and account

* Add ApiKeyAccess status in User InfoCard for Users if Api key is generated

* Return apiKeyAccess in user and account response only for Root Admin

* fixed noredist build failure

* Show apikeyaccess on the left panel in the user view for root admins as well

* don't show divider if apiKeyAccess is not shown to user

* Fix events generated to set Username, Account and Domain of the caller correctly

* Added DB upgrade path from 42000 to 42010

---------

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
2024-12-03 12:10:54 +05:30
Abhisar Sinha ef6c0c443d
Prepend VPC name to VPC network tier name (#9780)
* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>

* Prepend vpc name to vpc tier network name based on global setting

* Added UT for createVpcGuestNetwork

* rename connector to delimiter and add configKey.Category.Network

* Move setting the name to a new method

---------

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
2024-12-03 12:06:00 +05:30
Wei Zhou 8a1da3804c
Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration (#9761)
* server: add global settings for volume resize

* resizeVolume: support automigrate

* Address Suresh's comments

* Update api/src/main/java/org/apache/cloudstack/api/command/user/volume/ResizeVolumeCmd.java

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>

* address Suresh's comments

* UI: add autoMigrate to resizeVolume

* resizevolume: add unit tests

* resizevolume: add unit test for Allocated volume

---------

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-12-02 10:28:14 +05:30
Wei Zhou 34056d956c
Improvement: management server peer states (#9885)
* Improvement: management server peer states

* Update pr9885: consider new mgmt server node which has msId=managementServerNodeId

* Update pr9885: update global config description

* Update pr9885: update label on UI

* framework: Do not update mshost_peer when mgmt server is Up as it will be updated by status update

* mgmt: Update state to Up when mgmt server writes heartbeat to db

* mgmt: change Service IP to Management IP

---------

Co-authored-by: Boris Stoyanov - a.k.a Bobby <bss.stoyanov@gmail.com>
2024-12-02 10:26:20 +05:30
Lucas Martins 5886780240
Change vmsnapshot.max config to be dynamic (#9883)
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
2024-11-28 14:49:05 -03:00
Lucas Martins dc0ebe985c
Remove info about the environment from exception when revert VM snapshot (#9944)
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
2024-11-28 14:42:34 -03:00
Rodrigo D. Lopez 4189bac8e0
enable to create VPC portfowarding rules with source cidr (#7081)
Co-authored-by: Lopez <rodrigo@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-11-28 17:53:07 +01:00
Felipe a8cb7abca3
Add new column `last_id` to the table volumes (#9759)
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-28 17:51:53 +01:00
João Jandre d9774a8462 Updating pom.xml version numbers for release 4.21.0.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-27 11:47:06 -03:00
João Jandre c63c7ee63e Updating pom.xml version numbers for release 4.20.1.0-SNAPSHOT
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-27 11:40:45 -03:00
João Jandre 2fe3fcef7c Updating pom.xml version numbers for release 4.20.0.0
Signed-off-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-19 08:54:07 -03:00
João Jandre b38ee63c48 Merge branch '4.19' 2024-11-13 10:47:24 -03:00
Wei Zhou adbf370909
Revert "storage: fix private templates are not copied to new image store (#9206)" (#9839)
This reverts commit e06f80e899.
2024-11-13 10:31:31 -03:00
Daan Hoogland f7b7339bcf Merge branch '4.19' 2024-11-07 11:33:50 +01:00
Abhisar Sinha 22c6f08f77
engine-storage: Set SecretConsumerDetail for VM live migration with storage on shared NFS (#9222)
* engine-storage-datamotion: Set SecretConsumerDetail for VM live migration with storage on shared NFS

* VM live migration - powerflex encrypted volume

* rename isPowerFlex
2024-11-07 14:45:04 +05:30
Wei Zhou 019f2c60ce
server: fix vm_stats records are not removed if batchsize is 0 (#9818) 2024-10-18 14:52:46 +05:30
Wei Zhou f10d9fecbe
fix forward-merge errors 2024-10-15 09:21:33 +02:00
João Jandre 554ea220f0 Fix merge errors 2024-10-11 14:12:18 -03:00
Daan Hoogland dbfc7f23a7 Merge branch '4.19' 2024-10-11 17:59:46 +02:00
nvazquez 87a528d2b2
Merge branch 'main' into cks-enhancements-upstream 2024-10-03 00:00:10 -03:00