Commit Graph

11935 Commits

Author SHA1 Message Date
Vishesh 3923f80c22
Merge branch '4.19' 2024-06-25 18:53:57 +05:30
Vishesh f6ceeab3b3
server: Enforce strict host tag check (#9017)
Documentation PR: https://github.com/apache/cloudstack-documentation/pull/398

Currently, an administrator can break host tag compatibility for a VM administrator by certain operations:
* deploy/start VM on a specific host
* migrate VM
* restore VM
* scale VM

This PR allows the user to specify tags which must be checked during these operations.

Global Settings
1. `vm.strict.host.tags` - A comma-separated list of tags for strict host check (Default - empty)
2. `vm.strict.resource.limit.host.tag.check` - Determines whether the resource limits tags are considered strict or not (Default - true)

During the above operations, we now check and throw an error if host tags compatibility is being broken for tags specified in `vm.strict.host.tags`. If `vm.strict.resource.limit.host.tag.check` is set to `true`, tags set in `resource.limit.host.tags` are also checked during these operations.
2024-06-25 14:42:17 +05:30
Rene Glover 6ee6603359
Updates to HPE-Primera and Pure FlashArray Drivers to use Host-based VLUN Assignments (#8889)
* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* Updates to change PUre and Primera to host-centric vlun assignments; various small bug fixes

* update to add timestamp when deleting pure volumes to avoid future conflicts

* update to migrate to properly check disk offering is valid for the target storage pool

* improve error handling when copying volumes to add precision to which step failed

* rename pure volume before delete to avoid conflicts if the same name is used before its expunged on the array

* remove dead code in AdaptiveDataStoreLifeCycleImpl.java

* Fix issues found in PR checks

* fix session refresh TTL logic

* updates from PR comments

* logic to delete by path ONLY on supported OUI

* fix to StorageSystemDataMotionStrategy compile error

* change noisy debug message to trace message

* fix double callback call in handleVolumeMigrationFromNonManagedStorageToManagedStorage

* fix for flash array delete error

* fix typo in StorageSystemDataMotionStrategy

* change copyVolume to use writeback to speed up copy ops

* remove returning PrimaryStorageDownloadAnswer when connectPhysicalDisk returns false during KVMStorageProcessor template copy

* remove change to only set UUID on snapshot if it is a vmSnapshot

* reverting change to UserVmManagerImpl.configureCustomRootDiskSize

* add error checking/simplification per comments from @slavkap

* Update engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* address PR comments from @sureshanaparti

---------

Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-25 10:35:39 +05:30
Rohit Yadav 42e71175d7 Merge remote-tracking branch 'origin/4.18' into 4.19 2024-06-24 20:39:55 +05:30
Abhishek Kumar de266a4cb8
api,server: allow updating hypervisor capabilities with hypervisor and version (#8475)
* api,server: allow updating hypervisor capabilities with hypervisor and
version

hypervisor and hypervisorversion parameter added to the
updateHypervisorCapabilities API.

* param description

* Update server/src/main/java/com/cloud/server/ManagementServerImpl.java

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Henrique Sato <henriquesato2003@gmail.com>
Co-authored-by: Vishesh <vishesh92@gmail.com>
2024-06-24 14:41:01 +05:30
Wei Zhou fcfa371337
kvm: support Rocky/RHEL/OL/Alma in the same cluster (#8641)
* kvm: support Rocky/RHEL/OL/Alma in the same cluster

* Update PR#8641: add unit tests
2024-06-24 13:10:41 +05:30
Suresh Kumar Anaparti c17aa0d9ad
Import Remote KVM VM logging improvements (#9284) 2024-06-24 11:34:37 +05:30
dahn 7e00678691
set isSystem for SVM IPs (#9281)
* set isSystem for SSVM IPs

* Revert "set isSystem for SSVM IPs"

This reverts commit 4ba71b3d6b4ea377f6778d166ff9da7d41e5d007.

* set isSystem flag for SSVMs
2024-06-23 19:36:45 +05:30
Vishesh 59e9ab9efe
Fix volume response for service offering with disk offering (#9273) 2024-06-22 11:17:11 +05:30
Wei Zhou 313a165e62
server: add global setting consoleproxy.sslEnabled (#8809) 2024-06-21 15:56:20 +05:30
dahn 60f234c682
remove Project Template Permissions inhibition (#9196) 2024-06-21 15:20:49 +05:30
Gabriel Pordeus Santos 083ac069ca
fix assignvm template permission check (#8886) 2024-06-21 10:58:46 +02:00
Suresh Kumar Anaparti 5ab23cd9c9
Timeout config to copy the disks of remote KVM instance while importing the instance from an external host (#9213)
* Added timeout config to copy the disks of remote KVM instance while importing the instance from an external host

* Updated copy config units to mins

* Cleanup remote converted file and local file when copy failed
2024-06-21 10:28:18 +05:30
João Jandre ae3fa5d0de
Add configuration to limit the number of rows deleted from vm_stats (#8740)
Co-authored-by: João Jandre <joao@scclouds.com.br>
2024-06-20 14:26:36 +02:00
Wei Zhou d81ffd2d86
Merge remote-tracking branch 'apache/4.19' 2024-06-20 08:57:17 +02:00
Abhishek Kumar 3e6900ac1a
api,server: purge expunged resources (#8999)
This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine). There would be three mechanisms for purging removed resources:

    Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
    API - New admin-only API purgeExpungedResources. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate. Currently, API is not supported in the UI.
    Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.

Following new global settings have been added:

    expunged.resources.purge.enabled: Default: false. Whether to run a background task to purge the expunged resources
    expunged.resources.purge.resources: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the expunged resources. Currently only VirtualMachine is supported. An empty "value will result in considering all resource types for purging
    expunged.resources.purge.interval: Default: 86400. Interval (in seconds) for the background task to purge the expunged resources
    expunged.resources.purge.delay: Default: 300. Initial delay (in seconds) to start the background task to purge the expunged resources task.
    expunged.resources.purge.batch.size: Default: 50. Batch size to be used during expunged resources purging.
    expunged.resources.purge.start.time: Default: (empty). Start time to be used by the background task to purge the expunged resources. Use format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.
    expunged.resources.purge.keep.past.days: Default: 30. The number of days in the past from the execution time of the background task to purge the expunged resources for which the expunged resources must not be purged. To enable purging expunged resource till the execution of the background task, set the value to zero.
    expunged.resource.purge.job.delay: Default: 180. Delay (in seconds) to execute the purging of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.

Documentation PR: apache/cloudstack-documentation#397

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-20 11:34:44 +05:30
Wei Zhou 81269ad62a
VNF: add API listVnfAppliances (#8994) 2024-06-19 19:49:21 +05:30
Harikrishna 2315a73a20
User friendly name of Downloaded Templates Volumes and ISOs (#9252) 2024-06-19 12:47:43 +02:00
Suresh Kumar Anaparti 8b02624e61
User data content size validation, and related code improvements (#8418)
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-06-18 20:07:13 +02:00
Daan Hoogland 373f017002 Merge branch '4.19' 2024-06-18 19:58:43 +02:00
Daan Hoogland 050ee44137 Merge branch '4.18' into 4.19 2024-06-18 16:05:45 +02:00
Vishesh 6c7426e3a7
Prevent network disruption on adding a VPC tier for redundant VRs (#9251) 2024-06-18 13:47:45 +02:00
dahn 7c5b7ca077
Extra parameter for UpdateImageStore (#8941)
* Extra parameter for UpdateImageStore

* add name parameter

* ui

* cleanup

* update DB from storage stats results
2024-06-18 12:31:17 +05:30
Harikrishna bb0c1f93af
Add volume encryption checks during the disk offering change (#9209) 2024-06-17 10:36:47 +02:00
Wei Zhou aab1361949
server: fix mgmt server 503 error by renaming kubernetesClusterHelper to kubernetesServiceHelper 2024-06-15 08:35:36 +02:00
Bryan Lima 00fe25ab01
Fix allocation of VMs with multiple clusters (#8611)
* Fix allocation of VMs with multiple clusters

* Readd debug guard
2024-06-14 13:54:01 +03:00
dahn ed86dc973b
protect against missing service offering (#9235)
* protect agains missing service offering

* search removed before assuming none

* import

* javadoc
2024-06-14 15:25:46 +05:30
Daan Hoogland cb9b3134f7 Merge branch '4.19' 2024-06-14 10:30:10 +02:00
Abhishek Kumar ce9b2c52f3
cks: fix events (#9070)
Fixes #8043

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-14 12:22:39 +05:30
Abhishek Kumar df5c546559
server: remove username required condition for listVmsForImport (#9124)
* server: remove username required condition for listVmsForImport

Fixes #8414

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-14 12:14:59 +05:30
Rohit Yadav 2ca0857bd5
api: listVM API improvement followup, change returning of stats detail (#9177)
- Changes behaviour of details param handling via global setting:
  - listVirtualMachines API: when the details param is not provided, it returns whether stats are returned controlled by a new global setting `list.vm.default.details.stats`
  - listVirtualMachinesMetrics API: when the details param is not provided, it uses `all` details including `stats`
- Users who are affected slow performance of the listVirtualMachines API response time can set `list.vm.default.details.stats` to `false`
- Remove ConfigKey vm.stats.increment.metrics.in.memory which was renamed to `vm.stats.increment.metrics` in #5984 and also remove unused/unnecessary global settings via upgrade path
- Changes default value of VM stats accumulation setting `vm.stats.increment.metrics` to false until a better solution emerges. Since #5984, this is true and during the execution of listVM APIs the stats are clubbed/calculated which can immensely slow down list VM API calls. Any costly operations such as summing of stats shouldn't be done during the course of a synchronous API, such as the list VM API.
- Fix UI that uses listVirtualMachinesMetrics to not call `stats` detail when in list view without metrics selected.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-06-14 11:03:14 +05:30
Harikrishna 517cddcb15
Fix error message if specific host does not have capacity (#9218) 2024-06-13 12:38:51 +05:30
Vishesh 6fe835e118
Merge branch '4.19' 2024-06-13 11:57:55 +05:30
Suresh Kumar Anaparti 6fda757936
While starting VM with 'considerlasthost' enabled, don't load host tags/details for the last host when it doesn't exist [main] (#9063) 2024-06-12 17:03:18 +02:00
Vishesh faf94387a6
Disable logs to stdout and capture remaining stdout logs to a file (#8880)
* Log stdout to a file

* Add logrotation
2024-06-12 13:46:31 +03:00
Vishesh c756e4370a
List volumes by service offering id (#9211)
* Allow listing of volumes by service offering id

* Address comments
2024-06-12 16:13:56 +05:30
dahn 4de975ff25
list by isEncrypted (#8643)
* list by isEncrypted

* use filter on VO and cleanup

* add encryption type to volume response

* Update api/src/main/java/org/apache/cloudstack/api/command/user/volume/ListVolumesCmd.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-12 15:11:00 +05:30
Suresh Kumar Anaparti 503ae64a3d
Upload volume format check with the supported hypervisor, fail if supported hypervisor not found on zone (#9224) 2024-06-12 11:40:29 +02:00
Pearl Dsilva 0f770194df
Configure config drive ISO with password file on reinstallation of VM with password-enabled template (#9113)
* Configure config drive ISO with password file on reinstallation of VM with password-enabled template

* address comment
2024-06-12 12:00:08 +03:00
Abhishek Kumar 1577218999
server,event: fix events for storage maintenance (#9056)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-12 09:43:13 +02:00
Abhishek Kumar ca59720015
vpc,event: fix events for createVpc (#9055)
Fixes #8496

* vpc,event: fix events for createVpc
2024-06-12 12:56:08 +05:30
Suresh Kumar Anaparti 4e7c6682fd
While starting VM with considerlasthost enabled, don't load host tags/details for the last host when it doesn't exist (#9037) 2024-06-12 07:49:03 +02:00
Wei Zhou edf7394919
server: search template name and display text if keyword is passed (#9215) 2024-06-11 19:35:43 +05:30
Suresh Kumar Anaparti 2e3f76ec03
Improve error messaging / logs when listing VMs on the remote KVM host (for import) (#9204) 2024-06-11 14:48:21 +02:00
Harikrishna 6edcf32c5e
Fix vsphere storage policy when compute offering uses the disk offering attached to it (#9203) 2024-06-11 14:49:58 +03:00
Bernardo De Marco Gonçalves 1383625c93
Fix `*.smtp.useAuth`, `quota.usage.smtp.useStartTLS` and `*.smtp.enabledSecurityProtocols` settings definitions (#9031)
* change configs definitions

* add normalization query

* add ui support

* add labels

* add end of line to SQL script
2024-06-11 10:54:03 +03:00
Abhishek Kumar fcca3e8f39
api,server: encryptformat in volume response (#9106)
Fixes #9014

* api,server: encryptformat in volume response

* Apply suggestions from code review

* remove passphrase db change
2024-06-11 11:20:50 +05:30
Vishesh cc8dc84f64
server: fix resource reservation leakage (#9169)
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-10 12:30:06 +05:30
Abhishek Kumar be552fdce9
feature: webhooks (#8674)
* api,server,ui: weebhoks feature

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* registry of message busses

* test bus

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix and refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes for webhook dispatch history

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes, initial ui

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* improvements

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes for account webhook cleanup

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix remaining event bus usage

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes for testing webhook dispatch

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* make element

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* missing

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* buid fix

* fix lint

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes for project delete check

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add collapse in create

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui fix and refactor for eventditributor publish

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update org.json and add json validation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* schema fixes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wordings

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: improve progress button

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui improvements

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unrelated change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* search and count

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add payloadurl in info

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* positive progress

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix hmac key

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* create webhook form fixes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor, address feedback

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* indentation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix filters

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove test eventbus

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* default scope be Local

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add lifecycle smoke test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add test for webhook deliveries

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix lint

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor - losgs and others

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unit tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix lint

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* smoke test fix, log refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* get bean from all components

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: missing label

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address review comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add some more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* lint

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* rename setting

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* upgrade: move 4.19.0->4.20.0 to 4.19.1->4.20.0

* fix test delivery layout

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix webhook secret display

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add http to payloadurl when no scheme

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* allow removing secretkey for webhook

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix update sslverification

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* disallow same payload url for same account

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix delivery with url w/o scheme

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* api: listApis should return params based on caller

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wip changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update engine/schema/src/main/resources/META-INF/db/schema-41900to42000.sql

* remove unique constraint for now

Constraint is present in Java code validations

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fixes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: add option to delete multiple deliveries

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add filter for deliveries, delete api start/endtime support

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* do not throw error when no deliveries removed

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: fix deliveries table column sorting, time filter cancel

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove isDebugEnabled wrapping

* merge fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Wei Zhou <weizhou@apache.org>
2024-06-10 10:40:12 +05:30
Daan Hoogland b29391c329 Merge LTS branch '4.19' into main 2024-06-07 09:53:57 +02:00