Commit Graph

384 Commits

Author SHA1 Message Date
Daan Hoogland 75db42bca6 Merge branch '4.20' into 4.22 2026-01-27 15:23:23 +01:00
Wei Zhou 63c8b5fc56
api/server: support deploy-as-is template as VNF template (#12499) 2026-01-26 15:23:27 +01:00
Daan Hoogland edb62b490c Merge branch '4.20' into 4.22 2026-01-26 13:30:26 +01:00
Suresh Kumar Anaparti c8cadcb56e
NPE fix while deleting storage pool when pool has detached volumes (#12451)
* NPE fix while deleting storage pool when pool has detached volumes

* review

* unit tests

* Added log for volumes not attached to any VMs

* update filter, log and test

* updated volume dao method names returning non destroyed volumes

* build fix

---------

Co-authored-by: dahn <daan@onecht.net>
2026-01-26 10:31:14 +02:00
Nicolas Vazquez 04875f1517
Improve logs for VM migrations (#12332) 2026-01-09 17:50:27 +01:00
Daan Hoogland 55ab7c5589 Merge branch '4.20' into 4.22 2025-12-22 13:23:37 +01:00
vladimirpetrov b394b5ba74
Fix terms, typos and grammar mistakes in the API, error messages, events, etc. (#7857)
This PR aligns the use of terminology, renaming VM / virtual machine references to 'Instance' and also capitalising the terms Templates, Network, Snapshot, User, Account in CloudStack APIs, error and log messages, events, tooltips, etc. Many typos, grammar and spelling mistakes were fixed, also terms like IPv4, VPN, VPC, etc. were properly capitalised. Some error messages were cleaned for better readability. The test cases, expecting some exception strings were adjusted accordingly.

Here is the wiki page, describing the changes in details:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Object+Naming+and+Title+Case+Convention

---------

Co-authored-by: Manoj Kumar <manojkr.itbhu@gmail.com>
Co-authored-by: Harikrishna <harikrishna.patnala@gmail.com>
2025-12-22 15:18:58 +05:30
Abhisar Sinha 671d8ad704
Track volume usage data at a vm granularity as well (#11531)
Co-authored-by: Vishesh <8760112+vishesh92@users.noreply.github.com>
2025-11-12 09:32:01 +01:00
Abhisar Sinha 046014b4c5
NAS BnR: Create Instance from Backup issues (#11754)
* add createCrossZoneInstnaceEnabled to BackupOfferingResponse

* show use IP Address from Backup button when orignal instance is expunged

* Fix NPE in takeBackup if the  vm template is deleted.

* Add since to Cross zone instance creation in BackupOfferingResponse.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Store and show Guest os type in the backup metadata

* show warning in create instance from backup form if guest os type is different

* show warning in create instance from backup form if guest os type is different

* backupvmexpunged -> isbackupvmexpunged

* review comments

* fix npe

* improve err msg

* err msg

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-10-14 14:51:57 +05:30
Vishesh 0b9afe77ca
Enforce distinct hostnames network (#10212)
* Check for unique hostnames for all networks in the vpc

* Address comments
2025-10-13 12:38:31 +05:30
Pearl Dsilva 973819dad6
API: Add support to list all snapshot policies & backup schedules (#11587)
* API: Add support to list all snapshot policies & backup schedules

* Add support for backup policy listing without tying it to the vmid

* add tests for snapshot policy listing

* update tests for listbackupschedules

* remove trailing spaces and fix lint failure

* Add upgrade test

* remove unused import

* add create policy - snap/backup in the list view with resource (volume/vm) selection

* add translations

* refresh parent list

* remove unnecessary alert info

* fix checks for UI backup schedule list view

* fix checks for UI backup schedule list view

* add back access checks

* add since param

* fix failing test

* update snapshot policy and backup schedule ownership when VM is moved

* fix issue with showing vm selection

* fix unit test failure

* Update list snappolicy & backup schedule logic to list only those that belong to a proj or for root admin those that belong to it, unless listall & projid is passed

* fix test

* support snap / backup policy search using keyword

* fix tests
2025-10-09 17:22:17 +05:30
Daan Hoogland 5f63e8493c Merge branch '4.20' 2025-10-07 15:12:53 +02:00
Manoj Kumar 9bcd98876d
Make kvm domain persistent when unmanaged from CS (#11541)
CS creates transient KVM domain.xml. When instance is unmanaged from CS, explicit dump of domain has to be taken to manage is outside of CS.

With this PR

    domainXML gets backed up and becomes persistent for further management of Instance.
    Stopped instance also can be unmanaged, last host for instance is considered for defining domain
    hostid param is supported in unmanageVirtualMachine API for KVM hypervisor and for stopped Instances
    hostid field in response of unmanageVirtualMachine, representing host used for unmanage operation
    Disable unmanaging instance with config drive, can unmanage from API using forced=true param for KVM
2025-10-07 10:32:33 +05:30
Alexandru Bagu 8e4dc0a66d
VMware: match nic mac for ip address fetch (#10641) 2025-10-04 11:49:26 +02:00
Abhisar Sinha 70af55e848
UI support for extraconfig in deploy and update instance (#11719) 2025-09-30 09:20:44 +02:00
Abhisar Sinha 23c9e83047
Create Instance from backup on another Zone (DRaaS use case) (#11560)
* draas initial changes

* Added option to enable disaster recovery on a backup respository. Added UpdateBackupRepositoryCmd api.

* Added timeout for mount operation in backup restore configurable via global setting

* Addressed review comments

* fix for simulator test failures

* Added UT for coverage

* Fix create instance from backup ui for other providers

* Added events to add/update backup repository

* Fix race in fetchZones

* One more fix in fetchZones in DeployVMFromBackup.vue

* Fix zone selection in createNetwork via Create Instance from backup form.

* Allow template/iso selection in create instance from backup ui

* rename draasenabled to crosszoneinstancecreation

* Added Cross-zone instance creation in test_backup_recovery_nas.py

* Added UT in BackupManagerTest and UserVmManagerImplTest

* Integration test added for Cross-zone instance creation in test_backup_recovery_nas.py
2025-09-25 13:28:29 +05:30
Wei Zhou e78b5cc3cc
Merge remote-tracking branch 'apache/4.20' 2025-09-24 09:27:08 +02:00
Nicolas Vazquez 1a223fd2ba
server: Fix VM import DB sequence issue on import failure (#11659)
* Fix VM import DB sequence issue on import failure

* Remove ununsed imports

* Refactor to avoid duplicating the next ID for VM sequence
2025-09-23 08:48:18 +02:00
vishesh92 ada750e391
Merge branch '4.20' 2025-09-17 14:26:06 +05:30
Suresh Kumar Anaparti cd69f2ce16
server: Fix NPE during VM IP fetch for shared networks (#11389)
* Fix NPE during VM IP fetch for shared networks

* PR 11389: add missing import org.apache.commons.lang3.ObjectUtils

---------

Co-authored-by: Wei Zhou <weizhou@apache.org>
2025-09-15 10:14:06 +02:00
Suresh Kumar Anaparti 6d16ac2113
ScaleIO/PowerFlex smoke tests improvements, and some fixes (#11554)
* ScaleIO/PowerFlex smoke tests improvements, and some fixes

* Fix test_volumes.py, encrypted volume size check (for powerflex volumes)

* Fix test_over_provisioning.py (over provisioning supported for powerflex)

* Update vm snapshot tests

* Update volume size delta in primary storage resource count for user vm volumes only
The VR volumes resource count for PowerFlex volumes is updated here, resulting in resource count discrepancy
(which is re-calculated through ResourceCountCheckTask later, and skips the VR volumes)

* Fix test_import_unmanage_volumes.py (unsupported for powerflex)

* Fix test_sharedfs_lifecycle.py (volume size check for powerflex)

* Update powerflex.connect.on.demand config default to true
2025-09-12 16:17:20 +02:00
Wei Zhou 70a4503ea1
Merge remote-tracking branch 'apache/4.20' 2025-09-11 14:04:52 +02:00
Abhishek Kumar e3681a04bb
server: allow migration of vm with snapshots for vmware (#9305)
Fixes #9061

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-09-10 12:25:16 +02:00
slavkap dc5e475bd2
Fix of deployment VM from a copied snapshot in another zone (#11351)
* Fix of deploy VM with a snapshot that is copied to another zone
* Fix of creating StorPool volume from a snapshot if the size in the
offering is bigger than the snapshot size
2025-08-07 18:17:57 +05:30
Abhisar Sinha a87c5c2b3a
Create new Instance from VM backup (#10140)
This feature adds the ability to create a new instance from a VM backup for dummy, NAS and Veeam backup providers. It works even if the original instance used to create the backup was expunged or unmanaged. There are two parts to this functionality:
Saving all configuration details that the VM had at the time of taking the backup. And using them to create an instance from backup.
Enabling a user to expunge/unmanage an instance that has backups.
2025-07-31 15:47:22 +05:30
Vishesh f6ad184ea2
Feature: Add support for GPU with KVM hosts (#11143)
This PR allows attaching of GPU devices via PCI, mdev or VF to an Instance for KVM.

It allows the operator to discover the GPU devices on the KVM host and create a Compute Offering with GPU support based on the available GPU devices on the host. Once the operator has created the Compute offering, it can be used by users to launch Instances with GPU devices.
2025-07-29 13:46:24 +05:30
Harikrishna cca8b2fef9
Extensions Framework & Orchestrate Anything (#9752)
The Extensions Framework in Apache CloudStack is designed to provide a flexible and standardised mechanism for integrating external systems and custom workflows into CloudStack’s orchestration process. By defining structured hook points during key operations—such as virtual machine deployment, resource preparation, and lifecycle events—the framework allows administrators and developers to extend CloudStack’s behaviour without modifying its core codebase.
2025-07-28 10:41:17 +05:30
Abhishek Kumar 407072ffd4
schema,framework/db,server: fix user_vm_details usage (#11292)
* schema,framework/db,server: fix user_vm_details usage

Continuation of #10736

* python files

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2025-07-26 12:24:05 +05:30
Abhishek Kumar 83bccead3d
schema, refactor: rename cloud.user_vm_details to cloud.vm_instance_details (#10736)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2025-07-24 12:08:29 +02:00
slavkap 54b44cc316
KVM: Option to deploy a VM with existing volume/snapshot (#10503)
* Option to deploy a VM with existing volume/snapshot

* smoke test changes

check if the hypervisor is KVM
check if the primary storage's scope is ZONE wide

* skip all tests if the storage isn't Zone-Wide and the hypervisor isn't KVM

* support StorPool tags

add StorPool tags to a volume created from snapshot or to a volume which
will be attached as a ROOT to a new VM

* Add StorPool tags on the new ROOT volume

* Add the StorPool's tags when volume is created from a snapshot or a
volume is attached as a ROOT to a VM

* Addressed review
2025-07-14 15:10:45 +05:30
Nicolas Vazquez 6adfda2818
CKS Enhancements (#9102)
CKS Enhancements:

* Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd

* Ability to use CKS ready custom templates for CKS cluster nodes

* Add and Remove external nodes to and from a kubernetes cluster

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Update remove node timeout global setting

* CKS/NSX : Missing variables in worker nodes

* CKS: Fix ISO attach logic

* CKS: Fix ISO attach logic

* address comment

* Fix Port - Node mapping when cluster is scaled in the presence of external node(s)

* CKS: Externalize control and worker node setup wait time and installation attempts

* Fix logger

* Add missing headers and fix end of line on files

* CKS Mark Nodes for Manual Upgrade and Filter Nodes to add to CKS cluster from the same network

* Add support to deploy CKS cluster nodes on hosts dedicated to a domain

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* Support unstacked ETCD

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix CKS cluster scaling and minor UI improvement

* Reuse k8s cluster public IP for etcd nodes and rename etcd nodes

* Fix DNS resolver issue

* Update UDP active monitor to ICMP

* Add hypervisor type to CKS cluster creation to fix CKS cluster creation when External hosts added

* Fix build

* Fix logger

* Modify hypervisor param description in the create CKS cluster API

* CKS delete fails when external nodes are present

* CKS delete fails when external nodes are present

* address comment

* Improve network rules cleanup on failure adding external nodes to CKS cluster

* UI: Fix etcd template was not honoured

* UI: Fix etcd template was not honoured

* Refactor

* CKS: Exclude etcd nodes when calculating port numbers

* Fix network cleanup in case of CKS cluster failure

* Externalize retries and inverval for NSX segment deletion

* Fix CKS scaling when external node(s) present in the cluster

* CKS: Fix port numbers displayed against ETCD nodes

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* update column name

* CKS: Exclude etcd nodes when calculating port numbers

* update param name

* update param

* UI: Fix CKS cluster creation templates listing for non admins

* CKS: Prevent etcd node start port number to coincide with k8s cluster start port numbers

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* consolidate query

* Fix upgrade logic

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix CKS cluster version upgrade

* CKS: Fix etcd port numbers being skipped

* Fix CKS cluster with etcd nodes on VPC

* Move schema and upgrade for 4.20

* Fix logger

* Fix after rebasing

* Add support for using different CNI plugins with CKS

* Add support for using different CNI plugins with CKS

* remove unused import

* Add UI support and list cni config API

* necessary UI changes

* add license

* changes to support external cni

* UI changes

* Fix NPE on restarting VPC with additional public IPs

* fix merge conflict

* add asnumber to create k8s svc layer

* support cni framework to use as-numbers

* update code

* condition to ignore undefined jinja template variables

* CKS: Do not pass AS number when network ID is passed

* Fix deletion of Userdata / CNI Configuration in projects

* CKS: Add CNI configuration details to the response and UI

* Explicit events for registering cni configuration

* Add Delete cni configuration API

* Fix CKS deployment when using VPC tiers with custom ACLs

* Fix DNS list on VR

* CKS: Use Network offering of the network passed during CKS cluster creation to get the AS number

* CKS cluster with guest IP

* Fix: Use control node guest IP as join IP for external nodes addition

* Fix DNS resolver issue

* Improve etcd indexing - start from 1

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* simplify logic

* Tweak setup-kube-system script for baremetal external nodes

* Consider cordoned nodes while getting ready nodes

* Fix CKS cluster scale calculations

* Set token TTL to 0 (no expire) for external etcd

* Fix missing quotes

* Fix build

* Revert PR 9133

* Add calico commands for ens35 interface

* Address review comments: plan CKS cluster deployment based on the node type

* Add qemu-guest-agent dependency for kvm based templates

* Add marvin test for CKS clusters with different offerings per node type

* Remove test tag

* Add marvin test and fix update template for cks and since annotations

* Fix marvin test for adding and removing external nodes

* Fix since version on API params

* Address review comments

* Fix unit test

* Address review comments

* UI: Make CKS public templates visible to non-admins on CKS cluster creation

* Fix linter

* Fix merge error

* Fix positional parameters on the create kubernetes ISO script and make the ETCD version optional

* fix etcd port displayed

* Further improvements to CKS  (#118)

* Multiple nics support on Ubuntu template

* Multiple nics support on Ubuntu template

* supports allocating IP to the nic when VM is added to another network - no delay

* Add option to select DNS or VR IP as resolver on VPC creation

* Add API param and UI to select option

* Add column on vpc and pass the value on the databags for CsDhcp.py to fix accordingly

* Externalize the CKS Configuration, so that end users can tweak the configuration before deploying the cluster

* Add new directory to c8 packaging for CKS config

* Remove k8s configuration from resources and make it configurable

* Revert "Remove k8s configuration from resources and make it configurable"

This reverts commit d5997033ebe4ba559e6478a64578b894f8e7d3db.

* copy conf to mgmt server and consume them from there

* Remove node from cluster

* Add missing /opt/bin directory requrired by external nodes

* Login to a specific Project view

* add indents

* Fix CKS HA clusters

* Fix build

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>

* Add missing headers

* Fix linter

* Address more review comments

* Fix unit test

* Fix scaling case for the same offering

* Revert "Login to a specific Project view"

This reverts commit 95e37563f4.

* Revert "Fix CKS HA clusters" (#120)

This reverts commit 8dac16aa35.

* Apply suggestions from code review about user data

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/userdata/BaseRegisterUserDataCmd.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Refactor column names and schema path

* Fix scaling for non existing previous offering per node type

* Update node offering entry if there was an existing offering but a global service offering has been provided on scale

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-06-19 11:00:42 +05:30
Manoj Kumar 7632814cd2
Instance lease: Allow deployment of instances with lease duration and leaseexpiry action (#10560)
* FR-248: Instance lease, WIP commit

* insert lease expiry into db and use that to filter exiring vms, add asyncjobmanager

* Add leaseDuration and leaseExpiryAction in Service offering create flow

* Update listVM cmd to allow listing only leased instances

* Add methods to fetch instances for which lease is expiring in next days

* Changes included:
config key setup and configured for alert email
lease options in create and update vm screen
handle delete protection, edit vm, create vm
validated stop and detroy, delete protection

* Update UI screens for leased properties coming from config and service offering

* use global lock before running scheduler

* Unit tests

* Flow changes done in UI based on discussion

* Include view changes in schema upgrade files and use feature in various UI elements

* Added integration test for vm deployment, UI enhancements for user persona, bug fixes

* validate integration tests, minor ui changes and log messages

* fix build: moving configkey from setup to test itself

* Disable testAlert to unblock build and trim whitespaces in integration tests

* Address review comments

* Minor changes in EditVM screen

* Use ExecutorService instead of Timer and TimerTask

* Additional review comments

* Incorporate following changes:
1. Execute lease action once on the instance
2. Cancel lease on instance when feature is disabled
3. Relevant events when lease gets disabled, cancelled, executed
4. Disable associating lease after deployment
5. UI elements and flow changes
6. Changes based on feedback from demo

* Handle pr review comments

* address review comments

* move instance.lease.enabled config to VMLeaseManager interface

* bug fix in edit instance flow and reject api request for invalid values

* max allowed lease is for 100 years

* log instance ids for expired instance

* Fix config validation for value range and code coverage improvement

* fix lease expiry request failures in async

* dont use forced: true for StopVmCmd

* Update server/src/main/java/org/apache/cloudstack/vm/lease/VMLeaseManager.java

Co-authored-by: Vishesh <vishesh92@gmail.com>

* handle review comments

---------

Co-authored-by: Rohit Yadav <rohityadav89@gmail.com>
Co-authored-by: Vishesh <vishesh92@gmail.com>
2025-05-28 17:40:09 +05:30
Pearl Dsilva f496ed6eaf Merge branch '4.20' of https://github.com/apache/cloudstack 2025-05-22 12:05:47 +05:30
Fabricio Duarte 34896a1ada
Address `assignVm` regression (#10875)
* Address assignVm regression
2025-05-21 09:21:41 +05:30
Wei Zhou 842b2f8c24
Merge remote-tracking branch 'apache/4.20' 2025-05-19 21:25:37 +02:00
Harikrishna b17808bfba
Introducing Storage Access Groups for better management for host and storage connections (#10381)
* Introducing Storage Access Groups to define the host and storage pool connections

In CloudStack, when a primary storage is added at the Zone or Cluster scope, it is by default connected to all hosts within that scope. This default behavior can be refined using storage access groups, which allow operators to control and limit which hosts can access specific storage pools.

Storage access groups can be assigned to hosts, clusters, pods, zones, and primary storage pools. When a storage access group is set on a cluster/pod/zone, all hosts within that scope inherit the group. Connectivity between a host and a storage pool is then governed by whether they share the same storage access group.

A storage pool with a storage access group will connect only to hosts that have the same storage access group. A storage pool without a storage access group will connect to all hosts, including those with or without a storage access group.
2025-05-19 11:33:29 +05:30
Suresh Kumar Anaparti 90316b2e90
VMware 80u2 and 80u3 updates/fixes (#10586)
* VMware - Ignore disk not found error on cleanup when the VM disk doesn't exists

* VMware - Retry powerOn on lock issues

* addressed comments

* Update CPVM reboot tests - wait for the agent to Disconnect and back Up

* Retry moveDatastoreFile when any file access issue while creating volume from snapshot

* Update full clone flag when restoring vm using root disk offering with more size than the template size

* refactored (mainly,for diskInfo - causing NPE in some cases)

* Retry moveDatastoreFile when there is any file access issue
2025-05-17 00:39:34 +05:30
João Jandre 6fdaf51ddc
KVM incremental snapshot feature (#9270)
* KVM incremental snapshot feature

* fix log

* fix merge issues

* fix creation of folder

* fix snapshot update

* Check for hypervisor type during parent search

* fix some small bugs

* fix tests

* Address reviews

* do not remove storPool snapshots

* add support for downloading diff snaps

* Add multiple zones support

* make copied snapshots have normal names

* address reviews

* Fix in progress

* continue fix

* Fix bulk delete

* change log to trace

* Start fix on multiple secondary storages for a single zone

* Fix multiple secondary storages for a single zone

* Fix tests

* fix log

* remove bitmaps when deleting snapshots

* minor fixes

* update sql to new file

* Fix merge issues

* Create new snap chain when changing configuration

* add verification

* Fix snapshot operation selector

* fix bitmap removal

* fix chain on different storages

* address reviews

* fix small issue

* fix test

---------

Co-authored-by: João Jandre <joao@scclouds.com.br>
2025-05-12 10:50:30 -03:00
Pearl Dsilva a21f912be3 Merge branch '4.19' of https://github.com/apache/cloudstack into 4.20 2025-05-12 12:41:34 +05:30
Wei Zhou ea32a1a71a
server: fetch IP of VMs on L2 networks (#10431) 2025-05-09 10:19:30 +02:00
Wei Zhou 5d28e66b52
vTPM: support KVM and VMware (#10543)
* KVM: add Virtual TPM model and version

* KVM: add admin-only VM setting GUEST.CPU.MODE and GUEST.CPU.MODEL

* VMware: add vTPM

* vTPM: do not set Key due to 'Cannot add multiple devices using the same device key..'

* vTPM: add unit test testTpmModel

* engine/schema: remove user vm details for guest CPU mode/model

* vTPM: extra methods as Daan's requests

* vTPM: add unit tests in VmwareResourceTest

* vTPM: update unit tests in VmwareResourceTest

* vTPM: add unit test in LibvirtComputingResourceTest

* vTPM: use the default TPM version if an invalid version is passed

* vTPM: requires UEFI on vmware and do nothing if it is not enabled/disabled

* vTPM: let uses to add UEFI on vmware

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* vTPM: remove template details for guest CPU mode/model

* UI: boot vm from ISO into UEFI/SECURE mode

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-04-26 14:35:36 +05:30
Vishesh 45bac89b83
Fix NPE on updating security groups for an instance (#10493)
* Fix NPE on updating security groups for an instance

* addressed review comments

* Method refactoring

---------

Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
2025-04-22 03:12:30 -04:00
Daan Hoogland 4f3e8e8c5a Merge branch '4.19' into 4.20 2025-02-12 15:00:51 +01:00
Wei Zhou d920aba176
server: fix scale vm with same disk offering id (#10235) 2025-02-09 09:35:45 +01:00
Rene Glover 3337f425ff
Primera pure patches & various small fixes (#10132)
Co-authored-by: GLOVER RENE <rg9975@cs419-mgmtserver.rg9975nprd.app.ecp.att.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2025-02-07 13:19:34 +01:00
Abhishek Kumar 0b5a5e8043
api,agent,server,engine-schema: scalability improvements (#9840)
* api,agent,server,engine-schema: scalability improvements

Following changes and improvements have been added:

- Improvements in handling of PingRoutingCommand

    1. Added global config - `vm.sync.power.state.transitioning`, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
    2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
    3. Optimized scanning stalled VMs

- Added option to set worker threads for capacity calculation using config - `capacity.calculate.workers`

- Added caching framework based on Caffeine in-memory caching library, https://github.com/ben-manes/caffeine

- Added caching for account/use role API access with expiration after write can be configured using config - `dynamic.apichecker.cache.period`. If set to zero then there will be no caching. Default is 0.

- Added caching for account/use role API access with expiration after write set to 60 seconds.

- Added caching for some recurring DB retrievals

    1. CapacityManager - listing service offerings - beneficial in host capacity calculation
    2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
    3. DownloadListener - hypervisors for zone - beneficial for host joins
    5. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands

- Optimized MS list retrieval for agent connect

- Optimize finding ready systemvm template for zone

- Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks

- Changes in agent-agentmanager connection with NIO client-server classes

    1. Optimized the use of the executor service
    2. Refactore Agent class to better handle connections.
    3. Do SSL handshakes within worker threads
    5. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - `agent.ssl.handshake.min.workers` and `agent.ssl.handshake.max.workers` can be used to control number of new connections management server handles at a time. `agent.ssl.handshake.timeout` can be used to set number of seconds after which SSL handshake times out at MS end.
    6. On agent side backoff and sslhandshake timeout can be controlled by agent properties. `backoff.seconds` and `ssl.handshake.timeout` properties can be used.

- Improvements in StatsCollection - minimize DB retrievals.

- Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.

- Improvements in hosts connection for a storage pool. Added config - `storage.pool.host.connect.workers` to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.

- Minor improvements in resource limit calculations wrt DB retrievals

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* test1, domaindetails, capacitymanager fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test2 - agent tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* capacitymanagertest fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert marvin/setup.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix indent

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* use space in sql

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address duplicate

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update host logs

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert e36c6a5d07

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix npe in capacity calculation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* move schema changes to 4.20.1 upgrade

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix build

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add some more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* checkstyle fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unnecessary mocks

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* replace statics

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* engine/orchestration,utils: limit number of concurrent new agent
connections

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor - remove unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unregister closed connections, monitor & cleanup

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add check for outdated vm filter in power sync

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* agent: synchronize sendRequest wait

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-01 12:28:41 +05:30
Stephan Krug d9a7b6e511
Rollback of changes with errors during the VM assign (#7061)
Co-authored-by: Stephan Krug <stephan.krug@scclouds.com.br>
Co-authored-by: Gabriel <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2025-01-15 16:10:49 +01:00
Vishesh a4224e58cc
Improve logging to include more identifiable information (#9873)
* Improve logging to include more identifiable information for kvm plugin

* Update logging for scaleio plugin

* Improve logging to include more identifiable information for default volume storage plugin

* Improve logging to include more identifiable information for agent managers

* Improve logging to include more identifiable information for Listeners

* Replace ids with objects or uuids


* Improve logging to include more identifiable information for engine

* Improve logging to include more identifiable information for server

* Fixups in engine

* Improve logging to include more identifiable information for plugins

* Improve logging to include more identifiable information for Cmd classes

* Fix toString method for StorageFilterTO.java
2025-01-06 16:42:37 +05:30
Daan Hoogland b7f0aac519 Merge branch '4.19' into 4.20 2024-12-20 14:34:39 +01:00
Suresh Kumar Anaparti 188eacd9eb
Certificate and VM hostname validation improvements (#10051)
* Certificate and VM hostname validation improvements

* Improve certificate name validation and some code/log improvements
2024-12-20 10:18:11 +05:30