Commit Graph

590 Commits

Author SHA1 Message Date
Daan Hoogland 5ee61c226a Merge branch '4.19' into 4.20 2024-12-04 13:04:18 +01:00
Fabricio Duarte ef1a58d837
Remove user from project before deletion (#10008)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-12-04 08:06:22 +01:00
Daan Hoogland da54234585 Merge branch '4.19' into 4.20.merge 2024-12-03 16:32:15 +01:00
Abhisar Sinha d17de834a5
Disable API Key Access for users, accounts and domains (#9741)
* cli changes to update user/account, list by apikeyaccess, domain level setting

* UI changes for updating user/account and searchfilter in listview

* make the api parameters and setting accessible only to root admin

* revert changes to ui/package-lock.json

* minor changes to description strings

* UT for ApiServer and AccountManagerImpl classes

* fix pre-commit failure

* Added a constant for the string System

* UT for searchForUsers and searchForAccounts

* Fix marvin test error

* Update schema to use idempotent add column

* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>

* Added user name uuid to logging

* Add events when api key access is changed via api or config setting

* fix the userid for api key access update event

* Fix ut failure after event logging

* Convert drop down to radio-button in edit user and account

* Add ApiKeyAccess status in User InfoCard for Users if Api key is generated

* Return apiKeyAccess in user and account response only for Root Admin

* fixed noredist build failure

* Show apikeyaccess on the left panel in the user view for root admins as well

* don't show divider if apiKeyAccess is not shown to user

* Fix events generated to set Username, Account and Domain of the caller correctly

* cli changes to update user/account, list by apikeyaccess, domain level setting

* UI changes for updating user/account and searchfilter in listview

* make the api parameters and setting accessible only to root admin

* revert changes to ui/package-lock.json

* minor changes to description strings

* UT for ApiServer and AccountManagerImpl classes

* fix pre-commit failure

* Added a constant for the string System

* UT for searchForUsers and searchForAccounts

* Fix marvin test error

* Update schema to use idempotent add column

* Added user name uuid to logging

* Add events when api key access is changed via api or config setting

* fix the userid for api key access update event

* Fix ut failure after event logging

* Convert drop down to radio-button in edit user and account

* Add ApiKeyAccess status in User InfoCard for Users if Api key is generated

* Return apiKeyAccess in user and account response only for Root Admin

* fixed noredist build failure

* Show apikeyaccess on the left panel in the user view for root admins as well

* don't show divider if apiKeyAccess is not shown to user

* Fix events generated to set Username, Account and Domain of the caller correctly

* Added DB upgrade path from 42000 to 42010

---------

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
2024-12-03 12:10:54 +05:30
Rodrigo D. Lopez 4189bac8e0
enable to create VPC portfowarding rules with source cidr (#7081)
Co-authored-by: Lopez <rodrigo@scclouds.com.br>
Co-authored-by: Fabricio Duarte <fabricio.duarte.jr@gmail.com>
2024-11-28 17:53:07 +01:00
Felipe a8cb7abca3
Add new column `last_id` to the table volumes (#9759)
Co-authored-by: João Jandre <48719461+JoaoJandre@users.noreply.github.com>
2024-11-28 17:51:53 +01:00
Wei Zhou 019f2c60ce
server: fix vm_stats records are not removed if batchsize is 0 (#9818) 2024-10-18 14:52:46 +05:30
João Jandre bf2cedea79 Merge branch 4.19 to main 2024-09-26 09:01:22 -03:00
Abhishek Kumar 53aa92199b
server: fix nfs version option during mount (#9559)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-09-25 16:47:16 +02:00
Vishesh 0655075f51
Feature: Forgot password (#9509)
* Feature: Forgot password

* Address comments

* fixups

* Make forgot password disabled by default

* Apply suggestions from code review

* Address comments
2024-09-10 12:55:28 -03:00
Vishesh 1303a4f323
Feature: Allow adding delete protection for VMs & volumes (#9633)
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-09-09 14:44:50 +02:00
Nicolas Vazquez 8c8d115a1e
feature: Support Multi-arch Zones (#9619)
This introduces the multi-arch zones, allowing users to select the VM arch upon deployment. 

Multi-arch zone support in CloudStack can allow admins to mix x86_64 & arm64 hosts within the same zone with the following changes proposed:
- All hosts in a clusters need to be homogenous, wrt host CPU type (amd64 vs arm64) and hypevisor
- Arch-aware templates & ISOs:
   -  Add support for a new arch field (default set of: amd64 and arm64), when unspecified defaults to amd64 and for existing templates & iso
   -  Allow admins to edit the arch type of the registered template & iso
- Arch-aware clusters and host:
   - Add new attribute field for cluster and hosts (kvm host agents can automatically report this, arch of the first host of the cluster is cluster's architecture), defaults to amd64 when not specified
   - Allow admins to edit the arch of an existing cluster
- VM deployment form (UI):
   - In a multi-arch zone/env, the VM deployment form can allow some kind of template/iso filtration in the UI
   - Users should be able to select arch: amd64 & arm64; but this is shown only in a multi-arch zone (env)
- VM orchestration and lifecycle operations:
   - Use of VM/template's arch to correctly decide where to provision the VM (on the correct strictly arch-matching host/clusters) & other lifecycle operations (such as migration from/to arch-matching hosts)

Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 12:14:54 +05:30
Rohit Yadav f683d2dd24 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 10:59:36 +05:30
Wei Zhou 679ce1a639
feature: Dynamic and Static Routing (#9470)
This PR contains 3 features

- IPv4 Static Routing (Routed mode) #9346
Design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=306153967

- AS Numbers Management #9410
Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/BGP+AS+Numbers+Management


- Dynamic routing
Design Document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=315492858

- Document: https://github.com/apache/cloudstack-documentation/pull/419

Rename nsx mode to routing mode

by
```
git grep -l nsx_mode  |xargs sed -i "s/nsx_mode/routing_mode/g"
git grep -l nsxmode  |xargs sed -i "s/nsxmode/routingmode/g"
git grep -l nsxMode  |xargs sed -i "s/nsxMode/routingMode/g"
git grep -l NsxMode  |xargs sed -i "s/NsxMode/RoutingMode/g"
```
- re-organize sql changes
- fix NPE as rules do not have public ip
- fix missing destination cidr in ingress rules
- disable network usage for routed network
- fix DB exception as network_id is -1 during network creation
- apply ingress/egress routing rules
- VR changes to configure nft rules for isolated network
- VR: setup nft rule for control network
- VR: flush all iptables rules
- fix NPE which is because ingress rules do not have public ip associated
- fix dest cidr is missing in nft tables
- add ip4 routing and ip4 routes to list network and list vpc response
- fix ingress rule is missing when vr is restarted
- fix icmp types in nft rules
- add tab to manage routing firewall rules
- fix ingress rules are not applied when VR is restarted
- add default rules in FORWARD chain
- fix create vpc offerings
- fix public ip is not assigned to vpc
- fix network offering is not listed when create vpc tier
- add is_routing to boot args of vpc vr
- remove table ip4_firewall in vpc vr
- release or remove subnet when remove a network
- implemenent fw_vpcrouter_routing
- fix wrong ip familty when flush ipv4 rules
- fix acl rules are not applied due to wrong version (should be 6 which means ip6 rules are removed)
- add default rules for vpc tiers so that tcp connections (e.g. ssh) work
- append policy rules after default rules
- remove /usr/local/cloud/systemvm/ in routers
- throw an exception when allocate subnet with cidrsize
- fix some TODOs
- add new parameters to update API
- return type Ipv4GuestSubnetNetworkMap when get or create subnet
- fix firewall rules are broken
- add domain_id and account_id to db
- add domain/account/project to ipv4 subnet response
- create ipv4 subnet for domain/account/project
- check conflict when update ipv4 subnet
- ui changes
- add parent subnet to response
- add list for ipv4 subnet
- implement some methods
- fix list subnets for guest networks by zoneid
- UI changes
- fix delete ipv4 subnet for network
- fix ipv4 subnet is set to zone guest network cidr if cidrsize is specified
- add zone info to response if parent subnet is null but network is not
- fix gateway/cidr is not set when create network with cidrsize
- fix order of nft rules in the VRs

* Routed v24

- add classes in marvin base.py

* Routed v25

- add test_01_subnet_zone
- fix dedicate to domain/account failure
- list subnets for network by keyword and subnet

* Routed v26: implement subnet auto-allocation

- add utils for split ip ranges into small subnets
- add utils to get start/end ip of a cidr
- implement subnet auto-generation
- add global settings

* Routed 27: add subnet for VPC

- add db column for vpc_id
- add db record for vpc
- remove db record when delete a vpc
- add checkConflicts methods
- remove duplicated settings
- check ipv4 cidr when create subnet

* Routed v28: update smoke tests

- update test_ipv4_routing.py
- search subnets by networkid

* Routed 29: fix vpc and add more tests

- fix createnetwork in vpc
- add vpc id/name to response
- fix zone id/name are not displayed in some cases
- add smoke test for vpc
- add smoke tests for failed cases
- add smoke test for connectivity checks
- marvin: add "-q" to ssh command

* Routed 31: ui and smoke tests

- UI: add link to network in list view
- add nftables rules check in VRs

* Routed 32: add chain OUTPUT and more rules

- fix the issue 80/443/8080 is not reachable from VR itself

```
2024-06-27 10:21:52,121 INFO     Executing: systemctl start cloud-password-server@172.31.1.1
2024-06-27 10:21:52,128 INFO     Service cloud-password-server@172.31.1.1 start
2024-06-27 10:21:52,129 INFO     Executing: ps aux
2024-06-27 10:24:02,175 ERROR    Failed to update password server due to: <urlopen error [Errno 110] Connection timed out>
```

* Routed: fix dns search from VMs in Isolated networks

* Routed: fix VPC dns issue due to gateway IP is missing in cloud.conf

This is caused by NSX integration, and fixed by
https://github.com/apache/cloudstack/pull/9102/

* Routed: rename routing_mode to network_mode

* Routed: replace centos5.5 template in smoke test as dhclient does not work in the vms

// this does not work
refer to https://dominikrys.com/posts/disable-udp-checksum-validation/#ignoring-udp-checksums-with-nftables
and
https://forum.openwrt.org/t/udp-checksum-with-nftables/161522/11

the vm should have checksum offloading disabled

* Routed: fix smoke test due to wrong cidrlist of egress rules and missing ingress rule from VR

* PR 9346: fix lint error schema-41910to42000.sql

* PR 9346: ui polish v1

* PR 9346: create VPC with cidrsize

* Routed: fix test failures with test_network_ipv6 and test_vpc_ipv6 due to 'ssh -q'

* Routed: fix /usr/local/cloud/systemvm/ are removed after SSVM/CPVM reboot

* Routed: fix IP of additional nics of VPC VR is not gateway

* PR 9346: fix cidrsize check when create VPC with cidrsize

* Routed: fix test/integration/smoke/test_ipv4_routing.py:279:16: E713 test for membership should be 'not in'

* PR9346: fix/Update api

* PR 9346: set response object name

* PR9346: UI refactor and small fixes

* PR9346: change return type of getNetworkMode

* PR9346: move IPv4 subnet to seperated tab

* PR9346: revert IpRangesTabGuest.vue back to original

* PR9346: fix remove ipv4 subnet on UI

* PR9346: fix test_ipv4_routing.py

* AS Number Range Management

* Create AS Number Range for a Zone

* Fix build

* Add ListASNRange and fix create ASN range

* Add List AS numbers

* Add UI for AS Numbers

* Fix UI and filter AS Numbers

* Add AS Number on Isolated network creation and refactor UI and response

* Release AS Number

* Add network offering new columns

* Add UI support to view and add AS number and configure network offering

* Automatically assign AS Number if not specify AS number

* update variable name

* Fix routing mode check

* UI: Only allow selecting AS number when routing mode is Dynamic and specifyAsNumber is true

* UI: Only pass AS number when supported by the network offering

* Release AS number on network deletion

* Add deleteASNRange command (#81)

* API: List ASNumbers by asnumber (#83)

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* AS number management extensions

* Support AS number on VPC tier creation based on the offering

* Fix delete AS Range

* Fix UI values

* UI: Minor fix for releasing AS number

* UI: Move management of AS Range to Zone details view

* Fix specify_as_number column in network_offering table to set the default false

* Add events for AS number operations

* Allow users to list AS Numbers and fix network form for Normal users

* Add AS number details to list networks response

* Fix Allocated time format

* Fix Allocated time format

* support in details view too

* Fix: Do not release AS number if acquired network requires AS number

* Fix: Do not release AS number if acquired network requires AS number

* Fix typo

* Fix allocated release

* Fix event type

* UI: Add Routing mode and Specify AS to the network offering details

* UI: Add Routing mode and Specify AS to the network offering details

* Address comment

* Fix release AS number of network deletion

* Fix release AS number of network deletion

* Fix

* Restore release to its place based on the boolean

* Rename boolean

* API: Add networkId as listASNumber parameter

* Add Network name to the search view filter for AS numbers

* Present allocated time in human readable format - Pubilc IP / AS Numbers

* Add account / domain filter for AS numbers

* Add support for AS numbers on VPC offerings

* Refactor AS number allocation to VPC and non VPC isolated networks

* Checkstyle

* Add support for AS numbers on VPC offerings

* extend vpc offering view and vpcoffering response

* merge https://github.com/shapeblue/cloudstack-playtika/pull/115 and change network_id of as_numbers to include vpc_id

* Display AS number of VPC tiers as the AS number of the VPC

* extend asnumber response and ui support

* improve UI and as number response to view VPC details

* List only dynamic offerings for vpc tiers with specify as numbers

* Fix release AS number

* Fix AS number displayed as 0 when no AS number assigned

* Fix VPC offering creation without specify AS

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix release AS number on VPC deletion

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Update server/src/main/java/com/cloud/dc/BGPServiceImpl.java

* Fix missing column on asnumber table

* Fix listASNumbers API to support vpcid and obtain AS number from vpc for tiers

* Prevent listing 0 AS number for VPC

* Fix create Isolated Network form

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Update server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

* Dynamic: move routingmode/specifyasn after networkmode in AddNetworkOffering.vue on UI

* Dynamic: fix ip4routing in network response

* Dynamic/systemvm: add FRR to systemvm template

* Dynamic: BGP peers (DB,VO,Dao)

* Dynamic: BGP peers (VR/server)

* Dynamic: v3

- remove BgpPeer class
- fix vpc vr has bgp peers of only 1 tier
- rename ip4_cidr to guest_ip4_cidr
- rename ip6_cidr to guest_ip6_cidr
- generate /etc/frr/frr.conf
- apply BGP peers on Dynamic-Routed network even if there is no BGP peers

* Dynamic v4: fix vpc vr

- fix duplicated guest cidr in frr.conf in vpc vr

todo
- restart frr / reload frr (reload will cause bgp session to Policy state)
- apis for bgp peers
- assign/release bgp peer from/to network

* Dynamic v5: add apis for bgp peers

* Dynamic v6: fix bugs

- set response object name
- remove required as number when update
- fix checks when update
- allow regular users to list bgp peers

* Dynamic v7: move apis to bgp sub-dir

* Dynamic v8: add tab for manage BGP peers on UI

* Dynamic v9: fix update bgp with same config

* Dynamiv v10: add changeBgpPeersForNetworkCmd

* Dynamic v11: create network with bgppeerids

- create network with bgppeerids
- add marvin classes
- add smoke tests
- remove uuid from bgp_peer_network_map
- fix created/removed in bgp_peer_network_map
- remove bgppeers when remove a network
- UI: fix delete bgp peer

* Dynamic v12: add test for vpc tiers

* Dynamic v13: bug fixes

- fix change BGP peers for network in Allocated state
- fix listing network returns removed record
- fix all vpc tiers have the same settings
- remove BGP peers as part of network removal
- remove FRR settings for vpc tiers without any BGP peers
- UI: fix no error msg when change BGP peers

* Dynamic v14: assign BGP Peers for VPC instead of VPC tiers

- create vpc with bgppeerids
- do not allow create/update vpc tier with bgppeerids
- apply all bgp peers when create/delete a vpc tier
- UI: change bgp peers for vpc
- test: update tests on vpc

* Dynamic: fix build errors after merging as number PR

* Dynamic: fix TODOs

* Dynamic: fix smoke test on VPC

* Allow creation of networks by users with as numbers

* Address review comments

* Move BGPService to bgp package and inject it on BaseCmd

* Revert changes for CKS and address more comments

* Display left side menu option for AS number only for root admin

* Dynamic: create/update BGP peer with details

refer to https://docs.frrouting.org/en/latest/bgp.html

* Dynamic: fix build error and remove access to ListBgpPeers cmd for regular users

* Dynamic: assign all zone BGP peers to user networks

* Dynamic: show BGP peer info of networks only for root admin

* AS number: disable specifyasnumber for non-NSX offerings

* Dynamic: pass bgppeer details to command and fix typo with ip6 addr

* Dynamic: list BGP peers by isdedicated, and fix change bgppeers for network/vpc

* Dynamic: add UI labels

* Dynamic: add bgp peers to vpc response

* Dynamic: list bgp peers by keyword, fix list by asnumber

* Dynamic: fix list bgppeers by keyword and db schema

* Dynamic: fix list bgppeers do not return dedicated peers

* Dynamic: update UI when create network/vpc offering

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update tools/marvin/setup.py

* Dynamic: network mode must be same when update a network with new offering

* Dynamic: add method networkModel.isAnyServiceSupportedInNetwork

* Dynamic: rename APIs and classes

* Dynamic: fix unit tests due to previous changes

* Dynamic: validateNetworkCidrSize when auto-create subnet

* Dynamic: check AS number overlap

* Dynamic: add ActionEvent

* Dynamic: small code optimization

* Dynamic: fix ui bugs after api rename

* Dynamic: add marvin and test for ASN ranges and AS numbers

* Dynamic: add account setting use.system.bgp.peers

also
- change the default value of routed.ipv4.vpc.max.cidr.size and routed.ipv4.vpc.min.cidr.size
- change the category of settings

* static: fix ui error when delete zone ipv4 subnets

* static: small UI polish

* Dynamic: throw exception when as number is required but not passed

* Dynamic: fix typo when create FRR directory which causes network deletion failures

* Dynamic: connect to ALL (or ALL dedicated) BGP peers if no BGP peer mapping for the network/vpc

* Dynamic: throw exception when as number is required for VPC but not passed

* Dynamic: list bgp peers by useSystemBgpPeers

* Dynamic: fix frr config in VPC VR when change bgp peers

* Dynamic: create frr config even if there is no VPC tiers

* Dynamic: list bgp peers by zoneid (required for account) and account

* Dynamic: only apply FRR config for vpc tiers with dynamic routing

* Dynamic: donot send commands to router if commands size is 0

* Dynamic: fix 'new IPv6 address is not valid' when update bgp peer without IPv6

* Dynamic: throw exception if fail to allocate AS number when create network/vpc with dynamic routing

* Dynamic: enable ipv6 unicast and 'ip nht resolve-via-default'

* Dynamic: delete network/vpc if fail to allocate AS number when create network/vpc with dynamic routing

* test: add unit tests for ASN APIs

* test: add unit tests for core module

* test: add unit tests for API responses

* test: add unit tests for BgpPeerTO

* test: add minor changes

* test: add tests for create/delete/update/list RoutingFirewallRuleCmd

* Static: show ip4 routes for vpc tiers

* test: fix smoke test failure caused by type change of as number

* test: add test for Ipv4SubnetForZoneCmd

* test: add test for Ipv4SubnetForGuestNetworkCmd and BgpPeerCmd

* UI: do not show redundant router when network mode is ROUTED as RVR is not supported

* UI: hide 'Conserve mode' when networkmode is ROUTED

* test: add unit tests for ListASNumbersCmdTest

* Static: remove allocated IPv4 subnet when delete a network or vpc

* test: add unit tests for BgpPeersRules

* Dynamic: set ipv4routing from network offering

* server: list as numbers and ipv4 subnets by keyword

* server: remove dedicated bgp peers and ipv4 subnets when delete an account or domain

* server: fix dedicated ipv4 subnet is allocated to other accounts

* UI: fix allocated time format

* server: ignore project is projectid is -1 so bgppeers/ipv4subnets works in project view

* UI: add project column to bgp peers and ipv4 subnets

* server: fix list AS numbers by domain admin or normal user

* server: fix network creation when ipv4 subnet is dedicated

* UI: polish network.js

* Dynamic: fix frr config for ipv6 routing

* Static routing: support cks cluster

* Static: get/create IPv4 subnet from dedicated subnets at first

* Dynamic: add BGP peers tab

* Static: remove redundant loops

* api: add since to api and response

* server: add unit tests

---------

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-06 08:55:17 +05:30
Wido den Hollander c3f0d14d31
storage/object: Add support for Ceph RGW Object Store (#8389)
This feature adds support for Ceph's RADOS Gateway (RGW) support for the
Object Store feature of CloudStack.

The RGW of Ceph is Amazon S3 compliant and is therefor an easy and straigforward
implementation of basic S3 features.

Existing Ceph environments can have the RGW added as an additional feature to a
cluster already providing RBD (Block Device) to a CloudStack environment.

Introduce the BucketTO to pass to the drivers. This replaces just passing the bucket's name.

Some upcoming drivers require more information then just the bucket name to perform their actions,
for example they require the access and secret key which belong to the account of this bucket.

This is leftover code from a long time ago and this validation test has nu influence
on the end result on how a URL will be used afterwards.

We should support hosts pointing to an IPv6(-only) address out of the box.

For the code it does not matter if it's IPv4 or IPv6. This is the admin's choice.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-09-05 17:58:39 +05:30
dahn 0fae0c021f
propagate sort order through retrieval sequence (#9624) 2024-09-05 09:37:30 +02:00
Abhishek Kumar 7e085d5e1d
framework/db: use HikariCP as default and improvements (#9518)
Per docs, if the mysql connector is JDBC2 compliant then it should use
the Connection.isValid API to test a connection.
(https://docs.oracle.com/javase/8/docs/api/java/sql/Connection.html#isValid-int-)

This would significantly reduce query lags and API throughput, as for
every SQL query one or two SELECT 1 are performed everytime a Connection
is given to application logic.

This should only be accepted when the driver is JDBC4 complaint.

As per the docs, the connector-j can use /* ping */ before calling
SELECT 1 to have light weight application pings to the server:
https://dev.mysql.com/doc/connector-j/en/connector-j-usagenotes-j2ee-concepts-connection-pooling.html

Replaces dbcp2 connection pool library with more performant HikariCP.
With this unit tests are failing but build is passing.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohityadav89@gmail.com>
2024-09-05 09:36:58 +05:30
Abhishek Kumar 31b0ed0a18
framework/config,server: configkey caching (#9628)
Added caching for ConfigKey value retrievals based on the Caffeine
in-memory caching library.
https://github.com/ben-manes/caffeine
Currently, expire time for a cache is 30s and each update of the
config key invalidates the cache. On any update or reset of the
configuration, cache automatically invalidates for it.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-09-05 09:32:56 +05:30
Fabricio Duarte 7692b745e8
Fix NPE when sending copy command to least busy SSVM (#9125) 2024-08-28 12:06:16 +02:00
Henrique Sato c9f1c5790d
Fix snapshot scheduling with expired jobs (#8832)
Co-authored-by: Henrique Sato <henrique.sato@scclouds.com.br>
2024-08-23 11:19:13 +02:00
Daan Hoogland 287a47266b Merge branch '4.19' 2024-08-14 12:33:47 +02:00
Felipe cf1428ddcc
Fix column from op_dc_ip_address_alloc not being referenced correctly by its ORM class (#8812)
* refactored field instanceId to nicId in DataCenterIpAddressVO and AcquirePodIpCmdResponse

* refactored ocurrences of "instanceId" in DataCenterDaoImpl and DataCenterIpAddressDaoImpl
2024-08-13 16:26:55 -03:00
dahn 6e6a276663
add procedures procedure (#9385) 2024-08-12 14:47:34 +02:00
Suresh Kumar Anaparti 161d156af1
Support user resource name / displaytext with emoji, unicode chars, and some sql exception msg improvements (#9340)
* Added API arg validator for RFC compliance domain name, to validate VM's host name

* Added unit tests for vm host/domain name validation

* Don't send sql exception/query from dao to upper layer, log it and send only the error message

* Updated user resources name / displayname(/text) column's charset to utf8mb4 to support emojis / unicode chars

* Check and update char set for affinity group name to utf8mb4, from the data migration in upgrade path

* Added smoke test to check resource name for vm, volume, service & disk offering, template, iso, account(first/lastname)

* Updated resource annotation charset to utf8mb4

* Updated some resources description charset to utf8mb4

* Updated sql stmt with constant

* Updated modify columns char set with idempotent procedure

* Removed delimiter (for creating procedures)
2024-08-07 14:50:29 +05:30
Rohit Yadav 7f4a6861b3 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-07-29 09:46:43 +05:30
Rohit Yadav 3d32ab5a49 Merge remote-tracking branch 'origin/4.18' into 4.19 2024-07-29 09:42:38 +05:30
Vishesh 33be0ae745
Fix removal of usage records (#9442) 2024-07-26 11:40:02 +03:00
Vishesh 35fd17c62e
Merge branch '4.19' 2024-07-17 17:43:40 +05:30
Abhishek Kumar 6a8494be3b
engine-schema,cks: fix finding non removed network vms (#9339)
Fixes #9331

Only those VMs should be considered network VM which have a NIC entry
that is not marked removed.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-07-15 09:46:29 +05:30
Suresh Kumar Anaparti 7f8f113709
Merge branch '4.18' into 4.19 2024-07-15 00:59:06 +05:30
Vishesh 22ef08154d
Switch back to CLOUD_DB after purging usage records (#9367) 2024-07-14 14:58:36 +05:30
Vishesh 0ec7c72875
Merge branch '4.19' 2024-07-01 12:41:45 +05:30
Abhisar Sinha 063dc60114
Change storage pool scope from Cluster to Zone and vise versa (#8875)
* New feature: Change storage pool scope

* Added checks for Ceph/RBD

* Update op_host_capacity table on primary storage scope change

* Storage pool scope change integration test

* pull 8875 : Addressed review comments

* Pull 8875: remove storage checks, AbstractPrimayStorageLifeCycleImpl class

* Pull 8875: Fixed integration test failure

* Pull 8875: Review comments

* Pull 8875: review comments + broke changeStoragePoolScope into smaller functions

* Added UT for changeStoragePoolScope

* Rename AbstractPrimaryDataStoreLifeCycleImpl to BasePrimaryDataStoreLifeCycleImpl

* Pull 8875: Dao review comments

* Pull 8875: Rename changeStoragePoolScope.vue to ChangeStoragePoolScope.vue

* Pull 8875: Created a new smokes test file + A single warning msg in ui

* Pull 8875: Added cleanup in test_primary_storage_scope.py

* Pull 8875: Type in en.json

* Pull 8875: cleanup array in test_primary_storage_scope.py

* Pull:8875 Removing extra whitespace at eof of StorageManagerImplTest

* Pull 8875: Added UT for PrimaryDataStoreHelper and BasePrimaryDataStoreLifeCycleImpl

* Pull 8875: Added license header

* Pull 8875: Fixed sql query for vmstates

* Pull 8875: Changed icon plus info on disabled mode in apidoc

* Pull 8875: Change scope should not work for local storage

* Pull 8875: Change scope completion event

* Pull 8875: Added api findAffectedVmsForStorageScopeChange

* Pull 8875: Added UT for findAffectedVmsForStorageScopeChange and removed listByPoolIdVMStatesNotInCluster

* Pull 8875: Review comments + Vm name in response

* Pull 8875: listByVmsNotInClusterUsingPool was returning duplicate VM entries because of multiple volumes in the VM satisfying the criteria

* Pull 8875: fixed listAffectedVmsForStorageScopeChange UT

* listAffectedVmsForStorageScopeChange should work if the pool is not disabled

* Fix listAffectedVmsForStorageScopeChangeTest UT

* Pull 8875: add volume.removed not null check in VmsNotInClusterUsingPool query

* Pull 8875: minor refactoring in changeStoragePoolScopeToCluster

* Update server/src/main/java/com/cloud/storage/StorageManagerImpl.java

* fix eof

* changeStoragePoolScopeToZone should connect pool to all Up hosts

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-29 10:03:34 +05:30
Vishesh 90fe1d5fdc
Merge branch '4.19' 2024-06-29 03:35:24 +05:30
Vishesh a4e9d7f21a
Change vm.stats.remove.batch.size to delete.batch.query.size & allow delete of volume_stats in batches (#9283)
* Change vm.stats.remove.batch.size to delete.batch.query.size

* Add support for deletion of volume stats in batches

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Update configkey description

* Address comments

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-28 15:32:49 +05:30
Suresh Kumar Anaparti 46f672563e
Improve migration of external VMware VMs into KVM cluster (#8815)
* Create/Export OVA file of the VM on external vCenter host, to temporary conversion location (NFS)

* Fixed ova issue on untar/extract ovf from ova file
"tar -xf" cmd on ova fails with "ovf: Not found in archive" while extracting ovf file

* Updated VMware to KVM instance migration using OVA

* Refactoring and cleanup

* test fixes

* Consider zone wide pools in the destination cluster for instance conversion

* Remove local storage pool support as temporary conversion location
- OVA export not possible as the pool is not accessible outside host, NFS pools are supported.

* cleanup unused code

* some improvements, and refactoring

* import nic unit tests

* vmware guru unit tests

* Separate clone VM and create template file for VMware migration
- Export OVA (of the cloned VM) to the conversion location takes time.
- Do any validations with cloned VM before creating the template (and fail early).
- Updated unit tests.

* Check conversion support on host before clone vm / create template on vmware (and fail early)

* minor code improvements

* Auto select the host with instance conversion capability

* Skip instance conversion supported response param for non-KVM hosts

* Show supported conversion hosts in the UI

* Skip persistence map update if network doesn't exist

* Added support to export OVA from KVM host, through ovftool (when installed in KVM host)

* Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadvmfiles', to be generic

* Updated hardcoded UI messages with message labels

* Updated UI to support importvm api param - forcemstodownloadvmfiles

* Improved instance conversion support checks on ubuntu hosts, and for windows guest vms

* Use OVF template (VM disks and spec files) for instance conversion from VMware, instead of OVA file
 - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

* OVF export tool parallel threads code improvements

* Updated 'convert.vmware.instance.to.kvm.timeout' config default value to 3 hrs

* Config values check & code improvements

* Updated import log, with time taken and vm details

* Support for parallel downloads of VMware VM disk files while exporting OVF from MS, and other changes below.
- Skip clone for powered off VMs
- Fixes to support standalone host (with its default datacenter)
- Some code improvements

* rebase fixes

* rebase fixes

* minor improvement

* code improvements - threads configuration, and api parameter changes to import vm files

* typo fix in error msg
2024-06-27 21:14:13 +05:30
Wei Zhou 22cd00ffb1
veeam: fix issues with PreSetup and DVS and Solidfire (#9256)
* Veeam: find storage pool by path for PreSetup and VMFS

* Veeam: support VMware distributed virtual switch

* Veeam: sync volumes on Solidfire after backup restoration

user faced the issue that backup is restored but the DATA disk is gone (ROOT disk is ok)
```
2024-05-03 12:00:32,868 ERROR [o.a.c.b.BackupManagerImpl] (API-Job-Executor-13:ctx-aa8a1d85 job-149661 ctx-73328567) (logid:6510cf06) Failed to import VM [vmInternalName: i-169-9679-VM] from backup restoration [{"backupType":"Full","externalId":"821ca400-a5da-4282-bf3f-7c7e38a6cdb4","id":257,"uuid":"69399101-5cbd-461c-8a48-f0c70eac0b24","vmId":9679}] with hypervisor [type: VMware] due to: [Couldn't find storage pool -iqn.2010-01.com.solidfire:3p53.data-9679.221-0].
```

On managed storage, the datastore name of DATA disk is determined by the iscsi_name of the volume.

* Veeam: set correct path for DATA disks on solidfire
2024-06-26 18:02:25 +05:30
slavkap 6c06e85c80
Temporarily backup StorPool volume before expunge (#8843)
* Temporarily backup StorPool volume before expunge

Sometimes the users delete the volumes by mistake. This enhancment
provides a solution to backup the volume before it's deleted. The user
will be able to see the snapshot in CloudStack UI/CLI and create only a
volume from it.
A task will check (by default on every 5mins) if the snapshots are
deleted from StorPool

Global settings to enable the delay delete option:

`storpool.delete.after.interval` - The interval (in seconds) after the StorPool snapshot will be deleted
`storpool.list.snapshots.delete.after.interval` - The interval (in seconds) to fetch the StorPool snapshots with deleteAfter flag

Minor fix when deleting snapshots

* added Apache licence

* addressed comments
2024-06-26 13:58:04 +05:30
Vishesh 3923f80c22
Merge branch '4.19' 2024-06-25 18:53:57 +05:30
Vishesh f6ceeab3b3
server: Enforce strict host tag check (#9017)
Documentation PR: https://github.com/apache/cloudstack-documentation/pull/398

Currently, an administrator can break host tag compatibility for a VM administrator by certain operations:
* deploy/start VM on a specific host
* migrate VM
* restore VM
* scale VM

This PR allows the user to specify tags which must be checked during these operations.

Global Settings
1. `vm.strict.host.tags` - A comma-separated list of tags for strict host check (Default - empty)
2. `vm.strict.resource.limit.host.tag.check` - Determines whether the resource limits tags are considered strict or not (Default - true)

During the above operations, we now check and throw an error if host tags compatibility is being broken for tags specified in `vm.strict.host.tags`. If `vm.strict.resource.limit.host.tag.check` is set to `true`, tags set in `resource.limit.host.tags` are also checked during these operations.
2024-06-25 14:42:17 +05:30
Abhishek Kumar de266a4cb8
api,server: allow updating hypervisor capabilities with hypervisor and version (#8475)
* api,server: allow updating hypervisor capabilities with hypervisor and
version

hypervisor and hypervisorversion parameter added to the
updateHypervisorCapabilities API.

* param description

* Update server/src/main/java/com/cloud/server/ManagementServerImpl.java

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Henrique Sato <henriquesato2003@gmail.com>
Co-authored-by: Vishesh <vishesh92@gmail.com>
2024-06-24 14:41:01 +05:30
João Jandre ae3fa5d0de
Add configuration to limit the number of rows deleted from vm_stats (#8740)
Co-authored-by: João Jandre <joao@scclouds.com.br>
2024-06-20 14:26:36 +02:00
Abhishek Kumar 3e6900ac1a
api,server: purge expunged resources (#8999)
This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine). There would be three mechanisms for purging removed resources:

    Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
    API - New admin-only API purgeExpungedResources. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate. Currently, API is not supported in the UI.
    Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.

Following new global settings have been added:

    expunged.resources.purge.enabled: Default: false. Whether to run a background task to purge the expunged resources
    expunged.resources.purge.resources: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the expunged resources. Currently only VirtualMachine is supported. An empty "value will result in considering all resource types for purging
    expunged.resources.purge.interval: Default: 86400. Interval (in seconds) for the background task to purge the expunged resources
    expunged.resources.purge.delay: Default: 300. Initial delay (in seconds) to start the background task to purge the expunged resources task.
    expunged.resources.purge.batch.size: Default: 50. Batch size to be used during expunged resources purging.
    expunged.resources.purge.start.time: Default: (empty). Start time to be used by the background task to purge the expunged resources. Use format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.
    expunged.resources.purge.keep.past.days: Default: 30. The number of days in the past from the execution time of the background task to purge the expunged resources for which the expunged resources must not be purged. To enable purging expunged resource till the execution of the background task, set the value to zero.
    expunged.resource.purge.job.delay: Default: 180. Delay (in seconds) to execute the purging of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.

Documentation PR: apache/cloudstack-documentation#397

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-20 11:34:44 +05:30
Daan Hoogland 373f017002 Merge branch '4.19' 2024-06-18 19:58:43 +02:00
Bryan Lima a10eee25b3
Add method for decrypting values accordingly (#9088) 2024-06-18 10:31:35 +02:00
dahn ed86dc973b
protect against missing service offering (#9235)
* protect agains missing service offering

* search removed before assuming none

* import

* javadoc
2024-06-14 15:25:46 +05:30
Vishesh 6fe835e118
Merge branch '4.19' 2024-06-13 11:57:55 +05:30
dahn 4de975ff25
list by isEncrypted (#8643)
* list by isEncrypted

* use filter on VO and cleanup

* add encryption type to volume response

* Update api/src/main/java/org/apache/cloudstack/api/command/user/volume/ListVolumesCmd.java

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2024-06-12 15:11:00 +05:30
Daan Hoogland b29391c329 Merge LTS branch '4.19' into main 2024-06-07 09:53:57 +02:00
Abhishek Kumar 91c7bc722f
server,cks: check if vm is cks node during vm destroy (#9057)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-06-06 10:54:02 +02:00
Wei Zhou 9b3c318db6
upgrade: move 4.19.0->4.20.0 to 4.19.1->4.20.0 (#8893)
This PR move upgrade files for 4.19.0->4.20.0 to 4.19.1->4.20.0
2024-06-03 11:42:28 +05:30
Wei Zhou 5433e775e5
New feature: Implicit host tags (#8929)
* Merge two HostTagVO and HostTagDaoImpl

* Implicit host tags

* PR8929: add since

* Update variable names

* Update 8929: add unit test in LibvirtComputingResourceTest

* Update 8929: add explicithosttags in response

* Update 8929 UI: Update explicit host tags

* Update 8929: remove host tags and change labels on UI

* Update 8929: update host_view to use explicit_host_tags.is_tag_a_rule

* Update: ui polish for host tags

* Update 8929: fix UI error if no host tags
2024-05-30 17:21:13 +05:30
Daan Hoogland f1c3d2c4be Merge release branch 4.19 to main
* 4.19:
  engine-schema: add removed columne for cloud.user_data, fix delete op (#9120)
  ui: fix create menu item access (#9104)
  server: honor listall param for listiso api (#9064)
  ui: add move to top/bottom for acl rules list (#9119)
  ui: fix columns for exportacls csv (#9118)
  ui: fix create network access in deploy vm wizard (#9117)
  UI: Add search filters (#9068)
2024-05-28 09:44:05 +02:00
Abhishek Kumar c970141e25
engine-schema: add removed columne for cloud.user_data, fix delete op (#9120)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-05-28 09:43:14 +02:00
Daan Hoogland e61f3bae4d Merge branch '4.19' 2024-04-29 11:37:40 +02:00
Rohit Yadav 3de1f8b4ba Merge remote-tracking branch 'origin/4.18' into 4.19
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-04-29 13:44:34 +05:30
Wei Zhou 0e08a126df
systemvm: add template_zone_ref record when add a new zone with same hypervisor type (#8395) 2024-04-25 08:41:59 +02:00
Daan Hoogland 0af923e618 Merge branch '4.19' 2024-04-24 14:01:02 +02:00
SadiJr 96ae479000
[Usage] Create network billing (#7236)
Co-authored-by: Bryan Lima <bryan.lima@hotmail.com>
Co-authored-by: SadiJr <sadi@scclouds.com.br>
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
Co-authored-by: Henrique Sato <henriquesato2003@gmail.com>
2024-04-24 08:52:49 +02:00
Wei Zhou bf9fdaddbd
Fix build errors due to log4j 2.x changes 2024-04-23 11:43:08 +02:00
Wei Zhou 7b57480d60
Merge remote-tracking branch 'apache/4.19' into main 2024-04-23 10:06:16 +02:00
dahn 6502dde8c3
field enlarged and db upgrade (#8675)
* 4.19 -> 4.19.1 and enlarge url field
2024-04-23 13:01:08 +05:30
SadiJr 49d244f251
[Usage] Create VPC billing (#7235)
Co-authored-by: SadiJr <sadi@scclouds.com.br>
Co-authored-by: Bryan Lima <bryan.lima@hotmail.com>
2024-04-22 09:42:24 +02:00
Vishesh ebaf5a47b9
Speedup resource count calculation (#8903)
* Speed up resource count calculation

* Refactor resource count calculation

* Start transaction for updateCountByDeltaForIds
2024-04-17 14:21:30 +05:30
Vishesh 19f79b1d94
Merge branch '4.19' 2024-04-12 16:31:40 +05:30
Vishesh 730cc5d5b8
Change iops on offering change (#8872)
* Change IOPS on disk offering change

* Remove iops & bandwidth limits before copying template

* minor refactor

* Handle diskOfferingDetails

* Fixup
2024-04-11 17:01:55 +05:30
Vishesh 93f31828f5
Hypervisor as class (#8798)
* HypervisorType as a class

* Fixup

* fixup

* Add missing annotation

* Resolve comments

* Handle parallels typo

* Add missing @Convert annotation
2024-04-02 17:36:15 +05:30
Rohit Yadav 01d44347e7 Merge remote-tracking branch 'origin/4.19'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

 Conflicts:
	plugins/network-elements/juniper-contrail/pom.xml
	pom.xml
	tools/marvin/setup.py

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2024-03-21 18:38:01 +05:30
Marcus Sorensen 2e88eb45a3
Update mysql-connector version (#8753)
Co-authored-by: Marcus Sorensen <mls@apple.com>
2024-03-21 18:09:06 +05:30
Vishesh 24d5c45f7f
Add indexes for vm_stats (#8737)
* Add indexes for vm_stats

* Remove index on timestamp

* Chnage index from vm_id,timestamp to vm_id
2024-03-21 14:05:47 +05:30
Vishesh 33dc7465c2
Merge remote-tracking branch 'origin/4.19' 2024-03-19 15:18:02 +05:30
Vishesh 0043540fa3
Use join instead of views (#8321) 2024-03-18 18:08:19 +01:00
nvazquez db564b12b6
Merge branch '4.19' 2024-03-14 22:46:48 -03:00
Vishesh e87c6cfcb1
Fix resource count discrepancies (#8302)
* Fix resource count discrepancies

* Fixup while removing vm

* Fix discrepancies when starting VMs

* Fixup tests

* Fix failing tests

* Don't take lock when amount is negative

---------

Co-authored-by: dahn <daan@onecht.net>
2024-03-13 18:22:44 +05:30
Daan Hoogland d99b1b9c2d Merge branch '4.18' into 4.19 2024-03-08 08:19:49 +01:00
Pearl Dsilva 6dc3d06037
NSX integration (#7919)
* NSX integration - skeletal code

* Fix module not loading on startup

* add upgrade path and daos
\n add nsx controller command

* add support for adding and listing nsx provider to a zone

* add license

* add default VPC offering and update upgrade path

* add global setting to enable nsx plugin

* add delete nsx controller operation

* add nsxresource

* add NSX resource , api client, create tier1 gw

* update db

* update response and add license

* Add support to create and delete nsx tier-1 gateway

* add license

* cleanup and add skeletal code for network creation

* add create/delete segment and UI integration

* add license

* address code smells - part 1

* fix test / build failure

* NSX integration - skeletal code

* Fix module not loading on startup

* add upgrade path and daos
\n add nsx controller command

* add support for adding and listing nsx provider to a zone

* add license

* add default VPC offering and update upgrade path

* add global setting to enable nsx plugin

* add delete nsx controller operation

* add nsxresource

* add NSX resource , api client, create tier1 gw

* update db

* update response and add license

* Add support to create and delete nsx tier-1 gateway

* add license

* cleanup and add skeletal code for network creation

* add create/delete segment and UI integration

* add license

* address code smells - part 1

* fix test / build failure

* add ui changes + update nsx_provider table transport zones + use NSX broadcast domain for add nics to router

* ui: fix password field, and backend changes

* add route advertisement

* update offering

* update offering

* add sleep before deletion of vpc / tier g/w for ports to be removed

* move creation of segments to design phase

* change provider to VPC router for Dhcp & dns service in an nsx offering

* Add public nic for NSX

* reserve first IP (after g/w) of subnet for router nic - NSX

* revert reserving 1st IP in vpc segments

* [NSX] Create a DHCP relay and add it to a VPC tier segment (#107)

* Create DHCP relay command and execute request

* In progress integrate with networking

* Create DHCP relay config on the network VR allocation

* Revert domain router dao changes

* Create DHCP relay con VR nic plug to NSX network

* Link DHCP relay config to segment after creation

* [NSX] Cleanup DHCP Relay config on segment deletion (#108)

* Cleanup DHCP Relay config on segment deletion

* update segment & relay name generators and call delete dhcprelay after deletion of segment

* address comment

* [NSX] Fix DHCP relay config deletion was missing zone name (#8068)

* [NSX] Refactor API wrapper operations (#8059)

* [NSX] Refactor API wrapper operations

* Big refactor

* Address review comment

* change network cidr to cidr to prevent NPE

* add domain and zone names to the various networks - vpc & tier

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* Nsx unit tests (#8090)

* Add tests

* add test for NsxGuestNetworkGuru

* add unit tests for NsxResource

* add unti tests for NsxElement

* cleanup

* [NSX] Refactor API wrapper operations

* update tests

* update tests - add nsxProviderServiceImpl test

* add unit test - NsxServiceImpl

* add license

* Big refactor

* Address review comment

* change network cidr to cidr to prevent NPE

* add domain and zone names to the various networks - vpc & tier

* fix tests

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* modify NSX resource naming convention (#8095)

* modify NSX resource naming convention

* remove unused imports

* add a setup phase between desgin and implementation of a network for intermediary steps

* add method to all classes

* NSX: Refactor Network & VPC offering (#8110)

* [NSX] Refactor API wrapper operations

* Network offering changes for NSX

* fix services and provider combination

* address comments: rename param

* update nsx_mode parameter

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* fix test

* [NSX] Allow NSX isolated networks (#8132)

* Add network offerings for NSX on isolated networks

* Fix offerings creation

* In progress NSX isolated network

* Fixes

* Fix NIC allocation to router

* NSX: Add Step for Adding Public traffic network for NSX During zone creation (#8126)

* NSX: Add Step for Adding Public traffic network for NSX

* address comments and cleanup

* address comment

* remove indent

* NSX: Create and Delete static NAT & Port forward  rules (#8131)

* NSX: Create and delete NSX Static Nat rules

* fix issues with static nat

* add static nat

* Support to add and delete Port forward rules

* add license

* fix adding multiple pf rules

* cleanup

* fix lint check

* fix smoke tests

* fix smoke tests

* Nsx add lb rule (#8161)

* NSX: Create and delete NSX Static Nat rules

* fix issues with static nat

* add static nat

* Support to add and delete Port forward rules

* add license

* fix adding multiple pf rules

* cleanup

* NSX: Add support to create and delete Load balancer rules

* fix deletion of lb rules

* add header file and update protocol detail

* build failure fix

* [NSX] Add SNAT support (#8100)

* In progress add source NAT

* Fix after merge

* Fix tests

* Fix NPE on isolated network deletion

* Reserve source NAT IP when its not passed for NSX VPC

* Create source NAT rule on VR NIC allocation

* Fix update VPC and remove VPC to update and remove SNAT rule

* Fix packaging

* Address review comment

* Fix build

* fix build - unused import

* Add defensive checks

* Add missing design to NSX public guru

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* NSX: Fix VR public NIC allocation (#8166)

* NSX: fix LB member addition and deletion and add defensive checks (#8167)

* Fix public NIC NPE on broadcast URI

* NSX: Router Public nic to get IP from systemVM Ip range (#8172)

* NSX: Router Public nic to get IP from systemVM Ip range

* Fix VR IP address and setSourceNatIp command

* NSX: hide systemVM reserved IP range SourceNAT

* fix test

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* fix test failure

* test failure fix

* [NSX] Fix update source NAT IP (#8176)

* [NSX] Fix update source NAT IP

* Fix startup

* Fix API result

* NSX - add LB route Advertizement (#8192)

* [NSX] Add ACL types support (#8224)

* NSX: Create segment group on segment creation

* Add unit tests

* Remove group for segment before removing segment

* Create Distributed Firewall rules

* Remove distributed firewall policy on segment deletion

* Fix policy rule ID and add more unit tests

* Fix DROP action rules and transform tests

* Add new ACL rules

* Fixes

* associate security policies with groups and not to DFW and add deletion of rules

* Fix name convention

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* NSX: Fix creation of VPCs (#8320)

* Fix ACL rules creation (#8323)

* [NSX] Fix database views (#8325)

* NSX: Add CKS Support & Firewall rules for Isolated Networks (#8189)

* NSX: Add ALL LB IP to the list of route advertisements in tier1

* NSX: Support Source NAT on NSX Isolated networks

* NSX: Cks Support

* NSX: Create segment group on segment creation

* Add unit tests

* Remove group for segment before removing segment

* Create Distributed Firewall rules

* Remove distributed firewall policy on segment deletion

* Fix policy rule ID and add more unit tests

* Add support for routed NSX Isolated networks \n and non RFC 1918 compliant IPs

* Add support for routed NSX Isolated networks \n and non RFC 1918 compliant IPs

* Add Firewall rules

* build failure - fix unit test

* fix npes

* Add support to delete firewall rules

* update nsx cks offering

* add license

* update order of ports in PF & FW rules

* fix filter for getting transport zones

* CKS support changed - MTU updated, etc

* add LB for CKS on VPC

* address comments

* adapt upstream cks logic for vpc

* rever mtu hack

* update UI changes as per upstream fix

* change display test for CKS n/w offerings for isolated and VPC tiers

* add extra line for linter

* address comment

* revert list change

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* fix ui build failure

* [NSX] Address SonarCloud Bugs (#8341)

* [NSX] Address SonarCloud Bugs

* Fix NSX API connection issues

* NSX: Add unit tests to increase coverage (#8355)

* NSX: Add unit tests

* cleanup unused imports

* add more unit tests

* add tests for publicnsxnetworkguru

* add license

* fix build failures

* address sonar comment

* fix security hotspots

* NSX: Add more unit tests (#8381)

* NSX : Unit tests

* remove unused imports

* remove unused import causing build failure

* fix build failures due to unused imports

* fix build failure

* fix test assertion

* remove unused imports

* remove unused import

* Nsx UI zone bug (#8398)

* NSX: Attempt to fix NSX Zone creation bug for public networks

* fix zone wizard public traffic issue

* add proper filtering of offerings based on VPC nsx mode

* clean up console logs

* NSX: Fix code smells and reported bugs (#8409)

* NSX: Fix code smells and reported bugs

* fox override issue

* remove unused imports

* fix test

* refactor code to reduce complexity

* add lisence

* cleanup

* fix build failure

* fix build failure

* address comments

* test - add config to ignore certain files from test coverage

* test exclusion of classes from test cov

* rever pom changes

* [NSX] Add more unit tests (#8431)

* [NSX] Add more unit tests

* More tests

* Fix build errors

* NSX: Prevent creation of L2 and Shared networks for NSX (#8463)

* NSX: Prevent creation of L2 and Shared networks for NSX

* add checks to backend to prevent creation of l2 and shared networks in nsx zones and filter only nsx offerings when creating isolated networks

* cleanup

* NSX: Fix code smells (#8436)

* NSX: Fix code smells

* Add changes to service creation logic

* CKS: Add action to during firewall rule creation (#8498)

* NSX,UI: Deduplicate network list when creating kubernetes clusters (#8513)

* NSX: Make LB service selectable in network offering (#8512)

* NSX: Make LB service selectable in network offering

* fix label

* address comments

* address comments

* NSX: Add appropriate error message when icmp type is set to -1 for NSX (#8504)

* NSX: Add appropriate error message when icmp type is set to -1 for NSX

* address comments

* update text

* fix test

* fix test - build failure

* fix test - build failure

* NSX: Cleanup NSX resources during k8s cluster cleanup (#8528)

* fix test failure

* NSX: Improve segment deletion process (#8538)

* NSX: Add passive monitor for NSX LB to test whether a server is available (#8533)

* NSX: Add passive monitor for NSX LB to test whether a server is available

* Add active monitors too

* fix build failure

* NSX: Add check for ICMP code / type for NSX zones (#8542)

* NSX: Fix Routed Mode for Isolated and VPC networks (#8534)

* NSX: Fix Routed Mode for Isolated and VPC networks

* NSX: Fix Routed mode - add checks for ports added for FW rules

* clean up code

* fix build failure

* NSX: Add retry logic with sleep to delete segments (#8554)

* NSX: Add retry logic with sleep to delete segments

* add logs

* NSX: Fix custom ACL check (#2)

* NSX: Fix custom ACL check

* NSX: Fix custom ACL check

* Nsx vpc routed mode (#5)

* NSX: Fix VPC routed mode

* NSX: VPC route mode

* remove unnecessary changes

* Nsx: Support internal LB (#4)

* NSX: Support internal LB service in NSX

* add lb removal logic

* Fix UI issue hiding internal LB tab

* Refactor method name

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* NSX: Improve NSX resource cleanup process (#3)

* Fix unit test

* NSX: Add SourceNAT service to the default Routed offering for VPC (#13)

* Fix VPC restart with cleanup (#12)

* NSX: Fix ACL rule removal on replacement and fix rule order (#11)

* NSX: fix smoke test failure for ACLs (#9)

* Fix unit tests

* Fix NSX plugin pom XML

* NSX: Add support to re-order ACL rules (NSX FW rules) (#14)

* [WIP] NSX: Add support to re-order ACL rules (NSX FW rules)

* fix reordering of acl rules on all networks that it is associated to

* clean up and attempt test fix

* Fix tests

* Remove unused import

* tweak reorder logic

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix zone creation issue for internal load balancer

* Fix

* Fix unit test

* fix logger

* fix logger

* fix logger

* NSX: Fix VPC form to ignore source NAT IP when creating VPCs and fix label

* Move SQL changes to the newest schema file

* NSX: Last Fixes

* Fix build

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>
2024-03-07 23:34:10 -03:00
Bryan Lima b82ea3de2d
Update pre-populated quota tariffs' unit types (#8591) 2024-03-06 17:36:22 +01:00
Wei Zhou 986d754768
SG: fix SQL exception when add security group rule with allowed account/group (#8616)
This fixes https://github.com/apache/cloudstack/issues/8595

```
2024-02-01 16:23:52,473 INFO  [c.c.n.s.SecurityGroupManagerImpl] (AgentManager-Handler-16:null) (logid:) Network Group full sync for agent 1 found 3 vms out of sync
2024-02-01 16:23:52,473 DEBUG [c.c.n.s.SecurityGroupManagerImpl] (AgentManager-Handler-16:null) (logid:) Security Group Mgr v2: scheduling ruleset updates for 3 vms  (unique=3), current queue size=0
2024-02-01 16:23:52,473 DEBUG [c.c.n.s.SecurityGroupManagerImpl] (AgentManager-Handler-16:null) (logid:) Security Group Mgr v2: done scheduling ruleset updates for 3 vms: num new jobs=3 num rows insert or updated=0 time taken=0
2024-02-01 16:23:52,478 ERROR [c.c.n.s.SecurityGroupManagerImpl] (SecGrp-Worker-20:ctx-0aa3885d) (logid:472b30d2) Problem during SG work com.cloud.network.security.LocalSecurityGroupWorkQueue$LocalSecurityGroupWork@5
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT SQL_CACHE security_group_vm_map.id, security_group_vm_map.security_group_id, security_group_vm_map.instance_id, nics.ip4_address, vm_instance.state, security_group.name FROM security_group_vm_map INNER JOIN nics ON security_group_vm_map.instance_id=nics.instance_id  INNER JOIN vm_instance ON security_group_vm_map.instance_id=vm_instance.id  INNER JOIN security_group ON security_group_vm_map.security_group_id=security_group.id  WHERE security_group_vm_map.security_group_id = 3  AND vm_instance.state='Running'
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:424)
        at com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:938)
        at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:928)
        at com.cloud.network.security.dao.SecurityGroupVMMapDaoImpl.listBySecurityGroup(SecurityGroupVMMapDaoImpl.java:134)
        at jdk.internal.reflect.GeneratedMethodAccessor555.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy245.listBySecurityGroup(Unknown Source)
        at com.cloud.network.security.SecurityGroupManagerImpl2.generateRulesForVM(SecurityGroupManagerImpl2.java:246)
        at com.cloud.network.security.SecurityGroupManagerImpl2.sendRulesetUpdates(SecurityGroupManagerImpl2.java:177)
        at com.cloud.network.security.SecurityGroupManagerImpl2.work(SecurityGroupManagerImpl2.java:157)
        at com.cloud.network.security.SecurityGroupManagerImpl2$WorkerThread$1.run(SecurityGroupManagerImpl2.java:75)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.network.security.SecurityGroupManagerImpl2$WorkerThread.run(SecurityGroupManagerImpl2.java:72)
Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.id, security_group_vm_map.security_group_id, security_group_vm_map.instance_id,' at line 1
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
        ... 28 more
```
2024-03-06 08:25:00 +01:00
Fabricio Duarte 9a73a2f9e1
Display dates in Quota and Usage messages according to the timezone configurations (#8230)
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2024-03-04 12:44:03 +01:00
Abhishek Kumar 0926e5c15a Merge remote-tracking branch 'apache/4.19' 2024-02-26 12:23:34 +05:30
Vishesh 1a1131154e
Fixup vm powerstate update (#8545)
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2024-02-19 13:56:21 +01:00
Abhishek Kumar 592038a304
api,server,ui: granular resource limit management (#8362)
Feature spec: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Granular+Resource+Limit+Management

Introduces the concept of tagged resource limits for granular resource limit management. Limits can be enforced on accounts and domains for the deployment of entities for a tagged resource. Current tagged resource limits can be used for the following resource types,

Host limits
- user_vm
- cpu
- memory

Storage limits
- volume
- primary_storage

Following global settings can used to specify tags for which limit needs to be enforced,

Host: `resource.limit.host.tags`
Storage: `resource.limit.storage.tags`

Option for specifying tagged resource limits and viewing tagged resource usage are made available in the UI.

Enhances the use of templatetag for VM deployment and template creation

Adds option to list service/compute offerings that can be used with a given template. A new parameter named templateid has been added.

Adds option to list disk offering with suitability flag for a virtual machine. A new parameter named virtualmachineid has been added to the listDiskOfferings API which when passed returns suitableforvirtualmachine param in the response.
2024-02-19 14:17:34 +05:30
Wei Zhou 6af1c25f52 Merge remote-tracking branch 'apache/4.19' 2024-02-17 12:30:40 +01:00
dahn a0e592e945
prevent nic removal on out of bounds router stop (#8371)
Co-authored-by: Vishesh <vishesh92@gmail.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
2024-02-16 14:33:22 +01:00
João Jandre 49cecaed06
Normalize loggers and upgrade log4j 1.2 to log4j 2.19 (#7131)
* Normalize logs

All classes that could have their loggers inherited from their fathers had their own loggers deleted;
Most loggers didn't have to be static, so most of them were normalized so that they wouldn't be;
All loggers are protected now;
Static logger's name are now 'LOGGER';
Non-static logger's name are now 'logger';
New class DbUpgradeAbstractImpl created so that all Upgraders extend it and inherit its logger

* Upgrade log4j

* fix errors caused by the merge

* Refactor cglibThrowableRenderer functionality to log4j2 and upgrade the last configuration files

* fix sonarcloud bug

* Fix errors caused by merge, remove some unused loggers, and rename a variable that was mistakenly renamed on the normalization commit

* Readd snmpTrapAppender, remove TestAppender

* Regenerate changes

* regenerate changes

* refactor last custom appender

* fix systemvm configuration xml

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* Fix utils pom

* fix some tests

* regenerate changes

* Fix jar being printed on exception

* fix logging in system VMs, fix commands not having log4j2 classpath.

* regenerate changes

* Fix some unwanted renomeations

* fix end of file

* regenerate changes

* regenerate changes

* fix merge error

* regenerate changes

* fix tests

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* readd reload4j to tungsten as juniper depends on it

* Regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* re-add reload4j dependency to network-contrail, as juniper depends on it

* regenerate changes

* regenerate changes

* regenerate changes

* fix typo

* regenerate changes

* regenerate changes

* Fix end of files

* regenerate changes

* add logj42 to cloud-utils-SHADED.jar

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* Regenerate changes

* regenerate changes

* Regenerate changes

* Regenerate changes

* fix some tests

* Regenerate changes

* Regenerate changes

* fix test

* Regenerate changes

* Regenerate changes
2024-02-08 09:55:41 -03:00
Suresh Kumar Anaparti 8ea9fc911d
StoragePoolType as class (#8544)
* StoragePoolType as a class

* Fix agent side StoragePoolType enum to class

* Handle StoragePoolType for StoragePoolJoinVO

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fix UserVMJoinVO for StoragePoolType

* fixed missing imports

* Since StoragePoolType is a class, it cannot be converted by @Enumerated annotation.
Implemented conveter class and logic to utilize @Convert annotation.

* Fixed equals for the enum.

* removed not needed try/catch for prepareAttribute

* Added license to the file.

* Implemented "supportsPhysicalDiskCopy" for storage adaptor.

Co-authored-by: mprokopchuk <mprokopchuk@apple.com>

* Add javadoc to StoragePoolType class

* Add unit test for StoragePoolType comparisons

* StoragePoolType "==" and ".equals()" fix.

* Fix StoragePoolType for FiberChannelAdapter

* Fix for abstract storage adaptor set up issue

* review comments

* Pass StoragePoolType object for poolType dao attribute

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
Co-authored-by: mprokopchuk <mprokopchuk@gmail.com>
2024-02-05 13:27:15 +05:30
Abhishek Kumar d353fcc599
engine-schema: upgrade path 4.19.0.0 to 4.20.0.0 (#8608)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-02-05 13:20:20 +05:30
Wei Zhou 33bb92acce
Veeam: Support Veeam 11 and 12 (#8241)
This PR fixes several issues in the testing of Veeam 11 and Veeam12
- Import Veeam.Backup.PowerShell and silently ignore the warning messages
- Fix issue when assign vm to backup offerings, which caused by separator (\r\n)
- Fix authorization failure in veeam 12a, which is because v1_4 is not supported in veeam 12a any more
- Fix exception if backup name has space
- Fix backup metrics in veeam12, which is because powershell command does not return the values needed
- Fix Incorrect datetime value, which is because powershell command returns a datetime which is not supported in Java
- Fix issue during backup restoration if VM has both ROOT and DATA disks.

This PR also has the following update
- Add integration test test/integration/smoke/test_backup_recovery_veeam.py
- Make some UI changes
- Add zone setting backup.plugin.veeam.version. If it is not set, CloudStack will get veeam version via powershell commands.
- Add zone setting backup.plugin.veeam.task.poll.interval and backup.plugin.veeam.task.poll.max.retry
2024-01-19 18:42:01 +01:00
Suresh Kumar Anaparti e87ce0c723
Fix reorder/list pools when cluster details are not set, while deploying vm / attaching volume (#8373)
This PR fixes reorder/list pools when cluster details are not set, while deploying vm / attaching volume.

Problem:
Attach volume to a VM fails, on infra with zone-wide pools & vm.allocation.algorithm=userdispersing as the cluster details are not set (passed as null) while reordering / listing pools by volumes.

Solution:
Ignore cluster details when not set, while reordering / listing pools by volumes.
2024-01-10 18:13:32 +05:30
Nicolas Vazquez b8d3e342be
Fix KVM import unmanaged instances on basic zone (#8465)
This PR fixes import unmanaged instances on KVM basic zones, on top of #8433

Fixes: #8439: point 1
2024-01-10 13:21:00 +05:30
Abhishek Kumar 514d2c2a26
schema,engine-schema: explicit VMware 8.0.0.2, 8.0.0.3 support, logs (#8444)
Fixes #8412

Add support for 8.0.0.2 explicitly to prevent falling over to the parent version
Adds log when hypervisor capabilities fail over to the parent version

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-01-09 14:43:04 +05:30
Abhishek Kumar a68dc0bfb5 Merge remote-tracking branch 'apache/4.18' 2024-01-08 18:01:13 +05:30
Abhishek Kumar f023fc53c0
engine-schema: fix finding guestos mapping with parent version (#8426)
GuestOS mappings are retrieved from the parent hypervisor version when a minor, patch hypervisor version doesn't exist.

Fixes #8412

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2024-01-08 17:51:04 +05:30
Fabricio Duarte 16d45f731d
Save the account which deliberately removed a public IP from quarantine (#8339)
When a public IP gets removed from quarantine, the removal reason gets saved to the database; however, it may also be useful for operators to know who removed the public IP from quarantine. For that reason, this PR extends the public IP quarantine feature so that the account that deliberately removed an IP from quarantine also gets saved to the database.
2023-12-18 12:06:31 +05:30
kishankavala ab20b1220f
KVM Ingestion - Import Instance (#7976)
This PR adds new functionality to import KVM instances from an external host or from disk images in local or shared storage.
Doc PR: https://github.com/apache/cloudstack-documentation/pull/356
2023-12-14 13:08:56 +05:30
Vishesh a791d46abe
events,alerts: Add missing indexes (#8276)
This PR adds missing indexes on `alerts` & `events` tables.

For alerts table, some of the queries are part of a couple of APIs and some operations. I have added the index for the same. Ref: 
8f39087377/engine/schema/src/main/java/com/cloud/alert/dao/AlertDaoImpl.java (L40-L45)

For Events table, we query for `resource_id` & `resource_type` in the UI for a resource's events. Indexes were missing, so I have added those.
2023-12-13 10:24:36 +05:30
João Jandre 12dcf5cc6f
Move subdomains between domains (#7446)
Sometimes users have the need to move resources between domains, for example, in a big company, a department may be moved from one part of the company to another, changing the company's department hierarchy, the easiest way of reflecting this change on the company's cloud environment would be to move subdomains between domains, but currently ACS offers no option to do that.
This PR adds the moveDomain API, which will move domains between subdomains. Furthermore, if the domain that is being moved has any subdomains, those will also be moved, maintaining the current subdomain tree.
2023-12-10 12:18:15 +05:30
Nicolas Vazquez 371ad9f55b
New Feature: Import VMware VMs into KVM (#7881)
This PR adds the capability in CloudStack to convert VMware Instances disk(s) to KVM using virt-v2v and import them as CloudStack instances. It enables CloudStack operators to import VMware instances from vSphere into a KVM cluster managed by CloudStack. vSphere/VMware setup might be managed by CloudStack or be a standalone setup.

    CloudStack will let the administrator select a VM from an existing VMware vCenter in the CloudStack environment or external vCenter requesting vCenter IP, Datacenter name and credentials.
    The migrated VM will be imported as a KVM instance
    The migration is done through virt-v2v: https://access.redhat.com/articles/1351473, https://www.ovirt.org/develop/release-management/features/virt/virt-v2v-integration.html
    The migration process timeout can be set by the setting convert.instance.process.timeout
    Before attempting the virt-v2v migration, CloudStack will create a clone of the source VM on VMware. The clone VM will be removed after the registration process finishes.
    CloudStack will delegate the migration action to a KVM host and the host will attempt to migrate the VM invoking virt-v2v. In case the guest OS is not supported then CloudStack will handle the error operation as a failure
    The migration process using virt-v2v may not be a fast process
    CloudStack will not perform any check about the guest OS compatibility for the virt-v2v library as indicated on: https://access.redhat.com/articles/1351473.
2023-12-07 12:59:56 +05:30
Daan Hoogland d3cad4266a Merge release branch 4.18 to main
* 4.18:
  server: Initial new vpnuser state (#8268)
  UI: Removed redundant IP Address Column when create Port forwarding rules (#8275)
  UI: Removed ICMP input fields for protocol number from ACL List rules modal (#8253)
  server: check if there are active nics before network GC (#8204)
2023-12-04 10:44:43 +01:00
kishankavala 5651eab49c
ObjectStore Framework with MinIO and Simulator plugins (#7752)
This PR adds Object Storage feature to CloudStack.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+CloudStack+Object+Store
2023-12-01 17:51:00 +05:30
Bryan Lima cb62ce6767
Global ACL for VPCs (#7150) 2023-11-30 14:51:43 +01:00
João Jandre 26b01f6f3b
Flexible tags for hosts and storage pools (#7489)
Co-authored-by: João Jandre <joao@scclouds.com.br>
2023-11-30 09:36:47 +01:00
Wei Zhou cb2b6aca45
server: check if there are active nics before network GC (#8204) 2023-11-29 18:55:26 +01:00
Daniel Augusto Veronezi Salvador adbb5520ca
Database VIEW management via separated files (#7417)
Co-authored-by: Daniel Augusto Veronezi Salvador <gutoveronezi@apache.org>
2023-11-17 10:27:53 +01:00
Bryan Lima 1f29f6f040
Public IP quarantine feature (#7378) 2023-11-15 10:29:22 +01:00
John Bampton f090c77f41
misc: fix spelling (#7549)
Co-authored-by: Stephan Krug <stekrug@icloud.com>
2023-11-02 09:23:53 +01:00
Vishesh 5362bad442
Storage Management (#7949) 2023-11-01 10:46:22 +01:00
Harikrishna 235e4fe190
Oauth2 integration with CloudStack (#7996)
OAuth2, the industry-standard authorization or authentication framework, simplifies the process of
granting access to resources. CloudStack supports OAuth2 authentication wherein users can login into
CloudStack without using a username and password. Support for Google and Github providers has been added.
Other OAuth2 providers can be easily integrated with CloudStack using its plugin framework.

The login page will show provider options when the OAuth2 is enabled and corresponding providers are configured.

"OAuth configuration" sub-section is present under "Configuration" where admins can register the corresponding
OAuth providers.
2023-10-31 13:25:28 +05:30
Wei Zhou bd52fa8a12
New feature: VNF templates and appliances integration (#8022) 2023-10-27 10:23:00 +02:00
Abhishek Kumar 543c54c718
api,server,ui: snapshot copy, multi-zone replica (#7873)
This PR adds new functionality to copy snapshots across zones and take snapshots for multiple zones.

Copy functionality is similar to template copy. The source zone acts as the web server from where the destination zone(s) can download the snapshot files. For this purpose, a new API - `copySnapshot` has been added. The response for copySnapshot will be returning zone and download details from the first destination zone of the request. This behaviour is similar to the `copyTemplate` API.

In a similar manner, multiple zones can be selected while taking the snapshots or creating snapshot policies. For this snapshot will be taken in the base zone(in which volume is present) and then copied to the additional zones. A new parameter - `zoneids` has been added to `createSnapshot` and `createSnapshotPolicy` APIs.

As snapshots can be present on multiple zones (secondary stores), a new parameter `zoneid` has been added to delete the snapshot copy on a specific zone.

`listSnapshots` API has been updated to allow listing snapshot entries for different zones/datastores. New parameters - `showUnique`, `locationType` have been added.

Events generated during snapshot operations will now be linked to the snapshot itself rather than the volume of the snapshot.

`listSnapshotPolicies` and `createSnapshotPolicy` APIs will return zone details of the zones in which backup will be scheduled for the policy.

----
New API added
`copySnapshot`

Request and response params updated for APIs
```
- listSnapshots
- deleteSnapshot
- createTemplate
- listZones
- listSnapshotPolicies
- createSnapshotPolicy
```
UI updated for
- Snapshot detail view
- Create snapshot form
- Create snapshot policy form
- Create volume (from snapshot) form
- Create template (from snapshot) form

Doc PR: https://github.com/apache/cloudstack-documentation/pull/344
PR: https://github.com/apache/cloudstack/pull/7873
2023-10-23 09:01:58 +02:00
Abhishek Kumar 20046ffe61 Merge remote-tracking branch 'apache/4.18' into main 2023-10-19 10:39:40 +05:30
Harikrishna 0183e25279
Fix VM snapshot size during storage capacity check (#8101) 2023-10-18 11:49:26 +02:00
Abhishek Kumar 9df580cef4 Merge remote-tracking branch 'apache/4.18' into main 2023-10-12 16:54:19 +05:30
Abhishek Kumar 3e7f21a190
vm-import: fix stopped managed vms listing in unmanaged instances (#7606)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-12 09:21:47 +02:00
Bryan Lima ae5dda867f
Normalize encryption on global configurations values (#6812) 2023-09-28 16:44:36 +02:00
sato03 8d81ec425d
keep deleted projects name (#7858)
Co-authored-by: Henrique Sato <henrique.sato@scclouds.com.br>
2023-09-28 09:15:32 +02:00
Daan Hoogland 0375714ded Merge release branch 4.18 to main
* 4.18:
  ldap trust map cleanup on domain delete (#7915)
  upgrade: fix upgrade from 4.18.1.0 to 4.18.2.0-SNAPSHOT (#7959)
2023-09-19 11:20:58 +02:00
Wei Zhou 9c9b17885f
upgrade: fix upgrade from 4.18.1.0 to 4.18.2.0-SNAPSHOT (#7959)
The uprgade from 4.18.1.0 to 4.18.2.0-SNAPSHOT failed with error

```
2023-09-12 16:12:19,003 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.18.1.0 Code Version = 4.18.2.0
2023-09-12 16:12:19,004 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.18.1.0 to 4.18.2.0
2023-09-12 16:12:19,036 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Running upgrade Upgrade41800to41810 to upgrade from 4.18.0.0-4.18.1.0 to 4.18.1.0
...
2023-09-12 16:12:19,041 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- Schema upgrade from 4.18.0.0 to 4.18.1.0
...
2023-09-12 16:12:21,602 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Statement: CREATE INDEX i_cluster_details__name on cluster_details (name)
2023-09-12 16:12:21,663 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Created index i_cluster_details__name
2023-09-12 16:12:21,673 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 2632 Name =  Upgrade; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:97-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:215-$Proxy30.persist:-1-DatabaseUpgradeChecker.upgrade:319-DatabaseUpgradeChecker.check:403-CloudStackExtendedLifeCycle.checkIntegrity:64
```

It succeeded with this change.
2023-09-18 19:14:36 +02:00
John Bampton 4eb110af73
Remove unneeded duplicate words (#7850) 2023-09-18 13:16:33 +02:00
Vishesh 710f34ee90
api: Fix listLoadBalancerRuleInstances api call (#7892) 2023-09-09 07:45:42 +02:00
Wei Zhou f6b2a58727 Merge branch '4.18' 2023-09-07 08:56:35 +02:00
Marcus Sorensen 2cccd8f754
db: Add index on cluster_details.name for FirstFitPlanner speedup (#7922) 2023-08-31 21:43:23 +02:00
GaOrtiga 819dd7b75c
server: remove supportedOwner from Resource.ResourceType (#7416) 2023-08-30 11:29:16 +02:00
Rohit Yadav 7362e5338c Merge remote-tracking branch 'origin/4.18' 2023-08-28 12:54:36 +05:30
Wei Zhou 0dd6bb7c67
systemvm: update size and state of systemvm template during registration (#7874) 2023-08-25 16:46:32 +02:00
Daan Hoogland 27f148e79e Merge branch '4.18' 2023-08-23 11:15:56 +02:00
Wei Zhou e4117e68d2
engine/schema: fix duplicated guest OSes in 4.18.0.0 (#7799)
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-08-23 09:44:34 +02:00
Daan Hoogland ea832bce13 Merge branch '4.18' 2023-08-22 11:44:45 +02:00
Wei Zhou 78bdde9e98
AutoScaling: support Managed User Data (#7769) 2023-08-22 11:07:16 +02:00
João Jandre 1065e9046b
Fix backup dates (#6473)
Co-authored-by: João Paraquetti <joao@scclouds.com.br>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-18 10:51:36 +02:00
Wei Zhou c8d6e50539
VMware: add support for 8.0b (8.0.0.2), 8.0c (8.0.0.3) (#7380)
* VMware: add support for 8.0b (8.0.0.2)

* VMware 8: add new guest os mappings in VirtualMachineGuestOsIdentifier

The full list can be found at https://developer.vmware.com/apis/1355/vsphere

* VMware: get guest os mappings of parent version

* VMware8: remove guest os mappings for 8.0.0.2

* VMware8: fix code smells

* vmware: remove annotations in VmwareVmImplementerTest which caused 0.0% code coverage

* VMware8: add a unit test case

* VMware: add support for 8.0c (8.0.0.3)

* VMware8: move to CloudStackVersion.getVMwareParentVersion

* VMware: add support for 8.0u1 (8.0.1.0)

* Copy engine/schema/src/main/java/com/cloud/upgrade/GuestOsMapper.java from PR 6979

* Copy engine/schema/src/main/java/com/cloud/storage/dao/GuestOSHypervisorDao.java from PR 6979

* VMware: ignore the last number in VMware versions

* VMware: copy guest os mapping from 8.0 to 8.0.1

* VMware: add unit tests in VmwareVmImplementerTest.java

* Copy engine/schema/src/test/java/com/cloud/upgrade/GuestOsMapperTest.java from PR 6979

* VMware8: retry vm poweron if fails due to exception "File system specific implementation of Ioctl[file] failed"

This fixes a weird issue on vmware8. When power on a vm, sometimes it fails due to error

2023-04-27 07:04:43,207 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-442:ctx-cdd42b03 10.0.32.133, job-105/job-106, cmd: StartCommand) (logid:8a24a607) StartCommand failed due to [Exception: java.lang.RuntimeException
Message: File system specific implementation of Ioctl[file] failed
].
java.lang.RuntimeException: File system specific implementation of Ioctl[file] failed
        at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:426)
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.powerOn(VirtualMachineMO.java:288)

in vmware.log on ESXi host, it shows

2023-04-27T09:20:41.713Z In(05)+ vmx - Power on failure messages: File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to lock the file
2023-04-27T09:20:41.713Z In(05)+ vmx - Cannot open the disk '/vmfs/volumes/7b29c876-ac102328/i-2-167-VM/ROOT-167.vmdk' or one of the snapshot disks it depends on.
2023-04-27T09:20:41.713Z In(05)+ vmx - Module 'Disk' power on failed.
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to start the virtual machine.

There is a KB article for it, but I still do not know why and how to fix it.
https://kb.vmware.com/s/article/1004232

* VMware: extract to method powerOnVM

* vmware: fix mistake in logs

* vmware8: use curl instead of wget to fix test failures

Traceback (most recent call last):
  File "/root/test_internal_lb.py", line 555, in test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
    self.execute_internallb_roundrobin_tests(vpc_offering)
  File "/root/test_internal_lb.py", line 641, in execute_internallb_roundrobin_tests
    client_vm, applb.sourceipaddress, max_http_requests)
  File "/root/test_internal_lb.py", line 497, in run_ssh_test_accross_hosts
    (e, clienthost.public_ip))
AssertionError: list index out of range: SSH failed for VM with IP Address: 10.0.52.187

and

sshClient: DEBUG: {Cmd: /usr/bin/wget -T3 -qO- --user=admin --password=password http://10.1.2.253:8081/admin?stats via Host: 10.0.52.188} {returns: ["/usr/bin/wget: '/usr/lib/libpcre.so.1' is not an ELF file", "/usr/bin/wget: can't load library 'libpcre.so.1'"]}

* VMware: correct guest OS names in hypervisor mappings for VMware 8.0

el9 and variants were introduced by https://github.com/apache/cloudstack/pull/7059
they are supported with guest os identifiers since VMware 8.0

see https://vdc-repo.vmware.com/vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/vim.vm.GuestOsDescriptor.GuestOsIdentifier.html

* VMware: add Ubuntu 20.04 and 22.04 support for vmware 7.0+

* PR7380: only add guest os mappings for Ubuntu 20.04

* PR7380: Correct RHEL9 guest os names and others for VMware 8.0

* PR7380: correct guest os names on 8.0.0.1 as well

* PR7380: remove Windows 12 and Windows Server 2025 which are not released yet
2023-08-17 10:42:42 +02:00
Nicolas Vazquez 8b5ba13b81
plugins: Add Custom hypervisor minimal changes (#7692)
### Description

Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

##### Proposed Future work:
Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

#### Feature Specifications
- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.


#### Use cases
This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts
2023-08-16 20:53:24 +05:30
Daan Hoogland 7b64236469 Merge release branch 4.18 to main
* 4.18:
  server: remove registered userdata when cleanup an account (#7777)
  server: Use max secondary storage defined on the account during upload  (#7441)
  test: upgrade kubernetes versions to 1.25.0/1.26.0 (#7685)
  kvm: Added VNI Devices as normal bridge slave devs (#7836)
  noVNC: fix JP keyboard on vmware7+ which uses websocket URL (#7694)
2023-08-10 14:50:46 +02:00
Wei Zhou 1605b2f0db
server: remove registered userdata when cleanup an account (#7777) 2023-08-10 14:33:59 +02:00
Daan Hoogland 7ee697fcc5 Merge release branch 4.18 to main
* 4.18:
  UI: allow new keys for VM details (#7793)
  Refactoring StorPool's smoke tests (#7392)
  UI: decode userdata in EditVM dialog (#7796)
  packaging: unalias cp before package upgrade (#7722)
  make NoopDbUpgrade do a systemvm template check (#7564)
  UI unit test: fix expected values (#7792)
2023-08-03 15:59:01 +02:00
dahn 2bb079189c
make NoopDbUpgrade do a systemvm template check (#7564) 2023-07-31 16:08:14 +02:00
GaOrtiga 0bab0db881
Introducing concept of domain VPCs (#7153)
Co-authored-by: Gabriel Ortiga Fernandes <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Lopez <rodrigo@scclouds.com.br>
2023-07-29 20:06:49 +02:00
Daan Hoogland 6bb95c0200 Merge release branch 4.18 to main
* 4.18:
  Storage and volumes statistics tasks for StorPool primary storage (#7404)
  proper storage construction (#6797)
  guarantee MAC uniqueness (#7634)
  server: allow migration of all VMs with local storage on KVM (#7656)
  Add L2 networks to Zones with SG (#7719)
2023-07-19 10:59:19 +02:00
dahn 73a269e3b3
guarantee MAC uniqueness (#7634)
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
2023-07-19 10:25:01 +02:00
Vishesh 594c70dde0
Sync precommit config from main (#7732)
Co-authored-by: John Bampton <jbampton@users.noreply.github.com>
Co-authored-by: dahn <daan@onecht.net>
2023-07-07 11:18:16 +02:00
Daan Hoogland 2132f46fcb Merge branch '4.18' 2023-07-06 11:24:08 +02:00
Harikrishna 0cbe77024a
Fix foreign key constraints and the mysql procedure that is used (#7381) 2023-07-06 08:43:49 +02:00
SadiJr 3c5fdeafdc
[Veeam] restored VMs without NICs (#6282) 2023-07-03 14:35:48 +02:00
Wei Zhou 41403c9cf9 Merge remote-tracking branch 'apache/4.18' 2023-06-27 11:52:37 +02:00
dahn 06caf32bc8
Guest os mappings improvements (#6979)
* Guest OS mapping improvements
- Checks the OS mapping name in hypervisor (VMware, XenServer)
- Displays guest OS mappings in UI

* Added API getHypervisorGuestOsNames to list the guest OS names in the hypervisor, and code improvements

* Some static analysis fixes

* Removed commented code in listview

* Guest OS list

* UI changes for adding guest os and mappings

* Added guest os mappings in guest os form

* Added new filter to guest os mapping

* Name and description changes

* VMWare Host and cluster MO unit tests

* CheckGuestOsMapping command and answer unit tests

* GetHypervisorGuestOsNames command and answer unit tests

* VmwareResource unitests

* GuestOsMapper unittests

* icon changes

* Addressed review comments

* Renaming fixes

* Removed comments

* marvin tests for guest os operations

* Added marvin tests for OS mappings

* Document links and UI improvements

* Added deduplication for the list guest OS API

* Fixed linter failure

* Few bug fixes and UI changes

* Few improvements

* Addressed code smells

* Fixed UI issues after rebase

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
2023-06-27 13:49:20 +05:30
slavkap faaf72b1a4
Volume encryption support for StorPool plug-in (#7539)
Supported Virtual machine operations:
- live migration of VM to another host
- virtual machine snapshots (group snapshot without memory)
- revert VM snapshot
- delete VM snapshot
Supported Volume operations:
- attach/detach volume
- live migrate volume between two StorPool primary storages
- volume snapshot
- delete snapshot
- revert snapshot
2023-06-26 11:24:51 +02:00
Nicolas Vazquez c809201247
Fix: Volumes on lost local storage cannot be removed (#7594) 2023-06-23 12:22:15 +02:00
dahn ae10263b3b
Possibility to choose the source NAT IP address on a isolated network or VPC (#6442)
Co-authored-by: NuxRo <nux@li.nux.ro>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2023-06-09 14:51:53 +02:00
Nicolas Vazquez 2457d202dd
Fix acquire console session issue (#7558)
* Fix acquire console session issue

* Refactor
2023-05-26 07:20:45 +05:30
John Bampton 11d45654a6
misc: fix spelling (#7206)
This PR fixes spellings
2023-05-23 11:06:16 +05:30