Commit Graph

440 Commits

Author SHA1 Message Date
Bryan Lima cb62ce6767
Global ACL for VPCs (#7150) 2023-11-30 14:51:43 +01:00
João Jandre 26b01f6f3b
Flexible tags for hosts and storage pools (#7489)
Co-authored-by: João Jandre <joao@scclouds.com.br>
2023-11-30 09:36:47 +01:00
Wei Zhou cb2b6aca45
server: check if there are active nics before network GC (#8204) 2023-11-29 18:55:26 +01:00
Daniel Augusto Veronezi Salvador adbb5520ca
Database VIEW management via separated files (#7417)
Co-authored-by: Daniel Augusto Veronezi Salvador <gutoveronezi@apache.org>
2023-11-17 10:27:53 +01:00
Bryan Lima 1f29f6f040
Public IP quarantine feature (#7378) 2023-11-15 10:29:22 +01:00
John Bampton f090c77f41
misc: fix spelling (#7549)
Co-authored-by: Stephan Krug <stekrug@icloud.com>
2023-11-02 09:23:53 +01:00
Vishesh 5362bad442
Storage Management (#7949) 2023-11-01 10:46:22 +01:00
Harikrishna 235e4fe190
Oauth2 integration with CloudStack (#7996)
OAuth2, the industry-standard authorization or authentication framework, simplifies the process of
granting access to resources. CloudStack supports OAuth2 authentication wherein users can login into
CloudStack without using a username and password. Support for Google and Github providers has been added.
Other OAuth2 providers can be easily integrated with CloudStack using its plugin framework.

The login page will show provider options when the OAuth2 is enabled and corresponding providers are configured.

"OAuth configuration" sub-section is present under "Configuration" where admins can register the corresponding
OAuth providers.
2023-10-31 13:25:28 +05:30
Wei Zhou bd52fa8a12
New feature: VNF templates and appliances integration (#8022) 2023-10-27 10:23:00 +02:00
Abhishek Kumar 543c54c718
api,server,ui: snapshot copy, multi-zone replica (#7873)
This PR adds new functionality to copy snapshots across zones and take snapshots for multiple zones.

Copy functionality is similar to template copy. The source zone acts as the web server from where the destination zone(s) can download the snapshot files. For this purpose, a new API - `copySnapshot` has been added. The response for copySnapshot will be returning zone and download details from the first destination zone of the request. This behaviour is similar to the `copyTemplate` API.

In a similar manner, multiple zones can be selected while taking the snapshots or creating snapshot policies. For this snapshot will be taken in the base zone(in which volume is present) and then copied to the additional zones. A new parameter - `zoneids` has been added to `createSnapshot` and `createSnapshotPolicy` APIs.

As snapshots can be present on multiple zones (secondary stores), a new parameter `zoneid` has been added to delete the snapshot copy on a specific zone.

`listSnapshots` API has been updated to allow listing snapshot entries for different zones/datastores. New parameters - `showUnique`, `locationType` have been added.

Events generated during snapshot operations will now be linked to the snapshot itself rather than the volume of the snapshot.

`listSnapshotPolicies` and `createSnapshotPolicy` APIs will return zone details of the zones in which backup will be scheduled for the policy.

----
New API added
`copySnapshot`

Request and response params updated for APIs
```
- listSnapshots
- deleteSnapshot
- createTemplate
- listZones
- listSnapshotPolicies
- createSnapshotPolicy
```
UI updated for
- Snapshot detail view
- Create snapshot form
- Create snapshot policy form
- Create volume (from snapshot) form
- Create template (from snapshot) form

Doc PR: https://github.com/apache/cloudstack-documentation/pull/344
PR: https://github.com/apache/cloudstack/pull/7873
2023-10-23 09:01:58 +02:00
Abhishek Kumar 20046ffe61 Merge remote-tracking branch 'apache/4.18' into main 2023-10-19 10:39:40 +05:30
Harikrishna 0183e25279
Fix VM snapshot size during storage capacity check (#8101) 2023-10-18 11:49:26 +02:00
Abhishek Kumar 9df580cef4 Merge remote-tracking branch 'apache/4.18' into main 2023-10-12 16:54:19 +05:30
Abhishek Kumar 3e7f21a190
vm-import: fix stopped managed vms listing in unmanaged instances (#7606)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-10-12 09:21:47 +02:00
Bryan Lima ae5dda867f
Normalize encryption on global configurations values (#6812) 2023-09-28 16:44:36 +02:00
sato03 8d81ec425d
keep deleted projects name (#7858)
Co-authored-by: Henrique Sato <henrique.sato@scclouds.com.br>
2023-09-28 09:15:32 +02:00
Daan Hoogland 0375714ded Merge release branch 4.18 to main
* 4.18:
  ldap trust map cleanup on domain delete (#7915)
  upgrade: fix upgrade from 4.18.1.0 to 4.18.2.0-SNAPSHOT (#7959)
2023-09-19 11:20:58 +02:00
Wei Zhou 9c9b17885f
upgrade: fix upgrade from 4.18.1.0 to 4.18.2.0-SNAPSHOT (#7959)
The uprgade from 4.18.1.0 to 4.18.2.0-SNAPSHOT failed with error

```
2023-09-12 16:12:19,003 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.18.1.0 Code Version = 4.18.2.0
2023-09-12 16:12:19,004 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.18.1.0 to 4.18.2.0
2023-09-12 16:12:19,036 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Running upgrade Upgrade41800to41810 to upgrade from 4.18.0.0-4.18.1.0 to 4.18.1.0
...
2023-09-12 16:12:19,041 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- Schema upgrade from 4.18.0.0 to 4.18.1.0
...
2023-09-12 16:12:21,602 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Statement: CREATE INDEX i_cluster_details__name on cluster_details (name)
2023-09-12 16:12:21,663 DEBUG [c.c.u.d.DatabaseAccessObject] (main:null) (logid:) Created index i_cluster_details__name
2023-09-12 16:12:21,673 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 2632 Name =  Upgrade; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:97-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:215-$Proxy30.persist:-1-DatabaseUpgradeChecker.upgrade:319-DatabaseUpgradeChecker.check:403-CloudStackExtendedLifeCycle.checkIntegrity:64
```

It succeeded with this change.
2023-09-18 19:14:36 +02:00
John Bampton 4eb110af73
Remove unneeded duplicate words (#7850) 2023-09-18 13:16:33 +02:00
Vishesh 710f34ee90
api: Fix listLoadBalancerRuleInstances api call (#7892) 2023-09-09 07:45:42 +02:00
Wei Zhou f6b2a58727 Merge branch '4.18' 2023-09-07 08:56:35 +02:00
Marcus Sorensen 2cccd8f754
db: Add index on cluster_details.name for FirstFitPlanner speedup (#7922) 2023-08-31 21:43:23 +02:00
GaOrtiga 819dd7b75c
server: remove supportedOwner from Resource.ResourceType (#7416) 2023-08-30 11:29:16 +02:00
Rohit Yadav 7362e5338c Merge remote-tracking branch 'origin/4.18' 2023-08-28 12:54:36 +05:30
Wei Zhou 0dd6bb7c67
systemvm: update size and state of systemvm template during registration (#7874) 2023-08-25 16:46:32 +02:00
Daan Hoogland 27f148e79e Merge branch '4.18' 2023-08-23 11:15:56 +02:00
Wei Zhou e4117e68d2
engine/schema: fix duplicated guest OSes in 4.18.0.0 (#7799)
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-08-23 09:44:34 +02:00
Daan Hoogland ea832bce13 Merge branch '4.18' 2023-08-22 11:44:45 +02:00
Wei Zhou 78bdde9e98
AutoScaling: support Managed User Data (#7769) 2023-08-22 11:07:16 +02:00
João Jandre 1065e9046b
Fix backup dates (#6473)
Co-authored-by: João Paraquetti <joao@scclouds.com.br>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-18 10:51:36 +02:00
Wei Zhou c8d6e50539
VMware: add support for 8.0b (8.0.0.2), 8.0c (8.0.0.3) (#7380)
* VMware: add support for 8.0b (8.0.0.2)

* VMware 8: add new guest os mappings in VirtualMachineGuestOsIdentifier

The full list can be found at https://developer.vmware.com/apis/1355/vsphere

* VMware: get guest os mappings of parent version

* VMware8: remove guest os mappings for 8.0.0.2

* VMware8: fix code smells

* vmware: remove annotations in VmwareVmImplementerTest which caused 0.0% code coverage

* VMware8: add a unit test case

* VMware: add support for 8.0c (8.0.0.3)

* VMware8: move to CloudStackVersion.getVMwareParentVersion

* VMware: add support for 8.0u1 (8.0.1.0)

* Copy engine/schema/src/main/java/com/cloud/upgrade/GuestOsMapper.java from PR 6979

* Copy engine/schema/src/main/java/com/cloud/storage/dao/GuestOSHypervisorDao.java from PR 6979

* VMware: ignore the last number in VMware versions

* VMware: copy guest os mapping from 8.0 to 8.0.1

* VMware: add unit tests in VmwareVmImplementerTest.java

* Copy engine/schema/src/test/java/com/cloud/upgrade/GuestOsMapperTest.java from PR 6979

* VMware8: retry vm poweron if fails due to exception "File system specific implementation of Ioctl[file] failed"

This fixes a weird issue on vmware8. When power on a vm, sometimes it fails due to error

2023-04-27 07:04:43,207 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-442:ctx-cdd42b03 10.0.32.133, job-105/job-106, cmd: StartCommand) (logid:8a24a607) StartCommand failed due to [Exception: java.lang.RuntimeException
Message: File system specific implementation of Ioctl[file] failed
].
java.lang.RuntimeException: File system specific implementation of Ioctl[file] failed
        at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:426)
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.powerOn(VirtualMachineMO.java:288)

in vmware.log on ESXi host, it shows

2023-04-27T09:20:41.713Z In(05)+ vmx - Power on failure messages: File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of LookupAndOpen[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - File system specific implementation of Ioctl[file] failed
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to lock the file
2023-04-27T09:20:41.713Z In(05)+ vmx - Cannot open the disk '/vmfs/volumes/7b29c876-ac102328/i-2-167-VM/ROOT-167.vmdk' or one of the snapshot disks it depends on.
2023-04-27T09:20:41.713Z In(05)+ vmx - Module 'Disk' power on failed.
2023-04-27T09:20:41.713Z In(05)+ vmx - Failed to start the virtual machine.

There is a KB article for it, but I still do not know why and how to fix it.
https://kb.vmware.com/s/article/1004232

* VMware: extract to method powerOnVM

* vmware: fix mistake in logs

* vmware8: use curl instead of wget to fix test failures

Traceback (most recent call last):
  File "/root/test_internal_lb.py", line 555, in test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
    self.execute_internallb_roundrobin_tests(vpc_offering)
  File "/root/test_internal_lb.py", line 641, in execute_internallb_roundrobin_tests
    client_vm, applb.sourceipaddress, max_http_requests)
  File "/root/test_internal_lb.py", line 497, in run_ssh_test_accross_hosts
    (e, clienthost.public_ip))
AssertionError: list index out of range: SSH failed for VM with IP Address: 10.0.52.187

and

sshClient: DEBUG: {Cmd: /usr/bin/wget -T3 -qO- --user=admin --password=password http://10.1.2.253:8081/admin?stats via Host: 10.0.52.188} {returns: ["/usr/bin/wget: '/usr/lib/libpcre.so.1' is not an ELF file", "/usr/bin/wget: can't load library 'libpcre.so.1'"]}

* VMware: correct guest OS names in hypervisor mappings for VMware 8.0

el9 and variants were introduced by https://github.com/apache/cloudstack/pull/7059
they are supported with guest os identifiers since VMware 8.0

see https://vdc-repo.vmware.com/vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/vim.vm.GuestOsDescriptor.GuestOsIdentifier.html

* VMware: add Ubuntu 20.04 and 22.04 support for vmware 7.0+

* PR7380: only add guest os mappings for Ubuntu 20.04

* PR7380: Correct RHEL9 guest os names and others for VMware 8.0

* PR7380: correct guest os names on 8.0.0.1 as well

* PR7380: remove Windows 12 and Windows Server 2025 which are not released yet
2023-08-17 10:42:42 +02:00
Nicolas Vazquez 8b5ba13b81
plugins: Add Custom hypervisor minimal changes (#7692)
### Description

Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

##### Proposed Future work:
Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

#### Feature Specifications
- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.


#### Use cases
This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts
2023-08-16 20:53:24 +05:30
Daan Hoogland 7b64236469 Merge release branch 4.18 to main
* 4.18:
  server: remove registered userdata when cleanup an account (#7777)
  server: Use max secondary storage defined on the account during upload  (#7441)
  test: upgrade kubernetes versions to 1.25.0/1.26.0 (#7685)
  kvm: Added VNI Devices as normal bridge slave devs (#7836)
  noVNC: fix JP keyboard on vmware7+ which uses websocket URL (#7694)
2023-08-10 14:50:46 +02:00
Wei Zhou 1605b2f0db
server: remove registered userdata when cleanup an account (#7777) 2023-08-10 14:33:59 +02:00
Daan Hoogland 7ee697fcc5 Merge release branch 4.18 to main
* 4.18:
  UI: allow new keys for VM details (#7793)
  Refactoring StorPool's smoke tests (#7392)
  UI: decode userdata in EditVM dialog (#7796)
  packaging: unalias cp before package upgrade (#7722)
  make NoopDbUpgrade do a systemvm template check (#7564)
  UI unit test: fix expected values (#7792)
2023-08-03 15:59:01 +02:00
dahn 2bb079189c
make NoopDbUpgrade do a systemvm template check (#7564) 2023-07-31 16:08:14 +02:00
GaOrtiga 0bab0db881
Introducing concept of domain VPCs (#7153)
Co-authored-by: Gabriel Ortiga Fernandes <gabriel.fernandes@scclouds.com.br>
Co-authored-by: Lopez <rodrigo@scclouds.com.br>
2023-07-29 20:06:49 +02:00
Daan Hoogland 6bb95c0200 Merge release branch 4.18 to main
* 4.18:
  Storage and volumes statistics tasks for StorPool primary storage (#7404)
  proper storage construction (#6797)
  guarantee MAC uniqueness (#7634)
  server: allow migration of all VMs with local storage on KVM (#7656)
  Add L2 networks to Zones with SG (#7719)
2023-07-19 10:59:19 +02:00
dahn 73a269e3b3
guarantee MAC uniqueness (#7634)
Co-authored-by: Bryan Lima <42067040+BryanMLima@users.noreply.github.com>
2023-07-19 10:25:01 +02:00
Vishesh 594c70dde0
Sync precommit config from main (#7732)
Co-authored-by: John Bampton <jbampton@users.noreply.github.com>
Co-authored-by: dahn <daan@onecht.net>
2023-07-07 11:18:16 +02:00
Daan Hoogland 2132f46fcb Merge branch '4.18' 2023-07-06 11:24:08 +02:00
Harikrishna 0cbe77024a
Fix foreign key constraints and the mysql procedure that is used (#7381) 2023-07-06 08:43:49 +02:00
SadiJr 3c5fdeafdc
[Veeam] restored VMs without NICs (#6282) 2023-07-03 14:35:48 +02:00
Wei Zhou 41403c9cf9 Merge remote-tracking branch 'apache/4.18' 2023-06-27 11:52:37 +02:00
dahn 06caf32bc8
Guest os mappings improvements (#6979)
* Guest OS mapping improvements
- Checks the OS mapping name in hypervisor (VMware, XenServer)
- Displays guest OS mappings in UI

* Added API getHypervisorGuestOsNames to list the guest OS names in the hypervisor, and code improvements

* Some static analysis fixes

* Removed commented code in listview

* Guest OS list

* UI changes for adding guest os and mappings

* Added guest os mappings in guest os form

* Added new filter to guest os mapping

* Name and description changes

* VMWare Host and cluster MO unit tests

* CheckGuestOsMapping command and answer unit tests

* GetHypervisorGuestOsNames command and answer unit tests

* VmwareResource unitests

* GuestOsMapper unittests

* icon changes

* Addressed review comments

* Renaming fixes

* Removed comments

* marvin tests for guest os operations

* Added marvin tests for OS mappings

* Document links and UI improvements

* Added deduplication for the list guest OS API

* Fixed linter failure

* Few bug fixes and UI changes

* Few improvements

* Addressed code smells

* Fixed UI issues after rebase

---------

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
2023-06-27 13:49:20 +05:30
slavkap faaf72b1a4
Volume encryption support for StorPool plug-in (#7539)
Supported Virtual machine operations:
- live migration of VM to another host
- virtual machine snapshots (group snapshot without memory)
- revert VM snapshot
- delete VM snapshot
Supported Volume operations:
- attach/detach volume
- live migrate volume between two StorPool primary storages
- volume snapshot
- delete snapshot
- revert snapshot
2023-06-26 11:24:51 +02:00
Nicolas Vazquez c809201247
Fix: Volumes on lost local storage cannot be removed (#7594) 2023-06-23 12:22:15 +02:00
dahn ae10263b3b
Possibility to choose the source NAT IP address on a isolated network or VPC (#6442)
Co-authored-by: NuxRo <nux@li.nux.ro>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2023-06-09 14:51:53 +02:00
Nicolas Vazquez 2457d202dd
Fix acquire console session issue (#7558)
* Fix acquire console session issue

* Refactor
2023-05-26 07:20:45 +05:30
John Bampton 11d45654a6
misc: fix spelling (#7206)
This PR fixes spellings
2023-05-23 11:06:16 +05:30
Abhishek Kumar 8e7c1f7bd7
schema: acquired for console_session turned into datetime (#7414)
* schema: acquire for console_session turned into datetime

Fixes #7159

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

* Update schema-41810to41900.sql

* Update schema-41810to41900.sql

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-04-26 22:05:21 +05:30
Daan Hoogland 514df7b3a0 Merge release branch 4.18 to main
* 4.18:
  debian/control: build packages on ubuntu 22.04 (#7269)
  server,engine-schema: fix listing service offering for vm scale (#7398)
2023-04-17 17:44:50 +02:00
Abhishek Kumar afccde5cfc
server,engine-schema: fix listing service offering for vm scale (#7398)
Fixes #7389

Fixes listing of service offerings for VM scale when the current offering has  `disk_offering_strictness=true`

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-04-17 14:19:20 +05:30
John Bampton c2e17310d6
Add three more `pre-commit` checks (#7083)
Co-authored-by: dahn <daan@onecht.net>
2023-03-27 13:28:55 +02:00
David Jumani 2aa3f980e8
Fix schema path for 419 upgrade path (#7346) 2023-03-16 11:58:17 +01:00
Rohit Yadav 1d2e3fe24d Merge remote-tracking branch 'origin/4.18'
Fix 4.19/main build and add upgrade path from 4.18.1.0->4.19.0.0

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-03-16 13:36:04 +05:30
Rohit Yadav 74a414e76f Fix build failure and 4.18.0.0->4.18.1.0 upgrade path
Unblocks smoketests/health check runs on 4.18 branch.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-03-16 13:18:26 +05:30
dahn aa881f6bf3
avoid hard coded guest OS IDs (#7279) 2023-02-23 15:19:45 +01:00
Harikrishna a3feccf70c
User two factor authentication (#6924)
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-02-13 09:14:17 +01:00
Wei Zhou 62e342c1bc
utils,framework/db: Introduce new database encryption cipher based on AesGcmJce (#7003) 2023-02-02 16:25:49 +01:00
Nicolas Vazquez 89bf4750ab
Add console session cleanup task (#7132) 2023-02-01 16:53:54 +01:00
David Jumani c774b865c9
Tungsten integration (#7065)
Co-authored-by: rtodirica <rtodirica@ena.com>
Co-authored-by: Huy Le <huylm@unitech.vn>
Co-authored-by: radu-todirica <Radu.Todirica@ness.com>
Co-authored-by: Huy Le <minh.le@ext.ewerk.com>
Co-authored-by: Simon Weller <siweller77@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-02-01 09:19:53 +01:00
Suresh Kumar Anaparti d8c7e34b38
Improve global settings UI to be more intuitive/logical (#5797)
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: nvazquez <nicovazquez90@gmail.com>
Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-01-31 11:23:43 +01:00
Abhishek Kumar 3b6ce97097
infra: edge zones (#6840)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: dahn <daan@onecht.net>
2023-01-31 09:36:45 +01:00
nvazquez 83c2bfacd8
Merge branch '4.17' 2023-01-30 07:53:58 -03:00
Abhishek Kumar 028ca74fb6
ui,server,api: resource metrics improvements (#6803)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-01-30 09:48:03 +01:00
Nicolas Vazquez c78a777d3a
Fix: memory leak on volume allocation (#7136) 2023-01-30 09:44:50 +01:00
Abhishek Kumar 2dd29558fe
server: fix listnetworkofferings with domain, refactor listvpofferings (#6748) 2023-01-24 09:30:12 +01:00
dahn 221118289b
fix guestOsMapper and move mapping code to latest upgrade (#7095) 2023-01-23 09:03:55 +01:00
Daniel Augusto Veronezi Salvador 911f951e2a
Handle console session in multiple management servers (#7094) 2023-01-20 09:25:07 +01:00
Daan Hoogland 0c925be44d merge LTS 4.17 in main 2023-01-12 15:39:04 +01:00
Wei Zhou 1dd064c8a9
server: fix cannot update vm if it uses unconstraint dynamic offering (#7077)
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-01-12 13:40:53 +01:00
Daan Hoogland 2d1d2944e5 merge release branch 4.17 into main 2023-01-09 12:57:40 +01:00
Abhishek Kumar eb5c8a5a8c
server: correctly list suitable hosts for migration with uefi capability (#7024)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-01-09 11:03:10 +01:00
Sina Kashipazha 20306d6129
Allow creating atmost 1 physical network with null tag (#6781) 2023-01-05 21:36:31 +01:00
Pearl Dsilva 3044d63a8b
Configurable MTU for VR (#6426)
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2023-01-04 09:42:24 +01:00
dahn 41717b0977
allow a new upgrade version without DB changes to be inserted (#6858) 2022-12-19 11:03:07 +01:00
Rohit Yadav 72b6ab9c51 Merge remote-tracking branch 'origin/4.17' into main
Due to merge conflict, and schema changes in 4.17 branch the previous
4.17.1->4.18.0 DB upgrade path class was renamed to 4.17.2->4.18.0

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-12-13 07:50:36 +00:00
Rohit Yadav d331b2f98f
schema: Add upgrade path from 4.17.1.0 to 4.17.2.0 (#6981)
This implements a blank/noop upgrade path from 4.17.1.0 to 4.17.2.0
which implements DbUpgradeSystemVmTemplate to kick the systemvm template
upgrade.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-12-13 12:42:20 +05:30
Daniel Augusto Veronezi Salvador ed1ac1c161
schema: Fix migration path (#6927)
PR #5909 was created before the announce of release 4.17.1.0 and the changes in the databases were addressed in the 4.17.0.0 -> 4.18.0.0 migration path. However, #5909 was merged after 4.17.1.0 releasing, with the original migration path.

This PR intends to fix the migration path of PR #5909.

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2022-12-12 12:42:34 +05:30
Rohit Yadav 44e13900e6
schema: Add upgrade path from 4.17.2.0 same as 4.17.1.0 (#6965)
There's no DB upgrade path b/w 4.17.1.0 and 4.17.2.0, this adds the
same upgrade path of 4.17.1.0 when source version is 4.17.2.0.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-12-12 12:38:57 +05:30
João Jandre 7c61d8aeaf
Set root volume as destroyed when destroying a VM (#6868)
* Set root volume as destroyed when destroying a VM

* Address review

* Address review

Co-authored-by: João Jandre <joao@scclouds.com.br>
2022-12-06 17:48:35 -03:00
Wei Zhou a63b2aba7a
VM Autoscaling with virtual router (#6571) 2022-12-05 15:23:03 +01:00
John Bampton e65c22d883
Fix spelling (#6860) 2022-11-13 10:56:15 +01:00
John Bampton fa39e61a4c
Fix spelling (#6609) 2022-11-01 08:30:01 +01:00
dahn 4a06363749
Ova download fix (#6758) 2022-10-21 14:31:19 +02:00
Daniel Augusto Veronezi Salvador 2ca164ac96
Quota custom tariffs (#5909)
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-10-17 10:03:50 +02:00
Harikrishna 713a236843
UserData as first class resource (#6202)
This PR introduces a new feature to make userdata as a first class resource much like existing SSH keys.

Detailed feature specification document:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Userdata+as+a+first+class+resource
2022-10-05 17:34:59 +05:30
Sina Kashipazha 4e2f461b31
Prometheus exporter enhancement (#4438)
* Export count of total/up/down hosts by tags

* Export count of vms by state and host tag.

* Add host tags to host cpu/cores/memory usage in Prometheus exporter

* Cloudstack Prometheus exporter: Add allocated capacity group by host tag.

* Show count of Active domains on grafana.

* Show count of Active accounts and vms by size on grafana

* Use prepared statement to query database for a number of VM who use a specific tag.

* Extract repeated codes to new methods.
2022-09-30 17:02:01 +02:00
Marcus Sorensen 697e12f8f7
kvm: volume encryption feature (#6522)
This PR introduces a feature designed to allow CloudStack to manage a generic volume encryption setting. The encryption is handled transparently to the guest OS, and is intended to handle VM guest data encryption at rest and possibly over the wire, though the actual encryption implementation is up to the primary storage driver.

In some cases cloud customers may still prefer to maintain their own guest-level volume encryption, if they don't trust the cloud provider. However, for private cloud cases this greatly simplifies the guest OS experience in terms of running volume encryption for guests without the user having to manage keys, deal with key servers and guest booting being dependent on network connectivity to them (i.e. Tang), etc, especially in cases where users are attaching/detaching data disks and moving them between VMs occasionally.

The feature can be thought of as having two parts - the API/control plane (which includes scheduling aspects), and the storage driver implementation.

This initial PR adds the encryption setting to disk offerings and service offerings (for root volume), and implements encryption support for KVM SharedMountPoint, NFS, Local, and ScaleIO storage pools.

NOTE: While not required, operations can be significantly sped up by ensuring that hosts have the `rng-tools` package and service installed and running on the management server and hypervisors. For EL hosts the service is `rngd` and for Debian it is `rng-tools`. In particular, the use of SecureRandom for generating volume passphrases can be slow if there isn't a good source of entropy. This could affect testing and build environments, and otherwise would only affect users who actually use the encryption feature. If you find tests or volume creates blocking on encryption, check this first.

### Management Server

##### API

* createDiskOffering now has an 'encrypt' Boolean
* createServiceOffering now has an 'encryptroot' Boolean. The 'root' suffix is added here in case there is ever any other need to encrypt something related to the guest configuration, like the RAM of a VM.  This has been refactored to deal with the new separation of service offering from disk offering internally.
* listDiskOfferings shows encryption support on each offering, and has an encrypt boolean to choose to list only offerings that do or do not support encryption
* listServiceOfferings shows encryption support on each offering, and has an encrypt boolean to choose to list only offerings that do or do not support encryption
* listHosts now shows encryption support of each hypervisor host via `encryptionsupported`
* Volumes themselves don't show encryption on/off, rather the offering should be referenced. This follows the same pattern as other disk offering based settings such as the IOPS of the volume.

##### Volume functions

A decent effort has been made to ensure that the most common volume functions have either been cleanly supported or blocked. However, for the first release it is advised to mark this feature as *experimental*, as the code base is complex and there are certainly edge cases to be found.

Many of these features could eventually be supported over time, such as creating templates from encrypted volumes, but the effort and size of the change is already overwhelming.

Supported functions:
* Data Volume create
* VM root volume create
* VM root volume reinstall
* Offline volume snapshot/restore
* Migration of VM with storage (e.g. local storage VM migration)
* Resize volume
* Detach/attach volume

Blocked functions:
* Online volume snapshot
* VM snapshot w/memory
* Scheduled snapshots (would fail when VM is running)
* Disk offering migration to offerings that don't have matching encryption
* Creating template from encrypted volume
* Creating volume from encrypted volume
* Volume extraction (would we decrypt it first, or expose the key? Probably the former).

##### Primary Storage Support

For storage developers, adding encryption support involves:

1. Updating the `StoragePoolType` for your primary storage to advertise encryption support. This is used during allocation of storage to match storage types that support encryption to storage that supports it.

2. Implementing encryption feature when your `PrimaryDataStoreDriver` is called to perform volume lifecycle functions on volumes that are requesting encryption. You are free to do what your storage supports - this could be as simple as calling a storage API with the right flag when creating a volume. Or (as is the case with the KVM storage types), as complex as managing volume details directly at the hypervisor host. The data objects passed to the storage driver will contain volume passphrases, if encryption is requested.

##### Scheduling

For the KVM implementations specified above, we are dependent on the KVM hosts having support for volume encryption tools. As such, the hosts `StartupRoutingCommand` has been modified to advertise whether the host supports encryption. This is done via a probe during agent startup to look for functioning `cryptsetup` and support in `qemu-img`. This is also visible via the listHosts API and the host details in the UI.  This was patterned after other features that require hypervisor support such as UEFI.

The `EndPointSelector` interface and `DefaultEndpointSelector` have had new methods added, which allow the caller to ask for endpoints that support encryption.  This can be used by storage drivers to find the proper hosts to send storage commands that involve encryption. Not all volume activities will require a host to support encryption (for example a snapshot backup is a simple file copy), and this is the reason why the interface has been modified to allow for the storage driver to decide, rather than just passing the data objects to the EndpointSelector and letting the implementation decide.

VM scheduling has also been modified. When a VM start is requested, if any volume that requires encryption is attached, it will filter out hosts that don't support encryption.

##### DB Changes

A volume whose disk offering enables encryption will get a passphrase generated for it before its first use. This is stored in the new 'passphrase' table, and is encrypted using the CloudStack installation's standard configured DB encryption. A field has been added to the volumes table, referencing this passphrase, and a foreign key added to ensure passphrases that are referenced can't be removed from the database.  The volumes table now also contains an encryption format field, which is set by the implementer of the encryption and used as it sees fit.

#### KVM Agent

For the KVM storage pool types supported, the encryption has been implemented at Qemu itself, using the built-in LUKS storage support. This means that the storage remains encrypted all the way to the VM process, and decrypted before the block device is visible to the guest.  This may not be necessary in order to implement encryption for /your/ storage pool type, maybe you have a kernel driver that decrypts before the block device on the system, or something like that. However, it seemed like the simplest, common place to terminate the encryption, and provides the lowest surface area for decrypted guest data.

For qcow2 based storage, `qemu-img` is used to set up a qcow2 file with LUKS encryption. For block based (currently just ScaleIO storage), the `cryptsetup` utility is used to format the block device as LUKS for data disks, but `qemu-img` and its LUKS support is used for template copy.

Any volume that requires encryption will contain a passphrase ID as a byte array when handed down to the KVM agent. Care has been taken to ensure this doesn't get logged, and it is cleared after use in attempt to avoid exposing it before garbage collection occurs.  On the agent side, this passphrase is used in two ways:

1. In cases where the volume experiences some libvirt interaction it is loaded into libvirt as an ephemeral, private secret and then referenced by secret UUID in any libvirt XML. This applies to things like VM startup, migration preparation, etc.

2. In cases where `qemu-img` needs to use this passphrase for volume operations, it is written to a `KeyFile` on the cloudstack agent's configured tmpfs and passed along. The `KeyFile` is a `Closeable` and when it is closed, it is deleted. This allows us to try-with-resources any volume operations and get the KeyFile removed regardless.

In order to support the advanced syntax required to handle encryption and passphrases with `qemu-img`, the `QemuImg` utility has been modified to support the new `--object` and `--image-opts` flags. These are modeled as `QemuObject` and `QemuImageOptions`.  These `qemu-img` flags have been designed to supersede some of the existing, older flags being used today (such as choosing file formats and paths), and an effort could be made to switch over to these wholesale. However, for now we have instead opted to keep existing functions and do some wrapping to ensure backward compatibility, so callers of `QemuImg` can choose to use either way.

It should be noted that there are also a few different Enums that represent the encryption format for various purposes. While these are analogous in principle, they represent different things and should not be confused. For example, the supported encryption format strings for the `cryptsetup` utility has `LuksType.LUKS` while `QemuImg` has a `QemuImg.PhysicalDiskFormat.LUKS`.

Some additional effort could potentially be made to support advanced encryption configurations, such as choosing between LUKS1 and LUKS2 or changing cipher details. These may require changes all the way up through the control plane. However, in practice Libvirt and Qemu currently only support LUKS1 today. Additionally, the cipher details aren't required in order to use an encrypted volume, as they're stored in the LUKS header on the volume there is no need to store these elsewhere.  As such, we need only set the one encryption format upon volume creation, which is persisted in the volumes table and then available later as needed.  In the future when LUKS2 is standard and fully supported, we could move to it as the default and old volumes will still reference LUKS1 and have the headers on-disk to ensure they remain usable. We could also possibly support an automatic upgrade of the headers down the road, or a volume migration mechanism.

Every version of cryptsetup and qemu-img tested on variants of EL7 and Ubuntu that support encryption use the XTS-AES 256 cipher, which is the leading industry standard and widely used cipher today (e.g. BitLocker and FileVault).

Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2022-09-27 10:20:59 +05:30
Abhishek Kumar 7905a76bd2 Merge remote-tracking branch 'apache/4.17' into main
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-09-23 17:14:59 +05:30
Rohit Yadav dcffdc1c0b
DatabaseUpgradeChecker: add upgrade path from 4.17.0.1 to 4.17.1.0 (#6759)
Fixes DB upgrade for 4.17.0.1 to 4.17.1.0
2022-09-23 17:12:18 +05:30
Bryan Lima 7be7ef66fb
Improve error message on storage tags update (#6269)
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-09-14 09:06:20 +02:00
Abhishek Kumar 78b68fd7e6
api,server: custom dns for guest network (#6425)
Adds option to provide custom DNS servers for isolated network, shared network and VPC tier.
New API parameters added in createNetwork API along with the corresponding response parameters.

Doc PR: apache/cloudstack-documentation#276
2022-09-10 13:05:40 +05:30
Bryan Lima d4b8e2cbe8
Validate host tags on VM live scale (#6409)
* Validate host tags on VM live scale

* Remove extra spaces
2022-08-31 09:24:00 +02:00
Abhishek Kumar bd6bcffccc Merge remote-tracking branch 'apache/4.17' into main 2022-08-22 14:42:35 +05:30
Nicolas Vazquez 1fd1f0626a
Fix SQL query for uuid wrong format (#6654)
Fixes: #6652
2022-08-19 18:27:11 +05:30
Rohit Yadav 83008e55cf Merge remote-tracking branch 'origin/4.17' 2022-08-17 16:25:25 +02:00
Abhishek Kumar 010b47f82e
server: fix ipv6 network deployment with separate guest nw (#6588)
Fixes #6584

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-08-17 15:49:01 +05:30
Abhishek Kumar d9b770eb48 Merge remote-tracking branch 'apache/4.17' into main 2022-08-12 23:44:42 +05:30