Commit Graph

36071 Commits

Author SHA1 Message Date
John Bampton b0ae701ca4
Remove Super-Linter action and move `flake8` to pre-commit (#7861) 2023-08-17 09:37:10 +02:00
Nicolas Vazquez 8b5ba13b81
plugins: Add Custom hypervisor minimal changes (#7692)
### Description

Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

##### Proposed Future work:
Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

#### Feature Specifications
- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.


#### Use cases
This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts
2023-08-16 20:53:24 +05:30
Vishesh fe70f4d801
Remove powermock from services (#7860) 2023-08-16 09:05:18 +02:00
Daan Hoogland 5559668f12 Merge branch '4.18' 2023-08-15 09:15:17 +02:00
Wei Zhou aa02d9b3c1
test: skip live storage migration on CentOS 7 (#7862)
since #7570, The detail 'Host.OS' of centos7 host is changed from 'CentOS' to 'CentOS Linux'
2023-08-15 08:39:18 +02:00
Alexandru Bagu a47a4f4ad4
CPVM: fix cpu usage for console vm when using vnc over websockets (#6970)
Co-authored-by: Alexandru Bagu <alexandru.bagu@hotmail.com>
2023-08-14 15:45:04 +02:00
Fabricio Duarte 72e3491cef
server: Fix allocation of more public IPs than the account's limit (#7832) 2023-08-14 10:33:29 +02:00
Fabricio Duarte 2c6072273b
VMware: Fix casting exception during backup NIC synchronization (#7831) 2023-08-14 10:20:33 +02:00
Wei Zhou bf9c71430b
CKS: check access when get kubernetescluster config (#7854) 2023-08-14 10:02:20 +02:00
Nicolas Vazquez 0204377032
Fix UI unit tests (#7856) 2023-08-11 16:47:11 +02:00
Rohit Yadav d22a3d517d ui: fix conflict/regression found in AutogenView
After forward merged from 4.18->main, found a regression due to
3c25a35426 where any columns which are
custom functions don't render. Such as metrics columns.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-11 16:38:52 +05:30
Rohit Yadav 62feb24de6 Merge remote-tracking branch 'origin/4.18'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-11 16:27:57 +05:30
Rohit Yadav feb9509547
ui: assorted improvements (#7833)
This PR aims to polish the UI with following tweaks and changes:

- Increase resource and os-logo icons both in list view, user-menu bar and VM deployment form
- Fix css issues in VM deployment form when resource icons are on some of the templates/isos but not all
- Replace edit icon in the resource icon editting button on the infocard, in resource view
- Fix css marging/padding issue for nav bar and left-branding/logo
- Introduce a new Limits option in the user menu, to allow users to see their own limits when they log in
- Rename resource tab to limits tab for accounts, project and domains
- Introduce a new copy-label component, that can be clicked to copy strings; use in info-card and list view for entites such as IP addresses and UUIDs
- Add router-link to /zones/ in case of user-accounts (when /zone isn't routable in the UI)
- Show better list of nics and ssh keys pairs in infocard for VM resource view
- Standardise most resources to show state/status columns right after resource name (wherever applicable)
- Remove displayname column in VM list view, add cpu number and memory by default
- Add k8s version column in k8s list view
- Add size and phy size columns in case of template and ISOs list view, only for root/domain admins
- Add phy network router-link in case of guest VLAN list view; rearrange columns list for consistency
- Add snapshot phy size column in the snapshot list view; and router-link for volume in the snapshot list view; and missing/useful details in the volume snapshot details view
- Add a create and add data disk feature in Instances tab, just like we've add nic feature in the same

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-11 14:00:04 +05:30
Wei Zhou 540dcc1040 Merge remote-tracking branch 'apache/4.18' 2023-08-11 08:30:07 +02:00
Rohit Yadav 29c7b3167e
cloudstack: bump to use 4.18.1.0 systemvmtemplate (#7828)
* cloudstack: bump to use 4.18.1.0 systemvmtemplate

Use new 4.18.1.0 systemvmtemplate

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* fix cd ISO url

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

---------

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-11 07:38:47 +02:00
John Bampton 9bc0dd7a69
pre-commit fix line ending (#7851) 2023-08-10 20:14:47 +02:00
Wei Zhou d9319e645c
storpool: fix pom.xml which caused docker image build failures (#7845)
steps to reproduce the issue:
- git clone https://github.com/apache/cloudstack.git
- cd cloudstack
- rm -rf .git/
- run `mvn -P developer,systemvm clean install`

Without this PR, it fails with error

```
 > [ 8/10] RUN mvn -Pdeveloper -Dsimulator -DskipTests clean install:
668.1 [ERROR] Failed to execute goal pl.project13.maven:git-commit-id-plugin:4.9.10:revision (get-the-git-infos) on project cloud-plugin-storage-volume-storpool: .git directory is not found! Please specify a valid [dotGitDirectory] in your pom.xml -> [Help 1]
```
2023-08-10 16:28:49 +02:00
Daan Hoogland 7b64236469 Merge release branch 4.18 to main
* 4.18:
  server: remove registered userdata when cleanup an account (#7777)
  server: Use max secondary storage defined on the account during upload  (#7441)
  test: upgrade kubernetes versions to 1.25.0/1.26.0 (#7685)
  kvm: Added VNI Devices as normal bridge slave devs (#7836)
  noVNC: fix JP keyboard on vmware7+ which uses websocket URL (#7694)
2023-08-10 14:50:46 +02:00
Wei Zhou 1605b2f0db
server: remove registered userdata when cleanup an account (#7777) 2023-08-10 14:33:59 +02:00
João Jandre fdb23dae40
server: Use max secondary storage defined on the account during upload (#7441) 2023-08-10 11:39:40 +02:00
Wei Zhou b32ba953cf
test: upgrade kubernetes versions to 1.25.0/1.26.0 (#7685)
* test: upgrade versions to 1.25.0/1.26.0

* PR7685: move settings to test_data.py
2023-08-10 09:48:08 +02:00
Vishesh 82f9106633
Remove powermock from kvm hypervisor plugin (#7818)
* Remove powermock from kvm hypervisor plugin

* Add @RunWith

* Remove unnecessary mocks

* Remove commented code
2023-08-10 09:04:36 +02:00
fermosan fa58f59619
kvm: Added VNI Devices as normal bridge slave devs (#7836)
This will allow for VXLAN configurations to utilize tags on the physical network of a zone
2023-08-10 08:59:28 +02:00
kiranchavala 927818b55f
added docker command for ubuntu 22.04 to Building DEB packages (#7315) 2023-08-09 12:11:49 +02:00
Wei Zhou a1af03f413
noVNC: fix JP keyboard on vmware7+ which uses websocket URL (#7694)
* noVNC: fix JP keyboard on vmware7+ which uses websocket URL

* noVNC: cleanup rfb.js

* noVNC: fix < and > on JP keyboard

* noVNC: fix Caps lock on JP keyboard
2023-08-09 11:43:16 +02:00
Rohit Yadav ac882f3d07 Merge remote-tracking branch 'origin/4.18' 2023-08-08 15:56:19 +05:30
Sina Kashipazha 9df23f951b
Prometheus exporter fix cpu/memory usage labels (#7629) 2023-08-07 20:47:03 +02:00
Daan Hoogland 32448e1ac7 Merge branch '4.18' 2023-08-07 11:13:17 +02:00
dahn dea1373480
server: clean network offerings for domain on remove (#7775) 2023-08-07 09:56:58 +02:00
Vishesh 55f0801b18
Remove powermock from network elements (#7791) 2023-08-07 09:24:26 +02:00
dahn 0318e9fccb
test_vm_deployment_planner: fix using wrong hypervisor (#7718) 2023-08-07 10:20:42 +05:30
Rohit Yadav 4f85c5568f
engine/schema: fix CPU cap limitation for upgraded environment (#7808)
CPU cap limitation was enabled as part of
https://github.com/apache/cloudstack/pull/6420 that changes behaviour
for existing environments. The CPU cap limitation on KVM causes
systemvms to not start or be really slow in nested and virtualised
environments.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2023-08-07 10:19:33 +05:30
mprokopchuk ab0297ea9b
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted (#7760)
VM.CREATE/VOLUME.DELETE/VOLUME.DESTROY not being emitted

* Update server/src/main/java/com/cloud/vm/UserVmManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

---------

Co-authored-by: Maxim Prokopchuk <mprokopchuk@apple.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2023-08-07 10:18:17 +05:30
Daan Hoogland eb31e3d795 Merge release branch 4.18 to main
* 4.18:
  Allow KVM overcommit to work without reducing minimum VM memory when vm ballooning is disabled (#7810)
2023-08-06 10:41:00 +02:00
Rohit Yadav dc5e4f3ec6
Allow KVM overcommit to work without reducing minimum VM memory when vm ballooning is disabled (#7810)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daan Hoogland <daan@onecht.net>
2023-08-06 10:39:14 +02:00
Vishesh 84974a1e08
Remove powermock from vmware-base & vmware hypervisor plugin (#7798) 2023-08-04 10:40:29 +02:00
Vishesh 90443cdd43
Remove powermock from xenserver hypervisor plugin (#7806) 2023-08-04 10:25:58 +02:00
Daan Hoogland e3bd2c0a8c Merge branch '4.18' 2023-08-04 09:24:16 +02:00
dahn 6b21825746
make details optional for addGuestOs API (#7812) 2023-08-04 08:44:29 +02:00
kishankavala 57779a8586
Remove nic allocation txn during Vm deploy. During Vm deploy failure, this will allow cleanup of any nics that are successfully provisioned (#7809) 2023-08-04 08:42:38 +02:00
dahn b4032d9984
include all VMs when projects selected not just for admins (#7667) 2023-08-03 16:06:11 +02:00
Daan Hoogland 7ee697fcc5 Merge release branch 4.18 to main
* 4.18:
  UI: allow new keys for VM details (#7793)
  Refactoring StorPool's smoke tests (#7392)
  UI: decode userdata in EditVM dialog (#7796)
  packaging: unalias cp before package upgrade (#7722)
  make NoopDbUpgrade do a systemvm template check (#7564)
  UI unit test: fix expected values (#7792)
2023-08-03 15:59:01 +02:00
dahn c86684fbdd
UI: allow new keys for VM details (#7793) 2023-08-03 04:58:22 +02:00
John Bampton 6f4503488b
pre-commit: apply `end-of-file-fixer` to all files (#7551) 2023-08-02 13:47:21 +02:00
slavkap da7fc72c66
Refactoring StorPool's smoke tests (#7392)
* Removed the hardcoded StorPool endpoint from tests

- removed the hardcoded enpoint of StorPool primary storage from tests
- added the git commit information into the maven build

* Convert indents to spaces

* update git-commit-id-plugin version
2023-08-02 14:37:37 +08:00
John Bampton e876ef096e
Fix links to `release notes` in README and CHANGES (#7797) 2023-08-01 17:58:19 +02:00
Wei Zhou 87f218c68c
UI: decode userdata in EditVM dialog (#7796) 2023-08-01 09:10:26 +02:00
Vishesh 0d9292c9ea
Remove powermock from database/quota plugin (#7795) 2023-08-01 08:55:29 +02:00
dahn 60ab494c25
packaging: unalias cp before package upgrade (#7722) 2023-08-01 01:34:05 +02:00
dahn 2bb079189c
make NoopDbUpgrade do a systemvm template check (#7564) 2023-07-31 16:08:14 +02:00