Commit Graph

2067 Commits

Author SHA1 Message Date
Nicolas Vazquez b0c47f4a97
FR65: Secure KVM VNC connection using CA framework (#217)
* Support RFB 3.8 and VNC auth working

* IN PROGRESS: Add TigerVNC classes and authenticate, work left on the messaging

* Fix console display

* Cleanup

* Last in-progress work

* Don't block reads in case stream empty, use Link to init client certs (#215)

Co-authored-by: Marcus Sorensen <mls@apple.com>

* Unused files and methods cleanup

* Rewrite finished

* More cleanup

* Rename console server session to VM display name (#216)

* Rename console server session to VM display name

* Fix after rebasing

* Add encryption bar when available

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>

* Add missing license header

* Remove unused variable from the TLS security

* Address review comments and sonar cloud reports

* Address more review comments

* Last sonarcloud code smell fixes

* Automate VNC TLS provisioning and improve UI

* Add missing cases for sonarcloud report

* Fix certs renewal issue on configuring TLS

* Address review comments

* Refactor serviceConfig script

* Fix certs propagation TLS conf

* Fix unsecure host tests

Co-authored-by: Marcus Sorensen <marcus_sorensen@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-01-19 08:35:16 -03:00
Wei Zhou 11f38ad102
Apple FR66 - Host Control Plane Status (#213)
* Apple FR66 - Host Control Plane Status

* Apple FR66 - use Offline instead of Disconnected

* Apple FR66: fix smoke test test_router_host_control_state

* Apple FR66: reorder import

* Apple FR66: revert DetailsTab.vue and apply new changes

* Update PR: update en.json

* Update PR: add hostcontrolstate to routers/systemvms

* Update PR: test stop/start cloudstack-agent in smoke test

* Update PR: fix UI build error (The template root requires exactly one element  vue/valid-template-root)

* Update PR: update message on ui

* Update PR: Disable rebootVM and create vm/volume snapshot for KVM VMs

* Update PR: add more unit tests
2023-01-04 13:01:10 +01:00
Harikrishna d83c70cd25
User data as first class resource (#193)
* UserData as first clas resource

* Few fixes

* Added userdata id in deploy VM flow

* Fixed userdata append scenario between template userdata and user provided userdata

* UI: added a new section for userdata

* Added userdata details to the deployVM cmd flow

* Write userdata details into metadata VR

* Updated template response and views

* UI: added userdata id to deloyvm wizard

* Added userdata list to deploy VM form

* Added userdata params to registeruserdata UI form

* Small fixes and added userdata to updateVM flow

* Fixed unit tests and imports

* Userdata Navigation in template view

* Added userdata denyoverride flag and userdata params section

* Added ToolTips and fixes deploy VM, register userdata form

* added userdata policy list to register template form

* Allow override append of userdata in deploy VM flow

* update userdata linking to template

* Few UI fixes in deploy VM and edit template form

* fixes in deploy VM form to support deny userdata policy

* Added unit tests for userdata

* Added unit tests for linking userdata to template

* Remove unused imports

* Move test file to proper files

* Fix unused imports

* Fix Userdata delete flow

* Few improvements in the code

* Adding marvin tests for userdata

* Fixed marvin tests for registerd userdata

* Added few more marvin tests for userdata

* Few code fixes

* Few more code fixes

* Added userdata details to register and upload iso forms

* Added userdata selection in deploy VM form for ISOs

* Add comments section to userdata

* Added new API to reset UserData of a VM along with UpdateVM API

* Added new UnitTests for resetVMUserData

* Added resetVMUserdata in UI for stopped VM

* Added blank values for userdata in edit template form

* Added ISO id to the linkuserdatatotemplate API

* Added validation to userdata params so that it wont contain any VR metadafile names

* Removed required param for iso id in linkuserdatatotemplate cmd

* Added length to userdata param

* remove delete cascade on user_vm and vm_template tables for userdata id foreign key reference.

* Fix custom userdata params for config drive by adding userdata file name and value metadata json

* Fix marvin test case

* added comments to marvin test cases

* Fix document link in UI

* Added a check while deleting the userdata, to see if no VMs are using the userdata

* Added unit tests

* Removed labels added during merge

* added success message for userdata registration

* Added Schema changes to 4160 to 4161 upgrade path

* Fixed imports and some errors

* Fix service offering uuid in mysql view

* UI changes wrt to 4.16 branch mostly related to vue2

* Some UI fixes

* Register userdata and update template form fixes

* Fixed compute.js related to userdata

* UI fixes and user_vm_view wrt sshkey

* Fixed update template form

* Fix deploy VM and userdata reset forms

* Fixed Register and upload template and ISO forms

* Fixed getting params for userdata from template in instance creation form

* Removed CloudZonesNetworkElement.java as part of rebase, which is actually removed in https://github.com/shapeblue/cloudstack-apple/pull/191

* Fixed userdata selection in deploy vm and reset userdata vm forms

* Fixed method calls after rebase
2022-10-03 15:54:39 +05:30
dahn 5f63054ee0
Resource reservations (#192)
* reservation table

* try-with-resource

* domain added to reservation

* integration test for parallel vm creations
2022-09-15 09:06:53 +02:00
Nicolas Vazquez 00d8a13a52
Apple FR64: Standardise API driven console access (#188)
* Console access enhancements

* Remove extra logging

* Fix security hotspot

* Fix sonar cloud code smells

* Refactor API response

* Minor fix

* Refactor and increase timeout on ssh to cpvm

* Add marvin tests and extend permissions

* Fix account type

* Add unit tests

* Check vncport file exits on CPVM before attempting to add rules

* Change how vncport is read on cpvm

* Extra validation refactor

* Fix wrong token API param on UI

* Refactor vnc port selection to 8080 or 8443

* Do not display the input token modal and improve error message on console

* Improve error message and prevent opening blank popup when errors

* Fix logging exception due to algorithm
2022-09-07 17:47:13 +05:30
Marcus Sorensen ac3f3c8799
Allow option of exposing VM domain info via instance metadata (#191)
Signed-off-by: Marcus Sorensen <mls@apple.com>

Signed-off-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2022-09-06 08:47:11 -06:00
Marcus Sorensen ba7adfa6f0
Volume encryption (#135)
This PR introduces volume encryption option to service offerings and disk offerings. Fixes #136

There is a hypervisor component and a storage pool component. Hypervisors are responsible for being capable of running/using the encrypted volumes. Storage pools are responsible for being able to create, copy, resize, etc. Hypervisors will report encryption support in their details, storage pools are marked for encryption support by pool type.

The initial offering for experimental release of this feature will have support for encryption on Local, NFS, SharedMountPoint, and ScaleIO storage types.

When volumes choosing an encrypted offering are allocated to a pool, the pool type must be capable of supporting encryption and this is enforced.

When VMs are started and they have an encrypted volume, the hypervisor must be capable of supporting encryption. Also, if volumes are attached to running VMs, the attach will only work if the hypervisor supports encryption.

This change includes a few other minor changes - for example the ability to force the KVM hypervisor private IP. This was necessary in my testing of ScaleIO, where the KVM hypervisors had multiple IPs and the ScaleIO storage only functions if the hypervisor as a ScaleIO client matches IPs with what CloudStack sees as the hypervisor IP.

For experimental release of this feature, some volume workflows like extract volume and migrate volume aren't supported for encrypted volumes. In the future we could support these, as well as migrating from unencrypted to encrypted offerings, and vice versa.

It may also be possible to configure encryption specifics in the future, perhaps at the pool level or the offering level. Currently, there is only one workable encryption offering for KVM that is supported by Libvirt and Qemu for raw and qcow2 disk files, LUKS version 1. This PR ensures we at least store this encryption format associated with each volume, with the expectation that later we may have LUKS v2 volumes or something else. Thus we will have the information necessary to use each volume with Libvirt if/when other formats are introduced.

I think the most disruptive change here is probably a refactoring of the QemuImg utility to support newer flags like --object. I've tested the change against the basic Qemu 1.5.3 that comes with EL7 and I believe it is good, but it will be nice to see the results of some functional tests. Most of the other changes are limited to changing behavior only if volume encryption is requested.

Working on documentation for the CloudStack docs. One thing to note is that hypervisors that run the stock EL7 version of Qemu will not support encryption. This is tested to be detected and report properly via the CloudStack API/UI. I intend to like to have a support matrix in the CloudStack docs.

I may add a few more unit tests. I'd also like some guidance on having functional tests. I'm not sure if there's a separate framework, or if Marvin is still used, or what the current thing is.

* Add Qemu object flag to QemuImg create

* Add apache license header to new files

* Add Qemu object flag to QemuImg convert

* Set host details if hypervisor supports LUKS

* Add disk encrypt flag to APIs, diskoffering

* Schema upgrade 4.16.0.0 to 4.16.1.0 to support vol encryption

* Add Libvirt secret on disk attach, and refer to it in disk XML

* Add implementation of luks volume encryption to QCOW2 and RAW disk prep

* Start VMs that have encrypted volumes

* Add encrypt option to service offering and root volume provisioning

* Refactor volume passphrase into its own table and object

* CryptSetup, use key files to pass keys instead of command line

* Update storage types and allocators to select encryption support

* Allow agent.properties to define the hypervisor's private IP

* Implement createPhysicalDisk for ScaleIOStorageAdaptor

* UI: Add encrypt options to offerings

* UI module security updates

* Revert "UI module security updates" - belongs in base

This reverts commit a7cb7cf7f57aad38f0b5e5d67389c187b88ffd94.

* Add --target-is-zero support for QemuImg

* Allow qemu image options to be passed, API support convert encrypted

* Switch hypervisor encryption support detection to use KeyFiles

* Fixes for ScaleIO root disk encryption

* Resize root disk if it won't fit encryption header

* Use cryptsetup to prep raw root disks, when supported

* Create qcow2 formatting if necessary during initial template copy to ScaleIO

* Allow setting no cache for qemu-img during disk convert

* Use 1M sparse on qemu-img convert for zero target disks

* UI: Add volume encryption support to hypervisor details

* QemuImg use --image-opts and --object depending on version

* Only send storage commands that require encryption to hosts that support encryption

* Move host encryption detail to a static constant

* Update host selection to account for volume encryption support

Only attach volumes if encryption requirements are met

* Ensure resizeVolume won't allow changing encryption

* Catch edge cases for clearing passphrase when volume is removed

* Disable volume migration and extraction for encrypted volumes

* Register volume secret on destination host during live migration

* Fix configdrive path editing during live migration

* Ensure configdrive path is edited properly during live migration

* Pass along and store volume encryption format during creation

* Fixes for rebase

* Fix tests after rebase

* Add unit tests for DeploymentPlanningManagerImpl to support encryption

* Deployment planner tests for encryption support on last host

* Add deployment tests for encryption when calling planner

* Added Libvirt DiskDef test for encryption details

* Add test for KeyFile utility

* Add CryptSetup tests

* Add QemuImageOptionsTest

* add smoke tests for API level changes on create/list offerings

* Fix schema upgrade, do disk_offering_view first

* Fix UI to show hypervisor encryption support

* Load details into hostVO before trying to query them for encryption

* Remove whitespace in CreateNetworkOfferingTest

* Move QemuImageOptions to use constants for flag keys

* Set physical disk encrypt format during createDiskFromTemplate in KVM Agent

* Whitespace in AbstractStoragePoolAllocator

* Fix whitespace in VolumeDaoImpl

* Support old Qemu in convert

* Log how long it takes to generate a passphrase during volume creation

* Move passphrase generation to async portion of createVolume

* Revert "Allow agent.properties to define the hypervisor's private IP"

This reverts commit 6ea9377505f0e5ff9839156771a241aaa1925e70.

* Updated ScaleIO/PowerFlex storage plugin to support separate (storage) network for Host(KVM) SDC connection. (#144)

* Added smoke tests for volume encryption (in KVM). (#149)

* Updated ScaleIO pool unit tests.

* Some improvements/fixes for code smells (in ScaleIO storage plugin).

* Updated review changes for ScaleIO improvements.

* Updated host response parameter 'encryptionsupported' in the UI.

* Move passphrase generation for the volume to async portion, while deploying VM (#158)

* Move passphrase generation for the volume to async portion, while deploying VM.
* Updated logs, to include volume details.

* Fix schema upgrade, create passphrase table first

* Fixed the DB upgrade issue (as noticed in the logs below.)
DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLException: Failed to open the referenced table 'passphrase'
ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script

* Fixes for snapshots with encrypted qcow2
Fixes #159 #160 #163

* Support create/delete encrypted snapshots of encrypted qcow2 volumes
* Select endpoints that support encryption when snapshotting encrypted volumes
* Update revert snapshot to be compatible with encrypted snapshots
* Disallow volume and template create from encrypted vols/snapshots

* Disallow VM memory snapshots on encrypted vols. Fixes #157

* Fix for TemplateManagerImpl unit test failure

* Support offline resize of encrypted volumes. Fixes #168

* Fix for resize volume unit tests

* Updated libvirt resize volume unit tests

* Support volume encryption on kvm only, and  passphrase generation refactor (#169)

* Fail deploy VM when ROOT/DATA volume's offering has encryption enabled, on non-KVM hypervisors
* Fail attach volume when volume's offering has encryption enabled, on non-KVM hypervisors
* Refactor passphrase generation for volume

* Apply encryption to dest volume for live local storage migration
fixes #161

* Apply encryption to data volumes during live storage migration

Fixes #161

* Use the same encryption passphrase id for migrating volumes

* Pass secret consumer during storage migration prepare

Fix for #161

* Fixes create / delete volume snapshot issue, for stopped VMs

* Block volume snapshot if encrypted and VM is running

Fixes #159

* Block snap schedules on encrypted volumes

Fix for #159

* Support cryptsetup where luks type defaults to 2

Fixes #170

* Modify domain XML secret UUID when storage migrating VM

Fix for #172

* Remove any libvirt secrets on VM stop and post migration

Fix for #172

* Update disk profile with encryption requirement from the disk offering (#176)

Update disk profile with encryption requirement from the disk offering
and some code improvements

* Updated review changes / javadoc in ScaleIOUtil

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-06-29 15:51:00 +05:30
Abhishek Kumar 92931aeeb8
schema,server,api: events improvement (#127)
Add resource ID and resource type to event.
In UI, adds Events tab in resource view for the supporting resources.

Following SQL changes needed to support events with resource details in DB,
```
 -- Alter event table to add resource_id and resource_type
ALTER TABLE `cloud`.`event`
    ADD COLUMN `resource_id` bigint unsigned COMMENT 'ID of the resource associated with the even' AFTER `domain_id`,
    ADD COLUMN `resource_type` varchar(32) COMMENT 'Account role in the project (Owner or Regular)' AFTER `resource_id`;

DROP VIEW IF EXISTS `cloud`.`event_view`;
CREATE VIEW `cloud`.`event_view` AS
    SELECT
        event.id,
        event.uuid,
        event.type,
        event.state,
        event.description,
        event.resource_id,
        event.resource_type,
        event.created,
        event.level,
        event.parameters,
        event.start_id,
        eve.uuid start_uuid,
        event.user_id,
        event.archived,
        event.display,
        user.username user_name,
        account.id account_id,
        account.uuid account_uuid,
        account.account_name account_name,
        account.type account_type,
        domain.id domain_id,
        domain.uuid domain_uuid,
        domain.name domain_name,
        domain.path domain_path,
        projects.id project_id,
        projects.uuid project_uuid,
        projects.name project_name
    FROM
        `cloud`.`event`
            INNER JOIN
        `cloud`.`account` ON event.account_id = account.id
            INNER JOIN
        `cloud`.`domain` ON event.domain_id = domain.id
            INNER JOIN
        `cloud`.`user` ON event.user_id = user.id
            LEFT JOIN
        `cloud`.`projects` ON projects.project_account_id = event.account_id
            LEFT JOIN
        `cloud`.`event` eve ON event.start_id = eve.id;
```
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-05-05 13:44:33 +05:30
Nicolas Vazquez d6b9c65ce6
Backport: Direct download certificates additions and improvements (#6104) (#140)
* Direct download certificates additions and improvements (#6104)

* Add direct download certificates listing

* Restore class to original project

* Small refactor

* Register API

* Apply suggestions from code review

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Refactor after review

* Fix checkstyle

* Add hosts mapping to API response

* Improvements on revoke certificate

* Refactor revoke certificate API

* Fix condition

* Filter only certificates not revoked for revokeCertificate API

* Improve upload certificate and add provision certificate API

* Improve certificate response output

* Address review comments

* Refactor revoke cert test

* Fix marvin test

* Address review comments

* Fix issues

* Improvements

* Refactor upload template API response

* Fix response

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Fix pom version

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2022-04-12 10:49:08 -03:00
Pearl Dsilva e0a5df50ce
CKS Enhancements and SystemVM template upgrade improvements (#5863)
* This PR/commit comprises of the following:
- Support to fallback on the older systemVM template in case of no change in template across ACS versions
- Update core user to cloud in CKS
- Display details of accessing CKS nodes in the UI - K8s Access tab
- Update systemvm template from debian 11 to debian 11.2
- Update letsencrypt cert
- Remove docker dependency as from ACS 4.16 onward k8s has deprecated support for docker - use containerd as container runtime

* support for private registry - containerd

* Enable updating template type (only) for system owned templates via UI

* edit indents

* Address comments and move cmd from patch file to cloud-init runcmd

* temporary change

* update k8s test to use k8s version 1.21.5 (instead of 1.21.3 - due to https://github.com/kubernetes/kubernetes/pull/104530)

* support for private registry - containerd

* Enable updating template type (only) for system owned templates via UI

* smooth upgrade of cks clusters

* update pom file with temp download.cloudstack.org testing links

* fix pom

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* update to official links
2022-02-15 18:27:14 +05:30
dahn 122b018057
Quota test fixes (#5959)
* fixes and cleanups on quota integration test

* text changes in exception messages

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2022-02-10 18:57:21 +05:30
Wei Zhou ce81a8e708
test: sleep 30s after restarting mgt server in test_kubernetes_supported_versions.py to fix test failures with test_secondary_storage.py (#5962) 2022-02-09 09:37:28 +05:30
Abhishek Kumar 62b806ac13
api,server: add params for updatehypervisorcapabilities API (#5473)
* api,server: add params for updatehypervisorcapabilities API

Allows updating following capabilities for a hypervisor, version:
- Max DATA volumes limit
- Storage motion supported
- Max hosts per cluster
- VM snapshot enabled

* added test

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update test/integration/smoke/test_hypervisor_capabilities.py

Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-02-03 10:21:23 +05:30
Wei Zhou fadd74aaca
network: fix vm can be deployed on L2 network of other accounts (#5784)
* Update #5769: fix domain admin can deploy vm on L2 network of other users

* test: fix test_storage_policy.py

* Update #5784: revert part of changes in #2420
2022-01-11 12:16:00 +05:30
Rakesh 2bd1dc1e14
Enable resetting config values to default value (#4230)
* Enable resetting config values to default value

Provide reset button to zone,cluster,domain,account,
primary and secondary storage so that config values
can be reset to default value

* fix ui issue

* Update test/integration/smoke/test_reset_configuration_settings.py

* Update test/integration/smoke/test_reset_configuration_settings.py

Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-01-03 21:55:30 +01:00
Wei Zhou 4ba2ad5397
test: fix component test test_configdrive.py (#5528)
* test: fix test_configdrive.py and move it to smoke test

(sleep 30s as vm crashes after migration in ubuntu 20.04)

* Update test/integration/smoke/test_configdrive.py

Co-authored-by: dahn <daan.hoogland@gmail.com>

* test: fix test_configdrive.py

* Update #5528: move back to component test

Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-12-30 16:31:31 +05:30
dahn f03ed8c721
cleanup of unused code and cleanup of cleanup procedure (#5562)
* cleanup of unused code and cleanup of cleanup procedure

* remove unnecessary egress rule

* remove rule and add rule removal

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-12-23 10:10:38 +05:30
davidjumani 75a8c8ff7c
Fixing error in kube smoke tests (#5585) 2021-10-18 16:10:40 +05:30
davidjumani 6ac834a358
Adding AutoScaling for cks + CKS CoreOS EOL update + systemvmtemplate improvements (#4329)
Adding AutoScaling support for cks
Kubernetes PR : kubernetes/autoscaler#3629
Also replaces CoreOS with Debian
Fixes #4198

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-10-06 21:17:41 +05:30
Wei Zhou 09fce75047
OVS/GRE: bug fixes (#5446)
* OVS/GRE: fix failed to fetch cloudbr* info on host

* OVS/GRE: move ovstunnel.py to python3

* OVS: add test_privategw_acl_ovs_gre.py for automatic testing

(1) VR <-> VM should work
(2) Private GW should work

* OVS: fix an issue on xenserver

ovs bridges are deleted by xenserver/ovs automatically
```
[root@ref-trl-1797-x-M7-wei-zhou-xs2 ~]# grep -r xapi7 /var/log/ |grep del-br
/var/log/xensource.log:Sep 15 07:13:44 ref-trl-1797-x-M7-wei-zhou-xs2 xcp-networkd: [ info|localhost|611 |org.xen.xapi.xenops.classic events D:4a3d931cd89f|network_utils] /usr/bin/ovs-vsctl --timeout=20 -- --if-exists del-br xapi7
/var/log/daemon.log:Sep 15 07:13:45 ref-trl-1797-x-M7-wei-zhou-xs2 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=20 -- --if-exists del-br xapi7
```

which results that xe network exists but bridge does not exist, and operation stuck for 20 minutes at
```
2021-09-15 16:06:56    DEBUG [root] #### VMOPS enter  create_tunnel ####
2021-09-15 16:06:56    DEBUG [root] Creating tunnel from host 2 to host 1 with GRE key 2116
2021-09-15 16:06:56    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=0', 'wait-until', 'bridge', 'xapi7', '--', 'get', 'bridge', 'xapi7', 'name']
2021-09-15 16:26:56    DEBUG [root] bridge xapi7 for creating tunnel - VERIFIED
2021-09-15 16:26:56    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi7', 't2116-2-1', '--', 'set', 'interface', 't2116-2-1', 'type=gre', 'options:key=2116', 'options:remote_ip=10.0.34.230']
```

* OVS/GRE: component test test_vpc_distributed_routing_offering.py passes on xenserver

* OVS: support Shared network with GRE protocol

* update #5446: use lang3.StringUtils

* OVS/GRE: VPC with distributed routing on kvm (still not working)
2021-10-03 14:47:52 +05:30
Rakesh dcc02e0fbb
Enable account settings to be visible under domain settings (#4215)
* Enable account settings to be visible under domain settings

All the account settings can't be configured under domain
level settings right now.
By default, if account setting is not configured then
its value will be taken from global setting.
Add a global setting "enable.account.settings.for.domain"
so that if its enabled then all the account level settings
will be visible under domain levelsettings also.
If account level setting is configured then that value will
be considered else it will take domain scope value. If
domain scope value is not configured then it will pick
it up from global setting.

If domain level setting is not configured then by default
the value will be taken from global setting
Add another global setting "enable.domain.settings.for.child.domain"
so that when its true, if a value for domain setting is not
configured then its parent domain value is considered until
it reaches ROOT domain. If no value is configured till ROOT
domain then global setting value will be taken.

Also display all the settings configured under the domain level
in list domains api response

* rename variables

Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
2021-09-29 10:29:20 +02:00
nicolas 9de5ef9f2b
Merge branch '4.15' into main 2021-09-24 11:32:38 -03:00
Wei Zhou 6aa75cf671
test_vpc_redundant.py: reduce sleep time from 1 hour to 21 mins (#5503) 2021-09-24 11:31:47 -03:00
Pearl Dsilva d1f375f35a
marvin: Refactor - cleanup of resource after test run (#5505) 2021-09-24 13:42:15 +05:30
Pearl Dsilva 6ba656b664
marvin: Fix intermittent failure observed in test_02_list_snapshots_with_removed_data_store (#5483)
* marvin: Fix intermittent failure observed in test_02_list_snapshots_with_removed_data_store

* refactor test
2021-09-23 00:37:25 +05:30
dahn 3208929a97
move out broken tests (#5495)
Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-09-22 14:47:18 +05:30
Rohit Yadav 847078b3c8 Merge remote-tracking branch 'origin/4.15' into main 2021-09-22 14:38:56 +05:30
Wei Zhou 7444bc6cd5
travis: fix test/integration/component/test_public_ip.py (#5486) 2021-09-22 09:55:13 +02:00
dahn 35990d7ca3
tests: move misc tests to broken (#5456)
Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-09-16 11:10:59 +05:30
Peinthor Rene 66c39c1589
storage: Linstor volume plugin (#4994)
This adds a volume(primary) storage plugin for the Linstor SDS.
Currently it can create/delete/migrate volumes, snapshots should be possible,
but currently don't work for RAW volume types in cloudstack.

* plugin-storage-volume-linstor: notify libvirt guests about the resize
2021-09-16 10:50:58 +05:30
Nicolas Vazquez 413d10dd81
server: Extend the Annotations framework (#5103)
* Extend addAnnotation and listAnnotations APIs

* Allow users to add, list and remove comments

* Add adminsonly UI and allow admins or owners to remove comments

* New annotations tab

* In progress: new comments section

* Address review comments

* Fix

* Fix annotationfilter and comments section

* Add keyword and delete action

* Fix and rename annotations tab

* Update annotation visibility API and update comments table accordingly

* Allow users seeing all the comments for their owned resources

* Extend comments for volumes and snapshots

* Extend comments to multiple entities

* Add uuid to ssh keypairs

* SSH keypair UI refactor

* Extend comments to the infrastructure entities

* Add missing entities

* Fix upgrade version for ssh keypairs

* Fix typo on DB upgrade schema

* Fix annotations table columns when there is no data

* Extend the list view of items showing they if they have comments

* Remove extra test

* Add annotation permissions

* Address review comments

* Extend marvin tests for annotations

* updating ui stuff

* addition to toggle visibility

* Fix pagination on comments section

* Extend to kubernetes clusters

* Fixes after last review

* Change default value for adminsonly column

* Remove the required field for the annotationfilter parameter

* Small fixes on visibility and other fixes

* Cleanup to reduce files changed

* Rollback extra line

* Address review comments

* Fix cleanup error on smoke test

* Fix sending incorrect parameter to checkPermissions method

* Add check domain access for the calling account for domain networks

* Fix only display annotations icon if there are comments the user can see

* Simply change the Save button label to Submit

* Change order of the Tools menu to provent users getting 404 error on clicking the text instead of expanding

* Remove comments when removing entities

* Address review comments on marvin tests

* Allow users to list annotations for an entity ID

* Allow users to see all comments for allowed entities

* Fix search filters

* Remove username from search filter

* Add pagination to the annotations tab

* Display username for user comments

* Fix add permissions for domain and resource admins

* Fix for domain admins

* Trivial but important UI fix

* Replace pagination for annotations tab

* Add confirmation for delete comment

* Lint warnings

* Fix reduced list as domain admin

* Fix display remove comment button for non admins

* Improve display remove action button

* Remove unused parameter on groupShow

* Include a clock icon to the all comments filter except for root admin

* Move cleanup SQL to the correct file after rebasing main

Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
2021-09-08 10:14:06 +05:30
dahn 571a361926
tests: component test ports/fixes in python3 (#5082)
* cleanup plan

* more robust cleanup within member method

* ss_domain_limits fixed

* egress 'icmp' instead of 'all' and cleanup

* configdrive syntax, fixed to be up to par with py2

* cleanup fixed the lb secondary ip tests?

* deal with different base64 encoding in py3 for userdata

* cleanup of multiple_ips_per_nic

* cleanup and reformat of test_volumes

* cleanup and fixes for test_ps_domain_limits.py

* cleanup and fix of test_ps_limits.py

* fix occasional match of float against int

* cleanup and fix test_ps_resize_volume.py

* cleanup and fix test_snapshots

* cleanup ss_max_limits and fix for float vs int problem in API

* mere cleanup of test_volume_destroy_recover

* add missing command creation

* cleanup of test_vpc_on_host_maintenance

* cleanup test_vpc_network

* cleanup, comments and logging in test_vpc_network_lbrules

* cleanup of test_vpc_network_pfrules

* cleanup and format code for test_vpc_network_staticnatrule

* f string instead of conversion specifiers

* check http and ssh fix and cleanup (for vpc pfrules tests)

* generalise create network method

* make ip optional in creating webserver

* remove unused code and add rules to enable ssh

* more cleanup

* remove unused code and cleanup

* small cleanup, mostly precarous run environment required

* cleanup and removed unused code

* advancedsg only, cleanup, pulled in services

* reformat/cleanup

* log result of error after verify rule

* add nw_off_no_services

* tags=["TODO"] for escalations_networks

* tags=["TODO"] for organization_states

* tags=["TODO"] for browse_templates

* tags=["TODO"] for configdrive

* tags=["TODO"] for vpc_vms_deployment

* add remove network cleanup and fixes

* move tests that fail on all platforms out of the way

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-09-02 16:03:49 +05:30
davidjumani 8bc120b288
Merge remote-tracking branch 'apache/4.15' into main 2021-08-24 13:46:28 +05:30
davidjumani f822547c3b
securitygroups: Fix security_groups for c8/suse (#5354)
* Fix security_groups for c8/suse

* security_group.py: revert a minor change in #5348

Co-authored-by: Wei Zhou <weizhou@apache.org>
2021-08-24 13:07:52 +05:30
Rohit Yadav 57a55bd777 Merge remote-tracking branch 'origin/4.15' into main 2021-08-20 17:19:40 +05:30
Wei Zhou f9fe9da734
security group: fix component test test_multiple_nic_support.py failures (#5348)
This PR fixes component test failure with test_multiple_nic_support.py, in advanced zone with security groups.

Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-08-20 17:06:26 +05:30
Pearl Dsilva 25206e4fd3
test: Frix travis failure - test_outofbandmanagement.py (#5346) 2021-08-20 13:00:34 +02:00
Rohit Yadav 953f5b81ed Merge remote-tracking branch 'origin/4.15' into main 2021-08-18 16:56:19 +05:30
davidjumani 21d7854274
s2svpn: Set initial state as Connecting (#5327)
* s2svpn: Set initial state as Connecting

* Fix comments

* Adding connecting elsewhere
2021-08-18 16:50:38 +05:30
davidjumani 98d3231dbf
Adding support for RHEL8 binary-compatible variants (#5158)
* Adding support for RHEL binary-compatible variants

* Skip ipmi related tests

* Fixing security_groups.py
2021-08-18 10:03:03 +02:00
DK101010 664a46a525
PR multi tags in compute offering [#4398] (#4399)
* [#4398] adapt code to handle multi tag string with commas

* [#4398] remove trailing spaces

* [#4398] add multi host tag support for ingest process

* [#4398] add test for multi tag support in offerings

* [#4398]  update multitag support for DeploymentPlanningManagerImpl

encapsulate multi tag check from Ingest Feature, DepolymentPlanningManager into
HostDaoImpl to prevent code duplicates

* [#4398] move logic to HostVO and add tests

* rename test method

* [#4398] Change string method to apaches StringUtils

* [#4398] modify test for multi tag support

* adapt sql for double tags

Co-authored-by: Dirk Klahre <Dirk.Klahre@Itelligence.de>
2021-08-16 12:08:40 -03:00
Abhishek Kumar 6581a1bccd
test/vmware: add live migratevmwithvolume test and fix (#5289)
Co-authored-by: Gabriel Brascher <gabriel@apache.org>
2021-08-11 13:52:05 +02:00
davidjumani 3c8c704df3
kubernetes: Deploy kubernetes-provider when creating a cluster (#5254)
* kubernetes: Deploy kubernetes-provider when creating a cluster

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-08-09 14:45:20 +05:30
slavkap d6a77a72f0
Allow updating the storage/host tags of service offerings (#5043) 2021-08-02 15:48:07 +02:00
DK101010 c23cbdff8d
Remove condition that are prevent resizing for root volumes (vmware) (#5186)
* remove condition that are prevent resizing for root volumes

* adjust tests acordingly

* enable disk resize tests for VMware

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: mib1185 <mail@mib85.de>
2021-07-21 22:33:51 -03:00
Spaceman1984 96c9c5a5e2
Added disk provisioning type support for VMWare (#4640)
* Added disk provisioning type support for VMWare

* Review changes

* Fixed unit test

* Review changes

* Added missing licenses

* Review changes

* Update StoragePoolInfo.java

Removed white space

* Review change - Getting disk provisioning strictness setting using the zone id and not the pool id

* Delete __init__.py

* Merge fix

* Fixed failing test

* Added comment about parameters

* Added error log when update fails

* Added exception when using API

* Ordering storage pool selection to prefer thick disk capable pools if available

* Removed unused parameter

* Reordering changes

* Returning storage pool details after update

* Removed multiple pool update, updated marvin test, removed duplicate enum

* Removed comment

* Removed unused import

* Removed for loop

* Added missing return statements for failed checks

* Class name change

* Null pointer

* Added more info when a deployment fails

* Null pointer

* Update api/src/main/java/org/apache/cloudstack/api/BaseListCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Small bug fix on API response and added missing bracket

* Removed datastore cluster code

* Removed unused imports, added missing signature

* Removed duplicate config key

* Revert "Added more info when a deployment fails"

This reverts commit 2486db78dc.

Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-07-16 22:37:42 -03:00
Abhishek Kumar 5ac184edd6 Merge remote-tracking branch 'apache/4.15' into main 2021-07-15 14:02:49 +05:30
Wei Zhou cf0f1feb5e
configdrive: fix some failures in tests/component/test_configdrive.py (#5144)
* server: fix failed to apply userdata when enable static nat

* server: fix cannot expunge vm as applyUserdata fails

* configdrive: fix ISO is not recognized when plug a new nic

* configdrive: detach and attach configdrive ISO as it is changed when plug a new nic or migrate vm

* configdrive test: (1) password file does not exists in recreated ISO; (2) vm hostname should be changed after migration

* configdrive: use centos55 template with sshkey and configdrive support

* configdrive: disklabel is 'config-2' for configdrive ISO

* configdrive: use copy for configdrive ISO and move for other template/volume/iso

* configdrive: use public-keys.txt

* configdrive test: fix (1) update_template ; (2) ssh into vm by keypair
2021-07-15 12:49:37 +05:30
davidjumani 4645863b25
tests: Fix test failures for Local storage and Basic zones (#5106)
tests: Fix test failures for Local storage and Basic (SG) zones

* Skip migrate vm when localstorage enabled
* Fix test_direct_download.py
* Skip test_03_create_network_domain_network_offering in basic zones
* Skip test_03_create_vpc_domain_vpc_offering in basic zones
* Skip test_01_add_primary_storage_disabled_host if localstorageenabled
2021-07-01 09:45:21 +05:30
dahn 7835c08120
tests: formatting and cleanup fixes for test_volumes (#5146)
* (auto) formatting and cleanup fixes

* ssh fail and detach in all cases

* message format and removed overcleaning

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-06-29 13:06:08 +05:30