Commit Graph

4282 Commits

Author SHA1 Message Date
Rohit Yadav 840c3f6a7a Merge remote-tracking branch 'origin/4.17' 2022-08-10 23:11:09 +02:00
Harikrishna d3ec27dc79
Fixed Veeam listing restore points (#6555)
Fixes issue #6465 where listing backup restore points are failing with Veeam version v11.0.1.1261.

Though this version is not fully supported for backup and recovery, existing backups, restore points for the VMs can continue to work with the Veeam version v11.0.1.1261. I've created a separate ticket here to fully support the version #6554
2022-08-10 16:59:10 +05:30
slavkap 76f52af8f3
removed the use of SharedMountPoint storage type for the StorPool plugin (#6552)
Fixes #6455

The default storage adaptor - LibvirtStorageAdaptor - is used by different storage types and doesn't use the annotation @StorageAdaptorInfo. In this case, a storage plugin that wants to adopt one of the predefined storage pool types will override the default behaviour. If fixing the issue in general (for new storage plugins or current ones that want to reuse the existing storage pool types) would affect all volume/snapshot/VM cases. This will lead to the need of extensive testing for each storage plugin for which we don't have the resources to do it. That's why this patch fixes the old behaviour for the SharedMountPoint by adding a new storage pool type for the StorPool plugin.
2022-08-10 14:41:32 +05:30
João Jandre 9c63c39371
Add new parameter to createLoadBalancerRule API (#6460)
* Add new parameter to createLoadBalancerRule API

* address review

Co-authored-by: João Paraquetti <joao@scclouds.com.br>
2022-08-08 10:48:21 +02:00
John Bampton f9347ecf2c
Fix spelling (#6597) 2022-08-03 15:43:47 +05:30
Rohit Yadav 5f04018bf0 Merge remote-tracking branch 'origin/4.17' 2022-07-27 12:41:31 +02:00
Rohit Yadav 441edf3ca7
utils: use safer parsing utility across codebase (#6562)
This addresses SonarQube/SonarCloud quality checks to use safer xml
parser to resist potential XXE attacks.

https://sonarcloud.io/organizations/apache/rules?open=java%3AS2755&rule_key=java%3AS2755

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-27 14:31:51 +05:30
SadiJr 61e4e862c4
[Veeam] externalize restore timeout (#6320)
* [Veeam] add global timeout configuration for backup restore process

* Use 'this'

* Address reviews

* Address reviews

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-07-21 12:17:13 +05:30
Rohit Yadav c4c4c71591 cherry-pick ce7c3694c8
This fixes cherry-pick issue, while merging 4.17.0.1 on 4.17 branch

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-20 16:18:32 +05:30
Harikrishna d4460a8afc
Scope setting changes in ldap and utils pom.xml files (#6557) 2022-07-20 13:42:44 +05:30
Rohit Yadav 661956cc60 Merge remote-tracking branch 'origin/4.17' 2022-07-20 11:52:26 +05:30
Harikrishna 2c05b63495
kvm: Fix for Revert volume snapshot (#6527)
This PR fixes the issue #6209 where the snapshot revert operation fails after certain volume operations like Migrate VM with volume / migrate volume / reinstall VM.

The root cause of the issue after these volume operations, the primary storage entry is getting deleted for that volume. We have fixed it here to get the primary datastore entry wrt volume and continue the operation.
2022-07-20 11:34:02 +05:30
Bryan Lima 71bc088a70
Improve login time (#6412)
* Improve slow login

* Address review

* Address Daan's review

* Address Daniel reviews
2022-07-20 11:30:17 +05:30
Rohit Yadav 4baaf736b9 Merge remote-tracking branch 'origin/4.17'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-18 19:42:44 +05:30
Rohit Yadav 7a3e97d67e Tagging release 4.17.0.1 on branch b30a4a99d1.
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEXtHhEi3F6KSkURLCSEJIIQ7j2IQFAmLRYi0ACgkQSEJIIQ7j
 2ISTWxAAlozJuDMoRnr4D1TDbNCr2hzWSgVn5AK+IZGwnd22OnaZnS7tVQUheTCq
 t9aQgRLb7oUGAzNngHEjDaQBnxlHdLHMKby+QGe+RjX/d9urFoEyHe2xyvCJPkwM
 hFM1uesMqtH/HKwhIL3l8fATGPHlucdhQEZ+XA4bu91IVzxog0gikSnm7SjbaljF
 yYNkn9CgOWtZYFek7lcOM7iuKB79QSdpYxN8PYLpE7esyQSu4KjU4Ekufv1u6Tql
 ILsY5PA5tzzxS7ArfW5PICgSxkXOUIkflBbPHObGgduKw9Q36bmnRM/701lNb2re
 EWE4NMlM2PDn8kKZ2zULD2VBIq5tVdJuZjXbjDyD17z/KiU9pd6hGeHABSitnpDW
 vAS6rLJVY3YT9eqoVDVhpkpFQZmvdfDC8L4nYU2E7dCHj4lF9FlsgYO08SCfSgvP
 InAnfg1jZvbhA9EDL+LiuhxCStn6ZpjRuRCC89hYfRfRM1ZdrT2FazDj8KwPuC0P
 xfEr8eTnMm7xM+B9JCBQ2Lskl3jxQk3KAYQX13LtZCUj05Y1f3crx/iq6t0qIrAH
 PU9keojKMZffLz5MBlFU8qor32stw+uNMky8dZgtDIx6kRjnuYuPYOxpcPDzl+Cs
 KBRcwpIP+GR9mePU8PKBNDClLA45vDE1XqeK6KnOOf7MBSprU5o=
 =ETOD
 -----END PGP SIGNATURE-----

Merge tag '4.17.0.1' into 4.17

Tagging release 4.17.0.1 on branch b30a4a99d1.
2022-07-18 19:40:53 +05:30
Rohit Yadav 4ed1be821c Tagging release 4.16.1.1 on branch b7415bf127.
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEXtHhEi3F6KSkURLCSEJIIQ7j2IQFAmLQBOkACgkQSEJIIQ7j
 2IS+nRAAl8qEGHMCtbh64Uc517UwsATEZwEqm+s5cWrbi6GpkuJ00ITnchgK1QvY
 q9NQudL548oYWB8pvmeql8eeVJFGx4loh8e32GXdW5dNDcmIC/0YZ7VJFokPuHf6
 79GEDfcui5A15mvIL8DyTSHlZd/6x3LKusfM5Nu3f88B75yy1AkfxH8JcVTM8P9/
 ijtVTpy8zWkBWO+nnFUiwRjcQStOwNKd1jHxoapJIpFlNcUZw2DkRlaVIV7uU+ne
 Z7Y4JAJHzvki4ewkl/F5XwkRPiZlEMXVgEAb4dUmt5hg4GCWQvuDvHDio4fQ6Ws1
 CSNdiSV5rKMxa/fcE4l/oxvZ5oGxA7afbyJWo6Y4/s+UENKmZ3LiehkTptuOTSh0
 rgBTUKO7ZPtYuqD6kwaKYoxL431zYRJsdF1TUnts4nJTQQsWn6JlA3oTsX7nxlte
 qdxRtqmE7NTcpNH/+sU8MDKBYaHEqF3VOfzhw5Ta8ztQhebrGMHPJX0i3ypDBAll
 QEVH+cMpIoo5MQZWRFnIeKW/uTZuEZAMcJ8a/AS7gHSjLtiNGJExE5qvVXVnP8Im
 PruZSmk1ZovQ/XbtN0SD0DDS93CppYFH6fJRAeq6yqkEnYUm+dxFkBvEZqp8nr/X
 Z3vySZlH08+iz3fLlpbkGJRZSUfFXYKrGyZjLaFvKIf0FpedfOM=
 =/arX
 -----END PGP SIGNATURE-----

Merge tag '4.16.1.1' into 4.16

Tagging release 4.16.1.1 on branch b7415bf127.
2022-07-18 19:34:32 +05:30
Rohit Yadav 1c7efcbd0d Updating pom.xml version numbers for release 4.17.0.1
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-15 18:18:40 +05:30
dahn 731a83babf
add global setting to allow parallel execution on vmware (#6413)
* add global setting to allow parallel execution on vmware

* cleanup setting distribution for vmware.create.full.clone

* query setting in vmware guru

* don´t touch other hypervisor's commands

* guru hierarchy cleanup
2022-07-15 10:01:35 +02:00
Rohit Yadav ce7c3694c8 storpool: fix mvn pom.xml build issue
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-14 17:36:02 +05:30
Rohit Yadav f27de63644 saml: Safer DocumentBuilderFactory and ParserPool configuration
This implements safer DocumentBuilderFactory and ParserPool utilities
to be used throughout the codebase to prevent potential XXE exploits.

References:
https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html
https://www.blackhat.com/docs/us-15/materials/us-15-Wang-FileCry-The-New-Age-Of-XXE-java-wp.pdf

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 8e0e68ef368ebe2793ef80e2c3821eaecb47b593)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-14 17:31:12 +05:30
Rohit Yadav ebfdef5777 Updating pom.xml version numbers for release 4.16.1.1
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-14 17:28:08 +05:30
Rohit Yadav b7415bf127 saml: Safer DocumentBuilderFactory and ParserPool configuration
This implements safer DocumentBuilderFactory and ParserPool utilities
to be used throughout the codebase to prevent potential XXE exploits.

References:
https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html
https://www.blackhat.com/docs/us-15/materials/us-15-Wang-FileCry-The-New-Age-Of-XXE-java-wp.pdf

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 8e0e68ef368ebe2793ef80e2c3821eaecb47b593)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-14 16:58:08 +05:30
Rohit Yadav 35b5315dae
maven: update dependencies (#6539)
This upgrades mvn dependencies for the project.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-11 20:41:39 +05:30
Rohit Yadav 0cae4406fb Merge remote-tracking branch 'origin/4.17' 2022-07-07 14:37:34 +05:30
Rohit Yadav 3d316546bf
kvm: skip test that can't run and pass on M1 mac (#6537)
This fixes the build issue on M1 mac/osx.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-07 14:36:15 +05:30
Rohit Yadav 7ff3e7f968 Merge remote-tracking branch 'origin/4.17'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-07-06 12:53:45 +05:30
Abhishek Kumar 67e941f690
cks: fix k8s version upgrade (#6513)
Fixes #6514

On latest systemvm template used for CKS /usr/sbin is not present in the $PATH for normal user used during upgrade. This leads to failure for blkid command. Due to this during k8s version upgrade ISO is not being able to mount on the k8s cluster VMs and upgrade process is not carried out.
This PR fixes mounting of k8s version ISO and also returns failure for script when ISO mounting is failed.
Same failure is not seen during deployment of the cluster because setup-kube-system workflow is executed as ROOT user and it has a different value for $PATH.
From /etc/login.defs:

ENV_SUPATH	PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV_PATH	PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-07-06 12:34:49 +05:30
Rohit Yadav e57a0f9980 Merge remote-tracking branch 'origin/4.17' 2022-07-06 09:34:02 +05:30
Luis Moreira c6b611433b
saml: Fix SAML SSO plugin redirect URL (#6457)
This PR fixes the issue #6427 -> SAML request must be appended to an IdP URL as a query param with an ampersand, if the URL already contains a question mark, as opposed to always assume that IdP URLs don't have any query params.
Google's IdP URL for instance looks like this: https://accounts.google.com/o/saml2/idp?idpid=<ID>, therefore the expected redirect URL would be https://accounts.google.com/o/saml2/idp?idpid=<ID>&SAMLRequest=<SAMLRequest>

This code change is backwards compatible with the current behaviour.
2022-07-06 09:28:37 +05:30
Daan Hoogland a470f3353a Merge branch '4.17' 2022-07-05 09:11:45 +02:00
John Bampton 7d23a0a759
Fix spelling (#6272) 2022-07-05 09:08:53 +02:00
João Jandre e467ad02ce
Fix VMware memory retrieval (#6414)
* Fix VMware memory retrieval

* Address review

Co-authored-by: Joao <JoaoJandre@gitlab.com>
2022-07-01 23:36:40 -03:00
Rohit Yadav 6607a98597 Merge remote-tracking branch 'origin/4.17' 2022-07-01 10:44:29 +05:30
Harikrishna 637a102922
Fix for VMware VM migration with volume in local storage (#6483)
* Fix VMware VM migration with volume in case of local storage

* Break the loop once target host is found

* Code optimisations in getting the target host guid for local storage

* Fixed code smells and added unit test
2022-07-01 10:39:02 +05:30
Rohit Yadav 3fc825e246 Merge remote-tracking branch 'origin/4.17' 2022-06-30 12:17:29 +05:30
John Bampton 7ba43e447c
Enable flake8 rule W292 No newline at end of file (#6274)
https://www.flake8rules.com/rules/W292.html
2022-06-30 12:08:27 +05:30
Wei Zhou a9a2aa953c
CKS: add created to k8s cluster and k8s version (#6446)
This PR adds created info to the response of kubernetes clusters and kubernetes supported version.
2022-06-30 12:03:12 +05:30
SadiJr ea9124e49c
[KVM improve logs in migrate VM process (#6331)
* Add more logs to migrate VM process in KVM

* Remove unused imports

* Verify if debug is enable before write the log string

* Fix conflicts

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-06-28 08:58:19 +02:00
Wei Zhou ff7831d751 Merge remote-tracking branch 'apache/4.17' 2022-06-28 08:27:36 +02:00
Suresh Kumar Anaparti c70bc9d69c
kvm: Updated PowerFlex/ScaleIO storage plugin to support separate (storage) network for Hosts(KVM)/Storage connection. (#6367)
This PR enhances the existing PowerFlex/ScaleIO storage plugin to support separate (storage) network for Hosts(KVM)/Storage connection, mainly the SDC (ScaleIo Data Client) connection.
2022-06-27 14:42:51 +05:30
nvazquez 8333e8fbd7
Fix upgrade version on files 2022-06-06 12:26:52 -03:00
nvazquez 0bcc609f05
Updating pom.xml version numbers for release 4.18.0.0-SNAPSHOT
Signed-off-by: nvazquez <nicovazquez90@gmail.com>
2022-06-06 12:25:35 -03:00
nvazquez 7ed8d54db2
Fix version numbers 2022-06-06 12:22:11 -03:00
nvazquez 038a669d6b
Updating pom.xml version numbers for release 4.17.1.0-SNAPSHOT
Signed-off-by: nvazquez <nicovazquez90@gmail.com>
2022-06-06 12:19:44 -03:00
nvazquez c56220fcf2
Updating pom.xml version numbers for release 4.17.0.0
Signed-off-by: nvazquez <nicovazquez90@gmail.com>
2022-05-31 14:33:47 -03:00
David Jumani 0f4c3af935
cks: Fix when deployed on a nw without internet access (#6418)
* cks: Fix when deployed on a nw without internet access

* Revert "cks: Fix when deployed on a nw without internet access"

This reverts commit 40e3338001.

* cks: Fix issue when creating cluster in nw without internet access
2022-05-31 14:32:27 -03:00
Nicolas Vazquez 82e0d5d679
Fix UEFI detection on KVM and prevent deployments on non UEFI enabled hosts (#6423)
* Do not allow UEFI deployments on non UEFI enabled hosts

* Fix UEFI detection on KVM

* Refactor

* Improvement
2022-05-31 14:31:42 -03:00
Daniel Augusto Veronezi Salvador 81b7e6e750
Fix extract snapshot from vm snapshot on kvm (#6422)
* Fix extract snapshot from VM snapshot on KVM

* Fix validation expression - does not need to escape the slash

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2022-05-31 14:30:36 -03:00
dahn c123c3fd2f
remove request listener to prevent untimely session invalidation (#6393)
* login/-out constants

* no request listener

* store session as value, using id as key

* Apply suggestions from sonarcloud.io code review

three instances of unsafe parameters to logging

* new sonar issues

* sonar issues
2022-05-24 10:00:06 -03:00
nvazquez 96594aec28
Merge branch '4.16' 2022-05-23 08:16:52 -03:00
Nicolas Vazquez b1c8b5ab37
[KVM] Fix VM migration error due to VNC password on libvirt limiting versions (#6404)
* [KVM] Fix VM migration error due to VNC password on libvirt limiting versions

* Fix passwd value

* Simplify implementation
2022-05-23 08:12:49 -03:00
Nicolas Vazquez dc975dff95
[KVM] Enable IOURING only when it is available on the host (#6399)
* [KVM] Disable IOURING by default on agents

* Refactor

* Remove agent property for iouring

* Restore property

* Refactor suse check and enable on ubuntu by default

* Refactor irrespective of guest OS

* Improvement

* Logs and new path

* Refactor condition to enable iouring

* Improve condition

* Refactor property check

* Improvement

* Doc comment

* Extend comment

* Move method

* Add log
2022-05-23 08:11:14 -03:00
Nicolas Vazquez 363a2cff82
Backport: kvm: truncate vnc password to 8 chars (#6244) (#6402)
* kvm: truncate vnc password to 8 chars (#6244)

This PR truncates the vnc password of kvm vms to 8 chars to support latest versions of libvirt.

* Use lang3 string utils

Co-authored-by: Wei Zhou <weizhou@apache.org>
2022-05-20 11:36:43 -03:00
David Jumani 5fdd4d211d
cks: Get caller user keys if cluster belongs to project (#6392)
* cks: Get caller user keys if cluster belongs to project

* Refactor
2022-05-18 10:40:13 -03:00
Wei Zhou 8f39a049bb
agent: enable ssl only for kvm agent (not in system vms) (#6371)
* agent: enable ssl only for kvm agent (not in system vms)

* Revert "agent: enable ssl only for kvm agent (not in system vms)"

This reverts commit b2d76bad2e.

* Revert "KVM: Enable SSL if keystore exists (#6200)"

This reverts commit 4525f8c8e7.

* KVM: Enable SSL if keystore exists in LibvirtComputingResource.java
2022-05-12 07:01:55 -03:00
Marcus Sorensen 3dcb93d981
maven: Move apache DS dependencies to test scope (#6347)
Fixes: #6346

Move LDAP embedded server dependencies to test scope so they aren't packaged in final management server jar.

Co-authored-by: Marcus Sorensen <mls@apple.com>
2022-05-04 11:49:29 +05:30
Pearl Dsilva 3bd723ca06
Change patch path during live patching of systemVMs (#6328)
* Improve log when live patching fails

* change patching path from /tmp to /var/cache/clou

* add iptable rule for console proxy (novnc)

* temporary template paths

* revert pom xml to original paths
2022-04-29 00:01:11 -03:00
slavkap 453bb57fd2
Disable creating StorPool logs when there isn't StorPool primary storage (#6317)
There is not need to create log files for StorPool driver when there
isn't a StorPool primary storage
2022-04-27 07:24:44 -03:00
SadiJr 4cc830d04a
[VMWare] error when detaching volume (#6283)
* Fix exception, in VMWare 6.5, when trying to detach volume

* Address reviews

* Address reviews

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-04-26 23:29:17 -03:00
Abhishek Kumar 4a914aa88d
network: ipv6 static routes (#5786)
* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* IPv6: configure VR of isolated networks

* IPv6: add default IPv6 route in VR of isolated networks

* Reformat server/src/main/java/com/cloud/network/NetworkServiceImpl.java

* IPv6: update network to offering which support IPv6

* IPv6: update vm nic ipv6 address when update network to new offering

* IPv6: configure VPC VR to support multiple tiers with IPv6

* IPv6: add RDNSS in radvd.conf

* IPv6/UI: support ipv6 protocols in Network ACL

* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes for diagnostics

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* more import fromo #5594

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* IPv6: fix wrong public ipv6 in VPC VR

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* ui: fix add ipv6 prefix labels, message

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: label fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* logging fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* minor ui refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ip6 events

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ip6 usage

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* slaac based public ip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* diagnostics fix for vr

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* firewall changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* alert and show ipv6 usage

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change for network response

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ipv6 network test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: fix ipaddress listing

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix simulator

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test and fixes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test temp change revert

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fixes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* use uuid

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* event syntax fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* review comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* assign vlan public IP for dualstack only if both protocols present on same vlan

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* internetprotocol in networkofferingresponse

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add tcp, udp

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* support vpc with ipv6 only on same vlan

- adds new internet protocol param to createVpcOffering API
- When DualStack internet protocol is selected for the VPC offering, tiers with network with or without IPv6 support can be deployed.
- When IPv4 internet protocol is used for the VPC offering, tiers with network with only IPv4 support can be deployed

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change and fix

allow VPC with IPv4 protocol to deploy tiers with IPv6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui fixes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix multiple routes, network guest ipv6 gateway

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address review comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* stop radvd on backup VR

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix router redundant status with ipv6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* disable radvd for backup vr

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* correctly set ipv6 in redundant router case

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unused code

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix connection

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: don't show all protocol for egress

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix guest ipv6 for redundant VRs

Redundant VRs will not be assigned an IPv6 by ACS and guest netwrok gateway will be added as IPv6 for guest interface by systemvm scripts during setting redundant state of the VR.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing ipv6 on redundant vr

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix syntax

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: fix vpc tier redirect to show details

When redirecting to VPC tier, details tab should be active by default

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* restart radvd on primary redundant vr

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* check for ipv6 values

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove old ui change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix condition

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove gateway from backup vr

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* network upgrade fail early

when IPv6 network cannot be allocated fail before shutting down the network

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix radvd not running on RVR

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* prepare radvd.conf once

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix job polling

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix RVR for vpc with ipv6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix ipv6 network acls

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* Update CsConfig.py

* add check

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test: vpc offering test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test: add negative tests for guest prefix, public range

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add default ipv6 route for primary

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix dadfailed on vpc rvr

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: fix add iprange form, dedicate action visibility

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix adding, deleting ipv6 range

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix failing test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing destination cidr in ipv6 firewall

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix ipv6 nftables rules

Allow storing linger IPv6 CIDRs in DB
Specify all port range for TC{, UDP protocol rules withot ports
Fix adding nft rules by creating chains first

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix icmpv6 type, code

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix icmp type, code

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test: add more for ipv6 network

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add warning message for egress policy in ipv6 fw rule

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui,server: update ipv6 vlan range

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* subnet operations inside transaction

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* server: persistent public IPv6 for network

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: fix action alignment

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix vpc acl for tiers

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix removing network placeholder nic

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix acl rules for ip version

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix placeholder nic and nd-neighbor block issue

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test for redundant nw

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix ping

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* systemvm: uprgade to debian 11.3.0

* ipv6: enable ipv6 in sysctl config in bootstrap.sh

* VR: fix KeyError: 'nic_ip6_cidr'

* build fix for latest event changes

Signed-off-by: Abhishek Kumar <abhishek.kumar@shapeblue.com>

Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-04-25 22:51:32 -03:00
Abhishek Kumar 523805c8bc
schema,server,api: events improvement (#5997)
* schema,server,api: events improvement

Add resource ID and resource type to event.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wip

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor resourcetype association with API class

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add resource anme to the response

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* new line

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add resource test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* smoke test for events resource

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui improvements

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* api,ui: add support for listing events for a resource

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* since key

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* tests and permission changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* missing test

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* events for domain

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* improvements

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add missing license

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: fix js console errors

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* sort enumeration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix event resource for vpc

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* feedback changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix order

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* events with parent resource

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* missing UI labels

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* donot call cmd resource methods before dispatch

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add restore vm to procedure

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add missing imports

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* resource details for more events

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add test for changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* more test, license fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* wrong merge fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix for more event types

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-04-25 09:05:17 -03:00
JoaoJandre f4e95e3105
Fix VM stats inconsistencies (#6245)
* Fix VM stats inconsistencies

* Address reviews

* Address review

Co-authored-by: João Paraquetti <joao@scclouds.com.br>
Co-authored-by: Joao <JoaoJandre@gitlab.com>
2022-04-22 08:52:57 -03:00
dahn 545e89c1cb
Mshost stats (#5588)
* ms stats thread added

* initial data collection for management server

* empty list management server metrics command

* bean copy into MS metrics object

* ms status VO

* further API and DB plumbing

* minimal metrics response in API

* remove commented, refactor data collection plumbing

* javadocs

* surpress stacktrace on expected error

* update status experiment

* ms status publish framework added

* review comment addressed

* static data to DB and API, /proc/ reading

* addressing review comments

* ui for ms details

* small ui adjustment

* beanCopy

* agentcount response and system parameter

* labels

* package-lock

* add version strings to regular list API

* add shutdown time to DB

* add last start and last stop to regular list response

* distro info in regular response/session  count added

* metrics as details

* add heap used and remove details map

* thread-statusses

* move db upgrade to 4.17

* sysmem

* procmem

* ui demo comments applied

* javadoc

* get conf and log file locations

* loginfo

* cpuLoadStats

* no.remote

* extra spaces removed

* clusterlistener

* add unit to kb value

* revert accidental rename

* silly fqcn removed

* get mem info from bean is possible

* refactor long sequence for readability

* registerListener

* listUsageMetrics and isDbLocal

* rats

* local usage and db or not

* minimal listDbMetrics

* db vars and stats

* cleanup and #queries queried

* db stats calculation

* rat

* remove list response wrapper from sinlge details-lists responses

* rudimentary metrics view

* metrics table cleanup

* table makeup, collection dates

* move component to appropriate location

* capitalisation removed

* rebase error resolved

* rename deamon to daemon

* small style comments applied

* another merge issue

* naming comments and boot time

* stop/start prefixed with server

* layout-fix

* listMSMetrics test and test refactor

* usage metrics test

* db metrics test

* extra validations

* Update ui/public/locales/en.json

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

* descriptions of loadaverages and replica's

* collection time on top

* cpu load on metrics overview

* DbStatsCollection

* some parameter description texts

* labels adjusted

* new output 'kernelversion' and log info cleanup

* labels

* Update api/src/main/java/com/cloud/server/ManagementServerHostStats.java

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/response/DbMetricsResponse.java

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/dao/ManagementServerHostDao.java

Co-authored-by: Rodrigo D. Lopez <19981369+RodrigoDLopez@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/ClusterManager.java

Co-authored-by: Rodrigo D. Lopez <19981369+RodrigoDLopez@users.noreply.github.com>

* Update api/src/main/java/org/apache/cloudstack/api/response/ManagementServerResponse.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update api/src/main/java/org/apache/cloudstack/api/response/ManagementServerResponse.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update engine/schema/src/main/java/com/cloud/host/dao/HostDao.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/ClusterManager.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/ClusterManager.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/server/StatsCollector.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/ClusterManager.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/ClusterManager.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update framework/cluster/src/main/java/com/cloud/cluster/dao/ManagementServerHostDao.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/server/StatsCollector.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/server/StatsCollector.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update server/src/main/java/com/cloud/server/StatsCollector.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

* Update plugins/metrics/src/main/java/org/apache/cloudstack/metrics/MetricsServiceImpl.java

* some (more) refactorring suggestions applied

* human readable memory sizes

* rat

* actual collection time instead of query time, improved descriptions

* merge errors fixed

* optional metric values

* javadoc and logging

* names of jmx vars have changed

* vue3-compatibility

* new output parameter type

* lower retention default

* vue3 fixes

* polish comments

* polish comments 2, the reckoning

* note on usage servers

* merge conflict errors

* pollish

* conditional assertion to deal with simulator restart

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
Co-authored-by: Rodrigo D. Lopez <19981369+RodrigoDLopez@users.noreply.github.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-22 08:48:19 -03:00
Pearl Dsilva 830f3061bc
SystemVM optimizations (#5831)
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* refactor patching logic - certs

* move logic of fixing rootdiskcontroller from upgrade to kubernetes service

* add livepatch option to restart network & vpc

* smooth upgrade of cks clusters

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* address comments

* add temp links of download.cloudstack.org

* address part of the comments

* address comments

* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0

* address comments - simplify

* fix vue3 related icon changes

* allow network commands when router template version is lower but is patched

* add internal LB to the list of routers to be patched on network restart with live patch

* add unit tests for API param validations and new helper utilities - file scp & checksum validations

* perform patching only for non-user i.e., system VMs

* add test to validate params

* remove unused import

* add column to domain_router to display software version and support networkrestart with livePatch from router view

* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false

* use router software version instead of checksum

* show N/A if no software version reported i.e., in upgraded envs

* fix deb failure

* update pom to official links of systemVM template
2022-04-21 13:40:19 -03:00
DK101010 ccac1a383f
Feat/add vdisk UUID to list volume (#5848)
* get vdisk uuid from vcenter and store it into database

* add vdisk uuid as external_uuid to listVolume response

* add sql upgrade file

* Update vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* update sql add column external_uuid

* Update server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

Co-authored-by: Wei Zhou <weizhou@apache.org>

* adapt param description for externalUuid

* add 'idempotent column add' to create external_uuid col

* rename method to getExternalDiskUUID

* remove line disk_offering.system_use

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
2022-04-19 23:34:09 -03:00
Leo 70122007bb
Updated SAML2 auth sessionkey cookie path (#6149)
This change will set the sessionkey under the /client/api path. This commit should prevent duplicate sessionkey cookies from being set on both /client (incorrect) and /client/api (correct). Prior to this commit, the /client version was being set while the /client/api version remained unchanged with an invalid sessionkey. As a result, subsequent requests after the SAML2 authentication would immediately fail with an invalid session and results in the user being logged out.

The sessionkey is now set explicitly for the /client/api path which should fix this issue, regardless of the SSO URL and path that's being used.
2022-04-18 17:16:20 +05:30
John Bampton c265df965e
Fix spelling (#6185)
* Fix spelling

* Update services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java

Co-authored-by: SadiJr <sadijacinto@gmail.com>

Co-authored-by: Rohit Yadav <rohityadav89@gmail.com>
Co-authored-by: SadiJr <sadijacinto@gmail.com>
2022-04-16 00:55:11 +05:30
Wei Zhou aa0197c141
kvm: truncate vnc password to 8 chars (#6244)
This PR truncates the vnc password of kvm vms to 8 chars to support latest versions of libvirt.
2022-04-15 20:26:42 +05:30
slavkap 4004dfcfd8
StorPool storage plugin (#6007)
* StorPool storage plugin

Adds volume storage plugin for StorPool SDS

* Added support for alternative endpoint

Added option to switch to alternative endpoint for SP primary storage

* renamed all classes from Storpool to StorPool

* Address review

* removed unnecessary else

* Removed check about the storage provider

We don't need this check, we'll get if the snapshot is on StorPool be
its name from path

* Check that current plugin supports all functionality before upgrade CS

* Smoke tests for StorPool plug-in

* Fixed conflicts

* Fixed conflicts and added missed Apache license header

* Removed whitespaces in smoke tests

* Added StorPool plugin jar for Debian

the StorPool jar will be included into cloudstack-agent package for
Debian/Ubuntu
2022-04-14 11:12:01 -03:00
nvazquez 65dc2df896
Merge branch '4.16' 2022-04-13 08:45:48 -03:00
slavkap 42a92dcdd3
Extract the IO_URING configuration into the agent.properties (#6253)
When using advanced virtualization the IO Driver is not supported. The
admin will decide if want to enable/disable this configuration from
agent.properties file. The default value is true
2022-04-13 08:43:35 -03:00
Daniel Augusto Veronezi Salvador 39fad2d9d7
KVM disk-only based snapshot of volumes instead of taking VM's full snapshot and extracting disks (#5297)
* Refactor create volume snapshot with running VM

* Refactor create volume snapshot with stopped VM

* Refactor create volume from snapshot

* Refactor create template from snapshot

* Refactor volume migration (migrateVolume/ migrateVirtualMachineWithVolume)

* Refactor snapshot deletion

* Refactor snapshot revertion

* Adjusts and fix cherry-pick conflicts

* Remove diffuse tests

* Add validation to add flag '--delete' on command 'virsh blockcommand' only if libvirt version is equal or higher 6.0.0

* Expunge temporary snapshot only if template creation is from snapshot

* Extract strings to constant

* Remove unused imports

* Fix error on revert backed up snapshot

* Turn method's return to void as it is not used

* Rename method in SnapshotHelper

* Fix folder creation when using SharedMountPoint pool

* Remove static import

* Remove unnused method

* Cover take snapshot in centos 7

* Handle right snapshot flag according to qemu version

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2022-04-12 08:14:27 -03:00
Pearl Dsilva 431c352a6d
Synchronization of network devices on newly added hosts for Persistent Networks (#5977)
* Persistent Network feature & Marvin component tests

* Cleaned up comments and imports

* fixed small error

* add support to add setup persistent networks' resources when a disabled host is enabled

* small fix

* use wildcard instead of hard-coding the bridge name

* allow clean up of resources when removing a host in maintenance mode

* skip test for simulator hypervisor

Co-authored-by: shatoboar <sang-woo.bae@campus.tu-berlin.de>
2022-04-11 23:12:05 -03:00
José Flauzino 16f2896940
Persistence of VM stats (#5984)
* Add persistence of VM stats

* Fix API 'since' attribute

* Add license

* Address GutoVeronezi's reviews

* Fix the order of VM stats in the API response

* Fix msid in VM stats data

* Fix disk stats and add minor improvements

* Add log message

* Build string using ReflectionToStringBuilderUtils

* Rerun checks

Co-authored-by: joseflauzino <jose@scclouds.com.br>
2022-04-11 10:42:21 -03:00
slavkap 2b075ed39e
Storage-based Snapshots for KVM VMs (#3724)
* VM snapshots of running KVM instance using storage providers plugins for disk snapshots

Added new virtual machine snapshot strategy which is using storage providers plugins to take/revert/delete snapshots.
You can take VM snapshot without VM memory on KVM instance, using storage providers implementations for disk snapshots.
Also revert and delete is added as functionality. Added Thaw/Freeze command for KVM instance.
The snapshots will be consistent, because we freeze the VM during the snapshotting. Backup to secondary storage is executed after
thaw of the VM and if it is enabled in global settings.

* Removed duplicated functionality

Set few methods in DefaultVMSnapshotStrategy to protected to reuse them
without duplicating the code. Remove code that is actualy not needed

* Added requirements in global setting kvm.vmstoragesnapshot.enabled

Added more information in kvm.vmstoragesnapshot.enabled global setting,
that it needs installation of:
- qemu version 1.6+
- qemu-guest-agent installed on guest virtual machine

when the option is enabled

* Added Apache license header

* Removed commented code

* If "kvm.vmstoragesnapshot.enabled" is null should be considered as false

* removed unused imports, replaced default template

Removed unused imports which causing failures and replaced template to
CentOS8

* "kvm.vmstoragesnapshot.enabled" set to dynamic

* Getting status of freeze/thaw commands not the return code

Will chacke the status if freeze/thaw of Guest VM succeded, rather than
looking for return code. Code refactoring

* removed "CreatingKVM" VMsnapshot state and events related to it

* renamed AllocatedKVM to AllocatedVM

the states should not be associated to a hypervisor type

* loggin the result of "drive-backup" command

* Check which VM snapshot strategy could handle the vm snapshots

gets the best match of VM snapshot strategy which could handle the vm
snapshots on KVM.
Other storage plugins could integrate with this functionality to support group snapshots

* Added poolId in canHandle for KVM hypervisors

Added poolId into canHandle method used to check if all volumes are on
the same PowerFlex's storage pool

* skip smoke tests if the hypervisor's OS type is CentOS

This PR works with functionality included in qemu-kvm-ev which
does not come by default on CentOS. The smoke tests will be skipped if
the hypervisor OS is CentOS

* Added missed import in smoke test

* Suggested change to use ` org.apache.commons.lang.StringUtils.isNotBlank`

* Fix getting device on Ubuntu

On Ubuntu the device isn't provided and we have to get it from
node-name parameter. For drive-backup command (for Ubuntu) is needed and job-id which
is the value of node-name (this extra param works on Ubuntu and CentOS as well).

* Removed new snapshot states and functionality for NFS

* throw CloudRuntimeException

provide a properer error message when delete VM snapshot fails

* exclude GROUP snapshots when listing snapshots

* Skip tests if there is pool with NFS/Local

* address comments
2022-04-07 21:42:12 -03:00
Rohit Yadav 1cdacccdc0
maven: support JaCoCo and other quality checks (#6228)
This enables jacoco, which didn't run before with the -P quality due to
missing passing of jacoco arg line to surefire plugin.

This also adds support for jacoco/quality builds using Github action and
posting of the PR coverage data using a new action step.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2022-04-07 20:48:48 +05:30
David Jumani 5ab4309298
agent: Detect existing hosts with UEFI support (#6139)
* agent: Pass uefi enabled status as part of ready command

* Cleanup

* Fix checkstyle

* Save uefi status if different
2022-04-04 23:31:54 -03:00
Pearl Dsilva 2c8c476656
CKS: Support deployment of CKS clusters on Advanced zones with security groups (#6132)
* CKS: Support deployment of CKS clusters on Advanced zones with security groups

* use available constant

* address comments -
- Ingress sg rule for port 22 & 6443
- Use constant to define securityGroup Name
- rename variable name from type -> vmType

* unique name for security group + foreign key

* use constants
2022-04-01 13:19:54 -03:00
Nicolas Vazquez 2f2d6cbe38
KVM: Enhance CPU speed detection on hosts (#6175)
* KVM: Enhance CPU speed detection on hosts

* Update agent/conf/agent.properties

Co-authored-by: dahn <daan.hoogland@gmail.com>

Co-authored-by: dahn <daan.hoogland@gmail.com>
2022-04-01 12:12:34 -03:00
nvazquez 1c238e101d
Merge branch '4.16' 2022-03-30 00:00:34 -03:00
Wei Zhou ee27708ffb
SAML: replace first number with random alphabet if request ID starts with a number (#6165) 2022-03-29 23:59:44 -03:00
Wei Zhou 944b6037cc Merge remote-tracking branch 'apache/4.16' 2022-03-29 10:43:59 +02:00
Nicolas Vazquez a69ab3b28f
Ensure configdrive path is edited properly during live migration (#6173) 2022-03-29 10:09:08 +02:00
John Bampton 182899d0b5
Add Python flake8 linting for W291 trailing whitespace with Super-Linter (#4687)
* refactor: remove trailing whitespace from Python files

* Add the GitHub Super-Linter

Add Python flake8 linting for W291 trailing whitespace

* Add licenses
2022-03-28 11:40:26 -03:00
Wei Zhou 5845de4004
kvm: support multiple local storage pools (#6147)
* kvm: support multiple local storage pools

* Update #6147: add method configureLocalStorage

* Update #6147: add method validateLocalStorageUUID

* Update #6147: remove underscore

* Update #6147: add unit tests
2022-03-25 11:40:30 -03:00
Nicolas Vazquez 3fc31ca1e5
Revert "Honour isrecursive above listall (#6045)" (#6126)
This reverts commit 71056191f2.
2022-03-16 23:17:02 -03:00
David Jumani 704a939c65
cks: Fix missing .service files when bootstraping in cks (#6076)
* cks: Run 'systemctl daemon-reload' before setting up k8s

* add wait till service files exists
2022-03-14 14:07:34 -03:00
Bryan Lima afdc73f911
Update VM priority (cpu_shares) when live scaling it (#6031)
* Update VM priority (cpu_chares) when live scaling it

* Apply suggestions from code review

Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Addressing javadoc review

* Addressing review typo

Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2022-03-14 11:34:10 -03:00
David Jumani 71056191f2
Honour isrecursive above listall (#6045)
* Honour isrecursive above listall

* Revert "Honour isrecursive above listall"

This reverts commit 12bc26ae1c.

* Being smart about it

* Cleanup

* fixing npe
2022-03-10 23:59:42 -03:00
JoaoJandre 5f07ddaca9
Refactor account type (#6048)
* Refactor account type

* Added license.

* Address reviews

* Address review.

Co-authored-by: João Paraquetti <joao@scclouds.com.br>
Co-authored-by: Joao <JoaoJandre@gitlab.com>
2022-03-09 11:14:19 -03:00
John Bampton 6401c850b7
Fix spelling (#6064)
* Fix spelling

- `interupted` to `interrupted`
- `paramter` to `parameter`

* Fix more typos
2022-03-08 13:02:35 -03:00
dahn b014617416
no axis (#5993)
* no axis

* remove f5 from ui

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2022-03-07 22:46:29 -03:00
Rohit Yadav 2f6fc56e74
maven: Fix build on osx (#6056)
kvm: don't read /proc/meminfo on non-Linux environments as part of constructor

Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com>
2022-03-07 14:55:42 +05:30
John Bampton 980c0bff03
Fix spelling (#6041)
Changed `occured` to `occurred`
2022-03-03 13:10:14 -03:00
Suresh Kumar Anaparti bc70535ee5
Updating pom.xml version numbers for release 4.16.2.0-SNAPSHOT
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-03-03 18:15:33 +05:30
David Jumani 85c59979f7
Multiple SSH Keys support (#5965)
* keypairs added in api-constants

* names parameter added

* findbynames method added in dao

* change in impl to find and reset multiple keys

* findbynames method implemented

* log the publickeys, check the ssh keys given exists or not

* new ArrayList<>

* SQL IN toArray

* keypair

* null pointer exception solved with + concatanation

* null pointer exception solved with + concatanation

* error resolved

* keypair name to names in uservmresponse

* keypair name is set in the uservmresponse, from the details

* null checks are removed, keypairnames are stored in a string, sent to the resetvmsshinternal, and added in details

* commit first eval

* deploy vm takes multiple ssh-keys

* Deploy VM UI changed to accept multiple ssh keys

* Reset SSH UI API changed

* ResetSSH.vue

* ssh keys joined, ssh added in infocard

* changes made

* schema error resolved

* potential null pointer exception removed

* Update UserVmManagerImpl.java

unnecessary check removed.

* Update DeployVMCmd.java

* Update DeployVMCmd.java

* Update ResetVMSSHKeyCmd.java

* Update UserVmJoinDaoImpl.java

* .

* arraylist

* Update DeployVMCmd.java

* Update UserVmManagerImpl.java

* Update ResetVMSSHKeyCmd.java

* Update db

* Fix list vm by keypair

* ui fixes

* Fix typos

* ui fixes

* Cleanup

* Adding deprecated and since in api params

* Adding upgrade for existing vms with ssh keys

* Handle no key for cks

* Show existing keyparis in reset ssh key form

* get keys from the right account

Co-authored-by: bicrxm <bickrombishsass@gmail.com>
2022-03-01 21:30:55 -03:00
Suresh Kumar Anaparti cad9332082
Updating pom.xml version numbers for release 4.16.1.0
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-02-25 19:01:16 +05:30
Suresh Kumar Anaparti 1484f5a418
Merge branch '4.16' into main 2022-02-23 22:42:10 +05:30
Suresh Kumar Anaparti 027e6030af
[KVM] Disconnect the volumes with the proper storage adaptor. (#6029)
* [KVM] Disconnect the volumes with the proper storage adaptor.

* Improved / Added logs
2022-02-23 22:40:14 +05:30
Wei Zhou 5d1e54a4ab
server: fix host anti-affinity when start batch vms (#6002) 2022-02-21 09:54:03 +01:00
Pearl Dsilva 550c810538
[VMware] Support for removal of NIC on IP disassociation on the VR (#5985)
* [VMware] Support for removal of NIC on IP disassociation on the VR

* address comments - extract code + add appropriate operation name in logs

* Do not remove/unplug nic on removal of static nat on last IP (Public NIC)

* address comments

* address comment
2022-02-17 22:44:03 -03:00
Suresh Kumar Anaparti d4f40ecc6a
Fixed strings import issue (#5998)
* Fixed strings import issue in KubernetesClusterResourceModifierActionWorker

* Use isNoneEmpty
2022-02-15 13:32:41 -03:00
Suresh Kumar Anaparti b50542a11c
Merge branch '4.16' into main 2022-02-15 19:26:04 +05:30
Pearl Dsilva e0a5df50ce
CKS Enhancements and SystemVM template upgrade improvements (#5863)
* This PR/commit comprises of the following:
- Support to fallback on the older systemVM template in case of no change in template across ACS versions
- Update core user to cloud in CKS
- Display details of accessing CKS nodes in the UI - K8s Access tab
- Update systemvm template from debian 11 to debian 11.2
- Update letsencrypt cert
- Remove docker dependency as from ACS 4.16 onward k8s has deprecated support for docker - use containerd as container runtime

* support for private registry - containerd

* Enable updating template type (only) for system owned templates via UI

* edit indents

* Address comments and move cmd from patch file to cloud-init runcmd

* temporary change

* update k8s test to use k8s version 1.21.5 (instead of 1.21.3 - due to https://github.com/kubernetes/kubernetes/pull/104530)

* support for private registry - containerd

* Enable updating template type (only) for system owned templates via UI

* smooth upgrade of cks clusters

* update pom file with temp download.cloudstack.org testing links

* fix pom

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* update to official links
2022-02-15 18:27:14 +05:30
Pearl Dsilva 9f23fbe7b6
[Simulator] Fix NPE in SolidFireHostListener (#5955)
* [Simulator] Fix NPE in SolidFireHostListener

* address comments

* Update plugins/storage/volume/solidfire/src/main/java/org/apache/cloudstack/storage/datastore/provider/SolidFireSharedHostListener.java

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2022-02-12 08:32:19 -03:00
Suresh Kumar Anaparti 212a4dedfc
Merge branch '4.16' into main 2022-02-10 20:56:52 +05:30
Nicolas Vazquez a3bb84b495
[VMware] Update SCSI controllers for VMs (#5910) 2022-02-10 14:36:34 +01:00
nvazquez e9d8d7942d
Merge branch '4.16' 2022-02-10 09:48:49 -03:00
dahn a6d9fa61b9
Role escalation prevention (#5879)
* prevent role access escallation

* hierarchy issue fixed

* create api list in account manager for checking new account access

* full api list check

* strange role restriction removed for BareMetal

* add role check on upfdate account as well

* more selective use of api checkers

* error msg and var name

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2022-02-10 11:50:27 +05:30
Suresh Kumar Anaparti 850714c1fc
Merge branch '4.16' into main 2022-02-10 10:57:29 +05:30
dahn 143b72d67e
cleanup: Network Throttling for Additional Networks code in DirectVifDriver.java (#5875) 2022-02-09 10:19:47 +01:00
PJ Fanning c7885f55eb
maven: upgrade bouncycastle due to cve (#5949)
Fixes: #5948
2022-02-09 00:22:28 +05:30
Suresh Kumar Anaparti 208ae84dd7
Merge branch '4.16' into main 2022-02-08 19:01:34 +05:30
Rohit Yadav da56a2a806
maven: migrate short-term to reload4j v1.2.18 (#5878)
* maven: migrate short-term to reload4j v1.2.18

This migrate to log4j 1.x fork, reload4j 1.2.18.0 which is drop-in
replacement and addresses some immediate CVE and issues.

* log4j migration to reload4j in pom xmls

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Exclude log4j from transitive dependencies (#73)

Co-authored-by: Marcus Sorensen <shadowsor@gmail.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2022-02-08 15:00:38 +05:30
dahn 0f1cd6009d
add logging to deployment planners (#5859)
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2022-02-04 17:02:32 +01:00
Suresh Kumar Anaparti e3f0e3b544
Merge branch '4.16' into main 2022-02-04 19:04:08 +05:30
Harikrishna 56a8852c4c
Remove code that reserves extra interfaces based on the global setting router.extra.public.nics in case of VMware. (#5912)
Added hot plugging of vifs whenever required in case of isolated networks. VPC networks already uses hotplugging of vifs.
2022-02-04 12:41:56 +01:00
davidjumani 26e4e18218
kvm: Use lscpu to get cpu max speed (#5506)
* kvm: Use lscpu to get cpu max speed

* Fix str conversion

* Reorder

* Refactor

* Apply suggestions from code review

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Updated the calling method name getCpuSpeedFromCommandLscpu

* Make it more readable

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2022-02-04 11:37:14 +01:00
Suresh Kumar Anaparti bf70566c2c
Merge branch '4.16' into main 2022-02-02 17:30:21 +05:30
Nicolas Vazquez ddd311c695
[XenServer/XCP-ng] Sync the 'platform' setting according to the 'cpu.corespersocket' setting (#5892)
* Update platform setting if cpu.corespersocket is set

* Refactor conditions

* Allow empty map but not null
2022-02-02 12:21:47 +05:30
Nicolas Vazquez 3e92a63155
[XenServer/XCP-ng] Pass the image store NFS version on storage commands (#5886)
* Add NFS version to mount command

* Remove extra line

* Extend NFS version to mount secondary storage

* Unused import

* Refactor NFS version to be granular

* Make use of the ConfigKey on the NFS version setting value
2022-01-31 12:21:13 +05:30
Suresh Kumar Anaparti f610e89ea9
Merge branch '4.16' into main 2022-01-28 13:43:00 +05:30
davidjumani 6fa8538666
Adding placeholders for custom NSP vues (#5620)
* Adding placeholders for custom NSP vues

* Fix decorator

* Fixing gslbproviderprivateip description
2022-01-27 20:11:35 +05:30
Harikrishna f15cab16da
server: Decouple service (compute) offering and disk offering (#5008)
Currently, our compute offerings and disk offerings are tightly coupled with respect to many aspects. For example, if a compute offering is created, a corresponding disk offering entry is also created with the same ID as the reference. Also creating compute offering takes few disk-related parameters which anyway goes to the corresponding disk offering only. I think this design was initially made to address compute offering for the root volume created from a template. Also changing the offering of a volume is tightly coupled with storage tags and has to be done in different APIs either migrateVolume or resizeVolume. Changing of disk offering should be seamless and should consider new storage tags, new size and place the volume in appropriate state as defined in disk offering.

more details are mentioned here https://cwiki.apache.org/confluence/display/CLOUDSTACK/Compute+offering+and+disk+offering+refactoring

* Schema changes and disk offering column change from "type" to "compute_only"

* Few more changes

* Decoupled service offering and disk offering

* Remove diskofferingid from vminstance VO

* Decouple service offering and disk offering states

* diskoffering getsize() is only for strict disk offerings

* Fix deployVM flow

* Added new API params to compute offering creation

* Add diskofferingstrictness to serviceoffering vo under quota

* Added overrideDiskOfferingId parameter in deploy VM API which will override disk offering for the root disk both in template and ISO case

Added diskSizeStrictness parameter in create Disk offering API which will decide whether to restrict resize or disk offering change of a volume

* Fix User vm response to show proper service offering and disk offerings

* Added disk size strictness in disk offering response

* Added disk offering strictness to the service offering response

* Remove comments

* Added UI changes for Disk offering strictness in add compute offering form and Disk size strictness in add disk offering form

* Added diskoffering details to the service offering response

* Added UI changes in deployvm wizard to accept override disk offering id

* Fix delete compute offering

* Fix VM deployment from custom service offering

* Move uselocalstorage column access from service offering to disk offering

* UI: Separated compute and disk releated parameters in add compute offering wizard, also added association to disk offering

* Fixed diskoffering automatic selection on add compute offering wizard

* UI: move compute only toggle button outside the box in add compute offering wizard

* Added volumeId parameter to listDiskOfferings API and the disksizestrictness flag of the current disk offering is honored while list disk offerings

* Added configuration parameter to decide whether to check volume tags on the destination storagepool during migration

* Added disk offering change checks during resize volume operation

* Added new API changeofferingforVolume API and corresponding changes

* Add UI form for changeOfferingForVolume API

* Fix UI conflicts

* Fix service offering usage as disk offering

* Fix unit test failures

* fix user_vm_view

* Addressed review comments

* Fixed service_offering_view

* Fix service offering edit flow

* Fix service offering constructor to address custom offering

* Fix domain_router_view to get proper service offering id

* Removed unused import

* Addressed review comments and fixed update service offering flow with storage tags

* Added marvin test cases for checking disk offering strictness

* review comments addressed

* Remove system_use column from disk offering join

* update volume_view to update system_use column from service offering and not disk offering

* Fix changeOfferingForVolume API for custom disk offering

* Fix global setting implementation

* Fix list volumes, after changing system_use column from disk offering to service offering in volume_view

* Changes for override root disk offering in deployvm wizard in case of custom offering

* Fix a unit test case

* Fixed recent unit test cases with new serviceofferingvo constructor

* Fix unit test in VolumeApiServiceImpl

* Added storage id for the list disk offering API and corresponding UI changes in migrateVolume and changeOfferingForVolume flow

* Rename global configuration parameter from storage.pool.tags.disk.offering.strictness to match.storage.pool.tags.with.disk.offering

* Fix smoke test failures

* Added tool tip for migrate volume UI form

* Address review comments and fix UI form of deploy VM in case of ISO.

* Fixed resize volume UI form for data disk

* UI changes to disable override root disk size when override root disk offering is enabled

* UI fix in deploy vm wizard

* Fix listdiskoffering after rebasing with main

* Fixed UI in migrate and changeofferingfor volume to handle empty disk offering list
Removed the volume's current disk offering from listDiskOffering response list

* Added custom Iops to resize volume form and removed the current disk offering during change offering for volume UI form

* Fix false response on updateDiskOffering API

* Added search field for changeofferingforvolume UI form

* Fix resize volume and migrate volume to update volume path if DRS is applied on volume in datastore cluster

* Removed DB changes from 4.16 upgrade file

* Resolving merge conflicts with main 4.17

* Added support for auto migration and auto resize of the root volume upon changing the service offering for VM.

* UI: Added automigrate checkbox in scale VM form

* Addes since attributes to new API params

* Added shrinkOK parameter to changeofferingforvolume API

* Added shrinkOk param to UI in changeOfferingforVolume form

* Added shrinkOk flag to scaleVM and changeServiceForVirtualMachines and UI form

* Removed old foreign key constraint on IDs of service offering and disk offering

* Allow resize and automigrate of root volume if required in all cases of service offering change

* Allow only resize to higher disk size from UI

* Fixing vue syntax error

* Make UI changes to provide root disk size box when the linked disk offering is of custom

* Converted from check box to toggle in scale VM, changeoffering, resize and migrate volume forms

* Fix resize volume operation to update the VM settings

* Fix migratevolume form to pick selected storage pool id in list diskofferings API
2022-01-27 15:08:42 +05:30
Suresh Kumar Anaparti e1b56be67c
Merge branch '4.16' into main 2022-01-11 12:46:33 +05:30
dahn 7ea2cbd889
Storage pool absent (#5841)
* simple null check

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2022-01-11 12:18:55 +05:30
SadiJr 5d7ea30f87
vmware: Remove restriction, in VMWare, to resize only SCSI disks (#5650)
Remove restriction, in VMWare, to resize only SCSI disks, because VMWare supports resize of subtypes of SCSI, like LSILOGIC, LSILOGIC SAS, etc

In VMWare, there are different types of SCSI controllers, such as BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS, etc. All of these sub-types support the disk resize operation, the only exception is the IDE controller. However, in ACS, there is a logical restriction to allow only resizing of volumes of the specific SCSI type; thus, not allowing resizing of the SCSI sub-types. This PR corrects this behavior blocking the resize of volumes which uses IDE controller only.
More information can be found at:

https://www.starwindsoftware.com/blog/storage-controllers-in-vmware-vsphere
https://www.nakivo.com/blog/scsi-controller-and-other-vmware-controller-types/
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-5872D173-A076-42FE-8D0B-9DB0EB0E7362.html
https://communities.vmware.com/t5/ESXi-Discussions/Unable-to-increase-disk-Space/td-p/397823

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-01-08 09:02:00 +05:30
slavkap 48006de3e3
kvm: Set RAW format to RBD DATADISK (#5801)
While deploying a VM with data disk on Ceph the volume format was in
QCOW2.

Fixes #5531
2022-01-08 08:54:21 +05:30
SadiJr e993a20e5a
[Veeam] Force ACS to use the template repository to clone job (#5774)
* [Veeam] Force ACS to use the repository of the template to clone job

* Address reviews

* Fix checkstyle

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2022-01-07 11:12:57 +05:30
Suresh Kumar Anaparti 30ae9eedad
Merge branch '4.16' into main 2022-01-05 10:14:47 +05:30
dahn e06a66ba14
ldap: truststore per domain (#5816)
Co-authored-by: Daan Hoogland <dahn@onecht.net>
2022-01-03 21:01:51 +05:30
Rohit Yadav c84198d76d Merge remote-tracking branch 'origin/4.16'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-12-30 16:52:30 +05:30
Marcus Sorensen dcdcd09058
Randomize managed volume copy host (#5789)
* Randomize managed volume copy host

* Managed volume copy was always returning first host that could see storage pools

* Fix null value in logging for ScaleIOPrimaryDataStoreDriver due to if/else logic error

Signed-off-by: Marcus Sorensen <mls@apple.com>

* Use String.format for ScaleIO debug message

Signed-off-by: Marcus Sorensen <mls@apple.com>

* Update debug message for ScaleIO copy methods

Signed-off-by: Marcus Sorensen <mls@apple.com>

Co-authored-by: Marcus Sorensen <mls@apple.com>
2021-12-30 16:11:00 +05:30
Rohit Yadav 6728b696cd
kvm: don't always force scsi controller for aarch64 VMs (#5802)
* kvm: don't force scsi controller for aarch64 VMs

This would allow use of virtio disk controller with Ceph, etc or as
defined in the VM's root disk controller setting, rather than always
enforce SCSI.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* remove test that doesn't apply now

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* address review comment

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-12-30 15:19:32 +05:30
SadiJr 01f1aaee60
Improve VMWare logs (#5353)
* Improve VMWare logs

* Address reviews

* Address reviews and add more logs

* Improve VMWare logs

* Address reviews

* Address reviews and add more logs

* Address reviews

* Address reviews

* Try fix tests

* Refactor some logs to try fix tests

* Verifying if GSON is the cause of test exceptions

* Fix checkstyle

Co-authored-by: SadiJr <17a0db2854@firemailbox.club>
Co-authored-by: SadiJr <sadi@scclouds.com.br>
2021-12-22 13:56:00 +01:00
Wei Zhou 51d4e5475f Merge remote-tracking branch 'apache/4.16' 2021-12-10 17:50:25 +01:00
SadiJr 9363f6490b
Fix full backup VM restore (#5680)
Co-authored-by: SadiJr <sadi@scclouds.com.br>
2021-12-10 07:16:42 +01:00
Abhishek Kumar 93f4ecab84
vmware: fix cpu reservation during vm scale (#5708)
Fixes #5706

Correctly set CPU reservation while scaling a dynamically scalable VM.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-12-07 09:47:00 +05:30
sureshanaparti 00c6ad2733
vmware: Perform volume file search in the VM directory on the datastore while computing the VM snapshot chain size. (#5735)
This PR improves the volume file search on the datastore while computing the VM snapshot chain size in VMware. The file search is now performed in the VM directory on the datastore, instead of all the directories on the datastore (which is leading to incorrect VM snapshot chain size computation when VM id has 6 digits).
2021-12-03 16:46:31 +05:30
Rohit Yadav 2e9c9417be Merge remote-tracking branch 'origin/4.16' 2021-12-03 15:41:56 +05:30
davidjumani dfd75d7bc0
api: Fix wrong logger class in *Cmd.java (#5740)
This PR changes the incorrect class name used in the logger in ListHypervisorsCmd
2021-12-02 15:16:28 +05:30
Harikrishna 867ccf6f48
Merge pull request #5720 from shapeblue/remove-redundant-call-for-snapshot-chain-size
Removed redundant call for VM snapshot chain size, in VMware.
2021-12-02 10:08:28 +05:30
Suresh Kumar Anaparti e3459b5c01 Removed redundant call for VM snapshot chain size, in VMware. 2021-11-26 11:19:13 +05:30
José Flauzino 6e04f8711b
Fix HostMetricsResponse unit test (#5713)
Co-authored-by: José Flauzino <jose@scclouds.com.br>
2021-11-25 10:42:31 -03:00
Rohit Yadav 9c9de74966 Merge remote-tracking branch 'origin/4.16' 2021-11-24 17:56:14 +05:30
DH Park a85a00bed9
kvm: Fixes UEFI XML Definition Error (#5692)
UEFI bootMode 추가
2021-11-23 16:44:32 +05:30
Daniel Augusto Veronezi Salvador b4aabadc4d
Replace string libraries with org.apache.commons.lang3.StringUtils (#5386)
* Replace google lib for lang3 and adjust methods calls

* Replace string libs by lang3

* Prohibit others string libs

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-11-18 13:41:48 +05:30
nicolas 3f79436840
Updating pom.xml version numbers for release 4.17.0.0-SNAPSHOT
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-09 22:55:52 -03:00
nicolas 93c3c3b9ac
Updating pom.xml version numbers for release 4.16.1.0-SNAPSHOT
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-09 22:50:22 -03:00
nicolas 44c08b5acc
Updating pom.xml version numbers for release 4.16.0.0
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-04 14:14:57 -03:00
Peinthor Rene fe1ddfb1eb
linstor-volume-plugin: Only create diskless assignments on nodes (#5661)
With diskful set to true, linstor will fail if it cannot create local
storage for the resource. Which in turn will make it impossible to have a
setup with just compute nodes on cloudstack.
2021-11-03 21:24:48 -03:00
Rohit Yadav cb67da5bce
cks: refactor code to be architecture agnostic (#5629)
* cks: refactor code to be architecture agnostic

This makes the CKS setup/install/upgrades architecture agnostic and
works on both amd64 and arm64 platforms.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-10-29 15:15:34 -03:00
Rohit Yadav a8b19ee2d3
Merge pull request #5583 from apache/4.15
vmware: fix NPE for volume migration CLUSTER to ZONE-wide pool (#5582)
2021-10-17 06:56:43 +05:30
Abhishek Kumar 330e8e6e9d
vmware: fix NPE for volume migration CLUSTER to ZONE-wide pool (#5582)
Scope type of source pool is passed in the method, incorrectly, instead of the target pool.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-10-17 06:54:41 +05:30
Pearl Dsilva e73445e5d5
api: Fix response object for various APIs (#5573) 2021-10-13 22:26:53 -03:00
Wei Zhou 6b757d502e
CKS: use cluster-autoscaler-standard.yaml in kubernetes repo (#5574) 2021-10-13 12:46:52 -03:00
sureshanaparti 152159b13a
kvm: Add the source disk format for disk conversion/copy using 'qemu-img convert', when specified explicitly. (#5561)
This PR adds the source disk format (in KVM) for disk conversion/copy using 'qemu-img convert', when specified explicitly

Fixes: #5516
2021-10-11 12:00:49 +05:30
Abhishek Kumar 2a42a01728
xcp-ng: fix vm boot options (#5543)
* xcp-ng: fix vm boot options

Use value boot option values from VM details directly similar to other hypervisor plugins instead of relying on boot properties explicitly set for VirtualMachineTO.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Harikrishna Patnala <harikrishna.patnala@gmail.com>
2021-10-07 13:22:23 +05:30
davidjumani 6ac834a358
Adding AutoScaling for cks + CKS CoreOS EOL update + systemvmtemplate improvements (#4329)
Adding AutoScaling support for cks
Kubernetes PR : kubernetes/autoscaler#3629
Also replaces CoreOS with Debian
Fixes #4198

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-10-06 21:17:41 +05:30
sureshanaparti e198edfaf8
Report the PowerFlex/ScaleIO disk copy failure during volume migration and fail the migration (#5542)
* Report the PowerFlex/ScaleIO disk copy failure during volume migration and fail the migration.

* Code improvements

* Addressed review comments
2021-10-05 14:19:52 -03:00
div8cn 52a9dbdcd2
kvm available memory calculation optimization (#5540)
* Update LibvirtComputingResource.java

* Update LibvirtComputingResource.java

* Update LibvirtComputingResource.java

re

* Update MemStat.java

* Update MemStat.java

* Update MemStatTest.java
2021-10-05 12:34:28 -03:00
Harikrishna cd4e7e031a
Fix resize volume and migrate volume to update volume path if DRS is applied on volume in datastore cluster (#5539)
* Fix resize volume and migrate volume to update volume path if DRS is applied on volume in datastore cluster

* Change in constructors

* Naming changes

* Remove commented code

* Refactor code for more readability

* Addressed review comments on code refactor
2021-10-04 20:58:25 -03:00
Leo (Hsueh Yu-Min) 72a1c0e7f1
[KVM] Add MV Settings for virtual GPU hardware type and memory (#5513)
* KVM: Add MV Settings for virtual GPU hardware type and memory

* fix method createVideoDef argument in test package

* add available options for KVM virtual GPU hardware VM setting

* fix videoRam default value

* fix _videoRam is 0, it will use default provided by libvirt
2021-10-04 09:55:32 +05:30
Wei Zhou 09fce75047
OVS/GRE: bug fixes (#5446)
* OVS/GRE: fix failed to fetch cloudbr* info on host

* OVS/GRE: move ovstunnel.py to python3

* OVS: add test_privategw_acl_ovs_gre.py for automatic testing

(1) VR <-> VM should work
(2) Private GW should work

* OVS: fix an issue on xenserver

ovs bridges are deleted by xenserver/ovs automatically
```
[root@ref-trl-1797-x-M7-wei-zhou-xs2 ~]# grep -r xapi7 /var/log/ |grep del-br
/var/log/xensource.log:Sep 15 07:13:44 ref-trl-1797-x-M7-wei-zhou-xs2 xcp-networkd: [ info|localhost|611 |org.xen.xapi.xenops.classic events D:4a3d931cd89f|network_utils] /usr/bin/ovs-vsctl --timeout=20 -- --if-exists del-br xapi7
/var/log/daemon.log:Sep 15 07:13:45 ref-trl-1797-x-M7-wei-zhou-xs2 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=20 -- --if-exists del-br xapi7
```

which results that xe network exists but bridge does not exist, and operation stuck for 20 minutes at
```
2021-09-15 16:06:56    DEBUG [root] #### VMOPS enter  create_tunnel ####
2021-09-15 16:06:56    DEBUG [root] Creating tunnel from host 2 to host 1 with GRE key 2116
2021-09-15 16:06:56    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=0', 'wait-until', 'bridge', 'xapi7', '--', 'get', 'bridge', 'xapi7', 'name']
2021-09-15 16:26:56    DEBUG [root] bridge xapi7 for creating tunnel - VERIFIED
2021-09-15 16:26:56    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi7', 't2116-2-1', '--', 'set', 'interface', 't2116-2-1', 'type=gre', 'options:key=2116', 'options:remote_ip=10.0.34.230']
```

* OVS/GRE: component test test_vpc_distributed_routing_offering.py passes on xenserver

* OVS: support Shared network with GRE protocol

* update #5446: use lang3.StringUtils

* OVS/GRE: VPC with distributed routing on kvm (still not working)
2021-10-03 14:47:52 +05:30
Harikrishna df0c0045b5
Fix export snapshot and template to secondary storage to export only required disk (#5510)
* Fix export snapshot and export template to secondary storage in VMware to export only one required disk

* Move clone operation into virtual machine mo

* Code refactored for readability

* Added disk key check even for successful clone operation

* Delete dettached disks from cloned VM and added few logs
2021-09-30 12:44:02 -03:00
Rohit Yadav b955b38ccb Merge remote-tracking branch 'origin/4.15' into main 2021-09-27 16:17:49 +05:30
Abhishek Kumar 25d522fb3d
server: add vm boot details for start vm api (#5469)
Add vm boot details for start vm api

Fixes #5466

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-09-27 16:16:27 +05:30
SadiJr c3a1c7862f
backup: Improve Veeam Plugin logs (#5455)
* Improve Veeam Plugin logs

* Address reviews

* Address reviews

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2021-09-24 17:25:27 +05:30
Rohit Yadav b28dd81e8e Updating pom.xml version numbers for release 4.15.3.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-09-17 12:53:29 +05:30
Peinthor Rene 66c39c1589
storage: Linstor volume plugin (#4994)
This adds a volume(primary) storage plugin for the Linstor SDS.
Currently it can create/delete/migrate volumes, snapshots should be possible,
but currently don't work for RAW volume types in cloudstack.

* plugin-storage-volume-linstor: notify libvirt guests about the resize
2021-09-16 10:50:58 +05:30
Wei Zhou 95ef292860
kvm: honor migrate.wait and abort vm migration job (#5388)
* kvm: honor migrate.wait and abort vm migration job

* kvm: propogate migratewait to all cloudstack agents on kvm hosts

* update #5388

* update #5388: display error msg
2021-09-15 12:38:16 -03:00
Pearl Dsilva 3f827ef22b
UI: Support to upload resource icons (#5157)
* Resource Icon support - backend

* Add API support for resourceicon

* update reponse params + ui support

* Add exclusive list api for icons and UI changes

* refactor upload view

* UI changes to support resource icon wherever necessary

* convert api to POST + refactor icon view

* Add response name to list API + cosmetic changes in UI

* Added support for the following:
resource icon support for vpcs, networks, domains, and projects
add icons to list view if reosurces support icons to be added
support for showing project icons in the project switching drop-down menu

* List resourceicon cmds to be allowed for user role too
Users to inherit account icon if present (in listUsers response)
Move common code to plugin.js
Add icon to project list view - while switching between projects - Dashboard page
Show icons against zones - Capacity Dashboard view
Show user / account icon at the login button if present

* cosmetic changes

* optimize ui code

* fix reload issue for domain view

* add access check for delete operation

* ui-related changes to show iso icons

* iso image in uservm response

* add icons to custom form's list resources

* some more custom forms aligned to show icon for resources

* conmitic changes + add listing of icons to listdomainchildren cmd

* Add backend/server-side validation for base64 string passed for image

* change preview border

* preselect zone if there's only one

* add default icon

* show icon for network list in deploy vm view

* add custom icons if any to the import-export VM view

* preselect zone persistence on clearing cache

* prevent root vol from inheriting template/iso icon

* show tempalte icon in the info card details

* fix icon not being show on hard-refresh / initial traversal

* fx success message
2021-09-15 00:50:42 -03:00
Rohit Yadav 4aaa850b63 Updating pom.xml version numbers for release 4.15.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-09-10 21:19:58 +05:30
Nicolas Vazquez 413d10dd81
server: Extend the Annotations framework (#5103)
* Extend addAnnotation and listAnnotations APIs

* Allow users to add, list and remove comments

* Add adminsonly UI and allow admins or owners to remove comments

* New annotations tab

* In progress: new comments section

* Address review comments

* Fix

* Fix annotationfilter and comments section

* Add keyword and delete action

* Fix and rename annotations tab

* Update annotation visibility API and update comments table accordingly

* Allow users seeing all the comments for their owned resources

* Extend comments for volumes and snapshots

* Extend comments to multiple entities

* Add uuid to ssh keypairs

* SSH keypair UI refactor

* Extend comments to the infrastructure entities

* Add missing entities

* Fix upgrade version for ssh keypairs

* Fix typo on DB upgrade schema

* Fix annotations table columns when there is no data

* Extend the list view of items showing they if they have comments

* Remove extra test

* Add annotation permissions

* Address review comments

* Extend marvin tests for annotations

* updating ui stuff

* addition to toggle visibility

* Fix pagination on comments section

* Extend to kubernetes clusters

* Fixes after last review

* Change default value for adminsonly column

* Remove the required field for the annotationfilter parameter

* Small fixes on visibility and other fixes

* Cleanup to reduce files changed

* Rollback extra line

* Address review comments

* Fix cleanup error on smoke test

* Fix sending incorrect parameter to checkPermissions method

* Add check domain access for the calling account for domain networks

* Fix only display annotations icon if there are comments the user can see

* Simply change the Save button label to Submit

* Change order of the Tools menu to provent users getting 404 error on clicking the text instead of expanding

* Remove comments when removing entities

* Address review comments on marvin tests

* Allow users to list annotations for an entity ID

* Allow users to see all comments for allowed entities

* Fix search filters

* Remove username from search filter

* Add pagination to the annotations tab

* Display username for user comments

* Fix add permissions for domain and resource admins

* Fix for domain admins

* Trivial but important UI fix

* Replace pagination for annotations tab

* Add confirmation for delete comment

* Lint warnings

* Fix reduced list as domain admin

* Fix display remove comment button for non admins

* Improve display remove action button

* Remove unused parameter on groupShow

* Include a clock icon to the all comments filter except for root admin

* Move cleanup SQL to the correct file after rebasing main

Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
2021-09-08 10:14:06 +05:30
Abhishek Kumar 3d16584fa6
kvm: fix misleading error log (#5396)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-09-08 10:10:21 +05:30
Wei Zhou a755ecfce8
Migrate vm across clusters (#4534)
* server: Optional destination host when migrate a vm

* #4378: migrate systemvms/routers with optional host

* Migrate vms across clusters

After enabling maintenance mode on host, if no suitable hosts
are found in the same cluster then search for hosts in
different clusters having the same hypervisor type

set global setting migrate.vm.across.clusters to true

* search all clusters in zone when migrate vm across clusters if applicable

* Honor migrate.vm.across.clusters when migrate vm without destination

* Check MIGRATE_VM_ACROSS_CLUSTERS in zone setting

* #4534 Fix Vms are migrated to same clusters in CloudStack caused by dedicated resources.

* #4534 extract some codes to methods

* fix #4534: an error in 'git merge'

* fix #4534: remove useless methods in FirstFitPlanner.java

* fix #4534: vms are stopped in host maintenance

* fix #4534: across-cluster migration of vms with cluster-scoped pools is supported by vmware vmotion

* fix #4534: migrate systemvms is only possible across clusters in same pod to avoid potential network errors.

* fix #4534: code optimization

Co-authored-by: Rakesh Venkatesh <r.venkatesh@global.leaseweb.com>
Co-authored-by: Sina Kashipazha <s.kashipazha@global.leaseweb.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Sina Kashipazha <soreana@users.noreply.github.com>
2021-09-07 21:50:29 -03:00
nicolas 4de35426c4
Merge branch '4.15' into main 2021-09-02 22:04:12 -03:00
Rohit Yadav 2297c73c92
vmware: delete snapshot disk after backup to secondary storage (#5333)
* vmware: delete snapshot disk after backup to secondary storage

WIP - This ensures that worker VM is destroyed along with any of its own
disks that are backed up to secondary storage.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix for volume backup and confuding vm var name

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* tag as worker vm

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-09-03 00:38:44 +05:30
Abhishek Kumar 56f4da6dce Merge remote-tracking branch 'apache/4.15' into main 2021-09-02 16:13:33 +05:30
sureshanaparti fc285e23f3
vmware: Cancel the pending tasks for worker VM before destroying it (#5374)
Co-authored-by: nicolas <nicovazquez90@gmail.com>
2021-09-02 11:02:20 +05:30
Abhishek Kumar 8998479606
vmware: check checksum before copying systemvm ISO to decide if it is needed (#5380)
* vmware: fix copy systemvm.iso for same version

For VMware, systemvm.iso is copied from MS to secondary store. Current server checks if the desired file is present on the secondary store or not. If it is not present ISO is copied.
This change adds a check for checksum for source and destination ISO which would allow copying new ISO if there is a mismatch.
Useful in case when file is corrupted in secondary store or new systemvm.iso is generated for dev environment.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-09-02 10:53:29 +05:30
Slair1 76d5ce310a
allow cert renewal even if auth strictness is false (#4852)
includes updated tests and logging
2021-09-01 18:19:07 +05:30
Wei Zhou 4e53997ca2
server: do not remove volume from DB if fail to expunge it from primary storage or secondary storage (#5373)
* server: do not remove volume from DB if fail to expunge it from primary storage or secondary storage

* server/VolumeApiServiceImpl.java: move to method

* update #5373
2021-08-31 13:48:58 -03:00
Abhishek Kumar 73cabcd641
xcp-ng: allow passing vm boot options (#5335)
* xenserver: honor vm boot details

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* ui: allow boot option selection for xenserver

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix case

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* host uefi capability

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add detail only if uefi supported

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update host detail

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix version comparison

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-08-31 16:10:12 +05:30
Rohit Yadav a1a3aff2b5 Merge remote-tracking branch 'origin/4.15' into main 2021-08-31 14:29:30 +05:30
slavkap 961e85eb60
Fix of creating volumes from snapshots without backup to secondary storage (#5349)
* Fix of creating volumes from snapshots without backup

When few snaphots are created onyl on primary storage, and try to create
a volume or a template from the snapshot only the first operation is
successful. Its because the snapshot is backup on secondary storage with
wrong SQL query. The problem appears on Ceph/NFS but may affects other
storage plugins.
Bypassing secondary storage is implemented only for Ceph primary storage
and it didn't cover the functionality to create volume from snapshot
which is kept only on Ceph

* Address review
2021-08-31 12:46:57 +05:30
Rafael 14323c9db5
updated maven dependency due to #5363 (#5366)
Co-authored-by: Rafael del Valle <rvalle@privaz.io>
2021-08-31 12:39:18 +05:30
nicolas 3ddcf85876
Merge branch '4.15' into main 2021-08-30 22:32:27 -03:00
sureshanaparti e50a40ea08
Start VM with deploy-as-is template having multiple disk (SCSI) controller types. (#5311) 2021-08-30 21:52:24 -03:00
sureshanaparti 46ca853e55
Added Worker VM tags for few cloned VMs, created while performing some volume operations. (#5377)
Worker VM tags are missed for few cloned VMs in VMware, and so these are skipped when tracking / cleaning up of Worker VMs. Adding proper Worker VM tags to these VMs would make them trackable from CloudStack.
2021-08-30 20:22:06 -03:00
Spaceman1984 1d3083db86
Added support for removing unused port groups on VMWare (#4701)
* Added support for removing unused port groups on VMWare

* Fixed error handling around unavailable portgroup name

* Review changes, defaulting glovbal var to false, added warning to description, changed if statement.

* Cleanup unused network port groups on all the hosts.

Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: nicolas <nicovazquez90@gmail.com>
2021-08-28 09:04:50 -03:00
sureshanaparti 7f4f3f7f1a
[VMware DRS] Adding new host to DRS cluster does not participate in load balancing. (#1257)
* CLOUDSTACK-9175: [VMware DRS] Adding new host to DRS cluster does not participate in load balancing.

Summary: When a new host is added to a cluster, Cloudstack doesn't create all the port groups (created by cloudstack earlier in other hosts) present in the cluster. Since the new host doesn't have all the necessary networking port groups of cloudstack, it is not eligible to participate in DRS load balancing or HA.

Solution: When adding a host to the cluster in Cloudstack, use VMware API to find the list of unique port groups on a previously added host (older host in the cluster) if exists and then create them on the new host.

* Added few checks for cluster details
2021-08-27 14:22:47 -03:00
SadiJr 8e0e5acea7
Externalize VMWare stats time window config (#5357)
Co-authored-by: SadiJr <17a0db2854@firemailbox.club>
2021-08-26 08:27:48 -03:00
Daniel Augusto Veronezi Salvador 159c72fa97
Externalize KVM Agent's option to change migration thread timeout (#4570)
* Externalize KVM Agent's option to change migration thread timeout

* Update javadoc

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-26 08:18:17 -03:00
DK101010 9163013683
Feat/ram reservation (#4662)
* remove hot enable cpu und memory in case of reservation

ram and cpu reservation have not relation to ram and cpu hot add

* add custom ram_reservation and it to vm details

* system vms haven't this property, for this reason add additional check

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* replace 0.0 with NumberUtils

* remove default value and remove return MinRam(seems to be not necessary)

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/guru/VmwareVmImplementer.java

Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>

* Update plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: dahn <daan.hoogland@gmail.com>
Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
2021-08-24 14:15:52 -03:00
Daniel Augusto Veronezi Salvador 83c0b61ab2
Externalize KVM Agent storage's reboot configuration (#4586)
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-24 01:06:00 -03:00
Daniel Augusto Veronezi Salvador 8a16729fcf
Support vm dynamic scaling with kvm (#4878)
* Create utility to centralize byte convertions

* Add/change toString definitions

* Create Libvirt handler to ScaleVmCommand

* Enable dynamic scalling VM with KVM

* Move config from interface to class and rename it

As every variable declared in interfaces are already final,
this moving will be needed to mock tests in nexts commits

* Configure VM max memory and cpu cores

The values are according to service offering or global configs

* Extract dpdk configuration to a method and test it

* Extract OS desc config to a method and test it

* Extract guest resource def to a method and test it

Improve libvirt def

* Refactor LibvirtVMDef.GuestResourceDef

* Refactor ScaleVmCommand

* Improve VMInstaVO toString()

* Refactor upgradeRunningVirtualMachine method

* Turn int variables into long on utility

* Verify if VM is scalable on KVMGuru

* Rename some KVMGuruTest's methods

* Change vm's xml to work with max memory

* Verify if service offering is dynamic before scale

* Create methods to retrieve data from domain

* Create def to hotplug memory

* Adjust the way command was scaling the VM

* Fix database persistence before executing command

* Send more info to host to improve log

* Fix var name

* Fix missing "}"

* Undo unnecessary changes

* Address review

* Fix scale validation

* Add VM prepared for dynamic scaling validation

* Refactor LibvirtScaleVmCommandWrapper and improve unit tests

* Remove duplicated method

* Add RuntimeException check

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Update ByteScaleUtilsTest.java

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-08-21 09:29:02 +02:00
Rohit Yadav d23a9954e8 Merge remote-tracking branch 'origin/4.15' into main
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-08-19 17:46:58 +05:30
Wei Zhou 7fa63b3fe9
add license header in HostMetricsResponseTest.java (#5342) 2021-08-19 17:41:34 +05:30
Rohit Yadav c34a0c5f92 Merge remote-tracking branch 'origin/4.15' into main
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-08-19 16:07:35 +05:30