Commit Graph

2943 Commits

Author SHA1 Message Date
Wei Zhou 669ab73efe
server: check service offering (storage) tags when reallocate a ROOT disk (#5501)
* server: check service offering (storage) tags when reallocate a ROOT disk

* server: resize volumes in Allocated state
2021-10-03 19:45:59 -03:00
Abhishek Kumar 4a42e7ef9e
vmware, ui: update portgroup on network update (#5470)
Enhanced update network form in the UI.
On network offering change for an isolated network,

- VMware portgroup should be updated accordingly.
- VMs on the network should be placed on the correct VMware portgroup based on the network rate, https://docs.cloudstack.apache.org/en/latest/adminguide/service_offerings.html#network-throttling.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-10-03 14:41:42 +05:30
SadiJr 965a47fdfd
Create UpdateBackupOffering API (#5511)
* Create UpdateBackupOffering API

* Address reviews

* Address reviews

* Address reviews

Co-authored-by: SadiJr <sadi@scclouds.com.br>
2021-10-01 08:30:25 -03:00
Wei Zhou ea643a64e5
server: cannot deploy/start vm if service offering has multiple tags (#5521) 2021-09-29 14:59:48 +05:30
Rakesh dcc02e0fbb
Enable account settings to be visible under domain settings (#4215)
* Enable account settings to be visible under domain settings

All the account settings can't be configured under domain
level settings right now.
By default, if account setting is not configured then
its value will be taken from global setting.
Add a global setting "enable.account.settings.for.domain"
so that if its enabled then all the account level settings
will be visible under domain levelsettings also.
If account level setting is configured then that value will
be considered else it will take domain scope value. If
domain scope value is not configured then it will pick
it up from global setting.

If domain level setting is not configured then by default
the value will be taken from global setting
Add another global setting "enable.domain.settings.for.child.domain"
so that when its true, if a value for domain setting is not
configured then its parent domain value is considered until
it reaches ROOT domain. If no value is configured till ROOT
domain then global setting value will be taken.

Also display all the settings configured under the domain level
in list domains api response

* rename variables

Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
2021-09-29 10:29:20 +02:00
Pearl Dsilva 74bb80687d
resource limit: Fix resource limit check on VM start (#5428)
* resource limit: Fix resource limit check on VM start

* add check to validate if cpu/memory are within limits for custom offering + exception handling

* unit tests

Co-authored-by: utchoang <hoangnm@unitech.vn>
2021-09-24 09:51:16 +05:30
sureshanaparti 34bd92259a
New API endpoint to update pod management network IP range (#5458)
* Add UpdatePodManagementNetwork api endpoint

* Checkstyle changes and added a few methods

* Minor Checkstyle change

* Refactor UpdatePodManagementNetworkIpRangeCmd.java

* Added missing parameters

* Cleanup

* Addressed the review comments

Co-authored-by: kioie <kioieddy@google.com>
Co-authored-by: kioie <kioi@outlook.com>
2021-09-20 18:19:05 -03:00
Pearl Dsilva 93150f465b
api: Fix list templates when no secondary stores present (#5468) 2021-09-20 14:07:47 -03:00
dahn 22a0c0a46c
change logging during upgrade (#5474)
* reduce ERROR logging on ignored exception

* string concat formatting

Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: sureshanaparti <12028987+sureshanaparti@users.noreply.github.com>
2021-09-20 13:00:06 -03:00
Wei Zhou 747608f75f
Add New API endpoint: UpdateVlanIpRange (#5411)
* Added Logic to update the user_ip_address table

* Edited ConfigurationManagerImpl

* Refactor UpdateVlanIpRangeCmd location

* Checkstyle corrections

* Mock updateVlanAndPublicIpRange

* Changes:
- UpdateVlanIpRangeCmd - changed since to 4.15.0
- ConfigurationService - Updated Javadoc
- ConfigurationManager - Updated Javadoc
- Added Unit tests
- Added license
- Update server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java
- fix some bugs in #5411 and add support for ipv6 and forsystemvms
- fix #5411: disallow forsystemvms if ip range is dedicated
- update #5411: ui changes
- update #5411: support gateway/netmask change
- update #5411: change to sync call and fix bugs

Co-authored-by: kioie <kioieddy@google.com>
Co-authored-by: kioie <kioi@outlook.com>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-09-20 10:56:29 +05:30
Nicolas Vazquez 8228ecee43
[Vmware] Add missing condition to cleanup nics if there are commands to send (#5449)
* [Vmware] Add missing condition to cleanup nics if there are commands to send
2021-09-17 21:54:52 +05:30
Nicolas Vazquez 3ca3843b02
[Vmware] Fix for ovf templates with prefix (#5448)
* [Vmware] Fix for ovf templates with prefix

* Support multiple hardware versions
2021-09-16 16:16:41 -03:00
Peinthor Rene 66c39c1589
storage: Linstor volume plugin (#4994)
This adds a volume(primary) storage plugin for the Linstor SDS.
Currently it can create/delete/migrate volumes, snapshots should be possible,
but currently don't work for RAW volume types in cloudstack.

* plugin-storage-volume-linstor: notify libvirt guests about the resize
2021-09-16 10:50:58 +05:30
Wei Zhou 95ef292860
kvm: honor migrate.wait and abort vm migration job (#5388)
* kvm: honor migrate.wait and abort vm migration job

* kvm: propogate migratewait to all cloudstack agents on kvm hosts

* update #5388

* update #5388: display error msg
2021-09-15 12:38:16 -03:00
Pearl Dsilva 3f827ef22b
UI: Support to upload resource icons (#5157)
* Resource Icon support - backend

* Add API support for resourceicon

* update reponse params + ui support

* Add exclusive list api for icons and UI changes

* refactor upload view

* UI changes to support resource icon wherever necessary

* convert api to POST + refactor icon view

* Add response name to list API + cosmetic changes in UI

* Added support for the following:
resource icon support for vpcs, networks, domains, and projects
add icons to list view if reosurces support icons to be added
support for showing project icons in the project switching drop-down menu

* List resourceicon cmds to be allowed for user role too
Users to inherit account icon if present (in listUsers response)
Move common code to plugin.js
Add icon to project list view - while switching between projects - Dashboard page
Show icons against zones - Capacity Dashboard view
Show user / account icon at the login button if present

* cosmetic changes

* optimize ui code

* fix reload issue for domain view

* add access check for delete operation

* ui-related changes to show iso icons

* iso image in uservm response

* add icons to custom form's list resources

* some more custom forms aligned to show icon for resources

* conmitic changes + add listing of icons to listdomainchildren cmd

* Add backend/server-side validation for base64 string passed for image

* change preview border

* preselect zone if there's only one

* add default icon

* show icon for network list in deploy vm view

* add custom icons if any to the import-export VM view

* preselect zone persistence on clearing cache

* prevent root vol from inheriting template/iso icon

* show tempalte icon in the info card details

* fix icon not being show on hard-refresh / initial traversal

* fx success message
2021-09-15 00:50:42 -03:00
Rakesh 3c2360c9d2
VM has wrong network statistics with multiple nics in shared networks (#4741)
If vm has multiple nics belonging to different shared networks then
wrong statistics will be collected since network id is not considred
as primary key. Make the change so that primary key contains network
id so that traffic belonging to that corresponding network is shown

If network id is not added to primary key then all the traffic of all
shared networks will show up in one nic.

Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
2021-09-14 23:54:34 -03:00
Junxuan Wu f6073052aa
Fix potential NullPointerException in findStoragePool (VolumeOrchestrator) (#5358)
* fix null pointer exception when vm is null

* add null checker to getPreferredStoragePool method

Co-authored-by: junxuan <atrocitythemetms@gmail.com>
2021-09-08 23:43:33 +05:30
Daniel Augusto Veronezi Salvador 8ffba83214
Keep volume policies after migrating it to another primary storage (#5067)
* Add commons-lang3 to Utils

* Create an util to provide methods that ReflectionToStringBuilder does not have yet

* Create method to retrieve map of tags from resource

* Enable tests on volume components and remove useless tests

* Refactor VolumeObject and add unit tests

* Extract createPolicy in several methods

* Create method to copy policies between volumes and add unit tests

* Copy policies to new volume before removing old volume on volume migration

* Extract "destroySourceVolumeAfterMigration" to a method and test it

* Remove javadoc @param with no sensible information

* Rename method name to a generic name

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-09-08 09:13:41 -03:00
Nicolas Vazquez 413d10dd81
server: Extend the Annotations framework (#5103)
* Extend addAnnotation and listAnnotations APIs

* Allow users to add, list and remove comments

* Add adminsonly UI and allow admins or owners to remove comments

* New annotations tab

* In progress: new comments section

* Address review comments

* Fix

* Fix annotationfilter and comments section

* Add keyword and delete action

* Fix and rename annotations tab

* Update annotation visibility API and update comments table accordingly

* Allow users seeing all the comments for their owned resources

* Extend comments for volumes and snapshots

* Extend comments to multiple entities

* Add uuid to ssh keypairs

* SSH keypair UI refactor

* Extend comments to the infrastructure entities

* Add missing entities

* Fix upgrade version for ssh keypairs

* Fix typo on DB upgrade schema

* Fix annotations table columns when there is no data

* Extend the list view of items showing they if they have comments

* Remove extra test

* Add annotation permissions

* Address review comments

* Extend marvin tests for annotations

* updating ui stuff

* addition to toggle visibility

* Fix pagination on comments section

* Extend to kubernetes clusters

* Fixes after last review

* Change default value for adminsonly column

* Remove the required field for the annotationfilter parameter

* Small fixes on visibility and other fixes

* Cleanup to reduce files changed

* Rollback extra line

* Address review comments

* Fix cleanup error on smoke test

* Fix sending incorrect parameter to checkPermissions method

* Add check domain access for the calling account for domain networks

* Fix only display annotations icon if there are comments the user can see

* Simply change the Save button label to Submit

* Change order of the Tools menu to provent users getting 404 error on clicking the text instead of expanding

* Remove comments when removing entities

* Address review comments on marvin tests

* Allow users to list annotations for an entity ID

* Allow users to see all comments for allowed entities

* Fix search filters

* Remove username from search filter

* Add pagination to the annotations tab

* Display username for user comments

* Fix add permissions for domain and resource admins

* Fix for domain admins

* Trivial but important UI fix

* Replace pagination for annotations tab

* Add confirmation for delete comment

* Lint warnings

* Fix reduced list as domain admin

* Fix display remove comment button for non admins

* Improve display remove action button

* Remove unused parameter on groupShow

* Include a clock icon to the all comments filter except for root admin

* Move cleanup SQL to the correct file after rebasing main

Co-authored-by: davidjumani <dj.davidjumani1994@gmail.com>
2021-09-08 10:14:06 +05:30
Wei Zhou a755ecfce8
Migrate vm across clusters (#4534)
* server: Optional destination host when migrate a vm

* #4378: migrate systemvms/routers with optional host

* Migrate vms across clusters

After enabling maintenance mode on host, if no suitable hosts
are found in the same cluster then search for hosts in
different clusters having the same hypervisor type

set global setting migrate.vm.across.clusters to true

* search all clusters in zone when migrate vm across clusters if applicable

* Honor migrate.vm.across.clusters when migrate vm without destination

* Check MIGRATE_VM_ACROSS_CLUSTERS in zone setting

* #4534 Fix Vms are migrated to same clusters in CloudStack caused by dedicated resources.

* #4534 extract some codes to methods

* fix #4534: an error in 'git merge'

* fix #4534: remove useless methods in FirstFitPlanner.java

* fix #4534: vms are stopped in host maintenance

* fix #4534: across-cluster migration of vms with cluster-scoped pools is supported by vmware vmotion

* fix #4534: migrate systemvms is only possible across clusters in same pod to avoid potential network errors.

* fix #4534: code optimization

Co-authored-by: Rakesh Venkatesh <r.venkatesh@global.leaseweb.com>
Co-authored-by: Sina Kashipazha <s.kashipazha@global.leaseweb.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
Co-authored-by: Sina Kashipazha <soreana@users.noreply.github.com>
2021-09-07 21:50:29 -03:00
davidjumani a000361238
Merge remote-tracking branch 'apache/4.15' into main 2021-09-03 16:55:10 +05:30
Pearl Dsilva b2fd1f6d32
server: Allow public templates with no url to be migrated (#5404)
This PR allows migration of public templates that are created from snapshots / volumes. Data migration across secondary stores initially excluded all public templates on the pretext that public templates are automatically synced when a new image store is added; however, this assumption isn't true for templates marked as "public" when created from snapshots / volumes. Such templates can be identified if their url is null
2021-09-03 16:25:11 +05:30
davidjumani 0bc6e05bc4
ui: Honour default.ui.page.size (#5394)
* ui: Honour default.ui.page.size

* Moving db upgrade path
2021-09-03 16:23:26 +05:30
nicolas 13d8489478
Merge branch '4.15' into main 2021-09-03 00:06:08 -03:00
davidjumani a5e7e08a6b
Filter disk / service offerings by domain at DB level (#5307)
* Filter disk / service offerings by domain at DB level

* Search for tags in the db

* Update search to include host tags

* Differenciate between tags

* Refactor
2021-09-02 23:56:18 -03:00
nicolas eb64141785
Merge branch '4.15' into main 2021-09-02 12:16:06 -03:00
Nicolas Vazquez e507b57b41
Add 4.15.2 schema and upgrade path (#5403) 2021-09-02 20:27:38 +05:30
Abhishek Kumar 56f4da6dce Merge remote-tracking branch 'apache/4.15' into main 2021-09-02 16:13:33 +05:30
Pearl Dsilva 557dc5e1a0
api: List details of template download state for stores corresponding to a zone (#5379)
* api: List details of template download state for stores corresponding to a zone

* fix test
2021-09-02 10:58:58 +05:30
Rohit Yadav a1a3aff2b5 Merge remote-tracking branch 'origin/4.15' into main 2021-08-31 14:29:30 +05:30
slavkap 961e85eb60
Fix of creating volumes from snapshots without backup to secondary storage (#5349)
* Fix of creating volumes from snapshots without backup

When few snaphots are created onyl on primary storage, and try to create
a volume or a template from the snapshot only the first operation is
successful. Its because the snapshot is backup on secondary storage with
wrong SQL query. The problem appears on Ceph/NFS but may affects other
storage plugins.
Bypassing secondary storage is implemented only for Ceph primary storage
and it didn't cover the functionality to create volume from snapshot
which is kept only on Ceph

* Address review
2021-08-31 12:46:57 +05:30
Rohit Yadav 5837f9016e
engine/schema: Use same upgrade path as 4.15.1-4.16.0 as for 4.15.2 (#5385)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-08-31 12:35:47 +05:30
sureshanaparti 7f4f3f7f1a
[VMware DRS] Adding new host to DRS cluster does not participate in load balancing. (#1257)
* CLOUDSTACK-9175: [VMware DRS] Adding new host to DRS cluster does not participate in load balancing.

Summary: When a new host is added to a cluster, Cloudstack doesn't create all the port groups (created by cloudstack earlier in other hosts) present in the cluster. Since the new host doesn't have all the necessary networking port groups of cloudstack, it is not eligible to participate in DRS load balancing or HA.

Solution: When adding a host to the cluster in Cloudstack, use VMware API to find the list of unique port groups on a previously added host (older host in the cluster) if exists and then create them on the new host.

* Added few checks for cluster details
2021-08-27 14:22:47 -03:00
Daniel Augusto Veronezi Salvador 8a16729fcf
Support vm dynamic scaling with kvm (#4878)
* Create utility to centralize byte convertions

* Add/change toString definitions

* Create Libvirt handler to ScaleVmCommand

* Enable dynamic scalling VM with KVM

* Move config from interface to class and rename it

As every variable declared in interfaces are already final,
this moving will be needed to mock tests in nexts commits

* Configure VM max memory and cpu cores

The values are according to service offering or global configs

* Extract dpdk configuration to a method and test it

* Extract OS desc config to a method and test it

* Extract guest resource def to a method and test it

Improve libvirt def

* Refactor LibvirtVMDef.GuestResourceDef

* Refactor ScaleVmCommand

* Improve VMInstaVO toString()

* Refactor upgradeRunningVirtualMachine method

* Turn int variables into long on utility

* Verify if VM is scalable on KVMGuru

* Rename some KVMGuruTest's methods

* Change vm's xml to work with max memory

* Verify if service offering is dynamic before scale

* Create methods to retrieve data from domain

* Create def to hotplug memory

* Adjust the way command was scaling the VM

* Fix database persistence before executing command

* Send more info to host to improve log

* Fix var name

* Fix missing "}"

* Undo unnecessary changes

* Address review

* Fix scale validation

* Add VM prepared for dynamic scaling validation

* Refactor LibvirtScaleVmCommandWrapper and improve unit tests

* Remove duplicated method

* Add RuntimeException check

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Remove copyright from header

* Update ByteScaleUtilsTest.java

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-08-21 09:29:02 +02:00
Daniel Augusto Veronezi Salvador 9c51009134
Remove storage scope validation on KVM live migration (#5321)
Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-20 14:54:14 -03:00
Daniel Augusto Veronezi Salvador 65a48dcb74
Add SharedMountPoint to KVMs supported storage pool types (#4780)
* Add SharedMountPoint to KVMs supported storage pool types

* Fix live migration to iSCSI and improve logs

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-08-16 12:32:19 -03:00
DK101010 664a46a525
PR multi tags in compute offering [#4398] (#4399)
* [#4398] adapt code to handle multi tag string with commas

* [#4398] remove trailing spaces

* [#4398] add multi host tag support for ingest process

* [#4398] add test for multi tag support in offerings

* [#4398]  update multitag support for DeploymentPlanningManagerImpl

encapsulate multi tag check from Ingest Feature, DepolymentPlanningManager into
HostDaoImpl to prevent code duplicates

* [#4398] move logic to HostVO and add tests

* rename test method

* [#4398] Change string method to apaches StringUtils

* [#4398] modify test for multi tag support

* adapt sql for double tags

Co-authored-by: Dirk Klahre <Dirk.Klahre@Itelligence.de>
2021-08-16 12:08:40 -03:00
Abhishek Kumar 87ddc76277
db: make *_details.value non-nullable (#5274)
Fixes #4897
Some details tables were allowing null values for detail value which can cause NPE in some cases.
mysql> SELECT TABLE_NAME, COLUMN_NAME, COLUMN_TYPE FROM information_schema.columns WHERE table_schema='cloud' AND table_name LIKE'%_details' AND column_name='value' AND IS_NULLABLE='YES';
+-------------------------------+-------------+---------------+
| TABLE_NAME                    | COLUMN_NAME | COLUMN_TYPE   |
+-------------------------------+-------------+---------------+
| account_details               | value       | varchar(255)  |
| cluster_details               | value       | varchar(255)  |
| data_center_details           | value       | varchar(1024) |
| domain_details                | value       | varchar(255)  |
| image_store_details           | value       | varchar(255)  |
| storage_pool_details          | value       | varchar(255)  |
| template_deploy_as_is_details | value       | text          |
| user_vm_deploy_as_is_details  | value       | text          |
| user_vm_details               | value       | varchar(5120) |
+-------------------------------+-------------+---------------+
9 rows in set (0.00 sec)

Brings consistency for value column of *_details tables with preventing null values.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-08-13 21:37:43 -03:00
Daniel Augusto Veronezi Salvador 278658283e
Add new registers in guest_os (#4699)
* Add new registers in guest_os

* Create a procedure to insert guest_os and guest_os_hypervisor data

* Remove ';' as the last char of the procedure

* Set the right category_id on guest_os

Ubuntu 20.04 LTS - Ubuntu - Linux
Ubuntu 21.04 - Ubuntu - Linux
pfSense 2.4 - FreeBSD - Unix
OpenBSD 6.7 - Unix
OpenBSD 6.8 - Unix
AlmaLinux 8.3 - CentOS

* Fix SQL line's last character

* Add from with dummy table

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-08-12 11:32:09 -03:00
Rakesh 2a4c2c2506
Global setting to select preferred storage pool (#5249)
* Global setting to select preferred storage pool

Currently all the volumes are allocated on storage pools
based on the capacity or the algorithm selected. Sometimes
we need to deploy all volumes of particular account in a
specific storage pool and in that case its not possible.

with this change, we can specify the uuid of the preferred
storage pool, so that all volumes of the account will be
deployed in this pool

* code feedback

Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
2021-08-12 00:01:15 -03:00
davidjumani cb1078aa20
Merge remote-tracking branch 'apache/4.15' into main 2021-08-09 15:10:51 +05:30
Pearl Dsilva ea7d3b34d1
Cleanup volume information from db when deleted (#4551)
* Cleanup volume information from db when deleted

* reuse search builder

* revert change

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-08-09 14:21:07 +05:30
Rakesh 949cf71491
api: Display last updated time for VM (#4685)
Add a field in api response and ui to display last updated time
2021-08-09 14:17:53 +05:30
Rakesh 2f8d557f58
api: Change GET/POST request max length of VM user data to 4K/1M (#4737)
Currently we can send a default value of 4K/32K for GET/POST request of
user data field. Most new browsers and also nginx support till 1MB of
post data.

Added a new global setting `vm.userdata.max.length` with default value of
32KB which can be increased till 1MB.
2021-08-09 14:14:19 +05:30
Abhishek Kumar 1ccb42017f
server: skip zone check for PERHOST iso during attachIso (#5270)
* server: skip zone check for PERHOST iso during attachIso

Hypervisor tools ISO - vmware-toools.iso, xs-tools.iso are marked as PERHOST in DB. They are active but not downloaded to the secondary storages and hence no template-zone entry.
Skips the template-zone check for such templates.

Fixes #5265

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* inverted check

* use constants in TemplateManager

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-08-09 14:02:25 +05:30
Daniel Augusto Veronezi Salvador 1389862c22
engine/storage: Fix regression on create volume from snapshot (#5282)
* Fix regression on create volume from snapshot

* Log hidden exception

* Revert "Log hidden exception"

This reverts commit 70e655687f.

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-08-09 13:37:10 +05:30
slavkap d6a77a72f0
Allow updating the storage/host tags of service offerings (#5043) 2021-08-02 15:48:07 +02:00
Gabriel Beims Bräscher 0d8b4de1b2
Cleaning up code and enhancing a few IP management logs (#4714)
* Cleanup unnecessary code and enhance a few log messages at IpAddressManagerImpl

* Add toString method for DataCenterVO

* line too long
2021-07-30 16:38:11 +02:00
Sina Kashipazha 490658c5c3
display nics deviceid and order nics by deviceid on Nics tab of instances. (#4768) 2021-07-28 15:11:43 -03:00
Daniel Augusto Veronezi Salvador 1f5ee5b3e3
Improve logs on ConsoleProxyManagerImpl and refactor a few process (#4959) 2021-07-27 19:56:14 -03:00
Daniel Augusto Veronezi Salvador eb3acc334b
Externalize tls version and security protocols configuration on mail sending (#5119)
* Externalize configs to alert

* Externalize configs to project

Co-authored-by: GutoVeronezi <daniel@scclouds.com.br>
2021-07-21 09:01:11 -03:00
Pearl Dsilva 1f743e911a
api: Add 'created' field to API response (#5210)
* api: Add 'created' field to API response

* address comments
2021-07-20 18:04:13 -03:00
Spaceman1984 96c9c5a5e2
Added disk provisioning type support for VMWare (#4640)
* Added disk provisioning type support for VMWare

* Review changes

* Fixed unit test

* Review changes

* Added missing licenses

* Review changes

* Update StoragePoolInfo.java

Removed white space

* Review change - Getting disk provisioning strictness setting using the zone id and not the pool id

* Delete __init__.py

* Merge fix

* Fixed failing test

* Added comment about parameters

* Added error log when update fails

* Added exception when using API

* Ordering storage pool selection to prefer thick disk capable pools if available

* Removed unused parameter

* Reordering changes

* Returning storage pool details after update

* Removed multiple pool update, updated marvin test, removed duplicate enum

* Removed comment

* Removed unused import

* Removed for loop

* Added missing return statements for failed checks

* Class name change

* Null pointer

* Added more info when a deployment fails

* Null pointer

* Update api/src/main/java/org/apache/cloudstack/api/BaseListCmd.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

* Small bug fix on API response and added missing bracket

* Removed datastore cluster code

* Removed unused imports, added missing signature

* Removed duplicate config key

* Revert "Added more info when a deployment fails"

This reverts commit 2486db78dc.

Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-07-16 22:37:42 -03:00
Daniel Augusto Veronezi Salvador cbe380a068
Externalize secondary storage capacity threshold (#4790)
* Externalize secondary storage capacity threshold

* Use default value as threshold when config value is lower than 0.0

* Move config to CapacityManager

* Validate config in CapacityManagerImpl

* Use config in StorageOrchestrator

* Change config description

* Remove unused import

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-07-16 08:38:36 +02:00
Abhishek Kumar 50a16979c5
refactor: migrate vm with storage (#5030)
* refactor: migrate with storage host capability check

Refactors Boolean HypervisorCapabilitiesDao::isStorageMotionSupported to boolean HypervisorCapabilitiesDao::isStorageMotionSupported for simplifying callers.
Refactors log messages.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* simplify

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* review comments addressed

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* var rename

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-07-15 12:57:13 +05:30
Gabriel Beims Bräscher ca78f5b386
Enhance log messages with host name (#4575)
* Enhance log messages with hostName

* Use host.toString() on most of host logs.

* Remove redundant "Host" in logs and enhance logs

* duplicated "for"

* Adopt String.format, and enhance code

* Address reviews enhancing log messages

Update server/src/main/java/com/cloud/resource/ResourceManagerImpl.java
    -- server/src/main/java/com/cloud/vm/UserVmManagerImpl.java
    -- server/src/main/java/com/cloud/resource/RollingMaintenanceManagerImpl.java

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

* Fix String.format issue and change log message from debug to warn

* Fix checkstyle issue

* Fix string.format log

* Address review: enhance logs

* Enhance log of hosts in maintenance avoid list

* Remove "VM" on logs as vm.toString() already appends VM-<details>

* Add more details of the VM when postStateTransitionEvent

* Address reviewer and enhance VMInstanceVO.toString()

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
2021-07-13 17:35:59 -03:00
Pearl Dsilva 3fd92502bb
Prevent starting a VM in destroyed state (or any state but Stopped) (#5165)
* Prevent starting a VM in destroyed state (or any state but Stopped)
2021-07-09 15:11:30 +05:30
Rohit Yadav d916e416ec Updating pom.xml version numbers for release 4.15.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-07-02 22:59:07 +05:30
Abhishek Kumar 5fd970d88f
db, server: refactor host_view to prevent duplicate entries (#4796)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-07-01 11:00:51 -03:00
Rohit Yadav 379454caae Updating pom.xml version numbers for release 4.15.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-28 15:27:27 +05:30
Rohit Yadav f58b72f6f7 Merge remote-tracking branch 'origin/4.15' 2021-06-27 18:25:46 +05:30
DK101010 53963256d8
server: Bug/false positive success message vm start (#5148)
* add throws statement during the retry process

* Update engine/orchestration/src/main/java/org/apache/cloudstack/engine/cloud/entity/api/VMEntityManagerImpl.java

Co-authored-by: dahn <daan.hoogland@gmail.com>

Co-authored-by: DK101010 <dirk.klahre@itelligence.de>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-06-27 06:40:30 +05:30
slavkap d82909318f
server: Fix of delete of Ceph's snapshots from secondary storage (#5130)
This PR fixes the deletion will be handled by DefaultSnapshotStrategy::deleteSnapshot #4797
2021-06-25 12:04:36 +05:30
davidjumani 29109b4332
db: Cleanup obsolete tables (#5002)
* db: Cleanup unused tables

* Removing volume_host_ref references

* Removing template_host_ref references

* fix space issue

* Fix fk constraint

* Removing certificate table

* Revert "Removing certificate table"

This reverts commit fa24e6483f.

* Addressing comments
2021-06-24 16:50:31 -03:00
davidjumani 1a7bfa98d8
Follow consistent vo table name convention (#5004) 2021-06-24 16:46:50 -03:00
nicolas d2ab350a40 Merge branch '4.15' into main 2021-06-16 15:38:18 -03:00
sureshanaparti 07cabbe7ac
scaleio: Updated PowerFlex/ScaleIO gateway client with some improvements. (#5037)
- Added connection manager to the gateway client.
 - Renew the client session on '401 Unauthorized' response.
 - Refactored the gateway client calls, for GET and POST methods.
 - Consume the http entity content after login/(re)authentication and close the content stream if exists.
 - Updated storage pool client connection timeout configuration 'storage.pool.client.timeout' to non-dynamic.
 - Added storage pool client max connections configuration 'storage.pool.client.max.connections' (default: 100) to specify the maximum connections for the ScaleIO storage pool client.
 - Updated unit tests.
and blocked the attach volume operation for uploaded volume on ScaleIO/PowerFlex storage pool
2021-06-16 12:45:27 +05:30
Harikrishna 12b2e80d82
vmware: Fix fetching chain_info of the volumes. It is used to assume datastore names are in the form of UUIDs but it can be any name. So fetch chain_info based on the datastore name. (#5097)
his PR fixes the problem of not updating the chain info or setting chain info to null after volume migrations.

Problem: While fetching the volume chain info, management server assumes datastore name to be a UUID (this is true only for NFS storages added by CloudStack) but datastore name can be with any name.
Solution: To fetch the volume chain info, use datastore name instead of UUID.

The fix is made in the flow of following API operations

migrateVirtualMachine
migrateVirtualMachineWithVolume
migrateVolume
2021-06-11 20:06:06 +05:30
Daniel Augusto Veronezi Salvador bf6266188c
Improve logs on RemoteAccessVpnManagerImpl (#4960)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-06-10 16:51:45 +02:00
Harikrishna 083646b3c3
server: VM dynamic scaling option granularity (#4643)
This PR introduces new granularity levels to configure VM dynamic scalability. Previously VM is configured to be dynamically scalable based on the template and global setting. Now we bringing this option to configure at service offering and VM level also.

VM can dynamically scale only when all flags are ON at VM level, template, service offering and global setting. If any of the flags is set to false then VM cannot be scalable. This result will be persisted in DB for each VM and will be honoured for that VM till it is updated.

We are introducing 'dynamicscalingallowed' parameter with permitted values of true or false for deployVM API and createServiceOffering API.

Following are the API parameter changes:
createServiceOffering API: 
          dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.
deployVirtualMachine API: 
          dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.

Following are the UI changes:
Service offering creation has ON/OFF switch for dynamic scaling enabled with default value true
2021-06-08 17:10:00 +05:30
Suresh Kumar Anaparti 958182481e cloudstack: make code more inclusive
Inclusivity changes for CloudStack

- Change default git branch name from 'master' to 'main' (post renaming/changing default git branch to 'main' in git repo)
- Rename some offensive words/terms as appropriate for inclusiveness.

This PR updates the default git branch to 'main', as part of #4887.

Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-08 15:47:20 +05:30
Rohit Yadav d10cdb495f Merge remote-tracking branch 'origin/4.15' 2021-06-07 15:34:52 +05:30
Gabriel Beims Bräscher 3ee563905d
kvm: Check for VLAN or VXLAN in NetworkDaoImpl.listByPhysicalNetworkPvlan (#5074)
This PR fixes #5071; where it was reported an issue when creating a network with VXLAN.
2021-06-05 22:25:01 +05:30
Rohit Yadav d184e282b2 Merge remote-tracking branch 'origin/4.15' 2021-05-31 20:01:09 +05:30
Pearl Dsilva d04fa0201d
server: usage generated for destroyed VMs with no backups (#5017)
Fixes: #4990
When a VM associated with a backup offering is destroyed/expunged, the backup offering isn't unassigned, and despite the VM having no backups present, backup usage is generated. This PR prevent usage record generation when there are no backups present for a VM with a backup offering associated to it. This is done by ensuring that usage event for backups is generated only when a the backup size > 0
2021-05-31 18:59:48 +05:30
Rohit Yadav 57e4e3b6b1 Merge remote-tracking branch 'origin/4.15' 2021-05-31 16:04:42 +05:30
Rohit Yadav fbc8610f6e Merge remote-tracking branch 'origin/4.14' into 4.15 2021-05-31 15:54:56 +05:30
Gabriel Beims Bräscher a78f676037
engine: fix network with SG disabled still has security group script adding rules on KVM (#5049)
This PR fixes #5047 which can be reproduced on Zones with _(I) Advanced Networks, (II) Security Groups enabled for the Zone, (III)  network offering without Security Groups_; for instance, `DefaultSharedNetworkOffering` which does not list Security Group as supported service.

The issue is due to the following code inside the method `VirtualMachineManagerImpl.orchestrateReboot`:
[VirtualMachineManagerImpl.java#L3340](280c13a4bb/engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java (L3340)).

```
  final Answer rebootAnswer = cmds.getAnswer(RebootAnswer.class);
  if (rebootAnswer != null && rebootAnswer.getResult()) {
      if (dc.isSecurityGroupEnabled() && vm.getType() == VirtualMachine.Type.User) {
          List<Long> affectedVms = new ArrayList<Long>();
          affectedVms.add(vm.getId());
          _securityGroupManager.scheduleRulesetUpdateToHosts(affectedVms, true, null);
      }
      return;
  }
```
2021-05-31 15:52:26 +05:30
Gabriel Beims Bräscher a3cdd1f836
Allow deploy Admin VMs and VRs in disabled zones/pods/clusters (#3600) 2021-05-28 10:45:30 +02:00
Daniel Augusto Veronezi Salvador 6531ee5871
Externalize config to set min memory/cpu with division by overprovisi… (#4722)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: dahn <daan.hoogland@gmail.com>
2021-05-28 10:15:51 +02:00
Abhishek Kumar 4b7fa4e630 Merge remote-tracking branch 'apache/4.15'
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-24 16:43:53 +05:30
Pearl Dsilva 2eae0f5385
SystemVM: Set agent state to disconnected on Stopping the systemVM (#5010)
Fixes: #4972
This PR sets systevms' agent state to disconnected when it is stopped. Currently, when a systemVM (Console Proxy VM / Secondary storage VM) is stopped, the agent state still appears to be 'Up'
2021-05-19 13:00:17 +05:30
Abhishek Kumar 426f14b6ed Merge remote-tracking branch 'apache/4.15'
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-18 15:19:20 +05:30
Rohit Yadav 2286c8d2bf Merge remote-tracking branch 'origin/4.14' into 4.15 2021-05-14 23:19:06 +05:30
Abhishek Kumar dc91a1fd4d
server: destroy ssvm, cpvm on last host maintenance (#4644)
* server: destroy ssvm, cpvm on last host maintenance

When a single or last UP host enters into maintenance just stopping SSVM and CPVM will leave behind VMs on hypervisor side. As these system vms will be recreated they can be destroyed.
Fixes #3719

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix methods

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* immediately destroy systemvms

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix destroy

Added bypassHostMaintenance flag in Comma.java class to allow command to be handled by host agent even when host is in maintenace.
Flag is set true only for delete commands for ssvm and cpvm.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unit test fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing return statement

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix

VM should be stopped with cleanup before calling expunge else it server may through error with host in PrepareForMaintenance state.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* rename

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-05-14 23:16:15 +05:30
Wei Zhou e2183ed666
forceha: fix two issues when (1)stop vm from inside (2) force remove host (#4647)
* forceha: fix vm is not started if it is poweroff from inside

steps to reproduce the issue
(1) make sure force.ha is true in global setting. if not, change it to true, and restart mgt server
(2) create a service offering , ha is not enabled
(3) create a vm
(4) log into the vm, and power off via cli.

expected result: vm is started again by cloudstack
actual result: vm is not started.

* forceha: fix vms are still running if host is force-removed

when host can be force removed, however vms are stopped in cloudstack, but not stopped on host
```
(localcloud) 🐱 > delete host id="a5625393-444d-4d0a-b31d-62baf88a8be1" forced=true
{
  "success": true
}```

after some minutes, vms are still runnning on host
```
root@mgt01:~# ssh node63 virsh list
 Id   Name        State
---------------------------
 1    i-2-19-VM   running
 2    i-2-11-VM   running
```

error message are
```
Cannot transmit host 2 to Enabled state
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state = Enabled event = DeleteHost
        at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1216)
        at com.cloud.resource.ResourceManagerImpl$1.doInTransactionWithoutResult(ResourceManagerImpl.java:907)
```

* forceha: Make ForceHA dynamic
2021-05-14 23:14:39 +05:30
Wei Zhou 1b28ea1ebb
network: fix dhcp/password/metadata issues on shared networks with multiple subnets (#5013)
* #4943: apply iptables for password and metadata

* #4943: fix wrong ip alias

* #4943: revert previous change and add ip_aliases

Co-authored-by: Wei Zhou <weizhouapache@gmail.com>
2021-05-13 14:31:47 +05:30
Daniel Augusto Veronezi Salvador 3674e80219
Improve logs on SecondaryStorageManagerImpl and few refactors (#4955)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-05-10 09:47:42 +02:00
Rohit Yadav cb167072a1 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-05-07 16:37:42 +05:30
Harikrishna 32e3bbdcc5
VMware Datastore Cluster primary storage pool synchronisation (#4871)
Datastore cluster as a primary storage support is already there. But if any changes at vCenter to datastore cluster like addition/removal of datastore is not synchronised with CloudStack directly. It needs removal of primary storage from CloudStack and add it again to CloudStack.

Here synchronisation of datastore cluster is fixed without need to remove or add the datastore cluster.
1. A new API is introduced syncStoragePool which takes datastore cluster storage pool UUID as the parameter. This API checks if there any changes in the datastore cluster and updates management server accordingly.
2. During synchronisation if a new child datastore is found in datastore cluster, then management server will create a new child storage pool in database under the datastore cluster. If the new child storage pool is already added as an individual storage pool then the existing storage pool entry will be converted to child storage pool (instead of creating a new storage pool entry)
3. During synchronisaton if the existing child datastore in CloudStack is found to be removed on vCenter then management server removes that child datastore from datastore cluster and makes it an individual storage pool.
The above behaviour is on par with the vCenter behaviour when adding and removing child datastore.
2021-05-07 16:30:54 +05:30
Rohit Yadav fce84e62ab Merge remote-tracking branch 'origin/4.15' 2021-05-06 00:27:43 +05:30
Pearl Dsilva bc80815cf5
server: Adding VPN options for IKE version and IKE split connections (#4953)
IKE version allows selecting ike (autoselect), ikev1, or ikev2.
Split connections gives an option of separating the first right subnet from the rest, and kicking out individual statements for each right subnet for better cross-compatibility.

Backported from PR: #4137
update per PR suggestion

Fixes #3138

Co-authored-by: Greg Goodrich <ggoodrich@ippathways.com>
Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2021-05-05 12:54:23 +05:30
Rohit Yadav 1abd10199c Merge remote-tracking branch 'origin/4.15' 2021-05-04 19:37:45 +05:30
Gabriel Beims Bräscher ab790c11d5
server: Allow to upgrade service offerings from local <> shared storage pools (#4915)
This PR addresses the issue raised at #4545 (Fail to change Service offering from local <> shared storage).

When upgrading a VM service offering it is validated if the new offering has the same storage scope (local or shared) as the current offering. I think that the validation makes sense in a way of preventing running Root disks with an offering that does not match the current storage pool. However, the validation only compares both offerings and does not consider that it is possible to migrate Volumes between local <> shared storage pools.

The idea behind this implementation is that CloudStack should check the scope of the current storage pool which the ROOT volume is allocated; this, it is possible to migrate the volume between storage pools and list/upgrade according to the offerings that are supported for such pool.

This PR also fixes an issue where the API command that lists offerings for a VM should follow the same idea and list based on the storage pool that the volume is allocated and not the previous offering.

Fixes: #4545
2021-04-30 11:59:50 +05:30
Olivier Lemasle 72f6612971
server: Increase max length for VMInstanceVO.backupVolumes (#4967)
The default length is 255, which caused a truncation of data if
the JSON object representing the backup volumes is too big.
It caused errors when backups were made on VMs with 3 volumes
or more.

`vm_instance.backup_volumes` has the type TEXT, which has a
maximal length of 65535 characters.

Fixes #4965
2021-04-30 11:57:56 +05:30
Rohit Yadav 4742ac15f7 Merge remote-tracking branch 'origin/4.15' 2021-04-29 21:50:40 +05:30
dahn be255e4203
server: protect against stray snapshot-details without snapshot (#4924)
This PR makes sure no orphaned snapshot details are considered in the cleanup at startup job.
a real solution would be to implement some kind of cascading delete, but as the parent record is "only" marked as removed this would be a bit com

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-04-29 20:40:29 +05:30
Abhishek Kumar 42c83b08f5 Merge remote-tracking branch 'apache/4.15' 2021-04-26 14:33:58 +05:30
Nicolas Vazquez f728287aa2
server: Fix template garbage collection cleanup (#4944) 2021-04-24 18:57:47 +05:30
Abhishek Kumar a30d518e8a
vmware: fix stopped VM volume migration (#4758)
* prevent other vm disks getting deleted

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* vmware: fix inter-cluster stopped vm migration

Fixes #4838

For inter-cluster migration without shared storage, VMware needs a host to be specified. Fix is to specify an appropriate host in the target cluster.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix detached volume inter-cluster migration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* cleanup unused method

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* review changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* vmware: allow attached volume migration using VmwareStorageMotionStrategy

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* find vm clusterid with multiple ROOT volumes

VM can have multiple ROOT volumes and some can be on zone-wide store therefore iterate over all of them till a cluster ID is found.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix successive storage migration

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix intercluster check

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor vm cluster, host method

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove inter-pod check

Added by mistake, VMware won't have pods

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address review comment

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-04-24 18:55:25 +05:30