Commit Graph

554 Commits

Author SHA1 Message Date
Marcus Sorensen 2316e800ba
Apple base416 passphrase enc (#240)
* Move PassphraseVO to use String instead of byte[] to support Encrypt annotation

* Check for unencrypted passphrases before migrating passphrase table

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-03-03 08:40:46 -07:00
Wei Zhou 67b60385af
Apple FR68: New database encryption cipher (#214)
* Rough start swapping DB Encryption, add CLI PoC

* Enhance EncryptionCLI to have command line parsing

* Refactor new encryption behind AeadBase64Encryptor for every use

* Add comment about encryption passwords

* EncryptionSecretKeyChanger - use reflection to find all encrypted tables

Over the years this hasn't been updated properly. Use reflection to find
the tables with encrypted fields. This will also ensure any plugins in
the classpath that add tables will get their encrypted fields updated as well.

Table vpn_users has encrypted columns [password]
Table sslcerts has encrypted columns [password, key]
Table user_view has encrypted columns [secret_key]
Table account_details has encrypted columns [value]
Table domain_details has encrypted columns [value]
Table s2s_customer_gateway has encrypted columns [ipsec_psk]
Table ucs_manager has encrypted columns [password]
Table vm_instance has encrypted columns [vnc_password]
Table passphrase has encrypted columns [passphrase]
Table keystore has encrypted columns [key]
Table external_stratosphere_ssp_credentials has encrypted columns [password]
Table storage_pool has encrypted columns [user_info]
Table remote_access_vpn has encrypted columns [ipsec_psk]
Table user has encrypted columns [secret_key]
Table oobm has encrypted columns [password]

* Apple FR68: add new class CloudStackEncryptor

* Apple FR68: add interface com.cloud.utils.crypt.Encryptor

* Apple FR68: update com.cloud.utils.EncryptionUtil

* Apple FR68: add cloudstack-utils.jar to cloudstack-common package

* Apple FR68: use cloudstack-utils.jar in scripts

* Apple FR68: revert replace.properties to original version

* Apple FR68: update EncryptionSecretKeyChanger

* Apple FR68: Add EncryptorVersion to CloudStackEncryptor

* Apple FR68: Update com.cloud.utils.crypt.EncryptionCLI

* Apple FR68: Remove check on EncryptionSecretKeyChecker.useEncryption in CloudStackEncryptor

* Apple FR68: update EncryptionSecretKeyChanger part2

* Apple FR68: update EncryptionSecretKeyChanger part3 (force update)

* Apple FR68: move cloud-migrate-databases.in to deprecated and recreate it with java command

* Apple FR68: update EncryptionSecretKeyChanger part4 (add skip-database-migration)

* Apple FR68: set encryptor in first encryption in CloudStackEncryptor

* Apple FR68: save db.cloud.encryptor.version in db.properties

* Apple FR68: update EncryptionSecretKeyChanger part4 (clear db.cloud.encryptor.version)

* Apple FR68: load and save db.cloud.encryptor.version in db.properties

* Apple FR68: Add caller class name in debug messages

* Apple FR68: consider non-exist tables and columns

* Apple FR68: skip tables if no data exists

* Apple FR68: remove GeneralSecurityException from code

* Apple FR68: hide value with Asterisks in CloudStackEncryptor

* Apple FR68: log an error message when fail to load 'init'

* Apple FR68: remove setup/bindir/cloud-migrate-databases.deprecated.in which I think it is not needed

* Apple FR68: add new encryptor version to EncryptionSecretKeyChanger

* Apple FR68: use System.exit(1) in EncryptionSecretKeyChanger

* Apple FR68: check arguments in cloudstack-migrate-databases

* Apple FR68: remove all org.jasypt.* in code

* Apple FR68: initilize database encryptors by getting 'init'

* Apple FR68: migrate server.properties

* Apple FR68: load new management key from environment variable CLOUD_SECRET_KEY_NEW

* Apple FR68: fix unable to load 'init' in fresh installation

* Apple FR68: fix 'Rolling back the transaction' in txn.close

* Apple FR68: improve logging in cloudstack-migrate-databases

* Apple FR68: hide value with Asterisks in other encryptors

* Apple FR68: System.exit(1) if fail to migrate server.properties

* Apple FR68: migrate values from cluster_details,user_vm_details,etc

* Apple FR68: refactor EncryptionSecretKeyChanger

* Apple FR68: update user_vm_deploy_as_is_details values

* Apple FR68: update image_store.url (if protocol is cifs) and storage_pool.path (if pool_type is SMB)

* Apple FR68: minor improvement EncryptionSecretKeyChanger

* Apple FR68: add unit test EncryptionSecretKeyChangerTest

* Apple FR68: support encryption type 'env' in cloudstack-setup-databases to get env "CLOUD_SECRET_KEY" before passed value

* Apple FR68: rename Encryptor to Base64Encryptor

* Apple FR68: Backport community PR 6542

* Apple FR68: code optimization

* Apple FR68: use Options and StringUtils

* Apple FR68: add license headers

* Apple FR68: refactor CloudStackEncryptor as per Daan's review

* Apple FR68: refactor DatabaseUpgradeChecker as per Daan's review

* Apple FR68: show error message in usage.log if fail to get encrypted configurations

* Apple FR68: load new MS key from env before migration

* Apple FR68: return 1 if fail to parse arguments of EncryptionCLI

* Apple FR68: fix code smells

* Apple FR68: fix code smells (part2)

* Apple FR68: revert FOOTER of cloudstack-migrate-databases to use \n

* Apple FR68: update help message of cloudstack-setup-databases

* Apple FR68: fix code smells (part3)

* Apple FR68: make changes as per suggestions

* Apple FR68: migrate database if new encryptor version is set to different

Testing result: (assume db.cloud.encryptor.version=V1)

(1) migrate only db.properties (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is not migrated

(2) migrate only db.properties (same db key, new db encryptorversion)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V2
         cloudstack database is not migrated (mostly on secondary management servers)

(3) migrate only db.properties (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is not migrated

(4) migrate only db.properties (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is not migrated (mostly on secondary management servers)

(5) migrate only db.properties (different db key, new db version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2 --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V2
         cloudstack database is not migrated (mostly on secondary management servers)

(6) migrate only db.properties (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey --skip-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is not migrated (mostly on secondary management servers)

(7) migrate db.properties and database (same db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V1 --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is migrated using encryptor V1

(8) migrate db.properties and database (same db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V2
         cloudstack database is migrated using encryptor V2

(9) migrate db.properties and database (same db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey --force-database-migration
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is migrated using encryptor V1

(10) migrate db.properties and database (different db key, same db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V1
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is migrated using encryptor V1

(11) migrate db.properties and database (different db key, new db encryptor version)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey -v V2
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V2
         cloudstack database is migrated using encryptor V2

(12) migrate db.properties and database (different db key, db encryptor version is not set)
Command: /usr/bin/cloudstack-migrate-databases -m mgmtkey -d dbkey -n newmgmtkey -e newdbkey
Changes: db.cloud.encrypt.secret is encrypted by V2 (always)
         db.cloud.encryptor.version=V1
         cloudstack database is migrated using encryptor V1

* smoke test: fix test_primary_storage.py

* smoke test: Do NOT run tests in test_primary_storage.py in parallel

This also fixes an issue in detachvolume

'Failed to detach volume Test Volume-yyyyyy from VM VM-zzzzzz; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment'

* Update PR7003: rename method

---------

Co-authored-by: Marcus Sorensen <mls@apple.com>
2023-01-31 13:18:28 +01:00
Marcus Sorensen 1f0a85aed8
Refactor few process of VirtualMachineManagerImpl and improve logs (#4966) (#211)
* Improve logs

* Remove unnecessary comments

* Use diamond inference

* Fix some logs

* Remove unnecessary unboxing

* Create method to handle job result

* Remove unused vars and fix some logics

* Extract code to method and few adjusts

* Use CollectionUtils

* Extract pending work job validation to method

* Create new constructors

* Extract work job and info creation to a method

* Extract submit async job to a method

* Extract find vm by id to a method

* Change log level from trace to debug

* Remove unnused methods and add logs

* Undo code remotion

* Remove asserts and fix conditionals

* Address @GabrielBrascher reviews

* Remove double quotes from keys in manual json

* Undo code remotion

* Add object to log

* Remove statement from try/catch

* Implement toString with ReflectionToStringBuilderUtils

* Fix errors related to merge main

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>

Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2022-11-18 20:37:11 +05:30
Nicolas Vazquez 00d8a13a52
Apple FR64: Standardise API driven console access (#188)
* Console access enhancements

* Remove extra logging

* Fix security hotspot

* Fix sonar cloud code smells

* Refactor API response

* Minor fix

* Refactor and increase timeout on ssh to cpvm

* Add marvin tests and extend permissions

* Fix account type

* Add unit tests

* Check vncport file exits on CPVM before attempting to add rules

* Change how vncport is read on cpvm

* Extra validation refactor

* Fix wrong token API param on UI

* Refactor vnc port selection to 8080 or 8443

* Do not display the input token modal and improve error message on console

* Improve error message and prevent opening blank popup when errors

* Fix logging exception due to algorithm
2022-09-07 17:47:13 +05:30
Abhishek Kumar 92931aeeb8
schema,server,api: events improvement (#127)
Add resource ID and resource type to event.
In UI, adds Events tab in resource view for the supporting resources.

Following SQL changes needed to support events with resource details in DB,
```
 -- Alter event table to add resource_id and resource_type
ALTER TABLE `cloud`.`event`
    ADD COLUMN `resource_id` bigint unsigned COMMENT 'ID of the resource associated with the even' AFTER `domain_id`,
    ADD COLUMN `resource_type` varchar(32) COMMENT 'Account role in the project (Owner or Regular)' AFTER `resource_id`;

DROP VIEW IF EXISTS `cloud`.`event_view`;
CREATE VIEW `cloud`.`event_view` AS
    SELECT
        event.id,
        event.uuid,
        event.type,
        event.state,
        event.description,
        event.resource_id,
        event.resource_type,
        event.created,
        event.level,
        event.parameters,
        event.start_id,
        eve.uuid start_uuid,
        event.user_id,
        event.archived,
        event.display,
        user.username user_name,
        account.id account_id,
        account.uuid account_uuid,
        account.account_name account_name,
        account.type account_type,
        domain.id domain_id,
        domain.uuid domain_uuid,
        domain.name domain_name,
        domain.path domain_path,
        projects.id project_id,
        projects.uuid project_uuid,
        projects.name project_name
    FROM
        `cloud`.`event`
            INNER JOIN
        `cloud`.`account` ON event.account_id = account.id
            INNER JOIN
        `cloud`.`domain` ON event.domain_id = domain.id
            INNER JOIN
        `cloud`.`user` ON event.user_id = user.id
            LEFT JOIN
        `cloud`.`projects` ON projects.project_account_id = event.account_id
            LEFT JOIN
        `cloud`.`event` eve ON event.start_id = eve.id;
```
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2022-05-05 13:44:33 +05:30
Nicolas Vazquez d6b9c65ce6
Backport: Direct download certificates additions and improvements (#6104) (#140)
* Direct download certificates additions and improvements (#6104)

* Add direct download certificates listing

* Restore class to original project

* Small refactor

* Register API

* Apply suggestions from code review

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Refactor after review

* Fix checkstyle

* Add hosts mapping to API response

* Improvements on revoke certificate

* Refactor revoke certificate API

* Fix condition

* Filter only certificates not revoked for revokeCertificate API

* Improve upload certificate and add provision certificate API

* Improve certificate response output

* Address review comments

* Refactor revoke cert test

* Fix marvin test

* Address review comments

* Fix issues

* Improvements

* Refactor upload template API response

* Fix response

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>

* Fix pom version

Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
2022-04-12 10:49:08 -03:00
Suresh Kumar Anaparti cad9332082
Updating pom.xml version numbers for release 4.16.1.0
Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-02-25 19:01:16 +05:30
Pearl Dsilva 85ced4447b
API: Fix listSSHKeyPairs API when listing all resources (listall=true & projectid=-1) (#5958)
* API: Fix listSSHKeyPairs API when listing all resources (listall=true & projectid=-1)

* fix issue with indexing
2022-02-09 15:25:27 +01:00
Rohit Yadav da56a2a806
maven: migrate short-term to reload4j v1.2.18 (#5878)
* maven: migrate short-term to reload4j v1.2.18

This migrate to log4j 1.x fork, reload4j 1.2.18.0 which is drop-in
replacement and addresses some immediate CVE and issues.

* log4j migration to reload4j in pom xmls

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Exclude log4j from transitive dependencies (#73)

Co-authored-by: Marcus Sorensen <shadowsor@gmail.com>
Co-authored-by: Marcus Sorensen <mls@apple.com>
2022-02-08 15:00:38 +05:30
jairov4 6c9f0c1f3f
Fix #3448 quota calculation for monthly tariffs (#5517)
* Fix quota calculation for monthly tariffs

* Remove unused parameter
2022-02-07 13:34:37 +01:00
dahn 028d338aaa
remove VmWorkJob after adding a nic to a vm (#5658)
Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Co-authored-by: Wei Zhou <weizhou@apache.org>
2022-01-10 16:31:50 +01:00
nicolas 93c3c3b9ac
Updating pom.xml version numbers for release 4.16.1.0-SNAPSHOT
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-09 22:50:22 -03:00
nicolas 44c08b5acc
Updating pom.xml version numbers for release 4.16.0.0
Signed-off-by: nicolas <nicovazquez90@gmail.com>
2021-11-04 14:14:57 -03:00
Nicolas Vazquez 261fe326ff
Revert "parallel nic adding (#5541)" (#5665)
This reverts commit 3574d8d20b.
2021-11-04 13:29:59 -03:00
dahn 3574d8d20b
parallel nic adding (#5541)
* trace nics additions

* work queue patch for network to add

* add secondary key to job

* logging improvements and naming of field(s)

* several naming corrections

* extra check if net already exists for vm

* placeholder job with secondary object

* constraint on entering the same job multiple times

* error handling/warning message

* review comments applied

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Wei Zhou <wei.zhou@shapeblue.com>
2021-10-04 12:21:21 -03:00
Rakesh dcc02e0fbb
Enable account settings to be visible under domain settings (#4215)
* Enable account settings to be visible under domain settings

All the account settings can't be configured under domain
level settings right now.
By default, if account setting is not configured then
its value will be taken from global setting.
Add a global setting "enable.account.settings.for.domain"
so that if its enabled then all the account level settings
will be visible under domain levelsettings also.
If account level setting is configured then that value will
be considered else it will take domain scope value. If
domain scope value is not configured then it will pick
it up from global setting.

If domain level setting is not configured then by default
the value will be taken from global setting
Add another global setting "enable.domain.settings.for.child.domain"
so that when its true, if a value for domain setting is not
configured then its parent domain value is considered until
it reaches ROOT domain. If no value is configured till ROOT
domain then global setting value will be taken.

Also display all the settings configured under the domain level
in list domains api response

* rename variables

Co-authored-by: Rakesh Venkatesh <rakeshv@apache.org>
2021-09-29 10:29:20 +02:00
wx930910 7d5393d577
Refactor GroupByExtension to improve test logic (#5480) 2021-09-22 12:02:10 -03:00
Daniel Augusto Veronezi Salvador e962f0f271
Unify SMTP mail sending (#4954)
* Add mail dependencies

* Create util to send SMTP mail

* Add unit tests to SMTP mail sender

* Use SMTP mail util on quota alert

* Use SMTP mail util on alert

* Use SMTP mail util on project

* Use SMTP mail util on usage alert

* Remove copyright line in license header

Co-authored-by: Gabriel Beims Bräscher <gabrascher@gmail.com>

* Remove copyright line in license header

Co-authored-by: Gabriel Beims Bräscher <gabrascher@gmail.com>

* Remove copyright line in license header

Co-authored-by: Gabriel Beims Bräscher <gabrascher@gmail.com>

* Remove copyright line in license header

Co-authored-by: Gabriel Beims Bräscher <gabrascher@gmail.com>

* Remove copyright line in license header

Co-authored-by: Gabriel Beims Bräscher <gabrascher@gmail.com>

Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
Co-authored-by: Gabriel Beims Bräscher <gabrascher@gmail.com>
2021-06-15 23:18:26 +02:00
Harikrishna 083646b3c3
server: VM dynamic scaling option granularity (#4643)
This PR introduces new granularity levels to configure VM dynamic scalability. Previously VM is configured to be dynamically scalable based on the template and global setting. Now we bringing this option to configure at service offering and VM level also.

VM can dynamically scale only when all flags are ON at VM level, template, service offering and global setting. If any of the flags is set to false then VM cannot be scalable. This result will be persisted in DB for each VM and will be honoured for that VM till it is updated.

We are introducing 'dynamicscalingallowed' parameter with permitted values of true or false for deployVM API and createServiceOffering API.

Following are the API parameter changes:
createServiceOffering API: 
          dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.
deployVirtualMachine API: 
          dynamicscalingenabled: an optional parameter of type Boolean with default value “true”.

Following are the UI changes:
Service offering creation has ON/OFF switch for dynamic scaling enabled with default value true
2021-06-08 17:10:00 +05:30
Suresh Kumar Anaparti 958182481e cloudstack: make code more inclusive
Inclusivity changes for CloudStack

- Change default git branch name from 'master' to 'main' (post renaming/changing default git branch to 'main' in git repo)
- Rename some offensive words/terms as appropriate for inclusiveness.

This PR updates the default git branch to 'main', as part of #4887.

Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-06-08 15:47:20 +05:30
Rohit Yadav 4742ac15f7 Merge remote-tracking branch 'origin/4.15' 2021-04-29 21:50:40 +05:30
dahn be255e4203
server: protect against stray snapshot-details without snapshot (#4924)
This PR makes sure no orphaned snapshot details are considered in the cleanup at startup job.
a real solution would be to implement some kind of cascading delete, but as the parent record is "only" marked as removed this would be a bit com

Co-authored-by: Daan Hoogland <dahn@onecht.net>
2021-04-29 20:40:29 +05:30
Daniel Augusto Veronezi Salvador b28d638ade
Add support to StartTLS on Quota's mailing (#4573)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-04-13 09:55:02 +02:00
Daniel Augusto Veronezi Salvador 31771186fa
Externalize TLS version on Quota's mailing (#4583)
Co-authored-by: Daniel Augusto Veronezi Salvador <daniel@scclouds.com.br>
2021-02-24 11:15:12 +00:00
sureshanaparti eba186aa40
storage: New Dell EMC PowerFlex Plugin (formerly ScaleIO, VxFlexOS) (#4304)
Added support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack (for KVM hypervisor) and enabled VM/Volume operations on that pool (using pool tag).
Please find more details in the FS here:
https://cwiki.apache.org/confluence/x/cDl4CQ

Documentation PR: apache/cloudstack-documentation#169

This enables support for PowerFlex/ScaleIO (v3.5 onwards) storage pool as a primary storage in CloudStack

Other improvements addressed in addition to PowerFlex/ScaleIO support:

- Added support for config drives in host cache for KVM
	=> Changed configuration "vm.configdrive.primarypool.enabled" scope from Global to Zone level
	=> Introduced new zone level configuration "vm.configdrive.force.host.cache.use" (default: false) to force host cache for config drives
	=> Introduced new zone level configuration "vm.configdrive.use.host.cache.on.unsupported.pool" (default: true) to use host cache for config drives when storage pool doesn't support config drive
	=> Added new parameter "host.cache.location" (default: /var/cache/cloud) in KVM agent.properties for specifying the host cache path and create config drives on the "/config" directory on the host cache path
	=> Maintain the config drive location and use it when required on any config drive operation (migrate, delete)

- Detect virtual size from the template URL while registering direct download qcow2 (of KVM hypervisor) templates

- Updated full deployment destination for preparing the network(s) on VM start

- Propagate the direct download certificates uploaded to the newly added KVM hosts

- Discover the template size for direct download templates using any available host from the zones specified on template registration
	=> When zones are not specified while registering template, template size discovery is performed using any available host, which is picked up randomly from one of the available zones

- Release the VM resources when VM is sync-ed to Stopped state on PowerReportMissing (after graceful period)

- Retry VM deployment/start when the host cannot grant access to volume/template

- Mark never-used or downloaded templates as Destroyed on deletion, without sending any DeleteCommand
	=> Do not trigger any DeleteCommand for never-used or downloaded templates as these doesn't exist and cannot be deleted from the datastore

- Check the router filesystem is writable or not, before performing health checks
	=> Introduce a new test "filesystem.writable.test" to check the filesystem is writable or not
	=> The router health checks keeps the config info at "/var/cache/cloud" and updates the monitor results at "/root" for health checks, both are different partitions. So, test at both the locations.
	=> Added new script: "filesystem_writable_check.py" at /opt/cloud/bin/ to check the filesystem is writable or not

- Fixed NPE issue, template is null for DATA disks. Copy template to target storage for ROOT disk (with template id), skip DATA disk(s)

* Addressed some issues for few operations on PowerFlex storage pool.

- Updated migration volume operation to sync the status and wait for migration to complete.

- Updated VM Snapshot naming, for uniqueness in ScaleIO volume name when more than one volume exists in the VM.

- Added sync lock while spooling managed storage template before volume creation from the template (non-direct download).

- Updated resize volume error message string.

- Blocked the below operations on PowerFlex storage pool:
  -> Extract Volume
  -> Create Snapshot for VMSnapshot

* Added the PowerFlex/ScaleIO client connection pool to manage the ScaleIO gateway clients, which uses a single gateway client per Powerflex/ScaleIO storage pool and renews it when the session token expires.

- The token is valid for 8 hours from the time it was created, unless there has been no activity for 10 minutes.
  Reference: https://cpsdocs.dellemc.com/bundle/PF_REST_API_RG/page/GUID-92430F19-9F44-42B6-B898-87D5307AE59B.html

Other fixes included:

- Fail the VM deployment when the host specified in the deployVirtualMachine cmd is not in the right state (i.e. either Resource State is not Enabled or Status is not Up)

- Use the physical file size of the template to check the free space availability on the host, while downloading the direct download templates.

- Perform basic tests (for connectivity and file system) on router before updating the health check config data
	=> Validate the basic tests (connectivity and file system check) on router
	=> Cleanup the health check results when router is destroyed

* Updated PowerFlex/ScaleIO storage plugin version to 4.16.0.0

* UI Changes to support storage plugin for PowerFlex/ScaleIO storage pool.
- PowerFlex pool URL generated from the UI inputs(Gateway, Username, Password, Storage Pool) when adding "PowerFlex" Primary Storage
- Updated protocol to "custom" for PowerFlex provider
- Allow VM Snapshot for stopped VM on KVM hypervisor and PowerFlex/ScaleIO storage pool

and Minor improvements in PowerFlex/ScaleIO storage plugin code

* Added support for PowerFlex/ScaleIO volume migration across different PowerFlex storage instances.

- findStoragePoolsForMigration API returns PowerFlex pool(s) of different instance as suitable pool(s), for volume(s) on PowerFlex storage pool.
- Volume(s) with snapshots are not allowed to migrate to different PowerFlex instance.
- Volume(s) of running VM are not allowed to migrate to other PowerFlex storage pools.
- Volume migration from PowerFlex pool to Non-PowerFlex pool, and vice versa are not supported.

* Fixed change service offering smoke tests in test_service_offerings.py, test_vm_snapshots.py

* Added the PowerFlex/ScaleIO volume/snapshot name to the paths of respective CloudStack resources (Templates, Volumes, Snapshots and VM Snapshots)

* Added new response parameter “supportsStorageSnapshot” (true/false) to volume response, and Updated UI to hide the async backup option while taking snapshot for volume(s) with storage snapshot support.

* Fix to remove the duplicate zone wide pools listed while finding storage pools for migration

* Updated PowerFlex/ScaleIO volume migration checks and rollback migration on failure

* Fixed the PowerFlex/ScaleIO volume name inconsistency issue in the volume path after migration, due to rename failure
2021-02-24 14:58:33 +05:30
Rohit Yadav b482da8c91 Updating pom.xml version numbers for release 4.15.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-11 13:58:30 +05:30
Daan Hoogland 280c13a4bb Updating pom.xml version numbers for release 4.15.0.0
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2021-01-05 15:51:02 +00:00
Daan Hoogland 81e9e6809b Updating pom.xml version numbers for release 4.15.1.0-SNAPSHOT
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2021-01-04 11:34:46 +00:00
Daan Hoogland e26202f23e Updating pom.xml version numbers for release 4.16.0.0-SNAPSHOT
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2021-01-04 11:32:10 +00:00
Daan Hoogland 01b3e361c7 Updating pom.xml version numbers for release 4.15.0.0
Signed-off-by: Daan Hoogland <dahn@onecht.net>
2020-12-23 16:32:25 +00:00
lujiefsi 3689b9da41
jobs: The patch remove the password from resultObject and make it be humanreadable (#4538)
This PR fixes CLOUDSTACK-10425.

Co-authored-by: lujie <lujie@foxmail.com>
2020-12-14 11:39:54 +05:30
Gabriel Beims Bräscher 1ba6a49fa4
Fix JsonSyntaxException when creating API command response #4355 (#4387)
* Add missing \" on obfuscatePassword

* Add tests for AsyncJobManagerImpl.obfuscatePassword

* fix for "password},"
2020-10-20 16:49:55 +05:30
davidjumani aab8df09aa
Fixing searchAndCount searchAndDistinctCount when sc is null (#4374) 2020-10-13 11:30:04 +02:00
Rohit Yadav 86fcb14238 Merge remote-tracking branch 'origin/4.14'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-24 12:20:29 +05:30
Wei Zhou 534dd475d8
agent: Compare indirect agent lb algorithm when cloudstack agent connects (#4335)
Compare not only the list of management servers but also the lb algorithm when agent connects.

Fixes: #3895
2020-09-23 12:48:16 +05:30
Pearl Dsilva b464fe41c6
server: Secondary Storage Usage Improvements (#4053)
This feature enables the following:
Balanced migration of data objects from source Image store to destination Image store(s)
Complete migration of data
setting an image store to read-only
viewing download progress of templates across all data stores
Related Primate PR: apache/cloudstack-primate#326
2020-09-17 10:12:10 +05:30
Gabriel Beims Bräscher 3fe724bd32
db: Prevent NullPointerException on GenericDaoBase (#4268)
* Prevent NullPointerException on GenericDaoBase

* Fix checkstyle: remove unused import
2020-08-17 16:13:46 +05:30
Spaceman1984 b586eb22f1
Human readable sizes in logs (#4207)
This PR adds outputting human readable byte sizes in the management server logs, agent logs, and usage records. A non-dynamic global variable is added (display.human.readable.sizes) to control switching this feature on and off. This setting is sent to the agent on connection and is only read from the database when the management server is started up. The setting is kept in memory by the use of a static field on the NumbersUtil class and is available throughout the codebase.

Instead of seeing things like:
2020-07-23 15:31:58,593 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 8-1863645820801253428: Processing: { Ans: , MgmtId: 52238089807, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-224-VM","bytesSent":"106496","bytesReceived":"0","result":"true","details":"","wait":"0",}}] }

The KB MB and GB values will be printed out:

2020-07-23 15:31:58,593 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 8-1863645820801253428: Processing: { Ans: , MgmtId: 52238089807, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-224-VM","bytesSent":"(104.00 KB) 106496","bytesReceived":"(0 bytes) 0","result":"true","details":"","wait":"0",}}] }

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Human+Readable+Byte+sizes
2020-08-13 15:55:16 +05:30
Pearl Dsilva c578004fe5
projects: Role based users in Projects (#4128)
Enabling Role Based users in projects
Primate PR related to the FR: apache/cloudstack-primate#382
Doc PR: https://github.com/apache/cloudstack-documentation/pull/145

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2020-08-13 15:45:39 +05:30
Rohit Yadav 2c82aac5aa Merge remote-tracking branch 'origin/4.14' 2020-07-07 12:53:05 +05:30
davidjumani b64d0b2d92
quota: Adding pagination for quotaSummary and quotaTariffList (#4186)
Adds pagination and keyword search support to quotaSummary and quotaTariffList
Fixes: #4181
2020-07-06 14:54:04 +05:30
harikrishna-patnala a279d5c453
logging: Logging framework to use only log4j (#4003)
Currently CloudStack is using logging frameworks as log4j and Java util logging, logging wrappers as slf4j and Apache common logging.
Here changes are to made it uniform, using only log4j framework.
Removed Java util logging, slf4j and Apache common logging.
2020-06-17 07:11:23 +05:30
Rohit Yadav 567524a2a5 Merge remote-tracking branch 'origin/4.14' 2020-06-16 13:31:29 +05:30
Rohit Yadav b54d19b3b9
server: fix TransactionLegacy DB connection leaks due to DB switching by B&R thread (#4121)
BackupSync task would switch between databases to update backup usage
metrics in the cloud_usage.usage_backup table. The current framework
and the usage in ManagedContext causes database connection
(LegacyTransaction) leaks. When the thread runs faster, the issue is
easily reproducible and checking via heap dump analysis or using JMX
MBeans. This fixes by moving the task of backup data updation for
usage data to the usage server by publishing usage events instead of
switching between databases in a local thread while in a
ManagedContextRunnable.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-06-16 13:30:31 +05:30
andrijapanicsb 5f926c3353 Updating pom.xml version numbers for release 4.15.0.0-SNAPSHOT
Signed-off-by: andrijapanicsb <andrija.panic@shapeblue.com>
2020-05-23 10:18:39 +01:00
andrijapanicsb 05e9b11694 Updating pom.xml version numbers for release 4.14.1.0-SNAPSHOT
Signed-off-by: andrijapanicsb <andrija.panic@shapeblue.com>
2020-05-23 09:59:32 +01:00
andrijapanicsb 6f96b3b2b3 Updating pom.xml version numbers for release 4.14.0.0
Signed-off-by: andrijapanicsb <andrija.panic@shapeblue.com>
2020-05-11 15:03:14 +01:00
Daan Hoogland 8e4be6dc60 Merge branch '4.13' 2020-04-16 15:27:52 +02:00
dahn 1d34eed43c Cs 1268 gs
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-04-16 15:13:06 +02:00
Rohit Yadav 318924d801
CloudStack Backup & Recovery Framework (#3553) 2020-03-03 13:27:58 +01:00