Cloudstack Source code
Go to file
Marcus Sorensen ba7adfa6f0
Volume encryption (#135)
This PR introduces volume encryption option to service offerings and disk offerings. Fixes #136

There is a hypervisor component and a storage pool component. Hypervisors are responsible for being capable of running/using the encrypted volumes. Storage pools are responsible for being able to create, copy, resize, etc. Hypervisors will report encryption support in their details, storage pools are marked for encryption support by pool type.

The initial offering for experimental release of this feature will have support for encryption on Local, NFS, SharedMountPoint, and ScaleIO storage types.

When volumes choosing an encrypted offering are allocated to a pool, the pool type must be capable of supporting encryption and this is enforced.

When VMs are started and they have an encrypted volume, the hypervisor must be capable of supporting encryption. Also, if volumes are attached to running VMs, the attach will only work if the hypervisor supports encryption.

This change includes a few other minor changes - for example the ability to force the KVM hypervisor private IP. This was necessary in my testing of ScaleIO, where the KVM hypervisors had multiple IPs and the ScaleIO storage only functions if the hypervisor as a ScaleIO client matches IPs with what CloudStack sees as the hypervisor IP.

For experimental release of this feature, some volume workflows like extract volume and migrate volume aren't supported for encrypted volumes. In the future we could support these, as well as migrating from unencrypted to encrypted offerings, and vice versa.

It may also be possible to configure encryption specifics in the future, perhaps at the pool level or the offering level. Currently, there is only one workable encryption offering for KVM that is supported by Libvirt and Qemu for raw and qcow2 disk files, LUKS version 1. This PR ensures we at least store this encryption format associated with each volume, with the expectation that later we may have LUKS v2 volumes or something else. Thus we will have the information necessary to use each volume with Libvirt if/when other formats are introduced.

I think the most disruptive change here is probably a refactoring of the QemuImg utility to support newer flags like --object. I've tested the change against the basic Qemu 1.5.3 that comes with EL7 and I believe it is good, but it will be nice to see the results of some functional tests. Most of the other changes are limited to changing behavior only if volume encryption is requested.

Working on documentation for the CloudStack docs. One thing to note is that hypervisors that run the stock EL7 version of Qemu will not support encryption. This is tested to be detected and report properly via the CloudStack API/UI. I intend to like to have a support matrix in the CloudStack docs.

I may add a few more unit tests. I'd also like some guidance on having functional tests. I'm not sure if there's a separate framework, or if Marvin is still used, or what the current thing is.

* Add Qemu object flag to QemuImg create

* Add apache license header to new files

* Add Qemu object flag to QemuImg convert

* Set host details if hypervisor supports LUKS

* Add disk encrypt flag to APIs, diskoffering

* Schema upgrade 4.16.0.0 to 4.16.1.0 to support vol encryption

* Add Libvirt secret on disk attach, and refer to it in disk XML

* Add implementation of luks volume encryption to QCOW2 and RAW disk prep

* Start VMs that have encrypted volumes

* Add encrypt option to service offering and root volume provisioning

* Refactor volume passphrase into its own table and object

* CryptSetup, use key files to pass keys instead of command line

* Update storage types and allocators to select encryption support

* Allow agent.properties to define the hypervisor's private IP

* Implement createPhysicalDisk for ScaleIOStorageAdaptor

* UI: Add encrypt options to offerings

* UI module security updates

* Revert "UI module security updates" - belongs in base

This reverts commit a7cb7cf7f57aad38f0b5e5d67389c187b88ffd94.

* Add --target-is-zero support for QemuImg

* Allow qemu image options to be passed, API support convert encrypted

* Switch hypervisor encryption support detection to use KeyFiles

* Fixes for ScaleIO root disk encryption

* Resize root disk if it won't fit encryption header

* Use cryptsetup to prep raw root disks, when supported

* Create qcow2 formatting if necessary during initial template copy to ScaleIO

* Allow setting no cache for qemu-img during disk convert

* Use 1M sparse on qemu-img convert for zero target disks

* UI: Add volume encryption support to hypervisor details

* QemuImg use --image-opts and --object depending on version

* Only send storage commands that require encryption to hosts that support encryption

* Move host encryption detail to a static constant

* Update host selection to account for volume encryption support

Only attach volumes if encryption requirements are met

* Ensure resizeVolume won't allow changing encryption

* Catch edge cases for clearing passphrase when volume is removed

* Disable volume migration and extraction for encrypted volumes

* Register volume secret on destination host during live migration

* Fix configdrive path editing during live migration

* Ensure configdrive path is edited properly during live migration

* Pass along and store volume encryption format during creation

* Fixes for rebase

* Fix tests after rebase

* Add unit tests for DeploymentPlanningManagerImpl to support encryption

* Deployment planner tests for encryption support on last host

* Add deployment tests for encryption when calling planner

* Added Libvirt DiskDef test for encryption details

* Add test for KeyFile utility

* Add CryptSetup tests

* Add QemuImageOptionsTest

* add smoke tests for API level changes on create/list offerings

* Fix schema upgrade, do disk_offering_view first

* Fix UI to show hypervisor encryption support

* Load details into hostVO before trying to query them for encryption

* Remove whitespace in CreateNetworkOfferingTest

* Move QemuImageOptions to use constants for flag keys

* Set physical disk encrypt format during createDiskFromTemplate in KVM Agent

* Whitespace in AbstractStoragePoolAllocator

* Fix whitespace in VolumeDaoImpl

* Support old Qemu in convert

* Log how long it takes to generate a passphrase during volume creation

* Move passphrase generation to async portion of createVolume

* Revert "Allow agent.properties to define the hypervisor's private IP"

This reverts commit 6ea9377505f0e5ff9839156771a241aaa1925e70.

* Updated ScaleIO/PowerFlex storage plugin to support separate (storage) network for Host(KVM) SDC connection. (#144)

* Added smoke tests for volume encryption (in KVM). (#149)

* Updated ScaleIO pool unit tests.

* Some improvements/fixes for code smells (in ScaleIO storage plugin).

* Updated review changes for ScaleIO improvements.

* Updated host response parameter 'encryptionsupported' in the UI.

* Move passphrase generation for the volume to async portion, while deploying VM (#158)

* Move passphrase generation for the volume to async portion, while deploying VM.
* Updated logs, to include volume details.

* Fix schema upgrade, create passphrase table first

* Fixed the DB upgrade issue (as noticed in the logs below.)
DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) Error executing: CALL `cloud`.`IDEMPOTENT_ADD_FOREIGN_KEY`('cloud.volumes', 'passphrase', 'id')
ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:) java.sql.SQLException: Failed to open the referenced table 'passphrase'
ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to execute upgrade script

* Fixes for snapshots with encrypted qcow2
Fixes #159 #160 #163

* Support create/delete encrypted snapshots of encrypted qcow2 volumes
* Select endpoints that support encryption when snapshotting encrypted volumes
* Update revert snapshot to be compatible with encrypted snapshots
* Disallow volume and template create from encrypted vols/snapshots

* Disallow VM memory snapshots on encrypted vols. Fixes #157

* Fix for TemplateManagerImpl unit test failure

* Support offline resize of encrypted volumes. Fixes #168

* Fix for resize volume unit tests

* Updated libvirt resize volume unit tests

* Support volume encryption on kvm only, and  passphrase generation refactor (#169)

* Fail deploy VM when ROOT/DATA volume's offering has encryption enabled, on non-KVM hypervisors
* Fail attach volume when volume's offering has encryption enabled, on non-KVM hypervisors
* Refactor passphrase generation for volume

* Apply encryption to dest volume for live local storage migration
fixes #161

* Apply encryption to data volumes during live storage migration

Fixes #161

* Use the same encryption passphrase id for migrating volumes

* Pass secret consumer during storage migration prepare

Fix for #161

* Fixes create / delete volume snapshot issue, for stopped VMs

* Block volume snapshot if encrypted and VM is running

Fixes #159

* Block snap schedules on encrypted volumes

Fix for #159

* Support cryptsetup where luks type defaults to 2

Fixes #170

* Modify domain XML secret UUID when storage migrating VM

Fix for #172

* Remove any libvirt secrets on VM stop and post migration

Fix for #172

* Update disk profile with encryption requirement from the disk offering (#176)

Update disk profile with encryption requirement from the disk offering
and some code improvements

* Updated review changes / javadoc in ScaleIOUtil

Co-authored-by: Marcus Sorensen <mls@apple.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2022-06-29 15:51:00 +05:30
agent el8 fixes backport (#155) 2022-05-24 12:02:13 +05:30
api Volume encryption (#135) 2022-06-29 15:51:00 +05:30
build CLOUDSTACK-10012: Migrate to Embedded Jetty 2017-11-03 23:57:25 +05:30
client Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
cloud-cli packaging: Adding Centos8, Ubuntu 20.04, XCPNG8.1 Support (#4068) 2020-08-17 16:28:30 +05:30
core Volume encryption (#135) 2022-06-29 15:51:00 +05:30
debian Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
deps deps: fix pbm url download (#4440) 2020-11-03 18:13:31 +05:30
developer Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
engine Volume encryption (#135) 2022-06-29 15:51:00 +05:30
framework schema,server,api: events improvement (#127) 2022-05-05 13:44:33 +05:30
packaging packaging: display First Install and Onboarding Message (#5851) 2022-02-08 16:11:03 +01:00
plugins Volume encryption (#135) 2022-06-29 15:51:00 +05:30
python kvm: Add check if host meets the minimum requirements (#5369) 2021-09-17 10:21:05 +05:30
quickcloud Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
scripts CKS Enhancements and SystemVM template upgrade improvements (#5863) 2022-02-15 18:27:14 +05:30
server Volume encryption (#135) 2022-06-29 15:51:00 +05:30
services Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
setup Universal sshkey and password manager script (#4890) 2021-09-21 09:00:58 +02:00
systemvm Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
test Volume encryption (#135) 2022-06-29 15:51:00 +05:30
tools Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
ui Volume encryption (#135) 2022-06-29 15:51:00 +05:30
usage Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
utils Volume encryption (#135) 2022-06-29 15:51:00 +05:30
vmware-base Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
.dockerignore CLOUDSTACK-3049: Implemented role update for account. (#3058) 2019-01-25 10:02:56 -02:00
.gitignore CKS Enhancements and SystemVM template upgrade improvements (#5863) 2022-02-15 18:27:14 +05:30
.java-version cloudstack: add JDK11 support (#3601) 2020-02-12 12:58:25 +05:30
.python-version python3: Migrate Marvin and smoketests to python3 (#4727) 2021-05-04 23:19:37 +05:30
.travis.yml OVS/GRE: bug fixes (#5446) 2021-10-03 14:47:52 +05:30
CHANGES.md dummy commit to for PR close #95 2015-03-08 11:47:29 -04:00
CONTRIBUTING.md cloudstack: make code more inclusive 2021-06-08 15:47:20 +05:30
INSTALL.md INSTALL: updated maven download link (#4709) 2021-02-19 14:13:57 +05:30
ISSUE_TEMPLATE.md cloudstack: make code more inclusive 2021-06-08 15:47:20 +05:30
LICENSE CLOUDSTACK-10105: Use maven standard project structure in all projects (#2283) 2018-01-20 03:19:27 +05:30
LICENSE.header Add license check plugin to root pom. Disabled for the entire project, except for poms that override it (opendaylight, nvp) 2014-01-22 11:22:42 +01:00
NOTICE Update NOTICE to account for updated jQuery version 2014-08-26 13:13:29 -07:00
PULL_REQUEST_TEMPLATE.md cloudstack: make code more inclusive 2021-06-08 15:47:20 +05:30
PendingReleaseNotes Add a PendingRelease notes file (#2723) 2018-07-03 21:17:41 +02:00
README.md Update README.md (#5492) 2021-09-22 13:35:25 +05:30
pom.xml Updating pom.xml version numbers for release 4.16.1.0 2022-02-25 19:01:16 +05:30
requirements.txt CLOUDSTACK-8855 Improve Error Message for Host Alert State and reconnect host API. (#2387) 2018-03-14 15:27:43 -03:00

README.md

Apache CloudStack Build Status Quality Gate Status Lines of Code GitHub language count GitHub top language

Apache CloudStack

Apache CloudStack is open source software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. CloudStack is used by a number of service providers to offer public cloud services, and by many companies to provide an on-premises (private) cloud offering, or as part of a hybrid cloud solution.

CloudStack is a turnkey solution that includes the entire "stack" of features most organizations want with an IaaS cloud: compute orchestration, Network-as-a-Service, user and account management, a full and open native API, resource accounting, and a first-class User Interface (UI).

CloudStack currently supports the most popular hypervisors: VMware vSphere, KVM, XenServer, XenProject and Hyper-V as well as OVM and LXC containers.

Users can manage their cloud with an easy to use Web interface, command line tools, and/or a full-featured query based API.

For more information on Apache CloudStack, please visit the website

Who Uses CloudStack?

  • There are more than 150 known organizations using Apache CloudStack (or a commercial distribution of CloudStack). Our users include many major service providers running CloudStack to offer public cloud services, product vendors who incorporate or integrate with CloudStack in their own products, organizations who have used CloudStack to build their own private clouds, and systems integrators that offer CloudStack related services.

  • See our case studies highlighting successful deployments of Apache CloudStack.

  • See the up-to-date list of current users.

  • If you are using CloudStack in your organization and your company is not listed above, please complete our brief adoption survey. We're happy to keep your company name anonymous if you require.

Getting Started

  • Download a released version
  • Build from source with the instructions in the INSTALL.md file.

Getting Source Repository

Apache CloudStack project uses Git. The official Git repository is at:

https://gitbox.apache.org/repos/asf/cloudstack.git

And a mirror is hosted on Github:

https://github.com/apache/cloudstack

The Github mirror is strictly read only and provides convenience to users and developers to explore the code and for the community to accept contributions via Github pull requests.

Getting Involved and Contributing

Interested in helping out with Apache CloudStack? Great! We welcome participation from anybody willing to work The Apache Way and make a contribution. Note that you do not have to be a developer in order to contribute to Apache CloudStack. We need folks to help with documentation, translation, promotion etc. See our contribution page.

If you're interested in learning more or participating in the Apache CloudStack project, the mailing lists are the best way to do that. While the project has several communications channels, the mailing lists are the most active and the official channels for making decisions about the project itself.

Mailing lists:

Report and/or check bugs on Github and check our developer page for contributing code.

News and Events

Reporting Security Vulnerabilities

If you've found an issue that you believe is a security vulnerability in a released version of CloudStack, please report it to security@cloudstack.apache.org with details about the vulnerability, how it might be exploited, and any additional information that might be useful.

For more details, please visit our security page.

License

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Please see the LICENSE file included in the root directory of the source tree for extended license details.

Notice of Cryptographic Software

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software:

  • CloudStack makes use of JaSypt cryptographic libraries.
  • CloudStack has a system requirement of MySQL, and uses native database encryption functionality.
  • CloudStack makes use of the Bouncy Castle general-purpose encryption library.
  • CloudStack can optionally interact with and control OpenSwan-based VPNs.
  • CloudStack has a dependency on and makes use of JSch - a java SSH2 implementation.