Commit Graph

2094 Commits

Author SHA1 Message Date
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
Gabriel Beims Bräscher 57255ac72c CLOUDSTACK-9914: Update Quota plugin to support currency values up to 5 decimal places (#2123)
Summary: this commit alters column currency_value from table
cloud_usage.quota_tariff to support values up to 5 decimal places. The
current implementation allows up to 2 decimal places.

Issue: need to use more than 2 decimal places to define resources values
in Quota tariff.

Solution: modify column currency_value from table
cloud_usage.quota_tariff to support values up to 5 decimal places.
Values with more than 5 decimal places will be displayed with scientific
notation in the user interface.

SQL command: "ALTER TABLE cloud_usage.quota_tariff MODIFY currency_value
DECIMAL(15,5) not null"
2017-08-23 21:40:48 +02:00
Sigert Goeminne dc361e1903 CLOUDSTACK-10031: change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
Change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
(cherry picked from commit 17bc6afc82)

This fixes some test_nic failures caused due to short aggregation command timeout

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-17 23:05:23 +02:00
Sigert Goeminne 17bc6afc82 CLOUDSTACK-10031: change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
Change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
2017-08-11 12:26:24 +02:00
Rohit Yadav 98dc4eb96a CLOUDSTACK-9782: New Background Polling Task Manager (#2218)
CloudStack has several background polling tasks that are spread across
the codebase, the aim of this work is to provide a single manager to
handle submission, execution and handling of background tasks. With
the framework implemented, existing oobm background task has been
refactored to use this manager.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-03 11:53:45 +02:00
Harika Punna bef5c17baa CLOUDSTACK-9982: Introduce new response parameter: PhysicalSize in listTemplates API response 2017-07-24 09:28:25 +02:00
Wei Zhou 047a57ddf2 CLOUDSTACK-7984: Add vm.{disk,network}.stats.interval.min global setting 2017-07-23 17:15:23 +02:00
Rajesh Battala 884606f77b CLOUDSTACK-8672 : NCC Integration with CloudStack. 2017-07-20 12:42:43 +05:30
Rohit Yadav 322e18a7b6 CLOUDSTACK-9928: Allow native CloudStack users to change password in UI
This allows native CloudStack users to change password in UI when LDAP
is enabled. Overall changes:
- A new usersource returned in the listUsers response
- Removed ldap check in the UI, replaced with check based on user source
- DB changes to include user.source in user_view
- Changed UI error message for non-native users trying to change password

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-14 14:55:08 +05:30
dahn 3a70c5cd16 schema: 4.10 to 4.11 upgrade path (#2175)
Add 4.10 to 4.11 upgrade path

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-13 15:45:26 +05:30
Rajani Karuturi be6e1a8596 Merge pull request #1759 from yvsubhash/CLOUDSTACK-9589
CLOUDSTACK-9589 vmName entries from host_details table for the VM's w…
2017-06-07 11:03:49 +05:30
subhash yedugundla 49a661be5f CLOUDSTACK-9589 vmName entries from host_details table for the VM's whose state is Expunging should be deleted during upgrade from older versions 2017-06-06 16:27:54 +05:30
Jayapal 065fa1c849 CLOUDSTACK-9669:Advaced zone isolated network egress destination cidr support
CLOUDSTACK-9669:egress destination cidr VR python script changes
    CLOUDSTACK-9669:egress destination API and orchestration changes
    CLOUDSTACK-9669: Added the ipset package in systemvm template
    CLOUDSTACK-9669:Added licence header for new files
    CLOUDSTACK-9669: replacing 0.0.0.0/0 with the network cidr

     ipset member add with 0.0.0.0/0 fails. So 0.0.0.0/0 replaced with the network cidr.
     In source cidr 0.0.0.0/0 is nothing but network cidr.
     updated the default egress all cidr with network cidr
2017-06-06 15:51:00 +05:30
Rohit Yadav f6273caf54 Merge branch '4.9'
- Fixed conflicts in DatabaseUpgradeChecker
- Fixed upgrade path as: 4.9.2.0 -> 4.9.3.0 -> 4.10.0.0

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-19 17:25:43 +05:30
Rajani Karuturi 1cabd32a6d schema: add new guest os/hypervisors from eee1e1a6
- Fix now() to utc timestamp

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-19 16:50:40 +05:30
Fabrice Brazier 86660c17fa CLOUDSTACK-9792: Add support for VMware 6.5
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-18 13:35:57 +05:30
Rohit Yadav 95914eef1f CLOUDSTACK-9792: Add upgrade path for 4.9.3.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-18 13:26:37 +05:30
Jayapal 8ef52325e7 CLOUDSTACK-9317: When there 1 static nat removing PF rules is handled
There is 1 static nat rule and 2 PF rule. Removing 2 PF rules was deleting static nat rule.
    Fixed this issue in this commit.
2017-05-17 11:07:28 +05:30
Rajani Karuturi a4f3c71744 Merge pull request #1849 from sudhansu7/CLOUDSTACK-9690
CLOUDSTACK-9690: Scale CentOS7 VM fails with error
2017-05-16 11:24:50 +05:30
Sudhansu e7a7db3ef3 CLOUDSTACK-9690: Scale CentOS7 VM fails with error
1. Removed XenServerGuestOsMemoryMap from CitrixHelper.java
This java file was holding a static in memory map named XenServerGuestOsMemoryMap. This was the source for xenserver dynamic memory values(max and min). These values were moved to guest_os_details table.

2. DAO layer was modified to access these values.
3. VirtualMachineTo object was modified to populate the dynamic memory values.
4. addGuestOs and UpdateGuestOS api has been modified to update memory values.
2017-05-05 10:58:12 +05:30
Rajani Karuturi f0dd5994b4 Merge pull request #1582 from DaanHoogland/CLOUDSTACK-9408
CLOUDSTACK-9408 for the move away from download.cloud.com
2017-04-24 07:57:07 +05:30
Rajani Karuturi a4dd6bdeeb Merge pull request #1955 from myENA/virtio-scsi
CLOUDSTACK-8239 Add VirtIO SCSI support for KVM hosts
2017-04-20 15:36:34 +05:30
Daan Hoogland 70ef0788c9 CLOUDSTACK-9408: Fix download urls in sql and scripts
This fixes the agreed upon url on download.cloudstack.org in various
sql files and misc scripts.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-04-20 12:33:33 +05:30
nvazquez edf0e2b26f CLOUDSTACK-9827: Storage tags stored in multiple places 2017-03-24 13:37:04 -03:00
Nathan Johnson 5c476492b1 CLOUDSTACK-8239 - Adding support for virtio-scsi on KVM hosts
This adds support for virtio-scsi on KVM hosts, either
for guests that are associated with a new os_type of 'Other PV Virtio-SCSI (64-bit)',
or when a VM or template is regstered with a detail parameter rootDiskController=scsi.

Update cloudstack add template dialog to allow for selecting rootDiskController with KVM

Update cloudstack kvm virtio-scsi to enable discard=unmap
2017-03-12 10:54:43 -05:00
Rajani Karuturi 87f11044c8 Merge pull request #1379 from jayapalu/configdriveScripts
CLOUDSTACK-8324: config drive data set/get scripts for the guest vmAdded the guest vm scripts for set/get the vm data, password and ssh keys

* pr/1379:
  CLOUDSTACK-8324: updated the mount directory name and kvm virt device
  CLOUDSTACK-8324: config drive data set/get scripts for the guest vm

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-23 05:40:14 +05:30
Rajani Karuturi fe555e194e Merge pull request #1881 from Accelerite/CLOUDSTACK-9721
CLOUDSTACK-9721: Remove deprecated/unused global configuration parameter - consoleproxy.loadscan.interval

* pr/1881:
  CLOUDSTACK-9721: Remove deprecated/unused global configuration parameter - consoleproxy.loadscan.interval

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-21 17:30:30 +05:30
nvazquez 3a6d98289c CLOUDSTACK-9539: Support changing Service offering for instance with VM Snapshots 2017-02-16 10:49:09 -03:00
Rajani Karuturi c3c9b8f38b Merge pull request #1747 from nvazquez/storageTags
CLOUDSTACK-9574: Redesign storage views## Part 1: Redesign storage tags
### Actual behavior

Primary storage tags are being saved as an entry on `storage_pool_details` with:
* name = TAG_NAME
* value = "true"

When a boolean property is defined in {{storage_pool_details}} and has value = "true", it is displayed as a tag.

![](https://issues.apache.org/jira/secure/attachment/12836196/PS-DETAILS.PNG "Screenshot 1 - Primary storage details")
![](https://issues.apache.org/jira/secure/attachment/12836195/PS.PNG "Screenshot 2 - Primary storage view")

### Goal

Redesign `Storage Tags` for Primary Storage view, to list only tags, as it is done in Host Tags (Hosts view).

## Part 2: Remove details from listImageStores API call response and UI

### Description
In Secondary Storage view we propose removing `Details` field, as `Setting` tab list details for a given image store. We also remove details from response on `listImageStores` API method

* pr/1747:
  CLOUDSTACK-9574: Redesign storage tags and remove details from listImageStores response and UI

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-16 12:30:15 +05:30
Jayapal 6518607885 CLOUDSTACK-8324: updated the mount directory name and kvm virt device 2017-02-15 13:58:54 +05:30
Jayapal ea09e6c196 CLOUDSTACK-8324: config drive data set/get scripts for the guest vm 2017-02-15 12:02:58 +05:30
Rajani Karuturi eee1e1a694 Added more Guest OS and their mappings on the hypervisor 2017-02-13 15:41:43 +05:30
Suresh Kumar Anaparti da7148a13e CLOUDSTACK-9721: Remove deprecated/unused global configuration parameter - consoleproxy.loadscan.interval 2017-02-09 10:55:29 +05:30
nvazquez 13ccbda10e CLOUDSTACK-9574: Redesign storage tags and remove details from listImageStores response and UI 2017-02-07 17:50:09 -03:00
Wei Zhou a2428508e2 CLOUDSTACK-8746: vm snapshot implementation for KVM
(1) add support to create/delete/revert vm snapshots on running vms with QCOW2 format
(2) add new API to create volume snapshot from vm snapshot
(3) delete metadata of vm snapshots before stopping/migrating and recover vm snapshots after starting/migrating
(4) enable deleting of VM snapshot on stopped vm or vm snapshot is not listed in qcow2 image.
(5) enable smoke tests for vmsnaphsots on KVM
2017-01-24 21:47:30 +01:00
Rohit Yadav 342162bad7 Merge branch '4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:59:02 +05:30
Rohit Yadav a0e36b73ae Merge pull request #1711 from syed/xenserver7
XenServer 7 SupportThis PR adds support for XenServer 7. I have manually done the following tests
- Create a new cluster with XenServer7
- Add Primary storage: Should create an SR on XS7
- Add another XS7 host to the Pool
- Add host2 to Cloudstack
- Create VM1 from template
- Create VM2 from template
- Ping/SSH VM1 to VM2 and vice-versa
- Stop/Delete/Expunge VM2
- Create Data disk
- Attach it to VM1
- Create VM snaphsot of VM1
- Restore VM snapshot of VM1
- Delete VM snapshot of VM1
- Create Volume snapshot of Datadisk
- Create volume snapshot of Root disk
- Create new template from snapshot of root disk
- Create volume from snapshot of datadisk
- Detach datadisk volume
- Delete datadisk volume
- Aquire a public IP
- Create a static nat to VM1
- Live migrate VM1 while traffic on VM
- Delete VM1

* pr/1711:
  [CLOUDSTACK-9662] Add support for XenServer 7

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:18:57 +05:30
Rohit Yadav f1cc4d1f5d schema: Upgrade path from 4.9.1.0 to 4.9.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:12:45 +05:30
Syed eabf862ba9 [CLOUDSTACK-9662] Add support for XenServer 7 2016-12-21 16:58:10 -05:00
Rohit Yadav dd321a6de3 CLOUDSTACK-9671: Fix sql change to corresponding version paths
- Fixes issue of failing upgrade paths
- Moves schema changes from PR #1615 in the 4.9.1.0 to 4.10.0.0 sql path

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-15 12:00:47 +05:30
Rohit Yadav 7df21f9cce
Merge branch '4.9' 2016-12-09 00:40:05 +05:30
Sateesh Chodapuneedi a401a3db9d CLOUDSTACK-9654 Missing hypervisor mapping of various SUSE Linux guest os versions on VMware 6.0
Issue: Currently many versions of SUSE Linux does not have any hypervisor mapping entry in guest_os_hypervisor table in cloud database for VMware 6.0. Also observed that the guest_os_name field is incorrect for some SUSE Linux variants, which results in deployed instance (with SUSE Linux) set to guest OS type as "Other (64-bit)" on vCenter, which would not represent the guest OS accurately on hypervisor.

Fix: Add the missing hypervisor mappings
2016-12-07 22:35:05 +05:30
Frank Maximus 4c91c9c519 CLOUDSTACK-9403: Support for shared networks in Nuage VSP plugin
BUG-ID: CLOUDSTACK-9403
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Rahul Singal <rahul.singal@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
Co-Authored-By: Nick Livens <nick.livens@nuagenetworks.net>
2016-12-07 10:03:28 +01:00
Rohit Yadav 1de336b975
Merge branch '4.9' 2016-11-29 17:19:47 +05:30
Rohit Yadav edf4fe951d
Merge branch '4.9' 2016-11-28 16:01:54 +05:30
Rohit Yadav 7a96d32c7e
CLOUDSTACK-9584: run component tests in Travis run
This would run additional component tests in Travis run

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-28 13:47:50 +05:30
Sateesh Chodapuneedi d7ff6b75f0 CLOUDSTACK-9624 Incorrect hypervisor mapping of guest os Windows 2008 Server R2 (64-bit) for VMware
Issue
Guest OS Windows Server 2008 R2 (64-bit) is being mapped to incorrect guest os at hypervisor, which is winLonghorn64Guest, same as that of Windows Server 2008 (64-bit).
Due to this the VM's guest os type was set to "Other (64-bit)", which would not represent the guest OS accurately on hypervisor.

Solution
Fix is to update incorrect guest_os_name field value in DB table cloud.guest_os_hypervisor.
Th query is,
UPDATE IGNORE cloud.guest_os_hypervisor SET guest_os_name = 'windows7Server64Guest' WHERE guest_os_id IN (SELECT id FROM guest_os WHERE display_name LIKE 'windows%2008%r2%64%') AND hypervisor_type = 'VMware' AND hypervisor_version != 'default';
2016-11-28 03:50:53 +05:30
Nick Livens 8d4dc81223 CLOUDSTACK-9402 : Support for underlay features (Source & Static NAT to underlay) in Nuage VSP plugin
CLOUDSTACK-9402 : Marvin tests for Source NAT and Static NAT features verification with NuageVsp (both overlay and underlay infra).

Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>, Frank Maximus <frank.maximus@nuagenetworks.net>
2016-11-24 21:33:02 +01:00
Syed f46651e672 Support Backup of Snapshots for Managed Storage
This PR adds an ability to Pass a new parameter, locationType,
    to the “createSnapshot” API command. Depending on the locationType,
    we decide where the snapshot should go in case of managed storage.

    There are two possible values for the locationType param

    1) `Standard`: The standard operation for managed storage is to
    keep the snapshot on the device. For non-managed storage, this will
    be to upload it to secondary storage. This option will be the
    default.

    2) `Archive`: Applicable only to managed storage. This will
    keep the snapshot on the secondary storage. For non-managed
    storage, this will result in an error.

    The reason for implementing this feature is to avoid a single
    point of failure for primary storage. Right now in case of managed
    storage, if the primary storage goes down, there is no easy way
    to recover data as all snapshots are also stored on the primary.
    This features allows us to mitigate that risk.
2016-10-30 23:19:58 -06:00
Rajani Karuturi c9e7ccf46e Merge pull request #1615 from nvazquez/nfsConfigKey
CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UIJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9438

### Introduction

From #1361 it was possible to configure NFS version for secondary storage mount.
However, changing NFS version requires inserting an new detail on `image_store_details` table, with `name = 'nfs.version'` and `value = X` where X is desired NFS version, and then restarting management server for changes to take effect.

Our improvement aims to make NFS version changeable from UI, instead of previously described workflow.

### Proposed solution
Basically, NFS version is defined as an image store ConfigKey, this implied:
* Adding a new Config scope: **ImageStore**
* Make `ImageStoreDetailsDao` class to extend `ResourceDetailsDaoBase` and `ImageStoreDetailVO` implement `ResourceDetail`
* Insert `'display'` column on `image_store_details` table
* Extending `ListCfgsCmd` and `UpdateCfgCmd` to support **ImageStore** scope, which implied:
** Injecting `ImageStoreDetailsDao` and `ImageStoreDao` on `ConfigurationManagerImpl` class, on `cloud-server` module.

### Important
It is important to mention that `ImageStoreDaoImpl` and `ImageStoreDetailsDaoImpl` classes were moved from `cloud-engine-storage` to `cloud-engine-schema` module in order to Spring find those beans to inject on `ConfigurationManagerImpl` in `cloud-server` module.

We had this maven dependencies between modules:
* `cloud-server --> cloud-engine-schema`
* `cloud-engine-storage --> cloud-secondary-storage --> cloud-server`

As `ImageStoreDaoImpl` and `ImageStoreDetailsDao` were defined in `cloud-engine-storage`, and they needed in `cloud-server` module, to be injected on `ConfigurationManagerImpl`, if we added dependency from `cloud-server` to `cloud-engine-storage` we would introduce a dependency cycle. To avoid this cycle, we moved those classes to `cloud-engine-schema` module

* pr/1615:
  CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UI

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2016-10-27 11:48:23 +05:30
Rajani Karuturi 2a827b5ba0 CLOUDSTACK-9525 add support for windows 10 guest os in KVM 2016-10-10 11:26:40 +05:30
Rajani Karuturi f06352cc7a CLOUDSTACK-9525 add support for windows 10 guest os in vmware ESX 6.0
queries author @serg38
(https://github.com/apache/cloudstack/pull/1698#issuecomment-251672496)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2016-10-10 11:26:05 +05:30
Rajani Karuturi eba58db744 CLOUDSTACK-9525 add support for windows 10 guest os in xenserver 6.5.0 2016-10-10 11:25:14 +05:30
Rohit Yadav 6e7fdc404c Merge branch '4.9' 2016-10-01 00:09:12 +05:30
Rohit Yadav 335ad1a867 CLOUDSTACK-9364: Add Ubuntu 16.04 as a guest OS
Adds Ubuntu 16.04 as a supported guest os. This allows users to select
the OS when creating a template etc.

Note: As XenServer 6.5 does not have 16.04 in its list of known Ubuntu releases,
as a workaround 16.04 guest os refers to 14.04 for XenServer 6.5.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-09-29 22:09:30 +05:30
nvazquez 2e77496601 CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UI 2016-09-28 08:51:37 -07:00
Bharat Kumar 8805f955a3 Added license headders 2016-09-12 16:13:32 +05:30
Bharat Kumar 75f89c7676 CLOUDSTACK-8751
Minimise network downtime during network updates when redundant VR is being used.
database schema changes
Made changes to the updateNetwork API.
2016-09-12 15:24:47 +05:30
Rohit Yadav 95f1de89b4
Merge branch '4.9'
Conflicts:
	engine/schema/src/com/cloud/upgrade/DatabaseUpgradeChecker.java
	engine/schema/test/com/cloud/upgrade/DatabaseUpgradeCheckerTest.java
	tools/marvin/setup.py

This fixes class names to make things consistent as per the 4.9 PR on master.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-09-02 23:08:49 +05:30
John Burwell eabda0eb55 Renames of 4.9.0->4.9.1.0 upgrade scripts to match the four position version scheme
* Renames schema-490to491*.sql to schema490to4910*.sql
    * Renames the Upgrade490to491 class to Upgrade490to4910
    * Removes the unused s_logger contant from Upgrade490to4910
    * Updates the version in tools/marvin/setup to 4.9.1.0-SNAPSHOT
2016-09-01 21:19:29 -04:00
Rohit Yadav 19549f7b01
Merge branch '4.9' 2016-09-01 17:27:12 +05:30
Abhinandan Prateek 180406e895 CLOUDSTACK-9481: Convert MyISAM table to InnoDB for consistency
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-09-01 14:02:08 +05:30
Rohit Yadav 9555492b4d Merge branch '4.9' 2016-08-23 14:16:53 +05:30
Rohit Yadav b87eda230a cloudstack: upgrade path from 4.9.0 to 4.9.1
- Adds db upgrade path from 4.9.0 to 4.9.1
- CLOUDSTACK-9449: Fix typo in default user role description

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:15 +05:30
Rohit Yadav 546a3f8884 cloudstack: fix upgrade paths to 4.10.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-04 13:03:46 +05:30
Will Stevens 121b3d6403 Merge pull request #1567 from exoscale/CLOUDSTACK-9238
CLOUDSTACK-9238: Fix URL length to 2048 for all url fields in VOI will update the PR to add max field length in the API commands too

* pr/1567:
  API: update url field max length
  not needed on host table
  Fix URL length to 2048 for all url fields in VO

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-27 15:20:22 -04:00
Marc-Aurèle Brothier d0453fc296 not needed on host table 2016-05-27 08:26:09 +02:00
Marc-Aurèle Brothier a59ee03fd7 Fix URL length to 2048 for all url fields in VO 2016-05-27 08:16:05 +02:00
Will Stevens c122f4c879 Merge pull request #1552 from romain-dartigues/feature_dhcp_ubuntu
add DHCP lease folders for UbuntuAdd `/var/lib/dhcp/*` to the search path for Ubuntu; tested under Ubuntu Precise, Trusty and Xenial.

* pr/1552:
  add DHCP lease folders for Ubuntu

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:01:53 -04:00
Romain Dartigues 3bb70daecc add DHCP lease folders for Ubuntu
Add "/var/lib/dhcp/*" to the search path for Ubuntu; tested under
Ubuntu Precise and Trusty, not yet with newer releases.
2016-05-18 23:34:19 +02:00
SudharmaJain d64f224a7b CLOUDSTACK-8970 Centos 6.{1,2,3,4,5} guest OS mapping for vmware is not available 2016-05-17 10:10:41 +05:30
Rohit Yadav 4d5e8df2f9 travis: Use patched version of ipmitool for tests
- For out-of-band management feature (CLOUDSTACK-9299) use patched version of
  ipmitool that would work on trusty travis machines
- The ipmitool used is from xenial/16.04 release with patch from RedHat
  https://bugzilla.redhat.com/show_bug.cgi?id=1286035
- Installs ipmitool from xenial repositories to get all the dependencies
  and then install patched deb version
- Skip test if the known failure occurs

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav 07564469e9 CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav 4347776ac6 CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
This feature allows root administrators to define new roles and associate API
permissions to them.

A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack

DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
  current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
  Admin, Domain Admin and User) which maintains this association by requiring
  all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
  removal of roles and/or modifications of permissions, without the need
  of restarting management server(s)

Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
  deployments will continue to use the older static role based api access checker
  with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
  roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
  to add existing set of permissions to the default roles. cloud.account
  will have a new role_id column which will be populated based on default roles
  as well

Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:45:19 +05:30
Will Stevens ccf2265460 Merge pull request #1466 from nvazquez/graldboptimization
CLOUDSTACK-9340: General DB Optimization## Description
In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken and some deficiencies were discovered, we can group them in 4 groups:
* Incorrect PRIMARY key
* Duplicate PRIMARY KEY
* Missing indexes (Add indexes to avoid full table scans)
* Some views query (Change view to improve account retrieval speed)

* pr/1466:
  CLOUDSTACK-9340: General DB Optimization

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:57:39 -04:00
Wido den Hollander 92e3ba2053 CLOUDSTACK-8818: Use MySQL native connector with Python
MySQLdb has been deprecated and is also not supported in Python 3.

mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.

https://dev.mysql.com/doc/connector-python/en/
2016-05-04 23:28:52 +02:00
nvazquez 8b6e6185f5 CLOUDSTACK-9340: General DB Optimization 2016-05-04 09:13:17 -07:00
Will Stevens f960725dd5 Merge pull request #1457 from nvazquez/excludeclusters
CLOUDSTACK-9333: Exclude clusters from OVF operationsJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9333

## Introduction
In some environments there is a need to exclude certain VMware clusters from performing OVF operations. This operations are part of:
* create template
* create volume snaphsot
* copy template, volume, images from primary storage to secondary storage
* migrate volume
* participate when a template gets cached over to primary storage.

In ESX/ESXi, OVF operations are low priority and bound to a single CPU and most likely get throttled to certain IOPS and network limits.
If the hypervisor chosen for OVF operations is weak or overloaded this results in significantly longer execution of such OVF command and therefore degraded performance of underlying CloudStack API call.

### Proposed solution
It is proposed to add a way to exclude hosts from selected clusters for OVF operations.
To exclude a cluster, would be necessary to insert a record in <code>cluster_details</code> specifying property **vmware.exclude_from_ovf** in this way: (supposing we want to exclude cluster X)

| cluster_id | name| value |
|:-------------:|:-------------:|:-------------:|
|X|vmware.exclude_from_ovf|true|

* pr/1457:
  CLOUDSTACK-9333: Exclude clusters for OVF operations

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-07 13:30:20 -04:00
Rohit Yadav 87b431c0f3 travis: Fix simulator tests and optimize default global configs
- Migrate to trusty based Travis VMs
- Increase tests across five build matrices
- Fix xunit-reader output, include time
- Fix pip/python usage, pkg installation
- Build CloudStack in parallel with -T4
- Deploy database with optimized global settings

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-04-07 02:19:52 +05:30
nvazquez 4aae051558 CLOUDSTACK-9333: Exclude clusters for OVF operations 2016-04-05 08:21:36 -07:00
nvazquez c19d8b4739 CLOUDSTACK-9298: Remove user definer from view creations 2016-03-11 07:04:33 -03:00
nvazquez db54b26fed CLOUDSTACK-9298: Improve ListTemplatesCmd, ListVolumesCmd and ListVMsCmd performance 2016-03-10 06:07:29 -08:00
Remi Bergsma 14c837ab83 CLOUDSTACK-8300: Set indexes on event table 2016-01-30 16:50:46 +01:00
Rohit Yadav 431389da9d Merge pull request #1044 from ustcweizhou/keep-sshkey
CLOUDSTACK-5822: keep user-added sshkeys in authorized_keysFor now, if we add the ssh key inside the vm (not on cloudstack UI), the sshkey will be removed if we reset the sshkey on cloudstack UI.

After this commit, the sshkey (added by cloudstack) will end with cloudstack@apache.org.
We will only control the sshkeys with cloudstack@apache.org.

This will be used for multiple sshkey support for vm in the future.

* pr/1044:
  CLOUDSTACK-5822: keep user-added sshkeys in authorized_keys

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-01-27 16:12:06 +01:00
Remi Bergsma e1ecfad322 Implement upgrade path 4.8.1 -> 4.9.0 2016-01-26 15:21:23 +01:00
Remi Bergsma 95331aad30 Implement upgrade path 4.8.0 -> 4.8.1 2016-01-26 13:42:53 +01:00
Remi Bergsma 75b68c6829 Merge release branch 4.7 to master
* 4.7:
  Fix unable to setup more than one Site2Site VPN Connection
  FIX S2S VPN rVPC: Check only redundant routers in state MASTER
  PEP8 of integration/smoke/test_vpc_vpn
  Add S2S VPN test for Redundant VPC
  Make integration/smoke/test_vpc_vpn Hypervisor independant
  FIX VPN: non-working ipsec commands
  [UI] MADNESS
  [DB] Add force_encap field to s2s_customer_gateway table
  [ROUTER] Add forceencaps field to python router ipsec config method
  [TEST] unittest needs rework
  [MARVIN] Add forceencap field to VpnCustomerGateway class in marvin base
  [CORE] Add Force UDP Encapsulation option to Site2Site VPN
  CLOUDSTACK-9186: Root admin cannot see VPC created by Domain admin user
  CLOUDSTACK-9192: UpdateVpnCustomerGateway is failing
  CLOUDSTACK-6485 prevent ip asignment of private gw iface
  CLOUDSTACK-9204 Do not error when staticroute is already gone
  make both check lines consistent
  CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh
  CLOUDSTACK-9202 Bump ssh timeout
2016-01-16 19:54:41 +01:00
Wido den Hollander a171bbc96d CLOUDSTACK-9238: Increase URL fields to 2048 charachters from 255
255 characters is to small for various URLs like S3 pre-signed URLs.

This causes one or more characters to be chopped of the end of the URL
and this renders them useless.

Internally in the code all URLs are passed as Strings and they are not
sized limited. This was purely in the database.

Other URL fields in the database were already 2048 characters.

This limit was introduced in the 4.1 to 4.2 upgrade when Object storage
like S3 and Swift was introduced in CloudStack for Secondary Storage.
2016-01-14 14:06:04 +01:00
Michael Andersen 74f670dc3c [DB] Add force_encap field to s2s_customer_gateway table 2016-01-07 19:27:43 +01:00
nvazquez de23c94f33 CLOUDSTACK-9074: Move sql to 4.8.0 schema 2015-12-24 16:12:01 -03:00
nvazquez 07264204f5 CLOUDSTACK-9074: Drop nicira_nvp_router_map unique index on logicalrouter_uuid 2015-12-24 16:07:06 -03:00
Remi Bergsma 18fb10439b Implement upgrade path 4.7.1 -> 4.8.0 2015-12-21 23:02:43 +01:00
Remi Bergsma 9537f5d32b Merge release branch '4.7' into master
* 4.7:
  Debian changelog updated
  Implement upgrade path 4.7.0 -> 4.7.1
  Updating pom.xml version numbers for release 4.7.1-SNAPSHOT
  Implement upgrade path 4.6.2 -> 4.6.3
  Updating pom.xml version numbers for release 4.6.3-SNAPSHOT
2015-12-21 21:57:52 +01:00
Remi Bergsma 04c759390b Implement upgrade path 4.7.0 -> 4.7.1 2015-12-21 20:25:54 +01:00
Maneesha.P 626052128b CLOUDSTACK-9103 : Missing OS Mappings for VMware 6.0 2015-12-14 15:30:46 +05:30
Wei Zhou 24f1396ac1 CLOUDSTACK-9129: list vpc routers by keyword in Infrastructure -> Virtual Routers
and two more changes:
(1) add network name/vpc name in the listRouters response
(2) add network name/vpc id, vpc name in the router details page
2015-12-10 09:14:12 +01:00
Abhinandan Prateek 987fcbd441 CLOUDSTACK-8592: Implement Quota service
Quota service while allowing for scalability will make sure that the cloud is
not exploited by attacks, careless use and program errors. To address this
problem, we propose to employ a quota-enforcement service that allows resource
usage within certain bounds as defined by policies and available quotas for
various entities.  Quota service extends the functionality of usage server to
provide a measurement for the resources used by the accounts and domains using a
common unit referred to as cloud currency in this document. It can be configured
to ensure that your usage won’t exceed the budget allocated to accounts/domain
in cloud currency.  It will let user know how much of the cloud resources he is
using. It will help the cloud admins, if they want, to ensure that a user does
not go beyond his allocated quota. Per usage cycle if a account is found to be
exceeding its quota then it is locked. Locking an account means that it will not
be able to initiat e a new resource allocation request, whether it is more
storage or an additional ip. Needless to say quota service as well as any action
on the account is configurable.

Changes from Github code review:

- Added marvin test for quota plugin API
- removed unused commented code
- debug messages in debug enabled check
- checks for nulls, fixed access to member variables and feature
- changes based on PR comments
- unit tests for UsageTypes
- unit tests for all Cmd classes
- unit tests for all service and manager impls
- try-catch-finally or try-with-resource in dao impls for failsafe db switching
- remove dead code
- add missing quota calculation case (regression fixed)
- replace tabs with spaces in pom.xmls
- quota: though default value for quota_calculated is 0, the usage server
  makes it null while entering usage entries. Flipping the condition so
  as to acocunt for that.
- quotatypes: fix NPE in quota type
- quota framework test fixes
- made statement period configurable
- changed default email templates to reflect the fact that exhausted quota may not result in a locked account
- added quotaUpdateCmd that refreshes quota balances and sends alerts and statements
- report quotaSummary command returns quota balance, quota usage and state for all account
- made UI framework changes to allow for text area input in edit views
- process usage entries that have greater than 0 usage
- orocess quota entries only if tariff is non zero
- if there are credit entries but no balance entry create a dummy balance entry
- remove any credit entries that are before the last balance entry
  when displaying balance statement
- on a rerun the last balance is now getting added

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-+FS
PR: https://github.com/apache/cloudstack/pull/768

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-12-07 23:02:48 +05:30
Daan Hoogland 3de117c8be 4.6.0 upgrade path is to pass 4.6.1 to create the extra view in there 2015-12-04 10:06:19 +01:00
Remi Bergsma 7e902cd505 Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9075 - Uses the same vlan since it should have been already released
  CLOUDSTACK-9075 - Adds VPC static routes test
  CLOUDSTACK-9075 - Covers Private GW ACL with Redundant VPCs
  CLOUDSTACK-9075 - Add method to get list of Physical Networks per zone
  CLOUDSTACK-6276 Removing unused parameter in integration test for projects
  CLOUDSTACK-6276 Removing unused parameter in integration test
  CLOUDSTACK-6276 Fixing affinity groups for projects
2015-12-03 20:42:41 +01:00
Remi Bergsma a1d2c531f6 Merge pull request #1007 from ustcweizhou/dedicated-ip-for-domain
[4.7] CLOUDSTACK-8958: add dedicated ips to domain (account for now)For now, we dedicate ip pool to account, however, other accounts in the same domain cannot fetch the ip from this ip pool.
By dedicating ip pool to domain, accounts in the domain can fetch the public ip from same ip pool.

* pr/1007:
  CLOUDSTACK-8958: throw an exception if project account cannot be found
  CLOUDSTACK-8958: add dedicated ips to domain (account for now)

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-29 09:39:25 +01:00