Commit Graph

2116 Commits

Author SHA1 Message Date
Frank Maximus 3af54ec1a8 CLOUDSTACK-10382: Fix cloud-get-vm-data-configdrive.in (#2688) 2018-06-07 11:18:24 +05:30
Rafael Weingärtner f2efbcecec
[CLOUDSTACK-10240] ACS cannot migrate a local volume to shared storage (#2425)
* [CLOUDSTACK-10240] ACS cannot migrate a volume from local to shared storage.

CloudStack is logically restricting the migration of local storages to shared storage and vice versa. This restriction is a logical one and can be removed for XenServer deployments. Therefore, we will enable migration of volumes between local-shared storages in XenServers independently of their service offering. This will work as an override mechanism to the disk offering used by volumes. If administrators want to migrate local volumes to a shared storage, they should be able to do so (the hypervisor already allows that). The same the other way around.

* Cleanups implemented while working on [CLOUDSTACK-10240]

* Fix test case test_03_migrate_options_storage_tags

The changes applied were:
- When loading hypervisors capabilities we must use "default" instead of nulls
- "Enable" storage migration for simulator hypervisor
- Remove restriction on "ClusterScopeStoragePoolAllocator" to find shared pools
2018-03-07 18:23:15 -03:00
Frank Maximus b176648f90 CLOUDSTACK-9813: Extending Config Drive support (#2097)
Extending Config Drive support

* Added support for VMware
* Build configdrive.iso on ssvm
* Added support for VPC and Isolated Networks
* Moved implementation to new Service Provider
* UI fix: add support for urlencoded userdata
* Add support for building systemvm behind a proxy

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
2018-01-12 15:14:40 +05:30
Bitworks Software, Ltd a86160b389 Cloudstack 10170: Fix resource tags security bugs and add account tags support (#2350)
This PR introduces several features and fixes some bugs:
- account tags feature
- fixed resource tags bugs which happened during tags search (found wrong entries because of mysql string to number translation - see #905, but this PR does more and fixes also resource access - vulnerability during list resource tags)
- some marvin improvements (speed, sanity)

Improved resource tags code:
1. Enhanced listTags security
2. Added support for account tags (account tags are required to support tags common for all users of an account)
3. Improved the tag management code (refactoring and cleanup)

Marvin:
1. Fixed Marvin wait timeout between async pools. To decrease polling interval and improve CI speed.
2. Fixed /tmp/ to /tmp in zone configuration files.
3. Fixed + to os.path.join in log class.
4. Fixed + to os.path.join in deployDataCenter class.
5. Fixed typos in tag tests.
6. Modified Tags base class delete method.

Deploy Datacenter script:
1. Improved deployDatacenter. Added option logdir to specify where script places results of evaluation.

ConfigurationManagerImpl:
1. Added logging to ConfigurationManagerImpl to log when vlan is not found. Added test stubs for tags. Found accidental exception during simulator running after CI.

tests_tags.py:
1. Fixed stale undeleted tags.
2. Changed region:India to scope:TestName.
2018-01-09 13:55:34 +05:30
Rohit Yadav 9aa7d4e818 CLOUDSTACK-10013: SystemVM.iso refactoring and cleanups
- Removes old/dead files
- Refactors file path/location, backward compatible to filepaths in
  systemvm.isoa
- Fixes failures around apache2
2017-12-23 09:22:44 +05:30
Marc-Aurèle Brothier a372040798 CLOUDSTACK-10012: Load SQL schema scripts from JAR (#2247)
Load SQL schema scripts from the uber/fat JAR. This also removes the setup/db/db directory from centos packaging.
2017-12-19 12:11:38 +05:30
Paul Angus b417226d05 CLOUDSTACK-9839: Add XenServer 7.1 and 7.2 interoperablility (#2346)
Adds XenServer 7.1 and 7.2 interoperablility.
Adds hypervisor to capabilities list and add guest OS mappings (copied from 7.0.0 mappings initially).
Add newly supported OSes and remove deprecated ones (XenServer)
2017-12-15 17:37:19 +05:30
Paul Angus e2aee99623 Revert "add XenServer base 7.1 and 7.2 capabilities to database"
This reverts commit 1421a6e07a.
2017-11-29 16:36:31 +00:00
Paul Angus 1421a6e07a add XenServer base 7.1 and 7.2 capabilities to database 2017-11-29 16:25:56 +00:00
Sigert Goeminne 77864992fe CLOUDSTACK-9776: extra DHCP options support for Nuage VSP
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9776

Design-Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+extra+DHCP+option+support
2017-11-21 11:44:39 +01:00
Frank Maximus 14ab69e19a
Merge pull request #2005 from nuagenetworks/feature/nw_offering_for_vpc
CLOUDSTACK-9450: Network Offering for VPC based on DB flag
2017-11-16 17:33:07 +01:00
Abhinandan Prateek 4627fb2cd7 CLOUDSTACK-9972: Enhance listVolume API to include physical size and … (#2158)
* CLOUDSTACK-9972: Enhance listVolume API to include physical size and utilization.
Also fixed pool, cluster and pod info

* CLOUDSTACK-9972: Fix volume_view and duplicate API constant

* CLOUDSTACK-9972: Backport Do not allow vms to be deployed on hosts that are in disabled pod

* CLOUDSTACK-9972: Fix localization missing keys

* CLOUDSTACK-9972: Fix sql path
2017-11-05 21:44:43 +05:30
Rohit Yadav eda3b35bfa CLOUDSTACK-10012: Migrate to Embedded Jetty
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
  deprecates the need to create/store cloud.jks in conf folder and
  the db.cloud.keyStorePassphrase in db.properties file. This also
  remove the need of the --keystore-passphrase in the
  cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-03 23:57:25 +05:30
Frank Maximus d467a6b8ec CLOUDSTACK-9450: Network Offering for VPC based on DB flag
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-11-03 12:39:23 +01:00
Mike Tutkowski 4c89b5b97a Merge branch 'pr-2081' 2017-10-18 14:25:45 -06:00
dahn a379230e8e CLOUDSTACK-9957 Annotations (#2181)
* annotations on hosts

* Adding marvin tests

* rebase error

* review comments

* context for owner

* review

* illegal entity test

* entityType check on input

* Annotation events

* rebase issues
2017-10-13 11:55:26 +02:00
Rafael Weingärtner b044f1bc96 CLOUDSTACK-9859: Retirement of midonet plugin (final removal) (#2285)
Following the component retirement process defined in [1], a vote thread was started in [2]. The community decided to retire this Midonet plugin. This task represents the final step of the retirement, which is the removal of the plugin from CloudStacks code base.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68720798
[2] http://markmail.org/message/qigrtfirwnmct4hr
2017-10-10 10:08:23 +05:30
Harika Punna 6bb0ca2f85 This feature separates the snapshot creation on primary and its backing up on secondary.
As part of this, a new parameter, which is optional, is added to CreateSnapshotCmd, which seperates the creation and backup.

More details in the FS-
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Separate+creation+and+backup+operations+for+a+volume+snapshot
2017-10-04 14:39:03 +05:30
Nathan Johnson 3bc2341c64 CLOUDSTACK-9902: add consoleproxy.sslEnable global setting (#2078)
This adds the missing global configuration parameter consoleproxy.sslEnable required for enabling SSL on the console proxy VM.
2017-09-19 15:09:28 +05:30
Lars Tobias Skjong-Børsting a4036f7083 CLOUDSTACK-10067: Fix a case where a user 'ro' or 'roo' exists on the system, or other variants. (#2252)
If one sets user=roo, this will return two directories and then it will fail.
2017-09-18 12:15:36 +05:30
Rohit Yadav c0b33db5ce CLOUDSTACK-9782: Nested-oobm CloudStack plugin
Nested out-of-band management plugin to work with hosts that are VMs in
a CloudStack env.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
Gabriel Beims Bräscher 57255ac72c CLOUDSTACK-9914: Update Quota plugin to support currency values up to 5 decimal places (#2123)
Summary: this commit alters column currency_value from table
cloud_usage.quota_tariff to support values up to 5 decimal places. The
current implementation allows up to 2 decimal places.

Issue: need to use more than 2 decimal places to define resources values
in Quota tariff.

Solution: modify column currency_value from table
cloud_usage.quota_tariff to support values up to 5 decimal places.
Values with more than 5 decimal places will be displayed with scientific
notation in the user interface.

SQL command: "ALTER TABLE cloud_usage.quota_tariff MODIFY currency_value
DECIMAL(15,5) not null"
2017-08-23 21:40:48 +02:00
Sigert Goeminne dc361e1903 CLOUDSTACK-10031: change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
Change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
(cherry picked from commit 17bc6afc82)

This fixes some test_nic failures caused due to short aggregation command timeout

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-17 23:05:23 +02:00
Sigert Goeminne 17bc6afc82 CLOUDSTACK-10031: change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
Change default configuration for router.aggregation.command.each.timeout from 3 to 600 seconds (#2223)
2017-08-11 12:26:24 +02:00
Rohit Yadav 98dc4eb96a CLOUDSTACK-9782: New Background Polling Task Manager (#2218)
CloudStack has several background polling tasks that are spread across
the codebase, the aim of this work is to provide a single manager to
handle submission, execution and handling of background tasks. With
the framework implemented, existing oobm background task has been
refactored to use this manager.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-03 11:53:45 +02:00
Harika Punna bef5c17baa CLOUDSTACK-9982: Introduce new response parameter: PhysicalSize in listTemplates API response 2017-07-24 09:28:25 +02:00
Wei Zhou 047a57ddf2 CLOUDSTACK-7984: Add vm.{disk,network}.stats.interval.min global setting 2017-07-23 17:15:23 +02:00
Rajesh Battala 884606f77b CLOUDSTACK-8672 : NCC Integration with CloudStack. 2017-07-20 12:42:43 +05:30
Rohit Yadav 322e18a7b6 CLOUDSTACK-9928: Allow native CloudStack users to change password in UI
This allows native CloudStack users to change password in UI when LDAP
is enabled. Overall changes:
- A new usersource returned in the listUsers response
- Removed ldap check in the UI, replaced with check based on user source
- DB changes to include user.source in user_view
- Changed UI error message for non-native users trying to change password

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-14 14:55:08 +05:30
dahn 3a70c5cd16 schema: 4.10 to 4.11 upgrade path (#2175)
Add 4.10 to 4.11 upgrade path

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-13 15:45:26 +05:30
Rajani Karuturi be6e1a8596 Merge pull request #1759 from yvsubhash/CLOUDSTACK-9589
CLOUDSTACK-9589 vmName entries from host_details table for the VM's w…
2017-06-07 11:03:49 +05:30
subhash yedugundla 49a661be5f CLOUDSTACK-9589 vmName entries from host_details table for the VM's whose state is Expunging should be deleted during upgrade from older versions 2017-06-06 16:27:54 +05:30
Jayapal 065fa1c849 CLOUDSTACK-9669:Advaced zone isolated network egress destination cidr support
CLOUDSTACK-9669:egress destination cidr VR python script changes
    CLOUDSTACK-9669:egress destination API and orchestration changes
    CLOUDSTACK-9669: Added the ipset package in systemvm template
    CLOUDSTACK-9669:Added licence header for new files
    CLOUDSTACK-9669: replacing 0.0.0.0/0 with the network cidr

     ipset member add with 0.0.0.0/0 fails. So 0.0.0.0/0 replaced with the network cidr.
     In source cidr 0.0.0.0/0 is nothing but network cidr.
     updated the default egress all cidr with network cidr
2017-06-06 15:51:00 +05:30
Rohit Yadav f6273caf54 Merge branch '4.9'
- Fixed conflicts in DatabaseUpgradeChecker
- Fixed upgrade path as: 4.9.2.0 -> 4.9.3.0 -> 4.10.0.0

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-19 17:25:43 +05:30
Rajani Karuturi 1cabd32a6d schema: add new guest os/hypervisors from eee1e1a6
- Fix now() to utc timestamp

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-19 16:50:40 +05:30
Fabrice Brazier 86660c17fa CLOUDSTACK-9792: Add support for VMware 6.5
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-18 13:35:57 +05:30
Rohit Yadav 95914eef1f CLOUDSTACK-9792: Add upgrade path for 4.9.3.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-18 13:26:37 +05:30
Jayapal 8ef52325e7 CLOUDSTACK-9317: When there 1 static nat removing PF rules is handled
There is 1 static nat rule and 2 PF rule. Removing 2 PF rules was deleting static nat rule.
    Fixed this issue in this commit.
2017-05-17 11:07:28 +05:30
Rajani Karuturi a4f3c71744 Merge pull request #1849 from sudhansu7/CLOUDSTACK-9690
CLOUDSTACK-9690: Scale CentOS7 VM fails with error
2017-05-16 11:24:50 +05:30
Sudhansu e7a7db3ef3 CLOUDSTACK-9690: Scale CentOS7 VM fails with error
1. Removed XenServerGuestOsMemoryMap from CitrixHelper.java
This java file was holding a static in memory map named XenServerGuestOsMemoryMap. This was the source for xenserver dynamic memory values(max and min). These values were moved to guest_os_details table.

2. DAO layer was modified to access these values.
3. VirtualMachineTo object was modified to populate the dynamic memory values.
4. addGuestOs and UpdateGuestOS api has been modified to update memory values.
2017-05-05 10:58:12 +05:30
Rajani Karuturi f0dd5994b4 Merge pull request #1582 from DaanHoogland/CLOUDSTACK-9408
CLOUDSTACK-9408 for the move away from download.cloud.com
2017-04-24 07:57:07 +05:30
Rajani Karuturi a4dd6bdeeb Merge pull request #1955 from myENA/virtio-scsi
CLOUDSTACK-8239 Add VirtIO SCSI support for KVM hosts
2017-04-20 15:36:34 +05:30
Daan Hoogland 70ef0788c9 CLOUDSTACK-9408: Fix download urls in sql and scripts
This fixes the agreed upon url on download.cloudstack.org in various
sql files and misc scripts.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-04-20 12:33:33 +05:30
nvazquez edf0e2b26f CLOUDSTACK-9827: Storage tags stored in multiple places 2017-03-24 13:37:04 -03:00
Nathan Johnson 5c476492b1 CLOUDSTACK-8239 - Adding support for virtio-scsi on KVM hosts
This adds support for virtio-scsi on KVM hosts, either
for guests that are associated with a new os_type of 'Other PV Virtio-SCSI (64-bit)',
or when a VM or template is regstered with a detail parameter rootDiskController=scsi.

Update cloudstack add template dialog to allow for selecting rootDiskController with KVM

Update cloudstack kvm virtio-scsi to enable discard=unmap
2017-03-12 10:54:43 -05:00
Rajani Karuturi 87f11044c8 Merge pull request #1379 from jayapalu/configdriveScripts
CLOUDSTACK-8324: config drive data set/get scripts for the guest vmAdded the guest vm scripts for set/get the vm data, password and ssh keys

* pr/1379:
  CLOUDSTACK-8324: updated the mount directory name and kvm virt device
  CLOUDSTACK-8324: config drive data set/get scripts for the guest vm

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-23 05:40:14 +05:30
Rajani Karuturi fe555e194e Merge pull request #1881 from Accelerite/CLOUDSTACK-9721
CLOUDSTACK-9721: Remove deprecated/unused global configuration parameter - consoleproxy.loadscan.interval

* pr/1881:
  CLOUDSTACK-9721: Remove deprecated/unused global configuration parameter - consoleproxy.loadscan.interval

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-21 17:30:30 +05:30
nvazquez 3a6d98289c CLOUDSTACK-9539: Support changing Service offering for instance with VM Snapshots 2017-02-16 10:49:09 -03:00
Rajani Karuturi c3c9b8f38b Merge pull request #1747 from nvazquez/storageTags
CLOUDSTACK-9574: Redesign storage views## Part 1: Redesign storage tags
### Actual behavior

Primary storage tags are being saved as an entry on `storage_pool_details` with:
* name = TAG_NAME
* value = "true"

When a boolean property is defined in {{storage_pool_details}} and has value = "true", it is displayed as a tag.

![](https://issues.apache.org/jira/secure/attachment/12836196/PS-DETAILS.PNG "Screenshot 1 - Primary storage details")
![](https://issues.apache.org/jira/secure/attachment/12836195/PS.PNG "Screenshot 2 - Primary storage view")

### Goal

Redesign `Storage Tags` for Primary Storage view, to list only tags, as it is done in Host Tags (Hosts view).

## Part 2: Remove details from listImageStores API call response and UI

### Description
In Secondary Storage view we propose removing `Details` field, as `Setting` tab list details for a given image store. We also remove details from response on `listImageStores` API method

* pr/1747:
  CLOUDSTACK-9574: Redesign storage tags and remove details from listImageStores response and UI

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-16 12:30:15 +05:30
Jayapal 6518607885 CLOUDSTACK-8324: updated the mount directory name and kvm virt device 2017-02-15 13:58:54 +05:30
Jayapal ea09e6c196 CLOUDSTACK-8324: config drive data set/get scripts for the guest vm 2017-02-15 12:02:58 +05:30
Rajani Karuturi eee1e1a694 Added more Guest OS and their mappings on the hypervisor 2017-02-13 15:41:43 +05:30
Suresh Kumar Anaparti da7148a13e CLOUDSTACK-9721: Remove deprecated/unused global configuration parameter - consoleproxy.loadscan.interval 2017-02-09 10:55:29 +05:30
nvazquez 13ccbda10e CLOUDSTACK-9574: Redesign storage tags and remove details from listImageStores response and UI 2017-02-07 17:50:09 -03:00
Wei Zhou a2428508e2 CLOUDSTACK-8746: vm snapshot implementation for KVM
(1) add support to create/delete/revert vm snapshots on running vms with QCOW2 format
(2) add new API to create volume snapshot from vm snapshot
(3) delete metadata of vm snapshots before stopping/migrating and recover vm snapshots after starting/migrating
(4) enable deleting of VM snapshot on stopped vm or vm snapshot is not listed in qcow2 image.
(5) enable smoke tests for vmsnaphsots on KVM
2017-01-24 21:47:30 +01:00
Rohit Yadav 342162bad7 Merge branch '4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:59:02 +05:30
Rohit Yadav a0e36b73ae Merge pull request #1711 from syed/xenserver7
XenServer 7 SupportThis PR adds support for XenServer 7. I have manually done the following tests
- Create a new cluster with XenServer7
- Add Primary storage: Should create an SR on XS7
- Add another XS7 host to the Pool
- Add host2 to Cloudstack
- Create VM1 from template
- Create VM2 from template
- Ping/SSH VM1 to VM2 and vice-versa
- Stop/Delete/Expunge VM2
- Create Data disk
- Attach it to VM1
- Create VM snaphsot of VM1
- Restore VM snapshot of VM1
- Delete VM snapshot of VM1
- Create Volume snapshot of Datadisk
- Create volume snapshot of Root disk
- Create new template from snapshot of root disk
- Create volume from snapshot of datadisk
- Detach datadisk volume
- Delete datadisk volume
- Aquire a public IP
- Create a static nat to VM1
- Live migrate VM1 while traffic on VM
- Delete VM1

* pr/1711:
  [CLOUDSTACK-9662] Add support for XenServer 7

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:18:57 +05:30
Rohit Yadav f1cc4d1f5d schema: Upgrade path from 4.9.1.0 to 4.9.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:12:45 +05:30
Syed eabf862ba9 [CLOUDSTACK-9662] Add support for XenServer 7 2016-12-21 16:58:10 -05:00
Rohit Yadav dd321a6de3 CLOUDSTACK-9671: Fix sql change to corresponding version paths
- Fixes issue of failing upgrade paths
- Moves schema changes from PR #1615 in the 4.9.1.0 to 4.10.0.0 sql path

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-15 12:00:47 +05:30
Rohit Yadav 7df21f9cce
Merge branch '4.9' 2016-12-09 00:40:05 +05:30
Sateesh Chodapuneedi a401a3db9d CLOUDSTACK-9654 Missing hypervisor mapping of various SUSE Linux guest os versions on VMware 6.0
Issue: Currently many versions of SUSE Linux does not have any hypervisor mapping entry in guest_os_hypervisor table in cloud database for VMware 6.0. Also observed that the guest_os_name field is incorrect for some SUSE Linux variants, which results in deployed instance (with SUSE Linux) set to guest OS type as "Other (64-bit)" on vCenter, which would not represent the guest OS accurately on hypervisor.

Fix: Add the missing hypervisor mappings
2016-12-07 22:35:05 +05:30
Frank Maximus 4c91c9c519 CLOUDSTACK-9403: Support for shared networks in Nuage VSP plugin
BUG-ID: CLOUDSTACK-9403
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Rahul Singal <rahul.singal@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
Co-Authored-By: Nick Livens <nick.livens@nuagenetworks.net>
2016-12-07 10:03:28 +01:00
Rohit Yadav 1de336b975
Merge branch '4.9' 2016-11-29 17:19:47 +05:30
Rohit Yadav edf4fe951d
Merge branch '4.9' 2016-11-28 16:01:54 +05:30
Rohit Yadav 7a96d32c7e
CLOUDSTACK-9584: run component tests in Travis run
This would run additional component tests in Travis run

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-28 13:47:50 +05:30
Sateesh Chodapuneedi d7ff6b75f0 CLOUDSTACK-9624 Incorrect hypervisor mapping of guest os Windows 2008 Server R2 (64-bit) for VMware
Issue
Guest OS Windows Server 2008 R2 (64-bit) is being mapped to incorrect guest os at hypervisor, which is winLonghorn64Guest, same as that of Windows Server 2008 (64-bit).
Due to this the VM's guest os type was set to "Other (64-bit)", which would not represent the guest OS accurately on hypervisor.

Solution
Fix is to update incorrect guest_os_name field value in DB table cloud.guest_os_hypervisor.
Th query is,
UPDATE IGNORE cloud.guest_os_hypervisor SET guest_os_name = 'windows7Server64Guest' WHERE guest_os_id IN (SELECT id FROM guest_os WHERE display_name LIKE 'windows%2008%r2%64%') AND hypervisor_type = 'VMware' AND hypervisor_version != 'default';
2016-11-28 03:50:53 +05:30
Nick Livens 8d4dc81223 CLOUDSTACK-9402 : Support for underlay features (Source & Static NAT to underlay) in Nuage VSP plugin
CLOUDSTACK-9402 : Marvin tests for Source NAT and Static NAT features verification with NuageVsp (both overlay and underlay infra).

Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>, Frank Maximus <frank.maximus@nuagenetworks.net>
2016-11-24 21:33:02 +01:00
Syed f46651e672 Support Backup of Snapshots for Managed Storage
This PR adds an ability to Pass a new parameter, locationType,
    to the “createSnapshot” API command. Depending on the locationType,
    we decide where the snapshot should go in case of managed storage.

    There are two possible values for the locationType param

    1) `Standard`: The standard operation for managed storage is to
    keep the snapshot on the device. For non-managed storage, this will
    be to upload it to secondary storage. This option will be the
    default.

    2) `Archive`: Applicable only to managed storage. This will
    keep the snapshot on the secondary storage. For non-managed
    storage, this will result in an error.

    The reason for implementing this feature is to avoid a single
    point of failure for primary storage. Right now in case of managed
    storage, if the primary storage goes down, there is no easy way
    to recover data as all snapshots are also stored on the primary.
    This features allows us to mitigate that risk.
2016-10-30 23:19:58 -06:00
Rajani Karuturi c9e7ccf46e Merge pull request #1615 from nvazquez/nfsConfigKey
CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UIJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9438

### Introduction

From #1361 it was possible to configure NFS version for secondary storage mount.
However, changing NFS version requires inserting an new detail on `image_store_details` table, with `name = 'nfs.version'` and `value = X` where X is desired NFS version, and then restarting management server for changes to take effect.

Our improvement aims to make NFS version changeable from UI, instead of previously described workflow.

### Proposed solution
Basically, NFS version is defined as an image store ConfigKey, this implied:
* Adding a new Config scope: **ImageStore**
* Make `ImageStoreDetailsDao` class to extend `ResourceDetailsDaoBase` and `ImageStoreDetailVO` implement `ResourceDetail`
* Insert `'display'` column on `image_store_details` table
* Extending `ListCfgsCmd` and `UpdateCfgCmd` to support **ImageStore** scope, which implied:
** Injecting `ImageStoreDetailsDao` and `ImageStoreDao` on `ConfigurationManagerImpl` class, on `cloud-server` module.

### Important
It is important to mention that `ImageStoreDaoImpl` and `ImageStoreDetailsDaoImpl` classes were moved from `cloud-engine-storage` to `cloud-engine-schema` module in order to Spring find those beans to inject on `ConfigurationManagerImpl` in `cloud-server` module.

We had this maven dependencies between modules:
* `cloud-server --> cloud-engine-schema`
* `cloud-engine-storage --> cloud-secondary-storage --> cloud-server`

As `ImageStoreDaoImpl` and `ImageStoreDetailsDao` were defined in `cloud-engine-storage`, and they needed in `cloud-server` module, to be injected on `ConfigurationManagerImpl`, if we added dependency from `cloud-server` to `cloud-engine-storage` we would introduce a dependency cycle. To avoid this cycle, we moved those classes to `cloud-engine-schema` module

* pr/1615:
  CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UI

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2016-10-27 11:48:23 +05:30
Rajani Karuturi 2a827b5ba0 CLOUDSTACK-9525 add support for windows 10 guest os in KVM 2016-10-10 11:26:40 +05:30
Rajani Karuturi f06352cc7a CLOUDSTACK-9525 add support for windows 10 guest os in vmware ESX 6.0
queries author @serg38
(https://github.com/apache/cloudstack/pull/1698#issuecomment-251672496)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2016-10-10 11:26:05 +05:30
Rajani Karuturi eba58db744 CLOUDSTACK-9525 add support for windows 10 guest os in xenserver 6.5.0 2016-10-10 11:25:14 +05:30
Rohit Yadav 6e7fdc404c Merge branch '4.9' 2016-10-01 00:09:12 +05:30
Rohit Yadav 335ad1a867 CLOUDSTACK-9364: Add Ubuntu 16.04 as a guest OS
Adds Ubuntu 16.04 as a supported guest os. This allows users to select
the OS when creating a template etc.

Note: As XenServer 6.5 does not have 16.04 in its list of known Ubuntu releases,
as a workaround 16.04 guest os refers to 14.04 for XenServer 6.5.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-09-29 22:09:30 +05:30
nvazquez 2e77496601 CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UI 2016-09-28 08:51:37 -07:00
Bharat Kumar 8805f955a3 Added license headders 2016-09-12 16:13:32 +05:30
Bharat Kumar 75f89c7676 CLOUDSTACK-8751
Minimise network downtime during network updates when redundant VR is being used.
database schema changes
Made changes to the updateNetwork API.
2016-09-12 15:24:47 +05:30
Rohit Yadav 95f1de89b4
Merge branch '4.9'
Conflicts:
	engine/schema/src/com/cloud/upgrade/DatabaseUpgradeChecker.java
	engine/schema/test/com/cloud/upgrade/DatabaseUpgradeCheckerTest.java
	tools/marvin/setup.py

This fixes class names to make things consistent as per the 4.9 PR on master.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-09-02 23:08:49 +05:30
John Burwell eabda0eb55 Renames of 4.9.0->4.9.1.0 upgrade scripts to match the four position version scheme
* Renames schema-490to491*.sql to schema490to4910*.sql
    * Renames the Upgrade490to491 class to Upgrade490to4910
    * Removes the unused s_logger contant from Upgrade490to4910
    * Updates the version in tools/marvin/setup to 4.9.1.0-SNAPSHOT
2016-09-01 21:19:29 -04:00
Rohit Yadav 19549f7b01
Merge branch '4.9' 2016-09-01 17:27:12 +05:30
Abhinandan Prateek 180406e895 CLOUDSTACK-9481: Convert MyISAM table to InnoDB for consistency
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-09-01 14:02:08 +05:30
Rohit Yadav 9555492b4d Merge branch '4.9' 2016-08-23 14:16:53 +05:30
Rohit Yadav b87eda230a cloudstack: upgrade path from 4.9.0 to 4.9.1
- Adds db upgrade path from 4.9.0 to 4.9.1
- CLOUDSTACK-9449: Fix typo in default user role description

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:15 +05:30
Rohit Yadav 546a3f8884 cloudstack: fix upgrade paths to 4.10.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-04 13:03:46 +05:30
Will Stevens 121b3d6403 Merge pull request #1567 from exoscale/CLOUDSTACK-9238
CLOUDSTACK-9238: Fix URL length to 2048 for all url fields in VOI will update the PR to add max field length in the API commands too

* pr/1567:
  API: update url field max length
  not needed on host table
  Fix URL length to 2048 for all url fields in VO

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-27 15:20:22 -04:00
Marc-Aurèle Brothier d0453fc296 not needed on host table 2016-05-27 08:26:09 +02:00
Marc-Aurèle Brothier a59ee03fd7 Fix URL length to 2048 for all url fields in VO 2016-05-27 08:16:05 +02:00
Will Stevens c122f4c879 Merge pull request #1552 from romain-dartigues/feature_dhcp_ubuntu
add DHCP lease folders for UbuntuAdd `/var/lib/dhcp/*` to the search path for Ubuntu; tested under Ubuntu Precise, Trusty and Xenial.

* pr/1552:
  add DHCP lease folders for Ubuntu

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 23:01:53 -04:00
Romain Dartigues 3bb70daecc add DHCP lease folders for Ubuntu
Add "/var/lib/dhcp/*" to the search path for Ubuntu; tested under
Ubuntu Precise and Trusty, not yet with newer releases.
2016-05-18 23:34:19 +02:00
SudharmaJain d64f224a7b CLOUDSTACK-8970 Centos 6.{1,2,3,4,5} guest OS mapping for vmware is not available 2016-05-17 10:10:41 +05:30
Rohit Yadav 4d5e8df2f9 travis: Use patched version of ipmitool for tests
- For out-of-band management feature (CLOUDSTACK-9299) use patched version of
  ipmitool that would work on trusty travis machines
- The ipmitool used is from xenial/16.04 release with patch from RedHat
  https://bugzilla.redhat.com/show_bug.cgi?id=1286035
- Installs ipmitool from xenial repositories to get all the dependencies
  and then install patched deb version
- Skip test if the known failure occurs

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav 07564469e9 CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav 4347776ac6 CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
This feature allows root administrators to define new roles and associate API
permissions to them.

A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack

DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
  current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
  Admin, Domain Admin and User) which maintains this association by requiring
  all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
  removal of roles and/or modifications of permissions, without the need
  of restarting management server(s)

Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
  deployments will continue to use the older static role based api access checker
  with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
  roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
  to add existing set of permissions to the default roles. cloud.account
  will have a new role_id column which will be populated based on default roles
  as well

Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:45:19 +05:30
Will Stevens ccf2265460 Merge pull request #1466 from nvazquez/graldboptimization
CLOUDSTACK-9340: General DB Optimization## Description
In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken and some deficiencies were discovered, we can group them in 4 groups:
* Incorrect PRIMARY key
* Duplicate PRIMARY KEY
* Missing indexes (Add indexes to avoid full table scans)
* Some views query (Change view to improve account retrieval speed)

* pr/1466:
  CLOUDSTACK-9340: General DB Optimization

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:57:39 -04:00
Wido den Hollander 92e3ba2053 CLOUDSTACK-8818: Use MySQL native connector with Python
MySQLdb has been deprecated and is also not supported in Python 3.

mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.

https://dev.mysql.com/doc/connector-python/en/
2016-05-04 23:28:52 +02:00
nvazquez 8b6e6185f5 CLOUDSTACK-9340: General DB Optimization 2016-05-04 09:13:17 -07:00
Will Stevens f960725dd5 Merge pull request #1457 from nvazquez/excludeclusters
CLOUDSTACK-9333: Exclude clusters from OVF operationsJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9333

## Introduction
In some environments there is a need to exclude certain VMware clusters from performing OVF operations. This operations are part of:
* create template
* create volume snaphsot
* copy template, volume, images from primary storage to secondary storage
* migrate volume
* participate when a template gets cached over to primary storage.

In ESX/ESXi, OVF operations are low priority and bound to a single CPU and most likely get throttled to certain IOPS and network limits.
If the hypervisor chosen for OVF operations is weak or overloaded this results in significantly longer execution of such OVF command and therefore degraded performance of underlying CloudStack API call.

### Proposed solution
It is proposed to add a way to exclude hosts from selected clusters for OVF operations.
To exclude a cluster, would be necessary to insert a record in <code>cluster_details</code> specifying property **vmware.exclude_from_ovf** in this way: (supposing we want to exclude cluster X)

| cluster_id | name| value |
|:-------------:|:-------------:|:-------------:|
|X|vmware.exclude_from_ovf|true|

* pr/1457:
  CLOUDSTACK-9333: Exclude clusters for OVF operations

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-07 13:30:20 -04:00