Commit Graph

2676 Commits

Author SHA1 Message Date
Rohit Yadav acc5fdcdbd
CLOUDSTACK-10290: allow config drives on primary storage for KVM (#2651)
This introduces a new global setting `vm.configdrive.primarypool.enabled` to toggle creation/hosting of config drive iso files on primary storage, the default will be false causing them to be hosted on secondary storage. The current support is limited from hypervisor resource side and in current implementation limited to `KVM` only. The next big change is that config drive is created at a temporary location by management server and shipped to either KVM or SSVM agent via cmd-answer pattern, the data of which is not logged in logs. This saves us from adding genisoimage dependency on cloudstack-agent pkg.

The APIs to reset ssh public key, password and user-data (via update VM API) requires that VM should be shutdown. Therefore, in the refactoring I removed the case of updation of existing ISO. If there are objections I'll re-put the strategy to detach+attach new config iso as a way of updation. In the refactored implementation, the folder name is changed to lower-cased configdrive. And during VM start, migration or shutdown/removal if primary storage is enable for use, the KVM agent will handle cleanup tasks otherwise SSVM agent will handle them.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 14:27:23 +05:30
Nicolas Vazquez 06f7e495dc Host Affinity plugin (#2630)
This implements a new host-affinity plugin.
2018-05-21 12:49:08 +05:30
Rafael Weingärtner b9ed42bd29
Fix primary storage count when deleting volumes (#2629)
* Primary Storage count for an account does not decrease when a Data Disk is deleted

When a data disk is created and not attached in a running VM, the "deleteVolume" will not decrement the count for used primary storage in the VMs accounting information. The property that is not being decremented is called "primarystoragetotal"; this information can be retrieved via "listAccounts" API method.

Steps to reproduce this issue:
1 - Create an account, deploy a VM in it
2 - Check the primary storage count for the account with listAccounts API
3 - Create a data disk
4 - Check the primary storage count for the account with listAccounts API
5 - Delete the Data disk
6 - Check the primary storage count for the account with listAccounts API - It is the same as before deleting the data disk (it should not be the same as the value in step 2!)

* formatting and cleanups

* fix imports that were wrongly changed during rebase
2018-05-16 15:28:28 -03:00
Rohit Yadav f663b926c7
config-drive: use hostname of VM instance of internal VM id (#2645)
This fixes config drive to use VM's user provided host-name instead of
the internal VM instance ID for hostname related config in both
cloudstack and openstack metadata bundled in the ISO.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-16 13:19:21 +05:30
Rohit Yadav a77ed56b86
CLOUDSTACK-9114: Reduce VR downtime during network restart (#2508)
This introduces a rolling restart of VRs when networks are restarted
with cleanup option for isolated and VPC networks. A make redundant option is
shown for isolated networks now in UI.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-11 12:48:07 +05:30
Rohit Yadav 4534cefa40
backports for 4.11.1 from master (#2621)
* CLOUDSTACK-10147 Disabled Xenserver Cluster can still deploy VM's. Added code to skip disabled clusters when selecting a host (#2442)

(cherry picked from commit c3488a51db)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10318: Bug on sorting ACL rules list in chrome (#2478)

(cherry picked from commit 4412563f19)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10284:Creating a snapshot from VM Snapshot generates error if hypervisor is not KVM.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10221: Allow IPv6 when creating a Basic Network (#2397)

Since CloudStack 4.10 Basic Networking supports IPv6 and thus
should be allowed to be specified when creating a network.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
(cherry picked from commit 9733a10ecd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10214: Unable to remove local primary storage (#2390)

Allow admins to remove primary storage pool.
Cherry-picked from eba2e1d8a1

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* dateutil: constistency of tzdate input and output (#2392)

Signed-off-by: Yoan Blanc <yoan.blanc@exoscale.ch>
Signed-off-by: Daan Hoogland <daan.hoogland@shapeblue.com>
(cherry picked from commit 2ad5202823)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10054:Volume download times out in 3600 seconds (#2244)

(cherry picked from commit bb607d07a9)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* When creating a new account (via domain admin) it is possible to select “root admin” as the role for the new user (#2606)

* create account with domain admin showing 'root admin' role

Domain admins should not be able to assign the role of root admin to new users. Therefore, the role ‘root admin’ (or any other of the same type) should not be visible to domain admins.

* License and formatting

* Break long sentence into multiple lines

* Fix wording of method 'getCurrentAccount'

* fix typo in variable name

* [CLOUDSTACK-10259] Missing float part of secondary storage data in listAccounts

* [CLOUDSTACK-9338] ACS not accounting resources of VMs with custom service offering

ACS is accounting the resources properly when deploying VMs with custom service offerings. However, there are other methods (such as updateResourceCount) that do not execute the resource accounting properly, and these methods update the resource count for an account in the database. Therefore, if a user deploys VMs with custom service offerings, and later this user calls the “updateResourceCount” method, it (the method) will only account for VMs with normal service offerings, and update this as the number of resources used by the account. This will result in a smaller number of resources to be accounted for the given account than the real used value. The problem becomes worse because if the user starts to delete these VMs, it is possible to reach negative values of resources allocated (breaking all of the resource limiting for accounts). This is a very serious attack vector for public cloud providers!

* [CLOUDSTACK-10230] User should not be able to use removed “Guest OS type” (#2404)

* [CLOUDSTACK-10230] User is able to change to “Guest OS type” that has been removed

Users are able to change the OS type of VMs to “Guest OS type” that has been removed. This becomes a security issue when we try to force users to use HVM VMs (Meltdown/Spectre thing). A removed “guest os type” should not be usable by any users in the cloud.

* Remove trailing lines that are breaking build due to checkstyle compliance

* Remove unused imports

* fix classes that were in the wrong folder structure

* Updates to capacity management
2018-05-09 15:20:19 +05:30
Frank Maximus 85203248a4 ConfigDrive fixes: CLOUDSTACK-10288, CLOUDSTACK-10289 (#2566)
* CLOUDSTACK-10289: Config Drive Metadata: Use VM UUID instead of VM id

* CLOUDSTACK-10288: Config Drive Userdata: support for binary userdata

* CLOUDSTACK-10358: SSH keys are missing on Config Drive disk in some cases
2018-04-26 10:10:23 -03:00
Khosrow Moossavi b6d420bec3 CLOUDSTACK-9677: Adding storage policy support for swift as secondary storage (#2412)
Original-Author: @pdube on PR Fixes #1830.
2018-04-26 00:42:15 +02:00
Nicolas Vazquez b69c37862c [Vmware] Fix for OVF parsing error (#2567) 2018-04-13 11:19:30 +02:00
Nicolas Vazquez 6a75423779 CLOUDSTACK-10231: Asserted fixes for Direct Download on KVM (#2408)
Several fixes addressed:

- Dettach ISO fails when trying to detach a direct download ISO
- Fix for metalink support on SSVM agents (this closes CLOUDSTACK-10238)
- Reinstall VM from bypassed registered template (this closes CLOUDSTACK-10250)
- Fix upload certificate error message even though operation was successful
- Fix metalink download, checksum retry logic and metalink SSVM downloader
2018-03-20 19:24:46 +05:30
Nicolas Vazquez 74db647dbb CLOUDSTACK-10321: CPU Cap for KVM (#2482) 2018-03-14 18:21:24 +00:00
Mike Tutkowski a30a31c9b7 CLOUDSTACK-9620: Enhancements for managed storage (#2298)
Allowed zone-wide primary storage based on a custom plug-in to be added via the GUI in a KVM-only environment (previously this only worked for XenServer and VMware)

Added support for root disks on managed storage with KVM

Added support for volume snapshots with managed storage on KVM

Enable creating a template directly from a volume (i.e. without having to go through a volume snapshot) on KVM with managed storage

Only allow the resizing of a volume for managed storage on KVM if the volume in question is either not attached to a VM or is attached to a VM in the Stopped state.

Included support for Reinstall VM on KVM with managed storage

Enabled offline migration on KVM from non-managed storage to managed storage and vice versa

Included support for online storage migration on KVM with managed storage (NFS and Ceph to managed storage)

Added support to download (extract) a managed-storage volume to a QCOW2 file

When uploading a file from outside of CloudStack to CloudStack, set the min and max IOPS, if applicable.

Included support for the KVM auto-convergence feature

The compression flag was actually added in version 1.0.3 (1000003) as opposed to version 1.3.0 (1003000) (changed this to reflect the correct version)

On KVM when using iSCSI-based managed storage, if the user shuts a VM down from the guest OS (as opposed to doing so from CloudStack), we need to pass to the KVM agent a list of applicable iSCSI volumes that need to be disconnected.

Added a new Global Setting: kvm.storage.live.migration.wait

For XenServer, added a check to enforce that only volumes from zone-wide managed storage can be storage motioned from a host in one cluster to a host in another cluster (cannot do so at the time being with volumes from cluster-scoped managed storage)

Don’t allow Storage XenMotion on a VM that has any managed-storage volume with one or more snapshots.

Enabled for managed storage with VMware: Template caching, create snapshot, delete snapshot, create volume from snapshot, and create template from snapshot

Added an SIOC API plug-in to support VMware SIOC

When starting a VM that uses managed storage in a cluster other than the one it last was running in, we need to remove the reference to the iSCSI volume from the original cluster.

Added the ability to revert a volume to a snapshot

Enabled cluster-scoped managed storage

Added support for VMware dynamic discovery
2018-01-15 00:05:52 +05:30
Frank Maximus b176648f90 CLOUDSTACK-9813: Extending Config Drive support (#2097)
Extending Config Drive support

* Added support for VMware
* Build configdrive.iso on ssvm
* Added support for VPC and Isolated Networks
* Moved implementation to new Service Provider
* UI fix: add support for urlencoded userdata
* Add support for building systemvm behind a proxy

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
2018-01-12 15:14:40 +05:30
Bitworks Software, Ltd 7ca4582a85 CLOUDSTACK-10188 - Resource Accounting for primary storage is Broken when Domains are in use (#2362)
During storage expunge domain resource statistics for primary storage space resource counter is not updated for domain. This leads to the situation when domain resource statistics for primary storage is overfilled (statistics only increase but not decrease).

Global scheduled task resourcecount.check.interval > 0 provides a workaround but not fixes the problem truly because when accounts inside domains use primary_storage allocation/deallocation intensively it leads to service block of operation.

NB: Unable to implement marvin tests because it (marvin) places in database weird primary storage volume size of 100 when creating VM from template. It might be a sign of opening a new issue for that bug.
2018-01-10 22:11:26 +05:30
Abhinandan Prateek 64832fd70a CLOUDSTACK-4757: Support OVA files with multiple disks for templates (#2146)
CloudStack volumes and templates are one single virtual disk in case of XenServer/XCP and KVM hypervisors since the files used for templates and volumes are virtual disks (VHD, QCOW2). However, VMware volumes and templates are in OVA format, which are archives that can contain a complete VM including multiple VMDKs and other files such as ISOs. And currently, Cloudstack only supports Template creation based on OVA files containing a single disk. If a user creates a template from a OVA file containing more than 1 disk and launches an instance using this template, only the first disk is attached to the new instance and other disks are ignored.
Similarly with uploaded volumes, attaching an uploaded volume that contains multiple disks to a VM will result in only one VMDK to being attached to the VM.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Support+OVA+files+containing+multiple+disks

This behavior needs to be improved in VMWare to support OVA files with multiple disks for both uploaded volumes and templates. i.e. If a user creates a template from a OVA file containing more than 1 disk and launches an instance using this template, the first disk should be attached to the new instance as the ROOT disk and volumes should be created based on other VMDK disks in the OVA file and should be attached to the instance.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-10 22:10:41 +05:30
Bitworks Software, Ltd a86160b389 Cloudstack 10170: Fix resource tags security bugs and add account tags support (#2350)
This PR introduces several features and fixes some bugs:
- account tags feature
- fixed resource tags bugs which happened during tags search (found wrong entries because of mysql string to number translation - see #905, but this PR does more and fixes also resource access - vulnerability during list resource tags)
- some marvin improvements (speed, sanity)

Improved resource tags code:
1. Enhanced listTags security
2. Added support for account tags (account tags are required to support tags common for all users of an account)
3. Improved the tag management code (refactoring and cleanup)

Marvin:
1. Fixed Marvin wait timeout between async pools. To decrease polling interval and improve CI speed.
2. Fixed /tmp/ to /tmp in zone configuration files.
3. Fixed + to os.path.join in log class.
4. Fixed + to os.path.join in deployDataCenter class.
5. Fixed typos in tag tests.
6. Modified Tags base class delete method.

Deploy Datacenter script:
1. Improved deployDatacenter. Added option logdir to specify where script places results of evaluation.

ConfigurationManagerImpl:
1. Added logging to ConfigurationManagerImpl to log when vlan is not found. Added test stubs for tags. Found accidental exception during simulator running after CI.

tests_tags.py:
1. Fixed stale undeleted tags.
2. Changed region:India to scope:TestName.
2018-01-09 13:55:34 +05:30
Nicolas Vazquez e86bb41e0e CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates (#2379)
This feature allows using templates and ISOs avoiding secondary storage as intermediate cache on KVM. The virtual machine deployment process is enhanced to supported bypassed registered templates and ISOs, delegating the work of downloading them to primary storage to the KVM agent instead of the SSVM agent.

Template and ISO registration:
- When hypervisor is KVM, a checkbox is displayed with 'Direct Download' label.
- API methods registerTemplate and registerISO are both extended with this new parameter directdownload.
- On template or ISO registration, no download job is sent to SSVM agent, CloudStack would only persist an entry on template_store_ref indicating that template or ISO has been marked as 'Direct Download' (bypassing Secondary Storage). These entries are persisted as:
template_id = Template or ISO id on vm_template table
store_id NULL
download_state = BYPASSED
state = Ready
(Note: these entries allow users to deploy virtual machine from registered templates or ISOs)
- An URL validation command is sent to a random KVM host to check if template/ISO location can be reached. Metalink are also supported by this feature. In case of a metalink, it is fetched and URL check is performed on each of its URLs.
- Checksum should be provided as indicated on #2246: {ALGORITHM}CHKSUMHASH
- After template or ISO is registered, it would be displayed in the UI

Virtual machine deployment:
When a 'Direct Download' template is selected for deployment, CloudStack would delegate template downloading to destination storage pool via destination host by a new pluggable download manager.
Download manager would handle template downloading depending on URL protocol. In case of HTTP, request headers can be set by the user via vm_template_details. Those details should be persisted as:
Key: HTTP_HEADER
Value: HEADERNAME:HEADERVALUE

In case of HTTPS, a new API method is added uploadTemplateDirectDownloadCertificate to allow user importing a client certificate into all KVM hosts' keystore before deployment.
After template or ISO is downloaded to primary storage, usual entry would be persisted on template_spool_ref indicating the mapping between template/ISO and storage pool.
2018-01-09 12:22:18 +05:30
Rafael Weingärtner 1d0f2128f6 [CLOUDSTACK-9025][CLOUDSTACK-10128] solve problems of templates created based on snapshots (#2315)
The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only tackling the problem for CloudStack deployments that use single hypervisor types (restricted to XenServer). Additionally, the lack of information regarding that solution (poor documentation, test cases and description in PRs and Jira ticket) led the code to be removed in #1124 after a long discussion and analysis in #1056. That piece of code seemed logicless (and it was!).  It would receive a hostId and then change that hostId for other hostId of the zone without doing any check; it was not even checking the hypervisor and storage in which the host was plugged into.

The problem reported in #CLOUDSTACK-9025 is caused by partial snapshots that are taken in XenServer. This means, we do not take a complete snapshot, but a partial one that contains only the modified data. This requires rebuilding the VHD hierarchy when creating a template out of the snapshot. The point is that the first hostId received is not a hostId, but a system VM ID(SSVM). That is why the code in #1176 fixed the problem for some deployment scenarios, but would cause problems for scenarios where we have multiple hypervisors in the same zone. We need to execute the creation of the VHD that represents the template in the hypervisor, so the VHD chain can be built using the parent links.

This commit changes the method com.cloud.hypervisor.XenServerGuru.getCommandHostDelegation(long, Command). From now on we replace the hostId that is intended to execute the “copy command” that will create the VHD of the template according to some conditions that were already in place. The idea is that starting with XenServer 6.2.0 hotFix ESP1004 we need to execute the command in the hypervisor host and not from the SSVM. Moreover, the method was improved making it readable and understandable; it was also created test cases assuring that from XenServer 6.2.0 hotFix ESP1004 and upward versions we change the hostId that will be used to execute the “copy command”.

Furthermore, we are not selecting a random host from a zone anymore. A new method was introduced in the HostDao called “findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using this method we look for a host that is in the cluster that is using the storage pool where the volume from which the Snaphost is taken of. By doing this, we guarantee that the host that is connected to the primary storage where all of the snapshots parent VHDs are stored is used to create the template.

Consider using Disabled hosts when no Enabled hosts are found

This also closes #2317
2018-01-06 09:55:29 +01:00
PranaliM 5fad84174a CLOUDSTACK-10104: Optimize database transactions in ListDomain API to improve performance (#2282)
While creating the response object for the 'listDomain' API, several database calls are triggered to fetch details like parent domain, project limit, IP limit, etc. These database calls are triggered for each record found in the main fetch query, which is causing the response to slow down.

Fix:
The database transactions are reduced to improve response of the Listdomain API
2018-01-05 21:51:46 +05:30
dahn 6724a47122
CLOUDSTACK-10121 moveUser (#2301)
* internal service call for moveUser
* expose moveUser as API
* move uuid to external entity
2017-12-30 11:40:14 +00:00
Marc-Aurèle Brothier e27b3e120d CLOUDSTACK-9602: Improve resource limits comprehension in listResources (#1554)
Add resource type name in request and response for listResources API call.
This adds in the response a new attribute typename with the String value for the corresponding resource enum.

    {
      "capacitytotal": 0,
      "capacityused": 0,
      "percentused": "0",
      "type": 19,
      "typename": "gpu",
      "zoneid": "381d0a95-ed4a-4ad9-b41c-b97073c1a433",
      "zonename": "ch-dk-2"
    }

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2017-12-28 15:46:24 +05:30
Abhinandan Prateek 391952da5b CLOUDSTACK-9867: VM snapshot on primary storage usage metrics (#2035)
VM snapshot on primary storage usage metrics.
2017-12-28 14:57:10 +05:30
Sigert Goeminne 26759d1d13 CLOUDSTACK-10189: Adding nuage VSD managed network support to CloudStack (#2360)
Exposing externalId en domainId field in the UI to CS users.

Co-Authored-By: Sigert Goeminne sigert.goeminne@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net
2017-12-28 14:55:15 +05:30
Sigert Goeminne d49765619d CLOUDSTACK-10024: Network migration support
Co-Authored-By: Frank Maximus frank.maximus@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net

New API’s:

* migrateNetwork
* migrateVpc
2017-12-21 11:25:17 +01:00
Bitworks Software, Ltd 80a6961b5d CLOUDSTACK-10168: VR duplicate entries in /etc/hosts when reusing VM name (#2366)
Bug is fixed. Fixed typo 'adress' to 'address' everywhere which leads to correct generation for agent communication json dhcp command.
2017-12-20 23:19:12 +05:30
Nicolas Vazquez 13c325aad4 CLOUDSTACK-10102: New network type L2 (#2281)
This feature allows CloudStack administrators to create layer 2 networks on CloudStack. As these networks are purely layer 2, they don't require IP addresses or Virtual Router, only VLAN is necessary (provided by administrator or assigned by CloudStack). Also, network services should be handled externally, e.g. DNS, DHCP, as they are not provided by L2 networks.
As a consequence, a new Guest Network type is created within CloudStack: L2

Description:
Network offerings and networks support new guest type: L2.
L2 Network offering creation allows administrator to select Specify VLAN or let CloudStack assign it dynamically.
L2 Network creation allows administrator to specify VLAN tag (if network offerings allows it) or simply create network.
VM deployments on L2 networks:
VMs should not IP addresses or any network service
No Virtual Router deployed on network
If Specify VLAN = true for network offering, network gets implemented using a dynamically assigned VLAN
UI changes

A new button is added on Networks tab, available for admins, to allow L2 networks creation
2017-12-20 17:07:39 +05:30
Nitin Kumar Maharana 8acb0908c4 CLOUDSTACK-9880: Expansion of Management IP Range. (#2048)
At present, The management IP range can only be expanded under the same subnet. According to existing range, either the last IP can be forward extended or the first IP can be backward extended. But we cannot add an entirely different range from the same subnet. So the expansion of range is subnet bound, which is fixed. But when the range gets exhausted and a user wants to deploy more system VMs, then the operation would fail. The purpose of this feature is to expand the range of management network IPs within the existing subnet. It can also delete and list the IP ranges.

Please refer the FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Expansion+of+Management+IP+Range
2017-12-20 08:36:53 +00:00
Yoan Blanc 6ab2ba9230
doc: replace virutal by virtual (typo)
Signed-off-by: Yoan Blanc <yoan.blanc@exoscale.ch>
2017-12-13 09:59:15 +01:00
Wido den Hollander 885f561791 CLOUDSTACK-8313: Support overprovisioning for local storage (#2356)
This used to work in earlier version but was broken during a earlier
refactor.

This commit restores the functionality, but is also adds a boolean
per storage pool type if it supports overprovisioning.

This removes the nasty if which hard codes it.

UnitTests are also added to this commit to verify this functionality
remains stable in future releases.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-12-12 20:57:21 +05:30
Wido den Hollander 632479d8f8 CLOUDSTACK-9853: Add support for Secondary IPv6 Addresses and Subnets (#2028)
This commit adds support for passing IPv6 Addresses and/or Subnets as
Secondary IPs.

This is groundwork for CLOUDSTACK-9853 where IPv6 Subnets have to be
allowed in the Security Groups of Instances to we can add DHCPv6
Prefix Delegation.

Use ; instead of : for separating addresses, otherwise it would cause
problems with IPv6 Addresses.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-22 17:30:33 +05:30
subhash yedugundla 421517f949 CLOUDSTACK-10133: Local storage overprovisioning for ext file system (#2310)
Local storage overprovisioning for ext file system
2017-11-22 17:23:34 +05:30
Sigert Goeminne 77864992fe CLOUDSTACK-9776: extra DHCP options support for Nuage VSP
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9776

Design-Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+extra+DHCP+option+support
2017-11-21 11:44:39 +01:00
Frank Maximus 14ab69e19a
Merge pull request #2005 from nuagenetworks/feature/nw_offering_for_vpc
CLOUDSTACK-9450: Network Offering for VPC based on DB flag
2017-11-16 17:33:07 +01:00
Abhinandan Prateek 4627fb2cd7 CLOUDSTACK-9972: Enhance listVolume API to include physical size and … (#2158)
* CLOUDSTACK-9972: Enhance listVolume API to include physical size and utilization.
Also fixed pool, cluster and pod info

* CLOUDSTACK-9972: Fix volume_view and duplicate API constant

* CLOUDSTACK-9972: Backport Do not allow vms to be deployed on hosts that are in disabled pod

* CLOUDSTACK-9972: Fix localization missing keys

* CLOUDSTACK-9972: Fix sql path
2017-11-05 21:44:43 +05:30
Frank Maximus d467a6b8ec CLOUDSTACK-9450: Network Offering for VPC based on DB flag
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-11-03 12:39:23 +01:00
Rohit Yadav 41fdb88970 CLOUDSTACK-10047: DVSwitch fixes and improvements (#2293)
Allow security policies to apply on port groups:
- Accepts security policies while creating network offering
- Deployed network will have security policies from the network offering
  applied on the port group (in vmware environment)
- Global settings as fallback when security policies are not defined for a network
  offering
- Default promiscuous mode security policy set to REJECT as it's the default
  for standard/default vswitch

Portgroup vlan-trunking options for dvswitch: This allows admins to define
a network with comma separated vlan id and vlan
range such as vlan://200-400,21,30-50 and use the provided vlan range to
configure vlan-trunking for a portgroup in dvswitch based environment.

VLAN overlap checks are performed for:
- isolated network against existing shared and isolated networks
- dedicated vlan ranges for the physical/public network for the zone
- shared network against existing isolated network

Allow shared networks to bypass vlan overlap checks: This allows admins
to create shared networks with a `bypassvlanoverlapcheck` API flag
which when set to 'true' will create a shared network without
performing vlan overlap checks against isolated network and against
the vlans allocated to the datacenter's physical network (vlan ranges).

Notes:
- No vlan-range overlap checks are performed when creating shared networks
- Multiple vlan id/ranges should include the vlan:// scheme prefix

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-25 15:14:42 +05:30
Mike Tutkowski 4c89b5b97a Merge branch 'pr-2081' 2017-10-18 14:25:45 -06:00
dahn a379230e8e CLOUDSTACK-9957 Annotations (#2181)
* annotations on hosts

* Adding marvin tests

* rebase error

* review comments

* context for owner

* review

* illegal entity test

* entityType check on input

* Annotation events

* rebase issues
2017-10-13 11:55:26 +02:00
Rohit Yadav 0fedbdd7a9 CLOUDSTACK-9998: Prometheus Exporter for CloudStack (#2287)
This implements a CloudStack Prometheus exporter as a plugin, that serves
metrics on a HTTP port.

New global settings:

1. prometheus.exporter.enable - (default: false), Enable the prometheus
exporter plugin, management server restart needed.
2. prometheus.exporter.port - (default: 9595), The prometheus exporter
server port.
3. prometheus.exporter.allowed.ips - (default: 127.0.0.1), List of comma
separated prometheus server ips (with no spaces) that should be allowed to
access the URLs.

The following list  of  metrics are provided  per pop (zone)  with  the exporter:
• Per host:
o CPU cores:  used, total
o CPU usage:  used, total (in MHz)
o Memory  usage:  used, total (in MiBs)
o Total VMs running on  the host
• CPU cores:  allocated (per  zone)
• CPU usage:  allocated (per  zone, in  MHz)
• Memory  usage:  allocated (per  zone, in  MiBs)
• Hosts:  online, offline,  total
• VMs: in all states -- starting, running, stopping, stopped, destroyed,
       expunging, migrating,  error, unknown
• Volumes:  ready,  destroyed,  total
• Primary Storage Pool: (Disk size) used, allocated,  unallocated,  total (in GiBs)
• Secondary Storage Pool: (Disk size) used, allocated,  unallocated,  total (in GiBs)
• Private IPs:  allocated,  total
• Public  IPs:  allocated,  total
• Shared  Network IPs:  allocated,  total
• VLANs:  allocated,  total

Additional metrics for the environment:
• Summed  domain  (level=1) limit for CPU cores
• Summed  domain  (level=1) limit for memory/ram

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-11 17:24:22 +05:30
Harika Punna 6bb0ca2f85 This feature separates the snapshot creation on primary and its backing up on secondary.
As part of this, a new parameter, which is optional, is added to CreateSnapshotCmd, which seperates the creation and backup.

More details in the FS-
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Separate+creation+and+backup+operations+for+a+volume+snapshot
2017-10-04 14:39:03 +05:30
Nathan Johnson 2bad9a6c11 CLOUDSTACK-9949: add ability to specify mac address (#2143)
Added ability to specify mac in deployVirtualMachine and
addNicToVirtualMachine api endpoints.

Validates mac address to be in the form of:
aa:bb:cc:dd:ee:ff , aa-bb-cc-dd-ee-ff , or aa.bb.cc.dd.ee.ff.

Ensures that mac address is a Unicast mac.

Ensures that the mac address is not already allocated for the
specified network.
2017-09-16 12:16:42 +05:30
Nathan Johnson 2ccea134ae CLOUDSTACK-10056: Fix vm details usage (#2248)
Fix bug where disk controller specified via vm details throws
a NumberFormatException, since "scsi" is not a number.
2017-09-16 12:01:03 +05:30
mrunalinikankariya d1e61eb3ff CLOUDSTACK-9812: Update 'updatePortForwardingRule' api to include additional parameter end port (#1985)
Configure a PF rule Private port : Start port ; 20 ENd POrt 25 || Public Port : Start port 20 ; ENd Port : 25.
Trigger UpdatePortForwardingRule api
ApI fails with following error : " Unable to update the private port of port forwarding rule as the rule has port range "

Solution-
Port range gets modified
2017-08-31 09:44:51 +02:00
Rohit Yadav 212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
SudharmaJain 7f987ace35 CLOUDSTACK-9165: unable to use reserved IP range in a network for external VMs (#1246) 2017-08-22 21:11:45 +02:00
dahn 597df24b53 CLOUDSTACK-10007: Isolation methods (#2193)
Change isolation methods from an enum to a registry based construct to enhance pluggability
2017-08-03 12:20:37 +02:00
Bharat Kumar d98520a6da CLOUDSTACK-9667 Enable resourcecount.check.interval by default 2017-07-24 09:26:17 +02:00
Wei Zhou 960cb84083 CLOUDSTACK-7984: Collect network statistics for VMs on shared network (KVM implementation) 2017-07-23 17:15:23 +02:00
Wei Zhou 088cca2b28 CLOUDSTACK-9997: Add cpu cores information in CapacityResponse
This commit contains following changes
(1) add CPU CORE information in op_host_capacity
(2) add capacity name in the CapacityResponse
(3) add allocatedCapacity for CPU/MEMORY/CPU CORE for zones
(4) sort CapacityResponse by zonename and CapacityType
2017-07-22 11:26:37 +02:00