Commit Graph

3542 Commits

Author SHA1 Message Date
Rohit Yadav 30175d6879
CLOUDSTACK-10132: Extend support for management servers LB for agents (#2469)
The new CA framework introduced basic support for comma-separated
list of management servers for agent, which makes an external LB
unnecessary.

This extends that feature to implement LB sorting algorithms that
sorts the management server list before they are sent to the agents.
This adds a central intelligence in the management server and adds
additional enhancements to Agent class to be algorithm aware and
have a background mechanism to check/fallback to preferred management
server (assumed as the first in the list). This is support for any
indirect agent such as the KVM, CPVM and SSVM agent, and would
provide support for management server host migration during upgrade
(when instead of in-place, new hosts are used to setup new mgmt server).

This FR introduces two new global settings:

- `indirect.agent.lb.algorithm`: The algorithm for the indirect agent LB.
- `indirect.agent.lb.check.interval`: The preferred host check interval
  for the agent's background task that checks and switches to agent's
  preferred host.

The indirect.agent.lb.algorithm supports following algorithm options:

- static: use the list as provided.
- roundrobin: evenly spreads hosts across management servers based on
  host's id.
- shuffle: (pseudo) randomly sorts the list (not recommended for production).

Any changes to the global settings - `indirect.agent.lb.algorithm` and
`host` does not require restarting of the mangement server(s) and the
agents. A message bus based system dynamically reacts to change in these
global settings and propagates them to all connected agents.

Comma-separated management server list is propagated to agents on
following cases:
- Addition of a host (including ssvm, cpvm systevms).
- Connection or reconnection by the agents to a management server.
- After admin changes the 'host' and/or the
  'indirect.agent.lb.algorithm' global settings.

On the agent side, the 'host' setting is saved in its properties file as:
`host=<comma separated addresses>@<algorithm name>`.

First the agent connects to the management server and sends its current
management server list, which is compared by the management server and
in case of failure a new/update list is sent for the agent to persist.

From the agent's perspective, the first address in the propagated list
will be considered the preferred host. A new background task can be
activated by configuring the `indirect.agent.lb.check.interval` which is
a cluster level global setting from CloudStack and admins can also
override this by configuring the 'host.lb.check.interval' in the
`agent.properties` file.

Every time agent gets a ms-host list and the algorithm, the host specific
background check interval is also sent and it dynamically reconfigures
the background task without need to restart agents.

Note: The 'static' and 'roundrobin' algorithms, strictly checks for the
order as expected by them, however, the 'shuffle' algorithm just checks
for content and not the order of the comma separate ms host addresses.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-03-15 16:34:03 +05:30
Rafael Weingärtner 972b8b71d7
CLOUDSTACK-8855 Improve Error Message for Host Alert State and reconnect host API. (#2387)
* CLOUDSTACK-8855 Improve Error Message for Host Alert State

* [CLOUDSTACK-9846] create column to save the content of alert messages

Remove declaration of throws CloudRuntimeException
I also removed some unused variables and comments left behind

This closes #837

* Isolate a problematic test "smoke/test_certauthority_root"
2018-03-14 15:27:43 -03:00
Nicolas Vazquez 74db647dbb CLOUDSTACK-10321: CPU Cap for KVM (#2482) 2018-03-14 18:21:24 +00:00
Raf Smeets 19d6578732 CLOUDSTACK-10303 : Refactor test data to nuage_test_data.py runnable against simulator (#2483)
* Refactored nuage tests

Added simulator support for ConfigDrive
Allow all nuage tests to run against simulator
Refactored nuage tests to remove code duplication

* Move test data from test_data.py to nuage_test_data.py

Nuage test data is now contained in nuage_test_data.py instead of
test_data.py
Removed all nuage test data from nuage_test_data.py

* CLOUD-1252 fixed cleanup of vpc tier network

* Import libVSD into the codebase

* CLOUDSTACK-1253: Volumes are not expunged in simulator

* Fixed some merge issues in test_nuage_vsp_mngd_subnets test

* Implement GetVolumeStatsCommand in Simulator

* Add vspk as marvin nuagevsp dependency, after removing libVSD dependency

* correct libVSD files for license purposes

pep8 pyflakes compliant
2018-03-14 17:17:36 +05:30
Rohit Yadav 82bcc74679 Merge branch '4.11' 2018-03-12 16:15:46 +05:30
Rohit Yadav da8cf8c370
CLOUDSTACK-10319: Prefer TLSv1.2, deprecate TLSv1.0,1.1 (#2480)
This deprecates and remove TLS 1.0 and 1.1 from preferred list of
protocols and keeps only TLSv1.2.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-03-12 11:43:59 +01:00
Rohit Yadav 68251e8440 Merge branch '4.11' 2018-03-08 12:19:59 +01:00
Rafael Weingärtner c3e5c09ac6
[CLOUDSTACK-10197] Update DisplayText of XenServer tools ISO entry in the database when it already exists (#2470)
* Update the displayText of XenServer ISO when it already exist in the DB

Besides updating the ISO display text, I also created unit test cases for 'createXenServerToolsIsoEntryInDatabase' and 'getActualIsoTemplate' methods.

* Formatting and cleanups for checkstyle of changed classes
2018-03-08 06:59:52 -03:00
Nicolas Vazquez c0b920f740 CLOUDSTACK-10274: L2 network refused to be designed on VXLAN physical network (#2448)
L2 network refused to be designed on VXLAN physical network. Add fix for vxlan issue.
Add condition for L2 networks which do not allow specifying vlan.
2018-03-06 14:56:47 +01:00
mrunalinikankariya bb607d07a9 CLOUDSTACK-10054:Volume download times out in 3600 seconds (#2244) 2018-02-27 13:33:00 +01:00
Daan Hoogland cdddecb41e Merge release branch 4.11 to master
* 4.11:
  CLOUDSTACK-10306: Upgrade to VMware 6.5 vim jar dependency (#2467)
  CLOUDSTACK-10298: fix for recreation of an earlier deleted Nuage managed network (#2460)
2018-02-27 12:42:41 +01:00
Sigert Goeminne 3dfbcbc48a CLOUDSTACK-10298: fix for recreation of an earlier deleted Nuage managed network (#2460)
* CLOUDSTACK-10298: Recreation of an earlier deleted Nuage managed isolated or vpc tier network fails

Added negative tests for feature vsd managed subnets
Added fixes for reacreation of an earlier deleted Nuage managed network
Updated Nuage Vsp CloudStack client to 1.0.8
Co-authored by: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>

* removed commented code as per review comment
2018-02-27 12:25:15 +01:00
Rohit Yadav 0ece15f86e Updating pom.xml version numbers for release 4.11.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-02-26 16:57:48 +01:00
Rohit Yadav 3db1746c03 Merge branch '4.11' 2018-02-26 15:32:20 +01:00
Nicolas Vazquez 703054964a CLOUDSTACK-10305: Rare race condition in KVM migration (#2466)
There is a race condition in the monitoring of the migration process on KVM. If the monitor wakes up in the tight window after the migration succeeds, but before the migration thread terminates, the monitor will get a LibvirtException “Domain not found: no domain with matching uuid” when checking on the migration status. This in turn causes CloudStack to sync the VM state to stop, in which it issues a defensive StopCommand to ensure it is correctly synced.

Fix: Prevent LibvirtException: "Domain not found" caused by the call to dm.getInfo()
2018-02-26 15:31:51 +01:00
Rafael Weingärtner e532b574dd Forward merge #2402 merged on '4.11' branch
[CLOUDSTACK-10128] Simplify the search for hosts to execute commands
2018-02-19 18:05:00 -03:00
Rafael Weingärtner 264036bd2c
Merge pull request #2402 from rafaelweingartner/fixCLOUDSTACK-10128
[CLOUDSTACK-10128] Simplify the search for host to execute command
2018-02-19 17:54:01 -03:00
Rohit Yadav f96398c127 Merge branch '4.11' 2018-02-14 11:56:00 +01:00
Frank Maximus dee5bb527e CLOUDSTACK-10261: Libvirt metadata create only one nuage-extension xml tag (#2441) 2018-02-12 12:06:44 +01:00
Rohit Yadav b88681de18 Merge branch '4.11' 2018-02-10 18:32:46 +01:00
Sigert Goeminne 8949efe8d1 CLOUDSTACK-10218: Fix for forced network update in a nuage network (#2445)
Fix for forced network update to a nuage network offering with vr fails with IllegalArgumentException.
Addressed review comments DaanHoogland.
2018-02-10 18:24:01 +01:00
Rohit Yadav 1c26b2ec2e Merge remote-tracking branch 'origin/4.11'
CLOUDSTACK-10269: On deletion of role set name to null (#2444)
CLOUDSTACK-10146 checksum in java instead of script (#2405)
CLOUDSTACK-10222: Clean snaphosts from primary storage when taking (#2398)

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-02-09 11:53:20 +01:00
Khosrow Moossavi 94776fbfd0 CLOUDSTACK-10222: Clean snaphosts from primary storage when taking new (#2398)
When user creates a snapshot (manual or recurring), snapshot remains on
the primary storage, even if the snapshot is transferred successfully to
secondary storage. This is causing issues because XenServer can only hold
a limited number of snapshots in its VDI chain, preventing the user from
creating new snapshots after some time, when too many old snapshots are
present on the primary storage.
2018-02-06 20:09:40 +01:00
Rohit Yadav 6ffbce6159 Updating pom.xml version numbers for release 4.11.0.1-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-02-05 11:13:50 +01:00
Rohit Yadav 5dada1f7ed Updating pom.xml version numbers for release 4.11.0.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-26 13:13:37 +01:00
Rohit Yadav d6ce39b29b Merge branch '4.11'
- CLOUDSTACK-10239: Fallback to default provider if needed (#2430)
- CLOUDSTACK-10255: Fix agent logrotate (#2429)
2018-01-26 13:11:58 +01:00
dahn 6aadbc5219 CLOUDSTACK-10239: Fallback to default provider if needed (#2430)
Fallback to default provider if needed.
2018-01-26 13:03:11 +01:00
Rohit Yadav bc1b5fb98f Merge branch '4.11' 2018-01-24 13:11:55 +01:00
Rohit Yadav 170b6ce20d
CLOUDSTACK-10236: Enable dynamic roles for missing props file (#2426)
Automate dynamic roles migration for missing props file

- In case commands.properties file is missing, enables dynamic roles.
- Adds a new -D or --default flag to migrate-dynamicroles.py script
  to simply update the global setting and use the default role-rule
  permissions.
- Add warning message, ask admins to move to dynamic roles during upgrade

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-24 13:11:08 +01:00
Rafael Weingärtner c591c5ad3e CLOUDSTACK-10248: Fix errors that appeared after #2283 (#2417)
This fixes move refactoring error introduced in #2283 
For instance, the class DatadiskTO is supposed to be in com.cloud.agent.api.to package. However, the folder structure it was placed in is com.cloud.agent.api.api.to.

Skip tests for cloud-plugin-hypervisor-ovm3:
For some unknown reason, there are quite a lot of broken test cases for cloud-plugin-hypervisor-ovm3. They might have appeared after some dependency upgrade and was overlooked by the person updating them. I checked them to see if they could be fixed, but these tests are not developed in a clear and clean manner. On top of that, we do not see (at least I) people using OVM3-hypervisor with ACS. Therefore, I decided to skip them.

Identention corrected to use spaces instead of tabs in XML files
2018-01-23 12:19:36 +01:00
Rohit Yadav 7e39961f22 Merge branch '4.11'
- CLOUDSTACK-10244: Cache whether to migrate storage

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-22 09:05:26 +01:00
Mike Tutkowski 2ca264ca4d CLOUDSTACK-10244: Cache whether to migrate storage (#2416)
KVM online storage migration fails, this fixes use of whether to migrate storage using a cached variable.
2018-01-21 15:49:54 +05:30
Marc-Aurèle Brothier 893a88d225 CLOUDSTACK-10105: Use maven standard project structure in all projects (#2283)
Remove maven standard module (which only a few were using) and get ride of maven customization for the projects structure.

- moved all directories to src/main/java, src/main/resources, src/main/scripts, src/test/java, src/test/resources
- grep scan to search for src/com and src/org left over
- grep for <project>/scripts to fix pom.xml configuration
- remove custom <build> configuration in pom.xml

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2018-01-20 03:19:27 +05:30
Rohit Yadav f7f14bfcc5 Merge branch '4.11' 2018-01-19 13:46:58 +05:30
Frank Maximus cfff78d9a1 CLOUDSTACK-10233: Use namespace in Libvirt domain metadata. (#2410)
The documentation of Libvirt specifies the requirement of using an XML namespace,
when having metadata in the Domain XML. The Nuage extenstion metadata was not
adhering to this specification, and the lastest Libvirt version ignores it in that case.
2018-01-19 13:44:16 +05:30
Rohit Yadav 072dbc0720 Updating pom.xml version numbers for master to 4.12.0.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-15 17:43:45 +05:30
Mike Tutkowski a30a31c9b7 CLOUDSTACK-9620: Enhancements for managed storage (#2298)
Allowed zone-wide primary storage based on a custom plug-in to be added via the GUI in a KVM-only environment (previously this only worked for XenServer and VMware)

Added support for root disks on managed storage with KVM

Added support for volume snapshots with managed storage on KVM

Enable creating a template directly from a volume (i.e. without having to go through a volume snapshot) on KVM with managed storage

Only allow the resizing of a volume for managed storage on KVM if the volume in question is either not attached to a VM or is attached to a VM in the Stopped state.

Included support for Reinstall VM on KVM with managed storage

Enabled offline migration on KVM from non-managed storage to managed storage and vice versa

Included support for online storage migration on KVM with managed storage (NFS and Ceph to managed storage)

Added support to download (extract) a managed-storage volume to a QCOW2 file

When uploading a file from outside of CloudStack to CloudStack, set the min and max IOPS, if applicable.

Included support for the KVM auto-convergence feature

The compression flag was actually added in version 1.0.3 (1000003) as opposed to version 1.3.0 (1003000) (changed this to reflect the correct version)

On KVM when using iSCSI-based managed storage, if the user shuts a VM down from the guest OS (as opposed to doing so from CloudStack), we need to pass to the KVM agent a list of applicable iSCSI volumes that need to be disconnected.

Added a new Global Setting: kvm.storage.live.migration.wait

For XenServer, added a check to enforce that only volumes from zone-wide managed storage can be storage motioned from a host in one cluster to a host in another cluster (cannot do so at the time being with volumes from cluster-scoped managed storage)

Don’t allow Storage XenMotion on a VM that has any managed-storage volume with one or more snapshots.

Enabled for managed storage with VMware: Template caching, create snapshot, delete snapshot, create volume from snapshot, and create template from snapshot

Added an SIOC API plug-in to support VMware SIOC

When starting a VM that uses managed storage in a cluster other than the one it last was running in, we need to remove the reference to the iSCSI volume from the original cluster.

Added the ability to revert a volume to a snapshot

Enabled cluster-scoped managed storage

Added support for VMware dynamic discovery
2018-01-15 00:05:52 +05:30
Frank Maximus b176648f90 CLOUDSTACK-9813: Extending Config Drive support (#2097)
Extending Config Drive support

* Added support for VMware
* Build configdrive.iso on ssvm
* Added support for VPC and Isolated Networks
* Moved implementation to new Service Provider
* UI fix: add support for urlencoded userdata
* Add support for building systemvm behind a proxy

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
2018-01-12 15:14:40 +05:30
Rafael Weingärtner 1c8bbddb04 [CLOUDSTACK-10128] Simplify the search for host to execute command 2018-01-11 14:36:35 -02:00
Abhinandan Prateek 64832fd70a CLOUDSTACK-4757: Support OVA files with multiple disks for templates (#2146)
CloudStack volumes and templates are one single virtual disk in case of XenServer/XCP and KVM hypervisors since the files used for templates and volumes are virtual disks (VHD, QCOW2). However, VMware volumes and templates are in OVA format, which are archives that can contain a complete VM including multiple VMDKs and other files such as ISOs. And currently, Cloudstack only supports Template creation based on OVA files containing a single disk. If a user creates a template from a OVA file containing more than 1 disk and launches an instance using this template, only the first disk is attached to the new instance and other disks are ignored.
Similarly with uploaded volumes, attaching an uploaded volume that contains multiple disks to a VM will result in only one VMDK to being attached to the VM.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Support+OVA+files+containing+multiple+disks

This behavior needs to be improved in VMWare to support OVA files with multiple disks for both uploaded volumes and templates. i.e. If a user creates a template from a OVA file containing more than 1 disk and launches an instance using this template, the first disk should be attached to the new instance as the ROOT disk and volumes should be created based on other VMDK disks in the OVA file and should be attached to the instance.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-10 22:10:41 +05:30
Rohit Yadav b0d7844cf0
CLOUDSTACK-10109: Fix regression from PR #2295 (#2394)
This fixes regression introduced in PR #2295:
- Pass assign=true to fetch new public IP
- Use wait_until instead of sleep+wait in tests
- Loop through list of public IP ranges to match the systemvm gateway
- Fix potential NPE seen when adding simulator host(s)
- Removes aria2 installation from setup_agent.sh using yum, it's already
  dependency for cloudstack-agent package

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-10 00:44:00 +05:30
Nicolas Vazquez e86bb41e0e CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates (#2379)
This feature allows using templates and ISOs avoiding secondary storage as intermediate cache on KVM. The virtual machine deployment process is enhanced to supported bypassed registered templates and ISOs, delegating the work of downloading them to primary storage to the KVM agent instead of the SSVM agent.

Template and ISO registration:
- When hypervisor is KVM, a checkbox is displayed with 'Direct Download' label.
- API methods registerTemplate and registerISO are both extended with this new parameter directdownload.
- On template or ISO registration, no download job is sent to SSVM agent, CloudStack would only persist an entry on template_store_ref indicating that template or ISO has been marked as 'Direct Download' (bypassing Secondary Storage). These entries are persisted as:
template_id = Template or ISO id on vm_template table
store_id NULL
download_state = BYPASSED
state = Ready
(Note: these entries allow users to deploy virtual machine from registered templates or ISOs)
- An URL validation command is sent to a random KVM host to check if template/ISO location can be reached. Metalink are also supported by this feature. In case of a metalink, it is fetched and URL check is performed on each of its URLs.
- Checksum should be provided as indicated on #2246: {ALGORITHM}CHKSUMHASH
- After template or ISO is registered, it would be displayed in the UI

Virtual machine deployment:
When a 'Direct Download' template is selected for deployment, CloudStack would delegate template downloading to destination storage pool via destination host by a new pluggable download manager.
Download manager would handle template downloading depending on URL protocol. In case of HTTP, request headers can be set by the user via vm_template_details. Those details should be persisted as:
Key: HTTP_HEADER
Value: HEADERNAME:HEADERVALUE

In case of HTTPS, a new API method is added uploadTemplateDirectDownloadCertificate to allow user importing a client certificate into all KVM hosts' keystore before deployment.
After template or ISO is downloaded to primary storage, usual entry would be persisted on template_spool_ref indicating the mapping between template/ISO and storage pool.
2018-01-09 12:22:18 +05:30
koushik-das 4d7a9d82cc CLOUDSTACK-9892: Primary storage resource check is broken when using root disk size override to deploy VM (#2088)
This happens when the root disk size is overridden. The primary storage limit check should be performed based on overridden size instead of template size. Enabled root disk resize tests to run on simulator as well.
2018-01-07 16:18:58 +05:30
Khosrow Moossavi 25d7d741a7 CLOUDSTACK-10197: Rename xentools iso for XenServer 7.0+ (#2365)
The xentools iso has been renamed from xs-tools to guest-tools
starting from XenServer 7.0.
2018-01-07 16:00:39 +05:30
Marc-Aurèle Brothier 92a6bc27ff CLOUDSTACK-9896: listDedicatedXXX should respect pagination (#2073)
Fixes listDedicatedxxx APIs to respect pagination options.
2018-01-07 15:43:27 +05:30
Nicolas Vazquez 90ef67bab9 CLOUDSTACK-10109: Enable dedication of public IPs to SSVM and CPVM (#2295)
This feature allow admins to dedicate a range of public IP addresses to the SSVM and CPVM, such that they can be subject to specific external firewall rules. The option to dedicate a public IP range to the System VMs (SSVM & CPVM) is added to the createVlanIpRange API method and the UI.

Solution:
Global setting 'system.vm.public.ip.reservation.mode.strictness' is added to determine if the use of the system VM reservation is strict (when true) or preferred (false), false by default.
When a range has been dedicated to System VMs, CloudStack should apply IPs from that range to
the public interfaces of the CPVM and the SSVM depending on global setting's value:

If the global setting is set to false: then CloudStack will use any unused and unreserved public IP
addresses for system VMs only when the pool of reserved IPs has been exhausted
If the global setting is set to true: then CloudStack will fail to deploy the system VM when the pool
of reserved IPs has been exhausted, citing the lack of available IPs.
UI Changes
Under Infrastructure -> Zone -> Physical Network -> Public -> IP Ranges, button 'Account' label is refactored to 'Set reservation'.

When that button is clicked, dialog displayed is also refactored, including a new checkbox 'System VMs' which indicates if range should be dedicated for CPVM and SSVM, and a note indicating its usage.

When clicking on button for any created range, UI dialog displayed indicates whether IP range is dedicated for system vms or not.
2018-01-06 23:14:30 +05:30
dahn 45df928e04 CLOUDSTACK-10117: Account ldap binding (#2381)
Map an ldap group to an account. Ldap related settings on a domain level.
2018-01-06 22:58:28 +05:30
Rafael Weingärtner 1d0f2128f6 [CLOUDSTACK-9025][CLOUDSTACK-10128] solve problems of templates created based on snapshots (#2315)
The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only tackling the problem for CloudStack deployments that use single hypervisor types (restricted to XenServer). Additionally, the lack of information regarding that solution (poor documentation, test cases and description in PRs and Jira ticket) led the code to be removed in #1124 after a long discussion and analysis in #1056. That piece of code seemed logicless (and it was!).  It would receive a hostId and then change that hostId for other hostId of the zone without doing any check; it was not even checking the hypervisor and storage in which the host was plugged into.

The problem reported in #CLOUDSTACK-9025 is caused by partial snapshots that are taken in XenServer. This means, we do not take a complete snapshot, but a partial one that contains only the modified data. This requires rebuilding the VHD hierarchy when creating a template out of the snapshot. The point is that the first hostId received is not a hostId, but a system VM ID(SSVM). That is why the code in #1176 fixed the problem for some deployment scenarios, but would cause problems for scenarios where we have multiple hypervisors in the same zone. We need to execute the creation of the VHD that represents the template in the hypervisor, so the VHD chain can be built using the parent links.

This commit changes the method com.cloud.hypervisor.XenServerGuru.getCommandHostDelegation(long, Command). From now on we replace the hostId that is intended to execute the “copy command” that will create the VHD of the template according to some conditions that were already in place. The idea is that starting with XenServer 6.2.0 hotFix ESP1004 we need to execute the command in the hypervisor host and not from the SSVM. Moreover, the method was improved making it readable and understandable; it was also created test cases assuring that from XenServer 6.2.0 hotFix ESP1004 and upward versions we change the hostId that will be used to execute the “copy command”.

Furthermore, we are not selecting a random host from a zone anymore. A new method was introduced in the HostDao called “findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using this method we look for a host that is in the cluster that is using the storage pool where the volume from which the Snaphost is taken of. By doing this, we guarantee that the host that is connected to the primary storage where all of the snapshots parent VHDs are stored is used to create the template.

Consider using Disabled hosts when no Enabled hosts are found

This also closes #2317
2018-01-06 09:55:29 +01:00
jayakarteek 053b12c813 CLOUDSTACK-9908: Primary Storage allocated capacity goes very high after VM snapshot (#2104)
Added '-' chracter while reading Filenames so that filename match exactly with VM name.
2018-01-05 11:19:01 +05:30
dahn 8131bccd97 CLOUDSTACK-10205: LinkDomainToLdap returns UUID instead of internal id (#2378)
The internal id is not usefull to the user. It is a bug to return it instead of a uuid.
In the process of fixing the above "name" was deprecated in favour of "ldap_domain".
2018-01-04 13:26:21 +05:30
Wido den Hollander 9988c269b2 CLOUDSTACK-7958: Add configuration for limit to CIDRs for Admin API calls (#2046)
* Cleanup and Improve NetUtils

This class had many unused methods, inconsistent names and redundant code.

This commit cleans up code, renames a few methods and constants.

The global/account setting 'api.allowed.source.cidr.list' is set
to 0.0.0.0/0,::/0 by default preserve the current behavior and thus
allow API calls for accounts from all IPv4 and IPv6 subnets.

Users can set it to a comma-separated list of IPv4/IPv6 subnets to
restrict API calls for Admin accounts to certain parts of their network(s).

This is to improve Security. Should an attacker steal the Access/Secret key
of an account he/she still needs to be in a subnet from where accounts are
allowed to perform API calls.

This is a good security measure for APIs which are connected to the public internet.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-01-04 10:56:30 +05:30
Frank Maximus c436bc3ef9 CLOUDSTACK-10024: Network Migration (#2374)
Handle PR comments

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Kris Sterckx <kris.sterckx@nuagenetworks.net>
2018-01-03 13:50:19 +05:30
dahn 6724a47122
CLOUDSTACK-10121 moveUser (#2301)
* internal service call for moveUser
* expose moveUser as API
* move uuid to external entity
2017-12-30 11:40:14 +00:00
Sigert Goeminne 26759d1d13 CLOUDSTACK-10189: Adding nuage VSD managed network support to CloudStack (#2360)
Exposing externalId en domainId field in the UI to CS users.

Co-Authored-By: Sigert Goeminne sigert.goeminne@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net
2017-12-28 14:55:15 +05:30
WingKai Ho d95ba438f6 CLOUDSTACK-10208: Fix snmptrap alert bug (#2045)
Fix snmptrap alerts info omit to send the field of SnmpConstants.sysUpTime
Fix SnmpConstants.sysUpTime expression
Remove unused import java.util.Date
2017-12-27 13:27:16 +05:30
Rohit Yadav e0d5364b70
CLOUDSTACK-9456: Upgrade spring and misc dependencies (#2375)
Upgrades spring, jetty and misc dependencies

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-26 11:53:48 +05:30
Rohit Yadav d19629a115 CLOUDSTACK-10013: Fixes based on code review and test failures
This includes test related fixes and code review fixes based on
reviews from @rafaelweingartner, @marcaurele, @wido and @DaanHoogland.

This also includes VMware disk-resize limitation bug fix based on comments
from @sateesh-chodapuneedi and @priyankparihar.

This also includes the final changes to systemvmtemplate and fixes to
code based on issues found via test failures.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 17:51:42 +05:30
Rohit Yadav ce3303212b CLOUDSTACK-9953: Resize root disk for VMware when full clone is enabled
Resize for VMware root disk should only be performed during VM start
when vmware.create.full.clone is true i.e. the disk chain length is one.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav 85aee8d18d CLOUDSTACK-10013: SystemVM codebase refactorings and improvements
- Refactors and simplifies systemvm codebase file structures keeping
  the same resultant systemvm.iso packaging
- Password server systemd script and new postinit script that runs
  before sshd starts
- Fixes to keepalived and conntrackd config to make rVRs work again
- New /etc/issue featuring ascii based cloudmonkey logo/message and
  systemvmtemplate version
- SystemVM python codebase linted and tested. Added pylint/pep to
  Travis.
- iptables re-application fixes for non-VR systemvms.
- SystemVM template build fixes.
- Default secondary storage vm service offering boosted to have 2vCPUs
  and RAM equal to console proxy.
- Fixes to several marvin based smoke tests, especially rVR related
  tests. rVR tests to consider 3*advert_int+skew timeout before status
  is checked.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Remi Bergsma 551e11cf3e CLOUDSTACK-10013: Make the generated VR/json files unique (ports #1470)
This ports PR #1470 by @remibergsma.

Make the generated json files unique to prevent concurrency issues:
The json files now have UUIDs to prevent them from getting overwritten
before they've been executed. Prevents config to be pushed to the wrong
router.

2016-02-25 18:32:23,797 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) (logid:) Seq 2-4684025087442026584: Processing:  { Ans: , MgmtId: 90520732674657, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupA
nswer":{"results":["null - success: null","null - success: [INFO] update_config.py :: Processing incoming file => vm_dhcp_entry.json.4ea45061-2efb-4467-8eaa-db3d77fb0a7b\n[INFO] Processing JSON file vm_dhcp_entry.json.4ea4506
1-2efb-4467-8eaa-db3d77fb0a7b\n"],"result":true,"wait":0}}] }

On the router:
2016-02-25 18:32:23,416  merge.py __moveFile:298 Processed file written to /var/cache/cloud/processed/vm_dhcp_entry.json.4ea45061-2efb-4467-8eaa-db3d77fb0a7b.gz

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav a7b233c5dc Merge branch '4.10' 2017-12-23 09:20:24 +05:30
Rohit Yadav bb7d206dab Merge branch '4.9' into 4.10 2017-12-23 09:19:01 +05:30
Frank Maximus c12e1386d3 CLOUDSTACK-10127: For OVS to work use dev/nic id based on mac address for KVM (#2304)
This uses mac address based referencing for nic/dev id for KVM. This fixes openvswitch (ovs) regression.
2017-12-23 09:17:45 +05:30
Rohit Yadav 593ed59bcd
CLOUDSTACK-10173: Pick network rate from guest offering for VRs (#2351)
Per the following doc, VR's guest/public nic should pick up network
rate from its network offering:
http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/service_offerings.html#network-throttling

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-21 17:50:40 +05:30
Sigert Goeminne d49765619d CLOUDSTACK-10024: Network migration support
Co-Authored-By: Frank Maximus frank.maximus@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net

New API’s:

* migrateNetwork
* migrateVpc
2017-12-21 11:25:17 +01:00
Nicolas Vazquez 13c325aad4 CLOUDSTACK-10102: New network type L2 (#2281)
This feature allows CloudStack administrators to create layer 2 networks on CloudStack. As these networks are purely layer 2, they don't require IP addresses or Virtual Router, only VLAN is necessary (provided by administrator or assigned by CloudStack). Also, network services should be handled externally, e.g. DNS, DHCP, as they are not provided by L2 networks.
As a consequence, a new Guest Network type is created within CloudStack: L2

Description:
Network offerings and networks support new guest type: L2.
L2 Network offering creation allows administrator to select Specify VLAN or let CloudStack assign it dynamically.
L2 Network creation allows administrator to specify VLAN tag (if network offerings allows it) or simply create network.
VM deployments on L2 networks:
VMs should not IP addresses or any network service
No Virtual Router deployed on network
If Specify VLAN = true for network offering, network gets implemented using a dynamically assigned VLAN
UI changes

A new button is added on Networks tab, available for admins, to allow L2 networks creation
2017-12-20 17:07:39 +05:30
pedro-martins 7cb4551d75 CLOUDSTACK-10198: removed unused (#1437)
com.cloud.hypervisor.hyperv.resource.HypervDummyResourceBase
class and change the log message in
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2017-12-19 12:24:25 +05:30
kishankavala 49be7eecd2 CLOUDSTACK-9800: Enabled netscaler inline mode. Added corrsponding changes in UI (#1964)
NetScaler can work in inline mode also along with other Firewall devices. Enabled netscaler inline mode capability
2017-12-19 04:18:55 +05:30
Rafael Weingärtner 3c6df7c970 CLOUDSTACK-10196: Remove ejb-api 3.0 dependency (#2348)
Fixed QuotaAlertManagerImplTest, which was injecting mock objects manually.
2017-12-18 23:59:06 +05:30
ming416 973a9c1dce CLOUDSTACK-10003 automatic configure juniper srx/vsrx nat loopback, (#2184)
Automatic configure juniper srx/vsrx nat loopback,

Constraint condition that manual configure source nat in juniper srx,allowed vm vist public network :
zone : trust to {trust,untrust}
rule : source address {0.0.0.0/0},destination address{0.0.0.0/0} ,do source nat with pool {public network getway ip}.

code change for trust to trust destination or static nat:

1. add srxCommand :CHECK_PRIVATE_IF_EXISTS. for add/delete rule to detect whether exist or not contain DestinationNatRule or StaticNatRule (ruleName_private) in trust zone .
2. add DestinationNatRule  (ruleName_private) to trust zone when ADD DestinationNatRule to untrust .
3. delete DestinationNatRule  (ruleName_private) from trust zone when DELETE DestinationNatRule from untrust.
4. add StaticNatRule (ruleName_private) to trust zone when ADD  StaticNatRule to untrust .
5. delete  StaticNatRule (ruleName_private) from trust zone when DELETE  StaticNatRule from untrust.

Check private if exist.
2017-12-16 09:51:25 +05:30
Rohit Yadav fc49533a8e
CLOUDSTACK-9986: Use host table to iterate for zone/cluster metrics (#2353)
The host_view may contain duplicate entries when hosts have tags.
Changing the host_view may cause unseen regressions so
to fix the issues we've modified the zone/cluster metrics code to use
the `host` table (hostdao) to iterate through the list of hosts in a
cluster during zone/cluster metrics listing.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-15 17:17:17 +05:30
Rafael Weingärtner bac423fea0 CLOUDSTACK-10184: Re-work method QuotaResponseBuilderImpl.startOfNextDay (#2326)
Also, removed @Local annotation that is not needed and add test cases.
2017-12-11 22:32:23 +05:30
Wido den Hollander be3a39ea4f CLOUDSTACK-10177: Only pass IPv6 address to Security Group Python script if present (#2355)
Otherwise we send down a 'null' to a ProcessBuilder in Java instead of a String and this
causes a NPE.

We should check first if the Instance has a IPv6 address before sending it there.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-12-11 21:47:50 +05:30
Sigert Goeminne 8f35657ac2 CLOUDSTACK-10164: allow users to create a VPC through the UI. (#2345)
The listNuageVspDomainTemplat cmd was called by the ui when the create vpc dialog was opened. This command failed when no nuage vsp device was present. As a consequence the ui did not show the dialog. So currently it's not possible to create a vpc through the UI with a native CloudStack deployment. This bugfix, adds robustness to the ui that in case the cmd fails the dialog will appear nonetheless . Furthermore I also changed the listNuageVspDomainTemplateCmd to always return an empty result when there is no nuage zone.
2017-12-01 02:39:21 +05:30
Wido den Hollander fdf2509060 CLOUDSTACK-10160: Fix typo in Libvirt XML definition for Virtio-SCSI (#2341)
* CLOUDSTACK-10160: Fix typo in Libvirt XML definition for Virtio-SCSI

The attribute for the XML element 'controller' should be 'model' and
not 'mode'.

Source: https://libvirt.org/formatdomain.html#elementsControllers

  A scsi controller has an optional attribute model, which is one of
  'auto', 'buslogic', 'ibmvscsi', 'lsilogic', 'lsisas1068', 'lsisas1078',
  'virtio-scsi' or 'vmpvscsi'.

In the current state a regular SCSI device is attached and not a Virtio-SCSI
device.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* CLOUDSTACK-10160: Add UnitTest for LibvirtVMDef.SCSIDef

To make sure the XML output string is correct

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-28 09:42:15 +05:30
Rafael Weingärtner cb9c7adddf CLOUDSTACK-10156: Fix Coverity new problems CID(1349987, 1349986, 1347248) (#2332)
Fix Coverity new problems CID(1349987, 1349986, 1347248).
2017-11-23 11:40:53 +05:30
Wido den Hollander 632479d8f8 CLOUDSTACK-9853: Add support for Secondary IPv6 Addresses and Subnets (#2028)
This commit adds support for passing IPv6 Addresses and/or Subnets as
Secondary IPs.

This is groundwork for CLOUDSTACK-9853 where IPv6 Subnets have to be
allowed in the Security Groups of Instances to we can add DHCPv6
Prefix Delegation.

Use ; instead of : for separating addresses, otherwise it would cause
problems with IPv6 Addresses.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-22 17:30:33 +05:30
Sigert Goeminne 77864992fe CLOUDSTACK-9776: extra DHCP options support for Nuage VSP
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9776

Design-Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+extra+DHCP+option+support
2017-11-21 11:44:39 +01:00
Marc-Aurèle Brothier cd6288ecfb CLOUDSTACK-10012: Jetty 9.4 (#2329)
* Bump Jetty to 9.4

* Use new jetty gzip handler

* Redirect / to context

* Update wiremock but still not working

* Add session timeout configuration

* server.properties.in: Change default timeout to 30 (mins)

* cloudian: fix unit test failures

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* client: use older 9.2.x jetty-maven-plugin that works

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Moving jetty mvn plugin version in properties

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>

* Set default session timeout to 30mins
2017-11-17 23:26:02 +05:30
Frank Maximus 3eafd0ce52
Merge pull request #1981 from nuagenetworks/feature/nuage_vpc_selectable_domain_template
CLOUDSTACK-9806: Nuage domain template selection per VPC
2017-11-17 11:31:48 +01:00
Frank Maximus 14ab69e19a
Merge pull request #2005 from nuagenetworks/feature/nw_offering_for_vpc
CLOUDSTACK-9450: Network Offering for VPC based on DB flag
2017-11-16 17:33:07 +01:00
Rohit Yadav 4fda0c039e
CLOUDSTACK-9998: Fix metrics generation failurein prometheus exporter (#2325)
This fixes metrics generation failures when metrics capacities are null.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-16 10:13:53 +05:30
Sigert Goeminne 880e475f40 CLOUDSTACK-9806: Nuage domain template selection per VPC
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9806

Design Document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nuage+domain+template+selection+per+VPC
2017-11-15 14:40:30 +01:00
Abhinandan Prateek 4627fb2cd7 CLOUDSTACK-9972: Enhance listVolume API to include physical size and … (#2158)
* CLOUDSTACK-9972: Enhance listVolume API to include physical size and utilization.
Also fixed pool, cluster and pod info

* CLOUDSTACK-9972: Fix volume_view and duplicate API constant

* CLOUDSTACK-9972: Backport Do not allow vms to be deployed on hosts that are in disabled pod

* CLOUDSTACK-9972: Fix localization missing keys

* CLOUDSTACK-9972: Fix sql path
2017-11-05 21:44:43 +05:30
Rohit Yadav eda3b35bfa CLOUDSTACK-10012: Migrate to Embedded Jetty
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
  deprecates the need to create/store cloud.jks in conf folder and
  the db.cloud.keyStorePassphrase in db.properties file. This also
  remove the need of the --keystore-passphrase in the
  cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-03 23:57:25 +05:30
Frank Maximus d467a6b8ec CLOUDSTACK-9450: Network Offering for VPC based on DB flag
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-11-03 12:39:23 +01:00
Frank Maximus 2139dbe803
Merge pull request #2238 from nuagenetworks/feature/caching_and_statistics
[CLOUDSTACK-10053] Performance improvement: caching of NuageVsp ID
2017-11-02 11:13:15 +01:00
Sigert Goeminne 0871ff9eda CLOUDSTACK-10053: Performance improvement: caching of NuageVsp ID
* VSP ID Caching
* VSP call Statistics
* 5.0 Support

Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-10-31 17:03:11 +01:00
Rohit Yadav 41fdb88970 CLOUDSTACK-10047: DVSwitch fixes and improvements (#2293)
Allow security policies to apply on port groups:
- Accepts security policies while creating network offering
- Deployed network will have security policies from the network offering
  applied on the port group (in vmware environment)
- Global settings as fallback when security policies are not defined for a network
  offering
- Default promiscuous mode security policy set to REJECT as it's the default
  for standard/default vswitch

Portgroup vlan-trunking options for dvswitch: This allows admins to define
a network with comma separated vlan id and vlan
range such as vlan://200-400,21,30-50 and use the provided vlan range to
configure vlan-trunking for a portgroup in dvswitch based environment.

VLAN overlap checks are performed for:
- isolated network against existing shared and isolated networks
- dedicated vlan ranges for the physical/public network for the zone
- shared network against existing isolated network

Allow shared networks to bypass vlan overlap checks: This allows admins
to create shared networks with a `bypassvlanoverlapcheck` API flag
which when set to 'true' will create a shared network without
performing vlan overlap checks against isolated network and against
the vlans allocated to the datacenter's physical network (vlan ranges).

Notes:
- No vlan-range overlap checks are performed when creating shared networks
- Multiple vlan id/ranges should include the vlan:// scheme prefix

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-25 15:14:42 +05:30
Rohit Yadav b6dc40faef CLOUDSTACK-10103: Cloudian Connector for CloudStack (#2284)
Several organizations use Cloudian as S3 provider, this implements the
Cloudian Management Console connector for CloudStack that can do the
following:

- Provide ease in connector configuration using CloudStack global
  settings
- Perform SSO from CloudStack UI into Cloudian Management Console (CMC)
  when the connector is enabled
- Automatic provisioning and de-provisioning of CloudStack accounts and
  domains as Cloudian users and groups respectively
- During CloudStack UI logout, logout user from CMC
- CloudStack account will be mapped to Cloudian Users, and CloudStack
  domain will be mapped to Cloudian Groups.
- The CloudStack admin account is mapped to Cloudian admin (user name
  configurable).
- The user/group provisioning will be from CloudStack to Cloudian only,
  i.e. user/group addition/removal/updation/deactivation in Cloudian
  portal (CMC) won't propagate the changes to CloudStack.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Cloudian+Connector+for+CloudStack

New APIs:
- `cloudianIsEnabled`: API to check whether Cloudian Connector is enabled.
- `cloudianSsoLogin`: Performs SSO for the logged-in, requesting user
                      and returns the URL that can be used to perform
                      SSO and log into CMC.

New Global Settings:
- cloudian.connector.enabled  (false)
If set to true, this enables the Cloudian Connector for CloudStack.
Restarting management server(s) is required.
- cloudian.admin.host (s3-admin.cloudian.com)
The host where Cloudian Admin services are accessible.
- cloudian.admin.port (19443)
The admin service port.
- cloudian.admin.protocol (https)
The admin service API scheme/protocol.
- cloudian.validate.ssl (true)
 When set to true, this validates the certificate of the https-enabled
admin API service.
- cloudian.admin.user (sysadmin)
The admin user's name when making (admin) API calls.
- cloudian.admin.password (public)
The admin password used when making (admin) API calls.
- cloudian.api.request.timeout (5)
The API request timeout in seconds used by the internal HTTP/s client.
- cloudian.cmc.admin.user (admin)
The CMC admin user's name.
- cloudian.cmc.host (cmc.cloudian.com)
The CMC host.
- cloudian.cmc.port (8443)
The CMC service port.
- cloudian.cmc.protocol (https)
 The CMC service scheme/protocol.
- cloudian.sso.key (ss0sh5r3dk3y)
The Single-Sign-On shared key.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-25 10:49:45 +05:30
Rohit Yadav bd953d811f CLOUDSTACK-10107: For VMware VMs add devices without unit number (#2288)
When VMs are deployed or nics are plugged, using a static unit number
may cause device configuration errors. This fixes a previous limitation
that more than 7 nics/networks could not be added to a VM.

Per the API docs, `unitNumber` need not be set:
https://www.vmware.com/support/developer/converter-sdk/conv55_apireference/vim.vm.device.VirtualDevice.html

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-25 09:56:44 +05:30
Rohit Yadav c3ed1b38e5 CLOUDSTACK-9993: Have basic constraint in CA certificate (#2286)
- Refactors V3 x509 cert generator to put basic constraint and key usage
  extensions when CA cert is created
- Refactors root CA provider to use V3 generator to generate CA cert

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-11 17:41:37 +05:30
Rohit Yadav 0fedbdd7a9 CLOUDSTACK-9998: Prometheus Exporter for CloudStack (#2287)
This implements a CloudStack Prometheus exporter as a plugin, that serves
metrics on a HTTP port.

New global settings:

1. prometheus.exporter.enable - (default: false), Enable the prometheus
exporter plugin, management server restart needed.
2. prometheus.exporter.port - (default: 9595), The prometheus exporter
server port.
3. prometheus.exporter.allowed.ips - (default: 127.0.0.1), List of comma
separated prometheus server ips (with no spaces) that should be allowed to
access the URLs.

The following list  of  metrics are provided  per pop (zone)  with  the exporter:
• Per host:
o CPU cores:  used, total
o CPU usage:  used, total (in MHz)
o Memory  usage:  used, total (in MiBs)
o Total VMs running on  the host
• CPU cores:  allocated (per  zone)
• CPU usage:  allocated (per  zone, in  MHz)
• Memory  usage:  allocated (per  zone, in  MiBs)
• Hosts:  online, offline,  total
• VMs: in all states -- starting, running, stopping, stopped, destroyed,
       expunging, migrating,  error, unknown
• Volumes:  ready,  destroyed,  total
• Primary Storage Pool: (Disk size) used, allocated,  unallocated,  total (in GiBs)
• Secondary Storage Pool: (Disk size) used, allocated,  unallocated,  total (in GiBs)
• Private IPs:  allocated,  total
• Public  IPs:  allocated,  total
• Shared  Network IPs:  allocated,  total
• VLANs:  allocated,  total

Additional metrics for the environment:
• Summed  domain  (level=1) limit for CPU cores
• Summed  domain  (level=1) limit for memory/ram

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-11 17:24:22 +05:30
Gabriel Beims Bräscher f1c01a5fa2 CLOUDSTACK-10101: Present the full domain name when listing user's domains in SAML2 plugin (#2280)
This commit is related to the Saml2 user authentication plugin.

The user can list its domains. His/Her domains are presented as the
following example:
username/subdomainA
username/subdomainB

However, if a user has two subdomains of the same name Dom1/subdomainA
and Dom2/subdomainA, the list is presented as follows:
username/subdomainA
username/subdomainA

With this commit it shows the full domain name to avoid such cases.
Thus, the domains will be presented as follows:
username/Dom1/subdomainA
username/Dom2/subdomainA
2017-10-10 10:12:03 +05:30
Rafael Weingärtner b044f1bc96 CLOUDSTACK-9859: Retirement of midonet plugin (final removal) (#2285)
Following the component retirement process defined in [1], a vote thread was started in [2]. The community decided to retire this Midonet plugin. This task represents the final step of the retirement, which is the removal of the plugin from CloudStacks code base.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68720798
[2] http://markmail.org/message/qigrtfirwnmct4hr
2017-10-10 10:08:23 +05:30
Rohit Yadav 0dfdbe064c CLOUDSTACK-9993: With auth strictness stop SSL handshake for rogue clients (#2278)
When auth strictness is set to true, terminate SSH handshake for clients
that do not present valid certificates.

This uses the `setNeedClientAuth`, where if the option is set and the
client chooses not to provide authentication information about itself,
the negotiations will stop and the engine will begin its closure
procedure:
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLEngine.html#setNeedClientAuth(boolean)

During systemvm reboot, the conf folder is removed and certificate
re-setup is not done. This may cause the agent to not connect, this
fixes the case by backing up and restoring keystore and other config
files when re-patching is done after rebooting of a systemvm (cpvm, ssvm).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-04 09:53:49 +05:30
Bitworks Software, Ltd 3381c38cc7 CLOUDSTACK-10073: KVM host RAM overprovisioning (#2266)
Commit enables a new feature for KVM hypervisor which purpose is to increase virtually amount of RAM available beyond the actual limit.
There is a new parameter in agent.properties: host.overcommit.mem.mb which enables adding specified amount of RAM to actually available. It is necessary to utilize KSM and ZSwap features which extend RAM with deduplication and compression.
2017-09-29 11:46:09 +05:30
Wido den Hollander b130e55088 CLOUDSTACK-9397: Add Watchdog timer to KVM Instance (#1707)
The watchdog timer adds functionality where the Hypervisor can detect if an
instance has crashed or stopped functioning.
The watchdog timer adds functionality where the Hypervisor can detect if an
instance has crashed or stopped functioning.

When the Instance has the 'watchdog' daemon running it will send heartbeats
to the /dev/watchdog device.

If these heartbeats are no longer received by the HV it will reset the Instance.

If the Instance never sends the heartbeats the HV does not take action. It only
takes action if it stops sending heartbeats.

This is supported since Libvirt 0.7.3 and can be defined in the XML format as
described in the docs: https://libvirt.org/formatdomain.html#elementsWatchdog

To the 'devices' section this will be added:

In the agent.properties the action to be taken can be defined:

vm.watchdog.action=reset

The same goes for the model. The Intel i6300esb is however the most commonly used.

vm.watchdog.model=i6300esb

When the Instance has the 'watchdog' daemon running it will send heartbeats
to the /dev/watchdog device.

If these heartbeats are no longer received by the HV it will reset the Instance.

If the Instance never sends the heartbeats the HV does not take action. It only
takes action if it stops sending heartbeats.

This is supported since Libvirt 0.7.3 and can be defined in the XML format as
described in the docs: https://libvirt.org/formatdomain.html#elementsWatchdog

To the 'devices' section this will be added:

  <watchdog model='i6300esb' action='reset'/>

In the agent.properties the action to be taken can be defined:

  vm.watchdog.action=reset

The same goes for the model. The Intel i6300esb is however the most commonly used.

  vm.watchdog.model=i6300esb

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-09-28 13:56:15 +05:30
SudharmaJain c670691bfb CLOUDSTACK-8865: Adding SR doesn't create Storage_pool_host_ref entry for disabled host (#876)
This causes VM deployment failure on the host that was disabled while adding the storage repository.
In the attachCluster function of the PrimaryDataStoreLifeCycle, we were only selecting hosts that are up and are in enabled state. Here if we select all up hosts, it will populate the DB properly and will fix this issue. Also added a unit test for attachCluster function.
2017-09-21 10:49:11 +05:30
SudharmaJain 3f69c83f96 CLOUDSTACK-9956: File search on the vmware datastore may select wrong file if there are multiple files with same name (#2153)
If there are multiple files with the same name on vmware datastore, search operation may select any one file during volume related operations. This involves volume attach/detach, volume download, volume snapshot etc.

While using NetApp as the backup solution. This has .snapshot folder on the datastore and sometimes files from this folder gets selected during volume operations and the operation fails. Because of wrong selection of file following exception can be observed while volume deletion.

2017-02-23 19:39:05,750 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-304:ctx-a1dbf5d8 ac.local) delete volume failed due to Exception: java.lang.RuntimeException
Message: Cannot delete file [4cbcd46d44c53f5c8244c0aad26a97e1] .snapshot/hourly.2017-02-23_1605/r-97-VM/ROOT-97.vmdk

To fix this behavior I have added a global configuration by name vmware.search.exclude.folders which can be comma separated list of folder paths.

I have also added a unit test to test the new method.
2017-09-19 15:12:17 +05:30
vedulasantosh 48fa20698e CLOUDSTACK-9963 Root Disk controller value is changing while migrating VM 2017-09-11 16:19:10 +05:30
Rohit Yadav 774998d76b Updating pom.xml version numbers for release 4.9.4.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-09-07 20:21:43 +05:30
subhash yedugundla 0d81e88601 CLOUDSTACK-8939: VM Snapshot size with memory correctly calculated in cloud.usage_event (XenServer) (#914)
Fixing the error  with snapshot size calculation  in a vm with memory
2017-09-01 11:00:55 +02:00
Gabriel Beims Bräscher 01b040d19e CLOUDSTACK-9314: Remove unused code from XenServerStorageProcessor (#1443)
This commit removes unused code from
"com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor"
class and modifies some methods access level.

Remove unused methods:
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getSRByNameLabel(Connection,
String)
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getNfsSR(Connection,
StorageFilerTO)
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.directDownloadHttpTemplate(CopyCommand,
DecodedDataObject, DecodedDataObject)
- com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.execute(AttachPrimaryDataStoreCmd)


access level:
- from public to private:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.swiftUpload(Connection,
SwiftTO, String, String, String, Boolean, int)
- from public to protected:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.swiftBackupSnapshot(Connection,
SwiftTO, String, String, String, Boolean, int)
- from protected to private:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getSnapshotSize(Connection,
String, String, Boolean, int)
- from protected to private:
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.backupSnapshot(Connection,
String, String, String, String, String, String, Boolean, int)
2017-08-31 09:34:24 +02:00
Boris Stoyanov f917ab660e CLOUDSTACK-9782: Improve host HA tests
- All tests should pass on KVM, Simulator
- Add test cases covering FSM state transitions and actions

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav d2c3408da7 CLOUDSTACK-9782: Improve scheduling of jobs
- Removed three bg thread tasks, uses FSM event-trigger based scheduling
- On successful recovery, kicks VM HA
- Improves overall HA scheduling and task submission, lower DB access

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav c0b33db5ce CLOUDSTACK-9782: Nested-oobm CloudStack plugin
Nested out-of-band management plugin to work with hosts that are VMs in
a CloudStack env.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Jayapal 6e832b55dc Merge pull request #2109 from Accelerite/CLOUDSTACK-9922
CLOUDSTACK-9922 Unable to use 8081 port for Load balancing
2017-08-30 15:15:19 +05:30
Rohit Yadav d145944be0 Updating pom.xml version numbers for release 4.9.3.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 14:52:37 +02:00
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
Daan Hoogland 94718c0bd3 CE-110 markedForGC is ignored so delete the templates in the task that finds them 2017-08-25 08:57:52 +02:00
Daan Hoogland 16b34c9a11 CE-110 default interval of 0 minutes means do not mark for cleaning 2017-08-25 08:57:51 +02:00
Daan Hoogland c10c342d9e CE-110 task for marking cleaning fully cloned templates
implemented by marking them for GC
2017-08-25 08:57:51 +02:00
digitalgust a8c09f2a19 CLOUDSTACK-9996: Fix juniper vlan untagged jedgement and xml element replacment (#2174)
Fix  juniper vlan untagged jedgement and xml element replacment.
2017-08-24 13:22:06 +02:00
sureshanaparti 0c6cf69eee CLOUDSTACK-9717: [VMware] RVRs have mismatching MAC addresses for extra public NICs. (#1878)
Fix: When RVR is enabled and Peer Router is available, get the MAC addresses of the extra public NICs from the Peer Router and set them to the router.
2017-08-21 11:04:20 +02:00
SowjanyaPatha f66bbb9521 CLOUDSTACK-9922 Unable to use 8081 port for Load balancing 2017-08-08 11:57:45 +05:30
Wido den Hollander 2867080979 CLOUDSTACK-10034: Use libvirt to create new volumes and not rados-java (#2039)
Since libvirt 1.2.2 libvirt will properly create volumes
using RBD format 2.

We can use libvirt to creates the volumes which strips a bit of
code from the CloudStack Agent's responsbility.

RBD format 2 is already used by all volumes created by CloudStack.

This format is the most recent format of RBD and is still actively
being developed.

This removes the support for Ubuntu 12.04 as that does not have the
proper libvirt version available.

Signed-off-by: Wido den Hollander wido@widodh.nl

We can use libvirt to creates the volumes which strips a bit of
code from the CloudStack Agent's responsbility.

RBD format 2 is already used by all volumes created by CloudStack.

This format is the most recent format of RBD and is still actively
being developed.

This removes the support for Ubuntu 12.04 as that does not have the
proper libvirt version available.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-08-06 00:04:21 +02:00
dahn 597df24b53 CLOUDSTACK-10007: Isolation methods (#2193)
Change isolation methods from an enum to a registry based construct to enhance pluggability
2017-08-03 12:20:37 +02:00
Sateesh Chodapuneedi 6977cb3841 CLOUDSTACK-9684 Invalid zone id error while listing vmware zone
Issue
=====
While listing datacenters associated with a zone, only zone Id validation is required.
There is no need to have additional checks like zone is a legacy zone or not.

Fix
===
Removed unnecessary checks over zone ID and just checking if zone with specified ID exists or not.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>
(cherry picked from commit 0ef1c17541)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
subhash yedugundla 1b26a486d1 BUG-ID:CLOUDSTACK-8921
Summary: CLOUDSTACK-8921
     snapshot_store_ref table should store actual size of back snapshot in secondary storage
Calling SR scan to make sure size is updated correctly

(cherry picked from commit 4e4b67cd96)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Nitesh Sarda 9334e091c0 CLOUDSTACK-9937 : dedicateCluster API response does not return correct detail in response
(cherry picked from commit 7364616aa4)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Rajani Karuturi 9ce2dde030 CLOUDSTACK-9834: prepareTemplate API call doesn't work well
with XenServer & Local SR (Db_exn.Uniqueness_constraint_violation)

removed the host uuid from SR label so that any host which has access to
the SR(all the hosts in the same pool) can reuse the same SR

(cherry picked from commit 1aa6a72bc7)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Rene Moser 05f94b8ef7 Bug fix for CLOUDSTACK-9762 Management Server UI (VM statistics page) CPU Utilized value is incorrect.
(cherry picked from commit b676a8a7c3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Anshul Gangwar e6855db78a CLOUDSTACK-9685: delete snapshot on primary associated with a volume when that volume is deleted
as that snapshot will never be going to use again and also it will fill up primary storage

(cherry picked from commit 336df84f17)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Nick Livens 1f44e884ad CLOUDSTACK-9321 : Multiple Internal LB rules (more than one Internal LB rule with same source IP address) are not getting resolved in the corresponding InternalLbVm instance's haproxy.cfg file
CLOUDSTACK-9321 : Adding component tests for VPC Network functionality - Internal LB rules

CLOUDSTACK-9321 : Extending Nuage VSP Internal LB Marvin tests

Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>, Frank Maximus <frank.maximus@nuagenetworks.net>

(cherry picked from commit 62e858131f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
nvazquez 1d25582ed6 CLOUDSTACK-9752: [Vmware] Optimization of volume attachness to vm
(cherry picked from commit 49dadc5505)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Suresh Kumar Anaparti 0fbbbdb83a CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.
(cherry picked from commit 8676b20276)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Suresh Kumar Anaparti e61815a255 CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM
Updated hardcoded value with max data volumes limit from hypervisor capabilities.

(cherry picked from commit 93f5b6e8a3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-01 20:40:25 +02:00
Syed Mushtaq Ahmed a4cecd2366 solidfire: Add NULL checks for various objects in SolidFire integration test API (#2205) 2017-07-28 11:58:05 +02:00
Rohit Yadav 619da91386 Merge branch '4.10' 2017-07-28 11:28:20 +02:00
Rohit Yadav e9f526e221 Merge branch '4.9' into 4.10
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-28 10:49:34 +02:00
Frank Maximus 0ead11e8f1 CLOUDSTACK-9751: Fix public ip not applied, when added while VR is starting. (#1925)
Public IP state wasn't passed to the vsp client, making it ignore the apply public ip.

Reported-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-07-28 10:30:43 +02:00
Rohit Yadav c384239a05 Merge branch '4.10' 2017-07-24 12:44:25 +02:00
Rohit Yadav 83e9de4bfd Merge branch '4.9' into 4.10 2017-07-24 12:43:24 +02:00
Rohit Yadav 90e7cabf4e FIX3: Consider overcommit ratios with total/threshold values for host metrics
Consider the CPU and memory overcommit ratios with total cpu/ram values
or thresholds for host metrics. This will fix incorrect notification
(cells turning yellow/red) in the metrics view.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-24 12:42:02 +02:00
Wei Zhou c03923c7e7 CLOUDSTACK-9113: skip vm with inconsistent state when getVmNetworkStats 2017-07-23 17:15:23 +02:00
Wei Zhou 960cb84083 CLOUDSTACK-7984: Collect network statistics for VMs on shared network (KVM implementation) 2017-07-23 17:15:23 +02:00
Syed 829965771a Check for host before adding it to storagepaths 2017-07-22 11:21:00 +02:00
Nitin Kumar Maharana e243a31e41 CLOUDSTACK-8672 : NCC Integration with CloudStack.
Improvements.
2017-07-20 12:42:43 +05:30
Nitin Kumar Maharana b2b422c7d7 CLOUDSTACK-8672 : NCC Integration with CloudStack. 2017-07-20 12:42:43 +05:30
Priyank Parihar b2f7f9abb6 CLOUDSTACK-8672 : NCC Integration with CloudStack. 2017-07-20 12:42:43 +05:30
Rajesh Battala 884606f77b CLOUDSTACK-8672 : NCC Integration with CloudStack. 2017-07-20 12:42:43 +05:30
Rohit Yadav 445dbe7c53 Merge remote-tracking branch 'origin/4.10' 2017-07-14 15:00:11 +05:30
Rohit Yadav 32e96abea9 Merge remote-tracking branch 'origin/4.9' into 4.10 2017-07-14 14:59:17 +05:30
Wido den Hollander ca415e7436 CLOUDSTACK-9929: Do not gather statistics for CDROM or FLOPPY devices
Libvirt / Qemu (KVM) does not collect statistics about these either.

On some systems it might even yield a 'internal error' from libvirt
when attempting to gather block statistics from such devices.

For example Ubuntu 16.04 (Xenial) has a issue with this.

Skip them when looping through all devices.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-07-14 14:57:11 +05:30
Rajani Karuturi 4bc7c270fa Updating pom.xml version numbers for release 4.11.0.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-12 12:09:38 +05:30
Rajani Karuturi 4f229d19ad Updating pom.xml version numbers for release 4.10.1.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-10 10:36:24 +05:30
Rajani Karuturi 9d2893d44a Updating pom.xml version numbers for release 4.10.0.0
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-03 10:06:43 +05:30
Mike Tutkowski 330089a801
Only add the “.vmdk” suffix if need be
(cherry picked from commit b156d5de11)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-06-15 16:29:05 +05:30
Rohit Yadav ed376fcad6 Merge remote-tracking branch 'origin/4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-07 11:21:27 +05:30
Rohit Yadav e197652a28 CLOUDSTACK-9860: Fix stackoverflow issue
Fixes issue caused to a PR-refactoring from #2108, reported by
@borisstoyanov

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-06 16:14:03 +05:30
Rajani Karuturi 321712bcaf Merge pull request #1841 from sateesh-chodapuneedi/pr-cloudstack-9684
CLOUDSTACK-9684 Invalid zone id error while listing vmware zone
2017-06-06 16:04:24 +05:30
Rajani Karuturi c4c692afa5 Merge pull request #1862 from sateesh-chodapuneedi/pr-cloudstack-9704
CLOUDSTACK-9704 Remove dependency on VmwareContext object to fetch systemVM keyfile
2017-06-06 15:57:18 +05:30
Rajani Karuturi 7ee751e747 Merge pull request #1894 from Accelerite/CLOUDSTACK-9700
CLOUDSTACK-9700 Allow user to Register/Copy templates to multiple zones at the same time
2017-06-06 15:41:28 +05:30
Rajani Karuturi 51974cd568 Merge pull request #899 from yvsubhash/CLOUDSTACK-8921
BUG-ID:CLOUDSTACK-8921 snapshot_store_ref table should store actual size of backup snapshot in secondary storage
2017-06-06 15:38:42 +05:30
Rajani Karuturi 68d50fbfd8 Merge pull request #1996 from Accelerite/secretkey
CLOUDSTACK-9099: SecretKey is returned from the APIs
2017-06-06 15:35:11 +05:30
Nitesh Sarda 7364616aa4 CLOUDSTACK-9937 : dedicateCluster API response does not return correct detail in response 2017-06-01 14:03:06 +05:30
Rohit Yadav 8323a175f1 CLOUDSTACK-9860: Power off VMs when stopVM is called with forced=true
The 'force' option provided with the stopVirtualMachine API command is
often assumed to be a hard shutdown sent to the hypervisor, when in fact
it is for CloudStacks' internal use. CloudStack should be able to send
the 'hard' power-off request to the hosts.

When forced parameter on the stopVM API is true, power off (hard shutdown)
a VM. This uses initial changes from #1635 to pass the forced parameter
to hypervisor plugin via the StopCommand, and fixes force stop (poweroff)
handling for KVM, VMware and XenServer.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-25 17:25:22 +05:30
Rajani Karuturi e25a444a0c Merge pull request #2121 from bvbharatk/CLOUDSTACK-9641
CLOUDSTACK-9641 In KVM SSVM and CPVM may use the old cmdline data, if…
2017-05-23 10:07:55 +05:30
Bharat Kumar 3c80f00550 CLOUDSTACK-9641 In KVM SSVM and CPVM may use the old cmdline data, if we fail to fetch the new cmdline in the first pass. 2017-05-19 16:50:19 +05:30
Rajani Karuturi 701146ff28 Merge pull request #1908 from Accelerite/staticnat
CLOUDSTACK-9317: Fixed disable static nat on leaving ips on interface
2017-05-18 15:48:09 +05:30
Jayapal 87cf33ac5c CLOUDSTACK-9099: Added a separate API to apikey and secretkey 2017-05-17 14:16:12 +05:30
Rajani Karuturi 8b3cadb55e Merge release branch 4.9 to master
* 4.9:
  Do not set gateway to 0.0.0.0 for windows clients
  CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
  ignore bogus default gateway   when a shared network is secondary the default gateway gets overwritten by a bogus one   dnsmasq does the right thing and replaces it with its own default which is not good for us   so check for '0.0.0.0'
  Activate NioTest following changes in CLOUDSTACK-9348 PR #1549
  CLOUDSTACK-9828: GetDomRVersionCommand fails to get the correct version as output Fix tries to return the output as a single command, instead of appending output from two commands
  CLOUDSTACK-3223 Exception observed while creating CPVM in VMware Setup with DVS
  CLOUDSTACK-9787: Fix wrong return value in NetUtils.isNetworkAWithinNetworkB
2017-05-17 11:36:52 +05:30
Jayapal d04a3e842c CLOUDSTACK-9317: Update review comments and rule state column 2017-05-17 11:08:13 +05:30
Jayapal c20e0ef88f CLOUDSTACK-9317: Fixed disable static nat on leaving ips on interface 2017-05-17 11:03:50 +05:30
Rajani Karuturi 7434d91614 Merge pull request #1873 from Accelerite/dhcpOffloadFix
CLOUDSTACK-9709: Updated the vm ip fetch task to use the correct the …
2017-05-17 10:43:51 +05:30
Rajani Karuturi 9c4eb959c3 Merge pull request #2070 from shapeblue/hyperv-log-fix
CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
2017-05-17 09:49:33 +05:30
Rajani Karuturi 9b605b1a72 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9900: Fix high CPU deviation issues seen in metrics view
2017-05-16 12:44:53 +05:30
Rajani Karuturi 86eb5683dd Merge pull request #1840 from anshul1886/CLOUDSTACK-9685
CLOUDSTACK-9685: delete snapshot on primary associated with a volume …
2017-05-16 12:42:48 +05:30
Rajani Karuturi a4f3c71744 Merge pull request #1849 from sudhansu7/CLOUDSTACK-9690
CLOUDSTACK-9690: Scale CentOS7 VM fails with error
2017-05-16 11:24:50 +05:30
rashmidixit 49a862b223 CLOUDSTACK-9700: Allow user to Register/Copy templates to multiple zones at the same time 2017-05-15 14:07:22 +05:30
Rajani Karuturi 1aa6a72bc7 CLOUDSTACK-9834: prepareTemplate API call doesn't work well
with XenServer & Local SR (Db_exn.Uniqueness_constraint_violation)

removed the host uuid from SR label so that any host which has access to
the SR(all the hosts in the same pool) can reuse the same SR
2017-05-15 11:17:20 +05:30
Rajani Karuturi 45f62c3483 Merge pull request #1813 from priyankparihar/CLOUDSTACK-9604
CLOUDSTACK-9604: Root disk resize support for VMware and XenServer.
2017-05-08 10:15:57 +05:30
Sudhansu e7a7db3ef3 CLOUDSTACK-9690: Scale CentOS7 VM fails with error
1. Removed XenServerGuestOsMemoryMap from CitrixHelper.java
This java file was holding a static in memory map named XenServerGuestOsMemoryMap. This was the source for xenserver dynamic memory values(max and min). These values were moved to guest_os_details table.

2. DAO layer was modified to access these values.
3. VirtualMachineTo object was modified to populate the dynamic memory values.
4. addGuestOs and UpdateGuestOS api has been modified to update memory values.
2017-05-05 10:58:12 +05:30
Rohit Yadav c4326b09c6 CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
This fixes log4j xml to have @AGENTLOG@ replaced with values defined
in build/replace.properties.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-04 13:47:23 +05:30
Rohit Yadav efe7646977 CLOUDSTACK-9900: Fix high CPU deviation issues seen in metrics view
HostStats returns cpu usage in percentage while memory usage in bytes.
This fixes a regression in maximum CPU usage deviation that did not
assume the values to be in percentage and multiple the final ratios
with 100 which leads to 100x the actual deviation value.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-05-03 15:02:47 +05:30
Rajani Karuturi 57628b2dd0 Merge pull request #2030 from shapeblue/snapshot-housekeeping
CLOUDSTACK-9864 cleanup stale worker VMs after job expiry time
2017-05-02 11:08:50 +05:30
Rajani Karuturi 4ad29b474a Merge pull request #1918 from jayakarteek/cloudstack-9762
Management Server UI (VM statistics page) CPU Utilized value is incorrect.
2017-04-27 11:12:00 +05:30
Mowgli c2c1f01d2e Merge branch 'master' into CLOUDSTACK-9604 2017-04-26 11:48:56 +05:30
Rajani Karuturi fc8ed22abe Merge pull request #2060 from apache/4.9
Merge release branch 4.9 to master
2017-04-23 09:34:54 +05:30
Rajani Karuturi a4dd6bdeeb Merge pull request #1955 from myENA/virtio-scsi
CLOUDSTACK-8239 Add VirtIO SCSI support for KVM hosts
2017-04-20 15:36:34 +05:30
Daan Hoogland 084febb02b CE-113 use of duration (instead of the old clock-tick-based code 2017-04-18 18:11:00 +02:00
Daan Hoogland 8065aedf68 CE-113 extra trace log of worker VMs 2017-04-18 18:11:00 +02:00
Daan Hoogland 61189db1d9 CE-113 configure workervm gc based on job expiry 2017-04-18 18:11:00 +02:00
Rafael Weingartner a3f351c07e [CLOUDSTACK-9858] Retirement of midonet plugin (build disabling) 2017-04-11 16:04:10 -04:00
Suresh Kumar Anaparti bf77c07797 CLOUDSTACK-9719: [VMware] VR loses DHCP settings and VMs cannot obtain IP after HA recovery - Set high restart priority for the VR.
Test scenarios:
- Enable cluster HA after VR is created. Now stop and start VR and check its restart priority, should be High.
- Enable cluster HA before VR is created. Now create some VM and verify that VR created must have High restart priority.
2017-04-11 10:48:42 +05:30
Rajani Karuturi e1c76dcc23 CLOUDSTACK-9783: corrected the version number in metrics pom
master is broken after fwd-merging metrics PR #1944 from 4.9
This is due to the incorrect parent version number in the metrics pom.
Corrected the same to reflect master version number.
2017-04-07 09:52:27 +05:30
Rajani Karuturi 4db186ef6f Merge pull request #1880 from Accelerite/CLOUDSTACK-9720
CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.Updated the template_spool_ref table with the correct template (VMware - OVA file) size.

* pr/1880:
  CLOUDSTACK-9720: [VMware] template_spool_ref table is not getting updated with correct template physical size in template_size column.

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-04-06 12:11:22 +05:30
Rajani Karuturi 5c0979fff5 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9783: Improve metrics view performance
2017-04-06 12:04:55 +05:30
Rajani Karuturi 6548839417 Merge pull request #1944 from shapeblue/4.9-metrics-enhancement
CLOUDSTACK-9783: Improve metrics view performanceThis improves the metrics view feature by improving the rendering performance
of metrics view tables, by re-implementing the logic at the backend and data
served via APIs. In large environments, the older implementation would
make several API calls that increases both network and database load.

List of APIs introduced for improving the performance that re-implement the frontend logic at backend:

    listClustersMetrics
    listHostsMetrics
    listInfrastructure
    listStoragePoolsMetrics
    listVMsMetrics
    listVolumesMetrics
    listZonesMetrics

Pinging for review - @abhinandanprateek @DaanHoogland @borisstoyanov @karuturi @rashmidixit

Marvin test results:

=== TestName: test_list_clusters_metrics | Status : SUCCESS ===

=== TestName: test_list_hosts_metrics | Status : SUCCESS ===

=== TestName: test_list_infrastructure_metrics | Status : SUCCESS ===

=== TestName: test_list_pstorage_metrics | Status : SUCCESS ===

=== TestName: test_list_vms_metrics | Status : SUCCESS ===

=== TestName: test_list_volumes_metrics | Status : SUCCESS ===

=== TestName: test_list_zones_metrics | Status : SUCCESS ===

* pr/1944:
  CLOUDSTACK-9783: Improve metrics view performance

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-04-06 12:04:29 +05:30
Rene Moser b676a8a7c3 Bug fix for CLOUDSTACK-9762 Management Server UI (VM statistics page) CPU Utilized value is incorrect. 2017-04-05 15:37:40 +05:30
subhash yedugundla 4e4b67cd96 BUG-ID:CLOUDSTACK-8921
Summary: CLOUDSTACK-8921
     snapshot_store_ref table should store actual size of back snapshot in secondary storage
Calling SR scan to make sure size is updated correctly
2017-04-04 15:02:17 +05:30
Rajani Karuturi ec2d4dd422 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number>
2017-03-23 15:19:31 +05:30
Wei Zhou bf93b6313e CLOUDSTACK-9811: fix duplicated nics on VR caused by nic name p<slot_number>p<port_number> 2017-03-20 07:32:31 +01:00
Rajani Karuturi ad7ed7a178 Merge pull request #847 from kishankavala/CLOUDSTACK-8880
Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm.  free memory = total memory - (all vm memory)With memory over-provisioning set to 1, when mgmt server starts VMs in parallel on one host, then the memory allocated on that kvm can be larger than the actual physcial memory of the kvm host.

Fixed by checking free memory on host before starting Vm.
Added test case to check memory usage on Host.
Verified Vm deploy on Host with enough capacity and also without capacity

* pr/847:
  Bug-ID: CLOUDSTACK-8880: calculate free memory on host before deploying Vm.  free memory = total memory - (all vm memory)

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:20:58 +05:30
Rajani Karuturi 3f0fbf251c Merge pull request #1953 from Accelerite/CLOUDSTACK-9794
CLOUDSTACK-9794: Unable to attach more than 14 devices to a VMUpdated hardcoded value with max data volumes limit from hypervisor capabilities.

* pr/1953:
  CLOUDSTACK-9794: Unable to attach more than 14 devices to a VM

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:19:04 +05:30
Rajani Karuturi 09802e0f3c Merge pull request #1861 from sateesh-chodapuneedi/pr-cloudstack-9698
CLOUDSTACK-9698 [VMware] Make hardcorded wait timeout for NIC adapter hotplug as configurableJira
===
CLOUDSTACK-9698 [VMware] Make hardcoded wait timeout for NIC adapter hotplug as configurable

Description
=========
Currently ACS waits for 15 seconds (hard coded) for hot-plugged NIC in VR running on VMware to get detected by guest OS.
The time taken to detect hot plugged NIC in guest OS depends on type of VMware NIC adapter like (E1000, VMXNET3, E1000e etc.)
and guest OS itself. In uncommon scenarios the NIC detection may take longer time than 15 seconds,
in such cases NIC hotplug would be treated as failure which results in VPC tier configuration failure.
Alternatively making the wait timeout for NIC adapter hotplug as configurable will be helpful for admins in such scenarios. This is specific to VR running over VMware hypervisor.

Also in future if VMware introduces new NIC adapter types which may take time to get detected by guest OS, it is good to have flexibility of
configuring the wait timeout as fallback mechanism in such scenarios.

Fix
===
Introduce new configuration parameter (via ConfigKey) "vmware.nic.hotplug.wait.timeout" which is "Wait timeout (milli seconds) for hot plugged NIC of VM to be detected by guest OS." as fallback instead of hard coded timeout, to ensure flexibility for admins given the listed scenarios above.

Signed-off-by: Sateesh Chodapuneedi <sateesh.chodapuneedi@accelerite.com>

* pr/1861:
  CLOUDSTACK-9698 Make the wait timeout for NIC adapter hotplug as configurable

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:12:54 +05:30
Rajani Karuturi 56e851ca46 Merge release branch 4.9 to master
* 4.9:
  moved logrotate from cron.daily to cron.hourly for vpcrouter in cloud-early-config
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent
2017-03-13 22:09:27 +05:30
Rajani Karuturi 7b719c71fc Merge pull request #1856 from ustcweizhou/set-kvm-host-params
[4.9] CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agentThe router.aggregation.command.each.timeout in global configuration is only applied on new created KVM host.
For existing KVM host, changing the value will not be effective.
We need to propagate the configuration to existing host when cloudstack-agent is connected.

* pr/1856:
  CLOUDSTACK-9569: propagate global configuration router.aggregation.command.each.timeout to KVM agent

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-03-13 22:08:47 +05:30