Commit Graph

2232 Commits

Author SHA1 Message Date
Rohit Yadav 85aee8d18d CLOUDSTACK-10013: SystemVM codebase refactorings and improvements
- Refactors and simplifies systemvm codebase file structures keeping
  the same resultant systemvm.iso packaging
- Password server systemd script and new postinit script that runs
  before sshd starts
- Fixes to keepalived and conntrackd config to make rVRs work again
- New /etc/issue featuring ascii based cloudmonkey logo/message and
  systemvmtemplate version
- SystemVM python codebase linted and tested. Added pylint/pep to
  Travis.
- iptables re-application fixes for non-VR systemvms.
- SystemVM template build fixes.
- Default secondary storage vm service offering boosted to have 2vCPUs
  and RAM equal to console proxy.
- Fixes to several marvin based smoke tests, especially rVR related
  tests. rVR tests to consider 3*advert_int+skew timeout before status
  is checked.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav 0102e8593d CLOUDSTACK-10129: UX improvements and event timeline
- Fixes timezone issue where dates show up as nvalid in UI
- Introduces new event timeline listing/filtering of events
- Several UI improvements to add columns in list views
- Bulk operations support in instance list view to shutdown and destroy
  multiple-selected VMs (limitation: after operation, redundant entries
  may show up in the list view, refreshing VM list view fixes that)
- Align table thead/tbody to avoid splitting of tables

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav d0005d8353 CLOUDSTACK-9348: Improve Nio SSH handshake buffers
Use a holder class to pass buffers, fixes potential leak.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav 893f2af31f CLOUDSTACK-7853: Fix ping timeout edge case and refactor code
Refresh InaccurateClock every 10seconds, refactor code to get ping timeout
and ping interval.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Rohit Yadav bb965649d9 CLOUDSTACK-10013: Debian9 SystemVM appliance improvements
- Refactor cloud-early-config and make appliance specific scripts
- Make patching work without requiring restart of appliance and remove
  postinit script
- Migrate to systemd, speedup booting/loading
- Takes about 5-15s to boot on KVM, and 10-30seconds for VMware and XenServer
- Appliance boots and works on KVM, VMware, XenServer and HyperV
- Update Debian9 ISO url with sha512 checksum
- Speedup console proxy service launch
- Enable additional kernel modules
- Remove unknown ssh key
- Update vhd-util URL as previous URL was down
- Enable sshd by default
- Use hostnamectl to add hostname
- Disable services by default
- Use existing log4j xml, patching not necessary by cloud-early-config
- Several minor fixes and file refactorings, removed dead code/files
- Removes inserv
- Fix dnsmasq config syntax
- Fix haproxy config syntax
- Fix smoke tests and improve performance
- Fix apache pid file path in cloud.monitoring per the new template

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Sigert Goeminne d49765619d CLOUDSTACK-10024: Network migration support
Co-Authored-By: Frank Maximus frank.maximus@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net

New API’s:

* migrateNetwork
* migrateVpc
2017-12-21 11:25:17 +01:00
Bitworks Software, Ltd 80a6961b5d CLOUDSTACK-10168: VR duplicate entries in /etc/hosts when reusing VM name (#2366)
Bug is fixed. Fixed typo 'adress' to 'address' everywhere which leads to correct generation for agent communication json dhcp command.
2017-12-20 23:19:12 +05:30
Nicolas Vazquez 13c325aad4 CLOUDSTACK-10102: New network type L2 (#2281)
This feature allows CloudStack administrators to create layer 2 networks on CloudStack. As these networks are purely layer 2, they don't require IP addresses or Virtual Router, only VLAN is necessary (provided by administrator or assigned by CloudStack). Also, network services should be handled externally, e.g. DNS, DHCP, as they are not provided by L2 networks.
As a consequence, a new Guest Network type is created within CloudStack: L2

Description:
Network offerings and networks support new guest type: L2.
L2 Network offering creation allows administrator to select Specify VLAN or let CloudStack assign it dynamically.
L2 Network creation allows administrator to specify VLAN tag (if network offerings allows it) or simply create network.
VM deployments on L2 networks:
VMs should not IP addresses or any network service
No Virtual Router deployed on network
If Specify VLAN = true for network offering, network gets implemented using a dynamically assigned VLAN
UI changes

A new button is added on Networks tab, available for admins, to allow L2 networks creation
2017-12-20 17:07:39 +05:30
Nitin Kumar Maharana 8acb0908c4 CLOUDSTACK-9880: Expansion of Management IP Range. (#2048)
At present, The management IP range can only be expanded under the same subnet. According to existing range, either the last IP can be forward extended or the first IP can be backward extended. But we cannot add an entirely different range from the same subnet. So the expansion of range is subnet bound, which is fixed. But when the range gets exhausted and a user wants to deploy more system VMs, then the operation would fail. The purpose of this feature is to expand the range of management network IPs within the existing subnet. It can also delete and list the IP ranges.

Please refer the FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Expansion+of+Management+IP+Range
2017-12-20 08:36:53 +00:00
Marc-Aurèle Brothier a372040798 CLOUDSTACK-10012: Load SQL schema scripts from JAR (#2247)
Load SQL schema scripts from the uber/fat JAR. This also removes the setup/db/db directory from centos packaging.
2017-12-19 12:11:38 +05:30
subhash yedugundla 1edb3e8a45 CLOUDSTACK-9595: Avoiding the deadlocks in the code (#1762)
MySQLTransactionRollbackException is seen frequently in logs
Root Cause
Attempts to lock rows in the core data access layer of database fails if there is a possibility of deadlock. However Operations are not getting retried in case of deadlock. So introducing retries here
Solution
Operations would be retried after some wait time in case of dead lock exception.
2017-12-19 00:06:21 +05:30
Rafael Weingärtner 3c6df7c970 CLOUDSTACK-10196: Remove ejb-api 3.0 dependency (#2348)
Fixed QuotaAlertManagerImplTest, which was injecting mock objects manually.
2017-12-18 23:59:06 +05:30
subhash yedugundla 566238b08e CLOUDSTACK-8908: After copying the template charging for that template is stopped (#2308)
When a template is copied back to zone after it is deleted. deleted field gets reset to null. delete field is added to Search on template zone mapping table to take care of the existing mapping.
2017-12-16 09:50:11 +05:30
niteshsarda 44a7eb0d14 CLOUDSTACK-10176: VM Start Api Job returns success for failed Job (#2354)
ISSUE :

VM start API Job returns success for failed Job

STEPS TO REPRODUCE :

Stop VM instance.
Mark state of associate router as Stopping from DB
Execute startVirtualMachine Api from rest client.
VM will not start but still main job will return the status as SUCCEEDED, whereas sub-job for command VmWorkStart will return status as failed.
2017-12-15 17:18:51 +05:30
René Moser e5f0a68a4d CLOUDSTACK-10077: allow to have different VPN customer gateway configs for same gateway IP (#2267)
Related 86bbe211f2 and CLOUDSTACK-494. Currently we can not have serveral VPCs in one account with different VPN customer gateways configuration per same gateway IP.
2017-12-11 21:58:21 +05:30
Wido den Hollander 632479d8f8 CLOUDSTACK-9853: Add support for Secondary IPv6 Addresses and Subnets (#2028)
This commit adds support for passing IPv6 Addresses and/or Subnets as
Secondary IPs.

This is groundwork for CLOUDSTACK-9853 where IPv6 Subnets have to be
allowed in the Security Groups of Instances to we can add DHCPv6
Prefix Delegation.

Use ; instead of : for separating addresses, otherwise it would cause
problems with IPv6 Addresses.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-22 17:30:33 +05:30
Sigert Goeminne 77864992fe CLOUDSTACK-9776: extra DHCP options support for Nuage VSP
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9776

Design-Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+extra+DHCP+option+support
2017-11-21 11:44:39 +01:00
Marc-Aurèle Brothier 421f1072d5 CLOUDSTACK-10123: Entity should use GMT TZ for timestamp values (#2303)
Depending on the timezone you're running CS (before GMT timezones) you could experience that some jobs are marked as failed since the parent job got a null result despite its child job having successfully done the job. The child job got deleted by the CleanupTask ahead of time, due to a missing datetime conversion to GMT timezone.

Jobs are failing with this message: Job failed with un-handled exception

The fix intends to correct any datetime used in the code that should be using the GMT timezone instead of the local one since all DB datetime should be stored at GMT.
2017-11-18 09:14:57 +05:30
Rafael Weingärtner 7f6ae15972 Remove annotation not needed at cloud-engine-storage-image (#2324)
remove bogus depends-on declaration at spring-engine-storage-snapshot-core-context
2017-11-16 23:11:15 +05:30
Frank Maximus 14ab69e19a
Merge pull request #2005 from nuagenetworks/feature/nw_offering_for_vpc
CLOUDSTACK-9450: Network Offering for VPC based on DB flag
2017-11-16 17:33:07 +01:00
Rohit Yadav 3ee8d83621 CLOUDSTACK-10136: Fix RemoteHostEndPoint thread growth
This fixes the following:
- Unchecked thread growth in RemoteEndHostEndPoint
- Potential NPE while finding EP for a storage/scope

Unbounded thread growth can be reproduced with following findings:
- Every unreachable template would produce 6 new threads (in a single
ScheduledExecutorService instance) spaced by 10 seconds
- Every reachable template url without the template would produce 1 new
thread (and one ScheduledExecutorService instance), it errors out quickly without
causing more thread growth.
- Every valid url will produce upto 10 threads as the same ep (endpoint
instance) will be reused to query upload/download (async callback)
progresses.

Every RemoteHostEndPoint instances creates its own
ScheduledExecutorService instance which is why in the jstack dump, we
see several threads that share the prefix RemoteHostEndPoint-{1..10}
(given poolsize is defined as 10, it uses suffixes 1-10).

This fixes the discovered thread leakage with following notes:
- Instead of ScheduledExecutorService instance, a cached pool could be
used instead and was implemented, and with `static` scope to be reused
among other future RemoteHostEndPoint instances.
- It was not clear why we would want to wait when we've Answers returned
from the remote EP, and therefore a scheduled/delayed Runnable was
not required at all for processing answers. ScheduledExecutorService
was therefore not really required, moved to ExecutorService instead.
- Another benefit of using a cached pool is that it will shutdown
threads if they are not used in 60 seconds, and they get re-used for
future runnable submissions.
- Caveat: the executor service is still unbounded, however, the use-case
that this method is used for short jobs to check upload/download
progresses fits the case here.
- Refactored CmdRunner to not use/reference objects from parent class.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-10 22:04:09 +05:30
mrunalinikankariya 32807cb887 CLOUDSTACK-9958: Include tags of resources in listUsageRecords API (#2242)
Tags field to be included in the listusagerecords response such that it can be used in billing report. E.g.
"tags":[
{"key":"city","value":"Toronto","resourcetype":"UserVm","resourceid":"a0cca906-f985-4b56-ad11-f33e59c4c733","account":"admin","domainid":"dec39eb8-4f81-11e7-8315-067fa0000031","domain":"ROOT"}
,
{"key":"region","value":"canada","resourcetype":"UserVm","resourceid":"a0cca906-f985-4b56-ad11-f33e59c4c733","account":"admin","domainid":"dec39eb8-4f81-11e7-8315-067fa0000031","domain":"ROOT"}
2017-11-07 09:52:00 +05:30
Rohit Yadav eda3b35bfa CLOUDSTACK-10012: Migrate to Embedded Jetty
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
  deprecates the need to create/store cloud.jks in conf folder and
  the db.cloud.keyStorePassphrase in db.properties file. This also
  remove the need of the --keystore-passphrase in the
  cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-03 23:57:25 +05:30
Nicolas Vazquez 1315c947c0 CLOUDSTACK-10122: Unrelated error message (#2302)
Simply remove unrelated text from error message
2017-11-03 20:33:33 +05:30
Frank Maximus d467a6b8ec CLOUDSTACK-9450: Network Offering for VPC based on DB flag
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-11-03 12:39:23 +01:00
Frank Maximus d077b3efc6
Merge pull request #2004 from nuagenetworks/feature/vr_without_public_ip
CLOUDSTACK-9832: Do not assign public IP NIC to the VPC VR when the VPC offering does not contain VpcVirtualRouter as a SourceNat provider
2017-11-02 11:56:05 +01:00
Frank Maximus 2139dbe803
Merge pull request #2238 from nuagenetworks/feature/caching_and_statistics
[CLOUDSTACK-10053] Performance improvement: caching of NuageVsp ID
2017-11-02 11:13:15 +01:00
Sigert Goeminne 0871ff9eda CLOUDSTACK-10053: Performance improvement: caching of NuageVsp ID
* VSP ID Caching
* VSP call Statistics
* 5.0 Support

Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-10-31 17:03:11 +01:00
Rohit Yadav 41fdb88970 CLOUDSTACK-10047: DVSwitch fixes and improvements (#2293)
Allow security policies to apply on port groups:
- Accepts security policies while creating network offering
- Deployed network will have security policies from the network offering
  applied on the port group (in vmware environment)
- Global settings as fallback when security policies are not defined for a network
  offering
- Default promiscuous mode security policy set to REJECT as it's the default
  for standard/default vswitch

Portgroup vlan-trunking options for dvswitch: This allows admins to define
a network with comma separated vlan id and vlan
range such as vlan://200-400,21,30-50 and use the provided vlan range to
configure vlan-trunking for a portgroup in dvswitch based environment.

VLAN overlap checks are performed for:
- isolated network against existing shared and isolated networks
- dedicated vlan ranges for the physical/public network for the zone
- shared network against existing isolated network

Allow shared networks to bypass vlan overlap checks: This allows admins
to create shared networks with a `bypassvlanoverlapcheck` API flag
which when set to 'true' will create a shared network without
performing vlan overlap checks against isolated network and against
the vlans allocated to the datacenter's physical network (vlan ranges).

Notes:
- No vlan-range overlap checks are performed when creating shared networks
- Multiple vlan id/ranges should include the vlan:// scheme prefix

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-25 15:14:42 +05:30
Mike Tutkowski 4c89b5b97a Merge branch 'pr-2081' 2017-10-18 14:25:45 -06:00
dahn a379230e8e CLOUDSTACK-9957 Annotations (#2181)
* annotations on hosts

* Adding marvin tests

* rebase error

* review comments

* context for owner

* review

* illegal entity test

* entityType check on input

* Annotation events

* rebase issues
2017-10-13 11:55:26 +02:00
Rohit Yadav 0fedbdd7a9 CLOUDSTACK-9998: Prometheus Exporter for CloudStack (#2287)
This implements a CloudStack Prometheus exporter as a plugin, that serves
metrics on a HTTP port.

New global settings:

1. prometheus.exporter.enable - (default: false), Enable the prometheus
exporter plugin, management server restart needed.
2. prometheus.exporter.port - (default: 9595), The prometheus exporter
server port.
3. prometheus.exporter.allowed.ips - (default: 127.0.0.1), List of comma
separated prometheus server ips (with no spaces) that should be allowed to
access the URLs.

The following list  of  metrics are provided  per pop (zone)  with  the exporter:
• Per host:
o CPU cores:  used, total
o CPU usage:  used, total (in MHz)
o Memory  usage:  used, total (in MiBs)
o Total VMs running on  the host
• CPU cores:  allocated (per  zone)
• CPU usage:  allocated (per  zone, in  MHz)
• Memory  usage:  allocated (per  zone, in  MiBs)
• Hosts:  online, offline,  total
• VMs: in all states -- starting, running, stopping, stopped, destroyed,
       expunging, migrating,  error, unknown
• Volumes:  ready,  destroyed,  total
• Primary Storage Pool: (Disk size) used, allocated,  unallocated,  total (in GiBs)
• Secondary Storage Pool: (Disk size) used, allocated,  unallocated,  total (in GiBs)
• Private IPs:  allocated,  total
• Public  IPs:  allocated,  total
• Shared  Network IPs:  allocated,  total
• VLANs:  allocated,  total

Additional metrics for the environment:
• Summed  domain  (level=1) limit for CPU cores
• Summed  domain  (level=1) limit for memory/ram

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-11 17:24:22 +05:30
dahn ed7811a9a2 CLOUDSTACK-10046 checksum validation for any java supported Digests-type (#2246)
* CLOUDSTACK-10046 digest helper for calculating checksums

* CLOUDSTACK-10046 cleanup unused checksum code

* CLOUDSTACK-10046 padding method proof of concept

* CLOUDSTACK-10046 only compare checksums if old value is valid

* Adding positive and negative tests for md5, sha-1 and sha-256, for xen, vmware and kvm hypervisors.
KVM Results:

 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 189, in test_02_1_create_template_with_checksum_sha1_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{sha-1}bf580a13f791d86acf3449a7b457a91a14389264" didn\'t match the given value, "{sha-1}someInvalidValue"\n']
=== TestName: test_02_1_create_template_with_checksum_sha1_negative | Status : SUCCESS ===
=== TestName: test_02_create_template_with_checksum_sha1 | Status : SUCCESS ===.
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 203, in test_03_1_create_template_with_checksum_sha256_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{SHA-256}efc03633f2b8f5db08acbcc5dc1be9028572dfd8f1c6c8ea663f0ef94b458c5" didn\'t match the given value, "{SHA-256}someInvalidValue"\n']
=== TestName: test_03_1_create_template_with_checksum_sha256_negative | Status : SUCCESS ===
=== TestName: test_03_create_template_with_checksum_sha256 | Status : SUCCESS ===
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 217, in test_04_1_create_template_with_checksum_md5_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{md5}ada77653dcf1e59495a9e1ac670ad95f" didn\'t match the given value, "{md5}someInvalidValue"\n']
=== TestName: test_04_1_create_template_with_checksum_md5_negative | Status : SUCCESS ===
=== TestName: test_04_create_template_with_checksum_md5 | Status : SUCCESS ===

* CLOUDSTACK-10046 digest helper for calculating checksums

* CLOUDSTACK-10046 cleanup unused checksum code

* CLOUDSTACK-10046 padding method proof of concept

* CLOUDSTACK-10046 only compare checksums if old value is valid

* Adding positive and negative tests for md5, sha-1 and sha-256, for xen, vmware and kvm hypervisors.
KVM Results:

 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 189, in test_02_1_create_template_with_checksum_sha1_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{sha-1}bf580a13f791d86acf3449a7b457a91a14389264" didn\'t match the given value, "{sha-1}someInvalidValue"\n']
=== TestName: test_02_1_create_template_with_checksum_sha1_negative | Status : SUCCESS ===
=== TestName: test_02_create_template_with_checksum_sha1 | Status : SUCCESS ===.
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 203, in test_03_1_create_template_with_checksum_sha256_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{SHA-256}efc03633f2b8f5db08acbcc5dc1be9028572dfd8f1c6c8ea663f0ef94b458c5" didn\'t match the given value, "{SHA-256}someInvalidValue"\n']
=== TestName: test_03_1_create_template_with_checksum_sha256_negative | Status : SUCCESS ===
=== TestName: test_03_create_template_with_checksum_sha256 | Status : SUCCESS ===
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 217, in test_04_1_create_template_with_checksum_md5_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{md5}ada77653dcf1e59495a9e1ac670ad95f" didn\'t match the given value, "{md5}someInvalidValue"\n']
=== TestName: test_04_1_create_template_with_checksum_md5_negative | Status : SUCCESS ===
=== TestName: test_04_create_template_with_checksum_md5 | Status : SUCCESS ===

* Adding additional test with no checksum added when registering template
Result:
test_05_create_template_with_no_checksum (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_05_create_template_with_no_checksum | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 1 test in 42.320s

OK

* Fixing negative tests exception handling

* Adding tests for ISO checksum validation and fixing a zero prefix failure test in templates

* CLOUDSTACK-10046 padding

* CLOUDSTACK-10046 usability additions

* yet another IDE artifact hindering checkstyle
2017-10-11 11:49:06 +02:00
Frank Maximus 1d382e0cb4 CLOUDSTACK-9832: Remove public interface from VPC Virtual Router
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9832

Detail:
When the VPC offering does not contain VpcVirtualRouter as a SourceNat provider,
then we will not add the interface in the public network to the VpcVR.

CLOUDSTACK-9832: Move isSrcNat check to VpcManager
2017-10-11 11:35:53 +02:00
dahn 7ca5b535a4 CLOUDSTACK-9899 Url validation disabling (#2074)
* CLOUDSTACK-9899 adding a global setting for not checking URLs from the MS

* CLOUDSTACK-9899 refactor HttpTemplateDownloader contructor cleanup

* CLOUDSTACK-9899 refactor HttpTemplateDownloader.download() cleanup

* CLOUDSTACK-9899 add the new config key to configurable

* CLOUDSTACK-9899 refactor download method

* CLOUDSTACK-9899 less verbose setting comment

* CLOUDSTACK-9899 debug message to indicate checking happened

* CLOUDSTACK-9899 typi flase -> false
2017-10-10 19:56:00 +02:00
Harika Punna 6bb0ca2f85 This feature separates the snapshot creation on primary and its backing up on secondary.
As part of this, a new parameter, which is optional, is added to CreateSnapshotCmd, which seperates the creation and backup.

More details in the FS-
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Separate+creation+and+backup+operations+for+a+volume+snapshot
2017-10-04 14:39:03 +05:30
Marc-Aurèle Brothier e1cff7d435 CLOUDSTACK-10083: Allow ModifySshKeys to be forwarded to the agent in maintenance mode (#2269)
Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2017-09-28 12:39:46 +05:30
SudharmaJain c670691bfb CLOUDSTACK-8865: Adding SR doesn't create Storage_pool_host_ref entry for disabled host (#876)
This causes VM deployment failure on the host that was disabled while adding the storage repository.
In the attachCluster function of the PrimaryDataStoreLifeCycle, we were only selecting hosts that are up and are in enabled state. Here if we select all up hosts, it will populate the DB properly and will fix this issue. Also added a unit test for attachCluster function.
2017-09-21 10:49:11 +05:30
sureshanaparti f2584bb9e7 CLOUDSTACK-9182: Some running VMs turned off on manual migration when auto migration failed while host preparing for maintenance. (#1252)
Fix: Block VMOperations if Host in PrepareForMaintenance mode. VM operations (Stop, Reboot, Destroy, Migrate to host) are not allowed when Host in PrepareForMaintenance mode.
2017-09-20 16:23:24 +05:30
Nathan Johnson 2bad9a6c11 CLOUDSTACK-9949: add ability to specify mac address (#2143)
Added ability to specify mac in deployVirtualMachine and
addNicToVirtualMachine api endpoints.

Validates mac address to be in the form of:
aa:bb:cc:dd:ee:ff , aa-bb-cc-dd-ee-ff , or aa.bb.cc.dd.ee.ff.

Ensures that mac address is a Unicast mac.

Ensures that the mac address is not already allocated for the
specified network.
2017-09-16 12:16:42 +05:30
mrunalinikankariya e894022790 CLOUDSTACK-9886 : After restarting cloudstack-management , It takes time to connect hosts 2017-09-11 17:08:47 +05:30
Syed Mushtaq Ahmed f5cebeb71a CLOUDSTACK-10061: When starting a VM, make sure it is attached to correct VAG when using managed storage (#2253)
This can happen when you stop a VM in one cluster and start a VM in another cluster. When the VM starts in a new cluster, we don't add a new VAG and hence it fails to start. This PR ensures that we call grantAccess to the VM that gets started which will fix the access issue.
2017-09-01 21:10:44 +02:00
niteshsarda 74fe9e386c CLOUDSTACK-10004 : On deletion, Vmware volume snapshots are left behind with message 'the snapshot has child, can't delete it on the storage' (#2188)
Snapshots are not deleted resulting unexpected storage consumption in case of VMware.

Steps to reproduce this issue :

In VMware setup, create a snapshot of volume say Snap1.
After successful creation of snapshot Snap1, create new snapshot of same volume say Snap2.snapshots
While Snap2 is in BackingUp state, delete Snap1.
Snap1 will disappear from Web UI, but when we check secondary storage, files associated with Snap1 still persists even after cleanup job is performed.
In snapshot_store_ref table in DB, Snap1 will be in ready state instead of Destroyed.
Also, in snapshots table, status of Snap1 will be Destroyed but removed column will be null and will never change to the date of snapshot removal.
Fix for this issue :

In VMware, snapshot chain is not maintained, instead full snapshot is taken every time.
So, it makes sense not to assign parent snapshot id for the snapshot. In this way, every snapshot will be individual and can be deleted successfully whenever required.
2017-09-01 11:04:48 +02:00
Rohit Yadav c0b33db5ce CLOUDSTACK-9782: Nested-oobm CloudStack plugin
Nested out-of-band management plugin to work with hosts that are VMs in
a CloudStack env.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
Daan Hoogland 693d63e7c4 CE-110 remove duplicate-unused functionality 2017-08-25 08:57:51 +02:00
Daan Hoogland 426894411e CE-110 move config to public fields 2017-08-25 08:57:51 +02:00
Nitin Kumar Maharana 1f29313071 CLOUDSTACK-10002: Restart network with cleanup spawns Redundant Routers(In Default Network Offering) (#2186)
The setter name is different from what it should be used by update call.
2017-08-22 22:48:32 +02:00
Rohit Yadav cc749cca3e Merge branch '4.10' 2017-08-22 10:23:29 +02:00