This PR introduces several features and fixes some bugs:
- account tags feature
- fixed resource tags bugs which happened during tags search (found wrong entries because of mysql string to number translation - see #905, but this PR does more and fixes also resource access - vulnerability during list resource tags)
- some marvin improvements (speed, sanity)
Improved resource tags code:
1. Enhanced listTags security
2. Added support for account tags (account tags are required to support tags common for all users of an account)
3. Improved the tag management code (refactoring and cleanup)
Marvin:
1. Fixed Marvin wait timeout between async pools. To decrease polling interval and improve CI speed.
2. Fixed /tmp/ to /tmp in zone configuration files.
3. Fixed + to os.path.join in log class.
4. Fixed + to os.path.join in deployDataCenter class.
5. Fixed typos in tag tests.
6. Modified Tags base class delete method.
Deploy Datacenter script:
1. Improved deployDatacenter. Added option logdir to specify where script places results of evaluation.
ConfigurationManagerImpl:
1. Added logging to ConfigurationManagerImpl to log when vlan is not found. Added test stubs for tags. Found accidental exception during simulator running after CI.
tests_tags.py:
1. Fixed stale undeleted tags.
2. Changed region:India to scope:TestName.
This feature allow admins to dedicate a range of public IP addresses to the SSVM and CPVM, such that they can be subject to specific external firewall rules. The option to dedicate a public IP range to the System VMs (SSVM & CPVM) is added to the createVlanIpRange API method and the UI.
Solution:
Global setting 'system.vm.public.ip.reservation.mode.strictness' is added to determine if the use of the system VM reservation is strict (when true) or preferred (false), false by default.
When a range has been dedicated to System VMs, CloudStack should apply IPs from that range to
the public interfaces of the CPVM and the SSVM depending on global setting's value:
If the global setting is set to false: then CloudStack will use any unused and unreserved public IP
addresses for system VMs only when the pool of reserved IPs has been exhausted
If the global setting is set to true: then CloudStack will fail to deploy the system VM when the pool
of reserved IPs has been exhausted, citing the lack of available IPs.
UI Changes
Under Infrastructure -> Zone -> Physical Network -> Public -> IP Ranges, button 'Account' label is refactored to 'Set reservation'.
When that button is clicked, dialog displayed is also refactored, including a new checkbox 'System VMs' which indicates if range should be dedicated for CPVM and SSVM, and a note indicating its usage.
When clicking on button for any created range, UI dialog displayed indicates whether IP range is dedicated for system vms or not.
This extends work presented on #2048 on which the ability to extend the management range is provided.
Aim
This PR allows separating the management network subnet on which SSVM and CPVM are from the virtual routers management subnet.
Detailed use case
PCI compliance requires that network elements are defined as ‘in scope’ or ‘out of scope’, for compliance purposes. The SSVM and CPVM are both in scope as they allow public HTTP or HTTPS connections. The virtual routers have been defined as out of scope as they have been placed entirely in a firewalled network's segment. However, all of the system VM types share management network. As SSVM and CPVM are both in scope this would bring the virtual routers into scope as well, requiring individual audits of every virtual router. As this is not practical, the ‘management network’ which the SSVM and CPVM are on, and the management network which the virtual routers are on, must be separated by a firewall.
Description
By this feature it is possible to dedicate a created range for SSVM and CPVM (system vms) and provide a VLAN ID for its range.
A new boolean global configuration is added: system.vm.management.ip.reservation.mode.strictness. If enabled, the use of System VMs management IP reservation is strict, preferred if not. Default value is false (preferred).
Strict reservation: System VMs should try to get a private IP from a range marked for system vms. If not available, deployment fails
Preferred reservation: System VMS will try to get a private IP from a range marked for system vms. If not available, IP for range not marked for system vms is taken.
* Cleanup and Improve NetUtils
This class had many unused methods, inconsistent names and redundant code.
This commit cleans up code, renames a few methods and constants.
The global/account setting 'api.allowed.source.cidr.list' is set
to 0.0.0.0/0,::/0 by default preserve the current behavior and thus
allow API calls for accounts from all IPv4 and IPv6 subnets.
Users can set it to a comma-separated list of IPv4/IPv6 subnets to
restrict API calls for Admin accounts to certain parts of their network(s).
This is to improve Security. Should an attacker steal the Access/Secret key
of an account he/she still needs to be in a subnet from where accounts are
allowed to perform API calls.
This is a good security measure for APIs which are connected to the public internet.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
At present, The management IP range can only be expanded under the same subnet. According to existing range, either the last IP can be forward extended or the first IP can be backward extended. But we cannot add an entirely different range from the same subnet. So the expansion of range is subnet bound, which is fixed. But when the range gets exhausted and a user wants to deploy more system VMs, then the operation would fail. The purpose of this feature is to expand the range of management network IPs within the existing subnet. It can also delete and list the IP ranges.
Please refer the FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Expansion+of+Management+IP+Range
ISSUE :
Duplicate public VLAN for two different admin accounts.
STEPS TO REPRODUCE :
Start multiple threads for executing createVlanIpRange API.
Make sure multiple threads run in parallel.
Verify vlan table in DB, duplicate entry for same VLAN and IP address range will be encountered, just id and uuid will be different, rest all fields will have similar value.
Following entry will be observed in vlan table :
`mysql> select * from vlan where vlan_id like 'vlan://77' and removed is null;
+----+--------------------------------------+-----------+--------------+-----------------+---------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------------------+
| id | uuid | vlan_id | vlan_gateway | vlan_netmask | description | vlan_type | data_center_id | network_id | physical_network_id | ip6_gateway | ip6_cidr | ip6_range | removed | created |
+----+--------------------------------------+-----------+--------------+-----------------+---------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------------------+
| 15 | 6a205b78-d162-43e3-8da9-86a3ff60f40e | vlan://77 | 10.112.63.65 | 255.255.255.192 | 10.112.63.66-10.112.63.70 | VirtualNetwork | 1 | 200 | 200 | NULL | NULL | NULL | NULL | 2017-12-13 12:55:51 |
| 17 | ff8b5175-b247-45a5-b8d3-feb6a1ca64d0 | vlan://77 | 10.112.63.65 | 255.255.255.192 | 10.112.63.66-10.112.63.70 | VirtualNetwork | 1 | 200 | 200 | NULL | NULL | NULL | NULL | 2017-12-13 12:55:51 |
+----+--------------------------------------+-----------+--------------+-----------------+---------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------------------+
The description of the parameter was saying that the parameter has to be bigger than 6, however, in the code we were only accepting values bigger than 10. This PR changes the validation method to accept any number >= 6. We also change other inconsistencies in error messages presented to users. in near by validations.
Allow security policies to apply on port groups:
- Accepts security policies while creating network offering
- Deployed network will have security policies from the network offering
applied on the port group (in vmware environment)
- Global settings as fallback when security policies are not defined for a network
offering
- Default promiscuous mode security policy set to REJECT as it's the default
for standard/default vswitch
Portgroup vlan-trunking options for dvswitch: This allows admins to define
a network with comma separated vlan id and vlan
range such as vlan://200-400,21,30-50 and use the provided vlan range to
configure vlan-trunking for a portgroup in dvswitch based environment.
VLAN overlap checks are performed for:
- isolated network against existing shared and isolated networks
- dedicated vlan ranges for the physical/public network for the zone
- shared network against existing isolated network
Allow shared networks to bypass vlan overlap checks: This allows admins
to create shared networks with a `bypassvlanoverlapcheck` API flag
which when set to 'true' will create a shared network without
performing vlan overlap checks against isolated network and against
the vlans allocated to the datacenter's physical network (vlan ranges).
Notes:
- No vlan-range overlap checks are performed when creating shared networks
- Multiple vlan id/ranges should include the vlan:// scheme prefix
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Unable to create service offering with networkrate=0(Unlimited network throttling) with an error "Failed to create service offering xxxxxxx: specify the network rate value more than 0".
CLOUDSTACK-5806: add presetup to storage types that support over provisioning
Ideally this should be configurable via global settings
* pr/1958:
CLOUDSTACK-5806: add presetup to storage types that support over provisioning Ideally this should be configurable via global settings
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
This PR adds an ability to Pass a new parameter, locationType,
to the “createSnapshot” API command. Depending on the locationType,
we decide where the snapshot should go in case of managed storage.
There are two possible values for the locationType param
1) `Standard`: The standard operation for managed storage is to
keep the snapshot on the device. For non-managed storage, this will
be to upload it to secondary storage. This option will be the
default.
2) `Archive`: Applicable only to managed storage. This will
keep the snapshot on the secondary storage. For non-managed
storage, this will result in an error.
The reason for implementing this feature is to avoid a single
point of failure for primary storage. Right now in case of managed
storage, if the primary storage goes down, there is no easy way
to recover data as all snapshots are also stored on the primary.
This features allows us to mitigate that risk.
CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UIJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9438
### Introduction
From #1361 it was possible to configure NFS version for secondary storage mount.
However, changing NFS version requires inserting an new detail on `image_store_details` table, with `name = 'nfs.version'` and `value = X` where X is desired NFS version, and then restarting management server for changes to take effect.
Our improvement aims to make NFS version changeable from UI, instead of previously described workflow.
### Proposed solution
Basically, NFS version is defined as an image store ConfigKey, this implied:
* Adding a new Config scope: **ImageStore**
* Make `ImageStoreDetailsDao` class to extend `ResourceDetailsDaoBase` and `ImageStoreDetailVO` implement `ResourceDetail`
* Insert `'display'` column on `image_store_details` table
* Extending `ListCfgsCmd` and `UpdateCfgCmd` to support **ImageStore** scope, which implied:
** Injecting `ImageStoreDetailsDao` and `ImageStoreDao` on `ConfigurationManagerImpl` class, on `cloud-server` module.
### Important
It is important to mention that `ImageStoreDaoImpl` and `ImageStoreDetailsDaoImpl` classes were moved from `cloud-engine-storage` to `cloud-engine-schema` module in order to Spring find those beans to inject on `ConfigurationManagerImpl` in `cloud-server` module.
We had this maven dependencies between modules:
* `cloud-server --> cloud-engine-schema`
* `cloud-engine-storage --> cloud-secondary-storage --> cloud-server`
As `ImageStoreDaoImpl` and `ImageStoreDetailsDao` were defined in `cloud-engine-storage`, and they needed in `cloud-server` module, to be injected on `ConfigurationManagerImpl`, if we added dependency from `cloud-server` to `cloud-engine-storage` we would introduce a dependency cycle. To avoid this cycle, we moved those classes to `cloud-engine-schema` module
* pr/1615:
CLOUDSTACK-9438: Fix for CLOUDSTACK-9252 - Make NFS version changeable in UI
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9047 rename enumsmake enums adhere to best practice naming conventions
* pr/1049:
CLOUDSTACK-9046 rename enums to adhere to naming conventions
CLOUDSTACK-9046 renamed enums in kvm plugin
CLOUDSTACK-9047 use 'State's only with context there are more types called 'State' (or to be called so but now 'state') So remove imports and prepend their enclosing class/context to them.
Signed-off-by: Daan Hoogland <daan@onecht.net>
* 4.6:
CLOUDSTACK-9075 - Uses the same vlan since it should have been already released
CLOUDSTACK-9075 - Adds VPC static routes test
CLOUDSTACK-9075 - Covers Private GW ACL with Redundant VPCs
CLOUDSTACK-9075 - Add method to get list of Physical Networks per zone
CLOUDSTACK-6276 Removing unused parameter in integration test for projects
CLOUDSTACK-6276 Removing unused parameter in integration test
CLOUDSTACK-6276 Fixing affinity groups for projects
CLOUDSTACK-8868: use PasswordGenerator.generateRandomPassword() to generate systemvm passwordsgenerateRandomPassword() is supposed to create root user passwords. Right now it is only used on the guest VMs. The format of the passwords it creates are of the form "random 3-character string with a lowercase character, uppercase character, and a digit" + random n-character string with only lowercase characters".
For whatever reason it was that we use generateRandomPassword() for guest VM root user passwords(maybe more secure?) we should use the same function for system VM root user passwords.
Tests:
manually tested that password is generated in proper format and am able to login to cpvm with the new password. ex: zD2ztm, tR8snbwhq
```
$ mvn -pl server test -Dtest=ConfigurationServerImplTest#testUpdateSystemvmPassword
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.cloud.server.ConfigurationServerImplTest
log4j:WARN No appenders could be found for logger (com.cloud.utils.crypt.EncryptionSecretKeyChecker).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.487 sec - in com.cloud.server.ConfigurationServerImplTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.781 s
[INFO] Finished at: 2015-09-16T14:17:07+05:30
[INFO] Final Memory: 60M/466M
[INFO] ------------------------------------------------------------------------
```
* pr/841:
CLOUDSTACK-8868: change the default vm.password.length to 10
CLOUDSTACK-8868: use same method to generate passwords for system/guest vms
removed commented code
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.7] CLOUDSTACK-8958: add dedicated ips to domain (account for now)For now, we dedicate ip pool to account, however, other accounts in the same domain cannot fetch the ip from this ip pool.
By dedicating ip pool to domain, accounts in the domain can fetch the public ip from same ip pool.
* pr/1007:
CLOUDSTACK-8958: throw an exception if project account cannot be found
CLOUDSTACK-8958: add dedicated ips to domain (account for now)
Signed-off-by: Remi Bergsma <github@remi.nl>
Removed unnecessary @Local annotations and their respective importsFollowing @rafaelweingartner 's findings in PR #714 that many spring beans contained an @Local annotation, we've decided to remove said annotations and their imports from the ComponentLifecycleBase class and its subclasses seeking a reduction of a few hundred lines of useless code.
I had already opened a pull request for this (#853) but at some point my commit disappeared from the PR entirely, showing no new changes in code, which caused it to be merged automatically (with no changes).
* pr/1102:
Removed unnecessary @Local annotations and their respective imports from the ComponentLifecycleBase class and its subclasses.
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.7] CLOUDSTACK-8951: validation for config param "remote.access.vpn.psk.length"throwing error for value < 8 and value > 256
right now, 8, 256 are hardcoded in the code. They should be moved to a
constant and has to be reused everywhere.
will update with test cases/testing later.
* pr/929:
CLOUDSTACK-8951: validation for "remote.access.vpn.psk.length"
Signed-off-by: Remi Bergsma <github@remi.nl>
Bitworks Software, Ltd
Nicolas Vazquez
dahn
Wido den Hollander
Sigert Goeminne
Nitin Kumar Maharana
niteshsarda
Frank Maximus
Rafael Weingärtner
Frank Maximus
Sigert Goeminne
Rohit Yadav
Rohit Yadav
Wei Zhou
SowjanyaPatha
Bharat Kumar
Nitin Kumar Maharana
Rajani Karuturi
subhash_y
Rajani Karuturi
Abhinandan Prateek
Priyank Parihar
Jayapal
pallavi_dumbre
Suresh Kumar Anaparti
Syed
Mike Tutkowski
jeff
Daan Hoogland
Remi Bergsma
Patrick Dube
cirstofolini
Rajani Karuturi