* 4.6:
CLOUDSTACK-9022: move storage.cleanup related global configurations to StorageManager
CLOUDSTACK-9022: keep Destroyed volumes for sometime
Conflicts:
server/src/com/cloud/storage/StorageManagerImpl.java
CLOUDSTACK-8868: use PasswordGenerator.generateRandomPassword() to generate systemvm passwordsgenerateRandomPassword() is supposed to create root user passwords. Right now it is only used on the guest VMs. The format of the passwords it creates are of the form "random 3-character string with a lowercase character, uppercase character, and a digit" + random n-character string with only lowercase characters".
For whatever reason it was that we use generateRandomPassword() for guest VM root user passwords(maybe more secure?) we should use the same function for system VM root user passwords.
Tests:
manually tested that password is generated in proper format and am able to login to cpvm with the new password. ex: zD2ztm, tR8snbwhq
```
$ mvn -pl server test -Dtest=ConfigurationServerImplTest#testUpdateSystemvmPassword
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.cloud.server.ConfigurationServerImplTest
log4j:WARN No appenders could be found for logger (com.cloud.utils.crypt.EncryptionSecretKeyChecker).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.487 sec - in com.cloud.server.ConfigurationServerImplTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.781 s
[INFO] Finished at: 2015-09-16T14:17:07+05:30
[INFO] Final Memory: 60M/466M
[INFO] ------------------------------------------------------------------------
```
* pr/841:
CLOUDSTACK-8868: change the default vm.password.length to 10
CLOUDSTACK-8868: use same method to generate passwords for system/guest vms
removed commented code
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9080: Resource limits for Primary arent respected during attachprimary store resource limit check is not performed while attaching a
volume to a vm. Added them same.
Also added a marvin test case to verify the same.
Testing:
BEFORE
No error is shown in UI when trying to attach a volume even after reaching the resource limits.
```
mysql> select * from resource_limit where type="primary_storage";
+----+-----------+------------+-----------------+-------------+
| id | domain_id | account_id | type | max |
+----+-----------+------------+-----------------+-------------+
| 10 | NULL | 4 | primary_storage | 21474836480 |
+----+-----------+------------+-----------------+-------------+
1 row in set (0.00 sec)
mysql> select * from resource_count where account_id=4 and type='primary_storage';
+----+------------+-----------+-----------------+-------------+
| id | account_id | domain_id | type | count |
+----+------------+-----------+-----------------+-------------+
| 63 | 4 | NULL | primary_storage | 48318382080 |
+----+------------+-----------+-----------------+-------------+
1 row in set (0.00 sec)
```
AFTER
Following error message is shown in UI and the volume is not attached
The resource limits stays the same
```
mysql> select * from resource_limit where type="primary_storage";
+----+-----------+------------+-----------------+-------------+
| id | domain_id | account_id | type | max |
+----+-----------+------------+-----------------+-------------+
| 10 | NULL | 4 | primary_storage | 21474836480 |
+----+-----------+------------+-----------------+-------------+
1 row in set (0.01 sec)
mysql> select * from resource_count where account_id=4 and type='primary_storage';
+----+------------+-----------+-----------------+-------------+
| id | account_id | domain_id | type | count |
+----+------------+-----------+-----------------+-------------+
| 63 | 4 | NULL | primary_storage | 48318382080 |
+----+------------+-----------+-----------------+-------------+
1 row in set (0.00 sec)
```
Marvin test: nosetests --with-marvin --marvin-config=setup/dev/advanced.cfg --zone=xen-zone0 --hypervisor=xenserver test/integration/component/test_ps_resource_limits_volume.py
before the change
```
# do ... === TestName: test_attach_volume_exceeding_primary_limits | Status : FAILED ===
AssertionError: Resource count 23 should match with the expected resource count 22\n
```
After the change
```
# do ... === TestName: test_attach_volume_exceeding_primary_limits | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 1178.354s
OK
```
* pr/1107:
CLOUDSTACK-9080: Resource limits for Primary arent respected during attach.
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8866 : restart.retry.interval is being used instead of migrate.retry.interval during host maintenanceThe frequency at which Cloudstack tries to migrate the VMs is currently controlled by the global parameter "restart.retry.interval" which has a default value of 600 seconds or 10 minutes.This has to be changed to use "migrate.retry.interval" which by default is 120 seconds or 2 minutes . Cloudstack uses restart.retry.interval for all operations-migrate,restart,stop,destroy.
* pr/834:
CLOUDSTACK-8866 : restart.retry.interval is being used instead of migrate.retry.interval during host maintenance
Signed-off-by: Daan Hoogland <daan@onecht.net>
* 4.6:
CLOUDSTACK-9020: UI enhancements from metrics view
CLOUDSTACK-9064: The users should be able to create multiple Guest Shared Networks in same Vlan ID, same Physical Network and same network, just with a different IP ranges.
CLOUDSTACK-9078: Gave scripts executable permissions.
CLOUDSTACK-9076: Changed ownership of directory /var/lib/cloudstack to cloud.
Cannot list vlanipranges by keyword
[4.6] Cannot list vlanipranges by keywordBefore change:
cloudmonkey> list vlanipranges keyword=118
: Caught: com.mysql.jdbc.JDBC4PreparedStatement@18f36b6e: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.description, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan WHERE ( OR vlan.description LIKE ** NOT SPECIFIED ** ) AND vlan.removed IS NULL ORDER BY vlan.id ASC LIMIT 0, 500
After change:
cloudmonkey> list vlanipranges keyword='118'
count = 1
vlaniprange:
id = 0d80fd9c-cd6b-4f99-96c6-261420e75f58
account = system
domain = ROOT
domainid = 2044762d-c4a5-11e3-8379-005056ac4490
......
* pr/1085:
Cannot list vlanipranges by keyword
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.7] CLOUDSTACK-8958: add dedicated ips to domain (account for now)For now, we dedicate ip pool to account, however, other accounts in the same domain cannot fetch the ip from this ip pool.
By dedicating ip pool to domain, accounts in the domain can fetch the public ip from same ip pool.
* pr/1007:
CLOUDSTACK-8958: throw an exception if project account cannot be found
CLOUDSTACK-8958: add dedicated ips to domain (account for now)
Signed-off-by: Remi Bergsma <github@remi.nl>
Removal of DefaultUserAuthenticator empty class.The DefaultUserAuthenticator is an empty class, extending from the
AdapterBase and implementing the UserAuthenticator interface. The class
is not being used as a marker and it's sole use is to be extended by
other UserAuthenticators. Noticing that the class had no purpose, I
removed it and made it's children extend from it's superclass and
implement it's interface instead. Also, I removed the @Local tags from
those classes, since EJB is not used anymore.
* pr/1100:
Removal of DefaultUserAuthenticator empty class
Signed-off-by: Remi Bergsma <github@remi.nl>
* 4.6:
Fix event UUIDS missing on event bus
Add select template dropdown when reinstall VM
CLOUDSTACK-9068: Listing Port Forwarding Rules take too much time to load
The DefaultUserAuthenticator is an empty class, extending the
AdapterBase and implementing the UserAuthenticator interface. The class
is only being used as a marker and it's sole use is to be extended by
other UserAuthenticators. Noticing that the class had no purpose, I
removed it and made it's children extend from it's superclass and
implement it's interface instead. Also, I removed the @Local tags from
those classes, since EJB is not used anymore.
The fixing of CLOUDSTACK-8816 introduced a regression that removed the
first class entities in the event bus description property. This is
because everything was changed to use the Class as a key... Everything
but the populateFirstClassEntities method in ActionEventUtils.
primary store resource limit check is not performed while attaching a
volume to a vm. Added them same.
Also added a marvin test case to verify the same.
CLOUDSTACK-8485: listAPIs are taking too long to return results- Removed regex. based search/replace of sensitive data on API response introduced as part of commit b0c6d47347
- Added new response serializer to skip sensitive data from getting logged based on annotation present in resposne object fields
- Added annotation (@LogLevel(Log4jLevel.Off)) to sensitive response object fields
Ran the following tests on simulator:
test_vm_life_cycle.py
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 10 tests in 306.429s
OK
test_volumes.py
Download a Volume attached to a VM ... === TestName: test_03_download_attached_volume | Status : SUCCESS ===
ok
Delete a Volume attached to a VM ... === TestName: test_04_delete_attached_volume | Status : SUCCESS ===
ok
Detach a Volume attached to a VM ... === TestName: test_05_detach_volume | Status : SUCCESS ===
ok
Delete a Volume unattached to an VM ... === TestName: test_09_delete_detached_volume | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 4 tests in 184.132s
OK
test_network.py
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 4 tests in 783.726s
OK
test_routers.py
Test router internal advanced zone ... SKIP: Marvin configuration has no host credentials to check router services
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 7 tests in 42.958s
OK (SKIP=1)
test_global_settings.py
test update configuration setting at zone level scope ... === TestName: test_UpdateConfigParamWithScope | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 0.127s
OK
test_resource_detail.py
Test volume detail ... === TestName: test_01_updatevolumedetail | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 11.492s
OK
* pr/1021:
CLOUDSTACK-8485: listAPIs are taking too long to return results - Removed regex. based search/replace of sensitive data on API response introduced as part of commit b0c6d47347 - Added new response serializer to skip sensitive data from getting logged based on annotation present in resposne object fields - Added new parameter 'isSensitive' to @Param for marking a field as sensitive in response objects
Signed-off-by: Remi Bergsma <github@remi.nl>
Removed unnecessary @Local annotations and their respective importsFollowing @rafaelweingartner 's findings in PR #714 that many spring beans contained an @Local annotation, we've decided to remove said annotations and their imports from the ComponentLifecycleBase class and its subclasses seeking a reduction of a few hundred lines of useless code.
I had already opened a pull request for this (#853) but at some point my commit disappeared from the PR entirely, showing no new changes in code, which caused it to be merged automatically (with no changes).
* pr/1102:
Removed unnecessary @Local annotations and their respective imports from the ComponentLifecycleBase class and its subclasses.
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.7] CLOUDSTACK-8951: validation for config param "remote.access.vpn.psk.length"throwing error for value < 8 and value > 256
right now, 8, 256 are hardcoded in the code. They should be moved to a
constant and has to be reused everywhere.
will update with test cases/testing later.
* pr/929:
CLOUDSTACK-8951: validation for "remote.access.vpn.psk.length"
Signed-off-by: Remi Bergsma <github@remi.nl>
update for comment in cancelMaintin on start VRchange comment
//if the instance is of type ssvm, call the ssvm manager
to
//if the instance is of type domain router vm, call the network manager
* pr/1065:
update for comment in cancelMaintin on start VR
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9062: Improve S3 implementation.The S3 implementation is far from finished, this commit focuses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimizations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
Please review thoroughly, both code inspection and (automated) integration tests. Currently no integration tests are available specifically for S3. Therefore the implementation is needed to be tested manually, for now...
What I tested:
- Greenfield install -> will download latest systemvm template automatically to S3.
- Upload a template/iso
- Download a template/iso
- Restart of management server -> list available templates -> doesn't download them again if available.
* pr/1083:
CLOUDSTACK-9062: Improve S3 implementation.
Signed-off-by: Remi Bergsma <github@remi.nl>
* 4.6:
CLOUDSTACK-9053 security upgrade as per COLLECTIONS-580
CLOUDSTACK-9055: fix NPE in updating Redundant State of VPC networks
CLOUDSTACK-9057 remove old system vm upgrade code
- Removed regex. based search/replace of sensitive data on API response introduced as part of commit b0c6d47347
- Added new response serializer to skip sensitive data from getting logged based on annotation present in resposne object fields
- Added new parameter 'isSensitive' to @Param for marking a field as sensitive in response objects
The S3 implementation is far from finished, this commit focusses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimalisations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
Before change:
cloudmonkey> list vlanipranges keyword=118
: Caught: com.mysql.jdbc.JDBC4PreparedStatement@18f36b6e: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.description, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan WHERE ( OR vlan.description LIKE ** NOT SPECIFIED ** ) AND vlan.removed IS NULL ORDER BY vlan.id ASC LIMIT 0, 500
After change:
cloudmonkey> list vlanipranges keyword='118'
count = 1
vlaniprange:
id = 0d80fd9c-cd6b-4f99-96c6-261420e75f58
account = system
domain = ROOT
domainid = 2044762d-c4a5-11e3-8379-005056ac4490
......
generateRandomPassword() is supposed to create root user passwords.
Right now it is only used on the guest VMs. The format of the passwords
it creates are of the form "random 3-character string with a lowercase
character, uppercase character, and a digit" + random n-character string
with only lowercase characters".
For whatever reason it was that we use generateRandomPassword() for
guest VM root user passwords(maybe more secure?) we should use the same
function for system VM root user passwords.
throwing error for value < 8 and value > 256
right now, 8, 256 are hardcoded in the code. They should be moved to a
constant and has to be reused everywhere.
CLOUDSTACK-9006 - ListTemplates API returns result in inconsistent order when called concurrentlyThe order of templates returned in the response is based on a field called sortkey and by default value for the field is set to 0.
With more than 1000 templates, we tried listing the templates with different page sizes concurrently, and we noticed the results being inconsistent.
Thus we added a secondary order by clause to list templates call on tempZonePair column to make sure the results are consistent.
The addOrderby method of Filter class was also not appending , if we added more orderby clauses.
* pr/1009:
CLOUDSTACK-9006 - ListTemplates API returns result in inconsistent order when called concurrently
CLOUDSTACK-9006 - ListTemplates API returns result in inconsistent order when called concurrently
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8902 Restart Network fails in EIP/ELB zoneThe restart network was failing when using external loadbalencer. The failure was because of a number format exception. When BroadcastDomainType.getValue(guestConfig.getBroadcastUri() is executed this returns a string untagged. We were trying to parse this as long so there was a number pointer exception.
This happens only when the vlan uri is vlan://untagged. in other cases were there is a number instead of untagged (vlan tag) this used to succeed. Although we were trying to convert the number to long we were not really using it. we were converting the number to long and then back to string when creating the IpAddressTo. so I removed this unnecessary conversion in this case for fixing the issue at hand.
I did a manual restart of the network and checked for this number format exception in a EIP/ELB setup.
* pr/898:
CLOUDSTACK-89027 Restart Network fails in EIP/ELB zone
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9016 Fail to create VM instance within VPCbug link https://issues.apache.org/jira/browse/CLOUDSTACK-9016.
CS doe not allocate the ip of the form x.x.x.1 to a guest VM. We seem to incorrectly assume that the first ip in the subnet belongs to the gateway.
* pr/1020:
CLOUDSTACK-9016: Deploy vm with gateway ip address in VPC
CLOUDSTACK-9016 Fail to create VM instance within VPC
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACk-9002: VM deployment is successful even when dhcp entry command fails - Fixed
Reason: The return value of the call to accept() function in the applyRules() function of BasicNetworkTopology.java was not checked for success or failure. As a result even if it fails, exception was not thrown and VM deployment went ahead without any errors.
Fix: Added the necessary checks.
* pr/995:
CLOUDSTACk-9002: VM deployment is successful even when dhcp entry command fails - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8940: Wrong value is inserted into nics table netmask field when creating a VM - Fixed
Problem: When creating a VM in shared network with no service, the value of netmask is added in the table in the CIDR format unlike other cases where it is added as normal string in the format xxx.xxx.xxx.xxx. The netmask column in the nics table has a length of 15 chars which gets violated if the CIDR exceeds it(Max CIDR length can be 18).
Fix: Before storing the netmask convert from CIDR to native format.
* pr/916:
CLOUDSTACK-8940: Wrong value is inserted into nics table netmask field when creating a VM - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
Fixed: Network Update from RVR offering to Standalone offering failsProblem: Moving a RVR network offering to standalone makes the status of VR's as UNKNOWN and Redundant Router marked with YES.
Fix: The network's isRedundant was not getting updated.
* pr/818:
CLOUDSTACK-8844: Network Update from RVR offering to Standalone offering fails - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8793 Enable s2s VPN connection for projects
* pr/879:
CLOUDSTACK-8793 Added project id to create vpn customer gateway, and to the impl of list vpn connections and list vpn customer gateways
Signed-off-by: Remi Bergsma <github@remi.nl>
Daan Hoogland
Remi Bergsma
Wei Zhou
weingartner
jeff
Rajani Karuturi
cirstofolini
Koushik Das
Boris Schrijver
Rajani Karuturi
Daan Hoogland
wenwenxiong
Bharat Kumar
Kshitij Kansal
ramamurtis
