CLOUDSTACK-8793 Enable s2s VPN connection for projects
* pr/879:
CLOUDSTACK-8793 Added project id to create vpn customer gateway, and to the impl of list vpn connections and list vpn customer gateways
Signed-off-by: Remi Bergsma <github@remi.nl>
Pass LbProtocol down to the HAProxyConfiguratorThis will let us specify a new load balancer protocol (tcp-proxy) which enables HAProxy's `send-proxy` functionality.
`send-proxy` / [the PROXY protocol][1] will send the real connection origin IP through to the servers behind HAProxy, without requiring any protocol specific changes (such as HTTP header rewriting).
[1]: http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
This is also inline with what [Amazon ELB now supports][2].
[2]: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
* pr/848:
Pass LbProtocol down to the HAProxyConfigurator
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8962: Dedicated cluster is used for virtual routers that belong to non-dedicated account
Earlier the deployment planner was not handling the case of virtual routers.(In Explicit Dedication)
It was only handling for all instance VMs/user VMs.
Added code for checking the case of Virtual Routers.
* pr/945:
CLOUDSTACK-8962: Dedicated cluster is used for virtual routers that belong to non-dedicated account
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8911: VM start job got stuck in loop looking for suitable host
VM instance creation job get stuck in the loop, when VMs require local storage there are host that reached max guest limit and remain hosts does have storage available. This happens because the hosts that reach the max guest limit were not getting added to the avoid list and hence the cluster.
Verified the fix on my local setup.
Repro Steps:
1. Take an environment with single cluster and 2 hosts.
2. change the max guest limit for the hypervisor such that on one host max guest limit should reach.
3. change thresholds so that other host should not have enough storage. If required create a VM for sufficient bigger disk.
4. Now deploy a VM with local storage.
5. cluster will never be put in the avoid set and job will keep looking for suitable host.
6. once we increase the max guest limit, VM will deploy or will fail if there is a lack of storage.
* pr/895:
CLOUDSTACK-8911: VM start job got stuck in loop looking for suitable host
Signed-off-by: Remi Bergsma <github@remi.nl>
Cloudstack-8816 some of the events do not have resource uuidsThe key objects in the context map are sometimes String and sometimes object. This causes missing uuids when an entity put in the context map with key entity.toString is queried with key entity
Testing:
manually tested by deploying a vm and checked that the created events in rabbitmq now has uuids.
events before and after the change are update at https://issues.apache.org/jira/browse/CLOUDSTACK-8816?focusedCommentId=14805239
unittests
```
$ mvn -pl :cloud-api test -Dtest=CallContextTest
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.cloudstack.context.CallContextTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.152 sec - in org.apache.cloudstack.context.CallContextTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.445 s
[INFO] Finished at: 2015-09-18T14:58:53+05:30
[INFO] Final Memory: 55M/448M
[INFO] ------------------------------------------------------------------------
```
* pr/849:
CLOUDSTACK-8816 added missing events
CLOUDSTACK-8816: fixed missing resource uuid in delete network cmd
CLOUDSTACK-8816: fixed missing resource uuid in destroy vm event
Cloudstack-8816: Fixed missing resource uuid in delete snapshot events
CLOUDSTACK-8816: some of the events do not have resource uuids
CLOUDSTACK-8816: some of the events do not have resource uuids
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8917 : Instance tab takes long time to load with 12K Vmsmodified sql that is used for retrieving vm count .
In load test environment listVirtualmachine takes 8-11 sec to load. This environment has around 12k active VMs. Total number of rows is 190K.
Performance bottleneck in listVirtualmachine command is fetching the count and distinct vms.
{noformat}
// search vm details by ids
Pair<List<UserVmJoinVO>, Integer> uniqueVmPair = _userVmJoinDao.searchAndCount(sc, searchFilter);
Integer count = uniqueVmPair.second();
{noformat}
This takes 95% of the total time.
To fetch the count and distinct vms we are using below sqls.
Query 1:
{noformat}
SELECT DISTINCT(user_vm_view.id) FROM user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL ORDER BY user_vm_view.id ASC LIMIT 0, 20
{noformat}
Query 2:
select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL
Query 2 is a problematic query.
If we rewrite the query as mentioned below then it will be ~2x faster.
select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
Mysql Test result:
With 134 active Vms (total rows 349)
mysql> select count(*) from vm_instance;
+----------+
| count(*) |
+----------+
| 349 |
+----------+
1 row in set (0.00 sec)
mysql> select count(*) from user_vm_view;
+----------+
| count(*) |
+----------+
| 135 |
+----------+
1 row in set (0.02 sec)
mysql> select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL;
+--------------------+
| count(distinct id) |
+--------------------+
| 134 |
+--------------------+
1 row in set (0.02 sec)
mysql> select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
+----------+
| count(*) |
+----------+
| 134 |
+----------+
1 row in set (0.01 sec)
With 14326 active Vms (total rows 195660)
mysql> select count(*) from vm_instance;
+----------+
| count(*) |
+----------+
| 195660 |
+----------+
1 row in set (0.04 sec)
mysql> select count(*) from user_vm_view;
+----------+
| count(*) |
+----------+
| 41313 |
+----------+
1 row in set (4.55 sec)
mysql> select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL;
+--------------------+
| count(distinct id) |
+--------------------+
| 14326 |
+--------------------+
1 row in set (7.39 sec)
mysql> select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
+----------+
| count(*) |
+----------+
| 14326 |
+----------+
1 row in set (2.08 sec)
UI test Results:
Before:
After
* pr/894:
CLOUDSTACK-8917 : Instance tab takes long time to load with 12K active VM (total vms: 190K)
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
Primary Storage count for an account does not decrease when a Data Disk
is deleted belonging to the account unless the VM to which volume
belonged is destroyed
The resource counts are updated even before the disk is actually deleted
resulting in the same value.
Moved the resource counts updation to after the expunge operation as
thats when the disk is actually deleted.
all the tests in test/integration/component/test_ps_limits.py now pass
CLOUDSTACK-8964: Can't create template or volume from snapshot on KVM
* pr/954:
CLOUDSTACK-8964: Can't create template or volume from snapshot
Signed-off-by: Remi Bergsma <github@remi.nl>
uuid is missing in the first event of VM create as the entity is just
created and never put in the Context.
Added the entity uuid to context on successful creation.
This issue happens on KVM.
Normally the SSVM will process the CopyCommand from snapshot to template.
However, Ovm3HypervisorGuru chooses a KVM hypervisor to process the CopyCommand.
This is obviously wrong.
The labeling was broken. Only labels assigned at zone creation
were used, changing labels was not working. Tested with changing
a label and checking it.
As a bonus fixed the consistency of KVM in Dutch compared to other
traffic labels in dutch and copied in the OVM3 translated label
in other languages.
Earlier the deployment planner was not handling the case of virtual routers.(In Explicit Dedication)
It was only handling for all instance VMs/user VMs.
Added code for checking the case of Virtual Routers.
- The default is Accept and will be changed based on the configuration of the offering.
CLOUDSTACK-8934 - The default egress is set as Deny in the router.
- We had to change it on the Java side in order to make the apply it once the default is defined as allowed on the net offering
CLOUDSTACK-8864: Not able to add TCP port forwarding rule in VPN for specific ports
Setting port forwarding rules for port 500,1701 and 4500 after enabling VPN, gives the error message "The range specified, xxxx, conflicts with rule xxxx which has xxxx." This happens because the rules added for vpn doesn't have a matching condition to allow port forwarding rules.
Added a unit test to verify the detectRulesConflict function of FirewallManagerImpl.
* pr/851:
CLOUDSTACK-8864: Not able to add TCP port forwarding rule in VPN for specific ports
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8834: Fixed unable to download Template , when in multi zonesWe were listing image stores by zone id which was resulting in listing of only one image store
If in that image store its download state is not DOWNLOADED then download template is failing
* pr/804:
CLOUDSTACK-8834: Fixed unable to download Template , when in multi zones We were listing image stores by zone id which was resulting in listing of only one image store If in that image store its download state is not DOWNLOADED then download template is failing
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Cloudstack 8816 entityuuid missing in some of the eventsIn some of the events generated, entity uuid was missing making it difficult to find the entity. Fixed the same.
Tested it on rabbitmq instance.
There are the events before after the fix:
Before
--------------------------------------------------------------------------------
routing_key: management-server.ActionEvent.ACCOUNT-DELETE.Account.*
exchange: cloudstack-events
message_count: 2
payload:
{"eventDateTime":"2015-09-04 17:59:24 +0530","status":"Scheduled","description":"deleting User test4 (id: 28) and accountId \u003d 28","event":"ACCOUNT.DELETE","Account":"c09e2e81-8edc-4c27-b072-25005b522b63","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 304
payload_encoding: string
redelivered: False
--------------------------------------------------------------------------------
routing_key: management-server.AsyncJobEvent.complete.Account.*
exchange: cloudstack-events
message_count: 0
payload: {"cmdInfo":"{\"id\":\"9dd3abc2-3f8b-4852-aa60-a74b234acb13\",\"response\":\"json\",\"sessionkey\":\"5ig1ItP2_5v-mgY4cVJbJN5hw_w\",\"ctxDetails\":\"
{\\\"interface com.cloud.user.Account\\\":\\\"9dd3abc2-3f8b-4852-aa60-a74b234acb13\\\"}
\",\"cmdEventType\":\"ACCOUNT.DELETE\",\"expires\":\"2015-09-07T11:11:56+0000\",\"ctxUserId\":\"2\",\"signatureversion\":\"3\",\"httpmethod\":\"GET\",\"uuid\":\"9dd3abc2-3f8b-4852-aa60-a74b234acb13\",\"ctxAccountId\":\"2\",\"ctxStartEventId\":\"447\"}","instanceType":"Account","jobId":"5004989d-0cde-4922-8afa-66bf38b75ea7","status":"SUCCEEDED","processStatus":"0","commandEventType":"ACCOUNT.DELETE","resultCode":"0","command":"org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd","jobResult":"org.apache.cloudstack.api.response.SuccessResponse/null/
{\"success\":true}
","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 914
payload_encoding: string
redelivered: False
--------------------------------------------------------------------------------
After
--------------------------------------------------------------------------------
routing_key: management-server.ActionEvent.ACCOUNT-DELETE.Account.e5e2db91-414d-484c-99d5-c4e265c14ad8
exchange: cloudstack-events
message_count: 13
payload: {"eventDateTime":"2015-09-07 17:32:26 +0530","status":"Completed","description":"Successfully completed deleting account. Account Id: 45","event":"ACCOUNT.DELETE","entityuuid":"e5e2db91-414d-484c-99d5-c4e265c14ad8","entity":"com.cloud.user.Account","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 344
payload_encoding: string
redelivered: True
--------------------------------------------------------------------------------
routing_key: management-server.AsyncJobEvent.complete.Account.e5e2db91-414d-484c-99d5-c4e265c14ad8
exchange: cloudstack-events
message_count: 12
payload: {"cmdInfo":"{\"id\":\"e5e2db91-414d-484c-99d5-c4e265c14ad8\",\"response\":\"json\",\"sessionkey\":\"8AJVbn8HIpg5LZ_VaVfSPs_QN2k\",\"ctxDetails\":\"{\\\"interface com.cloud.user.Account\\\":\\\"e5e2db91-414d-484c-99d5-c4e265c14ad8\\\"}\",\"cmdEventType\":\"ACCOUNT.DELETE\",\"expires\":\"2015-09-07T12:17:42+0000\",\"ctxUserId\":\"2\",\"signatureversion\":\"3\",\"httpmethod\":\"GET\",\"uuid\":\"e5e2db91-414d-484c-99d5-c4e265c14ad8\",\"ctxAccountId\":\"2\",\"ctxStartEventId\":\"465\"}","instanceType":"Account","instanceUuid":"e5e2db91-414d-484c-99d5-c4e265c14ad8","jobId":"0bb08486-6d9f-4e9f-bfef-b7463c42e71b","status":"SUCCEEDED","processStatus":"0","commandEventType":"ACCOUNT.DELETE","resultCode":"0","command":"org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd","jobResult":"org.apache.cloudstack.api.response.SuccessResponse/null/{\"success\":true}","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 968
payload_encoding: string
redelivered: True
--------------------------------------------------------------------------------
* pr/782:
CLOUDSTACK-8816 Systemvm reboot event doesnt have uuids. Fixed the same
CLOUDSTACK-8816: Project UUID is not showing for some of operations in RabbitMQ.
CLOUDSTACK-8816: entity uuid missing in create network event
CLOUDSTACK-8816: instance uuid is missing in events for delete account
CLOUDSTACK-8816 Fixed entityUuid missing in some cases is events
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
We were listing image stores by zone id which was resulting in listing of only one image store
If in that image store its download state is not DOWNLOADED then download template is failing
CLOUDSTACK-8814 - Refactoring the configuration of Routers and VPC routers nicsHi there,
I refactored the configureDefaultNics() method in order to split the implementations for Routers and VPC Routers.
The following tests were executed:
* test_vm_life_cycle
* test_routers
* test_vpc_router_nics
* test_vpc_routers
* test_vpc_offerings
@remibergsma @bhaisaab @koushik-das @miguelaferreira @DaanHoogland @karuturi , could you please have a look/test this PR?
Thanks in advance.
Cheers,
Wilder
* pr/788:
CLOUDSTACK-8814 - Refactoring the configuration of Routers and VPC routers nics
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
context parameters is Map<object,Object>. This has been used
interchangeably with String and class object resulting the param value
not being available in some cases if its put with object and get with
class.forName()
made the put and get consistent by using Object as key everywhere.
CLOUDSTACK-8727: API call listVirtualMachines returns same keypairCurrently the user can register same key with different names. Upon listing the VM's the name which got registered first is being returned and not the actual one. Anyhow this behavior is rare and not good. I have added a UNIQUE constraint on the ssh_keypairs table and also made sure that the previous registered keys(with duplicates) get deleted.
* pr/685:
CLOUDSTACK-8727: API call listVirtualMachines returns same keypair: Corrected and test cases added
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
config: doc: add default value 10 in description for vm.network.nic.max.secondary.ipaddresses
value hardcoded by commit 4925b9f6a1
* pr/777:
config: doc: add default value 10 in description for vm.network.nic.max.secondary.ipaddresses
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-8759 - Destroying VPC router results in a new unusable VPC routerHow to reproduce the problem:
1. Stop/Destroy the VPC router
2. Add a virtual machine to one of the VPC tier - it will trigger a VPC router creation
3. Router is created, but the NICs are not configured
How to recover without this fix:
1. Stop/destroy the VPC router and restart the VPC
Side effects: private gateways could be misconfigured.
Root cause:
In the VpcNetworkHelperImpl.configureDefaultNics() method, the guest network nic was added in the map prior to the control and public NICs. The order in the map should not matter, however in the LibvirtComputingResource.createVifs() method, there is a logic that relies on the device index - the array index - in order to create the control nic. I advise a refactor on the data model in order to be able to identify the NIC type instead of relying in the array index.
An integration test was added to cover the fix:
* test_vpc_router_nics.py
Environment:
Management Server running on CentOS 7.1
KVM host running on CentOS 7.1
CloudStack Agent/Common 4.6.0-SNAPSHOT
Executing the test:
```
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone2-kvm2-ISOLATED.cfg -s -a tags=advanced,required_hardware=true component/test_vpc_router_nics.py
```
Remark: during the SSH there might be stack traces on the console due to the connection retry. It takes some time to get the PF rules in place and reach the VMs. So, just let the test run until the end.
```
Test results:
Create a vpc with two networks with two vms in each network ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 774.020s
OK
/tmp//MarvinLogs/test_vpc_router_nics_VH6E9S/results.txt (END)
```
* pr/773:
CLOUDSTACK-8759 - Fix guets nic allocation
CLOUDSTACK-8759 - Adding a marvin test in order to cover the fix
CLOUDSTACK-8759 - The guest nic has to be added after the control nic
Signed-off-by: Remi Bergsma <github@remi.nl>
Cloudstack:8647 LDAP Trust AD and AutoimportToday, CloudStack can automatically import LDAP users based on the configuration to a domain or an account. However, any new users in LDAP aren't automatically reflected. The admin has to manually import them again.
This feature enables admin to map LDAP group/OU to a CloudStack domain and any changes are reflected in ACS as well.
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/WIP%3A+LDAP%3A+Trust+AD+and+Auto+Import
testcases output:
```
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running groovy.org.apache.cloudstack.ldap.NoLdapUserMatchingQueryExceptionSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.216 sec - in groovy.org.apache.cloudstack.ldap.NoLdapUserMatchingQueryExceptionSpec
Running groovy.org.apache.cloudstack.ldap.LdapManagerImplSpec
log4j:WARN No appenders could be found for logger (org.apache.cloudstack.ldap.LdapManagerImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
using type:
using type: null
using type: TEST
using type: TEST TEST
using name:
using name: null
using accountType: -1
using accountType: 1
using accountType: 3
using accountType: 4
using accountType: 5
using accountType: 6
using accountType: 20000
using accountType: -500000
Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.387 sec - in groovy.org.apache.cloudstack.ldap.LdapManagerImplSpec
Running groovy.org.apache.cloudstack.ldap.LdapListUsersCmdSpec
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.041 sec - in groovy.org.apache.cloudstack.ldap.LdapListUsersCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapAddConfigurationCmdSpec
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in groovy.org.apache.cloudstack.ldap.LdapAddConfigurationCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapUserSpec
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec - in groovy.org.apache.cloudstack.ldap.LdapUserSpec
Running groovy.org.apache.cloudstack.ldap.LdapAuthenticatorSpec
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec - in groovy.org.apache.cloudstack.ldap.LdapAuthenticatorSpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationVOSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationVOSpec
Running groovy.org.apache.cloudstack.ldap.OpenLdapUserManagerSpec
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.094 sec - in groovy.org.apache.cloudstack.ldap.OpenLdapUserManagerSpec
Running groovy.org.apache.cloudstack.ldap.LdapDeleteConfigurationCmdSpec
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in groovy.org.apache.cloudstack.ldap.LdapDeleteConfigurationCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapUserResponseSpec
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in groovy.org.apache.cloudstack.ldap.LdapUserResponseSpec
Running groovy.org.apache.cloudstack.ldap.LdapUserManagerFactorySpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec - in groovy.org.apache.cloudstack.ldap.LdapUserManagerFactorySpec
Running groovy.org.apache.cloudstack.ldap.ADLdapUserManagerImplSpec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in groovy.org.apache.cloudstack.ldap.ADLdapUserManagerImplSpec
Running groovy.org.apache.cloudstack.ldap.LdapCreateAccountCmdSpec
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.168 sec - in groovy.org.apache.cloudstack.ldap.LdapCreateAccountCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapImportUsersCmdSpec
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 sec - in groovy.org.apache.cloudstack.ldap.LdapImportUsersCmdSpec
Running groovy.org.apache.cloudstack.ldap.LinkDomainToLdapCmdSpec
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in groovy.org.apache.cloudstack.ldap.LinkDomainToLdapCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapSearchUserCmdSpec
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in groovy.org.apache.cloudstack.ldap.LdapSearchUserCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapListConfigurationCmdSpec
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in groovy.org.apache.cloudstack.ldap.LdapListConfigurationCmdSpec
Running groovy.org.apache.cloudstack.ldap.NoSuchLdapUserExceptionSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in groovy.org.apache.cloudstack.ldap.NoSuchLdapUserExceptionSpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationResponseSpec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationResponseSpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationSpec
asserting for provider configuration: openldap
asserting for provider configuration: microsoftad
asserting for provider configuration:
asserting for provider configuration:
asserting for provider configuration: xyz
asserting for provider configuration: MicrosoftAd
asserting for provider configuration: OpenLdap
asserting for provider configuration: MicrosoftAD
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationSpec
Running groovy.org.apache.cloudstack.ldap.LdapContextFactorySpec
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.099 sec - in groovy.org.apache.cloudstack.ldap.LdapContextFactorySpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationDaoImplSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationDaoImplSpec
Running groovy.org.apache.cloudstack.ldap.LdapUtilsSpec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in groovy.org.apache.cloudstack.ldap.LdapUtilsSpec
Results :
Tests run: 156, Failures: 0, Errors: 0, Skipped: 0
```
* pr/755:
CLOUDSTACK-8647: linkdomaintoldap shouldnt fail when createuseraccount fails
CLOUDSTACK-8647 removed duplicate key in create sql of ldap_trust_map
CLOUDSTACK-8647: string formatting
CLOUDSTACK-8647: updated with review comments
CLOUDSTACK-8647: unittests for LdapAuthenticatorSpec
CLOUDSTACK-8647: formatted LdapAuthenticatorSpec
CLOUDSTACK-8647: UI for trust AD feature
CLOUDSTACK-8647 added unittests for new methods in ldapmanager
CLOUDSTACK-8647 unittests for LinkDomainToLdap api command
CLOUDSTACK-8647: fixed unittests
CLOUDSTACK-8647 support for assigning and admin to linked ldap domain
CLOUDSTACK-8647 added nested group enabled config in ldap
CLOUDSTACK-8647 added account_type to the linkDomainToLdap API
CLOUDSTACK-8647 changed the authentication flow
CLOUDSTACK-8647 added new api linkLdapToDomain
CLOUDSTACK-8647: added cmd and response class for the new api
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
- When stopping/detroying a VPC router and creating a new one via the addition os a VM, the networks are no plugged correctly
- The ETH1 ends up with the IP of the guest nic, which remains down. This causes routes problems and the VMs are not reachable via the pub IPs
- The test will create a VPC, add 2 tier, 2 VMs, ACL, PF and SSH into the VM
- Then it will stop the router, destroy the router, add another VM to 1 tier and check that we can reach all the VMs
Guys, can you review it? things need to be discussed:
(1) this supports KVM/QCOW2 only. Anyone want to implement for other Hypervisor/format ?
(2) The original data volume (on primary storage) will be removed.
(3) The script uses the default timeout in libvirtComputingResource. Do we need to add one in global configuration (like copy.volume.wait or backup.snapshot.wait, create.volume.from.snapshot.wait)
(4) In scripts/storage/qcow2/managesnapshot.sh, I use "qemu-img convert -f qcow2 -O qcow2" to copy the snapshot from secondary to primary (hence there is no base image file), instead of "cp -f", this is because convert is faster than cp in my testing.
* pr/732:
CLOUDSTACK-5863: revert volume snapshot for KVM/QCOW2
Signed-off-by: Wei Zhou <w.zhou@tech.leaseweb.com>
CLOUDSTACK-8785: Proper enforcement of retry count (max.retries) for all work type handled by HighAvailability manager
Retry count is properly enforced for all work types in HA manager. Also reorganized some of the code for easy testing.
Signed-off-by: Koushik Das <koushik@apache.org>
Made following fixes in simulator
- Support for ScaleVmCommand/NetworkRulesVmSecondaryIpCommand in resource layer
- Added support for scaling up a running VM in simulator
- Fixed some method names not following convention
In order to test PR #725 using simulator some of these changes are needed.
Based on the way HV check is present in the scale VM API, had to explicitly put simulator related check to allow support. The ideal way would be to remove all these HV specific check from code and made them some configuration (by putting them in hypervisor_capabilities table in DB). But that will be a bigger effort outside the scope of this PR.
Signed-off-by: Koushik Das <koushik@apache.org>
This reverts commit cd7218e241, reversing
changes made to f5a7395cc2.
Reason for Revert:
noredist build failed with the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloud-plugin-hypervisor-vmware: Compilation failure
[ERROR] /home/jenkins/acs/workspace/build-master-noredist/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[484,12] error: non-static variable logger cannot be referenced from a static context
[ERROR] -> [Help 1]
even the normal build is broken as reported by @koushik-das on dev list
http://markmail.org/message/nngimssuzkj5gpbz
CLOUDSTACK-8721: Fixed Setting details of VM through API results in removal of all other details except the one passed in API
* pr/673:
CLOUDSTACK-8721: Fixed Setting details of VM through API results in removal of all other details except the one passed in API
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Support for ScaleVmCommand/NetworkRulesVmSecondaryIpCommand in resource layer
- Added support for scaling up a running VM in simulator
- Fixed some method names not following convention
Fix for the NicVO.java regression.Renamed set*() methods to correct naming.
* pr/726:
Fix for the NicVO.java regression.
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8687: Prepare template only on a given storage poolUpdate prepare template api to seed/prepare a template only on a given primary storage. Currently, the prepare template api will seed/prepare a given template on all the primary storage pools in a zone. If however, a user wishes to prepare a template only a particular storage pool, it isn't possible. Updated the api to take storage pool id as an optional parameter. If the pool id is provided then the template is prepared only on the given primary storage pool. Also added unit tests to validate the functionality
* pr/715:
CLOUDSTACK-8687: Unit tests for validating the prepare template functionality. These tests validate that the templates get scheduled for seeding. Additionally, if a template is already seeded, we do not try and seed it again. Tests also validate that templates are seeded to storage pools which are available.
CLOUDSTACK-8687: Update prepare template api to seed/prepare a template only on a given primary storage. Currently, the prepare template api will seed/prepare a given template on all the primary storage pools in a zone. If however, a user wishes to prepare a template only a particular storage pool, it isn't possible. Updated the api to take storage pool id as an optional parameter. If the pool id is provided then the template is prepared only on the given primary storage pool
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
Refactored Nic.java for readability.Only changed names and reorganized them! No functional implementation change.
See issue CLOUDSTACK-8736 for ongoing effort to clean up network code.
* pr/707:
Renamed internal variables of NicProfile.java to adhere to the naming convention.
Refactored Nic.java for readability.
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
These tests validate that the templates get scheduled for seeding. Additionally,
if a template is already seeded, we do not try and seed it again. Tests also
validate that templates are seeded to storage pools which are available.
only on a given primary storage. Currently, the prepare template api
will seed/prepare a given template on all the primary storage pools in
a zone. If however, a user wishes to prepare a template only a
particular storage pool, it isn't possible. Updated the api to take
storage pool id as an optional parameter. If the pool id is provided
then the template is prepared only on the given primary storage pool
Changed methodnames according to Nic.java refactor.
Fixed NicVO.java due to regression from Nic.java refactor.
Fixed VmWareGuru.java after Nic.java refactor.
See issue CLOUDSTACK-8736 for ongoing effort to clean up network code.
Refactored NicProfile.java for readability.Only changed names and reorganized them! No functional implementation change.
See issue CLOUDSTACK-8736 for ongoing effort to clean up network code.
* pr/705:
Refactored NicProfile.java for readability.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Cloudstack 8656: do away with more silently ignoring exceptions.a lot of messages added.
some restructuring for test exception assertions and try-with-resource blocks
* pr/654: (29 commits)
CLOUDSTACK-8656: more logging instead of sysout
CLOUDSTACK-8656: use catch block for validation
CLOUDSTACK-8656: class in json specified not found
CLOUDSTACK-8656: removed unused classes
CLOUDSTACK-8656: restructure of tests
CLOUDSTACK-8656: reorganise sychronized block
CLOUDSTACK-8656: restructure tests to ensure exception throwing
CLOUDSTACK-8656: validate the throwing of ServerApiException
CLOUDSTACK-8656: logging ignored exceptions
CLOUDSTACK-8656: try-w-r removes need for empty catch block
CLOUDSTACK-8656: try-w-r instead of clunckey close-except
CLOUDSTACK-8656: deal with empty SQLException catch block by try-w-r
CLOUDSTACK-8656: unnecessary close construct removed
CLOUDSTACK-8656: message about timed buffer logging
CLOUDSTACK-8656: message about invalid number from store
CLOUDSTACK-8656: move cli test tool to separate file
CLOUDSTACK-8656: exception is the rule for some tests
CLOUDSTACK-8656: network related exception logging
CLOUDSTACK-8656: reporting ignored exceptions in server
CLOUDSTACK-8656: log in case we are on a platform not supporting UTF8
...
Signed-off-by: Remi Bergsma <github@remi.nl>
Interface changes related to CLOUDSTACK-8580See issue CLOUDSTACK-8580 and individual commits.
* pr/680:
Made interface changes related to CLOUDSTACK-8580
Signed-off-by: Remi Bergsma <github@remi.nl>
Added responses to ListCapabilities to reflect CLOUDSTACK-8580 changes.
This to add these options to the gui. See issue CLOUDSTACK-8580.
3be14e978a
Removed comments as proposed by Daan Hoogland.
See issue CLOUDSTACK-8133 for more information.
Added null check by comment of Koushik Das.
Added brackets by comment of Wido den Hollander.
Removed a call to findById() by comment of Koushik Das.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Implemented condition that only admin or owner of the template can change its permissions ..... using updateTemplatePermissions API
Consider this scenario :
In a domain, there are three User Accounts UA1, UA2,UA3
A private template is registered by UA1
Through the updateTemplatePermissions API, UA1 gives permission to both UA2 and UA3
Now, UA2, having been shared the template, can remove the permission of UA3(or add permissions to another account).
EXPECTED BEHAVIOR :
UA2 should not be able to to add/remove permissions of other accounts.
* pr/658:
Implemented condition that only admin or owner of the template can change its permissions using updateTemplatePermissions API
Signed-off-by: Remi Bergsma <github@remi.nl>
* pr/547:
CLOUDSTACK-8601. VMFS storage added as local storage can be re-added as shared storage. Fail addition of a VMFS shared storage pool in case it has already been added as local storage in CS.
Signed-off-by: Mike Tutkowski <mike.tutkowski@solidfire.com>
CLOUDSTACK-8704: Schedule restart of router VMs ahead of user VMs as part of HA
VRs are scheduled for HA ahead of user VMs.
Refer to the bug for more details.
* pr/656:
CLOUDSTACK-8704: Schedule restart of router VMs ahead of user VMs as part of HA VRs are scheduled for HA ahead of user VMs
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8711: public_ip type resource count for an account is not decremented upon IP range deletionProblem:
--------------------------
When you add an IP range and associate it to an account then resource count of public_ip will be updated to the range length.
After some time try to delete this range and the resource count of public_up for this account is not descremented and is causing account not to add any more public IPs to it once it reaches the resource limit.
RCA:
----------------
We were not decrement the count while deleting the IP range that was associated to an account.
Fix:
-------------
Up on deletion we are decrementing the resource count for public_up now.
* pr/662:
Bug-Id: CS-27335: public_ip type resource count for an account is not decremented upon IP range deletion
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8709 No out of band migrate alert for non-routers
* pr/666:
CLOUDSTACK-8709 No out of band migrate alert for non-routers
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8714 Restore VM (Re-install VM) with enable.storage.migration set to false fails
* pr/659:
Bug-ID:CS-27160: Restore VM (Re-install VM) with enable.storage.migration set to false fails, later fails to start up VM too
Signed-off-by: Remi Bergsma <github@remi.nl>
Reviewed-By: Anthony
Changes:
- Try to reuse the storage pools for READY disks if the pool fits the deployment plan
- Try to use the last_host if it has free capacity but no reserved capacity
Signed-off-by: Maneesha.P <maneesha.papireddygari@citrix.com>
Instead of putting the host to Alert state immediately, the investigators should be allowed to run for some time based on alert.wait global config.
At the end of this interval if the host state still cannot be determined then put the host in Alert. Also updated some of the log messages.
This closes#621
Also change the logging a bit so that you get useful logs when
not running on DEBUG level
Signed-off-by: Wido den Hollander <wido@widodh.nl>
This closes#580
If SSVM cannot be started then the existing entry is removed and a new SSVM gets created. Made similar changes for CPVM as well.
Also cleaned up some log messages in SSVM and CPVM manager code.
- Login is based on sessionkey HttpOnly Cookie
- ApiServlet does login verification using sessionKey from both the request cookies
and the API parameters. In both cases, if either or both are passed they should
match the sessionKey stored in the current session of the HttpRequest
- UI: it no longer needs to read or set sessionkey cookie
- UI: it no longer needs to return g_sessionKey value in the API requests, though
to support a sso mechanism g_sessionKey is still passed in the API is not null
- Secure jsessionid cookie is set to be HttpOnly and Secure
- SAML login should also set HttpOnly cookie before redirecting to UI
- SAML: listIdps & getSPMetadata APIs are readonly now, won't log out a logged in user
Performed tests (login, saml login if applicable, page refreshes, opening
multiple tabs, logout) with following combinations:
- SAML disabled, normal auth as admin, domain-admin and user
- SAML enabled, normal auth as admin, domain-admin and user; and saml sso as
admin, domain-admin and user
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This closes#574
This closes#308
- Make sure the new password replaces the old one in the queue
- Updated the patch files for XenServer
- Updated the script path on LibvirtComputing class
- Adding update_host_passwd to VRScripts
- Add implementation to CitrixUpdateHostPasswordCommandWrapper
- Improve testUpdateHostPasswordCommand() unit test on CitrixRequestWrapperTest
- Adding update_host_passwd.sh script
- Adding the host IP address as an instance variable on UpdateHostPasswordCommand
- Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
- Make sure doUpdateHostPassword() doesn't get called if flag is set to false
- Do not update XenServer hosts if the cluster ID is not informed
This closes#559
- Changed location of the update_host_passwd script
- Updated the patch files for XenServer
- Updated the script path on LibvirtComputing class
- Removed the hostIP from the LibvirtUpdateHostPasswordCommandWrapper execute() method
listing the hosts for migration for instance with tag
While preparing the suitable hosts we are accidentally removing the
incompatible (host does not have host tag) hosts from otherhost list(
incorrect use of List.retainAll).
- Modifying the LibvirtUpdateHostPasswordCommandWrapper in order to execute the script on the host
- Adding the script path to LibvirtComputingResource
- Adding the host IP address as an instance variable on UpdateHostPasswordCommand
- Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
* Move config options to SAML plugin
This moves all configuration options from Config.java to SAML auth manager. This
allows us to use the config framework.
* Make SAML2UserAuthenticator validate SAML token in httprequest
* Make logout API use ConfigKeys defined in saml auth manager
* Before doing SAML auth, cleanup local states and cookies
* Fix configurations in 4.5.1 to 4.5.2 upgrade path
* Fail if idp has no sso URL defined
* Add a default set of SAML SP cert for testing purposes
Now to enable and use saml, one needs to do a deploydb-saml after doing a deploydb
* UI remembers login selections, IDP server
- CLOUDSTACK-8458:
* On UI show dropdown list of discovered IdPs
* Support SAML Federation, where there may be more than one IdP
- New datastructure to hold metadata of SP or IdP
- Recursive processing of IdP metadata
- Fix login/logout APIs to get new interface and metadata data structure
- Add org/contact information to metadata
- Add new API: listIdps that returns list of all discovered IdPs
- Refactor and cleanup code and tests
- CLOUDSTACK-8459:
* Add HTTP-POST binding to SP metadata
* Authn requests must use either HTTP POST/Artifact binding
- CLOUDSTACK-8461:
* Use unspecified x509 cert as a fallback encryption/signing key
In case a IDP's metadata does not clearly say if their certificates need to be
used as signing or encryption and we don't find that, fallback to use the
unspecified key itself.
- CLOUDSTACK-8462:
* SAML Auth plugin should not do authorization
This removes logic to create user if they don't exist. This strictly now
assumes that users have been already created/imported/authorized by admins.
As per SAML v2.0 spec section 4.1.2, the SP provider should create authn requests using
either HTTP POST or HTTP Artifact binding to transfer the message through a
user agent (browser in our case). The use of HTTP Redirect was one of the reasons
why this plugin failed to work for some IdP servers that enforce this.
* Add new User Source
By reusing the source field, we can find if a user has been SAML enabled or not.
The limitation is that, once say a user is imported by LDAP and then SAML
enabled - they won't be able to use LDAP for authentication
* UI should allow users to pass in domain they want to log into, though it is
optional and needed only when a user has accounts across domains with same
username and authorized IDP server
* SAML users need to be authorized before they can authenticate
- New column entity to track saml entity id for a user
- Reusing source column to check if user is saml enabled or not
- Add new source types, saml2 and saml2disabled
- New table saml_token to solve the issue of multiple users across domains and
to enforce security by tracking authn token and checking the samlresponse for
the tokens
- Implement API: authorizeSamlSso to enable/disable saml authentication for a
user
- Stubs to implement saml token flushing/expiry
- CLOUDSTACK-8463:
* Use username attribute specified in global setting
Use username attribute defined by admin from a global setting
In case of encrypted assertion/attributes:
- Decrypt them
- Check signature if provided to check authenticity of message using IdP's
public key and SP's private key
- Loop through attributes to find the username
- CLOUDSTACK-8538:
* Add new global config for SAML request sig algorithm
- CLOUDSTACK-8539:
* Add metadata refresh timer task and token expiring
- Fix domain path and save it to saml_tokens
- Expire hour old saml tokens
- Refresh metadata based on timer task
- Fix unit tests
This closes#489
(cherry picked from commit 20ce346f3a)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Conflicts:
client/WEB-INF/classes/resources/messages_hu.properties
plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixCheckHealthCommandWrapper.java
plugins/user-authenticators/saml2/src/org/apache/cloudstack/api/command/SAML2LoginAPIAuthenticatorCmd.java
ui/scripts/ui-custom/login.js
- Make sure the doUpdateHostPassword() method returns the result from the answer, not always true
- Added the LibvirtUpdateHostPasswordCommandWrapper class, which will handle the changes in the KVM agent
That's will be further implemented in the future. The command will avoid the agent to complain about Unsupported command
- Added a test to make sure the current implementation of the comment works
- If changes in the future, it will also require changes. In that way, we make sure nobody will break it
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
This closes#527
- 2 years ago a change in that method caused an issue when trying to update the hosts password:
- they would be updated in the databse but not in the host it self, causing the hosts to disconnect after a management server resstart
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
- By doing so the check is simple and will require less effort when adding other hypervisors.
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
- The code was hard to maintain because updating a host or all the hosts in a cluster was handled in the same method
- Created updateHost and updateCluster password in both ResourceManager and ManagementServer interfaces/classes
- The chck for whihc method to use is done in the API level
- Started adding the support for KVM host passwd update
No API changes are needed and it will be backwards compatible.
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
- New implementation uses nanoseconds. Due to that, the places where the Profiler is used as a Monitor and/or
a stopwatch will suffer with the difference in the return
- Also added a getDuration(), which returns the time in nanoseconds in case someone wants to use it instead
- Added an extra test to check if the getDuration() works fine with nanoseconds
- Fixed the test that checks the time in milliseconds: I added an error margin to cover the test better
Signed-off-by: Daan Hoogland <daan@onecht.net>
Remi Bergsma
Kshitij Kansal
ramamurtis
Patrick Dube
Aaron Brady
Nick Livens
Rajani Karuturi
Wei Zhou
Funs Kessen
Nitin Kumar Maharana
Wilder Rodrigues
SudharmaJain
Sudhansu
Boris Schrijver
Bharat Kumar
Maneesha.P
Wido den Hollander
Ashwin Swaroop
Sanjay Tripathi
Anshul Gangwar
Kshitij Kansal
Rohit Yadav
Rene Moser
Koushik Das
Rafael Weingartner
Rajani Karuturi
radu-stefanache
Devdeep Singh
Milamber
Mike Tutkowski
Damodar
Prachi Damle
Daan Hoogland
Daan Hoogland
Jayapal
Likitha Shetty
Rene Moser
