CLOUDSTACK-8814 - Refactoring the configuration of Routers and VPC routers nicsHi there,
I refactored the configureDefaultNics() method in order to split the implementations for Routers and VPC Routers.
The following tests were executed:
* test_vm_life_cycle
* test_routers
* test_vpc_router_nics
* test_vpc_routers
* test_vpc_offerings
@remibergsma @bhaisaab @koushik-das @miguelaferreira @DaanHoogland @karuturi , could you please have a look/test this PR?
Thanks in advance.
Cheers,
Wilder
* pr/788:
CLOUDSTACK-8814 - Refactoring the configuration of Routers and VPC routers nics
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
context parameters is Map<object,Object>. This has been used
interchangeably with String and class object resulting the param value
not being available in some cases if its put with object and get with
class.forName()
made the put and get consistent by using Object as key everywhere.
sysctl: don't modify /etc/sysctl.confTo configure firewall rules, CloudStack modifies `/etc/sysctl.conf` and
execute those modifications. This may be harmful for several reasons:
1. `/etc/sysctl.conf` may be managed by some configuration management
system. Such a system will constantly restore the previous version.
2. `/etc/sysctl.conf` may contain additional properties that have been
changed later by some system administrator (for example, once a
firewall has been configured, forwarding may have been activated
while it is disabled in `/etc/sysctl.conf`). Executing the file
again at a later time may disrupt the system.
3. Entries are added again and again. `/etc/sysctl.conf` will contain
the same directives repeated several times.
Using a configuration file is not needed as `sysctl` is able to directly
modify sysctl values with `-w` flag.
Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* pr/776:
sysctl: don't modify /etc/sysctl.conf
Signed-off-by: Wido den Hollander <wido@widodh.nl>
CLOUDSTACK-8727: API call listVirtualMachines returns same keypairCurrently the user can register same key with different names. Upon listing the VM's the name which got registered first is being returned and not the actual one. Anyhow this behavior is rare and not good. I have added a UNIQUE constraint on the ssh_keypairs table and also made sure that the previous registered keys(with duplicates) get deleted.
* pr/685:
CLOUDSTACK-8727: API call listVirtualMachines returns same keypair: Corrected and test cases added
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
CLOUDSTACK-8688 - default policies for INPUT and FORWARD should be set to DROP instead of ACCEPT
- In order to be able to access the routers via the link local interface, we have to add a rules with NEW and ESTABLISHED state
* pr/765:
CLOUDSTACK-8688 - Adding Marvin tests in order to cover the fixes applied
CLOUDSTACK-8688 - default policies for INPUT and FORWARD should be set to DROP instead of ACCEPT
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
config: doc: add default value 10 in description for vm.network.nic.max.secondary.ipaddresses
value hardcoded by commit 4925b9f6a1
* pr/777:
config: doc: add default value 10 in description for vm.network.nic.max.secondary.ipaddresses
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
debian: Updated dependencies for PackagesA few dependencies have been updated to their latest version and some
have been removed.
The ordering for some dependencies has been changed so that we will
depend on Java 8 over Java 7 when doing a new install.
* pr/783:
debian: Updated dependencies for Packages
Signed-off-by: Wido den Hollander <wido@widodh.nl>
- Changing refactored the utils.get_process_status() function
- Adding 2 tests: test_01_single_VPC_iptables_policies and test_02_routervm_iptables_policies
A few dependencies have been updated to their latest version and some
have been removed.
The ordering for some dependencies has been changed so that we will
depend on Java 8 over Java 7 when doing a new install.
Fixed issue: UI labels are display with key. ex: label.yesdue to missing comma in the jsp, all the labels below it are broken leading to showing keys instead of strings in UI
* pr/781:
Fixed issue: UI labels are display with key. ex: label.yes
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
Fixing findbugs error due to PR #755 mergeissue: Boxing/unboxing to parse a primitive
* pr/779:
Fixing findbugs error due to PR #755 merge
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-8759 - Destroying VPC router results in a new unusable VPC routerHow to reproduce the problem:
1. Stop/Destroy the VPC router
2. Add a virtual machine to one of the VPC tier - it will trigger a VPC router creation
3. Router is created, but the NICs are not configured
How to recover without this fix:
1. Stop/destroy the VPC router and restart the VPC
Side effects: private gateways could be misconfigured.
Root cause:
In the VpcNetworkHelperImpl.configureDefaultNics() method, the guest network nic was added in the map prior to the control and public NICs. The order in the map should not matter, however in the LibvirtComputingResource.createVifs() method, there is a logic that relies on the device index - the array index - in order to create the control nic. I advise a refactor on the data model in order to be able to identify the NIC type instead of relying in the array index.
An integration test was added to cover the fix:
* test_vpc_router_nics.py
Environment:
Management Server running on CentOS 7.1
KVM host running on CentOS 7.1
CloudStack Agent/Common 4.6.0-SNAPSHOT
Executing the test:
```
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone2-kvm2-ISOLATED.cfg -s -a tags=advanced,required_hardware=true component/test_vpc_router_nics.py
```
Remark: during the SSH there might be stack traces on the console due to the connection retry. It takes some time to get the PF rules in place and reach the VMs. So, just let the test run until the end.
```
Test results:
Create a vpc with two networks with two vms in each network ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 774.020s
OK
/tmp//MarvinLogs/test_vpc_router_nics_VH6E9S/results.txt (END)
```
* pr/773:
CLOUDSTACK-8759 - Fix guets nic allocation
CLOUDSTACK-8759 - Adding a marvin test in order to cover the fix
CLOUDSTACK-8759 - The guest nic has to be added after the control nic
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8678: Reserve RAM for KVM host OSUse host.reserved.ram.mb agent property to modify total system RAM
before reporting to management server.
* pr/766:
CLOUDSTACK-8678: Reserve RAM for KVM host OS
Signed-off-by: Remi Bergsma <github@remi.nl>
Cloudstack:8647 LDAP Trust AD and AutoimportToday, CloudStack can automatically import LDAP users based on the configuration to a domain or an account. However, any new users in LDAP aren't automatically reflected. The admin has to manually import them again.
This feature enables admin to map LDAP group/OU to a CloudStack domain and any changes are reflected in ACS as well.
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/WIP%3A+LDAP%3A+Trust+AD+and+Auto+Import
testcases output:
```
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running groovy.org.apache.cloudstack.ldap.NoLdapUserMatchingQueryExceptionSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.216 sec - in groovy.org.apache.cloudstack.ldap.NoLdapUserMatchingQueryExceptionSpec
Running groovy.org.apache.cloudstack.ldap.LdapManagerImplSpec
log4j:WARN No appenders could be found for logger (org.apache.cloudstack.ldap.LdapManagerImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
using type:
using type: null
using type: TEST
using type: TEST TEST
using name:
using name: null
using accountType: -1
using accountType: 1
using accountType: 3
using accountType: 4
using accountType: 5
using accountType: 6
using accountType: 20000
using accountType: -500000
Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.387 sec - in groovy.org.apache.cloudstack.ldap.LdapManagerImplSpec
Running groovy.org.apache.cloudstack.ldap.LdapListUsersCmdSpec
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.041 sec - in groovy.org.apache.cloudstack.ldap.LdapListUsersCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapAddConfigurationCmdSpec
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in groovy.org.apache.cloudstack.ldap.LdapAddConfigurationCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapUserSpec
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec - in groovy.org.apache.cloudstack.ldap.LdapUserSpec
Running groovy.org.apache.cloudstack.ldap.LdapAuthenticatorSpec
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec - in groovy.org.apache.cloudstack.ldap.LdapAuthenticatorSpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationVOSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationVOSpec
Running groovy.org.apache.cloudstack.ldap.OpenLdapUserManagerSpec
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.094 sec - in groovy.org.apache.cloudstack.ldap.OpenLdapUserManagerSpec
Running groovy.org.apache.cloudstack.ldap.LdapDeleteConfigurationCmdSpec
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in groovy.org.apache.cloudstack.ldap.LdapDeleteConfigurationCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapUserResponseSpec
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in groovy.org.apache.cloudstack.ldap.LdapUserResponseSpec
Running groovy.org.apache.cloudstack.ldap.LdapUserManagerFactorySpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec - in groovy.org.apache.cloudstack.ldap.LdapUserManagerFactorySpec
Running groovy.org.apache.cloudstack.ldap.ADLdapUserManagerImplSpec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in groovy.org.apache.cloudstack.ldap.ADLdapUserManagerImplSpec
Running groovy.org.apache.cloudstack.ldap.LdapCreateAccountCmdSpec
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.168 sec - in groovy.org.apache.cloudstack.ldap.LdapCreateAccountCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapImportUsersCmdSpec
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 sec - in groovy.org.apache.cloudstack.ldap.LdapImportUsersCmdSpec
Running groovy.org.apache.cloudstack.ldap.LinkDomainToLdapCmdSpec
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in groovy.org.apache.cloudstack.ldap.LinkDomainToLdapCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapSearchUserCmdSpec
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in groovy.org.apache.cloudstack.ldap.LdapSearchUserCmdSpec
Running groovy.org.apache.cloudstack.ldap.LdapListConfigurationCmdSpec
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in groovy.org.apache.cloudstack.ldap.LdapListConfigurationCmdSpec
Running groovy.org.apache.cloudstack.ldap.NoSuchLdapUserExceptionSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in groovy.org.apache.cloudstack.ldap.NoSuchLdapUserExceptionSpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationResponseSpec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationResponseSpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationSpec
asserting for provider configuration: openldap
asserting for provider configuration: microsoftad
asserting for provider configuration:
asserting for provider configuration:
asserting for provider configuration: xyz
asserting for provider configuration: MicrosoftAd
asserting for provider configuration: OpenLdap
asserting for provider configuration: MicrosoftAD
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationSpec
Running groovy.org.apache.cloudstack.ldap.LdapContextFactorySpec
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.099 sec - in groovy.org.apache.cloudstack.ldap.LdapContextFactorySpec
Running groovy.org.apache.cloudstack.ldap.LdapConfigurationDaoImplSpec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec - in groovy.org.apache.cloudstack.ldap.LdapConfigurationDaoImplSpec
Running groovy.org.apache.cloudstack.ldap.LdapUtilsSpec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in groovy.org.apache.cloudstack.ldap.LdapUtilsSpec
Results :
Tests run: 156, Failures: 0, Errors: 0, Skipped: 0
```
* pr/755:
CLOUDSTACK-8647: linkdomaintoldap shouldnt fail when createuseraccount fails
CLOUDSTACK-8647 removed duplicate key in create sql of ldap_trust_map
CLOUDSTACK-8647: string formatting
CLOUDSTACK-8647: updated with review comments
CLOUDSTACK-8647: unittests for LdapAuthenticatorSpec
CLOUDSTACK-8647: formatted LdapAuthenticatorSpec
CLOUDSTACK-8647: UI for trust AD feature
CLOUDSTACK-8647 added unittests for new methods in ldapmanager
CLOUDSTACK-8647 unittests for LinkDomainToLdap api command
CLOUDSTACK-8647: fixed unittests
CLOUDSTACK-8647 support for assigning and admin to linked ldap domain
CLOUDSTACK-8647 added nested group enabled config in ldap
CLOUDSTACK-8647 added account_type to the linkDomainToLdap API
CLOUDSTACK-8647 changed the authentication flow
CLOUDSTACK-8647 added new api linkLdapToDomain
CLOUDSTACK-8647: added cmd and response class for the new api
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
To configure firewall rules, CloudStack modifies `/etc/sysctl.conf` and
execute those modifications. This may be harmful for several reasons:
1. `/etc/sysctl.conf` may be managed by some configuration management
system. Such a system will constantly restore the previous version.
2. `/etc/sysctl.conf` may contain additional properties that have been
changed later by some system administrator (for example, once a
firewall has been configured, forwarding may have been activated
while it is disabled in `/etc/sysctl.conf`). Executing the file
again at a later time may disrupt the system.
3. Entries are added again and again. `/etc/sysctl.conf` will contain
the same directives repeated several times.
Using a configuration file is not needed as `sysctl` is able to directly
modify sysctl values with `-w` flag.
Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
Use host.reserved.ram.mb agent property to modify total system RAM
before reporting to management server.
Remove dom0ram variable and its min/max calculation. Just reserve a
default of 1GB, unless overridden by host.reserved.mem.mb property.
Incase create useraccount fails with any runtime exception,
linkdomaintoldap api shouldnt fail. It just will not return the admin id
as it didnt create the account.
added test cases to verify this as well.
- When stopping/detroying a VPC router and creating a new one via the addition os a VM, the networks are no plugged correctly
- The ETH1 ends up with the IP of the guest nic, which remains down. This causes routes problems and the VMs are not reachable via the pub IPs
- The test will create a VPC, add 2 tier, 2 VMs, ACL, PF and SSH into the VM
- Then it will stop the router, destroy the router, add another VM to 1 tier and check that we can reach all the VMs
CLOUDSTACK-8762: Check to confirm disk activity before starting a VMImplements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* pr/753:
CLOUDSTACK-8762: Check to confirm disk activity before starting a VM
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
L10n update master from Transifex 20150901Update L10N resource files on master branch with 4.6 translation strings from Transifex (20150901)
Fix an issue with the key message.admin.guide.read on pt_BR L10N in web UI at the root cause
* pr/769:
Update L10N resource files on master branch with 4.6 translation strings from Transifex (20150901) Fix an issue with the key message.admin.guide.read on pt_BR L10N in web UI at the root cause
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Guys, can you review it? things need to be discussed:
(1) this supports KVM/QCOW2 only. Anyone want to implement for other Hypervisor/format ?
(2) The original data volume (on primary storage) will be removed.
(3) The script uses the default timeout in libvirtComputingResource. Do we need to add one in global configuration (like copy.volume.wait or backup.snapshot.wait, create.volume.from.snapshot.wait)
(4) In scripts/storage/qcow2/managesnapshot.sh, I use "qemu-img convert -f qcow2 -O qcow2" to copy the snapshot from secondary to primary (hence there is no base image file), instead of "cp -f", this is because convert is faster than cp in my testing.
* pr/732:
CLOUDSTACK-5863: revert volume snapshot for KVM/QCOW2
Signed-off-by: Wei Zhou <w.zhou@tech.leaseweb.com>
wilderrodrigues
Anshul Gangwar
Bharat Kumar
Boris Schrijver
Wei Zhou
Rajani Karuturi
Wido den Hollander
Nitin Kumar Maharana
Jayapal
Jayapal
Kshitij Kansal
Rohit Yadav
Daan Hoogland
Remi Bergsma
Rene Moser
Vincent Bernat
Kevin McCormick
shweta agarwal
Milamber