Commit Graph

657 Commits

Author SHA1 Message Date
Remi Bergsma be89f64731 Merge release branch 4.7 to 4.8
* 4.7:
  Display hostname the VPC router runs on
  CLOUDSTACK-9266: Make deleting static routes in private gw work
  CLOUDSTACK-9264: Make /32 static routes for private gw work
2016-02-04 09:26:53 +01:00
Remi Bergsma a243339faa Merge pull request #1386 from remibergsma/fix-del-private-gw-route
CLOUDSTACK-9266: Make deleting static routes in private gw workThe to-be-deleted static routes were removed from the json file, instead of putting them there with revoke=true. The script that parses the json now doesn't find it and thus does not delete it.

Example after adding/removing some:
```
root@r-3-VM:/var/cache/cloud# cat /etc/cloudstack/staticroutes.json
{
    "1.2.3.0/24": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.2.3.0/24",
        "revoke": true
    },
    "1.2.3.4/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.2.3.4/32",
        "revoke": true
    },
    "1.2.33.3/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.2.33.3/32",
        "revoke": true
    },
    "1.22.2.2/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.22.2.2/32",
        "revoke": true
    },
    "10.1.2.1/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "10.1.2.1/32",
        "revoke": true
    },
    "10.1.200.0/25": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "10.1.200.0/25",
        "revoke": true
    },
    "10.11.12.13/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "10.11.12.13/32",
        "revoke": true
    },
    "172.16.1.3/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "172.16.1.3/32",
        "revoke": true
    },
    "172.16.15.14/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "172.16.15.14/32",
        "revoke": false
    },
    "172.16.17.0/25": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "172.16.17.0/25",
        "revoke": false
    },
    "id": "staticroutes"
}
```

This results in:
```
root@r-3-VM:/var/cache/cloud# ip route show
default via 192.168.23.1 dev eth1
169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.1.67
172.16.0.0/24 dev eth2  proto kernel  scope link  src 172.16.0.2
172.16.15.14 via 172.16.0.1 dev eth2
172.16.17.0/25 via 172.16.0.1 dev eth2
192.168.23.0/24 dev eth1  proto kernel  scope link  src 192.168.23.4
```

Two static routes left, the rest deleted:
```
172.16.15.14 via 172.16.0.1 dev eth2
172.16.17.0/25 via 172.16.0.1 dev eth2
```

That also matches the UI:

<img width="1327" alt="screen shot 2016-01-30 at 06 34 06" src="https://cloud.githubusercontent.com/assets/1630096/12693933/83e67d80-c71b-11e5-9241-9f478522b7a4.png">

* pr/1386:
  CLOUDSTACK-9266: Make deleting static routes in private gw work

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-02-04 09:13:00 +01:00
Remi Bergsma a40415604a CLOUDSTACK-9266: Make deleting static routes in private gw work 2016-01-30 06:25:00 +01:00
Remi Bergsma 78c43669e5 CLOUDSTACK-9264: Make /32 static routes for private gw work 2016-01-29 19:25:29 +01:00
Remi Bergsma e93fd548d6 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9256 add unique key for static routes in json
2016-01-26 17:07:37 +01:00
Remi Bergsma 48672c7e4a Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9256 add unique key for static routes in json
2016-01-26 17:07:04 +01:00
Remi Bergsma 782d5d73f5 Merge pull request #1364 from remibergsma/CLOUDSTACK-9256
CLOUDSTACK-9256 add unique key for static routes in jsonStatic routes that are being set do not show up in the static_routes.json file. The reason for this is that the index that is used, is the gateway address, which is not unique. Hence stuff is overwritten and lost.

Ping @borisroman @wilderrodrigues @DaanHoogland

* pr/1364:
  CLOUDSTACK-9256 add unique key for static routes in json

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 17:06:21 +01:00
Remi Bergsma 43ab98d823 Updating pom.xml version numbers for release 4.9.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 15:12:20 +01:00
Remi Bergsma 32fcc47117 Updating pom.xml version numbers for release 4.8.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 09:39:00 +01:00
Remi Bergsma a4ac9d3fff Updating pom.xml version numbers for release 4.7.2-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 09:33:56 +01:00
Remi Bergsma 8c60ad214f Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9254: Make longer names display pretty
  CLOUDSTACK-9245 - Deletes ACL items when destroying the VPC or deleting the ACL itself
  CLOUDSTACK-9245 - Formatting NetworkACLServiceImpl class
  CLOUDSTACK-9245 - Formatting VpcManagerImpl class
  CLOUDSTACK-9245 - Formatting NetworkACLManagerImpl class
  More VR performance!
2016-01-26 08:39:28 +01:00
Remi Bergsma 62fc6fef97 CLOUDSTACK-9256 add unique key for static routes in json 2016-01-25 17:17:55 +01:00
Remi Bergsma 3d2bb285f0 Merge pull request #1356 from borisroman/4.7-vr-performance-wip
More VR performance!Ping @remibergsma

Integration tests runnning now!

* pr/1356:
  More VR performance!

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-23 19:16:48 +01:00
Boris Schrijver 3eceb60f5c More VR performance! 2016-01-21 11:51:03 +01:00
Remi Bergsma 62f218b7bd Updating pom.xml version numbers for release 4.8.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-20 23:43:35 +01:00
Remi Bergsma 6f9215cf39 Merge release branch 4.7 to master
* 4.7:
  Refactor public ip retrieval into method
  CLOUDSTACK-9244 Fix setting up RFC1918 routes
  CLOUDSTACK-9239 throw exception on deprecated command
  Enhance VR performance by selectively executing tasks instead of brute-forcing
  CLOUDSTACK-9236: Load Balancing Health Check button displayed when non-NetScaler offering is used
2016-01-20 14:18:09 +01:00
Remi Bergsma ed23ac627a Merge pull request #1346 from borisroman/4.7-vr-performance
Enhance VR performanceWill post integration test results soon!

Ping @wilderrodrigues @remibergsma

* pr/1346:
  Enhance VR performance by selectively executing tasks instead of brute-forcing

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-20 14:17:37 +01:00
Daan Hoogland 91f7fc05b6 Refactor public ip retrieval into method 2016-01-19 23:02:26 +01:00
Remi Bergsma d601570053 CLOUDSTACK-9244 Fix setting up RFC1918 routes 2016-01-19 18:29:31 +01:00
Boris Schrijver 73c0242df3 Enhance VR performance by selectively executing tasks instead of brute-forcing 2016-01-18 11:40:59 +01:00
Daan Hoogland addb26455a Merge release branch 4.7 to master
* 4.7:
  CLOUDSTACK-9154 - Sets the pub interface down when all guest nets are gone
  CLOUDSTACK-9187 - Makes code ready for more something like ethXXXX, if we ever get that far
  CLOUDSTACK-9188 -  Reads network GC interval and wait from configDao
  CLOUDSTACK-9187 - Fixes interface allocation to VRRP instances
  CLOUDSTACK-9187 - Adds test to cover multiple nics and nic removal
  CLOUDSTACK-9154 - Adds test to cover nics state after GC
  CLOUDSTACK-9154 - Returns the guest iterface that is marked as added

 Conflicts:
	engine/orchestration/src/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java
2016-01-17 20:04:11 +01:00
Remi Bergsma ff89587fd1 Merge pull request #1277 from ekholabs/fix/4.7-rvpc-net-gc-CLOUDSTACK-9154
[4.7] Critical VPCVR issues fixed: CLOUDSTACK-9154; CLOUDSTACK-9187; and CLOUDSTACK-9188This PR applies the same fixes as in the PR #1259, but against branch 4.7.

Please refer to PR #1259 for the tests results and all the comments already made there.

Issues fixed are:

* CLOUDSTACK-9154: rVPC doesn't recover from cleaning up of network garbage collector
* CLOUDSTACK-9187: rVPC routers in Master/Master due to concurrency problem when writing the keepalivd.conf
* CLOUDSTACK-9188: NetworkGarbageCollector is not using gc.interval and gc.wait from settings

Those changes have been covered by 2 new tests added to ```smoke/test_vpc_redundant.py```:

* test_04_rvpc_network_garbage_collector_nics
* test_05_rvpc_multi_tiers

The test ```test_04_rvpc_network_garbage_collector_nics``` depends on the global settings for the network.gc.interval and gc.wait. If one wants the test to run quicker, please change the settings (default is 600 seconds for each) and restart the Management Server before running the tests. I would suggest to set it to 60 seconds.

In addition, the NetworkGarbageCollector was redefining the settings above mentioned and not reading their values through ConfigDao. Due to that, the settings were not being applied properly and the test was waiting to long to check the VPC routers.

* pr/1277:
  CLOUDSTACK-9154 - Sets the pub interface down when all guest nets are gone
  CLOUDSTACK-9187 - Makes code ready for more something like ethXXXX, if we ever get that far
  CLOUDSTACK-9188 -  Reads network GC interval and wait from configDao
  CLOUDSTACK-9187 - Fixes interface allocation to VRRP instances
  CLOUDSTACK-9187 - Adds test to cover multiple nics and nic removal
  CLOUDSTACK-9154 - Adds test to cover nics state after GC
  CLOUDSTACK-9154 - Returns the guest iterface that is marked as added

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-17 19:12:43 +01:00
Remi Bergsma 24818af23a Merge release branch 4.7 to master
* 4.7:
  CLOUDSTACK-9222 Prevent cloud.log.1 filling up the disk
  Add integration test for restartVPC with cleanup, and Private Gateway enabled.
  Nullpointer Exception in NicProfileHelperImpl
2016-01-16 20:25:29 +01:00
Remi Bergsma c13c5540b0 Merge pull request #1329 from remibergsma/delay_compress_fix
CLOUDSTACK-9222 Prevent cloud.log.1 filling up the diskDelay Compress results in more space usage than needed. Since we have copy truncate we don't need it.

* pr/1329:
  CLOUDSTACK-9222 Prevent cloud.log.1 filling up the disk

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 20:13:49 +01:00
Remi Bergsma 75b68c6829 Merge release branch 4.7 to master
* 4.7:
  Fix unable to setup more than one Site2Site VPN Connection
  FIX S2S VPN rVPC: Check only redundant routers in state MASTER
  PEP8 of integration/smoke/test_vpc_vpn
  Add S2S VPN test for Redundant VPC
  Make integration/smoke/test_vpc_vpn Hypervisor independant
  FIX VPN: non-working ipsec commands
  [UI] MADNESS
  [DB] Add force_encap field to s2s_customer_gateway table
  [ROUTER] Add forceencaps field to python router ipsec config method
  [TEST] unittest needs rework
  [MARVIN] Add forceencap field to VpnCustomerGateway class in marvin base
  [CORE] Add Force UDP Encapsulation option to Site2Site VPN
  CLOUDSTACK-9186: Root admin cannot see VPC created by Domain admin user
  CLOUDSTACK-9192: UpdateVpnCustomerGateway is failing
  CLOUDSTACK-6485 prevent ip asignment of private gw iface
  CLOUDSTACK-9204 Do not error when staticroute is already gone
  make both check lines consistent
  CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh
  CLOUDSTACK-9202 Bump ssh timeout
2016-01-16 19:54:41 +01:00
Remi Bergsma 5fdc77e16c Merge pull request #1276 from michaelandersen/fix/site2sitevpn
[4.7] FIX Site2SiteVPN on redundant VPCThis PR:
- fixes the inability to setup more than one Site2Site VPN connection from a VPC
- fixes starting of Site2Site VPN on redundant VPC
- fixes Site2Site VPN state checking on redundant VPC
- improves the vpc_vpn test to allow multple hypervisors
- adds an integration test for Site2Site VPN on redundant VPC

Tested it on 4.7 single Xen server zone:

command:
```
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-xen1.cfg -a tags=advanced,required_hardware=true /tmp/test_vpc_vpn.py
```

results:
```
Test Site 2 Site VPN Across redundant VPCs ... === TestName: test_01_redundant_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_01_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test Site 2 Site VPN Across VPCs ... === TestName: test_01_vpc_site2site_vpn | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 3 tests in 1490.076s

OK
```

also performed numerous manual inspections of state of VPN connections and connectivity between VPC's

* pr/1276:
  Fix unable to setup more than one Site2Site VPN Connection
  FIX S2S VPN rVPC: Check only redundant routers in state MASTER
  PEP8 of integration/smoke/test_vpc_vpn
  Add S2S VPN test for Redundant VPC
  Make integration/smoke/test_vpc_vpn Hypervisor independant
  FIX VPN: non-working ipsec commands

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 19:51:46 +01:00
Remi Bergsma 66a933afab Merge pull request #1296 from remibergsma/fix-checkrouter-script47
CLOUDSTACK-9181 Prevent syntax error in checkrouter.shAdded quotes to prevent syntax errors in weird situations.

Error seen in mgt server:
```
2015-12-15 14:30:32,371 DEBUG [c.c.a.m.AgentManagerImpl] (RedundantRouterStatusMonitor-7:ctx-0dd8ef3e) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: UNKNOWN
/opt/cloud/bin/checkrouter.sh: line 28: [: =: unary operator expected
/opt/cloud/bin/checkrouter.sh: line 31: [: =: unary operator expected
```
Cause:
```
root@r-1191-VM:/opt/cloud/bin# ./checkrouter.sh
./checkrouter.sh: line 28: [: =: unary operator expected
./checkrouter.sh: line 31: [: =: unary operator expected
Status: UNKNOWN
```

Somehow a nic was missing.

After fix the script can handle this:

```
root@r-1191-VM:/opt/cloud/bin# ./checkrouter.sh
Status: UNKNOWN
```

The other states are also reported fine:
```
root@r-1191-VM:/opt/cloud/bin# ./checkrouter.sh
Status: MASTER
```

```
root@r-1192-VM:/opt/cloud/bin# ./checkrouter.sh
Status: BACKUP
```

While at it, I also removed the INTERFACES variable/constant as it was only used once and hardcoded the second time. Now both are hardcoded and easier to read.

* pr/1296:
  make both check lines consistent
  CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 19:46:06 +01:00
Remi Bergsma b4ebfb640e Merge pull request #1298 from remibergsma/staticroute_delete_47
CLOUDSTACK-9204 Do not error when staticroute is already goneWhen deleting a static route fails because it isn't there any more (KeyError), it should succeed instead.

Error seen:
```
[INFO] Processing JSON file static_routes.json.1451560145
Traceback (most recent call last):
File "/opt/cloud/bin/update_config.py", line 140, in <module>
process_file()
File "/opt/cloud/bin/update_config.py", line 52, in process_file
qf.load(None)
File "/opt/cloud/bin/merge.py", line 258, in load
proc = updateDataBag(self)
File "/opt/cloud/bin/merge.py", line 91, in _init_
self.process()
File "/opt/cloud/bin/merge.py", line 131, in process
dbag = self.process_staticroutes(self.db.getDataBag())
File "/opt/cloud/bin/merge.py", line 179, in process_staticroutes
return cs_staticroutes.merge(dbag, self.qFile.data)
File "/opt/cloud/bin/cs_staticroutes.py", line 26, in merge
del dbag[key]
KeyError: u'192.168.0.3'
```

* pr/1298:
  CLOUDSTACK-9204 Do not error when staticroute is already gone

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 19:43:24 +01:00
Michael Andersen 435a98cd3f Fix unable to setup more than one Site2Site VPN Connection 2016-01-12 11:24:35 +01:00
Michael Andersen dfa924bdee FIX VPN: non-working ipsec commands 2016-01-12 11:24:34 +01:00
Remi Bergsma 867be4145d CLOUDSTACK-9222 Prevent cloud.log.1 filling up the disk
Delay Compress results in more space usage than needed. Since we have copy truncate we don't need it.
2016-01-11 21:09:08 +01:00
Michael Andersen 21acc95d57 [ROUTER] Add forceencaps field to python router ipsec config method 2016-01-07 19:27:42 +01:00
Remi Bergsma 779f4b36e0 CLOUDSTACK-9204 Do not error when staticroute is already gone
When deleting fails because it isn't there any morei (KeyError), it should succeed instead.
2015-12-31 14:11:06 +01:00
Remi Bergsma 8fb677027d make both check lines consistent
No need to make a variable, use it on one place and hardcode it on another.
2015-12-28 17:06:58 +01:00
Remi Bergsma 525949e027 CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh 2015-12-28 17:06:43 +01:00
Wilder Rodrigues 5ef3144fdf CLOUDSTACK-9154 - Sets the pub interface down when all guest nets are gone
- Refactors the set_backup, set_master and set_fault methods to have better names for the variable
   - Increase the sleep on the test in order to wait for the routers to be ready. It's now 3 times the GC settings
2015-12-22 14:52:08 +01:00
Wilder Rodrigues f5a6dee8dd CLOUDSTACK-9187 - Makes code ready for more something like ethXXXX, if we ever get that far
- Adds log info to NetworkOrchestrator in order to make the work of the Net-Scavenger more visible.
2015-12-22 14:52:07 +01:00
Wilder Rodrigues c99d6f18c9 CLOUDSTACK-9187 - Fixes interface allocation to VRRP instances 2015-12-22 14:52:06 +01:00
Wilder Rodrigues 7988f51ac0 CLOUDSTACK-9154 - Returns the guest iterface that is marked as added
- Force a restart of keepalived if conntrackd is not running or configuration has changed
2015-12-22 14:52:05 +01:00
Remi Bergsma 8f5a2920e8 Updating pom.xml version numbers for release 4.8.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-21 22:09:31 +01:00
Remi Bergsma 5b112408c7 Updating pom.xml version numbers for release 4.7.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-21 19:55:04 +01:00
Remi Bergsma 2f26a859a9 Updating pom.xml version numbers for release 4.7.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-13 21:09:53 +01:00
Remi Bergsma 08a1b11c11 Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9155 make sure logrotate is effective
2015-12-13 14:31:24 +01:00
Remi Bergsma d7b7171870 Merge pull request #1235 from remibergsma/fix-passwd-server-logging
CLOUDSTACK-9155 make sure logrotate is effective for cloud.logMany processes on the VRs log to cloud.log. When log rotate kicks in, the file is rotated but the scripts still write to the old inode (cloud.log.1 after rotate). Tis quickly fills up the tiny log partition.

Using 'copytruncate' is a small tradeoff, there is a slight change of missing a log entry, but in the old situation nothing ended up in cloud.log after rotate (except for stuff that was (re)started) so I think this is the best solution until we properly rewrite the script to either use their own script or syslog.

More details: https://issues.apache.org/jira/browse/CLOUDSTACK-9155

* pr/1235:
  CLOUDSTACK-9155 make sure logrotate is effective

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-13 14:30:07 +01:00
Remi Bergsma 636efa2da3 CLOUDSTACK-9155 make sure logrotate is effective
Many processes on the VRs log to cloud.log. When logrotate
kicks in, the file is rotated but the scripts still write
to the old inode (cloud.log.1 after rotate). Tis quickly
fills up the tiny log partition.

Using 'copytruncate' is a tradeoff, there is a slight
change of missing a log entry, but in the old situation
we were missing all of them after logrotate.
2015-12-13 12:07:20 +01:00
Remi Bergsma 90cea824e7 Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9151 - Removes the replacement of the VRID in the CsRedundant file
2015-12-13 00:14:21 +01:00
Remi Bergsma fcccaaad3f Merge pull request #1231 from ekholabs/fix/vrid-rvpc-CLOUDSTACK-9151
CLOUDSTACK-9151 - As a Developer I want the VRID to be set within the limits of KeepaliveDThis PR fixes a blocker issue!

   - Just like with RVRs, use the VRID 51 instead of making it dependent on the VPCID
   - Reason: arbitary unique number 0..255 used to differentiate multiple instances of vrrpd running on the same NIC (and hence same socket). virtual_router_id 51

* pr/1231:
  CLOUDSTACK-9151 - Removes the replacement of the VRID in the CsRedundant file

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-13 00:13:37 +01:00
Daan Hoogland 1063508003 Merge pull request #1186 from remibergsma/462_version
Updating pom.xml version numbers for release 4.6.2-SNAPSHOTSet next version in 4.6 release branch to version 4.6.2-SNAPSHOT.

Using ` ./tools/build/setnextversion.sh`.

Ping @bhaisaab @DaanHoogland before we merge this, how will we be creating the upgrade paths from 4.6.2 to 4.7? After this PR is merged, we need to manually do a fwd-merge and make sure we keep the pom versions in master/4.7. Much like in #1071.

* pr/1186:
  Fixed typo in iam/pom.xml
  Updating pom.xml version numbers for release 4.6.2-SNAPSHOT

Signed-off-by: Daan Hoogland <daan@onecht.net>
2015-12-12 23:17:19 +01:00
Remi Bergsma 5147dec4ff Updating pom.xml version numbers for release 4.6.2-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-12 21:49:37 +01:00
Wilder Rodrigues 2bebb7f8a3 CLOUDSTACK-9151 - Removes the replacement of the VRID in the CsRedundant file
- Just like with RVRs, use the VRID 51 instead of making it dependent on the VPCID
   - Reason: arbitary unique number 0..255 used to differentiate multiple instances of vrrpd running on the same NIC (and hence same socket). virtual_router_id 51
2015-12-12 16:26:00 +01:00
Remi Bergsma 67b4a7bd78 Setup routes for RFC 1918 ip space
Setup general route for RFC 1918 space, as otherwise it will be sent to
the public gateway and not work. More specific routes that may be set
have preference over this generic routes.

When public network is RFC1918, we do not setup the routes to avoid
problems with internal-only deployments.
2015-12-11 11:58:06 +01:00
Remi Bergsma 738b78886e Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9118 - Adds integration test to cover routers state.
  CLOUDSTACK-9118 - Improves the existing checkrouter.sh script
2015-12-08 20:12:09 +01:00
Wilder Rodrigues f81a29351d CLOUDSTACK-9118 - Improves the existing checkrouter.sh script 2015-12-08 11:06:35 +01:00
Remi Bergsma e08294a95f Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9106 - Makes Enum name compliant with Java code conventions.
  CLOUDSTACK-9106 - Adds a test to cover the changes in the applyVpnUsers() method
  CLOUDSTACK-9106 - Makes the router commands call more consistent.
  CLOUDSTACK-9106 - Enables private gateway tests on Redundant VPCs
  CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
  CLOUDSTACK-9106 - Reduces the amount of iterations through the routers of a VPC
  Add support for not (re)starting server after cloud-setup-management.

Closed PRs that will not be considered for merge:
This closes #1158
This closes #1097
2015-12-07 21:36:36 +01:00
Wilder Rodrigues 24dafc2cc2 CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
- Use the router to retrieve the instance ID
    - Check if the VPC is redundant in order to reuse the private gateway address.
    - Brings the private gateways interfaces up.
2015-12-07 10:30:18 +01:00
Daan Hoogland c67d1da5dd Merge pull request #1167 from koushik-das/CLOUDSTACK-9105
CLOUDSTACK-9105: Logging enhancement: Handle/reference to track API calls end to end in the MS logs

Added logid to logging framework, now all API call logs can be tracked with this id end to end

* pr/1167:
  CLOUDSTACK-9105: Logging enhancement: Handle/reference to track API calls end to end in the MS logs Added logid to logging framework, now all API call logs can be tracked with this id end to end

Signed-off-by: Daan Hoogland <daan@onecht.net>
2015-12-06 20:41:28 +01:00
Remi Bergsma 58ba44a1c0 Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9097 Make public ip work immediately
  debian: allow rules to pick ACS_BUILD_OPTS from env
2015-12-04 11:05:23 +01:00
Remi Bergsma 4f6ff6ca08 Merge pull request #1163 from remibergsma/arping-to-gw
Send arping to the gateway instead of our own addressWe need to send an Unsolicited ARP to the gateway, instead of our own address. We now encounter problems when people deploy/destroy/deploy and get the same public ip.

Packets arrive, but with incorrect / cached mac and are ignored by the routervm kernel.
Run arping manually to update the arp-cache on the gateway and things start to work.

Then we discovered the `arping` is actually done, but sent to its own address. Therefore the gateway doesn't pick it up. We only saw this happening when rapid deploy tools are used, like Terraform that do deploy/destroy/deploy and might get the same ip but on a new router having a new mac.

```
2015-12-03 18:07:25,589  CsHelper.py execute:160 Executing: arping -c 1 -I eth1 -A -U -s 192.168.23.8 192.168.23.1
```

The integration tests seem happy, although the full run is still ongoing:

```
=== TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
```

Thanks @sspans for helping trouble shoot this. Ping @wilderrodrigues can you review please?

* pr/1163:
  CLOUDSTACK-9097 Make public ip work immediately

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-04 10:44:48 +01:00
Koushik Das a4dc271253 CLOUDSTACK-9105: Logging enhancement: Handle/reference to track API calls end to end in the MS logs
Added logid to logging framework, now all API call logs can be tracked with this id end to end
2015-12-04 12:45:57 +05:30
Remi Bergsma 7e902cd505 Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9075 - Uses the same vlan since it should have been already released
  CLOUDSTACK-9075 - Adds VPC static routes test
  CLOUDSTACK-9075 - Covers Private GW ACL with Redundant VPCs
  CLOUDSTACK-9075 - Add method to get list of Physical Networks per zone
  CLOUDSTACK-6276 Removing unused parameter in integration test for projects
  CLOUDSTACK-6276 Removing unused parameter in integration test
  CLOUDSTACK-6276 Fixing affinity groups for projects
2015-12-03 20:42:41 +01:00
Remi Bergsma 90e01c95a2 CLOUDSTACK-9097 Make public ip work immediately
We need to send an Unsolicited ARP to the gateway, instead of our own address. We now encounter problems when people deploy/destroy/deploy and get the same public ip.
2015-12-03 19:30:06 +01:00
Wilder Rodrigues 6d9a3d82f9 CLOUDSTACK-9075 - Uses the same vlan since it should have been already released
- After the first test is done, the clean up will delete the whole VPC, also releasing the VLAN that was in use.
2015-12-02 10:36:53 +01:00
Wilder Rodrigues a17fa48de1 CLOUDSTACK-9075 - Adds VPC static routes test
- Adds redundant VPC tests
   - Adds support to Static Routes on VPC private gatways
   - Removes the route configuration in case static route is deleted.
2015-12-02 10:36:52 +01:00
Remi Bergsma 7665bdc815 Merge pull request #1083 from borisroman/CLOUDSTACK-9062
CLOUDSTACK-9062: Improve S3 implementation.The S3 implementation is far from finished, this commit focuses on the bases.

 - Upgrade AWS SDK to latest version.
 - Rewrite S3 Template downloader.
 - Rewrite S3Utils utility class.
 - Improve addImageStoreS3 API command.
 - Split various classes for convenience.
 - Various minor improvements and code optimizations.

A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.

Please review thoroughly, both code inspection and (automated) integration tests. Currently no integration tests are available specifically for S3. Therefore the implementation is needed to be tested manually, for now...

What I tested:
 - Greenfield install -> will download latest systemvm template automatically to S3.
 - Upload a template/iso
 - Download a template/iso
 - Restart of management server -> list available templates -> doesn't download them again if available.

* pr/1083:
  CLOUDSTACK-9062: Improve S3 implementation.

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-20 22:27:17 +01:00
Remi Bergsma 76f430cfc3 Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9015 - Delete public IP in order to get both IP and NAT rule removed.
  CLOUDSTACK-9015 - Add test to cover the rVPC routers stop/start/reboot scenario
  CLOUDSTACK-9015 - Make sure the Backup router can talk to the Master router after a stop/start/reboot
2015-11-19 18:02:13 +01:00
Remi Bergsma 791f9dfe5f Merge pull request #1084 from ekholabs/improvement/remove_scripts-CLOUDSTACK-9067
CLOUDSTACK-9067 - As I developer I want to remove all the unused router-shell scripts from ACSThis PR removes the unused shell scripts that were present in the ACS project. Those script were replaced by the.

Some of the scripts are used by the HyperV Resource, which were hardcoded. I took the opportunity to use the Java constants over there as well, so the next one touching the code will know they exist and won't hardcode anything.

The following task were applied:

* Remove the shell files and the Java constants that were mapping them;
* Apply the use of the Java constants to the HyperV Resource class;
* Wrap the String.format() method in the StringUtils so we can test the changes in the HyperV Resource class.

The last point was added because I do not have a HyperV test environment. Hence, I wanted to make sure the tiny code I changed is covered at least by unit tests.

* pr/1084:
  CLOUDSTACK-9067 - Replaces hardcoded paths with the VRScripts constants.
  CLOUDSTACK-9067 - Fomatting the code of HypervDirectConnectResource class
  CLOUDSTACK-9067 - Remove old script file from the project

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-19 17:58:42 +01:00
Remi Bergsma 78e5518741 Merge pull request #1070 from ekholabs/fix/rVPC-CLOUDSTACK-9015
[4.6.1] CLOUDSTACK-9015 - Redundant VPC Virtual Router's state is BACKUP & BACKUP or MASTER & MASTERThis PR closes #1064

All the details can be found in the original PR, which won't be merged because it was created agains master. Once this PR is closed, the original one will be also closed.

* pr/1070:
  CLOUDSTACK-9015 - Delete public IP in order to get both IP and NAT rule removed.
  CLOUDSTACK-9015 - Add test to cover the rVPC routers stop/start/reboot scenario
  CLOUDSTACK-9015 - Make sure the Backup router can talk to the Master router after a stop/start/reboot

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-19 17:52:30 +01:00
Boris Schrijver 5c0366c99e CLOUDSTACK-9062: Improve S3 implementation.
The S3 implementation is far from finished, this commit focusses on the bases.

 - Upgrade AWS SDK to latest version.
 - Rewrite S3 Template downloader.
 - Rewrite S3Utils utility class.
 - Improve addImageStoreS3 API command.
 - Split various classes for convenience.
 - Various minor improvements and code optimalisations.

A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
2015-11-19 15:29:10 +01:00
Remi Bergsma 228935df4e Merge release branch 4.6 to master
* 4.6:
  CLOUDSTACK-9058
2015-11-18 11:22:16 +01:00
Remi Bergsma 37cee3309c Merge pull request #1079 from dsclose/CLOUDSTACK-9058
CLOUDSTACK-9058 - Respond with "saved_password" if no password is to be issued.The password server on the virtual router should respond with "saved_password" if no password is to be issued. This allows for backwards compatibility with Windows Guest VMs which require the "saved_password" response.

* pr/1079:
  CLOUDSTACK-9058

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-18 11:20:56 +01:00
Wilder Rodrigues 6477bd8ff7 CLOUDSTACK-9067 - Remove old script file from the project
- Java constants also removed
   - Project still compiling and all unit tests passing.
2015-11-17 15:58:22 +01:00
Remi Bergsma 1f53f2a93e Updating pom.xml version numbers for release 4.7.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-15 18:54:13 +01:00
Remi Bergsma b38c3bed0c Updating pom.xml version numbers for release 4.6.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-13 21:27:57 +01:00
Wilder Rodrigues cd05a252fb CLOUDSTACK-9015 - Make sure the Backup router can talk to the Master router after a stop/start/reboot
- Stop KeepaliveD/ConntrackD if the eth2 (guest) interface is not configured and UP
   - Only setup the redundancy after all the router configuration is done
   - Open the FW for the VRRP communitation
     - 224.0.0.18 and 225.0.0.50
   - Set keepalived.conf.templ by default to use interface eth2 (guest)
     - It will be reconfigured anyway, but having eth2 there is more clear
2015-11-12 18:13:51 +01:00
dsclose 8a7deefe64 CLOUDSTACK-9058
Respond with "saved_password" if no password is to be issued.
2015-11-12 08:05:57 +00:00
Remi Bergsma e31ade03c6 Updating pom.xml version numbers for release 4.6.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-11-10 15:45:34 +01:00
Wilder Rodrigues 79dabfdae4 CLOUDSTACK-8925 - Implement the default egress DENY/ALLOW properly
- Make tests work with right IP and rules
   - Add egress rule for port 53 protocol UDP when testing default egress DENY on RVR
2015-11-04 13:48:04 +01:00
Wilder Rodrigues caa0b4071c CLOUDSTACK-8925 - Drop the traffic when default egress is set to false
- The DROP rule should be appended and the other rules inserted.
2015-11-02 21:45:03 +01:00
Wei Zhou 9f7f42330a CLOUDSTACK-9013: Virtual router failed to start on KVM
This fix a typo of commit 4a177031b0
2015-10-30 20:04:56 +01:00
Remi Bergsma 323a2aaa2a Merge pull request #981 from serbaut/repopulate-dhcphost
CLOUDSTACK-8993: DHCP fails with "no address available" when an IP is reused

Repopulate /etc/dhcphosts.txt to remove old entries with the same IP address.

* pr/981:
  CLOUDSTACK-8993: DHCP fails with "no address available" when an IP is reused

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-10-29 15:56:54 +01:00
Wilder Rodrigues b2aa1f8417 CLOUDSTACK-8957 - Implement password server in configure.py 2015-10-29 07:14:51 +01:00
Wilder Rodrigues 7d40814ba1 CLOUDSTACK-8991 - Process the IPs that have been removed 2015-10-28 19:51:57 +01:00
Joakim Sernbrant f413f69985 CLOUDSTACK-8993: DHCP fails with "no address available" when an IP is reused
Repopulate /etc/dhcphosts.txt to remove old entries with the same IP address.
2015-10-28 15:05:59 +01:00
Wilder Rodrigues 7197cf2e24 CLOUDSTACK-8991 - Remove public IP form interface in case add = false 2015-10-27 19:46:13 +01:00
Wilder Rodrigues a01720b25b CLOUDSTACK-8935 - Filter the DNS list because it might contain 1 None entry which breaks the code. 2015-10-23 16:45:36 +02:00
Wilder Rodrigues 5ca32dfc82 CLOUDSTACK-8935 - Check if the key is available in the dictionary
- This will fix the cleanup problems.
2015-10-22 14:52:07 +02:00
Wilder Rodrigues 3725b3ee30 CLOUDSTACK-8935 - Add a check to avoid exception related to None value 2015-10-22 13:03:06 +02:00
Wilder Rodrigues 698171d210 CLOUDSTACK-8933 - Replace infinite loop by a for loop
- The loop will also sleep in between iterations, waiting for a couple of seconds and being multiplied by factor 2 per iteration
2015-10-21 11:10:37 +02:00
Bharat Kumar ac68f2a116 CLOUDSTACK-8933 SSVm and CPVM do not survive a reboot from API 2015-10-20 10:50:14 +02:00
Wilder Rodrigues fb33cb28ab CLOUDSTACK-8952 - Make the checkrouter.sh compatible with RVR as well 2015-10-17 15:52:28 +02:00
Wilder Rodrigues 2a747ca735 CLOUDSTACK-8952 - Reduce retried from 20 to 5
- We do not need to retry that much
2015-10-17 12:20:42 +02:00
Wilder Rodrigues 5b3c99031f CLOUDSTACK-8952 - Do not rely in the router state on the json file to report back to ACS
- If we stop/start a router, the state in the file will still say MASTER, when it is actually not
   - Checking the state based on the interface (eth1) state
   - Once master.py is called by keepalived, save the state in the json file to BACKUP just to make sure it's also written there
2015-10-17 12:20:41 +02:00
Wilder Rodrigues 41f4d8b58a CLOUDSTACK-8952 - Make the check for master more reliable
- Do not use the API call because it will read what is in the database, that might not have been updated yet
     * Check the status in the router directly instead
   - Remove all the sleeps
2015-10-17 12:20:41 +02:00
Wilder Rodrigues c7671f3cdd CLOUDSTACK-8952 - Restart dnsmasq everytime the configure.py runs 2015-10-17 12:20:41 +02:00
Wilder Rodrigues 2b286ecd73 CLOUDSTACK-8952 - Make sure the calls to CsFile use the new logic of commit/is_changed methods
- We now have to check if the file changed before commiting. Doesn't make sense to write on disk if there was nono change.
2015-10-17 12:20:40 +02:00
Wilder Rodrigues 1886c4a1b3 CLOUDSTACK-8952 - Make sure we restart dnsmasq if the configuration file changes
- It was working before because the Routers were restarting about 10 times for each operation
     e.g. adding a VM to a network ot acquiring a new IP.
   - Adding stat_rules of internal LB to iptables
     We needed one extra rule in the INPUT chain
2015-10-17 12:20:40 +02:00
Wilder Rodrigues d762dc8579 CLOUDSTACK-8952 - The public interface was comming UP in the Backup router
- There were too many places trying to put the pub interface UP. I centralised it now.
2015-10-17 12:20:40 +02:00
Wilder Rodrigues b4920aa028 CLOUDSTACK-8952 - Do not restart conntrackd unless it's needed
- With the keepalived fixed they should not be needed anymore. So first reducing them drasticaly
   - I am now making a backup of the template file, write to the template file and compare it with the existing configuration
   - The template file is recovered afer the process
   - I also check if the process is running
   - I fixed a bug in the compare method
   - I am now updating the configuration variable once the file content is flushed to disk
2015-10-17 12:20:40 +02:00
Wilder Rodrigues 5a216056b5 CLOUDSTACK-8952 - Do not replace the conntrackd config file unless it's needed
- With the new logic, the file will be replaced when the router starts, becasue the default
     conntrackd config file will be different.
2015-10-17 12:20:39 +02:00
Wilder Rodrigues 08b983fe02 CLOUDSTACK-8952 - Remove the '--vrrp' search criteria form the CsProcess constructor call
- There is no such process, which makes the CsProcess.find return false and restart keepalived all the time.
2015-10-17 12:20:39 +02:00
Wilder Rodrigues f35a16c19e CLOUDSTACK-8947 - Adding some logging to better understand whay is happening with the Processes 2015-10-14 07:44:39 +02:00
Wilder Rodrigues 80b51a7972 CLOUDSTACK-8947 - Adding some logging to better understand what's happening with the rules 2015-10-14 07:44:39 +02:00
Wilder Rodrigues 59bd935f3e CLOUDSTACK-8947 - Configure the firewall when the load balancer is setup
- Only restart HAproxy if it's not running yet
2015-10-14 07:44:38 +02:00
Wilder Rodrigues 4a177031b0 CLOUDSTACK-8947 - Avoid multiple entries in the FW_EGRESS_RULES table 2015-10-14 07:44:38 +02:00
Wilder Rodrigues 052c0dc4c9 CLOUDSTACK-8947 - Open the input chain to IP when loadbalancer is configured
- Also remove the chain rule when it is removed.
2015-10-14 07:44:38 +02:00
Wilder Rodrigues cbd6464b88 CLOUDSTACK-8947 - FW_EGRESS should be added only to filter table 2015-10-14 07:44:37 +02:00
Wilder Rodrigues cd8c3e97f5 CLOUDSTACK-8934 - Fix the AclIP class to make it configure the default FW policy 2015-10-11 14:57:33 +02:00
Wilder Rodrigues b4dc392bfd CLOUDSTACK-8934 - Fix default EGRESS rules for isolated networks
- The default is Accept and will be changed based on the configuration of the offering.

CLOUDSTACK-8934 - The default egress is set as Deny in the router.

   - We had to change it on the Java side in order to make the apply it once the default is defined as allowed on the net offering
2015-10-11 14:57:32 +02:00
Wilder Rodrigues 45642b8382 CLOUDSTACK-8934 - Add default gateway when the public interface is up again 2015-10-11 14:57:12 +02:00
Wilder Rodrigues 9fe040e96d CLOUDSTACK-8915 - Copy the conntrackd configuration every time _redundant_on() function is called
- Also refactored the copy() function under CsHelper.py
2015-10-06 16:41:15 +02:00
Wilder Rodrigues 260ff836b6 CLOUDSTACK-8915 - Reverting changes from commit id 1a02773b55
- That's not the place to fix the default routes for redundant VPC,
   - Adding tests to cover PF and FW in isolated networks
     * Will still add some tests for egress as well
2015-10-06 16:41:14 +02:00
Wilder Rodrigues 3d34e4a52c CLOUDSTACK-8915 - Reverting changes from commit id 18dbc0c4cb
- If the file is always copied, it will result in restarting keepalived everytime which makes the routers transit between master/backup
2015-10-06 16:41:13 +02:00
Wilder Rodrigues 094d29d7cd CLOUDSTACK-8915 - VRRP needs a cidr in order to work properly
- The cidr was replaced by the single IP, which broke the feature.
   - Wait during transition from master to backup otherwise the test fails due to wronge state
2015-10-06 16:41:13 +02:00
Wilder Rodrigues 0c752eab60 CLOUDSTACK-8915 - Rearrenging a bit the default route code in order to make it more clear 2015-10-06 16:41:13 +02:00
Wilder Rodrigues c17fb0ff28 CLOUDSTACK-8915 - Add the default route only on address that have not been configured yet.
- In case of rVPC we experienced the wrong route being added to the VPC tiers
2015-10-06 16:41:12 +02:00
Wilder Rodrigues 09e05f2a06 Fixing the index out of bounds error in the check_if_link_up() function 2015-09-26 20:43:15 +02:00
Remi Bergsma d83995e23c small cleanups 2015-09-26 20:33:21 +02:00
Wilder Rodrigues a8fa3374da Fixing the defaut route for VPC routers 2015-09-26 20:30:49 +02:00
Wilder Rodrigues 595fa50b3b Formatting the get_gateway() method in the CsDatabag.py file 2015-09-25 16:11:32 +02:00
Wilder Rodrigues 3cfc4cff80 Fixing the dhcpsrvr iptables file
- Instead of changing the router type in a local variable, lets have a dedicated file for the dhcpsrvr routers
   - The file is called iptables-dhcpsrvr, just like we have iptables-vpcrouter and iptables-router
2015-09-25 16:10:43 +02:00
Remi Bergsma 4c8f4ac341 Merge pull request #842 from jayapalu/shareNwVR
CLOUDSTACK-8843: Fixed issue in default iptables rules on shared network VROn basic zone share network VR default iptables rules are not applied correctly. Due to this ssh to VR got failed.
In shared network the VR type is 'dhcpsrvr' not router. So corrected it in the ''del_standard' method to select the correct type.

Testing:
1. VR is deployed correctly.
2. Tested restart, stop, start VR.
3. New VM deployment is success.
4. ssh to VR from the host is successful.
5. iptables rules on the VR came up correctly.
below is the output from the VR:
iptables -L INPUT -nv
Chain INPUT (policy DROP 16 packets, 1056 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            224.0.0.18
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            225.0.0.50
  104  9800 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
  281 36500 ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  eth2   *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    6   504 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    2   656 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
   13   780 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:3922 state NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 state NEW
    0     0 ACCEPT     tcp  --  eth0   *       10.147.40.0/23       0.0.0.0/0            state NEW tcp dpt:8080

* pr/842:
  CLOUDSTACK-8843: Fixed issue in default iptables rules on shared network VR

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-09-24 16:42:42 +02:00
Remi Bergsma 4420f48e3e Merge pull request #882 from jayapalu/CLOUDSTACK-8881
CLOUDSTACK-8881: Fixed Static and PF configuration issue1. For static nat filter rules are not configured in VR.
2. Corrected vm ip in PF rule.

* pr/882:
  CLOUDSTACK-8881: Fixed Static and PF configuration issue

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-09-24 15:36:18 +02:00
Remi Bergsma 649a4bdc76 Merge pull request #881 from jayapalu/CLOUDSTACK-8905
CLOUDSTACK-8905: Fixed hooking egress rulesAdded hooking the FIREWALL_EGRESS_RULES chain into FW_OUTBOUND chain.
With this egress rules will effective.

* pr/881:
  CLOUDSTACK-8905: Fixed hooking egress rules

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-09-24 15:36:04 +02:00
Remi Bergsma 4018d47ef8 Merge pull request #870 from jayapalu/CLOUDSTACK-8874
Configured dnsmasq to listen on all interfaces so that vpn  client gets dns1. Dnsmasq is not listening on the ppp+ interfaces due to this remote access vpn clients dns requests are  dropped.

2. Configured the dnsmasq to listen on all the interfaces except public. There is firewall to allow only specific cidr to allow the dns requests.

Tested from windows client nslookup.

* pr/870:
  Configured dnsmasq to listen on all interfaces so that vpn  client gets dns

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-09-24 15:35:48 +02:00
Remi Bergsma a5a5f612ea Merge pull request #867 from jayapalu/CLOUDSTACK-8891
CLOUDSTACK-8891: Fixed default iptables rules on VR  for guest trafficVR default iptables rules in INPUT chain are configured partially.
In CsAddress.py rules are configured while configuring public interface, guest interface post configuration is missed. Fixed to configure guest post configuration so that iptables rules are configured.

Testing:
1. Deployed vm in the network.
2.iptables rules on the VR configured correctly.
3.VM got the dhcp ip address from the VR.

* pr/867:
  CLOUDSTACK-8891: Fixed default iptables rules on VR  for guest traffic

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-09-24 15:35:33 +02:00
Jayapal 40138d2e99 CLOUDSTACK-8881: Fixed Static and PF configuration issue 2015-09-24 17:22:29 +05:30
Jayapal 2bf7fb4b63 CLOUDSTACK-8905: Fixed hooking egress rules 2015-09-24 17:06:11 +05:30
Jayapal 746a5dc48e CLOUDSTACK-8891: Fixed default iptables rules on VR for guest traffic 2015-09-24 12:49:43 +05:30
Jayapal dbedfe2557 Configured dnsmasq to listen on all interfaces so that vpn client gets dns 2015-09-22 14:10:48 +05:30
SudharmaJain 56d4429500 CLOUDSTACK-8863: VM doesn't reconnect to internet post VR RESTART/STOP-START/RECREATE 2015-09-17 11:50:21 +05:30
Jayapal a15df0569f CLOUDSTACK-8843: Fixed issue in default iptables rules on shared network VR 2015-09-16 15:24:34 +05:30
wilderrodrigues 6b9e2ac16a Merge pull request #800 from bvbharatk/CLOUDSTACK-8798
CLOUDSTACK-8798 Fixed the vrrp virtual ip config in case of rvr enablFixed the vrrp virtual ip config in case of rvr enabled isolated networks.
changed the CsRedundant.py to bring down the public interface when rvr changes state to
backup. Also fixed vrrp authentication for isolated networks.

This fix dose not effect the vpc networks. it is only meant for rvr isolated networks.

manullay deployed a vm in rvr enabled isolated network and ran the tests below.

nosetests --with-marvin --marvin-config=/marvin-config test/integration/component/test_redundant_router_services.py

Test redundant router internals ... === TestName: test_enableVPNOverRvR | Status : SUCCESS ===

----------------------------------------------------------------------
Ran 1 test in 633.336s

nosetests --with-marvin --marvin-config=/marvin-config test/integration/component/test_redundant_router_cleanups.py

Test network garbage collection with RVR ... === TestName: test_network_gc | Status : SUCCESS ===
ok
Test restarting RvR network without cleanup ... === TestName: test_restart_ntwk_no_cleanup | Status : SUCCESS ===
ok
Test restart RvR network with cleanup ... === TestName: test_restart_ntwk_with_cleanup | Status : SUCCESS ===

----------------------------------------------------------------------
Ran 3 tests in 2120.263s

* pr/800:
  CLOUDSTACK-8798 Fixed the vrrp virtual ip config in case of rvr enabled isolated networks. changed the CsRedundant.py to bring down the public interface when rvr changes state to backup. Also fixed vrrp authentication for isolated networks.

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-09-11 15:27:41 +02:00
Bharat Kumar b66dcda49f CLOUDSTACK-8799 fixed for vpc networks. 2015-09-11 14:57:32 +02:00
Bharat Kumar 1a02773b55 CLOUDSTACK-8799 fixed the defalut routes
CLOUDSTACK-8799 made changes to fix CsRedundant.py
2015-09-11 14:57:32 +02:00
Bharat Kumar 6f68ecc3bf CLOUDSTACK-8798 Fixed the vrrp virtual ip config in case of rvr enabled isolated networks.
changed the CsRedundant.py to bring down the public interface when rvr changes state to
backup. Also fixed vrrp authentication for isolated networks.
2015-09-10 22:42:12 -07:00
Jayapal 33f4f952cf CLOUDSTACK-8690: Updated the iptables order 2015-09-09 10:50:50 +05:30
Jayapal f661ac0a2a CLOUDSTACK-8690:Added remote access vpn and vpn users configuration 2015-09-09 10:50:50 +05:30
wilderrodrigues 237e0e4611 Merge pull request #765 from schubergphilis/fix/default_policies
CLOUDSTACK-8688 - default policies for INPUT and FORWARD should be set to DROP instead of ACCEPT

  - In order to be able to access the routers via the link local interface, we have to add a rules with NEW and ESTABLISHED state

* pr/765:
  CLOUDSTACK-8688 - Adding Marvin tests in order to cover the fixes applied
  CLOUDSTACK-8688 - default policies for INPUT and FORWARD should be set to DROP instead of ACCEPT

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-09-08 13:45:19 +02:00
wilderrodrigues f5e5f4d002 CLOUDSTACK-8688 - default policies for INPUT and FORWARD should be set to DROP instead of ACCEPT
- In order to be able to access the routers via the link local interface, we have to add a rules with NEW and ESTABLISHED state
2015-08-31 12:43:04 +02:00
Bharat Kumar 18dbc0c4cb CLOUDSTACK-8725 RVR functionality is broken in case of isolated networks, conntrackd fails to start.
Rebased with master. This closes #692
2015-08-31 11:54:11 +05:30
Michael Andersen a9eb4ba5c9 Fixed defaultroute on VPC routers in CsRoute and CsAddress
Refactored CsRoute
2015-08-24 13:35:29 +02:00
Michael Andersen bb93a61ee7 pep8 cleanup 2015-08-24 13:35:01 +02:00
Remi Bergsma 7ddec661ca Merge pull request #690 from jayapalu/vpn
CLOUDSTACK-8710: Fixed applying iptables rules for s2s vpn
@remibergsma @wilderrodrigues
Moved applying iptables rules apply after vpn configuration so that vpn specific rules also get applied

* pr/690:
  CLOUDSTACK-8710: Fixed applying iptables rules for s2s vpn

This closes #690

Signed-off-by: Remi Bergsma <github@remi.nl>
2015-08-14 09:11:21 +02:00
Remi Bergsma 4f8ab51f7f do not log sensitive site-to-site VPN PSK
Logging before:
2015-08-12 16:30:07,126 Searching for 192.168.23.6  and replacing with 192.168.23.6 192.168.23.5: PSK "preSharedKey"

Logging after:
2015-08-12 16:30:07,126 Searching for 192.168.23.6  and replacing with 192.168.23.6 192.168.23.5: PSK "****"
2015-08-14 09:07:25 +02:00
Remi Bergsma 9b97719c5c tighten security of site-to-site VPN
It was like this in 4.4 and 4.5
2015-08-14 09:05:59 +02:00
Remi Bergsma 382458317e CLOUDSTACK-8730: fix s2s iptables rules and ipsec config
For site2site VPN to work, we need a default gateway to be set.
See CLOUDSTACK-8685
2015-08-14 09:02:21 +02:00
Jayapal da9e757b8e CLOUDSTACK-8710: Fixed applying iptables rules for s2s vpn 2015-08-13 14:07:12 +05:30
Jayapal 97eb0f7528 CLOUDSTACK-8707: Removed convert sec to h on esp_lifetime
This closes #655
2015-08-05 14:14:55 +05:30
Jayapal c13201353c CLOUDSTACK-8707: Added missed esp policy esp is added with esp_lifetime instead of esp_policy, corrected it. 2015-08-05 14:14:46 +05:30
Kishan Kavala bccd344796 CLOUDSTACK-8683: process cmd_line.json for shared network VR in cloud-early-config
This closes #634
2015-07-29 15:57:19 +05:30
Kishan Kavala 90137acda2 VR type in shared network is dhcpsrvr. Ips are being removed due to this issue
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>

This closes #627
2015-07-28 13:43:09 +02:00
wilderrodrigues b29d8e7280 CLOUDSTACK-8616 - Adding important remark about the use of buildsystemvm.sh
- For package installation/update please refer to: cloud-tools/appliance/definitions/systemvmtemplate/install_systemvm_packages.sh

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>

This closes #587
2015-07-16 13:14:49 +02:00
wilderrodrigues 4b3db3d117 CLOUDSTACK-8616 - Add a cron job to restart ConntrackD on reboot
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-07-16 13:14:48 +02:00
wilderrodrigues f78a65c05a CLOUDSTACK-8616 - Installing KeepaliveD from Debian Wheezy backports
- preempt delay reverted on version 1.2.13 - from the backports
     - vrrp : Revert "Honor preempt_delay setting on startup.".
     - See changelog: http://www.keepalived.org/changelog.html
   - Refactoring some variable names to avoid misunderstanding

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-07-16 13:14:48 +02:00
wilderrodrigues 12ee36a55f CLOUDSTACK-8616 - Add keepalived start on reboot
- Runs check_heartbeat.sh every 30 seconds

CLOUDSTACK-861 - Copy/Paste error

   - Paste the wrong command in the crontab line.

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-07-16 13:14:48 +02:00
wilderrodrigues 20be22c698 CLOUDSTACK-8616 - Removing the Priority form KeepaliveD configuration
- We use no preempt mode with state set as EQUAL to both nodes, no need to have Priotities setup
   - Do not add IPs as comments to the configuration. If a new guest interface is added, the file will change anyway.
     - This was used in the past when keepalived would restart for each new interface added
   - Removed the long sleep form the tests: we now sleep 5 seconds per PF rule added

CLOUDSTACK-8616 - Fix keepalived.ts/2 files comparison

   - Add call to set_fault() in case of router transits to that state
   - Removing commented out code

CLOUDSTACK-8616 - Fixing check_heartbeat.sh.templ

CLOUDSTACK-8616 - Call set_fault from the check_heartbeat.sh script

Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-07-16 13:14:48 +02:00
Ian Southam b7b54f1680 Fixes to enable test to succeed. Small change to Marvin to be able to override retries
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
2015-07-08 11:19:12 +02:00
David Bierce 646e0d99dd Always enter chap-secrets as a quoted field. In the event of special characters it creates and deletes the entry properly, in the event there are not special characters there is no change behavior.
Signed-off-by: Remi Bergsma <apache@remi.nl>

This closes #307
2015-05-27 13:42:48 +02:00
Ian Southam 56483436f0 Pep8 changes that got lost with the strange double push request problem last time 2015-05-26 07:07:43 -04:00
Abhinandan Prateek ad6ac9bb10 CLOUDSTACK-8488: network with LB fails to restart as the bash script mangles the escape characters, fixed 2015-05-22 16:24:20 +05:30
Ian Southam ab915b6c92 Allow forward to fix port forwarding rules 2015-05-19 12:54:38 +00:00
Rajani Karuturi 0b8355920e Merge branch 'volume-upload' into master
This closes #206
2015-04-29 11:12:53 +05:30
Rohit Yadav 6829399647 CLOUDSTACK-7493: ignore errors on removal of iptables chains that don't exist
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit f26ae6eb39)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-23 14:19:46 +02:00
Rohit Yadav f55658dab2 vpn_l2tp: fix variable assignment, fails L2TP VPN
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 5cba410937)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-23 14:19:34 +02:00
wilderrodrigues 2fad87d3f3 Make the routers persistent
- After configuration save the ipdated in files
    * /etc/iptables/router_rules.v4 and /etc/iptables/router_rules.v6
    * Reload the configuration on reboot via the /etc/rc.local using iptables-restore
2015-04-14 15:09:47 +02:00
Rene Moser 0ada08aa85 CLOUDSTACK-6885: fix logrotate on VR to depend on size
In 6ac06e5e5e logrotate was changed to run hourly.
Some logrotate configs still have set `daily` only which results in logs not
rotated hourly. The only way to ensure the log is rotated is to use size.

This closes #162

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-04-14 00:49:28 +05:30
wilderrodrigues cb2b9e870b Fixing the communication with VM via Public IP
- Pub IP port forwarding and static NAT fixed for single VPCs
   - Pub IP port forwarding fixed for redundant VPCs

[wip] fix static NAT for redundant VPCs

This closes #150
2015-04-08 16:04:55 +02:00
wilderrodrigues 3d22a16c4f Bump priority stragety is no longer used for redundant virtual routers
- With the changes added by the rVPC work, the bump priority became deprecated.
     This commit includes a refactor to get it removed from the following resources:
     * Java classes
     * domain_router table - removing the is_priority_bumpup column
     * Fixing unit tests

All changes were tested with:

XenServer 6.2 running under our VMWare zone
CloudStack Management Server running on MacBook Pro
MySql running on MackBook Pro
Storage Type: Local
2015-04-02 21:50:49 +02:00
Rohit Yadav 0540ba1b30 systemvm: fix socat usage
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 888f67f0bd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-30 13:50:39 +05:30
Abhinandan Prateek 6ac06e5e5e CLOUDSTACK-6885: rotate logs hourly on VR 2015-03-30 13:49:23 +05:30
Rene Moser 33280ade2e systemvm: fix/improve irqbalance on multicore VR.
This is a known issue in irqbalance 1.0.3 and was partially fixed in 1.0.4.
Using the package 1.0.6 from wheezy backports helped on many VRs balancing the
interrupts, but not on all.

It seems only 1.0.7 fixes all issues regarding this, also see thread in user ML
http://mail-archives.apache.org/mod_mbox/cloudstack-users/201503.mbox/%3C5508540E.4090302%40renemoser.net%3E

This closes #119

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit a45f0e29e2)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-23 14:21:13 +05:30
Rohit Yadav eddcb101dc CLOUDSTACK-8331: have savepassword try all interfaces
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit b819211476)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-18 14:40:23 +05:30
Rohit Yadav b011c9bad4 Add missing license headers from recent branch merge 3c429ee
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-18 11:03:54 +05:30
Rohit Yadav f0da19a060 vpc_passwd_server: use the improved Python based password server
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 3262b0bfd9)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-18 10:43:20 +05:30
Rohit Yadav 3c429ee6b5 Merge remote-tracking branch 'sbp/feature/persisten-systemvm-redundant-vpc-REBASE'
This closes #118

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-17 15:44:57 +05:30
Rohit Yadav 83736ab534 CLOUDSTACK-8331: savepassword should try all IPs on eth0
The logic is same as passwd_server_ip script which runs password server on all
IPs on eth0 interface.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 294503a025)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-17 14:23:07 +05:30
Rohit Yadav e3c04c7147 savepassword: use eth0's IP address, as password server listens on that IP
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 488625b193)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2015-03-17 13:24:06 +05:30
Ian Southam 23c100d949 If master gets called and then within a few seconds backup (this can happen during provisioning), the master process will complete after the backup one. Not what we want. These commands must be serial
Used a wee Unix socket to create a lock

We still need to find out why the flip flop occasionally happens. nopreempt should stop this ...
2015-03-16 11:40:12 +01:00
Ian Southam 7bfccd699b Fixed Port forwarding (DNAT/SNAT) for isolated networks
Fixed failure on using eth10 (which CS now supports)
Refactored CSForward to us CsInterface object instead of teh way it was kludged together
Fixed hex conversion for device numbers and iptables marks
2015-03-16 11:40:12 +01:00
Ian Southam 6fc680be71 Fixed password server respawning issue
Some changes to isolated network acl generation
2015-03-16 11:40:11 +01:00
Ian Southam 5ddf87b2fd Use reload on keepalived
restart causes the master status to flip unecessarily during provisioning tasks
2015-03-16 11:40:11 +01:00
Ian Southam 585f5f7000 ACL issues
ACL order issues
Do not block multicast traffic for vrrp
Many smaller bug fixes
checkrouter provided in /opt/cloud/bin
2015-03-16 11:40:10 +01:00
Ian Southam 3cc0915a01 Work arounds for how CS adds and removes hosts 2015-03-16 11:40:08 +01:00
Ian Southam 748e2e51a9 Check both ip and hosts when building dhcp bag 2015-03-16 11:40:08 +01:00
Ian Southam c1ec5b0b16 nat rules added to incorrect interface 2015-03-16 11:40:07 +01:00
Ian Southam 7700e940ff dnsmasq should restart if config changes and hup if hosts change 2015-03-16 11:40:07 +01:00
wilderrodrigues e7969b640b No hardcoded passwords.
- If for some reason the cmdLine json doesn't contain the password key, which is almost impossible to happen,
    we generate a password based on other unique data per VPC
2015-03-16 11:40:06 +01:00
wilderrodrigues 4b6604318d generate random password for redundant VPC 2015-03-16 11:40:06 +01:00
Ian Southam baa758accc Utterley stupid 2015-03-16 11:40:06 +01:00
Ian Southam 639126da8b {ip: hosts} syntax goes wrong if ip contains 1::
Needs to anchore regexp
2015-03-16 11:40:05 +01:00
Ian Southam aa6fdf853b A tab got in there somehow! 2015-03-16 11:40:05 +01:00
Ian Southam b3da2c54c2 Add network stats for isalated networks 2015-03-16 11:40:04 +01:00
Ian Southam 57d3ffaef8 Fixed password server, fixed more firewall issues
Fixed issues with real IP and not virtual (gateway) IP being opened on the firewall
DNS now works on the vms
2015-03-16 11:40:03 +01:00
wilderrodrigues e6b3ee318c Fix the checkrouter.sh script in order to report the routers state 2015-03-16 11:40:02 +01:00
Ian Southam 9e82281cc1 Fixed rules for dhcp towards v,
Fixed fw rules for a bunch of things
Bring public interface up if no tier has been defined
2015-03-16 11:39:58 +01:00
wilderrodrigues 4c9f4fcdd3 Make virtual router id unique per redundant group
Customise the messages showed during VPC restart based on redundancy
Do not show "Make Redundant" option if VPC is already redundant
Show on VPC details if a VPC is redundant
2015-03-16 11:39:58 +01:00
Ian Southam 7a9a356094 Switch control interface for vrrp to guest network
Fix rRouter that was deleting the gateway interface
2015-03-16 11:39:57 +01:00
Ian Southam 87d4171c53 Who changed my condition!
De Morgans laws chaps
The negation of a conjunction is the disjunction of the negations.
The negation of a disjunction is the conjunction of the negations.
2015-03-16 11:39:56 +01:00
Ian Southam db48a99678 Restart keepalived if a pub interface gets added or removed 2015-03-16 11:39:55 +01:00
Ian Southam c828653a0a Another correction 2015-03-16 11:39:54 +01:00