Commit Graph

657 Commits

Author SHA1 Message Date
Rohit Yadav 495f7bb1f5
Merge branch '4.9' 2016-11-08 15:21:00 +05:30
Rohit Yadav c09fc27255
Merge branch '4.8' into 4.9 2016-11-08 15:20:45 +05:30
Rohit Yadav b704cefc1f
Merge branch '4.7' into 4.8 2016-11-08 15:20:07 +05:30
Rohit Yadav 0279ac20e4 Merge pull request #1744 from greenqloud/4.7
CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory

* pr/1744:
  CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-08 15:19:28 +05:30
Ólafur Stefán Arnarsson 420d06f4d3 CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory 2016-10-31 21:44:15 +00:00
Murali Reddy 9cc06a8fc8 CLOUDSTACK-9498: VR CsFile search utility methods fail when search string has
're' meta chars, and causing VPN user add/deelte to fail

    -there is no real use of python 're' in CsFile.py utility methods searchString, deleteLine
    Replacing with regular string search instead.

    -modifying the smoke test for VPN user add/delete to have all permissable chars
2016-10-28 17:45:15 +05:30
Rohit Yadav ccd1734dc4 Merge branch '4.9' 2016-10-21 10:15:29 +05:30
Rohit Yadav 9b9b49e10b Merge branch '4.8' into 4.9 2016-10-21 10:14:16 +05:30
Murali Reddy 93ac134f1e CLOUDSTACK-9533: gateway of public IP is not handled correctly when parsing the
cmd_line.json to create ips.json databag

using 'gateway' in cmd_line for the public ip gateway while creating ips.json databag
2016-10-06 16:41:29 +05:30
Murali Reddy 052a9af023 CLOUDSTACK-9515: internal LB vm is not handled when parsing cmd_line.json,
resulting in internal LB vm not come up

parsing cmd_line to create 'ips' data bag, never handled internal lb vm, but still
worked due to another bug. support for internal lb vm is added with this fix
2016-10-05 15:11:00 +05:30
Rajani Karuturi 43f82c991c Merge release branch 4.9 to master
* 4.9:
  fix egress rule incorrect behavior
2016-09-21 15:00:30 +05:30
Rajani Karuturi cc043e9f8f Merge pull request #1666 from murali-reddy/egress_rules
CLOUDSTACK-9480,  CLOUDSTACK-9495 fix egress rule incorrect behaviorWhen 'default egress policy' is set to 'allow' in the network offering, any egress rule that is added will 'deny' the traffic overriding the default behaviour.

Conversely, when 'default egress policy' is set to 'deny' in the network offering, any egress rule that is added will 'allow' the traffic overriding the default behaviour.

While this works for 'tcp', 'udp' as expected, for 'icmp' protocol its always set to ALLOW. This patch keeps all protocols behaviour consistent.

Results of running test/integration/component/test_egress_fw_rules.py.  With out the patch test_02_egress_fr2 test was failing. This patch fixes the test_02_egress_fr2  scenario.
-----------------------------------------------------------------------------------------------------
Test By-default the communication from guest n/w to public n/w is NOT allowed. ... === TestName: test_01_1_egress_fr1 | Status : SUCCESS ===
ok
Test By-default the communication from guest n/w to public n/w is allowed. ... === TestName: test_01_egress_fr1 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_1_egress_fr2 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_egress_fr2 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_1_egress_fr3 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_egress_fr3 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_1_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule without CIDR ... === TestName: test_06_1_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without CIDR ... === TestName: test_06_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without End Port ... === TestName: test_07_1_egress_fr7 | Status : EXCEPTION ===
ERROR
Test Create Egress rule without End Port ... === TestName: test_07_egress_fr7 | Status : SUCCESS ===
ok
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Delete Egress rule ... === TestName: test_09_1_egress_fr9 | Status : SUCCESS ===
ok
Test Delete Egress rule ... === TestName: test_09_egress_fr9 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_1_egress_fr10 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_egress_fr10 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_1_egress_fr11 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_egress_fr11 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_1_egress_fr12 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_egress_fr12 | Status : EXCEPTION ===
ERROR
Test Redundant Router : Master failover ... === TestName: test_13_1_egress_fr13 | Status : SUCCESS ===
ok
Test Redundant Router : Master failover ... === TestName: test_13_egress_fr13 | Status : SUCCESS ===
ok
-----------------------------------------------------------------------------------------------------

* pr/1666:
  fix egress rule incorrect behavior

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2016-09-21 14:54:31 +05:30
Murali Reddy a43abbe47b fix egress rule incorrect behavior
CLOUDSTACK-9480: Egress Firewall: Incorrect use of Allow/Deny for ICMP

     fix ensures, ICMP, TCP, UDP are handled similalry w.r.t egress rule action

CLOUDSTACK-9495: Egress rules functionalty broken when protocol=all specified

     when protocol=all specified, CIDR was ignored. Fix ensures if CIDR is specified
     its always used in configuring iptable rules

 2 new test cased to test /32 CIDR
2016-09-20 16:56:06 +05:30
Rohit Yadav 1d9735c346
Merge branch '4.9' 2016-08-30 22:44:33 +05:30
Rohit Yadav 14504dc7e3 CLOUDSTACK-6432: Prevent DNS reflection attacks
DNS on VR should not be publically accessible as it may be prone to DNS
amplification/reflection attacks. This fixes the issue by only allowing VR
DNS (port 53) to be accessible from guest network cidr, as per the fix in:
https://issues.apache.org/jira/browse/CLOUDSTACK-6432

- Only allows guest network cidrs to query VR DNS on port 53.
- Includes marvin smoke test that checks the VR DNS accessibility checks from
  guest and non-guest network.
- Fixes Marvin sshClient to avoid using ssh agent when password is provided,
  previous some environments may have seen 'No existing session' exception without
  this fix.
- Adds a new dnspython dependency that is used to perform dns resolutions in the
  tests.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 22:39:33 +05:30
John Burwell 8d11511b1f Adds support for four position versions and optional db upgrades
Often, patch and security releases do not require schema migrations or
data migrations.  However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break.  With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path.  This approach allows the upgrade to proceed and
for the database to properly reflect the installed version.  This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.

Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes.  The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers.  This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers.   It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.

  * Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
  a version that was not explicitly specified.  It determines the
  releases the first release before it with database migrations and uses
  that list as the basis for the list for version being calculated.  A
  noop upgrade is then added to the list which causes no schema changes
  or data migrations, but will update the database to the version.
  * Adds unit tests for the upgrade path calculation logic in
  DatabaseUpgradeChecker
  * Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
  versions of this patch
  * Introduces the CloudStackVersion value object which parses and
  compares three (3) and four (4) position version numbers.  This class
  is intended to replace com.cloud.maint.Version.
  * Adds the junit-dataprovider dependency -- allowing test data to be
  concisely generated separately from the execution of a test case.
  Used extensively in the CloudStackVersionTest.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 13:32:32 +05:30
Rohit Yadav 9555492b4d Merge branch '4.9' 2016-08-23 14:16:53 +05:30
Rohit Yadav f13c224da1 Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:39 +05:30
Will Stevens 740bd45be6 Updating pom.xml version numbers for release 4.8.2-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-15 10:31:09 -04:00
Will Stevens a63db21d16 Updating pom.xml version numbers for release 4.8.1
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-08 10:06:21 -04:00
Will Stevens 62aa3b2bfa Updating pom.xml version numbers for release 4.10.0-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-29 10:11:34 -04:00
Will Stevens 227ff3884d Updating pom.xml version numbers for release 4.9.0
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-25 16:56:04 -04:00
Will Stevens d302269fe5 CLOUDSTACK-9437: Create egress chain on upgrade and cleanup for allow all traffic
- Ensure that FW_EGRESS_RULE chain exists after upgrading the router
- Flush allow all egress rule on 0.0.0.0/0, if such a rule exists in the config
  it will be added later (CLOUDSTACK-9437)
2016-07-25 16:44:38 -04:00
Patrick Dube 9ab676206a Added missing rules on router config, fixed ordering of multiple rules, removed duplicate rules, added fix for network stats, added a check for b64 decoding (to pad incorrect b64). Also added a catch exception to be logged on the configure main. 2016-07-22 15:32:20 -04:00
Will Stevens a566cde145 Merge pull request #1609 from pdube/network-acl-add-order
[CLOUDSTACK-9430] Added fix for adding/editing Network ACL rule orderingBUG: https://issues.apache.org/jira/browse/CLOUDSTACK-9430

The issue occurred because all of the ACL rules get inserted before the old ones. Then, the cleanup deletes the duplicate rows, and leaves any new rule in front of the old ones.

Here is an example with a simplified iptables view for ACL
Ex: adding a rule 4
before add:
1,2,3

during add:
1',2',3',4',1,2,3

after add:
4',1,2,3

After fix:
before add:
1,2,3

during add:
1,2,3,1',2',3',4'

after add:
1',2',3',4'

* pr/1609:
  Added fix for adding/editing Network ACL rule ordering

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-18 14:11:13 -04:00
Will Stevens 49df7f283c Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
2016-07-18 14:07:09 -04:00
Will Stevens 01758ee867 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
2016-07-18 14:06:08 -04:00
Will Stevens adaf410109 Merge pull request #1480 from Slair1/S2S-VPN-PFS-Setting
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctlyBug in code set PFS to the same value (yes/no) as DPD.

file.addeq(" pfs=%s" % CsHelper.bool_to_yn(obj['dpd']))

* pr/1480:
  CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-07-18 14:05:18 -04:00
Patrick Dube 6dd6ef0c9a Added fix for adding/editing Network ACL rule ordering 2016-07-11 15:12:41 -04:00
Wido den Hollander 67eb6caa37
Remove /etc/apache2/sites-enabled/000-default in cloud-early-config 2016-06-07 13:03:18 +02:00
Ronald van Zantvoort 91a8faac31 SysVM various fixes to previous refactorings
* make CORS include a regular glob-matched one
* fix NameVirtualHost in CsApp.py as well
* even moar cleanups
2016-06-07 13:03:10 +02:00
Ronald van Zantvoort 58bdb44135 SysVM: Cleanup and removal of old (and dangerous) config files
* ports.conf
* default & default-ssl sites
* SSL config in httpd.conf
* deprecated & dead setup_redundant_router in cloud-early-config
2016-06-07 13:03:10 +02:00
Ronald van Zantvoort 72e83085e1 SysVM cloud-early-config: Intermediate fix for SecStore & CORS
* Take setup from vhost.template rather than default(-ssl)
  * should move into Python CS code as well
* Move CORS setup to separate conf
* Modify vhost template to Optionally include the cors file
* Add NameVirtualHost to vhost template for feature parity with ports.conf
* Take setup from vhost.template rather than default(-ssl)
2016-06-07 13:03:10 +02:00
Ronald van Zantvoort f49a4d944a VR cloud-early-config: Commonize Apache2 common setup 2016-06-07 13:03:10 +02:00
Ronald van Zantvoort 7e290979a9 VR cloud-early-config: Fix Apache2 alias cleanup 2016-06-07 13:03:10 +02:00
Ronald van Zantvoort e32cd1303a VR: consistent SSL setup, vhost is not an example, but a template 2016-06-07 13:03:10 +02:00
Ronald van Zantvoort f379df4bc2 VR CsConfig: reintroduce old get_dns() behaviour for redundant non-VPC's 2016-06-07 13:03:10 +02:00
Ronald van Zantvoort d14a484374 VR CsAddress fixes:
* cleanup imports,
* fix to_str(),
* improve & fix service post_config logic
* don't arpPing when there's no gateway
2016-06-07 13:03:09 +02:00
Ronald van Zantvoort 6055ed6ed6 VR CsApp: Expose config to classes, move vhost confs to proper location, allow for multiple IP's per intf, sanitize servername, don't open port 53 if no DNS is foreseen 2016-06-07 13:03:09 +02:00
Ronald van Zantvoort 748bf43530 VR CsConfig: Add is_router(), is_dns(), has_dns(), has_metadata(), use_extdns(), fix get_dns() with use_extdns() 2016-06-07 13:03:09 +02:00
Ronald van Zantvoort 875379042e VR CsDhcp: allow multiple ranges & finite lease time (fixes CLOUDSTACK-8303) 2016-06-07 13:03:09 +02:00
Ronald van Zantvoort 2790d7a69b VR CsGuestNetwork obey useextdns 2016-06-07 13:03:09 +02:00
Ronald van Zantvoort ed820bebe5 VR merge.py ipalias fix & dhcpconfig stub notification 2016-06-07 13:03:09 +02:00
Will Stevens 237bd46d55 Merge pull request #1423 from syed/ipsec-start-bug
[CLOUDSTACK-9296] Start ipsec for client VPNThis fix starts the IPSEC daemon when enabling client side vpn

* pr/1423:
  [CLOUDSTACK-9296] Start ipsec for client VPN

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-27 15:21:16 -04:00
Will Stevens ff0b095028 Merge release branch 4.8 to master
* 4.8:
  lower the time we wait for interfaces to appear
2016-05-26 15:51:31 -04:00
Will Stevens 7974bfd313 Merge release branch 4.7 to 4.8
* 4.7:
  lower the time we wait for interfaces to appear
2016-05-26 15:50:29 -04:00
Will Stevens d9429f6add Merge pull request #1471 from remibergsma/47_lower_interface_wait
Lower the time we wait for interfaces to appearWaiting for interfaces is tricky. They might never appear.. for example when we have entries in `/etc/cloudstack/ips.json` that haven't been plugged yet. Waiting this long makes everything horribly slow (every vm, interface, static route, etc, etc, will hit this wait, for every device). We've seen CloudStack send an `ip_assoc.json` command for `eth1` public nic only and then the router goes crazy waiting for all other interfaces that were there before reboot and aren't there. If only the router would return to the mgt server a success of `eth1`, it would get the command for `eth2` etc etc. Obviously, a destroy works much faster because no state services, so no knowledge of previous devices so no waits :-)

After a stop/start the router has state in `/etc/cloudstack/ips.json` and every commands waits. Eventually hitting the hardcoded 120 sec timeout.

* pr/1471:
  lower the time we wait for interfaces to appear

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-26 15:49:50 -04:00
Syed 26227768a1 [CLOUDSTACK-9296] Start ipsec for client VPN 2016-05-26 11:25:09 -04:00
Will Stevens 2d825dd07f Merge pull request #1555 from milamberspace/Improve-CS-1475
Add Java Default Certificat Authorities into the keystore if using a custom cert SSL

Related to CLOUDSTACK-1475

* pr/1555:
  Add Java Default Certificat Authorities into the keystore if using a custom cert SSL Related to CLOUDSTACK-1475 Fix some english message

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:57:40 -04:00
Will Stevens 678b28f273 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:54:23 -04:00
Will Stevens b80696cbc0 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-6928: fix issue disk I/O throttling not applied
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.
2016-05-25 22:53:42 -04:00
Will Stevens 5ccebf0f2b Merge pull request #1514 from dsclose/CLOUDSTACK-6975
CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvRRebase of PR #1509 against the 4.7 branch as requested by @swill

One LGTM from @ustcweizhou carried from previous PR. Previous PR will be closed.

Description from PR #1509:

CLOUDSTACK-6975 refers to service monitoring bringing up dnsmasq but this is no-longer accurate, as service monitoring is not active on the post-4.6 routers. These routers still suffer an essentially identical issue, however, because "dnsmasq needs to be restarted each time configure.py is called in order to avoid lease problems." As such, dnsmasq is still running on backup RvRs, causing the issues described in CLOUDSTACK-6975.

This PR is based on a patch submitted by @ustcweizhou. The code now checks the redundant state of the router before restarting dnsmasq.

RvR networks without this patch have dnsmasq running on both master and backup routers. RvR networks with this patch have dnsmasq running on only the master router.

* pr/1514:
  CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-25 22:52:55 -04:00
Milamber d8767ec13e Add Java Default Certificat Authorities into the keystore if using a custom cert SSL
Related to CLOUDSTACK-1475
Fix some english message
2016-05-25 06:34:15 +01:00
Will Stevens 1dd866068a Merge pull request #1414 from resmo/fix/cleanup
SystemVM cleanupsfrom the logrotate docs

> size - With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem).

> Note: If size and time interval options are specified at same time, only size option take effect. it causes log files to be rotated without regard for the last rotation time. If both log size and timestamp of a log file need to be considered by logrotate, the minsize option should be used. logrotate will rotate log file when they grow bigger than minsize, but not before the additionally specified time interval.

* pr/1414:
  systemvm, logrotate: remove daily explicitly as it is ignored

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-20 08:28:30 -04:00
Will Stevens 88cd182272 Forward merged 4.8 to master and fixed conflicts in #1331 2016-05-19 12:08:08 -04:00
Will Stevens 091ca567f0 Merge release branch 4.7 to 4.8
* 4.7:
  Revert "Merge pull request #1482 from remibergsma/iptables-fix"
2016-05-19 12:03:38 -04:00
Remi Bergsma 74f60df828 Revert "Merge pull request #1482 from remibergsma/iptables-fix"
Seems to have a license issue so reverting for now.

This reverts commit 9a20ab8bcb, reversing
changes made to 7a0b37a29a.
2016-05-19 11:04:46 +02:00
Will Stevens 8f330b0b92 Merge release branch 4.7 to 4.8
* 4.7:
  Fix Sync of template.properties in Swift
  Configure rVPC for router.redundant.vrrp.interval advert_int setting
  Have rVPCs use the router.redundant.vrrp.interval setting
  Resolve conflict as forceencap is already in master
  Split the cidr lists so we won't hit the iptables-resture limits
  Check the existence of 'forceencap' parameter before use
  Do not load previous firewall rules as we replace everyhing anyway
  Wait for dnsmasq to finish restart
  Remove duplicate spaces, and thus duplicate rules.
  Restore iptables at once using iptables-restore instead of calling iptables numerous times
  Add iptables copnversion script.
2016-05-18 15:54:32 -04:00
Will Stevens ebc70a51e2 Merge pull request #1486 from remibergsma/reimplement-vrrp-setting-47
Reimplement router.redundant.vrrp.interval settingGlobal setting `router.redundant.vrrp.interval` is not used any more and it is now set to a hardcoded 1.

This results in a failover from master->backup when the backup doesn't hear from the master in ~3.6sec. This is a bit too tight, as we've seen failovers during live migrations. We could reproduce it in about half of the cases. Setting this to setting to 2 (tested it by hardcoding it in the systemvms) gives twice as much time and we didn't see issues any more. Instead of updating the hardcoded setting from 1 to 2, I reimplemented the global setting by sending it to the router with the cmd_line, as the non-VPC router also does.

Background:
Why is the maximum failover time in the example 3.6 seconds? This comes from the advertisement interval and the skew time. The default advertisement interval is 1 second (configurable in keepalived.conf). The skew time helps to keep everyone from trying to transition at once. It is a number between 0 and 1, based on the formula (256 - priority) / 256

As defined in the RFC, the backup must receive an advertisement from the master every (3 * advert_int) + skew_time seconds. If it doesn't hear anything from the master, it takes over. With a backup router priority of 100 (as in the example), the failover will happen at most 3.6 seconds after the master goes down.

Source: http://www.hollenback.net/KeepalivedForNetworkReliability

* pr/1486:
  Configure rVPC for router.redundant.vrrp.interval advert_int setting
  Have rVPCs use the router.redundant.vrrp.interval setting

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-18 15:52:38 -04:00
Will Stevens 9a20ab8bcb Merge pull request #1482 from remibergsma/iptables-fix
Restore iptables at once using iptables-restore instead of calling iptables numerous timesThis makes handling the firewall rules about 50-60 times faster because it is generated in memory and then loaded once. It's work by @borisroman see PR #1400. Reopened it here because I think this is a great improvement.

* pr/1482:
  Resolve conflict as forceencap is already in master
  Split the cidr lists so we won't hit the iptables-resture limits
  Check the existence of 'forceencap' parameter before use
  Do not load previous firewall rules as we replace everyhing anyway
  Wait for dnsmasq to finish restart
  Remove duplicate spaces, and thus duplicate rules.
  Restore iptables at once using iptables-restore instead of calling iptables numerous times
  Add iptables copnversion script.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-18 15:50:20 -04:00
Remi Bergsma 9c0eee4387 Configure rVPC for router.redundant.vrrp.interval advert_int setting 2016-05-13 14:37:04 +02:00
Will Stevens 2b4b8aa40c Merge release branch 4.8 to master
* 4.8:
  This patch addresses two issues:
2016-05-12 18:25:08 -04:00
Will Stevens 3bbeb6af95 Merge release branch 4.7 to 4.8
* 4.7:
  This patch addresses two issues:
2016-05-12 18:23:37 -04:00
Will Stevens bbb2dd034e Merge pull request #1536 from ntavares/useextdns_rvmvip47
Honour GS use_ext_dns and redundant VR VIPThis patch addresses two issues:

On redundant VR setups, the primary resolver being handed out to instances is the guest_ip (primary IP for the VR). This might lead to problems upon failover, at least while the DHCP lease doesn't update (because the primary resolver will be checked first until times out, however it'll be gone upon failover).

If Global Setting use_ext_dns is true, we don't want the VR to be the primary resolver at all.

* pr/1536:
  This patch addresses two issues:

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 18:23:01 -04:00
Will Stevens 103d62ee02 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:04:53 -04:00
Will Stevens dbbc37ecc7 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:03:57 -04:00
Will Stevens 3fab75772f Merge pull request #1474 from remibergsma/47_private_gw_initial_config
Handle private gateways more reliablyWhen initialising a VPC router we need to know which IP/device corresponds to a private gateway. This is to solve a problem when stop/starting a VPC router (which gets the private gateway config as a guest network and as a result breaks the functionality). You read it right, the private gateway is sent as type=guest after reboot and type=public initially.

Before this change, you could add a private gw to a running router but you couldn't restart it (it would mix up the tiers). Now the private gateway is detected properly and it works just fine.

Booting without private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.0.42",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "None",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

Booting with private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.2.227",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "10.201.10.1",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

And:
```
cat cmdline
vpccidr=10.0.0.0/24 domain=cs2cloud dns1=8.8.8.8 privategateway=10.201.10.1 template=domP name=r-167-VM eth0ip=169.254.2.227 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w baremetalnotificationapikey=V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw host=192.168.22.61 port=8080
```

Logs:
```
2016-02-24 20:08:45,723 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-458d4c52 job-1402/job-1403 ctx-d5355fca) (logid:5772906c) Set privategateway field in cmd_line.json to 10.201.10.1
```

* pr/1474:
  Handle private gateways more reliably
  Add private gateway IP to router initialization config

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:02:55 -04:00
Will Stevens 919660d093 Merge pull request #1472 from remibergsma/47_fix_static_router_master_change
Apply static routes on change to master stateRefactored static routes for private gateways so they also get loaded when the router switches to master state. Otherwise they're lost and connections drop after fail over.

* pr/1472:
  apply static routes on change to master state

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:01:55 -04:00
Will Stevens 309a60ea71 Merge pull request #1483 from remibergsma/pr1413-wilder-47
CLOUDSTACK-9287 - Fix unique mac address per rVPC routerThis is work by @wilderrodrigues, see PR #1413 It contains important fixes and I think it needs to be included so I send the PR again.

* pr/1483:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:00:40 -04:00
dean.close 38b3bdd488 CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR. 2016-05-09 11:34:47 +01:00
Nuno Tavares c269097a27 This patch addresses two issues:
On redundant VR setups, the primary resolver being handed out to instances is the guest_ip (primary IP for the VR). This might lead to problems upon failover, at least while the DHCP lease doesn't update (because the primary resolver will be checked first until times out, however it'll be gone upon failover).

If Global Setting use_ext_dns is true, we don't want the VR to be the primary resolver at all.
2016-05-08 22:47:55 +02:00
Will Stevens 29ba71f2db Merge pull request #1463 from DaanHoogland/CLOUDSTACK-9336
CLOUDSTACK-9336 surround the execution of baremetal-vr.py with condition

* pr/1463:
  CLOUDSTACK-9336 surround the execution of baremetal-vr.py with condition

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-04 10:29:15 -04:00
Rene Moser 1889e399dd systemvm, logrotate: remove daily explicitly as it is ignored
If the size directive is used, logrotate will ignore the daily, weekly, monthly,
and yearly directives.

remove cloud-cleanup

This script does not do anything because it fails due missing /var/log/cloud directory. Logrotate is used for this functionality.
2016-05-04 00:23:30 +02:00
Remi Bergsma ebb7cb690d Resolve conflict as forceencap is already in master 2016-04-30 21:09:33 +02:00
Anshul Gangwar 7db5f1833e CLOUDSTACK-9164: Prevent firefox's quick search from opening in VM console 2016-04-25 11:37:43 +05:30
Will Stevens 37afba05f4 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9172 Added cross zones check to delete template and iso
  Check the existence of 'forceencap' parameter before use
  systemvm: set default umask 022 in injectkeys.sh
2016-04-21 16:32:36 -04:00
Will Stevens c2fc0c4cd3 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9172 Added cross zones check to delete template and iso
  Check the existence of 'forceencap' parameter before use
  systemvm: set default umask 022 in injectkeys.sh
2016-04-21 16:31:49 -04:00
Will Stevens a7fef86bb3 Merge pull request #1402 from remibergsma/forceencap_check
Check the existence of 'forceencap' parameter before useCheck the existence of 'forceencap' parameter before use.

Error seen:

```
Traceback (most recent call last):
  File "/opt/cloud/bin/update_config.py", line 140, in <module>
    process_file()
  File "/opt/cloud/bin/update_config.py", line 54, in process_file
    finish_config()
  File "/opt/cloud/bin/update_config.py", line 44, in finish_config
    returncode = configure.main(sys.argv)
  File "/opt/cloud/bin/configure.py", line 1003, in main
    vpns.process()
  File "/opt/cloud/bin/configure.py", line 488, in process
    self.configure_ipsec(self.dbag[vpn])
  File "/opt/cloud/bin/configure.py", line 544, in configure_ipsec
    file.addeq(" forceencaps=%s" % CsHelper.bool_to_yn(obj['encap']))
KeyError: 'encap'
```

* pr/1402:
  Check the existence of 'forceencap' parameter before use

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-04-21 16:30:16 -04:00
Remi Bergsma 53de78f642 Check the existence of 'forceencap' parameter before use 2016-04-19 11:11:19 +02:00
Slair1 7b9b328f30 CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
Bug in code set PFS to the same value (yes/no) as DPD.

file.addeq(" pfs=%s" % CsHelper.bool_to_yn(obj['dpd']))
2016-04-15 12:00:16 -05:00
Will Stevens 0dcaf197b4 Merge release branch 4.8 to master
* 4.8:
  speedup iptables by prefetching the variables
2016-04-12 10:11:08 -04:00
Will Stevens f8e71b0235 Merge release branch 4.7 to 4.8
* 4.7:
  speedup iptables by prefetching the variables
2016-04-12 10:10:17 -04:00
kollyma 0a01e82c23 speedup iptables by prefetching the variables 2016-04-11 20:58:03 +02:00
Remi Bergsma f4f9b3ab4e Handle private gateways more reliably 2016-04-10 20:06:44 +02:00
Wilder Rodrigues 78bbd498e7 CLOUDSTACK-9287 - Fix RVR public interface 2016-04-09 21:14:41 +02:00
Wilder Rodrigues c41edc1fe6 CLOUDSTACK-9287 - Refactor the interface state configuration
- This also refactors the CsAddress in order to offer better readability in a couple of methods.
2016-04-09 21:14:25 +02:00
Remi Bergsma 6a767732f9 CLOUDSTACK-9287 - Bring up the private gw interface on state change to master 2016-04-09 21:14:10 +02:00
Remi Bergsma 057b54aa3e CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw 2016-04-09 21:13:47 +02:00
Wilder Rodrigues d93b008deb CLOUDSTACK-9287 - Put private gateway interface down on backup router 2016-04-09 21:13:35 +02:00
Remi Bergsma b9feb39e17 apply static routes on change to master state 2016-04-07 20:57:58 +02:00
Remi Bergsma 3636ad1114 lower the time we wait for interfaces to appear
They might never appear.. for example when we have entries in
/etc/cloudstack/ips.json that haven't been plugged yet. Waiting
this long makes everything horribly slow (every vm, interface,
static route, etc, etc, will hit this wait, for every device).
2016-04-07 20:52:33 +02:00
dahn 2a88b11132 CLOUDSTACK-9336 surround the execution of baremetal-vr.py with condition 2016-04-05 13:28:39 +02:00
Boris Schrijver 30741a3309 Split the cidr lists so we won't hit the iptables-resture limits 2016-02-05 21:11:59 +01:00
Boris Schrijver 709be45cf0 Check the existence of 'forceencap' parameter before use 2016-02-05 21:06:53 +01:00
Boris Schrijver 18d5cd2855 Do not load previous firewall rules as we replace everyhing anyway 2016-02-05 21:04:06 +01:00
Boris Schrijver eb9706b655 Wait for dnsmasq to finish restart 2016-02-05 12:02:58 +01:00
Boris Schrijver 69e9348900 Remove duplicate spaces, and thus duplicate rules. 2016-02-05 12:02:57 +01:00
Boris Schrijver b857f79fcb Restore iptables at once using iptables-restore instead of calling iptables numerous times 2016-02-05 12:02:57 +01:00
Boris Schrijver 0f75042566 Add iptables copnversion script.
Source: https://raw.githubusercontent.com/sl0/conv/master/iptables_converter.py
2016-02-05 12:02:57 +01:00
Remi Bergsma 7017a829ea Merge release branch 4.8 to master
* 4.8:
  Display hostname the VPC router runs on
  CLOUDSTACK-9266: Make deleting static routes in private gw work
  CLOUDSTACK-9264: Make /32 static routes for private gw work
2016-02-04 09:27:40 +01:00