Commit Graph

738 Commits

Author SHA1 Message Date
Rohit Yadav 3587bc4b54 Merge remote-tracking branch 'origin/4.10' 2017-07-22 11:09:23 +02:00
Rohit Yadav ffddd6db09 Merge remote-tracking branch 'origin/4.9' into 4.10 2017-07-22 11:07:42 +02:00
Rohit Yadav aa8a721c39 CLOUDSTACK-9838: Allow ingress traffic between guest VMs via snat IPs
This enables the firewall/mangle tables rules to ACCEPT instead of RETURN, which
is the same behaviour as observed in ACS 4.5. By accepting the traffic, guest
VMs will be able to communicate tcp traffic between each other over snat public
IPs.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-22 10:53:21 +02:00
Rajani Karuturi 4bc7c270fa Updating pom.xml version numbers for release 4.11.0.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-12 12:09:38 +05:30
Rajani Karuturi 4f229d19ad Updating pom.xml version numbers for release 4.10.1.0-SNAPSHOT
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-10 10:36:24 +05:30
Rajani Karuturi 9d2893d44a Updating pom.xml version numbers for release 4.10.0.0
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
2017-07-03 10:06:43 +05:30
Sigert Goeminne 77da97aad9 CLOUDSTACK-9980: Fix for DNS after network restart with clean up.
Configure.py without arguments gives IndexError.
2017-06-28 12:21:14 +02:00
Rajani Karuturi d577adeaa8 Merge release branch 4.9 to master
* 4.9:
  Revert "Merge pull request #2084 from shapeblue/passwd-speedup"
2017-06-26 09:59:13 +05:30
Rajani Karuturi 24434beb42 Revert "Merge pull request #2084 from shapeblue/passwd-speedup"
This reverts commit 48f413a982, reversing
changes made to 5f35c15b6b.
2017-06-26 09:58:33 +05:30
Rajani Karuturi b9a4cb8770 Merge release branch 4.9 to master
* 4.9:
  vRouter defer configure: Resolve merge conflicts
  vRouter: vr_cfg: style consistency fixes
  vRouter: vr_cfg: Keep the original cfg
  vRouter: prevent fh leakage and use buffered writes in DataBags
2017-06-20 10:14:50 +05:30
Ronald van Zantvoort 330a0c7f2b vRouter defer configure: Resolve merge conflicts 2017-06-07 18:20:02 +02:00
Ronald van Zantvoort a2eb0f23a9 vRouter: vr_cfg: style consistency fixes 2017-06-07 18:11:33 +02:00
Ronald van Zantvoort 8f5275399e vRouter: vr_cfg: Keep the original cfg 2017-06-07 18:11:33 +02:00
Ronald van Zantvoort c10c3245d1 vRouter: prevent fh leakage and use buffered writes in DataBags 2017-06-07 18:11:33 +02:00
Rohit Yadav ed376fcad6 Merge remote-tracking branch 'origin/4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-06-07 11:21:27 +05:30
Jayapal 065fa1c849 CLOUDSTACK-9669:Advaced zone isolated network egress destination cidr support
CLOUDSTACK-9669:egress destination cidr VR python script changes
    CLOUDSTACK-9669:egress destination API and orchestration changes
    CLOUDSTACK-9669: Added the ipset package in systemvm template
    CLOUDSTACK-9669:Added licence header for new files
    CLOUDSTACK-9669: replacing 0.0.0.0/0 with the network cidr

     ipset member add with 0.0.0.0/0 fails. So 0.0.0.0/0 replaced with the network cidr.
     In source cidr 0.0.0.0/0 is nothing but network cidr.
     updated the default egress all cidr with network cidr
2017-06-06 15:51:00 +05:30
Rajani Karuturi c3d6b9e182 Merge pull request #2016 from Accelerite/CLOUDSTACK-9835
CLOUDSTACK-9835 : To make management server and SSVM to be in time sync
2017-06-06 15:33:18 +05:30
Rajani Karuturi 48f413a982 Merge pull request #2084 from shapeblue/passwd-speedup
Passwd speedup
2017-06-06 15:32:02 +05:30
Nitesh Sarda 10b25adc46 CLOUDSTACK-9017 : VPC VR DHCP broken for multihomed guest VMs 2017-05-23 17:34:20 +05:30
Rajani Karuturi 8b3cadb55e Merge release branch 4.9 to master
* 4.9:
  Do not set gateway to 0.0.0.0 for windows clients
  CLOUDSTACK-9904: Fix log4j to have @AGENTLOG@ replaced
  ignore bogus default gateway   when a shared network is secondary the default gateway gets overwritten by a bogus one   dnsmasq does the right thing and replaces it with its own default which is not good for us   so check for '0.0.0.0'
  Activate NioTest following changes in CLOUDSTACK-9348 PR #1549
  CLOUDSTACK-9828: GetDomRVersionCommand fails to get the correct version as output Fix tries to return the output as a single command, instead of appending output from two commands
  CLOUDSTACK-3223 Exception observed while creating CPVM in VMware Setup with DVS
  CLOUDSTACK-9787: Fix wrong return value in NetUtils.isNetworkAWithinNetworkB
2017-05-17 11:36:52 +05:30
Rajani Karuturi cbd3662482 Merge pull request #2055 from shapeblue/sharedisolated
CLOUDSTACK-9887 ignore bogus default gateway
2017-05-17 09:53:39 +05:30
Jayapal da8a3567a3 CLOUDSTACK-9848: Added exit status checking for the iptables commands 2017-05-09 17:40:02 +05:30
Daan Hoogland 710d3bff3f rat 2017-05-08 07:42:04 +02:00
Daan Hoogland 00add83749 remaining conflicting code for vm_passwd speedup 2017-05-06 22:12:36 +02:00
Remi Bergsma 9a21f56f8a Speedup vm start by making vm_passwd saving much faster
- do not keep passwords in databag (/etc/cloudstack/vmpasswd.json)
- process only the password we get in (vm_password.json) from mgt server
- lookup the correct passwd server instead of adding passwd to all of them

Example:
- 4 tiers and 199 VMs running
- Start vm 200 would cause new passwd from vm_password.json (1) to be merged with /etc/cloudstack/vmpasswd.json (199)
- A curl command was exected foreach password (200) foreach tier (4) resulting in 800 calls
- In fact, since passwds are never cleaned it could very well be even more as the ip address was the key in the json file so until the ip address was reused the original password would remain and be sent to passwd server every time another vm starts.
- This took ~40 seconds

Now we just figure out the right tier and only process the new password resulting in a single curl call.
- takes 0,03 seconds!
2017-05-06 21:48:25 +02:00
Will Stevens 45a96a7f95 remove unnecessary VPN drops 2017-04-27 13:19:25 -04:00
Harikrishna Patnala 8a4f3e53b6 CLOUDSTACK-9835 : Management server and SSVM should be in time sync
Added a new configuration paremetar "ntp.server.list" to configure NTP server ip in NTP settings of SSVM
2017-04-25 13:01:33 +05:30
Daan Hoogland 0db9c980a6 ignore bogus default gateway
when a shared network is secondary the default gateway gets overwritten by a bogus one
  dnsmasq does the right thing and replaces it with its own default which is not good for us
  so check for '0.0.0.0'
2017-04-20 09:36:17 +02:00
Daan Hoogland f07d729086 CLOUDSTACK-9408 for the move away from download.cloud.com
- commented some occurences of cloud.com as being harmless
  * examples
  * identifiers (internal)
 - changed the URL for vhd-util download
 - changed comments from 'cloud.com' to 'Apache CloudStack'
2017-04-20 11:35:36 +05:30
Will Stevens bb4087733b changed the order fix to be closer to the original code 2017-03-17 15:07:11 -04:00
Will Stevens 8d4855b4eb CLOUDSTACK-9811: fixed an issue if the dev is not in the databag 2017-03-14 09:23:47 -04:00
Abhinandan Prateek e303eee8e6 CLOUDSTACK-9828: GetDomRVersionCommand fails to get the correct version as output
Fix tries to return the output as a single command, instead of appending output from two commands
2017-03-09 14:15:45 +05:30
David Mabry 3e54388858 moved logrotate from cron.daily to cron.hourly for vpcrouter in cloud-early-config
(cherry picked from commit 450deed847)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-03-03 14:00:43 +05:30
Rajani Karuturi 017c42b625 Merge pull request #1907 from swill/fix_vr_ip
Fix public IPs not being removed from the VR when deprovisionedThis PR replaces #1706.  It does not remove the IP from the database, but it does deprovision the IP correctly from the VR when the public IP is removed.

* pr/1907:
  Fix public IPs not being removed from the VR when deprovisioned

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-28 05:56:03 +05:30
Rajani Karuturi 48cbef6d24 Merge pull request #1922 from Accelerite/vpcApub
CLOUDSTACK-9757: Fixed issue in traffic from additional public subnetAcquire ip from additional public subnet and configure nat on that ip.
After this pick any from that network and access additional public subnet from this vm. Traffic is supposed to go via additional public subnet interface in the VR.

* pr/1922:
  CLOUDSTACK-9757: Fixed issue in traffic from additional public subnet

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-28 05:53:59 +05:30
Rajani Karuturi fa85151be9 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
  CLOUDSTACK-9788: Fix exception listNetworks with pagesize=0
  CLOUDSTACK-8663: Fixed various issues to allow VM snapshots and volume snapshots to exist together
  Fix HVM VM restart bug in XenServer
2017-02-28 05:47:06 +05:30
Rajani Karuturi b95bf8fcd8 Merge release branch 4.8 to 4.9
* 4.8:
  CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
2017-02-28 05:45:31 +05:30
Rajani Karuturi f9ef6ca667 Merge pull request #1948 from greenqloud/pr-faster-static-nats
[CLOUDSTACK-9793] Faster IP in subnet checkThis change removes the conversion from IPNetwork to list in one of the router scripts. This makes the router faster at processing static NAT rules, which can prevent timeouts when attaching or detaching IPs.

With the `list` conversion, it has to potentially check a list of 65536 IP strings multiple times. We assume that the comparison implemented in the IPNetwork is far more efficient. We have seen speed-up from 218 seconds to enable static NAT with 18 IPs on the router to 2 or 3 seconds by removing this cast. This also fixes a potential bug where adding IPs to a router time out because the scripts are taking too long. 218 seconds, for example, is beyond the timeout on the KVM agent for script execution, and then all enableStaticNat operations will fail.

* pr/1948:
  CLOUDSTACK-9793: Faster ip in subnet check

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-28 05:33:38 +05:30
Jayapal baac747089 CLOUDSTACK-9757: Fixed issue in traffic from additional public subnet 2017-02-24 14:50:48 +05:30
Will Stevens 23f64a13a8 Fix public IPs not being removed from the VR when deprovisioned 2017-02-22 13:08:11 -05:00
Joakim Sernbrant 5604b4c59e CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
* rotate both daily and by size by using maxsize in stead of size
* decrease the max size to 10M for rsyslog files
* remove delaycompress for rsyslog files
* increase rotate to 10 for cloud.log
2017-02-22 11:04:52 +01:00
David Mabry 450deed847 moved logrotate from cron.daily to cron.hourly for vpcrouter in cloud-early-config 2017-02-20 09:06:46 -06:00
Stefania d5c5eb10f8 CLOUDSTACK-9793: Faster ip in subnet check
This change removes an unnecessary conversion from IPNetwork
to list in one of the router scripts. This makes the router
faster at processing static NAT rules, which can prevent
timeouts when attaching or detaching IPs.
2017-02-17 13:20:11 +00:00
Rajani Karuturi 13bfdd71e6 Merge pull request #1741 from swill/strongswanvpn
Updated StrongSwan VPN ImplementationThis PR is a merge of @jayapalu changes in #872 and the changes I had to make to get the functionality working.

I have done pretty extensive testing of this code so far and we are looking to be in pretty good shape.  One thing to note is that a `Diffie-Hellman` group **is required** in order for this feature to work correctly.  It is not highlighted in the tests below, but I have shown that the `PFS` is not required for this feature to work.  In #872 I have shown a more exhaustive set of tests of this code, but I have limited this set of tests to a recommended `IKE` and `ESP` configuration in order to reduce the noise and test the other areas of functionality.

**Test Results**
I am testing this functionality by creating two VPCs with VMs in each and creating a S2S VPN connection between the two VPCs. Then I SSH into a VM in one VPC and I ping the private IP of a VM in the other VPC. Then I tear it down and try a different configuration.

_Setup_

```
VPC 1                          VPC 2
=====                          =====
VPN Gateway                    VPN Gateway
VPN Customer Gateway           VPN Customer Gateway
VPN Connection        <--->    VPN Connection
 - Passive = True               - Passive = False
```

_Legend_
`SKIP` => At least one of the VPN Connections did not come up, so no test was run.
`OK` => The ping test was successful over the S2S VPN connection.
`FAIL` => The ping test failed over the S2S VPN connection.

`Passive` => Specifies if either the `<vpc_1> : <vpc_2>` sides of the VPN Connection is set to passive.
`Conn State` => Specifies the connection status of the `<vpc_1> : <vpc_2>` VPN Connection in the UI.
`Requires Reset` => If the ping test does not result in an `OK`, then a VPN Connection Reset is performed on either `<vpc_1> : <vpc_2>` sides of the VPN Connection based on which side is not showing `Connected`.  The results in the `Status` column is the final result after the reset is performed.

_Results_

```
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| Status | IKE & ESP            | DPD   | Encap | IKE Life | ESP Life | Passive       | Conn State                  | Requires Reset |
+========+======================+=======+=======+==========+==========+===============+=============================+================+
| OK     | aes128-sha1;modp1536 | True  | False | 86400    | 3600     | True : False  | Disconnected : Connected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | True  | 86400    | 3600     | True : False  | Disconnected : Connected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | False |          | 3600     | True : False  | Disconnected : Connected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | False | 86400    |          | True : False  | Disconnected : Connected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | False |          |          | True : False  | Disconnected : Connected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | False | 86400    | 3600     | False : False | Connected : Connected       | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | False | 86400    | 3600     | True : True   | Disconnected : Disconnected | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | True  | False | 86400    | 3600     | False : True  | Connected : Disconnected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | False | False | 86400    | 3600     | False : False | Connected : Connected       | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | False | False | 86400    | 3600     | True : False  | Disconnected : Connected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | False | False | 86400    | 3600     | True : True   | Disconnected : Disconnected | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK     | aes128-sha1;modp1536 | False | False | 86400    | 3600     | False : True  | Connected : Disconnected    | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| SKIP   | aes128-sha1          | True  | False | 86400    | 3600     | True : False  | Disconnected : Error        | True : False   |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| SKIP   | aes128-sha1          | False | False | 86400    | 3600     | True : False  | Disconnected : Error        | True : False   |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| FAIL   | aes128-sha1          | True  | False | 86400    | 3600     | True : True   | Disconnected : Disconnected | True : True    |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| SKIP   | aes128-sha1          | True  | False | 86400    | 3600     | False : False | Connected : Error           | False : False  |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
```

* pr/1741:
  complete implementation of the StrongSwan VPN feature

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-14 17:59:17 +05:30
Jayapal 45c3d94d83 CLOUDSTACK-9715: Update somaxconn value to default value 2017-02-13 15:10:10 +05:30
Rajani Karuturi 2d5054f655 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9770: fix missing ip routes in VR
2017-02-08 15:02:30 +05:30
Wei Zhou 8c69cb1c1f CLOUDSTACK-9770: fix missing ip routes in VR 2017-02-03 17:51:46 +01:00
Rajani Karuturi 8e069ed132 Merge release branch 4.9 to master
* 4.9:
  FIX issue on preshared key if we disable/enable remote access vpn
2017-02-03 06:38:14 +05:30
Rajani Karuturi c8916792c6 Merge pull request #1890 from ustcweizhou/vpn-preshared-key-issue
[4.9] CLOUDSTACK-9712: FIX issue on preshared key if we disable/enable remote access vpnWay to reproduce the issue
(1) enable remote access vpn
root@r-8349-VM:~# cat /etc/ipsec.d/ipsec.any.secrets
: PSK "mVSx5KDXCPYX7X5DGb2W8yNW"

(2) disable/enable vpn
root@r-8349-VM:~# cat /etc/ipsec.d/ipsec.any.secrets
: PSK "mVSx5KDXCPYX7X5DGb2W8yNW"
: PSK "HeV3dHZpZXt4chhfvhx8D83C"

Expected configuration:
root@r-8349-VM:~# cat /etc/ipsec.d/ipsec.any.secrets
: PSK "HeV3dHZpZXt4chhfvhx8D83C"

* pr/1890:
  FIX issue on preshared key if we disable/enable remote access vpn

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-03 06:37:50 +05:30
Rajani Karuturi 5309c42058 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9692: Fix password server issue in redundant VRs
2017-02-03 06:32:27 +05:30
Rajani Karuturi 42f0340a1d Merge pull request #1871 from ustcweizhou/RVR-passwd-server
[4.9] CLOUDSTACK-9692: Fix password server issue in redundant VRsThe password server in RVRs has wrong parameters as the gateway of guest nics is None.
In this case, we should get the gateway from /var/cache/cloud/cmdline.
This issue is caused by commit 45642b8382

* pr/1871:
  CLOUDSTACK-9692: Fix password server issue in redundant VRs

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2017-02-03 06:31:55 +05:30
Will Stevens f045d65b90 complete implementation of the StrongSwan VPN feature 2017-02-02 16:18:06 -05:00
Rohit Yadav 8b6e96bca9 Updating pom.xml version numbers for release 4.9.3.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-06 10:40:15 +05:30
Wei Zhou 16c2cd0244 FIX issue on preshared key if we disable/enable remote access vpn
Way to reproduce the issue
(1) enable remote access vpn
root@r-8349-VM:~# cat /etc/ipsec.d/ipsec.any.secrets
: PSK "mVSx5KDXCPYX7X5DGb2W8yNW"

(2) disable/enable vpn
root@r-8349-VM:~# cat /etc/ipsec.d/ipsec.any.secrets
: PSK "mVSx5KDXCPYX7X5DGb2W8yNW"
: PSK "HeV3dHZpZXt4chhfvhx8D83C"

Expected configuration:
root@r-8349-VM:~# cat /etc/ipsec.d/ipsec.any.secrets
: PSK "HeV3dHZpZXt4chhfvhx8D83C"
2017-01-05 12:14:13 +01:00
Rohit Yadav dfc39c1f08 Updating pom.xml version numbers for release 4.9.2.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-03 12:28:47 +05:30
Wei Zhou 066b374c35 CLOUDSTACK-9692: Fix password server issue in redundant VRs
The password server in RVRs has wrong parameters as the gateway of guest nics is None.
In this case, we should get the gateway from /var/cache/cloud/cmdline.
2016-12-30 09:35:00 +01:00
Rohit Yadav a9f45dfc5f
Merge branch '4.9' 2016-12-23 17:50:42 +05:30
Rohit Yadav 2088f0ad73 Merge pull request #1783 from jayapalu/CLOUDSTACK-9615
CLOUDSTACK-9615: Fixd applying ingress rules without portsWhen ingress rule is applied without ports (port start and port end params are not passed) then API/UI is showing rule got applied but in the VR, iptables rule not got applied.

Fixed this issue in the VR script.

* pr/1783:
  CLOUDSTACK-9615: Fixed applying ingress rules without ports

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 12:37:13 +05:30
Rohit Yadav 5e19e64f2f Updating pom.xml version numbers for release 4.9.2.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-16 20:48:16 +05:30
Jayapalu fed58eebdd CLOUDSTACK-9615: Fixed applying ingress rules without ports 2016-12-12 11:11:02 +05:30
Rohit Yadav af2679959b Updating pom.xml version numbers for release 4.9.1.0
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-10 08:38:03 +05:30
Rohit Yadav 11dd9fb1ca
Merge branch '4.9' 2016-12-08 17:18:46 +05:30
Murali Reddy 8b4c36ef50 CLOUDSTACK-9659: mismatch in traffic type in ip_associations.json and ips.json
As part of the bug 'CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly'
issue of mismatch of traffic type represented by 'nw_type' in config sent by management server in
ip_associations.json and how it is persisted in the ips.json data bag are differnet,
is addressed, however missed the change in final merge.
this bug is to add the functionality in cs_ip.py, to lower the traffic type sent by management server before persisting in the ips.json databag
2016-12-08 14:19:01 +05:30
Rohit Yadav 6bdc411ff2
Merge branch '4.9' 2016-12-08 00:04:26 +05:30
Murali Reddy 6749785cab CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly
-when processing static nat rule, add a mangle table rule, to mark the traffic
   from the guest vm when it has associated static nat rule so that traffic gets
   routed using the route tabe of the device which has public ip associated

  -fix the case where nic_device_id is empty when ip is getting disassociated
   resulting in empty deviceid in ips.json

  -add utility methods in CsRule, and CsRoute to add 'ip rule' and 'ip route' rules respectivley

  -ensure traffic from all public interfaces are connection marked with device number, and restored
   for the reverse traffic. use the connection marked number to do device specific routing table lookup
   fill the device specific routing table with default route

  -component tests for testing multiple public interfaces of VR
2016-12-07 14:33:24 +05:30
Rohit Yadav d0481723e3
Merge branch '4.9' 2016-11-27 15:50:18 +05:30
Rohit Yadav cc72e4da64 systemvm: Fix regression from 825935
Fixes merge conflict issue incorrectly fixed during a fwd-merge in 825935
from PR #1766

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-27 15:43:10 +05:30
Rohit Yadav 16c2761cea Merge PR #1545
CLOUDSTACK-8715: Add channel to Instances for Qemu Guest Agent

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-24 23:12:07 +05:30
Rohit Yadav c6bb8c6f41
Merge branch '4.9' 2016-11-24 12:45:01 +05:30
Rohit Yadav 825935da69
Merge branch '4.8' into 4.9 2016-11-24 12:44:19 +05:30
Rohit Yadav 90ae04b791
Merge pull request #1766 from murali-reddy/vr-default-network-gateway
CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network when guest VM has nic's in more than one guest network set the tag for each host in /etc/dhcphosts.txt, and use the tag to add exception in /etc/dhcpopts.txt to prevent sending default route, dns server in case if the nic is in non-default network

this was the behaviour with edithosts.sh prior to 4.6

* pr/1766:
  CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-24 12:39:36 +05:30
Wido den Hollander 2a5f37c1b1
CLOUDSTACK-8715: Add channel to Instances for Qemu Guest Agent
This commit adds a additional VirtIO channel with the name
'org.qemu.guest_agent.0' to all Instances.

With the Qemu Guest Agent the Hypervisor gains more control over the Instance if
these tools are present inside the Instance, for example:

* Power control
* Flushing filesystems
* Fetching Network information

In the future this should allow safer snapshots on KVM since we can instruct the
Instance to flush the filesystems prior to snapshotting the disk.

More information: http://wiki.qemu.org/Features/QAPI/GuestAgent

Keep in mind that on Ubuntu AppArmor still needs to be disabled since the default
AppArmor profile doesn't allow libvirt to write into /var/lib/libvirt/qemu

This commit does not add any communication methods through API-calls, it merely
adds the channel to the Instances and installs the Guest Agent in the SSVMs.

With the addition of the Qemu Guest Agent channel a second channel appears in /dev
on a SSVM as a VirtIO port.

The order in which the ports are defined in the XML matters for the naming inside
the SSVM VM and by not relying on /dev/vportXX but looking for a static name the
SSVM still boots properly if the order in the XML definition is changed.

A SSVM with both ports attached will have something like this:

  root@v-215-VM:~# ls -l /dev/virtio-ports
  total 0
  lrwxrwxrwx 1 root root 11 May 13 21:41 org.qemu.guest_agent.0 -> ../vport0p2
  lrwxrwxrwx 1 root root 11 May 13 21:41 v-215-VM.vport -> ../vport0p1
  root@v-215-VM:~# ls -l /dev/vport*
  crw------- 1 root root 251, 1 May 13 21:41 /dev/vport0p1
  crw------- 1 root root 251, 2 May 13 21:41 /dev/vport0p2
  root@v-215-VM:~#

In this case the SSVM port points to /dev/vport0p1, but if the order in the XML
is different it might point to /dev/vport0p2

By looking for a portname with a pre-defined pattern in /dev/virtio-ports we
do not rely on the order in the XML definition.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2016-11-23 16:01:08 +01:00
Rohit Yadav 97f9ef52a4
Merge branch '4.9' 2016-11-23 14:31:09 +05:30
Rohit Yadav 1e88ad45a7
Merge pull request #1680 from murali-reddy/vr_csfile_search
CLOUDSTACK-9498: VR CsFile search utility methods fail when search stThere is no real use of python 're' module  in CsFile.py utility methods searchString, deleteLine. Regular string search is sufficient. These methods are used only for VPN user add/delete. Since VPN user password can have python 're' module meta characters, it interfere with search functionality.

Replacing re.search() with regular string search instead.

Change is confined to VPN add/delete users. Have run the test/integration/component/test_vpn_users.py

VPN remote access user limit tests ... === TestName: test_01_VPN_user_limit | Status : SUCCESS ===
ok
Test create VPN when L2TP port in use ... === TestName: test_02_use_vpn_port | Status : SUCCESS ===
ok
Test create NAT rule when VPN when L2TP enabled ... === TestName: test_03_enable_vpn_use_port | Status : SUCCESS ===
ok
Test add new users to existing VPN ... === TestName: test_04_add_new_users | Status : SUCCESS ===
ok
Test add duplicate user to existing VPN ... === TestName: test_05_add_duplicate_user | Status : SUCCESS ===
ok
Test as global admin, add a new VPN user to an existing VPN entry ... === TestName: test_06_add_VPN_user_global_admin | Status : SUCCESS ===
ok
Test as domain admin, add a new VPN user to an existing VPN entry ... === TestName: test_07_add_VPN_user_domain_admin | Status : SUCCESS ===
ok

* pr/1680:
  CLOUDSTACK-9498: VR CsFile search utility methods fail when search string has 're' meta chars, and causing VPN user add/deelte to fail

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-23 14:31:00 +05:30
Rohit Yadav 15fd571f2b
Merge branch '4.9' 2016-11-23 13:39:02 +05:30
Rohit Yadav 1137a79ccc
Merge branch '4.8' into 4.9 2016-11-23 13:38:11 +05:30
Murali Reddy 7ab35e6616 CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network
when guest VM has nic's in more than one guest network set the tag for
each host in /etc/dhcphosts.txt, and use the tag to add exception in
/etc/dhcpopts.txt to prevent sending default route, dns server in case if the nic is in non-default network
this was the behaviour with edithosts.sh prior to 4.6

added new test case test_router_dhcp_opts to test DHCP option file use of cloudstack
2016-11-22 16:30:42 +05:30
Murali Reddy 4c4696e5e4 CLOUDSTACK-9583: VR: In CsDhcp.py preseed both hostaname and localhost to resolve to 127.0.0.1
The VR executes a ip route flush command as part of configurations. This command performs a
DNS lookup on the VR hostname. Since the VR does not have a DNS entry, the ip command would
wait 5 seconds before timing out and executing the flush operation. This fix adds the VR
hostname to /etc/hosts mapped to 127.0.0.1 to answer the DNS lookup – reducing the
execution time.
2016-11-10 13:25:22 +05:30
Rohit Yadav 96a2602148
Merge branch '4.9' 2016-11-09 12:22:53 +05:30
Rohit Yadav 8b786d1fb2
Merge pull request #1743 from wido/CLOUDSTACK-8326
CLOUDSTACK-8326: Always fill UDP checksums in DHCP replies in VRIn some cases the UDP checksums in packets from DHCP servers are
incorrect. This is a problem for some DHCP clients that ignore
packets with bad checksums. This patch inserts an iptables rule
to ensure DHCP servers always send packets with correct checksums.

Due to this bug DHCP offers are sometimes not accepted by Instances.

The end-result without this fix is no connectivity for the Instance
due to the lack of a IPv4 address.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* pr/1743:
  CLOUDSTACK-8326: Always fill UDP checksums in DHCP replies in VR

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-09 12:18:03 +05:30
Wido den Hollander fa56d0b3e6
CLOUDSTACK-8326: Always fill UDP checksums in DHCP replies in VR
In some cases the UDP checksums in packets from DHCP servers are
incorrect. This is a problem for some DHCP clients that ignore
packets with bad checksums. This patch inserts an iptables rule
to ensure DHCP servers always send packets with correct checksums.

Due to this bug DHCP offers are sometimes not accepted by Instances.

The end-result without this fix is no connectivity for the Instance
due to the lack of a IPv4 address.

This is also commited in OpenStack:
- https://github.com/projectcalico/felix/issues/40
- https://review.openstack.org/148718
- https://bugzilla.redhat.com/show_bug.cgi?id=910619

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2016-11-08 18:41:37 +01:00
Rohit Yadav 495f7bb1f5
Merge branch '4.9' 2016-11-08 15:21:00 +05:30
Rohit Yadav c09fc27255
Merge branch '4.8' into 4.9 2016-11-08 15:20:45 +05:30
Rohit Yadav b704cefc1f
Merge branch '4.7' into 4.8 2016-11-08 15:20:07 +05:30
Rohit Yadav 0279ac20e4 Merge pull request #1744 from greenqloud/4.7
CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory

* pr/1744:
  CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-08 15:19:28 +05:30
Ólafur Stefán Arnarsson 420d06f4d3 CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory 2016-10-31 21:44:15 +00:00
Murali Reddy 9cc06a8fc8 CLOUDSTACK-9498: VR CsFile search utility methods fail when search string has
're' meta chars, and causing VPN user add/deelte to fail

    -there is no real use of python 're' in CsFile.py utility methods searchString, deleteLine
    Replacing with regular string search instead.

    -modifying the smoke test for VPN user add/delete to have all permissable chars
2016-10-28 17:45:15 +05:30
Rohit Yadav ccd1734dc4 Merge branch '4.9' 2016-10-21 10:15:29 +05:30
Rohit Yadav 9b9b49e10b Merge branch '4.8' into 4.9 2016-10-21 10:14:16 +05:30
Murali Reddy 93ac134f1e CLOUDSTACK-9533: gateway of public IP is not handled correctly when parsing the
cmd_line.json to create ips.json databag

using 'gateway' in cmd_line for the public ip gateway while creating ips.json databag
2016-10-06 16:41:29 +05:30
Murali Reddy 052a9af023 CLOUDSTACK-9515: internal LB vm is not handled when parsing cmd_line.json,
resulting in internal LB vm not come up

parsing cmd_line to create 'ips' data bag, never handled internal lb vm, but still
worked due to another bug. support for internal lb vm is added with this fix
2016-10-05 15:11:00 +05:30
Rajani Karuturi 43f82c991c Merge release branch 4.9 to master
* 4.9:
  fix egress rule incorrect behavior
2016-09-21 15:00:30 +05:30
Rajani Karuturi cc043e9f8f Merge pull request #1666 from murali-reddy/egress_rules
CLOUDSTACK-9480,  CLOUDSTACK-9495 fix egress rule incorrect behaviorWhen 'default egress policy' is set to 'allow' in the network offering, any egress rule that is added will 'deny' the traffic overriding the default behaviour.

Conversely, when 'default egress policy' is set to 'deny' in the network offering, any egress rule that is added will 'allow' the traffic overriding the default behaviour.

While this works for 'tcp', 'udp' as expected, for 'icmp' protocol its always set to ALLOW. This patch keeps all protocols behaviour consistent.

Results of running test/integration/component/test_egress_fw_rules.py.  With out the patch test_02_egress_fr2 test was failing. This patch fixes the test_02_egress_fr2  scenario.
-----------------------------------------------------------------------------------------------------
Test By-default the communication from guest n/w to public n/w is NOT allowed. ... === TestName: test_01_1_egress_fr1 | Status : SUCCESS ===
ok
Test By-default the communication from guest n/w to public n/w is allowed. ... === TestName: test_01_egress_fr1 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_1_egress_fr2 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_egress_fr2 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_1_egress_fr3 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_egress_fr3 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_1_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule without CIDR ... === TestName: test_06_1_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without CIDR ... === TestName: test_06_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without End Port ... === TestName: test_07_1_egress_fr7 | Status : EXCEPTION ===
ERROR
Test Create Egress rule without End Port ... === TestName: test_07_egress_fr7 | Status : SUCCESS ===
ok
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Delete Egress rule ... === TestName: test_09_1_egress_fr9 | Status : SUCCESS ===
ok
Test Delete Egress rule ... === TestName: test_09_egress_fr9 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_1_egress_fr10 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_egress_fr10 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_1_egress_fr11 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_egress_fr11 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_1_egress_fr12 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_egress_fr12 | Status : EXCEPTION ===
ERROR
Test Redundant Router : Master failover ... === TestName: test_13_1_egress_fr13 | Status : SUCCESS ===
ok
Test Redundant Router : Master failover ... === TestName: test_13_egress_fr13 | Status : SUCCESS ===
ok
-----------------------------------------------------------------------------------------------------

* pr/1666:
  fix egress rule incorrect behavior

Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
2016-09-21 14:54:31 +05:30
Murali Reddy a43abbe47b fix egress rule incorrect behavior
CLOUDSTACK-9480: Egress Firewall: Incorrect use of Allow/Deny for ICMP

     fix ensures, ICMP, TCP, UDP are handled similalry w.r.t egress rule action

CLOUDSTACK-9495: Egress rules functionalty broken when protocol=all specified

     when protocol=all specified, CIDR was ignored. Fix ensures if CIDR is specified
     its always used in configuring iptable rules

 2 new test cased to test /32 CIDR
2016-09-20 16:56:06 +05:30
Rohit Yadav 1d9735c346
Merge branch '4.9' 2016-08-30 22:44:33 +05:30
Rohit Yadav 14504dc7e3 CLOUDSTACK-6432: Prevent DNS reflection attacks
DNS on VR should not be publically accessible as it may be prone to DNS
amplification/reflection attacks. This fixes the issue by only allowing VR
DNS (port 53) to be accessible from guest network cidr, as per the fix in:
https://issues.apache.org/jira/browse/CLOUDSTACK-6432

- Only allows guest network cidrs to query VR DNS on port 53.
- Includes marvin smoke test that checks the VR DNS accessibility checks from
  guest and non-guest network.
- Fixes Marvin sshClient to avoid using ssh agent when password is provided,
  previous some environments may have seen 'No existing session' exception without
  this fix.
- Adds a new dnspython dependency that is used to perform dns resolutions in the
  tests.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 22:39:33 +05:30
John Burwell 8d11511b1f Adds support for four position versions and optional db upgrades
Often, patch and security releases do not require schema migrations or
data migrations.  However, if an empty upgrade class and associated
scripts are not defined, the upgrade process will break.  With this
change, if a release does not have an upgrade, a noop DbUpgrade is added
to the upgrade path.  This approach allows the upgrade to proceed and
for the database to properly reflect the installed version.  This change
should make the release process simpler as RMs no longer need to
rememeber to create this boilerplate code when starting a new release.

Beginning with the 4.8.2.0 and 4.9.1.0 releases, the project will
formally adopt a four (4) position release number to properly accomodate
rekeases that contain only CVE fixes.  The DatabaseUpgradeChecker and
Version classes made assumptions that they would always parse and
compare three (3) position version numbers.  This change adds the
CloudStackVersion value object that supports both three (3) and four (4)
version numbers.   It encapsulates version comparsion logic, as well as,
the rules to allow three (3) and four (4) to interoperate.

  * Modifies DatabaseUpgradeChecker to handle derive an upgrade path for
  a version that was not explicitly specified.  It determines the
  releases the first release before it with database migrations and uses
  that list as the basis for the list for version being calculated.  A
  noop upgrade is then added to the list which causes no schema changes
  or data migrations, but will update the database to the version.
  * Adds unit tests for the upgrade path calculation logic in
  DatabaseUpgradeChecker
  * Removes dummy upgrade logic for the 4.8.2.0 introduced in previous
  versions of this patch
  * Introduces the CloudStackVersion value object which parses and
  compares three (3) and four (4) position version numbers.  This class
  is intended to replace com.cloud.maint.Version.
  * Adds the junit-dataprovider dependency -- allowing test data to be
  concisely generated separately from the execution of a test case.
  Used extensively in the CloudStackVersionTest.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-30 13:32:32 +05:30
Rohit Yadav 9555492b4d Merge branch '4.9' 2016-08-23 14:16:53 +05:30
Rohit Yadav f13c224da1 Updating pom.xml version numbers for release 4.9.1.0-SNAPSHOT
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-08-19 13:53:39 +05:30
Will Stevens 740bd45be6 Updating pom.xml version numbers for release 4.8.2-SNAPSHOT
Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-08-15 10:31:09 -04:00