From cc7e9eed7e1340729109983f79200557df22296b Mon Sep 17 00:00:00 2001 From: Sebastien Goasguen Date: Thu, 6 Jun 2013 09:29:57 -0400 Subject: [PATCH 01/15] [GSoC]: Added Ian Duffy's proposal to guide --- docs/en-US/CloudStack_GSoC_Guide.xml | 1 + docs/en-US/gsoc-imduffy15.xml | 395 +++++++++++++++++++++++++++ 2 files changed, 396 insertions(+) create mode 100644 docs/en-US/gsoc-imduffy15.xml diff --git a/docs/en-US/CloudStack_GSoC_Guide.xml b/docs/en-US/CloudStack_GSoC_Guide.xml index 91c2967fc45..b7ba61f8ee4 100644 --- a/docs/en-US/CloudStack_GSoC_Guide.xml +++ b/docs/en-US/CloudStack_GSoC_Guide.xml @@ -47,6 +47,7 @@ + diff --git a/docs/en-US/gsoc-imduffy15.xml b/docs/en-US/gsoc-imduffy15.xml new file mode 100644 index 00000000000..652152fcc4b --- /dev/null +++ b/docs/en-US/gsoc-imduffy15.xml @@ -0,0 +1,395 @@ + + +%BOOK_ENTITIES; +]> + + + + + Ians's 2013 GSoC Proposal + This chapter describes Ians 2013 Google Summer of Code project within the &PRODUCT; ASF project. It is a copy paste of the submitted proposal. +
+ LDAP user provisioning + + "Need to automate the way the LDAP users are provisioned into cloud stack. This will mean better + integration with a LDAP server, ability to import users and a way to define how the LDAP user + maps to the cloudstack users." + +
+
+ Abstract + + The aim of this project is to provide an more effective mechanism to provision users from LDAP + into cloudstack. Currently cloudstack enables LDAP authentication. In this authentication users + must be first setup in cloudstack. Once the user is setup in cloudstack they can authenticate + using their LDAP username and password. This project will improve Cloudstack LDAP integration + by enabling users be setup automatically using their LDAP credential + +
+
+ Deliverables + + + Service that retrieves a list of LDAP users from a configured group + + + Extension of the cloudstack UI "Add User" screen to offer user list from LDAP + + + Add service for saving new user it details from LDAP + + + BDD unit and acceptance automated testing + + + Document change details + + +
+
+ Quantifiable Results + + + + + Given + An administrator wants to add new user to cloudstack and LDAP is setup in cloudstack + + + When + The administrator opens the "Add User" screen + + + Then + A table of users appears for the current list of users (not already created on cloudstack) from the LDAP group displaying their usernames, given name and email address. The timezone dropdown will still be available beside each user + + + + + + + + + + Given + An administrator wants to add new user to cloudstack and LDAP is not setup in cloudstack + + + When + The administrator opens the "Add User" screen + + + Then + The current add user screen and functionality is provided + + + + + + + + + + Given + An administrator wants to add new user to cloudstack and LDAP is setup in cloudstack + + + When + The administrator opens the "Add User" screen and mandatory information is missing + + + Then + These fields will be editable to enable you to populate the name or email address + + + + + + + + + + Given + An administrator wants to add new user to cloudstack, LDAP is setup and the user being created is in the LDAP query group + + + When + The administrator opens the "Add User" screen + + + Then + There is a list of LDAP users displayed but the user is present in the list + + + + + + + + + + Given + An administrator wants to add a new user to cloudstack, LDAP is setup and the user is not in the query group + + + When + The administrator opens the "Add User" screen + + + Then + There is a list of LDAP users displayed but the user is not in the list + + + + + + + + + + Given + An administrator wants to add a group of new users to cloudstack + + + When + The administrator opens the "Add User" screen, selects the users and hits save + + + Then + The list of new users are saved to the database + + + + + + + + + + Given + An administrator has created a new LDAP user on cloudstack + + + When + The user authenticates against cloudstack with the right credentials + + + Then + They are authorised in cloudstack + + + + + + + + + + Given + A user wants to edit an LDAP user + + + When + They open the "Edit User" screen + + + Then + The password fields are disabled and cannot be changed + + + + + +
+
+ The Design Document + + + LDAP user list service + + + + name: ldapUserList + + + responseObject: LDAPUserResponse {username,email,name} + + + parameter: listType:enum {NEW, EXISTING,ALL} (Default to ALL if no option provided) + + + Create a new API service call for retreiving the list of users from LDAP. This will call a new + ConfigurationService which will retrieve the list of users using the configured search base and the query + filter. The list may be filtered in the ConfigurationService based on listType parameter + + + + LDAP Available Service + + + + name: ldapAvailable + + + responseObject LDAPAvailableResponse {available:boolean} + + + Create a new API service call veriying LDAP is setup correctly verifying the following configuration elements are all set: + + + ldap.hostname + + + ldap.port + + + ldap.usessl + + + ldap.queryfilter + + + ldap.searchbase + + + ldap.dn + + + ldap.password + + + + + + LDAP Save Users Service + + + + name: ldapSaveUsers + + + responseObject: LDAPSaveUsersRssponse {list]]>} + + + parameter: list of users + + + Saves the list of objects instead. Following the functionality in CreateUserCmd it will + + + Create the user via the account service + + + Handle the response + + + It will be decided whether a transation should remain over whole save or only over individual users. A list of UserResponse will be returned. + + + + Extension of cloudstack UI "Add User" screen + + + + Extend account.js enable the adding of a list of users with editable fields where required. The new "add user" screen for LDAP setup will: + + + Make an ajax call to the ldapAvailable, ldapuserList and ldapSaveUsers services + + + Validate on username, email, firstname and lastname + + + + + + Extension of cloudstack UI "Edit User" screen + + + + Extend account.js to disable the password fields on the edit user screen if LDAP available, specifically: + + + Make an ajax call to the ldapAvailable, ldapuserList and ldapSaveUsers services + + + Validate on username, email, firstname and lastname. Additional server validation will nsure the password has not changed + + + +
+
+ Approach + + To get started a development cloudstack environment will be created with DevCloud used to verify changes. Once the schedule is agreed with the mentor the deliverables will be broken into small user stories with expected delivery dates set. The development cycle will focus on BDD, enforcing all unit and acceptance tests are written first. + + + A build pipe line for continious delivery environment around cloudstack will be implemented, the following stages will be adopted: + + + + + + Stage + Action + + + + + Commit + Run unit tests + + + Sonar + Runs code quality metrics + + + Acceptance + Deploys the devcloud and runs all acceptance tests + + + Deployment + Deploy a new management server using Chef + + + + +
+
+ About me + + I am a Computer Science Student at Dublin City University in Ireland. I have interests in virtualization, +automation, information systems, networking and web development + + + I was involved with a project in a K-12(educational) environment of moving their server systems over +to a virtualized environment on ESXi. I have good knowledge of programming in Java, PHP and +Scripting langages. During the configuration of an automation system for OS deployment I experienced +some exposure to scripting in powershell, batch, vbs and bash and configuration of PXE images based +of WinPE and Debian. +Additionally I am also a mentor in an opensource teaching movement called CoderDojo, we teach kids +from the age of 8 everything from web page, HTML 5 game and raspberry pi development. It's really +cool. + + + I’m excited at the opportunity and learning experience that cloudstack are offering with this project. + +
+ From aa2fb311124599113ee73e00b583683976be615f Mon Sep 17 00:00:00 2001 From: Kishan Kavala Date: Wed, 5 Jun 2013 16:40:24 +0530 Subject: [PATCH 02/15] check for null protocol while validating ACL item --- .../network/vpc/NetworkACLServiceImpl.java | 40 ++++++++++--------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/server/src/com/cloud/network/vpc/NetworkACLServiceImpl.java b/server/src/com/cloud/network/vpc/NetworkACLServiceImpl.java index d6e86e2c811..4ad22d90770 100644 --- a/server/src/com/cloud/network/vpc/NetworkACLServiceImpl.java +++ b/server/src/com/cloud/network/vpc/NetworkACLServiceImpl.java @@ -303,28 +303,30 @@ public class NetworkACLServiceImpl extends ManagerBase implements NetworkACLServ } //Validate Protocol - //Check if protocol is a number - if(StringUtils.isNumeric(protocol)){ - int protoNumber = Integer.parseInt(protocol); - if(protoNumber < 0 || protoNumber > 255){ - throw new InvalidParameterValueException("Invalid protocol number: " + protoNumber); + if(protocol != null){ + //Check if protocol is a number + if(StringUtils.isNumeric(protocol)){ + int protoNumber = Integer.parseInt(protocol); + if(protoNumber < 0 || protoNumber > 255){ + throw new InvalidParameterValueException("Invalid protocol number: " + protoNumber); + } + } else { + //Protocol is not number + //Check for valid protocol strings + String supportedProtocols = "tcp,udp,icmp,all"; + if(!supportedProtocols.contains(protocol.toLowerCase())){ + throw new InvalidParameterValueException("Invalid protocol: " + protocol); + } } - } else { - //Protocol is not number - //Check for valid protocol strings - String supportedProtocols = "tcp,udp,icmp,all"; - if(!supportedProtocols.contains(protocol.toLowerCase())){ - throw new InvalidParameterValueException("Invalid protocol: " + protocol); + + // icmp code and icmp type can't be passed in for any other protocol rather than icmp + if (!protocol.equalsIgnoreCase(NetUtils.ICMP_PROTO) && (icmpCode != null || icmpType != null)) { + throw new InvalidParameterValueException("Can specify icmpCode and icmpType for ICMP protocol only"); } - } - // icmp code and icmp type can't be passed in for any other protocol rather than icmp - if (!protocol.equalsIgnoreCase(NetUtils.ICMP_PROTO) && (icmpCode != null || icmpType != null)) { - throw new InvalidParameterValueException("Can specify icmpCode and icmpType for ICMP protocol only"); - } - - if (protocol.equalsIgnoreCase(NetUtils.ICMP_PROTO) && (portStart != null || portEnd != null)) { - throw new InvalidParameterValueException("Can't specify start/end port when protocol is ICMP"); + if (protocol.equalsIgnoreCase(NetUtils.ICMP_PROTO) && (portStart != null || portEnd != null)) { + throw new InvalidParameterValueException("Can't specify start/end port when protocol is ICMP"); + } } //validate icmp code and type From 62e125c081cd46858fb835290e35e037922a941a Mon Sep 17 00:00:00 2001 From: Kishan Kavala Date: Thu, 6 Jun 2013 18:53:19 +0530 Subject: [PATCH 03/15] check for null ACL before creating ACL commands --- .../router/VpcVirtualNetworkApplianceManagerImpl.java | 6 +++--- server/src/com/cloud/network/vpc/NetworkACLManagerImpl.java | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/server/src/com/cloud/network/router/VpcVirtualNetworkApplianceManagerImpl.java b/server/src/com/cloud/network/router/VpcVirtualNetworkApplianceManagerImpl.java index 9992b7ca01e..711549903fa 100644 --- a/server/src/com/cloud/network/router/VpcVirtualNetworkApplianceManagerImpl.java +++ b/server/src/com/cloud/network/router/VpcVirtualNetworkApplianceManagerImpl.java @@ -949,9 +949,9 @@ public class VpcVirtualNetworkApplianceManagerImpl extends VirtualNetworkApplian if (router.getVpcId() != null) { if (_networkModel.isProviderSupportServiceInNetwork(guestNetworkId, Service.NetworkACL, Provider.VPCVirtualRouter)) { List networkACLs = _networkACLMgr.listNetworkACLItems(guestNetworkId); - s_logger.debug("Found " + networkACLs.size() + " network ACLs to apply as a part of VPC VR " + router - + " start for guest network id=" + guestNetworkId); - if (!networkACLs.isEmpty()) { + if ((networkACLs != null) && !networkACLs.isEmpty()) { + s_logger.debug("Found " + networkACLs.size() + " network ACLs to apply as a part of VPC VR " + router + + " start for guest network id=" + guestNetworkId); createNetworkACLsCommands(networkACLs, router, cmds, guestNetworkId, false); } } diff --git a/server/src/com/cloud/network/vpc/NetworkACLManagerImpl.java b/server/src/com/cloud/network/vpc/NetworkACLManagerImpl.java index 171b8b9a6bf..bf6b859f619 100644 --- a/server/src/com/cloud/network/vpc/NetworkACLManagerImpl.java +++ b/server/src/com/cloud/network/vpc/NetworkACLManagerImpl.java @@ -294,6 +294,9 @@ public class NetworkACLManagerImpl extends ManagerBase implements NetworkACLMana @Override public List listNetworkACLItems(long guestNtwkId) { Network network = _networkMgr.getNetwork(guestNtwkId); + if(network.getNetworkACLId() == null){ + return null; + } return _networkACLItemDao.listByACL(network.getNetworkACLId()); } From bdb5997cfc0b5e258e117a7c6c5de18d9dfce1f3 Mon Sep 17 00:00:00 2001 From: Jessica Wang Date: Thu, 6 Jun 2013 10:45:29 -0700 Subject: [PATCH 04/15] CLOUDSTACK-747: Internal LB detailView - Assigned VMs tab - Assign VMs action - fix a bug that select VM view failed to show VMs. --- ui/scripts/vpc.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ui/scripts/vpc.js b/ui/scripts/vpc.js index 580e004f66f..f0b449f1656 100644 --- a/ui/scripts/vpc.js +++ b/ui/scripts/vpc.js @@ -667,7 +667,7 @@ var items = []; if(instances != null) { for(var i = 0; i < instances.length; i++) { - if(instances[i]._isSelected = true) + if(instances[i]._isSelected == true) continue; else items.push(instances[i]); From 62025632f904b3f46f5d35b499235701be6f48cf Mon Sep 17 00:00:00 2001 From: Jessica Wang Date: Thu, 6 Jun 2013 10:55:12 -0700 Subject: [PATCH 05/15] CLOUDSTACK-747: Internal LB detailView - Assigned VMs tab - add source port, instance port to listView. --- ui/scripts/vpc.js | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/ui/scripts/vpc.js b/ui/scripts/vpc.js index f0b449f1656..082cc258b5f 100644 --- a/ui/scripts/vpc.js +++ b/ui/scripts/vpc.js @@ -359,6 +359,8 @@ fields: { name: { label: 'label.name' }, sourceipaddress: { label: 'Source IP Address' }, + sourceport: { label: 'Source Port' }, + instanceport: { label: 'Instance Port' }, algorithm: { label: 'label.algorithm' } }, dataProvider: function(args) { @@ -369,8 +371,15 @@ }, success: function(json) { var items = json.listloadbalancerssresponse.loadbalancer; - args.response.success({ data: items }); - + if(items != null) { + for(var i = 0; i < items.length; i++) { + var item = items[i]; + //there is only one element in loadbalancerrul array property. + item.sourceport = item.loadbalancerrule[0].sourceport; + item.instanceport = item.loadbalancerrule[0].instanceport; + } + } + args.response.success({ data: items }); } }); }, From 36ed93f8abd21e585e73d2070faca48b3cf52cfb Mon Sep 17 00:00:00 2001 From: Jessica Wang Date: Thu, 6 Jun 2013 11:39:05 -0700 Subject: [PATCH 06/15] CLOUDSTACK-2880: UI - zone detail - add VMware DC action - replace URL field with vcenter field and make vcenter required. --- ui/scripts/system.js | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/ui/scripts/system.js b/ui/scripts/system.js index c1eef690386..e3d7b4172fa 100644 --- a/ui/scripts/system.js +++ b/ui/scripts/system.js @@ -5388,9 +5388,9 @@ label: 'label.name', validation: { required: true } }, - url: { - label: 'label.url', - validation: { required: false } + vcenter: { + label: 'vcenter', + validation: { required: true } }, username: { label: 'label.username', @@ -5406,14 +5406,10 @@ action: function(args) { var data = { zoneid: args.context.physicalResources[0].id, - name: args.data.name - }; + name: args.data.name, + vcenter: args.data.vcenter + }; - if(args.data.url != null && args.data.url.length > 0) { - $.extend(data, { - url: args.data.url - }) - } if(args.data.username != null && args.data.username.length > 0) { $.extend(data, { username: args.data.username From 9a175f9306dacf998bd8417a238230d4f904b3c8 Mon Sep 17 00:00:00 2001 From: Jessica Wang Date: Thu, 6 Jun 2013 11:49:50 -0700 Subject: [PATCH 07/15] CLOUDSTACK-747: Internal LB detailView - Assigned VMs tab - Assign VMs action - listView will be freshed in widget level after action is done. Therefore, remove refresh from application level. --- ui/scripts/vpc.js | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/ui/scripts/vpc.js b/ui/scripts/vpc.js index 082cc258b5f..a087baa2630 100644 --- a/ui/scripts/vpc.js +++ b/ui/scripts/vpc.js @@ -710,11 +710,7 @@ var jid = data.assigntoloadbalancerruleresponse.jobid; args.response.success({ _custom: { - jobId: jid, - getUpdatedItem: function(json) { - $('.list-view').listView('refresh'); - //return json.queryasyncjobresultresponse.jobresult.volume; - } + jobId: jid } }); } From f4a1a2ff380478fcd23319f0fd2678c50b971e36 Mon Sep 17 00:00:00 2001 From: Alena Prokharchyk Date: Thu, 6 Jun 2013 11:52:15 -0700 Subject: [PATCH 08/15] Allow account to have multiple networks with customer defined cidrs as we already let it happen when the cidr is taken from the physical network config --- .../com/cloud/network/NetworkManagerImpl.java | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/server/src/com/cloud/network/NetworkManagerImpl.java b/server/src/com/cloud/network/NetworkManagerImpl.java index b92ef4b7dfa..cae4e8a87a7 100755 --- a/server/src/com/cloud/network/NetworkManagerImpl.java +++ b/server/src/com/cloud/network/NetworkManagerImpl.java @@ -1481,23 +1481,6 @@ public class NetworkManagerImpl extends ManagerBase implements NetworkManager, L s_logger.debug("Found existing network configuration for offering " + offering + ": " + configs.get(0)); } - if (errorIfAlreadySetup) { - InvalidParameterValueException ex = new InvalidParameterValueException("Found existing network configuration (with specified id) for offering (with specified id)"); - ex.addProxyObject(offering.getUuid(), "offeringId"); - ex.addProxyObject(configs.get(0).getUuid(), "networkConfigId"); - throw ex; - } else { - return configs; - } - } - } else if (predefined != null && predefined.getCidr() != null && predefined.getBroadcastUri() == null && vpcId == null) { - // don't allow to have 2 networks with the same cidr in the same zone for the account - List configs = _networksDao.listBy(owner.getId(), plan.getDataCenterId(), predefined.getCidr(), true); - if (configs.size() > 0) { - if (s_logger.isDebugEnabled()) { - s_logger.debug("Found existing network configuration for offering " + offering + ": " + configs.get(0)); - } - if (errorIfAlreadySetup) { InvalidParameterValueException ex = new InvalidParameterValueException("Found existing network configuration (with specified id) for offering (with specified id)"); ex.addProxyObject(offering.getUuid(), "offeringId"); From f61d61db9490eb28d2feedbef0d329479a66aede Mon Sep 17 00:00:00 2001 From: Wei Zhou Date: Thu, 6 Jun 2013 23:05:12 +0200 Subject: [PATCH 09/15] CLOUDSTACK-2875: allow port 8080 on virtual router so that vm can get password from virtual router --- patches/systemvm/debian/config/etc/iptables/iptables-router | 1 + 1 file changed, 1 insertion(+) diff --git a/patches/systemvm/debian/config/etc/iptables/iptables-router b/patches/systemvm/debian/config/etc/iptables/iptables-router index 3f5bc5f736b..b214e4025fe 100644 --- a/patches/systemvm/debian/config/etc/iptables/iptables-router +++ b/patches/systemvm/debian/config/etc/iptables/iptables-router @@ -37,6 +37,7 @@ COMMIT -A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i eth1 -p tcp -m state --state NEW --dport 3922 -j ACCEPT -A INPUT -i eth0 -p tcp -m state --state NEW --dport 80 -j ACCEPT +-A INPUT -i eth0 -p tcp -m state --state NEW --dport 8080 -j ACCEPT -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i eth0 -o eth0 -m state --state NEW -j ACCEPT From 554178582709e51ef934fa2684b281e83f3a3ed5 Mon Sep 17 00:00:00 2001 From: Jessica Wang Date: Thu, 6 Jun 2013 16:04:31 -0700 Subject: [PATCH 10/15] CLOUDSTACK-747: Internal LB detailView - Assigned VMs tab - Assign VMs action - Select VM view - refresh assignedVMs data by API call when popping up Select VM view. --- ui/scripts/vpc.js | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/ui/scripts/vpc.js b/ui/scripts/vpc.js index a087baa2630..399699d3948 100644 --- a/ui/scripts/vpc.js +++ b/ui/scripts/vpc.js @@ -651,7 +651,21 @@ listView: $.extend(true, {}, cloudStack.sections.instances.listView, { type: 'checkbox', filters: false, - dataProvider: function(args) { + dataProvider: function(args) { + var assignedInstances; + $.ajax({ + url: createURL('listLoadBalancers'), + data: { + id: args.context.internalLoadBalancers[0].id + }, + async: false, + success: function(json) { + assignedInstances = json.listloadbalancerssresponse.loadbalancer[0].loadbalancerinstance; + if(assignedInstances == null) + assignedInstances = []; + } + }); + $.ajax({ url: createURL('listVirtualMachines'), data: { @@ -663,11 +677,9 @@ // Pre-select existing instances in LB rule $(instances).map(function(index, instance) { - instance._isSelected = $.grep( - args.context.internalLoadBalancers[0].loadbalancerinstance, - - function(lbInstance) { - return lbInstance.id == instance.id; + instance._isSelected = $.grep(assignedInstances, + function(assignedInstance) { + return assignedInstance.id == instance.id; } ).length ? true : false; }); From 4a14ea8a4d0d8a729928b05f137ea6721acf02ea Mon Sep 17 00:00:00 2001 From: Sheng Yang Date: Thu, 6 Jun 2013 18:03:20 -0700 Subject: [PATCH 11/15] CLOUDSTACK-1771: Fix ipv6 address for router Now it won't change(as ipv4 address) after router is destroyed. --- .../ExternalFirewallDeviceManagerImpl.java | 2 +- ...ExternalLoadBalancerDeviceManagerImpl.java | 4 +- .../src/com/cloud/network/NetworkManager.java | 2 +- .../com/cloud/network/NetworkManagerImpl.java | 3 +- .../com/cloud/network/NetworkModelImpl.java | 17 +++++-- .../cloud/network/guru/DirectNetworkGuru.java | 5 +- .../guru/DirectPodBasedNetworkGuru.java | 2 +- .../VirtualNetworkApplianceManagerImpl.java | 46 ++++++++++--------- .../cloud/network/MockNetworkManagerImpl.java | 9 +--- .../com/cloud/vpc/MockNetworkManagerImpl.java | 2 +- 10 files changed, 48 insertions(+), 44 deletions(-) diff --git a/server/src/com/cloud/network/ExternalFirewallDeviceManagerImpl.java b/server/src/com/cloud/network/ExternalFirewallDeviceManagerImpl.java index 9d24e478fe2..9b190aa6b12 100644 --- a/server/src/com/cloud/network/ExternalFirewallDeviceManagerImpl.java +++ b/server/src/com/cloud/network/ExternalFirewallDeviceManagerImpl.java @@ -485,7 +485,7 @@ public abstract class ExternalFirewallDeviceManagerImpl extends AdapterBase impl if (add && (!reservedIpAddressesForGuestNetwork.contains(network.getGateway()))) { // Insert a new NIC for this guest network to reserve the gateway address - _networkMgr.savePlaceholderNic(network, network.getGateway(), null); + _networkMgr.savePlaceholderNic(network, network.getGateway(), null, null); } // Delete any mappings used for inline external load balancers in this network diff --git a/server/src/com/cloud/network/ExternalLoadBalancerDeviceManagerImpl.java b/server/src/com/cloud/network/ExternalLoadBalancerDeviceManagerImpl.java index 4b110e9596c..829ad3fdfe6 100644 --- a/server/src/com/cloud/network/ExternalLoadBalancerDeviceManagerImpl.java +++ b/server/src/com/cloud/network/ExternalLoadBalancerDeviceManagerImpl.java @@ -774,7 +774,7 @@ public abstract class ExternalLoadBalancerDeviceManagerImpl extends AdapterBase // If a NIC doesn't exist for the load balancing IP address, create one loadBalancingIpNic = _nicDao.findByIp4AddressAndNetworkId(loadBalancingIpAddress, network.getId()); if (loadBalancingIpNic == null) { - loadBalancingIpNic = _networkMgr.savePlaceholderNic(network, loadBalancingIpAddress, null); + loadBalancingIpNic = _networkMgr.savePlaceholderNic(network, loadBalancingIpAddress, null, null); } // Save a mapping between the source IP address and the load balancing IP address NIC @@ -1019,7 +1019,7 @@ public abstract class ExternalLoadBalancerDeviceManagerImpl extends AdapterBase if (add) { // Insert a new NIC for this guest network to reserve the self IP - _networkMgr.savePlaceholderNic(guestConfig, selfIp, null); + _networkMgr.savePlaceholderNic(guestConfig, selfIp, null, null); } else { // release the self-ip obtained from guest network Nic selfipNic = getPlaceholderNic(guestConfig); diff --git a/server/src/com/cloud/network/NetworkManager.java b/server/src/com/cloud/network/NetworkManager.java index 8627251e717..bc43daa9975 100755 --- a/server/src/com/cloud/network/NetworkManager.java +++ b/server/src/com/cloud/network/NetworkManager.java @@ -381,7 +381,7 @@ public interface NetworkManager { String allocatePublicIpForGuestNic(Long networkId, DataCenter dc, Pod pod, Account caller, String requestedIp) throws InsufficientAddressCapacityException; - NicVO savePlaceholderNic(Network network, String ip4Address, Type vmType); + NicVO savePlaceholderNic(Network network, String ip4Address, String ip6Address, Type vmType); DhcpServiceProvider getDhcpServiceProvider(Network network); diff --git a/server/src/com/cloud/network/NetworkManagerImpl.java b/server/src/com/cloud/network/NetworkManagerImpl.java index cae4e8a87a7..d6a64508ffe 100755 --- a/server/src/com/cloud/network/NetworkManagerImpl.java +++ b/server/src/com/cloud/network/NetworkManagerImpl.java @@ -4325,9 +4325,10 @@ public class NetworkManagerImpl extends ManagerBase implements NetworkManager, L @Override - public NicVO savePlaceholderNic(Network network, String ip4Address, Type vmType) { + public NicVO savePlaceholderNic(Network network, String ip4Address, String ip6Address, Type vmType) { NicVO nic = new NicVO(null, null, network.getId(), null); nic.setIp4Address(ip4Address); + nic.setIp6Address(ip6Address); nic.setReservationStrategy(ReservationStrategy.PlaceHolder); nic.setState(Nic.State.Reserved); nic.setVmType(vmType); diff --git a/server/src/com/cloud/network/NetworkModelImpl.java b/server/src/com/cloud/network/NetworkModelImpl.java index 6b63eadd4b2..21917f76351 100755 --- a/server/src/com/cloud/network/NetworkModelImpl.java +++ b/server/src/com/cloud/network/NetworkModelImpl.java @@ -2065,17 +2065,24 @@ public class NetworkModelImpl extends ManagerBase implements NetworkModel { public NicVO getPlaceholderNicForRouter(Network network, Long podId) { List nics = _nicDao.listPlaceholderNicsByNetworkIdAndVmType(network.getId(), VirtualMachine.Type.DomainRouter); for (NicVO nic : nics) { - if (nic.getReserver() == null && nic.getIp4Address() != null) { + if (nic.getReserver() == null && (nic.getIp4Address() != null || nic.getIp6Address() != null)) { if (podId == null) { return nic; } else { //return nic only when its ip address belong to the pod range (for the Basic zone case) List vlans = _vlanDao.listVlansForPod(podId); for (Vlan vlan : vlans) { - IpAddress ip = _ipAddressDao.findByIpAndNetworkId(network.getId(), nic.getIp4Address()); - if (ip != null && ip.getVlanId() == vlan.getId()) { - return nic; - } + if (nic.getIp4Address() != null) { + IpAddress ip = _ipAddressDao.findByIpAndNetworkId(network.getId(), nic.getIp4Address()); + if (ip != null && ip.getVlanId() == vlan.getId()) { + return nic; + } + } else { + UserIpv6AddressVO ipv6 = _ipv6Dao.findByNetworkIdAndIp(network.getId(), nic.getIp6Address()); + if (ipv6 != null && ipv6.getVlanId() == vlan.getId()) { + return nic; + } + } } } } diff --git a/server/src/com/cloud/network/guru/DirectNetworkGuru.java b/server/src/com/cloud/network/guru/DirectNetworkGuru.java index 84008c0f050..b4577ac096a 100755 --- a/server/src/com/cloud/network/guru/DirectNetworkGuru.java +++ b/server/src/com/cloud/network/guru/DirectNetworkGuru.java @@ -221,7 +221,6 @@ public class DirectNetworkGuru extends AdapterBase implements NetworkGuru { throws InsufficientVirtualNetworkCapcityException, InsufficientAddressCapacityException { - //FIXME - save ipv6 informaiton in the placeholder nic Transaction txn = Transaction.currentTxn(); txn.start(); _networkMgr.allocateDirectIp(nic, dc, vm, network, requestedIp4Addr, requestedIp6Addr); @@ -229,8 +228,8 @@ public class DirectNetworkGuru extends AdapterBase implements NetworkGuru { if (vm.getType() == VirtualMachine.Type.DomainRouter) { Nic placeholderNic = _networkModel.getPlaceholderNicForRouter(network, null); if (placeholderNic == null) { - s_logger.debug("Saving placeholder nic with ip4 address " + nic.getIp4Address() + " and ipv6 address " + requestedIp6Addr + " for the network " + network); - _networkMgr.savePlaceholderNic(network, nic.getIp4Address(), VirtualMachine.Type.DomainRouter); + s_logger.debug("Saving placeholder nic with ip4 address " + nic.getIp4Address() + " and ipv6 address " + nic.getIp6Address() + " for the network " + network); + _networkMgr.savePlaceholderNic(network, nic.getIp4Address(), nic.getIp6Address(), VirtualMachine.Type.DomainRouter); } } txn.commit(); diff --git a/server/src/com/cloud/network/guru/DirectPodBasedNetworkGuru.java b/server/src/com/cloud/network/guru/DirectPodBasedNetworkGuru.java index cf27986a69d..f21e352ff76 100755 --- a/server/src/com/cloud/network/guru/DirectPodBasedNetworkGuru.java +++ b/server/src/com/cloud/network/guru/DirectPodBasedNetworkGuru.java @@ -203,7 +203,7 @@ public class DirectPodBasedNetworkGuru extends DirectNetworkGuru { Nic placeholderNic = _networkModel.getPlaceholderNicForRouter(network, pod.getId()); if (placeholderNic == null) { s_logger.debug("Saving placeholder nic with ip4 address " + nic.getIp4Address() + " for the network " + network); - _networkMgr.savePlaceholderNic(network, nic.getIp4Address(), VirtualMachine.Type.DomainRouter); + _networkMgr.savePlaceholderNic(network, nic.getIp4Address(), null, VirtualMachine.Type.DomainRouter); } } txn.commit(); diff --git a/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java b/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java index e0ff1573143..db4786a7c7e 100755 --- a/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java +++ b/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java @@ -1740,30 +1740,34 @@ public class VirtualNetworkApplianceManagerImpl extends ManagerBase implements V s_logger.debug("Adding nic for Virtual Router in Guest network " + guestNetwork); String defaultNetworkStartIp = null, defaultNetworkStartIpv6 = null; if (!setupPublicNetwork) { + Nic placeholder = _networkModel.getPlaceholderNicForRouter(guestNetwork, plan.getPodId()); if (guestNetwork.getCidr() != null) { - Nic placeholder = _networkModel.getPlaceholderNicForRouter(guestNetwork, plan.getPodId()); - if (placeholder != null) { - s_logger.debug("Requesting ip address " + placeholder.getIp4Address() + " stored in placeholder nic for the network " + guestNetwork); - defaultNetworkStartIp = placeholder.getIp4Address(); - } else { - String startIp = _networkModel.getStartIpAddress(guestNetwork.getId()); - if (startIp != null && _ipAddressDao.findByIpAndSourceNetworkId(guestNetwork.getId(), startIp).getAllocatedTime() == null) { - defaultNetworkStartIp = startIp; - } else if (s_logger.isDebugEnabled()){ - s_logger.debug("First ip " + startIp + " in network id=" + guestNetwork.getId() + - " is already allocated, can't use it for domain router; will get random ip address from the range"); - } - } + if (placeholder != null && placeholder.getIp4Address() != null) { + s_logger.debug("Requesting ipv4 address " + placeholder.getIp4Address() + " stored in placeholder nic for the network " + guestNetwork); + defaultNetworkStartIp = placeholder.getIp4Address(); + } else { + String startIp = _networkModel.getStartIpAddress(guestNetwork.getId()); + if (startIp != null && _ipAddressDao.findByIpAndSourceNetworkId(guestNetwork.getId(), startIp).getAllocatedTime() == null) { + defaultNetworkStartIp = startIp; + } else if (s_logger.isDebugEnabled()){ + s_logger.debug("First ipv4 " + startIp + " in network id=" + guestNetwork.getId() + + " is already allocated, can't use it for domain router; will get random ip address from the range"); + } + } } - - //FIXME - get ipv6 stored in the placeholder + if (guestNetwork.getIp6Cidr() != null) { - String startIpv6 = _networkModel.getStartIpv6Address(guestNetwork.getId()); - if (startIpv6 != null && _ipv6Dao.findByNetworkIdAndIp(guestNetwork.getId(), startIpv6) == null) { - defaultNetworkStartIpv6 = startIpv6; - } else if (s_logger.isDebugEnabled()){ - s_logger.debug("First ipv6 " + startIpv6 + " in network id=" + guestNetwork.getId() + - " is already allocated, can't use it for domain router; will get random ipv6 address from the range"); + if (placeholder != null && placeholder.getIp6Address() != null) { + s_logger.debug("Requesting ipv6 address " + placeholder.getIp6Address() + " stored in placeholder nic for the network " + guestNetwork); + defaultNetworkStartIpv6 = placeholder.getIp6Address(); + } else { + String startIpv6 = _networkModel.getStartIpv6Address(guestNetwork.getId()); + if (startIpv6 != null && _ipv6Dao.findByNetworkIdAndIp(guestNetwork.getId(), startIpv6) == null) { + defaultNetworkStartIpv6 = startIpv6; + } else if (s_logger.isDebugEnabled()){ + s_logger.debug("First ipv6 " + startIpv6 + " in network id=" + guestNetwork.getId() + + " is already allocated, can't use it for domain router; will get random ipv6 address from the range"); + } } } } diff --git a/server/test/com/cloud/network/MockNetworkManagerImpl.java b/server/test/com/cloud/network/MockNetworkManagerImpl.java index 7ab322b3871..077395fca0e 100755 --- a/server/test/com/cloud/network/MockNetworkManagerImpl.java +++ b/server/test/com/cloud/network/MockNetworkManagerImpl.java @@ -66,15 +66,8 @@ import com.cloud.user.Account; import com.cloud.user.User; import com.cloud.utils.Pair; import com.cloud.utils.component.ManagerBase; -import com.cloud.vm.Nic; -import com.cloud.vm.NicProfile; -import com.cloud.vm.NicVO; -import com.cloud.vm.ReservationContext; -import com.cloud.vm.VMInstanceVO; -import com.cloud.vm.VirtualMachine; import com.cloud.vm.*; import com.cloud.vm.VirtualMachine.Type; -import com.cloud.vm.VirtualMachineProfile; import org.apache.cloudstack.api.command.admin.network.DedicateGuestVlanRangeCmd; import org.apache.cloudstack.api.command.admin.network.ListDedicatedGuestVlanRangesCmd; import org.apache.cloudstack.api.command.admin.usage.ListTrafficTypeImplementorsCmd; @@ -924,7 +917,7 @@ public class MockNetworkManagerImpl extends ManagerBase implements NetworkManage @Override - public NicVO savePlaceholderNic(Network network, String ip4Address, Type vmType) { + public NicVO savePlaceholderNic(Network network, String ip4Address, String ip6Address, Type vmType) { // TODO Auto-generated method stub return null; } diff --git a/server/test/com/cloud/vpc/MockNetworkManagerImpl.java b/server/test/com/cloud/vpc/MockNetworkManagerImpl.java index d46be7c9e22..b609022ff26 100644 --- a/server/test/com/cloud/vpc/MockNetworkManagerImpl.java +++ b/server/test/com/cloud/vpc/MockNetworkManagerImpl.java @@ -1393,7 +1393,7 @@ public class MockNetworkManagerImpl extends ManagerBase implements NetworkManage @Override - public NicVO savePlaceholderNic(Network network, String ip4Address, Type vmType) { + public NicVO savePlaceholderNic(Network network, String ip4Address, String ip6Address, Type vmType) { // TODO Auto-generated method stub return null; } From 6208a51283bfdb401f76f042296c16304ee8f4e4 Mon Sep 17 00:00:00 2001 From: Prasanna Santhanam Date: Fri, 7 Jun 2013 11:29:56 +0530 Subject: [PATCH 12/15] CLOUDSTACK-2889: Resize volume unsupported on vmware Skip the tests if the VM deploys on a VmWare host since we do not support resizing volumes on vmware (yet) Signed-off-by: Prasanna Santhanam --- test/integration/smoke/test_volumes.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/test/integration/smoke/test_volumes.py b/test/integration/smoke/test_volumes.py index d9c808a01d9..7039c6ffe66 100644 --- a/test/integration/smoke/test_volumes.py +++ b/test/integration/smoke/test_volumes.py @@ -591,6 +591,8 @@ class TestVolumes(cloudstackTestCase): if hosts[0].hypervisor == "XenServer": self.virtual_machine.stop(self.apiClient) + elif hosts[0].hypervisor.lower() == "vmware": + self.skipTest("Resize Volume is unsupported on VmWare") self.apiClient.resizeVolume(cmd) count = 0 @@ -638,6 +640,9 @@ class TestVolumes(cloudstackTestCase): if hosts[0].hypervisor == "XenServer": self.virtual_machine.stop(self.apiClient) + elif hosts[0].hypervisor.lower() == "vmware": + self.skipTest("Resize Volume is unsupported on VmWare") + self.debug("Resize Volume ID: %s" % self.volume.id) cmd = resizeVolume.resizeVolumeCmd() From e7d5ccaada697607260ab5ee8e8ff3b436973f88 Mon Sep 17 00:00:00 2001 From: Likitha Shetty Date: Fri, 7 Jun 2013 12:11:15 +0530 Subject: [PATCH 13/15] Fix DB warnings - txn: Commit called when it is not a transaction: -NetworkServiceImpl.dedicateGuestVlanRange:3050 --- server/src/com/cloud/network/NetworkServiceImpl.java | 1 + 1 file changed, 1 insertion(+) diff --git a/server/src/com/cloud/network/NetworkServiceImpl.java b/server/src/com/cloud/network/NetworkServiceImpl.java index 2bf9f402d34..29a36b9fe5f 100755 --- a/server/src/com/cloud/network/NetworkServiceImpl.java +++ b/server/src/com/cloud/network/NetworkServiceImpl.java @@ -3044,6 +3044,7 @@ public class NetworkServiceImpl extends ManagerBase implements NetworkService { _accountGuestVlanMapDao.update(guestVlanMapId, accountGuestVlanMapVO); } else { Transaction txn = Transaction.currentTxn(); + txn.start(); accountGuestVlanMapVO = new AccountGuestVlanMapVO(vlanOwner.getAccountId(), physicalNetworkId); accountGuestVlanMapVO.setGuestVlanRange(startVlan + "-" + endVlan); _accountGuestVlanMapDao.persist(accountGuestVlanMapVO); From c2ac4090d947dae21fdab31573b0a369de1fd2a2 Mon Sep 17 00:00:00 2001 From: Pranav Saxena Date: Fri, 7 Jun 2013 12:22:43 +0530 Subject: [PATCH 14/15] Changing URL to vcenter while adding a VmwareDC --- ui/scripts/zoneWizard.js | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/ui/scripts/zoneWizard.js b/ui/scripts/zoneWizard.js index 6631cbafe4c..5870a7bf418 100755 --- a/ui/scripts/zoneWizard.js +++ b/ui/scripts/zoneWizard.js @@ -3366,10 +3366,11 @@ zoneId: args.data.returnedZone.id, username: args.data.cluster.vCenterUsername, password: args.data.cluster.vCenterPassword, - name: args.data.cluster.vCenterDatacenter + name: args.data.cluster.vCenterDatacenter, + vcenter: args.data.cluster.vCenterHost }; $.ajax({ - url: createURL('addVmwareDc&url=' + todb(url)), + url: createURL('addVmwareDc'), data: vmwareData, success: function(json) { var item = json.addvmwaredcresponse.vmwaredc; From 1bdb6266c6b263684db229accd4f0a4a330f203a Mon Sep 17 00:00:00 2001 From: tuna Date: Thu, 6 Jun 2013 23:46:18 +0700 Subject: [PATCH 15/15] my gsoc proposal document Signed-off-by: Sebastien Goasguen --- docs/en-US/gsoc-tuna.xml | 203 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 203 insertions(+) diff --git a/docs/en-US/gsoc-tuna.xml b/docs/en-US/gsoc-tuna.xml index 68032a8d46d..0988734a465 100644 --- a/docs/en-US/gsoc-tuna.xml +++ b/docs/en-US/gsoc-tuna.xml @@ -25,4 +25,207 @@ Nguyen's 2013 GSoC Proposal This chapter describes Nguyen 2013 Google Summer of Code project within the &PRODUCT; ASF project. It is a copy paste of the submitted proposal. +
+ Add Xen/XCP support for GRE SDN controller + + "This project aims to enhance the current native SDN controller in supporting Xen/XCP and integrate successfully the open source SDN controller (FloodLight) driving Open vSwitch through its interfaces." + +
+
+ Abstract + + SDN, standing for Software-Defined Networking, is an approach to building data network equipments and softwares. It were invented by ONRC, Stanford University. SDN basically decouples the control from physical networking boxes and given to a software application called a controller. SDN has three parts: controller, protocols and switch; In which, OpenFlow is an open standard to deploy innovative protocols. Nowaday, more and more datacenters use SDN instead of traditional physical networking boxes. For example, Google announced that they completely built its own switches and SDN confrollers for use in its internal backbone network. + + + OpenvSwitch, an open source software switch, is widely used as a virtual switch in virtualized server environments. It can currently run on any Linux-based virtualization platform, such as: KVM, Xen (XenServer, XCP, Xen hypervisor), VirtualBox... It also has been ported to a number of different operating systems and hardware platforms: Linux, FreeBSD, Windows and even non-POSIX embedded systems. In cloud computing IaaS, using OpenvSwitch instead of Linux bridge on compute nodes becomes an inevitable trend because of its powerful features and the ability of OpenFlow integration as well. + + + In CloudStack, we already have a native SDN controller. With KVM hypervisor, developers can easily install OpenvSwitch module; whereas, Xen even has a build-in one. The combination of SDN controller and OpenvSwitch gives us many advanced things. For example, creating GRE tunnels as an isolation method instead of VLAN is a good try. In this project, we are planning to support GRE tunnels in Xen/XCP hypervisor with the native SDN controller. When it's done, substituting open-sources SDN controllers (floodlight, beacon, pox, nox) for the current one is an amazing next step. + +
+
+ Design description + + CloudStack currently has a native SDN Controller that is used to build meshes of GRE tunnels between Xen hosts. There consists of 4 parts: OVS tunnel manager, OVS Dao/VO, Command/Answer and Ovs tunnel plugin. The details are as follow: + + + OVS tunnel manager: Consist of OvsElement and OvsTunnelManager. + + + OvsElement is used for controlling Ovs tunnel lifecycle (prepare, release) + + + + prepare(network, nic, vm, dest): create tunnel for vm on network to dest + + + release(network, nic, vm): destroy tunnel for vm on network + + + + OvsTunnelManager drives bridge configuration and tunnel creation via calling respective commands to Agent. + + + + destroyTunnel(vm, network): call OvsDestroyTunnelCommand to destroy tunnel for vm on network + + + createTunnel(vm, network, dest): call OvsCreateTunnelCommand to create tunnel for vm on network to dest + + + + OVS tunnel plugin: These are ovstunnel and ovs-vif-flows.py script, writen as XAPI plugin. The OVS tunnel manager will call them via XML-RPC. + + + Ovstunnel plugin calls corresponding vsctl commands for setting up the OVS bridge, creating GRE tunnels or destroying them. + + + + setup_ovs_bridge() + + + destroy_ovs_bridge() + + + create_tunnel() + + + destroy_tunnel() + + + + Ovs-vif-flow.py clears or applies rule for VIFs every time it is plugged or unplugged from a OVS bridge. + + + + clear_flow() + + + apply_flow() + + + + OVS command/answer: It is designed under the format of requests and answers between Manager and Plugin. These commands will correspondence exactly the mentioned manipulations. + + + + OvsSetupBridgeCommand + + + OvsSetupBridgeAnswer + + + OvsDestroyBridgeCommand + + + OvsDestroyBridgeAnswer + + + OvsCreateTunnelCommand + + + OvsCreateTunnelAnswer + + + OvsDestroyTunnelCommand + + + OvsDestroyTunnelAnswer + + + OvsFetchInterfaceCommand + + + OvsFetchInterfaceAnswer + + + + OVS Dao/VO + + + + OvsTunnelInterfaceDao + + + OvsTunnelInterfaceVO + + + OvsTunnelNetworkDao + + + OvsTunnelNetworkVO + + +
+
+ Integrate FloodLight as SDN controller + + I think that we maybe deploy FloodLight Server as a new SystemVM. This VM acts like current SystemVMs. One Floodlight SystemVM per Zone, so it can manage for virtual switches under this zone. + +
+
+ Deliverables + + GRE has been used as isolation method in CloudStack when deploy with Xen/XCP hosts. + + + + User set sdn.ovs.controller parameter in Global Setting to true. He deploys Advance Networking and chooses GRE as isolation method + + + Make use of Floodlight instead of native SDN controller. + + +
+
+ About me + + My name is Nguyen Anh Tu, a young and enthusiastic researcher in Cloud Computing Center - Viettel Research and Development Institute, Vietnam. Since last year, we has built Cloud Platform based on CloudStack, starting with version 3.0.2. As the results, some advanced modules were successfully developed, consists of: + + + + Encrypt Data Volume for VMs. + + + Dynamic Allocate Memory for VMs by changing policy on Squeeze Daemon. + + + AutoScale without using NetScale. + + + Deploy a new SystemVM type for Intrustion Detection System. + + + + Given the working experience and recent researches, I have obtained remarkably the understanding of specific knowledges to carry on this project, details as follow: + + + + Java source code on CloudStack: Design Pattern, Spring framework. + + + Bash, Python programming. + + + XAPI plugin. + + + XML-RPC. + + + OpenVSwitch on Xen. + + + + Other knowledges: + + + + XAPI RRD, XenStore. + + + Ocaml Programming (XAPI functions). + + +