%BOOK_ENTITIES; ]> Welcome to the 4.2.0 release of &PRODUCT;, the second major release from the Apache CloudStack project since its graduation from the Apache Incubator. &PRODUCT; 4.2 includes more than 50 new features and enhancements. The focus of the release is on three major areas: Improved support for both legacy-style and cloud-style workloads New third-party plug-in architecture Networking enhancements In addition to these major new areas of functionality, &PRODUCT; 4.2 provides many additional enhancements in a variety of product areas. All of the new features are summarized later in this Release Note. This document contains information specific to this release of &PRODUCT;, including upgrade instructions from prior releases, new features added to &PRODUCT;, API changes, and issues fixed in the release. For installation instructions, please see the Installation Guide. For usage and administration instructions, please see the &PRODUCT; Administrator's Guide. Developers and users who wish to work with the API will find instruction in the &PRODUCT; API Developer's Guide If you find any errors or problems in this guide, please see . We hope you enjoy working with &PRODUCT;!

Apache CloudStack 4.2.0 includes many new features. This section covers the most prominent new features and changes.

CLOUDSTACK-3236:Portable IPs in &PRODUCT; are nothing but elastic IPs that can be transferred across geographically separated zones. As an administrator, you can provision a pool of portable IPs at region level and are available for user consumption. The users can acquire portable IPs if admin has provisioned portable public IPs at the region level they are part of. These IPs can be used for any service within an advanced zone. You can also use portable IPs for EIP service in Basic zones. Additionally, a portable IP can be transferred from one network to another network.

CLOUDSTACK-770:In &PRODUCT; 3.0.6, a functionality was added to allow users to create a multi-tier application connected to a single instance of a Virtual Router that supports inter-VLAN routing. Such a multi-tier application is called a virtual private cloud (VPC). Users were also able to connect their multi-tier applications to a private Gateway or a Site-to-Site VPN tunnel and route certain traffic to those gateways. For &PRODUCT; 4.2, additional features are implemented to enhance VPC applications. Internal Load Balancing between VPC tiers Source NAT and ACL support on private gateways Multiple private gateway support Support for ACL deny rules ACL support on all layer 4 protocols Support up to 8 VPN Gateways Support for blacklisting routes NetScaler support for VPC load balancing Support for KVM hypervisor Support for the ability to simultaneously deploy an instance on a VPC Tier and one or more Shared Networks

CLOUDSTACK-742:&PRODUCT; supports Cisco Virtual Network Management Center (VNMC) on Cisco Nexus 1000v dvSwich-enabled VMware hypervisors. &PRODUCT; supports Cisco ASA 1000v as an external Firewall provider when integrated with Cisco VNMC. When Cisco VNMC is integrated with ASA 1000v Cloud Firewall and Cisco Nexus 1000v dvSwitch in &PRODUCT; you will be able to: Configure Cisco ASA 1000v Firewalls Create and apply security profiles that contain ACL policy sets for both ingress and egress traffic, connection timeout, NAT policy sets, and TCP intercept Consider the following use cases before using this feature: A Cloud administrator adds VNMC as a network element by using the admin API addCiscoVnmcResource after specifying the credentials A Cloud administrator adds ASA 1000v appliances by using the admin API addCiscoAsa1000vResource. You can configure one per guest network. A Cloud administrator creates an Isolated guest network offering by using ASA 1000v as the service provider for Firewall, Source NAT, Port Forwarding, and Static NAT.

CLOUDSTACK-772:&PRODUCT; 4.2 supports VMware vSphere Distributed Switch (VDS) for virtual network configuration in a VMware vSphere environment. Each vCenter server instance can support up to 128 VDSs and each VDS can manage up to 500 VMware hosts.

VMware VDS is an aggregation of host-level virtual switches on a VMware vCenter server. VDS abstracts the configuration of individual virtual switches that span across a large number of hosts, and enables centralized provisioning, administration, and monitoring for your entire datacenter from a centralized interface. VDS is controlled as a single distributed switch at the datacenter level. So there needed a component to ensure that the network configurations on the source and the destination virtual switch are consistent and will allow the VM to operate without breaking connectivity or network policies. Particularly during migration of VM across hosts, the sync up among peers need to be taken care. However in case of distributed vSwitch during VMotion, the vCenter server, would update the vSwitch modules on the hosts in cluster accordingly.

To make a &PRODUCT; deployment VDS enabled, set the vmware.use.dvswitch parameter to true by using the Global Settings page in the &PRODUCT; UI and restart the Management Server. Unless you enable the vmware.use.dvswitch parameter, you cannot see any UI options specific to VDS, and &PRODUCT; ignores the VDS-specific parameters specified in the AddCluster API call. Additionally, &PRODUCT; uses VDS for virtual network infrastructure if the value of vmware.use.dvswitch parameter is true and the value of vmware.use.nexus.dvswitch parameter is false. &PRODUCT; supports configuring virtual networks in a deployment with a mix of Virtual Distributed Switch, Standard Virtual Switch and Nexus 1000v Virtual Switch.

CLOUDSTACK-4243: This feature is supported only on NetScaler version 10.0 and beyond. The Nitro API is not compatible with NetScaler 9.3 and therefore this version is not supported for this feature. CLOUDSTACK-816:(NetScaler load balancer only) A load balancer rule distributes requests among a pool of services (a service in this context means an application running on a virtual machine). When creating a load balancer rule, you can specify a health check which will ensure that the rule forwards requests only to services that are healthy (running and available). This is in addition to specifying the stickiness policy, algorithm, and other load balancer rule options. You can configure one health check policy per load balancer rule. When a health check is in effect, the load balancer will stop forwarding requests to any resources that it has found to be unhealthy. If the resource later becomes available again, the periodic health check (periodicity is configurable) will discover it and the resource will once again be added to the pool of resources that can receive requests from the load balancer. You can delete or modify existing health check policies. To configure how often the health check is performed by default, use the global configuration setting healthcheck.update.interval. This default applies to all the health check policies in the cloud. You can override this value for an individual health check policy.

###

CLOUDSTACK-618: Limits the number of API requests per second that can be placed against a management server to avoid DoS attacks via API requests. The throttling is controlled by the api.throttling.enabled, api.throttling.interval, and api.throttling.max configuration settings. Note that api.throttling.enabled is set to false by default.

Supported on XenServer, VMware, and KVM. Windows 8 and Windows Server 2012 can now be used as OS types on guest virtual machines. The OS would be made available the same as any other, by uploading an ISO or a template. The instructions for uploading ISOs and templates are given in the Administrator's Guide. Limitation: When used with VMware hosts, this feature works only for the following versions: vSphere ESXi 5.1 and ESXi 5.0 Patch 4.

CLOUDSTACK-509: This enhancement backs NFS secondary storage with an S3-compatible object store. Periodically, a reaper thread synchronizes the templates, ISOs, and snapshots stored on a NFS secondary storage mount with a configured S3 object store. In addition to permitting the use of commodity or IaaS storage solutions for static assets, it provides a means of automatically synchronizing template and ISO assets across multiple zones. See the &PRODUCT; wiki for more information on this feature, currently the documentation is incomplete.

CLOUDSTACK-437: This feature adds the ability for domain admins and users to create their own API Key and Secret. Domain admins can create keys for themselves, subdomain admins, and for regular users, but not for other domain admins.

CLOUDSTACK-306: For &PRODUCT; deployments using the Juniper SRX (firewall) and F5 Big IP (load balancer), &PRODUCT; 4.2.0 supports putting the firewall in front of the load balancer, making the firewall device the gateway and putting the load balancer behind the public network.

CLOUDSTACK-299: This feature allows users to create egress (exit) traffic rules from private networks to public networks (e.g. from your internal network to the public Internet). By default all traffic is blocked from internal networks to the public networks, this allows you to open ports as necessary. Egress traffic rules are suppored only on virtual routers at this time, physical devices are not supported.

CLOUDSTACK-297: &PRODUCT; 4.2.0 introduces a new API resetSSHKeyForVirtualMachine, that can allow them to set or reset the SSH keypair assigned to a virtual machine.

Apache CloudStack uses Jira to track its issues. All new features and bugs for 4.2.0 have been tracked in Jira, and have a standard naming convention of "CLOUDSTACK-NNNN" where "NNNN" is the issue number. This section includes a summary of known issues against 4.0.0 that were fixed in 4.2.0. Approximately 470 bugs were resolved or closed in the 4.2.0 cycle. Defect Description

Issue ID Description CLOUDSTACK-2709 Egress rules are are not supported on shared networks. CLOUDSTACK-1747 mvn deploydb only creates 4.0 DB, not 4.2 Due to tooling changes between 4.2 and 4.2, CloudStack's database is created using the 4.0 schema and updated to the 4.2 schema when the management server starts for the first time. It's OK to see the same schema if the management server has not started yet. CLOUDSTACK-1306 Better Error message when trying to deploy Vm by passing static Ipv4 addresses that are assigned to another VM/IP4 address is outside the iprange. CLOUDSTACK-1236 Warning while adding Xen 6.1 host [Unable to create local link network] CLOUDSTACK-969 api: zone response lists vlan in it as "vlan range of zone" but the vlan belongs to physical network CLOUDSTACK-963 [cloud.utils.AnnotationHelper] class java.lang.Stringdoes not have a Table annotation CLOUDSTACK-458 xen:snapshots:Storage gc fail to clean the failed snapshot images from secondarystorage CLOUDSTACK-315 Infrastructure view does not show capacity values CLOUDSTACK-300 Creation of compute offering allow combination of local storage + HA CLOUDSTACK-276 SSVM ID is exposed in the Error Message thrown by AddTrafficType API CLOUDSTACK-270 Ui should not ask for a vlan range if the physical network isolation type is not VLAN CLOUDSTACK-245 VPC ACLs are not stored and programmed consistently CLOUDSTACK-231 Tag creation using special charecters CLOUDSTACK-124 NetworkGarbageCollector not cleaning up networks CLOUDSTACK-62 console proxy does not support any keymaps besides us, jp

This section contains upgrade instructions from prior versions of CloudStack to Apache CloudStack 4.2.0. We include instructions on upgrading to Apache CloudStack from pre-Apache versions of Citrix CloudStack (last version prior to Apache is 3.0.2) and from the releases made while CloudStack was in the Apache Incubator. If you run into any issues during upgrades, please feel free to ask questions on users@cloudstack.apache.org or dev@cloudstack.apache.org.

This section will guide you from &PRODUCT; 4.0.x versions to &PRODUCT; 4.2.0. Any steps that are hypervisor-specific will be called out with a note. The package structure for &PRODUCT; has changed significantly since the 4.0.x releases. If you've compiled your own packages, you'll notice that the package names and the number of packages has changed. This is not a bug. However, this does mean that the procedure is not as simple as an apt-get upgrade or yum update, so please follow this section carefully. We recommend reading through this section once or twice before beginning your upgrade procedure, and working through it on a test system before working on a production system. Most users of &PRODUCT; manage the installation and upgrades of &PRODUCT; with one of Linux's predominant package systems, RPM or APT. This guide assumes you'll be using RPM and Yum (for Red Hat Enterprise Linux or CentOS), or APT and Debian packages (for Ubuntu). Create RPM or Debian packages (as appropriate) and a repository from the 4.2.0 source, or check the Apache CloudStack downloads page at http://cloudstack.apache.org/downloads.html for package repositories supplied by community members. You will need them for step or step . Instructions for creating packages from the &PRODUCT; source are in the Installation Guide. Stop your management server or servers. Run this on all management server hosts: # service cloud-management stop If you are running a usage server or usage servers, stop those as well: # service cloud-usage stop Make a backup of your MySQL database. If you run into any issues or need to roll back the upgrade, this will assist in debugging or restoring your existing environment. You'll be prompted for your password. # mysqldump -u root -p cloud > cloudstack-backup.sql If you have made changes to /etc/cloud/management/components.xml, you'll need to carry these over manually to the new file, /etc/cloudstack/management/componentContext.xml. This is not done automatically. (If you're unsure, we recommend making a backup of the original components.xml to be on the safe side. After upgrading to 4.2, API clients are expected to send plain text passwords for login and user creation, instead of MD5 hash. If API client changes are not acceptable, following changes are to be made for backward compatibility: Modify componentsContext.xml, and make PlainTextUserAuthenticator as the default authenticator (1st entry in the userAuthenticators adapter list is default) <!-- Security adapters --> <bean id="userAuthenticators" class="com.cloud.utils.component.AdapterList"> <property name="Adapters"> <list> <ref bean="PlainTextUserAuthenticator"/> <ref bean="MD5UserAuthenticator"/> <ref bean="LDAPUserAuthenticator"/> </list> </property> </bean> PlainTextUserAuthenticator works the same way MD5UserAuthenticator worked prior to 4.2. If you are using Ubuntu, follow this procedure to upgrade your packages. If not, skip to step . This section assumes you're using the community supplied packages for &PRODUCT;. If you've created your own packages and APT repository, substitute your own URL for the ones used in these examples. The first order of business will be to change the sources list for each system with &PRODUCT; packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.) Start by opening /etc/apt/sources.list.d/cloudstack.list on any systems that have &PRODUCT; packages installed. This file should have one line, which contains: deb http://cloudstack.apt-get.eu/ubuntu precise 4.0 We'll change it to point to the new package repository: deb http://cloudstack.apt-get.eu/ubuntu precise 4.2 If you're using your own package repository, change this line to read as appropriate for your 4.2.0 repository. Now update your apt package list: $ sudo apt-get update Now that you have the repository configured, it's time to install the cloudstack-management package. This will pull in any other dependencies you need. $ sudo apt-get install cloudstack-management You will need to manually install the cloudstack-agent package: $ sudo apt-get install cloudstack-agent During the installation of cloudstack-agent, APT will copy your agent.properties, log4j-cloud.xml, and environment.properties from /etc/cloud/agent to /etc/cloudstack/agent. When prompted whether you wish to keep your configuration, say Yes. Verify that the file /etc/cloudstack/agent/environment.properties has a line that reads: paths.script=/usr/share/cloudstack-common If not, add the line. Restart the agent: service cloud-agent stop killall jsvc service cloudstack-agent start During the upgrade, log4j-cloud.xml was simply copied over, so the logs will continue to be added to /var/log/cloud/agent/agent.log. There's nothing wrong with this, but if you prefer to be consistent, you can change this by copying over the sample configuration file: cd /etc/cloudstack/agent mv log4j-cloud.xml.dpkg-dist log4j-cloud.xml service cloudstack-agent restart Once the agent is running, you can uninstall the old cloud-* packages from your system: sudo dpkg --purge cloud-agent If you are using CentOS or RHEL, follow this procedure to upgrade your packages. If not, skip to step . This section assumes you're using the community supplied packages for &PRODUCT;. If you've created your own packages and yum repository, substitute your own URL for the ones used in these examples. The first order of business will be to change the yum repository for each system with &PRODUCT; packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.) Start by opening /etc/yum.repos.d/cloudstack.repo on any systems that have &PRODUCT; packages installed. This file should have content similar to the following: [apache-cloudstack] name=Apache CloudStack baseurl=http://cloudstack.apt-get.eu/rhel/4.0/ enabled=1 gpgcheck=0 If you are using the community provided package repository, change the base url to http://cloudstack.apt-get.eu/rhel/4.2/ If you're using your own package repository, change this line to read as appropriate for your 4.2.0 repository. Now that you have the repository configured, it's time to install the cloudstack-management package by upgrading the older cloud-client package. $ sudo yum upgrade cloud-client For KVM hosts, you will need to upgrade the cloud-agent package, similarly installing the new version as cloudstack-agent. $ sudo yum upgrade cloud-agent During the installation of cloudstack-agent, the RPM will copy your agent.properties, log4j-cloud.xml, and environment.properties from /etc/cloud/agent to /etc/cloudstack/agent. For CentOS 5.5, perform the following: Run the following command: rpm -Uvh http://download.cloud.com/support/jsvc/jakarta-commons-daemon-jsvc-1.0.1-8.9.el6.x86_64.rpm Upgrade the Usage server. sudo yum upgrade cloud-usage Verify that the file /etc/cloudstack/agent/environment.properties has a line that reads: paths.script=/usr/share/cloudstack-common If not, add the line. Restart the agent: service cloud-agent stop killall jsvc service cloudstack-agent start Once you've upgraded the packages on your management servers, you'll need to restart the system VMs. Make sure port 8096 is open in your local host firewall to do this. There is a script that will do this for you, all you need to do is run the script and supply the IP address for your MySQL instance and your MySQL credentials: # nohup cloudstack-sysvmadm -d IP address -u cloud -p -a > sysvm.log 2>&1 & You can monitor the log for progress. The process of restarting the system VMs can take an hour or more. # tail -f sysvm.log The output to sysvm.log will look something like this: Stopping and starting 1 secondary storage vm(s)... Done stopping and starting secondary storage vm(s) Stopping and starting 1 console proxy vm(s)... Done stopping and starting console proxy vm(s). Stopping and starting 4 running routing vm(s)... Done restarting router(s). This step is only for CloudStack installs that are using Xen hosts. Copy the file vhd-utils to /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver.

This section will guide you from Citrix CloudStack 3.0.2 to Apache CloudStack 4.2.0. Sections that are hypervisor-specific will be called out with a note. The following upgrade instructions apply only if you're using VMware hosts. If you're not using VMware hosts, skip this step and move on to . In each zone that includes VMware hosts, you need to add a new system VM template. While running the existing 3.0.2 system, log in to the UI as root administrator. In the left navigation bar, click Templates. In Select view, click Templates. Click Register template. The Register template dialog box is displayed. In the Register template dialog box, specify the following values (do not change these): Field Value Name systemvm-vmware-4.2 Description systemvm-vmware-4.2 URL http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova Zone Choose the zone where this hypervisor is used Hypervisor VMware Format OVA OS Type Debian GNU/Linux 5.0 (32-bit) Extractable no Password Enabled no Public no Featured no Watch the screen to be sure that the template downloads successfully and enters the READY state. Do not proceed until this is successful. Stop all Usage Servers if running. Run this on all Usage Server hosts. # service cloud-usage stop Stop the Management Servers. Run this on all Management Server hosts. # service cloud-management stop On the MySQL master, take a backup of the MySQL databases. We recommend performing this step even in test upgrades. If there is an issue, this will assist with debugging. In the following commands, it is assumed that you have set the root password on the database, which is a CloudStack recommended best practice. Substitute your own MySQL root password. # mysqldump -u root -pmysql_password cloud > cloud-backup.dmp # mysqldump -u root -pmysql_password cloud_usage > cloud-usage-backup.dmp Either build RPM/DEB packages as detailed in the Installation Guide, or use one of the community provided yum/apt repositories to gain access to the &PRODUCT; binaries. If you are using Ubuntu, follow this procedure to upgrade your packages. If not, skip to step . This section assumes you're using the community supplied packages for &PRODUCT;. If you've created your own packages and APT repository, substitute your own URL for the ones used in these examples. The first order of business will be to change the sources list for each system with &PRODUCT; packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.) Start by opening /etc/apt/sources.list.d/cloudstack.list on any systems that have &PRODUCT; packages installed. This file should have one line, which contains: deb http://cloudstack.apt-get.eu/ubuntu precise 4.0 We'll change it to point to the new package repository: deb http://cloudstack.apt-get.eu/ubuntu precise 4.2 If you're using your own package repository, change this line to read as appropriate for your 4.2.0 repository. Now update your apt package list: $ sudo apt-get update Now that you have the repository configured, it's time to install the cloudstack-management package. This will pull in any other dependencies you need. $ sudo apt-get install cloudstack-management You will need to manually install the cloudstack-agent package: $ sudo apt-get install cloudstack-agent During the installation of cloudstack-agent, APT will copy your agent.properties, log4j-cloud.xml, and environment.properties from /etc/cloud/agent to /etc/cloudstack/agent. When prompted whether you wish to keep your configuration, say Yes. Verify that the file /etc/cloudstack/agent/environment.properties has a line that reads: paths.script=/usr/share/cloudstack-common If not, add the line. Restart the agent: service cloud-agent stop killall jsvc service cloudstack-agent start During the upgrade, log4j-cloud.xml was simply copied over, so the logs will continue to be added to /var/log/cloud/agent/agent.log. There's nothing wrong with this, but if you prefer to be consistent, you can change this by copying over the sample configuration file: cd /etc/cloudstack/agent mv log4j-cloud.xml.dpkg-dist log4j-cloud.xml service cloudstack-agent restart Once the agent is running, you can uninstall the old cloud-* packages from your system: sudo dpkg --purge cloud-agent If you are using CentOS or RHEL, follow this procedure to upgrade your packages. If not, skip to step . This section assumes you're using the community supplied packages for &PRODUCT;. If you've created your own packages and yum repository, substitute your own URL for the ones used in these examples. The first order of business will be to change the yum repository for each system with &PRODUCT; packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.) Start by opening /etc/yum.repos.d/cloudstack.repo on any systems that have &PRODUCT; packages installed. This file should have content similar to the following: [apache-cloudstack] name=Apache CloudStack baseurl=http://cloudstack.apt-get.eu/rhel/4.0/ enabled=1 gpgcheck=0 If you are using the community provided package repository, change the baseurl to http://cloudstack.apt-get.eu/rhel/4.2/ If you're using your own package repository, change this line to read as appropriate for your 4.2.0 repository. Now that you have the repository configured, it's time to install the cloudstack-management package by upgrading the older cloud-client package. $ sudo yum upgrade cloud-client For KVM hosts, you will need to upgrade the cloud-agent package, similarly installing the new version as cloudstack-agent. $ sudo yum upgrade cloud-agent During the installation of cloudstack-agent, the RPM will copy your agent.properties, log4j-cloud.xml, and environment.properties from /etc/cloud/agent to /etc/cloudstack/agent. Verify that the file /etc/cloudstack/agent/environment.properties has a line that reads: paths.script=/usr/share/cloudstack-common If not, add the line. Restart the agent: service cloud-agent stop killall jsvc service cloudstack-agent start If you have made changes to your copy of /etc/cloud/management/components.xml the changes will be preserved in the upgrade. However, you need to do the following steps to place these changes in a new version of the file which is compatible with version 4.2.0. Make a backup copy of /etc/cloud/management/components.xml. For example: # mv /etc/cloud/management/components.xml /etc/cloud/management/components.xml-backup Copy /etc/cloud/management/components.xml.rpmnew to create a new /etc/cloud/management/components.xml: # cp -ap /etc/cloud/management/components.xml.rpmnew /etc/cloud/management/components.xml Merge your changes from the backup file into the new components.xml. # vi /etc/cloudstack/management/components.xml If you have more than one management server node, repeat the upgrade steps on each node. After upgrading to 4.2, API clients are expected to send plain text passwords for login and user creation, instead of MD5 hash. Incase, api client changes are not acceptable, following changes are to be made for backward compatibility: Modify componentsContext.xml, and make PlainTextUserAuthenticator as the default authenticator (1st entry in the userAuthenticators adapter list is default) <!-- Security adapters --> <bean id="userAuthenticators" class="com.cloud.utils.component.AdapterList"> <property name="Adapters"> <list> <ref bean="PlainTextUserAuthenticator"/> <ref bean="MD5UserAuthenticator"/> <ref bean="LDAPUserAuthenticator"/> </list> </property> </bean> PlainTextUserAuthenticator works the same way MD5UserAuthenticator worked prior to 4.2. Start the first Management Server. Do not start any other Management Server nodes yet. # service cloudstack-management start Wait until the databases are upgraded. Ensure that the database upgrade is complete. After confirmation, start the other Management Servers one at a time by running the same command on each node. Failing to restart the Management Server indicates a problem in the upgrade. Having the Management Server restarted without any issues indicates that the upgrade is successfully completed. Start all Usage Servers (if they were running on your previous version). Perform this on each Usage Server host. # service cloudstack-usage start Additional steps are required for each KVM host. These steps will not affect running guests in the cloud. These steps are required only for clouds using KVM as hosts and only on the KVM hosts. Configure a yum or apt repository containing the &PRODUCT; packages as outlined in the Installation Guide. Stop the running agent. # service cloud-agent stop Update the agent software with one of the following command sets as appropriate for your environment. # yum update cloud-* # apt-get update # apt-get upgrade cloud-* Start the agent. # service cloudstack-agent start Edit /etc/cloudstack/agent/agent.properties to change the resource parameter from "com.cloud.agent.resource.computing.LibvirtComputingResource" to "com.cloud.hypervisor.kvm.resource.LibvirtComputingResource". Start the cloud agent and cloud management services. When the Management Server is up and running, log in to the CloudStack UI and restart the virtual router for proper functioning of all the features. Log in to the CloudStack UI as administrator, and check the status of the hosts. All hosts should come to Up state (except those that you know to be offline). You may need to wait 20 or 30 minutes, depending on the number of hosts. Troubleshooting: If login fails, clear your browser cache and reload the page. Do not proceed to the next step until the hosts show in Up state. If you are upgrading from 3.0.2, perform the following: Ensure that the admin port is set to 8096 by using the "integration.api.port" global parameter. This port is used by the cloud-sysvmadm script at the end of the upgrade procedure. For information about how to set this parameter, see "Setting Global Configuration Parameters" in the Installation Guide. Restart the Management Server. If you don't want the admin port to remain open, you can set it to null after the upgrade is done and restart the management server. Run the cloud-sysvmadm script to stop, then start, all Secondary Storage VMs, Console Proxy VMs, and virtual routers. Run the script once on each management server. Substitute your own IP address of the MySQL instance, the MySQL user to connect as, and the password to use for that user. In addition to those parameters, provide the -c and -r arguments. For example: # nohup cloud-sysvmadm -d 192.168.1.5 -u cloud -p password -c -r > sysvm.log 2>&1 & # tail -f sysvm.log This might take up to an hour or more to run, depending on the number of accounts in the system. If needed, upgrade all Citrix XenServer hypervisor hosts in your cloud to a version supported by CloudStack 4.2.0. The supported versions are XenServer 5.6 SP2 and 6.0.2. Instructions for upgrade can be found in the CloudStack 4.2.0 Installation Guide under "Upgrading XenServer Versions." Now apply the XenServer hotfix XS602E003 (and any other needed hotfixes) to XenServer v6.0.2 hypervisor hosts. Disconnect the XenServer cluster from CloudStack. In the left navigation bar of the CloudStack UI, select Infrastructure. Under Clusters, click View All. Select the XenServer cluster and click Actions - Unmanage. This may fail if there are hosts not in one of the states Up, Down, Disconnected, or Alert. You may need to fix that before unmanaging this cluster. Wait until the status of the cluster has reached Unmanaged. Use the CloudStack UI to check on the status. When the cluster is in the unmanaged state, there is no connection to the hosts in the cluster. To clean up the VLAN, log in to one XenServer host and run: /opt/xensource/bin/cloud-clean-vlan.sh Now prepare the upgrade by running the following on one XenServer host: /opt/xensource/bin/cloud-prepare-upgrade.sh If you see a message like "can't eject CD", log in to the VM and unmount the CD, then run this script again. Upload the hotfix to the XenServer hosts. Always start with the Xen pool master, then the slaves. Using your favorite file copy utility (e.g. WinSCP), copy the hotfixes to the host. Place them in a temporary folder such as /tmp. On the Xen pool master, upload the hotfix with this command: xe patch-upload file-name=XS602E003.xsupdate Make a note of the output from this command, which is a UUID for the hotfix file. You'll need it in another step later. (Optional) If you are applying other hotfixes as well, you can repeat the commands in this section with the appropriate hotfix number. For example, XS602E004.xsupdate. Manually live migrate all VMs on this host to another host. First, get a list of the VMs on this host: # xe vm-list Then use this command to migrate each VM. Replace the example host name and VM name with your own: # xe vm-migrate live=true host=host-name vm=VM-name If you see a message like "You attempted an operation on a VM which requires PV drivers to be installed but the drivers were not detected," run: /opt/xensource/bin/make_migratable.sh b6cf79c8-02ee-050b-922f-49583d9f1a14. Apply the hotfix. First, get the UUID of this host: # xe host-list Then use the following command to apply the hotfix. Replace the example host UUID with the current host ID, and replace the hotfix UUID with the output from the patch-upload command you ran on this machine earlier. You can also get the hotfix UUID by running xe patch-list. xe patch-apply host-uuid=host-uuid uuid=hotfix-uuid Copy the following files from the CloudStack Management Server to the host. Copy from here... ...to here /usr/lib64/cloud/common/scripts/vm/hypervisor/xenserver/xenserver60/NFSSR.py /opt/xensource/sm/NFSSR.py /usr/lib64/cloud/common/scripts/vm/hypervisor/xenserver/setupxenserver.sh /opt/xensource/bin/setupxenserver.sh /usr/lib64/cloud/common/scripts/vm/hypervisor/xenserver/make_migratable.sh /opt/xensource/bin/make_migratable.sh (Only for hotfixes XS602E005 and XS602E007) You need to apply a new Cloud Support Pack. Download the CSP software onto the XenServer host from one of the following links: For hotfix XS602E005: http://coltrane.eng.hq.xensource.com/release/XenServer-6.x/XS-6.0.2/hotfixes/XS602E005/56710/xe-phase-2/xenserver-cloud-supp.tgz For hotfix XS602E007: http://coltrane.eng.hq.xensource.com/release/XenServer-6.x/XS-6.0.2/hotfixes/XS602E007/57824/xe-phase-2/xenserver-cloud-supp.tgz Extract the file: # tar xf xenserver-cloud-supp.tgz Run the following script: # xe-install-supplemental-pack xenserver-cloud-supp.iso If the XenServer host is part of a zone that uses basic networking, disable Open vSwitch (OVS): # xe-switch-network-backend bridge Reboot this XenServer host. Run the following: /opt/xensource/bin/setupxenserver.sh If the message "mv: cannot stat `/etc/cron.daily/logrotate': No such file or directory" appears, you can safely ignore it. Run the following: for pbd in `xe pbd-list currently-attached=false| grep ^uuid | awk '{print $NF}'`; do xe pbd-plug uuid=$pbd ; On each slave host in the Xen pool, repeat these steps, starting from "manually live migrate VMs." If passwords which you know to be valid appear not to work after upgrade, or other UI issues are seen, try clearing your browser cache and reloading the UI page.

Ensure that you query your IPaddress usage records and process them; for example, issue invoices for any usage that you have not yet billed users for. Starting in 3.0.2, the usage record format for IP addresses is the same as the rest of the usage types. Instead of a single record with the assignment and release dates, separate records are generated per aggregation period with start and end dates. After upgrading to 4.2.0, any existing IP address usage records in the old format will no longer be available. If you are using version 2.2.0 - 2.2.13, first upgrade to 2.2.14 by using the instructions in the 2.2.14 Release Notes. If KVM hypervisor is used in your cloud, be sure you completed the step to insert a valid username and password into the host_details table on each KVM node as described in the 2.2.14 Release Notes. This step is critical, as the database will be encrypted after the upgrade to 4.2.0. While running the 2.2.14 system, log in to the UI as root administrator. Using the UI, add a new System VM template for each hypervisor type that is used in your cloud. In each zone, add a system VM template for each hypervisor used in that zone In the left navigation bar, click Templates. In Select view, click Templates. Click Register template. The Register template dialog box is displayed. In the Register template dialog box, specify the following values depending on the hypervisor type (do not change these): Hypervisor Description XenServer Name: systemvm-xenserver-4.2.0 Description: systemvm-xenserver-4.2.0 URL:http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2 Zone: Choose the zone where this hypervisor is used Hypervisor: XenServer Format: VHD OS Type: Debian GNU/Linux 6.0 (32-bit) Extractable: no Password Enabled: no Public: no Featured: no KVM Name: systemvm-kvm-4.2.0 Description: systemvm-kvm-4.2.0 URL: http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2 Zone: Choose the zone where this hypervisor is used Hypervisor: KVM Format: QCOW2 OS Type: Debian GNU/Linux 5.0 (32-bit) Extractable: no Password Enabled: no Public: no Featured: no VMware Name: systemvm-vmware-4.2.0 Description: systemvm-vmware-4.2.0 URL: http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh7.ova Zone: Choose the zone where this hypervisor is used Hypervisor: VMware Format: OVA OS Type: Debian GNU/Linux 5.0 (32-bit) Extractable: no Password Enabled: no Public: no Featured: no Watch the screen to be sure that the template downloads successfully and enters the READY state. Do not proceed until this is successful WARNING: If you use more than one type of hypervisor in your cloud, be sure you have repeated these steps to download the system VM template for each hypervisor type. Otherwise, the upgrade will fail. Stop all Usage Servers if running. Run this on all Usage Server hosts. # service cloud-usage stop Stop the Management Servers. Run this on all Management Server hosts. # service cloud-management stop On the MySQL master, take a backup of the MySQL databases. We recommend performing this step even in test upgrades. If there is an issue, this will assist with debugging. In the following commands, it is assumed that you have set the root password on the database, which is a CloudStack recommended best practice. Substitute your own MySQL root password. # mysqldump -u root -pmysql_password cloud > cloud-backup.dmp # mysqldump -u root -pmysql_password cloud_usage > cloud-usage-backup.dmp Either build RPM/DEB packages as detailed in the Installation Guide, or use one of the community provided yum/apt repositories to gain access to the &PRODUCT; binaries. If you are using Ubuntu, follow this procedure to upgrade your packages. If not, skip to step . This section assumes you're using the community supplied packages for &PRODUCT;. If you've created your own packages and APT repository, substitute your own URL for the ones used in these examples. The first order of business will be to change the sources list for each system with &PRODUCT; packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.) Start by opening /etc/apt/sources.list.d/cloudstack.list on any systems that have &PRODUCT; packages installed. This file should have one line, which contains: deb http://cloudstack.apt-get.eu/ubuntu precise 4.0 We'll change it to point to the new package repository: deb http://cloudstack.apt-get.eu/ubuntu precise 4.2 If you're using your own package repository, change this line to read as appropriate for your 4.2.0 repository. Now update your apt package list: $ sudo apt-get update Now that you have the repository configured, it's time to install the cloudstack-management package. This will pull in any other dependencies you need. $ sudo apt-get install cloudstack-management On KVM hosts, you will need to manually install the cloudstack-agent package: $ sudo apt-get install cloudstack-agent During the installation of cloudstack-agent, APT will copy your agent.properties, log4j-cloud.xml, and environment.properties from /etc/cloud/agent to /etc/cloudstack/agent. When prompted whether you wish to keep your configuration, say Yes. Verify that the file /etc/cloudstack/agent/environment.properties has a line that reads: paths.script=/usr/share/cloudstack-common If not, add the line. Restart the agent: service cloud-agent stop killall jsvc service cloudstack-agent start During the upgrade, log4j-cloud.xml was simply copied over, so the logs will continue to be added to /var/log/cloud/agent/agent.log. There's nothing wrong with this, but if you prefer to be consistent, you can change this by copying over the sample configuration file: cd /etc/cloudstack/agent mv log4j-cloud.xml.dpkg-dist log4j-cloud.xml service cloudstack-agent restart Once the agent is running, you can uninstall the old cloud-* packages from your system: sudo dpkg --purge cloud-agent If you are using CentOS or RHEL, follow this procedure to upgrade your packages. If not, skip to step . This section assumes you're using the community supplied packages for &PRODUCT;. If you've created your own packages and yum repository, substitute your own URL for the ones used in these examples. The first order of business will be to change the yum repository for each system with &PRODUCT; packages. This means all management servers, and any hosts that have the KVM agent. (No changes should be necessary for hosts that are running VMware or Xen.) Start by opening /etc/yum.repos.d/cloudstack.repo on any systems that have &PRODUCT; packages installed. This file should have content similar to the following: [apache-cloudstack] name=Apache CloudStack baseurl=http://cloudstack.apt-get.eu/rhel/4.0/ enabled=1 gpgcheck=0 If you are using the community provided package repository, change the baseurl to http://cloudstack.apt-get.eu/rhel/4.2/ If you're using your own package repository, change this line to read as appropriate for your 4.2.0 repository. Now that you have the repository configured, it's time to install the cloudstack-management package by upgrading the older cloud-client package. $ sudo yum upgrade cloud-client For KVM hosts, you will need to upgrade the cloud-agent package, similarly installing the new version as cloudstack-agent. $ sudo yum upgrade cloud-agent During the installation of cloudstack-agent, the RPM will copy your agent.properties, log4j-cloud.xml, and environment.properties from /etc/cloud/agent to /etc/cloudstack/agent. Verify that the file /etc/cloudstack/agent/environment.properties has a line that reads: paths.script=/usr/share/cloudstack-common If not, add the line. Restart the agent: service cloud-agent stop killall jsvc service cloudstack-agent start If you have made changes to your existing copy of the file components.xml in your previous-version CloudStack installation, the changes will be preserved in the upgrade. However, you need to do the following steps to place these changes in a new version of the file which is compatible with version 4.0.0-incubating. How will you know whether you need to do this? If the upgrade output in the previous step included a message like the following, then some custom content was found in your old components.xml, and you need to merge the two files: warning: /etc/cloud/management/components.xml created as /etc/cloud/management/components.xml.rpmnew Make a backup copy of your /etc/cloud/management/components.xml file. For example: # mv /etc/cloud/management/components.xml /etc/cloud/management/components.xml-backup Copy /etc/cloud/management/components.xml.rpmnew to create a new /etc/cloud/management/components.xml: # cp -ap /etc/cloud/management/components.xml.rpmnew /etc/cloud/management/components.xml Merge your changes from the backup file into the new components.xml file. # vi /etc/cloud/management/components.xml After upgrading to 4.2, API clients are expected to send plain text passwords for login and user creation, instead of MD5 hash. Incase, api client changes are not acceptable, following changes are to be made for backward compatibility: Modify componentsContext.xml, and make PlainTextUserAuthenticator as the default authenticator (1st entry in the userAuthenticators adapter list is default) <!-- Security adapters --> <bean id="userAuthenticators" class="com.cloud.utils.component.AdapterList"> <property name="Adapters"> <list> <ref bean="PlainTextUserAuthenticator"/> <ref bean="MD5UserAuthenticator"/> <ref bean="LDAPUserAuthenticator"/> </list> </property> </bean> PlainTextUserAuthenticator works the same way MD5UserAuthenticator worked prior to 4.2. If you have made changes to your existing copy of the /etc/cloud/management/db.properties file in your previous-version CloudStack installation, the changes will be preserved in the upgrade. However, you need to do the following steps to place these changes in a new version of the file which is compatible with version 4.0.0-incubating. Make a backup copy of your file /etc/cloud/management/db.properties. For example: # mv /etc/cloud/management/db.properties /etc/cloud/management/db.properties-backup Copy /etc/cloud/management/db.properties.rpmnew to create a new /etc/cloud/management/db.properties: # cp -ap /etc/cloud/management/db.properties.rpmnew etc/cloud/management/db.properties Merge your changes from the backup file into the new db.properties file. # vi /etc/cloud/management/db.properties On the management server node, run the following command. It is recommended that you use the command-line flags to provide your own encryption keys. See Password and Key Encryption in the Installation Guide. # cloudstack-setup-encryption -e encryption_type -m management_server_key -k database_key When used without arguments, as in the following example, the default encryption type and keys will be used: (Optional) For encryption_type, use file or web to indicate the technique used to pass in the database encryption password. Default: file. (Optional) For management_server_key, substitute the default key that is used to encrypt confidential parameters in the properties file. Default: password. It is highly recommended that you replace this with a more secure value (Optional) For database_key, substitute the default key that is used to encrypt confidential parameters in the CloudStack database. Default: password. It is highly recommended that you replace this with a more secure value. Repeat steps 10 - 14 on every management server node. If you provided your own encryption key in step 14, use the same key on all other management servers. Start the first Management Server. Do not start any other Management Server nodes yet. # service cloudstack-management start Wait until the databases are upgraded. Ensure that the database upgrade is complete. You should see a message like "Complete! Done." After confirmation, start the other Management Servers one at a time by running the same command on each node. Start all Usage Servers (if they were running on your previous version). Perform this on each Usage Server host. # service cloudstack-usage start (KVM only) Additional steps are required for each KVM host. These steps will not affect running guests in the cloud. These steps are required only for clouds using KVM as hosts and only on the KVM hosts. Configure your CloudStack package repositories as outlined in the Installation Guide Stop the running agent. # service cloud-agent stop Update the agent software with one of the following command sets as appropriate. # yum update cloud-* # apt-get update # apt-get upgrade cloud-* Start the agent. # service cloudstack-agent start Copy the contents of the agent.properties file to the new agent.properties file by using the following command sed -i 's/com.cloud.agent.resource.computing.LibvirtComputingResource/com.cloud.hypervisor.kvm.resource.LibvirtComputingResource/g' /etc/cloud/agent/agent.properties Start the cloud agent and cloud management services. When the Management Server is up and running, log in to the CloudStack UI and restart the virtual router for proper functioning of all the features. Log in to the CloudStack UI as admin, and check the status of the hosts. All hosts should come to Up state (except those that you know to be offline). You may need to wait 20 or 30 minutes, depending on the number of hosts. Do not proceed to the next step until the hosts show in the Up state. If the hosts do not come to the Up state, contact support. Run the following script to stop, then start, all Secondary Storage VMs, Console Proxy VMs, and virtual routers. Run the command once on one management server. Substitute your own IP address of the MySQL instance, the MySQL user to connect as, and the password to use for that user. In addition to those parameters, provide the "-c" and "-r" arguments. For example: # nohup cloud-sysvmadm -d 192.168.1.5 -u cloud -p password -c -r > sysvm.log 2>&1 & # tail -f sysvm.log This might take up to an hour or more to run, depending on the number of accounts in the system. After the script terminates, check the log to verify correct execution: # tail -f sysvm.log The content should be like the following: Stopping and starting 1 secondary storage vm(s)... Done stopping and starting secondary storage vm(s) Stopping and starting 1 console proxy vm(s)... Done stopping and starting console proxy vm(s). Stopping and starting 4 running routing vm(s)... Done restarting router(s). If you would like additional confirmation that the new system VM templates were correctly applied when these system VMs were rebooted, SSH into the System VM and check the version. Use one of the following techniques, depending on the hypervisor. SSH in by using the link local IP address of the system VM. For example, in the command below, substitute your own path to the private key used to log in to the system VM and your own link local IP. Run the following commands on the XenServer or KVM host on which the system VM is present: # ssh -i private-key-path link-local-ip -p 3922 # cat /etc/cloudstack-release The output should be like the following: Cloudstack Release 4.0.0-incubating Mon Oct 9 15:10:04 PST 2012 SSH in using the private IP address of the system VM. For example, in the command below, substitute your own path to the private key used to log in to the system VM and your own private IP. Run the following commands on the Management Server: # ssh -i private-key-path private-ip -p 3922 # cat /etc/cloudstack-release The output should be like the following: Cloudstack Release 4.0.0-incubating Mon Oct 9 15:10:04 PST 2012 If needed, upgrade all Citrix XenServer hypervisor hosts in your cloud to a version supported by CloudStack 4.0.0-incubating. The supported versions are XenServer 5.6 SP2 and 6.0.2. Instructions for upgrade can be found in the CloudStack 4.0.0-incubating Installation Guide. Apply the XenServer hotfix XS602E003 (and any other needed hotfixes) to XenServer v6.0.2 hypervisor hosts. Disconnect the XenServer cluster from CloudStack. In the left navigation bar of the CloudStack UI, select Infrastructure. Under Clusters, click View All. Select the XenServer cluster and click Actions - Unmanage. This may fail if there are hosts not in one of the states Up, Down, Disconnected, or Alert. You may need to fix that before unmanaging this cluster. Wait until the status of the cluster has reached Unmanaged. Use the CloudStack UI to check on the status. When the cluster is in the unmanaged state, there is no connection to the hosts in the cluster. To clean up the VLAN, log in to one XenServer host and run: /opt/xensource/bin/cloud-clean-vlan.sh Prepare the upgrade by running the following on one XenServer host: /opt/xensource/bin/cloud-prepare-upgrade.sh If you see a message like "can't eject CD", log in to the VM and umount the CD, then run this script again. Upload the hotfix to the XenServer hosts. Always start with the Xen pool master, then the slaves. Using your favorite file copy utility (e.g. WinSCP), copy the hotfixes to the host. Place them in a temporary folder such as /root or /tmp. On the Xen pool master, upload the hotfix with this command: xe patch-upload file-name=XS602E003.xsupdate Make a note of the output from this command, which is a UUID for the hotfix file. You'll need it in another step later. (Optional) If you are applying other hotfixes as well, you can repeat the commands in this section with the appropriate hotfix number. For example, XS602E004.xsupdate. Manually live migrate all VMs on this host to another host. First, get a list of the VMs on this host: # xe vm-list Then use this command to migrate each VM. Replace the example host name and VM name with your own: # xe vm-migrate live=true host=host-name vm=VM-name If you see a message like "You attempted an operation on a VM which requires PV drivers to be installed but the drivers were not detected," run: /opt/xensource/bin/make_migratable.sh b6cf79c8-02ee-050b-922f-49583d9f1a14. Apply the hotfix. First, get the UUID of this host: # xe host-list Then use the following command to apply the hotfix. Replace the example host UUID with the current host ID, and replace the hotfix UUID with the output from the patch-upload command you ran on this machine earlier. You can also get the hotfix UUID by running xe patch-list. xe patch-apply host-uuid=host-uuid uuid=hotfix-uuid Copy the following files from the CloudStack Management Server to the host. Copy from here... ...to here /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver60/NFSSR.py /opt/xensource/sm/NFSSR.py /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/setupxenserver.sh /opt/xensource/bin/setupxenserver.sh /usr/lib64/cloudstack-common/scripts/vm/hypervisor/xenserver/make_migratable.sh /opt/xensource/bin/make_migratable.sh (Only for hotfixes XS602E005 and XS602E007) You need to apply a new Cloud Support Pack. Download the CSP software onto the XenServer host from one of the following links: For hotfix XS602E005: http://coltrane.eng.hq.xensource.com/release/XenServer-6.x/XS-6.0.2/hotfixes/XS602E005/56710/xe-phase-2/xenserver-cloud-supp.tgz For hotfix XS602E007: http://coltrane.eng.hq.xensource.com/release/XenServer-6.x/XS-6.0.2/hotfixes/XS602E007/57824/xe-phase-2/xenserver-cloud-supp.tgz Extract the file: # tar xf xenserver-cloud-supp.tgz Run the following script: # xe-install-supplemental-pack xenserver-cloud-supp.iso If the XenServer host is part of a zone that uses basic networking, disable Open vSwitch (OVS): # xe-switch-network-backend bridge Reboot this XenServer host. Run the following: /opt/xensource/bin/setupxenserver.sh If the message "mv: cannot stat `/etc/cron.daily/logrotate': No such file or directory" appears, you can safely ignore it. Run the following: for pbd in `xe pbd-list currently-attached=false| grep ^uuid | awk '{print $NF}'`; do xe pbd-plug uuid=$pbd ; On each slave host in the Xen pool, repeat these steps, starting from "manually live migrate VMs."