Sonarcube is great, but has no context and can be wrong, in this
case File.separator is nice if you're running platform independent,
the agent on OVM3 is however NOT platform independent and will break
if we feed it windows separators.
The copy command reply can have null size returned, so check and set values
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 53ca0b1861)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fix: RTNETLINK errors
- Management Server health check trying to create already existing interface
- Changes on update_config.py, cs_guestnetwork.py, merger.py
Fix: replace RRouTER_LOG in the CsRedundant.py per log file location
Fix: Guest Net address association during Router restart
- Changes on NicProfileHelper, NicProfileHelperImpl
Fix: aggregationExecution() method on VirtualNetworkApplianceManagerImpl
- Do not send an AggregationControlCommand to a non-configured router
Some classes have been formatted.
Added a source column to the user table.
Source now has only two values UNKNOWN,LDAP with UNKNOWN being the
default and is an enum is com.cloud.User.
When the source is UNKNOWN, the old method of authenticating against all
the available authenticators is used. If a source is available, only
that particular authenticator will be used.
added overloaded methods in AccountService to createUserAccount and
createUser with source specified.
(cherry picked from commit 5da733072e)
Added a source column to the user table.
Source now has only two values UNKNOWN,LDAP with UNKNOWN being the
default and is an enum is com.cloud.User.
When the source is UNKNOWN, the old method of authenticating against all
the available authenticators is used. If a source is available, only
that particular authenticator will be used.
added overloaded methods in AccountService to createUserAccount and
createUser with source specified.
The 19e3c0168e commit breaks SAML login and any
login where redirection is used.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit b79f13ccb5)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
MigrateVMWithVolumes-
1. If ESXi host version is below 5.1, ensure destination datastore(s) is mounted on the source host, then migrate the storage and then finally migrate the VM.
If destination storage(s) is not mounted on the source host,
- In case of NFS storage mount the storage(s).
- In case of VMFS storage fail the request for migration.
2. If EXi host version is 5.1 or above, simultaneously migrate the VM and its storage to the destination host and storage(s) respectively for both NFS and VMFS storage.
This is a plugin that puts in ovm3 support ranging from 3.3.1 to 3.3.2. Basic
functionality is in here, advanced networking etc..
Snapshots only work when a VM is stopped now due to the semantics of OVM's raw
image implementation (so snapshots should work on a storage level underneath the
hypervisor shrug)
This closes#113
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This commit produces event bus messages to a "cloudstack" topic
in Apache Kafka. Configuration is expected to be found in
/etc/cloudstack/management/kafka.producer.properties and will
generally be of the form:
bootstrap.servers=kafka-host1:9092,kafka-host2:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
There is no way to parameterize the topic yet, and the consuming
code is just place-holder. I think adding a consumer within cloudstack
is very debatable and likely not needed.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
When there is large size VR configuration (aggregate commands) copying data to VR using vmops plugin was failed
because of the ARG_MAX size limitation. The configuration data size is around 300KB.
Updated this to create file in host by scp with file contents. This will create file in host.
Then copy the file from the host to VR using hte vmops createFileInDomr method.
In host file get created in /tmp/ with name VR-<UUID>.cfg, once it copied to VR this file will be removed.
(cherry picked from commit 619f014255)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
When there is large size VR configuration (aggregate commands) copying data to VR using vmops plugin was failed
because of the ARG_MAX size limitation. The configuration data size is around 300KB.
Updated this to create file in host by scp with file contents. This will create file in host.
Then copy the file from the host to VR using hte vmops createFileInDomr method.
In host file get created in /tmp/ with name VR-<UUID>.cfg, once it copied to VR this file will be removed.
The test testFindSecurityProfileByUuid built an assumptio on the order of the arguments after transforming through a HashMap.
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Refactored to use the XPatch expressions to check the generated domain xml rathern than string comparison.
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Design Document:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Allow+VM+listing+by+User+ID
- Adds column to VMInstance DAO
- Adds column in vm_instance table
- Adds column in the UserVMJoinVO
- Adds default admin user which has UID = 2
- Adds migration path that sets user_id to first user of the accountId that
owns the vm in vm_instance table
- Add arg on list VMs API to query by userId, add support in query layer
- Refactor VMInstanceVO and child classes to accept userId
- Add code to let service layer pass userId if loggedIn user belongs to same
account as the owner executing an API call or use first user from owner account
- In case of CPVM and SSVM use system user ID
- Fix unit tests and spring injections
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Earlier host addition of multiple hosts with local storage failed due to
same local storage UUID being used where the storage path is same.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit bf17f640c6)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Earlier host addition of multiple hosts with local storage failed due to
same local storage UUID being used where the storage path is same.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fix cleanup of NFS mounts on management server during server starup by correcting how mount points are listed for a management server.
(cherry picked from commit 647532376f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
For AttachVolume/DetachVolume API command, improve user error message in case of RuntimeException by throwing the exception instead of 'Unexpected Exception'.
(cherry picked from commit 4d7ede535d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Conflicts:
server/src/com/cloud/storage/VolumeApiServiceImpl.java
If VM has been cold migrated across different VMware DCs, then unregister the VM from source host.
(cherry picked from commit 15b348632d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
1. If a VM by the same name exists on a different cluster in the VMware DC, unregister the existing VM and continue with the VM start.
2. If VM start succeeds, delete VM files associated with the unregistered VM.
3. If VM start fails, re-register the unregistered VM.
(cherry picked from commit 974b0180dd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
If an exact match is being done while locating disk chain by name, don't trim snapshot postfix appended to the disk name.
(cherry picked from commit ddcae8a930)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Before registering a VM check if a different CS VM with same name exists in vCenter.
(cherry picked from commit 33179cce56)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Look for a VM in vCenter based on both the vCenter name and CS internal name (required in case 'vm.instancename.flag' is enabled).
During Attach Volume and Volume Migration, for lookup and other operations use VM's name as obtained from vCenter instead of using the name set in the agent command.
(cherry picked from commit b8fdda0a34)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixed some issues with the polling logic to handle scenarios when SSVM is destroyed/down.
Also changed the status of the volume_store_ref entries so that subsequent operations work fine.
KVMStoragePoolManager is a singleton in practice, any plugin
or extension of LibvirtComputingResource will need to act on
the specific instance of KVMStoragePoolManager that LibvirtComputingResource
has initialized. Therefore, expose this variable for those who
wish to call storage commands from plugins or extensions.
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
KVMStoragePoolManager is a singleton in practice, any plugin
or extension of LibvirtComputingResource will need to act on
the specific instance of KVMStoragePoolManager that LibvirtComputingResource
has initialized. Therefore, expose this variable for those who
wish to call storage commands from plugins or extensions.
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
The fix generates X509Certificate if missing from DB and uses that for eternity.
SAML SP metadata remains same since it's using the same X509 certificate and
it remains same after restarts. The certificate is serialized, base64 encoded
and stored in the keystore table under a specific name. For reading, it's
retrieved, base64 decoded and deserialized.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 4358714381)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The fix generates X509Certificate if missing from DB and uses that for eternity.
SAML SP metadata remains same since it's using the same X509 certificate and
it remains same after restarts. The certificate is serialized, base64 encoded
and stored in the keystore table under a specific name. For reading, it's
retrieved, base64 decoded and deserialized.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
As per Version 1 cookies, certain characters are now allowed such as space,
colons etc but they should be url encoded using UTF8 encoding. The frontend
has a cookie value unboxing method that removes any double quotes that are added.
As per the doc http://download.oracle.com/javase/6/docs/api/java/net/URLEncoder.html
values are application/x-www-form-urlencoded and as per
http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4 whitespaces are encoded
as +, therefore '+' are replaced by %20 (whitespace).
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 734bd70173)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
As per Version 1 cookies, certain characters are now allowed such as space,
colons etc but they should be url encoded using UTF8 encoding. The frontend
has a cookie value unboxing method that removes any double quotes that are added.
As per the doc http://download.oracle.com/javase/6/docs/api/java/net/URLEncoder.html
values are application/x-www-form-urlencoded and as per
http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4 whitespaces are encoded
as +, therefore '+' are replaced by %20 (whitespace).
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The User table's UUID column is restricted to 40 chars only, since we don't
know how long the nameID/userID of a SAML authenticated user will be - the fix
hashes that user ID and takes a substring of length 40 chars. For hashing,
SHA256 is used which returns a 64 char length string.
- Fix tests, add test cases
- Improve checkSAMLUser method
- Use SHA256 one way hashing to create unique UUID for SAML users
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit b2b496288d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The User table's UUID column is restricted to 40 chars only, since we don't
know how long the nameID/userID of a SAML authenticated user will be - the fix
hashes that user ID and takes a substring of length 40 chars. For hashing,
SHA256 is used which returns a 64 char length string.
- Fix tests, add test cases
- Improve checkSAMLUser method
- Use SHA256 one way hashing to create unique UUID for SAML users
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
1. If a VM by the same name exists on a different cluster in the VMware DC, unregister the existing VM and continue with the VM start.
2. If VM start succeeds, delete VM files associated with the unregistered VM.
3. If VM start fails, re-register the unregistered VM.
For AttachVolume/DetachVolume API command, improve user error message in case of RuntimeException by throwing the exception instead of 'Unexpected Exception'.
on calling GetUploadParamsForTemplate, persisting the metadata to db
validating the account limits and incrementing the appropriate limits
encoded the metadata on management server using preshared key
On default iptables rules are updated to add ACCEPT egress traffic.
If the network egress default policy is false, CS remove ACCEPT and adds the DROP rule which
is egress default rule when there are no other egress rules.
If the CS network egress default policy is true, CS won't configure any default rule for egress because
router already came up to accept egress traffic. If there are already egress rules for network then the
egress rules get applied on VR.
For isolated network with out firewall service, VR default allows egress traffic (guestnetwork --> public network)
added 3 new method to strip partition information from VirtualServer, LBPool, VLAN api response.
With BigIP V11.x VirtualServer, LBPool, VLAN api response has been modified.
Now BigIP returns resource name with user partition information
ex: if vlanname is vlan-100 then the get_list() will return /Common/vlan-100 (/Common -> Suer portition)
This method will strip the partition information and only returns a list with vlan name (vlan-100)
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
On default iptables rules are updated to add ACCEPT egress traffic.
If the network egress default policy is false, CS remove ACCEPT and adds the DROP rule which
is egress default rule when there are no other egress rules.
If the CS network egress default policy is true, CS won't configure any default rule for egress because
router already came up to accept egress traffic. If there are already egress rules for network then the
egress rules get applied on VR.
For isolated network with out firewall service, VR default allows egress traffic (guestnetwork --> public network)
Changes;
- Upgrades maven-war plugin to 4.5 (faster war packaging)
- Upgrade spring framework to latest minor release
- Upgrade ehcache, jasypt, httpclient, httpcore and other core dependencies
- Upgrade to latest ipv6 library, fix unit test NetUtilsTest
- httpcore and httpclient are sharing same version variable
- commons-httpclient is different that httpclient, the fix gives it a separate var
- Apidocs failed to generate and get stuck with new reflections version, for now
we will continue using 0.9.8
Newer dependencies can be listed using:
mvn versions:display-dependency-updates -Dnoredist -Dsimulator -P developer,systemvm
Testing;
- Tested using Maven 3.2.1
- Local noredist build with unit tests succeeds
- CloudStack mgmt server started, basic business layer tests work
- Observed 10-15% build time improvement using new maven-war plugin
Branch: bugfix/4.5-8011 (commits are squashed in favour of a linear history)
Pull request:
https://github.com/apache/cloudstack/pull/50
This closes#50
TravisCI build summary:
https://travis-ci.org/shapeblue/cloudstack/builds/42902172
- Build passes with unit tests
- Apidocs generates successfully
- Most integration tests pass, some fail due to timeout errors, second re-run
passes some of them
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit fac7bfc5d5)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Conflicts:
pom.xml
Changes;
- Upgrades maven-war plugin to 4.5 (faster war packaging)
- Upgrade spring framework to latest minor release
- Upgrade ehcache, jasypt, httpclient, httpcore and other core dependencies
- Upgrade to latest ipv6 library, fix unit test NetUtilsTest
- httpcore and httpclient are sharing same version variable
- commons-httpclient is different that httpclient, the fix gives it a separate var
- Apidocs failed to generate and get stuck with new reflections version, for now
we will continue using 0.9.8
Newer dependencies can be listed using:
mvn versions:display-dependency-updates -Dnoredist -Dsimulator -P developer,systemvm
Testing;
- Tested using Maven 3.2.1
- Local noredist build with unit tests succeeds
- CloudStack mgmt server started, basic business layer tests work
- Observed 10-15% build time improvement using new maven-war plugin
Branch: bugfix/4.5-8011 (commits are squashed in favour of a linear history)
Pull request:
https://github.com/apache/cloudstack/pull/50
This closes#50
TravisCI build summary:
https://travis-ci.org/shapeblue/cloudstack/builds/42902172
- Build passes with unit tests
- Apidocs generates successfully
- Most integration tests pass, some fail due to timeout errors, second re-run
passes some of them
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This should fix that we can't gather CPU statistics on hypervisors
> Ubuntu 12.04
(cherry picked from commit 69ee01af9d)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Findbugs will report error on this as it is expecting true/false for Boolean value.
But we have diffrent meaning for null so it is false positive case from findbug
This closes#39
Findbugs will report error on this as it is expecting true/false for Boolean value.
But we have diffrent meaning for null so it is false positive case from findbug
This closes#39
Look for a VM in vCenter based on both the vCenter name and CS internal name (required in case 'vm.instancename.flag' is enabled).
During Attach Volume and Volume Migration, for lookup and other operations use VM's name as obtained from vCenter instead of using the name set in the agent command.
Web UI shows all DHCP/PXE providers in cloud when admin click DHCP/PXE IP for A zone
Conflicts:
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalKickStartServiceImpl.java
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
Also when the ssvm is destroyed all the download urls are expired to be cleaned up in the next run by the new ssvm.
(cherry picked from commit ce90837357)
Clearly show if a volume is found and if not, that the pool is being refreshed
and the fetch is tried again.
Due to my commit b53a9dcc9f the chance of a volume
not being found is slightly bigger, but the performance gain is enormous on larger
deployments.
This is why we clearly have to log that we are refreshing the pool information
when a volume is not found.
It could be that a volume is created on host A and a few seconds later host B tries
to access the volume. In that case host B's libvirt doesn't know about the volume
yet and has to refresh the pool before it does.
(cherry picked from commit 4ee82f1f40)
2.8.7 to 3.3.5
3.3.5 is the latest stable version of AMQP client, which is also
backward comaptible. Successfully tested with updated client library.
(cherry picked from commit ff797dfa59)
modules can use it.
(cherry picked from commit b6401b04f22b0a5b686c7c477da4c6e0fd18df84)
Conflicts:
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalKickStartServiceImpl.java
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
(cherry picked from commit 39fe766c2b)
Support offline volume resize on ESX by creating a worker VM to attach the unattached volume to and then resize it.
(cherry picked from commit 65ed25b7a6)
While expunging a volume, CS chooses the endpoint to perform delete operation by selecting any host that has the storage containing the volume mounted on it.
Instead, if the volume to be deleted is attached to a VM, the endpoint chosen by CCP should be the host that contains the VM.
(cherry picked from commit f1e3e83bbf)
If CCP thread local context which is used to handle connections to a vCenter is being re-used, validate that the context corresponds to the right vCenter API session.
(cherry picked from commit 6b06970366)
On larger (especially RBD) storage pools this can take a lot of
time slowing operations like creating volumes down.
The getStorageStats command will still ask a pool to be refreshed so
that the management server has accurate information about the storage pools.
On larger deployments, with thousands of volumes in one pool, this should
significantly improve storage related operations
(cherry picked from commit b53a9dcc9f)
cleaning up vmsync changes, checkvirtualmachine command was updated to return the
power state of the vm. The change was missed for Hyper-V. This caused migration to
fail on cloudstack even though it used to succeed on the hypervisor. Updated the
hyper-v agent code to return the cloudstack equivalent power state for check virtual
machine answer.
(cherry picked from commit 5350e61187)
Clearly show if a volume is found and if not, that the pool is being refreshed
and the fetch is tried again.
Due to my commit b53a9dcc9f the chance of a volume
not being found is slightly bigger, but the performance gain is enormous on larger
deployments.
This is why we clearly have to log that we are refreshing the pool information
when a volume is not found.
It could be that a volume is created on host A and a few seconds later host B tries
to access the volume. In that case host B's libvirt doesn't know about the volume
yet and has to refresh the pool before it does.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
While expunging a volume, CS chooses the endpoint to perform delete operation by selecting any host that has the storage containing the volume mounted on it.
Instead, if the volume to be deleted is attached to a VM, the endpoint chosen by CCP should be the host that contains the VM.
If CCP thread local context which is used to handle connections to a vCenter is being re-used, validate that the context corresponds to the right vCenter API session.
On larger (especially RBD) storage pools this can take a lot of
time slowing operations like creating volumes down.
The getStorageStats command will still ask a pool to be refreshed so
that the management server has accurate information about the storage pools.
On larger deployments, with thousands of volumes in one pool, this should
significantly improve storage related operations
cleaning up vmsync changes, checkvirtualmachine command was updated to return the
power state of the vm. The change was missed for Hyper-V. This caused migration to
fail on cloudstack even though it used to succeed on the hypervisor. Updated the
hyper-v agent code to return the cloudstack equivalent power state for check virtual
machine answer.
Fix the bug by removing the command CreateEntityDownloadURLCommand from the host delegation. This results in same ssvm for creating the symlink on ssvm and same public ip being used for generating the url on MS.
NPE can happen if Spring fails to inject api authenticator, so better check
and set list of commands if the authenticator is not null or returning null cmds
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Return super.true() even if plugin is not enabled
- Return empty list when getCommands is called
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Latest httpclient library suggests when creating Protocol object for use
with the apache common httpclient class, they should avoid using the
deprecated Protocol signature which takes in SecureProtocolSocketFactory
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
If session is null, probably logout (local) happened removing the name id and
session index which is needed for global logout. The limitation by design is that
local logout will void possibility of global logout. To globally logout, one
use the SLO api which would logout locally as well.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Fixes signatures on plugin manager for ease of testing
- Fixes authenticator
- Adds unit testing for getType and authenticate methods for all cmd classes
- Adds SAMLAuthenticator test
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This adds GetServiceProviderMetaDataCmd which returns SP metadata XML, since
this information should be public for IdPs to discover, we implement this as a
login/cmd api so this does not require any kind of authentication to GET this
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Use opensaml version from root pom
- Add utils and api as explicit dependency
- Add org.apache.cloudstack.saml.SAML2AuthServiceImpl bean
- Fix imports in all source files and resource xmls
- Use methods available from SAMLUtils to encode/decode SAML request/response
- SAML logout api is not the global logout api
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Move interfaces and classes from server to api module
- This can be then used for pluggable api authenticators
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This is a feature to handle DNS entries by means of an external DNS Provider,
such as Bind. These entries include DNS domains and reverse domains, VM records
and reverse records.
For a complete description, please refer to the design document available at
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Bind+and+PowerDNS+integration+by+Globo+DNSAPI
For the discussion about this feature on the dev mailing list, please refer to
http://markmail.org/thread/fvwf36hpxotiibka
Summary:
- new Network Service Provider called GloboDNS
- new Network Element to manage network domains and VM records (entries) on an external API
- new Network Resource to communicate with GloboDNS (open source)
- new API command to add DNS server
- new global option to determine if this provider should override VM entries on external DNS server
- changes in UI to include GloboDNS in Providers list
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
1. While destroying a ROOT volume do the lookup of the associated VM under the DC and not just cluster.
2. In case of VMware, during VM start if a volume is being recreated no need to detach the old volume because
we now expunge it immediately and don't wait for the storage cleanup task to run.
During VM start while configuring its disk devices, obtain the matching disk for a volume in storage
using both the volume's path and volume's datastore information.
While adding host to existing cluster which is using Nexus 1000v as a network backend, skip validation of Nexus VSM as it was already done while adding that cluster.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
determined. To exclude hosted system we filter the result on Caption='Virtual Machine'
but this string is locale dependent so it may not not work properly for locales other
than english. To overcome this now we started using ProcessId >= 0 filter
For ResizeVolume API command -
1. If hypervisor resource throws an exception, handle the NPE thrown by the job framework.
2. Improve user error message in case of RuntimeException by throwing the exception instead of 'Unexpected Exception'.
We don't need an external script to investigate the format of the RBD volume,
we only have to ask Libvirt to resize the volume and that will ask librbd to
do so.
1. Fixed JSON response deserialization. While creating a mock a JSON can be passed which will be deserialized into a response object and returned from agent layer.
For e.g. for a mock corresponding to StopCommand, a response like "{"com.cloud.agent.api.StopAnswer":{"result":false,"wait":0}}" can be passed.
2. Ability to mock PingCommand (returned as part of getCurrentStatus() agent method). As a part of this a mocked VM state report can be returned.
For e.g. {"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_hostVmStateReport":{"v-2-VM":{"state":"PowerOn","host":"SimulatedAgent.e6df7732-69b2-429b-9b6a-3e24dddfa2e0"},"i-2-5-VM":{"state":"PowerOff","host":"SimulatedAgent.e6df7732-69b2-429b-9b6a-3e24dddfa2e0"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":3,"contextMap":{},"wait":0}}
in the XenServer pool for GRE tunnel networks
Fix uses XenServer recommended way
Network.other_config:assume_network_is_shared=true
which ensures bridge is created automatically on hosts in the pool for
GRE tunnel networks. Fix also gets rid of error prone custom logic that ensures
bridge is created by plugging a VIF into the dom0 and connected to
GRE tunnel network.
Conflicts:
plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java
xen cluster
XenServer does not create a bridge automatically when VIF from domU is connected
to internal network. So there is logic to force bridge creation by
creating VIF in dom0 connected to GRE tunnel network. But there is no
logic to delete the VIF after bridge gets created. So this fix ensure
VIF is delted when atleast there is one domU VIF connected to the
network.
1. Adding the missing Template/Volume URLs expiration functionality
2. Improvement - While deleting the volume during expiration use rm -rf as vmware now contains directoy
3. Improvement - Use standard Answer so that the error gets logged in case deletion of expiration link didnt work fine.
4. Improvement - In case of domain change, expire the old urls
In situations where libvirt lost the storage pool the KVM Agent will re-create the
storage pool in libvirt.
This could be then libvirt is restarted for example.
The object returned internally was missing essential information like the sourceDir
aka the Ceph pool, the monitor IPs, cephx information and such.
In this case the first operation on this newly created pool would fail. All operations
afterwards would succeed.
The proxy-arp add/del is done on firewall rule add/del.
The proxy-arp rule is deleted only when there is no static nat or dest nat rule is not using the ip.
When there is static nat or PF and firewall rule
a. Delete firewall rule. It skips delete proxy-arp because the rule is used by static nat rule.
b. After deleting fw rule if we disable static nat there is no way to delete proxy-arp rule.
On VM expunge we are deleting firewall rules first then static nat rules. This caused the stale proxy-arp
rules.
With this fix adding/deleting proxy arp rule on static nat/PF rule add/del.
to get nic info of 0.0.0.0. To get it, it iterates through all nics and return the last NIC in
the list if it doesn't match with any IP address. In case last NIC doesn't have unicastAddress,
Hyper-V agent will fail to start. We don't need IP address during initialization. It get
initialized with startupcommand later.
last VM from the VPC is deleted on a host
OVS distributed routing: ensure bridge is deleted when last VM from the
VPC is deleted on a host. This fix ensures that bridge is
destroyed.
the folder column. For an smb share the smb credentials are in the query string of the path.
Before adding the path, smb shares query string should be cleaned up.
that case cloudstack was not doing anything and not updating the state of the vms to stopped.
Now the agent returns empty list of hostvmstatereport. Management server will then update the
vm state to stopped (instead of not acting upon the return state).
not created already when OvsVpcPhysicalTopologyConfigCommand update is
recived
Currently if the tunnel creation fails, there is no retry logic. Fix
ensures OvsVpcPhysicalTopologyConfigCommand updates as an opputiunity to ensure
proper tunnels are established between the hosts.
Check if switch name detected from traffic label for management, storage, control traffic is null before falling back to default value.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
We used to create the snapshot after the copy from Secondary Storage,
but it could be that we never use the snapshot.
Now we check if the snapshot exists prior to performing the cloning operation
Since we use qemu-img to copy from RBD to Secondary Storage we no
longer have to force to RAW images, but can stick with QCOW2
When the snapshot backups are QCOW2 format they can easily be deployed
again when restoring from a backup
The KVMStorageProcessor no longer has a hardcoded if-statement which sets
RBD volumes to RAW, this is now handled in the LibvirtStorageAdapter
The Management Server still sends QCOW2 as format. That's a fix for later.
templatefilter="shared" is used , we see public templates also being
included in the list. This commit reverts listTemplates behavior to 4.3
old logic without using consistent interpretation of list parameters
adopted in new IAM model.
invalid password is provided.
- AccountManager now works using accountId instead of accountType in
following methods too:
- isResourceDomainAdmin()
- isAdmin()
OvsVpcPhysicalTopologyConfigCommand and OvsVpcRoutingPolicyConfigCommand
fix ensures only latest updates are applied (new openflow rules) to the
bidge enabled for distributed routing.
fix mismatch of ovs-host-setup, ovs_host_setup used Libvirt resource and
scripts
plug the nic to OVS bridges created for the tunnel network.
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/OvsVifDriver.java
In case some environments has different performance or we found some commands
would took too long to execute, one global configuration item is introduced to
specify "time out in seconds per one command in aggregation commands".
By default it's 3 seconds. If admin feel it's too long, it can be adjust to as
low as 1 seconds, which runs still well in my machine.
Conflicts:
setup/db/db/schema-430to440.sql
Added a new flag 'checkBeforeCleanup' to StopCommand based on which check is done to see if VM is running in HV host.
If VM is running then in this case it is not stopped and the operation bails out.
Also modified the MS code to call the StopCommand with appropriate value for the flag based on the context.
Currently it is only set to 'true' when called from the new vmsync logic based on powerstate of VM. For rest it
is set to 'false' meaning no change in behaviour.
This reduces the amount of time and storage it takes dramatically. We no longer
do a full copy, but a sparse copy. The destination image is still in RAW
format, but we only copy over used blocks.
Qemu is also better in doing this then us doing it in Java code.
Otherwise a RBDException will be thrown with the message that the snapshot
isn't protected.
modified: plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
clusers
across the poll members an internal network created is visible to al the
members but bridge is not necessariliy created. This fix enables
plugging a temp VIF connected to internal network to dom0 and then
unplug-it. this action creates a bridge on the host of the network.
This saves the step of writing to a temporary image in /tmp first before
writing to RBD.
This is possible due to a new version in librbd. With the rbd_default_format
setting we can now force qemu-img to create format 2 RBD images.
This is available since Ceph version 0.67.5 (Dumpling).
introduce 'RegionLevelVpc' as capability of 'Connectivity' service. Add
support for CreateVPCOffering to take the 'regionlevelvpc' as capability
of service 'connectivity'.
introduces new capability 'StretchedL2Subnet' for 'Connectivity'
service. Also add support to createNetworkOffering api to allow
StretchedL2Subnet capablity for the connectivity service.
adds check to ensure 'Connectivity' service provider supports
'StretchedL2Subnet' and 'RegionLevelVpc' capabilities when specified in
createNetworkOffering and createVpcOffering respectivley
enable ovs plug-in to support both StretchedL2Subnet and RegionLevelVpc
capabilities
make zone id optional parameter in createVpc, zone id can be null only
if vpc offfering supports region level VPC
in region level vpc, let the network/tier to be created in any zone of
the region
keep zoneid as required param for createVpc
skip external guest network guru if 'Connectivy' service is present in
network offering
fix build break in contrail manager
permit VM's to be created in different zone that in which network is
created if the network support streched L2 subnet
add integration tests for region level VPC
rebase to master
Conflicts:
setup/db/db/schema-430to440.sql
Conflicts:
api/src/org/apache/cloudstack/api/ApiConstants.java
engine/schema/src/com/cloud/network/vpc/VpcVO.java
setup/db/db/schema-430to440.sql
Add executeInVR() with timeout interface to VirtualRouterDeployer
AggregationControlCommand with Action.Finish may take longer than normal command
since it would execute all the commands in one execution, and it may result in
SSH timeout for SshHelper or other mechanism communicate with VR.
Introduce an new executeInVR() interface with added timeout period for waiting
FinishAggregationCommand to complete execution.
to flow rules and applies them on the bridge
add event subscriber in OvsTunnelManager, that listens to
replaceNetworkAcl events. On event sends the updated policy info to all
the hosts in the VPC
- get the hosts on which VPC spans given vpc id
- get the VM's in the VPC
- get the hosts on which a network spans
- get the VPC's to which a hosts is part of
- get VM's of a VPC on a hosts
introduces capability to build a physical toplogy representation of a
VPC. This json file is encapsulated in
OvsVpcPhysicalTopologyConfigCommand, and is used to send full topology
to hypervisor hosts. On hypervisor this json config can be used to setup
tunnels, configure bridge, add flow rules etc
Ovs GURU, to use different broasdcast scheme VS://vpcid.gerkey for the
networks in VPC that use distributed routing
each VIF and tunnel interface to carry the network UUID in other/options
config
storage pool (SMB) and attached to a running vm can be live migrated to another shared storage
pool. Also a vm and its volumes can be live migrated to another host and storage pool respectively.
2) Corrected some logging in MidoNetPublicNetworkGuru - removed .toString method call on the objects in the log body as toString is called on the object by default when use log4j
CLOUDSTACK-4762 : Enabling VGPU support for XenServer.
This feature is to enable the GPU-passthrough and vGPU functionality,
with the help of this feature, admins/users will be able to leverage
the GPU graphics unit power by deploying a virtul machine with GPU or
vGPU support or by changing the service offering of an existing VM
at any later point of time. There GPU/vGPU enabled VMs are able to run
graphical applications.
For now, this feature is only supported with XenServer hypervisor but
can be extended to add the support of other hypervisors.
Moving default transport for console proxy, SSVM to http.
See
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Realhost+IP+changes
for more info.
jlk ported Amogh's patch for 4.3 to master - code base is different
enough that patch has multiple issues.
Author: Amogh Vasekar <Amogh Vasekar <amogh.vasekar@citrix.com>
Signed-off-by: John Kinsella <jlk@stratosec.co> 1394398017 -0700
In vCenter 5.5, once a volume is migrated the VMDKs are renamed to match the name of the VM.
Update volume path for every volume belonging to the VM to the corresponding new disk filename.
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
plugins/hypervisors/vmware/src/org/apache/cloudstack/storage/motion/VmwareStorageMotionStrategy.java
In vCenter 5.5, once a volume is migrated the VMDKs are renamed to match the name of the VM.
If a volume has been renamed upon migration update its volumePath to that of the new disk filename.
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
vmware-base/src/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java
With VirtIO enabled on KVM. FreeBSD 10 supports VirtIO for both the
network and the disks. This frees us from IDE and E1000 which should
also improve performance.
By default all network disks are in RAW format. Gluster works fine with
QCOW2 which has some advantages.
Disks are by default in QCOW2 format. It is possible to run into
a mismatch, where the disk is in QCOW2 format, but QEMU gets started
with format=raw. This causes the virtual machines to lockup on boot.
Failures to start a virtual machine can be verified by checking the log
of the virtual machine, and compare the output of 'qemu-img info'.
In /var/log/libvirt/qemu/<VM>.log find the URL for the drive:
-drive file=gluster+tcp://...,format=raw,..
Compare this with the 'qemu-img info' output of the same file, mounted
under /mnt/<pool-uuid>/<img-uuid>:
# qemu-img info /mnt/<pool-uuid>/<img-uuid>
...
file format: qcow2
...
This change makes passes the format when creating a disk located on RBD
(RAW only) and Gluster (QCOW2).
Signed-off-by: Niels de Vos <ndevos@redhat.com>
The support for Gluster as Primary Storage is mostly based on the
implementation for NFS. Like NFS, libvirt can address a Gluster environment
through the 'netfs' pool-type.
return null. Changed to retrieve the first entry in the map.
Removed the ExecutionException try catch, this would prevent the
unittest from giving accurate information on exceptions. Avoid catching
checked exceptions in a unittest, use expected keyword on @Test instead.
PrepareForMigrationCommand, so that destination hypervisor can
mount pool. This further exposed an issue for KVM where iso
was not getting cleaned up upon successful migration, fixed as well.
When project/domains are created, our plugin code won't get invoked. Our
contrail plugin uses Event infrastructure provided by cloudstack to
receive these events and handle them accordingly. It is must to create
domains/projects before creating a virtual network/vm object in contrail
implementation. Hence our plugin must need a way to get notified about
project/domain events.
By default only the Integers between -128..127 are cached (unless overridden by java.lang.Integer.IntegerCache.high system property)
If the inbound or outbound values are higher, the reference comparison won't work.
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
- minor resource leak cleaned up
- cpu-speed reading method extracted
- test added
- logging added in case of exception
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Replaced HttpClient#execute(HttpUriRequest) with
HttpClient#execute(HttpUriRequest,ResponseHandler<T>).
The former requires extra EntityUtils#consume(HttpEntity).
This saves us a lot of code and libvirt is probably a better
place to do this.
libvirt-java now has the support we want, so we can now resize volumes
with libvirt.
(C)LVM volumes can't be resized using libvirt, so we have to
invoke a resize script for that.
By default the client_mount_timeout setting in librados is 300 seconds,
but that causes the connect to the Ceph cluster to block for 5 minutes
if the Ceph cluster is not available.
This patch is not ideal, but it mitigates the problem for now.
At a later point all this librados/librbd code should go back to libvirt
again, but the current versions of libvirt in the distributions are
to old for all the features we require.
For now this should prevent the CloudStack agent blocking for 5 minutes
when the Ceph cluster isn't available.
This is also tracked at the Ceph tracker: http://tracker.ceph.com/issues/6507
To obtain network read/write statistics, multiply sample duration with the
average of the particular performance metric obtained over the sample period.
vxlan code. Users can set a physical network to isolation type 'vxlan',
put public traffic on that physical network, and it will still attempt
to use 'vlan' isolation on the KVM hosts. This is going to be an issue
for other isolation types as well, but I'm not familiar with them, so
I'm just fixing vxlan for now.
Added fix for exception and listing. Mentioned details under bug.
Post the fix, simulator works fine.
Signed-off-by: Santhosh Edukulla <Santhosh.Edukulla@citrix.com>
Signed-off-by: Koushik Das <koushik@apache.org>
wilderrodrigues
Daan Hoogland
Funs
Daan Hoogland
Rohit Yadav
Antonio Fornie
Hugo Trippaers
Rajani Karuturi
Sanjay Tripathi
Likitha Shetty
Pierre-Yves Ritschard
Laszlo Hornyak
punith-cloudbyte
Jayapal
Marcus Sorensen
Mike Tutkowski
Santhosh Edukulla
Damodar
Rajesh Battala
Anthony Xu
Sateesh Chodapuneedi
Koushik Das
Saksham Srivastava
Wei Zhou
Kishan Kavala
Sudhansu
Logan Barfield
Wei Zhou
Harikrishna Patnala
Wido den Hollander
Anshul Gangwar
Bharat Kumar
Edison Su
Frank Zhang
amoghvk
punith-cloudbyte
Nitin Mehta
Murali Reddy
Devdeep Singh
David Nalley
imduffy15
Fred Clift
Vania Xu
Kishan Kavala
Daniel Vega
Wido den Hollander
Min Chen
Ritu Sabharwal
Sheng Yang
Will Stevens
Suresh Ramamurthy
Marcus Sorensen
Joris van Lieshout
Kelven Yang
ynojima
Vania Xu
Sujaya Maiyya
Girish chaudhari
Tim Mackey
Sachchidanand Vaidya
Prachi Damle
Chris Suich
Ding Yuan
Kishan Kavala
sbalineni
Abhinandan Prateek
Alex Huang
Santhosh Edukulla
Alena Prokharchyk
John Kinsella
Mandar Barve
Niels de Vos
Miguel Ferreira
Alex Hitchins
Sander Botman
Darren Shepherd
Syed Ahmed
Hiroaki KAWAI