CLOUDSTACK-8964: Ovm3HypervisorGuru answer fix and snapshot to volume implementation.some clean up of code after the fix from #975
* pr/1015:
CLOUDSTACK-8964 side effect isolation extract side effect away to emphasize the main commandDelegation objective
host delegation logging for XenServerGuru.java
Getting volume from snapshot working again... odd
simple change to prevent failure and keep OVM3 snapshots working
This closes#975
Signed-off-by: Remi Bergsma <github@remi.nl>
These were lacking, but this helper is used in various places
inside the KVM code.
Some simple tests to verify the helper is doing what we expect it
to do.
CLOUDSTACK-8826: XenServer - Use device id passed as part of attach volume API properly
If device id passed as part of API and available then use it otherwise fallback on XS to automatically assign one.
For ISO device id used is 3 and it is processed before any other entry to avoid conflict.
Signed-off-by: Koushik Das <koushik@apache.org>
If device id passed as part of API and available then use it otherwise fallback on XS to automatically assign one.
For ISO device id used is 3 and it is processed before any other entry to avoid conflict.
Replaced all occurences of Charset.forName(UTF-8) with StringUtils.getPreferredCharset().
* pr/825:
Replaced all occurences of Charset.forName(UTF-8) with StringUtils.getPreferredCharset().
Signed-off-by: Daan Hoogland <daan@onecht.net>
sysctl: don't modify /etc/sysctl.confTo configure firewall rules, CloudStack modifies `/etc/sysctl.conf` and
execute those modifications. This may be harmful for several reasons:
1. `/etc/sysctl.conf` may be managed by some configuration management
system. Such a system will constantly restore the previous version.
2. `/etc/sysctl.conf` may contain additional properties that have been
changed later by some system administrator (for example, once a
firewall has been configured, forwarding may have been activated
while it is disabled in `/etc/sysctl.conf`). Executing the file
again at a later time may disrupt the system.
3. Entries are added again and again. `/etc/sysctl.conf` will contain
the same directives repeated several times.
Using a configuration file is not needed as `sysctl` is able to directly
modify sysctl values with `-w` flag.
Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
* pr/776:
sysctl: don't modify /etc/sysctl.conf
Signed-off-by: Wido den Hollander <wido@widodh.nl>
To configure firewall rules, CloudStack modifies `/etc/sysctl.conf` and
execute those modifications. This may be harmful for several reasons:
1. `/etc/sysctl.conf` may be managed by some configuration management
system. Such a system will constantly restore the previous version.
2. `/etc/sysctl.conf` may contain additional properties that have been
changed later by some system administrator (for example, once a
firewall has been configured, forwarding may have been activated
while it is disabled in `/etc/sysctl.conf`). Executing the file
again at a later time may disrupt the system.
3. Entries are added again and again. `/etc/sysctl.conf` will contain
the same directives repeated several times.
Using a configuration file is not needed as `sysctl` is able to directly
modify sysctl values with `-w` flag.
Signed-off-by: Vincent Bernat <Vincent.Bernat@exoscale.ch>
Use host.reserved.ram.mb agent property to modify total system RAM
before reporting to management server.
Remove dom0ram variable and its min/max calculation. Just reserve a
default of 1GB, unless overridden by host.reserved.mem.mb property.
CLOUDSTACK-8762: Check to confirm disk activity before starting a VMImplements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* pr/753:
CLOUDSTACK-8762: Check to confirm disk activity before starting a VM
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Guys, can you review it? things need to be discussed:
(1) this supports KVM/QCOW2 only. Anyone want to implement for other Hypervisor/format ?
(2) The original data volume (on primary storage) will be removed.
(3) The script uses the default timeout in libvirtComputingResource. Do we need to add one in global configuration (like copy.volume.wait or backup.snapshot.wait, create.volume.from.snapshot.wait)
(4) In scripts/storage/qcow2/managesnapshot.sh, I use "qemu-img convert -f qcow2 -O qcow2" to copy the snapshot from secondary to primary (hence there is no base image file), instead of "cp -f", this is because convert is faster than cp in my testing.
* pr/732:
CLOUDSTACK-5863: revert volume snapshot for KVM/QCOW2
Signed-off-by: Wei Zhou <w.zhou@tech.leaseweb.com>
Made following fixes in simulator
- Support for ScaleVmCommand/NetworkRulesVmSecondaryIpCommand in resource layer
- Added support for scaling up a running VM in simulator
- Fixed some method names not following convention
In order to test PR #725 using simulator some of these changes are needed.
Based on the way HV check is present in the scale VM API, had to explicitly put simulator related check to allow support. The ideal way would be to remove all these HV specific check from code and made them some configuration (by putting them in hypervisor_capabilities table in DB). But that will be a bigger effort outside the scope of this PR.
Signed-off-by: Koushik Das <koushik@apache.org>
This reverts commit cd7218e241, reversing
changes made to f5a7395cc2.
Reason for Revert:
noredist build failed with the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloud-plugin-hypervisor-vmware: Compilation failure
[ERROR] /home/jenkins/acs/workspace/build-master-noredist/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[484,12] error: non-static variable logger cannot be referenced from a static context
[ERROR] -> [Help 1]
even the normal build is broken as reported by @koushik-das on dev list
http://markmail.org/message/nngimssuzkj5gpbz
Implements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
messages: Removed unused agent command AttachVolumeCommand and corresponding answer AttachVolumeAnswer
These types are not used anywhere in the orchestration layer.
Signed-off-by: Koushik Das <koushik@apache.org>
Default to notify only script to handle non-CLVM/QCOW cases.This relates to [CLOUDSTACK-8732](https://issues.apache.org/jira/browse/CLOUDSTACK-8732)
Before this commit the call to `getResizeScriptType` would throw an exception (earlier versions returned `null`, which was fine) - this caused the RBD case to fail. By changing the default to notify only we fix the case for any non-CLVM and non-QCOW cases, too.
This is RBD for now, but this should extend to new storage types supported by Libvirt natively in future.
This is my first attempted contribution: I can see a case for adding RBD logic to the actual getResizeScriptType call, too, but I felt that putting it `LibvirtResizeVolumeCommandWrapper.java` kept the special-casing of RBD (and comments about that) in one place.
### Caveat:
With Libvirt 1.2.2 this actually doesn't do the right thing - but it does do what the documentation *says* should be the right thing, so I'm going to test if this is a Libvirt bug which is fixed in a later version.
(To make it work I need to execute something like:
virsh blockresize --path vda --size 100G i-7-44-VM
where vda is the path as far as the *guest* is concerned, and not an `rbd/` path - which *should* work, but doesn't.)
* pr/696:
Default to notify only script to handle non-CLVM/QCOW cases.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Support for ScaleVmCommand/NetworkRulesVmSecondaryIpCommand in resource layer
- Added support for scaling up a running VM in simulator
- Fixed some method names not following convention
CLOUDSTACK-8443: Support CentOS 7 as KVM hypervisorThis adds support for CentOS 7 as a KVM hypervisor. As discussed in the Jira issue, the root cause of the issue was the `cloudstack-setup-agent` script that added this line to `/etc/libvirt/qemu.conf`:
`cgroup_controllers=["cpu"]`
On CentOS 6 this 'works', but since by default no cgroups are mounted, this setting is ignored by `libvirtd`. On CentOS 7 is does not work, because the `cpu` and `cpuacct` are 'co-mounted'. This simpy means you cannot use one without the order. The config line above instructs `libvirtd` to only use the `cpu` cgroup, which it can't.
There was code added to make sure the `cpu` cgroup was not co-mounted. It did this with a shell script, but after a reboot it was gone. It wasn't reliable and not even a solution at all. I went ahead and removed the code, because the co-mounted cgroups work fine. The real issue was the config line we put in `/etc/libvirt/qemu.conf` so much better to fix that instead.
The default for `libvirtd` is to use only cgroups that are mounted. When you remove the 'cgroup_controllers' line, the default kicks in. According to the `qemu.conf` file, the default is:
`cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]`
Then again, if it's not mounted, it's not used. So, that's a nice way to control it.
This works just fine for both RHEL/CentOS 6 and RHEL/CentOS 7.
Along the debugging, I came across some small issues here and there which I also fixed in this PR.
If you test this, be sure to build RPMs from this branch and install those on the KVM hypervisor.
At Schuberg Philis we run KVM hypervisors on CentOS 7 for months with these fixes. Now I took the time to fix the issues in CloudStack, rather than work around them.
@bhaisaab @karuturi could you please review this? Thanks!
Please note:
The next step is to fix CLOUDSTACK-8625 (Systemd profile for KVM Agent) as currently a sysvinit script is still used. Although it works, we do not have any systemd benefits. As it is a separate issue, I'll try to address it in a separate PR.
* pr/731:
CLOUDSTACK-8443: detect CentOS 7.x as RHEL 7 alike
CLOUDSTACK-8443: mention the correct logfile
CLOUDSTACK-8443: display the right hypervisor type
CLOUDSTACK-8443: don't try to fix co-mounted cgroups
Signed-off-by: Remi Bergsma <github@remi.nl>
This setting works on CentOS 6 / RHEL 6 but does nothing, as
"cpu" cgroup is not mounted. On CentOS 7 / RHEL 7 systemd does
mount cgroups and "cpu" is co-mounted with "cpuacc". Hence, if
we specify "cpu" then this results in an error because it can
only use them both, or none.
By removing the setting, we rely on the default of qemu, which
is:
cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]
Only if they are really mounted, they will be used. So, this will
work on both version 6 and 7.
The 'fix script' didn't work well, as after a reboot you'd still have qemu
throwing errors. Now we can handle the co-mountedcgroups.
Removed duplicate code in CitrixResourceBase and its subclassesRemoved unnecessary duplicated code by having the body of the getPatchFiles method only in the CitrixResourceBase superclass. Given that all of its implementations consisted of the same code except for the path which contains the necessary file for that implementation. An abstract method getPatchFilePath was created in the CitrixResourceBase superclass so that each implementation may return the path containing the specific file needed by that implementation.
Test cases were created for each implementation, simple as they may be. One assert is made to verify that the path returned by each implementation corresponds to the path that was previously specified on each getPatchFiles implementation.
* pr/700:
Removed duplicate code in CitrixResourceBase.getPatchFiles
Signed-off-by: Remi Bergsma <github@remi.nl>
This tries to avoid cleaning by a device name.
This closes#718
(cherry picked from commit 74f697a2dd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Changed methodnames according to Nic.java refactor.
Fixed NicVO.java due to regression from Nic.java refactor.
Fixed VmWareGuru.java after Nic.java refactor.
See issue CLOUDSTACK-8736 for ongoing effort to clean up network code.
@cristofolini comments:
Removed unnecessary duplicated code by having the body of the
getPatchFiles method only in the
CitrixResourceBase superclass. Given that all of its implementations
consisted of the same code
except for the path which contains the necessary file for that
implementation. An abstract method
getPatchFilePath was created in the CitrixResourceBase superclass so
that each implementation may
return the path containing the specific file needed by that
implementation.
Test cases were created for each implementation, simple as they may be.
One assert is made to verify
that the path returned by each implementation corresponds to the path
that was previously specified
on each getPatchFiles implementation.
removed trailing whitespace
Cloudstack 8656: do away with more silently ignoring exceptions.a lot of messages added.
some restructuring for test exception assertions and try-with-resource blocks
* pr/654: (29 commits)
CLOUDSTACK-8656: more logging instead of sysout
CLOUDSTACK-8656: use catch block for validation
CLOUDSTACK-8656: class in json specified not found
CLOUDSTACK-8656: removed unused classes
CLOUDSTACK-8656: restructure of tests
CLOUDSTACK-8656: reorganise sychronized block
CLOUDSTACK-8656: restructure tests to ensure exception throwing
CLOUDSTACK-8656: validate the throwing of ServerApiException
CLOUDSTACK-8656: logging ignored exceptions
CLOUDSTACK-8656: try-w-r removes need for empty catch block
CLOUDSTACK-8656: try-w-r instead of clunckey close-except
CLOUDSTACK-8656: deal with empty SQLException catch block by try-w-r
CLOUDSTACK-8656: unnecessary close construct removed
CLOUDSTACK-8656: message about timed buffer logging
CLOUDSTACK-8656: message about invalid number from store
CLOUDSTACK-8656: move cli test tool to separate file
CLOUDSTACK-8656: exception is the rule for some tests
CLOUDSTACK-8656: network related exception logging
CLOUDSTACK-8656: reporting ignored exceptions in server
CLOUDSTACK-8656: log in case we are on a platform not supporting UTF8
...
Signed-off-by: Remi Bergsma <github@remi.nl>
Before this commit the call to `getResizeScriptType` would throw
an exception (earlier versions returned `null`, which was fine) -
this caused the RBD case to fail. By changing the default to
notify only we fix the case for any non-CLVM and non-QCOW cases.
This is RBD for now, but this should extend to new storage
types suppored by Libvirt natively in future.
When dumping XML use appropriate flags:
1, VIR_DOMAIN_XML_SECURE (dump security sensitive information too)
8, VIR_DOMAIN_XML_MIGRATABLE (dump XML suitable for migration)
Source:
https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainXMLFlags
This fixes CVE 2015-3252: VNC password lost during VM migration across KVM
hosts. The issue is also seen when a VM is rebooted.
(cherry picked from commit cb2aca7516)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* pr/547:
CLOUDSTACK-8601. VMFS storage added as local storage can be re-added as shared storage. Fail addition of a VMFS shared storage pool in case it has already been added as local storage in CS.
Signed-off-by: Mike Tutkowski <mike.tutkowski@solidfire.com>
On NFS we write a heartbeat, but without those we can not safely
fence off a host.
If we fence without knowing about a heartbeat we can cause a split-brain situation.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
- Make sure the new password replaces the old one in the queue
- Updated the patch files for XenServer
- Updated the script path on LibvirtComputing class
- Adding update_host_passwd to VRScripts
- Add implementation to CitrixUpdateHostPasswordCommandWrapper
- Improve testUpdateHostPasswordCommand() unit test on CitrixRequestWrapperTest
- Adding update_host_passwd.sh script
- Adding the host IP address as an instance variable on UpdateHostPasswordCommand
- Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
- Make sure doUpdateHostPassword() doesn't get called if flag is set to false
- Do not update XenServer hosts if the cluster ID is not informed
This closes#559
- Adding more unit tests as suggested by Daan and Rajani.
- Mocking SshHelper and the Script classes, used by Citrix and Libvirt resources respectively.
- Changed location of the update_host_passwd script
- Updated the patch files for XenServer
- Updated the script path on LibvirtComputing class
- Removed the hostIP from the LibvirtUpdateHostPasswordCommandWrapper execute() method
- Adding update_host_passwd to VRScripts
- Add accessor method to host password on CitrixResourceBase
- Add implementation to CitrixUpdateHostPasswordCommandWrapper
- Improve testUpdateHostPasswordCommand() unit test on CitrixRequestWrapperTest
- Add line to patch files on xenserver directory
Concerning the LibVirt change:
- I forgot to assing the return of the getDefaultHypervisorScriptsDir() method to the hypervisorScriptsDir variable
- Modifying the LibvirtUpdateHostPasswordCommandWrapper in order to execute the script on the host
- Adding the script path to LibvirtComputingResource
- Adding the host IP address as an instance variable on UpdateHostPasswordCommand
- Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
- Make sure the doUpdateHostPassword() method returns the result from the answer, not always true
- Added the LibvirtUpdateHostPasswordCommandWrapper class, which will handle the changes in the KVM agent
That's will be further implemented in the future. The command will avoid the agent to complain about Unsupported command
- Added a test to make sure the current implementation of the comment works
- If changes in the future, it will also require changes. In that way, we make sure nobody will break it
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
This closes#527
CLOUDSTACK-3317 - DVS does not support management\storage network
Added support for Management and Storage Network traffic over VMware DVS in CloudStack deployments. Also added support for storage VLAN over dvPortGroup.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
CLOUDSTACK-3317 - DVS does not support management\storage network
Use non-zero dvport count while updating dvportgroups of system traffic.
Updated configuration compare logic to avoid update dvportgroup operation unless required.
This would help improve speed in vm/network deployment as the update calls would reduce. Also improved logging.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
Adding unit tests in class HypervisorHostHelperTest
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
Added license header to new file being added to repo/branch.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
Assertion is not used in runtime, correct way is throw and handle exception without killing app
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
This closes#362
VmwareResource.java:315, MS_SHOULD_BE_FINAL, Priority: High
com.cloud.hypervisor.vmware.resource.VmwareResource.s_serviceContext isn't final but should be
VmwareResource.java:331, MS_SHOULD_BE_FINAL, Priority: High
com.cloud.hypervisor.vmware.resource.VmwareResource.s_powerStatesTable isn't final but should be
Signed-off-by: Daan Hoogland <daan.hoogland@gmail.com>
This closes#363
KVM hosts which are actuall up, but if their agents are shutdown should be put
in disconnected state. This would avoid getting the VMs HA'd and other commands
such as deploying a VM will exclude that host and save us from errors.
The improvement is that, we first try to contact the KVM host itself. If it fails
we assume that it's disconnected, and then ask its KVM neighbours if they can
check its status. If all of the KVM neighbours tell us that it's Down and we're
unable to reach the KVM host, then the host is possibly down. In case any of the
KVM neighbours tell us that it's Up but we're unable to reach the KVM host then
we can be sure that the agent is offline but the host is running.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
This closes#340
When executing the tests in an environment where Libvirt is also installed, it
caused errors.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This closes#342
Maven was bulding the project and executing the tests just fine because it uses "-sourcepath". However,
with IDEs and javac it would fail.
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
This closes#317
Pull average Cpu util report between polling intervals instead of since boot
instead of using values since uptime
(cherry picked from commit 04176eaf17)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
This closes#297
Passing the file argument to the xml break for EL 7.1, the fix removes
the argument as just passing rombar='off' with its file arg to be empty string.
This closes#290
(cherry picked from commit aafa0c80b3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
EL7 has a different output to 'free', use /proc/meminfo instead of a tool to be
more consistent across distros
(cherry picked from commit 212a05a345)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
Removing real IPs from the tests because they cause a long running time for LibvirtComputingResourceTest
- In a local machine it takes 1.977s, but in a KVM test environment it's taking 257.879 sec
Fixing typo on LibvirtRequestWrapper
- Replace linbvirtCommands by libvirtCommands on LibvirtRequestWrapper
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This closes#255
- The test was okay, but when running in an environment where a /root/.ssh/id_rsa existed, it would return true then fail
- We now mock the calls to methods that return the key paths, instead of relying in the static variables
- Adding LibvirtNetworkElementCommandWrapper and LibvirtStorageSubSystemCommandWrapper
- 2 unit tests added
- KVM hypervisor plugin with 22.2% coverage
I also refactored the StorageSubSystemCommand interface into an abstract class
- Remove the pseudo-multiple-inheritance implementation
- The StorageSubSystemCommand was an interface, not related to the Command class
and its implementation were extending the Command class anyway. The whole structure is better now.
- Addin LibvirtPvlanSetupCommandWrapper
- 6 unit tests added
- KVM hypervisor plugin with 21% coverage
From the 6 tests added, 2 were extra tests to increase the coverage of the LibvirtStopCommandWrapper
- Increased from 35% to 78.7%
- Adding LibvirtCopyVolumeCommandWrapper
Refactoring the LibvirtUtilitiesHelper
- Changing method name
Did not add any test to this commit due to the refactor mentioned abot.
Will proceed and add the tests
i# Please enter the commit message for your changes. Lines starting
- Gave it a better, more suggestive, name since I now added other methods to the class.
- It makes easier to mock objects and get a better coverage of the classes
- Adding LibvirtBackupSnapshotCommandWrapper, LibvirtCreatePrivateTemplateFromVolumeCommandWrapper and LibvirtManageSnapshotCommandWrapper
- 3 unit tests added
- KVM hypervisor plugin with 18.3% coverage
Less tests added to those classes because the code is quite complex and way too long.
The tests added are just covering the new flow, to make sure it works fine. I will come back to those classes later.
- Adding LibvirtOvsDestroyBridgeCommandWrapper, LibvirtOvsSetupBridgeCommandWrapper
- 4 unit tests added
- KVM hypervisor plugin with 13.9% coverage
More tests added to cover LibvirtPrepareForMigrationCommandWrapper
- Coverage of this wrapper broght from 37% to 90.6%
- 4 new tests added
- Adding LibvirtCheckConsoleProxyLoadCommandWrapper, LibvirtConsoleProxyLoadCommandWrapper, LibvirtWatchConsoleProxyLoadCommandWrapperand CitrixConsoleProxyLoadCommandWrapper
- 2 unit tests added
- KVM hypervisor plugin with 12% coverage
Refactored the CommandWrapper interface in order to remove the esecuteProxyLoadScan, which is now
implemented bu subclasses.
- Adding LibvirtGetHosStatsCommandWrapper
- 1 unit test added
- KVM hypervisor with 10.5% coverage
Tests are a bit limited on this one becuause of the current implementation. Would clean it up later in a separate branch
- Adding LibvirtGetVmStatsCommandWrapper
- 3 unit tests
Refactored the LibvirtConnectiobn by surrounding it with an wrapper.
- Make it easier to cover the static/native calls
- Added better coverage to StopCommand tests
- Adding LibvirtStopCommandWrapper
- LibvirtRequestWrapper
- 1 unit tests
Refactored the RequestWrapper to make it better.
- Changes also applied to the CitrixRequestWrapper
Linux kernel supports vmxnet3, allowing it in KVM plugin would allow us to
run ESX hosts on KVM hosts using CloudStack with vmxnet3 nic which can be
passed as VM's nicAdapter detail
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit e02d787f30)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This improvements checks for "guest.cpu.features" property which is a space
separated list of cpu features that is specific for a host. When added, it
will add <feature policy='require' name='{{feature-you-listed}}'/> in the
<cpu> section of the generated vm spec xml.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit ea7fd37783)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The only artifact resolved from libvirt.org was org.libvirt:libvirt:0.5.1
this artifact is now available from maven's default central repository
This closes#180
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
As suggested by Wido on the dev ML changing the repo to eu.ceph.com to avoid
build failures. Will revert if ceph.com is up again.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit c9fd57fff3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CentOS 7 does not ship with ifconfig anymore. We should use ip commands instead.
This also works on older versions, like CentOS 6 and Ubuntu 12.x/14.x, that we
support.
This closes#165
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- use sticky chmod 1777 on the mountpoint
- remove dead code
(cherry picked from commit eea716b791)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
1. provide compatibility with the Big Cloud Fabric (BCF) controller
L2 Connectivity Service in both VPC and non-VPC modes
2. virtual network terminology updates: VNS --> BCF_SEGMENT
3. uses HTTPS with trust-always certificate handling
4. topology sync support with BCF controller
5. support multiple (two) BCF controllers with HA
6. support VM migration
7. support Firewall, Static NAT, and Source NAT with NAT enabled option
8. add VifDriver for Indigo Virtual Switch (IVS)
This closes#151
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- With the changes added by the rVPC work, the bump priority became deprecated.
This commit includes a refactor to get it removed from the following resources:
* Java classes
* domain_router table - removing the is_priority_bumpup column
* Fixing unit tests
All changes were tested with:
XenServer 6.2 running under our VMWare zone
CloudStack Management Server running on MacBook Pro
MySql running on MackBook Pro
Storage Type: Local
Started the refactor of the XenServer56Resource class
- Unit test added
Changing the CitrixRequestWrapper in order to cope with multiple resources classes and commands
Still have to remove few methods form CitrixResourceBase
- Unit tests added
In the executeRequest I needed to keep the following:
// We need this one because the StorageSubSystemCommand is from another hierarchy.
if (cmd instanceof StorageSubSystemCommand) {
return storageHandler.handleStorageCommands((StorageSubSystemCommand) cmd);
}
- Added basic tests
- Changed the way RebootCommand gets called from RebootRouterCommand
- Made a couple of methods public in the CitrixResourceBase and its subclasses
Sonarcube is great, but has no context and can be wrong, in this
case File.separator is nice if you're running platform independent,
the agent on OVM3 is however NOT platform independent and will break
if we feed it windows separators.
The copy command reply can have null size returned, so check and set values
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 53ca0b1861)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fix: RTNETLINK errors
- Management Server health check trying to create already existing interface
- Changes on update_config.py, cs_guestnetwork.py, merger.py
Fix: replace RRouTER_LOG in the CsRedundant.py per log file location
Fix: Guest Net address association during Router restart
- Changes on NicProfileHelper, NicProfileHelperImpl
Fix: aggregationExecution() method on VirtualNetworkApplianceManagerImpl
- Do not send an AggregationControlCommand to a non-configured router
Some classes have been formatted.
Added a source column to the user table.
Source now has only two values UNKNOWN,LDAP with UNKNOWN being the
default and is an enum is com.cloud.User.
When the source is UNKNOWN, the old method of authenticating against all
the available authenticators is used. If a source is available, only
that particular authenticator will be used.
added overloaded methods in AccountService to createUserAccount and
createUser with source specified.
(cherry picked from commit 5da733072e)
MigrateVMWithVolumes-
1. If ESXi host version is below 5.1, ensure destination datastore(s) is mounted on the source host, then migrate the storage and then finally migrate the VM.
If destination storage(s) is not mounted on the source host,
- In case of NFS storage mount the storage(s).
- In case of VMFS storage fail the request for migration.
2. If EXi host version is 5.1 or above, simultaneously migrate the VM and its storage to the destination host and storage(s) respectively for both NFS and VMFS storage.
This is a plugin that puts in ovm3 support ranging from 3.3.1 to 3.3.2. Basic
functionality is in here, advanced networking etc..
Snapshots only work when a VM is stopped now due to the semantics of OVM's raw
image implementation (so snapshots should work on a storage level underneath the
hypervisor shrug)
This closes#113
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
When there is large size VR configuration (aggregate commands) copying data to VR using vmops plugin was failed
because of the ARG_MAX size limitation. The configuration data size is around 300KB.
Updated this to create file in host by scp with file contents. This will create file in host.
Then copy the file from the host to VR using hte vmops createFileInDomr method.
In host file get created in /tmp/ with name VR-<UUID>.cfg, once it copied to VR this file will be removed.
Refactored to use the XPatch expressions to check the generated domain xml rathern than string comparison.
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
Earlier host addition of multiple hosts with local storage failed due to
same local storage UUID being used where the storage path is same.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit bf17f640c6)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixed some issues with the polling logic to handle scenarios when SSVM is destroyed/down.
Also changed the status of the volume_store_ref entries so that subsequent operations work fine.
KVMStoragePoolManager is a singleton in practice, any plugin
or extension of LibvirtComputingResource will need to act on
the specific instance of KVMStoragePoolManager that LibvirtComputingResource
has initialized. Therefore, expose this variable for those who
wish to call storage commands from plugins or extensions.
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
1. If a VM by the same name exists on a different cluster in the VMware DC, unregister the existing VM and continue with the VM start.
2. If VM start succeeds, delete VM files associated with the unregistered VM.
3. If VM start fails, re-register the unregistered VM.
For AttachVolume/DetachVolume API command, improve user error message in case of RuntimeException by throwing the exception instead of 'Unexpected Exception'.
on calling GetUploadParamsForTemplate, persisting the metadata to db
validating the account limits and incrementing the appropriate limits
encoded the metadata on management server using preshared key
On default iptables rules are updated to add ACCEPT egress traffic.
If the network egress default policy is false, CS remove ACCEPT and adds the DROP rule which
is egress default rule when there are no other egress rules.
If the CS network egress default policy is true, CS won't configure any default rule for egress because
router already came up to accept egress traffic. If there are already egress rules for network then the
egress rules get applied on VR.
For isolated network with out firewall service, VR default allows egress traffic (guestnetwork --> public network)
Findbugs will report error on this as it is expecting true/false for Boolean value.
But we have diffrent meaning for null so it is false positive case from findbug
This closes#39
Look for a VM in vCenter based on both the vCenter name and CS internal name (required in case 'vm.instancename.flag' is enabled).
During Attach Volume and Volume Migration, for lookup and other operations use VM's name as obtained from vCenter instead of using the name set in the agent command.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
Clearly show if a volume is found and if not, that the pool is being refreshed
and the fetch is tried again.
Due to my commit b53a9dcc9f the chance of a volume
not being found is slightly bigger, but the performance gain is enormous on larger
deployments.
This is why we clearly have to log that we are refreshing the pool information
when a volume is not found.
It could be that a volume is created on host A and a few seconds later host B tries
to access the volume. In that case host B's libvirt doesn't know about the volume
yet and has to refresh the pool before it does.
(cherry picked from commit 4ee82f1f40)
modules can use it.
(cherry picked from commit b6401b04f22b0a5b686c7c477da4c6e0fd18df84)
Conflicts:
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalKickStartServiceImpl.java
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
(cherry picked from commit 39fe766c2b)
Support offline volume resize on ESX by creating a worker VM to attach the unattached volume to and then resize it.
(cherry picked from commit 65ed25b7a6)
While expunging a volume, CS chooses the endpoint to perform delete operation by selecting any host that has the storage containing the volume mounted on it.
Instead, if the volume to be deleted is attached to a VM, the endpoint chosen by CCP should be the host that contains the VM.
(cherry picked from commit f1e3e83bbf)
If CCP thread local context which is used to handle connections to a vCenter is being re-used, validate that the context corresponds to the right vCenter API session.
(cherry picked from commit 6b06970366)
On larger (especially RBD) storage pools this can take a lot of
time slowing operations like creating volumes down.
The getStorageStats command will still ask a pool to be refreshed so
that the management server has accurate information about the storage pools.
On larger deployments, with thousands of volumes in one pool, this should
significantly improve storage related operations
(cherry picked from commit b53a9dcc9f)
cleaning up vmsync changes, checkvirtualmachine command was updated to return the
power state of the vm. The change was missed for Hyper-V. This caused migration to
fail on cloudstack even though it used to succeed on the hypervisor. Updated the
hyper-v agent code to return the cloudstack equivalent power state for check virtual
machine answer.
(cherry picked from commit 5350e61187)
Clearly show if a volume is found and if not, that the pool is being refreshed
and the fetch is tried again.
Due to my commit b53a9dcc9f the chance of a volume
not being found is slightly bigger, but the performance gain is enormous on larger
deployments.
This is why we clearly have to log that we are refreshing the pool information
when a volume is not found.
It could be that a volume is created on host A and a few seconds later host B tries
to access the volume. In that case host B's libvirt doesn't know about the volume
yet and has to refresh the pool before it does.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
While expunging a volume, CS chooses the endpoint to perform delete operation by selecting any host that has the storage containing the volume mounted on it.
Instead, if the volume to be deleted is attached to a VM, the endpoint chosen by CCP should be the host that contains the VM.
If CCP thread local context which is used to handle connections to a vCenter is being re-used, validate that the context corresponds to the right vCenter API session.
On larger (especially RBD) storage pools this can take a lot of
time slowing operations like creating volumes down.
The getStorageStats command will still ask a pool to be refreshed so
that the management server has accurate information about the storage pools.
On larger deployments, with thousands of volumes in one pool, this should
significantly improve storage related operations
cleaning up vmsync changes, checkvirtualmachine command was updated to return the
power state of the vm. The change was missed for Hyper-V. This caused migration to
fail on cloudstack even though it used to succeed on the hypervisor. Updated the
hyper-v agent code to return the cloudstack equivalent power state for check virtual
machine answer.
Fix the bug by removing the command CreateEntityDownloadURLCommand from the host delegation. This results in same ssvm for creating the symlink on ssvm and same public ip being used for generating the url on MS.
1. While destroying a ROOT volume do the lookup of the associated VM under the DC and not just cluster.
2. In case of VMware, during VM start if a volume is being recreated no need to detach the old volume because
we now expunge it immediately and don't wait for the storage cleanup task to run.
During VM start while configuring its disk devices, obtain the matching disk for a volume in storage
using both the volume's path and volume's datastore information.
While adding host to existing cluster which is using Nexus 1000v as a network backend, skip validation of Nexus VSM as it was already done while adding that cluster.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
determined. To exclude hosted system we filter the result on Caption='Virtual Machine'
but this string is locale dependent so it may not not work properly for locales other
than english. To overcome this now we started using ProcessId >= 0 filter
For ResizeVolume API command -
1. If hypervisor resource throws an exception, handle the NPE thrown by the job framework.
2. Improve user error message in case of RuntimeException by throwing the exception instead of 'Unexpected Exception'.
We don't need an external script to investigate the format of the RBD volume,
we only have to ask Libvirt to resize the volume and that will ask librbd to
do so.
1. Fixed JSON response deserialization. While creating a mock a JSON can be passed which will be deserialized into a response object and returned from agent layer.
For e.g. for a mock corresponding to StopCommand, a response like "{"com.cloud.agent.api.StopAnswer":{"result":false,"wait":0}}" can be passed.
2. Ability to mock PingCommand (returned as part of getCurrentStatus() agent method). As a part of this a mocked VM state report can be returned.
For e.g. {"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_hostVmStateReport":{"v-2-VM":{"state":"PowerOn","host":"SimulatedAgent.e6df7732-69b2-429b-9b6a-3e24dddfa2e0"},"i-2-5-VM":{"state":"PowerOff","host":"SimulatedAgent.e6df7732-69b2-429b-9b6a-3e24dddfa2e0"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":3,"contextMap":{},"wait":0}}
in the XenServer pool for GRE tunnel networks
Fix uses XenServer recommended way
Network.other_config:assume_network_is_shared=true
which ensures bridge is created automatically on hosts in the pool for
GRE tunnel networks. Fix also gets rid of error prone custom logic that ensures
bridge is created by plugging a VIF into the dom0 and connected to
GRE tunnel network.
Conflicts:
plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java
xen cluster
XenServer does not create a bridge automatically when VIF from domU is connected
to internal network. So there is logic to force bridge creation by
creating VIF in dom0 connected to GRE tunnel network. But there is no
logic to delete the VIF after bridge gets created. So this fix ensure
VIF is delted when atleast there is one domU VIF connected to the
network.
In situations where libvirt lost the storage pool the KVM Agent will re-create the
storage pool in libvirt.
This could be then libvirt is restarted for example.
The object returned internally was missing essential information like the sourceDir
aka the Ceph pool, the monitor IPs, cephx information and such.
In this case the first operation on this newly created pool would fail. All operations
afterwards would succeed.
to get nic info of 0.0.0.0. To get it, it iterates through all nics and return the last NIC in
the list if it doesn't match with any IP address. In case last NIC doesn't have unicastAddress,
Hyper-V agent will fail to start. We don't need IP address during initialization. It get
initialized with startupcommand later.
last VM from the VPC is deleted on a host
OVS distributed routing: ensure bridge is deleted when last VM from the
VPC is deleted on a host. This fix ensures that bridge is
destroyed.
the folder column. For an smb share the smb credentials are in the query string of the path.
Before adding the path, smb shares query string should be cleaned up.
that case cloudstack was not doing anything and not updating the state of the vms to stopped.
Now the agent returns empty list of hostvmstatereport. Management server will then update the
vm state to stopped (instead of not acting upon the return state).
Check if switch name detected from traffic label for management, storage, control traffic is null before falling back to default value.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
We used to create the snapshot after the copy from Secondary Storage,
but it could be that we never use the snapshot.
Now we check if the snapshot exists prior to performing the cloning operation
Since we use qemu-img to copy from RBD to Secondary Storage we no
longer have to force to RAW images, but can stick with QCOW2
When the snapshot backups are QCOW2 format they can easily be deployed
again when restoring from a backup
The KVMStorageProcessor no longer has a hardcoded if-statement which sets
RBD volumes to RAW, this is now handled in the LibvirtStorageAdapter
The Management Server still sends QCOW2 as format. That's a fix for later.
fix mismatch of ovs-host-setup, ovs_host_setup used Libvirt resource and
scripts
plug the nic to OVS bridges created for the tunnel network.
Conflicts:
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/OvsVifDriver.java
In case some environments has different performance or we found some commands
would took too long to execute, one global configuration item is introduced to
specify "time out in seconds per one command in aggregation commands".
By default it's 3 seconds. If admin feel it's too long, it can be adjust to as
low as 1 seconds, which runs still well in my machine.
Conflicts:
setup/db/db/schema-430to440.sql
Added a new flag 'checkBeforeCleanup' to StopCommand based on which check is done to see if VM is running in HV host.
If VM is running then in this case it is not stopped and the operation bails out.
Also modified the MS code to call the StopCommand with appropriate value for the flag based on the context.
Currently it is only set to 'true' when called from the new vmsync logic based on powerstate of VM. For rest it
is set to 'false' meaning no change in behaviour.
This reduces the amount of time and storage it takes dramatically. We no longer
do a full copy, but a sparse copy. The destination image is still in RAW
format, but we only copy over used blocks.
Qemu is also better in doing this then us doing it in Java code.
Otherwise a RBDException will be thrown with the message that the snapshot
isn't protected.
modified: plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java
clusers
across the poll members an internal network created is visible to al the
members but bridge is not necessariliy created. This fix enables
plugging a temp VIF connected to internal network to dom0 and then
unplug-it. this action creates a bridge on the host of the network.
This saves the step of writing to a temporary image in /tmp first before
writing to RBD.
This is possible due to a new version in librbd. With the rbd_default_format
setting we can now force qemu-img to create format 2 RBD images.
This is available since Ceph version 0.67.5 (Dumpling).
Add executeInVR() with timeout interface to VirtualRouterDeployer
AggregationControlCommand with Action.Finish may take longer than normal command
since it would execute all the commands in one execution, and it may result in
SSH timeout for SshHelper or other mechanism communicate with VR.
Introduce an new executeInVR() interface with added timeout period for waiting
FinishAggregationCommand to complete execution.
to flow rules and applies them on the bridge
add event subscriber in OvsTunnelManager, that listens to
replaceNetworkAcl events. On event sends the updated policy info to all
the hosts in the VPC
- get the hosts on which VPC spans given vpc id
- get the VM's in the VPC
- get the hosts on which a network spans
- get the VPC's to which a hosts is part of
- get VM's of a VPC on a hosts
introduces capability to build a physical toplogy representation of a
VPC. This json file is encapsulated in
OvsVpcPhysicalTopologyConfigCommand, and is used to send full topology
to hypervisor hosts. On hypervisor this json config can be used to setup
tunnels, configure bridge, add flow rules etc
Ovs GURU, to use different broasdcast scheme VS://vpcid.gerkey for the
networks in VPC that use distributed routing
each VIF and tunnel interface to carry the network UUID in other/options
config
storage pool (SMB) and attached to a running vm can be live migrated to another shared storage
pool. Also a vm and its volumes can be live migrated to another host and storage pool respectively.
2) Corrected some logging in MidoNetPublicNetworkGuru - removed .toString method call on the objects in the log body as toString is called on the object by default when use log4j
CLOUDSTACK-4762 : Enabling VGPU support for XenServer.
This feature is to enable the GPU-passthrough and vGPU functionality,
with the help of this feature, admins/users will be able to leverage
the GPU graphics unit power by deploying a virtul machine with GPU or
vGPU support or by changing the service offering of an existing VM
at any later point of time. There GPU/vGPU enabled VMs are able to run
graphical applications.
For now, this feature is only supported with XenServer hypervisor but
can be extended to add the support of other hypervisors.
In vCenter 5.5, once a volume is migrated the VMDKs are renamed to match the name of the VM.
Update volume path for every volume belonging to the VM to the corresponding new disk filename.
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
plugins/hypervisors/vmware/src/org/apache/cloudstack/storage/motion/VmwareStorageMotionStrategy.java
In vCenter 5.5, once a volume is migrated the VMDKs are renamed to match the name of the VM.
If a volume has been renamed upon migration update its volumePath to that of the new disk filename.
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
vmware-base/src/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java
With VirtIO enabled on KVM. FreeBSD 10 supports VirtIO for both the
network and the disks. This frees us from IDE and E1000 which should
also improve performance.
By default all network disks are in RAW format. Gluster works fine with
QCOW2 which has some advantages.
Disks are by default in QCOW2 format. It is possible to run into
a mismatch, where the disk is in QCOW2 format, but QEMU gets started
with format=raw. This causes the virtual machines to lockup on boot.
Failures to start a virtual machine can be verified by checking the log
of the virtual machine, and compare the output of 'qemu-img info'.
In /var/log/libvirt/qemu/<VM>.log find the URL for the drive:
-drive file=gluster+tcp://...,format=raw,..
Compare this with the 'qemu-img info' output of the same file, mounted
under /mnt/<pool-uuid>/<img-uuid>:
# qemu-img info /mnt/<pool-uuid>/<img-uuid>
...
file format: qcow2
...
This change makes passes the format when creating a disk located on RBD
(RAW only) and Gluster (QCOW2).
Signed-off-by: Niels de Vos <ndevos@redhat.com>
The support for Gluster as Primary Storage is mostly based on the
implementation for NFS. Like NFS, libvirt can address a Gluster environment
through the 'netfs' pool-type.
PrepareForMigrationCommand, so that destination hypervisor can
mount pool. This further exposed an issue for KVM where iso
was not getting cleaned up upon successful migration, fixed as well.
By default only the Integers between -128..127 are cached (unless overridden by java.lang.Integer.IntegerCache.high system property)
If the inbound or outbound values are higher, the reference comparison won't work.
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
- minor resource leak cleaned up
- cpu-speed reading method extracted
- test added
- logging added in case of exception
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
This saves us a lot of code and libvirt is probably a better
place to do this.
libvirt-java now has the support we want, so we can now resize volumes
with libvirt.
(C)LVM volumes can't be resized using libvirt, so we have to
invoke a resize script for that.
By default the client_mount_timeout setting in librados is 300 seconds,
but that causes the connect to the Ceph cluster to block for 5 minutes
if the Ceph cluster is not available.
This patch is not ideal, but it mitigates the problem for now.
At a later point all this librados/librbd code should go back to libvirt
again, but the current versions of libvirt in the distributions are
to old for all the features we require.
For now this should prevent the CloudStack agent blocking for 5 minutes
when the Ceph cluster isn't available.
This is also tracked at the Ceph tracker: http://tracker.ceph.com/issues/6507
To obtain network read/write statistics, multiply sample duration with the
average of the particular performance metric obtained over the sample period.
Added fix for exception and listing. Mentioned details under bug.
Post the fix, simulator works fine.
Signed-off-by: Santhosh Edukulla <Santhosh.Edukulla@citrix.com>
Signed-off-by: Koushik Das <koushik@apache.org>
Introduced by:
commit ac65f8fddf
Author: Hugo Trippaers <htrippaers@schubergphilis.com>
Date: Mon Jan 20 18:03:02 2014 +0100
CLOUDSTACK-5884 make getTargetSwitch(NicTO nicTo) do all the work to select
switch name, type and vlan token. Change preference to use the tags set on the
physical network.
the management server was restarted. The template.properties file created for the
template has the format field in upper-case. This caused the template service to
not to recognise the format and it removed the entry from the template_store_ref
table in db. Fixed the format field in the templatee.properties.
null pointer exception was getting generated when a VolumeTO object was
serialized to create an answer object. If a local storage is used the uri
field will be null. Added null checks for the same.
We hit these excptions whenever a management server held session that was with the old 5.1 vCenter server
is used to make resource calls to the new 5.5 vCenter.
Validate a vCenter session context before it is being used to make a resource call.
And if the context is invalid then discard the context and retrieve a new one.
During the invalidation of an old context handle the context disconnect better
by catching the appropriate exception and returning a newly created context.
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareContextFactory.java
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
plugins/hypervisors/vmware/src/com/cloud/storage/resource/VmwareSecondaryStorageResourceHandler.java
vmware-base/src/com/cloud/hypervisor/vmware/util/VmwareContext.java
For a detached volume, don't try to find the associated VM on the hypervisor/peer hypervisor host.
By default create a worker VM to perform snapshot operations.
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/storage/resource/VmwareStorageProcessor.java
on hyperv. There were multiple issues here. Upload volume was actually
failing because the post download check for vhd on the cifs share was
unsuccessful. Also the agent code wasn't parsing the volume path correctly.
Fixed it too.
Don't package the OVF and VMDK files into OVA after a template is created from volume.
Since packaging process contains reading and writing from the NFS mount, it doubles the amount of data that needs to be moved around
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/storage/resource/VmwareStorageProcessor.java
Instead of injecting object of VolumeOrchestrationService into VmwareResource, we now populate the command object (MigrateVolumeCommand here) with required information. Thus we dont need volume orchestration service to query that information from resource.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
is attached a hard disk drive is created on the scsi controller. On detach
the data disk is removed from the drive but the disk drive is left behind.
On reattach the agent was again trying to create a disk drive while it was
already present. Fixed the agent code to look up for disk drive while
attaching and if one is not found then only to create the drive for
attaching a data disk.
The agent was always creating a disk with image format vhdx, but the
cloudstack management server defaults to image format vhd for hyperv.
Updated the agent code to be consistent with what cs expects. All disks
are now created with image format vhd.
When VM is not running, existing code is unable to retrieve associated cluster's Id. Now we will try to get this information using previous host where the VM was running.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java
DetachISO is succeeding even though detach opeartion is failing as cdrom is locked by VM as it was mounted inside VM.
Detect if cdrom is locked or not. If locked fail detach operation and warn user to unmount before detaching the iso/cdrom device.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
Conflicts:
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
plugins/hypervisors/vmware/src/com/cloud/storage/resource/VmwareStorageProcessor.java
vmware-base/src/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java
deployment fails because of that as cloudstack tries to deploy it on a host which is
ctually down. An investigator wasn't present for hyper-v; so cloudstack wasn't able to
determine the status of the host. Wrote an investigator for hyper-v which checks with
other hosts in the cluster for the status of the host being investigated.
host is put in maintenance mode. The migrate flag wasn't set to true in
the maintain answer. This caused cloudstack to not to schedule a migration
work item for vms on the host. Made a change to set the migrate flag to
true in migrate answer.
brought back up after being down for few hours,snapshot jobs do not get
triggered with reason "there is other active snapshot tasks on the
instance to which the volume is attached".
The systemvm iso file is copied only when a systemvm or router vm is to be started on
a host. The file gets copied to the secondary storage. The mount point used is the one
that has permissions for regular user to mount a share.
CLOUDSTACK-5275: The failure was because a secondary storage wasn't available when the
host was added. When a setup is done through wizard the hosts get added before the
secondary storage. CS was tying to copy yhe systemvm iso to secondary and it used to
fail if it wasn't available. Made a change to copy the iso only when a systemvm is
being started on a host.
CLOUDSTACK-5202: Made changes to clean up mount points on stop and start.
All the three are related fixes; so putting a fix in one commit.
While VRs upgrade, cloudstack should still be able to work with older
/newer version of the scripts within VRs. To allow this, the simulator
needs to send the version strings for the domr version response or the
VR start is interrupted.
Signed-off-by: Prasanna Santhanam <tsp@apache.org>
In VMware during VM start the existing disk information is used to configure the VMs. So even if a new disk is created using the new template VM continues to use the old disk.
Once the old root disk is marked for destroy force expunge it and sync the new disk into the VM folder before VM start
- the result of dividing long with long resulted in loss of precision both for network and IO
- unit tests included
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
replace vlanid wih broadcast uri to support vxlan to identify whether id is VLAN ID or VNI
Signed-off-by: ynojima <mail@ynojima.net>
Signed-off-by: Hugo Trippaers <htrippaers@schubergphilis.com>
Fixing rebase issues after integrating with wmi v2 implementation.
Removing the executable attribute from some files.
Remove the unused wmi v1 interface file.
Unit test for DestroyCommand implementation in hyperv agent.
Fixed VM state changes w.r.t wmi version 2 changes
If a VM is already running, deploy virtual machine shouldn't fail and throw an exception.
Don't run vhd-util on templates which are present on CIFS. Hyperv uses cifs as secondary storage
Add a SCSI controller by default. This is needed so that data volumes can be added/removed
on a running vm.
Remove the hard coded path in the agent code.
Rat fixes for hyper agent. Added the missing headers in files where it was missing.
Copy the iso to the secondary storage and let the hypervisor agent know of its
location during setup. The agent will copy it over once it handles the setup
command.
Changes for attaching the systemvm iso to virtual router will booting it -
part 2. The agent copies over the systemvm iso during setup. When a
virtual router is being booted it attaches the iso to it.
Hyperv unit tests for the agent. Unit tests are written using NSubstitute and XUnit and
they test the create, stop and start commands in the agent.
Fix to make sure the hyperv agent and the funcitonal tests are working after the unit tests update.
Fixing the warnings while running unit tests for hyper agent.
Added a new switch for functional tests.
Update the unit test to create a fake vhd file on the fly and run the test. The file is removed when the test completes.
Fix for functional tests. The test was failing to build on java 1.6.
Fix to bring up SSVM and Console Proxy systemvms
Fix to discover the seeded template to bring up the systemvm's for the first startup and fixed UNC path isues
Fixed the UNC path for copying the files from CIFS, and from seeded template
Fixed the issues for ssvm and cpvm to wait until it gets configured and then return the status. Made checksum method to return true.
Fixed HypervDirectConnect resource to figure out the status of systemvms, Need to fix this issue by connecting to public/control ip instead of local ip
checksum is failing for the copied system vm images, currently bypassing.
Implemented commands that are required for VR to bootup and Vm deployment to work
Modified hyperv agent code, to deploy VR with Boot Args, boot args passed to VR using KVP Exchange Component.
Fix for VR to boot up and get configured with boot args, Fixed issue in VolumeOrchestrator
Implemented SetFirewallRulesCommand in HyperV Resource
Implemented VR network commands to provide the necessary services from VR
Fixed hyperv localstorage path encode url issue. encode is converting space to '+'
architecture allows additional functionality to be easily added. Incorporating the plugin in CloudStack will allow
the community to participate in improving the features available with Hyper-V. The plugin uses a Director Connect
Agent architecture described here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Progress
Add ability to pass kvp data via the key cloudstack-vm-userdata
Rearrange code to make it clearer what .NET objects are being used.
Test failures are easier to deal with if test key is not deleted.
Acquire management/pod ip for control ip when VR deploys in HyperV
Fixed deletion on VM's on hyperv host when mgmt server gets restarted due to HA
Implementation for attach iso command. Attaches an iso to a given vm.
Detail: getPhysicalDisk() was not matching on volumes with .raw, so
instead setting disk format to QCOW2.
BUG-ID: CLOUDSTACK-5018
Bugfix-for:
Reviewed-by:
Reported-by:
Signed-off-by: John Kinsella <jlk@stratosec.co> 1383287538 -0700
Now VPN connection can be created as "passive", which would enable the ability
of remote peer initiate the connection. So it's possible for VPC VR to
establish the connection to another VPC VR of CloudStack.
Test case also included.
The test case would create 2 vpcs and using VPN to connect them.
1) vxlan will use bridge scheme 'brvx-<vni>'. Multiple physical networks can host guest
traffic type with vxlan isolation, so long as they don't use the same VNI range.
2) Guest traffic labels can be physical interface if bridge by given name is not found.
Normally we take traffic label name, find the matching bridge, then resolve that to a
physical interface. Then we create guest bridges on that interface. Now we can just
specify the interface.
When a ROOT volume is created from base template, if a folder already exists for the ROOT volume's VM then replace the old ROOT disk files with the new one.
The simulator uses the default planners of cloudstack and does not
require a separate planner context (as of now). This was just c&p from
baremetal planners.
Signed-off-by: Prasanna Santhanam <tsp@apache.org>
xs 6.1/6.2 introduce the new virtual platform, so there are two virtual platforms, windows PV driver version must match virtual platforms,
this patch tracks PV driver versions in vm details and template details.
Anthony
Detail: Checks for other Ethernet interface names uses startsWith(),
whereas the p1p1 style interface uses a regex that doesn't allow for
tailing characters, and so blocks vlan IDs. Fixed.
BUG-ID: CLOUDSTACK-4884
Bugfix-for: 4.2.1
Reviewed-by:
Reported-by:
Signed-off-by: John Kinsella <jlk@stratosec.co> 1381965250 -0700
Introduction of a new Transaction API that is more consistent with the style
of Spring's transaction managment. The existing Transaction class was renamed
to TransactionLegacy. All of the non-DAO code in the management server has been
updated to use the new Transaction API.
I don't think host kernel version has any bearing on it. Original code
was tested with CentOS 6.3 and 6.4, but it seems to succeed or fail per-host,
e.g. a fast host might work and a slow host might not. I was getting intermittent
failures with ubuntu 12.04.3 prior to this patch.
These changes are a joint effort between Edison and I to refactor some
of the code around snapshotting VM volumes and creating
templates/volumes from VM volume snapshots. In general, we were working
towards allowing PrimaryDataStoreDrivers to create snapshots on primary
storage and not requiring the snapshots to be transferred to secondary
storage.
High level changes:
-Added uuid to NfsTO, SwiftTO & S3TO to cut down on the requirement of
PrimaryDataStoreTO and ImageStoreTO which don't really serve much of a
purpose
-Initial work towards enable reverting VM volume from snapshots
-Added hypervisor commands for introducing and forgetting new hypervisor
objects (snapshots, templates & volumes)
Signed-off-by: Edison Su <sudison@gmail.com>
ACS is now comprised of a hierarchy of spring application contexts.
Each plugin can contribute configuration files to add to an existing
module or create it's own module.
Additionally, for the mgmt server, ACS custom AOP is no longer used
and instead we use Spring AOP to manage interceptors.
The managed context framework provides a simple way to add logic
to ACS at the various entry points of the system. As threads are
launched and ran listeners can be registered for onEntry or onLeave
of the managed context. This framework will be used specifically
to handle DB transaction checking and setting up the CallContext.
This framework is need to transition away from ACS custom AOP to
Spring AOP.
Initial patch for VXLAN support.
Fully functional, hopefully, for GuestNetwork - AdvancedZone.
Patch Note:
in cloudstack-server
- Add isolation method VXLAN
- Add VxlanGuestNetworkGuru as plugin for VXLAN isolation
- Modify NetworkServiceImpl to handle extended vNet range for VXLAN isolation
- Add VXLAN isolation option in zoneWizard UI
in cloudstack-agent (kvm)
- Add modifyvxlan.sh script that handle bridge/vxlan interface manipulation script
-- Usage is exactly same to modifyvlan.sh
- BridgeVifDriver will call modifyvxlan.sh instead of modifyvlan.sh when VXLAN is used for isolation
Database changes:
- No change in database structure.
- VXLAN isolation uses same tables that VLAN uses to store vNet allocation status.
Known Issue and/or TODO:
- Some resource still says 'VLAN' in log even if VXLAN is used
- in UI, "Network - GuestNetworks" dosen't display VNI
-- VLAN ID field displays "N/A"
- Documentation!
Signed-off-by : Toshiaki Hatano <haeena@haeena.net>
Libvirt reports:
org.libvirt.LibvirtException: Storage volume not found: no storage vol
with matching name
in some cases, if the volume is created on one kvm host, while accessed
from other host.
It's possible due to concurrent access(read/write) storage.
The current fix is to try serveral times, and wait for 30 seconds for
each retry.
If the issue still there, then need to sync the storage pool access
CLOUDSTACK-4457:
CLOUDSTACK-4459:
harden kvm getvolume. It's possible that one volume created on other kvm host, won't show up on another host, try more times to refresh storage pool if volume won't shown up
Conflicts:
engine/storage/integration-test/test/org/apache/cloudstack/storage/test/FakeDriverTestConfiguration.java
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
There still exist two issues after Edison's commits.
(1) Migration from new hosts to old hosts failed.
The bridge name on old host is set to cloudVirBr* if network.bridge.name.schema is set to 3.0 in /etc/cloudstack/agent/agent.properties, but the actual bridge name is breth*-* after running cloudstack-agent-upgrade.
(2) all ports of vms (Basic zone, or Advanced zone with security groups) on old hosts are open, because the iptables rules are binding to device (bridge) name which is changed by cloudstack-agent-upgrade.
After this, the KVM upgrade steps :
a. Install 4.2 cloudstack agent on each kvm host
b. Run "cloudstack-agent-upgrade". This script will upgrade all the existing bridge name to new bridge name, and update related firewall rules.
c. install a libvirt hook:
c1. mkdir /etc/libvirt/hooks
c2. cp /usr/share/cloudstack-agent/lib/libvirtqemuhook /etc/libvirt/hooks/qemu
c3. chmod +x /etc/libvirt/hooks/qemu
c4. service libvirtd restart
c5. service cloudstack-agent restart
Signed-off-by: Wei Zhou <w.zhou@leaseweb.com>
The migrate method from libvirt supports passing down a different XML for running
the instance of the target hypervisor.
This enables the VNC to bind to the private IP Address of the hypervisor and during
migration this will be changed to the private IP address of the target host.
This way VNC doesn't listen world wide and is much safer.
when secondary storage is mounted as read-only, changing permission of files on it will fail. But we should still stick to current mount point instread of
returning a wrong mount point /mnt/sec
Initial patch for VXLAN support.
Fully functional, hopefully, for GuestNetwork - AdvancedZone.
Patch Note:
in cloudstack-server
- Add isolation method VXLAN
- Add VxlanGuestNetworkGuru as plugin for VXLAN isolation
- Modify NetworkServiceImpl to handle extended vNet range for VXLAN isolation
- Add VXLAN isolation option in zoneWizard UI
in cloudstack-agent (kvm)
- Add modifyvxlan.sh script that handle bridge/vxlan interface manipulation script
-- Usage is exactly same to modifyvlan.sh
- BridgeVifDriver will call modifyvxlan.sh instead of modifyvlan.sh when VXLAN is used for isolation
Database changes:
- No change in database structure.
- VXLAN isolation uses same tables that VLAN uses to store vNet allocation status.
Known Issue:
- Some resource still says 'VLAN' in log even if VXLAN is used
- in UI, "Network - GuestNetworks" dosen't display VNI
-- VLAN ID field displays "N/A"
This failed due to a RAW -> QCOW2 conversion (again).
The current code still makes to much assumptions about everything always
being QCOW2 while that is not always true.
UI support for baremetal PXE server
CloudStack CLOUDSTACK-1364
UI support for baremetal DHCP server
Conflicts:
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BareMetalPingServiceImpl.java
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalKickStartServiceImpl.java
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalPxeManagerImpl.java
KVM - Create template from volume
Vmware - Create template from volume / Create template from snapshot
send the physical size in the copycommand which accordingly will populate template store ref and the usage_event tables with the right physical size
Signed off by : nitin mehta<nitin.mehta@citrix.com>
XS Creating templates from volume - send the physical size in the copycommand which accordingly will populate template store ref and the usage_event tables with the right physical size
Signed off by : nitin mehta<nitin.mehta@citrix.com>
If all the VM's volumes are on zone wide primary storage pool then live migration of the VM would not involve storage migration. Hence MigrateVM API would be called against MigrateVMWithVolume. So far PrepareForMigrationCommand handled scenarios of VM moving across hosts within a cluster, but with zone wide primary storage in picture this command need to handle scenarios of VM moving across clusters. Try to find the VM in datacenter if not found within cluster.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
Simulator should revert back to CLOUD_DB after its operations on
SIMULATOR_DB or the cloudstack connections go to the simulator instead
of cloud.
Signed-off-by: Prasanna Santhanam <tsp@apache.org>
(cherry picked from commit 3d39716c8f)
Remi Bergsma
Wido den Hollander
Daan Hoogland
Funs Kessen
David Mabry
Koushik Das
wilderrodrigues
Boris Schrijver
Satoru Nakaya
Vincent Bernat
Kevin McCormick
Rohit Yadav
Wei Zhou
Mike Tutkowski
Rajani Karuturi
Rafael Weingartner
Likitha Shetty
Miguel Ferreira
manuiiit
Aaron Brady
Daan Hoogland
Funs
Wido den Hollander
Jayapal
Kishan Kavala
Rafael da Fonseca
Sanjay Tripathi
Sateesh Chodapuneedi
Laszlo Hornyak
Anshul Gangwar
Daan Hoogland
Rajani Karuturi
KC Wang
afornie
Hugo Trippaers
punith-cloudbyte
Marcus Sorensen
Saksham Srivastava
Wei Zhou
Harikrishna Patnala
Santhosh Edukulla
Bharat Kumar
Frank Zhang
Anthony Xu
amoghvk
Logan Barfield
punith-cloudbyte
Edison Su
Damodar
Nitin Mehta
Devdeep Singh
David Nalley
imduffy15
Fred Clift
Kishan Kavala
Min Chen
Sheng Yang
Suresh Ramamurthy
Marcus Sorensen
Joris van Lieshout
Kelven Yang
ynojima
Murali Reddy
Girish chaudhari
Tim Mackey
Rajesh Battala
Chris Suich
Ding Yuan
Kishan Kavala
Abhinandan Prateek
Alex Huang
Santhosh Edukulla
Alena Prokharchyk
Mandar Barve
Niels de Vos
Sander Botman
Jayapal
Donal Lafferty
tuna
Prasanna Santhanam
Chris Suich
Pedro Marques
John Kinsella
Darren Shepherd
Darren Shepherd
Toshiaki Hatano
Edison Su