CLOUDSTACK-9231: Root volume migration from one primary to another primary storage within the same cluster is failingEXPECTED BEHAVIOUR:
====================
Root Volume migration within cluster should work.
ACTUAL BEHAVIOUR:
==================
Root volume migration within cluster failed.
This situation arises when there are two management server accessing the same database.
When the migration request comes the command is forwarded from one management server to another because the host is owned by the second management server. So, serialisation of map from one to another fails.
Fix:
===
This is fixed by converting the maps to lists.
* pr/1336:
CLOUDSTACK-9231: Root volume migration from one primary to another primary storage within the same cluster is failing
Signed-off-by: Remi Bergsma <github@remi.nl>
* 4.7:
Fix unable to setup more than one Site2Site VPN Connection
FIX S2S VPN rVPC: Check only redundant routers in state MASTER
PEP8 of integration/smoke/test_vpc_vpn
Add S2S VPN test for Redundant VPC
Make integration/smoke/test_vpc_vpn Hypervisor independant
FIX VPN: non-working ipsec commands
[UI] MADNESS
[DB] Add force_encap field to s2s_customer_gateway table
[ROUTER] Add forceencaps field to python router ipsec config method
[TEST] unittest needs rework
[MARVIN] Add forceencap field to VpnCustomerGateway class in marvin base
[CORE] Add Force UDP Encapsulation option to Site2Site VPN
CLOUDSTACK-9186: Root admin cannot see VPC created by Domain admin user
CLOUDSTACK-9192: UpdateVpnCustomerGateway is failing
CLOUDSTACK-6485 prevent ip asignment of private gw iface
CLOUDSTACK-9204 Do not error when staticroute is already gone
make both check lines consistent
CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh
CLOUDSTACK-9202 Bump ssh timeout
This situation arises when there are two management server accessing the same database.
When the migration request comes the command is forwarded from one management server to another because
the host is owned by the second management server. So, serialization of map from one to another fails.
This is fixed by converting the maps to lists.
* 4.7:
CLOUDSTACK-9220 Sort list of domains on Domain tab in UI
Admin cannot see VMs on port forwarding page
Fix mariadb related listCapacity bug (CLOUDSTACK-8966)
CLOUDSTACK-9213 - Split the ACL rules using comma instead of dash.
CLOUDSTACK-9213 - Formatting the code
* 4.6:
[UI] bug fix: Delete added ACL lists is not available for Domain Admin and normal users
CLOUDSTACK-4787: Allow users to select disk controller for VM/template
CLOUDSTACK-4787 Allow selection of scsi controller type in vSphere
- Adds new controller types in the UI, for selecting root disk controller while
registering templates
- Fixes bug to not override disk controller type if provided in the details (either
vm details or from template details)
(cherry picked from commit c7d67628b3)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
commit #7
So far only 1 controller (scsi or ide) is supported in Cloudstack for ide or
scsi, this is existing limitation. Added support for 2nd IDE controller. Support adding IDE
virtual disk to VM. Also added check if VM is running as IDE virtual disk cannot be attached
to VM if VM is runnning.If user detaches a virtual disk on lower unit number of controller,
then subsequent attach operation should find free unit number on the controller and attach
the virtual disk there.
commit #6
Let the controllers of existing VMs continue without flip, current busInfo retrieved from
chain_info field of volume record from database would be preferred over
controller settings from all configuration settings.
commit #5
Editing global configuration param vmware.root.disk.controller osdefault value results
in loss of previous root disk controller type. Hence root disk's controller type for legacy
VMs is unknow post that modificaiton by user. If VM is stop/start then we could get this
infromation from bus info of existing volume. But if user resets VM and then try to start VM.
The existing bus info would be lost. Hence existing disk info is not available to depend on.
Using lsilogic or generic scsi controller for ROOT disk of legacy VMs if reset.
commit #4
Avoid adding additional (>1) scsi controllers to system vms. While attaching volume to legacy VM
don't use osdefault optoin which applicable only for VM created with the option enabled, use
legacy data disk controller type (lsilogic)
commit #3
If root disk's controller type is scsi and data disk controller type condenses
to any of scsi sub-types then data disk controller type would fall back to root disk controller itself. This
ensures data volumes would be accessible in all cases as controller of root volume would be reliable
and it means VM has the supported controller. It also avoids mix of scsi controller sub-types in a user instance.
Also translating disk controller type scsi to lsilogic.
commit #2
Support auto detection of recommended virtual disk controller type for specific guest OS.
commit #1
Support granual controller types. Add support for controller types in template registration as well.
Fix white spaces.
Removed stale HEAD merge lines
Removed tail of merge lines
Fixed VmwareResource, removing storage commands that moved to VmwareStorageProcessor.
removed stale code of controller that is present in processor
Fixed check style errors.
Fixed injection.
Tested with Linux and windows templates. Unable to run iso based tests due to few bugs in register iso area.
Signed-off-by: Sateesh Chodapuneedi <sateesh@apache.org>
(cherry picked from commit a4cc987a6f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Storage motion of vm across clusters/xenserver-pools fails in a
clustered management server setup. In xen storage motion we have to send
a migrate_receive command to the destination host, followed by
migrate_send command to the source host. The sr and network detials of
the destination host have to be passed in migrate_send command on the
source. While migrating across clusters the source and destination
resources are seperate objects. To pass this information across
resources we have to send seperate migrate with storage receive and send
commands to the resource. In a clustered ms setup these commands may
have to be forwarded to another ms as the resource may be owned by it.
The serilization of the command and answer objects fails in such case as
it doesn't understand the xapi sr and network objects. Made a change to
serialize these objects in the resource layer and pass the around as
strings in the command and answer objects. Reviewed-By: Likitha
Removed unnecessary @Local annotations and their respective importsFollowing @rafaelweingartner 's findings in PR #714 that many spring beans contained an @Local annotation, we've decided to remove said annotations and their imports from the ComponentLifecycleBase class and its subclasses seeking a reduction of a few hundred lines of useless code.
I had already opened a pull request for this (#853) but at some point my commit disappeared from the PR entirely, showing no new changes in code, which caused it to be merged automatically (with no changes).
* pr/1102:
Removed unnecessary @Local annotations and their respective imports from the ComponentLifecycleBase class and its subclasses.
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9048: Fix typo for public network description
* pr/1051:
CLOUDSTACK-9048: Fix typo for public network description
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9062: Improve S3 implementation.The S3 implementation is far from finished, this commit focuses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimizations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
Please review thoroughly, both code inspection and (automated) integration tests. Currently no integration tests are available specifically for S3. Therefore the implementation is needed to be tested manually, for now...
What I tested:
- Greenfield install -> will download latest systemvm template automatically to S3.
- Upload a template/iso
- Download a template/iso
- Restart of management server -> list available templates -> doesn't download them again if available.
* pr/1083:
CLOUDSTACK-9062: Improve S3 implementation.
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9067 - As I developer I want to remove all the unused router-shell scripts from ACSThis PR removes the unused shell scripts that were present in the ACS project. Those script were replaced by the.
Some of the scripts are used by the HyperV Resource, which were hardcoded. I took the opportunity to use the Java constants over there as well, so the next one touching the code will know they exist and won't hardcode anything.
The following task were applied:
* Remove the shell files and the Java constants that were mapping them;
* Apply the use of the Java constants to the HyperV Resource class;
* Wrap the String.format() method in the StringUtils so we can test the changes in the HyperV Resource class.
The last point was added because I do not have a HyperV test environment. Hence, I wanted to make sure the tiny code I changed is covered at least by unit tests.
* pr/1084:
CLOUDSTACK-9067 - Replaces hardcoded paths with the VRScripts constants.
CLOUDSTACK-9067 - Fomatting the code of HypervDirectConnectResource class
CLOUDSTACK-9067 - Remove old script file from the project
Signed-off-by: Remi Bergsma <github@remi.nl>
The S3 implementation is far from finished, this commit focusses on the bases.
- Upgrade AWS SDK to latest version.
- Rewrite S3 Template downloader.
- Rewrite S3Utils utility class.
- Improve addImageStoreS3 API command.
- Split various classes for convenience.
- Various minor improvements and code optimalisations.
A side effect of the new AWS SDK is that it, by default, uses the V4 signature. Therefore I added an option to specify the Signer, so it stays compatible with previous versions.
again due to missing virtualsize property in template.properties
We have multiple file processors to process different types of image
formats. The processor interface has two methods getVirtualSize() and
process().
1. getVirtualSize() as the name says, returns the virtual size of
the file and is used at get the size while copying files from NFS to s3
2. process() returns FormatInfo struct which has fileType, size,
virutalSize, filename. on successfully downloading a template, each
file is passed to all the processors.process() and whichever returns a
FormatInfo, that will be used to create template.properties file. If
process() throws an InternalErrorException, template installation fails.
But, if process() returns null, template registration is successful with
template.properties missing some attributes like virtualSize, file
format etc. which results in this bug on restart of ssvm/cloud
service/management server.
failing the template download if virutalsize or some other properties
cannot be determined.
The following changes are done:
getVirtualSize() to always return size(if it can calculate, get virtual
size else return file size). This would mean the following changes
1. QCOW2Processor.getVirtualSize() to return file size if virtual
size calculation fails
2. VHDProcessor.getVirtualSize() to return file size if virtual size
calculation fails
process() to throw InternalErrorException if virtual size calculation
fails or any other exceptions occur. This would mean the following
changes
1. OVAProcessor to throw InternalErrorException if untar fails
2. QCOW2Processor to throw InternalErrorException if virtual size
calculation fails
3. VHDProcessor to throw InternalErrorException if virtual size
calculation fails
This reverts commit 6841ba61da, reversing
changes made to 13b29bac5a.
Master is currently frozen, no merges without RM approval.
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201509.mbox/browser
It also broke the build:
[INFO] Apache CloudStack Framework - Jobs ................ SUCCESS [3.448s]
[INFO] Apache CloudStack Cloud Engine Internal Components API SUCCESS [2.528s]
[INFO] Apache CloudStack Server .......................... FAILURE [24.769s]
[INFO] Apache CloudStack Usage Server .................... SKIPPED
Fixed secstorage.proxy is not setting proxy configuration on SSVM in some cases
* pr/802:
Fixed secstorage.proxy is not setting proxy configuration on SSVM in some cases
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
Guys, can you review it? things need to be discussed:
(1) this supports KVM/QCOW2 only. Anyone want to implement for other Hypervisor/format ?
(2) The original data volume (on primary storage) will be removed.
(3) The script uses the default timeout in libvirtComputingResource. Do we need to add one in global configuration (like copy.volume.wait or backup.snapshot.wait, create.volume.from.snapshot.wait)
(4) In scripts/storage/qcow2/managesnapshot.sh, I use "qemu-img convert -f qcow2 -O qcow2" to copy the snapshot from secondary to primary (hence there is no base image file), instead of "cp -f", this is because convert is faster than cp in my testing.
* pr/732:
CLOUDSTACK-5863: revert volume snapshot for KVM/QCOW2
Signed-off-by: Wei Zhou <w.zhou@tech.leaseweb.com>
This reverts commit cd7218e241, reversing
changes made to f5a7395cc2.
Reason for Revert:
noredist build failed with the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloud-plugin-hypervisor-vmware: Compilation failure
[ERROR] /home/jenkins/acs/workspace/build-master-noredist/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[484,12] error: non-static variable logger cannot be referenced from a static context
[ERROR] -> [Help 1]
even the normal build is broken as reported by @koushik-das on dev list
http://markmail.org/message/nngimssuzkj5gpbz
messages: Removed unused agent command AttachVolumeCommand and corresponding answer AttachVolumeAnswer
These types are not used anywhere in the orchestration layer.
Signed-off-by: Koushik Das <koushik@apache.org>
This patch makes it possible to expose VM UUID to subsystems, this can be
useful for implementing VM Snapshots for KVM in future.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This reverts commit 180afe52e5.
This broke the build on master:
master build broken with the below error http://jenkins.buildacloud.org/job/build-master-slowbuild/2101/consoleText
```
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/acs/workspace/build-master-slowbuild/plugins/hypervisors/xenserver/test/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixRequestWrapperTest.java:[1581,51] error: constructor CreateVMSnapshotCommand in class CreateVMSnapshotCommand cannot be applied to given types;
[ERROR] required: String,String,VMSnapshotTO,List<VolumeObjectTO>,String
found: String,VMSnapshotTO,List<VolumeObjectTO>,String
reason: actual and formal argument lists differ in length
/home/jenkins/acs/workspace/build-master-slowbuild/plugins/hypervisors/xenserver/test/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixRequestWrapperTest.java:[1623,53] error: no suitable constructor found for RevertToVMSnapshotCommand(String,VMSnapshotTO,List<VolumeObjectTO>,String)
[INFO] 2 errors
[INFO] -------------------------------------------------------------
```
This was PR #717 towards 4.5
This patch makes it possible to expose VM UUID to subsystems, this can be
useful for implementing VM Snapshots for KVM in future.
This was PR #717 towards 4.5
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 0062ff2672)
Signed-off-by: Remi Bergsma <github@remi.nl>
* pr/547:
CLOUDSTACK-8601. VMFS storage added as local storage can be re-added as shared storage. Fail addition of a VMFS shared storage pool in case it has already been added as local storage in CS.
Signed-off-by: Mike Tutkowski <mike.tutkowski@solidfire.com>
- Adding update_host_passwd to VRScripts
- Add accessor method to host password on CitrixResourceBase
- Add implementation to CitrixUpdateHostPasswordCommandWrapper
- Improve testUpdateHostPasswordCommand() unit test on CitrixRequestWrapperTest
- Add line to patch files on xenserver directory
Concerning the LibVirt change:
- I forgot to assing the return of the getDefaultHypervisorScriptsDir() method to the hypervisorScriptsDir variable
- Modifying the LibvirtUpdateHostPasswordCommandWrapper in order to execute the script on the host
- Adding the script path to LibvirtComputingResource
- Adding the host IP address as an instance variable on UpdateHostPasswordCommand
- Improving the Unit Test (LibvirtComputingResourceTest) to get it covering the new code
- Formatting code
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
CLOUDSTACK-8180 - RouterVM does no longer provide X-ForwardedFor header with Loadbalancer
- Make sure the "option forwardfor" is always available by not removing it
- If keepalived is enabled, then set "no option forceclose"
Signed-off-by: wilderrodrigues <wrodrigues@schubergphilis.com>
This closes#549
- Adding LibvirtNetworkElementCommandWrapper and LibvirtStorageSubSystemCommandWrapper
- 2 unit tests added
- KVM hypervisor plugin with 22.2% coverage
I also refactored the StorageSubSystemCommand interface into an abstract class
- Remove the pseudo-multiple-inheritance implementation
- The StorageSubSystemCommand was an interface, not related to the Command class
and its implementation were extending the Command class anyway. The whole structure is better now.
- Adding LibvirtCheckConsoleProxyLoadCommandWrapper, LibvirtConsoleProxyLoadCommandWrapper, LibvirtWatchConsoleProxyLoadCommandWrapperand CitrixConsoleProxyLoadCommandWrapper
- 2 unit tests added
- KVM hypervisor plugin with 12% coverage
Refactored the CommandWrapper interface in order to remove the esecuteProxyLoadScan, which is now
implemented bu subclasses.
- Adding LibvirtStopCommandWrapper
- LibvirtRequestWrapper
- 1 unit tests
Refactored the RequestWrapper to make it better.
- Changes also applied to the CitrixRequestWrapper
- With the changes added by the rVPC work, the bump priority became deprecated.
This commit includes a refactor to get it removed from the following resources:
* Java classes
* domain_router table - removing the is_priority_bumpup column
* Fixing unit tests
All changes were tested with:
XenServer 6.2 running under our VMWare zone
CloudStack Management Server running on MacBook Pro
MySql running on MackBook Pro
Storage Type: Local
Fix: RTNETLINK errors
- Management Server health check trying to create already existing interface
- Changes on update_config.py, cs_guestnetwork.py, merger.py
Fix: replace RRouTER_LOG in the CsRedundant.py per log file location
Fix: Guest Net address association during Router restart
- Changes on NicProfileHelper, NicProfileHelperImpl
Fix: aggregationExecution() method on VirtualNetworkApplianceManagerImpl
- Do not send an AggregationControlCommand to a non-configured router
Some classes have been formatted.
firewallrules implemented sperately from networkacl (ugly but best that can be done ATM)
Some refactoring in CsDHCP as it made no sense to do it that way anymore
processmonitoring implemented
Domain not correctly sourced in VR
Adding a couple of tests (ConfigHelperTest) to validate the following:
* Command is generated;
* JSON String is generated based on the Java Object
* JSON Object is generated based on the String content
Also fixed the SetPortForwardingRulesVpcConfigItem: implementation was missing.
Removed the code from ConfigHelper, which is now really clean
Added Flyweight to AbstractConfigItemFacada
Renamed 3 *ConfigItem classes that had typos in their names
Adding new ConfigItem classes, that will be used to decouple the configureation from the ConfigHelper class.
The idea is to use mix of Flyweight and Facade implementation in order to the correct classes without having so many ifs and private static methods.
on restart of management server, template sync runs. It checks for
templates in ssvm using the uniquename. If it doesnt find any, cleans
the directory. In case of uploaded templates, these are getting saved
using name instead on uniquename and hence template sync cant find them
and does cleanup. Using uniquename in template.properties now.
This is a plugin that puts in ovm3 support ranging from 3.3.1 to 3.3.2. Basic
functionality is in here, advanced networking etc..
Snapshots only work when a VM is stopped now due to the semantics of OVM's raw
image implementation (so snapshots should work on a storage level underneath the
hypervisor shrug)
This closes#113
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The test build on an assumption on the order of items in a Set, while this changed in java 1.8
Signed-off-by: Laszlo Hornyak <laszlo.hornyak@gmail.com>
The volume/template size and install path was not getting updated properly.
The upload status response should return these properties on completion of upload.
on calling GetUploadParamsForVolume, persisting the metadata to db
validating the account limits and incrementing the appropriate limits
encoded the metadata on management server using preshared key
on calling GetUploadParamsForTemplate, persisting the metadata to db
validating the account limits and incrementing the appropriate limits
encoded the metadata on management server using preshared key
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
modules can use it.
(cherry picked from commit b6401b04f22b0a5b686c7c477da4c6e0fd18df84)
Conflicts:
plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalKickStartServiceImpl.java
GPU enabled hosts from non-GPU VM deployment.
Cluster reordering is based on the number of unique host tags in a cluster,
cluster with most number of unique host tags will put at the end of list.
Hosts with GPU capability will get tagged with implicit tags defined by
global config param 'implicit.host.tags' at the time os host discovery.
Also added FirstFitPlannerTest unit test file.
(cherry picked from commit 39fe766c2b)
This reverts commit 809b47e4eb.
Because the fix cause CLOUDSTACK-7163. Would fix it in another way.
Conflicts:
core/src/com/cloud/agent/resource/virtualnetwork/ConfigHelper.java
1. Adding the missing Template/Volume URLs expiration functionality
2. Improvement - While deleting the volume during expiration use rm -rf as vmware now contains directoy
3. Improvement - Use standard Answer so that the error gets logged in case deletion of expiration link didnt work fine.
4. Improvement - In case of domain change, expire the old urls
Remi Bergsma
Daan Hoogland
Nitin Kumar Maharana
Michael Andersen
Wilder Rodrigues
Koushik Das
Rohit Yadav
Sateesh Chodapuneedi
Devdeep Singh
cirstofolini
Boris Schrijver
Carles Figuerola
Wido den Hollander
Aaron Brady
Nick Livens
Wei Zhou
Rajani Karuturi
Rajani Karuturi
Anshul Gangwar
Rafael Weingartner
Likitha Shetty
Mike Tutkowski
Daan Hoogland
Jayapal
Abhinandan Prateek
Daan Hoogland
Ian Southam
Hugo Trippaers
Sander Botman
Funs
Laszlo Hornyak
Santhosh Edukulla
amoghvk
Sanjay Tripathi
Edison Su
Anthony Xu
Fred Clift
Min Chen
Kishan Kavala
Sheng Yang
Murali Reddy
Damodar Reddy
Nitin Mehta
Marcus Sorensen