CLOUDSTACK-8758: Handle redirects in communication with NXS controller (a.k.a. Nicira NVP)When an NSX controller node is part of a cluster it will redirect API calls to the master controller. Because the ACS management server does not follow such redirects, if there is a change of master within a NSX cluster, the NSX device (a.k.a. NiciraNvp) needs to be reconfigured (via the management server DB).
The goal of this PR is to enable ACS management server to follow HTTP redirects sent by NSX controllers. However, other changes were made to the cloud-utils module that provides the REST client that the NSX plugin uses.
Cosmetic changes:
* Upgrade maven module structure for cloud-utils and cloud-plugin-netowkr-nvp to comply with maven default
* Several refactorings on both modules to consistently format the code, remove unused code, declare final when possible, remove auto generated comments, etc
Other changes:
* Upgrade HTTP library used in REST client to version 4.5 of Apache HTTP Components
* Implement generic REST client that supports HTTP redirects
* Implement NSX specific REST client
* Simplify NSX api implementation
* Previously existing unit tests for both the REST client and NSX api were either maintained in the same test classes, moved to new test classes (because code under tests also moved), or removed (because code under tests was also removed)
* New Marvin tests for NSX controllers
Testing:
* Ran all unit tests present in the project
* Ran Java Integration tests for NSX api targeting both a master and a slave controller
* Ran new Marvin test for NSX controller
* Manual inspection of logs to confirm redirection is taking place
* pr/737:
Use NSX specific RestClient in API implementation (NiciraNvpApi)
Refactor NSX api implementation (NiciraNvpApi)
Add NSX specific RestClient implementation
Delegate HTTP protocol activity in RESTServiceConnector to RestClient
Add basic RestClient implentation based on HTTP Components 4.5
Code clean up in cloud-plugin-network-nvp project
Code clean up in cloud-utils project
Refactor cloud-plugin-network-nvp project into Maven default structure
Refactor cloud-utils project into Maven default structure
Fix unit-test library dependencies
Add Marvin test for Nicira NVP plugin
Create Nicira NVP devices and enable plugin at deploy
Signed-off-by: Remi Bergsma <github@remi.nl>
- Add -noverify JVM arg to surefire plugin, to allow Powermockito to
de-encapsulate private methods
- Add dependency on cloud-utils test-jar to use custom HttpRequest
matchers
- All HTTP protocol activities are now handled by RestClient
- This service is now only responsible for creating requests, and
dispatching them to the client
- Provides a Simple API for creating, updating, retrieving and deleting
objects
- Upgrade version of HTTP Components to 4.5
- Add helper class to create Http clients
- Add helper class to build http requests
- Add enum with the different Http Methods
- Add constants class for HTTP related values
- XML formatting
- Fix license header
- Reorder hamcrest, junit, mockito and powermock dependencies
* Since several libraries include a subset of hamcrest classes,
hamcrest should be declared first in the pom, inorder for its
classes to be loaded first by the JVM
- Depdend on hamcrest-all and exclude hamcrest classes from other
libraries
CLOUDSTACK-8685: Fix for missing defaultroute on VPC router https://issues.apache.org/jira/browse/CLOUDSTACK-8685
Refactored CsRoute and CsAddress to:
- add methods to add and verify the defaultroute
- renamed methods flush, add, and routeTable to flush_table, add_route, add_table respectively.
Added some suboptimal unit tests to TestCsRoutes
* pr/738:
Fixed defaultroute on VPC routers in CsRoute and CsAddress Refactored CsRoute
pep8 cleanup
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8443: Support CentOS 7 as KVM hypervisorThis adds support for CentOS 7 as a KVM hypervisor. As discussed in the Jira issue, the root cause of the issue was the `cloudstack-setup-agent` script that added this line to `/etc/libvirt/qemu.conf`:
`cgroup_controllers=["cpu"]`
On CentOS 6 this 'works', but since by default no cgroups are mounted, this setting is ignored by `libvirtd`. On CentOS 7 is does not work, because the `cpu` and `cpuacct` are 'co-mounted'. This simpy means you cannot use one without the order. The config line above instructs `libvirtd` to only use the `cpu` cgroup, which it can't.
There was code added to make sure the `cpu` cgroup was not co-mounted. It did this with a shell script, but after a reboot it was gone. It wasn't reliable and not even a solution at all. I went ahead and removed the code, because the co-mounted cgroups work fine. The real issue was the config line we put in `/etc/libvirt/qemu.conf` so much better to fix that instead.
The default for `libvirtd` is to use only cgroups that are mounted. When you remove the 'cgroup_controllers' line, the default kicks in. According to the `qemu.conf` file, the default is:
`cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]`
Then again, if it's not mounted, it's not used. So, that's a nice way to control it.
This works just fine for both RHEL/CentOS 6 and RHEL/CentOS 7.
Along the debugging, I came across some small issues here and there which I also fixed in this PR.
If you test this, be sure to build RPMs from this branch and install those on the KVM hypervisor.
At Schuberg Philis we run KVM hypervisors on CentOS 7 for months with these fixes. Now I took the time to fix the issues in CloudStack, rather than work around them.
@bhaisaab @karuturi could you please review this? Thanks!
Please note:
The next step is to fix CLOUDSTACK-8625 (Systemd profile for KVM Agent) as currently a sysvinit script is still used. Although it works, we do not have any systemd benefits. As it is a separate issue, I'll try to address it in a separate PR.
* pr/731:
CLOUDSTACK-8443: detect CentOS 7.x as RHEL 7 alike
CLOUDSTACK-8443: mention the correct logfile
CLOUDSTACK-8443: display the right hypervisor type
CLOUDSTACK-8443: don't try to fix co-mounted cgroups
Signed-off-by: Remi Bergsma <github@remi.nl>
Git PR merge script improvementsProcessed feedback from @bhaisaab and (a colleague of) @karuturi
- make the help message better (I also updated Confluence)
- make sure we can use it on other repo's as well (Documentation and such)
Tested it and works fine for me.
* pr/734:
fixed hardcoded merge message location
fix bug where the countdown would be on multiple lines
make it work with any Apache repository mirrored on GitHub
Improve error/help message in case of missing git remote
Signed-off-by: Remi Bergsma <github@remi.nl>
Example of problem:
ATTENTION: Merging pull request #731 from remibergsma/centos7-kvm into 'master' branch in 5 seconds. CTRL+c to abort..
-n 5
-n 4
-n 3
-n 2
-n 1
-n 0
Should be compatible with more environments if printf is used instead.
This setting works on CentOS 6 / RHEL 6 but does nothing, as
"cpu" cgroup is not mounted. On CentOS 7 / RHEL 7 systemd does
mount cgroups and "cpu" is co-mounted with "cpuacc". Hence, if
we specify "cpu" then this results in an error because it can
only use them both, or none.
By removing the setting, we rely on the default of qemu, which
is:
cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]
Only if they are really mounted, they will be used. So, this will
work on both version 6 and 7.
The 'fix script' didn't work well, as after a reboot you'd still have qemu
throwing errors. Now we can handle the co-mountedcgroups.
This patch makes it possible to expose VM UUID to subsystems, this can be
useful for implementing VM Snapshots for KVM in future.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fix for the NicVO.java regression.Renamed set*() methods to correct naming.
* pr/726:
Fix for the NicVO.java regression.
Signed-off-by: Remi Bergsma <github@remi.nl>
Removed duplicate code in CitrixResourceBase and its subclassesRemoved unnecessary duplicated code by having the body of the getPatchFiles method only in the CitrixResourceBase superclass. Given that all of its implementations consisted of the same code except for the path which contains the necessary file for that implementation. An abstract method getPatchFilePath was created in the CitrixResourceBase superclass so that each implementation may return the path containing the specific file needed by that implementation.
Test cases were created for each implementation, simple as they may be. One assert is made to verify that the path returned by each implementation corresponds to the path that was previously specified on each getPatchFiles implementation.
* pr/700:
Removed duplicate code in CitrixResourceBase.getPatchFiles
Signed-off-by: Remi Bergsma <github@remi.nl>
Add scripts to merge PRs and release fixes forwardAs it is now, additions to CloudStack need to go through a GitHub PR and, if accepted, merged to the Apache remote (not in GitHub). This means that merging a PR for CloudStack is a bit more involved than merging a normal GitHub PR. In addition to that, a PR that fixes something in an older release needs to be merged against a release branch, and then forward merged up to master (if applicable).
To help with these two kinds of merges, @remibergsma, @karuturi, @DaanHoogland and myself, have created two scripts (contributed in this PR).
One script (**git-pr**) helps in merging a PR from GitHub to the Apache remote. The script will read the PR info using GitHub's API and will create a merge commit that contains the same commit message a normal GitHub PR merge commit, plus, as extra info, a list of commits that are being merged.
The other script (**git-fwd-merge**) will create a specific merge commit message that indicates a forward merge, and lists the commits being merged.
This Apache wiki [page](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61311655) describes how to use these two scripts, and points to an even more detailed [page](https://github.com/schubergphilis/cloudstack-git-tree-manipulation/wiki) that explains the merge scenarios step-by-step.
By the way, beacause both scripts follow the naming convention for git tools, as long as they are in your "path" you can execute them as simply as "git pr" and "git fwd-merge". But be aware that, in order to use these scripts, you should make sure to remove any aliases that can cause a name clash.
* pr/721:
Add git merge scripts
Signed-off-by: Remi Bergsma <github@remi.nl>
This reverts commit 180afe52e5.
This broke the build on master:
master build broken with the below error http://jenkins.buildacloud.org/job/build-master-slowbuild/2101/consoleText
```
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/acs/workspace/build-master-slowbuild/plugins/hypervisors/xenserver/test/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixRequestWrapperTest.java:[1581,51] error: constructor CreateVMSnapshotCommand in class CreateVMSnapshotCommand cannot be applied to given types;
[ERROR] required: String,String,VMSnapshotTO,List<VolumeObjectTO>,String
found: String,VMSnapshotTO,List<VolumeObjectTO>,String
reason: actual and formal argument lists differ in length
/home/jenkins/acs/workspace/build-master-slowbuild/plugins/hypervisors/xenserver/test/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixRequestWrapperTest.java:[1623,53] error: no suitable constructor found for RevertToVMSnapshotCommand(String,VMSnapshotTO,List<VolumeObjectTO>,String)
[INFO] 2 errors
[INFO] -------------------------------------------------------------
```
This was PR #717 towards 4.5
This patch makes it possible to expose VM UUID to subsystems, this can be
useful for implementing VM Snapshots for KVM in future.
This was PR #717 towards 4.5
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 0062ff2672)
Signed-off-by: Remi Bergsma <github@remi.nl>
This tries to avoid cleaning by a device name.
This closes#718
(cherry picked from commit 74f697a2dd)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Apply RHEL 7 fix for comounted cgroups to CentOS 7 KVM hypervisors as wellThe RHEL 7 version string is hard coded and misses CentOS 7.
* pr/719:
RHEL 7 and CentOS 7 need the same fix
fixing white space and formatting
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8687: Prepare template only on a given storage poolUpdate prepare template api to seed/prepare a template only on a given primary storage. Currently, the prepare template api will seed/prepare a given template on all the primary storage pools in a zone. If however, a user wishes to prepare a template only a particular storage pool, it isn't possible. Updated the api to take storage pool id as an optional parameter. If the pool id is provided then the template is prepared only on the given primary storage pool. Also added unit tests to validate the functionality
* pr/715:
CLOUDSTACK-8687: Unit tests for validating the prepare template functionality. These tests validate that the templates get scheduled for seeding. Additionally, if a template is already seeded, we do not try and seed it again. Tests also validate that templates are seeded to storage pools which are available.
CLOUDSTACK-8687: Update prepare template api to seed/prepare a template only on a given primary storage. Currently, the prepare template api will seed/prepare a given template on all the primary storage pools in a zone. If however, a user wishes to prepare a template only a particular storage pool, it isn't possible. Updated the api to take storage pool id as an optional parameter. If the pool id is provided then the template is prepared only on the given primary storage pool
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
CLOUDSTACK-8744 Add missing localization (l10n) for several parts in the UI
- l10n for the SSH Key Pairs behavior
- l10n for Autoscaling / LB sections
- l10n for Reset password
- l10n on some strings for the installation Wizard
- l10n on some strings in VPN/VPC section
- l10n on Service offerings sections
- improve some FR translations
* pr/712:
CLOUDSTACK-8744 Add missing localization (l10n) for several parts in the UI - l10n for the SSH Key Pairs behavior - l10n for Autoscaling / LB sections - l10n for Reset password - l10n on some strings for the installation Wizard - l10n on some strings in VPN/VPC section - l10n on Service offerings sections - improve some FR translations
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
Refactored Nic.java for readability.Only changed names and reorganized them! No functional implementation change.
See issue CLOUDSTACK-8736 for ongoing effort to clean up network code.
* pr/707:
Renamed internal variables of NicProfile.java to adhere to the naming convention.
Refactored Nic.java for readability.
Signed-off-by: Rajani Karuturi <rajanikaruturi@gmail.com>
These tests validate that the templates get scheduled for seeding. Additionally,
if a template is already seeded, we do not try and seed it again. Tests also
validate that templates are seeded to storage pools which are available.
only on a given primary storage. Currently, the prepare template api
will seed/prepare a given template on all the primary storage pools in
a zone. If however, a user wishes to prepare a template only a
particular storage pool, it isn't possible. Updated the api to take
storage pool id as an optional parameter. If the pool id is provided
then the template is prepared only on the given primary storage pool
Remi Bergsma
Miguel Ferreira
Michael Andersen
Rohit Yadav
Koushik Das
Rajani Karuturi
Boris Schrijver
radu-stefanache
Devdeep Singh