Commit Graph

31705 Commits

Author SHA1 Message Date
Marc-Aurèle Brothier 3101612ad1 docker: add compose file for database (#2898)
A docker-compose file to setup quickly the database for CS. This helps during local development and remove the need to install a local MariaDB/MySQL.
2018-10-29 21:25:48 +05:30
Gabriel Beims Bräscher cdc6e6e50a CLOUDSTACK-9315: Removed unused Classes (#1448)
* Remove some unused Classes

These classes were deleted because they have no references in our code base. They are not in Spring execution flow nor instantiated with "new":
- com.cloud.agent.api.CheckStateAnswer
- com.cloud.agent.api.StartupVMMAgentCommand
- com.cloud.agent.api.routing.UserDataCommand
	- remove from description at
com.cloud.configuration.Config.ExecuteInSequenceNetworkElementCommands
enum
- com.cloud.agent.api.storage.UpgradeDiskCommand
- com.cloud.agent.api.storage.CreatePrivateTemplateCommand
- com.cloud.agent.api.storage.DestroyAnswer
	- Note: "FIXME: Should have an DestroyAnswer" at
com.cloud.storage.resource.StoragePoolResource
- com.cloud.agent.api.storage.UpgradeDiskAnswer
- com.cloud.agent.api.storage.ManageVolumeAvailabilityAnswer
- com.cloud.agent.api.storage.ManageVolumeAvailabilityCommand
- com.cloud.exception.UsageServerException
- com.cloud.info.SecStorageVmLoadInfo
- com.cloud.serializer.SerializerHelper

* PR#1448 update description of 'execute.in.sequence.network.element.commands' param

Update description of 'execute.in.sequence.network.element.commands'parameter to reflect an unused command that has been removed. The removed class command is 'UserDataCommand'.

* Add cloud schema to update SQL
2018-10-29 08:27:00 -03:00
Rohit Yadav 323d381767 Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:27:08 +05:30
Rene Diepstraten f7bc5807a3 vr: defer was broken in VR because of json name change
Committed at f0491d5c72 (#2979).

After upgrade from CS 4.10 to CS 4.11, multiple VRs did not start through.
It did not properly defer the finalize config in update_config.py.
Apparently, the json files are now called differently: where it used to
be vm_dhcp_entry.json it now has a uuid added, for example
vm_metadata.json.4d727b6e-2b48-49df-81c3-b8532f3d6745.
The if statement that checks if the finalize can be safely deferred
therefore no longer matches. This PR contains a fix so finalize is
defered again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:19:33 +05:30
Nicolas Vazquez dffb430975 kvm: Fix migrating VM from ISO failures (#2928)
Prevents errors while migrating VM from ISO:

Test 1: Deploy VM from ISO -> Live migrate VM to another host -> ERROR
Test 2: Register ISO using Direct Download on KVM -> Deploy VM from ISO -> Live migrate VM to another host -> ERROR

- Prevent NullPointerException migrating VM from ISO
- Prevent mount secondary storage on ISO direct downloads on KVM
2018-10-29 16:14:20 +05:30
Rohit Yadav f0491d5c72
vr: defer was broken in VR because of json name change (#2979)
After upgrade from CS 4.10 to CS 4.11, multiple VRs did not start through.
It did not properly defer the finalize config in update_config.py.
Apparently, the json files are now called differently: where it used to
be vm_dhcp_entry.json it now has a uuid added, for example
vm_metadata.json.4d727b6e-2b48-49df-81c3-b8532f3d6745.
The if statement that checks if the finalize can be safely deferred
therefore no longer matches. This PR contains a fix so finalize is
defered again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:11:43 +05:30
Rohit Yadav 8738ca75b1
kvm: use libvirtd as service name for all distributions (#2925)
Since we support only Ubuntu 16.04+ on master/4.12+, we can now use
the libvirt service name `libvirtd` for all distributions. This also
fixes an optional package name for libvirtd installation on Debian 9+.
Fixes #2909

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 11:55:28 +05:30
Rohit Yadav e2ba934c19
server: fix unwanted txn commit warning messages (#2927)
This fixes unwanted transaction commit warning messages such:

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 02:49:54 +05:30
Gabriel Beims Bräscher bfc326384d saml: redirect saml2 failed login message to a configurable URL (#2185)
When a user fails to authenticate with SAML2, it returns an error page
showing the content of the attached image.

To make it more user-friendly and customizable, one could configure a
desirable URL to redirect when such authentication failure happens.

This ticket proposes a global settings variable
(saml2.failed.login.redirect.url). If null, the SAML2 authentication
flow does not change from the current; however, if the user configures
an URL then ACS redirects to that URL.
2018-10-28 00:55:06 +05:30
alexanderbazhenoff a87acf93d8 kvm: improved performance on creating VM (#2923)
Improved performance on creating VM for KVM virtualization.

On a huge hosts every "ifconfig | grep" takes a lot of time (about 2.5-3 minutes on hosts with 500 machines). For example: ip link show dev $vlanDev > /dev/null is faster than ifconfig |grep -w $vlanDev > /dev/null. But using ip command is much better. Using this patch you can create 500s machine in 10 seconds. You don't need slow ifconfig prints anymore.
2018-10-25 16:28:13 +05:30
Rohit Yadav 9cf57d2568
network: on rolling restart force stop old routers (#2926)
This force stops old VRs when performing rolling restart with
cleanup=true. This will ensure that VRs are powered off quickly than
wait longer for the normal ACPI shutdown. During testing, it was found
on VMware where VM stops are slow compared to XenServer and KVM.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-25 09:20:39 +05:30
Rohit Yadav b8ed159f47 Merge remote-tracking branch 'origin/4.11' 2018-10-25 08:14:49 +05:30
Simon Weller c4b621a418 kvm: HyperV Enlightment for Improved Windows Server 2008+ Performance (#2870)
Windows has support for several paravirt features that it will use when running on Hyper-V, Microsoft's hypervisor. These features are called enlightenments. Many of the features are similar to paravirt functionality that exists with Linux on KVM (virtio, kvmclock, PV EOI, etc.)

Nowadays QEMU/KVM can also enable support for several Hyper-V enlightenments. When enabled, Windows VMs running on KVM will use many of the same paravirt optimizations they would use when running on Hyper-V.

A number of years ago, a PR was introduced that added a good portion of the code to enable this feature set, but it was never completed. This PR enables the existing features. The previous patch set detailed in #1013 also included the tests.

By selecting Windows PV, the enlightenment additions will be applied to the libvirt configuration. This is support on Windows Server 2008 and beyond, so all currently supported versions of Windows Server.

In our testing, we've seen benchmark improvements of around 20-25% running on Centos 7 hosts and it is also supported on Centos/RHEL 6.5 and later. Testing on Ubuntu would be appreciated.
2018-10-25 06:54:13 +05:30
Rohit Yadav 9b35b64b3c
packaging: install plugins at /usr/share/cloudstack-management/lib (#2915)
Install any additional plugin jars in the lib directory to be picked up
by the classpath builder, otherwise one has to manually add the jar
to /etc/default/cloudstack-management after installation. This fixes
the issue for `mysql-ha` plugin.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-24 18:21:03 +05:30
Abhishek 6f44a8d7ba template: Return requireshvm key as part of list template response (#2921)
Added requireshvm key in the response of list template API.
Fixes #2821

Signed-off-by: Abhishek Kumar <abhishek.mrt@gmail.com>
2018-10-24 16:20:27 +05:30
Rohit Yadav 8d31024a60 Merge remote-tracking branch 'origin/4.11' 2018-10-24 11:08:00 +05:30
Abhishek c8ca9e2fa0 template: create/updateTemplate should allow to set/change sshKeyEnabled (#2922)
Added sshKeyEnabled key in createTemplate and updateTemplate API.
Fixes #2822.

Signed-off-by: Abhishek Kumar <abhishek.mrt@gmail.com>
2018-10-24 11:07:03 +05:30
Rohit Yadav e092529c98
systemvm: Ensure cloud service reboots after failure (#2916)
This fixes an issue for systemvms (CPVM and SSVM) on VMware, as eth0
is not programmed (link-local) the networking.service fails to start
which is a dependency for cloud-postinit service. When cloud-postinit
service fails to start/run, it fails to start the agent (cloud) process.
This fixes the smoketest failures we saw in case of VMware 6.5 with
4.11.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-23 23:33:08 +05:30
Rafael Weingärtner 71e5a88fc1 Forward merge client: mgmt server listen default to 0.0.0.0 (#2907) 2018-10-22 16:32:06 -03:00
Abhishek d0fbd00e9b tools: Added '-B' flag for mvn current version cmd (#2910) (#2914)
maven command for finding current version might need to download packages and without batch mode it will end in variable. Added '-B' to enable batch mode.
2018-10-22 22:04:42 +05:30
Rohit Yadav 47c9c1cb58
client: mgmt server listen default to 0.0.0.0 (#2907)
This makes the management server listen on all interfaces by default.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-22 20:00:51 +05:30
Abhishek f5d52421b9 kvm: Remove libvirt-bin check for Debian (#2909) (#2912)
Debian does not have libvirt-bin package. Therefore, only for Ubuntu host distro process name libvirt-bin is used, otherwise libvirtd will be used
2018-10-22 17:48:11 +05:30
Nicolas Vazquez 5cf163d888 server: Unify templates/ISOs checksum API output (#2911)
Unify checksum API output for templates and ISOs: not list the checksum algorithm on:
KVM direct downloads

On in progress normal template downloads. The algorithm is shown on the listtemplates API, but after it is downloaded it is not shown anymore.
2018-10-21 22:33:04 +05:30
Rohit Yadav 233f46c94b Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-17 20:29:58 +05:30
Rohit Yadav 5ce14df31f
network: Allow ability to disable rolling restart feature (#2900)
This adds a global setting for admins who may not want the rolling
restart of routers or are seeing any issues around it. In future, this
setting may be removed.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-17 20:27:08 +05:30
Rohit Yadav e871638dc8 Merge remote-tracking branch 'origin/4.11' 2018-10-17 13:25:06 +05:30
Rohit Yadav 1904a70512
agent: on shutdown don't allow server reconnection (#2904)
When agent is stopped, don't allow reconnection. Previously this would
send a shutdown command to the management server which would put the
host state to Disconnected but then agent's reconnection logic may kick
in sometimes which would connect the agent to the management server
but then the agent process would terminate causing the host to be
put in Alert state (due to ping timeout or it waiting too long).

This fixes the issue by ensuring that when the agent is stopped, it
does not reconnect to the management server.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-17 06:31:13 +05:30
Gabriel Beims Bräscher 143fe5375c Allow enable the Java remote Debug for CloudStack Agent (#2897)
The idea is to create a property in /etc/defaults/cloudstack-agent that
can be enabled/disabled (uncommented/commented) to control the enabling
of Java remote debug in CloudStack's JVM.
2018-10-16 15:57:13 -03:00
Rafael Weingärtner 92f8cda64a Merge forward #2902 (Add checksum sanity validation on template registration) 2018-10-16 10:25:50 -03:00
Nicolas Vazquez 9003c7bfdc Add checksum sanity validation on template registration (#2902)
* Add checksum sanity validation on template registration

* Refactor

* Rename checksum sanity method
2018-10-16 10:21:20 -03:00
Pierre-Yves Ritschard 58b4e71b0e
params: allow signatureversion and expires without logging
This patch considers the new expires and signatureversion parameters
valid. Without this, all calls log when using the V3 signature scheme.
2018-10-16 09:01:46 +02:00
Rohit Yadav 84994c841f Merge remote-tracking branch 'origin/4.11' 2018-10-16 10:54:39 +05:30
Nicolas Vazquez 11d83fab43 agent: set log level to INFO as default for http wire (#2903)
Avoid logging bytes on direct download on KVM.
2018-10-16 10:32:03 +05:30
Rohit Yadav 933ee23104
vr: memory and swap optimizations (#2892)
This tries to provide a threshold based fix for #2873 where swappinness of VR is not used until last resort. By limiting swappiness unless actually needed, the VR system degradation can be avoided for most cases. The other change is around not starting baremetal-vr by default on all VRs, according to the spec https://cwiki.apache.org/confluence/display/CLOUDSTACK/Baremetal+Advanced+Networking+Support only vmware VRs need to run it and that too only as the last step of the setup/completion, so we don't need to run it all the time.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-16 10:29:48 +05:30
Rafael Weingärtner 272167c3b3 merge forward #2876 -- PULL_REQUEST_TEMPLATE simplification 2018-10-15 15:33:13 -03:00
Rohit Yadav 63f4d852d5 PULL_REQUEST_TEMPLATE: simplify and remove unpopular sections (#2876)
This removes the section from the pull request template that is not very
popular or filled by the PR author.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-15 15:21:13 -03:00
Rafael Weingärtner ebc6b2f706 Remove push force test file 2018-10-13 09:40:23 -03:00
Rafael Weingärtner 1758a75a54 test force push 2018-10-12 10:58:41 -03:00
Rohit Yadav bd9880003f Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-10 16:10:26 +05:30
Rohit Yadav ea771cfda4
router: Fixes #2719 program VR nics by device id order for VPC (#2888)
This fixes #2719 where private gateway IP might be incorrectly
programmed on a guest network nic. The VR would now check ipassoc
requests by mac addresses than provided nic/device id in case they are
wrong.

The root cause is that the device id information is lost when aggregated
commands are created upon starting of a new VPC VR, without the correct
device id in ip_associations json it mis-programs the VR.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-10 15:20:36 +05:30
Frank Maximus a6196b0a60 Fixes: #2881 Improve Exception message (#2889)
Network.Service and Network.Provider were missing a toString() method.
Added this so appending (a list of) them will be understandable.
2018-10-09 15:43:48 +05:30
Paul Angus e73fb765d2 ui: Display i18n language name in the same language (#2883)
This update changes how the languages are shown on the UI's login page. Previously all languages were shown in the current language of the login page. This meant for instance, that a Japanese speaker would have to know English to find their language. Likewise if the language had been changed, a German speaker might need to speak Korean in order to find their language.

Therefore the UI has been altered to always show the name of each language in the appropriate language.
2018-10-09 05:57:10 +05:30
Rohit Yadav 80debcb409 Merge remote-tracking branch 'origin/4.11' 2018-10-09 05:44:10 +05:30
Paul Angus 37ecfe2d28
Merge pull request #2884 from shapeblue/usage-server-timstamp
add date to usage server logs

Merged based on 2x LGTM and checking errors in smoke tests - none in any way related to the logging output change.
2018-10-08 15:20:51 -04:00
Rohit Yadav 1fa4f10a9e Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-07 21:12:20 +05:30
Rohit Yadav f430f41edd
ca: Fixes #2877 mgmt server cert should have all addrs of default nic (#2879)
This fixes the default RootCA provider implementation to initiate
and issue certificate for mgmt server on startup for all the IP addresses
on the default nic of that host.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-07 21:07:10 +05:30
Paul Angus 35656553ca add date to usage server logs 2018-10-06 17:20:17 +01:00
Rohit Yadav 5313c04b21 Merge remote-tracking branch 'origin/4.11' 2018-10-05 18:36:37 +05:30
Simon Weller 5db65a6363 kvm: Fixes #2868 libvirt resize notify failure (#2878)
Incorrect diskpath information was being sent to virsh blockresize, so the block device size was never refreshed to reflect the new disk size.
Fixes #2868
2018-10-05 18:35:09 +05:30
Rohit Yadav b6302d4e90 Merge remote-tracking branch 'origin/4.11'
Conflicts resolved for:
	engine/orchestration/src/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java
	engine/schema/src/com/cloud/vm/dao/UserVmDaoImpl.java
server/src/com/cloud/network/element/VirtualRouterElement.java
server/src/com/cloud/vm/UserVmManagerImpl.java
tools/marvin/setup.py

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-05 05:18:42 +05:30