Commit Graph

421 Commits

Author SHA1 Message Date
Greg Goodrich 2bbc78170b
packaging: Issue #3974 Deploying mysql-ha jar file into its own path to allow for easy inclusion of just this jar file into multiple packages, such as cloudstack-management and cloudstack-usage. (#3975)
Support for centos7 and centos8

Prior PR #2915 causes the cloudstack-usage daemon to not be able to find the mysql-ha jar file.

This happens because the /etc/default/cloudstack-usage file points to the old location for the mysql-ha jar file, and thus is unable to load it. This prior PR installs this jar into a more common area with other jar files, and thus is not really a desired location for the cloudstack-usage daemon to get it from, as it will cause cloudstack-usage to load other plugins that it doesn't need.

Fixes: #3974
2021-09-08 10:31:11 +05:30
davidjumani f17683779c
packaging: Adding SUSE 15 support (#5110)
This PR adds support for SUSE Leap 15.2+ as a KVM hypervisor and Management / Usage server on Cloudstack
Related docs link apache/cloudstack-documentation#224

Co-authored-by: Michael <35783820+mib1185@users.noreply.github.com>
2021-08-19 13:28:54 +05:30
sureshanaparti 0d72886ee3
Replace the DB properties having master and slave(s), with source and replica(s) respectively, on Upgrade for inclusiveness (#5162)
Updates the DB properties (with strict / full property string search) having master and slave(s), with source and replica(s) respectively on upgrade (for inclusiveness).

Signed-off-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com>
2021-07-01 10:12:59 +05:30
Rohit Yadav f58b72f6f7 Merge remote-tracking branch 'origin/4.15' 2021-06-27 18:25:46 +05:30
davidjumani 5195f8b4b7
packaging: Create cloud user and group if not present (#5115) 2021-06-24 11:08:39 +05:30
dahn a1f825e7c7
python3: Migrate Marvin and smoketests to python3 (#4727)
This PR prepares marvin and tests for python3. it was part of #4479, until nose2 was decided to be abandoned from that PR.

Re-PR of #4543 and #3730 to enable cooperation

Co-authored-by: Daan Hoogland <dahn@onecht.net>
Co-authored-by: Gabriel Beims Bräscher <gabriel@apache.org>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-05-04 23:19:37 +05:30
Rohit Yadav 1abd10199c Merge remote-tracking branch 'origin/4.15' 2021-05-04 19:37:45 +05:30
Pearl Dsilva 2d176db9a4
centos: Install libgcrypt v1.8.5 required by libvirt 6.0 on CentOS8 (#4970)
Fixes: #4969
This PR upgrades the version of libgcrypt that is required by libevirt 6.0

Co-authored-by: Pearl1594 <pearl.dsilva@shapeblue.com>
2021-04-30 12:15:09 +05:30
Rohit Yadav 77290df0d5 Merge remote-tracking branch 'origin/4.15'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-02-26 12:09:11 +05:30
Abhishek Kumar d0e71adbbf
packaging: update Requirements in README (#4725)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2021-02-24 17:46:01 +05:30
Wei Zhou 51a0864937
packaging: new timestamp format and add output dir option (#4471)
* Packaging: new timestamp format and add output dir option

* #4471 packaging: echo message after moving packages
2021-02-19 14:26:13 +05:30
Rohit Yadav abec6d1fff
packaging: build and bundle UI using npm in deb and rpm packages (#4605)
This changes deb and rpm packaging to build the UI using npm and bundle
it in the `cloudstack-management` package and a new `cloudstack-ui`
package. The `cloudstack-ui` package will install the UI under
`/usr/share/cloudstack-ui/`. For both packages the config.json will not
be overridden on upgrade and hosted at /etc/cloudstack/management
for the cloudstack-mangement package, and at /etc/cloudstack/ui for the
cloudstack-ui package. The cloudstack-ui package is for advanced users
who only want the UI want to setup reverse proxy (separate hosting of UI).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2021-01-28 14:29:31 +05:30
Rohit Yadav db3fdf4142 Merge remote-tracking branch 'origin/4.14' 2020-10-19 13:15:59 +05:30
Rohit Yadav b27b8d0f0a
packaging: enable Parallel Collector GC for management server (#4407)
The default GC algorithm G1 that is enabled by default with Java11
serves well on multiprocessor machines with large amount of memory where
GC is probablistic with low pauses, where response time is more
important than throughput and GC is kept shorter.

The CloudStack management server is largely a multi-threaded server
application that handles and orchestrates several network requests, and
has the default max. heap size of only 2G that can be considered a
small/medium application from a heap size perspective. Perhaps a more
aggresive GC algorithm such as ParallelGC as used in Java8 and before
(that is previous CloudStack releases) would serve better for throughput
and cause more aggresive GC.

Reference: https://docs.oracle.com/en/java/javase/11/gctuning/available-collectors.html#GUID-13943556-F521-4287-AAAA-AE5DE68777CD

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-10-19 13:10:53 +05:30
slavkap d213a4d956
systemd: Binding listening socket to all address for remote debug (#4345)
Since Java 9 the notation 'address=port' only applies to localhost.
For remote debug you have to explicitly specify that you want to listen
to all IP addresses (e.g. address=*8000)
2020-09-29 12:26:14 +05:30
Andrija Panic 238eccc317
packaging: Minor message update (#4333)
adding quotes, to fix the "servers" to "server's"
2020-09-21 14:12:52 +05:30
Andrija Panic cddf69c0f8
Usage-server update message improvement (#4278)
* Update cloud.spec

* Update cloud.spec

* Update cloudstack-usage.postinst
2020-08-23 16:04:38 +05:30
davidjumani d949302d0f
packaging: Adding Centos8, Ubuntu 20.04, XCPNG8.1 Support (#4068)
* DB : Add support for MySQL 8

- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters

For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:

    server_id = 1
    sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
    innodb_rollback_on_timeout=1
    innodb_lock_wait_timeout=600
    max_connections=1000
    log-bin=mysql-bin
    binlog-format = 'ROW'

    default-authentication-plugin=mysql_native_password

Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.

Developers can set empty password as follows:

    > sudo mysql -u root
    ALTER USER 'root'@'localhost' IDENTIFIED BY '';

In libvirt repository, there are two related commits

2019-08-23 13:13 Daniel P. Berrangé            ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé            ● remote: forbid the --listen arg when systemd socket activation

In libvirt.spec.in

        /bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :

Co-authored-by: Wei Zhou <w.zhou@global.leaseweb.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-08-17 16:28:30 +05:30
harikrishna-patnala a279d5c453
logging: Logging framework to use only log4j (#4003)
Currently CloudStack is using logging frameworks as log4j and Java util logging, logging wrappers as slf4j and Apache common logging.
Here changes are to made it uniform, using only log4j framework.
Removed Java util logging, slf4j and Apache common logging.
2020-06-17 07:11:23 +05:30
NuxRo db55910f6b
packaging: missing python3 libvirt dependency for CentOS7 (#4124)
Missing python3 libvirt bindings on CentOS7 effectively break security groups.
There are 0 firewall rules added. The agent logs report:

```2020-06-02 10:58:34,346 DEBUG [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Traceback (most recent call last):  File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 26, in <module>    import libvirtModuleNotFoundError: No module named 'libvirt'
```

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-06-04 07:09:14 +05:30
Daan Hoogland da865f08d3 Merge release branch 4.13 to master
* 4.13:
  systemd dependency on db (#3973)
2020-03-25 19:59:58 +01:00
dahn d93c2459a4
systemd dependency on db (#3973) 2020-03-25 19:57:12 +01:00
Nicolas Vazquez efe00aa7e0
[KVM] Rolling maintenance (#3610) 2020-03-12 16:59:46 +01:00
Rohit Yadav d90341ebf1
cloudstack: add JDK11 support (#3601)
This adds support for JDK11 in CloudStack 4.14+:

- Fixes code to build against JDK11
- Bump to Debian 9 systemvmtemplate with openjdk-11
- Fix Travis to run smoketests against openjdk-11
- Use maven provided jdk11 compatible mysql-connector-java
- Remove old agent init.d scripts

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-02-12 12:58:25 +05:30
Wei Zhou cd55674afe packaging: install python-dnspython or python-dns to fix issue with cloudstack-setup-management (#3854)
Fixes #3817 #3841

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-02-07 11:18:00 +05:30
Rohit Yadav 493415d91b
packaging: install python-dnspython or python-dns to fix issue with cloudstack-setup-management (#3854) (#3869)
Fixes #3817 #3841

Co-authored-by: Wei Zhou <ustcweizhou@gmail.com>
2020-02-07 11:13:57 +05:30
Anurag Awasthi 1771727ce1 Revert "Extract systemvm.iso using bsdtar (#3536)" (#3800)
This reverts commit 8a55c931e8.
2020-01-13 12:24:43 +01:00
Kai Takahashi 8a55c931e8 Extract systemvm.iso using bsdtar (#3536)
* Extract systemvm.iso using bsdtar if available.

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>

* New dependency for CentOS 7 and Debian: bsdtar

bsdtar can extract iso images without mounting.

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>

* Remove all 'mount' and 'umount' command call(s).

Signed-off-by: Kai Takahashi <k-takahashi@creationline.com>
2020-01-06 11:46:20 +01:00
Andrija Panic 45503e104a
Revert "Revert "Fix virtual template size for managed storage for KVM / refactor cloud-install-sys-tmplt (#3371)" (#3771)" (#3772)
This reverts commit e319c8b8f0.
2019-12-19 19:50:30 +01:00
Andrija Panic e319c8b8f0
Revert "Fix virtual template size for managed storage for KVM / refactor cloud-install-sys-tmplt (#3371)" (#3771)
This reverts commit a0efbf9d09.
2019-12-17 15:29:37 +01:00
Sven Vogel a0efbf9d09
Fix virtual template size for managed storage for KVM / refactor cloud-install-sys-tmplt (#3371)
* remove unused chksum flag
* refactor cloud-install-sys-tmplt / add qemu-img command to calculate correct virtual size
* add qemu-utils dependency to debian build control file
* add qemu-utils dependency to centos spec file
2019-12-17 15:27:03 +01:00
Rohit Yadav e93f6bf3c2
utils: use iproute to get default network interface (#3704)
Use iproute to get default network interface

This uses `ip route` than `route` without an explicit path (/bin, /sbin
etc.) to query the default network interface on the system.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-11-27 09:35:22 +05:30
fc9a525adc New BuildRequires for CentOS 7: python-setuptools (#3630)
(It's already added for debian from
commit 4a77a79949)

Signed-off-by: Kai Takahashi <www.carrotsoft@gmail.com>
2019-11-21 11:32:20 +05:30
Wido den Hollander 899eab66c1 kvm/security_group: Make Security Group Python 3 compatible (#3589)
* kvm/security_group: Make Security Group Python 3 compatible

This script only runs on the KVM Hypervisors and these all support
Python 3.

As Python 2 is deprecated at the end of 2019 we need to fix these
scripts to work under Python 3.

CentOS 7, 8 and Ubuntu 16.04 and 18.04 all have Python 3 installed
by default.

Ubuntu 20.04 will no longer have Python 2 installed and therefor
this script needs to be modified to work with Python 3.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* Add dependency of python3 in packaging/centos7/cloud.spec
2019-09-26 18:17:03 +05:30
Rohit Yadav 89053c7612
packaging: deprecate el6 and add 4.13-4.14 upgrade path (#3591)
- Removes CentOS6/el6 packaging (voting thread reference https://markmail.org/message/u3ka4hwn2lzwiero)
- Add upgrade path from 4.13 to 4.14
- Enable live storage migration support for KVM by default as el6 is deprecated
- PRs using live storage migration
  #2997 KVM VM live migration with ROOT volume on file storage type
  #2983 KVM live storage migration intra cluster from NFS source and destination
  #2298 CLOUDSTACK-9620: Enhancements for managed storage

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-09-19 10:01:08 +05:30
Wido den Hollander 8da0556f63 kvm/cloudstack-guest-tool: Tool to query Qemu Guest Agent (#3519)
Using this tool on a hypervisor admins can query KVM Instances running
on that hypervisor if they have the Qemu Guest Agent installed.

All System VMs have this and they can be queried.

For example:

$ cloudstack-guest-tool i-2-25-VM

This will print some information about network and filesystem status.

root@hv-138-a05-23:~# ./cloudstack-guest-tool s-11-VM --command info|jq
{
  "network": [
    {
      "ip-addresses": [
        {
          "prefix": 8,
          "ip-address": "127.0.0.1",
          "ip-address-type": "ipv4"
        }
      ],
      "name": "lo",
      "hardware-address": "00:00:00:00:00:00"
    },
    {
      "ip-addresses": [
        {
          "prefix": 16,
          "ip-address": "169.254.242.169",
          "ip-address-type": "ipv4"
        }
      ],
      "name": "eth0",
      "hardware-address": "0e:00:a9:fe:f2:a9"
    },
 ...
 ...
  "filesystem": [
    {
      "mountpoint": "/var",
      "disk": [
        {
          "bus": 0,
          "bus-type": "virtio",
          "target": 0,
          "unit": 0,
          "pci-controller": {
            "slot": 7,
            "bus": 0,
            "domain": 0,
            "function": 0
          }
        }
      ],
      "type": "ext4",
      "name": "vda6"
    },

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-09-10 09:31:03 -07:00
Sven Vogel 9d02b3679b packaging: set correct logrotate file permissions for cloudstack-agent (#3347)
Correct the permissions on cloudstack-agent logrotate file (/etc/logrotate.d/).
2019-05-24 12:43:08 +05:30
Rohit Yadav 9488c6dcc5 Merge remote-tracking branch 'origin/4.11' into 4.12 2019-04-09 14:22:18 +05:30
Rohit Yadav 96611fc640
packaging: systemctl daemon-reload after agent install or upgrade (#3269)
This runs systemctl daemon-reload after cloudstack-agent is installed
or upgraded.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-04-09 14:21:09 +05:30
Rohit Yadav 55efaf14d9
packaging: don't skip unit tests while building packages (#3266)
This may slow down CI and release, but ensures that unit tests always
run as part of the packaging build process.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-04-08 13:51:30 +05:30
Rohit Yadav a6f4d6d312 Merge remote-tracking branch 'origin/4.11' 2019-03-14 17:55:33 +05:30
Rohit Yadav f7327c7457 systemd: Fix -Dpid arg passing to systemd usage service (#3210)
* systemd: Fix -Dpid arg passing to systemd usage service

This fixes regression introduced by refactoring PR #3163 where `-Dpid`
was incorrectly passed string `$$` instead of parent PID integer.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* fix systemd limitation, exec using /bin/sh instead and wrap in ${} syntax

https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* usage: don't hide exception from Gabriel's https://github.com/apache/cloudstack/pull/3207/files#diff-062fcf5ae32de59dfd6cd4f780e1d7cd

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-14 09:19:12 -03:00
GabrielBrascher 3f17671449 Fix conflict and merge forward PR #3163 from 4.11 to master (4.12)
# Conflicts:
#	packaging/debian/init/cloud-management
#	packaging/systemd/cloudstack-agent.default
#	packaging/systemd/cloudstack-agent.service
#	packaging/systemd/cloudstack-management.service
2019-02-04 23:53:19 -02:00
Rohit Yadav cb3fed0e4e systemd: fix services to allow TLS configurations via java.security.ciphers (#3163)
* systemd: fix services to allow TLS configurations via java.security.ciphers

This fixes the management server and systemd services to allow the
java.security.ciphers file to configure disabled TLS protocols and
algorithms. This also cleans up systemd service files for agent and
usage server.

This fixes #3140

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* configure: fix travis failure due pycodestyle error

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-02-04 19:51:30 -02:00
GabrielBrascher ff07c17a26 Foward merge #3139 from 4.11 to master (4.12) 2019-02-04 19:05:11 -02:00
Rohit Yadav 463372bc7e
packaging: management default file cleanup (#3139)
This cleanups management server default file, the `cloud.jks` is no
longer created by the management server but instead created in-memory
by the root CA plugin on management server startup.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-01-25 22:19:33 +05:30
Rafael Weingärtner df6288f4bc
Extend PR#2535 to enable remote debugging for CentOS63 as well (#3128)
This PR is an extension of #2535. By doing this, we also create an easy method to enable remote debug in CentOS63 as well.
2019-01-14 22:15:01 -02:00
Rohit Yadav 52f68a273a Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-12-04 16:39:21 +05:30
Sven Vogel 17097929b6 packaging: correct permissions in spec file and fix class path specified variable (#3030)
Install CentOS 7 e.g. Build 1804 and Java build 1.8.0_181

if you inspect systemd in debug mode you will see some errors
1.
permission of the cloudstack-managment.service are not corretly set
2.
invalid classpath specified. it seems the string which is used will be divided... we now we use ${..} like the lines above ... confused
2018-12-01 01:38:01 +05:30
Rohit Yadav b8ed159f47 Merge remote-tracking branch 'origin/4.11' 2018-10-25 08:14:49 +05:30
Rohit Yadav 9b35b64b3c
packaging: install plugins at /usr/share/cloudstack-management/lib (#2915)
Install any additional plugin jars in the lib directory to be picked up
by the classpath builder, otherwise one has to manually add the jar
to /etc/default/cloudstack-management after installation. This fixes
the issue for `mysql-ha` plugin.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-24 18:21:03 +05:30
Gabriel Beims Bräscher 143fe5375c Allow enable the Java remote Debug for CloudStack Agent (#2897)
The idea is to create a property in /etc/defaults/cloudstack-agent that
can be enabled/disabled (uncommented/commented) to control the enabling
of Java remote debug in CloudStack's JVM.
2018-10-16 15:57:13 -03:00
Rohit Yadav b6302d4e90 Merge remote-tracking branch 'origin/4.11'
Conflicts resolved for:
	engine/orchestration/src/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java
	engine/schema/src/com/cloud/vm/dao/UserVmDaoImpl.java
server/src/com/cloud/network/element/VirtualRouterElement.java
server/src/com/cloud/vm/UserVmManagerImpl.java
tools/marvin/setup.py

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-05 05:18:42 +05:30
Rohit Yadav c2f4b3653d
packaging: Fixes #2857 don't overwrite agent logrotate config (#2860)
This makes the agent logrotate config to `noreplace` so on upgrade
any changes to the file are not lost.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-09-25 11:25:06 -04:00
Wido den Hollander 7313a5ead2 deb: Remove Ubuntu 14.04 LTS support and update DEB dependencies (#2828)
Ubuntu 14.04 will go EOL in April 2019. With a new CloudStack
release close to that date we can drop support for this Ubuntu
version and the master branch of CloudStack.

Ubuntu 16.04 (Xenial) and 18.04 (Bionic) both have systemd and
more recent Java versions which make it easier to run the CloudStack
KVM Agent on them.

In addition libvirt and Qemu are more up to date with features
which allow VMs to run better.

Yet to be implemented features in KVM can also leverage the newer
version of Qemu and libvirt without the need of taking older
version of them into account.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-09-18 16:01:37 +05:30
Bruno P. Kinoshita 34ada079d3 s/pacakge/package (#2812) 2018-08-20 10:14:25 +02:00
Khosrow Moossavi 4a5fab0573 packaging: Catch error in packaging script and fail the build (#2649)
This is to fix the regression added in #2433. In this fix
we're going to fail the build early if --use-timestamp is
provided and working directory is *not* clean. And also
fail the build if something in the script has returned
non-zero value.
2018-07-13 16:47:34 +05:30
Rohit Yadav 85750f918b Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-06-20 12:31:52 +05:30
Rohit Yadav c87ca1b262 packaging: use libuuid x86_64 package for cloudstack-common (#2706)
* packaging: use libuuid x86_64 package for cloudstack-common

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* 64 bit links is packaged

* post scan filter to exclude libuuid.so.1

* Revert "packaging: use libuuid x86_64 package for cloudstack-common"

This reverts commit b3fb8957fe.

* post scan filter to exclude libuuid.so.1 (centos63)

* revert removal of 32 bit support for vhd-util libs
2018-06-19 13:34:44 +02:00
Rohit Yadav 7c6777b8d3 Merge branch '4.11': allow config drives on primary storage for KVM (#2651)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 14:50:55 +05:30
Rohit Yadav acc5fdcdbd
CLOUDSTACK-10290: allow config drives on primary storage for KVM (#2651)
This introduces a new global setting `vm.configdrive.primarypool.enabled` to toggle creation/hosting of config drive iso files on primary storage, the default will be false causing them to be hosted on secondary storage. The current support is limited from hypervisor resource side and in current implementation limited to `KVM` only. The next big change is that config drive is created at a temporary location by management server and shipped to either KVM or SSVM agent via cmd-answer pattern, the data of which is not logged in logs. This saves us from adding genisoimage dependency on cloudstack-agent pkg.

The APIs to reset ssh public key, password and user-data (via update VM API) requires that VM should be shutdown. Therefore, in the refactoring I removed the case of updation of existing ISO. If there are objections I'll re-put the strategy to detach+attach new config iso as a way of updation. In the refactored implementation, the folder name is changed to lower-cased configdrive. And during VM start, migration or shutdown/removal if primary storage is enable for use, the KVM agent will handle cleanup tasks otherwise SSVM agent will handle them.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 14:27:23 +05:30
Khosrow Moossavi 1cc571015c packaging: don't skip tests while packaging Centos7 (#2646) 2018-05-16 13:18:08 +05:30
Khosrow Moossavi 11075953f6 packaging: Catch error in packaging script and fail the build (#2627)
This is to fix the regression added in #2433. In this fix
we're going to fail the build early if --use-timestamp is
provided and working directory is *not* clean.
2018-05-09 21:43:07 +05:30
Rohit Yadav 50042680b0 Merge branch '4.11' 2018-04-10 00:31:38 +05:30
ernjvr bb09caf8dc packaging: remove deprecated tomcat based sysconfig file for centos (#2538)
Remove deprecated tomcat configuration file instead of moving it, since moving it prevented the 4.11 upgrade from overwriting it with the new jetty configurations. Also added systemctl daemon-reload to refresh daemon services so that management server can start up seamlessly after the upgrade.

Fixes #2537
2018-04-10 00:31:09 +05:30
Rafael Weingärtner e198ff21b3
Create an easy way to enable the Java remote Debug for ACS (#2535)
The idea is to create a property in /etc/defaults/cloudstack-management that can be enabled/disabled (uncommented/commented) to control the enabling of Java remote debug in CloudStack's JVM.
2018-04-06 07:17:17 -03:00
Daan Hoogland 9124f97447 Merge release branch 4.11 to master
* 4.11:
  CLOUDSTACK-10287 package require open jdk 1.8 (#2519)
  make Broadcast- and IsolationURI visible to admin (#2520)
2018-04-03 08:18:37 +02:00
dahn cff4032f00
CLOUDSTACK-10287 package require open jdk 1.8 (#2519) 2018-04-02 22:01:41 +00:00
Rohit Yadav 95d10c251a
[Merge 4.11] CLOUDSTACK-10227: Fix mkdir path for centos rpms (#2513)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-03-27 22:53:59 +05:30
Rohit Yadav 48e6989da4
CLOUDSTACK-10227: Fix mkdir path for centos rpms (#2513)
This fixes the mkdir path `/usr/bin` for el6/7 rpms.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-03-27 22:52:37 +05:30
Khosrow Moossavi af52b1a873 CLOUDSTACK-10268: Fix typo (#2495)
This fixes typo introduced in PR #2433
2018-03-19 11:46:29 +05:30
Khosrow Moossavi 1708838518 CLOUDSTACK-10268: Fix and enhance package script (#2433)
- new flag `-T, --use-timestamp` to use `timestamp` when POM version contains SNAPSHOT
  - in the final artifacts (jar) name
  - in the final package (rpm, deb) name
  - in `/etc/cloudstack-release` file of SystemVMs
  - in the Management Server > About dialog
- if there's a "branding" string in the POM version (e.g. `x.y.z.a-NAME[-SNAPSHOT]`),
the branding name will be used in the final generated pacakge name such as following:
  - `cloudstack-management-x.y.z.a-NAME.NUMBER.el7.centos.x86_64`
  - `cloudstack-management_x.y.z.a-NAME-NUMBER~xenial_all.deb`
- branding string can be overriden with newly added `-b, --brand` flag
- handle the new format version for VR version
- fix long opts (they were broken)
- tolerate and show a warning message for unrecognized flags
- usage help reformat

* Deprecate Version class in favor or CloudStackVersion
2018-03-15 10:55:29 +00:00
Marc-Aurèle Brothier 893a88d225 CLOUDSTACK-10105: Use maven standard project structure in all projects (#2283)
Remove maven standard module (which only a few were using) and get ride of maven customization for the projects structure.

- moved all directories to src/main/java, src/main/resources, src/main/scripts, src/test/java, src/test/resources
- grep scan to search for src/com and src/org left over
- grep for <project>/scripts to fix pom.xml configuration
- remove custom <build> configuration in pom.xml

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2018-01-20 03:19:27 +05:30
Rohit Yadav fa7c1e2e65
CLOUDSTACK-10227: Stabilization fixes for 4.11.0.0 (#2403)
This fixes regression failures seen in Trillian, fixes NPEs that cause Travis related failures.
This also removes the aria2 dependency from rpms that require users to enable/install epel-release.
This finally updates the checksums for 4.11 systemvmtemplates in db upgrade path.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-15 00:08:36 +05:30
Nicolas Vazquez e86bb41e0e CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates (#2379)
This feature allows using templates and ISOs avoiding secondary storage as intermediate cache on KVM. The virtual machine deployment process is enhanced to supported bypassed registered templates and ISOs, delegating the work of downloading them to primary storage to the KVM agent instead of the SSVM agent.

Template and ISO registration:
- When hypervisor is KVM, a checkbox is displayed with 'Direct Download' label.
- API methods registerTemplate and registerISO are both extended with this new parameter directdownload.
- On template or ISO registration, no download job is sent to SSVM agent, CloudStack would only persist an entry on template_store_ref indicating that template or ISO has been marked as 'Direct Download' (bypassing Secondary Storage). These entries are persisted as:
template_id = Template or ISO id on vm_template table
store_id NULL
download_state = BYPASSED
state = Ready
(Note: these entries allow users to deploy virtual machine from registered templates or ISOs)
- An URL validation command is sent to a random KVM host to check if template/ISO location can be reached. Metalink are also supported by this feature. In case of a metalink, it is fetched and URL check is performed on each of its URLs.
- Checksum should be provided as indicated on #2246: {ALGORITHM}CHKSUMHASH
- After template or ISO is registered, it would be displayed in the UI

Virtual machine deployment:
When a 'Direct Download' template is selected for deployment, CloudStack would delegate template downloading to destination storage pool via destination host by a new pluggable download manager.
Download manager would handle template downloading depending on URL protocol. In case of HTTP, request headers can be set by the user via vm_template_details. Those details should be persisted as:
Key: HTTP_HEADER
Value: HEADERNAME:HEADERVALUE

In case of HTTPS, a new API method is added uploadTemplateDirectDownloadCertificate to allow user importing a client certificate into all KVM hosts' keystore before deployment.
After template or ISO is downloaded to primary storage, usual entry would be persisted on template_spool_ref indicating the mapping between template/ISO and storage pool.
2018-01-09 12:22:18 +05:30
lmrv 6ab46b3c29 CLOUDSTACK-9501: route is not available by default on CentOS7 (#1637)
This adds the `net-tools` dependency on CentOS cloudstack-agent rpms.
This will provide ifconfig, route and other tools that may be used
by CloudStack scripts and utilities.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 09:22:44 +05:30
Marc-Aurèle Brothier a372040798 CLOUDSTACK-10012: Load SQL schema scripts from JAR (#2247)
Load SQL schema scripts from the uber/fat JAR. This also removes the setup/db/db directory from centos packaging.
2017-12-19 12:11:38 +05:30
Marc-Aurèle Brothier 9c067c0496 CLOUDSTACK-10137: Only link log4j if not already present (#2316)
Use ln -sf to add link log4j.xml.

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2017-11-10 19:59:59 +05:30
Rohit Yadav eda3b35bfa CLOUDSTACK-10012: Migrate to Embedded Jetty
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
  deprecates the need to create/store cloud.jks in conf folder and
  the db.cloud.keyStorePassphrase in db.properties file. This also
  remove the need of the --keystore-passphrase in the
  cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-03 23:57:25 +05:30
Marc-Aurèle Brothier e9f29c787d CLOUDSTACK-10012: Introduce Embedded Jetty with shaded jar
- Changes packaging to shaded jar
- Introduced embedded jetty
- Updates jetty dependencies version 9.2 (java 7+)
- Initial changes to debian config files
- Initial removal of tomcat dependency
- Introduce ServerDaemon class with customizations
2017-11-03 23:57:25 +05:30
Rohit Yadav 7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
Rohit Yadav ffddd6db09 Merge remote-tracking branch 'origin/4.9' into 4.10 2017-07-22 11:07:42 +02:00
Rohit Yadav 449ff979bd packaging: Improve post-upgrade processing
$1 is "2" during package upgrade in %post section, this fixes the
handling of $1 as per https://fedoraproject.org/wiki/Packaging:Scriptlets

This improves handling of $1 during %post upgrade step. Some of the
command/code are idempotent such as enabling and starting a service
and can be run without any $1 checks.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-07-22 10:54:44 +02:00
Rajani Karuturi 11bc262d4b Merge pull request #2053 from wido/deb-binary
deb: Only build binary packages
2017-04-27 14:33:54 +05:30
Rajani Karuturi f6edc1b63f Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9871: Set SQL Mode in SQL Session for MySQL 5.7 compatibility
2017-04-27 11:21:07 +05:30
Rajani Karuturi cab0869bd4 Merge pull request #2037 from wido/sql_mode
CLOUDSTACK-9871: Set SQL Mode in SQL Session for MySQL 5.7 compatibility
2017-04-27 11:16:33 +05:30
Rajani Karuturi f0dd5994b4 Merge pull request #1582 from DaanHoogland/CLOUDSTACK-9408
CLOUDSTACK-9408 for the move away from download.cloud.com
2017-04-24 07:57:07 +05:30
Rajani Karuturi 9cc3ae8a94 Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9857: With this change if agent dies the systemd will catch it properly and show process as exited
  CLOUDSTACK-9805: Display VR list in network details
  CLOUDSTACK-9356: FIX Cannot add users in VPC VPN
2017-04-20 19:32:20 +05:30
Daan Hoogland 70ef0788c9 CLOUDSTACK-9408: Fix download urls in sql and scripts
This fixes the agreed upon url on download.cloudstack.org in various
sql files and misc scripts.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-04-20 12:33:33 +05:30
Daan Hoogland f07d729086 CLOUDSTACK-9408 for the move away from download.cloud.com
- commented some occurences of cloud.com as being harmless
  * examples
  * identifiers (internal)
 - changed the URL for vhd-util download
 - changed comments from 'cloud.com' to 'Apache CloudStack'
2017-04-20 11:35:36 +05:30
Wido den Hollander 9e634d98cc
deb: Only build binary packages
We do not need to build a tarball before building the DEB packages

Saves a few minutes on building DEB packages

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-04-19 20:20:19 +02:00
Wido den Hollander 0c1c27ba85
CLOUDSTACK-9871: Set SQL Mode in SQL Session for MySQL 5.7 compatibility
MySQL 5.7 has a more strict SQL mode by default with which CloudStack
is not compatible.

By setting the SQL Mode to a more relaxed mode on run-time we can
run without changing any SQL server settings.

Admins could also apply this to the [mysqld] section of their my.cnf:

sql_mode = 'STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-04-12 15:13:08 +02:00
Wei Zhou 4a77a79949 CLOUDSTACK-9462: Build packages on Ubuntu 14.04/16.04 and support tomcat6/tomcat7
Changes
(1) add systemd support in debian/control and debian/rules for 16.04
(2) add python-setuptools in Build-Depends in debian/control
(3) seperate cloudstack-management.service and cloudstack-management.default for CentOS7 and Ubuntu 16.04
(4) add server7-ssl.xml and server7-nonssl.xml in management installation
(5) link /usr/share/cloudstack-management/lib and /usr/share/cloudstack-management/bin to correct path (tomcat6 or tomcat7)
(6) link /etc/cloudstack/management/server.xml to correct file path (server-nonssl.xml or server7-nonssl.xml)
(7) remove *.zip from .gitignore to avoid build error caused by missing /vhds/test.vhd.zip

Instruction
(1) build packages on Ubuntu 16.04: dpkg-buildpackage -uc -us
    Output on Ubuntu 16.04:
-rw-r--r-- 1 root root      4090 Feb 17 10:12 cloudstack_4.10.0.0-SNAPSHOT_amd64.changes
-rw-r--r-- 1 root root      1235 Feb 17 09:53 cloudstack_4.10.0.0-SNAPSHOT.dsc
-rw-r--r-- 1 root root   8018248 Feb 17 09:53 cloudstack_4.10.0.0-SNAPSHOT.tar.xz
-rw-r--r-- 1 root root  91868746 Feb 17 10:11 cloudstack-agent_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root     52882 Feb 17 10:12 cloudstack-cli_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root  98556216 Feb 17 10:08 cloudstack-common_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root     52864 Feb 17 10:12 cloudstack-docs_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root    585434 Feb 17 10:12 cloudstack-integration-tests_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root 323459934 Feb 17 10:10 cloudstack-management_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root    442656 Feb 17 10:12 cloudstack-marvin_4.10.0.0-SNAPSHOT_all.deb
-rw-r--r-- 1 root root  87037576 Feb 17 10:12 cloudstack-usage_4.10.0.0-SNAPSHOT_all.deb

(2) setup tomcat6/tomcat7 on management server:
    tomcat6: cloudstack-setup-management --tomcat6
    tomcat7: cloudstack-setup-management --tomcat7

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-04-07 18:41:24 +05:30
Abhinandan Prateek e5bfffb48c CLOUDSTACK-9857: With this change if agent dies the systemd will catch it properly and show process as exited
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-04-06 13:07:53 +05:30
Rajani Karuturi 1decf5366d Merge release branch 4.9 to master
* 4.9:
  CLOUDSTACK-9765: centos: packaging: fix missing path in spec file post install
2017-03-15 10:52:14 +05:30
Rene Moser bcc6b4fbaf CLOUDSTACK-9765: centos: packaging: fix missing path in spec file post install 2017-02-02 15:56:55 +01:00
Wido den Hollander 84e496b4f9
CLOUDSTACK-676: IPv6 Basic Security Grouping for KVM
This commit implements basic Security Grouping for KVM in
Basic Networking.

It does not implement full Security Grouping yet, but it does:
- Prevent IP-Address source spoofing
- Allow DHCPv6 clients, but disallow DHCPv6 servers
- Disallow Instances to send out Router Advertisements

The Security Grouping allows ICMPv6 packets as described by RFC4890
as they are essential for IPv6 connectivity.

Following RFC4890 it allows:
- Router Solicitations
- Router Advertisements (incoming only)
- Neighbor Advertisements
- Neighbor Solicitations
- Packet Too Big
- Time Exceeded
- Destination Unreachable
- Parameter Problem
- Echo Request

ICMPv6 is a essential part of IPv6, without it connectivity will break or be very
unreliable.

For now it allows any UDP and TCP packet to be send in to the Instance which
effectively opens up the firewall completely.

Future commits will implement Security Grouping further which allows controlling UDP and TCP
ports for IPv6 like can be done with IPv4.

Regardless of the egress filtering (which can't be done yet) it will always allow outbound DNS
to port 53 over UDP or TCP.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-01-26 15:36:08 +01:00
Rohit Yadav e6cc78f531 CLOUDSTACK-9710: Switch to JRE1.8
- Switches Travis to use jdk1.8
- Changes java-version to 1.8
- Change jdk/maven version to 1.8
- Switch to F5/java8 compatible library release
- Switch packaging to use jdk 1.8, and jre 1.8 in init/systemd scripts
- Switch systemvm to openjdk-8-jre

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-01-11 14:04:03 +05:30
Rohit Yadav 3fa461693c
Merge branch '4.9' 2016-11-09 14:54:20 +05:30
Rohit Yadav cf825eb044 systemd: Fix semicolon missing in b75e69
While forward merging PR #1728, and resolving merge issues a semi-colon was
not added causing cloudstack-agent to fail to start. This fixes the
issue of running agent on centos7.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-11-09 14:44:01 +05:30