This extends securing of KVM hosts to securing of libvirt on KVM
host as well for TLS enabled live VM migration.
Based on whether keystore and certificates files are available at
/etc/cloudstack/agent, the KVM agent determines whether to use TLS or
TCP based uris for live VM migration. It is also enforced that a secured
host will allow live VM migration to/from other secured host, and an
unsecured hosts will allow live VM migration to/from other unsecured
host only.
Post upgrade the KVM agent on startup will expose its security state
(secured detail is sent as true or false) to the managements server that
gets saved in host_details for the host. This host detail can be accesed
via the listHosts response, and in the UI unsecured KVM hosts will show
up with the host state of ‘unsecured’. Further, a button has been added
that allows admins to provision/renew certificates to KVM hosts and can
be used to secure any unsecured KVM host.
The `cloudstack-setup-agent` was modified to accept a new flag ‘-s’
which reconfigured libvirtd with following settings that enables only
TLS:
listen_tcp=0
listen_tls=1
tcp_port="16509"
auth_tcp="none"
tls_port=”16514”
auth_tls=”none”
key_file = "/etc/pki/libvirt/private/serverkey.pem"
cert_file = "/etc/pki/libvirt/servercert.pem"
ca_file = "/etc/pki/CA/cacert.pem"
For a connected KVM host agent, when the certificate are
renewed/provisioned a background task is scheduled that waits until all
of the agent tasks finish after which libvirt process is restarted and
finally the agent is restarted via AgentShell.
There are no API or DB changes.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This allows admins to add a KVM host using a sudoer user. This also
fails early when there is an issue with securing a KVM host on addition
than supress the information in logs.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
FR12 introduce basic support for comma-separated list of management
servers for agents, while makes an explicit LB unnecessary.
On the agent side, the 'host' is saved as:
<comma separated addresses>@<algorithm name>.
Where the algorithm name is the name of the lb algorithm. The
This FR introduces two new global settings:
- indirect.agent.lb.algorithm: The algorithm for the indirect agent LB.
- indirect.agent.lb.check.interval: The preferred host check interval
for the agent's background task that checks and switches to agent's
preferred host.
Any changes to the above two global settings and the 'host' setting does
not require restarting of the management server(s).
The indirect.agent.lb.algorithm supports following algorithm options:
- static: use the list as provided.
- roundrobin: evenly spreads hosts across management servers.
- shuffle: (pseudo) randomly sorts the list (not recommended for production).
From the agent's perspective, the first address in the propagated list
will be considered the preferred host. A new background task can be
activated by configuring the indirect.agent.lb.check.interval which is
a cluster level global setting from CloudStack or admins can override
this by configuring the 'host.lb.check.interval' in the host's
agent.properties file.
Comma-separated management server list is propagated to agents on
following cases:
- Addition of a host (including ssvm, cpvm systevms).
- Connection or reconnection by the agents to a management server.
- After admin changes the 'host' and/or the
'indirect.agent.lb.algorithm' global settings.
First the agent connects to the management server and sends its current
management server list, which is compared by the management server and
in case of failure a new/update list is sent for the agent to persist.
Every time agent gets a ms-host list and the algorithm, the host specific
background check interval is also sent and it dynamically reconfigures
the background task without need to restart agents.
The 'static' and 'roundrobin' algorithms, strictly checks for the order
as expected by them, however, the 'shuffle' algorithm just checks for
content and not the order of the comma separate ms host addresses.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.
This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.
Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
radomized comma separated list to which they will attempt connection
or reconnection in provided order. This removes need of a TCP LB on
port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
connecting agents will be required to present certificates issued
by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
authentication and connecting agents will not be required to present
certificates.
- A script `keystore-setup` is responsible for initial keystore setup
and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
SSH, and later provisioning is handled via an existing agent connection
using command-answers. The supported clients and agents are limited to
CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
connection, however rejects a revoked certificate used during SSL
handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
be used by mgmt server(s) for SSL negotiations and handshake. New
keystores will be named `cloud.jks`, any additional SSL certificates
should not be imported in it for use with tomcat etc. The `cloud.jks`
keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
the validity of them are same as the CA certificates.
New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial
Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates
Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed
UI changes:
- Button to download/save the CA certificates.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Implements a VM volume/disk file activity checker that checks if QCOW2 file
has been changed before starting the VM. This is useful as a pessimistic
approach to save VMs that were running on faulty hosts that CloudStack could
try to launch on other hosts while the host was not cleanly fenced. This is
optional and available only if you enable the settings in agent.properties
file, on per-host basis.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This allows non-root users to add KVM hosts, the user should be an admin or
added to sudoers to execute sudo cloudstack-setup-agent.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: Remi Bergsma <apache@remi.nl>
This closes#288
This improvements checks for "guest.cpu.features" property which is a space
separated list of cpu features that is specific for a host. When added, it
will add <feature policy='require' name='{{feature-you-listed}}'/> in the
<cpu> section of the generated vm spec xml.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Number of threads on SSVM agent for connection with MS (Agent->NioClient) should be configurable using global config 'workers'.
(cherry picked from commit bc235ed5eb)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
In init.d scripts, the LSB header may specify what kind of service is
provided by an init script. If spaces are used, this means the init
script is providing several boot facilities. We fix that by using an
hyphen.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 2401eb927b)
This allows us to have the Agent connect to the Management Server
over IPv6 if that is listening on :::8250
With this patch it is possible to deploy a IPv6-only KVM Agent where
IPv4 traffic is still forwarded over the bridges, but the KVM Agent
itself does not have IPv4 connectivity.
Replacing whatami with $0 which is how UNIX shell scripts should get the
script's name.
BUG-ID: CLOUDSTACK-6129
Bugfix-for:
Reviewed-by:
Reported-by:
Signed-off-by: John Kinsella <jlk@stratosec.co> 1392660036 -0800
Rohit Yadav
Rohit Yadav
Wido den Hollander
Santhosh Edukulla
Likitha Shetty
Edison Su
Vincent Bernat
Hugo Trippaers
Anthony Xu
Kishan Kavala
Marcus Sorensen
Daan Hoogland
John Kinsella
Alex Huang
wilderrodrigues
Kelven Yang