cloudstack/ui
Abhishek Kumar 0b5a5e8043
api,agent,server,engine-schema: scalability improvements (#9840)
* api,agent,server,engine-schema: scalability improvements

Following changes and improvements have been added:

- Improvements in handling of PingRoutingCommand

    1. Added global config - `vm.sync.power.state.transitioning`, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
    2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
    3. Optimized scanning stalled VMs

- Added option to set worker threads for capacity calculation using config - `capacity.calculate.workers`

- Added caching framework based on Caffeine in-memory caching library, https://github.com/ben-manes/caffeine

- Added caching for account/use role API access with expiration after write can be configured using config - `dynamic.apichecker.cache.period`. If set to zero then there will be no caching. Default is 0.

- Added caching for account/use role API access with expiration after write set to 60 seconds.

- Added caching for some recurring DB retrievals

    1. CapacityManager - listing service offerings - beneficial in host capacity calculation
    2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
    3. DownloadListener - hypervisors for zone - beneficial for host joins
    5. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands

- Optimized MS list retrieval for agent connect

- Optimize finding ready systemvm template for zone

- Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks

- Changes in agent-agentmanager connection with NIO client-server classes

    1. Optimized the use of the executor service
    2. Refactore Agent class to better handle connections.
    3. Do SSL handshakes within worker threads
    5. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - `agent.ssl.handshake.min.workers` and `agent.ssl.handshake.max.workers` can be used to control number of new connections management server handles at a time. `agent.ssl.handshake.timeout` can be used to set number of seconds after which SSL handshake times out at MS end.
    6. On agent side backoff and sslhandshake timeout can be controlled by agent properties. `backoff.seconds` and `ssl.handshake.timeout` properties can be used.

- Improvements in StatsCollection - minimize DB retrievals.

- Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.

- Improvements in hosts connection for a storage pool. Added config - `storage.pool.host.connect.workers` to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.

- Minor improvements in resource limit calculations wrt DB retrievals

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* test1, domaindetails, capacitymanager fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* test2 - agent tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* capacitymanagertest fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* change

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix missing changes

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert marvin/setup.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix indent

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* use space in sql

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address duplicate

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* update host logs

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* revert e36c6a5d07

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix npe in capacity calculation

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* move schema changes to 4.20.1 upgrade

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* address comments

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* fix build

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add some more tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* checkstyle fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* remove unnecessary mocks

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* build fix

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* replace statics

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* engine/orchestration,utils: limit number of concurrent new agent
connections

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* refactor - remove unused

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* unregister closed connections, monitor & cleanup

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* add check for outdated vm filter in power sync

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

* agent: synchronize sendRequest wait

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

---------

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2025-02-01 12:28:41 +05:30
..
docs Fix spelling in docs and code comments only (#8774) 2024-03-25 14:38:01 +01:00
nginx docker: add container build files and instructions (#22) 2021-01-20 07:06:14 +05:30
public Merge branch '4.19' into 4.20 2025-01-24 17:01:42 +01:00
src api,agent,server,engine-schema: scalability improvements (#9840) 2025-02-01 12:28:41 +05:30
tests UI: fix unit tests (#9798) 2024-10-16 10:44:45 +05:30
.babelrc pre-commit: apply `end-of-file-fixer` to all files (#7551) 2023-08-02 13:47:21 +02:00
.editorconfig editorconf: Remove CRLF editor config (#23) 2021-01-20 07:06:14 +05:30
.env.local.example README updates and CS_URL fix 2021-01-20 07:06:13 +05:30
.env.local.https.example vue: allow both http and https configuration (#2) 2021-01-20 07:06:14 +05:30
.env.primate-qa ui: show zones and offerings to users (#7557) 2023-05-25 13:44:44 +02:00
.gitattributes pre-commit: apply `end-of-file-fixer` to all files (#7551) 2023-08-02 13:47:21 +02:00
.gitignore local versions of .env ignored (#6256) 2022-04-13 09:46:07 -03:00
.prettierrc primate: initial UI v0.1 2021-01-20 07:06:11 +05:30
Dockerfile Removed deprecated instruction MAINTAINER (#8846) 2024-09-20 09:28:46 +02:00
LICENSE Create LICENSE 2021-01-20 07:06:13 +05:30
README.md Merge LTS branch '4.19' into main 2024-04-18 08:54:58 +02:00
babel.config.js Explore Test Automation (#320) 2021-01-20 07:06:24 +05:30
docker.sh Possiblity to choose between docker and podman from the command line (#5217) 2021-08-19 13:13:34 +05:30
index.html Use lowercase HTTP header field names so we are compatible with HTTP/2 (#7006) 2023-01-23 11:17:54 +01:00
jest.config.js Generate coverage for Simulator based integration tests & ui tests in github actions (#7784) 2023-09-11 14:14:12 +05:30
jsconfig.json primate: initial UI v0.1 2021-01-20 07:06:11 +05:30
package-lock.json ui: bump UI dependencies versions (#8923) 2024-04-17 09:08:56 +02:00
package.json Notify users when upgrades are available or restart is required for network or VPC (#7610) 2024-09-04 09:23:57 +02:00
postbuild.sh ui: prevent same string docHelp override (#5014) 2021-05-12 16:20:17 +05:30
prebuild.sh ui: prevent same string docHelp override (#5014) 2021-05-12 16:20:17 +05:30
theme.config.js UI: Upgrade to Vue3 library (#5151) 2022-03-09 09:47:09 -03:00
vue.config.js Fix Vue devServer after CSRF fix 2024-10-11 17:25:32 +02:00
webstorm.config.js src: add apache v2.0 license header to files 2021-01-20 07:06:14 +05:30

README.md

CloudStack UI

A modern role-based progressive CloudStack UI based on Vue.js and Ant Design.

Screenshot

Getting Started

Install node: (Debian/Ubuntu)

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt-get install -y nodejs
# Or use distro provided: sudo apt-get install npm nodejs

Install node: (CentOS/Fedora/RHEL)

curl -sL https://rpm.nodesource.com/setup_16.x | sudo bash -
sudo yum install nodejs

Install node: (Mac OS)

brew install node@16

Optionally, you may also install system-wide dev tools:

sudo npm install -g @vue/cli npm-check-updates

Development

Clone the repository:

git clone https://github.com/apache/cloudstack.git
cd cloudstack/ui
npm install

Override the default CS_URL to a running CloudStack management server:

cp .env.local.example .env.local
Change the `CS_URL` in the `.env.local` file

To configure https, you may use .env.local.https.example.

To use the community Simulator QA server you can do this:

echo "CS_URL=https://qa.cloudstack.cloud" > .env.local

Build and run:

npm run serve
# Or run: npm start

Upgrade dependencies to the latest versions:

ncu -u

Run Lint and Unit Tests:

npm run lint
npm run test:unit

Fix issues and vulnerabilities:

npm audit

A basic development guide and explaination of the basic components can be found here

Production

Fetch dependencies and build:

npm install
npm run build

This creates a static webpack application in dist/, which can then be served from any web server or CloudStack management server (jetty).

To use CloudStack management server (jetty), you may copy the built UI to the webapp directory on the management server host. For example:

npm install
npm run build
cd dist
mkdir -p /usr/share/cloudstack-management/webapp/
cp -vr . /usr/share/cloudstack-management/webapp/
# Access UI at {management-server}:8080/client in browser

If the webapp directory is changed, please change the webapp.dir in the /etc/cloudstack/management/server.properties and restart the management server host.

To use a separate webserver, note that the API server is accessed through the path /client, which needs be forwarded to an actual CloudStack instance.

For example, a simple way to serve UI with nginx can be implemented with the following nginx configuration (to be put into /etc/nginx/conf.d/default.conf or similar):

server {
    listen       80;
    server_name  localhost;
    location / {
        # /src/ui/dist contains the built UI webpack
        root   /src/ui/dist;
        index  index.html;
    }
    location /client/ {
        # http://127.0.0.1:800 should be replaced your CloudStack management
        # server's actual URI
        proxy_pass   http://127.0.0.1:8000;
    }
}

Docker

A production-ready Docker container can also be built with the provided Dockerfile and build script.

Make sure Docker is installed, then run:

bash docker.sh

Change the example configuration in nginx/default.conf according to your needs.

Run UI:

docker run -ti --rm -p 8080:80 -v $(pwd)/nginx:/etc/nginx/conf.d:ro cloudstack-ui:latest

Packaging

The following is tested to work on any Ubuntu 18.04/20.04 base installation or docker container:

# Install nodejs (lts)
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs debhelper rpm
# Install build tools
npm install -g @vue/cli webpack eslint
# Clone this repository and run package.sh
cd <cloned-repository>/packaging
bash -x package.sh

Documentation

Attributions

The UI uses the following:

History

The modern UI, originally called Primate, was created by Rohit Yadav over several weekends during late 2018 and early 2019. During ApacheCon CCCUS19, on 9th September 2019, Primate was introduced and demoed as part of the talk Modern UI for CloudStack (video). Primate was accepted by the Apache CloudStack project on 21 Oct 2019. The original repo was merged with the main apache/cloudstack repo on 20 Jan 2021.

License

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.