Address review feedback on #12843 (verified with a stubbed self-test):
- domjobinfo 'Failed' branch: exit after cleanup instead of looping until
BACKUP_TIMEOUT (the loop never broke on a failed virsh job).
- backup_stopped_vm qemu-img convert failure: exit after cleanup so a partial
backup is not treated as success; append to the agent log instead of truncating.
- backup_stopped_vm success: unmount before returning, mirroring backup_running_vm/
delete_backup. Without this the 'trap cleanup EXIT' fired on success and ran
'rm -rf $dest', deleting the just-completed backup.
- mount_operation: test mount's own exit in an if-condition so 'set -eo pipefail'
no longer aborts before the check and a piped tee can't mask a mount failure.
- cleanup: only umount when the path is actually a mount, so a non-mount umount
error doesn't wrongly report EXIT_CLEANUP_FAILED via the EXIT trap.
Three independent reliability fixes for the KVM NAS backup script, layered on
top of the existing quiesce + EXIT_CLEANUP_FAILED groundwork:
1. BACKUP_TIMEOUT env var (default 6h) bounds the libvirt domjobinfo wait
loop in backup_running_vm. Today a stuck QEMU backup holds the agent's
command slot until the orchestrator-level timeout fires. The new guard
issues domjobabort and exits non-zero so the agent reclaims the slot
promptly.
2. MIN_FREE_SPACE env var (default 1 GiB) + check_free_space() runs after
mount and before any qemu-img convert in both backup_running_vm and
backup_stopped_vm. Fail-fast on a near-full NAS instead of failing
mid-write halfway through a multi-GiB convert.
3. trap cleanup EXIT replaces the six explicit cleanup() call sites as the
primary cleanup mechanism so orphan NFS mounts no longer accumulate when
the script dies to SIGTERM, SIGINT, or any uncaught set -e failure
between the explicit call sites. cleanup() is now guarded by
CLEANUP_DONE so the trap doesn't re-run an already-completed cleanup
from an explicit call.
cleanup() additionally resumes the VM if it's still paused — backup-begin
holds the guest paused briefly and a failed backup mid-pause currently
leaves the guest stuck in 'paused' state until an operator intervenes.
Targets main; supersedes the 4.20-targeted version of this PR.
This feature adds the ability to create a new instance from a VM backup for dummy, NAS and Veeam backup providers. It works even if the original instance used to create the backup was expunged or unmanaged. There are two parts to this functionality:
Saving all configuration details that the VM had at the time of taking the backup. And using them to create an instance from backup.
Enabling a user to expunge/unmanage an instance that has backups.
* NAS B&R Plugin enhancements
* Prevent printing mount opts which may include password by removing from response
* revert marvin change
* add sanity checks to validate minimum qemu and libvirt versions
* check is user running script is part of libvirt group
* revert changes of retore expunged VM
* add code coverage ignore file
* remove check
* issue with listing schedules and add defensive checks
* redirect logs to agent log file
* add some more debugging
* remove test file
* prevent deletion of cks cluster when vms associated to a backup offering
* delete all snapshot policies when bkp offering is disassociated from a VM
* Fix `updateTemplatePermission` when the UI is set to a language other than English (#9766)
* Fix updateTemplatePermission UI in non-english language
* Improve fix
---------
* Add nobrl in the mountopts for cifs file system
* Fix restoration of VM / volumes with cifs
* add cifs utils for el8
* add cifs-utils for ubuntu cloudstack-agent
* syntax error
* remove required constraint on both vmid and id params for the delete bkp schedule command
This is a simple NAS backup plugin for KVM which may be later expanded for other hypervisors. This backup plugin aims to use shared NAS storage on KVM hosts such as NFS (or CephFS and others in future), which is used to backup fully cloned VMs for backup & restore operations. This may NOT be as efficient and performant as some of the other B&R providers, but maybe useful for some KVM environments who are okay to only have full-instance backups and limited functionality.
Design & Implementation follows the `networker` B&R plugin, which is simply:
- Implement B&R plugin interfaces
- Use cmd-answer pattern to execute backup and restore operations on KVM host when VM is running (or needs to be restored) - instead of a B&R API client, relies on answers from KVM agent which executes the operations
- Backups are full VM domain snapshots, copied to a VM-specific folders on a NAS target (NFS) along with a domain XML
- Backup uses libvirt feature: https://libvirt.org/kbase/live_full_disk_backup.html orchestrated via virsh/bash script (nasbackup.sh) as the libvirt-java lacks the bindings
- Supported instance volume storage for restore operations: NFS & local storage
Refer the doc PR for feature limitations and usage details:
https://github.com/apache/cloudstack-documentation/pull/429
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Co-authored-by: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
jmsperu
Suresh Kumar Anaparti
Abhisar Sinha
John Bampton
Pearl Dsilva
Rohit Yadav