0

I want to create a simple backup script to shutdown all running VM's, copy image files afterwards and starting auto-starting VM'S again. Sound simple, but when I shutdown a VM (using "virsh shutdown VMNAME") script does not wait for shutdown completely. Can I stop KVM service, resulting all VM's to shutdown, copy VM files and start again. This way only auto starting VM's are started. Sounds like stopping MySQL/MariaDB, backing-up files and starting it again. People may suggest LVM snapshots, it is not possible right now for this server.

bkilinc
  • 989
  • 2
  • 13
  • 28

2 Answers2

0

I had the same problem and so I made a small if statement to check the /var/log/libvirt/qemu/domain.com.log to see whether reason=shutdown was issued in the logs, which serves as a proxy for whether the domain actually shutdown or not. If the conditional fails, the script will email you, and if it succeeds, the conditional will continue with your backup script and/or call another. It is also provisioned for a loop/array with multiple VMs. Something like this should work:

#!/bin/bash
DATE=`date +"%Y%m%d-%H:%M:%S"`
IMG="guest1.qcow2 guest2.qcow2"

for i in $IMG;
do
virsh shutdown $i
#Adjust to what you expect is a reasonable shutdown time
sleep 2m

if
tail -n -2 /var/log/libvirt/qemu/$i.log | grep "reason=shutdown"
then

#enter your backup script inline here, or call it
/bin/bash /usr/local/bin/backup-script.sh

else
#notify of failure
echo "The VM failed to shutdown or is already off at $(date) so I can't run your backup script" | mail -s "[backup-failed]-$(hostname -f)-$(date)" alert@domain.com

fi
done

No script is perfect and you will still need to conduct regular maintenance to ensure that it runs properly. I will also note that my solution is quite primitive, but that it works reliably. For passerby, however, I will note that the OP question seems to be a duplicate to this question which has some very elegant proposals as to how to ensure and/or wait for your VMs to power down before executing scripts on them. I have a different script based off one of those here if it helps passerby.

oemb1905
  • 148
  • 1
  • 7
-1

"virsh shutdown" sends a shutdown command to the client OS, which may (or may not) actually shutdown the client. "virsh destroy" similarly sends a powerdown signal to the client, but clients that are not in a normal state (e.g., waiting on NFS unmounts or problematic disk synchs) will still not shut down.

The only way I've found to guarantee client shutdown is to reboot the KVM parent, wait for any KVM state to be recovered, and immediately shutting down any running clients. At that point, you can do the qemu-img backups or snapshots, then restart the clients

I have not tried the "virsh suspend" command (just looked it up researching this answer) -- that may be a help for my similar issues.

mpez0
  • 2,815
  • 17
  • 12
  • thanks. I found out that it is better do write down a script that waits for all servers to shutdown. Libvirt has PHP API, which I am familiar with. I think I managed to write the script that cleanly shutdown all servers, create backup and start again. I am thinking of sharing it, after some polishing. For now, It just works for me. – bkilinc Dec 05 '17 at 12:28
  • @bkilinc please share your script now; it should be polished after these 5 years! ;> – oemb1905 Dec 27 '22 at 18:42
  • 1
    here it is ! https://gist.github.com/bkilinc/17ecd3ce25e2a4d9ab6716834a53a615 @oemb1905 – bkilinc Dec 28 '22 at 19:18
  • @bkilinc very nicely done, thanks! – oemb1905 Dec 29 '22 at 21:59
  • 1
    thanks @oemb1905. I am using this script monthly. And I really forgot this question over years. So I guess there is no simpler solution for graceful shutdown of virtual machines in 2022. – bkilinc Dec 30 '22 at 12:07
  • Actually, you can just embed this simple while statement within a for loop and if then, ``virsh shutdown $i STATE=$(virsh dominfo $i | grep -w "State:" | awk '{ print $2}') while ([ "$STATE" != "" ] && [ "$STATE" == "running" ]); do sleep 10 STATE=$(virsh dominfo $i | grep -w "State:" | awk '{ print $2}') done;`` I found it in a related post here: https://stackoverflow.com/questions/37453525/how-can-i-check-specific-server-running-or-notusing-virsh-commands-before-i-s @bkilinc – oemb1905 Jan 04 '23 at 06:52
  • And my tail script above, although basic, is very good at ceasing the backup attempt if the logs don't produce the desired shutdown result. There are, however, false positives, e.g., machine is already shut off so should be excluded, or false negatives, e.g., log read failure or qemu changes the log. Your script is powerful and nice, though, and should help passerby! ;> – oemb1905 Jan 04 '23 at 06:54