KVM Service, aka Simple Virtualisation: emergency pandemic tips

Your colleagues are fighting off a zombie invasion, so YOU have to save the KVM service. Here's how.

Basic facts

  • The virtualisation software is called KVM and the server side daemon for this is called libvirtd.
  • The VMs run on several large servers ("the KVM servers").
  • Most of the KVM servers are run by the MPU.
  • RAT also has several KVM servers of its own, which are managed in almost the same way.
  • The MPU has:
  • Run this to get a full list of KVM servers: netgroup -M HOST_kvmserver
  • Virtual machines are known by various terms: VMs; guests; domains.

Where the config lives

  • The MPU KVM servers use live/mpu-kvm-server.h. This pulls in various headers including dice/options/kvm-server.h and dice/options/kvm-client.h, and it sets some common configuration for the servers, such as which subnets are available for VMs to use.

Where's the documentation?

Look in SimpleKVMDocs.

Local tools - kvmtool and kvmreport

  • kvmtool helps you manipulate a VM - for instance you can use it to create, delete, shutdown or boot a VM. See man kvmtool for help.
    It's really just a simpler front end to virsh. To find out more, a lot more, see man virsh.
  • kvmreport summarises the state of a KVM server for you. Run it as root on the KVM server.
  • There's also a KVM servers web report. (At some point this will move to a new address. If that link doesn't work any more, ask Alastair or MPU where the KVM servers web report has moved to.)

If kvmtool hangs, restart libvirtd

Sometimes libvirtd goes wrong on one of the KVM servers, and doesn't answer kvmtool. Running kvmtool with the --debug option makes it obvious which server is giving the problem. Once you know that, the solution is to restart libvirtd :
$ ssh <the affected KVM server>
$ nsu
# systemctl restart libvirtd
This is safe. It won't kill the VMs. (They're managed by a different systemd service called libvirt-guests.) See also SimpleKVMHost#When_you_can_t_connect_to_a_KVM.

How to reboot a server

MPUKVMServerUpgrade explains how to reboot a KVM server without breaking it.

How to migrate a VM to another server

This is called hot migration, and this page tells you how to do it: SimpleKVMHost#Hot_migrating_a_guest_from_one_s.
It's fairly safe and routine. For small VMs it takes a few minutes, for big ones up to an hour or more.
Limitations: the guest has to be running, and you should only migrate it to another server at the same site.

What if a VM doesn't resume? (and other questions)

When you're resuming VMs - for instance after having rebooted their server - it's possible that a VM will fail to resume. You might spot that this has happened thanks to an error on the server's console (such as error: Unable to read from monitor: Connection reset by peer) or you might simply realise at some point that the VM isn't running. The solution is:
  1. Delete the suspended session that won't resume: rvirsh server managedsave-remove VM
    For example rvirsh banjo managedsave-remove smith
  2. Start the VM. It'll boot from cold. kvmtool start VM
There are lots of answers to questions like this in SimpleKVMDocs and its child pages.
Topic revision: r5 - 09 Oct 2020 - 12:05:03 - StephenQuinney
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies