A complicated KVM server multiple upgrade

This was the plan we followed when we upgraded the OS and the memory and added a new /var/lib/libvirt/qemu partition, all at the same time.

  1. Install the gdisk package.
  2. Use gdisk to make a new disk partition (/dev/sda4).
  3. Pacify nagios for oyster.
  4. Pacify nagios for its MPU guests too.
  5. Migrate essential VMs elsewhere.
  6. Stop oyster's VMs from suspending and resuming at every reboot: !boot.services mREMOVE(rc_libvirtguests)
  7. Shut down every VM in rvirsh oyster list :
           kvmtool shutdown --name rembrandt
           kvmtool shutdown --name nimrodkvm
           kvmtool shutdown --name tetrakvm
           kvmtool shutdown --name praecox
           kvmtool shutdown --name knox
           kvmtool shutdown --name sanremo
           kvmtool shutdown --name tench
           kvmtool shutdown --name nescxldb
           kvmtool shutdown --name barb
           kvmtool shutdown --name tarnkvm
           kvmtool shutdown --name natter
           kvmtool shutdown --name charlar
           kvmtool shutdown --name canary
           kvmtool shutdown --name redmires
           kvmtool shutdown --name yubtest
           kvmtool shutdown --name dstwo
           kvmtool shutdown --name dsone
           kvmtool shutdown --name guppy
           kvmtool shutdown --name bakerstreet
           kvmtool shutdown --name huxley
           kvmtool shutdown --name dunkvm
           kvmtool shutdown --name banter
           kvmtool shutdown --name craic
           kvmtool shutdown --name lardup
           kvmtool shutdown --name gabble
           kvmtool shutdown --name heda
           kvmtool shutdown --name rhacotis
           kvmtool shutdown --name arachne
           kvmtool shutdown --name storck
    
  8. Check rvirsh oyster list to make sure they're all down.
  9. Check that /var/lib/libvirt/qemu is empty (apart from 3 empty directories).
  10. rfe lcfg/oyster and remove DICE_STICK_WITH_SL65
  11. Wait for the new profile to hit oyster and check /var/lcfg/log/client etc.
  12. Install any new packages before rebooting, to minimise later reboots: om updaterpms run -- -f
  13. Shut oyster down.
  14. Install the new memory.
  15. Switch oyster on.
  16. Ensure that the new memory is recognised and passed by the POST.
  17. If possible boot to single user.
  18. Make the new filesystem: /sbin/mke2fs -t ext3 /dev/sda4
  19. Get the new partition's UUID: /sbin/blkid /dev/sda4 -o value -s UUID
  20. Edit sda4 and its UUID into /var/lcfg/conf/fstab/fstab.sda
  21. om fstab configure or edit sda4 and its UUID into /etc/fstab
  22. mount -a to check the new partition.
  23. Go to multi-user or reboot.
  24. Start the VMs: /etc/init.d/libvirt-guests start
  25. Or if that doesn't work when VMs aren't running at all, start them up individually:
           kvmtool start --name rembrandt
           kvmtool start --name nimrodkvm
           kvmtool start --name tetrakvm
           kvmtool start --name praecox
           kvmtool start --name knox
           kvmtool start --name sanremo
           kvmtool start --name tench
           kvmtool start --name nescxldb
           kvmtool start --name barb
           kvmtool start --name tarnkvm
           kvmtool start --name natter
           kvmtool start --name charlar
           kvmtool start --name canary
           kvmtool start --name redmires
           kvmtool start --name yubtest
           kvmtool start --name dstwo
           kvmtool start --name dsone
           kvmtool start --name guppy
           kvmtool start --name bakerstreet
           kvmtool start --name huxley
           kvmtool start --name dunkvm
           kvmtool start --name banter
           kvmtool start --name craic
           kvmtool start --name lardup
           kvmtool start --name gabble
           kvmtool start --name heda
           kvmtool start --name rhacotis
           kvmtool start --name arachne
           kvmtool start --name storck
    
  26. Remove the oyster details from https://computing.help.inf.ed.ac.uk/scheduled-downtime
  27. Migrate the essential VMs back from hammersmith and jubilee.
  28. Remove the temporary storage and symlinks from hammersmith and jubilee.
Topic revision: r2 - 01 Sep 2015 - 13:04:48 - ChrisCooke
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies