Gas explosion drill - MPU report

This MPU kit was affected by the pretend AT gas explosion:

Machines lost

  • kubelik (student.ssh)
  • KVM servers circle and waterloo
  • wildcat (RPM cache and ?backup PXE)
  • With waterloo we have also lost vermeer aka lcfg1, one of the two main LCFG slave servers.

The loss of waterloo knocks out these virtual machines:

bank MPU test
barking RAT Trac server
borges Services backup print server
capon Infrastructure secondary Nagios server
wobleg Services test
spadina RAT projects.inf
vermeer MPU lcfg1.inf

These VMs were lost along with circle:

argus RAT testing forumtracker
arlott INF testing sl6 KDC
armitage Services student labs monitoring test
cardus INF testing sl6 cosign
circlevm0 MPU for testing
circlevm2 MPU for testing
circlevm3 MPU for testing
circlevm4 MPU for testing
circlevm5 MPU for testing
circlevm6 MPU for testing
circlevm7 MPU for testing
circlevm8 MPU for testing
circlevm9 MPU for testing
circlevm10 MPU for testing
dilley INF testing sl6 KDC
ekcof RAT testing Coltex
engadine RAT gdutton test
idoru Services gordon's test vm
keele RAT test portal
littlebird RAT iainr test
monmouth RAT iainr test
monty INF testing sl6 prometheus
moody ? Moodle

Services affected

One of the two main LCFG slave servers has been lost. The service will carry on more or less unaffected using the other slave server. The MPU is considering bringing up another slave server elsewhere. In the meantime the DNS aliases lcfg1 and lcfg3 have been moved to the other slave server rembrandt, safely in the Forum.
Package cache and updaterpms
We have lost wildcat, one of the two RPM cache servers serving This is the address from which updaterpms gets its RPMs on most DICE machines. We have altered the DNS to remove its IP address from cache.pkgs. An om dns update or waiting an hour should be enough to get updaterpms working on DICE machines outside the Tower.
SSH aka has gone. The MPU has brought its hot spare shrew at KB into use as the new temporary student ssh server.
KVM service
  • We have recovered the backup of /etc/libvirt for the lost KVM server waterloo, in case it should come in handy, though we hope that the waterloo wiki page and the LCFG should give sufficient detail to enable people to restore their VMs elsewhere. We have sufficient capacity on other KVM servers, partly thanks to waterloo having been underused. It only hosted seven VMs of which two were test VMs. The backup of waterloo_'s /etc/libvirt can be found in /etc/waterloo on _oyster if anyone needs it.
  • We do not expect to take any action to recover circle as it is only a test server.

-- ChrisCooke - 13 Jan 2014

Edit | Attach | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r2 - 13 Jan 2014 - 14:31:14 - ChrisCooke
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies