Services Unit Report Following the Flood Drill

On Sunday 26th of February a simulated Flood affected access and servers to the Forum Server room. Our report is here:

• flood-drill-report.pdf: PDF of report
• flood-drill-report.odt: Open Office Document

The Lessons and Actions are repeated below.

Lessons

1. Some system should be put in place to record and regularly update the contents of server room racks.
2. The configuration of the various disk arrays needs to be dumped to a secure location at regular intervals.
3. Service data should be backed up, wherever possible, directly from the host providing the service.
4. There needs to be a supply of server and fibre hardware available for cases where a VM cannot act as a suitable replacement.

Actions

1. Instigate a rack content recording system
2. Arrange for regular dumping of disk configuration information (services unit)
3. Arrange for service data to be backed up directly (services unit)
4. Carry out test restore of entire AFS partition (services unit)
5. Experiment with mounting AFS volumes on a different server (services unit)
6. Arrange for supply of replacement server and fibre hardware
7. Give consideration as to the best way to restore the AFS service to normality after promoting offsite RO volumes (services unit)
8. Need to think about what it means if ext3/4 journals being held on SSDs internal to the server when the “disks” are on a SAN elsewhere, and the machine and SSD (and so journals) get destroyed, but the file systems on the SAN survive. What would that mean when trying to bring the file system back online on another machine (without the journal)?

-- NeilBrown - 08 Mar 2012