Services Unit Report Following the Flood Drill

On Sunday 26th of February a simulated Flood affected access and servers to the Forum Server room. Our report is here:

The Lessons and Actions are repeated below.

Lessons

  1. Some system should be put in place to record and regularly update the contents of server room racks.
  2. The configuration of the various disk arrays needs to be dumped to a secure location at regular intervals.
  3. Service data should be backed up, wherever possible, directly from the host providing the service.
  4. There needs to be a supply of server and fibre hardware available for cases where a VM cannot act as a suitable replacement.

Actions

  1. Instigate a rack content recording system
  2. Arrange for regular dumping of disk configuration information (services unit)
  3. Arrange for service data to be backed up directly (services unit)
  4. Carry out test restore of entire AFS partition (services unit)
  5. Experiment with mounting AFS volumes on a different server (services unit)
  6. Arrange for supply of replacement server and fibre hardware
  7. Give consideration as to the best way to restore the AFS service to normality after promoting offsite RO volumes (services unit)
  8. Need to think about what it means if ext3/4 journals being held on SSDs internal to the server when the “disks” are on a SAN elsewhere, and the machine and SSD (and so journals) get destroyed, but the file systems on the SAN survive. What would that mean when trying to bring the file system back online on another machine (without the journal)?

-- NeilBrown - 08 Mar 2012

Topic attachments
I Attachment Action Size Date Who Comment
elseodt flood-drill-report.odt manage 17.3 K 08 Mar 2012 - 13:18 NeilBrown Open Office Document
pdfpdf flood-drill-report.pdf manage 71.6 K 08 Mar 2012 - 13:18 NeilBrown PDF of report
Topic revision: r1 - 08 Mar 2012 - 13:21:05 - NeilBrown
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies