AT Gas Explosion - this is a drill !
Affected services
What |
Backup/mirror |
Plan |
4 AFS file servers: naga, cetus, minotaur, gorgon |
tibs backups and mirror releases from 8pm Saturday |
1657 user volumes affected. Promote offsite ROs to RW, create new RW space downtown for shuffling later |
stoater - AIAI web server |
mirrored to nix mirror17 and then tibs |
Affected sites aiai binni ksco i-globe i-x.aiai vue.ed openvce.net atate.org youtute schooltute equipment.inf - rebuild and restore from mirror |
cigar - Plone WCMS server |
mirror nix rmirror16 and then tibs |
lots of sites - rebuild and restore from mirror |
Affected due to .202. network being unavailable
What |
Backup/mirror |
Plan |
toaster - groups.inf and various other web sites also so related NFS group space |
mirrored to nix mirror16 |
KVM on jubilee in Forum, but on 202 subnet. Plan would probably be to move machine to .33 (or create a new VM on .33) and update DNS for the various sites |
For the record, not affected
mail, www.inf, dice.inf, wiki.inf, printing,
most group file space - see groups detail below
Other kit affected, but not providing a live service
atabeast1, satablade1, woblog, borges
More detail on the affected services
AFS
The four AFS servers only use local disk storage, not from the SAN, so there's no option to remount their data. We'd have to rely on the nightly release and backups and make use the the DR data at KB.
Rather that listing all the affected user volumes here to be scraped, you can generate the list yourself with:
echo naga cetus gorgon minotaur | xargs -n1 /usr/sbin/vos listvldb -server | grep ^user\.
As mentioned in the chat room, there are about 13 computing staff affected by that. We'd probably have done "vos convertROtoRW" for those staff fairly quickly. To get them up and running again. Probably being more considered for the other users, as we may want to make sure we've got new AFS space on line to create new offsite RO copies first.
From AFSPartitions (and just local knowledge) we can see that the raw storage required would be 12x455GB of data, but we can get an "actually used" figure from the size of the offsite RO volumes. - TO DO
stoater - aiai web server
Contact: Austin/AIAI
- atate.org
- openvce.net
- oplan.aiai.ed.ac.uk
- vue.ed.ac.uk
- www.aiai.ed.ac.uk
- www.aiai.inf.ed.ac.uk
- www.i-globe.info
- www.i-x.info
- www.ksco.info
- www.openvce.org
Contact: John Lee
- youtute.inf.ed.ac.uk
- schooltute.inf.ed.ac.uk
Contact: Francesco Figari
- equipment-sharing.inf.ed.ac.uk
cigar - Plone WCMS web server
- wcms.inf.ed.ac.uk including / cisa, icsa, ipab, lfcs, hcrc, ilcc, speechlabs, pepa, idar09, sspnet, jast, dice, sandbox
- www.anc.ed.ac.uk
- www.cisa.inf.ed.ac.uk
- www.classic-project.org
- www.emime.org
- www.hcrc.ed.ac.uk
- www.ilcc.inf.ed.ac.uk
- www.mngu0.org
- www.not-a-service.inf.ed.ac.uk
- www.transfics.eu
- www.ultrax-speech.org
- migration.inf.ed.ac.uk
- pbf2013.inf.ed.ac.uk
toaster - groups web sites
In the mirror of the web config
for i in `grep -i servername *.conf | awk '{print $NF}' | sort | uniq`; do echo -n $i; host -t A $i | awk '/has address/ {printf(" %s",$NF)} END {printf "\n"}'; done | grep -E '129\.215\.202\.(26|60)\b'
- aicat.inf.ed.ac.uk
- aied.inf.ed.ac.uk
- conferences.inf.ed.ac.uk
- data.cstr.ed.ac.uk
- dbibd-05.inf.ed.ac.uk
- downloads.specknet.org
- events.inf.ed.ac.uk
- fordyce.inf.ed.ac.uk
- groups.inf.ed.ac.uk
- history.dcs.ed.ac.uk
- hoppers.inf.ed.ac.uk
- infcricket.inf.ed.ac.uk
- inf.statmt.org
- media.inf.ed.ac.uk
- newbuildpics.inf.ed.ac.uk
- openafs2012.inf.ed.ac.uk
- proofgeneral.inf.ed.ac.uk
- ref2014.inf.ed.ac.uk
- touchscreens.inf.ed.ac.uk
- uitp05.inf.ed.ac.uk
- waim-05.inf.ed.ac.uk
- workshops.inf.ed.ac.uk
- www.arcs.im
- www.bctcs.ac.uk
- www.cav2005.inf.ed.ac.uk
- www.computersciencepodcast.com
- www.ehmn.bioinformatics.ed.ac.uk
- www.entrepedia.org
- www.etaps05.inf.ed.ac.uk
- www.euphoria-project.eu
- www.hscma2011.org
- www.icdt2005.inf.ed.ac.uk
- www.ilsi.inf.ed.ac.uk
- www.inspace.ed.ac.uk
- www.neurogems.org
There are some NFS group areas web areas served from toaster:
rfe -g amdmap/group | awk '/^\[/ {sec=$1} /toaster1/ {printf("/group/%s%s\n",sec,$1)}' | tr -d [ | tr ]
/group/bctcs
/group/project/aicat
/group/project/bioinformatics
/group/project/entrepedia
/group/project/hoppers
/group/project/ilsi
/group/project/inspace
/group/project/nxt
/group/project/perlis
/group/project/proofgeneral
/group/project/sicsa
/group/cisa/web
/group/conference/cav2005
/group/conference/compsac2005
/group/conference/emnlp08
/group/conference/emnlp09
/group/social/infcricket