Note I'm mastering this in emacs ~neilb/work/dice/fileservices/afs/dec.2012/forum-upgrades and will only update it occasionally. Wiki changes will be lost the next time I sync.

December 2012 Forum AFS Server Upgrades/Replacements

By the end of the year we need to have replaced 3 of the four existing Forum AFS file servers: squonk, bunyip, cameleopard and crocotta.

With the new: nessie, yeti, kraken

and have them all running SL6x64

Initial state is that the original machines serve all their data from the SAN, ifevo1 and ifevo2.

The new machines have 1.5TB of local RAID 10 disks, and no FC cards.

The plan is to get one FC card and install it in kraken, and then migrate all volumes from one of the old servers to kraken, so the old machine can be decommissioned and it's FC card installed in yeti. And so on until the new machines all have FC cards from the 3 decommissioned old machines.

This would leave one old machine with no where to migrate to (if we want to SL6 it without disrupting users).

Another goal would be to free up ifevo1 of all data, so we can update its firmware and probably recreate its vdisks.

Some rough facts and figures, sizes in TBs U=user volumes G=group volumes

machine Total disk Used disk Total disk for Groups Total disk for users
squonk 8 5 5 2
bunyip 7 5 6 1
cameleopard 5 4 5 1
crocotta 9 4 6 2.5

ifevo1 has 10TB RAID5 + 6TB RAID10

bunyip is showing a DIMM Parity error - so best that that's not the one that survives.

Vague Plan

ifevo2 has 16x2TB unallocated disks. So initially I've grabbed 5 of them to make an 8TB RAID5 vdisk, with the intention to move volumes from the next fileserver to be replaced/upgraded to, and then back off again once replaced. ie just as some shuffle space. Possibly not moving stuff back that comes off the ifevo1 so it gradually empties.

Repartition R720 RAID 10

parted commands

mkpart sdb1 ext3 0% 33%
mkpart sdb2 ext3 33% 66%
mkpart sdb3 ext3 66% 99%
simpler but a bit wasteful
mkpart sdb1 ext3 0% 476416MiB
mkpart sdb2 ext3 476416MiB 952831MiB
mkpart sdb3 ext3 952831MiB 100%
more efficient use of space but harder to read!

Migrate Cameleopard to Nessie

Cameleopard -> Nessie
type Source partitions sizes   Dest Size Dest Partitions Notes
group a,b,c,d 4x250G -> 2x500G d, e  
group e,f,g,h 4x500G -> 2x1000G f, g  
user i,j 2x250G -> 1x450G b shuffle some to c
group k,l 2x250G -> 1x500G h  
user m 1x250G -> 1x450G c plus some over spll from b
group o 1x760G -> 1x1000G i  

Leaving vicepa empty for extra users from crocotta. A total 4.5TB SAN space.

Time take to do moves: user.moves real 850m29.464s = 14.25hrs - the user partitions group.moves1 real 1559m1.203s = 26hrs - group partitions a to g group.moves2 real 1117m29.952s = 18.5hrs - group partitions h to o

Migrate bunyip to kraken

bunyip -> kraken
type Source partitions sizes   Dest Size Dest Partitions Notes
user a,b,c,d 4x250G -> 3x500G a,b,c a and c to a, b to b, d to c
group e,f,g,h 4x540G -> 4x5400G e,f,g,h  
group i 1x950G -> 1x1000G i  
group j,k 2x500G -> 2x500G j,k  
group l 1x1000G -> 1x1000G l  
group m 1x1100G -> 1x1000G m  

mostly 1 to 1 mapping, some sizes tweeked a wee bit.

Note on moving partitions between servers

After some Googling and chats in the OpenAFS chat room. These are the steps to do to move vicep's from server A to B (and having the volumes they contain move too).

  • Shutdown A
  • Disconnect partitions from A and attach them to B (they don't have to use the same vicep name as when on A).
  • Restart file server on B (if you need to)
  • vos syncvldb -server B
  • vos syncserver -server B
  • Restart A (without the moved vicep's) - this is to keep clients happy for the next 2 hours, after which A can be turned off (if it has no more volumes)
  • Double check that the vldb records things where they should be.
  • that's it

Still to do

  • update AFS partitions wiki page with bunyip/kraken move
  • reclaim free SAN space
  • crocotta move to yeti - needs bunyip's FC card
  • SL6 squonk.

-- NeilBrown - 14 Dec 2012

Edit | Attach | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r2 - 14 Dec 2012 - 11:11:03 - NeilBrown
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies