Note I'm mastering this in emacs ~neilb/work/dice/fileservices/afs/dec.2012/forum-upgrades and will only update it occasionally. Wiki changes will be lost the next time I sync.

December 2012 Forum AFS Server Upgrades/Replacements

By the end of the year we need to have replaced 3 of the four existing Forum AFS file servers: squonk, bunyip, cameleopard and crocotta.

With the new: nessie, yeti, kraken

and have them all running SL6x64

Initial state is that the original machines serve all their data from the SAN, ifevo1 and ifevo2.

The new machines have 1.5TB of local RAID 10 disks, and no FC cards.

The plan is to get one FC card and install it in kraken, and then migrate all volumes from one of the old servers to kraken, so the old machine can be decommissioned and it's FC card installed in yeti. And so on until the new machines all have FC cards from the 3 decommissioned old machines.

This would leave one old machine with no where to migrate to (if we want to SL6 it without disrupting users).

Another goal would be to free up ifevo1 of all data, so we can update its firmware and probably recreate its vdisks.

Some rough facts and figures, sizes in TBs U=user volumes G=group volumes

machine	Total disk	Used disk	Total disk for Groups	Total disk for users
squonk	8	5	5	2
bunyip	7	5	6	1
cameleopard	5	4	5	1
crocotta	9	4	6	2.5

ifevo1 has 10TB RAID5 + 6TB RAID10

bunyip is showing a DIMM Parity error - so best that that's not the one that survives.

Vague Plan

ifevo2 has 16x2TB unallocated disks. So initially I've grabbed 5 of them to make an 8TB RAID5 vdisk, with the intention to move volumes from the next fileserver to be replaced/upgraded to, and then back off again once replaced. ie just as some shuffle space. Possibly not moving stuff back that comes off the ifevo1 so it gradually empties.

Repartition R720 RAID 10

parted commands

mkpart sdb1 ext3 0% 33%
mkpart sdb2 ext3 33% 66%
mkpart sdb3 ext3 66% 99%
simpler but a bit wasteful
 or
mkpart sdb1 ext3 0% 476416MiB
mkpart sdb2 ext3 476416MiB 952831MiB
mkpart sdb3 ext3 952831MiB 100%
more efficient use of space but harder to read!

Migrate Cameleopard to Nessie

Cameleopard -> Nessie
type	Source partitions	sizes		Dest Size	Dest Partitions	Notes
group	a,b,c,d	4x250G	->	2x500G	d, e
group	e,f,g,h	4x500G	->	2x1000G	f, g
user	i,j	2x250G	->	1x450G	b	shuffle some to c
group	k,l	2x250G	->	1x500G	h
user	m	1x250G	->	1x450G	c	plus some over spll from b
group	o	1x760G	->	1x1000G	i

Leaving vicepa empty for extra users from crocotta. A total 4.5TB SAN space.

Time take to do moves:

• user.moves real 850m29.464s = 14.25hrs - the user partitions
• group.moves1 real 1559m1.203s = 26hrs - group partitions a to g
• group.moves2 real 1117m29.952s = 18.5hrs - group partitions h to o

Migrate bunyip to kraken

bunyip -> kraken
type	Source partitions	sizes		Dest Size	Dest Partitions	Notes
user	a,b,c,d	4x250G	->	3x500G	a,b,c	a and c to a, b to b, d to c
group	e,f,g,h	4x540G	->	4x5400G	e,f,g,h
group	i	1x950G	->	1x1000G	i
group	j,k	2x500G	->	2x500G	j,k
group	l	1x1000G	->	1x1000G	l
group	m	1x1100G	->	1x1000G	m

mostly 1 to 1 mapping, some sizes tweeked a wee bit.

• group.ami8, the single 990GB volume on vicepm took 14h to move.
• user vol moves real 985m29.680s = 16h
• group2moves real 1311m36.477s
• group1moves real 1863m50.228s

Migrate crocotta to yeti

crocotta -> yeti
type	Source partitions	sizes		Dest Size	Dest Partitions	Notes
user	a,b,c,d,e,f	6x250G	->	3x500G	a,b,c	ab to a, cd to b, ef to c
group	g	1x250G	->	1x500G	d	mixed
user	h	1x250G	->	1x500G	d	mixed
group	i	1.2TB	->	1.2TB	e
group	j,k	2x600GB	->	2x600GB	f,g
group	n	1.2TB	->	1.2TB	h
group	o	1TB	->	1TB	i
group	p	1TB	->	1TB	j
mpu	squonk	580GB	->	580GB	k
user	l,m	800GB	->	800GB	l

# crocotta
./afsmigrate-partition crocotta a yeti a 
./afsmigrate-partition crocotta b yeti a a & b 711mins
./afsmigrate-partition crocotta c yeti b
./afsmigrate-partition crocotta d yeti b
./afsmigrate-partition crocotta e yeti c 3h:14m
./afsmigrate-partition crocotta f yeti c 2h:55m.28s

./afsmigrate-partition crocotta g yeti d
./afsmigrate-partition crocotta h yeti d
./afsmigrate-partition crocotta i yeti e
./afsmigrate-partition crocotta j yeti f
./afsmigrate-partition crocotta k yeti g all 1324mins

./afsmigrate-partition crocotta l yeti l
./afsmigrate-partition crocotta m yeti l
./afsmigrate-partition crocotta n yeti h 667m (11.1hr) for lmn
./afsmigrate-partition crocotta o yeti i 4h:53m.21s
./afsmigrate-partition crocotta p yeti j 2h:44m.21s



d 00c0ffd863b400007a95cf5001000000
 3600c0ff000d863b47a95cf5001000000
e 00c0ffd863b40000ab95cf5001000000
 3600c0ff000d863b4ab95cf5001000000
fg 00c0ffd863b40000d595cf5001000000
  3600c0ff000d863b4d595cf5001000000
h 00c0ffd863b40000fe95cf5001000000
 3600c0ff000d863b4fe95cf5001000000
i 00c0ffd863b400001f96cf5001000000
 3600c0ff000d863b41f96cf5001000000
j 00c0ffd863b400004396cf5001000000
 3600c0ff000d863b44396cf5001000000
k 00c0ffd8545a00007a4fcf5001000000
 3600c0ff000d8545a7a4fcf5001000000
l 00c0ffd863b40000a69bcf5001000000
 3600c0ff000d863b4a69bcf5001000000

nessie k 00c0ffd863b40000589fcf5001000000 - small squonk groups vols
        3600c0ff000d863b4589fcf5001000000

afsmigrate-partition squonk g nessie k
afsmigrate-partition squonk i nessie k
afsmigrate-partition squonk n nessie k
afsmigrate-partition squonk o nessie k
The above to 368mins = 6.1hrs

afsmigrate-partition squonk p yeti k took 4h:14m

Note on moving partitions between servers

After some Googling and chats in the OpenAFS chat room. These are the steps to do to move vicep's from server A to B (and having the volumes they contain move too).

• Shutdown A
• Disconnect partitions from A and attach them to B (they don't have to use the same vicep name as when on A).
• Restart file server on B (if you need to)
• vos syncvldb -server B
• vos syncserver -server B
• Restart A (without the moved vicep's) - this is to keep clients happy for the next 2 hours, after which A can be turned off (if it has no more volumes)
• Double check that the vldb records things where they should be.
• that's it

Still to do

• update AFSPartitions wiki page with bunyip/kraken move
• reclaim free SAN space
• crocotta move to yeti - needs bunyip's FC card
• SL6 squonk.

squonk moves

 900GB vicepn on kraken, move squonk a,b,c,d to it LUN27  674mins
1100GB vicepj on nessie, move squonk h,j,k,l LUN 6 726m
squonk f to nessie a 250GB