The "Top Five" things you need to know in case of emergency
We are gradually deprecating the use of storage arrays within the School in favour of servers with lots of disks. At KB, only the icsa-kbevo is in active use, and even that is debatable. Of the three
DotHill Evo arrays in the Forum, two of these will be withdrawn after the current spending round, with the last array going in 2020 when its warranty expires.
These arrays:
- have a bunch of disks which can be divided in to groups to create a Vdisk/Virtual Disk.
- these VDisks can then have Volumes created within them, which are the things that can then be mounted as a single disk to a client machine connected to the SAN infrastructure (Fibre channel switches).
- You use Volume Mapping to limit which clients can see/access which Volume, otherwise you run the risk of multiple machines mounting one single "disk".
- The choice of RAID level will automatically give you some redundancy from a single disk failure, but the arrays keep a couple of the physical disks in the enclosures as "hot spares", which can be brought into service if an active disk does fail. Note that some arrays using RAID10 do not have hot spares though there should be a cold spare nearby. This of course requires human intervention,
- All, of our storage arrays have dual controllers, which means that we can suffer the failure of one logic board, and they will keep running. As well as having redundant PSUs.
The Wiki page
ATABoxesInfo contains the current list of hardware.
Managing/Monitoring the boxes
All the machines should have web interfaces, though occasionally their
web servers do lock up. When this happens, you can restart the management controller (and hence the web interface) by ssh'ing into the appropriate array.
They all use the same username and a variation of the password to
manage them. The variation is listed on the
ATABoxesInfo page. Ask a
member of the services unit, though I think Tim and Alastair also
know, the base password. The Evos get unhappy if more than one person is logged into the web interface at a time. If this happens, it's worth noting that both the EVO
controllers are networked, so there are "a" and "b" versions to connect to.
Power cycling
Don't. Or at least don't just do it unless you have no other option,
ie it has completely wedged and is no longer serving file space. Power
cycling a SAN box while servers have volumes mounted, will cause the
servers (and you) lots of pain. To do things cleanly, you need to make
sure all volumes have been unmounted from the servers, typically this
means shutting them down.
Due to an incident in the past when turning off an Evo, we've now
taken to disconnecting the fibre connections (one at a time) from the
Evo ports (giving the SAN time to settle between each disconnect)
before turning off Evo hardware. Similarily when turning back on on
plug the FC connections back in (to their original ports) one at a
time, again giving the fabric a few moments to reconfigure between ports.
Identifying what's mounted where
These days, it should be obvious from the volume name. You can then check on that host to
confirm that it is actually mounting it. How that is done varies, but basic
pointers would be to use the commands "lsscsi -k", "multipath -l",
"df" and looking in /etc/fstab.
If the volume has no LUN mask set, then potentially any machine
physically connected to the SAN infrastructure could be mounting it.
LUN masking
The volumes defined on the storage arrays appear as SCSI type disks to the machines on the SAN. If you were
to do an "lsscsi -k" (to list all the "SCSI" devices it is connected to) on
squonk.inf
you might see something like:
[2:0:3:0] disk DotHill R/Evo 2730-2R J200 /dev/sde
[2:0:3:1] disk DotHill R/Evo 2730-2R J200 /dev/sdf
and on
crocotta.inf
[2:0:3:2] disk DotHill R/Evo 2730-2R J200 /dev/sdd
[2:0:3:3] disk DotHill R/Evo 2730-2R J200 /dev/sde
Here the 4 numbers in the square brackets [] are the [host bus adapter:controller:scsi target:LUN]. So even though both machines are connected to the same SAN and so should see the same volumes as defined on the Evo box, they are actually only seeing the devices each is meant to see. This is due to the LUN masking.
Each port connection on the SAN has a unique Port WWN (see
http://en.wikipedia.org/wiki/Fibre_Channel), think of it like a ether MAC address. So when setting up Volume Mapping on the Evo's, you are saying expose volumes foo and bar as LUNs 0 and 1 to squonk's HBA Ports WWN and similarily volumes wibble and bibble as LUNS 2 and 3 to crocotta's HBA Ports WWN.
Note that there is a limit to the number of LUNs you can allocate though this is probably not going to be of concern to anyone outside the Services Unit.
We are using multipathing (speak to MPU about what that is) were
possible, some hosts will have two WWNs, both need to be given access
to the LUN so that both paths can reach the
device/volume. Multipathing then ties the two fibre paths to that
single volume together to give us redundancy, eg:
[squonk]root: /sbin/multipath -l
3600c0ff000d863b4a5637f4c01000000 dm-3 DotHill,R/Evo 2730-2R
[size=1.1T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 2:0:6:11 sde 8:64 [active][undef]
\_ 1:0:9:11 sdi 8:128 [active][undef]
Here a volume on the Evo has been lun masked to both FC ports (two
WWNs) for squonk, both as LUN 11 to keep things sane. There are now
two physical paths to that volume from squonk, one out FC port 1
(adapter 1 as /dev/sdi) and the other via FC port 2 (adapter 2 as
/dev/sde). These two paths are then bonded together as
/dev/mapper/3600c0ff000d863b4a5637f4c01000000
As the
DotHill arrays have two controllers, each with 2 FC connections, the Forum servers will see 4 paths to the same volume.
Problems
- Complete wedges - rare, but not unheard off, if the box isn't responding to anything, serial console, web or serving data, then you have little choice but to power cycle the box. The damage to the clients mounting things from it has already been done, so there's little to lose
- Lost LUN masking - rare, but it has happened, the array loses its record of what machines can see which volumes, you have to go back into the web interface for the box and reconfigure the LUN masks for the affected volumes. Which you may be able to guess from the name of the volumes, or you need to find a record of what they were last set to. Currently we have no automated record of the lun masking, but Neil's likely to have a copy of a machines last config dump in ~neilb/san/ dated the last time a fault was reported with the hardware.
- Disk failures - these do happen, most machines can suffer at least 2 disk failures without data loss. However at the next disk failure data will be lost. So you need to arrange to replace the failed disks quickly. This can be done live, and is easier on some machines compared to others. The drives face out of the rack, and can just be popped out and in again. As mentioned above. we keep cold spares for the arrays in the server room on the Services Unit racks.
The boxes should send status alerts and updates to the mailing list
ata-info@inf, but this sometimes stops working (the sending of the mail).
Some links
Here are a selection of links to pages containing SAN info, not
already listed above. How current some are is up for debate!
--
CraigStrachan - 26 Feb 2019