The "Top Five" things you need to know in case of emergency

We are gradually deprecating the use of storage arrays within the School in favour of servers with lots of disks. At KB, only the icsa-kbevo is in active use, and even that is debatable. Of the three DotHill Evo arrays in the Forum, two of these will be withdrawn after the current spending round, with the last array going in 2020 when its warranty expires.

These arrays:

• have a bunch of disks which can be divided in to groups to create a Vdisk/Virtual Disk.
• these VDisks can then have Volumes created within them, which are the things that can then be mounted as a single disk to a client machine connected to the SAN infrastructure (Fibre channel switches).
• You use Volume Mapping to limit which clients can see/access which Volume, otherwise you run the risk of multiple machines mounting one single "disk".
• The choice of RAID level will automatically give you some redundancy from a single disk failure, but the arrays keep a couple of the physical disks in the enclosures as "hot spares", which can be brought into service if an active disk does fail. Note that some arrays using RAID10 do not have hot spares though there should be a cold spare nearby. This of course requires human intervention,
• All, of our storage arrays have dual controllers, which means that we can suffer the failure of one logic board, and they will keep running. As well as having redundant PSUs.

The Wiki page ATABoxesInfo contains the current list of hardware.

Managing/Monitoring the boxes

All the machines should have web interfaces, though occasionally their web servers do lock up. When this happens, you can restart the management controller (and hence the web interface) by ssh'ing into the appropriate array.

They all use the same username and a variation of the password to manage them. The variation is listed on the ATABoxesInfo page. Ask a member of the services unit, though I think Tim and Alastair also know, the base password. The Evos get unhappy if more than one person is logged into the web interface at a time. If this happens, it's worth noting that both the EVO controllers are networked, so there are "a" and "b" versions to connect to.

Power cycling

Don't. Or at least don't just do it unless you have no other option, ie it has completely wedged and is no longer serving file space. Power cycling a SAN box while servers have volumes mounted, will cause the servers (and you) lots of pain. To do things cleanly, you need to make sure all volumes have been unmounted from the servers, typically this means shutting them down.

Due to an incident in the past when turning off an Evo, we've now taken to disconnecting the fibre connections (one at a time) from the Evo ports (giving the SAN time to settle between each disconnect) before turning off Evo hardware. Similarily when turning back on on plug the FC connections back in (to their original ports) one at a time, again giving the fabric a few moments to reconfigure between ports.

Identifying what's mounted where

These days, it should be obvious from the volume name. You can then check on that host to confirm that it is actually mounting it. How that is done varies, but basic pointers would be to use the commands "lsscsi -k", "multipath -l", "df" and looking in /etc/fstab.

If the volume has no LUN mask set, then potentially any machine physically connected to the SAN infrastructure could be mounting it.

LUN masking

The volumes defined on the storage arrays appear as SCSI type disks to the machines on the SAN. If you were to do an "lsscsi -k" (to list all the "SCSI" devices it is connected to) on squonk.inf you might see something like:

[2:0:3:0]    disk    DotHill  R/Evo 2730-2R    J200  /dev/sde
[2:0:3:1]    disk    DotHill  R/Evo 2730-2R    J200  /dev/sdf

and on crocotta.inf

[2:0:3:2]    disk    DotHill  R/Evo 2730-2R    J200  /dev/sdd
[2:0:3:3]    disk    DotHill  R/Evo 2730-2R    J200  /dev/sde

Here the 4 numbers in the square brackets [] are the [host bus adapter:controller:scsi target:LUN]. So even though both machines are connected to the same SAN and so should see the same volumes as defined on the Evo box, they are actually only seeing the devices each is meant to see. This is due to the LUN masking.

Each port connection on the SAN has a unique Port WWN (see http://en.wikipedia.org/wiki/Fibre_Channel), think of it like a ether MAC address. So when setting up Volume Mapping on the Evo's, you are saying expose volumes foo and bar as LUNs 0 and 1 to squonk's HBA Ports WWN and similarily volumes wibble and bibble as LUNS 2 and 3 to crocotta's HBA Ports WWN.

Note that there is a limit to the number of LUNs you can allocate though this is probably not going to be of concern to anyone outside the Services Unit.

We are using multipathing (speak to MPU about what that is) were possible, some hosts will have two WWNs, both need to be given access to the LUN so that both paths can reach the device/volume. Multipathing then ties the two fibre paths to that single volume together to give us redundancy, eg:

[squonk]root:  /sbin/multipath -l
3600c0ff000d863b4a5637f4c01000000 dm-3 DotHill,R/Evo 2730-2R
[size=1.1T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 2:0:6:11 sde 8:64  [active][undef]
 \_ 1:0:9:11 sdi 8:128 [active][undef]

Here a volume on the Evo has been lun masked to both FC ports (two WWNs) for squonk, both as LUN 11 to keep things sane. There are now two physical paths to that volume from squonk, one out FC port 1 (adapter 1 as /dev/sdi) and the other via FC port 2 (adapter 2 as /dev/sde). These two paths are then bonded together as /dev/mapper/3600c0ff000d863b4a5637f4c01000000

As the DotHill arrays have two controllers, each with 2 FC connections, the Forum servers will see 4 paths to the same volume.

Problems

• Complete wedges - rare, but not unheard off, if the box isn't responding to anything, serial console, web or serving data, then you have little choice but to power cycle the box. The damage to the clients mounting things from it has already been done, so there's little to lose
• Lost LUN masking - rare, but it has happened, the array loses its record of what machines can see which volumes, you have to go back into the web interface for the box and reconfigure the LUN masks for the affected volumes. Which you may be able to guess from the name of the volumes, or you need to find a record of what they were last set to. Currently we have no automated record of the lun masking, but Neil's likely to have a copy of a machines last config dump in ~neilb/san/ dated the last time a fault was reported with the hardware.
• Disk failures - these do happen, most machines can suffer at least 2 disk failures without data loss. However at the next disk failure data will be lost. So you need to arrange to replace the failed disks quickly. This can be done live, and is easier on some machines compared to others. The drives face out of the rack, and can just be popped out and in again. As mentioned above. we keep cold spares for the arrays in the server room on the Services Unit racks.

The boxes should send status alerts and updates to the mailing list ata-info@inf, but this sometimes stops working (the sending of the mail).

Some links

Here are a selection of links to pages containing SAN info, not already listed above. How current some are is up for debate!

• ServicesSANStuff

-- CraigStrachan - 26 Feb 2019