SAN Stuff

I think we need a place to grub together useful nuggets about accessing/interrogating SAN attached hardware. Why not here as a starting point? We probably need to differentiate between linux and Solaris info. And point out anything that is machine specific e.g. due to different hardware/drivers

Todo list. ServicesSANToDo


Just to get the ball rolling:

  • lsscsi -- useful for listing all the old style /dev/sda type disk devices and seeing the scsi_host and LUN number for each [Host:Channel:Id:Lun]
  • lsscsi | sort -t: -k4 -n -- useful to list devices by LUN, so you see which LUNs are not being used
  • /sys/class/fc_host/host/X - Contains lots of info about the HBA installed. Rarely used.
  • See the old linux teams info on the subject now moved to
  • If you create a new volume on an array, then the fibre should notice the change automatically, but to add the device as a new scsi device then do echo "- - -" > /sys/class/scsi_host/host<controller>/scan . Where <controller> is the H field of the output of lsscsi.
  • dmesg -H | tail -- to see devices etc being detected when you do the scan, or setup the LUN mask on the SAN box.
  • multipath -l -- list all the /dev/mapper/DEVICE devices and their associated /dev/sdXX scsi devices. Check DEVICE bears some resemblance to the the volume id/serial number on the SAN box, and the LUNs of the scsi devices match what you set on the SAN box.

Removing SAN volumes/partitions

  • remove data from /vicep or /disk/ptn
  • locate corresponding SAN volume. Remember that a single SAN volume could be partitioned into several mount points on a single machine. So you need to make sure all the mounts are unused before deleting the volume.
    • comments in profile may say which SAN box the mount is from, if not you'll just have to log into each box and look for a volume with the right name (not always correct).
    • check you've identified the correct SAN volume by checking its
      • volume id should be similar to /dev/mapper/XXXX id eg 00c0ff1995e50000dc2eb75701000000 => 3600c0ff0001995e5dc2eb75701000000
      • lun mapping host WWN matches WWN of FC ports on host (~neilb/bin/share/fcwwns). Also note there's usual 2 per host eg 2100001b321cd40b => host1 0x2100001b321cd40b 21:00:00:1b:32:1c:d4:0b
      • lun mapping value on SAN with dev/mapper using (as root) multipath -l eg LUN 13 => multipath -l 3600c0ff0001995e5dc2eb75701000000
3600c0ff0001995e5dc2eb75701000000 dm-0 DotHill ,DH3000
size=2.4T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=active
| `- 1:0:0:13 sde 8:64   active undef unknown
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 1:0:3:13 sdo 8:224  active undef unknown
      note 13

Once you are happy you've removed all the data from a volume, and identifed the volume.

  • umount the partition(s)
  • remove partitions from the machine profile
  • remove the LUN mappings for the volume on the SAN box. You should see the paths failing, or being removed from "multipath -l DEVICEID"
  • multipath -f DEVICEID to remove DEVICEID from the list of multipath devices
  • If not rebooting, I like to remove the now orphaned /dev/sdX type devices with
    echo "scsi remove-single-device H C I L" > /proc/scsi/scsi
    where H C I L can been from multipath -l before it was removed, or lsscsi
    • remove-single etc seems to be the old way,
      echo 1 > /sys/class/scsi_device/H:C:I:L/device/delete
      seems to be the new approved way.
  • rename the volumes on the SAN box "del..." or "delete..." as a reminder. A few days later actually remove them. This gives you the possibility of undoing things if you accidentally removed the LUN masking from the wrong volume.
  • update wiki pages that may refer to the removed partitions.

Multipath funnies

Occasionally, eg if re-patching FC cables or changing LUN masking, deleting volumes on the SAN. You can leave multipath hanging on to devices that no longer exist, and though normally multipath -F would clear unused devices, sometimes it doesn't. Which then means you get nagios moaining about failed paths, that you don't care about any more.

Alastair reported in the chat room that echo 1 > /sys/block/sdb/device/delete (where 'sdb' is one of the reportedly failed paths) will remove that path from multipath's gaze. Not had a chance to try this for myself yet though.

Re-sizing a volume

This can be done (on the R/Evo) with the Volume Management: expand volume option. This bit appears to be file system safe, I did it to an ext3 FS and it survived - neilb.

Note, however, that where the volume is already mapped, utilities like fdisk may not see the change in size (the appropriate kernel values have not been updated). To fix this, delete and recreate the device using the kernel entry point /proc/scsi/scsi and re-scan.

Confirm ID(s) of device(s) to be deleted using "lsscsi -k" or "multipath -l", then:

  • for each Host:Channel:ID:LUN (usually in pairs), use:
         # echo "scsi remove-single-device H C I L" > /proc/scsi/scsi
This will delete the /dev/dm-N device and associated /dev/sd* devices.
...then add it back:
  • for each Host:Channel:ID:LUN, use:
        # echo "scsi add-single-device H C I L" >/proc/scsi/scsi
...and re-scan the SCSI bus:
          # echo "- - -" > /sys/class/scsi_host/host1/scan
          # echo "- - -" > /sys/class/scsi_host/host2/scan

fdisk should now see the change in size.

-- RogerBurroughes - 09 Jun 2010

But to enlarge an existing file system, there's more to do. I used as a guide, but it boils down to:

  • umount
  • fsck -- sanity check that it wasn't broken before the next bits
  • fdisk -- remove the existing partition, recreate at the same place but with the new end cyclinder (so this only really works on a volume with a single partition, or its the last partition you're expanding)
  • tune2fs -O ^has_journal -- remove journal
  • e2fsck -f -C 0 -- force check prior to ...
  • resize2fs -p -- the actual resizing, if brave then you can probably give it -f to skip needing the previous forced fsck
  • fsck -- just to make sure!
  • tune2fs -j -- put back journal
  • mount

-- NeilBrown - 21 Aug 2012

The ATAthings

Note that the Nexsan ATAthings use LUN masking, which may explain why you can't see the volumes you are expecting on the machine attached to the SAN. Check the ServicesUnitPortnameInfo page for the nodenames of the hardware.

Also check the ATABoxesInfo for details about the ATA hardware, including warranty and maintenance details.

There is a page detailing the volumes on each disk array at ServicesUnitSANVolumes

list of replaced Evo disks: ServicesUnitEvoDisks

EVO serial console adapter wiring: ServicesSANStuffEvoConsole

-- NeilBrown - 31 Jul 2006

Topic revision: r21 - 26 Jun 2019 - 12:22:24 - NeilBrown
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies