Attaching a linux box to a SAN

It is absolutely essential that all fibre connections are removed from a machine during installation

Introduction

  • It is absolutely essential that all fibre connections are removed from a machine during installation
  • Only the Qlogic 23XX and 24xx series cards are currently supported.
  • NEW Multipath notes NEW

There are three stages to configuring a machine to use a SAN :-

  • Configuring the fibre HBA (Host Bus Adapter)
  • Identifying the required SAN volume(s)
  • Configuring the fstab entries to mount the SAN Volume(s)

Configuring the fibre HBA

  1. Add one of the following include files to your machine's profile (under DICE, remember to use <dice/options...>)

#include <lcfg/options/qlogic_qla2300.h>
#include <lcfg/options/qlogic_qla2322.h>
#include <lcfg/options/qlogic_qla2400.h>

  1. Check the profile is compiled and received by machine.
  2. Reboot the machine. This reboot will compile the qlogic driver module for the current kernel. It will then trigger an additional reboot.

Identifying the required SAN volume(s)

The Linux SCSI stack, in particular its naming convention for SCSI devices, is extremely simplistic. Basically, it assigns device names to devices in the order that the kernel discovers them at boot (or module load) time, starting from /dev/sda, then /dev/sdb, through to /dev/sdz.

This is fine for static configurations where SCSI disks don't come and go but, in a SAN environment where new volumes and storage arrays are routinely added and removed, this simplistic naming convention creates real problems. If a new volume is created on a storage array, this will often result in many existing volumes being discovered after the new volume, and hence being assigned a different SCSI device name.

If you have just added a new volume to an array, you will probably need to get the HBA to rescan the bus to detect it. For QLogic cards, the following should work
  echo "scsi-qlascan" > /proc/scsi/qla2xxx/1
  echo "- - -" > /sys/class/scsi_host/host/scan
  or rescan-scsi-bus.sh -c -l -w
Though you will have to use the path appropriate to your HBA and port in use. And obtain the rescan-scsi-bus.sh script from the web.
Under FC5, the SCSI stack now creates a soft link to each partition it finds on any attached disk (volume), whether it be local or on a SAN, in the /dev/disk/by-id directory. These soft links are based on the unique SCSI identifiers for the associated volumes. In effect, the SCSI stack is now automatically doing the work that the FC3 udev service did, with no configuration required; with the slight difference that the links are now per partition rather than being per disk (volume).

In the following example, there are five volumes, the first three of which have just one partition, the fourth has two partitions and the fifth has currently no partitions :-


[blurt]ascobie: ls /dev/disk/by-id
scsi-36000402001ec048c6a04fc8b00000000
scsi-36000402001ec048c6a04fc8b00000000-part1
scsi-36000402001ec048c6a04fcbb00000000
scsi-36000402001ec048c6a04fcbb00000000-part1
scsi-36000402001ec048c6a05c0a700000000
scsi-36000402001ec048c6a05c0a700000000-part1
scsi-36000402001ec048c6a05c0f000000000
scsi-36000402001ec048c6a05c0f000000000-part1
scsi-36000402001ec048c6a05c0f000000000-part2
scsi-36000402001ec048c6c37286100000000
[blurt]ascobie: 

In this example, we will create one partition in a volume on the NexSAN ATAbeast atabeast1.inf.ed.ac.uk. The volume's LUN is 1. The partition will be mounted as /disk/testdata.

  1. Determine current SCSI device name for the volume.
The first task is to determine what device name the SCSI stack has currently allocated for the volume. The easiest way to do this is to use the lsscsi utility.

[bottle]ascobie: lsscsi -k
[0:0:6:0]    process PE/PV    1x3 SCSI BP      1.1   -       
[0:2:0:0]    disk    MegaRAID LD 0 RAID1   34G 412W  /dev/sda
[1:0:0:0]    disk    NEXSAN   SATABeast        Cr41  /dev/sdb
[1:0:1:0]    disk    NEXSAN   SATABeast        Cr41  -       
[1:0:1:1]    disk    NEXSAN   SATABeast        Cr41  /dev/sdc
[1:0:1:3]    disk    NEXSAN   SATABeast        Cr41  /dev/sdd
[1:0:2:0]    disk    NEXSAN   ATAboy2F         5p41  /dev/sde
[1:0:2:1]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdf
[1:0:2:2]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdg
[1:0:2:3]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdh
[1:0:2:4]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdi
[1:0:2:5]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdj
[1:0:2:6]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdk
[1:0:2:7]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdl
[1:0:2:8]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdm
[1:0:2:9]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdn
[1:0:2:10]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdo
[1:0:2:11]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdp
[1:0:2:12]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdq
[1:0:2:13]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdr
[1:0:2:14]   disk    NEXSAN   ATAboy2F         5p41  /dev/sds
[1:0:3:0]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdt
[1:0:3:1]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdu
[1:0:3:2]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdv
[1:0:3:3]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdw
[1:0:3:4]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdx
[1:0:3:5]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdy
[1:0:3:6]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdz
[1:0:3:7]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdaa
[1:0:3:8]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdab
[1:0:3:9]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdac
[1:0:3:10]   disk    NEXSAN   ATAbeastF        8r41  /dev/sdad
[1:0:3:11]   disk    NEXSAN   ATAbeastF        8r41  /dev/sdae

This command lists the SCSI volumes (devices) available.

The 4th number in the bracketed value in the first column indicates the LUN of each volume. The last column indicates the current SCSI device name for each volume.

We're after the volume with LUN 1 on the ATAbeast, which lsscsi shows to be currently /dev/sdu.


...
[1:0:3:1]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdu
...

If you have multiple NEXSAN arrays of the same type, you can no longer rely on using lsscsi to directly identify a specific array. Instead you must use the scli -t command to identify your target array using the "Port Name" of the array and map this to the SCSI stack's target id (the third value in the bracketed value in the first column in the output of lsscsi.

[bottle]root: scli -t
..
------------------------------------------------------------------------------
Path                       : 0
Target                     : 3
Device ID                  : 0x8a
Port ID                    : 65-0F-00
Product Vendor             : NEXSAN  
Product ID                 : ATAbeastF       
Product Revision           : 8r41
Node Name                  : 20-01-00-04-02-E8-08-5D
Port Name                  : 50-00-40-20-01-E8-08-5D
Product Type               : Disk
Number of LUN(s)           : 12
Status                     : Online
------------------------------------------------------------------------------
..

In this case, the ATAbeast has target id 3 - this maps onto the third value in the bracketed value in the first column in the output of lsscsi.

  1. Partition the volume
Partition the volume as you would any SCSI device. In this example by calling fdisk /dev/sdu.

  1. Create any filesystems for new partitions.
Create any filesystems. In this example by calling, eg, mke2fs -j /dev/sdu1.

  1. Determine the unique SCSI identifier for the volume.


[bottle]root: /sbin/scsi_id -g -s /block/sdu
36000402001e8085d6a1882b400000000

Note the use of /block/sdu rather than /dev/sdu here.

  1. Check entries in /dev/disk/by-id
Check that there are entries in /dev/disk/by-id by looking for entries of the form scsi- unique SCSI identifier .

[bottle]root: ls  /dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000*
/dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000
/dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000-part1

The second entry here (postfixed by part1) is the partition (/dev/sdu1) on the volume.

Configuring fstab entries

Ideally, one would create a symlink in the /dev directory to the long-winded partition alias. However, this directory is wiped at boot so this is not possible.

  1. Add fstab.entries entries to add to /etc/fstab


!fstab.entries          mADD(testdata)
fstab.spec_testdata     /dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000-part1
fstab.file_testdata     /disk/testdata
fstab.vfstype_testdata  ext3
fstab.passno_testdata   2

(Choose another passno for the next partition)

  1. reboot (after checking resources shipped)


Multipath

  • The <lcfg/options/multipath.h> header enables multipath support. Note that no multipath configuration is yet possible; a component may be written in the future if it proves necessary. (Remember to use <dice/options/multipath.h> under DICE)
  • If you are using a qlogic HBA, you must disable the driver's failover option (This is disabled by default for DICE)
  • The storage array must export the LUNs you want to access on all fibre paths. To avoid confusion, it's a very good idea to have the LUNs map to the same volumes on each path; otherwise it is very tricky to identify which volume is which.
  • The multipath system presents the storage volumes and partitions under /dev/mpath rather than under /dev/disk/by-id, and without a scsi prefix. The volumes are still presented under /dev/disk/by-id, but you should not use these devices as they are hard-wired to one fibre path and will not failover if that path fails.
  • Use the /sbin/multipath -l command to list the available volumes, instead of /sbin/lsscsi and /sbin/scsi_id used in the above instructions. Where there are many storage arrays connected, it will still be necessary to use /sbin/scli to identify the array required (as documented above).

In the following example, LUN 4 of the NexSAN SATABlade with IP address 129.215.46.190 (81D72EBE) is mapped to /dev/mpath/36000402001ec048c6a04fcbb00000000


[budapest]root: /sbin/multipath -l
.....
36000402001ec048c6a04fcbb00000000dm-2 NEXSAN,SATABl(81D72EBE)
[size=40G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:4  sdd 8:48  [active][undef]
 \_ 2:0:0:4  sdi 8:128 [active][undef]
36000402001ec048c6c37286100000000dm-1 NEXSAN,SATABl(81D72EBE)
[size=10G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:3  sdc 8:32  [active][undef]
 \_ 2:0:0:3  sdh 8:112 [active][undef]
....
[budapest]root:

Jan 09 - Note that for Nexsan products, you may have to add the following to /etc/scsi_id.config in order that the multipath software identifies the two paths as being to the same volume.

vendor="NEXSAN",options=-p 0x80

-- AlastairScobie - 27 Jul 2007

Edit | Attach | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | More topic actions...
Topic revision: r4 - 2009-04-29 - neilb
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback