(r2) FibreAttach < LCFG

Tags: view all tags
---++  Attaching a linux box to a SAN 




---++ %RED% *It is absolutely essential that all fibre connections are removed from a machine during installation* %ENDCOLOR%




---+++ Introduction 


   * %RED% *It is absolutely essential that all fibre connections are removed from a machine during installation* %ENDCOLOR%
   * Only the Qlogic 23XX and 24xx series cards are currently supported. 
   * %N%  Multipath [[FibreAttach#FibreMultiPath][notes]] %N% 

There are three stages to configuring a machine to use a SAN :- 


   * Configuring the fibre HBA (Host Bus Adapter) 
   * Identifying the required SAN volume(s) 
   * Configuring the fstab entries to mount the SAN Volume(s) 




---+++ Configuring the fibre HBA 


   1 Add one of the following include files to your machine's profile (under DICE, remember to use =<dice/options...>=) 
<verbatim>

#include <lcfg/options/qlogic_qla2300.h>
#include <lcfg/options/qlogic_qla2322.h>
#include <lcfg/options/qlogic_qla2400.h>

</verbatim>

   1 Check the profile is compiled and received by machine.
   1 Reboot the machine. This reboot will compile the qlogic driver module for the current kernel. It will then trigger an additional reboot.






---+++ Identifying the required SAN volume(s) 

The Linux SCSI stack, in particular its naming convention for SCSI devices, is extremely simplistic. Basically, it assigns device names to devices in the order that the kernel discovers them at boot (or module load) time, starting from */dev/sda*, then */dev/sdb*, through to */dev/sdz*. 

This is fine for static configurations where SCSI disks don't come and go but, in a SAN environment where new volumes and storage arrays are routinely added and removed, this simplistic naming convention creates real problems. If a new volume is created on a storage array, this will often result in many existing volumes being discovered after the new volume, and hence being assigned a different SCSI device name. 

Under FC5, the SCSI stack now creates a soft link to each _partition_ it finds on any attached disk (volume), whether it be local or on a SAN, in the =/dev/disk/by-id= directory. These soft links are based on the unique SCSI identifiers for the associated volumes. In effect, the SCSI stack is now automatically doing the work that the FC3 *udev* service did, with no configuration required; with the slight difference that the links are now per partition rather than being per disk (volume). 

In the following example, there are five volumes, the first three of which have just one partition, the fourth has two partitions and the fifth has currently no partitions :- 


<verbatim>

[blurt]ascobie: ls /dev/disk/by-id
scsi-36000402001ec048c6a04fc8b00000000
scsi-36000402001ec048c6a04fc8b00000000-part1
scsi-36000402001ec048c6a04fcbb00000000
scsi-36000402001ec048c6a04fcbb00000000-part1
scsi-36000402001ec048c6a05c0a700000000
scsi-36000402001ec048c6a05c0a700000000-part1
scsi-36000402001ec048c6a05c0f000000000
scsi-36000402001ec048c6a05c0f000000000-part1
scsi-36000402001ec048c6a05c0f000000000-part2
scsi-36000402001ec048c6c37286100000000
[blurt]ascobie: 

</verbatim>


In this example, we will create one partition in a volume on the NexSAN ATAbeast =atabeast1.inf.ed.ac.uk=. The volume's LUN is 1. The partition will be mounted as =/disk/testdata=. 




   1 Determine current SCSI device name for the volume. 
The first task is to determine what device name the SCSI stack has currently allocated for the volume. The easiest way to do this is to use the =lsscsi= utility. 
<verbatim>

[bottle]ascobie: lsscsi -k
[0:0:6:0]    process PE/PV    1x3 SCSI BP      1.1   -       
[0:2:0:0]    disk    MegaRAID LD 0 RAID1   34G 412W  /dev/sda
[1:0:0:0]    disk    NEXSAN   SATABeast        Cr41  /dev/sdb
[1:0:1:0]    disk    NEXSAN   SATABeast        Cr41  -       
[1:0:1:1]    disk    NEXSAN   SATABeast        Cr41  /dev/sdc
[1:0:1:3]    disk    NEXSAN   SATABeast        Cr41  /dev/sdd
[1:0:2:0]    disk    NEXSAN   ATAboy2F         5p41  /dev/sde
[1:0:2:1]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdf
[1:0:2:2]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdg
[1:0:2:3]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdh
[1:0:2:4]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdi
[1:0:2:5]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdj
[1:0:2:6]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdk
[1:0:2:7]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdl
[1:0:2:8]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdm
[1:0:2:9]    disk    NEXSAN   ATAboy2F         5p41  /dev/sdn
[1:0:2:10]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdo
[1:0:2:11]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdp
[1:0:2:12]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdq
[1:0:2:13]   disk    NEXSAN   ATAboy2F         5p41  /dev/sdr
[1:0:2:14]   disk    NEXSAN   ATAboy2F         5p41  /dev/sds
[1:0:3:0]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdt
[1:0:3:1]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdu
[1:0:3:2]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdv
[1:0:3:3]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdw
[1:0:3:4]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdx
[1:0:3:5]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdy
[1:0:3:6]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdz
[1:0:3:7]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdaa
[1:0:3:8]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdab
[1:0:3:9]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdac
[1:0:3:10]   disk    NEXSAN   ATAbeastF        8r41  /dev/sdad
[1:0:3:11]   disk    NEXSAN   ATAbeastF        8r41  /dev/sdae

</verbatim>


This command lists the SCSI volumes (devices) available. 

The 4th number in the bracketed value in the first column indicates the LUN of each volume. The last column indicates the current SCSI device name for each volume. 

We're after the volume with LUN 1 on the ATAbeast, which =lsscsi= shows to be currently =/dev/sdu=. 


<verbatim>

...
[1:0:3:1]    disk    NEXSAN   ATAbeastF        8r41  /dev/sdu
...

</verbatim>
If you have multiple NEXSAN arrays of the same type, you can no longer rely on using =lsscsi= to directly identify a specific array. Instead you must use the =scli -t= command to identify your target array using the "Port Name" of the array and map this to the SCSI stack's target id (the third value in the bracketed value in the first column in the output of =lsscsi=. 
<verbatim>

[bottle]root: scli -t
..
------------------------------------------------------------------------------
Path                       : 0
Target                     : 3
Device ID                  : 0x8a
Port ID                    : 65-0F-00
Product Vendor             : NEXSAN  
Product ID                 : ATAbeastF       
Product Revision           : 8r41
Node Name                  : 20-01-00-04-02-E8-08-5D
Port Name                  : 50-00-40-20-01-E8-08-5D
Product Type               : Disk
Number of LUN(s)           : 12
Status                     : Online
------------------------------------------------------------------------------
..

</verbatim>
In this case, the ATAbeast has target id 3 - this maps onto the third value in the bracketed value in the first column in the output of =lsscsi=. 



   1 Partition the volume 
Partition the volume as you would any SCSI device. In this example by calling =fdisk /dev/sdu=. 



   1 Create any filesystems for _new_ partitions. 
Create any filesystems. In this example by calling, eg, =mke2fs -j /dev/sdu1=. 



   1 Determine the unique SCSI identifier for the volume. 

<verbatim>

[bottle]root: /sbin/scsi_id -g -s /block/sdu
36000402001e8085d6a1882b400000000

</verbatim>
Note the use of =/block/sdu= rather than =/dev/sdu= here. 



   1 Check entries in =/dev/disk/by-id=
Check that there are entries in =/dev/disk/by-id= by looking for entries of the form <tt>scsi-</tt>
<i>unique SCSI identifier</i>
. 
<verbatim>

[bottle]root: ls  /dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000*
/dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000
/dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000-part1

</verbatim>
The second entry here (postfixed by =part1=) is the partition (=/dev/sdu1=) on the volume. 







---++++ Configuring fstab entries

Ideally, one would create a symlink in the =/dev= directory to the long-winded partition alias. However, this directory is wiped at boot so this is not possible. 




   1 Add =fstab.entries= entries to add to =/etc/fstab=

<verbatim>

!fstab.entries          mADD(testdata)
fstab.spec_testdata     /dev/disk/by-id/scsi-36000402001e8085d6a1882b400000000-part1
fstab.file_testdata     /disk/testdata
fstab.vfstype_testdata  ext3
fstab.passno_testdata   2

</verbatim>


(Choose another passno for the next partition) 



   1 reboot (after checking resources shipped) 




------------------------------------------------------------------------





---++ Multipath 


   * The =<lcfg/options/multipath.h>= header enables multipath support. Note that no multipath configuration is yet possible; a component may be written in the future if it proves necessary. (Remember to use =<dice/options/multipath.h>= under DICE) 
   * If you are using a qlogic HBA, you must disable the driver's failover option (This is disabled by default for DICE) 
   * The storage array must export the LUNs you want to access on all fibre paths. To avoid confusion, it's a very good idea to have the LUNs map to the same volumes on each path; otherwise it is very tricky to identify which volume is which. 
   * The multipath system presents the storage volumes and partitions under =/dev/mpath= rather than under =/dev/disk/by-id=, and without a =scsi= prefix. The volumes are still presented under =/dev/disk/by-id=, but you should not use these devices as they are hard-wired to one fibre path and will not failover if that path fails. 
   * Use the =/sbin/multipath -l= command to list the available volumes, instead of =/sbin/lsscsi= and =/sbin/scsi_id= used in the above instructions. Where there are many storage arrays connected, it will still be necessary to use =/sbin/scli= to identify the array required (as documented above). 

In the following example, LUN 4 of the !NexSAN SATABlade with IP address =129.215.46.190= (=81D72EBE=) is mapped to =/dev/mpath/36000402001ec048c6a04fcbb00000000= 
<verbatim>

[budapest]root: /sbin/multipath -l
.....
36000402001ec048c6a04fcbb00000000dm-2 NEXSAN,SATABl(81D72EBE)
[size=40G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:4  sdd 8:48  [active][undef]
 \_ 2:0:0:4  sdi 8:128 [active][undef]
36000402001ec048c6c37286100000000dm-1 NEXSAN,SATABl(81D72EBE)
[size=10G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:3  sdc 8:32  [active][undef]
 \_ 2:0:0:3  sdh 8:112 [active][undef]
....
[budapest]root:

</verbatim>

Jan 09 - Note that for Nexsan products, you may have to add the following to /etc/scsi_id.config in order that the multipath software identifies the two paths as being to the same volume.

<verbatim>
vendor="NEXSAN",options=-p 0x80

</verbatim>


-- Main.AlastairScobie - 27 Jul 2007
Topic revision: r2 - 2009-01-05 - ascobie
LCFG: Large Scale Unix Configuration System
~~Edit~~
~~Attach~~