|
RHAS / HP EVA SAN
Notes from attaching linux RHAS to an HP/Compaq SAN with an
Emulex LP950 HBA.
Background: The QLogic QLA2340/QLA2342 fibre channel HBAs are currently the
only supported cards for attaching a Linux server to an EVA. This document explains
how to use an Emulex LP950 HBA. There are some issues with using the Emulex HBA
which are documented along with work arounds in some cases.
Preliminary steps: Install the Emulex HBA and fibre attach to the SAN.
1. Get the open build driver and utility from Emulex.
2. Compile the driver:
a. Extract the driver & util tar file into a working directory.
b. Edit lpfc.conf.c as follows:
lpfc_network_on=0 (disables IP networking on the HBA)
lpfc_scandown=0 (Used for arbitrated loops)
lpfc_lun0_missing=0 (Scan past lun 0 id it is missing)
lpfc_automap=2 (Automatically bind using WWPN)
c. I used automap because I couldn't seem to get persistent binding to work.
d. make build
e. make install
At this point lpfcdd.o should be in the kernel modules directory.
3. Update modules.conf
alias scsi_hostadapter lpfcdd
4. Rebuild initrd
/sbin/mkinitrd /boot/initrd-2.4.9-e.27enterprise-lpfc.img 2.4.9-e.27enterprise
(NOTE: use "uname -r" or "cat /proc/version" to determine the running kernel version)
5. Update grub.conf or lilo.conf. The boot loader needs to know about the new initrd file
and needs to tell the kernel that support for multiple luns is required.
From updated grub.conf
title Red Hat Linux (2.4.9-e.27enterprise-lpfc)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.27enterprise ro root=/dev/cciss/c0d0p3 max_scsi_luns=128
initrd /initrd-2.4.9-e.27enterprise-lpfc.img
6. Some utilities seem to be required from the EVA platform kit for linux so install the
platform kit next. The platform kit will install scripts in /etc/init.d to load qlogic drivers.
They should be removed or disabled.
7. Reboot.
Adding Disks:
Next create vdisks on the EVA and present them to the linux host. This assumes that the
host was physically attached to the Fibre Channel switch and that a host entry for the
linux host was already created in the EVA. The platform kit I got from HP states that the
OS specified during the host entry creation should be "Sun Solaris". Also the linux host
should have been zoned such that it sees only one path on the FS switch.
On the linux box verify that the lpfcdd module loaded "lsmod". The lpfcdd module
requires the sg scsi generic module, sd_mod and the scsi_mod.
The probe-luns command will scan for new luns. Run "probe-luns -a" and then "probe-
luns -l". The output will show the EVA controller and each presented lun. The luns will
be associated with sg and sd devices. This will be reflected in the messages file.
At this point I could sfdisk the new sd devices and then use mkfs to create filesystems.
Problems:
1. I had to create a startup file to scan for luns and mount filesystems, otherwise when the
system booted no luns were available. See attached listing for san-attach.sh. This was
resolved. I simply needed to load the lpfc module and probe for luns in the
/etc/rc.d/rc.sysinit file. The change was added just before filesystems are checked
and mounted. Also this was where the QLogic drivers were being loaded. The
rc.sysinit file is apparently modified by HP when the platform kit is installed. The
part where the QLogic drivers are loaded should be commented out or removed. By
loading the lpfc module here it is possible to put the SAN filesystem entries in the
fstab file where they belong.
2. Half of the vdisks I create on the EVA do not appear to be usable on the linux box.
They are detected by probe-luns but the OS can not read or modify their partition tables. I
presented some of the problem luns to a Sun box and they worked fine. Half of the luns
presented to the linux box worked fine.
3. Sometimes when I add a new lun I simply have to re-run probe-luns which isn't
disruptive to users on the system. Other times I have to reload the lpfcdd module which
requires that I un-mount all the SAN filesystems. At least I don't have to reboot.
Misc details:
The probe-luns program is from the HP platform kit which includes other useful utilities
in /opt/local/hp-fibreutils/.
The "inquire" command is convenient in determining if a lun is attached along with
additional info about the lun. The ûa switch will provide info about a disk named in the
format hba.target.lun. For example:
[root@boulder engle005]# inquire -a 0.0.1
scsidev@0.0.1:COMPAQ HSV110 (C)COMPAQ3000|Disk, /dev/sg1
S/N: P5849D4AAOQ088
WWNN:600508B400010CF50000900001C00000
[root@boulder engle005]# inquire -a 0.0.2
scsidev@0.0.2:COMPAQ HSV110 (C)COMPAQ3000|Disk, /dev/sg2
S/N: P5849D4AAOQ088
WWNN:600508B400010CF50000900001D50000
[root@boulder engle005]# inquire -a 0.0.3
[root@boulder engle005]# inquire -a 0.0.4
scsidev@0.0.4:COMPAQ HSV110 (C)COMPAQ3000|Disk, /dev/sg3
S/N: P5849D4AAOQ088
WWNN:600508B400010CF50000900001FB0000
The command "cat /proc/scsi/scsi also provides useful information.
bash-2.05$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 01
Vendor: COMPAQ Model: HSV110 (C)COMPAQ Rev: 3000
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 00 Lun: 02
Vendor: COMPAQ Model: HSV110 (C)COMPAQ Rev: 3000
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 00 Lun: 04
Vendor: COMPAQ Model: HSV110 (C)COMPAQ Rev: 3000
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: COMPAQ Model: HSV110 (C)COMPAQ Rev: 3000
Type: Unknown ANSI SCSI revision: 02
San-attach
#!/bin/bash
#
#
# Reload the driver
rmmod lpfcdd
insmod lpfcdd
#
#
# Probe for new luns.
/opt/local/hp_fibreutils/probe-luns -a
/opt/local/hp_fibreutils/probe-luns -l
#
#
# Mount the SAN filesystems.
mount /dev/sda1 /ora99-raid5
mount /dev/sdb1 /opt/apache
mount /dev/sdc1 /ora97-raid1
|
You are visitor number 4437