Benutzerspezifische Werkzeuge
Sie sind hier: Startseite Support Dokumentation docs Howto install SCST over Infiniband with SRP on Debian Sqeeze

Howto install SCST over Infiniband with SRP on Debian Sqeeze

Here you may learn about SCST over infinband, its installation and configuration. How to compile the modules for scst, and the SRP-Target ib_srpt. How to configure SCST with scstadmin. How to configure the initiator side with ib_srp and multipathing.

This Blog is the result of the ongoing research to build a high reliable storage stack based on DRBD, Infiniband utilizing the SRP (SCSI RMDA Protcol) of SCST. Research is needed since the documentation on this subject is sparse. I grabbed information from various mailing lists, web sites, blogs and did often wild guessing.

Be warned: This is work in progress.


A) Howto build SCST without infiniband (dead end for me but maybe useful for others)

B) Howto build SCST with infiniband (including the precious ib_srpt module)

A) Building SCST without infiniband

I ran into this dead end since the Packaging of the SCST-Sources is suboptimal. The SCST.tar.gz file that you find on the sourceforge site e.g. here does not include the SRPT-Module.  You may follow the link on the SRPT-Subpage to find that package. I did not and get stuck. Procede to B) if you like to build with infiniband.

1) You may use the prebuild debian packages from but they do not work for me.
2) Building by hand
Loading the minimal build dependencies:
#aptitude install build-essentials linux-headers checkinstall
#tar -xzf scstadmin-2.0.0.tar.gz
#cd scst-
#checkinstall --exclude=/lib/modules/2.6.32-5-amd64/modules.softdep,/lib/modules/2.6.32-5-amd64/modules.devname

3) Trying to load the modules according to the manual
Then run the following commands:
modprobe scst
modprobe scst_vdisk
modprobe iscsi-scst
scstadmin -config /etc/scst.conf
# modprobe scst
# modprobe scst_vdisk
# modprobe iscsi-scst
FATAL: Module iscsi_scst not found.
download iscsi-scst:
#tar -xzf iscsi-scst-2.0.0.tar.gz
#cd iscsi-scst-2.0.0/
#make all
#checkinstall --exclude=/lib/modules/2.6.32-5-amd64/modules.softdep,/lib/modules/2.6.32-5-amd64/modules.devname

#dpkg -i iscsi-scst_2.0.0-1_amd64.deb
# modprobe iscsi-scst
# iscsi-scstd
process ist started successfully
# scstadmin
-bash: scstadmin: command not found
download scstadm:
#tar -xzf  scstadmin-2.0.0.tar.gz
#cd  scstadmin-2.0.0
#checkinstall --exclude=/lib/modules/2.6.32-5-amd64/modules.softdep,/lib/modules/2.6.32-5-amd64/modules.devname

Collecting current configuration: done.
No valid operations specified.
SCST Configurator v2.0.0
General Operations
-config <config>        : Configure SCST given the specified <file>.
-check_config <file>    : Checks the saved configuration <file>.
-write_config <file>    : Writes the current configuration to <file>.
-clear_config           : Clear all SCST configuration.

A) Building SCST with infiniband

I was not able to compile the SRPT-Module from the srpt.2.0.0 sources. The top level makefile from the svn-trunk is not included so the build-sequence given by the README.txt included in the package does not work. To cope with that I downloaded the complete developement trunk via svn
svn co
cd trunk
The toplevel makefile has lots of build targets and building the scst-modules
make scst

and srpt modules

make srpt
is no problem.
To build debian packages for the modules go into the respective folders and do:
#cd scst
#checkinstall --exclude=/lib/modules/2.6.32-5-amd64/modules.softdep,/lib/modules/2.6.32-5-amd64/modules.devname
#cd ../srpt
#checkinstall --exclude=/lib/modules/2.6.32-5-amd64/modules.softdep,/lib/modules/2.6.32-5-amd64/modules.devname
I would have been so nice if the scstadmin folder behaves after this scheme, too - but it did not.
I wasn't able to checkinst the scstadmin package from within the directory, so I used the debian package I builded in Section A. Since scstadmin is a userspace tool the minor differences between the official release and the trunk should not lead to problems.

Checking the installation

At this point you should be able to do the following:
#modprobe scs
#modprobe scst_vdisk
#modprobe ib_srpt
without any errors.
Finishing the installation:
scst needs the directory /var/lib/scst/pr to work properly.
so we create it:
# mkdir -p /var/lib/scst/pr

since we did not have patched the kernel the scst_disk module refuses to load. To be able to use the /etc/init.d/scst init-script it is necessary to remove this module from the module list.

Also it may comes handy to include the ib_srpt module in its module list.

# Modules to load/unload.
# !!! YET!
SCST_MODULES="ib_srpt scst_vdisk"

But the comment warned to include target drivers at this point, so we have to delegate the loading of the ib_srpt module to a init-script that will be called later.

Configuration of the first target:

We create a LVM device for testing :

lvcreate -L50G --name scst_test vgroot

Registering the device using scstadmin:

scstadmin -open_dev scst_test_dev  -handler vdisk_blockio -attributes filename=/dev/vgroot/scst_test

Collecting current configuration: done.

-> Making requested changes.
-> Opening device 'scst_test_dev' using handler 'vdisk_blockio': done.
-> Done.

All done.

Meanwhile in the kernel:

[1362022.061873] [12639]: dev_vdisk: vdisk_report_registering:3013:Registering virtual vdisk_blockio device scst_test_dev (BLOCKIO)
[1362022.061882] [12639]: scst_suspend_activity:714:suspend_count 0
[1362022.061887] [12639]: scst_susp_wait:680:wait_event() returned 0
[1362022.061890] [12639]: scst_suspend_activity:759:Waiting for 0 active commands finally to complete
[1362022.061894] [12639]: scst_susp_wait:680:wait_event() returned 0
[1362022.062449] [12639]: dev_vdisk: vdisk_attach:772:Attached SCSI target virtual disk scst_test_dev (file="/dev/vgroot/scst_test", fs=51200MB, bs=512, nblocks=104857600, cyln=51200)
[1362022.062473] [12639]: __scst_resume_activity:790:suspend_count 0 left
[1362022.062493] [12639]: scst: scst_register_virtual_device:1178:Attached to virtual device scst_test_dev (id 6)

Checking with scstadmin for the device:

# scstadmin -list_device

Collecting current configuration: done.

Handler           Device
vdisk_nullio     -
vdisk_fileio     -
vdisk_blockio    scst_test_dev
vcdrom           -

All done.

Checking for targets:

#scstadmin  -list_target
Collecting current configuration: done.

Driver  Target
ib_srpt ib_srpt_target_0

All done.

Defining a security group:

#scstadmin  -add_group HOST01 -driver ib_srpt -target ib_srpt_target_0
Collecting current configuration: done.
-> Making requested changes.
-> Adding new group 'HOST01' to driver/target 'ib_srpt/ib_srpt_target_0': done.
-> Done.                                                                       

All done.

Verifying the group:

#scstadmin  -list_group
Collecting current configuration: done.
Driver: ib_srpt
Target: ib_srpt_target_0
Driver/target 'ib_srpt/ib_srpt_target_0' has no associated LUNs.
Group: HOST01
Assigned LUNs:
LUN  Device
0    scst_test_dev
Assigned Initiators:
All done.

Assigning a LUN to our target:



scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group HOST01 -device scst_test_dev                                                                                          
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'scst_test_dev' at LUN 0 to driver/target/group 'ib_srpt/ib_srpt_target_0/HOST01': done.
-> Done.                                                                                                
All done.