SCSI RDMA Protocol

LIO Target
	; Fibre Channel fabric module(s)
Original author(s)	Vu Pham; Bart Van Assche; Nicholas Bellinger
Developer(s)	Mellanox Technologies, Ltd.
Initial release	February 15, 2011
Preview release	4.1.0-rc1 / February 21, 2011; ; 14 months ago
Development status	Production
Written in	C
Operating system	Linux
Type	Fabric module
License	GNU General Public License
Website	www.mellanox.com

From Linux-iSCSI

(Redirected from SRP)

Jump to: navigation, search

See Target for a complete overview over all fabric modules.

An SCSI RDMA Protocol (SRP, aka SCSI Remote Protocol) fabric module (ib_srpt) for Mellanox HCAs is supported. The SCSI RDMA Protocol defines a SCSI mapping onto the Infiniband architecture and/or functionally similar cluster protocols.

The InfiniBand/SRP fabric module went upstream into the Linux 3.3 kernel on 1/18/2012.^[1]

General

The use of RDMA generally allows higher throughput and lower latency than TCP/IP based communication. RDMA is only possible with network adapters that support RDMA in hardware. Examples of such network adapters are InfiniBand HCAs and 10 GbE network adapters with iWARP support. While the SRP protocol has been designed to use RDMA networks efficiently, it is also possible to implement the SRP protocol over networks that do not support RDMA.

SRP is easier to implement than iSER, but iSER offers more management functionality, e.g. the target discovery infrastructure enabled by the iSCSI protocol.

Setup

targetcli from RisingTide Systems is a comprehensive, powerful, easy storage management tool that can efficiently handle complex storage installations.

This code won't be included in Linux before kernel v2.6.40. However using Linux >= v2.6.38, this and other HW Target mode modules will be able to function.

Specifications

SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:

SCSI RDMA Protocol (SRP): SRP defines a SCSI protocol mapping onto the InfiniBand (tm) Architecture and/or functionally similar cluster protocols. ANSI INCITS 365-2002. Status: Final Draft. 7/3/2002

Getting the sources

Initial build: git setup

git clone lio-core from git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git as follows:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git
cd lio-core-2.6
git checkout --track -b tcm_ib_srpt-38 origin/tcm_ib_srpt-38
git pull origin tcm_ib_srpt-38
cp <your_working_config_file> .config
make menuconfig

In menuconfig:

Select device drivers
Select target core:

<M> Generic Target Core Mod (TCM) and ConfigFS Infrastructure  --->

Select the modules FILEIO, IBLOCK, PSCSI backend drivers, and iSCSI target module.

            --- Generic Target Core Mod (TCM) and ConfigFS Infrastructure     
                   <M>   TCM/IBLOCK Subsystem Plugin for Linux/BLOCK              
                   <M>   TCM/FILEIO Subsystem Plugin for Linux/VFS           
                   <M>   TCM/pSCSI Subsystem Plugin for Linux/SCSI                                        
                   < >   TCM/STGT Subsystem Plugin                             
                   <M>   Linux-iSCSI.org iSCSI Target Mode Stack
                   [ ]   LIO-Target iscsi_debug.h ring buffer messages
                   [ ]   LIO-Target ErrorRecoveryLevel debug code

Select the Infiniband stack:

<M> InfiniBand support  --->

) Select the necessary IB stack support, HCA specific HW drivers and SRP Target ULP driver:

                 --- InfiniBand support                                                                   
                 <M>   InfiniBand userspace MAD support                                                  
                 <M>   InfiniBand userspace access (verbs and CM)                                       
                 <M>   Mellanox HCA support                                                             
                 < >   QLogic PCIe HCA support                                                          
                 < >   Ammasso 1100 HCA support                                                          
                 <M>   Mellanox ConnectX HCA support                                                    
                 < >   NetEffect RNIC Driver                                                          
                 <M>   IP-over-InfiniBand                                                                
                 [ ]     IP-over-InfiniBand Connected Mode support                                      
                 -*-     IP-over-InfiniBand debugging                                                    
                 [ ]       IP-over-InfiniBand data path debugging                                        
                 < >   InfiniBand SCSI RDMA Protocol                                                    
                 <M>   InfiniBand SCSI RDMA Protocol target support                                      
                 < >   iSCSI Extensions for RDMA (iSER)

Save and return

If CONFIGFS_FS=y is set, a manual mount is required after the kernel boots:

mount -t configfs configfs /sys/kernel/config

Subsequent builds: git pull

Go to the directory where your lio-core kernel is located, and:

cd <lio-core-2.6>
git pull origin tcm_ib_srpt-38
make clean
make oldconfig

Building

Kernel and modules:

make
make modules
make modules_install
make install

Installing

New Linux kernel

Use lio-core-2.6 kernel:

Change /boot/grub/menu.lst to select your lio-core-2.6 kernel.

Then reboot:

reboot

and verify that the live kernel is indeed your new lio-core-2.6.git/tcm_ib_srpt-38 kernel:

uname -a

Loading LIO Target

Verify the presence of the following required modules with modprobe: ib_srpt, target_core_mod, configfs, e.g.:

modprobe ib_srpt srp_max_req_size=4200

Normally all these modules should be loaded. If one is missing, you can load it as follows:

modprobe configfs
# Or do the mount command as shown above (if .config file has CONFIGFS_FS=y)
modprobe target_core_mod
modprobe ib_srpt srp_max_req_size=4200

Starting LIO Target

The target will be automatically started if targetcli-frozen is installed. Otherwise, start the target manually:

/etc/init.d/target start

To stop target:

/etc/init.d/target stop

To check the target status:

/etc/init.d/target status

User-space tools

targetcli requires the libreadline6 package to be installed. The links for these rpm packages are:

Configuring

Please see SCSI_RDMA_Protocol/targetcli. The basic steps are as follows:

# targetcli
  ...
/> create hba fileio
  ...
/> create storage <mylun_name> /tmp/<myfile_name> <size>

Where, e.g.:

size = 10G
mylun_name could be my_lun
myfile_name could be my_file
This creates a storage object my_lun and a 10 GB FILEIO backstore in /tmp/my_file.

Once the explict Node ACL and MappedLUN have been for the initiator configured, it's time to login from the initiator side.

This includes using ibsrpdm -c to obtain the necessary string:

truelife:/usr/src/lio-core-2.6.git# ibsrpdm -c
id_ext=0002c903000e8acc,ioc_guid=0002c903000e8acc,dgid=fe800000000000000002c903000e8acd,pkey=ffff,service_id=0002c903000e8acc

And then echoing this into the sysfs add_target attribute on the SRP Initiator to perform the SRP_LOGIN:

truelife:/usr/src/lio-core-2.6.git# echo -n id_ext=0002c903000e8acc,ioc_guid=0002c903000e8acc,dgid=fe800000000000000002c903000e8acd,pkey=ffff,service_id=0002c903000e8acc > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target

Notes:

By default, FILEIO uses O_SYNC. It allows both, exporting a shared filesystem for mounting, and creating files in the shared filesystem to be used as the backstore for raw block devices.
Any number of virtual HBAs and assoicated devices is possible. The Target subsystem plugins (FILEIO, IBLOCK, pSCSI, RAMDISK, etc.) will report the underlying HW limitiations for things like TCQ depth, MaxSectors, TaskAbortedStatus, UA Interlocking, etc. All of these values are available as attributes in the targetcli device context:

# targetcli
  ...
/> fileio0/my_file(/tmp/my_file)> ls attribute
  ...

Results

configfs

Please see SCSI RDMA Protocol/configFS for the running layout of a /sys/kernel/config/target/srpt configuration

lsmod output

Module                  Size  Used by
ib_srpt                26028  6 
ib_umad                 9422  8 
iscsi_target_mod      201391  1 
mlx4_ib                38241  0 
target_core_pscsi      11045  0 
target_core_file        6307  1 
target_core_iblock      6606  1 
target_core_mod       206442  28 ib_srpt,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
configfs               19153  2 target_core_mod
ib_iser                23172  0 
rdma_cm                23833  1 ib_iser
ib_cm                  24878  2 ib_srpt,rdma_cm
iw_cm                   6019  1 rdma_cm
ib_sa                  15537  2 rdma_cm,ib_cm
ib_mad                 30330  5 ib_srpt,ib_umad,mlx4_ib,ib_cm,ib_sa
ib_core                39716  9 ib_srpt,ib_umad,mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
ib_addr                 3982  1 rdma_cm
ipv6                  242427  21 ib_addr
iscsi_tcp               7729  0 
libiscsi_tcp           10485  1 iscsi_tcp
libiscsi               29466  3 ib_iser,iscsi_tcp,libiscsi_tcp
scsi_transport_iscsi    24624  4 ib_iser,iscsi_tcp,libiscsi
dm_multipath           14016  0 
scsi_dh                 4656  1 dm_multipath
</code>

=== Interrupt stats ===

MSI-X interrupt stats for target side operation:

<pre>
lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | grep mlx4
43:   16864937   16864950   16864795   16864897   PCI-MSI-edge      mlx4-comp-0@pci:0000:00:05.0
44:          0          0          0          0   PCI-MSI-edge      mlx4-comp-1@pci:0000:00:05.0
45:          0          0          0          0   PCI-MSI-edge      mlx4-comp-2@pci:0000:00:05.0
46:          0          0          0          0   PCI-MSI-edge      mlx4-comp-3@pci:0000:00:05.0
47:       1377       1308       1338       1322   PCI-MSI-edge      mlx4-async@pci:0000:00:05.0

Initiator

Kernel ring buffer

Initiator side ring buffer output during initiator login:

scsi7 : SRP.T10:0002C903000E8ACC
scsi 7:0:0:0: Direct-Access     LIO-ORG  RAMDISK-MCP      4.0  PQ: 0 ANSI: 5
sd 7:0:0:0: Attached scsi generic sg2 type 0
sd 7:0:0:0: [sdb] 524288 512-byte logical blocks: (268 MB/256 MiB)
sd 7:0:0:0: [sdb] Write Protect is off
sd 7:0:0:0: [sdb] Mode Sense: 2f 00 00 00
sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 7:0:0:0: [sdb] Attached SCSI disk

Device identifiers

sg_inq -i initiator side output for EVPD=0x83 device identifiers:

truelife:/usr/src/lio-core-2.6.git# sg_inq -i /dev/sdb
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    designator_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x1405
      Vendor Specific Identifier: 0x929cc061d
      Vendor Specific Identifier Extension: 0xe082d4d98d810cdc
      [0x6001405929cc061de082d4d98d810cdc]
  Designation descriptor number 2, descriptor length: 61
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: LIO-ORG
      vendor specific: RAMDISK-MCP:929cc061-e082-4d98-810c-c90b421bd066
  Designation descriptor number 3, descriptor length: 8
    transport: Remote Direct Memory Access (RDMA)
    designator_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x2
  Designation descriptor number 4, descriptor length: 8
    transport: Remote Direct Memory Access (RDMA)
    designator_type: Target port group,  code_set: Binary
    associated with the target port
      Target port group: 0x0
  Designation descriptor number 5, descriptor length: 8
    designator_type: Logical unit group,  code_set: Binary
    associated with the addressed logical unit
      Logical unit group: 0x0
  Designation descriptor number 6, descriptor length: 32
    transport: Remote Direct Memory Access (RDMA)
    designator_type: SCSI name string,  code_set: UTF-8
    associated with the target port
      SCSI name string:
      0x2c903000e8acc,t,0x0001

ibsrpdm output

truelife:/usr/src/lio-core-2.6.git# ibsrpdm 
IO Unit Info:
    port LID:        0002
    port GID:        fe800000000000000002c903000e8acd
    change ID:       0001
    max controllers: 0x10
    controller[  1]
        GUID:      0002c903000e8acc
        vendor ID: 000002
        device ID: 00673c
        IO class : 0100
        ID:        Linux SRP target
        service entries: 1
        service[  0]: 0002c903000e8acc / SRP.T10:0002c903000e8acc

Timeline

History of LIO Target
Release	Details	2008												2009												2010												2011												2012
Release	Details	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12
2.x	Version	2.9
2.x	Feature	GPL release
3.x	Version								3.0													3.1		3.2						3.4					3.5
3.x	Feature								ConfigFS control plane, tcm_loop													PR ALUA		Cleanups						Open-FCoE					Backports
4.x	Version																																				4.0
	Feature																																				Target Core			tcm_loop			FCoE			iSCSI			Perf.		IB SRP		CMWQ			FC vSCSI
	Linux																																				2.6.38			2.6.39			3.0			3.1			3.2		3.3		3.4			3.5

Notes

↑ Linus Torvalds (1/18/2012). "InfiniBand/SRP merge". lkml.org.

External links

linux-rdma mailing list
SRP Wikipedia entry
InfiniBand Wikipedia entry
Mellanox website
T10 home page

[0] Linus Torvalds (1/18/2012). "InfiniBand/SRP merge". lkml.org.

[1]

SCSI RDMA Protocol

From Linux-iSCSI

Contents

General

Setup

Specifications

Getting the sources

Initial build: git setup

Subsequent builds: git pull

Building

Installing

New Linux kernel

Loading LIO Target

Starting LIO Target

User-space tools

Configuring

Results

configfs

lsmod output

Initiator

Kernel ring buffer

Device identifiers

ibsrpdm output

Timeline

See also

Notes

External links

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Toolbox

Google AdSense


Fibre Channel fabric module(s)
Original author(s)	Vu Pham Bart Van Assche Nicholas Bellinger
Developer(s)	Mellanox Technologies, Ltd.
Initial release	February 15, 2011 (2011-02-15)
Preview release	4.1.0-rc1 / February 21, 2011; 14 months ago (2011-02-21)
Development status	Production
Written in	C
Operating system	Linux
Type	Fabric module
License	GNU General Public License
Website	www.mellanox.com