Fibre Channel over Ethernet
The Linux SCSI Target Wiki
![]() | |
---|---|
Cisco Systems FCoE fabric module | |
Original author(s) |
Joe Eykholt Kiran Patil |
Developer(s) | Cisco Systems, Inc. |
Initial release | July 21, 2011 |
Stable release | 4.1.0 / June 20, 2012 |
Preview release | 4.2.0-rc5 / June 28, 2012 |
Development status | Production |
Written in | C |
Operating system | Linux |
Type | Fabric module |
License | GNU General Public License, version 2 (GPLv2) |
Website | www.cisco.com |
- See LIO for a complete overview over all fabric modules.
Fibre Channel over Ethernet (FCoE) provides FCoE SAN connectivity over Ethernet networks.
Contents |
Overview
The FCoE fabric module provides Fibre Channel (FC) access over Ethernet. It is based on a fabric module from open-fcoe.org. The Open-FCoE target code was adapted to use a modified upstream Linux libfc module. This includes a patch series to re-enable point-to-point module for libfc (merged upstream), and a set of hooks into libfc to be able to hand off incoming FLOGI
requests to Fibre Channel based LinuxIO modules.
Hardware support
The following Ethernet NICs are supported:
- 1 GbE: all usual on-board or PCI NICs are supported.
- 10 GbE: the Intel Ethernet X520 Adapters are recommended.
The fabric module (tcm_fc.ko, Linux kernel driver database) for LIO was released with Linux kernel 3.0 on July 21, 2011 .[1]
targetcli
targetcli from Datera, Inc. is used to configure FCoE targets. targetcli aggregates service modules via a core library, and exports them through an API to the LIO, to provide a unified single-node SAN configuration shell, independently of the underlying fabric(s).
![]() |
LIO Admin Manual The LIO Admin Manual provides comprehensive background and many examples on using targetcli and on programming the Datera library. |
Cheat sheet
Command | Comment |
---|---|
/backstores/iblock create my_disk /dev/sdb | Create the LUN my_disk on the block device /dev/sdb |
/tcm_fc create <WWPN> | Create a FCoE target |
In /tcm_fc/<WWPN>: luns/ create /backstores/iblock/my_disk | Export the LUN my_disk |
In /tcm_fc/<WWPN>: acls/ create <Initiator WWPN> | Allow access for the initiator at <WWPN> |
/saveconfig | Commit the configuration |
Startup
targetcli is invoked by running targetcli as root from the command prompt of the underlying LIO shell.
# targetcli Welcome to targetcli: Copyright (c) 2014 by Datera, Inc. All rights reserved. Visit us at http://www.datera.io. Using ib_srpt fabric module. Using qla2xxx fabric module. Using iscsi fabric module. Using loopback fabric module. Using tcm_fc fabric module. /> tcm_fc/ info Fabric module name: tcm_fc ConfigFS path: /sys/kernel/config/target/fc Allowed WWN list type: free sysFabric module specfile: /var/target/fabric/tcm_fc.spec Fabric module features: acls Corresponding kernel module: tcm_fc />
Upon targetcli initialization, the underlying RTSlib loads the installed fabric modules, and creates the corresponding ConfigFS mount points (at /sys/kernel/config/target/<fabric>), as specified by the associated spec files (located in /var/target/fabric/fabric.spec).
Display the object tree
Use ls to list the object hierarchy, which is initially empty:
/> ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- fileio ............................................... [0 Storage Object] | o- iblock ............................................... [0 Storage Object] | o- pscsi ................................................ [0 Storage Object] | o- rd_dr ................................................ [0 Storage Object] | o- rd_mcp ............................................... [0 Storage Object] o- ib_srpt ........................................................ [0 Target] o- iscsi .......................................................... [0 Target] o- loopback ....................................................... [0 Target] o- qla2xxx ........................................................ [0 Target] />
![]() |
Global parameter auto_cd_after_create After the creation of a new object, automatically enter its object context. |
Per default, auto_cd_after_create is set to true, which automatically enters an object context (or working directory) after its creation. The examples here are modeled after this behavior.
Optionally, set auto_cd_after_create=false to prevent targetcli from automatically entering new object contexts after their creation:
/> set global auto_cd_after_create=false Parameter auto_cd_after_create is now 'false'. />
Create a backstore
Create a backstore using the IBLOCK or FILEIO type devices.
For instance, enter the top-level backstore context and create an IBLOCK backstore from a /dev/sdb block device:
/> cd backstores/ /backstores> iblock/ create name=my_disk dev=/dev/sdb Generating a wwn serial. Created iblock storage object my_disk using /dev/sdb. Entering new node /backstores/iblock/my_disk. /backstores/iblock/my_disk>
targetcli automatically creates a WWN serial ID for the backstore device and then changes the working context to it.
The resulting object hierarchy looks as follows (displayed from the root object):
/> ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- fileio ............................................... [0 Storage Object] | o- iblock ............................................... [1 Storage Object] | | o- my_disk .......................................... [/dev/sdb activated] | o- pscsi ................................................ [0 Storage Object] | o- rd_dr ................................................ [0 Storage Object] | o- rd_mcp ............................................... [0 Storage Object] o- ib_srpt ........................................................ [0 Target] o- iscsi .......................................................... [0 Target] o- loopback ....................................................... [0 Target] o- qla2xxx ........................................................ [0 Target] />
Alternatively, any LVM logical volume can be used as a backstore, please refer to the LIO Admin Manual on how to create them properly.
For instance, create an IBLOCK backstore on a logical volume (under /dev/<volume_group_name>/<logical_volume_name>):
/backstores> iblock/ create name=my_disk dev=/dev/vg0/lv1 Generating a wwn serial. Created iblock storage object my_disk using /dev/vg0/lv1. Entering new node /backstores/iblock/my_disk. /backstores/iblock/my_disk>
Again, targetcli automatically creates a WWN serial ID for the backstore device and then changes the working context to it.
![]() |
More backstore examples More examples on creating backstores can be found in targetcli. |
Instantiate a target
Associate the three Ethernet network interfaces that are available on the storage array with FCoE via:
echo ethX > /sys/module/fcoe/parameters/create
The three then resulting FCoE ports are presented in the WWN context with the following WWNPs:
- 20:00:00:0c:fc:00:f5:7b
- 20:00:00:0c:fc:01:0c:0f
- 20:00:00:15:17:af:5b:30
Instantiate an FCoE target on the existing IBLOCK backstore device my_disk (as set up in targetcli):
/backstores/iblock/my_disk> /tcm_fc create 20:00:00:0c:fc:00:f5:7b Created target 20:00:00:0c:fc:00:f5:7b. Entering new node /tcm_fc/20:00:00:0c:fc:00:f5:7b. /tcm_fc/21:0...0c:fc:00:f5:7b>
targetcli automatically changes the working context to the resulting Endpoint.
Export LUNs
Declare a LUN for the backstore device, to form a valid SAN storage object:
/tcm_fc/21:0...0c:fc:00:f5:7b> luns/ create /backstores/iblock/my_disk Selected LUN 0. Successfully created LUN 0. Entering new node /tcm_fc/20:00:00:0c:fc:00:f5:7b/luns/lun0. /tcm_fc/21:0...5:7b/luns/lun0>
targetcli automatically assigns the default ID '0' to the LUN, and then changes the working context to the SAN storage object. The LIO is now created, and exports /dev/sdb as LUN 0.
Return to the underlying Endpoint as the working context, as no attributes need to be set or modified for standard LUNs:
/tcm_fc/21:0...5:7b/luns/lun0> cd < Taking you back to /tcm_fc/20:00:00:0c:fc:00:f5:7b. /tcm_fc/21:0...0c:fc:00:f5:7b>
Define access rights
Configure the access rights to allow logins from initiators. This requires setting up individual access rights for each initiators, based on its WWPN.
Determine the WWPN for the respective FCoE initiator. For instance, for Linux initiator systems, use:
# cat /sys/class/fc_host/host*/port_name | sed -e s/0x// -e 's/../&:/g' -e s/:$//
For a simple setup, allow access to the initiator with the WWPN as determined above:
/tcm_fc/21:0...0c:fc:00:f5:7b> acls/ create 21:00:00:24:ff:31:4c:4c Successfully created Node ACL for 21:00:00:24:ff:31:4c:4c. Created mapped LUN 0. Entering new node /tcm_fc/20:00:00:0c:fc:00:f5:7b/acls/21:00:00:24:ff:31:4c:4c. /tcm_fc/21:0...24:ff:31:4c:4c> cd / />
targetcli then automatically adds the appropriate mapped LUNs per default.
Display the object tree
The resulting FCoE SAN object hierarchy looks as follows (displayed from the root object):
/> ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- fileio ............................................... [0 Storage Object] | o- iblock ............................................... [1 Storage Object] | | o- my_disk .......................................... [/dev/sdb activated] | o- pscsi ................................................ [0 Storage Object] | o- rd_dr ................................................ [0 Storage Object] | o- rd_mcp ............................................... [0 Storage Object] o- ib_srpt ........................................................ [0 Target] o- iscsi .......................................................... [0 Target] o- loopback ....................................................... [0 Target] o- qla2xxx ........................................................ [0 Target] o- tcm_fc ......................................................... [1 Target] o- 20:00:00:0c:fc:00:f5:7b ....................................... [enabled] o- acls .......................................................... [1 ACL] | o- 21:00:00:24:ff:31:4c:4c .............................. [1 Mapped LUN] | o- mapped_lun0 ........................................... [lun0 (rw)] o- luns .......................................................... [1 LUN] o- lun0 .................................... [iblock/my_disk (/dev/sdb)] />
Persist the configuration
![]() |
Don't forget to use saveconfig! Without saveconfig, the LIO configuration will be lost upon rebooting or unloading the target service, as the target configuration will revert back to the last saved one. |
Use saveconfig from the root context to persist the target configuration across LIO reboots:
/> saveconfig WARNING: Saving rtsnode1 current configuration to disk will overwrite your boot settings. The current target configuration will become the default boot config. Are you sure? Type 'yes': yes Making backup of srpt/ConfigFS with timestamp: 2012-02-27_23:19:37.660264 Successfully updated default config /etc/target/srpt_start.sh Making backup of qla2xxx/ConfigFS with timestamp: 2012-02-27_23:19:37.660264 Successfully updated default config /etc/target/qla2xxx_start.sh Making backup of loopback/ConfigFS with timestamp: 2012-02-27_23:19:37.660264 Successfully updated default config /etc/target/loopback_start.sh Making backup of LIO-Target/ConfigFS with timestamp: 2012-02-27_23:19:37.660264 Successfully updated default config /etc/target/lio_backup-2012-02-27_23:19:37.660264.sh Making backup of Target_Core_Mod/ConfigFS with timestamp: 2012-02-27_23:19:37.660264 Successfully updated default config /etc/target/tcm_backup-2012-02-27_23:19:37.660264.sh Generated Target_Core_Mod config: /etc/target/backup/tcm_backup-2012-02-27_23:19:37.660264.sh Successfully updated default config /etc/target/lio_start.sh Successfully updated default config /etc/target/tcm_start.sh />
Spec file
Datera spec files define the fabric-dependent feature set, capabilities and available target ports of the specific underlying fabric.
In particular, the FCoE spec file /var/target/fabric/tcm_fc.spec is included via RTSlib. WWN are extracted via /sys/class/fc_host/host*/port_name in wwn_from_files_filter, once the Ethernet device has been created with fcoe.ko, and are presented in the targetcli WWN working context to register individual FCoE port GUIDs.
# WARNING: This is a draft specfile supplied for demo purposes only. # The tcm_fc fabric module uses the default feature set. features = acls # Non-standard module naming scheme kernel_module = tcm_fc # The module uses hardware addresses from there wwn_from_files = /sys/class/fc_host/host*/port_name # Transform '0x1234567812345678' WWN notation to '12:34:56:78:12:34:56:78' wwn_from_files_filter = "sed -e s/0x// -e 's/../&:/g' -e s/:$//" # The configfs group is default configfs_group = fc
Specifications
The full advanced LIO SPC-3/SPC-4 SCSI logic (such as PRs, ALUA, VAAI, etc.) is inherently working on FCoE Ports and (across) all other fabric modules, such as FCoE, Fibre Channel, iSCSI, iSER, vHost, etc.
The following specifications are available from the T11 Working Group:
- Fibre Channel - Backbone (FC-BB): to develop the mappings necessary to bridge between physically-separate instances of the same network definition, including MAC address mapping & translation, configuration discovery, management facilities and mappings of FC Service definitions. Candidate network definitions are all flavors of Ethernet (incl. Gigabit) & FDDI, but will be a function of resources provided. Murali Rajagopal (edt.), Status: Published, 8/1/1996
Acknowledgements
FCoE was made possible by Joe Eykholt's generous contributions to tcm_fc
, and his ongoing work of making the Linux libfc
target capable.
See also
- LIO
- LinuxIO, targetcli
- Fibre Channel, iSCSI, iSER, SRP, tcm_loop, vHost
- ConfigFS (low level kernel API)
Notes
- ↑ Linus Torvalds (2011-07-21). "Linux 3.0". lkml.org.
External links
- LIO Admin Manual
- RTSlib Reference Guide [HTML][PDF]
- FibreChannel (FC) Wikipedia entry
- FibreChannel over Ethernet (FCoE) Wikipedia entry
- Open-FCoE
Timeline of the LinuxIO | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Release | Details | 2011 | 2012 | 2013 | 2014 | 2015 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | ||
4.x | Version | 4.0 | 4.1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Feature | LIO Core | Loop back | FCoE | iSCSI | Perf | SRP | CM WQ | FC USB 1394 | vHost | Perf | Misc | 16 GFC | iSER | Misc | VAAI | Misc | DIF Core NPIV | DIF iSER | DIF FC vhost | TCMU Xen | Misc | Misc | virtio 1.0 | Misc | NVMe OF | ||||||||||||||||||||||||||||||||||||
Linux | 2.6.38 | 2.6.39 | 3.0 | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | 3.7 | 3.8 | 3.9 | 3.10 | 3.11 | 3.12 | 3.13 | 3.14 | 3.15 | 3.16 | 3.17 | 3.18 | 3.19 | 3.20 | 3.21 | 3.22 |