SCSI Target Subsystem
From Linux-iSCSI
The generic SCSI Target Subsystem for Linux (SCST) allows creation of sophisticated storage devices from any Linux box. We have received a significant number of technical questions regarding SCST, and thought a brief discussion might be helpful.
Comparison
We keep getting a number of questions regarding the SCST feature comparison table, and decided to provide a discussion of the pertinent points here, in the hope that this might be helpful.
The remainder of this section is organized according to the list of features that the SCST teams claims are supported by SCST and unsupported by LIO.
Automatic sessions reassignment
SCST describes this feature as "changes in the access control are immediatelly seen by initiators."
LIO can change Mapped LUNs on the fly. It looks like SCST also supports the REPORT_LUNS_DATA_HAS_CHANGED unit attention condition, which provides enhanced support for this feature, but isn't required for it. In any event, REPORT_LUNS_DATA_HAS_CHANGED should probably be added to LIO at some point.
Asynchronous Event Notifications (AEN)
LIO uses AENs in fabric module code for signaling target side connection failure with ERL=2 in iSCSI. SCSI level AENs, however, have been deprecated in the T10 SCSI-3 standard, and we thus don't plan to add further AEN support outside of fabric module dependent code.
Bidirectional commands
Proper BIDI commands are supported by TCM v4 and enabled in tcm_loop. They are not yet generally enabled in lio-target.ko.
Extended CDB (size >16 bytes)
Proper extended CDB support has been added for TCM v4 and enabled tcm_loop. It has not yet been generally enabled in lio-target.ko.
Transactional Persistent Reservation data
SCST further describes this as "durable, i.e. transactional, save of [persistance] through power loss persistent reservation data."
LIO supports this functionlity: it saves all PR APTPL metadata into a local file, and automatically rebuilds PR APTPL metadata state based on active configFS groups.
ALUA
SCST states that "all ALUA [state] [transitions] [are] implemented as empty placeholders, which means this implementation can't be used to build ALUA-based clusters without additional code."
First, target_core_alua.c clearly doesn't only contain "empty placeholders." Second, the LIO ALUA implementation fully supports ALUA LUN and Target Port Groups abstractions, including full support for explict ALUA (via SET_TARGET_PORT_GROUPS) and implict ALUA (via configfs operations). Third, at the very least, this implementation is sufficient to support VMware vSphere certification (which SCST can't support).
I/O context grouping between I/O threads
SCST claims that this is "a big performance win with CFQ."
LIO implements a threading model that entails a control flow from multi-threaded fabric to a single-thread per backend device across virtual HBAs. So architecturally, this is not an issue with the LIO stack.
Having multiple backend threads might still be a design consideration, however. With our performance measurements, we were not able to validate the SCST claim of a big performance win with thread grouping by I/O context. In our tests, LIO was able to exceed 100 Gb/sec with tcm_loop to/from local CPU cache memory, which seems to imply ample headroom.
Protection against invalid CDBs
SCST further describes this as "commands with [the] wrong transfer size or transfer direction (may lead to crash or hard lockup of the target)."
LIO has per CDB type sanity checks for incoming SCSI descriptors from a fabric, otherwise it would allow trivial DOS attacks.
Protection against memory overallocation
SCST further describes this as "crashing [the] target by making it to allocate too much memory for buffers and go into OOM state."
LIO, in contrast to SCST, is splitting received CDB across the limitiations for max_sectors of the backend device, so it has no hard limits regarding the CDB size. So architecturally, LIO doesn't need additional provisions to protect against this scenario (unlike SCST).
Latency measurements
LIO does in fact not yet have sophisticated latency measurement instrumentation. For the Linux kernel 2.6.39 target, we are planning to enable the modern advanced kernel mainline tracing functionality for LIO, and thus use the comprehensive Linux profiling infrastructure, including, e.g., for latency measurement instrumentation.
SCST, in contrast, is using its own non-standard profiling code (to support older kernel versions) which arguably is obsolete with modern Linux kernels.
Configuration tool
SCST further describes this as a "configuration tool with [the] ability to automatically apply changes in the config file on [the] fly without any restarts."
LIO obviously supports this with its high level targetcli management tool.
Support for limiting number of initiators
SCST further describes this as "support for limiting [the] number of initiators [that are] allowed to connect to a target"
This functionality is properly done using explict NodeACLs and MappedLUNs, which is functionality that is generic to all v4 fabric modules using target_core_fabric_configfs.c logic. Claiming that LIO doesn't support this standard functionality is misleading.
Support for AHS
LIO supports this feature in general in the target (with tcm_loop), but in fact hasn't enabled it in lio-target.ko yet.
Support for iSCSI redirects
LIO indeed doesn't yet support this functionality in lio-target.ko. However, we have not been able to find convincing use case for it. If we find one, iSCSI redirects will be pretty easy to add.
No support for iSNS
LIO supports iSNS. Also, iSNS is a userspace demon, so it shouldn't be part of a kernel feature comparison list.
Safe implementation of Task Management commands
LIO properly supports LUN_RESET with TAS=[1,0] (Task Aborted Status) emulation, and is used for SPC-3 PR-OUT PREEMPT_AND_ABORT. LIO currently doesn't support emulation for ABORT_TASK, ABORT_TASK_SET or CLEAR_ACA, because ACA is not supported in Linux/SCSI proper, so there is (currently) no easy way in mainline to test these.
LIO is planning to add more extensive emulated support for the non ACA related TMRs (Task Management Requests/Responses).
Multithreaded digest calculation
SCST describes this as "each connection multithreaded digest calculation."
LIO supports this functionality, and also supports it with modern SSE-4.2 CRC32C instructions. In fact, LIO supported it before SCST, and the SCST implementation shows many similarities to the one in LIO.
Contributing
SCST also provides a list of feature requests, for which it is soliciting community contributions. It is interesting to analyze and briefly discuss some of these SCST feature requests, as well.
Dynamic I/O flow control
We interpret this as meaning changing the TCQ depth of the device (or a specific Initiator Port). LIO can technically do things like this, however, the current code is preventing this from occuring with active target ports. We are planning to add this as a LIO v4.1 / Linux v2.6.39 feature, but feel that it is lower priority for now.
Solve SG IO count limitation
SCST describes this further as "solve [the] SG IO count limitation issue in pass-through mode (splitting of large commands)."
LIO already implements this functionality (as explained above) for all backend devices regardless of the max_sectors limitiations. This logic is transparent to all backend subsystem drivers (pSCSI, IBLOCK, FILEIO).
Usage with non-SCSI transports (e.g., AoE)
The LIO IBLOCK and FILEIO subsystem backend drivers for v4 are already SCSI independent (with help from the kernel community). The LIO team hasn't hooked up a non SCSI fabric module to this infrastructure just yet.
GET_CONFIGURATION command
This is CDB emulation for CD/DVD ROM devices. The better design and would appear to be pushing this up to userspace for complex non TYPE_DISK devices. LIO plans an implementation along these lines.
Per-device suspending
SCST further elaborates that "currently before doing any management operations[, the] SCST core performs so called 'activities suspending', i.e. it suspends new coming SCSI commands and wait[s] until currently being executed ones finished. It allows to simplify internal locking and reference counting a lot, but has a drawback that it is global, i.e. affects all devices and SCSI commands, even ones which don't participate in the management operation."
This sounds like SCST is using a global-locking structure, which usually isn't exactly the hallmark of a good design.
LIO implements a per-device lock granularity. In addition, the LIO team has been working with IBM and Intel Labs to continuosly improve the lock structure in the larger Linux storage subsystem, in order to make sure it can deliver the performance that modern backstore devices and LIO require and/or deliver.
See also
External links
- SCST Comparison table
