Kernel-based Virtual Machine

The Linux SCSI Target Wiki

(Redirected from KVM)
Jump to: navigation, search
KVM
Screenshot
Screenshot of qemu/KVM running NetBSD, OpenSolaris and Kubuntu on an Linux host.
Developer(s) Red Hat, Inc.
Written in C (programming language)
Operating system Linux kernel
Type Platform virtualization
License GNU General Public License or GNU Lesser General Public License
Website linux-kvm.org (unofficial)
LIO™ v3.0 across KVM Guests on Neterion X3110 10 Gb/sec Ethernet hardware using PCIe device passthrough.
TCM v4.0 QLAXXX prototype using PCIe device passthrough for target mode MSI-X interrupts, and QEMU Megasas HBA emulation into Windows7 64-bit guest.
TCM v4.0 QLAXXX prototype using PCIe device passthrough for target mode MSI-X interrupts, and QEMU Megasas HBA emulation into Debian guest.

Kernel-based Virtual Machine (KVM) is a Linux kernel virtualization infrastructure. KVM supports native virtualization on x86 processors that provide Intel VT-x or AMD-V extensions; it has also been ported to S/390,[1] PowerPC,[2] IA-64, and an ARM port[3] is in progress.

Using KVM, it is possible to use both Datera, Inc. SCSI initiators and Linux-IO Targets inside of KVM guests at high bandwith speeds using PCI Express (PCIe) device passthrough.

Although doing PCIe device passthrough is not a requirement for using initiators and targets in KVM guests, this page focuses on 10 Gb/sec Ethernet hardware capable of single function operation, multi-function operation, and future PCIsig SR-IOV (included in v2.6.30-rc) and MR-IOV capable networking hardware.

Contents

Target KVM

FILEIO and RAMDISK storages objects are exported from KVM guests to other Linux instances via using iSCSI in Demo Mode. RAMDISK storage objects are using a 64 slot queue depth.

Linux-IO Target can also bring legacy PCIe networking and storage HBAs into KVM guests, e.g.:

The struct socket->sock_ops->sendpage() has been disabled for outgoing bulk struct scatterlist TX traffic (eg: iSCSI READ) in LIO™ code (it is usually enabled by default). This was done because sendpage() currently triggers a bug with the X3110 adapter in the KVM guest, and has been disabled for these tests.

The KVM guest is started with:

qemu-system-x86_64 -m 3072 -smp 8 -pcidevice host=06:00.0 lenny64guest1-orig.img

Initiators KVM

Both Core-iSCSI and Open-iSCSI Initiators can be used on kernel v2.6.29.2 inside of KVM guest.

The iSCSI Initiator KVM guest is started with:

qemu-system-x86_64 -m 4096 -smp 8 -pcidevice host=07:00.0 lenny64guest0-orig.img

Core-iSCSI

Core-iSCSI is using between 1 and 4 iSCSI connections per iSCSI session.

Both Initiators are using digests disabled with:

Also, struct socket->sock_ops->sendpage() triggers a bug in KVM guest with vxge, so unforutately it is not possible to obtain Open-iSCSI WRITE and WRITE/READ numbers yet..

Core-iSCSI is passing:

These correspond to struct scsi_host_template->can_queue and struct scsi_host_template->can_queue->cmd_per_lun settings.

Open-iSCSI

The Open-iSCSI module is from upstream linux-2.6.29.y.git. Open-iSCSI is using 1 iSCSI connection per iSCSI session.

Open-iSCSI is using its default values for can_queue and cmd_per_lun which are 128 and 32 respectively.

Test environment

The first tests focus on using RAMDISK and FILEIO storage objects using 10 GbE adapters. Of course, IBLOCK storage objects (SAS, SATA, SSD, etc.) work across the 10 GbE and iSCSI fabric as well.

It is also possible to use Linux dm-multipath for scaling across multiple iSCSI sessions to the same Linux-IO Target and iSCSI Target Port using both Open-iSCSI and Core-iSCSI. However, tests here focus on iSCSI MC/S and SC/S to evaluate which configuration provides maximum bandwith to the underlying Linux/SCSI struct scsi_device referencing an iSCSI LUN in a KVM guest.

Configuration

Setup

Each KVM guest is assigned a Neterion X3110 10 GbE adapter running in single-function (SF) mode. Each of these adapters has a total of 8 Vpath pairs (a TX and RX MSI-X interrupt pair for each Vpath) and each Linux KVM guest is using 4 of the 8 VPaths available for each adapter.

Both KVM guests providing Linux-IO Target and initiator functionality below are using a 9000 byte MTU, and the following sysctl settings:

### IPV4 specific settings
# turns TCP timestamp support off, default 1, reduces CPU use
net.ipv4.tcp_timestamps = 0
# turn SACK support off, default on
net.ipv4.tcp_sack = 0
# on systems with a VERY fast bus -> memory interface this is the big gainer
# sets min/default/max TCP read buffer, default 4096 87380 174760
net.ipv4.tcp_rmem = 10000000 10000000 10000000
# sets min/pressure/max TCP write buffer, default 4096 16384 131072
net.ipv4.tcp_wmem = 10000000 10000000 10000000
# sets min/pressure/max TCP buffer space, default 31744 32256 32768
net.ipv4.tcp_mem = 10000000 10000000 10000000
                                                                                
### CORE settings (mostly for socket and UDP effect)
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 524287
# maximum send socket buffer size, default 131071
net.core.wmem_max = 524287
# default receive socket buffer size, default 65535
net.core.rmem_default = 524287
# default send socket buffer size, default 65535
net.core.wmem_default = 524287
# maximum amount of option memory buffers, default 10240
net.core.optmem_max = 524287
# number of unprocessed input packets before kernel starts dropping them, default 300
net.core.netdev_max_backlog = 300000

Benchmarking

The benchmarking tool was disktest from Linux-Test-Project (LTP), using pthread and O_DIRECT to RAMDISK and FILEIO backstore objects. Results summary:

KVM I/O bandwith

The first test is using disktest for iSCSI READs, then with iSCSI WRITEs, then with both iSCSI WRITEs/READs. This is done using the following LIP disktest parameters with a 256K block size and 64 pthreads with O_DIRECT access to struct block_device enabled:

disktest -K64 -ID -PTX -h3 -t30 -r -b262144 /dev/sda
disktest -K64 -ID -PTX -h3 -t30 -w -b262144 /dev/sda
disktest -K64 -ID -PTX -h3 -t30 -wr -b262144 /dev/sda

Open-iSCSI Initiator

Read: SC/S

| 2009/05/14-18:55:20 | START | 2767 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0)
| 2009/05/14-18:55:20 | INFO  | 2767 | v1.4.2 | /dev/sda | Starting pass
| 2009/05/14-18:56:20 | STAT  | 2767 | v1.4.2 | /dev/sda | Total bytes read in 98669 transfers: 25865486336
| 2009/05/14-18:56:20 | STAT  | 2767 | v1.4.2 | /dev/sda | Total read throughput: 431091438.9B/s (411.12MB/s), IOPS 1644.5/s.
| 2009/05/14-18:56:20 | END   | 2767 | v1.4.2 | /dev/sda | Test Done (Passed)

Write: SC/S

This test could not effectively be performed because an Open-iSCSI problem: Unable to use sendpage() for iSCSI WRITE.

Read/Write: SC/S

This test could not effectively be performed because an Open-iSCSI problem: Unable to use sendpage() for iSCSI WRITE.

Core-iSCSI Initiator

Read: SC/S

| 2009/05/14-17:58:31 | START | 3334 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0)
| 2009/05/14-17:59:31 | STAT  | 3334 | v1.4.2 | /dev/sda | Total bytes read in 176951 transfers: 46386642944
| 2009/05/14-17:59:31 | STAT  | 3334 | v1.4.2 | /dev/sda | Total read throughput: 773110715.7B/s (737.30MB/s), IOPS 2949.2/s.
| 2009/05/14-17:59:31 | END   | 3334 | v1.4.2 | /dev/sda | Test Done (Passed)

Read: MC/S with 2 connections

| 2009/05/14-18:03:33 | START | 3406 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0)
| 2009/05/14-18:04:34 | STAT  | 3406 | v1.4.2 | /dev/sda | Total bytes read in 211214 transfers: 55368482816
| 2009/05/14-18:04:34 | STAT  | 3406 | v1.4.2 | /dev/sda | Total read throughput: 922808046.9B/s (880.06MB/s), IOPS 3520.2/s.
| 2009/05/14-18:04:34 | END   | 3406 | v1.4.2 | /dev/sda | Test Done (Passed)

Read: MC/S with 3 connections

| 2009/05/14-18:06:32 | START | 3478 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0)
| 2009/05/14-18:07:32 | STAT  | 3478 | v1.4.2 | /dev/sda | Total bytes read in 194460 transfers: 50976522240
| 2009/05/14-18:07:32 | STAT  | 3478 | v1.4.2 | /dev/sda | Total read throughput: 849608704.0B/s (810.25MB/s), IOPS 3241.0/s.
| 2009/05/14-18:07:32 | END   | 3478 | v1.4.2 | /dev/sda | Test Done (Passed)

Read: MC/S with 4 connections

| 2009/05/14-18:12:56 | START | 3551 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -r /dev/sda (-N 1048576) (-c) (-p R) (-D 100:0) (-t 0:2m) (-o 0)
| 2009/05/14-18:13:56 | STAT  | 3551 | v1.4.2 | /dev/sda | Total bytes read in 194242 transfers: 50919374848
| 2009/05/14-18:13:56 | STAT  | 3551 | v1.4.2 | /dev/sda | Total read throughput: 848656247.5B/s (809.34MB/s), IOPS 3237.4/s.
| 2009/05/14-18:13:56 | END   | 3551 | v1.4.2 | /dev/sda | Test Done (Passed)

Write: SC/S

| 2009/05/14-18:19:52 | START | 3929 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0)
| 2009/05/14-18:20:53 | STAT  | 3929 | v1.4.2 | /dev/sda | Total bytes written in 124977 transfers: 32761970688
| 2009/05/14-18:20:53 | STAT  | 3929 | v1.4.2 | /dev/sda | Total write throughput: 546032844.8B/s (520.74MB/s), IOPS 2082.9/s.
| 2009/05/14-18:20:53 | END   | 3929 | v1.4.2 | /dev/sda | Test Done (Passed)

Write: MC/S with 2 connections

| 2009/05/14-18:21:51 | START | 3999 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0)
| 2009/05/14-18:22:52 | STAT  | 3999 | v1.4.2 | /dev/sda | Total bytes written in 151405 transfers: 39689912320
| 2009/05/14-18:22:52 | STAT  | 3999 | v1.4.2 | /dev/sda | Total write throughput: 661498538.7B/s (630.85MB/s), IOPS 2523.4/s.
| 2009/05/14-18:22:52 | END   | 3999 | v1.4.2 | /dev/sda | Test Done (Passed)

Write: MC/S with 3 connections

| 2009/05/14-18:23:12 | START | 4072 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0)
| 2009/05/14-18:24:13 | STAT  | 4072 | v1.4.2 | /dev/sda | Total bytes written in 131623 transfers: 34504179712
| 2009/05/14-18:24:13 | STAT  | 4072 | v1.4.2 | /dev/sda | Total write throughput: 575069661.9B/s (548.43MB/s), IOPS 2193.7/s.
| 2009/05/14-18:24:13 | END   | 4072 | v1.4.2 | /dev/sda | Test Done (Passed)

Write: MC/S with 4 connections

| 2009/05/14-18:26:24 | START | 4145 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -w /dev/sda (-N 1048576) (-c) (-p R) (-D 0:100) (-t 0:2m) (-o 0)
| 2009/05/14-18:27:24 | STAT  | 4145 | v1.4.2 | /dev/sda | Total bytes written in 133475 transfers: 34989670400
| 2009/05/14-18:27:24 | STAT  | 4145 | v1.4.2 | /dev/sda | Total write throughput: 583161173.3B/s (556.15MB/s), IOPS 2224.6/s.
| 2009/05/14-18:27:24 | END   | 4145 | v1.4.2 | /dev/sda | Test Done (Passed)

Read/Write: SC/S

| 2009/05/14-17:50:16 | START | 2815 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0)
| 2009/05/14-17:51:16 | STAT  | 2815 | v1.4.2 | /dev/sda | Total bytes read in 62973 transfers: 16507994112
| 2009/05/14-17:51:16 | STAT  | 2815 | v1.4.2 | /dev/sda | Total bytes written in 62973 transfers: 16507994112
| 2009/05/14-17:51:16 | STAT  | 2815 | v1.4.2 | /dev/sda | Total read throughput: 550266470.4B/s (524.77MB/s), IOPS 2099.1/s.
| 2009/05/14-17:51:16 | STAT  | 2815 | v1.4.2 | /dev/sda | Total write throughput: 550266470.4B/s (524.77MB/s), IOPS 2099.1/s.
| 2009/05/14-17:51:16 | END   | 2815 | v1.4.2 | /dev/sda | Test Done (Passed)

Read/Write: MC/S with 2 connections

| 2009/05/14-17:51:30 | START | 2891 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0)
| 2009/05/14-17:52:30 | STAT  | 2891 | v1.4.2 | /dev/sda | Total bytes read in 82009 transfers: 21498167296    
| 2009/05/14-17:52:30 | STAT  | 2891 | v1.4.2 | /dev/sda | Total bytes written in 82010 transfers: 21498429440
| 2009/05/14-17:52:30 | STAT  | 2891 | v1.4.2 | /dev/sda | Total read throughput: 716605576.5B/s (683.41MB/s), IOPS 2733.6/s.
| 2009/05/14-17:52:30 | STAT  | 2891 | v1.4.2 | /dev/sda | Total write throughput: 716614314.7B/s (683.42MB/s), IOPS 2733.7/s.
| 2009/05/14-17:52:30 | END   | 2891 | v1.4.2 | /dev/sda | Test Done (Passed)

Read/Write: MC/S with 3 connections

| 2009/05/14-17:52:41 | START | 2964 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0)
| 2009/05/14-17:53:41 | STAT  | 2964 | v1.4.2 | /dev/sda | Total bytes read in 83225 transfers: 21816934400
| 2009/05/14-17:53:41 | STAT  | 2964 | v1.4.2 | /dev/sda | Total bytes written in 83225 transfers: 21816934400
| 2009/05/14-17:53:41 | STAT  | 2964 | v1.4.2 | /dev/sda | Total read throughput: 727231146.7B/s (693.54MB/s), IOPS 2774.2/s.
| 2009/05/14-17:53:41 | STAT  | 2964 | v1.4.2 | /dev/sda | Total write throughput: 727231146.7B/s (693.54MB/s), IOPS 2774.2/s.
| 2009/05/14-17:53:41 | END   | 2964 | v1.4.2 | /dev/sda | Test Done (Passed)

Read/Write: MC/S with 4 connections

| 2009/05/14-17:55:27 | START | 3176 | v1.4.2 | /dev/sda | Start args: -K64 -ID -PTX -h3 -T60 -wr /dev/sda (-N 1048576) (-c) (-p R) (-D 50:50) (-t 0:2m) (-o 0)
| 2009/05/14-17:56:28 | STAT  | 3176 | v1.4.2 | /dev/sda | Total bytes read in 84816 transfers: 22234005504
| 2009/05/14-17:56:28 | STAT  | 3176 | v1.4.2 | /dev/sda | Total bytes written in 84817 transfers: 22234267648
| 2009/05/14-17:56:28 | STAT  | 3176 | v1.4.2 | /dev/sda | Total read throughput: 741133516.8B/s (706.80MB/s), IOPS 2827.2/s.
| 2009/05/14-17:56:28 | STAT  | 3176 | v1.4.2 | /dev/sda | Total write throughput: 741142254.9B/s (706.81MB/s), IOPS 2827.2/s.
| 2009/05/14-17:56:28 | END   | 3176 | v1.4.2 | /dev/sda | Test Done (Passed)

See also

Notes

  1. Sergey Kiselev (2007-04-26). "kvm-20 on SLES 10 fails". gmane.comp.emulators.kvm.devel. 
  2. Igor Serebryany (2010-11-20). "OOM kernel panic on bootup in debian/ubuntu guests". gmane.comp.emulators.kvm.devel. 
  3. "KVM for ARM wiki". 

External links

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense