Changpeng Liu, Cloud Software Engineer

Piotr Pelpliński, Cloud Software Engineer

• Introduction to VirtIO and Vhost

• SPDK Vhost Architecture

• Use cases for vhost

• Benchmarks

• Next steps

QEMU VIRTIO

Vhost (KERNEL)

vhost (USERSPACE)

Virtio• Paravirtualized driver specification

• Common mechanisms and layouts for device discovery, I/O queues, etc.

• virtio device types include:

• virtio-net

• virtio-blk

• virtio-scsi

• virtio-gpu

• virtio-rng

• virtio-cryptoHypervisor (i.e. QEMU/KVM)

Guest VM(Linux*, Windows*, FreeBSD*, etc.)

virtio front-end drivers

virtio back-end drivers

device emulation

virtqueuevirtqueuevirtqueue

5

QEMU

Kernel

Guest VM

Guest kernel

Application

virtqueue

I/O Processing

AIO

QEMU VirtIO SCSI

6

1. Add IO to virtqueue

2. IO processed by QEMU

3. IO issued to kernel

4. Kernel pins memory

5. Device executes IO

6. Guest completion interrupt

QEMU VIRTIO

Vhost (KERNEL)

vhost target (kernel or userspace)

Vhost• Separate process for I/O processing

• vhost protocol for communicating guest VM parameters

• memory

• number of virtqueues

• virtqueue locations

Hypervisor (i.e. QEMU/KVM)

Guest VM(Linux*, Windows*, FreeBSD*, etc.)

virtio front-end drivers

device emulation

virtio back-end drivers

virtqueuevirtqueuevirtqueue

vhostvhost

QEMU

Kernel

Guest VM

Guest kernel

Application

virtqueue

vhost-kernel AIO

Kernel VHOST

9

1. Add IO to virtqueue

2. Write virtio doorbell

3. Wake vhost kernel

4. Kernel pins memory

5. Device executes IO

6. Guest completion interrupt

kvm

QEMU VIRTIO

Vhost (KERNEL)

vhost (USERSPACE)

11

Host Memory

QEMU

Guest VM

virtio-scsi

Shared Guest VMMemory

SPDK vhost

vhost DPDK vhost

virtio-scsi

virtqueuevirtqueuevirtqueue

eventfd

UNIX domain socket

SPDK VHOST Architecture

QEMU

Kernel

Guest VM

Guest kernel

Application

virtqueue

SPDK Vhost

vhost i/o

SPDK VHOST

12

1. Add IO to virtqueue

2. Poll virtqueue

3. Device executes IO

4. Guest completion interrupt

kvm

SCSI

Block Device Layer (bdev)

scsi/bdev Translation

NVMebdev driver

mallocbdev driver

Ceph RBDbdevdriver

Linux AIO bdev driver

vhostDPDK

rte_vhost

JS

ON

RP

C

QEMU

SPDK VHOST Layers

13

COMPARISON with existing solutions

Host Kernel

QEMU

Guest VM

Guest Kernel

VIRTIO_SCSI

VIRTIO_SCSI_PCI

NVME_MOD

Host Kernel

QEMU

VHOST_SCSI_PCI

NVME_MOD

QEMU

VHOST_USER_SCSI_PCI

LIO

VHOST

SPDK VHOST

PMD_NVME

SCSI

VHOST_USER

Guest VM

Guest Kernel

VIRTIO_SCSI

Guest VM

Guest Kernel

VIRTIO_SCSI

IOCTL SOCKET

14

QEMU VIRTIO SCSI Target VHOST Kernel Target VHOST Userspace Target

SPDK

SCSI

BDAL

SPDK Vhost

BlobstoreBlob Bdev

NVMe Driver

BDAL

NVMeBdev

VM

Intel® SSD for Datacenter

VM EPHEMERAL STORAGE• Increased efficiency yields

greater VM density

16

Intel® SSD for Datacenter

SPDK

SCSI

SPDK Vhost

NVMe-oFInitiator

BDAL

NVMe-oFBD

VM

NVMe-oFTarget

VM Remote Storage• Enable disaggregation and

migration of VMs using remote storage

17

Ceph Cluster

SPDK

Intel® SSD for Datacenter

Ceph RBD Driver

BDAL

CephBdev

SCSI

SPDK Vhost

VM

VM CEph Storage• Potential for innovation in

data services

• Cache

• Deduplication

18

0

2

4

6

8

10

12

14

16

QEMU VirtIO Vhost Kernel SPDK Vhost

Co

res

VM IO Processing

System configuration: 44x Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (HT off); Cores per socket: 22; 8x Samsung 8GB DDR4 @2400 12x Intel SSD DC P3700 Series 1,5T @ FW 8DV101H0 DPDK: 17.02; Host Dist/Kernel: Fedora 25/Kernel 4.8.15-300; Guest Dist/Kernel: Ubuntu 16.04/Kernel 4.4.0-59-generic, mq enabled; Fio ver: fio-2.2.10; Fio workload: blocksize=4k, iodepth=512, iodepth_batch=128, iodepth_low=256, ioengine=libaio, size=10G, ramp_time=10, group_reporting, thread, numjobs=1, direct=1, rw=randread

Benchmarks

0

200000

400000

600000

800000

1000000

1200000

QEMU VirtIO Vhost Kernel SPDK Vhost

IO p

er

seco

nd

IO per second

20

• VFIO Support

• Support for vhost-blk protocol

• Live migration

• Performance tuning, including:

• multiqueue

• completion event coalescing

• Integration software

Next Steps

22

Unmodified guest VMs

Add data services and add value

Big improvement in VM I/O efficiency

Notices and DisclaimersIntel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.

No computer system can be absolutely secure.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.

Intel, the Intel logo, Xeon, and others are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

© 2017 Intel Corporation.