Hardware accelerated virtio networking for nfv linux con

TM

Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, t

he Energy Efficient Solutions logo, mobileGT, PowerQUICC, QorIQ, StarCore and Symphony

are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack,

ColdFire+, CoreNet, Flexis, Kinetis, MXC, Platform in a Package, Processor Expert, QorIQ

Qonverge, Qorivva, QUICC Engine, SMARTMOS, TurboLink, VortiQa and Xtrinsic are trademarks

of Freescale Semiconductor, Inc. All other product or service names are the property

of their respective owners. © 2014 Freescale Semiconductor, Inc.

TM

Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, C-Ware, the Energy Efficient Solutions logo, mobileGT, PowerQUICC, QorIQ, StarCore

and Symphony are trademarks of Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. BeeKit, BeeStack, ColdFire+, CoreNet, Flexis, Kinetis, MXC, Platform in a

Package, Processor Expert, QorIQ Qonverge, Qorivva, QUICC Engine, SMARTMOS, TurboLink, VortiQa and Xtrinsic are trademarks of Freescale Semiconductor, Inc.

All other product or service names are the property of their respective owners. © 2014 Freescale Semiconductor, Inc.

2

• Network Function Virtualization

• I/O Virtualization Mechanisms

• Vhost-net Details

• NFV with virtio

− Issues

− Workaround

• Proposed Architecture

TM





3

Network Function Virtualization

TM





4

• Dynamic network service launch restricted by existing hardware

− Increased capital investment

• New infrastructure = Increased power requirement

− Increased cost of energy

• Shorter procure-design-integrate-deploy cycle due to rapidly evolving hardware

− Shorter hardware lifecycles

TM





5

• Allows for dynamic network service launch

− Reduces time for deploying new network services

• Same infrastructure can be used/shared for hosting multiple services

− Reduced capital investment

− Reduced cost of energy

• Offers Agility and Flexibility

− Quickly scale up or down services based on the demand

− Reduces time to deploy new networking services

TM





6

• Network Functions

− Data plane and Control plane:Firewall, VPN , UTM etc.

• Virtualization

− Hypervisor managed VMs

• IT virtualization technology to consolidate network equipment .

OS

Network

Function

Network

Function

Network

Function

hardware

OS OS

Hypervisor

TM





7

VM1 VM2 VM3 VM4

Hypervisor

Switch

Server

Network Appliance

NFVM1 NFVM2 NFVM3 NFVM4

Switch

Hypervisor

Pkt

VM1 NFVM VM3 VM4

Hypervisor

Switch

Server

Pkt

Pkt

Shared NIC Shared NIC

Shared NIC

Pkt

Pkt

Pkt

Pkt

Pkt

TM





8

I/O Virtualization Mechanisms

TM





9

Guest OS

Hypervisor

(Full Virtualization)

Traps

Device

Emulation

Guest OS

Hypervisor

(ParaVirtualization)

Interfaces

Device

Emulation

Para-drivers

HardwareHardware

TM





10

Guest OS

Front end drivers

Back end Drivers

Device EmulationHV (Para)

Virtio

virtqueues

TM





11

► Each guest sees a private VirtIO

network device on PCI bus

► VirtIO network driver is needed in

guest

► Standard tun/tap interface is used by

QEMU to bridge to host network

stack

► Backend driver in QEMU

Partition 1

App

QEMU

Partition 2

OS

App

OS

App

QEMU

Linux Kernel / KVM

tun/tap

eth0

Hardware

Bridge

TM





12

Vhost-net is a module in

Linux Kernel.

Transfers buffers between host and

guest.

Bypasses QEMU for virtqueue

operations

Partition 1

App

QEMU

Partition 2

OS

App

OS

App

QEMU

tun/tap

Hardware

eth0

Bridge

vhost Linux Kernel /

KVM

TM





13

Vhost-net Details

TM





14

Guest

VNIC

vhost thread

TX vqRX vq

Linux Kernel/KVM

Tap

BridgeEthernet

Driver

NIC

TM





15

QEMU

vhost_dev_ioctl

kthread_create

User Space

ioctl

This thread is responsible for

handling both Rx and Tx jobs for

a vhost device

A worker thread is

created per vhost

instance

Kernel Space

TM





16

RX Packet Flow

TM





17

START

Hardware Interrupt

Interrupt Handler

irq_exit

napi_schedule

Other Routines

raise_softirq_irqoff

other routinesdo_softirq

netif_receive_skbnet_rx_action

dpaa_eth_poll

qman_p_poll_dqrr

priv_rx_default_dqrr

AHardirq

Softirq

TM





18

br_handle_frame

A

br_handle_frame_finish

br_forward

dev_hard_start_xmit

tun_net_xmit

skb_queue_tail

br_dev_queue_push_xmit

dev_queue_xmit

wakeup_poll

This is going to wake up

tun_chr_poll

Bridge

Handling

of skb

Softirq

Tun

receives

the skb

and

queues it

TM





19

START

Work dequeued

kthread_should_stop

False

False

list_empty

True

True

need_

schedule

schedule

TrueReturn False

RX processing

TM





20

sock_recvmsg()

calls tun_recvmsg

handle_rx()

sock_len

Socket refers to

tap socket

Poll

Updated used list

tun_do_read() Copy Packet

True

Return

FalseTrue

Return

False

vhost_worker()

Total bytes

received >=

Vhost Budget

False

TM





21

A

Free Rx Buffer > size/2

try_fill_rcvtry_fill_rcv

schedule_delayed

refill_work

Done. Added empty

buffers

received < budget

napi_complete

TM





22

TX Packet Flow

TM





23

Traffic from Guest OS

Virtio-net Driver

Write Desc in Available Ring

Ring_Full Report

Congestion

TM





24

iowrite16

start_xmit()

virtqueue_kick()

virtqueue_notify()

virtqueue_kick_prepareFalse

Return

True

kvm_io_bus_write

TM





25

NFV with VirtIO

TM





26

Shared NIC

Shared NIC

VM1 VM2 VM3 VM4

Hypervisor

Switch

Server

Network Appliance

NFVM1 NFVM2 NFVM3 NFVM4

Switch

Hypervisor

Pkt

Pkt

Pkt

Pkt Packet Flooding

Packet Drop

TM





27

TM





28

NIC driver

Bridge

Tap1 2 … 500

Tap Queue Full

Packets keep on

dropping at tap level

Physical NICNAPI

processing

Packet Flooding

Vhost Thread Fails to Schedule due to lower

Priority than SoftIRQ

Ingress Rate

TM





29

TM





30

Congestion

Send Buffer to

Bridge/Tap

FalseTrue

NAPI Handler

Disable HWI

Vhost_Rx

Rx Ring < Lower Watermark

Process Pkt

Enable HWI

NAPI SoftIRQ ProcessingVhost Rx Processing

TM





31

NIC driver

Bridge

Tap

1 2 … Last

indexNow Tap Queue Full

Physical NIC

Packet gets queued

Congestion Notification

sent to bridge

Packet Drop at HW

last index

Packet Flooding

Bridge stops forwarding

NIC driver Disable HWI

Vhost Thread Now Vhost can schedule

during packet flooding.

TM





32

0

100

200

300

400

500

600

700

800

900

1000

0 500 1000 1500 2000 2500

eg

ress R

ate

Ingress Rate

Current Design Modified Design

TM





33

TM





34

• Dynamic vhost thread controlled NIC driver instance.

• Vhost thread registers with NIC driver:

− Rx Function handle

− Buffer depletion handler

• Driver provides following during registration to Vhost:

− Tx Function Handler

− Buffer pool management handle

TM





35

Vhost Thread

New Network Acceleration Interface Driver

Ethernet Port

Rx FQ/BD

Ring

Tx FQ/BD

Ring

PCD block

Rx

Ring

Tx

Ring

GUEST

VNET1

KVMVhost

Initialization

Register for

Driver

Instantiation

BD Ring

Creation,

Buffer Pool

Creation

Fill Tx

Handler,

Buffer Pool

Management

handler and

Return

Buffer Pool

TM





36

Vhost Thread

New Network Acceleration Interface Driver

Ethernet Port

Rx FQ/BD

Ring

PCD block

Rx

Ring

Tx

Ring

GUEST

VNET1

KVM

RxTx

No Rx Buffers

Left in Buffer

Pool

Buffer Pool

Buffer Acquire

Failure

Packet Dropped

at HW

Buffers Seeded

by Vhost

Buffers

available in

Buffer Pool Buffer Acquire

Succeed

Packet DMAed

to buffer and

Sent to Rx Ring

Pkt

Buffer Acquire

Requested

TM





37

• Vhost-net interfaces can get saturated at high traffic rates

• No mechanism to communicate Vhost-net interface saturation information to host

• With a more integrated guest/host Virtio infrastructure, it’s possible to avoid system saturation and increase throughput

• Hardware acceleration can be leveraged for Vhost net interfaces

Hardware accelerated virtio networking for nfv linux con

Technology

Transcript of Hardware accelerated virtio networking for nfv linux con