NSDI April , Andromeda - USENIX · Andromeda Data Plane VM Host Management Plane Open vSwitch...

AndromedaPerformance, Isolation, and Velocit at

Scale in Cloud Net ork Virtualization

NSDIApril ,

Andromeda Goals

Performance and IsolationHigh throughput and lo latenc , regardless of the actions of other tenants

VelocitQuickl de elop and deplo ne features and performance impro ements

ScalabilitLarge net orks, man tenants, rapid pro isioning

Cluster xx10.1.0.0/16

Cluster yy10.2.0.0/16

vmA

vmX

vmY

vmB

vmC

vmD

vmE

vmZ

vmV

virtual switch

vmF

vmM

vmP

vmN

vmL

vmQ

virtual switch

virtual switch

virtual switch

virtual switch

Host 10.1.1.3

Host 10.1.2.4

Host 10.1.2.5

Host 10.2.1.7

Host 10.1.1.9

Virtual IP192.168.0.2192.168.0.3192.168.0.4192.168.0.5192.168.0.6192.168.0.710.240.0.310.240.0.610.240.0.7

vnid111111222

Host:keylocal:1710.1.1.3:110.1.2.4:110.1.2.4:210.1.2.4:310.2.1.7:110.1.1.3:210.1.1.3:210.1.1.

Net ork Virtualization

Andromeda ArchitectureVM ControllerVM ControllerVM Controller

OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM

VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller

OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM

VM Host

Virtual s itch

VM

VM VM

RPC

E tended OpenFlo

Match Actions Match Actions Match Actions


OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM


OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM

VM Host

VM

Virtual s itch

VM

VM VM

RPC

E tended OpenFlo


1. New VM added


OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM


OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM

VM Host

VM

Virtual s itch

VM

VM VM

RPC

E tended OpenFlo


2. Install flows from other VMs to the new VM.

1. New VM added


OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM


OpenFlo Front End

VM Host

VM

Virtual s itch

VM

VM VM

VM Host

VM

Virtual s itch

VM

VM VM

RPC

E tended OpenFlo


3. Install flows from the new VM to other VMs in the network.

2. Install flows from other VMs to the new VM.

1. New VM added

Scaling Goals

Global connecti it

Large irtual net orks k+ VMs

Rapid pro isioningEnable on-demand orkloads

Programming Time for Large Net orksSetup:❖ VMs are placed on , hosts❖ VM Controller partitions

Programming time is O n×H n = number of VMs H = number of hosts

Quadratic scaling leads to pro isioning challenges❖ Control plane CPU and memor❖ Dataplane memor

mA

mX

mY

mB

mD

irtual s itch

mC

irtual s itch

Host . . .

Host . . .

Scaling ith Ho erboards

HoverboardHoverboardHoverboard

vmE

vmZ

vmV

virtual switch

Host 10.1.2.5

low priority route

mA

mX

mY

mB

mD

irtual s itch

mC

irtual s itch

Host . . .

Host . . .

Ho erboard Offloading


vmE

vmZ

vmV

virtual switch

Host 10.1.2.5

vmX → vmZ offload flow

low priority route

mA

mX

mY

mB

mD

irtual s itch

mC

irtual s itch

Host . . .

Host . . .

Ho erboard Offloading


vmE

vmZ

vmV

virtual switch

Host 10.1.2.5

low priority route

vmX → vmZ offload flow

VM Controller

OpenFlow Front Endstats

flow prog

ramming

Ho erboards reduce time to program net ork connecti it for large net orks

❖ ✕ faster for a , -VM net ork

Programming Time for Large Net orks

Wh Ho erboards Are Effecti e

% of VM pairs ha e peak throughput < kbps

o er % of VM pairs ne er communicate

Peak throughput for all VM pairs in all virtual networks in one cluster over a 30-minute interval

Today, more than 99.5% of traffic is offloaded.

Andromeda Data Plane

VM Host Open vSwitch

Host OS Kernel

NIC

Extended OpenFlow

OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance

Userspace dataplane, live migration, and hitless upgrades for feature velocity

Manages on-host Flow Tables


VM Host Management Plane

Open vSwitch

Host OS Kernel

NIC

Extended OpenFlow



Manages on-host VMs



Open vSwitch

Andromeda Fast Path

Host OS Kernel

NIC

Extended OpenFlow



Busy polls physical & virtual NIC queues, forwards VM packets



Open vSwitch

Andromeda Fast PathMatch Action

Flo cache

miss insert

Host OS Kernel

NIC

Extended OpenFlow



Routes packet, applies per-flow Fast Path actions (encap, decap, etc)



Open vSwitch

Guest VM Coprocessor


Flo cache

miss insert

Host OS Kernel

NIC

Extended OpenFlow



Per-VM attributed threads for executing CPU-intensive packet ops (e.g., DoS)



Open vSwitchGuest VM



Flo cache

miss insert

Host OS Kernel

NIC

shared memory ring

Extended OpenFlow



Fast Path polls guest VM rings & copies packets to/from guest VM memory

Data Plane - Fast Path




Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow

OpenFlo Front End High performance traffic processed end-to-end on Fast Path

> 30Gb/s throughput & > 3M pps on one core

Flow Table performs routing, encap/decap, etc.

Fast Path polls virtual & physical NIC rings

PacketMatch Action

Flo cache

Pull packet from NICParse, TcpDump, ...





Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow





Packet

Match Action

Flow LookupRoute, decap, ...





Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow





Packet

Match Action

Deliver packet to VMCopy, update rings, ...

Flo cache

Data Plane - Coprocessor Path




Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow

OpenFlo Front End

PacketMatch Action

Flo cache

Pull packet from NICParse, TcpDump, ...

Coprocessors are per-VM threads CPU attributed to VM container

Coprocessors execute CPU-intensive packet ops such as DoS

Decouples feature growth from Fast Path speed





Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow

OpenFlo Front End

Packet

Match Action

Flow LookupRoute, decap, set Coprocessor stages...








Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow

OpenFlo Front End

Match Action

Flo cache

Packet

Send to CoprocessorApply Coprocessor stages (e.g., DoS)








Andromeda Fast Path

miss insert

Host OS Kernel

NIC

Extended OpenFlow

OpenFlo Front End

Packet

Match Action

Flo cache

Deliver packet to VMCopy, update rings, ...




VM-VM Throughput Single core per host for dataplane Fast Path. Sk lake testbed hosts.

Both hosts connected to same Top of Rack s itch.

VM-VM Round Trip Latenc Single core per host for dataplane Fast Path. Sk lake testbed hosts.

Both hosts connected to same Top of Rack s itch.

CPU Efficienc

Minimizing host and guest net ork CPU c cles per b te CPB is critical

Since initial production release, e ha e impro ed CPB b > 6 as measured on sender + recei er host during a multi-stream benchmark.

Andromeda . + use a single core per host for the dataplane Fast Path. Results from Sand bridge testbed hosts connected to same ToR s itch.

CPU Efficienc E olution

Host: 43.5Guest:16.0

Host: 30.4Guest:12.3

Host: 5.4Guest: 5.6

Host: 2.6Guest: 5.0

Host: 2.0Guest: 4.9

Andromeda 1.0Kernel datapath

Andromeda 1.5Optimize pipeline

Andromeda 2.0OS bypass, 1 thread hop

Andromeda 2.1Remove thread hop

Andromeda 2.2Memory copy offload

31

VelocitA rapid release c cle enables s ift deplo ment of features & bug fi es.

Our dataplane has eekl rollouts ia non-disrupti e upgrades.

Li e migration allo s VMs to be migrated bet een ph sical host ithout disruption, enabling transparent host maintenance.

Dataplane Hitless Upgrade /

Physical NIC

Upgrade Brownout

Old Dataplane state is transferred to New Dataplane in the background

Old Dataplane continues serving physical NIC & virtual NIC queues

State XferOld Dataplane New Dataplane

Guest VM


Physical NIC

State XferOld Dataplane New Dataplane

Guest VM

Upgrade Blackout

Old Dataplane stops serving virtual & physical NIC queues

Then, any updated (delta) Old Dataplane state is transferred to New Dataplane


Physical NIC

New Dataplane

Guest VM

Upgrade Complete

State xfer done. Median blackout time is 270ms.

New Dataplane starts serving VM virtual NIC & physical NIC queues

Old dataplane terminated

ConclusionWe ha e discussed the design and e olution of Andromeda

Control plane scalabilit & Rapid pro isioning

● Ho erboard model a oids programming long tail of mostl idle flo s on VM host. Scales to k VMs/net ork

High performance & Feature elocit

● OS B pass dedicated CPU dataplane pro ides high performance > Gb/s, > M pps ith core & eekl non-disrupti e updates

NSDI April , Andromeda - USENIX · Andromeda Data Plane VM Host Management Plane Open vSwitch...

Documents

Transcript of NSDI April , Andromeda - USENIX · Andromeda Data Plane VM Host Management Plane Open vSwitch...