AndromedaPerformance, Isolation, and Velocit at
Scale in Cloud Net ork Virtualization
NSDIApril ,
Andromeda Goals
Performance and IsolationHigh throughput and lo latenc , regardless of the actions of other tenants
VelocitQuickl de elop and deplo ne features and performance impro ements
ScalabilitLarge net orks, man tenants, rapid pro isioning
Cluster xx10.1.0.0/16
Cluster yy10.2.0.0/16
vmA
vmX
vmY
vmB
vmC
vmD
vmE
vmZ
vmV
virtual switch
vmF
vmM
vmP
vmN
vmL
vmQ
virtual switch
virtual switch
virtual switch
virtual switch
Host 10.1.1.3
Host 10.1.2.4
Host 10.1.2.5
Host 10.2.1.7
Host 10.1.1.9
Virtual IP192.168.0.2192.168.0.3192.168.0.4192.168.0.5192.168.0.6192.168.0.710.240.0.310.240.0.610.240.0.7
vnid111111222
Host:keylocal:1710.1.1.3:110.1.2.4:110.1.2.4:210.1.2.4:310.2.1.7:110.1.1.3:210.1.1.3:210.1.1.
Net ork Virtualization
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
VM
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
1. New VM added
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
VM
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
2. Install flows from other VMs to the new VM.
1. New VM added
Andromeda ArchitectureVM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM ControllerVM ControllerVM Controller VM ControllerVM ControllerVM Controller
OpenFlo Front End
VM Host
VM
Virtual s itch
VM
VM VM
VM Host
VM
Virtual s itch
VM
VM VM
RPC
E tended OpenFlo
Match Actions Match Actions Match Actions
3. Install flows from the new VM to other VMs in the network.
2. Install flows from other VMs to the new VM.
1. New VM added
Scaling Goals
Global connecti it
Large irtual net orks k+ VMs
Rapid pro isioningEnable on-demand orkloads
Programming Time for Large Net orksSetup:❖ VMs are placed on , hosts❖ VM Controller partitions
Programming time is O n×H n = number of VMs H = number of hosts
Quadratic scaling leads to pro isioning challenges❖ Control plane CPU and memor❖ Dataplane memor
mA
mX
mY
mB
mD
irtual s itch
mC
irtual s itch
Host . . .
Host . . .
Scaling ith Ho erboards
HoverboardHoverboardHoverboard
vmE
vmZ
vmV
virtual switch
Host 10.1.2.5
low priority route
mA
mX
mY
mB
mD
irtual s itch
mC
irtual s itch
Host . . .
Host . . .
Ho erboard Offloading
HoverboardHoverboardHoverboard
vmE
vmZ
vmV
virtual switch
Host 10.1.2.5
vmX → vmZ offload flow
low priority route
mA
mX
mY
mB
mD
irtual s itch
mC
irtual s itch
Host . . .
Host . . .
Ho erboard Offloading
HoverboardHoverboardHoverboard
vmE
vmZ
vmV
virtual switch
Host 10.1.2.5
low priority route
vmX → vmZ offload flow
VM Controller
OpenFlow Front Endstats
flow prog
ramming
Ho erboards reduce time to program net ork connecti it for large net orks
❖ ✕ faster for a , -VM net ork
Programming Time for Large Net orks
Wh Ho erboards Are Effecti e
% of VM pairs ha e peak throughput < kbps
o er % of VM pairs ne er communicate
Peak throughput for all VM pairs in all virtual networks in one cluster over a 30-minute interval
Today, more than 99.5% of traffic is offloaded.
Andromeda Data Plane
VM Host Open vSwitch
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Manages on-host Flow Tables
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Manages on-host VMs
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Andromeda Fast Path
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Busy polls physical & virtual NIC queues, forwards VM packets
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Andromeda Fast PathMatch Action
Flo cache
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Routes packet, applies per-flow Fast Path actions (encap, decap, etc)
Andromeda Data Plane
VM Host Management Plane
Open vSwitch
Guest VM Coprocessor
Andromeda Fast PathMatch Action
Flo cache
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Per-VM attributed threads for executing CPU-intensive packet ops (e.g., DoS)
Andromeda Data Plane
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast PathMatch Action
Flo cache
miss insert
Host OS Kernel
NIC
shared memory ring
Extended OpenFlow
OpenFlo Front EndOS bypass, busy polling dedicated CPU Fast Path for high performance
Userspace dataplane, live migration, and hitless upgrades for feature velocity
Fast Path polls guest VM rings & copies packets to/from guest VM memory
Data Plane - Fast Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End High performance traffic processed end-to-end on Fast Path
> 30Gb/s throughput & > 3M pps on one core
Flow Table performs routing, encap/decap, etc.
Fast Path polls virtual & physical NIC rings
PacketMatch Action
Flo cache
Pull packet from NICParse, TcpDump, ...
Data Plane - Fast Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End High performance traffic processed end-to-end on Fast Path
> 30Gb/s throughput & > 3M pps on one core
Flow Table performs routing, encap/decap, etc.
Fast Path polls virtual & physical NIC rings
Packet
Match Action
Flow LookupRoute, decap, ...
Data Plane - Fast Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End High performance traffic processed end-to-end on Fast Path
> 30Gb/s throughput & > 3M pps on one core
Flow Table performs routing, encap/decap, etc.
Fast Path polls virtual & physical NIC rings
Packet
Match Action
Deliver packet to VMCopy, update rings, ...
Flo cache
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
PacketMatch Action
Flo cache
Pull packet from NICParse, TcpDump, ...
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
Packet
Match Action
Flow LookupRoute, decap, set Coprocessor stages...
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
Match Action
Flo cache
Packet
Send to CoprocessorApply Coprocessor stages (e.g., DoS)
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
Data Plane - Coprocessor Path
VM Host Management Plane
Open vSwitchGuest VM
Guest VM Coprocessor
Andromeda Fast Path
miss insert
Host OS Kernel
NIC
Extended OpenFlow
OpenFlo Front End
Packet
Match Action
Flo cache
Deliver packet to VMCopy, update rings, ...
Coprocessors are per-VM threads CPU attributed to VM container
Coprocessors execute CPU-intensive packet ops such as DoS
Decouples feature growth from Fast Path speed
VM-VM Throughput Single core per host for dataplane Fast Path. Sk lake testbed hosts.
Both hosts connected to same Top of Rack s itch.
VM-VM Round Trip Latenc Single core per host for dataplane Fast Path. Sk lake testbed hosts.
Both hosts connected to same Top of Rack s itch.
CPU Efficienc
Minimizing host and guest net ork CPU c cles per b te CPB is critical
Since initial production release, e ha e impro ed CPB b > 6 as measured on sender + recei er host during a multi-stream benchmark.
Andromeda . + use a single core per host for the dataplane Fast Path. Results from Sand bridge testbed hosts connected to same ToR s itch.
CPU Efficienc E olution
Host: 43.5Guest:16.0
Host: 30.4Guest:12.3
Host: 5.4Guest: 5.6
Host: 2.6Guest: 5.0
Host: 2.0Guest: 4.9
Andromeda 1.0Kernel datapath
Andromeda 1.5Optimize pipeline
Andromeda 2.0OS bypass, 1 thread hop
Andromeda 2.1Remove thread hop
Andromeda 2.2Memory copy offload
31
VelocitA rapid release c cle enables s ift deplo ment of features & bug fi es.
Our dataplane has eekl rollouts ia non-disrupti e upgrades.
Li e migration allo s VMs to be migrated bet een ph sical host ithout disruption, enabling transparent host maintenance.
Dataplane Hitless Upgrade /
Physical NIC
Upgrade Brownout
Old Dataplane state is transferred to New Dataplane in the background
Old Dataplane continues serving physical NIC & virtual NIC queues
State XferOld Dataplane New Dataplane
Guest VM
Dataplane Hitless Upgrade /
Physical NIC
State XferOld Dataplane New Dataplane
Guest VM
Upgrade Blackout
Old Dataplane stops serving virtual & physical NIC queues
Then, any updated (delta) Old Dataplane state is transferred to New Dataplane
Dataplane Hitless Upgrade /
Physical NIC
New Dataplane
Guest VM
Upgrade Complete
State xfer done. Median blackout time is 270ms.
New Dataplane starts serving VM virtual NIC & physical NIC queues
Old dataplane terminated
ConclusionWe ha e discussed the design and e olution of Andromeda
Control plane scalabilit & Rapid pro isioning
● Ho erboard model a oids programming long tail of mostl idle flo s on VM host. Scales to k VMs/net ork
High performance & Feature elocit
● OS B pass dedicated CPU dataplane pro ides high performance > Gb/s, > M pps ith core & eekl non-disrupti e updates
Top Related