Build High Performance NSH-based SFC Solution with FD.io and OpenDaylight *Danny Zhou (danny.zhou@intel.com) , Yi Yang (yang.yi@intel.com)

Network Platforms Group, DCG, Intel

Acknowledgement: Hongjun Ni, Keith Burn, Brady Johnson, Anna Wan, John DiGiglio

2

Agenda

Service Function Chaining Overview

FD.io’s NSH_SFC Plugin

– Internals

– Evolving NSH_SFC Features

– NSH_SFC Performance and Analysis

– Functional and Performance Automation Test

OpenDaylight SFC Integration with VPP and NSH_SFC

Summary and Future Plan

3

Introducing SFC (Service Function Chaining)

SFF1

SF1(FW)

SFF2

SF2(LB)

Classifier1

Classifier2

Legend

SFC ComponentSFF: Service Function Forwarder

SF: Service Function

NSH: Network Service HeaderService Function Chain

Client1

Client2

Client3

Server

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

VER|O|C|R|R|R|R|R|R|R| Length | MD Type | Next Protocol

Service Path ID | Service Index

Mandatory Context Header

Mandatory Context Header

Mandatory Context Header

Mandatory Context Header

Proxy

SFC Use Cases Overview

4

Domain Use Cases Details

Telco Gi-LAN in Mobile Core Network

Internet Gateway in Fixed Broadband Network

Internet Gateway in Cloud CPE

Data Center

Internet Gateway

Customer

Network

Access

NodeBNG

Internet

Service Chain Network

L2-CPE

L2-CPE

Internet

Cloud CPE / SFC

FW

ACL

NAT

…

Server

Server

… Router Internet

Data Center

IDS/IPS FW NAT

SFC

UE SGW PGWInternet

SFC

ADCFW

DNS

PCEF

Video Opt.

DPI

FD.io’s NSH_SFC Plugin Internals

5L4 header

InnerIP header

Inner MACheader

UDP headerOuter

IP headerOuter MAC

headerNSH

headerVxLAN-GPE

headerPayload

• Typical NSH packet processed by VPP and NSH_SFC plugin

VxLAN-GPE

NSH_SFC Plugin

NSH Map NSH Entry

nsh-inputnsh-proxynsh-

classifier

nsh-aware-vnf-proxy

Supported Graph Nodes

nsh-inputnsh-proxynsh-

classifier

VxLAN-GPE

Maps one tunnel to another

Stores NSH header info

L2-input-classify

Ethernet-input

inputOtherGraph nodes

inputOtherGraph nodes

• NSH_SFC Plugin

Typical NSH_SFC Usage : NSH-Aware SF

VPPdpdk-input

ethernet-input

mpls-ethernet-input

ip6-input ip4-input arp-input llc-input

ip4-lookup

…

Packet Vector

NSH_SFC

VxLAN-GPE

FW

ip4-udp-lookup

ip4-local

NAT44

LB

ethernet-output

Legend

Ingress

Egress

ip4-rewrite-transmit

Evolving NSH_SFC Features

Service Function Forwarder Plugin framework in VPP and HC Integration with ODL SFC

16.09

NSH Classifier NSH Proxy Integration test with HC and ODL

17.01

NSH-aware SFs by collocating Proxy and SF (e.g. SNAT) Initial MD Type 2 support IOAM over NSH CSIT enabling (functional)

17.04

Under development ……. Enable Eth and Geneve as NSH transports Performance optimization Performance automation test Real NSH-aware VNF by enabling NSH_SFC to pass per-packet metadata to real SF (stretch goal)

17.07

NSH_SFC Performance and Analysis

9

• BIOS Configuration

Enhanced Intel Speedstep Enabled

Turbo Boost Enabled

Processor C3 Disabled

Processor C6 Disabled

Hyper-Threading Disabled

Intel VT-d Enabled

CPU Power and Performance Policy

Performance

Memory Freq. 2133 MHz

Total Memory Size 64 GB

Memory RAS and Performance Configuration -> NUMA Optimized

ENABLED

QPI B/W 9.6 GT/s

MLC Streamer ENABLED

MLC Spatial Prefetcher ENABLED

DCU Data Prefetcher ENABLED

DCU Instruction Prefetcher ENABLED

Direct Cache Access (DCA) ENABLED

• Software Configuration

OS Ubuntu 16.04 LTS

Kernel Linux version 4.4.0-21-generic

DPDK 16.11

VPP 17.01

NSH_SFC 17.01

Honeycomb 17.01

Device Under Test• Hardware Configuration

CPUIntel(R) Xeon(R) CPU E5-2699 v4 @

2.20GHz(Broadwell)

DIMM 2133 MHz, 64GB Total

NIC2x 82599ES 10-Gigabit SFI/SFP+

Network Connection

PacketGenIxia* 10 Gigabit Ethernet Traffic

Generator(16 ports)

DUT(VPP + NSH_SFC)

Traffic Gen(Trex)

Port A Port B

Port A Port B

Core 0

LegendFlow A

Flow B

• Network Topology

1C1T Throughput of NSH_SFC 17.01

72B 128B 256B 512B 1024B 1280B 1518B IMIX

NSH_SFC Classifier RFC2544 4934.449 7122.883 8257.485 8691.556 9777.756 9829.521 9841.643 8092.955

NSH_SFC Classifier 100% TX 5182.164 7174.001 9428.353 9669.746 9820.892 9854.23 9875.715 9425.316

Delta 5.02% 0.72% 14.18% 11.25% 0.44% 0.25% 0.35% 16.46%

Theoritical 10000 10000 10000 10000 10000 10000 10000 10000

0

2000

4000

6000

8000

10000

12000T

hro

ug

hp

ut

(Mb

ps)

NSH_SFC Classifier Throughput (RFC2544 vs. 100% TX Rate)

152B 256B 512B 1024B 1280B 1518B IMIX

NSH_SFC Proxy Inbound RFC2544 5401.172 8405.794 8269.983 9578.545 9661.54 9713.915 4566.067

NSH_SFC Proxy Inbound 100% TX 5401.565 8405.826 9172.955 9578.69 9661.58 9713.93 8633.353

Delta 0.01% 0.00% 10.92% 0.00% 0.00% 0.00% 89.08%

Theoritical 10000 10000 10000 10000 10000 10000 10000

0

2000

4000

6000

8000

10000

12000

Th

rou

gh

pu

t (M

bp

s)

NSH_SFC Proxy Inbound Throughput (RFC2544 vs. 100% TX Rate)

128B 256B 512B 1024B 1280B 1518B IMIX

NSH_SFC Proxy Outbound RFC2544 3843.331 8789.628 8729.212 9755.99 9819.183 9814.526 8700.796

NSH_SFC Proxy Outbound 100% TX 4807.208 8848.117 9640.68 9812.516 9848.719 9871.751 9422.431

Delta 25.08% 0.67% 10.44% 0.58% 0.30% 0.58% 8.29%

Theoritical 10000 10000 10000 10000 10000 10000 10000

0

2000

4000

6000

8000

10000

12000

Th

rou

gh

pu

t (M

bp

s)

NSH_SFC Proxy Outbound Throughput (RFC2544 vs. 100% TX Rate)

152B 256B 512B 1024B 1280B 1518B IMIX

NSH_SFC SFF RFC2544 4798.336 8623.209 8609.027 9739.464 9845.621 9869.394 8192.851

NSH_SFC SFF100% TX 5466.659 9205.381 9623.842 9808.213 9845.935 9869.751 9376.255

Delta 13.93% 6.75% 11.79% 0.71% 0.00% 0.00% 14.44%

Theoritical 10000 10000 10000 10000 10000 10000 10000

0

2000

4000

6000

8000

10000

12000

Th

rou

gh

pu

t (M

bp

s)

NSH_SFC SFF Throughput (RFC2544 vs. 100% TX Rate)

IMIX Profile for NSH_SFC

Packet Size(Bytes) 78 138 258 594 1518

Percentage 45.0% 23.5% 15.0% 11.0% 5.5%

NSH_SFC Performance Scaling and Performance Analysis

0

2

4

6

8

10

12

14

L2XC L2XC VxLAN IPFW VxLAN SFF NSH-aware SNAT

12.7

6.55.45

4.53.08T

hro

ug

hp

ut

(Mp

ps)

VPP and NSH_SFC Performance Comparision

(1C1T)

Source: VPP 17.01 CSIT Performance Report and NSH_SFC Performance Report

NSH ClassifierNSH Proxy

Inbound

NSH Proxy

Outound

NSH Proxy

BidirectionNSH SFF

1 core 5,232.27 5,404.78 4,785.75 4,835.17 5,504.20

2 cores 10,063.19 10,277.84 9,113.27 9,685.17 10,500.08

0.00

2,000.00

4,000.00

6,000.00

8,000.00

10,000.00

12,000.00

Th

rou

gh

pu

t (M

bp

s)

NSH_SFC Scaling Test Throughput

VPP & NSH_SFC Functionality Description

L2XC (64B) L2 Cross Connect

L2XC VxLAN (64B + 50B)

L2 Cross Connect + VxLAN Encap/Decap

SFF (128B)L2 Cross Connect + VxLAN-GPE

Encap/Decap + NSH manipulation

NSH-awareSNAT (128B)

L2 Cross Connect + VxLAN-GPE Encap/Decap + NSH manipulation + SNAT• 2 to 3 dedicated cores achieves 10G line

rate for smallest packet sizes

Test Case Packet size

NSH Classifier 72B

NSH Proxy Inbound 152B

NSH Proxy Outbound 128B

NSH SFF 152B

DUT2(VPP)

DUT1(VPP )

Traffic Gen

DUT(VPP + NSH_SFC)

Traffic Gen

End-to-End Throughput of NSH_SFC on Single Server

DUT

Niantic Port A Niantic Port B

VPP(L3 routing)

Vhost_user Port A

Container A

VPP(L2 xConnect)

Niantic PMD

Virtio_user

Traffic Gen

NianticPMD

DUT

Niantic Port A Niantic Port B

VPP(L3 routing)

Vhost_user Port A

Container A

VPP(L2 xConnect)

Niantic PMD

Virtio_user

Traffic Gen

Container B

VPP(L2 xConnect)

Niantic PMD

Virtio_user

Vhost_user Port B

Test NameVPP Functionality

(1C1T for each VPP instance)Packet Size (Byte) VPP-17.01

(Mpps)

CV L2XC + L3 routing 64 5.35

CVC L2XC + L3 routing + L2XC 64 3.59

CVC NSH-aware SNAT + SFF + NSH-aware SNAT 128 1.09

• CV setup • CVC setup

Function and Performance Automation Test

NSH_SFC Feature and Performance Test in CSIT Framework Example scripts to configure Service Function Forwarder on DUT1

/* Configure IP, routing and ARP tables */

set int state TenGigabitEthernet5/0/0 up

set int ip table TenGigabitEthernet5/0/0 0

set int ip address TenGigabitEthernet5/0/0 192.168.50.72/24

set int state TenGigabitEthernet5/0/1 up

set int ip table TenGigabitEthernet5/0/1 0

set int ip address TenGigabitEthernet5/0/1 192.168.50.73/24 ip route add 192.168.50.74/24 via 192.168.50.73

set ip arp TenGigabitEthernet5/0/1 192.168.50.74 02fe.8629.b438

/* Configure P2P VxLAN tunnels */

create vxlan-gpe tunnel local 192.168.50.72 remote 192.168.50.71 vni 9 next-nsh

encap-vrf-id 0 decap-vrf-id 0

create vxlan-gpe tunnel local 192.168.50.73 remote 192.168.50.74 vni 9 next-nsh

encap-vrf-id 0 decap-vrf-id 0

/* Configure NSH map and entry tables */

• create nsh entry nsp 185 nsi 255 md-type 1 c1 1 c2 2 c3 3 c4 4 next-ethernet

• create nsh entry nsp 185 nsi 254 md-type 1 c1 11 c2 12 c3 13 c4 14 next-ethernet

• create nsh map nsp 185 nsi 255 mapped-nsp 185 mapped-nsi 254 nsh_action swap

encap-vxlan-gpe-intf 4

Feature automation test enabled, performance automation test WIP

DUT1(Classifier/SFF/Proxy)

DUT2(L2Xconnect)

Traffic Gen(Trex)

192.168.50.73 192.168.50.74

VxLAN tunnel

192.168.50.71

192.168.50.72

OpenDaylight SFC Integration with VPP and NSH_SFC

16

Architecture: SFC VPP Renderer and Classifier in ODL SFC

SFC Provider

Data Store

SFC UI RESTCONF

OpenFlow Renderer VPP RendererSFC OvS*

Openflow PluginNetconfPlugin

OVSDB Plugin

Switch (OvS*)Netconf Device

(VPP node)Data Plane

Devices

ODL

SFC

SBI

Not used for VPP

Used for VPP

VPP Classifier

Legend

17

Architecture: OpenDaylight SFC and VPP/NSH_SFC Integration

VPP

OpenDaylight* SFC

Honeycomb

HC Core

VPP Renderer

Netconf/YANG

VPP Core

JVPP Core

Core Binary APIs

HC NSH Plugin

VxLAN-GPE port

configuration

NSH_SFC plugin

JVPP NSH Plugin

NSH Entry and Map Table

configuration

JVPP Registry

VPP ConnectionNSH Binary APIs

Data Broker

Config Data Tree Operational Data Tree

Translation Layer

JVPP APIs JVPP APIs

NSH-related

Contributions led by Intel

Legend

VPP Classifier

VxLAN-GPE

18

Summary and Future Plan

High performance NSH-based SFC solution within VPP and OpenDaylight

– NSH_SFC plugin to support Eth transport to eliminate per-hop encap/decap for SFC in NFV environment

– vHost_user performance needs optimization for East-West traffics

OpenDaylight SFC controls data planes mixed with VPP and DPDK OVS

Containerized SFC orchestrated by Kubernets and controlled by OpenDaylight

ONAP integration to provide orchestration support in SFC solution

– SDC supports SFC design

– MSO and Policy for SFC orchestration

– SDN-C and APP-C for data plane control

L4 packetinner

IP header

InnerMAC

header

NSH header,NP=0x3

Outer Ethernet,ET = 0x894F

Q&A

Summarized 1C1T Throughput of NSH_SFC 17.01

51205520

7088

9432

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

72 84 128 256

Th

rou

gh

pu

t (M

bp

s)

Packet Size (Bytes)

NSH Classifier Throughput for Different Packet Size

(1C1T)

7088

42504500

0

1000

2000

3000

4000

5000

6000

7000

8000

Th

rou

gh

pu

t (M

bp

s)

NSH Classifier NSH Proxy (inbound) SFF

NSH_SFC Throughput for 128B Packet

(1C1T)