0

SONA: ONOS SDN Controller based

OpenStack/Kubernetes Network Management Solution

Trellis: Multi-Purpose Leaf-Spine Fabric Solution

Sangho Shin

Feb 22, 2019

1

Introduction to SDN & ONOS

2

Software-Defined Network (SDN) (1/2)

Specialized Packet For

warding Hardware

App App App Specialized Packet For

warding Hardware

App App App

Specialized Packet For

warding Hardware

App App App

Specialized Packet For

warding Hardware

Operating

System

Operating

System

Operating

System

Operating

System

App App App

Other aspects of SDN follow

Control

Data

1

1

1

n

m

n

Specialized Packet For

warding Hardware

App App App

Operating

System

3

Software-Defined Network (SDN) (2/2)

Specialized Packet

Forwarding Hardware

App App App

Specialized Packet

Forwarding Hardware

App App App

Specialized Packet

Forwarding Hardware

App App App

Specialized Packet

Forwarding Hardware

App App App

Specialized Packet

Forwarding Hardware

Operating

System

Operating

System Operating

System

Operating

System

Operating

System

App App App

Network Operating System (Controller)

App App App

4

SDN Evolution and ONF

Platform

Development

2007 – Ethane

2008 – OpenFlow

2009 – FlowVisor,

Mininet, NOX

2010 – Beacon

2009 – Stanford

2010 – GENI started

and grew to 20

universities

2013 – 20 more cam

puses to be added

Deployments

Demonstrations

2008-2011 – SIGCOMM

2011 – Open Networking

Summit, Interop

2012 –Define SDN

research agenda

for the coming

years

And Beyond

Invention

2007 – Creation

of SDN Concept

2017 – ON.Lab and

ONF merger

Non-profit, carrier and

vendor neutral

Provide technical shepherding,

core team

Build community

Many organizations supports

5

Northbound(policy enforcement, conflict resolution)

OpenFlow NetConf . . .

Applications

Distributed Core(scalability, availability, performance, persistence)

Southbound(discover, observe, program, configure)

Provider Provider . . .

Contains user applications

E.g., reactive forwarding, ProxyARP,

segment routing, SDN-IP, etc.

Transfer network info to app layer

Provide management interface for

controlling lower layer component

Contains many core features

Provide distributed clustering func.

for supporting HA and scalability

Provide an abstracted interface for

controlling the network infrastructure

Network protocol implementation

for managing network elements

E.g., OpenFlow, NetConf

ONOS Architecture (1/2)

6

NB Core API

Distributed Core

(state management, notifications, high-availability & scale-out)

SB Core API

Protocols

Providers

Protocols

Providers

Protocols

Providers

Protocols

Providers

AppsApplications

ONOS Architecture (2/2)

7

Device Link Host

Topology

Flow Rule

Path

Packet

StatisticsIntent

Application

Leadership

Messaging

Storage Region

Mastership

Driver

Group

Security

Flow Objective

Event

OpenFlow NetConf OVSDB

Core Cluster

. . .

Proxy ARPSONA L2 Forwarding

REST API GUI CLI

Network Cfg.

SDN IP / BGP DHCP

Tunnel

. . .

OSGi / Apache Karaf

Network Virt.Device Cfg.

Config

UI Extension

Off-platform Apps

Graph

Discovery Tenant . . .

OSGi Framework

Non-networking core subsystem

South Bound

Interface module

Networking core subsystem

On-platform Applications

On-platform Application Interfaces

Off-platform Applications

ONOS Subsystems (Services)

8

ONOS Project at ONF

Mobile

ROADM

(Core)

PON

OLTs

PON

OLTs

ResidentialEnterprise

Shared Cloud Infrastructure

VOLTHA ODTNStratum

Trellis

XOS

ONOS

Residential

Services

Mobile

Services

Enterprise

Services

9

ONOS Community

Partners Collaborators

10

Q4/14 AvocetBase Architecture

Q1/15 BlackbirdPerformance

Q2/15 CardinalONS Use Cases

SDN-IP

Packet Optical

R-CORD

Q3/15 DrakeONF ATRIUM

Secure Mode ONOS

VxLAN

Device Configuration

Q4/15 EmuOPNFV

SONAAARNET

KREONET-S

Q1/16 FalconONS Use Cases

{A, E, M} CORD

Disaggregated ROADM

Global R&E Deployment

Q2/16 GoldeneyeCPMan Apps

Intents using Flow Objectives

P4 DEMO support

YANG tool chain

Q3/16 HummingbirdRabbitMQ, Kafka Message

YANG NBI, SBI CODECs

ACTN Traffic Engineering

Q4/16 IbisBUCK Build Tool

Trellis Fabric enhancement

LISP SBI support, REST Client,

FatTree simulator

Q1/17 JuncoTL1 SBI support

Virtualization support

Regionalization support

Dynamic conf. enhancement

Q2/17 KingfisherYANG Tools 2.0

OpenFlow 1.4 support

Intent F/W improment

vRouter, OpenROADM support

Q3/17 LoonOpenFlow 1.5 SBI

gRPC NBI support

P4 runtime initial support

Q4/17 MagpieTopo2 initial support

More switch driver

support

Q1/18 NightingaleISSU initial support

Trellis enhancement (T3)

P4 support enhancment

ONOS Release History

Q2/18 OwlComing soon…

11

Introduction to Trellis

12

Multi-purpose leaf-spinefabric designed for NFV

Bare-metal hardwareOpen-source software

SDN-based (built on ONOS)

Trellis Overview

13

● Bridging with Access & Trunk VLANs (within a rack)

● Routing (inter-rack)

○ IPv4 & IPv6 Unicast routing with MPLS Segment-Routing

○ IPv4 & IPv6 Multicast routing

● Dual-homing for compute-nodes and external routers

● Multi-stage fabrics (2 layers of spines)

● vRouter - entire fabric behaves as a single router

○ BGP (v4/v6) support for external (upstream) connectivity

○ Static routes, route blackholing

○ DHCP L3 relay (IPv4/v6)

● MPLS Pseudowires

● QinQ termination

● T3 - Trellis Troubleshooting Tool

● ASIC Support

○ Broadcom Qumran, Tomahawk, Trident2 switches from EdgeCore & QCT

○ Preliminary support for Cavium Xpliant switches and P4-based Tofino switches

Trellis Features

14

White Box SwitchEdgeCore 5712,5912

QCT LY8

Leaf Switch

48 x 10G, 6 x 40G/100G

GE mgmt.

White Box SwitchEdgeCore 6712,7712

Spine Switch

32 x 40G/100G ports downlink to leaf switches

GE mgmt.

BRCM ASIC

OF-DPA

Indigo OF Agent

OpenFlow 1.3

OCP: Open Compute ProjectONL: Open Network LinuxONIE: Open Network Install EnvironmentBRCM: Broadcom Merchant Silicon ASICs OF-DPA: OpenFlow Datapath Abstraction

Leaf/Spine Switch Software Stack

to controller

OCPSoftware

(ONL,ONIE)

OCP Bare Metal Hardware

White-Box = Bare-metal hw + Open-Source sw

Trident2, Tomahawk, Qumran

15

Ingress

Port Ta

ble

Phy

Por

t

Vlan T

able

Termin-

ation M

AC

Table

Multi-

cast Ro

uting Ta

ble

Unicast

Routing

Table

MPLS

Table

Bridging

Table

ACL

Policy

Table

L2 Floo

d

Group

L3 ECM

P

Group

Phy

Port

Phy

Port

Phy

Port

Phy

Port

Phy

Port

MPLSLabelGroup

MPLSLabelGroup

L3

Mcast

Group

L2 Interface

Group

L2 Interface

Group

Fabric ASIC Pipeline* (BRCM’s OF-DPA)

Vlan 1

Table

MPLS

L2 Por

t

Table

* Simplified view

Abstracts underlying ASICEnables programming of allflow-tables & port-groups

Why OF-DPA?

L2 Interface

Group

Phy

Port

L2 Interface

Group

15

16

16

ONOS Cluster

P4Runtime gNMI

Barefoot Tofino

Mellanox

Cavium Xpliant

fabric.p4 driver

Trellis & P4

OF-DPA driver

Brcm Qumran

Brcm Trident2

Brcm Tomahawk

OpenFlow NetConf

Segment Routing DHCP L3 Relay vRouter Multicast SPGW-app

Same set of Trellis applications on ONOS

P4 capable hardware

Allowing new functionality on hardware (demo at MWC ‘18)

P4

Enhanced with P4 program deployment and pipeline configuration

1717

Trellis @ Comcast

18

Introduction to SONA

19

Why SONA?

Compute Node

nova-compute

neutron-plugin-agent

Compute Node

nova-compute

neutron-plugin-agent

Compute Node

nova-compute

neutron-plugin-agent

Control Node

AMQP

nova-api

keystone

nova-scheduler

neutron-server

horizon

Network Node

neutron-L3-agent

neutron-*plugin-agent

neutron-DHCP-agent

neutron-metadata-agent

Compute Node

nova-compute

neutron-plugin-agent

Management Network

Data Network

External Network

• Limitation of Neutron network

– Limited visibility of VM traffic

– Limited scalability of network node

20

SONA (Simplified Overlay Networking Architecture)

• SONA: Overlay Network Management Solution for SDDC

– ONOS based Virtual Network Management solution (support VxLAN, VLAN, FLAT)

– Empowered by SDN controller, a better replacement of neutron, scalable gateway

– Fully compatible with OpenStack (mitaka, newton, ocata, pike, queens)

Highly

scalable

Better VM - VM Traffic

visibility at Control Plane

21

SONA (Simplified Overlay Networking Architecture)

• Integration with OpenStack

– OpenStack neutron

• Plugin: modular layer 2 plugin

– networking-onos

• ONOS L3 plugin

• Drivers for LBaaS, FWaaS, etc.

– SONA

• Northbound interacts with networking-onos

– https://github.com/openstack/networking-onos

• Southbound protocol

– OpenFlow: install/uninstall flow rules

– OVSDB: configure OpenvSwitch

» Add/delete virtual port

» Create/delete bridges (e.g., br-int, etc.)

Neutron

ONOS

ML2 Plugin

ONOS

Mech Driver

ONOS

L3/LBaaS/

Plugins/

Drivers

SONA Northbound

SONA Network/Rule Services

OpenFlow OVSDB

VM VM

ML2

DB

①

② ③

22

SONA Features

Direct communication

23

Scalable Gateway

SONA Features

24

SONA Features

UI based Flow Tracer

25

• SONA Fabric

– Pure OpenFlow based Leaf-Spine Fabric Solution

– Supports ECMP, Failure detection & auto recovery

– Physical + Virtual Network Integration

SONA Features

26

SONA Features

• vFlow Statistics

– Collect VM to VM real-time flow statistic

– Stats collection is realized using OpenFlow

standards protocol (no extra overhead!)

– Seamless integration with monitoring

systems through various NBIs

• REST, Kafka, gRPC, influxDB, etc.

– Realized through OpenstackTelemetry app

– No additional software installations are

required at OpenStack side

– No additional hardware installations are

required at compute/control node

– Open source!

SDN Controller

OpenstackTelemetry …

REST Kafka gRPC influxDB …

VM VM

OpenFlow

OVS

OVSOVS

OVS

27

SONA Features

• vTap

– Mirror VM to VM real-time traffic

– Leverage OVS’s traffic mirroring feature

– Two traffic mirroring schemes

• Port-based: specific to OVS

• Flow-based: uses OpenFlow group table

– Realized through OpenstackTelemetry app

– No additional software installations are

required at OpenStack side

– Further improve the mirroring performance by

leveraging data plane acceleration technology

– Open source!

OVS

VM VM

eth0

OVS

vDPI

eth0

SDN Controller

OpenstackvTap …

28

VM

SONA

ARP Request

VM

ARP Response

VM

SONA

ARP Request

VM

ARP Response

VM

SONA

ARP Request

VM

ARP Response

Crash

VM

SONA

ARP Request

VM

ARP Response

SONA Failover

Proxy Mode

Broadcast Mode

29

Spine

Leaf...

VM VM VM

VM VM VM

VM VM

VM VM

VM

VM

VM VM VM

VM VM VM

Baremetal

Baremetal

Tenant A Tenant B Tenant C Tenant D

SONA Fabric

Kuryr-

kubernetesNeutron

OVS

Kuryr CNI

Pods

VMVMVMContainerContainerContainer

SONA Features

• Kubernetes Support

30

OpenvSwitch

How does SONA Process Packets?

• SONA Pipeline

DHCP & ARP (vNet)

(table = 1)

FLAT

(table = 2)

vTAG

(table = 10)

ACL

(table = 20)

ConnTrack

(table = 21)

Jump

(table = 30)

Routing

(table = 40)

Switching

(table = 50)

Ingress

Port

Egress

Port

InboundStat

(table = 0)

OutboundStat

(table = 49)

31

SONA CI/CD

• Continuous Integration (CI)

– Fetch latest SONA source

– Build against stable ONOS

– Run unit test

– Package & deploy SONA

– Run integration test

– Notify the CI result via slack

– Deliver SONA container

https://hub.docker.com/r/opensona/onos-sona-nightly-docker/

Jenkins

Machine

ONOS Build &

Run Machine

OpenStack

Control Node

Tempest

Machine

OpenStack

Compute Node

OpenStack

Compute Node

Gateway

Node…

32

SONA CI/CD

• Integration Test

– Initialize environment to spawn ONOS cluster

– Tempest basic test

• OpenStack API test

• OpenStack scenario test

– ONOS failure test

• Terminate ONOS nodes, run tempest

– SONA app failure test

33

Open Source Contribution

• Open Source Strategy

– 100% open source

– 136 commits were upstreamed in 2018 (2018.01 ~ now)

• https://gerrit.onosproject.org/#/q/project:onos+branch:master+topic:sona

• Helps from Community

– More tests and feedback from community

– Code contributions are always welcomed :)

• Wiki

– https://wiki.onosproject.org/display/ONOS/SONA%3A+DC+Network+Virtualization

• Slack Channel

– #sonaproject @ onosproject.slack.com