SDN Practice for Cloud Overlay Network:

From Infrastructure to ProductsGang Cheng

Agenda

Introduction

1

SDN Overlay Network

2

SDN Products

3

Summary

4

About Alibaba Cloud

World Leading Public

Cloud Service Provider

47.6% Market share

104% Revenue growth

Our Datacenters

US West

US East

Germany

UAE

Singapore

Australia

Japan

MalysiaIndonesia

Hong KongIndia

China Mainland

Some Numbers

325K 162.8BPayments/s RMB256kOrders/s

Powered By Alibaba Cloud

The Network Behind the Scene

Physical Network

Underlay SDN Controller

Overlay Controller Virtual

SwitchGW SLB

CEN (Cloud

Enterprise Network)FuxiApplications

Unified API

…

Overlay Network

Applications such as cloud resource management,

obtain the network resources by calling the

north bound API of overlay network controller.

Applications

The overlay network of Alibaba Cloud is a SDN

network consisting of control plane, data

plane, and management plane.

Overlay Network

Alibaba physical network is one of the world

largest SDN network infrastructure.

Physical Network

Agenda

SDN Overlay Network

2

Introduction

1

SDN Products

3

Summary

4

Architecture Overview

Region/IDC

Virtual Switch

Gateway SLB Hybrid Cloud Gateway

Data plane

Regional SDN Controller

Global SDN Controller

Management

Plane

Various data plane acceleration technologies

are intensively used in Alibaba Cloud Overlay

network, e.g., user mode networking stack,

programmable white box switch.

Data plane

The SDN controller of Alibaba Cloud Overlay

network consists of three parts: SDN host

controller agent, regional controller and

global controller.

Control Plane

The management plane driven by machine

learning & big data technologies empower the

automation of SDN overlay network.

Management Plane

SDN Host Controller Agent

Apsara Virtual Switch (AVS)

Route

Apsara Network BaseOS

TAP QOS ACL Meter NAT

VM

Fastpath

Container Bare Metal Allow rich network functionalities and achieve

high performance

Separated slow path & fast path

Business & platform abstraction

enables AVS to run on different platform and

support different applications

Business & Platform Abstraction

Data plane acceleration with kernel-bypass

technology/user mode network stack, SR-IOV, or

SmartNiC

High Performance

SDN Regional Controller

SDN Controller

Configuration

Cache

AVS

VM VM VM

…

Host

Controller Agent

AVS

VM VM VM

…

Host

Controller Agent

SDN Controller

Configuration

AVS

VM VM VM

…

Host

Controller Agent

Self-learning

AVS

VM VM VM

…

Host

Controller Agent

• Supporting up to 1M virtual networks in a region (each virtual network with maximally 100K VMs) places great pressure on

the SDN controller performance

High Availability

Controller

GW Gateway/SLB

…

GW Gateway/SLB

…

Controller

GW Gateway/SLB

…

AZ 1 AZ 2 AZ 3

Region

• Redundancy on both control plane and data plane

Intelligent Management

Overlay SDN Controller

GW

Data plane

Underlay SDN Controller

VS

Log, Tracing… Big Data Analysis

Machine Learning

Dev/Ops

Tracing and logs from both underlay and overlay

networks are collected and fed into big data

analysis service.

Data Collecting

The big data analysis ensures log data can be

processed in a real time fashion so that

failures can be identified quickly enough.

Big Data Analysis

Machine learning helps reduce the recovery

time and reduce the human intervention.

Machine Learning

Performance

Virtual Switch

• High throughput

• Ultra low latency

• Hot upgrade

• Hot migration

Gateway

• Scale out

• Hot upgrade

• 5m connections/s

• 30M PPS

Controller

• 1M VPCs per Region

• 100K VMs per VPC

• 100K routes in 3 seconds

The Benefits from SDN Approach

scalability performance convenience reliability intelligence

Agenda

SDN Products

3

SDN Overlay Network

2

Summary

4

Introduction

1

Cloud-Network Convergence

洛神Cloud

Internet

IDC

Overlay Network Evolution

Classic Network VPC VPC

VPC Peering

VPC

VPC

VPC

VPC

VPC

Shanghai US West

GermanDubai

Japan

The First Generation

The first generation network, only

provides public internet access

capability.

VPC Network

Besides the functionalities provided the

first generation, the second generation

network provides point-to-point

connection between the VPCs.

CEN

The Third Generation

The third generation cloud overlay

network has the intelligence and

capability to provide global

connectivity.

Cloud Enterprise Network (CEN)

VPC

VPC

VPC

Shanghai US West

GermanDubai

Japan

CENVR

VS VS VS…

VPC1

VM VM VM VM VM

…

VR

VS VS VS…

VPC2

VM VM VM VM VM

…

VBRRegion A

VR

VS VS VS…

VPC3

VM VM VM VM VM

…

VBR Region B

CEN: a fully meshed backbone network

The Challenges

VPCVM

VMVPC

VM

VM

Region A Region B

GW GW

IDC

Cloud Services

Cross Region VPC Peering

SLB

The explosive business growth of Alibaba Cloud calls a solution with higher performance, better scalability,

and less cost.

Leased line access

CEN Architecture

GW VS

Underlay Controller

Overlay Controller

Regional Route Controller

Manage

ment

Leased line access point Overlay Network

A Cloud Based SD-WAN Network Solution

Global Route Controller

An Use Case of Programmable SDN Switch

GW

VM Docker

Bare Metal

VPC

Controller

GW

VM Docker

Bare Metal

VPC

Regional Route Compute & Controller

Overlay Network Controller

Control Plane

NFV

Apsara vswitch

virtio-net

queue queue queue queue queue

Virtio polling mode driver

NFV application

VM

DPDK

Agenda

Summary

4

SDN Products

3

SDN Overlay Network

2

Introduction

1

Our SDN Overlay Network

SDN Overlay Network

• Flexible

• Automated management

• Highly available

• Quick iteration

Rich Functionalities

• Xen， KVM，Docker…

• Hybrid cloud

• SD-WAN

Security

• Strong anti-DDoS capability

• Tenant isolation

Our Experience

Future，Globalization，More Intelligent

Thank You