When simplicity becomes complex

SURFnet7 indepth Wouter Huisman

On the road to a scalable and dynamic SURFnet7 network

What do we want from a network?

Scalable Flexible Cost efficient Endusers service requirements

Bottomline is keep it simple

SURFnet6 network

3

Mnster

Aachen

Hamburg

Geneva (CERN)

Dordrecht1

Breda1

Tilburg1

NLR

BT

Zutphen1

Lelystad1

Subnetwork 4: Blue Azur

Subnetwork 3: Red

Subnetwork 1: Green

Subnetwork 2: Dark blue

Subnetwork 5: Grey

Emmeloord

Zwolle1

Venlo1

LeeuwardenHarlingen

Den Helder

Alkmaar1

Haarlem1

Assen1

Meppel1

Emmen1

Arnhem

Apeldoorn1

Bergen-op-ZoomZierikzeeMiddelburg

Vlissingen Krabbendijke

Breukelen1

Ede

Heerlen2

DLO

Schiphol-Rijk

Hilversum1

Hoogeveen1

Dwingeloo1

Den Bosch1

Nieuwegein1

Heerlen1

Nijmegen1Rotterdam4

3XLSOP

Winschoten1Middenmeer1

NNIInHolland

SchipholOegstgeest

Lisse

ZoetermeerLSOP Rtd

SWOVNBD

DenHaag

Rotterdam1

Leiden1

Roermond1

Sittard1

Venlo2

Maastricht7

Muenster1

Almere1

TNO Soesterberg

Groningen1

Hamburg

Aaken

Maastricht1

Beilen1

Amersfoort1

Eindhoven1

ROADM

ROADM

ROADM

Maasbracht1

Amsterdam1

DAS3

NLRLelystad2

Wageningen1

Enschede1

Amsterdam2

Brussel

Paris

Geneva

Collapsed IP core

SURFnet SNE masters BGP in de praktijk

SURFnet AS1103

Customer

Border Routers

Core Routers

Located in Ams1 and Ams2

Transport layer

Juniper T-series: 4

Ciena OM5200: 20

Nortel MERS 8600: 20

Ciena OME6500: 100

Ciena CPL: 100

Nortel OME1010/1060: 350

What to optimize of SURFnet6?

And make SURFnet7 simple

Institute

Optisch Transport (Layer 0/1)

Next Generation Ethernet

(Layer 2)

Routed IP (Layer 3)

Institute

Bandwidth

Service

Bandwidth

Service

Applica

Concepts in the SURFnet7 tender

Single technology end-2-end transport Remove as many OEO as possible Introduce 40G and 100G Improve the BoD solution

Selected Ciena CES solution 5410, 3930, 5150 Based on PBB-TE

Ciena CESD solution

Ciena 5410

Core switch, 2 per main POP Modular switch with 400Gb/s per slot

32-port 1G 4-port 10G 10-port10G 40-port 10G/48-port 1G 4-port 100G/2-port 40G

Pluggable optics 1G SX, LX, CWDM, DWDM 10G SR, LR, CWDM, DWDM 40G LR4 CFP 100G LR4 CFP

Ciena 5150

Pizza box with extension slots 2U height 48x 1G SFP 4x 10G XFP Dual AC/DC power

Ciena 3930

Pizza box 1U height 8x 1G SFP 2x 1G/10G SFP+ Dual AC/DC power

Tiered Network Architecture

Tiered Network Architecture

Tiered Network Architecture

Tiered Network Architecture

Tiered Network Architecture

Tunable XFP & OTN XFP

CPL DWDM OADM 5410 - CES

Transponder

No transponder shelves required Tunable DWDM XFP used Save on power, footprint, OAM ~25% of the links require G.709

Scalability issue

Uptake lightpath services was significant

Technology limitation Bandwidth claimed for

100%, even though not all used.

Single service per port

!

!

a

b

c

26

a

b

c

a b c

a,b,c

EIR

CIR

Statistical Multiplexing

QoS

2 Color marking (Green and Yellow) 8 levels with .1p priority

Management mapped to 7 Lightpath mapped to 5 IP services mapped to 3

Strict queueing, which guarantees that Lightpaths suffer last from congestion

PBB-TE Encapsulation

28

DA SA

Payload

DA SA

Payload

VID

DA SA

Payload

S-VID C-VID

DA SA

Payload

S-VID C-VID

B-DA B-SA B-VID I-SID 802.1 802.1q

802.1ad

802.1ah

29

PBB-TE Protection

Connection Oriented Ethernet Primary & Backup Tunnels Monitored via 802.1ag CCMs CCMs configurable for 10 msec intervals

Ingress PBB-TE Edge Bridge

Egress PBB-TE Edge Bridge

Primary Tunnel

Protected Node

MEP A MEP B

MEP C MEP D

Node config of a tunnel, transits, services Concept VS

Creating a tunnel

A

Z B

C

Endnode

Transit node

Transit node

virtual-switch create vs TRANSIT-1334!pbt transit create pbt-transit TRANSIT-1334_503 parent-port 5/3!pbt transit add pbt-transit TRANSIT-1334_503 class-element 1 bvid 1334!virtual-switch interface attach pbt-transit TRANSIT-1334_503 vs TRANSIT-1334!pbt transit create pbt-transit TRANSIT-1334_101 parent-port 1/1!pbt transit add pbt-transit TRANSIT-1334_101 class-element 1 bvid 1334!virtual-switch interface attach pbt-transit TRANSIT-1334_101 vs TRANSIT-1334!

Startnode

vlan create vlan 1337 name TRANSIT-1337!vlan add vlan 1337 port 3.1!vlan add vlan 1337 port to_Asd001A_5410_01T_l1!

Creating a tunnel

A

Z B

C

Startnode Transit node

Transit node

pbt tunnel-group create group DH5150_5410s tunnel-sync on!pbt encap-tunnel create static-encap P_DH5150_5410s tunnel-group DH5150_5410s pair-index 4 port to_Asd001A_5150_02T_l1 bvid 1333 dest-bridge-name Asd001A_5150_02 weight 6!pbt decap-tunnel create static-decap P_DH5150_5410s tunnel-group DH5150_5410s pair-index 4 port to_Asd001A_5150_02T_l1 bvid 1333 src-bridge-name Asd001A_5150_02!cfm service create static-encap P_DH5150_5410s name PBT-1333 next-mepid 1333 level 2!cfm service set service PBT-1333 ccm-interval 100ms alarm-time 0!cfm service enable service PBT-1333!

Endnode

Creating a service

A

Z B

C

Startnode Endnode

virtual-circuit pbt create static-vc 3669IP tunnel 1Hedr1_Asd_D egress-isid 73860 ingress-isid 73860!virtual-circuit pbt set static-vc 3669IP retain-stag yes!virtual-switch add reserved-vlan 150!virtual-switch ethernet create vs 3669IP vc 3669IP reserved-vlan 150!virtual-switch ethernet set vs 3669IP description "3669IP"!virtual-switch ethernet set vs 3669IP encap-cos-policy fixed encap-fixed-dot1dpri 3!! MTU size increased to 9190 for CN 5150 Hedr001A_5150_01 port 1.1!port set port 1.1 max-frame-size 9190!port set port 1.1 vs-ingress-filter off!virtual-switch ethernet add vs 3669IP port 1.1!port set port 1.1 acceptable-frame-type all untagged-data-vs 3669IP!aggregation set port 1.1 agg-mode manual!lldp set port 1.1 mode rx-only notification off!virtual-switch ethernet set port 1.1 vs 3669IP encap-cos-policy fixed encap-fixed-dot1dpri 3!traffic-profiling set port 1.1 mode standard-dot1dpri!traffic-profiling standard-profile create port 1.1 profile 1 name 3669IP01 cir 0 cbs 16 pir 1000000 ebs 64!traffic-profiling set port 1.1 nonconform-standard-profile 3669IP01!traffic-profiling enable port 1.1!cfm service create vs 3669IP name 3669IP next 1 level 4!cfm service set service 3669IP alarm-priority 3!cfm service set service 3669IP alarm-time 10000!cfm service set service 3669IP reset-time 3000!cfm service set service 3669IP ccm-interval 1s!cfm service enable service 3669IP!! CN 5150 Hedr001A_5150_01 port 1.1 forced to Enabled!port enable port 1.1!

virtual-circuit pbt create static-vc 2013LP tunnel 0809prot egress-isid 1073041 ingress-isid 1073041 virtual-circuit pbt set static-vc 2013LP retain-stag yes virtual-switch add reserved-vlan 151 virtual-switch ethernet create vs 2013LP vc 2013LP reserved-vlan 151 virtual-switch ethernet set vs 2013LP description "2013LP" virtual-switch ethernet set vs 2013LP encap-cos-policy fixed encap-fixed-dot1dpri 5 virtual-switch l2-cft set vs 2013LP tunnel-method transparent virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol 802.1x disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lacp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lacp-marker disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lldp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol oam disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol rstp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-cdp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-dtp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-pagp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-pvst disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-stp-uplink-fast disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-udld disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-vtp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol gvrp disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol vlan-bridge disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol all-bridges-block disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol bridge-block disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol garp-block disposition forward virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol gmrp disposition forward virtual-switch l2-cft enable vs 2013LP port set port 1 vs-ingress-filter off virtual-switch ethernet add vs 2013LP port 1 port set port 1 acceptable-frame-type all untagged-data-vs 2013LP rstp disable port 1 mstp disable port 1 aggregation set port 1 agg-mode manual lldp set port 1 mode disable notification off port set port 1 untagged-ctrl-vs 2013LP virtual-switch ethernet set port 1 vs 2013LP encap-cos-policy fixed encap-fixed-dot1dpri 5 traffic-profiling set port 1 mode standard-dot1dpri traffic-profiling standard-profile create port 1 profile 1 name 2013LP01 cir 0 cbs 16 pir 1000 ebs 64 traffic-profiling set port 1 nonconform-standard-profile 2013LP01 traffic-profiling enable port 1 cfm service create vs 2013LP name 2013LP next 2 level 4 cfm service set service 2013LP alarm-priority 3 cfm service set service 2013LP alarm-time 10000 cfm service set service 2013LP reset-time 3000 cfm service set service 2013LP ccm-interval 1s cfm service set service 2013LP dmm-interval 1s cfm delay send service 2013LP local-mepid 2 mepid 1 repeat 1 cfm service set service 2013LP lmm-interval 1s cfm frame-loss send service 2013LP local-mepid 2 mepid 1 repeat 1 cfm service enable service 2013LP ! CN 3930 Asd001A_3930-08T port 1 forced to Enabled port enable port 1

Service CFM

A

Z B

C

Startnode

Endnode

MEP MEP

Service CFM is added to each service - Run from client sub port to client sub port - Takes the same path as customer data - Set at 1s interval, to allow hitless tunnel protec

Monthly Reporting

Availability is based on Service CFM alarms Time between raise and clear of the alarm is the down time of

the service

Volume reporting is based on the standard MIB entries Per port and per service (subservice of a physical port) is

possible

Considering reporting of latency and jitter

Can we manage elephant flows?

Unpredictability of the dynamic nature of BoD BoD tend to generate large flows (> 1Gb/s) Can we rely on the NMS?

BW enforcement for tunnels and services

Layered structure of tunnels and services Each tunnel has a bandwidth profile Services are added to tunnels as long as tunnel

can accommodate

Link capacity Tunnel profile Service BW

A B

Tunnel CIR: 2 node network

Consider: 10G Link between A and B Filled with single tunnel and consuming 50% of CIR bandwidth 5G SLP is to be reserved for new install Tunnel profile is changed to claim 100% CIR bandwidth In mean

Solution

DLP can only be transported across DLP tunnels Claim Bandwidth in advance

Create dummy tunnels with required CIR for new services

Or increase the CIR bandwidth of existing SLP tunnels

2 elevator dilemma

Assume 2 elevators, both with capacity for exact 10 people

2 elevator dilemma

Assume 2 elevators, both with capacity for exact 10 people

Drop at least 1 person

LAG problem

Consider Lightpath flows of 6 Gb/s and of 5 Gb/s A CES will autonomous decide, which LAG member to use A LAG of 2 x 10G should be sufficient But you have 50% chance on packet drop

Problem with LAGs with flows > 10% physical link speed

Solutions to the LAG issue

Use higher line rate instead of LAGs Use 100G

Use LAGs, but be conscious not to put high bandwidth flows on a LAG as you are loosing control

NMS implements Strict & Loose CAC flag

Strict CAC LAG LAG with 1 member Can accommodate any services Loose CAC LAG LAG with possibly more link members Accommodates Lightpaths upto 1Gb/s And IP

W

wouter.huisman[at]surfnet.nl www.surfnet.nl +31 30 2 305 305 Creative Commons Attribution license: http://creativecommons.org/licenses/by/3.0/