1

IETF 74, San FranciscoIAB Plenary

March 18, 2009

Network Operator Perspective

MPLS: 12 Years AfterTom Bechly

2

MPLS: A Successful Protocol

• MPLS has been and is a successful protocol– From perspective of RFC 5218 (What Makes for a Successful Protocol?),

MPLS was used for its intended purpose and at intended scale• Goal was to switch packets to support rapidly expanding global

networks– MPLS is “wildly successful” (RFC 5218) in that its use has exceeded its

original design goal thru development of numerous extensions– From service provider perspective MPLS was successful in supporting

growth, reducing cost, and providing basis for new services

• Original goal of bringing Layer 2 switching speed to Layer 3 was accomplished, but somewhat discounted over time due to hardware evolution– L2 was hardware switched and L3 was process switched

• MPLS was easily leveraged for traffic engineering, VPNs, and layer 2 transport.

• For the service provider, MPLS has become one the most reached for and extended tools in the tool chest (150+ RFCs)

3

MPLS

• Enables network edge routers to apply simple MPLS labels to packets or frames

• Forwards packets by swapping labels with minimal lookup

• Integrates Layer 2 switching and Layer 3 routing

Customer Edge (CE) Router

Provider Edge (PE) Router/Switch

Provider Core Router/Switch

MPLS COREMPLS CORECE Router

CE Router

CE Router

CE Router

PE RouterPE Router

P RouterP Router

PE RouterPE Router

PE RouterPE RouterPE RouterPE RouterP RouterP Router P RouterP Router

4

MPLS/RSVP-TE Benefits

• MPLS with RSVP-TE provides overall path control in network– Use with constraint based routing– Control over latency and delay variation– Bridges gap between ability to deploy capacity versus current demand in

existing network

• Use of MPLS allowed gathering measurement statistics on LSPs– Probably more important than actual path control– Provides ability to accurately measure traffic between router pairs

• Traffic volumes, latency, and delay variation– Measure traffic between hubs, metros, and regions– Measure asymmetry of flows, over time– A time series depiction can be built to trend traffic for efficient investment

and to provide required service

• MPLS became an enabler for the development of additional services– L2 VPNs and L3 VPNs

5

Verizon Public IP

• AS 701 was initially implemented as an overlay over a dedicated frame relay network– Path control was effected thru manipulating path of frame relay PVCs

• As capacity requirements increased, the network was migrated to an overlay over ATM– The cost of this became untenable, as capacity requirements continued

to increase

• MPLS with RSVP-TE deployed in EMEA (AS 702) in 1999– First deployment of RSVP-TE in production network– Deployed in US (AS 701) in 2000

• Deployed for traffic engineering to provide control over path selection that was not available thru L3 protocols– Shortest path algorithm did not always provide optimal route

• MPLS technology has enabled the Verizon Public IP network to grow to be one of the largest in the world

6

Verizon IP Network

– 410 unique switch/router hubs (PoPs)– Six continents, 150+ countries

7

Verizon Layer3 VPN Services: VBNS+ and Private IP

• vBNS (very-high-performance Backbone Network Service) was established in 1995– Cooperative research and development agreement between Verizon

(formerly MCI) and National Science Foundation (follow on to NSFnet)– Evolved to a commercial product: vBNS+ for gov and edu market

• MPLS routing/switching implemented in network in 1999– Initially MPLS was implemented for traffic engineering

• L3VPN (RFC 2547) was implemented in 2001– There are approximately 40 nodes in 19 US cities, full mesh of TE LSPs

• Verizon PIP (Private IP) was established in 1999– Layer 3 VPN (RFC 4364), wide area network for business customers– Quality of Service, strong SLAs, etc.

• Large global network– There are approximately 625 nodes across 162 cities in 59 countries

• Uses LDP for label distribution, with partial mesh of LSPs

8

Private IP Global Reach

MP10163v5.03

9

Verizon Layer 2 Services: MAE® Services and Converged Packet Architecture (CPA)

• MAE® Services established 1992 as metro Internet Exchange point• Evolved into MPLS based national service for extended peering and

L2 VPNs (VPWS), implemented in 2002– Service interworking (ATM, Frame Relay, and Ethernet), based on draft

Martini pseudowires and draft Shah ARP Mediation

• Implemented across public internet within full mesh of GRE tunnels– ISIS, RSVP-TE signaled LSPs, and LDP signaled pseudowires

• CPA supports Ethernet access and Ethernet services– L2 VPNs: both EVPL (PWE3) and VPLS (RFC 4762)– Quality of Service, strong SLAs, etc.

• Large global network– There are approximately 115 nodes across 27 countries

• RSVP-TE used to signal LSPs– Full mesh for EVPL and VPLS– Currently 10,000+ LSPs

10

Lessons Learned

• Implementation defects significantly impact early perception of technology– For AS 701, there was internal resistance to moving from ATM underlay

network to MPLS– When defects in the MPLS implementation on vendor equipment were

encountered these initially viewed by some as defects in the technology

11

Lessons not Learned (VPLS)

• RFC 4762: Virtual Private LAN Service (VPLS) Using Label Distribution Protocol (LDP) Signaling– Hierarchy is managed thru HVPLS, specified within RFC

• RFC 4761: Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling– Hierarchy is managed thru route reflectors and multi-segment

pseudowires

• Both approaches are currently in production in different service provider networks

• Some vendors have implemented both standards• This increases to overall cost and complexity of technology and

network development across the community– Resolution and mitigation of differences is far more economic during

protocol development than once into implementation– Gateway function has high development and operational cost– The added costs and complexity are continuously accretive

12

Lessons not Learned (RFC5085 – PW VCCV)

• Pseudowire Virtual Circuit Connectivity Verification (VCCV) – RFC 5085– Three modes of operation: (Type 1: PWE3 Control Word Bit, Type 2: MPLS

Router Alert Label, Type 3: MPLS PW Label with TTL == 1– Mode is negotiated, so all three are optional

• Vendors, to this point, have not implemented all modes nor the same modes

• This leads to interoperability issues in mixed vendor networks– Delays significantly availability of feature– Adds to development and integration costs

VCCV Mode Vendors Y Vendors X

Control Word* Yes No

Router Alert Label Yes No

TTL Expiry* No Yes

13

Continuing Challenges

• Latency sensitive customers– These are typically financial customers that are sensitive to a 2ms

increase or change in latency

• Require traffic to be on path with deterministic low latency– Due to network event traffic may be rerouted, via Fast Reroute and the

re-signaled LSP– Paths are recalculated periodically to ensure low latency path– Once optimal path is available, traffic is re-routed (make before break) to

this path– As this path could be significantly shorter (2 – 10ms), there will be out of

order packets that may impact some hosts

• Nodes in network within the core, may carry a high number of LSPs– Latency sensitive customers are requesting notification on any

maintenance that will impact LSPs carrying their traffic

14

MPLS Going Forward

• MPLS has been an extremely successful protocol– It has been widely deployed and extended

• MPLS based networks and facilities to continue to grow and expand – This growth is continuing and will continue for some time