Network Update, Building Tomorrow’s Big Science NetworksChris Tracy

Network Planning, ESnet

Lawrence Berkeley National Laboratory

Tech Exchange

San Francisco, CA

October 16, 2017

Mission: To Enable and Accelerate Scientific Discovery by Delivering Unparalleled Network Infrastructure, Capabilities, and Tools

2

Bulk Data Movement

Global Connectivity

Potential network service requirements to support tomorrow’s scientific collaborations include:

Remote Control Applications

Deadline SchedulingTele-Presence

Real Time Data Streaming

Network Content CachingNetwork Security Services

Virtual Private Networks

Superfacility Model

Named Data Networking

Virtual Private CloudsApplication-Network Interaction

Distributed Workflow Integration

ESnet Terrestrial Circuit Updates● Additional redundancy between ESnet and BTAA (aka OmniPoP)

– Starlight to BTAA (100G)– 600 W. Chicago to BTAA (100G)

● Nashville to McLean (100G Express Wave)● NCSA (new 100G Connection)● Equinix 100G upgrades

– High-speed access to shared fabric and cloud providers– Ashburn & Chicago: 100G to fabric, 200G to backbone– San Jose: 100G Upgrade in progress

● Chicago to McLean (100G direct + 100G through Equinix Chicago/Ashburn)● ESnet Measurement Boxes (Disk PTs aka DTNs)

– BNL, ANL, and LBL DTNs → moving to hubs (NEWY, STAR, and SUNN)

3

ESnet Transatlantic Circuit Updates

4

ESnet Transatlantic Circuit Updates

5

Integrate NSF-funded IRNC NEAAR circuit (completed)

ESnet6

Building Tomorrow’s Big Science Networks

6

Our Next-Generation Network will be ESnet6Project Mission Need

● Exponential growth in network CAPACITY needs– 72% year-on-year traffic growth since 1990– Cost effective solution to increase capacity as needed

● Network Life Cycle: Improve RELIABILITY– Replace aging infrastructure– Increase the cyber-resiliency of the network

● Network FLEXIBILITY– Support increasingly complex workflow models– Flexibility at all layers of the network is needed to support wide

spectrum of science requirements

7

ESnet6 Rough Project Timeline

CY2017 CY2018 CY2019 CY2020 CY2021 CY2022

8

May: R&D Conceptual-Arch Selection

Sept: In Process Design Review

More Reviews

Jan: R&D Architecture Review

Project Close-out

Network Build

ESnet6 Service Starts

ESnet5 DecommissioningStarts

Project Funding

ESnet6 Input Requirements

9

1. Capacity• Predict usage• Determine overheads (e.g. burst multiplier, resiliency requirements, short-term

growth trends)2. Services

• Document workflows• Develop service portfolio*

*NB: Service Portfolio in conjunction with architecture design drives the technical requirements

ESnet6 Capacity Planning Process

10

1. Determine predicted baseline usage (for 2020, 2025, and 2030)1. Perform best-fit growth curve of ingress traffic per router2. Adjust individual router predictions such that total of all router ingress traffic

matches ESnet’s 25+ year total traffic growth curve3. Using historical flow data and predicted ingress traffic data, perform full mesh

path computation to determine per link utilization from PE-to-PE2. Strategic capacity planning (for 2020 and 2025)

1. Add burst overhead bandwidth per link based on historical knowledge2. Add additional bandwidth to paths based on resiliency strategy3. Keep in view: new projects on the horizon, unpredictable events

11

LHCONE ramps up From 1.7 PB in December 2014~10x in 8 months To 18.4 PB in July 2015

Long term modeling and capacity prediction continues to be a challengeESnet6 Predicted Usage Map in Jan 2030

100+Tbps speeds at long-haul distances on a single fiber pair is outside the existing optical technology envelope

12

What does capacity really look like in the future?Jan 2021 Bandwidth Capacity Planning Predictions

ESnet6 Services Definition Process

13

• Determine workflows based on requirements workshops– 6 DOE Office of Science program offices:

- Advanced Scientific Computing Research (ASCR) - Basic Energy Sciences (BES)- Biological and Environment Research (BER) - Fusion Energy Sciences (FES)- High Energy Physics (HEP) - Nuclear Physics (NP)

– Two requirements workshops a year– 3 years to rotate through all 6 programs

• ESnet testbed research activities

ESnet SDN Testbed

ESnet6 Workflows, Services, and Technical Requirements

14

Input on requirements are documented as workflows, which are then formalized as services, driving the technical requirements.

Workflows

Example workflow:

Use of Cloud compute resources as an extension of the site’s resources.

Services

Example Service:

Virtual Private Cloud

• Service Description

• Service Attributes– Scale– Scope– Demarcation

TechnicalRequirementsExample Requirements:

• L2VPN– EVPN– MPLS– ISIS-SR– BGP-SR-TE

• e/iBGP

R&D Phase: Architecture and Technologies Matrix

15

Orchestrators

Packet Optical Integration Traditional Routed Software Defined Networking

– Architecture A&B (Packet Optical Integration)• Potentially highly scalable, cost compelling, power & space efficient• All-in-one box requires very careful resiliency planning• Separate transponder and amplifier chain (vendor) domains require some custom integration for optical end-to-end management

ESnet6 R&D Technology Findings in a Nutshell

16

– Architecture C&D (Traditional Routed)• Clean layer separation provides simpler resiliency planning• Cost challenged, potentially unsustainable power and space requirements moving towards 2025

– Architecture E&F (Software Defined Networking)• Highly flexible for resource slicing and service creations (i.e. NFV)• Potentially highly scalable, cost compelling• Non-trivial design complexity (e.g. distinct requirements and designs for management, control, and data planes)• Potential production support complications (e.g. troubleshooting, multi-vendor components, etc)• Lack of maturity for WAN scale solutions

Conclusion: ESnet6 will be a hybrid, containing certain aspects of each architecture and balancing the benefits and trade-offs.

ESnet6 (“Hollow-Core”) Conceptual Architecture Overview

Services Edge• Primary function is to provide customer service handoff• Instantiate services locally at point-of-use (where possible, and using the

Core only for connectivity to other service edges)• Coordination with Core for edge-to-edge services with TE constraints• Reactive functions performed locally with proactive functions orchestrated

centrally• Highly programmable data and control plane Network Elements (NEs)

leveraging SDN concepts to dynamic instantiate (new) services as needed

“Hollow” CoreProgrammable, Scalable, Resilient

17

Services EdgeProgrammable, Flexible, Dynamic

“Hollow” Core• Primary function is to maintain connectivity between Service Edge

nodes• No per-hop L3 routing within the Core, packets will be (label) switched• High capacity bandwidth paths with optical express and line sub-rate

support• Protection and restoration for (Service) Edge-to-Edge connections• Dynamically provisionable bandwidth paths• Centralized intelligence for traffic engineering paths• Low cost to add capacity as needed

ESnet6 Functional Architecture (Optical Core)

18

Packet CoreOptical Core

Open Line System

Colorless and directionless (not contention-less) programmable wavelength switching[Capacity/Flexibility]

Flexible grid system[Capacity]

Flexibility to support 3rd party transponders[Capacity/Flexibility]

Router or switch transponder interface[Capacity]

Dedicated transponder shelf with client Ethernet interfaces[Capacity]

Integrated management system to manage open line system and 3rd party transponders[Reliability]

“Hollow” Core

Services Edge

ESnet6 Functional Architecture (Packet Core)

19

Packet CoreOptical Core

Open Line System

Deep buffers for traffic aggregation and speed mismatches[Capacity]

Traffic engineering and resiliency functions to facilitate per service (instance) design engineering[Resiliency/Flexibility]

“Hollow” Core

Services Edge

Differentiated class of service for packet traffic[Flexibility]

ESnet6 Functional Architecture Services Edge

20

“Hollow” Core

Customer facing protocols[Resiliency]

Protocols to instantiate the services with WAN[Resiliency/Flexibility] Highly programmable and

flexible edge Network Elements (NE)[Flexibility]

Compute resources for Network Function Virtualization (NFV) and Virtual Network Functions (VNFs)[Flexibility]

Support for commercial network functions[Flexibility]Network

Appliance

Services Edge

ESnet6 Deployment Considerations

21

Packet CoreOptical Core

Open Line System

“Hollow” Core

Services Edge

Network Appliance

Single NE serving both Service Edge and Core (packet) functions

Distinct Service Edge NE with programmable control and data-plane (e.g. FPGA or NPU) to support custom encapsulations

Service Edge NE will be programmable control-plane (via an API), but with “fixed” data-plane functions (e.g. Broadcom Jericho)

Components in Service Edge node deployments may vary across customer/peer locations based on required functionality and expenditure

Orchestration to automate service provisioning, common configuration APIs to NEs is desired

Orchestration Monitoring and Measurement

Leverage standards based monitoring and measurements mechanisms and protocols

Dedicated testbed resources for internal/external research, and prototyping new services

• Integrated or turn-key solutions- Pros: - Cons: - Simplified deployment - High CAPEX/OPEX - Well defined support model - Flexibility may be constrained

• Disaggregated or Open Platform solutions- Pros: - Cons - Lower CAPEX/OPEX - Integration work needed - Minimal vendor lock-in - Support complexity

• Custom in-house solution- Pros: - Cons: - Highly customizable - Significant resources needed - Huge potential cost savings

Design Implementation Options

22

Disaggregated and Open Platform solutions have been progressing at a very rapid rate in the last few years and show great promise. ESnet6’s implementation solution may encompass components from all 3 options.

Your Favorite Vendor Name

Here

ESnet6 and Beyond: Build a Smarter Network (Breaking the 50% utilization ceiling)

23

• Informed: Understand past and present conditions– Fully instrumented– Highly granular telemetry information– Assimilates external information (e.g. network black-lists, attack signatures, etc)

• Proactive: Recognizes trends (resource optimizations) and patterns (threat mitigation)– Deep learning– Anomaly detection

• Adaptable: Reconfigure the network or services to optimize for efficiency, effectiveness, and security

All the above requires research, development, and integration to realize!!!!

Looking Towards ESnet7: Assimilates, Anticipates, Adapts

24

Level 0:No Automation

Now

Level 1:Function-specific

AutomationNow

Level 2:Combined Function

Automation2013+

Level 3:Limited Self-Driving

Automation2020+ ?

Level 4:Full Self-Driving

Automation2025+ ?

No Automation

Drive in complete and sole control at all times

Involves 1 or more specific control functions

(e.g. stability control, pre-charged brakes)

Involves automation of at least 2 primary control functions working in unison (e.g. adaptive

cruise control in combination with lane

centering)

Enables all safety-critical functions to be automated

(incl. steering, throttle, brake). The vehicle

monitor any changes in conditions that require a transition back to driver

control

Vehicle is designed to perform all safety-critical

driving functions and monitor road conditions

for an entire trip (includes both occupied and

unoccupied vehicles)

Drive can regain control or stop faster than if driving without the

special function

Driver is temporarily relieved of these driving

functions

Driver not expected to take control at any time

Driver is temporarily relieved of these driving

functions

Source: NHTSA (Modified)

Vehic

le >

< Dr

iver

Level of Driving Automation [NHTSA]Vehicle can:

Assimilates internal and external information (e.g. usage, maintenance schedules, component MTF, driver’s schedule, etc.)

Anticipates trips based on routines and disruptions (e.g. scheduled maintenance) Adapts route and departure time due to road, (current and expected) traffic, and weather conditions

Level “5”:Full-service

Transportation Orchestration and

Automation

• Networks of tomorrow will be cognitive (e.g. Level “5”)• Networks today are primarily directive (e.g. Level 3-4)

25

Thank You and Questions?