Evolution of the Internet2 Network

Dale Finkelson

CANS 2017

Topics:

• Current transition of the network – Short term – Motivation – Start point – End point – Demand expectations – Support for Layer 2, 3.

• Developing Evolution of the network – Long term – Goals:

• Closer coordination with US R&E network (Regionals and state networks) • Closer coordination with Global R&E on new technology

– Community involvement – POC efforts

Topics

• Support for SDN

• Supporting the research community

– Facilitation teams

• NS, Security, Research Support, CE

– Developing models for use of Internet2 and related global resources effectively

• Continued participation in GNA activities

CURRENT TRANSITION OF THE NETWORK – SHORT TERM

[ 5 ]

Background: Planning, Architecture, and Engineering

Developing Structure: Three interrelated, overlapping teams: • Planning and Architecture (Joint)

• Engineering (Joint)

• Routine 24x7 Operations and Break/fix One goal: Break down boundaries between teams as much as possible, while maintaining clear understandings of responsibilities.

Architecture Engineering

O perations

[ 6 ]

Background: Planning, Architecture, and Engineering

Ground Rules: Make smart, efficient investments on behalf of the community Support data movement & access to cloud services Provide abundant interconnect bandwidth, minimizing risk of congestion Accelerate continuous evolution and improvement of infrastructure Enable end-to-end innovation across all layers of the infrastructure Optimize and enhance the current infrastructure

[ 7 ]

Late 2015

B

J JJ

J

B

J J

B

J

B

J

JJ

B

B

J

J

J

JB

B

J

J

B

J

B

B

J

B

B

J

B

J

JB

B

B

J

J

J

J

J

JJ

B

J

J

J

J

J

J

J

J Juniper AL2S Switch

Optical Add/Drop, no L2/L3

L2 or L3 BB via 3rd Party Service

L2 or L3 BB via Internet2 Opt

J Juniper TR-CPS Router

B Brocade AL2S Node (replaced/consolidated)

J Juniper R&E Router (w, w/o TR-CPS vrf)

J

NEWY1118TH

NEWY32AOA

EQNY

HART2

BOSTALBA

CLEV

PITT

ASHB

RALE

WASH

CHAR

PHILINDI

LOUI

CINC

STAR EQCH

CHIC

KANS

SAIN

COLU4

TULS

NASH

ATLA

JACK

BATO

JCSNDALLDALL3

HOUS

HOUH

MINN

MISS2

SEAT

LACY

PORT

EUGE

BOIS

SACR

SUNN

PAIX

LOSA

WILC

SAND

SANL

PHOE TUCS

SALT

DENV

PUEB

ALBU

ELPA SANA

J

RENO

J

LASV

5x10G

LAG

[ 8 ]

2016-2017 Target areas

Core Network:

Move to single vendor platform based on MPLS (2016 – 2017)

Capacity:

Improve analytics; perform first wave of upgrades

Leverage DCI at key interconnect locations

Optical:

Auditing and Compliance

[ 9 ]

2016-2017 CORE Network Program

• Project Scope – Build a production quality single-vendor core network – Remove OpenFlow dependency from backbone transport – Retain API based provisioning, SDN functionality – Improve access to routed services – Eliminate as many “choke points” as possible – Pave the way for future technologies and deployments

• Goals/Success Criteria – All Layer 2 circuits on L2VPNs/VPLS/L2-CCC – Layer 3 Connectivity using VRFs and leveraging core iBGP instances (not

OpenFlow) – Reduced switch/router population – ENSURE MINIMAL COMMUNITY DISRUPTION

[ 10 ]

2016-2017 CORE Network Program: Requirements

• All Services at all nodes

– AL3S (R&E, TRCPS, LHCONE, other L3VPNs)

– AL2S (L2VPN, VPLS, Advanced L2-CCC)

• Allow for Explicit Path Selection

• Restore to Primary by Default

• Edge interface support all services on a single port

• Multipoint (VPLS, default MAC-Address Table Limits)

• Path Exposure (Traceroute, MPLS Ping)

• IPv6

• Multicast (ASM & SSM)

[ 11 ]

2016-2017 CORE Network Program

Consolidated AL2S/AL3S/TRCPS

J

J

J

J J

J

J J

JJ

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

J

JJ

J

J

J

JNEWY1118TH

NEWY32AOA

HART2

BOST

ALBA

CLEV

PITTASHB

RALE

WASH

CHAR

PHILINDI

LOUI

CINC

STAR

EQCH

CHIC

KANS

SAIN

COLU4

TULS

NASH

ATLA

JACK

BATO

JCSN

DALL

DALL3

HOUS

HOUH

MINN

MISS2

SEAT

LACY

PORT

EUGE

BOIS

SACR

SUNN

PAIX

LOSA

WILC

SAND

SANL

PHOE TUCS

SALT

DENV

PUEB

ALBU

ELPA SANA

RENO

LASV

J

J

J

J J TELX

Optical Add/Drop, no L2/L3

J R&E/TRCPS VRFs and OpenFlow

J OpenFlow only

J TRCPS VRF only

J Juniper MX960

J R&E/TRCPS VRF only

[ 12 ]

Current Status

• Transition of all vlans to MPLS is completed as of 8-15-17.

– Total of vlans transitioned in stages was 890.

– Took about 40 hours total

• The process occurred over 7 nights. No single vlan took more then a minute or so.

• No substantive issues came up during this process.

SDN SUPPORT

• SDN network where we can take risks without impacting production

– Reduce friction from thorough acceptance testing

– Use separate hardware to implement the overlay network

– Support direct remote access for OF controllers.

• Collocate compute with hardware based switching

– Support NFV exploration

– Support alternative approaches like NDN where hardware support not available.

• Support from the GENI Project Office to provide OpenFlow 1.3 capable overlay

Motivation

[ 14 ]

• 8 sites, national footprint

• Each site contains:

– Dell Server

– Corsa Switch

– Multiple 10GE interconnects provided via AL2S

– 10GE AL2S port for onramp/offramp

[ 15 ]

What it looks like

• AL2S circuits interconnect sites, circuits follow physical infrastructure

• Each slice gets its own set of logical circuits

• To the control plane, these look just like direct 10GE adjacencies

• Each slice has its own dedicated management network.

[ 16 ]

What it looks like (2)

• Each slice gets a dedicated private management network

• A virtual bastion server is provided for access if need be

• Bastion provides limited NAT support to allow OF connections to remote controllers

• Slice users, now have choice to run controller on our network or in their lab. [ 17 ]

Slice management network

• 3 node trial deployment demonstrated at March GENI conference

• Rest of the nodes have been deployed and configured

– Reworking the control plane to keep it separate from the rest of the infrastructure

• In process evaluating Corsa OpenFlow 1.3 support and features

• Pace may vary with MPLS mirgration demands.

Status and next steps

[ 18 ]

LONG TERM EVOLUTION OF THE NETWORK

Framing

• Community investment - not just backbone - and not normal refresh • Not “what Internet2 should do” but “how can we work differently together”

• Target for capital investment is 2+ years out • Practically - looking at technology that is not yet ready for prime time today • We should take the time to more efficiently make shared investments where practical

• Re-emphasize our commitment to our research support missions • Tailor services and infrastructure to do so in a more integrated fashion

• Workforce impact - skills, processes, etc. • This is an overarching issue

• This is not a firm plan - we need input and collaboration • Not everyone will be ready to engage day one • Maintain community mindset - all can join and benefit when ready

Key principles

• Developed from requirements gathering process

– Ecosystem - planning, service development and investment should proceed in an integrated fashion amongst ecosystem members (campuses, regionals, NRENs, etc.)

– Experimentation - development of the integrated infrastructure (ecosystem) should take place as a community-wide continuous development process

– End User Experience - Follow leading use cases and examples that enhance the user’s end to end experience

• Points to the need for coordinated, community experimentation efforts

Developing an action plan

• Goal: gain experience as a community with the technologies and collaborations that will allow us to serve our evolving needs – Technology evaluation - start with Core

• Evaluate current state of packet/optical, choose candidate technologies/products, test in lab & in situ, make informed analysis of where capabilities will be in 2 years

• Develop a model of deeper collaboration within the ecosystem – Pick candidate service, develop joint deployment and ops plan – Could be focused on core technology, or could be higher layer – Evaluate impact on business models, investments, etc.

• Technology evaluation phase 2 - Edge – Develop key technology partners - community members working in this space, industry, etc – Evaluate current state, choose candidate technologies/products, test in lab & in situ, make

informed analysis of where capabilities will be in 2 years

PoC proposals - initial straw man

• Core technology evaluation • End-to-end service delivery - technology, operations, business • Edge technology evaluation • Thoughts about partnering:

– For the technology evaluations: • Ideal candidates are community members who are already engaged in or planning

on looking into new technology options, or who have already established experimental platforms that can be operationalized

– For the end-to-end service delivery project – We may want to target research use cases that can benefit, and have some

tolerance for being on the front end of service development

PoC proposals - initial straw man

• Initial Cases

– Working with NYSERnet on Voyager evaluation

– Working with MAX on redesign of WIX

REASEARCH FACILITATION

Facilitation Team

• Led by John Moore and Jim Bottum

• Developing a program to support the research community in using the global infrastructure effectively.

– General idea is to develop local campus resources that can help with problem identification and resolution.

– Internet2 acts as a resource and facilitation group to assist as needed.

Facilitation Team

• Engage in outreach to the Research community to better understand their needs.

• Work to extend this activity to the whole of the Internet2 community.

– As well as their interactions globally.

• In the process of developing a program.

– Input on how best to achieve the goals is always welcome.

GNA

GNA

• Currently in the process of developing GNA 2.0 standards. • GNA tech is co-chaired by myself and Erik-Jan Bos • Many people from the community are engaged in the current activities,

these include: – Enhanced measurement/monitoring – Work on End-to-End capabilities – Agreements on baseline services and functionalities – Potential use of NFV tools – Non-networking capabilities – Security and Authentication

• More information on the GNA activities will be forthcoming from Erik-Jan and myself .