Inter-Domain ServicesInter-Domain ServicesPresented to Quilt Spring WorkshopPresented to Quilt Spring Workshop

Salt Lake City, UTSalt Lake City, UTByBy

Linda WinklerLinda Winklerwinkler@mcs.anl.govwinkler@mcs.anl.gov

April 4, 2007April 4, 2007

MotivationMotivation

Support applications that demand capabilities that are hard to support in a shared Support applications that demand capabilities that are hard to support in a shared packet infrastructurepacket infrastructure

• Large bandwidth applicationsLarge bandwidth applications• Applications don’t want to worry about their impact on other network users, or vice versaApplications don’t want to worry about their impact on other network users, or vice versa• Applications that benefit from circuit characteristics, and that may be low bandwidth in natureApplications that benefit from circuit characteristics, and that may be low bandwidth in nature

Dedicated network resources Dedicated network resources Deterministic performanceDeterministic performance

• Repeatable and predictable Repeatable and predictable • Very high performanceVery high performance• Multi gigabit flows, low latency, low loss, minimal jitterMulti gigabit flows, low latency, low loss, minimal jitter

• User community expectationsUser community expectations• Extreme bandwidth, dedicated and on-demandExtreme bandwidth, dedicated and on-demand• Inter-regional and international connectivity requirements Inter-regional and international connectivity requirements

Reserve and schedule network resources in advanceReserve and schedule network resources in advance• In conjunction with availability of non-network resources (e.g. instruments, clusters, viz devices)In conjunction with availability of non-network resources (e.g. instruments, clusters, viz devices)

Flexible and dynamicFlexible and dynamic• Capability to acquire resources on short notice from many potential service/resource providersCapability to acquire resources on short notice from many potential service/resource providers

E-Science services growthE-Science services growth• Broad scope of applicationsBroad scope of applications• Petabyte demandsPetabyte demands• Globally distributed applications, resources and collaboratorsGlobally distributed applications, resources and collaborators

Emerging large-scale, globally distributed applications require more sophisticated network services than have previously been delivered.

Trends=>Set the StageTrends=>Set the Stage Applications community desire to treat network resources as an integral Grid resourceApplications community desire to treat network resources as an integral Grid resource R&E Optical NetworksR&E Optical Networks

• Locally-managed fiber termination pointsLocally-managed fiber termination points• Locally-organized peering relationshipsLocally-organized peering relationships• Locally-controlled layer-0/1/2/3 servicesLocally-controlled layer-0/1/2/3 services• Community ownership and control of network infrastructureCommunity ownership and control of network infrastructure

Ubiquitous EthernetUbiquitous Ethernet• Most-requested client service interfaceMost-requested client service interface• Both point-to-point and virtual-LAN topologiesBoth point-to-point and virtual-LAN topologies• Apps consuming bandwidth in Ethernet-sized incrementsApps consuming bandwidth in Ethernet-sized increments• 10G has become a commodity10G has become a commodity• Application expectations for error-free connections keep rising. Apps with minimal error checking running on Application expectations for error-free connections keep rising. Apps with minimal error checking running on

pipes with almost no error correction, across inter-RON pipes with no error reporting pipes with almost no error correction, across inter-RON pipes with no error reporting a recipe for failure a recipe for failure Convergence Optical + Ethernet LayersConvergence Optical + Ethernet Layers

• Ethernet client interface, Optical transportEthernet client interface, Optical transport• Optical service established via GMPLS Optical service established via GMPLS

Control plane technologies emerging to help meet user expectations in the current environment Control plane technologies emerging to help meet user expectations in the current environment FPGA technologyFPGA technology

• Allowing WDM transport hardware to be built with software defined functionalityAllowing WDM transport hardware to be built with software defined functionality• Flexibility, fewer components, configurations, sparesFlexibility, fewer components, configurations, spares• Investment protection and future-proofingInvestment protection and future-proofing

Pluggable opticsPluggable optics• Flexibility, lower costsFlexibility, lower costs• Newer capabilitiesNewer capabilities

Technology ChoicesTechnology Choices Reasons you may want/need build circuits

• Dedicated bandwidth connections for deterministic file transfers• Dedicated bandwidth & low jitter for instrument control or interactive

applications• Connector backhaul to your IP Network• Traffic engineering of your IP Network

Dynamic router-to-router circuits for traffic cut through

• Computer to Computer communications Processor to memory or block data storage system access

• Setting up application specific topologies to create & optimize distributed application or data storage systems

There are many technologies available over which to construct these circuits

• IP router-based Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) “circuits”

• Ethernet VLAN based “circuits”• SONET/SDH TDM “circuits”• Wavelength Division Multiplexing (WDM) “circuits”

IP

Ethernet

SONET DMencap GFP

Lambda

Fiber

Network Technology Service

Dark Fiber

Wavelength

Lightpath (sub-lambda)

Ethernet VLAN

IPLayer 3

Layer 2

Layer 1

Layer 0

Mapping Technology to ServicesMapping Technology to Services

Starlight

iCAIR

UvA

NetherLight, Amsterdam

Nortel Research Labs Ottawa

Glimmerglass

Topology for L1 Dynamic Optical Multicast TestbedTopology for L1 Dynamic Optical Multicast Testbed• 750-~980+ Mbps Hi-Def Video• 3x1:2 Optical Multicast: StarLight OME

Nortel HDXc

OC192/STM64

10GE LAN

750-~980 Mbps

Nortel HDXc

UvANortel

NortelDX

Fujitsu

UvA

Nortel

iCAIR

UvA

iCAIR Nortel

3x750 MONS

3x750 M

3x1:2Opticalmultica

st

2

1

OMNInet

NortelDX

Nortel 8600

10 G WAN

Nortel 8600

1

2 ONS

1

2

CA*Net5

NortelOME6500

iCAIR

NortelOME6500

Nortel

Brno

LSU

StarlightNetherlight

SC NOC

Baton Rouge(LSU Class)

McLEAN

A-Wave

CAT 6506

3x1.5G

LSU

Topology for L2 Multicast DemoTopology for L2 Multicast Demo• 1.5 Gbps Hi-Def video• 3x1:2 Ethernet multicast: StarLight & SCinet• Layer1 and Layer2 networking

Nortel HDXc

Nortel HDXc

Raptor

CaveWave

Catalyst 65xx

e300

Brno Tampa

F10 E1200

Calient

CAT 6509

BrnoONS

Brno

LONI Tampa

Brno

LONI Tampa

3x1.5G

OC-192 GFP/VCAT

10GE LAN

10G WAN

1.5 Gbps

3 x 1.5G

10GE switch

OPTERON

1

2

1

4

2

12

1

3

1

2

1

2

CAT 6504

SLE1200

22

Calient

Tampa

LSU

NortelMERS8600

OPTERONAMD PC

ONS

OMNInet

OMNInet

EnLIGHTened Wave

CAT 6506

IP NetworkCircuit Switched

VLAN overMPLS

MPLS tunnelLayer-1/2 connection

VLAN over Ethernet orSONET

Alignmentof VLANS

Unification of Data Plane Technology Unification of Data Plane Technology Across Layers 1/2/3—One MethodAcross Layers 1/2/3—One Method

Dynamic Circuit Setup/AllocationDynamic Circuit Setup/Allocation

Lots of work in the community on dynamic resource allocation Lots of work in the community on dynamic resource allocation (lightpaths) and how we’re going to get there from here, eg GLIF (lightpaths) and how we’re going to get there from here, eg GLIF forum.forum.

Soon we will be to able to rely on multiprovider/multivendor Soon we will be to able to rely on multiprovider/multivendor control plane signaling for circuit establishment.control plane signaling for circuit establishment.

Simplify or add complexity?Simplify or add complexity? Dynamic resource mechanisms will increase our capabilities and Dynamic resource mechanisms will increase our capabilities and

enable new capabilites, like timesharing resources between enable new capabilites, like timesharing resources between large short-duration flows.large short-duration flows.

Cost TBDCost TBD

Connection Brokering per DomainConnection Brokering per Domain

Network DescriptionsNetwork Descriptions• Paths through networkPaths through network• Information about pathsInformation about paths

Common Services DefinitionCommon Services Definition• Description of connectivity serviceDescription of connectivity service

PredictablePredictable VerifiableVerifiable RepeatableRepeatable End-to-endEnd-to-end

• Agreement between user and provider(s)Agreement between user and provider(s)

Control Plane FunctionsControl Plane Functions Bandwidth requiredBandwidth required

• Allow for reservation of future bandwidthAllow for reservation of future bandwidth User authenticationUser authentication Report status of infrastructureReport status of infrastructure Retain and share information on topology resources and Retain and share information on topology resources and

reachabilityreachability Routing- topology discovery and path computationRouting- topology discovery and path computation Signaling- standard communications protocols between network Signaling- standard communications protocols between network

elements for the establishment and maintenance of connectionselements for the establishment and maintenance of connections Neighbor discovery- cross-domain sharing of connectivity details Neighbor discovery- cross-domain sharing of connectivity details

among neighborsamong neighbors Domain (local) resource management- accounting of available Domain (local) resource management- accounting of available

resourcesresources

Interdomain Connection SetupInterdomain Connection Setup

TDM (SONET/SDH Layer)

RON / NREN

Data Plane

TDM or L2 substrate

InterdomainControl Plane

Control Plane

ChallengesChallenges Engineering/configuration/setup Engineering/configuration/setup O&M complexityO&M complexity

• TroubleshootingTroubleshooting• Fault managmentFault managment• MonitoringMonitoring• PerformancePerformance

Debugging toolsDebugging tools End-to-end measurement for monitoring and performance analysisEnd-to-end measurement for monitoring and performance analysis PoliciesPolicies

• a way to enforce policies and filters at the domain boundaries to process the incoming inter-a way to enforce policies and filters at the domain boundaries to process the incoming inter-domain requests based on certain agreed trust and service levels/contracts between domainsdomain requests based on certain agreed trust and service levels/contracts between domains

Authentication, Authorization, Accounting (AAA)Authentication, Authorization, Accounting (AAA) SecuritySecurity SchedulingScheduling Service Level Agreements (SLAs)Service Level Agreements (SLAs) User interfaceUser interface Capacity planningCapacity planning Many lightpaths traverse a number of transport boxes and domains Many lightpaths traverse a number of transport boxes and domains Number of domains participating on the rise (backbone providers, RONs, campuses). Number of domains participating on the rise (backbone providers, RONs, campuses).

Are they prepared for the challenge?Are they prepared for the challenge?

Dynamic Resource Allocation ResearchDynamic Resource Allocation Research MAX DRAGONMAX DRAGON Internet2 Circuit Services- based on extension of the DRAGON control plane Internet2 Circuit Services- based on extension of the DRAGON control plane

architecturearchitecture Internet2 BRUW (Bandwidth Reservation for User Work)Internet2 BRUW (Bandwidth Reservation for User Work) ESNet Science Data Network and the OSCARS (On-demand Secure Circuits and ESNet Science Data Network and the OSCARS (On-demand Secure Circuits and

Advance Reservation System) projectAdvance Reservation System) project DANTE/GEANT JRA3 projectDANTE/GEANT JRA3 project SURFnet and collaboration with Nortel on the DRAC projectSURFnet and collaboration with Nortel on the DRAC project University of Amsterdam, Network Description LanguageUniversity of Amsterdam, Network Description Language European Union PHOSPHORUS ProjectEuropean Union PHOSPHORUS Project G-Lambda project (Japan)G-Lambda project (Japan) CANARIE- UCLPCANARIE- UCLP DOE UltraScience NetworkDOE UltraScience Network DOE Hybrid Multi-Layer Network Control Project (ESnet, ORNL, I2, ISI East, Tenn Tech)DOE Hybrid Multi-Layer Network Control Project (ESnet, ORNL, I2, ISI East, Tenn Tech) TeraPaths Brookhaven National LabTeraPaths Brookhaven National Lab Enlightened Computing Project (NCREN, MCNC, LSU, NCSU, RCI)Enlightened Computing Project (NCREN, MCNC, LSU, NCSU, RCI) Cheetah (Univ of Virginia)Cheetah (Univ of Virginia) UltraLightUltraLight DICE (DANTE, Internet2, CANARIE, ESnet)DICE (DANTE, Internet2, CANARIE, ESnet) OptIPuter/iCAIROptIPuter/iCAIR

Cisco/UltraLight wave

EnLIGHTened wave (Cisco/NLR)

LONI wave

Members:- MCNC GCNS- LSU CCT-NCSU-(Subcontract) RENCI

Official Partners:- AT&T Research- SURA- NRL- Cisco Systems- Calient Networks- IBM

NSF Project Partners- OptIPuter- UltraLight- WAN-in-LAB- DRAGON- Cheetah

International Partners •LUCIFER - EC•G-Lambda - Japan-GLIF

CHI

HOU

DAL

TUL

KAN

PIT

WDC

OGD

BOI

CLE

POR

DEN

SVL

SEA

Baton Rouge

Raleigh

To Asia To Canada To Europe

L.A.

San Diego

CAVE wave

Chicago

Enlightened Computing TestbedEnlightened Computing Testbed

HD Media Testbed to support class on Computational ScienceHD Media Testbed to support class on Computational Science

EnlightenedEnlightened ArchitectureArchitecture

Internet2 Circuit Service AttributesInternet2 Circuit Service Attributes

Physical connection Physical connection • 1G or 10G Ethernet 1G or 10G Ethernet • OC-48 or OC-192 SONETOC-48 or OC-192 SONET

Guaranteed bandwidth (STS-1 to STS-192)Guaranteed bandwidth (STS-1 to STS-192) Circuit Service TypeCircuit Service Type

• Point-to-point Ethernet framed SONET CircuitPoint-to-point Ethernet framed SONET Circuit• Point-to-point SONET CircuitPoint-to-point SONET Circuit

Deterministic behavior (bounded jitter, latency Deterministic behavior (bounded jitter, latency and packet loss)and packet loss)

Static or dynamicStatic or dynamic

NLRNLR

FrameNetFrameNet• Ethernet-based transport services over the Ethernet-based transport services over the

nationwide NLR optical infrastructure nationwide NLR optical infrastructure • National Exchange FabricNational Exchange Fabric• Dedicated FrameNetDedicated FrameNet• Non-Dedicated FrameNetNon-Dedicated FrameNet

Research with MCNCResearch with MCNC• Calient Optical SwitchCalient Optical Switch

ReferencesReferences dragon.maxgigapop.netdragon.maxgigapop.net www.es.net/oscarswww.es.net/oscars www.internet2.edu/network/library/internet2-network-www.internet2.edu/network/library/internet2-network-

service-descriptions-02272007-v1.1a.pdfservice-descriptions-02272007-v1.1a.pdf www.ist-phosphorus.eu/www.ist-phosphorus.eu/ www.mfaforum.org/interop/www.mfaforum.org/interop/

GMPLSwhitepaper_Final1009021.pdfGMPLSwhitepaper_Final1009021.pdf www.glif.iswww.glif.is enlightenedcomputing.orgenlightenedcomputing.org www.oiforum.comwww.oiforum.com IETFIETF ITUITU