ATLAS Tier 2 Paths Within ESnet

Mike O’Connor

ESnet Network Engineering Group

Lawrence Berkeley National Lab

moc@es.net

ATLAS Tier 2 Connectivity

• Typical ATLAS tier 2 site connectivity should be via Abilene and ESnet in order to reach Brookhaven Lab.

• “ESnet’s Goal is that DOE Lab ↔ Univ. connectivity should be as good as Lab ↔ Lab and Univ. ↔ Univ.” W. E. Johnston

ATLAS Tier 2 Support• The role of ESnet is to provide networking that supports and

anticipates the needs of the Office of Science Labs and their collaborators.

• ESnet support for ATLAS tier 2 site connectivity makes high bandwidth and fault tolerant paths between BNL and Abilene an absolute priority.

• Planning is underway to upgrade BNL’s ESnet OC48 circuit to a multi 10G lambda fault tolerant Metropolitan Area Network (MAN). (Targeted for Q1 2006)

• Today ESnet and Abilene peer in Atlanta, Chicago, New York and Sunnyvale with a combined bandwidth of 18.2Gbps.These four locations comprise a high bandwidth and fault tolerant interconnect between these two networks.

ES

ne

t S

cie

nc

e D

ata

Ne

two

rk (

SD

N)

co

re

LVMR-NOC

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPLLab DCOffices

MIT

ANL

BNL

FNALAMES

LLNL

GA

DOE-ALB

OSC GTNNNSA

International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)OC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

ESnet IP core

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet over SONET Optical

Ring and Hubs ELP

ATL

DC

commercial and R&E peering points

MAE-E

PAIX-PAEquinix, etc.

PN

WG

Po

P/

PA

cifi

cWav

e

SEA

ESnet Provides Global High-Speed Internet Connectivity forDOE Facilities and Collaborators (Spring, 2006 Configuration)

ESnet core hubs IP

Abilenehigh-speed peering pointswith Internet2/Abilene

Abilene

Ab

ilen

e

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

StarlightChi NAP

CH

I-SL

SNV

Ab

ilene

SNV SDN

JGI

LBNL

SLACNERSC

SDSC

Equinix

MA

N L

AN

Ab

ilen

e

MAXGPoP

SoXGPoP

SDN

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2, ASCC)

NA

SA

Am

es

AU

South AmericaWest South America

East

international R&E peers

AU

NREL

ESnet AS 293

Abilene AS 11537

10 Gbps 2.4 Gbps 1 Gbps

Atlanta GASunnyvale CA Chicago IL New York NY

ESnet & Abilene Peering points

*paths do not include all router hops**Example site logos chosen at random.

ESnet AS 293

Abilene AS 11537

Atlanta GASunnyvale CA Chicago IL New York NY

Abilene Interior Gateway Protocol (IGP) mileage based metrics determine the best exit path toward the ESnet. This common method also known as “hot potato routing” it will always compute an exit path to the nearest peering point.

Abilene Routing Toward ESnet

ESnet AS 293

Abilene AS 11537

Atlanta GASunnyvale CA Chicago IL New York NY

ESnet Routing Toward Abilene

Abilene Multi Exit Discriminators (MEDs) are used by ESnet to choose the best exit path toward Abilene. These MEDs are based on the same IGP metrics used in the interior of their network. This creates a symmetric path back to Abilene.

32 Avenue Of the Americas NYC

ESnet PeeringRouter

MANLAN

ESnet CoreAOA (NYC)

Abilene

OC48DC

OC192CHI

1GE

10GE

10GE

2GE

Peerings@MANLANnysernet 3754abilene 11537sinet 2907geant 20965

BNLOC48

The MANLAN peering is the primary path from ESnet to Abilene at AOA. The directly connected 1Gbps peering is a secondary failover connection.

Chicago

ESnet Starlight

ESnet CoreChicago

Abilene

Peerings@Starlight ANL 683 ASCC 9264 CANET 6509 CERN 513 DREN 668 FNAL 3152 GLORIAD 5568 KREONET 17579 MREN 22335 NASA 297 NWU 103 OPENTRANSIT 5511 SINET 2907 TANET 7539 UIUC 38 ULTRALIGHT 32361 USLHCNET 1297 UWMADISON 59 WISCNET 2381

OC192NY

OC192CA

OC48

10GE

Abilene is the only external peer connected directly to the ESnet Core router in Chicago.

ESnet Starlight

ESnet CoreChicago

Abilene

OC192NY

OC192CA

OC48

10GE

1.9 Gbps peak19% Load

0.8 Gbps peak 33% Load

Current Load in Chicago

Sunnyvale & Atlanta

ESnet and Abilene have direct OC48 peerings in Atlanta GA and Sunnyvale CA.

ESnet CoreSunnyvale CA

Abilene

OC192CHI

OC48

ESnet CoreAtlanta GA

Abilene

OC48

OC48DC

OC48

El PasoTX

OC48

Future Directions

• Long Island MAN Ring

• ESnet Lambda Infrastructure

MANLANAbileneGEANT

Manhattan Long IslandIP Core to Chicago

IP Core to DC

111th & 8th

BNL

60 Hudson

32 AOA

λ2 SDN/circuit

λ1 production IP

10 Gb/soptical

channels

• 2 λs (10 G circuits) delivered as two 10 GigEther circuits

-8 X current BNL bandwidth

• Physically diverse fiber paths forming a ring topology.

• Will be used as a10 Gb/s production IP and a10 Gb/s path for SDN circuit services

• Services switch over to the other service path during a circuit failure.

ESnet LI MAN Ring

Circuits based on WDM rings

*SDN – Science Data Network

ESnet 2010 Lambda Infrastructure and LHC T0-T1 Networking

Denver

Seattle

Su

nn

yv

ale

LA

San Diego

Chicago

Pitts

Raleigh

Jacksonville

Atlanta

KC

Baton Rouge

El Paso - Las Cruces

Phoenix

Pensacola

Dallas

San Ant.Houston

Albuq. Tulsa

New YorkClev

Boise

CE

RN

-1G

ÉA

NT

-1G

ÉA

NT

-2

Wash DC

CE

RN

-2

Tier 1 Centers

ESnet IP core hubs

New hubs

ESnet SDN/NLR hubsESnet Production IP core (10-20 Gbps)ESnet Science Data Network core (10G/link)(incremental upgrades, 2007-2010)Other NLR linksCERN/DOE supplied (10G/link)International IP connections (10G/link)

Cross connects with Internet2/Abilene

NLR PoPs

TRIUMF

FNAL

BNL

CE

RN

-3

Vancouver

Toronto

CANARIE

Conclusions

• ESnet provides an infrastructure and services that are critical to ATLAS science

• ESnet is well positioned to provide essential network services to the LHC tier 2 community

• ESnet is working on providing the DOE mission science networking requirements with several new initiatives and a new architecture