Post on 18-Apr-2022
1
The Energy Sciences NetworkBESAC August 2004
William E. Johnston, ESnet Dept. Head and Senior Scientist
R. P. Singh, Federal Project ManagerMichael S. Collins, Stan Kluz,
Joseph Burrescia, and James V. Gagliardi, ESnet Leads
Gizella Kapus, Resource Managerand the ESnet Team
Lawrence Berkeley National Laboratory
Mary Anne ScottProgram Manager
Advanced Scientific Computing Research
Office of ScienceDepartment of Energy
2
What is ESnet?
• Mission:• Provide, interoperable, effective and reliable
communications infrastructure and leading-edge network services that support missions of the Department of Energy, especially the Office of Science
• Vision:• Provide seamless and ubiquitous access, via shared
collaborative information and computational environments, to the facilities, data, and colleagues needed to accomplish their goals.
• Role: • A component of the Office of Science infrastructure critical
to the success of its research programs (program funded through ASCR/MICS; managed and operated by ESnet staff at LBNL).
3
Why is ESnet important?
• Enables thousands of DOE, university and industry scientists and collaborators worldwide to make effective use of unique DOE research facilities and computing resources independent of time and geographic location
o Direct connections to all major DOE siteso Access to the global Internet (managing 150,000 routes at 10
commercial peering points) o User demand has grown by a factor of more than 10,000 since
its inception in the mid 1990’s—a 100 percent increase every year since 1990
• Capabilities not available through commercial networks- Architected to move huge amounts of data between a small
number of sites- High bandwidth peering to provide access to US, European, Asia-
Pacific, and other research and education networks.
Objective: Support scientific researchSupport scientific research by providing seamless and ubiquitous access to the facilities, data, and colleagues
4
How is ESnet Managed?
• A community endeavoro Strategic guidance from the OSC programs
- Energy Science Network Steering Committee (ESSC)– BES represented by Nestor Zaluzec, ANL and Jeff Nichols, ORNL
o Network operation is a shared activity with the community - ESnet Site Coordinators Committee- Ensures the right operational “sociology” for success
• Complex and specialized – both in the network engineering and the network management – in order to provide its services to the laboratories in an integrated support environment
• Extremely reliable in several dimensions
Taken together these points make ESnet a unique facility supporting DOE science that is quite different from a commercial ISP or University network
5
…what now???VISION - A scalable, secure, integrated network network
environmentenvironment for ultra-scale distributed science is being developed to make it possible to combine resources and expertise to address complex questions that no single institution could manage alone.
• Network StrategyProduction network
- Base TCP/IP services; +99.9% reliableHigh-impact network
- Increments of 10 Gbps; switched lambdas (other solutions); 99% reliable
Research network- Interfaces with production, high-impact and other research
networks; start electronic and advance towards optical switching; very flexible [UltraScience Net]
• Revisit governance modelo SC-wide coordinationo Advisory Committee involvement
6
Where do you come in?• Early identification of requirements
o Evolving programso New facilities
• Participation in management activities• Interaction with BES representatives on ESSC• Next ESSC meeting on Oct 13-15 in DC area
7
What Does ESnet Provide?• A network connecting DOE Labs and their collaborators that
is critical to the future process of science
• An architecture tailored to accommodate DOE’s large-scale science
o move huge amounts of data between a small number of sites
• High bandwidth access to DOE’s primary science collaborators: Research and Education institutions in the US, Europe, Asia Pacific, and elsewhere
• Full access to the global Internet for DOE Labs
• Comprehensive user support, including “owning” all trouble tickets involving ESnet users (including problems at the far end of an ESnet connection) until they are resolved – 24x7 coverage
• Grid middleware and collaboration services supporting collaborative science
o trust, persistence, and science oriented policy
8
What is ESnet Today?
• ESnet builds a comprehensive IP network infrastructure (routing, IPv6, and IP multicast) on commercial circuitso ESnet purchases telecommunications services ranging
from T1 (1 Mb/s) to OC192 SONET (10 Gb/s) and uses these to connect core routers and sites to form the ESnet IP network
o ESnet purchases peering access to commercial networks to provide full Internet connectivity
• Essentially all of the national data traffic supporting US science is carried by two networks –ESnet and Internet-2 / Abilene (which plays a similar role for the university community)
9
How Do Networks Work?
• Accessing a service, Grid or otherwise, such as a Web server, FTP server, etc., from a client computer and client application (e.g. a Web browser_ involveso Target host nameso Host addresseso Service identificationo Routing
10
How Do Networks Work?
LBNL
Google, Inc.
ESnet(Core network)
Big ISP(e.g. SprintLink)
gatewayrouter
routerrouter
router
router
router
corerouter
router
peeringrouter
corerouter
borderrouter
border/gateway routers•implement separate site and network provider policy (including site firewall policy)
peering routersExchange reachability information (“routes”)
• implement/enforce routing policy for each provider
• provide cyberdefense
router
router
core routers•focus on high-speed packet forwarding
peeringrouter
DNS
11
ESnet Core is a High-Speed Optical Network
10GE
10GE
RTR
RTR
optical fiber ring
Wave division multiplexing
• today typically 64 x 10 Gb/s optical channels per fiber
• channels (referred to as “lambdas”) are usually used in bi-directional pairs
Lambda channels are converted to electrical channels
• usually SONET data framing or Ethernet data framing
• can be clear digital channels (no framing – e.g. for digital HDTV)
ESnet IP router
Site IP router
Site – ESnet network policy demarcation
(“DMZ”)
site LANESnet site
ESnet hub
ESnet core
RTR
RTR
RTRRTR
A ring topology network is inherently reliable – all single point failures are mitigated by routing traffic in
the other direction around the ring.
TWC
JGISNLL
LBNL
SLAC
YUCCA MTBECHTEL
PNNLLIGO
INEEL
LANL
SNLAAlliedSignal
PANTEX
ARM
KCP
NOAA
OSTIORAU
SRS
ORNLJLAB
PPPL
ANL-DCINEEL-DCORAU-DC
LLNL/LANL-DC
MIT
ANL
BNL
FNALAMES
4xLAB-DCNERSC
NR
EL
ALBHUB
LLNL
GADOE-ALB
SDSC
Japan
GTN&NNSA
International (high speed)OC192 (10G/s optical)OC48 (2.5 Gb/s optical)Gigabit Ethernet (1 Gb/s)OC12 ATM (622 Mb/s)OC12 OC3 (155 Mb/s)T3 (45 Mb/s)T1-T3T1 (1 Mb/s)
Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (3)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)
QWESTATM
42 end user sites
ESnet IP
GEANT- Germany- France- Italy- UK- etc Sinet (Japan)Japan – Russia(BINP)
CA*net4MRENNetherlandsRussiaStarTapTaiwan(ASCC)
CA*net4KDDI (Japan)FranceSwitzerlandTaiwan(TANet2)
AustraliaCA*net4Taiwan(TANet2)
Singaren
ESnet core: Packet over SONET Optical Ring and Hubs
ELP HUB
SNV HUB CHI HUB
NYC HUB
ATL HUB
DC HUB
peering points
MAE-EStarlightChi NAP
Fix-W
PAIX-W
MAE-W
NY-NAP
PAIX-E
Euqinix
PNW
G
SEA HUB
ESnet Provides Full Internet Serviceto DOE Facilities and Collaborators
with High-Speed Access to all Major Science Collaborators
hubsSNV HUB
Abilene
Abilene high-speed peering points
Abilene
Abile
ne MAN
LAN
Abile
ne
CERN
STARL
IGH
T
MAE-E
NY-NAP
PAIX-E
GA
LBN
L
ESnet’s Peering InfrastructureConnects the DOE Community With its Collaborators
ESnet Peering(connections to other networks)
NYC HUBS
SEA HUB
Japan
SNV HUB
MAE-W
FIX-
W
PAIX-W 26 PEERS
CA*net4CERNMRENNetherlandsRussiaStarTapTaiwan(ASCC)
Abilene +7 Universities
22 PEERS
MAX GPOP
GEANT- Germany- France- Italy- UK- etc SInet (Japan)KEKJapan – Russia (BINP)
AustraliaCA*net4Taiwan(TANet2)
Singaren
20 PEERS3 PEERS
LANL
TECHnet
2 PEERS
39 PEERS
CENICSDSC
PNW-GPOP
CalREN2 CHI NAP
Distributed 6TAP19 Peers
2 PEERS
KDDI (Japan)France
EQX-ASH
1 PEER
1 PEER
5 PEERS
ESnet provides access to all of the Internet by managing the full complement of Global Internet routes (about 150,000) at 10 general/commercial peering points + high-speed peerings w/ Abilene and the international R&E networks. This is a lot of work, and is very visible, but provides full access for DOE.
ATL HUB
University
International
Commercial
Abilene
EQX-SJ
Abilene
6 PEERS
Abilene
14
AS routes peer1239701
20933563561
701829143549551117464617473
349111537540043234200639528287132
SPRINTLINK6338451685
470634144035980
28728197231736981905492503244293529
332733212774247524082383
UUNET-ALTERNETQWESTLEVEL3CABLE-WIRELESSATT-WORLDNETVERIOGLOBALCENTEROPENTRANSITCOGENTCOABOVENETSINGTELCAIS
ABILENEBTTWTELECOMALERONBROADWINGXO
1961 SBC
What is Peering?
• Peering points exchange routing information that says “which packets I can get closer to their destination”
• ESnet daily peeringreport(top 20 of about 100)
• This is a lot of work
peering with this outfitis not random, it carriesroutes that ESnet needs(e.g. to the Russian Backbone Net)
15
What is Peering?
• Why so many routes? So that when I want to get to someplace out of the ordinary, I can get there. For example:http://www-sbras.nsc.ru/eng/sbras/copan/microel_main.html(Technological Design Institute of Applied Microelectronics, Novosibirsk, Russia)
Peering routers
Start: 134.55.209.5
134.55.209.90
63.218.6.65
63.218.6.38
63.216.0.53
63.216.0.30
63.218.12.37
63.218.13.134
195.209.14.29
195.209.14.153
195.209.14.206
Finish: 194.226.160.10
ESnet coresnv-lbl-oc48.es.net
snvrt1-ge0-snvcr1.es.net
pos3-0.cr01.sjo01.pccwbtn.net
pos5-1.cr01.chc01.pccwbtn.net
pos6-1.cr01.vna01.pccwbtn.net
pos5-3.cr02.nyc02.pccwbtn.net
pos6-0.cr01.ldn01.pccwbtn.net
rbnet.pos4-1.cr01.ldn01.pccwbtn.net
MSK-M9-RBNet-5.RBNet.ru
MSK-M9-RBNet-1.RBNet.ru
NSK-RBNet-2.RBNet.ru
ESnet peering at Sunnyvale
AS3491 CAIS Internet
“ “
“ “
“ “
“ “
AS3491->AS5568 (Russian Backbone Network) peering point
Russian Backbone Network
“ “
“ “
Novosibirsk-NSC-RBNet.nsc.ru RBN to AS 5387 (NSCNET-2)
16
Organized by Office of Science
Mary Anne Scott, Chair Dave Bader Steve Eckstrand Marvin Frazier Dale Koelling Vicky White
Workshop Panel ChairsRay Bair and Deb AgarwalBill Johnston and Mike WildeRick StevensIan Foster and Dennis GannonLinda Winkler and Brian TierneySandy Merola and Charlie Catlett
August 13-15, 2002Predictive Drivers for the Evolution of ESnet
•The network and middleware requirements to support DOE science were developed by the OSC science community representing major DOE science disciplines
o Climateo Spallation Neutron Sourceo Macromolecular Crystallographyo High Energy Physics
o Magnetic Fusion Energy Scienceso Chemical Scienceso Bioinformatics
Available at www.es.net/#research
The network is needed for:o long term (final stage) data analysiso “control loop” data analysis (influence an
experiment in progress)o distributed, multidisciplinary simulation
17
The Analysis was Driven by the Evolving Process of ScienceFeature Requirements
Discipline
Characteristics thatMotivate High Speed Nets
• A few data repositories, many distributed computing sites
• NCAR - 20 TBy
• NERSC - 40 TBy
• ORNL - 40 TBy
• Add many simulation elements/components as understanding increases
• 100 TBy / 100 yr generated simulation data, 1-5 PBy / yr (just at NCAR)
o Distribute large chunks of data to major users for post-simulation analysis
• 5-10 PBy/yr (at NCAR)
• Add many diverse simulation elements/components, including from other disciplines - this must be done with distributed, multidisciplinary simulation
• Virtualized data to reduce storage load
Networking
Climate(near term)
Analysis of model data by selected communities that have high speed networking (e.g. NCAR and NERSC)
• Authenticated data streams for easier site access through firewalls
• Server side data processing (computing and cache embedded in the net)
• Information servers for global data catalogues
• Reliable data/file transfer (across system / network failures)
Middleware
• Quality of service guarantees for distributed, simulations
• Virtual data catalogues and work planners for reconstituting the data on demand
Climate(5 yr)
Enable the analysis of model data by all of the collaborating community
• Robust access to large quantities of data
Climate(5-10 yr)
Integrated climate simulation that includes all high-impact factors
• Robust networks supporting distributed simulation -adequate bandwidth and latency for remote analysis and visualization of massive datasets
Vision for the Future Process of Science
analysis was driven by
18
Evolving Quantitative Science Requirements for Networks
Science Areas Today End2EndThroughput
5 years End2End Throughput
5-10 Years End2End Throughput
Remarks
High Energy Physics
0.5 Gb/s 100 Gb/s 1000 Gb/s high bulk throughput
Climate (Data & Computation)
0.5 Gb/s 160-200 Gb/s N x 1000 Gb/s high bulk throughput
SNS NanoScience Not yet started 1 Gb/s 1000 Gb/s + QoS for control channel
remote control and time critical throughput
Fusion Energy 0.066 Gb/s(500 MB/s burst)
0.198 Gb/s(500MB/20 sec. burst)
N x 1000 Gb/s time critical throughput
Astrophysics 0.013 Gb/s(1 TBy/week)
N*N multicast 1000 Gb/s computational steering and collaborations
Genomics Data & Computation
0.091 Gb/s(1 TBy/day)
100s of users 1000 Gb/s + QoS for control channel
high throughput and steering
19
Observed Drivers for ESnet Evolution
• Are we seeing the predictions of two years ago come true?
• Yes!
20
0
50
100
150
200
250
300
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
OSC Traffic Increases by 1.9-2.0 X Annually
Annual growth in the past five years has increased from 1.7x annually to just over 2.0x annually.
TByt
es/M
onth
ESnet is currently transporting about 250 terabytes/mo.(250,000,000 MBy/mo.)
ESnet Monthly Accepted Traffic
21
Fermilab
(US) →
CERN
SLAC (US) →
IN2P3 (FR)
1 Te
raby
te/d
ay
SLAC (US) →
INFN Padva (
IT)
Fermilab
(US) →
U. Chica
go (U
S)
CEBAF (US) →
IN2P3 (FR)
INFN Padva
(IT) →
SLAC (US)
U. Toro
nto (C
A) →Ferm
ilab (US)
Helmholt
z-Karls
ruhe (D
E)→SLAC (U
S)
DOE Lab →
DOE Lab
DOE Lab →
DOE Lab
SLAC (U
S) →JA
NET (UK)
Fermilab
(US) →
JANET (U
K)
Argonne
(US) →
Level3 (U
S)
Argonne
→SURFne
t (NL)
IN2P3 (FR) →
SLAC (U
S)
Fermilab
(US) →
INFN Padva (
IT)
A small number of science users
account for a significant
fraction of all ESnet traffic
Since BaBar data analysis started, the top 20 ESnet flows have consistently accounted for > 50% of ESnet’s monthly total traffic (~130 of 250 TBy/mo)
ESnet Top 20 Data Flows, 24 hr. avg., 2004-04-20
ESnet is Engineered to Move a Lot of Data
FNAL (US) →
IN2P3 (FR)
2.2 Terabytes
SLAC (U
S) →IN
FN Padua
(IT)
5.9 Tera
bytes
U. Toro
nto (C
A) →Ferm
ilab (US)
0.9 Terabytes
SLAC (U
S)→Helm
holtz-K
arlsruh
e (DE)
0.9 Terabytes
SLAC (U
S) →IN
2P3 (
FR)
5.3 Tera
bytes
CERN →FNAL (U
S)
1.3 Terabytes
FNAL (US) →
U. Nijm
egen (N
L)
1.0 Terabytes
FNAL (US)→
Helmholt
z-Karls
ruhe (
DE)
0.6 Terabytes
FNAL (US)→
SDSC (US)
0.6 Teraby
tes
U. Wisc
. (US)→
FNAL (US
0.6 Teraby
tes
The traffic is not transient: Daily and weekly averages are about the same.
SLAC is a prototype for what will happen when Climate, Fusion, SNS, Astrophysics, etc., start to ramp up the next generation science
ESnet Top 10 Data Flows, 1 week avg., 2004-07-01
23
ESnet is a Critical Element of Large-Scale Science
• ESnet is a critical part of the large-scale scienceinfrastructure of high energy physics experiments, climate modeling, magnetic fusion experiments, astrophysics data analysis, etc.
• As other large-scale facilities – such as SNS – turn on, this will be true across DOE
24
Science Mission Critical Infrastructure• ESnet is a visible and critical piece of general DOE science
infrastructureo if ESnet fails, tens of thousands of DOE and University users know it
within minutes if not seconds
• Requires high reliability and high operational security in theo network operations, ando ESnet infrastructure support – the systems that support the operation
and management of the network and services- Secure and redundant mail and Web systems are central to the operation
and security of ESnet– trouble tickets are by email– engineering communication by email– engineering database interface is via Web
- Secure network access to Hub equipment- Backup secure telephony access to all routers- 24x7 help desk (joint w/ NERSC) and 24x7 on-call network engineers
Automated, real-time monitoring of traffic levels and operating state of some 4400 network entities is the primary network
operational and diagnosis toolNetwork Configuration OSPF Metrics
(routing and connectivity)Performance
SecureNet IBGP Mesh(routing and connectivity)
Hardware Configuration
26
ESnet’s Physical Infrastructure
Picture detail
Equipment rack detail at NYC Hub, 32 Avenue of the Americas (one
of ESnet’s core optical ring sites)
27
Cisco 7206AOA-AR1
(low speed links to MIT & PPPL)($38,150 list)
Juniper M20AOA-PR1
(peering RTR)($353,000 list)
Juniper T320AOA-CR1
(Core router)($1,133,000
list)
Juniper OC192Optical Ring
Interface (the AOA end of
the OC192 to CHI
($195,000 list)
Juniper OC48Optical Ring
Interface (the AOA end of the OC48 to
DC-HUB($65,000 list)
AOAPerformance Tester
($4800 list)
Qwest DS3 DCX
DC / AC Converter($2200 list)
Lightwave SecureTerminal Server
($4800 list)ESnet core
equipment @ Qwest
32 AofA HUB NYC, NY
(~$1.8M, list)
Sentry power 48v 30/60 amp panel
($3900 list)
Sentry power 48v 10/25 amp panel
($3350 list)
Typical Equipment of an ESnet Core Network Hub
28
LBNLPPPL
BNL
AMES
Remote Engineer• partial duplicate infrastructure
DNS
Remote Engineer• partial duplicate
infrastructure
TWCRemoteEngineer
Disaster Recovery and Stability
• The network must be kept available even if, e.g., the West Coastis disabled by a massive earthquake, etc.
ATL HUB
SEA HUB
ALBHUB
NYC HUBS
DC HUB
ELP HUB
CHI HUB
SNV HUB Duplicate InfrastructureCurrently deploying full replication of the NOC databases and servers and Science Services databases in the NYC Qwest carrier hub
Engineers, 24x7 Network Operations Center, generator backed power
• Spectrum (net mgmt system)• DNS (name – IP address
translation)• Eng database• Load database• Config database• Public and private Web• E-mail (server and archive)• PKI cert. repository and
revocation lists• collaboratory authorization
service
Reliable operation of the network involves• remote NOCs• replicated support infrastructure• generator backed UPS power at all critical network and infrastructure locations
• non-interruptible core - ESnet core operated without interruption through
o N. Calif. Power blackout of 2000o the 9/11/2001 attacks, ando the Sept., 2003 NE States power blackout
29
ESnet WAN Security and Cyberattack Defense• Cyber defense is a new dimension of ESnet security
o Security is now inherently a global problemo As the entity with a global view of the network, ESnet has an
important role in overall security
30 minutes after the Sapphire/Slammer worm was released, 75,000 hosts running Microsoft's SQL Server (port 1434) were infected.
(“The Spread of the Sapphire/Slammer Worm,” David Moore (CAIDA & UCSD CSE), Vern Paxson (ICIR &LBNL), Stefan Savage (UCSD CSE), Colleen Shannon (CAIDA), Stuart Staniford (Silicon Defense), Nicholas Weaver (Silicon Defense & UC Berkeley EECS) http://www.cs.berkeley.edu/~nweaver/sapphire ) Jan., 2003
30
ESnet and Cyberattack Defense
Sapphire/Slammer worm infection hits creating almost a full Gb/s (1000 megabit/sec.) traffic spike on the ESnet backbone
31
Cyberattack Defense
LBNL
ESnet
router
router
borderrouter
X
peeringrouter
Lab
Lab
gatewayrouter
ESnet second response – filter traffic from outside of ESnet
Lab first response – filter incoming traffic at their ESnet gateway router
ESnet third response – shut down the main peering paths and provide only limited bandwidth paths for specific
“lifeline” services
Xpeeringrouter
gatewayrouter
border router
router
attack trafficX
ESnet first response –filters to assist a site
Sapphire/Slammer worm infection created a Gb/s of traffic on the ESnet core until filters were put in place (both into and out of sites) to damp it out.
Science Services: Support for Shared, Collaborative Science Environments
• X.509 identity certificates and Public Key Infrastructure provides the basis of secure, cross-site authentication of people and systems (www.doegrids.org)o ESnet negotiates the cross-site, cross-organization, and
international trust relationships to provide policies that are tailored to collaborative science in order to permit sharing computing and data resources, and other Grid services
o Certification Authority (CA) issues certificates after validating request against policy
o This service was the basis of the first routine sharing of HEP computing resources between US and Europe
33
Science Services: Public Key Infrastructure
* Report as of July 15,2004
34
Voice, Video, and Data Tele-Collaboration Service
• Another highly successful ESnet Science Service is the audio, video, and data teleconferencing service to support human collaborationo Seamless voice, video, and data teleconferencing is
important for geographically dispersed scientific collaborators
o ESnet currently provides to more than a thousand DOE researchers and collaborators worldwide- H.323 (IP) videoconferences (4000 port hours per month and rising)- audio conferencing (2500 port hours per month) (constant)- data conferencing (150 port hours per month)- Web-based, automated registration and scheduling for all of these
services
• Huge cost savings for the Labs
35
ESnet’s Evolution over the Next 10-20 Years
• Upgrading ESnet to accommodate the anticipated increase from the current 100%/yr traffic growth to 300%/yr over the next 5-10 years is priority number 7 out of 20 in DOE’s “Facilities for the Future of Science – A Twenty Year Outlook”
• Based on the requirements of the OSC Network Workshops, ESnet must addresso Capable, scalable, and reliable production IP networking
- University and international collaborator connectivity- Scalable, reliable, and high bandwidth site connectivity
o Network support of high-impact science- provisioned circuits with guaranteed quality of service
(e.g. dedicated bandwidth)
o Science Services to support Grids, collaboratories, etc
36
New ESnet Architecture to Accommodate OSC• The future requirements cannot be met with the
current, telecom provided, hub and spoke architecture of ESnet
• The core ring has good capacity and resiliency against single point failures, but the point-to-point tail circuits are neither reliable nor scalable to the required bandwidth
ESnetCore
New York (AOA)Chicago (CHI)
Sunnyvale (SNV)Atlanta (ATL)
Washington, DC (DC)
El Paso (ELP)
DOE sites
37
SCI
C&Cinstrument
compute
storage
cache &compute
Evolving Requirements for DOE Science Network Infrastructure
C
S
C
C
S
I
C
S
C
C
S
I
C
S
C
C
S
IC&C
C&C
C&C
C&
C
C&C
C&C
C
S
C
C
S
IC&C
C&C
C&C
C&
C
C&C
C&C
1-40 Gb/s,end-to-end
guaranteedbandwidthpaths
100-200 Gb/s,end-to-end
• In the near term applicationsneed higher bandwidth
• high bandwidth• QoS
• high bandwidth and QoS• network resident cache and compute
elements• robust bandwidth (multiple paths)
• high bandwidth and QoS• network resident cache and compute
elements
1-3
yr R
equi
rem
ents
3-5
yr R
equi
rem
ents
2-4
yr R
equi
rem
ents
4-7
yr R
equi
rem
ents
38
A New Architecture
• With the current architecture ESnet cannot addresso the increasing reliability requirementso the long-term bandwidth needs
(incrementally increasing tail circuit bandwidth is too expensive – it will not scale to what OSC needs)
- LHC will need dedicated 10 Gb/s into and out of FNAL and BNL
• ESnet can benefit fromo Engaging the research and education networking
community for advanced technologyo Leveraging the R&E community investment in fiber and
networks
39
A New Architecture
• ESnet new architecture goals: full redundant connectivity for every site and high-speed access for every site (at least 10 Gb/s)
• Three part strategy1) MAN rings provide dual site connectivity and much higher
site-to-core bandwidth2) A second core will provide
- multiply connected MAN rings for protection against hub failure- extra core capacity- a platform for provisioned, guaranteed bandwidth circuits- alternate path for production IP traffic- carrier neutral hubs
3) a high-reliability IP core (like the current ESnet core)
40
A New ESnet Architecture
EuropeAsia-
Pacific
ESnetExisting
Core
New York(AOA)
Chicago (CHI)
Sunnyvale(SNV)
Washington,DC (DC)
El Paso (ELP)
DOE/OSC Labs
New hubs
Existing hubs
2nd Core (e.g. NLR)
Possible new hubs
Atlanta (ATL)
MetropolitanArea
Rings
41
ESnet Beyond FY07
DENDEN
ELPELP
ALBALBATLATL
MANs
High-speed cross connects with Internet2/AbileneMajor DOE Office of Science Sites
Japan
CERNEurope
SDGSDG
AsiaPacSEASEA
NLR – ESnet hubsQwest – ESnet hubs
SNVSNV
Europe
10Gb/s30Bg/s40Gb/s
Japan
CHICHI
High-impact science coreLab suppliedMajor international
2.5 Gbs10 Gbs
Future phases
Production IP ESnet core
DCDC
Japan
NYCNYC
42
Conclusions• ESnet is an infrastructure that is critical to DOE’s
science mission
• Focused on the Office of Science Labs, but serves many other parts of DOE
• ESnet is working hard to meet the current and future networking need of DOE mission science in several ways:o Evolving a new high speed, high reliability, leveraged
architectureo Championing several new initiatives which will keep
ESnet’s contributions relevant to the needs of our community
43
Reference -- Planning Workshops
• High Performance Network Planning Workshop, August 2002http://www.doecollaboratory.org/meetings/hpnpw
• DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for Large-Scale Science Applications, April 2003
http://www.csm.ornl.gov/ghpn/wk2003
• Science Case for Large Scale Simulation, June 2003http://www.pnl.gov/scales/
• DOE Science Networking Roadmap Meeting, June 2003http://www.es.net/hypertext/welcome/pr/Roadmap/index.html
• Workshop on the Road Map for the Revitalization of High End Computing, June 2003
http://www.cra.org/Activities/workshops/nitrdhttp://www.sc.doe.gov/ascr/20040510_hecrtf.pdf (public report)
• ASCR Strategic Planning Workshop, July 2003http://www.fp-mcs.anl.gov/ascr-july03spw
• Planning Workshops-Office of Science Data-Management Strategy, March & May 2004
o http://www-conf.slac.stanford.edu/dmw2004 (report coming soon)