Dan MagorianDirector of
Engineering & Operations
November 30, 2005
What’s MAX Been Up to Lately?
Presentation to MAX membership
Welcome New MAX Participants!• Since April, 3 new participants joined for total of 38:
• USDA, Beltsville Agriculture Research Ctr• Library of Congress• Northrop Grumman
• Folks who are near term:• Health & Human Services (parent agency of NIH)• Lab for Telecom Science (LTS)• Venter Institute
• And folks that are in process:• Johns Hopkins APL campus • Inter-American Development Bank• DC networks (welcome back Chris Peabody)• Sun Microsystems Compute Grid @ 10G
• Max will likely be at 44 connectors by early 06• Forecast is for continued growth!!
Participant Consortium of 38 Participants Directly Connected Higher Education, Federal Agency, and Non-
Profit Institutions• Higher Education (18)
• Baltimore Education & Research Network (BERnet)• Catholic University• Georgetown University• George Washington University• Johns Hopkins University• Montgomery College• National Consortium for Supercomputing
Applications / ACCESS• Network Virginia (aggregating the State of Virginia)• Smithsonian Institution• Southern Universities Research Association
(SURA)• University of California, D.C. campus• University Consortium for Advanced Internet
Development (UCAID / Internet2)• University of Maryland, College Park• University of Maryland, Baltimore• University of Maryland, Baltimore Co.• Univ. System of Maryland Network (aggregating 11
campuses)• University of Southern California, Information
Sciences Institute / East• Washington Research Library Consortium
rev. 10/27
• Federal Labs and Agencies (14)• Library of Congress• NASA / GSFC• National Archives and Records Administration
(NARA)• National Science Foundation (NSF)• National Institutes of Health (NIH)• Nat’l Institutes of Standards and Tech. (NIST)• National Library of Medicine (NLM)• National Oceanic and Atmospheric Administration
(NOAA)• Naval Research Lab & ATDnet• U.S. Census• USDA, Beltsville Agriculture Research Ctr• U.S. Department of State (through GWU)• U.S. Geological Survey• U.S. Holocaust Memorial Museum
• Corporate and Non-profit (6)• Howard Hughes Med. Institute• Fujitsu Labs of America• Northrop Grumman• The Institute for Genomic Research• Windber Professional Services, Inc• World Bank
Introducing new MAX production engineers
• Dave Diller, Senior Network Engineer• Chris Tsonis, Services Engineer
• And most people know:• Quang Bach, Senior Network Engineer
• We’re very happy to have them, and we have more DRAGON and MAX production openings to fill soon!
MAXActivities
Production Net Activities
ATDNet-V2Research
Technologies
Internet2 HOPI Test
Bed support
MAX is More than Production IP Networking
NGIX/EastFedNet peer
point
DRAGONExperimental Networking
Related StrategicActivities
AtlanticWave,Quilt,
Research Activities
Layer 3 IP NetworkServices
Layer 1 Optical Services
Additional Funded
Research
For new folks:A Brief Background & History
• Founding consortium: Georgetown U., George Washington U., U. of Maryland, & Virginia Tech.
• UMD hosts MAX’s administration.• 1997: 155 mb atm network to vBNS connecting 4 univs.• 2000: Built regional dark fiber network lit by dwdm lambdas
connected to Abilene/Internet2 at 2.5 gb .• Evolved into a rapidly-growing group of universities, federal
agencies, and commercial and non-profit institutions emphasizing advanced networking.
• Currently operating optical OC-48 network in the DC, suburban MD, N Va, Baltimore metro areas.
• Doubled network size in 2-3 years
MAX Production Network Services
• IPv4 unicast, IPv4 multicast, and native IPv6 unicast.
• Internet2 (Abilene network) regional aggregation
• ISP resale (Qwest DIA, just added Cogent)
• NGIX/East FedNet peer point operation
• National Lambda Rail (NLR) aggregation and fanout
• Lambda transport on MAX dwdm optical network
• MPLS tunnel transport in MAX IP network
• Participant Colocation at UMD, Qwest, & Level3 pops
• Network monitoring and uptime/usage reporting
Strong Carrier Partnerships
• With 6 (soon 7) MAX pop locations around region, participants can connect at one or more locations via dark fiber or lit services via aggressive low-cost carriers, as well as traditional telcos.
• 80% & growing of participant circuits ethernet.• Strong partnerships with carriers (11 & growing!)
• Qwest, Level 3, Fibergate, Starpower/RCN, Yipes, Allied Telecom, AboveNet, Looking Glass Nets, Verizon, MCI, ATT.
Mid-Atlantic Crossroads
GEANT
Network Virginia
National/InternationalPeering Networks
Regional NetworkParticipants ATDnet
ESnet
NGIX-East
DRENvBNS
NISN
NRENAbilene
MAXMAXRegional
Infrastructure
Qwest ISP
Institutional Connectors
Univ System Md
DRAGON
Cogent
MAX upstream network peerings
• Abilene (diverse: 2.5 gb primary /1 gb backup, 10 gb)• vBNS network (1 gb)• Defense Research & Exp Net (DREN) (1 gb)• Energy Sciences net (ESnet) (1 gb, becoming 10 gb)• NASA’s NISN and NREN (1 gb ea)• USGS DOI net (1gb)• Qwest DIA ISP service offering (1 gb, adding 2nd 1 gb)• National Lambda Rail (lambdas, no peerings “yet”)• GEANT, “European Abilene”, (10 gb, spring 06)• Atlantic Wave distributed peer fabric, (summer 06)
MAX Commodity Internet Services
• 3rd year of Qwest DIA service, resold to any participant for primary or secondary ISP.
• ISP routes kept separate from I2 routes via RFC 2547 MPLS vpns (Separate routing tables, separate peerings)
• 13 participants, 650 mb aggregated and growing, added Census, NOAA, and raised JHU during 2004-5, up from 11 participants at 405 Mb. HHS coming onboard shortly with additional 400 mb.
• Dropped cost from $70 mb/m to $55 mb/m in year 3 (start Oct)• Talked Qwest into 2nd redundant burstable peering for minimum
incremental cost, prototype for Quilt offering. Online shortly.• Provided emergency outbound ISP to NOAA for Katrina relief.• Just added Cogent as very-low-cost ISP via resale through
GWU. Peering established, just waiting completion of MOU.
The Quilt Regions
SOX(AL)
(AZ)
GPN(AR)
Cenic(CA)
FRGP(CO)
(CT)
(FL)
SOX(GA)
MREN(IL)
(IN)
(IA)
GPN(KS)
Texas(LA)
NoX
(MA)
Merit(MI)
GPN(MO)
Texas(MS)
GPN(NE)
Cenic(NV)
(NH)
(NJ)
(NM)
NYSERnet(NY)
NC Giga(NC)
NorthernLights(ND)
OARnet(OH)
(OK)
OREGON(OR)
PSC(PA)
SOX (SC)
NorthernLights(SD)
SOX(TN)
Texas Gigapop(TX)
(UT)
(VT)
MREN(WI)
FRGP(WY)
PNWGP(WA)
PSC(WV)
NorthernLights(MN)
Net.VirgSoX(KY)
PNWGP(MO)
PNWGP(ID)
MAX(MD)
(AK)NET.Virg (KY)
N. Lights (ND, SD, MN)
NoX(ME,NH,VT,MA,CT,RI)
NWV (VI)
NYSERnet (NY)
OARnet (OH)
OREGON (OR)
PNWGP (WA,ID,MO) PSC (PS,WV)
SOX (TN,AL,GA,SC) Texas (TX,LA,MS)
MREN (WI, IL)
GPN (NE, KS, AR, IA)
MAX (MD, VI)
Merit (MI)
FRGP (CO, WY)
Cenic (CA, NV)
NC Giga (NC)
MAXNWV(VI)
(RI)
(DE)
(HI)
National Association of GigaPoPs
• MAX is a member of TheQuilt, a national association of GigaPoPs
• Provides collaboration on projects to benefit the individual GigaPoPs• E.g., Leverages multi-state buying power• ISP services with pricing based on aggregated
national bandwidth tiers• national pricing for commodity ISP services
Where we were Spring 05
OADM
BALT
Qwest
OC48c POS
OC48c POS
OC48c POS
Abilene
M160 Router
M160 Router
Aggregation Switch
T640 Router
M40e Router
ARLG
DCGW DCNE
CLPK
NGIX
Northern Virginia
Washington D.C.
College Park
Baltimore
MAX is in the middle of a huge expansion
• Upgrade and consolidation of layer 3 routers• Removed 3 Juniper M160s (EOLed)• Installed 2 Juniper T640s, cutting all customers over.• Router changeover essentially complete!
• Expansion of layer 1 core optical network• Backhaul with redundancy of switches at pops• Top-tier customer optical backhaul from any pop• Integrating second optical dwdm system (Movaz)• Optical backhaul of Abilene and Qwest ISP
• Expansion to new Level 3 pop (NLR) – already filled!• Easily 5x more complex than original network installation, esp
logistics to preserve service uptime.• It went well, other than some Qwest provisioning nightmares.
CLPKCLPK
CLPKCLPK
MCLNMCLN
ARLGARLG
DCGWDCGW DCNEDCNE
NLR
NGIX
T640T640T640
M40EM40E
Cogent
R&E Nets
Abilene
Qwest
Where we are Fall 051. Original Zhone dwdm over Qwest fiber2. Movaz dwdm over State Md fiber
Ring1
Ring2
OC48c w/ OC3 backup
What’s in Progress Now?• More optical rings ahead! • Large expansion of Movaz dwdm CLPK-LVL3 from 2 chassis to 8
• Upgrade to REII chassis, allowing protected lambdas• Changeout of 8 operating lambdas to new transceiver types• Ringout of path via strong partnership with NWMd (2nd MOU)
• New pop location at Equinix in Ashburn VA thanks to HHMI contribution! • AboveNet dwdm ring from Level3 Mclean to Equinix• Pick up second 1 gb Qwest ISP connection for protection/expansion
• New dwdm ring College Park to Baltimore replacing OC48, thanks to USM• Will be able to offer protected lambdas to Baltimore participants• Will offer protection to MAX backbone from fiber cuts etc.• Upgrade to 10G-capable Force10 ethernet switch in Baltimore
• Next phase Q1 06 will upgrade MAX backbone from OC48 to 10G, including Abilene connection. Already have added new top-tier OC48-10G participant bandwidth pricing ($200k/yr).
• High uptake on MAX lambda offering for NLR fanout, already 6+ 10Gs in planning for ESnet, GEANT, NOAA, NASA, etc
• Atlantic Wave distributed peerings to NYC and Miami for NGIX customers
CLPKCLPK
CLPKCLPK
CLPKCLPK
MCLNMCLN
BALTBALT
MCLNMCLN
ASHBASHB ARLGARLG
DCGWDCGW DCNEDCNE
NLR
NGIX
T640T640T640
M40EM40E
Cogent
R&E Nets
Abilene
Qwest
Qwest
Where we’ll be by Spring 061. Original Zhone dwdm over Qwest fiber2. Movaz dwdm over State Md fiber3. HHMI dwdm over Abovenet fiber4. USM dwdm over State Md fiber
Ring1
Ring2
Ring3
Ring4
Atlantic Wave Distributed Peering Topology
MIA
Miami, FL
MANLAN
NGIXATL
Atlanta, GA Washington, DC New York City, NY
ToSouth America
ToEurope &Canada,
Initial East Coast NLR 10G A-Wave Backbone
Future backbone Extensions within NA & EU
Ethernet fanout
10G ethernet via NLR
We want to encourage Redundant Peerings for participant protection (high-9’s)
LVL3T640
CLPKT640
USM NLM NetWVa
Movaz dwdm
Movaz dwdm
Already doing with USM and NetWorkVa, works well. NLM is in progress with theirs. Approach is to have 2 of your routers peer with 2 MAX routers over diverse fiber paths. MAX can provision lambdas to help accomplish this.
Offsite backup, the next “killer application”• Some MAX participants already have procured off-site backup of
critical services. This can be quite expensive through commercial offerings. Many other MAX folks are in process of figuring out options now. Offsite backup was one of biggest service requests spring MM.
• MAX already runs colo at UMD and Level3 (NOAA, HOPI, NASA). This has worked out well.
• Because of already being out of space in new Level3 suite, we’re intending to build a much larger expansion space nearby for our own and other’s racks. Northrop Grumman, right next to Level3 in Mclean VA, is a strong possibility for lots of racks/cages/cabinets and power.
• We would like to be able to offer rack/cage space to participants at low rates, using same simple “do-it-yourself” model with some “smart hands” help. Could provision lambdas back to your institutions to make racks become part of your networks w/o MAX L3 in middle.
• Beyond regional backup, potential for backup across NLR to remote areas of country. Those gigapops would probably be thrilled to do this.