Dan MagorianDirector of

Engineering & Operations

November 30, 2005

What’s MAX Been Up to Lately?

Presentation to MAX membership

Welcome New MAX Participants!• Since April, 3 new participants joined for total of 38:

• USDA, Beltsville Agriculture Research Ctr• Library of Congress• Northrop Grumman

• Folks who are near term:• Health & Human Services (parent agency of NIH)• Lab for Telecom Science (LTS)• Venter Institute

• And folks that are in process:• Johns Hopkins APL campus • Inter-American Development Bank• DC networks (welcome back Chris Peabody)• Sun Microsystems Compute Grid @ 10G

• Max will likely be at 44 connectors by early 06• Forecast is for continued growth!!

Participant Consortium of 38 Participants Directly Connected Higher Education, Federal Agency, and Non-

Profit Institutions• Higher Education (18)

• Baltimore Education & Research Network (BERnet)• Catholic University• Georgetown University• George Washington University• Johns Hopkins University• Montgomery College• National Consortium for Supercomputing

Applications / ACCESS• Network Virginia (aggregating the State of Virginia)• Smithsonian Institution• Southern Universities Research Association

(SURA)• University of California, D.C. campus• University Consortium for Advanced Internet

Development (UCAID / Internet2)• University of Maryland, College Park• University of Maryland, Baltimore• University of Maryland, Baltimore Co.• Univ. System of Maryland Network (aggregating 11

campuses)• University of Southern California, Information

Sciences Institute / East• Washington Research Library Consortium

rev. 10/27

• Federal Labs and Agencies (14)• Library of Congress• NASA / GSFC• National Archives and Records Administration

(NARA)• National Science Foundation (NSF)• National Institutes of Health (NIH)• Nat’l Institutes of Standards and Tech. (NIST)• National Library of Medicine (NLM)• National Oceanic and Atmospheric Administration

(NOAA)• Naval Research Lab & ATDnet• U.S. Census• USDA, Beltsville Agriculture Research Ctr• U.S. Department of State (through GWU)• U.S. Geological Survey• U.S. Holocaust Memorial Museum

• Corporate and Non-profit (6)• Howard Hughes Med. Institute• Fujitsu Labs of America• Northrop Grumman• The Institute for Genomic Research• Windber Professional Services, Inc• World Bank

Introducing new MAX production engineers

• Dave Diller, Senior Network Engineer• Chris Tsonis, Services Engineer

• And most people know:• Quang Bach, Senior Network Engineer

• We’re very happy to have them, and we have more DRAGON and MAX production openings to fill soon!

MAXActivities

Production Net Activities

ATDNet-V2Research

Technologies

Internet2 HOPI Test

Bed support

MAX is More than Production IP Networking

NGIX/EastFedNet peer

point

DRAGONExperimental Networking

Related StrategicActivities

AtlanticWave,Quilt,

Research Activities

Layer 3 IP NetworkServices

Layer 1 Optical Services

Additional Funded

Research

For new folks:A Brief Background & History

• Founding consortium: Georgetown U., George Washington U., U. of Maryland, & Virginia Tech.

• UMD hosts MAX’s administration.• 1997: 155 mb atm network to vBNS connecting 4 univs.• 2000: Built regional dark fiber network lit by dwdm lambdas

connected to Abilene/Internet2 at 2.5 gb .• Evolved into a rapidly-growing group of universities, federal

agencies, and commercial and non-profit institutions emphasizing advanced networking.

• Currently operating optical OC-48 network in the DC, suburban MD, N Va, Baltimore metro areas.

• Doubled network size in 2-3 years

MAX Production Network Services

• IPv4 unicast, IPv4 multicast, and native IPv6 unicast.

• Internet2 (Abilene network) regional aggregation

• ISP resale (Qwest DIA, just added Cogent)

• NGIX/East FedNet peer point operation

• National Lambda Rail (NLR) aggregation and fanout

• Lambda transport on MAX dwdm optical network

• MPLS tunnel transport in MAX IP network

• Participant Colocation at UMD, Qwest, & Level3 pops

• Network monitoring and uptime/usage reporting

Strong Carrier Partnerships

• With 6 (soon 7) MAX pop locations around region, participants can connect at one or more locations via dark fiber or lit services via aggressive low-cost carriers, as well as traditional telcos.

• 80% & growing of participant circuits ethernet.• Strong partnerships with carriers (11 & growing!)

• Qwest, Level 3, Fibergate, Starpower/RCN, Yipes, Allied Telecom, AboveNet, Looking Glass Nets, Verizon, MCI, ATT.

Mid-Atlantic Crossroads

GEANT

Network Virginia

National/InternationalPeering Networks

Regional NetworkParticipants ATDnet

ESnet

NGIX-East

DRENvBNS

NISN

NRENAbilene

MAXMAXRegional

Infrastructure

Qwest ISP

Institutional Connectors

Univ System Md

DRAGON

Cogent

MAX upstream network peerings

• Abilene (diverse: 2.5 gb primary /1 gb backup, 10 gb)• vBNS network (1 gb)• Defense Research & Exp Net (DREN) (1 gb)• Energy Sciences net (ESnet) (1 gb, becoming 10 gb)• NASA’s NISN and NREN (1 gb ea)• USGS DOI net (1gb)• Qwest DIA ISP service offering (1 gb, adding 2nd 1 gb)• National Lambda Rail (lambdas, no peerings “yet”)• GEANT, “European Abilene”, (10 gb, spring 06)• Atlantic Wave distributed peer fabric, (summer 06)

MAX Commodity Internet Services

• 3rd year of Qwest DIA service, resold to any participant for primary or secondary ISP.

• ISP routes kept separate from I2 routes via RFC 2547 MPLS vpns (Separate routing tables, separate peerings)

• 13 participants, 650 mb aggregated and growing, added Census, NOAA, and raised JHU during 2004-5, up from 11 participants at 405 Mb. HHS coming onboard shortly with additional 400 mb.

• Dropped cost from $70 mb/m to $55 mb/m in year 3 (start Oct)• Talked Qwest into 2nd redundant burstable peering for minimum

incremental cost, prototype for Quilt offering. Online shortly.• Provided emergency outbound ISP to NOAA for Katrina relief.• Just added Cogent as very-low-cost ISP via resale through

GWU. Peering established, just waiting completion of MOU.

The Quilt Regions

SOX(AL)

(AZ)

GPN(AR)

Cenic(CA)

FRGP(CO)

(CT)

(FL)

SOX(GA)

MREN(IL)

(IN)

(IA)

GPN(KS)

Texas(LA)

NoX

(MA)

Merit(MI)

GPN(MO)

Texas(MS)

GPN(NE)

Cenic(NV)

(NH)

(NJ)

(NM)

NYSERnet(NY)

NC Giga(NC)

NorthernLights(ND)

OARnet(OH)

(OK)

OREGON(OR)

PSC(PA)

SOX (SC)

NorthernLights(SD)

SOX(TN)

Texas Gigapop(TX)

(UT)

(VT)

MREN(WI)

FRGP(WY)

PNWGP(WA)

PSC(WV)

NorthernLights(MN)

Net.VirgSoX(KY)

PNWGP(MO)

PNWGP(ID)

MAX(MD)

(AK)NET.Virg (KY)

N. Lights (ND, SD, MN)

NoX(ME,NH,VT,MA,CT,RI)

NWV (VI)

NYSERnet (NY)

OARnet (OH)

OREGON (OR)

PNWGP (WA,ID,MO) PSC (PS,WV)

SOX (TN,AL,GA,SC) Texas (TX,LA,MS)

MREN (WI, IL)

GPN (NE, KS, AR, IA)

MAX (MD, VI)

Merit (MI)

FRGP (CO, WY)

Cenic (CA, NV)

NC Giga (NC)

MAXNWV(VI)

(RI)

(DE)

(HI)

National Association of GigaPoPs

• MAX is a member of TheQuilt, a national association of GigaPoPs

• Provides collaboration on projects to benefit the individual GigaPoPs• E.g., Leverages multi-state buying power• ISP services with pricing based on aggregated

national bandwidth tiers• national pricing for commodity ISP services

Where we were Spring 05

OADM

BALT

Qwest

OC48c POS

OC48c POS

OC48c POS

Abilene

M160 Router

M160 Router

Aggregation Switch

T640 Router

M40e Router

ARLG

DCGW DCNE

CLPK

NGIX

Northern Virginia

Washington D.C.

College Park

Baltimore

MAX is in the middle of a huge expansion

• Upgrade and consolidation of layer 3 routers• Removed 3 Juniper M160s (EOLed)• Installed 2 Juniper T640s, cutting all customers over.• Router changeover essentially complete!

• Expansion of layer 1 core optical network• Backhaul with redundancy of switches at pops• Top-tier customer optical backhaul from any pop• Integrating second optical dwdm system (Movaz)• Optical backhaul of Abilene and Qwest ISP

• Expansion to new Level 3 pop (NLR) – already filled!• Easily 5x more complex than original network installation, esp

logistics to preserve service uptime.• It went well, other than some Qwest provisioning nightmares.

CLPKCLPK

CLPKCLPK

MCLNMCLN

ARLGARLG

DCGWDCGW DCNEDCNE

NLR

NGIX

T640T640T640

M40EM40E

Cogent

R&E Nets

Abilene

Qwest

Where we are Fall 051. Original Zhone dwdm over Qwest fiber2. Movaz dwdm over State Md fiber

Ring1

Ring2

OC48c w/ OC3 backup

What’s in Progress Now?• More optical rings ahead! • Large expansion of Movaz dwdm CLPK-LVL3 from 2 chassis to 8

• Upgrade to REII chassis, allowing protected lambdas• Changeout of 8 operating lambdas to new transceiver types• Ringout of path via strong partnership with NWMd (2nd MOU)

• New pop location at Equinix in Ashburn VA thanks to HHMI contribution! • AboveNet dwdm ring from Level3 Mclean to Equinix• Pick up second 1 gb Qwest ISP connection for protection/expansion

• New dwdm ring College Park to Baltimore replacing OC48, thanks to USM• Will be able to offer protected lambdas to Baltimore participants• Will offer protection to MAX backbone from fiber cuts etc.• Upgrade to 10G-capable Force10 ethernet switch in Baltimore

• Next phase Q1 06 will upgrade MAX backbone from OC48 to 10G, including Abilene connection. Already have added new top-tier OC48-10G participant bandwidth pricing ($200k/yr).

• High uptake on MAX lambda offering for NLR fanout, already 6+ 10Gs in planning for ESnet, GEANT, NOAA, NASA, etc

• Atlantic Wave distributed peerings to NYC and Miami for NGIX customers

CLPKCLPK

CLPKCLPK

CLPKCLPK

MCLNMCLN

BALTBALT

MCLNMCLN

ASHBASHB ARLGARLG

DCGWDCGW DCNEDCNE

NLR

NGIX

T640T640T640

M40EM40E

Cogent

R&E Nets

Abilene

Qwest

Qwest

Where we’ll be by Spring 061. Original Zhone dwdm over Qwest fiber2. Movaz dwdm over State Md fiber3. HHMI dwdm over Abovenet fiber4. USM dwdm over State Md fiber

Ring1

Ring2

Ring3

Ring4

Atlantic Wave Distributed Peering Topology

MIA

Miami, FL

MANLAN

NGIXATL

Atlanta, GA Washington, DC New York City, NY

ToSouth America

ToEurope &Canada,

Initial East Coast NLR 10G A-Wave Backbone

Future backbone Extensions within NA & EU

Ethernet fanout

10G ethernet via NLR

We want to encourage Redundant Peerings for participant protection (high-9’s)

LVL3T640

CLPKT640

USM NLM NetWVa

Movaz dwdm

Movaz dwdm

Already doing with USM and NetWorkVa, works well. NLM is in progress with theirs. Approach is to have 2 of your routers peer with 2 MAX routers over diverse fiber paths. MAX can provision lambdas to help accomplish this.

Offsite backup, the next “killer application”• Some MAX participants already have procured off-site backup of

critical services. This can be quite expensive through commercial offerings. Many other MAX folks are in process of figuring out options now. Offsite backup was one of biggest service requests spring MM.

• MAX already runs colo at UMD and Level3 (NOAA, HOPI, NASA). This has worked out well.

• Because of already being out of space in new Level3 suite, we’re intending to build a much larger expansion space nearby for our own and other’s racks. Northrop Grumman, right next to Level3 in Mclean VA, is a strong possibility for lots of racks/cages/cabinets and power.

• We would like to be able to offer rack/cage space to participants at low rates, using same simple “do-it-yourself” model with some “smart hands” help. Could provision lambdas back to your institutions to make racks become part of your networks w/o MAX L3 in middle.

• Beyond regional backup, potential for backup across NLR to remote areas of country. Those gigapops would probably be thrilled to do this.

MAX networking efforts are directed toward extending the services & performance of the national high performance backbones into the region and to leverage the resources of MAX for the good of all participants.