Scaling the Internet CoreScaling the

Internet CoreWilliam B. Norton

Co-founder and Chief Technical LiaisonEquinix, Inc.

wbn@equinix.comwww.equinix.com

William B. NortonCo-founder and Chief Technical Liaison

Equinix, Inc.wbn@equinix.comwww.equinix.com

PerspectivePerspective

• Practical: Application of technology• Neutral Internet Exchanges• Environment for Exploitation of Optical

Efficiencies• ISP Peering and Transit• Massive Scaling of Content

• Environment for launching Next Generation Services

• Practical: Application of technology• Neutral Internet Exchanges• Environment for Exploitation of Optical

Efficiencies• ISP Peering and Transit• Massive Scaling of Content

• Environment for launching Next Generation Services

Service Provider ScalabilityService Provider Scalability

Position: The current service provider interconnection model does not scale well

Service providers need solutions that provide:• Control over bandwidth provisioning• Choice of providers• Efficient scaling

Position: The current service provider interconnection model does not scale well

Service providers need solutions that provide:• Control over bandwidth provisioning• Choice of providers• Efficient scaling

NSFNET to Post-NSFNET Era InternetNSFNET to Post-NSFNET Era Internet

No Internet presentation complete without…

BARRNet

NWNet

Sesqui

NEARNet

NyserNetMerit

NCAR

OarNet

Regional

RegionalRegionalNSFNET

SDSC

AT&T

MCI

Sprint

UUNet

ANSNWNet NyserNet

MeritOarNet

Regional

Regional RegionalRegional

NSFNET (1987-1994) Regional Tech Meetings• Performance &

Operations• Reporting &

Coordination• Routing & Network

PlanningPrivatization (1994-Now)

North American Network Operators Group

(NANOG) Meetings

NSFNET (1987-1994) Regional Tech Meetings• Performance &

Operations• Reporting &

Coordination• Routing & Network

PlanningPrivatization (1994-Now)

North American Network Operators Group

(NANOG) Meetings

Internet StatisticsInternet Statistics

• 87.5% of all Internet statistics are made up

• 87.5% of all Internet statistics are made up

Made Up

Real

Source: I made it up

Evolution of Interconnection StrategiesEvolution of Interconnection Strategies

• Carrier Internet Exchanges• Technologies: ATM, distributed (and colo)

• Network Access Points (NAPs)• Technologies: FDDI (with colo)• Congestion issues

• Direct (point-to-point) interconnections• Technologies: N x OC-3, N x OC-12

• Carrier Internet Exchanges• Technologies: ATM, distributed (and colo)

• Network Access Points (NAPs)• Technologies: FDDI (with colo)• Congestion issues

• Direct (point-to-point) interconnections• Technologies: N x OC-3, N x OC-12

OC-3=155Mbps, OC-12=622MbpsOC-48=2.5Gbps, OC-192=10Gbps

OC-768=40Gbps

OC-3=155Mbps, OC-12=622MbpsOC-48=2.5Gbps, OC-192=10Gbps

OC-768=40Gbps

Direct Circuit ISP Interconnection StrategyDirect Circuit ISP Interconnection Strategy

Point-to-Point Circuits• Regional

interconnects• Now: N x OC-12• Insufficient carrier

bandwidth• Carriers can’t

provision fast enough• No traffic aggregation

efficiency

Doesn’t scale well

Point-to-Point Circuits• Regional

interconnects• Now: N x OC-12• Insufficient carrier

bandwidth• Carriers can’t

provision fast enough• No traffic aggregation

efficiency

Doesn’t scale well

U U U U

G G G G

A A A A

C C C C

S S S S

Current Problems at the “Core”Current Problems at the “Core”• Time to provision services (9-18 months)

• Carrier• ISPs• Content (ASP, E*Com, hosting, etc.)

• Scaling at the Core• Interconnection bandwidth (OC-12->OC-48-

>OC-192)• Transport (try and order an OC-12!)• ISP Peering (exponential growth in

users/traffic)• ISP Transit (demand exceeds supply, choice)• Content Peering (unicast streaming, content

generation)• Edge (DSL – AOL+)

• Time to provision services (9-18 months)• Carrier• ISPs• Content (ASP, E*Com, hosting, etc.)

• Scaling at the Core• Interconnection bandwidth (OC-12->OC-48-

>OC-192)• Transport (try and order an OC-12!)• ISP Peering (exponential growth in

users/traffic)• ISP Transit (demand exceeds supply, choice)• Content Peering (unicast streaming, content

generation)• Edge (DSL – AOL+)

Tools for Service Provider ScalabilityTools for Service Provider Scalability

• Application of Technology• Dark Fiber Providers

• Control over provisioning of capacity

• WDM/DWDM• Huge capacity into

exchange• Enabled Bandwidth• Aggregation Efficiency

• Half filled pipes• Collocated Content

Providers• Environment to exploit

optical efficiencies

• Application of Technology• Dark Fiber Providers

• Control over provisioning of capacity

• WDM/DWDM• Huge capacity into

exchange• Enabled Bandwidth• Aggregation Efficiency

• Half filled pipes• Collocated Content

Providers• Environment to exploit

optical efficiencies WDM=Wave Division Multiplexing,

DWDM=Dense Wave Division Multiplexing

S

U U U U

G G G G

A A A A

C C C C

S S S

Neutral Internet Business Exchange

CP

CP

CP

Scaling of Interconnection MethodsScaling of Interconnection Methods

• Traffic (Content, Peering, Transit) Exchange at OC-3

• Traffic (Content, Peering, Transit) Exchange at OC-3

Cost Comparison of Interconnection Strategies

$-

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64

# of participants

Mo

nth

ly C

ost

of

Inte

rco

nn

ecti

on

Direct C

ircuit I

nterconnection: Linear

Exchange : Stepped Efficient

Fiber Exchange:Linear Efficient

Future of the Internet Exchange EnvironmentFuture of the Internet Exchange Environment

MarketPlace: Stratification of component operations, fast creation & provisioning of next generation packaged operations servicesMarketPlace: Stratification of component operations, fast creation & provisioning of next generation packaged operations services

DSL Access Providers

Dial-in Modem Operators

Email SvcsUSENET Svcs

WebHost SvcsSearch Engines Cache/Content

DistributionCarrier

InternetServiceProvider

Carriers

Carrier

CarrierInternetServiceProvider Carrier

CarrierInternetServiceProvider

Service Providers

InternetServiceProvider

Wireless Access Svc

VirtualISP

Sparing

Scaling Benefits of Neutral Co-location of Content and ISP Backbones

Scaling Benefits of Neutral Co-location of Content and ISP Backbones• Control

• Time to provision interconnect : 24 hours• Scalability

• Bandwidth scalability into the exchange• Bandwidth scalability within the exchange

• Choice• Ease of multi-homing• Open marketplace to buy/sell evolving

services• Stratification of the Internet service space

• Robust Interconnection Environment

• Control• Time to provision interconnect : 24 hours

• Scalability• Bandwidth scalability into the exchange• Bandwidth scalability within the exchange

• Choice• Ease of multi-homing• Open marketplace to buy/sell evolving

services• Stratification of the Internet service space

• Robust Interconnection Environment

SummarySummary

Application of existing technology can scale the core

Service provider tools include:• Dark fiber service providers; WDM into • Robust neutral exchange environment• Fiber cross-connects within exchange• Co-located content and multiple ISPs• Leads to: Control, choice, scalability at the

core

Application of existing technology can scale the core

Service provider tools include:• Dark fiber service providers; WDM into • Robust neutral exchange environment• Fiber cross-connects within exchange• Co-located content and multiple ISPs• Leads to: Control, choice, scalability at the

core