The Evolution of the U.S. Internet Peering Ecosystem

William B. Norton

Co-Founder and Chief Technical Liaison

Equinix, Inc.

Internet Researcher

• 90% with Internet Companies Engineers

• EQIX: Massive Carrier Neutral Colocation

• Lots of Blinking lights…

• Observe: documentation on HW&Protocols

• Lack of Operations documents

• Research How does Peering work?

White paper process..

Community Operations Research

• Ground Truth w/dozens of experts

• Write White Paper v0.1

• Walk community through WP for comments

• Revise White Paper into new version

• Present White Paper at conferences

• Solicit comments over lunches and dinners

White papers so far…

Internet Operations White Papers

1) “Interconnection Strategies for ISPs”2) “Internet Service Providers and Peering”3) “A Business Case for Peering”4) “The Art of Peering: The Peering

Playbook”5) “The Peering Simulation Game”6) “Do ATM-based Internet Exchanges

Make Sense Anymore?” Freely available. Send e-mail to wbn@equinix.com

New white Paper: The Evolution of the U.S. Peering Ecosystem, Agenda…

Agenda

• Definitions to level set– Peering, Transit, ISPs– Synch Point here.

• Introduction to notion:“Peering Ecosystem”– Internet=Many Internet Regions each with– Tier 1 ISPs, Tier 2 ISPs, Content Providers

• Evolution of the U.S. Peering Ecosystem– 3 major evolutions

Def: Internet Service Provider

• Def: Internet Service Provider is an entity that sells access to the entire Internet. This service is often called “Internet Transit”

• Def: Transit is a business relationship whereby one entity sells access to the entire Internet.

Definition of Transit

1) ISP A buys Transit Service

Upstream Transit ProviderISP A

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit

Providers

Def: Transit is the business relationship whereby one ISP

announces (sells) reachability to the *entire* Internet to a customer.

•$$$ ? Typically usage-based•Pricing ’04: $18-$200/Mbps•Volume based on 95th Percentile measure•Transit is Simple, Convenient:• Upstream handles the delivery of packets to the Internet by some meansUsage: “I’m purchasing transit from Level 3.”

3) Traffic Flows

INTERNET

NETWORKS

2) ReachabilityAnnouncement

Transit

0

50

100

150

200

2501 14 27 40 53 66 79 92 105

Traffic Exchanged (Mbps)

$/M

bp

s

Transit $/Mbps

Cost of Transit Traffic Exchange

2001 Pricing Sampling Range: $100-$1200/MbpsMy Financial Models used $388/Mbps

Source: 2004 Survey Avg. Range: $20-$400/Mbps 95th Percentile

Why Peer?

1/15/04 field spot quotes: GX: $65/Mbps, HE: $25/Mbps Cogent $20/Mbps UU: $45/Mbps, etc. varied commits 0-1000Mbps

Scaling at the Edge

Content Heavy ISP

Access Heavy ISP

UPSTREAMS

V

IGMBT

Streaming Media

56k 384k 1.5m

$$$

$$$

Broadcast.com: 100,000 concurrent unicast streams15 million streaming hrs/mo, 1300 Live events/day

Access

AOL+ DSL: 1,000/dayRoadrunner Cable Modems: 1M subscribers

Negotiate Peering with Upstream ISPsOr focus on peering with like-minded ISPs…

1 T1 consumer streaming 24/7 cost $388/mo in transit

Streaming:BroadcastTelephonyVideoMultimediaInteractiveGamingSPOT Events

Tier1 Equipment: $13B/’00 -> $42B/’04(Source: Infonetics)

The Zone: +1M gaming user/mo!

Seek transportInterconnection $

x

Why Peer? Motivations for Peering• Financial: Reduce load on expensive Transit

service• Traffic src/dest• Measure vs Intuit• Usage-based Billing

• Engineering: Lower latency

1st Stage of Peering:Top 10 destination ISP list

Transit$$$

Transit$$$

ISP A

ISP B

Transit ISP

Top 10, Def: Peering…

Sample Top 10 Destination ListA S N u m b e r M b p s D e s t i n a t i o n I S P C o n t a c t

6 1 7 2 2 4 . 3 5 H O M E - N E T - 1 [ H O M E - N O C - A R I N ]

7 0 1 8 . 9 0 A L T E R N E T - A S [ I E 8 - A R I N ]

1 6 6 8 8 . 1 4 A O L - P R I M E H O S T [ A O L - N O C - A R I N ]

4 7 6 6 7 . 0 8 A P N I C - A S - B L O C K [ S A 9 0 - A R I N ]

3 3 2 0 5 . 1 2 R I P E - A S N B L O C K 4 [ R I P E - N C C - A R I N ]

5 7 7 4 . 2 4 B A C O M [ E Q - A R I N ]

6 3 2 7 3 . 9 0 S H A W F I B E R [ I A S - A R I N ]

1 3 . 8 9 B B N P L A N E T [ C S 1 5 - A R I N ]

7 0 1 8 3 . 6 6 A T T - I N T E R N E T 4 [ J B 3 3 1 0 - A R I N ]

9 3 1 8 3 . 1 3 A P N I C - A S - 3 - B L O C K [ S A 9 0 - A R I N ]

5 7 6 9 2 . 6 7 V I D E O T R O N [ N A V 1 - A R I N ]

6 8 3 0 2 . 3 0 H C S N E T - A S N B L K [ M D 2 0 5 - A R I N ]

9 2 7 7 2 . 2 2 A P N I C - A S - 3 - B L O C K [ S A 9 0 - A R I N ]

1 0 9 9 4 2 . 0 8 T A M P A 2 - T W C - 5 [ J D 6 - A R I N ]

1 2 3 9 2 . 0 5 S p r i n t L i n k [ S P R I N T - N O C - A R I N ]

I n t e r n e t S e r v i c e P r o v i d e r A

Def: Peering

Definition of PeeringDef: Peering is the business relationship whereby ISPs reciprocally announce reachability to each others’ transit customers.

Peering

USNet EastNetWestNet

Peering

TransitRoutingTables

•Peering is *not* a transitive relationship•Peering *does not* provide access to the entire Internet

Cost Peering vs. Transit

Usage: “I buy transit from UUNet and Peer with EastNet, WestNet.”

The Cost of PeeringPeering

0

500

1000

1500

2000

25001 12 23 34 45 56 67 78 89 100

Traffic Exchanged (Mbps)

$/M

bp

s

Peering

1) Monthly Circuit Fee into IX2) Monthly IX Port Fees

When does Peering Make Sense Financially?

Peering vs. Transit

0

500

1000

1500

2000

2500

1 6 11 16 21 26 31

Traffic Exchanged (Mbps)

$/M

bp

s

Transit $/Mbps

Peering

Number of Mbps exchanged

Generalization:ISP Peering Breakeven Analysis Graphs

$/MbpsExchanged

Cost of Transit

Cost of Traffic Exchange in Peering Relationship

Breakeven Point (ISPs Indifferent between

Peering and Transit traffic exchange)

Prefer PeeringPeeringRisk

Synch Point

• You should have – a working lexicon at this point– A sense of the motivations for Peering &

Transit

• Any questions so far?

The Internet Peering Ecosystem

Pre-Crash : circa 2000

Peering Ecosystem

• System of interconnected (Peering and Transit) players

• Each group with homogenous peering motivations and observed behaviors– Tier 1 ISPs– Tier 2 ISPs– Content Providers

Def: Tier 1 ISP

• Definition: A Tier 1 ISP is an ISP that has access to the entire Internet Region solely through peering relationships. (i.e. doesn’t buy transit from anyone to see regional routes)

• Motivation: Doesn’t need to peer with anyone else, so

• Peering only with other Tier 1 ISPs

Def: Tier 2 ISP

• Definition: A Tier 2 ISPs is any ISP that has to purchase transit to reach destinations in an Internet Region

• Motivation: Peer to decrease costs, improve performance

• Peering: with Tier 1’s is preferable but hard to get, peering with Tier 2’s as appropriate

• Must purchase Transit

Def: Content Providers

• Def: A Content Provider is an entity that produces content for delivery over the Internet but does not sell transit.

• Motivation: Best Customer Experience (engineered performance)

• Peering: Not typically done, prefer SLAs.

• Transit from ISPs

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

The Evolution of the U.S. Peering Ecosystem

Act II: 2000-2003

Evolution of the U.S. Peering Ecosystem

Four key events leading to drastic disruption in the Peering Ecosystem:

1) The 1999/2000 Telecom Collapse2) The growth of the Used Equipment Market3) The Upstream Provider for the Cable Companies

(@Home) went bankrupt4) Peer-to-Peer file sharing systems (like Kazaa) grow

in popularity, traffic grows exponentially between Access Providers (1.5MB MP3 700MB AVI files)

5) Transit Prices Drop, Transport Prices Drop

Leads to 3 Major Evolutions

Evolution #1 – Cable Companies are Peering

• 40% of traffic Kazaa• 3-5 Gbps of transit

traffic-> Gbps of peering potential !

Significant because1) Volume2) Open Peering3) Kazaa Effect

(Optional)Backbone Backhaul

Cable Company

Transit Provider Interconnections (Upstream(s))

Peering

U.S. Cable Companies Eyeballs

MSO Country YE02 Change in 02Comcast USA 3,620,300 1,199,100Time Warner USA 2,613,000 1,027,000Cox USA 1,407,900 524,400Charter USA 1,180,000 572,300CableVision USA 770,100 263,500Shaw (Big Pipe) Canada 750,000 ?Rogers Canada 650,000 ?Adelphia USA 610,000 305,900Bright House USA 490,000 159,000Mediacom USA 191,000 76,000RCN USA 160,400 49,800Insight USA 144,800 56,700Cable One USA 78,100 45,200Total 12,665,600 4,278,900

Cable Modem Subscribers

Source: The Bridge & Leichtman ResearchSource: I$P HO$TING ReportMay 2003, Volume VII, Number 5, Page 8Article "Dialup Bakeoff - Is it really so bad at EarthLink? Or so good at United?"Source: Bill Norton conversations with Rogers and Shaw 2002 (numbers may be understated)

Cable Company

Cable Company

Cable Company

Cable CompanyCable Company

Cable Company

Cable Company

AOL

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

Content

2000-2003 Evolution #1 – Cable Companies Peering

Tier 2 ISPs

CableCos

Tier 1 ISPs

Peering

Evolution #2 – Network Savvy Large Scale Content Companies

get into Peering• To reduce transit costs • To improve the end user experience• They need to move out of bankrupt colo anyway

Significant because

1) Volume of traffic is huge

2) Content Providers have Open Peering

3) Leading players pave the way

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

Content

2000-2003 Evolution #2 – Large Scale Network Savvy Content

Peers

Tier 2 ISPs

LSNSC

Tier 1 ISPs

Other Broadband Players

• Significant, but…

• Not Open Peers so disincentive to peer

• So less volume and less disruptive

Evolution #3 – Cable Companies Freely Peer with Content

Companies• Content literally on the Cable Company

Network

• Lowest possible latency from content to eyeballs

• Internet Gaming, Broadband Streaming, etc.

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

Content

2000-2003 Evolution #1 – Cable Companies Peering

Tier 2 ISPs

CableCos

Tier 1 ISPs

Content

2000-2003 Evolution #2 – Large Scale Network Savvy Content

Peers

Tier 2 ISPs

LSNSC

Tier 1 ISPs

Content

2000-2003 Evolution #3 – Cable Peers with Large Scale Network

Savvy Content

Tier 2 ISPs

LSNSC

Tier 1 ISPs

CableCos

International Dynamics – Separate White Paper

Content

Tier 2 ISPs

LSNSC

Tier 1 ISPs

CableCos

Content

Tier 2 ISPs

Tier 1 ISPs

NTT

KDDI

IIJ

JTCableCos

DSL

JP U.S.

U.S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet RegionJapan Tier 1 ISPs are Tier 2 ISPs in the U.S. Internet Region

Q&A