A Business Case for Peering in 2004 (v0.7)

William B. NortonCo-Founder & Chief Technical Liaison

Equinix, Inc.

(V0.7), XChangePoint Europe October 28, 2005 Updated: Oct 26, 2004

Internet Researcher

• 90% externally focused• Many documents on Protocols• Lack of Operations documents• Research: Peering

– How does Peering work?– What are the definitions?– What are the “Tricks of the Trade?”

White paper process..

Community Operations Research

1. “Ground Truth” w/dozens of experts2. Write White Paper v0.13. Walk community through WP for

comments4. Revise White Paper into new version5. Present White Paper at conferences6. Solicit comments over lunches and

dinners

White papers so far…

Internet Operations White Papers1) “Interconnection Strategies for ISPs”2) “Internet Service Providers and Peering”3) “A Business Case for Peering”4) “The Art of Peering: The Peering Playbook”5) “The Peering Simulation Game”6) “Do ATM-based Internet Exchanges Make Sense Anymore?” 7) “Evolution of the U.S. Peering Ecosystem”8) “Asia Pacific Peering Guidebook”9) “A Business Case for Peering in 2004”

Freely available. See Web site or send e-mail to wbn@equinix.comOr Google for “William B. Norton”

Disclaimer: Building Models

• There is nothing about Equinix in this presentation

• We are simply building models here• All prices are estimates, averages,

surveyed values, or made up• Your mileage may vary• Adjust the models to meet your situation• Also, this is an early draft

Does Peering Make Sense Anymore?

Transit Prices per Mbps

$1,500

$1,200$1,013

$600$450 $425

$275$125

$0$200$400$600$800

$1,000$1,200$1,400$1,600

1997 1998 1999 2000 2001 2002 2003 2004

The 2004 U.S. Transit Model Transit Prices (August 2004) (TC )

1 Mbps Commit $125 per Mbps10 Mbps Commit $80 per Mbps

100 Mbps Commit $45 per Mbps1000 Mbps Commit $30 per Mbps

Transit Commit Prices

$0

$20

$40

$60

$80

$100

$120

$140

1 121 241 361 481 601 721 841 961 1081 1201 1321

Mbps

$/M

bps

The Optimal Transit Model

• What is the optimal commit level given an expected transit volume?

Transit Prices (August 2004) (TC )1 Mbps Commit $125 per Mbps

10 Mbps Commit $80 per Mbps100 Mbps Commit $45 per Mbps

1000 Mbps Commit $30 per Mbps

Optimal Transit Model

• What is the optimal commit level given an expected transit volume?

• The 100% Heuristic • the Leo Transit Commit Heuristic

“I Don’t want to pay for traffic I don’t use!”

Compare 1M vs. 10M Commit

Mbps

$/Mbps at 1M Commit

$/Mbps at 10M Commit

1 $125 $8002 $125 $4003 $125 $2674 $125 $2005 $125 $1606 $125 $1337 $125 $1148 $125 $1009 $125 $89

10 $125 $8011 $125 $8012 $125 $8013 $125 $8014 $125 $8015 $125 $80 Graphically…

Comparing 1M and 10M Commit

$0

$100

$200

$300

$400

$500

$600

$700

$800

$900

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

$/Mbps at 1M Commit$/Mbps at 10M Commit

Transit Breakeven Point 6.4 Mbps

Compare the heuristics…

Comparing Transit Commit Heuristics

Mbps

$/Mbps at 1M Commit

$/Mbps at 10M Commit

100% Heuristic

Leo Heuristic

Norton Transit Heuristic

1 $125 $800 $125 $125 $1252 $125 $400 $125 $125 $1253 $125 $267 $125 $125 $1254 $125 $200 $125 $125 $1255 $125 $160 $125 $125 $1256 $125 $133 $125 $125 $1257 $125 $114 $125 $125 $1148 $125 $100 $125 $125 $1009 $125 $89 $125 $125 $89

10 $125 $80 $80 $125 $8011 $125 $80 $80 $125 $8012 $125 $80 $80 $125 $8013 $125 $80 $80 $125 $8014 $125 $80 $80 $125 $8015 $125 $80 $80 $80 $80

Cost Savings of Optimal Transit

Optimal Transit : 1M commit v. 10M commit

$0.00$20.00$40.00$60.00$80.00

$100.00$120.00$140.00

1 6 11 16 21 26 31 36 41 46

Mbps

$/M

bps

Leo Heuristic(70%)

100% Heuristic

Norton Heuristic

Same effect between 10 and 100M commits…

1M vs 10M vs 100M CommitsTransit (1M v 10M v 100M)

$0

$20

$40

$60

$80

$100

$120

$140

$160

30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

105

110

115

120

125

130

Mbps

$/M

bps 56.25Mbps

1M Commit

100M Commit

Prefer 100M Commit

Algebraically find the crossover point…

Norton Heuristic

MbpsT

MbpsT

MbpsT

MbpsTiceTransit

Mbps

Mbps

Mbps

Mbps

Commit

/30$

/45$

/80$

/125$Pr

1000

100

10

1

)(

)1()1(

)1(

)1(

)(

)(

)(

)1()1()(

)1()1()()(

*)1(

Pr

Pr

*

**

c

ccc

c

c

c

c

BEc

ccc

cccc

TVT

iffVClCommitLeve

MinTransitVolNextCommitV

iceTransitNextCommitT

nsitVolumeCurrentTraV

icensitCurrentTraT

TTVT

V

VTVTtCostNextTransinsitCostCurrentTra

Q: When do we commit to the next higher commitment level?A: At the Transit Breakeven Point (TBE)

Plug in for T(c) and T(c+1)

Plug in for V(c+1)

Transit Breakeven PointsTransit vs. Optimal Transit

$0

$20

$40

$60

$80

$100

$120

$140

1 141 281 421 561 701 841 981 1121

Mbps

$/M

bps

CommitPrice

$/Mbps at OptimalTransit

6.4M

56Mbps

666Mbps

100M Commit

1000M Commit

The Peering ModelPeering isn’t free…Monthly Costs include:• Local Loop fees (the cost to get in IX),• IX collocation fees, and• IX peering port fees.Also consider Equipment Costs

which can be amortized into Monthly Cost

Monthly Peering Costs=LL+IX+IXP+E

But how do we compare Monthly Costs against $/Mbps per month?...

Peering versus (Optimal) Transit1) Normalize

the Peering and Transit units

Peering vs. Transit

Traffic Exchanged (Mbps)

$/M

bps

Fixed Monthly Cost of Peering

Metered Transit

Peering Breakeven Point (Peering=Transit)

$/Mbps (per month)

Vs.

Mbps ExchangedIn Peering or Transit Relationship(Per month)

All as 95th Percentile measures

Peering vs. Transit at TV

2) Compare Peering at PV and Transit at TV

Sample Blended Peering w/Optimal Transit vs. Optimal Transit(Assume 20% peering in Blend)

Peering (20%) Mbps

Peering at PV

Transit (80%) Mbps

Transit at TV

Peering and Transit Blend

Transit Only Mbps

Optimal Transit (100%)

Monthly Cost Savings

25 $360.00 100 $60.00 120.00$ 125 45.00$ -$9,37550 $180.00 200 $45.00 72.00$ 250 45.00$ -$6,75075 $120.00 300 $45.00 60.00$ 375 45.00$ -$5,625

100 $90.00 400 $45.00 54.00$ 500 45.00$ -$4,500125 $72.00 500 $45.00 50.40$ 625 45.00$ -$3,375150 $60.00 600 $45.00 48.00$ 750 40.00$ -$6,000175 $51.43 700 $45.00 46.29$ 875 34.29$ -$10,500200 $45.00 800 $45.00 45.00$ 1000 30.00$ -$15,000225 $40.00 900 $44.44 43.56$ 1125 30.00$ -$15,250250 $36.00 1000 $40.00 39.20$ 1250 30.00$ -$11,500275 $32.73 1100 $36.36 35.64$ 1375 30.00$ -$7,750300 $30.00 1200 $33.33 32.67$ 1500 30.00$ -$4,000325 $27.69 1300 $30.77 30.15$ 1625 30.00$ -$250350 $25.71 1400 $30.00 29.14$ 1750 30.00$ $1,500375 $24.00 1500 $30.00 28.80$ 1875 30.00$ $2,250400 $22.50 1600 $30.00 28.50$ 2000 30.00$ $3,000425 $21.18 1700 $30.00 28.24$ 2125 30.00$ $3,750450 $20.00 1800 $30.00 28.00$ 2250 30.00$ $4,500475 $18.95 1900 $30.00 27.79$ 2375 30.00$ $5,250500 $18.00 2000 $30.00 27.60$ 2500 30.00$ $6,000525 $17.14 2100 $30.00 27.43$ 2625 30.00$ $6,750550 $16.36 2200 $30.00 27.27$ 2750 30.00$ $7,500575 $15.65 2300 $30.00 27.13$ 2875 30.00$ $8,250 100M Peering vs. Optimal Transit at EPB

$0.00

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

$70.00

$80.00

$90.00

$100.00

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

Mbps Exchanged

Effective Peering Bandwidth@70.5Mbps

Transit

Example: Peering vs. Optimal TransitPeering Cost: $9000/Month

MbpsT

MbpsT

MbpsT

MbpsTiceTransit

Mbps

Mbps

Mbps

Mbps

Commit

/30$

/45$

/80$

/125$Pr

1000

100

10

1

Blend Peering and Optimal Transit vs. Optimal Transit (assuming only 20% Peering is attainable)

$-

$20.00

$40.00

$60.00

$80.00

$100.00

$120.00

$140.00

Mbps

$/M

bps

Peering and Transit Blend

Optimal Transit (100%)

2004 Peering vs. Transit

• Model I – Have to build into IX– Monthly Peering costs

• Local Loop• Peering Port• Rack Space• Peering Equipment

• Model II – Already at IX– Monthly Peering Costs

• Peering Port• Incremental Peering HW

Peering Equipment costs…

Peering Equipment CostsUsed Peering Equipment100M Peering EquipmentFoundry Big Iron 4000 w/24 FEs $2,4003yr Straight Line Depreciation $66.67 per month

1000M Peering EquipmentCisco 12008, loaded, with 2*GE $7,0003yr Straight Line Depreciation $194.44 per month

New peering Equipment1000M Peering EquipmentCisco 7301 $13,0003yr Straight Line Depreciation $361.11 per month

High End 1000M Peering EquipmentJuniper M7i, loaded, dual gigEs $28,9253yr Straight Line Depreciation $803.47 per month

Sources: Richard Steenbergen (nLayer)And Patrick Gilmore (Akamai)

Model I – Build into IX for 100M Peering

Model I : Monthly 100M Peering CostsTransport 100 Mbps 1,000.00$

IX Fees (Port + 1/2 rack) 100 Mbps 2,500.00$ Base IX Participation Peering Costs 3,500.00$

Amortized used Foundry Equipment Cost 66.67$

Total Monthly Cost of 100M Peering 3,566.67$ Local Loop Assumptions:100M Transport into IX: $1000 per month1000M Transport into IX: $4000 per month

Graphically: P vs.T…

Peering Rack & Port Assumptions:100M w/ ½ rack at IX: $2500 per month1000M w/ ½ rack at IX: $5000 per month

Model I – 100M Peering100M Blended Peering vs. Optimal Transit

$0.00$500.00

$1,000.00$1,500.00$2,000.00$2,500.00$3,000.00$3,500.00$4,000.00

1 28 55 82 109 136 163 190 217

Mbps

$/M

bps Peering

Optimal Transit

100M Peering at the EPB100M Peering vs. Optimal Transit at EPB

$0.00

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

$70.00

$80.00

$90.00

$100.00

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

Mbps Exchanged

$/M

bps

Effective Peering Bandwidth@70.5Mbps

Peering

Transit

$50.59

$45/Mbps

Why peer when P>T?

Why Peer when P100>T100 ?

1) Performance 2) Greater control over routing 3) Meeting Peering Prerequisites 4) Already at the IX 5) Marketing value 6) 100M Peering is a stepping stone 7) Currently paying high transit fees. 8) Peering Costs are immaterial.

100M Peering RevisitedModel I 100M Peering vs. Optimal Transit Only

$0.00

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

$70.00

$80.00

$90.0060 78 96 114

132

150

168

186

204

222

240

258

276

294

Mbps

$/M

bps Blended Peering

optimal transit only

Add FastE Ports

Commit to 1000M Transit

Assuming only 20%, at 33% peering is closer call. 4*100M=1000M, Peering at 1G…

Model I – Build into IX for 1000M Peering

Model I : Monthly 1000M Peering CostsTransport 1000 Mbps 4,000.00$

IX Fees (Port + 1/2 rack) 1000 Mbps 5,000.00$ Base IX Participation Peering Costs 9,000.00$

Amortized used Cisco 12008 Equipment Cost 194.44$

Total Monthly Cost of 1000M Peering w/used Cisco12008 9,194.44$

Graphically P vs T…

Peering Breakeven AnalysisModel I - 1000M Peering vs. Optimal Transit

$0.00

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

$70.0015

0

207

264

321

378

435

492

549

606

663

Mbps

$/M

bps

Optimal Transit(@4xP)

Model I 1000MPeering

Peering=Transit

306 Mbps

New vs. Used equipment?

Model I – 1000M Peering – new vs. used Equipment doesn’t matter

Peering 1000M: Comparison Between Used and New EquipmentTotal Monthly Cost of 1000M Peering w/used Cisco12008 9,194.44$ per MonthMinimum Cost of Peering (at 705Mbps) 13.04$ per MbpsPeering Breakeven Point (w/$30/Mbps transit) 306.5 Mbps

Total Monthly Cost of 1000M Peering with new Cisco 7301 9,361.11$ per MonthMinimum Cost of Peering (at 705Mbps) 13.28$ per MbpsPeering Breakeven Point (w/$30/Mbps transit) 312.0 Mbps

Total Monthly Cost of 1000M Peering w/new Juniper 7i 9,803.47$ per MonthMinimum Cost of Peering (at 705Mbps) 13.91$ per MbpsPeering Breakeven Point (w/$30/Mbps transit) 326.8 Mbps

Project forward – when gigE fills…

Adding 2nd gigE for PeeringPeering 2nd gigE

$0.00

$5.00

$10.00

$15.00

$20.00

$25.00

$30.00

$35.00

$40.00

$45.00

$50.0020

5

300

395

490

585

680

775

870

965

1060

1155

1250

1345

Mbps

$/M

bps Optimal Transit (@4xP)

Model I 1000M Peering

Peering Breakeven @306Mbps

2nd gigE

When does 10GE peering make sense?

10GigE Peering ModelModel I : Monthly 10-gigE Peering CostsTransport 10000 Mbps 16,000.00$

IX Fees (Port + 1/2 rack) 10000 Mbps 12,000.00$ Base IX Participation Peering Costs 28,000.00$

Amortized Cisco 6509, 4*10G, 4xXENPAK $1,527.78

Total Monthly Cost of 10000M Peering 29,527.78$

Graphically P v T…

Model I – gigEs vs 10gigE1G & 1G-10G vs Optimal Transit

$0.00

$20.00

$40.00

$60.00

$80.00

$100.00

$120.00

$140.00

$160.00

$180.00

$200.0050 485

920

1355

1790

2225

2660

3095

3530

3965

4400

4835

5270

5705

6140

6575

7010

Mbps

$/M

bps Optimal Transit (@4xP)

Model I 1000M Peering

Model I 1-10G Peering

Summary stats for Model I 10G Peering

10gigE Peering MetricsSummary Stats for Model I Peering 10000M w/new 6509

Effective Peering Bandwidth 7,050.0 MbpsMinimum Cost of Peering (at 7050Mbps) 4.19$ per MbpsPeering Breakeven Point (20% Peering) 306.48 MbpsEffective Peering Range 306.5 until 7050.00 MbpsAt the 1410 before using 2nd gigE 20.94$ per Mbps

Traffic exchange for less than $5/Mbps and covering all costs of peering!

Model II – What if I already have colo at an IX?

Model II – Already at IX Colo

Model II - 100M PeeringModel II : Monthly 100M Peering CostsTransport -$

IX Port Fees (rack present) 100 Mbps 1,500.00$ Base IX Participation Peering Costs 1,500.00$

Equipment presentAssume 24 Fast Ports already installedTotal Monthly Cost of 100M Peering 1,500.00$

Peering Port Fee Assumptions: 100M Port only: $1500 per month1000M Port only: $4000 per month

Graphically: P vs T…

Model II – Peering 100M PortsModel II+100M Peering

$-

$10.00

$20.00

$30.00

$40.00

$50.00

$60.0030 85 14

0

195

250

305

360

415

470

525

580

635

690

745

800

855

910

965

1020

1075

1130

1185

1240

1295

1350

1405

1460

1515

1570

1625

1680

Mbps

$/M

bps

Optimal Transit

Peering with FastE cards

Graphically 1000M and 10G Peering…

Model II – 1000 and 10G PeeringModel II : Peering vs. Transit

$0.00

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

73 383

693

1003

1313

1623

1933

2243

2553

2863

3173

3483

3793

4103

4413

4723

5033

5343

5653

5963

6273

6583

6893

Mbps

$/M

bps Optimal Transit

Model II : Peering gigEsModel II : Peering gigEs to 10GE

$1.92/Mbps

Model II Summary StatsSummary Stats for Model II Peering 10000M w/new Cisco 6509

Effective Peering Bandwidth 7,050.0 MbpsMinimum Cost of Peering (at 7050Mbps) 1.92$ per MbpsPeering Breakeven Point (20% Peering) 89.81 MbpsEffective Peering Range 89.8 until 7050.00 MbpsAt the 1410 before using 2nd gigE 9.59$ per Mbps

Summary

• Transit Prices Plummeted• Does Peering Make Sense?• Decision Makers: Peering best compared

against Optimal Transit• New vs. Used Peering Equipment doesn’t

matter• Peering Scales Well in the gigabit ranges

Acknowledgements

I am grateful to Richard Steenbergen (nLayer) and Patrick Gilmore (Akamai) for their data points and insights during the initial development of the white paper models. I also thank John Hardie and Lane Patterson (Equinix), Scott Bradner (Harvard), Joe McGuckin (via.net), Leo Bicknell (AboveNet), Dave Woodelet (Shaw Fiber), Rich Collela (AOL), Dan Golding (Burton Group) for their suggestions.