The Internet is Flat: Revisited...Sep 21, 2018 · A Small Transit Provider Case Study 2017 –...

The Internet is Flat: Revisited

A Small Transit Provider Case Study

Bob Bender Steve Surdock (CTS Telecom)

Amogh Dhamdhere

(CAIDA/UCSD)


The Internet is Flat: Revisited A Small Transit Provider Case Study

In2011atNANOG52,asmallTier3ISPAS19653joinedNANOG.Alsoin2011,thissmallISPreadthepaper-“TheInternetisFlat:ModelingtheTransitionfromaTransitHierarchytoaPeeringMesh”Theforecastsinthispaperwereusedtoinformbusinessandnetworkplanning.ActualnetworkdatawascollectedfromAS19653from2010topresent.Thissmalltransitproviderdataisavignetteofthefactorsthat“cantransformtheInternetecosystemfromamulti-tierhierarchythatreliesmostlyontransitlinkstoadensemeshofhorizontalinterconnectionsthatreliesmostlyonpeeringlinks”

The Internet is Flat: Modeling the Transition from a Transit Hierarchy to a Peering Mesh

OriginallyPresentedatACMCoNEXT2010Nov.30–Dec3,2010--Philadelphia,PAUSA

AmoghDhamdhere(CAIDA)ConstantineDovrolis(GeorgiaTech)



The Internet in 2011: What did the future hold?

InternetEcosystemEvents

2004-GoogleIPO

2007-AppleiPhoneIntroduced

2007-Netflixbeginsstreaming

2008-HuluLaunched

2011-PandoraIPO

2012-FacebookIPO

The“TheInternetisFlat”paperofferedananalysisofwhatwesawhappeninganecdotallyasasmallISP.AsaTier3ISPitbecameclearthatamovetobecomeaTier2ISPwouldbepossibleinthenewInternetecosystem.MostimportantlythepaperdrovehomethattheimportanceofTier1TransitwasdiminishedandpeeringwithcontentintheIXPwasparamount.

The ITER Model

5The Internet is Flat: Revisited

Agent-basedcomputationalmodeltoanswer“what-if”questionsaboutInternetevolution

Inputs•  Networktypesbasedonbusinessfunction•  Pricing/costparameters•  Interdomaintrafficmatrix•  Geographicalconstraints•  Peer/providerselectionmethods

Output:Equilibriuminternetworktopology,trafficflow,per-networkfitness

A.DhamdhereandC.Dovrolis.TheInternetisFlat:ModelingtheTransitionfromaTransitHierarchytoaPeeringMeshACMCoNEXT2010–Page2

ITERModelPreviousApplicationsareinDemography,Social,EconomicandEnvironmentalSciences.

The ITER Approach

Analyticallyintractable.Findequilibriumcomputationally,usingagent-basedsimulationsEquilibrium:nonetworkhastheincentivetochangeitsproviders/peers

TrafficFlowInterdomainTopology

Per-ASEconomicFitness

Cost/PriceParameters

BGP-likeRouting

InterdomainTrafficMatrix

ProviderSelection


ASOptimizations

PeerSelection


The “Hierarchical” and “Flat” Internet

TheHierarchicalInternet(late90s–2007)•  Topcontentprovidersgeneratedsmallfractionof

totaltraffic•  Contentprovidersweretypicallyservedfromorigin•  Peeringwasrestrictive

TheFlatInternet(2007onwards)•  Topcontentprovidersgeneratelargefractionof

totaltraffic•  Contentprovidershaveexpandedgeographically•  Peeringismoreopen

“InternetInterdomainTraffic”,Labovitzetal.,Sigcomm2010The Internet is Flat: Revisited

Content Consolidation


Interdomain Routing and Traffic flow

•  Moretrafficflowsoverpeeringlinksthantransitlinksinthe“Flat”Internet

•  Trafficfollowsshorterroutingpathsduetodirectpeeringinthe“Flat”Internet

•  Thiseffectisevenmorepronouncedwhenpathsareweightedbytrafficvolume:pathscarryingthemosttrafficareshorter


Simulatedtwo“instances”oftheITERmodel.Firstwasparameterizedtoresemblethe“HierarchicalInternet”.Secondwasparameterizedtoresemblethe“FlatInternet”.Thencomparedvariouspropertiesoftheequilibriumthatwegetfromthetwoinstancesofthemodel.

Predictions of Transition Impacts

ContenttrafficbypassesTier-1providersinthe“flat”Internet:ProducesconditionsforTier1consolidationItispossibleforaTransitProviderstoenhanceprofitabilityinthe“flat”bypeeringstrategicallywithlargeContentProvidersContentproviderscalepromotespeering

The Internet is Flat: Revisited A.DhamdhereandC.Dovrolis.TheInternetisFlat:ModelingtheTransitionfromaTransitHierarchytoaPeeringMeshACMCoNEXT2010–Page12

InboththeHierarchicalandFlatInternet,thereisastrongcorrelationbetweenaTransitProvider’sfitnessandthesizeofitscustomerbase.(need“eyeballs”topeer)

IntheFlatInternet,however,strategicpeeringbecomesmoreimportantforSmallTransitProviders(STP)andLTPs;bothcanbeprofitablebypeeringselectivelywiththelargestcontentproviders.

IntheFlatInternet,itispossibleforaTransitProvidertotransitionfromunprofitabilitytoprofitabilitybypeeringstrategically,particularlywithlargeContentProviders;suchatransitionislesslikelyintheHierarchicalInternet.

The Opportunity Presented by Peering Content instead of relying on Tier 1 Transit

The Internet is Flat: Revisited A.DhamdhereandC.Dovrolis.TheInternetisFlat:ModelingtheTransitionfromaTransitHierarchytoaPeeringMeshACMCoNEXT2010–Page12


2017–(after18NANOGs)PacketOpticalServiceProviderTier2ISP80%Peering20%transitMorethan100Ginupstreamports

2011–JoinedNANOGTelephoneCompany(ILEC-CLEC)Tier3ISP100%transit(twoOC-12s)


AS19653–SmallTransitProviderinClimax,MichiganFoundedin1911asClimaxTelephonedbaCTSTelecom

IndependentILEC-CLEC-ISP.CLLI=CLMXMIXI

Network Data Source for Graphs


DailySolarWindsNPM95thPercentilereportscollectedsince2010

SOURCE:CTSTelecom

0Mbps1,000Mbps2,000Mbps3,000Mbps4,000Mbps5,000Mbps6,000Mbps7,000Mbps8,000Mbps9,000Mbps10,000Mbps11,000Mbps12,000Mbps13,000Mbps14,000Mbps15,000Mbps16,000Mbps17,000Mbps18,000Mbps19,000Mbps20,000Mbps21,000Mbps22,000Mbps23,000Mbps24,000Mbps

Nov-10

Nov-11

Nov-12

Nov-13

Nov-14

Nov-15

Nov-16

Nov-17

CabinetinIXPPeeringExchange

1stGigPNItoContent

FirstCDNServer

CDNServer

CDNServer

Transitiontoall10GPorts

SecondIXPCabinet

L-RootHost

J-RootHostThirdIXP

PeeringDB

RadBRoutingRegistry

3rdPeeringExchange


“IntheFlatInternet,however,strategicpeeringbecomesmoreimportantforSmallTransitProviders…”



SOURCE:CTSTelecom

AS19653 Traffic Mirrors the US IP Traffic Curve


U.S.IPtrafficisprojectedtogrow2.5xinthenextfiveyears

AS19653 Evolution of Transit to Peering

A Small Transit Provider Case Study SOURCE:CTSTelecom

0Mbps

2,000Mbps

4,000Mbps

6,000Mbps

8,000Mbps

10,000Mbps

12,000Mbps

14,000Mbps

16,000Mbps

18,000Mbps

20,000Mbps

22,000Mbps

2010 2011 2012 2013 2014 2015 2016 2017 2018

Peering Transit

Peering 0% 0% 36% 66% 73% 80% 86% 88% 82%

Transit 100% 100% 64% 34% 27% 20% 14% 12% 18%

AS19653 From Transit to IXP Peering to CDN


0Mbps1,000Mbps2,000Mbps3,000Mbps4,000Mbps5,000Mbps6,000Mbps7,000Mbps8,000Mbps9,000Mbps10,000Mbps11,000Mbps12,000Mbps13,000Mbps14,000Mbps15,000Mbps16,000Mbps17,000Mbps18,000Mbps19,000Mbps20,000Mbps21,000Mbps22,000Mbps23,000Mbps24,000Mbps

Nov-10

Jan-11

Mar-11

May-11

Jul-1

1

Sep-11

Nov-11

Jan-12

Mar-12

May-12

Jul-1

2

Sep-12

Nov-12

Jan-13

Mar-13

May-13

Jul-1

3

Sep-13

Nov-13

Jan-14

Mar-14

May-14

Jul-1

4

Sep-14

Nov-14

Jan-15

Mar-15

May-15

Jul-1

5

Sep-15

Nov-15

Jan-16

Mar-16

May-16

Jul-1

6

Sep-16

Nov-16

Jan-17

Mar-17

May-17

Jul-1

7

Sep-17

Nov-17

Jan-18

Mar-18

May-18

Jul-1

8

Sep-18

Transit IXPeering CDNPeering

Percentage of Total Traffic AS19653 Transit/Peering/CDN


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

11/10/2010

1/10/2011

3/10/2011

5/10/2011

7/10/2011

9/10/2011

11/10/2011

1/10/2012

3/10/2012

5/10/2012

7/10/2012

9/10/2012

11/10/2012

1/10/2013

3/10/2013

5/10/2013

7/10/2013

9/10/2013

11/10/2013

1/10/2014

3/10/2014

5/10/2014

7/10/2014

9/10/2014

11/10/2014

1/10/2015

3/10/2015

5/10/2015

7/10/2015

9/10/2015

11/10/2015

1/10/2016

3/10/2016

5/10/2016

7/10/2016

9/10/2016

11/10/2016

1/10/2017

3/10/2017

5/10/2017

7/10/2017

9/10/2017

11/10/2017

1/10/2018

3/10/2018

5/10/2018

7/10/2018

9/10/2018

Transit IXPeering CDNPeering


Network Snapshot AS19653

CAIDAASrank:1483IPsinCustomerCone(v4):143,104InternetExchanges:3PrefixesOriginated(all):12PrefixesOriginated(v4):8PrefixesOriginated(v6):4PrefixesAnnounced(all):57PrefixesAnnounced(v4):46PrefixesAnnounced(v6):11BGPPeersObserved(all):423BGPPeersObserved(v4):417BGPPeersObserved(v6):261IPsOriginated(v4):90,624ASPathsObserved(v4):91,578ASPathsObserved(v6):19,424

SOURCE:A.DhamdhereCAIDA

PeersvisibleafterjoiningRouteViews


Game Changers

•  JoiningNANOGCommunity•  EstablishingIXPpresence•  JoinedPeeringExchange•  JoinedPeeringDB•  Read“TheInternetisFlat”•  ImplementedNetFlowanalysis•  DevelopingNANOG“savoirfaire”•  “Dr.Peering”Website(ThankstoBillNorton!)•  SupportofContentProviders•  MentoringfromtheNANOGcommunity


Challenges and Cautions for Small Providers

•  Unlessyouhavealargeenoughnumberof“eyeballs”onyournetworkandahighenoughtrafficlevel,peeringdoesnotmakeeconomicsense

•  Peeringrequiresasignificantamountoftechnicalexpertiseandcommitmentsofresources.

•  ConnectivitytoInternetExchangePointsisnottrivial.IdeallyaprovidershouldbeattwoIXPsandredundantnetworkconnectionsarebest.SelectiveContentProvidersrequirepeeringatmultiplelocations.

•  ThefallingpriceofTransitmakesthecaseforpeeringforasmallprovidereconomicallychallenging:sometimesbuyingTransitiseasier.

•  YoumusthaveeconomicalaccesstofibertransporttoreachtheIXP.

The Internet is Flat: Revisited...Sep 21, 2018 · A Small Transit Provider Case Study 2017 –...

Documents

Transcript of The Internet is Flat: Revisited...Sep 21, 2018 · A Small Transit Provider Case Study 2017 –...