Build vs. Buy: Internet Datacenter Operations Research
description
Transcript of Build vs. Buy: Internet Datacenter Operations Research
Build vs. Buy: Internet DatacenterOperations Research
INTAPMarch 6, 2001Tokyo
Agenda
• My Background and Who is Equinix? – 15 minutes• Recent Internet Operations Research – 20 minutes• “Build vs.Buy Internet Datacenter” White Paper – 40
minutes• Future: Where our Industry is heading – 5 minutes
NSFNET to Post-NSFNET Era InternetNSFNET to Post-NSFNET Era Internet
MY JOB: 5 Years as Chair of NANOG12 Years in Internet…No Internet Preso…
BARRNet
NWNet
Sesqui
NEARNet
NyserNetMerit
NCAR
OarNet
Regional
RegionalRegionalNSFNET
SDSC
AT&T
MCI
Sprint
UUNet
ANSNWNet NyserNet
MeritOarNet
Regional
Regional Regional
Regional
NSFNET (1987-1994) Regional Tech Meetings
–Performance & Operations
–Reporting & Coordination
–Routing & Network Planning
Privatization (1994-Now) North American Network Operators Group
(NANOG) Meetings
Internet Statistics• 87.5% of all Internet statistics are made up.
Made Up
Real
Source: I made it upSource: I made it up.
Equinix, Inc.
• Scale – size of Internet Business Exchanges, how many, where they are, future plans
• “Virtual Fly Through” Tour of the IBX - 6 minutes
IBX™ CentersIBX™ CentersIndustry Leading
SecurityIndustry Leading
Security• 5 security
checkpoints to customer equipment
• State of the art video imaging and archiving
• On site security personnel 24x365
• 5 security checkpoints to customer equipment
• State of the art video imaging and archiving
• On site security personnel 24x365
• Built to 3x power capacity
• 3 levels of power backup
• Forward looking design for growing industry
• Built to 3x power capacity
• 3 levels of power backup
• Forward looking design for growing industry
Fully RedundantBackup Power
Fully RedundantBackup Power
• 24x365 availability
• Customer trouble-shooting
• Smart Hands
• 24x365 availability
• Customer trouble-shooting
• Smart Hands
IP/CarrierKnowledgeable Staff
IP/CarrierKnowledgeable Staff
Customer FriendlySpaces – Conducting
Business & EquipmentPreparation
Customer FriendlySpaces – Conducting
Business & EquipmentPreparation• Designed from
customer experience
• Ergonomic work spaces
• Designed from customer experience
• Ergonomic work spaces
IBX™ GrowthIBX™ GrowthSites currently open or
Under DevelopmentSites currently open or
Under Development
Equinix Internet Business Exchange Model
Carriers
Internet Service Providers
ContentProviders Neutral Internet
Business ExchangeModel
• Infrastructure
• Players
• Neutrality…
Carrier Internet Exchange ModelCarrier Internet Exchange Model
Internet ServiceProviders
Carrier Operated Internet Exchange
• Infrastructure
• Players
Non Carrier-Neutral
No Content Support
Economic Distortions CarrierOperator
Job Title: Equinix Co-Founder & Chief Technical Liaison
• So, my job is to take friends out for lunches and dinners…
• Internet Operations Research &
• Write White Papers
• Summary: What did I learn?
No Internet Operations DocumentationTo make things more difficult, subtle language differences!
The Internet Operators: All about relationships
My Job: White Papers
Document Internet Operations Research– “Interconnection Strategies for ISPs”– “Internet Service Providers and Peering”– “Peering Case Studies”– “Peering Simulation Game”– “Build vs. Buy Internet Datacenter” today
New Job Title: Equinix Co-Founder & Chief Technical Liaison
White Paper: Interconnection Strategies for ISPs
SSS
GGG
UUU
A A A
S
G
U
A
CCCC
S
G
U
A
C
OC-12
OC-12
OC-12
OC-12
OC-12
Direct circuits replaced with:•OC-12 into Exchange•Fiber cross connects
Cost Drivers:•Big OC-12 Direct Circuit•Cheap Fiber Cross Connects•Aggregation Efficiency over OC-12 (2:1,3:1)•Rack FeesExchange Model Cost Formula:
ExchangeModelCost=BigDirectCircuit()+(n-1)*x/2+RackFees()
So does the Exchange-based model make sense?OC-12 $23,000/mo XC=$200/mo Rack=$1500/mo
Cost Comparison of Interconnection Strategies
$-
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64# of participants
Mo
nth
ly C
ost
of
Inte
rco
nn
ecti
on
White Paper: Internet Service Providers & Peering Transit
$$
Transit$$
ISP A
ISP B
Transit ISP
Time
Facilities Allocation
100%90%80%70%60%50%40%30%20%10%
First CarrierFirst ISP(s)First CP(s)
Secondary Carrier(s)Multiple ISPs provide CP redundancy
Facility becomes widely knownCritical mass is achievedBulk selling of rack 10-packsCross connect revenue grows rapidly Large Facility
Scaling
“The Startup Hump”
Facility fills
Peering
USNet EastNetWestNet
Peering
RoutingTable
White Paper: Case Study: Peering in the Bay Area
Transit vs. Peering Alternatives
$0
$500
$1,000
$1,500
$2,000
$2,500
$3,000
$3,500
1
11
21
31
41
51
61
71
81
91
10
1
Number of 10 Mbps
Co
st
pe
r M
bp
s
TRANSITIBXCustsGig
XODS3XOOC3XOOC12Telseon GigEXOOC48MAE-East OC-3MAE-East OC-12
Number of Mbps exchanged
Introducing:ISP Peering Breakeven Analysis Graphs
$/MbpsExchanged
Cost of Transit
Cost of Traffic Exchange in Peering Relationship
min(AccessMAX+PortMAX))
Breakeven Point(ISPs Indifferent between
Peering and Transit traffic exchange)
Initial Cost=f(Access,Rack,Port)
Allocated across the “useful capacity” at the exchange
Prefer PeeringPeeringRisk Expansion
at Exchange
Fe“Effective Peering BW”=100MbpsCost 155Mbps/100Mbps=$X/Mbps
Ge“Effective Peering BW”=155MbpsCost 155Mbps/1000Mbps=$Y/Mbps
“Effective Peering Bandwidth”
White Paper: The Peering Simulation Game
A
C
B
D
A
C
B
D
Transit Provider X
Transit Provider Y
IXW IXE
IXN
IXS
Y Y
XX
A A A A
A
A rolls 5, Pays Transit on others squares (3*$1000) Receives revenue on 6 squares (6*$2000)
$12,000 - $3,000 = $9,000
B rolls 3,Can get to IXE
Pays Transit on others squares (8*$1000) Receives revenue on 4 squares (4*$2000)
$8,000 - $8,000 = $0
B
B
B
C rolls 6,Can get to IXW, IXS
Pays Transit on others squares (11*$1000) Receives revenue on 4 squares (7*$2000)
$14,000 - $11,000 = $3,000
C
C
C
C C C
D rolls 1,Late entrant heading to IXE
Pays Transit on others squares (17*$1000) Receives revenue on 4 squares (2*$2000)
$2,000 - $17,000 = -$15,000
D
A rolls 3,Attaches to IXW
Pays Transit on others squares (13*$1000) Receives revenue on 9 squares (9*$2000)
$18,000 - $13,000 = $5,000Wants to peer with C – split costs?
YES: -$1,000 + both lose a turnNeither has to pay transit to each other!
A
AA
C
White Paper: Build Vs. Buy Internet DataCenter
• Modeling:• When does it
make sense to build your own?
• To Outsource?• Issues to consider• Economies of
Scale?• Multiple Business
Models
Service Level Agreements in Principle
• Guarantee a higher level quality• Define Measurement Criteria for adhering
to higher quality service Level• Meeting Performance Level critical to
success of business• Failure to meet Service Levels penalty
significantly worse than cost of service– eBay: $64,000/minute of downtime
SLAs in Practice• No SLA Pricing
– ISPPriceNO SLA=X
• SLA Pricing– ProbabilityOfMissingSLA=p– ISPPriceSLA1=X*(1+p)
• Network does not typically change to meet new criteria• CIO Security Audit Story• Equinix Switch SLA: 99.999% Uptime
SLA-Datacenters
• State-of-the-art Datacenters• Infrastructure Failure unlikely• SLAs rare, but…• Typically Limited to free month service• (Insufficient to potential loss customer faces)
• But what is an Internet Datacenter?• Include hosting? Include Internet Service? Include
Caching, Content Distribution?• Models help Define and Frame the Problem…
SLA Datacenter Parameters
• If there were Internet Datacenter SLAs• Parameters:
– Installation Lead Time– Service Availability– 24/7 Customer Support– Provisioning/Responsiveness
• Independent of Transport and Internet Service Issues.
Creating the Build vs. Buy Internet Datacenter Whitepaper
• Create a community white paper on issues associated with the building & scaling of Internet Datacenters
• Identify the issues that folks should consider when making the build vs. buy decision
Today:• Demonstrate the document & models to date and get
feedbackNOTE WELL:• EARLY STAGE OF BVB WHITE PAPER (v0.4)!• This Model may be of help to you in your SLA Discussions
Toured 32 (now) data centers
Some with very Secure Fiber Access…
Source: Technology Center of the Americas Tour
Some with less secure fiber entry…
Source: Don’t Ask.
Some Cat 5 Hurricane proof…
Some not quite cat proof.
Loading Dock
Great religious debate over ties
Velcro vs.Plastic cable ties vs.String
Time to white paper what I’ve seen & Facilitate BOF
Discussion• Datacenter BOF I NANOG Oct ’98 Atlanta
• Build vs. Buy White Paper
• Internet Datacenter Model– Definition of Internet Datacenter
• Focused discussion of Each Component of the Model
Internet Datacenter ModelDependence Relationship
IncreasedValue
Value
Dependence
Customer Business Model
Value
Dependence
Internet Datacenter Model Example
IXP
Ultimate Goal
• Build Financial models that allow an ISP to say:
• “Given our size requirements, it makes sense to build our own.”
Or• “Given our size requirements, it
probably makes sense to outsource this to someone who can realize economies of scale.”
+
+
=
Buy
Build
Size
Unit cost
Flavor of the Real Estate Model
• Place to put stuff
• Proximity to Fiber
• Space Size
• Utility Power
• Facilities Yard
Increased Value Relationship
Scaling of Datacenter Real Estate?
Scaling of the Real Estate Model
# Sq. Ft.
PremiumReal Estate Market(Tokyo, NYC,…)
Real Estate Market (everywhere else)
$/sq. ft.
Real Estate Size is Important
• Choose Too Small: Run out of space, lose economies of scale• Choose Too Big: Cost per Rack very high
Vs.
• Right of First Refusal (some Internet Datacenters allow this)• Expandability of Space
Commitment to DataCenter
• 1 yr contract
Vs.
• 10 year lease
Time to Market Issue – 6 – 9 months to build
Hidden Charges
• Meet Me Rooms
• Required products: – Must buy port on switch or certain volume of
BW
• Cross Connect Fees
Size & Finding Real Estate
• 100,000 sq ft in downtown DC?
• 10,000 sq ft in downtown DC
• Vs.
• Fiber Miles to Internet Datacenter
Flavor of the Infrastructure Model
• Heating Venting Air Conditioning (HVAC)
• Security Infrastructure
• Power Infrastructure
• Fire Suppression
Heating, Venting, Air Conditioning (HVAC) Infrastructure
• Controlled Humidity ~45%
• Too much Humidity: System Failure
• Too Little Humidity: Static Electricity
• Air Temperature:~72 (IBM: 65-85)
• Solutions:• DX Systems• Air Cooled• Water-Based
• How do these Scale?
HVAC ScalingDirect Expansion
Air Cooled
Chilled Water System
HVAC Cost
# Racks
$1,000/ton
$478/ton
$250/ton
HVAC System Cost: Economies of Scale
DX Systems: 25-300 tons of Cooling~ 10K sq ft or ~150-200 racks
Terramark Outsource of HVAC to FPL…
Crossover point where Chilled Water lowest per-rack solution
Cost Per Rack
$-
$5,000
$10,000
$15,000
$20,000
$25,000
$30,000
$35,000
$40,000
$45,000
$50,000
1 5 9 13 17 21 25 29 33 37 41 45 49 53Number of 10 Rack Bundles
DX System
Air Cooled
Water Cooled
Trend? Outsourcing of HVAC
FPL HVAC (Miami FL)Water Chiller
NOTA: 3 Blocks Away
Source: NAP of the Americas Tour
Security Infrastructure
• 24x7 Manned
• Access: Biometrics, card keys
• Video Surveillance
Sliding Glass
Power Infrastructure• Sun Netra 1U=90Watts
– 40U Rack=3600 Watts!
• Power Source– Power Grid (dual feeds)– Co-Generation– UPS/CPS/Generators
• Energy Storage– UPS Batteries– Flywheel (no UPS)– Fuel Cells (service)
• Monitoring Issues• Redundancy Issues• TBD: Scaling Issues
Fuel Cells for Generators
Fire Suppression Infrastructure
• Two Dominant Systems:• VESPA air sampling
• Dry pipe, pre action water system vs. FM-200
• Smoke & Heat Required to activate region
Infrastructure Operations
• Operations of HVAC, Security, Power systems
• N+1 Redundancy
Internet Datacenter Model
Transport Service Model
• Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc.
• Business Model?– Owned by Carrier? Will others come in?– Not owned by Carrier? Will Carriers come in?
• What type of services available?• How long to Provision?• Big Problem: Keeping up with BW Demand
Internet Datacenter Model
Internet Service Model
• ISP(s), Aggregator(s) like InterNAP?
• Business Model– Owned by ISP? Will others come in?– Not owned by ISP? Will others come in?
• Capacity Issues
• Latency Issues
• Interconnection Strategies Allowed?
Internet Datacenter Model
Content Provider Model
• Hoster Focus: – Installing New Servers, Provisioning Host Services,– Managing Server Farm, SLAs, Tech Support, 24/7
NOC, Back Office Solutions
• Hoster owns Datacenter?– Pro: Can buy Transit Services– Con: Expensive w/Bandwidth Hungry Apps– Con: Shared bandwidth issues (Yahoo! attack)– Con: Local Loop Expensive
• Business Model Content Distributor(s), Hoster(s)
Internet Datacenter Model
Future of the IXP?
MarketPlace: Stratification of component operations, fast creation & provisioning of packaged operations servicesImplication: IXP Content Sources, bandwidth demands
DSL Access Providers
Dial-in Modem Operators
E-Mail SvcsUSENET Svcs
WebHost SvcsSearch Engines Cache/Content
DistributionCarrier
InternetServiceProvider
Carriers
Carrier
CarrierInternetServiceProvider Carrier
CarrierInternetServiceProvider
Service Providers
InternetServiceProvider
Wireless Access Svc
VirtualISP
Sparing
Future SLA Challenges
1. Bundling of Internet Operations Services…2. Into Cohesive SLA-able services3. Notion: Trickle-down SLA
Bundle Participant Operations AgreementsFinancial Agreements
Example: Bundle Internet DataCenter, Transport, Internet, MSP. What happens when Transport service fails? Who covers the cost of the SLA?
BvB Document Availability
• All references are in the BvB white paper– Hyperlinks work in electronic version of
document– Early Stage of document: Comments Welcome!
• Send e-mail to [email protected] Subject: Build vs. Buy White Paper
Thanks to
• Sean Donelan (Equinix)• Jason Colangelo (APC)• Stephen Stuart/Phil
Steffora (MFN)• All Colo/ISPs tour
facilitators• Jay Adelson (Equinix)• Josh Horn (NOTA)• Justin Newton (NetZero)
• Friends at Exodus, Frontier Global Center, Level3, etc.
• Daniel Golding (NetRail)• Kenney Huang (AsiaInfra)• Sandip Gupta (Ensim)• Rob Glenn (Sun)• Everyone here! Please e-
mail feedback.