Copyright © 2014 BIGLOBE Inc.1
High speed fiber servicesand challenges to the core network
August 2014 @ MyNOG4
BIGLOBE Inc.
Seiichi Kawamura
Copyright © 2014 BIGLOBE Inc.2
Some figures about BIGLOBE (AS2518)
l 3 million consumer ISP users in Japan l 250Gbps total internet traffic
p including enterprise, data center, transit traffic
San Jose [vPOP] -‐‑‒Equinix IX(10G)
LA [vPOP] -‐‑‒Coresite Any2(10G)
Japan [Core] -‐‑‒ 3 million broadband customers -‐‑‒ 110 colocation customers -‐‑‒ 2 DCs fully owned by BIGLOBE -‐‑‒ Great connectivity -‐‑‒ Equinix(20G), JPIX(10G), BBIX(30G), JPNAP(30G) -‐‑‒ PNI with ALL major ISPs
Hong Kong[POP] -‐‑‒HKIX(10G)
Singapore [POP] -‐‑‒Equinix IX(10G) -‐‑‒ SGIX (1G)
Copyright © 2014 BIGLOBE Inc.3
Network Topology in Japan
Metro Network
Metro Network
Subscribers Peering
Subscribers Peering
Copyright © 2014 BIGLOBE Inc.4
Growth of Traffic
Metro Network (Tokyo)
Metro Network (Tokyo)
Subscribers Peering
Subscribers PeeringPeering traffic grows 1.37x every year (Mobile Traffic 1.8x, broadband 1.2x)
Copyright © 2014 BIGLOBE Inc.5
1.6x growth!
The impact of high speed services
1.2x growth
Copyright © 2014 BIGLOBE Inc.6
Metro Network
Metro Network
Challenge1 : Keeping the metro growth down
28*10G
Metro Network
Metro Network
10*10G 5 years
Core growth 1.2x every year Keeping growth down by
• Hot potato routing • ECMP tuning (enhanced hash) • Higher thresholds in capacity growth • Studying flows and moving peers to a closer exit
No MPLS Pure IP
Copyright © 2014 BIGLOBE Inc.7
Challenge 2: Streaming
l 30% of FTTH traffic is streaming l We better have a cost effective way of handling
streaming traffic (and all video traffic) l Cost effective peering was our answer
p Does not mean we love caches
Tactics
• Use netflow and PeeringDB to find where we can pick up the traffic
• Remote peering (unfortunately) to keep costs low • Extensive talks with content providers • Disclosed peering policy to pick up peering traffic
more effectively(more on this later) http://www.biglobe.co.jp/en/peering_policy.pdf
Copyright © 2014 BIGLOBE Inc.8
backbone
Streaming traffic basic principle
Minimize total cost of delivery and keep it from going to other links
IX ports, transit ports Keep traffic from overflowing into ports with high utilization. Can be used as backup links,
but should control traffic
Try to peer at router closest to end userLower backhaul cost
Lower peering costsCosts are fixed
But actual networks don’t look like this…
Copyright © 2014 BIGLOBE Inc.9
Actual Network
Access network concentrator
Core routingPeering
Ideally, we want video traffic to be here, but can we peer where the access concentrators are?
No.
Can we have caches here? Yes, but GGC, AANP, Open Connect, and what next? A
horrible cost model
More flexible DC interconnects may help
Copyright © 2014 BIGLOBE Inc.10
Challenge3 : Peering
l Domestic peering was REALLY an issue p IX port costs
Ø 2 years ago, 10G ports cost 20,000 usd per month
p the burden of peering, mistrust, lack of communication between peers Ø No place to talk about peering
l We wanted a better relationship with content providers
Copyright © 2014 BIGLOBE Inc.11
Changes coming from Mobile and Cloud area
l More mobile and cloud traffic p Broadband at 1.3x growth while mobile at
1.8x growth l Content providers have different
requirements than ISPs p semi-full route p simple route servers p latency aware p mtu9000 p fast detection using BFD p DoS protection as a service
Copyright © 2014 BIGLOBE Inc.12
Evolution of peering community
l Japan did not have a place to talk about peering until a few years ago p PAST: People disliked talking about peering at
JANOG l Peering relied heavily on ISPs and Telcos
• BoFs, study councils, talks at NOGs, and joint efforts by the IXPs and the community – more focus on content/dc and its requirements
Copyright © 2014 BIGLOBE Inc.13
community activities
l Google Groups : Peering in Japan p local Japanese language only p discussion on latest peering issues p no IX personnel on list p host Peering BoFs
l Tutorials p IXP provides low cost tutorials (available regularly) p free tutorials at JANOG (not always available)
l CloudIX Study Council p Group of members(BGP operators) in BBIX doing
technical experiments p sharing skills and helping out each other to peer
Copyright © 2014 BIGLOBE Inc.14
Challenge4 : Automation[ a work in progress ]
l Our data center is well automated p in the past
dc config change >>>>>>>>>> peering p dc requires orchestration with servers and other
virtual devices l Since we started peering more (and more
globally) we are now needing better automation tools
l Current implementation p Simple html(form) +mysql+ jinja2 to generate
configs via Web UI Ø Trying to shift to Django
p Exscript to push configs to router
Copyright © 2014 BIGLOBE Inc.15
Other misc challenges
l Unnecessary traffic avoidance p P2P services that have security problems are
controlled at the edge p BCP38 to stop sending unnecessary traffic
Ø still in initial deployment phase l Higher density with better power efficiency
p Working to keep PUE around 1.2-1.3 p Switching to higher density line cards
l Measuring actual user experience p “1Gbps service” is more marketing talk than actual
experience p Measuring response to popular sites (alexa) p Python script to crawl command and filter out noise
data “hping -p 80 -c 5 --syn [url]” works well with unix, PSPing works well with Windows
Copyright © 2014 BIGLOBE Inc.16
Copyright © 2014 BIGLOBE Inc.17
Next steps for evolution
l MPLS, TE, auto-bandwidth p Not necessary for domestic consumer service
backbone, but need TE to control global network p We did not need this in the past since we did not
provide VPN services (we may in the future) p we have a vxlan network running, so considering
EVPN as one of the solutions l Better network management and security
p BMP and other tools for better peering management p APIs to the router p BGP Flow Spec
l Differentiation of services inside the backbone p Last mile QoS and internet fast lane?
Copyright © 2014 BIGLOBE Inc.18
Summary
l Higher speed services to the consumers, means new challenges to the core network
l Peering really helps, but it requires a good ecosystem to be helpful p diversity in IX services, fair pricing, carrier
neutrality, open talks about peering l Handling streaming and video, is still a big
headache p is there a better way than everybody has their
own cdn? l Automation and tools are key to handling
bigger traffic in a cost effective way l Measurements make a difference in user
experience. “1Gbps marketing talks” don’t
Top Related