High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

Copyright © 2014 BIGLOBE Inc.1

High speed fiber servicesand challenges to the core network

August 2014 @ MyNOG4

BIGLOBE Inc.

Seiichi Kawamura


Some figures about BIGLOBE (AS2518)

l 3 million consumer ISP users in Japan l 250Gbps total internet traffic

p  including enterprise, data center, transit traffic

San Jose [vPOP] -‐‑‒Equinix IX(10G)

LA [vPOP] -‐‑‒Coresite Any2(10G)

Japan [Core] -‐‑‒ 3 million broadband customers -‐‑‒ 110 colocation customers -‐‑‒ 2 DCs fully owned by BIGLOBE -‐‑‒ Great connectivity -‐‑‒ Equinix(20G), JPIX(10G), BBIX(30G), JPNAP(30G) -‐‑‒ PNI with ALL major ISPs

Hong Kong[POP] -‐‑‒HKIX(10G)

Singapore [POP] -‐‑‒Equinix IX(10G) -‐‑‒ SGIX (1G)


Network Topology in Japan

Metro Network

Metro Network

Subscribers Peering

Subscribers Peering


Growth of Traffic

Metro Network (Tokyo)

Metro Network (Tokyo)

Subscribers Peering

Subscribers PeeringPeering traffic grows 1.37x every year (Mobile Traffic 1.8x, broadband 1.2x)


1.6x growth!

The impact of high speed services

1.2x growth


Metro Network

Metro Network

Challenge1 : Keeping the metro growth down

28*10G

Metro Network

Metro Network

10*10G 5 years

Core growth 1.2x every year Keeping growth down by

•  Hot potato routing •  ECMP tuning (enhanced hash) •  Higher thresholds in capacity growth •  Studying flows and moving peers to a closer exit

No MPLS Pure IP


Challenge 2: Streaming

l 30% of FTTH traffic is streaming l We better have a cost effective way of handling

streaming traffic (and all video traffic) l Cost effective peering was our answer

p  Does not mean we love caches

Tactics

•  Use netflow and PeeringDB to find where we can pick up the traffic

•  Remote peering (unfortunately) to keep costs low •  Extensive talks with content providers •  Disclosed peering policy to pick up peering traffic

more effectively(more on this later) http://www.biglobe.co.jp/en/peering_policy.pdf


backbone

Streaming traffic basic principle

Minimize total cost of delivery and keep it from going to other links

IX ports, transit ports Keep traffic from overflowing into ports with high utilization. Can be used as backup links,

but should control traffic

Try to peer at router closest to end userLower backhaul cost

Lower peering costsCosts are fixed

But actual networks don’t look like this…


Actual Network

Access network concentrator

Core routingPeering

Ideally, we want video traffic to be here, but can we peer where the access concentrators are?

No.

Can we have caches here? Yes, but GGC, AANP, Open Connect, and what next? A

horrible cost model

More flexible DC interconnects may help


Challenge3 : Peering

l Domestic peering was REALLY an issue p  IX port costs

Ø 2 years ago, 10G ports cost 20,000 usd per month

p  the burden of peering, mistrust, lack of communication between peers Ø No place to talk about peering

l We wanted a better relationship with content providers


Changes coming from Mobile and Cloud area

l More mobile and cloud traffic p Broadband at 1.3x growth while mobile at

1.8x growth l Content providers have different

requirements than ISPs p semi-full route p simple route servers p latency aware p mtu9000 p fast detection using BFD p DoS protection as a service


Evolution of peering community

l Japan did not have a place to talk about peering until a few years ago p PAST: People disliked talking about peering at

JANOG l Peering relied heavily on ISPs and Telcos

•  BoFs, study councils, talks at NOGs, and joint efforts by the IXPs and the community – more focus on content/dc and its requirements


community activities

l Google Groups : Peering in Japan p local Japanese language only p discussion on latest peering issues p no IX personnel on list p host Peering BoFs

l Tutorials p IXP provides low cost tutorials (available regularly) p free tutorials at JANOG (not always available)

l CloudIX Study Council p Group of members(BGP operators) in BBIX doing

technical experiments p sharing skills and helping out each other to peer


Challenge4 : Automation[ a work in progress ]

l Our data center is well automated p  in the past

dc config change >>>>>>>>>> peering p  dc requires orchestration with servers and other

virtual devices l Since we started peering more (and more

globally) we are now needing better automation tools

l Current implementation p Simple html(form) +mysql+ jinja2 to generate

configs via Web UI Ø Trying to shift to Django

p Exscript to push configs to router


Other misc challenges

l Unnecessary traffic avoidance p P2P services that have security problems are

controlled at the edge p BCP38 to stop sending unnecessary traffic

Ø  still in initial deployment phase l Higher density with better power efficiency

p Working to keep PUE around 1.2-1.3 p Switching to higher density line cards

l Measuring actual user experience p “1Gbps service” is more marketing talk than actual

experience p Measuring response to popular sites (alexa) p Python script to crawl command and filter out noise

data “hping -p 80 -c 5 --syn [url]” works well with unix, PSPing works well with Windows


Next steps for evolution

l MPLS, TE, auto-bandwidth p Not necessary for domestic consumer service

backbone, but need TE to control global network p We did not need this in the past since we did not

provide VPN services (we may in the future) p we have a vxlan network running, so considering

EVPN as one of the solutions l Better network management and security

p BMP and other tools for better peering management p APIs to the router p BGP Flow Spec

l Differentiation of services inside the backbone p Last mile QoS and internet fast lane?


Summary

l Higher speed services to the consumers, means new challenges to the core network

l Peering really helps, but it requires a good ecosystem to be helpful p  diversity in IX services, fair pricing, carrier

neutrality, open talks about peering l Handling streaming and video, is still a big

headache p  is there a better way than everybody has their

own cdn? l Automation and tools are key to handling

bigger traffic in a cost effective way l Measurements make a difference in user

experience. “1Gbps marketing talks” don’t

High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

Internet

Transcript of High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura