Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content...
Transcript of Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content...
![Page 1: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/1.jpg)
Scaling Internet TVContent Delivery
ALEX GUTARINDIRECTOR OF ENGINEERING, NETFLIX
![Page 2: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/2.jpg)
● Inventing Internet TV● Available in more than 190 countries
● 104+ million subscribers
![Page 3: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/3.jpg)
Lots of Streaming == Lots of Traffic
● Streaming 125+ million hours per dayof TV shows and movies
● Responsible for over one-third of North American peak internet traffic
![Page 4: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/4.jpg)
Architecture
![Page 5: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/5.jpg)
DATA PLANE
CONTROL PLANE
VIDEO STREAM
WEB SERVICES
WEB SERVICES
DEVICE RUNNINGA NETFLIX CLIENT APP
![Page 6: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/6.jpg)
before streaming starts = control plane =
![Page 7: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/7.jpg)
streaming = data plane =
![Page 8: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/8.jpg)
DATA PLANE
CONTROL PLANE
VIDEO STREAM
WEB SERVICES
WEB SERVICES
NETFLIXOPEN CONNECTDEVICE RUNNING
A NETFLIX CLIENT APP
![Page 9: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/9.jpg)
Open Connect is a Content Delivery Network (CDN)● A set of content delivery servers (appliances)
○ Geographically distributed
○ Attached to or embedded within ISP networks
● A way of routing requests to the optimal server
![Page 10: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/10.jpg)
We provide free Open Connect Appliances (OCA’s) for Internet Service Provider (ISP) data centers.
![Page 11: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/11.jpg)
We also provide settlement free interconnection to ISPs at Internet Exchange (IX) locations throughout the world.
![Page 12: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/12.jpg)
Wait, what’s an IX?“LIN
X” - London Internet Exchange
An Internet exchange point(IX or IXP) is a physical infrastructure through whichInternet Service Providers (ISPs) and Content Delivery Networks (CDNs) exchange Internet traffic between their networks
-Wikipedia
![Page 13: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/13.jpg)
![Page 14: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/14.jpg)
![Page 15: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/15.jpg)
![Page 16: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/16.jpg)
![Page 17: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/17.jpg)
Control Plane
![Page 18: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/18.jpg)
1. Where (on which OCAs) to pre-position the content?
2. Where to steer (which OCAs to hand out to) the client for streaming the content?
![Page 19: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/19.jpg)
Offload
![Page 20: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/20.jpg)
Pre-positioning the Content
For every file, figure out a list of OCAs it should be deployed to:
● Predict popularity
● Maximize (predicted) offload
● Minimize OCA requirements
![Page 21: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/21.jpg)
Steering Clients
For every playback session, return a set of OCAs based on:
● Content availability
● Network proximity
● OCA capacity to serve (health)
![Page 22: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/22.jpg)
Control Plane:Interesting Problems
![Page 23: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/23.jpg)
Predicting PopularityInputs● Observed popularity● View hour predictions for new titles
Goals● Maximize offload● Minimize churn (number of bytes that
need to be copied as popularity changes)
![Page 24: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/24.jpg)
Content ShardingInputs● Cluster of heterogeneous OCAs
○ Very different storage and throughput capabilities
Goals● Maximize offload (with fault tolerance)● Maximize load balance
![Page 25: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/25.jpg)
Tiered Content FillInputs● Network topology
Goals● Minimize number of bytes copied over
narrow / expensive links● Minimize deployment latency
![Page 26: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/26.jpg)
Scaling IssuesInputs● Up to hundreds of thousands of routes per
network location● Hundreds of thousands of files per OCA● Thousands of OCAs and locations
Goals● Perform steering quickly and efficiently
![Page 27: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/27.jpg)
Content DeliveryAppliances
![Page 28: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/28.jpg)
What is an OCA?● Massive web server● Serving static content● HTTP GET range requests● (Mostly) TLS connections
![Page 29: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/29.jpg)
Two Types of Appliances:“Storage”● Lots of storage, medium throughput
“Flash” (aka “offload”)● Limited storage, high throughput● Used to serve popular content where
traffic density allows for it
![Page 30: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/30.jpg)
16 core Intel Xeon E5 CPU, 128 GB RAM10 x 1 TB SSD; 24 x 10 TB HDD4 x 10GbE links
“Storage” OCA
![Page 31: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/31.jpg)
“Flash” OCA
16 core Intel Xeon E5 CPU, 128 GB RAM4 x 4 TB NVMe flash storage1 x 100GbE links
![Page 32: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/32.jpg)
OSS Stack● FreeBSD OS● NGINX web server● BIRD BGP daemon● Netflix apps to interface with
the control plane and fill content
![Page 33: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/33.jpg)
Flash OCAs● 90 Gbps of TLS encrypted traffic
○ Memory bandwidth limited● 100K TCP connections● 30-40K RPS● Most of the traffic not served from RAM
Storage OCAs● 20 - 30 Gbps (disk IO limited)
![Page 34: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/34.jpg)
OCA:Interesting Problems
![Page 35: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/35.jpg)
CPU & Memory Efficiency● Optimize
○ OS kernel○ Network adapter drivers○ Web server
● Remove bottlenecks that prevent us from achieving higher throughput
● Reduce CPU requirements (cost)
![Page 36: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/36.jpg)
IO Efficiency● Get more throughput from HDDs
○ Reduce IOPS and increase read sizes via intelligent pre-fetching
● Improve RAM and SSD caching of content
![Page 37: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/37.jpg)
Network Transport Algorithms● Improve TCP on the sender side
○ Loss detection and recovery○ Congestion control
● UDP-based transport deployable to clients● Goal is achieve higher goodput to clients
and improve customer quality of experience
![Page 38: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/38.jpg)
Open Connect Team 2016
![Page 40: Scaling Internet TV Content Deliverymascots2017.cs.ucalgary.ca/Gutarin-Keynote-Talk.pdf · Content Delivery Network (CDN) A set of content delivery servers (appliances) Geographically](https://reader033.fdocuments.net/reader033/viewer/2022051605/60074b683e258601e65e57f4/html5/thumbnails/40.jpg)
Thank you.Questions?