Netflix Viewing Data Microservices - How We Know Where ......Netflix Viewing Data Microservices -...
90
How We Know Where You Are in House of Cards Netflix’s Viewing Data Microservices @zimmermatt
Transcript of Netflix Viewing Data Microservices - How We Know Where ......Netflix Viewing Data Microservices -...
How We Know Where You Are in
House of Cards
Netflix’s Viewing Data Microservices
@zimmermatt
Netflix Scale
@zimmermatt
Netflix Scale
62,000,000
@zimmermatt
Netflix Scale
62,000,000
50+
@zimmermatt
Netflix Scale
62,000,000
50+
3,000,000,000+
@zimmermatt
Netflix Scale
62,000,000
50+
3,000,000,000+
1000+
@zimmermatt
Netflix Scale
62,000,000
50+
3,000,000,000+
1000+
37%
@zimmermatt
Viewing Data
@zimmermatt
Viewing Data: History
Who
@zimmermatt
Viewing Data: History
Who, What
@zimmermatt
Viewing Data: History
Who, What, When
6/10/2015 11:55
@zimmermatt
Viewing Data: History
Who, What, When, Where
6/10/2015 11:55
@zimmermatt
Viewing Data: History
Who, What, When, Where, How Long
6/10/2015 11:55
@zimmermatt
Viewing Data: Telemetry
@zimmermatt
What have I watched?
@zimmermatt
What have I watched?
@zimmermatt
Where was I at?
@zimmermatt
What else am I watching?
@zimmermatt
What Happened?
Video Bitrate Usage
@zimmermatt
Viewing Data Requests
@zimmermatt
Viewing Data Requests
@zimmermatt
Viewing Data Requests
@zimmermatt
Viewing Data Requests
@zimmermatt
Viewing Data Requests
@zimmermatt
Gen 3 (Existing) System
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Gen 3 System’s Architecture & Scale
Logs
@zimmermatt
Gen 3 System’s Architecture & Scale
Viewing History
Logs
@zimmermatt
Gen 3 System’s Architecture & Scale
Viewing History
His
tory
Cac
he
Logs
@zimmermatt
Gen 3 System’s Architecture & Scale
Viewing History
His
tory
Cac
he
Logs
Summarized Sessions
@zimmermatt
Gen 3 System’s Architecture & Scale
Viewing History
His
tory
Cac
he
Vie
win
g Se
rvic
e
Stateful Tier
…
Stateless Tier
(History, logs,
and fallback)
Logs
Summarized Sessions
@zimmermatt
Gen 3 System’s Architecture & Scale
Viewing History
His
tory
Cac
he
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Logs
Summarized Sessions
@zimmermatt
Growth
2015
@zimmermatt
Virtuous Cycle
Improved Personalization
Better Experience
Viewing
@zimmermatt
Stateful Instance Count Over Time
@zimmermatt
Guiding Lights
Externalize State…
Viewing History
His
tory
Cac
he
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Logs
Summarized Sessions
@zimmermatt
Externalize State… DENIED!
Viewing History
His
tory
Cac
he
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Logs
Summarized Sessions
@zimmermatt
Service Decomposition
Video Bitrate Usage
@zimmermatt
Service Decomposition
Start Stop
@zimmermatt
Active Sessions
Last Position
Viewing History
Data Feed
Service Decomposition
Start Stop
@zimmermatt
Active Sessions
Last Position
Viewing History
Data Feed
Service Decomposition
Start Stop
Event Stream
Stream State
Session Summary
@zimmermatt
Service Decomposition
Collect
Process
Provide
Events
Queries
@zimmermatt
Gen 3 Review
Viewing History
His
tory
Cac
he
Logs
Summarized Sessions
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Gen 4 Generic Architecture
Processor Provider Collector
@zimmermatt
Gen 4 Generic Architecture
Processor Provider Collector
Queue
@zimmermatt
Gen 4 Generic Architecture
Processor Provider Collector
Materialized Views
Raw Data
Queue
@zimmermatt
Gen 4 Generic Architecture
Processor Provider Collector
Materialized Views
Cache
Raw Data
Queue
@zimmermatt
Gen 4 Architecture
Session Summary Processor
Summarized Sessions
Materialized View
Session Events Collector
Event Data Raw Data
Log Collector
Logs Raw Data
Viewing History
Positions
Active Sessions
Materialized View
Session Accounting Provider
Playback History Provider
History Raw Data
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing
Edge
Se
rvic
e
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Log Collector
Logs
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Shadow Testing: Start Play Event
Edge
Se
rvic
e
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Start Play Event
Edge
Se
rvic
e
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Start Play Event
Edge
Se
rvic
e
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Start Play Event
Edge
Se
rvic
e
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
@zimmermatt
Shadow Testing: Start Play Event
Edge
Se
rvic
e
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Viewing History
Positions
Playback History Provider
History
Shadow Testing: Read Latest Position
Edge
Se
rvic
e
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Read Latest Position
Edge
Se
rvic
e
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Shadow Testing: Read Latest Position
Edge
Se
rvic
e
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Edge
Se
rvic
e
Summarized Sessions
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Shadow Testing: Stop Event
Summarized Sessions
Session Summary Processor
Summarized Sessions
@zimmermatt
Shadow Testing
Edge
Se
rvic
e
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
Session Summary Processor
Summarized Sessions
Session Events Collector
Event Data
Log Collector
Logs
Active Sessions
Session Accounting Provider
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Traffic Dial Session Summary
Processor
Summarized Sessions
Session Events Collector
Event Data
Log Collector
Logs
Active Sessions
Session Accounting Provider
Edge
Se
rvic
e
Viewing History
Positions
Playback History Provider
History
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Traffic Dial Session Events
Collector
Event Data
Log Collector
Active Sessions
Session Accounting Provider
Edge
Se
rvic
e
Positions
Playback History Provider
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
Session Summary Processor
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Traffic Dial Session Events
Collector
Event Data
Log Collector
Active Sessions
Session Accounting Provider
Edge
Se
rvic
e
Positions
Playback History Provider
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
Session Summary Processor
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
…
Active Sessions
Latest Positions
Session Summary
Stateless Tier
(History, logs,
and fallback)
Events
Traffic Dial Session Events
Collector
Event Data Ed
ge S
erv
ice
Summarized Sessions
Viewing History
His
tory
Cac
he
Logs
Session Summary Processor
@zimmermatt
Active Sessions
Session Accounting Provider
Positions
Playback History Provider
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
… Latest Positions
Stateless Tier
(History, logs,
and fallback)
Traffic Dial
Edge
Se
rvic
e
Viewing History
His
tory
Cac
he
@zimmermatt
Positions
Playback History Provider
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
… Latest Positions
Stateless Tier
(History, logs,
and fallback)
Traffic Dial Log
Collector
Edge
Se
rvic
e
Logs
@zimmermatt
Vie
win
g Se
rvic
e
Stateful Tier
0
1
n-2
n-1
… Latest Positions
Stateless Tier
(History, logs,
and fallback)
Traffic Dial
Edge
Se
rvic
e
Playback History Provider
Viewing History
His
tory
Cac
he
@zimmermatt
15
Migration Status Session Events
Collector
Event Data
Active Sessions
Session Accounting Provider
Edge
Se
rvic
e
Session Summary Processor
Summarized Sessions
Log Collector
Logs
Viewing History
Positions
Playback History Provider
History
@zimmermatt
Remember this?
@zimmermatt
This is a typical graph now…
@zimmermatt
The Road Ahead
@zimmermatt
Key Points
• Devour the whale a bite at a time.
• Use CQRS / Event Sourcing. Backup the raw event
data. Design for idempotency so it can be replayed.
• System architectures are throw-away artifacts, useful for
only a limited time.
@zimmermatt
“If you don’t like how the table is set, turn over the table.”
Frank Underwood
@zimmermatt
