ED BUKOSKI • SENIOR SOFTWARE ENGINEER • NETFLIX • @EBUKOSKI

Culture, Speed, Innovation

DevTools At Netflix

Just text by itself, for impact.

Production BitBucket (i-a1044e67)

October 15, 2015

WorkCoffee & Netflix WTF

October 15, 2015

Production BitBucket (i-a1044e67)

Technical

Culture

Scale

DevTools

S C A L E & C U LT U R E

D E V TO O L S AT N E T F L I X

S U P P O R T

Agenda

Q & A

T E A M S

I N N O VAT I O N & C H A N G E

Scale & Culture

0

3.5

7

10.5

14

2008 2010 2012 2014 2016

Streaming Hours

Netflix scale

13 billion streaming hours/month

83 million subscribers

35% peak downstream US internet traffic

0

125

250

375

500

2008 2010 2012 2014 2016

Build slaves

Netflix scale

473 build slaves

1,450 developers

7,700 builds per day

Netflix Culture

Freedom and responsibility

Context, not control

Highly aligned, loosely coupled

Operating Context for DevTools

High Availability

High Resiliency

High Demand

Netflix Scale Netflix Culture

Freedom & Responsibility

Context, Not Control

Highly Aligned, Loosely Coupled

DevTools at Netflix

Change & innovationSupportTeams

Change & innovationsSupportTeams

Teams

DEVELOPERS

Context over control

DEVTOOLS

HALC

Marketing

Microservices

F&R Choose own tech stack

Choose their own tools

#NEXTBIGTHING

The obvious choice

Friction free Supported Common

Code Build CI Scale

Resiliency

Instrumentation

more…

Deploy

Change & innovationsSupportTeams

Team support

Walk-insChat Email

Team support

What we do

Z z

What we don’t doFirst responder

1:1 sessionsHands-on

Feedback into training

TicketsAutomation

Formal SLAs24/7 support

Operational support

Automated metrics & monitoring

Automated backups

Automated alerts

Automated upgrades

Operational support

Bitbucket metrics • System • Load • Database • JMX

~100 metrics total

Realtime

Alert enabled

All automated

Change & innovationSupportTeams

Change

Replace obsolete technology

Cost reduction Regulatory change(SOX)

Change

OWNER _

DEVTOOLS

ON TRACK

IMPACTED

Change campaign tools

Identifies impacted assets

Locates asset owners

Notifies owners

Track progress

Just text by itself, for impact.

Innovation

Reward

Risk

DevTools

Innovation

Reward

Risk

DevTools

Pioneers

Settlers

Legacy

Innovation

Pioneers

DevToolsDevTools

Critical mass

Partner

Freedom and responsibility enables teams to pioneer

Freedom to pick tech stack

Responsibility to tool and maintain it

Pioneers

DevToolsDevTools

Critical mass

Partner

Reality

Velocity mismatch

Focus disconnect

Tech incompatibilities

Team constraints

DevTools at Netflix - Takeaways

Blend DevTools with culture

Build paved roadsPromote best practices, tools, and culture

Create tools to facilitate change

Embrace your customers/Empower innovators

Automate operational support

Alignment and communication are critical

Thank you!

ED BUKOSKI • SENIOR SOFTWARE ENGINEER • NETFLIX • @EBUKOSKI