8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
1/78
10 deploys per dayDev & ops cooperation at Flickr
John Allspaw & Paul Hammond
Velocity 2009
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
2/78
3 billion photos
http://flickr.com/photos/jimmyroq/415506736/
40,000 photos per second
http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
3/78
Dev versus Ops
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
4/78
Its not my machines,its your code!
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
5/78
Its not my code,its your machines!
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
6/78
Spock ScottyLittle bit weird
Sits closer to the boss
Thinks too hard
Pulls levers & turns knobs
Easily excited
Yells a lot in emergencies
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
7/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
8/78
Says No all the time
Afraid that new fangled things will break the site
Fingerpointy
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
9/78
They say NO all the time
Because no one tells
them anything
Because
They say NO all the time
Because the site breaks
unexpectedly
Ops stereotype
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
10/78
Devs job is to add new features
Ops job is to keep the site stable and fast
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
11/78
Ops job is NOT to keep the site stable and fast
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
12/78
Ops job is to enable the business(this is devs job too)
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
13/78
The business requires change
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
14/78
But change is the root cause of most outages!
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
15/78
Discourage change in the interests of stability
or
Allow change to happen as often as it needs to
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
16/78
Lowering risk of changethrough tools and culture
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
17/78
Dev and Ops
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
18/78
Ops who think like devsDevs who think like ops
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
19/78
But thats me!
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
20/78
You can always think more like them
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
21/78
Tools
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
22/78
1. Automated infrastructureIf there is only one thing you do
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
23/78
Chef
Puppet
CFengine
FAI
System Imager
Cobbler
BCfg2
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
24/78Role &configuration
management
OS imaging
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
25/78
2. Shared version control
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
26/78
http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
27/78
3. One step build
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
28/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
29/78
3. One step buildand deploy
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
30/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
31/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
32/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
33/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
34/78
http://www.flickr.com/photos/mauren/2429240906/
http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
35/78
4. Feature flags(aka branching in code)
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
36/78
1.0 1.1 1.2
1.0.1
1.1.1
1.0.2
Desktop software
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
37/78
r2301 r2302
Web software
r2306
http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
38/78
http://www.flickr.com/photos/stewart/461099066/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
39/78
http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
40/78
#php
if ($cfg['enable_feature_video']){ }
{* smarty *}
{if $cfg.enable_feature_beehive} {/if}
Feature flags
http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
41/78
http://www.flickr.com/photos/healthserviceglasses/3522809727/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
42/78
http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
43/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
44/78
http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
45/78
5. Shared metrics
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
46/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
47/78
Application level metrics
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
48/78
Application level metrics
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
49/78
Adaptive feedback loops
App System Metrics
RU ok?
maybe?
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
50/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
51/78
6. IRC and IM robots
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
52/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
53/78
Culture
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
54/78
1.RespectIf there is only one thing you do
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
55/78
http://www.flickr.com/photos/aaronjacobs/64368770/
http://www.flickr.com/photos/chrisdag/2286198568/
http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
56/78
http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
57/78
http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
58/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
59/78
Developers: Talk to ops about the impact of your code:
what metrics will change, and how?
what are the risks? what are the signs that something is going wrong? what are the contingencies?
This means you need to work this out before talking to ops
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
60/78
2.Trust
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
61/78
http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
62/78
http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
63/78
http://www.flickr.com/photos/williamhook/3468484351/
http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
64/78
Ops: Be transparent,
give devs access to systems
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
65/78
3.Healthy attitudeabout failure
http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
66/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
67/78
http://www.flickr.com/photos/toms/2323779363/
http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
68/78
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
69/78
http://www.flickr.com/photos/dnorman/2678090600
http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
70/78
4.Avoiding Blame
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
71/78
Fingerpointyness
http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
72/78
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,
finding fault
blaming,covering
ass
fixing
thing
s
fixed.
whining,hiding.
hurt egos
figuring itout
Being productive
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
73/78
Being productive
problem!!!argggh!
time
fixing
thing
s
fixed.
feelingguiltyfiguring itout moveon with
life
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
74/78
Developers: Remember that someone else will
probably get woken up when your code breaks
http://www.flickr.com/photos/alex-s/353218851/
http://www.flickr.com/photos/allspaw/2819774755/
http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
75/78
Ops: provideconstructive
feedback on
current achesand pains
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
76/78
1. Automated infrastructure
2. Shared version control3. One step build and deploy
4. Feature flags
5. Shared metrics
6. IRC and IM robots
1.Respect
2.Trust
3.Healthy attitude about failure4.Avoiding Blame
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
77/78
This is not easyYou could just carry on shouting at each other
8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation
78/78
(Thank you)