DevOps: It’s Not Just For WebOps And We Have The Metrics To Prove It

Gene  Kim  (IT  Revolu0on  Press)  Jez  Humble  (ThoughtWorks)  James  Turnbull  (Puppet  Labs)  John  Willis  (Enstra0us  /  Dell)  

Presenters James Turnbull, VP of Community, Evangelism, and Business

Development •  Prominent voice in DevOps community •  Author of several technical books including The LogStash

Book

Gene Kim, IT researcher and author •  Founder and former CTO of Tripwire for 13 years •  Author "The Phoenix Project: A Novel About IT, DevOps, and

Helping Your Business Win” and “Visible Ops Handbook”

Jez Humble, Principal, ThoughtWorks •  Co-author of Continuous Delivery •  Loud mouth on effective practices for software delivery

The Big Why?

• Desire  to  understand  the  health  and  habits  of  DevOps  community  

• Understand  the  link  between  behaviors  and  performance  

• Mi0gate  the  “echo  chamber”  effect:  what  is  the  evidence  of  improved  performance  

• Prove  that  DevOps  isn’t  just  for  unicorns…    it’s  for  the  horses,  too.  

Goals For This Presentation

• Show  you  how  to  do  cross-­‐sec0on  popula0on  studies  

• Survey  Design  • Survey  Execu0on  • Survey  Analysis  

• Share  our  top  findings  • Get  your  help  for  our  2013-­‐2014  study  

Where Did The High Performers Come From?

Over Ten Years, We Benchmarked 1500+ IT Orgs

Intuitive Analytical

Academic Study

Population Sample Population Sample

Academic Study

Population Sample

What Is Cross-Section Population Study

Population Sample

Academic Study

Population Study

Benchmark

ROI Story Best

Practice

Peer Recommendation

Anecdote

Prior Experience

We Are Here

Vendors Consultants Industry

Analyst

Tradeshows

Industry Press

Blogshere

High Performing IT Organizations

§  High  performers  maintain  a  posture  of  compliance  § Fewest  number  of  repeat  audit  findings  § One-­‐third  amount  of  audit  prepara0on  effort  

§  High  performers  find  and  fix  security  breaches  faster  § 5  /mes  more  likely  to  detect  breaches  by  automated  control  § 5  /mes  less  likely  to  have  breaches  result  in  a  loss  event  

§ When  high  performers  implement  changes…  § 14  /mes  more  changes  § One-­‐half  the  change  failure  rate    § One-­‐quarter  the  first  fix  failure  rate    § 10x  faster  MTTR  for  Sev  1  outages  

§ When  high  performers  manage  IT  resources…  § One-­‐third  the  amount  of  unplanned  work  § 8  /mes  more  projects  and  IT  services  § 6  /mes  more  applica0ons    

Source: IT Process Institute, 2008

Visible Ops: Playbook of High Performers

• The IT Process Institute has been studying high-performing organizations since 1999

• What is common to all the high performers?

• What is different between them and average and low performers?

• How did they become great?

www.ITPI.org

The Story Begins…

Picture credit: Flickr: Floralgal

I Know!

12  

Lesson:

You are only as smart as the average

of the top 5 people you hang out with

13  

2012 DevOps Survey

• Written by Puppet Labs, Gene Kim, and Jez Humble

•  Ran through December of 2012

•  Goal: Get a snapshot of the current state of the health

and habits of the DevOps community

How To Do A Cross Section Population Study

• Survey  design  • Survey  execu0on  • Survey  analysis  

The Vision: DevOps Survey Of Practice II

• Performance  • Behaviors  • Demographics  • Psychographics  

• Planned  target  was  30  ques0ons  • We  asked  for  a  budget  of  8  ques0ons  

16  

Survey Design

In  the  ideal,  we  want  to  be  able  to  say,  “the  more  you  do  [A,  B,  C],  the  more  performance  improves”  

X axis: independent variables DevOps behaviors hypothesized to improve performance

Y axis: dependent variables DevOps performance measures

Y-Axis: What Performance Looks Like

• How  ofen  do  you  deploy  code?  (e.g.,  daily,  weekly,  monthly,  yearly)  

• What  is  the  average  lead  0me  of  a  change?    (i.e.,  how  long  does  it  take  to  go  from  “code  commihed”  to  “code  successfully  running  in  produc0on?”)    

• What  percentage  of  your  changes  require  rollbacks  or  hoiixes?  

• On  average,  how  long  does  it  take  to  restore  service  when  something  goes  wrong?  (e.g.,  a  day,  a  week,  a  month,  a  year)  

18  

X-Axis: What Behaviors Result In Performance

• Are  environment  and  infrastructure  changes  (i.e.,  everything  except  for  code)  checked  into  revision  control?  

• Is  there  an  automated  process  to  deploy  environment  and  infrastructure  changes?  

• Who  performs  code  deployments?    (e.g.,  Dev,  Ops,  both)  

• Who  is  on  the  hook  for  produc0on  support  (e.g.,  Dev,  Ops,  both)  

19  

Next Step: Survey Execution

• Once  the  survey  instrument  is  ready,  then  you  need  people  to  take  the  survey  

• Michelle  Carroll  and  the  Puppet  Labs  marke0ng  machine  went  to  work  

• Mailing  lists  • Twiher  • Reddit  

21  

The Results

• 4039  completed  survey  responses  in  30  days!!!  

• This  is  amazing.    With  a  primary  research  firm,  this  type  of  popula0on  size  usually  would  cost  $100-­‐200K!  

22  

Lesson: Do Not Change Answers Mid-Survey

Uh, Oh… Now what?

Lesson:

Get data analyst on the team BEFORE you do survey design!

25  

Where Do You Find Your Own Wally?

• Steal  a  consultant  ;)  • R  User  Groups  • University  students  in  search  of  a  thesis  

• Economics  students  a  plus  (sta0s0cs,  applied)  • Ex-­‐Wall  Street  investors  (not  recommended)  

Wally Zabaglio, Puppet Labs Data Analyst –  Former specialized analytics consultant at The Nielsen

Company; analyst for the US Department of Energy's Office of Environmental Management

–  Primary analyst on DevOps survey results

27  

Data Cleanup

• Used  Google  Refine  to  clean  up  data  and  compensate  for  “Gene  screwup”  

• Used  R  to  do  analysis  • Look  for  signal  

• Explore  data  • Run  correla0ons  

Key Findings

High Performing DevOps Teams

• They’re  more  agile  • 30x  more  frequent  deployments  • 8,000x  shorter  lead  0me    (minutes/hours  vs.  months/quarters)  

• They’re  more  reliable  • 2x  the  change  success  rate  • 12x  faster  MTTR  

31

Technical Practices Matter

•  89% are using infrastructure version control •  82% are using automated code deployments

Deliberate Practice Matters

Organizations that implemented DevOps practices over 12

months ago were 5x more likely to be high performing than

organizations that weren’t implementing DevOps at all.

The Lost Hypotheses *

o  Who performs code deployments?

o  Who is on the hook for production support?

Measuring Culture

“I’ll tell you EXACTLY what devops means.

Devops means giving a shit about your job enough to not pass

the buck. Devops means giving a shit about your job enough to

want to learn all the parts and not just your little world.

Developers need to understand infrastructure. Operations

people need to understand code. People need to fucking work

with each other and not just occupy space next to each other.”

John Vincent | @lusis | http://bit.ly/12DkRhf

Measuring Culture

Trust (and Verify)

Continuous Improvement vs Risk Management Theater

“Human Error”

Heroes / "high performers"

The tribe is operationally strong *

   

70%  Opera0ons  people  

   

The tribe is operationally strong

   

This  is  not  a  Dev  takeover  

   

The tribe is enterprise too!

 

26%  -­‐  500  to  9999  16%  -­‐  10K  plus  

   

Overcoming barriers to DevOps adoption

Biggest barriers were cultural

•  Lack of manager buy-in

•  Lack of team buy-in

•  Value of DevOps not understood outside my

group

What are organizations looking for in DevOps skills?

•  Coding/scripting (84%)

•  People skills (60%)

•  Process re-engineering skills (56%)

•  Experience with specific tools (19%)

Challenges to culture

 It's  easier  to  teach  tools  than  teach  people  skills  

   

Preparing For DevOps Survey 2013-2014

• How  do  we  measure  culture?  • High  management  vs.  low  trust  management  styles  (and  link  to  team  size)  

•  Industry  code  • Beher  define  performance  variables  to  enable  regression  analysis  (Likert-­‐type  scale)  

• Business  cri0cality:  system  of  engagement  vs.  system  of  record  

• What  else?    We  want  to  hear  your  thoughts!  

If you … Then … … want to learn more about the findings •  Read the full-length report:

http://puppetlabs.com/devops •  Share the infographic:

http://puppetlabs.com/2013-state-of-devops-infographic

•  Get these slides: http://slideshare.net/realgenekim/DevOpsSurveyOfPractice

… have ideas on the DevOps Survey Of Practice 2013-2014

•  Come talk with us!

•  Join our Google+ Community: https://plus.google.com/communities/104363243715545285555

… share your stories of transformation •  Go to FlowCon! http://flowcon.org

… get a free copy of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

•  Go to the AppFirst booth at 2:40pm. Gene will be signing books!