Real World DevOpsRodrigo Campos (@xinu)

Velocity NY 2014

Hi I’m Rodrigo!

• Proudly crimping cables since 1992

• System Administrator

• Operations Director at walmart.com in Latin America

I’m not going to talk about

• Silver Bullets

• Unicorns

• Tools and Utilities

Instead I’ll focus on

• The challenges we faced

• How DevOps has helped us

• Bumps and bruises along the way

Our challenges

Our challenges

• Deploy a whole new platform before Black Friday

Our challenges

• Deploy a whole new platform before Black Friday

• Expand our private infrastructure to withstand web traffic spikes

Our challenges

• Deploy a whole new platform before Black Friday

• Expand our private infrastructure to withstand web traffic spikes

• Develop a company-wide agile mindset

Our challenges

• Deploy a whole new platform before Black Friday

• Expand our private infrastructure to withstand web traffic spikes

• Develop a company-wide agile mindset

• Rebuild trust between tech and business

Different companies, same problems

Different companies, same problems

• Silos not only between Developers and Operations, but within departments as well

Different companies, same problems

• Silos not only between Developers and Operations, but within departments as well

• BOFH attitude started to kick-in

Different companies, same problems

• Silos not only between Developers and Operations, but within departments as well

• BOFH attitude started to kick-in

• Over the wall production deployments

Different companies, same problems

• Silos not only between Developers and Operations, but within departments as well

• BOFH attitude started to kick-in

• Over the wall production deployments

• Tug of war between Dev and Ops

Different companies, same problems

• Silos not only between Developers and Operations, but within departments as well

• BOFH attitude started to kick-in

• Over the wall production deployments

• Tug of war between Dev and Ops

• Lots of bureaucracy and red-tape

Some phrases (roughly translated)

Some phrases (roughly translated)

• “I'll go to the 9th floor to help you kick everyone’s a**”

Some phrases (roughly translated)

• “I'll go to the 9th floor to help you kick everyone’s a**”

• “I don’t care if it’s stable as long as it passed QA testing”

Some phrases (roughly translated)

• “I'll go to the 9th floor to help you kick everyone’s a**”

• “I don’t care if it’s stable as long as it passed QA testing”

• “This is operations’ job”

Some phrases (roughly translated)

• “I'll go to the 9th floor to help you kick everyone’s a**”

• “I don’t care if it’s stable as long as it passed QA testing”

• “This is operations’ job”

• “Consider this a favor”

How DevOps was seen

So what is DevOps anyway?

So what is DevOps anyway?

• Being loosely defined can actually be a good thing!

So what is DevOps anyway?

• Being loosely defined can actually be a good thing!

• For us: “DevOps is a culture that fosters communication, transparency and collaboration"

So what is DevOps anyway?

So what is DevOps anyway?

• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!

So what is DevOps anyway?

• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!

• Transparency: I want to know your problems, I’ll make sure you know mine!

So what is DevOps anyway?

• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!

• Transparency: I want to know your problems, I’ll make sure you know mine!

• Collaboration: We’re in the same boat and I really mean it!

Operations Developers

UsThem InfoSec

Works on my Machine

We needed to change people's mindset

Source: http://youtu.be/Fx8OBeNmaWw

We needed to change people's mindset

We needed to change people's mindset

• Presentations to technology, business and human capital teams

We needed to change people's mindset

• Presentations to technology, business and human capital teams

• Reinforcing an inclusive and welcoming culture was essential

We needed to change people's mindset

• Presentations to technology, business and human capital teams

• Reinforcing an inclusive and welcoming culture was essential

• Everyone felt prepared and willing to change

We needed to change people's mindset

Typical Scenario

Infrastructure

Operations

Developers

Business

{ }

• Things go over the wall

• No alignment between tech and business

• One-way communications

• IT as a necessary evil (aka cost center)

We were already a little bit different…

Infrastructure

Operations

Developers

Business

{ }

DevOps Team

1st step

Infrastructure

Operations

Developers

Business

{ }Production Engineering

DevOps

2nd step

Infrastructure

Production ManagementDevelopers

Business

{ }

Production Engineering + { }

DevOps

InfoSec

• First and foremost: end the dossier culture

InfoSec

• First and foremost: end the dossier culture

• Communication + collaboration + transparency = safer systems

InfoSec

• First and foremost: end the dossier culture

• Communication + collaboration + transparency = safer systems

• Integrate security tests into continuous integration

InfoSec

• First and foremost: end the dossier culture

• Communication + collaboration + transparency = safer systems

• Integrate security tests into continuous integration

• Rugged DevOps: http://slidesha.re/1r87Vs7

InfoSec

• No more FUD

• What is riskier to the business?

• Lack of speed is a risk too!

InfoSec

• ITIL: “You keep using that word, I don’t think it means what you think it means”

• People often blame ITIL as an excuse for badly designed processes

• Surprisingly ITIL compliance was not our biggest problem

Compliance

• Audits are still our biggest challenge

• In our case, we have a sponsor (the CTO)

• Someone (who happens to be me) will be held accountable if something goes wrong

Compliance

The bad apples

• We’ve adopted the “No asshole rule"

• There will be obstructive people that won’t embrace change

• If you can't get rid of them, simply let them fade into oblivion

KPIs

KPIs

• Deployment Frequency

Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

KPIs

• Deployment Frequency

• Speed of Deployment

Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

KPIs

• Deployment Frequency

• Speed of Deployment

• Deployment Success Rate

Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

KPIs

• Deployment Frequency

• Speed of Deployment

• Deployment Success Rate

• Service Recovery

Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)

KPIs

• Deployment Frequency

• Speed of Deployment

• Deployment Success Rate

• Service Recovery

• CultureSource: Puppet Labs Blog (http://bit.ly/Z3gRpu)

Bumps & Bruises

Bumps & Bruises• DO:

Bumps & Bruises• DO:

• Understand beforehand what you want to achieve

Bumps & Bruises• DO:

• Understand beforehand what you want to achieve

• Use measurable KPIs

Bumps & Bruises• DO:

• Understand beforehand what you want to achieve

• Use measurable KPIs

• Explain to business areas what you’re doing

Bumps & Bruises• DO:

• Understand beforehand what you want to achieve

• Use measurable KPIs

• Explain to business areas what you’re doing

• Align with InfoSec and Compliance ASAP

Bumps & Bruises• DO:

• Understand beforehand what you want to achieve

• Use measurable KPIs

• Explain to business areas what you’re doing

• Align with InfoSec and Compliance ASAP

• Work hard on a sponsor

Bumps & Bruises

Bumps & Bruises• DON'T:

Bumps & Bruises• DON'T:

• Create a “Ninja Superstar team”

Bumps & Bruises• DON'T:

• Create a “Ninja Superstar team”

• Postpone communication

Bumps & Bruises• DON'T:

• Create a “Ninja Superstar team”

• Postpone communication

• Delay role changes

Bumps & Bruises• DON'T:

• Create a “Ninja Superstar team”

• Postpone communication

• Delay role changes

• Underestimate the managers' ability to influence behavior

• In the beginning it will be a lonely fight

• Some people will label you as the “hippie-anarchist”

Keep in mind

Results

Results• New eCommerce platform successfully deployed

Results• New eCommerce platform successfully deployed

• Only major eCommerce website in Brazil to have 100% uptime during Black Friday

Results• New eCommerce platform successfully deployed

• Only major eCommerce website in Brazil to have 100% uptime during Black Friday

• Increase in deployment frequency and success rate

Results• New eCommerce platform successfully deployed

• Only major eCommerce website in Brazil to have 100% uptime during Black Friday

• Increase in deployment frequency and success rate

• Business is incorporating agile methodologies

Any questions?

• Special thanks to: https://www.flickr.com/photos/51321237@N07/ https://www.flickr.com/photos/kh-67/ https://www.flickr.com/photos/jrparis/ https://www.flickr.com/photos/andrewdubock/ https://www.flickr.com/photos/rachelyoung/ !https://bitly.com/bundles/o_4p62vc3lid/5