Devops, it's not about tooling

But tooling is so much fun !

@KrisBuytaert

Kris Buytaert

I used to be a Dev,

Then Became an Op

Chief Trolling Officer and Open Source Consultant @inuits.eu

Everything is an effing DNS Problem

Building Clouds since before the bookstore

Some books, some papers, some blogs

Evangelizing devops

What's this Devops thing really about ?

World , 200X-2009Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and lots of others ..Gent , October 2009Mountain View , June 2010Hamburg , October 2010Boston, March 2011Mountain View, June 2011....

Amsterdam , 5th aniversery June 28-30Student Discounts Available

devops, a definition:

Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.

Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.

Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.

Institute training on the job.

Institute leadership The aim of supervision should be to help people and machines and gadgets do a better job.

Drive out fear, so that everyone may work effectively for the company.

Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.

Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.

Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership.

Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.

Remove barriers that rob people in management and in engineering of their right to pride of workmanship.

Institute a vigorous program of education and self-improvement.

Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job.

William Edwards Deming 1986, Out of the Crisis.

http://en.wikipedia.org/wiki/W._Edwards_Deming

C(L)AMS

Culture

(Lean)

Automation

Measurement

SharingDamon Edwards and John Willis

Gene Kim

DevOps is a cultural andprofessional movementAdam Jacob

Whats in it for you ?

Faster time to marketFeatures go live in hours vs years

In a more safe (Secure)

Reliable fashionFully automated

More happy {customers,developers,managers,investors}

How did we get here ?

The Old Days

Put this Code Live, here's a tarball NOW!

What dependencies ?

No machines available ?

What database ?

Security ?

High Availability ?

Scalability ?

My computer can't install this ?

Devs vs Ops

(Historically) Different Goals

DevelopmentNew releases

New Features

New platforms

New architectures

Functional Req

OperationsStable Platform

No Downtime

Scalable Platform

Non Functional Req

Culture,automation, Measturement, sharing

Analyze This

What are devs nagging aboutSlow builds ?

No enviroments ?

What are ops nagging aboutBad artefacts ?

No logs ?

What is mgmt nagging about Quality / Feedback ?

Are you speaking the same language ?

Configuration management vs configuration management

What is Operations ?

What is #devops ?

Crossfunctional Team

Build a project team with skills from all overDevelopment

Continuous Integration

Testing

Infrastructure (HA/ Scale/ Performance)Deployment

Measurement

Seat them together !

Goal = Help the business

Shared Backlog

CHALLENGE : Getting Non Functional Requirements in the development backlogOn team level

On management level

On business user level

Put the Product Owner on Call

There's a tool for that

Redmine

Gitlab

Trac

Phabricator

Taiga

Enable Communication

There's a tool for that

Good old Irc,

Xmpp (ejabberd,openfire, ),

Jitsi

Mattermost ,

.

Build Trust

ExperimentDev

Test

Prod

Automate all the things

Measure success

Measure Failure

With great power ...

Your code will go to production..You will be able to fix it ..You will have access to the logsAccess to the metrics...

While we talk about animals..

Your machines as Cattle

Treat your people as pets

Bond

Internal Devopsdays

Internal Open Source Days

Hack Days

Teach a collegue days

Ramdon Lunch meetups

Eat Cake

Both inside and outside the office

What tool?

Culture, Automation,Measurement,Sharing

Automate all the things

Buildreproducable builds are undiscussable

Testtesting reduces risk

automate deployments of your test infra

DeployInfrastructure as Code

100% automation

Can you rebuild your infrastructure ?

Nirvana

An ecosystem that supports continuous delivery, from infrastructure, data and configuration management to business.Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours sometimes even minutesno matter what the size of a project or the complexity of its code base.Continuous Delivery , Jez Humble

How do we get there ?

Use Version Control,

No Excuses

Also for scripts/config/cookbooks,manifests,etc

Infra Requirements:

Git Gitlab

Github

Gitolite

Gitblit

Gitorious

Continuous Integration

Builds

Nightly Builds

Builds with tests

Nightly Builds with tests

Frequent integration

Continuous Integration

CI Tools

Hudson

Jenkins

A zillion plugins

Go (Cruise Control)

Travis

GitlabCI

Also test your (Puppet/Chef/CFengine)

Infra Requirements:

Issuetracker / PM Tools

Communication Tools

Git

CI Tool

Build Pipelines

Jenkins Pipeline

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Test Automation

Unit tests

Regression tests

TDD

BDD

Selenium

Fitnesse

Cucumber

Can you Automate your Pipeline Creation ?

Puppetizing Pipelines

Templating Pipelines

Workflow Plugin

JJB

Can you Automate your Pipeline Creation ?

Pipeline as Code

Jenkins Job DSL

Pipeline Plugin

devops ( continuous delilvery

How many times a day ?

10 @ Flickr

Deployments used to be pain

Nobody dared to deploy a site

Practice makes perfect

Knowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to enable the business goals. We strongly feel that in order to do that you must have the ability to deploy code quickly and safely. Even if the business goals are to deploy strongly QAd code once a month at 3am (its not for us, we push all the time), having a reliable and easy deployment should be non-negotiable." Etsy Blog upon releasing Deployinatorhttp://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

Infrastructure as Code

Treat configuration automation as code

Development best practicesModel your infrastructure

Version your cookbooks / manifests

Test your cookbooks/ manifests

Dev/ test /uat / prod for your infra

Model your infrastructure

A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )

IAC -ne scripting

Surviving the 10th floor test

Pick your poison

CfEngine

Puppet

Chef

Ansible

SaltOr combine them

Infra Requirements:

Git

CI Tool

Test Frameworks

IAC Framework Includes Reporting

Also requires the above

Culture Hack:

Set up CI / CD for your infrastructure first, If the people running your infra don't know how CI/CD works , how do you expect them to support / teach your application teams ?You also get them to learn about the tooling they will need to support and they will share the pain and the joy of the application developers

Spinning up stacks

Foreman

Terraform ..

Pitfall : Require a CI Tool to deploy

Why ops like to package

Packages give you features

Consistency, security, dependencies

Uniquely identify where files come from

Package or cfg-mgmt

Source repo not always available

Firewall / Cloud etc ..

Weird deployment locations , no easy access

Little overhead when you automate

CONFIG does not belong in a package

#devopsdays 2010 Open Space Conclusions

Always package software YOU deployExceptions: code that changes faster than you can package it. (Very rare)

Do NOT package Config FILES ,Use a cfgmgmt tool for this

Languages are still reinventing the wheel :(

fpm

Say thanks to Jordan Sissel

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Package

Repository Management

Repository Management

PulpPro : MirroringLove

Con : Mongo, Stability, .deb, forge

PRM

Yum Repo Server by IS24

Aptly

Deploy Software

Repo per environment

Package {'mysoftware': ensure => present}

Strict versioning in config ?Ensure => '0.98.4'

Deployment isn't the End

Orchestration

Manage 1000 nodes,

Trigger

Upgrades

Config Runs

Service Changes

Think :

Mcollective

Consul

Rundeck

Ansible

Orchestration 2nd gen

Aka ChoreographyWhile ....

First install X

When it is ready configure Y

Then notify Z

Think : Zookeeper, Serf , Juju

The New Days

Put this Code Live, here's Docker Container NOW!

No machines available ?

What database ?

Security ? What distro is this even ?

High Availability ?

Scalability ?

How do we monitor his ?

How did you build this ?

If all you know is docker, every whale looks like a private cloud

Image Build by devs,
maintained by nobody

Closing the gaps between dev and ops
Again

Who build this ?

What about Security

How do you even build a container

How do you build the hosts that run the containers ?

Infrastructure as code ++

Now:

Use containers to build, test, package

Spin up development environments

Teach your developers about state

Learn how to monitor, manage containers

In 18 months:Move to production

Culture, Automation,Measurement :measure all the thingsSharing

Monitoring is usually an aftertoughtENOBUDGET, ENOTIME

An 2008 OLS Paper

We have bloated Java tools

Some open Core stuff

DYI folks want traditional Nagios

DBA Required

#monitoringsucks

John Vincent (@lusis), june 2011

A sub #devops movement

https://github.com/monitoringsucks/

Why #monitoringsucks

Manual config (gui)

Not in sync with reality

Hosts only

Services sometimes

Aplication never

Chaos or out of sync with reality

Alert Fatigue

What we want

Small , well suited componentsCollect

Transport / Mangle

Store

Analyse

Act / Alert

Visualize

#monitoringlove

Ulf Mansson #devopsdays Rome 2011

A new era of tooling

#monitoringlove hacksessions @inuits

#monitorama

Infra Requirement:

System Checks,

Basic Alerting,

Sensu

Awesome for non static environments

Scaling a clustered RabbitMQ ?

This is Europe, U no do cloud

Automation of #monitoring brought back the #love

Icinga

2009 Fork

I consider Nagios dead

Vibrant Community (or they stalk me)

Throw great parties in Nurnberg

Nobody can pronounce it anyhow

https://github.com/Inuits/puppet-icinga/

Automation

Automation gives , automated maintained monitoring of the services we define.

Monitoring a service vs Monitoring a Service

definition of done:

monitored and in production

A software project is not done untill your last end user is dead

CollectD all the metrics, at high intervals

Infra Requirements:

Metric Collection,TSDB,

Oldschool graphite

Graphite

Graphing at Scale

Graphing at Ease

Any metric is a graph

echo "somestring $somevalue $timestamp" | nc 2003

Graphite++

Dashboards Grafana

Engines : InfluxDB

Cyanite

Prometheus

Grafana

Triggers on Graphs

Export Java Metrics

JMXTrans

Export JMXConfigs

Configure NRPE Check

Export NagiosCheck

Collect JMX Exports on JMXTransNode

Graph EmCollect Icinga Configs on Icinga

Aggregation

Alert on streams

Alert on aggregated metrics

Math 101

f(x)

f'(x)

f''(x)

... statistics 101

But I have log files..

Logs and Metrics

Graylog2

ELSA (Enterprise Log Search and Archive)

ELK Stack

Collect from anywhere

Filter

Send anywhere

Queing

Self Service Metrics

Being able to add new metrics

Build your own dashboards

Look at metrics / logs on all platforms

Learn from the Logfiles

Learn from the platform

Culture, Automation, Measurement,Sharing

Dashboards

Visualize Business Metrics

$revenue

#sales

signups

conversions

Api calls

Application use

Sharing

Open Space

Open Source

Github

Talk about Experiences

Publish the code

Don't get Locked In !

Food

Quiz Time : Which tool did I forget ?

You

It's not about the tools It's about change It's about the people

Contact

Kris Buytaert Kris.Buytaert@inuits.eu

Further Reading@krisbuytaert http://www.krisbuytaert.be/blog/http://www.inuits.eu/

Inuits

Essensesteenweg 312930 BrasschaatBelgium891.514.231

+32 475 961221