DevOps is a ReorgHow To Accelerate The Change

© Copyright 2015 EMC Corporation. All rights reserved.

DevOps is a ReorgHow To Accelerate The Change

Don Demcsak

East Region – DevOps Process Lead

Twitter - @DonDemcsak


AGILITY – BAKING CHANGE INTO IT

Organization Design

OrgAgility

Business Agility

People Process Tools

ITAgility

© Copyright 2015 EMC Corporation. All rights reserved. 5© Copyright 2015 EMC Corporation. All rights reserved.

STRATEGIES FOR MANAGING CHANGE


• From Silos to Delivery Teams– Outcomes over Activities

• This change isn’t “solved” with a technology change

• This change is evolved through:

People + Process + Technology

Culture!

THE FOCUS OF IT NEEDS TO CHANGE


PEOPLE + PROCESS + TECHNOLOGIES

Aligning ALM and Cloud strategies


UNDERSTANDING CHANGES IN IT

• Fictional story about today’s IT challenges

• Explains how Operational and Cultural changes save the day.

• Based on “The Goal: A Process of Ongoing Improvement” book

• Mark Burgess, a pioneer of system configuration management. Creator of CFEngine.

• What Science tell us about Information Infrastructure video (Video)

https://vimeo.com/album/2573383/video/77207216


BI-MODAL or TRI-MODAL IT?

TRI-MODAL (SIMON WARDLEY)BI-MODAL (GARTNER)

BUSINESS

IT MANAGEMENT

LEGACYAPPLICATIONS

NEW APPS


• Eliminate non-value-added action (Waste/Muda)

• Pull over Push

• Kaizen (Continuous Improvement)

• Kaikaku (Disruptive Change)

• Small Batch + Experimentation

• Kanban (Try www.trello.com )

• Iterative vs. Incremental

ADOPTION OF LEAN PRACTICES

http://en.wikipedia.org/wiki/Lean_startup

http://www.trello.com

http://en.wikipedia.org/wiki/Lean_startup


BUILD PROJECTS ITERATIVE VS. INCREMENTAL

Painting of the Mona Lisa

Woman in Pastoral Settings

1 2 3

Incremental

Iterative


the “WHAT” – defines outcomes; setsenterprises standards to maintain qualityand provide a consistent work environmentacross the enterprise

EMPLOY A TOP DOWN & BOTTOMS UP CHANGE MODEL

SUCCESS REQUIRES A SHARED VISION

CHANGE ADOPTION

LEADERSHIP TEAM

DELIVERY TEAM

the “HOW” – determines the bestapproach for achieving the standardsdefined at the enterprise level


• Reduce cycle time by 300%

• Reduce recovery/rollback time to less than 4hrs

• Improve process efficiency by 10%

• Reduce break/fix workload from 50% to 30%

• Solution is extensible and parameterized

SUCCESS CRITERIA & MEASURESDEFINE THE OUTCOMES OF TRANSFORMATION


Release Candidate Pipelines

DELIVERY PIPELINES

VERSIONCONTROL

COMMITCREATECHANGE

• New software• Enhancements/fixes• Upgrades• Patches• Configurations

TEST UAT STAGE

PROD

Development & Build Process

1.1

1.2

1.3

1.4

1.5

1.2

Automated Tool Chain


START SMALL AND SCALE WITH CONFIDENCE

BUILDING PIPELINES - INCREMENTAL METHODOLOGY

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APPAPP

APP

APPAPP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APPAPP

APP

APP

APP

APPAPP

APP

APPAPP

APP

APP

Single Application(Project Team)

Dependent Applications(Program Teams)

10-100s Applications(Line-of-Business Portfolio)

100-1000s Applications(Enterprise)

PH

AS

E

1

PH

AS

E

2P

HA

SE

4

PH

AS

E

3

GOAL

GOAL

GOAL

GOAL

Goal+ +

+ +


OPTIMIZING CHANGE MANAGEMENTRelease Frequency

1 Release Every 100 Days


Daily Release 10 Releases a Day

100 Releases a Day

InfrastructureProduction-like environments can be prepared

thru version controlled scriptsHeterogeneous, managed

by operations

Production-Standardized PaaS or IaaS provided by delivery team






100 Releases a Day

Infrastructure

App Architecture

Production-like environments can be prepared thru version controlled scripts

Heterogeneous, managed by operations


Everything deployed together

Single packet per product/web property

Strict SOA with forward/backward compatibility






100 Releases a Day

Infrastructure

App Architecture

Testing







Separate QA department that owns functional test automation

Fast, comprehensive automated tests at unit and functional level organized in a deployment pipeline

Developers & Testers maintain automated functional test together






100 Releases a Day

Infrastructure

App Architecture

Testing

Databases










Manual migrations

Delta scripts for db, rollbacks practiced

Apps are forward/backward compatible with dbversions by design (using expand/contract)






100 Releases a Day

Infrastructure

App Architecture

Testing

Databases

Releases

Branch Model










Manual migrations

Delta scripts for db, rollbacks practiced

Apps are forward/backward compatible with dbversions by design (using expand/contract)

Release trains

Dark launching, Blue-green deployment, canary releasing

Developers push their own changes

Develop on branchesMerge to release branches

Then branch again after release

Develop on trunkBranch for Release

Develop on trunkRelease from trunk

Pull Requests toa Release Branch


WHAT DO WE DEFINE AS INFRASTRUCTURE?


• Deploy, monitor and connect together all the pieces needed to run services for the organization

• Processes and tools for faster end-to-end delivery of quality services

• Automation comes built-in

WHY DEFINE INFRASTRUCTURE AS CODE?


• Desired state specified in text files

• Autonomic (self-corrects to desired state)

• State should be known through monitoring

• Remove snowflake servers

HOW DO WE DO THIS?


• Easy to read and edit

• Shareable

• Can use standard version control like Git or SVN

• Becomes executable documentation

WHY STORE THEM IN TEXT FILES?


node 'www2' {

class { 'apache': } # use apache module

apache::vhost { 'awesomewebsite.com': # define vhost

port => '80',

docroot => '/var/www/html'

}

}

SIMPLE EXAMPLE


user { root:

ensure => present,

password => '$ecretP@ssw0rd',

}

CHANGE ROOT PASSWORD


MANUALLY CONFIGURED ENVIRONMENTS ARE LIKE A HOUSE

OF CARDS IN A CHINA SHOP

- NEAL FORD


WHAT CAN HAPPEN IF YOU DON'T TREAT

INFRASTRUCTURE AS CODE?


UP TO 60% OF FAILURES ARE CAUSED BY HUMAN ERROR, NOT

HARDWARE FAILURE


EXAMPLE - KNIGHT CAPITAL GROUP

https://infocus.emc.com/wp-content/uploads/2012/08/KCG.jpg


• Manual deployment of new trading software

• 7 of 8 servers correctly updated

• Old function still alive on the 8th server led to…

KNIGHT CAPITAL GROUP - WHAT HAPPENED?


$440 MILLION LOSSIN 45 MINUTES


SO


TREAT YOUR INFRASTRUCTURE AS

CODE

DevOps is a ReorgHow To Accelerate The Change

Technology

Transcript of DevOps is a ReorgHow To Accelerate The Change