Continuous Deliveryand Continuous Agile

From Andy Singleton, http://continuousagile.com

www.assembla.com

The secret weapon that Silicon Valley is using todisrupt and destroy competitors

• Retailer X deploys changes to their monolithic online ordering app once every six weeks. Ops holds for three weeks to make sure the complete system is stable.

• Amazon has thousands of services and more than 1000 service teams. They release something about once every 11.6 seconds. In the time that Retailer X takes to try one new release, Amazon has made 100,000 changes.

• Amazon hosting competitor: “It’s an emergency”.• 6 Apex Predators in the top 10 public companies worth

$2.4 trillion: Google, Amazon, Facebook, Microsoft, GE, Apple

Survey on Continuous Delivery

46% think their competitors

have adopted Continuous

Delivery

Problems solved by Continuous

• Stressful releases

• Speed to market• Competitiveness• Quality

• Scaling – building large systems

Not Scrum

• Continuous improvement versus CadenceUnblock your teams

• Web services versus monolithsSmaller, faster testing, faster releases

• Technical practices, not team practicesHUGE productivity gains from automation

• Tech leads versus self-managing teamsOne leader, ready to build services

• MAXOS versus SAFeScale different

改善1. リリースを頻繁にする2. 改善する

/ 月割りのリリース

Kaizen

1. Release more frequently2. Improve

Continuous Agile Agenda

1) DevOps - Continuous delivery of software – Build/Test/Deploy

2) Continuous, metric-driven product management

3) MAXOS enterprise management

4) Making the Move

Continuous Deliveryor “DevOps”

Scrum SprintValidate and

Sort DeliverablesSelect a

Sprint Plan

Completed = observed

velocity

Expand stories and estimate

tasks

CollectIdeas

Close Sprint

Into next sprint

Work on tasks

Scrumban Iteration

Validate and Sort Deliverables

Pull DeliverablesWhen Ready

CollectIdeas

StabilizeRelease

Into next release

Feature Freeze

Triage

Idealized ScrumBan Release

Copyright 2012 Perforce Software, Inc. - Assembla, Inc.

Kanban / ContinuousValidate and

Sort DeliverablesPull Deliverables

When ReadyCollectIdeas

Continuous Release

Stay within WIP limit

Release Features Continuously

Program Launch

Releases

Plan Program Test Doc Deploy

SkipAutomate& Blend Lag Automate

Pull

CI & CD

End up with

Waterfall to Continuous

Measure

Code Contribution PatternsManage code if possible. People are hard to manage and can’t be automated. They want to contribute.• Centralized continuous delivery

– No branches, finds and fixes problems as early as possible• Distributed continuous delivery

– Release every change with its own branch and test• Temporary branches

– Combines benefits of centralized and distributed• MAXOS

– Use centralized continuous integration to manage a massively scalable IT system

Centralized CI/CDContributor Commits – “as early as possible” to find problems

ContinuousIntegration tests

Fail - alarm

Release CandidateTest System Release

QA Testing

Pass

Continuous delivery at Edmunds.comwith PERFORCE

The big question: How to test?

• We release software in batches so that we can test it.

• In continuous delivery, we might get as little at 10 minutes to test a release candidate

Test Layering

Monitor your released software: Errors, Usage volume, usage patterns, user feedback

QA System with Human test consultants

Code review: Both a manual test, and a place to ask for test scripts.

Continuous integration: Run automated tests before using human review time

Unit tests in the development environment

Switch new features and architectureStart h

ere to

add layers

Start here to

release a change

More frequent releases can increase quality

9 (sparse) Layers at Edmunds

Feature Switch and Unveil

HiddenProgrammer sees a change locally. Change is tested in the main version but not seen.

Test Story Owner and testers see the change on test systems.

BetaInsiders see it and use it. Story Owner can show it to selected users for feedback or A/B testing.

UNVEIL! The big event. Communicate with all users. Measure reaction.

One code version

No special test builds

No long-running branches

Role: Developer

• Developers have more power and responsibility.• Developers have more responsibility for testing.• Developers (not QA or PM) decide when to release.

This is a strong finding.• Incentives are correct. Developer might have to

come back from Friday night beers to fix a problem. This provides a motivation to make good decisions and automate testing.

• Features can be released but hidden. Product Managers and Marketers will unveil when they are ready. Unblock!

Programmers approve releases, not QA

Role: IT VendorOld Agreement• Fixed Price• Fixed deliverables• Large probability of late delivery

New Agreement• Fixed price• Fixed Time• Variable deliverables

– Small guaranteed deliverable– Weekly meetings to agree on improvements

Advantages• No late delivery• Deliverable is better

than original specification

• Start sooner and finish sooner

Role: Product Manager/Owner

• Batch -> Continuous

• Requirements -> User Experience

• Strategy -> Measurement– Usage measurements are so important, so

underutilized– Double your productivity

Batch Agile Continuous

Explains the resurgence of vertical integration(eg Apple, Tesla). Requires CI

Making one good change is not a team sport

Product Owner -> Story Owner

What you realize when you begin to be very data-driven is a very large percentage of our ideas are bad.

Microsoft Product CTO Adam Pisoni

Measure everything• Measurements are power. They are impossible to ignore. They

win arguments.

• Double your development capacity for zero extra cost! Users ignore 50% of all new features. If you can find and ignore these features, you double your capacity.

• Measure everything possible about usage. Easy with an online service.

• Collect measurements from installed products and optionally transmit them.

• Use your marketing department. They measure.

Make Product, not Quality

ContinuousRelease

InventoryRelease

Improve Quality

Improve Product Usage

Matrix of Services(MAXOS)

Breaking the scale barrier

The Services MegatrendDesktop Web App Cloud

Services

App

DB

Service

Service

Service

Scale it like Google

• 15,000 developers, 5,000 projects, one current version of the code (2013). They can go from an idea to a release in 48 hours

• Vast Internet system divided into thousands of "services"• Most programming done by teams of 3-4• Centralized process with single version of the test system – run

100 million test cases daily• Before developers release a change, they test it with the most

recent version of all the other services. If a test script finds conflicts, it tells developers who to contact to resolve them

Matrix of Services - MAXOS

PrioritizedBacklog

CurrentWork

Each team releases

when ready

Hundredsof releases

per day

Service team Productionservice

Service team Productionservice

Service team Productionservice

Feedback on speed, errors, usage, and requests

Test as one system

Integrationtest env

Integrationtest env

Integrationtest env

Coordinate without big meetingsContinuous Integration between latest dev version of each service

• Continuous integration helps teams coordinate.

• See dependencies between “producers” and “consumers”

• Errors and conflicts show related team contact info

• Meetings and changes negotiated between two teams, not many

PrioritizedBacklog

CurrentWork

Service team

Service team

Service team

Integrationtest env

Integrationtest env

Integrationtest env

Machines can replace layers of management

Teams are largely self-managing

PrioritizedBacklog

CurrentWork

Service team Integrationtest env

Up to 50% of workfrom backlog

At least 50% of work is self-plannedProblems get fixed quickly

Productionservice

Productionservice

Productionservice

Feedback: quality, reliability, speed, user support

Productionservice

ProductionServer

Sense, respond, self manageminimize planning

Hubspot – Great at Mid-scale• Transformed a monolithic app to 200 services over one year• 3-person programming teams. Each of 20 teams is

responsible for about 10 services• Dev teams responsible for design, programming, testing,

release, monitoring, and responding to production problems. No full-time QA. Shared PM and UX. 4 Ops guys for 2000 servers.

• Lot’s of tooling and dashboards to help teams deploy, manage, and monitor their services

• Feedback from customer support also grouped by team

Team Building

PrioritizedBacklog

CurrentWork

Each team releases

when ready

Service team Productionservice

Service team Productionservice

Add capacity fast by building aroundSingle-function programmer/tech leads

Integrationtest env

Integrationtest env

Teams are not permanently multifunctional

Ways to Scale

Scrum + SAFe• Add more hierarchy• Complex multifunction

teams• Hold big meetings and

teleconferences• Block everyone into

one cadence• Coordinate big releases

Top Tech CompaniesAutomate management,

as well as testing and deployment.

Dev-lead teamsCommunicate peer to

peerUnblock teams to move as

fast a possibleRelease more frequently

SAFe versus MAXOS

How to move to CD and Services

• Measure• Release more frequently• Assign blame and other organizational and

authority hacks• Automate• Organize the fast layer• Reverse Conway’s Law• Manage products, not programs

Measure

People respond directly to measurements. They are very difficult to ignore. They are also MORE IMPORTANT than internal process discussions.• Usage and customer value• Errors• Fix times• Release frequency

Release more frequently

1. Release more frequently2. Improve

Authority Hacks

Training, persuasion, browbeating, and “culture change” will not work. They don’t change the game. You first change the game. Then, people figure out how to play the game and win.• Need automated tests? Mandatory code review• Need higher quality and shorter test cycles? Force

programmers to push the “release button”, not QA• Stop your boss from launching big waterfall

projects? Give him monthly budgeting, not annual.

Code Review gets you tests

AutomateIf the machine works, people will use it

Core ITannual budget

Reliability & security mission

WebAPI Layer

Mobile SaaS / Cloud

Marketing

Fast (Awesome Mode) ITmonthly budget

Mission to respond to opportunities

Service team Productionservice

Service team Productionservice

Productionservice

Integrationtest env

Integrationtest env

Integrationtest env

Fast IT

Core IT (stable service)

Reverse Conway’s Law

Boss

Dept A Dept B

PartA

PartB

Service A1

Service A2

Service B1

Service B2

Team A1

Team A2 Team B2

Team B1

Manage Products, not Programs

• Strong product managers deliver value to users as directly as possible

• The Matrix of Service teams respond to product managers. The front end teams provide the product. They act as “consumers” to pull what they need from back-end teams

• Service teams support multiple products, dissolving the program hierarchy

Product Matrix

Boss

Dept A Dept B

PartA

PartB

Service A1

Service A2

Service B1

Service B2

Team A1

Team A2 Team B2

Team B1

People first, or Technology first?

• Your comments