Deploy Early, Deploy Often, Deploy Safely copy · •Run on deploy and as part of monitoring...

Deploy Early, Deploy Often, Deploy Safely Andy Lowe

From User Story to Production Feature

@LoweKeyOne #AgileAZ

Basic Process…

Write Some Code

Commit UAT Deploy Acceptance Prod Deploy

Then There’s All The Other Stuff…

Staging Deploy

Code Review

Security Scans

Change Management

OSS License Review

Performance Testing

Integration Deploy

Integration Testing

Automated Tests

Why Automate Deployments?

• Multiple steps that happen exactly the same way each time• May need to deploy to multiple systems• Automated steps far easier to test

Manual deploys will go badly, your team will work nights and weekends, your users will be upset, and your business will be negatively impacted.


Would you rather fix an incident…

• By manually deploying• Changing configs by hand on

multiple servers• Documenting later what

changed• Without security controls


• Deploying using the same process you always do• With automated rollback• Using a fully tested and

understood process• With your usual controls

Or…

Deploying fixes can look just like deploying any other change.

Always have a widely understood exception process for any controls that may be in place.

How Often Do You Deploy?

Do you want to not hit your SLA because of maintenance windows?


SLAAllowed Yearly

Downtime (minutes)

Max Deploy Duration (minutes)

Once per Month Once per Week Once per Day

99% 5256 438 105.12 30.918

99.9% 525.6 43.8 10.512 3.092

99.99% 52.56 4.38 1.051 0.309

99.999% 5.256 0.438 0.105 0.031

99.9999% 0.526 0.044 0.011 0.003

Blue/Green Deploys

• Zero downtime deployments of a new version of an app.• Rollback strategy is simple.

• Requires the hardware resources to have two versions running simultaneously.


Blue/Green Deploys


Your app is running on on your servers.

MyBlueApp

Router

MyBlueApp.Example.com

MyApp.Example.com

Blue/Green Deploys


Deploy a new version.

MyBlueApp

Router


MyApp.Example.com

MyGreenAppMyGreenApp.Example.com

Blue/Green Deploys


Update main route to point both apps.

MyBlueApp

MyGreenApp

Router


MyGreenApp.Example.com

MyApp.Example.com

Blue/Green Deploys


Remove main route from blue app.

MyGreenApp

RouterMyGreenApp.Example.com

MyApp.Example.com

Blue/Green Deploys


Delete old version and route.

MyGreenApp

RouterMyGreenApp.Example.com

MyApp.Example.com

Blue/Green Deploys

$ cf push MyBlueApp –n MyApp

$ cf push MyGreenApp –n MyGreenApp$ cf map-route MyGreenApp example.com –n MyApp$ cf unmap-route MyBlueApp example.com –n MyApp$ cf delete MyBlueApp


Multi-Availability Zone Deploys – Baseline


MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp


Multi-Availability Zone Deploys – Push Green

MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp

MyGreenApp MyGreenApp



Multi-Availability Zone Deploys – Map New Routes

MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp




Multi-Availability Zone Deploys – Unmap Routes

MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp




Multi-Availability Zone Deploys – Delete Blue

Load Balancer

RouterRouter

Rout

erRo

uter



Health Check That


Aliveness checks• Check if app is responding• Should be very fast• Run on startup and every 30

seconds• TCP, Process, and http checks

Functional Checks• Check if app is functional• Should be fast• Check for connections to

external systems, environment validity, basic functionality

• Run on deploy and as part of monitoring solution

Health Checks During Deploy


MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp



HealthChecker

Health Check Each Green App


MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp



HealthChecker

Health Check Failed - Rollback


MyBlueApp

Load Balancer

MyBlueApp

RouterRouter

Rout

er

MyBlueApp

Rout

er

MyBlueApp

HealthChecker


Health Check Succeeded - Continue

Load Balancer

RouterRouter

Rout

erRo

uter



HealthChecker

Units of Deployment

SomeOtherSystem has to support MyBlueApp and MyGreenApp simultaneously. Options:1. Have immutable API. 2. Have versioned API with support for two versions.3. Combine deployment of new version of SomeOtherSystem with MyGreenApp.


Router

MyGreenApp

MyBlueApp

SomeOtherSystem

Devs Are More Focused Now…


Basic Process…

Write Some Code

Commit Acceptance

All The Other Stuff…

Code Review

Security Scans

Change Management

Third Party License Review

Performance Testing

Integration Testing

Automated Tests

UAT Deploy

Staging Deploy

Integration Deploy

Prod Deploy

Security Scans

• Static scans should happen for each artifact deployed to prod• May happen as part of deploying to lower environments• Make asynchronous, due to time required

• Active scans, taking care with environment in which they run• Set criteria for deploy to succeed• Start with scan results must exist for prod deploy• Add criteria around severity of issues over time• Always have an exception process in place


Security Scans

Security Scans

OSS Scans

Scan for OSS license compliance• Are you using GPL or LGPL?• What about BSD, MIT, WTFPL, APL, others?

Know which libraries are in use• Are you using outdated versions with security issues?• When a zero-day comes out, know which apps to patch




Change Management

Automated deployments mean automatically you know:• Who initiated the deployment• What artifacts and what configuration• When the deploy happens• Health checks on deploy give you deployment

validation.


Change Management

Change Management

Even More Focus


Basic Process…

Write Some Code

Commit Automated Tests

Dev Deploy Acceptance Prod Deploy

All The Other Stuff…

UAT Deploy Code Review

Security Scans

Change Management


Performance Testing

Integration Deploy

Integration Testing

Why Does All This Matter?


• This is real pain felt across the industry.• Freezes are a legitimate business

decision.

• All those freezes leave about 170 days on which you can deploy. Make the most of them.

Case Study: Allstate

Need: Centralized, opinionated deployment tool for deployments

• Deployadactyl – Open source tool for multi-available zone blue/green deploys• Conveyor – Allstate-specific deployment logic

• Metrics• Change management• Security scans• Compliance checks



Dev

UAT

Int

Staging

Prod

Dev

UAT

Int

Staging

Prod• Pipelines include a full automated test suite run prior to publishing to artifact repository and deployment.

• Dev pipelines run on commit to master in source control.• Prod pipelines are manually initiated.

Source Control

CI/CDArtifact

Repository



One month:87 product teams4000+ deployments1000+ production deployments

Easy verification that 32% of prod deploys had security scans.

Security scans on deploy success.

Easy verification that >90% of prod deploys had security scans.

Teams don’t wait for the weekend to deploy.

Deploy Early, Deploy Often, Deploy Safely Andy Lowe

Deploy Early, Deploy Often, Deploy Safely copy · •Run on deploy and as part of monitoring...

Documents

Transcript of Deploy Early, Deploy Often, Deploy Safely copy · •Run on deploy and as part of monitoring...