Releasing To Production Every Week India

Releasing to Production Every Week

Owen RogersTwitter: @exortech

http://exortech.com/blog



1 year

49 releases

49 releases(excluding patch releases)

~ 1 release / week

summary

lessons learned

process

practices

how long is your release cycle?

Vancouver

> 15 teams

Agile methods

Company

start up

real-time energy monitoring

building energy efficiency

save $

save kW

aggregate data

hosted solution

• 9 developers

• 9 developers• 1 product manager

• 9 developers• 1 product manager• 1 graphic designer

• 9 developers• 1 product manager• 1 graphic designer• 1 tester

what’s missing?

operations?

support?

project manager?

generalizing specialists

fungibility

rotating responsibility

daily support duty

few hand-offs

empowered

Process

Goal:• to surface risk as early

as possible to keep problems out of production

maximize feedback

shrink time

Continuous Integration

Continuous Integration

• 3-4 Commits/dev/day

Automated Build

Automated Build

• 3 minute build

Automated Build

• 3 minute build• 30-40 builds/day

Automated Test Deploy


• 4-5 times per day


• 4-5 times per day• Scheduled nightly at

11PM

Daily Standup

Daily Standup

• Weekly company standup

Daily Review

Daily Review

• Quick peer feature review

Weekly Demo

Weekly Demo

• 15 minute company-wide user-driven demo

Mon Tues Wed Thurs Fri

9:30AMPlanningMeeting

10AMUser-drivenDemo Release

Testing

5pmRelease

weekly cycle

11pmCut-off

Sustainable Pace

Sustainable Pace

• regular release window

Sustainable Pace

• regular release window• quality of life and code

Sustainable Pace

• regular release window• quality of life and code• part of culture

Sustainable Pace

• regular release window• quality of life and code• part of culture• company heartbeat

“agile”

Scrum?

what’s missing?

user stories?

story boards?

estimation?

estimation?fits or it doesn’t

“muda”

pairing?

pairing?as needed

pairing?as needed

code reviews

pairing?as needed

code reviewsopen workspace

TDD?52% coverage

TDD?52% coverage+ Selenium

retrospectives?

retrospectives?Release Review

retrospectives?Release Review

5 Whys

lightweight RCA

5 Whys

just-in-time

within 24 hours

1 Jira per question

Practices

Daily support rotation

5 Whys

Continuous Monitoring


App Server Socket Logger LogMaster

WARN | ERROR

(async)


• Proactive response to issues



• Clean logs



• Clean logs

• Message throttling



• Clean logs


• Gmail



• Clean logs


• Gmail

• Managing the signal-to-noise ratio


App Server

gmetad

web app

RRD

gmondUDP multicast

Zero-downtime Deployment

Zero-downtime Deploys

Load Balancer

App Server App Server App Server

Deploy


• Fully automated deployment



• One button deploy from TeamCity



• One button deploy from TeamCity

• Gracefully bring down servers


What about the DB?


Bering


012_rename_login_id_to_user_name

ALTER TABLE USER RENAME COLUMN login_id user_name


012_rename_login_id_to_user_name

012_add_user_name_column 02_remove_login_id_column

expansion contraction


• expansion


• expansion

• preserves backwards compatibility


• expansion


• contract


• expansion


• contract

• clean up


• Typically safe expansion operations:



• add table, add column, insert data




• Typically unsafe expansion operations:





• rename or remove columns or tables





• rename or remove columns or tables

• add constraint


/expand/.../143001_create_login_table002_add_login_id_index003_create_user_table004_create_group_table

/.../contract/.../143001_add_inv_constraint002_drop_alias_column

/...


• Database migration decoupled from the release



• Simplified rollback



• Simplified rollback

• Some additional complexity in writing migrations

Release Options

Traditional Release

Traditional Release

• high risk

Traditional Release

• high risk

• not scalable

Traditional Release

• high risk

• not scalable

• prolonged release: “Don’t release until it’s all ready”

Release Options

Release Options

• release to user subset (private beta)

Release Options


• split test (experimentation)

Release Options



• incremental rollout

Release Options




• selective enabling/downgrading

Release Options





• disable feature (feature darkmode)

Release Options






• defer commit

Release Options






• defer commit

• defer release

Release Options

Release Options

• Features running in production for weeks before fully available

Release Options


• lots of data

Release Options


• lots of data

• “Release is a marketing term”

Release Options


• lots of data


• Production levers

Release Options


• lots of data


• Production levers

• WANGMI: the discipline to defer

Use Production Data

Use Production Data

• Upload Nightly backups

Use Production Data


• Automated Test DB refresh after every release

Use Production Data



• Local DB refresh on demand

Use Production Data




• Cleansed

Use Production Data




• Cleansed

• Test data is a liability

Single Feature Release

• Daily Support Rotation

• 5 Whys

• Continuous Monitoring

• Zero-downtime deployment

• Release Options

• Use Production Data

• Single Feature Release

Questions?

[email protected]

mailto:[email protected]

mailto:[email protected]

Releasing To Production Every Week India

Technology

Transcript of Releasing To Production Every Week India