Continuous Delivery
The Path to Rapid Experimentation
Paul Coia
Deploy
Experiment
Example
Lessons
Global platform for independent artists and designers to sell
their work more easily, more oen.
10M designs on 40M products
500,000 artists all over the world.
40M pageviews from 7M Users/month
Deploy
Experiment
Example
Lessons
2000 releases 4 years
700 in 2012
Average 3 per day
cap production deploy
1. Continuous Integration success
2. Deploy code via Capistrano
3. Monitor vital signs
Jenkins Continuous Integration Server
New Relic Application Monitoring
New Relic Key Transaction Monitoring
AirbrakeError Reporting
Circonus and StatsdBusiness Metrics Monitoring
StatsdClient.increment("Payment OK")
if enrolled_in?(“new”).variant? // show variantelse // show normalend
Minimal cost of deploying
Rapid pace of change possible
Confidence make lots of changes
Try ideas quicker
Why we like it
Deploy
Experiment
Example
Lessons
1.Observe
2.Form a hypothesis
3.Devise an experiment to test it
4.Draw conclusions
Scientific Method
Quantitive Research - What?
Qualitative Research - Why?
“If the SEARCH box was bigger, then users will be more likely to search”
Split users at point of variation
Isolate experiments
Persist the decision across visits
Exclude crawlers and ‘old’ users
Enrollment
// ON LANDING PAGEif new_visitor() @guinea_pig = trueend
// ON EXPERIMENT PAGEif @guinea_pig && !retrieve_enrolment() experiment = enrol_in_experiment(“new”) persist_enrolment(experiment)end
if enrolled_in?(“new”).variant? // experiment logicelse // normal logicend
RUNNING_EXPERIMENTS ={ ... new: {description: 'New feature', enrolment_percent: 20}, ...}
Google AnalyticsData Collection
‘R’ Statistical LanguageGoogle Analytics API
Respect Statistics
Establish your test duration up front
Wait for the test to complete
Results
Model your Product funnel
Greater volume earlier in the funnel
Higher conversion rate between steps
Add lots of instrumentation
Useful metrics
Homepage
Sign up page
Signed up
1%
Homepage
Sign up page
Signed up
1%
Homepage
Sign up page
Signed up
20%
5%
‘Landing Page’ Optimizers
Client-side changes only
Simple experiments
Trivial deployment
Feature toggle
Experimentation platform
= Trivial feature experimentation
Deploy
Experiment
Example
Lessons
22% less likely to Add to Cart
50% less likely to Checkout
Mobile Visitors
No change to Checkout Completion
Improvement in Checkout Completion
Increase in Add to Cart rate
Further Increase in Add to Cart rate
Multiple lightweight experiments
A handful of weeks
Verified 30% increase in Purchasing
Deploy
Experiment
Example
Lessons
Invest in your deployment tools
Experiment on the small scale
Expect to be wrong
Build on the successes
Design for incremental change
Top Related