Text

What do you know?How Songkick uses continuous delivery and experimentation to learn from our users

About me: Dan Crow@crowquine

CTO, Songkick

!

Visiting Professor, University of Leeds co-Founder Blurb Apple, Google, Unicru, Verb

Songkick• The live music startup

• One of original Silicon Roundabout startups

• Second largest live music service globally

• Growth:

Back in 2012• Simplified product

• Simplified code

• Moved to SOA

• Moved to continuous integration

Results• Releases per month went way up

!

!

!

!

!

• Team happiness too

Text

Continuous Delivery

Aim• Fast feedback from users

• Get code live as fast as possible

• Minimize developer context switching

• Lots of small changes, instead of few large releases

Our principles• Create a continuous flow of work

• Fast feedback trumps exhaustive testing

• Waiting for builds kills focus

• Developers own product quality

• Business metrics tell you what is happening

What we do• Develop on main

• Jenkins CI server

• Automated tests on staging, 10 minute maximum

• Push live on green

• Features accumulate behind feature flippers

• Manual testing live in production

Feature flippers• Group changes together

• Control who can see changes: • No-one • VPN only • percentage of traffic • everyone

• Admin UI controls

Lessons learned• Speed is essential

• Small changes create speed

• Small changes easier to debug

• Immediate feedback leads to: • better product • happier team

It’s about culture, people• CD is cultural change, not a technical one

• Fast and small is (initially) scary

• Roll forwards becomes possible

• Minimize process once everyone is comfortable

• Developers own quality and testing

Text

Experiments

Everything is an experiment• Your startup

• Your products

• Each feature

• Each change

Embrace the scientific method

How we experiment• Answer a question

• Figure out the lowest cost way to answer the question • Example: sharing experiment

• Run as many experiments as possible

• Always throw away the code and design

Throw away the experiments• 80% of experiments fail, so do lots of experiments

• If the experiment fails • remove the changes asap

• If the experiment succeeds • remove the changes asap • rebuild the code

Mechanics• Document the question and the expected results

• A/B testing using feature flippers

• Variety of packages to analyse results: • Google Analytics • skab • MixPanel

It’s about culture, people• Always find the MVE

• Be comfortable with rough quality

• Get used to experiments failing

• Experiment often and fast

• Trust your users, not your instincts

• Trust your tests

Results• Run multiple experiments at once

• Experiments typically last 2-5 days

• Most web features now subject to tests

• Sometimes short-circuit formal A/B testing

Limitations• Your UI changes often

• Can get stuck in local minima

• Everyone needs to understand statistical significance

• Need lots of traffic to get fast results

• Mobile development cycles are too slow

Obligatory hiring appeal

Songkick are hiring: dancrow@songkick.com

@crowquine

!

Silicon Milkroundabout #7, May 10-11