Continuous Integration at SoundCloud

Istanbul Tech Talks 2015

Vincent Garrigues @garriguv

Why invest in a mobile CI pipeline?

• move fast • keep codebase healthy • confidence • ship reliable apps

• move fast • keep codebase healthy • confidence • ship reliable apps

• started from scratch • months in development • millions of users

iOS Crash Complaints (avg per Week)

0

35

70

105

140

April May June July August

2014 SoundCloud community team

iOS Crash Complaints (avg per Week)

0

35

70

105

140

April 2014 July 2014 October 2014 January 2015

2015 SoundCloud community team

How do we work?

• work on master • little to no branches • pairing • feature flags

- (id<AnalyticsProviderInterface>)soundcloudInternalProvider { if ([FlipTheSwitch isDevEventgatewayEnabled]) { return [self eventGatewayProvider]; } else { return [self eventLoggerProvider]; } }

Feature flagging

github.com/michaelengland/fliptheswitch

Feature flagging

Timeline

2012 2013 2014 2015

📱📱📱 📱

📱📱📱📱📱

📱📱📱📱📱📱📱📱📱📱

started very simple

⛔

• unit tests • α and β builds • app store build

Overview of the current

system

buildanalysis

unit tests

acceptance tests

AppStoreAdHocα and β

🚦 🚦

buildanalysis

unit tests

acceptance tests

AppStoreAdHocα and β

~ 5min 30sec ~ 7min ~ 3min

~ 15min

🚦 🚦

Let's have a look at the

details

buildanalysis

unit tests

acceptance tests

AppStoreAdHocα and β

🚦 🚦

build unit tests

acceptance tests build

linter

dependencies

local librariesunit tests

i18n push

build

local librariesunit tests

i18n push

unit tests

acceptance tests build

linter

dependencies

build unit tests

acceptance tests build

linter

dependencies

~2000

local librariesunit tests

i18n push

build unit tests

acceptance tests build

linter

dependencies

~2000

~5000

local librariesunit tests

i18n push

build unit tests

acceptance tests build

linter

dependencies

~2000

~5000

local librariesunit tests

i18n push

build unit tests

acceptance tests build

linter

dependencies

~5000

~2000

local librariesunit tests

i18n push

build unit tests

acceptance tests build

linter

dependencies

~5000

~2000

local librariesunit tests

i18n push

buildanalysis

unit tests

acceptance tests

AppStoreAdHocα and β

🚦 🚦

• frank • cucumber

http://www.testingwithfrank.comhttps://cukes.info

• flexible platform & OS • bandwidth and reachability

settings • launch arguments

stubby4j

✅

⛔datasets

mobile apiproxy

The proxy can change: • response status code • response body • record and undo actions

(like, repost…)

~360 acceptance

tests

~360 acceptance

tests~250 min

Sharding

distributed cucumber

distributed cucumber

x20

VM

buildanalysis

unit tests

acceptance tests

AppStoreAdHocα and β

🚦 🚦

~ 5min 30sec ~ 7min ~ 3min

~ 15min

acceptance tests

• iOS version • iPhone (4S, 5, 6, 6+) • iPad (retina and non retina) • feature flag configurations

What's the value of an acceptance

test?

avoid regressions on

hard to test flows/screens

keeping the build green

pre-ci

https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX

Flaky tests

Flaky tests• test driver is flaky • backend is flaky • app is unpredictable

Flaky tests• identify • isolate • fix

flakyrazor

flakyrazor1. Take failing test out of the test pool

2. Run the test multiple times (flaky or failing?)

3. Assign it to the author/committer

flakyrazor4. Assess test value 5. Act on test duration changes 6. Show statistics on why the test failed

Taking care of our build machines

❤

❤

❤

❤

❤

❤

❤ ❤

• we have images for all the machines

• CLIs to provision the machines remotely

• OS X server stores the images and controls the imaging process

What about Android?

Thank you!Istanbul Tech Talks 2015

Vincent Garrigues @garriguv