Wil Stuckey@quiiver

25 Million Members

25 Million Members

900,000 Shops

`

25 Million Members

900,000 Shops

1.4B Page Views / Month

Managing Experimentation

Managing Experimentation

ManagingLaunches

1. Experiments2. Rampups3. Communications

Life Before Continuous Deployments

Initial Idea &Planning

Geocities-ize

Initial Idea &Planning

Design & Development

Initial Idea &Planning

Deploy &Launch!

Design & Development

Pain?

Testing?

Communication?

How can we do better?

Living in a Continuous

Deployment World

Total Deploys: 4,040Total Commits: 45,447Total Unique Authors: 220

2013 ... So far

Total Deploys: 4,040Total Commits: 45,447Total Unique Authors: 220

Average Deploys per day: 35.75Average Authors per deploy: 7.01Average Commits per deploy: 11.25

2013 ... So far

That’s a lot of people

We are only halfway through the year!

CHAOS

Config System

$server_config['my_awesome_feature'] = [ 'enabled' => 10, 'admin' => 'on', 'groups' => [1234], 'users' => ['wstuckey', 'dmckinley'], ];

Branching in Code

if (Feature::isEnabled('my_awesome_feature')) { // display my_awesome_feature...} else { // continue on}

Typical LaunchCycle

Initial Idea &Planning

Initial Idea &Planning

Design & Development Launch!

Initial Idea &Planning

Design & Development Launch!

Internal Admin Launch

Initial Idea &Planning

Design & Development Launch!

Internal Admin LaunchPublic Prototype

Initial Idea &Planning

Design & Development Launch!

Internal Admin LaunchPublic Prototype50% Experiment

• Small iterative changes

• Small iterative changes

• Product exists in prod as soon as possible

• Small iterative changes

• Product exists in prod as soon as possible

• Can be available for small groups for testing and feedback.

• Small iterative changes

• Product exists in prod as soon as possible

• Can be available for small groups for testing and feedback.

• Easy to communicate milestones because they happen as part of the development process.

Because of the constant change you

have to constantly communicate.

Start Simple

Wiki + Email

• Works great in the beginning

Wiki

• Works great in the beginning

• Quickly becomes overwhelming as you increase your concurrent launches.

Wiki

• Works great in the beginning

• Quickly becomes overwhelming as you increase your concurrent launches.

• Requires manual labor to keep stakeholders informed

Wiki

FRUSTRATION

We can do better!

Frank Harris@hirefrank

Launch Calendar

Launch Calendar

• Simple web app

• Collect structured meta-data about the launch

• Central location for upcoming, current and past launch information

Launch CalendarCollect launch specific meta-data

Launch Calendar

• Owner

• Launch Date

• Internal Launch Date

• Team

• Expected Reaction

Collect launch specific meta-data

Launch Calendar

✉Make your email work for you

✉

Launch Calendar

• Automated emails to product owners around deadlines

• Daily emails to product groups containing upcoming and recently launched launches

Make your email work for you

Launch CalendarMake your email work for you

✉

Launch Calendar

• Highly utilized across the organization

• During the year+ that it was running we recorded and tracked over 275 launches

How did it go?

“This beats the crap out of the wiki page in every conceivable way.” - real

feedback

What could we do better?

What would a more integrated product look

like?

Can we automate more?

We can do better!

CatapultA unified launch management tool.

“I believe that by changing the ‘add to cart’ button to

blue we will increase conversion rate.”

Catapult

$server_config['my_awesome_button'] = [ 'enabled' => 50, 'admin' => 'on',];

Remember this?

Hack, Deploy, Hack, Hack Deploy!

Integration is awesome.

Communication is awesome.

Start Simple.

Build process that enable you to Ship.

Never stop deploying!

Thank You!

Wil Stuckey@quiiver