E X P E R I M E N TAT I O N F O R A N Y C I R C U M S TA N C E S

J O C E LY N G O L D F E I N / @ J G O L D F E I N W W W. J O C E LY N G O L D F E I N . C O M

“It might work for Facebook, but constant

experimentation couldn’t possibly work for my team.”

F O R M F O L L O W S F U N C T I O N

The more you control the deployment environment, the less friction there is to experimentation.

OPERAT ING

SYSTEMSDESKTOP

APPS MOB I LE

APPS WEB

T E C H N O L O G Y D R I V E S C O S T O F E X P E R I M E N T I N G

The more money someone pays you, the more likely your software is “mission critical” for them, and the more entitled they feel to predictability and reliability.

ENTERPR I SE :

$$$$CONSUMER :

$FREE /

AD -SUPPORTED

ENTERPR I SE :

$$$CONSUMER :

$$

B U S I N E S S M O D E L D R I V E S I M PA C T O F E X P E R I M E N T I N G

O T H E R D R I V E R S O F C O S T A N D B U S I N E S S I M PA C T

• Growth vs. Profit

• New customers vs. existing

• Caliber/experience of engineering team

• Partner Ecosystem

• etc…

• …left as exercise for the reader

E X A M P L E S

WEB

E N T E R P R I S E : $ $ $ $

E N T E R P R I S E : $ $ $

C O N S U M E R : $

OPERAT ING

SYSTEMS

DESKTOP APP

MOB I LE

APPS

C O N S U M E R : $ $

F R E E / A D - S U P P O R T E D

E X A M P L E S

WEB

E N T E R P R I S E : $ $ $ $

E N T E R P R I S E : $ $ $

C O N S U M E R : $

DESKTOP APP

MOB I LE

APPS

C O N S U M E R : $ $

F R E E / A D - S U P P O R T E D

OPERAT ING

SYSTEMS

1 X

1 0 X

1 0 X

1 0 0 X

WEB

E N T E R P R I S E : $ $ $ $

F R E E / A D - S U P P O R T E D

OPERAT ING

SYSTEMS

C O S T O F A M I S TA K E

Z O N E 1 : W E B / C O N S U M E R

• Experimentation: Easy

• Impact to users: Low

• Fixing mistakes: fast & easy

• Code bloat: doesn’t matter, your hardware

• Worst case scenario: hotfix

• Best practice:

• Go ahead and A/B test in production

• Rich server-side controls and visualization

N AT I V E C O D E M A K E S I T H A R D E R

• Experiments: more work (code bloat, multi-platform)

• Fixing mistakes: slow (app store approval)

• Worst case scenario: app unusable until users update

• Best practice:

• Server-side code, server-side killswitches - test in production w/ great caution

• Beta / “pre-release access program”

C U S T O M E R S R E LY I N G O N Y O U M A K E S I T H A R D E R

• User impact of experimentation: High

• They need your software to do their jobs

• Worst case scenario: customer abandons your product

• Best practice:

• Key customer input/feedback in design cycle

• Beta program - users expectations do a 180

• Free or cheap consumer version of your enterprise product as experimentation platform

P R O D U C T I O N A / B T E S T E V E R Y T H I N G :

• D E S I G N F E E D B A C K • B U G S • M E T R I C S I M PA C T

D I R E C T C U S T O M E R I N P U T: • D E S I G N F E E D B A C K

B E TA P R O G R A M • B U G S , D E S I G N F E E D B A C K

S E R V E R - S I D E A / B T E S T S : • J U D I C I O U S LY

C L I E N T B E TA : • D E S I G N F E E D B A C K • B U G S

S E R V E R - S I D E A / B T E S T S : • M E T R I C S I M PA C T • B U G S

WEB

E N T E R P R I S E : $ $ $ $

F R E E / A D - S U P P O R T E D

OPERAT ING

SYSTEMS

T Y P I C A L C H O I C E S

D I R E C T C U S T O M E R I N P U T: • D E S I G N F E E D B A C K

B E TA P R O G R A M • B U G S , D E S I G N F E E D B A C K

F R E E P R O D U C T C H A N N E L • B U G S , T E C H N I C A L F E A S I B I L I T Y

C U LT U R E W A R

WEBMOB I LE

APPS

F R E E / A D - S U P P O R T E D

C A S E S T U D Y: FA C E B O O K ’ S T R A N S I T I O N T O M O B I L E

Q U E S T I O N S ?

J O C E LY N G O L D F E I N / @ J G O L D F E I N / W W W. J O C E LY N G O L D F E I N . C O M