CULTURE OF TESTING

MY STORY OF SELF

• Not just a set of rules or best practices

• A commitment to:– Measure the effectiveness of your choices– Make decisions using data, not intuition

• So… how do we do that?

WHAT IS A CULTURE OF TESTING?

• Metric: numerical measure

of performance

• Email metrics:– Open rate– Click rate– Action rate– Unsubscribe rate

• Web page metrics:– Conversion rate (online actions per visitor)

METRICS, METRICS, METRICS

• Q: You have two ideas for an email – which one do you choose?

• A: Both!

• Steps to run an email A/B test:– Select two randomized email groups from your list– Send email A to the first group and email B to the second– Look at email metrics to determine which email did better– Send better email to the remainder of your list

METRICS IN ACTION:EMAIL A/B TESTS

• Example test:– Email A subject line about

puppies, email B subject line about cats

– List of 100,000 emails; select 10,000 randomly for A, 10,000 randomly for B

– Send emails, measure results:• 1,276 open A and 1,403 open B• 267 donate to A, 195 donate to B

– Which do you choose?

METRICS IN ACTION:EMAIL A/B TESTS

REAL EMAIL A/B TEST EXAMPLE

71% more in contributions from Version 1

• Higher test performance doesn’t always mean better – needs to be statistically significant

• Make sure test groups large enough for significance

• CALCULATOR DEMO

STATISTICAL SIGNIFICANCE

• Emails aren’t free – lose people who unsubscribe

• Need to balance benefit of actions against cost of unsubscribes

• “Trial balloon” test:– Send email to small, randomized group– Evaluate metrics and decide if benefits outweigh costs– If yes, send to remainder; if no, don’t send

TO SEND OR NOT TO SEND:UNSUBSCRIBES & TRIAL

BALLOONS

• Like email, possible to A/B test web pages

• Web page A/B test:– Create several variations of web page– For new users coming to page, randomly show one

variation– Record if visitor takes action– Calculate conversion rate for each variation – highest

conversation rate is best

• Need enough time for statistical significance

WEB PAGE A/B TESTING

29% increasedconversion rate

WEB PAGE A/B TESTING

• Until recently, not possible to do A/B testing on social media sharing

• With ShareProgress, can now test Facebook and Twitter share posts:– Create several variations of Facebook post (title,

description, thumbnail) or Twitter text– For users sharing from your website, randomly give them

one variation– Record if users’ friends click share link and take action– Calculate recruitment rate for each variation – share post

variation most likely to drive action is best

SOCIAL A/B TESTING

SOCIAL A/B TESTING

Facebook TitleShares Driving Action / Total

Shares

Success Rate

Improvement

Monsanto shouldn’t be above the law 341/1041

32.8% (±2.9%) --

“One of the most outrageous special interest provisions in years”

322/96233.5%

(±3.0%) 2.2%

Tell the Senate: Repeal the Monsanto Protection Act

458/97247.1%

(±3.1%) 43.8%

Sign the Petition: Repeal the Monsanto Protection Act

555/1037 53.5% (±3.0%) 63.4%

Facebook headline test by CREDO Action – 63% increased recruitment rate from Version 4

• Emails and web pages

• Social sharing

• Online ads

• Even offline actions

CULTURE OF TESTING INEVERYTHING YOU DO

MAINTAINING ACULTURE OF TESTING

• Get buy-in across the organization

• Run at least one test for every campaign you do

• Schedule periodic check-ins on what you've learned and accomplished

• Email A/B testing:– Check your email

management system

• Webpage A/B testing:– Google Website Optimizer– Optimizely

• Social A/B testing:– ShareProgress

• A/B Test Calculator

HELPFUL TOOLS

QUESTIONS?