What we learned from running 200 tests a year?

Martijn Scheijbeler / Director of Marketing / martijn@thenextweb.com / @MartijnSch

Emerce Conversion - @MartijnSch

Title

Martijn Scheijbeler

Marketing Director at The Next WebFormer Lead SEO & Analytics

martijn@thenextweb.com @MartijnSch

ProcessPROCESS

LearningsLEARNINGS

How it allStarted

@MartijnSch

THE START

Why we feelThe need for testing?

@MartijnSch

“So what would we have

to do to grow x10?”

Boris (CEO)

@MartijnSch

“I have no idea what I’m doing”

INDEX

20,000 140,000 6,000visitors startups listed square meters

7,500,000unique visitors

INDEX.COConference

Why am I showing you this slide?

@MartijnSch

MEDIA News

E-commerce

EVENTS Events

Conferences

WORK SPACES TQ

INTELLIGENCE Index

But how did we get to 200 A/B Tests a Year?

@MartijnSch

Testing Figures Data from 2015

2,5 CRO Team in 2015

38.000.000 Sessions with Tests

0 CRO Team in 2014

5-7 Tests per week

Priceless Amounts of fun

35% Traffic v.s. Tests

@MartijnSch

Quantity v.s. Quality?

@MartijnSch

“If you double the number of experiments you’re going to double your inventiveness”

- Jeff Bezos

@MartijnSch

What makes it different: Publishers v.s. E-commerce?

@MartijnSch

The opportunity is 350-400 tests a

year. As ROI per test is low we

need to run a lot to have a positive

return.

Efficiency <> Costs

@MartijnSch

How is our (marketing) team Structured?

@MartijnSch

SEOCRO/ANALYTICSTNW DEALS

INDEX.COCONTENT

GROWTH

@MartijnSch

Front-end Developers

“Better ask for forgiveness than permission”

- Boris Veldhuijzen van Zanten

@MartijnSch

F*CK it, we’ll do it liveIt’s not rocket science

“Innovate through in-house solutions”

@MartijnSch

We failed, not once but also not twice…

@MartijnSch

SO WE STARTED TESTING WITH GOOGLE ANALYTICS CONTENT EXPERIMENTS

GOOGLE TAG MANAGER IS MAKING OUR LIVES EASIER + SCALABLE

WE DIDN’T COMMUNICATE ENOUGH, WE SHOULD HAVE!

MOVING MOST OF OUR COMMUNICATION TO SLACK

What’s ourProcess

@MartijnSch

PROCESS

We needed to learn, Knowledge building?

@MartijnSch

Thanks for all your support!

We learn from you!

FIND THE PERFECT PROCESS TO FIX EVERYTHING > ALWAYS

WHY? MORE EFFICIENCY: THE MORE TESTS, THE HIGHER THE INVENTIVENESS

What are (y)our

Problems?@MartijnSch

“We can’t create any content”“No management approval”

“Whatever b*llshit reason”

@MartijnSch

“Our legal department is your Editorial team”

- Google Employee

@MartijnSch

How did we overcome our fails?

@MartijnSch

We didn’t move to any ‘real’

CRO tool but decided to

iterate on our own: GTM <> JS.

GTM

@MartijnSch

Download:

Github.com/MartijnSch/CRO

Building our own solutions for

saving testing documentation

+ analysis of tests.

Innovate

@MartijnSch

We moved most of our

communication towards

Slack. Making the teams more

involved in the whole process.

Always communicate

@MartijnSch

We hired two front-end

developers, to make sure in

the long-run we can test

bigger things.

Front-end Dev

@MartijnSch

Improvements

5-7 Tests per week

2-4 Tests per week

30-40 Tests per year

20% More efficient

@MartijnSch

What’s in our

Future?@MartijnSch

CONTINUOUS IMPROVEMENT

WHO’S ABLE TO DEPLOY WINNING VARIANTS WITHIN HOURS?

MOVING TESTING TO OUR BACK-END

COLLECTING DATA + SEGMENTATION MORE EASILY + LESS FLICKERING

GO BIG, BIGGER, BIGGEST!

TESTING SMALL DOESN’T WORK (THAT WELL), GO BIGGER!

TLDR: Process

1. Have the right company vision 2. Have the right team structure 3. Have access to top level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation

@MartijnSch

What testsdid we learn from

@MartijnSch

LEARNINGS

MEDIA News

eCommerce

EVENTS Events

Conferences

WORK SPACES TQ

INTELLIGENCE Index

Sharing

@MartijnSch

Sharing brings in more loyalty,

more engagement and

hopefully more traffic.

Share buttons

• Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users

Winner: Variant

Original:

Variant

Engagement

@MartijnSch

Engagement is important,

focus on getting more clicks

on the related stories in the

sidebar.

Rankings!

• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users

Winner: Variant

Original: Variant:

Engagement is important,

focus on getting more clicks

on the related stories below

the article.

Related Stories

• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users

Winner: Variant

Original:Variant

So what if we remove the

whole sidebar, wouldn’t people

be more focused on the stuff

that is out there?

Big, bigger

• Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users

Winner: Original

TLDR: Test learnings

1. Know what you’re testing 2. Focus on your user segments 3. Iterate on what you’re already testing 4. Focus on high significance 5. Test big, bigger, biggest!

@MartijnSch

Now you know how to run200 A/B Tests a Year?

@MartijnSch

But did we get to200 tests?

@MartijnSch

“Why didn’t we run 200 tests?”

“What went wrong?”

“Do I need to start crying?”

Boris (CEO)

“Well, if this guy wouldn’t have stuck

his fingers into a lawn mower, YES!”

Missing out on ±20 tests in 2015.

The Lawn Mower

Take Aways:

1. Focus on running X tests a year 2. Improve the process

3. Have the right (free) tools

4. Have the right people

5. Document/Communicate everything 6. Know what the future looks like 7. Have fun!

@MartijnSch

Emerce Conversion - @MartijnSch

Title

See you atConference?

Thanks! Have fun!

@MartijnSch

LET’S HAVE BEER AT THE ‘BORREL’ OR COFFEE IN THE NEAR FUTURE!

Do you have any

Questions?@MartijnSch