Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?
64
What we learned from running 200 tests a year? Martijn Scheijbeler / Director of Marketing / [email protected] / @MartijnSch
Transcript of Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?
What we learned from running 200 tests a year?
Martijn Scheijbeler / Director of Marketing / [email protected] / @MartijnSch
Emerce Conversion - @MartijnSch
Title
Martijn Scheijbeler
Marketing Director at The Next WebFormer Lead SEO & Analytics
[email protected] @MartijnSch
ProcessPROCESS
LearningsLEARNINGS
How it allStarted
@MartijnSch
THE START
Why we feelThe need for testing?
@MartijnSch
“So what would we have
to do to grow x10?”
Boris (CEO)
@MartijnSch
“I have no idea what I’m doing”
INDEX
20,000 140,000 6,000visitors startups listed square meters
7,500,000unique visitors
INDEX.COConference
Why am I showing you this slide?
@MartijnSch
MEDIA News
E-commerce
EVENTS Events
Conferences
WORK SPACES TQ
INTELLIGENCE Index
But how did we get to 200 A/B Tests a Year?
@MartijnSch
Testing Figures Data from 2015
2,5 CRO Team in 2015
38.000.000 Sessions with Tests
0 CRO Team in 2014
5-7 Tests per week
Priceless Amounts of fun
35% Traffic v.s. Tests
@MartijnSch
Quantity v.s. Quality?
@MartijnSch
“If you double the number of experiments you’re going to double your inventiveness”
- Jeff Bezos
@MartijnSch
What makes it different: Publishers v.s. E-commerce?
@MartijnSch
The opportunity is 350-400 tests a
year. As ROI per test is low we
need to run a lot to have a positive
return.
Efficiency <> Costs
@MartijnSch
How is our (marketing) team Structured?
@MartijnSch
SEOCRO/ANALYTICSTNW DEALS
INDEX.COCONTENT
GROWTH
@MartijnSch
Front-end Developers
“Better ask for forgiveness than permission”
- Boris Veldhuijzen van Zanten
@MartijnSch
F*CK it, we’ll do it liveIt’s not rocket science
“Innovate through in-house solutions”
@MartijnSch
We failed, not once but also not twice…
@MartijnSch
SO WE STARTED TESTING WITH GOOGLE ANALYTICS CONTENT EXPERIMENTS
GOOGLE TAG MANAGER IS MAKING OUR LIVES EASIER + SCALABLE
WE DIDN’T COMMUNICATE ENOUGH, WE SHOULD HAVE!
MOVING MOST OF OUR COMMUNICATION TO SLACK
What’s ourProcess
@MartijnSch
PROCESS
We needed to learn, Knowledge building?
@MartijnSch
Thanks for all your support!
We learn from you!
FIND THE PERFECT PROCESS TO FIX EVERYTHING > ALWAYS
WHY? MORE EFFICIENCY: THE MORE TESTS, THE HIGHER THE INVENTIVENESS
What are (y)our
Problems?@MartijnSch
“We can’t create any content”“No management approval”
“Whatever b*llshit reason”
@MartijnSch
“Our legal department is your Editorial team”
- Google Employee
@MartijnSch
How did we overcome our fails?
@MartijnSch
We didn’t move to any ‘real’
CRO tool but decided to
iterate on our own: GTM <> JS.
GTM
@MartijnSch
Download:
Github.com/MartijnSch/CRO
Building our own solutions for
saving testing documentation
+ analysis of tests.
Innovate
@MartijnSch
We moved most of our
communication towards
Slack. Making the teams more
involved in the whole process.
Always communicate
@MartijnSch
We hired two front-end
developers, to make sure in
the long-run we can test
bigger things.
Front-end Dev
@MartijnSch
Improvements
5-7 Tests per week
2-4 Tests per week
30-40 Tests per year
20% More efficient
@MartijnSch
What’s in our
Future?@MartijnSch
CONTINUOUS IMPROVEMENT
WHO’S ABLE TO DEPLOY WINNING VARIANTS WITHIN HOURS?
MOVING TESTING TO OUR BACK-END
COLLECTING DATA + SEGMENTATION MORE EASILY + LESS FLICKERING
GO BIG, BIGGER, BIGGEST!
TESTING SMALL DOESN’T WORK (THAT WELL), GO BIGGER!
TLDR: Process
1. Have the right company vision 2. Have the right team structure 3. Have access to top level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation
@MartijnSch
What testsdid we learn from
@MartijnSch
LEARNINGS
MEDIA News
eCommerce
EVENTS Events
Conferences
WORK SPACES TQ
INTELLIGENCE Index
Sharing
@MartijnSch
Sharing brings in more loyalty,
more engagement and
hopefully more traffic.
Share buttons
• Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users
Winner: Variant
Original:
Variant
Engagement
@MartijnSch
Engagement is important,
focus on getting more clicks
on the related stories in the
sidebar.
Rankings!
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users
Winner: Variant
Original: Variant:
Engagement is important,
focus on getting more clicks
on the related stories below
the article.
Related Stories
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users
Winner: Variant
Original:Variant
So what if we remove the
whole sidebar, wouldn’t people
be more focused on the stuff
that is out there?
Big, bigger
• Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users
Winner: Original
TLDR: Test learnings
1. Know what you’re testing 2. Focus on your user segments 3. Iterate on what you’re already testing 4. Focus on high significance 5. Test big, bigger, biggest!
@MartijnSch
Now you know how to run200 A/B Tests a Year?
@MartijnSch
But did we get to200 tests?
@MartijnSch
“Why didn’t we run 200 tests?”
“What went wrong?”
“Do I need to start crying?”
Boris (CEO)
“Well, if this guy wouldn’t have stuck
his fingers into a lawn mower, YES!”
Missing out on ±20 tests in 2015.
The Lawn Mower
Take Aways:
1. Focus on running X tests a year 2. Improve the process
3. Have the right (free) tools
4. Have the right people
5. Document/Communicate everything 6. Know what the future looks like 7. Have fun!
@MartijnSch
Emerce Conversion - @MartijnSch
Title
See you atConference?
Thanks! Have fun!
@MartijnSch
LET’S HAVE BEER AT THE ‘BORREL’ OR COFFEE IN THE NEAR FUTURE!
Do you have any
Questions?@MartijnSch
