Post on 21-Mar-2017
A GUIDE TO SPLIT-TESTING
THE NEXT ERA OF SEO
STANFORD STUDY
Frank George
Risk Aversity Performance
Cautious Fantastic. Recommend for awards.
Cautious Terrible. Close to getting fired.
FIREFIGHTER TRAITS
Frank
George
Students were now told:
We made it up. Entirely fictional!
!
How should a firefight treat risk?
They should be cautious!
They should embrace risk. It is a dangerous job!
STUDENTS NOW ASKED:
Frank
George
People cannot let go of old beliefs.
See what they want to see.
!
Have you recommended ALT attributes on images
for SEO reasons?
?
ALT ATTRIBUTES GOOD FOR SEO
TEST RESULT: ALT ATTRIBUTES HAD NO IMPACT
Google algorithm is too complex.
We are too biased.
!
Blog PostBlog Post
HOW YOU CAN TEST — FREE TOOL
‘TRADITIONAL’ CRO A/B TESTING IS USER ORIENTED
Visitors A & B: control version
of a page
Visitors C & D: test version
of the same page
SEO A/B TESTING IS PAGE ORIENTED
Pages 1, 2, 3: Control Layout
Pages 4, 5, 6: Test Layout
All users see the same as each other
SEO A/B TESTING INCLUDES GOOGLEBOT
Googlebot sees the same as users
Pages 1, 2, 3: Control Layout
Pages 4, 5, 6: Test Layout
OVERVIEW
SEO SHOULD BE INFORMED BY DATA & TESTS
1
2009: ALGORITHM IS PRETTY WELL UNDERSTOOD
No Panda No Penguin No RankBrain No Machine Learning
We know what to do.
AMIT SINGHAL - HEAD OF SEARCH, GOOGLE
See this fascinating comment from a user called Kevin Lacker (@lacker):
I was thinking about it like it was a math puzzle and if I just thought
really hard it would all make sense.
Kevin Lacker
Hey why don't you take the square root?
Amit Singhal according to Kevin Lacker
I was like, why a square root, it doesn't make much sense
Kevin Lacker
Multiply by 2 if it helps, add 5, whatever, just make things work and
we can make it make sense later.
Amit Singhal according to Kevin Lacker
THE ALGORITHM WAS MOSTLY HANDMADE
ALGORITHM IS TOO COMPLEX TO VISUALISE
EARLY 2016: AMIT SINGHAL RETIRES
JOHN GIANNANDREA TAKES OVER
PREVIOUSLY HEAD OF MACHINE LEARNING
IMPACT OF MACHINE LEARNING
IMPACT OF MACHINE LEARNING
Translate
10 YEARS RESEARCH
2 MONTHS MACHINE LEARNING
Imagine the same step change in the search algo…
IMPACT OF MACHINE LEARNING
It will be extremely hard to understand many rankings.
Even for Google.
!
No, but I’m still pretty good at this
You’re thinking this to yourself right now.
How good are we?
?
Google Information Needs Study (NBED LINK)
http://adage.com/article/media/future-tv/303508/
SEARCH: future of tv advertising
Google Information Needs Study (NBED LINK)
http://contentmarketinginstitute.com/2016/04/television-advertising-native/
SEARCH: future of tv advertising
DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 ?
http://dis.tl/rankq-1b
86 57 9 ?
SEARCH: future of tv advertising
DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 3
http://dis.tl/rankq-1b
86 57 9 8
SEARCH: future of tv advertising
https://www.jensinkler.com/fix-your-kettlebell-swing/
SEARCH: kettlebell technique
http://www.mensfitness.com/training/pro-tips/perfect-kettlebell-swing
SEARCH: kettlebell technique
DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 ?
http://dis.tl/rankq-2b
76 44 20 ?
SEARCH: kettlebell technique
DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 4
http://dis.tl/rankq-2b
76 44 20 6
SEARCH: kettlebell technique
http://www.macworld.co.uk/how-to/iphone/how-to-back-up-iphone-ipad-backup-3595444/
SEARCH: do I need to backup my iphone
http://www.idownloadblog.com/2013/08/28/not-enough-icloud-storage-iphone-cannot-be-backed-up/
SEARCH: do I need to backup my iphone
DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 ?
http://dis.tl/rankq-3b
67 34 3 ?
SEARCH: do I need to backup my iphone
DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 7
http://dis.tl/rankq-3b
67 34 3 5
SEARCH: do I need to backup my iphone
How do we compare?
?
WE SURVEYED 2000 PEOPLE FOR 10 QUERIES
% OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople
% OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO
% OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO
% OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip
% OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip DeepRank
TESTING - STOP GUESSING WHAT WORKED
Testing the same type of change on different sites can have differing effects.
THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
3) Machine Learning impact on algo will make this worse
THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
3) Machine Learning impact on algo will make this worse
4) Optimise further than previously possible
2
SOME RESULTS FROM OUR SEO TESTS
DistilledODN
Your Server Your Home Page
CDN
Distilled ODN
Your Optimized Home Page
2 IMPORTANT GRAPHS. THIS IS THE FIRST.
BLACK IS THE VARIANT, TREAT BLUE AS THE CONTROL
BLACK (ACTUAL) HIGHER THAN BLUE (CONTROL) - POSITIVE
THIS IS THE START DATE
SECOND GRAPH: TOTAL ADDITIONAL SESSIONS
SHADED AREA SHOWS CONFIDENCE INTERVAL
TEST IS SIGNIFICANT WHEN THE FAN CROSSES 0
IN THIS CASE IT HAPPENED HERE:
HERE THEY ARE TOGETHER.
ALT ATTRIBUTES
1
HYPOTHESIS: ALT attributes on images are good for SEO
RESULT: ALT attributes had no effect
SEO ‘CONTENT’
2
HYPOTHESIS: Text content on category pages is sort of good?
RESULT: No real impact when removed
RESULT: A 3.1% improvement in organic performance
Not every site is the same.
Identical changes can have different results.
!
JAVASCRIPT RENDERING
3
HYPOTHESIS: Javascript impacts Google’s crawling / indexation
MAY 2014 “our indexing system has been rendering a substantial number of web pages more like an average user’s browser with JavaScript turned on.
OCT 2015 “We are generally able to render and understand your web pages like modern browsers.”
Javascript EnabledJavascript Disabled
SITUATION: Javascript needed to render product list
RESULT: CSS fallback led to a 6.2% uplift
TITLES: HIGHEST SEARCH VOLUME
4
Title tag before: Which TV should I buy? - Argos Title tag after: Which TV to buy? - Argos
HYPOTHESIS: Title tags should match greatest search volume term
RESULT: Organic sessions decreased by average 8%
STRUCTURED MARKUP ON CATEGORY PAGES
5
SITUATION: Category pages have lots of images but little text
HYPOTHESIS: Structured Markup per product will help
RESULT: Organic sessions increase by 11%
3
HOW YOU CAN RUN TESTS YOURSELVES
(+ A FREE TOOL)
3 STEPS TO DIY SEO SPLIT-TESTS
1. Create two buckets of pages.
2. Make a change to all pages in one bucket.
3. Analyse which bucket performs better.
1. CREATE TWO BUCKETS OF PAGES.
A good DIY approach is using GA segments. Easy to make (see link). Suggestion: Create Segments by category (e.g. blog tag, product category).
At least 1,000 organic sessions per day to each bucket
!
2. MAKE A CHANGE TO ALL PAGES IN ONE BUCKET.
Control Variant
This will be your variant bucket. Utilise your CMS where possible. Ask Dev for scalable approach.
3. ANALYSE WHICH BUCKET PERFORMS BETTER.
distilled.net/diy-splittester
Typically takes 2 - 3 weeks to reach significance.
!
THEN READ THIS:
Blog Post
4
SUMMARY
SPLIT-TESTING IS BECOMING MORE NECESSARY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip
SPLIT-TESTING IS BECOMING MORE ESTABLISHED
SPLIT-TESTING MAKES YOU MORE CREDIBLE
SPLIT-TESTING IS ALREADY ACCESSIBLE
Step change in your SEO approach.
Start now.
!
WWW.DISTILLEDODN.COM
CHECK OUT OUR SEO A/B TESTING PLATFORM:
@TomAnthonySEO
@Dom_Woodman
@WillCritchlow
DIS.TL/SMXDE17GET THE DECK: