Debunking Ad Testing
-
Upload
hanapin-marketing -
Category
Marketing
-
view
62 -
download
3
Transcript of Debunking Ad Testing
Debunking Ad Testing
Mart in RöttgerdingH e a d o f S E M Logo Here
Learner Outcomes
• Adtesting best practices don‘t work
• What actually happens when you followbest practices
• What you should be doing instead
MartinRöttgerding
• Headof SEM@BloofusionGermany
• Judge @SEMY(GermanSearchAwards)
• Dad@Home
Who’s Talking?
@bloomarty
First off: No Worries
• I’mnotamathematician.
• Thisisnotamathlecture.
• Ibrought some data,though.
iStock.com/digitalgenetics
In the Beginning: Optimize for Clicks
% Served Impressions Clicks CTR
Ad1 96% 12,323 594 4.82%
Ad2 4% 536 37 6.90%
Alternative Approaches
• Gutfeeling
• Rules(„wait 100clicks,then decide“)
• StatisticalSignificance (usually:95%)
Statistical Significance in a NutshellImpressions Clicks
Ad1 200 10
Ad2 200 20
Statistical Significance in a Nutshell
0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%
CTR
Ad1
Impressions Clicks
Ad1 200 10
Ad2 200 20
Ad2
Statistical Significance in a Nutshell
0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%
CTR
Ad1
Impressions Clicks
Ad1 200 10
Ad2 200 20
Ad2
Statistical Significance in a Nutshell
0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%
CTR
Ad1
Impressions Clicks
Ad1 200 10
Ad2 200 20
Ad2
Statistical Significance in a Nutshell
0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%
CTR
Impressions Clicks
Ad1 2000 100
Ad2 2000 200
Ad1 Ad2
Pros and Cons
• You have the power.
• You know what to do.
• You look good.
• It doesn‘t work.
Waiting for SignificanceProblem #1
Is it significant yet?
no
no
no
no
no
no
no no
no yes
„We are 95%certain that this adis better than the other one.“
An Experiment
• 576adpairs
• Rotated evenly
• Left untouched for 12months
Analyzing the Data
• Scriptto evaluate the data
• Calculate level of significance for each day
• Visualization:– Ad1reaches statistical significance (95%)
– Ad2reaches statistical significance (95%)
The Result (a small part)
statistically significantno longer significant
stillsignificant…
waiting for significance
The Result (zoomed out)
The Result (zoomed out)
Results
• Mosttests reached asignificance level of 95%atsome point
MinimumtotalImpressions Teststo reach significance Stillsignificant inthe end
1,000 55% 13%
10,000 62% 12%
100,000 81% 11%
Not looking at all the piecesProblem #2
So this is what we see…Impressions Clicks CTR
Ad1 2,000 200 10%
Ad2 3,000 240 8%
How we tend to think of CTR
CTRYour Ad
What drives CTR?
CTR
Ad
Network(Googlevs.SearchPartners)
Search Partners?
Search Partners
Segmented by NetworkImpressions Clicks CTR
Ad1 2,000 200 10%Google
SearchPartners
Ad2 3,000 240 8%Google
SearchPartners
Segmented by NetworkImpressions Clicks CTR
Ad1 2,000 200 10%Google 1,000 180 18%
SearchPartners 1,000 20 2%
Ad2 3,000 240 8%Google 1,000 220 22%
SearchPartners 2,000 20 1%
Also possible…Impressions Clicks CTR
Ad1 2,000 200 10%Google 1,000 180 18%
SearchPartners 1,000 20 2%
Ad2 3,000 240 8%Google 1,000 220 22%
SearchPartners 2,000 20 1%
Also possible…Impressions Clicks CTR
Ad1 2,000 200 10%Google 1,000 180 18%
SearchPartners 1,000 20 2%
Ad2 3,000 270 9%Google 1,000 220 22%
SearchPartners 2,000 50 2.5%
How common is this?
Based onastudy of 6,500adpairs,compared with anAdWordsScript
• OverallwinnerlosesonGoogle • OverallwinnerlosesonGoogle&SearchPartners
32.74%
12.23%
Quick Win: Ignore Search Partners
• Well…
What drives CTR?
CTR
Ad
Network(Googlevs.SearchPartners)
Slot(topvs.other)
Same thing with slots
Based onastudy of 6,500adpairs,compared with anAdWordsScript
• Overallwinnerlosesinthetopslot • Overallwinnerlosesontop&other
18.46%
6.30%
What drives CTR?
CTR
Ad
Device
AdpositionNetwork(Googlevs.SearchPartners)
Slot(topvs.other)
CTR by avg. Position (Google top)
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0
Avg.Pos.CTR
Exact Positions‘ CTR‘s (Google top)
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0
Avg.Pos.CTR ExactPos.CTR
18.05%
13.64%12.05%
10.09%
Interpolated CTR‘s
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0
Avg.Pos.CTR ExactPos.CTRinterpolated ExactPos.CTR
18.05%
13.64%12.05%
10.09%
What happens at the impression level…
• Thisis justaglimpse into impression level data
• How would you assess the following scenarios?
Scenario 1
Your ad
Userclicks onthis ad…and never scrolls down
Scenario 2
Your ad
user clicks onthis ad…after0.2seconds
Scenario 3
Your ad
user clicks onyour ad…and onallother ads as well
Scenario 4
user clicks onyour ad…afterspending 35secondsreading allsearch results
Your ad
How would you evaluate these?
• 4impressions
• 2clicks
• CTR:50%
Ignoring FeedbackProblem #3
What drives CTR?
CTR
Ad
Slot(topvs.other) Device
AdpositionNetwork(Googlevs.SearchPartners)
AdAuction
Adranking
CTR
The position feedback
• Positivefeedback
• No loop:position effects donotaffect QS
Adposition
AdAuction
AdrankingHigherCTRHigherQualityScore
BetterPositionHigherCTR
The impressions feedback
The impressions feedback
• Negativefeedback:
• No loop:low relevance impressionsevaluated separately
HigherCTRHigherQualityScore
Morelessrelevant
impressionsLower CTR
CTR
Ad AdAuction
CTR Feedback
CTR
Ad
Slot(topvs.other) Device
AdpositionNetwork(Googlevs.SearchPartners)
AdAuction
Adranking
mostly invisible
Thinking you are better motivatedthan Google
Problem #4
Googledoesn‘tunderstand my
business Googledoesn‘t care
Adtesting isn‘timportant enough forGoogleto get it right
The AdWords business model
„How much would you give us if we gave you the click?“
„How much would you give us if we showed your ad?“
Selladclicks
Selladimpressions
Convert bids
No control over clicks
Advertisers want clicks
The ad auction
• 𝐴𝑑𝑅𝑎𝑛𝑘 = 𝐶𝑃𝐶 ∗ 𝑄𝑢𝑎𝑙𝑖𝑡𝑦𝑆𝑐𝑜𝑟𝑒
• = 𝐶𝑃𝐶 ∗ 𝐶𝑇𝑅
• = 89:;8<=>?:
∗ 8<=>?:@ABCD::=9E
• = 89:;@ABCD::=9E =„How much would you give us if we showed your ad?“
Getting CTR wrong…
The concept itselfProblem #5
Conflicting Mindsets
Have adedicated adfor everything.
Findthe best adand use only that.
[keywords]
? ?
??
?
??
?
Example: Location Context
• Arethey outor athome?
• Arethey moving or standing still?
• Arethey atafamiliar place?
Example: Search History
• Have they searched for this before?
• Did they interact with ads?
• Did they interact with organic results?
• Have they seen our adbefore?
Example: Personality
• Dothey take their timeto read the entire ad?
• How dothey respond to– discounts– reassurances– testimonials
Testing for Things We Can‘t See?
• Manually:Impossible
• Otherwise:
What should we do?So…
New Mindset
• You don‘t have control over adtesting.Let it go.
• There can be multiplewinners.
• Use Google‘s optimized adrotation by default.
Let the Machines Do Their Job
• Googleis well motivated
• Googleis really good with data and algorithms[citation needed]
• Let Googledecide which ads to show
Provide Meaningful Ad Variations
• Createads for client personas
• TryoutdifferentUSP‘s
• Bigstuff first
Know Google‘s Limits
• CTR&conversion rate
• Historicaldata
• Semantics
Keep an Eye on the Machines
• If necessary,force data collection
• Rotate atadgroup level
• Consider the cost of even rotation
• Alternative:Addthe adagain
iStock.com/RichVintage
To Sum Up…
• Nomoremicromanagingadtests
• Focusonmessagingandsupervisingthemachines
Thank You!
• AgencyBlog:DieInternetkapitäne
• Advanced AdWordsBlog:PPC-Epiphany.com
@bloomarty
Martin RöttgerdingHead of SEABloofusion Germany [email protected]