Test and learn strategy chris moffatt
-
Upload
projectmanagementbbp -
Category
Marketing
-
view
387 -
download
0
Transcript of Test and learn strategy chris moffatt
Best Practice is Dead: The Need for a Test and Learn Strategy
Chris Moffatt E-commerce Consultant
Other Media, London
Save me time. Save me money. Make me feel happier. Satisfy my thirst/hunger/desire. Give me more confidence. Make me laugh. Make me feel special. Make me feel intelligent. Solve a problem for me. Educate me. Help me with my research. Answer my question. Reduce my stress. Give me something before my friends have it. Keep my purchase a secret. Surprise me. Shock me. Let me share my experience. I want to compliment you. I want to complain. Reward my loyalty. Show me why you’re better than the competition. Tell me what’s new and hot. Inspire me. Show me something to aspire to. Make me feel famous. I want to feel part of something. I want to feel unique. I want to blend in with my peers. I want to stand out from my peers. I want to help somebody else. Relieve my boredom.
Ask your customers what they want
Some customer effort: Focus groups Surveys Feedback forms
No customer effort: Use analytics data Test and Learn
Few online retailers do it right…
8
6
5
2
NO TESTING HAVE SOFTWARE BUT NO TESTING
TESTING LACKS RIGOUR/VELOCITY
RIGOROUS, HIGH VELOCITY TESTING
…but it can yield great results
“Companies with a culture of optimisation had twice the conversion rate of their peers”
Adobe 2014 Digital Marketing Optimisation Survey
Barriers
• Cost of software • Lack of resource/time • Lack of skills • Change averse • Don’t know what to test • Inconclusive results • Don’t know how to measure success
What should I test?
The whole customer experience, end-to-end
• Acquisition (SEO, PPC, social media, email, affiliates, referrals)
• On-site: page content and design. Key modules e.g. carousel/banner. Search/navigation. Key page templates: Home page, Landing page, Listing page, Product page, Basket, Checkout, beyond Checkout
• Retention: Personalisation. Lifecycle emails and CRM.
People
• Testing Skills and Experience • Creativity • Good listener • Compelling communicator • Negotiation skills • Good presenter • Can explain results to a non-technical audience
Process
• Define KPIs • Identify pain points • Write and prioritise hypotheses • Devise test and success criteria • Manage risk (limit exposure/segment/time/product category) • Run test • Evaluate results in context • Share insight with stakeholders: propose change or new test
Tool
features
cost
Adobe Target
Adobe Target premium
Optimizely
Google Content Experiments
VWO Maxymiser
Autonomy Optimost
Monetate
Unbounce
WebTrends Optimize
Accenture optimisation
Test for free
• Pseudo tests/natural experiments (Example: GM cars) • Test using your Content Management System • Use a free trial of a test tool to justify its cost • Google Content Experiments
Pitfall 1: Starting too small
Small tests take longest to reach statistical significance Test for impact! Consider these first: • Acquisition methods • Home page content • Product page content • Basket and checkout design
Pitfall 2: Low test velocity
87% of online businesses run <5 tests per month… not enough! Some businesses run hundreds of tests per month Likely need at least 1 full time staff member for testing and analysis High ROI is only possible with high velocity testing Segment your audience, manage risk and do multiple concurrent tests
Source: Econsultancy CRO Report, 2013
Pitfall 3: Forgetting stakeholders
Brand Guardians
Offline Marke?ng
PR
E-‐commerce Team
Content Manager E-‐commerce Manager Online Merchandiser
Marke?ng/CRM Manager Op?misa?on Execu?ve
Budget holders
Commercial Director Retail/Online Director Marke?ng Director
Meet weekly to share results, agree changes and discuss next tests
Pitfall 4: Misinterpreting results
• Ensure other variables are controlled
• Don’t forget to segment data
• Test for statistical significance
• Misunderstanding of metrics e.g. conversion
Pitfall 5: Resistance to change
• Need stakeholder buy-in • Work closely with brand guardians • Manage risk • No-blame culture needed • Set expectations
“Risk-averse companies actually risk failure when they don't embrace a vigorous campaign of experiments in the name of superior customer experience.”
Adobe/Econsultancy Digital Trends 2015