Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testing summarised - craig...
-
Upload
craig-sullivan -
Category
Technology
-
view
106 -
download
1
description
Transcript of Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testing summarised - craig...
Craig SullivanBelronA year in testing
@OptimiseOrDie
Group CX Manager for Belron® (O’Brien ®)I get to do it with crowds
Naked self promotion
• Contact deflection and online self service
• Site search analytics
• Site, Page and Campaign optimisation
• Test design and execution
• A long usable site portfolio
• ROI on UX improvements
• Over 18 years of boring meetings
• Conversion Rate Optimisation
• A/B and Multi-variate testing
• Cross channel optimisation
• User Centered Design
• Usability testing
• Customer Research & Insight
• Web Analytics
• Browser and email compatibility
@OptimiseOrDie
Outline
• What is A/B and Multi Variate testing?• What is optimisation?• How does it all work?• What tools do I need?• Where do I start testing?• What results should I expect?• Case studies
"If you go to the men's washrooms at the Schiphol airport in Amsterdam, you may notice there's a fly in the urinals. It’s screen printed .
So what do you think most men do? That's right, they aim at the fly when they urinate.
They don't even think about it, and they don't need to read a user's manual; it's just an instinctive reaction that means 85% less spillage!
The interesting feature of these urinals is that they're deliberately designed to take advantage of this inherent human male tendency.“
This is my job.
Testing Primer
@OptimiseOrDie
Who designs it?s it?
Who designs and optimises
this page?
Who designs and optimises
this page?
Web/Online
Marketing
LegalCEO
IT
THE CUSTOMER
Pesky Meddling Kids?
Brand
Primer on testing
=
Control Version A Version B
A/B test
+9% +2%
Multi variate Pizza
=
ExperimentIngredients Recipes
+2% +3% +1% +2%
+9% +4% -2% +1%
+5% +13% -7% -9%
Month 1 - results
+20% -2% +5%
+3% +2% +6%
+6% +10% -18%
So what about ingredients?
+2% +3%
+9% +4%
+5% +13%
Weed out some losers…
+10% +13%
The final…
What’s optimisation?
@OptimiseOrDie
What is optimisation?We’re a customer experience team, but focused on how we can influence customer behaviour in nice ways!
So, our definition of Optimization:
“A set of techniques, implemented in order to influence customer behaviour towards these outcomes:”
•Increased revenue or profit•Removing cost from the business or contact center•Increasing productivity or labor flexibility•Increasing NPS (Customer Sat) scores •Simply delighting customers
All work is challenged on this basis!
Small team, big challenge
• Team of 3 – 2 x optimisers, 1 x part time designer• Higher ROI without spending more (or lower costs)• Stuff that doesn’t hinder SEO or PPC• Measurable ROI on changes, not ego or opinion• Tools with low or zero IT footprint, easy to deploy• Low cost or fast return techniques• Specialised tools for specific optimisation work• Optimisation is a culture, not a tool though• Stop guessing and start testing• No IT department!• MVT and A/B testing has the fastest ROI of *any* technique
How does it all work?
@OptimiseOrDie
Insight - Inputs
Insight
Usability testing
SurveysOps and CS
teams
Clicktale
Social analytics
Customer contact
Eye tracking
Remote testing
Forms analytics Search
analytics Depth interviews
Market research
A/B and MVT testing
Unstructured data
Web analytics
Competitor evalsCustomer
services
Insight - Inputs
ExplodingChannels
PPC
SEO
Direct
Affiliate
Display
TV
Retargeting
Phone RingbackEmail
Chat
Mobile handsets
Apps
Tablets
Desktops
Social & Micro
PartnersDM
Killer idea generation sources
• Analytics data• Customer insight – contact, customer feedback, research• Usability testing• Diary studies• Clicktale recordings• Business owners – market data, customer mix, local knowledge• Customer contact centre immersion (listening)• Books, Websites, Tips• Other optimisers• Test data – variables plotted on axis• Making lots of mistakes – Fail faster
Typical test lifecycle
Instrument Gather and review inputs
Other test results – good variables
& learnings
Brainstorm & Hypotheses
Design variables & creative elements
Test Plan Document
Signoff (Legal, Marketing,
Brand)
Instrument analytics
Wave design Results of wave Washup Next Wave (repeat)
Declare winnerDo verification wave + 90/10%
split
Business led reporting (with = £x, without = £y)
Repeat from start
Hub and Spoke model – homepage trafficWhat’s our testing mix – The traffic picture
What’s our testing mix?
• A/B testing – Homepage, Landing pages, Funnels• MVT testing – high traffic landing or home pages
– Mini (less than 8 variables)– Midi (8-32)– Maxi (>32 variables)
And 5 types of specialised testing:
• Funnel testing – bedding in new funnels• Landing page optimisation• Cross channel testing (e.g. Chat, Contact deflection)• New Feature testing• Telephony MVT
What tools do I need?
@OptimiseOrDie
Low cost of entry tools:
• Visual Website Optimizerwww.visualwebsiteoptimizer.com
• Google Website Optimizer www.google.com/websiteoptimizer
• Optimizelywww.optimizely.com
If you want to move up from here, try Autonomy Optimost or Webtrends Optimize.
Is a testing tool all you need?
• Analytics Talent• Session recording : Clicktale/Tealeaf• Performance optimisation :
Google/Strangeloop• Feedback : 4Q/Kampyle• Usability testing : See resources• Remote testing : See resources• Forms and search analytics : Custom or Bought• Testing tools : Browsercam
Deviceanywhere• Google speed test : Cool stuff!• Survey tools : See resources• Multi-variate and A/B testing : Free, Paid or Both
Where do I start?
@OptimiseOrDie
How do you find your test areas?
• Analytics talent, tools and instrumentation• Develop a leak model• Identify key loss or influence pages• Session recording tools• Usability testing & UCD• Contact centre feedback vital• Traffic * opportunity is the key• Grade targets by impact, time, resource, difficulty• Free evaluation for 2 people – be the first to ask me!
Analytics / Insight – where to test?
• It depends on your site...• High traffic entry pages• High dropout or bail pages• Key value pages (credit card)• High cost pages (PPC)• Conversion tipping points• Persuasion points• Anywhere there is traffic and CTA!• USE A LEAK MODEL!
How a leak model helps
• How many people continue?• How many leave?• How many channel switch?• Tip! Dynamic site numbering!• Where do they go?• Are there influence pages?• Any error clusters?• What are my pain points?
Leak Model Detail
Pipe diagram
Leak surprises!
• Confirmation page loses 8% visitors• One small change reduced this by 75% to 2%• 1 page drives 60% traffic flow to phone• Quote page has low switch – big test opportunity• Bugs in address lookup, branch finder• Forms errors contribute hugely to ‘leavers’• Losses aren’t always big – they’re just switching• In most cases, over 50% of funnel ‘losses’ are switch• Phone tracking!
Session recording
• Vital for optimizers• Fills in a ‘missing link’ for insight• Rich source of data• Segment by attribute/source/behavior• Can be used to optimise in real time!
• Tealeaf www.tealeaf.com• Clicktale www.clicktale.com• Clixpy www.clixpy.com
Usability Testing and UCD
• Lab based testing• Remote testing tools (see resources)• Eye tracking• Prototype testing• Low budget testing• Not what they say, watch what they do• Hurdles to conversion, comprehension, action• Excellent for ‘black box’ forms/funnels• Fills a serious gap in your knowledge
Performance?
• Huge impact on conversion and bailout• Enable Google site speed – get customer KPIs• Read anything by Steve Souders• Read Joshua Bixby (ace!) : bit.ly/8YB3ET• Use Keynote, Gomez for site monitoring• Consider CDN acceleration (Akamai etc.)• Who owns customer facing performance?
Forms and Search Analytics
• Ignore at your peril• Forms are often seen as a ‘technical’ area• What are the common errors?• What errors precede abandonment?• Validation improvements• Form design – Luke Wroblewski• Good example – Postcode Lookup• CR0 1XA CRO lXA• 2.5% loss due to strict validation• Search analytics – Hurol Inan
Browser Compatibility
• My challenge to you!• What browsers have problems?• Who doorsteps you?• The ‘Mac Users’ problem• Measure entry point stats only• Create a browser matrix• Use a testing service, not simulators• Test mobile devices too• Try the ‘Checkout browser’ test
“Anyone who slaps a ‘this page is best viewed with Browser X’ label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network.”Tim Berners-Lee
Where to test?
• High traffic, High impact areas• High ‘bounce index’ landing pages• Key conversion points or creatives• Funnels / Forms• Registration, Signup processes• PPC/SEO high traffic landing• Low hanging fruit first• Micro testing
Micro testing
What about us?
• Easy wins? In priority order:
• Homepage• Core funnel (common to all websites)• High traffic landing pages (hero SEO and PPC terms)• Local landing pages• Influencing pages• Lower traffic landing pages
The challenge
• Use MVT to increase conversion internationally • 35 websites, 19 languages, several bi or tri-lingual websites
• Core sites and pages are similar but there are:
— Cultural differences— Language differences— Different market conditions— Huge variations in traffic size— Very different local marketing approaches (traffic mix)
Homepage testing
• Run tests in key international hubs• Cascade using ‘hub and spoke’ model• Run A/B best hypothesis for smallest traffic
• Maxi (32+ variables), Midi (8-32), Mini (8), A/B (2+)
• UK, DE, FR = Maxi• AUS, IT, ES = Midi• NZ, AT, IE = Mini or A/B
The Results!International Multi-Variate testing
Tools & Techniques:ClicktaleGoogle AnalyticsAutonomy Optimost
UK Wave 2 – 21/9 to 26/10
+12.5%
+8.3%
+13.9%
+5.9%
What is the ROI then?
+11.2%
Another example
+1-2.5%
People images that suck…
Photo Guidelines
• Groups, vans, stores, corporate guff = NO• Single person – not groups or even person + customer• Open, Friendly, Natural, Smiling, Engaged = YES• Uniform – has huge effect, even wearing branded ball cap improves conversion• Look straight at viewer or slightly towards CTA• Plain backgrounds only• No folded arms, arms on hips, pointing, scowling, cheesy smiles, hands behind
back, hanging useless by the side, clasped by groin, fiddling with hands• Arms with prop instead – hold a clipboard, rag, tool – I’m working for *you*• Female images work best in almost every country – they slay the guys• Now we get slightly better images with the guidelines!• We still got a genital framing one last week• Photoshoots are now used to drive advertising strategy• Video tests of potential ‘actors’ as a trial
Key learnings – on page elements
• Less is more – simplicity, comprehension, clarity in design• Copy is king – not content, but key persuasive copy• Large, prominent, visible, button like CTA• CTA above fold, especially on landing page• Do not split attention on CTA (Save, Cancel, Print, Back, Help)• Use goal oriented keywords (Fix my glass!)• Where is the core value proposition?• Maintain a scent – PPC advert, Landing page, Funnel, Reflect the user• CTA wording, headlines, copy, framing = 50%+ of uplift• Think about emotional states and make use of them• Use badging, testimonials, social proofing and trust symbols• Run a betting game in the office – even UX experts little better than 60/40• Guidelines are good but every page and site will vary
Key learnings – test approach
• Stop testing rubbish subtleties (e.g. Button colour)• Cultural factors are minimal• Market and traffic composition are much larger drivers• Techniques work well, regardless of market or language• Huge similarities in testing, but not identical results• Hub and spoke works, but only at a traffic level• Faster cycles of smaller tests = better (fail faster)• A fast test that fails is better than a slow test• You must not test inside a bubble – keep challenging• Use A/B to test radical approaches/disruptive change
Summary
• Don’t overcook tests with too many variables• Don’t test things very similar (blue vs. green button?)• Don’t be timid – fail faster, be bolder• International tests show little cultural differences• Market conditions & traffic mix are more important• Biggest drivers on page = • Copy, CTA and CTA copy, Images, Badging, Nav/Layout• Biggest external factors =• Page and site performance, Browser compatibility• We are now doing up to 7M page tests per month
What’s the ROI like on this work?
• 12.4% from funnel work, 12.5% from MVT• 3.5% increase in downstream conversion rate• 30 seconds reduction in call time (1 sec = $20k p.a.)• Faster booking time (< 8%)• Greater customer delight – feedback. Less questions and worries• 5.5% increase in NPS (Customer sat) score • 4% increase from mobile optimised site• Total increase in conversion (measured) = 32%• Actual increase in conversion = 41%• NO additional marketing spend needed• Web Channel : ROI positive within 2 weeks
Cross channel optimisation
• Call centre optimisation– Phone tracking– Emergency phone switchoff– Telephony MVT– Chat testing– Usability testing of internal apps– Simple no training overspill systems– Contact deflection systems and self servicing– Scripting MVT– Customer reminders – Safelite – Technician profile and ‘on their way’ notice
Chat testing• What happened?• Ran a trial for 1 month• A/B split test – very detailed• One week – 17 calls avoided• But swapped that for 139 chats• As these were 8.25 minutes, much longer than phone• No impact on conversion• Assumption was big impact• Customers happy, yes, but not in a productive way• Like putting your customer service desk into the street• You must measure phone volumes if testing chat• Be very careful – prove your ROI on any pilot.
What’s next
• Segmentation, Segmentation, Segmentation• E.g. - screen size work on Mobile site, iPads• Better testing tools, self administered A/B• More funnel optimisation, increased volumes of page tests• Move from 8 tests -> 20+ tests per month• International MVT – now onto landing pages #1, homepage #3• Telephony and Email MVT• Hero term PPC, SEO + Local landing pages a priority• Dynamic Landing Page generation – human and rules based• The perfect clothes store journey
My advice• You’re crazy if you’re not testing• So easy to do A/B tests• Invest in optimisation tools and talent• Tools may be low cost – expertise is worth paying for• Biggest ROI is high traffic landing page optimisation• Funnel is good but page is quicker (weeks vs 6 mth)• Work on images, copy, call-to-action, layout, hurry up
messages, scarcity calls, badging, value proposition• Stop thinking about testing and doing it instead• Micro test, if you can’t do big stuff• Start a gambling club at work – have fun!
0 1 2 3 4 5 6 7 8 9 10
MVT
SEO
Devicetesting
UserCentered
Design
Analytics
Clicktale
Chat
Email testing
Funnel Optimisation
Targeting systems
Performancetuning
Analyticstalent
Series3(Size of bubble = ROI COST -->
EFFO
RT--
>
Further reading…
Slideshare
: @OptimiseOrDie
: linkd.in/pvrg14
: slidesha.re/nlCDm6
Questions? And some reading...
Resources…
CRO and Testing resources
• 101 Landing page tips : slidesha.re/8OnBRh • 544 Optimisation tips : bit.ly/8mkWOB• 108 Optimisation tips : bit.ly/3Z6GrP• 32 CRO tips : bit.ly/4BZjcW• 57 CRO books : bit.ly/dDjDRJ• CRO article list : bit.ly/nEUgui• Smashing Mag article : bit.ly/8X2fLk
Feedback tools
• Simplest example – 4q – bail from site• Deeper sampling:• Light touch – micro conversion (social, newsletter)• Starts key process – registration/purchase• Gets to tipping point – payment page• Get feedback at different funnel points• Review the user session if captured• Like 4Q but think of an onion
Onsite feedback tools
• Onsite feedback, always available, immediate, contextual
Feedback tools:• Kampyle• Opinionlab• Foresee• Polldaddy
Crowdsourcing:• Uservoice• Ideascale• Google moderator
Kampyle has excellent international support – trial in 2012.
Testing tools
• Browser testing www.browsercam.com
• Mobile devices www.perfectomobile.comwww.deviceanywhere.com
• Performance testing www.keynote.comwww.gomez.com
• New on the block strangeloopnetworks.com
End browser and device discrimination!
Survey tools
• Dust off those survey tools and ask stuff:
• My personal rating:
• Surveymonkey www.surveymonkey.com• Zoomerang www.zoomerang.com• Surveygizmo www.surveygizmo.com
Remote testing, usability, feedback
• Userzoom www.userzoom.com• Usabilla www.usabilla.com• Chalkmark www.chalkmark.com• Userfly www.userfly.com• Silverback www.silverbackapp.com• Usertesting www.usertesting.com• Loop11 www.loop11.com• Pidoco www.pidoco.com• Uservue www.techsmith.com• Ethnio www.ethnio.com• Feedback army www.feedbackarmy.com• Five second test www.fivesecondtest.com• Mechanical turk www.mechanicalturk.com• Open Hallway www.openhallway.com