MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B...
Transcript of MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B...
![Page 1: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/1.jpg)
MASTERING THE ART OF EXPERIMENTATION
Chris Gibbins - Director of UX & Optimisation
Biglight.co.uk | @cjgibbins | @biglightbuzz
1
![Page 2: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/2.jpg)
![Page 3: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/3.jpg)
The future belongs to brands that provide remarkable customer experiences.
We exist to help them do just this.
![Page 4: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/4.jpg)
SOME OF OUR CUSTOMER EXPERIENCE OPTIMISATION CLIENTS
![Page 5: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/5.jpg)
WORKSHOP AGENDA
15:40
15:50
16:10
16:20
16:55
17:20
18:00
START
INTRODUCTION TO EXPERIMENTATION
THE EXERCISE
- CUSTOMER PROBLEM 1
- CUSTOMER PROBLEM 2
- CUSTOMER PROBLEM 3 (If we have time)
END
![Page 6: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/6.jpg)
What about you?
![Page 7: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/7.jpg)
What comes to mind when you hear the word
“TESTING” “EXPERIMENTATION”
![Page 8: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/8.jpg)
INTRODUCTION TO EXPERIMENTATION
![Page 9: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/9.jpg)
CREATIVE UX
INSIGHT
EXPERIMENTATION
1 Learn what really works for users
4 Iterate as you can’t get everything right in your first attempt
5 Measure the ROI of all our work
2 Be more adventurous in our ideas / innovate
3 Fail fast6 Become better UXers and Designers
EXPERIMENTATION IS KEY TO CONTINUAL IMPROVEMENT OF THE UX
![Page 10: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/10.jpg)
“The most important single thing is to focus obsessively on the customer. Our goal is to be earth’s most customer-centric company.”
Jeff Bezos, CEO Amazon
![Page 11: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/11.jpg)
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day,”
Jeff Bezos, CEO Amazon
![Page 12: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/12.jpg)
OPTIMISATION / PERSONALISATION TOOLS
AB
C
Off-the-shelf solutions e.g. Develop your own e.g.
![Page 13: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/13.jpg)
LIVE EXAMPLE
Control The original website
Variation 1UI enhancements to filters
Variation 2Horizontal filter layout
![Page 14: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/14.jpg)
THE LEAN UX PROCESS - JEFF GOTHELF
![Page 15: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/15.jpg)
A/B Testing live on the
website
Roll-out winning experiences with a much greater degree of confidence
+Calculate the uplift and ROI
Design & Build the A/B test variations
Several concepts at
a time
OUR PROCESSES ARE NOT THAT DIFFERENT
![Page 16: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/16.jpg)
Last 4 months of UX and Optimisation activities for a client with 3 websites
35 A/B tests completed
Each one had on average of 3 design variations plus the control
12 significant winners (33% win rate)
TYPICAL PROGRAMME SHOWING HOW WE ITERATIVELY IMPROVE THE CUSTOMER EXPERIENCE
![Page 17: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/17.jpg)
The Experimentation Exercise
![Page 18: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/18.jpg)
Explore and investigate the customer problem making sure that you are clear on what the issues are.
Start sketching out ideas to solve the customer problem. Try to come up with at least 8 in total
Refine down to 4 ideas / variations which could be tested as an experiment. Be extra brave with the 4th ‘wildcard’ one!
1) Customer problem 2) Crazy Eights 3) Experiment design
10 mins
15 mins 10 mins
THE EXERCISE - 3 CUSTOMER PROBLEMS TO SOLVE
4) Present back
A couple of you to talk through the ideas
10 mins
![Page 19: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/19.jpg)
IDENTIFYING CUSTOMER PROBLEMS AND OPPORTUNITIES
![Page 20: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/20.jpg)
IDENTIFYING CUSTOMER PROBLEMS AND OPPORTUNITIES
![Page 21: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/21.jpg)
Long product listing pages and too much choice make it hard for users to find their ideal product.
The customer problem in detail:
- Website example: COS www.cosstores.com
- Usability testing has shown time and again that users struggle when there is too much choice on very long product listing page
- User Interviews have revealed a need for more direction and also more inspiration to help users choose especially when there are a large number of products in a category
- In store research has revealed how staff members guide users to a smaller range of products that better fit the users’ needs.
The challenge
- How do we improve the overall experience on the product listings pages to help users more easily find their ideal products?
CUSTOMER PROBLEM 1.
90x Men’s Knitwear items
![Page 22: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/22.jpg)
Takeaways- If you UXer or Product Designer who wants to make a measurable
difference, you really need to embrace online experimentation (e.g.
A/B testing)
- Push yourself and go beyond the obvious solution… be more
adventurous and you might just stumble across a solution that 10x
better for users.
- Be a little cautious of the phrase ‘best practice’. Often it’s used by
people wanting to push through their solution!
![Page 23: MASTERING THE ART OF EXPERIMENTATION - Tech Circus€¦ · THE LEAN UX PROCESS - JEFF GOTHELF. A/B Testing live on the website Roll-out winning experiences with a much greater degree](https://reader036.fdocuments.net/reader036/viewer/2022063008/5fbc30dd7653e03e261e98ff/html5/thumbnails/23.jpg)
THANK YOU
Biglight.co.uk | @cjgibbins | @biglightbuzz
23