1. Measuring Usability LLC Combining Quantitative & Qualitative Data in a Single Large-scale User Research Study Measuring Usability Airline Benchmark Data

2. Jeff Sauro Principal Measuring Usability Speaker Alfonso de la Nuez Co-Founder and Co-CEO UserZoom Moderator Speakers: 3. Measuring Usability LLC Meet UserZoom! All-in-One Software Solution to Cost-effectively Measure the Digital Customer Experience and Conduct Remote User Testing 4. Measuring Usability LLC Increase conversion rates by conducting cost- effective UX research, remote usability testing and voice of the customer studies Test any web-based products (live or prototype) and mobile apps, on a unified software platform. Meet UserZoom! Founded in 2007; launched SaaS platform in 2009 Our mission: To go beyond the Lab! International company: Silicon Valley, Spain, U.K., Germany 200 Enterprise customers, 50% of Fortunes Most Admired Brands 90% renewal rate, 40% yoy revenue growth, profitable business. 5. Measuring Usability LLC Participants take the study simultaneously In-the-wild testing (natural context) Geographic representation No moderation needed Many types of studies can be conducted Output combines Usability + Behavior + Survey Feedback What is Remote Unmoderated User Testing (URUT) Participant UZ Browser Bar 6. Measuring Usability LLC Select Clients using URUT 7. Measuring Usability LLC 1) Cost-effective No moderation needed Automated data analysis 2) Agile & Efficient Build studies and gather feedback in days 3) Qual + Quant Success rates, time on task, behavior, video, audio, heatmaps, verbatims, dendrograms, etc. 4) Easy Recruiting Geographic representation No travel needed 5) Feature Rich More flexible, customizable, robust, versitile than others 6) Team & Service Hands-on, highly experienced Customer Success Team Why is UZ unique? 8. Measuring Usability LLC Study Results 9. Measuring Usability LLC 10. Measuring Usability LLC RESEARCH APPROACH Tentative Hypothesis Observation Induction Deduction Pattern Theory Hypothesis 11. Measuring Usability LLC Qualitative Quantitative 12. Measuring Usability LLC STUDY STRUCTURE Retrospective SurveyN = 1000 N = 130 Asked about their most recent experience Randomly assigned 2 websites, asked to make a reservation SessionVideo and Audio Task Based Study 13. Measuring Usability LLC RESEARCH APPROACH Tentative Hypothesis Observation Induction Deduction Pattern Theory Hypothesis 14. Measuring Usability LLC AIRLINE WEBSITES INCLUDED N = 105 N = 121 N = 111 N = 110 N = 105 N = 102 N = 103 N = 107 DATES N = 108 15. Measuring Usability LLC SUPR-Q & FACTORS Standardized User Experience Percentile Rank Questionnaire Post Experience USABILITY This website is easy to use. It is easy to navigate within the website. LOYALTY How likely are you to recommend this website to a friend or colleague? I will likely visit this website in the future. APPEARANCE I find this website to have a clean and simple presentation. I find the website to be attractive. CREDIBILITY (trust, value & comfort) I feel comfortable purchasing from this website. I feel confident conducting business with this website. NPS PROVIDES STANDARDIZED SCORES RELATIVETO 200 OTHER WEBSITES 15 See Appendix for More Details 16. Measuring Usability LLC OVERALL SCORE BY SITE Industry Average = 83% Error Bars are 90% confidence intervals 91% 83% 65% 82% 90% 84% 87% 88% 79% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Poor Excellent 17. Measuring Usability LLC APPEARANCE BY SITE Industry Average = 75%Error Bars are 90% confidence intervals 82% 82% 35% 87% 78% 73% 83% 83% 77% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% I find this website to have a clean and simple presentation. I find the website to be attractive. Poor Excellent 18. Measuring Usability LLC Qualitative Insights 19. Measuring Usability LLC 20. Measuring Usability LLC 21. Measuring Usability LLC 22. Measuring Usability LLC UNITED APPEARANCE : HOMEPAGE What one thing would you improve on the United.com website? 41% MentionedAppearance UserComments Make the home page simpler, not have so much information condensed in such a small space. Cleaner look.There are so many different colors used on the main page that its distracting to look at & takes longer to figure out which part of the site is relevant. Its very busy on the home page.... lots and lots of tabs and place to click The homepage looks extremely cluttered. It is daunting. 23. Measuring Usability LLC Task Based Insights 24. Measuring Usability LLC STUDY LOGISTICS webcam microphone 25. Measuring Usability LLC 26. Measuring Usability LLC VALIDITY OF FINDINGS Internal Validity External Validity 27. Measuring Usability LLC ENCOURAGE USERSTOTHINK ALOUD 28. Measuring Usability LLC TASK BASED SUMMARY The success rates were essentially the same across all three sites 82% 84% 83% CONFIDENCE + EASE The experience across the three different websites were different, as seen in the confidence and ease scores Southwest was preferred 2:1 over United, further supporting the findings from the quantitative study American had the worst task performance of the three sites tested Flight selection and seat upgrades are overwhelming 29. Measuring Usability LLC LEVEL OF EASE 5.88 5.81 5.28 1 2 3 4 5 6 7 United Southwest American Overall, how difficult or easy was it to complete this task [make a reservation] ? 99% 89% 46% 20% 9% 4% 2% Average level of ease Very Easy Very Difficult American had significantly lower ease. 30. Measuring Usability LLC CONFIDENCE 6.00 6.05 5.43 1 2 3 4 5 6 7 United Southwest American How confident are you that you completed the task successfully [making a reservation] ? 99% 43% 12% 4% 2% 1% .3% Average Confidence Very Confident Not Confident At All 31. Measuring Usability LLC AMERICAN: FLIGHT SELECTION FRICTION This user seems confused when attempting to select a return flight and continues to select upgrades to their departing flight instead before ultimately giving up. 32. Measuring Usability LLC AMERICAN FLIGHT SELECTION 33. Measuring Usability LLC AMERICAN : DIFFICULTY MAKING RESERVATION Sometime actually too much info, I wish it would just give me top 5 flights within my time/price parameters with the option to expand if it doesnt meet my requirements. Although I like the layout, I am not a huge fan of the box in the middle of the page. It is the one feature of the website that makes it a little hard to read. The website layout is sort of confusing. Seat upgrade/upsell is confusing users 34. Measuring Usability LLC UNITED SEGMENT PRICING CONFUSION 35. Measuring Usability LLC UNITED: SEGMENT PRICING Large Study User: Search results for flights can be confusing; I like seeing departing flights from origin city and destination city at the same time. Its unclear what the final price will be because it gives individual price points for each flight, yet these prices aren't actually combined for a final cost. 36. Measuring Usability LLC UNITED: CHANGING RESERVATIONS when you are in the part where you are deciding on dates for a flight - to be able to change the dates without starting the search over would be nice It should be easier to modify search criteria without going back and forth. United had the lowest overall SUPRQ score as well as the lowest Usability Score. Southwest andAmerican have the highest Usability scores, where you are able to change your travel dates from the search results. 37. Measuring Usability LLC UNITED: STARTING SEARCH OVER 38. Measuring Usability LLC INTERACTION SATISFACTION Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles Sample Mean Overall Mean Standard deviation = Z Score = Percentile Rank 5.7 - 5.9 .2 -1.02= = 15% Accuracy Variability 39. Measuring Usability LLC INTERACTIONS: AIRLINES 94% Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles 81%67% 45%48% 26% 22% 5% 97%97% 62%77% 25% 19% 47% 69% 53%72% 19%15% 50% 15% 6% 70% 32%52% 9%13% 2% 0% 3% Green = Positive Interaction Red = Negative Interaction Orange = Neither strong negative or positive interaction 40. Measuring Usability LLC UNITED: CALENDAR The calendar feature on the United website was rated as the hardest to use. The Calendar does not automatically pop up when you click in the date box, therefore making some users manually enter the dates, which leads to error 41. Measuring Usability LLC SOUTHWEST: CALENDAR Uniteds calendar was rated as being significantly harder than Southwests calendar. When a user tabs through the details for their flight, the calendar automatically pops up, eliminating the thought that you would have to enter the dates manually, leading to a successful flight search. 42. Measuring Usability LLC 0% 5% 10% 15% 20% 25% Delta Southwest United American Orbitz Travelocity Kayak Priceline Expedia Booked the Wrong Date RECALLED MISTAKES Average 10% American Airlines users had significantly more mistakes than all other websites (p=.1). 43. Measuring Usability LLC LOYALTY & USABILITY Website usability explains 35% of likelihood to recommend scores Net Promoter, NPS, and Net Promoter Score are trademarks of Satmetrix Systems, Inc., Bain & Company, and Fred Reichheld Usability % Rank: 18% 69% 99% How likely are you to recommend to a colleague or friend? 0 1 2 3 4 5 6 7 8 9 10 Detractors Passives Promoters Not at all Likely Neutral Extremely Likely 44. Measuring Usability LLC KEY DRIVERS OF UX QUALITY Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles 45. Measuring Usability LLC KEY DRIVERS BY SITETYPE Leverage & ExpandDisasters Fix Maintain 46. Measuring Usability LLC KEY DRIVERS: AIRLINES Choosing a Seat 94% Changing Reservation 22% + - Checking In 66% Finding Contact Info 28% Checking In 92% Interacting with Calendar 7% Changing Reservation 0% Checking In 22% 47. Measuring Usability LLC SUMMARY & CONCLUSIONS Weve gone from general to particular and particular to general in understanding the user experience. Videos reinforced and illustrated quantitative patterns Southwest had high scores and United had low scores in both studies. Some evidence for lurking problems with American Airlines. Think aloud audio was essential in understanding some issues Induction Deduction Key drivers of UX differ by airline 48. Measuring Usability LLC @MeasuringU UX Boot Camp Denver CO, Aug 20th-22nd 2014 3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis. Participants receive a User Experience Research Certificate and 2.5 Continuing Education Credits from the University of Denver. DenverUX.com Measuring Usability LLC is a usability and quantitative research firm based in Denver, Colorado USA focusing on the statistical analysis of human behavior and quantifying the user experience for Fortune 1000 companies. www.MeasuringUsability.com 48 MEASURING USABILITY 49. Measuring Usability LLC Appendix 50. Measuring Usability LLC The SUPR-Q is a set of 8 standardized questions that gauge the usability, credibility, user loyalty, and appearance of a website SUPR scores are reported as a percentile ranking out of a database of over 200 websites For example, a score of 50% means that half of all website rank higher and half rank lower The 8 SUPR questions were selected after testing 48 candidate questions on thousands of users and dozens of websites The SUPR-Q usability factor exhibits strong correlation with the System Usability Scale (SUS) SUPR rankings are more useful than SUS scores because they offer a comparison with other websites More information is available online at www.suprq.com UNDERSTANDINGTHE SUPR-Q