Delivering Results: How Do You Report User Research Findings?

New Hampshire Usability Professional’s Association

January 2012 Meeting

Delivering Results: How Do You Report User Research Findings?

Jen McGinnEva KaniastyDharmesh MistryKyle SoucyCarolyn Snyder

www.nhupa.org

http://www.nhupa.org/


Introduction

About NH UPA We want to make New Hampshire a good place for user experience professionals to work and live

Join Us: www.nhupa.org or @nhupa or NH UPA on Facebook

Help Us: Volunteer or give us suggestions

Announcements• NH UPA Board of Directors:

Debra Arneson, Megan Fields, Rick La Vache, Bob Thomas

• Thanks to Mad*Pow

• Jobs

www.nhupa.org




Upcoming NH UPA Meetings

Tentative Date: Wednesday, February 15Tom Clancy, “Using Development Frameworks to Rapidly Prototype Applications”

• Have an idea for an upcoming meeting?• Want to practice an upcoming presentation or panel you’re

giving?• Want to give a 10-minute talk?

Contact Us: [email protected] or [email protected]

www.nhupa.org

mailto:[email protected]

mailto:[email protected]



Delivering Results: How Do You Report User Research Findings?

The long, textual written report is dead, isn’t it? So how do you deliver your findings to your clients? Is it PowerPoint? An e-mail? A spreadsheet? Post-it notes? And what do you include? Positive findings? Screenshots with callouts? Just issues? Or recommendations as well? Are they prioritized?


Panelists

If you ask our panelists, some of us have developed templates that we use and modify for each research activity, and others change the deliverable based on the activity and client. Each panelist will spend 5 minutes showing you their typical deliverables, and then we’ll open the floor for audience Q&A.

• Jen McGinn, Principal Usability Engineer, Oracle• Eva Kaniasty, Founding Principal, RedPill UX• Dharmesh Mistry, Usability Specialist, Acquia• Kyle Soucy, Founding Principal, Usable Interface• Carolyn Snyder, Founding Principal, Snyder Consulting


Jen McGinnPrincipal Usability Engineer, Oracle


Overview

• I present my research results via slides in 60-minute meeting (generally remote, via web conference)

• I’m going to spend 3-5 minutes walking you through the structure of one of my PowerPoint presentations

• Then I’ll summarize the take-aways


[Product ][Method] My name, title, and date

04/08/2023


Executive Summary• In [When?], the [what product?] was tested by

[number and type of participants] in [method type] to evaluate the ease of use of several features including [features or use cases].

• High level findings included [usually a total of 3 to 4 bullets]:• [ 1 - 2 biggest positive findings]• [ 1 - 2 biggest positive findings]• [ 2 or 3 biggest usability issues]• [ 2 or 3 biggest usability issues]

• This presentation covers all of the findings and subsequent recommendations. 04/08/2023


Agenda

• Goals • Tasks• Participants• Findings• Recommendations• Next Steps

04/08/2023


GoalsEvaluate the usability of the following features of the

U-Haul.com website:

– Are users confused about how to price a rental? A storage unit?– How do users react to the insurance options? Do they

understand the coverage?– How do users feel about the presentation of items for purchase

or for rent? – How effective is the shopping cart content? Are users confused

by when they need to pay for items?– Do users value the star ratings? U-Haul brand?– How do users feel about the targeted FAQ and search result

pages?– Does our online documentation help prevent calls to the service

center? Can they determine how to reach out to the U-Haul vendor nearest them?04/08/2023


Tasks1. Get the price of a 1-way move across country

2. Find a specific piece of information in the FAQ

3. Determine the size and cost of a storage unit needed to hold specific items

4. Find the phone number of a U-Haul location

5. Book the truck (and insurance), adding rental items and purchased items

6. Determine insurance coverage

7. Find the U-Haul location nearest you

04/08/2023


Participants

04/08/2023

ParticipantID

Gender Age Occupation Web-savvy

U1 Male 24 Missionary Average

U2 Male 52 Small business manager Average

U3 Female 62 62 Retired. Formerly television news producer, then licensed paralegal.

Average

U4 Female 36 Housewife Average

U5 Male 31 Sales and marketing Average


Findings

04/08/2023


Choosing a Truck

5/25/2011 Jen McGinn

Another issue

One participant suggested this fix

2 participants had this issue and did ‘x’ to work around it


Goals and Questions Re-visited

• [All the same as before] Are users confused about how to price a rental? A storage unit?

• How do users react to the insurance options? Do they understand the coverage?

• How do users feel about the presentation of items for purchase or for rent?

• How effective is the shopping cart content? Are users confused by when they need to pay for items?

• Do users value the star ratings? U-Haul brand?• How do users feel about the targeted FAQ and search result

pages?• Does our online documentation help prevent calls to the

service center? Can they determine how to reach out to the U-Haul vendor nearest them?04/08/2023


Positive Findings [these always come first]

04/08/2023

• All participants easily found the links to the FAQs and had no trouble finding the answer to the license question under FAQs

• All participants made use of the maps when comparing options.

• All participants did scroll down to compare prices, locations and reviews

• 4 participants valued the presence of the [higher] star ratings

• 2 participants valued U-Haul location more than the off-brand vendors

• 2 participants were pleased that the truck rental page "retained her information" -- the addresses and dates

• 2 participants appreciated the visuals of the items inside the storage units and the graphic of the person shown in the small unit icon

• 2 participants easily added the dolly, blankets and boxes during the truck rental task flow


Recommendations

04/08/2023

Priority Description Recommendation Location

HighParticipants don't understand what the purchased insurance actually covers

Re-format coverage and exclusions into bulleted lists; Don't use legal jargon

Damage coverage

HighParticipants have a very hard time estimating the storage unit size that would meet their needs

Provide more user assistance

Self Storage location details page

MediumUp-sell process for items to rent or purchase is confusing

Put the purchased items into another page in the flow, and make it clearer that users can opt out.

Additional rental items, Shopping cart

MediumParticipants are concerned that the site is incorrectly calculating the mileage and therefore overcharging

Add a link to display the map, so they can check it in place

Select your preferred pickup location

Low

Participants were not sure what location the giant thumbtack/pin was (address or zip code) or how far away the locations were

Display the distance "from" the specified location, like the Self-storage results page

Select your preferred pickup location, Location


Next Steps

• Work with [which stakeholders or teams] to prioritize changes

• Work with [stakeholders or teams] to design alternatives

• Validate that the new designs address the issues with users

04/08/2023


Summary

• Tell them what you’re going to tell them– Executive summary– Agenda– Goals/Questions

• Tell them – Tasks & participants (sometimes methodology)– Animated slides for progressive disclosure– Screen shots annotated with findings

• Tell them what you told them– Review goals of the research and the questions it was

intended to answer– Positive findings (go slowly here)– Prioritized opportunities for improvement


Eva KaniastyFounding Principal, RedPill UX


Report Formats

PPT: visually engaging but real-estate constrained (and will force you to be brief). Formatting can be time-consuming.

MS Word/Narrative: more room for context; quick, but can appear dry and boring.

3rd Option: No report.


Deciding Factors

• Time/Budget• (Mode of) Presentation of Results• Company Culture / Industry• Stakeholder Involvement• Deliverable Shelf Life


Reporting Findings (1)


Reporting Findings (2)


Dharmesh MistryUsability Specialist, Acquia

Content Management SystemOpen Source SoftwareCommunity

Products built on DrupalOpen Source/ ProprietaryStart-up


Supporting information Detailed Information Main Report

http://drupal.org/node/1399056 http://drupal.org/node/1399258http://drupal.org/node/1289476

Stakeholder behavior

Transparency


Main Report

Stakeholderscomments

http://drupal.org/node/1175694http://www.drupalusability.org/

Tracking

Recommendations


Credibility

Turn around time


Cool

Not Cool?


Stakeholders

Turn around time

Tracking

Presenting

Provide recommendations

Thousands of Stakeholders

Weeks-Months

High, Extensive

Twitter, Conferences, Front page on Drupal.org

No, never!

3-5

Hours-day-week

Moderate-Very High

Email, Conference meetings

Sometimes


Kyle SoucyFounding Principal, Usable Interface


Usable Interface Formal Usability Testing/Research Report


Executive Summary

3-4

Positive Findings

3-4

Negative Findings

When, What, Who, Where,

and Why Statement


Findings: Severity Ratings


Findings

Major Usability Problem


Findings: Recommendations


Presenting Findings…

Finding: All of the participants were disoriented after clicking the “View in 3D” button on the quote.

Recommendations:

• The navigation tabs should not disappear in the 3D view.

• If the page needs to be reloaded than the orientation should remain the same and not take the user back to the top of the screen.

• If possible, the “View in 3D” button should download the ActiveX plug-in.


Highlight Video


Observer Debrief Notes


Carolyn SnyderFounding Principal, Snyder Consulting

• There is no one “best” format• Do what works for the client, culture,

circumstances• Steal good ideas, drop losers


Text Report: “I’m not dead yet!”

Finding

Severity rating

Explanation of issue

Supporting observations from notes

Recommendations


PowerPoint with “report” in Notes Field


PowerPoint, Screen Shots with Callouts

44

Some people saw this as a bar graph, but others did not. One person read only [first 4 words]

Most people read this text; everyone drilled into [noun]

People seemed to understand the stacked bar graphs,

Amount isn’t explicit. The user must do the math.

Can’t explore [action]. People knew it was important.

Why just show [scenario]? Some misinterpreted it as “worst case scenario.”

People liked suggestions, but wanted concrete, prioritized advice. Order can imply priority.

Not clear why it showed [2 variations of graph]

People understood the purpose

Interest in theseInterest in these links

Important sentence buried in paragraph

Ambiguous


Sometimes the best report is…


Questions

1. Do you change your delivery of usability results depending on your role as a internal/external consultant or as a company employee?

2. How important are positive vs. negative findings?

3. How have your reports changed over the years? Is there anything you do differently than when you first started writing them?

4. How do you categorize the findings in your reports? For example, do you categorize them by the page/screen, by the step in a certain process (e.g. checkout process), or by the task?

5. Lean UX is a trending topic. Have you had experience with Lean UX or Agile methods, and had to change the way you conduct research and deliver results?

6. What guidelines do you follow when writing recommendations or proposed solutions to problems?

7. Do you decide ahead of time how long a report should be and make an effort to keep it that length? If so, what dictates the length?

8. If you think a report is too long and needs to be trimmed down, how do you decide what to cut out?

9. What part of a report is the hardest for you to write?

Delivering Results: How Do You Report User Research Findings?

Technology

Transcript of Delivering Results: How Do You Report User Research Findings?