Experience Research Best Practices - UX Meet Up Boston 2013 - Dan Berlin
Transcript of Experience Research Best Practices - UX Meet Up Boston 2013 - Dan Berlin
September 19, 2013
Dan BerlinExperience Research [email protected]@banderlin
Mad*Pow
EXPERIENCE RESEARCH BEST PRACTICES
Hi! I’m Dan Berlin @banderlin
BA in psychology from Brandeis U. Studied visual space perception
Seven years in technical support Sat as a participant for a usability study for a product I was working on Realized that user experience (UX) work is the perfect combination of computers and psychology
Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design
Two years at an interactive agency performing usability and neuromarketing research Then did some freelance UX consulting for about a year
Am now an Experience Research Director at Mad*Pow, an experience design agency based out of Portsmouth, NH
My passion is for research methodology and finding new ways to elicit data from participants
2
About this Presentation
Understanding your research goals Formative and evaluative studies
How to choose the right method(s) Methods chart
Gathering qualitative data Taking notes and organizing findings
3
UNDERSTANDING YOURRESEARCH GOALS
4
Understanding Your Research Goals
5
Preparing to prepare for your study
TechnologyUser
Needs
BusinessGoals
Your overall goal:
To uncover actionable business and design insights via user data
Start by understanding the business’ goals What is the overall goal of the interface? What are the interactions that drive the business? What are the important calls to action? Who are the target audiences? What do they want to know about their users? What do they think they already know about their users
How did they learn this?
Understanding Your Research Goals
6
Preparing to prepare for your study
Actionable = the data should indicate what exactly needs to change or be implemented in the interface to align it with users’ expectations and needs
This means you need to start with actionable (and attainable) goals, such as: How do perform a certain interaction today? Does the proposed IA align with user expectations? Are users able to complete a critical transaction? Do users understand the pricing schedule?
Goals will depend on the type of study Formative
Performed at the beginning of a project to learn how the project should proceed
Evaluative Performed during and towards the end of projects to determine if the design aligns with user expectations
Formative In-Person Usability Interviews Focus Groups Collaging Ethnography Surveys Diary Studies Card-Sorting
7
Evaluative In-Person Usability Remote Usability Unmoderated Usability Desirability Testing Eye-Tracking
Understanding Your Research GoalsStudy Types
HOW TO CHOOSE THERIGHT METHOD(S)
8
How to Choose the Right Method(s)
Make sure your research goals are clear Know what you want to learn Know what you will do with the information you gather Know what decisions the business needs to make
Know your constraints Timeline? Resources? Budget? Access to Users?
Create a Methods Chart The answers will become clear
9
Know Goals and Constraints
How to Choose the Right Method(s)
10
Method Pros Cons CandidateMethod 1 • List the advantages for
this method for this specific project
• List the disadvantages for this method for this specific project
Yes / No – Would this method be a good candidate for this specific project
Method 2Method 3Etc…
Methods Chart
How to Choose the Right Method(s)
Example Project Goals: Have 3 concepts for behavior change application. Which one is the best to
develop for maximum global appeal? Client wants answers as quickly as possible Client wants large numbers to provide confidence behind decision Client wants data collection in 5 countries around the world Designers want to know why or why not users chose each concept, to
provide additional design direction User population – adults who own a smart phone and want to get healthier
11
Methods Chart
Methods Chart Example
12
Method Pros Cons CandidateInterviews • Good for capturing
motivations for behavior change
• Can get good qualitative details on why users prefer each concept or not
• Good to explore issues to fine tune survey questions
• Can be done via phone (get broad geographic sample)
• Can be done quickly
• Small numbers – client wants large numbers
Y – would be good as a qualitative method to pair with a larger quantitative method
Focus Groups • Good for qualitative information gathering
• Could generate some interesting conversations about behavior change motivations
• Wouldn’t get as much detail as interviews
• Concern about group think when evaluating concepts
• Concern about not sharing details of personal goals in front of others
N – interviews would be better for qualitative
Survey • Good for large numbers• Easily replicated across
different countries/languages
• Can be done online for broad geographic distribution
• Can be done quickly
• Unclear what exact questions to ask
• Doesn’t provide detailed insights into qualitative topics
Y – good paired with qualitative method
How to Choose the Right Method(s)
Example Research Plan: Conduct telephone interviews in US with 12-15 participants Create online survey in US for 300 participants Use international research partners to conduct 12-15 interviews in 4
countries Use international research partners to conduct survey for 300 participants in
same 4 countries
Why this approach works: Interviews provide input into questions for survey Survey provides large numbers; interviews provide in-depth insight Both are commonly used methods and easy to replicate in different
countries Both methods can be done quickly
13
Methods Chart
GATHERING QUALITATIVE DATA
14
Gathering Qualitative Data
15
Usability Task Creation
Good usability study tasks are:
Non-leading: don’t give away the answer
Single tasks: don’t have users do two things in succession
Realistic: don’t have users do things they would not normally do
Self-explanatory: don’t have multiple sentences explaining the task
Achievable: always document what constitutes a “pass” for the task
Gathering Qualitative Data
16
Moderator’s Guide
A good study guide: Conveys the study goals and methodology to clients Serves as a quick reference to the moderator during the study Provides the moderator with a template to take hand-written notes, with ample space to
do so
The typical moderator’s guide contains:
Research method: a short paragraph explaining the usability study Study goals: a short bulleted list of the study goals Introduction: the moderator’s opening spiel when explaining the study to the participant Background questions: Typically demographic or product usage questions Tasks: The task, pass condition(s), notes for the moderator, and post-task questions Follow-up questions: The questions to be asked after all the tasks are complete
17
Gathering Qualitative Data
18
Notes Grid
Proper planning for taking notes is very underrated The organization and thoroughness of your notes will dictate the ease with which you
will create the final report Organized, complete notes = easy reporting Disorganized, incomplete notes = back to the video you go (ewww!)
Make your notes grid once your study & moderator’s guides are complete Give each question and subquestion its own row Put each participant in a new column (or vice-versa, if you like)
DO NOT put each participant/task in a new worksheet
Use data validation for quantitative data Task ease ratings, task success, multiple choice questions, etc. Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to
the question)
19
• Do not include the participant name in your notes grid
• Include a date/time cell to best align with the videos
• Visually separate sections of the study
• Fill data validated cells with a light color
• Hide columns when you have moved on to the next participant
Gathering Qualitative DataNotes Grid
20
Gathering Qualitative DataFindings Sheet
Scan the notes grid to determine findings Put these in a spreadsheet and assign and priority and category
In Conclusion
Set actionable and attainable research goals Know where you are in the project
Use a methods chart to weigh factors and choose a method Time, participants, goals, budget, etc.
Proper documentation = easier qualitative analysis Study guide notes grid findings spreadsheet report
21