Post on 10-May-2015
UX Assessment Techniques
Jen Romano Bergstrom
December 11, 2013
NOVA UX
@forsmarshgroup @romanocog
2
@forsmarshgroup @romanocog
Measuring the UX
• How does it work for the
end user?
• What does the user
expect?
• How does it make the user
feel?
“the extent to which a
product can be used by
specified users to
achieve specified goals
with effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
+ emotions
3
@forsmarshgroup @romanocog
Why is it important?
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not to
your users.
4
@forsmarshgroup @romanocog
Why is it important?
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not to
your users.
5
@forsmarshgroup @romanocog
Why is it important?
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not to
your users.
6
@forsmarshgroup @romanocog
Where to test
• Controlled environment
• All participants have the
same experience
• Record and
communicate from
control room
• Observers watch from
control room and provide
additional probes (via
moderator) in real time
• Incorporate physiological
measures (e.g., eye
tracking, EDA)
• No travel costs
LABORATORY REMOTE IN THE FIELD
• Participants tend to be
more comfortable in
their natural
environments
• Recruit hard-to-reach
populations (e.g.,
children, doctors)
• Moderator travels to
various locations
• Bring equipment (e.g.,
eye tracker)
• Natural observations
• Participants in their
natural environments
(e.g., home, work)
• Use video chat
(moderated sessions)
or online programs
(unmoderated)
• Conduct many sessions
quickly
• Recruit participants in
many locations (e.g.,
states, countries)
7
@forsmarshgroup @romanocog
Where to test
• Controlled environment
• All participants have the
same experience
• Record and
communicate from
control room
• Observers watch from
control room and provide
additional probes (via
moderator) in real time
• Incorporate physiological
measures (e.g., eye
tracking, EDA)
• No travel costs
LABORATORY REMOTE IN THE FIELD
• Participants tend to be
more comfortable in
their natural
environments
• Recruit hard-to-reach
populations (e.g.,
children, doctors)
• Moderator travels to
various locations
• Bring equipment (e.g.,
eye tracker)
• Natural observations
• Participants in their
natural environments
(e.g., home, work)
• Use video chat
(moderated sessions)
or online programs
(unmoderated)
• Conduct many sessions
quickly
• Recruit participants in
many locations (e.g.,
states, countries)
8
@forsmarshgroup @romanocog
How to test
• In-depth feedback from
each participant
• No group think
• Can allow participants to
take their own route and
explore freely
• No interference
• Remote in participant’s
environment
• Flexible scheduling
• Qualitative and
Quantitative
ONE-ON-ONE SESSIONS FOCUS GROUPS SURVEYS
• Representative
• Large sample sizes
• Collect a lot of data
quickly
• No interviewer bias
• No scheduling sessions
• Quantitative analysis
• Participants may be
more comfortable with
others
• Interview many people
quickly
• Opinions collide
• Peer review
• Qualitative
When to test
9
@forsmarshgroup @romanocog
10
@forsmarshgroup @romanocog
What to measure
OBSERVATIONAL + Ethnography
+ Time to complete task
+ Reaction time
+ Selection/click behavior
+ Ability to complete tasks
+ Accuracy
IMPLICIT + Facial expression analysis
+ Eye tracking
+ Electrodermal activity (EDA)
+ Behavioral analysis
+ Linguistic analysis of verbalizations
+ Implicit associations
+ Pupil dilation
EXPLICIT + Post-task satisfaction
questionnaires
+ In-session difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
11
@forsmarshgroup @romanocog
1. Participant repeatedly fixated the upper right hand corner. Participant said that he/she was looking for a search tool on the
page. The search tool was in a disappearing banner on the page.
2. Participants had similar fixation counts across bottom links, indicating uncertainty of where to click to get started.
Case 1
• Problems: What do users want? Does the new design work?
• Methods: Focus groups, one-on-one interviews, in-lab
usability testing with eye tracking
12
@forsmarshgroup @romanocog
1. Participant repeatedly fixated the upper right hand corner. Participant said that he/she was looking for a search tool on the
page. The search tool was in a disappearing banner on the page.
2. Participants had similar fixation counts across bottom links, indicating uncertainty of where to click to get started.
Case 1
• Problems: What do users want? Does the new design work?
• Methods: Focus groups, one-on-one interviews, in-lab
usability testing with eye tracking
13
@forsmarshgroup @romanocog
Case 2 • Problem: What parts of the form do people actually read?
• Method: In-lab usability testing with eye tracking
14
@forsmarshgroup @romanocog
Case 2
0
20
40
60
80
100
120
Page 1 Page 2 Page 3
Tim
e (
se
con
ds)
Participants did not read the
instructions in their entirety (page 1:
left; page 3: right); rather, they skimmed
and then moved on to the form where
they needed to enter information.
Length of time spent on each page of the instructions
before working on form.
Aggregate fixation count heat map across all
participants, Page 1. Participants looked at
‘Purpose of Form’ section the most often.
15
@forsmarshgroup @romanocog
Case 3
Prototype Grid Old Grid
16
@forsmarshgroup @romanocog
Case 4
Tablet
Smartphone
Gaze Plot: After getting an error message, the participant
had to search all over the screen to find the missing field.
“How do I advance to the next screen?”
“It seems like it's stuck on the screen.”
17
@forsmarshgroup @romanocog
Many ways to evaluate the UX
• Surveys
• Focus groups
• In-person one-on-one with eye tracking
• Analytics
18
@forsmarshgroup @romanocog
Obstacles to UX Testing
• There is no time.
– Start early in development process
– One morning a month with 3 users – Krug
– 12 people in 3 days – Anderson Reimer
– 12 people in 2 days – Lebson & Romano Bergstrom
• Can’t find representative users
– Everyone is important
– Travel
– Remote testing
• We don’t have a lab
– Test anywhere
Thank you!
• Twitter: @forsmarshgroup
• LinkedIn: http://www.linkedin.com/company/fors-marsh-group
• Blog: www.forsmarshgroup.com/index.php/blog
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
NOVA UX