Toolkit Support for Usability Evaluation

05-830 Spring 2013 – Karin Tsai

1

Overview

• Motivation• Definitions• Background from Literature• Examples of Modern Tools

2

Motivation

• To improve or validate usability• Comparison between products, AB tests, etc.• Measuring progress• Verify adherence to guidelines or standards• Discover features of human cognition

3

Usability Attributes

• Learnability – easy to learn• Efficiency – efficient to use• Memorability – easy to remember how to use• Errors – low error rate; easy to recover• Satisfaction – pleasant to use and likable

4

Evaluation Categories

• Predictive– psychological modeling techniques– design reviews

• Observational– observations of users interacting with the system

• Participative– questionnaires– interviews– “think aloud” user-testing

5

Challenges and Tradeoffs

• Quality vs. Quantity– “Quality” defined as abstraction, interpretability, etc.– User testing – high quality; low quantity– Counting mouse clicks – low quality; high quantity

• Observing Context • Abstraction

– Event reporting in applications places burden on developers

– Complicates software evolution

6

• Evaluation Type: Predictive• Description: Uses a predictive human

performance model (“cognitive crash dummy”) to evaluate designs.

7

CogTool

8

CogToolPros ConsFree Limited in “realisticness”Good for getting a baseline evaluation of prototypes

Quite confusing at first (extremely high learning curve)

Instantly accessible (not limited by participant availability or completion of the system’s functionality)

Documentation is “daunting”

Neat concept and insight into human cognition Limited usefulness

Overall Score: 6.5/10

• Evaluation Type: Observational• Description: Aggregates developer-defined

event data in useful ways.

9

Mixpanel

10

MixpanelPros ConsVery powerful built-in analysis tools High learning curveGood API for automated scripting Expensive

Scalable Application events = developer burden/maintainability issues

Flexible to fit needs of developers Rate-limited (one request at a time)

Overall Score: 9.5/10

• Evaluation Type: Observational• Description: Real-time data visualization.

11

Chartbeat

12

ChartbeatPros Cons

Data is real-time Does not scale well (financially) with huge sites

Captures data that is hard to obtain via events (reading, writing, idling, active time, referrals, social integration, etc.)

Limited in the data it captures (have to “hack” it if you want event-like data)

Great for site monitoring Only records “page-level” interactionsReally awesome visualization Limited historical data accessEasy to use Not built for usability evaluation

Overall Score: 7/10

• Evaluation Type: Participative• Description: Watch a user complete a task on

your system while thinking aloud.

13

User Testing

14

User TestingPros ConsProbably best method for catching usability issues Small sample size (hit or miss)

Most thorough recording of user interaction with the system Not easily scalable (expensive)

“Think aloud” allows data insights not otherwise attainable from just user interactions

Limited user availability

Can observe certain demographics without requesting personal information in the system itself

Sometimes, it’s painful to watch…

Overall Score: 8.5/10

15

Questions?