Interpretive Evaluation

Agenda• Simple user modeling• Interpretive Evaluation

- Motivation- Methods

Ethnography

• Fitt’s Law

Fall 2002 CS/PSY 6750 2

Simpler User Modeling

• How do attributes of users influence the design of user interfaces?

• Are there some design guidelines that we can derive from different attributes?

Fall 2002 CS/PSY 6750 3

User Profiles

• Attributes: attitude, motivation, reading level,

typing skill, education, system experience, task experience, computer literacy, frequency of use, training, color-blindness, handedness, gender,…

• Novice, intermediate, expert

Fall 2002 CS/PSY 6750 4

Motivation

• User Low motivation,

discretionary use Low motivation,

mandatory High motivation,

due to fear High motivation,

due to interest

• Design goal Ease of learning

Control, power

Ease of learning, robustness, control

Power, ease of use

Fall 2002 CS/PSY 6750 5

Knowledge & Experience

• Experience• task system

low low

high high

low high

high low

• Design goals Many syntactic and

semantic prompts Efficient commands,

concise syntax Semantic help

facilities

Lots of syntactic prompting

Fall 2002 CS/PSY 6750 6

Job & Task Implications

• Frequency of use High - Ease of use Low - Ease of learning & remembering

• Task implications High - Ease of use Low - Ease of learning

• System use Mandatory - Ease of using Discretionary - Ease of learning

Fall 2002 CS/PSY 6750 7

Evaluation

• Predictive Discount Usability (Mon) Cognitive Modeling-GOMS family (Wed) Cognitive Modeling-contextual (next

Mon)

• Interpretive (today)• User testing (later)

Fall 2002 CS/PSY 6750 8

Approaches

• Experimental (Lab studies, quantitative) Typically in a closed, lab setting

Manipulate independent variables to see effect on dependent variables

• Naturalistic (Field studies, qualitative) Observation occurs in “real life” setting

Watch process over time “Ecologically valid”

Fall 2002 CS/PSY 6750 9

Interpretive Evaluation

• Experimental: Formal and objective• Interpretive: More subjective

Concerned with humans, so no objective reality

Sociological, anthropological approach

• Users involved, as opposed to predictive approaches

Fall 2002 CS/PSY 6750 10

Beliefs

• Sees limitations in scientific hypothesis testing in closed environment Lab is not real world Can’t control all variables Context is neglected Artificial, short tasks

Fall 2002 CS/PSY 6750 11

IE Methods

• A number of different methods or techniques in this area exist Ethnography

Our main focus Contextual inquiry

More specific form of ethnography with a focus on asking questions

Field studyCommon notion, often equated with ethnography

Observational studyMuch more informal, just watching users

…

Fall 2002 CS/PSY 6750 12

Ethnography

• Deeply contextual study• Immerse oneself in situation you want

to learn about (anthropological and sociological roots) Observing people in their cultural context

• Interpretation of data is primary• Behavior is meaningful only in context

Fall 2002 CS/PSY 6750 13

Philosophy

• Argues that formal environment of controlled study is artificial --- Experimenter wields “power” over subject

• So … get into working environment of user

• -> Interpretation is primary, rather than data

Fall 2002 CS/PSY 6750 14

Objectives

• Understanding the user Understand goals and values Understand individual’s or group’s

interactions within a culture Try to make tacit domain knowledge explicit Do this in an unbiased fashion For UI designers: Improve system by finding

problems in way it is currently being used

Fall 2002 CS/PSY 6750 15

Techniques

• In person observation• Audio/video recording• Interviews

• “Wallow in the data”

Fall 2002 CS/PSY 6750 16

Observation is Key

• Carefully observe everything about users and their environment

• Think of describing it to someone who has never seen this activity before

• What users say is important, but also non-verbal details

Fall 2002 CS/PSY 6750 17

Observations

• Things of interest to evaluator Structure and language used in work Individual and group actions Culture affecting work Explicit and implicit aspects of work

• Example: Office work environment Business practices, rooms, artifacts, work

standards, relationships between workers, managers, …

Fall 2002 CS/PSY 6750 18

Interviews Important

• Have a question plan, but keep interview open to different directions

• Be specific• Create interpretations together with

users• At end, query “What should I have

asked?”• Record interviews

Fall 2002 CS/PSY 6750 19

Steps

• 1. Preparation Understand organization policies and work culture Familiarize yourself with system and its history Set initial goals and prepare questions Gain access and permission to observe & interview

• 2. Field study Establish rapport with users Observe/interview users in workplace and collect

all different forms of data Follow any leads that emerge from visits Record the visits

Rose et al ‘95

Fall 2002 CS/PSY 6750 20

Steps

• 3. Analysis Compile collected data in numerical, textual

and multimedia databases Quantify data and compile statistics Reduce and interpret data Refine goals and process used

• 4. Reporting Consider multiple audiences and goals Prepare a report and present findings

Fall 2002 CS/PSY 6750 21

One Technique

• Affinity Diagram• Process

Write down each quote/observation on a slip of paper

Put up on board Coalesce items that have affinity

If they are saying similar things about an issue

Give names to different groups (colors too) Continue grouping subgroups A hierarchy will be formed

Fall 2002 CS/PSY 6750 22

Why Useful?

• Can help designer gain a rich and true assessment of user needs Help to define requirements

• Uncovers true nature of user’s job Discovers things that are outside of job description or

documentation

• Allows you to play role of end-user Can sit in when real users not available

• Open-ended and unbiased nature promotes discovery Empirical study and task analysis are more formal –

ethnography may yield more unexpected revelations

Fall 2002 CS/PSY 6750 23

Types of Findings

• Can be both Qualitative

Observe trends, habits, patterns, …

QuantitativeHow often was something done, what

per cent of the time did something occur, how many different …

Fall 2002 CS/PSY 6750 24

Drawbacks

• Time required Can take weeks or months

• Scale Most use small numbers of participants just to

keep somewhat manageable

• Type of results Highly qualitative, may be difficult to present

and use

• Acquired skill Identifying and extracting “interesting” things is

challenging

Fall 2002 CS/PSY 6750 25

Ethnomethodology

• Concurrent/informed ethnography Study is being done in conjunction with

a system being developed + Helps keep focus on user throughout

design - Requires lots of time and coordination

Fall 2002 CS/PSY 6750 26

Cooperative Evaluation

• User is viewed as collaborator in evaluation, not a subject “Friendly approach”

• Relaxed version of think-aloud Evaluator and participant can ask each

other questions

Fall 2002 CS/PSY 6750 27

CE Methods

• Seeks to detect errors early in a prototype

• Experimenter uses tasks, also talks to participant throughout, asks questions

Fall 2002 CS/PSY 6750 28

Low Level Models

• Fitt’s Law Models movement times for selection tasks

• Basic idea: Movement time for a well-rehearsed selection task Increases as the distance to the target

increases Decreases as the size of the target increases

Fall 2002 CS/PSY 6750 29

Components

• ID - Index of difficulty

ID = log2 ( 2A/W )

bits result

Both quantities are distancesso unitless result

width toleranceof target

distance to move

Fall 2002 CS/PSY 6750 30

Components

• MT - Movement time

MT = b * ID orMT = a + b*ID

MT is a linear function of ID

Fall 2002 CS/PSY 6750 31

Original Experiment

• 1-DA W

Fall 2002 CS/PSY 6750 32

Exact Equation

• Run empirical tests to determine a and b in MT = a + b*ID

• Will get different ones for different input devices and ways the device is used

MT

ID

Fall 2002 CS/PSY 6750 33

Common Equation

• MT = a + b log2 (A/W + 1)

• Provides useful numbers

Fall 2002 CS/PSY 6750 34

Questions

• What do you do in 2D?

• Where can this be applied in user interface design?