Predictive Evaluation

Predicting performance

Predictive Models

Translate empirical evidence into theories and models that can influence design.

Performance measures– Quantitative– Time prediction– Working memory constraints

Competence measures Focus on certain details, others obscured

Two Types of User Modeling

Stimulus-Response– Practice law– Fitt’s law

Cognitive – human as interpreter/predictor – based on Model Human Processor (MHP)– Key-stroke Level Model

Low-level, simple– GOMS (and similar) Models

Higher-level (Goals, Operations, Methods, Selections)

Power law of practice

Tn = T1n-a

– Tn to complete the nth trial is T1 on the first trial times n to the power -a; a is about .4, between .2 and .6

– Skilled behavior - Stimulus-Response and routine cognitive actions

Typing speed improvement Learning to use mouse Pushing buttons in response to stimuli NOT learning

Power Law: Tn = T1n-a

n T_n1 52 3.795 2.62

10 1.9915 1.6925 1.37

0

1

2

3

4

5

6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

n

Tria

l Tim

e

If first trial (T1) takes 5 seconds, how long will future trials take? When will improvements level off? (a = -0.4)

Uses for Power Law of Practice

Use measured time T1 on trial 1 to predict whether time with practice will meet usability criteria, after a reasonable number of trials

– How many trials are reasonable? Predict how many practices will be needed for user

to meet usability criteria– Determine if usability criteria is realistic

Fitts’ Law

Models movement times for selection tasks Paul Fitts: war-time human factors pioneer

Basic idea: Movement time for a well-rehearsed selection task– Increases as the distance to the target

increases– Decreases as the size of the target increases

Moving

Move from START to STOPD

W

START STOP

Index of Difficulty:ID = log2 ( 2D/W ) (in unitless bits)

width of targetdistance

Movement Time

MT = a + b*IDor

MT = a + b log2 (2D/W)

MT

ID•Empirical measurement establishes constants a and b

•Different for different devices and different ways the same device is used.

Questions

What do you do in 2D?

– h x l rect:one way is ID = log2(d/min(w, l) + 1)

– Should take into account direction of approach

Applications

When does it apply?

How used in interface design?

Keystroke Level Model (KLM)

Also developed by Card, Moran, and Newell (1983)

Skilled users performing routine tasks– Assumes error-free performance

Analyze only observable behaviors– Keystrokes, mouse movements

Assigns times to basic human operations - experimentally verified

KSLM Accounts for

Keystroking TK

Mouse button press TB

Pointing (typically with mouse) TP

Hand movement betweenkeyboard and mouse TH

Drawing straight line segments TD

“Mental preparation” TM

System Response time TR

Step One : MS Word Find Command

Use Find Command to locate a six character word– H (Home on mouse)– P (Edit)– B (click on mouse button - press/release)– P (Find)– B (click on mouse button)– H (Home on keyboard)– 6K (Type six characters into Find dialogue box)– K (Return key on dialogue box starts the find)

Using KSLM - Step Two

Place M operatorsRule 0a. In front of all K’s that are NOT part of argument

strings (ie, not part of text or numbers)

Rule 0b. In front of all P’s that select commands (not arguments)

Step Two : MS Word Find Command

H (Home on mouse)MP (Edit)B (click on mouse button)MP (Find)B (click on mouse button)H (Home on keyboard)6K (Type six characters)MK (Return key on dialogue box starts

the find)

Rule 0b: Pselects command

Rule 0b: Pselects command

Rule 0a: Kis argument

Using KSLM - Step 3

Remove M’s according to heuristic rules (Rules relate to chunking of actions)Rule 1. Anticipated by prior operation

– H MP ->HP (pointing to menu item is anticipated by moving hand to mouse)

Rule 2. If string of MKs is a single cognitive unit (such as a command name), delete all but first– MKMKMK -> MKKK (same as M3K) (type “run is a chunk)

Rule 3. Redundant terminator, such as )) or rtn rtnRule 4. If K terminates a constant string, such as command-rtn, then

delete M M2K(ls)MK(rtn) -> M2K(ls)K(rtn) (typing “ls” command in Unix

followed by rtn is a chunk)

H (Home on mouse)MP (Edit)B (click on mouse button)MP (Find)B (click on mouse button)H (Home on keyboard)6K (Type six characters)MK (Return key on dialogue box starts

the find)

Step 3 : MS Word Find Command

Rule 4 Keep M

Rule 1 delete MH anticipates P

Using KSLM - Step 4

Plug in real numbers from experiments– K: .08 sec for best typists, .28 average, 1.2 if unfamiliar

with keyboard– B: down or up - 0.1 secs; click - 0.2 secs– P: 1.1 secs– H: 0.4 secs– M: 1.35 secs– R: depends on system; often less than .05 secs

Step 4 : MS Word Find Command

H (Home on mouse)P (Edit)B (click on mouse button - press/release)MP (Find)B (click on mouse button)H (Home on keyboard)6K (Type six characters into Find dialogue box)MK (Return key on dialogue box starts the find) Timings

– H = 0.40, P = 1.10, B = 0.20, M = 1.35, K = 0.28– 2H, 2P, 2B, 2M, 7K

Predicted time = 8.06 secs

Example: MS Windows Menu Selection

Get hands on mouse Select from menu bar with click of mouse

button The “pull down” menu appears Select desired item from the pull down menu

Step 1: MS Windows Menu

H (Home on mouse)

P (point to menu bar item)

B (left-click with mouse button)

P (point to menu item)

B (left-click with mouse button)

Step 2: MS Windows Menu - Add M’s

H (get hand on mouse)

MP (point to menu bar item)

B (left-click with mouse button)

MP (point to menu item)

B (left-click with mouse button)

Rule 0b: Pselects command

Rule 0b: Pselects command

Step 3: MS Windows Menu - Delete M’s

H (get hand on mouse) MP (point to menu bar item) B (left-click with mouse button) MP (point to menu item) B (left-click with mouse button)

Keep M

Rule 1 Manticipated by H

Step 4: MS Windows Menu Calculate Time

H (get hand on mouse) P (point to menu bar item) B (left-click with mouse button) MP (point to menu item) B (left-click with mouse button) Textbook timings (all in seconds)

– H = 0.40, P = 1.10, B = 0.20, M = 1.35– H, 2P, 2B, 1 M

Total predicted time = 4.35 sec

Alternative Menu Selection

Operator sequence– H(mouse)P(to menu item)B(down)PB(up)

Now place Ms– H(mouse)MP(to menu item)B(down)MPB(up)

Selectively remove Ms– H(mouse)MP(to menu item)B(down)MPB(up)

Textbook timings (all in seconds) H = 0.40, P = 1.10, B = 0.10 for up or down, M = 1.35 H, 2P, 2 B, 1 M

Total predicted time = 4.15 sec Alternative is predicted to be .2 secs faster than typical, about 5%

Rule 0b

Rule 0b

Rule 1 Delete H anticipates P

KSLM Comparison Problem

Are keyboard accelerators always faster than menu selection?

Use MS Windows to compare– Menu selection of File/Print (previous example estimated

4.35 secs.)– Keyboard accelerator

ALT-F to open the File pull down menu P key to select the Print menu item

Assume hands start on keyboard

KSLM Comparison:Keyboard Accelerator for Print

Use Keyboard for ALT-F P (hands already there)– K(ALT)K(F)K(P)

– MK(ALT)MK(F)MK(P)

– MK(ALT)K(F)MK(P) 2M + 3K = 2.7 + 3K

Times for K based on typing speed– Good typist, K = 0.12 s, total time = 3.06 s– Average typist, K = 0.28 s, total time = 3.54 s– Non-typist, K = 1.20 s, total time = 6.30 s– Time with mouse was 4.35 sec

Conclusion: Accelerator keys not faster than mouse for non-typists

First Kanticipatessecond K

What if you started on the mouse?

Use Keyboard for ALT-F P (hands already there)– H K(ALT)K(F)K(P)

– H MK(ALT) K(F)MK(P)

– H MK(ALT)K(F)MK(P) H + 2M + 3K = 3.1 + 3K

Times for K based on typing speed– Good typist, K = 0.12 s, total time = 3.46 s– Average typist, K = 0.28 s, total time = 3.94 s

Menu selection was 4.35 sec when starting on keyboard– To start on mouse its 4.35 – H = 3.95 seconds

Conclusion: Hmmm… not faster for average typists

Comparison

Consider: compare selecting a menu item in a right-click popup menu vs. selecting the same menu item from menu in menu bar

What would Fitt’s Law say? What would KSLM say?

One more practice

Draw through text and make it bold– By pointing to BOLD icon in floating palette– By selecting BOLD from pull-down menu