Observation

Watch, listen, and learn…

Agenda

Observation exercise Come back at 3:40.

Questions? Observation

Observing Users

Not as easy as you think

One of the best ways to gather feedback about your interface

Watch, listen and learn as a person interacts with your system

Qualitative & quantitative, end users, experimental or naturalistic

Observation

Direct In same room Can be intrusive Users aware of

your presence Only see it one

time May use 1-way

mirror to reduce intrusiveness

IndirectVideo or app recordingReduces intrusiveness, but doesn’t eliminate itCameras focused on screen, face & keyboardGives archival record, but can spend a lot of time reviewing it

Location

Observations may be In lab - Maybe a specially built usability

lab Easier to control Can have user complete set of tasks

In field Watch their everyday actions More realistic Harder to control other factors

UsabilityLab

http://www.surgeworks.com/services/observation_room2.htm

Large viewing area in this one-way mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms.

Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room.

ObservationRoom

State-of-the-art observation room equipped with three monitors to view participant, participant's monitor, and composite picture in picture.

One-way mirror plus angled glass captures light and isolates sound between rooms.

Comfortable and spacious for three people, but room enough for six seated observers.

Digital mixer for unlimited mixing of input images and recording to VHS, SVHS, or MiniDV recorders.

Task Selection

What tasks are people performing? Representative and realistic? Tasks dealing with specific parts of the

interface you want to test? Problematic tasks?

Don’t forget to pilot your entire evaluation!!

Engaging Users in Evaluation

What’s going on in the user’s head? Use verbal protocol where users describe

their thoughts

Qualitative techniques Think-aloud - can be very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions

“What did you like best/least?” “How would you change..?”

Think Aloud

User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do

Widely used, popular protocol Potential problems:

Can be awkward for participant Thinking aloud can modify way user performs

task

Cooperative approach

Another technique: Co-discovery learning (Constructive iteration) Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert

(coach) and one be novice More natural (like conversation) so removes

some awkwardness of individual think aloud Variant: let coach be from design team

(cooperative evaluation)

Alternative

What if thinking aloud during session will be too disruptive?

Can use post-event protocol User performs session, then watches

video afterwards and describes what s/he was thinking

Sometimes difficult to recall Opens up door of interpretation

What if a user gets stuck?

Decide ahead of time what you will do. Offer assistance or not? What kind of

assistance?

You can ask (in cooperative evaluation) “What are you trying to do..?” “What made you think..?” “How would you like to perform..?” “What would make this easier to accomplish..?” Maybe offer hints This is why cooperative approaches are used

Inputs / Outcomes

Need operational prototype could use Wizard of Oz simulation

What you get out “process” or “how-to” information Errors, problems with the interface compare user’s (verbalized) mental

model to designer’s intended model

Capturing a Session

1. Paper & pencil Can be slow May miss things Is definitely cheap and easy

Time 10:00 10:03 10:08 10:22

Task 1 Task 2 Task 3 …

Se

Se

Capturing a Session

2. Recording (audio and/or video) Good for think-aloud Hard to tie to interface Multiple cameras may be needed Good, rich record of session Can be intrusive Can be painful to transcribe and

analyze

Capturing a Session

3. Software logging Modify software to log user actions Can give time-stamped key press or

mouse event Two problems:

Too low-level, want higher level events Massive amount of data, need analysis

tools

Example logs2303761098721869683|hrichter|1098722080134|MV|START|5662303761098721869683|hrichter|1098722122205|MV|QUESTION|false|false|false|false|false|false| 2303761098721869683|hrichter|1098724978982|MV|TAB|AGENDA2303761098721869683|hrichter|1098724981146|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098724985161|MV|SLIDECHANGE|52303761098721869683|hrichter|1098724986904|MV|SEEK|PRESENTATION-A|566|604189|02303761098721869683|hrichter|1098724996257|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098724998791|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098725002506|MV|TAB|AGENDA2303761098721869683|hrichter|1098725003848|MV|SEEK|AGENDA|566|149613|6041892303761098721869683|hrichter|1098725005981|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725007133|MV|SLIDECHANGE|32303761098721869683|hrichter|1098725009326|MV|SEEK|PRESENTATION|566|315796|1496132303761098721869683|hrichter|1098725011569|MV|PLAY|566|3157962303761098721869683|hrichter|1098725039850|MV|TAB|AV2303761098721869683|hrichter|1098725054241|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725056053|MV|SLIDECHANGE|22303761098721869683|hrichter|1098725057365|MV|SEEK|PRESENTATION|566|271191|3157962303761098721869683|hrichter|1098725064986|MV|TAB|AV2303761098721869683|hrichter|1098725083373|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725084534|MV|TAB|AGENDA2303761098721869683|hrichter|1098725085255|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725088690|MV|TAB|AV2303761098721869683|hrichter|1098725130500|MV|TAB|AGENDA2303761098721869683|hrichter|1098725139643|MV|TAB|AV2303761098721869683|hrichter|1098726430039|MV|STOP|566|2711912303761098721869683|hrichter|1098726432482|MV|END

Analysis

Many approaches Task based

How do users approach the problem What problems do users have Need not be exhaustive, look for interesting

cases Performance based

Frequency and timing of actions, errors, task completion, etc.

Very time consuming!!

Example: Heather’s study

Software: MeetingViewer interface fully functional

Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing

Resources – subjects were students in a research group, just me as evaluator, plenty of time

Wanted completely authentic experience

Heather’s evaluation

Task: answer questions from a recorded meeting, use my software as desired

Think-aloud Video taped, software logs Also had post questionnaire Wrote my own code for log analysis Watched video and matched behavior to

software logs

Example: Analysis

Printed logs Watched video and marked actions,

thoughts, and video timing info on the logs

Wrote down interesting quotes from the think aloud

Took 2 to 3 times the session time just to do this analysis alone!