HCI460: Week 3 Lecture September 23, 2009. 2 © October 12, 2015 Project 1 recap Q & A Research...

HCI460: Week 3 LectureHCI460: Week 3 LectureSeptember 23, 2009September 23, 2009

2© April 21, 2023

Project 1 recap

Q & A

Research overview

Defining usability– Usability measures

Testing with users

Preparation for a usability study

Assignment for next week

Outline

3

Project 1 Recap

© April 21, 2023

4© April 21, 2023

No deductions for late submissions but from now on:– In class students: Projects are due Wed 11:59 pm Central Time– DL students: Projects are due Sun 11:59 pm Central Time

Please submit through COL. – One person should submit for the entire group.

Feedback on Project 1a (Individual Notes)Project 1 Recap

5© April 21, 2023

Assignments differed along two dimensions:– Number of issues found

• Some found only a few, some found a lot.– Level of polish

• Some turned in polished deliverables, some turned in “quick and dirty” work.

Real world: perfection vs. efficiency

Recognize when “quick and dirty” is needed and when professional and final deliverable is needed.

At this stage of the project (individual notes), it was better to have more issues and present them in a “quick and dirty” way than to have fewer issues but a more polished deliverable.

Feedback on Project 1a (Individual Notes)Project 1 Recap

6© April 21, 2023

Project 1b is due today at midnight for in-class students and on Sunday at midnight for DL students.– One person from each group should upload it to COL.

What have you learned?

What was difficult / challenging?– About the evaluation itself– About evaluating as a group

How did you overcome the challenges?

What will you do differently next time?

Did you see any value in conducting the evaluation as a group?

Project 1b: DiscussionProject 1 Recap

7

Q & A

© April 21, 2023

8© April 21, 2023

Interaction has been excellent thus far

Clarification will always be provided, just ask

Exercises allow for alternative feedback

Feedback

Opportunities– Push the User eXperience (UX) envelope– Professional development

Class Interaction ~ OpportunityQ & A

9© April 21, 2023

UXalliance.com

mobileHCI– Eating usability– Touch interface debate

Conference RecapQ & A

10

Research Overview

© April 21, 2023

11© April 21, 2023

What do we do?

What do you want to do?

Why Are You Here?Research Overview

12© April 21, 2023

Research background

Design background

Paths Into UX ResearchResearch Overview

13© April 21, 2023

If someone asked, what is it?

What we strive to do is answer questions.

Methodologies and techniques are just tools.

As practitioners, we must be focused on the questions.

Often, finding the answer will require much deeper and complex methodologies than discount usability testing will provide OR NOT!

Let’s explore and quite possibly push the boundaries of user experience research tactics.

Thinking About UX ResearchResearch Overview

14

May have mentioned….

Web sites Software applications Interactive Voice Response systems (IVRs) Speech recognition systems Text-to-speech (TTS) interfaces Voice mail systems Unified messaging interfaces Internet applications Games and gaming systems Telecommunication products and services Call center applications Consumer products Wireless devices Packaging

What Can’t We Evaluate?Research Overview

© April 21, 2023

Consider an iPhone…

15

What Can We Test?Research Overview

© April 21, 2023

Three Dimensions of Feedback

16

How people evaluate objects− Commercials− Packages− Online advertisements− Products− …

Research Overview

© April 21, 2023

Attitude What users “say” ...

Influencers– Social status– Emotion– Coolness / Hip

Reveal– Feature importance – Purchase intent

17

Research Overview

© April 21, 2023

Behavior What users actually “do”

Ultimately, behaviors are what we wish to shape

Give users context, a task and stimuli and then – Observe what users do

Behavior drives usage

18

Research Overview

© April 21, 2023

Attention What users “focus” on…

What happens inside the head

Sometimes users are unaware

Often attention measured by

eye tracking

19

Research Overview

© April 21, 2023

20© April 21, 2023

Defining Usability

21© April 21, 2023

What do we do?

What do you want to do?

Why Are You Here?Defining Usability

22© April 21, 2023

We want to make things better.

But how do you measure “better?”

Can usability be defined?

Can Usability Be Defined?Defining Usability

23© April 21, 2023

The UX HoneycombDefining Usability

24© April 21, 2023

Lights are on.

Turn off the light.

What Makes Something Usable?Defining Usability

25© April 21, 2023

Turn the knob to move the arrow to the right.

Are There Universal Rules or Stereotypes?Defining Usability

26© April 21, 2023

Make the shower temperature “just right.”

Interaction Not Always UniversalDefining Usability

27© April 21, 2023

How About More Complicated Interfaces?Defining Usability

28© April 21, 2023

Can Interfaces Be Learned?Defining Usability

29© April 21, 2023

Emphasis on Usability in Many FieldsDefining Usability

30© April 21, 2023

No measures exist.

Why?

What can we measure?– Success/Fail– Time– Accuracy– Satisfaction– ...

Naysayers: You Cannot Measure UsabilityDefining Usability

31© April 21, 2023

From UPA:– Usability is an approach to product development that

incorporates direct user feedback throughout the development cycle in order to reduce cost and create products and tools that meet user needs.

From Krug’s “Don’t Make Me Think:”– Usability really means making sure that something works well…

so that a person of average ability and experience can use the thing for its intended purpose without being hopelessly frustrated.

Definitions of UsabilityDefining Usability

32© April 21, 2023

From ISO 9241-11:– The extent to which a product can be used by specified users to

achieve specific goals with effectiveness, efficiency, and satisfaction in a specific context of use.

• Effectiveness: Being able to complete the task• Efficiency: Amount of effort required to complete task• Satisfaction: Degree of happiness or fulfillment while

performing task (or rather the absence of pain and frustration)

Definitions of UsabilityDefining Usability

33© April 21, 2023

There is just too much going on to get reliable data—NOT!

Measurement must:– Be observable– Be quantifiable– Have sound control– Have understanding of context

Naysayers: Usability Data Are Too NoisyDefining Usability

34© April 21, 2023

Not All Measures Are Created EqualDefining Usability

Bad Inappropriate Good

35© April 21, 2023

Performance metrics

Issues-based metrics

Self-reported metrics

Behavioral and physiological metrics

Combined and comparative metrics

Types of Usability MetricsDefining Usability

36© April 21, 2023

Task success– Binary: Success/fail– Levels of success: Complete success/partial success/…/failure

Time on task– Actual time– Thresholds

Errors– Actual number

Efficiency– Deviations from optimal path– Lostness (Smith, 1996)– Combination of task success and time (NIST, 2001)

Learnability– Metrics above taken from trials within same session, with breaks, between

sessions

Performance MetricsDefining Usability

37© April 21, 2023

Usability issues– Number of issues found– Percentage of participants who found

an issue

Severity ratings– High severity– Medium severity– Low severity

Issues-Based MetricsDefining Usability

59September 23, 2009

Usability Issue: The Rapid Part Add feature does not recognize part numbers if dashes are not entered. The items are added to the cart but they are not recognized.Recommendation: The search functionality should be able to handle part numbers without dashes. If it can’t, instructions should say that the part number has to include dashes. Alternatively, provide three input fields, separated by dashes. Automatically advance to the next field if maximum number of characters have been entered.

Usability Issue: The Rapid Part Add gets visually lost among all the other elements on the page and may be difficult to notice.Recommendation: Put the Rapid Part Add in a shaded box to make it stand out.

Good Practice: Rapid Part Add allows users to quickly enter their shopping list if they know the part numbers of the items without having to search for the items one by one. Also, users’ own part numbers are accepted in the Rapid Part Add tool, which is helpful.

Usability Issue: There is no error check when adding parts through Rapid Part Add. If users make an error in the part number, they will not have an opportunity to correct it.Recommendation: Provide users with feedback If the part number is not recognized. H

L

M

Presenting Results

Example

HIGH

MEDIUM

LOW

38© April 21, 2023

Post-task ratings– Likert

Post-session ratings– Likert

System Usability Scale (SUS) (Brook 1996)– Likert

NASA TLX– Likert

Specific attribute– Agreement

Answers to open-ended questions

Self-Reported MetricsDefining Usability

39© April 21, 2023

System Usability Scale (SUS)Defining Usability

40© April 21, 2023

Verbal and non-verbal– Positive / negative comments

Facial expressions– Computerized analysis

Eye movements– Attention

Pupil diameter– Workload

Skin conductance and heart rate– Stress

Behavioral and Physiological MetricsDefining Usability

41© April 21, 2023

Usability scorecards– Comparison– Harvey balls

– Radar charts

Combined and Comparative MetricsDefining Usability

42© April 21, 2023

Again, Metric Selection is Non-TrivialDefining Usability

Bad Inappropriate Good

43© April 21, 2023

Consider a Horse Race: Which Measure is Good?Defining Usability

44© April 21, 2023

Different Conditions: Yes, That is SnowDefining Usability

45© April 21, 2023

Appropriate Metrics Depend on Many FactorsDefining Usability

46© April 21, 2023

Objective: – Tons of user complaints that volume is too soft.– We have three new versions and want to know if the problem has

been fixed.

Measure: – Rating of the perceived sound level

1 2 3 4 5 6 7

Exercise: Measurement ScaleDefining Usability

47© April 21, 2023

Evaluating UsabilityDefining Usability

48© April 21, 2023

Testing with Users

49© April 21, 2023

Inspection methods = no users, only UX experts– Heuristic evaluation– Expert evaluation– Competitive evaluation– Cognitive walkthrough– Pluralistic walkthrough

Methods involving users– User testing (lab, longitudinal)– Eye tracking– Focus groups– Surveys– Ethnographic research

Methods with No Users vs. Methods with UsersTesting with Users

50© April 21, 2023

Artificial situations

Results cannot prove product will work

Participants not necessarily actual target market

Might not be the right thing to do

Others? (Beyond Rubin?)

Sample size considerations

Feature coverage

Limitations of Testing (Rubin … Lab Testing)

Testing with Users

51© April 21, 2023

Walk Up And UseTesting with Users

52© April 21, 2023

FrustrationsTesting with Users

53© April 21, 2023

Test Interval Long Enough?Testing with Users

54© April 21, 2023

Hypothesis testing

Participants are randomly chosen

Tight controls in place

Control groups employed

Sample size sufficiency

UX Research = Experimental Method (Rubin) Testing with Users

55© April 21, 2023

Study Preparation

56© April 21, 2023

Understand study objectives

Determine the type of test

Identify target users

Understand the stimuli

Create the test plan

Develop the screener

Determine analysis and expect results

Develop the moderator’s guide – next week

Test PlanningPreparation for a Usability Study

57© April 21, 2023

Identifying the research objectives will help you select appropriate approach and methodology for the study.

Ask stakeholders:– What are you hoping to learn from this study?– What decisions will you need to make based on the outcome of

the study?

Understand Research ObjectivesPreparation for a Usability Study

58© April 21, 2023

Is it a formative or summative test?

Determine the Type of TestPreparation for a Usability Study

59© April 21, 2023

Earlier in product life cycle

Diagnostic

Qualitative

Study goal: Determine key strengths and weaknesses of the system

Ultimate goal: Improve the user experience

E.g., “We want to improve the website we are developing, so that the users can easily find what they are looking for.”

Determine the Type of Test: FormativePreparation for a Usability Study

60© April 21, 2023

Later in product life cycle

Verification; quantitative

Study goal: Determine how the system compares to usability standards, benchmarks or competitors

Ultimate goal: To make sure the product is ready for launch (can involve making legally defensible statistical, marketing, or regulation claims)

E.g., “We need to be sure that the new drug label will not increase the number of dispensing errors.”

Results should often be statistically valid.

Determine the Type of Test: SummativePreparation for a Usability Study

61© April 21, 2023

Difference in terminology between many professionals and Rubin:– Formative tests: Include Rubin’s exploratory and

assessment tests (1st edition)– Summative tests: Include Rubin’s validation and

comparison tests (1st edition)

In his 2nd edition, Rubin calls exploratory tests “formative” and assessment tests “summative.”– This is not a typical classification and we will use this one

Project 2: Formative study

Project 3: Summative study

Determine the Type of TestPreparation for a Usability Study

63© April 21, 2023

Define the target user groups that should be represented in the study.

Identify Target UsersPreparation for a Usability Study

64© April 21, 2023

Become familiar with the product or interface of interest and understand its capabilities and constraints.

Explore more than just the product itself = understand the whole experience:– Look at the packaging.– Read the manual and other

accompanying materials.

If the stimulus is a prototype:– Make sure it works.– Suggest quick fixes prior to

testing if there are glaring usability issues.

Review the StimuliPreparation for a Usability Study

65© April 21, 2023

A test plan is a document that helps:– Organize all information about a study. – Communicate this information to the research team and study

stakeholders.

Test plan should be clear and detailed.– Bulleted lists and diagrams help convey important details in a

clear manner.

Create the Test PlanPreparation for a Usability Study

66© April 21, 2023

Objectives of the study broken up into specific questions that the study has been designed to answer.

Description of the stimuli, including version or model numbers where appropriate.

Description of the participants with an indication of strict requirements and allowable variances.

A Good Test Plan Contains…Preparation for a Usability Study

67© April 21, 2023

Description of the methodology, including:– The procedure (e.g., use

of a think-aloud protocol and within- or between-groups study design)

– Measures used (e.g., time on task, ratings)


68© April 21, 2023

Requirements for equipment needed for testing (e.g., size of TV or computer screen, computer operating system, and types of cameras – face shot, desktop shot etc.).

Pictures/sketches or description of the lab setup denoting how the stimuli should be arranged, where the moderator and participant should sit, and where the camera(s) should be set up. – Esp. beneficial if complex

setup with multiple elements.


69© April 21, 2023

The project logistics, including:– Test dates and times– Test location(s)– Deliverable deadlines for

the stakeholders and the research team

– Etc.


70© April 21, 2023

… and iterate it based on their feedback.

Have Stakeholders Review the Test PlanPreparation for a Usability Study

71© April 21, 2023

The Screener (or Screening Questionnaire) is a document according to which the study participants will be recruited.

Examples of screener elements?

Internal recruit vs. external recruit

Tasks of the recruiter– Screen– Schedule– Follow-up

List vs. database recruit

Develop the ScreenerPreparation for a Usability Study

What Makes A Screener Effective? It is clear for recruiters, clients, and respondents to understand.

– Participant should not have to ask “what do you mean?” – Questions should be able to be answered with minimal thought

and quickly.• The more prospective participants they “think about it” – the

more likely they will overthink it and not give an honest response.

– It should be easy enough for respondents to accurately remember details to answer questions.

– Summarize quotas, termination criteria, and nice-to-haves in a separate section.

Preparation for a Usability Study

© April 21, 2023 74

What Makes A Screener Effective? Not overly restrictive

– Strive for the least restrictive screener that will get the respondents you need to satisfy test objectives. This will reduce recruiting costs.

– Nested recruits make recruits restrictive.– Recruits of users of specific features or devices tend to be more

restrictive.

Provides directions to the facility.

Require tangible artifacts– State-issued ID– Related products (phone, medicine box, etc.)


© April 21, 2023 75

What Makes A Screener Effective? Efficient

– Aim for max 20 questions.– Minimum number of questions to effectively screen respondents.

Contains all the details– Reminder to recruit talkative participants– Phone numbers on the day of testing and before testing– Let participants know they will be videotaped– Under 16 consent form requirements

Enough soak time - appropriate for the difficulty of the recruit– Standard soak time of 2 weeks?– We should create a list of specific types of studies that need

long soak times


© April 21, 2023 76

77© April 21, 2023

Avoid yes/no questions unless absolutely necessary.– They reveal a pattern and can be considered leading.

• We want honest, knee jerk responses that reflect current behavior.

Avoid questions that allow one to figure out what you’re looking for and answer accordingly.– Self-identification questions – Have participants state the activities that they perform, instead of

having the recruiter read a list.

Questions To AvoidPreparation for a Usability Study

Test Plan Creation Detailed, clear, and explicit test plan is the key to a successful

global study.– It improves communication between teams.– It sets expectations.– It links the methods and procedures to study objectives.


© April 21, 2023 78

79© April 21, 2023

Control what you can control.

Lesson Learned: Anticipate outcomes and their implications.– Spend time to think hard about research.– Effort spent early will make the study better and might prevent

another study.

Ready, Fire, Aim…?!?!Preparation for a Usability Study

82© April 21, 2023

As a group, select a product / interface to test, for example:– Application on a mobile phone– Medical device (e.g., blood pressure monitor, glucose meter)– Small home appliance (e.g., coffee maker)– Instructions to a game / puzzle– Utility bill– Software application or a website (only if you must)

Bring it with you to class next week and on Oct 7th & 14th.

Project components:– 2a: Test Plan – Due Next Week– 2b: Screener, Moderator’s Guide– 2c: Data collection– 2d: Final Report

Project 2: Formative Usability StudyAssignment for Next Week

83© April 21, 2023

Develop a test plan for testing your chosen product / interface.

The test plan should describe the following:– Objectives of the study– Stimuli– Participants– Procedure– Measures – Equipment and Lab Setup– Logistics

Project 2a: Test Plan – Due Next WeekAssignment for Next Week

84© April 21, 2023

Grading criteria:– Are all the components included in the test plan? (see previous

slide)– Are the objectives realistic?– Are the participants described in enough detail?– Does the methodology (including measures) appropriately

address the objectives of the study?– Overall, is the test plan clear and precise?– Is the test plan well structured and laid out?

• Proper alignment• Consistency of fonts, sizes, spacing, phrasing etc.

Project 2a: Test Plan – Due Next WeekAssignment for Next Week

85© April 21, 2023

Reading for Next Week Handbook of Usability Testing by Rubin:

– Chapter 4: Skills for Test Moderators– Chapter 7: Find and Select Participants– Chapter 8: Prepare Test Materials


86© April 21, 2023

Recap of To-Dos for Next Week As a group, select a product / interface to test.

– Bring it to class on Sep 30th.

Develop the test plan for formative usability testing of the selected product / stimulus.– Submit through COL by midnight Sep 30th (in-class students) or

Oct 4th (DL students).

Do your reading.


HCI460: Week 3 Lecture September 23, 2009. 2 © October 12, 2015 Project 1 recap Q & A Research...

Documents

Transcript of HCI460: Week 3 Lecture September 23, 2009. 2 © October 12, 2015 Project 1 recap Q & A Research...