Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was...
-
Upload
stanley-holland -
Category
Documents
-
view
215 -
download
0
Transcript of Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was...
![Page 1: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/1.jpg)
Usability and Human Factors
Usability Evaluation Methods
Lecture b
This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003.
![Page 2: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/2.jpg)
Usability Evaluation MethodsLearning Objectives
2
• Conduct a cognitive walkthrough (Lecture b)
• Design appropriate tasks for a usability test (Lecture b)
• Describe the usability testing environment, required equipment, logistics, and materials (Lecture b)
Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 3: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/3.jpg)
Cognitive Walkthrough (Polson et al, 1992)
3Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
• Kind of cognitive task analysis• Assesses system usability• Criteria focuses on cognitive processes
needed to perform tasks• Identifying sequences of actions and
subgoals to successfully complete a task• Assigning causes to usability problems• Are the cues provided by the interface
sufficient to perform task?
![Page 4: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/4.jpg)
Cognitive Task Analysis (CTA)
• Tools and techniques for describing knowledge & strategies required for task performance– Hierarchical decomposition of goals and
component tasks
• Objective: – Yield information about the knowledge,
thought processes, and goal structures that underlie observable task performance
4Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 5: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/5.jpg)
Why Do a CTA?
• Develop theory of competent performance• Understand invariant features of a task
– There are invariant performance characteristics of any class of tasks
• Understand process of skill acquisition• Training and instructional resources
– e.g. manuals and tutorials• Develop methods for usability testing
– Design– Coding scheme for data analysis
5Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 6: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/6.jpg)
Cognitive Walkthrough Step 1: Preparations
6Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 7: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/7.jpg)
Step 2: Walkthrough Process
7Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 8: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/8.jpg)
Step 3: Explicate Sources of Potential
Problems
8Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 9: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/9.jpg)
ATMs: Goal Structure
9Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 10: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/10.jpg)
A Partial Walkthrough: ATMGoal: Obtain $80 Cash from Checking
Account
10Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 11: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/11.jpg)
Cognitive Walkthrough: Measure Glucose
11Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 12: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/12.jpg)
CW: Blood Glucose 2
12Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 13: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/13.jpg)
CW: Blood Glucose 3
13Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 14: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/14.jpg)
CW Glucose Results
14Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 15: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/15.jpg)
Usability Testing
• Gold standard for usability evaluation• Set of techniques to collect empirical data
– while observing representative end users using the system under study to perform representative tasks
• Video-recorded• Provide information that can lead to systems that:
– Easy to learn and use– Satisfying to use– Provide utility and functionality that are valued by the
target population– Characterize task-specific competencies
15Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 16: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/16.jpg)
Think-Aloud Protocol
• Method broadly used in cognitive research and usability testing
• User verbalizes his/her thoughts while performing a task– Report the contents of working memory– Session is audio and/or video recorded
• Transcript of think aloud is coordinated with video analysis
16Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 17: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/17.jpg)
Selection of Representative Users
• Users may differ:– including age, education, gender, computer
experience, etc.
• Select subjects based on relevant criteria (e.g., age, education)
• Fully representative not possible
• Convenience sample is less desirable
17Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 18: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/18.jpg)
Development of Test Plan
• Outline task and procedure– Informed by objectives– Prior testing– Constrained by time and setting– Ethical and IRB issues
• Exploratory – Characterize potential problems
• Controlled Experiment– Comparing 2 Interfaces
18Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 19: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/19.jpg)
Role of Researcher
• Neutral Observer vs. Active Participant
• Researchers may play a more interactive role in field testing – Guide the subject as necessary
• Skilled user will need a minimum• Novice may need step-by-step instructions
– Autonomy Rule: No more guidance than necessary
19Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 20: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/20.jpg)
Field Usability Testing
• Hybrid Method: Lab and Ethnography/ Field Study
• Naturalistic setting– Numerous constraints
• Proscribed set of tasks– Quasi- Experiment
• Video analysis is key– Intrusive
20Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 21: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/21.jpg)
Video-Analytic Usability Testing on Location: Old School
Microsoft Clipart
21Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 22: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/22.jpg)
Software-based Video Analysis
• Provides a video of all screen activity • Captures user via a webcam• Logs a wide range of events and system
interactions including mouse clicks, text entries, web-page changes and windows dialogue events (e.g., saving a document, selecting among a set of choices).
• Morae state of the art usability testing software
22Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 23: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/23.jpg)
Morae Video Analytic Usability Software
Khan, S.A., Ancker, J.S., Li, J., Kaufman, D., Hutchinson, C., Cohall, A., Kukafka, R. (2009)
23Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 24: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/24.jpg)
Data Analysis: Transcript
• Working document for video analysis
• Verbatim and Time Stamped– Every 10 to 30 seconds
• Iteratively modified document as coding categories become refined
• Add field notes and observations to the transcript
24Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 25: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/25.jpg)
Video Analysis: Granularity
25Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 26: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/26.jpg)
Macro-AnalysisMarch 18, 2002
I: … You can choose not to answer any questions. May I ask how old are you?
P: 74
I: What is or was your professional occupation.
P: I have always worked in stores as a salesperson
I: How long have you lived in this country?
P: 48, 49 years
I: If I may ask what level of education do you have?
P: I finished high school, but in Colombia. Before I got married, then I came here after I got married, a couple of years afterwards.
I: How long have you known that you have diabetes.
P: A couple of years. About two years or three. I have never felt that I have diabetes.
Profile:
74 years of age
Female
Native Spanish speaker
Originally from Colombia
48-49 years in US
High school education
Worked as salesperson in stores
Has two sons, one in NJ
Never used computer prior to IDEATel
Diabetes History:
Had it for 2-3 years
Did not have monitoring device prior to IDEATel
1y.2months in program
Good health/Has asthma
A lot of fluctuation in glucose values
Does not look at them over a period of time, watches it one day to the other.
Generally good memory remembers values for previous days
1.1 Table: (Kaufman, et al., 2003).
26Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 27: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/27.jpg)
Observations: Sending Results
27Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 28: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/28.jpg)
Micro-AnalysisTime
Dialogue and Conversational Code
Action Code System Response
Comments
7:28 R: Can you explain what you see now?P: It is calling the Internet. There it turns off, and leaves me with the system.
Body Position Change (BPC): Patient moves head, looks from side to sideGesture: Points at screen with index fingerAction: Takes mouse and clicks security screen awayBPC: keeps hand on mouse for a couple of seconds, then offAction: Takes mouse and clicks identification screen away.
Security screen/dialogue box appearsSec. Screen goes awayPassword screen appears
The task is to access the web and patient immediately assumes the control
7:54 R: So far, so good. Action: Takes hand off mouse, hunches over keyboard, and puts in password.Action: Takes mouse and clicks OK button and hand off mouse.Action: Takes mouse in hand immediately as DM page comes up.
Password screen transitions to Diabetes Manager Page
Patient performs all actions without any prompting.
1.2 Table: (Kaufman, et al., 2003).
28Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 29: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/29.jpg)
Triangulate
29Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 30: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/30.jpg)
Usability Evaluation Methods Summary
• The value of usability evaluations in healthcare contexts has been well established
• Wide range of methods which vary in terms of their advantages and disadvantages
• Lectures illustrated how to use someof these methods including the cognitive walkthrough and usability testing
30Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
![Page 31: Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.](https://reader035.fdocuments.net/reader035/viewer/2022062717/56649e3f5503460f94b307a7/html5/thumbnails/31.jpg)
Usability Evaluation MethodsReferences – Lecture b
References
1. Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. John Wiley & Sons, New York, NY.
2. Polson, P., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741–773.
3. Preece, J., Rogers, Y., & Sharp, H. (2007). Interaction Design: Beyond Human-Computer Interaction (2nd ed.). West Sussex, England: Wiley.
Images
Slide 21: Microsoft Clipart
Slide 23: Khan, S.A., Ancker, J.S., Li, J., Kaufman, D., Hutchinson, C., Cohall, A., Kukafka, R. (2009). GetHealthyHarlem.org: developing a web platform for health promotion and wellness driven by and for the Harlem community. AMIA Annu Symp Proc.317–21.
Charts, Tables & Figures
1.1 Table: Kaufman, D.R., Patel, V.L., Hilliman, C., Morin, P.C., Pevzner, J, Weinstock, Goland, R. Shea, S. & Starren, J. (2003). Usability in the real world: Assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics, 36, 45-60.
1.2 Table: Kaufman, D.R., Patel, V.L., Hilliman, C., Morin, P.C., Pevzner, J, Weinstock, Goland, R. Shea, S. & Starren, J. (2003). Usability in the real world: Assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics, 36, 45-60.
31Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b