b
b
© Know-Center GmbH, www.know-center.at
Usability Evaluation
Vedran Sabol (with input from Belgin Mutlu and Cecilia di Sciascio)
Web Technologies, 11-11-2019
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Overview
▪ Part I: Usability
▪ Definition and Methods
▪ Usability Metrics
▪ Data collection
▪ Part II: Evaluation task annoucement
▪ Optional! ▪ You can earn up to 5 additional points
▪ Visualizer Tool
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a
specified context of use
Usability Definition
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Why evaluating Usability?
▪ Helps to find the problems when they are easy and cheap to fix (early stage)
▪ Saves time, money and other precious resources
▪ Reduces the risk of building the product in the wrong way
▪ Helps to understand user’s success and time spent to complete tasks
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Tests
▪ Formative Tests: Finding and fixing usability problems (How?)
▪ forms and shapes a UI design
▪ Summative Tests: Describing the usability of an application using metrics (How much?)
▪ about measuring (e.g. time) and counting (e.g. success rate)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Formative Tests
▪ Helps to “form” the design for a product/service
▪ Involve evaluating during development
▪ Iterative
▪ Methods:
Heuristic evaluation: expert evaluates a UI against 10 well-known usability heuristics
Thinking-aloud testing: user comments while performing a task to identify issues
Cognitive walkthrough: evaluates how easy it is to learn using a system for new users
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Example - Nielsen’s Heuristics
1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Example - Nielsen’s Heuristics
1. Visibility of system status
• system should keep users informed about what is happaning
• provide appropriate feedback, within reasonable time
2. Match between system and the real world
• system should speak the users' language
• use words, phrases and concepts familiar to the user (no system-oriented terms)
• follow real-world conventions
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Example - Nielsen’s Heuristics
3. User control and freedom
• users choose functions by mistake, need a clear "emergency exit" (no complicated procedures)
• e.g. undo and redo
4. Consistency and standards
• use consistent vocabulary across the tool (no wondering whether different words, situations, or actions mean the same thing)
• follow platform conventions (fonts, date format etc.)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Example - Nielsen’s Heuristics
5. Error prevention:
• design to prevent problems from occurring (instead of error providing messages)
• in error-prone conditions check for a confirmation
6. Recognition rather than recall
• minimize the user's memory load
• make objects, actions, and options visible (instead of having to remember them)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Example - Nielsen’s Heuristics
7. Flexibility and efficiency of use
• Accelerators may speed up common interactions• not seen by novice users
• allow users to tailor/automate frequent actions
8. Aesthetic and minimalist design
• do not show information which is irrelevant or rarely needed
• useless information competes with the relevant info for attention and diminishes readability
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Example - Nielsen’s Heuristics
9. Help users recognize, diagnose, and recover from errors:
• error messages in plain language (no codes)
• precisely describe problems, suggest solutions
10. Help and documentation
• design for intuitiveness, but
• make documentation easy to access, search, and read (e.g. lists of concrete steps with screenshots)
• focused on the user's task
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Summative Tests
Benchmark test
used to compare results with usability requirements efficiency of use
Intuitiveness
low perceived workload etc.
User tasks need to be clearly defined, e.g. identify 5 most relevant documents on web technologies (max.
2 min time)
Find an author with most publications on web technologies (3 min time)
objective measurement of performance of a single design
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Summative Tests
Comparative test, e.g.
Comparison of a current with a prior version
Comparison of competing products
different sets of users can work with each product (between-subject design)
same users can attempt tasks on all products (within-subjects design) Preferred when small num. of participants perform small
tasks
may not always be possible
Objective comparison of the performance of two or more alternative designs
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Sample Size
The sample of users should represent the populationabout which we intent to make a statement
To draw conclusions about different types of users, all groups should be represented in your sample
e.g. include novice and experienced users in the sample
Example: 1936 Literary Digest Presidential Poll
2.4 million responses but incorrectly predicted the winner
Problem was not the sample size but of representativeness
Responders tended to have higher incomes and education level-not representative for ultimate voters
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Data Collection
Lab-based moderated sessions
• Moderator observes and interacts with users when performing the tasks
• Expensive and time consuming
Unmoderated Sessions
• Users perform task on their own computer
• Moderator observes and records their behavior
Crowd-based studies
• No moderator observes, user behavior is recorded
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Metrics
Effectiveness: The accuracy and completeness with which users achieve specified goals
Efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals
Satisfaction: The comfort and acceptability of use
[Source: https://usabilitygeek.com/usability-metrics-a-guide-to-quantify-system-usability/ ]
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Metrics for Effectiveness:Completion Rates
Most fundamental of usability metrics
Collected as a binary measure of task success (coded as 1) or failure (coded as 0)
Number of tasks 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑑 𝑠𝑢𝑐𝑐𝑒𝑠𝑠𝑓𝑢𝑙𝑙𝑦
𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑎𝑠𝑘𝑠 𝑢𝑛𝑑𝑒𝑟𝑡𝑎𝑘𝑒𝑛x 100%
[Source: https://usabilitygeek.com/usability-metrics-a-guide-to-quantify-system-usability/ ]
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Metrics for Effectiveness:Errors
Unintended action, slip, mistake, omission a user makes (or encounters) while working on a task
Provide diagnostic information on, why users are failing tasks
Can be analyzed as binary measures
1 for user encountered an error
0 for user who did not
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Metrics for Efficiency:Task Time
How long user spends on an activity
Amount of time it takes to successfully complete a predefined task scenario
𝑗=1𝑅 𝑖=1
𝑁 𝑛𝑖,𝑗
𝑡𝑖,𝑗
𝑁𝑅
[Source: https://usabilitygeek.com/usability-metrics-a-guide-to-quantify-system-usability/ ]
N = Total number of tasks
R = Number of users
𝑛𝑖,𝑗= Result of task i by user j, 1 if successfully completed, 0
otherwise
𝑡𝑖,𝑗 = Time spent by user j to complete the task i: if the task
not successfully completed, the time is measured till user quits
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Metrics for Efficiency:Overall Relative Efficiency
Ratio of time taken by the users who successfully complete the task in relation to the total time taken by all users
𝑗=1𝑅 𝑖=1
𝑁 𝑛𝑖,𝑗𝑡𝑖,𝑗
𝑗=1𝑅 𝑖=1
𝑁 𝑡𝑖,𝑗x 100%
[Source: https://usabilitygeek.com/usability-metrics-a-guide-to-quantify-system-usability/ ]
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Usability Metrics: Satisfaction Ratings
Questionnaires that measure the perception of the ease of use of a system
After a task (post-task questionnaires)
At the end of a usability session (post-test questionnaires)
Outside of a usability test
There are standardized questionnaires
Own questions are welcome
might have to be validated by psychometrics tests
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Post-Task Questionnaires
To provide insight into task difficulty as seen from the participant’s perspective
Typically consisting of a small number of questions
Often take the form of Likert scale ratings, e.g.
worst 1 … 7 best
Examples:
ASQ: After Scenario Questionnaire (3 questions)
SMEQ: Subjective Mental Effort Questionnaire (1 question)
UME: Usability Magnitude Estimation (1 question)
NASA-TLX: NASA’s task load index measures the subjective task workload (6 questions)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
NASA-TLX
Mental Demand
How mentally demanding was the task?
Physical Demand
How physically demanding was the task?
Temporal Demand
How hurried or rushed was the pace of the task?
Performance
How successful were you in performing what you were asked to do?
Effort
How hard did you have to work to accomplish your level of performance?
Frustration
How insecure, discouraged, irritated, stressed, and annoyed were you?
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Post-Test Questionnaires
To measure participants’ impression of the overall ease of use of the system being tested
Examples:
SUPR-Q: Standardized User Experience Percentile Rank Questionnaire (13 questions)
CSUQ: Computer System Usability Questionnaire (1 question)
QUIS: Questionnaire For User Interaction Satisfaction (24 question)
SUMI: Software Usability Measurement Inventory (50 question)
SUS: System Usability Scale (10 questions)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Positive Version of the Software Usability Scale (SUS)
[Source: https://link.springer.com/chapter/10.1007/978-3-319-20886-2_20 ]
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Interaction Logging
What can be logged?
Occurrence of interactions with UI components:
e.g. click on tag in tag cloud, click to open a document, drag mouse to move a slider etc.
Record: user id, timestamps, action
Task success
1=success, 0=failure
Task time
in seconds (or milliseconds)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer
Web-application for data visualization URL: https://visualizer.know-center.tugraz.at/
Uses CSV files as datasets Upload from a local file or a URL
Simple data processing methods Cleaning, filtering, aggregation, computing columns
Multiple Visualizations Coordinated multiple views
UI similar to BI industry leaders (Tableau or PowerBI)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer - Features
Secure – data management locally in the browser (compressed)
Smart – automatically selects suitable visualizations for selected data
Can be used stand-alone or integrated into existing web-pages
Customizable – by changing the look of the platform
Configurable – by creating and sharing “views” on the data
Collaborative – synchronize user interactions over multiple clients (even mobile)
Extendable – by uploading own visualizations
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer – Upcoming Features
Personalized Assistant for Interactive Data Analytics systems collects user interactions (e.g. mouse clicks)
learns how users analyze data (computes models)
Goal: guide users towards useful data insights
Approach: assist users when analyzing data based on previous user behavior e.g. user first selects some data fields
the system recommends (potentially useful) analytical workflows (i.e. a sequence of algorithmic and visual steps)
user can accept a recommendation automatically executes the workflows to create
visualization(s)
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer – Upcoming Features
Personalized Assistant for Interactive Data Analytics
Needs click data to learn how users analyze data
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer - Evaluation
Student: execute a few analytical tasks for a data set gain an overview
discover trends
identify correlations
find outliers
understand causal relationships
etc.
Fill out a questionnaire
Write a 2-page report (heuristic analysis)
It is all voluntary
© Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Visualizer - Evaluation
But, you can earn a few extra points!
Perform tasks and fill out a questionnaire duration 15-20 minutes
1 extra point
Write a 2-page report
Heuristic evaluation
5 extra points
Details will be presented during the last lecture of the course “Visualization in the Web”, to be held on 09.12.2019
Top Related