Subjective questionnaires

Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:

EXCHANGING KNOWLEDGE AND EXPERIENCES

Usability TestingSubjective Questionnaires

Cristina Cachero

This project has been funded with support from the European Commission under the Lifelong Learning Programme



Inquiry method: you ask your users about what they do, but you do not observe it directly.

There are several proposals that have shown enough internal and external validity, and that can be applied at different times during the performance test

After all the tasks have been finished (test-level questionnaires)

Before/after each task is performed (task-level questionnaires)

Also, questionnaires may include open questions that can be codified and analyzed.


2

Subjective Questionnaires



Test-level (post-test) questionnaires (1/2): CSUQ: Computer System Usability Questionnaire [Lewis 1995]: 19 sentences,

all positive. They measure four dimensions: system utility, information quality, interface quality and general satisfaction. It was devised to be administered off-line

PSSUQ: Post-Study System Usability Questionnaire: variation of CSUQ to be administered in person

QUIS: Questionnaire for User Interface Satisfaction. 27 sentences divided in five groups: general reaction, screen, terminology/system info, learning, system capabilities.

EUCS : End-user computing satisfaction [Abdinnour-Helm 2006]. 12 items representing five dimensions: content, accuracy, format, ease of use and timeliness. All the items are combined into a global EUCS measure. 5-point scale (Almost never… almost always)

USE: Usefulness, Satisfaction and Ease of Use [Lund 2001]. 30 items divided in four categories: utility, satisfaction, ease of use and ease of learning. For each one, seven-point scale items..


3




Test-level (post-test) questionnaires (2/2): SUS: Systems Usability Scale [Brooke 1996]: 10

questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability

Net promoter Scores (NPS) ©: A single score regarding the fidelity of the user

SUMI [Kirakowski 1996] Questionnaires that rank sites (based on proprietary DB)

SUPR-Q: UX questionnaire . Includes usability (4 items), credibility (trust, value & comfort, 5 items), loyalty (2 items, one of which is the Net Promoter Score) and Appearance (2 items). Five-point scale from strongly disagree to strongly agree except for Item 10 (net promoter question)

WAMMI, ACSI, Opinion Lab

Proprietary (beware of reliability and validity!!)


4


Source: ISO 13407




5

Post-test SQ: EUCS [Abdinnour-Helm 2006].




6

Post-test SQ: USE [Lund 2001]

http://hcibib.org/perlman/question.cgi?form=USE



SUS (Systems Usability Scale) [Brooke 1996] 10 questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability (it does not distinguish among different components). It is the most reliable [Tullis and Stetson, 2004]


7

Post-test SQ: SUS [Brooke 1996]

The mean SUS value is 66%, the 25 percentile is 57% and the 75 percentile is 77%. This means that we should get a SUS greater than 80% (taking into account confidence intervals) to be reasonably sure of the global satisfaction of our users.



Activity: Read the original SUS paper, published by Brooke in 1996. You have the paper available in the Moodle platform.


8

Post-test SQ: SUS




9

Post-test SQ: SUS: Exercise

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1. I think that I would like to use this system frequently

2. I found the system unnecessarily complex

3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system

5. I found the various functions in this system were well integrated

6. I thought there was too much inconsistency in this system

7. I would imagine that most people would learn to use this system very quickly

8. I found the system very cumbersome to use

9. I felt very confident using the system

10. I needed to learn a lot of things before I could get going with this system

Strongly Strongly Disagree Agree




10

Post-test SQ: SUS: Score?

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5














11

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5











Post-test SQ: SUS. Steps Reverse items 2,4,6,8,10 Sum positions for each item (0 to 4) Multiply by 2.5

4

1

1

4

1

2

1

1

4

3

TOTAL SCORE: 22

SUS SCORE: 22*2.5=55




12

Post-test SQ: SUS. Threshold value

Which should be our SUS objective?




13

Post-test SQ: SUS. Adaptation

The SUS vocabulary can be adapted to the idyosyncrasy of the particular system

Example: OHIM: system->web site




14

Post-test SQ: SUS. Calculator

Calculator: http://www.measuringux.com/SUS-scores.xls



Net Promoter Scores: measure obtained through a single question (loyalty of the user to the app):

How likely is it that you’ll recommend this product to a friend or colleague? (0..10)

Three segments: Promoters: Responses from 9 to 10 Passives: Responses from 7 to 8 Detractors: Responses from 0 to 6

Promoter score (-100..+100): %Promoters-%Detractors


15

Post-test SQ: Net Promoter Score



Questionnaires that provide rankings to compare your results against the results from other similar apps

SUPR-Q (Sauro) WAMMI (www.wammi.org) (SUMI

successor) ACSI (www.TheACSI.org): particularly

interesting form government websites OpinionLab (www.OpinionLab.com)


16

Post-test SQ: Rankings




17

SQ: Rankings. SUPR-Q (4 sub-scales)

NPS




18

Post-test SQ: Rankings. SUPR-Q. SUPR-Q Score: Sum all items + ½ item 10




19

Post-test SQ: Rankings. SUPR-Q. Besides the values for each factor, SUPR-Q is backed on a proprietary DB that allows to generate a ranking of percentiles

E.g. your app belongs to percentil 75 regarding usability, what means, it is among the best 25%. The SUPR-Q usability factor has a strong correlation with a SUS score, r = .96. p < .001, meaning just four questions account for 93 percent

of the variation in SUS (.96 squared). These questions are a good substitute for SUS on websites Validity and reliability: http://www.suprq.com/




20

Post-test SQ: Rankings. WAMMI

http://www.wammi.com/samples/index.html Results are divided into five areas:

attractiveness, controllability, efficiency, helpfulness and Learnability, plus an overall usability score.

The scores are standardized (from comparison to their reference database), so 50 is medium and 100 is perfect.




21

Post-test SQ: Rankings. WAMMI

Sample graphic showing how a give website positions in reference to average scores in each axis



Circle five words that describe what you think about this design:

What are the three things you like best and least

about the Web site? _____________________________________________ ______________________________________________ ______________________________________________


22

Post-test SQ: proprietary (OHIM 1/2)

CuteStableResponsiveFriendlyHelpful

ReputableApproachableReliableClutteredGood

ConfidentTrustworthyCurrent and coolService orientedBoring

Easy to useConfusingComfortableAnnoyingInformativeOut-of-date

High-techSensitiveSecureStraightforwardAmateurish



If you could make one significant change to this Web site, what change would you make?

(Ask the participant if he is using/knows other IP websites) How do you find the site in comparison to other IP web sites?

Do you feel this site is current? Why? If you were to describe this site to a colleague in a sentence or

two, what would you say? Do you use the current OHIM web site? If Yes then Do you

think that the new OHIM web site clarity of structure is: Better, Same As, Worse or Don’t know, than the current one?

Do you have any other questions or comments about the Web site or your experiences with it?


23

Post-test SQ: proprietary (OHIM 2/2)



Expectation Rating After Scenario Questionnaire (ASQ) Awareness-Usefulness gap


24

Task level (Post-Scenario) SQ



Pre-task: How easy/difficult do you thing that the following task is going to be?

Post-task: How easy/difficult has been for you to carry out this task?


25

Post-Scenario SQ: Expectation Rating



Three post-task questions: I feel satisfied with the easiness with which I have

completed the task. I feel satisfied with the time that it has taken me

to complete this task. I feel satisfied with the support information (online

help, messages, documentation, and so on) that I have had available while I was completing this task (only if your system offers online help)


26

Post-Scenario SQ: ASQ



Two post-task questions: ‘‘Were you aware of this functionality prior to this

study? (yes or no) On a 1 to 5 scale, how useful is this functionality to

you? Calculation:

Convert the likert item to a binary scale (e.g. 4 or 5 useful, rest not useful) and draw the graph.

Differences between awareness and usefulness indicate which tasks to promote (e.g. by redesigning the visual hierarchy) in your app.


27

Post-Scenario SQ: Awar-Usefuln Gap



Graph sample:


28

Post-Scenario SQ: Awar-Usefuln Gap



More questionnaires are appearing by the day in the literature. Many of them have been deemed necessary to cover the idiosyncrasy of new platforms/domains E.g. MPUQ: Mobile Phone Usability Questionnaire


29

SQ: Other questionnaires

Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ)



Can be post-test or post-task Reasons why the users are promoters or detractors of your

product Insights from users gathered from field studies Complaints about a product sent to the customer service Why the task was difficult to complete …

The way of processing this kind of questions is to turn the open comments into categories, quantify them and analyze them. http://www.measuringusability.com/blog/quantify-comments.php


30

SQ: Open Questions



All the questionnaires must be validated with respect to their reliability, validity and utility. Once validated, they still need to be revalidated when the context of application varies.


31

SQ: Validation



Lewis95] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.

[Lund 2001] Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2

[Chin 88] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In: CHI '88. Conference Proceedings on Human Factors in Computing Systems. New York: ACM, pp. 213–218.

[Brooke 1996] Brooke, J. (1996). SUS: A “quick and dirty” usability scale, In Jordan, P. W., Thomas, B. T., and Weerdmeester, B. A. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 189–194.

[Kirakowski 1996] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 169–177.

[Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.

[Abdinnour-Helm 2006]. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction with a Web Site.

[Tullis 2008] Thomas Tullis and , William Albert (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (Interactive Technologies). Morgan Kauffman.


32

References



These slides are made available under the license Creative Commons Attribution-NonCommercial-NoDerivs CC BY-NC-ND. More information about license: http://creativecommons.org/licenses/by-nc-nd/3.0/.These slides were created under Leonardo da Vinci Partnerships Project 2012-1-PL1-LEO04-28181 GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES (http://usability-accessibility.org/).


33

Attributions

Subjective questionnaires

Education

Transcript of Subjective questionnaires