Multimedia Specification Design and Production 2013 / Semester 1 / week 9 Lecturer: Dr. Nikos...

Multimedia Specification Design and Production

2013 / Semester 1 / week 9Lecturer: Dr. Nikos [email protected]

2

Learning outcomes

• Evaluation (more about this topic)

• Evaluation methods

• Empirical evaluation (necessary steps)

• Analytical evaluation

More about Evaluation

3

Reading List:

1.Notes for Lecture week_8: introduction to evaluation

2.Faulkner Chapter 6, pp 137 – 146 (stop before 6.5.1), 6.6. – 6.16, pp 156 – 173 and Chapter 7, pp 177 - 196


4

Evaluation

1.Central to user-centred iterative development, carried out throughout the development process (however often developers feel they can undertake by themselves)

2.Linked to every other activity in the design cycle

3.Developers are often tempted to skip itbecause it can add to the development time and costs money and effort


5

EVALUATION provides the opportunity to:

• help to ensure that the system is usable

• help to ensure what users want in the final system

• end up cheaper than fixing problems identified later on

Evaluation is the process of gathering data so we can answer the questions!


• why you are doing it

• what you hope to achieve within the inevitable practical constraints:

availability to facilities/equipment easy access to users expertise

time budget ethical issues

6

An introduction to Evaluation

Terminology

Evaluation

…the process of systematically gathering data at various stages within the

development process, which can be used to improve the designers’

understanding of the users’ requirements and amend the design to meet users’

needs. It can employ a range of techniques, some involving users directly, at

different stages, to examine different aspects of the design

7

Evaluation methods

1. Empirical 2. Analytical

…based on user experience …based on the expert view


3. Heuristic

8

1. Empirical evaluation: necessary steps

1. it should start with a clear understanding of what questions need to be

answered – i.e. what the evaluation aims to find out, set appropriate targets

2. developing the evaluation activity

selecting participants to perform tasks

developing tasks for participants to perform (benchmark and

representative tasks) – give the participants focused activities

determining protocol and procedures for the evaluation sessions

pilot testing may be necessary to improve experiment

3. directing the evaluation sessions


9


4. generating data by:

quantitative: benchmark tasks, user questionnaires

qualitative: concurrent verbal protocol, retrospective verbal protocol,

critical incident reporting, structured interviews

5. collecting data - in order to have evidence on which to base evaluation

real-time note-taking

audiotaping

videotaping

internal instrumentation of the interface


10


6. analyzing the data, comparing results with targets in usability

specifications

7. drawing conclusions to form a resolution of each design problem

8. depending on the outcome of the evaluation, it may be necessary to

redesign and implement the revisions


11

2. Analytical evaluation


The following section of notes has been compiled from papers on Jakob Nielson’s website http://www.useit.com. Faulkner Chapter 7 provides more detail about the various methods briefly outlined below.

Usability inspection is a set of methods that are all based on having

evaluators inspect and analyse a user interface.

Typically, usability inspection is aimed at finding usability problems in the

design, though some methods could evaluate the overall usability of an

entire system.

Inspection methods can be applied early in the interaction development

lifecycle, allowing feedback, iteration and improvement.

12



• Heuristic evaluation is the most informal method and involves having

usability specialists judge

• Heuristic estimation is a variant in which the inspectors are asked to

estimate the relative usability of two (or more) designs in quantitative terms

(typically expected user performance).

13



• Cognitive walkthrough uses a more detailed procedure to simulate a

user's problem-solving process at each step through the dialogue, checking

if the simulated user's goals and memory content can be assumed to lead to

the next correct action.

• Pluralistic walkthrough uses group meetings where users, developers,

and human factors people step through a scenario, discussing each

dialogue element.

14



• Feature inspection lists sequences of features used to accomplish typical

tasks, checks for long sequences, cumbersome steps, steps that would not

be natural for users to try, and steps that require extensive

knowledge/experience in order to assess a proposed feature set.

• Consistency inspection has designers who represent multiple other

projects inspect an interface to see whether it does things in the same way

as their own designs.

15



• Standards inspection has an expert on an interface standard inspect

the interface for compliance.

• Formal usability inspection combines individual and group inspections

in a six-step procedure with strictly defined roles to with elements of both

heuristic evaluation and a simplified form of cognitive walkthroughs.

16



Heuristic evaluation, heuristic estimation, cognitive walkthrough, feature

inspection, and standards inspection normally have the interface inspected

by a single evaluator at a time.

* In contrast

pluralistic walkthrough and consistency inspection are group inspection

methods. Many usability inspection methods are so easy to apply that it is

possible to have regular developers serve as evaluators, though better

results are normally achieved when using usability specialists.

17

3. Heuristic evaluation


• Heuristics – broad based rules or principles derived from theoretical

knowledge (e.g. cognitive psychology) and practical experience

• heuristic evaluation is the most popular usability inspection method

• heuristics can be used to inform the design, as well as providing a checklist

for evaluation

• heuristic evaluation allows quick, cheap and easy evaluation of a user

interface design, hence known as a ‘discount usability engineering’ method

• heuristic evaluation aims to identify usability problems which can then be

fixed within the iterative design process

• heuristic evaluation involves a small set of evaluators examining an

interface and judging its compliance with recognized usability principles

Multimedia Specification Design and Production 2013 / Semester 1 / week 9 Lecturer: Dr. Nikos...

Documents

Transcript of Multimedia Specification Design and Production 2013 / Semester 1 / week 9 Lecturer: Dr. Nikos...