HF750: Usability Testing & AssessmentHF750: Usability Testing & AssessmentKodak EASYSHARE ONEKodak EASYSHARE ONE

Usability Test Team 5:Usability Test Team 5:

Kevin BerniKevin Berni

Vanessa BrownVanessa Brown

Mike LedouxMike Ledoux

Pat SorensonPat Sorenson

Hanna YehudaHanna Yehuda

Usability Test ResultsUsability Test Results

We were fortunate to have access to a product that We were fortunate to have access to a product that

was soon to be released to the general public.was soon to be released to the general public.

The product offered some advancements in digital The product offered some advancements in digital

photography.photography.

Our team expressed an interest in testing a non-Our team expressed an interest in testing a non-

software based application.software based application.

It would offer an enjoyable test experience for our It would offer an enjoyable test experience for our

testing team and our participants.testing team and our participants.

Why the Kodak EasyShare One Why the Kodak EasyShare One camera was chosen for testingcamera was chosen for testing

Test DesignTest Design

RecruitmentRecruitment– 3 Males & 5 Females 3 Males & 5 Females – Ages early 20’s to early 40’sAges early 20’s to early 40’s– Professionals, categorized as “tech savvy”Professionals, categorized as “tech savvy”– Familiar with wireless internet technologyFamiliar with wireless internet technology

TasksTasks– Basic operational tasks to test key featuresBasic operational tasks to test key features– Probing usability tasks to test some of the concerns of Kodak and the Probing usability tasks to test some of the concerns of Kodak and the

Bentley test teamBentley test team Testing environmentTesting environment

– The Bentley usability lab (video & auditory recording capabilities)The Bentley usability lab (video & auditory recording capabilities)– One Bentley Graduate school moderator with each participantOne Bentley Graduate school moderator with each participant– One Bentley Graduate school team member managing the video & audioOne Bentley Graduate school team member managing the video & audio– One Bentley Graduate school team member taking notesOne Bentley Graduate school team member taking notes– Remote testing out of the home (3 participants)Remote testing out of the home (3 participants)

Test DesignTest Design

Testing protocolsTesting protocols– Participants were instructed to “Think aloud” during testingParticipants were instructed to “Think aloud” during testing– Survey questionsSurvey questions

After each task – 7 point Likert scale (1=Easy… 7=Difficult)After each task – 7 point Likert scale (1=Easy… 7=Difficult) After all tasksAfter all tasks

– Semantic Differential for Measuring Hedonic QualitySemantic Differential for Measuring Hedonic Quality

– Post Ease of Use SurveyPost Ease of Use Survey

– Post Test Satisfaction SurveyPost Test Satisfaction Survey

– AssistanceAssistance Product overview reference cardProduct overview reference card Moderator could give an assistModerator could give an assist

Measurement of ResultsMeasurement of Results

Participant tasks were measured byParticipant tasks were measured by– TimeTime– Successful completion of the taskSuccessful completion of the task– Number of assistsNumber of assists– Importance of the taskImportance of the task

Test ExecutionTest Execution The test consisted of 14 tasks that would test the product’sThe test consisted of 14 tasks that would test the product’s

– Features & FunctionalityFeatures & Functionality– Feature workflowFeature workflow– User conceptual understandingUser conceptual understanding– The physicality of operationThe physicality of operation– The camera’s operational abilities (performance, speed, user The camera’s operational abilities (performance, speed, user

manipulation).manipulation). The participant was instructed to read each task aloud and to The participant was instructed to read each task aloud and to

carry out the instructions. No break times were given during carry out the instructions. No break times were given during testing, although they were offered.testing, although they were offered.

All 8 participants completed (or attempted) all tasks without All 8 participants completed (or attempted) all tasks without rest breaks or without an early departure from testing. rest breaks or without an early departure from testing.

Test ExecutionTest Execution IssuesIssues

WI-FI connectivity with the printerWI-FI connectivity with the printer Handling the technological dependencies, e.g. the Bentley Handling the technological dependencies, e.g. the Bentley

labs network goes down, or performance is degradedlabs network goes down, or performance is degraded Initial Kodak EasyShare One camera broke during pilot testInitial Kodak EasyShare One camera broke during pilot test The male vs female ratio of participantsThe male vs female ratio of participants Half of the participants watched the Kodak EasyShare One Half of the participants watched the Kodak EasyShare One

training videotraining video The type and number of assists that could be givenThe type and number of assists that could be given Handling the dependency factor, e.g. the successful Handling the dependency factor, e.g. the successful

completion of one task needed for the subsequent taskcompletion of one task needed for the subsequent task

Test ResultsTest Results How we categorized the issues foundHow we categorized the issues found

Positive findingPositive finding: A camera feature or process that should be preserved : A camera feature or process that should be preserved or developed more.or developed more.

Good IdeaGood Idea: The participant and or test team found a feature or : The participant and or test team found a feature or approach that worked well, or received input that should be considered.approach that worked well, or received input that should be considered.

Minor ProblemMinor Problem: Caused the participant to hesitate for a few seconds.: Caused the participant to hesitate for a few seconds. Moderate ProblemModerate Problem: Delayed participants in their use of the camera, : Delayed participants in their use of the camera,

but didn’t cause any catastrophic problems.but didn’t cause any catastrophic problems. Serious ProblemSerious Problem: Delayed participants in their use of the camera for 1 : Delayed participants in their use of the camera for 1

to 5 minutes, but eventually were able to continue.to 5 minutes, but eventually were able to continue. Critical ProblemCritical Problem: Caused frequent catastrophes.: Caused frequent catastrophes. BugBug: Camera is not working as designed.: Camera is not working as designed.

Molich, R. & Dumas, J. (2005). Comparative usability evaluation (CUE-4), Behavior and Information Technology, In press, p. 11-12

Test ResultsTest Results The category break-down of issues found.The category break-down of issues found.

Test RecommendationsTest Recommendations Consider a menu option like Create or New, which could be Consider a menu option like Create or New, which could be

used for creating a new album.used for creating a new album. Add affordance to pictures that are favorites, versus having Add affordance to pictures that are favorites, versus having

the user clicking a picture to see if it’s a favourite.the user clicking a picture to see if it’s a favourite. Add a graphic of the camera with the WIFI card extended on Add a graphic of the camera with the WIFI card extended on

the “extend WIFI card” error message.the “extend WIFI card” error message. The edit/album screen icon name is confusing, especially if The edit/album screen icon name is confusing, especially if

you want to add a photo to the album. Consider modifying you want to add a photo to the album. Consider modifying the screen name.the screen name.

Submit icons for user testing. Better Submit icons for user testing. Better resemblanceresemblance icons are icons are needed.needed.

Consider a built-in WIFI adapter.Consider a built-in WIFI adapter. The camera uses some wizards in carrying out certain The camera uses some wizards in carrying out certain

functionality, consider the use of more wizards.functionality, consider the use of more wizards. Add an envelope button for email.Add an envelope button for email.