Feedback from Usability Testing

Semantico/Random House Widget

4th April 2008

Who’s here

• Pete Gale ~ Head of User Experience

• David Harley ~ User Experience Consultant

Overview

• Introduction• Details of usability testing conducted• Framework for understanding results• Specific issues identified• Suggestions

About Cogapp

• Founded in 1985• Decades delivering projects on demanding timescales• Long-term view, long-term value• Creative methodology• A user-centred approach supported by internal testing facilities• Growing internet consultancy practice

Our Clients

• Apple UK• Macmillan Cancer Support• BBC• The British Museum• Prudential• Department of Health• Art Institute of Chicago• HSBC• National Gallery, London• Cable & Wireless• MoMA• Design Museum London

• V&A• SONY • Ofcom• Arthritis Research Campaign• Microsoft• National Portrait Gallery• Channel 4• 10 Downing Street• Science Museum London• European CommissionI• CIS• The British Library

Usability & user centred design

• Usability techniques provide the best ROI when used as part of a user centred design process.

• User centred design tests early and often, to validate user needs and design solutions, ensuring designs are based on an understanding of real user needs and behaviours.

“We urge all UK technology companies to put User Centred Design at the heart of their R&D and innovation activities and promote a people-centred culture throughout their organisations. This will help ensure they develop the right products and services for

the right markets, first time and all the time.”

Department of Trade and Industry, Global Watch

ROI for user centred design (hidden)

• User centred design provides an opportunity to identify & validate user needs early in the project

• 80% of changes made to a project after launch are because of “unmet or unforeseen user requirements”.

Claire Karat,

Software Engineering in Dollars and Cents

• The cost of change requests increases exponentially as software moves through development and into a live environment.

Roger Pressman, Software Engineering: A Practitioner’s Approach, McGraw-Hill

Cost

of

makin

g c

han

ges

1xDuringDesign

100xAfter Launch

6xDuring

Development

Considering different levels of user experience

Proposition What is the value being delivered?

Concept How is that value being delivered?

Structure How is it organised?

Information What information is required?

Interaction How do users interact with the product?

Appearance What does it look like and how is it arranged?

• As proposed by Jesse James Garrett and others, user experience can be considered on a number of interdependent levels

The Random House Widget

The New Functionality:The Thumbnail Browser View

How is Value Derived by Random House ?

• Increased uptake of books via online interactions.

• Development of a better online relationship with Random House book buyers.

What is the Value Delivered to Users ?

• There’s something “special” going on.– The presence of the “Open the Book” banner denotes a different

treatment for this book perhaps something “special” worth investigating.

• The ability to read content.– The opportunity to engage with the material within the text of

the book.

• The ability to scan meta content to ascertain relevant purchases.– The opportunity to access supporting information about this

book to make an effective purchase , e.g. recommendations, table of contents, reviews.

• A faithful representation of a real artefact (that they might buy).– To get a feel for the real object through a virtual representation

of the book.

Usability Testing

User testing sessions were used to identify major obstacles to using the widget for particular tasks.

Users were asked to engage with the following tasks:

1. Make sense of the initial ‘small widget view’.2. Access the main browsing view from the initial view.3. Explore and read a book’s content.4. Navigate successfully around the main browser window. 5. Use the other functionality available through the main browser window.

Our Users

• Eight users. • Two women, six men. • Aged from 19 to 62 years with an average of 34.1 years. • Regular Internet users with broadband connections at

home, going online every day. • Familiar with buying books online.• 5 bloggers, 3 non-bloggers.

Testing Setup

Whilst engaging with the tasks our users’ actions and reactions were observed and captured in a number of ways:

• Their onscreen actions were recorded.• Their conversation and reactions were captured using the

computer’s webcam.• They were observed through a two-way mirror.• Notes were taken of salient interactions.

Throughout testing users were asked to “think aloud”, i.e. state explicitly what they were attempting to do.

User Session 1

“Explore the interface and identify content that is of interest” (Non-Blogging Book Enthusiasts)

3 users took part in the first user testing session. They were all non-bloggers and book enthusiasts.

• Undirected they did not read the book.

• They focussed on meta content (i.e. information about the content of the book ), particularly the recommendations.

What is the Value Delivered to these Users ?

• Something “special” going on

• Ability to read content

• Ability to scan meta content to ascertain relevant purchases

• Faithful representation of a real artefact (that they might buy)

User Session 2

“Read through the first page of the book” (Blogging Book Enthusiasts)

5 users took part in the 2nd user testing session. They were bloggers and book enthusiasts.

1. They were asked to compare different book browsing approaches looking through a similar novel on each website.– Amazon– Random House US– Random House UK

2. They were then asked to explore the full functionality of the Random House UK interface.

3. They were asked to rate the different interfaces in terms of ‘ease of use’ after step 1 and then to rate the Random House UK interface again after exploration (step 2).

Amazon

Random House U.S.

Random House U.K.

Ease of Use

After Reading

6.8

7

7.2

7.4

7.6

7.8

8

8.2

8.4

8.6

Amazon Random House UK Random House US

Ease of Use

After Exploring Other Functionality

6.8

7

7.2

7.4

7.6

7.8

8

8.2

8.4

8.6

Amazon Random House UK Random House US

Developing a Framework for Understanding These Results

Broad Findings:

• Users had preconceptions of expected use based on Amazon.

• Reading through a novel online favoured the Amazon interface.

• More learning was required to make the most of the Random House interface.

• Users could see the potential of the Random House interface when

shown more of its functionality.

Different Types of Book: Narrative Versus Information

• Different book content requires different interactions.

• Narrative ( fiction, children’s books and biographies ) are time-bound and sequential. You have to read page by page from beginning to end.

• Amazon and Random House US emphasise sequential interactions. They do this by restricting user options in the interface. The simplest and most immediate interactions move through the book page by page without access to an overview.

Different Types of Book: Narrative Versus Information

• Informational books ( non-fiction, manuals, cookery books, etc ) rely on access at multiple points rather than sequentially.

• Random House UK interface currently emphasises this multiple-point access ( and hence access to informational content). It discourages engagement with narrative.

Different Reading Behaviours

• We know from eye tracking research that there are different kinds of reading behaviour. 3 distinct behaviours have been identified:

– Reading– Scanning – Skimming (Ohno, 2004)

• These are dependent on the intent of the reader, the type of book they are reading and the context in which they are reading it, e.g. on a website, in a book or in a newspaper.

Reading Narrative in Print

Reading on the Web

Reading

Scanning

Skimming

What Kind of Reading Did We Observe?

User Session 1 User Session 2

Scanning Scanning

Skimming Skimming

Reading (prompted)

Selecting Books Through Different Kinds of Reading Behaviour

Academic research suggests that the most effective way of selecting books with informational content is through the scanning of meta content, in particular tables of contents. (Wacholder et al, 2006).

What is the Value Delivered to Users ?

1. There’s something “special” going on

2. The ability to read content.

3. The ability to scan meta content to ascertain relevant purchases.

4. A faithful representation of a real artefact (that they might buy).

Specific Issues Identified within the Random House Interface

Issues relating to tasks:• Exploring and reading the book’s content• Navigating around the main browser window• Other functionality

– Search– Add widget– Send to a friend– Buy– Book info

Exploring and Reading the Book’s Content

Scrolling

Reading narrative requires a smooth movement of attention from left to right and down the page. Obstacles to this stop the natural flow of reading.

Evidence– All 8 users could not initially find any means of scrolling within the browser

window.

Analysis– There is currently no visible scroll control.

Recommendations– Scrolling must be visible and accessible from within the viewed page.

!!!!

Users have expectations for what should happen when they ‘Open the Book’. This is dependent on the type of book being viewed.

Evidence– All 8 users found it difficult to read the book’s content when first presented with it.

Analysis– Expectations for narrative are reading rather than overview navigation.

Recommendations– The main browser window should start by displaying pages according to the

appropriate reading style for the book • Narratives should open ‘ready to read’ on the first page of content.• Informational books should open on the table of contents.

Book Entry point!!

The significance of the thumbnail navigator is likely to depend on the type of book.

Evidence– All the users who discovered that they could hide the thumbnail navigator

preferred reading without it. In terms of navigating through the book all 8 users liked the thumbnail navigator as an innovative feature.

Analysis– The style of reading (i.e. skimming or scanning for informational content, reading for narrative)

should also be reflected in the availability of the thumbnail versus single page view.

Recommendations– Narratives should open without the thumbnail navigator.Informational books

should open with it.– The ability to hide or display the thumbnail navigator should be more prominent,

i.e. the navigator open/close icon should be larger and clearer.

Thumbnail Navigator Visibility!!

Zooming is closely related to the use of the scroll, both being balanced for optimum viewing.

Evidence– All users identified the zoom control and were able to use it to view pages at a

reasonable size. Their inability to find the scroll meant that they struggled to find an optimum size, i.e. one that was visible but did not require scrolling. This meant continual readjustment of the zoom.

Analysis– The complexity of the zoom control was only necessary because of the invisibility

of the scroll function.

Recommendations– Pages should appear immediately at a viewable size. – Simpler one click controls should be made available to zoom in/out of the viewed

page.

Zoom Controls!

Navigating Around the Main Browser Window

Reading, scanning and skimming all follow a similar movement of attention from top left to bottom right. The virtual turning of pages must accommodate this.

Evidence– All users had to follow a sequence of manoeuvres in order to read through the book

page by page. Starting with the mouse pointer in the middle of the page for scrolling they had to interrupt their reading, locate the page forward button, move to it with the mouse pointer, click on it then return their pointer to the middle of the page to resume scrolling.

Analysis– The mouse movements involved in turning the page are excessive and interfere

with attention to reading.

Recommendations– The ‘page forward’ button should be made larger and be placed closer to the bottom

right of the displayed text (without obscuring the text).

‘Page Forward’ Control!!!!

The thumbnail navigator potentially enables greater control over the scanning of a book by traversing a number of pages at a time using the top arrow buttons. It is important that the user retains control over this at all times and can interrupt this process.

Evidence– Two users got into a situation where they were traversing pages using the top arrow

buttons and decided they wanted to stop on a particular page. This was not possible causing user frustration.

Analysis– Immediate control is an important aspect of scanning

Recommendations– Allow the user to select individual pages whilst traversing pages.

Retaining Control Whilst Traversing Pages!!

The arrow icons at the top and bottom of the main browser window are the same but offer different functionality. The top ones allow traversal of many pages (useful for scanning and skimming) whilst the bottom ones move one page at a time (and are important for reading).

Evidence– Through experimentation all users were able to effectively navigate within the book

but none of them were aware of this difference in function until it was pointed out to them.

Analysis– Users are likely to miss functionality which may be most appropriate to their

intended use of the interface, i.e. reading versus scanning or skimming.

Recommendations– The arrow icons should be changed to reflect the underlying difference of function.

Differentiation of Navigation Arrows!

The function for going to the ends of the book also shows the travel through the intervening pages. Those attempting to scan through the book are most interested in the meta content and are unlikely to want to read along the way.

Evidence– Three users found the journey through the pages frustrating because it was slow

and did not allow them to interrupt the process.Analysis– Users engaged in scanning behaviour are most interested in the meta content that

is usually on the back cover. They want access to this information as quickly as possible..

Recommendations– The arrows to the end points of the book should provide instant access to these

points and should not show the travel. The end point arrows should be moved to the bottom of the page so that they are accessible from within the narrative view ( see suggestions section).

Instant End Point Navigation!

Testing Other Functionality: “search”

The purpose and context of the search is unclear. The search is not relevant to narrative content.

Evidence– When asked to use the search facility all the users were unsure of its purpose.

“Why would I put any word in [the search], I don’t understand that” (User 2).

– Users’ initial expectations were that the search would be pertinent to book titles rather than book content. This expectation was reinforced by the appearance of the search on other pages, i.e. book info, send to a friend, add widget.

– Users could see the potential of search given a book with informational content, e.g. a cook book or a text book.

Analysis– Search is relevant to informational content and not narrative. The

misunderstanding of the search context could be disastrous for prospective sales is users interpret no results as ‘no book’.

Recommendations– Search should be removed from narrative content and the context of the search

should be clearly stated, e.g. “ Search this Book”. The results should reiterate this context, e.g. “(search term) “X” appears Y times in this book”

Search Context!!!!

Search results appear in two categories – one that is part of the available sample and that can be accessed and another that is contained within the rest of the book and that cannot be accessed. The distinction between these results is not apparent to users.

Evidence– No user worked out the distinction between the two sets of results.

“I don’t know why I can’t see what I’ve been searching for” (User 6)

Analysis– To be useful the search facility must support effective scanning and skimming. As

such it should be a selective questioning of the content which gives appropriate feedback at all times.

Recommendations– Feedback must be provided when users click on the results to explain why they

cannot access the content, e.g. “If you buy this book you will be able to read the whole paragraph”. The same is true for no results with feedback such as “Sorry, this word does not appear in the book”.

Search Results!!

When search terms are typed into the search box, a “clear search” icon appears to the right of the text. This is easily misinterpreted as a “search go” button.

Evidence– 2 users clicked on the “clear search” icon thinking that it was a “search go” icon.

Analysis– The placing of the “clear search” is contrary to existing conventions for search

functionality and hence user expectations.

Recommendations– Replace the “clear search” function with a “search go” function. Rely on the

keyboard delete for user control of text.

“Search Clear” Function!

This was understood by all but 2 users. Those that did not understand it found that it did not represent an obstacle for them as they just re-entered their search terms.

Search Again

How is Value Derived by Bloggers ?

1. Something “special” going on

2. Ability to read content

3. Ability to scan meta content to ascertain relevant purchases

4. Faithful representation of a real artefact (that they might buy)

5. Ability to share current or recent reading with others ?

Blogging and Narrative

• Like narrative, blogs are time-bound and sequential.

.• They differ from narrative in that they start with a sense of ‘now’ and

move backwards in time. Conversely narrative starts in the past and moves forwards towards now.

• Blogs capitalise on web reading behaviour. They make use of the characteristic ‘F shaped’ reading pattern which emphasises the top part of the screen (see earlier slide on reading behaviour).

• Blogs reintroduce aspects of ‘social reading’ which hark back to earlier reading practices, e.g. public readings.

.

Testing Other Functionality: “add widget”

The add widget function is accessed by clicking on the “+” icon. This does not describe the underlying function and is easily misunderstood.

Evidence– The non-bloggers could not ascertain what this function was about or how the “+”

symbol related to it. Bloggers could understand the concept of a widget but felt that it was inappropriately named and that the “+” symbol did not aid understanding. The “+” symbol was misinterpreted as a zoom by all 3 of the non-bloggers.

Analysis– The purpose of this function in relation to book browsing remains unclear.Recommendations– The “+” symbol should be changed to a more suitable one. See www.lastfm

/widgets for an example. The label for this function should also be changed to define its function more clearly, e.g. “Add to My Site”.

The Add Widget Icon “+”!!!!

The 2 colours used in the widget can both be changed by using the colour choosing interface. It is easy for users to choose colours that are not harmonious – these changes are not localised within the widget but change the look and feel of the whole browsing experience

Evidence– All users found changing the colours reasonably easy but once changed they had

great difficulty in choosing colours that they were happy with. The corresponding and simultaneous colour changes in the rest of the browser interface were unexpected and led users to think that they had misunderstood the purpose of the “add widget” function from the start.

Analysis– Extended interactions with the colour changing interface are a distraction from

bloggers’ intended aim of sharing their reading experience.Recommendations– Limit the choice of colours to discrete and harmonious combinations. See

www.lastfm/widgets for an example. Do not change the look and feel of the whole browser interface as a consequence of widget changes.

Colour Changing within the Widget!!!!

Testing Other Functionality: “send to a friend”

Currently the labels which explain the required user input are already present in the text boxes and are deleted by the user as they type their own information into the boxes. If users forget what they are typing then they are unsure of how to proceed.

Evidence– Two users assumed what information was required and started typing into the

boxes. They then realised that they had not checked what was actually required and could no longer check from the labels.

Analysis– The current set up does not allow for user error.

Recommendations– Move the labels for user input so that they are permanently placed adjacent to

the relevant input boxes, either to the left or above the user input boxes.

Text Input Labels for Send to a Friend!!!!

Currently users are not informed of what they are actually sending to a friend by email and in particular how that email relates to the book they were just reading.

Evidence– Although all users were able to go through the motions of sending an

email to a friend, when asked what they had sent none were able to answer the question with any certainty.

Analysis– The purpose of sending an email to a friend remains unclarified within

the interface..Recommendations– It should be stated clearly what it is that is being sent and users should

be able to tailor their email content.

Content of Sent Messages!!

What is the Value Delivered to Bloggers ?

1. Something “special” going on

2. Ability to read content

3. Ability to scan meta content to ascertain relevant purchases

4. Faithful representation of a real artefact (that they might buy)

5. Ability to share current reading with others ?

Suggestions

Increasing Value to Bloggers

• Engagement with and sharing of contentAllow users of Random House book browser to share passages from

the book rather than just the cover shot of the book.

• Encouragement of social aspects of readingIntegration with online book clubs, reading groups.

Suggestions

2 Browser Views:

• The readable one for narrative

&

• The navigable one for informational content

Suggestions

What Kind of Reading Leads to Book Buying ?

This remains unclear. It would be possible to analyse user interactions once the functionality has been launched online.

From measurements of user clicks and the timings between it would be possible to ascertain whether users are scanning, skimming or reading and which of these are most likely to result in a purchase.

References

Ohno, T. (2004) EyePrint: Support of Document Browsing with Eye Gaze Trace. Proceedings of the 6th International Conference on Multimodal interfaces. 2004, 16-23.

Rayner, K. (1998) Eye Movements in Reading and Information Processing: 20 Years of Research. Psychological Bulletin. 1998, Vol. 124, No. 3, 372-422.

Wacholder, N., Liu, L. and Liu, Y. (2006) Selecting Books: A Performance-based Study. Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries. 2006, 337.