Research to Inform Development DecisionsSteve Fadden, Ph.D.

Director, Analytics UX Research, Salesforce

Lecturer, Professional Faculty, UC Berkeley School of Information

Presented at the UserTesting Everyday Insights Roadshow, San Francisco, CA

August 23, 2016

Image: https://pixabay.com/en/photos/spiral%20staircase/

Image: https://www.flickr.com/photos/jiuguangw/5134925627

Agenda1. Introductions

2. Process

3. Formative Methods

4. Case study

About me

Steve Fadden

@sfadden

linkedin.com/in/stevefadden

About youFocus

● Research

● Design

● Engineering

● Marketing

● Management

● Other?

Research experience

● Over 5 years

● 2-5 years

● Up to 2 years

● None

About the research processGenerative: Creation

Formative: Evaluation

Summative: Assessment

Images: https://commons.wikimedia.org/wiki/File:Idea.png,

https://pixabay.com/en/selection-pick-chose-choice-select-443127/,

https://pixabay.com/en/graph-statistics-chart-infochart-29709/

About the research processGenerative: Creation

Formative: Evaluation

Summative: Assessment

Images: https://commons.wikimedia.org/wiki/File:Idea.png,

https://pixabay.com/en/selection-pick-chose-choice-select-443127/,

https://pixabay.com/en/graph-statistics-chart-infochart-29709/

Formative methods inform progressFormative

During development

Lower fidelity

Focused on improvement

Aid decision-making

Faster & lighter

Summative

Ready to launch

High fidelity

Focused on measurement

Understand current state

More comprehensive

Technique 1: Understand the problem

Image: https://pixabay.com/en/detective-magnifying-glass-viewing-788592

Gain insights:

● Evidence of problems

● Opportunities to solve

Based on:

● Recent events

● Specific details

● Feelings and perceptions

● Future behaviors, responses

Critical incident questions

Process1. Confirm user profile

2. Identify time since last experience

3. Gather details

a. Description

b. Actions taken

c. Feelings

d. Outcome

e. Future actions and responses

Reference: http://www.usabilitynet.org/tools/criticalincidents.htm

Applying your findings1. Validate problems

2. Identify opportunities

3. Understand expectations

4. Identify scenarios

5. Build empathy

Image: http://www.freestockphotos.biz/stockphoto/17339

“I needed to share a PDF with a friend, and we use Dropbox, but I hadn’t used it in a while. I logged in through my browser, dragged the PDF to Files, and then saw the PDF open. I expected Dropbox to start uploading. I hit back, created a folder in Dropbox, uploaded the PDF to it, clicked share to add my friend, and sent her the link.”

Technique 2: Assess expectations

Expectation testsGain insights:

● Mental model

● Categorization, labels

● Confusion areas

With:

● Nonworking interface

● No text/labels

● Basic task information

Image: https://pixabay.com/en/photos/man%20thinking/

“Greeking” technique1. Identify important tasks

(e.g. frequent, critical)

2. Write scenario

3. Write “first step”

questions

4. Gather feedback

5. Ask for group names

Example

Scenario: “You are exploring colleges, and need to make sure the courses look interesting and financial aid is available.”

Instruction: “How would you expect to start to apply for financial aid?”

Reference: Thomas S. Tullis, 1998. A method for evaluating web page design concepts. CHI 98 conference summary on human factors in computing systems

“Greeking” activityScenario: You are exploring colleges,

and need to make sure the courses look

interesting and financial aid is available.

How would you expect to:

1. Find out more about financial aid?

2. View information about courses

offered?

3. Contact the department in charge

of Athletics?

“Greeking” resultsHow would you expect to:

1. Find out more about financial aid?

2. View information about courses

offered?

3. Contact the department in charge

of Athletics?

What would you call each group?

12

3

“Academics” “Administrative”

“Connections”

Technique 3: Prioritize needs

Kano model

Absent Present

Dissatisfaction

Satisfaction

Adapted from: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results

Kano model

Absent Present

Dissatisfaction

Satisfaction

Adapted from: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results

Exciting

(truly unexpected)

Perform

ance

(m

ore is better) Basic

(required)

Kano model (automobile example)

Absent Present

Dissatisfaction

Satisfaction

Seat-warmers

Bluetooth

Intuitive GPS

H

orsepow

er

Fuel econom

y

Airbags

Adjustable seats

Kano question processProcess

● Assemble list of features

● Pose realistic scenarios for context

Assess for each:

● Importance of feature

● Satisfaction if available

● Satisfaction if not available

Scenario: “Imagine you are using our mobile app to find a restaurant.”

Question 1: “How would you feel if the app allowed you to filter results by distance?”

Question 2: “How would you feel if the app did not allow you to filter results by distance?”

Question 3: “How important is this function to you?”

Kano question results

If present :( If present :)

If absent :)

If absent :(

Map answers to “product

allowed you to...”

questions

Map answers to

“product did not allow

you to...” questions

Kano question results

If present :( If present :)

If absent :)

If absent :(

Providing

this feature is

dissatisfying

Providing

this feature

is satisfying

Not providing this feature is satisfying

Not providing this feature is dissatisfying

Kano question results pattern

If present :( If present :)

If absent :)

If absent :(

Providing

this feature is

dissatisfying

Providing

this feature

is satisfying

Not providing this feature is satisfying

Not providing this feature is dissatisfying

Technique 4: Evaluate the solution

Usability studyGoals

● Explore concepts

● Assess effectiveness

● Compare competitors

● Validate performance

Measures

● Learnability

● Efficiency

● Memorability

● Errors

● Satisfaction

Process

1. Plan (goals, participant criteria)

2. Prepare (recruit, test materials)

3. Conduct (active observers)

4. Analyze (issues, insights)

5. Report (fixes, future opportunities)

Reference: https://www.nngroup.com/articles/usability-101-introduction-to-usability/

Usability studyRITE - Rapid Iterative

Testing & Evaluation

(address problems)

Discount (find problems)

Traditional (benchmark

performance)

Aim to:

● Test representative users● Follow script with actual tasks● Use high-fidelity interface

Key elements:

● Observation by key decision makers (PM, design, development)

● Agree on critical problems and make changes to interface after each session

References: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (2nd Ed.), Rubin, J. & Chisnell, D. (2008)

Jakob Nielsen, 1989. Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using

human-computer interfaces and knowledge based systems (2nd ed.); Image: The RITE Way to Prototype, J. Shirey et al., UX Magazine, 2013

Using the RITE method to improve products; a definition and a case study, M. Medlock et al., Presented at UPA, 2002.

Example: Understanding data export

Approach used● Background

○ Urgently-needed integration with

a 3rd party product

○ Very visible launch event was

already scheduled

○ Small team, no dedicated research

support

○ “Boil the ocean” effort: Was

concept and functionality

understood, desired, and usable?

Process

Critical incident

Expectation

Prioritization

Usability evaluation

Critical incident elicited rich feedback“Consider the last time you had to export data. Describe why you

needed to export data, and list the steps you remember from that

process. (If you haven’t exported data before, or don’t remember

the last time, just skip to the next question).”

“I’m pretty old school, so I export my credit card transaction data about every quarter. My credit card site has a button to export to CSV, so I just click that and it downloads to my computer.”

“We have our marketing, sales, and inventory data in different systems. I have to export data from each system in order to combine it into a spreadsheet for my stakeholders. The export process is easy. Combining the data is more involved.”

Expectations gathered from storyboard“Consider the concept presented on the next 4 slides. As you

read through the concept, comment on anything you find to be

confusing, problematic, useful, or appealing about the concept.”

1. 2. 3. 4.

100%

Initial expectations highlight concerns1.

2.

3.

4.

100%

“Doing this would require a lot of clicks, even for a small number of columns.”

“You should embed best practices for naming here. Otherwise, the result could be messy.”

“Will we be able to save the mappings? That could save time in the future.”

Directed feedback emphasized value and priorities

“It’s great that you don’t have to jump around different parts of the system to do this. Very valuable to be able to complete this from one place.”

“It’s useful, but only if I can save it and reuse it in the future. Wouldn’t use it for a one-time export.”

“Hi, I wanted to follow up to reiterate that this is a REALLY COOL idea and it fills a much needed requirement for our use of the product. Please consider me for future studies like this, because we need this functionality!”

Usability sessions identified more issues

● RITE with 4 participants over 2 days

● Key changes: Terminology, embedded instructions, icons

● Stakeholder advocacy (and gratitude)

“It says ‘save’ but I’m just modifying the settings. Unless I can re-use this later?”

“I haven’t selected the columns yet! Why can’t I go back?”

Some formative evaluation results● Evidence of problems

● Value of overall solution

● Prioritizations and tradeoffs

● Issues with flow, terminology, design elements

● Importance of incorporating formative evaluation into

process

Thanks!

Image: https://en.wikipedia.org/wiki/File:TulipStair_QueensHouse_Greenwich.jpg