Webinar: Retail UX Benchmarking

Webinar: Retail UX Benchmarking: Which Brand Provides the Best Website Experience?

Nate Colker| UserZoom12/13/2012

Intros

Alfonso de la Nuez

Co-Founder & Co-CEO

Nate Colker

UX Researcher

Twitter Hashtag #uzwebinar

Introduction Study Design Results Tips

• Defining UX Benchmarking

• Luxury department store Competitive Industry Study

• Tips for building your own competitive benchmarking study

Overview


Define Benchmarking

“Benchmarking is the process of comparing one's business processes and performance metrics to industry bests or best practices from other industries.”

http://en.wikipedia.org/wiki/Benchmarking


UX Benchmarking is the process of comparing a website’s (or any digital product) performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception, to another that is widely considered to be an industry standard benchmark or best practice

Definining UX Benchmarking & Metrics

Task Completion

Task Time Satisfaction

Task ConfidenceTask Level - Single Ease Question Test Level - System Usability ScaleNet Promoter Score

Number of ClicksNumber of Page Views


Define measurable & comparable UX-related KPIs

Jeff Sauro, Measuringusability.com


Why do we Care about Measuring User Experience?Ordering a Pizza Online (1994)

http://articles.chicagotribune.com/1994-08-22/news/9408230031_1_pizza-hut-santa-cruz-operation-internet


Why do we Care about Measuring User Experience?Ordering a Pizza Online (2012)



The World of User Experience Testing

http://www.useit.com/alertbox/user-research-methods.html Chart by Christian Rohrer

http://www.useit.com/alertbox/user-research-methods.html

http://www.useit.com/alertbox/user-research-methods.html

Unmoderated Usability Testing


Case Study: 4 Brands


Study Summary

• Between subjects design• 50 Participants per brand (200 total)• 2 Tasks + survey questions• Randomized task order• Recruitment criteria

– Must shop online– Shopped at luxury department stores

• Total study length : ~14 minutes• Used a panel vendor• 3-4 days to field the study (11/20-11/23)


Task Descriptions

Task 1 Objective: measure findability of a common apparel item (“scavenger hunt task”)

Task DescriptionPlease find the price of a grey wool dress by Eileen Fisher. Please note down the price, as you will be asked for it later.

Task 2 Objective: Access desirability and perceived quality of the sales section (“open task”)

Task DescriptionPlease take your time to explore the sale section and look for things that are of interest to you. Once you feel you have spent an adequate amount of time exploring this section, click on the “I’m Finished” button. We will then ask you some questions about your experience with the sale section of Nordstrom.com.


Case Study Preview


https://s.userzoom.com/m/MSBDNTlTMzI3

Task 1 Completion Rates: Find a dress

Nordstrom Neiman Marcus Saks Lord & Taylor0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

72%

54%

74%

84%

Task Success Rate

The completion rates on Nordstrom, Saks and Lord & Taylor were significantly better than on Neiman Marcus (90% Confidence Level)


Disaster Rates (Highest Confidence but Failure)

Nordstrom Neiman Marcus Saks Lord & Taylor0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

72%

54%

74%84%

20%

32%

10%

12%8%

14% 16%4%

DisasterFailureSuccess


Task-Level Ease

Nordstrom Neiman Marcus Saks Lord & Taylor1

2

3

4

5

6

7

4.8

3.9

6 5.9

Very

Diffi

cult

Very

Eas

y

Overall, how difficult or easy was it to find the price of a grey wool dress?


Too many choices. Small print beneath the pictures.

I couldn’t find it and wasted a lot of time trying.

I never found the grey wool dress therefore I couldn't know the cost. This is obviously a failure at the task asked of the survey. I gave myself the lowest score possible, not for lack of trying thought!

I could not find the dress. I searched under eileen as the designer. I typed in grey wool dress in the search box and only two childrens dresses came up. I looked under sale items too and found three dresses by this designer non of which were wool.

I put Eileen Fisher grey wool dress in search bar and search query came up with "can not find Eileen Fisher grey wool dress" so I put in search Eileen Fisher wool dress and I found dresses none were grey.... I then looked through other pages and did not see a grey dress.

Search didn’t work - came back with not found, didn’t have intuitive drop-downs, under designers didn’t have dresses as option. No description said was a grey wool dress, closest was gray 3/4 length dress. Took to long to weed through options looking for dresses by Eileen.

Briefly describe why you chose the number you did.

Nordstrom

Neiman Marcus

Open-ended responses from participants who thought the task was difficult (1, 2 or 3)


The Saks website kept sending me to Saks Direct - it took a number of attempts (7? 8?) to get to dresses, then find the red wool dress assigned. It was very frustrating. Had the site worked well, it should have been easy.

Number of steps to complete task

It was not a difficult site to navigate, but I would rather narrow down a category rather than to search for a specific item using the search box

Its the only grey dress that said it was wool. it kept bringing up all different thing unrelated.

Because i could not figure out which brand of grey wool dress it was.

It took time to find the right dress, and I’m not completely sure it was the right one

Briefly describe why you chose the number you did.

Saks Fifth Avenue

Lord & Taylor

Open-ended responses from participants who thought the task was difficult (1, 2 or 3)


Task Time: Completed tasks only


50

100

150

200

250

300

350

236

293

146126

Sec

ond

s


Completion time on Lord & Taylor was significantly lower than it was on Nordstrom & Neiman Marcus (90% Confidence Level)

Post task questions

Nordstrom Neiman Marcus

Saks Lord & Taylor1

2

3

4

5

6

7

5.5

4.7

6.3 6.2

5.2

4.4

6.25.9

5.1

4.1

66.2

It was easy to navigate the websiteIt was clear how to start finding the price of the wool dressI am satisfied with the amount of time it took to complete the task

Str

ong

ly D

isa

gree

Str

ong

ly A

gree


Task 2

Evaluate the sales section

Please take your time to explore the sale section and look for things that are of interest to you. Once you feel you have spent an adequate amount of time exploring this section, click on the “I’m Finished” button. We will then ask you some questions about your experience with the sale section of Nordstrom.com


Sale Section Evaluation

How satisfied are you with the number of items that were on sale?


2

3

4

5

6

7

5.55.2

4.95.3

Not

at

all S

atis

fied

Ver

y S

atis

fied


Sale Section Evaluation

How satisfied are you with the sale prices?


2

3

4

5

6

7

3.9

3.33

3.7

Not

at

all S

atis

fied

Ver

y S

atis

fied


Sale Section EvaluationS

tron

gly

Dis

agr

eeS

tron

gly

Agr

ee


Saks Lord & Taylor1

2

3

4

5

6

7

6.2

5.76

6.2

5.75.4

5.1

5.5

4.8

4 3.9

4.5It was easy to navigate the sale sectionI found some items that I likedI would consider checking out the sale section again in the fu-ture


Final Questionnaire: What’s your attitude towards the brand?


Saks Lord & Taylor1

2

3

4

5

6

7

4.8 4.84.4

4.1

4.9

4.34.6

4.8

BeforeAfter


Final Questionnaire:Brand Perception

1. Trendy 30% 62% +32%

2. Good Value 8% 26% +18%

3. Friendly 12% 30% +18%

4. Exciting 18% 32% +14%

Pre Post1. Expensive 60% 80% +20% 2. Trendy 34% 48% +14% 3. Trustworthy 24% 18% -6%

4. Customer 20% 8% -12% Focused

Pre Post


1. Trendy 50% 62% +12%

2. Expensive 80% 88% +8%

3. Exciting 16% 12% -4%

4. Trustworthy 28% 12% -16%

Pre Post

Saks Fifth Avenue

1. Expensive 48% 82% +34%

2. Trendy 22% 52% +30%

3. Exciting 8% 26% +18%

4. Good Quality 42% 58% +16%

Pre Post

Lord & Taylor


Final Questionnaire

Nordstrom NeimanMarcus

Saks Fifth Avenue

Lord & Taylor

The website is easy to use 5.8 5.1 6.2 6.3

I feel comfortable purchasing from this website

5.2 4.3 5.1 5.2

I find the website to be attractive 5.5 5.3 5.5 5.6

The website has a clean and simple presentation 5.7 5.3 5.1 5.9

I will likely visit this website in the future 4.5 3.6 3.9 4.4

On a scale from 1 to 7Introduction Study Design Results Tips

Final Questionnaire:Net Promoter Score

0 1 2 3 4 5 6 7 8 9 10 NPS

Nordstrom10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6%

-38%

Neiman Marcus

8% 10% 10% 4% 8% 18% 10% 12% 18% 6% 6%-

56% Saks Fifth Avenue

12% 2% 10% 0% 2% 18% 16% 10% 10% 6% 14%

-40%

Lord & Taylor10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6%

-38%

Detractors Promoters

NPS% of Promoters

(9s and 10s)Detractors

(0 through 6)= -


Final Questionnaire:What did you like the most about the site?

Nordstrom Neiman Marcus“I really liked that you can change the colors of different items without being forced to open that particular item, taking you to another screen.”

“That I could click on the pictures and see the back of the garment.”

“The search function quickly found what I was looking for.”

“I like how the pictures of the apparel would show front and back when I hovered the mouse over the picture.”


Saks Fifth Avenue Lord & Taylor

“The amount of items I could see on one window. I hate hitting next over and over again.”

“Garment being worn by a moving person”“Easy to find anything. love the search/browse by size, color etc.”

“It was easy to navigate and find products I was looking for.”

“Search options. By Designer By Color By Price. Very Easy and nice to use.”

Final Questionnaire:What did you like the least about the site?

Nordstrom

Saks Fifth Avenue

Neiman Marcus

Lord & Taylor

“Small print beneath the pictures.”“Hard to search compared to other sites I have been on.”

“The models are ultra slim, it can be discouraging for someone who isn’t a size 2 or size 0.”

“The filters by designer only includes size color etc. not dresses, tops etc.”

“Drill through for finding the dress was difficult.”“It was difficult to navigate when searching for something specific.”


“The prices were a little steep.”

“They were very expensive. I also felt that for the price you would think they would have more trendy items of clothing. I didn’t find anything thatInterested me.”

“A little too expensive to buy more than one or two items.”

“The prices, but that is not the website”

We have an exciting announcement about a new feature…

Exciting announcement of new feature

Some Final Tips for Creating your own Benchmark

Task Selection

Standardized design

Randomize Conditions

Cleaning the Data


Selecting the Right Tasks

A Balance of User Data and Business Goals

What are users coming to the sites to do (core tasks) ?

Are there any tasks that are required to use your site?

What drives revenue on your site?


Selecting the Right Tasks


A Balance of User Data and Business Goals

What are users coming to your site to do (core tasks) ?

- Browsing new sale items

Are there any tasks that are required to use your site?

- Creating an account?

What drives revenue on your site?

- Completing a purchase

Standardize Tasks Across Competitors


Introduction Case Study Results Analysis

Randomize Task Order

Screener Pre-task questions

Task 1

Task 2

Final QuestionsRandomized

Control for ordering effects• Participants become more proficient as they use the site etc.

There are exceptions, sometimes tasks must be completed in

order

Clean the Data

Watch out for Cheaters

• Very Low task times

• 0 Clicks

• Nonsense Answers

• Enable Quality

Controls


http://en.wikipedia.org/wiki/File:Cheating.JPG by Hariadhi

http://en.wikipedia.org/wiki/File:Cheating.JPG

http://en.wikipedia.org/wiki/File:Cheating.JPG

http://commons.wikimedia.org/wiki/User:Hariadhi

Take this with you today:

• Triangulate, use multiple metrics

• Experimental design

• Clean data before analysis

• Want to learn more?

Thanks so much for your attention!

Time for Q&A

Follow us on Twitter @userzoom

Webinar: Retail UX Benchmarking

Documents

Transcript of Webinar: Retail UX Benchmarking