Optimizing eBay - Architecture · eBay Inc. confidential eBay manages … – Over 276,000,000...

Optimizing eBay Improving customer experience at the world’s online marketplace

Elissa Darnell, Director, User Experience Research

Deepak Nadig, eBay Principal Architect

eMetrics Marketing Optimization Summit May 06, 2008

eBay Inc. confidential 2

In the beginning…

“I started eBay as an experiment” — Pierre Omidyar

eBay Inc. confidential

An Amazing Story

1996 Registered eBay

users: 41,000

2008 Registered eBay users: 276 Million


eBay manages … – Over 276,000,000 registered users – Over 1 Billion photos

– eBay users worldwide trade more than $2039 worth of goods every second – eBay averages well over 1 billion page views per day – At any given time, there are over 113 million items for sale on

the site in more than 50,000 categories – eBay stores over 2 Petabytes of data – over 200 times the size of the Library of Congress! – eBay analytics processes over 25 Petabytes of data on any day – The eBay platform handles 4.4 billion API calls per month

The eBay context

Over ½ Million pounds of Kimchi are sold every year!

A sporting good sells every 2 seconds

In a dynamic environment – 300+ features per quarter – We roll 100,000+ lines of code every two weeks

In 39 countries, in seven languages.

>44 Billion SQL executions/day!

An SUV is sold every 5 minutes


Site Statistics: in a typical day…

June 1999

Q1 2007

Growth

Outbound Emails 1 M 41 M 41x

Total Page Views 54 M >1 B 19x

Peak Network Utilization 268 Mbps 16 Gbps 59x

API Calls 0 150 M N/A

Availability ~97% 99.94% 50x

43 mins/day 50 sec/day


Meet the Buyers and Sellers

Some people buy on eBay and other people sell, and some do both

eBay users trade in more than 50,000 categories

eBay’s buyers want a fun shopping experience that provides a great deal

Approximately 1.3M people around the world make all or part of their living selling on eBay


Personal Accounts

7

Who is eBay’s Community?

“eBay Community” is everyone who has a relationship with eBay, Inc.

New Buyers

Collectors

Frequent buyers

One-time buyers

Small Businesses

Experienced Buyers

In-active buyers

Store Sellers

Power Sellers

New Sellers

Casual Sellers

Business Sellers

Hobby Sellers

Merchant Accounts

Pay a friend

Buy online

Top Buyers


What we mean by the whole user experience

It starts by being useful…

Functionally, people must be able to use it…

The way it looks must be pleasing…

This helps create an overall brand experience

It is useful to me. It meets my needs.

I am able to use the product easily (I can perform the tasks it supports).

I like the way the product looks and feels.

My overall feeling about a brand/ product (in the abstract) is good.

Executing well on all of these areas is what creates a great user experience. Research is

needed for each.


Assortment of User Research methods

Participatory Design

Card sorting

Field Visits Lab Testing

Eye Tracking Surveys


Lab-based Usability Testing

• Usability or “lab-testing” – We bring in representative users

individually to our usability labs

– Observe them while they perform assigned tasks

– Use prototypes or the live site

• Enables direct observation of target users as they interact with our web site or a design prototype

– Observe what users ACTUALLY do, rather than relying on what they SAY they do

– Understand WHY people do what they do, not just what they do

• Identify areas that are confusing and potential fixes

• Usability testing done iteratively, throughout the design process


Variations on the standard lab study

Some variations on standard lab tests

• Low Fidelity (Paper) Lab Testing

– Designs are shown on paper

– Researcher or Designer acts as the computer

– Participant uses their finger as the mouse

• RITE Lab Testing

– Stands for Rapid Iterative Test and Evaluation

– Focuses more on rapidly iterating and re-testing the design based on very small sample

– Can be performed on lo-fi (paper) or high fidelity interactive prototypes


Visits

Seeing through the eyes of our customers


Visits: What is it?

• Also known as a Field Study or Ethnography

• Involves going into people’s home, office, where ever they use eBay

• Spending two to three hours with them, observing them use eBay

(or shop online) and listening to them, and sometimes conducting

structured interview

• Unlike Lab Testing, typically formal tasks are not given

• Data is collected through videotaping and taking extensive notes

• Findings are summarized across participants


Visits allow us to observe how people use eBay in a natural context


We get to see people use eBay…

• On their own computers (PCs, laptops, old, new) • With various connection speeds (some dial-up)

• To perform their own tasks (such as selling a camera or book)

• With their own cameras and workspace (living room, office)

• With life’s normal interruptions (talking parrot, cluttered desk)

They tend to let their guard down and we learn things we might not otherwise learn online or in the lab


The focus of Visits

• Questions we focus on in Visits are: – “What is the larger context of use?” – “What issues exist, and WHY?” – “What can we do to address the issues?”

– Visits are NOT about the numbers, or the question, “How many users experienced that?”

– If pervasiveness of a particular issue is of interest, we supplement with survey or analytics data to help quantify the issue


Research methods span the Product Development

Lab Testing RITE Testing Paper Prototypes Eye Tracking Card sorting

Strategic research to inspire Design research to inform Assessment research to track

Understand Conceive Design Develop Launch

Surveys Live-to-site Testing Longitudinal Studies Diary Studies

Field Visits Participatory Design Competitive Evaluation


Research methods

Scripted or lab-based use of product

Natural use of product

Qualitative Quantitative

Self-reported (stated)

Observed Behavior

Mixture

APPROACH

DAT

A SO

UR

CE

Combination / hybrid De-contextualized / not using product

KEY – Context of data collection with respect to product use

Focus Groups / “Voices” Phone Interviews Cardsorting Diary/Camera Study

Intercept Surveys Desirability studies Email Surveys

Message Board Mining

“Visits” / Ethnographic Field Studies Usability Lab Studies (task-based)

Quantitative user experience assessments (e.g., Keynote/Vividence)

(Onsite interviews)

(Extended observation) /

Eyetracking

Usability benchmarking (in lab)

Experimentation

Clickstreams

Data mining

Why How to fix

Behaviors | What people do

What How much

Attitudes | What people believe


Case Study Using Qualitative and Quantitative Research to create a New “View Item” Page

Why redesign the “View Item” Page? • Increase BID/BIN efficiency

• Improve the user experience: Reduce complexity and clutter and inspire buyers to convert

Research Approach • Qualitative and quantitative research to understand user needs and inform design decisions

• Incorporate User Experience Research, Market Research, and Analytics

• Institute research as an integral component of the redesign process (from start to finish)

View Item 2008 View Item 2000 View Item 2003 View Item 2006


Understanding User Needs (Qualitative) • “Compelled” lab study (US) to understand the current experience and how it may be improved

• Participatory design (US, UK, DE) to understand the “ideal” experience to gather user requirements and inform product design and innovation

Concept Testing (Qualitative)

• Focus groups (US) to gauge user reaction to early design concepts and isolate aspects of alternative designs that resonate with users

Iterative Design (Qualitative & Quantitative)

• Rapid Iterative Testing (RITE) (US, DE, IT) to gauge user reaction to an early View Item prototype and make rapid improvements to the design based on user feedback and behavior

• Maximum Differential Survey (US) to determine the relative perceived usefulness of features to Buyers in an effort to streamline and simplify the experience

• Analytics (US) to understand current usage of existing features

20

Research Overview

Visual Design Research (Quantitative)

• Desirability study (US) to determine which visual design approach best conveys target brand attributes (e.g. convenient, clean, safe, fun)

• Tab Visual Design Eyetracking study (US) to explore different visual design treatments for View Item page tabs and determine which best captures and maintains attention as measured by eyetracking data

Longitudinal Research (Q2 2008) (Qualitative)

• Longitudinal Diary Study (US) with functional prototype to understand “real world” usage of View Item over time and determine areas for improvement prior to launch


Compelled Lab Study

Objective

Surface usability issues and perceived strengths and weaknesses of the current View Item page and larger transaction flow

Approach

A “compelled” lab study with users genuinely interested in purchasing items of interest on eBay. Users were asked to find and purchase items of interest on eBay with their real account and payment information in the lab setting

Participants

12 - Mix of experienced and less experienced eBay buyers

All participants were pre-screened to ensure they had a genuine interest in purchasing specific items of interest on eBay in the coming days


Feature Prioritization Maximum Differential Survey

• A key goal was site simplification: Provide buyers with the most important information they need to make their purchase decision

• Traditional surveys focused on understanding feature usefulness typically do not effectively differentiate among features (e.g. majority of features considered useful by 90% or more of buyers)

• A Max Differential survey was conducted in the U.S. to determine the relative usefulness of View Item features to understand which features to include and the relative prominence that should be assigned

• Approximately 5000 buyers and 5000 sellers completed the survey which involved asking users to choose the most useful and least useful feature among different combinations of features to yield a score for each feature

Current bid

Auction countdown

End time

Time left

Real-time updating of bid

Questions and Answers

Your maximum bid

Number of unique bidders

Total Buyers Top 30% Buyers Lower 70% Buyers EXAMPLE

Background and Objectives


Tab Visual Design Eyetracking Study

Goal: To explore different visual design treatments for View Item page tabs and determine which best captures and maintains attention as measured by eyetracking data

On Hover Off

On Hover Off

On Hover Off

Sample Heat map summarizing overall viewing pattern

Sample scanning pattern


How do we know the new “View Item” page will be a success?

Research was conducted throughout the product lifecycle to evaluate the current strategy and design at each stage (prior to further dedication of design or engineering resources)

• Focus groups on early concepts

• Rapid Iterative Testing of early prototype

• Max Differential survey (and Analytics data) to ensure that reduction of page complexity targeted the least useful features

• Quantitative “Desirability” study to ensure that the chosen design approach best conveyed target brand attributes (such as “fun” and “unique”)

• Quantitative Eyetracking study of tab design alternatives to ensure the chosen treatment would best capture and maintain user attention

• A longitudinal diary study (planned in Q2 2008) to gather qualitative real-world usage feedback to ensure a good user experience prior to launch (to be interpreted in conjunction with A/B Test data)


Experimentation

• Setup Experiment Samples Treatments, Factors • Implementation

• User (Experiment, Treatment) • Serve Treatment

• Tracking • Monitoring

• DOE • Define Samples, Treatments, Factors

• Metrics • Reporting • Idea (!)

• Learning

1. Hypothesis

2. Experimental Design

3. Setup Experiment

4. Launch Experiment

5. Measurement

7. Analysis & Results

eBay Experimentation

Platform


Automation

• Dynamically adapt experience – Choose page, modules, and inventory which provide best experience for that user and

context – Order results by combination of demand, supply, and other factors (“Best Match”)

• Feedback loop enables system to learn and improve over time – Collect user behavior – Aggregate and analyze offline – Deploy updated metadata – Decide and serve appropriate experience

• Best Practices – “Perturbation” for continual improvement – Dampening of positive feedback


Experimentation Platform

27

Experience

Page, Module

Experiment Metadata Experience

Response

Metrics / Experience

Results Observations

Design

Access

Results Observations

Access

Metrics / Experience


What we think about

28

Type/Level of Experiment

Signal/Noise Ratio

Concurrency

Iteration time

Cost of Experiments

Fidelity of Experiments

The total cost of designing, building, running, and analyzing an experiment

The time from hypothesis to when the analyzed results are available for planning the next iteration

Number of experiments that can be run at the same time

The extent to which the metrics are obscured by experimental noise

Extent to which the model and its conditions represent the final feature or product

Supporting different types (A/B, 1-FAT, DOE) and levels (page, module, page flow) of experiments


Challenges …

• Sticky-ness to user

• What, not why

• Duration and long term effects

• Minor vs. major differences

• Extent of generalization


Qualitative research such as lab tests and field visits give us rich data about • Usability problems • Discoverability • Navigation • Terminology • More complex problems

Quantitative research such as surveys, usage studies, and log analysis tells us • How pervasive a problem is • What features on the site are actually being used

The combination of qualitative and quantitative data give us a more complete picture that is most powerful

The Power of Combining Data


How we get our data differs…

Analytics Measure what customers do through serving experiences and tracking responses

Research Work directly with customers by…

• Talking to them

• Observing them

• Surveying them

The real power is from looking at our user’s from both sides!


Summary

• Understanding the customer experience requires insights into – What they do – Why they do it – Attitudes, motivations and behaviors

• A variety of research methods, both qualitative and quantitative, can be used.

• Each of the methods – qualitative and quantitative - have their advantages and limitations.

• Using them together continues to help eBay gain a holistic understanding of the user experience and aspects that need to be improved.


Questions

Elissa Darnell

[email protected]

Deepak Nadig

[email protected]

Optimizing eBay - Architecture · eBay Inc. confidential eBay manages … – Over 276,000,000...

Documents

Transcript of Optimizing eBay - Architecture · eBay Inc. confidential eBay manages … – Over 276,000,000...