Jon Bach

eBay, Inc.

With more than eighteen years of experience in software testing, Jon Bach has held technical

and managerial positions in companies including Hewlett-Packard and Microsoft. In his current

role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests

(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.

He is most notable for creating, with his brother James, Session-Based Test Management, a

method to manage and report exploratory testing.

9/23/2013

1

Selling (and Buying)

Live Site Quality) at

Jon Bach

QE Director, Live Site Quality

[email protected]

STAR West 2013

February issue

mailto:[email protected]

9/23/2013

2

What’s the quality right NOW?

Selling and Buying Live Site Quality

Is it good enough?

Good enough quality bar

Further time and effort

is crucial.

Further time and effort

is a waste of resources.

Unacceptable Quality

Too Much Quality

floating line


9/23/2013

3

Heuristics of quality

Bad Buyer Experiences

Gross Merchandise Volume (+)

Net Promoter Score

Earnings Per Share

Social Media chatter

Morning Mission Report

eWatch’s Most Wanted

Re-rolls and Rollbacks

Call volume

Mean Time to Detect

Mean Time to Mitigate

5 Selling and Buying Live Site Quality

Heuristics of quality

Overdue bug fixes

Survey results

Tell Us What You Think

Metrics: DoD / Wow / MoM / YoY

Key Flow Test Results

eCommerce Bytes

#ebaysux

“Be The Customer”

Discussion Forum topics

Calls to eBay Radio

Emails from VOICES


9/23/2013

4

What I do (“Director of Live Site Quality”)


“Director of Live Site Quality”


Jon the Bug Hunter on eBay Radio

Call customers who participate in our NPS callback program

Speak at VOICES meet-ups

9/23/2013

5



Live Site Ledger

P1 Bugs found this week (with Root Cause Analysis)

Overdue bugs that need fixing

What bad things happened in the last 24 hours?

Live Site Ledger

P1 Bugs found this week (with Root Cause Analysis)

Overdue bugs that need fixing

What bad things happened in the last 24 hours?

Teach New Hires How to Explore



Make sure eWatch is in the loop on issues

Find out who *really* owns this bug

Drive issues that threaten the notion of “site readiness”

9/23/2013

6



Coordinate teams in end-to-end testing

Host bug bashes that focus on exploration

Manage crowdsourced testing efforts in production

What WE do … especially after release

Monitoring

Experimentation

Surveys

Scenario

Follow-up

Investigation

Competitive

Events

Reputation

Dogfooding


9/23/2013

7

Monitoring (Checking)


Experimentation (A/B testing)

14

A

B


9/23/2013

8

Surveys

When site visitors click this…


…we see this:


9/23/2013

9


Follow-up on “UFO reports”

-------------------------------------------------------------------------------

“I have attached the screen shot for you. Try as I might, I can't get

the selling reminders to go to the top. The arrows don't work, but I

can move sold, unsold and active selling. I even set the default to

summary, but no luck. FYI- I am using Safari on my trusty Mac.

Also, the default font size is too big. When I zoom it out, it's too

small. It was a better size the way it was before. I sent a message

to the team through the 'what do you think' for that page.”


9/23/2013

10

Investigative Testing

What questions would you have to troubleshoot a dip in metrics like

this?


Topics for investigation (FORCOSTTTIP)

•Feature – Do we know what happens when people bid?

•Origin -- When did this start happening?

•Recent – Did we just push code to Production?

•Consistency / Correlation -- What happened yesterday, tomorrow?

Are there other graphs and metrics?

•Overlap – Is there any other process happening?

•Security – Is it malicious? Intentional?

•Terminology -- What kinds of “bids”? US, UK, International?

•Third-Party / Silent Partner – What’s not in the room? What sort of

information have we NOT heard from?

•Truth is out there – does someone already know about this?

• Instrumentation – is the graph wrong? Can I get more data?

•Prediction – can we test a conjecture with a prediction


9/23/2013

11

Competitive Testing

21 Testing… After You’ve Finished Testing

“Site Readiness”

A large seller on eBay wants to have a “Flash Sale”.

Are we ready?


9/23/2013

12

Reputation


Dogfooding

Video of a flow test:


C:/Users/jobach/Documents/E2ESOL docs/video specs/flow1_--_search-VI-sellonelikethis-signin-shortform-myebay-searchtoverifyitemID.swf

9/23/2013

13

Ask my brother to round up a posse…

Rapid Testing Intensive

July 2012

• 80 testers from around the world – in-person and online

* 4 days of testing eBay Motors: Tire Finder / Wheel Center

•How to jump into a product and find bugs while you learn all about

it, while working toward "deep coverage" testing.

•Charter-based exploration, modeling, usability testing, intense

coverage to deliver notions of quality and risk


What I’ve learned

Once they see it, they may want it

There is no sale without a buyer

A buyer can return an item being sold


9/23/2013

14

Most Important Message…

27

I AM LIVE SITE QUALITY

(AND SO CAN YOU!)

JON BACH



Back at your desk on Monday

•Run Experimentation??? Try crowdsourced charters

•Create monitoring??? Take an Ops guy to lunch

•What to investigate… Customer Support’s Most Wanted

•Make surveys??? Interview a Business Analyst

•How to make scenarios… Try a flow exercise with legos

•What’s our reputation??? Scan user groups

•Follow-up on what??? Talk to an early adopter

•Dogfood a medical device? Watch a live study or trial

•Capabilities? What??? See a marketing roadmap

•Competitive what now? Explore a competitor URL

•Usability, but no lab? Try Open-Book Testing


Keynote: Selling (and Buying) “Live Site Quality” at eBay

Technology

Transcript of Keynote: Selling (and Buying) “Live Site Quality” at eBay