World-Class Web Metrics by Dan Olsen

Copyright © 2009 YourVersion

Dan OlsenDan OlsenCEO, YourVersionCEO, YourVersionJune 17, 2009June 17, 2009

World‐Class Web Metrics:How to Optimize Your Product & Business

http://www.yourversion.com/


World‐Class Metrics:Amazon.com

$37 B Market Cap$19 B Revenue ‘08

•Automated testing of page elements & suggested products

•High degree of user personalization

•Real‐time recommendations


World‐Class Metrics:Google

$137 B Market Cap$22 B Revenue ‘08

•Auto‐testing ofUI elements & search results

•Automatically learns from user mistakes

•Optimize ads to maximize revenue


OverviewOverview

What is being World‐Class in Metrics?

Optimizing Business Value

Optimizing the Customer Value ofYour Product


My Web Metrics EpiphanyQuicken (1998 – 2001)

New version every year: shrink‐wrapped CD‐ROMPre‐launch

Before development: Validated product ideas with usersDuring development: Usability testing & user feedback

Post‐launchNo “real‐time” metrics dataSales data: 1‐2 month lag, not actionableUser surveys: actionable, but not until next product cycle

Quicken Brokerage (2001 – 2003)Online brokerage launched in September 2002The day after launch: LOTS of metrics data → sought moreMuch faster iteration: metrics → hypothesis → improvement


Metrics Maturity ModelUnaware Aspiring World‐Class

MetricsTracking None Sporadic Comprehensive

breadth & depth

Analysis None Ad hoc Recurring

Testing & Optimization None Some A/B

tests

Automated multivariate

testing

Organization Lack of ownership

Partial ownership

Clear, dedicated owners

Culture Intuition‐based

Data‐driven

Continuous improvement


Optimizing Business ValueOptimizing Business Value


Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise

Given reality as it exists today,Given reality as it exists today,

optimize our business resultsoptimize our business results

subject to our resource constraints.subject to our resource constraints.


( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conve( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Ratersion Rate

Paying UsersPaying Users x x Revenue per Paying UserRevenue per Paying User

New Paying UsersNew Paying Users + + Repeat Paying UsersRepeat Paying Users

Previous Paying Users x ( 1 Previous Paying Users x ( 1 –– Cancellation Rate )Cancellation Rate )Trial UsersTrial Users x x ConvConv RateRate

Profit = Profit = RevenueRevenue ‐‐ CostCost

Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””


How to Track Your MetricsTrack each metric as daily time series

Create ratios from primary metrics: X / YExample: How good is your registration page?Okay: # of registered users per dayBetter: registration conversion rate =

# registered users / # uniques to reg page

DateUnique Visitors

Page views

Ad Revenue

New User Sign‐ups …

4/24/08 10,100 29,600 25 490

4/25/08 10,500 27,100 24 480

…


Sample Signup Page Yield DataSample Signup Page Yield Data

Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10

Dai

ly S

ignu

p Pa

ge Y

ield

Changedmessaging

Added questionsto signup page

Started requiringregistration


Identifying the Identifying the ““Critical FewCritical Few”” MetricsMetrics

What are the metrics for your business?Where is current value for each metric? How many resources to “move” each metric?

Developer‐hours, time, moneyWhich metrics have highest ROI opportunities?

Return

Return

InvestmentInvestment

Return

Return

InvestmentInvestmentRe

turn

Return

InvestmentInvestment

Metric AMetric AGood ROIGood ROI

Metric BMetric BBad ROIBad ROI

Metric CMetric CGreat ROIGreat ROI


Turning Metrics into Meaning & Action

• Clear owner for each key metric

•Focus on trend & changes

•Discuss business & customer impact

•Propose ideas to improve

From Avinash Kaushik’s Occam’s Razor at www.kaushik.net


Optimizing the Customer Optimizing the Customer Value of Your ProductValue of Your Product


Using Analytics to Clarify TheFuzzy Math of Customer Value

You create customer value with a product thatSatisfies customers’ needs

Is easy to use

Has a good price

Is better than other alternatives

Offline: hard to measure

Online: can measure everythingMeasuring value: # of users, repeat use, revenue

Can achieve clarity with metrics and analytics


Once You’re Measuring Everything, Where Do You Focus?

How do you decide which user benefits and features you should focus on?

Need framework to make decisions“Problem space” vs. “Solution space”

Importance vs. SatisfactionImportance of user need (problem space)

Satisfaction with how well your product meets the user’s need (solution space)


Importance vs. SatisfactionAsk Users to Rate for Each Feature

98

8784

8679 847055 80

7280

75

4150

55

60

65

70

75

80

85

90

95

100

40 50 60 70 80 90 100

Satisfaction

Impo

rtan

ce

Recommended reading: “What Customers Want” by Anthony Ulwick

BadBad

GreatGreat


Kano Model: User Needs & Satisfaction

User SatisfactionUser Satisfaction

User DissatisfactionUser Dissatisfaction

Performance (more is better)

Delighter (wow)

NeedNeednot metnot met

NeedNeedfully metfully met

Must Have

Needs & features migrate over time


Is the site up when I want to use it?

Is the site fast enough?

Does the functionality work?

Does the functionality meet my needs?

Olsen’s Hierarchy of Web User Needs(adapted from Maslow)

Customer’s Perspective What does it mean to us?

Uptime

Page Load Time

Absence of Bugs

Feature Set

Usability & Design

Decreasing

Dissatisfaction

Increasing S

atisfaction

How easy to use is it?


Using Web Analytics Tools to Understand Your Users

Tracking visitors and traffic

Seeing where users are clicking

Measuring key conversions

Monitoring user feedback


Basic Tracking of Traffic:Google Analytics

•Unique visitors

•New vs. returning

•Pageviews

•Time on site

•Top referrers

•Top geos


Seeing Where Users are Clicking: CrazyEgg Heatmap

•Shows Click Density

•Color indicates % of clicks

•See which links perform best

•See impact of UI changes


Measuring Key Conversions:Conversion Funnel

•Tie user actions to business goals

•Instrument key steps in user flow

•See where users are dropping off

•Quantify improvement from changes


Monitoring User Feedback Quantitatively with Kampyle

Simple popup

UnobtrusiveSolicitation Average Grade over time

Average Grade for each page on your site


The UI Design IcebergThe UI Design Iceberg

VisualDesign

InteractionDesign

InformationArchitecture

ConceptualDesign

Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net

What most people seeand react to

What good PMs and Designers think about


UI questions are never yes/no! (not binary)Should ask: “What percentage of users …?”UI changes impact your metrics

Impact can be positive, negative, small, largeSeek high‐ROI UI changes

Typical UI design question:“When using web pages, do users scroll down?”

‐ Yes‐ No

Approaching UI Design Analytically

Copyright © 2009 YourVersionCopyright Copyright ©© 2009 Olsen Solutions LLC2009 Olsen Solutions LLC

Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold

Landing Page ALanding Page A Landing Page BLanding Page B

Key conversion action is above the fold

Key conversion action is below the fold

The Fold


Marrying UI Design with MetricsThink of “Ease of use”as distinct from the functionalityUser Interface elements matter

PositionLayoutSizeColorFontText copyInteraction designNavigation


Levels of Optimization Maturity

A/B testing vs. Multivariate testingA/B: test 2 alternatives (ideally same time)

Multivariate: test multiple variables at once

Degree of automation & frequencyManual → Automated → Continuous real‐time

Level of personalizationUniversal: same for all users

Personalized: varying based on user, user attributes, or user segment


Automated Multivariate Testing:Google Website Optimizer


Organization and CultureImpact Product Success

Roles and Skills requiredProduct Management (PM)UI DesignDevelopmentQuality Assurance (QA)Operations

Cultural traits requiredCustomer‐centricData‐driven decision‐making with clear ownershipRapid, iterative developmentContinuous improvement mindset


The Product Manager’s Job:A Successful Product

Be the expert on the market and the customer

Translate business objectives and customer needs into product requirements

Be the clearinghouse for all product ideas

Work with team to design & build great product

Define and track key metrics

Use metrics to drive continuous improvement

Identify, plan & prioritize product ideas to maximize ROI on engineering resources


Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI

Investment (developer‐weeks)

Return (V

alue

Created

)

Idea C

Idea B

Idea D

Idea A

Idea F

1

1

2

3

4

2 3 4

?


OpenAccount

Sign in

Account Selection

Register

56%

44%

Forget Password

Registration Process

45% drop off(20% of total)

36% overall drop off for this step

70%(32% of Total)

17% drop off (10% of total)

20% drop off(6% of total)

30%(14% of Total)

80%(26% of Total)

55%(24% of Total)

64%of Total

Case Study 1: Quicken BrokerageCase Study 1: Quicken BrokerageOptimizing Sign In/Registration FlowOptimizing Sign In/Registration Flow

Change Password

83%(46% of Total)


Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion Rate 37%Registration Conversion Rate 37%

37% improvement in conversion rate

ReleasedNew Design


• Which metric offers the highest ROI opportunity?

Case Study 2: FriendsterCase Study 2: FriendsterOptimizing Viral GrowthOptimizing Viral Growth

ActiveUsers

Prospective Users

Invite Click

Succeed

Inviteclick-through rate

Conversion rate

Don’t Click

Fail

Invites per sender

% of users sending invites

• Multiplied together, these metrics determine your viral ratio

Users

% of users who are active

= 15% = 2.3

= 85%

Registration Process

= 20%

= 50%


Case Study 2: FriendsterCase Study 2: FriendsterDoubled Number of Invitations Sent per SenderDoubled Number of Invitations Sent per Sender


ContinuousContinuousImprovementImprovement


Adding Metrics and Optimization to your Product Process

PlanPlan

DesignDesign

DevelopDevelop

BusinessObjectives

ProductObjectives

Prioritized Feature List

Scoping

Requirements & Design

Code Test Launch

Site Level

Feature Level

OptimizeOptimize Metrics & User Feedback


Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement

Measurethe metric

Analyzethe metric

Identify top opportunitiesto improve

Design & develop the enhancement

Launch theenhancement

Learning

Gaining knowledge:

• Market

• Customer

• Domain

• Usability


Summary:How to be World‐Class in Web MetricsDefine the equation of your business

Identify and track your key metrics

Use analytics to gain customer insights

Establish clear roles and data‐driven culture

Identify opportunities and prioritize by ROI

Launch, learn, and iterate


Questions? [email protected]

http://www.yourversion.com/

World-Class Web Metrics by Dan Olsen

Business

Transcript of World-Class Web Metrics by Dan Olsen