www.leananalyticsbook.com@leananalytics

@byosko | @acroll

Lean Analytics

Use data to build abetter business faster.

Some background on Lean, analytics, metrics, and

segmentation.

Hotmailwas a

database company

Flickrwas going to be an

MMO

Twitterwas a

podcasting company

Autodeskmade

desktop automation

Paypalfirst built for Palmpilots

Freshbooks

was invoicing for

a web design firm

Wikipediawas to be written by

experts only

Mitelwas a

lawnmower company

Most startups don’t know what they’ll be when they grow up.

Possible problem space

Product/market

hypothesis

Trial startup

Product/market

hypothesis

Trial startup

Product/market

hypothesis

Tria

l sta

rtup

Product/market

hypothesis

Tria

l sta

rtup

You are

herePIVOT

Kevin Costner is a lousy entrepreneur.Kevin Costner is a lousy entrepreneur.

Don’t sell what you can make.Make what you can sell.

5 things you need to know about metrics

Qualitative or Quantitative

Exploratory or Reporting

Vanity or Actionable

Correlated or Causal

Leading or Lagging

QualitativeQualitative Quantitative

Quantitative

Unstructured, anecdotal, revealing, hard to aggregate.

Unstructured, anecdotal, revealing, hard to aggregate.

Numbers and stats; hard facts but less insight.

Numbers and stats; hard facts but less insight.

http://www.flickr.com/photos/zooboing/8388257248/ http://www.flickr.com/photos/x1brett/4665645157/

Warm and fuzzy.Warm and fuzzy.

Cold and hard.Cold and hard.

Exploratory ReportingSpeculative, trying to find unexpected or interesting insights.

Predictable, keeping you abreast of normal, managerial operations.

http://www.flickr.com/photos/elwillo/4737933662/http://www.flickr.com/photos/50755773@N06/5415295449/

Donald Rumsfeld on analytics

(Or rather, Avinash Kaushik channeling Rumsfeld)

Things we

know

don’tknow

we know

we don’tknow

we know

we don’tknow

Are facts which may be wrong and should be checked against data.

Are facts which may be wrong and should be checked against data.

Are questions we can answer by reporting, which we should baseline & automate.

Are questions we can answer by reporting, which we should baseline & automate.

Are intuition which we should quantify and teach to improve effectiveness, efficiency.

Are intuition which we should quantify and teach to improve effectiveness, efficiency.

Are exploration which is where unfair advantage and interesting epiphanies live.

Are exploration which is where unfair advantage and interesting epiphanies live.

VanityVanity

Makes you feel good, but doesn’t change how you’ll act.

Picks a direction.

ActionableActionable

http://www.flickr.com/photos/lostseouls/807253220/ http://www.flickr.com/photos/aussiegall/6382775153/

HitsA metric from the early, foolish days of the Web. Count people instead.

Page viewsMarginally better than hits. Unless you’re displaying ad inventory, count people.

VisitsIs this one person visiting a hundred times, or are a hundred people visiting once? Fail.

Unique visitorsThis tells you nothing about what they did, why they stuck around, or if they left.

Followers/friends/likes

Count actions instead. Find out how many followers will do your bidding.

Time on site, or pages/visit

Poor version of engagement. Lots of time spent on support pages is actually a bad sign.

Emails collected

How many recipients will act on what’s in them?

Number of downloads

Outside app stores, downloads alone don’t lead to lifetime value. Measure activations/active accounts.

Correlated CausalTwo variables that change in similar ways , perhaps because they’re linked to something else.

An independent factor that directly impacts a dependent one.

Summer

Ice creamconsumption

DrowningCorrelated

Causa

l Causal

Causality is a superpower, because it lets you change the future.

Correlation lets you predict the

future

Causality lets you change the future

“I will have 420 engaged users and 75 paying customers next month.”

“If I can make more first-time visitors stay on for 17 minutes I will increase sales in 90 days.”

Find correlation

Test causality

Optimize the causal

factor

Leading LaggingNumber today that shows metric tomorrow—makes the news.

Historical metric that shows how you’re doing—reports the news.

A leading indicator for e-commerce

Acquisition

Hybrid

Loyalty

70%of retailers

20%of retailers

10%of retailers

You are just like

How many of your customers buy a second

time in 90 days?

Once

2-2.5per year

>2.5per year

Your customers will buy from

you

Then you are in this mode

1-15%

15-30%

>30%

Low CAC, high

checkout

Increasing returns

Loyalty, inventory expansion

Focus on

(Thanks to Kevin Hilstrom for this.)

Segments, cohorts, A/B, and multivariates

Segment:Cross-sectional comparison of

all people divided by

some attribute (age, gender,

etc.)

☀

☁

Cohort:Comparison of similar groups along a timeline.

A/B test:Changing one thing (i.e. color) and measuring the result (i.e. revenue.)

MultivariateanalysisChanging several things at once to see which correlates with a result.

☀☁☀☁

Why use cohorts? Here’s an example.

  JanuaryFebruar

yMarch April May

Rev/customer

$5 $5 $4 $4 $5

Is this company growing

or stagnatin

g?

Cohort 1 2 3 4 5

January $5 $3 $2 $1 $1

February $6 $4 $2 $1

March $7 $6 $5

April   $8 $7

May       $9

How about now?

Why use cohorts? Here’s an example.

Cohort 1 2 3 4 5

January $5 $3 $2 $1 $1

February $6 $4 $2 $1  

March $7 $6 $5    

April $8 $7      

May $9        

Averages $7 $5 $3 $1 $1

Look at the same data in cohorts

Frameworks we borrowed from.

AARRR

Dave McClure’s Pirate metrics

Acquisition

How do your users become aware of you?SEO, SEM, widgets, email, PR, campaigns,

blogs ...

Activation

Do drive-by visitors subscribe, use, etc?Features, design, tone, compensation, affirmation

...

Retention

Does a one-time user become engaged?

Notifications, alerts, reminders, emails, updates...

Revenue

Do you make money from user activity?Transactions, clicks, subscriptions, DLC,

analytics...

Referral

Do users promote your product?Email, widgets, campaigns, likes, RTs, affiliates...

Eric Ries’Three engines

Virality

Make people invite friends

How many they tell, how fast they tell

them

Price

Spend revenue getting

customers

Customers are worth more

than they cost to get

Stickiness

Keep people coming back

Approach

Get customers faster than you

lose them

Math that matters

Ash Maurya’sLean canvas

Lean Canvas box Some relevant metrics

Problem Respondents who have this need; respondents who are aware of having the need.

Solution Respondents who try the MVP; engagement; churn; most-used/least-used features; people willing to pay.

Unique Value Proposition

Feedback scores; independent ratings; sentiment analysis; customer-worded descriptions; surveys; search and competitive analysis.

Customer Segments How easy it is to find groups of prospects; unique keyword segments; targeted funnel traffic from a particular source.

ChannelsLeads/cust per channel; viral coefficient and cycle; net promoter score; affiliate margins; open, click-through rate; PageRank; message reach.

Unfair AdvantageRespondents’ understanding of the USP; patents; brand equity; barriers to entry; number of new entrants; exclusivity of relationships.

Revenue Streams Lifetime customer value; Average revenue per user; Conversion rate; Shopping cart size; Click-through rate.

Cost Structure Fixed costs; cost of customer acquisition; cost of servicing the nth customer; support costs; keyword costs.

Sean Ellis’Startup growth pyramid

Scalegrowth

Stack the odds

Product/market fitDecide what you sell to whom, then prove it.

Find a defensible unfair advantage and tweak it.

Step on the gas in new markets, products, channels.

The business model flipbook.

SoftwarePlatformMerchandisingUser-generated content MarketplaceMedia/contentService

Oracle’s accounting suiteAmazon’s EC2 cloudThinkgeek’s retail storeFacebook’s status updateAirBnB’s list of house rentalsCNN’s news pageA hairstylist

Pro

du

ctty

pe

What the startup does in return. May be a product or service; may be hardware or software; may be a mixture.

One-time transactionRecurring subscriptionConsumption chargesAdvertising clicksRe-sale of user dataDonation

Single purchase from FabMonthly charge from FreshbooksCompute cycles from RackspacePPC revenue on CNET.comTwitter’s firehose licenseWikipedia’s annual campaignR

even

ue

mod

el How the startup

extracts money from its visitors, users, or customers.

Paid advertisingSearch Engine Mgmt.Social media outreach Inherent viralityArtificial viralityAffiliate marketingPublic relationsApp/ecosystem mkt.

Banner on Informationweek.comHigh pagerank for ELC in kid’s toysActive on Twitter i.e. KissmetricsInviting team member to AsanaRewarding Dropbox user for others’ signupsSharing a % of sales with a referring bloggerSpeaker submission to SXSW Placement in the Android market

Acq

uis

itio

nch

an

nel

How the visitor, customer, or user finds out about the startup.

Hosted serviceDigital deliveryPhysical delivery

Salesforce.com’s CRMValve purchase of desktop gameKnife shipped from Sur La Table

Deliv

ery

mod

el

How the product gets to the customer.

Simple purchaseDiscounts & incentives Free trialFreemiumPay for privacyFree-to-play

Buying a PC on Dell.comBlack Friday discount, loss leader, free shipTime-limited trial such as fitbit PremiumFree tier, relying on upgrades, like EvernoteFree account content is public, like SlideshareMonetize in-app purchases, like Airmech

Selli

ng

tact

ic

What the startup does to convince the visitor or user to become a paying customer.

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

Inherent virality.

Artificial virality.

Sharing files with others.

Free storage when others sign up.

Freemium.Limited-capacity accounts are free; subscribe when you need more.

Recurring subscription.

$99/year, monthly fees, enterprise tiers.

Platform.Storage-as-a-service with APIs, collaboration, synchronization tools.

Hosted service.

Digital delivery.

Cloud storage, web interface.

Desktop client software.

Dropbox example

The six business models.

E-commerce

TL;DR:•Are you focused on loyalty or acquisition?•Pricing matters more than you think•Don’t overlook logistics, delays, and ratings•Old “average conversion rates”

e-Commerce: metrics that matter

• Conversion rate (the number of visitors who buy something.)• Purchases/year (the number of purchases made by each customer

per year)• Average shopping cart size (the amount of money spent in a

purchase.)• Abandonment (the percentage of people that begin to make a

purchase, and then don’t.)• Cost of customer acquisition (the money spent to get someone to

buy something.)• Top keywords driving traffic to the site.• Top search terms that lead to revenue; top search terms that don’t

have any results.• Effectiveness of recommendation engines (how likely a visitor is to

add a recommended product to their cart.)• Virality (word of mouth, and sharing per visitor.)• Mailing list effectiveness (click-through rates and ability to make

buyers return and buy.)

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

Acquisition-focused (Virality, SEO/SEM, paid advertising).

Word of mouth, sharing purchases, affiliates. Less emphasis on loyalty programs or return purchases.

Simple purchase, with discounts.

Seasonal specials, free shipping.

One-time transaction.

Price of purchase.

Merchandising.Bulk-breaking, assortment, and distribution of physical goods.

Physical delivery.Ground shipping of purchases via courier service.

E-commerce example

Software-as-a-Service

TL;DR:•Eventually, focus on Customer Acquisition Payback•Engagement varies by intended use of the app (i.e. CRM versus travel booking)•Credit cards up front have a huge effect•Freemium is a sales tactic, not a business model•Churn, acquisition cost, and lifetime value•Subscriptions may be a bad thing (per-transaction pricing is an option too.)

SaaS: metrics that matter

• Attention (how effectively can you get people to visit?)• Enrollment (how many visitors become users?)• Stickiness (how much do customers use the product?)• Conversion (how many users become paying customers?)• Revenue per Customer (how much money do you make from a

customer per period?)• Customer Acquisition Cost (how much does it cost to get a paying

user?)• Virality (how likely are customers to invite others and spread the

word?)• Upselling (what makes customers increase their spending, and how

often does that happen?)• Uptime and reliability (how many complaints, problem escalations, or

outages do you have?)• Churn (how many users and customers leave in a given time period?)• Lifetime Value (how much are customers worth from cradle to grave?)

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

Inherent virality.Invite others to a collaborative project as part of the natural use of the tool.

Free trial.Credit card up front, no charge for 30 days.

Recurring subscription.

$20 a month, per seat.

Software.Collaborative project management tool.

Hosted service.Run on a cloud platform, delivered via a web browser.

SaaS example

Mobile

TL;DR:•App stores make or break you•In-app revenue is the only way to make a living

Mobile: metrics that matter

• Downloads: how many people have downloaded the application• Related metrics: app store placement, downloads and ratings

• Customer Acquisition Cost: How much does it cost to get a paying use?

• Launch rate: What percentage launch the app and create an account?• Percent Active Users/Players: What percent use it on a daily/monthly

basis?• This is your daily active users (DAU), and monthly active users (MAU)

• Percentage of users who pay• Time to first purchase• Monthly average revenue per user (or ARPU)

• Also track ARPPU, which is the average revenue per paying user.• Ratings click-through: Percent that review an app• Virality: How many other users does a user invite?• Churn: How many customers have uninstalled the application

• Or haven’t launched it in 90 days?• Customer Lifetime Value: How much is a user worth from cradle to

grave?

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

App/ecosystem market, paid ads in other games.

Lobbying Apple & Android; paid downloads.

Free to play. No charge for basic game.

One-time transactions.Re-selling user data.

Purchase of in-game components.Licensing of usage data to 3rd party.

Software.Collaborative project management tool.

Digital delivery.Game and extra content delivered to the mobile device.

Mobile app example

Media

TL;DR:•Advertising is a complicated, many-faced beast

Media: metrics that matter

• Audience and churn: How many people visit the site, and do they keep coming back?

• Ad inventory: How many impressions do we have that we can monetize?

• Ad rates: What might we be paid for those impressions, based on the content we cover the people who visit us?

• Click-through rates: How many of those impressions actually turn into money?

• The content/advertising balance: Are we properly balancing money, ad inventory, and content to maximize overall performance?

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

SEO/SEM, social media outreach, app/ecosystem market.

Publishing to social and RSS feeds; promoting tablet/mobile app.

N/A (no transaction).

Have good content; make it relevant to desirable ad placement.

Ad clicks.Banner, PPC, and affiliate links generate revenue.

Media/content. News site with editorial content.

Hosted service, digital delivery.

Media is online but also available through downloadable app.

Media example

User-generated content

TL;DR:•Content virality and user virality•Trying to move users up the engagement funnel•More time on fraud than you expect•Notifications and email are the real user interface•Passive content creation is on the horizon

UGC: metrics that matter

• Number of engaged visitors: How often do people come back, and how long do they stick around?

• Content creation: What percentage of visitors interacts with content in some way, from creating it to voting on it?

• Engagement funnel changes: How well is it moving people to more engaged levels of content over time?

• Value of created content: What is the business benefit of content?

• Content sharing and virality: How does content get shared, and how does this drive growth?

• Notification effectiveness: When we tell a user something, do they act on it?

UGC engagement funnels

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

SEO/SEM, social media outreach, natural/artificial virality.

Publishing to social and RSS feeds; encouraging inviting & sharing with friends.

N/A (no transaction).

Have good content; make it relevant to desirable ad placement.

Ad clicks, re-sale of user data.

Banner, PPC, and affiliate links; licensing user data feed.

User-generated content.

Visitors submit and vote on content.

Hosted service.Available via web and mobile browsers.

UGC example

2-sided market

TL;DR:•Focus on the money side•Breaking the chicken-and-egg

2-sided market: metrics that matter

• Buyer and seller growth: The rate at which you’re adding new buyers and sellers, as measured by return visitors.

• Inventory growth: The rate at which sellers are adding inventory (such as new listings,) as well as completeness of those listings.

• Search effectiveness: What buyers are searching for, and whether it matches the inventory you’re building.

• Conversion funnels: The conversion rates for items sold, and any segmentation that reveals what helps sell items (such as the professional photographs of a property mentioned in the AirBnB example earlier in the book.)

• Ratings and signs of fraud: The ratings for buyers and sellers, signs of fraud, and tone of the comments.

• Pricing metrics: If you have a bidding method in place (as eBay does) then you care whether sellers are setting prices too high or leaving money on the table.

Acquisitionchannel

Sellingtactic

Revenuemodel

Producttype

Deliverymodel

Business aspect

Flipbookpage(s)

Loyalty-focused (Virality, email, engagement).

More emphasis on returning users (stickiness) than acquiring new ones.

Simple purchase.Emphasis on search results, accuracy of buyer/seller matches.

One-time transaction; subscription

Percentage of each transaction as a fee; premium vendor services/prominent listing are extra.

Marketplace.Bringing together buyers and sellers for a price.

Physical delivery.Labelling and logistics services for sellers run through the site.

2-sided market example

The five stages of Lean Analytics

Where is the risk?

Healthy market?

Sustainable biz?

Good product?

Right solution?

Real need?

Successful exit?

Key: EmpathyKey: Empathy

Key: Stickiness

Key: Stickiness

Key: ViralityKey: Virality

Key: RevenueKey: RevenueK

ey:

Gro

wth

rate

Key:

Gro

wth

rate

Key: ScaleKey: Scale

Combining stage and model to find your

OMTM.

What’s your OMTM?

Empathy

Stickiness

Virality

Revenue

Scale

E-commerc

eSaaS Media

Mobileapp

User-gencontent

2-sidedmarket

Interviews; qualitative results; quantitative scoring; surveys

Loyalty, conversion

CAC, shares, reactivation

Transaction, CLV

Affiliates, white-label

Engagement, churn

Inherent virality, CAC

Upselling, CAC, CLV

API, magic #, mktplace

Content, spam

Invites, sharing

Ads, donations

Analytics, user data

Inventory, listings

SEM, sharing

Transactions, commission

Other verticals

(Money from transactions)

Downloads, churn, virality

WoM, app ratings, CAC

CLV, ARPDAU

Spinoffs, publishers

(Money from active users)

Traffic, visits, returns

Content virality, SEM

CPE, affiliate %, eyeballs

Syndication, licenses

(Money from ad clicks)

Example: a restaurant

• Empathy: Before opening, the owner first learns about the diners in its area, their desires, what foods aren’t available, and trends in eating.

• Stickiness: Then he develops a menu and tests it out with consumers, making frequent adjustments until tables are full and patrons return regularly. He’s giving things away, testing things, asking diners what they think. Costs are high because of variance and uncertain inventory.

• Virality: He starts loyalty programs to bring frequent diners back, or to encourage people to share with their friends. He engages on Yelp and Foursquare.

• Revenue: With virality kicked off, he works on margins—fewer free meals, tighter controls on costs, more standardization.

• Scale: Finally, knowing he can run a profitable business, he pours some of the revenues into marketing and promotion. He reaches out to food reviewers, travel magazines, and radio stations. He launches a second restaurant, or a franchise based on the initial one.

Example: a software company

• Empathy: The founder finds an unmet need, often because she has a background in a particular industry or has worked with existing solutions that are being disrupted.

• Stickiness: She meets with an initial group of prospects, and signs contracts that look more like consulting agreements, which she uses to build an initial product. She’s careful not to commit to exclusivity, and tries to steer customers towards standardized solutions, charging heavily for custom features. She supports the customers directly from the engineering team until the product is stable and usable.

• Virality: Product in hand, she asks for references from satisfied customers, and uses them as testimonials. She starts direct sales, and grows the customer base. She launches a user group, and starts to automate support. She releases an API, encouraging third-party development and scaling potential market size without direct development.

• Revenue: She focuses on growing the pipeline, sales margins, and revenues while controlling costs. Tasks are automated, outsourced, or offshored. Feature enhancements are scored based on anticipated payoff and development cost. Recurring license and support revenue becomes an increasingly large component of overall revenues.

• Scale: She signs deals with large distributors, and works with global consulting firms to have them deploy and integrate her tool. She attends trade shows to collect leads, carefully measuring cost of acquisition against close rate and lead value.

Lean Analytics lifecyclefor an enterprise-focused startup

EmpathyConsulting to test ideas and bootstrap the business

Lock-in, IP control, overfitting

StickinessStandardization and integration; shift from custom to generic

Ability to integrate; support

ViralityWord of mouth, references, case studies

Bad vibes; exclusivity

RevenueGrowing direct sales, professional services, support

Pipeline, revenue recognition, comp

ScaleChannels, analysts, ecosystems, APIs, vertically targeted products

Crossing the chasm; Gorillas

Stage Do this Fear this

The Lean Analytics lifecyclefor an Intrapreneur

EmpathyFind problems; don’t test demand. Skip the business case, do analytics

Entitled, aggrieved customers

StickinessKnow your real minimum based on expectations, regulations

Hidden “must haves”, feature creep

ViralityBuild inherent virality in from the start; attention is the new currency

Luddites who don’t understand sharing

RevenueConsider the ecosystem, channels, and established agreements

Channel conflict, resistance, contracts

ScaleHand the baton to others gracefully

Hating what happens to your baby

Stage Do this Fear this

BeforehandGet buy-in Political fallout

Tool: scoring problem interviews.

How to score problem interviews

• Did the interviewee successfully rank the problems you presented?

• Are they actively trying to solve the problems (or have they done so in the past)?

• Were they engaged and focused throughout the interview?

• Did they agree to a follow-up meeting or interview (to present your solution)?

• Did they offer to refer others to you for interviews?

• Did they offer to pay you immediately for the solution?

Did the interviewee successfully rank the problems you presented?

YES 10 points

Sort of 5 points

NO 0 points

• If the interviewees ranked the problems with strong interest (irrespective of the ranking) that’s a good sign. Score 10 points.

• If they couldn’t decide which problem was really painful, but they were still really interested in the problems, that’s OK but you’d rather see more definitive clarity. Score 5 points.

• If they struggled with this, or they spent more time talking about other problems they have, that’s a bad sign. Score 0 points.

Are they actively trying to solve the problems (or have they done so in the past)?

YES 10 points

Sort of 5 points

NO 0 points

• If they’re trying to solve the problem with Excel and fax machines, you may have just hit on the Holy Grail. Score 10 points.

• If they spend a bit of time fixing the problem, but just consider it the price of doing their job, they’re not trying to fix it. Score 5 points.

• If they don’t really spend time tackling the problem, and are okay with the status quo, it’s not a big problem. Score 0 points.

Were they engaged and focused throughout the interview?

YES 8 points

Sort of 4 points

NO 0 points

• If they were hanging on your every word, finishing your sentences, and ignoring their smartphone, score 8 points.

• If they were interested, but showed distraction or didn’t contribute comments unless you actively solicited them, score 4 points.

• If they tuned out, looked at their phone, cut the meeting short, or generally seemed entirely detached—like they were doing you a favor by meeting with you—score 0 points.

Did they agree to a follow-up meeting or interview (to present your solution)?

Yes, without being asked 4 pointsYes, when you asked them

to2 points

No 0 points• If they are demanding the solution “yesterday”, score 4 points.• If they’re OK with scheduling another meeting, but suddenly their

calendar is booked for the next month or so, score 2 points.• If you both realize there’s no point showing them anything in terms

of a solution, score 0 points.

Did they offer to refer others to you for interviews?

Yes, without being asked 4 pointsYes, when you asked them

to2 points

No 0 points• If they actively suggested people you should talk to without being

asked, score 4 points.• If they suggested others at the end, in response to your

question, score 2 points.• If they couldn’t recommend people you should speak with, score 0

points (and ask yourself some hard questions about whether you can reach the market at scale.)

Did they offer to pay you immediately for the solution?

Yes, without being asked 3 pointsYes, when you asked them

to1 point

No 0 points• If they offered to pay you for the product without being asked, and

named a price, score 3 points.• If they offered to pay you for the product, score 1 point.• If they didn’t offer to buy and use it, score 0 points.

How to score problem interviews

• A score of 31 or higher is a good score. Anything under is not.

• Try scoring all the interviews, and see how many have a good score.

• This is a decent indication of whether you’re onto something or not with the problems you want to solve.

• Then ask yourself what makes the good score interviews different from the bad score ones.

• Maybe you’ve identified a market segment; maybe you have better results when you dress well; maybe you shouldn’t do interviews in a coffee shop. Everything is an experiment you can learn from.

How to score problem interviews

• You can also sum up the rankings for the problems that you presented. If you presented three problems, which one had the most first place rankings? That’s where you’ll want to dig in further and start proposing solutions (during Solution Interviews.)

• The best-case scenario is very high interview scores within a subsection of interviewees where those interviewees all had the same (or very similar) rankings of the problems. That should give you more confidence that you’ve found the right problem and the right market.

Tool: scoring problem interviews.

Draw a new line

Pivot orgive up

Try again

Success!

Did we move the needle?

Measure the

results Make changes in production

Design a test

Hypothesis

Metrics in practice:The Lean Analytics Cycle

With data:find a

commonality

Without data: make

a good guess

Find a potential

improvement

Draw a linein the sand

Pick OMTM

Tool: scoring problem interviews.

Three threes

Threeassumptions

What big bets are you making?•“People will answer questions”•“Organizers are frustrated with how to run conferences”•“We'll make money from parents”•“Amazon is reliable enough for our users.”

Three actionsto take

What are you doing to make these assumptions happen (or identify they’re wrong and change course?)•Product enhancements•Marketing strategies

Three experimentsto run

•Feature tests•Continuous deployment•A/B testing•Customer survey

Three threes

Threeassumptions

Three actionsto take

Three experimentsto run

Monthly

Weekly

Daily

Board, investors, founders

Executive team

Employees

Strategy

Tactics

Execution

Three threes

Threeassumptions

Three actionsto take

Three experimentsto run

Get more people

Increase answer %

Test betterquestions

Change the UI

Test timings

Questions from peers

Many people will answer questions

Tool: the problem/solution canvas