Metrics Driven Prioritization
35
Metrics Driven Prioritization Engineering For Product Success — #QConSF 2014 Sam McAfee, Principal - @sammcafee
-
Upload
sam-mcafee -
Category
Internet
-
view
779 -
download
0
description
Prioritization is one of the toughest aspects of software product-development. In these days of being agile and running lean, we don't want to prioritize our backlog by fiat, politics, or gut-feel. Teams need to be aligned to accomplish core business objectives and need tight feedback loops with the customer and stakeholders in order to achieve those objectices. To accomplish this level of cross-functional product development maturity, we must combine data from a number of sources to quantitatively optimize development. In this talk, I will outline an approach we are using at Neo to help our clients integrate business metics and probabilistic modeling into their prioritization process. Leveraging data from business KPIs, the product development team's own performance metics, and feedback from in-production experiments and A/B tests, I'll illustrate a straightforward yet rigorously quantitative method of prioritizing feature development to focus on achieving business and product outcomes, not simply producing technology outputs.
Transcript of Metrics Driven Prioritization
Metrics Driven PrioritizationEngineering For Product Success — #QConSF 2014Sam McAfee, Principal - @sammcafee
Innovate or die.
Who is this guy?• Dot-com boom survivor. • Ran a dev shop for 10 years after that. • Agile has always been the “norm”. • Read ‘4 Steps’ before #LeanStartup. • Participated in 3 different startups. All dead. • Ran the Change.org engineering team. • Learned LeanUX from @clevergirl herself. • Now a Principal at Neo Innovation (neo.com)
Tweet every word! @sammcafee #qconsf
Slides available at bit.ly/metricsdriven
“If you build it…”Is anyone even going care about this product?
Nope.
Competing priorities
Everyone loves quarterly planning meetings
A Metrics Driven Approach
Pre-requisitesWhat is product success, anyway?
A successful product solves a painful problem for a specific customer in a
sizable reachable market.
UsableDoes it delight the user?
FeasibleCan we build it?
1
2
ValuableDoes anyone want it?
3
Test Your Assumptions• Use lightweight personas to test the customer segment.
• Use problem interviews to test the problem.
• Use sketches and paper prototypes to test the proposed solution.
• Use landing pages to test the market.
• And do all this before you build anything!
Cross-functionalThe team includes design, engineering, and product.
CollocatedEvery works in the same location.
DedicatedThe team only works on one product at a time.
Assemble the Team
A Metrics Driven Approach
Measure the BusinessUse an economic framework to measure the
impacts of your decisions on core KPIs.
Measure the TeamsMeasure lead time from concept to cash, and use
outliers to set realistic expectations for delivery.
Run ExperimentsYour backlog is a list of experiments you need to run
in order to validate your business assumptions.
Measure the Business
Business Model Canvas
Model Your Funnel
Build an Economic Framework
An Example Product Funnel
• $20 / month subscription.
• 10% conversion from landing page.
• 5% referral rate
• 10% churn rate (or 90% retention rate)
Information Value = Economic impact *
uncertainty.
An Example Product Funnel
• Each visit is worth $2.00
• Each share is worth $0.10
• Customer lifetime value (LTV) is depleted $1 per month, on average, per customer.
Quantify Cost of Delay
Example: Cost of Delay
Measure the Teams
Most delay in product development is wait time.
Measuring Cycle Time
Weighted Shortest Job First
Running Experiments
The Value of Experiments• Each visit is worth $2 (10% of visitors sign up for $20/month).
• Increasing conversion by 50% is worth $1 per user to us.
• We have only a 10% chance of being able to increase it by 50%.
• Historically, only 10% of our experiments on that page succeed.
• $2 * 1.5 * 0.1 * 0.1 = $0.01 = experiment is only worth a penny!!
Prioritizing an experiment requires data from the
funnel, the team, and any past experiments.
Thank you!
@sammcafee
