Lean Analytics: How to find & test innovative Growth Hacks using Analytics
-
Upload
phil-pearce -
Category
Data & Analytics
-
view
417 -
download
1
Transcript of Lean Analytics: How to find & test innovative Growth Hacks using Analytics
Lean Analytics
Workout
DAA Hub
Phil Pearce
April 2014
Fitness Consultant
linkedin.com/in/philpearce
Harder, Better, Faster, Stronger
Leaner!
Who have we got in the room?
1. Entrepreneur – Started company 2. Corporation – Work in big company 3. Agency – Help startups
Have you read any of these?
Quick Quiz
https://www.youtube.com/watch?v=usdeiJP7xh0 and http://prezi.com/lw-arulaenh4/copy-of-lean-startup-buzz-words/
START
START
GH
PvT
PMF
LS
Cdev
FINISH
FINISH
FINISH
IyBITwC
TwCIyBi
P2 E4
MVP
BML
VM
Web Analytics Exchange
mentor 750 GA
questions answered
Tracking protection group
About Me
Phil Pearce Analytics Consultant
linkedin.com/in/philpearce
I`m not an entrepreneur
Apart from
this one
AdWords
But... I have done alot of agency consulting & I worked for some innovative startups
Sold for €16m
Pivoted
Changed business
model IPO in ~1yrs
Funded by
Gwyneth Paltrow
Sold for €37m
Crazy growth & IPO plans
IPO soon
Metrics Plan
Massive Revenue
understanding own sites digital value
to understand investments
Grew Taxi booking
Revenue by €10m in 2yrs
Intrapreneur & Technical marketer
1. Build PPC reporting platform MS access
2. Enabled KW level ROI bidding in 2007.
3. Managed £600K pm Adwords account & out-
performed market leader.
4. Built end-to-end affiliate tracking system.
5. Reverse engineered Adwords Algo.
6. Built mathematical ClickFraud detection tool for
mobile
7. Built free version of SpeedPPC
8. Building “4clicks” SaaS for Magneto (KPIs,
dataLayer, Dashboards, Remarketing -> all auto-
enabled)
GH GH
GH GH
GH GH
GH GH
GH GH
...and closet growth hacker FINISH
GH
... I have author-ed a book on Amazon
Agenda • Start: 9:30am-12:30am
• Introduce Lean Analytics terminology
– (e.g. MVP, Iterations, Agility)
• Explain why obsessing over the performance of one key metric is vital
• Describe the difference between website and product innovating and
testing?
• Look at some examples of successful (and unsuccessful) analytics hacks
• Develop a super analytics hack for your business
• Define a process for testing and refining your hacks
If you build it … they will come.
They will come ... if you build it
Because…
Fast Feedback = Build what customers want
Favourite Food
Most startups don’t know what their
customers will consume
(or what they are good at making)
Hotmail
was a
database
company
Flickr
was going to
be an Video
Game platform
was a
podcasting
company
Autodesk
made
desktop
automation
Paypal
first built for
Palmpilots
Freshbooks
was invoicing
for a web
design firm
Wikipedia
was to be
written by
experts only
Mitel
was a
lawnmower
company
Fast iterations/sprints using…
Build > Measure > Learn BML
Build
• (products)
Measure
• (data)
Learn
• (ideas)
Fast iterations/sprints using…
Build > Measure > Learn (repeat) BML
Build
• (products)
Measure
• (data)
Learn
• (ideas)
Even the book uses lean
principles… 1. 5th edition in 8months (new
edition every built
1.5months!)
2. “We liked to hear from you”
feedback section in front &
online blog comments
encouraged.
3. Learnings have spawned
start-up conferences
Build > Measure > Learn
Measure
Problem:
We lie to ourselves
“We” are amazing!
Reality check…
…Analytics to the rescue
Analytics is the measurement of
movement towards your business
goals.
In a startup, the purpose of analytics is
to iterate to product/market fit
before the money runs out.
I have two coins.
Atleast one of them is heads.
What is the % probability
that the other is tails?
Guess…
Tails
Tails
Heads
Tails
Tails
Heads
Heads
Heads
Heads
Tails
Tails
Heads
Heads
Heads
2 of 3 (66%)
are tails.
Some fundamentals.
A good metric is:
Understandable
If you’re busy
explaining the
data, you won’t
be busy acting
on it.
Comparative
Comparison is
context.
A ratio or rate
The only way to
measure
change and roll
up the tension
between two
metrics
(Miles Per Hour)
Behavior
changing
If you’re busy
explaining the
data, you won’t
be busy acting
on it.
simplest
rule
Not a good metric.
If metric won’t
change how you
behave, it’s…
Metrics help you know yourself.
You are
just like
Customers that
buy >1x in 90d
Your customers
will buy from you
Then you are
in this mode
Acquisition 70% of retailers
Once 1-15% Low acquisition
cost, high
checkout
Hybrid 20% of retailers
2-2.5 per year
15-30% Increasing return
rates, market share
Focus on
Loyalty 10% of retailers
>2.5 per year
>30% Loyalty, selection,
inventory size
(Thanks to Kevin Hillstrom for this.)
Qualitative
Unstructured, anecdotal,
revealing, hard to
aggregate, often too
positive & reassuring.
Warm and fuzzy.
Quantitative
Numbers and stats.
Hard facts, less insight,
easier to analyze; often
sour and disappointing.
Cold and hard.
Exploratory
Speculative. Tries to find
unexpected or
interesting insights.
Source of unfair
advantages.
Cool.
Reporting
Predictable. Keeps you
abreast of the normal,
day-to-day operations.
Can be managed by
exception.
Necessary.
Rumsfeld on Analytics
(Or rather, Avinash Kaushik channeling Rumsfeld)
Things we
know
don’t
know
we know Are facts which may be wrong and
should be checked against data.
we don’t
know
Are questions we can answer by
reporting, which we should baseline
& automate.
we know Are intuition which we should
quantify and teach to improve
effectiveness, efficiency.
we don’t
know
Are exploration which is where
unfair advantage and interesting
epiphanies live.
May
A/B test:
Changing one thing
(i.e. color) and
measuring the
result (i.e. revenue.)
Apr Mar
0
Jan Feb
Segment:
Cross-sectional
comparison of all
people divided by
some attribute (age,
gender, etc.)
Slicing and dicing data 5,000
Acti
ve
users
Cohort:
Comparison of
similar groups
along a timeline. (this is the April cohort)
Multivariate
analysis
Changing several
things at once to
see which correlates
with a result.
☀
☁
☀
☁
☀
☁
Which of these two companies
is doing better?
Is this company
growing or stagnating?
Which of these two companies has the best
Revenue/Customer?
January February March April May
Rev/customer $5.00 $ 4.50 $4.33 $4.25 $4.50
Cohort January February March April May Averages Cohort
group5 € 5.00 € 6.00 € 7.00 € 8.00 € 9.00 € 7.00
group4 € 3.00 € 4.00 € 6.00 € 7.00 € 5.00
group3 € 2.00 € 2.00 € 5.00 € 3.00
group2 € 1.00 € 1.00 € 1.00
group1 € 0.50 € 0.50
Lagging
Historical. Shows you
how you’re doing;
reports the news. Example: sales.
Explaining the
past.
Leading
Forward-looking.
Number today that
predicts tomorrow;
reports the news. Example: pipeline.
Predicting the
future.
• A Facebook user reaching 7 friends within 10 days of signing up
(Chamath Palihapitiya)
• If someone comes back to Zynga a day after signing up for a game,
they’ll probably become an engaged, paying user
(Nabeel Hyatt)
• A Dropbox user who puts at least one file in one folder on one device
(ChenLi Wang)
• Twitter user following a certain number of people, and a certain
percentage of those people following the user back
(Josh Elman)
• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
Some examples
(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)
Which means it’s time to talk
about correlation.
Number of Analysts
Chess StarTrek correlated Liked Maths
causal
Number of Analysts
Correlated vs Causal P2 E4
Correlated vs Causal
But it is not the cause!
Strong Correlation
Correlated
Two variables that are
related (but may be
dependent on
something else.)
Ice cream &
drowning.
Causal
An independent variable
that directly impacts a
dependent one.
Summertime &
drowning.
A leading, causal metric
is a superpower. h" p ://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/
Growth hacking, demystified.
Find
correlation
Test
causality
Optimize the
causal factor
Pick a metric
to change
Why is Nigerian spam so badly
written?
Aunshul Rege of Rutgers University, USA in 2009
Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages
emailed; they expect to land 2 or 3 “Mugu” (fools) each week.
One scammer boasted “When you get a reply it’s 70% sure you’ll get the money” “By
sending an email that repels all but the most gullible,” says [Microsoft Researcher
Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts
the true to false positive ratio in his favor.”
1000 emails
1-2 responses
1 fool and their money, parted.
Bad language (0.1% conversion)
Gullible (70% conversion)
1000 emails
100 responses
1 fool and their money, parted.
Good language (10% conversion)
Not-gullible (.07% conversion)
This would be horribly inefficient since
humans are involved.
Turns out the word “Nigeria” is the best
way to identify promising prospects.
Nigerian spammers
really understand their target market.
They see past vanity metrics.
The Lean Analytics framework.
Eric’s three engines of growth
Virality
Make people
invite friends.
How many they
tell, how fast they
tell them.
Price
Spend money to
get customers.
Customers are
worth more than
they cost.
Stickiness
Keep people
coming back. Approach
Get customers
faster than you
lose them.
Math that
matters
@agatestudio
Lean Analytics Stages
Empathy • I’ve found a real, poorly-met need & reachable market faces
Stickiness • I’ve figured out how to solve the problem, in a way they will adore and pay for!
Virality • I’ve built the right product/features/functionality that keeps users around.
Revenue • The users and features fuel growth organically and artificially.
Scale • I’ve found a sustainable, scalable business with right margin in a healthy ecosystem.
1. Ecommerce
2. Two sided marketplace
3. SaaS
4. Mobile app
5. Media/Publishing
6. User generate content
Six business model types
Model + Stage = One Metric That Matters.
One Metric
That Matters.
The business you’re in
E-Com 2-Sided SaaS Mobile Media UCG
Empathy
Stickiness
Virality
Revenue
Scale
Th
e s
tag
e y
ou
’re a
t
Really? Just one?
Yes, one!
Because… In a startup`s “focus" is
hard to achieve.
Having only one metric
resolves this problem.
www.theeastsiderla.com
Prevents distraction
Metrics are like squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/
Revenue stage:
CompareAndSave.com
(2-sided marketplace)
• Focus on one metric of CTR
• Reduced CPC
• Increased RPC (Effected of reverse economies of scale &
tiered cpa volumes)
• Marketplace: Consumers + Banks
Technically a “comparison engine”
Empathy
Stickiness
Virality
Revenue
Scale
E-
commerce SaaS Media
Mobile
app
User-gen
content
2-sided
market
Loyalty,
conversion
CAC, shares,
reactivation
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
(Money from transactions)
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
(Money from active users)
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses
(Money from ad clicks)
Workshop Task: 1. Select business type
(E-Com, 2-Sided, SaaS, Mobile, Media, UCG) 2. Determine Stage
(Empathy, Stickiness, Virality, Revenue, Scale) 3. Pick one metric 4. Set line in the sand (benchmark)
Useful sheet bit.ly/BigLeanTable
Other measurement models bit.ly/kpishake
What other metrics
do you want to know about?
Drawing some lines in the sand.
A company loses a quarter of its
customers every year.
Is this good or bad?
Baseline:
10% visitor engagement/day
30% of users/month use web or mobile app
10% of users/day use web or mobile app
1% of users/day use it concurrently
Fred Wilson’s social ratios
Baseline:
2-5% monthly churn
• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s
on the job.
• Get below a 5% monthly churn rate before you know you’ve got a
business that’s ready to grow (Mark MacLeod) and around 2%
before you really step on the gas (David Skok)
• Last-ditch appeals and reactivation can have a big impact.
Facebook’s “don’t leave” reduces attrition by 7%.
Who is worth more?
Lifetime:
$200
Lifetime:
$200
Today
A
Roberto Medri, Etsy
B
Visits
The Lean Analytics cycle
Did we move the
needle?
Make changes
in production
Hypothesis
Design a test
Make changes
in production
Measure the
results
Success!
Pivot or give
up
Pick a KPI
Find a potential
improvement
Draw a line
With data: find
a commonality
Without data:
make a good
guess
Draw a new line
Repeat test Did we
move the
needle?
Do AirBnB hosts
get more business
if their property is
professionally
photographed?
Gut instinct (hypothesis)
Professional photography helps AirBnB’s business
Candidate solution (MVP)
20 field photographers posing as employees
Measure the results
Compare photographed listings to a control group
Make a decision
Launch photography as a new feature for all hosts
5,000 shoots per month
by February 2012
Draw a new line
Pivot or
give up Find a potential
improvement Try again
Success!
Did we move the
needle?
Measure
the results
Make changes
in production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Draw a line Pick a KPI
“G e e, t ho s e
ho u se s t ha t d o
w e ll l o o k r e a l l y
n i c e.”
Ma y be i t ’s t he
c a m er a .
“C o m pu ter : Wha t
d o a l l t he
h ig hl y r e n te d
ho u se s ha v e i n
c o m m o n ?”
C a m er a m o d e l .
With data:
find a commonality
Without data: make a
good guess
Some non-tech
examples.
I lied. Everyone is a tech company.
http://www.flickr.com/photos/puuikibeach/4789015423
Cost of attention: way up.
http://www.flickr.com/photos/elcapitanbsc/3936927326
Cost of experiments:down.
Let’s pick on restaurants
for a while.
A line in the sand
Labor costs
Gross revenue
30%
20%
Just right
Understaffed?
= 24%
Too costly?
A leading indicator
http://www.flickr.com/photos/avlxyz/4889656453 http://www.flickr.com/photos/mysticcountry/3567440970
50 reservations
at 5PM
250 covers
that night
(Varies by
restaurant.
McDonalds
≠ Fat Duck.)
http://www.flickr.com/photos/southbeachcars/6892880699
Restaurant MVP
Is tip amount a leading indicator of long-
term revenue?
Why does every table get the same
menu?
Is purple ink better? http://tippingresearch.com/uploads/managing_tips.pdf
Growth hacking
(is a word you should hate but will hear a lot about.)
Growth hacking, demystified.
Find
correlation
Test
causality
Optimize the
causal factor
Pick a metric
to change
Guerrilla
marketing
Data-
driven
learning
Subversiveness
GROWTH
HACKING
• A Facebook user reaching 7 friends within 10 days of signing up
(Chamath Palihapitiya)
• If someone comes back to Zynga a day after signing up for a game, they’ll
probably become an engaged, paying user (Nabeel Hyatt)
• A Dropbox user who puts at least one file in one folder on one device (ChenLi
Wang)
• Twitter user following a certain number of people, and a certain percentage of
those people following the user back (Josh Elman)
• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
(These are also great segments to analyze.)
Leading indicators: Growth Hacks
Read more examples:
http://www.slideshare.net/mattangriffel/growth-hacking
• Growth hacking is simply what marketing should have been
doing, but it fell in love with Don Draper and opinions along the
way
• Optimize a factor you think is correlated with growth
The growth hack
Growth Hacking examples
• Hotmail – P.S. I love you
• Drobbox – Refer a friend
• Facebook – Exclusive network appeal
• Twitter – follow celebrities
Read more examples:
http://www.slideshare.net/mattangriffel/growth-hacking
AirBnB and Craigslist
What is PPC Growth?
Adwords Marketing
Conversion Data
Sales Growth
Classic model 1. Create new marketing campaign. 2. If CPA data is within target threshold = then growth achieved 3. Then Increase marketing. Repeat above steps, until threshold reached.
Repeat until CPA threshold reached
And… needs to be Scalable, Repeatable and Sustainable.
Growth Hacking for PPC SaaS
Adwords
Marketing Product Data
Growth Hacking
Note: “Product” could be value-proposition or incentives
SaaS hack
Have you Heard of “Battleships”?”
Growth Hacking a PPC tool…
Growth Hacking a PPC tool…
Download CTR battleships http://bit.ly/battleshipsctr
And… needs to be Scalable, Repeatable and Sustainable.
Growth Hacking a PPC tool…
Adwords
Marketing
(capture/test initial lead)
Product
(Variation on QS grader)
Adwords API Data
Growth Hacking
Same product but different UI/Landing page - value proposition tweaked and made fun / viral… e.g every one likes to show off :)
Growth Hacking a PPC tool…
Winner must tweet winning strategy #ctrbattleships Or take a photo them holding the Prize! #ctrbattleships
Download CTR battleships http://bit.ly/battleshipsctr
Some tools and traps
Traction graphs
Your business model
The stage you’re at
Your one metric
... change often if
you’re doing it right.
So how do you track
that over time?
Traction graphs
Jan
Signup
s per
day
Feb Mar
Conversio
n rate
Apr
Chur
n rate
May Jun
Viral
coefficient
This axis changes for
each metric
Traction graphs
Jan
Signups
per day
Feb Mar
Conversion
rate
Apr
Churn
rate
May Jun
Viral
coefficient
0%
Use vanity to get to
meaningful metrics
• Your goal is to produce outcomes
• If the outcomes require action, and vanity
motivates actors, use it!
• But show how the vanity metric is a leading
indicator of the real one! x
Web traffic
Revenue
Activation
Cart
Size
Conversion
rate
VM
“The most important figures that one
needs for management are unknown
or unknowable, but successful
management must nevertheless take
account of them”
Lloyd S. Nelson
Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844
ARCHIMEDES HAD TAKEN BATHS BEFORE
Once, a leader leader/king
convinced others in the
absence of data.
Now, a leader/king knows what
questions to ask.
Alistair Croll & Ben Yoskovitz
Special thanks to…
Please buy
their book if you
want more info!
Copy of these slides
bit.ly/leananalytics2
Appendix:
Useful videos
15min Lean Start-up Video
https://www.youtube.com/watch?v=zOX1vC7_n6s
60min Lean Analytics
https://www.youtube.com/watch?v=-CB4w_OtrKw
50min podcast
www.twistimage.com/podcast/mp3/SPOS_351_-_Alistair_Croll.mp3