Building a data-driven culture
-
Upload
olof-hoverfaelt -
Category
Data & Analytics
-
view
1.409 -
download
3
Transcript of Building a data-driven culture
May 3, 2023Presentation name 1
Building a Data-Driven CultureOlof HoverfältDirector, Sanoma Games September 2, 2015
@Hoverfalt
Sanoma Games is a diverse yet compact and lean unit
B2C & B2B, web & mobile Autonomous unit Key functions “in-house”
Business development
Marketing
Design
Development
Sales
DESIGN
BUSINESS ANALYTICS
UX Business
Being data-driven requires much more than the right tools
http://toolmonger.com/category/85/computer/page/2/
Handheld USB Snake
Scope
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
http://theleanstartup.com/
http://www.startupolic.com/launching-your-service-think-about-ux/; https://en.wikipedia.org/wiki/Lean_startup
An MVP is “a version
of a new product
which allows a team
to collect the
maximum amount of
validated learning
about customers with
the least effort"
The MVP approach is useful at all levels of development, from single product feature to new business concepts
Adopt MVP thinking in all development
http://www.yesware.com/blog/how-to-close-more-deals-with-lean-email/
A continuous Build-Measure-Learn cycle enables fast learning but is really sensitive to friction in workflow and cooperation
Ensure all key functions are fully dedicated and the workflow is frictionless
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
A data-driven culture cannot be created by coercion, but must be built through intrinsic motivation from its visible benefits
Transparency Autonomy Ownership
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
xxx
Improvement actions
xxx
Revenue
ROS&packag. campaign vol.
Videocampaign vol.
Pricing
Discounts
Campaign volume
Optimization
Direct
Referral
SEO
Ret. visitors
Potential issues
Site problems
Ad blockers
New visitors
Reach
Monetization
Visits
Premiumfill rate
Premium eRPM
Remnant eCPM
Unique visitors
Media sales breakdown by contributing component
Impression success rate
nn%
nn%
nn% mm%
nn%
nn%
nn% lost
Ensure overall understanding of how the business works and of the contribution of and the dependencies between its components. Make deep data available
In order for a team to develop and fine tune an engine, everyone must know how it works
Adoption of a data-driven approach requires for people to see its benefits; how they could kick ass even more at what they do
Image: http://blog.crazyegg.com/lessons/show-benefits/
Make the personal benefits of being data-driven visible.Show how its autonomous pursuit enhances mastery.
Super You!Data-drivenness
You
Ownershipis not about doing what I tell you to do
is not about not doing what I tell you not to dois about doing what I don’t tell you to do
Ensure ownership of benefits, not deliverablesFocus on why, not what
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
Data-drivenness, at both individual and team level, is a capability that can and should be developed
http://skepdic.com/tencommands.html
+Encourage autonomous reflection but ensure conceptual guidance
Reflection Key concepts
Being data-driven is not only about sensibly answering questions, but about finding them in the first place
It cannot itself be standardized into rules or processes, but it
can be guided by high-level key concepts
A core principle of learning is to be hypothesis-driven. Random experimentation will yield various outcomes but little insight
Make continuous hypothesis-driven split testing the standard approach in all development activities across functions
Hypothesis:[A statement that can be shown true or false]
Design a test and a metric that confirms the hypothesis at a certain pre set limit
Test in a real setting with real usersPay attention also to
what is NOT happening
Confirm or reject the hypothesis based on
the chosen metric and pre set limit
Add the learnings to your body of
knowledge, also the invalidated hypotheses
Build on the knowledge to form the
next hypothesis
Data without insight can be destructive as it may lead development the wrong way look like improvement
DAU(daily active users)
Downloads(i.e. not even activations)
Activity delta(change in user engagement)
Churn(users who quit)
>Vanity metricIndicator of
actual use
>Leading indicator of churn
ActionableLagging indicator
Not actionable
Put a lot of effort into designing KPIs (Key Performance Indicators) that are both sensible and actionable; such that steer development initiatives toward your ultimate business goal
Main KPIs should be static but reflect the current growth phase, while tactical tools can be whatever helps improve the KPIs
High-level (growth-stage-specific) KPIs to continuously optimize
Tactical tools do explore user behavior and to measure impact from change on KPI(s)
http://lanesplitter.jalopnik.com/x-ray-artist-reveals-the-inner-beauty-of-classic-motorc-1698564514
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
A simple tool that is actually used provides infinitely more value than a sophisticated one that is not
https://xplr.com/xplr-umbrella-dataviz-on-top-of-unsupervised-machine-learning/
>Make sure everyone has easy access to all tools they might need and know how to use them
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
Goals set at function level can often be unaligned or conflicting, breaking joint effort and leading to sub-optimal performance
UX
Set high level common goals to ensure everyone works and cooperates to optimize the whole as opposed to sub-optimizing their own components
Business Manager goal:Visit eRPM
(effective revenue per session)
Product Manager goal:Avg. DAU(daily active users)
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
Example:Continuous improvement and A/B testing
A/B testing is about continuous and systematic gradual improvements that over time build increased performance
2.5x+ Improvement
Seemingly small but sustained improvements can have a very large impact on overall performance
49% Improvementmin to max
25% Improvementmean to max
Example:Retention (how well you keep your users)
6.1%Retention D1
Count of users by level
Count of users by level
Count of users by level
6.1%Retention D1
47%Retention D1
25%Retention D7
6.2%Retention D90
Compare with6.1% D1 Retention
in the previous example
Example:LTV ((Customer) Life Time Value)
User profitability is basically determined by LTV* and CAC*. Being able to even roughly predict LTV can be very insightful.KPIsDAU
Monthly rev.
Retention D1 43 %
Retention D7 24 %
Retention D14 19 %
Retention D30 12 %
Conversion
ARPDAU
ARPPDAU
LTV
CAC
ARPDAU nn € nn €
Extr. tot. days (≤120th) nn nn
LTV nn € nn €
* LTV = (Customer) Life Time Value ; CAC = Customer Acquisition Cost
Example:Liigapörssi (KPIs and tools)
Direct
Org. search
SEM
Display
Other
Our main time unit of comparable comparison is a game “phase”. The key desired user behavior varies within the phase
Free
Trades
1
Pool Free
Trades
2
Pool Free
Trades
6
Pool
Direct
Org. search
SEM
Display
Other
Pay(Revenue)
Join pool(Activation)
Sign up (Acquisition)
Play the whole phase(Retention)
Continue to next phase(Retention)
Sign up or reactivate
(Acquisition)
Desired user behavior
Create pool& invite
(Referral)
Invite friends(Referral)
Make trades(Activation)
Buy trades(Revenue)
Create team (Acquisition)
Phase 1 Phase 2 Phase 3-6Season
AcquisitionActivation
RetentionRevenue
Referral
Acquisition
ReferralRevenue
Retention
Activation Focus area
We use various tools to identify development opportunities, run tests and measure impact on the main high-level KPIs set
Area KPI
Acquisition
Signed up (%)
Created team (%)
CAC (€)
ActivationActive user (%)
In pool (%)
RetentionIn-phase retention (%)
Phase change retention (%)
Revenue
LTV
ARPPU
Conversion
Referral Invited friend (%)
High-level (growth-stage-specific) KPIs to continuously optimize
Tactical tools do explore user behavior and to measure impact from change on KPI(s)
… …
Funnels
Emails
Anything!
Custom Dimensions using hashed user IDs enables user
level analysis, which is VERY powerful
Some tools we are currently using
Tool Domain Sought benefits
Email Improve open rate and CTR
Email (transactional) Identify issues in use and funnels
Funnels Improve user onboarding and payments
Ecommerce Research revenue by product, user type, etc. (Enh. Ecom. Trac.)
Engagement Research anything at the user level!
Engagement Retention, User level progression
User Acquisition Volume and CPI (Cost Per Install)
Landing pages Optimize CTA (Call To Action) conversion
UI heat maps Research behavior and improve UI
KPI dashboard Communication and awareness
Some suggested further reading
* http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version
Slideshare*
...Design Product Marketing Business
Being data-driven requires much more than the right tools
Organization supporting lean development
Data-driven culture
Key concepts
Accessible tools
Shared goals
Management fostering self-direction
Key concepts Key concepts
@Hoverfalt
May 3, 2023Presentation name 52