SLIDE NO. 1

TAKE CONTROL OVER YOUR KPIsHilda KosorusSenior Data Scientist

SLIDE NO. 2

About me

2017

Runtastic: Senior data scientist

2015

Runtastic:Data scientist

2010

JKU: Research assistant

Academia Industry

2013

JKU:Senior researcher

SLIDE NO. 3

ABOUT RUNTASTIC

SLIDE NO. 4

4We are

founders

8We are

years old20

We were profitable after just

months

235We are

employees

40+We come from

countries

3We have

offices in Linz, Viennaand Salzburg

15Our products are

available in

languages

1We are

team with a shared vision

RUNTASTIC BY THE NUMBERS

SLIDE NO. 5

SLIDE NO. 6

SLIDE NO. 7

EACH DAY WE HAVE

150,000NEW DOWNLOADS

WE HAVE MORE THAN

130MREGISTERED USERS

WE BOAST OVER

246MTOTAL APP DOWNLOADS

WE HAVE MORE THAN

4.8MFANS AND FOLLOWERS

TAKE CONTROL OVER YOUR KPIsAdvice from a Data Scientist

PRESENTATION TITLESLIDE NO. 9

WHAT and WHY?

PRESENTATION TITLESLIDE NO. 10

What are KPIs?

KPI = Key Performance Indicator

“a quantifiable measure used to evaluate the success of a business”

PRESENTATION TITLESLIDE NO. 11

Why are KPIs useful?

measurable / quantifiable

data-driven

gauge the effectiveness of processes

business health indicators

setting goals

common language → common goal

PRESENTATION TITLESLIDE NO. 12

Examples

● Number of newly acquired customers● Revenue● ROI● Retention/churn rate● Conversion rate● NPS● Sales growth

PRESENTATION TITLESLIDE NO. 13

HOW: 7 STEPS

PRESENTATION TITLESLIDE NO. 14

1. CHOOSE WISELY

PRESENTATION TITLESLIDE NO. 15

Choose SMART

● Specific - has a clear purpose for the business

● Measurable

● Achievable

● Relevant to the success of the organization

● Time phased - the value represents a predefined and relevant period

PRESENTATION TITLESLIDE NO. 16

You advise, management decides

● Be proactive● Do your homework● Communicate & consult● Make it work either way

PRESENTATION TITLESLIDE NO. 17

2. UNDERSTAND THOROUGHLY

PRESENTATION TITLESLIDE NO. 18

Why should you care?

● You can’t optimize what you don’t understand

● Won’t know why they perform good/bad

● Can’t act upon them effectively

● Waste of resources → invested in the wrong place for the wrong reasons

● Most probably won’t reach your goals

PRESENTATION TITLESLIDE NO. 19

What should you do?

1. Where is the data coming from? ● Is the source reliable?

2. How is the metric calculated?3. Analyze

● components, trend, seasonality, edge cases● relationships to other company metrics & drivers● at different granularity levels

4. Monitor● seasonality effects, patterns, outliers

PRESENTATION TITLESLIDE NO. 20

Focus on the basic components

Almost all KPIs are aggregates

● Count of non-repeatable events○ without churn (e.g. # registered users)○ with churn (# active subscribers)

● Windowed aggregates (e.g. # monthly active users)● Ratio (e.g. % conversion rate)

PRESENTATION TITLESLIDE NO. 21

What a DS should see:

Example: # active subscribers

new

churn

What a managers sees:

PRESENTATION TITLESLIDE NO. 22

Monitor

PRESENTATION TITLESLIDE NO. 23

3. ESTIMATE FUTURE BEHAVIOR

PRESENTATION TITLESLIDE NO. 24

After all, they are just indicators...

● KPIs are similar to measuring instruments (e.g. thermometer)● useful to know past/current state (temperature increased or decreased)

It is more critical to know what opportunities and dangers lie ahead.

Drivers are your most valuable allies.

PRESENTATION TITLESLIDE NO. 25

How to predict?

● Forecasting methods○ growth-based○ curve fitting (linear, polynomial, regression methods, etc.)○ probabilistic models (survival analysis, HMM)○ time series forecasting (ARIMA)○ machine learning (e.g. classification)

● A plethora of libraries (R, python)

PRESENTATION TITLESLIDE NO. 26

Time series forecasting

● developed by Facebook DS team● open source● python & R● facebook.github.io/prophet

PRESENTATION TITLESLIDE NO. 27

How does it work?

● decomposable time series model○ trend, seasonality, holiday

○ models non-period changes○ represents periodic change○ represents effects of irregular

events● interpretable parameters

PRESENTATION TITLESLIDE NO. 28

Capabilities

● Growth (linear, logistic)● Changepoints● Seasonality components

○ yearly, monthly, weekly, daily○ additive, multiplicative

● Holidays○ past and future dates○ lower and upper window of effect

● Deals with missing values● Deals with irregularly spaced data● Diagnostics● Visualizations

PRESENTATION TITLESLIDE NO. 29

Why prophet?

● automatic forecasting techniques○ hard to tune○ too inflexible to incorporate useful assumptions or heuristics

● prophet○ flexible, easy to use○ fast○ interpretable○ it works!

PRESENTATION TITLESLIDE NO. 30

Example in R

PRESENTATION TITLESLIDE NO. 31

4. AUTOMATE

PRESENTATION TITLESLIDE NO. 32

ETL

Reports

R + T-SQL

Data sources

Interactive dashboard

Deployment infrastructure

PRESENTATION TITLESLIDE NO. 33

1

2

R and Transact-SQL

PRESENTATION TITLESLIDE NO. 34

Learnings

● Communication & collaboration○ Define clear responsibilities (DE, DS)

● Efficient deployment○ Define process○ Empower DS to deploy independently○ Code reusability○ Standardize forecasting data products

● Platform stability○ R launchpad management on SQL server

PRESENTATION TITLESLIDE NO. 35

5. INSPIRE TRUST

PRESENTATION TITLESLIDE NO. 36

Evaluate for model selection

● Historical data for training● Parameter selection

○ changepoint prior○ seasonality prior○ growth method○ etc.

● Cross-validation● Evaluation metrics (e.g. MAPE)● Visualization approaches

PRESENTATION TITLESLIDE NO. 37

Evaluate past predictions

● How accurate were our past predictions?● Monitoring● Sanity checks

○ trend changes○ seasonal changes○ other interferences

● Re-evaluate parameter settings

PRESENTATION TITLESLIDE NO. 38

6. ACT

PRESENTATION TITLESLIDE NO. 39

Making use of predictions

● Help understand trends and seasonality● Defining goals

○ realistic○ ambitious

● Detect trend changes early on○ automate

● Derive meaningful actions○ resource allocation○ identify measures with high impact

PRESENTATION TITLESLIDE NO. 40

7. OPTIMIZE

PRESENTATION TITLESLIDE NO. 41

Good enough, but not great

● Granularity○ Break down metric for consistent behaviour ○ Segments (e.g. product, platform, acquisition source)○ Components (e.g. new and churned customers)

● Events -- non-organic interferences (e.g. ads, campaigns, offers)○ Identify effect○ Collect data & consider future occurrences○ Plan & optimize

● Parameters○ Monitor: identify potential improvements○ Re-evaluate regularly○ Future work: automate

PRESENTATION TITLESLIDE NO. 42

Example: using holidays

PRESENTATION TITLESLIDE NO. 43

To sum up...

1. Choose wisely2. Understand thoroughly3. Monitoring is not enough4. Automate5. Inspire trust6. Act7. Optimize continuously

PRESENTATION TITLESLIDE NO. 44

SLIDE NO. 45

THANK YOURuntastic.com

Place photo over me

Hilda KosorusSenior Data Scientist

hilda.kosorus@runtastic.com

SLIDE NO. 46

We are hiringDATA ENGINEERS