OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2...

46
SLIDE NO. 1 TAKE CONTROL OVER YOUR KPIs Hilda Kosorus Senior Data Scientist

Transcript of OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2...

Page 1: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 1

TAKE CONTROL OVER YOUR KPIsHilda KosorusSenior Data Scientist

Page 2: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 2

About me

2017

Runtastic: Senior data scientist

2015

Runtastic:Data scientist

2010

JKU: Research assistant

Academia Industry

2013

JKU:Senior researcher

Page 3: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 3

ABOUT RUNTASTIC

Page 4: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 4

4We are

founders

8We are

years old20

We were profitable after just

months

235We are

employees

40+We come from

countries

3We have

offices in Linz, Viennaand Salzburg

15Our products are

available in

languages

1We are

team with a shared vision

RUNTASTIC BY THE NUMBERS

Page 5: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 5

Page 6: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 6

Page 7: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 7

EACH DAY WE HAVE

150,000NEW DOWNLOADS

WE HAVE MORE THAN

130MREGISTERED USERS

WE BOAST OVER

246MTOTAL APP DOWNLOADS

WE HAVE MORE THAN

4.8MFANS AND FOLLOWERS

Page 8: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

TAKE CONTROL OVER YOUR KPIsAdvice from a Data Scientist

Page 9: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 9

WHAT and WHY?

Page 10: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 10

What are KPIs?

KPI = Key Performance Indicator

“a quantifiable measure used to evaluate the success of a business”

Page 11: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 11

Why are KPIs useful?

measurable / quantifiable

data-driven

gauge the effectiveness of processes

business health indicators

setting goals

common language → common goal

Page 12: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 12

Examples

● Number of newly acquired customers● Revenue● ROI● Retention/churn rate● Conversion rate● NPS● Sales growth

Page 13: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 13

HOW: 7 STEPS

Page 14: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 14

1. CHOOSE WISELY

Page 15: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 15

Choose SMART

● Specific - has a clear purpose for the business

● Measurable

● Achievable

● Relevant to the success of the organization

● Time phased - the value represents a predefined and relevant period

Page 16: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 16

You advise, management decides

● Be proactive● Do your homework● Communicate & consult● Make it work either way

Page 17: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 17

2. UNDERSTAND THOROUGHLY

Page 18: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 18

Why should you care?

● You can’t optimize what you don’t understand

● Won’t know why they perform good/bad

● Can’t act upon them effectively

● Waste of resources → invested in the wrong place for the wrong reasons

● Most probably won’t reach your goals

Page 19: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 19

What should you do?

1. Where is the data coming from? ● Is the source reliable?

2. How is the metric calculated?3. Analyze

● components, trend, seasonality, edge cases● relationships to other company metrics & drivers● at different granularity levels

4. Monitor● seasonality effects, patterns, outliers

Page 20: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 20

Focus on the basic components

Almost all KPIs are aggregates

● Count of non-repeatable events○ without churn (e.g. # registered users)○ with churn (# active subscribers)

● Windowed aggregates (e.g. # monthly active users)● Ratio (e.g. % conversion rate)

Page 21: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 21

What a DS should see:

Example: # active subscribers

new

churn

What a managers sees:

Page 22: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 22

Monitor

Page 23: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 23

3. ESTIMATE FUTURE BEHAVIOR

Page 24: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 24

After all, they are just indicators...

● KPIs are similar to measuring instruments (e.g. thermometer)● useful to know past/current state (temperature increased or decreased)

It is more critical to know what opportunities and dangers lie ahead.

Drivers are your most valuable allies.

Page 25: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 25

How to predict?

● Forecasting methods○ growth-based○ curve fitting (linear, polynomial, regression methods, etc.)○ probabilistic models (survival analysis, HMM)○ time series forecasting (ARIMA)○ machine learning (e.g. classification)

● A plethora of libraries (R, python)

Page 26: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 26

Time series forecasting

● developed by Facebook DS team● open source● python & R● facebook.github.io/prophet

Page 27: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 27

How does it work?

● decomposable time series model○ trend, seasonality, holiday

○ models non-period changes○ represents periodic change○ represents effects of irregular

events● interpretable parameters

Page 28: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 28

Capabilities

● Growth (linear, logistic)● Changepoints● Seasonality components

○ yearly, monthly, weekly, daily○ additive, multiplicative

● Holidays○ past and future dates○ lower and upper window of effect

● Deals with missing values● Deals with irregularly spaced data● Diagnostics● Visualizations

Page 29: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 29

Why prophet?

● automatic forecasting techniques○ hard to tune○ too inflexible to incorporate useful assumptions or heuristics

● prophet○ flexible, easy to use○ fast○ interpretable○ it works!

Page 30: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 30

Example in R

Page 31: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 31

4. AUTOMATE

Page 32: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 32

ETL

Reports

R + T-SQL

Data sources

Interactive dashboard

Deployment infrastructure

Page 33: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 33

1

2

R and Transact-SQL

Page 34: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 34

Learnings

● Communication & collaboration○ Define clear responsibilities (DE, DS)

● Efficient deployment○ Define process○ Empower DS to deploy independently○ Code reusability○ Standardize forecasting data products

● Platform stability○ R launchpad management on SQL server

Page 35: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 35

5. INSPIRE TRUST

Page 36: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 36

Evaluate for model selection

● Historical data for training● Parameter selection

○ changepoint prior○ seasonality prior○ growth method○ etc.

● Cross-validation● Evaluation metrics (e.g. MAPE)● Visualization approaches

Page 37: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 37

Evaluate past predictions

● How accurate were our past predictions?● Monitoring● Sanity checks

○ trend changes○ seasonal changes○ other interferences

● Re-evaluate parameter settings

Page 38: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 38

6. ACT

Page 39: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 39

Making use of predictions

● Help understand trends and seasonality● Defining goals

○ realistic○ ambitious

● Detect trend changes early on○ automate

● Derive meaningful actions○ resource allocation○ identify measures with high impact

Page 40: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 40

7. OPTIMIZE

Page 41: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 41

Good enough, but not great

● Granularity○ Break down metric for consistent behaviour ○ Segments (e.g. product, platform, acquisition source)○ Components (e.g. new and churned customers)

● Events -- non-organic interferences (e.g. ads, campaigns, offers)○ Identify effect○ Collect data & consider future occurrences○ Plan & optimize

● Parameters○ Monitor: identify potential improvements○ Re-evaluate regularly○ Future work: automate

Page 42: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 42

Example: using holidays

Page 43: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 43

To sum up...

1. Choose wisely2. Understand thoroughly3. Monitoring is not enough4. Automate5. Inspire trust6. Act7. Optimize continuously

Page 44: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

PRESENTATION TITLESLIDE NO. 44

Page 45: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 45

THANK YOURuntastic.com

Place photo over me

Hilda KosorusSenior Data Scientist

[email protected]

Page 46: OVER YOUR KPIs TAKE CONTROL Senior Data Scientist Hilda ... · Senior Data Scientist. SLIDE NO. 2 About me 2017 Runtastic: Senior data scientist 2015 Runtastic: Data scientist 2010

SLIDE NO. 46

We are hiringDATA ENGINEERS