+

Making Data Driven DecisionsWith Eden Kung, Box and Skye Gilbert, PATH

November 16, 2017

The image part with relationship ID rId33 was not found in the file.

The Agenda

1. Introduction

2. Problem Summary

3. Creating a data driven process

4. Tackling the Problem

5. Lessons Learned

6. Q&A

2

Introduction

Eden Kung Skye Gilbert Deputy Director

Digital Health Solutions, PATHSenior Director

Business Analytics, Box

The image part with relationship ID rId40 was not found in the file.

Content

Metadata

Collaboration

WorkflowZones

KeySafe

Protection Policies

Governance

Compliance

Insights

API

Cloud Content Management from Box

The simple and secure way to bring all of your people, information and applications together to revolutionize

how you work

The image part with relationship ID rId40 was not found in the file.

We will……. Put the user first.… Collaborate for impact.… Focus on sustainability.… Uphold country ownership.… Evaluate and evolve solutions.

PATH vision:

PATH envisions a world where innovation ensures that health is within reach for everyone.

Ensure digital innovations improve health.

Guiding principles:

Digital Health mission:

6

Problem Summary

The image part with relationship ID rId40 was not found in the file. 8

The image part with relationship ID rId40 was not found in the file.

Balance Sales Potential versus Workload for each Sales Territory

• Prioritizing a rep’s accounts takes a lot of data and a lot of time• Newer reps don’t know what fields to consider

• Different reps using different prioritization methodologies, potentially missing on high potential accounts• Sales Potential: How much Total Addressable Market (TAM) is there in the territory's accounts?• Workload: How hard will it be to win deals with these accounts?

The Challenge

• Combine Predicted Priority Score with a sales rep’s input to increases “win” rates

The Vision

• Learn from experienced reps on which variable to use for prioritization•Develop and scale algorithmic score for all accounts and rank each rep's account from highest to lowest•Allow reps to modify Predicted Priority Score based on contextual knowledge

The Approach

The image part with relationship ID rId40 was not found in the file. 10

The image part with relationship ID rId40 was not found in the file.

• There are strong challenges related to data quality around immunizations in Africa, yet few can identify which problems matter most and where.

• Lack of reliable, accessible, actionable data on the barriers impeding immunizations coupled with trained and empowered data users at all levels.

The Challenge

• Empower countries to enhance immunization and overall health service delivery through improved data collection, quality, and use.

The Vision

• Partner with demonstration countries Tanzania and Zambia to:• Identify the most pressing routine immunization service delivery problems.• Develop, perfect, and scale solutions with the users on the ground

throughout the health system.• Facilitate peer learning with other sub-Saharan African countries in

design, testing, and applying interventions.

The Approach

The Data Driven Process

The image part with relationship ID rId40 was not found in the file.

Step 1

UNDERSTANDING THE TARGETS

Step 2

FINE TUNING THE DATA

Step 3

BUILDING THE DATA'S REPUTATION

Step 4

TEACHING THEM HOW TO FISH

• Who the data analytics will support and what level & kind of data is appropriate

• How to match the data to actions users can take

• How to get people to believe in the power of data

• Pass on learnings and data analytics tools so targets can make decisions with their own data

13

Tackling the Problem

1. Understanding the Targets: who the data analytics will support and what level & kind of

data is appropriate

The image part with relationship ID rId40 was not found in the file.

Youtube embed https://youtu.be/bpMKSiAgs2c?t=11s

The image part with relationship ID rId40 was not found in the file.

The image part with relationship ID rId40 was not found in the file.

The image part with relationship ID rId40 was not found in the file.

What digital health means to Lucy→ Time-consuming paper and pencil registries are

automated by time-saving digital records.→ Immunization program reports can be created and

sent quickly, building trust with supervisors→ Equipment inventory and order planning

is simplified.→ Much easier to identify children who are

overdue, and how to reach them.

Lucy is a nurse at a local clinic.

She manages the clinic’simmunization program,and is responsiblefor record-keeping, reporting, and ordering equipment. She uses paper and pencil to track allher records and must compile a handwritten report each month to the district.

The image part with relationship ID rId40 was not found in the file.

What digital health means to Lucy→ Time-consuming paper and pencil registries are

automated by time-saving digital records.→ Immunization program reports can be created and

sent quickly, building trust with supervisors→ Equipment inventory and order planning

is simplified.→ Much easier to identify children who are

overdue, and how to reach them.

Lucy is not (yet) interested in:• Demand forecasting

Lucy is a nurse at a local clinic.

She manages the clinic’simmunization program,and is responsiblefor record-keeping, reporting, and ordering equipment. She uses paper and pencil to track allher records and must compile a handwritten report each month to the district.

The image part with relationship ID rId40 was not found in the file.

What digital health means to Lucy→ Time-consuming paper and pencil registries are

automated by time-saving digital records.→ Immunization program reports can be created and

sent quickly, building trust with supervisors→ Equipment inventory and order planning

is simplified.→ Much easier to identify children who are

overdue, and how to reach them.

Lucy is not (yet) interested in:• Demand forecasting• GIS mapping of local area

Lucy is a nurse at a local clinic.

She manages the clinic’simmunization program,and is responsiblefor record-keeping, reporting, and ordering equipment. She uses paper and pencil to track allher records and must compile a handwritten report each month to the district.

The image part with relationship ID rId40 was not found in the file.

What digital health means to Lucy→ Time-consuming paper and pencil registries are

automated by time-saving digital records.→ Immunization program reports can be created and

sent quickly, building trust with supervisors→ Equipment inventory and order planning

is simplified.→ Much easier to identify children who are

overdue, and how to reach them.

Lucy is not (yet) interested in:• Demand forecasting• GIS mapping of local area• Predictive analytics for patient

outcomes

Lucy is a nurse at a local clinic.

She manages the clinic’simmunization program,and is responsiblefor record-keeping, reporting, and ordering equipment. She uses paper and pencil to track allher records and must compile a handwritten report each month to the district.

2. Finetuning the Data: how to match the data to actions users can take

The image part with relationship ID rId40 was not found in the file.

Goal of Territory Design is to balance Sales Potential versus Workload for each territory

Sales PotentialHow much upside is there in the

territory’s accounts?

Primarily driven by potential seatsAKA “Total Addressable Market”

WorkloadHow hard will it be to win deals

with these accounts?

Primarily driven by # of customers vs prospects and how good a “fit”

they are for Box

The image part with relationship ID rId40 was not found in the file.

Adjusted TAM

Total Addressable Market (TAM)

Likelihood to win a deal= X

To create equitable territories, give them equal Adjusted TAM

The image part with relationship ID rId40 was not found in the file.

→ Abdu decides which facilities require technical supervision, but does not manage payments

→ Abdu decides how to allocate stock across facilities, but not which vaccines to provision.

Abdu is a District Immunization Officer.

He plans, manages, and monitors immunization programs in his district and coordinates with the Ministry of Health. Hespendsmany hours working at the district level to improve data management

and quality.

The image part with relationship ID rId40 was not found in the file.

→ Abdu decides which facilities require technical supervision, but does not manage paymentsAbdu’s data dashboard shows service delivery indicators…

→ Abdu decides how to allocate stock across facilities, but not which vaccines to provision.

Abdu is a District Immunization Officer.

He plans, manages, and monitors immunization programs in his district and coordinates with the Ministry of Health. Hespendsmany hours working at the district level to improve data management

and quality.

The image part with relationship ID rId40 was not found in the file.

→ Abdu decides which facilities require technical supervision, but does not manage paymentsAbdu’s data dashboard shows service delivery indicators……but not financials.

→ Abdu decides how to allocate stock across facilities, but not which vaccines to provision.

Abdu is a District Immunization Officer.

He plans, manages, and monitors immunization programs in his district and coordinates with the Ministry of Health. Hespendsmany hours working at the district level to improve data management

and quality.

The image part with relationship ID rId40 was not found in the file.

→ Abdu decides which facilities require technical supervision, but does not manage paymentsAbdu’s data dashboard shows service delivery indicators……but not financials.

→ Abdu decides how to allocate stock across facilities, but not which vaccines to provision.

Abdu’s data dashboard shows stock availability at all facilities…

Abdu is a District Immunization Officer.

He plans, manages, and monitors immunization programs in his district and coordinates with the Ministry of Health. Hespendsmany hours working at the district level to improve data management

and quality.

The image part with relationship ID rId40 was not found in the file.

→ Abdu decides which facilities require technical supervision, but does not manage paymentsAbdu’s data dashboard shows service delivery indicators……but not financials.

→ Abdu decides how to allocate stock across facilities, but not which vaccines to provision.

Abdu’s data dashboard shows stock availability at all facilities…

…but not disease morbidity outside the immunization program.

Abdu is a District Immunization Officer.

He plans, manages, and monitors immunization programs in his district and coordinates with the Ministry of Health. Hespendsmany hours working at the district level to improve data management

and quality.

3. Building the Data's Reputation: how to get people to believe in the power of data

The image part with relationship ID rId40 was not found in the file.

Corporate Mid-Market Enterprise Large Ent & Verticals

% of US Prospect accounts we sold to by Predicted Priority Score

A & B C D

The image part with relationship ID rId40 was not found in the file.

Corporate Mid-Market Enterprise L Ent & Vert Corporate Mid-Market Enterprise L Ent & Vert Corporate Mid-Market Enterprise L Ent & Vert

A & B C D

% of US Prospect Accounts we sold to by Predicted Priority Score vs Rep-Entered Score

Predicted Rep-entered

Predictive score better at ID’inglow priority accounts across all

segments

Predictive score better when reps have too many accounts

to individually prioritize

The image part with relationship ID rId40 was not found in the file.

Corporate Mid-Market Enterprise Large Ent & Verticals

% of US Prospects we sold to where both Predicted and Rep-entered score were A or B

Predicted Rep-entered Both

The image part with relationship ID rId40 was not found in the file.

"I like this system because…”

Lucy’s journey from time-savings to trust to excitement

“…I no longer have to spend 2-3 days per month preparing reports.”

The image part with relationship ID rId40 was not found in the file.

"I like this system because…”

Lucy’s journey from time-savings to trust to excitement

“…I no longer have to spend 2-3 days per month preparing reports.”

“…I know that when a child shows up on my outreach list, it’s because that child truly needs outreach.”

The image part with relationship ID rId40 was not found in the file.

"I like this system because…”

Lucy’s journey from time-savings to trust to excitement

“…I no longer have to spend 2-3 days per month preparing reports.”

“…I know that when a child shows up on my outreach list, it’s because that child truly needs outreach.”

“…it tells me which mothers I need to call in order to clear my queue.”

(55) 55-55-555

4. Teach Them How to Fish: pass on learnings and data analytics tools so targets can make decisions with

their own data

38

Sales Account Dashboard

The image part with relationship ID rId40 was not found in the file.

Joseph works for the Ministry of Health.

He is responsible for estimating and ordering vaccines and immunization equipment for the country’s immunization program and ensuring that the national medical store is

fully stocked.

“I don’t have visibility into stock at point of care.”1

1. From TiMR demo (Tanzania) 2. This screenshot comes from the Logistimo user guide: https://logistimo.freshdesk.com/support/solutions/articles/24000004489-stock-out-prediction-activity-overview-export ; Logistimo is currently being rolled out in Zambia as part of the BID package at facility level 3. Orange used their data to map migration patterns in Senegal (https://phys.org/news/2016-11-migrations-mobile.html); We are currently explore machine learning solutions as well as data mashups with mobile network operators to further improve demand forecasting

Joseph drives next-generation analytics

The image part with relationship ID rId40 was not found in the file.

Joseph works for the Ministry of Health.

He is responsible for estimating and ordering vaccines and immunization equipment for the country’s immunization program and ensuring that the national medical store is

fully stocked.

“I don’t have visibility into stock at point of care.”1

1. From TiMR demo (Tanzania) 2. This screenshot comes from the Logistimo user guide: https://logistimo.freshdesk.com/support/solutions/articles/24000004489-stock-out-prediction-activity-overview-export ; Logistimo is currently being rolled out in Zambia as part of the BID package at facility level 3. Orange used their data to map migration patterns in Senegal (https://phys.org/news/2016-11-migrations-mobile.html); We are currently explore machine learning solutions as well as data mashups with mobile network operators to further improve demand forecasting

“Okay, but this doesn’t tell me where stock-outs are likely.”2

Joseph drives next-generation analytics

The image part with relationship ID rId40 was not found in the file.

Joseph works for the Ministry of Health.

He is responsible for estimating and ordering vaccines and immunization equipment for the country’s immunization program and ensuring that the national medical store is

fully stocked.

“I don’t have visibility into stock at point of care.”1

“Okay, but this doesn’t tell me where stock-outs are likely.”2

“This helps, but I need more to forecast demand.”3

1. From TiMR demo (Tanzania) 2. This screenshot comes from the Logistimo user guide: https://logistimo.freshdesk.com/support/solutions/articles/24000004489-stock-out-prediction-activity-overview-export ; Logistimo is currently being rolled out in Zambia as part of the BID package at facility level 3. Orange used their data to map migration patterns in Senegal (https://phys.org/news/2016-11-migrations-mobile.html); We are currently explore machine learning solutions as well as data mashups with mobile network operators to further improve demand forecasting

Joseph drives next-generation analytics

The image part with relationship ID rId40 was not found in the file.

Joseph works for the Ministry of Health.

He is responsible for estimating and ordering vaccines and immunization equipment for the country’s immunization program and ensuring that the national medical store is

fully stocked.

“I don’t have visibility into stock at point of care.”1

“Okay, but this doesn’t tell me where stock-outs are likely.”2

“This helps, but I need more to forecast demand.”3

1. From TiMR demo (Tanzania) 2. This screenshot comes from the Logistimo user guide: https://logistimo.freshdesk.com/support/solutions/articles/24000004489-stock-out-prediction-activity-overview-export ; Logistimo is currently being rolled out in Zambia as part of the BID package at facility level 3. Orange used their data to map migration patterns in Senegal (https://phys.org/news/2016-11-migrations-mobile.html); We are currently explore machine learning solutions as well as data mashups with mobile network operators to further improve demand forecasting

Joseph drives next-generation analytics

Lessons Learned

The image part with relationship ID rId40 was not found in the file.

Step 1

UNDERSTANDING THE TARGETS

Step 2

FINETUNING THE DATA

Step 3

BUILDING THE DATA'S REPUTATION

Step 4

TEACHING THEM HOW TO FISH

• Who the data analytics will support and what level & kind of data is appropriate

• How to match the data to actions users can take

• How to get people to believe in the power of data

• Pass on learnings and data analytics tools so targets can make decisions with their own data

For more on PATH/BID lessons learned, visit here: http://bidinitiative.org/wp-content/uploads/VAD_BID_LessonsLearned_DATAUSE_v1_rev06.pdf

Q&A

+

Thank you!@ Eden: ekung@box.com@ Skye: sgilbert@path.org