7 Steps for Data-Driven Decision Making

Questions? [email protected] #7steps

7 Steps for Data-Driven Decision-Making

Sacha Litman, Managing Director, Measuring Success

José Fernández, Director of GuideStar Exchange


Agenda

5 minutes Opening poll and introduction5 minutes Benefits to using data15 minutes Illustrative Case study5 minutes Q&A and concurrent poll25 minutes 7 Steps to data-driven decisions5 minutes Take-away principles & talking points 10 minutes Q&A and concurrent closing poll1 minute Conclusion


A. Benefits


Benefits of data-driven decision making to Customers

Success matters because your mission is important!If you don’t measure and use data, you are not achieving the full potential of your mission.


Benefits to Management & StaffFocus limited time and energies on key activities that are real issues and have impact on our mission

Set goals for staff and hold them accountable

“Cannot manage what you do not measure” – Peter Drucker

“What gets measured gets done”

“High performing organizations use data analytics roughly three times as extensively as the lower performers in their field” – Competing on Analytics, HBS Press


Benefits to Boards and Donors

BoardsFocus their energies away from emotions & anecdotes

Decisions not made by loudest or wealthiest voice

“Analytical projects aimed at improved outcomes had a median ROI of 55% for CRM and 139% for financial management”– Competing on Analytics, HBS Press

DonorsAbility to show “return on investment”

Attract large gifts and grants by confidence engendered by data

Data analytics was one of the top three areas of increased investment by companies during the recession.


Popular Press ExamplesProfessional sports: Oakland A’s (Moneyball)

Maximize team wins while minimizing payrollBy hiring players as indicated by data, not scouts’ gut

Health Care: Intermountain Health Care (NY Times Magazine Nov 9, 2009)Improve patient outcomes (survival, longevity, quality of life) while minimizing cost By selecting treatment/procedure supported by evidence not Doctor’s gut instinct or unneccessary procedures

Technology: GoogleIncrease advertising revenues ($21 billion) by targeting ads to right customersBy mining data on all users of free Google consumer software


B. Quick Case Study


1. Chapter-based community centers

Experiencing some with member drops, financial insolvency. Challenge to business model & brand consistency.

Association HQ

Local Affiliates or Regions(N=150)

Individual Members / Users(N=1000 to 10,000 per affiliate)


2. Association sought “early warning system” & understanding of what led to

successful outcomes


3. Build measurement and dashboard system as pilot experiment

Engaged 6 willing chaptersBuilt:

Customer surveyEmployee surveyFinancial analysis toolMember participation tool


4. Focused on rankings and metrics that were statistically valid

Regression analysis ties activities to outcomesComparison

Against peer chapters in other geographiesAgainst local competition from other organizationsAgainst own prior measures (longitudinal)Within demographic segments of member base(see following slides)


4a. Identify activities associated with outcomes (multiple regression analysis)

Budget Management versus Value for Membership DollarCorrelation

Perceived Budget Management & Transparency(1= strongly disagree, 5 = strongly agree)

Valu

e fo

r Mem

bers

hip

Dol

lar

Ave

rage

Sco

re 1

-5 S

cale


5. Like doctor, ran diagnostics on each chapter every 2 years

Rolled out pilot to all chapters; readiness factors to participateMutually agreed on a plan for improvementSome consulting support

Select Measures from Customer Survey

Rank (of 15)

Score% StrAgree

Priority Goals & Strategy

Membership Value for the Dollar 7 25% Medium Focus on quality, budget perceptions

Professionals welcoming 2 35% Low

Budget Perceived as well managed

14 16% High Double scores in 2 years.


2. Consumer Latent Needs(Purpose-Driven)

3. InstitutionalStrategic Vision

4. FinancialSustainability

(Surplus/Deficit)5. Resource Intensiveness

(Mgmt Time, Sqft, Foot Traffic)

6. Contribution to People Pipeline

(Intake, Attrition)

1. Consumer Expressed

Wants

Data Creates OrganizationalAlignment & Intentionality


6. Turnaround: focus lead to improvement from low to average in 2 years; now aiming for top

Personal Conversations with CustomersMonthly or more often

27%


Data encourages prioritization:80% of board & mgmt team hypotheses about what we

anecdotally “believe” is a problem is not supported by data!


7. Chapter ruled by anecdotes: 80% of assumptions were not supported by data

Assumed they were strongest with eldest & wealthiest portions ofparticipant base

Shock, challenged the data, acceptanceManagement team focused energies on improvement with eldest and wealthiest. 2 years later, scores significantly higher there.

50% 43% 50% 48% 45% 40%

Less than $100,000

$100,000-$199,999

$200,000-$299,999

$300,000-$399,999

$400,000-$499,999

$500,000 and over

40% 42% 67% 63% 42% 46%

25-34 35-44 45-54 55-64 65-74 Over 75

Likelihood to recommend to a friend% Strongly Agree


8. Rising Tide Lifts All Boats: top, average, and low performers all improved

XX

Surplus Margin for Early Childhood ProgramSurplus as % of Expenses, after allocating all overhead


9. New policy: value, not priceAssumed that key driver of member retention was priceAnalysis shows not price, but perceived quality and value-for-dollarResult: association stopped encouraging price subsidization, encouraged perceived quality improvementPerceived value and value for dollar are tracked carefully and promoted system wide

Highest WillingnessTo Pay (center)

A. FinancialAbility

B. CommitmentTo Issue

C. PerceivedQuality of Services Least Ability to Affect

Demand Function


10. Outcome improvement: participationChapters that embraced this approach outperformed others significantly in participant enrollment and financial sustainability, despite the recessionParticipant Retention Rate:

91% to 96%New Participant Rate:

5% to 10%

Net Participation: 96% to 106%

Financial sustainability increased: coverage ratio (% expenses from fees & membership) grew from 74% to 80%


C. 7 Steps to Data-Driven Decision Making


Data driven decision making is parallel to what we learned in middle school science class

Identify issueState hypothesis: “I believe…”Perceived mechanism/ causeDesign experimentExamine dataConfirm or reject hypothesis


7. Commun-

ication

6. Decision Making

5. Inter-

pretation

4. Data Analysis

3. Data Collection

7 Stages of Data-Driven Decision Making

2. Hypothesis Develop-

ment

1. Framing

the Problem


2. Hypothesis Development

Brainstorm hypotheses. Many will turn out to be incorrect, but that’s ok so long as you can articulate a plausible mechanism.

Remember, our consistent experience with organizations is that over 80% of their initial hypotheses as to what is driving a problem are not supported by the data!

During brainstorm define:Hypothesis (may need to revise several times)Mechanism (cause & effect)How could measure it


2. Hypothesis Development: Gann Academy

Initial Hypothesis: unevenness in students’ experience getting their individual learning needs met by the schoolPlausible Mechanism: additional academic support (learning center).

Major differentiator in students’ learning experience (for those in the same classrooms)

Final Hypothesis: students who received additional academic support from learning center felt their individual learning needs were better met by the school.


2. Hypothesis Development: Gann Academy

How measure? Parent surveyQuestion on individual learning needs met by schoolCut data by whether student used learning center

t

Learning Center Yes

Learning Center No

84 2403.5 4.0

Scale 10a. Teachers Understand

Needs & Employ Effective

Techniques

Strongly Disagree 3%Disagree 9%Neither Agree Nor Disagree 16%Agree 44%Strongly Agree 28%

Average 3.8

CountAverage


3. Data CollectionDo…

Prioritize hypotheses Work backwards: figure out how will use results of data first before collect it Consider several methods of data collection Save time and money by considering simple, pre-built tools allowing some customization (instead of building own or buying highest end product you won’t use)

Don’t…Just go after what’s easiestOvertax your resources –data collection can be very time intensiveGo after more data than you can act upon –Garbage In Garbage Out (GIGO)


3. Data CollectionTool Pro Con

Survey Data collection phase is automated and not resource intensive

Obtain perceptions of quality

To get quality results and high response rates, need to put lots of time into asking the right questions, making it look professional, and assuring confidentiality

Financial analysis Data your organization already has

Requires reorganizing and “fine slicing” the financials, so need a strong CFO

Tracking system (CRM)

Integrated into daily efforts Takes a lot of discipline and time to build data

Self-reported from existing databases

Already have the database Time consuming if database does not contain the right information may end up with lots of estimates


3. Data Collection : Gann AcademyGood: financial modeling

Clarified board’s needs up front: 5 year projections, key levers of sustainabilityUsed already-built but simple model (in Excel) instead of building ownWorked hard to gather consistent financial data (counting $ in development and CFO’s office)“What if” scenario building enabled board buy-in

Bad: admission trackingPurchased high end database systemNot trained properlyOrphan Excel spreadsheetsNever clarified what management needed for decision making, so “GIGO”


3. Data Collection : Gann AcademyWork backwards. Decide how you will use the data first, as this will dictate whether and how to ask for the data.


You don’t need to be a statistician to correctly interpret dataHowever, do need to understand basic concepts

MeanStandard deviationCounts (sample size)Benchmarking

Frequently, someone will “find” something in the data that is not valid but it supports their personal views

5. Interpretation: draw correct conclusions from data. Danger in “seeing what want to believe”


Board member who was convinced the most severe threat on enrollment was from charter and public schools, not area private schools Hypothesis: public schools are better at supporting individual student needs and thus pose the greatest risk of attrition for Gann Academy families. Data source: teacher attention to individual needs cut by most attractive alternative option

5. (Mis)Interpretation: Gann Academy

Private Schools

Charter or

magnet public

schools

General public

schools

50 4 513.8 3.4 3.6

CountAverage

Relative Strength of Gann Academy’s Attention to Individual Needs vs.

Most Attractive Alternative School (higher score is better)

Gann vs. Gann vs. Gann vs.


Contextualization is critical:Peer group (similar organizations nationally/internationally)Local competition*Own longitudinal historyAcross demographic groups*

*Depicted on a prior slide

5. Interpretation: Benchmarking Relative to What?


Relative to Peer Group of other similar high schools around the countryNote if had used primary schools (K-8) as peer group

5. Interpretation: Benchmarking


7. Communication

Sharing with board, participants, donors, employeesFear: sharing data will undermine management’s authorityReality: sharing data engenders trust

Diffuse concerns through transparencyEven if current scores are not as high as would like, donors value a roadmap toward and metrics for success


7. Commun-

ication

6. Decision Making

5. Inter-

pretation

4. Data Analysis

3. Data Collection

7 Stages of Data-Driven Decision Making

2. Hypothesis Develop-

ment

1. Framing

the Problem


Common Non-Profit Applications of Data-driven Decision Making

Outcome measurement

Identifying activities that give greatest “bang for buck” in limited resource environment

Financial sustainability

Setting program fees or membership dues levels

Identifying most susceptible demographic groups

Donor prioritization using predictive models

Board reporting

Program evaluation


If your organization makes data-driven decisions you will be able to…

Focus limited energies and resources on only the 20% of hypotheses that are supported by the data

Motivate staff and volunteers because they know they are investing in what works

Align efforts of staff, volunteers, board, donors, and customers

Monitor progress toward goals

Have a dashboard for your performance

Make strategic board decisions more efficiently and gain more buy-in

Distinguish yourself against and outperform your competition by developing a core competence few non-profits have

Observe improvements in key areas such as: financial sustainability, donations, participation, mission impact


7 Steps for Data-Driven Decision-Making

Sacha Litman, Managing Director, Measuring Success

José Fernández, Director of GuideStar Exchange

7 Steps for Data-Driven Decision Making

Business

Transcript of 7 Steps for Data-Driven Decision Making