Net Impact Evaluation Framework For Minnesota With preliminary results as of May 2014

Net ImpactEvaluation Framework

For Minnesota

With preliminary results as of May 2014

A Standardized

Getting Started

Nick MarynsSenior Policy Analyst

Governor’s Workforce Development [email protected]

Raymond RobertsonProfessor of Economics

Macalester [email protected]

mailto:[email protected]

mailto:[email protected]

Motivations, History, Partners

Overview and Basic Parameters

Pilot ProjectEvaluation Design

Standardized

Net Impact Evaluation Framework

Preliminary Results


“Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.”

- Peter Orszag and John Bridgeland,The Atlantic Monthly, July 2013

Motivations

History

Advisory Group


Motivations

History

Advisory GroupThe National Conversation

Pew-MacArthur Results First Initiative

Results for America

Social Impact Bonds / Pay for Success


Motivations

History

Advisory Group

Apples and Oranges Approaches across the State


Motivations

History

Advisory Group The UPAM law required the development of uniform ROI measure.


Motivations

History

Advisory Group(d) Functions. The State Board shall assist the Governor in—(6) development and continuous improvement of comprehensive State performance measures, including State adjusted levels of performance, to assess the effectiveness of the workforce investment activities in the State as required under section 2871 (b) of this title;

-Section 111 of the Workforce Investment Act

Subd. 3. Purpose; duties. (c) “Advise the governor on the development and implementation of statewide and local performance standards and measures relating to applicable federal human resource programs and the coordination of performance standards and measures among programs”

-Minnesota Statute 116L.665 Subd. 3c

The GWDC’s Role


Motivations

History

Advisory Group

Community Organizations•Greater Twin Cities United Way•Lukeworks•Twin Cities RISE!

State Agencies•Department of Employment and Economic Development•Department of Corrections•Department of Education•Department of Human Services•MN State Colleges and Universities

Researchers / Evaluators•Anton Economics•Invest in Outcomes•Macalester College•Minneapolis Federal Reserve Bank•Wilder Research

Local Workforce Boards•City of Minneapolis Employment and Training Program•Minnesota Workforce Council Association•Workforce Development, Inc.

Business / Employers•Dolphin Group•MN Chamber of Commerce


Framework Design

Values

Objectives

Scope of Programs

Collaborations

Considerations and Trade-Offs

GoalA framework for measuring and

understanding the net impacts and social ROI of all publicly-funded

workforce programs that is standardized and credible, and that

informs strategy and continuous improvement


Framework Design

Values

Objectives

Scope of Programs

Collaborations


Supportive Policies and Infrastructure

Cost-Benefit Analysis

Net Impact Analysis

Oversight / ManagementFramework


Framework Design

Values

Objectives

Scope of Programs

Collaborations


Manageable, feasible to administer

Useful, relevant, timely

Credible, transparent, trusted

Adaptable, sensitive to change


Framework Design

Values

Objectives

Scope of Programs

Collaborations


Improving Services, Driving Value“What works, and for whom?”

“What disparities exist?”

Making Smarter Investments“How do current investments align to what works,

and to disparities in our community?”

Communicating Value“How do workforce services benefit

participants and taxpayers?”

Standardizing the Approach

Strengthening Transparency/Accountability


Framework Design

Values

Objectives

Scope of Programs

Collaborations


Publicly-administered and funded workforce programsDEED and other state agencies

Non-profit passthroughs

Public education (elements of K-12 and PS)

State and federal competitive grants and special initiatives

Programs serving targeted populations (e.g. people with disabilities, veterans)

Long-Term VisionIndependent nonprofits and

education providers

Other service areas


Framework Design

Values

Objectives

Scope of Programs

Collaborations


United Way

Wilder Research

Invest in Outcomes/State Pay for Performance

National Governors Association


Framework Design

Values

Objectives

Scope of Programs

Collaborations


One methodology, many programs

Ensuring usefulness to program managers and policy makers

(Unintended) incentives created by measure

Evaluation Design


Benefits

Costs

B/C Perspectives

What’s Not Included

Net Impact Analysis

Data, Not Assumptions

Net Impact

Other Features

Policy Framework

Management Framework


Net Impact Analysis

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features Costs

Benefits

Break-Even Point

ROI = (Benefits – Costs) / Costs

Time

$

(Return) (Investment)

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

EmploymentIncome / Fringe Benefits

TaxesIncome / Payroll / Sales

Public Assistance SavingsMFIP / SNAP / UI

Healthcare SavingsMinnesotaCare / Medical Assistance

Incarceration Avoidance

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Program CostsTime-weighted / Service-weighted

(where possible)

Cost to Participant

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Benefits and Costs to Participants

+Benefits and Costs

to Taxpayers=

Total Social Benefits and Costs

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Some public benefitsSubsidized housing costs

Prescription Drug Program costsChild Support payments

Other important but difficult-to-quantify effectsChange in mental and physical health

Change in worker productivityReduction in criminal activity

Economic multipliers

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Causality / True AttributionApproaching

Approaching

Kernel Density Propensity Score Matching

Difference-in-Difference Estimator

“As good as random”

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Foundation: administrative data at the

individual level

Avoid broad assumptions wherever possible

Not Used: Self-reported program

performance indicators, e.g. entered employment ratesix-month retention rate

earnings change

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features Net Impact

Comparison Group

Treatment Group

Earnings

Time

Not the relevant comparison

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Accounts for Many FactorsPersonal Characteristics

GeographyLocal Economic Conditions

Services Received

Also allows us to analyze performance by these categories

Evaluation Design


Benefits

Costs

B/C Perspectives


Net Impact Analysis


Net Impact

Other Features

Contextualized Performance GoalsAdjusted for population served, local conditions

Leading IndicatorsFor near-term relevance; based on statistical

relationships between near-term indicators and long-term outcomes

Pilot Project

Scope of Programs

Treatment

Comparison

Data Sharing

Timeframe PurposePrimarily for internal use, to test concept,

methodology, data process

The pilot evaluation comprises 950,000 individuals and roughly

50 million data points

Pilot Project

Scope of Programs

Treatment

Comparison

Data Sharing

Timeframe

Initial Cohorts (2007-08 and 2009-10)

WIA Adult ProgramWIA Dislocated Worker Program

Twin Cities RISE!

New Cohorts (2010-11)

FastTRAC I&B GranteesMFIP / DWP Employment Services

Adult Basic EducationSNAP Employment and Training

Pilot Project

Scope of Programs

Treatment

Comparison

Data Sharing

TimeframeRegistrants at WorkForce Centers

and on MnWorks.net

Unemployment Insurance Applicants

Pilot Project

Scope of Programs

Treatment

Comparison

Data Sharing

Timeframe

Pilot Project

Scope of Programs

Treatment

Comparison

Data Sharing

TimeframeNew round of Data Sharing

Agreements recently finalized

Data are currently coming in

Results this Fall

DISCLAIMER: The results reported here are preliminary and are subject to further testing and refinement that could alter the direction and magnitude of the results. A final report is forthcoming later in 2014.

Preliminary Net Impact Results

Summary StatisticsEarnings ImpactsEmployment ImpactsFastTRAC: Initial FindingsWhat’s Next

Preliminary results address earnings and employment impacts across two programs: •WIA Adult•Dislocated Worker (both WIA and MN)

Treatment cohorts are defined as such:

CohortWIA Adult 0708WIA Adult 0910DW 0708DW 0910

Exit DatesJuly 2007 – June 2008July 2009 – June 2010 July 2007 – June 2008July 2009 – June 2010

Additionally, some initial findings on FastTRAC data are also provided.

See disclaimer to the left.




The analysis that has produced the following preliminary results has been guided by the GWDC Net Impact Advisory Group and is still under development.

The preliminary results that follow have been reviewed by program directors and relevant staff at DEED, who emphasized the value of the findings and voiced their support for the continuation of the effort.




INTERPRETATION:Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

INTERPRETATION:Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.




0.2

.4.6

Ke

rne

l De

nsi

ty E

stim

ate

0 4 8 12Log of Quarterly Wage

UI

AD

kernel = epanechnikov, bandwidth = 0.0282

WIA AD 2007-2008 Entrance

PreEntrance Log Wages

0.2

.4.6

Ke

rne

l De

nsi

ty E

stim

ate


UI

AD


WIA AD 2009-2010 Entrance


Figure 1a: WIA AD 0708 Kernel Density Distribution of Pre Wages

Figure 1b: WIA AD 0910 Kernel Density Distribution of Pre Wages

INTERPRETATION:These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer.





0.2

.4.6

.8K

ern

el D

en

sity

Est

imat

e


UI

DW


WIA DW 2007-2008 Entrance


0.2

.4.6

.8K

ern

el D

en

sity

Est

imat

e


UI

DW


WIA DW 2009-2010 Entrance


Figure 2a: DW 0708 Kernel Density Distribution of Pre Wages

Figure 2b: DW 0910 Kernel Density Distribution of Pre Wages






INTERPRETATION:This table tells us the average time in program is between three quarters and a year, with a lot of variation.

INTERPRETATION:This table tells us the average time in program is between three quarters and a year, with a lot of variation.




Figure 3: Unmatched Wage Distribution: WIA Adult 0708

INTERPRETATION:In the earnings charts that follow, 0 represents time of enrollment. We worked to match earnings in the pre-period. The net impact on earnings is the average difference in the post-period, specifically quarters 5-8.

INTERPRETATION:In the earnings charts that follow, 0 represents time of enrollment. We worked to match earnings in the pre-period. The net impact on earnings is the average difference in the post-period, specifically quarters 5-8.

Matched Average Net Impact





INTERPRETATION:For WIA AD 0708, the results at right translate to a net 30% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress.





Figure 5: Unmatched Wage Distribution: Dislocated Worker 0708

INTERPRETATION:For DW 0708, the results at right translate to a net 52% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant.





Figure 6: Unmatched Wage Distribution: Dislocated Worker 0910






Figure 7: Unmatched Employment Distribution: WIA Adult 0708

INTERPRETATION:For AD 0708, the results at right translate to a net 30% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.





Figure 8: Unmatched Employment Distribution: WIA Adult 0910






Figure 9: Unmatched Employment Distribution: Dislocated Worker 0708

INTERPRETATION:For DW 0708, the results at right translate to a net 6% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.





Figure 10: Unmatched Employment Distribution: Dislocated Worker 0910






A statistical analysis of FastTRAC is forthcoming; FastTRAC data have presented a number of challenges, many of which illustrate common data challenges we face.

Funded as a pilot project through the Joyce Foundation, the MN FastTRAC model was not initially designed to measure outcomes based on placement, but instead focused on educational attainment among a hard-to-serve population (likely MFIP participants).

The model also allowed flexibility across local service providers, which created greater differences in self-reporting outcomes by each provider.




Specifically, gathering data on program participants has presented the following challenges:

1.Participant data did not require entry into one database but relied on local systems. Data entry is now entered into WF1.

– Some participants are excluded from data altogether depending on program completion, placement, or continuation of their academic program.

– Entrance/exit dates may be defined inconsistently across programs

– Program activities/services may be used and/or defined inconsistently (trying to adapt to other programs within WF1)

2.FastTRAC participants are characterized in part by the multiple barriers they face; accordingly, it may be more difficult to find strong control group matches for them.

3.Small sample sizes and variance among FastTRAC participant characteristics make statistically significant results harder to obtain.




As FastTRAC has evolved, the pilot recognized challenges with collecting data to measure program impacts.

Progress is being made to make data collection practices more complete and consistent across FastTRAC programs.

Data practices are improving, but it will take time for those changes to be reflected in net impact analyses since those analyses require at least one year of post-enrollment data.




Further analysis is currently underway. Here’s what to expect:

1.Further refinement of matching and estimation, to improve the statistical significance of the results.

2.Additional net outcomes measured over longer time periods, including usage of public benefits, reincarceration, and associated monetary (ROI) impacts.

3.Additional programs to be analyzed, including FastTRAC, Adult Basic Education, MFIP Employment Services, and SNAP Employment and Training.

4.Results disaggregated by participant characteristics (e.g. race, gender) and other factors.

Wrapping Up

Discussion and Questions

Raymond RobertsonProfessor of Economics

Macalester [email protected]

Nick MarynsSenior Policy Analyst

Governor’s Workforce Development [email protected]

Net Impact Evaluation Framework For Minnesota With preliminary results as of May 2014

Documents

Transcript of Net Impact Evaluation Framework For Minnesota With preliminary results as of May 2014