The Disability Employment Initiative (DEI): Impact Evaluation Design October 21, 2015 Sung-Woo Cho,...

The Disability Employment Initiative (DEI): Impact Evaluation Design

October 21, 2015

Sung-Woo Cho, Ph.D.

Abt Associates | pg 2

Introducing Myself

Associate at Abt Associates

Project Director of the DEI impact evaluation

Most of my work is in impact evaluation of programs in community colleges and K-12

Past work has been in community college students and their outcomes

Teach applied statistics at The George Washington University


DEI Round 5 Overview

Six grantees in six different states

Intervention consists of services that are designed to assist people with disabilities who are seeking employment

Flagged DEI Round 5 participants: TREATMENT group

Using a rigorous evaluation design, we would want to compare these people with a comparison group, tracking their outcomes along the same points in time


Design Phase

Not randomized – we are using two quasi-experimental designs (QEDs) to determine impact of DEI interventions as a whole, and impact of a career pathways component (part of the intervention)

Outcomes include postsecondary credentials, employment, wages

Will incorporate a survey (Abt SRBI) to determine disability type and ADL (adult daily living) information

Treatment and comparison – at the LWIA level. Services provided at AJCs (One-Stops)


DEI Round 5 Interventions

The actual interventions across the six grantees vary

However, career pathways component is consistent throughout the grantees

Some of the interventions are based on previous rounds’ interventions

Examples of interventions for DEI Round 5:

– Wrap-around services (South Dakota)

– Remedial skill development (California)

– Disabilities resource coordinator (Minnesota)


Quasi-Experimental Design (QED)

The basic idea is to match a treatment group of customers to a comparison group of similar customers– Match by using their characteristics: gender, age,

ethnicity, other demographic and wage information, and disability type

– Pre-enrollment wage information would be very good to have, since there’s a good amount of variation here

• Wages are also an outcome of interest

In the end, you have a treatment group and comparison group of customers that look similar to one another on key characteristics – except only the treatment group received the DEI Round 5 services


Comparison Groups

Our calls with the grantees indicated that the participants in other local LWIAs could serve as our comparison group

All else equal, similar participants in the comparison LWIAs received a different set of services compared to participants in treatment LWIAs

We ruled out trying to create a comparison group directly from community and technical colleges

– The data are difficult to collect directly from these institutions, and general population may be hard to match


Disability Type and Matching

Comparison group – would need to match on people with self-reported disabilities

The treatment LWIAs will have a participant tracking system (PTS) that will collect information on disability type

However, the comparison LWIAs will not have a PTS– Solution: A survey that will collect information on

disability type among comparison LWIA participants who disclose a disability


QED and Baseline Equivalence

Once you have a treatment group and comparison group that look similar to one another, we want to measure their baseline (that is, pre-intervention) characteristics

– Ex: using wage information prior to start of the intervention to measure baseline equivalence

If you can show that the treatment and comparison groups are very similar at baseline, you can look at the difference in the two groups’ outcomes to measure the impact of the intervention


QED VisualO

utco

me

TimeStart of services

Treatment

ComparisonBaseline equivalence established

Impactestimate


QED Matching Strategy

10 Treatment Customers

30 Comparison Customers


Match Based on Characteristics




20 Comparison Customers are Not Matched (in red)




And They are Left Out of the Sample in a 1:1 Matching Strategy




Important Notes on Matching

The matching is done at the person (participant) level

– We should use information about the LWIAs when we do the matching – but keep in mind that we are not matching LWIAs themselves

– However, we keep in mind geography of the LWIAs

Each analysis would first be run within each state

– We will eventually group the six states together, to determine the overall impact of the Round 5 interventions


Measuring Outcomes

Outcomes will be measured by WIASRD and Wagner-Peyser reported outcomes

These can include, but are not limited to, the following:

– Employment

– Wages

– Credential completion (self-reported)


Potential Limitations

Interventions vary in strength and have different approaches to operationalizing career pathways programs. Grantees typically focus on adding services/components to existing services.

May be difficult to detect impacts on outcomes given nature of the interventions

Small sample sizes could contribute to issues in power

– Makes it harder to determine if impacts are actually present


Why is this Important?

People with disabilities who are seeking employment – an important group to help

DOL has demonstrated this with large financial support Want to figure out if the initiatives are making an impact on education and employment outcomes

Survey information will give us more info on disability type, which is not collected in enough detail at AJCs

Through the evaluation, we are setting up data collection systems that will allow for easier information retrieval on this population and the types of services they receive


Concluding Remarks

A rigorous QED is possible, assuming that we will have information on an appropriate comparison group of non-Round 5 customers in other local WIAs

We will use data from WIASRD and Wagner-Peyser, with information on demographic characteristics and outcomes

We would use a matching strategy to create a comparison group for each treatment group

– A survey will help collect information on comparison group disability type

Implementation study will be important for understanding how to provide career pathways programs to this population

The Disability Employment Initiative (DEI): Impact Evaluation Design October 21, 2015 Sung-Woo Cho,...

Documents

Transcript of The Disability Employment Initiative (DEI): Impact Evaluation Design October 21, 2015 Sung-Woo Cho,...