A Framework for Studying Variation in Program Effects June 14, 2015 Michael J. Weiss Based on:...

1

A Framework for StudyingVariation in Program Effects

June 14, 2015Michael J. Weiss

Based on: Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A Conceptual Framework for Studying the Sources of

Variation in Program Effects. Journal of Policy Analysis and Management, 33(3).

2

Outline

Section I: Introduction

Section II: Proximal Sources of Variation in Program Effects

Section III: Connecting the Treatment Contrast to Program Implementation & Treatment Fidelity

3

Section 1

Introduction

4

Consequences of Effect Variation

• Targeting services (types of clients / locations)

• Equity / fairness• Best case / worst case scenarios• Generalizability / Scaling• Research design–What research questions to address?– Sample size– Etc.

5

Three Motivating Studies

• After School Programs– certain types “work” (sequenced activities, active

learning, limited # of goals, activities tied to goals)

• Charter Schools– certain types “work” (clear behavior standards,

zero-tolerance, coaching new teachers)

• Welfare-To-Work– certain types “work” (counseling to get jobs quick,

personalized attention)– context mattered (low unemployment rate larger

effects)

6

DEFINING A PROGRAM EFFECT FOR AN INDIVIDUAL

Section 1: Introduction

7

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnNeyman

Neyman

8

ESTIMATING THE AVERAGE PROGRAM EFFECT FOR A GROUP OF INDIVIDUALS


9

Estimating the Average Program Effect

Randomize individuals (or groups) to: 1. Receive program offer2. Not receive program offer

Compare two group’s average future outcomes

10

DEFINING VARIATION IN PROGRAM EFFECTS


11

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnNeyman

Neyman

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnRubin

Rubin

The Framework

Outcomes

Treatment Contrast

Program Effect

12

Treatment Offered

Take-up

Treatment Offered

Treatment Fidelity

Implementation Process

Client Characteristics and Context

Characteristics of Implementing Organization

Implementation Plan

Treatment Planned

Prog

ram

Mod

el

Take-up

OutcomesTreatment Received

Treatment Received

13

Section 2

Proximal Sources of Variation in Program Effects

14

The Three C’s

1. Treatment Contrast

2. Client Characteristics

3. Program Context

15

1. THE TREATMENT CONTRAST

Section 2: Proximal Sources of Variation in Program Effects

16

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnNeymanContrast 1

Neyman

Weight Loss Program

Alternate Program

Potential Outcome

Potential Outcome

ProgramEffectOnNeymanContrast 2

Neyman

Treatment Contrast and Program Effect

Treatment Received

Outcomes

Outcomes

Treatment Received

Treatment Contrast

Program Effect

17 Client Characteristics and Context


Treatment Fidelity

Implementation ProcessImplementation

Plan

Treatment Planned

Prog

ram

Mod

el Treatment Offered

Take-up

Take-up

Treatment Offered

Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod

el

Outcomes

- Employment - Achievement - Recidivism - etc.

Outcomes

- Employment - Achievement - Recidivism - etc.

Treatment Received

- Content - Quantity - Quality - Conveyance

Mediators

Treatment Received

- Content - Quantity - Quality - Conveyance

Mediators

18

2. CLIENT CHARACTERISTICS


19

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnHigh-risk Type

High-Risk

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnLow-risk Type

Low-Risk

Client Characteristics as Moderators

Treatment Received Outcomes

Treatment Contrast

Program Effect

OutcomesTreatment Received



Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod


Take-up

Take-up

Treatment Offered

Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod

el

21

Author’s Suggestions (Client Characteristics as Moderators)

1. Risk• E.g., pre-test, pre-RA income level

2. Readiness• E.g., acknowledgment of problem

(substance abuse)

3. Program Theory• E.g., age (adult education that relies on

technology)

22

3. PROGRAM CONTEXT


23

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnNeymanContext 1

Neyman

Weight Loss Program

No Program

Potential Outcome

Potential Outcome

ProgramEffectOnNeymanContext 2

Neyman

Context as a Moderator

Treatment Received

Outcomes

Outcomes

Treatment Received

Treatment Contrast

Program Effect



Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod


Take-up

Take-up

Treatment Offered

Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod

el

25

Section 3

Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity

26

FROM TREATMENT OFFERED TO TREATMENT RECEIVED: CLIENT TAKE-UP

Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity

Client Take-Up

Treatment Received

Outcomes

Outcomes

Treatment Received

Treatment Contrast

Program Effect



Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod


Take-up

Take-up

Treatment Offered

Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod

el

28

Author’s Suggestions (Strategies to Increase Client Take-Up)

1. Incentives / Inducements • ASAP’s MetroCard tied to attending advising

sessions

2. Requirements / Mandates • Opening Doors required student success course

3. Remove Barriers• H&R Block’s FAFSA simplification• 401(k) saving plans “opt-out”

4. Improved Outreach• High school to college transition, summer

outreach

29

FROM TREATMENT PLANNED TO TREATMENT OFFERED: PROGRAM IMPLEMENTATION

Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity

Program Implementation

Treatment Received

Outcomes

Outcomes

Treatment Received

Treatment Contrast

Program Effect



Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod


Take-up

Take-up

Treatment Offered

Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod

el

1

2

3

31

Treatment Plan

Definition: Planned content, quantity, quality & conveyance of treatment services for clients

• The plans for “what, how much, how, and by whom?” of treatment

• Varies in specificity

1

32

Implementation Plan

Definition: Set of instructions for how the treatment plan is to be realized

Worth Considering:• Staff recruitment and selection• Staff training• Staff monitoring and supports

2

33

Implementing Organization(Author’s Suggestions )

1. Strong Leadership

2. Resources / Capacity

3. Supportive Culture / Climate

4. Involvement of Outside Monitor or “Fixer”

3

Treatment Fidelity

Treatment Received

Outcomes

Outcomes

Treatment Received

Treatment Contrast

Program Effect



Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod


Take-up

Take-up

Treatment Offered

Treatment Fidelity


Plan

Treatment Planned

Prog

ram

Mod

el

35

A Framework for StudyingVariation in Program Effects

Michael J. Weiss Howard S. Bloom Thomas Brock [email protected] [email protected] [email protected]

mailto:[email protected]



A Framework for Studying Variation in Program Effects June 14, 2015 Michael J. Weiss Based on:...

Documents

Transcript of A Framework for Studying Variation in Program Effects June 14, 2015 Michael J. Weiss Based on:...