A Short Comparative Interrupted Time-Series Analysis · PDF fileA Short Comparative...

download A Short Comparative Interrupted Time-Series Analysis · PDF fileA Short Comparative Interrupted Time-Series Analysis of the Impacts of Jobs-Plus ... What is short comparative interrupted

If you can't read please download the document

  • date post

    10-Apr-2018
  • Category

    Documents

  • view

    223
  • download

    2

Embed Size (px)

Transcript of A Short Comparative Interrupted Time-Series Analysis · PDF fileA Short Comparative...

  • 1

    A Short Comparative Interrupted Time-Series

    Analysis of the Impacts of Jobs-Plus

    Howard S. Bloom

    MDRC

    Presented at the HHS Conference on Building Strong Evidence in Challenging Contexts:

    Alternatives to Traditional Randomized Control Trials, Washington, DC, September 23,

    2016.

  • Introduction

    What is short comparative interrupted time-series (CITS)

    analysis?

    It compares deviations from trends for a treatment and comparison

    group

    It is an extension of difference-in-differences analysis

    When might we use such an analysis?

    For a retrospective study of a policy change (e.g. raising or lowering speed limits or drinking ages)

    For a small-N study of a place-based initiative (e.g. a community employment, crime or health intervention)

    To study the impacts of environmental, economic or social

    disruptions (e.g. storms, earthquakes, plant closings or wars)

    For a longitudinal comparison-group study of a social program (e.g. federal employment and training programs)

    2

  • Some Benefits of Short CITS Analysis

    What you see is what you get!

    You can use it prospectively or retrospectively.

    You can use it with administrative data.

    Aggregate level

    Individual level

    You can use it when a conventional RCT is not feasible.

    3

  • 4

    A Hypothetical Killer Application of Short CITS:

    Measuring the Impact of an Oprah Book Endorsement

    0

    200

    400

    600

    800

    1,000

    1,200

    An

    nu

    al S

    ale

    s (

    1,0

    00

    s)

    Anna Karenina (Endorsed) War and Peace (Not Endorsed)

    Before Oprah Endorses After Oprah Endorses

  • 5

    Conditions for a Successful

    Short CITS Analysis

    An outcome that is measured consistently over time

    A baseline trend that is sufficiently long, frequent and stable

    Impacts that are sufficiently pronounced and immediate

    A follow-up period that is long enough to account for program

    implementation but short enough to avoid other major changes

    A comparison group with the same data (matching can help but is not always necessary)

    Covariates can be used to account for sample composition that

    changes markedly over time

  • 6

    Estimating Intervention Effects

    The basic estimator is a treatment- and comparison-group difference in deviations from their baseline trends.

    Baseline trends can be simple means or linear and (infrequently) non-linear functions of time.

    Serial correlation can sometimes be accounted for.

    Multi-level data can be accommodated.

    Matching can be used to choose a comparison group.

    Covariate adjustments can be made, if needed.

  • Origins of the Jobs-Plus Community Revitalization

    Initiative for Public Housing Families

    Jobs-Plus was an MDRC demonstration project designed to build mixed-income communities from within Response to growing concentration of joblessness, underemployment,

    welfare receipt, and poverty in public housing and surrounding neighborhoods

    Public and private Jobs-Plus sponsors US Dept. of Housing and Urban Development (HUD)

    The Rockefeller Foundation

    Other public and private funders

    7

  • 8

    The Jobs-Plus Program Model

    Three intervention components focused on selected public housing developments:

    Employment and training services

    Convenient on-site job centers

    New rent rules to make work pay

    Rents rise less than usual as earnings grow

    Community support for work

    Neighbor-to-neighbor outreach; sharing work leads,

    babysitting for working mothers, etc.

    Saturation-level outreach

    Aimed at all working-age residents

  • The Jobs-Plus Sites

    The local public housing authorities (PHAs) involved

    50 PHAs invited

    42 PHAs applied

    7 PHAs selected

    6 PHAs began participation

    3 PHAs had stronger implementation (LA, Dayton and St. Paul)

    1 PHA had stronger implementation but could not continue (Seattle)

    2 PHAs had very weak implementation (Baltimore and Chattanooga)

    Selection of treatment and comparison developments

    2 or 3 candidate developments per site

    Random assignment to Jobs-Plus of one candidate development per site

    Remaining candidate developments formed the comparison group for each site

    9

  • The Jobs-Plus Short CITS Analysis

    Time series

    Baseline: 4 6 years before 1998 launch

    Follow-up: 6 8 years after 1998 launch

    Outcome measures

    Quarterly earnings and employment rates

    Quarterly welfare receipt and receipt rates

    Analytic perspectives

    People (1998 cohort members)

    Place (local public housing developments)

    Substantive focus

    Implementation

    Impacts

    Findings reported

    Overall, by site implementation level, and by site

    For sample subgroups10

  • Pooled Average Quarterly Earnings for the 1998 Cohort

    At the Three Stronger Implementation Sites

    Rollout period

    0

    500

    1,000

    1,500

    2,000

    2,500Q

    1 1

    99

    2

    Q1

    19

    93

    Q1

    19

    94

    Q1

    19

    95

    Q1

    19

    96

    Q1

    19

    97

    Q1

    19

    98

    Q1

    19

    99

    Q1

    20

    00

    Q1

    20

    01

    Q1

    20

    02

    Q1

    20

    03

    Time Period

    Mea

    n Q

    ua

    rter

    ly E

    arn

    ing

    s R

    ecei

    pt

    (in

    20

    03

    do

    lla

    rs)

    Jobs-Plus Group

    Comparison GroupDifference due

    to Jobs-Plus =

    +$ 1,141/year

    or + 14%

  • Pooled Difference in Average Quarterly Earnings for the

    1998 Cohort at the Three Stronger Implementation Sites

    -600

    -400

    -200

    0

    200

    400

    600

    Q 1

    19

    92

    Q 1

    19

    93

    Q 1

    19

    94

    Q 1

    19

    95

    Q 1

    19

    96

    Q 1

    19

    97

    Q 1

    19

    98

    Q 1

    19

    99

    Q 1

    20

    00

    Q 1

    20

    01

    Q 1

    20

    02

    Q 1

    20

    03

    Time Period

    Dif

    feren

    ce in

    Mean

    Qu

    arte

    rly

    Earn

    ing

    s R

    eceip

    t (in

    20

    03

    do

    llars)

    Jobsplus Program vs Comparison Developments

  • Jobs-Plus Quarterly Impact Estimation Model

    Model of Quarterly Mean T/C Earnings Differences

    = + + and

    = 1+

    where

    = the difference in mean earnings for the treatment and

    comparison groups in quarter t,

    = one if quarter t is follow-up quarter m and zero otherwise,

    and 1= error terms with a first-order autoregressive structure,

    = a random error term that is independently and identically

    distributed.

    13

  • Additional Impact Estimation Steps

    Estimate annual impacts: by summing quarterly impact estimates

    Estimate standard errors of annual impact estimates: based on estimated standard errors and covariances of the quarterly impact

    estimates.

    Adjust p-values for the multiplicity of annual impact

    estimates: using a layered Bonferroni approach

    14

  • Jobs-Plus Earnings Impacts for the 1998 Cohort from the

    Three Stronger Implementation Sites

    15

    Estimated

    Percentage

    Observed Estimated Estimated Change

    Outcome with Effect of Outcome Without in Outcome Due to

    Follow-Up Period Jobs-Plus Jobs-Plus Jobs-Plus Jobs-Plus

    1998 6,089 173 5,916 2.9

    1999 7,619 209 7,410 2.8

    2000 8,793 714 ** 8,079 8.8

    2001 9,256 1,135 *** 8,121 14.0

    2002 9,419 1,171 *** 8,248 14.2

    2003 9,443 1,543 *** 7,900 19.5

    2000-2003 9,228 1,141 *** 8,087 14.1

  • Selected References

    Original Sources

    Campbell, D.T. and J.C. Stanley (1963) Experimental and Quasi-experimental Designs

    for Research. Chicago: Rand McNally, 37 43.

    Cook, T.D. and D.T. Campbell (1979) Quasi-Experimentation: Design and Analysis

    Issues for Field Settings, Chicago: Rand McNally, 207 232.

    Recent Sources:

    St. Clair, T., K. Hallberg and T.D. Cook (2016) The Validity and Precision of the

    Comparative Interrupted Time-Series Design, Journal of Educational and Behavioral

    Statistics, 41(3): 269 299.

    Wong, M., T.D. Cook and P.M. Steiner (2015) Adding Design Elements to Improve

    Time-Series Designs: No Child Left Behind as an Example of Causal Pattern

    Matching, Journal of Research on Educational Effectiveness, 8(2): 245 279.

    16

  • Selected References

    (continued)

    Recent Sources (continued)

    St. Clair, T., T.D. Cook and K. Hallberg (2014) Examining the Internal Validity and

    Statistical Precision of the Comparative Interrupted Time Series Design by Comparison

    with a Randomized Experiment, American Journal of Evaluation: 1098214014527337.

    Dee, T.S. and B. Jacob (2011) The Impact of No Child Left Behind on Student

    Achievement. Journal of Policy Analysis and Management. 30 (3): 418 486.

    Personal Sources

    Bloom, H.S., J. Riccio and N. Verma (2005) The Effectiveness of Jobs-Plus. New York:

    MDRC.

    Bloom, H.S. (1984) Estimating the Effect of Job-training Programs Using Longitudinal

    Data: Ashenfelters Findings Reconsidered, Journal of Human Resources (Fa