1
A Short Comparative Interrupted Time-Series
Analysis of the Impacts of Jobs-Plus
Howard S. Bloom
MDRC
Presented at the HHS Conference on Building Strong Evidence in Challenging Contexts:
Alternatives to Traditional Randomized Control Trials, Washington, DC, September 23,
2016.
Introduction
What is short comparative interrupted time-series (CITS)
analysis?
It compares deviations from trends for a treatment and comparison
group
It is an extension of difference-in-differences analysis
When might we use such an analysis?
For a retrospective study of a policy change (e.g. raising or lowering speed limits or drinking ages)
For a small-N study of a place-based initiative (e.g. a community employment, crime or health intervention)
To study the impacts of environmental, economic or social
disruptions (e.g. storms, earthquakes, plant closings or wars)
For a longitudinal comparison-group study of a social program (e.g. federal employment and training programs)
2
Some Benefits of Short CITS Analysis
What you see is what you get!
You can use it prospectively or retrospectively.
You can use it with administrative data.
Aggregate level
Individual level
You can use it when a conventional RCT is not feasible.
3
4
A Hypothetical Killer Application of Short CITS:
Measuring the Impact of an Oprah Book Endorsement
0
200
400
600
800
1,000
1,200
An
nu
al S
ale
s (
1,0
00
s)
Anna Karenina (Endorsed) War and Peace (Not Endorsed)
Before Oprah Endorses After Oprah Endorses
5
Conditions for a Successful
Short CITS Analysis
An outcome that is measured consistently over time
A baseline trend that is sufficiently long, frequent and stable
Impacts that are sufficiently pronounced and immediate
A follow-up period that is long enough to account for program
implementation but short enough to avoid other major changes
A comparison group with the same data (matching can help but is not always necessary)
Covariates can be used to account for sample composition that
changes markedly over time
6
Estimating Intervention Effects
The basic estimator is a treatment- and comparison-group difference in deviations from their baseline trends.
Baseline trends can be simple means or linear and (infrequently) non-linear functions of time.
Serial correlation can sometimes be accounted for.
Multi-level data can be accommodated.
Matching can be used to choose a comparison group.
Covariate adjustments can be made, if needed.
Origins of the Jobs-Plus Community Revitalization
Initiative for Public Housing Families
Jobs-Plus was an MDRC demonstration project designed to build mixed-income communities from within Response to growing concentration of joblessness, underemployment,
welfare receipt, and poverty in public housing and surrounding neighborhoods
Public and private Jobs-Plus sponsors US Dept. of Housing and Urban Development (HUD)
The Rockefeller Foundation
Other public and private funders
7
8
The Jobs-Plus Program Model
Three intervention components focused on selected public housing developments:
Employment and training services
Convenient on-site job centers
New rent rules to make work pay
Rents rise less than usual as earnings grow
Community support for work
Neighbor-to-neighbor outreach; sharing work leads,
babysitting for working mothers, etc.
Saturation-level outreach
Aimed at all working-age residents
The Jobs-Plus Sites
The local public housing authorities (PHAs) involved
50 PHAs invited
42 PHAs applied
7 PHAs selected
6 PHAs began participation
3 PHAs had stronger implementation (LA, Dayton and St. Paul)
1 PHA had stronger implementation but could not continue (Seattle)
2 PHAs had very weak implementation (Baltimore and Chattanooga)
Selection of treatment and comparison developments
2 or 3 candidate developments per site
Random assignment to Jobs-Plus of one candidate development per site
Remaining candidate developments formed the comparison group for each site
9
The Jobs-Plus Short CITS Analysis
Time series
Baseline: 4 6 years before 1998 launch
Follow-up: 6 8 years after 1998 launch
Outcome measures
Quarterly earnings and employment rates
Quarterly welfare receipt and receipt rates
Analytic perspectives
People (1998 cohort members)
Place (local public housing developments)
Substantive focus
Implementation
Impacts
Findings reported
Overall, by site implementation level, and by site
For sample subgroups10
Pooled Average Quarterly Earnings for the 1998 Cohort
At the Three Stronger Implementation Sites
Rollout period
0
500
1,000
1,500
2,000
2,500Q
1 1
99
2
Q1
19
93
Q1
19
94
Q1
19
95
Q1
19
96
Q1
19
97
Q1
19
98
Q1
19
99
Q1
20
00
Q1
20
01
Q1
20
02
Q1
20
03
Time Period
Mea
n Q
ua
rter
ly E
arn
ing
s R
ecei
pt
(in
20
03
do
lla
rs)
Jobs-Plus Group
Comparison GroupDifference due
to Jobs-Plus =
+$ 1,141/year
or + 14%
Pooled Difference in Average Quarterly Earnings for the
1998 Cohort at the Three Stronger Implementation Sites
-600
-400
-200
0
200
400
600
Q 1
19
92
Q 1
19
93
Q 1
19
94
Q 1
19
95
Q 1
19
96
Q 1
19
97
Q 1
19
98
Q 1
19
99
Q 1
20
00
Q 1
20
01
Q 1
20
02
Q 1
20
03
Time Period
Dif
feren
ce in
Mean
Qu
arte
rly
Earn
ing
s R
eceip
t (in
20
03
do
llars)
Jobsplus Program vs Comparison Developments
Jobs-Plus Quarterly Impact Estimation Model
Model of Quarterly Mean T/C Earnings Differences
= + + and
= 1+
where
= the difference in mean earnings for the treatment and
comparison groups in quarter t,
= one if quarter t is follow-up quarter m and zero otherwise,
and 1= error terms with a first-order autoregressive structure,
= a random error term that is independently and identically
distributed.
13
Additional Impact Estimation Steps
Estimate annual impacts: by summing quarterly impact estimates
Estimate standard errors of annual impact estimates: based on estimated standard errors and covariances of the quarterly impact
estimates.
Adjust p-values for the multiplicity of annual impact
estimates: using a layered Bonferroni approach
14
Jobs-Plus Earnings Impacts for the 1998 Cohort from the
Three Stronger Implementation Sites
15
Estimated
Percentage
Observed Estimated Estimated Change
Outcome with Effect of Outcome Without in Outcome Due to
Follow-Up Period Jobs-Plus Jobs-Plus Jobs-Plus Jobs-Plus
1998 6,089 173 5,916 2.9
1999 7,619 209 7,410 2.8
2000 8,793 714 ** 8,079 8.8
2001 9,256 1,135 *** 8,121 14.0
2002 9,419 1,171 *** 8,248 14.2
2003 9,443 1,543 *** 7,900 19.5
2000-2003 9,228 1,141 *** 8,087 14.1
Selected References
Original Sources
Campbell, D.T. and J.C. Stanley (1963) Experimental and Quasi-experimental Designs
for Research. Chicago: Rand McNally, 37 43.
Cook, T.D. and D.T. Campbell (1979) Quasi-Experimentation: Design and Analysis
Issues for Field Settings, Chicago: Rand McNally, 207 232.
Recent Sources:
St. Clair, T., K. Hallberg and T.D. Cook (2016) The Validity and Precision of the
Comparative Interrupted Time-Series Design, Journal of Educational and Behavioral
Statistics, 41(3): 269 299.
Wong, M., T.D. Cook and P.M. Steiner (2015) Adding Design Elements to Improve
Time-Series Designs: No Child Left Behind as an Example of Causal Pattern
Matching, Journal of Research on Educational Effectiveness, 8(2): 245 279.
16
Selected References
(continued)
Recent Sources (continued)
St. Clair, T., T.D. Cook and K. Hallberg (2014) Examining the Internal Validity and
Statistical Precision of the Comparative Interrupted Time Series Design by Comparison
with a Randomized Experiment, American Journal of Evaluation: 1098214014527337.
Dee, T.S. and B. Jacob (2011) The Impact of No Child Left Behind on Student
Achievement. Journal of Policy Analysis and Management. 30 (3): 418 486.
Personal Sources
Bloom, H.S., J. Riccio and N. Verma (2005) The Effectiveness of Jobs-Plus. New York:
MDRC.
Bloom, H.S. (1984) Estimating the Effect of Job-training Programs Using Longitudinal
Data: Ashenfelters Findings Reconsidered, Journal of Human Resources (Fall):
544 556.
Bloom, H.S. and H.F. Ladd (1982) Property Tax Revaluation and Tax Levy Growth,
Journal of Urban Economics, Vol. 11.
17
Selected References
(continued)
Personal Sources (continued)
Jacob, R., M.A. Somers, P. Zhu and H. Bloom (2016) The Validity of the
Comparative Interrupted Time Series Design for Evaluating the Effect of
School-Level Interventions, Evaluation Review. DOI: 10.1177/0193841X16663414.
Jobs-Plus Long-Term Follow-up Source
Riccio, J.A. (2010) Sustained Earnings Gains for Residents in a Public Housing
Jobs Program: Seven-Year Findings From the Jobs-Plus Demonstration. New
York: MDRC (January).
18
Top Related