Using School Choice Lotteries to Test Measures of School Effectiveness

David DemingHarvard University and NBER

Measuring School Effectiveness

• School rankings, ratings, league tables

• Gain score or “value-added” modeling approach (VAMs)– School VAMs now in ~30 U.S. states (Blank 2010)– Teacher VAMs used in evaluation, retention

• Accuracy of VAMs is important for incentive design and student welfare (Baker 2002, Rothstein 2010)

VAM Research• Large literature on measurement / technical issues

• First order issue mostly untested - is assignment of teachers to classes within schools as good as random?

• Kane and Staiger (2008), Kane et al (2013) randomly assign classes to teachers, test validity of VAMs– Chetty, Friedman and Rockoff (2013) use quasi-experimental design with

teacher mobility

• School VAMs require conditionally exogenous sorting of students across schools– Would you consent to this experiment?

Using School Choice Lotteries

• Oversubscribed public schools in Charlotte-Mecklenburg– Random assignment, within a self-selected group of

applicants

• Estimate VAMs using data from prior cohorts– Vary model specification, sample, outcome– Out-of-sample predictions of “school effects”

• Use VAM estimates to predict the treatment effect of winning the lottery

Data and VAMs• Grades 4-8• Covariates - test scores from 1996-97 to 2001-2002, demographics

– Models with nothing in X (levels), lagged scores only, lagged scores + demographics

• Outcomes - 2003 and later test scores

;

• Obtain by computing average residual (random effects)– Results are very similar with fixed effects

• “School-by-grade” effects

𝐴𝑖𝑗𝑡 = 𝑋𝑖𝑗𝑡𝛽+ 𝜔𝑖𝑗𝑡 𝜔𝑖𝑗𝑡 = 𝜇𝑗 + 𝜃𝑗𝑡 + 𝜀𝑖𝑗𝑡

Lottery Data and Sample• 2,599 students in 118 separate lotteries for “marginal” priority groups• Top 3 choices but nearly all randomization was over 1st choices

Table A1: Descriptive Statistics

Non-lottery Lottery(1) (2) (3)

Male 0.506 0.468 -0.007*[0.003]

African-American 0.449 0.619 0.043**[0.004]

Latino 0.070 0.056 0.009[0.006]

Free / Reduced Lunch 0.542 0.638 0.003[0.004]

2002 Math Score 0.074 -0.037 0.001[0.002]

2002 Reading Score 0.001 -0.070 0.007**[0.002]

Grade-by-Neighborhood School Fixed Effects

X

Sample Size 31,455 2,599 35,596

Baseline Means Pr (lottery sample)

Empirical Strategy2SLS with VAM in Fall 2002 school as the endogenous variable:

1 = 1st choice schoolN = neighborhood school = indicator for winning lottery j = lottery fixed effects (unit of randomization)

If = 0.1, implied causal effect of attending school j relative to average school is 0.1.

Thus if is unbiased, = 1. < 1 means VAM is upward biased…

isn’t obvious - what is lottery applicant’s outside option? We try a few different things to check this...

𝑉𝐴𝑀𝑖𝑗𝐴 = 𝜃𝑊𝑖𝑗𝑉𝐴𝑀𝑖𝑗1 + 𝛾൫1− 𝑊𝑖𝑗൯𝑉𝐴𝑀𝑖𝑗𝑁+ 𝜋𝑋𝑖𝑗 + Γ𝑗 + 𝜀𝑖𝑗 𝐴𝑖𝑗 = 𝛿𝑉𝐴𝑀𝑖𝑗𝐴 +𝛽𝑋𝑖𝑗 +Γ𝑗 + 𝜀𝑖𝑗

4 Possible Explanations for Bias

1. Sorting on unobserved determinants of student achievement (Rothstein 2010)

2. Estimation error

3. “True” school effects may vary over time independent of estimation error

4. Lottery sample is self-selected, so treatment effect is different for them

No correlation between average test scores (in levels) and lottery impacts

Table 1: Validating Models of "School Effects" Using Lottery Data

Outcome is Spring 2003 Test Scores(1) (2) (3) (4) (5) (6)

0.025 0.034 0.014 0.531* 0.807** 0.966**[0.077] [0.070] [0.072] [0.208] [0.236] [0.342]

Years of Prior Data 2002 only 2001-2002 1998-2002 2002 only 2001-2002 1998-2002

Covariates in VAM none none noneprior scores

(1 lag)prior scores

(1 lag)prior scores

(1 lag)

Shrinkage Adjustment none none none none none none

p-value on F(VA=1) 0.000 0.000 0.000 0.024 0.413 0.920SD of school effects 0.431 0.417 0.397 0.110 0.096 0.073Observations 2,599 2,599 2,599 2,599 2,599 2,599

School "Value-Added" in 2003

1. Huge improvement from adding lagged scores (gains model)

2. Need 2+ years of data to fail to reject unbiasedness (triple negative!)

Table 1: Validating Models of "School Effects" Using Lottery Data

Outcome is Spring 2003 Test Scores(1) (2) (3) (4) (5) (6)

0.025 0.034 0.014 0.531* 0.807** 0.966**[0.077] [0.070] [0.072] [0.208] [0.236] [0.342]

Years of Prior Data 2002 only 2001-2002 1998-2002 2002 only 2001-2002 1998-2002

Covariates in VAM none none noneprior scores

(1 lag)prior scores

(1 lag)prior scores

(1 lag)

Shrinkage Adjustment none none none none none none

p-value on F(VA=1) 0.000 0.000 0.000 0.024 0.413 0.920SD of school effects 0.431 0.417 0.397 0.110 0.096 0.073Observations 2,599 2,599 2,599 2,599 2,599 2,599

School "Value-Added" in 2003

1. Adding demographics to increases coefficients modestly in all specifications

2. School effects “too small” when all prior years of data are included

Table 1: Validating Models of "School Effects" Using Lottery DataOutcome is Spring 2003 Test Scores

(7) (8) (9) (10) (11) (12)

0.547** 0.908** 1.237** 0.627** 1.602** 1.185**[0.194] [0.231] [0.347] [0.207] [0.450] [0.323]

Years of Prior Data 2002 only 2001-2002 1998-2002 2002 only 1998-2002 1998-2002

Covariates in VAMdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scores

Shrinkage Adjustment none none noneEmpirical

BayesEmpirical

Bayes"Drift"

Adjustment

p-value on F(VA=1) 0.020 0.689 0.495 0.071 0.180 0.567SD of school effects 0.102 0.088 0.067 0.088 0.047 0.054Observations 2,599 2,599 2,599 2,599 2,599 2,599

School "Value-Added" in 2003

“Shrinkage”

• Attenuate teacher/school effects toward zero based on transitory variance in the estimate– Estimate a constant from noisy data

• Empirical Bayes (e.g. Kane et al 2013) weighs all past years of data equally

• Weigh based on autocovariance – teacher effects “drift” over time (Chetty, Friedman and Rockoff 2013)

1. Shrinkage improves prediction when only one year of prior data is used – with longer panel unshrunken more accurate

2. “Drift” adjustment is best here

3. Shrinkage overcompensates if there is true variation in school effects – CFR call this “teacher bias”

Table 1: Validating Models of "School Effects" Using Lottery DataOutcome is Spring 2003 Test Scores

0.547** 0.627** 1.237** 1.602** 1.185**[0.194] [0.207] [0.347] [0.450] [0.323]

Years of Prior Data 2002 only 2002 only 1998-2002 1998-2002 1998-2002

Covariates in VAMdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scoresdemogs +

prior scores

Shrinkage Adjustment noneEmpirical

Bayesnone

Empirical Bayes

"Drift" Adjustment

p-value on F(VA=1) 0.020 0.071 0.495 0.180 0.567SD of school effects 0.102 0.088 0.067 0.047 0.054Observations 2,599 2,599 2,599 2,599 2,599

School "Value-Added" in 2003

Concluding thoughts

• Despite sorting concerns, school VAMs are surprisingly accurate– Best fit was gains only on full panel, but that’s not a theorem– Large standard errors admit non-trivial bias – need to

replicate in other settings

• Good news for policies that use VAMs, but:1. Biases might be offsetting 2. Other outcomes are important too!3. “School effects” may include contextual factors that are

beyond school’s control (peers, neighborhoods)

Thanks!