Interrupted Time Series and Regression Discontinuity 2019-12-07¢  Interrupted time series...

download Interrupted Time Series and Regression Discontinuity 2019-12-07¢  Interrupted time series Interrupted

of 90

  • date post

    13-Apr-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Interrupted Time Series and Regression Discontinuity 2019-12-07¢  Interrupted time series...

  • Interrupted Time Series and Regression Discontinuity Design

    Marcelo Coca Perraillon

    University of Colorado Anschutz Medical Campus

    March 2019

    1 / 90

  • Outline

    Interrupted time series

    Basic design

    Estimation

    Validity issues

    Extensions

    Overview of regression discontinuity

    Meaning and validity of RDD

    Several examples from the literature

    Estimation (where most decisions are made)

    Sharp RRD example: Nursing home ratings

    Fuzzy RDD example: Early intervention therapies

    2 / 90

  • Code/data

    You can get code and example dataset from my website (click on Code on left menu):

    http://tinyurl.com/mcperraillon

    3 / 90

    http://tinyurl.com/mcperraillon

  • ITS

    Interrupted time series

    Interrupted time series is a fairly basic before and after analysis

    You have probably learned, for good reasons, that before and after analyses are suspicious

    Yet, in some circumstances, and under some assumptions, they can be fairly convincing designs

    Today, I’ll separate design and estimation which in general is the way to go

    See Rubin (2007,2008). Bottom line: “[O]bservational studies can and should be designed to approximate randomized experiments as closely as possible. In particular, observational studies should be designed using only background information to create subgroups of similar treated and control units, where similar here refers to their distributions of background variables.”

    4 / 90

  • ITS

    Interrupted time series

    Time series: observations for a single variable made consecutively over time

    Could be same units or different units

    Example: prescription numbers/rates over time for different people in each time period or the same group of people measured at different points over time

    At some point during the observation period there is change that creates an “interruption” in the time series

    For example, a policy change like Medicare part D implementation in 2006 or the release of a black label warning for antidepressants use in children would create an “interruption” or a change in payment policy in Medicaid

    The idea is to use that interruption to measure the effects of policies

    5 / 90

  • ITS

    Example Medicaid in New Hampshire imposed a three-drug limit that restricted medication reimbursement among among chronically ill poor patients with cardiac and other chronic illnesses The outcome: the mean number of prescriptions declined by half Soumerai et al (2017, 1987).

    6 / 90

  • ITS

    Example Medicare Part D changes in prescriptions for Medicare beneficiaries Schneeweiss et al (2009) Use pharmacy claims for the elderly–problems with measurement and establishing insurance coverage

    7 / 90

  • ITS

    Thinking about effects

    Note that in the two examples effects can be described in different ways, which has implications when thinking about the estimation part

    Is a change in the level? Is it a change in the rate? In a model, intercept vs slope changes

    Is the change continuous or does it decay? In the Medicaid example, there was another change prompted by the concerns about prescription declines

    Is the effect immediate or delayed?

    In health policy, changes often happen before the actual policy implementation. Example, Medicaid expansion or Part D

    8 / 90

  • ITS

    Estimation

    Estimation is (relatively) simple with models parametrized in different ways depending on whether one assumes a change in level or a change in slope or if other features are also incorporated

    For example: y = β0 + β1t + β2post + β3t ∗ post + �, where post = 1 if after the policy intervention and t is time

    In the pre-period: E [ypre |x] = β0 + β1t In the post-period: E [ypost |x] = (β0 + β2) + (β1 + β3)t Note that several tests are possible. If β2 = β3 = 0 then there is no policy effect

    If β2 = 0 but β3 6= 0 then there was a change in the slope, but not the level after the interruption. β3 is the change in the rate (remember, interactions in the linear model are differences of differences). The slope after the policy is given by β1 + β3

    9 / 90

  • ITS

    Estimation

    For the Medicaid example above, a Wald test for β2 would be more relevant since there doesn’t seem to be much happening in the slope

    A model like y = β0 + β2post + � could be a better fit

    If you think that a change in slope and not level is a better assumption, splines are an option.

    With splines, we can model a change in the slope with no change in the level

    10 / 90

  • ITS

    Splines

    The policy change (“interruption”) happens at time t = k . In splines lingo k is the knot

    The model is y = β0 + β1time + β2(time − k)+� The (z)+ is called a truncated line function and is defined as being equal to z if z is positive and zero otherwise

    So (time − k)+ will be equal to time (time − k) after the policy change and and zero if at policy knot or before

    The only difficult part about splines is to get the coding right, the rest is (relatively) easy

    See Stata’s mkspline command. You can do all sort of things with splines (they don’t have to be linear). Sometimes they are called piecewise regression

    11 / 90

  • ITS

    Splines

    Model: y = β0 + β1time + β2(time − k)+ + � Before the policy change the model is A) : E [y ] = β0 + β1age

    After: If time > k the model is: E [y ] = β0 + β1time + β2(time − k) Same as centering (more on it soon) so when time > k we can rewrite is B): E [y ] = (β0 − β2 ∗ k) + (β1 + β2)time If β2 = 0, then the slope before and after is the same (so A and B are the same)

    Note that β2 is the incremental change in slope

    The trick of using the truncated function is that it allowed us the possibility of a different slope after k

    See Lopez Bernal (2017) for other parametrizations

    12 / 90

  • ITS

    Statistical issues

    The models I wrote above are linear models implicitly assuming errors are iid and �i ∼ N(0, σ2) or, equivalently, �i ∼ N(β0 + β1X1i + · · ·+ βpXpi , σ2) Of course, two issues: 1) The outcomes drives distributional assumptions. If counts, for example, a negative binomial or Poisson model would better

    2) The errors cannot be iid since they can’t be independent. Autocorrelation is a feature of time series

    Although the linear model is unbiased, the standard errors would be wrong

    Autoregressive integrated moving average (ARIMA) or other solutions are commonly used. There are test for autocorrelation, like the Breusch-Godfrey test

    Another issue (seasonality)

    13 / 90

  • ITS

    Validity

    Intuitively, the main validity issue is whether other changes happening after the policy/interruption could have caused the change in y

    Your textbook also mentions instrumentation (changes in procedures changes measurement) and

    Selection (the composition of the group changes after policy change)

    Compelling interrupted time series designs tend to be short-term, with changes that are hard to explain otherwise

    14 / 90

  • ITS

    Extensions

    Adding a nonequivalent no-treatment control group time series

    As the duck test describing an example of abductive reasoning goes: If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck

    Yes, this is the same as difference-in-difference models... See Wing et al. (2018)

    Note how the parallel trend and the common shocks assumptions of DiD are extensions of the issues discussed in Chapter 6

    Please read Chapter 6. Don’t underestimate your textbook

    15 / 90

  • ITS

    And now for something completely different...

    Switching gears: regression discontinuity design

    16 / 90

  • Introduction

    Basics

    Method developed to estimate treatment effects in non-experimental settings

    Provides causal estimates of treatment effects. Those estimates are Local Average Treatment Effects (LATE)

    Design limits external validity in some cases

    Good internal validity; some assumptions can be empirically verified

    Relatively easy to estimate but it has some complications

    First application: Thistlethwaite and Campbell (1960) (Does the last name Campbell sound familiar?)

    17 / 90

  • Introduction Thistlethwaite and Campbell

    Thistlethwaite and Campbell

    They studied the impact of merit awards on future academic outcomes

    Awards allocated based on test scores

    If a person had a score greater than c , the cutoff point, then she received the award

    The wrong way of analyzing: compare those who received the award to those who didn’t

    Thistlethwaite and Campbell realized they could compare individuals just above and below the cutoff point

    By now I find this idea intuitive but it’s not at first. It helps if you think that choosing the point c is arbitrary or measured with error

    Say, it’s 1200. But why not 1210? Or 1190? We know that the test scores used to give scholarships is related to future academic outcomes but there is no solid relationship between 1200 and the