AIM-CDD

Quasi-Experimental Methods

Florence Kondylis (World Bank)

AIM-CDD

Objective

• Find a plausible counterfactual»Reality check

• Every method is associated with an assumption

• The stronger the assumption the more we need to worry about the causal effect

»Question your assumptions

2

AIM-CDD

3

Program to evaluateFertilizer vouchers Program (2007-08)–Main Objective• Increase maize production

– Intervention: vouchers distribution–Target group:• Maize producers• Farmers owning >1 Ha, <3 Ha land

– Indicator: Yield (Maize)

AIM-CDD

I. Before-after identification strategy

Counterfactual:

Yield before program started

» EFFECT = After minus Before

Counterfactual assumption:

There is no other factor than the vouchers affecting yield from 2007 to 2008

years

4

AIM-CDD

5

AIM-CDD

6

Questioning the counterfactual assumption

Question: what else might have happened in 2007-2008 to affect maize yield ?

AIM-CDD

7

Examine assumption with prior data

Assumption of no change over time not so great ! >> There are external

factors (rainfall, pests…)

AIM-CDD

II. Non-participant identification strategy

Counterfactual:

Rate of pregnancy among non-participants

Counterfactual assumption:

Without vouchers, participants would as

productive as non-participants in a given year

8

AIM-CDD

9

AIM-CDD

Questioning the counterfactual assumption

10

Question: how might participants differ from non-participants?

AIM-CDD

Test assumption with pre-program data

11

REJECT counterfactual hypothesis of same productivity

AIM-CDD

III. Difference-in-Difference identification strategy

Counterfactual:

1.Non-participant maize yield, purging pre-program differences between participants/nonparticipants

2.“Before vouchers” maize yield, purging before-after change for nonparticipants (external factors)

• 1 and 2 are equivalent

12

AIM-CDD

13

AIM-CDD

14

57.50 - 46.37 = 11.13

66.37 – 62.90 = 3.47

Non-participants

Participants

Effect = 3.47 – 11.13 = - 7.66

AIM-CDD

15

After

Before

Effect = 8.87 – 16.53 = - 7.66

66.37 – 57.50 = 8.87

62.90 – 46.37 = 16.53

AIM-CDD

16

Counterfactual assumption:

Without intervention participants and nonparticipants’ pregnancy rates follow same trends

AIM-CDD

17

74.0

16.5

AIM-CDD

18

74.0 -7.6

AIM-CDD

19

Questioning the assumption

• Why might participants’ trends differ from that of nonparticipants?

AIM-CDD

20

Examine assumption with pre-program data

counterfactual hypothesis of same trends doesn’t look so believable

Average rate of teen pregnancy in

2004 2008 Difference (2004-2008)

Participants (P) 54.96 62.90 7.94

Non-participants (NP) 39.96 46.37 6.41

Difference (P=NP) 15.00 16.53 +1.53 ?

AIM-CDD

21

IV. Matching with Difference-in-Difference identification strategy

Counterfactual:

Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants

AIM-CDD

22

Counterfactual assumption:

Question: how might participants differ from matched nonparticipants?

Unobserved characteristics do not affect outcomes of interest

Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset

AIM-CDD

23

73.36

66.37Matched

nonparticipant

Participant

Effect = - 7.01

AIM-CDD

24

Can only test assumptionwith experimental data

Apply with care – think very hard about unobservables

Studies that compare both methods (because they have experimental data) find that:

unobservables often matter!

direction of bias is unpredictable!

AIM-CDD

Summary

• Randomization requires minimal assumptions needed and procures intuitive estimates (sample means !)

• Non-experimental requires assumptions that must be carefully assessed

»More data-intensive

25

AIM-CDD

Example: Irrigation for rice producers + Enhanced Market Access• Impact of interest measured by:

– Input use & repayment of irrigation fee– Rice yield– (Cash) income from rice– Non-rice cash income (spillovers to other value chains)

• Data: 500 farmers in project area / 500 random sample farmers– Before & after treatment

»Can’t randomize irrigation so what is the counterfactual?

AIM-CDD

Plausible counterfactuals• Random sample difference in difference

– Are farmers outside the scheme on the same trajectory ?

• Farmers in the vicinity of the scheme but not included in scheme– Selection of project area needs to be carefully documented

(elevation…)

– Proximity implies “just-outside farmers” might also benefit from enhanced market linkages

» What do we want to measure?

• Propensity score matching

– Unobservables determining on-farm productivity ?

27

AIM-CDD

28

Thank You