TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia,...

TRANSLATING RESEARCH INTO ACTION

What is Randomized Evaluation?Why Randomize?

J-PAL South Asia, April 29, 2011

Life Cycle of an Evaluation

< Needs assessment > <~~~~~~~~~~~ Process Evaluation ~~~~~~~~~~~~> <~~~~~~~~~ Impact Evaluation ~~~~~~~~>Other types of Evaluations: Review / Cost-benefit analysis / Cost-effectiveness Analysis

Balsakhi Program

Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program

Incorporating random assignment into the program



• Class sizes are large• Many children in 3rd and 4th standard are not

even at the 1st standard level of competency• Social distance between teacher and some of

the students is large

What was the issue?

• Hire local women (balsakhis) from the community

• Train them to teach remedial competencies– Basic literacy, numeracy

• Identify lowest performing 3rd and 4th standard students– Take these students out of class (2 hours/day)– Balsakhi teaches them basic competencies

Proposed solution





Pros• Reduced class size• Teaching at appropriate

level• Reduced social distance

• Improved learning for lower-performing students

• Improved learning for higher-performers

Cons• Less qualified• Teacher resentment• Reduced interaction with

higher-performing peers

Possible outcomes

• We want to look at impact• What do we do?

Study design

What is the impact of the balsakhi?

What would have happened without the balsakhi program?

• Look at average change in test scores over the school year for the balsakhi children

Pre-post (Before vs. After)

What was the impact of the balsakhi program?

Before vs. AfterImpact = 26.42 points?

75

50

25

0

02002 2003

26.42 points?

Impact: What is it?

Time

Prim

ary

Out

com

e

Impact

Counterfactual

Intervention

Impact: What is it?

Time

Prim

ary

Out

com

e

ImpactCounterfactual

Intervention

Impact is defined as a comparison between:

1. the outcome some time after the program has been introduced

2. the outcome at that same point in time had the program not been introduced

the ”counterfactual”

14

How to measure impact?

1. Randomized Experiments • Also known as:

– Random Assignment Studies – Randomized Field Trials – Social Experiments– Randomized Controlled Trials (RCTs)– Randomized Controlled Experiments

15

Impact evaluation methods

2. Non- or Quasi-Experimental Methods a. Pre-Postb. Simple Differencec. Differences-in-Differencesd. Multivariate Regressione. Statistical Matchingf. Interrupted Time Seriesg. Instrumental Variablesh. Regression Discontinuity

16

Impact evaluation methods

• There are more sophisticated non-experimental methods to estimate program impacts:– Multivariable Regression– Matching– Instrumental Variables– Regression Discontinuity

• These methods rely on being able to “mimic” the counterfactual under certain assumptions

• Problem: Assumptions are not testable

Other Methods

• We constructed results from the different ways of estimating the impacts using the data from the schools that got a balsakhi

1. Pre – Post (Before vs. After)2. Simple difference3. Difference-in-difference4. Other non-experimental methods5. Randomized Experiment

Methods to estimate impacts

TRANSLATING RESEARCH INTO ACTION

II – What is a randomized experiment?

The Balsakhi Example

povertyactionlab.org

The basics

Start with simple case: • Take a sample of schools• Randomly assign them to either:

Treatment Group – is offered treatment Control Group - not allowed to receive treatment

(during the evaluation period)

20

• Suppose we evaluated the balsakhi program using a randomized experiment

• QUESTION #1: What would this entail? How would we do it?

• QUESTION #2: What would be the advantage of using this method to evaluate the impact of the balsakhi program?

21

6 – Randomized Experiment

Source: www.theoryofchange.org

How to Randomize, Part I - 22

Random assignment in Vadodara

2006

Baseline Test Scores

20

10

0Treat Compare

28 28

Key advantage of experiments

Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,

any difference that subsequently arises between them can be attributed to the program rather than to other factors.

23

• Pratham had received funding to introduce balsakhis in all schools

• Principals wanted balsakhis in their school• Teachers wanted balsakhis in their class• If denied a balsakhi, they may block data

collection efforts

Constraints of design

Random assignment

Round 1Red: Std. 3Blue: Std. 4


Rotation design

Round 2Std 3 from Round 1 Std 4 in Round 2——————————————————————————

Std 4 from Round 1 Std 3 in Round 2


ComparisonRed: TreatmentBlue: ControlStd 3Std 4

Rotation design

The counterfactual:

School Groups

Year 1 (2001-2002) Year 2 (2002-2003)

Grade 3 Grade 4 Grade 3 Grade 4

Group A Balsakhi No Balsakhi No Balsakhi Balsakhi

Group B No Balsakhi Balsakhi Balsakhi No Balsakhi

Key steps in conducting an experiment

1. Design the study carefully

2. Collect baseline data

3. Randomly assign people to treatment or control

4. Verify that assignment looks random

5. Monitor process so that integrity of experiment is not compromised

29

Key steps in conducting an experiment (cont.)

6. Collect follow-up data for both the treatment and control groups

7. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group.

8. Assess whether program impacts are statistically significant and practically significant.

30

Impact of Balsakhi - Summary

Method Impact Estimate

(1) Pre-post 26.42*

(2) Simple Difference -5.05*

(3) Difference-in-Difference 6.82*

(4) Regression 1.92

*: Statistically significant at the 5% level



(1) Pre-post 26.42*



(4) Regression 1.92

(5)Randomized Experiment 5.87*


Bottom Line: Which method we use matters!



(1) Pre-post 26.42*



(4) Regression 1.92

(5)Randomized Experiment 5.87*


• This program improved average test score by nearly 10%

• The lowest-performing kids in balsakhi schools improved the most (relative to the control group)

• Highly cost effective: intervention cost less than Rs.100 per child and results far outweigh the cost

• Can be scalable, especially in light of RTE Act: remedial education

Results

TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia,...

Documents

Transcript of TRANSLATING RESEARCH INTO ACTION What is Randomized Evaluation? Why Randomize? J-PAL South Asia,...