How to randomize? povertyactionlab.org. Quick Review: Why Randomize Choosing the Sample Frame...

How to randomize?

povertyactionlab.org

• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization

• Options• How to Choose

• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control

Lecture Overview

• Ensure that we compare oranges and oranges• Randomization: randomly select treatment and

comparison group before program starts• If we construct a comparison group after the

program has started Bias– Before and After

• Other things may be going on

– Participants and Non participants (Simple comparison; Diff-in-Diff; Regression)

• Selection Bias

Why randomize?

Selection Bias

Difference of Means

Intrinsic Differences Impact of the program

Observables Non Observables

Selection Bias

Baseline

Randomization

Assignment to Treatment Assignment to Control

Sample frame

Compare T and C

How to Randomize, Part I - 6

Random assignment

2006

Income per person, per month, rupees

1000

500

0Treat Compare

1457 1442

Baseline

Randomization

Assignment to Treatment Assignment to Control

Endline

Compare T and C

Sample frame

Intervention

Compare T and C




Lecture Overview

• Geographic area and Population• Where will the project occur?

– Example: Purnea, Ajmer Districts• Who is the target population? What types of

tolas/villages will we be working in? • Choice of area and population is usually not

random– Convenience, eligible population

• How representative do we want to be?

Area and Population




Lecture Overview

1. Randomizing at the individual level2. Randomizing at the group level

“Cluster Randomized Trial”

Unit of Randomization: Options

11

Unit of Randomization: Individual?

“Groups of individuals”: Cluster Randomized Trial

Unit of Randomization: Clusters?

Unit of Randomization: Class?

Unit of Randomization: School?




Lecture Overview

• Spillovers and Crossovers– How wide is the potential impact?

• How is the intervention administered? – What is the mobilization method? – What is the catchment area of each

“unit of intervention”? • Others

– Logistical constraints– Budget and Sample size requirements

How to Choose the Level

• Remember the counterfactual!• If control group is different from the

counterfactual, results will be biased• Can occur due to

• Spillovers• Crossovers

“Contamination” of the Control

A Deworming Program

Treatment Control

Spillovers

Counterfactual

• Positive spillovers (treatment -> control)– Medicines or vaccines– District gives more resources to control schools

• Negative spillovers (control -> treatment)– Re-infection

• Negative spillovers (treatment -> control)– Less diseases in the area so parents put les efforts in

treating kids • Mother Literacy?• How do we ensure this does not happen?

Spillovers (Indirect “treatment”)

• Control individuals get directly treated – E.g Balsakhis: Control kids get treated (e.g.

transfer to treatment schools)– E.g: Microfinance evaluation in urban areas

• Mother Literacy? • How do we ensure this does not happen?

Crossovers (Direct treatment of control)

• What is the mobilization method?• What is the catchment area of each

“unit of intervention”? • Usually best to keep the randomization unit as

the unit of intervention• Given this, also useful to keep the unit of

randomization as small as possible

Unit and method of Intervention

• Logistics– Large villages as unit of randomization: more

travel, difficult to find volunteers etc.• Budget and sample size

– Individual randomization is cheaper (tomorrow!)– There may not be enough clusters of a certain

type available for the intervention• Villages vs tolas

Logistics, Budget and Sample Size




Lecture Overview

• Most programs have limited resources– Vouchers, Farmer Training Programs– Results in more eligible recipients than resources

will allow services for• Programs are often piloted in a smaller scale

before being scaled up • Lottery can be a fair way to select participants

Lottery

• What if you have just enough resources for the number of recipients in your work area?– 500 seats for a training, and 500 applicants

• Increase outreach of activities• May not be possible• Control group may be upset anyway• Phase-in Design

Lottery

• Everyone gets program eventually• Natural approach when expanding program

faces resource constraints• What determines which schools, branches,

etc. will be covered in which year?– Some choices based on need, geography, etc.– Others largely arbitrary

Phase-in: takes advantage of expansion

Phase-in design

Round 1Treatment: 1/3Control: 2/3


Round 3Treatment: 3/3Control: 0 1

1

11

1

1

1

1

1

11

1

1

1

2

2

22

2

2

22

2

2

2

22

2

2

2

3

333

3

33

33

3

33

3

3

3 3

3



Randomized evaluation endsRandomized evaluation ends

Advantages• Everyone gets something eventually

Concerns• Can complicate estimating long-run effects• Natural expansion timeline may be too short • Do expectations of change changes behavior

today?

Phase-in designs

• Groups get treatment in turns• Advantages• Concerns

Rotation design



Rotation design

Round 2Treatment from Round 1 Control——————————————————————————

Control from Round 1 Treatment


• Phase-in may not provide enough benefit to late round participants

• Cooperation from control group may be critical

• Consider within-group randomization• E.g., balsakhi program• All participants get some benefit• Concern: increased likelihood of contamination

“Want to survey me? Then treat me”

• Sometimes it’s practically or ethically impossible to randomize program access

• But most programs have less than 100% take-up

• Randomize encouragement to receive treatment

• Encouragement: Something that makes some folks more likely to use program than others

Encouragement design: What to do when you can’t randomize access

Encouragement design

Encourage

Do not encourage

participated

did not participate

Complying

Not complying

compare encouraged to not encouraged

do not compare participants to non-participants

adjust for non-compliance in analysis phase

These must be correlated

Summary

Method Most useful when Advantages Disadvantages

Lottery •Limited resources•Piloting a new program

Easy to understand and implement

Control group may be upset

Phase in •Program expanding over time•Everyone must get treatment

•Easy to understand and explain •Control group not upset

•Anticipation of treatment •Long term effects

Rotation Everyone must get something

No control group upset •Spillovers more likely •More difficult to measure long term

Encouragement •Program has to be done everywhere•Take up low but can easily be increased

Can easily randomize at individual level

•Need very effective incentive•Impact on those who respond to incentive




Lecture Overview

• Need sample frame• Pull out of a

hat/bucket

Mechanics of randomization

40

Source: Chris Blattman

• Need sample frame• Pull out of a

hat/bucket• Use random number

generator in spreadsheet program to order observations randomly

• Stata program code


41


• Need sample frame• Pull out of a hat/bucket• Use random number

generator in spreadsheet program to order observations randomly

• Stata program code• What if no existing list?


43





Lecture Overview

• Objective: balancing your sample

• Ultimately increases your power

• What is it: – dividing the sample into different subgroups– selecting treatment and control from each subgroup

• What happens if you don’t stratify?

Stratification

45

Stratification

Village Type Status

uyufssg Low lit T

Adjkjdk Low lit C

afiudiufa High Lit T

aduyu Low lit C

afjh High lit T

askfjkfjk Low lit C

fdkvofj High lit T

daisudimv Low lit C

vadlsvkoi Low lit T

Vaofdivofdai Low lit C

caidsudiau High Lit T

Stratification

Village Type Status

uyufssg Low lit T

Adjkjdk Low lit C

afiudiufa Low lit T

aduyu Low lit C

afjh Low lit T

askfjkfjk Low lit C

fdkvofj Low lit T

daisudimv High Lit T

vadlsvkoi High Lit C

Vaofdivofdai High Lit T

caidsudiau High Lit C

• Stratify on variables that could have important impact on outcome variable (bit of a guess)

• Logistics• Stratify on subgroups that you are particularly

interested in (where may think impact of program may be different)

• Stratification more important when small data set• Can get complex to stratify on too many variables• Makes the draw less transparent the more you

stratify

When to stratify

48




Lecture Overview

• Sometimes core question is deciding among different possible interventions

• You can randomize these programs• Test different components of treatment in

different combinations• Test whether components serve as substitutes

or compliments• What is most cost-effective combination?

Multiple treatments


Treatment 1

Treatment 2

Treatment 3

Multiple treatments

Mother Literacy Research Design

Randomization

Intervention 1:

Mothers Literacy Classes

60 villages

Intervention 2:

Train Mothers to Interact with

Children’s Education

60 Villages

Intervention 3:

Combine Intervention 1 and

2

60 Villages

Comparison group

60 Villages

240 Elligible Villages

In Each State:

• Some schools are assigned full treatment– All kids get pills

• Some schools are assigned partial treatment– 50% are designated to get pills

• Testing subsidies and prices• E.g: Information campaign - how much do you

need to spread information?

Measuring Spillovers: Varying levels of treatment

• Combined

Measuring Spillovers: Double randomization

155 clusters

Treatment schools Control schools

Random Assignment

Treatment individualsControl individuals

Control individuals

How to randomize? povertyactionlab.org. Quick Review: Why Randomize Choosing the Sample Frame...

Documents

Transcript of How to randomize? povertyactionlab.org. Quick Review: Why Randomize Choosing the Sample Frame...