Evaluating impact of Humanitarian Action: A science and an art?

Jo (Jyotsna) PuriHead of Evaluation

Deputy Executive Director, 3iewww.3ieimpact.org

WHAT IS AN IMPACT EVALUATION?

Group Exercise I

Lets write down a definition.

What is impact evaluation?

Impact evaluations answer the question about the extent the intervention being evaluated altered the state of the world

= the (outcome) indicator with the intervention compared to what it would have been in the absence of the intervention

= Yt(1) – Yt(0)

We can see this

But we can’t see this

So we use a comparison group

THE FIRST STEP: THE THEORY OF CHANGE

Starting with a theory of Change

Cash transfers designed

Increased purchasing

power

Improved livelihood indicators

People record high

levels of satisfaction

Households are targeted Money is

sent

No leakage

Access to markets

Behavioral attributes

ensure correct spending

Households correctly id-ed

Group exercise question II

• What is the theory of change/causal chain for this project that you are interested in?

• Write one outcome that is important?• What were the assumptions and risks in

various stages?

6

The counterfactual

Outcome monitoringBefore After

InterventionControl

www.3ieimpact.org

924084

Before vs. after (single difference) = 92-40 = 52 (outcome monitoring)Post-treatment comparison = 92-84 = 8

The counterfactual

Outcome monitoringBefore After

InterventionControl

www.3ieimpact.org

92408426

Before vs. after (single difference) = 92-40 = 52 (outcome monitoring)Post-treatment comparison = 92-84 = 8Double difference = (92-40)-(84-26) = -6

Group Exercise III

• Write a matrix for the outcome you are interested in examining.

• Write (hypothetical) numbers in the matrix.• Calculate the following:– Single difference– Single ex post difference– Double difference

Overall Aim: Improve lives• Evidence on what works

and why, how;• Improve awareness and

accountability on impact and process;

• Effective allocation of funds;

• Increase likelihood that humanitarian interventions are effective and efficient;

The counterfactual

Outcome

Time

Factual

Counterfactual

The essence of large n designBefore After

Project

Comparison

Large n• n is the number of units of assignment, e.g.

schools, villages, sub-districts (the unit of assignment can be different from the unit of analysis)

• If n is large then we create treatment (project) and comparison groups which are identical prior to the intervention…– And use statistical analysis to assess post-

intervention differences between treatment and comparison: we say these differences are caused by the intervention

So in factBefore After

Project

Comparison

Eligible

Step IEligible units

Step IIEvaluation sample

Step IIIRandom Assignment

Ineligible

Control

Treatment

Internal ValidityExternal Validity

The why of impact evaluation

Did the program cause the change?

Impact evaluation and policies

Would it have happened anyway?

If the program caused the effect, how much was the effect?

Are there other ways, that are cheaper to get the same impact?

Impact Evaluations for development assistance

Efficacy: Does it work in laboratory conditions?

Theory of changeCounterfactualMixed methodsOutcome variablesInternal validity: power, sample size, spill overs, john henry effectsExternal validity: Heterogeneity, representativeness, context

Working definition: Humanitarian action

Response to an emergency, to protect human life, health and subsistence.

The emergency can be the consequence of a natural disaster or a conflict.

-Slow onset disasters- Short term or longer term

Did the program increase the resilience of populations?

Impact evaluation and policies

Was it timely? How/ was adequate coverage ensured?

Did the affected population recover?

Are there other ways, that are cheaper to get the same impact? (cost-effectiveness)

Impact Evaluations for humanitarian assistance

Who lost most? Who recovered best? Unintended consequences.

Real-time Evaluations

Impact Evaluations

• Includes real time evaluations• Measures net change in welfare

levels; Measurement biases.• Expensive but low cost

evaluations too.• Robust evidence (relief,

recovery, resilience) unintended• Vulnerable populations• Long term policy

• Evaluates processes• Focuses on development

and implementation of the program

• Examines targets were met

• Is cheap (?)• Controversial

vs

Humanitarian Crisis: Heterogeneity of impacts

NATURAL DISASTER ARMED CONFLICT

POVERTY SOCIAL INEQUALITYPOOR GOVERENANCE STATE FRAGILITYFOOD INSECURITY

WELL BEING OUTCOMES

CLEAR AND PRESENT NEED FOR IMPACT EVALUATIONS IN HUMANITARIAN ASSISTANCE

Some facts

• In 2011, 62 million people were affected by crises across the world

• Natural disasters, alone, killed almost 26,000 people.

Humanitarian Window – a need.• “Understanding the impact

of humanitarian assistance is another area where much work is needed….Linking impact measurement and accountability better to the funds agencies receive is a key recommendation of this review.”

Need for impact evaluations

• There is a big gap between the requirement and availability of funds. – In 2011, shortage of funds

amounted to $3.4 billion

Critical that we know the efficiency and effectiveness of interventions

METHODOLOGY

Humanitarian vs. Development IEsHumanitarian interventions are more complex to evaluate than development interventions

Development EvaluationsSelection bias

Fragile states and vulnerable populations

Multiple concurrent interventions

Inadequate data

Humanitarian Evaluations

All development evaluations; plus

Rapid onset

High covariance

Disrupted communities

Difference in resources and need

Absence of baseline data

Difficulty in counterfactual selection

Design

TREATED COMPARISON UNAFFECTED

t-1 Baseline (pre-disaster)

T-1 C-1 U-1

t0 Emergency (immediate post-disaster)

T0 C0 U0

t1 Relief/Reconstruction (post-intervention #1)

T1 C1 U1

t2 Recovery (post-intervention #2)

T2 C2 U2

AID

Baseline Emergency Relief/Reconstruction Recovery

Single difference

Baselinet-1

Emergencyt0

Relieft1

Recoveryt2

Disaster related losses

Restoration to baseline

Recovery from disaster

Sustained restoration to baseline

Persistence of recovery

Sustained Recovery

Proportionate changes

Proportionate disaster loss

𝑡−1−𝑡 0𝑡− 1

𝑑𝑖𝑠𝑎𝑠𝑡𝑒𝑟 𝑙𝑜𝑠𝑠𝑒𝑠𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒

Also called “vulnerability of exposure to uninsured risk”

Proportionate changesWhy is it important? Heterogeneity!!

3−03

Case 1Baseline= 3 buffalos

Case 2Baseline = 6 buffalos

𝑇− 1−𝑇 0

𝑇−1

6−36

100% loss of large livestock 50% loss of large livestock

Pakistan Earthquake 2005: Background

• Struck on 8th October, 2005

• 7.6 on the Richter scale• Immediate toll – • 73,000 deaths• 128,000 injured• 600,000 houses

destroyed• Estimated damages were

US $5.8 billion

Pakistan Earthquake 2005: ERRA evaluation

• The ERRA was set up: primary responsibility for allocating reconstruction funds;

• ERRA undertook a “social impact assessment”

• Conducted a pre-/post assessment (no counterfactual)

• PROBLEMS?• Selection Bias• Information Bias• Contamination Bias

Pakistan Earthquake 2005:World Bank Evaluation

• They compare recovery in villages that were more vs. less affected (use as counterfactual)

• The evaluation focused on • Recovery for

households and educational facilities

• Access/quality of schooling

• Effects of grants• Has limitations, should

compliment ERRA study

1. Identify the long term household-level outcomes of interest• Clarify what questions

an IE is designed to answer

• Create a focused list of outcomes to guide the evaluation. For example (next slide)

Immediately after rescue efforts -

Suggested steps – Looking back

Impact Evaluation: Outcome indicators for hypothetical evaluation designEducation:· Net and gross enrollment rates at primary, middle and matric levelsHealth:· Infant/child immunization coverage· Diarrheal prevalence, last 30 days, children under 5· Provider consultation and treatment rates for recent illness/injury· % of women with recent birth receiving tetanus toxoid injection· Skilled attendance and location of childbirthHousing, water supply and sanitation:· Roof and wall materials· Number of rooms· Source of drinking water· Type of toiletHousehold perception of economic situation and satisfaction with facilities and service use:· Perception of economic situation of household compared to one year ago· Perception of economic situation of community compared to one year ago· Satisfaction with local services basic health unit, family planning services, school, veterinary hospital, agricultural extension, and police

2. Obtain a pre-earthquake area-representative household sample • In Pak, a good measure would have been the 2004-

05 PSLM • Importance of baseline data

Immediately after rescue efforts - Suggested steps – Looking back

Suggested steps – Looking backImmediately after rescue efforts -

3. Collect data on the pre-earthquake sample immediately post-earthquake• Use the PSLM and re-

interview households• Expand sample if

necessary• Post-disaster surveying

includes unaffected households as well.

Immediately after rescue efforts -

4. Design interventions for staged roll out or other variations• Provides a

counterfactual• Allows comparing two

or more interventions, and understanding best practices

• Ethical

Suggested steps – Looking back

Some variations

• Theory of change (without contamination bias)• Basic care package– Factorial design = A vs. A+B vs. A+C

• Cluster randomized designs can help determine the effectiveness of packages

• Examples– Rwanda – messaging– Sierra Leone – Paired matching– Aceh – Documentation documentation!– Burundi – Phased roll out amongst ex-combatants.

IEs• Provide insight regarding the losses resulting

from an emergency and compare them with a baseline;

• Test innovative programmes in real-life situations;

• See what difference assistance has made; • Examine whether recovery is sustained;• Examine the cost-effectiveness of

interventions.

Conclusions

WWW.3IEIMPACT.ORG

Thank you.@jo_puri

jpuri@3ieimpact.org