Howard White International Initiative for Impact Evaluation
description
Transcript of Howard White International Initiative for Impact Evaluation
![Page 1: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/1.jpg)
www.3ieimpact.orgHoward White
Getting what we pay for: impact evaluation for better planning and budgeting
Regional conference on public sector management in support of the MDGs
Bangkok, June 2012
Howard WhiteInternational Initiative for Impact Evaluation
![Page 2: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/2.jpg)
www.3ieimpact.orgHoward White
Why did the Bangladesh Integrated Nutrition Program (BINP) fail?
The case of the Bangladesh Integrated Nutrition Project (BINP)
Impact evaluation: an example
![Page 3: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/3.jpg)
www.3ieimpact.orgHoward White
Comparison of impact estimates
![Page 4: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/4.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
![Page 5: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/5.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
Right target group for nutritional counsellingPARTICIPATION
RATES WERE UP TO 30% LOWER FOR WOMEN LIVING WITH THEIR MOTHER-IN-LAW
![Page 6: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/6.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
Knowledge acquired and used
![Page 7: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/7.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
The right children are enrolled in the programme
![Page 8: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/8.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
Supplementary feeding is supplementary
![Page 9: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/9.jpg)
www.3ieimpact.orgHoward White
Lessons from BINP• Apparent successes can turn out to be
failures • Outcome monitoring does not tell us
impact and can be misleading• A theory based impact evaluation shows if
something is working and why• Quality of match for rigorous study• Independent study got different findings
from project commissioned study
![Page 10: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/10.jpg)
www.3ieimpact.orgHoward White
Stipends in rural China
• Enrolments rose from 40 to 92 percent in project areas
• So stipends “caused” growing enrolments amongst girls
0
20
40
60
80
100
boys
girls
![Page 11: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/11.jpg)
www.3ieimpact.orgHoward White
“Results reporting”
Results… cannot as a rule be attributed specifically, either wholly or in part, to the Netherlands. (Results report 2005-06)
![Page 12: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/12.jpg)
www.3ieimpact.orgHoward White
• Development effectiveness = how effective are development programmes = what difference did they make
• To measure this we need impact evaluation
• Results are what we achieved, not what would have happened anyway
• So outcome monitoring is not enough
![Page 13: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/13.jpg)
www.3ieimpact.orgHoward White
Take away message number 1:Results means impact, so only impact evaluation can tell us if we are achieving results. Results are not captured by outcome monitoring
![Page 14: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/14.jpg)
www.3ieimpact.orgHoward White
So, what is impact evaluation?
![Page 15: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/15.jpg)
www.3ieimpact.orgHoward White
What is impact evaluation?
Impact evaluations answer the question as to what extent the intervention being evaluated altered the state of the world
= the (outcome) indicator with the intervention compared to what it would have been in the absence of the intervention
= Yt(1) – Yt(0)
We can see this
But we can’t see this
So we use a comparison
group
![Page 16: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/16.jpg)
www.3ieimpact.orgHoward White
What do we need to measure impact?
Before After
Project
Comparison
![Page 17: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/17.jpg)
www.3ieimpact.orgHoward White
Girl’s secondary enrolment in rural China
Before AfterProject (treatment) 92
Comparison
The majority of evaluations have just this information … which means we can say absolutely nothing about impact
What do we need to measure impact?
![Page 18: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/18.jpg)
www.3ieimpact.orgHoward White
Before versus after single difference comparisonBefore versus after = 92 – 40 = 52
Before AfterProject (treatment) 40 92
Comparison
This ‘before versus after’ approach is outcome monitoring, which has become popular recently. Outcome monitoring has its place, but it is not impact evaluation
“scholarships have led to rising schooling of young girls in the project villages”
![Page 19: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/19.jpg)
www.3ieimpact.orgHoward White
Rates of completion of elementary male and female students in all rural China’s poor areas
0
20
40
60
80
100Share of rural children
1993 1993 20082008
![Page 20: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/20.jpg)
www.3ieimpact.orgHoward White
Post-treatment comparison comparisonSingle difference = 92 – 84 = 8
But we don’t know if they were similar before… though there are ways of doing this (statistical matching = quasi-experimental approaches)
Before AfterProject (treatment) 92
Comparison 84
![Page 21: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/21.jpg)
www.3ieimpact.orgHoward White
Double difference =(92-40)-(84-26) = 52-58 = -6
Before AfterProject (treatment) 40 92
Comparison 26 84
Conclusion: Longitudinal (panel) data, with a comparison group, allow for the strongest impact evaluation design (though still need matching). SO WE NEED BASELINE DATA FROM PROJECT AND COMPARISON AREAS
![Page 22: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/22.jpg)
www.3ieimpact.orgHoward White
Take away message number 2:Impact evaluation requires a valid comparison group, and baseline data really help. So ex ante design is best
![Page 23: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/23.jpg)
www.3ieimpact.orgHoward White
Comparison group: an identical group of individuals, or households, or firms, or sub-districts, but NOT subject to the programme.
Where do we get the comparison group from?
![Page 24: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/24.jpg)
www.3ieimpact.orgHoward White
RANDOMIZATION RANDOMIZATION RANDOMIZATION
RANDOMIZATION RANDOMIZATIONRANDOMIZATION RANDOMIZATION
RANDOMIZATION RANDOMIZATION
RANDOMIZATION RANDOMIZATION RANDOMIZATION RANDOMIZATION RANDOMIZATION
RANDOMIZATION
RANDOMIZATION
![Page 25: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/25.jpg)
www.3ieimpact.orgHoward White
Random assignment of the intervention…
Not the same as taking a random sample of the ‘treated’
Some examples….
![Page 26: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/26.jpg)
www.3ieimpact.orgHoward White
Voter education (Rajasthan and Delhi)
• Outcomes: voter turnout, vote share of incumbent, politician behavior, service delivery
• Intervention: pre-election voter awareness campaigns (report cards)
• Unit of assignment: 375 GPs, half to get intervention
![Page 27: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/27.jpg)
www.3ieimpact.orgHoward White
Schooling and early marriage
• Outcome: marriage, school attendance and attainment
• Intervention: in-kind transfer for girl remaining in education and unmarried
• Unit of assignment: village
![Page 28: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/28.jpg)
www.3ieimpact.orgHoward White
Health-based education programs
Eyeglasses Vitamin pills
![Page 29: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/29.jpg)
www.3ieimpact.orgHoward White
Some different ways to randomize
Pipeline Raised threshold
![Page 30: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/30.jpg)
www.3ieimpact.orgHoward White
Overcoming resistance to randomization
• There is probably an untreated population anyway
• Need not randomly allocate whole programme just a bit
• Exploit– Roll out– Raised threshold– Encouragement designs
• Don’t need ‘no treatment’ control• RCTs are not unethical, spending money on
programmes that don’t work is unethical
![Page 31: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/31.jpg)
www.3ieimpact.orgHoward White
Take away message number 3: RCTs are possible in a large range of settings… though it is not the only way to conduct IE
![Page 32: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/32.jpg)
www.3ieimpact.orgHoward White
Well designed IEs lead to more nuanced questions
• E.g. conditional cash transfer second generation questions:– Conditions or not?– What sort of conditions?– Who to give money to?– How to give the money?– When and how often to give money?
![Page 33: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/33.jpg)
www.3ieimpact.orgHoward White
Second generation questions: computer-assisted learning, CAL
• Most cost effective number of children per computer?
• What sort of software?• How much teacher training required?• What technological back up needed?• What age groups totarget?
![Page 34: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/34.jpg)
www.3ieimpact.orgHoward White
So conduct studies to get inform design to get better results
![Page 35: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/35.jpg)
www.3ieimpact.orgHoward White
And which policies are most cost effective
![Page 36: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/36.jpg)
www.3ieimpact.orgHoward White
Take away message number 4: Impact evaluation is not just about what works, but why, where and at what cost, and offers insights on intervention design, and so delivers better results
![Page 37: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/37.jpg)
www.3ieimpact.orgHoward White
Implications for results-based budgeting
In principle can identify priority outcomes, and what interventions are most cost effective in achieving these outcomes, and so allocate budget to things that work
This IS being done in some countries…
![Page 38: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/38.jpg)
www.3ieimpact.orgHoward White
But it’s not happening in most• “Evaluation is not systematically embedded in the
GoU’s management practices…Because evaluation addresses issues such as actual progress in attainment of program objectives, cost effectiveness, and value for money, it responds to some of the aspects of Uganda’s M&E system that are most critically lacking.”
• “There has been a general tendency to monitor rather than evaluate.” (Sri Lanka)
• “…the distortion found in most countries of an excess of monitoring and a dearth of genuine evaluation.” (World Bank)
![Page 39: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/39.jpg)
www.3ieimpact.orgHoward White
And attribution is not addressed
• “…M&E is not geared toward understanding causality and attribution between the stages of development change.” (Uganda)
• “Furthermore, while national and provincial treasuries have emphasized an approach to collecting information that is based on logical framework (log-frame) results chain, they have not focused on attribution or causality.” (South Africa)
![Page 40: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/40.jpg)
www.3ieimpact.orgHoward White
But there are growing cases..
![Page 41: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/41.jpg)
www.3ieimpact.orgHoward White
Evidence into practice examples
![Page 42: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/42.jpg)
www.3ieimpact.orgHoward White
Recommendations• Review current M&E systems and how it
aligns with requirements for “results”• Identify some priority areas for impact
evaluation, and commission a small number of studies (both ex post and ex ante)
• Start development of national framework to build systematic impact evaluation into M&E, and budgeting to ‘performance’ meaning results, meaning impact
![Page 43: Howard White International Initiative for Impact Evaluation](https://reader035.fdocuments.net/reader035/viewer/2022062310/56815f12550346895dcdd543/html5/thumbnails/43.jpg)
www.3ieimpact.orgHoward White
Thank you
Visit www.3ieimpact.org