• Development Impact Evaluation • in Finance and Private Sector

•1

Workshop Development Impact Evaluation in Finance and Private SectorDakar February 2010With generous support from Gender Action Plan

Steps in Implementing an Impact Evaluation

Arianna LegoviniHead, Development Impact Evaluation InitiativeThe World Bank

Steps

Step 1. Build capacity for IE

Objectives: Become informed consumers of impact

evaluation Set the learning agenda Use it as an internal management tool to

improve program over time

How Training Learning by doing

Step 2: Set learning agenda

Objective: Get answers to relevant policy and operational

questions How?

Dialectic discussion involving key policy makers and program managers

Technical facilitation to structure framework of analysis

Focus on few critical policy (what) and operational (how to) questions

Discuss agenda with authorizing environment and constituencies

Cont. 2: Questions Operational: design-choices of program

▪ Institutional arrangements, Delivery mechanisms, Packages, Pricing/incentive

Management purpose Use random trials to test alternatives Measure effects on short term outcomes (months)▪ take up rates, use, adoption

Scale up better implementation modalities

Policy: effectiveness of program Accountability purpose Use random assignment or next best method Measure effects medium to long term Scale up/down, negotiate budget, inform

Step 3: Design IE

Exploit opportunities: Will roll-out take time? Is the budget allocated insufficient to cover everyone? Are there quantitative eligibility rules? If the program has universal access, does it have imperfect

take-up? Set scale:

Pilot to try out an intervention Large scale w. representative sample: more costly, externally

valid Large scale with purposeful sample: less costly, indicative

Do power calculation to determine minimum sample size

Cont. Step 3Select “best” method for each of your

questions Feasible Requires least assumptions

Ethics No to deny access to something for which

there is irrefutable evidence Test interventions before scale up when

you have no solid evidence

Step 4: Planning implementation

Budget cost items▪ Staff time (PROJECT FUNDS) and training (DIME)▪ Analytical services and field coordination (DIME)▪ Data collection (PROJECT FUNDS)▪ Discussions and dissemination (shared)

Timeline▪ Use it to organize activities, responsibilities and work

backwards to know when to start Team

▪ Government (program manager, economist/statistician); WB Project team (Task manager or substitute); Research team (Lead researcher, co-researchers, field coordinator); Data collection agency

Step 5: Assignment to treatment and control The smallest unit of assignment is the unit

of intervention Credit: individual or group

SME services: enterprise

Municipal registration system: municipality

Create listing of treatment units assigned to the intervention and control units that are not

Explain assignment to responsible parties to avoid contamination

Step 6: Baseline data

Quality assurance : IE team (not data collection agency) to Design questionnaire and sample Define terms of reference for data

collection agency Train enumerators Conduct pilot Supervise data collection

Do not collect data before your design is ready and agreed

Cont. Step 6: Baseline data

Contract data collection agency Bureau of Statistics: Integrate with existing

data Ministry concerned: Ministry of

Agriculture/Water Resources/Rural Development Private agency

Analyze baseline data a feed back into program and evaluation design if needed

Check for balance between treatment and control group: do they have similar average characteristics?

Step 7: Roll out intervention

Conduct intensive monitoring of roll-out to ensure evaluation is not compromised

What if treatment and control receive the intervention?

What if all the control group receive some other intervention?

Step 8: Follow-up dataCollect follow-up data with the same

sample and questionnaire as baseline data

At appropriate intervals

Step 9: Estimate program effects

Randomization: compare average outcomes for treatment and control group

Other methods: Use relevant econometric analysis , test assumptions, check robustness

Are the effects statistically significant? Basic statistical test tells whether differences are due to

the program or to noisy data Are they significant in real terms?

If a program is costly and its effects are small, may not be worthwhile

Are they sustainable? Is the trajectory of results sustained?

Step 10: Discuss, Disseminate and Feedback into policy Are you thinking about this only now? Discuss what are the policy implications of the results What actions should be taken How to present them to higher ups to justify

changes/budget/scale up? Talk to policy-maker and disseminate to wider audience If no one knows about it, it won’t make a difference Make sure the information gets into the right policy

discussions Real time discussions Workshops Reports Policy briefs

Final step: Iterate

What do you need to learn next?