Impact Evaluation and Impact Evaluation and the Project Cyclethe Project Cycle

Arianna LegoviniArianna LegoviniPREM WEEK May 2, 2006PREM WEEK May 2, 2006This presentation is based on work by the This presentation is based on work by the

Thematic Group on Impact EvaluationThematic Group on Impact Evaluation

Objective of the presentationObjective of the presentation

Walk you through what it takes to do an Walk you through what it takes to do an impact evaluation for your project from impact evaluation for your project from Identification to ICRIdentification to ICR

Persuade you that impact evaluation will Persuade you that impact evaluation will add value to your projectadd value to your project

We will talk about…We will talk about…

Objective of the evaluationObjective of the evaluation General PrinciplesGeneral Principles Evaluation activities – the core issues for Evaluation activities – the core issues for

evaluation design and implementation, evaluation design and implementation, andand

Housekeeping activities—procedural, Housekeeping activities—procedural, administrative and financial management administrative and financial management issuesissues

Why do an impact evaluation of Why do an impact evaluation of your project?your project?

Provide a sound basis for policy developmentProvide a sound basis for policy development

Measure the impact of the project on intended and Measure the impact of the project on intended and unintended outcomes for:unintended outcomes for:

Making budget decisions and reallocating resources (fiscal Making budget decisions and reallocating resources (fiscal accountability)accountability)

Scaling up (provisos apply)Scaling up (provisos apply)

Measure the relative effectiveness of alternatives Measure the relative effectiveness of alternatives (modes of delivery, packages, pricing schemes) to:(modes of delivery, packages, pricing schemes) to:

Modify project features overtime (managing by results)Modify project features overtime (managing by results) Inform future project designs Inform future project designs

Some general principlesSome general principles

Government ownership—what matters is Government ownership—what matters is institutional buy-ininstitutional buy-in

Relevance and applicability—asking the Relevance and applicability—asking the right questions right questions

Flexibility and adaptabilityFlexibility and adaptability Horizon mattersHorizon matters

Country ownershipCountry ownership Ensure Government involvement at all stages to Ensure Government involvement at all stages to

build institutional capacity and a culture of build institutional capacity and a culture of managing-by-results.managing-by-results.

Agree on a dissemination plan to maximize use of Agree on a dissemination plan to maximize use of results for policy development. results for policy development.

Identify entry points in project and policy cycles Identify entry points in project and policy cycles midpoint and closing, for project; midpoint and closing, for project; sector reporting, CGs, MTEF, budget, for policysector reporting, CGs, MTEF, budget, for policy

Use partnerships with local academics to build Use partnerships with local academics to build local capacity for impact evaluation. local capacity for impact evaluation.

Example, Kenya educationExample, Kenya education

Relevance and ApplicabilityRelevance and Applicability

For an evaluation to be relevant, it must be For an evaluation to be relevant, it must be designed to respond to the policy questions that designed to respond to the policy questions that are of importance to the clients. are of importance to the clients.

Clarifying early what it is that the client wants to Clarifying early what it is that the client wants to learn and designing the evaluation to that end learn and designing the evaluation to that end will go some way to ensure that the will go some way to ensure that the recommendations of the evaluation will feed into recommendations of the evaluation will feed into policy making. policy making.

Example, devolution and decentralization in Example, devolution and decentralization in Zambia Zambia

Flexibility and adaptabilityFlexibility and adaptability The evaluation must be tailored to the specific The evaluation must be tailored to the specific

project and adapted to the specific institutional project and adapted to the specific institutional context.context.

The project design must be flexible to secure our The project design must be flexible to secure our ability to learn in a structured manner, feed ability to learn in a structured manner, feed evaluation results back into the project and evaluation results back into the project and change the project mid-course to improve change the project mid-course to improve project end results.project end results.

Example, Ethiopia energyExample, Ethiopia energy

This is an important point: In the past projects This is an important point: In the past projects have been penalized for affecting mid-course have been penalized for affecting mid-course changes in project design. Now we want to changes in project design. Now we want to

make change part of the project design.make change part of the project design.

Horizon mattersHorizon matters The time it takes to achieve results is an important The time it takes to achieve results is an important

consideration for timing the evaluation. Conversely, the consideration for timing the evaluation. Conversely, the timing of the evaluation will determine what outcomes timing of the evaluation will determine what outcomes should be focused on.should be focused on.

Early evaluations should focus on outcomes that are quick to Early evaluations should focus on outcomes that are quick to show changeshow change

For long-term outcomes, evaluations may need to span beyond For long-term outcomes, evaluations may need to span beyond project cycle. Example, Madagascar early childhood project cycle. Example, Madagascar early childhood developmentdevelopment

Think through how things are expected to change over Think through how things are expected to change over time and focus on what is within the time horizon for the time and focus on what is within the time horizon for the evaluationevaluation

Do not confuse the importance of an outcome with the time Do not confuse the importance of an outcome with the time it takes for it to change—some important outcomes are it takes for it to change—some important outcomes are

obtained instantaneously !obtained instantaneously !

Identification to PCNIdentification to PCN

Get an Early StartGet an Early StartHow do you get started?How do you get started?

Get help and access to resources: contact Get help and access to resources: contact person in your region or sector responsible for person in your region or sector responsible for impact evaluation and/or Thematic Group on impact evaluation and/or Thematic Group on Impact Evaluation and HDN CEOImpact Evaluation and HDN CEO

Define the timing for the various steps of the Define the timing for the various steps of the evaluation to ensure you have enough lead time evaluation to ensure you have enough lead time for preparatory activities (e.g. baseline goes to for preparatory activities (e.g. baseline goes to the field before program activities start)the field before program activities start)

The evaluation will require support from policy-The evaluation will require support from policy-makers: start building and maintaining makers: start building and maintaining constituents, dialogue with relevant part of constituents, dialogue with relevant part of government, build a broad base of support, government, build a broad base of support, include stakeholders include stakeholders

Build the TeamBuild the Team Select impact evaluation team and define Select impact evaluation team and define

responsibilities of:responsibilities of: program managers (government), program managers (government), project team, and other donors, project team, and other donors, lead researcher (impact evaluation specialist), lead researcher (impact evaluation specialist), local research team, andlocal research team, and data collection agency or firmdata collection agency or firm

Selection of lead researcher is critical for ensuring Selection of lead researcher is critical for ensuring quality of product, and so is the capacity of the data quality of product, and so is the capacity of the data

collection agencycollection agency

Partner with local researchers and research Partner with local researchers and research institutes to build local research capacityinstitutes to build local research capacity

Shift ParadigmShift Paradigm

From a project design based on “we know what’s best” From a project design based on “we know what’s best” To project design based on the notion that “we can learn To project design based on the notion that “we can learn

what’s best in this context, and adapt to new knowledge as what’s best in this context, and adapt to new knowledge as needed” needed”

Work iteratively:Work iteratively: Discuss what the team knows and what it needs to learn–the Discuss what the team knows and what it needs to learn–the

questions for the evaluation—to deliver on project objectivesquestions for the evaluation—to deliver on project objectives Discuss translating this into a feasible project designDiscuss translating this into a feasible project design Figure out what questions can feasibly be addressedFigure out what questions can feasibly be addressed Housekeeping: Include these first thoughts in a paragraph in the Housekeeping: Include these first thoughts in a paragraph in the

PCNPCN

Example: Incorporate learning in an Example: Incorporate learning in an energy access in Ethiopiaenergy access in Ethiopia

One component of the project will distribute Compact One component of the project will distribute Compact Fluorescent Light bulbs (CFLs) at subsidized prices to increase Fluorescent Light bulbs (CFLs) at subsidized prices to increase energy efficiency and reduce energy costs for the poor.energy efficiency and reduce energy costs for the poor.

The team wants to know how to deliver the greatest number of The team wants to know how to deliver the greatest number of CFLs under budget constraints. What is the optimal value for CFLs under budget constraints. What is the optimal value for the subsidy value? What is the best delivery mechanism? the subsidy value? What is the best delivery mechanism?

During the first six months of CFL distribution, the electric During the first six months of CFL distribution, the electric company will experiment with alterative subsidy values (high, company will experiment with alterative subsidy values (high, medium, low) and distribution mechanisms (local market, medium, low) and distribution mechanisms (local market, company distribution) in different localities. Localities will be company distribution) in different localities. Localities will be randomly selected to one or the other treatment.randomly selected to one or the other treatment.

At the end of the first six months, the company with evaluate At the end of the first six months, the company with evaluate which model was most cost-efficient (use of CFLs per dollar which model was most cost-efficient (use of CFLs per dollar spent), and implement that model for the rest of the project.spent), and implement that model for the rest of the project.

Preparation through appraisalPreparation through appraisal

Define project development Define project development objectives and results frameworkobjectives and results framework

This activity This activity clarifies the results chain for the project, clarifies the results chain for the project, identifies the outcomes of interest and the indicators identifies the outcomes of interest and the indicators

best suited to measure changes in those outcomes, best suited to measure changes in those outcomes, and and

the expected time horizon for changes in those the expected time horizon for changes in those outcomes. outcomes.

This will provide the lead researcher with the This will provide the lead researcher with the project specific variables that must be included project specific variables that must be included in the survey questionnaire and a notion of in the survey questionnaire and a notion of timing for scheduling data collection. timing for scheduling data collection.

Work out project design features Work out project design features that will affect evaluation designthat will affect evaluation design

Target population and rules of selection Target population and rules of selection This provides the evaluator with the universe for the This provides the evaluator with the universe for the

treatment and comparison sampletreatment and comparison sample

Roll out planRoll out plan This provide the evaluation with a framework for This provide the evaluation with a framework for

timing data collection and, possibly, an opportunity to timing data collection and, possibly, an opportunity to define a comparison group define a comparison group

The impact evaluator will work iteratively with the The impact evaluator will work iteratively with the team to agree on selection and roll out that team to agree on selection and roll out that allows for rigorous evaluation (F&A)allows for rigorous evaluation (F&A)

Narrow down the questions for the Narrow down the questions for the evaluationevaluation

Questions aimed at measuring the impact of Questions aimed at measuring the impact of the project on a set of outcomes, andthe project on a set of outcomes, and

Questions aimed at measuring the relative Questions aimed at measuring the relative effectiveness of different features of the effectiveness of different features of the projectproject

What is your hypothesis? (Results framework)What is your hypothesis? (Results framework) By expanding water supply, the use of clean water will increase, By expanding water supply, the use of clean water will increase,

water borne disease decline, and health status improvewater borne disease decline, and health status improve What is the evaluation question?What is the evaluation question?

Does improved water supply result in better health outcomes?Does improved water supply result in better health outcomes? How can do you test the hypothesis?How can do you test the hypothesis?

The government might randomly assign areas for expansion in water The government might randomly assign areas for expansion in water supply during the first and second phase of the programsupply during the first and second phase of the program

What will you measure?What will you measure? Measure the change in health outcomes in phase I areas relative to Measure the change in health outcomes in phase I areas relative to

the change in outcomes in phase II areas. Outcomes will include use the change in outcomes in phase II areas. Outcomes will include use of safe water (S-T), incidence of diarrhea (S/M-T), and health status of safe water (S-T), incidence of diarrhea (S/M-T), and health status (L-T, depending on when phase II occurs). Add other outcomes.(L-T, depending on when phase II occurs). Add other outcomes.

What will you do with the results?What will you do with the results? If the hypothesis proves true go to phase II; if false, modify policy.If the hypothesis proves true go to phase II; if false, modify policy.

Questions aimed at measuring the impact of Questions aimed at measuring the impact of the projectthe project are relatively straightforwardare relatively straightforward

Questions aimed at measuring the relative Questions aimed at measuring the relative effectiveness of different project features effectiveness of different project features require identifying the tough design choices on the require identifying the tough design choices on the

table…table…

What is the issue? What is the issue? What is the best package of products or services? What is the best package of products or services?

Where do you start from? Where do you start from? What package is the government delivering now? What package is the government delivering now?

Which changes do you or the government think could Which changes do you or the government think could be made to improve effectiveness? be made to improve effectiveness?

How do you test it?How do you test it? The government might agree to provide a package to a The government might agree to provide a package to a

randomly selected group of households and another randomly selected group of households and another package to another group of households to see how the package to another group of households to see how the two package performtwo package perform

What will you measure?What will you measure? The average change in relevant outcomes for The average change in relevant outcomes for

households receiving one package versus the same for households receiving one package versus the same for households receiving the other package households receiving the other package

What will you do with the results?What will you do with the results? The package that is most effective in delivering The package that is most effective in delivering

desirable outcomes becomes the one adopted by the desirable outcomes becomes the one adopted by the project from the evaluation onwardsproject from the evaluation onwards

Application, features that should be Application, features that should be tested early ontested early on

Early testing of project features (say 6 months to Early testing of project features (say 6 months to 1 year) can provide the team with the 1 year) can provide the team with the information needed to adjust the project early on information needed to adjust the project early on in the direction most likely to deliver success. in the direction most likely to deliver success.

Features might include:Features might include: alternative modes of delivery (e.g. school-based vs. alternative modes of delivery (e.g. school-based vs.

vertical delivery), vertical delivery), alternative packages of outputs (e.g. awareness alternative packages of outputs (e.g. awareness

campaigns vs. legal services), or campaigns vs. legal services), or different pricing schemes (e.g. alternative subsidy different pricing schemes (e.g. alternative subsidy

levels). levels).

Example: Incorporate learning in an Example: Incorporate learning in an energy project in Ethiopiaenergy project in Ethiopia

The project will distribute CFLs at subsidized prices to The project will distribute CFLs at subsidized prices to increase energy efficiency and reduce energy costs for the increase energy efficiency and reduce energy costs for the poor.poor.

The team wants to know how to deliver the greatest number The team wants to know how to deliver the greatest number of CFLs under budget constraints. What is the optimal value of CFLs under budget constraints. What is the optimal value for the subsidy value? What is the best delivery mechanism? for the subsidy value? What is the best delivery mechanism?

During the first six months of CFL distribution, the electric During the first six months of CFL distribution, the electric company will experiment with alterative subsidy values (high, company will experiment with alterative subsidy values (high, medium, low) and distribution mechanisms (local market, medium, low) and distribution mechanisms (local market, company distribution) in different localities. Localities will be company distribution) in different localities. Localities will be randomly selected to one or the other treatment.randomly selected to one or the other treatment.

At the end of the first six months, the company with evaluate At the end of the first six months, the company with evaluate which model was most cost-efficient (use of CFLs per dollar which model was most cost-efficient (use of CFLs per dollar spent) and market friendly, and implement that model for the spent) and market friendly, and implement that model for the next 4 years.next 4 years.

Develop identification strategyDevelop identification strategy((to identify the impact of the project separately from changes due to to identify the impact of the project separately from changes due to

other causesother causes ) )

One the questions are defined, the lead One the questions are defined, the lead researcher selects one or more comparison researcher selects one or more comparison groups against which to measure results in the groups against which to measure results in the treatment group. treatment group.

The “rigor” with which the comparison group is The “rigor” with which the comparison group is selected will determine the reliability of the selected will determine the reliability of the impact estimates. impact estimates.

Rigor? Rigor? More-same observables and unobservables More-same observables and unobservables

(experimental), (experimental), Less-same observables (non-experimental)Less-same observables (non-experimental)

Explore Existing DataExplore Existing Data Explore what data exists that might be relevant Explore what data exists that might be relevant

for use in the evaluation. for use in the evaluation. Discuss with the agencies of the national statistical Discuss with the agencies of the national statistical

system and universities to identify existing data system and universities to identify existing data sources and future data collection plans. sources and future data collection plans.

Record data periodicity, quality, variables Record data periodicity, quality, variables covered and sampling frame and sample size, covered and sampling frame and sample size, forfor CensusesCensuses Surveys (household, firms, facility, etc)Surveys (household, firms, facility, etc) Administrative dataAdministrative data Data from the project monitoring systemData from the project monitoring system

New DataNew Data Start identifying additional data collection needsStart identifying additional data collection needs. .

Data for impact evaluation must be representative of treatment and Data for impact evaluation must be representative of treatment and comparison group comparison group

Questionnaires must include outcomes of interest (consumption, Questionnaires must include outcomes of interest (consumption, income, assets etc), questions about the program in question and income, assets etc), questions about the program in question and questions about other programsquestions about other programs

The data might be at household, community, firm, facility, or farm The data might be at household, community, firm, facility, or farm levels and might be combined with specialty data such as those levels and might be combined with specialty data such as those from water or land quality tests.from water or land quality tests.

Investigate synergies with other projects to combine data Investigate synergies with other projects to combine data collection efforts and/or explore existing data collection collection efforts and/or explore existing data collection efforts on which the new data collection could piggy back efforts on which the new data collection could piggy back

Develop a data strategy for the impact evaluation including:Develop a data strategy for the impact evaluation including: The timing for data collectionThe timing for data collection The variables neededThe variables needed The sampleThe sample Plans to integrate data from other sources Plans to integrate data from other sources

Prepare for collecting dataPrepare for collecting data Identify data collection agency Identify data collection agency Lead researcher will work with the data Lead researcher will work with the data

collection agency to design sample, and train collection agency to design sample, and train enumeratorsenumerators

Lead researcher will prepare survey Lead researcher will prepare survey questionnaire or questionnaire module as questionnaire or questionnaire module as neededneeded

Pre-testing survey instrument may take place at Pre-testing survey instrument may take place at this stage to finalize instrumentsthis stage to finalize instruments

If financed with outside funds, baseline can now If financed with outside funds, baseline can now go to the field. If financed by project funds, go to the field. If financed by project funds, baseline will go to the field just after baseline will go to the field just after effectiveness but before implementation startseffectiveness but before implementation starts

Develop a Financial PlanDevelop a Financial Plan Costs:Costs:

Lead researcher and researcher team, Lead researcher and researcher team, Data collection, Data collection, Supervision and Supervision and DisseminationDissemination

Finances: Finances: BB, BB, Trust fund, Trust fund, Research grants, Research grants, Project funds, or Project funds, or Other donor fundsOther donor funds

HousekeepingHousekeeping Initiate an IE activity. The IE code in SAP is a way of Initiate an IE activity. The IE code in SAP is a way of

formalizing evaluation activities. The IE code recognizes formalizing evaluation activities. The IE code recognizes the evaluation as a separate AAA product. the evaluation as a separate AAA product.

Prepare concept note Prepare concept note Identify peer reviewers –impact evaluation and sector specialistIdentify peer reviewers –impact evaluation and sector specialist Carry out review process Carry out review process

Appraisal documentsAppraisal documents Include in the project description plans to modify project Include in the project description plans to modify project

overtime to incorporate resultsovertime to incorporate results Work the impact evaluation into the M&E section of the PAD and Work the impact evaluation into the M&E section of the PAD and

Annex 3 Annex 3 Include the impact evaluation in the Quality Include the impact evaluation in the Quality

Enhancement Review (TTL). Enhancement Review (TTL).

Negotiations to CompletionNegotiations to Completion

Ensure timely implementationEnsure timely implementation

Ensure timely procurement of evaluation Ensure timely procurement of evaluation services especially contracting the data services especially contracting the data collection, andcollection, and

Supervise timely implementation of the Supervise timely implementation of the evaluation includingevaluation including Data collectionData collection Data analysisData analysis Dissemination and feedbackDissemination and feedback

Data collection agency/firmData collection agency/firm

Data collection agency or firm must have Data collection agency or firm must have technical knowledge and sufficient technical knowledge and sufficient logistical capacity relative to the scale of logistical capacity relative to the scale of data collection requireddata collection required

The same agency or firm should be The same agency or firm should be expected to do baseline and follow up data expected to do baseline and follow up data collectioncollection

Baseline data collection and analysisBaseline data collection and analysis Baseline data collection should be carried Baseline data collection should be carried

out before program implementation out before program implementation begins; optimally even before program is begins; optimally even before program is announcedannounced

Analysis of baseline data will provide Analysis of baseline data will provide program management with additional program management with additional information that might help finalize information that might help finalize program designprogram design

Follow-up data collection and Follow-up data collection and analysisanalysis

The timing of follow-up data collection The timing of follow-up data collection must reflect the learning strategy adoptedmust reflect the learning strategy adopted

Early data collection will help modifying Early data collection will help modifying programs mid course to maximize longer-programs mid course to maximize longer-term effectivenessterm effectiveness

Later data collection will confirm Later data collection will confirm achievement of longer-term outcomes and achievement of longer-term outcomes and justify continued flows of fiscal resources justify continued flows of fiscal resources into the programinto the program

DisseminationDissemination Implement plan for dissemination of evaluation results Implement plan for dissemination of evaluation results

ensuring that the timing is aligned with government’s ensuring that the timing is aligned with government’s decision making cycle. decision making cycle.

Ensure that results are used to inform project Ensure that results are used to inform project management and that available entry points are management and that available entry points are exploited to provide additional feedback to the exploited to provide additional feedback to the governmentgovernment

Ensure that wider dissemination takes place only after Ensure that wider dissemination takes place only after the client has had a chance to preview and discuss the the client has had a chance to preview and discuss the resultsresults

Nurture collaboration with government and local Nurture collaboration with government and local researchers throughout the processresearchers throughout the process

HousekeepingHousekeeping

Involve local project implementation unit Involve local project implementation unit and PIU person responsible for monitoring and PIU person responsible for monitoring and evaluation and evaluation

Put in place arrangements to procure the Put in place arrangements to procure the impact evaluation work and fund it on time impact evaluation work and fund it on time

Use early results to inform mid-term Use early results to inform mid-term review review

Use later results to inform the ICR, CAS Use later results to inform the ICR, CAS and future operationsand future operations

Concluding remarksConcluding remarks

Making evaluation work for you requires a Making evaluation work for you requires a change in the culture of project design and change in the culture of project design and implementation, one that maximizes the use of implementation, one that maximizes the use of learning to change course when necessary and learning to change course when necessary and improve the chances for successimprove the chances for success

Impact evaluation more than a tool is an Impact evaluation more than a tool is an organizing analytical framework for doing thisorganizing analytical framework for doing this

Thank youThank you