How we can use impact evaluation to assure effective use of resources for development

How we can use impact evaluation to assure effective use of resources

for development

Maximo Torero, [email protected]

DirectorMarkets, Trade and Institutions Division

(IFPRI)

IFAD-IFPRI Partnership, January 31st. 2012

Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation

approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments

Need for impact evaluation Helps identify and measure the results Helps identify the causal link between

intervention and results Provides a systematic and objective assessment

of program impacts Helps determine if interventions are relevant and

cost effective Promotes accountability, evidence-based

policymaking, and learning.

Need for impact evaluation Over past decade, increased demand from

governments, donor agencies and general public, for evidence of Impact of development policies.

Political tool: Brings accountability regarding the use of development money

Fiscal tool / budgetary tool: Allocate resources across different sectors or programs

Management tools: Understand how to better reach the objectives.

Monitoring and Impact Evaluation:Monitoring A tool that provides regular information on:

How a project is being implemented How a project is operating in the field How a project is progressing relative to targets What is the quality of service delivery (where applicable)

Rationale for Monitoring:

Provides basis for corrective action Holds implementers accountable for delivery of inputs Provides assessment of continued relevance Provides critical information for decision-making

Monitoring and Impact Evaluation:Evaluation Impact Evaluation:

Measures effectiveness and impact of programs or policies on outcomes of interest

Seeks to establish causality

Not all programs need to be evaluated; not all outcomes need to be measured in all evaluations

Indicators for Monitoring and Evaluation

IMPACT

OUTPUTS

OUTCOMES

INPUTS

Effect on living standards - better welfare impacts (e.g literacy, health) - increase in participation, happiness

Financial and physical resources - track resources used in the intervention

-e.g. budget support for local service delivery

Goods and services generated- more local government services delivered- e.g., textbooks, food delivered, roads built

Access, usage and satisfaction of users- e.g. school attendance, vaccination rates, - food consumption, number of mobile phones

Eva

luat

ion

Mon

itorin

g

Guiding Principles of our IE Approach1. Generate information to influence decisions

2. Specify which indicators and methods are most suitable for each type of projects

3. Identify impact pathways

4. Evaluation activities must be built into the project design

4. Consider direct and indirect beneficiaries of projects

5. Evaluation at different levels of aggregation: Individual, thematic, and overall program

6. Incorporate complementarities and substitution among project impacts

Stage 1: Consultation

Stage 2: Development

IFAD’s:• Objectives• Activities• Information

needs to

Identify:• Indicators• Methods

• Monitor performance• Evaluate Effectiveness• Asses Impact

Target PEOPLE and vulnerable GROUPS:• Poor and Women

by Theme:•Technology• Productivity• Market Access• Nutrition

by level of Aggregation:• Individual projects• Theme• Agricultural Development

program

Stage 3: Feedback

• Governments, • IFAD• Implementers• CSO

Description of the Project

Impact Evaluation: Impact Pathway

The expected causal chain of events leading from project activities to outputs, to changes in the target population, and to the achievement of project objectives:

From INPUTS OUTCOMES IMPACTS

Focus on the impact pathway allows to: Understand how impacts are (or are not) achieved Allows generalizability of findings Provides key information for scaling up Identification of indicators for each step along the impact pathway

Scholarships for plant breeders & grants for agronomic research

number and quality of varieties released

availability and adoption ofimproved crop varieties

Higher yields for farmers whoadopted improved varieties

income, poverty among farmer households

IMPACT PATHWAY INDICATORS METHODS

Spending on scholarships& research grants

No. new varietiesApproved & released

% male, female farmersUsing improved varieties

Average yields amongadopting farmers

Income, expenditure,Well-being indicatorsamong target groups (poor, women, etc.)

Internal programmonitoringPR

OC

ESS

IMPA

CT

Intra-HH surveys:Before/After, Beneficiary/Control (Diff in Diff)

Illustrative Impact Pathway, Indicators, Methods Example from: Science & Technology

Applying the Methodology to specific types of interventions

Technology Example: Bio-fortification

Productivity Example: Grants to crop breeding programs

Market Access Example: Participation of small holders in the dairy value chain, “chilling

plant hubs”

Nutrition interventions Example: Evaluation of specific interventions to improve nutrition of the

most vulnerable

Bio-fortification

Adoption of bio-fortified varieties

Greater yields for farmers who adopted bio-fortified varieties


Spending on bio-fortification R&D

Average yields among adopting farmers

Income, expenditure,Well-being indicators

Internal programmonitoring

PRO

CES

SIM

PAC

T

Production of bio-fortified varieties

Consumption of bio-fortified varieties

Change in micronutrient status

Improvements in health, work performance, cognitive ability

income, poverty among farmer households

No. of farmers and land adopting bio-fortified varieties.

Total production of bio-fortified varieties

No. of individuals and average consumption (by type of individual))…

Reducing micronutrient malnutrition

Morbidity, mortality, enrollment ratio in primary

Bio-fortification Project (Science and Technology)Assumption: No price effect…

HH surveys•Beneficiary, control• Farmers, consumers•DD estimator•Randomization

•Panel: first round effect vs. second round effects

•Qualitative information: two-way calling with the poor

consumption of animal products, fruits, and vegetables

Creation of farmer groups as dairy farmer business associations (DFBA)

Chilling plant construction


Number of DFBA created and number of farmers participating (by gender)

Number of plants and milk capacity

Income, expenditure,Well-being indicators

Internal programMonitoring

Qualitative Assessment: organizational capacity

PRO

CES

SIM

PAC

T

Increase milk production of member farmers

Sales to formal markets and traditional markets

income, poverty among farmer members

milk production of farmer members

Volume of loss due to spoilage

Value of sales to formal processors and to traditional markets

Chilling Plant Hubs (Market Access)

HH surveys•Beneficiary, control•DD estimator•Non-experimental design

•Qualitative information: two-way calling with the poor

Reduction in loss through spoilage

M&E at Different Levels of Aggregation

What needs to be learned at the theme level?

Theme specific indicators:

Market access Productivity Science and tech. Data Analysis

Meta analysis within theme

Database at project level

within themes

What needs to be learned at the strategy level?

Cross theme indicators:

poverty

Meta analysis at the strategy

level

Evaluation strategy Indicators Methods

Project level

Themelevel

Programlevel

What needs to be learned at the project level?

Project Indicators:

Process indicators Outcome

indicators

Quantitative

Qualitative analysis

Impact Evaluation: Concepts

Impact evaluation hinges on determining what would have happened if the program had not existed.

Good practice involves a comparison of outcome before and after intervention with those with and without intervention

Problem is identifying valid counterfactual

Impact Evaluation: Methods

Quantitative Methods

Pre and post intervention, no control group Pre and post intervention, with control group, but

no randomization Pre and post intervention, with control group

and randomization

cost

reliability

Qualitative Methods - complementary – help:

Interpret of quantitative results Identify unexpected impacts, or effects on groups that are not captured

by quantitative surveys, etc.

Counterfactual

Ideally: Observe the outcome variable for those in the

program and For those same individuals had they NOT

participated in the program (the counterfactual)

So, constructing the counterfactual is the key issue that any empirical method must effectively handle.

Before the program

Beneficiaries:

Non-Beneficiaries:

A: “Treatment” Status

B: “Non Treatment” Status

C: “Treatment” Status

D: “Non Treatment” Status

E: Status beforethe program

F: Status beforethe program

Shaded boxes are Observable situations

Unshaded boxes are Unobservable

After the program

RealCounterfactual

EstimatedCounterfactual

Impact Evaluation: Finding a Counterfactual

Concept: How is the outcome different than it would have been if the project had not been implemented? = A – B (but cannot be observed)So estimated impact is based on double difference: (A-E) – (D-F)

Supposed we observe an increase in outcome Y for beneficiaries over time after an intervention

Y0

Y1

baseline(t0) follow-up(t1)

Intervention

(observed)

To measure impact, we need to remove the counterfactual from the observed outcome

Y0

Y1

baseline(t0) follow-up(t1)

Intervention

(observed)

Y1*

Impact=Y1-Y1

*

(counterfactual)

Comparison

Treatment Effects: key obstacles

Experimental vs. Non-Experimental Data Experimental data rules out self-selection into the

program (according to observables or unobservables) as a source of bias in measuring the treatment effect

So, this contribution of experimental data brings into high relief the two key obstacles that non-experimental data methods must overcome in order to avoid biased estimates of the average treatment effect:

Treatment Effects: Key obstacles (cont)

1. Self-selection into the program due to observables characteristics

2. Self-selection into the program due to unobservable characteristics

Accounting for #1 is often difficult (or impossible) to accomplish.

Even if #1 is accounted for in the method but # 2 is not, then bias in the result will inevitably occur.

Similarly if the control and treatment groups are randomly selected from a population then there is no bias in the initial characteristics

The impact of the procedure X can be attributed to the differences in the variable Y between the control and treatment group.

Population Random selection

Treatment Group (receives procedure

X)

Control group

(not receives procedure X)

Y Exp – Y Control

Although normally experimental methods are not applied ¿Why we can not then apply a direct comparison between the control

and treatment group? Because differences in characteristics of subjects, or what is called selection bias.

Population NO random selection

Quintile I (more poor)

Quintile II Quintile III Quintile IV QuintileV (more richer)

Because of initial differences between

both groups, the effects of the treatment can not be identified by directly comparing the groups


X)

Control group


Selection bias: “Graphically”

SB = 0

G=ATT

SB > 0

G>ATT

SB < 0

G<ATT

Observed difference (G)Impact on the treated (ATT) = true effect of the program on its recipientsSelection Bias (SB)

No selection bias Selection on “better-off” with respect to the

outcome

Selection on “worse-off” with respect to the

outcome

Observed G

Overcoming selection bias

Ex-ante

Experimental approach: the design of the program allows to introduce randomness in its allocation

Ex-post

Natural experiment approaches: there are events that allow to simulate “exogeneity in the choice of treatment”

Control approaches: try to neutralize (reduce) as much as possible the selection bias

Experimental approaches

Randomly allocate “Treatment” into a population. Eliminates selection bias:

Sometimes ethical critics If the exclusion of some beneficiaries is only due to the evaluation, while benefits

are well known In reality, resource constraints are the limiting factor. Then, random selection can

be considered a fair process (every potential beneficiary has same chance of being selected)

Must be designed before the start of the program

Remains the best approach.

C Ci iE Y | = E Y | 0,T C SB G ATT

How to randomize?

Randomize program as a whole. E.g. oversubscription: when there are limited supply and excess demand select recipients by lotteries.

Randomize phasing-in Program cannot reach all intended beneficiaries the first year. select first year recipients randomly

Randomize encouragement. Cannot randomize treatment for ethical or practical reasons. Randomly allocate encouragement (e.g. vouchers).

Only increases the probability that a treatment is received without changing it from zero to one specific analytical challenges (partial (or imperfect) compliance).

Natural experiment approaches Use the fact that the program was allocated to some potential beneficiaries

and not to others, for reasons that have nothing to do with the outcome itself. Find variable that is strongly linked to participation (fully or partially)

but not to outcome.

Pipeline comparisons when administrative delays. Compare current participants to prospective participants who also qualify.

Regression discontinuity when program selection based on clear threshold on a given variable. Compare people just before threshold to people just above.

Instrumental variables Use predicted participation as given by a variable linked to participation

but not to outcome

Limitations of These Methods of Impact Analysis

Impact evaluation focuses on program benefits, ignoring costs. Measures one side of cost effectiveness.

This limitation provides motivation for cost studies (Caldés, Coady and Maluccio, 2004)

Methods provide estimates of average impact in a ‘black box’ form. Good for demonstrating impact, but limited for broader policy analysis (Ravallion, 2005)

Controls approaches

Matching: compare people with similar ex-ante observable characteristics Control for the effects of observable characteristics that may affect hh

outcome. Assumption: All components of selection bias are observable and

measured (no omitted variables).

Difference in difference: compare the evolution of the hh with treatment to the evolution of the hh without treatment Neutralize time-invariant individual characteristics (observable and

unobservable). Neutralize effect due to other external events that may have affected

outcome since the program started. Assumption: absent the treatment, the outcomes in the two groups

would have followed parallel trends

Mixed: difference in difference on matched households

SummaryWelfare measure

Before… After…With WithoutWith Without

2

Problem with “before / after” measure

Difference could be driven by other events

Problem with “with / without” measure

Difference could be driven by selection

Double difference 1: “differences in evolution”Impact = (1) – (2). Controls for other events and self selection if the latent heterogeneity is additive and time invariant.

1

Double difference 2: “differences in evolution”Impact = (1’) – (2). Where initial differences are controlled for. E.g. matching and difference in difference

1’

If randomization or natural experiment approach, then original differences should not exist. In such cases, with/without measures can be sufficient

Different Levels of Aggregation: A Common Evaluation Measure

ERR, PRRR

ERR, PRRR

∑ theme level + complementarities –substitution (potential GE effects)

∑ project level + complementarities -substitution

Careful evaluation at this level is the foundation of higher-level evaluations

Economic Rate of Return ERR

Poverty Reduction Rate of Return PRRR

Project Level:- Program Logic Diagram- Impact evaluation- Cash flow analysis

Theme level

Program level

The concept of (stochastic) profit frontier toassure external validity This approach is based on a

simple economic concept: theProduction Possibility Frontier(PPF).

All the possible productioncombinations are found withinthe PPF.

Outside of the boundary are combinations which are notachieveable under currentconditions

The efficient use of resources isalong the boundary.

C

Milkproduction

Cornproduction

ProductionPossibilityFrontier

Land use

Roads

Water bodies

Altitude

Accessibility

Advantages of Micro-Region Typology

44

Typology

Diagnostic from Poverty map

High potential and low average efficiency

Low potential and low average efficiency

High poverty areas High poverty areas

What are the principal differences between high and low efficiency households in the area?

Productive projects differentiated to meet local needs and problems

Conditional Cash Transfers and Nutritional Programs

The inclusion of socioeconomic characteristics and access in the analysis allows for the identification of bottlenecks in areas of high potential but low or medium efficiency

Productive and Efficiency potential based on market, socioeconomic, bio-physical and access characteristics.

Advantages of a micro-region typology: classification

Micro-Regions Poverty Potential EfficiencyCritical, lacking agricultural potential High Low High-Medium-LowMedium priority, no agricultural opportunities Medium Low High-Medium-LowLow priority Low Low High-Medium-LowHigh priority High Medium-High High-Medium-LowMedium priority, with agricultural opportunities Medium Medium-High Medium-LowLow priority, with agricultural opportunities Low Medium-High Medium-lowHigh performance Low Medium-High High

Estimation Methodology

Potencial: Precios de productos (P) y insumos (W), beneficios reportados por el hogar (π).

Eficiencia: tierra, valor de los activos, características socioeconómicas (Z), condiciones biofísicas (G), acceso a mercado (A).

INSUMOS PARA LA ESTIMACION

Econometría Modelo de fronteras

estocásticas de beneficios

ESTIMACION OUTPUT DE LA ESTIMACIÓN

Pesos asignados a los insumos de acuerdo a teoría económica y evidencia empírica

PASO 2: PREDICCIÓN (NIVEL REGIONAL)

Resultado de la estimación (pesos)

Frontera: Precios de productos (P) y insumos (W).

Eficiencia: tierra, valor de activos características socioeconomicas (Z), condiciones biofísicas (G), acceso a mercado (A).

INSUMOS PARA LA PREDICCION

Potencial productive a nivel regional; eficiencia

de acuerdo a las características

socioeconómicas, condiciones biofísicas,

acceso a mercado dentro del área

PASO 1: ESTIMACION (NIVEL DEL HOGAR)

RESULTADO DE LA PREDICCION

Potencial productive y eficiencia a

nivel regional

Pesos

RESULTADO FINAL

PotentialPrices of products (P) and inputs (W), profits reported by household (π).

Efficiency Land, value of activities, socioeconomic characteristics (Z), biophysical conditions (G), market access (A).

Econometric Model of the

stochasticProfit frontier

Weights assigned to inputs following

economic theory and empirical evidence

Estimation results (weights)

Boundary Product prices (P) and inputs (W)

Efficiency: Land, value of activities, socioeconomic characteristics (Z), biophysical conditions (G), market access (A).

Productive potential at the regional level;

Efficiency according to socioeconomic characteristics, biophysical conditions, market access within the area

Region level productive

potential and efficiency

Estimation inputs Estimation Estimation output

Final result Prediction result Prediction inputs

Step 1:Estimation(Household Level)

Step 2:Prediction(Regional Level)

Recap (1)… Data

Available datasets:Land characteristics, biophysical conditions, socioeconomic characteristics, assets, market access, etc.

0.1

.2.3

.4D

ensi

ty

4 6 8 10 12values X

Group 1Group 2

Variable X

.7.8

.91

Cum

ulat

ive

Den

sity

0 2 4 6 8 10Values Z

Group 1Group 2

Variable Z

PPF:Input, Output, Profits

Geo Layers

Targeting Criteria based on Efficiency

Estimated cost of Market Access

Agricultural Profit Frontier

Efficiency in Agricultural Profits

Recap (2)…E

ffici

ency

Allo

catio

n C

riter

iaE

quity

Allo

catio

n C

riter

ion

Typology combines all these criteria

MULTIPLE TARGETING DIMENSIONS

49

Recap (3)…

Recall the initial objective….



Micro-Regions

Critical, lacking agricultural potential

Medium priority, no agricultural opportunities

Low priority

High priority

Medium priority, with agricultural opportunities

Low priority, with agricultural opportunities

High performance

Recap (4): Grouping diverse criteria into

seven microregions…

Recap (5)… How does this translate into policies?


What are the principal differences between high and low efficiency households in the area?

Productive projects differentiated to meet local needs and problems

Recap (6)… How does this translate into policies?


Conditional Cash Transfers and Nutritional Programs

Can be applied to other settings? Guatemala

Poverty Map Efficiency in Agricultural Profits

Cost of Market Access

Agricultural Profit Frontier

Guatemala: Seven-Class Typology

Without agricultural potential

With agricultural potential

Some examples Extension services

Market information

Infrastructure in rural areas

Property rights – land titling

Diff-in-Diff, FE (Dercon et al 2008) Impact of road quality improvements and increased access to

agricultural extension services on consumption and poverty in rural Ethiopia.

Dependent variables: household is poor, consumption growth Treatment: receiving at least one extension visit, and access to all-

weather roads (=1 if road to nearest town is all-weather road) Identification: IV model using GMM and controlling for household fixed

effects Instrument for consumption in time t-p: fertile land holdings, number

of adult equivalents and number of livestock units (all in logs) at time t-p.

Receiving at least one extension visit reduces headcount poverty by 10 percentage points and increases consumption growth by 7 percent. Access to all-weather roads reduces poverty by 6.9 percentage points and increases consumption growth by 16.3 percent.

Ex post Supply driven


Market information



Institutional arrangement for a simple price information system

Source: Hernanini (2007), World Bank

Flow of information and Institutional agreements for virtual markets

Source: Hernanini (2007), World Bank

Due partly to costly information, price dispersion across markets is common in developed and developing countries

Between 2001 and 2006, cell phone service was phased in throughout Niger, providing an alternative and cheaper search technology to grain traders and other market actors

The author constructs a novel theoretical model of sequential search, in which traders engage in optimal search for the maximum sales price, net transport costs

The model predicts that cell phones will increase traders’ reservation sales prices and the number of markets over which they search, leading to a reduction in price dispersion across markets.

To test the predictions of the theoretical model, they use a unique market and trader dataset from Niger that combines data on prices, transport costs, rainfall and grain production with cell phone access and trader behavior.

Pseudo randomized IV - Mobile phones: the impact of Cell phones on grain Markets in Nigeria (Jenny C. Aker - 2008)

The results provide evidence that cell phones reduce grain price dispersion across markets by a minimum of 6.4 percent and reduce intra-annual price variation by 10 percent

Cell phones have a greater impact on price dispersion for market pairs that are farther away, and for those with lower road quality. This effect becomes larger as a higher percentage of markets have cell phone coverage.

They provide empirical evidence in support of specific mechanisms that partially explain the impact of cell phones on market performance.

The primary mechanism by which cell phones affect market-level outcomes appears to be a reduction in search costs, as grain traders operating in markets with cell phone coverage search over a greater number of markets and sell in more markets.

The results suggest that cell phones improved consumer and trader welfare in Niger, perhaps averting an even worse outcome during the 2005 food crisis.

Main results

Beginning in October 2000, it set up 1700 internet kiosks and 45 warehouses in Madhya Pradesh that provide wholesale price information and an alternative marketing channel to soybean farmers in the state

Dependent variables: wholesale price of soybeans, sales in traditional markets, soybean cultivation

Treatment: presence of internet kiosks and price warehouses

Identification: variation in timing of the introduction of kiosks and warehouses

Equivalent to randomization at the village level Ex post Demand driven

Pseudo randomized IV: Internet: Internet kiosks in India to provide wholesale price information (Aparajita Goyal, 2008)

The estimates suggest an immediate and significant increase in the monthly wholesale market price of soybeans by 1-5 percent after the introduction of kiosks, lending support to the predictions of the theoretical model

While the presence of warehouses appears to have no effect on price, warehouses are associated with a dramatic reduction in the volume of sales in the traditional markets

Moreover, there is a significant increase in the area under soy cultivation. The estimates are robust to disaggregated measures of treatment and comparisons with alternative crops grown in the same season as soy

The results suggest that information can enhance the functioning of rural markets by making buyers more competitive.

Main results


Market information



Pipeline comparisons when administrative delays (Torero 2008)

The road to be improved was split in the following segments:

A B C D ETable 1. Timeline

Section

Scheduled Start Date

Scheduled End Date

A July 2008 June 2010

B Completed Completed

C July 2008 December 2009

D October 2008 April 2010

E October 2008 February 2010

Based on the geographic location and the timelines, the following treatment-control groups are suggested:

Test Number

Control Group

Treatment Group

1 B A

2 B C

3 D C

4 E D

Table 2. Treatment and Control Groups

Pipeline comparisons when administrative delays

Pipeline comparisons when administrative delays.

Randomized- Barriers to connection in Ethiopia (Bernard and Torero 2009)

Connection fees range between USD 50 and USD 150 (drop down line and meter). Need to find ways to facilitate connection for the poorer.

Can CFL (energy-saving light bulb) positively influence energy use? How to promote the use of energy-saving light bulbs (consumes 4 times less, but costs 8 times more)?

What is best: 2 years loan or 5 years loan for connection fee?

Pilot study on 20 towns to assess optimal subsidies. Experimental approach (randomize encouragement through distribution of vouchers).

This image cannot currently be displayed.

Public distribution

Public distribution

Random selection…


Market information



Matching, IV on cross-sectional data - Land property rights on productivity (- Markussen 2008)

Dependent Variable: (log) value of output per hectare

Treatment: The plot is held with a paper documenting ownership (titles, application receipts)

Identification: IV mode of plot acquisition (dummies to indicate if the plot was given by the State, inherited, bought, donated, occupied for free) as instrument for the dummy “plot held with paper”

Ex-post Demand driven: households and landholders

apply for titles

Pseudo-Randomized - Land titling on rural households (Torero and Field 2007)

Dependent variables: household expenditure, change in rent/market value of dwelling, risk of expropriation, production, trade of land, collateral and credit markets, land ownership and tenancy, permanent ant transitory crops

Treatment: to receive land title Identification: quasi random program

implementation, kernel matching Ex post Demand driven, but few requirements and

virtually free

The survey covered 3204Peruvian rural households:521 from rural coast, 1622from rural highlands and1061 form rural jungle.

The next map plots thetowns covered by the surveyand the valleys reached bythe PETT program. Fromthese 3204 households 1793match with at least oneprevious national survey.

The Database

If control and treatment groups are randomly selected from a population then there is no bias in the initial characteristics

The impact over income can be attributed to the access to title

Population Random selection


X)

Control group


Y Exp – Y Control

¿Why we can not then apply a direct comparison between the control and treatment group? Because differences in characteristics of subjects, or what is called selection bias.

Population Pseudo random

selection

Quintile I (more poor)

Quintile II Quintile III Quintile IV QuintileV (more richer)

Because of initial differences between

both groups, the effects of the treatment can not be identified by directly comparing the groups


X)

Control group


Identify comparable pairs (with similar initial characteristics) and that differ only on the procedure

We will use Propensity score matching.

Population pseudo random

selection

Find the pair to assure comparability Treatment

Control

Impact of the

procedure

Final comments The impact evaluation must be a part of the program

design It is very important to identify how to incorporate it

now that the program already exists For new programs it is necessary to invest in the

design so that an impact evaluation is also part of it

It is essential to identify the impact pathways, i.e. the expected causal chain of events leading from project activities to outputs, to changes in the target population, and to the achievement of project objectives

Since the beginning of the program is necessary to specify the expected “outcomes” and the control group

Final comments It will be ideal to have an autonomous and external

laboratory of impact evaluation

Communications among all stakeholders is central

Not all interventions need to be evaluated, it will be ideal to do it before scaling up so there is assurance that the intervention works

Alignment of proper incentives – to contractors, evaluators, to implementers and to USAID country offices

Finally, policy requires a causal model; “without it, we cannot understand the welfare consequences of a policy” (Deaton 2009)

Recommended readingsCaldés, Natalia, David Coady and John Maluccio. 2004. The Cost of Poverty

Alleviation Transfer Programs: A Comparative Analysis of Three Programs in Latin America. IFPRI FCND Discussion Paper No. 174., Washington, DC.

Duflo, Esther; Rachel Glennerster and Michael Kremer (2007):“Using Randomization in Development Economics Research: a Toolkit”CEPRdiscussion paper no. 6059

Feder et al. 2004. Review of Agricultural Economics.

Godtland et al. 2004. Economic Development and Cultural Change.

Heckman, J.J., H. Ichimura, and P.E. Todd. 1997. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program.” Review of Economic Studies 64:605-654.

Hirano and Imbens. 2004. The Propensity Score with Continuous Treatments. In Gelman & Meng, eds.

Miguel and Kremer. 2004. Econometrica.

Ravallion, Martin. 2005. Evaluating Anti-poverty Programs. World Bank Working Paper Series 3625, Washington, DC. Martin Ravallion (2003): “The Mystery of the Vanishing Benefits: An introduction to Impact Evaluation” The World Bank Economic Review, volume 15, no 1, pp115-140

How we can use impact evaluation to assure effective use of resources for development

Technology

Transcript of How we can use impact evaluation to assure effective use of resources for development