Impact Evaluation for policy-making

Promoting uptake of

impact evaluation

findings: the importance

of relevance, utility and

engagement

What 3ie is learning

Beryl Leach, deputy director, 3ie

18 February 2015

Istanbul, Turkey

Overview

• Who we are and how we promote high-quality,

relevant and useful studies that are taken up

and used

• What we are learning about impact evaluation

and policy influence

• How think tanks can produce, engage and use

impact evaluation in policy advocacy

What is 3ie?• International grant-making NGO founded in 2008

• Filling evidence gaps in what we know works,

why, how and the costs in development

• We fund particular types of evidence production

– high-quality experimental and quasi-experimental,

theory-based, mixed-method impact evaluations

– systematic reviews

– evidence syntheses

Why are IEs important for policy-making?

Policy relevance and usefulness

• Evidence that can improve the

effectiveness of development

policies and programmes

• Must be policy relevant

• We expect policy influence and

impact on policy and programming

because of how we approach

design and implementation

What do we mean by useful?

RELEVANT: Helps answer a

specific policy question

CONTEXTUAL: Appropriate

political context—it makes

sense

CLEAR: message gives

options

FEASIBLE: Affordable and

possible

TIMELY: interest exists

3ie funds high-quality policy-relevant studies

3ie encourages researchers to engage with key stakeholders

Uptake of study findings and improved policies and practice

Applications

- Ask policy

relevant

question(s)

- Have potential

for policy impact

- Study team has

experience in

policy influence

ASSUMPTIONS

Researchers

-Committed

- Understand how

policy influence

happens

- Have the tools

and resources to

invest in policy

engagement

- Decision-makers

are interested

Study

-Makes policy-

relevant

recommendations

– Answers main

IE questions

- Proposes

feasible

solutions

- Had ongoing

engagement

ASSUMPTIONS ASSUMPTIONS

What we learned from analysing early

influencing in our grants

• 3ie needed to be directly engaging with

researchers and much earlier

• Researchers needed to be engaging with

implementers much earlier and differently

• Paper-driven processes were not effective

• 3ie approach needed to be based on evidence

about policy change and research uptake

Main ways we are ensuring policy

relevance and usefulness

• Early and direct engagement with implementing agencies

during the development of funding windows

• Require country nationals on the research teams in

substantive roles; more funding is for country-based

teams – role and opportunity for think tanks

• Preparation phase

• Direct engagement with researchers about PIPs from the

start and ongoing dialogue with the team

Communicating impact evaluation

evidence: ongoing, integrated, multi-level

• Engage from the start

• Explain the study and why it will be useful--build interest

• Report preliminary findings-get feedback-promotes

ownership

• Engage a range of key actors: beneficiaries and other

local actors, civil society, media from early phases

• Translate into plain language and produce in a variety of

formats and disseminate through multiple channels:– Meetings, conferences, social media, multiple briefs, papers, reports

Think

Tanks

Impact evaluation and

think tank policy

influencing: opportunities

and constraints

Bridging researchers & policymakers

Translating, relationship building, networking

Using impact evaluation evidence for

policy advocacy: continued

• Constraints

• Two-sided coin– Rigor as the basis for being high quality

and for sound decision-making

– IEs seek to measure causal

relationships, we need to be able to

assess validity

• Two-part problem– Poor or weak designs mean you can’t

really know which cause

– Users have limited critical appraisal

capacity

Using impact evaluations for advocacy

• How much can you generalise based on one IE study?– Not much, even though we stress generalisability a lot

– More than IE evidence needed to know about potential for scaling up

– Researchers should always bring knowledge of existing evidence to the

analysis, so that the new evidence is situated in the existing body of

evidence, doesn’t always happen

• Evidence for programme change can be more

immediately useful than for policy change– Especially the case with theory-based evaluation, which is what we

require

– Modification to existing programme theory of change and operations

Benefits of increased production and use

of impact evaluations by think tanks• IEs particularly need a strong emphasis on policy relevance and

utility that think tanks can provide

• More think tanks doing quality IEs and being effective

intermediaries in policymaking will help increase uptake

• Participating in IE teams and commissioning can strengthen

relevance and utility

• Big potential for improving think tank understanding of IE evidence

and critical appraisal to be knowledge translators in advocacy

process

Thank you

www.3ieimpact.org#3ieNews #TTIX2015

END OF TTIX PRESENTATION

What are we looking at next?

• Developing our approaches to working with implementers

• Doing case studies on the cumulative impact of IEs done

over time: direct and indirect contributions to what types of

change

• Evaluating evidence uptake from 3ie studies– Finding an appropriate method and framework

– Exploring QCA

– Why we probably will not be doing IEs of policy influence

– Why we will look at synthesising evidence

Preparation phase-why is it so important

• Implementing agencies do not necessarily understand

impact evaluations

• Researchers do not necessarily speak the implementers’

language or understand their evidence needs well enough

• Fund working together

• Monitor through direct contact

• Have inception workshops

Policy influencing using impact

evaluations• Evidence does not even play a major role in most

decision-making

• Research uptake is a political process, not a technical one

• Researchers are vital to translating and building trust and

credibility

• Single study evidence most often does not result in major

policy change, nor should it

• 3ie does not advocate for wider change based on single

studies, nor should you

Limitations

• Expensive

• Work best with large

programmes

• Highly specialised

methods

• Demands of

counterfactuals

• Need to translate for policy

and programming uptake

Prevailing view of policy engagement in

the IE community

Get evidenceDisseminate

as reports and papers

Change happens

Assumption: You need the evidence

before you engage and promote uptake

3ie funds high-quality policy

relevant studies

Applications

- Ask policy relevant

question(s)

- Have potential for

policy impact

- Study team has

demonstrated

experience in

policy influence

ASSUMPTIONS

Increased scoring for policy aspects

National researchers on team

Preparation phase

How we

support

3ie requires stakeholder engagement

Researchers

-Committed

- Understand how

policy influence

happens

- Have the tools

and resources to

invest in policy

engagement

- Decision-makers

interested

ASSUMPTIONS

Inception workshop includes implementing agencies

Identify useful evaluable questions

Develop a policy influence plan

Earmark study budget

How we

support

Uptake of study

findings and

improved policies

and practice

Study

-Makes policy-

relevant

recommendations

- Answers what

works and why

- Proposes

feasible solutions

- Has had ongoing

engagement

ASSUMPTIONS Ongoing engagement with 3ie as part of grant monitoring– graduating to a more dynamic model of interaction

Intensive review and feedback of study reports by internal and external reviewers

How we

support

Using impact evaluation evidence for

policy advocacy: 3ie perspective

• Benefits

– Policymakers can understand numbers

– Talking in effect size and costs can be

compelling

– Policymakers often are looking for cost

effectiveness evidence

– Programme managers want the IE and

are invested in the questions and want

to be able to act on findings

– They have credibility in today’s climate