Evaluation and impact assessment David Souter. BCO Impact Assessment session, Bern, September 2007...

Evaluation and impact assessment

David Souter

BCO Impact Assessment session, Bern, September 2007

Why evaluate? It is dangerous to assume that what has been

decided will be achieved …… or that what happens is what was intended.

Accountability : to report to stakeholders on achievements against objectives

Learning : to improve practice in future, particularly programme design and implementation

Promotion : to demonstrate success to potential donors


Why impact assessment? “The purpose of most impact assessments is to

demonstrate past impact and to improve future practice, and there may be tension between the two.”

“Proof of effectiveness is not the main goal of impact assessment: it is concerned more with understanding, and learning from, the processes and approaches that have led to achieving a particular impact so that such knowledge and experience could be used in comparable situations.”


A hierarchy of outcomes : the MDG perception


Output, outcome/effect and impact

Output measures the extent to which an activity’s inputs have been made and services deployed

Outcome (or effect) measures the use of inputs/services and the short-term results arising from these

Impact measures the long-term changes in people’s lives, social behaviour and economic and political structures that can be attributed – in whole or part – to the activity


Activities

A hierarchy of complexity

Inputs

Outputs

Outcomes or

Effects

Impact

Increasing influence of context (social, economic, political, etc.)

Source: Roche


An example:


What is impact assessment? “Impact assessment refers to an evaluation of

how, and to what extent, development interventions cause sustainable changes in living conditions and behaviour of beneficiaries and the differential effects of these on women and men.

“Impact assessment also refers to an evaluation of how, and the extent to which, development interventions influence the socio-economic and political situation in a country.”

Peter Oakley et al., Outcomes and Impact


What is impact assessment?

“Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions.”

Novib/Oxfam research project, reported in C. Roche, Impact Assessment for Development Agencies


What is impact assessment? Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions.


Why assess impact? Impact assessment “is concerned … with

understanding, and learning from, the processes and approaches that have led to achieving a particular impact so that such knowledge and experience could be used in comparable situations.”

Impact assessment should both: look back at experience to date look forward to improving programme design and

implementation, and in helping to predict future impact


How do we assess impact? : a view

Stages of M&E development Degree of attention

Design Extensive

Implementation Substantial

Data generation More

Data interpretation Less

Use Some

Evaluation Minimal

R. Davies, assessment of Bangladesh projects, 1995


Causation is complex


Impact is long-term


The challenges of impact assessment A two tier process

The challenge of context

The challenge of the baseline

The challenge of understanding change

The challenge of time

The challenge of attribution

The challenge of aggregation

The challenge of unexpected impact

The challenge of stakeholders

The challenge of perception

The challenge of distribution


Two tiers in establishing impact

that it has had an effect on the behaviour of actors relating to the intervention itself

that this effect has led to change in the circumstances of those actors

that it has had an effect on policy

that the policy change concerned has resulted in change in people’s lives

To establish that an intervention has had impact, it is necessary to establish two things:

Where advocacy initiatives are concerned, this can be described as follows:


The challenge of context Impact measures change in context Understanding context is therefore

essentially to identifying impact Context is complex:

social, gender, cultural, political, economic, environmental etc. – and technological

contextual factors inter-relate Context is changing Context can be disrupted


The challenge of the baseline ‘… without some sort of baseline, it is not really possible to

assess impact.’ ‘… unless detailed and continuous M&E takes place at

[programme and project] level, it will be impossible to build up the body of knowledge which is necessary to understand the wider outcomes.”

Measuring change requires understanding of the starting point from which change is measured

Most impact assessment suffers from the lack of baseline data

Impact assessment needs to be built into processes from the beginning stage if it is to be undertaken meaningfully

Measures can be taken to try to recover baseline, but these are difficult and relatively weak


The challenge of change Impact assessment is essentially the

measurement of change Change – particularly social change – results from

a large number of diverse and differently inter-related factors

It is subject to unexpected externalities It may be ‘sudden, discontinuous and unpredictable’ Chance is a significant factor in change processes

Understanding change requires an in-depth understanding of: context and the baseline from which change is to be measured


The challenge of time Lasting change cannot be established quickly. Short-term impact assessments can identify

near-project effects, rather than lasting change Short-term assessments also lack perspective

regarding the complexity of change Impact varies over time:

short-term impacts can be few and differ from long-term impacts

and vice versa


The challenge of attribution Just because change occurs, it does not

mean you caused it. Many factors influence change: many people

may wish to claim credit. Attribution of change is complex. There is a temptation to claim credit in order

to emphasise the value of work done … … but this does not help to understand what

has been (or not been) achieved, and so does not contribute to the quality of future

work


The challenge of aggregation Individual interventions usually target

specific aspects of change Other interventions target similar and

related aspects of change Change may result from the cumulative

impact of numerous interventions “the sum is more than the total of the parts” implying that it is the “sum” rather than the

“parts” that drives the impact


The challenge of unexpected impact All initiatives have unexpected impacts, e.g.

diversion of resources changes in power relationships or the distribution of wealth redistribution of income to other purposes

which may be positive or negative It is as important to understand these unexpected

impacts as to understand whether intended impacts are achieved

Negative impacts tend to be under-reported, with the result that important lessons are not learnt for future programme design and implementation


The challenge of identifying stakeholders (illustrative)

Donors

Non-bene-ficiaries

Local inter-mediaries

Politicalauthorities

Otherdevelopment

initiatives

Projectmanagers

Target beneficiaries

Age group

Education Caste orclass

Land-holding

Wealth

Gender

Level of use


The challenge of perceptions Different stakeholders have different perceptions of impact.

In particular:

target beneficiaries may perceive impact differently from project managers

target beneficiaries are likely to focus on individual, family and community benefits

project managers are likely to have broader objectives for socio-economic or political change

non-beneficiaries are likely to perceive change differently from beneficiaries

Enabling different stakeholders to express divergent views of impact is crucial to successful impact assessment


The challenge of distribution Impacts affect non-target beneficiaries as well as target

beneficiaries Need to establish whether impact is favourable to target

beneficiaries and also if it is favourable to target beneficiaries and

neutral otherwise favourable to the community in general unfavourable to others or to the community in general favourable to target beneficiaries at the expense of others in

the community or neighbouring communities

Non-target beneficiaries may be reticent about negative impacts on themselves e.g. women may be reticent about changes in the balance of

power within households


Impact assessment and advocacy

The impact of advocacy work requires both: that advocacy leads to policy change and that policy change leads to changes in people’s lives

Advocacy work requires a wide range of interactions, many of which will be unproductive

Much advocacy work is long-term and may have few short-term effects

Much advocacy work is undertaken through networks in which different actors play different roles

Impact assessment requires particular attention to aggregation issues


Oxfam’s hierarchy of advocacy impact

Heightened awareness

Contribution to debate

Change in opinions

Change in policy

Implementation of change in policy

Positive change in people’s lives


Impact assessment as process

Diagnostics

where are we now?

Strategydesign

where do wewant to go?

Development and

implementation

how do we get there?

Monitoring and

evaluation

how do we know we are getting there?

Impact assessment needs to take place throughout the project cycle


The impact assessment cycle Understanding of context (diagnostics) Development of baseline and indicators, which

should be: achievable and adequate capable of providing data re inputs, outputs, effects and

impact Monitoring of indicators

including regular assessment (and storage) Analysis Interpretation Input into subsequent project cycles


Design of IA processes Establishment of:

context within assessment is to be made (impact where

scope and scale of assessed intervention (impact of what)

level and target (impact on whom, at individual, household or community level, etc.)

timescale (impact when) interaction with other influences/factors selection of illustrative examples


IA process requirements Commitment to learn from experience Adequate baseline data and indicators A management framework which

recognises real levels of resources Continuous engagement and continued

commitment Evaluative rigour Determination to complete the process Honesty in interpreting results


Impact assessment of ICD projects Impact assessment of ICD projects does not

obviously differ in principle from impact assessment in other development areas It should therefore gain from sharing experience with the

wider development community However, there are some aspects of ICD projects

which would seem to exacerbate the challenges of impact assessment: the lack of baseline data the very limited range of existing substantive impact

assessment the unpredictability of change in ICT technology and

behaviour uncertainty about the implications of different policy options


Ways forward There is a good deal of experience of

impact assessment in other development sectors on which to draw

This experience emphasises: development of baselines continuous monitoring and assessment stakeholder participation exploration of unexpected and negative

outcomes openness to diverse conclusions


Ways forward “Lessons are more likely to be about the

relative merits of various strategies in addressing specific audiences, as well as different sequences of tools and methods, than about developing a single model of change or assessment.”

C. Roche, Impact Assessment for Development Agencies


The opportunity for BCO The BCO Impact Assessment offers BCO

partners an opportunity to: learn from the experience in other

development sectors apply that experience to their own experience

with ICD projects develop approaches to impact assessment

which have specific relevance to ICD, for their own future use and the wider ICD community

Evaluation and impact assessment David Souter. BCO Impact Assessment session, Bern, September 2007...

Documents

Transcript of Evaluation and impact assessment David Souter. BCO Impact Assessment session, Bern, September 2007...