Institute for Development Studies International Workshop

March 26-27, 2013

Impact, Innovation and Learning Towards a Research and Practice Agenda for the Future

Are development evaluators fighting the last war?

Robert Picciotto King's College London

“Evaluation cannot hope for perfect objectivity but neither does this mean that it should slump into

rampant subjectivity” Ray Pawson

1

The story line • The past is not always prologue

• The impact evaluation debate is over

• Experimental methods have formidable advantages… and disadvantages

• Complexity/system thinking: a theory?

• A new development evaluation agenda: values; human well being; new development architecture; evaluation beyond aid

2

The risks of “path dependence” • Past decisions lose their relevance, distort

today’s decisions or impose limits on them: – What worked yesterday may not work tomorrow

– What did not work yesterday may, with luck, work tomorrow

• Summative evaluation does not necessarily translate into formative evaluation: good evaluators exercise caution when offering recommendations

• Path dependence also affects evaluation models and approaches – this is why we are here!!!

3

The impact evaluation debate is over

• Methods have to do with tactics

• Even the best tactics fail without a strategy

• Attribution analysis of discrete interventions has dominated recent debates

• The Stern gang has produced a definitive “state of the art” review: many questions beyond attribution need to be tackled by development evaluators

• It is time to move on...

4

The potential of randomisation

• Experimental methods have formidable advantages: they can help establish causality by providing a plausible measure of what results would have been observed had the intervention not taken place

• This approach is ideal for tackling selection bias and addresses the counterfactual dilemma by ensuring that all the other factors that may affect outcomes are identical except for stochastic errors

• The statistical significance of findings can be measured by statistical testing techniques

5

Society is not a laboratory • Redundancy or feasibility issues

• External validity: context matters

• Failure to address the distinctive accountabilities of partners, replication, implementation (who? Why? How ?)

• Ethical issues

• Experiments are costly, require superior skills, large studies, large samples and specialized quality assurance

6

Qualitative approaches contribute to evaluation rigour

• Without theory there is no credible causal explanation that adds to development knowledge

• Understanding of causal relationships calls for substantive knowledge of the intervention and its context

• Qualitative methods involve participation, observation, text based information, village meetings, open ended interviews, etc.

• Quality can and should be quantified. • It is also important to figure out why things

did not happen

7

Mixed methods together with (or

beyond) the experiment – Regression and factor analysis

– Quasi-experimental designs

– Multivariate statistical modelling

– Participatory approaches

– Qualitative impact assessment

– Beneficiaries surveys

– General elimination methodology

– Expert panels

– Benchmarking

– Case studies

8

Complexity models and systems

thinking: not yet a theory? • Pluralism and inclusivity in methods

• Networks as privileged units of account

• Interrelationships

• Engagement with multiple actors

• Symbols and diagrams to guide evaluative inquiry

• From ‘attribution’ to ‘contribution'.

• Rely selectively on all evaluation traditions – theories of change, realist and critical evaluation, democratic evaluation, etc.

9

Towards a new development

evaluation agenda A relevant development evaluation

strategy for our troubled times should:

put values at the very centre of the strategy

reflect the new development agenda likely to emerge beyond the MDGs

reconsider the ultimate goals of development cooperation.

10

Why should values come first?

• The social arrangements that allow a few individuals to accumulate enormous wealth while depriving large segments of the population of basic necessities have spawned widespread popular anger and resentment

• The richest 0.5% hold well over a third of the world's wealth while 68% share only 4%

• Differences in income attributable to effort, skill or entrepreneurship are not widely resented – unless the rules of the game are rigged

• Evaluators should examine these rules! 11

Evaluators can no longer sit on the

fence

• Current utilization driven evaluation models are client-controlled and tend to be value free

• Evaluators should be independent and seek inspiration from the ethical imperatives of social justice

• They should master recent policy research findings about inequality

• They should revise their guiding principles and metrics in ways consistent with the pursuit of more equitable, inclusive and environmentally sustainable growth

• They should give higher priority to the political dimensions of policies and programs

12

We need new metrics • GNI measures fail to capture services provided

within the household or to reflect environmental losses, social inequities and human insecurities

• Five planets would be needed to get the world up to US living standards

• The spectre of global warming is haunting the poorest and most vulnerable countries

• The reality of poverty encompasses factors that reach well beyond material deprivation

• The profile of risk management should rise given the rise of national disasters and the insecurities of the interconnected global system

13

Towards a 3D (McGregor/Sumner) evaluation model

14

Evaluation characteristics

Material well-being

Relational well-being

Perceptual well-being

Major discipline Economics Sociology Psychology

Dominant evaluation

approach

Cost benefit

analysis

Participatory

evaluation

Empowerment evaluation

Investment focus Physical capital Social capital Human capital

Main unit of account Countries Communities Individuals

Main types of

indicators

Socio-economic Resilience Quality of life

Taking account of a new aid

architecture • Aid is once again an instrument of geopolitics and

new public and private actors have joined the fray

• Aid is increasingly channelled through large, diffuse and adaptable country based programmes or through vertical multi-country initiatives involving multiple actors.

• Aid coalitions are increasingly operating across borders to tackle “problems without passport”

• The complex political equations that underlie their operations cannot be solved through abstract, politically neutral approaches

15

Evaluating development cooperation beyond aid

The footprint of development cooperation is far more significant than the aid footprint:

• Exports (about $5.8 trillion)- 45 times official aid flows.

• Remittances ($283 billion) - 2.2 times

• FDI ($594 billion) - 4.6 times as large

• Royalty and licence fees($27 billion) - a fourth

• Huge damage caused by climate change as a result of OECD countries’ energy policies

16

Conclusions

• Even the best methods cannot compensate for a misguided strategy

• The new development research and evaluation agenda: • emphasize ethical standards and democratic

values

• reflect the new human well being paradigm

• come to terms with the new aid architecture

• reach well beyond aid

• experiment with complexity and systems thinking

• promote independence in evaluation processes

• devote more attention to the political dimension

17