Participatory approaches to evaluation

Puja MylesPuja.myles@nottingham.ac.uk

Session outline

• What is Participatory evaluation?• Philosophical underpinnings • Barriers to participatory approaches• Participatory Action Research (PAR)• Most Significant Change technique (MSC)

Participatory evaluation-1

“Participatory evaluation is a partnership approach to evaluation in which stakeholders actively engage in developing the evaluation and all phases of its implementation”

-Zukoski and Luluquisen (2002)

Participatory evaluation-2

“Participatory monitoring and evaluation is not just a matter of using participatory techniques within a conventional monitoring and evaluation setting. It is about radically rethinking who initiates and undertakes the process, and who learns or benefits from the findings.”

-Institute of Development Studies (1998)

Participatory evaluation-3

“Participatory evaluation means more than just a way of seeing how much has been achieved or produced at what cost and with what effect…Participatory evaluation is not just to do with the development of things. It is to do particularly with the development of people.”

-Feuerstein (1986)

Historical and Philosophical basis

“The basic ideology of PAR is that self-conscious people, those who are currently poor and oppressed, will progressively transform their environment by their own praxis. In this process others may play a catalytic and supportive role but will not dominate"

-Fals-Borda, 1991

Freirean approach to community development

• Paulo Freire (Pedagogy of the Oppressed)• An educational philosophy that actively

involves the poor in critically analysing their social situation (conscientisation), thus creating the potential for them to transform their environment.

• Praxis: reflection-action dynamic

Group reflection

• How participatory should participatory evaluation be?

• What are the advantages of using participatory approaches to evaluation?

• Can a participatory evaluation replace traditional evaluation methods?

• Are participatory approaches to evaluation valid?

• What are the barriers to participatory approaches?

Participatory Action Research (PAR)

“Put simply, action research is ‘learning by doing’ - a group of people identify a problem, do something to resolve it, see how successful their efforts were, and if not satisfied, try again.”

-Rory O’Brien (1998)

The action research process

• Identify problem• Collect data for detailed diagnosis• Collective postulation of possible solutions• Agree and implement an action plan• Collect data on results of intervention• Analyse data• Interpret findings in relation to success of

action• Re-assess problem• Begin another PAR cycle

Principles of action research (Winter 1989)

• Reflexive critique• Dialectical critique• Collaborative resource• Risk• Plural structure• Theory, practice, transformation:

theory informs practice and practice refines theory

Application of PAR in health

• Health Needs Assessment• Planning health care services• Service evaluation• As a process of empowerment• Can employ both qualitative and

quantitative methods• Qualitative tools include: journals,

interviews, focus groups, filming, use of pictures or drawings

• Quantitative tools: surveys, monitoring data

RAP, RRA, PRA, PLA

• Rapid Assessment Procedure (RAP)• Rapid Rural Appraisal (RRA)• Participatory Rural Appraisal (PRA)• Participatory Learning and Action

(PLA)

RAP techniques for data collection

• Proportional piling: visual form of estimating proportions.

• Community maps: Drawn in groups; Useful to learn about an area and how different groups use it

• Transect walk: observation technique• Diagramming: Venn (chapati) diagrams

to represent power structures (power represented by size; relationships by overlap; exclusion by distance)

Most Significant Change

Aka ‘Monitoring-without-indicators’, ‘the story approach’, ‘evolutionary approach

to organisational learning’…

The kernel of the MSC process

• A question:“Looking back over the last month, what do you think was the most significant change in [particular domain of change]?”

• A second question:“From among all the significant changes, what do you think was the most significant change of all?”

Steps

• Start and raise interest• Define domains of change (e.g. quality of life, health,

participation/empowerment)• Define reporting period• Collect significant change (SC) stories• Select the most significant of the stories (hierarchical

level)• Feed back results of the selection process• Verification of stories• Quantification• Secondary analysis and meta-monitoring• Revising the system

When to use MSC

• Complex programmes with diverse and unexpected outcomes

• Focussed on social change• Participatory in ethos• Highly customised services• Focus on learning rather than just

accountability• Enablers: organisational culture of ‘no-

blame’; willingness to try something different; commitment by senior managers

Defining domains of change

• Domains are broad and often fuzzy categories of possible SC stories

• A domain of change is not an indicator (deliberately fuzzy as opposed to SMART)

• Include an ‘open window’ domain• Could include a domain for negative

changes ‘lessons learned’ or ‘changes that reflect an area to improve’

• Could be based on programme objectives• Between 3-5 domains are best

Collecting the SC stories

• Information to be documented includes:• Who collected the story and where the

events occurred• Description of the story itself- what

happened (sequence of events)• Significance (to the storyteller) of the

events described in the story• Stories could be 1-2 pages long

Selecting the most significant of stories

• Hierarchy of selection processes• Criteria for selecting stories: Majority

vote, iterative voting, scoring• Documenting the results of the selection

process: the reasons for selecting an SC story as the most significant should be documented and attached to the story

Secondary analysis and meta-monitoring

For all collected stories:• Thematic coding• Look for positive and negative changes• Analyse changes mentioned in MSC

stories against a logic model• Analyse genre• Analyse differences between selected

stories and those not selected

What can MSC offer as an evaluation technique?

• Uses an inductive approach to help identify unexpected outcomes

• Helps monitor messy impacts, intangible and indirect consequences

• The hierarchical selection process of sorting through diverse SC stories becomes an opportunity for making implicit values explicit, communicating shared values and agreeing a direction for the project

Validity and voice in MSC

Thick description: richness of detail and context

Systematic process of selection (compare to case studies)

Transparency (systematically records interpretation)

VerificationParticipation in keeping with the broader

programme philosophy of empowermentPurposeful sampling

Further reading

1. Davies, R. and Dart, J. (2005).The ‘Most Significant Change’ (MSC) Technique: A guide to its use

(http://www.mande.co.uk/docs/MSCGuide.pdf)

2. Action research resources

(http://www.scu.edu.au/schools/gcm/ar/arhome.html)