Participatory Monitoring and Evaluation

Participatory Monitoring and

Evaluation (PM&E)

CED 246-Evaluation in Rural Development

Anna Merlinna T. Fontanilla MS DEVCOM

PM&E, CED 246-Evaluation in Rural Development, 10/28/14

What’s in the presentation

• Background and concepts

• Goal of PM&E

• The PM&E Process

• Checklist in designing PM&E

• Sample case


PM&E roots

Participatory Action

Research

Farming Systems

Research or FPR

Participatory Learning and

Action (including PRA)

Participatory Monitoring

and Evaluation

PM&E, CED 246-Evaluation in Rural Development, 10/28/14 Conventional PM&E WHY Accountability, usually

summary judgment about the project to determine if funding continues

To empower local people to initiate, control and take corrective action

WHO External experts Community members, project staff, facilitator

WHAT Predetermined indicators of success, principally cost and production output

People identify their own indicators of success

HOW Focus on “scientific objectivity” distancing of evaluators from other participants; uniform complex procedures; delayed limited access to results

Self evaluation; simple methods adapted to local culture; open immediate sharing of results through local involvement in evaluation processes

WHEN Midterm and completion Any assessment for program improvement; merging of monitoring and evaluation, hence frequent small evaluations Narayan (1993) as cited by Sartorius, R./Social Impact


A process through which stakeholders at

various levels engage in monitoring or

evaluating a particular project, program or

policy, share control over the content, the

process and the results of the M&E activity

and engage in taking or identifying

corrective actions.


Common principles

Estrella and Gaventa


Goal of PM&E

Make the M&E process more

participatory and effective by including

a wider range of stakeholders at every

stage of the process.


The PM&E Process

1. Identifying and engaging

stakeholders

2. Building stakeholder’s

capacity for PM&E

3. Defining and agreeing on what to

monitor and evaluate: objectives

4. Developing and formulating indicators

5. Gathering information

6. Managing and analyzing data

8. Learning and change

7. Reflection, sharing and

using the results of PM&E

(Reporting)

Njuki, J. et al., 2006


Identifying and engaging stakeholders

Stakeholders include all those who affect,

and/or affected by the policies, decisions,

and actions of a system.


Building capacity for PM&E

1. Develop a common understanding of PM&E concepts and goals

2. Identify local vocabulary and terms equivalent to technical terms

3. Use of methods and tools that encourage participation

4. Discuss why PM&E is important to their lives and projects


Defining objectives

• Develop a common vision

• Agreeing on measurable

results and processes

that need to be

monitored and evaluated

• Identify what information

should be monitored, for

whom, and who should

be involved

• How results and findings

would be applied


Selecting indicators Roche, forthcoming


Gathering and analysis of data

• Analyze what is working,

what is not working and

why

• Reflect on the progress of

the project towards

achieving its goals and to

adjust activities as

required

• Allows community

members to

systematically reviews

their activities

Stakeholders decide:

• Which tools should be

used to collect

information on which

indicators,

• How sampling will be

done,

• Who should collect and

analyze the information,

• How frequently this will

be done


PM&E Plan/Framework/Worksheet

Results/ processes, activities

Indicators Info reqts.

Baseline Targets Data source

Who collects? Analyses? Reports?

How often? When?

Tools

Njuki, J. et al., 2006


Project Objectives (Goal, Purpose, Outputs)

Indicators Data Collection Data Analysis and Use

Source of Info

Baseline data needed

Who is involved

Tools and methods

How often

Added info needed

How often

Who is involved

How info is to be used

Who gets info

PM&E Plan/Framework/Worksheet

Sartorius, R./Social Impact


Reflection and learning

• Problems and successes

• Impacts of their efforts

• Act on what they have learned (how

stakeholders use information in making

decisions and identifying future action)

• Sharing results with the rest of the

community

• What do we need to change?


Checklist for designing PM&E

Will it be sustainable once the project has ended?

Do the people responsible for PM&E have all the necessary skills?

Can the system be incorporated into the structure of collaborating agencies?

Is the system based on a clear understanding of project objectives?

Is it based on a clear understanding of the information needs of key stakeholders?




Is the system based on indicators defined by

program participants?

Does the system involve the participation of all

key stakeholders in every stage of the PME cycle—

planning, data collection, analysis and use?

Do data collection tools fit the skills of the

collectors?

Is it cost-effective?

Is the amount of data collection manageable and

conducive to timely analysis and use of the results?




Is the amount of data collection manageable and

conducive to timely analysis and use of the

results?

Is the system documented so everyone knows

what it contains?

Is there a plan for testing and adjusting the

system?

Have annual self-assessments been planned?

Have impact evaluations been scheduled?

Others



PM&E Application

Participatory Evaluation in Human Resource Development: A Case Study

from Southeast Asia Gary Anderson and Deborah Gilsig

1998


Project background

Project Title SEAMEO-Canada Program of Cooperation in Human Resource Development

Location South East Asia

Time Frame Phase I (1985-1989) Phase II (1990-1995)

Goal Human resource development in the region through provision of short-term and long-term training courses, seminars, workshops, international and regional conferences, and information dissemination activities.


About the evaluation

Project Title SEAMEO-Canada Program of Cooperation in Human Resource Development

WHEN 5th and final year of the project

Objectives Provide information to SEAMES, CIDA, PAC, involved SEAMEO centers. Provide information that could guide future SEAMEO capacity development. Fulfil the conditions of the agreement between CIDA and SEAMES Provide accountability for CIDA’s investment in the project.

Design Employed external evaluators which provided general guidance. All stakeholders took part in the process by providing data; Each stakeholder participated in one or more of the other evaluation activities: developing the Terms of Reference, making revisions to the work plan, developing data-collection instruments, collecting data, analyzing data, and discussing and suggesting revisions to the evaluation report.


1. To what extent did the

project in general

represent a sound

development

investment?

1.1 Did it have a sound

rationale from the

perspectives of CIDA and

SEAMEO?

1.2 Was the project

efficiently implemented?

1.3 What were its

effectiveness and effects

in developing

organizational

capacities of SEAMEO? Do

these effects represent

good value for dollar?

2. To what extent did the institutional linkages help develop the capacities

of the involved SEAMEO centers?

2.1 Did they help centers to understand their internal and external environments?

2.2 Did they help develop needed center capacities?

2.3 Did they contribute to center performance?

3. To what extent was support for the regular SEAMEO training programs

worthwhile?

3.1 What were graduates’ perceptions of its efficiency, effectiveness, and

effects in making specialists more competent in their jobs?

3.2 Did the investment in regular training programs improve their quality in

a sustainable way?


Summary of major stakeholders and their roles

Stakeholder Principal Major Activities

CIDA • Defining evaluation questions related to compliance • Receiving the evaluation report and acting on its recommendations

• Facilitating implementation of the evaluation • Providing data • Providing feedback on draft report

PAC • Acting as the mandated authority • Commissioning the evaluation and receiving its reports

• Approving the terms of reference • Discussing the draft report • Receiving and approving the final report

SEAMEO/ SEAMES

• Facilitating data collection • Assuming responsibility for data collection from training of graduates • Providing data • Assisting with interpretation of analyses

• Helping design training graduate survey questionnaire • Distributing and collecting surveys • Providing data on project implementation


Summary of major stakeholders and their roles

Stakeholder Principal Major Activities

SEAMEO Centers

• Participating and assisting in data collection • Providing data • Assisting with interpretation of analyses

• Helping design training graduate survey questionnaire • Distributing and collecting surveys • Conducting center self-assessments • Providing data on project effectiveness, effects, and impacts • Interpreting data on project effectiveness, effects, and impacts

Canadian Institutions (Colleges and Universities)

• Providing data • Providing feedback on draft report

• Providing data on project effectiveness, effects, and impacts, and implementation through interviews and Canadian Technical Assistant Questionnaire

SEAMEO Training Graduates

• Providing data • Completing Regular Program Graduate Questionnaire


Summary of data collection

Source of Data Data Collection Instruments Proposed Sample Actual Sample (Actual #)

SEAMEO Centers Center Self-Assessment Guide SEAMEO-CIDA Project Questionnaire

12 centers +SEAMES 12 centers

92% (12) 100% (12)

SEAMEO Center Staff Trainees

Center Staff Trainee Questionnaire

n/a (56)

CIDA PTLs Interview Former & current PTLs

100% (2)

Canadian Institutions Telephone Interview 13 project heads 84% (12)

Canadian (TAs) Canadian Technical Assistant Questionnaire

68 Canadian TAs 59% (42)

Chairs of SEAMEO Governing Boards

Questionnaire for Members of SEAMEO Governing Boards

12 75% (9)

SEAMEO Center General Program Trainees

Regular Program Graduate Questionnaire

1,000 former trainees

53% (533)

PAC Members, SEAMES, CIDA

Numerous interviews were conducted to collect data on the functioning of the PAC, project management, etc.


Data analysis

• Statistical and content analysis

• Group analysis/interpretation

• Discussion about culturally-based findings

• Comparative levels of performance of

different training programs

• Center directors met, individually, with a

Universalia team member to discuss the

self-assessments (confidential, this was

owned exclusively by the centers)


Reflection and use of results

1. The process appears to have given the stakeholders a much deeper understanding of their organization (SEAMEO as a whole) and how it could link to Canada’s development interest.

2. It helped to build evaluation capacity.

- Center directors discussed about a possible phase III of the project.

- A more competitive or selective process for funding was being considered i.e. funding may not be available for all centers and will not be provided to be used as input substitution.

- Some of the centers plan on using the self-assessment tool ( or an equivalent) on a regular basis.

- A few of the centers are using the information from the evaluation in their strategic planning.

- The center’s participation in the process gave them a sense of ownership and the success has encouraged them to continue using the data collection instruments for future training reaction assessments.


References

• Adams, J. and A. Garbutt, 2008. Participatory monitoring and evaluation in practice:

Lessons learnt from Central Asia.. Praxis Paper 21: Participatory Monitoring and Evaluation

in Practice. International NGO Training and Research Center. Retrieved on October 26, 2014

from http://www.intrac.org/data/files/resources/420/Praxis-Paper-21-PME-in-Practice.pdf

• Estrella, M. 1998. Learning from change: Issues and experiences in participatory monitoring

and evaluation. Retrieved on October 6, 2014 from www.ids.ac.uk/files/dmfile/PB12.pdf

• Njuki, J. et al. 2006. Participatory monitoring and evaluation for stakeholder engagement,

assessment of project impacts, and for institutional and community learning and change.

Retrieved on October 20, 2014 from http://ciat-library.ciat.cgiar.org/Articulos_ciat/Njuki-

rev.pdf

• Sartorius, R. (n.d.). Participatory monitoring and evaluation systems: Improving the

performance of poverty reduction programs. Social Impact. Retrieved on October 20, 2014

from www.rrojasdatabank.info/wpover/06Rolf-Latest.pdf

• Victoria, The State of. 2013. Effective engagement. Retrieved on October 26, 2014 from

http://www.dse.vic.gov.au/effective-engagement/toolkit/tool-stakeholder-analysis-

stakeholder-matrix

• World Agroforestry Center, 2003. Carrying out a stakeholder analysis. Retrived on October

26, 2014 from

http://www.cglrc.cgiar.org/icraf/toolkit/Carrying_out_a_stakeholder_analysis.htm

• World Bank, The. 2003. Participation, monitoring and evaluation. Participation and Civic

Engagement. Retrieved on October 26, 2014 from

http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG

/0,,contentMDK:20509352~menuPK:1278203~pagePK:148956~piPK:216618~theSitePK:410306

,00.html

Participatory Monitoring and Evaluation

Education

Transcript of Participatory Monitoring and Evaluation