M&E Manual GuidE for ifad fundEd ProjEcts in ViEtnaM · 2013-05-11 · M&E Manual GuidE for ifad...

M&E Manual GuidE for ifad fundEd ProjEcts in ViEtnaM

1

Managing for iMpact in rural DevelopMent

M&E Manual GuidE forifad fundEd ProjEcts in ViEtnaM


3

Table of ConTenTs

LIST OF TABLES ............................................................................................................................................. 7LIST OF FIGURES ........................................................................................................................................... 7LIST OF BOXES .............................................................................................................................................. 8ABBREVIATION .............................................................................................................................................. 9

ChApTER I. hOw TO SET Up M&E SySTEM

1. Basic M&E concepts ................................................................................................................... 15

1.1. M&E Terms ................................................................................................................................. 15

1.2. PurposeandobjectivesofIFADfundedprojects’M&E .............................................................. 16

1.3. Result based management ......................................................................................................... 17

1.4. ParticipatoryM&E ...................................................................................................................... 18

1.5. IFADM&EMethodology ............................................................................................................. 19

2. M&Esystem ................................................................................................................................ 20

2.1. WhatisM&Esystem ................................................................................................................... 20

2.2. SettinguptheM&Esystem ........................................................................................................ 23

3. M&E planning ............................................................................................................................. 26

3.1. Thestructureofthemonitoringplan ......................................................................................... 26

3.2. Thestructureoftheevaluationplan .......................................................................................... 27

3.3. M&Eandprojectlogicalframework ........................................................................................... 29

3.4. DetailingM&Eplanbasedonprojectappraisalreport .............................................................. 31

3.5. UsingtheM&EMatrixfordetailedplanning .............................................................................. 32

3.6. IndicativecontentsforanM&EPlan .......................................................................................... 33

4. Datacollectionmethods ............................................................................................................. 36

4.1. Quantitativeandqualitativemethods ........................................................................................ 36

4.2. Collectingsecondarydataandinformation ................................................................................ 37

4.3. Survey ......................................................................................................................................... 38

4.4. Keyinformantinterview ............................................................................................................. 39

4.5. Focusgroupdiscussion ............................................................................................................... 39

4.6. Observationmethod ................................................................................................................... 39

4.7. Annualsupervisionmission ........................................................................................................ 39

5. Gender and M&E ........................................................................................................................ 39

5.1. TheimportanceofgenderinM&E ............................................................................................. 39

5.2. WhatwillagendersensitiveMonitoringsystemmonitor? ........................................................ 40

5.3. HowdowedogenderandM&E ................................................................................................ 41

6. M&Eimplementationresponsibilities ........................................................................................ 41


4

6.1. SeniorM&EOfficerinthePCU ................................................................................................... 416.2. KnowledgeManagementOfficer ................................................................................................ 426.3. Districtlevel ................................................................................................................................ 426.4. Communelevel ........................................................................................................................... 42

ChApTER II. MEASURING ANd REpORTING RIMS FIRST ANd

SECONd LEVEL RESULTS1. Overview .................................................................................................................................... 452. Whytomeasureandreportthefirstandsecondlevelresults? ................................................. 463. Collectinginformationformeasuringandreportingfirstandsecondlevelresults.................... 463.1. Selectingfirstandsecondlevelresults ....................................................................................... 473.2. Measuringandreportingfirstlevelresults ................................................................................. 483.3. Measuringandreportingsecondlevelresults ........................................................................... 52

ChApTER III. hOw TO CONdUCT ANNUAL OUTCOME SURVEy

1. Introduction ................................................................................................................................ 591.1. Whatistheannualoutcomesurveyandwhytodoit? .............................................................. 591.2. Howandwhentoconductthesurvey? ...................................................................................... 592. Howtoprepareandconductanannualoutcomesurvey?......................................................... 602.1. Step1–Fine-tuningthestandardquestionnaire ....................................................................... 612.2. Step2-Selectingthesample ..................................................................................................... 632.3. Step3–Enumerators’trainingandfieldtestingofthequestionnaire ....................................... 642.4. Step4–Logisticalplanning ........................................................................................................ 682.5. Step5–Datacollection .............................................................................................................. 692.6. Step6–Dataentry ..................................................................................................................... 702.7. Step7–Dataanalysis ................................................................................................................ 712.8. Step8–Reportwriting ............................................................................................................. 71

ChApTER IV. hOw TO CONdUCT RIMS ANd RIMS+ BASELINE ANd ENd-

LINE SURVEy1. WhatistheRIMSandRIMS+surveyandwhytodoit? ...................................................................... 752. Howandwhentoconductthesurvey ................................................................................................ 763. HowtoprepareandconductaRIMSand/orRIMS+survey ............................................................... 76

3.1.Step1–Developthequestionnaireandinterviewguide ................................................................ 763.2.Sampleselection.............................................................................................................................. 803.3.Step3–Enumerators’trainingandfieldtestingofthequestionnaire ............................................ 833.4.Step4–Logisticalplanning ............................................................................................................. 873.5.Step5–Datacollection ................................................................................................................... 893.6.Step6–Dataentry .......................................................................................................................... 903.7.Step7–Dataanalysis ...................................................................................................................... 913.8.Step8–Reportwriting .................................................................................................................... 91

tablE of contEnts

5

ChApTER V. SAMpLING METhOdOLOGy

1. Definitions .......................................................................................................................................... 951.1. Population .......................................................................................................................................... 951.2. Sample ................................................................................................................................................ 951.3. Samplingframe .................................................................................................................................. 961.4. Sample size ......................................................................................................................................... 961.5. Samplingmethod ............................................................................................................................... 962. SamplingmethodinprojectsfundedbyIFAD .................................................................................... 982.1. Thefirststage:Selectvillage .............................................................................................................. 982.2. Thesecondstage:Selecthousehold................................................................................................. 1003. Stand-byhouseholds ........................................................................................................................ 1034. Respondentselection ....................................................................................................................... 103

ChApTER VI. hOw TO CONdUCT kEy INFORMANT INTERVIEw

1. Definitionofkeyinformantinterview .......................................................................................... 1072. Classificationofkeyinformantinterview ..................................................................................... 1072.1. Unstructuredinterview ................................................................................................................ 1072.2. Semi-structuredinterview ............................................................................................................ 1083. Stepsconductingkeyinformantinterviews .................................................................................. 1083.1. Indentifyinterviewtopics ............................................................................................................. 1093.2. Identifykeyinformants ................................................................................................................. 1093.3. Developquestions ........................................................................................................................ 1093.4. Designkeyinformantinterviewguide .......................................................................................... 1103.5. Pre-testandrevisethequestionsorinterviewguide ................................................................... 1103.6. Conducttheinterview .................................................................................................................. 1113.7. Analyseandwritereport .............................................................................................................. 112

ChApTER VII. hOw TO CONdUCT FOCUS GROUp dISCUSSION

1. FocusGroupDiscussion ................................................................................................................ 1152. Stepstofacilitateafocusgroup ................................................................................................... 1162.1. Identifytopicsandinformationneeded ....................................................................................... 1162.2. Specifyinformationneedsforeachtopic ..................................................................................... 1162.3. Preparethediscussionguide ........................................................................................................ 1172.4. Selectparticipants ........................................................................................................................ 1182.5. Pre-testandrevisethediscussionguide ...................................................................................... 1182.6. Conductfocusgroupdiscussion ................................................................................................... 1192.7. Analyzeinformationandwritereport .......................................................................................... 122


6

ChApTER VIII. hOw TO ENTRy dATA USING CSpRO

1. WhatisCSPro? ................................................................................................................................. 1272. DataEntryapplication ...................................................................................................................... 1273. AccessingDatathathasbeensaved ................................................................................................. 1294. Modifyingdataandaddingnewcases ............................................................................................. 1305. Compare data tool ............................................................................................................................ 1306. Mergingdata(ConcatenateDataTool) ............................................................................................. 1347. ExportingDataTool .......................................................................................................................... 1378. SomenotesondevelopingdatabaseusingCSPro ............................................................................ 140

ChApTER IX. hOw TO ANALySIS dATA By SpSS

1. OpeninganexistingSPSSdataset ............................................................................................ 1432. Setupvariablenameandcodevariable ................................................................................... 1433. Dataanalysis ............................................................................................................................. 1453.1. Frequency ................................................................................................................................. 1453.2. Calculatethemean(Descriptive).............................................................................................. 1473.3. Crosstab(useforordinalandnominalvariables) ..................................................................... 1503.4. Comparethemeansofvariables .............................................................................................. 1523.5. Dealingwithmissingvalues ...................................................................................................... 1533.6. Dealingwithoutliners:Onlyappliedtoquantitativevariables................................................. 1533.7. Notes ........................................................................................................................................ 157

Annexes........................................................................................................................................................ 159

tablE of contEnts

7

list of taBlEsTable1.1ComparisonofMonitoringandEvaluation .................................................................................... 15

Table1.2ResultChain .................................................................................................................................... 18

Table1.3DistinguishM&EandparticipatoryM&E ...................................................................................... 19

Table1.4KeyM&Etasks ................................................................................................................................ 22

Table1.5ExampleofCriticalReflectionScheduleforaproject ..................................................................... 25

Table1.6QuestionstoguidethedetailedplanningofM&Econditionsandcapacities ................................. 26

Table1.7Examplefordeterminingthemonitoringmethodofgroupdiscussion ......................................... 27

Table1.8Illustrationofthestructureofanevaluationplan......................................................................... 28

Table1.9Exampleoflogframe ..................................................................................................................... 30

Table1.10TaskneededwhendetailingtheM&Eplanbasedonaprojectappraisalreport ......................... 31

Table1.11ContentsoftheM&Ematrix ......................................................................................................... 32

Table1.12IndicativecontentsforanM&EoperationalPlan ......................................................................... 34

Table1.13Examplesofmulti-purposeM&Emethods ................................................................................... 36

Table1.14Comparisonofquantitativeandqualitativemethods .................................................................. 37

Table2.1.Exampleofreportingfirstlevel ...................................................................................................... 52

Table2.2.Exampleofreportingfirstlevel ...................................................................................................... 52

Table2.3RatingScaleforsecond-levelresults ............................................................................................... 52

Table2.4.Listofperformancequestionsforeachfactor ............................................................................... 53

Table2.5Aframeworkoffactorsandguidingperformancequestions ........................................................ 55

Table4.1Suggestionofadditionalinformation/questionsofRIMS+questionnaire ..................................... 78

Table5.1ThesamplingframeofTNSPinTuyenQuang ............................................................................... 79

list of fiGurEsFigure1.1TheM&Esystemandhowitlinkstotheprojectstrategyandoperations .................................... 21

Figure3.1AnoverviewoftheAnnualOutcomeSurveydesign ..................................................................... 60

Figure5.1Theuseofcontrolgroup .............................................................................................................. 96


8

list of BoXEsBox1.1StepstosetuptheM&Esystem ........................................................................................................ 23

Box1.2Thebasicperformancequestionsperleveloftheobjectivehierarchy ............................................. 24

Box1.3Examplesofperformancequestions ................................................................................................. 24

Box1.4Exampleforgatheringinformationonfoodsecuritychanges .......................................................... 33

Box2.1ResultsChainTerminology ................................................................................................................. 46

Box3.1ExampleforthenecessaryofFocusGroupDiscussion ..................................................................... 60

Box3.2Somepointsshouldbekeptinmindwhendesigningadditionalquestions ...................................... 62

Box3.3Questionstobeansweredbypilot .................................................................................................... 62

Box3.4Thedutiesofenumerators ................................................................................................................ 65

Box3.5Thedutiesoftheenumeratorteamleader ....................................................................................... 65

Box3.6Thedutiesofsupervisors ................................................................................................................... 66

Box4.1Requirementsforagoodquestionnaire ............................................................................................ 77

Box4.2Somepointsshouldbekeptwhendesigningadditionalquestions ................................................... 77

Box4.3.Questionstobeansweredbypilot ................................................................................................... 80

Box4.4Thedutiesofenumerators ............................................................................................................... 83

Box4.5Thedutiesoftheenumeratorteamleader ....................................................................................... 84

Box4.6Thedutiesofsupervisors ................................................................................................................... 84

Box6.1Examplesoftopicsforkeyinformantinterview ............................................................................. 109

Box6.2Examplequestionsforthetopic“Communityparticipationinlocalplanning” ............................. 110

Box6.3.Tipstohaveagoodkeyinformantinterview .................................................................................. 112

Box7.1Exampleoftopicsforfocusgroupdiscussion ................................................................................. 116

Box7.2.ExampleforquestionsdesignedforFocusgroupdiscussionsuggestedbyIFAD ............................ 117

Box7.3.HintidentifyparticipantsforgroupdiscussionswithIFADprojects .............................................. 118

Box7.4.ExampleofFocusgroupdiscussionintroduction .......................................................................... 121

Box7.5.Somesuggestionstohaveagoodfocusgroupdiscussion ............................................................. 122

tablE of contEnts

9

aBBrEViation

AwpB Annual Work Plan and Budget

CdB Commune Development Board

CdF Community Development Fund

CIG Common Interest Group

CSpro Name of Data Entry Software

dASU District Agricultural Support

FG Focus Group

FGd Focus Group Discussion

GIS Geographical Information System

GpS Global Positioning System

IFAd International Fund for Agricultural Development

kI Key Informant

kII Key Informant Interview

M&E Monitoring &Evaluation

MTR Mid-Term Review

NA No Available

pCU Provincial Coordination Unit

pMU Provincial Management Unit

RBM Result Based Management

RdS Respondent-driven sampling

RIMS Results and impacts measuring system

SEdp Socio-Economic Development Plan

SI Sampling Interval

SpSS Name of statistical analysis software

SwOT Strength Weakness Opportunities and Threats

TNSp Tam Nong Support Project

TOR Term of Reference

Vdp Village Development Plan

Vdp Village Development Plan

VNd Vietnam Dong


10


11

Introduction about the Manual Guide

IFAD has been working for and with the poorest people in Viet Nam, including ethnic minorities, small-scale farmers and households headed by women. At present, IFAD is working at ten provinces in Viet Nam (09 projects), and some other projects will be implemented soon. All IFAD-funded projects, no matter what their specific focus is (e.g. natural resources, micro-enterprises value chains, microfinance) are expected to bring positive changes in the beneficiaries’ lives in terms of food security, incomes and livelihoods. To address these objectives, the projects should be managed and implemented in a proper result-oriented way. .

In order to assess its progress against implementation plans (monitoring), and to assess the outcomes and impact it has achieved (evaluation), the project’s monitoring and evaluation (M&E) system is ex-tremely important. The purpose of monitoring and evaluation is to obtain accurate information that will be used by decision-makers in the project to improve and ensure proper use of resources as well as to maximize the impact achieved from these resources. In 2003, IFAD adopted a Results and Impact Management System (RIMS), which is a comprehensive system to monitor and document the outputs, outcomes and impacts of IFAD-funded projects. In addition, the last two years have seen the quasi-experimental approach for rigorous impact evaluation being integrated into the IFAD RIMS (commonly referred to as RIMS+).

However, there is a fact that most of IFAD project management staff in Viet Nam have limited exposure to IFAD M&E system, including the RIMS and RIMS+ (while those have some experience are moving around). Therefore, building and improving capacity for IFAD project’s management staff is extremely im-portant. The management staff needs to be equipped with knowledge and practices on project manage-ment and M&E, especially they have to familiar with the RIMS/RIMS+, in order to organize and manage resources so the project is completed within defined scope, quality, time and cost constraints.

Development and Policy Research Center (DEPOCEN) is assigned by IFAD Vietnam to prepare a Manu-al Guide for IFAD funded projects in Viet Nam, to equipped the IFAD partners in Vietnam with knowledge and practices on M&E.

Therefore, this Manual Guide is prepare mainly for the two major audiences:

n Project managers: the people responsible for managing the various aspects of project imple-mentation. This includes the project director, managers of project components and the respon-sible managers of partner or contracting organizations who are implementing a specific element of the project;

n M&E staff: the staff of a project or implementing partners and contractors who have responsibili-ties for setting up and/or implementing M&E systems.

Besides above major audience, the Manual Guide is also useful for the consultants, who providing ex-ternal assistance on project M&E and information management, and anyone in IFAD and cooperating institution staff, who is providing guidance, supervision or support to the project.

The ideas in this Manual Guide are not a mandatory M&E system with which all projects must comply. The Manual Guide describes what is considered-and has proven to be-good practice in project M&E. The development of this Manual Guide is mainly based on series of IFAD technical guidance/paper, including


12

“Managing for Impact in Rural Development – A Guide for Project M&E”, “Result and Impact Management System – Practical Guidance for Impact Survey”, “Result and Impact Management System – First and Second Level Results Handbook”, Technical Guidelines No.1 to No.5 and other literatures about M&E in general.

Structure of the Manual GuideThe Manual Guide has nine chapters. The first Chapter provides basic M&E concepts and guidance on how to set up M&E system, gender and M&E, and a brief about method to collect data for the purpose of M&E.

Chapter II to Chapter VII are about selected methods and tools to collect data for M&E, e.g. Measur-ing and Reporting first and second level of results (Chapter II), How to conduct annual outcome survey (Chapter III), How to conduct RIMS and RIMS+ baseline and end-line survey (Chapter IV), Sampling methodology (Chapter V), How to conduct key informant interview (Chapter VI), and How to conduct focus group discussion (Chapter VII).

Chapter VIII and Chapter IX provide guideline to use the two software CSPro and SPSS for data entry and analysis.

The Annexes provide number of template questionnaires and forms developed by IFAD and DEPOCEN to facilitate the M&E system of IFAD projects in Viet Nam.

How to sEt uP M&E systEM

13

chapter i.



14


15

Chapter I.

How to set up M&e systeM1. Basic M&e concepts

1.1. M&E Terms

Monitoring is a continuous management function that aims primarily at providing programme manag-ers and key stakeholders with regular feedback and early indications of progress or lack thereof in the achievement of intended results. Monitoring tracks the actual performance against what was planned or expected according to pre-determined standards. It generally involves collecting and analysing data on programme processes and results, and recommending corrective measures.

Evaluation is a time-bound exercise that attempts to assess systematically and objectively the rele-vance, performance and success, or the lack thereof, of ongoing and completed programmes. Evaluation is undertaken selectively to (i) Answer specific questions to guide decision-makers and/or programme managers, and (ii) Provide information on whether underlying theories and assumptions used in pro-gramme development were valid, what worked and what did not work and why. Evaluation commonly aims to determine the relevance, validity of design, efficiency, effectiveness, impact and sustainability of a programme.

Table 1.1. Comparison of Monitoring and Evaluation

Monitoring Evaluation

Continuous.Periodic: at important milestones such as the mid-term of programme implementation; at the end or a substantial period after programme conclusion.

Keeps track; oversight; analyses and documents progress. In-depth analysis; Compares planned with actual achievements.

Focuses on inputs, activities, outputs, implementation processes, continued relevance, likely results at outcome level.

Focuses on outputs in relation to inputs; results in relation to cost; processes used to achieve results; overall relevance; impact; and sustainability.

Answers what activities were implemented and results achieved.

Answers why and how results were achieved. Contributes to building theories and models for change.

Alerts managers to problems and provides options for corrective actions. Provides managers with strategy and policy options.

Self-assessment by programme managers, supervisors, community stakeholders, and donors.

Internal and/or external analysis by programme managers, supervisors, community stakeholders, donors, and/or external evaluators.

The Relationship between Monitoring and Evaluation

Monitoring and evaluation are intimately related. Both are necessary management tools to inform deci-sion-making and demonstrate accountability. Evaluation is not a substitute for monitoring nor is monitor-ing a substitute for evaluation. Both use the same steps however, they produce different kinds of informa-tion. Systematically generated monitoring data is essential for successful evaluations.


16

1.2. Purpose and objectives of IFAD funded projects’ M&E

Purpose of M&E

M&E is an important management function to track progress and support the decision-making process. It aim to to meet the requirements of a number of donors, but also of partners and targeted communities that work directly with your organization, who benefit most from the communicating progress through the M&E system. Through a thorough review of work progress, your organization can design programs and activities more effectively and ensure greater benefits to the project target group.

For IFAD funded projects, all projects gather information to assess progress against implementation plans (monitoring), and all projects gather information to assess the outcomes and impact it has achieved (evaluation). Good projects collect accurate information and produce timely reports. Great projects collect accurate information, and not only produce good reports, but also use the information to make decisions that improve efficiencies and enhance impact. The purpose of monitoring and evaluation is to obtain accurate information that will be used by decision-makers in the project to improve and ensure proper use of resources as well as to maximize the results and benefits achieved from these resources.

Objectives of M&E

The M&E system in the IFAD funded projects are expected to achieve the following four objectives:

- Monitor and Manage Project Progress. Information will be obtained to measure the progress of the project against planned activities and production of outputs. This will also include assessing the quality of the activities and outputs toward achieving the desired outcomes. The information will be used by decision-makers to adjust plans by expanding activities that produce results at a reason-able cost, modifying activities that need to produce greater results or be implemented efficiently and discontinuing activities that are producing unsatisfactory results.

- - Assess Project Outcomes and Impact. Project M&E systems will obtain information to enable decision-makers to assess the short- and medium-term effects of project interventions (outcomes), towards achieving impact on targeted impact groups (development objective) and in making con-tributions toward higher level impact (project goal). Outcome-level results assessment will focus on changes in behaviour or systems as a result of project outputs. At the objective-level, the M&E systems will obtain information to assess the likely long-term impact of the project on the lives of targeted impact groups. At the goal-level, the project M&E systems will provide information to as-sess the overall change, from all combined factors, toward meeting the goals of the program. The information obtained will be used by decision-makers to make modifications to the project strategy, reallocating resources, as required.

- Capture and Disseminate Lessons Learned and Good Practices. The project will generate useful lessons learned, both positive and negative, on implementation and impact as well develop good practices for achieving impact. The project M&E systems will systematically capture these and ensure that they are not only well documented, but widely disseminated within the project as well as outside of the project. In addition, good practices and lessons learned from outside the project will be channeled into the project to be adapted and tested.

- Build Local Capacities for M&E. Finally, the project M&E systems have a specific objective to build capacities among stakeholders to be able to undertake effective monitoring and evaluation using cutting edge ideas and technologies. The project will produce clear guidelines for gathering,


17

recording, analyzing and presenting information tailored to stakeholder needs, and for sharing such information across departments at different levels and with community and other stakeholders.

1.3. Result based management

Results Based Management (RBM) is a management strategy by which an organization ensures that its processes, products and services contribute to the achievement of desired results (outputs, outcomes & impacts). RBM rests on stakeholder participation, on clearly defined accountability for results, and on ef-fective communication that ensures constraints to achieving objectives are resolved in good time. It also requires monitoring of progress towards results and reporting on performance/feedback which is carefully reviewed and used to further improve the design or implementation of the programme.

The purpose and benefits of results-based monitoring

• It supports achievement of positive results and objectives;

• It facilitates the identification of negative results and risks and in this way allows introducing measures to correct them;

• It clarifies tasks and responsibilities and establishes feedback and discourse mechanisms be-tween all stakeholders;

• It creates a well structured and transparent basis for decisions, based on real data and informa-tion;

• It facilitates communication about achieved results and therefore:

- Improves communication with the contractor;

- Facilitates co-financing.

• It facilitates understanding and presentation of contributions to the Millennium Development Goals.

RBM involves three steps

1) Performance planning (e.g. project design or preparation of the annual work plan and budget): use Project Logframe; Cost Tables; Results-Oriented Annual Work Plan.

2) Performance measurement (e.g. through periodic M&E or ad hoc studies): using M&E Tools

3) Performance evaluation and reporting: use Supervision reports; Annual Progress Reports; Mid-Term and Completion review reports.

Project management need to facilitate or undertake the regular collection, analysis and evaluation of information on results, in order to ensure that resources are used in the best way for expected objectives to be achieved, at all levels.

The Results’ Chain concept

Results’ Chain is the core of RBM, it consists of series of results: result level 1– outputs, result level 2 –outcomes and result level 3 – impact (intended or unintended, positive or negative) of a development intervention. The outputs, outcomes and impact are described in Table 1.2.


18

Table 1.2 Result Chain

Results’ level Definition Example

Level 3

(Impact)

Positive and negative, long-term effects produced by a development intervention, directly or indirectly, intended or unintended

Positive impact: increased farmers’ income

Level 2

(Outcomes)The likely (or achieved) short-term and medium-term effects of an intervention

Increased capacity by small farmers to grow high-value crops;

Increased production of high value crops in project area

Level 1

(Outputs)

The products and services produced (through activities) by the project and provided to direct beneficiaries

Activity of project: organized training course à result: Farmers trained in the production of high-value crops

Input Financial, material and human resources used for implementing activities Human, material, technical

Process to achieve the results

The transformation of inputs into outputs, through activities, is quasi automatic (e.g. if a training course is organized, then the output “people trained” is achieved). Outputs do not get automatically translated into the expected outcomes or impact. For example, it may not be sufficient to teach farmers how to grow high-value crops and unless they also have access to improved seeds, they will not necessarily be able to grow high-value crops. External factors cause a positive or negative impact on the expected results. Although project management may be able to control certain factors, some other external factors will be largely beyond project management’s control, e.g. natural disasters, epidemics….

1.4. Participatory M&E

Participatory M&E (PM&E) advocates stakeholder engagement and inputs into the planning and imple-mentation stages of a project, as well as during evaluations. This has the benefits of:

- Identifying opportunities for better targeting and design of activities to met local needs and prefer-ences;

- Encouraging local ownership and support to project activities by developing project-related stake-holder relationships that are more focused on equal partnerships;

- Gaining the support of project gatekeepers, such as government staff, that may be need to provide access to some regions, organisations, and people;

- Empowering and building the capacity of project beneficiaries and local implementing partners;

- Assessing benefits to project beneficiaries, as they themselves perceive these benefits;

- Directing interventions in a way that maximizes local benefits, particularly direct benefits to the target group.

Core principles of PM&E

- Primary stakeholders are active participants – not just sources of information;

- A focus on building the capacity of local people to analyze, reflect and take action;


19

- There are opportunities for joint learning of stakeholders at various levels;

- It aims to create greater stakeholder commitment to take corrective actions to help themselves.

Participatory Project Management and PM&E need a different mindset than traditional approaches to project management do. Project staff need to be prepared to regularly share their knowledge and infor-mation openly with key project actors.

Table 1.3. distinguish M &E and participatory M & E

M&E participatory M&E

• Aims at making a judgment on the programme for accountability purposes rather than empowering programme stakeholders

• Strives for “scientific” objectivity of M&E findings thereby distancing the external evaluator(s) from stakeholders

• Tends to emphasise the needs for information of programme funders and policy makers rather than programme implementers and people affected by the programme

• Focuses on measurement of success according to predetermined indicators

• Is a process of individual and collective learning and capacity development through which people become more aware and conscious of their strengths and weaknesses, their wider social realities, and their visions and perspectives of development outcomes. This learning process creates conditions conducive to change and action

• Emphasises varying degrees of participation (from low to high) of different types of stakeholders in initiating, defining the parameters for, and conducting M&E

• Is a social process of negotiation between people’s different needs, expectations and worldviews. It is a highly political process which addresses issues of equity, power and social transformation

• Is a flexible process, continuously evolving and adapting to the programme specific circumstances and needs

Types of Stakeholders could participate in M&E

- The community whose situation the programme seeks to change

- Project Field Staff who implement activities

- Programme Managers who oversee programme implementation

- Funders and other Decision-Makers who decide the course of action related to the programme

- Supporters, critics and other stakeholders who influence the programme environment..

1.5. IFAD M&E Methodology

The proposed M&E system is oriented around a three-level framework within which to monitor implemen-tation progress toward producing proposed outputs and assessing the impact of these outputs in inducing behavioral or systemic change and impact on targeted impact groups and wider development goals.

i. Output Monitoring: The first level refers to the regular and continuous monitoring of project ac-tivities, project processes, and the outputs produced as a result.

ii. Outcome Assessment: The second level relates to the monitoring and assessment of behavioral or systemic change that is being induced by the outputs produced by the project.

iii. Impact Assessment: Finally, the third level refers to the assessment of changes occurring in


20

the quality of life on targeted impact groups as a result of the behavioral or systemic changes caused by the project. This impact includes both the impact on specific targeted impact groups as reflected in the project’s development objective, as well as the contributions that the project is making towards the achievement of its overall goal.

The M&E system within the IFAD funded projects is a results-based system which will focus not just on how well the activities of the project are being executed, but also on answering the question “have the goals and objectives of the project been achieved; and how is this supported by evidence?”. Rather than focusing on how much output is being generated by the project, the monitoring and evaluation system will use survey data, qualitative information and self-assessments by participants to measure behavioural and systematic changes, as well as how these changes are affecting the lives of targeted beneficiaries.

In 2004, IFAD introduced a methodology for tracking the impact of IFAD assisted projects called the Results and Impact Management System (RIMS). RIMS is a comprehensive system for measuring and reporting on the results and impact of IFAD supported country programmes. RIMS consists of a stand-ardized list of common indicators that can be applied across all IFAD projects and programmes. The list includes 1st level indicators (of outputs), 2nd level indicators (of outcomes), and 3rd level indicators of (impact). See Annex 1 for the 1st and 2nd level indicators.

Within the system, four RIMS standardized indicators are mandatory for all projects:

i. Number of households with improvement in household assets ownership index (anchor impact level indicator);

ii. Reduction in the prevalence of child malnutrition (anchor impact level indicator);

iii. Number of households that have improved food security (compulsory outcome level indicator);

iv. Number of persons receiving project services (direct beneficiaries - compulsory output level indicator).

v. In addition to these mandatory indicators, other performance indicators at the different levels have been identified during the project design phases as shown in the project Logical Frame-work.

2. M&e system

2.1. What is M&E system

The M&E system is the set of planning, information gathering and synthesis, reflection and reporting processes, along with the necessary supporting conditions and capacities required for M&E to make a valuable contribution to planning, decision-making and learning. Key project stakeholders need to de-velop the different elements of the system together if they are all to use the M&E outputs to improve implementation.

The project strategy (the plan for what will be achieved and how it will be achieved is the starting point for project implementation and setting up the M&E system. The strategy is the basis for working out the project operations required to efficiently and effectively achieve project objectives. The completion of project activities leads to a series of actual outputs, outcomes and impacts. A core function of M&E is to comparethe actual outputs, outcomes and impacts with what was planned in the project strategy, and understanding the differences in order to identify necessary changes in strategy and operations to ensure that project purpose is achieved..


21

The M&E system consists of four interlinked parts: (i) setting up the M&E system; (ii) implementing the M&E system means gathering and managing information; (iii) involve project stakeholders in reflecting critically; and (iv) communicating and reporting the results of M&E. Ultimately the results from M&E – both the communication processes and the information- will feed into results-based planning processes and improve the project strategy and operations.

For M&E to work as a tool for managing for impact, project management and the M&E staff need to be clear on how to identify, agree upon and follow up on project improvements. If this process for guiding changes is not in place, even a very good M&E system will not have much value for the project.

Figure 1.1. The M&E system and how it links to the project strategy and operations


22

Table 1.4. key M&E tasks

Early design phase (formulation and appraisal)

Establish the scope and purpose of the M&E system

Indicate key performance questions and indicators, plus associated monitoring mechanisms

Identify organizational arrangements for M&E

Develop terms of reference for M&E staff

Indicate the process for how M&E is to be established during start-up

Establish an indicative M&E budget

Document the above in the M&E framework

Start-up prior to loan effectiveness

Revise performance questions, indicators and monitoring mechanisms after reviewing the project strategy

Organize training with staff and partners likely to be involved in M&E

Initiate a project implementation manual with key staff

Prepare a project implementation manual with key staff

Start-up after loan effectiveness

Review project design in relation to M&E with key stakeholders

Develop a detailed M&E plan, taking into consideration existing mechanisms with partners

Put in place necessary conditions and capacities for M&E to be implemented

Main implementation

Ensure information needs for management are met

Coordinate information gathering and management

Facilitate informal information gathering and communication

Support regular review meetings and processes with all implementers

Prepare for supervision missions

Prepare for and facilitate the annual project review

Conduct focused studies on emerging questions

Communicate results to stakeholders

Prepare annual progress reports

Mid-term review

Collate information for the mid-term review

Facilitate the internal review process to prepare for the external review process

Help respond to MTR feedback

Adjust the M&E system. As necessary

Phasing-out and completion

Assess what the implementers can do to sustain impact and sustain M&E after closing down-and implement these ideas

Hold workshops and do field studies with key stakeholders to assess impacts

Identify lessons learned for the next phase and/or other projects


23

2.2. Setting up the M&E system

Setting up an M&E system involves six steps that need to be dealtwith twice – generally at initial design and in detail at start-up- constructed around the project results chain:

Box 1.1. Steps to set up the M&E system

1. Establishing the purpose and scope – Why do we need M&E and how comprehensive should our M&E system be?2. Identifying performance questions, information needs and indicators – What do we need to know to monitor and

evaluate the project in order to manage it well?3. Planning information gathering and organising – How will the required information be gathered and organised?4. Planning critical reflection processes and events – How will we make sense of the information gathered and use it to

make improvements?5. Planning for quality communication and reporting – What, how and to whom do we want to communicate in terms of

our project activities and processes?6. Planning for the necessary conditions and capacities – what is needed to ensure that the M&E system actually

works?

2.2.1. Establishing the purpose and scope

Clear definition of the purpose and scope of the intended M&E system helps when deciding on issues such as budget levels, number of indicators to track, type of communication needed and so forth. The appraisal report will include a brief M&E purpose statement but you will need to revisit this question at start-up with representatives of implementing partners and primary stakeholders.

When formulating the purpose at appraisal or revisiting it during start-up, ask yourself the following question:

• What are the main reasons to set up and implement M&E, for us – as implementing partners and primary stakeholders – and for other key stakeholders?

Define the scope of the M&E system by asking:

• What level of funding is potentially available?

• What level of participation in M&E by primary stakeholders and partner organizations is desir-able and feasible?

• How detailed does the M&E information have to be, either in terms of quantitative or qualitative data?

• What sort of baseline study is desirable and feasible?

• What are the current M&E capacities among primary stakeholders and partners organizations, and how will this affect the desired levels of M&E?

2.2.2. Identifying performance questions, information needs and indicators

A performance question helps focus your information-seeking and information analysis processes on what is necessary in order to know if the project is performing as planned or, if not, why not. Once you have your performance questions, you can more easily decide what information you need to track rather than what is nice to track.


24

Working with performance questions to guide indicator analysis will give you more integrated and mean-ingful picture of overall project achievements. Answering these questions requires descriptive analysis and quantitative information. Starting by identifying performance question make it easier to recognise which specific indicators are really necessary. Sometimes a performance question can be answered di-rectly with a simple quantitative indicator. However, very often the question can only be answered with a range of qualitative and quantitative information.

Box 1.2. The basic performance questions per level of the objective hierarchy

n Activities – What have we actually done?n Outputs – What have we delivered as a result of project activities (e.g., number of people trained)?n Outcomes (results) – What has been achieved as a result of the outputs (e.g., extent to which those trained are

effectively using new skills)?n Impacts – What has been achieved as a result of the outcomes (e.g., to what extent are NGOs more effective)?

What contribution is being made to the goal? Are there any unanticipated positive or negative impacts? n Lessons – What has been learned from the project that can contribute to improved project implementation or to

building relevant fields of knowledge?

Box 1.3. Examples of performance questions

Performance questions are not just about what has been achieved. They also ask why there is success or failure and what has been learned to improve future action. Examples of performance questions include:n How has the purchasing power of target households as compared to non-target groups? What external factors

have influenced any changes?n To what extent are target households better able to meet their housing, education and health needs than non-

target households?n How have the diversity, production and productivity of agriculture in the target area changed as a result of project

activities and as a result of external factors?

2.2.3. Planning for information gathering and organising

For each information need or indicator, you must establish how the information will be collected and organized. Not only will each indicator require choosing a different method, but for each indicator or infor-mation need you will usually present several options. There are many methods – qualitative and quantita-tive and individual versus group-based.

Particularly critical at the moment of method selection is knowing who will be involved in collecting, compiling and analyzing. The more that the intended users of the methods can involved in selecting or developing the methods, the more chance there is that they will understand them and use them correctly. If methods are selected by someone not using them, then training users in the method will be essential.

Gathering data is one thing. But each bit of information also needs to be collated, perhaps summarised and certainly analysed by the right people.


25

2.2.4. Planning critical reflection processes and events

Critical reflection can occur formally and informally. Formally, it can be facilitated during project meetings, workshops with partners and primary stakeholders or as part of external evaluation. Informally, it can occur in ongoing discussions between project stakeholders.

Table 1.5 Example of Critical Reflection Schedule for a project

Note: This is a project-centric list. Primary stakeholders will have their own schedule of events as will each of the implementing partners,

The art of participatory M&E is to link these different schedules for shared learning.

Critical Reflection processes and

Eventspurpose and description whom to involve Timing

Participatory review of project strategy

Update the situation analysis, revise problems/visions, adjust objective hierarchy and assumptions

Representatives of intended primary stakeholders, staff or participant organization, all project staff, facilitator

Three-day workshop at start-up

Development of M&E plan with stakeholders

Assess different information needs, take stock of who is already doing what, agree on priority information areas, refine questions/indicators, decide on methods, agree on responsibilities

Representatives of intended primary stakeholders, staff or participant organization, all project staff, facilitator

Four or five full-day meetings during the first six months of the project

Quarterly progress reviews by PMU

staff

Discussion of key successes and problems Senior staff of PMU and partner organization

One-day meeting every three months

Field visits Firsthand look at what is happening in the field, informal chats about how activities are being implemented

Field staff, supervisors of field staff, project director

Weekly visit for field staff, monthly for the project director

Annual project review

Summary of key successes and problems, ideas for changing project activities/outputs and assumptions, review of implications for the project log frame, identification of lessons learned about project implementation, M&E system adjustment

Representatives of intended primary stakeholders, staff of implementing partners, all project staff, facilitator

Once a year

Periodic review workshops of key

project components

Focused discussion about the strategy and operations of key components to adjust the objective hierarchy, solve problems and identify lessons learned

Key stakeholders of the project component: intended primary stakeholders, implementing partners, field and senior project staff

Once a year in the first two years; after that, once every two years

Preparation for supervision

missions

Explain the mission purpose, agree on what the project and stakeholders would like to get out of the mission, identify who needs to prepare what before the mission, organize the logistics

Small group of primary stakeholders representatives, senior staff of the participant organization, senior project staff

One month prior the supervision mission


26

2.2.5. Planning for quality communication and reporting

M&E –related findings have many potential audiences. When reporting on progress with the AWPB, you will direct yourself to funding agencies, steering committees, cooperating institutions and implementing partners. Primary stakeholders have a right to knowing overall how the project is progressing and they deserve the opportunity to react to initial findings. Funding agencies and managers need information on impact, while all implementing partners need to understand problems in order to find solutions. Two sets of M&E findings will need to be communicated.

First, it is good practice to discuss draft M&E findings with implementing partners and primary stakehold-ers in order to get feedback on accuracy, reach joint conclusions and agree on next steps. Once the M&E findings are agreed upon, these can be communicated to funding agencies, cooperating institutions, gov-ernment departments and other projects. This second set of final findings will fulfill accountability needs but can also serve for advocacy purposes.

2.2.6. Planning for the necessary conditions and capacities

Usually, the appraisal report has indicate budget for M&E and describe how M&E should or could be organized. But getting the M&E system working means thinking of appropriate incentives, ensuring you have the right and enough human capacity at hand and thinking about ways of storing and sharing infor-mation. Table below lists key questions that need answers when detailing M&E to make it operational.

Table 1.6. Questions to guide the detailed planning of M&E conditions and capacities

Conditions and Capacities Questions to Guide detailed planning

Human capacity for M&E - What are the existing M&E capacities?

- What training will be necessary?

- What consultancy support will be required?

Incentives for implementing M&E

- Are M&E responsibilities included in job descriptions and terms of reference?

- How will reflection and learning among staff, partners and the intended primary stakeholders be encouraged?

Organizational structures - Will there be an M&E unit or will M&E be spread among all parties? If there is a unit, how many people will it have and where will it be located, under whose authority?

- How closely connected will M&E staff be with project management?

Management information systems (MIS)

- What information must be stored and accessible, when, how and for whom?

- What level of computerization is required and appropriate?

- What expertise will be required to set up the information management system?

Financial resources - Is there a separate M&E budget and have sufficient resources been allocated?

- Has the staffing allocation for the project taken into account time for all relevant staff to undertake M&E activities?

3. M&e planning

3.1. The structure of the monitoring plan

The monitoring plan specifies 5 issues: (i) what are the monitored subjects?; (ii) When is the monitoring conducted?; (iii) How is the monitoring conducted?; (iv) Who carry out the monitoring?; and (v) monitor-ing cost.


27

(i) What are the monitored subjects?

Important factors in determining the monitored subjects, including:

- - Main subjects that receive information;

- Questions;

- Indicators (for changes);

- Information on the implementation;

- Unexpected changes;

- General problems;

- Initial evaluation on the effectiveness of the results.

(ii) When is the monitoring conducted?

- After identifying the monitoring objectives, indentify the timeline for the monitoring. It is necessary to determine a timeline suitable with the timeframe of the whole project.

(iii) How is the monitoring conducted?

- Group discussion to determine the monitoring method, example:

Table 1.7. Example for determining the monitoring method of group discussion

Monitored issues Indicators how to measure the indicators how to report the results

Build models for crop production for project households

Number of participants (households)

Number of households that apply the model

Report of project officials on the number of participants

Questionnaires

Were the objectives achieved?

Adjustment measures (if objectives were not achieved)?

(iv) Who conduct the monitoring?

Who conduct the monitoring? In the monitoring plan, it is necessary to present the name and position of the monitoring official. This presentation has two effects, (i) link the monitoring official with specific activi-ties and hold him/her responsible for his/her activities; and (ii) the monitoring official knows what, how and when to do.

(v) Monitoring cost

Whether there are expenditures for the monitoring. If there are, estimate the monitoring cost, via monitoring activities and supports, such as means, for best results.

3.2. The structure of the evaluation plan

The structure of an evaluation plan is similar to that of a monitoring plan. Accordingly, the evaluation is also based on an established plan. An evaluation plan specifies 5 issues: (i) The evaluation objectives; (ii) Evalu-ation methods; (iii) Organization of the evaluation; (iv) Evaluation cost; and (v) Evaluation information-use plan.


28

The key point of an evaluation plan is to identify what are evaluated and how to do it. Generally, an evalu-ation plan contains the purpose and the evaluated subjects, questions, indicators to measure results, data-collecting methods, work plan and resources to implement the evaluation.

The evaluation plan must identify explicitly what are evaluated, such as a project, an organization, a sector or a programme. As demonstrated above, an evaluation plan includes information like contexts, objectives, specific activities, costs, expected impacts and results.

Table 1.8. Illustration of the structure of an evaluation plan

Requirement Content

Objectives, evaluation content What are evaluated? – What we need to know? Base on the project design – the Objectives section; or the Project Evaluation Plan – Evaluation Objectives section; or the Term of Reference to identify the evaluation objectives and content.

Evaluation method Present the evaluation methods used to collect information in order to specify the evaluation objectives and content.

Organize the evaluation Who will do what? What is his/her role? What are the roles of the partners, the relevant stakeholders and the evaluation team?

Evaluation cost Full estimated expenditure for each evaluation activity.

Evaluation information-use plan Structure of the evaluation report.

Expected purposes of information.

Plan to use and share information.

The five key questions

Followings are the five standard types of evaluation questions:

1. Relevance – Was/Is the project a good idea given the situation needing improvement? Does it deal with target group priorities? Why or why not?

2. Effectiveness – Have the planned purpose and component objectives, outputs, and activities been achieved? Why or why not? Is the intervention logic correct? Why or why not?

3. Efficiency – were inputs (resources and time) used in the best possible way to achieve out-comes? Why or why not? What could we do differently to improve implementation, thereby maximizing impact, at an acceptable and sustainable cost?

4. Impact – To what extent has the project contributed towards its longer-term goals? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise? To what extent has the project contributed towards poverty reduction (or other long-term goals)? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise?

5. Sustainability – Will there be continued positive impacts as a result of the project once it has finished? Why or why not?

The M&E operations will focus on the questions of “effectiveness” and “efficiency”. More strategic reflec-tions, like during annual reviews and supervision mission, will look at the questions of “relevant”, “impact” and “sustainability”. Some projects are also asked to prove their cost-effectiveness.


29

3.3. M&E and project logical framework

The logical framework (or logframe for short) is a tool to present objectives into specific and measureable indicators. In addition, the logframe specifies how to address those objectives as well as assumptions to achieve them. This tool also clarifies the relation between the objectives of the project and the activities in the project design. A specific log frame can be described as follows:

Long-term objectives: impacts, purposes, expected long-term results – contributions of the pro-ject.

Medium-term objectives: outcomes- these are intermediate resultsshort-term and medium-term benefits that the project aims to secure. Outcomes are a result of local changes in people’s be-haviors and in the performance of institutional services. They are the change effected in actors’ behaviours and capabilities (i.e. in actors’skills, attitude, knowledge, practices), and the change in performance (efficiency and effectiveness) of local systems (e.g learning systems, extension, banking systems), that occur as a result of outputs delivered by the project. Outcomes are generally measured in terms of what the actors within the sphere of the project have done or are now prepared/able to do. Their achievement increases the likelihood of contributing to the overall (longer-term) goal.

Short-term objectives: quick outputs and results – products or services that are directly provided by the project. Outputs are generally measured in terms of what project teams have done or delivered.

Indicators: To monitor and evaluate the results of a project/programme, measure impacts, results and outputs as expected to be built. These measuring methods are usually called indicators.

Criteria: Once the project objectives and relevant indicators are built, project management officials and M&E officials need to identify annual project evaluation method or criteria on the basis of final expected project’s results. If the progress criteria, final output criteria and result criteria have been identified in the log frame in the drafting of the project’s plan, M&E officials still have to evaluate their appropriateness in the operational environment. In case the final output criteria are agreed by the sponsor but the annual progress criteria have not been established, the project officials and M&E officials have to use completion criteria of the project to determine suitable annual criteria. Practical criteria can be built by dividing indicators into different levels of completion.

Risk: All projects face a certain kind of risks that hamper the achievement of objectives and/or expected results. Thus, these risks need to be identified, monitored and evaluated closely by M&E officials to propose necessary responses to reduce the impacts of any internal or external risk. Regarding internal risks, their impacts lie within the project. Internal risks may be delays in signing contract that affect important decisions, delays in disbursement for stakeholders’ opera-tion, or change in number of staff, etc. This kind of risks need to be controlled in order to achieve the objectives and expected results in a timely manner. On the other hand, external risks lie out-side the project but still affect its operational environment. Thus, they are often out of the control of the project. However, they still need to be monitored, evaluated and controlled right after they are recognized as potential threats to the project.

Risk management and risk planning: need to be built, especially for projects in unstable environ-ment or potentially unstable environment. In those projects, specific actions should be planned in preparation for worse conditions. Evaluating the level of risks is part of this process. Project


30

officials, particularly M&E officials, should be required to notify the Project Management Board on the unsolved risks for timely responses.

Assumption: External factors (e.g. events, conditions or decisions) may affect the progress or the success of a project/programme. The initial assumptions are critical requirements for the success of a project/programme. Risk assumptions are conditions that may threaten the imple-mentation of a project/programme.

Table 1.9. Example of log frame

project description Indicator Indicator verification Risk

Objectives • Support the establishment of CIGs of farmers on crop production and animal husbandry.

• Number of CIGs established in project area.

• CIGs’ work reports. • People are still hesitated/shy to take part

Purposes • Share working experiences for households.

• CIGs receive supports from the project: financial and technical supports

• Improve income

• Number of CIGs offered loans

• Number of training courses organized

• Household’s income (in comparison that previously)

• Reports of the Policy Bank.

• Reports on the progress of the project

• Annual survey on the project results.

• People are still hesitated/shy to take part

• Limited fund-managing capacity of the people and the Project Management Board.

Outcomes • CIGs have access to loans and other financial services

• Farmers have access to knowledge and information on improved crop husbandry practices

• Number of CIGs offered loans

• Number of CIGs that apply farming techniques


• Surveys


• People are not used to new cultivation customs.

Outputs • Establish CIGs in project area.

• Number of established CIGs in project area.

• CIGs’ work reports. • People are still hesitated/shy to take part

Activities • Offering loans to CIGs

• Provide farming-technical training to CIGs

• Volume of CIG loan disbursements; and repayment rate

• Number of farmers attending training


• Quarterly reports of the Project Management Board.


Inputs • Finance

• Human resources

• Training equipment

• Quarterly disbursement

• Quarterly participation of the members of the Project Management Project.

• Number of teaching materials prepared for distribution

• Project information management system

• Reports on the progress of the project



31

3.4. Detailing M&E plan based on project appraisal report

A good appraisal report will include an indicative M&E framework that provides enough detail about these questions to enable budgeting and allocation of technical expertise, give funding agencies an overview of how M&E will be undertaken, and guide project and partner staff during start-up. But this will only be indicative and needs to be adjusted and detailed further during the start-up phase. Below table presents the tasks for each step in taking the outline of an M&E system from an appraisal report and designing the details that make it operational.

Table 1.10. Task needed when detailing the M&E plan based on a project appraisal report

M&E design StepsOutputs in project

Appraisal Report (the M&E Framework)

Task during project Start-up to develop a detailed M&E plan

1. Establishing the purpose and scope

• Broadly defined purpose and scope of M&E in the project context

• Review the purpose and scope with key stakeholders.

2. Identifying performance question, indicators and information needs.

• List of indicative key questions and indicators for the goal, purpose and output levels

• Assess the information needs and interests of all key stakeholders.

• Precisely define all questions, indicator and information needs for all levels of the objective hierarchy.

• Check each bit of information for relevance and end-use.

3. Planning information gathering and organizing

• Generally described information gathering and organizing methods to enable resource allocation

• Plan information gathering and organizing in detail (who will do use which method to gather/synthesize what information, how often and when, where, with whom, with what expected information product).

• Check the technical and resource feasibility of information needs, indicators and methods.

• Develop formats for data collection and synthesis

4. Planning for quality communication and reporting

• Broad description of key audiences and types of information that should be communicated to them to enable resource allocation

• Make a precise list of all the audiences, what information they need, when they need it and in which format.

• Define what is to be done with the information – simply send it, provide a discussion for analysis, seek relevant feedback for verification, etc.

• Make a comprehensive schedule for information production, showing who is to do what by when in order to have the information ready on time.

1. Planning critical reflection processes and events

• General outline of key processes and events

• Precisely detail which methods/approaches are to be used, with which stakeholder groups and for what purpose.

• Identify who is responsible for which reflective events

1. Planning for the necessary conditions and capacities

• Indicatives staffing levels and types, clear description of organizational structure of M&E, indicative budget

• Come to a precise definition of: the number of M&E staff, their responsibilities and their linkages, incentive needed to make M&E work, organizational relationships between key M&E stakeholders, the type of information management system to be established and a detailed budget.


32

3.5. Using the M&E Matrix for detailed planning

To make M&E operational you need much more detail, which can be summarized in the “M&E matrix”. Here we will briefly outline the M&E matrix, looking at each column in turn.

Table 1.11 Contents of the M&E matrix

Performance questions

Information needs and indicators

Baseline information

requirements status and

responsibilities

Data-gathering methods,

frequency and responsibilities

Required forms, planning, training, data management,

expertise, resources and responsibilities

Analysis, reporting, feedback and

change processes and responsibilities

Step 1. Identifying Performance Questions

Rather than starting with indicators, first identify performance questions. This helps you focus your infor-mation gathering on what you will really use for understanding and improving project performance. Iden-tifying performance questions (indicators and selection methods) will be iterative: make an initial choice, assess its feasibility, accept and use it or reject it and find the next option.

Step 2. Identifying Information Needs and Indicators

Using your performance questions, you can more easily identify useful indicators and other information need for which you will need to collect data. Only data that help answer your performance questions are necessary. This helps avoid collecting information that is difficult to use to guide the project strategy and operations. .

Step 3. Knowing what Baseline Information You Need

Many baseline studies suffer from information overload and lack of use. When deciding whether you need to collect baseline data for a particular performance question, ask yourself if you need to compare infor-mation to be able to answer the question. If not, or if information already exists, then you will not need to collect baseline data.

Step 4. Selecting Which Data Collection Methods to Use, by Whom and How Often

Once you have decided what information is needed and what indicators will be use, you need to decide which methods will be used for gathering the data. You have many options: methods that are more qualitative or more quantitative, more or less participatory, and more or less resource intensive. Each will provide information of varying degrees of accuracy and reliability.

Deciding which methods to use requires balancing these different factors. When you examine the conse-quences of a particular performance questions or indicator, you may need to change it if it is impractical or too expensive. This includes looking at who will be using the method and how often it will be applied. For example, if you have no existing capacity to use your preferred method, you need to plan training – or choose another method if you have no resources for this.

Frequency of collection also needs to be established. This will vary per question and indicators. If one critical indicator needs to be collected often, then you may need to reduce the frequency of another less important indicator or delete it altogether.


33

Box 1.4. Example for gathering information on food security changes

For gathering information on food security changes you could consider three different methods (1) a detailed household survey conducted by independent researchers, (2) a participatory assessment process where women household members do their own monitoring and discuss their

findings, or (3) focus-group meetings to discuss changes that specific social groups have experienced. The first method would the most resource intensive but may have yield the most quantifiable outputs. If well facilitated, the second method can also yield precise results but at a lower cost than the first method and perhaps with interesting discussions from which new project by village women. The third method would yield the least precise and least quantitative information but would be the least resource intensive. Before embarking on resource intensive data-collection exercises, carefully consider whether a simpler method would yield sufficient information of good enough quality for your purpose.

Step 5. Identifying the Necessary Practical Support for Information Gathering

For a method to lead to the information you require, you will need to organize the conditions to make it work. These are often forgotten in the focus on identification of indicators but are critical to success. For each method, consider if and how you need to:

• Develop forms to record data;

• Develop forms, filling systems and databases for collating and storing information;

• Train staff, partners or community members who will be involved;

• Check and validate data;

• Organize external M&E or research expertise that may be needed;

• Agree on responsibilities for different tasks;

• Ensure everyone has sufficient financial resources and equipment.

Step 6. Organizing Analysis, Feedback and Change

In the rush to get out and start collecting data, many M&E units pay insufficient attention to the process of using the information for analysis and directing changes in the project.

To make sure that data will be used – and not just collected – think about how you will organize the analysis of information for each performance question. Sometimes a performance question cannot be answered without prior analysis of several bits of information. Who will do it? When will it happen? Also consider what form information should be in so that it can be used by different stakeholders. For example, will it be useful to present information visually, in graphs or maps? Or do you need to organize several community meetings to get more feedback on the initial analysis of the information?

Most importantly, consider how the generated information can be used to check progress and make im-provements as the projects proceeds.

3.6. Indicative contents for an M&E Plan

The operational plan for project M&E provides the conceptual and, above all, practical basis for plan-ning, monitoring and evaluation within the project. The document also defined the main concepts related to M&E, the redesigned log frame matrix, the adjustment of information collection systems and the database of primary stakeholders.


34

The M&E operational plan will be the reference point for stakeholders throughout the project life. So it needs to be comprehensive enough, at macro level, to provide a clear picture of the overall project intentions and how the M&E system will serve this. At the micro level, it must give fine detail on sched-ules, responsibilities, budgets and so on, which will help guide the drawing-up of AWPBs..

Table 1.12. Indicative contents for an M&E operational plan

Topic description

• Purpose and scope

• Project overview and objectives, rationale and justification for the design of the M&E system

• How the M&E system will support project management and meet the reporting requirements and information needs of different stakeholders.

• Summary of overall experience of M&E undertaken with key stakeholders

• Discussion of extent of participation, balance between qualitative/quantitative approach, resource intensiveness and the intended poverty focus of the M&E system

• Approach Overview of how stakeholders will be involved, what learning-oriented approaches will be used and, in general terms, what information gathering and analyzing methods will be used; for example, the extent of use of participatory approaches, geographic information systems, computer-based informati

M&E Manual GuidE for ifad fundEd ProjEcts in ViEtnaM · 2013-05-11 · M&E Manual GuidE for ifad...

Documents

Transcript of M&E Manual GuidE for ifad fundEd ProjEcts in ViEtnaM · 2013-05-11 · M&E Manual GuidE for ifad...