M&E Handbook 2006

8/6/2019 M&E Handbook 2006

1/59

47 Temple Street, Pietermaritzburg 3201P O Box 157, Pietermaritzburg 3200 E-mail: [email protected]

Tel: +27 33 345 7994 Fax: +27 33 345 7272 Website: www. cindi.org.zaNPO 011-496 PBO 930008976

HANDBOOK

on

for the CINDI Network

Compiled by Liz Clarke, CINDI M&E Advisor

March 2006

Monitoring andEvaluation

The development of this programme and document was made possible

through the generosity of the Rockefeller Brothers Fund

8/6/2019 M&E Handbook 2006

2/59

Contents

1 Background ........................................................................................................................................3

2 The CINDI M&E journey .................................................................................................................... 3

3 The current buzz around M&E ............................................................................................................5

4 From programme cycle to results based M&E ..................................................................................... 8

5 Mapping the journey information needs and indicators .................................................................13

6 Tools for gathering information .........................................................................................................16

7 Critical reflection: thinking through the journey ................................................................................ 21

8 Disseminating information ................................................................................................................ 24

9 Measuring, demonstrating or assessing impact? ................................................................................ 26

10 The basics of evaluation ................................................................................................................... 31

11 Developing an M&E Framework for CINDI Network ........................................................................35

12 Acknowledgements and References ..................................................................................................36

Annex 1: Glossary .................................................................................................................................39

Annex 2: Ten criteria for assessing indicators ......................................................................................... 43

Annex 3: Draft M&E Framework ............................................................................................................ 44

Annex 4: UNICEF Framework ...............................................................................................................59

How this Handbook has been put together ...

The different sections draw on information that was distributed at learningsessions to promote understanding of M&E concepts and processes. Someof the issues raised by participants are also included as there was a genuineeffort to make the system relevant to the CINDI Network and the environ-ment in which it works.

8/6/2019 M&E Handbook 2006

3/59

3Handbook on monitoring and evaluation for the CINDI Network

1. BackgroundChildren in Distress (CINDI) is a network of over 100 concerned organisations seekingto effectively respond to the growing numbers of children affected by HIV and AIDS.CINDIs vision is to be a multi-sectoral well-resourced network of civil society capable ofimplementing diverse, effective, sustainable care and preventative programmes for chil-

dren affected or orphaned by HIV and AIDS. The networks mission is to foster amongpartners a spirit ofUbuntu the principal of caring for each others well being with anattitude of mutual support.

CINDI Members:

acknowledge the AIDS epidemic is a problem too large for any individual, agencyor government department to meet;

have found a working framework for their alliance;

believe children need protection but fare best within their own families, communitiesand circles of friends;

understand extended families, caregivers and communities need external help if theyare to be responsible for meeting the needs of vulnerable children ;

emphasise development, not charity.

As a collection of independent initiatives, CINDI Members retain individual character,flexibility and the ability of individuals to respond to individual children, but are strength-ened and guided by their link to the Network. Their aim is to identify and help childrenin distress. The scale of the epidemic and the poverty surrounding it is overwhelming.In terms of the scale of need available financial support is grossly inadequate. Yet theinnovative elements of CINDIs work are important guides to the challenge of servingaffected children in a manner that is acceptable to communities and sustainable for all

the years it will take to raise a generation of children made extremely vulnerable by HIVand AIDS.

2. The CINDI monitoring and

evaluation (M&E) journeyThe design of the M&E framework is an initiative that follows on from an initial

2 year project funded by the Rockefeller Brothers Fund (RBF) which assisted Members with organisational development, strategic planning and monitoring and evaluation.This funding coincided with an allocation of funding from Irish Aid (formerly known asDevelopment Cooperation Ireland) for operationalising the CINDI Network Office andsupporting selected projects undertaken by nine CINDI Members. A further allocation offunding for 2005-2006 was made to facilitate the development of an M&E frameworkand support the training of approximately 20 Members of the CINDI Network.

Eleven member organisations mostly funded by Irish Aid participated in the pro-gramme.

The exercise was planned and implemented as a participatory process so that participat-

ing organisations help shape the framework to ensure it addresses their needs and reflectsthe realities of the external environment in which organisations work. From the visits

8/6/2019 M&E Handbook 2006

4/59


to the organisations during the inception phase of the project it seemed that some toolshad been developed to monitor programmes. Some of the tools had been developed bytrial and error while other tools were borrowed from other programmes. The stipulatedreporting requirements of the donors also seemed to fulfil an important monitoring role.Thus the development of the CINDI M&E framework has not happened in a vacuumbut builds on experience already gained. The Handbook covers the different elements ofthe framework that were covered by the learning sessions as well as a proposed resultsbased framework drawn up by participants.

Participating organisations shared valuable information on a number of challenges thatthe development of an M&E system would need to contend with. These are summarisedbelow:

2.1. User friendlinessOrganisations generally felt the monitoring tools developed should be user friendly andnot riddled with technical M&E jargon. It was felt that some of the terminology in theliterature is very abstract and the tendency to strive towards alignment with corporate

values was often at the expense of ensuring key development concepts particularly relat-ing to the well beingof children and families. Critical issues of well being are not easilymeasured or accommodated in log frames and reporting formats.

2.2. Data and the location of the data baseMany organisations wanted to have instant access to a very wide range of data as theybelieved this would make monitoring and evaluation easier and would also help themwith funding proposals. However, it was conceded that the capacity (particularly IT skills)to sustain such a system is currently not available in all organisations. It was felt that theCINDI Network Office was an ideal location for such a centralised data base. Critical

issues in the design of M&E framework is deciding on what data is needed, how it is col-lected, how it is organised, how it is stored and how it will be updated and accessed.

2.3. Monitoring non quantitative aspects of the workMany organisations are working with children and families and are endeavouring to pro-vide holistic care for them. This calls for specialised qualitative techniques of monitoringwell being, growth and development which do not translate easily on to the usual M&Eformats. Furthermore, if the M&E system is to provide a tool for empowerment of com-munities and capacitation of CBOs and other community structures innovative tools thatare user friendly for people with low levels of literacy must be developed.

2.4. Linking monitoring and evaluation to planningOrganisations that had developed comprehensive strategic/operational plans seemed tohave made the most progress with the regular monitoring of their work. It seemed mucheasier for them to meet with reporting requirements and also to reflect on their own prog-ress meaningfully. Those organisations that had grasped and implemented such planningand monitoring systems seemed to enjoy working with them.

2.5. Gaps between management and fieldworkersThere sometimes seemed to be a gap between fieldworkers who contended on a daily

basis with a lot of difficulties in their work and managers who wrote reports and focussedmostly on the outputs which did not always represent the realities in the external envi-

8/6/2019 M&E Handbook 2006

5/59


ronment. Such a gap may also exist between NGOs and the CBOs they mentor. Thereseems to be a we and they mindset which may result in the monitoring tools havinglimited value as an instrument of planning and change.

2.6. Issues of capacityThe AIDS pandemic, growing poverty, unemployment and bottlenecks, inefficiencyand corruption in the public sector programmes have combined to put a great deal ofstrain on organisations that are trying to maintain high standards while at the same timeexpand their work. NGOs have difficulty retaining the services of professional peoplebecause salaries are not really competitive. Organisations want to see the M&E systemset up and working but do not want too many additional demands on their time to betaken up servicing the system.

2.7. Involvement of beneficiaries in M&EIn some programmes, beneficiaries e.g. families, are involved in the assessment of theprogrammes. Some organisations wanted to try to expand the involvement of partici-

pants in the monitoring and assessment of the programmes.

2.8. Gender equalityMost organisations had complied with the requirement that a Gender Policy be put inplace. However, there was a view that it had been imposed and was not necessarilyrelated to the realities within which most organisations work. Generally the internationalbest practice of integrating a gender perspective into both the internal and external workof the organisation was not really understood or was viewed as unnecessary because ofhow some organisations are already structured (e.g. most home based care workers arewomen).

3. The current buzz around M&E

3.1. The buzz around M&EIn recent years there has been an upsurge of thinking around monitoring and evaluation.With the adoption of the New Public Management Framework during the 1990s, cor-porate values and performance management frameworks were increasingly adopted bygovernment and development organisations and eventually this became a requirementby funders of NGOs. Project management, financial management and organisationaldevelopment have become fields of specialised knowledge and the expectation is thatNGOs reflect these concepts in their management and work.

It was assumed that rigorous monitoring and evaluation would ensure governmentofficials, development managers and NGOs improve service delivery, the allocation ofresources and planning. It would also ensure greater accountability in the use of donorresources. Almost all major initiatives such as those supported by the Global Fund, andinternational frameworks for responding to HIV and AIDS and poverty have very elabo-rate M&E systems. M&E has thus become a highly specialised field forming an integralpart of strategic planning and programme management. However, it is still viewed by

some as a stand alone which must meet funders reporting requirements. For manysmaller NGOs with limited capacity the new M&E requirements have proved some-

8/6/2019 M&E Handbook 2006

6/59


what daunting. For CBOs and community structures working in lesser developed areasthe challenge of meeting M&E requirements is a critical one as they are already beingexcluded from accessing much needed funding because of their low levels of capacity.

Amongst those development workers who understand and support people centred devel-opment there is a growing uneasiness with some of the rigorous reporting requirementsof funders. The concerns arise from the shift to a preoccupation with service delivery

andproductat the expense ofdevelopment process which shifts power relations in thefavour of the less powerful.

In the learning sessions there was much discussion about the need to create opportuni-ties for local community structures to fulfil a more empowered role in the response to HIVand AIDS and poverty. It is hoped that the framework will ultimately reflect not only anM&E system that improves accountability and informs planning but that also alters thepower relations within the CINDI Network and mobilises and integrates CBOs and localcommunity structures into the mainstream of the response to HIV and AIDS and poverty,especially from the point of view of children in KwaZulu-Natal.

3.2. Defining M&EDefinitions of monitoring and evaluation differ widely but in the context of the work ofthe CINDI Network the following definitions capture the scope of the M&E Framework.

Monitoring refers to the systematic and continuous collecting and analysing of infor-mation about the progress of a project over time which helps an organisation assessits effectiveness in achieving its goals. Information gained from the monitoring processshould be integrated into the planning process.

Evaluation refers to a systematic assessment of a planned, ongoing or completed proj-ect which aims to answer specific management questions. Information gathered shouldhelp organisations assess what has been learnt which can inform future planning andactions. Issues of efficiency, effectiveness, sustainability and relevance of an organisa-tions programme objectives are commonly central to evaluation exercises.

3.3. The M&E System1

M&E is no longer an add on to a project or programme but requires the development ofa system and an M&E Plan.

Setting up the system requires the following steps:

1. Defining the purpose and scope of the M&E systemWhat is the purpose of the M&E system and how comprehensive should it be?

2. Identifying performance questionsWhat do we need to know to monitor and evaluate interventions and to manage

projects well?

3. Planning information gathering and organisationHow will the required information be gathered and organised?

4. Planning critical reflection processes and eventsHow will we analyse the information and use it to make improvements?

5. Planning for quality communication and reporting

How and to whom do we want to communicate what in terms of our project activi-ties and processes?

1 The design of theframework drawsheavily on thework of IFAD asoutlined in IFAD.2002.IFAD Managingfor Impact

in RuralDevelopment.Rome. Italy

8/6/2019 M&E Handbook 2006

7/59


6. Planning the development of the necessary capacitiesWhat is needed to ensure our M&E system actually works?

3.4. Different M&E Models

3.4.1. Traditional M&E

Traditionally M&E focussed on inputs (resources) and outputs (goods and services) Datacollection was haphazard and there was no systematic and long term plan.

3.4.2. Results Based M&E

The shift has been away from inputs and outputs towards outcomes and impacts. Thisshift comes about as a result of the growing complexity of development issues, growingcompetition for scarce resources and a growing need to integrate with higher level strate-gic objectives e.g. the Millennium Development Goals. However, measuring/assessingoutcome and impact is still an area in which new thinking is emerging all the time

3.4.3. Participatory M&EAs a result of HIV and AIDS and poverty, the situation in KwaZulu-Natal requires con-tinual commitment to scaling up programmes especially in lesser developed rural areasand poor urban informal settlements. This calls for the greater involvement of peoplewithin these communities in all aspects of project planning and development, monitoringand evaluation.

Increasingly participatory methodologies are being used to ensure local communities arefully engaged in development activities that affect themselves.

Conventional M&E focuses on measurement of results of activities such as service deliv-

ery, information dissemination, behaviour change etc.

Participatory M&E focuses on processes and results such as inclusion, collaboration, col-lective action. It encourages dialogue at local level and attempts to transform local peoplefrom being passive beneficiaries to active participants.

3.4.3. Developmental M&E

Developmental M&E places great emphasis on changes within people as indicators ofdevelopment. These changes are critical to development but cannot be easily measuredor log framed. A couple of important areas of change in people are listed below:

Relationships

More solidarity openness and trust

Mutual co-operation

Integrity and credibility

Understanding

Understand their world internal and external

Understand their needs

Ability to reflect and learn from experience

8/6/2019 M&E Handbook 2006

8/59


3.4.4. Integrating different approaches

The CINDI M&E Framework is strongly driven by the Results Based Model but if it isto ensure it promotes development and empowerment it needs to integrate aspects ofparticipatory and developmental M&E. This will be crucial to building capacity of theCBOs and scaling up.

4. From programme cycle to results

based M&E

4.1. The Programme CycleInitially the Programme Cycle formed the basis of conceptualising project and pro-gramme work. It was included in the learning sessions to help participants understandhow monitoring and evaluation was originally conceptualised in the programme cycle.

The main phases of the programme cycle included the following:

1. Needs assessment which is essentially an exercise to get information about com-munity problems. This information has generally made some assumptions abouthow the needs/problems could be addressed. This was later expanded to being asituation analysis which included needs and resources.

2. Planning uses those assumptions to draw a programme of activities. This later gaveway tostrategic and operational planningwhich is an integral part of current pro-gramme development

3. Implementation refers to the follow through of the activities in accordance with

the planning. Implementation is now expressed in operational or action planswhich commonly outline activities, time frames, responsibilities and budgets for theachievement of various objectives.

4. Monitoring and review and evaluation are processes which originally assessedthe use of inputs and successful completion of activities (outputs) in accordance withthe plan.

Needsassessment

Implementation

Planning

Monitoring

Review andEvaluation

8/6/2019 M&E Handbook 2006

9/59

8/6/2019 M&E Handbook 2006

10/59


1. Inputs and processes/activities

This refers to the resources allocated and processes used to carry out an activity,project or programme. Inputs commonly include resources such as funds, equip-ment, training aids etc. Processes refer to the activities used to achieve the objec-tives such as training, capacity building, service provision, information disseminationetc.

For example: A project may have an objective of training home based care workers.

The funds and training aids are the input and the actual training is theprocess/

activity.

Inputs usually produce results immediately (01 years).

2. Outputs

This refers to the consequences of the inputs and processes.

For example: The outputs of the above project may be 10 home based care workers

having an expanded understanding of issues relating to those in need of home based

care.

Depending on the nature of the project outputs usually reflect a result achieved ina relatively short time period (02 years).

3. Outcomes

This refers to recognizable changes in conditions in the targeted area. Outcomesmay reflect behavior or economic change, or expanded service delivery and helpus assess whether the activities have contributed toward development outcomes.Outcomes go further than outputs and attempt to answer the question so what?In this way one gets some understanding of the longer term consequences of the

outputs.

For example: 10 people have increased knowledge as a result of their training as

home based care workers. So what? What difference has the training made to those

infected or affected by HIV and AIDS? What changes has this brought in the targeted

area?

Outcomes usually reflect a result achieved over an intermediate time period (25years).

4. Impacts

Impacts are the overall and long-term effects of an intervention. Impacts are theultimate result probably attributable to interventions over an extended period.

For example: After 5 years the 10 home based care workers may have left the

programme and thus the long term impact may be negligible alternatively they

may have formed the nucleus of a comprehensive programme which contributed to

improving the quality of life of those affected and infected by HIV and AIDS.

Depending on the nature of the programme impacts usually reflect a result achievedover a longer time period (5-10+ years.)

8/6/2019 M&E Handbook 2006

11/59


Inputs/Processes Outputs Outcomes(So what?)

Impacts

Resources andprocesses utilised toproduce a result

Short term changeeffects and results

Intermediate changeeffects and results

Long term changeeffects and results

Inputs commonlyinclude:

Staff

Funds

Equipment

Facilities

Training materials

Processes/Activities:

Training

Mentoring

Communitymobilisation

Establishment ofservices

Establishment ofnetworks

Research

Outputs commonlyinclude:

Knowledge,awareness

understandingchange

Improved orexpansion ofaccess to services,programmes,networks,materials,information

Quality change(improvedprogrammes or

services)

Outcomes commonlyinclude:

Increased coverage oftarget populations by

interventions leadingto:

Behaviour changesuch as improvedHIV and AIDSprevention practice

Improvement ineconomic situationof children/caregivers

Impacts commonlyinclude:

Quality of lifechange

Overall healthstatus changedsuch as increasedsurvival

Politicalchange humanrights policiesimplemented

Socio/cultural/economic andempowermentchange,

Resourcemanagementchange

which leads to ... which leads to ... which leads to ...

What participants said during the learning session

Participants felt the Result Based Management Framework is logical, focussed and futureoriented. While the Logical Framework was currently used by many funders and partici-

pants it was felt that it had many weaknesses. There was a concern with regard to the time frame around impact. Some interventions

may be able to demonstrate impact in less than five years but there were reservationsabout the longer term impacts. Participants felt that there will be a need to re-evaluate orre-assess the changes. Some organisations providing care for children were concernedthat it would only be in adolescence and early adulthood that one would be able to trulyassess the impact of the care received in childhood.

Some of the terms seem to be used interchangeably and some of the funders including thegovernment departments seem to use different terminology altogether.

Consensus was reached that there is a need to have a system that works for CINDI

Members and meets the requirements of funders.

4.3. The results chain

8/6/2019 M&E Handbook 2006

12/59


Reflections on a CINDI Big Issue!

The issue of collaboration

CINDIs mission is to foster a spirit of Ubuntu amongst CINDI Members,encourage collaboration, coordinate research and unlock resource opportu-nities for the benefit of all Members

Participants raised questions about the meaning of collaboration

- What is collaboration?

- Are we collaborating?

- How do we collaborate?

- How do we know if we are really collaborating?

Collaboration defined

Collaboration basically means working together. At a community level col-

laboration is an essential part of achieving project goals. Individuals andorganisations rely on collaborative efforts and resources of others to achievemeaningful and sustainable outcomes. Collaboration can also be an out-come in itself, as it can be a key element for creating social infrastructurethat is sustainable beyond the duration of the project.

The main activities in the context of CINDI that facilitate collaborationinclude:

Networking and facilitating dialogue to ensure common understanding.

Disseminating information and creating a base for support.

Cooperation. Aligning needs to limit duplication, clarifying roles, modify-ing activities to ensure tasks are completed and goals achieved.

Coordination. Sharing resources, formalised links, group decision mak-ing such as the clusters, remaining focused on CINDI priorities.

Coalition. Shared resources from the existing systems, formalised agree-ments, Shared decision-making and the development of new resources.

Multi-sector collaboration. Shared vision, risks and rewards, highlydeveloped communication.

8/6/2019 M&E Handbook 2006

13/59


5. Mapping the journey information

needs and indicators

5.1. Why are indicators important?

Indicators are quantitative or qualitative factors that provide a simple and reliable meansto measure achievement, to reflect the changes connected to an intervention, or to helpassess the performance of a development actor. They can also serve as an early warningsystem to indicate future outcomes and the need for corrective action.

5.2. Counting measures versus proxy indicators

5.2.1. Indicators as counting measures

An indicator is ideally meant to provide evidence of change in conditions in a relativelysimple form ofcountingmeasures.

For example, number of home-based care workers trained, number of families being

supported with food parcels, etc.

5.2.2. Proxy indicators

Alternate or indirect measures are sometimes used to stand in for another indicator whenusing counting measures is too difficult, time consuming, or sensitive.

For example, the number of people undergoing VCT could be seen as an indirect mea-

sure of awareness. These are referred to asproxy indicators.

5.2.3. Indexed measuresSome results are complex enough or have enough identifiable steps that they lend them-selves to being measured by an index. An index value is achieved by developing a scaleof results and assigning points to the level of completion of each, and then adding thetotal to determine the level of measurement.

For Example:

Indicator: Level of CBO Capacity in Project Planning

Directions: Score the following statements on a scale of 1 to 5, with 1 indicating little

or no capacity and 5 indicating extensive capacity. Average the score to determine the

organisations project planning capacity.

CBO can formulate a project plan

1. Project planning is done in a participatory manner.

2. Project plan is available in writing with appropriate levels of formats and details.

3. Project plan outlines achievable and appropriate goals, objectives and activities.

4. Project plan reflects appropriate linkages with other key stakeholders.

5. Project plan includes a budget appropriate to the scope of the project.

6. Project reflects a basic plan for monitoring and evaluation.

Indexed measures are very useful in many field programmes when a complex series of

activities that jointly respond to an anticipated result for instance, quality of life of vul-nerable children and quality of patient care need to be measured.

8/6/2019 M&E Handbook 2006

14/59


5.3. Steps in selecting indicatorsDeveloping appropriate and useful indicators is a fairly straightforward process, butrequires collaboration and consensus-building and needs to be interactive. The follow-ing suggestions can be helpful in the selection process. Although presented as sequentialsteps, some can be effectively undertaken simultaneously.

Step 1: Clarify the results statements; identify what needs to be measured.Good indicators start with good results statements that people can understand and agreeupon. In developing a good results statement the following is important:

Desired results should be carefully considered

What is expected to change a situation, a condition, the level of knowledge, atti-tude, behaviour?

Is change expected to occur amongst individuals, families or communities?

Before developing indicators be clear about the relationship between the activitiesand their intended results.

In the context of poverty and HIV and AIDS in KZN there are a number of factors thatcan impact on results.

Step 2: Create an initial list of possible indicators by:

Internal brainstorming

Identifying and building on the experience of other similar organisations

Consulting with beneficiaries and experts

Identifying existing secondary data sources (data collected by someone else that youuse to measure a result). Key data should be available from the local municipalityand Statistics South Africa.

Referring to those developed by the CINDI Network and existing indicators devel-oped by UNICEF, MDG, Global Fund.

Step 3: Select the best indicators.

Consider carefully what the indicators should be achieving and narrow the list to thefinal indicators that will be used in the monitoring system. Remember to take into con-sideration the realities of the environment you work in and the costs and time involvedin gathering information. Limit the number of indicators used to track each objectiveor result to two or three. Select only those that represent the most basic and importantdimensions of your objectives.

Step 4: Draft indicator protocols.

Protocols are instruction sheets that capture the reason for selecting indicators, describethe indicator in precise terms, and identify the plans for data collection, analysis report-ing and review.

Step 5: Collect baseline data.

The baseline is a record of what exists in an area prior to an action. The baseline valuesestablish the starting point from which change can be measured.

Step 6: Refine indicators and protocols and finalize your selection.

Based on initial data collection efforts, refine your indicators and/or your data collectioninstructions.

8/6/2019 M&E Handbook 2006

15/59


What participants said and did during the learning session

1. The participants drew up a list of indicators for the CINDI Network. Thesewere aligned to the specific needs of CINDI, as well as making use of theUNICEF Framework for responding to children in a world of AIDS.

2. There was a lot of discussion around the indicators particularly aroundthe issue ofimproved quality of life which is summarised below:

Improved educational status of children should include additional indi-cators about absenteeism and age for grade.

Emotional well being of vulnerable children should include an indicatorabout children having access to a caring, consistent significant other.

Improved protection of vulnerable children should include an indicatorabout the proportion of children aware of childrens rights and avail-able resources.

The issue of spiritual and cultural well being as an integral part ofquality of life was discussed at length. Some of the points put forwardinclude:

African culture is in such a state of flux that it would be difficult toidentify specific patterns of behaviour and label them as cultural.

Self esteem and a sense of belonging were closely linked with cul-tural attributes derived from the immediate social environment

Spirituality deriving from whatever belief system helped peoplecope when the going was tough and therefore was very impor-

tant

What was critical was that children growing up should exerciseinformed options which would sometimes embrace western cultureand sometimes embrace traditional culture. The question was whowould help children understand the options?

It was felt that the output indicatorincreased quality of service and careshould also be represented as an outcome.

3. The indicators outlined should also be linked with the work of the fiveCINDI Network Clusters i.e. the Psychosocial Cluster, the Community

Development Cluster, Children in Care Cluster, School and YouthDevelopment Cluster, and the Home Based Care Cluster.

8/6/2019 M&E Handbook 2006

16/59


6. Tools for gathering information3

There are a number of tools that can be used to gather information for monitoring andevaluation purposes. Some tools are designed to be used to assess information relatingto specific focus areas. The table below outlines a number of tools and the specific focusareas in which they are most appropriate. Their purpose and use in a monitoring and

evaluation context including how they may be useful in the context of CINDI is set outbelow. The steps for using these tools are also set out.

Method Purpose and possible rel-evance to CINDI Network

Steps

Core M&E MethodsThe 10 methods that follow reflect the standard core methods commonly used in M&E exercises

StakeholderAnalysis

In the context of M&E this helps decidewho should be involved in the design ofan M&E plan. It is helpful in providing a

foundation and strategy for participationthroughout the project.

In the context of CINDI it is critical toidentify the range of stakeholders thatneed to be included in the planning,implementation and M&E. It is particu-larly important to include project ben-eficiaries and representatives from localcommunity structures.

1. Clarify the main purpose of the stakeholderanalysis

2. Decide on criteria for including stakeholders3. List all organisations and individuals that fit your

criteria

4. Reach agreement on how to involve people

DocumentationReview

This helps to understand the historicalevolution and performance of a project.In this way important baseline informa-tion can be obtained which helps toinform the development of an M&E sys-tem.

The changing role of CINDI in responseto a changing external environment high-lights the need for on going transforma-tion and alignment which is captured inthe documents.

1. Be sure about which issues you want to under-stand better

2. List all possible sources of existing information

3. Prioritise those most likely to provide therequired information

4. Collect documentation and check reliability-analyse it in terms of the questions you weretrying to answer

5. Identify other collection methods to address gaps

Biophysicalmeasurements

This helps measure physical changesrelated to any indicator e.g. weight forage etc. From an M&E perspective it canprovide statistically reliable informationfor measuring change and impact.

In the CINDI context the use of healthrelated indicators are important formonitoring the progress of vulnerablechildren.

1. Be sure about the information being provided bythe indicator

2. Ensure availability and standardisation of equip-ment

3. By comparing recorded data, comparisons can

be made over a period of time

4. The Road to Health cards for under fives frompublic sector clinics are useful in monitoringhealth related progress

Direct observa-tion

To obtain useful and timely informationby observing what people do or behav-iours. From an M&E point of view thismethod is often used to complement col-lected data, understand context and helpexplain results.

In the CINDI context this is an importanttool for psycho-social assessments and

the work of the Psychosocial Cluster

1. Agree on guidelines for what needs to beobserved

2. Choose an appropriate observer or observers

3. If necessary train in observational skills

4. Collect and record data as agreed

5. Organise time to discuss observations with rel-evant stakeholders

3 This section isadapted from thework of IFAD asoutlined in IFAD.2002.IFAD Managingfor Impact

in Ruraldevelopment.Rome. Italy

8/6/2019 M&E Handbook 2006

17/59


Cost-BenefitAnalysis

To provide a format for assessing therange of benefits and costs surrounding adecision. In the M&E context this methodis commonly used to evaluate a projectby comparing final costs and benefitsagainst those proposed in the design ofthe project.

In the context of CINDI it is importantfor assessing the value of training andmonitoring costs of services.

1. You may need someone with the relevant finan-cial background

2. List all project activities

3. Calculate all project costs for the period

4. For each project activity estimate the benefitswhich may continue into the future

5. If possible include beneficiaries so that they toocan comment on outcomes

Questionnairesand Surveys

To gain data from a large number ofpeople in a structured way. Commonlyused in M&E exercises as they allow forfocussed data collection around indica-tors.

Potentially a very important tool formonitoring and evaluating programmes

within the CINDI Network.

1. Agree on purpose and information needs of thequestions

2. Decide whether a questionnaire (just a set ofquestions) or survey (which includes questionsand observations is required)

3. Ensure questions are focused and well formu-lated

4. Agree on target group and number of respon-dents to be included in sample

5. Pretest questionnaire to ensure they are appro-priate and give required information

6. Collect and use information

Semi-structuredinterviews

To gain information face to face from anindividual or small group using a seriesof broad questions to guide the con-

versations but allowing for new ones toarise during the course of the discussion.From an M&E perspective this tool helpsto develop an in depth understanding ofqualitative issues.

Semi-structured interviews could be veryuseful when trying to understand atti-tudes, behaviours and beliefs. They canbe used in conjunction with other meth-

ods. They therefore could be a useful toolfor the CINDI Network.

1. Define the purpose and information needs ofthe inquiry and formulate an interview checklistof open-ended questions. The questions shouldencourage expression of opinions

2. Agree on who and how many should be inter-viewed

3. Train people to understand the purpose anddevelop the proper skills

4. Pretest the interview questions

5. Undertake the interviews and analyse outcomes

Case Studies To document life stories or sequenceof events over a period of time relatedto person, location, and organisationin order to understand impact. Froman M&E perspective they help to give aface to data and help to reflect contextof some of the data.

A very important tool for CINDI to con-textualise local realities. Members shouldattach case histories to reports to give anidea of process and context remembering

to respect confidentiality at all times.

1. Define the purpose and precise informationneeds of the case study

2. Decide how individuals, households, organisa-tions etc will be selected

3. Decide how the information will be obtainedfrom households, individuals etc.

4. Develop question checklist to guide informationcollection

5. Collect information and compile case history

Discussion Methods for GroupsGroup based methods are important when working with participatory projects. Basically the tools are used toencourage discussion and sharpen the focus on issues. In this way they can help an organisation gain an under-standing of community perspectives and build these into M&E frameworks.

Focus Groups To use group discussion to collect gen-eral information, clarify details or gatheropinions in a smaller group situation.In the M&E context they are useful forassessing opinions about change, orassessing the quality of project servicesor providers and identifying areas ofimprovement.

1. Determine the participants (4-8 people is ideal)which can be homogeneous or heterogeneous

2. Present the group with a broad question e.g.what impact is the CINDI mentoring programmehaving

3. Discuss the question for the time agreed before-hand (one or two hours maximum) with minimal

intervention from the facilitator

8/6/2019 M&E Handbook 2006

18/59


Focus Groups(continued)

Focus groups can be very important inassessing perceived outcome and impact.The tool can be used to help get infor-mation during the planning stages of anintervention. For this reason in the con-text of CINDI they can used to assesslonger term outcomes and impact of pro-grammes.

4. Take detailed notes of the discussion. Focusgroups are best if facilitated in pairs one facili-tates and one takes notes

5. To ensure reliability it is suggested that differ-ent focus group discussions be held around thesame topic

SWOT Analysis

(Strengths,Weakness,Opportunities,Threats)

To identify the strengths, weaknesses,

opportunities and threats in relation toa project or group and how such anassessment will change over time. Froman M&E perspective this method is useful

when qualitatively assessing the servicesprovided by a service provider, exploringrelationships etc.

This tool can help CINDI when involvingstakeholders in participatory processes.

1. The group defines, discusses, and records as

many factors as possible under the variousheadings

2. One way of using the headings is as follows:Strengths: Aspects that are working well in aprojectWeaknesses: Aspects that have not worked sowell in a projectOpportunities: How to build on strengths andovercome weaknessesThreats: The aspects that constrain or threatenthe range of opportunities

Dreamsrealised or

visioning

To have informed discussion aboutpeoples dreams or shared visions for

the future of the project or other activity.From an M&E perspective this is a goodmethod for identifying if primary stake-holders feel their well being is increasingor not or whether a project is based onpeoples visions for development.

This is a powerful tool for capturing com-munity opinions and understandings,planning and developing a M&E systemall in one.

In the CINDI context this tool is veryrelevant and can be used to help planprojects.

1. Start by asking people through meetings heldwith individuals, households, communities

etc how they would like to be in the future inresponse to certain focus areas

2. In practical terms ask for people to reflect on anindividual basis for about 15 minutes and thenget sharing in sub groups so that some consen-sus emerges

3. The dreams can be written down or representedas a symbol. Dreams can be specified with cleartime frames for achievement

4. Once articulated dreams can be expressed asindicators

5. The discussion can be repeated every 6-12months as a means of monitoring progress

Drama or roleplays

To encourage groups of people to enactscenes from their lives concerning per-ceptions, issues and problems that haveemerged relating to a project interven-tion. From an M&E perspective theycan also help a group identify possibleindicators for a project. A very valuabletool when working with rural people whoseem not to articulate problems as wellas they can perform them! Drama androle plays can often create a non con-frontational way of dealing with differ-ences e.g. generational and gender.

In the CINDI context this tool is valuablemeans of capturing local peoples per-ceptions of development challenges.

1. Choose the central theme whether it is an M&Eindicator or question or scenario to be drama-tised

2. Decide who is to work on a drama piece or getdifferent groups e.g. women, young people,elderly each to present their views on a topiclike the prevention of HIV and AIDS

3. Participants should construct their drama, inwhich they present their opinions and thoughtson the topic being discussed

4. Facilitators should record key issues in the per-formance

5. The group then discusses the issues emergingfrom the play and conclusions drawn

Methods for Spatially-Distributed InformationThese methods deal with information that has some level of geographic distribution which could relate to health,education or economic issues. Maps can help locate biophysical, economic and social indicators. They can be veryelementary such as drawn in sand at a neighbourhood meeting or make use of GPS systems. (CINDI Membersshould bear in mind Valley Trusts offer to integrate information into their GIS system)

Sketch mapping To provide a visual representation in aparticular geographical context based onstakeholders perceptions of any focusissue or indicator that is being monitored

and evaluated.

1. Ask participants to draw boundaries of the areaunder discussion. This can be represented onpaper or using local materials such as sticks,stones seeds etc.

8/6/2019 M&E Handbook 2006

19/59


Sketch mapping(continued)

Perhaps in the HIV and AIDS context itis important to map areas that are notbeing adequately served or lack waterand sanitation.

In the context of CINDI this tool maybedifficult to implement and maintain but itis important that projects reach the most

vulnerable households and communities.

2. Participants should draw the main features suchas roads, rivers etc. and reflect what is of inter-est to them or significant to the developmenteffort.

3. Once a base map is in place and translated intoa paper map it can be used for social mappingof households and the monitoring and evalua-tion of the spread of programmes

Transects To undertake a structured walk through

an area to observe particular indicators(such as places that present health andsafety risks or difficulty in accessing ser-

vices).

In the context of CINDI this can reallyhelp NGOs and CBOs to develop a com-mon understanding of local situations. Ithas the potential of being an importantmonitoring and evaluation tool. It canalso help service providers develop acommon understanding of issues withlocal participants.

1. After noting topics and indicators to be observed

decide who could provide relevant information2. If a map is available use it to identify what

the route will be. (Can be an hour or a daydepending on the size of the area)

3. Already available information should be usedto form the basis of observations and measure-ments during the walk

4. As the walk proceeds the observations shouldbe probed to give further insights and should berecorded

5. Record what has been seen and discussed on aschematic diagram to be used for future moni-toring transect walks

6. The frequency of the walks can vary to provide ameans of monitoring and evaluation

7. Compare the different observations over time togive an indication of progress

Methods for Time Based Patterns of ChangeThese methods help us understand changes related to specific blocks of time e.g. September of one year toSeptember of another year or simply critical events that have occurred during the time span of a project.

Diaries To help record events, facts, reactionsand/or opinions over time by individualstakeholders or groups. From an M&Eperspective, this method is useful forcapturing details that might otherwise be

missed and that might explain the contextin which change occurred. The methodmight promote understanding of how achange came about or focus on specificperformance areas or indicators.

In the context of CINDI this tool is prob-ably a useful means of gathering infor-mation to support statistical informationespecially at local community level.

1. Introduce the diary early in the life of the project

2. The form and focus of the diary and who recordsthe entries

3. The diaries can be individually written documen-

tation or based on group discussion4. Diaries can then be used to stimulate discussion

and identify changes that might be particularlysignificant

5. Can be used in conjunction with other statisticalmethods

6. Reviewing the diary can also be a helpful meansof supervision

Historicaltrends andtimelines

To obtain historical understanding ofsequential changes that have occurred,relating to particular points of interest.From an M&E perspective this could focuson specific indicators which can be usedas triggers in discussions to assess certainchanges that can be attributed to projectactivities.

In the context of CINDI this can be avery helpful tool to monitor communityperceptions of change and historicalaspects of problems. Particularly in rela-tion to the changing environment in

which CINDI works this could be veryhelpful e.g. changes in attitude to VCTfollowing the introduction of ARVs.

1. Agree on events/indicators important to the situ-ation at hand. (e.g. the prevalence of AIDS)

2. On a large sheet of paper draw rows and col-

umns to make a matrix. List dates at the tope.g. 1 year ago, 5 years ago, 10 years ago

3. Write in key events/influences that might haveimpacted on the spread of the disease

4. Work with groups to fill in the information anduse it for discussion

5. You might add a fourth column and label it thefuture in which people identify what they wouldlike to see

8/6/2019 M&E Handbook 2006

20/59


Methods for Analysing Linkages and RelationshipsFundamental for all projects is an understanding of changes in relationships and linkages between groups, such asprimary stakeholders and organisations and also between issues, activities, causes and effects. This cluster of meth-ods provides ideas on how to analyse such issues by using different visualisation techniques. (Perhaps very impor-tant for CINDI in relation to their members and their members in relation to CBOs).

Rich Pictures orMind maps

To make a pictorial representation of theelements that need to be considered or

are important to a particular (project) sit-uation, including stakeholders and issues,and the interactions and connectionsbetween them. From an M&E perspec-tive, a rich picture can help identify whataspects of a situation need to be moni-tored, which change indicators to trackand/or which key stakeholders need tobe included in the M&E effort.

In the context of CINDI this may be auseful tool for analysing relationships

within the CINDI Network.

1. Using a large sheet of paper and symbols, pic-tures and words, draw a rich picture (or mind

map) of the situation (project/group) that youwish to evaluate. This is best done with about 4to 8 people and it takes a half to 2 hours

2. Start by asking people to note all the physicalentities involved, for example, the critical people,organisations or aspects of the landscapes

3. Ask people to present their rich picture bydescribing the key elements and key linkagesbetween them

4. If there is more than one group, compare theirpictures and cluster the ideas that are similarand those that diverge. In this way you can iden-tify the most important issues to discuss, such as

critical topics to focus on in an evaluation, pos-sible indicators or key stakeholders to include inM&E

M&E Wheel(Spider Web)

To provide a visual index that helps inassessing the issue being monitored orevaluated in terms of its ideal, or in com-paring two or more monitoring sites andhow they change over time. This methodcan also be used to measure how well aproject is meeting anticipated targets, orhow an organisations capacity changesover time. From an M&E perspective, thespider web provides a visual means of

measuring changes in ratings on chosenindicators.

In the context of CINDI this tool could beuseful for analysing transformation andrelationships within the CINDI Network.See example.

1. Make sure that the topic being assessed is clear.For example, the idea of the capacity of anorganisation must be very clear and understoodby all of the participants. Have the participantsagree on which criteria to use to assess thequality of the topic. These are, in fact, the indi-cators

2. The selected indicators are arranged in theform of a wheel, with each indicator being onespoke as on a bicycle wheel. The spokes arespaced equidistant to one another. The indica-tors can be represented by words or symbols

3. Next, participants agree on how to rank eachindicator from 0 as the lowest/worst level to100 (or 25, 10, etc.) representing the highest/best level. It does not matter if 0 is on the outeredge of the wheel and 100 in the centre or theother way around, as long as all the spokes onthe wheel are the same

4. Once the wheel has been made, assess eachindicator. If doing this with a group, then there

will need to be consensus on the final score(or an average figure). Indicate the place on

the spoke that corresponds with the final scoregiven. Then join all the scores, which aremarked as points on the spokes, to show whatends up looking like a spider web. A look at thespider web gives a quick overview of key weak-nesses and key strengths. The weaker aspectsof the issue being assessed are those that havescores closest to 0

5. Previously made wheels can be revisited at sub-sequent monitoring sessions in order to comparehow the situation changes over time

Assessment of Organisational

Competence of CBO

0

10

20

ProjectManagement

Financial management

Community linkages

Alignment

M&E

Communication

Fundraising

Capacity development

Areas oforganisationalcompetence

Organisationalscore

8/6/2019 M&E Handbook 2006

21/59


7. Critical reflection: thinking through the

journey

7.1. What is critical reflection?

Critical reflection means creating the time and space in a project to reflect on experiences and data andlook for new insights into the situations the interventions are addressing.

All too often the commitment is to collecting, processing and reviewing of data that is part of the M&Esystem. Critical reflection goes beyond all the data and reports and asks the question, What is really hap-pening in this project?

Other questions which can be asked are:

Why is it happening?

What are the implications for the project?

What do we need to do next?

Some of the issues that are of particular importance in critical reflections include:

What assumptions were made about the project during the planning stages?

Are they still valid?

Are the interventions having the predicted outcome? If not, why not?

7.2. The importance of critical reflection for M&EM&E is often viewed as primarily an administrative/bureaucratic obligation to meet the needs of thefunders. It is not often viewed as an exercise to benefit the project. Commonly there is little appreciationof what can be gained from the frank analysis of projects with disappointing outcomes. This is of particularvalue when further planning is being considered.

Changing attitudes towards the value of M&E for projects and project staff requires a commitment tobuilding capacity, sensitisation to the realities in the internal and external environment and putting in placeeffective incentives.

Project stakeholders will improve their interventions by regularlysetting aside time to reflect on the data andacknowledge the lessons that are being learnt.

7.3. Process aspects of critical evaluation

Project M&E requires planning a series of reflective events including regular staff meetings and informalsessions to more formal supervision sessions and mid term reviews. The events whether self organisedor formally organised should happen alongside the data gathering exercises and make use of some of thedata gathered. The learning will evolve as participants explore important issues and question assumptionsin a group setting.

7.4. Importance of individual reflectionWhile group processes are important, individual reflection should not be overlooked which includes someof the following activities:

Review and reflection of job descriptions

Encouraging reporting that reflects their opinions particularly in relation to achievements, problemsand proposed solutions,

8/6/2019 M&E Handbook 2006

22/59


Regularly asking project stakeholders their views

Providing constructive feedback

Seeking feedback from the people you deal with if you are a manager

Valuing field visits and exchange

Rewarding critical reflection

7.5. Capturing lessons learned with stakeholdersIn the current climate wherein a number of new approaches are being explored there area lot of opportunities to reflect on the lessons learntwhich it is important to recogniseand document.

When formulating a lesson learnt consider the following:

Include a generalised principle that can be applied to other situations

Explain the lesson in the context of the project

Justify the lesson by means of providing evidence of the learning

Check the lesson is not too general or too specific to be useful

Lessons learnt should be documented with at least five elements:

The theme of the lesson learned

What was the original understanding or assumption

What is the revised understanding or assumption

One or two examples to demonstrate the revised understanding

How the project obtained this insight

7.6. Developing a plan for critical processes and eventsCritical processes and events should be planned throughout the life cycle of the projectand a plan for such events should form part of the M&E system.

Critical reflec-tion Processesof Events

Purposes and descrip-tion

Whom to Involve Timing

Participatoryreview of proj-ect strategy

Update the situationanalysis, revise problems/

visions and assumptions

Representatives ofintended primarystakeholders, proj-ect staff and facili-

tator

Three dayworkshop atstart up

Development ofM&E Plan withstakeholders

Assess different informa-tion needs, take note of

who is doing what agreeon priorities, revisit indica-tors, agree on responsibili-ties

Representatives ofintended primarystakeholders, proj-ect staff and facili-tator

Four orfive fullday meet-ings duringthe first sixmonths ofthe project

Quarterly prog-ress reviews byproject man-

agement

Discussion of successesand problems

Senior project man-agement staff andpartner organisa-

tions

One daymeetingevery three

months

8/6/2019 M&E Handbook 2006

23/59


Field visit First hand observation of what is happening in thefield, informal discussionsabout how activities arebeing implemented.

Field staff, supervi-sors of field staff,and project man-ager

Weekly visitfor field staff,monthly forproject direc-tor

Annual project

review

Summary of key successes

and challenges. Ideas forchanging project outputsand assumptions andreview of implications ofchanges

Representatives of

intended primarystakeholders, staffof implementingpartners, all projectstaff

Once a year

Periodic reviewworkshops ofkey projectcomponents

Focussed discussion aboutthe strategy and key activi-ties and revision of priori-ties and identification oflessons learned

Key stakehold-ers of the project,intended primarystakeholders, fieldand project andmanagement staff.

Once a yearin the firsttwo years,after thatonce everytwo years

Preparation forsupervision ses-sion

Explain the purpose of thesession, agree on what theproject and stakeholders

would like to get out of thesession and what prepara-tion is required by whom

Key stakeholdersof the project com-ponent, intendedprimary stakehold-ers, implementingpartners field andsenior project staff

One monthprior tosupervisionmission.

What participants said:

1. Participants acknowledged that critical events seemed to be a very impor-tant of way of bringing depth to monitoring and evaluation efforts. Itwas felt that sometimes organisations embarked on strategies withoutfully understanding the implications and this resulted in difficulties andmiscommunication.

2. There is a need to develop a plan for critical reflection processes or eventsas an integral part of M&E activities within the CINDI Network.

3. Potential funders, managers from the CINDI Network Office and CEOs ofmember organisations needed to visit programmes on the ground so

that they could have first hand experience of the realities and constraintsthat organisations and field workers are faced with.

4. Participants also felt that when issues are not being resolved and createtension within the work setting it was felt that well planned critical reflec-tion processes could help to assess progress and identify challenges. Itwas felt that some of the tensions between CBOs and NGOs and some ofthe issues around scaling up would benefit from a series of critical reflec-tion processes.

8/6/2019 M&E Handbook 2006

24/59


8. Disseminating information

8.1. Importance of communicating M&E findingsGenerally speaking two sets of information will need to be communicated:

A wide range of stakeholders including participants should have the opportunityto give feedback on M&E findings. This will help to check accuracy and increaseownership of the M&E exercise.

Once there is consensus on the findings they can be communicated to funders,project beneficiaries, government departments and other projects. In this way notonly funders needs for accountability will be met but also the findings will fulfil anadvocacy role.

8.2. Developing a strategy to communicate M&Efindings

Know your audience and ensure your presentation is appropriate to the level of theaudience

Build a communication plan into your M&E system

Ensure the resources are in your budget for a communication plan.

Things to think about when disseminating information

Ensure your message is clear. Different stakeholders will inevitably focus on differenttypes of information.

Agree on frequency for communicating information. Try to organise feedback ses-

sions for key meetings at the different levels.

Ensure Timeliness. Ensure feedback sessions are timely so that issues can be dealtwith when there is still a level of momentum in project activities.

Consider location. Think about the various opportunities for communication at thedifferent levels. Make it a point to ensure information reaches people so that theycan give feedback.

Make effective use of graphic information to facilitate analysis

Visually presented information is easier to understand.

When addressing boards and funders learn to use Power Point.

At local level, maps and other visual aids may help with analysis and communi-cate findings

Keep focussed on your task. Feedback sessions can easily become a talking eventwith no real outcomes. Plan the event carefully to include clarifications, additionalinsights, conclusions and action steps. Make it sincere and thought provoking notjust a glib exercise.

8/6/2019 M&E Handbook 2006

25/59


8.3. Using different media to communicate findings

Written Reporting

Remember to put a human face on impact reports by including stories from peoplethemselves. Work out a reporting strategy that fits the organisation and the level of thefield workers. Templates are helpful but they should be drawn up with the local realitiesin mind. Think through the information that you need to know and that which would

be nice to know and keep it simple. Dont let the reporting become the main thrust ofthe project and an end in itself.

Oral Presentations

Important information is obtained through personal contacts and oral presentations.Do not underestimate their value and ensure time is given for discussion and feedback.

Visual presentations

Visual displays such as graphs, charts and maps help illustrate data presented in oralpresentations. Photographs, video clips etc also help to convey a picture of what has

been done. At local level dramatic presentations can also be a powerful means of com-municating issues around a project.

The CINDI Network and communication

The issue of communication

1. Remember the CINDI Web Site is there and has approximately 5000page requests a month. It thus provides a very effective means of com-municating your work and sharing experiences.

2. The CINDI Network relies heavily on electronic communication which is

making it very difficult for the CBOs who have no access to the internetto keep up with developments. This remains a critical communicationchallenge for the CINDI Network.

3. To translate materials into isiZulu or not is a big CINDI issue! There is aview that people at local community level are excluded from broaden-ing their knowledge and expressing their own views because of theirdifficulties in expressing themselves in English. For this reason materialsand information must be available in isiZulu. There is another view thatpeople must get to grips with communicating in English which appearsto be the language of globalisation. Costs of translation are prohibitively

expensive and with increasing access to education within a few yearslack of fluency in English will be a thing of the past.

The debate continues ...

8/6/2019 M&E Handbook 2006

26/59


9. Measuring, demonstrating, or

assessing impact?4

9.1. Introduction and definitions

As has been pointed out in the learning sessions the focus of M&E in recent years hasshifted from inputs and outputs to impact giving rise to the results based approach toM&E. The shift to assessing impacts has been a feature of both the public and privatesector. In terms of the various frameworks for strategic planning and M&E most peopleare generally comfortable with some of the terminology. The terms inputs, activitiesand outputs for example, are all fairly well understood. Monitoring these dimensionsof a project or activity is fairly easy as it is basically a counting exercise and results arecommonly expressed in numbers.

Outcomes refer to the benefits or changes for project participants. They are harder tomeasure as they are less tangible and not readily countable. Some of the outcomes of

projects within the CINDI Network might include, for example, improved care of childrenorphaned or made vulnerable by HIV and AIDS, or increased understanding and capac-ity amongst youth to prevent the spread of HIV and AIDS etc. Outcomes are commonlyset out in an organisations objectives.

Impact is an even more difficult to define and measure. Commonly the term is usedas a verb meaning to have an effect on something e.g. How many children does theCINDI Network impact on?

However, as an M&E term and noun meaning effect it is used to describe the allimportant longer term (hopefully developmental) effect of an intervention. One defini-tion that has been compiled describes impact as any change resulting from an activity,

project or organisation. It includes intended as well as unintended effects, negative aswell as positive and long term as well as short term.

This very broad definition will probably need narrowing down to make it operational ina given context.

9.2. Measuring or demonstrating impact?As has been mentioned earlier the shift to funders wanting evidence of impact is stillfairly new and the topic of much debate. The aim is to ensure efficient resource allocationand utilisation. The emphasis on measurement usually comes as part of a top down

approach with funders, governments and the media wanting measurable results as ameans of assessing performance.

Demonstratingimpact is more likely to reflect a more community driven bottom upapproach whereby stakeholders including project participants assess what has beenachieved towards a set of goals with a certain level of resources. If efficient resource allo-cation is the most critical issue to the assessment of impact then demonstrating impactmay not succeed in proving this point.

4 Much of thissection hasbeen adaptedfrom MeasuringImpact: CaseStudies of impactassessmentin small andmedium sizedvoluntaryorganisations.The UK VoluntarySector researchGroup. May 2003

8/6/2019 M&E Handbook 2006

27/59


9.3. Some of the difficulties with current approachesThere are a number of difficulties with current approaches to measuring impact includingthe following:

There are no tools that can assess all aspects of impact as usually a framework isset at the beginning of the process resulting in unanticipated impacts not being cap-

tured. The short time scales and the expectation that organisations can report on impact as

soon as a project is completed makes it difficult to assess long term impact.

The difficulty in proving a causal relationship between a particular activity and aparticular outcome especially when measuring long term impact.

Organisations working with prevention programmes will have a lot of difficulty prov-ing positive impacts when success for them is something not happening e.g. preven-tion of drug addiction, teenage pregnancy etc.

It is very difficult to measure/demonstrate or assess impact if there has not beena base line survey and generally this is not easy for smaller resource constrained

organisations.

There is still a negative attitude towards M&E. Funders expectations are often unre-alistic which leads to reports being drawn up to suit the funder rather than to assessprogress for the sake of the project.

Generally there are difficulties in reflecting unanticipated impact with the result thatthese tend to be ignored or denied, e.g. the widespread fraud around the child caregrant.

Impact assessment is generally a costly and time consuming exercise which createsproblems for small organisations.

Current approaches seem geared primarily to quantitative data. There is oftendifficulty with qualitative approaches and much of the work undertaken by CINDIorganisations does not readily translate into quantitative data.

9.4. Dimensions of impactImpact has a number of dimensions as indicated below:

Impact can have economic, social or political dimensions

It can be individual or collective

It can have a geographic and specific locality focus

It can be time specific

It can have intended and unintended dimensions

It can be short term or long term.

8/6/2019 M&E Handbook 2006

28/59


9.5. Examples of basic models of impact evaluation5

5 Adapted fromMonitoring &Evaluation: Sometools, Methodsand Approaches.World Bank.2004

Model Design Comment

1. Randomized pre-test post-test

evaluation.

Subjects (families, schoolscommunities etc) are

randomly assigned toproject and control groups.Questionnaires or otherdata collection instrumentsare applied to both groupsbefore and after the projectintervention. Additionalobservations may bemade during projectimplementation.

1. Some of our communitieshave had enough of

being part of studiesand research. It may bedifficult to sell this model

2. May be costly and timeconsuming

3. The debate will behow much time needsto elapse betweenprogramme interventionand impact?

4. Nevertheless if resourceswere available it could

be used as a model forimpact evaluation

2. Quasi-experimentaldesign withbefore and aftercomparisons ofproject and controlpopulations.

If randomisation is notpossible then a controlgroup can be set up whichmatches the project group

Similar concerns asabove. Can you just set upcontrol and experimentalgroups as required in ourcommunities?

3. Rapid assessmentex-post impact

evaluations

Some evaluations onlystudy groups affected by the

project while others includematched control groups.Participatory methods canbe used to allow groups toidentify changes resultingfrom the project, who hasbenefited and who hasnot, and what were theprojects strengths and

weaknesses. Triangulationis used to compare thegroup information with the

opinions of key informantsand information availablefrom secondary sources.Case studies on individualsor groups may be producedto provide more in-depthunderstanding of theprocesses of change.

Could be used to assesswide range of programmes

relevant to the CINDINetwork

8/6/2019 M&E Handbook 2006

29/59


9.6. Some of the tools used in assessing impact

Questionnaires and Surveys

These can designed to collect standardised information. They can serve as an importantmeans of obtaining baseline information as well as changes over a period of time.

Impact Indicators

Indicators to demonstrate impact can be designed to assess psycho-social impact and theimpact of training e.g. I feel less isolated in the community or I am better able to copewith my problems.

Participatory approaches

Participatory approaches are those that engage participants and other stakeholders inthe design and implementation of M&E exercises. They often generate a real sense ofownership of the M&E processes. They are an important means of building capacity andempowering poor people. The disadvantage is that they are sometimes not consideredto be objective and can be manipulated by powerful stakeholders.

Rapid appraisal methods

Rapid appraisals are quick and usually low cost methods to gather the views and feed-back on the impact of a project. In this way information can be obtained to guide furtherpolicy decisions. They methods allow flexibility to explore new ideas. They tend torepresent very localised views and are viewed as less credible and reliable than formalsurveys.

9.7. Basic learnings from various case studies

Set realistic goals!

One can be tempted to try to measure/demonstrate/assess every aspect of an organi-sations work. However, this is an almost impossible exercise as there is no single toolthat can be used and assessing impact is a costly business. Thus it is important to beselective about which impacts you want to measure and it is helpful to ask yourself afew questions:

What are the reasons for assessing impact?

If it is for a funder what is it they want to know?

Is it for internal learning so as to effect improvements?

Is it necessary to measure it or will it be more meaningful to reflect it?

Such guiding questions can help with the design.

What has worked well?6

Following the documenting of best practice in a number of case studies the followingobservations were made about what works well.

One thing that worked well was that impact assessment was a process that contrib-uted to capacity building. By using a participative approach we were able to align

our aims with those of the major stakeholders, which meant they were willing to take

part in the process.

Bringing people together in focus groups promoted the development of inter-organi-sational working, reflective learning and strategic planning. Therefore, undertaking

6 Summarydrawing from fourcase historiesin MeasuringImpact: casestudies of impactassessmentin small andmedium sizedvoluntaryorganisations.The UK Voluntary

Sector ResearchGroup. May2003.

8/6/2019 M&E Handbook 2006

30/59


an impact assessment was potentially a very useful process.

Think of an organisation as part of a system or network, taking into account all

stakeholders and activities carried out by the organisation. This will help to highlight

impactees, and prevent you from thinking about the intended beneficiary group as

the only people who are impacted on;

If interviews are being used, keep the structure flexible, allowing unanticipatedimpacts to be uncovered;

Use visual aids such as a network map which represents where the organisation sits

in relation to its stakeholders. This is a useful prompt during interviews;

Be realistic about what is to be achieved with the time and resources available.

9.8. Learning for the CINDI Network Dont fall into traps trying to measure unmeasurable or very complex impact!

Be sure when planning a project what impact you want to achieve and how you willassess/demonstrate/measure that impact!

Be aware of and deal honestly with unintended outcomes.

Try to work hard at getting baseline data so that you can make comparisons.

Always keep things realistic.

The more participatory M&E actions are the greater ownership and empowermentthere will be as long as organisations do not become a target of funding relatedpower games and manipulation.

8/6/2019 M&E Handbook 2006

31/59


10. The basics of evaluation7

10.1. What is evaluation?Earlier evaluation was described as:

... a systematic assessment of a planned, ongoing or completed project which aims

to answer specific management questions. Information gathered should help organ-isations assess what has been learnt which can inform future planning and actions.

Issues of efficiency, effectiveness, sustainability and relevance of an organisations

programme objectives are commonly central to evaluation exercises.

Evaluation is most helpful when an organisation wants to assess results on a deeper leveland try to understand the underlying reasons why change is occurring or not occurringin the field. This information is then used to further develop both the activities and theconceptual (results) framework.

Evaluation focuses on why results are or are not being achieved, on unintended con-

sequences, or on issues of interpretation, relevance, effectiveness, efficiency, impact, orsustainability.

10.2. Types and purposes of evaluation

1. Formative Evaluation:

Formative evaluation normally takes place at the concept and design phase of aprogramme. Formative evaluations help organisations understand the following:

The operational setting of a potential programme

What the focus of an intervention or programme should be

What exactly is required in terms of inputs and activities

Where the geographical focus of the programme should be

Who should be involved in the intervention

How the intervention will be carried out.

How the intervention should be managed

Formative evaluation provides the information needed to define realistic goals, objec-tives, and strategies for a programme. Methods to undertake formative evaluation mayinclude:

Reviews of existing information and knowledge (literature reviews and discussionwith potential beneficiaries and partners)

Focus group or small group discussions with relevant stakeholders

Individual in-depth interviews or short surveys

2. Process Evaluation

Process evaluations take place once activities are underway and focus on tracking theefficiency of a given programme or organisation. Process evaluations focus on:

Providing information relating to what extent planned activities are being imple-mented,

How well services are being provided, in what timeframe, at what cost, and withwhat result.

7 Much of thissection derivesfrom the followingpublication: PACT.2005. BuildingMonitoring,Evaluation, and

reporting Systemsfor HIV/AIDSProgrammes.

8/6/2019 M&E Handbook 2006

32/59


Analyse how efficiently inputs (money, time, equipment, personnel, etc.) are beingused in creation of outputs (products, results, etc.).

Process evaluations help organisations analyse what they planned to do versus what theyactually are achieving and are used to adjust implementation strategies. Process evalua-tions are often conducted informally (staff meetings, etc.) at regular intervals during theprogramme year to assess progress toward achieving the results. They need to be based

on performance data (results from indicator data collection) as well as staff observationof projects and programmes.

Methods for process evaluation include:

Reviews of service records and regular reporting systems

Key stakeholders interviews including project participants

Direct observations

Population-based surveys, etc.

3. Mid-Term Evaluations

Most organisations are also familiar with mid-term evaluations, which are processevaluations conducted halfway through a strategic cycle. Mid-term evaluations may berequired or planned for some projects but should also be considered as a way to reviewprogramme or strategic plans. Mid-term evaluations usually:

Assess the organisations progress in implementing activities

Assess progress toward achievement of objectives or benchmarks

Assess whether interventions and budgets are sufficient to reach desired results

Identify barriers to achievement of results, objectives, and activities

Identify opportunities, unanticipated accomplishments, or innovation

Specify course correction or changes required

Mid-term evaluations usually suggest actions to bring mid-course adjustments in the lasthalf of the programme. Mid-term evaluations should endeavour to be participatory innature and include stakeholders in the analysis of programmes.

4. Effectiveness Evaluation

Effectiveness or impact evaluations normally take place toward the end of a pro-gramme intervention and focus on assessing the overall outcomes and impacts attained.Effectiveness evaluations focus on questions pertaining to what results have beenachieved, what short-term and intermediate effects were observed as a result of pro-

gramme effort, and what the outcomes mean. Does the programme make a differencetowards the larger development impact sought?

5. Final Evaluations

Most organisations are familiar with final evaluations, which are normally effectivenessevaluations. Final evaluations may be required or planned for some projects but shouldalso be considered as a way to review and update your strategic plan. Final evaluationsnormally:

Assess why progress toward planned results has been positive or negative

Test the validity of assumptions underlying a results framework

8/6/2019 M&E Handbook 2006

33/59


Assess how well needs of different project participants were met (e.g., by gender,age, ethnic group)

Identify and analyse unintended consequences and effects of activities

Assess sustainability of activities and their results

Identify lessons learned that may be useful elsewhere and/or by others

Where possible final evaluations should be participatory in nature and include stake-holder (target population) analysis of programmes.

Methods for effectiveness evaluation include:

Reviews of service records and regular reporting systems

Key stakeholder interviews

Direct observations

Population-based surveys, longitudinal studies, review of secondary data, etc.

10.3 Developing a learning agenda and evaluationschedule

Irrespective of the type of evaluation format it has planned to use, an organisation ini-tially needs to identify the questions it wants to ask and when it wants to collect the data in other words, develop a learning agenda.

Developing a learning agenda includes:

1. Determining what components of your programme you want to learn about (iden-tifying what needs to be evaluated). In deciding what needs to be reviewed checkout the implementation plan, deliverables and results framework and identify keycomponents, sub-components, or aspects of the programme which will be analysed

to assess the organisations effectiveness in achieving their goals.

2. Clarifying what exactly you want to learn about each component identified. To helpdetermine what question to ask review the planned deliverables, results frameworksand indicators. Some questions that might be asked include:

M&E Handbook 2006

Documents

Transcript of M&E Handbook 2006