May 3, 2023

MONITORING & EVALUATION OF HIV & AIDS PROGRAMS - FEBRUARY 2011

DEVELOPING M & E PLANS by DR DC TSHIBANGU

Objectives of the Session

By the end of the session, participants should be able to:

Describe the functions of an M&E plan

Identify the main elements of an M&E plan

Describe the process of developing an M&E plan

Understand how to go about implementing an M&E plan

Overview

Definition: What is an M&E Plan? Function: What is the Plan supposed to

do? Elements: What comprises an M&E

Plan? Standards: What constitutes a good M&E

Plan? Developing and implementing an M&E

plan Introduction to workshop group work:

Developing an M&E plan

M&E Plans – DefinitionComprehensive document that describes all

M&E activities in an M&E system, including: Program objectives, interventions developed

to achieve these objectives, & procedures to be implemented to determine whether or not the objectives are met

Expected results of the program and how they relate to goals and objectives

Data needs, how they will be collected & analyzed

Information use, including resources needed to do so

How the program will be accountable to stakeholders

M&E Plan: Functions States how program will measure

achievements Accountability

Documents consensus Transparency Responsibility

Guides M&E implementation Standardization Coordination

Preserves institutional memoryAn M&E plan is a living document that needsto be adjusted when a program is modified

M&E Plan: ElementsI. IntroductionII. Description of the overall program- including

problem statement and framework(s)III. Indicators- including definitions (presented in

indicator matrix and/or indicator reference sheets: very detailed!)

IV. Data sources and reporting systems (including management/roles and responsibilities)

V. Plans for demonstrating program outcome/impact

VI. Plans for dissemination and use of informationVII. Analysis of data quality constraints & potential

solutionsVIII. Implementation plan (aka- M&E action plan or

road map-should include budget and timeline)

Elements of an M&E Plan: Introduction

A. Purpose of the planB. Description of how it was developed

Stakeholders involved Consensus process

Elements of an M&E Plan: Program Description

A. Problem Statement What is the nature of the HIV-related issue

being addressed?B. Conceptual FrameworkC. Goal and Objectives

What is the ultimate outcome of the program (goal)

What are the shorter-term aims (objectives)D. Program Description:

Interventions Geographic scope Target population Duration

E. Logical Framework/Results Framework

Elements of an M&E Plan: Indicators

A. Selection of indicators based on: Conceptual and logic frameworks Strategic information needed for decision

making at appropriate level (country/state/local) Donor requirements Existing data (really needs to be based in reality) Funding (available and dedicate to M&E)

B. Presented in 2 ways: Indicator Matrix- a table presenting indicators

including information on data source, frequency and who is responsible

Indicator Reference Sheets- detailed sheets describing each indicator, how to measure it, underlying assumptions & interpretation considerations (may be included as appendices)

Elements of an M&E Plan:Data Sources and Reporting Systems

A. Sources of data for indicatorsB. Diagram of data collection, processing,

analysis, and reporting systemC. Data collection tools

Patient records or registers Survey instruments Commodity management forms (e.g.,

condoms) Others?

D. Management Roles and responsibilities of each

group/member of the M&E system1. Data Flow

Elements of an M&E Plan: Plan for demonstrating program outcome/impact

A methodology for measuring program impact (the evaluation)

Protocols for special studies

Elements of an M&E Plan: Dissemination and Use Plan

A. Clearly defined usersB. Databases for information storageC. Dissemination methods

Reports (schedule and audience) Media Speaking events Others?

Elements of an M&E Plan: Data Quality Constraints

Describe known constraints to data quality and/or system performance and what will be done to address these

Elements of an M&E Plan: Implementation Plan

A. Assessment of feasibility to implement plan

B. A detailed work plan for the M&E Plan to include: Each M&E activity (including

update of M&E Plan) Timing of each activity Party responsible for each activity Budget necessary for each activity

M&E Plan: Standards

Utility - serve practical information needs of intended users

Feasibility - be realistic, prudent, diplomatic and frugal

Propriety - conducted legally, ethically, and with regard to those involved in and affected by the evaluation(s)

Accuracy - reveal and convey technically accurate information

Developing an M&E Plan: Inputs

Authority and mandate to implement the M&E plan

Human resources with M&E technical capacity

Understanding of the program Commitment to accountability (not only to

donors, but to the people whom the interventions are to serve) and program improvement

Infrastructure (Technology, space, etc..)

Developing an M&E Plan: Process

Advocate for the need for M&E Assess strategic information needs Assess existing information systems’ capabilities to

address strategic information needs Achieve consensus and commitment among

stakeholders on: Indicators and their definitions Reporting structure, roles and responsibilities Establish a common “language”

Develop mechanism for M&E plan review Prepare document for final approval for all

stakeholders

M&E plan should be developed during theinitial stages of program development

Developing an M&E Plan: Output

A comprehensive document that describes the M&E system and that: Includes all the necessary elements; Has the approval of the governing

authority; Has the consensus of primary

stakeholders.

Implementing the M&E System: Costs

How much will it cost? Need to budget: Costs of information systems (costs of

data collection, processing, and analyzing)

Costs of information dissemination and use

Costs of the data quality control system Costs of coordination and capacity

building

Sample Costs for Selected Data Sources (example from a national system)

Information System Total Annual Costs (2001 US$)

Frequency of Data Collection

National Sentinel System $513,682 Continuous

HMIS $2,119,941 Continuous Integrated Disease Sample Costs for Selected Data Sources (example from a national system) Surveillance

$4,270,943 Continuous

Vital Registration $719,427 Continuous

DHS $854,164 Periodic (every 4 years)

National Census $8,244,114 Periodic (every 10 years)

Source: Rommelmann, et. al. Costs and Results of Information Systems for Poverty Monitoring, Health Sector Reform, and Local Government Reform In Tanzania.

2003

Developing & Implementing an M&E Plan: Role of the M&E Unit

Consensus building – among all stakeholders (donors to program managers to clients)

Coordination Data management and manipulation

(might include data entry, data formatting, data analysis and interpretation…)

Reporting Data auditing Information dissemination Training and capacity building

Consensus Building Tips

Make sure everyone understands what you are trying to achieve consensus on

Ensure that all stakeholders are involved early in the process

Provide stakeholders with the opportunity to provide input and receive feedback

Consistently promote the message that M&E provides the means to demonstrate the extent to which a program is achieving its objectives (“It is everyone’s job and to everyone’s benefit’)

Developing & Implementing an M&E Plan: Dos and Don’ts

Do: Start early Involve stakeholders at all stages in the

process Assess current capacity and use what is

already available Avoid duplication of data collection and

reporting

Do not: Collect information that will not be used Underestimate the importance of stakeholder

buy-in and ownership at every juncture

M&E for Program Improvement- An Iterative Process

Data collection methods and toolsHuman resources

and training

Indicators

Source of data

Implementation

Data use

May 3, 2023

WHAT IS PLANNING?

Planning is a process concerned with: Developing and achieving a vision Understanding some of the wider drivers of change A process that is flexible in light of these changes A process that is achievable in light of available

resources A process that is continuously reviewed as part of a

learning process A vision that is, importantly, implementable

May 3, 2023

PRINCIPLES FOR A SUCCESSFUL PLANNING PROCESS

Include at least the following four principles:

The problem identified (ensure that the right question is the focus of the planning)

The people involved (ensure that the right people are in the team)

The process used (ensure that this is a learning process that helps key players debate things that matter)

The resources available (it’s essential to match the size of the problem with the resources needed to address it)

May 3, 2023

WHAT IS A PLAN? It is a course of action one intends to follow in order to solve a

problem It is a purposeful, detailed outline of actions to allocate

resources and coordinate efforts, to achieve desired change (e.g. solve a problem, achieve an objective).

It is also regarded as a document that describes a system that links strategic information obtained from various data collection systems to decisions that will improve programs.

It ensures that objectives are set to deal with problems and to make best use of available resources

PRACTICALLY WHAT DOES THIS MEAN?

This implies the use of conceptual framework to develop a clear understanding of all the factors likely to positively or negatively influence the problem under investigation

Able to derive few models/scenarios from the conceptual framework at hand

CONCEPTUAL FRAMEWORK FOR HIV/AIDS CHALLENGES

Individual

Characteristics

Community

Characteristics

Programmatic

Characteristics

Not Infected

Infected

Not Infected

Sick

Infected

Death

THIS CONCEPTUAL FRAMEWORK SHOWS

SOME FACTORS THAT ARE LIKELY TO INFLUENCE THE TRANSMISSION OF HIV

THE OPTIMUM LEVEL OF APPLICATION OF SOME PREVENTION STRATEGIES:1. Primary Prevention: ABC; VCT; School-based life skill

education; PMTCT (infant); BCC.

2. Secondary Prevention: PMTCT (mother); ART

3. Tertiary Prevention: Palliative Treatment/Community- based support and care

PURPOSES OF STRATEGIC FRAMEWORK

To provide a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact

To clarify project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends

Strategic/Results FrameworkYouth HIV Prevention Program

SO1 : Increased use of VCT services among young people

IR1 : Availability of youth friendly VCT services

IR2 : Demand for VCT services

IR1.1 : Information and services increased for youth

IR1.2 : Practioners’ skills and knowledge increased for counseling and testing youth

IR1.3 : Sustainable effective management

IR2.1 : Customer knowledge of HIV/AIDS transmission improved

WHO/WHAT DETERMINES THE AREAS OF THE PROGRAM TO BE MEASURED?

The program goals/results indicate the areas in which progress can be expected, hence, the areas in which it can be measured.

How to measure progress in these areas? By selecting appropriate and relevant indicators (country’s goals, objectives, activities, local epidemiology, nature of problem and risk behaviors)

May 3, 2023

WHAT IS THE PLAN MAIN PURPOSE?

To help monitoring and evaluating activities through:

M&E BUY IN Efficiency – saves time Effectiveness – doing what we want to do Data management Data Quality & Data Use

May 3, 2023

Some References

Adamchak S et al. (2000). A Guide to Monitoring and Evaluating Adolescent Reproductive Health Programs. Focus on Young Adults, Tool Series 5. Washington D.C.: Focus on Young Adults.

Bertrand J et al. (1996). Evaluating Family Planning Programs. The Evaluation Project.

Rommelmann V, P Setel, Y Hemed, H Mponezya, G Angeles, T Boerma (2003). Costs and Results of Information Systems for Poverty Monitoring, Health Sector Reform, and Local Government Reform In Tanzania.

Lusthaus, et. al. (1999) Enhancing Organizational Performance: A toolbox for Self-Assessment.

Loos G.P. (1996). Field guide for international health project planners & managers. Janus Publishing Company, London England.

Hadridge P (2004). Strategic approaches to planning health care. Oxford Handbook of Public Health Practice. Oxford University Press.

May 3, 2023

THANK YOU