Download - Monitoring and Evaluation Overview. Monitoring & Evaluation Workshop MONITORING AND EVALUATION / General Objectives FOR MONITORING: Ensure adequate programme.

Monitoring and Evaluation Overview

Monitoring & Evaluation Workshop

MONITORING AND EVALUATION / General Objectives

FOR MONITORING:

• Ensure adequate programme management. Regular monitoring of inputs and outputs; Information-based management

decisions.

• Ensure adequate quality of service. Definition of minimum service standards

and quality of service indicators/goals; Process monitoring; Real-time, dynamic correction

mechanisms.


WHY MONITORING?

To improve the delivery process of services to beneficiaries.

To inform policy makers, wider public, stakeholders on the progress and achievement.


MONITORING AND EVALUATION / General Objectives

FOR EVALUATION:

• Measure programme impact on selected outcomes. Results focus vs process focus.

• Uphold accountability and compliance. Whether or not the work has been carried

out as agreed; Is in compliance with established standards.

• Provide opportunities for stakeholder feedback. To provide input, and as result, improve the

overall programme.


MONITORING VS. EVALUATION

EVALUATION MONITORINGFREQUENCY Periodic Regular, Continuous

COVERAGE Selected aspects of the programme

All project cycle phases

DATA Sample-based Sample-based and Universal

DEPTH OF INFORMATION

Tailored, often to performance and impact (WHY)

Tracks implementation (WHAT)Identifies operational challenges (HOW)

COST Can be high Spread Out

UTILITY Major programme decisions (POLICY, OBJECTIVES)

Continuous improvement, management

WHO Independent (FIRMS, CONSULTANTS)

Internal (Ministry and programme M&E unit)


INTERNAL MONITORING IN SCTP /Project Cycle Process

Targeting

Eligible/ ineligible householdsAppellant householdsRe-ranked eligible householdsComplete/ incomplete forms

Transfers

Collected/ uncollected transfers per termHouseholds with arrears per term

Case Management

UpdatesModified transfer amountsPending/ resolved claimsClaims regarding non-transfers/ wrong transfer amount/ transaction errors

Enrolment

Attendance of selected households at the third community meetingNew enrolments (replacements)Households with main and/ or alternative receivers


ANALYSIS OF MIS-BASED INDICATORS / Example

PROCESS NAME OF THE INDICATOR MEASURE FILTER

GeneralMain Receivers who are

household headsTotal Number

By Districts Gender: female

GeneralSecondary school-going

childrenTotal Number

By Districts: Mchinji, Machinga, Salima and Neno

General Single and double orphans Total Number and % By Gender By Districts

TargetingBeneficiaries household

membersTotal Number

By DistrictsGender: male

EnrolmentBeneficiary children

information enrolled in secondary level

Total NumberBy Age: 15-17 years oldDistrict: Chitipa By TAs

Transfers Amount paid Amount By Transfer Agency

PROCESS MONITORING Overview


PROCESS MONITORING OVERVIEW / Definition

An independent verification at various stages of the SCTP Project Cycle, to identify whether the process in the field is being carried out according to the Operational Manual (OM)/respective technical annexes.

TRANSFERS COMPLIANCEVERIFICATION

CASEMANAGEMENT

ENROLMENTTARGETING

MONITORING AND EVALUA

Differences could lead to identify potential problems either in the operational design or in the way the operation is carried out in the field.

Solutions should be suggested accordingly.


PROCESS MONITORING OVERVIEW / Process Monitoring and Process Evaluation Differences

ITEM PROCESS MONITORING PROCESS EVALUATION

WHO M&E Unit of the Ministry with District Officials

Independent consultant/firm

SCOPE Sub-processes within a project cycle phase Processes and sub-processes of a project cycle

COVERAGE Clusters and TAs Districts, entire programme areas

METHODS/TOOLS Mainly qualitative Qualitative and quantitative

WHEN Specific issue/problem/challenge is identified.

Upon contract, regular reviews, macro issues/challenges identified

EXAMPLE Somebody denounced that all the steps for the targeting process were not followed in a given cluster.

E-payment process is not appropriate for SCTP transfer receivers.


PROCESS MONITORING / How to Execute Process Monitoring

Identification Stage

STEP 1Identification of the need

STEP 2 Establish Questions

Planning Stage

STEP 3 Specify Design/

Methodology

STEP 4 Create Work

plan

Implementation Stage

STEP 5 Field Research

STEP 6Analyze Data

STEP 7 Document

Findings

Reporting/Feedback Stage

STEP 8 Disseminate Information

STEP 9Feedback for programme Operational Improvement


CONCLUSIONS

Process Monitoring can only uncover possible reasons why a programme activity is working or

not

The findings from a Process Monitoring are the basis behind improvements in effectiveness and

efficiency of safety net programmes

A sound evaluation can be done with careful planning and some basic math skills


PROCESS MONITORING OVERVIEW / Methodology

Ministry and District Officials identify issues/challenges/problems relating the designed project cycle in a given area.

Both agree and specify the process to be done:Issue identified, justification and context;Review of the process, using the OM and corresponding technical annex to understand the possible differences that may be occurring with respect to the issue that was identified;Question to be answered is defined;Source of the information to sustain the investigation;Method to be used: interviews, observations, focus groups, and the examination of the records of the programme;Personnel involved; and Possible coverage.

Work plan and budget


PROCESS MONITORING OVERVIEW / Methodology

Definition of Qualitative analysis

METHODS

• Individual interviews• Group interviews• Direct observation• Focus group• Beneficiary Score cards

SPOT CHECKSOverview


SPOT CHECKS OVERVIEW / Definition

Spot Checks allow verifying data quality received by the programme through the

SCTP-MIS.

To do this, a sample of the total population is taken and compared with information available at the source to verify if both coincide

with each other.

To test the truthfulness of information recorded in the MIS, qualitative an quantitative verification methods are applied:

Statistical formulas applied to see if there are any significant differences

Quantitative methodsFocus groups, interviews, and

other qualitative methods applied in order to comprehend

and/or confirm significant statistical differences.

Qualitative methods


SPOT CHECKS OVERVIEW /Advantages and Disadvantages

ADVANTAGES DISADVANTAGES

• Helps verifying that information entered into the MIS is accurate

• Promotes accountability

• Reasons for differences are analysed, and measures are identified to correct it

• Verification is located and limited

• Relative expensive and time consuming


SPOT CHECKS OVERVIEW / Objectives

1. Evaluate, through sampling techniques of probabilities, accuracy, and validity, data registered in the MIS, which was reported by Programme actors.

Accuracy Validity

Data in the MIS matches data

gathered during

Spot Checks implementatio

n

Verified data must be

right–processes applied

efficiently


SPOT CHECKS OVERVIEW / Objectives continued2. If significant differences are found, reasons must be analysed using process

monitoring techniques.

Example:

Are beneficiary households actually receiving the respective benefit amount?

Detect possible errors that may have occurred during data collection and entry.

Possible fraud

Others?


SPOT CHECKS OVERVIEW / Methodology

Ministry and District Officials identify possible problems in the internal monitoring SCTP-MIS module

Question (hypothesis) to be analysed is definedType of analisis method selected to verify statistical differences

Work plan and budget

Field workCalculation of statistical differencesQualitative work to understand/confirm statistical differences


SPOT CHECKS / How to Execute Spot Check Process

Identification Stage

STEP 1 Identifying the problem and

consequences/risks

STEP 2 Defining possible reasons for the

problem

Planning Stage

STEP 3 Formulating questions and

statistical hypothesis

STEP 4 Designing an instrument

(form) to collect information during field work

STEP 5Defining possible sources of

actual valid information

STEP 6Determining geogrraphic area

for intervention and investigation’s scope

STEP 7Defining the sample size

STEP 8Creating a work plan

Implementation Stage

STEP 9Field Work

STEP 10Analysing Results

STEP 11Accepting or rejecting possible reasons for

the problem

STEP 12Qualitative

investigation to confirm reasons for the

problem

Reporting/Feedback Stage

STEP 13Feedback for Operational Programme Improvement

IMPACT EVALUATIONOverview


IMPACT EVALUATION / Definition

- Assesses the changes in participants’ well-being that can be attributed to a specific programme/intervention;

- Determined by comparing the outcomes of programme participants with the outcomes other individuals experience in the absence of the programme (non-participants).

Participants = Treatment Group

Non-participants = Control /Comparison

Group


IMPACT EVALUATION / Objectives- To provide feedback to help improve the design of

programmes and policies;

- Estimate the magnitude of effects with clear causation:• Did impacts vary across different groups,

regions or over time?• How could program design be modified to

increase impact?

- Learn how effective is the program compared to alternative interventions.

- Generate lessons learned to inform the decision making process.


IMPACT EVALUATION / Why Conduct an Impact Evaluation?- The information generated as a result of the execution of an impact

evaluation enables policy-makers and programme managers to answer the following questions:

1. Does the programme achieve its intended goal (s)?

3. Can the changes in outcomes as a result of the programme or are they a result of some other factors

occurring simultaneously?

4. Do programme impacts vary across groups of intended beneficiaries, across regions, and over

time?

5. Does the programme have any unintended effects, either positive or negative?

6. How effective is the programme in comparison with alternative interventions?

7. Is the programme worth the resources it costs?

2. Should this pilot programme be scaled up? Should this large scale programme be continued?


IMPACT EVALUATION / Methodology

Inputs Activities Outputs Outcomes

ASSUMPTIONS

Factual Analysis Counterfactual Analysis

- A good impact evaluation examines all elements in the causal chain. This involves using mixed methods.

White, H (2012) “The use of Mixed Methods in randomized control trials” , 3ie


IMPACT EVALUATION PROCESS

STEP 1 : Initial Considerations

STEP 2 : Preparation of Evaluation Tasks

STEP 3 : Evaluation Research

STEP 4 : Reporting and Dissemination

STEP 5 : Management Response