Monitoring and Evaluation Overview. Monitoring & Evaluation Workshop MONITORING AND EVALUATION /...

download Monitoring and Evaluation Overview. Monitoring & Evaluation Workshop MONITORING AND EVALUATION / General Objectives FOR MONITORING: Ensure adequate programme.

of 27

  • date post

    21-Jan-2016
  • Category

    Documents

  • view

    227
  • download

    1

Embed Size (px)

Transcript of Monitoring and Evaluation Overview. Monitoring & Evaluation Workshop MONITORING AND EVALUATION /...

Presentacin de PowerPoint

Monitoring and Evaluation Overview

Monitoring & Evaluation WorkshopMONITORING AND EVALUATION / General ObjectivesFOR MONITORING:

Ensure adequate programme management.Regular monitoring of inputs and outputs;Information-based management decisions.

Ensure adequate quality of service.Definition of minimum service standards and quality of service indicators/goals;Process monitoring;Real-time, dynamic correction mechanisms.

Monitoring & Evaluation WorkshopWHY MONITORING?

Monitoring & Evaluation WorkshopMONITORING AND EVALUATION / General ObjectivesFOR EVALUATION:

Measure programme impact on selected outcomes.Results focus vs process focus.

Uphold accountability and compliance.Whether or not the work has been carried out as agreed;Is in compliance with established standards.

Provide opportunities for stakeholder feedback.To provide input, and as result, improve the overall programme.

Monitoring & Evaluation WorkshopMONITORING VS. EVALUATIONEVALUATIONMONITORINGFREQUENCYPeriodicRegular, ContinuousCOVERAGESelected aspects of the programmeAll project cycle phasesDATASample-basedSample-based and UniversalDEPTH OF INFORMATIONTailored, often to performance and impact (WHY)Tracks implementation (WHAT)Identifies operational challenges (HOW)COSTCan be highSpread OutUTILITYMajor programme decisions (POLICY, OBJECTIVES) Continuous improvement, managementWHO Independent (FIRMS, CONSULTANTS)Internal (Ministry and programme M&E unit)

Monitoring & Evaluation WorkshopINTERNAL MONITORING IN SCTP /Project Cycle ProcessTargetingEligible/ ineligible householdsAppellant householdsRe-ranked eligible householdsComplete/ incomplete formsTransfersCollected/ uncollected transfers per termHouseholds with arrears per termCase ManagementUpdatesModified transfer amountsPending/ resolved claimsClaims regarding non-transfers/ wrong transfer amount/ transaction errorsEnrolmentAttendance of selected households at the third community meetingNew enrolments (replacements)Households with main and/ or alternative receivers

Monitoring & Evaluation WorkshopANALYSIS OF MIS-BASED INDICATORS / Example PROCESSNAME OF THE INDICATORMEASUREFILTERGeneralMain Receivers who are household headsTotal NumberBy Districts Gender: femaleGeneralSecondary school-going childrenTotal NumberBy Districts: Mchinji, Machinga, Salima and NenoGeneralSingle and double orphans Total Number and %By Gender By DistrictsTargetingBeneficiaries household membersTotal NumberBy DistrictsGender: maleEnrolmentBeneficiary children information enrolled in secondary levelTotal NumberBy Age: 15-17 years oldDistrict: Chitipa By TAsTransfersAmount paidAmountBy Transfer AgencyPROCESS MONITORING

Overview

Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / DefinitionAn independent verification at various stages of the SCTP Project Cycle, to identify whether the process in the field is being carried out according to the Operational Manual (OM)/respective technical annexes.

Differences could lead to identify potential problems either in the operational design or in the way the operation is carried out in the field. Solutions should be suggested accordingly.

Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / Process Monitoring and Process Evaluation DifferencesITEMPROCESS MONITORINGPROCESS EVALUATIONWHOM&E Unit of the Ministry with District OfficialsIndependent consultant/firmSCOPESub-processes within a project cycle phase Processes and sub-processes of a project cycleCOVERAGEClusters and TAsDistricts, entire programme areasMETHODS/TOOLSMainly qualitativeQualitative and quantitativeWHENSpecific issue/problem/challenge is identified.Upon contract, regular reviews, macro issues/challenges identifiedEXAMPLESomebody denounced that all the steps for the targeting process were not followed in a given cluster.E-payment process is not appropriate for SCTP transfer receivers.

Monitoring & Evaluation WorkshopPROCESS MONITORING / How to Execute Process MonitoringIdentification StageSTEP 1Identification of the need

STEP 2 Establish QuestionsPlanning StageSTEP 3 Specify Design/ Methodology

STEP 4 Create Work planImplementation StageSTEP 5 Field Research

STEP 6Analyze Data

STEP 7 Document FindingsReporting/Feedback StageSTEP 8 Disseminate Information

STEP 9Feedback for programme Operational ImprovementPlanning Stage: Key to a successful Process EvaluationDevelopment Stage: Refers to the who, what and how to collect the data need to answer the evaluation questions.Implementation Stage: Field exercise and Data AnalysisReporting/Feedback Stage: Presentation of findings

11

Monitoring & Evaluation WorkshopCONCLUSIONSProcess Monitoring can only uncover possible reasons why a programme activity is working or not

The findings from a Process Monitoring are the basis behind improvements in effectiveness and efficiency of safety net programmes

A sound evaluation can be done with careful planning and some basic math skills

Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / MethodologyMinistry and District Officials identify issues/challenges/problems relating the designed project cycle in a given area.

Both agree and specify the process to be done:Issue identified, justification and context;Review of the process, using the OM and corresponding technical annex to understand the possible differences that may be occurring with respect to the issue that was identified;Question to be answered is defined;Source of the information to sustain the investigation;Method to be used: interviews, observations, focus groups, and the examination of the records of the programme;Personnel involved; and Possible coverage.

Work plan and budget .13

Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / MethodologyDefinition of Qualitative analysis

METHODS

Individual interviewsGroup interviewsDirect observationFocus groupBeneficiary Score cards

.14SPOT CHECKS

Overview

Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / Definition Spot Checks allow verifying data quality received by the programme through the SCTP-MIS.To do this, a sample of the total population is taken and compared with information available at the source to verify if both coincide with each other.To test the truthfulness of information recorded in the MIS, qualitative an quantitative verification methods are applied:Statistical formulas applied to see if there are any significant differences Quantitative methodsFocus groups, interviews, and other qualitative methods applied in order to comprehend and/or confirm significant statistical differences.Qualitative methods

Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW /Advantages and DisadvantagesADVANTAGESDISADVANTAGESHelps verifying that information entered into the MIS is accurate

Promotes accountability

Reasons for differences are analysed, and measures are identified to correct it

Verification is located and limited

Relative expensive and time consuming

Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / ObjectivesEvaluate, through sampling techniques of probabilities, accuracy, and validity, data registered in the MIS, which was reported by Programme actors.Data in the MIS matches data gathered during Spot Checks implementationVerified data must be rightprocesses applied efficiently

Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / Objectives continued2. If significant differences are found, reasons must be analysed using process monitoring techniques.Example:Are beneficiary households actually receiving the respective benefit amount?Detect possible errors that may have occurred during data collection and entry.Possible fraudOthers?

Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / MethodologyMinistry and District Officials identify possible problems in the internal monitoring SCTP-MIS module

Question (hypothesis) to be analysed is definedType of analisis method selected to verify statistical differences

Work plan and budget

Field workCalculation of statistical differencesQualitative work to understand/confirm statistical differences.20

Monitoring & Evaluation WorkshopSPOT CHECKS / How to Execute Spot Check ProcessIdentification StageSTEP 1 Identifying the problem and consequences/risks

STEP 2 Defining possible reasons for the problem

Planning StageSTEP 3 Formulating questions and statistical hypothesis

STEP 4 Designing an instrument (form) to collect information during field work

STEP 5Defining possible sources of actual valid information

STEP 6Determining geogrraphic area for intervention and investigations scope

STEP 7Defining the sample size

STEP 8Creating a work plan Implementation StageSTEP 9Field Work

STEP 10Analysing Results

STEP 11Accepting or rejecting possible reasons for the problem

STEP 12Qualitative investigation to confirm reasons for the problem

Reporting/Feedback StageSTEP 13Feedback for Operational Programme Improvement.21IMPACT EVALUATION

Overview

Monitoring & Evaluation WorkshopIMPACT EVALUATION / DefinitionAssesses the changes in participants well-being that can be attributed to a specific programme/intervention;

Determined by comparing the outcomes of programme participants with the outcomes other individuals experience in the absence of the programme (non-participants).

Participants = Treatment GroupNon-participants = Control /Comparison Group

Monitoring & Evaluation WorkshopIMPACT EV