Monitoring and Evaluation Overview. Monitoring & Evaluation Workshop MONITORING AND EVALUATION /...
Embed Size (px)
Transcript of Monitoring and Evaluation Overview. Monitoring & Evaluation Workshop MONITORING AND EVALUATION /...
Presentacin de PowerPoint
Monitoring and Evaluation Overview
Monitoring & Evaluation WorkshopMONITORING AND EVALUATION / General ObjectivesFOR MONITORING:
Ensure adequate programme management.Regular monitoring of inputs and outputs;Information-based management decisions.
Ensure adequate quality of service.Definition of minimum service standards and quality of service indicators/goals;Process monitoring;Real-time, dynamic correction mechanisms.
Monitoring & Evaluation WorkshopWHY MONITORING?
Monitoring & Evaluation WorkshopMONITORING AND EVALUATION / General ObjectivesFOR EVALUATION:
Measure programme impact on selected outcomes.Results focus vs process focus.
Uphold accountability and compliance.Whether or not the work has been carried out as agreed;Is in compliance with established standards.
Provide opportunities for stakeholder feedback.To provide input, and as result, improve the overall programme.
Monitoring & Evaluation WorkshopMONITORING VS. EVALUATIONEVALUATIONMONITORINGFREQUENCYPeriodicRegular, ContinuousCOVERAGESelected aspects of the programmeAll project cycle phasesDATASample-basedSample-based and UniversalDEPTH OF INFORMATIONTailored, often to performance and impact (WHY)Tracks implementation (WHAT)Identifies operational challenges (HOW)COSTCan be highSpread OutUTILITYMajor programme decisions (POLICY, OBJECTIVES) Continuous improvement, managementWHO Independent (FIRMS, CONSULTANTS)Internal (Ministry and programme M&E unit)
Monitoring & Evaluation WorkshopINTERNAL MONITORING IN SCTP /Project Cycle ProcessTargetingEligible/ ineligible householdsAppellant householdsRe-ranked eligible householdsComplete/ incomplete formsTransfersCollected/ uncollected transfers per termHouseholds with arrears per termCase ManagementUpdatesModified transfer amountsPending/ resolved claimsClaims regarding non-transfers/ wrong transfer amount/ transaction errorsEnrolmentAttendance of selected households at the third community meetingNew enrolments (replacements)Households with main and/ or alternative receivers
Monitoring & Evaluation WorkshopANALYSIS OF MIS-BASED INDICATORS / Example PROCESSNAME OF THE INDICATORMEASUREFILTERGeneralMain Receivers who are household headsTotal NumberBy Districts Gender: femaleGeneralSecondary school-going childrenTotal NumberBy Districts: Mchinji, Machinga, Salima and NenoGeneralSingle and double orphans Total Number and %By Gender By DistrictsTargetingBeneficiaries household membersTotal NumberBy DistrictsGender: maleEnrolmentBeneficiary children information enrolled in secondary levelTotal NumberBy Age: 15-17 years oldDistrict: Chitipa By TAsTransfersAmount paidAmountBy Transfer AgencyPROCESS MONITORING
Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / DefinitionAn independent verification at various stages of the SCTP Project Cycle, to identify whether the process in the field is being carried out according to the Operational Manual (OM)/respective technical annexes.
Differences could lead to identify potential problems either in the operational design or in the way the operation is carried out in the field. Solutions should be suggested accordingly.
Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / Process Monitoring and Process Evaluation DifferencesITEMPROCESS MONITORINGPROCESS EVALUATIONWHOM&E Unit of the Ministry with District OfficialsIndependent consultant/firmSCOPESub-processes within a project cycle phase Processes and sub-processes of a project cycleCOVERAGEClusters and TAsDistricts, entire programme areasMETHODS/TOOLSMainly qualitativeQualitative and quantitativeWHENSpecific issue/problem/challenge is identified.Upon contract, regular reviews, macro issues/challenges identifiedEXAMPLESomebody denounced that all the steps for the targeting process were not followed in a given cluster.E-payment process is not appropriate for SCTP transfer receivers.
Monitoring & Evaluation WorkshopPROCESS MONITORING / How to Execute Process MonitoringIdentification StageSTEP 1Identification of the need
STEP 2 Establish QuestionsPlanning StageSTEP 3 Specify Design/ Methodology
STEP 4 Create Work planImplementation StageSTEP 5 Field Research
STEP 6Analyze Data
STEP 7 Document FindingsReporting/Feedback StageSTEP 8 Disseminate Information
STEP 9Feedback for programme Operational ImprovementPlanning Stage: Key to a successful Process EvaluationDevelopment Stage: Refers to the who, what and how to collect the data need to answer the evaluation questions.Implementation Stage: Field exercise and Data AnalysisReporting/Feedback Stage: Presentation of findings
Monitoring & Evaluation WorkshopCONCLUSIONSProcess Monitoring can only uncover possible reasons why a programme activity is working or not
The findings from a Process Monitoring are the basis behind improvements in effectiveness and efficiency of safety net programmes
A sound evaluation can be done with careful planning and some basic math skills
Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / MethodologyMinistry and District Officials identify issues/challenges/problems relating the designed project cycle in a given area.
Both agree and specify the process to be done:Issue identified, justification and context;Review of the process, using the OM and corresponding technical annex to understand the possible differences that may be occurring with respect to the issue that was identified;Question to be answered is defined;Source of the information to sustain the investigation;Method to be used: interviews, observations, focus groups, and the examination of the records of the programme;Personnel involved; and Possible coverage.
Work plan and budget .13
Monitoring & Evaluation WorkshopPROCESS MONITORING OVERVIEW / MethodologyDefinition of Qualitative analysis
Individual interviewsGroup interviewsDirect observationFocus groupBeneficiary Score cards
Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / Definition Spot Checks allow verifying data quality received by the programme through the SCTP-MIS.To do this, a sample of the total population is taken and compared with information available at the source to verify if both coincide with each other.To test the truthfulness of information recorded in the MIS, qualitative an quantitative verification methods are applied:Statistical formulas applied to see if there are any significant differences Quantitative methodsFocus groups, interviews, and other qualitative methods applied in order to comprehend and/or confirm significant statistical differences.Qualitative methods
Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW /Advantages and DisadvantagesADVANTAGESDISADVANTAGESHelps verifying that information entered into the MIS is accurate
Reasons for differences are analysed, and measures are identified to correct it
Verification is located and limited
Relative expensive and time consuming
Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / ObjectivesEvaluate, through sampling techniques of probabilities, accuracy, and validity, data registered in the MIS, which was reported by Programme actors.Data in the MIS matches data gathered during Spot Checks implementationVerified data must be rightprocesses applied efficiently
Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / Objectives continued2. If significant differences are found, reasons must be analysed using process monitoring techniques.Example:Are beneficiary households actually receiving the respective benefit amount?Detect possible errors that may have occurred during data collection and entry.Possible fraudOthers?
Monitoring & Evaluation WorkshopSPOT CHECKS OVERVIEW / MethodologyMinistry and District Officials identify possible problems in the internal monitoring SCTP-MIS module
Question (hypothesis) to be analysed is definedType of analisis method selected to verify statistical differences
Work plan and budget
Field workCalculation of statistical differencesQualitative work to understand/confirm statistical differences.20
Monitoring & Evaluation WorkshopSPOT CHECKS / How to Execute Spot Check ProcessIdentification StageSTEP 1 Identifying the problem and consequences/risks
STEP 2 Defining possible reasons for the problem
Planning StageSTEP 3 Formulating questions and statistical hypothesis
STEP 4 Designing an instrument (form) to collect information during field work
STEP 5Defining possible sources of actual valid information
STEP 6Determining geogrraphic area for intervention and investigations scope
STEP 7Defining the sample size
STEP 8Creating a work plan Implementation StageSTEP 9Field Work
STEP 10Analysing Results
STEP 11Accepting or rejecting possible reasons for the problem
STEP 12Qualitative investigation to confirm reasons for the problem
Reporting/Feedback StageSTEP 13Feedback for Operational Programme Improvement.21IMPACT EVALUATION
Monitoring & Evaluation WorkshopIMPACT EVALUATION / DefinitionAssesses the changes in participants well-being that can be attributed to a specific programme/intervention;
Determined by comparing the outcomes of programme participants with the outcomes other individuals experience in the absence of the programme (non-participants).
Participants = Treatment GroupNon-participants = Control /Comparison Group
Monitoring & Evaluation WorkshopIMPACT EV