Planning for Measurement, Monitoring and Evaluation the Reform

Assoc. Prof. Dr. Wongsa Laohasiriwong

Faculty of Public Health, Khon Kaen University

Planning forPlanning for Measurement, Measurement, Monitoring and Evaluation the Monitoring and Evaluation the

Reform Reform

1

2

What is it?What is it?

Monitoring and Evaluation Monitoring and Evaluation (M&E) Plan(M&E) Plan

is a guide a guide as to what what you should evaluateyou should evaluate, what information what information you needneed, and whowho you are evaluating for.

3

Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

4

Characteristic of M&E Characteristic of M&E PlanPlan

UtilityUtility

- serve practical information needs of intended users.

Feasibility Feasibility - be realistic, prudent, diplomatic and frugal.

Propriety Propriety

- conducted legally, ethically, and with regard to those involved in and affected by the evaluation .

Accuracy Accuracy - reveal and convey technically accurate information.


5

Developing & Developing & Implementing Implementing

an M&E Plan: Logic an M&E Plan: Logic ModelModel

Inputs

•Human resources

•Understanding of the program •Authority and mandate •Stakeholders

Processes•Advocate•Assess strategic information needs•Assess information systems capabilities •Achieve consensus and commitment•Develop mechanism for M&E plan review•Prepare document for final approval

Impacts

Improved health status

Long-term

•Evidence-based decisions for improving programs

OutcomesOutput

•M&E Plan Document

Short-term

•M&E System for obtaining Strategic Information decision making

Performance Performance MeasurementMeasurement

6

is the ongoing monitoring ongoing monitoring and reporting of program and reporting of program accomplishments, accomplishments, particularly progress towards towards pre-established goals.pre-established goals.

Source: Stephanie Shipman and Joseph Wholey. 1998. Performance Measurement and Evaluation: Definitions and Relationships

How to Develop a How to Develop a Monitoring and Evaluation Monitoring and Evaluation

Plan?Plan?

Step 1. - Step 1. - Identify your evaluation audienceevaluation audience.

Step 2. - Step 2. - Define the evaluation questions.evaluation questions.

Step 3. - Step 3. - Identify the monitoring questions.monitoring questions.

Step 4. - Step 4. - Identify the indicators and data indicators and data sourcessources..

Step 5. - Step 5. - Identify who is responsiblewho is responsible for data collection and timelines.

Step 6. - Step 6. - Identify who will evaluate the who will evaluate the data, howdata, how it will be reported, and when.

Step 7. - Step 7. - ReviewReview the M&E plan.7


Characteristics of Characteristics of Good Good

Monitoring SystemsMonitoring SystemsFollows the reform Follows the reform process.

Cost-effectiveCost-effective for the operating unit.

Describes the stages or stages or events used for rating events used for rating progressprogress .

Describes the methodology or process methodology or process used for rating the current stage of the reform.

8

Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

Characteristics of Characteristics of Good Good Monitoring Systems (cont.)Monitoring Systems (cont.)

Defines key terms Defines key terms e.g. “operational,” “fully functioning,” or “fully implemented”.

Specifies which activities activities are most critical to the are most critical to the strategic objective. strategic objective.

9


Provides a rationale Provides a rationale for how future performance targets are set.

Includes reform stakeholders as partners stakeholders as partners in reviewing the process of adoption and implementation and in setting future performance targets.

Approaches to Approaches to MonitoringMonitoring

10


1) Simple Quantitative 1) Simple Quantitative Approaches (SQA)Approaches (SQA)

11

Using a purely quantitative approach is a common way common way of showing the progressof showing the progress.

Typically the agency promotes a list of reforms during a three-year periodthree-year period, and each year each year it records records the number of reformsthe number of reforms that have been successfully successfully completedcompleted.Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

2) Descriptive Approaches 2) Descriptive Approaches (DA)(DA)

12

The progress in adopting and implementing reforms is reforms is frequently not readily frequently not readily quantifiablequantifiable. .

It needs clearly define in advance the key events the key events in the reform process reform process and the planned planned results results and then monitor their monitor their achievementachievement.

This permits a relatively objective comparison of planned with actual results throughout actual results throughout the performance period.the performance period.


3) Composite Approaches 3) Composite Approaches (CA) (CA)

(Qualitative and Quantitative Methods) (Qualitative and Quantitative Methods)

13

Based on constructing more complex complex indicesindices.

These indices are frequently (but not necessarily) based on phases or stages phases or stages identified as milestone eventsidentified as milestone events and are expressed numericallyexpressed numerically.

Each phase or stage in the reform process is assigned a number or valuenumber or value..

Missions should provide a detailed Missions should provide a detailed explanation explanation to define each stagedefine each stage in a given index and to describe the related scoring scale. If not thoughtfully considered and well documented, can be imprecise and be imprecise and subjective.subjective.Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

MonitoringMonitoring

14

is collecting, recording, and collecting, recording, and reportingreporting information concerning performance that tracking of services and programmes performances.

Routine WorkingRoutine Working

Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

15

Why do we monitor?Why do we monitor?Simply because we know that things we know that things

don’t always go according to plandon’t always go according to plan (no matter how much we prepare).

To better assess how well a policy or how well a policy or programme is achievingprogramme is achieving its intended target.

To detect and reactdetect and react appropriately to deviations and changes to planschanges to plans


16

What do we What do we monitormonitor??

Men (human resources)

MachinesMaterialsMoneyTechnology

SpaceTimeTasksQuality/Technical

Performance

What we looking for?

Progress, Schedule, Costs, Performances: Work begin,

the completionSource: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

17

When do we When do we monitor?monitor?

End of the Plan Continuously Regularly Logically While there is still time to react As soon as possible At task completion At pre-planned decision points (milestones)


18

How do we How do we monitor?monitor?

Through information collection.Through meetings with stakeholders.For schedule – Update Gantt ChartsUsing Earned Value AnalysisCalculate Critical RatiosMilestonesReportsTests, Inspections, Auditing


EvaluationEvaluation is the systematic process of systematic process of the merit, value and worththe merit, value and worth of someone or something.

Evaluation TermsEvaluation Terms EfficacyEfficacy : Can it work? (normally for

medicine) EffectivenessEffectiveness : Does it work in reality?

EfficiencyEfficiency : Is it worth doing it, compared to other things we could do with the same money? (Cost-effectiveness)

EvaluatiEvaluationon

EffectivenessEffectivenessEffectiveness refers to the connection the connection between inputs and outputsbetween inputs and outputs, the achievement of objectivesachievement of objectives, means how well the job gets done i.e. the quality of the output.

EfficiencyEfficiency Efficiency refers essentially to the

extent to which outputs are attained outputs are attained while minimizing production costs, while minimizing production costs, achievement of the ends with the least amount of resources.least amount of resources.

There are 2 types of EfficiencyThere are 2 types of Efficiency1. Technical efficiency 2. Allocative efficiency

Cost-Benefit and Cost-Effectiveness Analyses

23

Types of Types of EvaluationEvaluation

Outcomes

Impacts

Inputs

Activities

Outputs

Implementation

Results

Impact Evaluation

Outcome Evaluation

Process Evaluation

(1) Define the study question and perspectiveDefine the study question and perspectiveDescribe alternative health programs to be considered determine study perspective

(decision making context of the study)

(4) Derive decision rulesDerive decision rulesCompare cost–benefit ratios , identify other decision criteria address implementation

issues

(2) Identify, measure, and value costs and benefitsIdentify, measure, and value costs and benefitsIdentify relevant costs and benefits for each program measure costs and benefits for each

program in appropriate physical units value costs and benefits for each program, using market valuation of factor inputs, stated preference valuations of health gains, or other

valuation methods

(3) Analyse costs and benefitsAnalyse costs and benefitsAdjust costs and benefits for differential timing (discounting) where appropriate

perform incremental analysis of costs and benefits of alternatives perform sensitivity analysis to allow for uncertainty in estimates of costs and consequences

Four Stages of EvaluationFour Stages of Evaluation

Once an organization has analyzed its mission, identified all its stakeholders, and defined its goals, it needs a way to measure progress toward those goals.

25

• Key Performance Key Performance Indicators Indicators are those are those measurements.measurements.

Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

Key Performance IndicatorsKey Performance Indicators, also known as KPIKPI or Key Key Success Indicators (KSI), Success Indicators (KSI), help an organization define and measure progress toward organizational goals.

26

Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

Key Performance Indicators (KPI)

“Measurable indicators that will be used to report progress that is chosen to reflect the critical success strategic”

27

Source: John McGerty, Key Performance Indicator

Specific Measurable Acceptable Realistic Time frame

Source: Clara Freeman. http://www.aboutfreelancewriting.com/2012/03/what-working-smarter-means-for-the-freelance-entrepreneur/

Specific Specific There has to be one

widely accepted definition of the KPI to make sure the different make sure the different users interpret it the users interpret it the same way same way and, as a result, come to the same result, come to the same and right conclusions and right conclusions which they can act on.


Measurable Measurable The KPI has to be measurable to define a standard, define a standard, budget or norm, to budget or norm, to make it possible to make it possible to measure the actual measure the actual valuevalue and to make the actual value comparable to the budgeted value.


AchievableAchievableIt is really important for the important for the acceptance of KPI’s and acceptance of KPI’s and Performance Management in Performance Management in general within the general within the organization that it possible to organization that it possible to achieved.achieved. Nothing is more discouraging than striving for a goal that you will never obtain.


RealisticRealistic The KPI must give

more insight in the performance of the organization in obtaining its goal and strategy. Therefore the action action need to happen.need to happen.


Time FrameTime FrameIt is important to important to express the value express the value of the KPI in time. of the KPI in time. The realization and standardization of the KPI therefore has to be time frame.


A business A business – the percentage of income that comes from return customers.

34

• A Health Promotion Hospital A Health Promotion Hospital – The percentage of coverage Polio vaccination in children age under 5 years old.

• A school A school – the graduation rates of students.• A Customer Service Department A Customer Service Department – the number of monthly customer complaint.

Benefits of Evaluation Benefits of Evaluation FindingFinding

Important input for government Important input for government decision making and prioritizationdecision making and prioritization, particularly in the budget process.

Assists managers by revealing the revealing the performance of ongoing activities performance of ongoing activities at the project, program or sector levels.

Used to assess the performance of assess the performance of organizations and institutional organizations and institutional reform processes.reform processes.

Contribute to accountability Contribute to accountability mechanismsmechanisms, whereby managers and governments can be held accountable for the performance of their activities.35

Source: Keith Mackay. 1998. Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C.

References: John McGerty, Key Performance Indicator F. John Reh, Key Performance Indicators (KPI) How an organization

defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

Clara Freeman. http://www.aboutfreelancewriting.com/2012/03/what-working-smarter-means-for-the-freelance-entrepreneur

Patricia Vondal. USAID. 2000. Monitoring the policy reform process. Developing a Monitoring & Evaluation Plan. From

evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159.

Stephanie Shipman and Joseph Wholey. 1998. Performance Measurement and Evaluation: Definitions and Relationships.

Keith Mackay. 1998. Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C.

Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

36

Planning for Measurement, Monitoring and Evaluation the Reform

Documents

Transcript of Planning for Measurement, Monitoring and Evaluation the Reform