20.04.23 Page 1
The impact oriented M&E-system of the German-costarican program on waste management
Dr. Stefanie KrappEvaluation Unit, German Technical Cooperation (GTZ)
Annual Conference of the Canadian Evaluation Society Victoria, CanadaMay 4th, 2010
20.04.23 Page 2
1. Evaluation system of GTZ
2. The waste management program in Costa Rica
3. The program M&E-System
4. Evaluation design
5. Some selected results
6. Conclusions
Contents
20.04.23 Page 3
Planning a development measure
Implementation Completion
e.g. project progress review
e.g. final evaluation
e.g. ex-post evaluation
Sustainability
Results
e.g. ex-ante evaluation
Results-based monitoring
Evaluations
Managing for development results
20.04.23 Page 4
Project progress review (PPR)
e-VAL
End of project evaluation
e-VAL
Ex-post evaluation2-5 years after completion
of project
Continuous results-based monitoring in every development measure is the prerequisite for all evaluations.
Self-evaluationSteering by
operative units
Independent evaluationSteering by
Evaluation Unit
External evaluation
GTZ instruments
Evaluation by the Ministry of
Economic Cooperation
and Development
(BMZ)
Evaluation by auditors
on behalf of BMZ
Evaluation system
GTZ Evaluation system
Decentralised evaluationSteering by
operative units
Mostly interim evaluation
20.04.23 Page 5
Component 1: Comunication and
dialogue
Public and private sector and the civil
society work constructively together in the
waste management.
Component 2: Strategies, plans and legal frame
Relevant stakeholders
accomplish the new legal frame conditions and
political development strategies in
waste management.
Component 4: Competition and
environment friendly behavior of the industrie
Industry, services and trade
implement production
methods and measures for
resource recuperation.
Component 3: Waste
management in municipalities
Improved abilities of selected
municipalities to carry out an
integrated waste management.
German-costarican program “Competition and Environment”
Goal: Improvement of waste management in municipalities and the industry in selected regions of Costa Rica (04/2005 – 03/2013)
20.04.23 Page 6
Intended Impacts
Diffusion: The implemented structures and processes in selected municipalities are taken over by other municipalities and companies which lead to the improvement of waste management in other regions and sectors.
Positive image of Costa Rica as an ecotourism country
Improvement of health
Costa Rica as knowledge carrier in waste management and innovative environmental technologies
Growing environmental awareness
Growing resource efficiency
….
German-costarican program “Competition and Environment”
20.04.23 Page 7
Performance Framework:
Result chains for each component with indicators on each level (output, use of
output, outcomes, impacts)
Internal Monitoring:Baselines & continuous
performance self-assessment with data collection,
time series analysis,
household survey, foto
documentation (output, use of
output, outcome)
Internal and external Eval.:
a) Project Progress Reviews
b) Specific component related
topicsc) Anual panel
according to outcome and
impact indicators
¡Participatively developed and implemented!¡Linked with the program management system!
The impact oriented M&E-System of the program CYMA
M&E System
20.04.23 Page 8
Time series analysis Ex. 1: Development of the quota of correctly deposited, composted or recycled waste in the municipalities
Before Aftert1
70%
t2
75%
t3
78%
t4
85%
t5
90%
Intervention:Implementation of waste management plans
in the municipalities
Ex. 2: Development of the quota of households connected to the waste management system in the municipalities
Internal Monitoring
20.04.23 Page 9
Program group (P) = 8 pilot municipalities
Counterfactual: Comparison group by matching procedure (C) = 3 municipalities not related to the program
Baseline and continuous collection of data according to relevant indicators of P and C
Standardised telephone survey of municipal households about the waste situation in the municipalities (P and C)
Randomwalk: Photographical documentation of the „visual“ waste situation in the municipalities (P and C)
Internal Monitoring C3
20.04.23 Page 10
Panel evaluation design
Impact model:
Capacity of the
counterparts, Outcome and
Impact
Longitudinal study, incl.
expert assessment,
guided interviews
Evaluation frame:
indicators of the impact
model
External Evaluation
20.04.23 Page 11
Outcome
CYMA
Fin. ResourcesCommunication
Staff
Organisational structure
Goal system
Impacts
Component 4Component 1
Component 2
Counterparts
Component 3
Improved waste management in municipalities and industries in selected regions of CR.
Partner structure
Impact Model
20.04.23 Page 12
Baseline
2006
2nd survey
2007
3rd survey
2008
4th survey
2010
The same variables
with the same operationalisation (guided interviews)
are measured with the same people
at different points of time.
Before After
Panel groups
External Evaluation: Panel
20.04.23 Page 13
(2006, 2007, 2008, 2010)
Experts
Stakeholders
Counter-parts
3 Ministries (Planning, Health, Environment), Chamber of Industries, IFAM, 4 pilot municipalities, program respresentatives
Municipalities, Tourism, university professors, Journalists
Gtz-Program PRODELO, 4 municipalities, chambers, ministeries (Economy, Agriculture), NGOs, Institutes (CNP+L, CEGESTI, CICAP), Associations (ACEPESA, ACIPLAST, FEMETROM, FEDEMUR, CONADECO), local respresentatives (PRU-GAM, UNGL), political respresentatives, companies (Holcim, Dos Pinos, Prolusa, Sur Química, Gente Reciclando, Fundación Escazú Recicla)
Panel groups
20.04.23 Page 14
Sumary
Monitoring data:
Baseline Time series Standardised telephone household survey Foto documentation of the waste situation in municipalities
Double-Difference Analysis:Comparison of data collected at t3 with data collected at t1 (baseline data) and t2 of the program and comparison group
Data of the Panel survey
Comparison of data collected at t4 with data collected at t1 (baseline), t2 and t3
comparison with monitoring results at different points of time
Determine the attribution of program impacts
20.04.23 Page 15
Law on waste management almost ratified, National Plan of Waste Management, Municipal Waste Management Plans
Diffusion of resource gentle and environment friendly production in the private industry
Increased separated disposal of waste
Intensified interaction in the waste sector and therefore continuing presence of the topic in the public
Appreciation of the political role of the chamber of industries
Costa Rica as knowledge carrier
BUT: still critical pollution (mainly oncerning water, waste disposal and recycling)
Selected results
20.04.23 Page 16
Main functions and goals of the M&E-System Joint learning
Information on how the program is doing
Steering information for the program management
Evidence of program results (outcomes and impacts)
Basis for the development of a national waste management monitoring system in Costa Rica
Main conclusion:
The M&E system is not perfect but tries to show how the program is doing and gives valid information on outcomes and hints on impacts under the given circumstances by applying different methods as „rigorous“ as posible!
Conclusions
20.04.23 Page 17
logical result chains and useful indicators flexible system that allows adaptations baselines continuous data collection good data basis in the municipalities participation of counterparts in the development and
implementation of the M&E-system (ownership) willing program manager M&E system connected with the management system staff responsible for the monitoring system combination of internal and external evaluations anchored in
the monitoring system pretension of rigorous evaluation designs to attribute impacts
(determination of a comparison group right at the beginning)
Conclusions: Prerequisites
20.04.23 Page 18
Thank you! Merci! Gracias! Danke!
Dr. Stefanie KrappEvaluation Unit, German Technical Cooperation (GTZ)
www.gtz.de
Top Related