Post on 24-May-2015
description
Logo
Zenebe Uraguchi
HELVETAS Swiss Intercooperation Bangladesh
Regional Workshop25.09.2013
Ben Tre, Vietnam
Monitoring and Results Measurement (MRM) System
Logo Why implement MRM system based on the DCED standard ?
Internally & externally
Cornerstone of good project steering Guidance for strategic direction for
achieving impacts sustainably
Externally
Demonstrates impact to the donors & other stakeholders
Contributes to credibility of projects
Internally
Fundamental to the learning process by providing vital information that projects need
Enhances making good decisions in order to maximize impacts
DCED: Donor Committee for Enterprise Development
Logo What is included in the (DCED) standard
The DCED is a forum; 23 members (bilateral donors, including SDC, multilateral agencies and private foundations
Member agencies and others share experiences and formulate a common understanding of good practices
7. Reporting
The DCED Standard
8. Managing the
System
4. Estimating attributable changes
5. Capturing wider changes in the system
1. Articulating the
results chains
2. Defining indicators
of change
3. Measuring changes in indicators
6. Tracking programme costs
Logo What is new as a system?
• Do events unfold as planned? Yes? No? Why?• An internal system for MRM instead of external consultants
One-off events
External consultants
Internal MRM system
Improving intervention logics
Logo
Source: Peter Roggekamp, Hans Posthumus
What we don’t want to do?
series of changes resulting from activities of the project
local systems –people, businesses, government, civil society organizations and the relationships, rules, policies and cultural norms that govern how they interact and work together
Goal
Logo
Source: Peter Roggekamp, Hans Posthumus
What we want to do?
LogoWhy MRM is important!
Source: Peter Roggekamp, Hans Posthumus
Logo
Source: Peter Roggekamp, Hans Posthumus
LogoLet us talk about the ball & goal
Systematic Intervention
Market trigger
Market uptake
Enterprise performance
Impact
The ball is the series of changes resulting from activities of the project
A results chain or impact logic clearly outlines the series of changes project staff expect, starting with project activities and ending with goal level impacts
LogoAn example from RLDP, Tanzania
Farmers attend demo plot sessions
Farmers understand technology
Farmers apply technology
MSK establishesdemo plots
Farmers‘ groups sign contract with MSK
MSK distributes inputs
Farmers apply inputs
Farmers increase yield
Farmers increase net income
MSK sensitize farmers and fgroups
LogoAn example from Samriddhi, Bangladesh
Facilitating technical capacity of market
actors
Improving access to technologies/good practices & inputs
Forming & strengthening
producers’ groups
Facilitating access to adapted financial
services & products
Systemic intervention
Market trigger
Enhanced capacity of local service providers
on technical knowledge & skills
Increased engagement of private sector
enterprises
Enhanced capacity of local service providers & producers’ groups in business planning & financial literacy
Improved organizational &
business skills of local service
providers
Service provision for affordable & quality services & inputs available; improved linkages with private sector enterprises & financial service providers established; improved &/or new technologies &
practices in bull fattening available
Better functioning market system
Market uptake
Increased production & productivity
Increased additional employment & net income by producers
Improved performance
Impact
Producers start establishing functional
relationships with private sector
enterprises
Producers apply improved bull rearing
practices using quality inputs & services
tPoducers start developing & implementing business plans by
identifying & negotiating with a range of financial service providers
Logo
An example on results chain: direct & indirect outreach
Training manual developed with the support of the project / other players e.g., Department of Livestock Services
Activity
Market trigger
Market uptake
Improved performance
Impact
Appropriate private sector company identified; deal made
Local service providers selected
Private sector company provided training to local service providers on improved bull fattening technology
Local service providers improved their skills & knowledge on improved bull fattening technology
Local service providers provided information on services to producers
Producers are aware of service availability & are willing to receive & pay for the services
Producers started using the services & adopted improved bull fattening technology
Farmers fattened bulls in short cycles
Farmers received better price for their bulls
Increased profit for farmers
Additional income of xx Taka for yy farmers
Other farmers are influenced to fatten bulls using new
technology
Additional farmers received better price for their bulls
Increased profit for indirect farmers
Indirect
outreach
LogoHow about developing a results chain from your project?
Systemic intervention
Market system changes
Market uptake
Improved performance
Impact
Tips:
1)Please select one intervention 2)Please be as detailed as possible
in the impact logic (changes at each level)
LogoResults Chain and Logframes
Compatible
Results chain increase the relevance of Logframe as management tool
Results chain provide details of the process for achieving results
LogoCompatible with Logframe
•
• • •
• • •
• • •
Log Frame
Results chain
LogoDefining Indicators
# box
indicator
12
11
10
9
8
7
6
5
4
3
2
12
3
4
5 6
7 8
9
1
10
11
12
LogoHow to formulate indicators
• Each box needs at least 1 indicator• Use quantitative and qualitative indicators
- Behaviour, perception, motivation, …• Indicators should be S M A R T !
- Specific- Measureable- Attributable- Relevant- Timely
LogoIndicators….
We should always ask: “Is each piece of information essential to decision making?”
What is a good indicator?
Relevance
Precision
% growth in a business membership organization
% change (year-on-year) of total businesses paying full membership fees in business
membership organization
Lazy & loose
Tight & precise
Time bounded-ness (e.g., profits in past month vs.
in past year )
Realistic
LogoDefining indicators of change
An indicator
shows what success looks like
outputs are produced & outcomes are achieved
if
indicators specify how the project will measure
therefore
the changes anticipated in the results chains are really occurring
whether
if so to what extent?+ how?
if not then why?
LogoIndicators
indicators to be derived from the logic of the results chain
Farmers reduced post-harvest loss of vegetables
Farmers adopt improved plastic crates for harvesting
and grading vegetables
Number of farmers who adopted plastic crates
% reduction of wastage or % increase in the sales of
produce
Quality of produce
+
Quantitative indicator
Quantitative indicator
tracking these indicators very important frequency of tracking will depend on the specific targets or projections
set by the project
Quantitative indicator
LogoIndicators & quantitative projection
Projections: change in the value of an indicator due to the project by a specific date
What to do?
1) At the start of activities, make clear projections about expected impacts
Why we do this?
give staff targets to aim forprovide staff with feedback if an
intervention is on track
2) Clear projection of key quantitative indicators
give an indication of whether particular investments are worth their cost & impacts
3) Do market research, have field observations or other credible sources (e.g., secondary resources)
provide realistic / credible projection
4) Periodically update projections (actual vs. projection)
Help reflect new data collected on indicators of change
LogoAn example from PakistanAssumption Calucation Projection
50% success rate 445*50% 222 garment producers improveOct-13
GPs produce 6 units per month
150Rs/unit*6units
Garment producers increase by 900Rs/month. Apr-13
Margins increase by 150Rs/unit
445 garment producers earn +150/unit Apr-13
75% success rate 594*75% 445 garment producers produceApr-13
90% success rate 660*90% 594 garment producers improveFeb-13
Each sales agent services 30 garment producers
22*30 660 garment producers servicedFeb-13
75% uptake rate 30*75% 22 women adoptDec-12
30 women trainedOct-12
Women adopt new service agents business model
Sales agents deliver services to garment producers
Garment producers improve knowledge of production techniques
Garment producers produce high quality garments
Garment producers earn higher margins
Garment producers increase incomes
Garment producers improve social status in house holds & community
Project trains
women to adopt new
sales agents business model
Constraint: Limited mobility of female garment producers (GPs)Identified solution: Sales agents (SAs) travel and provide services
LogoData collection tools
ObservationIn-depth
interview
Key informant interview (KII)
Company / partner records
Focus Group Discussions
(FGD)
Outsourced surveys
Secondary
Sources
In-house
surveys
LogoObservation
Source: HPC 2012
LogoSecondary sources
Source: HPC 2012
LogoCompany records
Source: HPC 2012
LogoFocus Group Discussion (FGD)
Source: HPC 2012
LogoStakeholder consultation
Source: HPC 2012
LogoIn-depth interviews
Source: HPC 2012
LogoIn house surveys
Source: HPC 2012
LogoOutsourced surveys
Source: HPC 2012
LogoBalance between rigor and cost
Rigour
Costs
Observations
Big surveys
LogoMix of methods and triangulation
2
3
4
5 6
7 8
9
1
10
11
12
Same tool for several boxes For one box: use several simple tools for
triangulation
Tool 1
Tool 2
Tool 3
Document the process, not only findings; state the limitations
LogoMeasuring change• Necessary to measure change• Ideally Pre-Intervention• Sources:- Previous monitoring data- Previous market analyses- Additional data collection and
analysis e.g. through surveys, interiviews, FGD, etc.
• Usually multiple baseline are needed across interventions
Impact
Indicator
LogoMissing baseline
• What if baseline has not been collected?• What if the beneficiaires have not been known at the start of
the intervention?
- Use secondary data;- Derive a baseline from retrospective research;- Use data from other regions with similar demographics etc.- Use partners‘ records
LogoResults Measurement Plans: example
Level Impact chain Questions Indicators How? Who ? When? What do we have?
Impact Additional income
Whether farmers and laborers have increased income?
Net increase in income
Survey; secondary sources
Research firm; sector team
June xx Census from xx; secondary source
Market uptake
Farmers cultivated early varietyVegetables
Do the farmers cultivate early variety vegetable?Do they get proper information?
No. of farmers who cultivate earlyvariety vegetable?Farmers' knowledge level
Survey Research firm; sector team
July XX
Producers improved their knowledge oncultivation techniques of early varietyVegetables
How many producers have been reached throughpromotional activities?How much have they learnt from the promotional activities?
No. producers aware;change in knowledge of producers
Observation of theorganizers, reports
Sector team
June xx
LogoEstimating attributable changes
“it is better to be approximately right than precisely wrong.“John Maynard Keynes
The outcomes might have happened without
the intervention (counterfactual)
Target groups might have
been better off without the intervention
(displacement or unintended
consequences)
The outcomes might not be due to the intervention, but due to other factors (e.g.,
projects, weather changes)
Logo
b= before
Impact= a-c
a= after
c= changes without the project
Time
Impact
Intervention starts Intervention ends
LogoAttribution: Influencing factors
Impact
External factors
Causality
Many
Few
Easy to establish
More difficult
to establish
Adapted from A. Miehlbradt
LogoAttribute plausibly
Impact
External factors
Causality
Many
Few
Easy to establish
More difficult
to establish
Increasing complexity
Demonstrate rigorously
Validate transparently
Estimate contribution transparently
Adapted from A. Miehlbrandt
LogoLessons from Samriddhi
Samriddhi had its “mock” audit for the compliance of its MRM system with the DCED standard in 2013.
= Complies with the Standard = Additional work required = Current system does not address
The project was assessed to have 75% of compliance with the standard, which was good.
Areas of improvement include:
1)Developing results chain not per value chain but by intervention2)Collecting specific and detailed information from individual
producers, and not from producers’ groups
LogoSuccess factors for Samriddhi’s MRM system
1) Designing MRM system to support decision making at all levels
2) Availability of MRM manual describing and elaborating the “nitty-gritty” of the MRM system
3) Making MRM system part of everyone’s job
4) Giving equal emphasis to qualitative and quantitative data with core indicators
5) Developing an integrated MRM tool
6) Ensuring staff training as a priority
7) Cooperation from the donor and good steering of the project
Logo
Thank you
LogoComparing different tools
Tool What is it Purpose Sampling guide Tips
Observation quick assessment of activity effectivenessgathers qualitativeinformation; collects preliminary information before the use of other tools
validate findings from other tools; understand the process of change
required for regular monitoring purposes
undertaken in the normal course of work or plannedeffective if a checklist of required information is prepared before a visitMay include informal discussions; familiarity with context required
In-depth interviews
are detailed and require a lot of time toextract explanations
In-depth analysis of change Validating facts through details
number depends on the context of thestudy or impact assessmentrespondents are intentionally chosen for relevant information
watch out for coached responses from the respondentsuse follow up questions and probes to elicit more extensive responses
LogoComparing different tools
Tool What is it Purpose Sampling guide Tips
Key Informant interviews (KII)
qualitative in‐depth interviews with individuals whohave first‐hand knowledge of the issues
collect information from a wide range ofindividuals who can provide important and significant insight into the issue
Sample size ranges from 2‐5 individuals
must have first‐hand knowledge about the issues associated with theIntervention diverse set of representatives are preferable
Pocket surveys
gather information on quantitative indicators when significant rigor isnot needed
validate the detailed findings of in‐depth interviews, with a largersample size
sample size ranges from 20‐40
questions should be straight‐forward and generally directed at getting facts
Focus Group Discussions (FGD)
conducted when consensus is needed on a topic, or to get a common view of attendants
explore the process of change extensively; provide deeper understanding ofwhy changes are taking place
the group consists of 8‐15 people
guideline should be prepared on need of info
LogoComparing different tools
Tool What is it Purpose Sampling guide Tips
Formal surveys
conducted for 'special cases' whichrequire us to generate specific quantitative information
validate the findings of an informal survey or an FGD
Depends on the size of the population; usually between 200-400
Questionnaire is structured with closed ended questions, based on the checklist Simple questions to get facts