Jy Recoup Mand E
-
date post
22-Oct-2014 -
Category
Technology
-
view
1.222 -
download
1
description
Transcript of Jy Recoup Mand E
Making a difference:M&E of policy researchJohn Young: ODI, [email protected]
What should you measure?It depends what you’re trying to do….
“If you don't know where you are going, any road will get you there”
Should be:
Specific
Measurable
Achievable
Realistic
Time-bound
(Objective)
Whatever you measure
M&E of policy research1. Strategy and direction: Logframes;
Social Network Analysis; Impact Pathways; Modular Matrices
2. Management: ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits; Horizontal Evaluation; Appreciative Inquiry
3. Outputs: Evaluating academic articles and research reports; Evaluating policy and briefing papers; Evaluating websites; Evaluating networks; After Action Reviews
4. Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys
5. Outcomes and impacts: Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies
www.odi.org.uk/RAPID/Publications/RAPID_WP_281.html
Logical frameworksGoal Indicator MOV Assumptions/Risks
Purpose Indicator MOV Assumptions/Risks
Output 1 Indicator MOV Assumptions/Risks
Output 2 Indicator MOV
Output 3 Indicator MOV
Output 4 Indicator MOV
√√
...and many projects fail when the inputs cease...
Change takes a long time
Inputs Activities Outputs Outcomes Impact
Other Actors
Project Effort
Behaviour Change
Focusing on change
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Outcome_mapping.html
Emphasis on “learning”
“…every time we do something again, we should do it better than the last time…”
Goals ResultsActivities
Learnduring
Learnafter
Learnbefore
External networks; Colleagues; Information assets; Own knowledge
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Index.html
Starts with the attitude that someone has probably already done what I am about to do.
I wonder who?”
Learning before: Peer Assist
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Peer_assists.html
• What was the situation?
• What was the challenge?
• What was done?
• What was the result?
• What lessons can be drawn?
Learning During: Stories
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Stories.htmlwww.mande.co.uk/docs/MSCGuide.pdf
Most significant change1.Best stories at each level
2.Synthesis
Stories of change
Horizontal evaluation• Peer review
– Choose the moment– Choose your peers– Limited criteria– e.g. ODI Peer Review
Diagram 8. The Appreciative Inquiry ‘5-D’ model
• Appreciative enquiry– Self-evaluation– CGIAR/CIAT– Workshop
www.odi.org.uk/RAPID/Tools/Toolkits/KM/AAR.html
An after action review asks 4 simple questions:
15 minute team debrief, conducted in a “rank-free” environment.
Learning after: AAR
• What was supposed to happen?
• What actually happened?• Why was there a
difference?• What can we learn from it?
Case & Episode Studies• Classical case studies: how did
evidence shape policy decisions?– e.g. IFPRI & IDRC– Overestimate the role of research
www.idrc.ca/en/ev-26606-201-1-DO_TOPIC.html
www.ifpri.org/impact/impact.htm
www.odi.org.uk/RAPID/Publications/BRP_ITDG.html
www.odi.org.uk/RAPID/Projects/PPA0104/Index.html
www.gdnet.org/middle.php?oid=175
• Episode studies: retrospective tracking back from policy change– e.g. PRSPs, SL, AHC– Underestimate the role of research
Social Network Analysis
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysis.html
RAPID Outcome Mapping
www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html
Impact log• Partner’s research quoted in media or on a
blog: ebpdn research on Japanese Aid on BBC Afrique; Waldo Mendoza of CIES quoted in Géstion.
• ebpdn used to find position: MSc student in US got intern position in Uganda; a member cites active membership of ebpdn in his on-line CVs (on other web sites)
• RAPID framework in academia: Garrett, J. “Improving Results for Nutrition” in the Journal of Nutrition (2008)
• RAPID approaches incorporated into organisations: CIDA research studies to make it more participatory
Stories
RSS Feeds
Webstats
Other approaches: Public• Citations, webstats, media
logs etc• Surveys
– Quantitative– Qualitative
• Distribution lists and attendance records
• Meeting evaluations• Logs:
– The expected– The unexpected– How you have changed
Evaluation: Practical Guidelines, Research Councils UK. 2002
www.rcuk.ac.uk/cmsweb/downloads/rcuk/publications/evaluationguide.pdf