Dark Matter Search with CsI ( Tl ) crystal scintillators : KIMS
Making csi matter project impact - j rangasami
description
Transcript of Making csi matter project impact - j rangasami
![Page 1: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/1.jpg)
© IMPACT CONSULTING 2009
CSI Matters Conference, Johannesburg
25 May 2011
Project impact:Beyond the talk of ‘what we invest in’ to
‘what our investments have achieved’
![Page 2: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/2.jpg)
© IMPACT CONSULTING 2009
Contents
1. Introduction:• Impact Consulting• Defining impact assessment/evaluation
2. Rationale for undertaking project impact evaluations
3. Is impact evaluation neglected/inadequately addressed with the CSI and NPO arena?
4. Whose responsibility?
5. Options for impact evaluation and can they be done affordably?
![Page 3: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/3.jpg)
© IMPACT CONSULTING 2009
Impact Consulting
• We provide services to social development projects and organisations to assist them to maximise their positive social impact
• Services: o M&E: from impact planning to impact evaluation to mentoring
and capacity buildingo Strategy and programme design o Social researcho Information management (DATAFOX)
• Clients: NPOs, donors and government
• Conducted over 40 evaluations in the last 6 years across sectors
![Page 4: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/4.jpg)
© IMPACT CONSULTING 2009
Introduction to impact evaluation
• Impact assessment vs impact evaluation
• Theory of change and theory of action
• Impact vs outcomes
![Page 5: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/5.jpg)
© IMPACT CONSULTING 2009
Theory of change and theory of action
Theory of change
Theory of action
Positive reinforcement
Negative reinforcement
Gold star on chart
Go out for ice-cream
Withhold pocket money
![Page 6: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/6.jpg)
© IMPACT CONSULTING 2009
Theories of change/logic models--------------- Change theory -----------------
----------- Action theory ----------
![Page 7: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/7.jpg)
© IMPACT CONSULTING 2009
Impact evaluation vs other types of M&E
Evaluation Activities
Programme outcomes/
effects
Outcome Evaluation/
ImpactEvaluation (Summative)
Conceptualisationand design of
programme
Clarificatory/theory evaluation
(also needs ax,feasibility study)
First version(s)of programmeimplemented
Processevaluation(Formative)
“Mature”version of
programmeimplemented
ProgrammeMonitoring(Formative/Summative)
Project Cycle
![Page 8: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/8.jpg)
© IMPACT CONSULTING 2009
Why evaluate?
Other intentions?
• marketing/fundraising
• decision-making (re-funding)
Source: Davies/Dart (2005): MSC guide, on http://www.mande.co.uk/docs/MSCGuide.pdfx
![Page 9: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/9.jpg)
© IMPACT CONSULTING 2009
Case study: Infant mortality and nutrition
![Page 10: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/10.jpg)
© IMPACT CONSULTING 2009
Choosing to conduct an impact evaluation
• Is this the appropriate type of evaluation for this project?
• Am I trying to test a theory? (finding gaps, improving)
• Am I trying to maximise the positive impact of my project?
• Am I trying to prove a replicable model?
• Am I wanting to take this project to scale?
![Page 11: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/11.jpg)
© IMPACT CONSULTING 2009
Is impact evaluation a neglected area for CSI and NPOs?
Often not – appropriateness due to short-term funding or
type of project (eg advocacy)
For impactful projects, barriers include:
• Funding
• Technical skills/knowledge
• Capacity (staff and time)
• Data and documentation (primary and secondary)
• Motivation/desire/buy-in (NPO and CSI)
• No culture of learning
![Page 12: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/12.jpg)
© IMPACT CONSULTING 2009
Action learning
![Page 13: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/13.jpg)
© IMPACT CONSULTING 2009
Roles of CSI and NPOs in impact evaluation
Some issues
• Who pays?
• Who decides on evaluation terms and design?
• What methods and questions are valued?
• Which impact/outcomes are valued?
• Who chooses the evaluator?
• Who uses evaluation?
• Who does the M&E?
NPO is ultimate user and driver
Partnership?
![Page 14: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/14.jpg)
© IMPACT CONSULTING 2009
Uses for external evaluators
• Mentoring/capacity building
• Independent “objective” evaluations
• Highly technical studies
NOT IN PLACE OF INTERNAL M&E/LEARNING
![Page 15: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/15.jpg)
© IMPACT CONSULTING 2009
Options for impact evaluation
“Gold standard” approach, focus on quantitative
• NB: mixed methods
• Monitoring to be done internally
• Incorporate with other kinds of M&E
• Focus on learning
• Key impact indicators AND emergent (NB!!)
• Use secondary data where possible (eg education
data)
• Needs investment
![Page 16: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/16.jpg)
© IMPACT CONSULTING 2009
Case study: Infant mortality and nutrition
Case study: Housing and quality of life
![Page 17: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/17.jpg)
© IMPACT CONSULTING 2009
Q
Case study: Impact evaluation design
Rapid evaluation to identify
domains of change:
grounded, exploratory, qualitative
IMPACT EVALUATION
Baseline Year 1 Year 2
300 shack
dwellers
300 shack
dwellers
Community 1
Community 2
100 shack
100 org
100 RDP
100 shack
100 org
100 RDP
100 shack
100 org
100 RDP
100 shack
100 org
100 RDP
Quantitative: Household survey with impact indicators
Qualitative: MSC stories, focus groups, informant interviews
![Page 18: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/18.jpg)
© IMPACT CONSULTING 2009
Results: expected/theoretical
Psychological wellbeing
Safety and
security
Comfort
Health
EducationEconomic wellbeing
Social cohesion?
But was this the NGO’s expectation? Or the donor’s?
...Time...
![Page 19: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/19.jpg)
© IMPACT CONSULTING 2009
• Disappointment that the only significant
QUANTITATIVE findings were for shorter-
term domains of change, despite amazing
qualitative results and indicators towards
longer-term changes (qual not valued)
• Looking for methodological faults to
explain perceived disappointing results
NPO’s response to results
![Page 20: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/20.jpg)
© IMPACT CONSULTING 2009
In conclusion
• Short-term funding = short-term change – this also has to be
reflected in what is “sold”
• Vital to measure expected change and specifically look for
unexpected
• Vital to evaluate throughout the process, not wait till the end
• Funding must be flexible to allow for learning and formative
changes
• Impact evaluation needs to be budgeted for and included
from the beginning
![Page 21: Making csi matter project impact - j rangasami](https://reader036.fdocuments.net/reader036/viewer/2022062405/55783808d8b42a1f5b8b4885/html5/thumbnails/21.jpg)
© IMPACT CONSULTING 2009
Contact details
Jerushah Rangasami
Director: Impact Consulting
082 776 2286
021 424 4166
www.impactconsulting.co.za