X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE...
-
Upload
leo-morris -
Category
Documents
-
view
231 -
download
0
Transcript of X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE...
![Page 1: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/1.jpg)
v
MEASURING IMPACT: USING QUANTITATIVE RESEARCHEibhlín Ní Ógáin
London, November 2012
![Page 2: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/2.jpg)
AGENDA
1 What is impact measurement
2 NPC’s approach
3 Quantitative approaches
4 Tips and things to remember
![Page 3: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/3.jpg)
TRANSFORMING THE CHARITY SECTOR
Increasing the impact of charities
eg, outcomes frameworks
Charity
Funder Sector
We are a think tank and
consultancy working at the nexus between charities and
funders
Increasing the impact of funders
eg, effective commissioning
Strengthening the sector
eg, collaboration towards shared
goals
![Page 4: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/4.jpg)
WHAT IS IMPACT MEASUREMENT?
![Page 5: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/5.jpg)
5
Impact: The difference an organisation or activity makes to the people or problem it aims to help
Impact measurement: Planning, managing, tracking and reviewing how much difference you are making
![Page 6: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/6.jpg)
NPC’S APPROACH TO MEASURING SOCIAL IMPACT
![Page 7: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/7.jpg)
7
NPC’S APPROACH TO MEASURING SOCIAL IMPACT
Strategic vision / goals
A well-developed Theory of Change
Existing evidence Appropriate measurement tools
![Page 8: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/8.jpg)
QUANTITATIVE RESEARCH
8
![Page 9: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/9.jpg)
9
WHY USE QUANTITATIVE RESEARCH
• Qualitative: understand something in detail. Rich information and insight.
• Quantitative: how many people? How large an effect? How much impact and how robust?
• Use quantitative research when dealing with large numbers of people and when you want to show a consistent effect.
• E.g does new type of psychotherapy lead to improved mental well-being? Need consistent measure and need large sample.
• Depends on type of ‘intervention’ and questions you are trying to answer.
• Know a lot, measure a little, know a little, measure a lot.
![Page 10: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/10.jpg)
10
DIFFERENT METHODS IN QUANTITATIVE RESEARCH
• Ranges from easy to measure to hard to measure
• Easier: Hard and objective outcomes: move into employment, gain a qualification, reduction in reoffending.
• Harder: Softer, difficult to observe outcomes: changes in feelings, personal growth, well-being.
• NPC’s preference is for previously tested scales and questionnaires.
• If there is a gap, develop own questions.
![Page 11: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/11.jpg)
11
TYPES OF QUANTITATIVE TOOLS AND THEIR USES
11
Type of toolIdentify needs of group
Measures progress for individuals
Aggregate to show change
Robust measure of change
Explain why change has happened
Admin. data ? ? ? Case work tool (eg, Recovery Star) ? Clinical tools Scales (developed by researchers)
Single questions
Hard outcomes (eg, statutory data)
![Page 12: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/12.jpg)
12
WHEN TO MEASURE
• Pre-post design: how outcomes change over time by collecting data before (pre) and after (post) the programme.
• When you think change happens influences your decision about:
– When to collect the before measure
– When to collect the after measure
12
Before measure
After measure
Programme / service
![Page 13: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/13.jpg)
13
THINK ABOUT ATTRIBUTION
• Is the programme the only factor influencing any changes? What would have happened anyway?
• Control groups
• Compare before and after data for participants and non-participants, ideally matching on their key characteristics
• Holy grail: Randomised Control Trial, where potential participants randomly assigned to control group
13
![Page 14: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/14.jpg)
14
THINK ABOUT SAMPLING
• Representativeness
– Does your sample represent your population?
• Response rates
– Is your response rate high enough to avoid bias?
• Statistical validity
– Are your sample sizes large enough for analysis?
• Oversampling (segmentation)
14
Biased/small sample => Avoid percentages
![Page 15: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/15.jpg)
15
YOU DON’T NEED TO MEASURE EVERYTHING
![Page 16: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/16.jpg)
16
WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF:
Has anyone already proved the causal link you want to make?
Yes
No
Is it really important you need data on this outcome?
Do you already collect this data?
Don’t collect unless you decide it is really important
Use research to limit your data collection
Yes
No
Yes
Can you access this data?
No
Develop your own data source
Use this data (if it is fit for purpose)
Does anyone else collect this data?
NoNo
Yes
Yes
![Page 17: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/17.jpg)
17
PRIORITISE OUTCOMES THAT:
• you directly influence (rather than indirectly support)
• are important / material to your mission
• are not too costly to measure
• will produce credible data?
![Page 18: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/18.jpg)
18
HOW ROBUST DOES EVIDENCE OF YOUR SOCIAL IMPACT NEED TO BE?
• What do your target stakeholders (investors?) think?
• What is possible, given your resources etc?
![Page 19: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/19.jpg)
19
LEVELS OF EVIDENCE
Randomised control trial
Anecdotes / quotes
Before and after survey
Self-reported change
Case studies
Control groups
Credibility
Basic Advanced
Nesta level 1 Nesta level 2 Nesta levels 3, 4, 5
![Page 20: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/20.jpg)
20
6 TIPS FOR MEASURING YOUR IMPACT
2. Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation.
1. Decide what’s important. Define your research question and be clear what you want to achieve.
3. Know your limits. You need to choose an approach that matches your resources.
4. Consider investing in training. If you are not confident, get training or seek external help.
5. Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs.
6. Use the information that you collect: If you don’t use it, don’t measure.
![Page 21: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/21.jpg)
21
RESOURCES
• Review of evidence of ICT and learning: http://educationendowmentfoundation.org.uk/toolkit/ict
• A good website to look for measures and indicators: http://wilderdom.com/tools/ToolsSummaries.html
• How to communicate your results: Hedley, S et al (2010) Talking about results. New Philanthropy Capital.
• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco.
• http://www.theoryofchange.org/• The innovation network’s logic model workbook:
http://www.innonet.org/client_docs/File/logic_model_workbook.pdf• The Centre for What works (http://www.whatworks.org/) has an outcome portal where
you can browse different outcomes and their corresponding indicators. • TRASI (http://trasi.foundationcenter.org/) has a database where you can search
different approaches to impact assessment. • NPC has a page on shared measurement where it outlines some indicators for different
sectors. http://www.philanthropycapital.org/publications/improving_the_sector/measuring_together.aspx
![Page 22: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/22.jpg)
22
RESOURCES
• Survey question bank: A website where you can research widely used surveys and single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/
• CES’s guide to theory of change: http://www.ces-vol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan-800-808.pdf
• Forum for Youth Investment’s guide to measuring soft skills in youth programmes: http://www.forumfyi.org/files/Soft_Skills_Hard_Data.pdf
• A list of the most common psychological scales and questionnaires: http://www.ull.ac.uk/subjects/psychology/psycscales.shtml
• Harvard Family Research Project’s tools for out-of-school time programmes: http://www.hfrp.org/out-of-school-time/publications-resources/measurement-tools-for-evaluating-out-of-school-time-programs-an-evaluation-resource2
• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco.
• Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science Students and Researchers. Sage Publications Ltd
• Measurement and assessment tools for family and parenting interventions: http://www.actionforchildren.org.uk/media/3986553/outcome_tools_1.pdf
![Page 23: X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASURING IMPACT: USING QUANTITATIVE RESEARCH Eibhlín Ní Ógáin London, November 2012 NPC - title.](https://reader034.fdocuments.net/reader034/viewer/2022050908/56649e365503460f94b258ec/html5/thumbnails/23.jpg)
vEibhlín Ní Ógá[email protected]