An Introduction to Monitoring & Evaluation
-
Upload
robin-beveridge -
Category
Government & Nonprofit
-
view
102 -
download
3
Transcript of An Introduction to Monitoring & Evaluation
Proving Your Impact: Pain-free Monitoring and Evaluation
10th November 2014Robin Beveridge and Tony Sacco
• Domestics• Who’s who• What’s in store today• Ground rules• Networking and learning from each other
Introduction
Structure of the day: AM9.30 Introductions and warm up
9.45 Monitoring & Evaluation: Key Concepts
10.00 Practical session: developing project ‘logic chains’
Break
11.00 Capturing information from beneficiaries – what works?
11.30 ‘Telling the story’ using case studies
Lunch
Structure of the day: PM
13.10
13.50
14.10
14.50
Workshop feedback and close
Quantitative Evaluation techniques
Evaluation in practice
Monitoring & Evaluation plans
Wowing the World
13.00 Recap and Review
15:30
Getting started
• Over the next 5 minutes…Write down 3 things you hope to get out of today on ‘post-its’Speak with someone you have never met before todayIntroduce yourselves and swap one of your ‘hopes’Repeat this with two more people
• Then… …post your ‘post-its’ up on the wall.
Setting the ground rules…
Setting the ground rules…
Key Concepts
• Monitoring• Evaluation• Outputs• Outcomes• Impact• Aims • Objectives• Quantitative
• Qualitative• Beneficiaries• Stakeholders• Value for Money• Additionality• Social Return
Logic chain example 1: This workshop
Caused by
Output
Need Activity
Impact Outcome
Limited ability to demonstrate
impactWorkshops and peer learning
People trainedHigher level of skills and
knowledge
Better demonstration of
impact
Lack of Monitoring and
evaluation knowledge
Rationale
Logic chain example 2: Streetwise
Caused by
Output
Need Activity
Impact Outcome
High level of teenage
pregnancyProviding
C-cards
Young people accessing condoms
Less un-protected sex
Lower level of teenage
pregnancy
Poor access to contraceptives
Logic chain example 3: Bright Futures
Caused by
Output
Need Activity
Impact Outcome
Young women in South Tyneside
facing health and social problems
Courses, advice sessions, peer
education, volunteering
Young women engaged in
programmes/courses or given
advice
Young women better equipped to succeed in life
Reduced health and social
problems for young women in South Tyneside
Lack of life skills and support
systems
Logic chain example 4: Action Foundation
Caused by
Output
Need Activity
Impact Outcome
Vulnerable asylum –seekers
without accommodation
Provision of free emergency
accommodation
Asylum-seekers housed and supported
Asylum-seekers have stability and ability to access
services for basic needs
Chronic cycle of poverty / poor health / illegal
work prevented
No recourse to public funds and no permission to
work
Working in pairs
• Using templates provided, develop a ‘Logic Chain’ for a project you are involved with
• Share that with someone on your table• Feed back to the workshop:– What was easy, what was hard?– Does it make sense?
Break
Capturing information from beneficiaries – What works?
• In groups consider…–What are the different ways of capturing
information from participants / beneficiaries?– Identify some ‘pros and cons’ of each
approach.
Capturing information from beneficiaries – Some thoughts
Questionnaires • Wide coverage• Tick box / narrative approach?• Accessibility• Self-completed or ‘managed’• ‘Survey Monkey’• Before / after / distance travelled• Confidentiality• Response rate
Capturing information from beneficiaries – Some thoughts
Capturing information from beneficiaries – some thoughts
Feedback forms• Immediate feedback from ‘captive
audience’?• More of / less of• Most useful / least useful• Improvements• Can be superficial and ‘too immediate’• Follow up again later?
Capturing information from beneficiaries – some thoughts
In-depth interview• Richer content• Particular insights• Range of views / perspectives• Resource intensive
Capturing information from beneficiaries – some thoughts
Focus Groups• A group with similar characteristics (e.g.
single parents)• ‘Managing’ the group• Recording the outcomes
Round Table• Different perspectives
Capturing information from beneficiaries – some thoughts
In-depth interview• Richer content• Particular insights• Different perspectives• Resource intensive
Capturing information from beneficiaries – some thoughts
Comments / concerns / compliments • Forms / Box?• Managing the process• The feedback loop
Timing issues…
• Captive audience• Getting buy-in from the outset• ‘When they’re gone they’re gone!’• Bias?• Incentives?
Storage issues…
• Paper based• Spreadsheets• Databases• Bespoke approaches?
What more creative ways can you come up with to gather information from beneficiaries?
Outside the box…
Telling the StoryUsing Case Studies
Spend a couple of minutes discussing with one or two others…
When and why might we use a case study?
Telling the StoryUsing Case Studies
• The power of the story• The power of the actual words…• … and a picture paints 1,000 of them
The Power of the Story
The Power of the Actual Words• ‘If I didn’t have this I would
have reoffended. I would have ended up getting drunk, fighting with someone and be back in prison.’
• ‘I felt miserable, I had no money and I was on benefits but now I’m independent. I feel good about working for my money.’
…and a picture…
Case Studies: ESF Approach
About the person: • Details of the client’s background, education/training
and employment history and current situation.Training/support: • Details of what activity took place during the project• What the client feels s/he achieved in regard to soft
skills (e.g. more confidence) or hard outcomes (e.g. a qualification, employment, or volunteering).
Case Studies: ESF Approach
Plans for the future• How does the client hope to progress their
employability, training or job searching? • Have they been referred to another organisation for
ongoing support – if so which and what type ?Quote• A short quote from the client on how the project has
made a difference. Signature
Case Studies: A Quick Checklist
• Before you start• Setting up the interview• On the day• After the event• Other thoughts
Brandon’s Story
Lunch
Recap
Structure of the day: PM
13.10
13.50
14.10
14.50
Workshop feedback and close
Quantitative Evaluation techniques
Evaluation in practice
Monitoring & Evaluation plans
Wowing the World
13.00 Recap and Review
Quantitative techniques
• Why numbers matter• What numbers do we
need?• How to get useful
numbers• Cross-tabulation• Sampling and
extrapolating• How to present data
Quantitative techniques
Ar-rgh!!
Zzzzz OK, if I
have to
Woo hoo, data
0
10
20
30
40
50
Reaction to the word “Spreadsheet”
MenWomen
%
n.b. fictional data!
Key numbersThe project has achieved significant positive outcomes, with 447 young people supported to become enterprise ready up to December 2011. To date, roughly 1 in 4 of these beneficiaries have gone on to establish a business, and around 30% have moved into education, training or employment.”
“An independent evaluation found that the service costed £145,000 over
three years, and had resulted in significant cost savings to statutory
services: conservatively estimated at £1,211,630.”
• How many beneficiaries?
• Of what sort?
• What happened as a result?
• What did it cost?
• How much did it save?
• Was it good value for money?
Getting the numbers
• Excel– Pivot Tables– Filters
• Survey Monkey• Database Systems
% of beneficiaries Under 25 26-50 Over 50 Total
Northumberland 20
Tyne & Wear 40
County Durham 25
Tees Valley 15
TOTAL 25 50 25 100
% of beneficiaries Under 25 26-50 Over 50 Total
Northumberland 0 5 15 20
Tyne & Wear 10 20 10 40
County Durham 15 10 0 25
Tees Valley 0 15 0 15
TOTAL 25 50 25 100
Cross tabulation
Sampling and extrapolation
• Most evaluation is based on a sample• How representative is your sample?• How diverse is your ‘population’?• Evaluation is NOT a science• Be aware of potential ‘bias’ in the sample• Use caveats in extrapolating
Sampling example
1 2 3 4-6 7-12 12+0
20
40
60
80
100
120
Length of time to find work
Months from completion
We found that 100% of respondents had found work within 3 months of completing the course. If this is true for all beneficiaries, we will have helped 300 people to find work”
Presenting Data – some tips
• Not too much information• Order bars from high to low• Use a variety of formats, but not too many• Keep a consistent colour scheme• Remember titles, axes and legends• Keep it as clean as possible
For example
Presenting Data – some tips
Use free tools, eg:• Google Fusion
Tables• Tableau public• Any others?
That’s quite enough about data
Developing an Evaluation Plan:Some things to think about
• Scope and Purpose– What?– Level – e.g. Project, Programme, Organisation– Formative / Summative– Audience?
Developing an Evaluation Plan:Some things to think about
• Timing and Phasing– When?– Single phase or multiphase?• How do they fit together?
– Previous evaluations?• The same approach or different?
– Key deadlines?• ‘Doomed to success’
Developing an Evaluation Plan:Some things to think about
• Collecting Data– What is required?• Back to the logic chain
– Where can you get the data– What have you got already?– Storage and analysis– Any barriers to using data?
Developing an Evaluation Plan:Some things to think about
• Who does it?– Internal / External?– Buy-in?– Objectivity– Budget
Developing an Evaluation Plan:Some things to think about
• Collecting views– Options for beneficiaries?– Key stakeholders?– Options for Stakeholders
Developing an Evaluation Plan:Some things to think about
• Managing the evaluation process– Who is responsible?– Sign off?
• Dissemination– Who and how?
Developing an Evaluation Planfor your project
Wowing the World
• Why are we doing this again?– Because we have to?– To improve our services to clients? – To attract new clients?– To demonstrate to ourselves and / or others that
what we’re doing is effective?– To show Value for Money?– To convince people to give us some more?
Wowing the World
• Whatever the reason we need to ‘broadcast it’ in some way
• What approaches have you used that have been particularly effective in broadcasting what you’ve achieved and the impact that it has made?
Wowing the World
The word on the street
• Evaluation Reports
• Summary Documents– Selected highlights
• Newspapers / Newsletters
• Annual Reports
The word on the street
The word on the street
The writing on the wall…
Summary and questions
Your Evaluation!
• Any immediate thoughts?
• Evaluation sheet – Free prize draw!
• A more detailed conversation?
• Peer support network?