An Introduction to Monitoring & Evaluation

Proving Your Impact: Pain-free Monitoring and Evaluation

10th November 2014Robin Beveridge and Tony Sacco

• Domestics• Who’s who• What’s in store today• Ground rules• Networking and learning from each other

Introduction

Structure of the day: AM9.30 Introductions and warm up

9.45 Monitoring & Evaluation: Key Concepts

10.00 Practical session: developing project ‘logic chains’

Break

11.00 Capturing information from beneficiaries – what works?

11.30 ‘Telling the story’ using case studies

Lunch

Structure of the day: PM

13.10

13.50

14.10

14.50

Workshop feedback and close

Quantitative Evaluation techniques

Evaluation in practice

Monitoring & Evaluation plans

Wowing the World

13.00 Recap and Review

15:30

Getting started

• Over the next 5 minutes…Write down 3 things you hope to get out of today on ‘post-its’Speak with someone you have never met before todayIntroduce yourselves and swap one of your ‘hopes’Repeat this with two more people

• Then… …post your ‘post-its’ up on the wall.

Setting the ground rules…

Key Concepts

• Monitoring• Evaluation• Outputs• Outcomes• Impact• Aims • Objectives• Quantitative

• Qualitative• Beneficiaries• Stakeholders• Value for Money• Additionality• Social Return

Logic chain example 1: This workshop

Caused by

Output

Need Activity

Impact Outcome

Limited ability to demonstrate

impactWorkshops and peer learning

People trainedHigher level of skills and

knowledge

Better demonstration of

impact

Lack of Monitoring and

evaluation knowledge

Rationale

Logic chain example 2: Streetwise

Caused by

Output

Need Activity

Impact Outcome

High level of teenage

pregnancyProviding

C-cards

Young people accessing condoms

Less un-protected sex

Lower level of teenage

pregnancy

Poor access to contraceptives

Logic chain example 3: Bright Futures

Caused by

Output

Need Activity

Impact Outcome

Young women in South Tyneside

facing health and social problems

Courses, advice sessions, peer

education, volunteering

Young women engaged in

programmes/courses or given

advice

Young women better equipped to succeed in life

Reduced health and social

problems for young women in South Tyneside

Lack of life skills and support

systems

Logic chain example 4: Action Foundation

Caused by

Output

Need Activity

Impact Outcome

Vulnerable asylum –seekers

without accommodation

Provision of free emergency

accommodation

Asylum-seekers housed and supported

Asylum-seekers have stability and ability to access

services for basic needs

Chronic cycle of poverty / poor health / illegal

work prevented

No recourse to public funds and no permission to

work

Working in pairs

• Using templates provided, develop a ‘Logic Chain’ for a project you are involved with

• Share that with someone on your table• Feed back to the workshop:– What was easy, what was hard?– Does it make sense?

Capturing information from beneficiaries – What works?

• In groups consider…–What are the different ways of capturing

information from participants / beneficiaries?– Identify some ‘pros and cons’ of each

approach.

Capturing information from beneficiaries – Some thoughts

Questionnaires • Wide coverage• Tick box / narrative approach?• Accessibility• Self-completed or ‘managed’• ‘Survey Monkey’• Before / after / distance travelled• Confidentiality• Response rate

Capturing information from beneficiaries – Some thoughts

Capturing information from beneficiaries – some thoughts

Feedback forms• Immediate feedback from ‘captive

audience’?• More of / less of• Most useful / least useful• Improvements• Can be superficial and ‘too immediate’• Follow up again later?


In-depth interview• Richer content• Particular insights• Range of views / perspectives• Resource intensive


Focus Groups• A group with similar characteristics (e.g.

single parents)• ‘Managing’ the group• Recording the outcomes

Round Table• Different perspectives


In-depth interview• Richer content• Particular insights• Different perspectives• Resource intensive


Comments / concerns / compliments • Forms / Box?• Managing the process• The feedback loop

Timing issues…

• Captive audience• Getting buy-in from the outset• ‘When they’re gone they’re gone!’• Bias?• Incentives?

Storage issues…

• Paper based• Spreadsheets• Databases• Bespoke approaches?

What more creative ways can you come up with to gather information from beneficiaries?

Outside the box…

Telling the StoryUsing Case Studies

Spend a couple of minutes discussing with one or two others…

When and why might we use a case study?

Telling the StoryUsing Case Studies

• The power of the story• The power of the actual words…• … and a picture paints 1,000 of them

The Power of the Story

The Power of the Actual Words• ‘If I didn’t have this I would

have reoffended. I would have ended up getting drunk, fighting with someone and be back in prison.’

• ‘I felt miserable, I had no money and I was on benefits but now I’m independent. I feel good about working for my money.’

…and a picture…

Case Studies: ESF Approach

About the person: • Details of the client’s background, education/training

and employment history and current situation.Training/support: • Details of what activity took place during the project• What the client feels s/he achieved in regard to soft

skills (e.g. more confidence) or hard outcomes (e.g. a qualification, employment, or volunteering).

Case Studies: ESF Approach

Plans for the future• How does the client hope to progress their

employability, training or job searching? • Have they been referred to another organisation for

ongoing support – if so which and what type ?Quote• A short quote from the client on how the project has

made a difference. Signature

Case Studies: A Quick Checklist

• Before you start• Setting up the interview• On the day• After the event• Other thoughts

Brandon’s Story

Structure of the day: PM

13.10

13.50

14.10

14.50

Workshop feedback and close

Quantitative Evaluation techniques

Evaluation in practice

Monitoring & Evaluation plans

Wowing the World

13.00 Recap and Review

Quantitative techniques

• Why numbers matter• What numbers do we

need?• How to get useful

numbers• Cross-tabulation• Sampling and

extrapolating• How to present data

Quantitative techniques

Ar-rgh!!

Zzzzz OK, if I

have to

Woo hoo, data

0

10

20

30

40

50

Reaction to the word “Spreadsheet”

MenWomen

%

n.b. fictional data!

Key numbersThe project has achieved significant positive outcomes, with 447 young people supported to become enterprise ready up to December 2011. To date, roughly 1 in 4 of these beneficiaries have gone on to establish a business, and around 30% have moved into education, training or employment.”

“An independent evaluation found that the service costed £145,000 over

three years, and had resulted in significant cost savings to statutory

services: conservatively estimated at £1,211,630.”

• How many beneficiaries?

• Of what sort?

• What happened as a result?

• What did it cost?

• How much did it save?

• Was it good value for money?

Getting the numbers

• Excel– Pivot Tables– Filters

• Survey Monkey• Database Systems

% of beneficiaries Under 25 26-50 Over 50 Total

Northumberland 20

Tyne & Wear 40

County Durham 25

Tees Valley 15

TOTAL 25 50 25 100

% of beneficiaries Under 25 26-50 Over 50 Total

Northumberland 0 5 15 20

Tyne & Wear 10 20 10 40

County Durham 15 10 0 25

Tees Valley 0 15 0 15

TOTAL 25 50 25 100

Cross tabulation

Sampling and extrapolation

• Most evaluation is based on a sample• How representative is your sample?• How diverse is your ‘population’?• Evaluation is NOT a science• Be aware of potential ‘bias’ in the sample• Use caveats in extrapolating

Sampling example

1 2 3 4-6 7-12 12+0

20

40

60

80

100

120

Length of time to find work

Months from completion

We found that 100% of respondents had found work within 3 months of completing the course. If this is true for all beneficiaries, we will have helped 300 people to find work”

Presenting Data – some tips

• Not too much information• Order bars from high to low• Use a variety of formats, but not too many• Keep a consistent colour scheme• Remember titles, axes and legends• Keep it as clean as possible

http://office.microsoft.com/en-gb/excel-help/available-chart-types-HA001233737.aspx

For example

Presenting Data – some tips

Use free tools, eg:• Google Fusion

Tables• Tableau public• Any others?

That’s quite enough about data

Developing an Evaluation Plan:Some things to think about

• Scope and Purpose– What?– Level – e.g. Project, Programme, Organisation– Formative / Summative– Audience?


• Timing and Phasing– When?– Single phase or multiphase?• How do they fit together?

– Previous evaluations?• The same approach or different?

– Key deadlines?• ‘Doomed to success’


• Collecting Data– What is required?• Back to the logic chain

– Where can you get the data– What have you got already?– Storage and analysis– Any barriers to using data?


• Who does it?– Internal / External?– Buy-in?– Objectivity– Budget


• Collecting views– Options for beneficiaries?– Key stakeholders?– Options for Stakeholders


• Managing the evaluation process– Who is responsible?– Sign off?

• Dissemination– Who and how?

Developing an Evaluation Planfor your project

Wowing the World

• Why are we doing this again?– Because we have to?– To improve our services to clients? – To attract new clients?– To demonstrate to ourselves and / or others that

what we’re doing is effective?– To show Value for Money?– To convince people to give us some more?

Wowing the World

• Whatever the reason we need to ‘broadcast it’ in some way

• What approaches have you used that have been particularly effective in broadcasting what you’ve achieved and the impact that it has made?

Wowing the World

The word on the street

• Evaluation Reports

• Summary Documents– Selected highlights

• Newspapers / Newsletters

• Annual Reports

The word on the street

The writing on the wall…

Multi Media

http://www.actionfoundation.org.uk/about_history.html

Summary and questions

Your Evaluation!

• Any immediate thoughts?

• Evaluation sheet – Free prize draw!

• A more detailed conversation?

• Peer support network?

An Introduction to Monitoring & Evaluation

Government & Nonprofit

Transcript of An Introduction to Monitoring & Evaluation