vMEASUREMENT SYSTEM FOR CHAPLAINCY SERVICESJames Noble

London, May 2015

2

CONTENTS

1Context 01

2Recap on theory of change (principles)

02

3Review of the chaplaincy theory of change

03

4Measurement principles 04

5Developing a measurement framework for chaplaincy projects

05

6Iizuka! 06

7Questions / discussion 07

3

WHY IS MEASURING IMPACT IMPORTANT?

Saves staff time

Influences the debate on

“what works”Improved services

Raises profile Motivates staff /

volunteers

Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater Impact. New Philanthropy Capital.

Helps secure funding

4

WHAT ARE THE CHALLENGES OF IMPACT MEASUREMENT?

Lack of skills

CostComplexity

Risk of weak results

Lack of confidence in value of soft outcomes

Risk of negative impact

on service delivery

Lack of technology

Wary of losing sight of ethos

CONTEXT

NPC: TRANSFORMING THE CHARITY SECTOR

6

NPC works at the nexus between charities and

funders

Charity

SectorFunder

Increasing the impact of charities

eg, impact-focused theories of change

Strengthening the partnership

Eg, collaboration towards shared

goals

Increasing the impact of funders

eg, effective commissioning

ConsultancyThink tank

7

“IMPROVING YOUR EVIDENCE”

Develop the evaluation capability and expertise of providers who offer services and

interventions to offenders.

http://www.clinks.org/support/evaluation-and-effectiveness

@imp_evidence

RESEARCH WITH TIER 1S

RECAP ON THEORY OF CHANGE

10

COMPONENTS OF NPC’S APPROACH TO MEASURING SOCIAL IMPACT

Map your theory of change

Prioritise what you measure

Choose your level of

evidence

Select your sources and

tools

Effective measurement framework developed

Strategic visionLeadership

Case for impact measurement

a) articulate how you think a service or intervention will have an impact;

b) and test your thinking against the

evidence you have available.

11

The essence of impact measurement:

Teaching assistants (TAs) trained to provide 1-1 support

Process for identifying children & families who will benefit

Pupils eat the breakfasts

Workplans better tailored to individual pupils needs

Engagement and enjoyment in maths lessons generally

More confident in Maths

Maths results amongst targeted pupils show improvement

Minimum of one hour of 1-1 tuition with TA per week

Inputs

Activities

Intermediate outcomes

Final outcomes

Description of resources deployed and targeting process

- Attendance records- Feedback from pupils,

teachers & parents

- Pupil behaviour measures (hard and soft)- Teachers perceptions- Mock test scores

Improved Key Stage test scores

LOGIC MODEL FOR A SCHOOL BREAKFAST CLUB

- Description of activities- Log of contact time- Example work plans

Healthy breakfasts available

Focus on activities/games to enthuse pupils

Pupils engage & enjoy a) tailored learning; b) group activities

Outputs

Targeted pupils attend and continue to attend

Behaviour improves

Accountability line

13

VALUE OF THE THEORY OF CHANGE APPROACH

• Give an account of impact. By this we mean providing a logical reason, or set of reasons, for why your products/service could have impact on outcomes.

NESTA: Standards for Impact Investing

• There should be a clear description of the service, including (i) the outcomes it aims to deliver, and (ii) who it is for

NOMS Commissioning Intentions

• Organisations applying for Level 1 need to demonstrate emerging evidence, clearly explaining the logic or the theory behind their work.

Project Oracle: What are the Standards of Evidence?

Which, if any, of the following techniques have you used in the last year?

Developed a theory of change

17% 49%YesNo - but would have liked to

14

NESTA / SRU / PROJECT ORACLE STANDARDS OF EVIDENCE

• Level 1: You can describe what you do and why it matters logically, coherently, and convincingly

• Level 2: You capture data that shows positive change but you cannot confirm you caused this

• Level 3: You can demonstrate causality using a control or comparison group

• Level 4: You have one + independent replication evaluations that confirm these conclusions

• Level 5: You have manuals, systems and procedures to ensure consistent replication and positive impact (fidelity)

http://www.nesta.org.uk/publications/standards-evidence-impact-investing

REVIEW OF THE CHAPLAINCY THEORY OF CHANGE

http://www.clinks.org/support-evaluation-and-effectiveness/demonstrating-outcomes

MEASUREMENT PRINCIPLES

http://www.clinks.org/support/evaluation-and-effectiveness

Specific Aim(middle of triangle)

Outcome(s) Outcome indicator(s)

Data collection method(s)

When and by whom

How and for whom will the data be used

RUN THE PROGRAMME AND COLLECT THE ASSOCIATED EVIDENCE

17http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Monitoring%20and%20evaluation%20framework.pdf

18

IMPACT MEASUREMENT TIPS

1. Use theory of change or logic model to articulate how your project or programme works

2. Make sure it’s informed by existing evidence/research

3. Focus on the immediate, short-term effects you have

4. Collect good quality qualitative evidence. Don’t cherry-pick

5. What administrative data is available? Only collect new quantitative data if you have to.

6. Try to measure the progress of service users longitudinally. Consider sampling rather than measuring everyone.

7. Consider your opportunities to measure the counterfactual

8. Publish and promote what you have learned

http://www.clinks.org/sites/default/files/Introduction%20to%20evaluation_final_0.pdf

INTRODUCTION TO THE CURRENT PROJECT

20

IMPACT READINESS FUND

CONTENT

FORMAT

23

REQUIRED FEATURES

• Monitor changes in outcomes between sessions: Internal and external

• Intuitive and easy to use

• Visual and numeric outputs. Exportable to Excel.

• Different views and levels of access

• High-level access to sign-off entries from low level users

• Personal data needs to be stored securely

• Flexible, adaptable

• Possible to quickly register an individual and add detailed information later

• Software should be usable on tablet and desktop platforms

• Unlimited users

• Link mentors, mentees and area administrators

• Off-line data entry

24

25

26

THOUGHTS?

vJames.noble@thinknpc.org