v

WORKING TOGETHER TO MEASURE IMPACTANNE KAZIMIRSKI, NPC

SIAA conference, Cergy-Pontoise, 10th December 2013

TRANSFORMING THE UK CHARITY SECTOR

2

NPC works at the nexus between charities and

funders

Charity

SectorFunder

Increasing the impact of charities

eg, impact-focused theories of change

Strengthening the partnership

Eg, collaboration towards shared

goals

Increasing the impact of funders

eg, effective commissioning

ConsultancyThink tank

3

• Charities and social enterprises under pressure to demonstrate impact

• New funding mechanisms require robust and comparable measurement

• But:

• Impact information is often of low quality, not embedded, not easily comparable, and rarely standardised

• Charities are spending resources developing custom measurement frameworks and tools

• Shared measurement is a potential solution

BACKGROUND

4

WHAT IS SHARED MEASUREMENT?

Shared measurement is using common tools to track outcomes across similar organisations and settings:

• a process: understanding a sector’s shared outcomes

• a product: any tool used by more than one organisation to measure impact.

5

WHO HAS TRIED TO IMPLEMENT SHARED MEASUREMENT?

Reviews 13 attempts at implementing shared measurement

6

HOW DO YOU SUCCESSFULLY IMPLEMENT SHARED MEASUREMENT WITHIN A SECTOR?

7

IS SHARED MEASUREMENT A GOOD IDEA?

Disadvantages

• Takes time to agree on shared values and goals

• Doesn’t reflect nuances of different organisations

• May not meet funder / commissioner requirements

• Risk comparing apples: oranges

• Reputational risk of comparability

• Data protection prevents sharing

Advantages

• Saves time and resources

• Reduces duplication in reporting

• Improves standards of impact measurement

• Externally validates impact data

• Promotes system thinking

• Allows comparing what works best for whom

8

• Funded by the UK government to develop a toolkit that will measure the outcomes of rehabilitation services

• Partnership with an academic institute and several charities

• Toolkit is for prison and post-prison programmes focusing on offenders’ relationships with their family and with their peers

OFFENDERS’ FAMILY RELATIONSHIPS

9

10

PROJECT PROCESS

Evidence review – identifying outcomes and toolkits

Consultation with providers & commissioners to select outcomes to focus on

Development and piloting of toolkit

Consultation with providers & commissioners to inform guidance

Analysis and reporting

12

DEVELOPMENT OF RELATIONSHIPS TOOLKIT

• Desistance theory:

• Link between creation/ investment in family relationships / reduction in peer pressure with reduced offending

• Consultation:

• Outcomes differed depending on the type of intervention & beneficiary

• Needed to reach a balance between being prescriptive and allowing for flexibility

• We researched existing scales and evaluations in each outcome area and prioritised:

• good validity and reliability; recently developed; brevity.

Toolkit: 25 standardised scales. For each outcome area: short general scales & longer specialised scales.

13

RELATIONSHIPS TOOLKIT STRUCTURE

We have identified 5 areas of outcomes shown below, supplemented by a personal development module developed by a parallel project:

Partner relationships

Child relationshipsWider family relationships

ParentingPeer

relationships

Personal development

14

OUTCOME AREAS

Partner relationships

Increased satisfaction with partner relationship, reduced conflict and improved commitment.

Child relationships

Quality of relationships with children, children’s well-being and behaviour.

Wider family relationships

Family functioning, resilience, conflict and communication, satisfaction with relationships in the family and quality of relationships

Parenting

Improved satisfaction with the parent-child relationship and parenting skills

Peer relationships

Satisfaction with peer relationships and reduction of negative peer relationships.

15

CHARACTERISTICS OF TOOLKIT

• The questions can be used as monitoring or as part of an evaluation with a ‘before and after’ (pre/post) research design, with a comparison group where possible

Before measure

After measure

Programme / service

Before measure

After measure

“Intervention” group

“Comparison” group

16

PILOTING DESIGN

Agreeing outcome areas and scales to include

Agreeing demographics/ risk factor questions

Tailoring to client population and service

Agreeing timing of post-questionnaires

Materials include:• Tailored pre and post questionnaires• Data collection sheets• Staff and client information sheets• Consent procedure

The timing of the post-questionnaires varies by provider, from one month to two.

Initial pilot feedback:• Suitability of peer relationship questions for older offenders• Capturing the end of negative partner relationships for women offenders

17

DISCUSSION

Working together across your sector to develop a shared measurement approach:

• What are/ would be the main challenges?

• What are the solutions to these challenges?