2012SFF - Made to measure

Made to Measure: Tools and Practices

• Introduction and Theory of change

• Tools overview

• Randomized control trial case

• Wrap up and Questions

Theory of change

Impact Reporting and Investment Standards

Global Impact Investing Ratings System

Social Return on Investment

Demonstrating Value

Randomized Control Trials

Measuring Social Value

Dr. Tessa HebbMeasuring Up, Social Finance ForumCentre for Impact Investing, MaRS, TorontoNovember 8th 2012

What is Blended Value“The Blended Value Proposition states is

that all organizations, whether for-profit or not, create value that consists of economic, social and environmental value components—and that investors (whether market-rate, charitable or some mix of the two) simultaneously generate all three forms of value through providing capital to organizations.” Jed Emerson

Metrics and Measurement

Blended value requires measurement of all

three forms of value economic, social and environmental.

“Measurement should be viewed as a process whereby the greatest value is achieved through organizations building up and learning from data and evidence over time.” (Measuring the Value of Corporate Philanthropy).”

What are social metrics?

Social metrics are measurement tools that can be used to define and articulate social value, social outcomes and the results generated by investment and activities in the social sector.

You manage what you measure “A review of measurement methodologies did not

turn up a “silver bullet” or single numeric against which performance can be universally gauged. Rather, this reading reinforced the notion that, to an extent, measurement is its own reward. It encourages improvement, management, and the explicit formulation of assumptions and expectations.” (Measuring the Value of Corporate Philanthropy)

Start with your mission

1. What are the results for which you will hold yourselves accountable?

2. How will you achieve them?3. What will they really cost?4. How do you build the organization you

need to deliver these results?

What to Measure (Measuring the Value of Corporate Philanthropy).

• Links among the mission, programs, and measures must be clearly defined and articulated in order to narrow the number of required indicators.

• The measures should be easily collectible and communicable.

• The measures should be strategically designed and applicable across the organization at all levels, while also encouraging of operating units to focus on high-level strategies.

• Above all, the measures must address progress toward the mission and illustrate whether and how the organization’s actions make a difference.

Inputs, Activities, Outcomes, and ImpactsInputs, activities, outputs, outcomes, and impacts over the short, medium and long term are the building blocks of social metrics.

The building blocks are linked to the organizations mission through its theory of change. “Activities,” such as the number of staff trained or amount of goods purchased, and “outputs,” such as the number of clients served, products distributed, and areas reached may be the extent of measurement for short-term, one-off projects. Sometimes simply identifying activities and measuring output may be all that is feasible.

Theory of Change• “Built around the pathway of

change, a Theory of Change describes the types of interventions (a single program or a comprehensive community initiative) that bring about the outcomes depicted in the pathway of a change map. Each outcome in the pathway of change is tied to an intervention, revealing the often complex web of activity that is required to bring about change.” (TheoryofChange.org)

Logic Model

Input/Resources What are the resources that are needed to accomplish the activities?

Activities What are the activities/products that will generate the outputs?

Outputs What is the evidence of service delivered to the intended audience at the intended dose? (what others will be able to see, touch, count)

Outcomes What is the change that will happen (short term or medium term) to the target group or individuals?

Impact What will happen over the long term

Indicator How would you measure the outcome? How will you know if change happened?

Stakeholder 1

Conclusion

•Blended value requires economic, social and environmental values be measured.

•Social metrics help organizations understand their strengths.

•Social metrics link mission, activities, outputs, outcomes and impacts.

•No one size fits all.

Dr. Tessa [email protected]

Director, Carleton Centre for Community Innovation

www.carleton.ca/3ci

Contact

mailto:[email protected]

http://www.carleton.ca/3ci

http://www.amazon.ca/Next-Generation-Responsible-Investing/dp/9400723474/ref=sr_1_1?ie=UTF8&qid=1321909368&sr=8-1

Impact Map

Theory of change

Impact Reporting and Investment Standards

Global Impact Investing Ratings System

Social Return on Investment

Demonstrating Value


Inputs Activities Outputs Outcomes Impact

SROI

IRIS

GIIRS

Demonstrating Value


Pg 18

The Impact Reporting and Investment Standards (IRIS) is a catalogue of

metrics that can be used to describe an organization’s social, environmental,

and financial performance.

Full Time Employees = Full Time Employees GHG Emissions Reduced = GHG Emissions Reduced

My Metrics

Your Metrics

IRIS is…

20

IRIS Framework

http://iris.thegiin.org

21

IRIS Framework

Inputs

Activities

Output

s

Outco

mes

Impact

SROI

IRIS

GIIRS

Demonstrating Value


Global Impact Investing Rating System

Pg 23

377 B Corporations I $1.82B Revenues I 54 Industries I

$1M Annual Savings

Environment

Workers

Governance

Community

25

B Impact Assessment

Inputs

Activities

Output

s

Outco

mes

Impact

SROI

IRIS

GIIRS

Demonstrating Value


What is SROI?

Social Return on Investment is a framework for measuring and accounting for the value created or destroyed by our activities – where the concept of value is much broader than that which can be captured by market prices.

val·ue/ˈvalyo) o/noun

the regard that something is held to deserve; the importance, worth, or usefulness of something

the material or monetary worth of something

29

Return on Investment

(Net Profit) Cost of Investment

ROI =

SROI example: Calgary Youth Justice

7 Principles of SROI

• Involve stakeholders

• Understand what changes

• Value the things that matter

• Only include what is material

• Do not overclaim

• Be transparent

• Verify the result

Inputs

Activities

Output

s

Outco

mes

Impact

SROI

IRIS

GIIRS

Demonstrating Value


Clear picture of informatio

n needs

Capacity to collect informati

on

Capacity to use

information

Information Blueprint

Support for Monitoring

Systems Developme

nt

‘Snapshot’ report

Capacity building

Performance Snapshot example

www.demonstratingvalue.orgThe workbook, snapshot gallery,

blog, newsletter and more

Twitter @demvalue


Library/Catalogue of indicators

Rating/Benchmarking system

FrameworkSet of principles

Framework Toolbox

Study design/methodology

The use of RCTs for Social Impact Assessment: An ExamplePresentation to the 2012 Social Finance Forum, November 8 and 9, 2012

Challenges for Social Impact Assessment

WHAT outcomes to measure – that will best align with your program goal

WHICH methods to use – that will best isolate program effects and allow the attribution of these efects to your program

HOW to quantify and monetize these effects – to demonstrate financial viability

39

Demonstrating Impact

The capacity to measure the difference a new approach or program makes.

To measure the true impact of a new program, we need to know what would have happened if the program had not been introduced.........we need a counterfactual:• to account for natural maturation processes, e.g., children skills are

constantly improving• to account for factors external to the program, e.g., state of the

economy fluctuates and influences labour market outcomes• to account for “regression to the mean” phenomenon, e.g., lone

mothers on welfare eventually go back to work

Displaced workers re-integrating labour market as a result of being offered an earnings supplement

0%

10%

20%

30%

40%

50%

60%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Full-

Tim

e Em

ploy

men

t Ra

te

Month From Random Assignment

Supplement GroupProgram Group

Displaced workers being offered supplement versus those not receiving the same offer

0%

10%

20%

30%

40%

50%

60%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Full-

Tim

e Em

ploy

men

t Ra

te

Month From Random Assignment

Supplement Group

Control Group

Program Group

Control Group

Why Randomized Experiments?

To measure the true impact of a new program, we need to know what would have happened if the program had not been introduced.........we need a counterfactual:

Random assignment of participants to a treatment group and a control group provides the best method to create a counterfactual • Not the only method available• Not always possible to do RA

How Random Assignment WorksRecruitment of participants

A random sample of individuals from the population that is targeted for the program intervention is recruited and interviewed

Informed Consent

Potential participants sign an informed consent agreeing to be part of the experiment and provide information for research purposes

Random Assignment

Volunteers are assigned at random to either the program group or the control group

Program Group

Eligible to receive program intervention

Informed of eligibility for the program intervention and the conditions

attached to recipiency

Meet conditions for recipiency

Receive program intervention

Do not meet conditions for recipiency, therefore, do not receive program intervention

Control Group

Ineligible

Informed of ineligibility status

Not eligible for program intervention, but continue to be part of the study

for research purposes

Note: Both program and control group members continue to have access to government programs and services available to members of their community.

With a large enough sample, random assignment insures that the two groups of individuals are identical, on average. • This is true for all observable and unobservable characteristics

(such as motivation, self-confidence, determination, and all other personal attributes that can explain why an intervention will succeed with one individual but not another)

Unlike nonexperimental methods, properly implement social experiments are guaranteed to provide internally valid impact estimates – no selection bias.

Nonexperimental methods may be equally reliable in any given application; we simply cannot know a priori that they are reliable, as we can with experimental methods – you can only match participants on measured characteristics.

Properties of RCTs / Social Experiments

A research and demonstration project testing the value of community-based employment as an alternative to income transfers in areas of chronic high unemployment

Two parallel but related studies

• Individuals: • Aims to preserve employability, through faster re-employment• Provides opportunities for skill development and strengthening of social capital

• Communities: • Study of a model which utilizes strengths of local communities to create jobs• Aims to support their capacity growth and improve the social economy

The Community Employment Innovation Project

The Offer to Individuals

• EI and IA recipients were offered 3 years of full-time employment, on locally developed projects in exchange for their entitlements to EI or IA

• Employment was designed to replicate full-time market jobs• 35 hours per week, at $325 a week, EI/CPP insurable, 15 days annual leave,

medical benefits • Support Services: some job-readiness and transferable skills training

The Offer to Communities

• 6 communities in the Cape Breton Regional Municipality were offered a free workforce of 750 workers for up to five years

• Each community was required to elect a representative board, develop a strategic plan, and approve projects

• Local control given to communities – explicitly links projects to local needs

The Program Model

Key outcomes of interest

• Economic well-being• Employment, earnings, transfer receipt, income, poverty, and

hardship• Longer-term employability: Skills and experience

• Characteristics of post-CEIP employment, employability skills, attitudes to work

• Social Capital, volunteering, life satisfaction

Participants Impact Study

-20-15-10

-505

101520253035404550556065707580859095

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53

Months From Random Assignment

Program GroupControl GroupImpacts

Perc

enta

ge E

mpl

oyed

Ful

l Tim

e

A 53 percentage point impact at peak

No significant impacts a year after end of CEIP eligibility

CEIP impacts on EI sample

-20-15-10

-505

101520253035404550556065707580859095

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53


Program GroupControl GroupImpacts

Perc

enta

ge Em

ploy

ed F

ull T

ime

Nearly 80 percentage point impact at peak

No significant impacts a year after end of CEIP eligibility

IA Sample: Large in-program impacts on employment, but not sustained

-60

-40

-20

0

20

40

60

80

100

1 11 21 31 41 51 61 71


Program GroupControl GroupImpact

Perc

enta

ge R

ecei

ving

IA

A stable 42 percentage point reduction in IA receipt during program

A sustained 12 percentage point reduction in IA receipt

IA Sample: Permanent reductions in IA receipt three years after CEIP comes to an end

Help with chores Emotional support

Specialized advice

Help finding a job

0.0

0.5

1.0

1.5

2.0

2.5

Network SizeChange in number of contacts

from baseline to 54 months

Program Control

54-m

onth

gai

n

Impacts on Social Capital:Sustained increase in the size of networks

Volunteering is important to both individuals and communities Can be an avenue to skill development, improves social inclusion,

and is a large resource for many community organizations

05

101520253035404550

Volunteered in the past 12 months

Volunteered in the past 12 months

EI Sample IA Sample

Perc

ent

54-Month: Impacts on Formal volunteering with groups or organizations

Program

Control

Community Engagement:Sustained increase in volunteering and social contact

Prod

uct e

ffect

s Product effectsCommunity Response

Organizing, Planning, MobilizationOrganizational

Mobilization

CEIP ProjectsProvide work experience and

valued community Services

Individual Engagement

Well Being and Community Capacity

Improves

Social Inclusion, Cohesion

Employment Levels, Market Conditions

Improves

Skill Gains, Social Capital

Improves

Process effects

Process effects

Early Mobilization

Years 1-3

Project Development and Interim

Effects

Years 4-5

Post-program Long-run Effects

Years 6-7

CEIP Theory of Change:Analytical framework of expected change

The two largest project categories were similar across communities Youth projects were most prevalent in New Waterford and Whitney Pier Seniors projects were at greatest scale in Sydney Mines

Figure ES.5: Percentage of Work Years Assigned, by Community and Sector Served

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Glace Bay NewWaterford North Sydney Sydney Mines Whitney Pier

Community

Per

cen

tag

e o

f to

tal

wo

rk y

ears

ass

ign

ed t

o a

sec

tor

Health, Environment,Beautification

Recreation, Arts andCulture

Services for the Poor,Unemployed

Supports for Seniors

Supports for Youth

Other: Services forthe Disabled, CEDAgencies, CEIPBoards

Targeted Community Sectors

Social Capital

Social Cohesion

Social Inclusion

Third Sector Relative Size

Organizational Capacity

Economic Outcomes

Social Conditions

Youth Effects

Senior Effects

Effects on the Poor

New Waterford

+

+

+

+

+

+

+

Sydney Mines

+

+

+

+

+

+

+

+

+

Whitney Pier

+

+

+

+

+

+

+

Dominion

-

+

Glace Bay

+

+

+

+

+

North Sydney

+

+

+

+

-

+

+

Results: Theory Supports ObservationPreponderance of positive change in more successful communities

Net benefit-cost per IA program group member over the full 54-month follow-up

Component of Analysis Individuals Communities Government Society

Monetized componentsParticipant Impacts CEIP earnings 34 344 0 -34 344 0 Foregone non-CEIP earnings -10 974 0 0 -10 974

Transfer payments (EI & IA) -11 836 0 11 836 0Tax payments (taxes and premiums) -3 559 0 2 921 -638

Other household member earnings 2 035 0 0 2 035Third Sector Organizational Effects Value from CEIP jobs (to sponsors) 0 20 024 0 20 024 Volunteering (CEIP induced) 0 2 404 0 2 404CEIP administrative costs 0 0 -4 274 -4 274Admin costs of EI & IA transfers 0 0 471 471

Net Benefit/Cost per Program Group Member 10 010 22 428 -23 390 9 048

Accounting Perspective

Cost-Benefit Analysis Results (IA Sample)

0.21

0.81

1.02

0.43

0.96

1.39

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

Participants Communities Society

Bene

fit-C

ost R

atio

Perspective

Total net benefit for every dollar that government spent on CEIP

EI Sample IA Sample

Positive Net Present Value

CEIP is very cost effective when one considers the combined benefits to individuals and community -- $1.39 in net benefits per dollar for IA recipients

Benefit-Cost Ratio

Cost-Benefit Analysis Accounting framework

Component of Analysis Individuals Communities Government Society

Non-Monetized componentsParticipant Impacts Reductions in Hardship, Stress + 0 0 + Improved Social Capital + 0 0 + Increased Trust in Networks + 0 0 + Foregone Leisure – 0 0 –

Community Effects

Increased Social Capital of Residents 0 + 0 + Improved Community Cohesion 0 + 0 + Increased Social and Civic Participation 0 + 0 + Foregone Leisure 0 – 0 –

Accounting Perspective

Social Capital – each additional contact is valued at 7% of income; therefore CEIP impact is worth $3,808 per participant

Trust – each additional percentage point increase in trust is valued at 2.5% of income; CEIP impact is worth $2,401 per participant

Hardship – the reductions in stress associated with lower hardship during CEIP is valued at $3,379

Valuing Intangibles – Examples

0.40

0.81

1.20

0.65

0.96

1.61

0.000.200.400.600.801.001.201.401.601.80

Participants Communities Society

Bene

fit-C

ost R

atio

Perspective

Total net benefit for every dollar that government spent on CEIP

EI Sample IA Sample

Positive Net Present Value

Including the intangible impacts improves the benefit cost ratio to $1.61 in net benefits per dollar spent

About a 50 percent improvement in overall net benefit to society

Benefit-Cost Ratio

To find out what difference a program makes, one needs to find an appropriate counterfactual. RCTs provide the best approach to set up a counterfactual and derive an Impact.

An Impact measure is required to do a proper Cost-Benefit Analysis; and a sound CBA is needed to report on the financial viability of a program

Analytical frameworks and Cost-Benefit Analysis should and can incorporate social benefits

Short of convincing private sector investors or government authorities to transform their accounting or fiscal framework to include environmental and social benefits, attempts should be made to monetize these benefits.

Take aways

WWW.SRDC.ORG

67

Questions?

2012SFF - Made to measure

Documents

Transcript of 2012SFF - Made to measure