Evaluating Collective Impact - OPEC 2018 · Evaluating Collective Impact Community Systems Group,...

Evaluating Collective Impact

Community Systems Group, LLC. 1

Community Systems Group, LLC 2

Should I Stay?

1. What does it mean to establish shared measures?

• For the our coalition’s capacity and process?

• For our coalition’s accomplishments (the dose)?

• For our coalition’s outcomes?

2. How do we connect these measures to show

collective impact?

3. What software is available to support data collection

and analysis.

Goals




Goals

4

Evidence-Based Processes


5

Lit Review: Quality / Degree


6

Behavior vs.

Perception


7

Benchmark


1. What is our shared (and evidence-based) definition of capacity?

2. What are our agreed upon measures?

3. How could having common measures of capacity benefit our work?

4. Would if be beneficial if all coalitions in our community used the same measures?


Critical Reflection:





Goals

Presenting Question:

What is it that partnerships can do that their

members cannot do (or do as well) alone?


1. Community Change: Instances

New or improved policies, programs and practices facilitated by the initiative in the community.

Definitions


12

Falmouth Prevention Partnership

• History of the Partnership

• Relationship with CSG


13

Community Changes are new or modified programs, policies and

practices facilitated by the partnership to reduce substance abuse in Falmouth. This includes modifications to the physical environment.

Examples: • For the first time a Social Host Lesson at Falmouth High School was implemented as part

of the health curriculum for students in the 9th through the 12th grade. 375 students attended the social host laws class. (New Program: May 1, 2012)

• The Falmouth Police Department dispatching added a new category to track liquor law

violations for underage persons. Prior to this there was not a specific way to collect data on underage liquor law violations. (New Practice: September 13, 2011)

• The FPP, Falmouth Public Schools and the Falmouth Police Department co-sponsored and required attendance at a Social Host Pre Prom Assembly for the parents/caretakers of children attending the Junior Prom. (New Program: May 9, 2012)

13 13 Community Systems Group, LLC 13


The Falmouth Prevention Partnership facilitated:

64 Community changes intended to reduce underage alcohol use.

42 Community changes intended to build opportunities for pro-social involvement or that targeted all drug use.

30 Community changes intended to reduce Rx abuse.

28 Community changes to increase community capacity (such as new partnerships).

164 Documented new or modified programs, policies, and practices facilitated by the Partnership from 2009 to 2013 in order to reduce substance abuse in Falmouth, MA.

15

Dimensions of Intensity2

Dimension Scoring Rationale

Duration High Medium Low

Ongoing More than Once One Time

Emerging evidence and expert consensus suggest that community changes that are sustained are more likely to affect behavior change and population-level outcomes.

Reach High Medium Low

≥ 21% of population = 6% to 20% of population ≤ 5% of population

Emerging evidence and expert consensus suggests that exposure to community changes is important to affect outcomes.

Strategy High Medium Low

Policy, Consequences, Access Social Support Information, Skills

Emerging evidence and expert consensus suggests that community changes addressing policy and access are more likely to achieve outcomes.

2 Adapted from Collie-Akers, V.L., Fawcett, S.B., and Schultz, J.A. (2013). Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica, 34(6): 422-428.


Helping ou create a healthier communities Helping you create healthier communities 16

Dimensions of Intensity2

Community Change Duration Reach Strategy Score

Launch of the sticker shock campaign in 18 liquor retail stores with 10, 000 stickers.

Medium Medium Low 1.20

Pre Prom Assembly which is mandatory for all parents of students going to the Junior prom.

Medium Medium High 2.10

For the first time participated in "Credit for Life" Community Service Booth.

Low Low Low 0.30

2 Adapted from Collie-Akers, V.L., Fawcett, S.B., and Schultz, J.A. (2013). Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica, 34(6): 422-428.

Helping ou create a healthier communities 17 n = 15

i = 15.75 n = 16

i = 23.70 n = 36

i = 38.70 n = 26

i = 37.95 n = 32

i = 45.15 n = number of changes i = total intensity score

Intensity of Changes to Reduce Underage Alcohol Use facilitated by the

Falmouth Prevention Partnership

0

5

10

15

20

25

30

35

40

45

50

2009 2010 2011 2012 2013

Provide Information

Provide Support

Build Skills

Change the Physical Design

Access & Barriers

Policy Change

Rewards & Consequences

sum

an

nu

al in

ten

sity

sco

re

Year (First 3 Quarters)

1. Community Change: Instances

New or improved policies, programs and practices facilitated by the initiative in the community.

2. Services Provided: Contact Hours

Events that give information, build skills or provide social support for people in the community.

Definitions


Helping ou create a healthier communities Community Systems Group, LLC 19

Services Provided are events that impart information, develop skills,

and give social support such as through classes, screenings, alternative activities, and workshops. Records on services provided include the number of classes conducted, participants, length, and location.

Examples: • The Falmouth Prevention Partnership provided an in-service training for K-12

Adjustment and Guidance Counselors in Falmouth Public Schools on use of evidenced-based prevention strategies.

• FPP participated in a "Credit for Life" Community Service Booth. The event is for high school seniors to prepare them for financial literacy and community volunteerism.

• The Alcohol and Marijuana Action Team brought the "It's All About YOU" presentation by Richard Campbell to the Junior and Senior Falmouth High School students in a one-hour assembly format.

20


0

200

400

600

800

1000

1200

1400

1600

1800

2009 2010 2011 2012 2013*

Tota

l C

on

tact

Ho

urs

FPP: Services Provided 2009-2013

Provide SocialSupport

ProvideInformation

Build Skills

3212 youth & 3120 adults received services facilitated by the Partnership.

*2013 data are for the first 9 months of the year.

Collecting Output Data

3. Media: Impressions

Coverage of the initiative, its projects or issues in any media channel such as through newspapers, newsletters, on-line, radio, and television.

Definitions



Media are instances of coverage of the initiative, its projects, or issues

in the newspaper, on-line (“new media”), radio, television, and other channels. Records on media include the number of impressions and the media channel.

Examples:

• "Program Aims To Educate Parents Who Serve Alcohol“ article in Falmouth Enterprise describes how parents of high school students will receive a letter in the mail warning them of the legal consequences of allowing teenagers to drink alcohol in their homes.

• "Parents Who Host Lose The Most" public service announcement ran 55 times from Dec.1 -Dec.24, 2011 on local public access television, Channel 13.

• Dr. Mike Bihari, FPP Chair, wrote the Friday, September 27, 2013 “Risky Business” column in the Falmouth Enterprise. The article contained statistics on underage drinking and the correlation between teen perception of parental disapproval and a decrease in underage drinking.


0

200

400

600

800

1000

1200

2009 2010 2011 2012 2013*

Tota

l Im

pre

ssio

ns

in T

ho

usa

nd

s

FPP: Media Impressions 2009-2013

Risky Business column

Community Education

Sticker Shock

Parents Who Host Losethe Most

*2013 data are for the first 9 months of the year.


“Parents Who Host Lose the Most” Campaign

Average Impressions per Falmouth Resident

2011 2012 2013

6.11 22.32 9.88

Collecting Output Data

3. Media: Impressions

Coverage of the initiative, its projects or issues in any media channel such as through newspapers, newsletters, on-line, radio, and television.

4. Resources Generated: Dollar Value

Acquisition of funding for the initiative and the community through grants, donations or gifts in-kind.

(usually cash, materials or time)

Definitions


1. Catalyst for needed change.

Community Change

2. Assure essential supports.

Services Provided

3. Foster an ongoing community conversation.

Media

4. Generate and focus resources.

Resources Generated

The Four Metrics


1. What is our shared (and evidence-based) definition of accomplishments /outputs?

2. What are our agreed upon measures?

3. How could having common measures of outputs benefit our work?

4. Would if be beneficial if all coalitions in our community used the same measures?


Critical Reflection:






Goals






2. How do we connect these measures to show

collective impact?

Goals

30

Analysis of Contribution

1. Collect output data (measure the dose).

2. Establish a time sequence.

3. Demonstrate a plausible mechanism.

4. Account for alternative explanations.

5. Show similar effects in similar contexts.


31

Establishing a Time Sequence

Method

• Document when the intervention took place.

(e.g. the four outputs of change, service, media and resources)

• Document when the outcomes took place.

(e.g. local conditions, risk factors, behaviors, health consequences, etc.)

• Visually plot these to establish a time sequence.


32

Falmouth implements a survey of students modeled on the Communities That Care instrument (PRIDE Survey).

Measures the incidence and prevalence of substance use, delinquency, and related problem behaviors and the risk and protective factors that predict those problems in communities.

Surveys were conducted in 2009, 2011, and 2013.

Students in 8th through 12th grades were surveyed (In 2009 students in 7th grade were also included).



33 i = 15.75 i = 23.70 i = 38.70 i = 37.50 i = 42.60

Relationship in Time Between Changes to Reduce Underage Alcohol Use and

Self-Reported Rates of Alcohol Use in the Past 30 Days by 11th Graders

0

5

10

15

20

25

30

35

40

45

50

2009 2010 2011 2012 2013

Sum Intensity

Score

sum

an

nu

al in

ten

sity

sco

re

100

90

80

70 60 50

40

30

20

10

0

% of 11th Grade Students Reporting Alcohol Use in the Past 30 Days.

% = 58.30 % = 56.50 % = 33.50


0

10

20

30

40

50

60

70

80

90

100

7th 8th 9th 10th 11th 12th

2009

2010

2011

2012

2013

2014

2015

2016

2017

Change occurs around high school with Classes 2014, 2015, and 2016. (p = .05)

Class of 2017 is showing change starting in the 8th grade. (p = .05)

Past 30 Day Use of Alcohol by Class Cohort

Baseline Data

Intervention Data

35

• There has to be a relationship in time in order for

there to be a contribution.

• However, a relationship in time does not prove a

contribution.

• If improvements happen after the “dose” then an

exploration is appropriate.



Most Common Error

Assuming a causal relationship exists just because one thing follows another.

ex post facto error

Superstitious sports fans who believe their lucky hat helped the team win the game.

Coalitions that see 30 day use of alcohol go down and suggest they are the cause.


36 Helping ou create a healthier communities Helping you create healthier communities 36

37






38

Mechanism:

“the means by which the intervention

could have achieved intended effects.”

Mechanism of Effect


39

Mechanism of Effect


By what means could the changes, services, and media implemented by the Partnership have affected rates of underage alcohol use?

1. Facilitated changes, services, and media implemented interventions with an existing evidence-base.

• Social Host Laws • Parents Who Host Lose the Most • Sticker Shock • Guiding Good Choices • Pro-Social Opportunities

40

Mechanism of Effect



2. Facilitated changes, services, and media implemented a comprehensive set of behavior change strategies.

• Provide Information • Provide Social Support • Build Skills • Change Access / Barriers • Alter Rewards / Consequences • Change the Physical Design of the Environment • Change Rules & Policies

41

Mechanism of Effect



3. Pathway (logic model)

Past 30 Day Alcohol Use

Availability

Retail

Home Pro-Social Supports

Parental Attitudes

42

Alcohol Use and Related Factors by Class Cohort

7.9

26.1

33.5

95.6

90.7

92.9

29.3

55.5 65.6

56 56 58

44 48

48

0

10

20

30

40

50

60

70

80

90

100

7th 9th 11th

Class of 2014

31.8

56.5

86.5

78.9

44.2

68.1

48 51

43

45

0

10

20

30

40

50

60

70

80

90

100

9th 11th

Class of 2012

Past 30-DayUse

ParentalDisapproval

Availability

Pro-social:School

Pro-social:Community

Baseline or “Pre” After Interventions or “Post”

Mechanism of Effect

42 42 Helping ou create a healthier communities Helping you create healthier communities 42

43


3. Pathway (logic model)

Past 30 Day Alcohol Use

Availability

Retail

Home Pro-Social Supports

Parental Attitudes

Mechanism of Effect

43 43 43 Helping ou create a healthier communities Helping you create healthier communities 43

44







45

What else could it have been?

In research this is called “threats to validity.”

45 Community Systems Group, LLC 45

Account for Alternatives



What else could it have been? [validity threats]

1. History: Events that co-occur with the

intervention and might account for the

observed change in outcomes.

2. Maturation: Processes within respondents

that co-occur with the intervention and might

account for the observed change in

outcomes.

47


3. Testing: The effects of taking a test upon

the scores of a second or additional testing.

4. Instrumentation: Changes in calibration of

instruments or in observers or scorers can

produce changes in measured results.



48




5. Regression: Groups or outcomes selected

for their extreme scores are likely to see

natural movement toward normal scores.

6. Selection: Differences between comparison

groups and intervention groups can bias

results.

49




7. Mortality: Lost cases or cases with only

partial data can alter scores across time and

cause different results across groups.

50

A. Secular Trends

- National data from Monitoring the Future survey shows a downward trend in 30 day use of alcohol over the same time period.

- State data from the Youth Health Survey shows a downward trend in 30 day use of alcohol over the same time period.

+ Trends in Falmouth exceed those observed nationally or at the state level.

+ Secular trends not followed on other substances such as marijuana.

50 50 50 Community Systems Group, LLC 50


51 51 51 51 Community Systems Group, LLC 51


B. Concurrent Efforts

To better understand concurrent efforts in Falmouth key informant interviews were conducted with:

• State prevention staff • Local leaders in youth work and prevention




State informants report no concurrent implementation of prevention efforts reaching Falmouth youth and families (one effort in Vineyard located elsewhere on the Cape).

Local informants reported three major efforts: • VIPS (focused on skills and mentoring) • Jack’s PACT • Together We Can (after-prom; recreation center)

FPP Partnered with all of these (funded, coordinated, co-implemented, helped to establish).

54

Conclusions

Influences on the national and state trends in rates of alcohol use may also be influencing rates in Falmouth.

VIPS, Jack’s Pact, and Together We Can are also making contributions to improved youth outcomes.

Much of these partner’s work is accounted for in the Partnership’s documented accomplishments / dose.

Additional dosage data from these community partners is required to quantify their potential contribution.



55



Conclusions

Only the Partnership was found to have implemented interventions consistent with the pathway effect observed.

National trends are inconsistent with the amount, timing, and specific nature of the observed effects (sudden, marked, and alcohol/parental attitudes specific).

56






5. Show similar effects in similar contexts.


Evaluating Collective Impact - OPEC 2018 · Evaluating Collective Impact Community Systems Group,...

Documents

Transcript of Evaluating Collective Impact - OPEC 2018 · Evaluating Collective Impact Community Systems Group,...