LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009.

LiveWell Colorado Evaluation

LiveWell Colorado Taskforce Meeting

July 16, 2009

LWCO Evaluation Team: Mission

To guide and support LiveWell Colorado

communities in evaluation and to report the overall outcomes and impact of

LiveWell initiatives.

LWCO Evaluation Team: Who We Are

Sr. Research Asst.Deanna McQuillan, MA

Sr. Research SpecialistErica Morse, MA

Sr. Research Asst.Lisa Harner, MA

Tristan Sanders, BS

Sr. Evaluation ManagerBonnie Leeman-Castillo, PhD

Data AnalysisData Collection

Community Reports & TAData Management

& Reports

Statistical AnalystSophia Newcomer, MPH

KPCO Community & Local Gov’t Relations

Corina Lindley, MPHMonica Buhlig, MPH

Principal Investigator Diane King, PhD

What We Do

Collect Enter Analyze Report

…………………………………………………Data

Primary Sources of Data

Demographic & Process Data Community Action Plan (year end) Continuation Progress Report (year end) Annual Site Visits (summer) Key informant interviews with key community

coalition members (fall) TA Reports (monthly)

Outcomes Data Continuation Progress Report (year end) IVR Surveys (varies by community)

Other Sources of Data

Media and Website Hits Community Highlights Local Evaluators’ Reports School surveys CDPHE surveys

e.g., BRFSS and Child Health Survey

Today’s Plan

LWC IVR Survey: Analytic Update Progress Milestones

Current indicators of future success Community Reports

Small group discussion

LiveWell Colorado IVR Survey:Analytic Update

Sophia Raff Newcomer, MPHBiostatisticianInstitute for Health Research, Kaiser Permanente Colorado

Agenda

What’s new with the IVR survey? IVR survey updates Timing of community surveys

Analytic update Discussion of raw vs. weighted data Examples

What’s new with the IVR survey? Timing of survey

Survey will be conducted: M&P Year Year 1 (first year of implementation) Year 5 Year 10

Questions Consistent “core” questions Limiting number of questions on survey

Unweighted data In the LiveWell Colorado IVR survey, a sample (n) of

community members participate in a phone survey.

The intent of the survey is to use the responses from that sample of community members (n) to better understand key health and behavior indicators of the entire community (N).

Community (N)

Sample (n)

Unweighted data (cont.) The data collected is raw data

Raw data: the actual responses from survey participants Sometimes refered to as “unweighted”

If we use the raw data from the sample (n) to draw conclusions about the community (N), then we assume that: Non coverage and non response bias is the same

throughout all demographic groups in the community The sample (n) is representative of the community

These assumptions are violated in the LWCO IVR

Example: Park Hill 2006Question: # of fruits and vegetables/dayN=19,315 (2000 Census) n= 321

18-24 25-34 35-44 45-54 55-64 65+

Population (N) 963 2003 1947 1769 1009 1159Population (%) 4.99% 10.37% 10.08% 9.16% 5.22% 6.00%


MA

LE

FE

MA

LE

18-24 25-34 35-44 45-54 55-64 65+


Sample (n) 1 7 10 20 13 20Sample (%) 0.31% 2.18% 3.12% 6.23% 4.05% 6.23%


Sample (n) 4 19 44 42 56 85Sample (%) 1.25% 5.92% 13.71% 13.08% 17.45% 26.48%

MA

LE

FE

MA

LE

18-24 25-34 35-44 45-54 55-64 65+


Sample (n) 1 7 10 20 13 20Sample (%) 0.31% 2.18% 3.12% 6.23% 4.05% 6.23%

ICW = Population (N) / Sample (n) 963.00 286.14 194.70 88.45 77.62 57.95


Sample (n) 4 19 44 42 56 85Sample (%) 1.25% 5.92% 13.71% 13.08% 17.45% 26.48%

ICW = Population (N) / Sample (n) 247.25 102.16 49.64 52.40 24.27 21.07

MA

LE

FE

MA

LE

Weighted survey analysis (Park Hill data) Weighted survey analysis adjusts the raw,

collected data from the sample (n) to try to “look” like the population (N)

Weighting on age and sex, the individual case weights (ICW) are: Small cell sizes; collapse

with next age group

Definitions…and then some data! Survey sample size: number of respondents that started

the IVR survey (provided a valid age and pressed a key for the first question)

Item sample size: number of respondents who provided a valid response for that question

95% confidence interval: range of values in which the “true” weighted percentage is expected to occur, with 95% probability If we drew 100 samples from that same community, we would

expect the weighted percentage from 95 of those samples to fall within the reported 95% confidence interval

Have you heard of Park Hill Thriving Communities?

2006 2007 2008

Survey Sample Size 420 524 529

Item Sample Size n=408 n=514 n=521

Response= Yes 20.1% 37.0% 41.8%

Response= Yes

Weighted %(95% C.I.)

17.0% ( 12.4% - 21.6% )

33.4% ( 28.0% - 38.7% )

39.3% ( 33.8% - 44.8% )

RAW DATA

WEIGHTED DATA

Have you heard of Park Hill Thriving Communities?

Weighted percentage for response=“Yes”

2006

10%

20%

30%

40%

17.0%

12.4% 21.6%

2007

33.4%

28.0% 38.7%

2008

39.3%33.8% 44.8%

Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?PARK HILL 2006 2007 2008



Meeting guidelines 33.1% 41.4% 31.9%

Meeting guidelines


37.3%( 25.4% - 49.2% )

39.9%( 33.7% - 46.0% )

34.9%( 28.7% - 41.1% )

RAW DATA

WEIGHTED DATA

Meeting guidelines of 5 or more servings of fruits and vegetables per day?

PARK HILL 2006 2007 2008




Meeting guidelines


44.7%( 37.4% - 51.9% )

41.3%( 35.5% - 47.2% )

38.7%( 32.6% - 44.7% )

RAW DATA

WEIGHTED DATA

Have you heard of LiveWell Commerce City?

2006 2007 2008



Response= Yes 20.6% 15.9% 20.5%

Response= Yes


14.0% ( 9.36% - 18.7% )

15.1% ( 10.1% - 20.0% )

19.1%( 13.7% - 24.4% )

RAW DATA

WEIGHTED DATA

Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?COMMERCE CITY 2006 2007 2008




Meeting guidelines


26.7%( 18.9% - 34.6% )

27.3%( 20.4% - 34.2% )

35.5%( 27.9% - 43.1% )

RAW DATA

WEIGHTED DATA

Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?

Weighted percentage of respondents meeting moderate PA guidelines

COMMERCE CITY

2006

20%

30%

40%

26.7%18.9% 34.6%

2007

27.3%20.4% 34.2%

2008

35.5%27.9% 43.1%

Meeting guidelines of 5 or more servings of fruits and vegetables per day?

COMMERCE CITY 2006 2007 2008




Meeting guidelines


29.1%( 21.5% - 36.8% )

38.8%( 31.0% - 46.6% )

25.8%( 19.6% - 31.9% )

RAW DATA

WEIGHTED DATA

Data weighting summary Advantages:

Allows the collected IVR data to “look” more like the population from which it is sampled

Limitations 2000 Census data Weighting on other sociodemographic factors,

such as race, is impractical Small sample sizes and “extreme” ICWs

Any questions?

Progress Milestones

How do we answer that question????? Collaborative Partnerships & Community

Capacity

4 Intermediate Outcomes that Predict Impact Volume of changes: number of new or modified community &

systems changes. Intensity of changes: modifying access & removing barriers

more intense than providing information about services. Permanence (durability) of changes: ongoing versus once Penetration of changes: was contact made (reach) through

multiple sectors & settings (adopters) or geographic areas.

Collaborative Partnerships & Capacity Building

Are Partners Broadly Representative? e.g., 4 core sectors, parents, policy makers, police,

target populations, non-profit organizations

Are Partners (including residents) Engaged? e.g., shared leadership & work teams

Is Community Capacity Increasing? e.g., amount of resources leveraged

Volume & Intensity

How many organizational systems changes adopted?

How many environmental changes made?

How many sectors are you working in?

Permanence (a.k.a “maintenance”)

Are strategies durable? e.g., policy change > curriculum change > events

Has HEAL mission been adopted by partner organizations?

Has HEAL language been incorporated into planning & policy documents?

Broad Penetration (a.k.a. “reach”)

What proportion of target population(s) have been reached?

Are residents being reached at multiple levels of the ecological model? e.g., cooking class (individual / program);

corner store/farmer’s markets (family / environment); school rules involving snacks (organization / policy)

Broad Penetration (a.k.a. “adoption”) Are changes holistic (i.e., inter-linked)?

e.g., park improvements + safe routes to park > park improvements alone

Are single goals addressed in multiple sectors? e.g., breastfeeding addressed through:

healthcare (training providers to counsel, changing formula bags for lactation kits),

social marketing (norms & attitudes),

workplaces (space, time, supervisor awareness)

Community Reports

Community Reports: ActivityQuestions for Discussion:

Who would use this information: partners, stakeholder, and/or residents?

How might it be used? Is it a good way to display this information?

Does it need more/less text?

What would you add or subtract on this topic/subsection of the report?

LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009.

Documents

Transcript of LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009.