Dawnovise N. Fowler, Ph.D., A.M.

CDC/NCIPC/DVP

Presented at the RPE Grantee Meeting

August 21, 2012

Evaluation for RPE

Overview

Intro. to Evaluation Revisited

Who, What, When, Why

Differences between Research and Evaluation

RPE-specific Evaluation Components

A Conceptual Framework

Moving Upstream

Reach and Impact

Outer Layer Evaluation

INTRO. EVALUATION

Definitions

Program evaluation – the systematic investigation of the

merit, worth, or significance of a program

Process evaluation – assesses the extent to which a

program, its activities, and operations are implemented

as intended

Outcome evaluation – assesses the extent to which a

program/strategy produces the intended change (i.e., in

knowledge, skills, attitudes, behavior, norms, etc.)

CDC. Framework for program evaluation in public health. MMWR 1999;48(No.

RR-11).

Differences between Evaluation and

Research

Evaluation

Focuses on a

Program

Seeks to Improve

Determines Value

Asks “is it working?”

Research

Focuses on a

Population

Seeks to Prove

Value-free

Asks “did it work?”

http://www.priyasmall.wordpress.com

Why Evaluate?

Are we doing what we said we would?

To compare actual outcomes to intended

outcomes

What is the implementation process?

Is it working?

Why does this work or why not?

Informs program planning, improvement and needs

Demonstrates accountability

To retain or increase funding

Promotes sustainability Rossi, P.H, Lipsey, M.W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. CA: Sage Pubs.

Riger, S. et al. (2002). Evaluating Services for Survivors of Domestic Violence and Sexual Assault. CA:

Sage Pubs.

Who Evaluates?

EVERYONE

• Internal evaluation capacity

o Personnel

o Data infrastructure

o Prioritization

• Evaluation Consultants

o Being an educated consumer

When to Evaluate?

Always engage in organized reflection of program

for continuous improvement

• Integrating processes to achieve continuous program improvement

– Planning What actions will best reach our goals and objectives.

– Performance measurement How are we doing?

– Evaluation Why are we doing well or poorly?

What do

we do?

How do

we do it?

How are

We doing?

Why are we doing

well or poorly?

What to evaluate?

The implementation, change, outcomes, and impact

of programs, strategies, interventions, processes,

policies on knowledge, skills, attitudes, norms,

behavior, processes, policies, climate, and systems

Actual Violence (WHO, 2009)

RPE-SPECIFIC EVALUATION COMPONENTS

RPE/FOA Requirements

Measurable outcomes…will be in alignment

with the following…:

Increase capacity to address prevention…

CDC goals for Healthy People in Every Stage of Life,

…Healthy Places…*:

• Indicators based on decreased prevalence and

risks, and increased protective factors

RPE Program Trajectory

Foundation

Permitted Uses

Current

-Capacity bldg

-Success

Stories

Upstream

-Outcomes

2002 2012 10+ Years

-Permitted Uses

-Community Change

Strategies:

-Coalition Bldg

-Community Mobiliz

-Policy Change

-Norms Change

-Evidence Informed/

Based Strategies:

-Bystander

-SafeDates

-4th R

-Norms Change

-Increased

Awareness,

Knowledge, Skills

-Increased Capacity

-Partnership/

Leadership

Development

-SV Prevention

-Safe, equal, respect

-Health promotion

PROGRAM PROXIMAL EFFECTS DISTAL EFFECTS

RPE PROGRAM EFFECTS

The following information was abstracted from RPE Grantees’ 2010 Annual Reports:

HOW IS DATA USED BY GRANTEES

(EB) CONTINUUM OF STRATEGIES REPORTED

MOVING

Reach and Impact

Concise, objective, quantifiable measures of

a program progress

Reach: the proportion of intended target audience

that participates/receives the program

Reach Formula:

• Actual # served/ Potential # served

• Ex.: Program serves 1,800 survivors with a potential popul.

= 3,000. The reach is 60%.

Impact: the effect that interventions have on people,

organizations, or systems to influence health;

outcomes

Refer to CDC/NCCDPHP Program Evaluation Tip Sheet

The following fictitious scenarios for Grantees A and B

illustrate examples for framing reach and impact:

Organizational

Reach

25 out of the 30 local programs

funded by Super State are

implementing healthy relationship

programs in elementary schools

Individual

Reach

25,000 students received healthy

relationship programs out of a

possible 30,000 students who attend

schools in the areas served by

Super State’s local RPE programs

Impact 23,000 out of the 25,000 students

who received the healthy rlx

program were infraction-free over

the next year

Grantee A

Example: Grantee B

Organizational

Reach

Local programs partnered with bars

and nightclubs within a specific

geographic radius and close

proximity to area(s) of high reported

sexual assault rates to implement

bars campaigns and bystander

training

Individual

Reach

15 bars/nightclubs out of 18

participated; 180 employees were

trained out of a possible 225.

Impact Reported SA rates decreased in that

specific geographic area by 10% the

year following campaign/training

implementation

Template

EVALUATION FOR OUTER LAYERS

The Social-Ecological Model

Dahlberg, L.L., Krug, E.G. Violence—a global public health problem. In Krug, E., Dalhlberg, L.L. et al. (2002). World Report on

Violence and Health. Switzerland, WHO, 1- 56.

Individual Relationship Community Societal

• Antisocial beliefs

and attitudes

• Associating with

delinquent peers

• Diminished

economic

opportunities

• Cultural norms that

support aggression

and violence

• Previous hx of

victimization

• Involvement with

gangs

• High concentration

of poverty

• Norms supportive of

male superiority and

sexual entitlement

• Hx of engaging in

early aggressive

behavior

• Disruption

(divorce/separation/

loss)

• High levels of

residential

instability

• Weak laws, health,

educ’al, economic,

and social policies

• Poor behav control • Poor family functioning • Low social capitol

• Social cognitive

deficits

• Poor-quality parent-

child rlxs

• Social

disorganization

• Psychol/mental

health problems

• Marital/family conflict • Lack of institutional

support

• Involvement w/

alcohol and drugs

• Emotionally unsupp.

environment

• Weak community

sanctions

• Witnessed family

violence as a child

• Parental/partner

substance abuse

• General tolerance for

violence

• Low educ.

achievem

• Econ./other stress

• Low income

• Social isolation

Risk Factors for Perpetration of Violence (Haegerich & Dahlberg, 2011)

Outer Layer Strategies and Activities

Strategies Activities

• STRATEGIC USE OF

MEDIA

• NORMS CHANGE

EFFORTS

• POLICY-BASED

EFFORTS

-Mass/Social

media/marketing campaigns

-Peer education

-Community

organizing/mobilizing

Media Campaigns

Strategic use of news media to advance an issue or

initiative

Designed to influence quantity & quality of media

coverage

Presumes that the framing of a news story effects

the way people perceive the issue

Approach to Media Evaluation

Media Tracking

Systematic tracking of media coverage

• Carefully select search terms and criteria

• Search articles (electronic and non-electronic)

• Select a sample within a specific time frame

• Code and assess

Amt of Coverage Frames Used Sources used

Issues Covered Messages Conveyed

Mentions of Groups Affected

Mapping Analyses

Illustrates

Linkages between systems and organizations

Linkages between strategies, change, and outcomes

The existence of social capital, other strengths as well

as gaps

IPV-Related Death Rates per 100,000 Population, 2006-2009

IPV-Related Death Rates per 100,000 Population 2006-2009. Source: Georgia Violent Death Reporting System 2010. Department of Public Health. U.S. Census 2010. Analyzed Jan and Feb 2012 by GCADV.

Southwest

South Southeast

Coastal

South Central/

Dublin

West Central/

Columbus

North Central/

Macon

East Central/

Augusta LaGrange

Clayton Fulton

Cobb/

Douglas

East

Metro

Clayton

Northeast

North North GA

Northwest

Mapping Analyses

Potential questions to answer:

Are the right programs in the right places? How to

increase reach and improve impact?

Where are viable partnerships formed? Where are

they needed?

How are policies impacted/changed by the

presence/absence of programs and partnerships in

particular areas?

Overarching Considerations

Prioritize evaluation

Be conventional AND creative in your measures

Give respect to ‘interim’ outcomes

Be attentive to reach and impact

Utilize pre/post evaluation designs in relevant/reliable/valid ways

Remember/utilize the theory of change model

Existing Strengths

Expertise and commitment to prevention, programs/services,

and partnerships

Opportunities exist and can be cultivated to improve

collaboration, coordination, and alignment across efforts/entities .

General Evaluation Resources

http://www.cdc.gov/eval/framework/index.htm

[CDC evaluation framework standards]

http://www.cdc.gov/eval/resources/index.htm

[CDC resource page on evaluation]

http://comm.eval.org/eval/resources/librarydocumentlist/

?LibraryKey=1eff4fd7-afa0-42e1-b275-f65881b7489b

[American Evaluation Association public library search page]

National Center for Injury Prevention and Control

Division of Violence Prevention

For more information please contact Centers for Disease Control and Prevention 1600 Clifton Road NE, Atlanta, GA 30333 Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348 E-mail: cdcinfo@cdc.gov Web: http://www.cdc.gov The findings and conclusions in this presentation are those of the author and do not necessarily represent the official position of the Centers for Disease Control and Prevention.