Prevention Education Meeting

May 29, 2013

Evaluation 101

What isEvaluation?

Definitions of Evaluation

Examine and judge carefully.

Determine value or worth.

Process that attempts to determine, systematically and objectively, the relevance, effectiveness, and impact of activities in light of their objectives.

Purpose of Evaluation

Use evidence to prove that a program has delivered intended services.

Show that program has made an impact on the health of a certain population.

Windsor, Clark, Boyd, & Goodman. Evaluation of Health Promotion, Health Education, & Disease Prevention Programs

Purpose of Evaluation

To determine achievement of objectives related to improved health status

To improve program implementationTo provide accountability to funding

agencies, community, and other stakeholders

McKenzie, Neiger, & Smeltzer. Planning, Implementing, & Evaluating Health Promotion Programs

Outputs

The amount of product or service that the program intends to provide Type of activity, product or service

and target audienceParticipants ClientsAgencies Decision makersCustomers PublicHealth care providers

Type of evaluation process evaluation Can also include satisfaction with activity/product/service

Short Term Outcomes

Immediate effects of activities including changes in

Awareness KnowledgeAttitudes BeliefsValues OpinionsAspirations MotivationSkills (knowledge)

Type of evaluation impact evaluation

Intermediate Outcomes

Changes in behavior

Risk factor reduction NormsSelf management PoliciesSocial action PracticeProcedures Decision-

makingDemonstration of skills

Type of evaluation impact evaluation

Long Term Outcomes

Changes in burden of disease, health status, or quality of life prevalence or incidence mortality also social, economic, civic, environmental factors

Type of evaluation outcome evaluation

Data Sourcesfor Evaluation

Evaluation Terminology

Process evaluation measurement of activities during implementation

Impact evaluation measurement of immediate observable effects of a

program including awareness, knowledge, attitudes, skills, and behaviors

Outcome evaluation measurement of the ultimate goal of a program

Process Evaluation Data

Quantitative Counts

# meetings held, # training sessions conducted, # of people who attended

Descriptive Statistics mean satisfaction rating for event, percent of people who

responded they liked the eventQualitative

Descriptive information description of event, date of event, location of event,

description of who attended, what information was included during event

Process Evaluation Data

Monthly Progress Report Example:

# professional training sessions/workshops conducted # professionals trained # informational materials/brochures distributed

# primary prevention education sessions conducted for students # students attended primary prevention education sessions # informational materials/brochures distributed

# sessions conducted with faith-based organizations descriptions of topics covered during sessions

Impact Evaluation Data

Short term outcomes on a logic model awareness, knowledge, attitude change data

Quantitative data collected from surveys/questionnaires, written

tests

Qualitative data collected from surveys/questionnaires,

interviews, focus groups

Impact Evaluation Data

Intermediate term outcomes on a logic model behavior change data

Quantitative BRFSS data

Qualitative interviews, focus groups, observations

Impact Evaluation Data

Example: The results of the pre/post knowledge tests indicted that

participants’ knowledge about healthy relationships increased by 500%.

The results indicated that students learned to use negotiation techniques as assessed by post-test.

Responses on the modeling questionnaire indicate that 85% of students would intervene when other students act disrespectfully.

School administrators at 10 schools strengthened their anti-bullying policies.

Outcome Evaluation Data

Long term outcomes Changes in health statusQuantitative

YRBS data BRFSS data Vital data Hospital Discharge data ER or ED data

Qualitative Interviews, focus groups

Outcome Evaluation Data

Example: 2007 YRBS

29.9% of high school students had been in a physical fight at least once in the past year

11.8% of high school students had been hit by their boy/girlfriend in the past year

9.5% of high school students have been forced to have sexual intercourse against their will

2008 BRFSS 12.5% of adult females have been victims of forcible rape 16.7% of adult females have either been raped or have

experienced an attempted rape 4.7% of adult males have either been raped or have

experienced an attempted rape

Evaluation Toolsand Resources

ABCDE Method of Writing Outcomes

A – Audience (Who will change?)B – Behavior (What will change?)C – Condition (By When?)D – Degree (How Much?)E – Evidence (How will the change be

measured?)

Evaluation Activity

Developing a data collection tool

General Evaluation Guides

The Program Managers Guide to Evaluation http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eva

l/reports/pmguide/pmguide_toc.html

Kellogg Foundation Logic Model Development Guide http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

Developing a Logic Model: Teaching and Training Guide http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.

pdf

Understanding Evaluation: The Way to Better Prevention Programs http://www.ed.gov/PDFDocs/handbook.pdf

Kellogg Foundation Evaluation Handbook http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

Topical Evaluation Guides

Core Performance Measures for Steps to a HealthierUS http://www.cdc.gov/eval/steps/Steps%20Core%20Performance%20Me

asures%2007-2007.pdf

Introduction to Program Evaluation for Comprehensive Tobacco Control Programs http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_e

valuation/evaluation_manual/00_pdfs/Evaluation.pdf

Physical Activity Evaluation Handbook http://www.cdc.gov/nccdphp/dnpa/physical/health_professionals/interv

entions/handbook.pdf

CDC DASH Evaluation Briefs http://www.cdc.gov/HealthyYouth/evaluation/resources.htm#4

Strategic Planning Kit for School Health Programs http://www.cdc.gov/HealthyYouth/evaluation/pdf/sp_kit/sp_toolkit.pdf

SMART Cards http://www.cdc.gov/HealthyYouth/evaluation/pdf/SMARTcards.pdf

Violence Prevention Evaluation Guides

Sexual and Intimate Partner Violence Prevention Programs Evaluation Guide Book by CDC, 2007, can be ordered from http://wwwn.cdc.gov/pubs/ncipc.aspx

Measuring Violence-Related Attitudes, Behaviors, and Influences Among Youths: A Compendium of Assessment Tools, Second Edition http://www.cdc.gov/ncipc/pub-res/pdf/YV/YV_Compendium.pdf

Evaluation Websites

CDC’s Evaluation Framework http://www.cdc.gov/eval/framework.htm

Community Tool Box http://ctb.ku.edu/en/

American Evaluation Association http://www.eval.org/

Coalitions Work http://coalitionswork.com/

Writing Good Goals and Objectives http://apps.nccd.cdc.gov/dashoet/writing_good_goals/menu.html

Basic Guide to Program Evaluation http://www.managementhelp.org/evaluatn/fnl_eval.htm

University of Wisconsin Extension – Program Development and Evaluation http://www.uwex.edu/ces/pdande/index.html

Kellogg Foundation http://www.wkkf.org/default.aspx?

tabid=75&CID=281&NID=61&LanguageID=0