Gina Baral Abrams, MPH, LSW, CHESAmy Melichar, MEd, CHES
AMERICAN COLLEGE HEALTH ASSOCIATION ANNUAL MEETING
SAN FRANCISCO, CAMAY 28, 2009
Evaluation Essentials for Health Promotion in
Higher Education: A Primer
Learning Objectives
Describe the uses and purposes of program evaluation in health promotion.
List conceptual approaches to evaluation.List the key qualities of measurable evaluation
questions.List qualitative and quantitative data collection
methods.Describe key qualities of an effective evaluation
report.
Evaluation Defined
“Examining the worth of a program, usually measuring it against a set of
predetermined objectives or a standard of acceptability.”
Green & Kreuter, 1999.
Uses and Purposes of Program Evaluation
Determine the rate and level of attainment of program objectives.
Assess the strengths and weaknesses of a program.
Help make decisions. Monitor standards of
performance. Establish quality assurance
and control mechanisms. Determine the generalizability
of an overall program or program elements to other populations.
Meet the demand for public or fiscal accountability.
Improve professional staff skill in program planning, implementation, and evaluation activities.
Promote positive public relations and community awareness.
Fulfill grant or contract requirements.
Contribute to scientific knowledge.
Indentify hypotheses for future study.
Conceptual Approaches to Evaluation
Objectives-based evaluation Determine extent to which objectives are achieved.
Management-oriented evaluation Provide information to aid in making decisions.
Consumer-oriented evaluation Provide information about products to aid decisions about
purchases or adoptions.Expertise-oriented evaluation
Provide professional judgments of quality.Participant-oriented evaluation
Understanding the complexities of programmatic activity, responding to an audience’s requirements for information.
The Evaluator’s Role
Objectivity is key when strategically selecting the evaluator.
Possible evaluators include: External evaluator Internal evaluator Internal evaluator with external consultant
Basic Types of Evaluation
Formative evaluation To provide information for program improvement.
What is working? What needs to be improved? How can it be improved?
Summative evaluation To provide information to make decisions about the
program’s future or adoption. What results occur? With whom? Under what conditions? With what training? At what cost?
Levels of Evaluation in Health Promotion
Process evaluation: to determine feasibility.“Implementation evaluation”“Quality assurance review”
What was delivered and how it was delivered? Did program follow protocol?
Applies non-experimental designs.
Impact evaluation: to provide information to make decisions about the program’s future or adoption.
What were short-term (immediate-24 months) effects of the program on target behaviors and their predisposing, enabling and reinforcing antecedents or on environmental factors?
Applies quasi-experimental and experimental designs.
Outcome evaluation: to assess changes in overall health status and quality of life indicators.
What were long-term changes (1-10 years)?Applies quasi-experimental and experimental designs.
Types of Evaluation Designs
Non-experimental Does not include random assignment or a control group. Asserts little or no control over confounding factors.
Quasi-experimental Includes experimental and control group. Does not include random assignment. Offers some control over confounding factors.
Experimental Experimental and control group. Includes random assignment. Asserts greatest degree over confounding factors. Evidence is most interpretable, defensible, and definitive.
Evaluation Plan: Key Steps
1. Gather background information2. Identify stakeholders and audiences3. Budget for evaluation4. Develop evaluation question(s)5. Determine sources of information required6. Select data collection methods 7. Conduct evaluation (collect, analyze and
interpret data)8. Communicate findings -- provide final report
Step 1: Gather Background Information
Why is the program needed? General background, research on other programs
and approaches.What are the goals and objectives of the
program? What is the program context?What are the program activities?
Step 1: Case Study Example
Required orientation program for all first-year students.One-hour workshop/discussion.Delivered by residential advisors in small groups.Conducted concurrently in residential setting.Resident advisors trained in advance to lead the workshops.
Program goals: Upon completion of the program, participants will understand the
role of drinking motives in decision-making concerning alcohol. Upon completion of the program, participants will be aware of the
role of social norms in decision-making concerning alcohol. Upon completion of the program, participants will be able to
identify “decision-making” as part of their own drinking behaviors.
Step 2: Identify Stakeholders and Audiences
Who are the stakeholders and audiences? What are their roles?
Stakeholder: anyone who has a stake in the program to be evaluated or the evaluation’s results.
Audiences: groups which have an interest in the evaluation and receive its results.
Consider stakeholder and audience roles: To make policy To make operational decisions To provide input to evaluation To react to For interest only
Step 2: Case Study Example
To Make Policy
To Make Operational Decisions
To Provide Input to Evaluation
To React
For Interest Only
Health Services X X X
Student Affairs Office X X X
Residential Life Office X X X
Resident Advisors X X
Student participants (freshmen)
X X
Parents and family of University freshmen
X
University community X
Other University students
X
University faculty and staff
X
Step 3: Budget for Evaluation
What resources are available to dedicate to the evaluation?
Estimate 7 – 10% of overall project budget for evaluation. Components may include, but are not limited to:
Staff salary with benefits Consultants Travel Communications Printing and duplication Program materials Supplies and equipment Food Incentives
Components may be paid in-kind, with operational dollars, external funding, etc. List all regardless of funding source.
Step 3: Case Study Example
Expenses Description Amount TotalStaff Salary
Health Educator .01 FTE $600Fringe 27.60% $166
Personnel Total $766
ConsultantsEvaluator $150/hour x 30 hours $4,500Statistician $60/hour x 10 hours $600
Focus Group Facilitators $20/hour x 10 groups (20 hours total) x 2 facilitators $800
Outside Consultants Total $5,900
TravelTravel/Meals $500Business Meeting Expenses $150
Travel Total $650
Printing and DuplicationPrinting and Mailing Based on historical average $150
Printing and Duplication Total $150
Program MaterialsSurvey Administration $500
Program Materials Total $500
Supplies and EquipmentMeeting Rooms Based on historical average $100General Supplies Based on historical average $250Computer Supplies $200Telephone Calls $100
Supplies and Equipment Total $650
FoodCatered Meals and Meetings Based on historical average $500
Food Total $500
IncentivesFocus Group Incentives $500Survey Incentives $250
Incentives Total $750
Total Funds Needed to Complete Project 9,866$
Step 4: Develop Evaluation Questions
What questions need to be answered?
Seek input from stakeholders and audiences.Consider process, impact and outcome
evaluation needs.Divergent phase:
Create a laundry list of potentially important questions
Convergent phase: Select most critical questions from laundry list
Step 4: Case Study Example
Process Evaluation Question: To what fidelity was the program implemented?
Does the content of the workshop cover the intended objectives? Are resident advisors sufficiently prepared to lead the
workshops? Do first-year students participate in the workshops in the
intended manner?
Impact Evaluation Question: Did the students who participated in the workshop
achieve the desired results? Do participants understand the role of drinking motives in
decision-making concerning alcohol? Do participants understand the role of social norms in decision-
making concerning alcohol? Do participants identify “decision-making” as part of their own
drinking behaviors?
Step 5: Determine Sources of Information
What information do you need in order to answer the evaluation questions?
Peer review, quality
control, accreditation,
audit, certification, satisfaction
Predisposing, enabling and
reinforcing factors
Protective behavior or
environment
Health Social benefit
Changes in knowledge,
attitudes, beliefs, skills,
resources, social support, policy
Changes in frequency,
distribution, timing of
behavior, or quality of the environment
Changes in mortality, morbidity,
disability or risk factors
Changes in quality of life
Health promotion program
Process Impact Outcome
Adapted from Green & Kreuter, 1999.
Step 6: Select Data Collection Methods
What data will you collect and how will you collect them?
Strategies may include, but are not limited to: Tests and assessments Written questionnaires Policy audit Interviews with key participants Observations of behavior or environmental conditions Self-report logs or diaries Focus groups Community-level indicators of impact Service utilization rates Case studies Document review
Steps 5 & 6: Case Study Example
Determine information required and source/data collection method for each question.
Selected questions appear below. Note that there would be many more data collection methods if all questions were presented.
Research Question Information RequiredInformation Source/
Data Collection Method
To what fidelity was the program implemented?
Does the content of the workshop cover the intended
objectives?
Workshop objectives; workshop components.
Document review: Workshop lesson plan, resources,
handouts, activities.
Did the students who participated in the
workshop achieve the desired results?
Do participants understand the role of drinking motives in decision-making concerning
alcohol?
Change in knowledge of drinking motives.
One group, pre-test post-test design: Web-based self-report
survey of workshop participants.
Step 7: Conduct Evaluation Plan
June 2009
July 2009
August 2009
September 2009
October 2009
November 2009
December 2009
Contract with consultants
Meet with program developers and
stakeholders
Develop data collection methods
Resident advisor training
Program implementation and participant
observation
Administer surveys
Conduct focus groups
Data analysis
Deliver final report
Step 8: Communicate Findings
Components of the final report may include: Front matter (front cover, etc.) Executive summary Program background/evaluation description Present the results Discussion, conclusion, recommendations
Report in ways that encourage follow-through by stakeholders, to increase the likelihood that the evaluation will be used.
Re-visit list of stakeholders and audiences to determine how to communicate findings and to whom.
References
Fitzpatrick, JL, Sanders, JR & Worthen, BR. (2004). Program evaluation: Alternative approaches and practical guidelines, 3rd ed. Boston, MA: Pearson.
Green, LW & Kreuter, MW. (1999). Health promotion planning: An educational and ecological approach, 3rd ed. Mountain View, CA: Mayfield Publishing Company.
McDermott, R., & Sarvela, P. (1999). Health education evaluation and measurement, 2nd ed. The McGraw-Hill Companies.
Modeste, NN & Tamayose, TS. (2004). Dictionary of public health promotion and education terms and concepts, 2nd ed. San Francisco, CA: Jossey-Bass.
WK Kellogg Foundation. (1998). The W.K. Kellogg evaluation handbook. Available at www.wkkf.org.
Windsor, R., Baranowski, T., Clark, N., & Cutter, G., (1994). Evaluation of health promotion, health education, and disease prevention programs, 2nd ed. Mountain View, CA: Mayfield Publishing Company.
Questions?
Gina Baral Abrams, MPH, LSW, CHESPrinceton [email protected]
Amy Melichar, MEd, CHESMarquette [email protected]
Top Related