Coalitions & Collaborations Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.
Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.
-
Upload
jacob-manning -
Category
Documents
-
view
215 -
download
0
Transcript of Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.
![Page 1: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/1.jpg)
Process Evaluation
Intermediate Injury Prevention CourseAugust 23-26, 2011
Billings, MT
![Page 2: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/2.jpg)
Session Goal
To provide Participants with the information to design a process evaluation for an injury prevention project.
![Page 3: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/3.jpg)
Session Objectives• Define Process Evaluation• Describe why and when to use process
evaluation• Recognize the type of data collected
during Process Evaluation• Describe Process Evaluation methods• Design a Process Evaluation for an
injury prevention project
![Page 4: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/4.jpg)
Formative Evaluation
Process Evaluation
Impact Evaluation
Outcome Evaluation
Short-term:
Long-term:
Timely Feedback
Baseline Data Collection Project Implementation
The 4 Stages of EvaluationReview
Pre-Testing MonitoringHealth Outcomes
Knowledge, Attitudes & Practice
Changes in the Target Audience
![Page 5: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/5.jpg)
Process EvaluationDefined
The type of evaluation used to determine if your project is:
• Being implemented as planned, and
• Reaching its target audience.
![Page 6: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/6.jpg)
Process EvaluationDefined
“Did I do what I set out to do,
and did it make a difference?”
Process Evaluation
Impact/Outcome Evaluation
![Page 7: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/7.jpg)
Process Evaluation asks:– What was actually done?
– Where and when was it done?
– How often was it done?
– Who did it, and who did they do it for?
Process EvaluationDefined
![Page 8: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/8.jpg)
• Process evaluation planning takes place during project planning and before formative evaluation starts.
• Begins immediately after your project is implemented.
• Continues throughout the life of your project.
Process EvaluationWhen its Used
![Page 9: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/9.jpg)
• Allows you to make adjustments in a timely manner.
• Needs of target population might change, and project may need to adapt.
• Identifies any problems that occur in reaching the target population.
Process EvaluationWhy it’s Used
![Page 10: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/10.jpg)
• Can be used to show funding agencies the project’s level of activity.
• Tells you how well your project is being implemented.
• Tells other interested programs, the “how” and “why” your program works.
Process EvaluationWhy it’s Used
![Page 11: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/11.jpg)
• Is necessary but not sufficient in evaluating a program’s effectiveness.
• Depends upon accurate record keeping and effective communication.
Process EvaluationLimitations
![Page 12: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/12.jpg)
Process EvaluationMeasures
Implementation Target Audience
Process Evaluation
Measures if your project is reaching its target audience
Measures & evaluates project implementation
![Page 13: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/13.jpg)
Process EvaluationImplementation
Implementation activities:• Developing project’s goals and objectives• Creation of an implementation protocol• Monitoring daily operations • Data collection • Coalition building • Ensuring staffing is at proper level to meet
program needs• Ensuring that staff are sufficiently trained
![Page 14: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/14.jpg)
Process EvaluationImplementation Questions
• What was actually done?• When and where was it done?• Who did it?• Who did they do it for?• Were any materials distributed?• What barriers or challenges were
discovered?• What was the cost?
![Page 15: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/15.jpg)
Process EvaluationTarget Audience
Reaching the Target Audience activities:• Ensuring the target pop. is being reached• Measuring program participation by target pop.• Making materials and resources available and
understood by target pop.• Determining if program is relevant to target pop. • Tracking the distribution of materials
![Page 16: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/16.jpg)
Process EvaluationTarget Audience Questions
• Is the target population being reached?
• What was the nature of this contact?
• How often and for how long was the target population involved?
• Are the project’s messages and materials appropriate for the target population?
![Page 17: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/17.jpg)
Process EvaluationData Collection Techniques
A. Project Exposure
B. Progress Review
C. Internal Audit
D. Target Population Survey
E. Project Site Survey
![Page 18: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/18.jpg)
Process EvaluationData Collection Techniques
A. Project Exposure• Monitors all project contacts (telephone,
email, classes, etc.) and materials distributed (brochures, products, etc.)
• Monitors who utilizes project information (e.g., sign-in sheets)
![Page 19: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/19.jpg)
Process EvaluationData Collection Techniques
B. Progress Review• Reviews program activities to determine if
goals and objectives are being met (e.g., meetings with coalition, tribal members, courses provided)
• Conducts interviews with staff and coalition members
• Conducted by project staff and interested parties
![Page 20: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/20.jpg)
Process EvaluationData Collection Techniques
C. Internal Audit• Compares implementation plan to actual
activities • Documents staff efforts, resources,
amount of time devoted to each task, and date of task completion
• E.g., Sleep Safe Coordinator’s quarterly reports
![Page 21: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/21.jpg)
Process EvaluationData Collection Techniques
D. Target Population Survey• Measures whether program is reaching
target audience • Describes target population’s awareness
of project, level of interest in the project, number who utilize project
![Page 22: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/22.jpg)
Process EvaluationData Collection Techniques
E. Project Site Survey• Determines if materials are being
distributed effectively• Monitors the number of materials used
over a certain period of time • E.g., Sleep Safe smoke alarm
follow-up data collection form
![Page 23: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/23.jpg)
Process EvaluationInterpreting the Data
The 5 process evaluation methods are now producing results…
Results of Internal Audit are used to inform project staffNo problems discovered: status-quoProblems discovered: timely revisions to projectMay require Formative evaluation to determine the problem’s cause
![Page 24: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/24.jpg)
Formative Evaluation
Process Evaluation
Impact Evaluation
Outcome Evaluation
Short-term:
Long-term:
Timely Feedback
Baseline Data Collection Project Implementation
Process EvaluationInterpreting the Data
Pre-Testing MonitoringHealth Outcomes
Knowledge, Attitudes & Practice
Changes in the Target Audience
![Page 25: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/25.jpg)
• Process evaluation is a management tool that is used to make sure that your project is implemented as planned and on schedule.
• Information collected during process evaluation can be used to make adjustments to your project.
• Tells you how well your project is being implemented.
Process EvaluationConclusion
![Page 26: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/26.jpg)
Process EvaluationConclusion
“The only difference between stumbling blocks and stepping stones is how you use them.”
(And your ability to recognize them.)
![Page 27: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/27.jpg)
• Using the provided goal & objectives and the Process Evaluation worksheet, your group should:– Select an objective.– Describe one way you will measure how
well your project is implemented and how it will reach the target audience.
– Describe how you will measure the objective using one of the process evaluation data collection techniques.
Process EvaluationExercise
Time allowed: 15 minutes
![Page 28: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/28.jpg)
Child Passenger Safety• Goal: ..to increase the use of child
passenger safety seats & correct use of seats
• Objectives:– Coalition to increase number of CPST and
Instructors by end of this year.– Media advertisement of clinics,
checkpoints, and importance of car seats by March 2007.
– Conduct a clinic & checkpoint per community yearly, starting next year.
![Page 29: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/29.jpg)
Formative Evaluation
Process Evaluation
Impact Evaluation
Outcome Evaluation
Short-term:
Long-term:
Timely Feedback
Baseline Data Collection Project Implementation
The 4 Stages of Evaluation
Pre-Testing MonitoringHealth Outcomes
Knowledge, Attitudes & Practice
Changes in the Target Audience
![Page 30: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/30.jpg)
Impact Evaluation
• Collects baseline information of people’s knowledge, attitudes, beliefs or behaviors.
• Long-term data collection (3 to 5 years) to measure and reduce morbidity mortality.
![Page 31: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/31.jpg)
Impact Evaluation
• Changes in elders use (observed, self-reported) of walkers.
• Changes in community members behavior in using occupant restraints (seat belt wearing for adults or car seats for children.
![Page 32: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/32.jpg)
Impact Evaluation
• Children bicycle helmet use (observed, self-reported) before/after a bicycle campaign.
• Changes in elders use (observed, self-reported) of walkers.
• Changes in number of operable smoke detectors installed/maintained in homes after a comprehensive fire prevention campaign.
![Page 33: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/33.jpg)
Impact Evaluation Example: Alaska PFD Promotion Project
• Drowning rate in Alaska’s YK Delta more than 3 times greater than the State average
• PFD promotion project began in YK Delta
• Observational surveys of PFD use were conducted after baseline data was collected
• Timely changes made to increase sales and PFD use
![Page 34: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/34.jpg)
Impact Evaluation Example: Alaska PFD Promotion Project
Usage Rates %
# Sales
4930 29
77
125
241200
300
59 58 58
898891
0
50
100
150
200
250
300
350
90 91 92 93 94 95 96 97 98
0102030405060708090100
![Page 35: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/35.jpg)
Formative Evaluation
Process Evaluation
Impact Evaluation
Outcome Evaluation
Short-term:
Long-term:
Timely Feedback
Baseline Data Collection Project Implementation
The 4 Stages of Evaluation
Pre-Testing MonitoringHealth Outcomes
Knowledge, Attitudes & Practice
Changes in the Target Audience
![Page 36: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/36.jpg)
Outcome Evaluation
• Usually requires significant resources, long periods of time, and ongoing data monitoring.
• Used less frequently than impact evaluation in NA injury prevention programs.
• Focuses on the program’s long-term effect on its target audience.
• Conducted after a program has been completed
![Page 37: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/37.jpg)
Outcome Evaluation Example: Drowning Rates YK Delta vs. All-Alaska
0
10
20
30
40
50
60
70
80
90
88 89 90 91 92 93 94 95 96
YK Starts PFD Program
YK Delta
All Alaskan
43% Reduction
Rate Per 100,000
![Page 38: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/38.jpg)
Evaluation Summary:Identify the evaluation type
• Number of PFDs distributed • Decrease in the number of drinking and
driving violations • Review of IP materials describing storage of
poisons • Number of meetings with Tribal council to
discuss possible speed limit ordinance• Number of completed suicides
![Page 39: Process Evaluation Intermediate Injury Prevention Course August 23-26, 2011 Billings, MT.](https://reader034.fdocuments.net/reader034/viewer/2022050714/56649e265503460f94b1621d/html5/thumbnails/39.jpg)
Additional Resources
www.nhtsa.dot.gov
www.cdc.gov/eval/resources.htm