Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of...
-
Upload
lena-bumstead -
Category
Documents
-
view
216 -
download
1
Transcript of Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of...
![Page 1: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/1.jpg)
Gathering Evidence of Impact: A Continued Conversation
Jan Middendorf
Cindy Shuman
Office of Educational Innovation and Evaluation
![Page 2: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/2.jpg)
Using a logic model to describe your program Using a logic model to describe your program
INPUTS OUTPUTS OUTCOMES
Program
investmentsActivities Participation Short Medium Long-term
What you do, with whomWhat you do, with whom What your clients do as a resultWhat your clients do as a result
ImpactImpact
Learning Behavior Change
Condition Change
![Page 3: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/3.jpg)
Four Levels of Evaluation
Level 1: Reaction Level 2: Learning/Skill Building
– (short term outcome…ask at end of meeting, lesson, workshop, etc.)
Level 3: Transfer…Behavior Change– (medium term outcome… ask sometime after…maybe 6
months later ) Level 4: Results… Impact
– (long term outcome)
Kirkpatrick Model of Evaluation
![Page 4: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/4.jpg)
Evaluation of Outcomes
How will you know what your clientele did as a result of your educational program?
What is the evidence that they used the information?
Why Evaluate?
![Page 5: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/5.jpg)
4 main purposes of evaluation4 main purposes of evaluation
Improvement: to improve the program; to enhance quality; to manage more effectively and efficiently. The effort to enhance programs.
Accountability: to assess merit or worth; to assess effects; to assess costs and benefits. The effort to make judgments about the value of a policy or program.
Knowledge development: to gain new insights. The effort to add to the knowledge base about effective practice or to add to policy debate.
Oversight and compliance: to assess the extent to which a program follows rules, regulations, mandates or other formal expectations.
Source: Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding and improving policies and programs. San Francisco: Jossey-Bass.
![Page 6: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/6.jpg)
Indicators: Evidence of Achieving Indicators: Evidence of Achieving OutcomesOutcomes
• What would it look like?• How would I know it?• If I were a visitor, what would I see, hear, read,
and/or smell that would tell me this “thing” exists?
If the outcome is achieved, how will you know it? What will it look like? What is the
evidence?
![Page 7: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/7.jpg)
Indicators - EvidenceIndicators - Evidence
The information needed to answer your evaluation questions
Example: Did participant land owners or managers improve their land management practices?
Evidence: # acres or % of acres managed according to guidelines
# or quality of conservation plans implemented
![Page 8: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/8.jpg)
Indicators - EvidenceIndicators - Evidence
The information needed to answer your evaluation questions
Example: Did participants increase their ability to achieve financial self-sufficiency?
Evidence: #, % who increased financial knowledge, #, & who reduced debt,
#,% who established an emergency fund
![Page 9: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/9.jpg)
Have the pets been fed today?Have the pets been fed today?
How would you know that the animals have been fed? What is the evidence?
![Page 10: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/10.jpg)
Let’s practice….Let’s practice….
What is the evidence of…– High blood pressure?– A clean neighborhood?– A popular movie?– A good carpenter?– Learning at the workshop?
Would the evidence be different for young people vs. seniors, high- vs. low-income neighborhoods, rural vs. urban residents, or by ethnicity?
![Page 11: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/11.jpg)
Evidence is often expressed as numbers or percentages (number of…, percent of…, ratio of…, incidence of…, proportion of…).
However, not all evidence is numbers; qualitative evidence may be important.
Remember, "Not everything that counts can be counted."
![Page 12: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/12.jpg)
Work on Evaluation Plans
Develop a question or two to understand what clientele did differently as a result of your educational program
Develop a plan for how you will gather the evidence…
– Get names and phone numbers and call them 6 months later– Ask a third party what they see that is different– Observe differences yourself– Other?
![Page 13: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/13.jpg)
How good is the indicator?
Tangible – be able to “touch/know” the information in some way– See (observable) – Read (survey, records, etc.)– Hear (from individuals, others)– Tips: direct, specific, useful, practical, culturally
appropriate, adequate, clearly defined
![Page 14: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/14.jpg)
What is an indicator?
An indicator is the specific information, or evidence, that represents the phenomenon you are asking about.
Indicator of fire = smoke Indicator of academic achievement = grades
![Page 15: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/15.jpg)
How good are your questions?
Can the questions be answered given the program?
Are the questions truly important? Will the questions provide new insights? Can the questions be answered given your
resources and timeline? Have the concerns of key users been
included?
![Page 16: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/16.jpg)
Identify key evaluation questions
Who wants to know what about this program?
![Page 17: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/17.jpg)
Evaluation Questions
Clarify your evaluation questions Make them specific What do you need to know vs. what you
would like to know Prioritize Check: Will answers to these questions
provide important and useful information?
![Page 18: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/18.jpg)
Components of a program
Situation Resources (Inputs) Outputs
– Activities – Participants
Outcomes– Chain of outcomes from short- to long- term
External Factors and Assumptions
![Page 19: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/19.jpg)
What is your purpose for evaluating?
We are conducting an evaluation of _____ (program name) because ______ in order to __________.
Example: We are conducting an evaluation of the Money Quest Program because we want to know to what extent youth who participate learn and use recommended money management skill in order to report program outcomes to our funder.
![Page 20: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/20.jpg)
![Page 21: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/21.jpg)
![Page 22: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/22.jpg)
![Page 23: Gathering Evidence of Impact: A Continued Conversation Jan Middendorf Cindy Shuman Office of Educational Innovation and Evaluation.](https://reader035.fdocuments.net/reader035/viewer/2022070308/551beb2f550346af588b63ac/html5/thumbnails/23.jpg)