Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden,...
-
Upload
dylan-potter -
Category
Documents
-
view
221 -
download
0
Transcript of Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden,...
![Page 1: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/1.jpg)
Earmark Grant Evaluation: An Introduction and Overview
May 19, 2005
Presented by:Jeff Padden, President
Public Policy Associates, Inc.119 Pere Marquette Drive
Lansing, Michigan 48912-1231(517) 485-4477
www.publicpolicy.com
![Page 2: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/2.jpg)
Presentation Topics The evaluation requirement for earmark
grants Evaluation overview – or – “Where’s the
upside?” Planning the evaluation The evaluation process for earmark grants Discussion
We’ll do clarifications on the fly, broader discussion at the end.
![Page 3: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/3.jpg)
The Evaluation Requirement
![Page 4: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/4.jpg)
Each grantee must …
Conduct or commission an evaluation
Submit evaluation plan
Use the evaluation template
Submit evaluation report shortly after completion of project activities
![Page 5: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/5.jpg)
Evaluation Overview– or –
“Where’s the upside?”
![Page 6: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/6.jpg)
Program evaluation is ...
The systematic collection of information about the subject of the evaluation
Used to make decisions about organization’s or program’s: Creation
Improvement
Effectiveness
![Page 7: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/7.jpg)
Evaluation is a mindset …
We are all evaluators
Evaluation is continuous
Evaluation looks forward, not just backward
Involves organizational learning
Means people working together
![Page 8: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/8.jpg)
Evaluation allows you to examine ...
What’s working well
What is not
How to improve
There is no bad news, only news!
![Page 9: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/9.jpg)
Evaluation requires comparison ...
of the same group over time• pre- and post-tests
• trends in community-level data
of two comparable groups at one point in time over time
of your group to a larger group county compared to state
![Page 10: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/10.jpg)
Our Approach: Utilization-Focused Evaluation
Focuses on intended uses and users
Is inherently participatory and collaborative by actively involving primary intended users in all aspects of the evaluation
Leads to ongoing, longer-term commitment to using evaluation logic and building a culture of learning in a program or organization
Symbiotic rather than parasitic
![Page 11: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/11.jpg)
Benefits of Evaluation
Program/organizational improvement Accountability to funders and others Planning Program description for stakeholders Public relations Fund raising Policy decision making
Evaluation has lots of upside!
![Page 12: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/12.jpg)
Planning the Evaluation
![Page 13: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/13.jpg)
Elements of the Evaluation Plan Who conducts the evaluation?
Internal or external?
Experienced or novice?
When do they do it? Along the way or after the fact?
How much do they do? The level of intensity must fit the project
Too much diverts resources, too little leaves unanswered questions
What exactly do they do? Six major steps
![Page 14: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/14.jpg)
Evaluation Steps
1. Clarify project & goals 2. Establish measures
3. Collect data
4. Analyze data
5. Prepare reports6. Improve project
![Page 15: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/15.jpg)
Step 1: Clarify Project & Goals
Thinking about goals What are you trying to accomplish?
What would success look like?
What is the difference between the current state of affairs and what you are trying to create?
Example of a goal statement:“Increase incomes of low-income families in the region through training for entry-level jobs that have career ladders leading to good jobs.”
![Page 16: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/16.jpg)
Does the Project Hang Together? Are the expected outcomes realistic?
Are there enough resources?
Do the customers like the product?
Does the organization have the right skills?
Logic models help answer these questions.
![Page 17: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/17.jpg)
A Simple Logic Model
Things needed to run the project:
People, resources, money, etc.
What you do:
Market, recruit, design, train, place, etc.
Direct results of activities:
Training completers,credentialsawarded, etc.
Changes caused by the project:
Jobs, wages, promotions, etc.
Inputs Activities Outputs Outcomes
![Page 18: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/18.jpg)
Step 2: Establish Measures
Determine performance measures Must be quantifiable
Data must be available, reliable, and valid
Examples of measures:
Activity: Number of training sessions
Output: Number of trainees
Outcome: Skill and credential gains
Impact: Stronger local workforce
![Page 19: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/19.jpg)
Step 3: Collect Data
Identify data sources, such as: Administrative records Surveys, interviews, focus groups Observation
Gather dataDesign the instruments and procedures for collection.Conduct data collection periodically.
Record data Organize data. Create database. Verify data.
Remember the measures!
![Page 20: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/20.jpg)
Step 4: Analyze and Interpret Data
Sort and sift: organize data for interpretation Cross-tabs
Modeling
Conduct data analysis to look for: Changes over time
Progress relative to goals or standards
Differences between groups
Test preliminary interpretation
This is the most creative step.
![Page 21: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/21.jpg)
Step 5: Prepare Reports
Determine reporting schedule
Report preliminary findings to key stakeholders and other audiences
Gather reactions
Incorporate reactions
Finalize reporting products
Different audiences need different types of reports.
![Page 22: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/22.jpg)
Step 6: Improve Project
Deliver reporting products internally.
Facilitate strategic and operational planning.
Improve processes and results.
A good evaluation will be more valuable to you than to DOL!
![Page 23: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/23.jpg)
The Evaluation Process for Earmark Grants
![Page 24: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/24.jpg)
Use the DOL Tools
“The Essential Guide for Writing an Earmark Grant Proposal”
“Evaluation Template for Earmark Grantees” (to be provided later)
![Page 25: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/25.jpg)
Discussion
![Page 26: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/26.jpg)
Thanks to …
… for the use of the “Demystifying Evaluation” materials.
Useful evaluation links:
• W.K. Kellogg Foundation: www.wkkf.org/Programming/Overview.aspx?CID=281
• American Evaluation Association: www.eval.org/EvaluationLinks/default.htm
• Western Michigan University Evaluation Checklists: www.wmich.edu/evalctr/checklists/checklistmenu.htm
![Page 27: Earmark Grant Evaluation: An Introduction and Overview May 19, 2005 Presented by: Jeff Padden, President Public Policy Associates, Inc. 119 Pere Marquette.](https://reader035.fdocuments.net/reader035/viewer/2022062313/56649cf45503460f949c189f/html5/thumbnails/27.jpg)
Earmark Grant Evaluation:An Introduction and Overview
May 2005
Presented by:Jeff Padden, President
Public Policy Associates, Inc.119 Pere Marquette Drive
Lansing, Michigan 48912-1231(517) 485-4477
www.publicpolicy.com