Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome...
-
Upload
sarah-andersen -
Category
Documents
-
view
222 -
download
2
Transcript of Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome...
![Page 1: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/1.jpg)
Program Evaluation
![Page 2: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/2.jpg)
Program Evaluation
Overview and Discussion of:• Objectives of evaluation• Process evaluation• Outcome evaluation• Indicators & Measures• Small group discussions
![Page 3: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/3.jpg)
Program Evaluation: Here we go again!
WHY WHY WHY do evaluation?• Time• Resources• Validity and reliability• Action
![Page 4: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/4.jpg)
Why Evaluate?
• Increase knowledge & understanding of the program being evaluated
• Identify specific problems/needs, as well as effective practices
• Develop capacity for ongoing assessment and improvement
• Evaluation is of little value unless it leads to further decisions and activities re: program improvement
![Page 5: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/5.jpg)
Why Evaluate?In the past…
Most often, the focus of evaluation was on what agencies did – how many clients did they see, how many clinical hours were spent, how many sessions did they deliver
Now…There is an increasing focus on what happens when agencies deliver services – what changes, what improves for the client
![Page 6: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/6.jpg)
Thorny Issues in Program Evaluation
• ?
• ?
• ?
• ?
• ?
• ?
![Page 7: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/7.jpg)
Thorny Issues in Program Evaluation
• Cost and human resources• Buy-in at all levels• Logic models that defy logic (overly
ambitious)• Process versus outcomes/impacts• Choosing the right indicators/measures• Moving data to action
![Page 8: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/8.jpg)
Capacity to do evaluation
Capacity to use evaluation
Human Resources•Staffing•Technical Skills•Communication Skills•Professional Development•Leadership
Organizational Resources•Budget•Ongoing Data Collection•Infrastructure
Evaluation Literacy•Results-Management Orientation•Involvement-Participation
Organizational Decision-Making•Management Processes•Decision Support
Learning Benefits•Instrumental-Conceptual Use•Process Use
{
{
Evaluation Planning and Activities•Evaluation Plan•Use of Consultants•Information Sharing•Organizational Linkages•External Supports
Framework of Evaluation Capacity, Isabelle Bourgeois. Do not cite without author’s permission.
![Page 9: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/9.jpg)
Ensuring support & buy-in
• Efforts to ensure support & buy-in for evaluation from staff & stakeholders should happen from the onset of deciding to conduct an evaluation
• Hold preliminary meetings• Clarify roles & responsibilities over the evaluation• Identify why the evaluation is being done; clarify any
misconceptions; provide reassurance• Identify who the end-user will be• Develop agreement on what is to be evaluated• Assess readiness for evaluation
![Page 10: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/10.jpg)
Logic Models
“And this is our new logic model!”
Keep it as simple and as focused as possible!
![Page 11: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/11.jpg)
Process Evaluations
Key Issues:• Service Delivery – the extent to which
program components are being delivered as planned
• Coverage – extent of target population participation in program
• Bias – extent to which subgroups of the designated target population participate (or don’t participate)
![Page 12: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/12.jpg)
Process Evaluations – Strengths & Weaknesses
Strengths:• Measures how well program meets client needs• Measures the extent to which the program was
implemented as intended• Ensures a consumer-driven program• Easy & inexpensive
Weaknesses:• Bias can be significant
o Few dissatisfied participants return surveyso Clients may report inflated levels of satisfactiono Very subjective
![Page 13: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/13.jpg)
Outcome Evaluations
Focus: program results, program effectivenessFocus: program results, program effectiveness
Examples of OutcomesExamples of Outcomes: : • Change in circumstances• Change in understanding/knowledge• Change in attitude• Change in behaviour
![Page 14: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/14.jpg)
What Are Some of Your Outcomes?How are you measuring them?
• ?• ?• ?• ?• ?• ?
![Page 15: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/15.jpg)
Indicators and Measures – Measurement StrategyOutcome Indicator(s) Source of data
(records, clients, caregivers, etc.)
Method to Collect Data
& Frequency
Who collects
data
When collects
data
EXAMPLE:Skill acquisition – self management, adaptive
Increase in scores on administered psychometric test, caregiver survey
Caregivers,Psychometric measures
Pre and post administered tests, surveys
Evaluation Researcher
At the beginning and end of program, and at six months post program
![Page 16: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/16.jpg)
Moving Results Into ActionUse results
• Develop action plans for improving program effectiveness
• Develop plans to disseminate and share findings – be innovative!
• Plans should be developed from the beginning and continue throughout
• Linkages & partnerships developed in the initial stages of the evaluation form the basis of plans for information sharing
![Page 17: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/17.jpg)
Centre of Excellence Activities
• Consultation services– Development of evaluation framework, logic
model, indicators and measures– Training or workshops– Facilitating or developing networks/ communities
of practice (e.g., Triple P)– Review of grant applications
• Evaluation grants
![Page 18: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/18.jpg)
Objectives – Evaluation Grants
• Build skills– Developing evaluation framework and logic model– Implementing evaluation plan and analyzing
results– Using evaluation findings
• Build support– Frequent communications between the Centre and
agencies– Engage internal and external stakeholders
• Build communities– Establishing communities of practice
![Page 19: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/19.jpg)
Communities of Practice – Triple P
• Nine agencies – key contacts for Triple P within Ontario
• Developing a common approach to evaluation – measuring same set of outcomes with common measures
• Facilitating a provincial roll-out of Triple P data
![Page 20: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/20.jpg)
Communities of Practice – Triple PWhat made it work?• Clearly identified purpose• Willingness to work together• Face-to-face meeting at strategic points• Communication portal to facilitate active
learning and sharing of ideas• Centre as facilitator
![Page 21: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/21.jpg)
Small Group Discussion
1. Think of a program in your agency. List all the external and internal stakeholders for the evaluation.
• What are some challenges in meeting the interests of all stakeholders in your program?
• What are some ways your program has been successful in getting buy-in from key stakeholders?
![Page 22: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/22.jpg)
Small Group Discussion
2. In your local area or region, what community of practice would you like to be involved in?
• What topics or issues would you like the community of practice to discuss and meet about?
• How can the Centre of Excellence help you in forming and maintaining this community of practice?
![Page 23: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/23.jpg)
Program Evaluation
Contact InformationContact Information:
Susan Kasprzak
Research Associate(613) 737-7600 ext. 3320
Tanya Witteveen
Research Associate
(613) 737-7600 ext. 3483
![Page 24: Program Evaluation. Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions.](https://reader035.fdocuments.net/reader035/viewer/2022062511/55150d595503465e608b4a5c/html5/thumbnails/24.jpg)
Visit our website for more information:
www.onthepoint.ca