Program Evaluation In the Non-Profit Sector
description
Transcript of Program Evaluation In the Non-Profit Sector
The American Nonprofit Sector and Philanthropy IThe American Nonprofit Sector and Philanthropy I
Fall 2010Fall 2010 Program EvaluationProgram Evaluation
Kelly Wishart, MA, CECKelly Wishart, MA, CECAdministrator, Professional Development and QualityAdministrator, Professional Development and Quality
Children’s FriendChildren’s Friend
Providence, RIProvidence, RI
Overview of SessionsOverview of Sessions
11-23-10 Session I:11-23-10 Session I: Evaluation BasicsEvaluation Basics Current thinking about evaluationCurrent thinking about evaluation Uses of evaluationUses of evaluation When to evaluate and when not toWhen to evaluate and when not to Types of evaluationTypes of evaluation Vocabulary of evaluationVocabulary of evaluation Logic Models and Theory of Change ModelLogic Models and Theory of Change Model Creating Program Level OutcomesCreating Program Level Outcomes
11-30-10 Session II:11-30-10 Session II: Evaluation Planning and DesignEvaluation Planning and Design Creating Program Level OutcomesCreating Program Level Outcomes Data, Data, Data: So What, Now WhatData, Data, Data: So What, Now What Data Collection ToolsData Collection Tools Avoiding the DRIP: Data Rich Information Poor SyndromeAvoiding the DRIP: Data Rich Information Poor Syndrome
12-7-10 Session III:12-7-10 Session III: Student Presentations and FeedbackStudent Presentations and Feedback Identify and articulate at least three outcomes for a program with Identify and articulate at least three outcomes for a program with
which you have familiarity.which you have familiarity. Design at least one data collection instrument and describe how the Design at least one data collection instrument and describe how the
tool will be administered, how the data will be analyzed, used in any tool will be administered, how the data will be analyzed, used in any public reporting, etc.public reporting, etc.
Each student will present for not more than ten minutesEach student will present for not more than ten minutes
How do you know you know?How do you know you know?
““The quicker picker upper.”The quicker picker upper.”
BountyBounty
““We make a lot of the things We make a lot of the things you buy, better.”you buy, better.”
BASFBASF
Evaluation…the old days:Evaluation…the old days:
To ensure fiscal accountability (audits are To ensure fiscal accountability (audits are type of evaluation tool)type of evaluation tool)
Imposed by outsiders (funders say they Imposed by outsiders (funders say they want to know how their money was spent, want to know how their money was spent, did you do what you said you were going did you do what you said you were going to?)to?)
Conducted at the end of the program (did Conducted at the end of the program (did you do what you intended or set out to do?)you do what you intended or set out to do?)
Findings are usually used to answer simple Findings are usually used to answer simple questions and not for program decision-questions and not for program decision-making or program improvement.making or program improvement.
Evaluation today:Evaluation today:
Valued by those doing the work (direct Valued by those doing the work (direct line workers know if they are making an line workers know if they are making an impact).impact).
Looks at the process of the doing, as Looks at the process of the doing, as well as the end result. well as the end result.
Is part of the planning and Is part of the planning and implementation process and not an after implementation process and not an after thought; questions are asked before thought; questions are asked before implementation. implementation.
Organizations use the information to Organizations use the information to improve practice.improve practice.
What can evaluation tell you:What can evaluation tell you: Is the program worth doing?Is the program worth doing? How can you improve your program?How can you improve your program? How well does the program deliver How well does the program deliver
services?services? What effect does the program have?What effect does the program have? What do we need to know for decision-What do we need to know for decision-
making?making? How do program costs compare with How do program costs compare with
benefits?benefits? How does the program comply with How does the program comply with
particular standards?particular standards? What’s important to people about this What’s important to people about this
program?program?
In other words, types of In other words, types of evaluation:evaluation:
Process Outcome Formative Summative Cost-effectiveness Cost-benefit evaluation
Other common terms:Other common terms:
QualitativeQualitative QuantitativeQuantitative
Refer to the Refer to the types of informationtypes of information or data collected in an or data collected in an evaluation, not the type of evaluation, not the type of evaluationevaluation
Preparing for EvaluationPreparing for Evaluation
What do you want to know?What do you want to know?
Who will use, view the results?Who will use, view the results?
How will the information be How will the information be
used?used?
Who needs to be involved in Who needs to be involved in your evaluation?your evaluation?
Program ProvidersProgram Providers Current fundersCurrent funders Potential donorsPotential donors Program participantsProgram participants ColleaguesColleagues ResearchersResearchers PartnersPartners CompetitorsCompetitors Others…?Others…?
Program EvaluationProgram Evaluation
Program evaluation is a
systematic method for
collecting, analyzing, and using
information to answer basic
questions about a program. ACF Program Evaluation GuideACF Program Evaluation Guide
The basic steps:The basic steps:
Assemble an evaluation teamAssemble an evaluation team
Prepare for the evaluationPrepare for the evaluation
Develop an evaluation planDevelop an evaluation plan
Collect evaluation informationCollect evaluation information
Analyze your evaluation informationAnalyze your evaluation information
Prepare the evaluation reportPrepare the evaluation report
Program Objectives, 1:Program Objectives, 1:
Implementation objectives refer to what you plan
to do in your program, how you plan to do it, and
who you want to reach.
The services or training you plan to implement
The characteristics of the participant population
The number of people you plan to reach
The staffing arrangements and staff training
The strategies for recruiting participants.
Often referred to as implementation evaluationOften referred to as implementation evaluation
Program Objectives, 2:Program Objectives, 2:
Participant outcome objectives describe what
you expect to happen to your participants as a
result of your program.
Can refer to the agencies, communities, and
organizations as well as individuals.
Measure your expectations about how your
program will change participants' knowledge,
attitudes, behaviors, or awareness are your
participant outcome objectives.
Often referred to as an outcome evaluation. Often referred to as an outcome evaluation.
What kind of objectives are What kind of objectives are these:these:
Family safety will improve. Examples: prevention of
physical, sexual or emotional abuse of children;
child neglect; domestic violence.
Improve referral and service connections among
DCYF and community agencies.
Children will show greater engagement in learning
and early literacy activities.
True partnerships between target schools and local
communities are realized in the design,
implementation, and governance of the project.
Who evaluates?Who evaluates?
An outside evaluator (which may be an
individual, research institute, or consulting firm)
who serves as the team leader and is
supported by in-house staff.
An in-house evaluator who serves as the team
leader and is supported by program staff and
an outside consultant.
An in-house evaluator who serves as the team
leader and is supported by program staff.
How much does it cost?How much does it cost?
Most experts say expect the Most experts say expect the cost of evaluation to be about cost of evaluation to be about 15-20% of your total program 15-20% of your total program budget.budget.
Some funders will require a Some funders will require a minimum amount to be included minimum amount to be included in your budget.in your budget.
Who evaluates?Who evaluates?
Depends on your program’s Depends on your program’s
funding requirementsfunding requirements
Your resources and capabilitiesYour resources and capabilities
Decision time:Decision time:
Does your agency or program have funds designated for evaluation purposes?
Have you successfully conducted previous evaluations of similar programs, components, or services?
Are existing program practices and information collection forms useful for evaluation purposes?
Can you collect evaluation information as part of your regular program operations (at intake, termination)?
Are there agency staff who have training and experience in evaluation-related tasks?
Are there advisory board members who have training and experience in evaluation-related tasks?
Deciding what to evaluate.Deciding what to evaluate.
Evaluation Logic ModelEvaluation Logic Model
The logic model serves as a basis for identifying your program's implementation and participant outcome objectives.
Focus your evaluation on assessing whether implementation objectives and immediate participant outcome objectives were attained.
Deciding what to evaluate:Deciding what to evaluate:
Source: Measuring Program Outcomes: A Practical Approach, United Way of America (1996)
Key to good program evaluation is Key to good program evaluation is clearly articulated outcomes!clearly articulated outcomes!
Outcomes are benefits or changes for Outcomes are benefits or changes for individuals or populations during or after individuals or populations during or after participating in program activities. participating in program activities.
They are influenced by a program’s outputs.They are influenced by a program’s outputs. They may relate to behavior, skills, They may relate to behavior, skills,
knowledge, attitudes, values, condition, knowledge, attitudes, values, condition, status, etc. status, etc.
They are what participants know, think, or They are what participants know, think, or can do; or how they behave; or what their can do; or how they behave; or what their condition is, that is different following the condition is, that is different following the program.program.
Examples:Examples:
Financial management Financial management counseling programcounseling program OutputsOutputs = number of classes and = number of classes and
families seenfamilies seen OutcomesOutcomes = families living within = families living within
a budget, making additions to a budget, making additions to savings accounts, increased savings accounts, increased financial stabilityfinancial stability
Examples:Examples:
Neighborhood clean up Neighborhood clean up campaigncampaign Outputs = number of organizing Outputs = number of organizing
meetings held, number of meetings held, number of weekends dedicated to the workweekends dedicated to the work
Outcomes = reduced exposure to Outcomes = reduced exposure to safety hazards, increased feeling safety hazards, increased feeling of pride in neighborhoodof pride in neighborhood
What is not an outcome:What is not an outcome:
If it does not represent benefits to or If it does not represent benefits to or changes in participants.changes in participants.
If it tells nothing about whether or not If it tells nothing about whether or not participants benefitted from the participants benefitted from the service, e.g. number of participants service, e.g. number of participants reflects only volume.reflects only volume.
Participant satisfaction generally not Participant satisfaction generally not an outcome since it does not indicate an outcome since it does not indicate whether or not the participant's whether or not the participant's condition improved as a result of the condition improved as a result of the service.service.
ObjectivesObjectives
Describe what you plan to do in your program and how you expect the participants to change in a way will allow you to measure these objectives. What you plan to doWhat you plan to do Who will do itWho will do it Who you plan to reach and how manyWho you plan to reach and how many
S.M.A.R.T. ObjectivesS.M.A.R.T. Objectives
SpecificSpecific MeasurableMeasurable Action-orientedAction-oriented RealisticRealistic Time-boundTime-bound
Measurable Objectives:Measurable Objectives: Stating participant outcome objectives
in measurable terms requires you to be specific about the changes in knowledge, attitudes, awareness, or behavior that you expect to occur as a result of participation in your program.
Be able to answer the question: How will we know that the expected
changes occurred? To answer this question, you will have to
identify the evidence needed to demonstrate that your participants have changed.
AssignmentAssignment
In small groups review the In small groups review the program description.program description.
Using the Logic Model Using the Logic Model worksheet identify assumptions, worksheet identify assumptions, potential activities, outputs and potential activities, outputs and outcomes.outcomes.
Present to group.Present to group.