Evaluation Plans & Performance Indicators Office of Research, Evaluation, and Policy Studies...
-
date post
21-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of Evaluation Plans & Performance Indicators Office of Research, Evaluation, and Policy Studies...
Evaluation Plans &Performance Indicators
Office of Research, Evaluation, and Policy Studies
Marcella M. Reca Zipp
November 30, 2010
Necessity of evaluation plan Types of evaluation plans Components of an evaluation plan Performance indicators Reporting requirements Sample evaluation plans
Purpose of Evaluation Plan
Cohesive approach to conducting evaluation and using results
Explains what, when, how, why, who Documents the evaluation process Ensures implementation fidelity Promotes a participatory approach
Source: University of Toronto
3 Levels of Evaluation
Project-Level Evaluation Context Implementation Outcome
Cluster Evaluation Program and Policymaking Evaluation
Project-Level Evaluation Context
Needs, assets, and resources of community Political atmosphere, social and environmental
strengths/weaknesses Implementation
Critical components/activities of project Aspects that are strengths and weaknesses How do the components connect to goals and
outcomes Outcome
Critical outcomes you are trying to achieve Impact on clients, community, etc. Unexpected impact
Cluster Evaluation Determines how well the
collection of projects fulfills the objective of systemic change.
Not a substitute for project-level evaluation.
Looks across a group of projects to identify common themes.
Information reported in aggregate form to granting organization.
EVALUATOR
PROJECT A PROJECT B PROJECT C
Program and Policy Making Evaluation
Macro form of evaluation. Utilizes information gathered from
both project-level and cluster evaluation to make effective decisions about program funding and support.
Supports communities in creating policy change at local, state, and federal levels.
Elements of an Evaluation
Introduction Project Objectives Logic Model Partnership Roles and Responsibilities Intervention Programming/Research Methodology/Data Collection Instrumentation Measures
Introduction Provides background information for
the evaluation, identifies purpose and goal, sets the course on the evaluation road map. Evaluation purpose and goals
What does the evaluation strive to achieve? Evaluation team
Who is the evaluation coordinator? Who are the members of the evaluation
team?
Logic Model
Graphic depiction of the program description.
Links needs, objectives, activities, measurements.
Provides scope of program. Ensures systematic decisions are
made about what will be measured. Identifies and organizes indicators.
Objectives PIMO method The number of objectives will be
determined by purpose (i.e., intervention, treatment, prevention).
Interrelated to your projected activities (i.e., education, service, research).
Feasible to collect, will provide accurate results.
Partnership Roles and Responsibilities
Project partners are expected to provide certain, unique expertise to your project activities, either in direct service function or as advisory units.
Identify each partner in their role and responsibility in terms of involvement for your project.
Partner-cited activities must be evaluated formatively and summatively.
Intervention Programming Identify one or more intervention strategies
used to support project activities and anticipated outcomes.
Cite if the program is on the federal evidence-based initiative list: EBI
For market-available programs that require training and certification of direct service providers, provide a timetable for acquiring training before intervention can be used.
Performance Indicators
Visible, measurable signs of program performance.
Relevant, understandable and useful. Reflect program objectives, logic
model and evaluation questions. Define success Reasonable expectations of program
performance.Source: University of Toronto
Performance Indicators cont.
Other terms – industry jargon Key Performance Indicator (KPI) Performance metric Performance standard Balanced Scorecard Quality indicators
All are different words for the same thing: measure performance.
Data Collection What methods will be used? How often will data be
collected? Who will collect the data? Validity and reliability of
data sources Baseline data Outcomes-based triangulation Quality assurance Design (experimental, quasi-
experimental, etc.)
Instrumentation Measures Tools for data collection
Only collect the information you need Easy to administer and use Pilot test tools before use in the
evaluation Human Subjects Considerations
IRB, school board approval Data management and storage
Confidentiality and data quality
Tips & Helpful Hints
Be realistic In your assessment of resources In your timeline
Seek help Use templates, tables, or guides that
may be provided in the RFP or model after past funded proposals.
Reporting and Dissemination Dissemination
How will you disseminate findings? Who is responsible? How, where, when will findings be used?
Reporting Formative reports – quarterly, biannually Summative reports – final report/end of
project Project deliverables
Sample RFP Evaluation Plan
Two examples of an evaluation plan within an RFP General, limited specifications Complex, very detailed
Evaluation Resources CDC: www.cdc.gov/eval University of Toronto: www.utoronto.ca/shp/hcu W.K. Kellogg Foundation:
www.wkkf.org/Publications/evalhdbk
Connell, J.P., Kubisch, A.C., Schorr, L.B., Weiss, C.H. (1995). New Approaches to Evaluating Community Initiatives, New York, NY: Aspen Institute.
Shadish, W.R., Cook, T.D., Leviton, L.C. (1991). Foundations of Program Evaluation. Newbury Park, CA: Sage Publications.
Taylor-Powell, E., Steele, S., Douglas, M. (1996). Planning a Program Evaluation. Madison, WI: University of Wisconsin Cooperative Extension.