Promoting a Culture of Evidence and Use of Data through Program Evaluation Session Theme 2...
-
Upload
hugo-obrien -
Category
Documents
-
view
214 -
download
1
Transcript of Promoting a Culture of Evidence and Use of Data through Program Evaluation Session Theme 2...
Promoting a Culture of Evidence and Use of Data through Program
Evaluation Session Theme 2
Presentation to:
OSEP Project Directors’ ConferenceJuly 21, 2009
Presentation by:
Thomas Fiore, Ph.D.Project DirectorCenter to Improve Project PerformanceWestat
2
Session Theme 2
Evaluation tools and processes can help project directors do the best job with project planning, implementation, and accountability.
3
Presentation Outline
Presentation outline• Keeping a project focused• Logic model as the starting point• Evaluation plan• Evaluation design• Evaluation implementation• CIPP’s role providing general TA• CIPP contact information
4
Keeping a Project Focused
Project purpose is to do a good job of the right thing.• Need data to document the activities (outputs) of a
project.
• Need data to document the overall success or lack of success (outcomes) of a project.
• Most important, need data to determine whether innovations should be scaled up, changed, or abandoned.
• Need a tool to understand which data are important—that tool is a logic model.
5
Logic Model as the Starting Point
A logic model . . .
• Portrays a project’s overall plan;
• Clarifies the relationships among a project’s goals, activities, outputs, and outcomes; and
• Displays the connections between those defining features of a project.
6
Logic Model Structure
Create a coordination hub where the various TA&D centers funded by OSEP and other federal agencies can find and share information and resources, collaborate and problem-solve together.
Provide logistical support for coordination, communication and collaboration maintaining/
expanding workgroups
developing listservs
maintaining and updating an integrated events calendar
Number/types of workgroups created and maintained
Number/types of listservs and listserv participants
Up-to-date events calendar
Goals
Strategies/ Activities
Outputs
Direct Intermediate
Long-term
Outcomes
An effective single point of entry for network resources is implemented/maintained
Beneficial connections exist among Network participants
Successful ways of locating and sharing information and resources among network members and others are implemented/maintained
The capacity of the TA&D Network members to serve clients increases steadily
Inputs• OSEP funding • Project staff• Prior experience• Research-based
policy and practices
External Factors/Context: Other federal initiatives; OSEP policy environment; grantee’s accumulated experience and visibility.
7
Logic Model
• Thus, logic models can be used as a starting point to plan data collection and analysis aimed at measuring project processes and performance.
• Systematically measuring project processes and performance is evaluation.
• A logic model implies a causal relationship that flows from goals to outcomes.
• Evaluation can be viewed as a test of the logic model’s implied hypotheses of this causal relationship.
8
Evaluation Plan
From the logic model, develop a plan for collecting and analyzing data.
• Focus on outputs and outcomes
• Useful for formative [define] or summative purposes [define], or both
9
Evaluation Plan
Develop the specific plan by answering these questions:
• What data are required to demonstrate project effectiveness or to provide information on the overall program effectiveness?
• What are the strategies/activities that should be given priority—that is, which ones should be evaluated because they are important?
• Who are the targeted recipients of interest and in what settings?
10
Evaluation Plan
Develop the specific plan by answering these questions: [more]
• What are the evaluation questions?
• What data collection activities are needed?
• How will the data be analyzed?
• What are the necessary timelines, staff assignments, and cost allocations across years?
• How will this be documented and reported?
11
Evaluation Plan
Logic model leads to evaluation questions:
→ Relevant goals (not necessarily all)
→ Salient strategies/activities related to those goals
→ Outputs associated with the strategies/activities
→ Outcomes (the most consequential ones)
→ Evaluation questions
12
Evaluation Plan
Goal: Create a coordination hub…
Strategies/activities: Provide logistical support…
Outputs: Number/types of workgroups, etc.
Outcomes: Successful ways of locating and sharing information and resources…
Evaluation question: To what extent have activities supporting coordination, communication, and collaboration been effective in enabling TA&D Network members to do their work efficiently and without duplication?
13
Evaluation Design
Evaluation plan leads to an evaluation design:
→ Evaluation questions
→ Measurable outputs or outcomes
→ Methods that capture change
→ Types of data collection
→ Instruments
14
Evaluation Design
Evaluation question: To what extent have activities…been effective in enabling TA&D Network members to do their work efficiently and without duplication?
Measurable outcome: An integrated technology system is implemented and used.
Types of data collection: Record review, survey
Instruments: Record Review and Web Statistics Protocol, Annual TA&D Network Survey
15
Evaluation Design
Evaluation design continues with:
• Instrument development
• Sampling
• Data collection scheduling
16
Evaluation Implementation
Evaluation implementation requires:
• Data Collection
• Analysis
• Reporting
17
Summary
• Sounds complicated, but projects are doing much of this already.
• Best when integrated into the overall implementation of the project.
• Best when formative and summative evaluations are integrated.
• Doesn’t need to be comprehensive—don’t need to measure everything in the most rigorous way to have information that can be useful.
• Overall, evaluation answers the question of great interest to funders and to clients: What good is this doing?
18
CIPP’s Role
CIPP role is two-fold:
• To guide, coordinate, and oversee the summative evaluations of 12 large grant-funded projects selected by OSEP.
• To provide technical assistance to current OSEP grantees in conducting formative and summative evaluations of their projects funded through the following programs: Parent Information Centers, Technical Assistance and Dissemination, Personnel Development, and Technology and Media Services.
19
CIPP’s Role Providing General TA
Vehicles for accessing TA:• CIPP website (www.cippsite.org) provides an
opportunity for staff of OSEP-funded projects to pose questions or ask for assistance.◦ CIPP website contains information and reference
materials on formative and summative evaluation. ◦ Evaluation Briefs will be available that discuss
information requested with some frequency and that can be downloaded from the website.
• Toll-free telephone line (1-888-843-4101) also enables project staff to pose questions or ask for assistance.
20
CIPP Contact Information
Website: www.cippsite.org
Toll-free telephone: 1-888-843-4101