Program Evaluation EDL 832 Jeffrey Oescher, Instructor 6 June 2013.
-
Upload
rosemary-henderson -
Category
Documents
-
view
224 -
download
1
Transcript of Program Evaluation EDL 832 Jeffrey Oescher, Instructor 6 June 2013.
Introduction
Objectives Describe the course Understand why evaluation is important Define evaluation Differentiate evaluation and research Discuss the jargon of evaluation
Who Is He?
Jeffrey Oescher Background Current responsibilities Experience as an evaluator Important things
Course Description
Catalogue description Rationale and goals Course organization Required text Course assignments Class schedule Important dates Moodle acknowledgment
The Need for Evaluation
What programs are currently being used in your school? Which ones are working well? Poorly? Are different parts of a program working better than
others? What can be done to improve those parts that are not contributing effectively?
What are the relative costs and benefits of these programs?
Have these programs been planned carefully? Is more planning needed?
What adaptations might make the program more effective?
Why is a program effective?
Evaluation Defined
OK, so there is a need to evaluate, but what does that term mean?
How would you define the term evaluation?
Evaluation Defined
There are many definitions of this term even among experts in the field.
The identification, clarification, and application of defensible criteria to determine an evaluation object’s value (i.e., worth and merit) in relation to those criteria. Fitzpatrick, Sanders, and Worthen, 2011
Evaluation Defined
Evaluation uses inquiry and judgment to determine standards for judging quality
relatively or absolutely; collect relevant information; apply the standards to determine value,
quality, utility, effectiveness, or significance; and
offer recommendations intended to optimize the evaluation object in relation to its purpose or help stakeholders make decisions.
Evaluation and Research
Purpose Research
Add knowledge to the field through the development or validation of theory
Establish cause and effect relationships Seek conclusions
Evaluation Help those who hold a stake in whatever is
being evaluated make decisions Describe aspects of the evaluation object Seek judgments
Evaluation and Research
Setting the agenda Research
The researcher sets the agenda through the choice of phenomena or hypotheses being investigated
Evaluation The stakeholders set the agenda by
determining the focus of the evaluation
Evaluation and Research
Standards used to judge adequacy Research
Validity Internal External
Reliability Evaluation
Accuracy – information is an accurate reflection of reality
Utility – the extent to which the results serve the information needs of the stakeholders
Feasibility – the extent to which the evaluation is realistic, diplomatic, prudent, etc.
Propriety – the extent to which the evaluation is done legally and ethically
Evaluation and Research
Preparation Research
Researcher’s training is focused on methodology within the specific discipline of study
Evaluation Evaluator’s training and expertise is inter-
disciplinary
Roles and Activities
Roles Researcher
Conduct the research according to plan Evaluator
Advocate for the use of evaluation Interact with stakeholders
Critical friend to help stakeholders learn Help develop critical judgment Help develop organizational learning Stimulate dialogue among various groups
Roles and Activities
Activities Researcher
Conduct the research (i.e., sample; structure the study; and collect, analyze, and interpret data)
Evaluator Negotiate with stakeholder groups to define the purpose
of an evaluation Develop contracts Hire and oversee staff Manage budgets Work with advisory groups Collect and analyze data Communicate with stakeholders Write reports Disseminate information
The Jargon of Evaluation
Formative The primary purpose is program
improvement Summative
The primary purpose is to provide information to help make decisions about program adoption, continuation, or expansion
The Jargon of Evaluation
Identify a specific situation where you would conduct a formative evaluation? A summative evaluation?
How do you see formative and summative evaluation interacting in the evaluation of a program in your school?
See Figure 1.1, p. 24
The Jargon of Evaluation
Needs assessment Establish whether a problem or need exists
and, if so, describe that need or problem as well as make recommendations for ways to reduce the need or correct the problem
Process Describe how a program is delivered
Outcome Describing, exploring, or determining changes
that occur in program participants See Table 1.3, p. 27
The Jargon of Evaluation
Internal evaluations Conducted by program employees
External evaluations Conducted by outsiders
Advantages of either approach See Figure 1.2, p. 31
Using the Jargon
An internal, formative evaluation An external, formative evaluation An internal, summative evaluation An external, summative evaluation
Where to Now?
Course Review the materials provided for you
The text Chapter 1
Review this material in light of our discussion tonight Chapter 2
Read this chapter to appreciate the history of evaluation so you can understand the current context in which evaluators operate
Chapter 3 Read this chapter to become familiar with some of the
issues that can become problematic when conducting an evaluation
Goal 1 Goal 2