Post on 12-Feb-2016
description
CECV Intervention Framework
Module 6 Evaluation
Purpose of this Module
As a result of participating in this module, you will:• Evaluate the effectiveness of the Intervention Framework in guiding
your school through the process of:
Identifying students with additional learning needs; Assessing students with additional learning needs; Analysing & interpreting the data collected; Designing & carrying out the teaching; and Evaluating & monitoring the student’s progress and the
effectiveness of the teaching.
2
Foundations of The Framework
3
Core Principles1. All students can succeed2. Effective schools promote a culture of learning3. Effective teachers are critical to student learning
success4. Teaching and learning is inclusive of all5. Inclusive schools actively engage and work in
partnership with the wider community6. Fairness is not sameness7. Effective teaching practices are evidence-based
4
“…research seeks to prove, evaluation seeks to improve…”M Q Patton (2003)
5
Focus Questions
• How do you define evaluation?
1. Why do you evaluate?2. What do you evaluate?3. For whom do you evaluate?4. How do you evaluate?
Educational Evaluation Around the World, Danish Evaluation Institute, 2003
6
How do you define
Evaluation?
7
In implementing any change it is necessary to evaluate the effect.
• In considering implementation of the Intervention Framework it is necessary to evaluate the effect on individual student outcomes and more broadly on teacher practice, teacher knowledge, school policies and processes.
8
Defining Evaluation
Defining Evaluation
Evaluation involves assessing the strengths and weaknesses of programs, policies,
personnel, products, and organizations to improve their effectiveness.
The American Evaluation Association
9
Defining Evaluation
Evaluation is the systematic collection and analysis of data needed to make decisions, a process in which most well-run programs
engage from the outset.
The American Evaluation Association
10
Defining Evaluation
Evaluation is about finding answers to questions such as,
“are we doing the right thing” and “are we doing things right?”
The American Evaluation Association
11
Defining Evaluation
Rossi and Freeman (1993) define evaluation as "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of ... programs." 12
Defining Evaluation• appraise• assess• critique• judge• justify• predict• prioritise• choose
Source: Anderson & Krathwohl, 2001
• monitor• select• rate• rank• prove• decide• conclude• argue
as cited in Biggs & Tang, 2007.
13
Steps in Evaluation
Step 1: Define what you hope to achieveStep 2: Collect data (pre & post)Step 3: Analyse the dataStep 4: Formulate conclusionsStep 5: Modify the program
14
Types of Evaluation
Process Evaluation Process Evaluations describe and assess the actual program materials and activities.
Outcome EvaluationOutcome Evaluations study the immediate or direct effects of the program on participants.
Impact Evaluation Impact Evaluations look beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.
15
Process Evaluation to Inform School Improvement
Phases of the process of improvement0 Preparation/ 1. Identification1 Diagnostic phase / 2. Assessment2 Strategic planning phase / 3. Analysis and
Interpretation3 Developmental phase / 4. Teaching and
Learning4 Evaluation phase / 5. Evaluation
R Bollen 199716
Outcome Evaluation
The ultimate goal of the Intervention Framework process is to improve student outcomes. How do you know whether it did?
• One commonly used way to find out whether the process (i.e. the T&L cycle) improved student outcomes is to ask whether the process caused the expected outcome.
• If the process caused the outcome, then one could argue that the process improved student outcomes.
• On the other hand, if the process did not cause the outcome, then one would argue that, since the process did not cause the outcome, then the process did not improve student outcomes.
17
Outcome Evaluation How to figure this out
Determining whether a process caused the outcome is one of the most difficult problems in evaluation, and not everyone agrees on how to do it. The approach you take depends on how the evaluation will be used, who it is for, what the evaluation users will accept as credible evidence of causality, what resources are available for the evaluation, and how important it is to establish causality with considerable confidence. Michael Quinn Patton
One way could be to evaluate the teaching programs implemented.
18
Impact Evaluation
Impact Evaluations look beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.
19
1. Why Evaluate?
20
Why Evaluate?
It is important to evaluate programs/the teaching for many reasons:– to ensure that the program is not creating any
unintended harm;– to determine if the program is making a positive
contribution (improved student outcomes); and– to improve and learn (i.e. to learn what were the
positive elements, how it can be replicated, how challenges can be overcome in the future and how to make the process sustainable).
21
Why Evaluate?
The four main reasons evaluation is conducted:accountability; learning; program management and development;ethical obligation.
Green and South, 2006.
22
2. What do you Evaluate?
23
3. For whom do you Evaluate?
24
4. How do you Evaluate?
25
What & How?
• How does your school evaluate its current programs?
• How would you evaluate whether the child/children progressed as a result of participation in this intervention process.
26
• Is the student progressing satisfactorily against the set goals?
• How will you monitor and interpret the student’s progress against the set goals?
• How will you evaluate the effectiveness of the program/approach?
27
What & How?
Making the results useful (student outcomes)
• How will you use the results to inform future program development for students?
• How will the results be reported so that they can be used by the school to make improvements?
28
What & How?
“…evaluation seeks to improve…”
29
Next Session - 2
30
EvaluationEffect Sizes
31
“evaluation seeks to improve”
32
Effect Sizes
1. What is an effect size?2. Why use effect sizes?
3. How can schools use effect sizes to evaluate the effectiveness the intervention?
33
Effect Sizes (d)
1a. What is an effect size?An effect size provides a common expression of the magnitude of study outcomes, across variables, such as improving reading levels in accuracy and comprehension.
An effect size of 1.0 indicates an increase of one standard deviation (1SD) on the outcome. One SD increase is typically associated with advancing students’ reading levels by two to three years, improving the rate of learning by 50%.
34
Effect Sizes (d)
1b. What is a reasonable effect size?Cohen (1988) suggests that:d = 0.2 is small, d = 0.5 is medium, d = 0.8 is large
Whereas the results from Hattie’s meta-analysescould suggest when judging educationaloutcomes:d = 0.2 is small, d = 0.4 is medium, d = 0.6 is large Reference: Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd ed.). Hillsdale, NJ: L. Erlbaum
Assoc.
35
John Hattie - Visible Learning
What is John Hattie on about, in a nutshell?
15 years of research800+ meta-analyses50,000 studies200+ million students
Outcome: What are the major influences on student learning?
36
Hattie’s Effect Sizes (d)
37
Effect Sizes (d)
The Formula
Effect size = Average (post) - Average (Pre) (d) Average Standard Deviation
(the spread)
38
Effect Sizes (d)
2. Why use effect sizes?• To compare progress over time on the
same test.• To compare results measured on different
tests.• To compare different groups doing the
same test.
39
Effect Sizes (d)
3. How can schools use effect sizes?
Discussion in school groups
40
EvaluationSelf Reflection
41
What is
Self Reflection?
42
Self Reflection
"Test all things; hold fast what is good"
I Thessalonians 5:21
43
What is Self Reflection?
Self Reflection is simply a form of self evaluation undertaken by participants in social situations in order to improve the rationality and justice of their own practices, their understanding of these practices, and the situations in which the practices are carried out. Adapted from Carr and Kemmis, 1986
44
Self Reflection in Schools
Self reflection is a process in which teachers examine their own educational practice systematically and carefully, using the techniques of action research.
45
Self Reflection Leads to Improvement?SELF REFLECTION
46
Follow effective action with quiet reflection.
Out of the quiet reflection will come even more
effective action.Peter. F. Drucker
47
Teachers account for about 30% of the variance in student achievement
“It is what teachers know, do, and care about, which is very powerful in this learning equation, not just what they know” (p. 2). They must put this knowledge into practice if they are to produce gains in student learning outcomes.
Hattie (2003)
48
Where to from here?
49
Where to from here?
Each school to reflect on:• Existing evaluation processes of:
– existing intervention programs currently in use in the school.
– teacher performance– Student performance
50
In your Context
Has this process highlighted the need to review your school’s policies and/or processes?
51
In your Context
• Has this process made a difference to your students’ performances?
Evaluation of change is fundamental to the process.
52
“When ignorance is bliss, tis folly to be
wise”Thomas gray from ‘Ode on a Distant Prospect of Eton College’.
53
“self reflection seeks to improve”
54
References
• Educational Evaluation Around the World: An International Anthology• http://english.eva.dk/publications/educational-evaluation-around-the-world-an-internatio
nal-anthology/download• An Education Primer: An overview of education evaluation
http://www2.ed.gov/offices/OUS/PES/primer1.html
• Evaluation definition: http://www.evaluationwiki.org/index.php/Evaluation_Definition
• Evaluation Toolkit: http://mypeer.org.au/monitoring-evaluation
• Introduction to program Evaluation: http://www.cdc.gov/tb/programs/Evaluation/Guide/Webinar/Eval_101_1_AP22.ppt
• What is Program Evaluation? A beginner’s guide: http://gsociology.icaap.org/methods/evaluationbeginnersguide.pdf
• Patton, M Q 2002, Qualitative research & evaluation methods (3rd ed.), Thousand Oaks, CA: Sage Publications
55