Gathering Feedback for TeachingCombining High-Quality Observations with
Student Surveys and Achievement Gains
Closing the effectiveness gap
2
Progressing beyond “The Plateau”
3
Ensuring Reliable & Trustworthy ObservationsEnsuring Reliable & Trustworthy Observations
The First of Three QuestionsHow many distinct levels of teacher performance do you think an evaluation system should recognize?
5
The Second of Three QuestionsWhat performance levels would you assign to what fraction of your district’s teachers on the following competencies?A.Classroom Management (time, behavior, materials)
B.Goals & Tasks (clear, appropriate, rigorous, interesting)
C.Supporting Student Understanding (content depth, feedback, questioning and discussion, instructional dialogue)
6
The Third of Three QuestionsHow closely associated are teaching behaviors with student outcomes (academic growth over time)?
7
Stud
ent P
erfo
rman
ce
Teacher’s Classroom Observation Score
highly
moderately
weakly
8
Validation
Teachers with Higher Observation Scores Had Students Who Learned More
Observation Score DistributionsFramework for Teaching
9
PLATO Prime, CLASS and MQI Lite
10
Observation Score Distributions UTeach Observation Protocol
11
12
13
14
Measure Predictive power ReliabilityPotential for
Diagnostic Insight
Value-added
Student survey
Observation
Measures have different strengths …and weaknesses
15
16
The Importance of Multiple Measures
Compared to What?
Compared to MA Degrees and Years of Experience, the Combined Measure Identifies Larger Differences
17
… on state tests
Compared to What?
…and on low stakes assessments
18
Compared to What?
…as well as on student-reported outcomes.
19
The Measures of Effective Teaching ProjectThe Measures of Effective Teaching Project
Two school years: 2009–10 and 2010–11 >100,000 students Grades 4–8: ELA and Math High School: ELA I, Algebra I and Biology
Participating Teachers
Research PartnersResearch Partners
22
The MET Project is ultimately a research project. Nonetheless, participants frequently tell us they have grown professionally as a result of their involvement. Below is a sampling of comments we received.
From Teachers:•“The video-taping is what really drew me in, I wanted to see not only what I’m doing but what are my students doing. I thought I had a pretty good grasp of what I was doing as a teacher, but it is eye opening … I honestly felt like this is one of the best things that I have ever done to help me grow professionally. And my kids really benefited from it, so it was very exciting.”
•"With the videos, you get to see yourself in a different way. Actually you never really get to see yourself until you see a video of yourself. I changed immediately certain things that I did that I didn't like.”
•“I realized I learned more about who I actually was as a teacher by looking at the video. I learned of the things that I do that I think that I’m great at I was not so great at after all.”
•“Even the things I did well, I thought, ok that's pretty good, why do I do that, and where could I put that to make it go farther. So it was a two-way road, seeing what you do well, and seeing the things that have become habits that you don't even think about anymore."From Raters:•“Being a rater has been a positive experience for me. I find myself ‘watching’ my own teaching more and am more aware of the things I should be doing more of in my classroom.”•“I have to say, that as a teacher, even the training has helped me refine my work in the classroom. How wonderful!”•“I have loved observing teachers, reflecting on my own teaching and that of the teachers teaching in my school.”
What the Participants Said….
MET Extension: A Library of Teaching Practice
Additional Data Collection•Subset of 360 MET Teachers (disproportionately highly effective)•50 lessons taped (18,000 total lessons)•100% teacher & parental consent (allowing for broader public use)•Cheaper cameras with potential for scale
Library of Practice•Searchable database (tagged by Common Core standards, teaching practices, etc.)•Tagging to be done in partnership with schools of education (w/ teachers-in-training)
Potential Uses:•Rater training, certification, and calibration•School districts – professional development•Teacher training institutions – teaching methods•Observation instrument developers – validation of new & existing tools
MET Logical SequenceMET Logical Sequence
24
Measures predict
Measures predict
Measures fairly reflect teacherMeasures fairly reflect teacher
Measures combine
Measures combine
Effective Teaching Index
Effective Teaching Index
Teaching Effectiveness Dashboard
Teaching Effectiveness Dashboard
Research Use
Measures reliable
Measures reliable
Measures improve
effectiveness
Measures improve
effectiveness
Measures stable under pressure
Measures stable under pressure
Measures communicated
effectively
Measures communicated
effectively
Validation EngineValidation Engine
System picks observation rubric & trains raters
Raters score MET videos of instruction
Software provides analysis of:•Rater consistency•Rubric’s relation to student learning
26
Four Steps
Four Steps to High-Quality Classroom Observations
27
Step 1: Define ExpectationsFramework for Teaching (Danielson)
28
Four Steps
Uns
atisf
acto
ry
Yes/no Questions, posed in rapid succession, teacher asks all questions, same few students participate.
Basic
Some questions ask for student explanations, uneven attempts to engage all students.
Profi
cien
t
Most questions ask for explanation, discussion develops/teacher steps aside, all students participate.
Adva
nced
All questions high quality, students initiate some questions, students engage other students.
Step 2: Ensure Accuracy of Observers
29
Four Steps
Step 3: Monitor Reliability
30
Four Steps
31
Four Steps
Multiple ObservationsLeads to Higher Reliability
NOTES: The numbers inside each circle are estimates of the percentage of total variance in FFT observation scores attributable to consistent aspects of teachers’ practice when one to four lessons were observed, each by a different observer. The total area of each circle represents the total variance in scores. These
estimates are based on trained observers with no prior exposure to the teachers’ students, watching digital videos. Reliabilities will differ in practice. See the research paper, Table 11, for reliabilities of other instruments.
32
Students with Most Effective Teachers Learn More in School
Student Perceptions
33
CareCare
ControlControl
ClarifyClarify
ChallengeChallenge
CaptivateCaptivate
ConferConfer
TestPrepTestPrep
ConsolidateConsolidate
CareCare
• My teacher makes me feel that s/he really cares about me
• My teacher seems to know if something is bothering me
• My teacher really tries to understand how students feel about things
• My teacher makes me feel that s/he really cares about me
• My teacher seems to know if something is bothering me
• My teacher really tries to understand how students feel about things
ControlControl
• Students in this class treat the teacher with respect
• My classmates behave the way the teacher wants them to
• Our class stays busy and doesn’t waste time
• Students in this class treat the teacher with respect
• My classmates behave the way the teacher wants them to
• Our class stays busy and doesn’t waste time
ClarifyClarify
• If you don’t understand something , my teacher explains it a different way.
• My teacher knows when the class understands, and when we do not.
• My teacher has several good ways to explain each topic that we cover in the class.
• If you don’t understand something , my teacher explains it a different way.
• My teacher knows when the class understands, and when we do not.
• My teacher has several good ways to explain each topic that we cover in the class.
ChallengeChallenge
• My teacher asks students to explain more about the answers they give.
• My teacher doesn’t let people give up when the work gets hard.
• In this class, we learn to correct our mistakes.
• My teacher asks students to explain more about the answers they give.
• My teacher doesn’t let people give up when the work gets hard.
• In this class, we learn to correct our mistakes.
CaptivateCaptivate
• My teacher makes learning enjoyable• My teacher makes learning interesting• I like the way we learn in this class
• My teacher makes learning enjoyable• My teacher makes learning interesting• I like the way we learn in this class
ConferConfer
• My teacher wants us to share our thoughts• Students get to decide how activities are done in
this class
• My teacher wants us to share our thoughts• Students get to decide how activities are done in
this class
ConsolidateConsolidate
• My teacher takes the time to summarize what we learn each day
• The comments that I get on my work in this class help me understand how to improve
• My teacher takes the time to summarize what we learn each day
• The comments that I get on my work in this class help me understand how to improve
TestPrepTestPrep
• I have learned a lot this year about [the state test]• Getting ready for [the state ] test takes a lot of
time in our class
• I have learned a lot this year about [the state test]• Getting ready for [the state ] test takes a lot of
time in our class
Student Perceptions Top 5 Correlations
34
Survey StatementSurvey Statement CategoryCategoryRankRank
11
22
33
44
55
• Students in this class treat the teacher with respect• Students in this class treat the teacher with respect
• My classmates behave the way my teacher wants them to• My classmates behave the way my teacher wants them to
ControlControl
ControlControl
ControlControl• Our class stays busy and doesn’t waste time• Our class stays busy and doesn’t waste time
ChallengeChallenge• In this class, we learn a lot every day• In this class, we learn a lot every day
ChallengeChallenge• In this class, we learn to correct our mistakes• In this class, we learn to correct our mistakes
3333 • I have learned a lot this year about [the state test]• I have learned a lot this year about [the state test] Test PrepTest Prep
3434 • Getting ready for [the state test] takes a lot of time in our class• Getting ready for [the state test] takes a lot of time in our class Test PrepTest Prep
Dynamic Trio
Combining Measures Improved Reliabilityas well as Predictive Power
35
Note: For the equally weighted combination, we assigned a weight of .33 to each of the three measures. The criterion weights were chosen to maximize ability to predict a teacher’s value-added with other students. The next MET report will explore different weighting schemes.
FFT alone
Tripod alone
VA alone
Combined(Equal Weights)
Combined(Criterion Weights)
.05
.1.1
5.2
.25
Diff
ere
nce
in M
ath
VA
(T
op
25
% v
s. B
ott
om
25
%)
0 .1 .2 .3 .4 .5 .6 .7Reliability
Note: Table 16 of the research report. Reliability based on one course section, 2 observations.
The Reliability and Predictive Power of Measures of Teaching:
Top Related