BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 1
The Marzano Model: How Much Does
It Really Matter?
Dr. Lindsey Basileo and Dr. Robert Marzano
Learning Objectives
1. Understand how your district or school
compares to others who implement the
Marzano Teacher Evaluation Model
2. Understand the relationship between teacher
observation scores and value-added metrics
(VAM)
3. Understand which elements are more highly
correlated to VAM
iObservation User?
• Open Explorer Internet Browser
– Do NOT use Chrome
• Log into your iObservation Account to follow
along
• www.effectiveeducators.com/login/auth
• Organizational/Principals Users
– Reports
– Search for Dashboards
– Building Leadership- District View/ Building
Leadership
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 2
iObservation User?
1. Organization/building
2. Start Date = 08/01/2014
3. End Date= 07/31/2015
4. Select the Domain 1 Forms that Apply to that School YearIf more than one version select V3 or V2
5. Select Domains 2, 3, 4 Forms
6. Compared Start Date = 08/01/2013
7. Compared End Date= 07/31/2014
8. Select the Domain 1 Forms that Apply to that School YearIf more than one version select V3 or V2
9. Select Domains 2, 3, 4 Forms for Comparison Date Range
10. Evaluative Option: Count toward annual evaluation
11. Click Create
BACKGROUND
iObservation™
Technology Platform
• Supports schools and educational initiatives in
– 11 countries
– 3 provinces in Canada
– 43 states in the United States
• Serves over 420 school districts
– 5,500 school buildings
– 311,000 teachers
– 4 million students
iObservation collects millions of records
each year on observations
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 3
iObservation Customers
Across the Nation
There are iObservation customers in 43 states
iObservation Customers
Across the Nation
• Florida has the largest
number of customers that
use iObservation
• 40% of the districts in
Florida implement the
Marzano Teacher
Evaluation Model
THE DATA
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 4
Observation Data
• Three years of data were analyzed in the
State of Florida to asses scoring trends
SCORING FREQUENCY
Evaluative Score Distribution By
Lesson Segment
The Lesson Segment, Addressing Content, was observed most
often in the 2014-15 and 2013-14 school years
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 5
Evaluative Distribution By
Design Question
Design Questions 2 and 5 were observed most often
Distribution of Elements Observed
Addressing Content
12%
11%
13%12%
7%6%
7%
5%
7%
3%
5%
2% 2%
4%
1% 1% 1% 1%
14%
13%
11%
10%9%
7%
5% 5% 5%
4% 4% 4%
2% 2% 2%1% 1% 1%
0%
2%
4%
6%
8%
10%
12%
14%
16%
2012-13 2013-14 2014-15
Florida Distribution
Addressing Content (Design Questions 2, 3 & 4)
Providing resources and guidance for cognitively complex tasks consistently had the highest score
Element Count Freq. Avg. Count Freq. Avg. Count Freq. Avg.
Helping Students Practice Skills, Strategies & Processes 73,949 12% 3.11 94,559 14% 3.12 82,979 14% 3.09
Reviewing Content 68,165 11% 3.04 89,049 13% 3.04 76,889 13% 3.00
Identifying Critical Content 82,705 13% 3.07 73,558 11% 3.09 69,098 11% 3.01
Chunking Content into Digestible Bites 77,543 12% 3.07 68,285 10% 3.11 60,982 10% 3.07
Organizing Students to Practice & Deepen Knowledge 43,541 7% 3.12 60,548 9% 3.13 55,825 9% 3.10
Helping Students Record & Represent Knowledge 41,036 6% 3.06 40,344 6% 3.09 43,578 7% 3.08
Organizing Students to Interact with New Content 47,620 7% 3.09 37,497 6% 3.13 33,696 5% 3.08
Helping Students Process New Content 35,238 5% 3.02 27,721 4% 3.03 32,038 5% 3.01
Previewing New Content 47,136 7% 3.02 39,136 6% 3.04 31,637 5% 3.02
Helping Students Examine Similarities & Differences 17,558 3% 3.09 25,138 4% 3.09 23,541 4% 3.09
Helping Students Elaborate on New Content 32,630 5% 3.06 28,334 4% 3.06 23,310 4% 3.06
Helping Students Examine Their Reasoning 12,868 2% 3.02 18,607 3% 3.03 22,650 4% 3.07
Helping Students Reflect on Learning 14,778 2% 3.00 14,080 2% 3.02 12,544 2% 3.06
Providing Resources & Guidance for Cognitively Complex
Tasks22,474 4% 3.22 21,731 3% 3.27 12,237 2% 3.29
Helping Students Revise Knowledge 6,155 1% 3.07 9,756 1% 3.05 12,156 2% 3.08
Engaging Students in Cognitively Complex Tasks 6,604 1% 3.25 9,533 1% 3.21 7,104 1% 3.27
Using Homework 6,577 1% 3.05 7,876 1% 3.09 6,819 1% 3.10
Organizing Students for Cognitively Complex Tasks 5,135 1% 3.17 7,107 1% 3.21 6,350 1% 3.27
Florida Distribution
2012-13
Florida Distribution
2013-14
Florida Distribution
2014-15
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 6
Turn & Talk:
What are some similarities and
differences that you see?
OBSERVER
PERFORMANCE
Observation Count for Teachers in
Domain 1
The majority of teachers received 3 observations in the 2014-15 school year
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 7
Number of Scored Elements for
Teachers ─ Domain 1
The majority of teachers had 20 elements scored in the 2014-15 school year
Turn & Talk:
Do you have the same number of
observations and/or scored
elements?
SCORE DISTRIBUTION
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 8
Score Distribution in Domain 1
The majority of scores were at the applying level for all three years
Average Teacher Score in Domain 1
The average score for teachers was 3.07 in 2012-13, 3.10 in 2013-14 and
3.09 in 2014-15
14,529
4 0 1 5 6 5 7 8 16 21 43 23 40 73 74 97104
143184
231370
395458 770
9081,228
1,8152,188
3,731
5,587
10,075
7,308
5,076
4,502
3,3352,717
2,193
1,658
1,5291,287
1,197
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
0.0
0
0.1
0
0.2
0
0.3
0
0.4
0
0.5
0
0.6
0
0.7
0
0.8
0
0.9
0
1.0
0
1.1
0
1.2
0
1.3
0
1.4
0
1.5
0
1.6
0
1.7
0
1.8
0
1.9
0
2.0
0
2.1
0
2.2
0
2.3
0
2.4
0
2.5
0
2.6
0
2.7
0
2.8
0
2.9
0
3.0
0
3.1
0
3.2
0
3.3
0
3.4
0
3.5
0
3.6
0
3.7
0
3.8
0
3.9
0
4.0
0
2012-13 2013-14 2014-15
Turn & Talk:
How does your score
distribution and average
teacher score compare?
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 9
DO TEACHERS WITH
HIGHER OBSERVATION
SCORES HAVE HIGHER
VAM SCORES?
Matching Procedure
• Teachers were matched to the VAM scores
by first and last name, building and district
School Year% of Teachers
with VAM
Count of
Teachers with
VAM
Total
Teachers
2014-15 20.6% 12,248 59,412
2013-14 21.2% 12,379 58,520
2012-13 21.2% 13,316 62,742
Valued-Added Metrics
• Florida Department of Education’s Aggregated
VAM Score (1 Year)
– Standard score
– Student weighted average of the Teachers VAM
estimates divided by the average student growth for
that grade and subject
– 3 years of data used (2014-15, 2013-14 and 2012-13)
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 10
Correlation Results
Average Observation Score &
Aggregated 1 Year VAM
• Positive and small correlations were found for 2014-15 and 2013-14
• These compares with a 2013-14 pilot study where observers and teachers
were trained on the model
Observation
ScoreRead VAM
Math
VAM
Combined
VAM
Observation
Score1.00 .173** .226** .199**
N 59,412 9,888 6,624 12,248
Observation
Score1.00 .150** .208** .186**
N 58,520 10,245 6,750 12,379
Observation
Score1.00 .145** .185** .173**
N 62,742 10,727 7,192 13,316
2013-14
2014-15
2012-13
Observation
Score1.00 .168 .444** .239*
N 249 61 40 75
PCS
2013-14
Correlation Coefficients
Corrected for Measurement Error
• Once corrected for measurement error,
coefficients increased in size
Read VAM Math VAMCombined
VAM
2014-15 .232 .303 .267
2013-14 .201 .279 .250
2012-13 .195 .248 .232
PCS
2013-14.225 .596 .321
Correlation Results:
Compared with Pilot Findings
• Out of the 26 assessments that had a control
group match, 21 showed positive and significant
growth for students at treatment schools (p < .10)
• Independent sample t-tests showed that students
who attended treatment schools had
significantly increased growth scores (.32
standard deviations above prediction) compared to
students at control schools, accounting for student
characteristics (Hedges’ g= .35)
• Fixed effects models showed similar results as
independent sample t-tests
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 11
Turn & Talk:
How could you improve your
correlations in your data?
DO TEACHERS’
OBSERVATION SCORES
PREDICT THEIR VALUE-
ADDED SCORE?
OLS Regression Model
• OLS regression was used to assess whether
teachers’ observation score predicted their VAM
controlling for teacher characteristics
• Teacher characteristics included:
– Education
– Gender
– Race
– Ethnicity
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 12
Teacher Observation Score Predicting
Aggregated Year 1 VAM
• The teacher
observation score
was the largest
predictor in the
model (β=.161)
• This held true
across subjects
and years
Variable Name β SE p-value
Constant - .078 .000
Education .024 .013 .043
Black -.056 .022 .000
Hispanic .000 .027 .978
Female .032 .024 .006
Observation Score .161 .021 .000
R2
N
2014-15 VAM Combined
.034
12,248
WHICH ELEMENTS ARE
MORE HIGHLY
CORRELATED TO VAM?
Correlations Between Average
Teacher Element Scores &
Aggregated VAM
• Each teacher had an average score per element
they were observed on and these were correlated
with Aggregated VAM estimates
• Correlation coefficients between average element
score and combined VAM were significant all years
Min. Max. Avg.
2014-15 .061 .200 .136
2013-14 .064 .186 .122
2012-13 .060 .169 .118
Avg. .062 .185 .125
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 13
Highest VAM Correlations
The table shows the largest correlation coefficients by element
using the combined VAM from 2014-15
2014-15Read
VAM
Math
VAM
Combined
VAM
Applying Consequences for Lack of Adherence to Rules and Procedures*** .171 .224 .200
Helping Students Practice Skills, Strategies, and Processes*** .149 .182 .173
Organizing Students to Interact with New Content .161 .159 .172
Organizing Students to Practice and Deepen Knowledge** .151 .190 .172
Identifying Critical Content** .154 .161 .164
Probing Incorrect Answers*** .146 .169 .164
Maintaining a Lively Pace*** .144 .160 .160
Helping Students Record and Represent Knowledge .139 .172 .159
Establishing Classroom Routines*** .134 .191 .158
Noticing When Students are Not Engaged*** .127 .190 .153
Tracking Student Progress** .133 .174 .153
Demonstrating Withitness*** .132 .170 .152
Engaging Students in Cognitively Complex Tasks** .131 .127 .150
* indicates a high correlation in 2013-14; ** indicates a high correlation in 2012-13; ***indicates high correlations for both years
Lowest VAM Correlations
Red indicates that the element did not have a significant
correlation
2014-15Read
VAM
Math
VAM
Combined
VAM
Using Verbal and Nonverbal Behaviors that Indicate Affection for Students** .118 .103 .115
Providing Opportunities for Students to Talk** .111 .097 .113
Using Academic Games*** .077 .152 .109
Acknowledging Adherence to Rules and Procedures** .071 .154 .102
Demonstrating Intensity and Enthusiasm** .083 .119 .097
Demonstrating Value and Respect for Low Expectancy Students*** .068 .141 .095
Celebrating Success* .078 .108 .085
Using Friendly Controversy*** .065 .150 .085
Displaying Objectivity and Control*** .059 .125 .075
Understanding Students Interests and Backgrounds*** .047 .101 .061
* indicates a low correlation in 2013-14; ** indicates a low correlation in 2012-13; ***indicates low correlations for both years
Turn & Talk:
How could you use
this information at your
district or school?
BUILDING EXPERTISE 2016Annual International Marzano Conference
June 15-17, 2016
©2016 Learning Sciences International 14
MarzanoCenter.com
LearningSciences.com
1.877.411.7114
Top Related