Assessing assessment: the role of student effort in comparative studies
description
Transcript of Assessing assessment: the role of student effort in comparative studies
Assessing assessment: the role of student effort in comparative studies
Ray Adams Jayne Butler
2
Participation
• 2000– OECD 28, non-OECD 4– PISA+, 11 non-OECD
• 2003– OECD 30, non-OECD 12
• 2006– OECD 30, non-OECD 28
3
Content
Coverage of domains
• Reading literacy
• Mathematical literacy
• Scientific literacy
4
Methods
Pencil and paper test
• Multiple choice questions
• Constructed responses
Student background questionnaire
School Principal survey
5
CYCLE OF MAJOR DOMAINS
2000 2003
Reading literacyMathematical literacy
Scientific literacy
2006
6
PISA outcomes
• Profiles of knowledge and skills
• Contextual indicators
• Trends and league tables
7
Instructions to students before the testYou have been chosen to take part in an important international education study. This study is called the Programme for International Student Assessment, ‘PISA’ for short. Its goal is to find out what students your age all around the world know about reading, mathematics and science. There are about <number of> students representing <country>. Around the world there are about 200,000 students involved, from more than 7000 schools in 40 countries. The results of the study will help countries determine what students are learning. Because the study may affect students all over the world in the future, we ask that you do the very best that you can.
8
Instructions to students at the end of the cognitive session
Please stop.
Now turn to the last page or so in your booklet, where there is a question about calculator use and a question about effort. Please answer these now, and then close your booklet.
9
The Effort Thermometer
How much effort did you invest?
Please try to imagine an actual situation (at school or in some other context) that is highly important to you personally, so that you would try your very best and put in as much effort as you could to do well.
In this situation you would mark the highest value on the “effort thermometer,“ as shown below:
Compared to the situation you have just imagined, how much effort did you put into doing this test?
How much effort would you have invested if your marks from the test were going to be counted in your school marks?
10 10 10
9 9 9
8 8 8
7 7 7
6 6 6
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1
11
Combined sample sizes for Country X and Y
Year No of students
2000 9597
2003 15997
Total 25594
12
Percentage of invalid cases
0
2
4
6
8
10
12
14
Country X2000
Country X2003
Country Y2000
Country Y2003
PISA effort
School mark effort
13
New variable
Relative Effort = School mark effort – PISA effort
Idealists: negative relative effort
Supporters: zero relative effort
Realists: positive relative effort
Cynics: extreme relative effort
14
Relative Effort Distribution
0
5
10
15
20
25
30
X-2000 Y-2000
15
Relative Effort Distribution
0
5
10
15
20
25
30
Idea
lists
Suppo
rter
s
Dilige
nt rea
lists
Realis
ts
Cynics
X-2000 X-2003
16
Relative Effort Distribution
0
5
10
15
20
25
30
Y-2000 Y-2003
17
Relative Effort Distribution
0
5
10
15
20
25
30
Idea
lists
Suppo
rter
s
Dilige
nt rea
lists
Realis
ts
Cynics
X-2000 Y-2000 X-2003 Y-2003
18
Means for Relative Effort
Country X Country Y
2000 1.74 2.14
2003 1.90 1.85
Note: higher scores mean lower effort
Significant decline in effort
Significant increase in effort
19
Means for Relative Effort by gender
1
1.2
1.4
1.6
1.8
2
2.2
2.4
Country X2000
Country Y2000
Country X2003
Country Y2003
FemaleMale
Note: higher scores mean lower effort
23
40
1623
Country X
2003
idealists
supporters
diligent realists
realistscynics
Relative effort
2000
Pla
usi
ble
val
ue
in R
ead
ing
idealists
supporters
diligent realists
realistscynics
idealists
supporters
diligent realists
realists
cynics
Country Y
2003
Relative effort
2000
Pla
usi
ble
val
ue
in R
ead
ing
idealists
supporters
diligent realists
realists
cynics
Relative effort by gender for reading
idealistssupporters
diligent realists
realistscynics
Pla
usi
ble
val
ue
in r
ead
ing Female
Male
Relative effort
23
Transformed mean scores for reading
0
20
40
60
80
100
120
140
160
Country X2000
Country X2003
Country Y2000
Country Y2003
female
male
overall
24
Does effort count?
25
Males
-10
0
10
20
30
40
50
2000 2003
X-Raw X-Adjusted Y-Raw Y-Adjusted
Consistent reduction in performance after adjustment for effort
Decrease rather than increase in performance after adjustment for effort
26
Females
0102030405060708090
2000 2003
X-Raw X-Adjusted Y-Raw Y-Adjusted
Relatively consistent performances, both with and without adjustment
Improvement (some, but not all) accounted for by effort
27
Country X
0102030405060708090
2000 2003
Male-Raw Female-Raw Male-Adjusted Female-Adjusted
Decline in boys over time leading to increased gender differences. Effort not affecting the pattern
28
Country Y
-10
0
10
20
30
40
50
2000 2003
Male-Raw Female-Raw Male-Adjusted Female-Adjusted
Effort explains some of an increase in female performance
Effort disguises a decline in male performance
Adjusted results show a widening of gender differences that is more consistent with Country X
29
Summary
• Effort is related to achievement but the relationship is non-linear
• Effort is less in Country Y than Country X but Country Y improved its effort investment from 2000 to 2003.
• Effort is less for boys than girls• Effort explains some but not all of the differences
between the achievement of boys and girls• Effort explains some but not all of the improved
performance of Country Y
30
Future Direction
• Compare students matched by effort• Compare effort distributions across
countries• Examine variance explained by
effort• Explore group differences after
controlling for effort
31