Understanding International Understanding International AssessmentsAssessments

Tom LovelessTom Loveless

UNLVUNLV

Las Vegas, NevadaLas Vegas, Nevada

January 17, 2012January 17, 2012

Outline of TalkOutline of Talk

1. The First International Tests1. The First International Tests

2. Why International Test Scores 2. Why International Test Scores MatterMatter

3. Comparing PISA and TIMSS3. Comparing PISA and TIMSS

4. Sample Items4. Sample Items

5. Results from TIMSS and PISA5. Results from TIMSS and PISA

6. Some surprises6. Some surprises

11stst International Mathematics Study, Population 1B International Mathematics Study, Population 1B

Country Mean SD

Israel 32.3 14.7

Japan 31.2 16.9

Belgium 30.4 13.7

Finland 26.4 9.6

Germany 25.4 11.7

England 23.8 18.5

International Avg. 23.0 15.0

Scotland 22.3 15.7

The Netherlands 21.4 12.1

France 21.0 13.2

Australia 18.9 12.3

United States 17.8 13.3

Sweden 15.3 10.8

International Project for the Evaluation of Educational Achievement, International Project for the Evaluation of Educational Achievement, International Study of International Study of Achievement in Mathematics: A Comparison of Twelve CountriesAchievement in Mathematics: A Comparison of Twelve Countries, edited by, edited by Torsten Husén (New York, Torsten Husén (New York, John Wiley and Sons, 1967).John Wiley and Sons, 1967).

Economic Growth and Education, 1960-2000Economic Growth and Education, 1960-2000Hanushek et al (2008)Hanushek et al (2008)

Comparing PISA and TIMSSComparing PISA and TIMSS

GovernanceGovernance

PISAPISA TIMSSTIMSS Government Government representativesrepresentatives

Researchers and Researchers and Government Government RepresentativesRepresentatives

SampleSample

PISAPISA TIMSSTIMSS Age-based; 15-year-Age-based; 15-year-oldsolds

Grade-based; 4th and Grade-based; 4th and 8th grades8th grades

Comparing PISA and TIMSS (cont.)Comparing PISA and TIMSS (cont.)

Philosophy of Assessment

PISA TIMSS

•Measures the ability to apply what has been learned to real-world situations (socio-constructivist)

•Measures what has been learned in the school curriculum

ScopeScope

PISAPISA TIMSSTIMSS Learning inside and Learning inside and outside of school, outside of school, including attitudes, including attitudes, values, and beliefsvalues, and beliefs

Topics in school Topics in school curriculumcurriculum

Comparing PISA and TIMSS (cont.)Comparing PISA and TIMSS (cont.)

Content-MathContent-Math

PISAPISA TIMSSTIMSS Mathematical literacy: Mathematical literacy: Space and shape, Space and shape, change and change and relationships, quantity, relationships, quantity, uncertaintyuncertainty

Grade 4 mathematics: Grade 4 mathematics: Number, geometric Number, geometric shapes and measures, shapes and measures, data displaydata display

Grade 8 mathematics: Grade 8 mathematics: Number, algebra, Number, algebra, geometry, data and geometry, data and chancechance

Sample Item—TIMSS 4Sample Item—TIMSS 4thth Grade Math Grade Math

Sample Items: PISA 2006 Sample Items: PISA 2006 Mathematics LiteracyMathematics Literacy

TIMSS 2007 Math Scores (Grade 4)TIMSS 2007 Math Scores (Grade 4)

Hong Kong SARHong Kong SAR 607607 GermanyGermany 525525

SingaporeSingapore 599599 DenmarkDenmark 523523

Chinese TaipeiChinese Taipei 576576 AustraliaAustralia 516516

JapanJapan 568568 HungaryHungary 510510

KazakhstanKazakhstan 549549 ItalyItaly 507507

Russian FederationRussian Federation 544544 AustriaAustria 505505

EnglandEngland 541541 SwedenSweden 503503

LatviaLatvia 537537 SloveniaSlovenia 502502

NetherlandsNetherlands 535535 ArmeniaArmenia 500500

LithuaniaLithuania 530530Slovak Slovak RepublicRepublic 496496

United StatesUnited States 529529

Countries in blue are significantly above the international average scale score (M = 500, sd = 100).

Source: TIMSS & PIRLS International Study Center, TIMSS 2007 International Mathematics Report: Findings from IES’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades (2008).

TIMSS 2007 Math Scores (Grade 4)TIMSS 2007 Math Scores (Grade 4)

ScotlandScotland 494494

New Zealand New Zealand 492492

Czech RepublicCzech Republic 486486

NorwayNorway 473473

UkraineUkraine 469469

GeorgiaGeorgia 438438

Iran, Islamic Rep. of Iran, Islamic Rep. of 402402

AlgeriaAlgeria 378378

ColombiaColombia 355355

MoroccoMorocco 341341

El SalvadorEl Salvador 330330

Tunisia Tunisia 327327

KuwaitKuwait 316316

QatarQatar 296296

Countries in green are significantly below the international average scale score (M = 500, sd = 100).

Source: TIMSS & PIRLS International Study Center, TIMSS 2007 International Mathematics Report: Findings from IES’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades (2008).

TIMSS 2007 Math Scores (Grade 8)TIMSS 2007 Math Scores (Grade 8)

Chinese TaipeiChinese Taipei 598598

Korea, Rep. ofKorea, Rep. of 597597

SingaporeSingapore 593593

Hong Kong SARHong Kong SAR 572572

JapanJapan 570570

Hungary Hungary 517517

EnglandEngland 513513

Russian Russian 512512

United StatesUnited States 508508

LithuaniaLithuania 506506

Czech RepublicCzech Republic 504504

SloveniaSlovenia 501501

Armenia Armenia 499499

AustraliaAustralia 496496

Countries in blue are significantly above the international average scale score (M = 500, sd = 100).

Source: TIMSS & PIRLS International Study Center, TIMSS 2007 International Mathematics Report: Findings from IES’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades (2008).

TIMSS 2007 Math Scores (Grade 8)TIMSS 2007 Math Scores (Grade 8)

SwedenSweden 491491 Tunisia Tunisia 491491

MaltaMalta 488488 GeorgiaGeorgia 488488

ScotlandScotland 487487 IranIran 487487

SerbiaSerbia 486486 Bahrain Bahrain 486486

ItalyItaly 480480 IndonesiaIndonesia 480480

MalaysiaMalaysia 474474 Syrian Arab RepublicSyrian Arab Republic 474474

NorwayNorway 469469 EgyptEgypt 469469

CyprusCyprus 465465 Algeria Algeria 465465

BulgariaBulgaria 464464 ColombiaColombia 464464

IsraelIsrael 463463 OmanOman 463463

Ukraine Ukraine 462462 PalestinePalestine 462462

Romania Romania 461461 BotswanaBotswana 461461

BosniaBosnia 456456 KuwaitKuwait 456456

Lebanon Lebanon 449449 El SalvadorEl Salvador 449449

ThailandThailand 441441 Saudi ArabiaSaudi Arabia 441441

TurkeyTurkey 432432 GhanaGhana 432432

JordanJordan 427427 QatarQatar 427427

Countries in green are significantly below the international average scale score (M = 500, sd = 100).

Source: TIMSS & PIRLS International Study Center, TIMSS 2007 International Mathematics Report: Findings from IES’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades (2008).

What Can We Learn from International What Can We Learn from International Tests?Tests?

Trends in National AchievementTrends in National Achievement

Top Scoring NationsTop Scoring Nations

Some SurprisesSome Surprises

Trends in Mathematics Scores, 8Trends in Mathematics Scores, 8thth grade gradeby country: 1995 to 2007by country: 1995 to 2007

* p<.05. Within-country difference between 1995 and 2007 average scores is significant.

Country difference in scores between 1995 and 2007 is greater than U.S.

Country difference in scores between 1995 and 2007 is less than U.S.

Country difference in scores between 1995 and 2007 is not measurably different from U.S.

1995 2007Colombia 332 380 47 *Lithuania 472 506 34 *Korea, Rep. of 581 597 17 *United States 492 508 16 *England 498 513 16 *Slovenia 494 501 7 *Hong Kong SAR 569 572 4Cyprus 468 465 -2Scotland 493 487 -6Hungary 527 517 -10 *Japan 581 570 -11 *Russian Federation 524 512 -12Romania 474 461 -12 *Australia 509 496 -13 *Iran, Islamic Rep. of 418 403 -15 *Singapore 609 593 -16 *Norway 498 469 -29 *Czech Republic 546 504 -42 *Sweden 540 491 -48 *Bulgaria 527 464 -63 *

Average score Difference

2007-1995Country

TIMSS 2007 Mathematics Grade 8TIMSS 2007 Mathematics Grade 8

• U.S. average score lower than average scores of 5 countries

• Top countries in Asia

Average score is higher than U.S. average score

Average score is lower than U.S. average score

Average is not measurably different from U.S. average

Why Do Top Scoring Nations Do So Well?Why Do Top Scoring Nations Do So Well?My Own SpeculationMy Own Speculation

1. Culture—emphasis on academic 1. Culture—emphasis on academic achievement in childhood.achievement in childhood.

2. Time—longer day and year + time outside 2. Time—longer day and year + time outside of schoolof school

3. Curriculum—lean, focused, sequential3. Curriculum—lean, focused, sequential

4. Teachers—trained in academic 4. Teachers—trained in academic disciplines disciplines rather than education schoolsrather than education schools

What Do Foreign Exchange Students Think?

Brown Center Reports 2001, 2002. Sample of approximately 380 students from

abroad (2001) and about the same from U.S (2002).

Asked a series of questions comparing schools and peers in U.S. and abroad.

A SurpriseA SurpriseAchievement and EnjoymentAchievement and Enjoyment

Between Country Relationship, r = -0.76Between Country Relationship, r = -0.76

300

350

400

450

500

550

600

-0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20

Index of Enjoyment of Science

Sci

ence

Sco

re

TunisiaNetherlands

USA

How Does Clark County Stack Up?

ConclusionConclusion

• The two international tests, TIMSS and PISA, differ in The two international tests, TIMSS and PISA, differ in content, sampling design, and age of students tested.content, sampling design, and age of students tested.

• The U.S. scores near the international average on The U.S. scores near the international average on mathematics tests. Mediocre but not awful.mathematics tests. Mediocre but not awful.

• The U.S. has The U.S. has nevernever scored at the top on international tests scored at the top on international tests of math and science. of math and science.

• The U.S. has improved (modestly) on international tests of The U.S. has improved (modestly) on international tests of math since 1995.math since 1995.

• National test scores in math are inversely correlated with National test scores in math are inversely correlated with national indicators of student confidence and enjoyment of national indicators of student confidence and enjoyment of subject.subject.

Conclusion (cont.)Conclusion (cont.)

• Culture seems to matter a lot in promoting academic Culture seems to matter a lot in promoting academic achievement.achievement.

•Extrapolations of state to national to international scores Extrapolations of state to national to international scores are fraught with error.are fraught with error.

• That said, Clark County almost certainly would score That said, Clark County almost certainly would score significantly below average on a test of international significantly below average on a test of international achievement.achievement.