PISAProgramme for International Student

Assessment

PISA teamDepartment of Education – Ghent

University – BelgiumBeijing – July 24-25, 2009

http://allserv.ugent.be/~mvalcke/CV/CVMVA.htmMartin.Valcke@ugent.be

(Partly based on Schleicher, A., 2006)

Structure

• Starting grounds

• Objectives

• Samples – Population

• Quality benchmarks

• Framework

• Typical assessment approach

• National Centres

PISA: starting grounds

• OECD: Organisation for Economical Organisation and Development

• Basis of PISA: original work of OECD related to statistics and indicators about education– late 1980s

– voluntary contributions

– Member engagement through networks

PISA: starting grounds

• Network on educational outcomes– Proposal to study educational outcomes

– Formally started with 11 members in 1996

– Expanded in 1997• New name PISA

• Members bear costs and risks

– Number of countries grows every PISA edition: 2000, 2003, 2006, 2009, …

PISA 2000

OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Liechtenstein, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Spanje, Zweden, Zwitserland, Verenigd Koninkrijk, Verenigde StatenPartnerlanden (niet-OESO): Albanië, Argentinië, Brazilië, Bulgarije, Chili, Hong Kong-China, Indonesië, Israël, Letland, Voormalige Joegoslavische Republiek Macedonië, Peru, Roemenië, Rusland, Thailand

PISA 2003

OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Liechtenstein, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Slovakije, Spanje, Zweden, Zwitserland, Turkije,

Verenigd Koninkrijk, Verenigde StatenPartnerlanden (niet-OESO): Brazilië, Hong Kong-China, Indonesië, Letland, Macao-China, Rusland, Servië &

Montenegro, Thailand, Tunesië, Urugay

PISA 2006

OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Slovakije, Spanje, Zweden, Zwitserland, Turkije, Verenigd Koninkrijk, Verenigde StatenPartnerlanden (niet-OESO): Argentinië, Azerbeidzjan, Brazilië, Bulgarije, Chili, Colombia, Kroatië, Estland, Hong Kong-China, Indonesië, Israël, Jordanië, Kyrgyzstan, Letland, Liechtenstein, Litouwen, Macao-China, Qatar, Montenegro, Servië, Roemenië, Rusland, Slovenië, Taipei, Thailand, Tunesië, Uruguay

OECD countries participating from PISA 2000

OECD countries participating from PISA from 2003

OECD partner countries participating from PISA 2000

OECD partner countries participating from PISA 2003

OECD partner countries participating from PISA 2006

PISA country participation

PISA 2000 (32 countries)

PISA 2003 (41 countries)

PISA 2006 (57 countries)

• A three-yearly global educational assessment• What did they learn?• Performance of 15-year-olds• key subject areas, and a range of educational outcomes

• Additionally:• student attitudes to learning, self efficacy beliefs, and

learning strategies • contextual data from students, schools, parents and

systems: policy levers

PISA: Objectives

See further

• Comparing performance within and between countries

• Cross-cultural study

• Central concept: LITERACY• Mathematical literacy• Scientific literacy• Reading literacy

PISA: Objectives

Example: scientific literacy Is defined in terms of an individual’s:• Scientific knowledge and use of that knowledge to…

… identify scientific issues, … explain scientific phenomena, and … draw evidence-based conclusions about science-related issues

• Understanding of the characteristic features of science as a form of human knowledge and enquiry

• Awareness of how science and technology shape our material, intellectual and cultural environments

• Willingness to engage with science-related issues .

PISA:Objectives

• Population of 15 year old pupils

• National samples Representative samples between 3,500 - 50,000 pupils Most federal countries: regional samples

e.g., Flanders versus Wallonia within Belgium PISA covers roughly 90% of the world economy .

PISA: Population versus samples

13

German speaking commmunity

WalloniaFrench speaking commmunity

FlandersDutch speaking commmunity

BelgiumFederal state(10,511.382 inhabitants)

PISA sampling requirements• Population: all 15-year-olds in school

– excludes 15-year-olds out of school– includes 15-year-olds in special education institutions– could exclude up to 5% of 15-year-olds in school

• difficult to reach (e.g. remote schools)• non-participation

– few countries fail to reach required sample in 2003• NZ (5.1%), Denmark (5.3%), UK (5.4%), Canada (6.8%),

Spain (7.3%), US (7.3%)• Sample

– minimum of 150 schools per country– two random samples: schools and replacement schools– if school declines, replacement school is invited– requirements set by countries

• International expertise network building on the participating countries…– Instruments ~ input of > 40 countries (see next presentation)– Cross-national and cross-cultural validity – Analysis of results

• International, natiopnal, regional analyses and reports • Country reviews

• Consortium of research institutionsACER, CITO, ETS, NIER, WESTAT

• Coordinated by OECD and international organisations

PISA: Networks

• Focus on performance in subject areas:– Languages: Reading literacy

• Using, interpreting and reflecting on written material.

– Mathematics: Mathematical literacy• Recognising problems that can be solved mathematically, representing them

mathematically, solving them.

– Sciences: Scientific literacy• Identifying scientific questions, recognising what counts as scientific evidence, using

evidence to draw conclusions about the natural world.

PISA: Objectives

PISA Objectives: cycles

2000 2003 2006 2009 2012

Reading Reading Reading Reading Reading

Maths Maths Maths Maths Maths

Science Science Science Science Science

• Focus on performance in additional domains subject areas:– 2000: Problem Solving

– 2003: ICT literacy

– 2006: Attitudes towards science

– 2009: ERA

PISA: Objectives

PISA key quality benchmarks• Overall performance of education systems

• Equity in the distribution of learning opportunities– Measured by the impact students’ and schools’ socio-economic

background has on performance…

…not merely by the distribution of learning outcomes

• Consistency of performance standards across schools

• Gender differences

• Foundations for lifelong learning

PISA

2000 2003 2006 2009 2012

Reading Reading Reading Reading Reading

Maths Maths Maths Maths Maths

Science Science Science Science Science

Problem solving

ICT literacyAttitudes towards sciences

ERA

PISA framework

Outputs and Outcomes

impact of learning

Quality and distribution of

knowledge & skills

PISA framework

Student background

variables

Mediating variables

Literacy

PISA framework: complex interplay variables

Student background

variables

Mediating variables

Literacy

PISA framework

Social & economic outcomes of

education

Output and performance of

institutions

Quality of instructional delivery

Quality and distribution of

knowledge & skills

Outputs and Outcomes

impact of learning

Instructional settings

LevelB

Schools, other institutions

LevelC

200

300

400

500

600

700

800

-3 -2 -1 0 1 2 3

HighStu

dent

perf

orm

ance

AdvantagePISA Index of social backgroundDisadvantage

Low

PISA framework

Structures, resource allocation

and policies

Social & economic outcomes of

education

The school learning environment

Teaching, learning practices and

classroom climate

Individual attitudes, engagement and

behaviour

Output and performance of

institutions

Quality of instructional delivery

Quality and distribution of

knowledge & skills

Policy Leversshape educational

outcomes

Outputs and Outcomes

impact of learning

Instructional settings

LevelB

Schools, other institutions

LevelC

Domain 2

PISA framework

National education, social & economic

context

Structures, resource allocation

and policies

Social & economic outcomes of

education

Community and school

characteristics

Student learning, teacher working

conditions

Socio-economic background of

learners

Antecedentscontextualise or

constrain ed policy

The school learning environment

Teaching, learning practices and

classroom climate

Individual attitudes, engagement and

behaviour

Output and performance of

institutions

Quality of instructional delivery

Quality and distribution of

knowledge & skills

Policy Leversshape educational

outcomes

Outputs and Outcomes

impact of learning

Instructional settings

LevelB

Schools, other institutions

LevelC

Domain 3Domain 2

Typical PISA assessment

• Information collection– From students

• 3½ hours of main domain assessment

• 1 hour in relation to other subdomain

• 2 hours on paper-and-pencil tasks (subset of all questions)

• ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation

– From school principals• Questionnaire (school demography, learning environment quality)

• Indirect assssment of classroom variables (teacher, class)BEWARE!! Only adequate if grade based sampling has been applied

PISA National Centre

• Linking with international consortium

• Implementation of framework ~national level

• Reporting to consortium

• Representation during international meetings

PISA International

• Strong prescriptive framework

• Framework, timing, procedures, tools

• National data gathering

• International processing of dataPriority in international release of results

• National (regional) processing of datanext priority level

• Secondary analysis of data: data available

New developments

• Towards electronic assessment

• 2009: first full scale trial– ERA Electronic Reading Assessment

PISAProgramme for International Student

Assessment

PISA teamDepartment of Education – Ghent

University – BelgiumBeijing – July 24-25, 2009

http://allserv.ugent.be/~mvalcke/CV/CVMVA.htmMartin.Valcke@ugent.be

(Partly based on Schleicher, A., 2006)