How Well Do Our How Well Do Our Assessment ProgramsAssessment Programs

REPORT TO SCHOOLSREPORT TO SCHOOLS??

How Well Do Our How Well Do Our Assessment ProgramsAssessment Programs

REPORT TO SCHOOLSREPORT TO SCHOOLS??

Jocelyn CookJocelyn CookManager, Educational MeasurementManager, Educational Measurement

Department of Education and Training Department of Education and Training Western AustraliaWestern Australia

Programs reported to schoolsPrograms reported to schoolsPrograms reported to schoolsPrograms reported to schools

• State and Territory Literacy and State and Territory Literacy and Numeracy population testing Numeracy population testing programsprograms

• Random sample testing programsRandom sample testing programs– State (Monitoring Standards in Education)– National (Primary Science Assessment

Program, Civics and Citizenship National Assessment Programs

– PISA, TIMSS

The aim of assessment programsThe aim of assessment programsThe aim of assessment programsThe aim of assessment programs

To contribute to the improvement of To contribute to the improvement of student learning …student learning …

How well do our population testing How well do our population testing programs and sample programs programs and sample programs contribute to the improvement of contribute to the improvement of student learning? student learning?

Improving student learningImproving student learningImproving student learningImproving student learning

Three stepsThree steps1.1. Agree and explicate what students Agree and explicate what students

should know and be able to doshould know and be able to do

2.2. Measure extent to which this is Measure extent to which this is being achievedbeing achieved

3.3. Efforts of educational enterprise Efforts of educational enterprise directed to ensuring students reach directed to ensuring students reach those goalsthose goals

Quality of monitoring programsQuality of monitoring programsQuality of monitoring programsQuality of monitoring programs

Margaret Forster, Margaret Forster, Research DirectorResearch Director,, Assessment and ReportingAssessment and Reporting

((2002 Seventh Roundtable on Assessment in 2002 Seventh Roundtable on Assessment in

Canberra)Canberra)

Framework for judging quality of Framework for judging quality of system-wide monitoring programs.system-wide monitoring programs.

Margaret Forster’s ChecklistMargaret Forster’s ChecklistMargaret Forster’s ChecklistMargaret Forster’s Checklist

·· Planning the program Planning the program (clarity of purpose, (clarity of purpose, resourcing and sustainability)resourcing and sustainability)

·· Collecting the data Collecting the data (validity and reliability)(validity and reliability)

·· Using the data Using the data (informing policy and (informing policy and reform)reform)

Planning the program: Resourcing Planning the program: Resourcing Planning the program: Resourcing Planning the program: Resourcing

All jurisdictions sustainAll jurisdictions sustain

• internationalinternational• nationalnational• State/Territory programs State/Territory programs

by providing technical, logisticalby providing technical, logisticalfinancial resources financial resources

Planning the Program: Clarity of PurposePlanning the Program: Clarity of PurposePlanning the Program: Clarity of PurposePlanning the Program: Clarity of Purpose

PISA:PISA:PISA was designed to help governments … PISA was designed to help governments … enhance the effectiveness of their enhance the effectiveness of their educational systemseducational systems

VCAAVCAA::… … The results provide information used to The results provide information used to plan new programs and a useful source of plan new programs and a useful source of feedback and guidance to students, parents feedback and guidance to students, parents and teachers. and teachers.

Tasmania’s Dept of Education:Tasmania’s Dept of Education:

… … student learning outcomes will be improved student learning outcomes will be improved by assessment, monitoring and reporting by assessment, monitoring and reporting practices that:practices that:

• inform decision-making about teaching and inform decision-making about teaching and learning;learning;

• provide useful and timely feedback to provide useful and timely feedback to students, parents and teachers; andstudents, parents and teachers; and

• enable accountability requirements to be enable accountability requirements to be met at student, school, Department and met at student, school, Department and government levels.government levels.

Collecting the data Collecting the data (validity & reliability)(validity & reliability)Collecting the data Collecting the data (validity & reliability)(validity & reliability)

State and Territory programs:State and Territory programs:

Intense scrutiny test quality & Intense scrutiny test quality & technical processes since technical processes since inception of National Benchmark inception of National Benchmark Reporting Reporting

Collecting the data Collecting the data (validity & reliability)(validity & reliability)Collecting the data Collecting the data (validity & reliability)(validity & reliability)

International and national International and national assessment programsassessment programs

national committees oversee and national committees oversee and endorse:endorse:

• assessment contentassessment content

• psychometric and technical psychometric and technical processes and proceduresprocesses and procedures

If accountability mechanism does not If accountability mechanism does not positively affect quality of:positively affect quality of:

• classroom teachingclassroom teaching• school practiceschool practice• public policy public policy

the accountability mechanisms the accountability mechanisms themselves are failing. themselves are failing.

Using the Data Using the Data Using the Data Using the Data

Good reporting Good reporting

• guides teachers to intelligent guides teachers to intelligent interpretation of data that isinterpretation of data that is

• useful to their work: teaching useful to their work: teaching students effectivelystudents effectively

Using the DataUsing the DataUsing the DataUsing the Data

Harnessing efforts of educational Harnessing efforts of educational enterprise to improve students enterprise to improve students learninglearning

• technical – technical – agree goals; measure; agree goals; measure; respondrespond

• symbolic –symbolic – motivating force (Joan Herman 2005)

Symbolic Function Symbolic Function Symbolic Function Symbolic Function

Motivation

• stimulates purposeful reform

• provides incentives and/or sanctions.

Technical FunctionTechnical FunctionTechnical FunctionTechnical Function

• measure performance and provide data to support improvement

Using the data Using the data Using the data Using the data

Sample programs Sample programs

PISA, TIMSS, PSAP, CCNAP, MSEPISA, TIMSS, PSAP, CCNAP, MSE

Limited technical and symbolic functionLimited technical and symbolic function

• No valid individual student level information No valid individual student level information • Limited school level information Limited school level information • Time-lag between testing and reportingTime-lag between testing and reporting• Few resources dedicated to maximising use Few resources dedicated to maximising use

of data at classroom and school levelof data at classroom and school level

Working with dataWorking with dataWorking with dataWorking with data

• Teachers trained for decades to Teachers trained for decades to mistrust test datamistrust test data

• Limited training in educational Limited training in educational measurementmeasurement

• Some extreme reactionsSome extreme reactions

Data Club (1999)Data Club (1999)Data Club (1999)Data Club (1999)

• Target group - school leaders Target group - school leaders

• Purpose – support them to judge Purpose – support them to judge school’s performance based on school’s performance based on their school’s Western Australian their school’s Western Australian Literacy and Numeracy Literacy and Numeracy Assessment (WALNA) dataAssessment (WALNA) data

Assessment for Improvement (2002)Assessment for Improvement (2002)Assessment for Improvement (2002)Assessment for Improvement (2002)

Target group – classroom teachersTarget group – classroom teachers

Purpose – build confidence in Purpose – build confidence in interpreting their class data derived interpreting their class data derived from WALNA from WALNA

• build teachers’ ability to blend their build teachers’ ability to blend their classroom monitoring with WALNA classroom monitoring with WALNA resultsresults

• plan for future teaching and learningplan for future teaching and learning

Teachers’ viewsTeachers’ viewsTeachers’ viewsTeachers’ views

• Prior usePrior use

• The power of talk and workThe power of talk and work

• Diagnostic useDiagnostic use

• Thinking about improvementThinking about improvement

Help schools provide better literacy & numeracy teachingHelp schools provide better literacy & numeracy teachingHelp schools provide better literacy & numeracy teachingHelp schools provide better literacy & numeracy teaching

8% 5%

42%

27%

92% 95%

58%

73%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

1999 2002 1999 2002

Agree Disagree

Parent Teacher

Evaluation FindingsEvaluation FindingsEvaluation FindingsEvaluation FindingsQuestion: The report gave me additional information not available in the regular school report

Question: The test results provided me with valuable diagnostic information about my students

38%

27%

63%

38%

62% 73%

37%

62%

0% 10% 20%

30% 40% 50% 60% 70% 80%

1999 2002 1999 2002 Parent Teacher

Agree

Disagree

Question: The data on individual students is useful for diagnostic purposes

Question: The test results provided me with valuable diagnostic information about my students

11%

38%

89%

62%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Principal Teacher

disagree agree

Evaluation FindingsEvaluation FindingsEvaluation FindingsEvaluation Findings

Evaluation FindingsEvaluation FindingsEvaluation FindingsEvaluation Findings

principals with no Data Club principals with no Data Club lessless likely to: likely to:

• provide the data to teachers or school provide the data to teachers or school communitycommunity

• judge their staff as confident users of datajudge their staff as confident users of data• use it when reviewing curriculum plans use it when reviewing curriculum plans • find the data useful for diagnostic purposesfind the data useful for diagnostic purposes• use it to track student performance.use it to track student performance.

Sample ProgramsSample Programs Sample ProgramsSample Programs

What has been learnt from the What has been learnt from the population testing experience is population testing experience is instructive for improving instructive for improving reporting to schools on sample reporting to schools on sample programs.programs.

Not about ‘their kids’ in Not about ‘their kids’ in immediate senseimmediate sense

……..but ..but information intrinsically information intrinsically

interesting. interesting.

• All jurisdictions committed to deep All jurisdictions committed to deep reform that positively alters reform that positively alters learning outcomes for studentslearning outcomes for students

BUTBUT

accountability mechanisms will fail accountability mechanisms will fail if teachers are expected to work it if teachers are expected to work it out all by themselvesout all by themselves