Developing Writing in a High-Stakes Environment

Marian Sainsbury

National Foundation for Educational Research

1

2

Summary

• High-stakes testing systems• Conceptual framework – validity theory• Threats to validity in actual high-stakes

systems• Discussion – what stance should we take?

The literature review:high stakes testing

‘roller-coaster economic conditions, dramatic swings in political ideology and leadership and an eroding consensus about societal values. Rising levels of education have led to declining public confidence in institutions, an escalation of mistrust in public figures and an irresistible demand for greater accountability in public institutions’. (Earl et al, 2003, p 23)

• Policy requirement for accountability• Schools must continuously improve

3

High-stakes testing systems

• England – national curriculum testing, league tables, targets

• USA – No Child Left Behind• Australia – national system under

development• Canada – Ontario attempt at ‘benign’

accountability

4

Validity theory

• 21st century validity theory is complex and powerful

• Validity is not a property of a test, rather there is a process of validation

• Validation concerns the inferences that can justifiably be made from test outcomes

• and the uses to which the test can be put• Validation is a process of argument

5

Kane (2006)

• The interpretive argument sets out the claims made by a test or testing system

• The validation argument examines each of those claims

6

Validating high-stakes systems

Claims• Tests ‘stand for’

pupil attainment

• Tests lead to improvement

Threats• Construct under-

representation• Marking bias• Mark scheme

ambiguity• Negative impact on

teaching and learning

7

Tests ‘stand for’ attainment

• Individual performances can be taken as an indicator of underlying abilities across– Purpose– Audience– Form– Structure

• What is the construct of writing development?– Creativity + crafting (Myhill, 2001)

8

Threat 1Construct under-representation

• England– two different pieces of writing– varied purpose/audience/genre– composing complete pieces– stranded mark scheme– timed writing– no choice– not very authentic

9

10

Threat 1Construct under-representation

• USA: NAEP– ‘variety’ but actually three purposes:

narrative, informative, persuasive– variation in audience and form– composing complete pieces– holistic mark scheme– timed writing– no choice– not very authentic

• Many states have similar approach11

12

In this paper, rated "Excellent," the writer uses dialogue effectively, develops characters, and provides a coherent plot. The student shows good control of language for a fourth grader and includes vivid details to illustrate the appearance of the characters -- "He was dressed in royalty with a purple cape and a crown of jewels."

Threat 1Construct under-representation

Hillocks (2002)• Major limitations in Texas and Illinois• Scoring rubrics privilege formulaic writing

– ‘Five paragraph essay’

13

The Kentuckyportfolio assessment

• Pieces of writing selected by student but number and type prescribed

• Contributes to high-stakes accountability score as part of a complex formula

• Better, more open, potential for authentic tasks (Hillocks, 2002; Hill, 2000; Lewis, 2001)

• But in practice tasks are often teacher-directed and 5-paragraph essay can still be found (Lewis, 2001)

14

Threat 2Marking

• Test performance does not represent writing ability if score results from biased or inaccurate marking

• Mark scheme at least as important as tasks

• Must reward what is valued• Reliability of writing marking very difficult

to achieve (Congdon & McQueen, 2000)

15

Threat 2Marking

• Problems of off-task responses• Lead to unreliable marking (Lumley, 2002)• Undermine confidence in system: ‘utter

waste of education funding on a test producing spurious data’ (Allingham, 2000)

16

Threat 3Impact and consequences

• Hillocks (2002): teachers internalise test rubrics and teach five-paragraph essay

• Earl et al (2003): narrowing of curriculum and focus on test skills in England

• Anger and dismay:– Loss of: voice and ownership; time for reflection;

faith in own ability; team working; the desire to teach (McCracken and McCracken, 2001)

– Loss of creativity and emotional engagement (Nelson, 2001; Pullman, 2003)

17

Nelson (2001)

…we can create classrooms and curricula that celebrate our students’ writing and give them a place to tell their stories and turn their fear and anger and confusion into art, a place to give them hope in the form of their inborn gift of language as an instrument of creation. (Nelson, 2001, p. 58)

18

Pullman (2003)

Proper writing just doesn’t happen like that. … There are no rules. Anything that’s any good has to be discovered in the process of writing it. … some things require to be private and tentative. Teaching at its best can give pupils the confidence to discover this mysterious state, and to begin to explore the things that can be discovered there. (Powling et al, 2003, p. 8)

19

But …

I ask you – whose fault is that? Just who do you think is implementing that test prep? It certainly isn’t the textbook companies or even the principals. It’s the teachers.

(McClaskey, 2001, p. 93; author’s italics)

20

McClaskey’s 3 options

• Embrace the test– Narrow teaching to meet test expectations

• Ignore the test– Requires support from school management

• Embed the test– Teach well– But recognise reality and prepare children for

the test

21

Embedding the test

• Apthorp (2001)– Immerse students in reading and writing of whole, personally

meaningful texts– Help students transfer knowledge between reading and

writing– Teach the whole writing process as reiterative, cyclical– Expect students to apply skills to reading and writing

extended texts– Allow for student choice– Engage students in thoughtful conversations about both

strategies and content– Provide sufficient time for thoughtful conversations and

practice– Prepare students for high-stakes tests

22

Embedding the test

• Wollman-Bonilla (2004) • Persuasive essays are often set as test pieces, but can be

taught in a ‘principled’ way by using authentic contexts and workshop process approach.

• Demystify the genre by teaching techniques at the time when children need them

• Peer conferencing at relevant moments• Children set topics themselves; personal interest is a great

motivator.• ‘We can subvert the current tendency to standardize and stultify

education while also preparing our students to succeed in a world of standardized writing tests.’

23

Embedding the test

• Langer (2001) – ‘succeeding against the odds’– Total English programme is what is important – integrated

experiences across whole school– Proactive focus on knowledge and skills. Teachers unpicked

the standards and tests to understand fundamentals, rather than surface features

– Engaging and challenging activities which may be individual, pair, small group, whole class, longer projects

– A focus on generative learning. That is, applying and deepening what has been learned in new contexts

24

Discussion

• Validation theory shows heavy demands on any high-stakes system if it is to be valid:– Good construct representation in tasks– Good construct representation in mark

schemes– Marker training and standardisation– Avoid negative backwash

25

Discussion:How should academic and professional subject leaders respond?

• Should writing be assessed?• Should we engage with high-

stakes issue and put forward the embrace / ignore / embed choice explicitly?– Argue against high-stakes

testing, but accept that it exists and that children and teachers must cope with it

26

Department for Research in Assessment and Measurement

www.nfer.ac.uk

27