AAAL 2012 Annual Conference

Diagnostic Testing of ESL Reading Skills:Diagnostic Testing of ESL Reading Skills: Relating Test-taking Strategies to

R di AbilitiReading Abilities

Alan UrmstonHong Kong Polytechnic Universityg g y y

DELTA is …

an online English language proficiency assessment system designed specifically for the Hong Kong academic context

d d h d k

designed specifically for the Hong Kong academic context which can

diagnose students’ strengths and weaknessestrack and report on students’ language gainsbe linked to curriculum/teaching/materialsbe linked to curriculum/teaching/materials

help students to plan their language learning

inform curriculum developmentinform curriculum development

DELTA Track and Component Skills Profile DELTA Track and Component Skills Profile

‘It is common to assert that diagnostic tests are i d d b h h d k f intended to probe the strengths and weaknesses of learners, but there is virtually no description, much less discussion of what the underlying much less discussion of what the underlying constructs might be that should be operationalized in valid diagnostic tests’ (p.28). g p

(Alderson, 2007)

Validating the DELTA: Th di The reading test

Jang (2009) identifies two key issues in the diagnostic assessment of reading comprehension: ‘the characteristics of reading skill profiles ‘the characteristics of reading skill profiles … and the usefulness of the resulting diagnostic information for students and teachers’ (p 39) information for students and teachers (p.39), reiterated by Alderson (2007) who suggests that ‘central to diagnosis must be the provision of usable feedback either to the learners themselves or to the diagnoser’ (p.30).

The DELTA reading subskillsDrawing on taxonomies of reading subskills (Munby, 1978; Weir, 1983; Alderson & Lukmani, 1989; Jang, 2009), knowledge of the Hong Kong tertiary-level English language learning and teaching context the DELTA Team initially

Identifying specific information

level English language learning and teaching context, the DELTA Team initially identified eight (8) reading subskills to be tested in the DELTA:

y g pInterpreting a word or phrase as used by the writerUnderstanding main ideas and supporting ideasUnderstanding information and making an inferenceUnderstanding information and making an inferenceUnderstanding an argument made by the writerInterpreting an attitude or intention of the writerUnderstanding grammatical relationships of words or phrases across a textUnderstanding grammatical relationships of words or phrases across a textIdentifying text type

The DELTA reading componentg pconsists of

four testlets (reading text + test items)increasing difficulty level (e.g. easy, medium, difficult)items of calibrated difficulty level (Winsteps)reading texts of the type (topic, theme, genre) that undergraduate students would be expected to read (both undergraduate students would be expected to read (both academic and non-academic)

Component Diagnostic Report: Reading

‘What a test of reading tests is not simply what its i h f j d constructors say it tests, nor what a set of judges

considers it to test. It must surely and crucially relate to what happens inside a test-taker’s head when he to what happens inside a test-taker s head when he or she responds to an item’ (p.478).

(Alderson 1990)(Alderson, 1990)

Test-taker stimulated recallsPost-test interviews were conducted with 10 students who had taken the DELTA a few days ypreviously.

Students were asked to describe the strategies they had used to answer the questions.

In some cases students did live think-aloud while answering reading questions.

Strategies used were identified and coded using the di h d i d b C h d U t ( )coding scheme devised by Cohen and Upton (2007).

Identifying specific information (Ability to locate and recognize information in one spot/in more

Text level: Easy (DELTA 1 65) Item

(Ability to locate and recognize information in one spot/in more than one spot with or without rephrasing within specific context.)

Text level: Easy (DELTA 1-65) Item

Although the cities that the Maya built were beautiful and the people

Even though the workers built the cities, built were beautiful and the people

worked hard to build them, usually, only the priests lived in the cities and other people lived in small villages in

A. they were reserved only for priests.* B. the ordinary people didn't like the

________

other people lived in small villages in the forests. They lived in small huts with no windows with the walls made of poles covered with dried mud and

cities. C. it took them a long time to construct

the cities. l h h hof poles covered with dried mud, and

the roof made of grass or leaves. D. people thought they were not

comfortable enough.

Identifying specific information

S2. Annie (average ability): I read the question and the options and then I scanned through the q p gparagraph and I found the word 'priests'. I noticed that the first option has the same word and so I chose A.

S3 Joyce (high ability): S3. Joyce (high ability): I read the whole passage quickly and so I knew the answer should be A. It’s an easy passage so I understand very well.

S5. Bill (average ability): I read the whole passage. I understand it easily coz the sentences are short and the words are easy to understand. I eliminated the other short and the words are easy to understand. I eliminated the other options as I knew they don’t make sense.

C i f d if i ifi i f iCommon strategies for Identifying specific information

Code Strategy descriptionCode Strategy description

T5 Reads the question and then reads the passage/portion to look for clues to the answer, either before or while considering options.

R7 Reads a portion of the passage rapidly looking for specific informationR7 Reads a portion of the passage rapidly looking for specific information.

T24 Selects options through elimination of other option(s) as unreasonable based on paragraph/overall passage meaning.

R5 Reads the whole passage rapidly.

TW1 Uses the process of elimination (i.e., selecting an option even though it is not understood out of a vague sense that the other options couldn’t be correct)understood, out of a vague sense that the other options couldn t be correct).

R4 Reads the whole passage carefully.

TW3 Selects the option because it appears to have a word or phrase from the passage in it –ibl k dpossibly a key word.

R6 Reads a portion of the passage carefully.

T22 Selects options through vocabulary, sentence, paragraph, or passage overall meaning T22 p g y, , p g p , p g g(depending on item type).

So, for Identifying specific information (and for the other subskills) the other subskills) …

students use a combination of reading test management students use a combination of reading, test management and test wiseness strategies to answer the items.

So how do reading strategies translate into di kill b kill ? reading skills or subskills?

‘Reading strategies are deliberate, goal-directed attempts to control and modify the reader’s efforts to decode text understand words and construct meanings decode text, understand words, and construct meanings of text. Reading skills are automatic actions that result in decoding and comprehension with speed, efficiency, in decoding and comprehension with speed, efficiency, and fluency and usually occur without awareness of the components or control involved’ (p.368).

Afflerbach et al (2008)

Just reads from the start and straight to the end; reading for no purpose [she] wouldn't stopno purpose, [she] wouldn't stop or pause. If it's something [she] can't understand then [she'll] stop once in a while to check what [she’s] read.

Reads all the questions first then reads the relevant part of the text.

DELTA Reading Text Comparison

DELTA R di P It MDELTA Reading Person-Item Map

More-proficient Readers

GRO

Less-proficient Readers

OWTH

DELTA R di P It MDELTA Reading Person-Item Map

more proficient readers

what the more proficient readers proficient readers can do

less proficient readers

The challenge The challenge …

to provide our learners with usableto provide our learners with usable information to help them know what they can and cannot do nowthey can and cannot do now.

Component Diagnostic Report: Reading (revised)

Tentative descriptions of proficiency

Proficient readers (DELTA > 110) can:

Identify specific information in feature articles on less familiar topics

Understand the main and supporting ideas in academic articles in fields outside of their course of study

h d f f l Interpret the attitude or intention of a writer in fictional or biographical texts

The challenge … to predict growth in our learners learners.

200 DELTA TRACK

160

180DELT

113118 120

125120

140A

MEA 113

80

100S

U

R

E

60

08/2011 08/2012 08/2013 Graduation

DELTA Measure Predicted Measure Target Measure

The challenge …

to help our learners to grow gas readers.

ICOSAICOSAReading RequirementReading Requirement

ReferencesAfflerbach P Pearson P D & Paris S G (2008) Clarifying Differences between Reading SkillsAfflerbach, P., Pearson, P.D. & Paris, S.G. (2008). Clarifying Differences between Reading Skills 

and Reading Strategies. The Reading Teacher, (61)5, International Reading Association Alderson, J.C. (1990). Testing Reading Comprehension Skills (Part 2): Getting Students to Talk 

about Taking a Reading Test (A Pilot Study) Reading in a Foreign Language (7)1 pp 465‐about Taking a Reading Test (A Pilot Study).  Reading in a Foreign Language (7)1, pp.465‐503. 

Alderson, J.C. (2007). The Challenge of Diagnostic Testing: Do We Know What We Are Measuring? In J Fox M Wesche D Bayliss L Cheng C Turner & C Doe (Eds ) LanguageMeasuring? In J. Fox, M. Wesche, D. Bayliss, L. Cheng, C. Turner & C. Doe (Eds.) Language Testing Reconsidered. Ottawa: University of Ottawa Press, pp. 21‐40. 

Alderson, J.C. & Lukmani, Y. (1989). Cognition and Reading: Cognitive Levels as Embedded in Test Questions Reading in a Foreign Language (5)2 pp 253 270Test Questions. Reading in a Foreign Language (5)2, pp.253‐270.

Jang, E.E. (2009). Demystifying a Q‐Matrix for Making Diagnostic Inferences about L2 Reading Skills. Language Assessment Quarterly, (6), pp.210‐238. 

Munby J L (1978) Communicative Syllabus Design Cambridge University PressMunby, J.L. (1978). Communicative Syllabus Design. Cambridge University PressWeir, C.J. (1983). Identifying the language problems of overseas students in tertiary education in 

the U.K. Unpublished PhD thesis, Institute of Education, University of London. 

For more information on DELTA For more information on DELTA

http://gslpa.polyu.edu.hk/eng/delta web/http://gslpa.polyu.edu.hk/eng/delta_web/