Copyright 2006 Stephen G. Sireci Test Accommodations and Test Validity: Issues, Research Findings,...

Copyright 2006 Stephen G. Sireci

Test Accommodations and Test Validity: Issues, Research Findings,

and Unanswered Questions

Stephen G. Sireci

Center for Educational Assessment

University of Massachusetts Amherst

Presentation delivered as part of the May 1, 2006 National Center on Educational Outcomes

Teleconference


Accommodated Standardized Test– Promotes fairness in testing?

Or– Provides an unfair advantage to some

examinees?

The Psychometric Oxymoron

What do the Standards for Educational and Psychological Testing say on this issue?


Standard 10.1: “In testing individuals with disabilities, test developers, test administrators, and test users should take steps to ensure that the test score inferences accurately reflect the intended construct rather than any disabilities and their associated characteristics extraneous to the intent of the measurement” (AERA, et al., p. 106).

Standards for Educational and Psychological Testing


Test accommodations may promote valid score interpretationsby removing a barrier associated

with the testing situation that impedes proper measurement of knowledge, skills, or abilities.– E.g., if a student can’t concentrate on

test material in a group setting, test her in a separate room.

– E.g., visually impaired person unable to read standard-size print, takes a large print or Braille version of the test.


Accommodated Tests/Administrations have the Potential to Undermine Validity in at Least 2 Ways:

1. Construct underrepresentation

2. Construct-irrelevant variance

As stated by Messick (1989):

“Tests are imperfect measures of constructs because they either leave out something that should be included…or else include something that should be left out, or both” (p. 34)


Standardized testing for SWDWhen standardized tests are NOT

accommodated for SWD– “Construct-irrelevant variance” can

interfere with test performance• e.g. ability to see, hear, focus, interferes with

measurement of math or reading proficiency

When standardized tests ARE accommodated– “Construct underrepresentation” may

occur• e.g., read-aloud for a reading assessment


Thus, the “validity dilemma” in testing SWD isIf you don’t accommodate,

constraints of the standardized testing situation will interfere with proper measurement of student’s knowledge, skill, ability, etc.

If you do accommodate, skills measured may change and interpretation of test score may change


Questions to ask in considering accommodations1. Will the accommodation(s) lead to

more accurate measurement of the student’s knowledge, skills, and abilities?

2. Will the accommodation(s) change the knowledge, skills, and abilities measured?

– i.e., will it change the construct measured?


What accommodations do states use in pursuit of more valid inferences about students’

knowledge, skills, and abilities?


Categories of Accommodations

PresentationTimingResponseSetting

Thompson, Blount, and Thurlow (2002)


Presentation Accommodations

•Oral (read-aloud, audiocassette)• Paraphrasing• Technological• Braille/large print• Sign language interpreter• Encouragement (redirecting)• Cueing• Spelling assistance• Use of manipulatives


Extended timeMultiple days/sessionsSeparate sessions

Timing Accommodations

Timing accommodations are not so much an issue on state standards-based assessments because most have generous time limits.


ScribeBooklet versus answer sheetMarking booklet to maintain placeTranscription

Response Accommodations

Setting Accommodations

• Individual administration

•Administration in a separate room


Other Accommodations

Alternate assessmentOthers?


Psychometric Research on Test Accommodations Has Focused On

•Has the accommodation changed the construct measured?

•Speed, Different skill

•Do accommodations help only those who need them?

–“Interaction hypothesis”

•Do test scores from accommodated and

non-accommodated administrations have the same meaning?

Interaction Hypothesis

Figure 1

Illustration of Interaction Hypothesis

Accommodation Condition

ACCNo ACC

Me

an

Sco

re

60

50

40

30

20

10

GROUP

GEN

SWD/ELL


Research on Test Accommodations for Individuals With Disabilities:

•Little empirical study•Some literature reviews

–Willingham et al. (1988) ─Chiu & Pearson (1999) –Tindal & Fuchs (2000) ─Pitoniak & Royer (2001)–Thompson et al. (2002) ─Bolt & Thurlow (2004)–Sireci, Scarpati, & Li (2005)

•Psychometric issues (Geisinger, 1994)

•Legal issues (Phillips, 1994)

•Also: Keeping Score for All (Koenig & Bachman, 2004)


Do test accommodations improve the scores of students with disabilities (SWD)?

If so, do such score gains reflect increased validity or unfair advantage?– Interaction hypothesis

What specific types of accommodations are best for specific types of students?

Sireci, Scarpati, & Li (2005)Research Questions


Most common findings were gains for both SWD and and non-SWD.

Most studies of extended time (6 of 8) looked at students with learning disabilities (SWLD).

Results: Extended Time


Results depend on subject– Gains for SWD only in Math– No differential gain in other subject

areas– Tends to support oral accommodation

for math tests

Results: Oral Accommodations


Research on “Equivalence of Test Structure”Do standard and accommodated

tests “measure the same thing?”– Do they have the same (factor)

dimensional structure?– One aspect of “construct equivalence”

Some studies in this area:– Rock, Bennett, Kaplan, & Jirele (1988)– Tippets & Michaels (1997)– Huynh, Meyer, & Gallant (2004)– Huynh & Barton (2006)– Cook, Eignor, Sawaki, Steinberg, & Cline (2006)


Research on Equivalence of Test Structure

Results tend to support similarity of test structure across accommodated and standard test administrations (oral, extended time, various).


Do accommodations hurt or promote valid score interpretations for students with disabilities?– Accommodations are designed to promote

validity by removing barriers (irrelevant variance)

– In general, the research suggests the accommodations being used are sensible and defensible.

Conclusions from the Literature


Extended time seems to be a valid accommodation.– Unintended test speededness could

explain results for students w/o disabilities

– Result support “differential boost” hypothesis over interaction hypothesis.

Conclusions (2)


Differential Boost Hypothesis(Fuchs, Fuchs, Eaton, Hamlett, & Karns, 2000)

Illustration of “Revised Interaction” Hypothesis

Accommodation Condition

ACCNo ACC

Mea

n S

core

60

50

40

30

20

10

GROUP

GEN

SWD/ELL


Other accommodations have less consistent and convincing results, but no evidence of “harm” or “unfairness.”

It should be noted that lots of solid and ingenious experimental research has been done in this area.– Small n, but intense with respect to data

collection

Conclusions (3)


Oral accommodation for math seems valid.

Oral accommodation for reading involves consideration of specific construct changes– Fletcher et al. (2006) results indicate

matching disability and accommodation to one aspect of construct promotes validity

Conclusions (4)


Looking across various studies and accommodation conditions– Lots of variability across studies

with respect to • accommodation conditions and how

they were implemented• Student groups (within and between)• Results

Conclusions (5)


What Should States Do In Deciding On Accommodations?The accommodations policies for

many states may not be based on research.

Research to validate accommodations policies is scarce.

So, states should conduct such research!


But how?Experimental studies

– Repeated measures designs: Test SWD with and without accommodation.

– If comparisons to non-disabled students is an issue, include those students in both conditions.

– Could also conduct between groups studies (SWD, non-SWD, random assignment to standard and accommodated conditions)

– Covariate adjustments may be helpful.


But how (2)?

Any studies conducted should look not only at students’ score differences, but also at student preferences.– “If given the choice, would you like the

accommodation?”– “Why”


Really, Steve, it’s all we can do to administer accommodated tests how can we conduct such studies?Sharron and Melissa will discuss

one example when support and state resources are available.

But small-sample studies can also be conducted.

Data can be aggregated at the state and national levels.


The key will be

to gather the data.– Keep good records.– Test students in both conditions, if

appropriate.– Carry out small-scale studies designed

to support or refute specific (potential) policy decisions.

– Consider diversity within “disability” categories.


Meta-analysis based on practice– Non-published test accommodations

being conducted in states– Establish a data warehouse for

teachers and test administrators to record results and make comments?

– Would address the small-n issue

Future Directions for Research


Larger sample sizes due to inclusion, coupled with improved school data management systems should promote more research on– Differential item functioning– Structural equivalence– Analysis of educational gains


Lot’s of work to do.

So let’s do it!

[email protected]

Thanks.

Copyright 2006 Stephen G. Sireci Test Accommodations and Test Validity: Issues, Research Findings,...

Documents

Transcript of Copyright 2006 Stephen G. Sireci Test Accommodations and Test Validity: Issues, Research Findings,...