Defining Reading Proficiency.pdf

7/27/2019 Defining Reading Proficiency.pdf

1/12

National Accessible Readi

Assessment Projects

Defining Reading Proficiency for

Accessible Large-Scale Assessments:

Some Guiding Principles and Issues


2/12

The University o Minnesota is committed to the policy that all persons shall have equal access to its programs,

acilities, and employment without regard to race, color, creed, religion, national origin, sex, age, marital status, dis-

ability, public assistance status, veteran status, or sexual orientation.

This document is available in alternative ormats upon request.

National Accessible Reading Assessment Projectshttp://www.narap.info

This work is supported, in part, by the

U.S. Department o Education, Institute o

Education Sciences, National Center or

Special Education ResearchGrant Numbers

H324F040001 and H324F040002. Opinions expressed do not necessarily reect those o the U.S.

Department o Education or ofces within it. Endorsement by the Federal Government should not

be assumed.

Partnership for Accessible Reading Assessment

University of Minnesota (NCEO and Dept. of C & I) / CRESST (UCLA & UCD) / Westat

350 Elliott Hall

75 E. River Road

Minneapolis, MN 55455

(612) 625-7241

Fax: (612) 624-0879

http://www.readingassessment.info

[email protected]

Designing Accessible Reading Assessments

Educational Testing Service

Rosedale Road, MS 09R

Princeton, NJ 08541

(609) 734-1347

Fax: (609) 734-1755

http://www.ets.org/dara

[email protected]


3/12

Defning Reading Profciency

or Accessible Large-Scale Assessments:

Some Guiding Principles and Issues

Prepared by:

National Accessible Reading Assessment Projects

Designing Accessible Reading Assessments (DARA)Educational Testing Service (ETS)

and

Partnership or Accessible Reading Assessment (PARA)University of MinnesotaNCEO (National Center on Educational Outcomes) andDepartment of Curriculum and Instruction

CRESST (Center for Research on Evaluation, Standards, and Student TestingUniversity of California, Los Angeles; and University of California, Davis)

Westat

February 17, 2006

Please use the following citation for this paper:

National Accessible Reading Assessment Projects. (2006). Defning reading profciency or accessible large-scale assessments: Some guiding principles and issues. Minneapolis, MN: Author.

Available on the World Wide Web at www.narap.info.


4/12

TheNationalAccessibleReadingAssessmentProjects

Introduction

Te National Accessible Reading Assessment Projects (NARAP) is a collaboration otwo projects unded by the U.S. Department o Education to conduct research anddevelopment on accessible reading assessments or students with disabilities that aect

reading. Te goal o these projects is to produce research ndings and assessmenttechniques that demonstrate how large-scale assessments o reading prociency canbecome more accessible and valid or all students, while also meeting the assessmentrequirements o the No Child Le Behind Act o 20011 (NCLB).

Understanding the need or accessible assessments is key to determining the stepsthat must be taken to develop such assessments. NCLBs critical provision that allstudents be included in state accountability systems, including the ull range o studentswith disabilities, brings to the oreront assessment challenges that have not yet beenadequately addressed. o be air to schools and to students, assessments need to allowall students the best opportunity to show what they know and what they can do. Allstudents have skills and knowledge in reading that reect varying levels o prociency byreading component. Tat is, each student may have some reading skills and knowledgethat reect what is considered to be the procient level, other skills and knowledge thatare emerging, and other reading skills and knowledge that have not yet emerged. Somestudents with disabilities may have skills and knowledge that are expressed in ways thatare not measured in current assessments; that is, they may have skills and knowledgethat in many assessments are integrated with skills and knowledge they do not have (e.g.,when decoding o passages is required in order to demonstrate comprehension). Tus,the denition o reading prociency must be stated in such a way that allows assessmentsto reect what students are able to do, not just what students are not able to do.

Te desire or all students with disabilities to be able to show their prociency asreaders does not indicate a desire to lower expectations or the prociency levels othese students. Rather, it is important or students to be held to the same standards oreading prociency whenever possible. Still, it will likely be necessary to provide exibleexpressions o reading. Flexible expressions o reading may include accommodationssuch as letting students give oral responses to questions, but may also require anapproach that is built into the assessment itsel universal design. By universal designwe mean assessments that are designed rom the beginning, and continually rened,to allow participation o the widest possible range o students, resulting in more validinerences about perormance (Tompson, Johnstone, & Turlow, 2002). echnologymay be an element o providing universal design and exible access to all students,

including students with disabilities.Seeking exibility also may suggest the need or large-scale assessments that do notrely solely on one way or students to demonstrate prociency. Students who are able todemonstrate skills on a range o important reading standards may be considered readerseven though they may not be procient on all components. Alternative approaches

This includes using the defnition o reading that is included in the Reading First program o NCLB.


5/12

DefningReadingProfciency

to scoring assessments such as the use o compensatory approaches rather thanconjunctive approaches may be another avenue to realizing a denition o readingprociency. By this we mean, or example, being able examine comprehension skillsseparately rom decoding skills. Tese alternative approaches recognize that studentswith disabilities deserve the opportunity to show the ull prole o their developing

prociency as readers.

Consequently, one o the rst steps in designing accessible assessments is the precisedenition o constructs being measured. Such denitions permit construct-relevantactors to be distinguished rom construct-irrelevant actors, thus acilitating the removalo barriers to accessibility while maintaining valid measurement o the constructs ointerest. Similarly, distinguishing various components o the construct may acilitatedevelopment o assessments that can reveal strengths in some components that wouldotherwise be masked by weaknesses in other components. Tus, one o NARAPs rsttasks was to develop a denition o reading prociency. In order to accomplish this wecollected inormation in a variety o ways, including: (a) reviewing existing denitions

o reading prociency (e.g., various reports about reading including the NRP, RAND,PISA, PIRLS), (b) convening a panel o experts to provide input, and (c) conductingocus groups. (Te inormation and materials used during this process as well as theocus group ndings are available at www.narap.ino .) Tis paper is a synthesis othe inormation collected and describes a set o principles and key unresolved issuesdescribed under each principle. Tese principles and issues will guide the next phases oour project (research and development). Tis paper does not ocus on the assessmentitsel; rather, the assessment developed will be inormed by our research and thenaddressed in a separate paper ocusing on the principles and guidelines o accessibleassessments.

Principles or Dening Reading Prociency

NARAP has identied three principles or dening reading prociency. Each o theseprinciples is discussed here, with issues identied as appropriate.

Principle 1: Denitions o reading prociency must be consistent with core NCLBprovisions.

Te ederal priority under which NARAP was unded cited two required purposes oNCLB assessments that must be met by accessible large-scale tests o reading prociency.Tese purposes are to provide (a) a valid measure o prociency against academic

standards, and (b) individual interpretive, descriptive, and diagnostic reports or the ullrange o students with disabilities that aect reading. With regard to the rst o thesetwo purposes, NCLB requires each state to establish grade-level academic standards.Because each state sets its own standards, the denition o reading prociency used byNARAP must not impose a particular standard but rather allow or variability amongstates. o be consistent with NCLB and its requirement or grade-level standards,


6/12


NARAP must base its research and development on grade-level denitions o readingprociency. Access, participation in, and progress in the general curriculum is aoundational requirement o the Individual with Disabilities Education Act (IDEA) since1997, reinorced as access, participation, and progress in grade-level curriculum andstandards by NCLB law and regulation and IDEA 2004 law and dra regulations.

States address areas o reading prociency and component skills through grade-leveldenitions o content standards (Tompson, Johnstone, Turlow, & Clapper, 2004). Forexample, phonemic knowledge is on the standards lists o 47 states at grades 1 and 2,and 27 states at grade 8. Fluency standards are listed or 39 states in grades 1 and 2 and30 states at grade 8. Skill standards or expository elements o text are listed by 20 statesat grades 1 and 2, 33 states at grade 4, and 30 states at grade 8. Te number o stateslisting the same standards or all grades (1-8) on other aspects o reading skills is airlyconsistent (e.g., inerential comprehension: grades 1 and 2on the lists o 31 states,grade 4on the lists o 37 states, grade 8on the lists o 40 states and word recognitionis on the lists or 45 states or grades 1 and 2 and remains on the lists o 40 states or

grade 8). Such general statements, however, should be read with caution, since themethod o simply counting reerences to terms in standards may not capture the relativeemphasis o standards in particular grades. Tere are additional caveats to this analysis.First, an analysis o state standards does not necessarily indicate the competencies statesactually include on their assessments. Second, current practice in dening state readingstandards may not be best practice in some cases. In act, some states do not measureappropriate skills and are conservative about what skills are listed or measured.

Grade-level skills are important and necessary oundations or any accessible assessment.Tis is evident in NAEP, which all states now are required to administer. Te NAEP 2009ramework provides a model or dening reading prociency in grades 4, 8, and 12 byrequiring students to read both literary (ction, nonction, and poetry) and inormationtexts (exposition, argumentation, persuasive, and procedural texts or documents).Readers must access words in texts, use the structure o texts, make sense o vocabularyas it is embedded in a text, understand sentences and paragraphs, and comprehendwhat they read. NAEP assessments expect that students text comprehension will beinuenced by their ability to apply the oundational components o reading: phonemicawareness, phonics knowledge, uency, and vocabulary. In addition, NAEP 2009 ocuseson grade-level cognitive targets and denes these as the mental processes or kinds othinking that underlie reading comprehension; the cognitive targets serve to guide thetest development process in that item writers target these processes or kinds o thinkingas they write items (2009 NAEP Reading Framework, p. 39).

Te second aspect o NCLB cited above, that assessments should provide individualinterpretive, descriptive, and diagnostic reports or the ull range o students withdisabilities that aect reading, emphasizes that assessments should be as worthwhileas possible or the ull range o students. o the extent that this is possible, assessmentsused primarily or accountability purposes (e.g., NCLB), should also provide useulinormation to educators as they plan instructional improvements. Tis inormation


7/12


would be coupled with other reading assessments used at a local level that areindividually administered and diagnostic in nature (e.g., inormal reading inventories;running records). Literature on developing and using tests provides cautions about theextent to which a single test can be used or multiple purposes (AERA, APA, NCME,1999).

Issues related to Principle 1:

A. How do the important reading skills vary as a unction o grade level?

B. How do we determine which measure may be appropriate or use at a specic gradelevel?

C. How much can achievement levels vary and still meet the requirements o grade-levelcontent?

D. How are dierences in reading achievement standards (e.g., modied or alternate

achievement standards) developed and dened? How are these varying achievementstandards reected in denitions o reading prociency?

Principle 2: Reading prociency must be dened in such a way that fexibleexpressions o reading are allowed while preserving the essential nature oreading. This is crucial as we seek to make assessments accessible to studentswith a variety o disabilities.

Flexibility in how reading prociency is dened and measured is demanded by thepractice o states developing their own standards and the reality that students dierin their reading strengths and weaknesses in component processes. Procient readers

draw upon a range o processing abilities ranging rom lower-level processes to higher-order processes. At the lower level they recognize individual words, and at higher levelsthey assemble inormation rom multiple sources into meaningul representations otext and relate it to background knowledge. No one component process can account oroverall prociency. Rather, both lower-level and higher-level processes contribute toindividual dierences in reading prociency (Daneman,1996). Te reliance on any seto component prociencies and the use o compensations are aimed at allowing readersto achieve overall prociency in understanding a given text. ypical procient readersrapidly recognize words thus reeing up processing capacity or comprehension. Onthe other hand, readers who have visual or auditory disabilities may compensate or alack o rapid recognition o orthographic patterns through more ecient memory o

written discourse structure or expository texts or stories than other readers. A readerusing braille can demonstrate overall reading comprehension prociency, albeit bymeans o an alternate ormat. Students with congenital deaness may achieve overallreading comprehension prociency, but may need to deploy alternative strategies tounderstanding sound-symbol relationships as a basis or reading.


8/12


Public expectations and perceptions about reading oen are reected in practice andpublic policy. We live in a world o printed text, but also a world with growing availabilityo text in other ormats, such as books on tape or CD, printed text translated into speechby reading machines, text transmitted electronically via the internet and readable byscreen reading soware, and materials in accessible electronic ormats (e.g. digital

talking book and the National Instructional Materials Accessibility Standard) that canbe presented to the reader in various visual or auditory modalities. Individuals who areblind or partially sighted oen reer to their activity as reading when they are in actlistening to text, and people without disabilities also take advantage o these new ormats,or example by listening to books while engaging in other activities.

Yet, the public is more likely to view the individual who is reading braille as a readerthan they are to reer to the individual who is reading American Sign Language (ASL)as a reader. Similarly, the reader o ASL is more likely to be considered a reader thanthe person who is reading by having a screen reader read a page o text to him orher. Tese public views are reected in the number o states that allow these types o

approaches on their state assessments braille is allowed without restriction in 39 states;sign language interpretation o the reading test is allowed without restriction in 13 states;and reading the questions aloud to the test taker is allowed without restriction in 3 states(Clapper, Morse, Lazarus, Tompson, & Turlow, 2005).

Federal statutes such as NCLB and IDEA allow a range o options in the types oassessments used and the achievement standards applied to students with disabilities.In addition to taking general assessments based on grade-level achievement standardswith or without accommodations, some students with disabilities can take assessmentsbased on modied achievement standards, or alternate assessments based on grade levelstandards or (or students with the most signicant cognitive disabilities) on alternateachievement standards. Modied or alternate achievement standards must be aligned tograde-level content standards, but may dier rom the grade-level achievement standardsin breadth, depth, or complexity. All o these are possible means o exibility or studentswith disabilities.


A. How broadly can we dene what constitutes reading and still have the denitionbased on grade-level achievement standards?

B. Can what constitutes reading or standards-based assessments dier by disabilitycategory or by needed accommodation?

C. How do the concepts o modied and alternate achievement standards apply to gradelevel reading standards?

D. How do students with disabilities compensate or weaknesses in specic readingprociency components due to their disability or multiple disabilities?


9/12


Principle 3: Denitions o reading prociency must refect both comprehensionand oundational skills.

Te inormation we collected by reviewing existing denitions o reading prociency,obtaining input rom a panel o experts and conducting ocus groups helped developa rm base upon which to anchor the exibility described in Principle 2. Although theemphasis varied, a consistent message was that the core construct o a denition oreading reerences both oundational reading skills and comprehension. Te ollowingdenition o reading, included in the Reading Firstprogram oNo Child Le Behind(2001), places a clear emphasis on oundational reading skills.

Te term reading means a complex system o deriving meaning rom print that requiresall o the ollowing:

(A) Te skills and knowledge to understand how phonemes, or speech sounds, areconnected to print.

(B) Te ability to decode unamiliar words.

(C) Te ability to read uently.

(D) Sucient background inormation and vocabulary to oster reading comprehension.

(E) Te development o appropriate active strategies to construct meaning rom print.

(F) Te development and maintenance o a motivation to read.

Other denitions such as the 2009 NAEP ramework stress that reading is an active andcomplex process that involves understanding written text; developing and interpretingmeaning; and using meaning as appropriate to type o text, purpose, and situation. Inthis denition it is assumed that students must apply oundational skills to comprehenda variety o texts. Tus, an assessment based on the NAEP ramework is a meaningulindicator o reading comprehension that indirectly measures oundational skills. Itcould be argued that i a child can demonstrate prociency on such an assessment, noadditional assessment o oundational skills is required. On the other hand, deningthe oundational skills involved in the reading processes is particularly important orthe population under study by NARAP (students with disabilities that aect reading)because we cannot assume that these oundational skills have been or are in the processo being acquired. Dening oundational skills will give NARAP the exibility to

develop component level measures that provide inormation on what students can do,rather than a single score o non-procient. Tis principle suggests the need or large-scale assessments that are exible (perhaps technology-based) and able to assess readingcomprehension and/or oundational skills based on student perormance.


10/12



A. o what degree can component skills be measured independently?

B. Comprehension is the primary goal or readers. I students are procient in this areawith accommodations, do we need to measure the oundational skills?

C. I comprehension is our primary goal, should the comprehension score be weightedmore heavily when oundational skills are also assessed?

D. What easible techniques are available or measuring oundational skills such asuency or phonics knowledge in the context o large-scale assessment?

E. For students who do not achieve grade-level prociency, what processes can bedeveloped or applied to aggregate their perormance on component skills into anoverall measure o reading prociency?

F. Some components appear to be problematic or certain disabilities (e.g. phonemesand dea students). Do we develop alternate denitions o procient readingor these populations based on a better understanding o reading processes andperormances?

G. Are some skills less critical to measure than others?

H. I oundational skills were only assessed aer a student had perormed belowprocient on comprehension, what proportion o ALL students (with and withoutdisabilities) would be assessed on each o the oundational skills? Is this proportionsmall enough to assess students in small groups, individually, or via computer?

I. Can some oundational skills be assessed together (e.g., decoding and phonemes)?

J. I oundational skills are going to be measured only or students who are notprocient on an assessment o comprehension, can accommodations be allowedthat invalidate the oundational skills (e.g., read aloud or decoding or extra time oruency)?

K. I a student is not procient on a measure o reading comprehension, should listeningcomprehension be assessed prior to measuring oundational skills?


11/12


Conclusion

Tis paper ocuses on several o the complex issues in creating accessible and validmeasures o reading prociency or students with disabilities that aect reading. Teprinciples included in this paper will guide NARAP in ormulating the denitions o

reading prociency we will use within our studies. A denition o reading prociency,compatible with all three principles outlined in this paper, would be an organizingstructure or ramework that would support each state as educators indicate how muchemphasis various components o reading have in their grade-level reading standards.During the next phase o our project we will ocus our eorts on conducting researchthat willbe consistent with Principle 1 and address some o the issues listed underprinciples 2 and 3. Several o these issues cannot be resolved through empirical research.Instead, they require that we ocus our eorts on providing inormation that can be usedto set policies on assessing students with disabilities that impact reading.


12/12


Reerences

American Educational Research Association, American Psychological Association, andNational Council o Measurement in Education (1999). Standards or educational andpsychological testing.

Clapper, A., Morse, A., Lazarus, S., Tompson, S., & Turlow, M. (2005). 2003 statepolicies on assessment participation and accommodations or students with disabilities(Synthesis Report 56). Minneapolis, MN: University o Minnesota, National Center onEducational Outcomes.

Daneman, M. (1996). Individual dierences in reading skills. In R. Barr, M. Kamil, P.Mosenthal, & P. D. Pearson (Eds.), Handbook o reading research (vol. 2) (pp. 512538).Mahwah, NJ: Erlbaum.

Individual with Disabilities Education ACT o 1997, 20 U.S.C. 1412 (a) (17) (A) (1997).

National Assessment Governing Board. (Spring, 2005). Specifcations or the 2009 NAEPReading Assessment. Washington, DC: Author. Retrieved February 18, 2006, romhttp://www.nagb.org/pubs/reading_w_08_05_prepub_edition.doc[Also, see Salinger, ., Kamil, M. L., Kapinus, B., & Aferbach, P. (2005). Developmento a new ramework or the NAEP reading assessment. 54rd Yearbook o the nationalreading conerence(pp. 334348). Oak Creek, WI: National Reading Conerence.]

No Child Le Behind Act o 2001, 20 U.S.C. 6301 e seq (2001) PL 107110).

Tompson, S. J., Johnstone, C. J., & Turlow, M. L. (2002). Universal design applied tolarge scale assessments (Synthesis Report 44). Minneapolis, MN: University o Minnesota,

National Center on Educational Outcomes.

Tompson, S. J., Johnstone, C. J., Turlow, M. L., & Clapper, A. . (2004). Stateliteracy standards, practice, and testing: Exploring accessibility(echnical Report 38).Minneapolis, MN: University o Minnesota, National Center on Educational Outcomes.

Defining Reading Proficiency.pdf

Documents

Transcript of Defining Reading Proficiency.pdf