Practical Issues in Computerized Testing:

A State Perspective

Patricia Reiss, Ph.DHawaii Department of Education

Agenda

Contextual Information on CAT Implementation in Hawaii Shadow Tests Scoring of Incomplete Adaptive Tests Speededness

Contextual Information

- Hawaii State Assessment: Online CAT since 2010- Smarter Balanced Assessment: CAT and PT- Computer Adaptive Algorithm

- Blueprint alignment: content (claims, targets, CCSS), cognitive complexity, and item types

- Maximize achievement level classification for total test

and reporting categories - Minimize overall measurement error- Stopping rule – test length

Flow

Select Eligible Items (not seen before)

Assign Starting Scores

Calculate Value for Each Item

Recalculate score

Administer item

Select highest valuedItem (or randomly from

highest)

Recalculate score

Field Test?

Go Back?

Recalculate score

Capture responsechange

Next item alreadyAdministered?

Recalculate score

Select field testitem

Update Constraints

Used with permission from American Institutes for Research

Practical Considerations

To allow for adaptations items in CAT are selected sequentially. But each of these practical problems implies constraints on item selection that can only be realized if all items are selected simultaneously.

Wim van der Linden

Practical ConsiderationsComputing resourcesNumber and cost of serversNetwork Bandwidth Reduce number of reassembled shadow

tests by requiringfixed intervals between reassembliesa certain minimal change in the interim

ability estimate

AIR Summative Simulation Report for Smarter Balanced March 2015

AIR Summative Simulation Report for Smarter Balanced March 2015

AIR Summative Simulation Report for Smarter Balanced March 2015

AIR Summative Simulation Report for Smarter Balanced March 2015

Scoring of Incomplete Adaptive Tests

For the Smarter Balanced online assessments, A student is required to provide a

response to an item before s/he can go on to the next item

However, the student is able to go back and change his or her answers

Scores for Incomplete Tests

For incomplete Smarter Balanced Assessments for which students answer at least 10 CAT items, the Consortium will derive a score based on the students’ actual responses to items and will consider as incorrect any item to which students do not provide a response. The difficulty of a student’s omitted items from the combination of online computer adaptive and performance tasks will be estimated based on the average difficulty of the items in the item pool.

For incomplete tests for which students answer fewer than 10 CAT items, the Consortium will assign the lowest possible scale score for the grade and content area.

Sub-scores are not reported for incomplete tests.

Differential Speededness

“Test items vary considerably in the amount of time they require.”

Examinees vary in how much time they require to answer test items.

Actual Testing TimesSmarter

Average testing time Max Test Time Min Test Time StdDev of Test TimeGrade 3 ELA 01:54:45 10:43:05 00:01:30 00:58:50Grade 3 Math 01:29:11 14:30:45 00:01:40 00:52:51Grade 4 ELA 02:05:19 12:09:37 00:02:10 00:59:41Grade 4 Math 01:39:38 10:51:03 00:01:42 00:54:07Grade 5 ELA 02:13:39 17:56:36 00:02:35 01:13:40Grade 5 Math 01:49:35 10:53:13 00:01:46 01:01:33Grade 6 ELA 02:08:01 12:17:17 00:01:44 01:01:32Grade 6 Math 01:43:07 13:58:16 00:01:36 00:48:51Grade 7 ELA 01:44:40 07:23:41 00:03:59 00:42:54Grade 7 Math 01:39:49 09:19:03 00:04:03 00:42:49Grade 8 ELA 01:45:30 09:19:16 00:04:11 00:42:56Grade 8 Math 01:39:58 19:20:24 00:02:22 00:45:04Grade 11 ELA 01:25:54 06:15:35 00:02:14 00:36:26Grade 11 Math 01:14:00 05:46:47 00:01:59 00:35:25

All test taken by 6/12/2015

Implementation Issues Related to “Time” Number of Computers

Bandwidth Prioritization and Management Highest priority given to assessments

Gaming, audio and video sites were blocked

Start times of tests staggered

School level feedback

Impact on duty free time for teachers

Bathroom breaks for teachers

Move students who require more time or keep all students in the same room

Conflict with lunch schedule

Loss of instructional time, extra-curricular activities, assemblies, field trips, etc.

Thank you!

Patricia_Reiss@notes.k12.hi.us

(808) 733-4100