Recommending Ohio Computer Based Assessment (OCBA) Performance … · 2015. 9. 2. · Based...

Recommending Ohio Computer Based Assessment (OCBA) Performance Standards Social Studies: Grades 4 and 6, and End-of Course Assessments in US History and US Government

Science: Grades 5 and 8, and End-of-Course Assessments in Physical Science and Biology Technical Report

September 2, 2015

Ohio Computer Based Assessments

Summer 2015 Standard Setting

Recommending Performance Standards for Ohio’s Computer Based Assessments

Social Studies: Grades 4 and 6, and End-of-Course Assessments in US History and US Government

Science: Grades 5 and 8, and End-of-Course Assessments in Physical Science and Biology

Technical Report

September 2, 2015

Prepared by American Institutes for Research (AIR) in collaboration with the Ohio Department of Education

OCBA Standard Setting Technical Report

ii

TABLEOFCONTENTS

Introduction .................................................................................................................................... 1

Performance Standards and Validity of Test Score Interpretations ........................................... 1

Overview of Standard Setting Approach ........................................................................................ 5

Workshop Design ........................................................................................................................ 5

Workshop Location ..................................................................................................................... 6

Workshop Staffing ...................................................................................................................... 6

Workshop Panelists .................................................................................................................... 6

Workshop Training ...................................................................................................................... 8

Standard Setting Materials and Procedures ................................................................................... 9

Performance Level Descriptors ................................................................................................... 9

Ordered Item Booklet ............................................................................................................... 10

OCBA Bookmark Placement ...................................................................................................... 12

Benchmark Information ............................................................................................................ 13

Panelist Feedback and Impact Data .......................................................................................... 14

Vertical Articulation .................................................................................................................. 15

Workshop Evaluation ................................................................................................................ 16

Recommended Performance Standards and Impact Data ........................................................... 17

Evaluation of the Standard Setting Workshop ............................................................................. 21

Panelist Evaluation of Standard Setting Workshop .................................................................. 21

References .................................................................................................................................... 23

Appendix A – Workshop Agendas ................................................................................................. 24

Appendix B – Composition of Panels ............................................................................................ 25

Appendix C – Training Presentations ............................................................................................ 26

Appendix D – Performance Level Descriptors .............................................................................. 27

Appendix E – Summary of Ordered Item Booklets ....................................................................... 28


iii

Appendix F – OIB Item Data Plots ................................................................................................. 29

Appendix G –Bookmark Placement Readiness Forms .................................................................. 30

Appendix H – Panelist Evaluation Forms ...................................................................................... 31

Appendix I – Recommend Performance Standards by Round ..................................................... 32

Appendix J – Convergence of Bookmarks across Rounds ............................................................. 33

Appendix K – Estimated Percentage of Students at Each Performance Level for Panelist Recommended Performance Standards, Overall and by Gender and Ethnicity .......................... 34

Appendix L – Summary of Panelist Evaluations ............................................................................ 35


1 American Institutes for Research

Introduction Ohio adopted new academic content standards in social studies and science, in 2010 and 2011, respectively. Ohio’s New Learning Standards in social studies and science are designed to ensure that students across grades are receiving the instruction they need to become scientifically literate and civic minded citizens equipped with knowledge and skills for the 21st century workforce and able to successfully transition to higher education. In spring 2015, the Ohio Department of Education (ODE) administered for the first time Ohio’s Computer Based Assessments (OCBA) in social studies and science to assess proficiency with respect to Ohio’s New Learning Standards. The OCBA assesses student achievement of social studies standards in grades 4 and 6, and following high school coursework in US History and US Government. Science achievement is assessed in grades 5 and 8, and following instruction in physical science and biology in high school.

The OCBA is a series of fixed form assessments that are intended to be administered online, although the assessment is offered as a dual mode, online and paper, assessment to accommodate schools that are not ready to transition to the online testing environment. For spring 2015, the OCBA assessments were administered in two parts. A performance‐based assessment (PBA), which included any items requiring hand scoring, was administered during an initial test administration window. A second test window was used to administer the end‐of‐year (EOY) assessments, which included only machine scored item types.

The first operational administration of the OCBA assessment took place in spring 2015. The paper and online administrations of the PBA occurred from March 2 through April 10. The paper and online administrations of the EOY were administered between April 4 and April 22. Following the close of the test administration windows, the American Institutes for Research (AIR), under contract to ODE, convened eight panels of Ohio educators to recommend performance standards on the assessments. This document describes the procedures used to conduct the standard setting workshops as well as the recommended performance standards and resulting impacts.

Performance Standards and Validity of Test Score Interpretations

Validity refers to the degree to which test score interpretations are supported by evidence, and speaks directly to the legitimate uses of test scores. Establishing the validity of test score interpretations is thus the most fundamental component of test design and evaluation. The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014) provide a framework for evaluating whether claims based on test score interpretations are supported by evidence. Within this framework, the Standards describe the range of evidence that may be brought to bear to support the validity of test score interpretations.



The kinds of evidence required to support the validity of test score interpretations depend centrally on the claims made for how test scores may be interpreted. Moreover, the standards make explicit that validity is not an attribute of tests, but rather test score interpretations. Some test score interpretations may be supported by validity evidence, while others are not. Thus, the test itself is not considered valid, but rather the validity of the intended interpretation and use of test scores is evaluated.

Central to evaluating the validity of test score interpretations is determining whether the test measures the intended construct. Such an evaluation in turn requires a clear definition of the measurement construct. For Ohio’s new OCBA assessments in science and social studies, the definition of the measurement construct is provided by Ohio’s New Learning Standards.

Ohio’s New Learning Standards specify what students should know and be able to do by the end of each grade level in order for students to graduate ready for post‐secondary education or entry into the workforce. Because directly measuring student achievement against each benchmark in the Standards would result in an impractically long test, each test administration is designed to measure a representative sample of the content domain defined by the Standards. To ensure that each student is assessed on the intended breadth and depth of the Standards, test form construction is guided by a set of test specifications, or blueprints, which indicate the number of items that should be sampled from each content strand, standard, and benchmark. Thus, the test blueprints represent a policy statement about the relative importance of content strands and standards in addition to meeting important measurement goals (e.g., sufficient items to report strand performance levels reliably). Because the test blueprint determines how student achievement of Ohio’s New Learning Standards is evaluated, alignment of test blueprints with the content standards is critical. ODE has published the OCBA test blueprints that specify the distribution of items across reporting strands and depth of knowledge levels.

Alignment of test content to Ohio’s New Learning Standards ensures that test scores can serve as valid indicators of the degree to which students have achieved the learning expectations detailed in the Standards. However, the interpretation of the OCBA test scores rests fundamentally on how test scores relate to performance standards which define the extent to which students have achieved the expectations defined in the Standards. OCBA test scores are reported with respect to five proficiency levels, demarcating the degree to which Ohio students have achieved the learning expectations defined by Ohio’s New Learning Standards. The cut score establishing the Proficient level of performance is the most critical, since it indicates that students are meeting grade level expectations for achievement of Ohio’s New Learning Standards and that they are prepared to benefit from instruction at the next grade level. The Accelerated level is also of critical importance, because performance at this level is intended to indicate that students are on track to pursue post‐secondary education. Procedures used to adopt performance standard for the OCBA assessments are therefore central to the validity of test score interpretations.



Following the first operational administration of the OCBA assessments in spring 2015, a standard setting workshop was conducted to recommend to the Ohio State Board of Education a set of performance standards for reporting student achievement of Ohio’s New Learning Standards. This document describes the standardized and rigorous procedures that Ohio educators, serving as standard setting panelists, followed to recommend performance standards. The workshops employed the Bookmark procedure, a widely used method in which standard setting panelists use their expert knowledge of Ohio’s New Learning Standards and student achievement to map the performance level descriptors adopted by the Ohio State Board of Education onto an ordered item book based on the first operational test form administered to students in spring 2015.

Panelists were also provided with contextual information to help inform their primarily content driven performance standard recommendations. Panelists recommending performance standards for the high school EOC assessments were provided with information about the approximate location of the relevant ACT college ready performance standard. Panelists recommending performance standards for American History and American Government were provided with the college ready benchmark for the ACT reading assessment. Panelists recommending performance standards for the grade level summative assessments were provided with the approximate location of relevant NAEP performance standards. As with the EOC assessments, panelists recommending performance standards for social studies were provided with NAEP reading benchmarks, using an interpolated value for grade 6. Panelists recommending performance standards for the science assessments were provided with science assessment benchmarks for NAEP and TIMSS at grade 8, and an interpolated TIMSS benchmark for grade 5. Additionally, panelists were provided the corresponding locations for the previous OAA and OGT performance standards. Panelists were asked to consider the location of these benchmark locations when making their content‐based cut‐score recommendations. When panelists are able to use benchmark information to locate performance standards that converge across assessment systems, validity of test score interpretations is strengthened.

In addition, following recommendation of performance standards in each of the panels, panelists were provided with feedback about the vertical articulation of their recommended performance standards so that they could view how the locations of their recommended performance standards for each of the grade level and EOC assessments sat in relation to the cut score recommendations for the other assessments. This approach allowed panelists to view their cut score recommendations as a coherent system of performance standards, and further reinforces the interpretation of test scores as indicating not only achievement of current grade level standards, but also preparedness to benefit from instruction in the subsequent grade level.

Based on the recommended cut scores, Table 1 shows the estimated percentage of students meeting or exceeding the OCBA proficient and accelerated standards for each assessment in spring 2015. Table 1 also shows the approximate percentage of Ohio students that would be expected to meet the ACT college ready standard, and the percentage of Ohio students meeting the NAEP proficient standards at grades 4 and 8, and as interpolated for grades 5 and



6. Table 1 also presents the expected proficient rate for the TIMSS science assessment at grade 8. As Table 1 indicates, the accelerated performance standards recommended for the OCBA assessments are generally consistent with relevant ACT college‐ready and NAEP proficient benchmarks. Moreover, the rates for proficient and accelerated across grade levels are generally consistent.

Table 1. Estimated Percentage of Students Meeting or Exceeding OCBA and Benchmark Proficient Standards.

Percent of Students Meeting or Exceeding Standard

Assessment OCBA

Proficient OCBA

AcceleratedOhio ACT

College ReadyOhio NAEP Proficient

National TIMSS High

Social Studies

Grade 4 70 29 37 Grade 6 57 36 38 American History 71 35 37 American Government 67 18 37

Science

Grade 5 62 38 47 Grade 8 60 37 38 40 Physical Science 63 22 26 Biology 60 27 26



Overview of Standard Setting Approach The Bookmark method (Mitzel, Lewis, Patz, & Green, 2001) was used to recommend performance standards for the OCBA. ODE previously used the Bookmark method to recommend performance standards for the OGT and OAA assessments. The Bookmark method was implemented in two rounds, providing panelists with benchmark information during OIB review prior to Round 1, impact data during Round 1, and panelist feedback prior to Round 2.

Panelists were tasked with recommending four performance standards (Basic, Proficient, Accelerated, and Advanced) that resulted in four performance levels (Limited, Basic, Proficient, Accelerated, and Advanced). ODE has designated the Accelerated performance level as indicating that students are on track for success in college level coursework.

Workshop Design

To recommend performance standards for each of the OCBA assessments, ODE convened eight panels, one for each assessment. The panels consisted of educators representing the respective grade level and content area assessments. The panelists recommended performance standards based primarily on content considerations with additional context provided by relevant benchmark information from national (NAEP), international (TIMSS), and college entrance (ACT) exams, as well as estimated student performance at the achievement level represented by each page in the OIB. Panelists used Ordered Item Booklets (OIBs) and Performance Level Descriptors (PLDs) to place performance standards demarcating all five performance levels, Limited, Basic, Proficient, Accelerated, and Advanced, in two rounds. After recommending performance standards for their designated assessments, a moderation session was conducted with the table leaders from each of the panels to review the articulation of performance standards across the grade level and end‐of‐course assessments, and to implement any adjustments to their recommendations to facilitate articulation across grade levels.

The OCBA Standard Setting workshops were conducted over two days, beginning Thursday, August 6, and concluding Friday, August 7. Detailed agendas for the standard setting workshops are included as Appendix A.

The workshops began with a brief table leader orientation to review with table leaders their role and responsibilities. The workshop proper began with a large group training to provide panelists with an overview of the workshop activities and initial training in the bookmarking procedures. Following the large group session, the workshop panels convened in their meeting rooms, and began their work by participating in the same OCBA online assessment that was administered to their students in the spring. Because the Biology EOC exam was not offered in the spring, panelists reviewed the test items using the web preview tool in the item banking system. Panelists then spent several hours working through the performance level descriptors (PLDs) developed by ODE, and developing modified descriptors to characterize the special subset of students who just barely qualify for entry into each of the performance levels. After



developing descriptors for the just barely students, panelists spent the remainder of day one reviewing their ordered item books (OIBs).

Panelists did not begin recommending performance standards until day two, which began with training on the bookmark placement task. Panelists then worked through their OIBs and placed their bookmarks for Round 1. After Round 1, panelists were provided feedback about the bookmark placements of the other panelists and discussed those bookmark placements at their tables and across the room more generally. Panelists were then also provided with impact data showing the estimated percentage of students who would meet each of the Round 1 recommended performance standards and engaged in panel discussions about any implications of those proficiency rates. Upon completion of panel discussions, panelists made a second round of bookmark placements.

Workshop Location

The workshops were held at the Quest Center, located at 8405 Pulsar Place in Columbus, Ohio. The location provided meeting spaces to hold the OCBA workshop panels, as well as a psychometric work room for completion of analysis activities and storage space for secure materials throughout the workshop.

Workshop Staffing

A senior workshop coordinator was tasked with leading the cross‐workshop introductory training and cross‐grade moderation meetings, and was responsible for working with each facilitator and monitoring the flow of activities across workshops. AIR test development staff served as workshop facilitators, leading each panel through training activities and execution of the standard setting process. Additionally, an AIR content area specialist was assigned to each panel to support the workshop facilitator. Because test development staff served as workshop facilitators, they were highly qualified to facilitate the development of just barely performance level descriptors, and to serve as a subject matter resources for panelists as they navigated the OIB. A team of two AIR psychometricians managed psychometric activities in support of the workshop, including ensuring accurate data capture of bookmark placements, presentation of cross‐grade articulation results for moderation meetings, and production of final results for the standard setting technical report. In addition, AIR project staff facilitated organization of meeting space and provided support to panelists as necessary.

ODE staff monitored all standard setting activities, and also addressed any policy or test development questions for panelists. While ODE staff answered specific, direct questions, they were not actively involved in the facilitation of the meeting.

Workshop Panelists

ODE worked to obtain broadly representative panels for the standard setting workshops that reflected the teacher population in the state of Ohio in terms of gender, race, ethnicity, and geographical representation. Diverse groups of panelists bring a wide range of perspectives and



experience to the standard setting effort, ensuring that the recommendations that are forwarded to the State Board of Education are thoughtful and representative of broad educational constituencies, and represent the range of expertise and experiences found in the educator population across the state.

Within each of the science and social studies panels, a total of 10 panelists per panel were recruited to recommend standards. ODE targeted the number of male and female panelists to mirror the population of educators. In the same way, ODE worked to recruit proportional representation of American Indian/Native American, Asian/Pacific Islanders, Black (Non‐Hispanic), Hispanic and White (Non‐Hispanic) panelists, and a proportional number of panelists from rural, urban, and suburban districts. In addition, ODE worked to include special education and English Language Learners (ELL) teachers.

Within each panel, tables were balanced to include panelists with varying content expertise and demographic representation in each group.

ODE designated two table leaders for each panel. Table leaders attended an additional orientation meeting and were tasked with assisting standard setting staff by

facilitating discussions within their table;

distributing and collecting readiness and recording sheets and secure materials;

alerting workshop staff of confusion or concerns within their tables; and

representing their table and panel during vertical articulation meetings.

Electronic invitations containing logistical information and reminders about the purpose of the workshop were emailed to confirmed panelists. Ten days prior, panelists received an email confirming their attendance along with hotel confirmation numbers if applicable. Throughout the process, ODE continued to recruit replacements for panelists who withdrew their participation.

Appendix B presents the composition of the standard setting panels. For each panel, the table includes a record for each panelist and indicates the district he or she represents and his or her gender, ethnicity, and current position and main area of expertise. While it is critically important to include a range of stakeholders in the standard setting process, experience has shown that it is essential for panelists to have direct knowledge of academic standards and student grade‐level performance to participate meaningfully in the Bookmarking procedure. For this reason, panel participation was restricted to classroom teachers and curriculum specialists with expertise in science and social studies curriculum and instruction.



Workshop Training

Thorough training is an essential element of a standard setting workshop. Training at the meetings helped panelists become familiar with the assessment system and the standard setting process. It also involved a review and discussion of the assessments, the student populations that participated in each, and the performance level descriptors (PLDs). In addition, training included in‐depth discussion of concepts key to bookmark placement, such as the notion of what would constitute a student “just barely” in a performance level. All panelists were administered an operational test in order to understand the test content, the testing interface, and various item types through which student knowledge and skills were assessed. A sample of the presentation slides used to conduct the introductory training, and those used to facilitate each workshop are provided in Appendix C.

To begin the workshop, the panelists were convened for a brief introductory training that focused on the purpose of the standard setting workshop and a review of the main workshop activities. Following this large group introduction, panelists joined their assigned workshop panels where the workshop leader for each assessment guided panelists through the standard setting activities and provided in‐depth training throughout the course of the workshop.

Table leaders had the additional responsibilities of ensuring that table activities remained focused on the task at hand, helping to verify that panelists understood their tasks, and alerting workshop leaders to any issues encountered by panelists as they engaged in their workshop tasks. Table leaders were not expected to provide training to panelists but rather serve as liaisons between the panelists and workshop leaders to ensure that workshop activities were implemented correctly, alerting workshop leaders to any issues that arose during the course of conducting workshop activities, and representing their tables in the cross‐panel moderation deliberations. A table‐leader orientation meeting was convened prior to the standard setting workshop to familiarize table leaders with their roles and responsibilities, including suggestions on how to provide leadership at the tables during the standard setting process and how to manage the secure materials.



Standard Setting Materials and Procedures

Performance Level Descriptors

Performance level descriptors (PLDs) define the content area knowledge and skills that students at each performance level are expected to demonstrate. The standard setting panelists based their judgments about the location of the performance standards on the PLDs as well as Ohio’s New Learning Standards.

Prior to convening the standard setting workshops, ODE, in consultation with AIR, drafted PLDs for each test that described the range of achievement encompassed by each performance level on the test. The PLDs were designed to be clear, concrete, and reflect Ohio’s expectations for proficiency based on Ohio’s New Learning Standards. Performance level descriptors that were used by panelists in the standard setting workshop are presented in Appendix D.

Central to their training in the bookmark method, panelists used the PLDs to develop a representation of students who just barely qualify for entry into each of the performance levels. During this training task, panelists learned that while PLDs are written to characterize the range of achievement characterized by each performance level, their bookmark placements would be directed toward identifying and characterizing the most minimally qualified members of each performance level. Characterizing just barely meets students is not an intuitive judgment and panelists worked to identify the minimum characteristics of student achievement to qualify for entry into each performance level. Each panel produced a just barely PLD to help guide their discussions and bookmark placements. To develop a common understanding among panelists, each panel was asked to

1. review and parse performance level descriptors;

2. discuss characteristics of students classified near thresholds of performance standards;

3. identify the characteristics that distinguish students just above the performance standard from those just below;

4. determine what evidence was necessary to conclude that a student possessed the minimum knowledge and skills needed to meet the performance standard; and

5. summarize knowledge and skills of students who “just barely” meet each performance standard, or are “just barely” described by each performance level descriptor

These discussions yielded common descriptions of students just barely characterized by each performance level descriptor within each room.



Ordered Item Booklet

Following review of performance level descriptors and development of “just barely” performance level descriptors, panelists reviewed ordered item booklets (OIBs). An OIB is a collection of test items ordered from easiest to most difficult. Each page in the OIB corresponds to a level of achievement on the OCBA, and panelists used the OIB to recommend the minimum level of achievement required to enter into each performance level.

Composition of OIB

Within each of the OCBA science and social studies assessments, all online test takers were administered a test form with a common set of items used for operational scoring, as well as a set of embedded items used for field testing new items. The operational test form was also administered on paper with item substitutions for a few technology‐enhanced items that could not be represented on paper. The operational items administered online served as the basis for the ordered item book.

To minimize gaps in the ordered item booklets, the OIBs were augmented by additional field‐test items to more fully represent the range of academic achievement encompassed within those item banks. Each science OIB was augmented with 16‐20 field test items, and each social studies OIB was augmented with 6‐14 field test items. All field test items selected for inclusion in the OIB were reviewed for statistical integrity; items flagged for further review due to low discrimination were excluded from the OIB. It is important to note that when selecting items to augment the OIBs, the alignment of items to the blueprint was considered to maintain to the degree possible proportional representation of items to the test blueprint.

Increasing the number of items across the range of item difficulties provides panelists with greater context to identify important shifts in the knowledge and skill requirements of test items. Often panelists become focused on the cognitive demands of a single item when deliberating on the location of a performance standard. This propensity is exacerbated when there are relatively few items in a given location, which can cause judgments about one item to take on too much importance. Even when there are sufficient items to establish reliable performance standards for a central proficient performance standard, there are typically fewer items available in locations associated with performance standards categorizing achievement well below or above the proficient standard; thus, movement of the bookmark by even a page or two may result in very large increases or decreases in the percentage of students meeting the standard. Augmenting the OIB moderates the impact associated with each OIB page, especially for performance standards in the tails of the ability distribution.

Items were ordered according to their response probability (RP) level based on their Item Response Theory (IRT) parameters. In IRT, the item characteristic curve for each item indicates the likelihood of responding correctly for each point along the student achievement distribution. The response probability criterion refers to the location on the achievement scale that corresponds to a given probability of success. In the context of the standard setting workshop, this criterion is used to develop a common understanding of what constitutes



mastery when evaluating whether a student can respond successfully to an item. An RP value of 0.67 was used as the mastery criterion for all of standard setting workshops except the high school end of course assessments in science. Panelists were asked to consider whether, for example, a just barely proficient student had a 0.67 likelihood of answering the item correctly. They were also encouraged to ask this question in other related ways, including whether ⅔ of just barely proficient students would answer the item correctly, or whether a just barely proficient student would respond correctly to the item two of three times. The end‐of‐course science tests were very difficult and the number of items on which students could demonstrate that level of mastery was quite low, resulting in a very short functional OIB. Thus, an RP value of 0.50 was adopted for the EOC science tests, meaning that a just barely proficient student, for example, had a 0.50 likelihood of responding correctly, or that ½ of just barely students could respond successfully to an item, or that a just barely student could respond successfully to the item at least one of two times.

Dichotomously scored (e.g., incorrect vs. correct) OCBA items were calibrated using the Rasch model. Multi‐point, partial credit items were calibrated using Masters’ partial credit model with ordering of score point pages in the OIB based on step‐level difficulties.

The ordered item booklets were presented online, allowing panelists to view items in the same context as student test takers. The composition of the science and social studies ordered item booklets by assessment and grade are summarized in Table 3 below. A technical summary of the OIBs is presented in Appendix E, including for each page in the OIB, the item score point associated with the presented item, the difficulty represented by the page, and the standard error of the difficulty. In addition, the appendix indicates the overall percent of students who would score at or above the standard associated with each OIB page, and the location of external benchmarks within the booklet.

Table 3. Composition of Ordered Item Booklets

Number of Items In OIB

Pages in OIB (Total Points)Test Operational Field Test Total

Social Studies

Grade 4 Social Studies 45 9 54 74 Grade 6 Social Studies 42 6 48 70 American History 45 11 56 79 American Government 44 14 58 81

Science

Grade 5 Science 43 16 59 74 Grade 8 Science 44 20 64 77 Physical Science 44 16 60 73 Biology 43 20 63 78



Review of Ordered Item Booklets

For each item in the OIB, panelists were instructed to ask what a student must know and be able to do to answer the question (or achieve the score point, for partial credit items) and what makes each item in the OIB more difficult than the preceding items. This review of the OIB allowed panelists to gain new perspectives on the knowledge and skill requirements of items and to share information regarding their thoughts on the location of threshold regions in the OIB. During this discussion, the workshop leader circulated through the room to monitor progress, to assist panelists who might have had trouble with the task, and to answer any questions.

On each page in the OIB, panelists viewed the content of the item, content alignment, and the scoring key or rubric. Panelists were also provided with a paper item map which they could use while navigating the OIB, which also included content alignment and scoring key information for each page in the OIB. In addition, panelists were presented with an item plot that provided a graphical representation of the difficulty of each page in the OIB; this tool showed where page item difficulties were clustered together versus spread out. OIB item plots are presented in Appendix F.

OCBA Bookmark Placement

Prior to making their Round 1 bookmark placements, panelists were provided training in the identification of performance standards in the ordered item booklets. As part of this training, panelists learned to identify a location in the OIB that best delineates two performance levels (e.g., between pages on which students must demonstrate mastery to meet the minimum requirements for membership in the Basic level from those items on which demonstration of mastery is not necessary).

Using their just barely PLDs as a guide, the panelists were then instructed to set a bookmark on the item that best delineated each of the performance levels. Panelists were reminded how to set bookmarks, and prior to making initial placements, facilitators led a group activity that reviewed the key concepts of the bookmark procedure, allowing facilitators to provide additional training if necessary. Prior to placing recommended performance standards in each round, panelists were asked to complete a readiness form to indicate their preparedness to recommend performance standards. This form asked panelists to assert their understanding of the tools used to recommend performance standards in each round. If a panelist indicated that they did not feel prepared to recommend performance standards, the workshop leader provided additional training and opportunities for discussion. All panelists had to indicate that they felt prepared to move forward before they could place a bookmark. All OCBA standard setting panelists indicated they understood the task at hand and felt ready to recommend performance standards. Samples of readiness forms used for completing the bookmark task are presented in Appendix G.

Bookmark placement was conducted in two rounds, allowing panelists to make independent judgments while still benefiting from discussion with their fellow panelists. Panelists were



instructed to identify their recommended cuts for Accelerated, Proficient, Basic, and Advanced, in that order, in each round. The placement of the bookmark is illustrated in Figure 1. Each panelist used their just barely PLDs to identify which item represented the lower bound of each performance level. In the example, a panelist concluded that students who were just barely at the “Proficient” level would demonstrate mastery on the item on the page indicated by the arrow, while students below the “Proficient” level would not. Therefore, the panelist decided that the Proficient performance level would begin on the page indicated by an arrow. The panelist believed that students below the “Proficient” performance level would not be able to demonstrate mastery of items beyond the indicated page in the ordered‐item booklet.

Figure 1. Example of Bookmark Placement

Benchmark Information

Panelists were charged with recommending an Accelerated performance standard comparable to other important assessment systems, including national and international assessments such as NAEP and TIMSS, and the ACT college entry exams. To facilitate comparisons of Ohio performance standards with other national and international benchmarks, panelists were provided with the locations of performance standards from these other assessments systems in their OIBs. In particular, performance standard locations for the following assessments were provided as part of panelists’ OIB review:



NAEP performance standards in reading in grades 4 and grade 6 (interpolated) for social studies, and grade 8 science,

TIMSS performance standards in grade 5 (interpolated) and grade 8,

ACT college ready performance standard in science for science EOC tests, and reading for social studies EOC tests, and

Ohio’s previous OAA and OGT assessments.

Panelist Feedback and Impact Data

Prior to Round 2, panelists were provided feedback about the bookmark placements made by fellow panelists. After making their Round 1 bookmark placements, panelists reconvened and began with a discussion of panelist feedback about the bookmark locations recommended by each panelist, beginning with table level feedback and discussion, and progressing to room level discussion. Each table spent time reviewing and discussing cut score placements, focusing on the lowest and highest recommended performance standards both at the table and across the panel. Panelists were asked to review the items between the lowest and highest performance standards at their table, discussing the standards and the just barely PLDs. Discussion was then expanded to the room level, with each table reviewing the basis for their own recommendations for the group at large.

Following discussion of panelist feedback, panelists were presented with impact data, the percentage of students expected to score at or above the recommended Round 1 performance standards. Panelists discussed any implications of the impact data, both at their tables and across the panel more generally, focusing on whether the impact was in line with their expectations. Following presentation of impact data, panelists were provided, for each item in the OIB, the percentage of students expected to achieve the ability level indexed by that page.

After completing their discussions, panelists again worked through the OIB, placing their Round 2 bookmarks for all four performance levels, beginning with Accelerated and Proficient, and followed by Basic and Advanced.

Estimating Student Performance Data

While the OCBA OIBs were constructed based on calibration of the online testing population, the percentage of students within the state meeting or exceeding each potential performance standard (i.e., each page in the ordered item booklet) was estimated based on all students participating in the first operational administration of the assessment, including students who tested online and students who tested on paper.

A matched samples approach was used to estimate the effects of mode on student performance. Previous year student achievement results, as well as demographic information, including gender, ethnicity, income level status, English language learner (ELL) status,



Individualized Education Program (IEP), were used to identify matched samples for the mode comparability analyses.

With matched samples in hand, item parameters were calibrated separately for the matched samples of paper and online test administrations, and the linking constants necessary to bring the paper item parameters onto the online reference scale were identified. The mode linking constants were generally quite small, indicating little or no effect of test administration mode on student performance. Nevertheless, for the purpose of estimating student impact for the standard setting workshop, the mode linking constants were applied to the paper item parameters to estimate student ability for paper test administrations. Thus, the percentage of students estimated to meet or exceed each potential performance standard on the OCBA was based on all students who participated in the operational assessment.

Prior to Round 2 of the Bookmark procedure, the percentage of students meeting the standards, based on the Round 1 median cut score, was presented to panelists.

Vertical Articulation

Performance standards should ideally be articulated across grades. Unless there are systematic differences in the quality of instruction across grades, the expectation is that students who meet the standards and are prepared for instruction in the subsequent grade will likely continue to meet standards as they progress through their school years, and that therefore we would not expect to see large changes in the proficiency rates from grade to grade. While this vertical articulation is incorporated into the development of Ohio’s New Learning Standards, because the assessments are administered only occasionally across the elementary and middle school grades, and are not reported on a common vertical reporting scale, the degree to which the assessments can be vertically articulated is limited. Nevertheless, lack of articulation across grade can result in confusion, especially when there are unreasonably large shifts in student performance‐level classifications from grade to grade. Thus, to the extent possible, there should be consistency in the percentage of students meeting the critical performance standards across the grade level and EOC assessments.

In the absence of a vertical scale, articulation was considered only from the perspective of the percent of students meeting standards across grades and courses.

To help foster consistency in the identification of performance standards across grades, after performance standards were recommended for each of the science and social studies assessments, table leaders were convened to participate in a vertical moderation session. Table leaders were shown the percentage of students scoring at or above each of the performance standards, and the percent of students classified at each performance level across tests. Where the percentage of students expected to meet standards varied greatly between grade‐ or course‐based assessments, table leaders were asked to consider modifications to the recommended standards that would achieve a more articulated system of standards. In these instances, table leaders reviewed the ordered item booklets and considered whether the content of the OIB supported the adjustment. Thus, while table leaders worked to articulate



standards across grades, they also ensured that any changes resulting from the moderation meeting be consistent with the knowledge and skills described in the PLDs.

Workshop Evaluation

Panelists were encouraged to provide feedback concerning the procedures and outcomes of the standard setting workshop throughout the process, via group discussions, practice activities, and completion of readiness forms prior to placing their bookmarks.

At the completion of the workshop, panelists were asked to complete a workshop evaluation form designed to elicit feedback on all aspects of the workshop, including clarity of training and tasks, appropriateness of the time spent on activities, and satisfaction with the outcome of the workshop. Samples of the evaluation forms are presented in Appendix H.



Recommended Performance Standards and Impact Data

For the OCBA assessments in science and social studies, Appendix I presents the minimum, maximum, and median bookmark placements for each round of bookmark placements, as well as any bookmarks placed during the moderation session, and the resulting final recommendations following the standard setting workshops. As panelists discussed the reasons for their bookmark placements in the context of feedback from other panelists and impact data, variability across tables often decreased across rounds. The figures in Appendix J, Convergence of Bookmarks across Rounds, illustrate variability in median table bookmark placements for the three performance standards over the two rounds. These figures illustrate how variability in bookmark decisions changed from the first to the second round. In general, there was considerable consistency in the placement of performance standards across rounds.

For each test, final recommended performance standard is the outcome from the final moderation, or in the absence of moderation, the median bookmark page following Round 2.

The final recommended performance standards for each assessment are presented in Table 4, along with the projected impact each performance standard would have on Ohio public school students tested in 2015. The final recommended OIB page numbers are the median bookmarks of each panel following Round 2 bookmark placement, and any subsequent moderation.

Table 4. Final Recommended Performance Standards for OCBA

Test Performance

Level

Ordered Item

Booklet Page Theta

Estimated Percentage of Students At or

Above Performance Standard

Approximate Percentage of

Raw Score Points

Grade 4 Social Studies

Basic 8 ‐0.92 88 33% Proficient 19 ‐0.4 70 44% Accelerated 41 0.57 29 64% Advanced 62 1.58 5 81%


Basic 15 ‐0.22 77 44% Proficient 30 0.36 57 58% Accelerated 44 0.97 36 70% Advanced 60 1.71 13 83%

American History


America Basic 10 ‐1.11 90 27%



Test Performance

Level

Ordered Item

Booklet Page Theta

Estimated Percentage of Students At or

Above Performance Standard

Approximate Percentage of

Raw Score Points

n Government

Proficient 22 ‐0.41 67 39% Accelerated 49 0.92 18 69% Advanced 69 1.66 4 81%

Grade 5 Science


Grade 8 Science


Physical Science

Basic 6 ‐1.56 87 20%

Proficient 19 ‐0.94 63 29%

Accelerated 45 0.02 22 48%

Advanced 63 0.95 4 70%

Biology


Note: Following the standard setting workshop, recommendations are submitted to Ohio’s State Board of Education. Performance standards are not final prior to approval and adoption by the Board.



Table 5 shows the estimated percentage of student classified at each performance level based on final panelist‐recommended standards for each of the OCBA science and social assessments. The results of Table 5 are represented graphically in Figure 2, for science, and Figure 3 for social studies. Appendix K presents the estimated percentage of students classified at each performance level disaggregated by gender and ethnicity.

Table 5. Percentage of Students at Each Performance Level based on Final Recommended Performance Standards

Test Limited Basic Proficient Accelerated Advanced

Science

Grade 5 Science 12% 26% 24% 21% 17% Grade 8 Science 18% 22% 23% 27% 10% Physical Science 13% 24% 41% 18% 4% Biology 21% 19% 33% 9% 17%

Social Studies

Grade 4 Social Studies 12% 19% 41% 24% 5% Grade 6 Social Studies 23% 20% 21% 22% 13% American History 12% 18% 36% 17% 18% American Government 10% 23% 49% 14% 4%

Figure 2. Percentage of Students at Each Performance Level based on Final Recommended Performance Standards—OCBA Science

12.2 18.0 13.321.5

25.921.9

23.719.0

23.9 23.240.6 32.8

20.727.1

18.0

9.5

17.39.8 4.4

17.2

G5 Science G8 Science Physical Science Biology

Percentage of Students at Each Performace Level for OCBA Science

Level 1 - Limited Level 2 - Basic Level 3 - Proficient Level 4 - Accelerated Level 5 - Advanced



Figure 3. Percentage of Students at Each Performance Level based on Final Recommended Performance Standards—OCBA Social Studies

11.523.3

11.7 10.1

18.8

20.1

17.5 23.2

40.721.0

36.2

48.6

24.3

22.3 16.5

13.6

4.713.3 18.0

4.5

G4 Social Studies G6 Social Studies American History American Government

Percentage of Students at Each Performace Level forOCBA Social Studies

Level 1 - Limited Level 2 - Basic Level 3 - Proficient Level 4 - Accelerated Level 5 - Advanced



Evaluation of the Standard Setting Workshop

Panelist Evaluation of Standard Setting Workshop

Following the completion of standard setting activities, panelists were asked to evaluate different aspects of the workshop, and the resulting recommendations. Evaluation results were generally consistent across panels, so Appendix L summarizes panelists’ responses to the evaluation forms across all panels. Most panelists indicated that training on the main components and tools of the bookmark procedure was adequate, and that they understood how to use each component. There were a few panelists, from across all panels, who evaluated the workshop materials and training negatively, mainly with the PLDs. However, we also note that there were two panelists who indicated strong disagreement for items in the first section, but strong agreement to items in subsequent section, suggesting perhaps mistakes in completing their evaluation form.

Generally, panelists indicated that the amount of time allotted for different activities within the standard setting workshop was “about right.” Overall, panelists expressed general satisfaction with the workshop and offered suggestions for improving the experience in future meetings.

Across all panels, all but one participant indicated that they agreed that students classified at accelerated and proficient performance levels are classified accurately based on the knowledge and skills described in the Ohio’s New Learning Standards, as summarized in Table 7.

Table 7. Summary of Panelist Evaluation of Recommended Performance Standards

Strongly Disagree Disagree Agree Strongly

Agree

a.

A student performing at the Accelerated – Level 4 demonstrates a strong command of Ohio’s Learning Standards for Science and Social Studies. (Level 4)

1 0 36 39

b.

A student performing at the Proficient – Level 3 demonstrates an appropriate command of Ohio’s Learning Standards for Science and Social Studies. (Level 3)

1 0 43 32

c.

A student performing at the Basic – Level 2 demonstrates a partial command of Ohio’s Learning Standards for Science and Social Studies. (Level 2)

1 2 44 29

d. A student performing at the Advanced – Level 5 demonstrates a distinguished 2 0 29 45



command of Ohio’s Learning Standards for Science and Social Studies. (Level 5)



References Ferrara, S., Perie, M., & Johnson, E. (2008). Matching the judgmental task with standard setting

panelist expertise: The Item‐Descriptor (ID) Matching procedure. Journal of Applied Testing Technology, 9(1).

McLaughlin, D., Scarloss, B. A., Stancavage, F. B., & Blankenship, C. D. (2005). Using State Assessments to Impute Achievement of Students Absent from NAEP: An Empirical Study in Four States. Washington, DC: American Institutes for Research. Retrieved from www.air.org/files/McLaughlin_AbsentStudents.pdf

Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The Bookmark procedure: Psychological perspectives. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Erlba

http://www.air.org/files/McLaughlin_AbsentStudents.pdf


Appendix

Appendix A – Workshop Agendas


A-1 American Institutes for Research

Document A1. OCBA Social Studies Grades 4 & 6, American History & American Government and Science Grade 5 & 8, Physical Science & Biology Standard Setting Agenda August 6 - 7, 2015

Meeting Location: Quest Business Center; Columbus, Ohio

Day 1: August 6, 2015

Standard Setting Workshop Orientation for Table Leaders

7:45 – 8:00 Registration

8:00 – 8:30 Table Leader Orientation

Standard Setting Introduction – all panelists in large group room

8:00 – 8:30 Registration

8:30 – 8:45 Welcome and introductions from Ohio Department of Education

8:45 – 9:45 Introduction to Standard Setting from AIR

• Welcome and Introductions • Purpose of standard setting workshop • General overview of standard setting procedures and key concepts

o Performance Level Descriptors o Ordered Item Book o Recommending Standards o Panelist feedback and impact data

9:45 – 10:00 Break, and separate into breakout rooms

Standard Setting Workshop – in breakout rooms

10:00 – 10:15 Introductions within panel, and signing of non-disclosure forms

10:15 – 11:15 Experience operational test in online testing environment

11:15 – 12:15 Review Performance Level Descriptors (PLDs) and summarize skills of students that are “just barely” described by each PLD

• Training on comparing PLDs across performance level • Discussion of just barely • Panelists review PLDs and summarize “just barely” skills

12:15 – 1:15 Lunch

1:15 – 2:45 Continue review and discussion of of just barely summary skills

2:45 – 4:30 Review of the Ordered Item Book (OIB)

• Training on composition of the OIB • What do students need to know and be able to do to respond successfully to

each page in the OIB?


A-2 American Institutes for Research


• Why is each page more difficult than the previous pages?

4:30 Adjourn


Standard Setting Workshop

8:00 – 8:30 Registration and morning refreshments 8:30 – 9:15 Group discussion of OIB and areas of transition

9:15 – 10:15 Training on bookmark placement task

• Discussion of mastery/ response probability • Introduction to Bookmark Placement procedures • Practice bookmark placement activity

10:15– 11:45 Round 1 bookmark placement (4 cuts)

• Review of bookmark procedures and key concepts • Completion of Bookmark Placement Readiness Form • Round 1 Bookmark Placement (4 bookmarks)

11:45 – 12:45 Lunch, and concurrent production of feedback data

12:45 – 1:45 Presentation and discussion of outcomes following Round 1

• Panelist agreement data • Impact data

1:45 – 2:45 Round 2 bookmark placement

• Completion of Bookmark Placement Readiness Form • Round 2 bookmark placement

2:45 – 3:15 Panelist evaluation of standard setting workshop, and concurrent production of final results

3:30 – 4:30 Debrief of workshop results and cross-panel moderation with table leaders, if required

*Only table leaders are required to participate in final debrief/ Moderation

4:30 Adjourn


Appendix

Appendix B – Composition of Panels


B-1 American Institutes for Research

Table B1. Composition of Grade 4 Social Studies Panel

Table Table Leader District Gender Ethnicity Current Position Education

Experience/Certification

1 Yes Avon Lake City Schools Female White Teacher Elementary, Secondary

1 No LaBrae Local Schools Female White 4th Grade Social Studies Elementary

1 No Ohio History Connection Female White Project Coordinator No Response

1 No Cleveland Metropolitan Schools Female Black Instructional Coach Elementary,

Instructional Coach

1 No Fort Frye Local Schools Female White 4th Grade Social Studies and

Language Arts Elementary, Secondary,

Special Education

2 Yes Loveland City Schools Male White 4th Grade Teacher Elementary

2 No Antioch University Midwest No

Response No Response Faculty Elementary

2 No Imagine Schools Female White Testing and Title I Coordinator Elementary,

Administrative, Gifted and Talented

2 No Plain Local School District Female White 4th Grade Teacher Elementary

2 No Wadsworth City Schools Female White 4th Grade Social Studies Teachers Middle Childhood

Education



Table B2. Composition of Grade 6 Social Studies Panel



1 Yes Plain Local School District Male White Teacher Elementary

1 No Bloom-Carroll Female White 6th Grade social Studies Elementary, Special

Education

1 No Brookville Local Schools Female White Teacher, 6th Grade Social Studies Elementary, Secondary

1 No Cleveland Metropolitan Schools Female White Teacher Elementary

1 No Crestview local schools Female White 6th Grade Social Studies Elementary

2 Yes The University of Findlay Female White Professor of Education, Chairperson

for Licensure

Elementary, Secondary, Middle School, Administrative

2 No Sidney City Female White 6th Grade Social Studies Elementary

2 No Grand Valley Local Schools Female White 5th/6th Grade Social Studies Elementary

2 No Metro Early College Middle School Female White 6th Grade Social Studies teacher Middle Level Grades 4-9

2 No Ridgewood Local Schools Male White Director of Curriculum, Assessment

and Special Education Administrative



Table B3. Composition of American History Panel



1 Yes West Geauga Local Schools Male White American History

Teacher/Department Head Secondary

1 No ECOT Female White Instructional Supervisor Secondary,

Administrative

1 No Black River Local Schools Male White History Teacher Secondary

1 No Cincinnati Public Schools Male White Teacher Secondary

1 No Stow Munroe Falls City Schools Female No Response Teacher Secondary

2 Yes Indian Creek Female White Social Studies Department Chair, US

history Teacher Secondary

2 No Toledo Federation of Teachers,

Toledo Public Schools Female Arab

History teacher, Vice President, Toledo Federation of Teachers (TFT),

Director of Human Trafficking and Social Justice Issues (TFT)

Secondary , National Board Certified in

History

2 No East Cleveland Female Black Intervention Specialist Secondary, Special

Education

2 No Boardman Local Male White Social Studies Department Chair Secondary

2 No Mohawk Local School Male White Social Studies Teacher Secondary, Special

Education, Post-Secondary



Table B4. Composition of American Government Panel



1 Yes Huron City Schools Female White

HS American Government & Economics, Dual Enrollment

Government, AP Macroeconomics, Honors US History

Secondary

1 No Fort Frye Local School District Male White Social Studies Teacher 7-12 Elementary,

Administrative, Assistant Principal

1 No Akron Public Schools No

Response No Response Teacher/Department Chair Secondary

1 No Perrysburg Local Schools Male White American Government Teacher Secondary

1 No Garfield Heights City Schools Female White Curriculum Leader, Teacher Secondary

2 Yes Northwest Local School District Female White Social Studies Instructional Specialist Secondary

2 No Mahoning County Educational

Service Center Male White Instructional Supervisor Secondary

2 No Electronic Classroom of Tomorrow Female White Curriculum Developer Secondary

2 No Akron Public Schools No

Response No Response Teacher/Department Chair Secondary

2 No Perrysburg Local Schools Male White American Government Teacher Secondary



Table B5. Composition of Grade 5 Science Panel



1 Yes Cincinnati Public Schools Female White 5th Grade Teacher Elementary

1 No Cincinnati Public Schools Female White 6th Grade Science and Social Studies

Teacher Elementary

1 No Huron City Schools Female Hispanic or Latino 5th Grade Science and Social Studies Elementary

1 No Rocky River City Schools Female White 5th Grade Science Teacher Elementary

1 No Jackson Local Schools Female White Grade 5 Science Teacher Elementary

2 Yes Anna Local Schools Female White Director of Student Achievement Secondary,

Administrative, College Adjunct

2 No ESC of Central Ohio Female White Innovation Specialist Elementary, Secondary,

Administrative, Curriculum & Instruction

2 No Ridgewood Local Schools Female White 5th Grade Teacher Elementary

2 No Firelands Female White 4th Grade Math Elementary

2 No Northwest Local School District Female White Curriculum Supervisor Secondary



Table B6. Composition of Grade 8 Science Panel



1 Yes Clark-Shawnee Local Female White 7th and 8th Grade Science Elementary

1 No Pickaway-Ross Career & Technical

Center Male White Science Instructor Secondary

1 No Plain Local School District Female White Science Instructor Secondary

1 No Columbus Catholic Diocese Female White 6-7-8 Science Teacher Elementary, Middle

School

1 No Hubbard City Schools Female White Middle School Math Elementary, Secondary

2 Yes Beavercreek City Schools Female White Curriculum Supervisor Elementary, Special Education (Gifted)

2 No Bright Local School District Male White Jr High Science Teacher Secondary

2 No Huntington Local Schools Female White 9th Grade Physical Science teacher Secondary,

Administrative

2 No TRECA Digital Academy Female White Middle School Math/Science Secondary

2 No Granville Exempted Village

Schools Female No Response Science 8 Teacher No Response



Table B7. Composition of Physical Science Panel



1 Yes Miamisburg City Schools Female White Teacher Secondary

1 No Buckeye Local Schools Male White Teacher Secondary

1 No Ohio Connections Academy Female White Physical Science Teacher Secondary

1 No Granville Exempted Village School

District Female No Response Physics Teacher No Response

1 No Batavia Local Schools Male White Science Teacher Secondary

2 Yes Findlay City Schools Female No Response Science teacher No Response

2 No Triway Female White Teacher Secondary

2 No Columbus Male White Teacher Secondary

2 No Ohio Virtual Academy Female White Science Teacher Secondary

2 No Lima City Schools Female White Intervention Specialist Elementary, Special

Education



Table B8. Composition of Biology Panel



1 Yes Fairland Male White Teacher Secondary

1 No Alexander Local Schools Female White Biology teacher Secondary

1 No Archdiocese of Cincinnati Female White Biology teacher/Department Chair Secondary

1 No Dover City Schools Female White Teacher Secondary

1 No Worthington City Schools Female White Biology Teacher Secondary

2 Yes Firelands Local Schools Male White Biology Teacher Secondary

2 No Toronto City Schools Female White Biology/Anatomy Teacher Secondary

2 No South Point Local Female Asian Administrator Secondary,

Administrative

2 No James A Garfield Female White Science Teacher Secondary

2 No Cleveland Metropolitan School

District Female Black

Classroom Biology/Physical Science Teacher

Secondary


Appendix

Appendix C – Training Presentations

The purpose of this workshop is to recommend a system of performance standards to the State Board of Education. For each of the OCBA assessments, the workshop panels will recommend four performance standards or cut scores: Basic - Level 2, Proficient - Level 3, Accelerated - Level 4, and Advanced - Level 5. These will be used to classify students into one of five performance levels: Limited - Level 1, Basic - Level 2, Proficient - Level 3, Accelerated - Level 4, and Advanced - Level 5.

To set valid, meaningful cut scores, that are publicly verifiable, standard setting workshops are conducted. The standard setting workshops employ research-based procedures that are used by committees of educators to establish cut scores on a state's assessments. Performance standards impact students and the education system statewide. The procedures you will engage in are designed to give you the tools to make informed judgments that yield defensible recommendations that can be submitted to state board of education for adoption.

2

OCBA

Presentation C.1. Large Group Training Presentation and Script Standard Setting Workshop

C-1 American Institutes for Research

In this workshop, we will be recommending performance standards for Social Studies and Science. There are 8 separate panels which will work independently.

Each panel is comprised of about 10 panelists, split into two tables.

Each table has a designated table leader who will help to distribute and collect materials, and who can serve as a liaison between your table and the workshop staff. Please make sure to leave all secure materials in the rooms or turn into your table leader if directed by the workshop leader.

3

We will begin the workshop with an overview of the standard setting process. The panels will employ the Bookmark procedure to recommend standards. You will be trained on the specifics of these methods as you go through each step of the process within your own groups.

Following this introduction, each of you will have an opportunity to participate in the same assessments administered to students this spring. You will take the grade and subject test for your assigned committee in the online testing environment that students experienced.

Following that, you will work with other members at your table to review the Performance Level Descriptors which define the knowledge and skill requirements of students at each level.

Next, you will review a book of test items ordered from easiest to most difficult based on actual student performance to recommend performance standards, thinking about what students have to know and be able to do in order to respond successfully to each item. This is referred to as your Ordered Item Booklet or OIB. You’ll then recommend performance standards by identifying pages in the OIB that serve as cuts for different levels of achievement. You’ll receive and discuss feedback on your initial recommendations with your fellow panelists, and then make

4

OCBA



another individual recommendation. Then, you’ll receive further feedback and other performance information to provide additional context to your recommendations.

Following final recommendations, we will conduct a debriefing and we will ask you to complete a workshop evaluation form before you leave. The evaluation forms you will receive throughout the process are an important part of the standard setting and you are asked to complete them thoroughly and thoughtfully.

American Institutes for Research 4

Facilitator Note: Please stress the importance of security. These are operational items that will be used on future administrations. Absolutely no picture taking, cell phones are to be put away and not out at the table, no texting while working. We understand emergencies may happen, please take all calls outside the meeting room. Do not surf other websites while using the laptops.

We can’t stress the importance of security enough. We’ll be working with live test items that will be administered to students again in the future, and it’s important from a test score validity perspective that items remain secure. That means that we ask that you keep cell phones, tablets, laptops away, and step outside if you need to take a call.

5

OCBA



Standard setting refers generally to the process of identifying a passing score on a test. The central question of the standard setting process is to identify the level of performance on a test that indicates a passing, or good enough, performance.

A passing or good enough performance is determined by the purpose of the assessment. Tests may, for example, certify minimum competence or select out only the highest performers.

The OCBA are criterion referenced tests, meaning that they directly measure a representative sample of the knowledge and skills that students are expected to achieve by the end of each school year. Therefore, we will employ a test-centered approach to setting performance standards. In test-centered approaches, cut scores are established based on the degree to which students demonstrate achievement of knowledge and skills measured directly in the assessment. For this reason, test-centered approaches depend critically on having participants in the standard setting who are very knowledgeable about the state's content standards and willing to help the state define the level of knowledge and skill expected of a student at each performance level demonstrated by the cut scores.

6

The standard setting process that we will be engaged in during the next several days is designed to translate the Ohio academic content standards in Science andSocial Studies into a set of performance standards, or cut scores. Two important documents, the Ordered Item Booklet and the Performance Level Descriptors, will be your primary tools for translating the academic content standards into performance standards.

Throughout this workshop, we will refer to different types of “standards.”

Academic content standards specify what students should know and be able to do by end of each academic year.

Performance standards specify how much of the content standards students must know and be able to do in order to meet each performance level. You will recommend four performance standards, or cut scores, for each grade.

7

OCBA



Performance levels are regions on the achievement scale demarcated by the performance standards. They classify students by how much of the content standards they know and are able to do. The four performance standards will result in five performance levels.

8

There are many methods for setting performance standards, including examinee-centered and test-centered. In some employment applications, for example, tests may be administered to groups of people who are more or less successful, and a cut score is identified that best differentiates the two groups. This is an example of an examinee-centered approach.

However, because the OCBA are criterion-referenced assessments, meaning that they measure a representative sample of the academic content that students are expected to know and be able to do by the end of each school year, we are employing a test-centered approach to recommended performance standards. In other words, successful performance of items on the test speaks directly to students’ performance of the standards.

Modern standard setting approaches generally use an Ordered Item Booklet, or OIB, with test items ordered from easiest to most difficult to help panelists and to streamline the standard setting process.

The Bookmark method is widely used for many years in statewide assessment programs, and has been used to set performance standards for various other state assessments.

9

OCBA



You will discuss the mechanics of the Bookmark procedures in much greater detail within your own groups.

Performance Level Descriptors, or PLDs, are detailed descriptions of the knowledge and skills students are able to demonstrate with respect to the academic content standards at each level. In particular, we’ll be concerned with a special group of students, those who just barely meet the performance level descriptors.

The Ordered Item Booklet, or OIB, contains operational and other test items that were administered in spring 2015 as well as other items to fill in information gaps, ordered from easiest to most difficult.

Once you have developed descriptions of students who just barely meet the PLDs and complete your review of the OIB, you will be ready to recommend performance standards for each of the proficiency levels. You will recommend performance standards in multiple rounds.

Although you will have plenty of opportunities to discuss bookmark placements with our fellow panelists, your bookmark placements represent you individual recommendations and you will make those recommendations independently from

10

the other panelists. After your initial bookmark placements, you will have an opportunity to discuss your recommendations in context of your fellow panelists’ recommendations, and affirm or revise your own recommendations in round two.


OCBA



Performance Level Descriptors will serve as your guide for identifying performance standards. The PLDs describe what skills students in each level are expected to have. When you recommend a standard, you are asserting that students who meet that level of performance fit the description of the student’s abilities provided in the PLDs. This link is critical because it allows teachers and families to understand what a student’s test results mean.

To thoroughly review the PLDs, it is helpful to parse the standards. You may find, for example, performance levels are differentiated by the verbs used in the PLDs – for example, students may recognize, identify, understand, explain, and so on.

You will be asked to pay careful attention to the content and skill demands required at each performance level. It is critical for you to understand that when you recommend a cut score, you are asserting that students who perform at that achievement level meet the content and skill requirements described in the PLD.

11 12

OCBA



We will compare the performance level descriptors across performance levels.

13

OIB is the other primary tool you will use. The Ordered Item Booklets will present you with the items from the spring 2015 assessments. The content of the items is proportional to the test blueprint.

The OIB consists of all operational items as well as 15-20 field test items to fill in information gap. What that means is that we added items where there was a large gap in item difficulty between operational items so that the items in the OIB appear more fluid in terms of difficulty.

All items will be presented in order of difficulty; page 1 will present the easiest item, and the last page will present the hardest item. Some items will be represented more than once. These items are more engaging than others and are worth more than a single score point. Each “page” in the ordered item booklet will refer to specific score point of the item, In each room you will review the series of items in an online environment.

14

OCBA



15

Items are ordered from easiest to most difficult. The OIB provides a picture of the range of knowledge and skills encompassed by the items on the test,and is a vehicle use to make cut score judgments.

The OIB should provide useful information about natural breaks in the knowledge and skill requirements necessary to consistently perform successfully across a range of item content.

For each performance standard, you will place a bookmark on the page that divides the OIB into two sets of page ranges: pages that students at a particular level can reliably respond correctly to, and the pages that the students at that level cannot respond correctly to.

When studying the OIB, it is important to understand the difference between items in context of the whole OIB. Items that appear earlier in the book are easier, despite perceptions otherwise, than items that appear later in the book.

For each item, ask yourself two questions:1) What do students need to know and be able to do to respond successfully to

this item?2) Why is this item more difficult than the previous items?

While the items will be presented to you online, you will be able to record your notes about these questions to refer to as you decide where to place your bookmark.

16

OCBA



The difficulty of these items is based on students’ performance during the 2015 operational assessment. So while a particular item may seem easier or harder to you, the placement in the ordered item book reflects how easy or hard it actually was for students to perform successfully on these items.

If an item seems out of order to you, remember that an item may not measure what you think it measures. For example, an item may intend for a student to have to know a particular piece of information, but perhaps the students were able to answer using recall from a lesson that was taught recently.

Instead of focusing on one item that may seem out of place to you, try to identify natural breaks or thresholds for groups of items.

17 American Institutes for Research 18

OCBA



There are two important concepts that you will discuss in order to understand your bookmark task:1. The idea of “just barely” meeting a standard, or a student that is “just barely”described by a performance level descriptor, and 2. A common understanding of what mastery means.We will discuss each of these in turn.

19

“Just Barely”Even within each performance level, students vary in the degree to which they have mastered the Ohio State Standards. Some students have just barely crossed the line between Basic - Level 2 and Proficient - Level 3, while others are getting ready to cross the line between Proficient - Level 3 and Advanced - Level 4. In general, the PLDs are written to describe the performance of students in the middle of the category. However, we actually want to focus our attention on a specific subset of students within each performance level, those who have “just barely” entered into the performance level. They are not the typical example of a Level 4 student, and may not be what you picture when asked to describe a Level 4 student, but they do still just meet the criteria described by the PLD, and Level 4 student.

20

OCBA



You will spend time in your workshop rooms reviewing the PLDs, and thinking about what knowledge and skills students that are just barely described by PLD have to have, and what separates them from students who are not described by the PLD.

To frame this, we’ll think about:1. Students who fall near each performance standard or level – what characterizes

these students?2. What differentiates students who just meet the performance standard from those

that do not – what can they do, or not do, that categorizes them on either side of the standard?

Each room will produce a summary of “just barely” skills for each performance level.

21

To place bookmarks, you will find the location in the OIB that differentiates students who are “just barely” from those that are not. To do this, you will evaluate whether “just barely” students can respond successfully to each item in the OIB.

In order to make this judgment, we need to develop a common understanding of what it means to perform successfully on an item. When we say that “just barely” students can perform successfully on an item, do we really mean that such students will always get the item correct? We don’t typically operate in absolutes. Students don’t always get items correct, for a variety of reasons. Instead, we say that students consistently perform successfully on items or tasks. In a similar vein, for the purpose of this workshop, we will define successful performance as a response probability of 67%, which is referred to as RP67, meaning that we wish to identify the location in the OIB where students who are just barely Proficient have a 2/3 chance of responding correctly to the item. You can think about this as a way to define what it means to say that a student can reliably answer an item correctly –they won’t always answer it correctly, but they can reliably answer it.

We can think about this concept in two different ways – if you picture one “just barely” student, they have a 67% chance of responding correctly to the item. Alternatively, if you visualize a group of 100 “just barely” students, two thirds of the group will respond to the item correctly.

22

OCBA



When you place bookmarks, you will work through each page of the OIB and determine whether 2/3 of just barely Level 3 students, for example, can respond successfully to the item on each page. This judgment will be the basis for recommending a bookmark.


In each room you will review the Ordered Item Book which presents a long series of items, ordered from easiest to most difficult. While reviewing, remember your focus will be to determine what students need to know and be able to do in order to respond to each item successfully, and why each item is more difficult for students than the items before.

In the coming days you will make performance standards recommendations by identifying a page number of the OIB that will serve as the cut.

For each performance level, you will work through the OIB and consider whether 2/3 of “just barely” students can respond successfully to the item. You will place your bookmark on the last page where 2/3 of students who just barely meet the

performance standard will answer correctly. This means that fewer than 2/3 of just barely meets students would be expected to respond successfully to the next item in the OIB.

23

OCBA



24 25

OCBA



After you complete your initial bookmark placement, you will be provided feedback about how other panelists placed their bookmarks. You will receive feedback about the bookmark placements for your table and also see how the bookmark placements across tables compare.

This data can serve as a start for discussion about bookmark placements and help panelists to develop common understandings of the skills a “just barely” student has, taking into account the varied backgrounds and expertise of your fellow panelists. Once you have had a chance to review the feedback data within your table, we will expand the discussion to other tables in your grade.

From these discussions, panelists may revise their judgments and choose to move their bookmark placement in Round 2, but there is no expectation that panelists will move bookmarks. Generally, we do see convergence from Round 1 to Round 2, but consensus is not a goal.

26

You will also be presented with impact data for each subject after the first round of cuts. This is the percentage of students who would reach or exceed the standard based on the item page in the Ordered Item Booklet. With this information, you will ask yourself if the outcome seems reasonable. While impact data can be informative, placement of your bookmarks should always be guided by content considerations to ensure that students meeting the performance standard are accurately described by the PLD for each level.

When you receive the impact data, you will want to ask yourself whether the impact of the current cut score placement seems reasonable and in the neighborhood of what you were expecting. If the impact data seems out of line with your expectations, consider why that might be. If the impact is that fewer students meet the standards than you expected, might it be, for example, that the new academic content standards are more rigorous and require students to demonstrate greater knowledge and skills than previously? If the impact is that more students meet the standards than you expected, could it be that you underestimated the knowledge and skills that students can reliably demonstrate?

Consider your cut score recommendations in the light of the impact data, and discuss the implications of the current cut score placements with your fellow panelists. Remember, while you may choose to modify your cut scores in light of the

27

OCBA



impact data, your rationale for making each cut score should be based on content considerations.

American Institutes for Research 27 28

OCBA




This concludes our large group training session. Please break into your assigned groups. Your panel assignments should be included in your folders as well as room numbers, which are also currently displayed on the screen. Please locate an AIR or ODE employee, as indicated by our badge, if you require any assistance. Thank you.

30

OCBA



31

Note for facilitator: Introduce workshop staff, and have panelists introduce themselves to the group. Encourage panelists to share names, school district/region, and what grades and subjects each panelist works with.

Let’s go around the room and introduce yourself, and share what school district or area in the state you are coming from, and what students you primarily work with.

32

OCBA

Presentation C.2. Sample Workshop Presentation and Script (Grade 4 Social Studies) Standard Setting Workshop


In the left side of your folder, there is a non-disclosure forms. The first order of business is to sign those and hand them in.

33

Facilitator Note: Please stress the importance of security. These are operational items that will be used on future administrations. Absolutely no picture taking, cell phones are to be put away and not out at the table, no texting while working. We understand emergencies may happen, please take all calls outside the meeting room. Do not surf other websites while using the laptops.

We can’t stress the importance of security enough. We’ll be working with live test items and past OCBA items that will be administered to students again in the future, and it’s important from a test validity perspective that items remain secure. That means that we ask that you keep cell phones, tablets, laptops away, and step outside if you need to take a call.

34

OCBA



Notes to elaborate on:• You will experience a subset of items administered in spring 2015• The interface is almost identical to the online test environment that the student

view and experience• You cannot see your scores for hand scored items because they are scored at a

later time• There is only an hour reserved for experiencing the Online assessment• The purpose is not to complete the test, but to get an idea of what the students

experienced

35

You will now have the opportunity to experience test questions similar to what was administered to students this spring.

Note: Secure browser should be deployed on each panelist’s computer. Additional workshop staff will circulate rooms to expedite log-in process.

36

OCBA



American Institutes for Research 37 American Institutes for Research 38

OCBA



39 40

OCBA



The standard setting process is designed to translate the Ohio's New Learning Standards in Social Studies into a set of performance standards, or cut scores, on each of the assessments. Two important documents, the Ordered Item Booklet and the performance level descriptors, will be your primary tools for translating the academic content standards into performance standards.

Throughout this workshop, we will refer to different types of “standards.”

Ohio's New Learning Standards specify what students should know and be able to do by end of each academic year.

We are going to use two tools, the OIB and PLD’s.

41

Performance level descriptors will serve as your guide for identifying performance standards. The PLD is a summary of what students within each achievement level are expected to know and be able to do. The PLD’s are the link between content and performance standards. This means that when you recommend a standard, you are asserting that students who meet that level of performance fit the description of the student provided in the PLDs. This link is critical because it allows teachers and families to understand what a student’s test results mean.

42

OCBA



The Ordered Item Book, or OIB, is the other primary tool you will use. The OIB contains operational test items administered in Spring 2015 ordered from easiest to most difficult. Each page is a score point on an item. Certain items appear multiple times in the OIB, once for each score point. The number of pages in the book is equal to the number of points in the OIB, not the number of items on the OIB.

You will identify how much a student should know and be able to do to meet the description for each performance level in the PLDs by placing a bookmark in the OIB that divides the book into two groups: items that students described by the performance level descriptor can respond successfully to, and items that students in that performance level cannot respond successfully to

You will have two different opportunities to make individual recommendations – you’ll make an initial judgment, and then you will receive feedback showingthe bookmarks of your fellow panelists. We’ll discuss everybody’s bookmarks, and we’ll also look at the percentage of students in the state who would meet or exceed each of the recommended standards, or impact. Then, you’ll make individual recommendations again – you can change your bookmarks, but you don’t have too.

43

Performance levels are regions on the performance scale demarcated by the performance standards. They classify students by how much of the content standards they know and are able to do. The four performance standards will result in five performance levels.

44

OCBA



Begin PLD review with panelists.


We can compare the Performance level descriptors across Performance levels for each content standard..

46

OCBA



This is an example of the skills students must present in order to reach the Level 4 PLD.


Let’s compare similar constructs across performance levels. You can see how skills are similar but different across performance levels. Questions about this idea?

(If asked for examples, original text included below:)

Limited - Level 1 – Name a First Amendment right (e.g., freedom of religion, speech, press,petition and assembly);Basic - Level 2 – Define a constitution (e.g., a written plan for government);Proficient - Level 3 – Identify the purpose of a constitution (e.g., organizing the structure of government, protecting basic rights);Accelerated - Level 4 – Describe the ways laws are used to protect the rights of citizens;Advanced - Level 5 – Explain how the Constitution limits the power of government;

48

OCBA




Even within each performance level, students vary in the degree to which they have mastered Ohio’s New Learning Standards. Some students have just barely crossed the line between Basic - Level 2 and Proficient - Level 3, while others are getting ready to cross the line between Accelerated - Level 4 and Advanced - Level 5. In general, the PLDs are written to describe the performance of students in the middle of the category. However, we actually want to focus our attention on a specific subset of students within each performance level, those who have “just barely” entered into the performance level.

To frame this, we’ll think about:1. Students who fall near each performance standard – what characterizes these

students?2. What differentiates students who just meet the performance standard from those

that do not – what can they do, or not do, that categorizes them on either side of the standard?

3. Descriptions of how much of the content standards students who just barelymeet the performance standard have to know and be able to do in order to be categorized in each performance level.

50

OCBA



As we go through and review the PLDs we will think about what high level skills are necessary for a student just entering each performance level.

Remember, the student has “just barely” crossed into the performance level. They demonstrate just enough to be considered Basic – Level 2, Proficient – Level 3, Accelerated – Level 4, and Advanced – Level 5.

Lets review the PLDs in the BLANK reporting category as a group to come up with a summary of the overarching skills necessary for the “just barely” students in each performance level.

Then we will assign different strands to each table to produce their own “just barely” summary statements, then come back together as a group and share what each table has produced and discuss any questions. When thinking about what a “just barely Accelerated” student can do, ensure you are not describing a Proficient –Level 3 student.Facilitator – instruct panelists to review the PLD then raise their hands to provide their input during the group activity

51

(example based on the activity)

(CHANGE THIS TO MATCH YOUR SUBJECT AREA)

Instructions for facilitator: talk through the PLDs for the performance level first. Have some examples of high level skills that could be necessary for a just barely student. Throw them out to the committee to comment on. This activity shouldn’t take more than 20 min but it should help them to see what their task is when they are asked to complete the reporting categories by table.

Ask the committee:

What is important for a just barely student to be able to demonstrate? How does this differ from the upper range of the adjacent performance level?

You should have something prepared for EACH reporting category even through you will only do the first together. This is to help individual tables think through this

52

OCBA



task so they don’t get stuck.


OCBA



Discuss the just barely summaries across tables.Encourage the tables to take notes on these documents and ask questions if they do not understand or agree with the descriptions.Everyone should be on the same page in understanding the skills of “just barely” students.**Ensure discussion is moving along and not stuck on one particular strand

54

Now we’ll turn our attention to your next task: review the Ordered Item Book.

The Ordered Item Booklets will present you with a subset of items from the spring 2015 assessments. All items will be presented in order of difficulty; page 1 will present the easiest item, and the last page will present the hardest item. Each “page” in the ordered item booklet refers to one score point of one item.

Typically, the OIB has been presented literally as a bound paper book. However, because the OCBA includes technology enhanced items that cannot be represented well on paper, you will use an electronic OIB. The OIB is ordered from the easiest to most difficult item, so the first item is the easiest, and the last item is the most difficult. You page forward to see increasingly more difficult items, and you can page backward to see progressively easier items. You will page through the OIB from easiest to the most difficult item. For each item, you will ask what students need to know and be able to do to respond successfully to the item, and what makes this item more difficult than the preceding items.

You can use the accompanying OIB map to keep notes.

Next, we will review:1. What to consider when reviewing the OIB2. How to log into the OIB, and how to navigate through each page of the OIB3. How your OIB Map corresponds to the OIB and your review of each item

55

OCBA



OIB is the other primary tool you will use. The Ordered Item Booklets will present you with the items from the spring 2015 assessments. The content of the items is proportional to the test blueprint.

The OIB consists of all operational items as well as 15-20 field test items to fill in information gap. What that means is that we added items where there was a large gap in item difficulty between operational items so that the items in the OIB appear more fluid in terms of difficulty.

All items will be presented in order of difficulty; page 1 will present the easiest item, and the last page will present the hardest item. Some items will be represented more than once. These items are more engaging than others and are worth more than a single score point. Each “page” in the ordered item booklet will refer to specific score point of the item. We will review the series of items in an online environment.

56

When studying the OIB, it is important to understand the difference between items in context of the whole OIB. Items that appear earlier in the book are easier, despite perceptions otherwise, than items that appear later in the book.

For each item, ask yourself two questions:1) What do students need to know and be able to do to respond successfully to this

item?2) Why is this item more difficult than the previous items?

57

OCBA



The difficulty of these items is based on students’ performance during the 2015 operational assessment. So while a particular item may seem easier or harder to you, the placement in the ordered item book reflects how easy or hard it actually was for Ohio students to perform successfully on these items.

If an item seems out of order to you, remember that an item may not measure what you think it measures. For example, an item may intend for a student to have to know a particular piece of information, but perhaps the students were able to answer using recall from a lesson that was taught recently.

Instead of focusing on one item that may seem out of place to you, try to identify natural breaks or thresholds for groups of items.


OCBA




OCBA



NOTE: Slides regarding benchmark data will be updated following identification of various benchmarks, and approval of what should be given to educator panels.

As you review the OIB, we are going to provide you some contextual information to help you better understand the achievement levels associated with each page in the OIB.

One piece of feedback you will see is the general location of the ACT college-ready cut score.The other is the

You will use this for information of the general location of other important achievement indicators.


The item map will guide your review of the OIB.

• Remember that items are presented in order from easiest to most difficult. Eachpage number represents one item.

• The ITS ID is shown on both the item map and in your online OIB. You can jumpto an OIB page by selecting the ITS ID from the dropdown in the top right portionof the screen.

• The Content Alignment column shows you the benchmark of each item. Thiscorresponds to the benchmarks in your test specifications document.

• Item format displays whether an item is multiple choice, indicated by “MC”, or anitem that requires a student to construct a response, such as “GI” meaning “griditem”, “NL” meaning “natural language,” where students type free responses. Asyou review the OIB, items that are not multiple choice will contain instructions onhow students are to answer.

• Space for notes – as you review each item in the OIB, remember to think abouttwo questions – 1) What do students need to know and be able to do in order torespond successfully to this question?, and 2) Why is this question more difficultthan the one before?

63

OCBA



FACILITATORS: Move to step “review of OIB”

Facilitators – username is “firstname_lastname” and your personal password is written on your agendas.

Walk through the different “Review Panel” options• More about this item• Notes• Marks• Impact• Feedback• Moderation• Prior Feedback

64

1. Panelist name should appear2. You will see the title of the grade/subject you are working on3. Step should say “Review of Ordered Item Booklet”4. Page forward/backward

65

OCBA



66

(Leave slide up while panelists review OIB)(Update with time and room information for reconvening)

67

OCBA



68 69

OCBA



NOTE: Panelist computers will be logged into the ITS and OIB prior to meeting start. Once panelists arrive, they can immediately start reviewing the OIB. As needed, assist panelists with logging into ITS and display password at front of room. Once all panelists arrive, review the activities for the morning and help panelists pace their review of the OIB, and reiterate key concepts on following slide.

We will spend the morning working through the Ordered Item Book. First, we will ask you to move through the OIB and review all items independently at a fluid pace. We will then review items within your tables. Following review of the OIB, we’ll discuss how to locate the “just barely” in the OIB and recommend cut scores. We will recommend cut scores and spend time reviewing feedback discussion our recommendations as a group before making another round of recommendations. Follow the second round of recommendations, you will receive performance information such as the percent of students estimated to meet performance standards based on the room’s median recommended bookmark page numbers. We will discuss the implications of the performance information, and you will again make a final round of recommendations.


OCBA



To place bookmarks, you will find the location in the OIB that differentiates students who are “just barely” from those that are not. To do this, you will evaluate whether “just barely” students can respond successfully to each item in the OIB.

In order to make this judgment, we need to develop a common understanding of what it means to perform successfully on an item. When we say that “just barely” students can perform successfully on an item, do we really mean that such students will always get the item correct? We don’t typically operate in absolutes. Students don’t always get items correct, for a variety of reasons. Instead, we say that students consistently perform successfully on items or tasks. In a similar vein, for the purpose of this workshop, we will define successful performance as a response probability of 67%, which is referred to as RP67, meaning that we wish to identify the location in the OIB where students who are just barely proficient have a 2/3 chance of responding correctly to the item. You can think about this as a way to define what it means to say that a student can reliably answer an item correctly –they won’t always answer it correctly, but they can reliably answer it.


72

When you place bookmarks, you will work through each page of the OIB and determine whether 2/3 of just barely proficient students, for example, can respond successfully to the item on each page. This judgment will be the basis for recommending a bookmark.


OCBA



The OIB is ordered from easiest to most difficult. This fosters an integrated concept of how the test reflects the performance standards. The OIB is the vehicle to make cut score judgments and communicates how the trait increases in difficulty as items ascend the scale.

73

Yesterday, we reviewed the Ordered Item Book which presented a long series of items, ordered from easiest to most difficult. While reviewing, remember our focus was determining what students need to know and be able to do in order to respond to each item successfully, and why each item was more difficult for students than the items before.

Today, we will make performance standards recommendations by identifying a page number of the OIB that will serve as the cut.

For each performance level, you will work through the OIB and consider whether 2/3 of “just barely” students can respond successfully to the item. You will place your bookmark on the last page where 2/3 of students who just barely meet the

performance standard will answer correctly. This means that fewer than 2/3 of just barely meets students would be expected to respond successfully to the next item in the OIB.

74

OCBA



We will use the “Marks” tab to submit recommendations. You will set a mark at the page number for the recommended cut score for all four different performance standards: Basic - Level 2, Proficient - Level 3, Accelerated - Level 4, Advanced -Level 5.

Remember that you will have an opportunity to discuss your recommendations among your group after this task, and to change your recommendation.

75

It is important to understand that a page number in the OIB does not equate to a number of items a student must get correct to meet a standard at that bookmark. There is no relationship between the pages in the OIB and the number of points needed to achieve a standard.

76

OCBA



[Group Activity]

We will now log into the online ordered item book to practice placing the Level 4 –Accelerated bookmark. We will set ONE bookmark at Level – 4 Accelerated.

Remember to focus our attention on a specific subset of students within each performance level, those that “just barely” Accelerated- Level 4. Work through the OIB and consider whether 2/3 of “just barely” students can respond successfully to the item.

77

Next, we’re going to complete a practice worksheet; the goal is to assess whether the training so far has clearly explained the mechanics of the bookmark procedure. This will help to identify which concepts that need additional clarification before we place our bookmarks. Please take a few minutes to review the worksheet, and then we will review it as a group.

(Give panelists time to complete worksheet. Then walk through worksheet and discuss results, and identify which concepts panelists do not grasp yet.)

78

OCBA



Next, we’re going to complete a practice worksheet; the goal is to assess whether the training so far has clearly explained the mechanics of the bookmark procedure. This will help to identify which concepts that need additional clarification before we place our bookmarks. Please take a few minutes to review the worksheet, and then we will review it as a group.

(Give panelists time to complete worksheet. Then walk through worksheet and discuss results, and identify which concepts panelists do not grasp yet.)

79

As necessary, review that “just barely” means they are just barely categorized as being described by the performance level descriptors. These are not the average student.

80

OCBA




81

The item on page 6 is MORE DIFFICULT than the item on page 5. Review with panelists as necessary:- Items are presented from easiest to most difficult.- Difficulty is based on student performance on 2015 operational assessments. It is not based on test developers’ judgments.

82

OCBA




83 84

OCBA



Review that bookmark should be placed on the last page where 2/3 of students described by the Just Barely PLD can be expected to respond successfully to the item. On the following page, less than 2/3 of just barely students would respond successfully. (Or just barely students would respond successfully less than 2/3 of the time).

85

Review that bookmark should be placed on the last page where 2/3 of students described by the Just Barely PLD can be expected to respond successfully to the item. On the following page, less than 2/3 of just barely students would respond successfully. (Or just barely students would respond successfully less than 2/3 of the time).

86

OCBA



87

Remember, we will focus our attention on a specific subset of students within each performance level, those that “just barely” Accelerated- Level 4. Based on your parsing of the PLDs, you all have a list of the characteristics that differentiate students who are “just barely” Accelerated- Level 4 from those who are not at the performance level, and a descriptor for students that “just barely” meet the performance levels.

In addition, it is important that we define successful performance on test items uniformly. For purposes of this workshop, we define successful performance on an item as a response probability of 67%. We wish to identify the location in the OIB where students who are just barely proficient have a 2/3 chance of responding correctly to the item.


Remember that the page numbers in the OIB have no relationship to the number or percent of items that at student has to perform successfully on in order to meet the recommended standard.

88

OCBA



Fill out your readiness form and hand it to your table leader.


Remove the readiness form from your folders. This form should say “Preparation for Round 1: Basic - Level 2, Proficient - Level 3, Accelerated - Level 4, Advanced -Level 5” near the top.

Review the specific bullets on the form, and please indicate whether you feel you understand, and are ready to place your bookmark. If you answer “No” to any questions, please notify a workshop staff member before continuing.

Let’s take a minute to complete this form. Please turn them into your table leaders.

(Note: In the event that a panelist indicates they are not ready, the facilitator will work with the individual(s) to ensure they understand the procedures.)

90

OCBA



Remember, you are seeking to divide the OIB into two sets of items, so don’t get hung up if you find what appears to be a particularly difficult item in the middle of otherwise relatively easy items. You are seeking to identify the set of items that students who are “just barely” Accelerated- Level 4 can respond to successfully from those items that students who may not meet the standards can also respond to successfully. You will identify the last page in the OIB that 2/3 of “just barely” Accelerated- Level 4 students can successfully respond to; fewer than 2/3 of “just barely” students will be able to successfully respond to the very next item.

It is important that everyone start with the Level 4 standard, then move to the Level 3 standard and finish your bookmark placement with the Level 2 and Level 5.

Note: Continue to display this slide while panelists are placing their bookmarks.

91 92

OCBA



93

From these discussions, you may revise your judgment about the bookmark placement and choose to move your bookmark placement in Round 2. However, there is no requirement or expectation that you will move your bookmarks. From experience, we do expect convergence from Round 1 to Round 2, but consensus is not a goal.

This can serve as a good jumping off point for discussion; for example, panelists who placed their bookmarks at the lowest and highest pages can share why they felt those were the right cut points. After discussing the feedback with your table members, you will discuss your bookmarks with the room as a whole.

NOTE: Have panelists discuss their findings within their tables. They should be able to see the cuts set by the other panelists at their table, all of the table medians and room medians.

94

OCBA



NOTE: Facilitate discussion within the room. Panelists should share what knowledge and skills required by the items or the PLDs led to recommendations.

95

You will also be presented with impact data for each subject. This is the percentage of students who would reach or exceed the standard based on the item page in the Ordered Item Booklet. With this information, you will ask yourself if the outcome seems reasonable. While impact data can be informative, placement of your bookmarks should always be guided by content considerations to ensure that students meeting the performance standard are accurately described by the PLD for each level.

96

OCBA



97

(graph median cut)Benchmark included as well.


OCBA




NOTE: Collect readiness forms with Round 2 initialed by panelists

100

OCBA



Following this meeting, another panel will convene that includes the table leaders. The purpose of this final meeting is to allow table leaders to review the system of standards as a whole so that they can review the appropriateness of the recommended cut scores as they relate to the performance level descriptors and their impact on students who meet each performance standard.

101

Thank you for your participation and hard work these past few days. Please remember to keep the content of the test items and the discussions about specific recommendations secure, and please feel free to share information about the standard setting process and your experience with colleagues and other individuals.

PLEASE take your time to fill out the workshop evaluation. We know that you all often have to fill out evaluations to conclude meetings or trainings, but please take the time to thoughtfully fill out the evaluation. The results of the evaluation will be included in the Standard Setting Technical Report that will be available to the public. Filling out the evaluation also provides us with feedback to improve the standard setting process and to assure the validity of the whole process.

102

OCBA




Appendix

Appendix D – Performance Level Descriptors

Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Grade 4 Social Studies. A student at this level has an emerging ability to use historical materials to understand events in Ohio history, use geographic knowledge to describe the physical environment, use knowledge of government to understand civic rights and responsibilities, and use economic reasoning to make financial choices.

A student at the Limited Level can:

• Define the Underground Railroad; • Locate Ohio on a map; • Follow the four cardinal directions (i.e., north, south, east and west); • Distinguish between fact and opinion; • Name the three branches of government; • Name a First Amendment right (e.g., freedom of religion, speech, press,

petition and assembly); • Group different Ohio resources into sources of food or energy (e.g., corn,

soybeans, coal); • Distinguish between examples of saving money and spending money; • Determine whether or not a given way of settling differences is an

example of a compromise.

D-1

Basic

A student performing at the Basic Level demonstrates partial command of Ohio’s Learning Standards for Grade 4 Social Studies. A student at this level has a general ability to use historical materials to understand events in Ohio history, use geographic knowledge to describe the physical environment, use knowledge of government to understand civic rights and responsibilities, and use economic reasoning to make financial choices.

A student at the Basic Level can:

• Place an event on a timeline; • Distinguish between primary and secondary sources; • Identify evidence of cooperation among prehistoric groups; • Identify sources of colonial frustration; • Identify rights contained in the Northwest Ordinance; • Identify a sectional issue that divided the nation in the early 1800s (e.g.,

slavery); • Identify examples of human modifications of the physical environment

(e.g., construction of dams, use of fertilizers); • Define a civic right (e.g., freedom of religion, freedom of speech); • Define the elements of compromise (e.g., taking turns, looking for

common goals); • Define a constitution (e.g., a written plan for government); • Recognize information found in a chart or table; • Identify productive resources (e.g., natural resources, capital goods).

D-2

Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Grade 4 Social Studies. A student at this level has a consistent ability to use historical materials to understand events in Ohio history, use geographic knowledge to describe the physical environment, use knowledge of government to understand civic rights and responsibilities, and use economic reasoning to make financial choices.

A student at the Proficient Level can:

• Use a timeline to show correct chronological order among several events; • Organize information from primary and secondary sources; • Describe points of cooperation or conflict among different groups of

people in Ohio’s past (e.g., hunting strategies); • Explain why the American colonists united to fight for independence from

Great Britain; • Sequence the steps in the process to statehood; • Explain Ohio's role in the Underground Railroad; • Locate features on a map using cardinal directions; • Describe how human activity has modified the environment; • Describe a way citizens can participate in government; • Compare points of agreement and disagreement to make an informed

decision; • Describe how First Amendment rights are exercised (e.g., freedom of

religion, speech, press, petition and assembly); • Identify the purpose of a constitution (e.g., organizing the structure of

government, protecting basic rights); • Describe the roles of the three branches of government; • Define entrepreneurship.

D-3

Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Grade 4 Social Studies. A student at this level has a superior ability to use historical materials to understand events in Ohio history, use geographic knowledge to describe the physical environment, use knowledge of government to understand civic rights and responsibilities, and use economic reasoning to make financial choices.

A student at the Accelerated Level can:

• Create a timeline with appropriate intervals; • Explain the influence of the Northwest Ordinance; • Explain the significance of the Battle of Lake Erie; • Describe how Ohio became a center for the anti-slavery movement; • Describe the relative location of physical characteristics using scale and

intermediate directions; • Explain a positive consequence of a human modification to the

environment; • Explain how citizens exercise rights to influence their government; • Analyze the perspective or purpose of sources of information to make an

informed decision; • Explain how Ohio’s agriculture, industry, and natural resources can both

influence and be influenced by the economic development of the United States;

• Apply a strategy for compromise in a given situation; • Describe the ways laws are used to protect the rights of citizens; • Summarize trends or patterns in a table; • Explain the risks and benefits of entrepreneurship.

D-4

Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Grade 4 Social Studies. A student at this level has a sophisticated ability to use historical materials to understand events in Ohio history, use geographic knowledge to describe the physical environment, use knowledge of government to understand civic rights and responsibilities, and use economic reasoning to make financial choices.

A student at the Advanced Level can:

• Evaluate information from primary and secondary sources; • Explain how issues left unresolved between American Indians and

European settlers surfaced again in the War of 1812; • Analyze British policy from an American Revolutionary perspective; • Explain the relationship between a region’s physical characteristics and

the region’s economy; • Explain both a positive and negative consequence of a human

modification to the environment; • Explain how Ohio’s evolving transportation systems have influenced the

movement of people and products throughout history; • Distinguish civic rights from responsibilities; • Evaluate information from multiple sources effectively to make informed

decisions; • Explain how the Constitution limits the power of government; • Interpret data to draw conclusions from tables or charts; • Apply the risks and benefits of entrepreneurship to a situation.

D-5

Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Grade 6 Social Studies. A student at this level has an emerging ability to describe the geographic impacts and cultural developments of early civilizations in the Eastern Hemisphere; use appropriate geographic tools to classify and evaluate regions in the Eastern Hemisphere; evaluate how traditions and cultures spread throughout the Eastern Hemisphere; examine sources for accuracy; and explain how scarcity influences economic decision-making.


• Recognize and understand the purpose of a timeline (e.g., to display historical

events); • Demonstrate limited understanding of the conventions of time (e.g., place some

dates in chronological order, place all dates in ascending order on a timeline, etc.);

• Recognize major locations in the Eastern Hemisphere on a map (e.g., continents, rivers, major countries [India, Saudi Arabia, China]);

• Recognize the use of a geographic map to identify locations; • Recall some physical and cultural features of ancient Eastern Hemisphere

civilizations (e.g., Ancient Mesopotamia was located on the Tigris and Euphrates rivers; it developed the cuneiform writing system);

• Identify a source that can be used for information on a historical event; • Recognize how societies and individuals make economic decisions (e.g., high

prices lead to lower demand, high cost leads to lower production).

D-6

Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for Grade 6 Social Studies. A student at this level has a general ability to describe the geographic impacts and cultural developments of early civilizations in the Eastern Hemisphere, use appropriate geographic tools to classify and evaluate regions in the Eastern Hemisphere, evaluate how traditions and cultures spread throughout the Eastern Hemisphere, examine sources for accuracy, and explain how scarcity influences economic decision-making.


• Apply the conventions of one system used to identify and place dates in

chronological order (i.e., B.C.E. and C.E. or B.C. and A.D.); • Demonstrate a simple understanding of the location where ancient river

civilizations in the Eastern Hemisphere developed (e.g., Ancient India developed along the Ganges River);

o Identify the influence of geography (e.g., rivers, climate, terrain); o Identify characteristics of the ancient civilizations of the Eastern

Hemisphere (e.g., language, foods, technologies); • Identify geographic tools that can be used to understand people, places, and

environments (e.g., a variety of maps, aerial photographs); • Demonstrate a partial understanding of using longitude and latitude to identify

absolute location (e.g., use one correctly but not the other); • Demonstrate rudimentary understanding of the characteristics of a region (e.g.,

geographic features, cultural and economic features, cultural diffusion); • Recognize how to determine the accuracy of a source; • Identify some features of different types of governments (e.g., monarchies have

kings or queens, theocracies have religious leaders, dictatorships have a single leader, democracies are ruled by the people);

• Recognize how economic data can be used; • Identify ways that societies handle the scarcity of resources (e.g., decide what to

produce, how to produce and for whom to produce); • Identify some ways in which consumers impact the interaction of markets,

resources and competition (e.g., the demand for silk has led to trade).

D-7

Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Grade 6 Social Studies. A student at this level has a consistent ability to describe the geographic impacts and cultural developments of early civilizations in the Eastern Hemisphere, use appropriate geographic tools to classify and evaluate regions in the Eastern Hemisphere, evaluate how traditions and cultures spread throughout the Eastern Hemisphere, examine sources for accuracy, and explain how scarcity influences economic decision-making.


• Apply the conventions of both systems of historical time (i.e., B.C.E. and C.E., and

B.C. and A.D.) to place events in chronological order on a timeline; • Describe the influence of geography on ancient civilizations in the Eastern

Hemisphere (e.g., rivers, climate, terrain); • Describe characteristics of the ancient civilizations of the Eastern Hemisphere

and relate them to traditions found in the region today (e.g., government, language, religion, economic systems, technologies, food);

• Use different kinds of geographic tools to understand people, places, and environments (e.g., globes, maps, photographs, diagrams);

• Use longitude and latitude to describe absolute location; • Use various criteria to describe, classify, and compare regions in the Eastern

Hemisphere (e.g., geographic, population, cultural, economic); • Explain how different physical environments have influenced human activities

(e.g., river valleys, deserts, mountains); • Explain how tradition and diffusion have influenced modern cultural practices

(e.g., religions found in regions outside of their original development, how trade route impacted diffusion);

• Examine sources for accuracy (e.g., can the information be verified in other sources, is the information a primary or secondary source);

• Describe how governments can be organized in different ways (e.g., monarchies, theocracies, dictatorships, democracies);

• Explain ways that societies handle the scarcity of resources (e.g., how global trade develops, how societies answer the fundamental questions of economics);

• Explain how the interaction of buyers and sellers determines market prices.

D-8

Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Grade 6 Social Studies. A student at this level has a superior ability to describe the geographic impacts and cultural developments of early civilizations in the Eastern Hemisphere, use appropriate geographic tools to classify and evaluate regions in the Eastern Hemisphere, evaluate how traditions and cultures spread throughout the Eastern Hemisphere, examine sources for accuracy, and explain how scarcity influences economic decision-making.


• Explain how geography has influenced ancient civilizations in the Eastern

Hemisphere (e.g., how river valleys affected agriculture and led to crop surpluses);

• Compare the characteristics of the ancient civilizations of the Eastern Hemisphere and their influences in the regions today (e.g., location of civilizations, religion, trade routes);

• Select and use a variety of geographic tools that can be used to understand people, places and environments (e.g., use maps with multiple types of information compiled [e.g., a landform map and an industry map]);

• Apply criteria that describe, classify and compare regions in the Eastern Hemisphere (e.g., what are similarities and differences);

• Evaluate how variations among physical environments in the Eastern Hemisphere have influenced ancient and modern human activities (i.e., how does where you live influence how you live?);

• Evaluate how human activities altered physical environments (e.g., how successful are alterations to the environment, what larger impacts do those alterations have);

• Evaluate the influence of tradition and diffusion on modern cultural practices; • Evaluate religious diffusion in the modern world of the Eastern Hemisphere; • Distinguish between historic and contemporary perspectives of sources; • Explain why sources are accurate or inaccurate; • Explain the relationship between those in power and citizens in different types of

governments; • Use and compare economic data to identify and explain relationships; • Evaluate ways that societies handle the scarcity of resources (e.g., the degree of

success in finding trading partners, why countries will give up some of one resource for another);

• Evaluate how the interaction of buyers and sellers determines market prices and thereby allocates scarce resources, goods and services.

D-9

Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Grade 6 Social Studies. A student at this level has a sophisticated ability to describe the geographic impacts and cultural developments of early civilizations in the Eastern Hemisphere, use appropriate geographic tools to classify and evaluate regions in the Eastern Hemisphere, evaluate how traditions and cultures spread throughout the Eastern Hemisphere, examine sources for accuracy, and explain how scarcity influences economic decision-making.


• Evaluate the influence of geography on ancient civilizations in the Eastern

Hemisphere (e.g., why civilizations settled where they did, how crop surpluses led to specialized industry and societal development);

• Analyze the characteristics of the ancient civilizations of the Eastern Hemisphere and explain their influence today;

• Apply different maps and a variety of geographic tools to gather, process, and report information to understand people, places, and environments;

• Evaluate and apply criteria that describe, classify and compare regions within the Eastern Hemisphere (e.g., when is one set of criteria more appropriate than another, why might it be the incorrect criteria to use);

• Analyze and evaluate how variations among physical environments in the Eastern Hemisphere influence human activities;

• Analyze and evaluate how human activities altered physical environments (e.g., what are the short- and long-term consequences of alterations);

• Analyze and evaluate the influence of tradition and diffusion on modern cultural practices (e.g., what set of circumstances led to the lasting impact of certain products or practices);

• Analyze and evaluate religious diffusion in the modern world of the Eastern Hemisphere (e.g., how Islam spread faster along trade routes because the Arabian Peninsula was a center for trade);

• Evaluate historic and contemporary sources that reflect multiple perspectives on a topic;

• Analyze and evaluate how different types of governments can have distinct or overlapping characteristics;

• Analyze and evaluate economic data to identify and explain relationships; • Analyze and evaluate ways that societies handle the scarcity of resources and

the degree of success to which they engage in global trade; • Explain how competition can exert strong influences over market prices (i.e., can

dictate prices when supply and demand stay constant).

D-10

Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for American History. A student at this level has an emerging ability to use and evaluate primary and secondary sources, use evidence to support or refute a thesis, analyze cause, effect, sequence and correlation in historical events, interpret historic Ohio and U.S. documents, understand the history of U.S. foreign affairs, describe social change in the United States during the 19th and 20th centuries, analyze important events related to the Cold War and describe challenges facing U.S. society in the post- Cold War era.


• Identify one historic era in the United States from 1877–present (e.g., IndustrialRevolution, WWI, Great Depression, Cold War);

• Identify sources as primary or secondary;• Identify at least one important state or federal historic document (e.g.,

Declaration of Independence, Northwest Ordinance, Articles of ConfederationFederalist/Anti-Federalist Papers, U.S. Constitution and U.S. Bill of Rights);

• Identify different groups of people who have immigrated to the United Statesfrom 1877–present (e.g., Irish, Russian, African, Latin American, EasternEuropean);

• Identify segregation as a system that limited the civil rights of African Americansduring the first half of the 20th century;

• Identify one new method of production during the late 19th/early 20th centurythat showed the transition to an industrial based economy (e.g., factory system,assembly line);

• Identify a social, political, or economic event during WWI or WWII in the UnitedStates (e.g., WWI-Red Scare, WWI-Selective Service Act, WWII-JapaneseInternment Camps, WWII-Rosie the Riveter Campaign);

• Identify a factor that led to the beginning of the Cold War (e.g., disagreementover postwar European recovery plans, nuclear arms race, world power status).

American History Performance Level Descriptors

D-11

Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for American History. A student at this level has a general ability to use and evaluate primary and secondary sources, use evidence to support or refute a thesis, analyze cause, effect, sequence and correlation in historical events, interpret historic Ohio and U.S. documents, understand the history of U.S. foreign affairs, describe social change in the United States during the 19th and 20th centuries, analyze important events related to the Cold War and describe challenges facing U.S. society in the post- Cold War era.


• Identify historical eras and when they occurred along a chronological timeline(e.g., Industrial Revolution, WWI, Great Depression, Cold War);

• Identify one difference between a primary and secondary source;• Identify one provision from an important state or federal historic document (e.g.,

Declaration of Independence, Northwest Ordinance, Articles of ConfederationFederalist/Anti-Federalist Papers, U.S. Constitution and U.S. Bill of Rights);

• Identify one pattern of immigration to the United States from 1877 to the present(e.g., European immigration during the Industrial Revolution, African and LatinAmerican immigration throughout the Cold War);

• Identify one historic example of institutionalized racism in the United States (e.g.,poll taxes, literacy tests, Jim Crow Laws);

• Identify one factor of the rise of industrialization in the late 19th century (e.g.,technological advances, rise of corporations, mechanized farming);

• Identify one effect of WWI or WWII on American society (e.g., racialdiscrimination, industrialization, suburbanization, migration patterns, economicopportunities for minority groups);

• Identify factors that led to the beginning and end of the Cold War (e.g., MarshallPlan, Truman Doctrine, NATO, Warsaw Pact, nuclear arms race, Vietnam War,Eastern European independence movements).


D-12

Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for American History. A student at this level has a consistent ability to use and evaluate primary and secondary sources, use evidence to support or refute a thesis, analyze cause, effect, sequence and correlation in historical events, interpret historic Ohio and U.S. documents, understand the history of U.S. foreign affairs, describe social change in the United States during the 19th and 20th centuries, analyze important events related to the Cold War and describe challenges facing U.S. society in the post-Cold War era.


• Describe short and long term effects of major historical events in Americansociety (e.g., European Immigration and its relation to the Industrial Revolution,WWI, Great Depression, Cold War, etc.);

• Use primary, secondary and other credible sources of information to support orrefute a thesis or position;

• Describe the governing precedents set by important state and federal historicdocuments (e.g., Declaration of Independence, Northwest Ordinance, Articlesof Confederation Federalist/Anti-Federalist Papers, U.S. Constitution and U.S. Billof Rights);

• Summarize social struggles of the numerous groups who immigrated to theUnited States from 1877 to the present (e.g., European Immigrants’ fight againstnativism and the KKK, Japanese-Americans’ captivity in WWII relocation camps,Mexican-Americans’ subjection to unsanitary working conditions on Californiafruit plantations);

• Describe legislation/court rulings that sought to reverse long-standingdiscriminatory policies and laws in the United States (e.g., Brown v. Board ofEducation of Topeka Kansas, Civil Rights Acts, 1965 Immigration Act, 19th

Amendment and 26th Amendment);• Describe how the Industrial Revolution transformed the United States from an

agrarian to an urban society;• Describe how events following the end of WWI influenced the events that would

ultimately lead to the beginning of WWII (e.g., Treaty of Versailles, United States’decision not to join the League of Nations, American isolationism, postwareconomic struggles in Europe, etc.);

• Explain how U.S.-Soviet tensions led to major geopolitical crisis in Europe, Asia,and Latin America during the second half of the 20th century.


D-13

Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for American History. A student at this level has a superior ability to use and evaluate primary and secondary sources, use evidence to support or refute a thesis, analyze cause, effect, sequence and correlation in historical events, interpret historic Ohio and U.S. documents, understand the history of U.S. foreign affairs, describe social change in the United States during the 19th and 20th centuries, analyze important events related to the Cold War and describe challenges facing U.S. society in the post-Cold War era.


• Analyze the correlation between important events in American society thatoccurred over a prolonged time (e.g., provisions of the Treaty of Versaillesleading to the outbreak of WWII, late 19th century improvements in agriculturaltechnology contributing to the Dust Bowl);

• Analyze and evaluate the credibility of primary and secondary sources;• Analyze the fundamental ideas behind state and federal historic documents that

have shaped the political landscape of Ohio and the nation (e.g., Declarationof Independence, Northwest Ordinance, Articles of ConfederationFederalist/Anti-Federalist Papers, U.S. Constitution and U.S. Bill of Rights);

• Analyze how a Civil Rights organization fought to extend equal rights to minoritygroups in the United States (e.g., NOW’s contribution to the passage of the EqualPay Act and Title IX, NAACP’s efforts leading to the passage of the Civil RightsAct and the ruling in Brown v. Board of Education of Topeka, Kansas);

• Analyze significant events that impacted legislation/court rulings that sought toreverse long-standing discriminatory policies and laws in the United States;

• Analyze the policies, laws, people and events that sought to reform negativesocial, political, environmental and economic effects of industrial capitalism(e.g., labor organizations, anti-trust legislation, consumer protection legislation,progressivism and muckrakers);

• Analyze and explain the motivations behind the construction of the atomicbomb and its long-term effect on establishing the United States as a world super-power;

• Analyze how the American policy of containment was carried out in a specificforeign nation or region during the second half of the 20th century (e.g., WesternEurope, Korea, Vietnam and Cuba).


D-14

Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for American History. A student at this level has a sophisticated ability to use and evaluate primary and secondary sources, use evidence to support or refute a thesis, analyze cause, effect, sequence and correlation in historical events, interpret historic Ohio and U.S. documents, understand the history of U.S. foreign affairs, describe social change in the United States during the 19th and 20th

centuries, analyze important events related to the Cold War and describe challenges facing U.S. society in the post-Cold War era.


• Analyze the long-term sequence of events that has led to a modern era political,social, economic, or environmental issue impacting the United States (e.g.,political upheaval and terrorist threats in the Middle East);

• Support/refute stances on historic and government issues through the use ofprimary, secondary and other credible sources;

• Identify and differentiate conflicting ideologies involved in the creation of a stateor federal historic document (e.g., Declaration of Independence, NorthwestOrdinance, Articles of Confederation Federalist/Anti-Federalist Papers, U.S.Constitution and U.S. Bill of Rights) that shaped the political landscape of Ohioand the nation (e.g., the differing viewpoints and concerns of the Federalists andAnti-Federalists in the creation of the U.S. Constitution and Bill of Rights, theinfluence of Enlightenment philosophies on the creation of the Declaration ofIndependence and the United States Constitution);

• Analyze the current and future impact on American culture generated by newdemographic groups immigrating to the United States as a result of the 1965Immigration Act (e.g., Hispanic cultural impact on American society);

• Examine and analyze the opposing views on the government’s role in theeconomy, environmental protection, social welfare and national securitybetween 1945 and 1994 (e.g., environmentalism vs. energy independence);

• Analyze how the American economy has been impacted by improved globalcommunications, international trade, transnational business organizations,overseas competition and the shift from manufacturing to service industries;

• Analyze and explain how the American government’s investigations andenforcement of the policy of containment reflected Cold War fears in Americansociety (e.g., Second Red Scare, McCarthyism);

• Analyze the political, national security and economic challenges the UnitedStates has faced in the post-Cold War period (e.g., changing role of the UnitedStates in global politics, controlling of nuclear weapons, political tension andterrorism in the Middle East).


D-15

Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for American Government. A student at this level has an emerging ability to evaluate the credibility and relevance of sources, explain the basic principles of government in the United States and understand how they have been applied and changed over time, interpret historical documents, explain the structure and function of government bodies, explain how citizens can influence government decisions and understand the government’s role in the economy.

Students at the Limited Level can:

• Identify the criteria used to determine the level of credibility of a source;• Recall the main ideas or general themes of the following:

o U.S. Constitution;o The Federalist and Anti-Federalist Papers;o The Bill of Rights;o Civil War amendments;o Progressive reform amendments;o Suffrage amendments;

• Identify all three branches of state and federal government;• Identify basic civic responsibilities of all U.S. citizens (e.g., obey the law, vote,

serve on juries).

D-16

Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for American Government. A student at this level has a general ability to evaluate the credibility and relevance of sources, explain the basic principles of government in the United States and understand how they have been applied and changed over time, interpret historical documents, explain the structure and function of government bodies, explain how citizens can influence government decisions and understand the government’s role in the economy.

Students at the Basic Level can:

• Determine how sources of information are used to support theories, ideas and

policy positions; • Identify the following:

o Basic principles of the U.S. Constitution; o Purpose of the Federalist and Anti-Federalist Papers; o Purpose of the Bill of Rights;

• Identify the importance of the following: o Civil War amendments including 13-15; o Progressive reform amendments including 16-19; o Suffrage amendments including 15, 19, 24, and 26; o Presidential amendments including 12, 20, 22, 23, and 25;

• Identify the basic responsibilities of each branch of the state and federal government;

• Identify examples of constitutional rights and civic responsibilities; • Identify issues with Ohio’s original constitution that led to the creation of the 1851

Ohio Constitution (e.g., legislative powers, selection of political leaders, powers of the governor);

• Identify methods by which citizens can address leaders at each branch of state and federal government;

• Identify how federal tax policy impacts the U.S. economy.

D-17

Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for American Government. A student at this level has a consistent ability to evaluate the credibility and relevance of sources, explain the basic principles of government in the United States and understand how they have been applied and changed over time, interpret historical documents, explain the structure and function of government bodies, explain how citizens can influence government decisions and understand the government’s role in the economy.

Students at the Proficient Level can:

• Identify the differences in the methods by which citizens, government and

political organizations try to resolve conflict (e.g., persuasion, compromise, consensus, negotiation);

• Identify and explain the circumstances around the following: o Basic principles of the U.S. Constitution; o Positions on limited government argued by the Federalists/Anti-

Federalists; o Historic changes in U.S. constitutional government; o Civil War amendments including 13-15; o Progressive reform amendments including 16-19; o Suffrage amendments including 15, 19, 24, and 26; o Presidential amendments including 12, 20, 22, 23, and 25;

• Compare the powers of each branch of the federal government as they pertain to law and public policy (e.g., Legislative-passing resolutions, Executive-issuing executive orders, Judicial-sentencing offenders of the law);

• Identify the relationship between the exercise of constitutional rights and civic responsibility (e.g., right to jury trial-serving on juries, right to vote-becoming informed on public issues);

• Identify several examples of how the 1851 Ohio Constitution addressed the problems of governing Ohio at the time of its adoption (e.g., election of government officials by popular vote, addition of district courts, banned poll taxes);

• Identify a public policy position and determine the most appropriate level and branch of government to address the issue;

• Identify how federal tax policy and the Federal Reserve System impact the U.S. economy.

D-18

Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for American Government. A student at this level has a superior ability to evaluate the credibility and relevance of sources, explain the basic principles of government in the United States and understand how they have been applied and changed over time, interpret historical documents, explain the structure and function of government bodies, explain how citizens can influence government decisions and understand the government’s role in the economy.

Students at the Accelerated Level can:

• Explain how political parties, special interest groups and/or the media influence

public policy making; • Cite specific examples regarding the following:

o Checks on power amongst the three branches of government; o Anti-Federalist ideas implemented in the language of the Bill of Rights; o Changes in the Constitution resulting from evolving social/public demands

for governing; o Consequences of the ratification of Amendments 13-15; o Consequences of the ratification of Amendments 16-19; o Consequences of the ratification of Amendments 15, 19, 24, and 26; o Consequences of the ratification of Amendments 12, 20, 22, 23, and 25; o Circumstances and consequences of the ratification of Amendments 11,

21, and 27; • Identify a specific historical or contemporary issue that required interaction

amongst the branches of government and analyze the dynamics involved (e.g., circumstances surrounding the 11th Amendment, the impeachment process, the federal government’s actions related to desegregation);

• Cite examples of civil rights being extended to minority groups; • Identify symmetry between the structure of the federal government and the

state government of Ohio; • Identify various methods by which individuals and organizations can engage

government officials and evaluate the effectiveness of each; • Explain a specific action taken by the federal government/Federal Reserve

System to manipulate the fiscal/monetary policy and determine its impact on consumer and business spending.

D-19

Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for American Government. A student at this level has a sophisticated ability to evaluate the credibility and relevance of sources, explain the basic principles of government in the United States and understand how they have been applied and changed over time, interpret historical documents, explain the structure and function of government bodies, explain how citizens can influence government decisions and understand the government’s role in the economy.

Students at the Advanced Level can:

• Examine a set of documents pertaining to a civic issue from at least two distinct

information types (e.g., public records, surveys, research data, etc.) and explain the relevancy and credibility of each;

• Cite specific examples regarding the following: o Federalist and Anti-Federalist ideas implemented in the language of the Bill of

Rights; o Changes in the Constitution resulting from evolving social/public demands for

governing; o Precedent and consequences of the ratification of Amendments 13-15; o Precedent and consequences of the ratification of Amendments 16-19; o Precedent and consequences of the ratification of Amendments 15, 19, 24

and 26; o Precedent and consequences of the ratification of Amendments 12, 20, 22,

23 and 25; o Precedent and consequences of the ratification of Amendments 11, 21,

and 27; • Examine and explain specific examples of checks and balances implemented

between the three branches of government and how those oversight powers uphold the principles of the U.S. Constitution (e.g., presidential veto, veto override, Supreme Court rulings against executive orders or legislation);

• Analyze an issue related to the denial of civil rights to a specific minority group and explain how at least one branch of government worked to extend civil rights to that group of people (e.g., Supreme Court and the 14th Amendment, Congress and the passing of Civil Rights amendments);

• Compare the structures, powers and relationships between the Ohio and federal governments based on their respective constitutions;

• Determine the monetary action that should be taken by the Federal Reserve to respond to the forces of inflation and deflation and how such actions will meet the desired economic goal.

D-20

Grade 5 Science Performance Level Descriptors Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Grade 5 Science. A student at this level has an emerging ability to demonstrate knowledge of the solar system and use knowledge of the relationship between Earth and the sun to describe the physical environment, demonstrate knowledge of the roles of organisms in ecosystems and use knowledge of motion and light to describe physical situations.


• Recall that the sun is the only star in the solar system; • Describe characteristics of planets such as distance from the sun, size,

movement, composition, and temperature; • Recognize Earth’s seasons, day and night, and the motion of the sun in the sky; • Recognize that organisms within an ecosystem have roles and relationships; • Recall the sun is the primary source of energy for most ecosystems; • Describe that objects at rest require a force to start moving; • Investigate how the color of objects is related to light; • Identify behaviors of light and sound.

D-21

Grade 5 Science Performance Level Descriptors Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for Grade 5 Science. A student at this level has a general ability to demonstrate knowledge of the solar system and use knowledge of the relationship between Earth and the sun to describe the physical environment, demonstrate knowledge of the roles of organisms in ecosystems and use knowledge of motion and light to describe physical situations.


• Recall that the orbital paths of planets, moons and celestial bodies are due to

gravitational attraction; • Select tools and technology needed to study the solar system, including Earth

(e.g., telescopes, satellites, probes); • Explain that other stars are much farther away from Earth than the sun, which

causes them to appear much smaller; • Recall the size and composition (e.g., made of gas) of stars, including the sun; • Identify general information about asteroids, meteoroids, comets, and dwarf

planets (e.g., composition, relative size, orbits); • Identify that roles of living organisms are determined by how they acquire energy

(e.g., producers, consumers, decomposers); • Recognize that producers are the foundation of the food web; • Identify the roles and relationships (e.g., symbiotic) of organisms within an

ecosystem; • Recall that photosynthesis is the process plants use to produce food from

sunlight; • Identify the effects of relative mass/weight and force (amount and direction) on

an object’s change in motion; • Recognize measurements of motion involving speed, distance and time; • Recall that an object moving at constant speed has no change in speed or

direction if no force is acting on it; • Investigate the behavior of light and sound when encountering a new medium

(e.g., absorption, reflection, pass, or travel through).

D-22

Grade 5 Science Performance Level Descriptors Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Grade 5 Science. A student at this level has a consistent ability to demonstrate knowledge of the solar system and use knowledge of the relationship between Earth and the sun to describe the physical environment, demonstrate knowledge of the roles of organisms in ecosystems and use knowledge of motion and light to describe physical situations.


• Compare or summarize general characteristics (e.g., size, composition, distance,

temperature, motion) of objects in the universe including, stars, planets, moons, asteroids and comets;

• Explain that seasonal weather patterns in specific regions (e.g., hurricane, monsoon, rainy or dry seasons) are predictable, due to the yearly solar cycle;

• Relate Earth’s tilt and revolution to direct sunlight and seasons; • Compare roles of living organisms, determined by how they acquire energy

(e.g., producers, consumers, decomposers); • Explain that photosynthesis is the process through which sunlight is transformed

by producers into energy; • Create a food web to illustrate how energy is transferred and transformed in an

ecosystem through interactions of organisms; • Describe that an object moving at constant speed has no change in speed or

direction if no force is acting on it; • Compare the speed of objects, given distance and time data; • Plan an investigation to explore the behavior of light when encountering a new

medium (e.g., absorption, reflection, refraction, pass, or travel through); • Explain how the color of objects relates to reflection and absorption; • Explain how the pitch of sound and the vibration rate of an object are related; • Interpret the behavior of sound when encountering a new medium (e.g.,

absorption, reflection, pass, or travel through).

D-23

Grade 5 Science Performance Level Descriptors Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Grade 5 Science. A student at this level has a superior ability to demonstrate knowledge of the solar system and use knowledge of the relationship between Earth and the sun to describe the physical environment, demonstrate knowledge of the roles of organisms in ecosystems and use knowledge of motion and light to describe physical situations.


• Create a model to show the relationship between size, distance and

appearance of stars; • Explain how using data about the compositions of planets can indicate distance

from the sun; • Examine relationships and draw conclusions between direct sunlight and

temperature, and the angle/altitude of the sun and amount of direct sunlight; • Explain and provide evidence to support why producers are the foundation of

the food web; • Develop a model to illustrate the flow of energy based on a scenario and

explain the roles and relationships (e.g., symbiotic) of organisms within an ecosystem;

• Predict the impact on the ecosystem as species are introduced or removed (e.g., endangered or threatened species, invasive species);

• Draw conclusions based on data and/or diagrams showing movement of an object over time;

• Compare and rank the relative change in motion for objects of different masses/weights that experience the same force;

• Compare and explain the differences between objects that emit light (e.g., the sun) and objects that reflect light (e.g., an apple, the moon);

• Predict how absorbed light causes objects to warm and the effects of the material, light intensity, angle, and time of exposure on the amount of heating;

• Plan an investigation to explore and predict movement of sound as it travels outward from its source through different media.

D-24

Grade 5 Science Performance Level Descriptors Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Grade 5 Science. A student at this level has a sophisticated ability to demonstrate knowledge of the solar system and use knowledge of the relationship between Earth and the sun to describe the physical environment, demonstrate knowledge of the roles of organisms in ecosystems and use knowledge of motion and light to describe physical situations.


• Given data, provide evidence to support the fact that Earth orbits the sun in a

nearly circular path; • Use data and evidence to make a conclusion about how the positions of the

Earth and the sun relate to seasonal weather patterns in specific regions; • Given a scenario, determine relationships between direct sunlight and

temperature and the angle/altitude of the sun and amount of direct sunlight; • Use data from an ecosystem to interpret the change of energy flow in an

ecosystem when species are introduced or removed; • Use data to form conclusions about the roles of organisms within a given

ecosystem and explain how the evidence supports that conclusion; • Plan an investigation based on data from a real-world scenario to determine the

impact of the introduction of an invasive species on the population of a local species;

• Predict changes that take place when an object experiences differing magnitudes of forces and/or masses/weights;

• Design an investigation that determines how the mass/weight of an object (or amount of force acting on an object) affects how the motion of an object changes;

• Test a design to explore how absorbed light causes objects to warm and the effects of the material, light intensity, angle, and time of exposure on the amount of heating;

• Design an object to illustrate the relationships between the pitch of a sound and the vibration rate of an object;

• Design an engineering solution to a real-world scenario involving light absorption and reflection of heat.

D-25

Grade 8 Science Performance Level Descriptors Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Grade 8 Science. A student at this level has an emerging ability to explain how geologic processes formed the Earth’s surface and interior and explain how the geologic record contains evidence of these changes, communicate that the characteristics of organisms are a result of inherited traits passed on by reproduction, describe how fossils provide evidence of the changes and diversity of life over geologic time, predict how forces with magnitude and direction affect the movements of objects, and explain the differences in types of potential energy.


• Recognize that seismic data can be used to determine the composition of

Earth’s interior; • Recall that geologic and fossil records serve as evidence for past environmental

conditions; • Recall that tectonic plates move; • Identify tectonic plate boundaries; • Define tectonic activity, erosion, and deposition; • Define relative and absolute age of geologic features; • Describe sexual and asexual reproduction; • Recall that mitosis and meiosis are processes by which genetic material is copied

and divided; • Complete a monohybrid cross Punnett square; • Identify gravitational, electric, and magnetic fields; • Identify that motion is relative and dependent on the position of the observer; • Define potential energy (e.g., gravitational, elastic, chemical, electric).

D-26

Grade 8 Science Performance Level Descriptors Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for Grade 8 Science. A student at this level has a general ability to explain how geologic processes formed the Earth’s surface and interior and explain how the geologic record contains evidence of these changes, communicate that the characteristics of organisms are a result of inherited traits passed on by reproduction, describe how fossils provide evidence of the changes and diversity of life over geologic time, predict how forces with magnitude and direction affect the movements of objects, and explain the differences in types of potential energy.


• Recall that movement of tectonic plates is caused by convection currents; • Identify types and characteristics of tectonic plate boundaries and the resulting

features or events; • Describe major geologic processes that form specific features on the surface of

Earth (e.g., tectonic activity, erosion, deposition); • Identify the processes that cause the formation of various types of surface

features (e.g., rivers, streams, deserts, coastlines); • Identify and recall factors (e.g., topography, climate, soil, rock characteristics)

that affect the surface patterns associated with streams, floodplains, glaciers, coastlines, flooding, and deserts;

• Describe how to determine relative age of geologic features; • Analyze the inheritance patterns shown in pedigrees based on relationships

between phenotypes and genotypes; • Use diagrams to show the genetic differences between the daughter cells

produced by mitosis and meiosis; • Explain the advantages and disadvantages of sexual and asexual reproduction; • Identify behavior of objects with mass, charge and/or magnetic properties in

gravitational, electric or magnetic fields; • Interpret the magnetic field from drawings/pictures of a magnet and iron filings; • Interpret force/free-body diagrams; • Describe different types of potential energy (e.g., gravitational, elastic,

chemical, electric).

D-27

Grade 8 Science Performance Level Descriptors Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Grade 8 Science. A student at this level has a consistent ability to explain how geologic processes formed the Earth’s surface and interior and explain how the geologic record contains evidence of these changes, communicate that the characteristics of organisms are a result of inherited traits passed on by reproduction, describe how fossils provide evidence of the changes and diversity of life over geologic time, predict how forces with magnitude and direction affect the movements of objects, and explain the differences in types of potential energy.


• Use seismic data, graphs, and charts to interpret the structure of Earth’s interior; • Explain and justify conclusions based on data, maps, and diagrams about the formation

and boundaries of geologic features due to tectonic plate movement; • Explain the characteristics of rocks and soil, climate, location, and topography as they

relate to constructive and destructive processes occurring between the hydrosphere and lithosphere;

• Explain the connection between the presence of specific rock types, rock features, or fossils and environmental conditions at the time of formation (e.g., rocks with ripple marks and moving water, basalt and volcanic activity);

• Apply the concept of uniformitarianism to determine the relative age of geologic features using the law of superposition, index fossils or crosscutting relationships);

• Use diagrams or data from geologic columns and glacial cores to interpret and compare relative and absolute age and environmental conditions;

• Predict how genetic variation (e.g., beak structure, coloration) affects the survival or extinction of a species when environmental conditions change gradually or suddenly;

• Compare end products of sexual and asexual reproduction with an emphasis on their advantages and disadvantages in relation to the continuation of the species;

• Use pedigrees to explain the principles of Mendelian genetics, law of segregation, and law of independent assortment;

• Explain the relationship between electric currents and magnetic fields; • Explain that generators and motors transform electrical and mechanical energy using

electric currents and magnetic fields; • Compare the properties of gravitational, electric, and magnetic fields; • Complete and analyze simple force/free-body diagrams; • Explain the changes in motion in a scenario involving balanced or unbalanced forces; • Compare the different types of potential energy and their attributes.

D-28

Grade 8 Science Performance Level Descriptors Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Grade 8 Science. A student at this level has a superior ability to explain how geologic processes formed the Earth’s surface and interior and explain how the geologic record contains evidence of these changes, communicate that the characteristics of organisms are a result of inherited traits passed on by reproduction, describe how fossils provide evidence of the changes and diversity of life over geologic time, predict how forces with magnitude and direction affect the movements of objects, and explain the differences in types of potential energy.


• Analyze or complete a diagram that shows how constructive or destructive

processes affect the lithosphere; • Use evidence from a geologic column to make conclusions about Earth’s

geologic history and make comparisons between the past and present; • Analyze data to justify conclusions that the fossil record serves as evidence for

biodiversity and/or diversity within a species and the fact that most species that have lived on Earth are now extinct;

• Interpret data from real-world scenarios or experiments showing outcomes (e.g., survival and reproduction rates) of sexual and asexual reproduction under varying environmental conditions;

• Design an experiment to determine possible genotypes and their probabilities of occurring in offspring from a cross;

• Use diagrams of motors, generators, or electromagnets to demonstrate the relationships between current and magnetic field;

• Predict changes in motion given a scenario involving balanced or unbalanced forces;

• Describe various forms of potential energy within a system in terms of the positions of the objects involved (e.g., elastic potential energy in a spring, electric potential energy associated with charges).

D-29

Grade 8 Science Performance Level Descriptors Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Grade 8 Science. A student at this level has a sophisticated ability to explain how geologic processes formed the Earth’s surface and interior and explain how the geologic record contains evidence of these changes, communicate that the characteristics of organisms are a result of inherited traits passed on by reproduction, describe how fossils provide evidence of the changes and diversity of life over geologic time, predict how forces with magnitude and direction affect the movements of objects, and explain the differences in types of potential energy.


• Use evidence from aerial photographs and/or topographical maps to generate

and justify a conclusion about how specific land features were formed; • Evaluate rock and fossil data to generate and justify conclusions about past

environmental conditions; • Trace climate change as documented by the geologic record and ice cores; • Evaluate graphs showing population data related to environmental changes; • Interpret and explain data from real-world scenarios or experiments showing

outcomes (e.g., survival and reproduction rates) of sexual and asexual reproduction under varying environmental conditions;

• Interpret a pedigree to determine the genotype of an individual within that pedigree;

• Design an experiment to assess how objects would behave in electric, magnetic, or gravitational fields;

• Design an experiment to test the effect of multiple forces on the motion of an object;

• Make and justify conclusions using data collected from an experiment that studies the relationship between position and potential energy in a system.

D-30

Physical Science Performance Level Descriptors Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Physical Science. A student at this level has an emerging ability to demonstrate knowledge of the classification properties and interactions among matter, apply knowledge of conservation of energy, transfer and transformation of energy, and forces and motion to describe physical situations, utilizing graphs, diagrams, and equations, and demonstrate knowledge of the evolution and expansion of the universe.


• Given a phase change graph, identify the temperature of the melting and/or

boiling point; • Given elemental properties, classify elements in the same group; • Given a chemical name for a compound, identify the elements involved; • Recognize that electrons can be lost, gained, or shared when atoms bond; • Given a chemical equation, identify the reactants and products; • Calculate for gravitational potential energy and kinetic energy; • Identify the portion of the electromagnetic spectrum that has the lowest

frequency, wavelength, or energy; • Recognize the properties of an object that affect absorption and radiation rates

of thermal energy; • Recognize that electrons flow through a circuit; • Interpret a free-body diagram to identify normal and tension forces; • Recall that interactive force pairs are equal in magnitude but act in opposite

directions; • Recognize that the universe is expanding; • Recognize that galaxies moving away from Earth have an observed redshift; • Recall that fusion occurs in stars; • Recall that the Hertzsprung-Russell (HR) diagram provides information about stars.

D-31

Physical Science Performance Level Descriptors Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for Physical Science. A student at this level has a general ability to demonstrate knowledge of the classification properties and interactions among matter, apply knowledge of conservation of energy, transfer and transformation of energy, and forces and motion to describe physical situations, utilizing graphs, diagrams, and equations, and demonstrate knowledge of the evolution and expansion of the universe.


• Describe, using properties, the difference between homogenous and

heterogeneous matter; • Given a phase change graph, identify where the phase changes occur; • Given data, determine values for various atomic properties (e.g., atomic

number, mass number); • Distinguish between the isotopes and ions of an element and describe properties

of elements based on their position in the periodic table; • Determine the chemical names of simple compounds given their formulas (or

vice versa); • Given a pair of elements, determine the formula of a compound between them

and/or whether the bond would be covalent or ionic; • Describe endothermic or exothermic reactions; • Recognize when work is done on an object; • Compare relative energies, frequencies and wavelengths of the different bands

of the electromagnetic spectrum, including the colors of visible light; • Explain the dissipations of energy from systems due to transformation into thermal

energy; • Describe the origin, motion and energy of electrons in circuits; • Recall that interactive force pairs can never cancel each other; • Interpret a free-body diagram to determine forces, including normal (for surfaces

at any angle) and tension forces; • For problems involving mass, weight and gravitational field strength, calculate

one when given the other two; • Calculate distance, displacement, average velocity, and acceleration based

on graphical and tabular motion data; • Identify information provided by a Hertzsprung-Russell diagram; • Recall a major difference between nuclear fission and fusion.

D-32

Physical Science Performance Level Descriptors Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Physical Science. A student at this level has a consistent ability to demonstrate knowledge of the classification properties and interactions among matter, apply knowledge of conservation of energy, transfer and transformation of energy, and forces and motion to describe physical situations, utilizing graphs, diagrams, and equations, and demonstrate knowledge of the evolution and expansion of the universe.


• Use one designated property (e.g., solubility, density, boiling/melting point) to

separate a mixture; • Interpret phase change graphs to identify changes in kinetic and/or potential

energy; • Create a diagram or model atoms, ions, isotopes, and various chemical bonds; • Balance chemical equations; • Describe properties of elements that lead to radioactive decay, interpret half-life

graphs and produce a graph from half-life data; • Given a real-world scenario, calculate values involving work or values involving

conservation of energy in a closed system; • Describe the characteristics and behaviors (e.g., superposition/interference,

diffraction) of waves as a form of energy transfer; • Compare radiant energy interactions between objects with differing

characteristics that influence the rate of thermal absorption and emission (e.g., temperature, color, texture, exposed surface area in the system);

• Describe the relationships between voltage, current, and resistance in circuits; • Represent, analyze and interpret data from diagrams, graphs, charts, and tables

related to position vs. time, velocity vs. time, acceleration, and motion; • Calculate force, mass or acceleration using values drawn from tables, graphs

and free-body diagrams; • Identify or describe interactive force pairs, and compare their magnitudes and

directions; • Explain how redshift provides information about the distances of galaxies and

how this can be used as evidence for the Big Bang model of the universe; • Interpret a Hertzsprung-Russell diagram in terms of mass, luminosity, temperature,

and evolutionary stages of the main sequence stars; • Explain the role of fusion in stars in the formation of elements.

D-33

Physical Science Performance Level Descriptors Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Physical Science. A student at this level has a superior ability to demonstrate knowledge of the classification properties and interactions among matter, apply knowledge of conservation of energy, transfer and transformation of energy, and forces and motion to describe physical situations, utilizing graphs, diagrams, and equations, and demonstrate knowledge of the evolution and expansion of the universe.


• Predict how unknown elements react when given properties; • Given an ionic formula with an unknown, identify ionic charge and/or the

elemental group and its location on the periodic table (e.g., XF2); • Describe the changes in motion and relative position of particles when given

data or graphs (e.g., phase change graphs); • Given half-life data of radioactive elements, evaluate which elements are

appropriate for various applications and justify this using evidence; • Calculate gravitational potential energy when the reference point is not the

ground or lowest level; • Given a real-world scenario, calculate values involving both work and

conservation of energy in a closed system; • Create a model to visually represent diffraction, superposition, constructive and

destructive interference, or a change in wavelength due to the Doppler effect; • Given a real-world scenario, recommend specific design features that relate to

thermal energy absorption and emission (e.g., temperature, color, texture, exposed surface area in the system);

• Explain observed changes in current and voltage in a circuit in terms of electrons and energy transfer;

• Design and/or evaluate a circuit to meet real-world conditions and constraints; • In a real-world scenario, construct a free-body diagram using information from

motion vs. time graphs (or vice versa); • Given a real-world context, interpret position vs. time, velocity vs. time and/or

motion data to create a scenario that describes possible forces responsible for the motion (or vice versa);

• Compare and interpret spectroscopic data indicating the Doppler shift of various galaxies to determine relative motion and distances.

D-34

Physical Science Performance Level Descriptors Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Physical Science. A student at this level has a sophisticated ability to demonstrate knowledge of the classification properties and interactions among matter, apply knowledge of conservation of energy, transfer and transformation of energy, and forces and motion to describe physical situations, utilizing graphs, diagrams, and equations, and demonstrate knowledge of the evolution and expansion of the universe.


• Design a solution to a real-world problem involving mixtures that need to be

separated based on more than one property (e.g., solubility, density, boiling/melting point);

• Plan an experiment to determine values related to energy transformation and energy transferred through work on a system;

• Design or improve a system that involves work and energy transformation that meets certain constraints (e.g., height, speed, force, displacement);

• Design or critique solutions to engineering problems involving forces and motion; • Design an experiment to measure the velocity of objects in a real-world scenario; • Design an experiment using dynamics to determine a specific force in a given

system of forces (e.g., friction force from spring scale); • Relate red and blue shift to relative galaxy motion and distance by constructing

a shifted spectrum.

D-35

Limited

A student performing at the Limited Level demonstrates a minimal command of Ohio’s Learning Standards for Biology. A student at this level has an emerging ability to describe genetic patterns of inheritance and how traits are defined by instructions encoded in many discrete genes, explain that Earth’s present-day species descended from common ancestral species due to variation and the natural selection of heritable characteristics, construct classification systems based on anatomical and molecular analysis to indicate the degree of relatedness between organisms, model how organisms transform matter and energy as part of complex ecosystems, and describe the cellular functions associated with cell division and homeostasis.


• Recall that DNA is the blueprint for cell structures and processes; • Name the complementary base pairs in DNA; • Recall that populations can change from generation to generation due to

selective pressures; • Given a scenario, predict which traits may increase survivability in a species; • State that organisms are classified on morphological and molecular evidence; • Recall that ecosystems change as geological and biological conditions vary; • Identify different types of cells (e.g., prokaryotic vs. eukaryotic, plant vs. animal); • Identify the products of photosynthesis and cellular respiration (e.g. oxygen,

carbon dioxide, glucose, etc.); • Recall that cell components work together to perform the functions of the cell.

D-36

Basic

A student performing at the Basic Level demonstrates a partial command of Ohio’s Learning Standards for Biology. A student at this level has a general ability to describe genetic patterns of inheritance and how traits are defined by instructions encoded in many discrete genes, explain that Earth’s present-day species descended from common ancestral species due to variation and the natural selection of heritable characteristics, construct classification systems based on anatomical and molecular analysis to indicate the degree of relatedness between organisms, model how organisms transform matter and energy as part of complex ecosystems, and describe the cellular functions associated with cell division and homeostasis.


• Recognize the structural and base pairing differences between DNA and RNA; • Identify that the sequence of DNA bases in a chromosome determines the

sequence of amino acids in a protein; • Recall that the insertion, deletion or substitution of segments of DNA are

mutations which alter genes; • Identify how an altered gene may be passed on to every cell that develops from

that cell; • Describe the concept of differentiation—although all cells have identical

genetic information, different genes are active in different types of cells; • Identify that variations within a population in a changing environment can lead

to evolution; • Provide an example of natural selection; • Describe one process by which an organism transforms energy and matter as

part of an ecosystem; • Recall how carrying capacity limits a population; • Explain how the functions of cellular parts are made possible by their structures

(e.g., the pores in the nuclear envelope, the channel proteins of the cell membrane);

• Use a diagram to model the transport of particles into and out of the cell.

D-37

Proficient

A student performing at the Proficient Level demonstrates an appropriate command of Ohio’s Learning Standards for Biology. A student at this level has a consistent ability to describe genetic patterns of inheritance and how traits are defined by instructions encoded in many discrete genes, explain that Earth’s present-day species descended from common ancestral species due to variation and the natural selection of heritable characteristics, construct classification systems based on anatomical and molecular analysis to indicate the degree of relatedness between organisms, model how organisms transform matter and energy as part of complex ecosystems, and describe the cellular functions associated with cell division and homeostasis.


• Explain the difference in possible outcomes when gene mutations occur in

gametes or somatic cells; • Predict and interpret non-Mendelian monohybrid and Mendelian dihybrid

crosses; • Recognize which tests are used to compare actual results to predicted results in

genetic investigations (e.g. chi-squared, gel electrophoresis); • Explain the steps of protein synthesis from DNA transcription to translation; • Explain how sorting and recombination of genes in sexual reproduction and

meiosis result in variations of traits in offspring; • Determine how variations within populations in a changing environment can

lead to evolution; • Given a real-world example, predict and provide evidence to support how a

population has responded to environmental changes; • Interpret a cladogram showing relationships among species; • Given population graphs or charts containing data, determine the changes to

an ecosystem, including carrying capacity; • Predict the effect of geological, biological and/or environmental changes on a

population within an ecosystem (e.g., shifting weather patterns, loss of habitat); • Interpret graphs or data (e.g., temperature, pH, light, concentration) to explain

the rate of enzyme activity in a cell; • Describe regulation of the cellular environment (e.g., homeostasis); • Determine an amino acid sequence from a DNA/RNA base sequence using a

codon chart; • Describe how cell components work together to perform the functions of

the cell; • Plan and conduct an experiment to determine the effects of external factors

(e.g., pH, temperature, concentration) on cellular functions (e.g., transport, enzyme rate, photosynthesis, cellular respiration).

D-38

Accelerated

A student performing at the Accelerated Level demonstrates a strong command of Ohio’s Learning Standards for Biology. A student at this level has a superior ability to describe genetic patterns of inheritance and how traits are defined by instructions encoded in many discrete genes, explain that Earth’s present-day species descended from common ancestral species due to variation and the natural selection of heritable characteristics, construct classification systems based on anatomical and molecular analysis to indicate the degree of relatedness between organisms, model how organisms transform matter and energy as part of complex ecosystems, and describe the cellular functions associated with cell division and homeostasis.


• Investigate Mendelian and non-Mendelian inheritance (e.g., dihybrid crosses,

sex-linked traits, linkage) to compare predicted results to actual results (e.g., chi- square goodness of fit);

• Interpret data from a real-world scenario involving biotechnology (e.g., gel electrophoresis);

• Explain why a population is or is not in Hardy-Weinberg equilibrium based on given data or conditions (e.g., no mutation is occurring, mating is random);

• Given scenarios in which technological innovations could potentially influence selective pressure on a population, outline possible advantages and disadvantages for use of the technology using scientific concepts;

• Given data and/or a scenario, make and justify a conclusion about evolutionary mechanisms in a population;

• Given population graphs or charts containing data, analyze current and past data to predict and provide evidence to support their predicted changes in an ecosystem;

• Use mathematical models (e.g. exponential or logistic growth) to explain carrying capacity and homeostasis within ecosystems;

• Complete cladograms to determine relationships among organisms; • Design a model to explain the transformation of energy through ATP and cycling

of carbon through cellular processes in cells (e.g., photosynthesis, cellular respiration);

• Explain how the cellular environment affects the mechanical operation of an enzyme (e.g., reactant, active site, product).

D-39

Advanced

A student performing at the Advanced Level demonstrates a distinguished command of Ohio’s Learning Standards for Biology. A student at this level has a sophisticated ability to describe genetic patterns of inheritance and how traits are defined by instructions encoded in many discrete genes, explain that Earth’s present-day species descended from common ancestral species due to variation and the natural selection of heritable characteristics, construct classification systems based on anatomical and molecular analysis to indicate the degree of relatedness between organisms, model how organisms transform matter and energy as part of complex ecosystems, and describe the cellular functions associated with cell division and homeostasis.


• Analyze data and provide evidence for the selection of an evolutionary

mechanism that has occurred in a given population; • Design or simulate a population growth model by manipulating environmental

conditions; • Develop and test a hypothesis to improve the design of an industrial application

of cellular processes (e.g., optimal environment for fermentation, genetically modified organisms);

• Use mathematical reasoning to analyze graphs displaying data about enzyme activity and how it impacts a cell;

• Explain the relative effects of different mutations (e.g., substitution or deletion resulting in a frameshift) in DNA on the operation of protein because of changes in sequence of amino acids;

• Interpret the results of experiments to identify evidence for core ideas in biology (e.g., structure of DNA, DNA semi-conservative replication, mRNA read three bases at a time).

D-40


Appendix

Appendix E – Summary of Ordered Item Booklets


E-1 American Institutes for Research

Table E. Summary of Ordered Item Booklets by Test

Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of

Students At or Above Standard

Location of External Benchmarks

Grade 4 Social Studies 1 1 -1.59 99 NAEP Ohio Reading: below Basic; OAA G5

SS (2009): Limited Grade 4 Social Studies 2 1 -1.44 98 Grade 4 Social Studies 3 1 -1.22 96 Grade 4 Social Studies 4 1 -1.16 94 Grade 4 Social Studies 5 1 -1.14 94

Grade 4 Social Studies 6 1 -1.03 92 OAA G5 SS (2009): Basic;

Grade 4 Social Studies 7 1 -0.99 91 Grade 4 Social Studies 8 2 -0.92 88 Grade 4 Social Studies 9 1 -0.85 86 Grade 4 Social Studies 10 1 -0.82 86 Grade 4 Social Studies 11 1 -0.79 86 Grade 4 Social Studies 12 1 -0.72 84 Grade 4 Social Studies 13 1 -0.68 81 Grade 4 Social Studies 14 1 -0.60 78 Grade 4 Social Studies 15 1 -0.60 78 Grade 4 Social Studies 16 1 -0.53 76 Grade 4 Social Studies 17 1 -0.50 76 Grade 4 Social Studies 18 1 -0.46 73

Grade 4 Social Studies 19 1 -0.40 70 NAEP Ohio Reading: Basic

Grade 4 Social Studies 20 1 -0.27 66 Grade 4 Social Studies 21 1 -0.27 65 Grade 4 Social Studies 22 2 -0.26 63 Grade 4 Social Studies 23 1 -0.24 63 Grade 4 Social Studies 24 2 -0.24 63

Grade 4 Social Studies 25 1 -0.14 60 OAA G5 SS (2009): Proficient;

Grade 4 Social Studies 26 1 -0.11 57 Grade 4 Social Studies 27 1 -0.04 54 Grade 4 Social Studies 28 2 0.03 51 Grade 4 Social Studies 29 1 0.05 51 Grade 4 Social Studies 30 1 0.06 51 Grade 4 Social Studies 31 1 0.13 48 Grade 4 Social Studies 32 1 0.18 44



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 4 Social Studies 33 1 0.19 44

Grade 4 Social Studies 34 2 0.21 44 OAA G5 SS (2009): Accelerated;

Grade 4 Social Studies 35 2 0.29 41

Grade 4 Social Studies 36 1 0.33 38 NAEP Ohio Reading: Proficient

Grade 4 Social Studies 37 1 0.42 35 Grade 4 Social Studies 38 1 0.47 32 Grade 4 Social Studies 39 1 0.53 31 Grade 4 Social Studies 40 1 0.56 29 Grade 4 Social Studies 41 1 0.57 29 Grade 4 Social Studies 42 1 0.63 26 Grade 4 Social Studies 43 2 0.73 23

Grade 4 Social Studies 44 2 0.77 20 OAA G5 SS (2009): Advanced;

Grade 4 Social Studies 45 1 0.93 17 Grade 4 Social Studies 46 1 0.93 17 Grade 4 Social Studies 47 1 0.94 17 Grade 4 Social Studies 48 1 0.94 16 Grade 4 Social Studies 49 2 0.96 16 Grade 4 Social Studies 50 1 1.02 15 Grade 4 Social Studies 51 1 1.05 13 Grade 4 Social Studies 52 2 1.12 13 Grade 4 Social Studies 53 1 1.13 11 Grade 4 Social Studies 54 1 1.14 11 Grade 4 Social Studies 55 1 1.16 11

Grade 4 Social Studies 56 1 1.27 9 NAEP Ohio Reading: Advanced

Grade 4 Social Studies 57 1 1.31 9 Grade 4 Social Studies 58 1 1.34 8 Grade 4 Social Studies 59 1 1.52 6 Grade 4 Social Studies 60 1 1.55 5 Grade 4 Social Studies 61 2 1.56 5 Grade 4 Social Studies 62 1 1.58 5 Grade 4 Social Studies 63 2 1.59 4 Grade 4 Social Studies 64 1 1.79 3 Grade 4 Social Studies 65 2 1.83 2 Grade 4 Social Studies 66 2 1.85 2



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 4 Social Studies 67 2 2.08 1 Grade 4 Social Studies 68 2 2.22 1 Grade 4 Social Studies 69 2 2.40 0 Grade 4 Social Studies 70 3 2.52 0 Grade 4 Social Studies 71 1 2.66 0 Grade 4 Social Studies 72 1 3.03 0 Grade 4 Social Studies 73 2 3.40 0 Grade 4 Social Studies 74 4 3.66 0

Grade 6 Social Studies 1 1 -1.56 99 OAA G5 SS (2009): Limited

Grade 6 Social Studies 2 1 -1.26 96 Grade 6 Social Studies 3 1 -1.20 95 Grade 6 Social Studies 4 1 -1.10 95

Grade 6 Social Studies 5 1 -0.95 93 OAA G5 SS (2009): Basic

Grade 6 Social Studies 6 1 -0.85 91 Grade 6 Social Studies 7 1 -0.83 90 Grade 6 Social Studies 8 1 -0.76 89 Grade 6 Social Studies 9 1 -0.51 84 Grade 6 Social Studies 10 2 -0.43 82 Grade 6 Social Studies 11 1 -0.37 81 Grade 6 Social Studies 12 1 -0.33 79 Grade 6 Social Studies 13 1 -0.25 77 Grade 6 Social Studies 14 1 -0.23 77 Grade 6 Social Studies 15 1 -0.22 77 Grade 6 Social Studies 16 1 -0.20 75 NAEP Ohio: Basic Grade 6 Social Studies 17 1 -0.18 75 Grade 6 Social Studies 18 1 -0.18 75 Grade 6 Social Studies 19 1 -0.10 73 Grade 6 Social Studies 20 1 -0.02 70 Grade 6 Social Studies 21 1 0.03 68 Grade 6 Social Studies 22 1 0.05 68 Grade 6 Social Studies 23 1 0.09 66 Grade 6 Social Studies 24 1 0.15 64 Grade 6 Social Studies 25 1 0.17 64

Grade 6 Social Studies 26 2 0.22 61 OAA G5 SS (2009): Proficient



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 6 Social Studies 27 1 0.24 61 Grade 6 Social Studies 28 1 0.30 59 Grade 6 Social Studies 29 1 0.35 57 Grade 6 Social Studies 30 2 0.36 57 Grade 6 Social Studies 31 2 0.41 55 Grade 6 Social Studies 32 1 0.43 54 Grade 6 Social Studies 33 1 0.49 52 Grade 6 Social Studies 34 2 0.50 52 Grade 6 Social Studies 35 1 0.53 52 Grade 6 Social Studies 36 1 0.62 47 Grade 6 Social Studies 37 1 0.64 47 Grade 6 Social Studies 38 2 0.64 47 Grade 6 Social Studies 39 1 0.65 46

Grade 6 Social Studies 40 1 0.74 44 OAA G5 SS (2009): Accelerated

Grade 6 Social Studies 41 1 0.79 41 Grade 6 Social Studies 42 2 0.87 38 NAEP Ohio: Proficient Grade 6 Social Studies 43 2 0.91 38 Grade 6 Social Studies 44 1 0.97 36 Grade 6 Social Studies 45 1 0.98 36 Grade 6 Social Studies 46 2 1.04 33 Grade 6 Social Studies 47 1 1.06 33 Grade 6 Social Studies 48 2 1.09 30 Grade 6 Social Studies 49 1 1.09 30 Grade 6 Social Studies 50 1 1.11 30 Grade 6 Social Studies 51 2 1.14 30 Grade 6 Social Studies 52 1 1.16 27 Grade 6 Social Studies 53 2 1.17 27 Grade 6 Social Studies 54 2 1.33 22

Grade 6 Social Studies 55 1 1.35 22 OAA G5 SS (2009): Advanced

Grade 6 Social Studies 56 2 1.41 19 Grade 6 Social Studies 57 1 1.41 19 Grade 6 Social Studies 58 2 1.42 19 Grade 6 Social Studies 59 1 1.46 19 Grade 6 Social Studies 60 2 1.71 13 Grade 6 Social Studies 61 3 1.74 12



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 6 Social Studies 62 1 1.75 12 Grade 6 Social Studies 63 1 1.81 11 Grade 6 Social Studies 64 2 1.89 9 Grade 6 Social Studies 65 1 1.91 9 NAEP Ohio: Advanced Grade 6 Social Studies 66 4 2.11 6 Grade 6 Social Studies 67 1 2.69 2 Grade 6 Social Studies 68 2 2.78 2 Grade 6 Social Studies 69 2 3.08 1 Grade 6 Social Studies 70 2 3.41 0

American History 1 1 -2.47 100 ACT: below College

ReadyOGT SS (2015 All Students): Limited

American History 2 1 -1.74 99 American History 3 1 -1.50 97 American History 4 1 -1.41 96 American History 5 1 -1.26 94 American History 6 1 -1.19 93

American History 7 1 -1.12 92 OGT SS (2015 All Students): Basic

American History 8 1 -1.00 89 American History 9 1 -0.98 88 American History 10 1 -0.90 88 American History 11 1 -0.86 85 American History 12 1 -0.79 83 American History 13 1 -0.72 81

American History 14 1 -0.66 81 OGT SS (2015 All Students): Proficient

American History 15 1 -0.60 79 American History 16 1 -0.56 76 American History 17 1 -0.47 74 American History 18 1 -0.42 72 American History 19 1 -0.41 72 American History 20 1 -0.38 72 American History 21 1 -0.37 71 American History 22 1 -0.32 69 American History 23 1 -0.24 67 American History 24 1 -0.19 64 American History 25 1 -0.13 62



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



American History 26 1 -0.06 59 American History 27 2 -0.03 58 American History 28 1 0.04 56

American History 29 1 0.08 54 OGT SS (2015 All Students): Accelerated

American History 30 2 0.19 51 American History 31 1 0.20 51 American History 32 1 0.22 49 American History 33 1 0.23 49 American History 34 1 0.33 45 American History 35 1 0.34 45 American History 36 1 0.41 43 American History 37 1 0.44 42 American History 38 1 0.44 40 American History 39 1 0.47 40 ACT: College Ready American History 40 2 0.50 40 American History 41 1 0.51 37 American History 42 1 0.54 37 American History 43 1 0.60 35 American History 44 1 0.66 34 American History 45 2 0.68 32 American History 46 2 0.74 31 American History 47 1 0.76 29 American History 48 1 0.79 28 American History 49 1 0.80 28 American History 50 1 0.87 26 American History 51 1 0.88 26 American History 52 2 0.94 24 American History 53 2 0.94 24

American History 54 1 0.97 23 OGT SS (2015 All Students): Advanced

American History 55 1 1.04 21 American History 56 3 1.05 21 American History 57 1 1.08 19 American History 58 1 1.12 18 American History 59 2 1.14 18 American History 60 2 1.18 16



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



American History 61 2 1.26 14 American History 62 1 1.30 13 American History 63 1 1.32 13 American History 64 2 1.34 13 American History 65 2 1.40 11 American History 66 1 1.41 11 American History 67 1 1.58 8 American History 68 1 1.71 6 American History 69 2 1.71 6 American History 70 2 1.95 3 American History 71 2 2.01 3 American History 72 2 2.05 3 American History 73 2 2.09 2 American History 74 4 2.14 2 American History 75 2 2.33 1 American History 76 2 2.80 0 American History 77 1 2.94 0 American History 78 2 3.32 0 American History 79 2 4.08 0

American Government 1 1 -1.89 99 ACT: below College

ReadyOGT SS (2015 All Students): Limited

American Government 2 1 -1.86 99 American Government 3 1 -1.78 99 American Government 4 1 -1.49 96 American Government 5 1 -1.47 96 American Government 6 1 -1.32 95 American Government 7 1 -1.28 93 American Government 8 1 -1.23 93

American Government 9 1 -1.16 91 OGT SS (2015 All Students): Basic

American Government 10 1 -1.11 90 American Government 11 1 -1.05 88 American Government 12 1 -0.99 86 American Government 13 1 -0.89 83

American Government 14 1 -0.84 82 OGT SS (2015 All Students): Proficient

American Government 15 1 -0.79 80



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



American Government 16 1 -0.76 79 American Government 17 1 -0.71 77 American Government 18 1 -0.64 76 American Government 19 1 -0.60 73 American Government 20 1 -0.57 73 American Government 21 1 -0.49 70 American Government 22 1 -0.41 67 American Government 23 1 -0.36 64 American Government 24 1 -0.28 60 American Government 25 2 -0.25 57 American Government 26 1 -0.19 57 American Government 27 1 -0.14 54 American Government 28 2 -0.14 54

American Government 29 1 -0.07 51 OGT SS (2015 All Students): Accelerated

American Government 30 1 -0.05 51 American Government 31 1 -0.03 51 American Government 32 1 0.07 46 American Government 33 1 0.12 43 American Government 34 1 0.20 40 American Government 35 2 0.24 40 ACT: College Ready American Government 36 2 0.30 38

American Government 37 1 0.37 35 OGT SS (2015 All Students): Advanced

American Government 38 1 0.46 32 American Government 39 1 0.48 30 American Government 40 1 0.54 29 American Government 41 2 0.58 27 American Government 42 1 0.62 25 American Government 43 1 0.66 25 American Government 44 1 0.71 23 American Government 45 1 0.74 23 American Government 46 1 0.74 23 American Government 47 1 0.87 19 American Government 48 1 0.91 19 American Government 49 1 0.92 18 American Government 50 1 0.94 17



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



American Government 51 1 0.95 17 American Government 52 1 0.95 17 American Government 53 1 1.00 16 American Government 54 1 1.07 15 American Government 55 2 1.20 11 American Government 56 1 1.20 11 American Government 57 2 1.22 11 American Government 58 1 1.22 11 American Government 59 2 1.27 10 American Government 60 3 1.27 10 American Government 61 1 1.28 10 American Government 62 2 1.35 8 American Government 63 1 1.36 8 American Government 64 2 1.44 7 American Government 65 1 1.51 7 American Government 66 2 1.54 6 American Government 67 2 1.60 6 American Government 68 1 1.61 6 American Government 69 1 1.66 4 American Government 70 1 1.67 4 American Government 71 2 1.68 4 American Government 72 2 1.75 4 American Government 73 2 1.78 4 American Government 74 1 1.82 4 American Government 75 2 1.86 3 American Government 76 2 1.93 3 American Government 77 4 2.32 1 American Government 78 2 2.80 0 American Government 79 2 2.95 0 American Government 80 2 3.09 0 American Government 81 2 4.44 0

Grade 5 Science 1 1 -1.51 97 TIMMS National:

LowOAA G5S (2014): Limited

Grade 5 Science 2 1 -1.35 96 Grade 5 Science 3 1 -1.25 94 OAA G5S (2014): Basic Grade 5 Science 4 1 -1.16 93



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 5 Science 5 1 -1.02 91 Grade 5 Science 6 1 -0.97 89 Grade 5 Science 7 1 -0.92 88 Grade 5 Science 8 1 -0.91 88 Grade 5 Science 9 1 -0.84 87 Grade 5 Science 10 1 -0.80 85 Grade 5 Science 11 1 -0.71 83 Grade 5 Science 12 1 -0.66 81

Grade 5 Science 13 1 -0.62 80 TIMMS National: Intermediate

Grade 5 Science 14 1 -0.56 78 Grade 5 Science 15 1 -0.51 78 Grade 5 Science 16 1 -0.49 76 Grade 5 Science 17 1 -0.43 75 Grade 5 Science 18 1 -0.39 73

Grade 5 Science 19 1 -0.33 71 OAA G5S (2014): Proficient

Grade 5 Science 20 1 -0.29 70 Grade 5 Science 21 1 -0.25 68 Grade 5 Science 22 1 -0.20 67 Grade 5 Science 23 1 -0.14 65 Grade 5 Science 24 1 -0.08 62 Grade 5 Science 25 2 -0.07 62 Grade 5 Science 26 1 -0.04 62 Grade 5 Science 27 1 -0.03 62 Grade 5 Science 28 1 -0.03 62 Grade 5 Science 29 1 0.01 59 Grade 5 Science 30 1 0.09 56 Grade 5 Science 31 1 0.15 53 Grade 5 Science 32 1 0.15 53

Grade 5 Science 33 1 0.16 53 TIMMS National:

HighOAA G5S (2014): Accelerated

Grade 5 Science 34 1 0.19 53 Grade 5 Science 35 1 0.30 47 Grade 5 Science 36 1 0.37 45 Grade 5 Science 37 1 0.39 44 Grade 5 Science 38 2 0.41 44



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 5 Science 39 1 0.41 44 Grade 5 Science 40 1 0.45 42 Grade 5 Science 41 1 0.57 38 Grade 5 Science 42 1 0.62 36 Grade 5 Science 43 1 0.66 35 Grade 5 Science 44 1 0.66 35 Grade 5 Science 45 1 0.71 33 Grade 5 Science 46 2 0.75 32 Grade 5 Science 47 1 0.76 32 Grade 5 Science 48 1 0.78 32 Grade 5 Science 49 1 0.79 30 Grade 5 Science 50 1 0.83 29 Grade 5 Science 51 1 0.90 26 Grade 5 Science 52 1 0.91 26 Grade 5 Science 53 3 0.91 26 Grade 5 Science 54 2 0.97 24 Grade 5 Science 55 2 1.06 23

Grade 5 Science 56 1 1.08 21 OAA G5S (2014): Advanced

Grade 5 Science 57 1 1.12 20

Grade 5 Science 58 1 1.13 20 TIMMS National: Advanced

Grade 5 Science 59 2 1.17 18 Grade 5 Science 60 1 1.25 17 Grade 5 Science 61 1 1.34 15 Grade 5 Science 62 1 1.37 15 Grade 5 Science 63 1 1.42 13 Grade 5 Science 64 1 1.58 10 Grade 5 Science 65 2 1.71 8 Grade 5 Science 66 1 1.74 7 Grade 5 Science 67 2 1.81 7 Grade 5 Science 68 2 1.84 6 Grade 5 Science 69 1 1.85 6 Grade 5 Science 70 4 2.49 1 Grade 5 Science 71 2 2.64 1 Grade 5 Science 72 2 2.71 1 Grade 5 Science 73 2 3.89 0



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of




Grade 8 Science 1 1 -1.66 93

NAEP Ohio: below BasicTIMMS National: LowOAA G8S (2014):

Basic Grade 8 Science 2 1 -1.51 91 Grade 8 Science 3 1 -1.39 89 Grade 8 Science 4 1 -1.36 88 Grade 8 Science 5 1 -1.34 88 Grade 8 Science 6 1 -1.30 86 Grade 8 Science 7 1 -1.23 85 Grade 8 Science 8 1 -1.19 83 Grade 8 Science 9 1 -1.14 82 Grade 8 Science 10 1 -1.08 80 Grade 8 Science 11 1 -1.01 79 Grade 8 Science 12 1 -0.94 76

Grade 8 Science 13 1 -0.86 73 NAEP Ohio: Basic; TIMMS National:

Intermediate Grade 8 Science 14 1 -0.78 69

Grade 8 Science 15 1 -0.71 67 OAA G8S (2014): Proficient

Grade 8 Science 16 1 -0.67 66 Grade 8 Science 17 1 -0.62 64 Grade 8 Science 18 1 -0.58 63 Grade 8 Science 19 1 -0.53 60 Grade 8 Science 20 1 -0.52 60 Grade 8 Science 21 1 -0.51 60 Grade 8 Science 22 1 -0.48 58 Grade 8 Science 23 1 -0.42 57 Grade 8 Science 24 1 -0.41 57 Grade 8 Science 25 1 -0.33 54 Grade 8 Science 26 1 -0.26 51 Grade 8 Science 27 1 -0.21 48 Grade 8 Science 28 1 -0.21 48 Grade 8 Science 29 1 -0.20 48 Grade 8 Science 30 1 -0.16 46 Grade 8 Science 31 2 -0.15 46



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 8 Science 32 1 -0.12 45 Grade 8 Science 33 1 -0.11 45 Grade 8 Science 34 1 -0.06 42 Grade 8 Science 35 1 -0.05 42 TIMMS National: High Grade 8 Science 36 1 0.01 40 NAEP Ohio: Proficient Grade 8 Science 37 1 0.06 38 Grade 8 Science 38 2 0.08 38

Grade 8 Science 39 1 0.09 37 OAA G8S (2014): Accelerated

Grade 8 Science 40 1 0.12 37 Grade 8 Science 41 1 0.16 34 Grade 8 Science 42 1 0.25 32 Grade 8 Science 43 1 0.25 32 Grade 8 Science 44 1 0.32 30 Grade 8 Science 45 1 0.35 29 Grade 8 Science 46 2 0.38 27 Grade 8 Science 47 1 0.40 27

Grade 8 Science 48 1 0.48 25 OAA G8S (2014): Advanced


Grade 8 Science 50 1 0.53 22 TIMMS National: Advanced

Grade 8 Science 51 3 0.70 18 Grade 8 Science 52 1 0.80 16 Grade 8 Science 53 1 0.87 14 Grade 8 Science 54 2 0.88 14 Grade 8 Science 55 1 0.88 14 Grade 8 Science 56 1 0.90 14 Grade 8 Science 57 1 0.91 14 Grade 8 Science 58 1 0.92 14 Grade 8 Science 59 1 0.99 12 Grade 8 Science 60 1 1.06 10 Grade 8 Science 61 2 1.08 10 Grade 8 Science 62 2 1.19 8 Grade 8 Science 63 1 1.21 8 Grade 8 Science 64 1 1.29 7 Grade 8 Science 65 1 1.32 7



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Grade 8 Science 66 2 1.40 5 NAEP Ohio: Advanced Grade 8 Science 67 1 1.46 5 Grade 8 Science 68 2 1.53 4 Grade 8 Science 69 4 1.64 3 Grade 8 Science 70 1 1.65 3 Grade 8 Science 71 1 2.12 1 Grade 8 Science 72 2 2.16 1 Grade 8 Science 73 1 2.48 0 Grade 8 Science 74 2 2.69 0 Grade 8 Science 75 1 2.80 0 Grade 8 Science 76 1 2.83 0 Grade 8 Science 77 2 3.71 0

Physical Science 1 1 -2.22 98 ACT: below College

ReadyOGT Sci (2015 All Students): Limited

Physical Science 2 1 -2.10 98

Physical Science 3 1 -1.81 93 OGT Sci (2015 All Students): Basic

Physical Science 4 1 -1.73 93 Physical Science 5 1 -1.63 90 Physical Science 6 1 -1.56 87 Physical Science 7 1 -1.48 87 Physical Science 8 1 -1.47 87 Physical Science 9 1 -1.43 83 Physical Science 10 1 -1.38 83 Physical Science 11 1 -1.32 82 Physical Science 12 1 -1.26 78 Physical Science 13 1 -1.22 77

Physical Science 14 1 -1.17 74 OGT Sci (2015 All Students): Proficient

Physical Science 15 1 -1.14 72 Physical Science 16 1 -1.09 72 Physical Science 17 1 -1.04 68 Physical Science 18 1 -1.00 68 Physical Science 19 1 -0.94 63 Physical Science 20 1 -0.91 63 Physical Science 21 1 -0.87 58 Physical Science 22 1 -0.82 58



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Physical Science 23 1 -0.75 54 Physical Science 24 1 -0.72 54 Physical Science 25 1 -0.71 54 Physical Science 26 1 -0.69 50 Physical Science 27 1 -0.68 50 Physical Science 28 1 -0.62 50 Physical Science 29 1 -0.57 46 Physical Science 30 1 -0.56 46

Physical Science 31 1 -0.53 45 OGT Sci (2015 All Students): Accelerated

Physical Science 32 2 -0.48 42 Physical Science 33 1 -0.47 42 Physical Science 34 1 -0.42 38 Physical Science 35 1 -0.41 38 Physical Science 36 1 -0.33 35 Physical Science 37 1 -0.26 32 Physical Science 38 1 -0.25 32 Physical Science 39 1 -0.19 31 Physical Science 40 1 -0.11 28 ACT: College Ready Physical Science 41 2 -0.07 25 Physical Science 42 1 -0.03 25 Physical Science 43 1 0.00 23

Physical Science 44 1 0.00 23 OGT Sci (2015 All Students): Advanced

Physical Science 45 1 0.02 22 Physical Science 46 1 0.06 21 Physical Science 47 1 0.10 20 Physical Science 48 1 0.24 16 Physical Science 49 2 0.30 15 Physical Science 50 1 0.31 15 Physical Science 51 1 0.34 14 Physical Science 52 1 0.34 14 Physical Science 53 1 0.36 13 Physical Science 54 2 0.39 12 Physical Science 55 1 0.40 12 Physical Science 56 1 0.46 11 Physical Science 57 1 0.53 10 Physical Science 58 2 0.59 8 Physical Science 59 3 0.72 7 Physical Science 60 1 0.77 6 Physical Science 61 1 0.86 5 Physical Science 62 1 0.89 5



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Physical Science 63 1 0.95 4 Physical Science 64 2 1.05 4 Physical Science 65 1 1.06 4 Physical Science 66 1 1.14 3 Physical Science 67 2 1.39 2 Physical Science 68 2 1.42 2 Physical Science 69 4 1.44 2 Physical Science 70 2 1.48 1 Physical Science 71 2 1.67 1 Physical Science 72 1 1.89 1 Physical Science 73 2 2.09 0

Biology 1 1 -2.29 98 ACT: below College

Ready; OGT Sci (2015 All Students): Limited

Biology 2 1 -1.89 94

Biology 3 1 -1.79 92 OGT Sci (2015 All Students): Basic

Biology 4 1 -1.71 91 Biology 5 1 -1.64 90 Biology 6 1 -1.58 89 Biology 7 1 -1.54 88 Biology 8 1 -1.47 86 Biology 9 1 -1.43 85 Biology 10 1 -1.36 84 Biology 11 1 -1.32 83 Biology 12 1 -1.26 81 Biology 13 1 -1.19 79 Biology 14 1 -1.10 76 Biology 15 1 -1.08 75

Biology 16 2 -1.01 73 OGT Sci (2015 All Students): Proficient

Biology 17 1 -1.00 72 Biology 18 1 -0.99 72 Biology 19 1 -0.92 69 Biology 20 1 -0.88 68 Biology 21 1 -0.85 67 Biology 22 1 -0.81 65 Biology 23 1 -0.78 64 Biology 24 2 -0.78 64 Biology 25 1 -0.73 62 Biology 26 1 -0.67 60 Biology 27 1 -0.65 58 Biology 28 1 -0.57 56



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Biology 29 1 -0.52 53 Biology 30 1 -0.48 51 Biology 31 1 -0.48 51 Biology 32 1 -0.39 48 Biology 33 1 -0.39 48 Biology 34 1 -0.39 48 Biology 35 1 -0.38 48 Biology 36 1 -0.36 47

Biology 37 1 -0.26 43 OGT Sci (2015 All Students): Accelerated

Biology 38 1 -0.21 41 Biology 39 1 -0.17 39 Biology 40 2 -0.09 36 Biology 41 1 -0.08 36 Biology 42 1 -0.06 35 Biology 43 1 -0.05 35 Biology 44 1 -0.01 33 Biology 45 1 0.01 32 Biology 46 1 0.07 30 Biology 47 2 0.09 30 Biology 48 1 0.10 29 Biology 49 1 0.18 27 Biology 50 1 0.18 26 ACT: College Ready Biology 51 1 0.19 26 Biology 52 1 0.23 25

Biology 53 1 0.32 22 OGT Sci (2015 All Students): Advanced

Biology 54 1 0.33 22 Biology 55 1 0.34 22 Biology 56 1 0.36 21 Biology 57 1 0.39 20 Biology 58 1 0.39 20 Biology 59 1 0.44 19 Biology 60 1 0.46 19 Biology 61 1 0.47 18 Biology 62 1 0.50 17 Biology 63 2 0.51 17 Biology 64 1 0.57 16 Biology 65 2 0.71 13 Biology 66 1 0.93 9 Biology 67 1 0.99 8 Biology 68 2 1.25 5



Test

Item Map

Order

Item Score

Category

RP67/ RP50 (High

School Science)

Overall Percent of



Biology 69 1 1.26 5 Biology 70 1 1.47 3 Biology 71 2 1.48 3 Biology 72 2 1.61 2 Biology 73 2 1.63 2 Biology 74 2 1.65 2 Biology 75 2 1.94 1 Biology 76 2 1.98 1 Biology 77 2 2.43 0 Biology 78 2 2.43 0


Appendix

Appendix F – OIB Item Data Plots


F-1 American Institutes for Research

Figure F1. OIB Item Data Plot – Grade 4 Social Studies



Figure F2. OIB Item Data Plot – Grade 6 Social Studies



Figure F3. OIB Item Data Plot – American History



Figure F4. OIB Item Data Plot – American Government



Figure F5. OIB Item Data Plot – Grade 5 Science



Figure F6. OIB Item Data Plot – Grade 8 Science



Figure F7. OIB Item Data Plot – Physical Science



Figure F8. OIB Item Data Plot – Biology


Appendix

Appendix G –Bookmark Placement Readiness Forms


G-1 American Institutes for Research

Document G. OCBA Bookmark Placement Readiness Form Test: ______________________________ Panelist ID number _______________ Preparation for Round 1 – Basic - Level 2, Proficient - Level 3, Accelerated - Level 4, and Advanced - Level 5

Yes No

a. The workshop training has prepared me to review the Performance Level Descriptors

b. The training fully explained the concept of a student who just barely meets the criteria described in the Ohio Performance Level Descriptors.

c. The workshop training has prepared me to review the Ordered Item Book (OIB).

d. The workshop training has prepared me to set my and confirm my bookmarks.

I have answered, “Yes” to the above questions and I understand what I need to do to place my Bookmarks. Yes _____ No _____ Initials __________ If I answered “No” to any of the above questions, I received additional training. Yes _____ No _____ Initials __________ Following the additional training, I now feel sufficiently trained on what I need to do to place my Bookmarks. Yes _____ No _____ Initials __________


G-2 American Institutes for Research

Test ____________________________ Panelist ID number _____________ Preparation for Round 2 – Basic - Level 2, Proficient - Level 3, Accelerated - Level 4, and Advanced - Level 5

Yes No

a. The workshop training has prepared me to review the Performance Level Descriptors

b. The training fully explained the concept of a student who just barely meets the criteria described in the Ohio Performance Level Descriptors.

c. The workshop training has prepared me to review the Ordered Item Book (OIB).

d. The workshop training has prepared me to set and confirm and bookmarks.

e. The training fully explained the agreement data (room medians and bookmarks of fellow panelists) that was presented.

f. The training fully explained the impact data (percent of students estimated to meet the standards) that was presented.

I have answered, “Yes” to the above questions and I understand what I need to do to place my Bookmarks. Yes _____ No _____ Initials __________ If I answered “No” to any of the above questions, I received additional training. Yes _____ No _____ Initials __________ Following the additional training, I now feel sufficiently trained on what I need to do to place my Bookmarks. Yes _____ No _____ Initials __________


Appendix

Appendix H – Panelist Evaluation Forms


H-1 American Institutes for Research

Document H. Overall Workshop Evaluations

Please take your time to carefully complete the information below. It is important that you answer this evaluation thoroughly as the results will be used to improve the Standard Setting process. The information gathered from this evaluation will be reported in the Standard Setting Technical Report that will be available to the public. Panelist ID Number: __________

1. At the end of the workshop,


Agree

a. I understood the purpose of this standard setting workshop.

b. The procedures used to recommend performance standards were fair and unbiased.

c. The training provided me with the information I needed to recommend performance standards.

d. Taking the online assessment helped me to better understand what students need to know and be able to do to answer each item.

e.

The Performance Level Descriptors (description of what students within each performance level are expected to know and be able to do) provided a clear picture of expectations for student achievement at each level.

f.

I was able to develop an understanding of the knowledge and skills demonstrated by students who are “just barely” described by the Performance Level Descriptors.

g.

I understood how to review each page in the Ordered Item Book (OIB) to determine what students must know and be able to do to answer each item correctly.

h. I was able to interpret having a two-thirds likelihood of answering an item correctly as indicating mastery.

i. I understood how to place my bookmarks.

j. I found the benchmark data and discussions helpful in my decisions about where to place my bookmarks.




Agree

k.

I found the panelist agreement data (room medians and individual bookmark placements) and discussion helpful in my decisions about where to place my bookmarks.

l.

I found the impact data (percentage of students that would achieve at the level indicated by the OIB page) and discussions helpful in my decisions about where to place my bookmarks.

m. I felt comfortable expressing my opinions throughout the workshop.

n. Everyone was given the opportunity to express his or her opinions throughout the workshop.

2. Please rate the clarity of the following components of the workshop.

Very Unclear

Somewhat Unclear

Somewhat Clear

Very Clear

a. Instructions provided by the Workshop Leader

b. Performance Level Descriptors (PLDs)

c. Ordered Item Booklet (OIB)

d. Panelist agreement data

e. Impact data (percentage of students that would achieve at the level indicated by the OIB page)



3. How important was each of the following factors in your placement of the bookmarks?

Not Important

Somewhat Important

Very Important

a. Performance Level Descriptors (PLDs)

b. Your perception of the difficulty of the items

c. Your experiences with students

d. Discussions with other panelists

e. External benchmark data

f. Room agreement data (room medians and individual bookmark placements)

g. Impact data (percentage of students that would achieve at the level indicated by the OIB page)

4. How appropriate was the amount of time you were given to complete the following components of the standard setting process?

Too Little

About Right

Too Much

a. Large group orientation

b. Experiencing the online assessment

c. Review of the Performance Level Descriptors

d. Discussion of skills demonstrated by students who are “just barely” described by each PLD

e. Review of the Ordered Item Booklet (OIB)

f. Placement of your bookmarks in each round

g. Round 1 discussion

5. Please read the following statement carefully and indicate your response.




Agree

a.

A student performing at the Proficient - Level 3 demonstrates an appropriate command of Ohio’s Learning Standards for Science and Social Studies. (Level 3)

b. A student performing at the Basic – Level 2 demonstrates a partial command of Ohio’s Learning Standards for Science and Social Studies. (Level 2)

c. A student performing at the Accelerated – Level 4 demonstrates a strong command of Ohio’s Learning Standards for Science and Social Studies. (Level 4)

d.

A student performing at the Advanced – Level 5 demonstrates a distinguished command of Ohio’s Learning Standards for Science and Social Studies. (Level 5)

6. What suggestions do you have to improve the training or standard setting process?

7. Do you have any additional comments? Please be specific.

Thank you for participating in the Standard Setting Workshop!


Appendix

Appendix I – Recommend Performance Standards by Round


I-1 American Institutes for Research

Table I. Recommended Cuts By Round

Form Round Measure Level 2 - Basic

Level 3 - Proficient

Level 4 - Accelerated

Level 5 - Advanced


Round 1 Minimum 5 11 34 52 Maximum 14 29 60 70 Median 8 21 41 63


Moderation -- -- -- -- -- Final -- 8 19 41 62





American History



Moderation -- -- -- 43 58 Final -- 9 21 43 58

American Government




Grade 5 Science


Round 2 Minimum 3 16 35 58 Maximum 5 24 41 68


I-2 American Institutes for Research

Form Round Measure Level 2 - Basic

Level 3 - Proficient

Level 4 - Accelerated

Level 5 - Advanced

Median 4 20 38 60 Moderation -- 7 26 41 -- Final -- 7 26 41 60

Grade 8 Science




Physical Science




Biology





Appendix

Appendix J – Convergence of Bookmarks across Rounds


J-1 American Institutes for Research

Figure J1. Convergence of Bookmarks across Rounds – Grade 4 Social Studies

The figure illustrates how variability in panelists’ bookmark decisions changed from the first to the second round and displays the median bookmark for each table, from Round 1 of bookmark placement to Round 2 of bookmark placement.

Round 1 Bookmark Placement Round 2 Bookmark PlacementTable 1 Advanced 61 61Table 2 Advanced 63 62Table 1 Accelerated 40 40Table 2 Accelerated 41 41Table 1 Proficient 19 19Table 2 Proficient 22 19Table 1 Basic 14 8Table 2 Basic 8 8

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

OIB

Pag

e N

umbe

r




Figure J2. Convergence of Bookmarks across Rounds – Grade 6 Social Studies



0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

OIB

Pag

e N

umbe

r




Figure J3. Convergence of Bookmarks Across Rounds – American History



0

5

10

15

20

25

30

35

40

45

50

55

60

65

OIB

Pag

e N

umbe

r

American History



Figure J4. Convergence of Bookmarks Across Rounds – American Government



0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

OIB

Pag

e N

umbe

r

American Government



Figure J5. Convergence of Bookmarks Across Rounds – Grade 5 Science



0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

OIB

Pag

e N

umbe

r

Grade 5 Science



Figure J6. Convergence of Bookmarks across Rounds – Grade 8 Science



0

5

10

15

20

25

30

35

40

45

50

55

60

65

OIB

Pag

e N

umbe

r

Grade 8 Science



Figure J7. Convergence of Bookmarks across Rounds – Biology



0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

OIB

Pag

e N

umbe

r

Biology



Figure J8. Convergence of Bookmarks across Rounds – Physical Science



0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

OIB

Pag

e N

umbe

r

Physical Science


Appendix

Appendix K – Estimated Percentage of Students at Each Performance Level for Panelist Recommended Performance Standards, Overall and by Gender and Ethnicity


K-1 American Institutes for Research

Table K. Estimated Percentage of Students at Each Performance Level for Panelist Recommended Performance Standards, Overall and by Gender and Ethnicity

Test Performance Level Overall Female Male White

African American Hispanic Asian

Multi-Ethnic

Native American Other


Limited 12 11 12 7 28 18 8 15 14 26 Basic 19 19 19 16 32 27 10 24 23 48 Proficient 41 42 40 43 32 40 32 40 41 21 Accelerated 24 24 25 29 7 14 34 18 20 5 Advanced 5 4 5 6 1 2 15 3 1 0



American History


American Government


Grade 5 Science



K-2 American Institutes for Research

Test Performance Level Overall Female Male White

African American Hispanic Asian

Multi-Ethnic

Native American Other

Grade 8 Science


Physical Science


Biology



Appendix

Appendix L – Summary of Panelist Evaluations


L-1 American Institutes for Research

Document L. Summary of Panelist Evaluations 1. At the end of the workshop,


Agree

a. I understood the purpose of this standard setting workshop. 2 0 17 57

b. The procedures used to recommend performance standards were fair and unbiased. 3 2 31 40

c. The training provided me with the information I needed to recommend performance standards. 2 1 20 53

d. Taking the online assessment helped me to better understand what students need to know and be able to do to answer each item.

2 1 15 58

e.

The Performance Level Descriptors (description of what students within each performance level are expected to know and be able to do) provided a clear picture of expectations for student achievement at each level.

1 13 26 36

f.

I was able to develop an understanding of the knowledge and skills demonstrated by students who are “just barely” described by the Performance Level Descriptors.

2 3 34 37

g.

I understood how to review each page in the Ordered Item Book (OIB) to determine what students must know and be able to do to answer each item correctly.

2 0 23 51

h. I was able to interpret having a two-thirds likelihood of answering an item correctly as indicating mastery.

2 1 36 37

i. I understood how to place my bookmarks. 2 0 20 54

j. I found the benchmark data and discussions helpful in my decisions about where to place my bookmarks.

3 4 16 53

k.

I found the panelist agreement data (room medians and individual bookmark placements) and discussion helpful in my decisions about where to place my bookmarks.

2 0 24 50

l.

I found the impact data (percentage of students that would achieve at the level indicated by the OIB page) and discussions helpful in my decisions about where to place my bookmarks.

4 3 26 43




Agree

m. I felt comfortable expressing my opinions throughout the workshop. 2 1 16 57

n. Everyone was given the opportunity to express his or her opinions throughout the workshop. 2 0 12 62

1. Please rate the clarity of the following components of the workshop.

Very Unclear

Somewhat Unclear

Somewhat Clear

Very Clear

a. Instructions provided by the Workshop Leader 0 1 6 69

b. Performance Level Descriptors (PLDs) 1 4 22 49

c. Ordered Item Booklet (OIB) 0 0 3 73

d. Panelist agreement data 0 0 3 73

e. Impact data (percentage of students that would achieve at the level indicated by the OIB page) 1 2 10 63

2. How important was each of the following factors in your placement of the bookmarks?

Not Important

Somewhat Important

Very Important

a. Performance Level Descriptors (PLDs) 1 12 63

b. Your perception of the difficulty of the items 5 26 45

c. Your experiences with students 1 23 52

d. Discussions with other panelists 1 23 52

e. External benchmark data 3 48 25

f. Room agreement data (room medians and individual bookmark placements) 3 47 26

g. Impact data (percentage of students that would achieve at the level indicated by the OIB page) 4 34 38



3. How appropriate was the amount of time you were given to complete the following components of the standard setting process?

Too Little

About Right

Too Much

a. Large group orientation 1 49 26

b. Experiencing the online assessment 1 69 6

c. Review of the Performance Level Descriptors 8 63 5

d. Discussion of skills demonstrated by students who are “just barely” described by each PLD 8 63 5

e. Review of the Ordered Item Booklet (OIB) 13 58 5

f. Placement of your bookmarks in each round 0 60 16

g. Round 1 discussion 4 70 2

4. Please read the following statement carefully and indicate your response.


Agree

a.

A student performing at the Proficient - Level 3 demonstrates an appropriate command of Ohio’s Learning Standards for Science and Social Studies. (Level 3)

1 0 43 32

b. A student performing at the Basic – Level 2 demonstrates a partial command of Ohio’s Learning Standards for Science and Social Studies. (Level 2)

1 2 44 29

c. A student performing at the Accelerated – Level 4 demonstrates a strong command of Ohio’s Learning Standards for Science and Social Studies. (Level 4)

1 0 36 39

d.

A student performing at the Advanced – Level 5 demonstrates a distinguished command of Ohio’s Learning Standards for Science and Social Studies. (Level 5)

2 0 29 45

Recommending Ohio Computer Based Assessment (OCBA) Performance … · 2015. 9. 2. · Based...

Documents

Transcript of Recommending Ohio Computer Based Assessment (OCBA) Performance … · 2015. 9. 2. · Based...