Guidance Document Quick Start-FEB14-FINAL

QUICK START

Guidance

Document

2014

PART OF THE ASSESSMENT LITERACY SERIES THE RIA GROUP

| 16407 Highland Club Avenue Baton Rouge, LA 70817

Guidance Document: Quick Start©

1

Quick Start© Program Guide

Introduction

The purpose of this document is to provide guidance for developing measures of student

performance that will meet the criteria within the Performance Measure Rubric. The rubric is a

self-assessment tool used to ascertain the technical quality of locally-developed performance

measures. The process used to “design”, “build”, and “review” teacher-made performance

measures is contained within the Quick Start program. Quick Start delivers a

foundational understanding of the procedures necessary to create these performance measures,

which teachers may then use to assess their students’ skills, knowledge, and concept mastery of

targeted content standards.

Figure 1. Process Components

Contact

For more information:

www.hr.riagroup2013.com

[email protected]

Des

ign

• Purpose Statement

• Targeted Content Standards

• Test Blueprint

Bu

ild

• Items/Tasks

• Scoring Keys & Scoring Rubrics

• Test Forms

Rev

iew

• Item/Task Reviews

• Alignment Reviews

• Data Reviews

• Refinements

http://www.hr.riagroup2013.com/

mailto:[email protected]


2

Phase I: Design

1.1 Goal Statement

Understand and apply the techniques used to design measures of student performance.

1.2 Objectives

The professional will successfully:

o Create a “purpose statement” for a specific performance measure;

o Identify an “Enduring Understanding/Key Concept” and its associated content

standards for a specific course/area of study; and,

o Develop a test blueprint outlining the performance measure’s structure.

1.3 Guiding Questions

What is the performance measure intended to measure and at what grade?

What are the developmental characteristics of test-takers?

Which areas will be targeted among the various content standards?

How will educators use the results (overall score and “growth” inferences)?

When will the performance measure be administered?

Do the items/tasks capture the content standards within the key concept?

Is the number of items/tasks sufficient so that students at varying levels can demonstrate

their knowledge?

What are the time demands for both teachers and students?

How does the design reflect the areas of emphasis in the standards?

1.4 Resources

State Content Standards, National Standards, and Common Core Standards

Teacher-made items/tasks


3

Department/grade-level projects, experiments, portfolios, performance demonstrations,

and writing journals

Textbooks and other ancillary materials

Handout #1 – Designing the Assessment-Examples

Handout #3 – Performance Measure Rubric [Scored Example]

Template #1 – Designing the Assessment

1.5 Procedural Steps

STEP 1. Convene a group of professional educators (e.g., Professional Learning

Communities) who are familiar with the content standards and grade level to be

assessed. A lead facilitator should be designated and agreed upon within the

group.

STEP 2. Decide how participants will reach consensus (100% agreement, majority, or

another measure).

STEP 3. Develop a plan and timeline for completing the development and review of

assessment measures.

STEP 4. Create a purpose statement for the assessment using the procedural steps below.

See Handout #1 for examples. Refer to Template #1 for more information.

Procedural Steps: Create a Purpose Statement

1. Individually create a statement about the performance measure in terms of

the content standards it purports to measure.

2. Build consensus by focusing on three components of the statement: What,

How, and Why.

3. Draft three (3) sentences reflecting the group’s consensus for each

component and review.

4. Merge sentences to create a single paragraph “statement”. Again, review

to ensure that the statement reflects the group’s intent.

5. Finalize the statement and double-check for editorial soundness.

Framework: Create a Purpose Statement

What-___________________________________________________________

How-____________________________________________________________

Why-____________________________________________________________


4

STEP 5. Identify targeted content standards using the procedural steps below. See

Handout #1 for examples. Refer to Template #1 for more information.

Procedural Steps: Select Targeted Content Standards

1. Place the course/subject’s name and Enduring Understanding/Key

Concept statement above the Targeted Content Standards table.

2. Place the code for each standard/content strand in the Content ID column

along with a description for each content standard in the Content

Statement column.

3. Have subject matter experts work collaboratively to identify an initial (i.e.,

draft) set of content standards associated with the Big Idea/Key Concept.

4. Review the list of targeted content standards. Look for gaps and/or

redundancies and then finalize the list by placing an “X” in the Final

column.

5. Verify that the “final” targeted content standards will be those used to

develop the test blueprint.

Framework: Select Targeted Content Standards

Subject/Course: ___________________

Enduring Understanding/Key Concept: _______________________________

Targeted Content Standards

Content ID Content Statement Draft Final

STEP 6. Develop a test blueprint based on the targeted content standards identified in Step

5. Using the procedural steps on the next page, determine the number of items

assigned to each targeted content standard and the associated levels of cognitive

demand. See Handout #1 for examples. Refer to Template #1 for more

information.


5

Procedural Steps: Develop a Test Blueprint

1. Review the selected targeted content standards.

2. Insert selected “Big Idea”/Key Concept and targeted content standards

(numeric code only) into the test blueprint table.

3. Determine the number of items/tasks across the four (4) cognitive levels.

4. Tally the rows and place the values in the Total column. Tally each

cognitive level column and place the resultant values in the Grand Totals

row.

5. Report the total number of items/tasks and the total possible points

available.

Framework

Subject/Course: __________________________________

Enduring

Understanding/

Key Concept

Targeted Content Standard

Item/Task Cognitive Level

Level

1

Level

2

Level

3

Level

4 Total

Grand Totals

Technical Note: This assessment is comprised of ___________ items [worth _____ point each],

________items [worth ____ points each], and _______ extended performance task [worth

______points] for a total of_____ total items/tasks. The maximum score possible on this

assessment is ____ points.

1.6 Quality Reviews

The Performance Measure Rubric is designed to help the educator review items/tasks, scoring

rubrics, and assessment forms to create high-quality performance measures. Applying the

criteria within the Performance Measure Rubric allows the educator to evaluate further the

assessment quality.

Strand 1 of the Performance Measure Rubric evaluates the Design phase of the assessment

process (purpose statement, targeted content standards, and test blueprint). Use the following

procedural steps to perform a quality review of this phase. Refer to Handout #3 Performance

Measure Rubric-Scored Example for more information.


6

Rating Steps: Performance Measure Rubric Strand 1: Design

STEP 1. Review information, data, and documents associated with the

development of the selected performance measure.

STEP 2. Assign a value in the “Rating” column for each aspect within a

particular strand using the following rating scale:

a. (1) = fully addressed

b. (.5) = partially addressed

c. (0) = not addressed

d. (N/A) = not applicable at this time

STEP 3. Reference supporting information associated with each assigned rating

in the “Evidence” column.

STEP 4. In the bottom row, add any additional notations and/or comments that

articulate important nuances of the performance measure.

STEP 5. Compile assigned values and place in the “Strand Summary” row.

STRAND 1: DESIGN

Task

ID

Descriptor Rating Evidence

1.1 The purpose of the performance measure is explicitly stated (who, what, why).

1.2 The performance measure has targeted content standards representing a range

of knowledge and skills students are expected to know and demonstrate.

1.3

The performance measure’s design is appropriate for the intended audience

and reflects challenging material needed to develop higher-order thinking

skills.

1.4

Specification tables articulate the number of items/tasks; item/task types,

passage readability, and other information about the performance measure –

OR – Blueprints are used to align items/tasks to targeted content standards.

1.5

Items/tasks are rigorous (designed to measure a range of cognitive

demands/higher-order thinking skills at developmentally appropriate levels)

and of sufficient quantities to measure the depth and breadth of the targeted

content standards.

Strand 1 Summary __out

of 5

Additional Comments/Notes


7

Phase II: Build

2.1 Goal Statement

Understand and apply the techniques used to build measures of student performance.

2.2 Objectives


o Create the necessary items/tasks to address the test blueprint;

o Develop scoring keys and/or scoring rubrics; and,

o Organize items/tasks and administration guidelines into a test form.


Are the items aligned with targeted content standards?

Do the selected items/tasks allow students to demonstrate content knowledge by:

o Responding to questions and/or prompts?

o Performing tasks, actions, and/or demonstrations?

Do the items/tasks measure content knowledge, skill, or process and not an external or

environmental factor (e.g., guessing)?

Is the number of items/tasks sufficient to sample the targeted content?

Are the items/tasks developmentally appropriate for the intended test-takers?

Are the correct answers and/or expected responses clearly identified?

Do the performance measure’s directions specify:

o What the test-taker should do, read, or analyze?

o Where and how the test-taker should respond or demonstrate the task?

o How many points a correct/complete response is worth towards the overall score?

Are there directions for different item/task types?


8

2.4 Resources

Teacher-made items/tasks/projects

Textbooks and other ancillary materials

Formative assessment materials, curriculum-based measures

Handout #2 – Building the Assessment-Examples

Handout #3 – Performance Measure Rubric [Scored Example]

Template #2 – Building the Assessment


STEP 1. Develop items/tasks according to the test blueprint, using the procedural steps for

Multiple Choice (MC) items, Short Answer (SA) items, Extended Answer (EA)

items, and Extended Performance (EP) tasks as listed below. See Handout #2 for

examples. Refer to Template #2 for more information.

Procedural Steps: Multiple Choice (MC) Items

1. Review the targeted content standard.

2. Determine which aspects of the standard can be measured objectively.

3. Select the focused aspect and determine the cognitive demand reflected in

the standard’s description.

4. Create a question (stem), one correct answer, and plausible (realistic)

distractors.

5. Review the item and answer options for grammatical soundness.

Framework: MC

1. ___<Item Stem>______________________________________________

A. __<Answer Option>__

B. __<Answer Option>__

C. __<Answer Option>__

D. __<Answer Option>__ (___<Item Tag>___)


9

Procedural Steps: SA/EA Items

1. Review the targeted content standard(s).

2. Determine which aspects of the standard(s) can be best measured by

having students “construct” a response.

3. Select and list aspects of the targeted content standard(s) to be measured.

4. Create a prompt, select a passage, or develop a scenario for students.

5. Develop a clear statement that articulates specific criteria for the test-taker

to provide.

Framework: SA/EA Items

Directions:

__________________________________________________________________

__________________________________________________________________

<Task/Passage/Scenario>

___<Item Stem>____________________________________________________

__<Response Area>_________________________________________________

(___<Item Tag>___)

Procedural Steps: Extended Performance (EP) Tasks

1. Review the targeted content standard(s).

2. Determine which aspects of the standard(s) can be best measured by

having students “develop” a complex response, demonstration, or

performance over an extend period of time (e.g., two weeks)

3. Select and list all aspects of the targeted content standard(s) to be

measured.

4. Create a project, portfolio, or demonstration expectation statement that

includes subordinate tasks, which are aligned to the test blueprint.

5. Develop a clear statement for each subordinate task that articulates

specific criteria for the test-taker to provide.


10

Framework: EP Tasks

__________________________________________________________________

__________________________________________________________________

<Project-Portfolio-Demonstration Description>

___<Task __ of __ >_________________________________________________

__<Criteria>_______________________________________________________

(___<Item Tag>___)

STEP 2. Develop scoring keys/rubrics using the procedural steps below. See Handout #2

for examples. Refer to Template #2 for more information.

Procedural Steps: MC Items Score Key

1. Enter the assessment information at the top of the Scoring Key.

2. Record the item number, item tag (optional), item type, and point value.

3. Record the MC answers in the Answer column.

4. Repeat Steps 1-4 until all items on the test blueprint are reflected within

the Scoring Key.

5. Validate that each question-to-answer relationship is recorded correctly.

Item # Item Tag Item Type Point Value Answer

1

2

Procedural Steps: SA/EA/EP Scoring Rubrics

1. Review the SA, EA, or EP task and the criteria articulated in the

stem/directions.

2. Select a “generic” rubric structure (see Template #2: Building the

Assessment) based upon (a) scoring criteria and (b) the number of

dimensions being measured.

3. Modify the rubric language using specific criteria expected in the response

to award the maximum number of points.

Assessment Name Grade/Course Administration Total Possible

Points


11

4. Determine how much the response can deviate from “fully correct” in

order to earn the next (lower) point value. [Continue until the full range of

possible scores is described]

5. During the review, ensure that the response expectation, scoring rubric,

and test blueprint are fully aligned.

Frameworks: SA/EA/EP Scoring Rubrics

Short Answer (SA) – Single Dimension

Item # _____ Sample Response for: _________________________________________

2 points

1 point

0 points

Extended Answer (EA) – Single Dimension

Item #_____ Sample Response for: ____________________________________________

4 points

3 points

2 points

1 point

0 points

Extended Performance (EP) Task – Multi-dimensional

Item #_____ Sample Response for: ____________________________________________

Dimension Advanced

(4 points)

Proficient

(3 points)

Basic

(2 points)

Below Basic

(1 points)

STEP 3. Develop administration guidelines using the procedural steps below. See

Handout #2 for examples. Refer to Template #2 for more information.

Procedural Steps: Administration Guidelines

1. Create a series of administrative steps for before, during, and after the

assessment window.


12

2. Explain any special requirements and/or equipment necessary, including

establish testing accommodations. State any ancillary materials (e.g.,

calculators) needed or allowed by the test-takers.

3. Identify the approximate time afforded to complete the assessment,

including each subtask in an EP task.

4. Include detailed “scripts” articulating exactly what is to be communicated

to students, especially when administering performance tasks over a long

period of time.

5. Include procedures for scoring, administering make-ups, and handling

completed assessments.

Framework: Administration Guidelines

Preparation

STEP 1. ______________________________________________________

STEP 2. ______________________________________________________

STEP 3. ______________________________________________________

Administration

STEP 1. ______________________________________________________

STEP 2. ______________________________________________________

STEP 3. ______________________________________________________

After Testing

STEP 1. ______________________________________________________

STEP 2. ______________________________________________________

STEP 3. ______________________________________________________

STEP 4. Organize test forms to include administration guidelines, items/tasks, and scoring

keys/rubrics.

2.6 Quality Reviews




assessment quality.

Strand 2 of the Performance Measure Rubric evaluates the Build phase of the assessment process

(items/tasks, scoring keys/rubrics, test forms). Use the following procedural steps to perform a

quality review of this phase. Refer to Handout #3 – Performance Measure Rubric-Scored

Example for more information.


13

Rating Steps: Performance Measure Rubric Strand 2: Build

STEP 1. Review information, data, and documents associated with the development

of the selected performance measure.

STEP 2. Assign a value in the “Rating” column for each aspect within a particular

strand using the following rating scale:





STEP 3. Reference supporting information associated with each assigned rating in

the “Evidence” column.




STRAND 2: BUILD

Task ID Descriptor Rating Evidence

2.1

Items/tasks and score keys are developed using standardized procedures,

including scoring rubrics for human-scored, open-ended questions (e.g.,

short constructed response, writing prompts, performance tasks, etc.).

2.2

Items/tasks are created and reviewed in terms of: (a) alignment to the

targeted content standards, (b) content accuracy, (c) developmental

appropriateness, (d) cognitive demand, and (e) bias, sensitivity, and fairness.

2.3

Administration guidelines are developed that contain the step-by-step

procedures used to administer the performance measure in a consistent

manner, including scripts to orally communicate directions to students, day

and time constraints, and allowable accommodations/adaptations.

2.4

Scoring guidelines are developed for human-scored items/tasks to promote

score consistency across items/tasks and among different scorers. These

guidelines articulate point values for each item/task used to combine results

into an overall score.

2.5

Summary scores are reported using both raw score points and a performance

level. Performance levels reflect the range of scores possible on the

assessment and use terms or symbols to denote performance levels.

2.6

The total time to administer the performance measure is developmentally

appropriate for the test-taker. Generally, this is 30 minutes or less for young

students and up to 60 minutes per session for older students (high school).


of 6



14

Phase III: Review

3.1 Goal Statement

Understand and apply the techniques used to review and refine measures of student

performance.

3.2 Objectives


o Review developed items/tasks for validity threats, content and cognitive match;

o Examine test alignment to ensure: (a) all items/tasks match the skills, knowledge,

and concepts in the targeted content standards; (b) all rubrics match the targeted

content standards; and, (c) all items/tasks reflect higher order thinking.

Note: A supplement to this document will address alignment (Step 8), data reviews (Step 9), and

refinement (Step 10), which the presenter addressed in the Orientation presentation.


Does each item/task clearly address the standard?

Is the reading difficulty and vocabulary appropriate?

Is the language clear, consistent, and understandable?

Are charts, tables, graphs, and diagrams clear and understandable?

Is there only one (1) correct answer?

Have the items been reviewed for bias and sensitivity?

o Items provide an equal opportunity for all students to demonstrate their

knowledge and skills. The stimulus material (e.g., reading passage, artwork, and

diagram) does not raise bias and/or sensitivity concerns that would interfere with

the performance of a particular group of students.

Are the items developmentally appropriate for test-takers?

Does the blueprint reflect the test form?


15

Does the scoring rubric provide detailed scoring information?

Does the assessment have at least two (2) performance levels?

3.4 Resources

Test blueprint

Draft operational form

Draft scoring sheet and rubric(s)

Template #3- Performance Measure Rubric


STEP 1. Review each item/task for content, bias, fairness, sensitivity, accessibility, and

editorial soundness using Strand 3 of the Performance Measure Rubric.

STEP 2. Review each item/task for alignment with the targeted content standards for

content match and cognitive demand.

STEP 3. Review the test form for alignment in terms of: (a) content patterns of emphasis;

and (b) content range.

STEP 4. Conduct a performance level review of the assessment to identify a preliminary

cut score. The assessment should have at least two (2) performance levels.

3.6 Quality Reviews




assessment quality.

Strand 3 of the Performance Measure Rubric evaluates the Review phase of the assessment

process (validity threat, content and cognitive matches of items/tasks, and test alignment). Use

the following procedural steps to perform a quality review of this phase. Refer to Handout #3 –

Performance Measure Rubric-Scored Example for more information.

Rating Steps: Performance Measure Rubric Strand 3: Review

STEP 1. Review information, data, and documents associated with the development of

the selected performance measure.


16

STEP 2. Assign a value in the “Rating” column for each aspect within a particular

strand using the following rating scale:





STEP 3. Reference supporting information associated with each assigned rating in the

“Evidence” column.




STRAND 3: REVIEW

Task

ID


3.1

The performance measures are reviewed in terms of design fidelity –

Items/tasks are distributed based upon the design properties found

within the specification or blueprint documents.

Item/task and form statistics are used to examine levels of difficulty,

complexity, distracter quality, and other properties.

Items/tasks and forms are rigorous and free of bias, sensitive, or unfair

characteristics.

3.2

The performance measures are reviewed in terms of editorial soundness, while

ensuring consistency and accuracy of other documents (e.g., administration) –

Identifies words, text, reading passages, and/or graphics that require

copyright permission or acknowledgements

Applies Universal Design principles

Ensures linguistic demands and/or readability is developmentally

appropriate

3.3

The performance measures are reviewed in terms of alignment characteristics –

Pattern consistency (within specifications and/or blueprints)

Matching the targeted content standards

Cognitive demand

Developmental appropriateness

3.4

Cut scores are established for each performance level. Performance level

descriptors describe the achievement continuum using content-based

competencies for each assessed content area.

3.5

As part of the assessment cycle, post-administration analyses are conducted to

examine aspects as items/tasks performance, scale functioning, overall score

distribution, rater drift, content alignment, etc.

3.6

The performance measure has score validity evidence that demonstrated item

responses were consistent with content specifications. Data suggest the scores

represent the intended construct by using an adequate sample of items/tasks

within the targeted content standards. Other sources of validity evidence such

as the interrelationship of items/tasks and alignment characteristics of the

performance measure are collected.


17

Task

ID


3.7

Reliability coefficients are reported for the performance measure, which

includes estimating internal consistency. Standard errors are reported for

summary scores. When applicable, other reliability statistics such as

classification accuracy, rater reliabilities, and others are calculated and

reviewed.


of 7


Note: A supplement to this document will address Tasks 3.5, 3.6, and 3.7 of the Performance Measure

Rubric.

Guidance Document Quick Start-FEB14-FINAL

Education

Transcript of Guidance Document Quick Start-FEB14-FINAL