CONSTRUCTEDRESPONSE PALOOZA

WINTER 2010

THINK, PAIR, SHARE

What are some of the challenges you face when grading constructed responsequestions?

OBJECTIVES

• Provide teachers with a firm understanding of the different types of rubrics and how to use them with different student writing samples.

• Provide an opportunity to examine and discuss an anchor set for a given writing prompt.

• Provide teachers with the opportunity to examine student samples from an administered constructed response item and norm along the ANet rubric.

OVERARCHING GOAL: To leave with a better understanding of using rubrics to effectively score constructed responses and to take this back to your schools to improve your school’s overall use of constructed response data.

AGENDA

I. Rubrics 101 15 min.

• Different types of rubrics• Different types of questions/prompts• Scoring

II. Norming around an Anchor Set 45 min.

• Review of passage, item, and rubric• Model of Scoring• Paired Scoring and Debrief

II. Practice Norming around ANet Responses 1 hour

• Groups use rubrics to score student responses• Debrief

IV. Next Steps 10 min.

RUBRICS 101HOLISTIC V. ANALYTIC

Holistic Rubrics

- provide a single score based on an overall impression of a student’s performance on a task.

DC Uses: to score constructed response items aimed at assessing reading comprehension

Advantages: quick scoring, provides overview of student achievement

Disadvantages: does not provide detailed information, may be difficult to provide one overall score

Analytic Rubrics

- provide specific feedback along several dimensions.

DC Uses : to score writing compositions aimed at assessing writing

Advantages: more detailed feedback, scoring more consistent across students and graders

Disadvantages: time consuming to score

ELA COMPOSITION VS. CONSTRUCTED RESPONSE QUESTIONS

Composition• Assess writing

• A starting point to initiate a student’s own thoughts on the topic, supported by their experiences and ideas

• No “correct” answer (but must address topic specified by the prompt)

• Style and organization are important to the score

• Uses Analytic Rubric and scored in two domains: Topic Development and Conventions

• Scored out of 10 points in grades 4,7 and 10

Constructed response• Assess reading comprehension

• Student must answer the question asked and answer must be supported with details from the text

• Answers found in reading the passage or can be inferred from information in the passage

• The explanation and support from the text matter much more than style and organization

• Uses Holistic Rubric, conventions are not scored

• Worth 9 total points (3 CRs, each worth 3 pts.)

SAMPLE COMPOSITION WRITING PROMPT

Think about a time you were helpful. For example, maybe you helped a new student feel comfortable in school, helped a teacher with a job in the classroom, helped with a project in your neighborhood or school, or helped to care for an animal.

Write a story about a time you were helpful. Give enough details for readers to understand how you were helpful.

Example is from the 2009 4th grade ELA MCAS

ANALYTIC RUBRIC

Score Description

6 •Rich topic/idea development•Careful and/or subtle organization•Effective/rich use of language

5 •Full topic/idea development•Logical organization•Strong details•Appropriate use of language

4 •Moderate topic/idea development and organization•Adequate, relevant details•Some variety in language

3 •Rudimentary topic/idea development and/or organization•Basic supporting details•Simplistic language

2 •Limited or weak topic/idea development, organization, and/or details•Limited awareness of audience and/or task

1 •Limited topic/idea development, organization, and/or details•Little or no awareness of audience and/or task

Topic/Idea Development

ANALYTIC RUBRIC

Standard English Conventions

Score Description

4 •Control of sentence structure, grammar and usage, and mechanics (length and complexity of essay provide opportunity for student to show control of standard English conventions)

3 •Errors do not interfere with communication and/or•Few errors relative to length of essay or complexity of sentence structure, grammar and usage, and mechanics

2 •Errors interfere somewhat with communication and/or•Too many errors relative to the length of the essay or complexity of sentence structure, grammar and usage, and mechanics

1 •Errors seriously interfere with communication AND•Little control of sentence structure, grammar and usage, and mechanics

SAMPLE CONSTRUCTED RESPONSE QUESTION

Based on the article, describe what peoplehave learned from studying slave quilts.

Support your answer with importantinformation from the article.

Example is question 35 from the 2008 6th grade ELA MCASPassage “Pattern for Freedom: Women’s Quilt as Art”

HOLISTIC RUBRIC

Score Description

3 The response demonstrates a complete understanding of what people have learned from slave quilts. The response includes support that• is clear and complete• provides relevant and specific details/information from the text

2 The response demonstrates a partial understanding of what people have learned from slave quilts. The response includes support that• is partially clear and/or partially complete• provides mostly relevant but somewhat general and/or inaccurate details/information from the text

1 The response demonstrates a minimal understanding of what people have learned from slave quilts. The response includes support that• is minimally correct or incomplete• provides inadequate, incorrect, or no relevant details/information from the text

0 The response demonstrates no understanding of what people have learned from slave quilts. Any details/information that is included isincorrect or irrelevant.

SCORING BIAS IN CONSTRUCTED RESPONSE ITEMS

It’s important for validity not to let writing skills

impinge on reading comprehension scores.

Things to look past (to avoid scorer bias)

• Penmanship• Neatness• Mechanical errors• Spelling• Length

PROCESS FOR SCORING STUDENT RESPONSES TO CONSTRUCTED RESPONSE PROMPTS

1. Read the passage and the accompanying prompt.

2. Review the rubric for the prompt and highlight key words.

3. Create a list of scoring notes.

4. Begin to build an anchor set.

5. Score remaining student responses using rubric and anchor set.

READ PASSAGE AND ACCOMPANYING PROMPT

• Read the passage “Pattern for Freedom: Women’s Quilt as Art”

• Read the prompt:

Based on the article, describe what peoplehave learned from studying slave quilts.

Support your answer with importantinformation from the article.

REVIEW THE RUBRIC

Score Description

3 The response demonstrates a complete understanding of what people have learned from slave quilts. The response includes support that• is clear and complete• provides relevant and specific details/information from the text

2 The response demonstrates a partial understanding of what people have learned from slave quilts. The response includes support that• is partially clear and/or partially complete• provides mostly relevant but somewhat general and/or inaccurate details/information from the text

1 The response demonstrates a minimal understanding of what people have learned from slave quilts. The response includes support that• is minimally correct or incomplete• provides inadequate, incorrect, or no relevant details/information from the text

0 The response demonstrates no understanding of what people have learned from slave quilts. Any details/information that is included isincorrect or irrelevant.

HIGHLIGHT KEY WORDS IN THE RUBRIC

Score Description

3 The response demonstrates a complete understanding of what people have learned from slave quilts. The response includes support that• is clear and complete• provides relevant and specific details/information from the text

2 The response demonstrates a partial understanding of what people have learned from slave quilts. The response includes support that• is partially clear and/or partially complete• provides mostly relevant but somewhat general and/or inaccurate details/information from the text

1 The response demonstrates a minimal understanding of what people have learned from slave quilts. The response includes support that• is minimally correct or incomplete• provides inadequate, incorrect, or no relevant details/information from the text

0 The response demonstrates no understanding of what people have learned from slave quilts. Any details/information that is included isincorrect or irrelevant.

CREATE SCORING NOTES

What people have learned from slave quilts:• Quilts were forms of self-expression: used to pass on emotion,

histories and religious beliefs• Quilts were a way to convey messages in African culture:

• Historical symbols: such as X for a crossroads, snake motif for the West African God of fertility, flower patterns for the Haitian goddess of love, etc.

• Quilts were a way for slaves to leave a record of their lives (personal histories):

• The use of color: for example red for a woman’s birth process or a man’s role as hunter and warrior, etc.

• Quilts were a way to guide runaway slaves (use in the Underground Railroad):

• Black fabric symbolized a safe house, the “Drunkard’s Path” told slaves to take an indirect route, etc.

CREATE AN ANCHOR SET

What is an Anchor set?

•An anchor paper is a sample response that illustrates student work at a given score point on a rubric.

•An anchor set will include examples of student responses at each score point at the rubric.

•These student samples help illustrate what is provided in the rubric.

STUDENT RESPONSE #1

Score: 0

Why a 0?

The response is incorrect.

Why not a 1?

The information contained in the response is incorrect and not supported by the article.

STUDENT RESPONSE #2

Score: 1

Why a 1?

There is minimal attempt to explain that slave quilts contain meaningful designs. There are no supporting details from the article included.

Why not a 0?

The information provided was not incorrect. The response, while lacking details from the passage, does show understanding of quilts helping slaves escape from slavery and that designs had different meanings.

STUDENT RESPONSE #3

Score: 2

Why a 2?

The response is a general description of what people have learned from studying slave quilts. The focus of the discussion is on the symbolism of the colors and patterns used in slave quilts.

Why not a 3?

The response is partially complete in that it lacks any reference to the use of quilts for passing down history. Details from the story are general.

STUDENT RESPONSE #4

Score: 3

Why a 3?

The response clearly, completely, and accurately explains that the study of slave quilts has taught people about the personal history of slaves, aspects of African culture, and symbols used in the Underground railroad. Relevant quotes from the article are included to support each idea. The introduction and conclusion do not contribute any additional information to this response.

Why not a 2?

The essay is clear and complete. Specific details, including quotes from the article, are used in the response. No inaccurate details are included.

THINK, PAIR, SCORE

Read student samples #5 and #6.

Score each sample using the rubric.

Justify your score.

Share your thoughts with your neighbors.

STUDENT RESPONSES #5 AND #6

Student Response #5

Score: 3

Why a 3?The response is a clear, complete and accurate description of what people have learned from studying slave quilts. Details are included to explain the meaning of several historical symbols as well as quilt patterns designed to guide runaway slaves. The symbolic meaning of different colors are also discussed.

Student Response #6Score: 2

Why a 2?This response provides a partial explanation of what people have learned from studying slave quilts. Information is provided about the symbolism of the color black and the significance of certain patterns, but overall details are limited.

SCORING AN ANET PROMPT

1. Read the passage and the accompanying prompt.

2. Review the rubric for the prompt and highlight key words.

3. Create a list of scoring notes.

4. Begin to build an anchor set.

5. Score remaining student responses using rubric and anchor set.

ANET WRITING PROMPT

Read the passage:Take a moment to read the passage from Lost in the Grand Canyon

by Deborah Hopkinson.

Read the promptAccording to the article, Powell was “a born leader with

courage andambition.” Based on the article do you agree or disagree

with thisstatement? Why? Use evidence from the article to support

youranswer.

REVIEW THE RUBRIC

Score Description of Student Response

3 The response demonstrates a complete description of whether or not Powell was a born leader. The response includes support that• is clear and complete• provides relevant and specific details/information from the text

2 The response demonstrates a partial description of whether or not Powell was a born leader. The response includes support that• is partially clear and/or partially complete• provides mostly relevant but somewhat general and/or inaccurate details/information from the text

1 The response demonstrates a minimal description of whether or not Powell was a born leader. The response includes support that• is minimally correct or incomplete• provides inadequate, incorrect, or no relevant details/information from the text

0 The response demonstrates no description of whether or not Powell was a born leader. Any details/information that is included is incorrect or irrelevant.

HIGHLIGHT KEY WORDS IN THE RUBRIC

Score Description of Student Response

3 The response demonstrates a complete description of whether or not Powell was a born leader. The response includes support that• is clear and complete• provides relevant and specific details/information from the text

2 The response demonstrates a partial description of whether or not Powell was a born leader. The response includes support that• is partially clear and/or partially complete• provides mostly relevant but somewhat general and/or inaccurate details/information from the text

1 The response demonstrates a minimal description of whether or not Powell was a born leader. The response includes support that• is minimally correct or incomplete• provides inadequate, incorrect, or no relevant details/information from the text

0 The response demonstrates no description of whether or not Powell was a born leader. Any details/information that is included is incorrect or irrelevant.

SCORING NOTES (THINGS TO LOOK FOR)

Powell is a born leader

1. Willing to search last unknown area (Grand Canyon).

2. Fought in the Civil War- lost arm, unable to row, but still leads journey.

3. Shouts orders and warnings when river turns wild.

4. Races downstream searching for the boat.

5. Comes up with plan to ‘run down a chute’.

6. Shoots off rifles and waits for the men who left- before continuing on the journey.

7. Powell and crew make it safely down the Colorado River.

Powell is not a born leader

8. Places blame on others when things go wrong.

9. Orders crew members around.

10.Powell is unprepared- doesn’t know how long trip will take and there isn’t enough food.

11. Tries to persuade crew members to stay, but is unsuccessful.

MODEL SCORING- RESPONSE A

Please refer to Response A

Score: 0

Why a 0? All details/information provided in response are incorrect.

Why not a 1?Student does not include any relevant and correct details in response to support a position.

MODEL SCORING- RESPONSE B

Please refer to Response B

Score 2:

Why a 2?Student provides a partial explanation of why Powell was a born leader. Details are included about Powell attempting to save men after the boat went over the waterfall and that he did not allow his men to starve. The overall details are limited and the response is not complete.

Why not a 3? The response includes some relevant details from the passage to support the position, however response is not complete.

BEGIN TO BUILD AN ANCHOR SET

You will be receiving a set of papers for thisprompt.

Unfortunately, each scoring point on the rubric does not have a student sample.

Independently and then in pairs take some time toreview each of the papers in light of the rubricand provide a score for each one.

Please make notes for each example as to why youbelieve it deserves that score.

ANCHOR SET

Response C

Score: 2Most examples contained in response were relevant and supported the position of Powell being a born leader. Response was partially complete.

Response D

Score: 1Response is a minimal description of Powell being a born leader. Supporting details from article are limited and there is little attempt to explain how or why Powell should be viewed as a born leader.

Response E

Score: 1This response includes a minimal description of Powell being a born leader. Some details are incorrect or irrelevant.

ANCHOR SET

Response F

Score: 0Response only restates the prompt. No information from passage is included. Response demonstrates no description of whether Powell is a born leader.

Response G

Score: 2Response includes multiple relevant details to support Powell being a born leader. Specific details were included to support thoughts presented. Response was not complete.

Response H

Score: 2This response includes relevant details. Final section seems to include inaccuracies and is incomplete.

TAKEAWAYS

BEST PRACTICES

USEFUL RESOURCE FOR CONSTRUCTED RESPONSE

• Find sample prompts and student responses on the Massachusetts Dept. of Elem. and Secondary Ed. website: Link to MA DOE Released CRs

MA Score DC Score4 3

3 - 2 21 10 0

NEXT STEPS

• Feel free to exchange contact information with the individuals in your scoring group to keep in contact about constructed response items and scoring.

• Please take the anchor set, passages, rubrics and student responses with you.

• Please contact us with any questions you might have.

• Please bring this back to your school so that over time you can collect increasingly valid constructed response data.

THANK YOU SO MUCH FOR COMING.

WE WANT YOUR FEEDBACK!