The LCSD APPR StoryThe Local Assessment ChapterDecember 13, 2012

The Third Piece 15-20pts

The straw that broke. . . The local assessment decision

more NYS assessmentan approved 3rd party assessmentregion or district developed

Our own assessmentswe decide what learning is importantdiversify measures of student learning

Measures student growth for SLOs too

Two Parts

I - Best practices in assessment development

II - Superintendent Approval Process Assurances Assessment Maps/Blueprints/Specs

Scoring Guides Test Specifications Testing Guides

Creating Summative Assessments

Cheryl Covell

Tompkins-Seneca-Tioga BOCES Model Schools Program

Accountability Based Assessment Instructionally Supportive Assessment

• Sorts, sanctions, rewards • Informs curriculum and instruction

• Public • Private

• Assesses broad standards • Assess particular skills and content

• Reliability most important • Validity most important

• Evaluate if students are “meeting standard” • Evaluate students at all levels

• Summative • Formative

• Infrequent • Frequent

What kind of test?

Summative Standardized Coverage of the “most important” standards Selected response and constructed response

Summative Assessment

Summarizes learning at a particular point in time.

Can be used to compare how well students mastered a set of learning goals.

Standards-based Assessment

Student performance is compared to a desired level of performance on the content.

Questions on the assessment are selected to cover important content.

Questions on the assessment are selected to match national, state or local content standards.

APPR Evidence – Considerations

9

• How aligned and authentic are the assessment items to the learning content?

• How valid and reliable are the assessments?

• Are they verified as comparable and rigorous?

• What, if any, administration accommodations must legally be made for students?

• How are the assessments scored in terms of point values assigned per item and method of summarizing scores?

• Have procedures been established to ensure those with vested interest do not score students’ assessments?

Selected Response

Any item which asks students to select from several options.

Multiple choiceTrue/falseMatchingFill in the blank, with word bank

Multiple Choice – Stems

Make the question as clear and direct as possible. Make sure students know what you are asking by

just reading the question – a proficient student should be able to come up with the right answer without first reading all of the possible choices.

Use negatives sparingly and emphasize them if used. Which of the following choices is NOT a mammal?

Ask more than one question when students must first take in a lot of information.

Multiple Choice – Alternatives

Make sure there is only one unambiguously right answer.

Create plausible distracters. Common misconceptions Common mistakes Technical jargon

Do not use emphasis (bold, capital letters). Avoid repeating words from the stem. Use “none of the above” with caution. List choices in a logical order.

All of the following are correct procedures for putting out a fire in a pan on the stove except:

a. Do not move the pan.*b. Pour water into the pan.c. Slide a fitted lid onto the pan.d. Turn off the burner controls.

All of the following are correct procedures for putting out a fire in a pan on the stove except:

a. Do not move the pan.*b. Pour water into the pan.c. Slide a fitted lid onto the pan.d. Turn off the burner controls.

California:

a. Contains the tallest mountain in the United Statesb. Has an eagle on its state flag.c. Is the second largest state in terms of area.*d. Was the location of the Gold Rush of 1849.

What is the main reason so many people moved to California in 1849?

a. California land was fertile, plentiful, and inexpensive.*b. Gold was discovered in central Californiac. The east was preparing for a civil war.d. They wanted to establish religious settlements.

When conducting library research in education, which of the following is the best source to use for identifying pertinent journal articles?

a. A Guide to Sources of Educational Information.*b. Current Index to Journals in Education.c. Resources in Educationd. The International Encyclopedia of Education.

When conducting library research in education, which of the following is the best source to use for identifying pertinent journal articles?

a. A Guide to Sources of Educational Information.*b. Education Index.c. Resources in Education.d. The International Encyclopedia of Education.

Suppose you are a mathematics professor who wants to determine whether or not your teaching of the unit on probability has had a significant effect on your students. You decide to analyze their scores from a test they took before the instruction and their scores from another exam taken after the instruction. Which of the following t-tests is appropriate to use in this situation?

*a. Dependent samples.b. Heterogeneous samples.c. Homogeneous samples.d. Independent samples.

When analyzing your students’ pretest and posttest scores to determine if your teaching has had a significant effect, an appropriate statistic to use is the t-test for:

*a. Dependent samples.b. Heterogeneous samples.c. Homogeneous samples.d. Independent samples.

How long does an annual plant generally live?

a. It dies after the first year.b. It lives for many years.c. It lives for more than one year.d. It needs to be replanted each year.

How long does an annual plant generally live?

a. Only one year.b. Only two years.c. Several years.

Constructed Response

A constructed response requires students to use creativity, organization skills, and logic to develop an answer.

EssayShort answerDiagramPerformanceFill in the blank – no word bank

Four Elements

1. Task What students should do.

2. Evaluative Criteria The essential ingredients of the skill being

measured.

3. Quality Definitions The description of how a student’s performance on

each evaluative criteria would look at each scoring level.

4. Scoring Strategy How a final score will be derived for the task.

Smarter Balanced Assessments

Analyze/Integrate Information2 The response gives sufficient evidence of the

ability to gather, analyze and integrate information within and among multiple sources of information.

1 The response gives limited evidence of the ability to gather, analyze and integrate information within and among multiple sources of information.

0 A response gets no credit if it provides no evidence of the ability to gather, analyze and integrate information within and among multiple sources of information.

Smarter Balanced Consortium

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/05/TaskItemSpecifications/EnglishLanguageArtsLiteracy/ELARubrics.pdf

Rubistar

http://rubistar.4teachers.org/

Lansing Central School DistrictCriteria and Assurances for Superintendent Approval Of Local End-of-Course Assessments Teacher(s) ________________________________ Course Name __________________________________ Assurances of Validity – Please check all boxes. 1. I have reviewed this assessment to determine that each item measures the learning target that I intend to measure. 2. I have reviewed this assessment to determine that, when taken together, these items assess curriculum that I have prioritized as important to students’ adult

lives, important to students’ future learning, important to students’ learning in other content areas, or relevant to NYS Assessments. Assurances of Reliability – Please check all boxes. 3. I have reviewed this assessment, and I expect that this collection of assessment items will reliably assess student learning for the affiliated course. I.e., student

scores on this assessment should correctly classify student learning (e.g., as developing, proficient, mastered, advanced). 4. I will immediately report any indications that student scores on this assessment do not reliably describe levels of student learning, if such irregular results occur. 5. This assessment includes scoring materials (e.g., answer key, rubric(s), points distribution for short answers, etc.) that support consistent scoring of all students’

assessments. Assurances of Bias Review – Please check all boxes. 6. I have reviewed this assessment for assessment bias and have taken all measures available to me to minimize bias in every item. Assurances of Rigor and Comparability – Please check box 7 and either 8 or 9. 7. I have reviewed this assessment and assure that it is aligned with the appropriate NYS Learning Standards, including grade-level aligned CCLS Literacy Standards

for Social Studies or or for Technical Subjects. 8. I have reviewed this assessment for rigor, using the levels in Bloom’s Taxonomy, and assure that these items address a range of thinking skills, but most items

require students to work above Bloom’s Comprehension level. OR 9. I assure that the assessment requires students to deeply comprehend complex grade level text (as defined in the CCLS) and to demonstrate that comprehension

through informational or argumentative writing using text-based references. (Grades K-2 tasks may require narrative writing.) The signature(s) below indicate that the end-of-course assessment for the course listed above has been reviewed and revised for validity, reliability, bias, and rigor. Teacher Signature(s) ________________________________________ Date _____________________

________________________________________ Date _____________________ The signature below indicates Superintendent approval for the end-of-course assessment listed above. Superintendent (or designee) Signature ______________________________________ Date _____________________

Validity and Reliability

Validity = Accurate InferencesReliability = Consistent Results

A Valid Assessment….

Assurances of Validity 1. I have reviewed this assessment to determine

that each item measures the learning target that I intend to measure.

2. I have reviewed this assessment to determine that, when taken together, these items assess curriculum that I have prioritized as important to students’ adult lives, important to students’ future learning, important to students’ learning in other content areas, or relevant to NYS Assessments.

Standards

Statements of what we want students to know and be able to do.

Learning Targets

The specific knowledge and skills a student must have in order to master a given standard.

A Reliable Assessment….

Assurances of Reliability

3.  I have reviewed this assessment, and I expect that this collection of assessment items will reliably assess student learning for the affiliated course. I.e., student scores on this assessment should correctly classify student learning (e.g., as developing, proficient, mastered, advanced).

4. I will immediately report any indications that student scores on this assessment do not reliably describe levels of student learning, if such irregular results occur.

5. This assessment includes scoring materials (e.g., answer key, rubric(s), points distribution for short answers, etc.) that support consistent scoring of all students’ assessments.

For objective assessments like multiple choice tests, to increase reliability, you should:

Include enough items. Allow enough time for students to complete

the test. Standardize administration.

For papers, essays, and projects, to increase reliability, you should:

Have clear enough directions for students that all are likely to produce work you can score.

Have a systematic scoring procedure. Have multiple markers (scorers) when

possible. Use broad categories for classification. Standardize administration.

Assurances of Bias Review

6. I have reviewed this assessment for assessment bias and have taken all measures available to me to minimize bias in every item.

Bias

Anything in the assessment or way the assessment is carried out that might not allow students to be able to adequately demonstrate what they know and can do.

offensive language or topics assumed background knowledge requiring skills outside of the area being assessed noise – typos, poor layout, confusing directions

Topics to Avoid abortion abuse of people or animals contraception deportation of immigrants experimentation on people

or animals that is dangerous or painful

killing of animals for sport the occult, witches,

ghosts, vampires pregnancy of human

beings rape

sexual behavior or sexual innuendo

suicide torture euthanasia gun control climate change caused by

human behavior prayer in school current or recent partisan

political issues, ethnic conflicts, and religious disputes

Topics to be Treated with Care Accidents and natural disasters Advocacy Alcohol, tobacco, and illegal

drugs Animals that are frightening to

children Antisocial, criminal, or

inappropriate behaviors Biographical materials Dancing Dangerous activities Death and dying Evolution Family problems Gambling

Holidays and birthdays Homelessness and evictions Immigration Junk food Luxuries Medicines, including diet

supplements Obesity and body-image

problems Personal questions Religion Serious illnesses Slavery Terrorism, wars, violence,

suffering

Looking for bias…. Shaquan helps assemble food packages for

poor people at Christmas. Each box holds 6 cans in a row. There is room for 4 rows in a box. Write the expression that best describes the number of cans in one full box.

Unacceptable. The first sentence adds to the reading load but adds no useful information. The references to “Christmas” and “poor people” are inappropriate and unnecessary.

Looking for bias…. Lee’s father and Juan’s father are both

policemen. Unacceptable. Even though both officers are

male, “police officers” is preferred to “policemen” to avoid the impression that only men are police officers.

Looking for bias…. If one card is taken at random from a deck of

playing cards, what is the probability that the card will be an ace?

Unacceptable. The question assumes knowledge of the number of aces and the total number of cards in a deck of playing cards. It is acceptable to ask about probability, and it is acceptable to use playing cards in math problems. According to the guideline about gambling, however, it is not acceptable to assume that test takers have knowledge of the characteristics of a deck of playing cards.

Looking for bias…. When Ms. Luna pulled her car into the parking

garage, the machine at the gate issued a ticket stamped with the time, 11:30 a.m. When she left the garage that afternoon, her ticket was stamped with the time she left, 12:15 p.m. What was the total length of time that Ms. Luna’s car was in the parking garage?

Unacceptable. The question is very wordy and uses an unfamiliar context for many children. In addition, “pulled her car” is an idiom that children may not know.

Looking for bias…. People who drive gas-guzzling SUVs

contribute to global warming. Unacceptable. This excerpt is a clear violation

of the guideline against advocating for one side in a controversial situation.

Assurances of Rigor and Comparability7. I have reviewed this assessment and assure that it is

aligned with the appropriate NYS Learning Standards, including grade-level aligned CCLS Literacy Standards for Social Studies or for Technical Subjects.

8. I have reviewed this assessment for rigor, using the levels in Bloom’s Taxonomy, and assure that these items address a range of thinking skills, but most items require students to work above Bloom’s Comprehension level. ****OR****

9. I assure that the assessment requires students to deeply comprehend complex grade level text (as defined in the CCLS) and to demonstrate that comprehension through informational or argumentative writing using text-based references. (Grades K-2 tasks may require narrative writing.)

Rigor…

Rigor Definition – APPR Field Guidance

Rigorous means that the locally-selected measure is aligned to the New York State learning standards or, in instances where there are no such learning standards that apply to a subject/grade level, evidence of alignment to research-based learning standards.

Common Core Learning Standard

7.RP - Ratios & Proportional Relationships

“Use proportional relationships to solve multistep ratio and percent problems.”

1. 50% of 20:2. 67% of 81:3. Shawn got 7 correct answers out of 10 possible answers on his science test. What

percent of questions did he get correct?4. J.J. Redick was on pace to set an NCAA record in career free throw percentage.

Leading into the NCAA tournament in 2004, he made 97 of 104 free throw attempts. What percentage of free throws did he make?

5. J.J. Redick was on pace to set an NCAA record in career free throw percentage. Leading into the NCAA tournament in 2004, he made 97 of 104 free throw attempts. In the first tournament game, Redick missed his first five free throws. How far did his percentage drop from before the tournament game to right after missing those free throws?

6. J.J. Redick and Chris Paul were competing for the best free-throw shooting percentage. Redick made 94% of his first 103 shots, while Paul made 47 out of 51 shots. – Which one had a better shooting percentage?– In the next game, Redick made only 2 of 10 shots while Paul made 7 of 10 shots.

What are their new overall shooting percentages? Who is the better shooter?– Jason argued that if Paul and J.J. each made the next ten shots, their shooting

percentages would go up the same amount. Is this true? Why or why not?

Common Core Learning Standard

Reading Standards for Informational TextGrade 3

“Determine the main idea of a text; recount the key details and explain how they support the main idea.”

LITTLE RED RIDING HOOD:  1. What is the main idea? 2. This story is mostly about: 

A. Two boys fighting B. A girl playing in the woods C. Little Red Riding Hood’s adventures with a wolf D. A wolf in the forest

  3. This story is mostly about: 

A. Little Red Riding Hood’s journey through the woods B. The pain of losing your grandmother C. Everything is not always what it seems D. Fear of wolves

In an open-ended question, the rubric defines the rigor.

In a multiple choice question, the options define the rigor.

Test Map

A summary of the content and method of assessment.

What are the prioritized standards? What type of items are included and how

many of each? How will the items be weighted?

Lansing Test Map

Category of Assessed Student Learning Percent of Total Credit Assigned to Area

Question Type: (e.g., 2 credit multiple choice, 4 credit short response, 20 credit essay, etc.)

Number of Questions

Total Credits

Some Details

Assessment Item Security Proctoring/Administrating Exams Assessment Maps/Blueprints/Specs

NYS Scoring Guides NYS Test Specifications NYS Testing Guides

First Assessment due date – January 31, 2013 All students in a course, e.g., 4th Grade ELA,

must take the same assessment.

The End

Next steps = begin work in grade level/department teams tomorrow

Ask lots of questions