Summative Assessment

Clarissa DirksThe Evergreen State College

Olympia, Washington

Smith et al 2005

PREVIOUS THINKING ABOUT TEACHING AND LEARNING

OUTCOME

OUTCOME

CURRENT THINKING ABOUT TEACHING AND LEARNING

Misconceptions (formative assessments)

Conceptual Frameworks (backward design; scholarly teaching)

Metacognition (formative assessments; testing effect)

AND SOMETHING ELSE . . .

Summative Assessments Drive Learning

THE MONTILLATION AND USES OF TRAXOLINE

It is very important to learn about traxoline. Traxoline is a new form of zionter. It is montilled in Ceristanna. The Ceristannians found that they could gristerlate large amounts of fervon and then bracter it to quasel traxoline. This new, more efficient bracterillation process has the potential to make traxoline one of the most useful products within the molecular family of lukizes snezlaus.

QUIZ:1. What is traxoline?

2. Where is it montilled?

3. How is traxoline quaseled?

4. Why is traxoline important?

What were the learning outcomes related to this content?

MisalignmentsMisalignments

Practice

HOC

LOC or none

HOC

Testing

LOC

HOC

HOC

Outcome for Student

Frustration

Frustration

Excitementand Learning

Lower Order Cognitive Skills = LOCSHigher Order Cognitive Skills = HOCS

Benjamin S. Bloom Taxonomy of educational objectives. Published by Allyn and Bacon, Boston, MA. Copyright (c) 1984 by Pearson Education.

Higher order cognitive skills(HOCS)

Lower order cognitive skills(LOCS)

Bloom’s TaxonomyBloom’s Taxonomy

Cognitive Level

Bloom LevelA Simple Phrase to Guide CategorizationVerbs Typically Associated with the Category

HOC Evaluate“Defend or judge a concept or idea”

Synthesize“Create something new”

Analyze“Distinguish parts and make inferences”

LOC/HOC Apply“Use information or concepts in new ways”

LOC Comprehend“Explain information or concepts”

Know “Recall information”

Knowledge

Comprehension

Application

Analysis Synthesis Evaluation

Knowledge

Comprehension

Application

Analysis Synthesis Evaluation

LOCS

HOCS

Partial Hierarchy of Bloom’s TaxonomyPartial Hierarchy of Bloom’s Taxonomy

Practice Using Bloom’s Taxonomy

Work with a neighbor to categorize your first five test questions as either a LOC or HOC.

Practice Using Bloom’s Taxonomy

Work with a neighbor to categorize your first five test questions as either a LOC or HOC.

What did you have to consider when doing this?

Could you easily categorize someone else’s questions in biology?

What about in another discipline?

Writing Multiple-Choice QuestionsThat Test High Order Cognitive Skills

Warning! These activities may be hazardous to your

colleague’s health as a result of undue stress.

Faculty often get distracted by the content of a question and miss out on the process of what is being presented.

Let’s focus on the process!

Structures of Multiple-Choice Questions (MCQs)Structures of Multiple-Choice Questions (MCQs)

Which of the following explains the many unique properties of water?

a. it has many phases

b. it is a universal solvent

c. its abundance on earth

d. dipole moments

Distractors

Key

Stem Which of the following explains the many unique properties of water?

a. it has many phases

b. it is a universal solvent

c. its abundance on earth

d. dipole moments

Distractors

Key

Stem

Standard MCQ

MCQ Format Advantages Disadvantages

Standard Useful for measuring HOC skills if a strong distractors are used

If not well written the structure can cue a student to the answer

Context Dependent

Useful for measuring HOC skills if strong distractors are used

Not time efficient

Two-Tiered The second tier may survey students’ misconceptions

Not time efficientAnswer to one tier can cue the answer to the other tier.

Complex (K-Type)

Requires analytical skills of comparing and contrasting

Not time efficientTends to be very misleading

Matching Time efficient Typically not useful for measuring HOC skills

True/False Time efficient True/false with less than 5 items have increased errors associated with guessing

Alternate Choice

Useful for measuring HOC skills

Not time efficientLess discriminating than standard format

Take a few minutes and evaluate your exam. What kinds of questions are you asking?

Free response, multiple-choice, or both?

Do you have questions that help you to identify student misconceptions?

Do you have any free response questions that requires a student to explain their reasoning or two-tiered MCQs.

Using Two-tiered questions:A MCQ followed by a free response requiring students to explain why they answered the question the way they did will be very revealing!

Enchanted Valley is located within the Olympic National park of Washington state and is home to many black bears. A researcher is studying the different types of berries that 20 Enchanted Valley black bears eat.

Which of the following passages describes the researcher’s data that would be graphed as a histogram?

A) amount of each type of berry the bears ate in a week

B) amount of salmon berries eaten by each bear in a week

C) amount of huckleberries eaten by male and female bears in a week

In the space provided below, please explain your reasoning for selecting the answer to the above question.

Review the graph right.

0 5 10 15 20 25 30

Temperature (C)

Me

mb

ran

e P

erm

ea

bili

ty t

o U

rea no

cholesterol

25%cholesterol

50% cholesterol

Which of the following passages describes the graph?A) membrane permeability to urea is only dependent on the amount of membrane cholesterol

B) amount of membrane cholesterol is only dependent on membrane permeability to urea

C) membrane permeability to urea is dependent on temperature and the amount of membrane cholesterol

D) amount of membrane cholesterol and membrane permeability to urea are dependent on one another

Which of the following best describes why you selected the answer that you did?

The graph shows . . .

A) two dependent variables and one independent variable

B) one dependent variable and two independent variables

C) one dependent variable and one independent variable

D) two dependent variables and two independent variables

0 5 10 15 20 25 30

Temperature (C)

Me

mb

ran

e P

erm

ea

bili

ty t

o U

rea no

cholesterol

25%cholesterol

50% cholesterol

Q: Why do we care about student misconceptions?

A1: They make us crazy!

A2: We would like to engage in Scholarly Teaching and/or Discipline-Based Education Research

Concept Inventories

Two ways of thinking about this terminology:

1. An outline of the key concepts that a course or curriculum will cover.

2. A MCQ test based of a specific content area that uses common student misconceptions as distractors.

Concept Inventories in Biology

Genetics Concept Assessment (GCA) Smith, et al. 2008.

Genetics Literacy Assessment Instrument 2 (GLAI-2) Bowling et al., 2008a; 2008b

Genetics Concept Inventory Elrod, 2007

Conceptual Inventory of Natural Selection (CINS) Anderson, 2003Anderson et al., 2002

Biology literacy (http://bioliteracy.net/) Klymkowsky and Garvin-Doxas, 2008

Diagnostic Question Clusters: Biology Wilson et al., 2006D’Avanzo, 2008

Host-Pathogen Interactions (HPI) Marbach-Ad et al., 2009

Introductory Molecular and Cell Biology Assessment (IMCA)

Shi et al., 2010

You too can create a concept inventory. If you do, I hope you would publish it!

It would require that the inventory is valid and reliable.

VALIDITY

Validity: Addresses whether the test measures what is suppose to measure.

Content Validity: The test presents the key concepts and misconceptions for a given domain as affirmed by experts.

Construct Validity: The item measures what it is intended to measure. It requires affirmation by experts.

Face Validity: The test has the appearance of measuring what it intends to measure, thereby motivating students to persevere in doing their best.

RELIABILITY

Reliability: Addresses the consistency of a set of measures; it is based on whether the test results are repeatable and not based on random error.

Inter- and Intra-Rater Reliability: The consistency of scoring free-response items by different instructors or the same instructor, respectively. (GREAT FOR RUBRIC DEVELOPMENT!)

Internal Consistency: It measures whether pair-wise items on the same test, measure the same latent variable.

Test-Re-Test Reliability: Each student should give relatively the same result when they take the test at two different times.

Alternate Form Reliability: Each student should give relatively the same result when they take alternate forms of the same test at two different times.

What are we doing for the rest of the day?What are we doing for the rest of the day?

Applying the concepts you learned this morning for creating teaching materials.

Work in groups to do the following in this order:

1. Create a few learning outcomes using Bloom’s TaxonomyTarget misconceptions? HOCS?

2. Write 2 summative assessments that test the learning outcomesHOCS? MCQS?

3. Design 1 formative assessment that gives students practice at working with the material at the same cognitive level at which you are testing them

Help students address their misconceptions, build a framework and develop metacognition.

POST THESE TO THE WIKI PAGE!