Designing, Implementing, and Assessing Student-Student Collaboration
Assessing the development of student ideas over time
description
Transcript of Assessing the development of student ideas over time
Assessing the development of student ideas over time
Karen DraneyUniversity of California, Berkeley
What is meant by student growth?
• Change in average score over time?– Groups of students– Pretest/posttest model– T-test or ANOVA
• Maintaining steady level each year in standardized testing?– Normative definition– Staying at the same percentile level
What is meant by student growth, continued
• Meeting some defined set of criteria?– Reaching a set percentage (e.g. 80% correct) on
series of tests• Piagetian-style developmental levels?
– Strong theoretical background– Certain kinds of performances on specified sets
of tasks• Elimination of misconceptions?
– E.g. Minstrell’s work
Our approach
• Wilson’s BEAR Assessment System• 4 “building blocks”
– Progress variables – Items design– Outcome space– Measurement model
Connection to NRC Assessment Triangle
Observations
Cognition
NRCAssessment
Triangle
CognitionA model of student cognition and learning in the domain.
Observation
A set of beliefs aboutthe kinds of
observations that willprovide evidence of
studentsÕcompetencies.
Interpretation
An interpretationprocess for making
sense of theevidence.
Progress map
• The heart of the system• Big ideas, central concepts, organizing
principles…• It is critical to pick the right ones -- but not
too many (or teachers will not be able to use)
• Developmental in nature
“Developmental” can mean…Following a (perhaps) necessarily ordered development of more complex ideas:
Progress map for speed
Or, displaying increasing amounts of complexity:
5. Generation: Students use the models to generate new knowledge and to extend models. (~graduate school)
4. Construction: Students integrate scientific understanding into full working models of the domain. (~upper division)
3. Formulation: Students combine unirelational ideas, building more complex knowledge structures in the domain. (~lower division)
2. Recognition: Students begin to recognize normative scientific ideas, attaching meaning to unirelational concepts. (~high school)
1. Notions: Students bring real-world ideas, observation, logic and reasoning to explore scientific problem-solving. (~middle-school)
Or even, the order in which a curriculum is arranged:
Levels of Understanding
Buoyancy depends on thedensity of the object relative
to the density of the medium.
Lessons
12: Relative Density
Assessment Activities
Reflective Lesson @10
Reflective Lesson @7
Reflective Lesson @6
Post test
Buoyancy depends on thedensity of the object.
11: Density of Medium
10: Density of Object
Buoyancy depends on themass and volume of the object.
7: Mass and Volume
Reflective Lesson @11
Buoyancy depends on thevolume of the object.
Buoyancy depends on themass of the object.
6: Volume
Reflective Lesson @4
Pretest
4: Mass
1: Introduction
Why things sink and float:
Items design
• Each item directly linked to one or more of the progress variables
• Each item displays a clearly defined relationship to two or more levels of a progress variables
• The levels of a progress variable remain consistent across the set of items (and their responses)
• Relationship between items and progress variables is empirically tested
Outcome space
• Specifies the relationship between every class of responses to given items and the levels of the progress variables
• For every item, these must be ordered, finite, and exhaustive
• This takes work. Lots of work. Looking at student responses, and ideally, talking with students, is critical here
ChemQuery
Examples of items from our instrument:
Both of the solutions have the same molecular formulas, but butyric acid smells bad and putrid while ethyl acetate smells good and sweet. Explain why these two solutions smell differently.
C4H8O4
butyric acidC4H8O4
ethyl acetate
Level 1:Notions
Response 1: I think there could be a lot of different reasons as to why the two solutions smell differently. One could be that they're different ages, and one has gone bad or is older which changed the smell.
Response 2: Using chemistry theories, I don't have the faintest idea, but using common knowledge I will say that the producers of the ethyl products add smell to them so that you can tell them apart.
Level 2:Recognition
Response: "They smell differently b/c even though they have the same molecular formula, they have different structural formulas with different arrangements and patterns.”
Does the mass of the solution change after the two solutions are mixed and a solid forms?
Levels Score Student Responses
Notions1 “…it is solid so it will have more
mass.”
1+ “The mass may change …because there may be a chemical reaction…”
Recognition2- “No, you still have the same
amount.”
2 “No, it doesn’t change because the same amount of molecules...”
Please answer the following question. Write as much information as you need to explain your answer. Use evidence, examples and what you have learned to support your explanations.
Why do things sink and float?
Level What the Student Knows Example Responses (WTSFessay)
RD
Relative DensityStudent knows that floatingdepends on having less density thanthe medium.
ŅAn object floats when its density isless than the density of the medium.Ó
DDensityStudent knows that floatingdepends on having a small density.
“An object floats when its density issmall.”
MV
Mass and VolumeStudent knows that floatingdepends on having a small massand a large volume.
“An object floats when its mass issmall and its volume is large.”
M V
MassStudent knowsthat floatingdepends onhaving a smallmass.
Vo lumeStudent knowsthat floatingdepends onhaving a largevolume.
“An object floats when its mass issmall.”
“An object floats when its volume islarge.”
PM
Productive MisconceptionStudent thinks that floatingdepends on having a small size,heft, or amount, or that it dependson being made out of a particularmaterial.
“An object floats when it is small.”
“An object floats when it is a lightmaterial.”
UF
Unconventional FeatureStudent thinks that floatingdepends on being flat, hollow, filledwith air, or having holes.
“An object floats when it is hollow.”
“An object floats when it is flat.”
OT
Off TargetStudent does not attend to anyproperty or feature to explainfloating.
“I have no idea.”
The measurement model• Provides statistical evidence of assessment quality
(reliability, validity)• Provides assessment of student growth, even across
non-identical groups of items• Provides empirical evidence on the fit of the items &
outcome space to theory of progress variable• Provides evidence of the performance of particular
items and persons (including fit)• Provides the ability to develop a variety of graphical
summaries of student development• The key to making sure the system is working; drives
our process of revision
Dotted line indicates expected performance; blue line is actual performance
Gains from Pre- to Post-test
The process
Construct Map
Items Design
Outcome Space
Measurement Model
LBC initial constructs
5 Integration across variables
4 Bonding (ionic, covalent molecules, perhaps collections of molecules
3 Model of the atom (including elements and periodicity
2 Particulate view (definition of matter as particulate)
1 Macroscopic observationssimple
complex
build
ing
ma t
ter
Second round
5 Integratethe domain
4 PredictScientific models
3 RelatePatterns andEquations
2 RepresentScientificdefinitions
1 DescribeObservations& experience
0A
CharacterizingMatter
Matter is composed of
atoms arranged in various ways
BMeasuring
MatterMass is
used to accountfor matter
CCharacterizing
ChangeChange is
associated withrearrangements
of atoms
DQuantifying
ChangeMass is
used to keep track of change
EEvaluatingEnergies
Energy transfer used to analyze
tendency for change
FQuantizing
EnergyInteraction of
light with matter elucidates structure
Quantitative vs Qualitative
5 Integratethe domain
4PredictScientific models
3RelatePatterns andEquations
2RepresentScientificdefinitions
1DescribeObservations& experience
0
ACharacterizing
MatterMatter is
composed of atoms arranged in various ways
BMeasuring
MatterMass is
used to accountfor matter
CCharacterizing
ChangeChange is
associated withrearrangements
of atoms
DQuantifying
ChangeMass is
used to keep track of change
EEvaluatingEnergies
Energy transfer used to analyze
tendency for change
FQuantizing
EnergyInteraction of
light with matter elucidates structure
bonding &reactivityadvanced bonding models, nucleophiles, electrophiles
phase &compositionbond strengths, intermolecular attractions, polarity
properties & atomic viewsoctet rule, ionic and covalent bonds
matter with chemical symbolselements, compounds,valence electrons, periodic trends
properties of mattersolids, liquids, gases, mixtures
models & evidenceevidence about things we can’t observe directly
limitations of modelsexamining evidenceassumptions
measured amountsof matterdensity, grams permole,molarity
mass with a particulate viewatoms, isotopes, moles
amounts of mattermass, weight, volume, pressure
kinetics &changes in bondingrxn mechanisms, activation energy
products of reactionsolubilities, relative acid strengths
change &reaction typesprecipitation, acid-base, redox
change with chemical symbolsbalanced equations, phys,vs chem change
attributes of changemixing, dissolving, color change,change in form
stoichiometry & equilibriumweak acids &bases, solubility of salts, gases
amounts ofproductslimiting reagents,strong acid/base titrations, % yield
amounts ofreactants & productsreaction stoichiometry, pH
change with a conservation viewconservation of mass inchemical reactions
changes in amountchanges in mass,weight, volume
particle & energy viewsstatisical mechanics KE & temperature
degrees of changeentropy, free energy & equilibrium
energy transfer& changeenthalpy changes Hess’s law, bond breaking
heats &temperatureheat capacity,calorimetry,exo(endo)thermic
measures of energytemperature, scalesmeasures of energy
spectroscopy & structuregroup theory, transition probabilities
electronic structurequantum model,atomic & molecular orbitals, ionization energy
color with light absorptionabsorption &emission spectra
energies associatedwith lightfrequency, speed, Planck’s constant
lightproduction of light,color
Final version
5 Generation: Research
4 Construction: Examining assumptions, relating models
3 Formulation: Relating ideas and concepts, simple models
2 Recognition: Language, definitions, symbols algorithms
1 Notions: Everyday experience, logical reasoningnovice
expert
stud
e nt u
nder
sta n
ding
Focus on Student Understanding