Analyzing students’ learning performances Analyzing students’ learning performances in terms of practices for developing in terms of practices for developing

accounts accounts

Hui Jin, Jiwon Kim and Charles W. AndersonHui Jin, Jiwon Kim and Charles W. Anderson

Research GoalResearch Goal

• To develop a teaching-oriented learning To develop a teaching-oriented learning progression framework for carbon-progression framework for carbon-transforming processes in socio-ecological transforming processes in socio-ecological systems. systems.

COMMON FRAMEWORK: Learning Progression Framework

Assessment Curriculum Instruction

General approachGeneral approach

PHASE 1: Use assessment PHASE 1: Use assessment data to develop/revise the data to develop/revise the learning progression learning progression framework framework

PHASE 2: Use the learning PHASE 2: Use the learning progression framework to progression framework to guide curriculum guide curriculum development; Implement development; Implement the curriculum and collect the curriculum and collect interview and written data interview and written data before and after the before and after the teaching interventionteaching intervention

PHASE 3: Analyze assessment PHASE 3: Analyze assessment data and use results of data and use results of analysis to revise learning analysis to revise learning progression frameworkprogression framework

Data sources (2009-10)Data sources (2009-10)

2009-10 Data Source

Tests Interviews

Elementary Schools 167 149 12 12

Middle Schools 288 338 16 16

High Schools 262 195 16 16

College 728 713 0 0

ASSESSMENT QUESTION 1: What is ASSESSMENT QUESTION 1: What is the scientific reasoning about the scientific reasoning about

carbon-transforming processes?carbon-transforming processes?

Three scientific elements Two characteristic ways of scientific reasoning

Scientific explanations of carbon-Scientific explanations of carbon-transforming processestransforming processes

MATTER

ENERGY

SCALE

Using principles of matter and energy to constrain

accounts

Reasoning across scales

Using principles of matter and Using principles of matter and energy to constrain accountsenergy to constrain accounts

Using Electric appliances

Driving Vehicles

Burning fossil fuels

Body Movement; Dead Organism Body Decay

Plant Growth Animal Growth

Organic Carbon O

xidation(Com

bustion)

Organic Carbon Oxidation (Cellular Respiration)

Organic Carbon Generation(Photosynthesis)

Organic Carbon Transformation (Biosynthesis, digestion)

CO2

Light Energy

Organic Carbon

Chemical Energy

Chemical Energy

Organic Carbon

Chemical Energy

Organic Carbon

Chemical Energy

Organic Carbon

Heat

CO2CO2

Heat

PRINCIPLESMatter Conservation, Energy Conservation, Energy

Degradation

Human Socio-economical Systems

Ecosystem

Atmosphere

Reasoning across scalesReasoning across scales

Atomic-molecular Atomic-molecular Macro-Scale Large-scale Macro-Scale Large-scale

3. Animal/People bodily functions4. Dead body decay5. Burning fossil fuels

2. Animal/People Growth

1. Plant Growth

Organic-carbon oxidation—cellular respiration and combustion

Organic-carbon transformation—digestion & biosynthesis

Organic-carbon generation—photosynthesis

Carbon Cycle

Energy flow

Macroscopic Scale

An example of Reasoning across An example of Reasoning across scales: Human Breathingscales: Human Breathing

CO2

O2

Atomic-molecular Scale

O2 + organic molecules CO2 + H2O

• Reasoning across scales: Patterns observed at the Reasoning across scales: Patterns observed at the macroscopic scale may be different from macroscopic scale may be different from processes at the atomic-molecular and large processes at the atomic-molecular and large scalesscales

ASSESSMENT QUESTION 2ASSESSMENT QUESTION 2How do students reason about How do students reason about

carbon-transforming processes? carbon-transforming processes? How shall we assess their How shall we assess their

understanding?understanding?

Students’ informal ExplanationStudents’ informal Explanation

Force-dynamic Explanations: Actors, Enablers, Results

WaterWater

SoilSoil

SunlighSunlightt

AirAir

ASSESS STUDENT UNDERSTANDING: ASSESS STUDENT UNDERSTANDING: Process dimensionProcess dimension

Atomic-molecular Atomic-molecular Macro-Scale Large-scale Macro-Scale Large-scale

3. Animal/People bodily functions4. Dead body decay5. Burning fossil fuels

2. Animal/People Growth

1. Plant Growth

Organic-carbon oxidation—cellular respiration and combustion

Organic-carbon transformation—digestion & biosynthesis

Organic-carbon generation—photosynthesis

Carbon Cycle

Energy flow

The first dimension of

the assessment:

6 Macroscopic Processes

6. Cross processes

ASSESSMENT QUESTION 3ASSESSMENT QUESTION 3How shall we analyze complex How shall we analyze complex

accounts from students?accounts from students?

Assess student understanding: Assess student understanding: Practice dimensionPractice dimension

Scale Macro-scale Micro-scale Large-scale

Matter

Explaining Materials Practice: Explain changes of matter qualitatively

Explaining Subsystems Practice: Explain macroscopic processes in terms of processes at a smaller scale; Explain macroscopic substance properties in terms of structure or properties at a smaller scale

Explaining Large-scale systems Practice: Classify macroscopic processes and explain their connections in terms of patterns at large-scale. Classify macroscopic substances and explain their similarities.

Explaining Mass Practice: Explain changes of matter quantitatively

Energy

Explaining Energy Practice: Explain changes of energy qualitatively

• BEAR Assessment System: Progress Variables; Levels of BEAR Assessment System: Progress Variables; Levels of AchievementAchievement

• Use a fine-grained measure to assess students’ understanding of Use a fine-grained measure to assess students’ understanding of the three scientific elements and two characteristic ways of the three scientific elements and two characteristic ways of reasoning. reasoning.

Use matter conservation to constrain accounts about processes

Use energy conservation and degradation to constrain accounts about processes

The second dimension of the assessment:

5 Practices

Assess student understanding from two Assess student understanding from two dimensionsdimensions

Learning Progression FrameworkLearning Progression Framework

Explaining Materials PracticeExplaining Materials Practice

Level Description Examples

4 Explaining about chemical identity of molecules and atom re-arrangement in chemical reaction

Carbon dioxide reacts with water to produce glucose and oxygen is also released.

3 Explaining about changes involving atoms and molecules

The sugar is energy. It turns into energy to help it [the tree] grow.

2 Explaining changes of the stuff that make up actors and enablers

The wood turns into ashes and smoke which is released into air.

1 Explaining about actors and enablers

You have to have a wick, or else it couldn’t burn on and it would be nothing for the fire to stay on.(Flame—Actor; Wick—Enabler that holds the flame)

Explaining Large-scale systems PracticeExplaining Large-scale systems Practice

Level Description

Examples

4 Carbon cycle & energy flow

Light energy goes into the EcoSphere and heat is released out of the EcoSphere.

3 Carbon cycling without transformation; Energy cycling

Explain how three things—algae, shrimp, and bacteria—stay alive in an EcoSphere. Energy cycles between shrimp and the algae it’s taking, like chemical energy from the algae and using it to live. Shrimp breathe, right. It would probably provide the oxygen for the algae to live so that would be like chemical energy I guess.

2 Stuff moving in food chain, gas cycle, etc.

Explain the arrows in the diagram (loop diagram about interactions between human system, ecosystem and atmosphere)The boy is breathing out the carbon dioxide. The trees collect the carbon dioxide from the atmosphere and then produce oxygen, clean oxygen that we can breath.

1 Sequences of events; Perceptions

Explain similarities and differences among tree growth, car running and flame burningThe car has gas in it, so that it changes the air. You can smell the gas. You can smell trees. If you are buy like a maple tree candle, you would smell the maple sort of.

Additional SlidesAdditional Slides

An example of emergent property: An example of emergent property: Chemical energyChemical energy

• Organic carbon-containing substances contain energy Organic carbon-containing substances contain energy due to the configuration of atoms in molecules (C-C and due to the configuration of atoms in molecules (C-C and C-H bonds); Carbon atoms do not contain energy. C-H bonds); Carbon atoms do not contain energy.

Consistency of students’ reasoningConsistency of students’ reasoning

• Consistency across the five practicesConsistency across the five practices• Correlation coefficients range from .789 to .907Correlation coefficients range from .789 to .907

• Consistency across different contexts (eight Consistency across different contexts (eight interview tasks)interview tasks)• Correlation coefficients range from .690 to .908Correlation coefficients range from .690 to .908

Validity Analysis between interview data Validity Analysis between interview data and written dataand written data

Boxplots for Interview vs. Written Assessment

Validity Analysis between interview data Validity Analysis between interview data and written dataand written data

Validity Analysis between interview data Validity Analysis between interview data and written dataand written data

Validity Analysis between interview data Validity Analysis between interview data and written dataand written data

Scatter plot of Average Interview Scores and Written Assessment Ability Estimates(r=.526)