AC-2009-492:

Analysis of Middle- & High School Students’

Learning of Science, Mathematics, and Engineering Concepts Through a LEGO Underwater Robotics Design Challenge

NSF ESI-0624709

Beth McGrath, Susan Lowes, Peiyi Lin, Jason Sayres

Build IT: NSF ITEST Project 2006-09

• Targets science, engineering, IT (programming) learning

• Career awareness of STEM and IT

• In-school implementation• Underrepresented groups

Why LEGO and Underwater Robotics?

• Grew out of ocean engineering research interests

• Presents unique, complex design challenges (e.g., buoyancy, control in 3-D)

• LEGO enables rapid prototyping, testing, redesign

Theoretical Framework• Robotics as “mindtools” (Jonassen, 2000)

• Problem-based learning: students “own” their learning

• Design-based science

Engineering Design in Science• Interaction: individual contributions to

collective product is paramount

• Artifact development: displays communal learning

• Critical analysisiterative design

• Impacts on science, mathematics learning

• Suggests narrowing gaps

Straight Line Challenge: One motor diameter of pool on surface; optimize gearing to achieve best propeller speed.

Slalom Challenge: Two motors enable steering; maneuver on surface to complete slalom course around two buoys in shortest time.

Submerge Challenge: Three motors, propellers, and other materials to control buoyancy.

Grabber Challenge: Design a motorized mechanical manipulator which can grasp specified objects.

Final Challenge: Timed competition to drive ROVs to pick up objects and place in goal at bottom of pool.

Curricular Match (HS)

Buoyancy Critical Thinking

Newton’s Laws Problem-Solving

Density/Volume Synthesis/Analysis of Problems

Gear ratio Testing

Torque and force

Basic circuits

Curricular Match (MS)

Forces Energy Motion

Density Buoyancy Volume

Mass-weight distribution

Ratio and proportion

Electricity

Data Sources: 2007-08 SY

• 36 schools (7-12)

• 50% from lowest SES

• 36 teachers from 31 schools taught Build IT to 1 or more one class

• 1/3 taught it twice or more

• Data from 40 classes: 22 MS, 18 HS

Eval: Was Build IT effective?• Formative data to improve program,

curriculum, assessments, and implementation

• What is working well, for whom and what should be improved?– Student learning of concepts– Student enjoyment of science– Engineering career awareness/interest

Eval: Was Build IT effective?• Can very diverse group of teachers teach it?

– Middle vs. high– Prior/no experience w/LEGO or robotics– Science, math, tech ed, pre-engineering, other

• Can very diverse students learn and enjoy it?

Instruments• Pre/post test (quiz) data on 2 concepts:

-buoyancy

-gearing

• Baseline/Final survey– Perceptions of engineering– Engineering career awareness/interest– Enjoyment of science– Teamwork– Iterative design

“Like Science Best” (MS)% all

students% students

in SCIENCE classes

% females in science

% males in science

Pre- 34% 44% 40% 47%

Post- 35% 50% 58% 42%

Reflection on Experience by SES

  Enjoyed (A or B)Learned (A or B)

All MS 72% 80%

Low SES MS 86% 85%

All HS 82% 78%

Low SES HS 100% 94%

Reflections by Gender

Enjoyed (A) Learned (A)

MS Girls 42% 44%

MS Boys 51% 37%

HS Girls 41% 44%

HS Boys 58% 39%

Liked Best (Boys)• Designing, building

• Hands-on, different than other classes

• See how robotics worked

Liked Best (Girls)

• Creativity, invention

• Seeing how things work

• “…I got to design robots and think technically and physically. I liked that we were able to have freedom with it. There was no specific design we had to follow.”

• Working without a teacher

Teamwork• “how important (it) is…”

• Problem-solving with group

• “Freeloader” syndrome

• Structured vs. self-selected groups

• Forced rotation of roles

Concept Learning: Gears

The mean class scores increased by almost 40 percent from pre-test to post-test (highly significant).

Assessment MeanN

(Classes)Std.

DeviationStd. Error

Mean

Pre-test 1.8833 30 .47856 .47856

Post-test 2.6267 30 .66898 .66898

Concept Learning: Buoyancy

• Buoyancy scores increased by 33 percent from pre- to post-test (highly significant)

• No correlation between a class’s mean post-test scores and the school’s SES

Assessment MeanN

(Classes)Std.

DeviationStd. Error

Mean

Pre-test .9280 25 .45873 .09175

Post-test 1.8880 25 .69541 .13908

GH

A BI GH A DE A A

FGGHGHCDB J J A A

A A CD J I

A IA GH

GH I

I

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

The same pattern (or lack of pattern) holds true for the buoyancy tests, and for both middle and high school.

Gear Test Mean Scores – All Classes

Conceptions of Engineering (Post)

• Students practiced iterative design, but only 26% could define it.

• MS: designing and building, less frequency of “structures”

• HS: more designing than building

• HS Girls: improving life for people

Baseline Final survey

Doctor, dentist, surgeon, vetn. 38% 32%Teacher 17% 11%Lawyer 13% 12%Scientist 13% 11%Engineer 11% 21% Subset of Girls 6% 11%Sports-related 9% 11%Computer-related 6% 8%Architect 5% 4%

Impact on Career Choices: MS

Baseline Final survey

Engineer 34% 42%

Subset of Girls 25% 30%

Scientist 26% 23%

Doctor, dentist, surgeon, veterinarian 25% 21%

Teacher 11% 10%

Computer-related 8% 9%

Architect 7% 4%

Lawyer 4% 6%

Sports-related 2% 4%

Impact on Career Choice - HS

Discussion• Build IT had statistically significant impact

on student learning of science concepts (gears, buoyancy)

• Positive impact on engineering career interest, perceptions of engineering (low SES and girls)

• Measuring hands-on learning in paper-pencil test remains an issue (implementation, vocabulary, curricular enhancements, assessments)

Next Steps• Data from 2008-09 school

year will yield new insights:– Teacher comfort and

implementation bugs addressed

– Programming using NXT-G and Mindstorms an added dimension

Questions?