Ocean Discovery Institute/Rokenbok Curriculum Pilot, Summer 2012

Evaluation Report

Prepared for Ocean Discovery Institute September 19, 2012

by Roxanne Ruzic, EdD

Ruzic Consulting, Inc.

10601 Tierrasanta Blvd., Suite G136 San Diego, CA 92124

rox@roxanneruzic.com

(858) 384-7237

2

Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report Prepared by Roxanne Ruzic, September 19, 2012

Introduction During the summer of 2012, representatives from Ocean Discovery Institute and Ruzic Consulting, Inc. collaborated to conduct a small pilot evaluation of a new curriculum for high school students that utilizes Rokenbok toys to teach principles of engineering through instruction coupled with hands-on, collaborative bridge building activities. The pilot evaluation was designed to investigate the ways in which students worked together and engaged with the curriculum activities using the Rokenbok toys, as well as how successful the curriculum was in helping students master the engineering principles presented. Methodology We piloted the full, four-lesson curriculum with 20 students, 10 male and 10 female, over the course of two days during a low-cost summer enrichment science program offered by a nonprofit organization in Southern California. Students who participated in the pilot ranged in age from 14-17 and represented a wide range of ethnicities. Students were divided into teams of three or four by the enrichment program directors, rather than Ocean Discovery Institute instructors, and stayed with these groups throughout the full, four-lesson unit, with two exceptions. One student arrived late and was assigned to a group part way through the first lesson. Two students from another single group were absent for the fourth lesson. The remaining student from that group was reassigned to another group for that final lesson. During the pilot evaluation, data were collected in five ways:

1. Observations of student activities and student conversations during the lessons; 2. Informal interviews with students during the lessons; 3. Examination of artifacts, such as worksheets and Rokenbok toy bridge models,

created by students during the lessons; 4. Short content exams given to students three times during the full unit (before the

first lesson, midway through the full unit, and after the final lesson); 5. Surveys of students at the conclusion of the full unit.

Findings We identified five primary findings from the pilot evaluation:

1. The curriculum using Rokenbok toys was highly engaging for students; 2. The curriculum encouraged high-level problem solving and collaboration; 3. The majority of students improved their understanding of engineering principles

presented in the unit; 4. Instructor guidance was a critical component of the student learning experience; 5. Students enjoyed the experience.

Each of these findings is discussed in more depth below.

3

Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report Prepared by Roxanne Ruzic, September 19, 2012

The curriculum using Rokenbok toys was highly engaging for students

“My favorite part of this experience was when we were testing out our bridges. The suspense in the air as the weights were being placed was kind of funny.” —Female pilot evaluation student

• All student groups were able to design and build a full bridge during a single 45-minute class period using the Rokenbok toys.

• Students were able to prototype rapidly while still creating bridges that

differed from one another using the Rokenbok toys. When students entered the room for the first activity, there was little interaction. Students did not appear to know each other well and—with the exception of a single group—sat quietly, not talking with each other, not looking at each other, and not interacting. In the short initial introduction to the unit, most students did not volunteer to speak. Once students were presented with the first activity, however, all the students in the room engaged almost immediately with the task of working together to choose and build a bridge with the Rokenbok toys, modeled after one of the types of bridges presented. Not all students participated in the same way, and the dynamics in each group were different. Some students put pieces of the bridges together, some watched and gave suggestions or direction to their teammates, and some alternated between participating in building and giving suggestions. Students sometimes took on different building tasks, and sometimes worked on a single piece of the bridge together. Students in some groups talked extensively with each other, while other groups spoke barely at all. All participants except one (95%), however, were fully engaged in the process of building the bridge throughout the entire lesson. This pattern of total engagement by all students except one during bridge building was consistent across all four lessons, including when members were moved or added to new groups. In addition, the student who was partially rather than fully engaged during the first two lessons also engaged more fully when her group shrunk to only one other member during the third lesson. During the bridge breaking process, students again were focused entirely on the activity, with all eyes in the room on each bridge being tested each time. There was a sense of excitement and suspense, and there was cheering or laughter each time a bridge broke during the first day. During the second day, when bridges were able to hold significantly more weight, the bridge breaking process took longer. Three additional students were less engaged during parts of the longer wait for each bridge to break, but the room remained almost entirely silent throughout the bridge breaking process until each bridge broke. We did not test potential variations in the curriculum empirically, and therefore cannot make definitive claims about the causes of these high levels of engagement. We hypothesize that a number of aspects of the activity we witnessed may have contributed. These include the requirement that students work in teams; the use of hands-on, visible manifestations of the engineering problem being considered; the

4

Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report Prepared by Roxanne Ruzic, September 19, 2012

specification of a desired end-goal (a bridge that supported the most weight); testing in the form of competition; and the use of building tools that enabled students to prototype rapidly and test variations in design while still creating structures that differed from one another.

Requiring that students work on a concrete, visible, well-defined problem may have allowed students to more easily picture and discuss their conceptual activity and remain clear about their goals and progress. Requiring that students work in teams and engage in public competition may have provided motivation through interpersonal interaction and the desire to succeed in a public forum. Using tools that allowed for rapid prototyping may have made the building periods more motivating for students because they could create their designs relatively quickly. The potential for variability in design in the building tools have made both the design and testing phases more interesting for students because they could make authentic choices and see the effects of those choices. The building materials used with the curriculum unit during the pilot study consisted of pre-formed Rokenbok toys: plastic pieces including beams, blocks, risers, braces, and supports. With these toys, students were able to construct their bridges rapidly, rather than having to create their bridges brick-by-brick. During the pilot study, all student groups were able to design and build a full bridge during a single 45-minute class period using the Rokenbok toys. In addition, students had time to experiment during the process. We observed members of each team creating and taking apart portions of their bridges throughout each activity as they tested, discussed, and refined their bridge designs on the fly. The fact that the Rokenbok toys used during the activities were able to fit together in different ways may have made the bridge breaking portions of the curriculum more interesting for students as well. During the pilot study, different groupsʼ final bridge designs varied from one another, even when all groups were asked to work on the same type of bridge during a particular lesson. This variation in final designs may have made the tests to see which bridge would hold the most weight more suspenseful than if all the bridges tested had been essentially the same.

The curriculum encouraged high-level problem solving and collaboration

“My favorite part was getting to build the bridges because they challenged our group and made us work as a team.” —Male pilot evaluation student

• The bridge building curriculum using Rokenbok toys fully and consistently engaged all students but one in deep reasoning and problem solving activities directly related to the content being presented.

Many school activities that engage students fall into one of two categories: assignments that engage students but do not require them to think deeply, and assignments that engage students in activities that do not focus directly on the academic content of a particular unit.

5

Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report Prepared by Roxanne Ruzic, September 19, 2012

For example, students are sometimes asked to write down information individually, follow directions using individual sets of tools, or work as members of a group to repeat a pre-determined process using a set of tools for the full group. These types of lessons often lead to high levels of engagement while students participate in hands-on activity, but the lessons themselves often do not require the students to think deeply about the content being presented. Instead, the focus is on following directions and completing the task. Alternately, students are sometimes divided into groups and asked to work on projects or create presentations with more freedom. These types of activities also often lead to high levels of engagement while students focus on completing the project, but the activities often do not require students to deeply engage with the content being explored, as when students spend time engaged in tasks such as looking for images and videos to complete a presentation rather than focusing on making sense of new content or solving authentic problems. Unlike these examples, the bridge building curriculum using Rokenbok toys tested in this pilot evaluation fully and consistently engaged all students but one in deep reasoning and problem solving activities directly related to the content being presented. Every group, over each of the four lessons, spent 100% of the time for each activity deeply engaged in the process of deciding how to build the different kinds of bridges and why. We observed no off-task activities during the building process, and noted that, with the exception of the one student already mentioned, 100% of students were fully engaged in the process with other group members throughout each of the four units. The work we witnessed was truly collaborative, with students creating plans in their notebooks, discussing options, building, testing, taking designs apart and re-building, all with the full participation and involvement of all group members throughout. The bridge building tasks were intellectually challenging without being prescriptive, and students were fully focused on those tasks. The Rokenbok toys used during the unit supported this curricular approach, providing structure and constraints while still allowing for creativity in the final structures. While we did not test variations in the curriculum empirically, we hypothesize that this combination of constraints and flexibility within a hands-on curriculum focused on a concrete task may have provided the opportunity for students to engage thoughtfully and authentically with their designs as opposed to following a lock-step procedure or engaging with activities outside the core learning goals.

The majority of students improved their understanding of engineering principles presented in the unit

“My favorite part was just learning about bridges in general and getting to build each type and the learn the flaws/strengths of each type.” —Female pilot evaluation student

• The majority of students demonstrated an increased understanding of engineering principles presented in the curriculum unit after participating in the four lessons using the Rokenbok toys.

6

Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report Prepared by Roxanne Ruzic, September 19, 2012

Before the first lesson, 7 of 19 students (37%) who took the pre-assessment were not able to identify the three different kinds of bridges—beam, arch, and truss—presented in the unit. By the end of the second unit, 100% of students were able to identify all three. Similarly, before the first lesson, no student was able to accurately describe the dissipation of forces in the three different kinds of bridges presented in the unit. Before the first lesson, 8 of 19 students (42%) were unable to accurately identify any of the forces acting on any type of bridge presented in the unit. By the end of the full unit, 17 of the 20 students (85%) had demonstrated an increased understanding of dissipation and the forces acting on the bridges. After the final lesson, 3 of 16 students who took the final assessment (19%) were able to accurately depict how dissipation functions in an arch bridge; 7 of 16 students (44%) how dissipation functions in a truss bridge; 9 of 16 students (56%) how dissipation functions in a beam bridge; and 2 of 16 students (13%) how dissipation functions in all types of bridges presented in the unit.

Instructor guidance was a critical component of the student learning experience

“My favorite was building the truss bridge. I liked it because it was challenging and seemed like the strongest.” —Female pilot evaluation student

• Instructor guidance was important in helping students understand the engineering principles presented in the unit and how to design effective bridges using the Rokenbok toys.

An important aspect of the curriculum evaluated in this pilot study was the guided nature of the instruction. In addition to the short informational sessions at the beginning of each lesson and the structure created by the specific bridge building activities, the instructor created additional support and structure for student learning by asking thoughtful questions of the students throughout the bridge building process. As students were involved in the hands-on group building activities, the lead instructor moved about the room, asking open-ended questions of the students about their designs. On multiple occasions, we observed the conversation and building activities shift immediately in response to these questions. The questions did not seem designed to test studentsʼ knowledge or check understanding by eliciting a correct answer for all members of the group to hear. Instead, the questions seemed designed to shift student focus to important but neglected aspects of the engineering process in an open-ended way. In most cases, the instructor did not wait for an answer to her question before moving on to the next group. Instead, she examined the bridge and the building process and then posed a question or questions that she left the group to discuss and consider. Questions asked in the pilot (and included as examples in the curriculum unit) addressed such topics as where studentsʼ bridges were weak and why; why students chose particular design elements and how those elements affected bridge strength and stability; and how students might use the engineering principles presented in the unit to strengthen their designs.

7

Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report Prepared by Roxanne Ruzic, September 19, 2012

Students enjoyed the experience

“Great thing to do in a science or engineering class!” —Male pilot evaluation student

• A significant proportion of students found it challenging to work with the Rokenbok toys.

In surveys at the conclusion of the program, students were upbeat about the unit, with 100% of students reporting that they enjoyed the experience. In fact, 5 of the 16 students (31%) who completed the final survey asked that the session be longer or that they be allowed to complete additional activities. Students cited the hands-on nature of the activity, the chance to learn more about bridges and the principles behind them, and the opportunity to collaborate and work as a team as positive aspects of the curriculum. Students had few suggestions for improvements, though six students explicitly mentioned that they struggled when working with the Rokenbok toys. During the lessons, we observed members of every team having difficulty putting pieces together and pulling pieces apart. In addition, several groups of student did not seem to distinguish between the 30-degree and 60-degree connectors, which led to unexpected issues with at least three bridge designs.