A Pilot Study on Pearson’s Interactive Science Program · 2015. 9. 29. · Prepared by PRES...

Prepared by PRES Associates, Inc. – An Independent Evaluation Company 1

A Pilot Study on Pearson’s Interactive Science Program

Final Report

Prepared by:

Miriam Resendez, Senior Researcher

Mariam Azin, President

Submitted on June 28, 2009

For inqui r ies , p lease contact PRES Assoc iates at :

[email protected]

(307) 733-3255

© 2009 PRES Assoc iates, Inc .


Table of ContentsTable of ContentsTable of ContentsTable of Contents

Executive Summary ........................................................................................................................... 5

Project Background......................................................................................................................... 10

Project Overview.............................................................................................................................. 11

Methodology ..................................................................................................................................... 13

Sites and Sample Characteristics ................................................................................................... 15

About the Schools .......................................................................................................................... 15 About the Pilot Classes .................................................................................................................. 15 Description of the Regular (non-PIS) Science Curriculum and Resources ................................... 16

Results ............................................................................................................................................... 17

1) What preliminary relationships are observed between use of the Pearson Interactive Science

program and key student and teacher affective outcomes? ......................................................... 21 2) What do users of the Pearson Interactive Science program think about the program? What

aspects of the program do they find most useful? Least useful? ................................................. 29 3) How do teachers use the Pearson Interactive Science program in their classrooms? Do

teachers of differing pedagogical approaches and philosophies implement the program differently across science classrooms? ........................................................................................ 40

4) How should the Pearson Interactive Science program best be used in order to maximize its impact on student performance? .................................................................................................. 48

5) What type(s) of training and preparation is needed in order to promote effective implementation of the Pearson Interactive Science program? Are the built-in teacher resources useful to teachers in helping them prepare to effectively deliver science instruction in their classrooms? .................................................................................................................................. 49

6) What preliminary impact does the Pearson Interactive Science program have on student science outcomes?........................................................................................................................ 52

7) Which type(s) of assessments and outcome measures will be most sensitive to picking up the effects of the Pearson Interactive Science program? ................................................................... 56

Conclusion ........................................................................................................................................ 57

References ......................................................................................................................................... 60

Appendix: Table of Statistical Results ........................................................................................... 61


Table of Figures and TablesTable of Figures and TablesTable of Figures and TablesTable of Figures and Tables Tables Table 1. Study Site School Level Student Demographics ...........................................................15 Table 2. Pilot Classes and Pearson Interactive Science Chapters Taught ...................................16 Table 3. Ratings of Usefulness of the Pearson Interactive Science Program by Pilot

Teachers ..........................................................................................................................40 Table 4. Frequency of Use of Pearson Interactive Science Components ....................................43 Table 5. Average Number of Labs and Time to Complete Labs & Chapters..............................44 Table 6. Difference Scores (Post minus Pre) by Chapter Tests and Type of Test Item ..............56 Table 7. Psychometric Properties of the PIS Chapter Tests ........................................................56 Table A1. Pre and Post PIS Chapter Test Statistics by Item Type .................................................63

Figures Figure 1. Student Perceptions of the Degree to Which They Were Interested in Science During

Their Use of Pearson Interactive Science Program Relative to Other Science Program........................................................................................................................... 22

Figure 2. Teacher and Student Perceptions of the Degree to which the Pearson Interactive Science Program Affected Student Engagement Relative to Other Science Program ... 22

Figure 3. Percent of Teachers and Students who Agree that the Pearson Interactive Science Content and Labs Facilitate Student Interest .................................................................. 22

Figure 4. Percent who Agree to Looking Forward to Science and Taking Pride in Work While Using Pearson Interactive Science.................................................................................. 23

Figure 5. Teacher and Student Perceptions of the Degree to which the Pearson Interactive Science Program Affected Science Connections and Applications Relative to Other Science Program ............................................................................................................. 24

Figure 6. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Affected Other Academic Areas Relative to Other Science Program............................ 24

Figure 7. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped them Determine Student Understanding Relative to Other Science Program ... 25

Figure 8. Percent of Students who Agree that the Pearson Interactive Science Program Assists with Organization ........................................................................................................... 26

Figure 9. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped them on Tests and Future Science Classes Relative to Other Science Program 26

Figure 10. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped them with their Preparation and Practices Relative to Other Science Program. 27

Figure 11. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped with Individualizing Instruction Relative to Other Science Program ................ 27

Figure 12. Teacher and Student Perceptions of the Degree to Which They Engaged in Science Class Activities During Their Use of Pearson Interactive Science Program Relative to Other Science Program............................................................................................... 28

Figure 13. Teacher and Student Perceptions of the Degree to Which They Engaged in Lab Activities During Their Use of Pearson Interactive Science Program Relative to Other Science Program ................................................................................................... 28


Figure 14. Teacher and Student Perceptions of the Degree to Which Teacher Engaged in Other Activities During Their Use of Pearson Interactive Science Program Relative to Other Science Program ................................................................................................... 28

Figure 15. Percent of Teachers and Students who Liked the Pearson Interactive Science Program........................................................................................................................... 29

Figure 16. Average Descriptive Ratings by Students on the Pearson Interactive Science Program........................................................................................................................... 30

Figure 17. Teacher Rating of Preference for Pearson Interactive Science Program versus Other Science Program ............................................................................................................. 30

Figure 18. Student Perceptions of the Degree to which the Pearson Interactive Science Program is Better Relative to Other Science Program .................................................................. 31

Figure 19. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program is Better Relative to Other Science Program .................................................................. 32

Figure 20. Percent of Students who Liked the Pearson Interactive Science Student Worktext ..... 33 Figure 21. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program

is Better in terms of Ease of Use Relative to Other Science Program............................ 34 Figure 22. Percent of Teachers and Students who Liked the Pearson Interactive Science

Illustrations ..................................................................................................................... 36 Figure 23. Percent of Teachers and Students who Liked the Pearson Interactive Science Labs..... 37 Figure 24. Teacher Ratings of the Pearson Interactive Science Labs by Type................................ 37 Figure 25. Frequency of Teacher Instructional Practices ................................................................ 41 Figure 26. Percent of Teachers Who Felt Training and Teacher’s Edition were Helpful ............... 51 Figure 27. Student Performance on the Pearson Interactive Science Chapter Test: Earthquakes ... 52 Figure 28. Student Performance on the Pearson Interactive Science Chapter Test: Populations

& Communities............................................................................................................... 53 Figure 29. Student Performance on the Pearson Interactive Science Chapter Test: Work &

Machines......................................................................................................................... 53 Figure 30. Student Performance on the Pearson Interactive Science Earthquakes Test by Type

of Test Items ................................................................................................................... 53 Figure 31. Student Performance on the Pearson Interactive Science Populations & Communities

Test by Type of Test Items ............................................................................................. 53 Figure 32. Student Performance on the Pearson Interactive Science Work & Machines Test by

Type of Test Items .......................................................................................................... 54 Figure 33. Percent of Students who Felt that the Pearson Interactive Science Program Helped

Them Learn About Science ............................................................................................ 54 Figure 34. Student Performance on the Pearson Interactive Science Tests by Gender ................... 55 Figure 35. Student Performance on the Pearson Interactive Science Tests by Minority Status ...... 55 Figure 36. Student Performance on the Pearson Interactive Science Tests by Performance

Categories ....................................................................................................................... 55 Figure 37. Student Performance on the Pearson Interactive Science Test – Constructed

Response Items by Teacher Pedagogy ........................................................................... 55


Executive SummaryExecutive SummaryExecutive SummaryExecutive Summary

As advances in technology and science continue to alter the world we live in, it is imperative that our youth have a strong science foundation. Unfortunately, research continues to show that U.S. students are not being provided with a solid science education that offers the tools they need to think critically and analyze both simple and complex situations. In an effort to meet the needs of students so that they can become successful in science, and indeed their future, Pearson Education is finalizing a new comprehensive science program for middle school students – the Pearson Interactive Science (PIS) program. This program, which incorporates Understanding by Design, represents a model of inquiry-learning that lends itself to higher-order thinking skills and understanding.

Planning, Research, and Evaluation Services

(PRES) Associates, Inc. conducted a pilot study on the Pearson Interactive Science program during the Spring of 2009 in order to provide formative feedback about the program. In addition to preliminary descriptive and outcome information on this new science program, data was collected to inform the design and implementation of: 1) a larger-scale pilot study planned for Spring 2010; and 2) a multi-site, year-long randomized control trial (RCT) to be conducted during the 2010-2011 school year. The pilot study consisted of 435 students in grades 6-8 and 7 science teachers spread across 4 middle schools. Three PIS chapters were piloted and consisted of Earthquakes, Populations and Communities, and Work and Machines.

This report provides information on the

design of the pilot study and procedures used; background information on the pilot study sites; and preliminary findings from the study. The report also summarizes the lessons learned during this pilot study and its implications for

the future pilot and RCT on the Pearson Interactive Science program. What follows is a summary of the key findings from the study arranged by research questions.

♦ What preliminary relationships are

observed between use of the Pearson

Interactive Science program and key

student and teacher affective outcomes?

The Pearson Interactive Science program was associated with changes in a variety of important areas – many of which are related to educational outcomes. In particular, qualitative and quantitative data indicated relationships between use of the PIS program and: (a) interest and engagement in science; (b) sense of ownership and pride; (c) application of science; (d) performance in other academic areas; (e) monitoring science learning; (f) organizational skills; (g) predicted performance on state assessments and future science courses; and (h) teacher practices and levels of preparedness.

♦ What do users of the Pearson Interactive

Science program think about the

program? What aspects of the program

do they find most useful? Least useful?

What, if any, suggestions for program

improvement do they have?

Overwhelmingly, teachers and students liked using the program. They felt the program as a whole was very engaging and well put together. They also would strongly recommend the program to others and several students indicated that they would like to use PIS during the upcoming school year. When students were asked to compare PIS to other science programs they have used regularly during the school year, students overwhelmingly indicated that they


preferred the new Pearson Interactive Science program. In particular, many students said that, in comparison to their other science program, the PIS program was more entertaining, did not include extraneous content (i.e., “gets to the point”), was easy to understand and follow, provided more examples, contained more lab opportunities, was more interactive, and was portable (lightweight). Moreover, when asked what they liked best about the program, the most oft-cited features were the writing opportunities and the labs.

Teachers also clearly indicated preference for PIS. They felt that, compared to their other science program, the PIS program was easier to use, facilitated a sense of student ownership because students had their own worktexts they could write in, had what they needed to effectively teach science, and promoted better science understanding. Other general themes that emerged from respondents with regard to the various features of the program included: � Teachers and students loved the ability

to write in the worktext -- the majority of students indicated that they felt more invested in learning and learned more as a result of being able to interact and use their worktext in this manner.

� Overall, teachers who used the Pearson Interactive Science program noted that the program was easy to use.

� Teachers generally felt that pacing was appropriate or easily modified.

� Teachers and students loved the layout of the program (i.e., design and organization).

� Teachers and students also thought the illustrations in the text were visually-appealing and helped promote student interest and learning.

� Teachers and students found the labs fun and engaging. They also felt that the labs related to the content in the book and helped enhance comprehension. That said, some users indicated that the directions were vague or confusing for some of the labs.

� Although most teachers commented that the content and readability of the program was at-grade level (i.e., geared towards on-level students), there were some comments that the program was too easy.

� Some recommendations noted by students and teachers included: (1) providing more challenging activities for more advanced students; (2) allow for greater spaces for writing; and (3) make directions for labs more student-friendly.

♦ How do teachers use the Pearson

Interactive Science program in their

classrooms? Do teachers of differing

pedagogical approaches and philosophies

implement the program differently across

science classrooms?

Participating teachers closely followed the Teacher Edition (TE) and attempted the vast majority of activities (including labs) that were noted in their TEs. That said, teachers reported that they used the Differentiated Instruction, 21st Century Skills, and Remediate sections of their TEs with less frequency and more on an as needed basis. With regards to labs, in most cases the time it took them to complete the labs was greater, and for some significantly so, than what was estimated. Indeed, this concurs with the feedback that was obtained from teachers with regard to lab pacing. Teachers thought the time estimates were oftentimes unrealistic and that, as a result, getting through lessons and the chapter was at times difficult and subsequently slowed down their pacing.


Furthermore, teachers of different pedagogies (inquiry and mixed instructional approach) did not demonstrate any discernible differences in the way they implemented their PIS lessons. That is, regardless of their instructional philosophy, teachers used the PIS program comparably. The information detailed in this report on how teachers used the program will be helpful in the development and piloting of implementation guidelines to be used for the larger scale pilot study in Spring 2010.

♦ How should the Pearson Interactive

Science program best be used in order to

maximize its impact on student

performance?

Based on information gathered during the pilot study, it is suggested that in the Spring 2010 pilot study, teachers engage in the lesson plan as directed in the TE (e.g., Being lessons with Planet Diary, point out and discuss the Big Question, etc.). However, it is also suggested that narrower guidelines be provided to teachers with respect to labs so that teachers are able to cover as many topic areas as they can while still providing engaging, hands-on lab opportunities. In addition, if differentiated instruction activities/worksheets and 21st Century skill sections are deemed essential to student science performance, then there will need to be greater emphasis on these activities during the training and within the implementation guidelines since these activities were not completed with much frequency when teachers were left to their own devices. During the upcoming pilot study, researchers can more thoroughly examine the feasibility of draft implementation guidelines which will be used and, to the extent variation occurs, look at whether differences in how teachers use the program influences teacher and student outcomes.

♦ What type(s) of training and preparation

is needed in order to promote effective

implementation of the Pearson Interactive

Science program? Are the built-in teacher

resources useful to teachers in helping

them prepare to effectively deliver science

instruction in their classrooms?

Since one exploratory question embedded in this smaller pilot study was to determine how teachers would typically use the program on their own (i.e., without highly detailed implementation guidelines), trainings on the program was focused on providing a general overview of the key components and the theoretical basis behind the program. Trainings did not provide highly detailed information on how to implement the program nor did they convey “must use” components. That said, the pilot study provided information on how the program should ideally be used in future research studies. With this in mind, it will be important for future studies of the PIS program to include a more structured, formal training so as to ensure that teachers use the program in a way that will optimally benefit students. Such training may include, for example, a thorough discussion of the program’s research base and philosophy, and a detailed walk through of all the teaching resources available as part of the Pearson Interactive Science program, where to find them, and how to use them (e.g., lesson modeling). For the Spring 2010 pilot, we can also examine in more detail the training model and its sufficiency to prepare teachers to hit the ground running. Furthermore, teachers felt that they were supplied with sufficient resources to effectively implement the PIS program. They also reported that they were quickly able to learn how to use the program, and


the lessons and labs were noted as being easy to implement. Due to the more limited amount of training provided to the North Carolina teachers, these teachers felt that the training was lacking as they were unable to go over all the materials in much detail. That said, they also indicated that the TE was extremely helpful and facilitated their implementation of the science program through self-study.

♦ What preliminary impact does the

Pearson Interactive Science program

have on student science outcomes?

It should be noted that the pilot study does not include a comparison group and was only conducted over a limited period of time; therefore, while preliminary data on student learning gains and other data on participant perceptions and feedback were obtained, the pilot study does not allow for strong causal inferences to be made in regards to the effectiveness of the program –the 2010-11 RCT will be designed to do this. With this in mind, results showed that students demonstrated significant growth from pre- to post-testing on the Earthquakes and Populations & Communities chapter tests. No significant difference was observed on the Work & Machines chapter test. This lack of significant difference may be due to several factors (and combinations thereof) such as: (1) a small sample was tested; (2) a Work & Machines teacher commented that some of the concepts covered in this chapter were too advanced and that many of her students did not have the prerequisite knowledge; and (3) teacher instructional factors. To the extent that this chapter will be included in the larger-scale pilot study, researchers will examine if similar results are obtained and also gather more detailed

information that can be used to inform the content and development of this chapter. Students also showed significant growth on the Earthquakes and Populations & Communities tests as measured by the different types of test items: multiple-choice, fill-in-the-blank, and constructed response. That is, students showed improvement on all three different types of items. However, mixed findings were obtained on the Work and Machines test. While students showed significant improvement on the multiple-choice items, there was also a significant decrease on the constructed-response items, and no significant change on the fill-in-the-blank items. Furthermore, pilot students showed significant growth on the science chapter tests regardless of whether they were male or female, minority or not a minority, and low to high performing. Interestingly, results also showed that on the constructed-response items, students of teachers who employed an inquiry-based approach to instruction showed greater growth than students of teachers who used a more blended approach. Given observed differences in student test scores by teacher pedagogy, it will be important to measure teachers’ instructional approach in future research studies to ascertain the extent to which usage of the program and student performance is affected by this.

♦ Which type(s) of assessments and

outcome measures will be most sensitive

to picking up the effects of the Pearson

Interactive Science program? What are

the psychometric properties of the

instruments used during the pilot study?

Examination of difference scores by the various types of items included in the


chapter tests showed that, with the exception of the Work & Machines test, students did equally well on the multiple-choice and constructed response tests. That is, students generally improved at a similar rate. Furthermore, they showed the greatest improvement on fill-in-the-blank test items which tended to focus on vocabulary. This suggests that the program may be quite helpful in improving students’ knowledge of science terminology. As previously noted, the PIS program also seems to impact other areas such as student engagement and application of science. It will be important to continue to measure these constructs during future research studies in order to more fully examine the impacts of this program.

In summary, the pilot study proceeded

smoothly despite initial hurdles associated with obtaining necessary permissions for the videotaping and student focus groups. Teachers commented that they appreciated the opportunity to pilot the PIS program and as noted earlier, strongly recommended the program. Moreover, the majority of students indicated that they wished they could use Pearson Interactive Science as their core science program.


Project BackgroundProject BackgroundProject BackgroundProject Background

"A decade ago, the Trends in International

Mathematics and Science Study (TIMSS)

benchmarking study showed that our best

districts could compete with anyone in the

world, but our worst districts—which, of

course, were in low-income communities—

were on par with third-world countries. This

kind of extreme inequity in education is not

unique to science, but it has enormous

repercussions in the workforce where

science-based industries are desperate for

skilled workers…You need to make inquiry-

based science relevant to kids—stimulate

their curiosity—connect it with their lives.

Together we need to change the national

dialog about science—to prepare our kids to

be honestly critical and technically

competent.” (U.S. Secretary of Education

Arne Duncan, March 2009)

The world is being transformed due to ever-increasing advances in technology and science. As a result, in order to succeed in future educational and career endeavors, a strong science foundation is no longer a bonus, but a necessity. It is essential that our youth be able to think critically and be provided with the tools they need to analyze both simple and complex situations.

Unfortunately, according to the National Assessment of Education Progress report (NAEP, 2005), 8th grade students in the U.S. showed no significant gains in science performance from 1996 to 2005. Similarly, results from the Trends in International Mathematics and Science Study (TIMSS, 2009) showed that among 8th graders, there was a lower percentage of U.S. students performing at or above the advanced benchmark in science in 2007 than in 1999 (10 v. 12 percent). This is cause for concern, especially given the backdrop of the No Child Left Behind Act (NCLB) of 2001, which enacted the requirement that by the

2007-2008 school year, states must administer science assessments at least three times during a student’s academic career. In an effort to further support education, the Obama administration recently enacted the American Recovery and Reinvestment Act (Feb. 2009), which includes the $5 billion Race to the Top fund, awarding districts who make advanced reforms, including in science education.

Such emphasis on attainment of state

standards and performance on state assessments has highlighted a profound need to learn about “what works” in science education. Accordingly, educators are requiring evidence that educational curricula produced by publishers are based upon research-proven interventions and have documented evidence that they will produce a demonstrable, positive impact on student science achievement.

“The preponderance of evidence

provided by meta-analyses and

evaluations of individual curricula seem

to confirm that inquiry-based science

curricula produce larger effects on

student achievement than do the more

‘traditional’ science curricula (Clewell et

al., 2004, p. 9).

The 2011 Pearson Interactive Science program (PIS) incorporates prior research on effective science instruction and seeks to improve upon how science is being taught in classrooms by: 1) embedding inquiry-based teaching strategies into science instruction to promote higher-order thinking skills and student engagement; 2) designing a science curriculum that addresses the latest state and national science standards; and 3) promoting real-world connections so that students have ample opportunity to apply what they learn – thereby continually reinforcing and emphasizing the relevance of science to their everyday life.


The 2011 Pearson Interactive Science program consists of 12 modules in the areas of Earth Science, Life Science and Physical Science, which are distributed as interactive worktexts1. The program contains nearly 1,000 individual lessons – allowing schools and teachers a great deal of flexibility in tailoring content, scope and sequence so as to address state and local standards and ensure that necessary content is covered prior to state assessment cycles.

For teachers, the Pearson Interactive Science program provides a comprehensive resource for lesson planning, devising lab activities, and engaging students in science content that may be outside their area of expertise or training. The program incorporates Understanding By

Design, an instructional model that places big science ideas into student-friendly all encompassing questions about science. Moreover, the PIS program employs an inquiry-based process which focuses upon the 5Es: 1) Engage; 2) Explore; 3) Explain; 4) Extend; and 5) Evaluate. Together, these features represent a model of inquiry-learning that lends itself to higher-order thinking skills, understanding and critical thinking. For students, the Pearson Interactive Science program presents real-world information that is personally relevant and socially engaging. The text is designed so students can make their own interpretations of the material and self-assess their understanding of the information presented. In sum, every page of this program is designed to get students engaged and excited about science and the student worktext serves as an individual record of student learning that students can continually use to measure and reflect upon their own progress.

1 “Worktexts” are a combination of a student text and a workbook. Students are able to use their worktext in an interactive manner, including writing in their worktext, and can keep it as a journal of their science learning.

Other key features of the Pearson Interactive Science program include:

� Engaging content and active learning opportunities to help teachers engage students along the road to scientific understanding.

� Each lesson features rich content, daily ongoing assessment, built-in student help, problem solving, and vocabulary support, all of which facilitate increased student confidence and independence in science.

� The program also focuses on helping teachers provide clear, effective, differentiated instruction to reach all learners through the program’s many technology and print resources available for planning, teaching, and assessing.

� Integrated Lab Zones allow the teacher to engage students in hands-on activities applying what they have learned to real-world problem solving scenarios.

Project OverviewProject OverviewProject OverviewProject Overview

Planning, Research, and Evaluation Services (PRES Associates), an external, independent, educational research firm with 20 years of experience in applied educational research and evaluation conducted a pilot study on the new 2011 Pearson Interactive Science program during the Spring, 2009. This pilot study was designed to provide preliminary descriptive and outcome information on the new PIS program and inform the design and implementation of: 1) a larger-scale pilot study planned for Spring 2010; and 2) a multi-site, year-long randomized control trial (RCT) to be conducted during the 2010-2011 school year.

Of note is that this pilot study was not designed to produce conclusive evidence on the effectiveness of the PIS program. That is, due to the short duration of the pilot study and the


fact that randomization did not take place, it was not expected to produce rigorous quantitative evidence upon which strong conclusions could be drawn regarding the effects of the program on student learning. Nevertheless, researchers were able to obtain preliminary information about the relationship between the new Pearson Interactive Science program and student science performance.

The research questions and study activities

subsequently described were designed to accomplish two overarching goals. First, they were designed to provide preliminary data on teacher and student outcomes associated with using the PIS program. Second, the information obtained from the pilot study and preliminary activities undertaken during this time period will be used to inform the design and methods used during the larger-scale pilot study (Spring 2010), which will ultimately lead into a more rigorous experimental study on the effectiveness of this program (2010-2011 school year). Together, the results from these pilot studies will enhance the likelihood that the RCT will be designed in such a fashion so as to detect any positive effects that occur as a result of using the Pearson Middle Grade Science program. Specifically, the pilot study was designed to address the following research questions:

1. What preliminary relationships are

observed between use of the Pearson Interactive Science program and key student and teacher affective outcomes?

2. What do users of the Pearson Interactive

Science program think about the program? What aspects of the program do they find most useful? Least useful? What, if any, suggestions for program improvement do they have?

3. How do teachers use the Pearson

Interactive Science program in their classrooms? Do teachers of differing

pedagogical approaches and philosophies implement the program differently across science classrooms?

4. How should the Pearson Interactive

Science program best be used in order to maximize its impact on student performance?

5. What type(s) of training and preparation

is needed in order to promote effective implementation of the Pearson Interactive Science program? Are the built-in teacher resources useful to teachers in helping them prepare to effectively deliver science instruction in their classrooms?

6. What preliminary impact does the

Pearson Interactive Science program have on student science outcomes?

7. Which type(s) of assessments and

outcome measures will be most sensitive to picking up the effects of the Pearson Interactive Science program? What are the psychometric properties of the instruments used during the pilot study?

This report provides information on the

procedures used for the pilot study, background information on the pilot sites (including its teachers and students), and the findings of the pilot study. The report concludes with a list of accomplishments, the lessons learned during the pilot study and the implications of these findings to the RCT.


MethodologyMethodologyMethodologyMethodology

The Pearson Interactive Science pilot study commenced in late February, 2009 and concluded in mid-May, 20092. Both qualitative and quantitative instruments were developed to provide a comprehensive picture of program implementation as well as preliminary outcome data on a diverse set of student and teacher outcomes.

Four middle schools located in North

Carolina and Massachusetts participated in the pilot study. The pilot study included 435 students spanning grades 6-8 and seven middle school science teachers. Details on participating schools, students, and teachers are provided in the section entitled Site and Sample Characteristics.

Three chapters were available for use in the

pilot study and consisted of:

♦ Earthquakes

♦ Populations and Communities

♦ Work and Machines Teachers were provided with the following

PIS resources during the pilot study: a) student worktexts; b) teacher edition and resources (e.g., lab worksheets); and c) lab materials for each of the chapters. It should be noted that no digital path materials or science videos were yet available for the Spring 2009 pilot study---however, it is anticipated that the technology components will be available for the 2010-2011 RCT. Participating teachers selected one or two chapters and taught this science topic area using the new PIS program.

2 Of note, and as discussed later in the report, teachers took longer than anticipated to complete chapters.

Methods

A range of data was collected in the pilot study, including descriptive information, program implementation data, and preliminary outcome data. Data from both quantitative and qualitative sources were triangulated to extract recurrent themes. In particular, analysis conducted across these multiple data sources focused on the identification of themes, rather than relying on a single source or measure. The following presents details on the types of data collected during the pilot study. Quantitative Methods Teacher Survey: All participating teachers completed a teacher survey after using PIS. The survey, constructed by PRES Associates, was developed to collect information on:

♦ Classroom and instructional science practices following implementation of the PIS program as compared to prior science practices;

♦ Pedagogical approach to science instruction used by teachers participating in the study;

♦ Perceptions of the PIS science program (including comparisons to other types of science curricula); and

♦ Estimated versus actual time to complete labs and lessons

Student Survey: Participating students also completed a survey at the conclusion of the pilot study. The survey was developed to gather information on the PIS science activities they participated in as compared to prior science activities, their attitudes about the PIS program (including various components) and how it compared to their prior science curriculum. Demographic information was obtained so that to the extent there were sufficient subgroups (n>10), subgroup analysis could be conducted.


Student Assessments: As a result of the specific content taught via the PIS chapters used in the pilot study, it was deemed necessary to create chapter assessments that would directly test the content taught. Assessments were designed for each chapter using the Pearson ExamView test generator provided as well as released items from the Massachusetts state assessment (MCAS). After a thorough review of earlier versions for item quality, assessments were then compared with the drafts of the worktext samples developed to ensure that the information being assessed was directly aligned to the content presented in the actual chapter. Assessments were developed for three pilot chapters: Earthquakes, Populations & Communities, and Work & Machines. The latter two assessments were worth 50 score points and contained 20 multiple choice items, 5 completion items, and 10 constructed-response items. The Earthquakes chapter test was worth 43 score points and contained 20 multiple choice items, 5 completion items, and 8 short-answer items. Scoring guides were developed for each of the assessments. Pre/post assessments were administered to students to obtain preliminary information on the effects of the program. As stated earlier, more rigorous data involving the use of comparison groups will be collected as part of the experimental study undertaken during the 2010-2011 school year. Qualitative Methods Classroom Observations (including video documentation): One classroom observation was conducted at the NC schools and two classroom observations were conducted at the MA school. Observations focused on how teachers used PIS in their classrooms and the characteristics and behaviors of the students (including student engagement) as they used PIS. In addition, researchers took videos of a

sample of science classrooms so as to capture images of how the program is used in real-life situations by teachers and their students3. Teacher Interview: An interview was conducted with all participating teachers following the classroom observation. The purpose of these interviews was to gather detailed information on their perceptions of the PIS program and its various components (e.g., labs, worktexts, etc.), how they used it within their classrooms—including which components they tended to use most, which component they found most or least useful, and any barriers they encountered as they tried to implement PIS. Teachers were also asked about the organization and design of the program, ease-of-use, pacing, and the extent to which the program helped their students’ science understanding and engagement. The information obtained from these interviews also helped inform how the program should ideally be used with students during the upcoming larger-scale pilot study and subsequent RCT in order to produce the intended results. Student Focus Groups: Focus groups were conducted with a group of select students at each site, with the exception of one middle school in NC4. Teachers were asked to nominate students who were verbal and would express candid opinions about the program to participate in the focus groups. The purpose of the focus groups was to obtain information from students on their perceptions of the PIS program (likes and dislikes), including what they thought about the worktexts, labs, organization and pacing of the program, and perceived impacts that the program had on their engagement and understanding of science.

3 Necessary legal release and consent forms were obtained prior to any videotaping taking place. Students who did not return the video release form were excluded. 4 A focus group could not be conducted at this site due to scheduling conflicts.


Student Worktext Review: In order to determine how students used their PIS worktexts, content review and analyses were conducted on samples of student worktexts. Researchers examined the types of notes students took and the amount of writing, highlighting, and other note-taking strategies students engaged in as they used the worktext.

Sites and Sample CharacteristicsSites and Sample CharacteristicsSites and Sample CharacteristicsSites and Sample Characteristics About the Schools

The pilot study sites consisted of four middle schools, one located in Massachusetts and three located within the same school district in North Carolina. Schools are located in suburban and urban areas. The Massachusetts school is a brand new middle school and serves students in grades 6-8. The North Carolina schools were all middle schools also serving grades 6-8 and are all older but in fair condition. School B is a magnet school that emphasizes learning through visual and performing arts. The school offered a core program and an accelerated learning program for students. The participating study classes were part of the core track.

In terms of school-wide academic performance, there was variation between each of the schools. School A students showed lower performance on statewide assessments; of 8th graders who took the Massachusetts science proficiency test, 19% of students at School A were proficient compared to the state average of

39%. North Carolina students in grades 6-8 were not assessed in science. However, on the North Carolina End of Grade test in reading and math, School C students showed above average performance, School D was average to below average, and School B showed below average performance in math. School-level demographics are shown in Table 1.

About the Pilot Classes

Approximately 435 sixth to eighth grade students in 22 classes, taught by 7 teachers participated in the pilot study. There were 16 sixth grade classes taught by 5 different teachers as well as 5 seventh grade classes and 1 eighth grade class that participated in the pilot study. The average class size was 19 students with a range from 8 to 24 students.

All of the North Carolina participating

classes completed pre and post tests for each chapter taught as well as student surveys. As a demonstration pilot site because of its close proximity to Pearson offices, students in Massachusetts were not required to complete the assessments; that said, the teacher agreed to administer the surveys. As shown in Table 2, three North Carolina 6th grade teachers taught the two chapters of Earthquakes and Populations & Communities. Another North Carolina 6th grade teacher taught only the Earthquake chapter as she had already covered Populations and Communities. Two North Carolina 7th grade

Table 1. Study Site School Level Student Demographics

Category School A School B School C School D

White 56% 2% 72% 42%

Hispanic 33% 46% 5% 13%

African American 3% 51% 20% 43%

Asian 3% <1% 3% <1%

Other <1% 0% <1% 0%

Free/ Reduced Lunch 43% 98% 19% 51%


teachers and the Massachusetts teacher covered the Work and Machines chapter.

Teacher experience varied somewhat among

the participating teachers. Most teachers (6 of 7) taught science at the middle grades for 3-9 years, with one teacher indicating that she had been teaching science for 33 years. Half of the teachers also tended to lean towards an inquiry-based instructional approach, while the other half used a more blended approach.

Teachers were primarily on a traditional schedule and taught science everyday for approximately 45 minutes. However, the teacher at school C taught on a block schedule and saw her students every other day. All teachers were dedicated science teachers who primarily taught science5. There were no major science initiatives under way in the schools and the science curriculum had not been updated at the North Carolina schools (B, C and D) for many years.

Description of the Regular (non-PIS) Science Curriculum and Resources

The teachers at the North Carolina schools C and D had available to them an “issue-oriented” inquiry-based science curriculum that the district has had in place for nine years. The

5 One teacher also taught a math section.

goal of the program is to provide students with engaging real-world scenarios involving personal and societal issues that students can use to make investigations and observations. The program included teacher’s materials, a large student edition (500 page hard-cover book) and lab materials. However, since the program had been in place for a long period of time, lab/activity materials were not always readily available to teachers. Students referred to it as the “big book” and noted that there were few pictures and a lot of reading and note-taking. While this was the primary program available, teachers relied heavily on teacher-created worksheets, lab activities, and lesson plans, as well as other supplemental materials identified by teachers on an as needed basis. The teacher at school B did not use any specific program but rather followed the district’s science curriculum pacing guide and drew upon existing materials from a variety of sources to teach towards the North Carolina state science standards.

The curriculum used by the teacher at

School A consisted of the 2005 edition of the Prentice Hall Science Explorer program. Both teacher and student texts provided by the program were comprised of a variety of ancillary resources including lab materials, videos and other resources. The program features integrated math and reading support activities and differentiated instruction. This

Table 2. Pilot Classes and Pearson Interactive Science Chapters Taught

Chapter(s) Taught/Number Classes State Teacher Grade Work &

Machines Earthquakes Populations &

Communities

School A 1 6 3

School A 1 8 1

School B 2 7 1

School C 3 6 1 1

School D 4 6 4 4

School D 5 6 4

School D 6 6 4 4

School D 7 7 4


teacher, like those at the other schools also relied on teacher created and collected lab activities and supplemented to meet the needs of students. It should be noted that the new PIS program represents a major departure from the 2005 Prentice Hall Science Explorer program. The new program is specifically based on Understanding by Design and 5Es: Engage, Explore, Explain, Extend, and Evaluate conceptual framework for instructional design. The program provides a format that creates and focuses on a variety of opportunities for students to think and write about science. For example, students are able to depart from standard note taking practices and are actually encouraged to write in their worktexts. As well, features like Big Ideas and the Big Question expose students to critical questions and statements throughout the chapter versus at the end only. Another unique feature of the new PIS program is the uniform lab format and teacher lab resources that are provided together in the lab book. The combination of these distinct program components separates the PIS program from the 2005 Science Explorer program.

ResultsResultsResultsResults This section is organized by the key

questions from the Spring 2009 pilot study and reviews major findings first, followed by a more detailed presentation of results.

Major Findings

♦ What preliminary relationships are

observed between use of the Pearson

Interactive Science program and key

student and teacher affective outcomes?

The Pearson Interactive Science program was associated with changes in a variety of important areas – many of which are related to educational outcomes. In particular, qualitative and quantitative data indicated relationships between use of the PIS

program and: (a) interest and engagement in science; (b) sense of ownership and pride; (c) application of science; (d) performance in other academic areas; (e) monitoring science learning; (f) organizational skills; (g) predicted performance on state assessments and future science courses; and (h) teacher practices and levels of preparedness.

♦ What do users of the Pearson Interactive

Science program think about the

program? What aspects of the program

do they find most useful? Least useful?

What, if any, suggestions for program

improvement do they have?

Overwhelmingly, teachers and students liked using the program. They felt the program as a whole was very engaging and well put together. They also would strongly recommend the program to others and several students indicated that they would like to use PIS during the upcoming school year. When students were asked to compare PIS to other science programs they have used regularly during the school year, students overwhelmingly indicated that they preferred the new Pearson Interactive Science program. In particular, many students said that, in comparison to their other science program, the PIS program was more entertaining, did not include extraneous content (i.e., “gets to the point”), was easy to understand and follow, provided more examples, contained more lab opportunities, was more interactive, and was portable (lightweight). Moreover, when asked what they liked best about the program, the most oft-cited features were the writing opportunities and the labs.

Teachers also clearly indicated preference for PIS. They felt that, compared to their


other science program, the PIS program was easier to use, facilitated a sense of student ownership because students had their own worktexts they could write in, had what they needed to effectively teach science, and promoted better science understanding. Other general themes that emerged from respondents with regard to the various features of the program included: � Teachers and students loved the ability

to write in the worktext -- the majority of students indicated that they felt more invested in learning and learned more as a result of being able to interact and use their worktext in this manner.

� Overall, teachers who used the Pearson Interactive Science program noted that the program was easy to use.

� Teachers generally felt that pacing was appropriate or easily modified.

� Teachers and students loved the layout of the program (i.e., design and organization).

� Teachers and students also thought the illustrations in the text were visually-appealing and helped promote student interest and learning.

� Teachers and students found the labs fun and engaging. They also felt that the labs related to the content in the book and helped enhance comprehension. That said, some users indicated that the directions were vague or confusing for some of the labs.

� Although most teachers commented that the content and readability of the program was at-grade level (i.e., geared towards on-level students), there were some comments that the program was too easy.

� Some recommendations noted by students and teachers included: (1) providing more challenging activities for more advanced students; (2) allow for

greater spaces for writing; and (3) make directions for labs more student-friendly.

♦ How do teachers use the Pearson

Interactive Science program in their

classrooms? Do teachers of differing

pedagogical approaches and philosophies

implement the program differently across

science classrooms?

Participating teachers closely followed the Teacher Edition (TE) and attempted the vast majority of activities (including labs) that were noted in their TEs. That said, teachers reported that they used the Differentiated Instruction, 21st Century Skills, and Remediate sections of their TEs with less frequency and more on an as needed basis. With regards to labs, in most cases the time it took them to complete the labs was greater, and for some significantly so, than what was estimated. Indeed, this concurs with the feedback that was obtained from teachers with regard to lab pacing. Teachers thought the time estimates were oftentimes unrealistic and that, as a result, getting through lessons and the chapter was at times difficult and subsequently slowed down their pacing. Furthermore, teachers of different pedagogies (inquiry and mixed instructional approach) did not demonstrate any discernible differences in the way they implemented their PIS lessons. That is, regardless of their instructional philosophy, teachers used the PIS program comparably. The information detailed in this report on how teachers used the program will be helpful in the development and piloting of implementation guidelines to be used for the larger scale pilot study in Spring 2010.


♦ How should the Pearson Interactive

Science program best be used in order to

maximize its impact on student

performance?

Based on information gathered during the pilot study, it is suggested that in the Spring 2010 pilot study, teachers engage in the lesson plan as directed in the TE (e.g., Being lessons with Planet Diary, point out and discuss the Big Question, etc.). However, it is also suggested that narrower guidelines be provided to teachers with respect to labs so that teachers are able to cover as many topic areas as they can while still providing engaging, hands-on lab opportunities. In addition, if differentiated instruction activities/worksheets and 21st Century skill sections are deemed essential to student science performance, then there will need to be greater emphasis on these activities during the training and within the implementation guidelines since these activities were not completed with much frequency when teachers were left to their own devices. During the upcoming pilot study, researchers can more thoroughly examine the feasibility of draft implementation guidelines which will be used and, to the extent variation occurs, look at whether differences in how teachers use the program influences teacher and student outcomes.

♦ What type(s) of training and preparation

is needed in order to promote effective

implementation of the Pearson Interactive

Science program? Are the built-in teacher

resources useful to teachers in helping

them prepare to effectively deliver science

instruction in their classrooms?

Since one exploratory question embedded in this smaller pilot study was to determine how teachers would typically use the program on their own (i.e., without highly

detailed implementation guidelines), trainings on the program was focused on providing a general overview of the key components and the theoretical basis behind the program. Trainings did not provide highly detailed information on how to implement the program nor did they convey “must use” components. That said, the pilot study provided information on how the program should ideally be used in future research studies. With this in mind, it will be important for future studies of the PIS program to include a more structured, formal training so as to ensure that teachers use the program in a way that will optimally benefit students. Such training may include, for example, a thorough discussion of the program’s research base and philosophy, and a detailed walk through of all the teaching resources available as part of the Pearson Interactive Science program, where to find them, and how to use them (e.g., lesson modeling). For the Spring 2010 pilot, we can also examine in more detail the training model and its sufficiency to prepare teachers to hit the ground running. Furthermore, teachers felt that they were supplied with sufficient resources to effectively implement the PIS program. They also reported that they were quickly able to learn how to use the program, and the lessons and labs were noted as being easy to implement. Due to the more limited amount of training provided to the North Carolina teachers, these teachers felt that the training was lacking as they were unable to go over all the materials in much detail. That said, they also indicated that the TE was extremely helpful and facilitated their implementation of the science program through self-study.


♦ What preliminary impact does the

Pearson Interactive Science program

have on student science outcomes?

It should be noted that the pilot study does not include a comparison group and was only conducted over a limited period of time; therefore, while preliminary data on student learning gains and other data on participant perceptions and feedback were obtained, the pilot study does not allow for strong causal inferences to be made in regards to the effectiveness of the program –the 2010-11 RCT will be designed to do this. With this in mind, results showed that students demonstrated significant growth from pre- to post-testing on the Earthquakes and Populations & Communities chapter tests. No significant difference was observed on the Work & Machines chapter test. This lack of significant difference may be due to several factors (and combinations thereof) such as: (1) a small sample was tested; (2) a Work & Machines teacher commented that some of the concepts covered in this chapter were too advanced and that many of her students did not have the prerequisite knowledge; and (3) teacher instructional factors. To the extent that this chapter will be included in the larger-scale pilot study, researchers will examine if similar results are obtained and also gather more detailed information that can be used to inform the content and development of this chapter. Students also showed significant growth on the Earthquakes and Populations & Communities tests as measured by the different types of test items: multiple-choice, fill-in-the-blank, and constructed response. That is, students showed improvement on all three different types of items. However, mixed findings were obtained on the Work and Machines test. While students showed significant improvement on the multiple-

choice items, there was also a significant decrease on the constructed-response items, and no significant change on the fill-in-the-blank items. Furthermore, pilot students showed significant growth on the science chapter tests regardless of whether they were male or female, minority or not a minority, and low to high performing. Interestingly, results also showed that on the constructed-response items, students of teachers who employed an inquiry-based approach to instruction showed greater growth than students of teachers who used a more blended approach. Given observed differences in student test scores by teacher pedagogy, it will be important to measure teachers’ instructional approach in future research studies to ascertain the extent to which usage of the program and student performance is affected by this.

♦ Which type(s) of assessments and

outcome measures will be most sensitive

to picking up the effects of the Pearson

Interactive Science program? What are

the psychometric properties of the

instruments used during the pilot study?

Examination of difference scores by the various types of items included in the chapter tests showed that, with the exception of the Work & Machines test, students did equally well on the multiple-choice and constructed response tests. That is, students generally improved at a similar rate. Furthermore, they showed the greatest improvement on fill-in-the-blank test items which tended to focus on vocabulary. This suggests that the program may be quite helpful in improving students’ knowledge of science terminology.


As previously noted, the PIS program also seems to impact other areas such as student engagement and application of science. It will be important to continue to measure these constructs during future research studies in order to more fully examine the impacts of this program.

Detailed Findings

What preliminary relationships are observed between use of the Pearson Interactive Science program and key student and teacher affective outcomes?

Information obtained from interviews, focus groups, and surveys were analyzed to obtain preliminary information on the impacts of the Pearson Interactive Science program on student and teacher affective outcomes. Data from these sources were triangulated so as to identify recurrent themes.

Results showed that according to both

teachers and students the Pearson Interactive Science program was associated with changes in a variety of important areas – many of which are related to educational outcomes. In particular, qualitative and quantitative data indicated relationships between use of the PIS program and: (a) interest and engagement in science; (b) sense of ownership and pride; (c) application of science; (d) performance in other academic areas; (e) monitoring science learning; (f) organizational skills; and (g) predicted performance on state assessments and future science courses. The following section presents information on how the PIS program contributed to positive changes on these other important student outcomes.

Theme: Students had a higher degree of interest in science and were more

engaged in their science lessons.

Overall, students reported more interest in learning science when using the Pearson Interactive Science program. Feedback obtained from students indicates that the PIS science activities and labs provided daily opportunities for students to engage themselves in the learning process. They also liked the fact that in many cases they got to work with science partners.

♦ I think it makes science more fun. – MA Middle School A Student

♦ It’s not boring, like just taking notes, you

actually get engaged in the book, doing

all these fun activities. I find that a lot of

kids in our class like science more

because of all the activities. – MA Middle School A Student

As shown in Figure 1, when students were

asked to compare their experiences during the time they used the Pearson Interactive Science to those when they used their other science program, they felt that the PIS program afforded them with greater opportunities to be engaged in their science learning. They also indicated looking forward to science class more when they used PIS. Figure 1. Student Perceptions of the Degree to Which They Were Interested in Science During Their Use of Pearson Interactive Science Program Relative to Other Science Program

Engagement

A lot less About the A lot more

while using PIS same while using PIS

3.46

3.62

3.58

3.70

3.41

I w as interested in w hat

w e learned in science

class.

I paid attention in science

class.

My science materials kept

me interested during

science class.

I looked forw ard to

science class.

I felt more involved in

learning about science.


♦ I think the program is highly interactive.

I think the kids are highly interested in

it. It helps. – MA School A Teacher

Furthermore, when asked to specifically compare the PIS program to their other science curriculum, on average, pilot teachers and their students felt that the Pearson Interactive Science program had a positive impact on students’ interest in science and motivation to learn and complete assignments as compared to the other science curriculum they have used in the past.

Figure 2. Teacher and Student Perceptions of the Degree to which the Pearson Interactive Science Program Affected Student Engagement Relative to Other Science Program

Other Program Both helped PIS

helped about the same helped

more more

4.00

4.43

4.14

4.00

4.00

3.60

3.56

Engage students in learning about science. (Teachers)

Interest students in conducting hands-on activities or

lab w ork. (Teachers)

Increase your students’ interest in science. (Teachers)

Motivate students to complete homew ork assignments.

(Teachers)

Help students take pride in their science w ork.

(Teachers)

Motivation to learn more about science. (Students)

Interest in science. (Students)

♦ They participate more, they were

involved, they were focused, they

actually looked through the book before

we even started it and they looked

through the book and they were

pointing, and yeah, they were very into

the book. – NC School D Teacher

Theme: Teachers indicated that the primary reason for higher levels of

student engagement was because of students’ increased interaction with

the Pearson Interactive Science program materials, especially writing

in the worktext and the labs.

When asked for factors that may have

contributed to students’ higher level of engagement, teachers indicated that this was

primarily the result of students’ increased interaction with the material through writing in the worktext and the labs, see Figure 3. In particular, teachers noted the students’ proclivity to participate in science, draw pictures, and answer questions in class when using the Pearson Interactive Science program. Figure 3. Percent of Teachers and Students who Agree that the Pearson Interactive Science Content and Labs Facilitate Student Interest

100% 100% 100%

80%78%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

The content present in the

Worktext keeps students

interested in science class.

The lab zones integrated w ith

the Worktext keeps students

interested in science class.

My students enjoy reading their

science Worktext to learn.

(Teachers)

Teacher

Student

♦ …As the kids get more excited about it

[PIS], I get more excited about it, and

that makes me happy. If they’re having a

good day, then I’m having a good day. – NC School D Teacher

Theme: Due to the interactive nature of the Pearson Interactive Science program, students felt more pride and ownership over their science

work and learning.

As a result of students being able to write in the worktext and using it as their own personal learning journal, and the frequent interaction with the material both via interactivities and labs, teachers indicated that the students generally felt more pride in their work, had a better understanding of how well they were doing, and looked forward to doing science.


Figure 4. Percent who Agree to Looking Forward to Science and Taking Pride in Work While Using Pearson Interactive Science

100% 100% 100%

73%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I looked forw ard to science

class w hen w e w ere using

Pearson Middle Grades

Science.

My students took pride in their

science w ork. (Teachers)

Having a complete record of

their science w ork and learning

w as important to my students.

Teacher

Student

♦ I think all the activities in the worktext

and the fact that they get to write in it

and keep it, has really helped them feel

proud of their work. You can see it in

their worktexts—they use it everyday,

are able to jot down their notes, and you

can tell that some students really take

care of and protect their books. – NC School B Teacher

♦ I like that it’s yours and you get to write

on it. – NC Middle School D Student

♦ I like individual student workbooks--

students can take ownership of their

work. – NC School D Teacher

Theme: The Pearson Interactive Science program al lowed students to more easily apply science to real l ife.

Students noted the connection between the My Planet Diary sections and the content they were learning as being a strong factor in relating content knowledge to their everyday lives. Teachers also reported that their students could connect their science lessons to the real-world, as well as apply what they were learning to real life. They also felt that this had a positive effect on students’ understanding of science.

♦ It gives you a personal look at the

outside world, about what you’re

studying. – MA Middle School A Student

♦ I think they’re very able to relate it. I

mean, just from the responses I get from

the kids. Applying to real world

situations---that’s just been awesome. – NC School C Teacher

Teachers and students were asked via survey to compare how well the PIS program assisted them in making connections and applying science learning versus their other science program. As shown in Figure 5, teachers and students felt that the PIS program was more capable of facilitating connections and application to the real world.

Figure 5. Teacher and Student Perceptions of the Degree to which the Pearson Interactive Science Program Affected Science Connections and Applications Relative to Other Science Program


helped more about the same helped more

3.46

3.50

4.29

4.14

Application of science (Students)

Understanding how science is relevant to my

everyday life (Students)

Help students make connections betw een science

content and real-w orld applications. (Teachers)

Help students make connections betw een science

and other subject areas. (Teachers)

♦ I like the way it refers to something real

life, like the chart with the biggest

earthquakes in the last 2 years--the way

that you can say, “Oh yeah, I remember

that.” – MA Middle School A Student

♦ Love the real-world application, having

the pictures to refer back to, having

everyday meaning versus scientific

meaning. – NC School D Teacher


Theme: The Pearson Interactive Science program was more helpful in

developing students reading and writing skills, and problem-solving/

reasoning skills, as compared to other science programs they have

used.

Teachers indicated that, in addition to assisting students with their science understanding, PIS was also helpful in developing students reading and writing skills, and problem-solving/ reasoning skills. Indeed, survey data showed that they felt that they gained more skills in these areas from the PIS program as compared to their other science program. Figure 6. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Affected Other Academic Areas Relative to Other Science Program

Other Science Program Both helped PIS


4.00

3.86

4.14

Increase your students'

higher order cognitive

skills (problem solving,

critical thinking).

Influence students to

formulate questions about

science content.

Help students develop

reading/w riting skills

relevant to science.

♦ I like the fact that it’s using some

common language that the kids are

seeing in Language Arts and also here.

We’re talking about inferences. Our

Language Arts teacher, she looked at the

book, and she was like, “Great.” When

it talked about compare and contrast,

how it’s showing sequencing, she was

like, “great!” She was glad of the fact. – NC School D Teacher

♦ I like the PIS program a lot more

pictures, concepts, and reading (writing

skills) opportunities. It was helpful. – NC School B Teacher

Theme: The Pearson Interactive Science program al lows students and

teachers to easily monitor and assess comprehension of science

content.

Both teachers and students felt the program components were very effective in assisting students with understanding the material on an on going basis. Particularly, the components they felt help enhance student comprehension included:

� Writing in worktext--as a running record

of their learning, students and teachers could closely monitor their understanding.

� Activities and questions embedded within the worktext helped them assess their own understanding.

� Worktext facilitated note-taking so that they could review material as needed.

� The way in which material was revisited throughout the chapter allowed for continual review.

They also appreciated the Got It section; in particular, teachers commented that some students do not raise their hand or speak out when they don’t understand material but by providing an opportunity within the worktext for these students to show and express their level of understanding, this can be easily determined without the student feeling uncomfortable.

♦ It also asks questions throughout to

show that you’re learning. It’s like a

review, so you know it right away and

you can learn it faster. – MA Middle School A Student

♦ I like how if you understand it you have

to answer the questions, so if some of the


questions are blank, and if I don’t know

them then I know I have to read over so

that I know what I’m not getting or

understanding. – MA Middle School A Student

♦ They can write their answers, and then

when I ask for responses I know that

they’ve written something down. I love

“I Got It.” Did you get the concept or

did you not get it. And the kids can

check. And if they check no I can spend a

little more time with them on it the next

day, or go back and say, “let’s look at

this,” or get a peer. – NC School D Teacher

As shown in Figure 7, teachers also expressed that the PIS program was more helpful as compared to their other science program for identifying and assessing their students’ level of understanding both during and after science lessons. Figure 7. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped them Determine Student Understanding Relative to Other Science Program

Other Program Both helped PIS helped more

helped more about the same

4.57

4.43

3.86

Help you assess student

understanding of material

during the lesson.

Help you assess student

understanding of material after

the lesson.

I w as able to clearly identify

w hich students needed more

help in science.

♦ [PIS] gives students an opportunity for

self assessment reflections. – MA School A Teacher

♦ It is a change from since I transferred

schools and never really used a textbook

at all. It helps you understand science

better and lets you see where you are at

in science and you are not rushed either.

Its great! – NC Middle School D Student

Theme: The Pearson Interactive Science program helps students stay more organized as compared to other

science programs.

Students reported being better organized, in part because they were able to write in the book and did not have to copy things into separate note books. Consequently, they were better able to more easily keep track of their science notes.

♦ Helps us. A lot easier to go back over

and look at it, rather than having to go

back through your different sets of

papers that you wrote stuff on. – NC Middle School D Student

♦ I see it like a study guide: we actually

write all our notes in the book, rather

than on separate paper. You don’t lose

your notes. – NC Middle School D Student

This is further substantiated from survey

results. As shown in Figure 8, the majority of students agreed that the PIS program helped them keep track of their learning and to take good science notes, as well as helping them organize science information mentally.


Figure 8. Percent of Students who Agree that the Pearson Interactive Science Program Assists with Organization

77%

83%78%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

My Worktext helped me keep

good science notes.

My notes and w riting in the

Worktext helped me to keep

track of w hat I learned.

The organization of the Worktext

around Big Questions helped me

organize new information in my

mind.

Theme: The Pearson Interactive Science program helps students on tests and in future science courses.

On average, students also felt that the PIS

program did more in preparing them for state and class exams, and for future science classes, see Figure 9. Note that this finding is quite striking because students were only exposed to the program for 1-2 months.

Figure 9. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped them on Tests and Future Science Classes Relative to Other Science Program



4.29

4.17

4.29

Prepare your students to dowell on science class tests.

Prepare your students to do

well on state/national tests.

Prepare your students to do

well in future sciencecourses.

♦ My grade went up – I don’t know why – I

wasn’t doing good in science and then my

grade went up with this book and it was

easier. – NC Middle School D Student Teacher Practices and Preparation

The teachers noted that the program helped them with their pedagogical practices as well. All pilot teachers agreed that the Pearson Interactive Science program provided them with useful information to effectively teach science. Since the majority of teachers used an inquiry-based science program or used a blended approach, there was not a major departure from their prior approach to instruction in terms of the inclusion of inquiry-based activities. That said, the structure of their classes and of course, the materials used were different. As one teacher notes:

• The kids have their own work books, it’s

in smaller increments, it was almost like

a little warm-up, a mini-lesson, a little

activity—whether it’s math, or writing a

sentence—then we can do a quick lab,

summarize, and we’re done. Next day,

we’re on to something else. – NC School D Teacher

As previously noted, teachers reported that

the program helped them assess student


knowledge on an on going basis and helped them make connections between science and the real-world. In addition to this, they also noted that it saved them time as it relates to lesson preparation, helped them plan and execute lab activities, and helped them teach science vocabulary (see Figure 10).

Teachers noted that the Pearson Interactive Science program helped

them: a) save on time as it relates to lesson preparation, b) plan and

execute lab activities, and c) teach science vocabulary.

Figure 10. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped them with their Preparation and Practices Relative to Other Science Program



4.00

4.57

4.43

Minimize preparation andplanning time.

Provide you with good ideas

for hands-on scienceactivities.

Help you teach science

vocabulary.

• Now, I’m comparing it to the way I

normally teach. I usually use problem-

based learning: very open-ended sort-of

inquiry. So it’s had me to focus a lot

more than I normally focus. But they

seem to respond to this. – NC School C Teacher

Teachers also reported that it provided

greater assistance, as compared to their other science program, for individualizing instruction to students at all levels (low, average, and advanced). That said, it is interesting to note that teachers did not use the differentiated activities with great frequency. As such, this may be more due to teachers being able to gauge their students’ level of understanding more easily via the Assess Your Understanding/Got It sections,

and subsequently, providing more instruction on an individual basis as needed. Figure 11. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program Helped with Individualizing Instruction Relative to Other Science Program



4.57

4.43

4.29Help provide differentiatedinstruction to students of

average ability.

Help provide differentiated

instruction to students ofbelow-average ability

(remediation).

Help provide differentiatedinstruction to students of

above-average ability(challenge/enrichment).

Teachers also reported that Pearson Interactive Science program provided

greater assistance, as compared to their other science program, for

individualizing instruction to students at all levels.

In order to obtain more detailed information

on exactly what typical science activities may have changed (either by doing more or less), teachers and students were asked the extent to which they participated in specific classroom and instructional activities when they used PIS as compared to when they used their other science program (in other words, did they do less, about the same, or more of these activities). Figure 12 shows that for the most part, students felt they: a) completed their assignments on time; b) reviewed work in class; c) answered textbook questions; and d) worked with other students more while they used the PIS program. Teachers on the other hand, rated a number of other activities as being more prevalent during the time they used Pearson Interactive Science. These additional activities include: a) doing mathematics related to science; b) applying what they learned to real-world scenarios; c) students discussing science topics with each other; d) reading from worktexts; e) listening and taking notes during a presentation and


lecture; and f) explaining their answers in both written and oral form.

Figure 12. Teacher and Student Perceptions of the Degree to Which They Engaged in Science Class Activities During Their Use of Pearson Interactive Science Program

Did activities...



3.7

3.9

3.9

3.9

3.6

3.4

4.1

4.1

3.7

3.2

3.8

3.9

3.0

3.2

3.3

3.1

3.0

3.9

3.4

3.4

3.6

3.1

3.3

3.5

Students listened to a

science lecture.

Students explained in

writing their answers to

questions in science

class.

Students explained orally

their answers to

questions in science

class.

Students listened and

took notes during a

science presentation.

Students worked in

groups with other

students.

Students read from a

science

Worktext/textbook in

class.

Students answered

textbook or worksheet

questions.

Students discussed

science topics with other

students.

We reviewed science

work in class

Students applied what

they learned to other

real-world scenarios.

Students used

mathematics as a tool

when working on science

problems.

Students completed my

science homework

assignments on time.

Teachers (Blue) Students (White)

In addition, teachers and students were asked to also make comparisons in terms of labs and lab-related activities. As Figure 13 shows, both students and teachers agreed that they engaged in more hands-on/lab activities, followed specific instructions during these lab activities, and wrote explanations about observations made during labs, during their use of PIS as compared to when they used their other science program.

Figure 13. Teacher and Student Perceptions of the Degree to Which They Engaged in Lab Activities During Their Use of Pearson Interactive Science Program

Did activities...



3.7

3.3

3.4

3.3

3.0

3.7

3.3

3.4

3.7

3.2

3.0

3.7

3.2

3.3

Students watched a

science demonstration

by me.

Students did hands-

on/laboratory scienceactivities.

Students designed or

planned experiments.

Students came up with

hypotheses or

predictions to be tested

during an experiment.

Students followed

specific instructions in

lab activities or

investigations.

Students recorded,

represented, and/or

analyzed data.

Students wrote

explanations about

observations they made

during

activities/experiments.

Teachers (Blue) Students (White)

In terms of other teaching practices, both teachers and students also agreed that teachers spent more time teaching science vocabulary and introducing new science concepts via labs and exploring Big Questions, see Figure 14. Figure 14. Teacher and Student Perceptions of the Degree to Which Teacher Engaged in Other Activities During Their Use of Pearson Interactive Science Program

Did activities...


w hile using PIS same w hile using PIS

4.4

4.0

3.7

3.7

3.7

3.4

Teacher introduced

new science topics by

exploring big-picture

questions.

Teacher introduced

new science topics by

assigning a lab or

hands-on activity.

Teacher spent time

teaching new science

vocabulary terms.

Teachers (B lue) Students (White)


What do users of the Pearson Interactive Science program think about the program? What aspects of the program do they find most useful? Least useful? What, if any, suggestions for program improvement do they have?

Theme: Overall, teachers and students really enjoyed using the

Pearson Interactive Science program and would recommend it.

Overwhelmingly, teachers and students liked using the program. Overall, they felt the program as a whole was very engaging and well put together. They also would strongly recommend the program and some students indicated that they hoped to use it during the upcoming school year.

♦ I enjoyed it very, very much. I hope to

get to use the PIS program again. – NC Middle School D Student

♦ It is a wonderful program that should be

considered for our permanent science

program in school. – NC Middle School D Student

♦ They’ve been very positive. I don’t know

if it’s the novelty of it, or what, but they

have been most positive. They love the

activities. – NC School C Teacher

♦ Overall I felt the text was informative, it

covered the standards. It appealed to the

students which is a big factor. I

understand it is a work in progress, but

once you iron out the kinks, it will be a

success in the classroom. – NC School D Teacher

Student and teacher comments are also

supported from the survey data as well. As

shown in Figure 15, all pilot teachers and approximately 80% of students surveyed enjoyed using the PIS program and would have liked to have used it for the remainder of the school year. Figure 15. Percent of Teachers and Students who Liked the Pearson Interactive Science Program

100% 100%

77%80%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Overall, I liked the Pearson Middle Grade

Science materials w e have been using.

I w ould like to use the Pearson Middle Grades

Science program for the rest of the school year.

Teacher

Student

♦ Keep up the good work, this was the best

textbook I think I have ever used! – NC Middle School B Student

♦ I showed my mom, because when I first

got it I was really excited about all the

pictures and stuff. She thought it was

really good…She’s a principal, so she

was like, “Wow, this is really good!” – NC Middle School C Student

Students were also asked to rate the PIS

program according to specific adjectives. Specifically, students were asked to rate the program on a scale from boring to interesting, bad to good, difficult to easy, useless to useful, and not fun to fun. Figure 16 shows the results of these ratings. In general, students rated the program as somewhat interesting, good, easy, useful, and fun (averages are in right side of middle line).


Figure 16. Average Descriptive Ratings by Students on the Pearson Interactive Science Program

Comparisons between Pearson Interactive Science and Other Science Programs

As previously noted, three general types of science programs were used by the pilot teachers. These consisted of an inquiry-based program, the 2005 Science Explorer program used by the Massachusetts teacher with some supplementation, and a teacher-created program that drew upon multiple resources (used by one NC pilot teacher).

Interestingly, although these three curricula

were distinct, comments by students based on comparisons made between these programs and PIS were fairly consistent. When students were asked to compare PIS to science programs they used regularly during the school year, students definitely preferred the new Pearson Interactive Science program. In particular, students indicated that in comparison to their other science program, the PIS program was more entertaining, did not include extraneous content (i.e., “gets to the point”), was easy to understand and follow, provided more examples, contained more lab opportunities, was more interactive, and was portable (lightweight).

♦ The other books are filled with stuff we

don’t really need. – NC Student Using Inquiry-Based Program

♦ It’s [other science program] so boring

you don’t even focus in anymore, but

these [PIS], since they’re like simple to

understand and explains things—it keeps

you into the book. – NC Student Using Inquiry-Based Program

♦ In the old book, at the end of the section

there were like 20 to 40 questions that

we would have to answer, but in this

book every few paragraphs we have to

answer questions that way we don’t

forget anything. – MA Middle School Student

♦ You get to do a lot of labs instead of

doing just book work where you have to

sit down. – MA Middle School Student

♦ [This is] much more effective, more

interactive. – MA Middle School Student

♦ I like it more because it has fun things in

it. – NC Student Using Teacher-Created Science Program

Teachers were also asked to rate the Pearson

Interactive Science program relative to the science program used regularly throughout the school year. As shown in Figure 17, on average teachers rated the overall PIS program as much better in comparison to their other science program.

Figure 17. Teacher Rating of Preference for Pearson Interactive Science Program versus Other Science Program


Furthermore, as shown in Figure 18, on average, students preferred the PIS program in a number of areas as compared to their other science program. In particular, students rated the PIS program more favorably in the areas of: 1) format of their worktext; 2) science lab and investigations; 3) how science is explained; 4) presentation/ design of the student book; and 5) types of science exercises/questions. Figure 18. Student Perceptions of the Degree to which the Pearson Interactive Science Program is Better Relative to Other Science Program

Other Program About the Same PIS

Better Better

3.7

3.7

3.6

3.6

3.4

3.5

3.6

3.5

3.5

Format of your student

book (i.e., PMGS Worktext

v.s. textbook)

Science

labs/investigations

Types of science

exercises/questions

Overall

presentation/design of

your student book

Type of math included to

do science

Writing activities

How science is explained

How questions are

presented in science

materials

Ease of use for note-

taking

In general, students rated the format of their student book, labs,

explanation of science, presentation/ design of the Pearson Interactive

Science program, and types of science questions most favorably as

compared to their other science program.

Teachers also commented on how they enjoyed using Pearson Interactive Science as compared to their other science programs. As the following quotes illustrate, teachers clearly indicated preference for PIS. They felt that compared to their other science program, the PIS program was easier to use, facilitated a sense of ownership because students had their own worktexts they could write in, had what they needed to effectively teach science, and promoted better science understanding.

♦ I used the Prentice Hall for years, when

they had the little blue books. I loved

those, because you could just hand them

out and not have to carry a big book

around. I liked those, but I like this

better because of the way the kids can

write in it, and it’s so concise. I think

they’ve almost taken the same idea and

just improved it. – NC School C Teacher

♦ With the old books, the Prentice Hall

ones—those were almost at a level that

didn’t hit. So I always piecemeal

between things or found my own labs to

add in. So it’s nice having labs there I

can just pick up and have the worksheets

to go along with. – NC School D Teacher

♦ The program that we’re supposed to be

using is [inquiry-based science

program] and the books that we do use,

because I pull them in, because I feel

like kids need to read. They need to have

something to grasp onto. I like that a lot

[about PIS]--that they have some

reading and individual books that they

can have.– NC School D Teacher

♦ I like it [PIS], I do. It’s a little more of

the concepts, and the questions go a

little more closely aligned with the state

test. I have used [inquiry-based science

program] before and lab aids, and those


were at times, a little too broad, or they

got too narrow. I like the fact that it’s

going back and forth and it hits what it

needs to hit. And it uses the terms that it

needs to use—the vocabulary. With

[inquiry-based science program], it

would get to the concept, but it was too

broad of a concept, and the kids didn’t

always make the connection of why they

were doing what they were doing. – NC School D Teacher

A couple of teachers also commented how they liked how the PIS program seems to blend both a basal and inquiry-based approach to instruction.

♦ I’m thinking, this is not just a book,

it’s a book and activities, so it’s a

good blend between the two. Because

you can’t just have one…and you

can’t have that boring text. So I think

it’s a great blend. – NC School C Teacher

♦ There’s no, what I call, meat [in

describing inquiry-based science

program]. It’s all inquiry and I

understand that. But kids need the

basic knowledge before they can

jump into the higher level thinking,

which this [PIS] tends to do.– NC School D Teacher

The pilot teachers were also asked via

survey to compare their other science program to the PIS program in several areas6. As shown in Figure 19, in all areas surveyed, teachers showed a preference for the PIS program. The highest rated areas were: 1) presentation/design of teacher book; 2) presentation/design of student book; 3) format of the student book; 4) student ease of use; 5) writing activities; and 6)

6 Due to the limited number of teachers (7), comparisons are not made between the different types of science programs they regularly use.

types of math included. In sum, as compared to their regular programs, teachers showed a preference for the overall “look and feel” of the PIS program and also liked that it made connections to and emphasized other academic areas such as writing and math.

Figure 19. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program is Better Relative to Other Science Program


Better Better

4.4

4.6

4.3

4.3

3.9

4.6

4.0

4.1

4.6

4.9

4.7

4.1

4.3

4.9

Format of student book (i.e.,PMGS Worktext v.s. textbook)

Science labs/investigations

Types of scienceexercises/questions

Overall presentation/design ofstudent book

Overall presentation/design ofteacher book

Type of math included to doscience

Teacher ease of use

Writing activities

Amount of space available forwriting in student book

How science is explained

How questions are presented inscience materials

Student ease of use

Amount of planning/preparationrequired

Quality of resources provided

According to teachers, the highest rated features of the Pearson

Interactive Science program as compared to their other science

programs were the design and format of the Pearson Interactive Science program, including the writing and

math activities.


Feedback on Design and Features of the Pearson Interactive Science Program

Theme: Both teachers and students loved the ability to write in the

worktext.

The most favorably cited feature among

both teachers and students was the ability to write in their worktexts. As will be discussed and shown in the section on how students used their worktexts, students reported writing in the worktext to answer questions, take notes, draw pictures, construct graphs, and highlight vocabulary. More importantly, the majority of students indicated that they felt more invested in learning and learned more as a result of being able to interact with the material.

♦ I like that you can write in it. The old

textbooks you had to copy everything

down into your notebook. I like this one

a lot better because it saves time. You

can just say, “I know this” and write it

down. – MA Middle School A Student

♦ Basically, it really helps that you get to

fill it in in the book. I think that’s the

coolest feature. In the old book there

used to be: copy it down, and then you

wouldn’t because you would have to take

notes on everything. But in the book we

have now we can highlight something we

think is important, and the questions are

really close so we can answer, so it

really helps to learn it better. – MA Middle School A Student

As shown in Figure 20, 84% of students

surveyed liked being able to write in their worktexts, and 78% enjoyed having their worktext as their source for notes and learning. The majority of students (69%) also noted that they like to read from their worktext—which indicates that the text was interesting and engaging.

Figure 20. Percent of Students who Liked the Pearson Interactive Science Student Worktext

78%

69%

84%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I liked having the Worktext as

a single source for my

science notes and learning.

I liked reading from my

Worktext.

I liked being able to w rite in my

Worktext.

♦ [Students enjoy] the fact that they can

write in the book. They love the

interaction. – NC School B Teacher

♦ I really think it makes them interact with

it more. Because every time in reading,

and in math, you know when they take a

test we expect them to write in the book:

to underline it, to draw on the graphs. So

this is just carrying that over. Makes a

big difference. – NC School D Teacher

Theme: Although most teachers commented that the content and

readability of the program was at-grade level (i .e., geared towards on-

level students), there were some comments that the program was too

easy.

During interviews, teachers were asked about the readability of the PIS program. While most teachers agreed that the language used in the text is student-friendly and that the program seemed to be designed for “regular” students, some teachers wished it were more challenging for advanced students.

♦ It’s at a level that’s a little bit easier for

the kids. Not that it’s at a lower reading

level, but it’s broken up enough that the

kids can digest what they’ve gotten and


go onto the next concept part of it.– NC School D Teacher

♦ I think maybe the questions could be a

little more challenging, but this is an

advanced group, so, you know, in a

regular group, I think it would be

perfect. – NC School C Teacher

The vast majority of students felt that this science program, in comparison to those they had used before, was easier to understand because of the reading level and vocabulary used. However, like some of the teachers, other students also pointed out that it was too easy and would have liked more of a challenge. This included the labs. That said, a few students commented that at times the text and questions were difficult.

♦ It was like they took it and they put it in

our kind of language so we could

understand more. – NC Middle School D Student

♦ I liked everything because it told us

exactly, straight out what we needed to

know. And it showed us that in a way

better because you can write in it. – NC Middle School D Student

♦ I wish that the book would challenge me

more. – NC Middle School D Student

♦ Well sometimes I don’t get it, because it

doesn’t explain it sometimes very well.

Like I have to read it a couple more

times to get it. – MA Middle School A Student

Theme: Overall, teachers who used the Pearson Interactive Science

program noted that the program was easy to use.

Teachers commented that the program was very straightforward and everything they needed was contained in the program. Teachers also noted how little planning time it involved. They could quickly jump into the lesson.

♦ Well, I feel better going home at night,

knowing that I’ve got it all laid out for

me. If I was out…I know that whoever

came in could just pick this up and

go…this is very, would be very easy…It

would be easy to have this for a

curriculum to come in and pick up. – NC School D Teacher

♦ I think it’s shorter for my planning time.

I really do. I think it was, because it’s

all there. I had to put everything in my

lesson book, but everything is right

there. And the kids know exactly where

everything is in the book, and we just

did our thing. – NC School D Teacher

Such qualitative feedback is supported by survey data. As shown in Figure 21, the Pearson Interactive Science program was rated by teachers as more teacher-friendly, and reduced the amount of planning and preparation time as compared to their other science program. Figure 21. Teacher Perceptions of the Degree to which the Pearson Interactive Science Program is Better in terms of Ease of Use Relative to Other Science Program


Better Better

4.0

3.9

4.0

Teacher ease of use

Amount of

planning/preparation

required

Minimize preparation

and planning time.


♦ There was no having to dig through the

book and having to find questions; you

guys had them. It was easy to open up

and start going and just reading

through. – NC School D Teacher

Theme: Overall, teachers felt that pacing was appropriate or easy to

modify.

The majority of teachers and students also felt that the Pearson Interactive Science chapters they piloted were appropriately paced—not too long and not too short. Furthermore, when teachers felt that more or less detail was needed, they found it easy to supplement (e.g., doing more labs, expanding on concepts, providing additional worksheets, etc.). Indeed, 6 out of 7 pilot teachers indicated that the program did a better job at pacing than their other science program.

♦ Well, at first I thought it was too fast,

because of the prior knowledge. But I

think you can tweak it, as the teacher,

either way, to whatever your kids need

[sic]. – NC School D Teacher

♦ When you really focus on everything, it’s

not too long and not too short at the

same time, because you have stuff to do.

You’re not just reading it really quick

and then you’re done. There are

questions and pictures at the bottom

where you have to match stuff up, so it

gives you something to do and it helps

you study and it’s fun. – MA Middle School A Student

Theme: Teachers and students loved the layout of the program (i.e.,

design and organization).

The structure of the program, including how it was organized and formatted was also

rated highly by teachers and students. As previously noted, students felt that they were more organized because the worktext allowed them to easily keep notes and was a personal journal of their learning. Other elements of the worktext noted by teachers and students as being important included the highlighting of vocabulary, Keys (bolding of key concepts), and the activities/questions embedded in the worktext. Furthermore, all teachers liked how the worktext was organized around Big Questions—according to the teachers, this made their teaching more seamless as the goal of each chapter was clearly stated and revisited throughout.

♦ I like the way it’s organized with like the

bolded, bright words, so you can really

just study that, or you know just to read

that or something like that. You don’t

have to scan the entire page to find the

one sentence that you need. – NC Middle School C Student

♦ My favorite part is the EQ (essential

questions). You answer the questions

with a key, you don’t have to look for it

too much. – NC Middle School C Student

♦ I like that [the organization of the program]. I really do. Every one has a

graph. It’s sort of using what you know.

Also I like the spiraling that they’re

using… It’s sort of like they take an

example, and they’ll talk about the

example in one way, and they’ll use it

later on. So it spirals beautifully. – NC School C Teacher

♦ [Re: Organization of program:] I like it.

I love it. Excellent. And the little key

points—they know that that’s the

important facts. And the graphic

organizers… it’s just awesome. That’s

part of our Learning Focus-- to do


graphic organizers, so this fits our

curriculum and the programs that our

school system has said that they want us

to do. – NC School D Teacher

Theme: Teachers and students also thought the il lustrations in the text were visually-appealing and helped

promote student interest and learning.

Another important feature of the Pearson

Interactive Science program that was oft-cited was the illustrations. Teachers and students alike felt that they enhanced their learning by showing students important science concepts (and not simply explaining it in text), see Figure 22. They also served as a memory aid. That is, students commented that associating science concepts with certain pictures helped them with recall.

♦ I think for a lot of people, pictures help.

Like, it just really helps you remember

what you were talking about if you can

remember the picture. And so I like the

pictures. – NC Middle School C Student

♦ It was very colorful. It attracted their

eyes, and they could put the reading with

the pictures, and they could follow

along. And then how you made things

some larger some things smaller, and

they understood. – NC School D Teacher

Figure 22. Percent of Teachers and Students who Liked the Pearson Interactive Science Illustrations

100%

82%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

The diagrams and visual representations in the student book help students to learn science

Teacher

Student

♦ They love the pictures. That is a strong

point. They love the pictures.—NC School C Teacher

♦ I’m just amazed at how, in textbooks,

lots of times kids are bored. They don’t

even look at the pictures. And so, but

they have really been focused on the

pictures. I am really surprised with that. – NC School D Teacher

Theme: Teachers and students find the labs fun and engaging. The labs relate to the content in the book and

enhance comprehension.

In general, teachers and students enjoyed using the PIS labs for the chapters that were piloted. They noted that most labs were engaging and helped students obtain a better understanding of the science content being taught by providing a hands-on opportunity to explore the concepts. As shown in Figure 23, over 80% of students and 100% of teachers who participated in the pilot study liked the labs and investigations.

♦ They [labs] helped me understand what

we were learning better. You got to see

and then do it. And also read about it too.

– NC Middle School D Student


♦ You get to see what you’re studying.

When you’re reading you’re not really

seeing, you’re just imagining it, but when

you’re doing you get a lot more. – MA Middle School A Student

Figure 23. Percent of Teachers and Students who Liked the Pearson Interactive Science Labs

100%

81%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I liked the labs and investigations w e did from the Pearson Middle Grades Science program.

Teacher

Student

Teachers were also asked to rate the different types of labs on a scale from 1=did not like to 5=liked very much. Results showed that the Quick Labs were rated most positively, followed by the Inquiry Warm Up Labs, see Figure 24. The Lab Investigations were regarded less favorably, with the exception of the Populations and Communities investigation. Indeed, across the different chapter labs, the Populations and Communities’ labs were rated highest, followed by the Earthquake and, Work and Machines’ labs. Figure 24. Teacher Ratings of the Pearson Interactive Science Labs by Type

44.2

2.3

3

3.7

2.7

2

4.7 4.74.5

1

2

3

4

5

Inquiry Warm Up Quick Lab Directed Inquiry Lab

Investigation

Open Inquiry Lab

Investigation

Sca

le 1

=D

id N

ot

Lik

e t

o 5

=L

ike

d V

ery

Mu

ch

Earthquakes Work & Machines Populations & Communities

Teachers also felt the labs were easy to implement. All the resources needed to complete the labs were accessible and easy to use. They also liked how the labs were organized and referred to within their Teacher’s Edition at the appropriate instructional point—as one teacher noted “every activity is just

placed exactly right.”

♦ I love them. I love it. And it’s all

together; it’s there in their little packet

box, ready to go. It’s very easy to use:

just grab it and organize it and put it

out. – NC School D Teacher

♦ I do like the lab activities, because

they’re to the point, and they relate-- the

kids love to do labs, they love hands on.

It doesn’t matter what they do, they just

love hands-on. – NC School D Teacher

Theme: Some users of the labs thought the directions were vague or

confusing.

While the vast majority of students commented that they enjoyed the labs, there were a few students who noted that the labs were at times boring, confusing, and/or difficult. Similarly, although teachers appreciated how seamless most labs fit in with the lessons and students noted they loved participating in investigation activities frequently, teachers noted that the directions were sometimes difficult to understand and would prefer a more guided approach to the setup.

♦ They can be a little bit confusing…So it

maybe needs to be cleared up a little bit. – NC School C Teacher

♦ I feel like the instructions are made for

adults, not for kids. The directions are

not kid-oriented. – MA Middle School A Student


Table 3. Ratings of Usefulness of the Pearson Interactive Science Program by Pilot Teachers

Pearson Interactive Science Components Min. Max. Mean*

Big Question at the beginning of a new topic. 5 5 5.0

Lab Zone embedded in the Worktext. 5 5 5.0

Content or material present in the Worktext. 5 5 5.0

Lab Zones in the Worktext to reinforce your instruction or lecture. 5 5 5.0

Figures/Activity Art/Animations in Worktext to reinforce instruction. 5 5 5.0

After the Inquiry Warm-Up worksheet 5 5 5.0

Lesson quizzes 5 5 5.0

Key Concepts summaries 5 5 5.0

Assess Your Understanding sections of the Worktext. 4 5 4.7

Vocabulary section of the Worktext. 4 5 4.7

Apply It! activity sections in the Worktext. 4 5 4.7

“Do The Math” activity sections in the Worktext. 4 5 4.7

Understanding by Design (i.e., 5Es of Engage, Explore, Explain, Elaborate, Evaluate). 4 5 4.5

My Planet Diary. 4 5 4.5

“Explore the Big Question” activity sections in the Worktext. 4 5 4.3

Differentiated Instruction activities noted in your TE. 3 5 4.3

Study Guide 4 4 4.0

References to the Big Question throughout the chapter. 4 4 4.0

Check Your Understanding sections of the chapter. 3 5 4.0

21st Century Skills sections of your TE to reinforce these skills. 3 5 4.0

Differentiated-activities worksheets (Review & Reinforce and Enrich) 2 5 3.5

NOTE: Based on data from 7 pilot teachers. *Based on scale from 1=not at all useful to 5=very useful.

♦ It was confusing. Like they gave you the

first step and the last step but nothing in-

between. – MA Middle School A Student

Feedback on Other Specific Components of the Pearson Interactive Science Program

Table 3 presents results from teacher ratings of the individual components of the Pearson Interactive Science program. Teachers were asked to rate each component on a scale from 1=not at all useful to 5=very useful. As shown, with the exception of one item, all components of the PIS program were, on average, rated as useful. Only one teacher rated the Differentiated Activities worksheets as not useful.

♦ We do the … Planet Diaries that are

really fun, because it has facts in it that

are interesting. – NC Middle School C Student

♦ The vocabulary is a lot more

appropriate, a lot easier to understand. – NC Middle School D Student

♦ I like the Got It thing at the bottom,

because you can really check if you got

what it was talking about in the chapter,

and then you can ask the teacher if

there’s something you didn’t quite

understand, or something like that. – NC Middle School D Student

♦ I think the Fun Facts are fun, because

this kind of relates, it gets them to do the

higher level thinking. – NC School D Teacher

♦ I love I Got It, because that kind of tells

that this is what the lesson was about. I

love that. – NC School D Teacher

♦ I liked this: the everyday meaning and

scientific meaning. The kids kind of got

that. – NC School D Teacher

♦ I love how they, because I’m also a math

teacher, I love how they put the math in

there... – NC School C Teacher


Recommendations

Teachers and students were also asked to provide feedback with respect to areas of the program that could be improved. The following provides a summary of their recommendations for improvement of the PIS program. Student Recommendations

♦ Directions for labs need to be more student-friendly. � The steps of the activities are way too

long and sometimes the steps are kind of

complicated. – MA Middle School A Student

♦ A glossary is needed.

♦ Provide better labels on the figures. � I have a problem with the pictures, in the

reading it’s like, “look at figure 5” let’s

say, and there’s all these pictures and

figure 5 is over here and they don’t use

enough descriptions so it makes it more

confusing. – MA Middle School A Student

Teacher Recommendations

♦ Recommends having the Teacher’s Notes appear closer or adjacent to the lessons. � Put all teacher notes and materials with

the lesson. Teachers tend to look on the

page where the lesson is being taught

only. They do not flip back and forth.

The answers were very clear to

questions. – NC School D Teacher

♦ Allow for greater spaces for writing. � Maybe in the book, if they had a few

blank sheets of paper, like a “Notes”

area. Where if I had them say, Lesson 1

extension: Write your two sentences.

And then I can pick it up quickly and

maybe do a little check. I’m not sure if

you have any rubrics? – NC School D Teacher

♦ Would like more challenging activities for more advanced students.

♦ Provide a separate Lab book. � If we ever do get it, I would have a lab

book separate, instead of having to make

all these copies, but we can do that at

the beginning of the year, knowing that

stuff. – NC School D Teacher

♦ Include more math. � I’d like to see even more Do the Math.

Having, maybe some extra sheets to

reinforce, like the enrich ones. Having

just a few more…tying in together the

science and math and integrating in any

way possible to help out the math

teachers. – NC School D Teacher

♦ Do more organization of the TE. For example, add page numbers for Quick Lab in TE so that it is easier to find (and reference these in TE lessons). � Note on which page the labs are located.

– NC School D Teacher � Put the inquiry-warm up, lab sheets, etc.

in order of the teacher book…makes it

easier to reference. – NC School D Teacher

♦ Correct errors and typos. � In some parts, TE was worded different

than the students. Also, the lab

instructions were vague for both teacher

and student. – NC School B Teacher

♦ Provide graphic organizers for the students to use.

♦ Provide multiple forms of assessment/ quizzes.

♦ Estimated time noted on lab instructions need to be more realistic. In particular, Quick Labs and Inquiry Warm Up Labs took longer than noted. This is discussed in more detail in the following section.


How do teachers use the Pearson Interactive Science program in their classrooms? Do teachers of differing pedagogical approaches and philosophies implement the program differently across science classrooms?

In general, teachers followed their Teacher’s Edition closely. Teachers estimated they got through 2-3 worktext pages per day and typically included completion of a referred lab (e.g., Quick Lab), as participating teachers attempted most of the Inquiry/Warm-Up and Quick Labs. Therefore, depending on where they were at within a lesson, the lesson began with a class discussion (e.g., of My Planet Diary, review of previously read material), an activity (e.g., Do the Math or Apply It) or reading from the Student Worktext. Teachers would either have students read from their worktexts silently or round-robin. During the readings, the teacher would point out vocabulary words that appeared in the text and generally, asked the questions noted in their TEs under the 5Es. Indeed, the teachers were quite adept at following the elements of Understanding by Design for each chapter taught, which included introducing, exploring, reviewing and applying the “Big Question.” Furthermore, the vast majority of activities embedded within the worktext (e.g., Apply It!, Assess Your Understanding, Do the Math, etc.) were completed by students (individually or with partners) as they appeared in the worktext. Most teachers also encouraged students to make notes or mark their worktext (e.g., highlighting or underlining) as they came across important information. Following the reading and activities in the Student Worktext, a lab might be completed. Depending on the complexity of the lab, these took anywhere from 20 to 45 minutes. If time was available following lab clean-up, a class discussion would ensue about what they had learned during the lab.

These observations are further supported

from the quantitative data obtained from teachers. As Table 4 on the following page shows, the majority of pilot teachers tended to do PIS activities on a daily to weekly basis as directed by their TE. That is, the majority of teachers engaged in most PIS activities with appropriate frequency. The activities that were least frequently used consisted of the Differentiated Instruction activities/worksheets and 21st Century Skills section in the TE7. With regard to the former, teachers indicated that for the most part, their students understood the material and that there was no need for remediation. While several teachers indicated that the content was at times easy, teachers rarely used the enrichment materials.

Of note is that teachers of different

pedagogies (inquiry and mixed instructional approach) did not show any significant differences in the way they implemented their PIS lessons. That is, regardless of their instructional philosophy, teachers used the PIS program comparably. As shown in Figure 25, teachers engaged in various instructional practices with a similar amount of frequency (note that 14% reflects one teacher out of seven). For the most part, teachers allowed students to work collaboratively during labs, encouraged students to help each other, and provided students with opportunities to explain their reasoning on an almost daily basis.

7 In addition, although the After Warm Up Lab worksheet is noted as a component of the program, these would not necessarily need to be done with great frequency and the teachers considered these more optional.


Table 4. Frequency of Use of Pearson Interactive Science Components

Never-Rarely

Sometimes (1-2 times per

chapter) Often (1-3

times/week) Almost to Everyday

Often-Everyday

Introduce the Big Question at the beginning of a new topic. 14% 0% 43% 43% 86%

Use embedded Lab Zones in the Worktext to reinforce your instruction or lecture. 0% 14% 43% 43% 86%

Use the lesson quizzes 0% 14% 72% 14% 86%

Reference the Big Question throughout the chapter. 0% 14% 86% 0% 86% Review answers to Assess Your Understanding sections of the Worktext. 0% 14% 57% 29% 86%

Begin each lesson with My Planet Diary. 0% 17% 33% 50% 83%

Introduce a new topic or lesson by using a Lab Zone embedded in the Worktext. 14% 14% 0% 72% 72%

Plan your lessons according to Understanding by Design (i.e., 5Es of Engage, Explore, Explain, Elaborate, Evaluate). 14% 14% 14% 58% 72% Use Figures/Activity Art/Animations in Worktext to reinforce instruction. 14% 14% 14% 58% 72%

Assign the Apply It! activity sections in the Worktext. 0% 29% 0% 71% 71%

Review the Vocabulary section of the Worktext. 0% 29% 14% 57% 71% Assign the “Do The Math” activity sections in the Worktext. 0% 29% 14% 57% 71%

Have students complete the Check Your Understanding sections of the chapter. 0% 29% 29% 42% 71% Assign the “Explore the Big Question” activity sections in the Worktext. 0% 29% 42% 29% 71%

Use the Study Guide 0% 29% 42% 29% 71%

Use the Key Concepts summaries 0% 29% 57% 14% 71%

Use the After the Inquiry Warm-Up worksheet 0% 57% 29% 14% 43%

Use the 21st Century Skills sections of your TE to reinforce these skills. 0% 66% 17% 17% 34%

Use the differentiated-activities worksheets (Review & Reinforce and Enrich) 17% 49% 17% 17% 34%

Use the Differentiated Instruction activities noted in your TE. 29% 42% 29% 0% 29%

NOTE: Percents exclude missing data (e.g., 1 teacher not responding to an item).

Figure 25. Frequency of Teacher Instructional Practices

86%

29%

86% 86%

14% 14%14%

0%

71%

0%0%0%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I allow ed students to

w ork collaboratively on

Lab Zones activities.

I allow ed students to

explore a topic of

interest in their ow n

w ay.

I encouraged students

to help each other

learn the material.

I provided students the

opportunity to explain

their thinking/answ er.

All the time to Often (1-2 times per w eek) Sometimes (1-2 times per chapter) Rarely to Not at All

In sum, participating teachers closely followed the TE and attempted the vast majority of activities (including labs) that were noted in their TEs. That said, teachers reported that they used the Differentiated Instruction, 21st Century Skills, and Remediate sections of their TEs with less frequency and more on an as needed basis. Such information will be helpful in the development and piloting of implementation guidelines to be used for the larger scale pilot study in Spring 2010.


Table 5. Average Number of Labs and Time to Complete Labs and Chapters During Pilot Study

Earthquakes Work & Machines Populations & Communities

Noted in Program Piloted



Inquiry Warm Up Lab: Count 3 3 4 3 4 4

Time 10 to 15 minutes

26 minutes average

10 to 20 minutes 45 minutes


Quick Lab: Count 7 6 8 5 8 6

Time 10 to 20 minutes 36 minutes



Lab Investigations: Count 2 1 2 2 2 1

Time 40 to 60 minutes 68 minutes


130 minutes

120 minutes

Completion of Chapter -- 4 weeks -- 4 weeks -- 3.6 weeks

Detailed Information on Lab Implementation

As previously noted, most teachers aimed to have students complete all of the labs that were available in the piloted chapters. Indeed, researchers and Pearson representatives encouraged teachers to use as many labs as they could fit into their schedule so as to better inform time estimates, ratings of labs, and qualitative feedback from both teachers and students. For this reason, along with the fact that labs took longer than projected, teachers took more time to complete chapters.

Table 5 shows detailed information on the

average number of labs teachers completed during the pilot study. The average amount of time it took them to complete the various types of labs and chapters are also noted. In addition, for reference, the numbers of labs and estimated time to complete (as noted in lab instructions) are also included (see blue-colored columns). As shown, nearly all teachers attempted the majority of labs and in most cases the time it took them to complete the labs was greater, and for some significantly so, than what was estimated. Indeed, this concurs with the feedback that was obtained from teachers with

regard to lab pacing. Teachers thought the time estimates were not realistic and that getting through lessons was at times difficult to maintain as a result. It should be noted that this had implications for the duration of the pilot study; it took teachers longer than the 3 weeks that was initially estimated to complete a chapter. Instead, teachers took on average about 4 weeks to complete each chapter.

Although overall pacing was considered fine by teachers, they

indicated that the time estimates for the labs were not always accurate

and that it took them longer to complete labs. As a result, it also

took more time to complete chapters.

♦ The Quick Labs aren’t too quick, as

you saw today, 20 minutes is not 20

minutes… [The labs take] 35 to 40.

I’ve gone to expect that now with the

labs. – NC School B Teacher

♦ The individual labs and the activities.

They took a little bit longer than what’s

anticipated. – NC School D Teacher


Students Usage of Worktexts

Students write in the worktext in different ways. They take notes,

answer questions, draw pictures, draw arrows, underline, and

highlight.

Researchers reviewed the worktexts that were used by students to obtain information on how students used them. While there was variation in how students used their worktexts according to teachers (e.g., some teachers graded worktexts and therefore, there was greater completion of the activities within the worktext), it was also clear that all teachers encouraged students to write in their worktexts.

♦ I have seen them do both: [writing and]

underlining. Especially if I tell them to,

you know, they’ll do it. – NC School D Teacher

♦ They’re using them. They’re not just

looking at the lines. They can actually

point arrows, underline, highlight

whatever they need to highlight. So they

enjoy that. I think they do. They say they

do. – NC School D Teacher

♦ We read the explanations, then we talk

about the answers, then we come up with

our own answers and write them down. – NC Middle School D Student

While most of the writing/drawing was

concentrated on the activities and questions noted in the worktext, some students also made other notations such as underlining or circling key phrases or words, adding “study” notes, noting sections with astericks *, and highlighting. In all, it is estimated that these activities (underlining or highlighting) were centered on vocabulary definitions and were done 4-5 times per chapter. There was also the occasional doodling, but for the most part they

focused their writing to those sections where they were asked to respond to questions. Quite a few of the worktexts also had student names written on them (written by student) and had personal statements (e.g., Duke University is #1), which are indicative of students feeling a sense of ownership and possession of the worktext (i.e., it is their own).

Some students also got very involved in their drawings—some of which were quite elaborate, using different kinds of ink or being very detailed in their drawings. Some of the writing activities (e.g., writing a letter to editor in the Populations and Communications chapter about Pandas) also allowed students to show their creativity. Teachers thought these activities were very important and helped tie science to other skill areas and standards (e.g., writing / reading / math).

As previously noted, some teachers graded the completion of worktexts—though they did not necessarily grade them for correctness. Thus, a lot of the writing and activities completed in the worktext are likely attributable to teachers making this a requirement. That said, anecdotal information obtained from interviews indicated that teachers would walk around and check the Assess Your Understanding sections as students complete these, and try to address those areas that students did not get.

The following pages include snapshots of worktexts to illustrate the type of writing that students did in their worktexts.


As shown, students made notations on their worktexts, such as study

notes and highlighting key phrases to aid studying and understanding

of science concepts.


Some students would write their names, indicating a

sense of ownership.

Teachers emphasized completion of the writing

activities within the worktext, such as Assess Your Understanding and Apply It! Students found the ability to write in their worktexts very helpful for their learning of science.


How should the Pearson Interactive Science program best be used in order to maximize its impact on student performance?

Results from the pilot study indicate that teachers can and will likely use the program as directed in the Teacher’s Edition. Recall that teachers were allowed to use those components that they felt would positively effect their students’ learning of science. There were no prescribed guidelines provided. Even so, teachers used the vast majority of components of the program. In particular, teachers tended to engage in the following activities several times per week:

♦ Introduce the Big Question at the beginning of a new topic

♦ Begin each lesson with My Planet Diary

♦ Use embedded Lab Zones in the Worktext to reinforce instruction or lecture

♦ Use the lesson quizzes

♦ Reference the Big Question throughout the chapter

♦ Review answers to Assess Your Understanding sections of the Worktext

♦ Plan your lessons according to Understanding by Design (i.e., 5Es of Engage, Explore, Explain, Elaborate, and Evaluate).

♦ Use Figures/Activity Art/Animations in Worktext to reinforce instruction

♦ Assign the Apply It! activity sections in the Worktext

♦ Review the Vocabulary section of the Worktext

♦ Assign the “Do The Math” activity sections in the Worktext

♦ Have students complete the Check Your Understanding sections of the chapter

♦ Assign the “Explore the Big Question” activity sections in the Worktext

♦ Use the Study Guide

♦ Use the Key Concepts summaries

In addition, teachers were asked to identify the key program components that they felt were essential to teaching students effectively about science. Specific key components noted by teachers included vocabulary, labs, questions/activities in student worktext, and the questions under the 5Es as noted in the Teacher’s Edition. The following are comments teachers made with respect to the key components of the program.

♦ I exactly followed the program. I was

very careful about that. If you skip stuff,

it wouldn’t make sense. – NC School D Teacher

♦ The vocabulary, I would say. The

activities…I would say the quick labs.

And then the vocabulary in the text. And

the examples. – NC School C Teacher

♦ I think the top 3 parts of the program are

My Planet Diary, guided reading

questions (5Es), and vocabulary. – NC School B Teacher

♦ I wouldn’t [eliminate labs or activities],

you gotta do ‘em! I mean, that’s what

gets the kids involved!...I wouldn’t cut

them out. They’re important. – NC School D Teacher

♦ I think that’s important for students to

answer all the questions [in worktext]. – NC School D Teacher

♦ I definitely would not skip the Apply It

because that has a lot to do with the

math, and the math and science goes

together. – NC School D Teacher

Although teachers felt that the labs were essential to student learning, requiring completion of all the labs during future research


studies may negatively impact pacing as teachers may not get through much of the science program. Therefore, while it is suggested that in the Spring 2010 pilot study, teachers engage in the activities as directed in the TE (see listing on prior page), narrower guidelines should be provided to teachers with respect to labs so that teachers are able to cover as many topic areas as they can while still providing engaging, hands-on lab opportunities. In addition, if differentiated instruction activities/worksheets and 21st Century skill sections are deemed essential to student science performance, then there will need to be greater emphasis on these activities during the training and within the implementation guidelines. Recall that teachers did not engage in these activities with much frequency. To conclude, during the upcoming pilot study, researchers can more thoroughly examine the feasibility of the implementation guidelines and to the extent variation occurs, we can also examine whether differences in how they use the program affects outcomes.

During future studies it is suggested that teachers engage in key Pearson Interactive Science components as directed in the TE with an emphasis

placed on providing as many engaging, hands-on lab opportunities

as possible while stil l covering a profusion of topics.

What type(s) of training and preparation is needed in order to promote effective implementation of the Pearson Interactive Science program? Are the built-in teacher resources useful to teachers in helping them prepare to effectively deliver science instruction in their classrooms?

In order to help ensure that participating

pilot teachers could use the PIS program, teachers met with representatives from the Pearson Science team. Trainings were conducted with pilot teachers in a North Carolina school (approximately one hour8) and at the Massachusetts pilot school (one hour and 20 minutes (though trainings at both sites were scheduled for three hours). The training sessions were primarily the same between the North Carolina and Massachusetts sites, with one exception. Because there was more time available for the training session in Massachusetts the Pearson representatives were able to include a slightly larger overview of the Lab Resources.

Overall, the trainings9 consisted of Pearson representatives walking the teachers through the various program components and pointing out what they viewed as highlights of the program. Since one goal of the pilot was to determine how teachers would use the program on their own (i.e., without specific guidelines), trainings did not provide any type of specific training related to how to implement the program nor did they convey “must use” components. The

8 Several teachers left after 20 minutes due to miscommunication between school liaison and teachers about the length of the training. 9 There was a follow up Webex training conducted on March 9th for one of the North Carolina teacher who could not attend the initial training. This was scheduled for two hours and was conducted by Pearson representatives. Though scheduled for two hours the Webex took one hour and was similar in format to the actual training, consisting of a material walk through and overview of the key elements of the program.


training began with an overview of the Teacher Edition and Student Worktexts. The representative went through these materials and pointed out content while teachers followed along. This included an introduction of the program which emphasized the use of “The Big Question” chapter opener as a way to get students thinking about what they’d be learning. The Pearson representatives emphasized that the program was based on Understanding by Design and the five E’s and that it was intended to be a more interactive program that gets students engaged, thinking and reflecting on science lessons and related topics. As they walked teachers through the Teacher and Student Editions they pointed out all the places where students had been given a place to write in the book and how the program really tried to get students to “read, reflect and develop an opinion” about what they were learning.

The Pearson representative then primarily walked teachers through the Getting Started, lesson, and labs in the teacher and student worktexts. This was followed by general comments related to various parts of the program they found especially useful or interesting. For example the representative pointed out that there is “reading and doing on every page” making the program “more interactive”, Blackline Masters could be easily copied and that the worktext was designed to “speak right to” the students. They also stressed the importance of doing as many of the labs as possible, but did not provide training specific to lab implementation.

In sum, while the Pearson representatives

provided a show and tell of the PIS program, a more formal training was not provided so that teachers would have the flexibility to implement the program as they wished. That said, it will be important for future studies of the PIS program to include a more structured, formal training so as to ensure that teachers use the program in a way that will optimally benefit students.

Feedback on Resources and Training

Theme: While teachers felt that they had sufficient resources to teach

with the Pearson Interactive Science program, they felt the training could

have been stronger.

Teachers felt that they were supplied with sufficient resources to effectively implement the PIS program. As previously noted, they were quickly able to learn how to use the program, and the lessons and labs were noted as being easy to implement. Due to the more limited amount of training provided to the North Carolina teachers, these teachers felt that the training was lacking as they were unable to go over all the materials in much detail. That said, they also indicated that the TE was extremely helpful and facilitated their implementation of the science program through self-study (see Figure 26). A few teachers commented that the training would have been more helpful if it included a lab and lesson demonstration.

♦ I wish it had a sink! I’m very well

supplied. Yes [the supplies are]

sufficient. Very much sufficient. – NC School D Teacher

♦ [With PIS] you have consumables, non-

consumables, everything was there for

us, and I didn’t have to go out and get

anything, some plants, but besides that

the company sent them. – NC School D Teacher

♦ Our training was cut short because of a

miscommunication with other teachers.

[Would also suggest to] do the labs--have

them demonstrated. – NC School D Teacher


Figure 26. Percent of Teachers Who Felt Training and Teacher’s Edition were Helpful

86%

57%

100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

The training I received prepared

me to implement the Pearson

Middle Grade Science program.

The training I received covered

all aspects of the Pearson

Middle Grade Science program I

used during the study.

The Teacher Edition Worktext

w as a helpful resource w hen I

had questions or concerns

about implementation.

Recommendations for Future Training

In order for teachers to effectively implement the PIS program in future research studies, it will be critical to more specifically define what components of the program are essential and what are superfluous. This information is important to the development of implementation parameters which will need to be clearly communicated to teachers to ensure that teachers implement the program with fidelity.

The following are items that are

recommended to be incorporated in trainings associated with future research studies.

♦ A training model should be developed to promote consistency in trainings across different trainers. The model should be clearly defined and articulated, and should include follow-up sessions to help facilitate successful use of the program as well as implementation fidelity. Trainings should also make use of all the scheduled time.

♦ Training should include a thorough discussion of the program’s research base and philosophy. It would be helpful for teachers to more fully understand the background related to why/how Pearson created this program in order to obtain a greater buy-in from them; if a teacher understands why the program was designed

the way it is, they are more likely to emphasize important components and direct more attention towards implementing the program as prescribed. In addition, this will help set the stage for how the program works to improve student science skills.

♦ Teachers would also benefit from having a general overview of all the teaching resources available as part of the Pearson Interactive Science program, where to find them and how to use them. Trainers should walk teachers through the typical instruction of a chapter while being open to modifications according to teachers’ needs or preferences.

♦ Teachers also recommended that trainings conducted involve demonstrations of ways to utilize the components designated as essential to implementation. For example, modeling a PIS lesson was noted by a teacher as useful for future trainings. Similarly, since labs are an important part of the program, future trainings could include specific training where teachers are walked through an actual lab and shown all the available lab resources and where to find them.

♦ To the extent that it is feasible, the initial training should occur at least 2 weeks prior to the start of the study and teachers should have access to the materials at least one month prior to this training so that they can review the materials.

♦ It will also be important to plan for training on the technology to be released as part of the PIS program. These recommendations will help ensure

that teachers participating in future research studies are ready to begin implementing the PIS program effectively at the start of the study.


What preliminary impact does the Pearson Interactive Science program have on student science outcomes?

It should be emphasized that the 2009 small-scale pilot study does not include a comparison group and was conducted over a limited period of time (e.g., teachers used the program from a range of 3 weeks to 8 weeks, depending on number of chapters piloted). Therefore, while preliminary data on student learning gains and other data on participant affective outcomes were obtained, the pilot study does not allow for causal inferences to be made in regards to the effectiveness of the program. Although no conclusive findings about the effectiveness of the program are possible given the nature of the pilot study, analyses were performed to provide preliminary,

descriptive information on the relationship between the PIS program and student science performance.

Data was analyzed via paired sample t-tests

to determine if students showed significant growth in their science performance. Recall that students took tests before and after each chapter that was taught. Thus, there was approximately a 4 week window between pre and post-testing which constitutes a limited time period to demonstrate significant change. After all, within a content area (e.g., Life Science), science concepts build upon one another and as such, greater growth is expected over time as students progress through various chapters and modules within a science content area. Therefore, the larger scale pilot and subsequent RCT will allow for a more sensitive analysis growth.

With this in mind, results indicate that

students showed significant growth from pre- to post-testing on 2 of the 3 chapter tests. Specifically, significant growth was observed on the Earthquakes, t(226)=-13.92, p<.01, and Populations and Communities, t(159)=-12.49, p<.01, tests but not on the Work and Machines test, t(51)=1.57, p=.12, see

Figures 27-29. That said, it should be noted that complete test data for the Work and Machines test was available from only 52 students. Such a small sample may contribute to spurious results. Furthermore, the teacher who had the greatest number of classes (n=4) that took this exam commented that some of the concepts covered in the Work & Machines chapter were too advanced (e.g., Joules and Newtons is taught in high school physics) and that many of her students did not have the necessary prerequisite knowledge. Therefore, it is unclear whether these results are attributable to the structure of the chapter itself, the limited sample, or some other factor (e.g., teacher’s ability to use the program effectively). To the extent that this chapter will be included in the larger-scale pilot study, it is suggested that researchers examine if similar results are obtained and also gather more detailed information that can be used to inform the content and development of this chapter.

Students showed significant growth from pre- to post-testing on the Earthquakes and Populations & Communities chapter tests. No

significant difference was observed on the Work & Machines chapter test.

Figure 27. Student Performance on the Pearson Interactive Science Chapter Test: Earthquakes

*Results statistically significant, p<.05.


Figure 28. Student Performance on the Pearson Interactive Science Chapter Test: Populations & Communities

*Results statistically significant, p<.05.

Figure 29. Student Performance on the Pearson Interactive Science Chapter Test: Work & Machines

Not significantly different.

In addition to examination of overall test

scores, researchers also conducted analyses on the different portions of the tests. In particular, growth scores were calculated for the three types of items contained in the tests: a) multiple-choice; b) fill-in-the-blank; and c) constructed-response. The results are presented in Appendix A and in Figures 30-32. As shown, students showed significant growth on the Earthquakes and Populations and Communities tests, regardless of the type of item. That is, students improved their science performance in these areas as measured by multiple-choice, fill-in-the-blank, and constructed response items. However, mixed findings were obtained on the Work and Machines test. While students

showed significant improvement on the multiple-choice items, there was also a significant decrease on the constructed-response items, and no significant change on the fill-in-the-blank items. To reiterate, however, this may be due to the limited sample available for this test.

Students showed significant growth on the Earthquakes and Populations & Communities tests across all the

different types of test items: multiple-choice, fi l l- in-the-blank, and

constructed response. However, mixed findings were obtained on the

Work and Machines test.

Figure 30. Student Performance on the Pearson Interactive Science Earthquakes Test by Type of Test Items

Earthquakes Chapter Test

17.3%

28.5% 27.8%

4.4%

52.7%

42.2%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Multiple-Choice Constructed Response Fill-in-the-Blank

Pe

rce

nt C

orr

ect

Pre Post Figure 31. Student Performance on the Pearson Interactive Science Populations & Communities Test by Type of Test Items

Populations and Communities Chapter Test

26.6%

45.1%

33.4%

12.7%

46.7%

68.7%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%


Pe

rce

nt C

orr

ect

Pre Post


Figure 32. Student Performance on the Pearson Interactive Science Work & Machines Test by Type of Test Items

Work and Machines Chapter Test

27.2%

13.6%

31.2% 30.0%32.6%

3.8%0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%


Pe

rce

nt C

orr

ect

Pre Post

Theme: Students feel that they perform better in science overall

using the Pearson Interactive Science program.

Results showing improvement in test scores

are also supported by self-reported information obtained from student surveys and focus groups. As shown in Figure 33, the majority of students (over 80%) felt that the PIS program helped them improve upon their science understanding. Students also reported doing better in their science class during their use of the Pearson Interactive Science program. Students and teachers noted that this was aided by the way the material was presented (e.g., questions embedded in text and ease of understanding).

♦ I know a lot of kids who say they hate

science, and I usually didn’t really love

it before because I didn’t really get it.

But because of this book, I can get it

more now, understanding it so much

quicker because of all the activities, and

I think it will have an effect on more

students. If it could happen to me, I bet it

could happen to a lot of other kids, too.

So the quicker this book comes out, I

think it will be better for the system, to

get more kids to understand science. – MA Middle School A Student

♦ I asked them today, “Did this [lesson]

help?” And some of them, who usually

don’t get into it, were like, “I

understand!” – NC School D Teacher Figure 33. Percent of Students who Felt that the Pearson Interactive Science Program Helped Them Learn About Science

83% 82%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

The Pearson Middle Grade Science program

helped me to learn science.

The Worktext w as really helpful in my science

learning.

Subgroup Differences: Student Subpopulations and Teacher Pedagogy

In addition to examining overall

performance of students on the science tests, data was analyzed to examine if there were differences in growth by subgroups of students (i.e., females versus males, minority versus non-minorities, and low-, average-, and high-performing students)10. Results showed no significant interactions between these variables and growth, p>.05 for all comparisons. This means that all types of students showed significant growth regardless of whether they were male or female, minority or not a minority, and low to high performing (see Figures 34-36).

Across all three chapters, pilot students showed significant growth

regardless of whether they were male or female, minority or not a minority,

and low to high performing.

10 To increase the analytical sample, analyses were conducted on the total pre and post tests scores across all three tests (scores were standardized to allow for comparisons). In addition, performance was categorized so that the bottom 33% on the pretests were identified as low-performing, then next 33% were identified as average, and the top 33% were identified as high-performing.


Figure 34. Student Performance on the Pearson Interactive Science Tests by Gender

21.1% 20.9%

43.0%38.2%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Female Male

Pe

rce

nt C

orr

ect

Pre Post

Figure 35. Student Performance on the Pearson Interactive Science Tests by Minority Status

20.0% 21.2%

40.9%40.9%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Non-Minority Minority

Pe

rce

nt C

orr

ect

Pre Post

Figure 36. Student Performance on the Pearson Interactive Science Tests by Performance Categories

21.0% 20.6% 21.2%

41.4%40.5%35.7%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Low Average High

Pe

rce

nt C

orr

ect

Pre Post

Analyses were also conducted to examine if

learning outcomes varied by teacher pedagogy. It should be first noted that in general, teacher’s instructional philosophies were similar and tended to lean towards inquiry-based instruction. That said, researchers identified 3 teachers who used a more blended approach,

combining elements from inquiry and traditional approaches. Comparisons were made between these two types of teachers. Results showed a significant interaction between teacher pedagogy and growth. In particular, results showed that on the constructed-response items only, teachers who employed a more inquiry-based approach to instruction had students who showed greater growth than students of teachers who used a blended approach, F(1, 436) = 4.33, p=.04,

see Figure 37. No significant differences were observed on the multiple-choice or fill-in-the-blank items. Figure 37. Student Performance on the Pearson Interactive Science Test – Constructed Response Items by Teacher Pedagogy

7.1%

11.0%

27.6%25.4%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

Inquiry Blended

Perc

ent C

orr

ect

Pre Post

Results showed that on the constructed-response items,

students of teachers who employed an inquiry-based approach to

instruction showed greater growth than students of teachers who used a

blended approach.


Which type(s) of assessments and outcome measures will be most sensitive to picking up the effects of the Pearson Interactive Science program? What are the psychometric properties of the instruments used during the pilot study?

Based on feedback obtained from the Pearson Science team, it was expected that students would perform well on multiple types of assessment items due to the emphasis placed by the program on preparing students to: a) explain their reasoning via writing; b) apply their science learning to the real world; and c) respond to various types of questions. Examination of change scores by the various types of items included in the chapter tests showed that, with the exception of the Work & Machines test, students did equally well on the multiple-choice and constructed response tests, see Table 6. Furthermore, they showed the greatest improvement on fill-in-the-blank test items which tended to focus on vocabulary. This suggests that the program may be quite helpful in improving students’ knowledge of science terminology.

Table 6. Difference Scores (Post minus Pre) by Chapter Tests and Type of Test Item

Earthquakes

Populations &

Communities Work &

Machines

Multiple-Choice 24.4 19.2 2.9 Constructed Response 23.1 18.2 -10.8 Fill-in-the-blank 31.4 27.6 -3.5

As previously noted, however, the Work and

Machines test did not show the same pattern of growth. On this test, students performed more poorly on the constructed-response and fill-in the blank items than the multiple-choice test. Of note is that this may be a spurious finding based

on the more limited sample who completed this test (n=52). That said, and as previously described, this may be also due to students having difficulty with the chapter due to the higher level science concepts that were introduced and the students’ general lack of prerequisite knowledge. This is a topic to be explored in greater depth during the larger scale Spring 2010 pilot study.

Although it is not anticipated that these

specific tests will be used in the RCT planned for the 2010-2011 school year, reliability analyses were conducted to obtain preliminary information on the technical qualities of the assessments used as part of the pilot study. Results are shown on Table 7. It is a general rule of thumb that instruments with reliability coefficients of .70 and above are considered reliable. As shown, with the exception of the Work and Machines test, the PIS chapter tests showed high levels of both internal consistency (alpha) and split-half reliability11. Table 7. Psychometric Properties of the PIS Chapter Tests

Earthquakes

Populations &

Communities Work &

Machines

Alpha .91 .89 .62

Split-half .82 .80 .30

Information was also gathered from the

Pearson Science team with regard to key constructs that they felt would be affected through use of the Pearson Interactive Science, other than science knowledge and skills. These included the following (Note: constructs measured in the present pilot study are identified with an *):

11 Reliability refers to the property of a measurement instrument that causes it to give similar results for similar inputs. Alpha (Cronbach) is a model of internal consistency, based on the average inter-item correlation. Split-half is a model that splits the scale into two parts and examines the correlation between the parts. The range of this value is 0 (no relationship) to 1 (complete relationship).


♦ Student self-assessment of science learning*;

♦ Sense of ownership towards worktext*;

♦ Student interest and engagement in science*;

♦ Performance on state exams;

♦ Parental awareness of student science learning;

♦ Usage of worktexts by students (how do they use the worktext?)*; and

♦ Critical thinking/reasoning skills In addition to these areas, this pilot study

also revealed that the PIS may affect the following areas: a) application of science, (b) performance in writing and math, (c) monitoring of science learning, (d) organizational skills, (e) predicted performance on state assessments and future science courses, and (f) teacher practices and preparation levels. It will be important to continue to measure these constructs during future research studies in order to more fully examine the impact of this program. In addition, to increase the sensitivity of the RCT to detect differences, and increase the validity of the results (e.g., if positive findings are found across different measures), a battery of outcome tests using multiple test item types is recommended to better capture how the program impacts different areas of scientific understanding and knowledge.

In sum, while the pilot study was designed to obtain formative feedback and preliminary efficacy data, the randomized control trial will be focused on summative information to determine the effectiveness of the PIS program on student science learning, and as such, requires reliable and valid instruments measuring multiple outcomes, using multiple methods.

ConclusionConclusionConclusionConclusion

The pilot study served to obtain formative feedback about the new Pearson Interactive Science program and to get preliminary data on the effectiveness of the program. It also allowed researchers to develop and test various procedures and instruments that could be used during future research studies (e.g., focus group protocols, videotaping procedures). In this section, we summarize what was accomplished and the lessons we learned and their implications for future research.

Accomplishments

♦ Instruments and assessments were created to obtain preliminary efficacy data for this pilot study. These instruments will be modified and refined for future studies so that they are more aligned to the anticipated outcomes identified in this study.

♦ Information was obtained to inform the development of implementation guidelines so as to ensure that teachers use the program successfully, consistently, and with fidelity;

♦ Preliminary efficacy data was obtained from students which showed significant growth from pre- to post-testing on the Earthquakes and Populations & Communities chapter tests. As well, additional constructs that should be measured in future studies were identified.

♦ Triangulation of data from multiple sources and methods was conducted to identify emergent themes which are summarized in this report. Of note is that the feedback obtained from teachers and students was overwhelmingly positive—they enjoyed using the PIS program and especially liked the writing and lab activities.


♦ New procedures were developed to allow for the videotaping of classrooms. Parental consent and video release forms were developed and translated into Spanish by Pearson Education.

♦ Similarly, focus group protocols were developed that may also be used in future research studies. The student focus groups, which could only be conducted with students who had signed informed consent forms, allowed for rich, qualitative information to be obtained and are summarized throughout this report.

Lessons Learned & Implications for Future Studies

♦ Due to observed differences in student test scores by teacher pedagogy, it will be important to measure teachers’ instructional approach to ascertain the extent to which usage of the program and student performance is affected by this.

♦ Teachers noted that their pacing was impacted by the number of labs they completed, which in the majority of cases included all of the labs. Thus, greater flexibility and a smaller number are recommended for the implementation guidelines. For example, teachers could be asked to complete at least 4 Quick Labs per chapter (or half) of their choosing.

♦ A more formal training will be needed for future studies, and should include for example: a thorough discussion of the program’s research base and philosophy, and a detailed walk through of all the teaching resources available as part of the Pearson Interactive Science program, where to find them, and how to use them (e.g., lesson modeling). For the Spring 2010 pilot, we can also examine in more detail the

training model and its sufficiency to prepare teachers to hit the ground running.

♦ If future studies employ the use of focus groups and/or videotaping, it will be essential to do site recruitment and obtain research approval even earlier as this typically takes longer when these procedures are planned. This is because there are more protections required of students who participate in these activities.

♦ Additional potential outcomes that should be measured in future studies on PIS were identified. For example, results showed preliminary relationships between use of the Pearson Interactive Science program and: (a) interest and engagement in science; (b) sense of ownership and pride; (c) application of science; (d) performance in other academic areas; (e) monitoring science learning; (f) organizational skills; (g) predicted performance on state assessments and future science courses; and (h) teacher practices and preparation levels.

♦ Inclusion of assessment(s) using multiple types of items will be important in future studies. Results from the pilot showed that students performed equally well on multiple-choice and constructed response items, and showed the greatest change with fill-in-the-blank items which measured science vocabulary. Such variation in test items is important because a key aspect of the PIS program is to teach students how to read and write about science.

♦ For the larger-scale pilot study, we can also re-examine the Work & Machines chapter to try to determine why students’ did not show growth. In particular, it would be beneficial to ensure that teachers taught at least from two PIS chapters so that they can compare and contrast across multiple PIS chapters. This would help determine if there are any


notable differences between the various chapters piloted (including whether some were more difficult for students).

In conclusion, the pilot study served to provide Pearson Education with valuable, rich feedback on their new Pearson Interactive Science program that will help inform product refinement and marketing. In addition, the pilot study provided researchers with more detailed information about the program and its likely impacts that can be used in future research studies on the Pearson Interactive Science program.


ReferencesReferencesReferencesReferences

Clewell, B. C., Consentino de Cohen, C., Campbell, P. B., & Perlman, L. (2004). Review of

Evaluation Studies of Mathematics and Science Curricula and Professional Development

Models. Washington, DC: Urban Institute. Retrieved online September 16, 2006, from http://www.urban.org/UploadedPDF/411149.pdf.

National Assessment of Educational Progress (2005). The Nation’s Report Card. Washington,

DC: National Center for Education Statistics. U.S. Department of Education (2009). Secretary Arne Duncan Speaks at the National Science

Teachers Association Conference. Washington, DC. Retrieved online May 20, 2009 from http://www.ed.gov/news/speeches/2009/03/03202009.html.

U. S. Department of Education National Center for Education Statistics (2009). TIMSS: Science

Achievement of Fourth- and Eighth-Graders between 1995 and 2007. Washington, DC. Retrieved online June 2, 2009 from http://nces.ed.gov/timss/results07_science95.asp.


Appendix: Table of Statistical ResultsAppendix: Table of Statistical ResultsAppendix: Table of Statistical ResultsAppendix: Table of Statistical Results

Table A1. Pre and Post Pearson Interactive Science Chapter Test Statistics by Item Type

Timing Mean N Std. Deviation t df Sig.

Populations & Communities

Pre 26.55 160 9.64

Multiple-Choice Post 46.65 160 16.46 -19.71 159 0.00

Pre 45.13 160 14.92

Constructed Response Post 68.72 160 17.74 -18.53 159 0.00

Pre 12.72 159 11.08

Fill in the Blank Post 33.37 159 25.12 -11.43 158 0.00

Pre 39.50 160 29.16

Overall Post 71.25 160 31.14 -12.49 159 0.00

Earthquakes

Pre 17.30 227 7.74

Multiple-Choice Post 42.17 227 22.30

-18.50

226

0.00

Pre 28.48 227 11.19

Constructed Response Post 52.69 227 21.77

-16.55

226

0.00

Pre 4.36 227 7.47

Fill in the Blank Post 27.83 227 24.99

-16.03

226

0.00

Pre 19.21 227 20.89

Overall Post 51.72 227 34.15 -13.92 226 0.00

Work & Machines

Pre 27.12 52 9.36

Multiple-Choice Post 32.60 52 13.56

-3.05

51

0.00

Pre 13.56 52 13.41

Constructed Response Post 3.75 52 9.39

5.93

51

0.00

Pre 31.15 52 28.95

Fill in the Blank Post 30.00 52 29.51

0.27

51

0.79

Pre 19.38 52 8.69

Overall Post 17.54 52 8.26 1.57 51 0.12

PO Box 10730 Jackson, WY 83002 Phone: 307-733-3255, Fax: 307-733-2017 [email protected]

PRES Associates, Inc.

A Pilot Study on Pearson’s Interactive Science Program · 2015. 9. 29. · Prepared by PRES...

Documents

Transcript of A Pilot Study on Pearson’s Interactive Science Program · 2015. 9. 29. · Prepared by PRES...