The Effects of Pre-Engineering Studies on Mathematics...

The Effects of Pre-Engineering Studies onMathematics and Science Achievement forHigh School Students*

NATALIE A. TRANDepartment of Secondary Education, California State University-Fullerton, College Park, Office 600-17,

P.O. Box 6868, Fullerton, CA 92843, USA. E-mail: [email protected]

MITCHELL J. NATHAN

Department of Educational Psychology, University of Wisconsin-Madison, 1025 West Johnson Street,

Madison, WI 53706, USA. E-mail: [email protected]

In this study, we examine the relationship between student enrollment in a pre-college engineeringcourse, Project Lead The Way, and student achievement in science and mathematics. Usingmultiple regression analysis (N = 176), controlling for prior achievement, free/reduced luncheligibility, and gender, students enrolled in PLTW courses performed marginally significant betterin mathematics than those who did not enroll in the course at 0.10 alpha level. However, there is nosignificant relationship between PLTW course enrollment and student achievement in science. Wediscuss the implications for these findings and provide recommendations focusing on: makingexplicit integration between academic content and pre-engineering principles; developing assess-ments that adequately represent students’ learning experiences in pre-college engineering; andexamining the impact of student prior achievement and social backgrounds on students’ lateracademic development and career opportunities in engineering.

Keywords: pre-engineering studies; mathematics achievement; science achievement

1. INTRODUCTION

IN EVERY INDUSTRIALIZED nation engin-eering serves as one of the primary vehicles fortechnological innovation, economic prosperity,national security, and advancements in publichealth. However, current educational trends inthe U.S. portend a decline in these areas as themathematical and scientific preparation of Amer-ican K-12 students decline in relation to otherindustrialized nations, and US students opt outof engineering programs and careers. To addressboth the preparedness for and the appeal of collegeprograms and future careers in engineering, federalmandates, such as the National Academy ofSciences’ Rising Above the Gathering Storm [1]and recent amendments to the Carl D. PerkinsVocational Education Act, stipulate that voca-tional and academic education must be integrated.Substantial resources have been allocated ‘toprovide vocational education programs that inte-grate academic [math and science] and vocationaleducation . . . so that students achieve bothacademic and occupational competencies.’[2]. Toaddress this mandate, structured ‘pre-engineering’education programs have emerged to providehands-on, project-based curricula that focus onthe explicit integration of college preparatory

mathematics and science knowledge with engineer-ing activities.However, this is not an easy task. As Rose [3]

points out, there is a fundamental challenge forTechnical Education (TE): Public education insti-tutions are responsible for both the intellectualgrowth and economic preparation of the youth inour society. On one hand, TE programs often lacka strong theoretical framework and a focus on theformal reasoning needed to facilitate generaliza-tion and life long learning in the face of rapidtechnological advancements, or to gain entry indemanding academic programs. On the otherhand, general education programs often fall shortin providing students adequate opportunities toapply the formal knowledge, emphasizing insteadabstract descriptions and symbolic representationsof observable phenomena [4]. Rather than resol-ving this paradox, the tendency in general educa-tion is to classify students into varying abilitygroups and project their career orientationstoward college preparatory courses or career-oriented training [3]. While the potential for posi-tive learning outcomes for students who experiencethe integration between formal education and TEis often discussed, little empirical evidence is avail-able to assess the effectiveness of such integration.The current research explores the extent to whichparticipating in these programs is successful atimproving high school students’ general math

* Accepted 15 October 2009.

1049

Int. J. Engng Ed. Vol. 26, No. 5, pp. 1049–1060, 2010 0949-149X/91 $3.00+0.00Printed in Great Britain. # 2010 TEMPUS Publications.

and science achievement. We focus on ProjectLead The Way (PLTW) as an exemplar of theclass of pre-college engineering programs thatstrives to implement the integration for TE andformal, college preparatory education.

1.1 Project lead the wayPLTW is one of the most widely used pre-college

engineering programs in middle schools and highschools throughout the U.S. It is a multi-year,problem-based/project-based curriculum that hasbeen adopted by over 1,400 schools (over 7% of allUS high schools) in all 50 states and the District ofColumbia [5, 6]. PLTW has two curricularprograms. The Gateway to Technology programat the middle school level provides five nine-weekcourses for students in grades six through eightaimed at showing students how engineering skills,including those from math, science and technol-ogy, are used to solve everyday problems. The highschool program, Pathway to EngineeringTM offersthree one-year foundation courses: Introduction toEngineering Design, Principles of Engineering, andDigital Electronics. Specialization courses such asAerospace Engineering, Biotechnical Engineering,Civil Engineering and Architecture, and ComputerIntegrated Manufacturing, as well as an engineer-ing research capstone course entitled, EngineeringDesign & Development are also available. ThePathway to EngineeringTM curriculum is designedto target students in secondary education who areaspiring to pursue postsecondary engineeringstudies.In 2005 PLTW showed a significant pattern of

growth, with 75% of the states reporting that theyhad already implemented the program, indicatingan increased awareness and demand for theprogram and pre-college engineering courses [7].Sander’s investigation [8] indicates that 85% of allteachers who teach PLTW are TE teachers whohave had TE teaching experiences. The PLTWcurriculum extends beyond just course materials.Everyone teaching PLTW courses must attend atwo-week professional development trainingprovided by PLTW’s affiliated colleges anduniversities. The professional development train-ing aims to make teachers proficient with thePLTW curriculum [5]. Professional training forhigh school guidance counselors is also availablefor participating schools. In addition to hostingsummer training institutes and ongoing profes-sional development, national affiliates offer grad-uate college credit opportunities for teachers.According to the PLTW website, when

combined with academic mathematics and sciencecourses, Pathway to EngineeringTM strives to intro-duce students to the scope, rigor and discipline ofengineering and engineering technology [5]. Giventhe broad content embedded in PLTW, thosestudents who do not intend to pursue furtherformal education can also benefit greatly fromthe technical knowledge and skills, as well as thelogical thought processes, which result from enrol-

ling in some or all of the courses provided in thecurriculum. The NRC report [1], Rising Above theGathering Storm, explicitly identifies PLTW asoffering a model curriculum for providing rigorousK-12 content needed to improve math and sciencelearning and increase America’s technologicaltalent pool.As described above, PLTW is an appropriate

choice of curriculum for our studies of the impactof pre-college engineering on student learning inscience and math because of its widespread use, theextensive teacher training program, and theprogram’s stated focus on integration engineeringwith science and mathematics. However, it isimportant to acknowledge that PLTW is onlyone program out of many for understandingimpact of the curricular integration between TEand formal learning. We are aware that PLTW isunique in some respects, and fosters its ownapproach to teaching and learning. Therefore,any findings associated with PLTW cannot beimmediately generalized to other TE programs,though they will contribute to our understandingof the complexities involved in pre-college engin-eering studies more generally.

1.2 Prior empirical research on pre-collegeengineeringContemporary empirical studies have made

noticeable contributions to the research involvingpre-college engineering. One study found no signif-icant differences by race or gender between TEparticipants and the current general student popu-lation [9]. Contrary to the general perception, themajority of TE concentrators (i.e., those takingthree or more courses in a common career track)go on to college, not directly to work; 80%complete the same number of high school mathand science course credits as their non-TE peers;and although TE concentrators as a group enterhigh school less well prepared than academic-onlystudents, that gap is narrowed and may even beeliminated by the time they reach graduation [10,11]. A number of studies examining career acade-mies—college-preparatory curricula with careerthemes and established partnerships with commu-nity businesses—have shown that career academiesenhance interpersonal social relations and engage-ment among teachers and students. However, theseeffects do not transfer to students’ achievement onstandardized tests, high school graduation rates, orhigher educational attainment [12, 13]. The resultsof these studies suggest that the belief that TEstudents are different from those perusing purelyacademic programs is unfounded.Only recently have there been studies specifically

directed at pre-college engineering curricula. In areport comparing student achievement and learn-ing experiences between PLTW students and otherTE students enrolled in the High School ThatWorks (HSTW) network, Bottom and Uhn [14]found that PLTW students who completed at leastthree PLTW courses scored higher in math and

N. A. Tran et al.1050

science on the National Assessment of EducationalProgress (NAEP), a set of national, standardizedtests that contribute to the ‘nation’s report card’provided by the National Center of EducationalStatistics, compared to other TE students in theHSTW network. According to this report, PLTWstudents in the sample were more likely than theirHSTW counterparts to: complete four years ofmathematics and three years of science courses;experience engaging instructional practices in theircourses; integrate reading, math, and scienceknowledge into their TE courses; and perceivehigh school as an important preparation for theirfuture. It is important to note that the sample ofstudents in this study composed of 65% White,22% African American, and 13% Other. A largeproportion of the students’ parents pursued post-secondary education (72%). Female students repre-sent 14% of the sample. Because students were onlysampled from the HSTW network and representeda specific demographic, these findings may notgeneralize to high schools more broadly.However, a number of recent studies have found

equivocal results. Our previous work on curricu-lum analysis specific to PLTW [15] revealed thatthe foundation courses provide very limited expo-sure to the mathematical content identified by theNational Council of Teachers of Mathematics [16].A much broader study of twenty-two pre-K–12engineering curricula, including PLTW, producedsimilar findings [17]. These investigators bemoanedthe prevalence of very basic mathematics and datagathering activities at the expense of more sophis-ticated and more relevant emphases on mathema-tical modeling and engineering design. The lack ofemphasis on engineering preparation is also foundin higher education. A recent Carnegie report [18]indicated that engineering schools, still heavilyinfluenced by the academic traditions, are notpreparing their students for the engineering profes-sion which requires the acquisition of technicalskills. Our subsequent analyses of the PLTWfoundation curriculum showed that on the occa-sions when the mathematics concepts do arise, theyare often implicitly embedded in the technologyand tools of the course activities [19] rather thanpresented as pedagogical opportunities to expli-citly integrate mathematical ideas into engineeringcontexts.

2. HYPOTHESES OF THE STUDY

Drawing on prior work in this area, we considerthree major hypotheses to explain the possibleoutcomes resulting from the integration of mathand science to pre-college engineering studies.First, because a pre-college engineering courselike PLTW is an elective course, we assume thatstudents enrolled in these courses will receive addi-tional instruction and practice time in science andmathematics that are above and beyond that of theregular academic program. It follows that pre-

college engineering course enrollment may be asso-ciated with higher levels of subject area-specificachievement. That is, students enrolled in PLTWcourses could benefit more from the exposure ofmath and science content knowledge in an engin-eering context that is different from the academiccourses taken by them and their non-PLTW peers.This leads us to our prediction that, controlling forprior achievement and other individual character-istics, students enrolled in PLTW courses areexpected to demonstrate greater improvements intheir achievement in mathematics and sciencecompared to their non-PLTW peers. Followingour previous work [20], we call this the enrichedintegration hypothesis. This hypothesis is supportedby previous research indicating that the bestpredictor of achievement is time on task [21, 22].Therefore, if the integration of science and mathtopics is effectively implemented, and if this inte-gration can yield positive learning outcomes, thenat a minimum, those taking PLTW courses shouldexperience increased time spent learning mathe-matics and science, thus resulting in improvementin student achievement. This integration providesstudents with the opportunity to engage in hands-on and real-world projects, and make connectionsbetween the knowledge and skills taught in theiracademic math and sciences classes and theirapplication to engineering projects. This compre-hensive approach to instruction using collabora-tive, technology oriented, project-based activitiesenables students to synthesize and construct newknowledge in various contexts [23].Two alternative hypotheses must also be consid-

ered. The insufficient integration hypothesis ad-dresses the possibility that there may be little orno integration between math and science contentknowledge and the engineering activities in PLTWcourses. Even when there is integration, if it is doneat a superficial level it may result in little or noimpact on student learning. This hypothesispredicts that controlling for prior achievementand student characteristics, students enrolled inPLTW course will perform the same as theirpeers who do not enroll in these courses. Finally,it is important to acknowledge that pre-collegeengineering experiences have many unique qual-ities that are different from the typical math andscience classroom. As such, the emphasis on colla-borative design, engineering skills such as drafting,computer-aided design (CAD), measurement andfabrication may interfere with the analytical andabstract exercises that typically make up math andscience assessments. The adverse integrationhypothesis recognizes that interference from pre-college engineering could lead to changes in atti-tudes, confusion, or even misconceptions thathinder student performance. This hypothesispredicts that, controlling for prior achievementand student characteristics, students in PLTWcourses will show lower science and mathematicsachievement gains compared to students who arenot enrolled in these courses.

Pre-College Engineering Studies 1051

Our recent study examined student math andscience achievement (N = 140) as a function ofPLTW enrollment for an ethnically and socio-economically diverse school district (72% offamilies qualified for the Free/Reduced LunchProgram; 57% African American, 22% Hispanic,12% White, 4% Asian, and 4% Other). We foundthat, while students generally gained in math andscience achievement from 8th to 10th grade,students enrolled in PLTW foundation coursesshowed significantly smaller math gains thanthose in a matched group that did not enroll inPLTW courses [20]. We also found no measurableadvantages on science assessments, after control-ling for prior achievement, student characteristics,and teacher experience. We concluded that thedata provided no support for what they termedthe enriched integration hypothesis, which statesthat, after controlling for prior achievement andstudent characteristics, students taking one ormore engineering courses will be expected toscore higher on science and mathematics standard-ized tests than the students with no engineeringcoursework. Rather, we found the data mostclosely aligned with the insufficient integrationhypothesis, in the case of science achievement,and the adverse integration hypothesis, for mathachievement.The results of our study mentioned above

suggest not only a lack of integration betweenacademic and pre-college engineering studies butalso a deficiency in engineering experiences thatare necessary to prepare students for futureadvanced studies and careers in engineering.When engineering experiences are put in place,results show mixed learning outcomes. The currentstudy adds onto the existing body of research onthe impact of pre-college engineering studies byinvestigating whether the results produced inprevious studies can be replicated using a samplewith different demographics—a school districtwith a higher socioeconomic population—thusexpanding the generalizability of previous researchand contributing to a richer understanding of thenature of engineering education at the high schoollevel.This study expands on our previous work on the

effects of a pre-engineering curriculum on studentachievement in science and mathematics [20] in thefollowing ways: a) it uses different student demo-graphics (larger proportion of students fromhigher social economic background); b) arandom sampling technique is used to generate acomparison non-PLTW group; and c) multipleregression, rather than multi-level modeling, isused to estimate the relationship between PLTWand student achievement. It is important to notethat because we did not randomly assign studentsto be in or out of these classes, causal claims arenot supported by the current design and analysistechniques. However, the findings of this study willprovide us with greater insights about pre-collegeengineering studies and factors that are associated

with student achievement in science and mathe-matics.

3. METHOD

3.1 SampleThe current study reports findings from a

sample of participants who are completely differ-ent from the participants analyzed in our previousinvestigation [20]. It consists of 176 eleventh-graders from four high schools in a moderatelydiverse urban school district in the Midwestern citywith 30% students of color, 2% English LanguageLearners, 24% eligible for free/reduced lunchprograms, and 15% identified for special educationservices. The students in this sample reside in anaffluent community with median income for afamily is around $60,000. Surrounded by a majoruniversity, 48.2% of city’s population over the ageof 25 holds at least a bachelor’s degree. In recentyears, Forbes magazine reported that the city hasthe highest percentage of individuals holding PhDsin the United States and having the lowest unem-ployment in the nation: 2.5%, less than half of thegeneral U.S. population in 2004.Using the school district information provided

on course enrollment, the PLTW group was iden-tified—those who enrolled in PLTW courses in the2007-2008 academic year. This sample composedof 57 students enrolled in one or more PLTWcourses offered and 119 students in the samegrade level who did not enroll in PLTW coursesin the 2007–2008 academic year. The non-PLTWgroup was identified using a random samplingtechnique generated by the statistical softwareSPSS where 119 students were randomly chosento represent the students in non-PLTW group.Following the selection, cross-tabulation wasimplemented to summarize information pertainingto gender, ethnicity, free/reduced lunch, andspecial education. The chi-square procedure wasused to test the null hypothesis that students in thePLTW group and those in the non-PLTW sharesimilar characteristics with the alpha level set at0.05. Overall, the results showed that–with theexception of gender (p < 0.000)–differences infree/reduced lunch eligibility (p = 0.273), specialeducation status (p = 0.575), and English profi-ciency (p = 0.749) were not statistically differentbetween PLTW students and their non-PLTWcounterparts. The analysis above suggests thatthe random sampling was a success resulting intwo groups with similar observable characteristics.Table 1 provides a description of the students inthe sample.

3.2 MeasuresStudent achievement. Data on student achieve-

ment including 2005–06 and 2007–08 results fromstate standardized tests for the tenth-gradestudents in mathematics and science were providedby the school district. Both math and science


assessments were administered to students inNovember, 2005 (8th grade) and again in Novem-ber, 2008 (10th grade). Using multiple-choice andshort-answer questions, these standardized testsare designed to measure the state academic stan-dards in mathematics and science. The scale scoresfor math range 350–730 for 8th grade and 410-750for 10th grade. For science, the scale scores range230–560 for 8th grade and 240–610 for 10th grade.The proficiency categories are advanced, profi-cient, basic, and minimal performance for allassessments.Student background variables. The district also

provided data on student characteristics includingfree/reduced-price lunch eligibility, and courseenrollment. This information was used toconstruct a set of dummy variables for gender(female =1), free/reduced lunch (eligible = 1), andPLTW enrollment (students enrolled in at leastone PLTW courses = 1). These variables, alongwith student prior achievement (2005-06 in 8thgrade) in mathematics and science, were includedas predictors in the regression analysis described inthe following section. Table 2 provides descriptivestatistics of the variables used in the analysis.The table of descriptive statistics shows that the

mean and standard deviation of 10th grade mathachievement is 582.37 and 49.557 respectively, andfor 10th science achievement it is 463.20 and49.892 respectively. The mean (563.69) and stand-ard deviation (47.058) for 8th grade math achieve-ment is slightly lower. The same pattern is foundfor 8th grade science with a mean of 417.29 andstandard deviation equals 48.463. Free/reducedlunch has a mean of 0.24, which implies that 24%of the sample is qualified for free/reduced lunchprograms. The mean for female is 0.43, whichindicates that 43% of the sample is female.Lastly, PLTW enrollment has a mean of 0.32,

which represents 32% of the students in thesample enrolled in PLTW courses.

3.3 AnalysisFirst, correlation analyses were conducted to

examine the relationship between student enroll-ment in PLTW courses and achievement in scienceand mathematics. Second, multiple regressiontechniques were used to examine the extent towhich student characteristics—prior achievement(8th grade) on state standardized assessment, elig-ibility for free or reduced lunch programs (ameasure of family socio-economic status), gender,and enrollment in one or more PLTW courses, asindependent variables—would predict students’math and science achievement in the 10th grade.Third, standardized achievement scores were usedto compare achievement test scores across gradesand subject areas. The model is specified as:

YAchievement = �0 + �1 Prior Achievement + �2Free/Reduced Lunch + �3 Female + �4 PLTW + "

Where Y (achievement), the dependent variable, isan estimate of student state standardized test scoreat 10th grade (in either math or science). Themodel also has four independent variables, orfactors: prior achievement, a student’s test score(in math or science) at 8th grade; free/reducedlunch status, a measure of family socio-economicstatus, with a value of 1 if a student is eligible forfree/reduced lunch program, and 0 if not; female,with a 1 for females and 0 for males; and PLTWenrollment, with a 1 for those enrolled in one ormore PLTW courses, and a 0 if not. �0 representsthe constant of the equation and is the estimatedvalue of 10th grade student achievement (Y; typi-cally the grand mean) when �1, �2, �3, �4 = 0 (i.e.,as if we did not have any of the informationobtained from knowing the values of the indepen-

Table 1. Characteristics of the students in the sample

EnrollmentAfricanAmerican Asian Hispanic White Female

SpecialEducation

Free/ReducedLunch

PLTW (N = 57)Raw(%)

6(3.4)

8(4.5)

2(1.1)

41(23.3)

9(5.1)

10(5.7)

1(0.01)

Non-PLTW (N = 119)Raw(%)

24(13.6)

10(5.7)

3(1.7)

82(46.6)

67(38.1)

17(9.7)

3(1.7)

Table 2. Descriptive statistics for student data (N = 176)

Variables Minimum Maximum Mean Std. Deviation

10th Grade Math Achievement 410 750 582.37 49.55710th Grade Science Achievement 240 591 463.20 49.8928th Grade Math Achievement 397 679 563.69 47.0588th Grade Science Achievement 284 707 417.29 48.463Free/Reduced Lunch 000 001 000.24 0.431Female 000 001 000.43 0.497PLTW Enrollment 000 001 000.32 0.469


dent variables for each student). �1, �2, �3, �4 arethe slopes of the regression line, also known as theregression coefficients. Each regression coefficienttells us how much change we will see in Y (pointsstudents will increase/decrease in 10th gradeachievement) for each unit change in the respectiveindependent variable. This model representsachievement regressed on the prior achievementscore, gender, free/reduced lunch status, PLTWenrollment, and the disturbance term (e), whichcaptures the influence on student achievement ofeverything other than the independent variablesspecified in the equation.

4. RESULTS

Paired sample t-tests for the group as a whole(N = 176) indicate significant gains of student

achievement in mathematics (p < 0.01) and science(p < 0.01) from 8th grade to 10th grade. Moder-ately high correlations (0.77 for math and 0.55 forscience, respectively) between 8th grade and 10thgrade achievement tests suggest that students whodid well on their subject-specific test in 8th grade,tended to do well on the respective 10th grade test.Figures 1 and 2 provide a summary of the overallmath and science achievement for 8th and 10thgrades.Table 3, the correlation analysis between PLTW

enrollment and math achievement, shows thatthese two factors were positively related: r =0.14, p = 0.057. Another correlation analysisshowed that PLTW enrollment and scienceachievement were positively related: r = 0.08, p =0.270; however, this correlation is not statisticallysignificant. We recognize that while the correla-tions between PLTW enrollment and studentachievement are very low, based on previousresearch we have reason to suspect that statisticalrelationships can be found between these variables,once other factors are controlled for. Therefore, weproceeded with the subsequent analysis.Regression analysis was used to examine the

relationships among the variables. This techniqueis used to predict student achievement in mathe-matics and science given information about studentprior achievement and other characteristics. Sincethe assessment scales may vary from year to year, itis necessary that we conduct subsequent analysesusing standardized scores. A standard score is atransformed score that relates a raw score to themean and standard deviation of the distribution.The standard score allows comparison of observa-tions from different normal distributions. In thiscase, it enables us to compare the various types ofoutcomes related to mathematics and scienceachievements. The most common standard scoreis z-score with a mean of 0 and a standard deviationof 1. A raw score can be transformed to a z-scoreusing the formula: zi = (xi –M)/s, where xi is the rawscore for student i, M is the mean of all studentscore, and s is the standard deviation for all scores.Positive z-scores are above average, and negative z-scores are below average. Standardized (z) scoreswere computed using SPSS. Effect size can beinterpreted as a kind of z-score. Therefore, effectsize uses the concept of ‘standard deviation’ toexplain the difference between two groups.The results indicate that collectively, prior

Fig. 1. Prior and current math achievement of PLTW and non-PLTW students.

Fig. 2. Prior and current science achievement of PLTW andnon-PLTW students.

Table 3. Correlations between student factors

10th GradeMath

10th GradeScience

8th GradeMath

8th GradeScience

Free/ReducedLunch Female

10th Grade Science 0.77**8th Grade Math 0.77** 0.65**8th Grade Science 0.48** 0.55** 0.74**Free/Reduced Lunch –0.39** –0.42** –0.38** –0.34**Female –0.09 –0.05 –0.13 –0.17* 0.12PLTW Enrollment 0.14* 0.08 0.10 0.15* –0.08 –0.38**

NB. * Correlation is significant at the 0.05 level (2-tailed). ** Correlation is significant at the 0.01 level (2-tailed).


achievement, free/reduced lunch eligibility, andPLTW enrollment account for 60.8% of the vari-ation in math achievement, a significant relation-ship between prior achievement, free/reduced luncheligibility, gender, PLTW enrollment and studentmath achievement, R2 = 0.61, F (4, 171) = 66.21, p< 0.001. Prior achievement alone accounted for59.0% of the variation in math achievement.Adding free/reduced lunch status to the regressionmodel helped explain 60.1% of the achievementvariation. The addition of gender to the modelexplained only a small amount of additional vari-ation of achievement (60.2%). Adding PLTWenrollment variable increased the percentage ofvariance explained to 60.8%. To assess the statis-tical significance of the unique contribution ofPLTW enrollment, we examine its coefficient.Controlling for prior achievement, eligibility forfree lunch, and gender, we found that studentsenrolled in PLTW courses scored marginally(0.10 alpha level) significantly higher than non-PLTW students (effect size = 0.18). An effect sizeof 0.18 falls in the range of a ‘small effect.’Specifically, it means that the score of the averagestudent in the PLTW group exceeds the scores of57% of the students in the non-PLTW group.For science, prior achievement, student charac-

teristics, and PLTW enrollment explained 61%of the science achievement variation, R2 = 0.370,F (4, 171) = 25.14, p < 0.001. Prior achievementexplained 30.7% of the variance in student achieve-ment. Adding free/reduced lunch status to theregression model increased this number to 36.7%.Similar to math, the addition of gender did notexplain much additional variance found in studentachievement (37.0%). Including PLTW enrollmentdid not improve the model (37.0%). The resultsshow no significant relationship between PLTWenrollment and student achievement in science,after controlling for prior achievement and studentcharacteristics, p = 0.85. That is, students enrolledin PLTW courses did not score significantly higheron science assessment than their non-PLTW coun-terparts.While the study’s focus is on the effects of

PLTW enrollment and student achievement inscience and mathematics, we want to highlightthe significant contributions of other factors inexplaining student achievement. Our results showthat controlling for free/reduced lunch status,gender and PLTW enrollment, student prior

achievements (8th grade) are significant predictorsfor current student achievement (10th grade), withthe effect size of 0.723 (p < 0.001) for mathematics(the upper end or a ‘medium’ effect size) and 0.474(p < 0.001) for science (the upper end or a ‘small’effect size). This suggests that students who didwell in 8th grade assessments tended to do well inthe 10th grade science and mathematics assess-ments. Another significant predictor is studenteligibility for free/reduced lunch programs (anindicator of socioeconomic status). The resultsindicate that after controlling for prior achieve-ment, gender, and PLTW enrollment, studentsqualified for free/reduced lunch programs (thosefrom lower socioeconomic background) scored0.269 standard deviation lower on standardizedmath assessment (p = 0.027) and 0.614 standarddeviation lower in science (p < 0.001) compared tostudents who are not eligible for free/reduced lunchprograms (those from higher socioeconomic back-ground). It is also important to note that aftercontrolling for other factors, gender does notappear to have an effect on student achievementin science (p = 0.358) or mathematics (p = 0.305).Table 4 provides a summary of the results.The multiple regression equations for predicting

student achievement in science and mathematicsusing standardized scores are specified below.

Math achievement =–0.039 + 0.723*Prior Math Achievement

–0.269*Free/Reduced Lunch + 0.108*Female+ 0.178*PLTW + �

Science achievement = 0.088 + 0.474*Prior ScienceAchievement–0.614*Free/Reduced Lunch + 0.123*Female +

0.026*PLTW + �

The findings for mathematics performance supportthe enriched integration hypothesis in math achieve-ment. The relationship between PLTW enrollmentand math achievement is marginally significant atthe alpha level 0.10, suggesting that the relation-ship is notable and might be stronger with a largersample. However, enrollment in PLTW coursesshowed no measurable benefit for 10th gradescience achievement. This supports the insufficientintegration hypothesis and suggests two possibili-ties: a) little integration between engineering andscientific concepts is taking place to connect to oradvance the scientific knowledge and reasoningabilities of students; or b) if the connection

Table 4. Regression analyses predicting student achievement in science and mathematics

Mathematics Science

Predictor Coefficient SE P Coefficient SE P

Intercept –0.039 0.087 0.659 0.088 0.110 0.423Prior achievement 0.723* 0.052 0.000 0.474* 0.065 0.000Free/Reduced lunch –0.269* 0.120 0.027 –0.614* 0.150 0.000Female 0.108 0.105 0.305 0.123 0.133 0.358PLTW enrollment 0.178** 0.111 0.110 0.026 0.141 0.854

NB. * Indicates significance at or below the 5% level. ** Indicates marginal significance at the 10% level.


between pre-college engineering content and scien-tific concepts is made, little impact is shown instudent achievement on standardized tests. Belowwe discuss the implications of these findings.

5. DISCUSSION AND CONCLUSIONS

Overall, PLTW enrollment has a marginal, butpositive impact for 10th grade math achievementscores, but no measurable effect on science perfor-mance. These analyses use correlational techni-ques, and thus cannot tell us about the causalrelationship between PLTW enrollment andstudent achievement in science and mathematics.Similar findings in science achievement have beenreported in our previous work with a differentdistrict serving a more ethnically diverse studentpopulation and a larger number of students whoqualified for free/reduced lunch [20]. There havealso been alternative studies examining thepresence and level of integration of mathematicsin PLTW and other pre-college engineering curri-cula [15, 17, 19], suggesting that the pattern is notspecific to any one pre-engineering curriculum.While important engineering education objectivesin areas of design, analysis and testing are beingmet, it appears the clarion call for TE to advanceacademic goals in science and mathematics as wellas contributing to one’s TE is not being respondedto adequately. This is particularly notable becauseof the high academic standards in math and sciencefor admission to and success in engineeringprograms at the university level [24]. We concludethe article with specific recommendations for curri-culum design and professional development aimedat improving the integration of core science andmath concepts within pre-college engineeringlessons, and development of alternative assess-ments that adequately measure student outcomesfor pre-college engineering studies. Lastly, weexamine some of the policy implications of thesefindings as educational leaders consider reformingsecondary education.

5.1 PLTW and achievement in science andmathematicsIn our study, students as a group showed

measurable gains in science and math from 8thto 10th grade. It is important to consider studentachievement gains in light of other factors thatmay enhance test performance, such as generaldevelopmental maturation and learning from thegeneral middle school and high school curriculum.The role of the comparison group allows us to askwhether the gains observed by students enrolled inPLTW courses are significant, above and beyondthose exhibited by a similar group of students. Inthis data set, PLTW enrollment is associated withno achievement gains in science tests, above andbeyond those gains obtained by students in thecomparison group, while math achievement showsmodest gains beyond those students who do not

take engineering courses. The findings on therelationship between PLTW science achievementare consistent with those found in our previousinvestigation using a different sample [20].However, the relationship between PLTW andmath achievement is inconsistent in these twostudies—with a positive association betweenPLTW enrollment and math achievement foundin this study and a negative association betweenthese two variables exhibited in our previous study[20]. We suspect that differences in these findingsmay be attributed to the demographics of thestudents in the samples, a topic we discuss in thefollowing section. Meanwhile, curriculum analysesmay show a low level of integration of math andscience in pre-college engineering activities, theresults reported here and elsewhere [15, 17, 20]are, on the balance, consistent with the insufficientintegration hypothesis, which suggests that theabsence of additional gains in achievement scoresmay be attributed to little or no integration of thescience content knowledge and engineeringconcepts made for students. We offer three otheralternative explanations for the lack of associationbetween PLTW and science achievement.First, we discuss the use of standardized assess-

ment in measuring student learning outcomes.Under the federal requirement of No Child LeftBehind Act in 2001 (NCLB), the use of standard-ized tests has increased drastically to reinforceaccountability standards for teachers and admin-istrators. Linn and Gronlund [25] have helped toarticulate two sides of the accountability debate.Proponents of high-stakes testing believe that thetests measure objectives that are important forstudents to learn and the tests can guide teachersto focus their attention on those objectives. Oppo-nents of standardized assessments have pointedout that the tests can be biased against certaingroups of students and tend to focus more on(easier to score) factual knowledge and low-levelskills at the expense of assessing deep, conceptualreasoning. Therefore, in the current climate, it isnot uncommon for important content knowledgeto be excluded from a curriculum if it is notincluded in the assessment—so-called ‘teaching tothe test.’ This underscores the importance ofunderstanding the alignment between standardizedtests that purport to measure achievement, and thespecific activities and learning objectives of thecourses that students enroll in. Poor alignmentbetween what is taught in these engineering coursesand what is assessed in general tests of high schoolscience and math must be considered as a factorwhen interpreting the relationship between courseenrollment and achievement scores.A second alternative explanation explores the

degree to which student learning can be transferredacross contexts. When there is a close matchbetween the learning activities and the assessment,it is easy to measure the transfer of learning acrossthese two contexts. This ‘near transfer’ is madepossible because of the similarities between tasks


that students are asked to perform in these settings[26, 27]. In contrast, ‘far transfer’ takes place whenstudents acquire deeper understanding of theconcepts and thus are able to apply this knowledgein a different context than the one in whichlearning took place. Thus, the results of thisstudy suggest that we are more likely to be ableto detect an effect that is ‘near transfer’ (whenthere is good alignment between the learningactivity and assessment) than ‘far transfer’ (whenthere is poor alignment between the learningactivity and assessment).Third, we suspect student demographics and

district characteristics may be attributed tothe observed outcomes. Compared to previousresearch focusing on a more diverse populationin an urban setting [20], this study focused on asample of students coming from a more affluentcommunity. The different results found in thesetwo studies suggest that PLTW may have differ-ential effect on student achievement depending onthe student demographics and their local context—that is, the enrichment integration hypotheses maybe more likely to be realized for students fromhigher social economic backgrounds. In otherwords, it is plausible to suspect that because ofdifferent values, expectations, communitydemands, parental influences, or even teachingpractices, PLTW yields greater benefits forstudents from more affluent backgrounds. Thisfinding is supported by previous research samplingPLTW students with similar racial/ethnicitycomposition showing positive learning outcomesfor students enrolled in PLTW courses [14].Further investigation is needed to assess the poten-tial differential impact of PLTW.Since the results of this study were not consistent

with those found in the previous investigation [20],in the following section we use descriptive informa-tion to explore how the results may vary acrosscontexts (one school district located in a metropo-litan area while the other is in an urban setting).Descriptive information on Table 5 shows thatstudents in these two districts vary in studentscience and mathematics achievement and free/reduced lunch eligibility (an indicator of socio-economic status). While this information onlyprovides us with descriptive characteristics of thedistricts and no statistical association can bedrawn, in the following section we highlight thesignificance of student characteristics in predictingtheir achievement in science and mathematics.

The current results show student prior achieve-ment and socioeconomic status are significantfactors explaining a large proportion of the vari-ation of the student achievement in science andmathematics. These findings are consistent withprevious research [20] and large-scale policyresearch in the field of education, supporting theidea that, coupled with prior achievement, theaccumulation of characteristics that define socialclass differences influence student achievement[28]. It is not difficult to imagine how students’current learning experiences in the classroom—their mastery of the content knowledge, engage-ment level, and attitudes about the subject—canaffect how they perform in the future. In additionto the educational experiences that the schools canprovide, student social background including theireconomic status, learning opportunities in out-of-school settings, and parental engagement, just toname a few can have an impact on their learningoutcomes. The significant contributions of thesetwo factors—student prior achievement and socialbackground– alone in students’ achievement makeit a challenging task for the educators to developinstructional practices and curricular approachesthat can have long lasting impacts on students’future scholastic and economic opportunities.Thus developers of pre-college engineeringprograms like PLTW must closely examine howthe content and learning activities presented inthese curricula are connected to student priorknowledge and experiences outside of the class-room. Only then will pre-college engineeringexperiences in the classroom significantly contri-bute to the academic preparation needed to pursueadvanced studies and careers in engineering for allstudents.

5.2 Limitations and future workEven though PLTW is a widely distributed

curriculum, there is a disproportionately smallnumber of students enrolled in the school ordistrict at any given time. In addition, analysesusing gains in student achievement scores requirethat students have complete data for multipleyears. These two factors alone make it challengingfor the researcher to obtain a large sample size toconduct meaningful statistical analysis. This studyused correlational techniques to examine the rela-tionship between PLTW enrollment and studentachievement in science and mathematics. Sincestudents were not randomly assigned to PLTW

Table 5. Descriptive information of two districts in previous and current studies

MetropolitanDistrict

UrbanDistrict

8th Grade Math Achievement 563.69 509.238th Grade Science Achievement 417.29 369.9210th Grade Math Achievement 582.37 523.3110th Grade Science Achievement 463.20 416.66Free/reduced Lunch Eligibility 24% 77%


courses, causal inferences cannot be made based onthe outcomes produced from these analyses. It isrecommended that future studies use larger samplesizes to examine the differential effects of PLTWenrollment among students in various sub groups.Also, when feasible, an experimental study shouldbe conducted to determine the causal relationshipbetween PLTW and student achievement byassigning students to be in PLTW treatment andnon-PLTW control groups. However, courseenrollment is not something casually chosen sinceit can have profound effects on a student’s future.In practice, parents, teachers, guidance counselorsand students all weigh in on these decisions. Forthis reason, there are ethical considerations thatalso come into play for studies that contemplaterandomly assigning students to different coursesfor purposes of supporting causal inference andhypothesis testing. In this sense, correlationalstudies based on prior enrollment decisions provideonly a partial portrait of the forms of learning thatare evident in school settings.

5.3 ConclusionsBased on the results of this study, we provide

four important recommendations that can advanceour knowledge in the area of pre-college engineer-ing studies and future assessments of their effec-tiveness. First, we recommend that pre-engineeringcurricula like PLTW make explicit and extensiveconnection between engineering principles andmath and science concepts through various learn-ing activities. This integration can be enhanced byproviding on-going professional developmenttraining throughout the school year that is speci-fically aimed at improving the integration of corescience and math concepts within pre-college en-gineering lessons for TE, mathematics and scienceteachers. Previous investigations on curricularanalysis indicate insufficient integration of engin-eering activities with the central concepts andprocedures in college preparatory science andmath courses [15, 19]. Yet studies explicitly inte-grating math instruction with TE courses (inagriculture, auto mechanics, business and market-ing, health sciences, and information technology)have been shown to significantly improve students’math performance, without any drop in perfor-mance in technical skills [29]. Results from thisstudy are consistent with these findings, suggestingthat teachers and staff providing professionaldevelopment should consider making the integra-tion a more explicit instructional goal. For effec-tive integration to take place, school and districtleaders ought to consider professional develop-

ment approaches that model how pre-engineeringconcepts can be integrated in math and scienceacademic courses and vice versa. Following theprofessional development training, school leaderswill need to provide on-going instructional supportfor teachers to implement the newly acquiredcontent and pedagogical skills in the classroomand discuss improvements that can be made foreffective integration.Second, the development of alternative assess-

ments that better align and reflect the skills andknowledge that students are learning in these pre-college engineering courses is much needed.Currently, researchers rely heavily on state andnational standardized assessments as indicators toassess instructional effectiveness and measurestudent learning outcomes. While these assess-ments are widely utilized as common assessments,they are not designed to assess students’ perfor-mance skills and comprehension of engineeringprinciples in particular. Course specific assess-ments are likely to be better aligned with curricularcontent, but are not likely to be taken by thegeneral student population, most of whom arenot enrolled in the specific TE courses. Thus, thedesign of alternative assessments is crucial toaccurately determine the effectiveness of pre-college engineering studies on improving studentlearning relative to the general student population.Third, these data provide value to district

leaders and policy makers who may be consideringrevising graduation requirements in science toinclude pre-engineering courses. While this inves-tigation provides no causal claims about PLTWand science achievement, it does suggest that thesepre-engineering courses, and perhaps others likethem, have limited impact on the forms of scienceknowledge that may be assessed in current stateassessments. In this regard, pre-engineeringcourses may serve as valuable science electives,perhaps as a precursor to fulfilling graduaterequirements. This would allow policy makers toproceed cautiously until data on the impact of pre-engineering on students’ scientific achievement aremore conclusive.Lastly, these data re-iterate the strong impact

that both academic factors (e.g., prior math andscience achievement) and social factors (e.g.,family socioeconomic status) have on students’later test performance. Teachers and district staffmust continue their efforts to provide qualityscience and mathematics education in the earlygrades, and to create a school climate where allstudents see competency in science and technologyis within reach.

REFERENCES

1. NRC, Rising Above The Gathering Storm: Energizing and Employing America for a BrighterEconomic Future Committee on Prospering in the Global Economy of the 21st Century: An Agendafor American Science and Technology. National Academy of Sciences, National Academy ofEngineering, Institute of Medicine, National Academy Press, Washington, DC, 2007.


2. Public Law pp. 105–332, 1998, http://www.cccs.edu/Docs/SBCCOE/PerkinsAct1998.pdf (lastaccessed March 2009).

3. M. Rose, The Mind at Work. Viking, New York, 2004.4. D. Laurillard, Rethinking University Teaching: A conversational framework for the effective use of

learning technologies (Second edition). Routeledge, New York, 2002.5. Project Lead The Way, 2008, http://www.pltw.org/index.cfm (last accessed July 2008).6. D. Walcerz, 2009, https://www.trueoutcomes.net/pltw/index.jsf, Accessed on April 27, 2009.7. L. A. Phelps, J. Johnson and K. Alder, The emerging profile of Project Lead the Way in

engineering and technology education: Findings from a national survey of state leaders, FinalReport, Madison, WI: Center on Education and Work, University of Wisconsin-Madison (2006).

8. M. Sanders, The nature of technology education in the U.S., American Society of EngineeringEducation (ASEE), 2008.

9. K. Gray, Is high school career and technical education obsolete? Phi Delta Kappan, 86 (2), 2004,pp. 128–134.

10. K. Levesque, D. Lauen, P. Teitelbaum, M. Alt, and S. Librera, Vocational education in the UnitedStates: Toward the Year 2000 (NCES 2000–029). U. S. Department of Education, National Centerfor Education Statistics, Washington, DC (2000), Retrieved May 21, 2009, from http://nces.ed.gov/pubs2000/2000029.pdf

11. S. Plank, Career and technical education in the balance: An analysis of high school persistence,academic achievement, and post-secondary destinations. Columbus: Ohio State University,National Center for Dissemination, 2001.

12. J. J. Kemple and J. C. Snipes, Executive summary: Career academies impacts on students’engagement and performance in high school, Manpower Development Research Corporation,New York, 2000.

13. J. J. Kemple, Career Academies: Impacts on Labor Market Outcomes and Educational Attain-ment, Manpower Demonstration Research Corporation, New York, 2004.

14. G. Bottom and J. Uhn, Project Lead The WayWorks: A new type of career and technical program.Southern Regional Education Board, September, 2007, Retrieved May 20, 2009, from http://www.sreb.org/programs/hstw/publications/2007pubs/07V29_Research_Brief_PLTW.pdf.

15. M. J. Nathan, N. Tran, L. A. Phelps and A. Prevost, The structure of high school academic andpre-engineering curricula: Mathematics, In Proceedings of the American Society of EngineeringEducation Annual Conference and Exposition, Pittsburgh, PA, 2008.

16. National Council of Teachers of Mathematics, Principles and standards for school mathematics.Reston, VA, 2000.

17. K. Welty, L. Katehi, G. Pearson and M. Feder, Analysis of K-12 Engineering Education Curriculain the United States—A Preliminary Report, In Proceedings of the American Society forEngineering Education Annual Conference and Exposition, Pittsburgh, PA, 2008.

18. S. D. Sheppard, K. Macatangay, A. Colby and W. Sullivan, Educating Engineers: Designing for theFuture of the Field. The Carnegie Foundation for the Advancement of Teaching (2009), http://www.carnegiefoundation.org/dynamic/publications/elibrary_pdf_769.pdf (last accessed March2009).

19. A. Prevost, M. J. Nathan, B. Stein, N. Tran and L. A. Phelps, Integration of mathematics in pre-engineering: the search for explicit connections, In Proceedings of the American Society ofEngineering Education (ASEE), Austin, TX, 2009.

20. N. A. Tran and M. J. Nathan, An investigation of the relationship between pre-college engineeringstudies and student achievement in science and mathematics, Journal of Engineering Education,99(2): 143–158.

21. J. Carroll, A model of school learning, Teachers College Record, 64, 1963, pp. 723–733.22. E. Forman and C. B. Cazden, Exploring Vygotskian perspectives in education: The cognitive

values of peer interaction. In J. Wertsch (ed.), Culture, communication and cognition, pp. 323–347.Cambridge University Press, Cambridge, England, 1985.

23. J. Bransford, A. Brown and R. Cocking, How People Learn: Brain, Mind, Experience and School.National Academy Press, Washington, DC, 2000.

24. R. M. Felder and R. Brent, Designing and teaching courses to satisfy the ABET engineeringcriteria, Journal of Engineering Education, 92 (1), 2003, pp. 7–25.

25. R. Linn and N. Gronlund, Measurement and Assessment in Teaching (8th ed.). Prentice-Hall/Merrill, Upper Saddle River, NJ, 2000.

26. S. M. Barnett and S. J. Ceci, When and where do we apply what we learn? A taxonomy for fartransfer. Psychological Bulletin, 128, 2002, pp. 612–637.

27. D. N. Perkins and G. Salomon, Transfer of Learning, International Encyclopedia of Education,(Second Edition). Pergamon Press, Oxford, England, 1992.

28. R. Rothstein, Class and schools: Using social, economic, and educational reform to close the black-white achievement gap. Economic Policy Institute, Washington, DC, 2004.

29. J. R. Stone, III, C. Alfeld and D. Pearson. Rigor and relevance: Enhancing high school students’math skills through career and technical education. American Educational Research Journal, 45,2008, pp. 767–795.

Natalie A. Tran is an Assistant Professor of Secondary Education at California StateUniversity-Fullerton. Her research focuses on instructional practices and social contextsaffecting student achievement in science. Her methodological interests include hierarchicallinear modeling, experimental design, quasi-experimental design, and case studies.


Mitchell J. Nathan is Professor of Educational Psychology, Curriculum & Instruction, andPsychology at the University of Wisconsin-Madison, and Chair of the Learning Scienceprogram in the School of Education. He is a research fellow at both the Wisconsin Centerfor Education Research and the Center on Education and Work. He uses experimental,quasi-experimental, and discourse-based research methods to understand the nature oflearning and instruction.


The Effects of Pre-Engineering Studies on Mathematics...

Documents

Transcript of The Effects of Pre-Engineering Studies on Mathematics...