A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal...

46
Emilyn Ruble Whitesell, PhD IESP Working Paper #03-15 July 2015 INSTITUTE FOR EDUCATION AND SOCIAL POLICY 665 BROADWAY, SUITE 805 NEW YORK, NY 10012 | 212 998 5880 | 212 995 4564 FAX STEINHARDT.NYU.EDU/IESP A Day at the Museum: The Impact of Field Trips to Informal Science Education Institutions on Middle School Science Achievement

Transcript of A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal...

Page 1: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

Emilyn Ruble Whitesell, PhD IESP Working Paper #03-15 July 2015

INSTITUTE FOR EDUCATION AND SOCIAL POLICY 665 BROADWAY, SUITE 805 NEW YORK, NY 10012 | 212 998 5880 | 212 995 4564 FAX STEINHARDT.NYU.EDU/IESP

A Day at the Museum: The Impact of Field Trips to Informal Science Education Institutions on Middle School Science Achievement

Page 2: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

ABSTRACT

Field trips are a common feature of public education in the United States, but there is little

research on the effect of field trips on student achievement. Using six years of student-level data

from a large-scale program in New York City (approximately 200 schools per year), I estimate

the impact of field trip exposure on students’ performance on New York State’s standardized

eighth-grade science exam. Using school fixed effects to capitalize on variation in field trip

participation within schools over time, I find small positive effects of exposure to field trips on

science test scores and proficiency. These results are important for policy, as they demonstrate

that enrichment and informal learning experiences can contribute positively to student

achievement.

Key words: field trip, informal learning, science achievement, middle school

Page 3: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

Direct correspondence to the author at [email protected]. The author would like to thank the staff of the Urban Advantage program at the Gottesman Center for Science Teaching and Learning at the American Museum of Natural History and the New York City Department of Education Research and Policy Support Group for their help with this study. The author would also like to thank Meryle Weinstein, Leanna Stiefel, Amy Ellen Schwartz, Sean Corcoran, and Jan Blustein for their helpful comments. Major public support for Urban Advantage is provided by the Speaker and the City Council of New York and the New York City Department of Education. Urban Advantage is a program of the Gottesman Center for Science Teaching and Learning at the American Museum of Natural History. Launched in 2004, Urban Advantage is a partnership program among the AMNH, Brooklyn Botanic Garden, New York Botanical Garden, New York Hall of Science, Queens Botanical Garden, Staten Island Zoological Society, the Wildlife Conservation Society’s Bronx Zoo and New York Aquarium, and the New York City Department of Education. The research reported here was also supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B080019 to New York University. The opinions expressed are those of the author and do not represent views of the Institute, the U.S. Department of Education or Urban Advantage

Editorial Board

The editorial board of the IESP working paper series is comprised of senior research staff and affiliated faculty members from NYU's Steinhardt School of Culture, Education, and Human Development and Robert F. Wagner School of Public Service. The following individuals currently serve on our Editorial Board:

Amy Ellen Schwartz Director, IESP

Daniel Patrick Moynihan Professor of Public Policy and Professor of Economics and Public Administration and International Affairs, Syracuse University

Sean Corcoran

Associate Director for Research, IESP NYU Steinhardt

Leslie Santee Siskin

NYU Steinhardt

Leanna Stiefel Associate Director for Education Finance, IESP

NYU Wagner and NYU Steinhardt

Beth Weitzman NYU Steinhardt

Page 4: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 1

I. Introduction

School field trips have long been a feature of public education in the United States, with

teachers across the country occasionally taking students beyond school walls to have informal

learning experiences. In the wake of test-based accountability, however, many schools have

shifted away from field trips and other enrichment (e.g., art, music, physical education) in favor

of in-school practices thought to raise scores, such as allocating more time to tested subjects

and exam preparation. Many parents push back against this emphasis on test preparation and

advocate for a more well-rounded education. For example, in the 2012 New York City (NYC)

parent survey, when asked what single improvement they would most like to see in their

schools, 15.8% of parents indicated more or better enrichment programs; next to smaller class

sizes (23.1%), this was the single most-requested improvement. Furthermore, 13.8% prioritized

more hands-on learning, and 8.5% chose more or better art programs.1

Discussions about field trips and other enrichment activities are often framed as either-or

debates, with an implicit trade-off between the difficult-to-quantify benefits of informal learning,

such as fostering student engagement and developing non-cognitive skills, and performance on

standardized exams. In this context, the potential benefits of informal learning for students’

academic outcomes are often overlooked. While some field trips are primarily recreational, trips

to informal learning institutions (e.g., museums, zoos) in particular often serve specific academic

purposes. Research indicates that these types of informal learning environments contribute to

students’ knowledge and skills differently than school-based instruction (Adams, Gupta, &

DeFelice, 2012; National Research Council, 2009; Phillips, Finkelstein, & Wever-Frerichs,

2007), but no studies have estimated the impact of field trips on standardized test scores.

In this study, I estimate the impact of field trips to informal science education institutions

(ISEIs) on eighth-grade students’ science achievement. Specifically, I use data from the Urban

1 Author’s calculations

Page 5: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 2

Advantage (UA) program in NYC to determine how organized class trips affect science test

scores and proficiency on New York State’s standardized science exam. Using a rigorous

identification strategy and six years of student-level data, I find small positive effects of

exposure to field trips, with larger effects for Hispanic students and those who qualify for free or

reduced-price lunch. These results suggest that schools do not need to sacrifice academic

learning in order to provide well-rounded enrichment experiences; in fact, for disadvantaged

students in particular, field trips can be an important contributor to academic outcomes.

This study is a significant contribution to the literature on field trips, as it is one of few

studies to explore the impact of field trips in a causal framework. Rare in field trip research, I

use a large student-level sample (more than 200 unique schools and approximately 200,000

unique students), follow schools over time, and link field trips to student performance on

standardized tests. Furthermore, I address the endogeneity of exposure to field trips (e.g., more

motivated teachers may take more field trips) more rigorously than most previous studies.

Including school fixed effects partly addresses this issue by controlling for all time-invariant

school characteristics, such as teacher quality. Additionally, in a falsification test I show that

student exposure to field trips after the exam is not related to student test scores.

This paper proceeds as follows: I provide background and context in Section II and

explain the conceptual framework that guides this research in Section III. In Section IV I

describe data and measures, and I present the empirical approach and results in Section V.

Finally, I discuss and conclude in Section VI.

II. Background and context

A. The Urban Advantage program and field trips to ISEIs

The field trips discussed in this paper are made possible by vouchers provided as a

component of the UA program. This section provides an overview of UA, highlighting

Page 6: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 3

programmatic details that are important to the study and paying special attention to field trip

vouchers. A more detailed description of UA is included in a previous paper, which identifies the

positive impact of the overall UA program on students’ eighth-grade science achievement

(Weinstein, Whitesell, & Schwartz, 2014).

Urban Advantage is a partnership between the NYC Department of Education (DOE)

and eight ISEIs in NYC, including the American Museum of Natural History (AMNH), the New

York Hall of Science (NYHS), the Staten Island Zoo (SIZ), the Bronx Zoo (BZ), the New York

Aquarium (NYA), the Brooklyn Botanic Garden (BBG), the Queens Botanical Garden (QBG),

and the New York Botanical Garden (NYBG). In response to a report from the City Council

documenting poor science performance (Miller & Moskowitz, 2004), AMNH spearheaded efforts

to develop a formal-informal partnership2 between these ISEIs and NYC middle schools. UA

focuses on improving instruction and boosting students’ inquiry skills in middle school, with the

proximal goal of improving middle school science achievement and the longer-term aims of

increasing success in high school science and ultimately improving college readiness. UA was

implemented in 36 middle schools in the 2004-05 academic year and has grown considerably

over time. In the analytic dataset used in this study, as of the 2011-12 academic year 320

teachers in 119 schools (approximately 25% of all NYC middle schools) were actively

participating in the program; all together, 21,458 eighth-graders attended UA schools in 2011-

12.

The three key components of UA are professional development (PD), classroom

materials, and access to ISEIs. Teachers new to UA undergo 40 hours of PD across three

cycles in their first year, 22.5 hours of PD in years two and three, and 12.5 hours in year four

and beyond.3 Professional development takes place at participating ISEIs and is administered

2 Formal-informal partnerships are collaborations between formal and informal institutions; in this case, schools (formal education) and ISEIs (informal education). 3 This reflects current attendance requirements, though PD hours have changed slightly over time.

Page 7: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 4

by UA program staff, lead UA teachers, and informal science educators from the host ISEIs.

The PD model uses an immersion-in-inquiry strategy, with teachers engaging in hands-on

activities and learning how to integrate inquiry-based instruction in their classrooms (Loucks-

Horsley & Matsumoto, 2010). To help support inquiry-based pedagogy, participating schools

receive science kits with materials that are geared toward supporting long-term scientific

investigations.4 Each UA teacher also receives funds to purchase additional classroom

materials of her choice.5

Finally, UA teachers and schools receive vouchers that provide free access to any of the

eight participating ISEIs. There are several different types of vouchers: class vouchers allow

teachers to take their classes on field trips during the school day; family field trip vouchers allow

parent coordinators6 to lead family trips to ISEIs on weekends and school holidays;

student/family vouchers allow students to attend ISEIs outside of school; and teacher/parent

coordinator vouchers allow teachers and parent coordinators to attend ISEIs independently and

are intended to facilitate field trip planning.7 Only teachers currently participating in UA receive

vouchers, but students who do not have UA teachers may also benefit. For example, any

student at the school can attend a weekend family field trip, regardless of whether or not she

has a UA teacher. Furthermore, occasionally UA teachers will share class field trip vouchers

with other teachers at their schools.8 Thus, while vouchers are associated with participating

teachers, they can be a broader, school-level resource.

4 In NYC public schools, eighth-grade students are required to complete a long-term science “exit project.” 5 Through 2011, funding was based on each teacher’s number of students; since 2012, yearly funding has been the same for all teachers. 6 Parent coordinators are full-time school staff who support family engagement with schools. In UA schools, parent coordinators help facilitate the UA program, for example by hosting UA information sessions, helping teachers plan class trips, and arranging family field trips. Parent coordinators attend UA workshops to train them for this role. 7 Current guidelines stipulate that up to 40 individuals can attend a class or family field trip using one class or family field trip voucher. Student/family vouchers and teacher/parent coordinator vouchers provide free access for up to 4 individuals (the student, teacher, or parent coordinator plus three guests). 8 The UA program discourages UA teachers from sharing vouchers with non-UA teachers, as field trips are intended to be embedded in the larger UA program. In particular, because UA teachers receive professional development around facilitating high-quality field trips, they may be able to use the class field trip vouchers more effectively.

Page 8: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 5

In all cases, vouchers provide access to resources that would otherwise cost schools or

families money. Most ISEIs charge a base admission fee and also have additional charges for

special exhibits, class workshops, etc. In the few cases where admission to an ISEI is always

free for students, UA vouchers provide access to additional services that would otherwise incur

a fee.9 Because the DOE provides free yellow bus transportation and free subway transportation

for school field trips, UA teachers can take their students on class trips entirely for free.10

Teachers describe using vouchers in a variety of ways. Some UA teachers interviewed

in a previous study (Weinstein, Whitesell, & Leardo, 2014) indicate they use vouchers to

facilitate field trips that are structured to fit into the science curriculum. For example, some

teachers take students to ISEIs in order to collect data for long-term science investigations,

while others plan hands-on learning experiences that correspond to specific units and

standards. For these types of trips, teachers typically share learning goals with students,

prepare students for the excursion, and have post-trip activities to build on learning. Other

teachers, however, report using field trips for more of an enrichment experience, with the goal of

increasing student engagement. For these types of trips, teachers may not have a specific

learning objective. Of course, teachers may take multiple field trips throughout the year, and

each may have a unique purpose and structure.

While many factors influence teachers’ ability and motivation to take class field trips, UA

vouchers can help facilitate class trips by alleviating some challenges. For example, research

suggests that rising costs and budget restrictions impose considerable financial difficulties for

teachers who hope to take field trips (Anderson, Kisiel, & Storksdieck, 2006; DeWitt &

Storksdieck, 2008). By providing free access to ISEIs, vouchers relieve some of this financial

pressure. Additionally, many teachers report working closely with ISEI staff to coordinate trip

9 For example, admission to AMNH is free for all NYC public school groups, but the UA voucher provides access to specific resources that would otherwise have a fee (e.g., IMAX movies, the planetarium, workshops). 10 Current guidelines for securing free bus or subway transportation from the DOE are provided on the website of the Office of Pupil Transportation: http://www.optnyc.org/schools/FieldTrips.htm. UA provides free bus transportation for family field trips organized by parent coordinators.

Page 9: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 6

logistics, which can be onerous.11 Finally, administrators and teachers may be hesitant to take

field trips if they perceive trips to detract from necessary class time – particularly in response to

accountability concerns (Anderson, Kisiel, & Storksdieck, 2006; DeWitt & Storksdieck, 2008).

The extensive PD UA teachers receive around integrating inquiry experiences and planning field

trips may increase confidence in their ability to design trips that contribute meaningfully to

students’ education outcomes, including test scores.

B. Literature review

1. Informal and supplemental learning experiences

Many different types of policies and programs seek to increase students’ access to

informal and out-of-school learning opportunities. Support for this strategy is found in research

on in-school and out-of-school contributions to learning. Several studies find schools increase

achievement for high- and low-income students at a similar rate and conclude achievement

gaps are largely driven by children’s early childhood experiences and how they spend their time

after school and in the summer (Alexander, Entwisle & Olson, 2001, 2007; Downey, von Hippel,

& Broh, 2004; Entwisle & Alexander, 1992, 1994; Heyns, 1978, 1987; Murnane, 1975).

Citing this research, Alan Krueger published an editorial in the New York Times

describing how out-of-school inequity (and particularly disparate summer experiences) is a key

driver of achievement gaps in the United States. To address this problem, he proposed Summer

Opportunity Scholarships that could be used to provide disadvantaged students with access to

summer learning experiences (Krueger, 2000). Similarly, Alexander, Entwisle, and Olson (2001)

argue that achievement gaps could be greatly diminished by providing low-income students with

the types of enriching out-of-school experiences their wealthier counterparts receive, such as

11 See the appendix for UA’s 2014-15 guidelines for taking class trips.

Page 10: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 7

“small group reading experiences, visitations to art museums and other cultural settings,

attendance at summer camps, and one-on-one tutoring.”

One prominent example of providing out-of-school education resources to low-income

students is found in the Supplemental Educational Services (SES) provision of the No Child Left

Behind (NCLB) Act of 2001. Federal law requires schools deemed low performing for three

consecutive years to offer low-income students options for free before- or after-school tutoring

with a state-licensed provider. The SES provision aims to level the playing field by providing

low-income students with access to resources that have typically only been available to mid-

and high-income students (Henig, 2006). Research on the impact of SES tutoring is mixed but

does not provide strong evidence of positive effects. Many studies find no impact of SES on

student test scores (Burch, 2007; Deke, Gill, Dragoset, & Bogen, 2014; Heinrich, Meyer, &

Whitten, 2010; Muñoz, Potter, & Ross, 2008, Ross et al., 2008), though in some districts and in

particular for students with high participation, SES can have a positive effect on student

achievement (Rickles, & Barnhart, 2007; Rickles & White, 2006; Ryan & Fatani, 2005; Zimmer

et al., 2007).

2. Field trips

Field trips are typically intended to strengthen students’ understanding of content, to

expose them to broader educational settings, or to provide rewards (Gottfried, 1980; Greene,

Kisida, & Bowen, 2014). Much of the research on field trips focuses on characteristics of high-

quality experiences, identifing appropriate pedagogical approaches for informal education

settings. In their summary of the literature, DeWitt and Storksdieck (2008) conclude that

teachers should integrate trips into their class curriculum, preview and follow up trips during

class time, and provide opportunities for students to engage in inquiry-based exploration and

hands-on learning while on excursions.

There is little rigorous research on how field trips affect student outcomes. Many studies

use qualitative methods to understand students’ experiences on field trips, while quantitative

Page 11: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 8

studies that connect field trips to students’ affective and cognitive outcomes are mostly

descriptive or correlational (DeWitt & Storksdieck, 2008). Research on affective outcomes

typically uses student surveys and finds that field trips can improve motivation and increase

interest in the subject area (e.g., Anderson, Kisiel, & Storksdieck, 2006; Gottfried, 1980; Jarvis &

Pell, 2002, 2005; Knapp, 2000; Sibthorp & Knapp, 1998). Studies of cognitive outcomes tend to

use targeted post-tests and focus narrowly on knowledge and skills related to specific exhibits,

such as recalling particular facts and describing processes (e.g., Anderson, 1999; Anderson &

Lucas, 1997; Bamberger & Tal, 2007; Falk, 1983; Falk & Dierking, 1997; Gottfried, 1980;

Miglietta, Belmonte, & Boero, 2008; Orion & Hofstein, 1991).

The most rigorous research on the impact of field trips on students’ cognitive outcomes

uses an experimental design to randomly assign classes to visit an art museum. In addition to

fact recall, this study identifies impacts on students’ ability to interpret and describe art, finding

positive effects that are largest for disadvantaged students (Greene, Kisida, & Bowen, 2014).

While this study is encouraging and the methods support causal inference, the outcome it

explores is narrowly defined, and it is unclear whether or not effects would translate to critical

thinking more broadly or students’ test scores. Taken together, the body of research indicates

field trips can influence students’ knowledge and skills, though no known previous studies link

field trips to traditionally measured academic outcomes, such as standardized test scores.

III. Conceptual framework

Though UA provides multiple types of vouchers that allow students to attend ISEIs, I

focus on the relationship between class field trips and students’ science test scores for several

reasons. First, field trips are important from a policy perspective, as they are a specific type of

intervention that can be facilitated during school time. Second, field trips may be more closely

related to student achievement than other types of visits, as they are planned and led by

Page 12: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 9

teachers and may be directly integrated into the curriculum. Third, class trips are appealing in a

study of student outcomes because they are driven by teacher decisions – not student

decisions. (Of course, teachers’ willingness to take class trips may be related to their students’

characteristics, including behavior, interest, and responses to previous class trips.) By contrast,

student/family trips will be endogenous, as the students who visit ISEIs on the weekends may

be more interested in science or have more invested parents than their peers.

The conceptual framework that guides this research is shown below. Class trips to ISEIs

may affect students’ science achievement through two key mechanisms. First, class trips to

ISEIs may influence students’ cognitive skills (arrow A). For example, exposure to ISEIs may

contribute to students’ conceptual understanding, develop students’ ability to engage in

scientific inquiry, or increase students’ knowledge of key facts and principles. The development

of these cognitive skills may then lead to improved student science outcomes (arrow B).

Second, class trips to ISEIs may improve students’ affective characteristics, such as motivation

or interest in science (arrow C).12 This in turn could influence students’ behavior (arrow D), for

example causing them to participate more actively in class or to undertake more challenging

projects. These changes in behavior may directly improve students’ performance on the

standardized science exam if, for example, students are better able to focus or if they exert

more effort on the test (arrow E). Additionally, improved behaviors can lead to improved

cognitive outcomes; for example, completing homework at higher rates can improve students’

understanding of the science material (arrow F).

12 Changes in affective characteristics, such as motivation and interest in science, is a key mechanism linking exposure to ISEIs to student science achievement. This highlights the endogeneity issue that would arise when studying the impact of student-initiated trips to ISEIs: trips may affect motivation/interest, but motivation/interest can also affect trip-taking. As previously described, this is one reason I focus on class trips, not student/family trips.

Page 13: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 10

Finally, field trip quality is a key moderator. Though I do not have measures of field trip

quality in this study, the extent to which a trip is aligned with the curriculum, is well planned, is

integrated with class lessons, and encourages students to participate in hands-on activities are

all theoretically important factors affecting the extent to which field trips improve students’

cognitive characteristics, affective characteristics, and ultimately test scores

IV. Data, measures, and descriptive statistics

A. Data

This study uses data from four main sources: administrative records from UA, student-

level administrative files from the DOE, the New York State School Report Cards (SRCs), and

the National Center for Education Statistics’ Common Core of Data (CCD). UA program staff

provided data on participating schools in each year, including the number of teachers at each

school participating in UA. Additionally, UA provided voucher-level data from 2007-2012; for

each voucher used, data include the type of voucher (e.g., class or student), the number of

visitors, the date of the trip, the ISEI visited, and the UA teacher and school. Voucher data do

not identify specific students who attend class trips or who use student/family vouchers.

Page 14: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 11

Student-level data from the DOE include information on race, poverty status

(qualification for free or reduced-price lunch), special education eligibility (SPED), limited

English proficiency (LEP), and standardized science, math, and English language arts (ELA)

test scores.13 School-level data from the SRCs provide school characteristics, including the

percent of students in the school who are black, Hispanic, Asian, white, poor, and LEP. Average

teacher characteristics include the percent with fewer than three years of experience in the

school and the percent with Master’s degrees or higher. The SRCs also include total enrollment,

the student attendance rate, and the teacher turnover rate. The CCD provides schools’ latitude

and longitude coordinates, which I use to calculate the distance between schools and ISEIs.

Using these data, I create a student-level panel for the years 2007-2012 that includes

eighth-graders attending schools that ever participate in UA during this time period.14 Schools

may enter and/or leave UA between 2007 and 2012, so the sample includes some students at

non-active schools. In each year, between 57 and 73 percent of the schools in the sample are

actively participating in UA.15 To be clear, all schools in the sample participate in UA at some

point between 2007 and 2012. By limiting the sample to these schools, I guard against the fact

that UA schools may be different in some ways than non-UA schools; in effect, this holds

constant the factors that are associated with UA participation.

The analytic sample does not include schools from Staten Island (though they are

included in a robustness test), as Staten Island is relatively isolated from the other boroughs in

NYC, includes only ten schools that ever participate in UA, and features just one participating

ISEI. Transportation from Staten Island to other NYC boroughs is logistically challenging and

13 Standardized science exams are given in grades 4 and 8; math and ELA exams are given in grades 3-8. Unique student identifiers allow me to follow students over time and control for prior achievement. 14 While UA has been in operation since the 2004-05 academic year, voucher data were not systematically collected and made available until 2006-07. 15 Of the 171 schools in the sample in 2007, 109 (63.7%) were actively participating in UA. Of the 195 schools in the sample in 2012, 112 (57.4%) were actively participating. See Appendix Table A1 for additional information.

Page 15: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 12

time consuming, and field trip behavior in Staten Island is likely different than in other boroughs.

The analytic data set includes 196,764 unique students and 209 unique schools.

To reflect students’ science achievement, I use two different measures of performance

on the eighth-grade Intermediate Level Science (ILS) exam, which is administered across New

York State. First, I use z-scores, which are created from yearly scale scores and are

standardized to have a mean of zero and a standard deviation of one; positive (negative) values

reflect above-average (below-average) performance citywide. Additionally, I use science

proficiency as a dichotomous outcome. Students are proficient if they earn a performance level

of 3 or 4 out of 4 on the ILS, based on their overall exam score.

Unfortunately, middle school students cannot be matched to their science teachers in

this time period; furthermore, UA does not have historic data on the grades taught by UA

teachers, so students and teachers cannot be matched by grade within schools. Due to these

data limitations, I define UA participation at the school level and classify any school that has at

least one participating teacher as a UA school. This is a conservative definition, as there will be

many students who attend UA schools but who do not have UA teachers.

Field trip measures are also created at the school level. First, exposure to field trips

before the science exam is calculated by dividing the total number of student field trip attendees

before the exam by the number of students in grades that are eligible to participate in UA. Pre-

exam field trips are identified using New York State testing windows. The exam was given in

early May from 2007-2009 and in early June from 2010-2012. Thus, pre-exam field trip

measures include trips taken from September to April for 2007-2009 and trips taken from

September to May for 2010-2012. Because only seventh- and eighth-grade teachers were

eligible to participate in UA from 2007-2009, the exposure rate in these years is based on

enrollment in grades seven and eight. Sixth-grade teachers were invited to participate in UA

starting in 2010, so enrollment in grades six through eight is used as the denominator for 2010-

Page 16: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 13

2012.16 The second key field trip measure is the number of class trips per UA teacher (the

school’s total number of class trips taken before the science exam divided by the school’s total

number of UA teachers).

Each measure of field trip activity presents strengths and limitations. The student

exposure measure reflects the average number of times students in eligible UA grades attend

UA field trips before the science exam, which is theoretically important but imprecisely

measured. For example, imagine a school where all eighth-grade students have UA teachers

and attend class trips, but seventh-grade teachers do not participate in the program. For these

eighth-graders, the student exposure measure will underestimate participation in field trips. At

the same time, there will be many schools where the exposure rate overestimates eighth-

graders’ participation in field trips. Given the measurement error for individual student exposure,

the exposure rate should be thought of as capturing the school’s overall participation in science

field trips. While the number of class trips taken per UA teacher is straightforward and precisely

measured, it is a measure of teacher behavior, not student exposure; thus, it is used as a

robustness test.

Two other important measures are used in robustness tests. First is students’ exposure

to independent trips (using student/family vouchers) before the exam, which may be a substitute

or complement for field trips. This rate is calculated by dividing the total number of independent

student visits before the exam by the number of students in eligible UA grades.17 Second, I

calculate the estimated percentage of science teachers in each school participating in UA.18

16 To be clear, from 2007-2009 student exposure is calculated by dividing the total number of visitors on class trips from Sept. to Apr. by enrollment in grades seven and eight. Student exposure from 2010-2012 is calculated by dividing the total number of visitors on class trips from Sept. to May by enrollment in grades six through eight. 17 I assume each student/family trip includes one student who is enrolled in a UA grade at the associated school. 18 Given the average middle school science class size is 27 students (NYC IBO, 2011) and assuming each teacher has five classes, middle school science teachers have 135 students, on average. I assume each sixth- through eighth-grade student takes one science class. To calculate the approximate number of science teachers in a school, I divide each school’s sixth- through eighth-grade enrollment by 135.

Page 17: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 14

C. Descriptive statistics

As shown in Figure 1, participating ISEIs and UA schools are spread across the city in

all five boroughs. Most schools are far enough away from even the closest ISEI to require

transportation for field trips. On average, schools in the analytic sample are 2.52 miles away

from the nearest participating ISEI, with schools at the 25th percentile 1.63 miles away and

schools at the 75th percentile 3.16 miles away from the closest ISEI.19

Figure 2 shows the distribution of total class trips (excluding those taken by Staten

Island schools) taken in each year, by ISEI; numbers include all trips taken between September

and June in each year. The total number of field trips increases over time, from 497 in 2007 to

771 in 2012. In all years, the American Museum of Natural History (AMNH) in Manhattan, the

New York Hall of Science (NYHS) in Queens, and the New York Aquarium (NYA) in Brooklyn

are the most popular destinations for class trips. In 2012, for example, 28% of all class trips

were to AMNH, 23% were to NYHS, and 18% were to NYA. The Staten Island Zoo (SIZ) and

Bronx Zoo (BZ) each typically host 5-15% of all class trips, and in 2012 each had 11% of field

trips. The botanical gardens are the least popular destinations for field trips. In 2012, the New

York Botanical Garden, the Queens Botanical Garden, and the Brooklyn Botanic Garden each

hosted approximately 2-4% of all class trips. This uneven distribution of trips across sites could

be due to a variety of factors, including fit with the middle school science curriculum, teacher or

student interest, and convenience (e.g., proximity to subways).

Figure 3 shows the distribution of class trips over time, by month. Teachers take very

few class trips early in the academic year and are much more likely to take field trips at the end

of the school year. For example, in 2012 1-2% of trips were taken in September and October, 5-

15% of all trips were taken in each month from November to April, and 44% percent of trips

were taken in May. Note that across all years, approximately 35-45% of all trips are taken in

19 Mileage is based on a direct-line route and does not reflect driving distances or public transportation times.

Page 18: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 15

May and June, combined. Beginning in 2012, vouchers expired at the end of May, and the

number of trips taken in May increased dramatically as teachers shifted trips earlier in the year.

This uneven distribution of trips throughout the school year is likely due to several

factors. First is an administrative point: vouchers are not typically distributed from UA until

October. Second, field trips require considerable advance planning and coordination. The UA

class trip guidelines (see appendix) recommend that planning begins at least two months in

advance, and there are several logistical tasks to complete (e.g., obtaining principal approval,

gathering signed permission slips from parents, securing transportation). Furthermore, ISEIs

typically require teachers to schedule field trips 2-5 weeks in advance.20 Third, teachers may

plan field trips at strategic times to integrate with curricular goals (e.g., collecting data for

science investigations), to attend specific exhibits, or to align with the school schedule (e.g.,

avoiding testing dates).

Turning to school-level measures of voucher use, Figure 4 shows average student

exposure to class field trips and the average number of science trips per UA teacher in active

UA schools before the exam. Student exposure more than doubles from 2007-2012, increasing

from an average of 0.21 trips per student in 2007 to 0.51 trips per student in 2012. The number

of class trips per UA teacher also increases dramatically over time, from 1.34 trips per UA

teacher in 2007 to 2.50 trips in 2012. There are relatively large increases in these field trip

measures from 2011 to 2012, due in part to changes in voucher use guidelines. As previously

described, beginning in 2012 vouchers expired at the end of May, and distributions over time

suggest schools shifted trips from June (after the exam) to May (before the exam) in order to

use their vouchers before the expiration date.

Table 1 shows the characteristics of active UA schools in 2012 by quartile of student

exposure to class trips. In quartile 1 schools, students attend an average of 0.02 trips, while

20 Guidelines for planning and scheduling field trips to each of the ISEIs can be found on their websites.

Page 19: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 16

students in quartile 2 schools attend 0.28 trips, those in quartile 3 schools attend 0.55 trips, and

those in the highest quartile attend 1.18 trips before the exam. Student exposure to independent

trips (student/family trips) also increases by quartile,21 as does the number of class trips per UA

teacher. There are no consistent differences in student characteristics across quartiles, with

similar percentages of students by race, poverty status, and LEP as well as similar attendance

rates across the distribution. In terms of teacher characteristics, schools with lower percentages

of teachers with Master’s degrees, lower teacher turnover rates, and higher percentages of

teachers participating in UA have greater student exposure to field trips. Schools with greater

field trip participation tend to have lower previous science, math, and ELA proficiency rates.

Finally, smaller schools, elementary-middle schools (e.g., K-8 schools), and those in Manhattan

and Brooklyn have relatively high field trip exposure.

V. Estimating the effect of field trips to ISEIs on student outcomes

A. Model

To estimate the impact of exposure to field trips on students’ science outcomes, I use an

ordinary least squares (OLS) model with school fixed effects, as follows:

(1) 𝑆𝑐𝑖𝑒𝑛𝑐𝑒𝑖𝑠𝑡 = 𝛽0 + 𝛽1𝐹𝑖𝑒𝑙𝑑𝑇𝑟𝑖𝑝𝑠𝑡 + 𝑆𝑡𝑢𝑑𝑒𝑛𝑡𝑖𝑠𝑡′ 𝛽2 + 𝑆𝑐ℎ𝑜𝑜𝑙𝑠𝑡

′ 𝛽3 + 𝛿𝑡 + 𝜃𝑠 + 휀𝑖𝑠𝑡

Here, Science is a science outcome (either a z-score or proficiency indicator) for student i in

school s in year t. When using the proficiency indicator as an outcome, this is a linear probability

model.22 FieldTrip captures field trip participation for school s in year t: either the continuous

measure of visits per student or indicator variables reflecting ranges of field trip exposure.

21 The correlation between student exposure to field trips and student exposure to independent trips is 0.37 overall and 0.27 for active UA schools in the analytic sample. 22 Because of the large number of school effects, it is inappropriate to use a Maximum Likelihood Estimator, such as probit or logit, as these would provide inconsistent estimates (Neyman & Scott, 1948; Greene, 2004).

Page 20: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 17

Student is a vector of student characteristics, including lagged math z-score, race,

poverty status, participation in SPED, and LEP classification. School is a vector of time-varying

school characteristics, including UA status (an indicator variable that takes a value of one in

years the school has at least one active UA teacher) and traditional controls: average student

characteristics (percent black, Hispanic, Asian, poor, and LEP), average teacher characteristics

(percent with fewer than three years of experience in the school, percent with Master’s

degrees), and other school factors (student attendance rate, teacher turnover rate, natural log of

total enrollment). Finally, the model includes year effects (δ) and school effects (θ) to remove

any systematic effects of years (e.g., a particularly difficult exam) or schools (e.g., culture,

location) and to isolate within-school variation over time. Robust standard errors are clustered at

the school-year level.

Though school fixed effects effectively control for all time-invariant school

characteristics, such as culture, teacher autonomy, and grade span, one may still be concerned

that field trip participation is endogenous. For example, field trip activity may be a proxy for

teacher quality, as the teachers who are more motivated or skilled may be more likely to take

class trips and also better able to contribute to students’ science learning. Teachers may also

be more likely to take students on class trips if their students are particularly interested in

science or well behaved. For these reasons, students in schools with high field trip exposure

rates may be more successful in science, regardless of whether or not they attend field trips.

Note, however, that to bias estimates of the effect of field trips, these factors (e.g., teacher

motivation) must vary within schools over time in a way that is related to both field trip exposure

and student outcomes.

In order to gauge the extent to which this endogeneity is an issue, I use a falsification

test to estimate the impact of class trips after the science exam on students’ science test

scores. The logic of this falsification test is that trips after the exam cannot have a causal effect

on student test scores, and so any relationship between post-test field trips and student

Page 21: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 18

achievement would indicate that field trip measures are correlated with other factors that affect

student achievement. To be clear, a positive “impact” of post-exam field trips would lead one to

be concerned that field trip measures simply reflect other school characteristics, such as

teacher quality. Finding no relationship between post-test field trips and student outcomes

would support the fixed effects identification strategy and increase confidence that the estimated

effects from the main models reflect the causal effect of class field trips.23

B. Results

Table 2 shows estimates of the impact of student exposure to class trips before the

science exam on science z-scores (columns 1 – 3) and proficiency (column 4 – 6). In all cases,

field trip exposure has a small but statistically insignificant positive relationship with test

performance. The model presented in column 1, which does not include school fixed effects or

students’ lagged math z-scores, indicates that one additional visit per student is associated with

an increase in science test scores of 0.049 standard deviations (SDs); once lagged math test

scores are included (column 2), the coefficient is reduced to 0.018, and the point estimate is

similar (0.016) after including school fixed effects (column 3). Columns 4 – 6 present similar

specifications, but because the outcome is science proficiency, these are linear probability

models and coefficients reflect predicted probabilities. In the specification without lagged math

test scores (column 4), each additional visit per student is associated with an increased

probability of being proficient on the exam of 1.9 percentage points (pp). This point estimate is

reduced after including lagged math test scores (0.7 pp more likely to be proficient, column 5)

and school fixed effects (0.3 pp, column 6).

Results from these models suggest a positive relationship between field trips and test

scores, but the impact of one additional class trip per student is statistically insignificant. The

23 One assumption of this falsification test is that the timing of the trip (before or after the exam) is random with respect to other school factors that may influence test scores; that is, if higher-quality teachers are more likely to take trips before the exam (while lower-quality teachers are more likely to take trips after the exam), estimates of the impact of pre-test field trips may be biased.

Page 22: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 19

continuous measure may not be the most appropriate way to operationalize exposure, however,

as one additional trip per student is a relatively large increase. For students in UA schools in

2012, one additional trip per student is similar to moving from the 10th percentile of exposure to

class trips (0.00) to the 95th percentile (1.04). It is potentially more meaningful to group schools

based on ranges of field trip exposure. Accordingly, I construct indicator variables reflecting

exposure rates of at least 0.25, 0.25 to 0.50, and greater than 0.50 trips per student.24 Across all

years and schools (including non-active UA schools), 22.9% of students attend a school with

exposure rates of 0.25 or greater, with 10.7% of students in the 0.25-0.50 group and 12.2% of

students exposed to at least 0.50 trips before the test, on average.25

Table 3 presents results from models that use indicator variables to capture student

exposure; all models include lagged math z-scores and school fixed effects. As shown in

column 1, students attending schools with at least 0.25 visits per student perform 0.026 SDs

higher on the eighth-grade science exam than students attending schools with lower exposure

rates. As shown in column 2, point estimates are similar for students attending schools with 0.25

to 0.50 trips per student (0.027) and for students attending schools with more than 0.50 trips per

student (0.025), though the effect is only statistically significant for the 0.25-0.50 group.

Results from proficiency models are shown in columns 3 and 4. In schools with at least

0.25 visits per student, students are 1.2 pp more likely to be proficient on the science exam than

students in schools with fewer visits per student (column 3). This effect is driven by schools with

moderate voucher usage. In schools with 0.25 to 0.50 visits per student, students are 1.6 pp

more likely to be proficient, and there is no statistically significant effect of attending a school

with greater than 0.50 visits per student (column 4).

For both z-score and proficiency outcomes, the estimated coefficients for attending a

school with 0.25 to 0.50 trips per student and for attending a school with greater than 0.50 trips

24 In robustness tests I show results are not sensitive to these cut-points. 25 The number and percent of schools and students in each of these categories are shown in appendix Table A2.

Page 23: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 20

per student are statistically indistinguishable from one another. Thus, my preferred specification

includes only one indicator variable, which takes a value of one if a school has at least 0.25 trips

per student. Table 4 presents results for this model estimated on different student subgroups.

Panel A shows effects on z-scores, and results can be compared to the main effect in Table 3,

column 1. The only statistically significant effects are for Hispanic and poor students, who

perform 0.043 SDs and 0.033 SDs higher on the science exam, respectively, when they are

exposed to an average of at least 0.25 trips. Estimated coefficients are positive but not

statistically significant for LEP (0.022), SPED (0.016), white (0.016), black (0.015), and non-poor

students (0.006). I estimate a negative and statistically insignificant effect for Asian students (-

0.015).

Results from proficiency models are similar. As shown in Panel B, estimated effects are

positive and statistically significant for white (0.020), Hispanic (0.019), and poor students

(0.017), who are approximately 1.7 to 2.0 pp more likely to be proficient on the science exam

when they attend schools with exposure rates of at least 0.25. Small, statistically insignificant

point estimates are positive for LEP (0.012), SPED (0.008), and black students (0.006) but

negative for Asian (-0.008) and non-poor students (-0.008).

C. Robustness tests

I estimate a series of robustness tests to determine if results are sensitive to alternative

sample, modeling, or measurement decisions. Results from these robustness tests are shown in

Table 5, with the main estimated effect of attending a school with at least 0.25 trips per student

on students’ science z-scores reproduced in column 1 for comparison. First, to ensure results

are not driven by the choice of analytic sample, I include Staten Island schools in the regression

sample. In this specification (column 2), the estimated coefficient on exposure to at least 0.25

trips of 0.019 is similar to the main effect, though it is not statistically significant.

Next, because field trips and independent student trips are positively correlated, one

may be concerned that the estimated effect of exposure to class field trips at least partially

Page 24: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 21

reflects participation in student/family trips. To address this, I control for the number of

independent trips per student. In this specification (column 3), the estimated effect of exposure

to class trips of 0.024 is quite similar to the main results. Similarly, because the percent of

science teachers participating in UA is potentially related to both field trip exposure and student

test scores, I estimate a model controlling for the percent of science teachers in UA. In this

specification (column 4), the estimated effect of 0.029 is essentially unchanged from the main

results.

I next address alternative measures of field trip participation, including different cut-

points for grouping students by field trip exposure. Column 5 shows the impact of attending a

school with at least 0.20 trips per student (0.029), while column 6 shows the impact of attending

a school with at least 0.33 trips per student (0.028); results indicate findings are robust to these

alternative measurement choices. Finally, instead of student exposure to field trips, I estimate

the impact of attending a school with at least one field trip per UA teacher (column 7). While not

statistically significant, the point estimate of 0.021 is of a similar magnitude to the student

exposure results. Taken together, results from these robustness tests indicate the main

estimated effects are not driven by sample, modeling, or measurement choices.26

D. Falsification test

As a falsification test, I estimate the impact of class trips taken after the science exam on

students’ science test scores. Because of changes in the testing dates, trips taken in May and

June are included in the post-test measures from 2007-2009, and trips taken in June are

included for 2010-2012.27 Results from this falsification test are shown in Table 6. Columns 1

and 2 present results for z-scores, while columns 3 and 4 show effects on proficiency. For both

outcomes, I estimate a specification using the continuous measure of student exposure

26 Results for the robustness tests using proficiency as the outcome are shown in the appendix, Table A3. 27 In 2012, vouchers expired at the end of May and there were no June field trips; thus, the post-exam field trip measure equals zero for all schools in 2012.

Page 25: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 22

(columns 1 and 3) and a model using an indicator for at least 0.25 visits per student after the

exam (columns 2 and 4). Across all models, point estimates are small, statistically insignificant,

and negative, indicating that trips taken after the exam are not systematically related to science

performance. Results from these models provide evidence that the main estimated effects

reflect the impact of field trips, not simply unobserved school characteristics correlated with both

field trip exposure and test scores. That is, effects capture the unique impact of field trips, not

simply the effect of attending the type of school where teachers take field trips.

VI. Discussion and conclusion

A. Contributions and limitations

This study makes an important contribution to the literature on school field trips and has

implications for research on informal education experiences more broadly. This is one of a very

small group of studies that identifies the impact of field trips in a causal framework. Previous

quantitative studies suggest relationships between field trip participation and both cognitive and

affective outcomes, but they are narrowly focused, tend to use small sample sizes, and have not

to my knowledge been able to link field trip data to standardized test scores. This study is

notable for exploring a large-scale program, using a large sample with six years of data, linking

field trip data to administrative records on participating schools and students, and employing a

rigorous identification strategy to identify plausibly causal effects on academic performance.

I find small positive effects of student exposure to field trips, as attending a school with

at least 0.25 pre-exam field trips per student improves eighth-graders’ science test scores by

approximately 0.026 SDs and increases their likelihood of being proficient on the exam by 1.2

percentage points. Furthermore, I find that trips to ISEIs are most effective for relatively

disadvantaged students. Estimated effects are largest for Hispanic and poor students, who

Page 26: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 23

perform 0.033 – 0.043 SDs higher and who are 1.7 to 1.9 percentage points more likely to be

proficient on the science exam when they attend schools with at least 0.25 trips per student

While these effects are small, results are robust to different samples, covariates, and

measures of field trip participation. Furthermore, finding no impact of trips after the exam on

student achievement suggests the main results reflect true effects – that is, field trip measures

are not simply a proxy for school or teacher quality. It is also important to emphasize that the

measures I am able to construct are school-level measures that assign all students in a school

the same level of field trip exposure; because many students who do not experience class trips

are considered treated, the results likely underestimate the impact of field trip attendance. That

is, results should be considered lower bound estimates for the impact of field trips in this

context.

To understand the practical significance of these findings, it is useful to consider the

nature of the intervention. This study focuses on a group of schools that are already

participating in a science program that provides high-quality professional development and

science materials. My results suggest that even among these schools that are receiving multiple

science supports, field trips to science institutions positively influence student science

performance. Additionally, class trips are relatively non-invasive and inexpensive. Taking a field

trip does not require an ongoing commitment from teachers or any specific curriculum or

pedagogical approaches in class. The eight ISEIs participating in UA have different policies and

costs for class trips, but they typically charge between $0 and $9 per student for admission, and

additional workshops (free with vouchers) otherwise cost between $150 and $500 per class.28

This study has several key limitations. Critically, the data do not indicate which students

attend field trips, and field trip measures are necessarily created at the school level.

Additionally, data do not include quality measures for ISEI visits, but as described in the

28 Specific policies and prices are available on the websites of participating ISEIs.

Page 27: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 24

conceptual framework, field trip quality may moderate the relationship between field trips and

student outcomes. In the future, it would be useful to collect quantitative and qualitative data on

how teachers plan for trips, how trips are related to the class curriculum, which content areas

are addressed, what activities students engage in while at ISEIs, and the length of class visits in

order to inform the conceptual framework and provide recommendations for field trip design.

A third limitation is that the mechanism through which visits to ISEIs affect students’

performance on science exams is unclear. Exposure to ISEIs can influence cognitive outcomes

(e.g., conceptual understandings, knowledge of key facts) and affective outcomes (e.g.,

motivation, interest). This study finds an overall positive effect of field trips to ISEIs on students’

science test performance, but it does not speak to the mechanism through which this occurs.

Future research should measure intermediate outcomes (e.g., understanding of the scientific

method, interest in science) in order to test the relative contribution of cognitive and affective

changes.

Finally, results from this study may not generalize to field trips more broadly. The field

trips explored in this study are nested in a broader science intervention that connects teachers

with ISEIs and features intensive and ongoing professional development. Notably, PD takes

place at ISEIs, familiarizes teachers with the resources of these institutions, provides training on

leading effective trips, and helps teachers establish relationships with informal educators at

participating ISEIs. On the one hand, this is an ideal setting for studying field trips, as it provides

evidence about the potential class field trips have for improving student achievement. On the

other hand, the UA context limits the generalizability of findings, as this is not the typical setting

in which teachers organize field trips. To be clear, simply providing free access to ISEIs without

training teachers to lead high-quality field trips may not have similarly positive effects.

B. Conclusion

Students’ educational outcomes are influenced by many different factors, both in and out

of school. Much research focuses explicitly on in-school resources, such as class size and

Page 28: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 25

teacher quality, while other studies address outside factors, including family and neighborhood

characteristics. Increasingly, however, formal-informal partnerships provide a multi-pronged

approach to improving student outcomes. Urban Advantage is one such program, bringing ISEIs

together with middle school science teachers in NYC to improve science instruction and

outcomes. In addition to PD and classroom materials, UA provides vouchers that allow teachers

to take their classes on field trips to ISEIs for free.

Policymakers are perpetually engaged in debates about what school outcomes matter

and how to allocate resources in order to reach educational goals. To oversimplify, some are

focused on traditional academic achievement measures and direct school-based resources to a

few inputs that seem closely related to test scores (e.g., teacher quality, longer school days);

others seek more inclusive definitions of student success and call for well-rounded educational

experiences. Though it is often framed as such, this is not necessarily an either-or policy issue.

With unique, large-scale data from a middle school science program in NYC, I provide some of

the first evidence that access to informal education resources can be leveraged within a formal

school context to improve students’ academic outcomes. I provide heartening evidence that

there is no inherent trade-off between enriching out-of-school experiences and student test

scores; at least with respect to field trips, there is common ground.

Page 29: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 26

References Adams, J., Gupta, P., & DeFelice, A. (2012). Schools and informal science settings: Collaborate, co-exist, or assimilate? Cultural Studies of Science Education, 7(2): 409-416. Alexander, K. L., Entwisle, D. R., & Olson, L. S. (2001). Schools, achievement, and inequality: A seasonal perspective. Educational Evaluation and Policy Analysis, 23(2), 171-191. Alexander, K. L., Entwisle, D. R., & Olson, L. S. (2007). Lasting consequences of the summer learning gap. American Sociological Review, 72(2), 167-180. Anderson, D. (1999). The development of science concepts emergent from science museum and post-visit activity experiences: Students’ construction of knowledge. Unpublished doctoral dissertation, Queensland University of Technology, Brisbane, Australia. Anderson, D., Kisiel, J., & Storksdieck, M. (2006). Understanding teachers’ perspectives on field trips: Discovering common ground in three countries. Curator: The Museum Journal, 49(3): 365-386. Anderson, D., & Lucas, K. B. (1997). The effectiveness of orienting students to the physical features of a science museum prior to visitation. Research in Science Education, 27(4), 485-495. Bamberger, Y., & Tal, T. (2007). Learning in a personal context: Levels of choice in a free choice learning environment in science and natural history museums. Science Education, 91(1), 75-95. Burch, P. (2007). Supplemental education services under NCLB: Emerging evidence and policy issues. Education Policy Research Unit, Arizona State University, Tempe, AZ. Deke, J., Gill, B., Dragoset, L., & Karen, B. (2014). Effectiveness of supplemental educational services. Journal of Research on Educational Effectiveness, 7(2), 137-165. DeWitt, J., & Storksdieck, M. (2008). A short review of school field trips: Key findings from the past and implications for the future. Visitor Studies, 11(2): 181-197. Downey, D. B., von Hippel, P. T., & Broh, B. A. (2004). Are schools the great equalizer? Cognitive inequality during the summer months and the school year. American Sociological Review, 69(5), 613-635. Ellerson, N. M. (2012). Weathering the storm: How the economic recession continues to impact schools. American Association of School Administrators. Entwisle, D. R., & Alexander, K. L. (1992). Summer setback: Race, poverty, school composition, and mathematics achievement in the first two years of school. American Sociological Review, 57(1), 72-84. Entwisle, D. R., & Alexander, K. L. (1994). Winter setback: School racial composition and learning to read. American Sociological Review, 446–460.

Page 30: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 27

Falk, J. H. (1983). Field trips: A look at environmental effects on learning. Journal of Biological Education, 17(2), 137-142.

Falk, J. H., & Dierking, L. D. (1997). School field trips: Assessing their long-term impact. Curator: The Museum Journal, 40(3), 211-218. Gottfried, J. (1980). Do children learn on school field trips? Curator: The Museum Journal, 23(3), 165-174. Government Accountability Office (2006). No Child Left Behind Act: Education actions needed to improve local implementation and state evaluation of supplemental educational services. GAO Report 06-758. Washington, D.C. Greene, J. P., Kisida, B., & Bowen, D. H. (2014). The educational value of field trips. Education Next, 14(1), 78-86. Greene, W. H. (2004). Fixed effects and bias due to the incidental parameters problem in The tobit model. Econometric Reviews, 23(2), 125-147. Henig, J. (2006). The political economy of supplemental education services. Paper presented at the American Enterprise Institute / Thomas B. Fordham Foundation Conference: “Fixing Failing Schools: Is NCLB Toolkit Working?” Washington, D.C. Heinrich, C. J., Meyer, R. H., & Whitten, G. (2010). Supplemental education services under No Child Left Behind: Who signs up, and what do they gain? Educational Evaluation and Policy Analysis, 37(2), 273-298. Heyns, B. (1978). Summer learning and the effects of schooling. New York: Academic. Heyns, B. (1987). Schooling and cognitive development: Is there a season for learning?. Child Development, 1151-1160. Jarvis, T., & Pell, A. (2002). Effect of the Challenger experience on elementary children's attitudes to science. Journal of Research in Science Teaching, 39(10), 979-1000. Jarvis, T., & Pell, A. (2005). Factors influencing elementary school children's attitudes toward science before, during, and after a visit to the UK National Space Centre. Journal of Research in Science Teaching, 42(1), 53-83. Knapp, D. (2000). Memorable experiences of a science field trip. School Science and Mathematics, 100(2), 65-72. Krueger, A. P. (2000, August 17). Vouchers for summer school could help halt the learning slide. The New York Times. Retrieved from http://www.nytimes.com Loucks-Horsley, S., & Matsumoto, C. (2010). Research on professional development for teachers of mathematics and science: The state of the scene. School Science and Mathematics, 9(5): 258-271. Miglietta, A. M., Belmonte, G., & Boero, F. (2008). A summative evaluation of science learning:

Page 31: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 28

A case study of the marine biology museum “Pietro Parenzan” (South East Italy). Visitor Studies, 11(2), 213-219. Miller, G. and Moskowitz, E. S. (2004). Lost in space: Science education in New York City public schools. The City Council of the City of New York: Committee on Education. Muñoz, M., Potter, A. P., & Ross, S. M. (2008). Supplemental educational services as a consequence of the NCLB legislation: Evaluating its impact on student achievement in a large urban district. Journal of Education for Students Placed at Risk, 13(1), 1-25. Murnane, R. J. (1975). The impact of school resources on the learning of inner city children. Cambridge, MA: Ballinger. National Research Council (2009). Learning science in informal environments: People, places and pursuits. Committee on Learning Science in Informal Environments. Washington, D.C.: National Academy Press. New York City Independent Budget Office (2011). New York City Public School indicators: Demographics, resources, outcomes. NYC IBO Annual Report, 2011. Neyman, J., & Scott, E. (1948). Consistent estimates based on partially consistent observations. Econometrica, 16, 1-32. Orion, N. and Hofstein, A. (1991). The measurement of students’ attitudes towards scientific field trips. Science Education, 75(5), 513-523. Peterson, P, (2005). Making up the rules as you play the game. Education Next, 5(4), 42-49. Phillips, M., Finkelstein, D., & Wever-Frerichs, S. (2007). School site to museum floor: How informal science institutions work with schools. Journal of Science Education, 29(12): 1489-1507. Rickles, J. H., & Barnhart, M. K. (2007). The impact of supplemental educational services participation on student achievement: 2005-06. Report of the Los Angeles Unified School District Program Evaluation and Research Branch Planning, Assessment, and Research Division Publication 352. Rickles, J. H., & White, J. A. (2006). The impact of supplemental educational services participation on student achievement. Los Angeles, CA: Los Angeles Unified School District. Program and Evaluation and Research Branch. Planning, Assessment and Research Division., Los Angeles, CA. Ross, S. M., Potter, A., Paek, J., McKay, D., Sanders, W., & Ashton, J. (2008). Implementation and outcomes of supplemental educational services: The Tennessee state-wide evaluation study. Journal of Education for Students Placed at Risk, 13(1), 26-58. Ryan, S., & Fatani, S. (2005). SES tutoring programs: An evaluation of the second year—Part one of a two part report (Policy report). Chicago: Office of Research. Evaluation and

Page 32: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 29

Accountability, Chicago Public Schools. Sibthorp, J., & Knapp, D. (1998). Evaluating short term and long term retention of experiences

associated with an interpretive school field trip. Paper presented at the Coalition of Education in the Outdoors Research Symposium, Bradford Woods, Indiana.

Weinstein, M., Whitesell, E. R., & Leardo, M. (2014). Successful schools: How school-level factors influence success with Urban Advantage. IESP Working Paper No. 01-14. Weinstein, M., Whitesell, E. R., & Schwartz, A. E. (2014). Museums, zoos, and gardens: How formal-informal partnerships can impact urban students’ performance in science. Evaluation Review, 38(6), 514-545. Zimmer, R., Gill, B., Booker, K., & Lockwood III, J. R. (2007). State and local implementation of the No Child Left Behind Act. Volume I. A Report from the National Longitudinal Study of No Child Left Behind. U.S. Department of Education.

Page 33: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 30

Figures and Tables

Figure 1: Location of ISEIs and UA schools

Note: ISEIs are shown in large black dots and labeled. The map includes all schools in the sample (light dots); all schools participate in UA at some point from 2007-2012.

Page 34: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 31

Figure 2: Count of total class trips taken to ISEIs by year, 2007-2012

Note: American Museum of Natural History (AMNH), New York Hall of Science (NYHS), New York Aquarium (NYA), Staten Island Zoo (SIZ), Bronx Zoo (BZ), New York Botanical Garden (NYBG), Queens Botanical Garden (QBG), Brooklyn Botanic Garden (BBG). Figure created using voucher-level data, excluding Staten Island schools. Figure includes trips taken between September and June, though beginning in 2012, vouchers expired at the end of May and no trips were taken in June.

0

50

100

150

200

250

AMNH NYHS NYA SIZ BZ NYBG QBG BBG

2007 2008 2009 2010 2011 2012

Page 35: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 32

Figure 3: Distribution of total class trips by month and year, 2007-2012

Note: Figure created using voucher-level data, excluding Staten Island schools. Figure includes trips taken between September and June, though beginning in 2012, vouchers expired at the end of May and no trips were taken in June.

Figure 4: Voucher use before the science exam in UA schools, 2007-2012

Note: Figure created using school-level data, excluding Staten Island schools. Figure includes trips taken from September to April (2007-2009) and from September to May (2010-2012) in active UA schools. Exposure is calculated by dividing the total number of class visitors by 6th-7th grade enrollment in 2007-2009 and by 6th-8th grade enrollment in 2010-2012.

0

50

100

150

200

250

300

350

Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May June

2007 2008 2009 2010 2011 2012

0.21 0.33 0.38 0.42 0.410.51

1.34

1.681.87 1.86 1.89

2.50

0.00

0.50

1.00

1.50

2.00

2.50

2007 2008 2009 2010 2011 2012

Visits per student Trips per teacher

Page 36: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 33

Table 1: School characteristics by quartile of student exposure to class trips in active UA schools, 2012

All schools Q1 Q2 Q3 Q4

Voucher usage

Student exposure - class trips 0.51 0.02 0.28 0.55 1.18

Student exposure - independent trips 0.32 0.20 0.25 0.31 0.53

Class trips per UA teacher 2.53 0.42 2.46 3.18 4.07

Student characteristics (%)

Black 33.13 34.82 31.07 31.61 35.04

Hispanic 42.73 38.61 46.75 41.89 43.68

Asian 13.67 15.93 11.93 15.89 10.93

White 10.40 10.50 10.11 10.64 10.36

Free/Reduced lunch 77.83 80.04 76.46 76.86 77.96

Limited English proficient 13.29 13.89 13.00 14.39 11.86

Attendance rate 91.89 91.25 92.25 92.21 91.86

Teacher characteristics (%)

< 3 years experience 10.10 10.68 8.82 11.89 9.00

Master's + 39.27 42.32 40.18 37.57 37.00

Turnover rate 17.92 23.43 15.39 18.79 14.07

% science teachers in UA 62.53 45.77 47.95 68.22 88.17

School characteristics

Science proficiency (lagged) 52.38 54.94 55.37 51.20 48.24

Math proficiency (lagged) 53.17 54.43 54.08 54.36 49.89

ELA proficiency (lagged) 36.47 36.64 37.22 37.17 34.84

Total enrollment 682.96 823.68 812.71 598.82 496.61

Distance to nearest ISEI (miles) 2.57 2.73 2.52 2.70 2.32

Elementary-middle 22.32 17.86 14.29 17.86 39.29

Middle 58.04 57.14 71.43 67.86 35.71

Middle-high 19.64 25.00 14.29 14.29 25.00

Manhattan (%) 23.21 14.29 10.71 32.14 35.71

Bronx (%) 20.54 17.86 17.86 32.14 14.29

Brooklyn (%) 34.82 32.14 39.29 25.00 42.86

Queens (%) 21.43 35.71 32.14 10.71 7.14

N 112 28 28 28 28 Note: Characteristics are based on all students in the school, not just 6th-8th graders. Science proficiency is based on the 8th-grade Intermediate Level Science exam. Quartiles are based on class visits per student. Quartile ranges: 1 (0 – 0.129), 2 (0.157 – 0.422), 3 (0.424– 0.720), 4 (0.736 – 2.353). Sample excludes Staten Island schools.

Page 37: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 34

Table 2: Impact of student exposure to class trips on 8th-grade science outcomes, 2007-2012

Z-score Proficient

(1) (2) (3) (4) (5) (6)

Class visits per student 0.049 0.018 0.016 0.019 0.007 0.003

(0.030) (0.024) (0.017) (0.013) (0.011) (0.008)

Active UA school -0.014 -0.002 0.012 -0.011 -0.006 0.004

(0.022) (0.018) (0.013) (0.009) (0.008) (0.006)

Black -0.375*** -0.163*** -0.172*** -0.158*** -0.070*** -0.073***

(0.013) (0.009) (0.009) (0.006) (0.005) (0.005)

Hispanic -0.206*** -0.081*** -0.091*** -0.084*** -0.031*** -0.035***

(0.012) (0.008) (0.008) (0.005) (0.004) (0.004)

Asian 0.196*** 0.015 0.013 0.081*** 0.006 0.006

(0.014) (0.009) (0.009) (0.006) (0.005) (0.005)

Poor -0.120*** -0.074*** -0.037*** -0.050*** -0.030*** -0.016***

(0.016) (0.011) (0.008) (0.006) (0.005) (0.004)

SPED -0.613*** -0.289*** -0.288*** -0.275*** -0.142*** -0.142***

(0.010) (0.009) (0.008) (0.005) (0.004) (0.004)

LEP -0.822*** -0.387*** -0.397*** -0.339*** -0.164*** -0.168***

(0.016) (0.013) (0.012) (0.007) (0.006) (0.006)

% black -0.009*** -0.003*** -0.000 -0.003*** -0.001*** -0.000

(0.001) (0.001) (0.004) (0.000) (0.000) (0.002)

% Hispanic -0.007*** -0.002*** 0.002 -0.003*** -0.001** 0.001

(0.001) (0.001) (0.003) (0.000) (0.000) (0.001)

% Asian -0.004*** -0.000 0.002 -0.001*** 0.000 0.000

(0.001) (0.001) (0.003) (0.000) (0.000) (0.001)

% LEP -0.004*** 0.000 0.005** -0.002*** 0.000 0.003***

(0.001) (0.001) (0.002) (0.000) (0.000) (0.001)

% poor -0.002*** -0.001** 0.000 -0.001*** -0.001*** -0.000

(0.001) (0.001) (0.000) (0.000) (0.000) (0.000)

% < 3 yrs exp 0.004*** 0.002** -0.000 0.002*** 0.001** -0.000

(0.001) (0.001) (0.001) (0.001) (0.000) (0.000)

% Master’s + 0.001 -0.000 -0.002** 0.001 0.000 -0.001

(0.001) (0.001) (0.001) (0.000) (0.000) (0.000)

Turnover rate -0.006*** -0.004*** -0.001* -0.003*** -0.002*** -0.001**

(0.001) (0.001) (0.001) (0.000) (0.000) (0.000)

Attendance rate 0.012*** 0.008*** 0.002* 0.005*** 0.004*** 0.001*

(0.004) (0.002) (0.001) (0.002) (0.001) (0.001)

Ln(enrollment) 0.024 0.018 0.015 0.010 0.008 0.005

(0.024) (0.019) (0.037) (0.010) (0.008) (0.017)

Year fixed effects Y Y Y Y Y Y

Lagged math z-score - Y Y - Y Y

School fixed effects - - Y - - Y

Observations 185,055 185,055 185,055 185,055 185,055 185,055

R-squared 0.293 0.516 0.548 0.212 0.356 0.379 Robust standard errors clustered at schl-yr level in parentheses *** p<0.01, ** p<0.05, * p<0.1 Note: Field trip measures include trips taken before the exam. Sample excludes Staten Island. Constant not shown.

Page 38: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 35

Table 3: Impact of medium and high student exposure to class trips before the science exam on 8th-grade science outcomes, 2007-2012

Z-score Proficient

Class visits per student (1) (2) (3) (4)

>=0.25 visits (indicator) 0.026* 0.012*

(0.014) (0.006)

0.25– 0.50 visits (indicator) 0.027* 0.016**

(0.016) (0.007)

> 0.50 visits (indicator) 0.025 0.007

(0.018) (0.008)

Student controls Y Y Y Y

School controls Y Y Y Y

Year FE Y Y Y Y

School FE Y Y Y Y

Observations 185,055 185,055 185,055 185,055

R-squared 0.548 0.548 0.379 0.379

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Note: All models include student controls, school controls, year effects, and school effects. Student-level controls include: lagged

math z-score, race, poverty (qualifying for free or reduced-price lunch), SPED, LEP. School-level controls include UA status, the

percent black, Hispanic, Asian, LEP, poverty, percent of teachers with fewer than 3 years of experience, percent of teachers with

Master’s degrees, teacher turnover rate, student attendance rate, and school size (log of enrollment). Class trips include trips

taken from September to April (2007-2009) and from September to May (2010-2012). Sample excludes Staten Island. Robust

standard errors are clustered at the school-year level.

Page 39: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 36

Table 4: Subgroup results: Impact of student exposure to ISEIs before the science exam on 8th-grade science outcomes, 2007-2012

Panel A: Outcome: Z-score (1) (2) (3) (4) (5) (6) (7) (8)

Black Hispanic Asian White Poor Not poor LEP SPED

>= 0.25 visits per student 0.015 0.043*** -0.015 0.016 0.033** 0.006 0.022 0.016

(0.024) (0.016) (0.022) (0.026) (0.014) (0.025) (0.030) (0.022)

Observations 53,727 78,417 33,359 19,478 156,277 28,778 22,552 20,233

R-squared 0.458 0.505 0.533 0.566 0.535 0.589 0.334 0.400

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Panel B: Outcome: Proficient (1) (2) (3) (4) (5) (6) (7) (8)

Black Hispanic Asian White Poor Not poor LEP SPED

>= 0.25 visits per student 0.006 0.019** -0.008 0.020* 0.017*** -0.008 0.012 0.008

(0.011) (0.007) (0.010) (0.011) (0.006) (0.010) (0.012) (0.011)

Observations 53,727 78,417 33,359 19,478 156,277 28,778 22,552 20,233

R-squared 0.316 0.349 0.338 0.370 0.372 0.400 0.229 0.254

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Note: All models include student controls, school controls, year effects, and school effects. Student-level controls include: lagged math z-score, race, poverty (qualifying for free or reduced-price lunch), SPED, LEP. School-level controls include UA status, the percent black, Hispanic, Asian, LEP, poverty, percent of teachers with fewer than 3 years of experience, percent of teachers with Master’s degrees, teacher turnover rate, student attendance rate, and school size (log of enrollment). Sample excludes Staten Island. Robust standard errors are clustered at the school-year level.

Page 40: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 37

Table 5: Robustness tests: Impact of student exposure to ISEIs before the science exam on 8th-

grade science z-scores using alternative samples, controls, and measures, 2007-2012

(1) (2) (3) (4) (5) (6) (7)

>=0.25 trips per student 0.026* 0.019 0.024* 0.029**

(0.014) (0.013) (0.014) (0.015)

>=0.20 trips per student 0.029**

(0.013)

>= 0.33 trips per student 0.028*

(0.014)

>=1 trip per UA teacher 0.021

(0.014)

Student and schl controls Y Y Y Y Y Y Y

Year and school FE Y Y Y Y Y Y Y

Staten Island - Y - - - - -

Independent visits / stu - - Y - - - -

% UA teachers - - - Y - - -

Observations 185,055 207,510 185,055 185,055 185,055 185,055 185,055

R-squared 0.548 0.552 0.548 0.548 0.548 0.548 0.548

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Note: Student-level controls include: lagged math z-score, race, poverty (qualifying for free or reduced-price lunch), SPED, LEP. School-level controls include UA status, independent trips per student, the percent black, Hispanic, Asian, LEP, poverty, percent of teachers with fewer than 3 years of experience, percent of teachers with Master’s degrees, teacher turnover rate, student attendance rate, and school size (log of enrollment). Class trips include trips taken from September to April (2007-2009) and from September to May (2010-2012). Robust standard errors are clustered at the school-year level.

Page 41: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 38

Table 6: Falsification test: Impact of class trips after the science exam on 8th-grade science outcomes, 2007-2012

Z-score Proficient

(1) (2) (3) (4)

Class visits per student -0.004 -0.005

(0.032) (0.014)

>=0.25 visits per student -0.002 -0.006

(0.019) (0.009)

Student and school controls Y Y Y Y

Year FE Y Y Y Y

School FE Y Y Y Y

Observations 185,055 185,055 185,055 185,055

R-squared 0.548 0.548 0.379 0.379

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Note: Student-level controls include: lagged math z-score, race, poverty (qualifying for free or reduced-price lunch), SPED, LEP. School-level controls include UA status, independent trips per student, the percent black, Hispanic, Asian, LEP, poverty, percent of teachers with fewer than 3 years of experience, percent of teachers with Master’s degrees, teacher turnover rate, student attendance rate, and school size (log of enrollment). Class trips after the exam include trips taken in May and June (2007-2009) and in June (2010-2012). Sample excludes Staten Island schools. Robust standard errors are clustered at the school-year level.

Page 42: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 39

Appendix Table A1: Number and percent of schools in the analytic sample participating in UA by year, 2007-2012

Non-active Active Total

2007 # 62 109 171

% 36.3% 63.7%

2008 # 60 125 185

% 32.4% 67.6%

2009 # 76 117 193

% 39.4% 60.6%

2010 # 52 143 195

% 26.7% 73.3%

2011 # 73 125 198

% 36.9% 63.1%

2012 # 83 112 195

% 42.6% 57.4%

Note: All schools in the sample participate in UA at some point between 2007 and 2012. Active schools have at least one teacher participating in UA in that year. Sample excludes Staten Island.

Table A2: Student exposure to class trips before the science exam, analytic sample, 2007-2012

School-level Total Student-level Total

<0.25 0.25-0.50 >0.50 Schools <0.25 0.25-0.50 >0.50 Students

2007 # 139 12 20 171 28,798 1,195 2,334 32,327

% 81.3 7.0 11.7 89.1 3.7 7.2

2008 # 135 19 31 185 26,374 2,701 2,558 31,633

% 73.0 10.3 16.8 83.4 8.5 8.1

2009 # 148 14 31 193 25,954 2,995 3,966 32,915

% 76.7 7.3 16.1 78.9 9.1 12.0

2010 # 116 26 53 195 20,843 3,587 5,539 29,969

% 59.5 13.3 27.2 69.5 12.0 18.5

2011 # 131 25 42 198 21,310 4,790 3,777 29,877

% 66.2 12.6 21.2 71.3 16.0 12.6

2012 # 122 29 44 195 19,555 4,446 4,333 28,334

% 62.6 14.9 22.6 69.0 15.7 15.3

Pooled # 791 125 221 1,137 142,834 19,714 22,507 185,055

% 69.6 11.0 19.4 77.2 10.7 12.2

Note: Figures include trips taken from September to April (2007-2009) and from September to May (2010-2012). From 2007-2009, total class visitors are divided by 7th-8th grade enrollment. From 2010-2012, total class visitors are divided by 6th-8th grade enrollment. Distributions are based on the analytic sample of 8th-graders with science test scores, excluding Staten Island schools.

Page 43: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 40

Table A3: Robustness tests: Impact of student exposure to ISEIs before the science exam on 8th-grade science proficiency using alternative samples, controls, and measures, 2007-2012

(1) (2) (3) (4) (5) (6) (7)

>=0.25 trips per student 0.012* 0.009 0.012* 0.014**

(0.006) (0.006) (0.006) (0.007)

>=0.20 trips per student 0.013**

(0.006)

>= 0.33 trips per student 0.009

(0.007)

>=1 trip per UA teacher 0.012*

(0.006)

Student and schl controls Y Y Y Y Y Y Y

Year and school FE Y Y Y Y Y Y Y

Staten Island - Y - - - - -

Independent visits / stu - - Y - - - -

% UA teachers - - - Y - - -

Observations 185,055 207,510 185,055 185,055 185,055 185,055 185,055

R-squared 0.379 0.382 0.379 0.379 0.379 0.379 0.379

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Note: Student-level controls include: lagged math z-score, race, poverty (qualifying for free or reduced-price lunch), SPED, LEP. School-level controls include UA status, independent trips per student, the percent black, Hispanic, Asian, LEP, poverty, percent of teachers with fewer than 3 years of experience, percent of teachers with Master’s degrees, teacher turnover rate, student attendance rate, and school size (log of enrollment). Class trips include trips taken from September to April (2007-2009) and from September to May (2010-2012). Sample excludes Staten Island. Robust standard errors are clustered at the school-year level.

Page 44: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 41

Exhibit A1: UA information on how to use a class voucher, available on UA website, 2014-15 academic year

Page 45: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

IESP WORKING PAPER #03-15 42

Exhibit A2: UA guidelines for class trips, available on UA website, 2014-15 academic year

1. Please check and follow your school trip procedures before planning your UA class trips. In the absence of such procedures, you should refer to Chancellor’s Regulations A-670, which were given to you in our UA teacher handbook and can be found at www.urbanadvantagenyc.org under the Teacher tab. You may follow these guidelines below.

2. All UA class trips should be approved by your principal and assistant principal or department supervisor at least

two weeks in advance. The trip approval form from your school or Chancellor’s Regulations A-670 must be

completed and signed by the coordinating teacher, department supervisor, and the principal before making any reservations. Please refer to our UA partner institution information on how to make reservations including any special instructions in our UA teacher handbook. This information can also be found in the Vouchers and Trips section under the Teacher tab at www.urbanadvantagenyc.org.

. 3. Before planning your UA class trips, please speak to your department supervisor or principal to discuss the

nature of your UA class trips. All UA class trips are to be curriculum-based and of an educational nature. Suggested pre- and post- trip activities may be obtained from our UA partner institutions. This should be done at about two months in advance of the trip dates. Planning cross-content trips is suggested if you need to take

large groups of students. 4. After your UA class trip approval, please obtain and complete transportation forms (school buses or public

transportation passes) from your main office. Sample copies are in our UA teacher handbook. Transit directions can be found at www.hopstop.com. UA only provides school buses for UA family field trips. If you wish to travel to an institution farther from your school, please consider using a school bus one way and using public transportation for the return trip. If you choose this option, be sure to have the proper consent from the each student’s parent or guardian.

5. If you wish to take school lunches for your students, please complete a school field trip meal request form from

your main office. It must be signed by your principal and submitted to the kitchen at least two weeks prior to the UA class trips. If no school form is available, please ask your principal to contact your School Food District Supervisor.

6. Please distribute and collect all signed parent or guardian class trip consent forms from Chancellor’s Regulations

A-670 and keep these forms on file. You should bring a copy of these forms with you. Please communicate to the parents or guardians the intended mode of transportation, costs, provisions for meals, whether the class will leave before school hours or return after school hours, etc.

7. Please make sure that the UA class trip is included in your school’s calendar of events. UA class trips should not

to be scheduled around the time of standardized testing and parent-teacher conferences without the prior approval of your principal. UA class trips should not be scheduled in June.

8. Please bring and share with your chaperones the attendance sheet and emergency contact information of your

students on your UA class trips. An attendance sheet of the students attending the trip must be given to someone designated by the principal in your main office on the day of the trip. As required by Chancellor’s Regulations A-670, please plan to have the required ratio of one adult to ten students for adequate supervision for the trips. Also, you should have the cell phone numbers of your school bus driver (if using a school bus) and chaperones for constant communication. Chaperones should have your cell phone number and a list of their students.

9. Please inform someone designated by your principal in your main office when you and your students arrive at

your destination and also when you and your students are ready to return. All trips must occur during school hours unless prior approval from your principal has been obtained.

10. Please note where the students who are not going on the UA class trip are being held. Please consult with your

dean and department supervisor in deciding where to place these students. You must account for all students. 11. Please consult your department supervisor on making arrangements and leaving assignments for your covering

teachers.

12. Please do not release your students early from your UA class trips. They are our responsibility (and must be actively supervised) until the dismissal time. In the event of a medical emergency, please call 911 immediately and then your main office. For a non-medical emergency, please contact your main office first and then call 911.

13. If you have any questions about these guidelines, please contact John Tom at [email protected].

Page 46: A Day at the Museum: The Impact of Field Trips to Informal ... · of Field Trips to Informal Science Education ... In this study, I estimate the impact of field trips to informal

The Institute for Education and Social Policy is a joint research center of NYU’s Wagner and Stein-hardt Schools. Founded in 1995, IESP brings the talents of a diverse group of NYU faculty, graduate students and research scientists to bear on questions of education and social policy. We are one of the nation’s leading academic research centers addressing urban education issues and a substantial amount of our work focuses on New York City public schools and students. More information is available on our website at http://steinhardt.nyu.edu/iesp.

665 Broadway, Suite 805, New York, NY 10012 p 212.998.5880 f 212.995.4564