Rochester Arts Impact Study

Deborah Harloff, RCSDExecutive Director of Visual and Performing Arts

Andrew MacGowan , IIIProject Administrator and Arts Impact Study Evaluator

The research on arts integration is based on the hypotheses that there is a positive impact between integrated arts instruction and increased student achievement .

What is arts integration? Arts Integration is the process of combining

content of the arts with other classroom content such as English Language Arts, Science, Social Studies, or mathematics. This is done by making clear content connections between the various curricular areas. Visual art

DanceTheatre

Music

The Arts as an interventionLittle empirical dataStudent motivation

1st Time 20062006-2009Over 200 applicantsRanked 13th nationally of 25 awardeesPerfect score on evaluation design

“True Experimental Design”What and Why• “Medical model,” “True Experimental,”

“Randomized Trial” all mean the same thing• Until NCLB, very few (< dozen) randomized

trials in education• The most rigorous form of evaluation• “quasi-experimental” is the next best form of

evaluation

The RCSD Design9 randomly selected elementary schools (out

of 39)180 teachers4,000 students 7 grade levels (K-6)Remaining 29 schools served as the control

groupApproximately 30,000 students in all, over

the course of four years (baseline year + three years of treatment)

Statistically “robust”: results can be believed

MeasuresSixteen major New York and national

standardized tests were used, in English Language Arts, Mathematics, Science and Social Studies

Components of the grantTen session teaching artist residency Additional arts experience Supplies to implement the residency Ten hours of PD for teachers

Year One: SelectionComputer-generated randomization of

schoolsIntroduce project to building administratorsInterview and hire teaching artists Create planning documentGrade level planning sessions

4 Schools Opt OutBased on the following rational:

the school based planning team (SBPT) was concerned about the ten hours PD requirement

the teachers at one of the schools were concerned that they would be writing the curriculum for the teaching artist

another school cited the concern about accepting a new program when the school had a new principal.

Years Two & ThreeYear two one school decided to opt out of the

program During year three, some of the faculty at one

school refused to meet thereby essentially opting out of the program.

Due to scheduling conflicts and organizational skills one teaching artist was not retained.

A different teaching artist moved out of state and could not complete the project.

Year One: Implementation Grade level planning meetings were

scheduled

Schedules were arranged to accommodate teaching artist availability as well as classroom continuity.

Materials and supplies needed for the project were discussed.

Years Two & Three: Implementation Due to inconsistencies in planning meetings,

a planning document was developed to create uniform lesson and unit plans

Schedule protocol remained the same for year two.

Introduction of the 4 Art FormsThe art form that was integrated at the

school was randomly selected . Two schools were assigned as music

two were assigned as danceThree schools were assigned as theatre

Three were assigned as visual art

Skills and Content Taught Visual Art

MusicTheatre

Dance

Arts Impact Study PDProfessional development (PD) to reinforce

understanding of integration techniques and strategies in music, dance, theater and visual arts.

Professional reading 96% approval ratingHands onModels strategiesProvides materials

PD for Teaching Artists Program objectives, program implementation and

classroom residencies. Operational practices were taught to the teaching

artistsTechniques to reinforce ELA, Math, Science and

Social Studies with the teaching artists’ area of expertise.

Collegial sharing - discussed best practices and successful strategies for co-teaching and co-planning lessons with classroom teachers.

Main ConclusionsUsing both national and NY tests, positive and

significant effects, from 5% to 30% improvements

Across most grade levels K – 6Students with Disabilities showed gainsEnglish Language Learners showed gainsGains in both low and high performing schoolsGains in ELA and MathScience and Social Studies (not statistically

significant)

More conclusionsThe lower the grade level, the greater the effects

(e.g., the younger the student, the greater the effects);

The more economically disadvantaged the student, the greater the effects.

Major, unexpected finding: Arts Impact acts as a protective factor for students who did not have the benefit of attending a higher performing school.

Top conclusions (charts to follow)Kindergarten through 2nd grade students in the

treatment schools enjoyed the greatest degree of gains, including Students With Disabilities.

By Year 3, Kindergarten students realized a 14% differential, 56% of the treatment students reached Hi/Scope Child Observation Record (COR) Proficiency Level 4 (on the COR 5-point scale), versus 41% of the control students.

The gains in Year 3 for Kindergarteners With a Disability were even greater, with 25% more gaining on the COR Proficiency Level 4 (on a 5-point scale), 30% for the control students vs. 55% for treatment students.

The comparative gains for Kindergarten English Language Learners in Year 2 saw an approximate 4% advantage.

By Year 3, 1st grade students on Free or Reduced-Price Lunch (an indication of poverty), realized a 6% comparative gain in Reading, Language Arts and Mathematics (identical in all three) as measured by the CTB Terra Nova (which is configured to New York State Standards.)

More conclusions . . .By Year 3, 1st grade English Language Learners realized a 12% comparative

gain in Reading, an 18% comparative gain in Language Arts and a 13% comparative gain in Mathematics.

By Year 3, 1st grade Students With Disabilities realized a 15% comparative gain in Reading, a 6% gain in Language Arts and an 8% gain in Mathematics, compared to control school students.

For reasons we do not yet understand the intermediate grades (3 – 6) see greater gains for students in 4th and 6th grade than in 3rd and 5th grade.

In low performing schools, 4th grade students realized a 13% higher rate of English Language proficiency and a 10% higher rate of Mathematics proficiency (as shown by the New York State Tests), compared to low performing control schools.

In low performing schools, 6th grade student gains were noteworthy -- a 13% higher rate of treatment students achieving English Language Arts proficiency (New York State Levels 3 and 4), than control school students, and a 20% higher rate of treatment students achieved Mathematics proficiency compared to students in the control schools.

Gains in Kindergarten – all studentsVII: Rochester Kindergarten Students

Improvement in English Language Arts, Math, Science and Behavior

0

10

20

30

40

50

60

70

80

90

Fall, 2006 Spring, 2007 Fall, 2007 Spring, 2008(p<.01)

Fall, 2008 Spring, 2009(p<.001)

Perc

ent S

corin

g at

or a

bove

4 o

n th

e To

tal K

-CO

R 5

leve

l Sca

le

Control SchoolsProgram Schools

(p<.001)

Martin F. Gardiner, Ph.D.

Gains for Grade 1 ELLV: Rochester 1st Grade Students With Disabilities

Progress in English Language Arts and Math

0

5

10

15

20

25

30

35

40

45

Reading2007-2008

(p< .05)

Reading2008-2009

NYLanguage2007-2008

NYLanguage

2008-2009

NY Math2007-2008

(p < .01)

NY Math2008-2009

Perc

ent S

corin

g at

or a

bove

Gra

de L

evel

Control Schools Program Schools

Year 1

Reading (2006-2007)

NY Lang.(2006-2007)

(p < .01)

NY Math(2006-2007)

Martin F. Gardiner, Ph.D.

Gains for Grade 1 Students with Disabilities

V: Rochester 1st Grade Students With DisabilitiesProgress in English Language Arts and Math

0

5

10

15

20

25

30

35

40

45

Reading2007-2008

(p< .05)

Reading2008-2009

NYLanguage2007-2008

NYLanguage

2008-2009

NY Math2007-2008

(p < .01)

NY Math2008-2009

Perc

ent S

corin

g at

or a

bove

Gra

de L

evel

Control Schools Program Schools

Year 1

Reading (2006-2007)

NY Lang.(2006-2007)

(p < .01)

NY Math(2006-2007)

Martin F. Gardiner, Ph.D.

Quantitative Data143 teacher interviews were conducted at the

end of the third year of the grant project1. What worked best?

the arts activity 30 21%supported content goals 26 18%the integration and the artist sessions

21 15%

student performance 12 8%movement activities 5 3%hands-on aspect of the program 5 3%teaching artist was organized 5 3%Particular teaching artist 5 3%flexibility of the teaching artist 4 2.7%class participation 3 2%artist modeling for the students 3 2%PD session 2  Culturally based activities 1  Planning together 1  Picture books 1  Illustrating 1  Engage students 1  Team work activities 1  

2. What is your evidence of student learning?Response # of

responsesPercentage

Student performance or project produced

31 22%

Students engagement increased 25 17%Application of knowledge from lesson to classroom discussion

14 10%

Student recall 40 28%Use of vocabulary 10 7%Students transfer of knowledge form one content area to another

2  

Test scores 2  Student writing 1  Student confidence increased 1  

3. What activities that the teaching artist modeled have you tried or would you do?

Music activity 36 25%Movement/ dance activity 34 24%Theatre activity 34 24%Visual art activity 31 22%Changing environment – took kids to another space

2  

Cooperative activities 4 3%Make costumes 1  Make musical instruments 3  

4. What support do you need to continue with arts integration?

Responses # of Responses

Percentage

Materials, equipment, resources 45 31%Professional development 40 28%Funding 7 5%More time during the day ( time constraints)

18 13%

More Teaching Artist sessions 17 12%Support for using integrations 7 5%

Did it work?In short, Yes. 30,000 students and 16

standardized tests.

SummaryOverall, we have concluded the earlier the

intervention, the greater were the effects sizes: We observe statistically significant positive effects in Kindergarten, first and second grade. We also observed the higher the poverty as determined by the Free and Reduced Price Lunch code the greater the effect size

Evaluation TipsIf you are contracting with an independent

evaluation outfit, get your Non-Disclosure Agreement done ASAP;

Work with US DoE on this – and feel free to borrow from Rochester’s model (available upon request)

Go with unique IDs, so you can use your Lunch Code data – which can reveal big differences in effects size

NEVER, EVER release Lunch Code information with personally-identifiable data

What Next?The results of this project – and the

independent evaluation of this project – pose significant policy implications for high-poverty urban schools, including those with high populations of English Language Learners and Students with Disabilities.