11 Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 2 Report Research and Policy Support...

11

Evaluating the NYC Core KnowledgeEarly Literacy Pilot: Year 2 Report

Research and Policy Support Group

CONFIDENTIAL – PLEASE DO NOT DISTRIBUTE

2

Executive Summary – Achievement Gains

By nearly all measures, Core Knowledge Reading (CKR) students made significantly greater gains in Year 2 than their peers Overall Achievement

When compared to peers (both peers at demographically similar schools and peers at Reading First schools), 1st graders taught with the CKR program made more progress in reading and social studies.

Spring science scores were significantly greater for students in pilot schools than comparisons schools, but not significantly different from students in Reading First comparison schools.

High vs. Low Performers High and low performing CK students made larger gains than comparison

students

Year 2 vs. Year 1 of the Program While significant gains were found in almost all domains, the magnitude of

these gains was smaller in Year 2 than in Year 1. Both students with only one year of CKR instruction and those with two

years of CKR instruction made greater gains than their peers at comparison schools.

3

Surveys and case studies indicate overall high levels of administrator and teacher satisfaction with the CKR Program and high levels of fidelity in program implementation.

While survey respondents had favorable reactions to CKR, they indicated that they felt CKR rated similarly to other programs they had used.

Surveys and case studies revealed that teachers are struggling with how to differentiate instruction with CKR, in particular for special populations.

Case study schools’ fidelity to the program aligned with their achievement gains: schools with low fidelity demonstrated less gains

Executive Summary – Surveys & Case Studies

4

Methodology: A multi-method, longitudinal research designYEAR 2

Assessments (at 10 CKR schools & 10 comparison schools, 3* of which are Reading First schools):

• Pre- and post-test of literacy skills • Additional tests of literacy skills at end of each year• Tests of science and social studies skills at end of each year

Teacher and Administrator Surveys (at 10 CKR schools):• Assesses satisfaction with and impact of CKR

Case studies (at 5 CKR schools):• Classroom observations, administrator & teacher interviews

Hypothesis: First Graders taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers.

Focus of the Evaluation

* Last year there were 4 Reading First comparison schools. This year there were 3 Reading First comparison schools: one comparison school used Reading First in Year 1 but not in Year 2.

6.2

64.2

14.7

69.4

11.8

80.0

5.7

66.9

89.5

16.6

8.5

75.8

ELL SpecialEd.

Free/Reduced

Lunch

Black/Hispanic

Level 3/4Students

ELA

SchoolSize (in

Hundreds)

5

Overall, CKR and comparison schools had similar demographic profiles. However, the differences between groups, especially % of Black/ Hispanic students, was significantly larger than last year.

CKR Students (N = 781)

Comparison Students

Note: Data from 08-09 was used to select comparison schools (data presented here are from the 2009-10 school year).

Percent of Students(Number of Students for School Size)

(N = 343)*

•N = the number of students for whom both fall and spring data were available.

A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested.

As a result, all analyses control for student demographics.

6

Evaluation of

Achievement Gains

7

CKR students displayed greater gains & higher Spring scores in nearly all literacy and content knowledge domains relative to comparison students

Woodcock Johnson (Fall and Spring) Terra Nova (Spring Only)

Test W-J Brief Reading

W-J Word Attack

(Decoding)

W-J Spelling of Sounds

(Written Spelling)

Terra Nova Reading

(Oral Reading Comprehension,

Vocabulary, Basic Reading,

Decoding)

Terra Nova Social Studies

Terra Nova Science

W-J Letter Word

Identification

(Basic Reading Skills)

W-J Passage Comprehension

(Oral Reading Comprehension)

CKR Schools Compared to…

All Comparison

Schools

CKR Sig. Greater Gains/ Spring Scores


CKR Sig. Greater Gains/

Spring Scores


CKR Sig. Greater Spring Scores

CKR Sig. Greater Spring

Scores


Scores

Reading First Comparison

Schools



CKR Sig. Greater Gains/

Spring Scores


CKR Sig. Greater Spring Scores


Scores

X

No Significant Difference

Note: All analyses control for student demographic characteristics.

8

Average Fall-Spring Gainin Scale Score Points

Woodcock-Johnson (Brief Reading Test)

CKR Students Significantly Higher

p < .0001

6.6

3.7

CKR Comparison Schools

CKR students displayed nearly 2x greater literacy gains relative to students at demographically similar comparison schools

(N = 758) (N = 342)

The magnitude of the group difference was smaller than last year (in Year 1, CKR students’ gains were 5X greater than their peers).

9

(N = 233)* (N = 100)*

* Data presented represent the three comparison schools that are Reading First schools and the three CKR schools who were statistically comparable matches.

15.5

2.4

CKR Reading First Comparison

CKR Students Significantly Higher

p < .0001

Average Fall-Spring Gainin Scale Score Points

Woodcock-Johnson (Brief Reading Test)

CKR students displayed 6x greater literacy gains relative to students at Reading First comparison schools

The magnitude of the group difference was smaller than last year (in Year 1, CKR students’ gained 14.3 pts while Reading First schools declined 4.1 pts).

School Type Average Gain

Pilot 19.1

Pilot 10.2

Pilot 13.1

Pilot 3.5

Pilot 2.6

Pilot 3.7

Comparison 2.6

Comparison 2.1

Comparison 2.2Pilot 1.2

Comparison 2.4

Comparison 1.3

Comparison 1.1

Comparison 1.1

Comparison 1.3Pilot 0.5

Comparison 0.3Pilot -1.5

Comparison -6.0Pilot -13.2

10

Gains at Each School: All of the schools making statistically significant gains were CKR pilot schools

Significantly Positive Gain

Significantly Negative Gain

Non-significant Gain

School Type Average Gain Rank in 2010 Rank in 2009

Pilot 19.1 1 2

Pilot 10.2 2 5

Pilot 13.1 3 9

Pilot 3.5 4 18

Pilot 2.6 5 11

Pilot 3.7 6 7

Comparison 2.6 7 13

Comparison 2.1 8 8

Comparison 2.2 9 14Pilot 1.2 10 12

Comparison 2.4 11

Comparison 1.3 12 10



Comparison 1.3 15 4Pilot 0.5 16 3

Comparison 0.3 17 20Pilot -1.5 18 1

Comparison -6.0 19 6Pilot -13.2 20 15

11

Gains at Each School in Year 1 and Year 2: Some CKR schools that made significant gains in Year 1 did not make significant gains in Year 2 and vice versa




10.49.3

3.6

15.9 15.9 16.7

8.29.0 8.0

1.1

5.2

0.1

-0.4

-4.7 -6.6

-0.6

Lowest 1/4

Fall Scores

2nd 1/4 3rd 1/4 Highest 1/4

Fall Scores

Lowest 1/4

Fall Scores

2nd 1/4 3rd 1/4 Highest 1/4

Fall Scores

12

CKR & All Comparison Schools CKR & Reading First Schools

CKR Schools

Reading First Schools

Average Fall-Spring Gainin Scale Score Points Woodcock-Johnson

(Brief Reading Test w/ Demographic Controls)

Comparison Schools

At all achievement levels, CKR students demonstrated greater literacy gains than students at comparison and Reading First schools

*Significantly different at p < .05

*

*

*

*

**

*

While the lowest performing students made greater gains than students at comparison schools, this difference was not significant. All other differences were statistically significant.

CKR students made significant gains regardless of whether it was their first or second year in the program

13

104.4

101.0

104.6

108.5

106.8

99.8

98.8

106.9

Fall Year 2 Spring Year 2

New in Year 2

Years 1 & 2

CKR + 4.6 + 3.9

Comparison + 2.2 - 0.1

CKR students with two years of the program made larger gains than their peers. Similarly, CKR students with one year of the program made larger gains than their peers.

CKR Years 1 and 2 (n = 527)

Comparison Years 1 and 2 (n = 154)

CKR New in Year 2 (n = 237)

Comparison New in Year 2 (n = 189)

14

Administrator and Teacher

Surveys

14

1515

Administrators reported satisfaction with the CKR program

No

6Yes

1

Will your 1st grade classrooms be using the CK

Reading program next year?

(n = 7)

Would you recommend the CK Reading curriculum

to other administrators you

know?(n = 7)

1

NotSure

6

Yes

Administrators’ overall

satisfaction with CK Reading

(n = 7*)

* Three administrators did not complete the survey.

“Although the CK program has a lot of good components, its lack of a writing curriculum and differentiated instruction is a major concern for a program being used for 1st grade.”

Very

SomewhatSatisfied

Satisfied

3

3

VeryDissatisfied

1

1616

Administrators reported that 1st grade CKR teachers’ practice was about the same as it was the previous year (prior to CKR)

Discussing/ sharing ideas on teaching strategies w/ other

teachers(n = 7)

Collaborating with teachers in other

grades(n = 7)

Collaborating with other first grade

teachers(n = 7)

Using assessment data

to drive instruction

(n = 7)

Somewhat less than last year

Much more than last year

2

3About the same as last year

1

Administrators’ Views:

“I like CKR. I see improvement from last year…However…it’s the balance that I would like to see CK achieve. More of the sight words that children in early childhood should acquire…I would like to see complete alignment between Pre-K, K, and 1st, and now we’re going to 2.”

About the same as last year

4


2

Somewhat more than last year

1


3


2Somewhat more than last year 1

Much less than last year

1


3


3

Somewhat more than last year

1

1

N/A

37.9

21.4

44.8

28.6

1717

Teachers’ overall satisfaction with CK

Reading(n = 29)a

Percent of Respondents

50.0%

82.7%

Much Better

Somewhat Better

Teachers’ overall opinion of CK Reading compared with other

reading programs(n = 28)b

Very Satisfied

Somewhat Satisfied

Number of teachers selecting the “neutral” response: question a = 1 (3.2%); question b = 6 (19.4%).

Teachers’ Views:“I think the skills training really works…the Skills Strand was kind of similar [to last year] because we did Foundations [which also had] lots of cards and decoding.”

“I used the Guided reading program. I liked the small groups and being able to move students faster according to their ability with the Guided Reading but I love the phonics and comprehension part of the Skills Strand.”

Most teachers were satisfied with the CKR overall but only half reported that CKR was better than other reading program

51.7 48.3

37.9

17.2 24.1

55.2

1818

60.7

41.4

41.417.9

31.0

55.2

Goals of lessons are

clear(n = 29) a

I have enough time to

complete daily lesson(n = 29) c

Students find activities engaging

(n = 29) b

Somewhat Agree

Strongly Agree

Goals of lessons are

clear(n = 29) d

I have enough time to

complete daily lesson(n = 29) f

Students find activities engaging

(n = 29) e

Skills Strand Listening and Learning Strand


Number of teachers selecting the “neutral” response: question a n = 0; question b n = 2 (7.1%); question c n = 2 (6.9%); question d n = 0; question e n = 6 (20.7%); question f n = 4 (13.8%).

96.6%

78.6%72.4%

93.1%

68.9%72.4%

Teachers thought both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 1, teachers rated the Skills Strand more positively than the L&L Strand and had concerns about time.

Number of teachers selecting the “about the same” response: question a n = 4(13.8%); question b n = 8 (27.6%); question c n = 3 (13.8%); question d n = 9 (31.0%); question e n = 11 (37.9%); question f n = 9 (31.0%).

17.2 20.7 20.727.6

20.7

31.0

44.8 34.5 31.0 17.2

10.3

37.9

Teachers compare CKR with other programs they have taught: As in Year 1, decoding skills are a strength of CKR while differentiation of instruction is rated less highly.

Somewhat Better

Much Better

Accommodations for different

learning needs(n = 29) f

Ability to engage students and spark

enthusiasm for reading(n = 29) e

Compehen-siveness

of program(n = 29) d

Teaching content/

background knowledge(n = 29) c

Teaching decoding

skills(n = 29) a

Sequence of

instruction(n = 29) b


68.9%

55.2%

62.0%

51.7%

44.8%

31.0%

20

Site Visits and

Interviews

20

21

Measuring fidelity to the CKR curriculum

• Examining implementation fidelity allows us to better determine:• 1) whether achievement gains can be attributable to the CKR

program and• 2) which components of the CKR program teachers are

struggling to implement and those they are implementing successfully

• We took several different approaches to measuring fidelity in the CKR Pilot schools:

• Site visits with classroom observations• Site visits were conducted at 5 randomly selected Pilot schools• 15 classrooms were observed

• Interviews with administrators and teachers• Interviews were conducted with 13 teachers and 7 administrators

• Additional teacher and administrator survey questions

Examples from Checklist

• Check to see that CKR materials are being used and CKR visuals are posted

• Verify that the daily schedule allows for both the Skills and the Listening and Learning Strands to last a full 60 minutes each

• Whether or not students are familiar with language introduced in the domain

• Teacher engaging all students in exercises, practice, class discussions, and games

• Whether or not students are attentive to the teacher during instruction

• Whether or not students are eager to participate in the daily activities of the program

• Whether or not students are generally on task and complete assignments in a timely fashion

22

During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form

These components were taken from the Core Knowledge Reading Pilot

Observation Form

These components were added as additional measures of student engagement

23

Students are attentive to

their teacher during

instruction

Students respond appropriately to their teacher’s

directions, questions, and assignments

Teachers have Skills Strand

materials posted and used Word Walls effectively

Small group and/or

individual practice time

Full 60 minutes is allotted for

both the Listening and Learning and Skills Strands

Number of Classrooms Out of 15

Observed During Visit

Not Observed During Visit

Most classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons. We observed fewer classrooms engaging in the full 60 minutes of recommended CKR instruction.

24

Overall, teachers and administrators reported that high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction.

Fidelity Survey Questions June 2010 Teacher Responses

Overall, I feel that the CK Curriculum is being implemented consistently in the First Grade classes at my school.

86% Strongly Agreed or Somewhat Agreed

Typically, I find I have enough time to complete the daily Skills Strand Lesson

72% Strongly or Somewhat Agreed

Typically, I find I have enough time to complete the daily Listening and Learning lesson


I am able to find time for daily small group instruction 46% Strongly or Somewhat Agreed

I check with the CK Learning Coordinator before making modifications to the program


Overall level of satisfaction with the CKR Program 83% Very or Somewhat Satisfied

Anecdotes from Classroom Observation

• The first classroom had a vibrant atmosphere. Approximately 30 students, all of whom were seated on the carpet, appeared completely absorbed, watching their teacher and the images about the U.S. space program that were projected on the screen beside her.

• The classroom was covered with student work, Core Knowledge posters and chaining charts, and a hanging model of our solar system.

• The students were eager to share their answers, some of them using challenging vocabulary words that they had learned from earlier lessons like “perseverance.”

• After a writing/drawing exercise, in the final minutes of the lesson the teacher played a song about the sun that she had found on the internet. Some students pretended to be conductors, which they had learned about in the Mozart unit they had recently completed.

• This school had a teacher-coordinator who had participated as a kindergarten teacher the year before and was providing specific support for both the Core Knowledge pilot in first grade and the kindergarten classrooms, which had continued to use the curriculum.

• The school is also using the Core Knowledge curriculum for all grades.

25

Example of a school with high fidelity

26

Administrators’ Views

Teachers’ Views

• Both teachers were very positive about the program’s approach to developing early childhood literacy in their 1st graders.

• They liked using the program, its structure, content, and activities, and they also could see the students benefiting from it.

• Their criticisms of the program were of more “mechanical” elements (e..g, lack of page numbers in the readers, would have liked captions to accompany the pictures and make them more interactive, they felt rushed to get through all the material for any give day)

Administrators’ Views

• Administrators were very supportive of the program

• They perceived large benefit from the program’s “multi-intelligence” approach and have been excited to watch the students make connections between lessons and to their lives outside the classroom as well.

• They perceived more parent involvement—children dragging their parents to the library, parents commenting that they are learning from their children.

• They felt the program worked well largely because their teachers are well supported and have fully embraced the program.

• Criticisms were that the program needs to include more “higher-order thinking” questions and provide material tailored for special populations (e.g., ELLs).

Example of a school with high fidelity

Anecdotes from Classroom Observation

• The structure of the classrooms themselves created a challenge for the three first grade teachers—they only have three walls, and are open onto a hallway on the fourth side. Not only did the teachers have to contend with noise from the other classes, but also with the distractions of passersby using the hallway.

• In two of the three classes, students were having a difficult time responding appropriately to their teacher’s directions, questions, and assignments.

• The students in one of these classrooms also had a difficult time listening respectfully to each other during the discussion time at the end of a Listening and Learning lesson. This was the only example we saw of this kind of behavior across all 15 classrooms.

• One of the classrooms did not have the Core Knowledge materials posted and was missing the chaining and tricky words.

• Toward the end of the year, the CKR literacy coordinators worked with the teachers to take a different approach to differentiation. They divided the students into three homogenous skill groups and then each teacher worked with one group three days a week for a 45 min. session.

27

Example of a school with lower fidelity

28

Teachers’ Views

• Did not feel supported by the school’s administrators. This made it that much more difficult to be successful in their implementation of the program because the administrators were out of touch with what the teachers needed in order to be fully successful in their implementation.

• Sometimes found it difficult to fit in all the content for both the Listening and Learning and the Skills Strand, and would resort to picking out the highlights for a reading or selecting the most feasible activities in order to optimize their time.

• For all the challenges, the teachers were positive about the program itself—they saw their students engaging with it and connecting it to their lives outside the classroom.

Administrator Views

• The administrator we interviewed was very enthusiastic about the program.

• Confirmed that the teachers like the program, but needed a little extra support.

• Researchers and implementation managers also had difficulty communicating with administrator (e.g., email inbox was always so full that any emails we sent were returned).

Example of a school with lower fidelity

29

There was an alignment between achievement gains and fidelity to the curriculum

Year 2 School Gains on WJIII Brief Reading Assessment

Pilot/Comparison School

Average Fall to Spring Gain,

Year 2

Pilot 19.1

Pilot 10.2

Pilot* 13.1

Pilot* 3.5

Pilot 2.6

Pilot* 3.7

Comparison 2.6

Comparison 2.1

Comparison 2.2

Pilot 1.2

Comparison 2.4

Comparison 1.3

Comparison 1.1

Comparison 1.1

Comparison 1.3

Pilot* 0.5

Comparison 0.3

Pilot -1.5

Comparison -6.0

Pilot* -13.2




• One of these schools was the low fidelity school described previously.

• One of these schools had a teacher new to teaching first grade who had been in and out of the classroom, and the relationship between the principal and this teacher was strained.

• All schools spent less than the required 60 minutes on the Listening and Learning Strand and the Skills Strand.

• One of these schools also had a very strong focus on writing, which would occasionally cut into their other strand time.

Three Site Visit Schools Did Not Significantly Gain or Declined in Performance from Fall to Spring:

30

Core Components of Year 3 Evaluation

• CKR assessments and longitudinal examination of relationship between literacy and content knowledge

• Survey/ site visits at comparison schools

• Assess administrators’ and teachers’ thoughts/ plans re: 3rd grade

• Assess achievement gains of Kindergarten and 1st graders - CKR grades that are not receiving as much intensive support

• (Year 4) Track student achievement through 3rd grade (NYS tests)

Recommendations for Year 3 Implementation

Provide targeted support to teachers on:

• Differentiating instruction for students requiring different accommodations

• Finding time to complete lessons and have them allot a full 60 minutes to lessons (or adjust appropriately if not allotting full 60)

Next Steps: Year 3

11 Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 2 Report Research and Policy Support...

Documents

Transcript of 11 Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 2 Report Research and Policy Support...