Avoidable Losses

Rice UniversityCreekmore Symposium

April 29, 2008

The Problem

• 135,000 youth are lost from Texas high schools every year prior to graduation.

• These are disproportionately African-American, Latino, and English Language Learners

High Stakes Accountability

• More than a decade of reform

• Aimed at: increasing student achievement closing the achievement gap

• Became the model for the nation

Avoidable Losses: Our Findings

• The high stakes accountability system is directly connected to the severity of the dropout problem

• This occurs when administrators are in compliance with the accountability system School ratings go up as weak students exit the

system Students are viewed as assets or liabilities to the

school ratings

How did we discover this connection?

• Data from 271,000 students over 7 years.

• Analysis from inside 7 schools

• Multi-year case study of a school working to comply with the accountability system

• Student interviews to learn their perspectives on life in school under this system

Avoidable Losses: The Study

• Accountability Miracle in Brazos City

• Educate or Comply

• The Culture of Accountability: Setting Kids Up to Fail

Accountability Miracle in Brazos City

Julian Vasquez Heilig, Ph.D.University of Texas at Austin

Texas Miracle

• In 1993, the Texas Legislature enacted a ground-breaking statute that mandated the creation of the Texas public school accountability system to rate school districts and evaluate campuses

• During the 2000 presidential campaign, George W. Bush touted the “Texas educational miracle” as a model for the rest of the nation

• George W. Bush and Rod Paige, two primary arbiters of NCLB, lassoed their ideas for federal education policy from Texas

The Miracle

• Closing achievement gap

• Low dropout rate

• Soaring Exit test scores

• Rising graduation rates

• Rising accountability ratings

Closing the Achievement Gap?

BCSD Reported Dropout by Year

Graduation Rates and Exit

• BCSD reported: Graduation rate had soared 21 percentage points over five years, from 54 percent in 1997 to 75.3 percent by 2002

• Local Newspaper characterized the high school achievement gains with the headline: “Sophomores soar in Exit TAAS exam”

TEA Accountability Ratings for BCSD High Schools (1996-2002)

0%

20%

40%

60%

80%

100%

1996-1997 1997-1998 1998-1999 1999-2000 2000-2001 2001-2002

School Year

Percent Rated

Low -Performing

Acceptable

Recognized

Exemplary

TEA Accountability Ratings for BCSD High

Schools (1996-2002)

• Considering BCSD’s closing achievement gap and rising accountability ratings (ultra low dropout and high graduation rates) it might be suggested that BCSD schools dramatically improved urban minority students’ academic and enrollment success.

• Indicative of an education miracle- worthy of national acclaim and replication into a comprehensive national educational policy (NCLB).

Accountability Miracle Reconsidered

• Testing analysis

• Longitudinal cohort analysis

• Grade retention

• Graduation

BCSD Testing: High-stakes vs. Low-stakes

• On average, about 12% of African American and Latino elementary and middle school students were excluded from the TAAS (high-stakes) but not Harcourt (low-stakes) testing from 1997-2002.

• In contrast to the TAAS, there is little difference in test-taking patterns on Harcourt tests by race/ethnicity.

• As a result, greater proportions of Latino and African American students’ test scores were excluded from TEA accountability ratings

TAAS and Harcourt (SAT-9 or Aprenda) Mathematics Testing Grades 3 -8, By Race/Ethnicity (1997-2002)

No Harcourt Math Harcourt Math Tested 1997-2002

Asian American No Math TAAS 3% 9%

TAAS Math Tested 1% 87%

African American No Math TAAS 3% 12% TAAS Math Tested 2% 83%

Latino No Math TAAS 2% 12%

TAAS Math Tested 1% 85% White No Math TAAS 3% 6%

TAAS Math Tested 2% 89%

Mean Scores by Participation

SAT/9 Mean Scores, by English TAAS Participation Status (1997 -2002), Grades 3 -8

SAT/ 9 Reading Score SAT/9 Math Score Took TAAS No TAAS Took TAAS No TAAS Overall Mean Scores (N)

459.70 (341,169)

210.54 (65,307)

504.9 (346,663)

273.75 (59,813)

African American Mean Scores (N)

439.95 (125,008)

196.78 (25,737)

467.12 (127,584)

233.91 (23,161)

Latino Mean Scores (N)

415.29 (163,522)

201.21 (33,097)

479.22 (166,004)

282.49 (30,615)

Native American Mean Scores (N)

556.33 (253)

354.90 (28)

588.76 (258)

427.38 (23)

Asian American Mean Scores (N)

598.29 (11,144)

272.27 (1,614)

705.97 (11,221)

455.31 (1,537)

White Mean Scores (N)

654.97 (41,242)

327.76 (4,831)

664.86 (41,596)

361.09 (4,477)

Those excluded from the English TAAS, the basis of the state and district accountability rankings, scored significantly lower on the Stanford-9 than those who took the TAAS across all racial/ethnic groups (p<.001)*.

*A GLS regression analysis examined the effects of a dummy variable representing inclusion or exclusion in the English TAAS (controlling for race / ethnicity, language status, income status, and school-level proportions of at-risk students).

BCSD High School Cohort Progression

14320

7532

6307 6447

18292

8552

7190

16707

8517

6645

6875

6901

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

1995-1996 1996-1997 1997-1998 1998-1999 1999-2000 2000-2001 2001-2002

School Year

Number of Students

9th Grade 1996-1997

9th Grade 1997-1998

9th Grade 1998-1998

BCSD High School Cohort Progression: Latinos and African Americans

100

50.4

42.2 43.4

100

44.9

37.335.8

100

48.2

38.7 37.5

0

10

20

30

40

50

60

70

80

90

100

1995-1996 1996-1997 1997-1998 1998-1999 1999-2000 2000-2001 2001-2002

School Year

Percent in Grade

9th Grade 1996-1997

9th Grade 1997-1998

9th Grade 1998-1999

BCSD Grade Retention by Year

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

1996-1997 1997-1998 1998-1999 1999-2000 2000-2001 2001-2002

School Year

Percent Retained

3

4

5

6

7

8

9

10

11

12

BCSD 9th Grade Retention by Race/Ethnicity

BCSD 1997-2001 Cohort Testing on the Exit

19.420.3

21.4

40.339.4

40.3 40.3 40.3

38.3

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

Writing Read Math

TAAS Exit Test

Percent of Students

No Pass Pass No Test

Retention?: Only 209 of 3,489 retained students (about 8% of the total) ever became eligible to graduate by passing all three subjects on the spring Exit TAAS.

BCSD Graduation Rate of 1997-2001 9th Grade Cohort (5-Yr)

Student Characteristics

% Graduation Eligible within

5 Years

% Coded Graduated

within 5 Years Overall 30.1 32.7 Anglo 44.9 43.3 Latino 26.1 24.8 African American 29.3 39.4 Asian American 53.1 49.4 Econ Disadvantage 26.3 28.3 LEP 14.1 20.0 No Pass 8th Read TAAS 7.3 19.3 No Pass 8th Math TAAS 9.7 22.3

• Large number of students are being retained and otherwise not advancing with their peers through school

• Many students not testing on the Exit

• Cohort graduation rates should be correspondingly low

BCSD Graduation Rates

• Vast discrepancy between the cohort graduation eligibility and status rates and publicly released rates

• District hemorrhaged students, especially between the 9th and 10th grades

• A counselor at King high school estimated from her experience, “I would think that the graduation rate is closer to 40-45%, not 85%.”

She related,

Ultimately what’s happening is that we’re letting kids down. We’re using some kind of system to disguise where they are. If you’ve got 600 kids in your school and 300 graduate….they’re somewhere...

Left Behind?

• Is it possible that high-stakes testing and accountability may not be as good as advertised?

• Voices from BCSD high schools on their response to the pressure of high-stakes testing and accountability

Inferential Regression Analyses

• Do actions that impact student progress spur increases in high-stakes test scores and TEA accountability ratings?

• GLM Random-Effect Regression Increasing retention and disappearance of students

increases average Exit scores

• Multinomial Logistic Regression By increasing 9th grade retention, schools increase and

maintain accountability rating

• Robustness of GLM random-effect and multinomial logistic regressions suggest that manipulating student progress can increase Exit scores and accountability ratings

Summary of Findings

• High-stakes test scores and accountability ratings steadily rise as students steadily leave

• Incentives to game and reduce educational opportunity for low-performing students

• High-stakes testing and accountability policies led to an escalation of deleterious outcomes disproportionately impacting minority students

To Comply or To Educate

Judy Radigan , Ph.D.

A Case Study in Compliance

• Edgeview H S, representative of Brazos City high schools with losses of low-income and minority youth

• 75% Latino, 20% African American, 4% White, 10% English Language Learners

• Reforming school working to personalize the learning environment with small learning communities

• Spring 1997 – Principal targeted to improve scores markedly or face TERMINATION

Ninth Grade Waiver• Based grade promotion from 9th grade on

passing of core courses: English, social studies, math, science rather than number of credits gained

• All the schools had it. In fact, we were one of the last schools to accept the waiver because philosophically we thought [to do so] would be cheating. We were testing everybody. We felt we had no choice but to move to that waiver in order to save face and get our scores up.

• A lot of the kids were defeated, and they dropped out.

Texas Miracle

TAAS 98/99 99/00 00/01 01/02All 52.4 76.3 78.1Reading 69.6 86.1 94.5 92.6Writing 79.6 87 86.4 87.6Math 60.7 87.5 87.6 89.2

Edgeview’s accountability rating moved from “acceptable with acceptable progress” to “recognized with exemplary progress.”

Withdrawal from High School

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Fine Oak

Edgeview

Crockett

King

Carver

Clearbend

Patterson

Withdrew99 Withdrew00 Withdrew01

9th Grade Retention

0

10

20

30

40

50

60

70

80

90

100

Fine Oak

Edgeview

King

Carver Clearbend

Crockett Douglass Patterson

Pro

po

rtio

n R

eta

ine

d

1997-1998

1998-1999

1999-2000

2000-2001

Structural Change Small Learning Communities

1994-2000

2000-2001 2001-2002

Magnet: International Studies Engineering & Industrial Technology Governmental & Public Service Environmental & Technology Business & Commerce

Magnet: International Studies 9th Grade Track Three Track Four

Magnet: International Studies 9th-10th Track T hree Track Four (Dropbacks)

Accountability Incentives: From System to Principal

Federal

State

school

District

schoolschool

schoolschool

vendors consultants

Texas Miracle?

It’s not a miracle to manipulate things.

A miracle is saving kids actually, in reality—that’s what miracles are.

It’s not to manipulate things so that it appears—it’s a facade.

The Culture of Accountabilty:Setting Up Kids to Fail

Eileen M. Coppola, Ed.D.

Expanding the Lens/Drilling Down into Student Experience

• How does what we’ve seen at Edgeview become systematically embedded?

• How does the operation of the accountability system become “normal” or unquestioned?

• How do students experience school under this system?

• How does their experience affect their motivation and ability to stay in school?

• Focus groups and individual interviews with school staff and students in 7 Brazos City High Schools. 3 schools majority African-American; 4 majority Latino; all majority low SES.

• 14 Administrators, 24 Math and English Teachers.

• 122 students selected by teachers from senior English classes and volunteers from a program for former dropouts.

Data

Other Components of the Accountability System

• Zero-tolerance attendance policy.

• Grade retention.

• Curriculum degradation.

Beneath the Surface: Connecting Dropping Out with the Accountability System

• Extensive grade retention leads to frustration.

• Texas’ 90% rule on attendance mean that students are penalized and lose credit for courses.

• The demands of high-stakes testing degrades the curriculum over time.

• Punitive culture and rules conflict with cultural norms and adult status of students.

Grade Retention

Considerable research shows an increased likelihood of dropping out each time a student is retained in grade.

High levels of frustration, disorientation, and alienation reported by students, leading to other choices than continuing school.

(Focus group with former dropouts who returned to school)

Jose: I’ve repeated ninth grade four – four times.

Arturo: Same thing with me. I repeated ninth grade four years because I just slacked off. I was lazy and –

Int: You repeated ninth grade four years?

Arturo: Yes. …Well part of my problem was my freshman year it was – if I didn’t know anything I wouldn’t do it, or I wouldn’t go to class because, you know, I was afraid if they’d call on me and have me go up I wouldn’t know how to work it so I just wouldn’t go to class. So that was just maybe the reason.

The Effects of Retention

The Effects of Retention, Continued

Paolo: Oh, yeah, they had me taking Algebra for, like, three years straight. I passed the first year, so in the second year I just decided

not to go. I tried to get it fixed, but they wouldn’t fix it. So after the third week trying to get it fixed I just

stopped going.

Int: So did they count you as absent?

Paolo: Yeah, and when they tried doing that I just got more aggravated and stopped coming to school period.

Attendance Policies

• 90% rule

• Loss of credit

• Evolution over time/credit recovery courses

• Intervention of Justice of Peace and Juvenile Court – ticketing

• Yet, many reasons for attendance issues: family, working, need for flexibility.

(Focus group with former dropouts who returned to school)

Carla: I went to three elementary schools…then Middle School. Then I came to ninth grade. When I got to ninth grade, you know, I just made it as a fun thing, you know. I had more freedom than what I had in other schools. Learned – made new friends, you know.

Well I flunked ninth grade and back to ninth grade again. Kept going in circles and circles, and my mom used to get tickets and tickets so, you know, it’s just left for me to drop out instead of me just giving my mom nothing but tickets. So I dropped out, and I stayed, like, a whole year without being in school.

Int: So what did they actually say – I mean, what happened?

 Carla: They sent me a truancy letter saying that if I missed one more day of school that I would be fined….A $500 ticket and a court date.

 

The Effects of Attendance Policy

Lavone: Per absence.

Carla: Per absence, and I was – but I had my excuse. But most students –

Int: See, I was wondering if that $500 fine, if they would (that would make you really stop coming to school?

Lavone: That would be a lot.

Sean: Some people – rich people really don’t care. For that reason they know if they’re still going to miss another day of school they’ll probably okay with it.

Carla: I might as well drop out.

Sean: I might as well drop out.

 Carla: And no one’s going to pay $500 a day.

 

Effects of Attendance Policy, Continued

Curriculum Degradation

• Curriculum that is increasingly fragmented and shaped by the forms and priorities of the standardized tests themselves.

• This fragmentation leads to inauthentic curriculum that further alienates students, who become bored and frustrated.

• Schools continued to have limited capacity to help struggling students. Students complain that they struggle, and worry about the tests, but cannot understand the work.

Example: Curriculum Degradation

Irma: Instead of teaching us the real life things that we going to need for college and stuff, they started zeroing in just on that test. So it makes everybody nervous, and it threw everybody off. So, like our curriculum is thrown off, ‘cause what they originally were teaching us in the subjects, all of the sudden they switched and then they were just zeroing into this test.

Fernando: Math is – its not that it’s boring, it’s just hard. I’ll get it in the beginning, and I’ll get lost. I won’t know what’s

going on…[The teacher] helps but, you know, it just –

John: He [the teacher] doesn’t break it down a lot.

Fernando: Yeah, he don’t – and then his examples are wack. He takes it out of his mind.

Int: How does he decide what math you guys should learn?

Fernando: Right now he’s teaching us the TAKS.

John: Yeah, right now he’s preparing us for TAKS….

John: They teach us what’s on the test – what’s going to be on the test.

Fernando: You need that to graduate.

Second ExampleCurriculum Degradation

Culture of Accountability

• Culture of accountability: alienates students can degrade curriculum so it is overly oriented to

tested skills does not address instructional capacity of

schools or faculty to undertake challenging teaching

Is technical, bureaucratic, and punitive, not developmental

Has NCLB Made a Difference?

• NCLB has exacerbated the pressure for increased school ratings.

• Schools at risk of losing funding

• NCLB provisions raise the stakes by putting the school at risk of closure.

Conclusions

• These losses are avoidable by rebuilding a system that: is less rigid, less punitive, more driven by

developmental concerns, makes school engaging places for young people

dramatically improves the capacity of schools to serve the students most at-risk of dropping out

conceives accountability as running in both directions: communities must adequately fund and support schools

communities become totally involved in thinking about what we want from our schools

Avoidable Losses: High-Stakes Accountability and the

Dropout Crisis

Education Policy Analysis Archives

January, 2008

http://epaa.asu.edu/epaa/v16n3/