SECOND REPORT ON THE ACPP: TEACHER

PERSPECTIVES & IMPLEMENTATION

LESSONS PRESENTED AT

THE STATE OF EDUCATION IN ARKANSAS 2008

Marc J. Holley, MEdDoctoral Fellow Dept. of Education Reform University of Arkansas

April 22, 2008

1

Presentation Overview

1) ACPP Description

2) Teacher Views: Research Questions

3) Teacher Surveys: Methods & Results

4) Teacher Interviews: Methods & Results

5) Bonus Amounts Awarded

6) Teacher Feedback

7) Conclusions

Why try merit pay?Of all the inputs we can control, teacher quality matters most for

student success.“…a good teacher will get a gain of 1.5 grade

level equivalents while a bad teacher will get .5 year for a single academic year.”

“…having five years of good teachers in a row (1.0 standard deviation above average, or at the 85th percentile) could overcome the average seventh-grade mathematics achievement gap between lower-income kids (those on the free or reduced-price lunch program) and those from higher-income families.”

Hanushek, E.A. & Rivkin, S.G. (2004). Similar findings by Sanders, W.L. & Rivers, J.C. (1996)

Compensation Reform as Policy Tool

The Single Salary Schedule which rewards teachers for credential and experience does not align pay with the outcome we care about most – Student Achievement.

Teacher compensation reform focused on pay-for-performance can have motivational (short term) and compositional effects (long range)

Achievement Challenge Pilot Project Description (ACPP)

ACPP - Little Rock, Arkansas Merit pay program for all staff members in a

school Reward personnel solely for increases in student

achievement

3-Year Program Created in 2004-05 at Meadowcliff Elementary Expanded to include Wakefield Elementary, 2005-

06 Expanded to include 3 more schools (Romine,

Mabelvale, & Geyer Springs), 2006-07

ACPP – Well Designed Plan

Straightforward – Based on growth, not complicated statistics

Non-Competitive for Teachers – Everyone could earn

Significant Awards - up to $10,000 for teachers

Focus on Growth of Students – address student placement

Everyone Participated – from secretaries to principals

Achievement GAINS! …BUT … An Unqualified Success???

Evaluation FocusTwo years of modest positive student gains

(Winters et al., 2008); (Barnett et al., 2007)

Should we think about other costs and benefits when evaluating a program?

What can we learn from implementation?

The Hypotheses We Tested: Proposed Benefits

Possible advantages of merit pay:1)Leads to greater innovation in

teaching2)Leads to teachers working

harder3)Leads to greater salary

satisfaction

The Hypotheses We Tested: Proposed Drawbacks

Possible disadvantages of merit pay:

1) Leads to a degraded school climate

2) Leads to an increase in counterproductive competition

3) Leads to the neglect of low-performing students

Teacher Views: Research Questions

Teacher Surveys (Compelling Control Groups)1) Do teachers in schools with a long-term merit pay (IA)

program have differing attitudes and perceptions than teachers in new merit pay schools (LA)?

2) Do the attitudes and perceptions of teachers in schools with a long-term merit pay program (IA) change over time?

Teacher Interview3) Were teachers supportive of merit pay and the ACPP,

and did their attitudes and behaviors change as a result of the merit pay program?

4) Did teachers feel the goals of the ACPP were fair and attainable, and how would they improve the program?

Teacher Surveys: MethodsSurveys were administered to teachers in

Fall ’06 & Spring ’07 at all five participating schools Survey participation was voluntary

Teachers responses were measured on a scale of 1 – 4, with a higher score representing stronger levels of agreement with that question

Questions were grouped into constructs that examined attitudes about oft-cited positives and negative associated with merit pay

Teacher SurveysComparison of IA vs. LA – Fall 2006

Construct IA Mean Score

LA Mean Score

Difference

ItI-value

p

1) I am innovative. 2.96 2.96 0.00 0.03 0.97

2) I work hard. 3.06 3.09 -0.03 0.27 0.79

3) I am satisfied with my salary.

2.34 1.88 0.46 3.47 0.00

4) My school does not suffer from negative competition.

3.33 3.20 0.13 1.78 0.08

5) My school has a positive work environment.

3.04 2.57 0.47 4.39 0.00

6) I view low-performing students as a positive challenge.

3.01 2.56 0.45 4.06 0.00

7) I am an effective teacher. 3.25 2.84 0.41 3.70 0.00

RQ1: Do teachers in schools with a long-term merit pay (IA) program have differing attitudes and perceptions than teachers in new merit pay schools (LA)?

*

**

*

Teacher Surveys: Comparison of IA vs. IA Fall 2006 to Spring 2007

Construct IA Fall Mean Score

IA Spring Mean Score

Difference ItI-value p

1) I am innovative. 2.96 2.91 0.05 0.57 0.57

2) I work hard. 3.06 3.15 -0.09 0.61 0.54

3) I am satisfied with my salary. 2.34 2.44 -0.10 0.63 0.53

4) My school does not suffer from negative competition.

3.33 3.35 -0.02 0.16 0.87

5) My school has a positive work environment.

3.04 2.96 0.08 0.65 0.52

6) I view low-performing students as a challenge.

3.01 2.87 0.14 1.10 0.28

7) I am an effective teacher.

3.25 3.21 0.04 0.32 0.75

RQ2: Do the attitudes and perceptions of teachers in schools with a long-term merit pay program (IA) change over time?

…ATTITUDES PERSISTED AFTER FIRST YEAR

Synthesis of Survey FindingsPredictions Concerning Merit Pay

Does Prediction Hold?

IA vs. LAFall 2006

Does Perception Continue?

IA vs. IA Spring 2007

Conclusion Concerning the

ACPP’s Effect on Teacher Attitudes

1) Teachers will report more innovation.

No Yes Not associated

2) Teachers will report working harder.

No Yes Not associated

3) Teachers will report more salary satisfaction.

Yes Yes Possible association

4) Teachers will report increased counterproductive competition.

No Yes Not associated

5)Teachers will report a more negative work environment.

No Yes Not associated

6) Teachers will report an increased view of low-performing students as burdensome.

No Yes Not associated

7) Teachers will report being more effective.

Yes Yes Possible association

Teacher Interviews: Methods Conducted in Fall 2007 after bonuses were disbursed Questions explored the same areas as the surveys :

Teacher salary satisfaction was not covered in the interviews

43 interviews were conducted (40 teachers, 3 principals) at the 5 ACPP schools

Responses were coded as positive, negative, or neutral A positive response would advocate for performance pay

Ex. Do teachers work harder at your school as a result of merit pay? Yes = positive response No = negative responseNo answer = neutral response

21 coded as positive, 16 were negative, & 6 were neutral

Teacher Interviews: Results

Question Positive Response

s

Negative Response

s

Overall Attitude

Confirms Survey Results

Did teachers initially support the ACPP? 17 8 Positive NA*

Did teachers support merit pay as a reform? 15 11 Positive NA*

Did the ACPP encourage teachers to become more innovative in the classroom? 5 10 Negative Yes

Did the ACPP encourage teachers to work harder? 13 10 Positive No

Did the ACPP lead to counterproductive competition amongst teachers? 10 1 Positive Yes

Did the ACPP have a negative effect on the climate of the school? 7 13 Negative No

Did the ACPP encourage teachers to work with lower-performing students? 6 2 Positive Yes

Did teachers feel that the ACPP was fair during the 2006-07 school year? 4 13 Negative NA*

RQ3: Were teachers supportive of merit pay and the ACPP, and did their attitudes and behaviors change as a result of the merit pay program?

RQ4: Did teachers feel the goals of the ACPP were fair and attainable, and how

would they improve the program? Highlighted are different findings.

*Not evaluated on the survey

Teacher Bonus Amounts: 2004-2007School Year Total

Bonus $Highest Teacher

Bonus

Lowest Teacher Bonus*

Average Teacher

Bonus

% of Teachers Receivin

g Bonuses

Meadowcliff 2004-05 $134,800 $8,600 $1,800 $5,417 100%

Meadowcliff 2005-06 $200,926 $7,300 $3,700 $5,928 93%

Wakefield $228,300 $9,200 $4,000 $6,709 100%

Meadowcliff 2006-07 $101,535 $5,100 $1,100 $3,153 100%

Wakefield $51,716 $7,600 $300 $2,335 48%

Mabelvale $39,550 $6,400 $450 $2,111 56%

Geyer Springs $64,530 $7,600 $350 $3,105 92%

Romine $12,450 $5,200 $450 $2,713 27%

* Other than $0

Ways to Improve the Program: Teacher Responses

• “I think it should have been consistent across the board. The rules changed.”

• “I don’t know if it was a lack of understanding, miscommunication, or it was literally changed after we were told that it would be this way, and then it wasn’t done that way. It was done a different way. That caused a lot of conflict.”

• “I like the pretesting in the Fall when they [the students] got here. I like the post-testing at the end of the year because that told me how good of a teacher…or what I needed to work on.”

• “I believe that no teacher should have gone without a check.”• “They [teachers] would have to be in on the planning/making.

And you would have the exact same discussion with them…what do you think? What do we want? And they would have to have so much ownership of it.”

• “Have one set of rules for the entire district. Everything has to be transparent. Our situation wasn’t transparent.”

Conclusions: Teacher Views of Merit Pay

1) Teachers initially supported the ACPP and were somewhat supportive of merit pay as a compensation reform over time.

2) Teachers did not associate merit pay with being more innovative, and responses varied in regards to working harder.

3) Teachers did not associate merit pay with counterproductive competition or viewing low-performing students as a burden, but did feel that the school environment became more negative after bonuses were disbursed.

4) Teachers reported higher salary satisfaction and felt more effective.

5) Teacher expectations play a significant role in the “success” of a merit pay program.

Take away messageMerit pay that ties bonuses directly to

student achievement gains can lead to student improvement.

Merit pay does not necessarily lead to divisive competition and nor undermine collaboration.

Participant expectations are very important, so implementers must communicate clearly and openly to set and maintain expectations for participants.

Contact Information:Marc Holley

Office for Education PolicyUniversity of Arkansas

http://www.uark.edu/ua/oepEmail: mjholley@uark.eduPhone: (479) 575-3773