Evaluating the Impact of Performance- related Pay for teachers in England Adele Atkinson, Simon...

Evaluating the Impact of Performance-related Pay for teachers in England

Adele Atkinson, Simon Burgess, Bronwyn Croxson, Paul Gregg, Carol Propper, Helen Slater, Deborah Wilson

12 April 2007 www.bris.ac.uk/Depts/CMPO/ 2

Background• Improving education outcomes key priority for

governments, but evidence suggests poor returns from simply raising school resources.

• One alternative mechanism: incentives for teachers, but rel. little evidence on impact.

• 1999: UK government introduced performance related pay scheme for teachers (the “Performance Threshold”).

• Performance assessed across five criteria, inc. pupil progress (value-added).


What we do in this paper• Quantitative evaluation of the impact of this PRP

scheme for teachers on pupil test score gains.• Design

– Longitudinal teacher-level data and a difference-in-difference research design.

– Link pupils to their teachers for each subject; collect prior attainment data for each pupil.

– So control for teacher and pupil fixed effects. – Also control for differences in teacher experience.

• Incentive scheme had significant effects on pupil progress.


Outline of the talk

• Current evidence (in paper, not here)

• The National Curriculum

• The PRP scheme

• Data

• Evaluation methodology

• Results

• Conclusion


The National Curriculum

• Centralised system of control over national exams and teacher pay scales.

• All pupils tested at the end of each Key Stage of the National Curriculum.

• KS1 and KS2 tests taken at ages 7 and 11; KS3 and KS4 (GCSE) taken at ages 14 and 16.

• KS1, 2, 3 tests taken in English, maths, science.• These subjects compulsory also at KS4.• We focus on KS4 and value added between KS3

and KS4.


The PRP scheme

• Labour administration 1998 Green Paper: range of reforms to education, inc. performance-related element to teacher pay.

• The ‘Performance Threshold’ introduced in 1999/2000; first applications in July 2000.

• The Performance Threshold itself was one element of larger pay reform, designed to affect teacher effort, as well as recruitment and retention.


The PRP scheme II

• Prior to the PRP scheme, all teachers paid on unified basic salary scale which had 9 full points.

• Position on scale depended on qualifications and/or experience; progress through annual increments. Plus additional management points available.

• 1999/2000: approx. 75% of teachers at top of scale, at spine point 9.


The PRP scheme III

• After the reforms, teachers on spine point 9 could apply to pass the Performance Threshold. 2 effects:– Annual bonus of £2,000.– Move onto the Upper Pay Scale (UPS):

additional spine points, each of which related to performance.


The PRP scheme IV• To pass the Threshold, teachers had to

demonstrate effectiveness in five areas, including pupil progress (value added).

• Forms submitted by July 2000. Assessed by headteacher and external assessor.

• Initial Threshold payments funded out of a separate, central budget; no quota or limit.

• The vast majority of eligible teachers both applied and were awarded the bonus.


Was it incentive pay?

• Wragg et al (2001) survey of 1000 schools– In these schools, 88% of the eligible teachers applied,

and of these 97% were awarded the bonus

– Unconditional pay increase - little effect on teacher effort.

• But– Ex ante (Marsden) survey suggests teachers believed it

to be ‘real’

– UPS element clearly performance related


Teacher survey before implementation

Disagree (%)

Neutral (%)

Agree (%)

“The Green Paper pay system is a device to avoid paying more money to all teachers”

9 18 68

“In practice, many excellent teachers will not pass the Threshold because there is certain to be a quota on places available”

3 8 82


Data requirements

• Control for pupil prior attainment to measure progress or value added:– KS3-GCSE; English, maths, science.

• Longitudinal element:– Follow same teachers through two complete KS3-

GCSE teaching cycles (before and after scheme introduced).

• Link pupils to teachers:– Obtain class lists direct from schools.


Sample

• First approached schools in 2000.• Onerous data requirements; problems with school

information systems; teacher and headteacher turnover.

• Final sample:– 18 schools.– 181 teachers (145 eligible; 36 not eligible).– Approx. 23,000 pupils.

• No presumption that sample is representative.


Evaluation Methodology

• Pupil i; teacher j; teaching cycle t.

• Teacher effectiveness, X; test score, g; value-added v.

tjiijiitji TSXeZg )()(321)(

jtjtjjt WfbIbX 1


• Teacher mean scores:

• Difference between two tranches:

TSWfbIbeZg jtjtjjjjt 133321

TWfbIeZg jjjjj 13321


• Differencing between eligible and ineligible teachers. D(x) operator means:D(x) E(x|I=1) – E(x|I=0)

• This is the difference-in-difference. If

• This yields:

WfDbeDZDgD 13321

021 eDZDE

WfDb 133


• For value-added:

• And same steps as before yield the following as the diff-in-diff:

tjiijitji TSXev )()(21)(

WfDbeDvD 1221


Key issues

• Parameters of interest are: – for gross test score

– for value added

• Role of experience profile:– If f(W) is linear, no problem, as Df(W) = 0– If concave, diff-in-diff underestimates

parameters of interest, as Df(W) < 0.


Experience Profile


Key issues (cont.)

• Experimental design and pupil assignment:

• No grouping on effort.

• Timing of class assignment.

021 eDZDE

01 eDE


Results

• Difference-in-difference results

• Regressions

• Robustness checks

• Interpretation and evaluation


Table 2: D-inD analysis: GCSEs

Overall

Percentiles

Tranche Eligible Sample

Size Mean Standard Deviation Min 25th 50th 75th Max

Output Measure

1 Yes 145 4.84 1.30 1.33 3.99 4.90 5.58 7.74 GCSE

2 Yes 145 5.05 1.28 1.62 4.12 5.12 5.90 7.79

Difference 0.21 1.24 -3.62 -0.29 0.16 0.75 4.05

1 No 36 4.78 1.50 1.08 3.81 5.26 5.71 6.90 GCSE

2 No 36 4.91 1.05 2.33 4.30 4.71 5.73 7.08

Difference 0.13 1.53 -1.91 -0.88 -0.64 1.23 3.70

Difference in

Difference 0.08

(0.17) 0.59

(0.10) 0.81

(0.11) -0.47 (0.38)


Table 3: D-in-D VA Means

Overall

Percentiles

Tranche Eligible Sample

Size Mean Standard Deviation Min 25th 50th 75th Max

Output Measure

1 Yes 145 -0.08 0.54 -1.93 -0.46 -0.11 0.30 1.38 Value added

2 Yes 145 0.05 0.57 -1.57 -0.35 0.05 0.51 1.50

Difference 0.13 0.56 -1.58 -0.13 0.16 0.42 2.03

1 No 36 -0.04 0.62 -1.42 -0.39 -0.07 0.49 1.29 Value added

2 No 36 0.06 0.53 -0.89 -0.31 -0.07 0.59 1.25

Difference 0.10 0.66 -1.12 -0.40 0.05 0.43 1.92

Difference in

Difference 0.03

(0.08) 0.28

(0.10) 0.11

(0.11) -0.01 (0.12)


Experience Difference

• Potentially need to control for systematic differences in experience: ideal: non-parametrically defined experience-effectiveness profile.

• Not enough data to do that, so define a ‘novice’ teacher dummy picking out teachers with least experience.


Table 4: GCSE AnalysisUnit of observation is a teacher Dependent variable is teacher change in GCSE score (1) (2) (3) (4) Eligible 0.706 0.286 0.890 (0.371) (0.298) (0.366)* Novice 1.333 (0.489)** Spine point -0.092 -0.181 (0.065) (0.080)* Years in School -0.001 -0.114 -0.120 -0.008 (0.015) (0.251) (0.254) (0.015) Leadership 0.831 2.144 0.277 0.873 (0.927) (1.150) (0.789) (0.803) Female teacher 0.062 -0.020 0.096 -0.003 (0.220) (0.223) (0.222) (0.220) Maths dummy -0.174 -0.191 -0.132 -0.135 (0.250) (0.248) (0.251) (0.246) English dummy -0.129 -0.114 -0.120 -0.121 (0.253) (0.251) (0.254) (0.249) Constant 0.433 0.685 -0.532 -1.096 (0.659) (0.667) (0.451) (0.488)* Observations 180 180 181 181 R-squared 0.15 0.17 0.14 0.18


Table 5: Value Added Analysis Unit of observation is a teacher Dependent variable is teacher change in VA score (1) (2) (3) (4) Eligible 0.193 0.063 0.422 (0.173) (0.128) (0.152)** Novice 0.539 (0.203)** Spine point -0.022 -0.057 (0.027) (0.033) Years in School -0.003 -0.005 -0.006 -0.005 (0.006) (0.006) (0.006) (0.006) Leadership 0.045 0.556 -0.005 0.236 (0.381) (0.473) (0.327) (0.333) Female teacher -0.014 -0.046 -0.001 -0.040 (0.091) (0.092) (0.092) (0.091) English dummy -0.145 -0.139 -0.134 -0.134 (0.104) (0.103) (0.105) (0.103) Maths dummy -0.337 -0.344 -0.315 -0.317 (0.103)** (0.102)** (0.104)** (0.102)** Constant 0.308 0.407 -0.021 -0.249 (0.271) (0.275) (0.187) (0.202) Observations 180 180 181 181 R-squared 0.27 0.29 0.26 0.29


Table 6: Subject Differences GCSE Value added Eligible 1.339 0.653 (0.437)** (0.180)** Novice 1.313 0.511 (0.480)** (0.202)* Interaction: Eligible* -0.771 -0.394 English

(0.667) (0.275)

Interaction: Eligible* -1.138 -0.588 Maths

(0.625) (0.257)*

Years in School -0.010 -0.006 (0.015) (0.006) Leadership 0.531 0.062 (0.862) (0.355) Female Teacher -0.059 -0.070 (0.223) (0.092) English dummy 0.554 0.211 (0.612) (0.252) Maths dummy 0.791 0.162 (0.563) (0.232) Constant -1.468 -0.441 (0.524)** (0.216)* Observations 181 181 R-squared 0.20 0.31


Robustness checks

• Leave out novices to just compare eligibles and ineligibles with pretty similar experience.

• Ceiling effects on marks: just look at pupils in bottom 75% of KS3 distribution

• Robust to these


Evaluation

• One standard deviation in the teacher-mean change in GCSE is 1.29, and 0.58 for VA– Coefficients on eligibility of 0.890 for GCSE

change and 0.422 for VA change– As percentages of a standard deviation these are

69% and 73%

• Alternatively, eligibility dummy is 67% of the novice teacher dummy for GCSE change, and 78% for VA change.


Conclusions

• Rich data, research design which controls for teacher and pupil effects

• Results: around 0.5 GCSE grade per pupil

• Caveats– Was it incentive pay and the experience-

effectiveness profile– Extra effort or effort diversion?


Additional slides


Table 2: Summary teacher stats Means Missing

data Missing

Eligibility Info

All Eligible Ineligible Teacher Characteristics

Age

41.9 (80)

44.8 (61)

32.1 (18)

102 1

Female dummy

0.57 (182)

0.59 (145)

0.47 (36)

0 1

Years at School

10.6 (182)

13.2 (145)

5.44 (36)

0 1

Spine point

8.59 (181)

9 (145)

7.26 (35)

1 1

Leadership Scale 0.02 (182)

0 (145)

0.11 (36)

0 1

Eligibility 0.8 (181)

1 (145)

0 (36)

0 1

Mean KS3

5.15 (182)

5.24 (145)

5.08 (36)

0 1

Mean GCSE

4.65 (182)

4.81 (145)

4.65 (36)

0 1

Mean Value added

-0.09 (182)

-0.03 (145)

-0.02 (36)

0 1


Table 3: Summary pupil stats Tranche 1 Tranche 2 Pupil Characteristics

Gender 0.53 (10777)

0.50 (12170)

KS3

5.29 (10611)

5.34 (11902)

English

5.12 (2706)

5.13 (2865)

Science

5.34 (5515)

5.53 (6418)

Maths

5.39 (2386)

5.53 (2619)

GCSE 4.88 (10717)

5.02 (11559)

English 5.04 (2805)

5.16 (2879)

Science 4.92 (5493)

5.11 (6044)

Maths 4.62 (2415)

4.65 (2636)

Value added

-0.04 (9830)

0.03 (11140)

English

0.33 (2593)

0.39 (2786)

Science

-0.07 (4970)

0.09 (5809)

Maths

-0.38 (2263)

-0.49 (2545)


Table 4: Comparative stats National

Estimation

Tranche 1 Tranche 2 Difference Tranche 1 Tranche 2 Difference KS3

English

4.66 (0.0022)

4.82 (0.0021)

0.16 (0.0031)

5.12 5.13 0.01

Science

4.93 (0.0017)

4.88 (0.0018)

-0.05 (0.0024)

5.39 5.53 0.14

Maths

5.02 (0.0019)

5.24 (0.0019)

0.22 (0.0027)

5.33 5.36 0.03

Overall 4.87 (0.0013)

4.99 (0.0011)

0.12 (0.0016)

5.29 5.34 0.05

GCSE

English

4.82 (0.0024)

4.81 (0.0022)

-0.01 (0.0003)

5.04 5.16 0.12

Science

4.32 (0.0026)

4.21 (0.0024)

-0.11 (0.0035)

4.92 5.11 0.19

Maths

4.13 (0.0027)

4.21 (0.0025)

0.08 (0.0037)

4.62 4.65 0.03

Overall 4.42 (0.0015)

4.41 (0.0014)

-0.01 (0.0020)

4.88 5.02 0.14

Value added

English

0.553 (0.0017)

0.504 (0.0016)

-0.049 (0.002)

0.33 0.39 0.06

Science

-0.132 (0.0017)

-0.074 (0.0015)

0.058 (0.002)

-0.07 0.09 0.16

Maths

-0.425 (0.0016)

-0.427 (0.0014)

-0.002 (0.002)

-0.38 -0.49 -0.11

Overall -0.002 (0.0010)

0.002 (0.0009)

0.004 (0.0014)

-0.04 0.03 0.07


Data requested from schools

Data requested Level Class lists for year 10 in 1997/8 and year 11 in 1998/9, with pupil identifiers and teacher identifiers

pupil

Class lists for year 10 in 2000/1 and year 11 in 2001/2, with pupil identifiers and teacher identifiers

pupil

Pupil test/exam scores for Key Stage 3 in 1996/7 and GCSE 1998/9, for all English, maths and science subjects, with pupil identifiers

pupil

Pupil test/exam scores for Key Stage 3 in 1999/00 and GCSE 2001/02, for all English, maths and science subjects, with pupil identifiers.

pupil

Supplementary information for each pupil: date of birth, gender, postcode, with pupil identifiers.

pupil

Teachers characteristics at 1 September 1999: age, gender, salary, experience, spine point, whether applied for PT, with teacher identifiers.

teacher

Information about school policy: exam boards used, streaming/setting policy, pre-existing performance management system.

school


0

2

4

6

8

GC

SE

score in tranche 2

0 2 4 6 8 GCSE score in tranche 1

Eligible Not eligible

Mean GCSE score in tranche2 against Mean GCSE in tranche1


-2

-1

0

1

2

GC

SE

score in tranche 2

-2 -1 0 1 2 GCSE score in tranche 1

Eligible Not eligible

Mean VA in tranche 2 against Mean VA in tranche 1


0

2

4

6

8

Teacher m

ean GC

SE

score

0 5 10 15 20

Pay spine point in tranche 1


Table 12: Distributional Impacts


Table 13: Robustness checks

No Novices Lower 75% KS3 only GCSE VA GCSE VA Eligible 1.389 0.786 0.996 0.605 (0.487)** (0.204)** (0.421)* (0.191)** Novice 1.099 0.511 (0.447)* (0.203)* Interaction: Eligible* -1.980 -0.962 -0.560 -0.328 English

(0.922)* (0.386)* (0.606) (0.276)

Interaction: Eligible* -0.877 -0.772 -0.955 -0.559 Maths

(0.824) (0.345)* (0.579) (0.264)*

Years in School -0.017 -0.007 -0.018 -0.010 (0.014) (0.006) (0.015) (0.007) Leadership -0.329 -0.267 0.381 0.139 (0.927) (0.388) (0.813) (0.370) Female teacher 0.166 -0.059 0.041 -0.002 (0.233) (0.097) (0.209) (0.095) English dummy 1.691 0.791 0.274 0.100 (0.888) (0.372)* (0.553) (0.251) Maths dummy 0.568 0.365 0.560 0.062 (0.778) (0.325) (0.517) (0.235) Constant -1.497 -0.569 -0.924 -0.301 (0.560)** (0.234)* (0.507) (0.231) Observations 164 164 178 178 R-squared 0.20 0.32 0.17 0.32

Evaluating the Impact of Performance- related Pay for teachers in England Adele Atkinson, Simon...

Documents

Transcript of Evaluating the Impact of Performance- related Pay for teachers in England Adele Atkinson, Simon...