Special Data Opportunities in Florida

Special Data Opportunities in Florida

David N. FiglioUniversity of Florida and

National Bureau of Economic Research

What’s so special about Florida?

• Florida has developed a remarkable ability to analyze data across a wide variety of settings

• Facilitated through legislative authority – but reflective of exceptional nurturing of interagency relationships that facilitate data-sharing

• Easier also to collect large datasets directly from school districts in areas not present in state data

• Due in part to the relatively small number of highly-organized school districts that are acculturated to data-sharing and facilitating policy-motivated research

Two main special data opportunities in Florida

• Florida Education and Training Placement Information Program (FETPIP) and K-20 Education Data Warehouse– Established in 1988 to document compliance with

vocational education performance requirements– Expanded to become a system of collecting and

sharing individual-level data across all education sectors (K-12, vocational education, community colleges, state universities)

– Follows education, employment and earnings outcomes longitudinally for students – integrated into a relational database

Two main special data opportunities in Florida

• Independent extensive surveys of school policies and practices– Collaborative effort between David Figlio (U. Florida),

Dan Goldhaber (U. Washington), Jane Hannaway (Urban Institute) and Cecilia Rouse (Princeton U.)

– Attempted census of all public schools in Florida – First conducted in 1999-2000, then followed up in

2001-02 and 2003-04– Surveys of teachers in 2000-01, 2002-03, and 2004-

05 in 288 representative elementary schools– Better than 70 percent response rates in all waves;

over 80 percent longitudinal response rates

Examples of FETPIP data

• Data from public two- and four-year institutions, as well as in-state private colleges and universities

• In-state, data are collected on courses taken, programs of study, and attainment

• Florida is working with the National Student Clearinghouse to include enrollment and credential records for a large fraction of out-of-state college students

Examples of FETPIP data

• Using social security numbers, FETPIP marches student records with other outcome data for all students exiting Florida public institutions as well as some private exits

• Examples include information on further education, job placement, compensation, military service, incarceration

• Result: a remarkable tool for policy generation, evaluation and research

School Surveys

• All “regular” public schools in 2001-02 and 2003-04;

• 70%+ response rate in each year;

• 2,095 schools responded in 2002; 81% of these responded in 2004.

School policies/practices can be grouped into domains

• Policies to improve low-performing students

• Lengthening instructional time• Reduced class size for subject• Minimum time required for

tested subject instruction• Minimum time required for non-

tested subject instruction• Scheduling systems• Additional school resources

• Policies to improve low-performing teachers

• Teacher resources• Teacher incentives• Teacher autonomy• District control• Principal control• School climate

Feeling the Florida Heat? How Low-Performing Schools Respond to

Voucher and Accountability Pressure

Cecilia Elena RousePrinceton University & NBER

Jane HannawayThe Urban Institute

Dan GoldhaberUniversity of Washington

David N. FiglioUniversity of Florida & NBER

There is little evidence on how schools respond to accountability pressure…

• Improved teacher effectiveness and greater focus on basics;

• Teaching to the test;• Cheating;• Reclassification of students;• “Strategic” suspension of

students.

Our question: Do schools change their policies and practices in

response to school accountability and voucher pressure?

Our approach:

• Study effects of school accountability on student test score performance in Florida with a change in the A+ Plan for Education;

• Analyze effects of accountability on schools using longitudinal data on school policies collected from surveys of school principals in 1999-00, 2001-02, and 2003-04;

• Attempt to determine if the policy changes explain the test score effects.

We find…

• Among elementary schools, student achievement significantly increases among F-graded schools;

• F-graded schools appear to respond with policy changes;

• These policy changes appear to explain non-trivial portions of the student gains, particularly in math.

Since 2002, Grade Points =

percent students meeting levels 3+ in reading, writing, and math

+

percent students making “learning gains” in reading and math

+

the percentage of the bottom 25% that have improved scale points in reading.

Table 1: The Distribution of Elementary School Grades, by Year

School Year

School Grade Summer

1999 Summer

2000 Summer

2001 Summer

2002 Summer

2003 Summer

2004

Elementary Schools

A 119 485 389 623 928 974

B 214 180 324 368 360 333

C 713 614 636 452 299 284

D 448 260 215 124 63 67

F 61 4 0 35 18 9

N 0 0 46 68 2 0

Total 1555 1543 1610 1670 1670 1667

Table 2: Transition Matrix in Predicted Grades Based on 2002 Grade Change, Elementary Schools

(row percentages)

Grade in 2002 based on new (summer 2002) grading system

A B C D F

A 0.89 0.11 0.00 0.00 0.00

B 0.48 0.38 0.13 0.26 0.00

C 0.23 0.25 0.46 0.05 0.28

D 0.02 0.01 0.38 0.44 0.15

Simulated grade in

2002 based on

old (summer

2001) grading system F 0.00 0.00 0.00 0.29 0.71

Appendix Table 1 (part 1): Mean School Characteristics in 2002

2002 2004

Schools in

Frame

Schools in Survey

Schools in

Frame

Schools in Survey

Variable (1) (2) (3) (4) Proportion Classified

Disabled 0.156 0.155 0.156 0.159

Proportion Free or Reduced Lunch Eligible

0.541 0.534 0.539 0.531

Proportion Gifted Students 0.041 0.043 0.041 0.042 Proportion LEP 0.087 0.084 0.087 0.084 Stability Rate 0.931 0.932 0.931 0.933 Number of Students 687.289 685.795 687.339 693.963 Special Education

Expenditure per Pupil 8564.039 8458.324 8554.113 8507.815

Regular Education Expenditure per Pupil

4532.851 4495.650 4524.682 4479.649

At-risk Students Expenditure per Pupil

5624.279 5632.600 5644.243 5483.582

Vocational Education Expenditure per Pupil

127.674 120.055 128.304 120.149

Appendix Table 1 (part 2): Mean School Characteristics in 2004

2002 2004

Schools in

Frame

Schools in Survey

Schools in

Frame

Schools in Survey

Variable (1) (2) (3) (4)

School Grade in 2002 = A 0.355 0.366 0.352 0.358

School Grade in 2002 = B 0.220 0.221 0.220 0.228

School Grade in 2002 = C 0.291 0.286 0.289 0.288

School Grade in 2002 = D 0.072 0.068 0.072 0.066

School Grade in 2002 = F 0.024 0.023 0.024 0.022

Number of Schools 2699 2095 2837 2088

Table 3 (part 1): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on

Student Performance

Standardized test score

Model specification High-stakes

reading High-stakes

math Low-stakes

reading Low-stakes

math All elementary cohorts with lagged test scores

Standardized test performance in 2002-03

0.096 (0.029)

0.140 (0.032)

0.059 (0.021)

0.069 (0.025)

Fifth-grade cohort of 2002-03


0.099 (0.030)

0.142 (0.045)

0.069 (0.021)

0.076 (0.028)

Standardized test performance in 2002-03 – controlling instead for third grade test scores

0.098 (0.037)

0.136 (0.048)

0.071 (0.027)

0.075 (0.035)


Student Performance



reading High-stakes

math Low-stakes

reading Low-stakes

math

Fifth-grade cohort of 2002-03


0.058 (0.029)

0.104 (0.046)

0.044 (0.022)

0.047 (0.026)


0.079 (0.029)

0.096 (0.039)

0.058 (0.024)

0.054 (0.026)

-.5

0.5

200 300 400 500 600points_in_2002

Test scores of fifth graders in 2002-03math: quintic in summer 2002 points

-.4

-.2

0.2

.4

200 300 400 500 600points_in_2002

Test scores of fifth graders in 2002-03reading: quintic in summer 2002 points


Student Performance

Standardized test score Model specification High-stakes

reading High-stakes

math Low-stakes

reading Low-stakes

math

Performance of 2002-03 fifth-grade cohort in three years following policy change (as compared with the preceding cohort)

Fifth grade (or equivalent) performance

0.140 (0.036)

0.212 (0.047)

0.074 (0.026)

0.122 (0.029)

Sixth grade (or equivalent) performance

0.051 (0.026)

0.119 (0.039)

0.053 (0.022)

0.101 (0.024)

Seventh grade (or equivalent) performance

0.088 (0.028)

0.118 (0.034)

0.081 (0.021)

0.078 (0.024)


Subsequent Student Performance: Alternative specifications



reading High-stakes

math Low-stakes

reading Low-stakes

math

Performance of 2002-03 fifth-grade cohort in third year (seventh grade or equivalent) following policy change (as compared with the preceding cohort): Alternative specifications

Comparing “F” schools to “D” schools in summer 2002 only

0.079 (0.045)

0.150 (0.064)

0.151 (0.037)

0.135 (0.039)

Restricting sample to those predicted to earn “D” grade under old system

0.120 (0.034)

0.139 (0.044)

0.117 (0.035)

0.127 (0.037)

Falsification test: Successive cohorts in fourth grade

-0.022 (0.029)

n/a 0.028

(0.029) 0.018

(0.028)

Table 7 (part 1): Seemingly-Unrelated Regression and OLS Results of the Effect of Receiving an F Grade in

Summer 2002 on School Policy in 2003-04 Seemingly-Unrelated Regression Index F vs. All F vs. D Domain (1) (2) (3) (4) (5) Policies to Improve Low-

Performing Students 0.438

(0.097) 0.354

(0.123) 0.339

(0.143) 0.347

(0.143)

0.207 (0.163)

Lengthening Instructional Time 0.278 (0.095)

0.205 (0.105)

0.283 (0.125)

0.285 (0.125)

0.249

(0.136) Reduced Class Size for Subject 0.559

(0.191) 0.535

(0.215) 0.329

(0.244) 0.335

(0.243)

0.082 (0.253)

Minimum Time Required for Tested Subject Instruction

0.213 (0.154)

0.058 (0.167)

0.027 (0.177)

0.028 (0.177)

0.238

(0.237) Minimum Time Required for Non-

Tested Subject Instruction -0.028

(0.185) -0.287 (0.223)

-0.290 (0.259)

-0.273 (0.260)

-0.217 (0.253)

Scheduling Systems 0.208 (0.120)

0.349 (0.116)

0.349 (0.139)

0.355 (0.139)

0.260

(0.139) School Resources 0.034

(0.240) -0.131 (0.354)

-0.193 (0.512)

-0.182 (0.497)

0.254

(0.374) Specifications also control for: 2002 “Simulated Grade” N Y Y Y Y Lagged dependent variable N Y Y Y Y Other school characteristics N Y Y Y Y 2002 grade points N N Y Y Y

Table 7 (part 2): Seemingly-Unrelated Regression and OLS Results of the Effect of Receiving an F Grade in

Summer 2002 on School Policy in 2003-04 Seemingly-Unrelated Regression Index F vs. All F vs. D Domain (1) (2) (3) (4) (5) Policies to Improve Low-

Performing Teachers 0.216

(0.081) 0.123

(0.086) 0.128

(0.093) 0.133

(0.094)

0.192 (0.151)

Teacher Resources 0.728 (0.391)

0.866 (0.453)

1.042 (0.525)

1.021 (0.530)

0.711

(0.651) Teacher Incentives 0.221

(0.161) 0.210

(0.171) 0.102

(0.203) 0.107

(0.203)

-0.042 (0.196)

Teacher Autonomy -0.060 (0.129)

-0.193 (0.141)

-0.083 (0.158)

-0.090 (0.157)

-0.153

(0.193) District Control -0.167

(0.172) -0.029

(0.179) 0.119

(0.209) 0.127

(0.208)

0.083 (0.197)

Principal Control 0.168 (0.129)

-0.067 (0.140)

-0.267 (0.161)

-0.264 (0.161)

-0.383 (0.200)

School Climate -0.455 (0.125)

-0.028 (0.136)

0.108 (0.157)

0.104 (0.156)

0.073

(0.163) Specifications also control for: 2002 “Simulated Grade” N Y Y Y Y Lagged dependent variable N Y Y Y Y Other school characteristics N Y Y Y Y 2002 grade points N N Y Y Y

Table 8 (part 1): OLS Results of the Impact of Receiving an F Grade in Summer 2002 on School

Selected Individual Policies in 2003-04

F vs. All F vs. D

No Covariates With Covariates Variable (1) (2) (3)

Average 1st and 4th grade class size -2.934 (0.733)

-0.501 (0.912)

-0.488 (0.913)

Require tutoring for low-performing students

0.303 (0.095)

0.204

(0.123)

0.200 (0.123)

Require Saturday classes for low-performing students

0.228 (0.055)

0.111

(0.067)

0.115 (0.067)

Use block scheduling 0.250

(0.097)

0.227 (0.115)

0.227

(0.115)

Use common prep period 0.010

(0.054)

0.110 (0.070)

0.113

(0.070)

Use other scheduling structure 0.067

(0.077)

0.179 (0.105)

0.182

(0.105)

Table 8 (part 2): OLS Results of the Impact of Receiving an F Grade in Summer 2002 on School

Selected Individual Policies in 2003-04

F vs. All F vs. D No Covariates With Covariates Variable (1) (2) (3)

Minutes per week for collaborative planning/class preparation

21.829 (45.396)

117.834 (61.733)

116.716 (61.782)

District control of budget spending -0.181 (0.208)

0.418

(0.267)

0.432 (0.267)

Reduced class size gifted academic performance

-0.245 (0.099)

-0.051 (0.127)

-0.051 (0.127)

Whole school reform model -0.056 (0.085)

-0.025 (0.110)

-0.026 (0.110)

Minimum time spent on science 0.102

(0.097)

-0.094 (0.120)

-0.085 (0.120)

Table 9 (part 1): The Effect of Including School Policy/Practice Variables on Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student

Performance: Fifth-Grade Cohort of 2002-03 Standardized test score


reading High-stakes

math Low-stakes

reading Low-stakes

math Fifth grade (or equivalent) performance Models excluding school policy/practice variables

0.100

(0.029) 0.135

(0.045) 0.067

(0.021) 0.073

(0.028)

Models including school policy/practice variables

0.090

(0.036) 0.102

(0.056) 0.056

(0.031) 0.018

(0.032)

p-value of joint significance of policy variables requiring resources

0.004 0.117 0.010 0.198

p-value of joint significance of policy variables aimed at greater efficiency

0.001 0.002 0.002 0.000

Sixth grade (or equivalent) performance Models excluding school policy/practice variables

0.059

(0.029) 0.102

(0.045) 0.047

(0.022) 0.048

(0.027) Models including school policy/practice variables

0.048

(0.036) 0.055

(0.060) 0.038

(0.028) 0.022

(0.033)

p-value of joint significance of policy variables requiring resources

0.000 0.001 0.019 0.011

p-value of joint significance of policy variables aimed at greater efficiency

0.004 0.000 0.006 0.029

Table 9 (part 2): The Effect of Including School Policy/Practice Variables on Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student

Performance: Fifth-Grade Cohort of 2002-03



reading High-stakes

math Low-stakes

reading Low-stakes

math Seventh grade (or equivalent) performance

Models excluding school policy/practice variables

0.079

(0.030) 0.094

(0.039) 0.059

(0.025) 0.045

(0.027)

Models including school policy/practice variables

0.070

(0.037) 0.068

(0.043) 0.039

(0.027) 0.016

(0.029) p-value of joint significance of policy variables requiring resources 0.018 0.032 0.008 0.076

p-value of joint significance of policy variables aimed at greater efficiency 0.005 0.001 0.047 0.011

In sum…

• We estimate effect sizes in reading test scores among “F-graded” schools between 6-10σ and effect sizes in math between 7-14σ.

• We also find that “F-graded” schools appear to focus on low-performing students, lengthen the amount of time devoted to instruction, adopt different ways of organizing the instructional environment of students and teachers, increase resources available to teachers, and decrease principal control.

• These policies may explain at least 10% of the gains in reading and at least 25% of the gains in math.

Caveats….

• F-graded schools receive additional state assistance (e.g., assessment and course materials, increased professional development for teachers);

• While our results suggest that schools respond to accountability in potentially educationally meaningful ways, we do not observe student performance along all dimensions.

Special Data Opportunities in Florida

Documents

Transcript of Special Data Opportunities in Florida