A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009...

download A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 by M Cahalan and S Stout

of 42

  • date post

  • Category


  • view

  • download


Embed Size (px)


This AEA presentation gives results from systematic review of all published reports from 2002 to 2009

Transcript of A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009...

  • A 8Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 Prepared by: Margaret Cahalan Sharon Stout Prepared for: AEA Meeting November 13 2009 Draft not for Quotation or Citation: All views and data presented are solely those of the authors and do not reflect any review or authorization from the US Department of Education
  • A Review of IES/NCEE & OPEPD/PPSS Research Reports
    • Part 1: Overview---All reports listed on web sites published between 2002 and 2009--Looked at level, methods, topics, findings
    • Part 2: Case Study Example of difficulties in using evaluation research for PART, budget and program reform ----college access and persistence
    • Part 3: Lessons Learned--New Directions/Models for Evaluation
  • Part I: Overview of Studies of Two ED-Offices
    • 2002-2009Period of transition between offices
      • OPEPD/ PPSS --Office Planning Evaluation Policy Development/Policy and Planning Studies Services ( known as PPSS in this presentation )
        • Earlier office responsible for most evaluations prior to 2002
        • Reports to --OPEPD Assistant Secretary
        • Informal agreement would do primarily Implementation evaluation studies after 2002
      • IES /NCEE---Institute for Education Sciences/ National Center Education Evaluation (known as IES in this presentation)
        • Created in re-organization in 2002 former OERI
        • Independent with Commissioner confirmed by congress
        • Legislative mandate for evaluations--scientific approachmedical model-- experimental designrandom assignment
  • Published Reports from IESNCEE and OPEPD/PPSS: 2002-2009 117 ---Evaluation Reports IES/NCEE: Primarily experimental design 27 reports (2003 to 2009) OPEPD/PPSS: Primarily Implementation Studies 90 reports (2002-2009) 15 Studies (study with multiple reports counted once) 50 Studies (study with multiple reports counted once)
  • IES/NCEE Studies: 2003-2009 27 reports of 15 studies (Largely Practice Based)
    • Teacher Induction
    • Supplemental Reading Comprehension Interventions-
    • DC Opportunity Scholarship Program
    • Student Mentoring Program
    • Reading and Mathematics Software Products
    • Teachers Trained Through Different Routes to Certification
    • Professional Development Strategies to Improve Early Reading
    • Reading First Impact Study Comprehension
    • Classroom Literacy Interventions and Outcomes in Even Start (Started PPSS)
    • Enhanced Reading Opportunities Study
    • Four Early Elementary School Math Curricula
    • Enhanced Academic Instruction in After-School Programs48 more hours
    • Striving Readers Study Closing the Reading Gap
    • Early Reading First
    • National Evaluation of 21st Century Community Learning Centers (started by PPSS)
  • PPSS50 studies (Largely Program Based)
    • Career and Technical Education (CTE)-3
    • Federal college access and retention programs TRIO and GEAR UP12
    • Title 1 (general, SES, Choice, funding, accountability, AYP & improvement)9
    • Single sex schools 1
    • Private schools under Title 1-1
    • Literacy through Libraries Program-1
    • Reading First-2
    • Teacher Quality-4
    • Comprehensive School Reform2
    • Class Size Reduction-1
    • Voluntary school choice-1
    • Small Learning Communities-1
    • Faith Based-1
    • Technology & Data System Use5
    • Charter Schools-2
    • Sex Misconduct-1
    • Magnet Program-1
    • Migrant ProgramTitle 1
    • Even Start Program-1
  • Distribution of Studies By School Level
    • IES/NCEE 15 Studies
    • OPEPD/PPSS 50 Studies
  • Topic Summary: IES/NCEE
  • Topics for OPEPD/PPSS Elementary-Secondary Studies
  • Topics/Programs of Studies with Reports Published in Secondary-Postsecondary by PPSS: 2002-2009
  • Major Method IES/NCEE: 15 Studies
  • OPEPD/PPSS Major Methods: 50 Studies
  • Summary of Effects IES/NCEE: 15 Studies
  • Effects Summary 23 PPSS Studies that report comparative effects or outcomes
  • 15 IES/NCEE Studies Topics Effects Summary Reading 6 studies (one 9 th grade positive; 3 small positive on decoding not comprehension; 2 not detected; findings that practices were different but not outcomes; the one study with positive involved more time on reading9 th grade study) DC Voucher 1 study --First two years no effects; third year began to see some effects for reading not for math; 26 percent did not take opportunity Mentoring 1 study-- not overall positive effects--some positive for some subgroups60 percent of controls also had mentoring; Reading and Math Software 1 study ( not detected positive overall ) TeacherReading PD, Induction, Alternative Certification 3 studies (not detected overall on student outcomes; some on teacher practice for reading) 4 Math Curricula 1-study 2 curriculums had positive effectsrelative to each other Afterschool 2 studies1--not effects on academics or behavior; second studypositive on math not reading48 hours more instruction
  • Observations 15 IES/NCEE Studies
    • Topics
      • Lack of direct link to federal program but sought more understanding of interventionmost funded by program fundsTitle1and Title 2 of ESEA/NCLB
    • Effects
      • Few overall effects especially on students when counterfactual was existing practice
      • Observe some effects on teacher practices associated with interventionnot translate into changes for students test scores
      • Some effects seen when increase time spent on subject
      • Effects sometimes emerge over time when the intervention is sustained
    • Issues
      • Counterfactual why would not assume would use science based practices
      • Contamination of control group; spill over issues
      • Intent to treat drop outs
      • Implementing intervention issues
      • Lack of external validity time and space----does the same circumstances ever happen again? Moving targets for curriculums and technology
  • K-12 PPSS Studies Topics Effects Summary Title 1-general & Funding Proficiency increased; however most states will not meet goals by 2013-14; Over 1100 schools in need of improvement by 2006 and most were in advanced stages of corrective action; did not show changes in time on subjects; poorest schools get less per capita; federal more targeted than state and local Teacher Over 90 percent of teachers met highly qualified; however those in high poverty have less experience and less likely to have degree in subject teaching and less professional development than less poverty schools; Supplemental ED Services (SES) Increase in number of schools eligible; however participation rate is stable--about 17 percent taking advantage ; Those who participated were more likely to show growth than eligible non-participants in sub-study of districtsselection effects issues; Waiver studiesmixed resultshigher proportion eligible participating when district servicein two large districts do better in two small did not do better Choice/Voluntary School Choice About 1 percent of students participating; Mixed resultssome show no difference in outcomes when compared with those not participating; one that students did better. Most districts kept choice within district and most did not go to high performing schools when transferred out of low performing schools Reading First More time on reading and adopted practices; limited evidence 3rd and 4th graders increased reading more than those not in programs; regression discontinuity
  • K-12 PPSS Studies (continued) Topics Effects Summary Charter Facilities study found loans were given to schools that served more needy population than other charter schools in areas; most would not have qualified for private loans-too high risk Descriptive study on characteristics focus on accountability Comprehensive School Reform No effects for those who got grants; evidence th