Upward Bound Evaluation Flaws & Strong Positive Reanalysis Results

download Upward Bound Evaluation Flaws & Strong Positive Reanalysis Results

of 13

  • date post

    31-Aug-2014
  • Category

    Education

  • view

    1.698
  • download

    0

Embed Size (px)

description

Presentation on the Hill by Department of ED's Technical Monitors details flaws in published Mathematica reports and presents strong positive impact estimates from the study in more credible standards based re-analysis correcting for identified report flaws. Contrary to the conclusions put forth by Mathematica for almost a decade the re-analysis found strong positive impacts for Upward Bound.

Transcript of Upward Bound Evaluation Flaws & Strong Positive Reanalysis Results

  • April 19 2012 Briefing 210 Cannon House Office Building David Goodwin, Ph.D. Technical Monitor, First UB Evaluation Contract; Former Division Director Policy Analysis Studies (PAS); US Department of Education; Retired Gates Foundation, currently Independent Consultant Margaret Cahalan, Ph.D. Technical Monitor, Final UB Evaluation Contract; Currently Senior Scientist, PellInstitute for the Study of Opportunity in Higher Education
  • Topics/Purposes of Presentation1. Basic problem2. Re-analysis results3. Why is this important?
  • Extreme unequal weighting and serious representation issues Project with 26 percent of Figure 1. Percentage of sum of the weights by project of the 67 projects making up thestudy sample: National Evaluation of Upward Bound, study conducted 1992-93-2003-04 weight (known as 69) was sole representative of 4- 30 26.38 year public strata, but was 25 a former 2-year school 20 with largely less than 2- 15 Percent of weight year programs 10 5 Project partnered with job 0 training program 1 3 6 8 0 2 4 7 9 2 4 6 3 4 9 1 3 5 7 9 1 8 0 5 7 9 1 4 6 8 0 2 7 P1 P1 P1 P1 P2 P2 P2 P2 P2 P3 P3 P3 P3 P4 P4 P6 P6 P7 P7 P7 P7 P7 P8 P4 P4 P4 P5 P5 P5 P5 P6 P6 P6NOTE: Of the 67 projects making up the UB sample just over half (54 percent) have less than 1 percent of the weights each and one Inadequate representationproject (69) accounts for 26.4 percent of the weights.SOURCE: Data tabulated December 2007 using: National Evaluation of Upward Bound data files, study sponsored by the Policy andPlanning Studies Services (PPSS), of the Office of Planning, Evaluation and Policy Development (OPEPD), US Department of Education,:study conducted 1992-93-2003-04. of 4-year
  • Severe non-equivalency in project 69 in favor of control groupexplains observed negative results from project 69 Project 69 Other 66 projects in sample 100 100 Control, 20 Control, 23 90 90 80 Control, 49 Control, 49 Control, 51 80 70 70 60 Control, 79 60 50 50 40 Treatment, 80 Treatment, 77 30 Treatment, 51 Treatment, 51 Treatment, 49 40 20 30 10 20 0 Treatment, 21 High academic In 9th (younger) Expect advanced 10 risk grade in 1993-94 degree 0 Treatment Control High academic In 9th (younger) Expect advanced risk grade in 1993-94 degree Treatment Control The Pell Institute 4
  • 10090 Control, 42 Control, 4480 Control, 5870605040 Treatment, 58 Treatment, 5630 Treatment, 422010 0 High academic In 9th (younger) Expect advanced risk grade in 1993-94 degree Treatment Control The Pell Institute 5
  • Re-analyses corrected for identified issues Used similar statistical analysis procedures but unlike published impact estimates the re-analyses: 1. Presented results with and without project 69 2. Standardized outcomes to expected high school graduation year for sample that spanned 5 years of high school graduation dates 3. Used all applicable follow-up surveys (3 to 5) and 10 years of federal aid files for source of data 4. Used National Student Clearinghouse (NSC) data only for BA degree and not for enrollment or 2-year or less degrees because coverage too low or non-existent in applicable period
  • Figure 3. Treatment on the Treated (TOT) and Intent to Treat (ITT) estimates of impact of Upward Bound (UB) on postsecondary entrance within +1 year (18 months) of expected high school graduation year (EHSGY) 1992-93 to 2003-04 Not UB participant (control) UB participant (treatment) Difference 14.2**** TOT (excludes project 69) 60.4 74.6 64.3 Difference ITT (excludes project 69) 73.3 9.0*** Difference TOT (includes project 69) 62.5 11.0**** 73.5 ITT (includes project 69 ) 66 Difference 72.9 6.9**** 0 20 40 60 80*/**/***/**** Significant at 0.10/0.05/. 01/00 level. NOTE. Model based estimates based on STATA logisticand instrumental variables regression and also taking into account the complex sample design. Based onresponses to three follow-up surveys and federal student aid files. SOURCE: Data tabulated January 2008using: National Evaluation of Upward Bound data files, study sponsored by the Policy and Program StudiesServices (PPSS), US Department of Education: study conducted 1992-93 to 2003-04; and federal StudentFinancial Aid (SFA) files 1994-95 to 2003-04. (Excerpted from the Cahalan Re-Analysis Report, Figure IV)
  • Figure 4. Impact of Upward Bound (UB) on Bachelors (BA) degree attainment: estimates based on 66 of 67 projects in UB sample: National Evaluation of Upward Bound, study conducted 1992-93 to 2003-04 TOT (Longitudinal file BA in +8 years of EHSGY- evidence from 14.6 any Followup Survey (Third to Fifith) or NSC; no evidence set 21.7 to 0)**** TOT(BA by end of the survey period, Fifth Follow-Up 21.1 Control responders only-adjusted for 28.7 Treatment non-response)**** ITT (Longitudinal file BA in +8 years of EHSGY- evidence from 13.7 any F