The Implementation and Effectiveness of Supplemental Educational Services

8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services

1/28

The Implementation and Efectiveneo Supplemental Educational Service

A Review and Recommendations or Program Improvement

Carolyn J. Heinrich University of Texas, Austin

Patricia Burch University of Southern California

March 2012

www.americanprogress.org www.aei.o

American Enterprise Institutefor Public Policy Research

Tightening Up Title I


2/28


3/28

1 Introduction and summary

4 Participation in SES and its impacts

6 Evidence on SES eects

10 Whats happening during an hour o SES?

14 Recommendations or policy and program change in SE

17 Conclusion

18 Reerences

21 Endnotes

22 About the authors and acknowledgements

Contents


4/28


5/28

Introduction and summary | American Enterprise Instit

Introduction and summary

Te 1965 Elemenary and Secondary Educaion Ac, or ESEA, creaed ile I ed-

eral unding o help provide all sudens wih an equal opporuniy o receive he

highes-qualiy educaion possible. Providing uoring or sruggling sudens is

one o many possible uses o ile I unds. Oher uses include eacher proessional

developmen, compuer labs, insrucional maerials, eacher assisans, and more.

ESEA was renamed he No Child Le Behind Ac, or NCLB, when i was reau-

horized in 2002. NCLB aimed o close he achievemen gap in public educaion.I requires public schools ha have no made adequae yearly progress on es

scores or a leas wo consecuive years o oer parens o children in low-income

amilies he opporuniy o receive exra academic assisance, oherwise known

as supplemenal educaional services, or SES. SES consiss primarily o uoring

oered ouside he regular school-day hours. Consisen wih he inen o he law

o promoe accounabiliy, exibiliy, and choice, SES is implemened a he local

level and draws largely on he privae secor o oer eligible sudens a range o

choices or ree uoring ouside o regular school hours.

No new ederal monies were allocaed o suppor he delivery or managemen

o SES. Te law lays ou crieria and guidelines or sae and local educaional

agencies in approving SES providers, arranging or heir services, and managing

conracs and nancial sysems. School disrics wih eligible schools are obligaed

o se aside 20 percen o heir ile I unding or SES and o measure provider

eeciveness in increasing suden achievemen.1

In arranging or SES, sae and local educaional agencies are able o draw on a

airly well-esablished marke o aer-school uoring programs. As SES expanded

uoring opporuniies or low-income sudens a subsanial number o diverseorganizaions enered he marke o compee or available SES unds. Tey

adverise widely varying hourly raes, uor qualicaions, uoring session lengh,

insrucional sraegies, and curriculums.2


6/28

2 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services

Te ux in he SES vendor marke is considerable. Many smaller organizaions

ener and leave aer atracing ew sudens, while ohers have rapidly expanded

heir share o sudens served. Some school disrics have also operaed heir own

SES programs, hough as his is condiional on he disric making adequae yearly

progress, disric roles as providers also come and go. Te subsanial year-o-year

ucuaions complicae sae and local educaional agency eors o comply wihNCLB requiremens in ideniying organizaions ha provide services consisen

wih sae and local insrucional programs and wihdrawing approval rom pro-

viders ha ail o increase suden academic achievemen or wo years.3

In heory, parens and sudens should be holding SES vendors accounable

hrough heir choices o providers. Tey osensibly use inormaion disribued

by school disrics and SES providers o ideniy he bes provider o mee he

childrens needs. Sudens who become aware o heir eligibiliy may choose

o regiser or SES wih a specic SES provider, and SES providers invoice he

school disric or he number o hours SES sudens atend, up o a maximumper-suden dollar allocaion.

Te more eecive providers would increase heir share o sudens atending

SES over ime i he program worked as he law inended. Te service agreemen

beween a disric and is SES providers is, eecively, a cos-reimbursemen con-

rac, however, wih no perormance coningencies. In addiion, only sae educa-

ional agencies, no disrics, have auhoriy o approve SES providers and esablish

program crieria, such as an accepable suden/uor raio or providers o mee.

Te ac ha SES akes place ouside o he regular school classroom and ha

insrucional pracices are known o vary signicanlyno only beween pro-

viders, bu also wihin he same provider depending on he seting and specic

uorurher challenges sae and local agency eors o acquire knowledge o

SES conen and eeciveness.

Paricia Burch and Annalee Good poin ou ha he eaures o SES ha are key

o is eecivenessaciviies and resources used in insrucion, he naure o

ineracions beween sudens and providers, and insiuional and srucural

elemens ha inuence uoring pracicesare among he leas visible o saesand school disrics. 4 Ta said, aer-school sudy and uoring programs have

long been in operaion, including ederally unded programs, and here is a large

body o research on heir implemenaion and eecs, include sudies specically

ocused on SES.


7/28

Introduction and summary | American Enterprise Instit

In his paper, we review sudies on he eecs o SES on suden achievemen and

updae he evidence on wha makes SES eecive or why i ails. We do his by

drawing on expanding disric evaluaion eors and he published lieraure in

his area as well as our own recen and ongoing mulisie, mulimehod sudies o

he implemenaion and eeciveness o SES.

In paricular, we address hese key quesions:

Who atends SES and or how many hours?Wha are he esimaed eecs o SES (rom our sudy and ohers), and how do

hey compare o hose o alernaive inervenions?Wha is happening in an invoiced hour o SES?Wha policy changes or levers migh improve SES?

We conclude wih recommendaions or program and policy change o make SES

more eecive in ligh o he expeced reauhorizaion o ESEA and ile I his year.

Our basic recommendaions include:

Sudens paricipaing in SES need o ge more hours o higher-qualiy and

appropriaely diereniaed insrucion or SES o be eecive. Saes and school disrics need o beter monior and conrol service qualiy

and delivery and ake a closer look a wha online providers are doing in an hour

o SES insrucion. SES resources should be direced primarily o sudens in lower grades, sudens

who are English language learners or who have disabiliies, and hose who are

mos severely underperorming in school. School disrics, SES providers and heir uors, and regular school-day eachers

and parens need o beter coordinae heir eors o increase he success o SES

in raising suden achievemen and closing achievemen gaps.


8/28


Participation in SES and its impacts

Since SES began school disrics have been under pressure o comply wih ederal

requiremens o make SES available o as many eligible sudens as unding allows

and o assess vendor eeciveness in increasing he achievemen o paricipaing

sudens. Some school disric accounabiliy and evaluaion unis atemp o mea-

sure program eeciveness and in some cases SES provider ecacy. Boh disric

sa and researchers , however, ace numerous challenges o properly evaluaing

suden- and vendor-level SES eecs.

One o he bigges challenges is ha who ges uored in SES programs and or

how long is inuenced by a variey o acors, including suden and amily char-

acerisics and program adminisraion.

Paricipaion in SES is volunary among sudens eligible or SES. NCLB requires

school disrics o use he same daa o deermine eligibiliy or SES ha hey use

or making wihin-disric ile I allocaionshisorically hey use inormaion

on ree school lunch eligibiliyand school disrics are required o noiy ami-

lies o heir childrens eligibiliy and he availabiliy o approved SES providers.

Disrics have o esablish addiional crieria o deermine which eligible sudens

ge access o services i more sudens are expeced o sign up or SES han here

are unds available o serve hem (preerably beore regisraion opens). Bu no

all sudens decide o ollow hrough in atending wih a chosen provider even

i hey are eligible and given he opporuniy o regiser or SES. And some sop

atending beore heir oal SES dollar allocaion is expended.

Dierences exis beween SES-eligible sudens who regiser and atend SES and

hose who do no or a number o reasons. Sudens more requenly absen dur-

ing he regular school day, or example, are more likely o orego an aer-schoolopion. Analyses rom our mulisie sudy show ha he percen o days absen

in he prior school year is one o he mos consisen, negaive predicors o boh

regisraion or SES and SES atendance across he sudy disrics. We also ound

ha one o he mos consisen, posiive predicors o SES regisraion and aten-

dance is wheher he SES-eligible suden atended SES in he prior school year.


9/28

Participation in SES and its impacts | American Enterprise Instit

As long as one can measure dierences such as hese in sudens who regiser/

atend and hose who do no, hey can be adjused in esimaing SES eecs. I

hose regisering and atending dier in ways ha are no observed or measured

and ha relae o suden achievemenin he level o encouragemen hey

receive rom regular school-day eachers, or examplegeneraing accurae esi-

maes o SES eecs may insead prove very dicul.

Our research also shows ha i is imporan o separaely model and accoun or

he muliple sages o SES selecionregisraion, atendance, and he number

o hours atendedbecause he inuence o suden characerisics diers across

hem. For insance, we nd across muliple sies and years ha whies, Hispanics,

and Asians are signicanlyless likely o regiser or or atend SES. Bu i hey

atend hey are signicanly more likely han Arican Americans o atend 40 or 60

hours. English language learners, or ELL sudens, are more likely o regiser and

atend more hours han non-ELL sudens.

Ideally we would like o know he added value o each addiional hour o SES

atended. Pu anoher way, wha number o hours o SES do sudens need o

atend beore we sar o see eecs on heir reading and mah achievemen?

o esimae his saisically, we need o have sucien numbers o sudens

atending a range o SES hours. In pracice, he number o hours sudens atend

SES is inuenced by acors such as he rae per hour charged by SES providers

and he dollars allocaed per suden by disrics or SES, as well as suden and

program characerisics.

One school disric in our mulisie sudy, or example, allocaed approximaely

$1,300 per suden or SES while over 70 percen o he paricipaing sudens

received SES rom a provider charging $75 or more per hour. Te maximum

hours o uoring a SES provider could oer a suden a his rae per hour and per

suden allocaion was abou 18 hours over he school year.

Across he disrics in our sudy, we observed he same peaks (or clumping) o

sudens in he number o hours hey atended SES, wih he larges peaks close

o 0 or 20 hours (depending on grade level), and smaller spikes around 40 and60 hours o SES atended. Te implicaion is ha we are able o more reliably

esimae SES eecs a hese common hresholds or levels o atendance where

here are sucien numbers o sudens observed (using rigorous mehods ha

eecively adjus or selecion).


10/28


Evidence on SES effects

Evaluaions o SES and/or provider-specic eecs are noably decien in many

saes and school disrics. Some saes and disrics rely only on inormaion

sel-repored by providers or rom relaively eeble daa-gahering eors such as

paren saisacion raes rom volunarily compleed surveys (wih very low and

selecive paricipaion).Oher disrics atemp o make he bes use o he daa

hey collec on suden SES atendance and provider invoices or operaing SES

o evaluae is eeciveness. Sill, only a handul o he larger disrics have he

in-house capabiliies o apply more rigorous approaches ha accoun or sudenselecion and oher esimaion problems.

In ac, some consisency in sudy ndings can be ound among he more

advanced disric evaluaion eors. Chicago Public Schools or CPS evaluaions

in 2003-04, 2004-05, 2006-07, and 2007-08 repored larger gains in reading and

mahemaics or sudens receiving a leas 40 hours o uoring and or sudens

in grades 4 hrough 8 who were notELL and who received at least30 hours o

SES uoring.5

Consisen wih oher CPS ndings, a Los Angeles Unied School Disric

sudy ound low SES paricipaion, and airly small program eecs even

among sudens wih he highes levels o SES atendance. Te eecs were

atribued primarily o improved perormance by elemenary sudens.6 In addi-

ion, sudies in Minneapolis and Milwaukee Public Schools, where average SES

hours atended are paricularly low, did no nd saisically signican, posiive

eecs o SES paricipaion.7

Mathew Springer and co-auhors poin ou ha very ew sudies rigorously adjus

or dierences in he characerisics o sudens who choose o paricipae in SES.8

Tey ideniy only our sudies besides heir own ha did so.9 I is imporan o

mach up sudens who look similar in heir characerisics in order o draw cred-

ible conclusions abou SES eeciveness based on comparisons o sudens who

paricipae in SES wih hose who orego he opporuniy o atend SES.


11/28

Evidence on SES eects | American Enterprise Instit

Ta said, an imporan insigh rom he broader lieraure on aer-school uor-

ing programswhich is consisen wih ha o SES evaluaions o daeis

ha reaching some minimal hreshold o uoring hours appears o be criical o

producing measurable eecs on sudens achievemen (as measured primarily

by es scores).

Paricia Lauer and co-auhors conduced a synhesis o research on ou-o-school-

ime programsspecically in response o NCLB requiremens o oer SES

and ound ha eec sizes were larger or programs ha were more han 45 hours

in duraion, hough hey became smaller or hose longes in duraion. 10

In our own ongoing research esimaing he eecs o SES we nd 40 hours o

uoring o be a criical hreshold. Below 40 hours we do no ideni y any saisi-

cally signican eecs o SES on sudens mah and reading gains (as measured

by changes in es scores). In addiion, we nd eecs on boh mah and reading

achievemen or elemenary sudens who receive a leas 40 hours o SES buonly gains in mah or middle school sudens. Sudies by Mathew Springer and

co-auhors and on Zimmer and co-auhors likewise ound more consisen, posi-

ive eecs o SES on sudens mahversus readinggains in heir sudies o

SES in large, urban school disrics.11

Anoher common nding in sudies o SES is ha younger childrenspecically,

elemenary school sudensare more likely o atend SES (aer regisering or a

program) and o atend more hours han middle school or high school sudens.12

Unorunaely, some sudies, including one ha explored he eecs o SES across

muliple school disrics (Balimore, Chicago, Long Beach, Los Angeles, Palm

Beach, Philadelphia, San Diego, and Washingon, D.C.), did no esimae SES

eecs by grade level (or elemenary, middle, or high school).13

Sill, hese sudies did explore he poenial cumulative eecs o SES or sudens

who atended more han one year. Tey ound subsanially larger eecs on su-

den mah and reading es score gains associaed wih atending more han one

year o SES. Tis suggess ha more hours o SES are needed o produce larger

program eecs even i hey occur over more han one school year.

Tis evidence, along wih oher corroboraing ndings in he sudies discussed

here, poin o he essenial role o he number o SES hours received in generaing

eecs on sudens mah and reading gains.


12/28


As indicaed above, an imporan challenge in geting more hours o SES o su-

dens is he hourly rae SES providers charge. Tis, in combinaion wih disric

(per-suden) maximum allocaions o SES dollars, limis he oal number o

hours o uoring sudens can receive. Sae and local educaional agencies do no

have auhoriy o proscribe or conrol he hourly raes charged by SES providers

(oher han disric-operaed ones), hough hey may speciy a range.

One would expec ha providers charging higher hourly raes would be delivering

higher-qualiy uoring services. In our curren and prior research, however, we

see litle correlaion beween provider characerisicssuch as suden-eacher

raios, oal hours oered, suden atendance, curriculum design, and ohers

and hourly raes charged oher han wheher a provider is online. 14

Furher, in our ongoing sudyincluding approximaely 300 providerswe nd

ha even when conrolling or suden selecion ino online vendors and he num-

ber o hours o SES sudens atend, sudens atending wih online SES providersare less likely o gain in mah and reading (relaive o ofine providers). Tis

nding is disconcering given ha in our sample, online vendors charged signi-

canly more or heir services ($24 per hour more) han oher providers.

We now look a eec sizes because hey are sandardized and can hereore be

compared across sudies. An eec size is he changemeasured in sandard

deviaionsin an average sudens oucome ha can be expeced i he suden

paricipaes in SES. A sandard deviaion ells you how dieren (or ar away) a

value is rom a given average (or a given sample). So a larger sandard deviaion

implies a bigger eec.

While here are some dierences in esimaed SES-eec sizesor mah and read-

ing and across sudies wih dieren samples, reamen measures, and approaches

o esimaing eecshere is also clearly some congruence in ndings.

Te average increase in mah es score gains o 0.09 sandard deviaions repored

by on Zimmer and co-auhors or sudens atending any SES is approximaely

he same ha we nd or middle school sudens who atend a leas 40 hours o

SES in our mulisie sudy.15

Te sudy by Mathew Springer and co-auhors alsonds increases in es score gains o abou 0.09 sandard deviaions in mahema-

ics (and 0.076 sandard deviaions in reading).16 In an alernaive specicaion

ha accouned or hose who regisered bu did no atend SES, however, hey

did no nd saisically signican eecs on reading or sudens atending SES

(comparable o our mulisie sudy resuls).


13/28

Evidence on SES eects | American Enterprise Instit

Our mulisie sudy separaely esimaes SES eecs or elemenary school

sudens. We nd comparably sized eecs or mah and reading (eec sizes o

approximaely 0.06 sandard deviaions, range: 0.054-0.076) ha are jus slighly

smaller han hose or middle school sudens.

Summaries o he eeciveness o similar educaional inervenions ha are ar-geed o elemenary and middle school sudens and inend o supplemen regular

school-day insrucion sugges ha hese eec sizes or SES are small. Based

on eec sizes o similarsupplemenal educaional inervenions compiled by

Carolyn Hill and co-auhors rom randomized sudies and a mea-analysis o oher

mea-analyses, an esimaed (one-year) eec size o 0.06 sandard deviaions is

one-ourh o one-h he size o mean eec size esimaes rom educaional iner-

venions argeed oward elemenary school sudens (0.23 o 0.33).17 Mean-eec

sizes rom randomized sudies o inervenions or middle school sudens, ranging

rom 0.27 o 0.51, imply he average SES-eec sizes range rom one-enh o one-

hird he size o eecs o oher similar educaional inervenions or hese sudens.

A he same ime, he ndings on SES eecs are very consisen wih hose

repored or inervenions more generally under NCLB ha rely on sandardized

achievemen ess o hold disrics and saes accounable or resuls. A Naional

Academy o Sciences sudy concluded ha school-level, es-based programs and

incenives under NCLB have eec sizes, on average, o approximaely 0.04 o 0.08

sandard deviaions, wih measurable eecs o dae ha are concenraed in ele-

menary grade mahemaics and are small in relaion o expeced improvemens. 18

Finally, o pu hese eec sizes ino perspecive wih regular school achievemens,

we esimae ha on average, not including administrative costs, school disrics

spend an amoun equal o abou 15 percen o heir average annual per-pupil coss

on SES (or hours o SES invoiced by providers). Using 0.06 sandard deviaions

as he average gain rom SES paricipaion, his is equal o abou 11 percen o 16

percen o he average annual gains in mah (0.54) and reading (0.38) by hird

and h graders on naionally-normed ess.

Wha his means is ha or elemenary sudensor whom sudies are mos likely

o nd saisically signican impacs o SESSES is, a bes, nearly as cos eec-ive in producing suden achievemen gains as spending on regular school-day

aciviies. In oher words, he gains rom paricipaion in SES relaive o average or

regular school-day gains in mah and reading are, a beswihou ull accouning

o cossapproximaely equal o or slighly less han hose achieved wih he same

racion o spending by schools disrics on regular school-day aciviies.


14/28


Whats happening during an

hour of SES?

NCLBs inenion was o aciliae as ree a choice as possible or sudens and

parens selecing SES providers and program ypes. Under he law, school disrics

canno impose requiremens on uors, and he only auhoriy hey have in ermi-

naing a providers conrac occurs when he provider violaes disric policies or

oher such erms o a conrac.

Disric sa who are responsible or he adminisraion o SES conend ha heir

hands are ied in monioring providers. Tey also poin ou ha SES uors dono have o mee highly qualied sandards or have specic raining. Furher,

hey allege ha sae educaional agencies are lax in evaluaing providers, se-

ing minimum sandards or uoring qualiy or requesing essenial inormaion

on applicaions or assessing and monioring qualiy, or ollowing hrough on

disric complains abou provider incompeence or misconduc. And wih very

ew resources or program adminisraion, le alone monioring and evaluaion,

disric sa are sreched o nd ime o observe SES providers and beter under-

sand wha is aking place in an hour o SES or which disrics are invoiced.

A disinguishing eaure o our mulisie, mulimehod sudy o he implemena-

ion and eeciveness o SES is an in-deph qualiaive componen designed o

dene key elemens o SES program models and o ideniy how policy and imple-

menaion poenially mediae or inuence SES impacs. Wha do we see happen-

ing in pracice (a he classroom level) in an invoiced hour o SES? How does his

vary across dieren SES provider setings, disrics, ormas, and approaches o

uoring? And how does i relae o program eeciveness?

Because he law inenionally oers SES providers wide-ranging exibiliy in he

design o heir programs, assessing program qualiy or deliy o evidence-basedpracices is a somewha elusive ask. Sill, considering he law and inormaion

specied in provider conracs wih disrics, some direcives or conen ocus,

locaion, and he use o research-based pracices are clear.


15/28

Whats happening during an hour o SES? | American Enterprise Institu

For one, SES providers should make reading and mahemaics he conen ocus

o insrucion, and insrucion has o be provided ouside o he regular school day.

Providers are no required o oer services o sudens wih disabiliies, or SWD,

or ELL sudens. Bu i providers oer hese services he law requires hem o be

adverised, and disrics are responsible or providing hese services i no provider

is able o or agrees o do so.

In addiion, he law saes ha SES uoring should be high-qualiy, research-

based, and specically designed o increase suden academic achievemen

[Secion 1116(e)(12)(C)]. In our sudy, we inerpre research-based pracices

as bes pracices idenied as making a measurable impac on suden achieve-

men by ou-o-school-ime or OS lieraure or conen area specialiss.

Best practices for out-of-school time tutoring and their use in SES

Litle research exiss on bes pracices specic o SES. Even so, prior research on

OS programs ells us ha high-qualiy programs are characerized by:

Consisen and susained insrucional ime Small grouping paterns (no larger han a suden-o-eacher raio o 10:1, bu

smaller is beter) Curriculum ha is conen rich, diereniaed o suden needs, and conneced

o sudens regular school-day learning Insrucion (or conen delivery) ha is varied (srucured and unsrucured,

independen and collecive), acive (no desk ime or workshees), ocused on

skills developmen, sequenced o achieve skill developmen objecives, and

explici in is argeing o specic skills Posiive relaionships beween uors, sudens, and peers eachers/uors wih boh conen and pedagogical knowledge and coninuous

suppor, as well as consrucive evaluaion, rom heir adminisraors 19

o ideniy hese bes pracices in SES sessions Burch and colleagues designed

a sandardized observaion insrumen o sysemaically collec inormaion on

eaching mehods and insrucional maerials in use and o ideniy he impaco dieren ormas, resources (curriculum maerials, sang), and insrucional

mehods on sudens observed levels o engagemen.20


16/28


o dae, Burch and colleagues have observed 99 uoring sessions across a range o

providers in ve, large urban school disrics in our saes. Te providers included

online, in-home, in-school, and communiy-based uors; or-pro, no-or-pro,

disric-provided, and aih-based organizaions; providers wih large marke

share (in erms o sudens served), wo or more years o SES provision, and

wih higher-han-average levels o suden atendance; and providers adverisingservices o SWD and ELL populaions.

In general, he model o uoring observed ended o ake he orm o radiional

academic learning environmens, wih sudens being uored in esed subjecs

mahemaics and readingand ypically insruced in a whole group orma wih

more han one suden and one ocal aciviy.

In oher words, raher han providing somehing innovaive, acive, and very di-

eren rom he regular school day, SES was based on radiional orms o eacher-

direced insrucion.21 For insance, uoring sessions o wo or more sudens weresrucured ino a single aciviy wih no op-ou aciviy in abou wo-hirds o

sessions observed. esearch on OS argues, wihou qualicaion, or diereni-

aed programming ha responds o sudens dieren learning syles or needs.

In addiion, sudens atending SES who migh learn bes via projec-based learning,

ars inegraion, or links o communiy-based aciviies encounered ew oppor-

uniies o his sor in our mulisie sudy observaions. Perhaps mos roubling,

however, very ew uors wih raining or experience in ELL or SWD insrucion

were presen during uoring, and wih very ew excepions, neiher curriculum nor

insrucion was ailored in any way o he unique needs o hese sudens.

On a more encouraging noe, uors were observed engaging wih sudens in

a predominanly posiive way across disrics and ormas. SES consisenly

occurred in small groups wih uoring sessions raing highly on indicaors o bes

pracices such as provide consrucive criicism; encourage paricipaion rom

disengaged sudens; and lisen acively and atenively o sudens.

Te observaion insrumen developed by Burch and colleagues also allowed

or assessmen o how consisen uors were in heir uoring pracices acrossan enire session and how much instructional ime sudens were receiving.

Irrespecive o he orma, sudens received less insrucional ime han wha was

adverised or invoiced by providers, hough he magniude o hese dierences


17/28

Whats happening during an hour o SES? | American Enterprise Institu

varied by orma. In more han hal o all observaions wih wo or more su-

densprimarily o-line, school-based setingssudens ha sared a session

were observed missing par o he session or leaving he uoring session alo-

geher, or sudens came in lae. We call his atendance ux.

We conrmed hrough inerviews wih uors and provider adminisraors haschool-based SES programs oen compee wih oher aer-school programs or

sudens ime, and classrooms wih muliple sudens required coordinaion and

se-up ha cu ino insrucional ime.

egardless, he quaniaive and qualiaive ndings o our mulisie sudy sugges

ha sudens are no geting enough hours o high-qualiy, diereniaed SES

insrucion o produce signican gains in heir learning. Tis is no a problem

ha will be resolved only by seting minimum hours sandards or SES providers

given ha invoiced hours do no equal insrucional ime.


18/28


Recommendations for policy

and program change in SES

Public suppor exiss, including among many parens, or coninuing an iner-

venion in an ESEA reauhorizaion ha provides ree, exra academic assisance

o sruggling sudens. Bu he bes available evidence o dae suggess ha SES

has been minimally eecive. I produces only small eecs or a relaively small

racion o sudens, primarily elemenary aged, who ge a sucien number o

hours o uoring.

Sudens paricipaing in SES need o ge more hours oinstructional ime, andhey need o receive higher-qualiy and appropriaely diereniaed insrucion.

We oer he ollowing recommendaions or improving he eeciveness o SES

ha address he program design, he responsibiliies o sae and local educa-

ional agencies in implemenaion, and he ron-line role o SES providers in

delivering services.

States and districts need more levers of control over SES providers

Te curren division o responsibiliies beween sae and local educaional

agencies or monioring and accounabiliy o SES providers is no working. Te

disrics conrac wih and pay SES vendors o provide services, bu hey have no

auhoriy o se sandards or uor or service qualiy or minimum hours o SES

per enrolled suden or hourly raes charged.

esearch shows a srong relaionship beween he inensiy o SEShe num-

ber o hours atendedand is eeciveness, and SES provider hourly raes

direcly inuence he number o hours o SES sudens can atend. Te ac ha

we observed widely varying hourly raes across and wihin providerswih oneprovider charging less han hal he rae per hour in one disric han i charged in

anohersuggess ha saes and disrics need o exer more conrol over hourly

raes wih he accepable ranges hey speciy.


19/28

Recommendations or policy and program change in SES | American Enterprise Institu

More generally, we recommend ha disrics be allowed and encouraged o

negoiae perormance-based conracs wih SES providers ha aciliae greaer

conrol over hourly raes and minimum SES hours provided. Tey also need more

say in uor qualicaions and curriculum (paricularly or serving ELL and SWD

populaions) and oher programmaic and nancial managemen acors.

Saes could similarly negoiae perormance-based conracs wih disric-

operaed providers. Some racion o ile I resources should be allocaed o

managing SES providers i perormance-based conracs are o be eecively

designed and managed.

Redirect SES resources toward students in lower grades and ELL and

students with disabilities

Anoher consisen nding in disric-based sudies and oher research is ha parici-paion and atendance in SES are signicanly lower or high school sudens han

or elemenary and middle school sudens. o dae, no sudies have shown SES o

be eecive or high school sudens, and hey show greaer atendance ux as well.

Alernaively, we ound ha ELL sudens were more likely o regiser or and atend

SES, and ye here was litle knowledge o or accommodaion or he special curricu-

lum and insrucional needs o ELL or sudens wih disabiliies in SES sessions.

Tese ndings sugges a policy change ha would redirec SES resources rom he

high school level o lower grades and oward new eors and programming o be-

er serve ELL and SWD enrollees.

Take a closer look at online providers

Online SES providers generally charge more or heir services. Tis is possibly

because hey ypically provide compuers or insrucion ha heir sudens are

subsequenly allowed o keep. No surprisingly, he share o sudens signing

up wih online providers has grown as sae and local educaional agencies havegenerally cracked down on providers use o incenives o atrac sudens o

heir programs and encourage atendance. Ofine providers are no longer able

o oer comparably atracive enrollmen incenives as he online providers who


20/28


can promise a compuer. Disric sa expressed concern ha some sudens

atend online SES only long enough o earn he compuer, and sudens ge

signicanly ewer hours o insrucion wih online providers due o heir higher

hourly raes.

We recommend ha sae and local educaional agencies underake a horoughassessmen o wha online providers oer in an SES sessionincluding qual-

iy and diereniaion in he curriculumand consider which crieria or key

elemens should bear on he hourly rae-seting o SES providers. Tis should be

done in combinaion wih heir assessmens o oher providers.

Inormaion gahered on provider perormance on hese crieria should be com-

municaed widely o sudens and parens. Again, disrics will need addiional

nancial and echnical suppor o underake hese evaluaions and eecively dis-

seminae he ndings o sudens and parens.

Focus on the students who need the most academic help

Large urban school disrics in paricular are increasingly sreched o provide

SES o all eligible sudens who sign up. Is likely ha in he absence o addiional

unding even ewer sudens will be served i policies are implemened o increase

he number o hours a given SES paricipan atends, as we recommend.

Some disrics have already had o esablish addiional eligibiliy crieria beyond

sricer low-income requiremens, and we urher recommend ha hese crieria

emphasize reaching hose sudens who are mos severely underperormingwih

adequae levels o uoring.

Better coordination among SES providers and other parties

Finally, i will likely ake greaer coordinaion and communicaion beween SES

providers and heir uors, school adminisraors, regular school-day eachers,

and parens o eecively serve hese sudens mos in need o exra academicassisance. Increased cooperaion, ransparency, and openness in program

managemen a boh sae and local levelswhich we saw aking roo in some

disricswill also be essenial o improving SES eeciveness.


21/28

Conclusion | American Enterprise Institu

Conclusion

Our own research and oher sudies conrm a srong relaionship beween he

inensiy o SES, or he number o hours o SES atended, and is eeciveness.

Te inensiy o he inervenion is parly ied o he hourly rae charged or

SES, which deermines he number o possible hours o SES available o a given

suden. SES providers use o ime or insrucion and suden atendance ux

likewise aec he level and qualiy o SES received.

Sill, even among hose receiving a level o uoring necessary o generae eecs,he magniude o hese eecs is small as gauged by eec sizes or similar kinds o

ou-o-school-ime inervenions. Findings rom observaions o uoring sessions

sugges ha he minimal eecs may also sem rom criical omissions in he qual-

iy and characer o insrucional pracices. Among hese criical omissions is pro-

gramming ha addresses he special needs o ELL and SWD sudens, a problem

made more roubling by he ac ha hese sudens are signing up and atending

SES a higher raes han oher sudens.

We recommend program changes ha should ulimaely help sudens parici-

paing in SES ge more hours o higher-qualiy and appropriaely diereniaed

insrucion. Saes and school disrics need o ake seps o beter monior and

conrol service qualiy and delivery, and hey also need o invesigae wha is ak-

ing place in insrucion provided by online vendors, who operae he mos cosly

and dicul o monior programs.

In disrics aced wih increasingly sreched resources, SES should be direced

primarily o sudens in lower grades, ELL and SWD sudens, and hose who are

mos severely underperorming in school. o be successul in heir service o su-

dens wih he greaes need or academic assisance, school disrics, SES provid-ers and uors, regular school-day eachers, and parens need o beter coordinae

heir eors o increase he insrucional ime received by sudens and he qualiy

o SES insrucion delivered.


22/28


References

Burch, P. and ohers. 2011. Preliminary Findings o aMulisie Sudy o he Implemenaion and Eecs oSupplemenal Educaional Services (SES). Workingpaper. Universiy o Souhern Caliornia and Universiyo Wisconsin.

Burch, P., A. Good, and M. Sewar. 2010. Mulisie Eval-uaion o Supplemenal Educaional Services Observa-ion Insrumen. Universiy o Souhern Caliornia andUniversiy o Wisconsin.

Burch, P. 2009. Hidden Markets: Te New Education Priva-tization. New York: ouledge.

Burch, P. and A. Good. 2009. Geting o he Core: Teole o Insrucional Seting in Federally Mandaeduoring. Working paper presened a he 2009

American Educaional esearch Associaion Coner-ence, San Diego, CA.

Burch, P., M. Seinberg, and J. Donovan. 2007. Supple-menal Educaional Serv ices and NCLB: Policy

Assumpions, Marke Pracices, Emerging Issues. Edu-

cational Evaluation and Policy Analysis 29 (2):115-133.

Chicago Public Schools. 2009. Te 2008 SupplemenalEducaional Services Program: Year 5 SummaiveEvaluaion. Chicago: Chicago Public Schools Oce oExended Learning Opporuniies, esearch, Evalua-ion and Accounabiliy.

Chicago Public Schools. 2008. Te 2007 SupplemenalEducaional Services Program: Year 4 SummaiveEvaluaion. Chicago: Chicago Public Schools Oce oExended Learning Opporuniies, esearch, Evalua-ion and Accounabiliy.

Chicago Public Schools, Oce o esearch, Evaluaion,and Accounabiliy. 2007. SES uoring Programs:

An Evaluaion o Year 3 in he Chicago Public Schools.Chicago: Chicago Public Schools.

Chicago Public Schools, Oce o esearch, Evaluaion,and Accounabiliy. 2005. SES uoring Programs:

An Evaluaion o he Second Year. Chicago: ChicagoPublic Schools.

Durlak, . and . Weissberg. 2007. Te impac o aer-school programs ha promoe personal and socialskills. Chicago: CASEL.

Governmen Accounabiliy Oce. 2006. No Child Le

Behind Ac: Educaion Acions Needed o ImproveLocal Implemenaion and Sae Evaluaion o Supple-menal Educaional Services. Washingon: GAOepor 06-758.

Heinrich, C.J., .H. Meyer, and G. Whiten. 2010. Supple-menal Educaion Services under No Child Le Behind:

Who Signs Up, and Wha Do Tey Gain?EducationalEvaluation and Policy Analysis 32 (June): 273-298.

Heisad, D. 2005. Analysis o 2005 supplemenal educa-ion services in Minneapolis public schools: An applica-ion o mached sample saisical design.Minneapolis:

Minneapolis Public Schools.

Hill, C.J., H.S. Bloom, A. . Black, and M.W. Lipsey. 2008.Empirical Benchmarks or Inerpreing Eec Sizes inesearch. Child Development Perspectives 2 (3): 172-177.

Lauer, P.A. and ohers. 2006. Ou-o-School-ime Pro-grams: A Mea-Analysis o Eecs or A-isk Sudens.Review of Educational Research 76 (2): 275-313.

Litle, P., Wimer, C., and Weiss, H. 2008. Programs inhe 21s Cenury: Teir Poenial and Wha i akeso Achieve I. Cambridge: Harvard Family esearch

Projec.

Naional Academy o Sciences. 2011. Incenives andes-based Accounabiliy in Educaion.


23/28

Reerences | American Enterprise Institu

ickles, J.H. and M.K. Barnhar. 2007. Te Impac oSupplemenal Educaional Services Paricipaion onSuden Achievemen: 2005-06. Los Angeles: Los

Angeles Unied School Disric Program Evaluaionand esearch Branch, Planning, Assessmen andesearch Division.

Springer, M.G., M.J. Pepper, and B. Ghosh-Dasidar. 2009.

Supplemenal Educaional Services and Suden esScore Gains: Evidence rom a Large, Urban SchoolDisric. Working paper, Vanderbil Universiy.

Vandell, D., E. eisner, and K. Pierce. 2007. Oucomeslinked o high-qualiy aerschool programs: Longiu-dinal ndings rom he sudy o promising pracices.Irvine: Universiy o Caliornia; Washingon: PolicySudies Associaes.

Zimmer, . and ohers. 2006. Evaluaion o wo Ou-o-School Program in Pitsburgh Public Schools: NoChild Le Behinds Supplemenal Educaional Servicesand Sae o Pennsylvanias Educaional AssisanceProgram. Working Paper, RND Corporaion.

Zimmer, . and ohers. 2007. Sae and Local Imple-menaion o he No Child Le Behind Ac: Volume

Iile I School Choice, Supplemenal EducaionalServices, and Suden Achievemen. Washingon: U.S.Deparmen o Educaion, Oce o Planning, Evalua-ion and Policy Developmen.


24/28



25/28

Endnotes | American Enterprise Institu

Endnotes

1 In addition, 5 percent o this unding can be set aside or transporta-tion or students eligible or choice programs.

2 Patricia Burch, Matthew Steinberg, and Joseph Donovan, Supple-mental Educational Services and NCLB: Policy Assumptions, MarketPractices, Emerging Issues, Educational Evaluation and Policy

Analysis 29 (2) (2007):115-133; Patricia Burch, Hidden Markets: TheNew Education Privatization (New York: Routledge, 2009).

3 The Government Accountability Oce reported in 2006 thatthree-ourths o states experienced challenges in developing datasystems or tracking and analyzing SES inormation and assessingstudents academic progress, with no state producing a conclusiveassessment o SES providers eects on student academic outcomes.

4 Patricia Burch and Annalee Good, Getting to the Core: The Role oInstructional S etting in Federally Mandated Tutoring. Workingpaper (University o Southern Caliornia, 2009).

5 Chicago Public Schools, Oce o Extended Learning Opportunities,Research, Evaluation and Accountability, The 2008 S upplementalEducational Services Program: Year 5 Summative Evaluation (2009);Chicago Public Schools Oce o Extended Learning Opportunities,Research, Evaluation and Accountability, The 2007 S upplementalEducational Services Program: Year 4 Summative Evaluation (2008);Chicago Public Schools, Oce o Research, Evaluation and Ac-countability, SES Tutoring Programs: An Evaluation o Year 3 in theChicago Public Schools (2007); Chicago Public Schools, O ce oResearch, Evaluation, and Accountability, SES Tutoring Programs:An Evaluation o the Second Year (2005).

6 Jordan H. Rickles and Melissa K. Barnhart, The Impact o Supple-mental Educational Services Participation on Student Achievement:2005-06 (Los Angeles: Los Angeles Unifed School District Program

Evaluation and Research Branch, Planning, Assessment and Re-search Division, 2007).

7 David Heistad, Analysis o 2005 supplemental education servicesin Minneapolis public schools: An application o matched samplestatistical design (Minneapolis: Minneapolis Public Schools, 2005);Carolyn J. Heinrich, Robert H. Meyer, and Greg Whitten, Supple-mental Education Services under No Child Let Behind: Who SignsUp, and What Do They Gain? Educational Evaluation and Policy

Analysis 32 (2) (2010): 273-298.

8 Matthew G. Springer, Matthew J. Pepper, and Bonnie Ghosh-Dastidar,Supplemental Educational Services and Student Test Score Gains:

Evidence rom a Large, Urban School District. Working paper(Vanderbilt University, 2009).

9 The our other studies identifed by Springer, Pepper, and Ghosh-Dastidar are: Ron Zim mer and others, Evaluation o Two Out-o-School Program in Pittsburgh Public Schools: No Child Let Behind's

Supplemental Educational Services and State o Pennsylvania'sEducational Assistance Program. RAND Working Paper (RANDCorporation, 2006); Ron Zimmer and others, State and LocalImplementation o the No Child Let Behind Act: Volume ITitleI School Choice, Supplemental Educational Services, and StudentAchievement (Washington: U.S. Department o Education, Oceo Planning, Evaluation and Policy Development, 2007); Heinrich,Meyer and Whitten, Supplemental Education Services under NoChild Let Behind; Heistad, Analysis o 2005 supplemental educa-tion services in Minneapolis public schools.

10 Patricia A. Lauer and others, Out-o-School-Time Programs: AMeta-Analysis o Eects or At-Risk Students,Review of EducationalResearch 76 (2) (2006): 275-313.

11 Springer, Pepper and Ghosh-Dastidar, Supplemental EducationalServices and Student Test Score Gains; Zimmer and others, Evalua-tion o Two Out-o-School Program in Pittsburgh Public Schools.

12 Patricia Burch and others, Preliminary Findings o a Multisite Studyo the Implementation and Eects o Supplemental EducationalServices (SES). Working paper (University o Southern Caliornia andUniversity o Wisconsin, 2011); Springer, Pepper and Ghosh-Dastidar,

Supplemental Educational Services and Student Test Score Gains;Zimmer and others, State and Local Implementation o the NoChild Let Behind Act.

13 Zimmer and others, State and Local Implementation o the No ChildLet Behind Act; Springer, Pepper and Ghosh-Dastidar, Supple-mental Educational Services and Student Test Score Gains.

14 Burch and others, Preliminary Findings o a Multisite Study o theImplementation and Eects o Supplemental Educational Services(SES): Heinrich, Meyer and Whitten, Supplemental EducationServices under No Child Let Behind.

15 Zimmer and others, State and Local Implementation o the No ChildLet Behind Act.

16 Springer, Pepper and Ghosh-Dastidar, Supplemental EducationalServices and Student Test Score Gains.

17 Carolyn J. Hill and others, Empirical Benchmarks or InterpretingEect Sizes in Research, Child Development Perspectives2 (3) (2008):172-177.

18 National Academy o Sciences Report o the Committee on Incen-tives and Test-based Accountability, Incentives and Test-basedAccountability in Education, (2011).

19 Joseph A. Durlak and Roger P. Weissberg, The impact o ater-school programs that promote personal and social skills (Chicago:CASEL, 2007); Priscilla Little, Christopher Wimer, and Heather Weiss,

Programs in the 21st Century: Their Potential and What it takes toAchieve It (Cambridge: Harvard Family Research Project, 2008) ;Debra Lowe Vandell, Elizabeth Reisner, and Kim M. Pierce, Out-comes linked to high-quality aterschool programs: Longitudinalfndings rom the study o promising practices. (Irvine: University oCaliornia and Washington: Policy Studies Associates, 2007).

20 Patricia Burch, Annalee Good, and Mary Stewart, Multisite Evaluationo Supplemental Educational Services Observation Instrument(University o Southern Caliornia and University o Wisconsin-Madison, 2010).

21 Homework help and test preparation activities were present in somesessions but were not the dominant orm o academic support.


26/28


About the authors

Carolyn Heinrich is he Sid ichardson Proessor o Public Aairs and he direc-

or o he Cener or Healh and Social Policy a he Lyndon B. Johnson School

o Public Aairs a he Universiy o exas a Ausin. Her research ocuses on

social welare and educaion policy, public managemen and perormancemanagemen, and social-program evaluaion. She requenly works direcly in

her research wih governmens a all levels, including wih he ederal govern-

men on evaluaions o workorce developmen programs, wih saes on heir

social welare and child suppor programs, school disrics in he evaluaion o

supplemenal educaional services and oher educaional inervenions, and

governmens such as Brazil and Souh Arica on heir povery reducion and

human capial developmen programs. In 2004, Proessor Heinrich received he

David N. Kershaw Award or disinguished conribuions o he eld o public

policy analysis and managemen by a person under age 40. Prior o her appoin-

men a he Universiy o Wisconsin-Madison in 2003, Proessor Heinrich wasan assisan proessor a he Universiy o Norh Carolina-Chapel Hill and held

an academic research appoinmen a he Universiy o Chicago. She received

her docorae in public policy sudies rom he Universiy o Chicago.

Patricia Burch is an associae proessor a he ossier School o Educaion,

Universiy o Souhern Caliornia. Prior o his, she was an assisan proessor in

he Deparmen o Educaional Policy Sudies, Universiy o Wisconsin, Madison.

Burch received her B.A. in English rom Oberlin College in 1985, her M.A. in

educaion rom Harvard Universiy in 1991, her M.A. in sociology rom Sanord

Universiy in 1998, and her Ph.D. in educaion rom Sanord Universiy in 2000.

Over he pas decade, Dr. Burch has conduced major sudies and evaluaions o

K-12 educaion reorm such as SAGE (he Wisconsin class-size iniiaive), sys-

emic insrucional reorm in large urban school disrics (Seatle, Chicago, and

Philadelphia), proessional developmen reorms (Chicago, Philadelphia, and New

York Ciy), and school-linked services (Wes Virginia, New Mexico, Boson, and

San Diego). Her newes work examines he role o privae rms as inuences in he

design and implemenaion o K-12 educaion policy and how public educaion

is being ransormed below he radar by new orms o privaizaion. Her book

Hidden Markets: Te New Education Privatization was published by ouledgein 2009. She currenly serves as a co-principal invesigaor on a our-year sudy

iled, A Muli-Sie Evaluaion o he Implemenaion and Impac o Supplemenal

Educaional Services, unded by he Insiue o Educaion Sciences.
http://www.lafollette.wisc.edu/calendar-news/2004/2004-fall.html#heinrichhttp://www.lafollette.wisc.edu/calendar-news/2004/2004-fall.html#heinrich


27/28

About the authors and acknowledgements | American Enterprise Institu

Acknowledgments

Te auhors hank he major under o heir ongoing research on supplemenal

educaional services, he Insiue or Educaional Sciences, P/Award number:

305A090301, Educaion Policy, Finance, and Sysems esearch Program, Goal 3.


28/28

About the Center or American Progress

The Center or American Progress is a nonpartisan re-

search and educational institute dedicated to promoting

a strong, just and ree America that ensures opportunityor all. We believe that Americans are bound together by

a common commitment to these values and we aspire

to ensure that our national policies reect these values.

We work to fnd progressive and pragmatic solutions

to signifcant domestic and international problems and

develop policy proposals that oster a government that

is o the people, by the people, and or the people.

Center or American Progress

1333 H Street, NW, 10th Floor

Washington, DC 20005

Tel: 202.682.1611 Fax: 202.682.1867

www.americanprogress.org

About the American Enterprise Institute

The American Enterprise Institute is a community o

scholars and supporters committed to expanding libe

increasing individual opportunity, and strengtheninree enterprise. AEI pursues these unchanging ideal

through independent thinking, open debate, reasone

argument, acts, and the highest standards o researc

and exposition. Without regard or politics or prevailin

ashion, we dedicate our work to a more prosperous

saer, and more democratic nation and world.

The American Enterprise Institute

1150 Seventeenth Street, N.W.

Washington, DC 20036

Tel: 202.862.5800 Fax: 202.862.7177

www.aei.org

American Enterprise Institutefor Public Policy Research

The Implementation and Effectiveness of Supplemental Educational Services

Documents

Transcript of The Implementation and Effectiveness of Supplemental Educational Services