Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager...
-
Upload
rosemary-maxwell -
Category
Documents
-
view
223 -
download
3
Transcript of Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager...
![Page 1: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/1.jpg)
Positive Impacts and Return Positive Impacts and Return on Investment: What We on Investment: What We
Can Learn from Head Start Can Learn from Head Start ResearchResearch
Positive Impacts and Return Positive Impacts and Return on Investment: What We on Investment: What We
Can Learn from Head Start Can Learn from Head Start ResearchResearch
Hilary ShagerHilary Shager
WI Department of Children and WI Department of Children and FamiliesFamilies
April 25, 2012April 25, 2012
![Page 2: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/2.jpg)
Overview
• Being a good consumer of Head Start research
• Early cohort research• The National Head Start Impact
Study• Early Head Start Research• What’s new in Head Start Research• Remaining research questions
![Page 3: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/3.jpg)
“...finally there is indisputable evidence about the program's effectiveness, provided by the Department of Health and Human Services: Head Start simply does not work.”
-Joe Klein, Time Magazine,
July 2011
![Page 4: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/4.jpg)
“Head Start improves the odds and the options for at-risk kids for a lifetime. Kids that have been through Head Start and Early Head Start are healthier, more academically accomplished, more likely to be employed, commit fewer crimes, and contribute more to society. Simply put: Head Start works. It’s been proven.”
-Letter to Congress, signed by over 270 top early childhood researchers
March, 2011
![Page 5: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/5.jpg)
Being a good consumer of Head Start research• Lots of Head Start research• Lots of rhetoric
– Which research is good research?– What do the results tell us about
Head Start’s effectiveness?– What are the caveats?
• Bottom line…– There is a lot to be excited about!
![Page 6: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/6.jpg)
Challenges in Head Start Research
• Lots of things affect educational, health, family outcomes– Hard to isolate the “true” impact of
Head Start programming
• Finding the appropriate control group– Head Start targeted at particularly
disadvantaged children and families
![Page 7: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/7.jpg)
Questions to ask1) Is the evaluation design strong enough to
produce trustworthy evidence?2) What program services were actually
received by participating children and families in the comparison group?
3) How big an impact did the program have?
4) Do the program’s benefits exceed its costs?
5) How similar are the programs, children, and families in the study to those in my community?
![Page 8: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/8.jpg)
Research 101: Research Design
• Gold Standard=Experiment– Randomized controlled trial
• Quasi-experimental approaches– Does the control group really look like
the Head Start population?– What services do children and families
in the control group get?• External validity
– If I reproduced the study in my community, would I get the same results?
![Page 9: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/9.jpg)
Research 101: Effect Size
• What is an effect size?– Measure of “magnitude” of impact
• The bigger, the better!• More meaningful than “statistically
significant”
– Fraction of the difference between the treatment and control group over variation• Allows comparison across different
measures• Need this to do a cost/benefit analysis
![Page 10: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/10.jpg)
Early cohort research: Some history
• The Westinghouse study (1968)– Quasi-experimental approach– Focus on cognitive outcomes only– Almost defunds Head Start– Consortium of researchers rallies
against the study
• We need better research on more outcomes!
• We waited a long time...
![Page 11: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/11.jpg)
Janet Currie and colleagues: Sibling fixed effect studies
(1990s)
• Use nationally representative data sets• Include families in which one sibling
went to Head Start and one didn’t• Teases out all the “fixed” family-
related factors that influence outcomes• Still questions about why one sibling
attended and not the other
![Page 12: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/12.jpg)
Currie et al. findings• Large gains in vocabulary test scores
– Equal to about 1/3 of achievement gap– Effects persist into adulthood for whites, but fade quickly for
African-Americans• Reduced probability of retention (47%)
– Again, only for whites• More likely to be immunized (8-11%)
– Both whites and African-Americans• Cost-benefit implications
– Increased test scores would increase expected future wage by 4%
– Reduction in retention would lead to 5% decline in probability of high school drop-out
– Preventative health savings– Definitely passes test for whites, question for African-Americans
![Page 13: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/13.jpg)
More Currie et al. findings
• Educational attainment (whites only)– 22 percentage points more likely to complete high school– 19 percentage points more likely to attend some college
• Less likely to be arrested or charged for crime– 12 percentage points, African-Americans only
• Positive effects for Hispanic children– Closes 1/4 gap in test scores– Closes 2/3 gap in probability of grade repetition
• Maybe effects less, fade for African-American children because they attend lower quality schools later
![Page 14: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/14.jpg)
Ludwig & Miller (2007)• 50 to 100% increase in Head Start funding in
1960s and 70s associated with– Increase of schooling attainment of about 1/2 year– 15% increase in likelihood of attending some
college– Found for both African-Americans and whites– Decrease in infant mortality
• Benefit/cost ratio as large as 7 to 1– Decline in mortality $120 per 4-year-old– Extra half year of schooling increases earnings 2%
or $15,000– Even higher if consider increased schooling
reduces crime
![Page 15: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/15.jpg)
Deming, 2009• Uses same method as Currie et al., but more
recent cohort• Effects on math and reading that fade
– 0.15 at age 5-6– 0.13 at age 7-10– 0.06 at age 11-14– Larger for African-Americans at first, but fade more
quickly• Larger positive effects on long-term outcomes
– 18.6 percentage points more likely to complete high school (African-Americans)
– “Index” of long-term outcomes, 0.23• Mechanism = less retention, special education
![Page 16: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/16.jpg)
Hard to generalize these findings to today
• More working parents• More early childhood
programming options• Decrease in achievement gaps• Increase in Head Start quality
![Page 17: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/17.jpg)
The National Head Start Impact Study
(NHSIS)
• Randomized controlled trial required by Congress in 1990s
• Nationally representative sample– 84 Head Start programs– Approximately 4,600 children– Begin attending in 2002
• Follow kids through end of 1st grade
![Page 18: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/18.jpg)
First year findings: 4-year-olds
• 7 positive cognitive effect sizes – Effect sizes = 0.16 to 0.31
• More likely to visit the dentist– (73.2% vs. 56.9%)
• Parents more likely to read to children– (3x/wk vs. 2.8x/wk)
• No significant findings for more advanced reading skills, math, or behavior problems
![Page 19: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/19.jpg)
First year findings: 3-year-olds
• Same positive impacts as 4-year-olds, but bigger, plus...– Math– Fewer problem behaviors– Parents spanked children less– Overall health
• Still no effects on more advanced reading skills
![Page 20: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/20.jpg)
Lots of Caveats
• “Intent to treat” analysis– 17% of 3-year-olds and 14% of 4-year-
olds never got Head Start services • Crossovers
– 18% of 3-year-olds and 14% of 4-year-olds in control group attended Head Start
• 60% of control kids in other programs• Evidence of circumventing
randomization– Center directors enroll those with highest
needs
![Page 21: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/21.jpg)
Ludwig & Phillips re-analysis
• “Treatment on the treated” (TOT) – Adjust for crossovers and no-shows– Larger effect sizes (0.14 to 0.50)
• Pool together 3- and 4-year-olds– Larger sample, more “power”– Statistically significant impacts for
nearly all main cognitive outcomes
![Page 22: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/22.jpg)
First grade follow-up
• Significant effects from first year mostly disappear– For 4-year-olds, only impact on
receptive vocabulary remains (0.09)– For 3-year-olds, only impact on oral
comprehension remains (0.08)
![Page 23: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/23.jpg)
Why we shouldn’t compare these results to pre-k study
results
• Pre-k studies use different research design that is not as strong as NHSIS
• Fewer kids in pre-k control groups attend other programs
• Different populations• Pre-k only looks at academic
outcomes• No long-term follow-up of pre-k
(therefore, no info about “fade out”)
![Page 24: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/24.jpg)
So, what does this study really tell us?
• Impacts represent short-term effects of offer of one year of Head Start
• Measures impact of Head Start versus everything else available
• Initial impacts found are about the same size as those found for children attending Head Start in the 1960s-1980s
• For those cohorts, Head Start seems to have produced long-term benefits large enough to outweigh program costs, despite fade-out of initial test score impacts (nothing like that measured here)
• But test score impacts do seem to fade more quickly in new cohort—not sure why or whether we should worry about this
![Page 25: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/25.jpg)
Does Head Start pass the benefit/cost test today?
• Ludwig & Phillips (2008) – Cost = $9,000 per child– Head Start might pass a benefit-cost
test if short-term effects on achievement test scores were as small as 0.1 to 0.2
![Page 26: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/26.jpg)
Early Head Start research
• Randomized controlled trial• Follow through 5th grade• Positive impacts at age 3
– Cognitive and language development– Socio-emotional development– Home environment and parenting behavior– Parents’ progress toward self-sufficiency– Positive impacts specifically for fathers
• “Sleeper” effects• Differences across different service models
![Page 27: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/27.jpg)
Early Head Start Home Visiting Evidence
![Page 28: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/28.jpg)
What’s new in Head Start research?
• Focus on “quality”• Curriculum enhancements• Professional development• Early Head Start and home visiting• Head Start Family and Child
Experiences Survey (FACES)– New! “Baby FACES”
![Page 29: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/29.jpg)
Meta-analysis (Shager et al., 2012)
• Study of 33 Head Start evaluations• Overall, positive effect of Head Start on
cognitive and achievement outcomes (effect size = 0.27)
• What happens in the control group matters a lot!– Effect sizes for studies that have an “active”
control group are 0.33 smaller– Type of skill, measures used, and where
published also matter
![Page 30: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/30.jpg)
Conclusions• Some high quality Head Start research exists,
but you need to be an alert consumer• Head Start and Early Head Start have had
positive impacts on children and families across a variety of domains
• Head Start appears to pass the benefit/cost test
• We still have a lot to learn about how and why early childhood programs improve long-term outcomes
![Page 31: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/31.jpg)
Remaining questions• Instead of asking whether Head Start is
effective at all, we must ask how effective Head Start is compared to the range of other options available.
• What happens to kids after they attend Head Start?– Do effects really “fade out,” or do other kids
“catch up”?• Which behavioral and socio-emotional
skills are important, and how can Head Start foster those?
• Other benefits? – Health, parenting, family self-sufficiency
![Page 32: Positive Impacts and Return on Investment: What We Can Learn from Head Start Research Hilary Shager WI Department of Children and Families April 25, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022052510/56649e635503460f94b5fb9e/html5/thumbnails/32.jpg)
What are your questions?