Head Start National Reporting System

Assessment Development, First-Year Results, Future Plans

Nicholas Zill, Ph.D. Westat

Presented at the National Head Start Association32nd Annual Training Conference

May 26, 2005

2

Purpose & Functions

• The Head Start National Reporting System (NRS) was designed to address the President’s Good Start, Grow Smart initiative and 1998 Congressional mandates for program.

• NRS provides indicators of the progress children are making on key early literacy and math skills for all local programs.

3

Purpose & Functions (2)

• NRS supplements local child assessment and program self-assessment efforts.

• For the first time, provides Head Start with consistently-collected, comparable measures for all programs.

• To be used in planning training and technical assistance for programs.

• Incorporate in future program monitoring.

4

How NRS Works

• All kindergarten-eligible 4- and 5-year-old children are given the same brief one-on-one assessment.

• Fall and spring assessments are reported to show progress over course of year.

• 3-day regional training sessions for trainers from 2,000 Head Start programs.

• They train local staff to conduct child assessments.

5

How NRS Works (2)

• Child’s responses recorded by assessor on scannable answer sheets.

• Internet-based Computer-Based Reporting System (CBRS) supports assessment management and use.

• Programs may import data from local information-management systems into CBRS.

6

NRS Data ProcessingLocal StaffEnter Data into CBRS

XtriaCBRS data received

into Internet data base

Local AssessorsComplete

Direct Child Assessments

PearsonAnswer Forms

Scanned

WestatHSNRS data received into analysis data base

and analyzed

Head Start Bureaureceives and reviews

HSNRS report

Local Programs

RegionalOffices

CBRS generates ID numbers based on class rosters

7

HSNRS Assessment Components

• English Language Screener

• Vocabulary

• Letter Naming

• Early Math Skills

8

Spanish Assessment Components

• Spanish Language Screener

• Spanish Vocabulary

• Spanish Letter Naming

• Spanish Early Math Skills

9

Criteria for test components

• Skills that Congress and the President expect Head Start children to learn.

• Can be reliably measured in relatively brief assessment.

• Contribute to achievement in school.

• Can be enhanced by activities in Head Start.

• Skills that parents want children to learn.

10

Developing Assessments

• Westat used extensive assessment data base from longitudinal studies on samples of Head Start children (FACES, QRC interventions, Head Start Impact Study)

• Select subsets of items with difficulty levels suitable for Head Start children.

• Simplify assessment procedures to enhance reliability with local assessors.

11

Developing Assessments (2)

• Development process guided by expert advice of Technical Work Group.

• National field test, public comment preceded first year of implementation.

• Extensive quality-control effort, including monitoring study by independent contractor and comparisons with longitudinal sample survey (FACES 2003).

12

Field Test

• National field test of 36 Head Start programs and 1,430 children conducted in April and May of 2003

• Test of child assessment and “training of trainers” approach

• Parallel assessments by local Head Start staff and trained and experienced Westat field assessors

13

Field Test Results• All proposed components had acceptable

internal consistency reliability.

• All components except phonological awareness (Elision task) had acceptable inter-assessor reliability.

• Good agreement between mean scores from experienced assessors & local staff.

• Similar means from teachers and non-teachers.

14

Field Test Results (2)

• Language screener worked well with children who spoke a non-English language at home.

• Technical Work Group recommended deletion of Elision task, but approved moving forward with remainder of NRS battery.

• HSB decided on national implementation.

15

National Implementation

• Summer “training of trainers” held in 7 cities and Puerto Rico.

• 2,831 local staff members trained and certified as English trainers.

• 576 staff certified as Bilingual trainers.

• Trainers returned to their programs and trained and certified more than 20,000 local assessors.

16

First Year Data Collection• Approximately 429,000 answer forms

successfully processed in Fall 2003 and 410,000 in Spring 2004.

• Assessment data processed for 1,790 Head Start programs.

• Matching fall and spring assessments obtained for 343,260 children (80%).

• Largest assessment of preschool children ever carried out.

17

How Assessments Are Scored

• Each assessment component scored separately.

• No attempt to produce single summary score for entire assessment.

• Mean percent of items correct and percentage of children at different skill levels reported to local programs.

• IRT scale scores developed for cross-program analysis and reporting to national and regional offices of Head Start Bureau.

18

Growth Analysis Results

• Head Start children showed significant growth in all English-language skill areas.

• Reliability of program-level fall-spring growth rates based on NRS data was good for both English and Spanish assessments.

• Virtually no program showed zero growth or negative growth in English skills.

19

Majority of English-Language Learners in Head Start Comprehend Spoken English Well by End of Program Year

35%

28%

13%

53%

0%

20%

40%

60%

80%

100%

Limited or No Understanding Comprehends Directions Well

Understanding Spoken English Skill Levels

Mea

n P

erce

nt o

f E

nglis

h-La

ngua

ge

Lear

ners

Fall 2003

Spring 2004

Majority of English-Language Learners in Head Start Comprehend Spoken English Well by End of Program Year

35%

28%

13%

53%

0%

20%

40%

60%

80%

100%

Limited or No Understanding Comprehends Directions Well

Understanding Spoken English Skill Levels

Mea

n P

erce

nt o

f E

nglis

h-La

ngua

ge

Lear

ners

Fall 2003

Spring 2004

20

Proportion of Head Start Children who are One Year or More Behind in Vocabulary Development Declines from

Nearly Half to One-Third

48%

20%

35% 31%

0%

20%

40%

60%

80%

100%

Vocabulary size one year ormore behind child's age

Vocabulary size at or abovechild's age

Vocabulary Skill Levels

Per

cent

of

Chi

ldre

n

Fall 2003

Spring 2004

21

Majority of Head Start Children At or Above Congressionally-Mandated Skill Level for Letter Recognition by End of Program Year

74%

16%

39%

46%

0%

20%

40%

60%

80%

100%

Knows Less Than 10 Letters Knows 17 or More Letters

Letter Recognition Skill Levels

Per

cent

of

Chi

ldre

n

Fall 2003

Spring 2004

Majority of Head Start Children At or Above Congressionally-Mandated Skill Level for Letter Recognition by End of Program Year

74%

16%

39%

46%

0%

20%

40%

60%

80%

100%

Knows Less Than 10 Letters Knows 17 or More Letters

Letter Recognition Skill Levels

Per

cent

of

Chi

ldre

n

Fall 2003

Spring 2004

22

Proportion of Head Start Children at Higher Math Skill Level Increases to a Majority

66%

14%20%

61%

0%

20%

40%

60%

80%

100%

Counts 11 or fewer objects Counts 15 or more objects

Early Math Skill Levels

Per

cent

of

Chi

ldre

n

Fall 2003

Spring 2004

23

Spanish Growth Analysis• Head Start children assessed in Spanish

showed growth in all skill areas.

• Growth was greater in programs in Puerto Rico, where instruction is in Spanish.

• Spanish growth rates comparable to English, except for vocabulary.

• Some programs on U.S. mainland showed average Spanish growth rates at or below zero.

24

Growth in Spanish Vocabulary Knowledge in Head Start Programs in Puerto Rico and on U.S. Mainland

61%

54%

78%

60%

0%

20%

40%

60%

80%

100%

Puerto Rico U.S. Mainland

Program Location and Time of Assessment

Mea

n P

erce

nt o

f It

ems

Cor

rect

Fall 2003

Spring 2004

Growth in Spanish Vocabulary Knowledge in Head Start Programs in Puerto Rico and on U.S. Mainland

61%

54%

78%

60%

0%

20%

40%

60%

80%

100%

Puerto Rico U.S. Mainland

Program Location and Time of Assessment

Mea

n P

erce

nt o

f It

ems

Cor

rect

Fall 2003

Spring 2004

25

Mean Program Fall-Spring Growth Rates (as Effect Sizes) by Skill Area and Language of Assessment

60%

68%

93%89%

59%

28%

96%

86%

0%

20%

40%

60%

80%

100%

UnderstandingSpoken Language

Vocabulary Letter Recognition Early Math

Skill Area and Language

% o

f P

opul

atio

n S

tand

ard

Dev

iati

on

English

Spanish

26

Current Activities• Second year of national implementation of

NRS underway. Thus far, numbers comparable to first year.

• Improvements in training and assessment procedures in response to feedback from Quality Assurance Study, TWG, and local staff.

• Distribution of materials proceeding in smoother, more timely fashion.

27

Ongoing Analysis• Continuing scrutiny of reliability and

validity of NRS data.

• Multilevel regression analysis of spring achievement levels and fall-spring growth rates (program, center, class, child).

• Using independent variables from CBRS and Program Information Report.

• Qualitative study of high- and low-performing programs.

28

Future Plans

• Possible expansion of NRS to encompass domains of phonological awareness, social-emotional development, child health.

• Exploration of alternative assessment techniques (e.g., computer-assisted personal assessment).

• Feasibility of using sampling to reduce assessment burden.

29

In Conclusion

• Despite considerable controversy, a National Reporting System for Head Start has been implemented and appears to be working reasonably well.

• The system involves in-person assessment of all 4- and 5-year-old children on a limited but important set of early literacy and math skills.

30

In Conclusion (2)• Assessment results will be scrutinized

carefully for several years before NRS data are used for any administrative decisions affecting individual programs.

• The Head Start Bureau will be examining the feasibility of expanding the scope of the NRS to encompass other domains, and use of alternative methods to improve accuracy and reduce burden.