Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the...
-
Upload
chloe-joseph -
Category
Documents
-
view
219 -
download
1
Transcript of Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the...
![Page 1: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/1.jpg)
Approaches to Measuring Approaches to Measuring Child OutcomesChild Outcomes
Approaches to Measuring Approaches to Measuring Child OutcomesChild Outcomes
Kathy HebbelerECO at SRI International
Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes, Albuquerque, NM April 2006
![Page 2: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/2.jpg)
2
What is happeningWhat is happening
Outcomes measurement is difficult and very complex
No one group has all the answers – or even most of the answers
There are some exciting things going on around the country
![Page 3: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/3.jpg)
3
Purpose of the meetingPurpose of the meeting
Share our challenges and what we have learned so far
Contribute to the collective knowledge base
Advance the discussion incrementally moving closer to producing outcomes data
*** for the ultimate good of children and families ***
![Page 4: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/4.jpg)
4
Critical eventsCritical events Spring 2005 – ECO submitted
recommendations to OSEP on what should be collected with regard to child and family outcomes
Summer 2005 – OSEP released the reporting requirements
December 2005 – States submitted their plans for outcome data collection in their State Performance Plan
Spring 2006 – States are collecting data
![Page 5: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/5.jpg)
5
The Number One Question*:The Number One Question*:
What are other states doing?
Have other states done X?
*In two forms
![Page 6: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/6.jpg)
6
Purpose: The Overriding Purpose: The Overriding QuestionQuestion
Why is a state collecting data on child outcomes?
Context, resources, values, etc. enter into the answer
![Page 7: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/7.jpg)
7
To respond to federal
reporting requirements
To meet provider/teacher, local and/or state need for outcome
information and
to respond to federal reporting
requirements
Purpose
WHY?
![Page 8: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/8.jpg)
Context & Context & Values Drive Values Drive
DecisionsDecisions
Resources?
Stakeholder input?
Burden on locals?
Standardized assessment?
Authentic assessment?
Interagency issues?
Local control?
Minimize change?Early learning guidelines?
Multiple sources of information?
Policymakers want data?
Other early childhood initiatives?
![Page 9: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/9.jpg)
9
Who is included in the outcomes Who is included in the outcomes system?system?
Pt. C system; Pt. B system Some blending of C and B
Same assessment Data sharing Data linking
Early Childhood System that includes C and B
![Page 10: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/10.jpg)
10
How does the state get data on How does the state get data on outcomes?outcomes?
Who provides? What assessments are used? How often is data collected? When is data collected? (When is it
reported?) Dealing with multiple sources? Dealing with different assessments?
![Page 11: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/11.jpg)
How Outcomes Data Get to the State AgencyHow Outcomes Data Get to the State Agency
![Page 12: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/12.jpg)
12
Analysis of SPPsAnalysis of SPPs
Analyses are based on SPP reports submitted in December 2005
Pt. C N=56; Pt B N= 58 Limitations
Variation in level of detail provided Landscape keeps changing
Analysis done by Lynne Kahn and staff at UNC/FPG
![Page 13: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/13.jpg)
13
““Camera” Issue: Capturing Child Camera” Issue: Capturing Child FunctioningFunctioning
What are the sources of the information on child functioning?
What kind of assessment tools are states planning to use?
![Page 14: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/14.jpg)
14
Capturing Child Functioning: Capturing Child Functioning: How many sources?How many sources?
Multiple sources Pt. C: 50% (28 states) Pt. B: 16% (16 states)
One data source Pt C: 39% (22 states)
Assessment instrument (21 states) Pt B: 55% (32 states)
Assessment instrument (31 states)
![Page 15: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/15.jpg)
15
Issues RaisedIssues Raised Data needs to reflect a child’s
functioning in each broad outcome area Functional outcomes summarize each
child’s current functioning across settings and situations
Best practice for assessing young children recommends the use of multiple measures
Will single sources (= assessment tool) Will single sources (= assessment tool) produce valid data on functional produce valid data on functional outcomes? How good is the camera?outcomes? How good is the camera?
![Page 16: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/16.jpg)
16
The Child OutcomesThe Child Outcomes
Children have positive social relationships
Children acquire and use knowledge and skills
Children take appropriate action to meet their needs
![Page 17: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/17.jpg)
17
Part C Outcomes Data SourcesPart C Outcomes Data Sources
Data Source # %
Formal assessment instruments
45 80%
Parent report 25 45%
Observation 14 25%
Clinical opinion 10 18%
IFSP goals & objectives 6 11%
Record review 4 7%
Not reported 6 11%
![Page 18: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/18.jpg)
18
Preschool Outcomes Data SourcesPreschool Outcomes Data Sources
Data Source # %
Formal assessment instruments
45 80%
Observation 12 21%
Parent report 11 19%
Teacher/provider report 8 14%
IEP goals & objectives 1 2%
Clinical opinion 1 2%
Not reported 10 17%
![Page 19: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/19.jpg)
19
Role of FamiliesRole of Families Impossible to understand how a child is
functioning across a variety of everyday settings and situations without family input
Options Incorporated into the assessment tool Collected through a parent-completed tool Incorporated into a summary rating
Issue: How is information from families Issue: How is information from families being included?being included?
![Page 20: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/20.jpg)
20
Capturing Child Functioning: Capturing Child Functioning: Approaches to identifying Approaches to identifying assessment toolsassessment tools
One assessment selected by state
List of assessments developed by state; programs pick
Programs can use whatever they have been using
![Page 21: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/21.jpg)
21
Capturing Child Functioning: Capturing Child Functioning: Assessment Tools Being UsedAssessment Tools Being Used
Part C – 20 different assessment tools identified 3 states using state developed tool
Part B – 43 different assessment tools identified 7 states using state-developed tool
![Page 22: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/22.jpg)
22
Commonly Reported Commonly Reported Assessment Instruments: Part CAssessment Instruments: Part C
Of 28 states who listed specific assessment instruments: HELP - 15 states BDI/BDI2 - 13 states AEPS - 11 states Creative Curriculum - 6 states ELAP- 6 states
Not reported – 30 states Not yet determined - 23 states
![Page 23: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/23.jpg)
23
Commonly Reported Commonly Reported Assessment Instruments: PreschoolAssessment Instruments: Preschool
Of 31 states who listed specific assessment instruments: BDI/BDI2 - 9 states Creative Curriculum - 8 states Brigance- 7 states High Scope COR - 6 states AEPS - 5 states State developed assessments - 7 states
Not reported - 27 states Not yet determined – 21 states
![Page 24: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/24.jpg)
24
Capturing Child Functioning: Capturing Child Functioning: Combining Information from Multiple Combining Information from Multiple SourcesSources
Part C : Using ECO Summary Form – 52% (29
states) Developing own summary tools – 7%
(4 states) Part B:
Using ECO Summary Form – 29% (17 states)
Developing own summary tools – 10% (6 states)
![Page 25: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/25.jpg)
25
Capturing Child Functioning: Capturing Child Functioning: TimingTiming
When and how often outcome information is being collected is related to why state is collecting data
What assessment is also related to why
OSEP requirement is entry and exit
![Page 26: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/26.jpg)
26
When will data be “collected”?When will data be “collected”?
Aligned around the naturally occurring data review points in programs
“Collected” may mean Data reviewed/summarized to determine
a functional level for each of the outcomes
Summary rating or other data reported to state or OSEP
Some states did not report anything beside at entry and exit; (C – 28 states; B - 15 states)
![Page 27: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/27.jpg)
27
When entry data will be collectedWhen entry data will be collected(three general patterns)(three general patterns)
Referral Eligibility Initial IFSP- e.g. goals, services, settings
IFSP 6 month review- intervention planning
Around eligibility (based on evaluation data)
Initial IFSP (based on eval and assessment data)
After services begin (based on eval, assess, and/or ongoing progress monitoring data
![Page 28: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/28.jpg)
28
Part C examples of when data Part C examples of when data will be collectedwill be collected
W/in 45 days of referral- 7 states W/in 1 month of IFSP- 2 states W/in 6 months of enrollment- 1 At initial IFSP and 6 month and
annual reviews- 21 W/in 2 months, 45 days, 3 months,
6 months of exit
![Page 29: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/29.jpg)
29
Preschool examples of when Preschool examples of when data will be collecteddata will be collected
Initial evaluations/eligibility - 7 states Initial IEP development -4 states Annual IEP reviews - 11 states Time periods prescribed by curriculum
referenced tools (2 or 3 times a year)- 8 states
Annually at the end of the school year- 6 states
![Page 30: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/30.jpg)
30
Which Children Will Be Included: Which Children Will Be Included: Part C Part C
All children – 40 states After a pilot or phase in period- 16 states
Sampling – 7 states 1 sampling at exit (all children will have
entry data) 1 sampling at entry (will only collect entry
and exit data on children in sample) Other 5 - could not tell from SPP
Not reported or undecided - 9 states
![Page 31: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/31.jpg)
31
Which Children will be Included: Which Children will be Included: PreschoolPreschool
All children – 42 states After a pilot or phase in period- 15 states
Sampling - 8 states 3 will collect data on ALL children, but select
a sample to report to OSEP 1 sampling at entry (will only collect entry
and exit data on children in sample) Other 4 - could not tell from SPP
Not reported or undecided- 8 states
![Page 32: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/32.jpg)
32
Collaboration between C and B Collaboration between C and B
25 states reported in Part C SPP collaborating with Part B on outcomes
21 states reported in Part B SPP collaborating with Part C on outcomes
![Page 33: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/33.jpg)
33
Collaboration with Other Early Collaboration with Other Early Childhood InitiativesChildhood Initiatives
Collaborate with or align outcome efforts with broader early childhood accountability initiatives in their state Part C: 3 states Part B: 18 states
Issue: What are the outcomes being Issue: What are the outcomes being assessed in the broader initiatives?assessed in the broader initiatives?
![Page 34: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/34.jpg)
34
Role of the Early Learning Role of the Early Learning GuidelinesGuidelines
May change or add to the outcomes questions
Are children meeting the ELGs?Are children meeting the ELGs? May mean mapping the ELGs to
the 3 OSEP outcomes Aligning with ELGs:
Part C – 8 states; Part B – 18 states
![Page 35: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/35.jpg)
How Outcomes Data Get to the State AgencyHow Outcomes Data Get to the State Agency
![Page 36: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/36.jpg)
36
Transfer Issues: How does Transfer Issues: How does information move?information move?
In what form? At what level of
detail? With what level of
identification?
![Page 37: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/37.jpg)
37
In what form?In what form?
Online In an electronic
file On paper
![Page 38: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/38.jpg)
38
At what level of detail?At what level of detail?
Child Level Data Item level data on the child (from an online
assessment system) Scores on assessment tool ECO Summary Rating OSEP Categories (a, b, c) Other?
Aggregated Data Scores, Rating, OSEP categories, etc.
![Page 39: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/39.jpg)
39
With what level of With what level of identification?identification?
Only relevant for child-level data Can state link outcome data be linked to
other information though an ID? Does it enter the system already linked?
Linkage to other data has major Linkage to other data has major implications for analysis and implications for analysis and questions state will be able to questions state will be able to answeranswer
![Page 40: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/40.jpg)
40
Training Training
Focused on various topics Training in assessment tools Training in use of the ECO Summary Form
Various approaches Various levels of investment ECO is developing materials and
compiling training materials for web site (including materials designed for parents)
Contact NECTAC or ECO for help
![Page 41: Approaches to Measuring Child Outcomes Kathy Hebbeler ECO at SRI International Prepared for the NECTAC National Meeting on Measuring Child and Family Outcomes,](https://reader035.fdocuments.net/reader035/viewer/2022062718/56649e865503460f94b88716/html5/thumbnails/41.jpg)
41
ConclusionsConclusions
States are building many different kinds of outcomes measurement systems
Features of the system reflect the contexts and values of the state
We know some things about what states are doing but the landscape keeps changing