Outcomes and Evidence Statements
description
Transcript of Outcomes and Evidence Statements
Outcomes and Evidence Outcomes and Evidence Statements Statements
Outcomes and Evidence Outcomes and Evidence Statements Statements
Prepared for the Michigan Results Group Lansing, MI July 2005
Kathy HebbelerSRI International
2
Objectives for this presentationObjectives for this presentation
Review the ECO outcomes Introduce some concepts related to
evidence statements to assist Michigan in formulating its evidence statements
Share ECO’s recommendations for evidence statements
Describe what OSEP will be requiring Introduce the measurement approach
ECO is developing
3
Early Childhood Outcomes (ECO) Early Childhood Outcomes (ECO) Center: MissionCenter: Mission
Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in national and state accountability systems
4
ECO CenterECO Center
5-year project funded by OSEP in October 2003Collaboration among: SRI International Frank Porter Graham Child Development
Institute (U of North Carolina, Chapel Hill) Juniper Gardens Children’s Project (U of
Kansas) National Association of State Directors of
Special Education University of Connecticut
5
Public Policy ContextPublic Policy Context
Age of accountability Accountability increasingly means
looking at results – not just process Applies across all private and
public human service and education programs
6
President’s Commission on President’s Commission on Excellence in Special Education Excellence in Special Education (2002)(2002)
Major Recommendation 1: Major Recommendation 1: Focus on results – not processFocus on results – not process
“IDEA will only fulfill its intended purpose if it raises expectations for students and becomes result-oriented—not driven by process, litigation, regulation and confrontation. In short, the system must be judged by the opportunities it provides and the outcomes achieved for each child.”
7
Federal push for outcome data: Federal push for outcome data: Intro to AcronymsIntro to Acronyms
GPRA =GPRA =Government Performance Government Performance
and Results Actand Results Act PART = Program Assessment Rating PART = Program Assessment Rating
ToolTool OMB = Office of Management and OMB = Office of Management and
BudgetBudget IDEA = Individuals with Disabilities IDEA = Individuals with Disabilities
Education ActEducation Act
8
GPRA (Government Performance GPRA (Government Performance and Results Act)and Results Act)
IDEA goals and indicators established IDEA goals and indicators established Indicators and data collection further along for Indicators and data collection further along for
school age population than for ECschool age population than for EC For early childhood, data have been collected For early childhood, data have been collected
on: on:
• Number of children served (Part C)Number of children served (Part C)
• Settings (both Part C and 619)Settings (both Part C and 619) Note: Part C = Programs for 0-3 year olds; 619 = Programs for 3-5
9
PART (Program Assessment PART (Program Assessment Rating Tool)Rating Tool)
Tool used to review federal programsTool used to review federal programs Four critical Assessment Areas Four critical Assessment Areas
includingincluding
““results and accountability”results and accountability” Programs given ratings from ineffective Programs given ratings from ineffective
to effectiveto effective Purpose – “to enhance budget analysis”Purpose – “to enhance budget analysis”
10
PART Evaluation: Findings and PART Evaluation: Findings and RecommendationsRecommendations
Part C and 619 Findings: No long-term child outcome goals or data
The PART required the Department to: 1. Establish long-term outcome-oriented
objectives 2. Develop a strategy to collect performance
data Office of Special Education Programs (OSEP)
asked states to provide outcome data in their Annual Performance Reports (APRs)
11
Reality Check (aka Challenges)Reality Check (aka Challenges) Michigan will be required to submit data to
OSEP OSEP has recently released the indicators on
which it will be requiring data and the timeline is impossible
ECO is simultaneously working on many of the same issues the states are. We have a measurement approach in development but it is far from complete.
We have no choice but to live with the uncertainties and move forward as best we can
12
Steps in Development of an Steps in Development of an Outcomes SystemOutcomes System
1. Identify overall purpose for the system (including priorities)
2. Identify outcome areas for children and families
3. Develop outcome statements4. Formulate the evidence statements 5. Identify measurement approaches (e.g.,
select instruments) ***Steps 6 to 11 after that
13
Need to acknowledge the range of Need to acknowledge the range of emotional responses to outcomes emotional responses to outcomes systemssystems
Hate it, have to do it so I will Hate it, have to do it but I won’t do a very
good job of it because it is a waste of my time Neutral, just another fad that will pass Not sure, might be good but not likely and
probably will be useless to harmful Worried that this will take time away from
children and families Hopeful that this will live up to the potential Excited about the prospect of having data on
child outcomes
14
Identify outcomes areasIdentify outcomes areas
What are the important outcomes areas? Child Family
Everyone wants to know “How are children doing?” but need to decide “with regard to what?”
15
DefinitionsDefinitions
Outcome—a statement of a measurable condition(s) desired for the population of children or their families Children take appropriate action to
meet their needs Families know how to advocate for
the services they need
16
Identify outcome areasIdentify outcome areas
Outcome statements are the system’s overall vision for children and for families
Not the same as outcomes on an IFSP or IEP which are very specific outcomes for an individual child or family
Outcomes in an accountability system are global statements of what we are trying to do for children and families
17
Identify outcome areasIdentify outcome areas
How should we think about child outcomes? “…for more than three decades,
researchers and service providers have struggled with both the identification of significant child outcomes and their valid and reliable measurement.”
National Research Council and Institute of Medicine, 2000
18
Identify outcome areasIdentify outcome areas
How should we think about child outcomes? Domains (language, cognitive, adaptive,
etc.) Functional skills (feeding, dressing, mobility,
etc.) Underlying functional capacities (Self-
regulation, engagement, knowledge acquisition skills, etc.)
Pre-academic skills (Early reading, math, science, etc.)
19
Pitfall Alert!Pitfall Alert! What we want for children and families
(desired outcomes) reflects our values What we want for young children may not be
the same as what can be easily measured When what is valued and what can be
measured easily or well do not line up, EITHER: Outcomes will be determined by what can be
measured easily or well, OR Some of the outcomes will be more easily measured
or more adequately measured than others
20
ECO Family OutcomesECO Family Outcomes
1. Families know their rights and advocate effectively for their children.
2. Families understand their children's abilities and special needs
3. Families help their children develop and learn.
4. Families have social supports. 5. Families are able to gain access to desired
services, programs, and activities in their community.
21
Need for an overarching goal for Need for an overarching goal for childrenchildren
The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community
22
ECO Child OutcomesECO Child Outcomes
1. Children have positive social relationships
2. Children acquire and use knowledge and skills
3. Children take appropriate action to meet their needs
23
Issues from stakeholder Issues from stakeholder discussionsdiscussions
Functional outcomes Best practice – kind of outcomes
recommended for IFSPs and IEPs Consistent with transdisciplinary
service delivery Not reflected well in a pure domains
framework Not captured well in current
assessment tools
Children have positive social relationships
Children acquire and use knowledge and skills
Children take appropriate action to
meet their needs
Relation-ships with
adultsRelation-
ships with
peers
Follows group rules
Masters the environ-ment
Symbol use, abstract thinking
Appliesknowledge
Listening PlayExploring
Playing
Being curious
Practicing
Touching
Attending
EngagingPersisting
Knowledge of physical world & culture
Self-care, health and safety
Elaboration of the ECO OutcomesElaboration of the ECO Outcomes
Children will be active and successful participants now and in the future in a variety of settings.
ECO Outcomes:
Overarching Goal:
Children have positive social relationships
Children acquire and use knowledge and skills
Children take appropriate action to meet their needs
Domains: Communication Cognition
Motor
Social-Emotional
Approaches to Learning
Content Areas:
Language Arts Math
Science Social Studies Art
Music
Note: Each of these can be broken down further into sub-areas
Processes: Memory Self-regulation Recognizing and interpreting sensory input
Listening Attending
Body Functions: Vision Seeing
Movement-flexibility-strength-postural response
Etc.
Speech production Etc.
Self Help
Example of how movement can be sub-divided
Alternative Ways of Thinking about Child OutcomesAlternative Ways of Thinking about Child Outcomes
26
What is happening with these What is happening with these outcome areas and statements?outcome areas and statements?
ECO’s work is independent of OSEP OSEP has been involved in ECO’s work
from the beginning ECO submitted these outcomes to OSEP
as its recommendations ECO submitted recommended indicators
based on these outcomes to OSEP for the APR/SPP instructions that were available for public comment
27
Evidence Statements(or what do you want to know about those
outcomes)
28
The Importance of Evidence The Importance of Evidence Statements Statements
What constitutes evidence of good outcomes?
If you were asked to testify before your state legislature to demonstrate the effectiveness of EI, what kind of evidence do you want to have? (Purpose = accountability)
What kind of evidence about outcomes is needed by states and local programs to improve services for children and families? (Purpose = program improvement)
29
Definition of Evidence StatementDefinition of Evidence Statement
Evidence Statement - a statement that incorporates a statistic and provides evidence as to whether not an outcome has been achieved
% of children who have positive social relationships
% of children who show progress toward acquiring knowledge and skills
Note: States will be asked to submit a specified kind of evidence to OSEP. Each state will need to be able to produce this kind of evidence but might want other kinds of evidence as well.
30
Features of a Good Evidence Features of a Good Evidence StatementStatement
Credible: Based on valid data Meaningful: The evidence can be
interpreted Powerful: The evidence is convincing
to those who will be receiving it (purpose = accountability)
Useful: The evidence is helpful to those who will use it (purpose = program improvement)
31
MeasurementMeasurement Evidence Evidence Statements Statements
Evidence statements are the ultimate product in an outcomes-based accountability system
The nature of the evidence statement that can be produced will depend on what is measured and how often
32
Possible Categories of Evidence Possible Categories of Evidence Statements for Outcome DataStatements for Outcome Data
1. Status (achievement at one point in time)
Example: “% of children who [achieved this..]
2. Progress (change relative to earlier status)
Example: “% of children who made progress in..]
33
Two Ways to Think about Two Ways to Think about Progress with Young ChildrenProgress with Young Children
Continuous Progress = acquisition of new skills and behaviors (almost all children show this)
Or Age-anchored Progress = change in
developmental trajectory (i.e., narrowing the gap between a child with a delay and typical development; “less delayed” after intervention)
34
Examples of Two Kinds of Examples of Two Kinds of ProgressProgress
Continuous– increase in vocabulary Age-anchored– change from being
1.5 standard deviations below norm to .5 standard deviation
Hypothetical Language Acquisition Rates for Three Groups of Children: Change in
Developmental Trajectory (Progress toward Closing the Gap)
0
10
20
30
40
50
60
70
1 6 11 16 21 26 31 36 41 46 51 56Age in Months
La
ng
au
ge
Sc
ore
Typically developing children
Children with delays without intervention
Children with delays after intervention
36
Advantages and Disadvantages of Different Evidence Statements
37
Status (where children are)Status (where children are)
Reports the percentage of children who achieved or could do X
Would likely involve some kind of comparison to expectations for children of a given age
Examples: 53% of early intervention graduates were
rated as “typical and proficient” with regard to having positive social relationships
38
Status Evidence StatementStatus Evidence Statement
Pro Requires one measurement point Could be strong evidence Nearly all K-12 evidence statements are of this
form (grad. rates, “proficient” reading)Con What is the benchmark that makes sense for
all children with disabilities? How does one interpret the data (is this finding
good news or bad news?)
39
Continuous Progress (change Continuous Progress (change compared to earlier status)compared to earlier status)
Reports % of children who improved Progress on a curriculum-based
assessmentExamples:
% of preschool children who made progress toward having positive social relationships
% of preschool children who showed an increase of 3 of more objectives related to positive social relationships scale after one year
40
Continuous ProgressContinuous Progress
Pro Straightforward Could quantify extent of child growth Incorporates all levels of gain, even the very
small gains made by children with severe impairments
Con Requires at least 2 points of data Nearly all children will improve over time Weak evidence. What constitutes “good
news”?
41
Age-Anchored Progress (closing the Age-Anchored Progress (closing the gap toward typical development)gap toward typical development)
Captures progress relative to same age peers
Examples: 23% of children moved from low performing
to age appropriate in positive social relationships by kindergarten entry
With regard to acquiring and using knowledge and skills, 75% of children made progress sufficient to maintain their functioning at an age appropriate level
42
Age-Anchored Progress (closing the Age-Anchored Progress (closing the gap toward typical development)gap toward typical development)
Pro Consistent with the intent of intervening Could produce strong evidence
Con Requires at least 2 time points of data Closing the gap is not an expectation for all
children with disabilities. How is the evidence to be interpreted?
If some children don’t close the gap, is this interpreted as lack of success for the program?
43
Deciding on desired evidenceDeciding on desired evidence
Not an either-or Different levels will want different
kinds of evidence Many different kinds of evidence
can be produced from the same set of data If the desired evidence statements
are identified as part of the planning
44
What OSEP will be requiring from states for Part C and Preschool
Special Education
45
Reporting Requirement: Part CReporting Requirement: Part C
Percent of infants and toddlers with IFSPs who demonstrate improved:
A. Positive social-emotional skills (including social relationships)
B. Acquisition and use of knowledge and skills (including early language/communication)
C. Use of appropriate behaviors to meet their needs
46
The detailsThe details
a. % of infants and toddlers who reach or maintain functioning at a level comparable to same-aged peers
b. % of infants and toddlers who improve functioning
c. % of infants and toddlers who did not improve functioning.
If children meet the criteria for a, report them in a. Do not include children reported in a in b or c.
Illustration of 5 Possible Paths
0
10
20
30
40
50
60
70
1 6 11 16 21 26 31 36 41 46 51 56Age in Months
Sco
re
Maintained functioning comparable to age peers
Achieved functioning comparable to age peers
Moved nearer functioning comparable to age peersMade progress; no change in trajectory
Did not make progress
48
3 OSEP Evidence Statements3 OSEP Evidence Statements
OSEP a Maintained functioning comparable to age
peers Achieved functioning comparable to age peersOSEP b Moved nearer functioning comparable to age
peers Made progress; no change in trajectoryOSEP c Did not make progress
49
Two kinds of outcomes questions for Two kinds of outcomes questions for each child with a disabilityeach child with a disability
1. How is this child doing relative to the individualized outcomes that were established through the IFSP or IEP process?
2. How is this child doing relative to same-aged peers?
50
Part C Indicators Part C Indicators Family OutcomesFamily Outcomes
Percent of families participating in Part C who report that early intervention services have helped the family Know their rights; Effectively communicate their
children’s needs; and Help their children develop and learn.
51
Measurement guidelinesMeasurement guidelines
State selected data source, sampling permitted
% of families who report that early intervention services have helped their family in each of the three areas
52
SPP/APRSPP/APRMeasurement implications Measurement implications Requires data collection from family
members/caregivers Data at one time period
Asks families to recognize that early intervention services contributed to the change or status listed.
Does not require family data to be linked with child data
53
Key Questions Key Questions
How can Michigan provide the indicators being requested by OSEP?
What kinds of evidence statements does Michigan want to be able to produce for its own use?
What do the answers mean for measurement?
54
Producing Data on the Outcomes
55
Rolling up assessment data to Rolling up assessment data to the outcomesthe outcomes
Given: Different children will have different kinds of
assessment data. Also:
The same child may have several different kinds of assessment data.
Challenge: How to roll up multiple pieces of information (e.g., data from different assessment tools) to a single score?
56
Children will be active and successful participants now and in the future in a variety of settings.
ECO Outcomes:
Overarching Goal:
Children have positive social relationships
Children acquire and use knowledge and skills
Children take appropriate action to meet their needs
Domains: Communication Cognition
Motor
Social-Emotional
Approaches to Learning
Content Areas:
Language Arts Math
Science Social Studies Art
Music
Note: Each of these can be broken down further into sub-areas
Processes: Memory Self-regulation Recognizing and interpreting sensory input
Listening Attending
Body Functions: Vision Seeing
Movement-flexibility-strength-postural response
Etc.
Speech production Etc.
Self Help
Example of how movement can be sub-divided
Different assessments on different childrenDifferent assessments on different children
Roll up Process
Assessment 1
Assessment 2
Assessment 3
Assessment 4
Single Score
Roll up Process
Assessment 1
Assessment 2
Parent Report
Informed Clinical Opinion
Single Score
Different pieces of information on the same childDifferent pieces of information on the same child
59
Rolling up from the different Rolling up from the different assessmentsassessments
ECO is working on a scale that will provide a way to give a child a numerical rating on each outcome It is NOT an assessment tool. It assumes teachers and providers are
already using an assessment tool to collect information about how children are doing.
The scale is a way to take what those closest to the child know and put a number on it.
60
What does the outcomes scale What does the outcomes scale look like?look like?
5 pt and a 7 pt version The precision question….
The high point (5 or 7) indicates outcome achieved at an age-expected level
The lowest point (1) indicates the farthest distance from age-expectations
61
What does the outcomes scale What does the outcomes scale look like?look like?
Challenge is in how to anchor and provide the descriptors for the middle points on the scale.
Progress is moving up the scale between time 1 and time 2.
62
Measuring the Family OutcomesMeasuring the Family Outcomes
ECO working on a tool ECO working on a document with
guidance for states that reviews key decisions and includes several options
63
Staying in touchStaying in touch
Web site: the-eco-center.org Follow developments related to work of the Center Obtain ECO resource documents Obtain other related resources
Email: [email protected] ECO sends to existing listservs Can join the ECO mailing list
64
VisionVision
To build a system where all the pieces will fit together smoothly
Meaningful outcomes data collected regularly that can used in programs by programs AND reported to the state (and feds)
Local data
State needs
68
When Michigan can realize this vision, children and families will
reap the benefits.