Post on 17-Dec-2015
Christina KasprzakECTA/ECO/DaSy
Lauren BartonECO/DaSy
Ruth Chvojicek WI Statewide Part B Indicator 7 Child Outcomes Coordinator
September 16, 2013Improving Data, Improving Outcomes Conference - Washington, DC
Where the Rubber Hits the Road:
Tools and Strategies for Using
Child Outcomes Data for
Program Improvement
Purposes
• To describe national resources for promoting data quality and supporting program improvement
• To share Wisconsin 619 experience and strategies to promote data quality and program improvement
• To discuss potential approaches for examining data quality and using data in your state 2
Quality Assurance: Looking for Quality Data
I know it is in here somewhere
4
Do Ratings Accurately Reflect Child Status?
Pattern Checking
• We have expectations about how child outcomes data should look– Compared to what we expect– Compared to other data in the state– Compared to similar states/regions/school
districts• When the data are different than
expected ask follow up questions
Questions to Ask
• Do the data make sense?– Am I surprised? Do I believe the data?
Believe some of the data? All of the data?
• If the data are reasonable (or when they become reasonable), what might they tell us?
5
Pattern Checking for Data Quality
Strategies for using data analysis to improve the quality of state data by looking for patterns that indicate potential issues for further investigation.
http://ectacenter.org/~pdfs/eco/pattern_checking_table.pdf
7
Predicted Pattern
3b. Large changes in status relative to same age peers between entry and exit from the program are possible, but rare.
Most children served in EI and ECSE will maintain or improve their rate of growth in the three child outcomes areas over time given participation in intervention activities that promote skill development.
8
Rationale
Analysis
1. Crosstabs between entry and exit ratings for each outcome, best for COS ratings.
2. Exit minus Entry numbers.
For COS ratings we would expect most
cases to increase by no more than 3 points.
Question: Is the distribution sensible?
9
Entry
Exit 1 2 3 4 5 6 7 total
1 1 4 2 7
2 1 1 5 6 9 3 1 26
3 2 15 14 27 19 6 83
4 4 4 21 39 28 12 108
5 1 12 14 71 86 48 232
6 1 3 21 48 63 136
7 2 18 23 56 99
Review Total 2 13 38 60 185 207 186 691
Outcome 3: Crosstabs Between Entry and Exit Ratings
Outcome 1: Children that increased by 4 or more points
from entry to exit
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 State0%
5%
10%
15%
20%
25%
30%
35%
% children who increased 4 or more points from entry to exit by district with 30 or more children (N=25)
Analyzing Child Outcomes Data for Program Improvement
• Quick reference tool• Consider key issues,
questions, and approaches for analyzing and interpreting child outcomes data.
http://www.ectacenter.org/~pdfs/eco/AnalyzingChildOutcomesData-GuidanceTable.pdf
Steps in Using Data for Program Improvement
Defining Analysis Questions
Step 1. What are your crucial policy and programmatic questions?
Step 2. What is already known about the question?
Clarifying Expectations
Step 3. Describe expected relationships with child outcomes.
Step 4. What analysis will provide information about the relationships? Do you have the necessary data for that?
Step 5. Provide more detail about what you expect to see. With that analysis, how would data showing the expected relationships look?
13
Steps in Using Data for Program Improvement
Analyzing Data
Step 6. Run the analysis and format the data for review.
Testing Inferences
Step 7. Describe the results. Begin to interpret the results. Stakeholders offer inferences based on the data.
Step 8. Conduct follow-up analysis. Format the data for review.
Step 9. Describe and interpret the new results as in step 7. Repeat cycle as needed.
Data-Based Program Improvement Planning
Step 10. Discuss/plan appropriate actions based on the inference(s).
Step 11. Implement and evaluate impact of the action plan. Revisit crucial questions in Step 1.
Defining Analysis Questions
What are your crucial policy and programmatic questions?
Example:
1. Does our program serve some children more effectively than others?
a. Do children with different racial/ethnic backgrounds have similar outcomes?
15
What do you expect to see?
Do you expect children with racial/ethnic backgrounds will have similar outcomes? Why? Why not?
16
Clarifying Expectations
1. Compare outcomes for children in different subgroups:
a. Different child ethnicities/races (e.g. for each outcome examine if there are higher summary statements, progress categories, entry and/or exit ratings for children of different racial/ethnic groups).
17
Analyzing Data
Outcome 1: Summary Statements by Child’s
Race/Ethnicity
18
Summary Statement 1 Greater Than Expected
Growth
Summary Statement 2 Exit at Age Expectations
0
10
20
30
40
50
60
70
80
90
100
68
61
6764
74
69
6259
57
51
72
63
NationalStatewide (4824)Caucasian (2496)Hispanic/Latino (1018)African-American (1134)Multiple/Other (176)
Perc
enta
ge o
f Chi
ldre
n
Outcome 1: Progress Categories by Child’s Race/Ethnicity
19a - no progress b - progress
compared to selfc - narrowed the gap d - closed the gap e - maintained
0
5
10
15
20
25
30
35
40
CaucasianHispanic/LatinoAfrican AmericanMultiple/Other
• Is the evidence what you expected? • What is the inference or interpretation?• What might be the action?
20
Describing and Interpreting Results
Guidance Table
21
USING DATA FOR STATE & LOCAL IMPROVEMENT WISCONSIN’S PART B
Ruth Chvojicek – WI Statewide Part B Indicator 7 Child Outcomes Coordinator
KEY POINTS ABOUT WISCONSIN’S SYSTEM
Sampling strategy until July 1, 2011 Part B Child Outcomes Coordinator position funded
through preschool discretionary funds – focus on training and data
Statewide T/TA system with district support through 12 Cooperative Educational Service Agency’s – Program Support Teachers
1 2 3 4 5 6 7
Outcome 1 0.036392615221145
0.108115287554788
0.0931066542701554
0.138265373887635
0.213972639128702
0.208527028821889
0.201620401115686
Outcome 2 0.0468853765440298
0.141984327267898
0.16323548944083
0.197104529153938
0.277593305883916
0.141718687740736
0.0314782839686546
Outcome 3 0.0332049408952053
0.0753088059503256
0.0848718289281448
0.11475627573383
0.17293133218223
0.211980342674991
0.306946473635278
2.5%
7.5%
12.5%
17.5%
22.5%
27.5%
32.5%
State 11-12 Entry Rating Distribution
GERMANTOWN SCHOOL DISTRICT – LESSON’S LEARNED Jenni Last – Speech Language Pathologist
Lisa Bartolone
School Pyschologist
RESULT OF GERMANTOWN’S WORK IN JUST 2 YEARS
1 2 3 4 5 6 7
2011-2012
0.0263157894736842
0.105263157894737
0.0526315789473684
0.184210526315789
0.263157894736842
0.157894736842105
0.210526315789474
2012-2013
NaN NaN 0.0769230769230769
0.269230769230769
0.576923076923077
0.0769230769230769
NaN
5.0%
15.0%
25.0%
35.0%
45.0%
55.0%
65.0%
Germantown Outcome OneEntry Rating Comparison
GERMANTOWN – OUTCOME TWO123456
2011-2012
0.0263157894736842
0.0789473684210526
0.0526315789473684
0.394736842105263
0.394736842105263
0.0526315789473684
2012-2013
2.5%
7.5%
12.5%
Germantown Outcome TwoEntry Rating Comparison
GERMANTOWN – OUTCOME THREE
1 2 3 4 5 6 7
2011-2012 0.0263157894736842
0.0526315789473684
0.0526315789473684
0.105263157894737
0.157894736842105
0.184210526315789
0.421052631578947
2012-2013 NaN NaN 0.0384615384615385
0.269230769230769
0.346153846153846
0.269230769230769
0.0769230769230769
2.5%
7.5%
12.5%
17.5%
22.5%
27.5%
32.5%
37.5%
42.5%
Germantown Outcome ThreeEntry Rating Comparison
STATE PROGRESS IN TWO YEARS – OUTCOME ONE
1 2 3 4 5 6 7
11-12 Out-come 1
0.0400000000000001
0.11 0.0900000000000001
0.14 0.21 0.21 0.2
12-13 Out-come 1
0.0336343410548358
0.107577542206518
0.120926580290538
0.166208611438294
0.246957204554378
0.193691925140688
0.131003795314749
3%
8%
13%
18%
23%
28%
Outcome 1 Entry Rating Comparison
STATE PROGRESS – OUTCOME TWO
1 2 3 4 5 6 7
11-12 Out-come 2
0.05 0.14 0.16 0.2 0.28 0.14 0.03
12-13 Out-come 2
0.0443717277486912
0.149083769633508
0.182984293193718
0.23717277486911
0.270418848167539
0.0973821989528796
0.018586387434555
3%
8%
13%
18%
23%
28%
Outcome 2 Entry Rating Comparison
STATE PROGRESS – OUTCOME THREE
1 2 3 4 5 6 7
11-12 Out-come 3
0.03 0.08 0.08 0.11 0.17 0.21 0.310000000000001
12-13 Out-come 3
0.0261814373609112
0.0747480036654013
0.100798533839508
0.142165204869747
0.233145699698913
0.215604136667103
0.207356983898416
3%
8%
13%
18%
23%
28%
33%
Outcome 3 Entry Rating Comparison
BUT … OUTCOME ONE EXIT RATING
1 2 3 4 5 6 7
11-12 Out-come 1 Exit Rating
0.01 0.032 0.039 0.07 0.158 0.253 0.438000000000001
12-13 Out-come 1 Exit Rating
0.00668073136427568
0.019338959212377
0.0365682137834038
0.0618846694796062
0.146272855133615
0.284810126582279
0.444444444444444
2.5%
7.5%
12.5%
17.5%
22.5%
27.5%
32.5%
37.5%
42.5%
47.5%
Outcome 1 Exit Rating Comparison
OUTCOME THREE
1 2 3 4 5 6 7
11-12 Out-come 3 Exit Rating
0.013 0.025 0.03 0.039 0.09 0.238 0.564
12-13 Out-come 3 Exit Rating
0.00597959901512491
0.0140696447414703
0.0235666549419626
0.0425606753429478
0.0960253253605351
0.236721772775237
0.58107632782272
5.0%
15.0%
25.0%
35.0%
45.0%
55.0%
65.0%
Outcome 3 Exit RatingComparison
WISCONSIN PART B DATA REVIEWS
11-12 – Piloted process individually with 20 districts Discovered differences in how districts were determining
eligibility S/L and SDD Two districts who used criterion referenced tool
consistently AND provided PD on using tool showed more appropriate pattern than other 18 districts
Next steps identified by districts: Mentoring and pd for new staff More attention to formative assessment process Work on internal data tracking system
WISCONSIN PART B 12-13 DATA REVIEW
Looked at 8 data patterns including: Entry Rating Distribution Entry Rating Distribution by Disability* Comparison Entry Ratings by Outcome Exit Rating Distribution Entry / Exit Comparison* Race/Ethnicity Comparison* State Progress Categories* Summary Statements*
LOOKING AT RACE/ETHNICITY
Outcome 1 1 2 3 4 5 6 7Asian 4.3% 18.4% 9.9% 9.2% 23.4% 15.6% 19.1%Black 6.4% 15.3% 11.3% 16.7% 20.1% 15.2% 15.2%Hispanic 4.9% 11.9% 12.1% 13.5% 21.5% 21.7% 14.5%American Indian Alaskan 2.7% 10.0% 15.5% 19.1% 24.5% 17.3% 10.9%Hawaiian Other Pacific Islander 0.0% 31.6% 21.1% 10.5% 31.6% 0.0% 5.3%Two or More Races 4.5% 7.2% 11.7% 18.9% 26.1% 18.9% 12.6%White 2.8% 9.5% 8.1% 13.2% 21.4% 22.2% 22.8%
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Outcome 1 State 11-12 Entry
State BlackState HispanicState White
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Outcome 1 CESA 1
CESA 1 BlackCESA 1 HispanicCESA 1 White
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
Outcome 1 CESA 2
CESA 2 BlackCESA 2 HispanicCESA 2 White
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
Outcome 1 District M 11-12 Entry
District M BlackDistrict M HispanicDistrict M White
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
Outcome 1 District B 11-12 Entry
District B BlackDistrict B HispanicDistrict B White
TRYING OUT THE NEW TOOL - DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?
1 2 3
State Target 0.796 0.825 0.824
State n = 2830 0.782 0.798 0.791
Speech Language n = 2062 0.801 0.827 0.833
SDD n = 341 0.793 0.752 0.783
Autism n = 133 0.702 0.721 0.733
OHI n = 142 0.767 0.721 0.727
5.00%
15.00%
25.00%
35.00%
45.00%
55.00%
65.00%
75.00%
85.00%
Summary Statement One 12-13 Data
DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?
1 2 3
State Target 0.701 0.703 0.806
State n = 2830 0.73 0.612 0.818
Speech Language n = 2062 0.848 0.693 0.914
SDD n = 341 0.528 0.437 0.674
Autism n = 133 0.173 0.278 0.368
OHI n = 142 0.394 0.387 0.546
5.00%
15.00%
25.00%
35.00%
45.00%
55.00%
65.00%
75.00%
85.00%
95.00%
Summary Statement Two 12-13 Data
DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?
A B C D E
State n = 2830 0.007 0.114 0.15 0.282 0.448
Speech Language n = 2062 0.005 0.078 0.069 0.266 0.582
SDD n = 341 0.015 0.173 0.284 0.434 0.094
Autism n = 133 0 0.293 0.534 0.158 0.015
OHI n = 142 0.014 0.197 0.394 0.303 0.092
5.00%
15.00%
25.00%
35.00%
45.00%
55.00%
65.00%
Outcome One 12-13 Progress
DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?
A B C D E
State n = 2830 0.006 0.153 0.228 0.4 0.213
Speech Language n = 2062 0.003 0.127 0.177 0.447 0.246
SDD n = 341 0.015 0.211 0.337 0.349 0.088
Autism n = 133 0 0.256 0.466 0.195 0.083
OHI n = 142 0.028 0.211 0.373 0.246 0.141
2.50%
7.50%
12.50%
17.50%
22.50%
27.50%
32.50%
37.50%
42.50%
47.50%
Outcome Two12-13 Progress
DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?
A B C D E
State n = 2830 0.008 0.082 0.092 0.25 0.568
Speech Language n = 2062 0.005 0.042 0.038 0.199 0.715
SDD n = 341 0.018 0.155 0.152 0.472 0.202
Autism n = 133 0.015 0.218 0.398 0.241 0.128
OHI n = 142 0.035 0.199 0.22 0.404 0.142
5.00%
15.00%
25.00%
35.00%
45.00%
55.00%
65.00%
75.00%
Outcome Three 12-13 Progress
WISCONSIN NEXT STEPS
Looking at the data – does the type of setting impact the progress children make? (District level analysis)
As a state T&TA system, we’re operating as a PLC to guide the work and support the District What will the Districts want to focus on? E.g. settings,
race/ethnicity, curriculum use
Local Contributing Factors Tool
Provides ideas for the types of questions a local team would consider in identifying factors impacting performance.
http://www.ectacenter.org/~meetings/outcomes2012/Uploads/ECO-C3-B7-LCFT_DRAFT-10-19-2012.docx
Relationship of Quality Practices to Child and Family Outcome
Measurement Results
Designed to assist states in identifying ways to improve results for children and families through implementation of quality practices.
51
http://ectacenter.org/~docs/eco/QualityPracticesOutcomes_4-29-11-Final.doc
Next Steps?
• Try out using these resources• Send feedback to ECO Center about the
new Analysis tool • What are your ‘take aways’ and next
steps related to analyzing your data for data quality and/or program improvement? (notes for State Team time)
52
53
Find more resources at: http://www. the-eco-center-
org