Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot
description
Transcript of Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot
![Page 1: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/1.jpg)
Building and Deploying an Early Warning System: Lessons Learned from a
Large Scale PilotJared Knowles
Research AnalystWisconsin Department of Public
InstructionSTATS-DC July 2013
Washington, D.C.
![Page 2: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/2.jpg)
Agenda• Principles for a Dropout Early
Warning System• Building a Statewide DEWS• Piloting a DEWS• Learning from the Pilot• Deploying Statewide?
![Page 3: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/3.jpg)
Why DEWS?• Push early warning systems to be earlier• Focus on extracting information from data
already reported by LEAs to the SEA• Statewide scale, leveraging multiple years
of data in the LDS• Built on open-source software that can
adapt to additional data and different contexts
![Page 4: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/4.jpg)
Challenges with DEWS?• Resonance – school principals have to
see this fitting in their work• Ease of Use – minimal barriers to opening
up and using the system• Accuracy – cannot give bad information
to LEAs• Coverage– as many students included as
possible• Transparency – needs to be a system
LEAs trust
![Page 5: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/5.jpg)
DEWS Refresher• DEWS score calculated using a
combination of demographic and student outcome measures to improve accuracy– Attendance, disciplinary events, assessment
scores, and student mobility• Student risk is calculated individually for
each student• Students are classified as at risk if their
score crosses a threshold set by DPI; districts can use this or ignore it
![Page 6: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/6.jpg)
DEWS Refresher• DPI early warning system is called the
Dropout Early Warning System, or DEWS• DEWS provides a score from 0-100 for
current 6th, 7th, and 8th graders• The score represents the rate at which
students similar to the current student in previous cohorts graduated
• A score of 75 means that 75% of prior students with similar characteristics graduated on time
![Page 7: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/7.jpg)
DPI’s System is in Development
• More than 60% of students who eventually do not graduate after 4 years of high school can be identified with current data before the start of 7th grade
• DPI is working to improve this through better techniques to allow students to be identified earlier and with more accuracy
• The system will continually improve with better data, better mathematical models, and more real time results
![Page 8: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/8.jpg)
Classification
![Page 9: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/9.jpg)
Project Plan• DEWS was developed during the 2012-13
school year• Pilot group of 34 schools identified in early
2013• Pilot materials delivered electronically in
mid-April 2013; participation in follow-up survey too – Interpretative guide– Student reports for all current 7th graders– School report– School roster
• Pilot materials mimic WISEdash, final scheduled for September 2013 rollout in WISEdash
![Page 10: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/10.jpg)
Awareness and Communications
• Title I Coordinators• Accountability Trainers• Statewide PBIS
Network• CESA Support Network• SSEDAC• School Administrators
Alliance• School Counselors
Association• WERAC• National Forum on
Education Statistics
• REL Midwest• Partners at WCER• Department of Children
and Families• Members of
WISEexplore
![Page 11: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/11.jpg)
DEWS Process
DemographicsAttendance
Assessments
Disciplinary EventsMobilityLocation
STATE DATA
Student Risk Identification
Teacher / program context
Parent input
Special circumstances
CONTEXT
LOCAL KNOWLEDGE
Intervention Strategies
![Page 12: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/12.jpg)
Pilot Reports
![Page 13: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/13.jpg)
Pilot Reports - Student
![Page 14: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/14.jpg)
Pilot Reports - School
![Page 15: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/15.jpg)
Survey Results• Survey sought to identify the utility of the
DEWS reports in relation to existing Early Warning System / identification measures
Asked about:• usefulness of DEWS report• usefulness of interpretation guides• desire to have DEWS available• WISEdash usage• Likelihood to use WISEdash if DEWS
included
![Page 16: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/16.jpg)
Survey Summary• 18 of the 34 participating pilot schools
have responded to the survey so far (52.9%)
• 15 of the respondents indicated they “fully reviewed” the results
• 6 schools have been interviewed• 5 schools said staff reviewed the reports
individually, 11 said staff reviewed them in a group working together
![Page 17: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/17.jpg)
DEWS Overall Valuable!
0
2
4
6
Not valuable Slightly valuable Somewhat valuable Very valuableScale
Coun
t
Overall, how valuable were the DEWS reports to your school?
![Page 18: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/18.jpg)
DEWS Identifies Students Missed
0.0
2.5
5.0
7.5
10.0
0 25 or less 26 - 100 100 +Number of Students
Res
pond
ents
How many students did the DEWS reports identify... which your school had not previously identified...
![Page 19: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/19.jpg)
DEWS Does Not Miss Many Students
0.0
2.5
5.0
7.5
0 10 or less 10 + NANumber of Students
Resp
onde
nts
How many students has your school identified
which were not identified in the DEWS reports?
![Page 20: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/20.jpg)
Student reports are most positive element
0
2
4
6
8
Slightly Valuable Somewhat Valuable Very ValuableScale
Res
pons
esStudent Reports
![Page 21: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/21.jpg)
The Student Roster is Valuable
0
1
2
3
4
5
Slightly Valuable Somewhat Valuable Very ValuableScale
Res
pons
esStudent Roster
![Page 22: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/22.jpg)
School reports are well liked
0
2
4
6
Slightly Valuable Somewhat Valuable Very ValuableScale
Resp
onse
s
School Report
![Page 23: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/23.jpg)
Most respondents expect DEWS to be used at least
annually
0
3
6
9
No YesScale
Resp
onse
s
Annual Use?
![Page 24: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/24.jpg)
Annual Delivery Before School Year is Strongly Preferred
0.0
2.5
5.0
7.5
- Before school beginsScale
Res
pons
es
Report Timing
![Page 25: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/25.jpg)
DEWS can drive WISEdash usage
0.0
2.5
5.0
7.5
10.0
No YesScale
Resp
onse
s
Use WISEdash Regularly?
0.0
2.5
5.0
7.5
No YesScale
Resp
onse
s
Use WISEdash More?
![Page 26: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/26.jpg)
Principals and Student Services Staff Must Have WISEdash
access
0
5
10
1 NAScale
Resp
onse
s
Principals
0.0
2.5
5.0
7.5
10.0
12.5
1 NAScale
Resp
onse
s
Student Services
0
2
4
6
8
1 NAScale
Resp
onse
s
Teachers
0
5
10
1 NAScale
Resp
onse
s
Other
![Page 27: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/27.jpg)
DEWS Beyond Fall 2013DEWS as it exists is just a start.
Several extensions for DEWS may be desired:
• Deeper WISEdash integration?• Communication and professional
development to raise awareness and use for informing interventions?
• Extend coverage to earlier and later grades
• Increase accuracy?• Add college-enrollment as a secondary
warning?
![Page 28: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/28.jpg)
Interview Quotes• “Great to have a system that doesn’t require
additional reporting or information from us!”• “This would be a fantastic resource to be able to
share with parents in fall teacher conferences.”• “Inclusion of mobility in the system covers a big
blind spot for us as an LEA.”• “Thank you for getting feedback early in the
process from us!”• “Want to use this list as a starting point for our
intervention planning teams.”• “Having an 8th grade score for incoming 9th
graders will be crucial to planning!”
![Page 29: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/29.jpg)
Interview Quotes• “This information is easy to understand and quick
to be processed so we can hit the ground running with ours staff.”
• “Summarizing all this data in one place for us saves us time so we can get to identifying additional data needed to identify interventions.”
![Page 30: Building and Deploying an Early Warning System: Lessons Learned from a Large Scale Pilot](https://reader035.fdocuments.net/reader035/viewer/2022062323/568165a0550346895dd87c3a/html5/thumbnails/30.jpg)
Questions?Contact Me• E-mail: [email protected]• GitHub: www.github.com/jknowles• Twitter: @jknowles• Google + : profiles.google.com/jknowles• Web: www.jaredknowles.com