Effectively Utilizing Data Collection: A Case Study of a Functioning System
description
Transcript of Effectively Utilizing Data Collection: A Case Study of a Functioning System
Effectively Utilizing Effectively Utilizing Data Collection: A Case Data Collection: A Case Study of a Functioning Study of a Functioning
SystemSystem
Robert BarteltRobert Bartelt&&
Kristen GayKristen Gay
Silver Springs – Martin Luther Silver Springs – Martin Luther SchoolSchool
Where to StartWhere to Start
Agency ValuesAgency Values Frame QuestionsFrame Questions Work with OthersWork with Others
LeadersLeaders Direct Care StaffDirect Care Staff Families and ChildrenFamilies and Children
PilotPilot Improve
Assess
Measure
Design
Data CollectionData Collection
What you get out of yourWhat you get out of your
data is only asdata is only asgood as what you put good as what you put
in….in….
Data Collection ProcessData Collection Process
Choosing What Data to CollectChoosing What Data to Collect
Clinically MeaningfulClinically Meaningful
Does the data add to your understanding of Does the data add to your understanding of
how clients are doing? how clients are doing?
Will your findings help you effect change in Will your findings help you effect change in
your program?your program?
Data Collection ProcessData Collection Process
Choosing What Data to CollectChoosing What Data to Collect
User Friendly for StaffUser Friendly for Staff
Who is going to be collecting the data?Who is going to be collecting the data?
Are measures easy to understand?Are measures easy to understand?
How much training is required?How much training is required?
How many measures do you need?How many measures do you need?
How much time is available for data collection?How much time is available for data collection?
Choosing What Data to CollectChoosing What Data to Collect
““Crunchable”Crunchable”
Does the measure provide information that is Does the measure provide information that is meaningful for individual analysis?meaningful for individual analysis?
Does the measure provide information that is Does the measure provide information that is meaningful for aggregate analysis?meaningful for aggregate analysis?
Do you have/how will you get the tools to manage Do you have/how will you get the tools to manage the data?the data?
Data Collection ProcessData Collection Process
Choosing What Data to CollectChoosing What Data to Collect
Information Required from Third PartiesInformation Required from Third Parties
What are the requirements from third What are the requirements from third parties?parties?
Can your data collection system be set up so Can your data collection system be set up so as to satisfy these requirements?as to satisfy these requirements?
How will the data represent your agency and How will the data represent your agency and services?services?
Data Collection ProcessData Collection Process
Choosing What Data to CollectChoosing What Data to Collect
Is the data collection system flexible?Is the data collection system flexible?
When a managed care company or a county When a managed care company or a county calls to request data on only “their” children, calls to request data on only “their” children, can you do that easily?can you do that easily?
Can you respond to new requirements without Can you respond to new requirements without expending a lot of additional resources? expending a lot of additional resources?
Data Collection ProcessData Collection Process
Data Collection ProcessData Collection Process
Choosing What Data to CollectChoosing What Data to Collect
Do you want to engage in benchmarking?Do you want to engage in benchmarking?
Other similar organizationsOther similar organizations MCO expectationsMCO expectations County expectationsCounty expectations National normsNational norms
Examples of Data to CollectExamples of Data to Collect
Behavioral Rating ScalesBehavioral Rating Scales
Frequencies/Counts (i.e., Lengths of Stay)Frequencies/Counts (i.e., Lengths of Stay)
Patterns of Diagnostic Groups/Presenting ProblemsPatterns of Diagnostic Groups/Presenting Problems
Demographics of PopulationDemographics of Population
Contacts with FamiliesContacts with Families
Critical IncidentsCritical Incidents
Satisfaction SurveysSatisfaction Surveys
Aftercare DataAftercare Data
County, MCO, etc. RequestsCounty, MCO, etc. Requests
Evolution of Data CollectionEvolution of Data Collection As the agency and/or service provided As the agency and/or service provided
becomes more sophisticated, you want to becomes more sophisticated, you want to know more information.know more information.
Start collecting dataStart collecting data
Ask new questionsAsk new questions
What other data will What other data will you want to collect?you want to collect?
Do you have the flexibility to ask more Do you have the flexibility to ask more questions and gather more data?questions and gather more data?
Improve
Assess
Measure
Design
A Look at a Specific A Look at a Specific OutcomeOutcome
RestraintsRestraints A high-risk activityA high-risk activity Upfront goal to reduce the use of restraintsUpfront goal to reduce the use of restraints
What data is needed by internal committees:What data is needed by internal committees: How do you provide the most meaningful data to How do you provide the most meaningful data to
internal committees?internal committees?
How frequently will the data be required?How frequently will the data be required?
What variables are important?What variables are important?
Restraint Reduction Restraint Reduction
Collection of DataCollection of Data What do you want to know?What do you want to know?
Frequency of RestraintsFrequency of Restraints ChildChild Staff Who InitiatesStaff Who Initiates Time of DayTime of Day Length of RestraintLength of Restraint Result (injury of child, injury of staff)Result (injury of child, injury of staff) Children with multiple restraintsChildren with multiple restraints Type of restraint usedType of restraint used
Restraint ReductionRestraint Reduction
Flowchart: Developing a data collection Flowchart: Developing a data collection systemsystem
Data InputData Input
Data InputData Input
Data InputData InputList Manager
Data InputData Input
Data InputData Input
Data InputData Input
Data InputData Input
Data InputData Input
Data InputData Input
Data Input Data Input
Data OutputData Output
Data OutputData Output
Restraint Data
Restraint ReductionRestraint Reduction Different Types of ReportsDifferent Types of Reports
Trends Over TimeTrends Over Time
Restraints - January 2009 through November 2009
0
5
10
15
20
25
30
Jan-09 Feb-09 Mar-09 Apr-09 May-09 Jun-09 Jul-09 Aug-09 Sep-09 Oct-09 Nov-09
Nu
mb
er o
f R
estr
ain
ts
Unit A Unit B Unit C Unit D
Restraint ReductionRestraint Reduction
Different Types of ReportsDifferent Types of Reports By UnitBy Unit
Unit A Unit B Unit C Unit D
January 2 10 2 1
February 9 8 8 2
March 27 15 8 1
April 28 7 7 2
May 16 6 18 1
June 13 10 11 1
July 4 9 11 1
August 14 9 7 0
September 15 9 9 1
October 9 8 6 1
November 6 3 7 1
Different Types of ReportsDifferent Types of Reports Restraints by Type of RestraintRestraints by Type of Restraint
Restraint ReductionRestraint Reduction
Number of Restraints - All Types
0
10
20
30
40
50
60
Jan-
09
Feb
-09
Mar
-09
Apr
-09
May
-09
Jun-
09
Jul-0
9
Aug
-09
Sep
-09
Oct
-09
Nov
-09
Nu
mb
er
of
Re
str
ain
ts
Total Number Physical Restraints Prone Restraints Supine Restraints
Seated Restraints Standing Restraints Moving
Restraint ReductionRestraint Reduction Different Types of ReportsDifferent Types of Reports
Restraints by ChildRestraints by Child
Pareto Chart of Restraints
11%
66%
21%
30%
37%45%
63%60%56%
51%
0
50
100
150
200
250
Dav
id
Gle
nn
Am
ir
Des
tiny
And
reis
Ant
oine
Sha
rif
Kha
leem
Jahl
eel
Cas
sidy
Nu
mb
er o
f R
estr
ain
ts
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Per
cen
tag
e o
f T
ota
l R
estr
ain
ts
Restraint ReductionRestraint Reduction Different Types of ReportsDifferent Types of Reports
Restraints and InjuryRestraints and Injury
Number of Restraints and Child Injury from Restraints
0
10
20
30
40
50
60
Jul-
08
Au
g-0
8
Sep
-08
Oct
-08
No
v-08
Dec
-08
Jan
-09
Feb
-09
Mar
-09
Ap
r-09
May
-09
Jun
-09
Jul-
09
Au
g-0
9
Sep
-09
Restraints Child Injury
Restraint ReductionRestraint Reduction Different Types of ReportsDifferent Types of Reports
Children with Frequent RestraintsChildren with Frequent Restraints
Restraints D. S. - Unit A
0
2
4
6
8
10
12
Sep
-08
Oct
-08
Nov
-08
Dec
-08
Jan-
09
Feb
-09
Mar
-09
Apr
-09
May
-09
Jun-
09
Jul-0
9
Aug
-09
Sep
-09
Oct
-09
Nov
-09
RT MLS
Reporting to Third PartiesReporting to Third Parties HCSISHCSIS reporting reporting
Reporting to Third PartiesReporting to Third Parties•Reports to MCOsReports to MCOs
Reporting to Third PartiesReporting to Third Parties Reports to MCOsReports to MCOs
Reporting to Third PartiesReporting to Third Parties Reports to MCOsReports to MCOs
Reporting to Third PartiesReporting to Third Parties Reports to MCOsReports to MCOs
Responding to Requests by Responding to Requests by FundersFunders
Dilemma:Dilemma: Multiple expectations for reportingMultiple expectations for reporting Various requirements re data structureVarious requirements re data structure
Example:Example: Specific request from DHS to produce Specific request from DHS to produce
structured case notesstructured case notes
Structured Structured Case Note:Case Note:A Request A Request
by DHSby DHS
Measuring Symptom Reduction: An Measuring Symptom Reduction: An Example of the Use of Individual vs. Example of the Use of Individual vs. Aggregate DataAggregate Data
Youth Outcomes Questionnaire Youth Outcomes Questionnaire (YOQ)(YOQ)
64-item standardized measure64-item standardized measure
Total Score and Six SubscalesTotal Score and Six Subscales Intrapersonal DistressIntrapersonal Distress SomaticSomatic Interpersonal RelationsInterpersonal Relations Social ProblemsSocial Problems Behavioral DysfunctionBehavioral Dysfunction Critical ItemsCritical Items
Measuring Measuring Symptom Symptom ReductionReduction
Youth Outcomes
Questionnaire
(YOQ)
Individual Results
Measuring Symptom Measuring Symptom ReductionReduction
Y-OQ Total Score Over Time NM
65
91
106
86
67
65
0
20
40
60
80
100
120
Mar-08
Jun-08
Jul-08
Oct-08
Feb-09
Jun-09
To
tal
Sco
re
Inpatient
Outpatient
Community
Individual Results
Measuring Symptom Measuring Symptom Reduction: Reduction:
A Look at One Child - Two A Look at One Child - Two MeasuresMeasures
YOQ Scores and Number of Restraints for A.C.
0
20
40
60
80
100
120
May
-08
Jun-
08
Jul-0
8
Aug
-08
Sep
-08
Oct
-08
Nov
-08
Dec
-08
Jan-
08
Feb
-08
Mar
-08
Apr
-09
May
-09
Jun-
09
Jul-0
9
Aug
-09
Sep
-09
Oct
-09
Nov
-09
Dec
-09
Month
YO
Q S
core
s
0
1
2
3
4
5
Nu
mb
er of R
estraints
YOQ Restraints
YOQ Community Range
YOQ Outpatient Range
YOQ Inpatient Range
Measuring Symptom ReductionMeasuring Symptom Reduction
YOQ – Aggregate ResultsYOQ – Aggregate Results
Of the children discharged in 2008-2009, 42% Of the children discharged in 2008-2009, 42% showed a decrease in YOQ scores from the showed a decrease in YOQ scores from the second administration of the YOQ until the second administration of the YOQ until the time they were discharged. time they were discharged.
Children with higher YOQ Discharge Scores Children with higher YOQ Discharge Scores (suggesting more clinical symptoms) have a (suggesting more clinical symptoms) have a higher numbers of restraints.higher numbers of restraints.
Development of a New Development of a New Measure: Family WorkMeasure: Family Work
What do we want to know?What do we want to know? How often do families have therapy sessions?How often do families have therapy sessions?
Do families see therapy sessions as helpful?Do families see therapy sessions as helpful?
Do children participate in the sessions?Do children participate in the sessions?
Do meetings address issues that are important Do meetings address issues that are important to the family?to the family?
How can we measure it?How can we measure it?
Development of a New Development of a New Measure: Family WorkMeasure: Family Work
Design Design MeasureMeasure AssessAssess ImproveImprove
Improve
Assess
Measure
Design
Development of a New Development of a New Measure: Family WorkMeasure: Family Work
Discharge and Aftercare Discharge and Aftercare DataData
At DischargeAt Discharge Restrictiveness of Living Scale (ROLES)Restrictiveness of Living Scale (ROLES)
Aftercare SurveyAftercare Survey Interviews occur at four points in time:Interviews occur at four points in time:
3-6 months post discharge3-6 months post discharge 1 year post discharge1 year post discharge 2 years post discharge2 years post discharge 3 years post discharge3 years post discharge
At DischargeAt Discharge The Restrictiveness of Living Scale The Restrictiveness of Living Scale
(ROLES)(ROLES)Discharge Location for FY 2008-2009 Using the
Restrictiveness of Living Scale (ROLES)
Family Member (including adoption)
3255%
Foster Family Care10
18%
Therapeutic Foster Care
712%
Group Home2
4%
Residential Treatment
59%Juvenile Detention
12%
Therapeutic Foster Care Foster Family Care
Family Member (including adoption) Juvenile Detention
Residential Treatment Group Home
85% were discharged to a home setting
AftercareAftercare The intent of the Aftercare Study is The intent of the Aftercare Study is
to look at whether children are:to look at whether children are: At HomeAt Home
Still in less restrictive settingStill in less restrictive setting Disruptions in placementDisruptions in placement
In SchoolIn School SuspensionsSuspensions ExpulsionsExpulsions
Out of TroubleOut of Trouble Reported Drug/Alcohol UseReported Drug/Alcohol Use Involvement with Legal SystemInvolvement with Legal System
HealthyHealthy Have regular medical and dental appointmentsHave regular medical and dental appointments
Aftercare Data CollectionAftercare Data Collection
3 Month
6 Month
12 Month
24 Month
36 Month
80%
20%
79%
21%
83%
17%
85%
15%
94%
6%0%
20%
40%
60%
80%
100%
Follow-Up on Children Who Were Discharged to a Less Restrictive Setting -
Where Children Are Up to 3 Years Post-Discharge
Less Restrictive
More Restrictive
At HomeAt Home
Aftercare Data CollectionAftercare Data Collection
In School
At the first interview, 84% of families report being satisfied with their child’s school placement.
Two years after discharge, 74% of families report no disruptions in their child’s education.
Aftercare Data CollectionAftercare Data Collection
Out of Trouble
• A large percentage of the children were reported not to be using either drugs or alcohol. 96% of the children at the 24-month interval reportedly were not using either drugs or alcohol.
Aftercare Data CollectionAftercare Data Collection
Percentage of Children with No Legal Problems
0%
20%
40%
60%
80%
100%
3-6 Months 12 Months 24 Months 36 Months
Month of Interview
No Legal Problems Involvement with LawIn Juvenile Detention Facility On Probation
Out of Trouble
Aftercare Data:Aftercare Data:Representation of NeedsRepresentation of Needs
Leads to a referral to Family Resource Services
More Sophisticated Data More Sophisticated Data AnalysisAnalysis
Use of Excel or SPSS to explore Use of Excel or SPSS to explore trends in data:trends in data: Run ChartsRun Charts Pareto ChartsPareto Charts CorrelationsCorrelations RegressionsRegressions T-testsT-tests
Effectively Utilizing Effectively Utilizing Data CollectionData Collection
Developing an effective data Developing an effective data collection system is a critical part collection system is a critical part of providing quality services of providing quality services today.today.
Performance-based service Performance-based service delivery is essential to our delivery is essential to our practice. practice.
Questions?Questions?