Assessment and Progress Monitoring in School-Based

Mental Health

October 16, 2012

Aaron Lyon, PhD (UW/SCH)Jessica Knaster Wasse, MPH (PHSKC)

Kristy Ludwig, PhD (UW/SCH)

A collaboration between Public Health of Seattle-King Country, the Seattle Public Schools, and the University of Washington School of Medicine/Seattle

Children’s Hospital

Acknowledgments

• Ann Vander Stoep• Elizabeth McCauley• Eric Bruns• Evalynn Romano• Jane Koltracht• Kelly Thompson• Seattle Public Schools• TJ Cosgrove• School-based practitioners!

Collaborators: Funding:• National Institute of Mental

Health (K08MH095939)• Institute of Education Sciences

(R305A120128)• American Psychological

Foundation• Public Health of Seattle & King

County• Bill and Melinda Gates

Foundation

Dr. Lyon is an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25MH080916-01A2) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).

Overview

1. Background & goals of standardized assessment / progress monitoring initiative in Seattle’s school-based health centers (SBHCs)

2. Public Health of Seattle & King County’s (PHSKC) interest in / support for progress monitoring in SBHCs

3. Current training approach / activities

4. Future directions

School-Based Health Centers (SBHCs)

• Integrated care clinics situated in schools• Operate in nearly 2,000 schools in the US (NASBH,

2008)

• Typically provide primary care and mental health services (Brown & Bolen, 2003)

• Well-substantiated as a mechanism to increase service accessibility to underserved and under/uninsured (Gance-Cleveland & Yousey, 2005; Kaplan et al., 1999; Wade et al., 2008)

Evidence-Based Practice in SBMH

• EBP utilized inconsistently in SBMH (Evans & Weist, 2004)

• Implementation of EBP in SBMH should focus on low-cost, high-yield practice improvement targets• Disseminate / Implement key competencies instead of full Tx

packages (Beidas et al., 2011; Rotheram-borus et al., 2012)

• Evidence-based assessment may be the “low-hanging fruit” of EBP implementation• School-based providers unlikely to use standardized assessment

and progress monitoring routinely• Previous work has demonstrated good uptake of SA following

training (Lyon et al., 2011)

Evidence-Based Assessment

Evidence-Based Assessment (EBA): “Assessment methods and processes that are based on empirical evidence in terms of both their reliability and validity as well as their clinical usefulness for prescribed populations and purposes” (Mash & Hunsley, 2005, p.364)

• Evidence-based interventions commonly rely on EBA

Standardized assessment (SA), or the use of measurement tools with empirical support for their reliability, validity, etc, is at the core of EBA (Jensen-Doss & Hawley, 2005)

Value of Standardized Assessment

Initial Assessment• Rating scales can increase the ease and accuracy of

clinical diagnosis (e.g., Jenkins et al., 2011; Youngstrom et al., 2005)

• Psychological assessment carries positive, clinically meaningful effects (Posten & Hanson, 2010)

Progress Monitoring• Clinicians are often not able to detect client

deterioration (Hannan et al., 2005)

• Providing assessment results to clinicians can result in improved outcomes (Bickman et al., 2011; Lambert et al., 2003)

Steps of Standardized Assessment Use

1. Selection

2. Administration

3. Scoring

4. Interpretation

5. Feedback to clients

6. Treatment planning

7. Monitoring (over time w/ feedback)

8. Treatment plan adjustment

SA / Progress Monitoring: SBMH v. CMH

SBHC(n = 16)

CMH(n = 58)

Percent of Caseload: 0% 1-39% 0% 1-39%

Gave a SA measure in initial 1-2 meetings 47% 27% 16% 32%

Gave a SA measure at termination 63% 36% 52% 23%

Gave a SA measure 38% 44% 42% 42%

Gave feedback about a SA measure 50% 31% 46% 30%

Changed Tx plan based on SA data 69% 31% 64% 32%

Changed indiv. session plan based on SA 56% 38% 64% 27%

Progress Monitoring & Feedback

• Progress monitoring may use SA tools or ideographic monitoring targets

• Value of ideographic targets (Weisz et al., 2011)…• Add specificity to ID’d problems• Focus on client concerns / priorities• Give clients a voice• Enhance rapport / alliance• Provide foci for ongoing assessment• Provide info about treatment termination• Easy to fit into everyday practice

• Combination of SA and ideographic is likely optimal

Progress Monitoring & Feedback

• Feedback Intervention Theory (Kluget & DeNisi, 1996): Bx is regulated by comparisons of feedback to goals

• Two levels (at least!): (1) feedback to the clinician & (2) feedback to the client• Most research has focused on feedback to clinicians

• Feedback is associated with higher rapport with provider, client self-understanding (Allen et al., 2003), and increased engagement in therapy (Ackerman et al., 2000)

• Academic outcomes are an essential domain for youth success, but rarely included in MH research (Atkins et al., 2010; Franklin et al., 2009; Hoagwood et al., 2007)

• Youth w/ combined MH and academic risk have a greater likelihood of long-term negative outcomes (Roeser et al., 1999)

• School-based clinicians rate school attendance as the most clinically-useful academic indicator (Lyon, Ludwig, & McCauly, in prep)

Monitoring School Outcomes in Tx

Potential Assessment / Monitoring Targets

Mental Health•Depression symptoms•Disruptive behavior•Self-injurious behavior

School Engagement•Attendance•Homework completion•Class participation•School connectedness

Academic Outcomes•Grades•Credits earned•Standardized test scores

Social Functioning•Interpersonal conflicts•Positive social experiences•Disciplinary events

Physical Health•Sleep•Diet & Exercise

Services•Satisfaction with treatment•Engagement in intervention

1. Engage stakeholders in planning / adapting a computerized measurement feedback system (MFS) to support SBMH providers

2. Provide training to SBMH providers in the use of standardized and ideographic assessment / progress monitoring

3. Implement, evaluate, and iterate the MFS, incorporating the ability to monitor academic outcomes

Current Goals & Activities

Engaging Stakeholders in the Adaptation of a Measurement Feedback System for SBHCs

Policy/funding context •Seattle Families & Education Levy

• Property tax levy, renewed for 4th 7 year term starting 2012• SBHC funding managed by PHSKC• Levy performance goals are academic in nature

•Successfully lobbied for Levy $ for MH enhancement• Goal: Enhance the academic impact of Levy health investments

by improving the quality of school mental health services • Increase use of evidence-based practice, with focus on

standardized assessment and outcome monitoring

SBHCs in Seattle / King County

SBHCs in Seattle / King County

Vision: Collaborative stepped care model• SA data used to support clinical decision making• Track clinical improvement and identify patients who are not

improving as expected

Aligns with Seattle Public Schools’ Response to Intervention (RtI) strategy

Graphic from the National Center on Response to Intervention

• “When developing an evaluation system, two of the most challenging components are getting buy-in from all partners involved and then selecting the measurement tools” (Sander et al., 2011, p.15).

• Gates Foundation funding for one-year planning process• Stakeholder committee comprised of: school-based MH

providers, supervisors, evaluators, program managers, computer programmers, school district

Stakeholder Committee

Monthly meetings designed to:

• Build initial consensus about data fields and other user-interface issues

• Identify initial battery of standardized assessment measures

• Identify SA toolkit for various MH domains• Develop and refine implementation / evaluation plan• Enhance provider buy-in

Stakeholder Committee

Challenges/hurdles

• Differing documentation requirements• Duplicate data entry EHR and MFS• Varying degrees of technical knowledge• Evaluation design: impact of randomization

Stakeholder Committee

Results/Successes

•Data sharing agreement with Seattle Public Schools•Specifications completed on schedule•Negotiated acceptable evaluation plan•Providers engaged and excited

Stakeholder Committee

Training in Standardized Assessment and Progress

Monitoring

In promoting data-based decision making, SBMH professionals often struggle to (Kelly, 2011)…

1. Clearly define the reason for collecting the data

2. Ensure that data-collection procedures are manageable / user-friendly

3. Map out a clear system to use the data for continuous improvement

Training in SA & Progress Monitoring

• Over the past 2 years, our group has provided a training and consultation series to local SBHC MH providers covering…• Standardized assessment principles, psychometrics,

and normative data• Initial assessment using standardized tools• Feedback to students on the basis of assessment• Identifying and tracking progress monitoring targets

Training in SA & Progress Monitoring

Principles of progress monitoring:•Monitoring of some type is a good idea for ALL youth receiving services•Only track those targets that are meaningful to the specific case/match treatment goals•Graphical feedback increases

understanding and makes the feedback more memorable (Kluger & DiNisi, 1996; Miller & Watkins, 2010)

•Can be used as a tool to validate changes in a student’s experience

Training in SA & Progress Monitoring

Training in SA & Progress Monitoring

Moving from the WHAT to WHY… •Progress monitoring is the first step and gives you the WHAT•Next steps include identifying the WHY (i.e., why is a target changing or not changing?)

• Moving beyond just “success” or “failure”

•The WHY is important to the kid feeling successful• Kid’s feelings of agency and control (therapy isn’t being done to

them or for them but something they are willfully and actively participating in)

Assessment Flow Chart

1) Decrease anxiety symptoms

2) Improved school engagement

3) Improved sleep hygiene

Student Presentation Assess

Treatment Goal ID

SA

Ideo-graphic

SA: MASC-10 IG: Anxiety experienced when arriving at school?

SA: School Connectedness ScaleIG: # on-time arrivals for 1st period

SA: None?IG: Nights/wk w/o FB or texting after 8pm

Current Findings and Future Evaluation Directions

Focus Groups • Focus groups conducted to evaluate the context in which the

MFS will be implemented.• Transcribed and evaluated using a conventional content

analysis approach. • EMERGING THEME: Factors which facilitate or inhibit the utility

of a technology in SBMH (Lyon et al., in prep)

• Accessibility• Impact on Youth Engagement• Confidentiality• Knowledge or Attitudes (Provider)• Integration of Aspects of Care• Relevance to MH Care• Speed of the Technology

Pilot of the Un-adapted MFS

• Anonymous surveys of youth receiving SBMH services supported by the un-adapted MFS (n = 34)

• 71% of youth respondents stated that answering questionnaires about emotions made it easier to tell their providers how they were feeling

• 55% said answering the questionnaires (and receiving feedback) made the time spent with their SBMH providers either a little or much easier/better.

• 45% said it did not affect their interactions (no students reported that it made it worse)

Many Questions Remain…

Selected findings from Key Informant individual interviews from a related project…•Who should provide feedback about progress?

• Teachers, students, school teams

•Who should receive / review feedback?• In addition to clinician, principal, teachers, school staff, the

academic counselor, school teams• Allow student to determine who will review

•Other challenges• Unique data systems and data sharing rules

• May require individualized approach district-by-district• Providers not used to collecting/using data systematically• Transitioning from health/MH to academic outcomes

Next Steps

1. Complete initial adaption of the MFS

2. Implement the MFS with randomly-selected providers

3. Compare provider behavior (e.g., use of standardized or ideographic assessment/monitoring) between MFS and non-MFS groups

4. Conduct independent evaluations of youth outcomes by recruiting from the caseloads of MFS and non-MFS providers

Questions and Discussion