www.TheNationalCouncil.org

Improving the Quality of Services Through Measurement and Feedback

Supported by the National Institute of Mental Health grant (RO1 MH068589) and a grant from the Leon Lowenstein Foundation .

Leonard Bickman, Ph.D., Susan Douglas Kelley, Ph.D., and Susan Casey, MSW

www.CFSystemsOnline.com

1

Outline• The challenge and opportunity• The theory of change• Contextualized Feedback

Systems and its components• Preliminary results of evaluation• Benefits• Plans

2

The Challenge:A Quality Chasm Exists for Mental

Health Services

“The greatest obstacle to discovery is not ignorance - it is the illusion of knowledge”Daniel Boorstin, 1984 Librarian of Congress

3

Little Research Support for the Effectiveness of Services in

Community Settings& Little Change in a Decade

• “The findings to date offer little support for the effectiveness of conventional clinical intervention for young people.” (Weisz & Jensen, 1999, p. 133)

• Some common elements of effective practice have been observed in treatment as usual but not in the depth and breadth needed for effective treatment. (Garland, Bickman, & Chorpita, 2010)

• But lack of evidence doesn’t necessarily mean lack of effectiveness 44

Little Accuracy About Client Progress & Focus of Treatment

• No correlation between standardized measures and clinicians’ perceptions of progress (Love et al., 2007)

• Difficulty predicting and detecting worsening of symptoms and functioning over the course of treatment (Hannan et al., 2005; Hatfield et al., 2009) – Yet, 89% of clinicians said they would know based on their

clinical judgment

• Little agreement on target problems by parents and therapists, and children and therapists (Hawley, KM & Weisz, JR. (2003).

• Chance Agreement on Level of Care Assignment (Bickman, L., Karver, M., & Schut, L. J. A. (1997).

5

Typical Community Based Care has Significant limitations

• “The findings to date offer little support for the effectiveness of conventional clinical intervention for young people.” (Weisz & Jensen, 1999, p. 133)

• Some common elements of effective practice have been observed in treatment as usual but not in the depth and breadth needed for effective treatment. (Garland, Bickman, & Chorpita, 2010)

• But lack of evidence doesn’t necessarily mean lack of effectiveness

66

Clinical Intuition and Regulations are Insufficient to Produce Good

Outcomes • Fifty years of research does not support

relying only on clinical judgment for effective practices

• Reliance on presumed standards of quality such as licensing and accreditation may also impede development of effective services

• Unmonitored “single shot” training on EBTs are not effective

7

Systematic and Accurate Feedback Can

Improve Outcomes

8

9

“Thus, use of patient monitoring and feedback in routine practice is imperative.

The focus on the individual rather than the average patient empowers the patient and encourages dialogue about progress, the direction of treatment and achievement of treatment goals”.

Newnham & Page. (2010). Bridging the gap between best evidence and best practice in mental health. Clinical Psychology Review, 30, 127–142 (p.138).

Strong Support for Feedback Concept

• Measurement and feedback are the core of all management and learning theories.

• Thousands of studies outside of mental health show that improvement is minimal without measuring performance and providing feedback.

• Direct feedback occupations show improvement with experience. However, clinician experience alone is not a good predictor of client outcomes.

10

Providence Service Corporation delivers home and community based social services to government beneficiaries and privatized social service projects.

Providence Service Corporation

12

Home and Community

Based Services

Management Services

Foster Care Services

PRSC ServiceOffering

.

Providence Overview

• Providence Service Corporation was aware of the need for feedback and was committed to implementing a quality enhancement initiative

• In 2004 we began a collaboration with Vanderbilt University to combine their efforts to create an evidence-based feedback system

13

Providence Core Values

• Community-based and Multi-systemic Services

• Identifying and Building on Strengths• Local Viability that is Nationally

Supported• Respectful Organizational Culture• Best Practice Models • Cultural Diversity

Providence Service Corporation

Quality Enhancement Initiative

15

• 12 states, 38 sites, 287 clinicians

• 836 youth 11-18 years of age and their caregivers

• 10,431 client weeks of Providence provided services

CFS - A Concurrent, Systematic Monitoring and Formative Feedback

Practice Improvement Tool

Practice Without Feedback Does Not Lead To Improvement

16

CFS is a Practice Improvement Strategy

• is part of treatment – provides an opportunity for brief reflection at the close of a session

• is primarily designed to support clinicians – provides guidance for those moments when little seems to be working

• supports supervisors – helps supervisors identify areas where clinicians need extra guidance to ensure they feel confident

• is flexible – optional custom measures, schedules, reports can be tailored for office workflow and QI initiatives

• can be used for administrative and reporting functions –provides tools for leadership to become or remain successful at meeting the needs of their funders and clients

17

18

CFS Theory of Change is on Two Levels

• The individual or psychological level

• The group or organizational level

19

Key Domains of CFS

• System Development• Measurement• Feedback• Training in Common Factors• Comprehensive Support

Dashboard-driven System

20

Easy to navigateHighlights common actions by userDashboards are designed for clients and

caregivers, clinicians, supervisors, administrative assistants, program directors, etc.

Sample Clinical Alerts

21

New = most recent alerts; not yet viewed

Pending = old alert; not yet viewed

Link to the feedback report containing the alert; Archive alerts

Sample Client Login

22

After logging in, client is presented with questionnaire to be completed.

23

Barry’s Sample Case – Clinician Judgment is Necessary - What Do You Notice?

24

What do you see here?

25

Weeks1 2 3 4 5 6 7

Measurement: The Peabody Treatment Progress Battery (PTPB)

• Domains include processes and outcomes• Includes strength-based measures• Brief – each takes 15 seconds to 2 minutes• Reliable • Show convergent and divergent validity• Information from youth, clinician, and caregiver• Easy to score and interpret• Sensitive to change• Available in English and Spanish• Free for anyone to use – now in second edition• http:\\peabody.vanderbilt.edu/ptpb

26

CFS Measures Map onto the Relevant Questions

• How is the youth doing overall?– Severity of symptoms and functioning (y, cg, cl)

– Life satisfaction (y)

– Hope (y)

• How is treatment going with the youth?– Therapeutic alliance (y, cl)

– Motivation for treatment (y)

– Counseling impact (y)

– Service satisfaction (y)

• What are important caregiver issues?– Caregiver strain– Life satisfaction– Therapeutic alliance– Motivation for treatment– Service satisfaction

27

28

Organizational Measures• Services, staffing, any data already

collected; experience with other initiatives, etc.

• Culture, climate, adaptability, leadership, organizational learning, & initial perceptions of CFS

• Current attitudes toward CFS, barriers/supports, value of reports, self-efficacy, & goal commitment

CFS Consultation and Training Model

• CFS is a tool for transforming your agency into a learning organization

• Ongoing support to– Promote sustainability through local ownership– Contextualize CFS operations and feedback to

your agency’s needs and resources– Apply feedback to inform clinical sessions,

supervision, program planning, professional development, and more

– Empower practice improvement leadership at all levels of your agency

29

30

Analyses of Current Project• Randomized experiment in 33 sites -

Analyses ongoing• Randomly assigned sites to feedback and no

feedback conditions• Started with most direct question: Does

feedback appear to affect clinician behavior?• The answer to this question does not depend

upon the effectiveness of the treatment provided

CFS Feedback Influences Clinician Behavior in Sessions• Cases selected where clinician had not previously reported

a problem with youth symptoms and functioning in a specific domain (e.g., behavioral issues, emotional problems, friend/peer issues)

• When a client or caregiver alert was present for that domain– Viewing clinical feedback reports associated with addressing that

domain in a future session– More feedback viewed more times addressed– Not explained by more reporting of problems by clients or

caregivers

31

Feedback Reduces Symptoms and Improves Functioning

• Used HLM random effects with new clients, controlled for nesting within sites, and number of sessions.

• Youth respondent (ages 11-18): 340 clients• Caregiver respondent: 308 clients• Clinician respondent: 294 clients• There were no significant differences at baseline

between the two groups for any respondent• Results showed Feedback Group improved more

than the non-Feedback Group according to all three respondents.

32

Modeled Values of Youths’ SFSS p = 0.025

33

30

35

40

45

50

55

60

65

70

75

80

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Yout

hs’ S

FSS

in C

BC

L U

nits

Months in CFS

Feedback

No Feedback

Modeled Values of Caregivers’ SFSS p = 0.05

34

30

35

40

45

50

55

60

65

70

75

80

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Car

egiv

ers’

SFS

S in

CB

CL

Uni

ts

Months in CFS

Feedback

No Feedback

Modeled Values of Clinicians’ SFSS p = .0005

35

30

35

40

45

50

55

60

65

70

75

80

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Clin

icia

ns’

SFSS

in C

BC

L U

nits

Months in CFS

Feedback

No Feedback

What Can CFS DoFor Clients and Caregivers?

• Provide a way to raise issues they may not feel comfortable addressing aloud or in the presence of caregiver or youth

• Reassure them that the clinician is paying attention to their needs and is contextualizing services to their concerns

• Demonstrate that effective services are a priority

• CFS is a tool that takes the consumer’s voice beyond the session and into all aspects of care

36

What Can CFS DoFor Clinicians?

• Better identify thoughts, events, and feelings of clients and caregivers

• Identify successes and problem areas to focus sessions• Provide consistent and systematic feedback of clinically

relevant information• Inform treatment planning and goal setting• Focus sessions on clients’ issues to show that you are

attending to their concerns• Check on how well treatment is working• CFS is a tool that enhances the clinician’s ability to tailor

treatment as it progresses37

What Can CFS DoFor Clinical Supervisors?

• Provide clinical data on each case’s progress and areas that seem to be improving or declining

• Provide a tangible framework from which to guide clinical supervision that is independent of the clinician

• Provide the needed information for a continuous quality improvement effort that facilitates accountability

• Most of all…CFS is a tool that provides the supervisor with needed resources to promote evidence-based practice

38

What Can CFS DoFor an Agency?

• Serve as evidence to funding agencies that quality of services and effectiveness are a priority

• Manage clinical services on their impact as well as their cost

• Provide data on the effectiveness of services, how clients are improving, the typical problems being encountered, and where needs are not being met

• CFS is a tool that supports overall practice improvement

39

What CFS Can Do for the Funder?

• Determine who is being served and their progress

• Provide a Quality Assurance mechanism• Compare services and agencies on cost

effectiveness to optimize value• Reduce costs of some regulations and

documentation activities such as excessively detailed treatment planning and progress notes

40

Plans• Continued partnership between Vanderbilt and Providence in

training, development, and research• Continue software development• Use with adults as well as children and youths• Continue development of new measures• Proposal pending to test CFS with different types of

feedback with youth with substance abuse problems• Integrate CFS with other mental health EBTs (new NIMH

grant for Functional Family Therapy combined with CFS)• Develop web-based “treatment” intervention with CFS

(Australia Project)• Use CFS in primary care settings to bridge medical and

mental health fields

41

42

Web Site: www.CFSystemsOnline.com