Implementing Evidence-Based Practices

Our Obligation to Program Fidelity

Kimberly Gentry Sperber, Ph.D.

Efforts To Date“What Works” Literature

Principles of Effective InterventionsGrowing evidence based on individual program evaluations and meta-analyses

Continuing Gap Between Science and Practice

Few programs score as satisfactory on CPAI

CPAI Data

Hoge, Leschied, and Andrews(1993) reviewed 135 programs assessed by CPAI

35% received failing score; only 10% received score of satisfactory or better.

Holsinger and Latessa (1999) reviewed 51 programs assessed by CPAI

60% scored as satisfactory but needs improvement or unsatisfactory; only 12% scored as very satisfactory.

CPAI Data Continued

Gendreau and Goggin (2000) reviewed 101 programs assessed by CPAI

Mean score of 25%; only 10% scored received satisfactory score

Matthews, Hubbard, and Latessa (2001) reviewed 86 programs assessed by CPAI

54% scored as satisfactory or satisfactory but needs improvement; only 10% scored as very satisfactory.

Fidelity Research

Landenberger and Lipsey (2005)Brand of CBT didn’t matter but quality of implementation did.Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers.

Schoenwald et al. (2003) Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients.

Henggeler et al. (2002)Supervisors’ expertise in the model predicted therapist adherence to the model.

Sexton (2001)Direct linear relationship between staff competence

and recidivism reductions.

More Fidelity Research

Schoenwald and Chapman (2007)A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatmentA 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment.

Schoenwald et al. (2007)When therapist adherence was low, criminal outcomes for substance abusing youth were worse relative to the outcomes of the non-substance abusing youth.

Washington State Example(Barnowski, 2004)

• For each program (FFT and ART), an equivalent comparison/control group was created

• Felony recidivism rates were calculated for each of three groups, for each of the programs

• Youth who received services from therapists deemed ‘competent’

• Youth who received services from therapists deemed ‘not competent’

• Youth who did not receive any services (control group)

Family Functional Therapy Results: % New Felony

139 6

2519

11

3227

17

0

10

20

30

40

50

60

70

6 Months 12 Months 18 Months

FFT Not Competent

Control group

FFT Competent

Results calculated using multivariate models in order to control for potential differences between groups

UC Halfway House/CBCF Study in Ohio: A Look at Fidelity Statewide

Average Treatment Effect was 4% reduction in recidivismLowest was a 41% Increase in recidivismHighest was a 43% reduction in recidivism

Programs that had acceptable termination rates, had been in operation for 3 years or more, had a cognitive behavioral program, targeted criminogenic needs, used role playing in almost every session, and varied treatment and length of supervision by risk had a 39% reduction in recidivism

What Do We Know About Fidelity?

Fidelity is related to successful outcomes (i.e., reductions in recidivism, relapse, and MH instability).Poor fidelity can lead to null effects or even iatrogenic effects.Fidelity can be measured and monitored.Fidelity cannot be assumed.

Why Isn’t It Working?Latessa, Cullen, and Gendreau (2002)

Article notes 4 common failures of correctional programs:

Failure to use research in designing programsFailure to follow appropriate assessment and classification practicesFailure to use effective treatment modelsFailure to evaluate what we do

Ways to Monitor Fidelity

Training post-testsStructured staff supervision for use of evidence-based techniquesSelf-assessment of adherence to evidence-based practicesProgram audits for adherence to specific models/curriculaFocus review of assessment instrumentsFormalized CQI process

Ensuring Training Transfer

Use of knowledge-based pre/post-testsUse of knowledge-based proficiency testsUse of skill-based rating upon completion of trainingMechanism for use of data

Staff must meet certain criteria or score to be deemed competent.Failure to meet criteria results in consequent training, supervision, etc.

Staff Supervision

Staff supervision is a “formal process of professional support and learning which enables individual practitioners to develop knowledge and competence, assume responsibility for their own practice and enhance [client]… care in complex … situations.”

Modified from Department of Health, 1993

Performance Measurement for Staff

Standardized measurementConsistencyEveryone measured on same items the same way each time

Consistent meaning of what is being measured

Everyone has same understanding, speaks the same language

Sample Measures

Uses CBT language during encounters with clients.Models appropriate language and behaviors to clients.Avoids power struggles with clients.Consistently applies appropriate consequences for behaviors.Identifies thinking errors in clients in value-neutral way.

Sample Employee Observation Evaluation

Employee Name: Supervisor Name: Site: Observation Time (amount): Start Time: Stop Time: Place of Observation: Activity Observed: Behavior Below

Expectations Needs Improvement

Meets Expectations

Exceeds Expectations

Comments on Direct Observations

Uses corrective thinking language during encounters with clients.

Models appropriate language and behavior to clients. Includes: Speaking positively about program, law, courts, etc. Does not use derogatory language/jokes op sarcasm.

Avoids power struggles with clients (e.g., does not argue with clients, raise voice at clients, antagonize clients)

Consistently applies appropriate consequences for behaviors (both positive and negative)

Identifies thinking barriers in clients in value-neutral way

Overall Score

Agency Self-Assessment:Assessing Best Practices at 17

Sites

Use of ICCA Treatment Survey to establish baselineComplete again based on best practicePerform Gap AnalysisAction Plan

ICCA Treatment Survey

CQI Manager and Clinical Director met with key staff from each program to conduct self assessment of current practices.

Evaluated performance in 6 key areasStaffAssessment/ClassificationProgrammingAftercareOrganizational ResponsivityEvaluation

Agency Response:Strategic Plan

FY2006Required to submit at least 1 action plan to “fix” an identified gap.Gaps in the areas of risk and need to be given priority.

FY2007Required to submit 2 action plans.One on use of role-plays and one on appropriate use of reinforcements.

FY2008Proposed focus on fidelity measurement at all sites.Creation of checklists and thresholds.

Program Audits:CBIT Site Assessments

Cognitive Behavioral Implementation TeamSite visits for observation and ratingStandardized assessment processStandardized reports back to sitesCombination of quantitative data and qualitative data

Individual LSI ReviewsSchedule of videotaped interviewsSubmitted for reviewUse of standardized audit sheetFeedback loop for staff developmentAggregate results to inform training efforts

LSI Audit Form Reviewer: Date: Interviewer:

1 = Poor 2 = Fair/Needs Improvement 3 = Good 4 = Excellent 1 2 3 4

1. Explanation of the purpose of the interview. ❑ ❑ ❑ ❑

2. Established structure for the interview. ❑ ❑ ❑ ❑

3. Adequate use of open-ended questions. ❑ ❑ ❑ ❑

4. Avoidance of double-barreled questions. ❑ ❑ ❑ ❑

5. Avoidance of biased/leading questions. ❑ ❑ ❑ ❑

6. Adequate use of follow-up questions. ❑ ❑ ❑ ❑ 7. Avoided barriers to listening (such as moralizing, disagreeing,

Blaming, shaming, reinforcing). ❑ ❑ ❑ ❑

8. Interview overcame problems such as silence or excessive talking. ❑ ❑ ❑ ❑

9. Interviewer used the interview guide. ❑ ❑ ❑ ❑

10. Notes were made indicating why items were or were not scored. ❑ ❑ ❑ ❑

11. Adequate documentation in the case of an override. ❑ ❑ ❑ ❑

12. Treatment plan clearly relates to information captured in the LSI. ❑ ❑ ❑ ❑

Total score: ______ divided by _______ = Reviewer Comments:

Formal CQI Model

Data Collection/Review Requirements:Peer Review (documentation)MUI’s/IncidentsComplaints/GrievancesEnvironmental ReviewClient SatisfactionProcess IndicatorsOutcome Indicators

Formal CQI Model

Programs required to review data monthly and to action plan accordingly.Each program’s data and action plans reviewed once per quarter by agency’s Executive CQI Committee

CQI Committee Infrastructure

Oversight CQI Committee

Risk Management Committee

Safety Committee Human Subjects Committee

Diversity Committee Corporate Compliance Committee

Cluster CQI Committees

Program Peer Review Committees

Morbidity & Mortality Conference

The Talbert House Strategic PlanFocus on Fidelity

FY2008 – FY2010 Objective:Improve Quality of Client Services

FY2008 – FY2010 Goal:Exceed 90% of quality improvement measures annually

FY2008 – FY2010 Strategy:Talbert House programs demonstrate fidelity to best practice service/treatment models as demonstrated by site specific best practice fidelity check sheet.

100% of programs create a Fidelity measurement tool by 12/31/07. 100% of programs establish and measure its site-specific fidelity threshold by 1/30/08. Programs will be expected to meet/exceed established fidelity thresholds by 6/30/09.Programs will be expected to meet/exceed established fidelity thresholds by 6/30/10.

Getting Started

What services do you say/promise that you deliver?

What does your contract say?What do referral sources expect?

List all programming componentsWhat is the model (e.g., CBT, MI, IDDT, IMR, TFM, etc.)?What curricula are in use?

Identify which is most importantMake selection for measurement

Creating a Tool for Measurement

Scale should adequately sample the critical ingredients of the EBP.Need to be able to differentiate between programs/staff that follow the model versus those that do not.Scale should be sensitive enough to detect progress over time.Need to investigate what measurement tools may already exist.

Sample Measures

CBT Group Observation FormTFM Fidelity Review Sheets and DatabaseIMR Fidelity Rating ScaleIDDT Fidelity ScaleMotivational Interviewing Treatment Integrity (MITI) Code

Sample Project - TFM

4 residential adolescent programs implemented Teaching Family Model.Required to record all teaching interactions with all clients.Required to record data on standardized form and to enter into Fidelity database.CQI Indicator = percentage of staff achieving 4:1 ratio.

Sample Project – CBT Groups

Several programs conducting group observations using standardized rating form.Needed to operationalize who would do observations and how frequently.Needed to operationalize how data would be collected, stored, analyzed, and reported.CQI Indicator = percentage of staff achieving a rating of 3.0. (on scale of 0-3).

Measuring CBT in GroupsYear One

Chose 5 items from observation tool:Use of role plays/or other rehearsal techniquesAbility of the group leader to keep participants on task Use of peer interaction to promote prosocial behaviorUse of modelingUse of behavioral reinforcements

Measuring CBT in GroupsYear Two

Refinement of role-play indicators:Percentage of groups observed where staff modeled the skill prior to having clients engage in role-playPercentage of role-plays containing practice of the correctivesPercentage of role-plays that required observers to identify skill steps and report back to the group

Sample Project – Dosage by Risk and Need

Program created dosage grid by LSI-R risk category and criminogenic need domains.

Requires prescribed set of treatment hours by risk

Program created dosage report out of automated clinical documentation system.Review monthly to insure clients are receiving prescribed dosage.Also review individual client data at monthly staffings.CQI Indicator = percentage of successful completers receiving prescribed dosage (measured monthly).

Sample Dosage ProtocolCommunity Correctional Center Risk Level Structure Guide

MediumHigh High Medium Low/ Moderate Low

LSI Score Range 34+ 31-33 24-30 19-23 0-18Length of Stay Target (days) 147 133 119 105 60

Corrective Thinking 200 180 132 92 52AOD 62 54 46 38 28Individualized Relapse Prevention 21Anger Management 24 24 24 24 if neededDomestic Violence 24 15 15 15 if neededVocational* 15 15 15 15 8Life Skills* 16 16 16 16 8Personal Development* 10 10 10 10 if needed

*not counted in dosage total

Total hours available: 351 314 258 210 117

Sample Dosage Protocol

CT Groups

Total progam hours LSI range

CT--1 8 19 and below CT--2 12 20--26

CT--3 16 26 and above

CD Groups

Total program

hours Diagnosis

Level 1 12 Abuse Level 2 18 Dependence

Relationship Between Evaluation and

Treatment Effect (based on UC Halfway House and CBCF study)

6

1

0

2

4

6

8

Internal QA No Internal QA

% C

hang

e in

Rec

idiv

ism

NPC Research on Drug Courts

Drug Court Uses Evaluation Feedback to Make Modifications

11%

44%

0%

10%

20%

30%

40%

50%

YesN=4

NoN=6

* "Percent improvement in outcome costs" refers to the percent savings for drug court compared to business-as-usual

Per

cen

t Im

pro

vem

ent

in O

utc

om

e C

ost

s*

Conclusions

Many agencies are allocating resources to selection/implementation of EBP with no evidence that staff are adhering to the model.There is evidence that fidelity directly affects client outcomes.There is evidence that internal CQI processes directly affect client outcomes.Therefore, agencies have an obligation to routinely assess and assure fidelity to EBP’s.Requires a formal infrastructure to routinely monitor fidelity performance.