Evidence-based Application of Evidence-based Treatments

Peter S. Jensen, M.D.President & CEO

The REACH InstituteREsource for Advancing Children’s Health

New York, NY

Effect Sizes of PsychotherapiesEffect Sizes of Psychotherapies

00.10.20.30.40.50.60.70.80.9

1

Smith &Glass,1977

Shapiro&

Shapiro,1982

Casey &Berman

Weisz etal., 1987

Kazdinet al.,1990

Weisz etal., 1995

Weisz etal, 1995M

ean

Eff

ect

Size

s

Weisz et al., 1995

Children & AdolescentsAdults

University

“Real World”

Three Levels:

Child & Family Factors: e.g., Access & Acceptance

Provider/Organization Factors: e.g., Skills, Use of EB

Systemic and Societal Factors: e.g., Organiz., Funding Policies

Barriers vs. “Promoters” to Barriers vs. “Promoters” to

Delivery of Effective Services Delivery of Effective Services (Jensen, 2000)(Jensen, 2000)

“Effective” Services

Efficacious Treatments

0.5

1

1.5

2

2.5

0 100 200 300 400

Assessment Point (Days)

Ave

rage

Sco

re

CC-NOMEDS

CC-MEDS

BEH

MED

COMB

Key Differences, MedMgt vs. CC:

Initial Titration

Dose

Dose Frequency

#Visits/year

Length of Visits

Contact w/schools

Teacher-Rated InattentionTeacher-Rated Inattention

(CC Children Separated By Med Use)(CC Children Separated By Med Use)

Would You RecommendWould You Recommend Treatment? (parent) Treatment? (parent)

Medmgt Comb Beh

Not recommend 9% 3% 5%

Neutral 9% 1% 2%

Slightly Recommend 4% 2% 2%

Recommend 35% 15% 24%

Strongly recommend 43% 79% 67%

Key ChallengesKey Challenges

Policy makers and practitioners hesitant to implement change

Vested interests in the status quo Researchers often not interested in promoting

findings beyond academic settings Manualized interventions perceived as difficult

to implement or too costly Obstacles and disincentives actively interfere

with implementation

Key ChallengesKey Challenges

Interventions implemented but “titrate the dose”, reducing effectiveness

“Clients too difficult”, “resources inadequate” used to justify bad outcomes

Research population “not the same” as youth being cared for at their clinical site

Having data and “being right” neither necessary nor sufficient to influence policy makers

The Good and the Bad: Effectiveness of Interventions The Good and the Bad: Effectiveness of Interventions by Intervention Type by Intervention Type

05

101520253035

Positive Negative

Davis, 2000

No. of Interventions demonstrating positive or negative/inconclusive change

Little or No Effect (Provider & Little or No Effect (Provider & Organization-focused) :Organization-focused) :

Educational materials (e.g., distribution of recommendations for clinical care, including practice guidelines, AV materials, and electronic publications)

Didactic educational meetings

Bero et al, 1998

Effective Provider & Organizational Effective Provider & Organizational Interventions:Interventions:

Educational outreach visits Reminders (manual or computerized) Multifaceted interventions Sustained, interactive educational meetings

(participation of providers in workshops that include discussion and practice)

Bero et al, 1998

Implications re: Changing Provider Implications re: Changing Provider BehaviorsBehaviors

• Changing professional performance is complex - internal, external, and enabling factors

• No “magic bullets” to change practice in all circumstances and settings (Oxman, 1995)

• Multifaceted interventions targeting different barriers more effective than single interventions (Davis, 1999)

• Little to no theory-based studies

• Consensus guidelines approach necessary, but not sufficient.

• Lack of fit w/HCP’s mental models

Additional PerspectivesAdditional Perspectives Messenger of equal importance as the message

Trusted Available Perceived as expert/competent

Adult Learning Models Tailored to learner’s needs Learner-defined objectives Hands-on, with ample opportunities for practice Sustained over time Skill-oriented Feedback Attention to Maintenance and sustaining change

Dissemination and Adoption of Dissemination and Adoption of New Interventions New Interventions

Source: Backer, Liberman, & Kuehnel (1986) Dissemination and Adoption of Innovative Psychosocial Interventions. Journal of Consulting and Clinical Psychology, 54:111-118; Jensen, Hoagwood, & Trickett (1997) From Ivory Towers to Earthen Trenches. J AppliiedDevelopmental Psychology

Sustained Interpersonal contact Organizational support Persistent championship of the intervention Adaptability of the intervention to local situations Availability of credible evidence of success Ongoing technical assistance, consultation

Science-based Plus Necessary “-abilities” Science-based Plus Necessary “-abilities”

• Palatable• Affordable• Transportable• Trainable• Adaptable, Flexible• Evaluable • Feasible• Sustainable

Models for Behavior Change:(Jaccard et al, 2002)

The Theory of Reasoned Action (Fishbein & Ajzen, 1975)Self-efficacy Theory (Bandura, 1977)The Theory of Planned Behavior (Ajzen, 1981)Diffusion of Innovations (Rogers, 1995)

Influences on Provider BehaviorInfluences on Provider BehaviorPatient & Family Factors:•Stigma

•Adherence

•Negative attitudes

•Rapport, engagement

Provider Factors:•Knowledge, training

•Self-efficacy

•Time pressures

•Fear of litigation

•Attitudes & beliefs

•Social conformity

•Lack of information

Economic Influences:•Compensation

•Reimbursement

•Incentives

Systemic & Societal Factors:•Organizational standards

•Staff support/resistance

•Staff Training

•Funding policy

Prescribing Practices

First, Use an Atypical vs. TypicalFirst, Use an Atypical vs. Typical

Favor/Unfavor

Easy/Hard

Improve/No

Agree/Disagree

543210-1-2-3-4-5

2

323

362

Descriptives (n=19)

Min/Max Mean(SD)

Favor/Unfavor 0/5 3.73(1.61)

Easy/Hard -1/5 4.16(1.64)

Improve/No 0/5 2.84(1.57)

Agree/Disagree 0/5 4.05(1.27)

First Use Atypical--AdvantagesFirst Use Atypical--Advantages

Advantages Count Percent of Responses

Avoids typicals' side effects 13 59.1%Better patient approval/compliance 5 22.7%Atypicals effective in treating aggression 2 9.1% Other (i.e. looks better politically) 2 9.1%

Total responses 22 100.0%

First Use Atypical – DisadvantagesFirst Use Atypical – Disadvantages

Disadvantage Count Percent of Responses

Typicals may work better for some patients 6 23.1% Avoids atypicals' side effects 6 23.1%If need to sedate patient, typicals may be better 6 23.1% More is known about typicals in kids 4 15.4% Can not be administered as IM’s 3 11.5% Other 1 3.8% Total responses 26 100.0%

First Use Atypical—ObstaclesFirst Use Atypical—Obstacles

Obstacle Count Percent of Responses

Cost 5 23.8% More data supporting typicals 5 23.8%Patient history of non-response to atypicals 4 19.1% Patient resistance 3 14.3%Less available 2 9.5% Other 2 9.5% Total responses 21 100.0%

Favor/Unfavor

Easy/Hard

Improve/No

Agree/Disagree

543210-1-2-3-4-5

2

202

2912

Limit the Use of Stat’s & P.R.N.’s

Descriptive Statistics (n=19)

Min/Max Mean (SD)

Favor/Unfavor -5/5 2.63(2.89)

Easy/Hard -5/5 -0.38(3.22)

Improve/No -2/5 2.44(1.92)

Agree/Disagree -2/5 3.86(1.81)

Limit Stat‘ & P.R.N.’s -- AdvantagesLimit Stat‘ & P.R.N.’s -- Advantages

Advantage Count Percent of Responses

Other (i.e avoids traumatizing patient, 6 27.3% Avoids unnecessary medication 5 22.7% Avoids unnecessary side effects 4 18.2% Allows doctor to better understand patient’s condition 4 18.2% Patient learns techniques they can apply in ‘real life’ 3 13.6% Total responses 22 100.0%

Limiting Stat’s & P.R.N.'s — Limiting Stat’s & P.R.N.'s — DisadvantagesDisadvantages

Disadvantage Count Percent of Responses

Possible safety risk to patient and others 9 2.9%Other (i.e. does not address biological factors 6 28.6%Difficult for staff, who may feel less in control 4 19.0% May need to rapidly sedate patient 2 9.5%

Total responses 21 100.0%

Limiting Stat’s & P.R.N.'s--Limiting Stat’s & P.R.N.'s--ObstaclesObstacles

Obstacle Count Percent of Responses

Safety 8 33.3%Other (i.e.patient belief

that p.r.n.’s condone behavior; 5 20.8%Staff resistance 4 16.7% Patient too aggressive 4 16.7% Staff availability and training 3 12.5%

Total responses 24 100.0%

Favor/Unfavor

Easy/Hard

Improve/No

Agree/Disagree

543210-1-2-3-4-5

25

Monitor Side Effects

Descriptives (n=19)

Min/Max Mean(SD)

Favor/Unfavor 3/5 4.57(.69)

Easy/Hard -2/5 2.94(2.4)

Improve/No 1/5 4.0(1.15)

Agree/Disagree 3/5 4.68(.58)

Use Standardized Scales for Side Use Standardized Scales for Side Effects -- AdvantagesEffects -- Advantages

Advantage Count Percent of Responses

Helps captures side effects you might otherwise miss 8 27.6%Other (i.e. increases patient compliance; improves 6 20.7%

communication between doctors; helps assess severity of side effects)

Provides objective measure 4 13.8%Keeps doctors’ focus on side effects 4 13.8%Determines drug effectiveness for specific symptoms 4 13.8%Enables doctor to track side effects over time 3 10.3% Total responses 29 100.0%

Use Standardized Scales for Side Use Standardized Scales for Side Effects--DisadvantagesEffects--Disadvantages

Disadvantage Count Percent of Responses

Doctor may ignore side effects not on scale 3 27.3% May minimize importance of clinical evaluations 3 27.3% Other (i.e. may make patient more aware of side effects) 3 27.3% Methodological problems (i.e. inter-rater reliability) 2 18.2%

Total responses 1 100.0%

Scales for Side Effects--ObstaclesScales for Side Effects--Obstacles

Obstacle Count Percent of Responses

Time 8 25.0% Scales are complicated/require training 6 18.7%Instrument availability 5 15.6% Other (i.e. staff resistance; instrument availability; 5 15.6%

cost) Administrative barriers 3 9.4% Laziness 3 9.4%Clinician resistance 2 6.3% Total responses 32 100.0%

Self-Efficacy Beliefs

Expected - Values

Behavioral Intention

Normative Beliefs

OBSTACLES Cognition, Habits & Automatic Processes, Knowledge, Behavioral Skills, Decisional Style, Behavioral Salience

Mental Contrasting

Implementation Intentions

Behavior

Expected Effects

Possible Effects

INTERVENTION

New Models for Behavior Change: TMC, TII (Gollwitzer, Oettingen, Jaccard, Jensen et al, 2002; Perkins et

al., 2007)

Mental Contrasting/Implementation Mental Contrasting/Implementation IntentionsIntentions

1. Use mental contrasting to strengthen behavioral intentions:

“What are the advantages or positive consequences associated with the use of Guideline X”

2. Identify Obstacles:“What gets in the way of implementing guideline X”3. Form Implementation Intentions to overcome

obstacles:“If I encounter obstacle Y, then I will X.”

Track Target SymptomsTrack Target Symptoms

Favor/Unfavor

Easy/Hard

Improve/No Improve

Agree/Disagree

543210-1-2-3-4-5

Pre-Intervention Post-Intervention

Descriptive Statistics (n=4)

1/5 3.5(1.9)0/4 1.8(1.7)1/4 2.5(1.3)3/5 4.3(1.0)

Favor/UnfavorEasy/HardImprove/No ImproveAgree/Disagree

Min/Max Mean(SD)

Descriptive Statistics (n = 4)

1/5 3.0(1.6)-3/1 -0.5(1.9) 2/3 2.8(0.5) 3/4 3.3(0.5)

Favor/UnfavorEasy/HardImprove/No ImproveAgree/Disagree

Min/MaxMean(SD)

Favor/Unfavor

Easy/Hard

Improve/No Improve

Agree/Disagree

543210-1-2-3-4-5

Descriptive Statistics (n = 4)

4/5 4.8(0.5) -5/5 3.3(2.9) 4/5 4.8(0.5) 5/5 5.0(0.0)

Favor/UnfavorEasy/HardImprove/No ImproveAgree/Disagree

Min/MaxMean(SD)

Descriptive Statistics (n=4)

5/5 5.0(0.0)1/5 3.5(1.9)5/5 5.0(0.0)5/5 5.0(0.0)

Favor/UnfavorEasy/HardImprove/No ImproveAgree/Disagree

Min/Max Mean(SD)

Use A Conservative Dosing StrategyUse A Conservative Dosing Strategy

Favor/Unfavor

Easy/Hard

Improve/No Improve

Agree/Disagree

543210-1-2-3-4-5

Favor/Unfavor

Easy/Hard

Improve/No Improve

Agree/Disagree

543210-1-2-3-4-5

Pre-Intervention Post-Intervention

Limit the Use of P.R.N.sLimit the Use of P.R.N.s

Favorable/Unfavorabl

Easy/Hard

Improve/No Improve

Agree/Disagree

543210-1-2-3-4-5

Descriptive Statistics (n=4)

3/5 4.5(1.0)0/4 2.0(1.8)1/5 3.3(1.7)4/5 4.8(0.5)

Favor/UnfavorEasy/HardImprove/No ImproveAgree/Disagree

Min/Max Mean(SD)

Descriptive Statistics (n = 4)

-3/5 2.5(3.8)-5/5 -0.8(4.2) 2/5 3.8(1.5) 3/5 4.5(1.0)

Favor/UnfavorEasy/HardImprove/No ImproveAgree/Disagree

Min/MaxMean(SD)

Favor/Unfavor

Easy/Hard

Improve/No Improve

Agree/Disagree

543210-1-2-3-4-5

Pre-Intervention Post-Intervention

Intention to Use Guidelines in the Intention to Use Guidelines in the Next Month (n=4)Next Month (n=4)

Guideline Pre-Intervention Post-Intervention

Track Target Symptoms 4.6(2.89) 8.25(2.1)

Conservative Dosing Strategy

8.8(1.30) 10.00(.0)

Limit P.R.N. 5.6(3.64) 8.75(.96)

Track Side Effects 9.6(.89) 8.75(1.5)

Three Levels:

Child & Family Factors: e.g., Access & Acceptance

Provider/Organization Factors: e.g., Skills, Use of EB

Systemic and Societal Factors: e.g., Organiz., Funding Policies

Barriers vs. “Promoters” to Barriers vs. “Promoters” to

Delivery of Effective Services Delivery of Effective Services (Jensen, 2000)(Jensen, 2000)

“Effective” Services

Efficacious Treatments

CLINIC/COMMUNITY INTERVENTION CLINIC/COMMUNITY INTERVENTION DEVELOPMENT AND DEPLOYMENT MODEL DEVELOPMENT AND DEPLOYMENT MODEL

(CID) (Hoagwood, Burns & Weisz, 2000)(CID) (Hoagwood, Burns & Weisz, 2000) Step 1: Theoretically and clinically-informed construction, refinement, and manualizing of the protocol within the context of the practice setting where it is ultimately to be deliveredStep 2: Initial efficacy trial under controlled conditions to establish potential for benefitStep 3: Single-case applications in practice setting with progressive adaptations to

the protocolStep 4: Initial effectiveness test, modest in scope and costStep 5: Full test of the effectiveness under everyday practice conditions, including cost effectiveness Step 6: Effectiveness of treatment variations, effective ingredients, core potencies, moderators, mediators, and costsStep 7: Assessment of goodness-of-fit within the host organization, practice setting, or communityStep 8: Dissemination, quality, and long-term sustainability within new organizations, practice settings, or communities

Partnerships & Collaborations Partnerships & Collaborations in Community-Based Research in Community-Based Research

Why Partnerships? partnerships -- not with other scientists per se, but with experts

of a different type -- experts from families, neighborhoods, schools, in communities.

Only from these experts can we learn what is palatable, feasible, durable, affordable, and sustainable for children and adolescents at risk or in need of mental health services

“Partnership” - changes in typical university investigator - research subject relationship

Practice – based Research Networks Bi-directional learning

Traditional approach research question posed, building on theory and body of

previous research logical next step in elegant chain of hypotheses, tests, proofs,

and/or refutations isolation of variables from larger context; limit potential

confounds and alternative explanations of findings study designed, investigator then looks for “subjects” who will

“recipients of the bounty” cannot answer questions about sustainability unidirectional blind to issues of ecological validity

Partnerships & Collaborations Partnerships & Collaborations in Community-Based Research in Community-Based Research

Alternative (collaborative) approach expert-lay distinction dissolved both partners bring critical expertise to research agenda

research methods and technical expertise from the university investigator

systems access and local-ecological expertise from the community collaborator

so-called “confounds” can provide useful “tests” of the feasibility, durability, and generalizability of the intervention

hence, importance of replication improved validity of knowledge obtained?

Partnerships & Collaborations Partnerships & Collaborations in Community-Based Research in Community-Based Research

The REACH Institute….The REACH Institute….Putting Science to WorkPutting Science to Work

- Problem area identification

- Bring key “change agents” and gatekeepers to the table (federal or state partners, consumer and professional organizations)

- Identify “actionable” knowledge among experts and “consumers”

- Identify E-B QI procedures that are feasible, sustainable, palatable, affordable, transportable

- Consumer and stakeholder “buy-in” & commitment to E-B practices

- Dissemination via partners across all 3 system levels - “with an edge” (policy/legislative strategy with relevant federal/state partners)- Training and TA/QI intervention;

all sites eventually get intervention.

- Monitoring/fidelity

- Report preparation

- Results fed back into Step II.

Step I Step II

- Site recruitment and preparation within “natural replicate” settings

- Tool preparation, fidelity/monitoring

- ”Skimming the cream,” first taking those sites most ready

Step III

Step IV

Design ConsiderationsDesign Considerations

“Begin with the end in mind” – CID model Enemy of the good is the perfect: raise the floor, not the ceiling “Randomized encouragement trials” vs. randomized controlled

trials Quality Improvement group vs. TAU

How does one know the necessary ingredients of change? Attention – Expectations – Hawthorne effects? Measure them Attention dose, time in treatment? Measure them Measure change processes

Assuring fidelity to model? Measure it Ensure therapeutic relationship…and measure it Ensure family buy-in and therapist buy-in. Measure it

Need for two controls? TAU, attention control group

Overcoming Challenges: Overcoming Challenges: A Motivational ApproachA Motivational Approach

Change implementation strategies based on motivational approaches - William Miller

Practice what you preach Express empathy

to challenges of policy makers and practitioners in implementing change with population

Develop discrepancy between ideal and current Success of evidence-based treatment must be

explainable, straightforward, simply stated, meaningful

Overcoming Challenges: Overcoming Challenges: A Motivational ApproachA Motivational Approach

Avoid argumentation Clinician scientists credible to policy makers

and community-based practitioners Avoid overstating the case and “poisoning

the well” Roll with resistance

Develop strategies for engagement, prepare for possible resistance

Foundation of Collaborative EffortsFoundation of Collaborative Efforts

Researcher driven

Research retains

Research skills designated as primary

One-wayUnbalanced

Continual suspicion

Shared; equal investment

Recognition of contribution by community member &

& researchers

Open; opportunities to discuss & resolve conflict

Belief in the good faith of partners; room for mistakes

Fairly distributed

Goals

Power

Skills

Communication

Trust

Degrees of collaborationDegrees of collaboration

Focus groupsCommunity Advisors or Advisory

Board

Community partners as paid

staffCollaboration

(+) identificationof pressing community/familyneeds(+) definition ofacceptable researchprojects or serviceinnovations

(+) provides ongoing input regarding various stages ofresearch process

(+) collaboration regarding implementation of project(+) access to researchers to provide guidance asobstacles encountered

(+) co-creationco-implementationco-evaluationco-dissemination

Points of Collaboration in the Points of Collaboration in the Research ProcessResearch Process

Study Aims Research design& sampling

Measurement& Outcomes

Procedures (recruit, retain, data

collectionImplementation Evaluation Dissemination

Definedcollaboratively

OR

Advice sought

OR

Researcherdefined

Decision madejointly

OR

Researchereducateson methods &advice sought

OR

Methods pre-determined

Defined withinpartnership

OR

Advicesought

OR

Researcherdefined

Shared responsibility (e.g. communityto recruit, researchstaff to collect data)

OR

Designed withinput

OR

Designed byresearchers

Projects areco-directed

OR

Researcherstrain community members asco-facilitators

OR

Research staffhired for project

Plans for analysisco-created to ensurequestions of bothcommunity & researchers answered

OR

Community members assist in interpretationof results

OR

Researchers analyzedata

Members ofpartnershipdefine disseminationoutlets

OR

Members of community fulfill co-author& co-presenter roles

OR

Researchers present at conferences &publish

The REACH Institute….The REACH Institute….Putting Science to WorkPutting Science to Work

- Problem area identification

- Bring key “change agents” and gatekeepers to the table (federal or state partners, consumer and professional organizations)

- Identify “actionable” knowledge among experts and “consumers”

- Identify E-B QI procedures that are feasible, sustainable, palatable, affordable, transportable

- Consumer and stakeholder “buy-in” & commitment to E-B practices

- Dissemination via partners across all 3 system levels - “with an edge” (policy/legislative strategy with relevant federal/state partners)- Training and TA/QI intervention;

all sites eventually get intervention.

- Monitoring/fidelity

- Report preparation

- Results fed back into Step II.

Step I Step II

- Site recruitment and preparation within “natural replicate” settings

- Tool preparation, fidelity/monitoring

- ”Skimming the cream,” first taking those sites most ready

Step III

Step IV