The importance of therapist effects in the treatment of children and adolescents G.S. (Jeb) Brown,...

The importance of therapist effects in the treatment of children and adolescents

G.S. (Jeb) Brown, Ph.D.

Center for Clinical Informatics

Overview

• Findings based on analyses of data collected through the PacifiCare Behavioral Health/United Behavioral Health ALERT clinical information system

• Novel methodology demonstrates that much of the variance in outcomes resides with the individual psychotherapist.

Therapist effects

• Therapist differ in their “effectiveness” resulting in wide differences in outcomes. Results cannot be explained by theoretical orientation,

treatment methods, years of training or experience.

• The effectiveness of all treatments, including medications, are mediated by therapist effects.

• Failure to measure and account for therapist effects is simply BAD SCIENCE!

Outcome Measures

• 30-item patient self-report questionnaires Life Status Questionnaire (adults) Youth Life Status Questionnaire (children)

• Administered at 1st, 3rd, 5th sessions and every 5th session there after

• Over 20,000 questionnaires per month come into the system

Case mix

• Case mix variables are those variables present at the beginning of the treatment episode that are predictive of the outcome

• Intake score accounts for 18% of variance in change scores in PBH data

• Addition of age, sex and diagnosis to predictive model accounts for < 2% additional variance

Where’s the variance?% of variance in change scores

Unexplained 80%(external stressors,health condition,social supports, etc.)

Acuity (intake score)18%

Diagnosis, age & sex2%

Case mix adjustment

Where else is the variance?

Unexplained 74% (externalstressors, health condition, socialsupports, etc.)

Acuity (intake score) 18%

Diagnosis, age & sex 2%

Clinician 6%

Treatment method 1%

Clinicians & treatments

• 1046 Provider Locations on Honor Roll

• 56 Group Practices

• 740 Individuals (not including those in groups)

• These providers accounted for about 30% of PBH outpatient episodes in 2004

Honors for Outcomes

• Honors for Outcomes Selection Criteria: – Minimum of 10 cases with two Y/LSQ data points in past

3 years– Average patient change must be reliably above average:

65% confidence that the provider’s Change Index >0 – Change Index is a case-mix adjusted measure, compares

outcomes to PBH’s large normative database

• Honors for Outcomes is updated quarterly

Honor Rating Method

Website

Honors for Outcomes - Search

Honors for Outcomes - Results

• Honors for Outcomes depends on predictive validity of Honors rating; prior performance predicts future performance

• Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents?

• Implication if yes: Therapists’ effectiveness is likely to be global in nature rather than specific to age and or diagnostic group.

Study Question 1

• Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents on medications?

• Implication if yes: The therapist effectiveness of the therapists is apparently mediating the effect of the medication(s).

Study Question 2

• Use Honors for Outcomes methodology to rank clinicians based on their outcomes with adult patients only.

• Therapist included in the study if they treated at least one child/adolescent with psychotherapy only and one with psychotherapy plus medication. (929 Honors, 1352 Non-Honors)

• Compare outcomes for children and adolescents for Honors clinicians to other clinicians.

Study Method

Result: Outcomes for adults predicts outcomes for children

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0-41 42-120

Intake scores

Eff

ec

t s

ize

Honors-psychotherapyonly

Honors-psychotherapyand medication

Non-Honors-psychotherapy only

Non-Honors-psychotherapy andmedication

mild symptoms moderate to severe symptoms

Results after adjusting for intake score, age, sex, diagnosis and

prior treatment history.

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0-41 42-120

Re

sid

ua

l eff

ec

t s

ize



Non-Honors-psychotherapyonly

Non-Honors-psychotherapyand medication

moderate to severe symptomsmild symptoms

All diagnoses and medications

Delta residual N Delta residual N

Honors-psychotherapy only

1.9 2.3 430 12.5 2.3 286

Honors-psychotherapy and medication

-1.3 -1.3 79 15.3 2.7 134


-0.9 -0.5 565 8.3 -2.2 449

Non-Honors-psychotherapy and

medication-1.2 -2.7 102 10.3 -1.5 186

Intake score below mean Intake score at mean or above

Children diagnosed with depression and treated with psychotherapy alone or in combination with an

antidepressant

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0-41 42-120

Re

sid

ua

l eff

ec

t s

ize

Honors-psychotherapyonlyHonors-psychotherapyand medicationNon-Honors-psychotherapyonlyNon-Honors-psychotherapyand medication

Depression & antidepressants



2.6 3.5 77 15.4 4.6 84


0.11 0.3 28 15.5 2.9 41


-1.7 -2.9 87 9.2 -2 123


medication-1.7 -3.2 27 11.1 -0.9 53


Implications for clinicians

• The effectiveness of the individual clinician is the major source of variance in outcomes.

• All treatments (including medications) are only as effective and the clinicians delivering the treatment.

• Clinicians have an ethical responsibility to assess and improve their personal effectiveness as clinicians… they cannot rely on the treatments alone to be curative.

Implications for administrators & policy makers

• Exclusive focus on the effectiveness of treatments rather than clinicians limits the potential to improve outcomes.

• Administrators and policy makers have an obligation to consumers to assure that they have access to effective clinicians.

• Failure to monitor outcomes at the clinician level places consumers at risk.

http://[email protected]

1821 Meadowmoor Rd.Salt Lake City, UT 84117

Voice 801-541-9720

Suggested readings

Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. Journal of Counseling Psychology; 2001: 48, 251-257.Blatt SJ, Sanislow CA, Zuroff DC, Pilkonis PA. Characteristics of effective therapists: Further analyses of data from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology; 1996: 64, 1276-1284.Brown GS, Burlingame GM, Lambert MJ, et al. Pushing the quality envelope: A new outcomes management system. Psychiatric Services; 2001: 52 (7), 925-934.Brown GS, Herman R, Jones ER, Wu J. Improving substance abuse assessments in a managed care environment. Joint Commission Journal on Quality and Safety; 2004: 30(8), 448-454.Brown GS, Jones ER, Betts W, Wu J. Improving suicide risk assessment in a managed-mare environment. Crisis; 2003: 24(2), 49-55. Brown GS, Jones ER, Lambert MJ, Minami T. Identifying highly effective psychotherapists in a managed care environment. American Journal of Managed Care, 2005:11(8):513-20 . Brown GS, Jones ER. Implementation of a feedback system in a managed care environment: What are patients teaching us? Clinical Psychology/In Session: 2005: 61(2), 187-198.Burlingame GM, Jasper BW, Peterson G, et al. Administration and Scoring Manual for the YLSQ. Wilmington, DL, American Professional Credentialing Services; 2001.Crits-Christoph P, Mintz J. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 1991; 59, 20-26.Crits-Christoph P., Baranackie K., Kurcias JS et al. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research; 1991: 1, 81-91.Elkin I. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical Psychology: Science and Practice; 1999: 6, 10-32.

• 1046 Provider Locations on Honor Roll

• 56 Group Practices

• 740 Individuals (not including those in groups)

• These providers accounted for about 30% of PBH outpatient episodes in 2004

Honors for Outcomes

• Honors for Outcomes Selection Criteria: – Minimum of 10 cases with two Y/LSQ data points in past

3 years– Average patient change must be reliably above average:

65% confidence that the provider’s Change Index >0 – Change Index is a case-mix adjusted measure, compares

outcomes to PBH’s large normative database

• Honors for Outcomes is updated quarterly

Honor Rating Method

Website

Honors for Outcomes - Search

Honors for Outcomes - Results

• Honors for Outcomes depends on predictive validity of Honors rating; prior performance predicts future performance

• Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents?

• Implication if yes: Therapists’ effectiveness is likely to be global in nature rather than specific to age and or diagnostic group.

Study Question 1

• Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents on medications?

• Implication if yes: The therapist effectiveness of the therapists is apparently mediating the effect of the medication(s).

Study Question 2

• Use Honors for Outcomes methodology to rank clinicians based on their outcomes with adult patients only.

• Therapist included in the study if they treated at least one child/adolescent with psychotherapy only and one with psychotherapy plus medication. (929 Honors, 1352 Non-Honors)

• Compare outcomes for children and adolescents for Honors clinicians to other clinicians.

Study Method

Result: Outcomes for adults predicts outcomes for children

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0-41 42-120

Intake scores

Eff

ec

t s

ize




Non-Honors-psychotherapy andmedication

mild symptoms moderate to severe symptoms

Results after adjusting for intake score, age, sex, diagnosis and

prior treatment history.

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0-41 42-120

Re

sid

ua

l eff

ec

t s

ize



Non-Honors-psychotherapyonly

Non-Honors-psychotherapyand medication

moderate to severe symptomsmild symptoms

All diagnoses and medications



1.9 2.3 430 12.5 2.3 286


-1.3 -1.3 79 15.3 2.7 134


-0.9 -0.5 565 8.3 -2.2 449


medication-1.2 -2.7 102 10.3 -1.5 186


Children diagnosed with depression and treated with psychotherapy alone or in combination with an

antidepressant

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0-41 42-120

Re

sid

ua

l eff

ec

t s

ize

Honors-psychotherapyonlyHonors-psychotherapyand medicationNon-Honors-psychotherapyonlyNon-Honors-psychotherapyand medication

Depression & antidepressants



2.6 3.5 77 15.4 4.6 84


0.11 0.3 28 15.5 2.9 41


-1.7 -2.9 87 9.2 -2 123


medication-1.7 -3.2 27 11.1 -0.9 53


Implications for clinicians

• The effectiveness of the individual clinician is the major source of variance in outcomes.

• All treatments (including medications) are only as effective and the clinicians delivering the treatment.

• Clinicians have an ethical responsibility to assess and improve their personal effectiveness as clinicians… they cannot rely on the treatments alone to be curative.

Implications for administrators & policy makers

• Exclusive focus on the effectiveness of treatments rather than clinicians limits the potential to improve outcomes.

• Administrators and policy makers have an obligation to consumers to assure that they have access to effective clinicians.

• Failure to monitor outcomes at the clinician level places consumers at risk.

http://[email protected]

1821 Meadowmoor Rd.Salt Lake City, UT 84117

Voice 801-541-9720

Suggested readings

Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. Journal of Counseling Psychology; 2001: 48, 251-257.Blatt SJ, Sanislow CA, Zuroff DC, Pilkonis PA. Characteristics of effective therapists: Further analyses of data from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology; 1996: 64, 1276-1284.Brown GS, Burlingame GM, Lambert MJ, et al. Pushing the quality envelope: A new outcomes management system. Psychiatric Services; 2001: 52 (7), 925-934.Brown GS, Herman R, Jones ER, Wu J. Improving substance abuse assessments in a managed care environment. Joint Commission Journal on Quality and Safety; 2004: 30(8), 448-454.Brown GS, Jones ER, Betts W, Wu J. Improving suicide risk assessment in a managed-mare environment. Crisis; 2003: 24(2), 49-55. Brown GS, Jones ER, Lambert MJ, Minami T. Identifying highly effective psychotherapists in a managed care environment. American Journal of Managed Care, 2005:11(8):513-20 . Brown GS, Jones ER. Implementation of a feedback system in a managed care environment: What are patients teaching us? Clinical Psychology/In Session: 2005: 61(2), 187-198.Burlingame GM, Jasper BW, Peterson G, et al. Administration and Scoring Manual for the YLSQ. Wilmington, DL, American Professional Credentialing Services; 2001.Crits-Christoph P, Mintz J. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 1991; 59, 20-26.Crits-Christoph P., Baranackie K., Kurcias JS et al. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research; 1991: 1, 81-91.Elkin I. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical Psychology: Science and Practice; 1999: 6, 10-32.

Suggested readings (continued)

Hannan C, Lambert MJ, Harmon C, Nielsen SL, Smart DW & Shimokawa K, Sutton SW. A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology/In Session: 2005: 61(2), 155-164.Harmon C, Hawkins, EJ, Lambert, MJ, Slade K & Whipple JL. Improving outcomes for poorly responding clients: the use of clinical support tools and feedback to clients. 61(2), 175-186.Huppert JD, Bufka LF, Barlow DH, Gorman JM, Shear MK, Woods SW. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology; 2001: 69, 747-755.Kim DM, Wampold BE, Bolt DM. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research: 2006: 12(2), 161-172.Lambert MJ., Whipple J., Smart, DW et al (Vermeersch, D. A., Nielsen, S.L., & Hawkins, E. J.) The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research; 2001: 11, 49-68.Lambert MJ, Harmon C, Slade K, Whipple JL & Hawkins EL. Providing feedback to psychotherapists on their patients’ progress: clinical results and practice suggestions. Journal of Clinical Psychology/In Session: 2005: 61(2), 165-174. Lambert MJ, Hatfield DR, Vermeersch DA., et al. Administration and scoring manual for the LSQ (Life Status Questionnaire). East Setauket, NY: American Professional Credentialing Services; 2001.Lambert MJ, Whipple JL, Hawkins EJ et al. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science & Practice; 2003: 10, 288-301.Lambert MJ. Emerging methods for providing clinicians with timely feedback on treatment effectiveness. Journal of Clinical Psychology/In Session: 2005: 61(2), 141-144.

Suggested readings (continued)

Luborsky L, Crits-Christoph P, McLellan T et al. Do therapists vary much in their success? Findings from four outcome studies. American Journal of Orthopsychiatry; 1986: 56, 501-512. Luborsky L, Rosenthal R, Diguer L et al. The dodo bird verdict is alive and well--mostly. Clinical Psychology: Science & Practice; 2002: 9(1) 2-12.Matsumoto K, Jones E, Brown, GS. Using clinical informatics to improve outcomes: A new approach to managing behavioral healthcare. Journal of Information Technology in Health Care; 2003: 1(2), 135-150Okiishi J, Lambert MJ, Nielsen SL, Ogles BM. Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology and Psychotherapy; 2003: 10, 361-373.Porter ME & Teisberg, EO. Redefining competition in health care. Harvard Business Review,2004:65-76.Shapiro DA, Shapiro, D. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Journal of consulting and Clinical Psychology; 1982: 92, 581–604.Vermeersch DA, Lambert MJ, Burlingame GM. Outcome Questionnaire: Item sensitivity to change. Journal of Personality Assessment; 2002: 74, 242-261. Wampold BE, Brown GS. Estimating therapist variability: A naturalistic study of outcomes in private practice. Journal of Consulting and Clinical Psychology; 2005: 75(5) pp 914-923. Wampold BE, Mondin GW, Moody M et al. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “all must have prizes.” Psychological Bulletin; 1997: 122, 203-2154.

About the presenter

G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 19987. He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country.

In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include PacifiCare Behavioral Health/ United Behavioral Health, Department of Mental Health for the District of Columbia, Accountable Behavioral Health Care Alliance, Resources for Living and assorted treatment programs and centers throughout the world.

Dr. Brown continues to work as a part time psychotherapist at behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.

The importance of therapist effects in the treatment of children and adolescents G.S. (Jeb) Brown,...

Documents

Transcript of The importance of therapist effects in the treatment of children and adolescents G.S. (Jeb) Brown,...