Becoming An Outcomes Informed Clinician

G.S. (Jeb) Brown, Ph.D.Center for Clinical InformaticsDirector & Senior Consultant

Scott Williams, Ph.D.Center for Clinical InformaticsSenior Consultant

An outcomes informed clinician…• Uses the best available data on treatment outcomes

to inform the treatment for each client/patient

• Recognizes the importance of clinician skill in providing effective treatments.

• Accepts personal responsibility for evaluating and improving his or her outcomes.

• Believes that clinicians should be able to demonstrate their value providing evidence of effectiveness.

Sources of outcome data…

• Clinical trials designed to draw causal conclusions regarding the efficacy of various treatments.

• Meta analyses designed to draw conclusions based on a large sample of clinical trials.

• Studies evaluating the effectiveness of treatments delivered in the real world.

• Data from our own practices!

Clinical trials

• Random assignment and experimental controls designed to control of sources of variance in outcomes.

• Double blind placebo controlled considered the “gold standard”

• Use of analysis of variance to determine if differences between treatments are “statistically significant”, i.e. unlikely to occur by chance alone.

Patient factors Treatment method, dose, duration, etc. Dose, duration, etc.

Where’s the variance

• Analysis of variance permits us to estimate of the contribution of each variable to the outcome of care.

• Analysis of variance can only calculate the percentage of variance for those variables specified in the hypotheses or “model”

Warning!Failure to specify all sources of variance in the model may lead to erroneous and misleading findings.

Hierarchical Linear Modeling

• Traditional analysis of variance evaluating treatments as the primary source of variance is correct only if the researcher is sure that the clinician does not matter!

• Psychotherapy research shows that the clinician matters…. A lot! 1-9

• If the clinician may be a source of variance, then it is necessary to use a hierarchical linear model which specifies the clinician as a variable and possible source of variance.

Reanalyzes – using HLM

• 3.4% of variance due to medication; 9.1% due to the psychiatrist

• Placebo - imipramine comparison performed including the 9 psychiatrists as a variable

• Top third of psychiatrists had a better outcome with placebo than the bottom third with imipramine

McKay, K. M., Imel, Z. E., & Wampold, B. E. (2006). 10 Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders, 92, 287-290.

• Re-analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data

Real world data

• Analysis of PacifiCare Behavioral Health (PBH) data reveals 5% of variance due to therapist for patients receiving psychotherapy alone.

• 35 % of variance due the therapist if patient receiving psychotherapy in combination with a medication!

Wampold BE & Brown GS. 2005. 11

Estimating variability in outcomes due to the therapist: A naturalistic study of outcomes in managed care. J Consul Clin Psychol. 73(5): 914-923.

Evidenced based psychotherapy

• For several decades psychotherapy researchers have attempted to design randomly controlled trials (RCT) to investigate the effectiveness of specific methods of psychotherapy.

• Study design analogous to pharmacy trials, except that designing credible “placebo treatments” is much more problematic.

• Various treatment methods are being touted as “evidenced based” by citing the number of RCTs providing evidence that the treatment exceeded placebo (or some other treatment).

Brand differentiation

• Advocates of psychotherapy brands insist on the uniqueness of their therapy and the need to adhere to specific treatment procedures

• Research methodology requires the use of manuals and other techniques to standardize treatments

• Treatment effectiveness presumed to be dependent on the correct application of the “active ingredients” in the psychotherapy method.

Recommended reading

We are not merely technicians!

We are the active ingredient!

Much more variance resides with the clinician than with the treatments.

Clinical trials – feedback studies• Michael Lambert, PhD and colleagues at the

Brigham Young Comprehensive Clinic conducted a series of controlled studies investigating whether providing clinician’s feedback the client’s trajectory of change would improve outcomes. 12-17

• Feedback found to significant reduce early dropout and treatment failures.

• Clinicians’ judgment alone, in the formed in absence of information from the questionnaires, proved to be a poor predictor treatment failures.

Outcomes informed care• Frequent administration of patient self report outcome

questionnaires to monitor patient response to treatment leads to improved outcomes for at risk cases.

• Recognition of the clinician as the most important “active ingredient” in the treatment process leads to the use decision support tools and performance feedback to improve the effectiveness of clinicians.

• Use of practice based evidence (as opposed to evidence base practices) permits an organization to evaluate what works and identify pathways to improved outcomes.

• ACORN non-profit organization offers tools for the outcomes informed clinician.

ACORN Questionnaires

• Brief, reliable and valid

• Online manual

• Flexibility to choose items that best meet your measurement needs

• Normative information updated regularly

Traditional Psychotherapy Process

Therapist applies Evidence-Based Treatment model

Outcomes Informed Care

The therapist uses the information to inform the treatment process.

The therapist may also use an session feedback questionnaire to gather more information on the clients perception of the working alliance (a strong predictor of treatment outcome).

Outcomes Informed Care

What’s in it for the clinician?

??

?? ? ?

Effectiveness studies

• Effectiveness studies evaluate outcomes in real world treatment settings.

• Heterogeneous outpatient treatment populations require use of statistical methods for case mix adjustment in order to compare results across sites or clinicians.

• Outcomes “benchmarking” refers to the practice of comparing outcomes from one sample to outcomes of another sample used for comparison purposes.

Benchmarking outcomes

• Takuya Minami, PhD and colleagues have published a series of articles describing one benchmarking methodology. 18-20

• Meta-analyses of controlled psychotherapy studies used to establish effect size benchmarks for the treatment of depression.

• Outcomes for treatment of depression by PacifiCare Behavioral Health’s network providers found clinical equivalent to benchmark from clinical trials.

Better outcomes?

• Regence outcomes informed care initiative tailored for clinicians who wish to demonstrate the value of their services

• Self selection may result in a sample of highly effective clinicians

Regence Outcomes for Depression(Adults)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Clinical trialsof

psychotherpyfor

depression(Minami et al,

2007)

Managed CareNetw ork

(Mianmi et al,2008)

RegenceClinicians

Eff

ect

Siz

e

Effective range

Highly effective range

• The results?

References

1. Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary much in their success? Findings from four outcome studies. Am J Orthopsychiatry 56:501-12.

2. Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1:81-91.

3. Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consul Clin Psychol 59:20-6.

4. Wampold BE. 1997. Methodological problems in identifying efficacious psychotherapies. Psychother Res 7:21-43,

5. Elkin I. 1999. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32.

References (continued)

6. Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested factor on measures of effect size in analysis of variance designs. Psychol Methods 4:425-33.

7. Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55.

8. Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink: An empirical analysis of therapist effects. Clin Psychol Psychother 10:361-73.

9. Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychother Res. 16(2): 161-172.

10. McKay, K. M., Imel, Z. E., & Wampold, B. E. (2006). Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders, 92, 287-290.

References (continued)

11. Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to the therapist: A naturalistic study of outcomes in managed care. J Consul Clin Psychol. 73(5): 914-923.

12. Lueger RJ. 1998. Using feedback on patient progress to predict the outcome of psychotherapy. J Clin Psychol 54:383-93.

13. Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.

14. Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74.

15. Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85.

References (continued)

16. Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74.

17. Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85.

18. Minami, T., Serlin, R. C., Wampold, B. E., Kircher, J. C., & Brown, G. S. (2008). Using clinical trials to benchmark effects produced in clinical practice, Quality and Quantity

19. Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E., Brown, G. S., & Kircher, J. (2007). Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment.

20. Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007). Benchmarks for psychotherapy efficacy in adult major depression, Journal of Consulting and Clinical Psychology, 75, 232-243.