Quality Measures for Rehabilitation: Policy, Provider and Patient Perspectives

Measuring Clinical Change: Quality Indicators

ACRM-ASNR Pre-Conference Institute

Kenneth J. Ottenbacher

University of Texas Medical Branch

Disclosures

National Institutes of Health Grants: R24 HD065702; R01 AG017638; K12 HD055929; T32 HD007539PI: Kenneth J. Ottenbacher

National Center for Medical Rehabilitation Research (NICHD), National Institute on Neurological Disorders and Stroke, and National Institute on Aging

Department of Education Grants: H133G080163; H133P110012 PI: Kenneth J. Ottenbacher

National Institute on Disability and Rehabilitation Research 

Learning Objectives

• Describe three approaches to measuring clinical change

• Discuss the difference between clinical change at the individual level and facility level

• Identify challenges in measuring clinical change

How NOT to measure clinically important change:

Measuring clinically important change is complex. Multiple methods are available with various advantages and disadvantages.

p<.05

Statistical Significance versus Clinical Importance

Design sensitivity: statistical power for

experimental research Mark W. Lipsey

The Cult of Statistical Significance

Stephen T. Ziliak & Deirdre N. McCloskey

Investigators and clinicians have a long history of interpreting statistically significant findings as also being clinically significant.

Example: Smith reports a rehabilitation outcome study with statistically significant results. Jones conducts a replication study with (null) non-significant results.

Assume that both trials are well-designed and their internal validity is good.

Results:

Smith, t = 2.21 (df = 78, p < 0.05)

Jones, t = 1.06 (df = 18, p = ns)

The interpretation is that Smith found a statistically significant difference (change) in the outcome measure and Jones did not find a statistically significant difference (change).

And… that the treatment produced a significant change in the Smith study, but not the Jones study.

An alternative way to look at the results of the Smith and Jones studies is to examine the degree to which the null hypothesis is false in both investigations.

Cohen (1988) refers to this as the “effect size.”

If we compute an effect size for each study, we will find the following:

Formula: d = 2(t)/ df

Smith d = 0.50

Jones d = 0.50

The effect size is the same for the two trials - the degree to which the null hypothesis is false is identical for both the Smith and Jones study.

The amount of change associated with the outcome is the same in both studies.

Why are the findings in one study statistically significant and the other study non-significant?

Things are not always what they appear to be.

Power for the Jones trial = 0.18

The chance of a Type 2 error is 0.82 (82%)

From: Day, R.H. (1972). Science, Vol. 175, p.1335.

Reading made Don Quixote a gentleman, but believing what he read made him insane.

G. B. Shaw

Who is the Target:

1. Assessment of individual change

2. Assessment of facility change

3. Assessment of population change

 

Measuring Clinical Change

Approaches to Assessing (individual) Change

• Comparison of change in raw scores or ratings

• Effect size (standardized mean difference)

• Reliable change index (Minimal detectable change)

• Minimal clinically important difference• Standardized regression-based change

scores

There are two approaches to measuring (individual) change.

• Distribution-based approaches that rely on expressing change scores in terms of an underlying sampling distribution.

• Anchor-based approaches that require an external, independent standard to ‘anchor’ the meaning of clinical

importance.

Examples:

Reliability Change Index (RCI)

Need the following information: a) baseline score, b) follow-up score, and c) the standard error of measurement (SEM) for the test. (SEM = S1 1-r) where S1 = standard deviation, r = reliability coefficient.

RCI = (X2 – X1) / SEM

The RCI provides an estimate of whether a change is greater than would be expected by chance (measurement error).

Examples:

Minimal Clinically Important Difference (MCID)

Requires identifying a clinically important external standard. For example, 5 motor FIM points. 5 FIM points represents ~ 5% of total scale range on the motor domain. This standard can be based on clinical experience, research literature, patient goals, or some combination.

Use of UDS burden of care values to determine external anchor.

Mean Decrease in Minutes of Help Per Day per FIMTM Point Change

Spinal Cord Injury Study, 1990-91 Hamilton et. al.

13 - 26 26 - 39 39 - 52 52 - 65 65 - 78 78 - 91

0

1

2

3

4

55

6

Min

ute

s o

f H

elp

Minutes per FIM Point

Motor FIM Score Ranges

UDSMR

Converting individual measures of clinical change to facility or population measures of clinical change (quality indicators).

Examples:

Percent of persons in a Case Mix Group achieving Minimal Detectable Change or Reliability Change Index criteria.

Proportion of persons in a Case Mix Group achieving a Minimal Clinically Important Difference.

Examples (continued):

Percent of persons in a Case Mix Groupachieving above benchmark expectations.

Polar GraphAdmission

Discharge

Benchmark

Ea GrBa

DU

DL

To

Bl

BoCh TToT

TuTWC

St

Co

Ex

Si

PsM e

UDSMR

Challenges:

1. How to risk Adjust?

2. How to make the process patient centered?

3. What are the unintended consequences?

The End

Doing outcomes research is a lot like raising children… you always think you are going to do a better job next time.

http://rehabsciences.utmb.edu/R24/ACRM.asp