2 Test Utility Statistics

Medical Decision Making and

Laboratory Statistics

Gregory Tetrault, M. D.Associate Professor

Pathology and Laboratory Medicine

UTHSC

2

Objectives

Bayes’ Theorem and medical decision making

Truth tables and test utility statistics

Assess clinical utility of a new test

Calculate confidence intervals

Receiver-Operator Characteristic curves

3

Bayes’ Theorem Use of symbols and logic to describe the

relationships among simple and conditional probabilities Simple probability: disease prevalence 5% Conditional probability: likelihood that a

test with 95% sensitivity will detect disease

€

P A | B( ) =P(B | A)P(A)

P(B | A)P(A) +P(B | ′ A )P( ′ A )

P = probability

A = condition (such as patient has disease)

B = condition (such as test is positive)

′ A = not A (such as patient does not have disease)

4

Lab Test Utility Statistics Sensitivity and specificity

Definitions: Sensitivity: Ability of the test to detect disease Specificity: Ability of the test to exclude disease

Depend only on nature of test and the cutoff chosen to discriminate two statessuch as disease versus no disease

Independent of disease prevalence Useful for picking best test or best cutoff

5

Lab Test Utility Statistics

Predictive values (positive and negative)

Definitions: PV+: Likelihood that a patient with a positive test

actually has the disease

PV-: Likelihood that a patient with a negative

test truly does not have the disease

Depend on sensitivity and specificity

Depend on disease prevalence

6


Efficiency (diagnostic accuracy)

Definition: The proportion of properly classified patients

Depends on sensitivity and specificity

Depends on disease prevalence

7


Disease

No disease

Totals

90 10 100

45 855 900

135 865 1000

Positive Negative TotalsTP FN

FP TN

SensitivitySpecificity

PV +PV –

EfficiencyDisease Prevalence

90.0 %95.0 %66.7 % 98.8 % 94.5 %

10 %

Truth Table:

TP: True positives

FP: False positives

FN: False negatives

TN: True negatives

8


Formulas:

Sensitivity: TP/Disease

Specificity: TN/No disease

Efficiency: (TP + TN)/Total

Predictive Value of Positive Test: TP/(TP + FP)

Predictive Value of Negative Test: TN/(TN + FN)

9


More Formulas:

Disease Prevalence: Disease/Total Patients Proportion of patients with disease

Positive Test Odds Ratio: PV+/Prevalence Impact of testing on diagnosing disease. A value

of 1.0 means that test did not help. A high value

means test was very useful.

Negative Test Odds Ratio: PV-/Prevalence Impact of testing on ruling-out disease.

10

An Example: Cancer test “X”

Screen 10,000 patients Prevalence of cancer is 0.5%

50 cancer patients True positives: 49 True negatives: 9,751 False positives: 199 False negatives: 1

11

An Example: Cancer test “X”Positive Test Negative Test Totals

Disease 49 1 50

No Disease 199 9751 9950

Totals 248 9752 10000

Results

Sensitivity 98.0%

Specificity 98.0%

12

An Example: Cancer test “X” Sensitivity and Specificity seem high, but…

19.8%

100.0%

Predictive Value of Positive Posttest odds of Disease if test Positive

Predictive Value of Negative Posttest odds of No Disease if test Negative

39.5

1.005

Positive Test Odds Ratio

Negative Test Odds RatioPosttest changes in the likelihoods of Disease or No Disease

13

An Example: Cancer test “X”

What does it all mean?

98% sensitivity means that 98% of cancer

patients will have abnormal “X” result

98% specificity means that 98% of cancer-

free patients will have normal “X” result

98% efficiency (proportion of correct

diagnoses) is great for a screening test

14

An Example: Cancer test “X” What does it all mean?

However, positive predictive value is <20% <1 in 5 patients with abnormal “X” have cancer Additional tests are needed to confirm cancer 80% of patients with abnormal “X” will worry

needlessly until other tests rule out cancer

Positive odds post-test are very high: A positive result increases the odds of having

cancer by a factor of 40. A negative test adds almost no new information

15

How do we get the numbers?

Test patient population of interest

Apply a “gold standard” for diagnosis

Retrospectively after surgery or autopsy

Create from other diagnostic procedures (lab tests,

imaging studies, biopsies, etc.)

Classify patients as “Disease” and “No disease”

based on gold standard

Prepare truth table from data

Use my Excel template

Can work in reverse

16

Reverse Stats Example

Test NamePopulation

Disease

0.05%100,000

98.5%99.0%

50

989514.7%

100.0%

99.0%

5049

10000.0490

Prevalence of DiseasePopulation Size

Sensitivity of TestSpecificity of Test

True NegativesPositive Predictive Value

Negative Predictive Value

False PositivesTrue Pos: False Pos Ratio

Efficiency of Test

Patients with Disease

Predictive Value and Efficiency of a Diagnostic Test

True Positives

HIV ELISA ScreenMarriage license applicantsHIV infection

Persons with Disease

17

New Example: Serum Hic Test

Hic test designed to detect propensity

towards prolonged hiccups

Developer believes it will be sensitive and

specific

“Gold standard” is self-reporting of hiccup

propensity

My Excel template used for this example

18

Test Name

Population

Positive Test Negative Test Totals

Disease 9 3 12

No Disease 6 182 188

Totals 15 185 200

Results

Sensitivity 75.0% 50.5 - 99.5

Specificity 96.8% 94.3 - 99.3

60.0% 35.2 - 84.8

98.4% 96.6 - 100

95.5% 92.6 - 98.4

6.00%

94.0%

10.0 5.9 - 14.1

1.05 1.03 - 1.06

Efficiency of Test

Adults

Predictive Value of Negative Posttest odds of No Disease if test Negative

95% Confidence Limits

Bayesian Analysis of a Diagnostic Test

Predictive Value of Positive Posttest odds of Disease if test Positive

Serum Hic

Posttest changes in the likelihoods of Disease or No Disease

Pretest odds of No DiseaseNo Disease Prevalence

Percentage of correct test results

Disease Prevalence

Negative Test Odds Ratio

Positive Test Odds Ratio

Pretest odds of Disease

19

Truth Tables for Test Method Comparisons

Compare qualitative tests with Truth Tables

Both methods run on patient samples N may be high if disease prevalence is low

Old method is considered correct

Calculate sensitivity and specificity of

new method

Accept if test statistics are good enough

20

Qualitative Method Comparison Clostridium difficile toxin detection tests

Positive or negative are only results

Bayesian Analysis of a Diagnostic Test

Test NamePopulation

Positive Test Negative Test TotalsDisease 17 5 22

No Disease 1 70 71Totals 18 75 93

Results

Sensitivity 77.3%Specificity 98.6%

Samples tested with C. diff. A kitClostridium difficile Toxin A & B kit

21

Confidence Intervals for Test Stats

Confidence intervals easily found for:

Sensitivity

Specificity

Predictive values

Efficiency

Based on binomial probabilities and

estimation of standard deviation

22

Bayesian Analysis Template

Test NamePopulation

Disease

0.02%100,000

99.0%98.5%

20

984801.3%

100.0%

98.5%

2020

15000.0133






Efficiency of Test



True Positives

HIV ELISA ScreenMarriage license applicantsHIV infection


23

Bayesian Analysis Template

Test NamePopulation

Disease

1.32%1,520

99.0%99.2%

20

1,48862.5%

100.0%

99.2%

202012

1.67






Efficiency of Test



True Positives

Western BlotPositive HIV ELISA ScreenHIV/AIDS


24

Receiver-Operator Characteristic Curves Developed in World War II to determine

optimal gain settings for radar

In laboratory world, ROC curves plot

information derived from Bayes’ theorem

truth tables

Can determine best cutoff for a test

Can determine which test performs best

25

Receiver-Operator Characteristic Curves Commonest ROC curve in medicine: plot of

False Positive fraction versus True Positive

fraction

Can also plot disease probability versus

Predictive Value of Positive or Negative test

Shows how test performance changes as

disease prevalence changes

26

Lab Test ROC

0.00

0.20

0.40

0.60

0.80

1.00

0.00 0.20 0.40 0.60 0.80 1.00

False Positive Fraction

True Positive Fraction

Cutoff False Pos. True Pos.129 0.00 0.14124 0.03 0.21119 0.03 0.35114 0.03 0.35109 0.08 0.50104 0.13 0.6699 0.20 0.7294 0.23 0.8389 0.30 0.9084 0.43 0.9379 0.55 1.00

27

ROC Analysis for 3 Tests

Sensitivity

1 - Specificity0 0.2

0.4

0.6

0.8

1.0

0

0.2

0.4 0.6 0.8 1.0

A

B C

False Positive Fraction

TruePositiveFraction

28

What does ROC curve tell us?

Good test has high true positive fraction and

low false positive fraction across range of

cutoff values

Screening test: emphasize sensitivity

Definitive test: emphasize specificity

Poor test not much better than chance

Use area under curve to quantify test utility

29

Conclusions

Bayes’ Theorem for conditional probabilities

foundation for Truth Tables and calculations

of diagnostic sensitivity, specificity,

predictive values, and efficiency.

Information value from a test relates to pre-

and post-test probabilities

ROC curves help determine best tests and

best cutoff values

30

Resources on CD-ROM

This PowerPoint presentation

“Evaluating Articles about Diagnostic Tests”

PowerPoint presentation

Bayes’ Theorem Excel template

Other Excel statistics templates

Materials from entire lab statistics course

2 Test Utility Statistics

Documents

Transcript of 2 Test Utility Statistics