Critical Appraisal of Articles About Diagnostic Tests

7/30/2019 Critical Appraisal of Articles About Diagnostic Tests

1/43

Critical Appraisal of

Articles About

Diagnostic Tests

Rani Gereige, M.D., MPHAssociate Professor

University of South Florida


2/43

Learning Objectives Test validity What is it? and how to ensure it?

Putting diagnostic tests to the test: Critical appraisal ofstudies about a diagnostic test

Be familiar with the concepts of

Treatment and Test thresholds Pre-test and post-test probabilities and odds

Likelihood Ratios (LRs)

Evaluate a VALID evidence using The 2x2 table

Calculations and interpretation of Sensitivity, Specificity, PPV,NPV, pre and post test probabilities and odds, LR(+) and LR(-)

Apply the evidence


3/43

QuestionsA patient presents to us with a chief

complaint

Why do we order tests?

What tests to order? Based on what?What do we hope to achieve as we get the

result of the test?

What if there are multiple tests that arerelated to this complaint?

What if we are considering 6 or 7 possiblediagnoses that might explain this chiefcomplaint?


4/43

Hurricane Fever

What ifthere is a new test that claims to

accurately pinpoint hurricanes location of the

attack??What would you want to know about this test?

Anything you want to know about the hurricane itself?

What about if the hurricane is right at your doorstep?

What if this new test can only be done at anelevation of 15000 ft above sea level and you

live in Florida?


5/43

Diagnostic Test

Validity Definition

The validity of a diagnostic test is the

extent to which the resultsrepresent an unbiased estimate of

the underlying truth

(it is testing what it supposed to test)


6/43

Validity versus Relevance

Validity Is it true?

Was the study appropriatelyrandomized, blinded?

Did they use appropriate goldstandard?

Was the population similarenough to your own?(generalizable)

Was follow-up adequate?

Recommended first by thetext

Relevance Does it matter to my

patients/population?

Is the test feasible for my patients?

Will it affect something thatMATTERS to patients? (Morbidity,mortality, quality of life)

Will it change my practice?

Most clinicians look atrelevance first

Easy and quick to find(abstract)

If does not matter do not botherwith validity testing


7/43

How to Use an Article about a

Diagnostic Test??Three main questions:

Is this evidence about the accuracy of a

diagnostic test valid? Does this (valid) evidence demonstrate an

important ability of this test to accuratelydistinguish patients who do or do not have a

specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test

to a specific patient?

If answer isNOStop


8/43

I- Is this evidence about the accuracy

of a diagnostic test VALID?

Measurement

Representative

Ascertainment

Reproducibility


9/43


of a diagnostic test VALID? Measurement: Was there an independent, blind

comparison with a reference Gold Standard? Patients should have undergone both the test and the

reference (gold standard) Confirmation by surgicalfindings, biopsy, autopsy, long term follow-up

Investigator should not know the result of goldstandard

Limitations:

Gold standard is not always possible (psych disorders) Bias cannot always be avoided

Some gold standards are for research only


10/43


of a diagnostic test VALID? Representative: Was the diagnostic test

evaluated in an appropriate spectrum of patients(like those in who we would use it in practice)?Was the spectrum of disease appropriate?

Florid cases vs asymptomatic volunteers

Disease severity: Mild, severe, early or late cases, bothtreated and untreated

Apply the test to patients with different disorder commonlyconfused with the target disorder

Was the spectrum of patients appropriate? Similar ages, sexes, co-morbidities

Enough patient numbers?


11/43


of a diagnostic test VALID?Ascertainment: Was the reference

standard ascertained regardless of the

diagnostic test result? Important to confirm or refute the test on ALL

patients, not just those with abnormal test

May use different method of gold standard toavoid harming patients (Surgery vs close

follow-up)


12/43


of a diagnostic test VALID? Reproducibility: Was the test (or cluster of

tests) validated in a second, independent

group of patients?Diagnostic tests are predictors, not explainers

of the diagnosis

Same results of the test in a separateindependent group will reassure us on the

accuracy

Not always done initially


13/43

Eligible Patients

n =

Excluded Patients

Reasons n =

Index test

n =

Positive

n =

Negative

n =

Inconclusive

Missing

n =

Missing

n =

Missing

n =

Reference Standard Reference StandardReference Standard


14/43



Is this evidence about the accuracy of a

diagnostic test valid? Does this (valid) evidence demonstrate an

important ability of this test to accuratelydistinguish patients who do or do not have a

specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test

to a specific patient?

If answer isNOStop


15/43

Probability versus Odds

Odds of disease does not equal

probability of disease

Example: 10 patients 3 have anemia and7 do not

Probability of having anemia is 3/10 = 30% =

Prevalence = Pretest probabilityOdds of having anemia are 3:7 or 0.43


16/43

Probabilities and Odds

Can be calculated back and forth Odds of a:b will give probability of a/(a+b)

Probability of X% will give Odds of X/(100-X)

You start withPre-test probability (Prevalence)

Pre-test Odds

Post-test Odds

Post-test probability


17/43

Pre-test Probability and Odds

Pre-test probability = Prevalence = The

proportion of people with the target

disorder in the population at risk at aspecific time

Pre-test Odds = The odds that the patient

has the target disorder before the test iscarried = Pre-test Probability/[1- Pre-test

Probability]


18/43

Post-test Probability and

Odds Post-test probability = The proportion of

patients with that particular test result who

have the target disorder = Post-test odds/[1+ Post-test Odds]

Post-test Odds = The odds that the patient

has the target disorder after the test iscarried out


19/43

Sources of Pre-test Probability Clinical experience

Biased by frequency of diagnosis, last patient, fear ofmissing a diagnosis

Easily and quickly accessed

Use with caution

Not for young careers Regional/ national prevalence statistics on

disorder frequency (as good as accuracy ofdiagnosis)

Local/ regional/ national practice databases

Calculated from the study being appraised

Track research report of a study devoted to

pretest probability documentation (Best - if welldone)


20/43

Inputs Into Pre-Test Probability

The frequency of the disease in

populations or groups of individuals similar

to a particular patient Risk factor exposure of the individual

Pattern of symptoms presented by the

patient

Use of a prior screening test


21/43

Test and Treatment Thresholds

Test Threshold probabilities below which a

clinician would dismiss a diagnosis and order

no further tests

Treatment Threshold probabilities above

which a clinician would consider the diagnosis

confirmed, and would stop testing


22/43

Test accuracy

Ability of a test to change our minds from

what we thought before the test (pre-test

probability) to what we think afterwards

(post-test probability)

Diagnostic tests that produce big changes

from pre-test to post-test probability, are

important and likely to useful in practice


23/43

Clinical Implications

When the probability of the target disorder

falls between the test and treatment

thresholds, further testing is mandated

Once test and treatment thresholds are

determined, the post-test probabilities

have direct treatment implications


24/43

Evaluating the Evidence Diagnostic Test

Construct the 2x2 table

Sensitivity = a /a+c = P(+/D)= P(TP amongdiseased)=PID

Specificity = d /b+d = P(-/D)= P(TN among non diseased)

PPV = a /a+b = P(D/+)= P(TP among all Positives)

NPV= d /c+d = P(D/-)= P(TN among all Negatives)

SnNout = Result of a test with high Sensitivity, whenNegative, rules out the diagnosis

SpPin = Result of a test with high Specificity, when

Positive, rules in the diagnosis


25/43

More Definitions

Sensitivity = PID = Positive in Disease

Specificity = NIH = Negative in Health

Discriminant ability of a test =

(sensitivity+specificity)/2


26/43


27/43

Likelihood Ratios

Indicate by how much a given diagnostictest result will raise or lower the pretest

probability of the target disorderLR = 1 means the post-test probability is

exactly the same as the pre-test probability

LR > 1 increases the probability that the target

disorder is presentLR < 1 decreases the probability that the

target disorder is present


28/43

Guide to the Significance of LRs

LR > 10 or < 0.1 generate large and often conclusive changes frompre-test to post-test probability

LR = 5 - 10 or 0.1 - 0.2 generate moderate shifts pre-test to post-test

LR = 2 5 or 0.5 0.2 generate small, but sometimes importantchanges in probability

LR = 1 2 or 0.5 1 are rarely important shifts


29/43

How to Use LRs?? Bayes

Theorem Used to calculate Post-test probability

Nomogram

(Textbook card 3A) Table

LR calculator

PDAMed Calc

LR calculator: http://araw.mede.uic.edu/cgi-alansz/testcalc.pl

Pre-test probability (Prevalence) - X

Pre-test Odds: Y = X/(100-X)

Post-test Odds: Z = Y x LR

Post-test probability = Z/(Z+1)

LR
http://araw.mede.uic.edu/cgi-alansz/testcalc.plhttp://araw.mede.uic.edu/cgi-alansz/testcalc.plhttp://araw.mede.uic.edu/cgi-alansz/testcalc.plhttp://araw.mede.uic.edu/cgi-alansz/testcalc.pl


30/43


31/43

LR Nomogram & Table


32/43


33/43


34/43

The Ferritin Systematic

Review DiseasePresent

Disease

Absent

Totals

Ferri t in

65 mmo l/L

78

(c)

1500

(d)

1578

(c+d)

Totals 809

(a+c)

1770

(b+d)

2579

(a+b+c+d)

Sensitivity = 731/809 = 90%

Specificity = 1500/1770 = 85%

PPV = 731/1001 = 73%

NPV = 1500/1578 = 95%

LR(+) = Sens/(1-Spec) = 90%/15% = 6

LR(-) = (1-Sens)/Spec = 10%/85% =

0.12

Pre-test Probability = Prevalence =

(a+c)/(a+b+c+d) = 809/2579 = 31%

What do you think of these LRs??


35/43

Calculating Post-Test

Probability Pre-test Odds =Prevalence/(1-Prevalence) =

31%/69% = 0.45

Post-test odds=pre-test oddsx LR

For (+) = 0.45x6=2.7

For (-) = 0.45x0.12=0.054

Post-test probability= Post-

test odds/(post-test odds+1)

For (+) = 2.7/3.7=72.99%

For (-) = 0.054/1.054=5.12%

(+)

(-)


36/43

What Conclusions Can You Draw

About Ferritin? High sensitivity and specificity

LR(+) of 6 means that using Ferritin level allowsto increase the probability of iron deficiency

anemia from 31% (or 50%) to 73% (or 86%) forlevels < 65 mmol/L

LR(-) of 0.12 means that using Ferritin levelallows to decrease the probability of iron

deficiency anemia from 31% (or 50%) to 5% (or11%) for levels > 65 mmol/L

This is known as sensitivity analysis


37/43

Multilevel Likelihood Ratios

Provide superior information to sensitivity and Specificity

Allow detailed and specific clinical conclusions

809 1770


38/43


39/43



Is this evidence about the accuracy of adiagnostic test valid?

Does this (valid) evidence demonstrate animportant ability of this test to accuratelydistinguish patients who do or do not have aspecific disorder? i.e. Magnitude

Can I apply this valid, important diagnostic testto a specific patient?

If answer isNOStop


40/43

Applying the Valid Evidence to

YOUR PatientThree Questions Q1. Is the diagnostic test available,

affordable, accurate and precise in

our setting?Requires knowledge of your clinical

capacity

Do you have the test? Personnel?

Cost? Insurance coverage?

What is your patient population? And status

of disease presentation?


41/43

Q2. Can we generate a clinically sensibleestimate of our patients pre-test probability? From personal experience (H&P, memory),

prevalence statistics (texts, government), practicedatabases (Hospitals, med records), primarystudies (validity testing)

Are the study patients similar to our own?

Is it unlikely that the disease possibilities or

probabilities have changed since this evidence wasgathered? (e.g. study from 1996 on AIDSdiagnosis)

Applying the Valid Evidence to YOUR

PatientThree Questions


42/43

Q3. Will the resulting post-test probabilitiesaffect our management and help our patient? Could it move us across a test-treatment threshold?

Depends on the disease and the relative risks of treatment vsno treatment

May move us from one test to another

Post-test odds after one test become the pre-test odds for thenext

Would our patient be a willing partner in carrying itout? (This is critical)

Would the consequences of the test help our patientreach his or her goals in all this?

Applying the Valid Evidence to YOUR

PatientThree Questions


43/43

Look at things CRITICALLY

Critical Appraisal of Articles About Diagnostic Tests

Documents

Transcript of Critical Appraisal of Articles About Diagnostic Tests