Diagnostic tests evaluation & uses

Charles Charles CaraguelCaraguelCentre for Veterinary Epidemiological Research

Centre for Aquatic Health SciencesCentre for Aquatic Health SciencesAtlantic Veterinary CollegeAtlantic Veterinary College

University of Prince Edward IslandUniversity of Prince Edward Island

Diagnostic testsDiagnostic tests

evaluation & usesevaluation & uses

Why is it so exciting?“Test performance is

critical to start a new life.” Me (2008)

1

Basics (1): Variation & errors in estimation

SourcesSources• Sampling error (selection bias)

• Biological variation (time & space)

• Measurement error (diagnostics)

SemanticsSemantics

Health status (Binary outcome)

Test result (Binary outcome)

POSITIVE (T+)

NEGATIVE (T-)

DISEASED (D+)

NON-DISEASED (D-)

2

Phase 1 - ”Laboratory“ validation

• Optimisation & standardisation of reagents

• Analytical sensitivity (ASe) & specificity (ASp)

Validation ObjectivesValidation Objectives““evaluation of a process to determine its evaluation of a process to determine its fitness for a particular usefitness for a particular use””

Dilution

3

Phase2 - “Field” validation

• “Precision” (agreement sensus stricto)

Repeatability (overall intra-lab)

Reproducibility (overall inter-lab)

• “Accuracy” (performances)

Diagnostic Sensitivity (DSe)

Diagnostic Specificity (DSp)

Degree of mistakes made by the test

Phase 1 - ”Laboratory“ validation

• Optimisation & standardisation of reagents

• Analytical sensitivity (ASe) & specificity (ASp)

Validation ObjectivesValidation Objectives““evaluation of a process to determine its evaluation of a process to determine its fitness for a particular usefitness for a particular use””

4

Basics (2): Basics (2):

Conditional ProbabilityConditional ProbabilityPr(Pr(AA//BB) = given ) = given BB, the probability of , the probability of AA

i.i. First set the primary information or condition (First set the primary information or condition (BB))

Health status (Health status (Se&SpSe&Sp); Test result (Predictive values)); Test result (Predictive values)

ii.ii. Then defined the component of interest (Then defined the component of interest (AA))

Health status (Predictive values); Test result (Health status (Predictive values); Test result (Se&SpSe&Sp))

5

Tested positiveTested positive

Tested negativeTested negative

““ Non diseasedNon diseased ”” ““ DiseasedDiseased ””

Sp = Sp = Negative & Non dis. Negative & Non dis.

Non diseasedNon diseased

Se = Se = Positive & Diseased Positive & Diseased

DiseasedDiseased

““ True True PositivePositive ””

““ False False NegativeNegative ””

““ True True NegativeNegative ””

““ FalseFalsePositivePositive ””

6

Accuracy DefinitionAccuracy DefinitionSe & SpSe & Sp

Diagnostic Sensitivity Diagnostic Sensitivity (true positive)(true positive)

Pr(Pr(TT++//DD++) =) =

Diagnostic SpecificityDiagnostic Specificity (true negative)(true negative)

Pr(Pr(TT--//DD--) =) =

the probability of the probability of TT++given given DD++, ,

the probability of the probability of TT--given given DD--, ,

7

Legend of theLegend of the

2X2 TABLE2X2 TABLE

Sensitivity Sensitivity (true positive fraction)(true positive fraction)

Pr(Pr(TT++//DD++) = ) = a / (a + b)a / (a + b)

Specificity Specificity (true negative fraction)(true negative fraction)

Pr(Pr(TT-- //DD--) =) = d / (c + d)d / (c + d)

HealthStatus

Test Results

DD++

DD--

TT++ TT--

aa bb

cc dd

a + ca + c b + db + d

a + ba + b

c + dc + d

n = n = a+b+c+da+b+c+d

TP

FP

FN

TN

8

Real applications in the Real applications in the ““fieldfield””

1. 1. Prevalence estimatePrevalence estimate

““farmers & veterinariansfarmers & veterinarians””

““ApparentApparent””

prevalence (prevalence (APrAPr))““TrueTrue””

prevalence (prevalence (TPrTPr))

Se / SpSe / Sp

Prevalence definitionPrevalence definition

APrAPr == TPrTPr = =

True prevalence formulaeTrue prevalence formulae

TPrTPr ==APrAPr + Sp + Sp -- 11

Se + Sp Se + Sp -- 11 True positive among D+

False positive among D-

Apparent prevalence formulaeApparent prevalence formulae

APrAPr = Pr Se + (1 = Pr Se + (1 -- TPrTPr) (1 ) (1 –– Sp)Sp)

# Positive# Positive

# Total (n)# Total (n)# Diseased# Diseased

# Total (n)# Total (n)

9

Exercise (1):True prevalence estimation

During a surveillance screening, 18 cows tested positive

for Staphylococcus aureus out of 100 tested

(Test performances: Se = 72 % & Sp = 98 %)

Compute the following value:

Apparent Prevalence =

True Prevalence =

10


1. 1. Prevalence estimatePrevalence estimate

2. 2. Decision makingDecision making



prevalence (prevalence (APrAPr))

““TrueTrue””


Se / SpSe / Sp

Positive Predictive Value (PPV)Positive Predictive Value (PPV)

Given test is positive, what probability to be diseased?Given test is positive, what probability to be diseased?

Negative Predictive Value (NPV)Negative Predictive Value (NPV)

Given test is negative, what probability to be nonGiven test is negative, what probability to be non--diseased?diseased?

Primary Information: TEST RESULTS!Primary Information: TEST RESULTS!

(condition)(condition)

11

NegNeg NegNeg NegNeg NegNeg NegNeg





PosPos PosPos PosPos PosPos PosPos





DiseasedDiseased

NonNon--diseaseddiseased

““ True True PositivePositive ””

““ False False PositivePositive ””

““ True True NegativeNegative ””

““ FalseFalseNegativeNegative ””

PPV = PPV = Positive & Diseased Positive & Diseased

PositivePositive

NPV = NPV = Negative & Non dis. Negative & Non dis.

NegativeNegative

NegativeNegative PositivePositive

12

Positive Predictive value Positive Predictive value (PPV)(PPV) Strongly influenced by SpStrongly influenced by Sp

Pr(Pr(DD++//TT++) =) =

Negative Predictive value Negative Predictive value (NPV)(NPV) Strongly influenced by SeStrongly influenced by Se

Pr(Pr(DD--//TT--) =) =

the probability of the probability of DD++Given Given TT++, ,

the probability of the probability of DD--Given Given TT--, ,

Predictive Values DefinitionsPredictive Values Definitions

13

Predictive Values FormulaePredictive Values Formulae

Positive Predictive ValuePositive Predictive ValuePr(Pr(DD++//TT++) = ) = a / (a + c)a / (a + c)

Negative Predictive ValueNegative Predictive ValuePr(Pr(DD-- //TT--) =) = d / (b + d)d / (b + d)

Test Results

DD++

DD--

TT++ TT--

aa bb

cc dd

a + ca + c b + db + d

a + ba + b

c + dc + d

n = n = a+b+c+da+b+c+d

TP

FP

FN

TN

14

You test 100 cows for Staph. aureus infection. Sixty-five cows tested

negative by PCR and 35 tested positive. However, using culture as

a gold standard, 5/35 of the PCR-positives were disease-free and

14/97 of the seronegatives were diseased.

Fill the 2X2 table and compute AP, TP, Se, Sp, PPV, NPV:

Se =

Sp =

PPV =

NPV =

Exercise (2):2X2 table and more

TT++ TT--

DD++

DD--

15

n*n*PrPr*Se*Sen*n*Pr(Pr(DD++&&TT++) )

n*n*Pr(Pr(DD--&&TT++) ) n*n*Pr(Pr(DD--&&TT++) )

n*n*Pr(Pr(DD++&&TT--) ) aa bb

cc dd

a + ca + c b + db + d n = n = a+b+c+da+b+c+d

Positive Predictive ValuePositive Predictive Value

Pr(Pr(DD++//TT++) =) =

Negative Predictive ValueNegative Predictive Value

Pr(Pr(DD-- //TT--) =) =

DD++

DD--

TT++ TT--

PrPr*(1*(1--Se)Se)

(1(1--Pr)Pr)*(1*(1--Sp)Sp) (1(1--Pr)Pr)*Sp*Sp

Pr(Pr(DD++) = ) = Prevalence(Prevalence(PrPr))

Pr(Pr(DD--) = ) = 1 1 -- PrPr

Pr(Pr(TT++//DD++) = Se) = SePr(Pr(TT-- //DD--) =) = SpSp

Bayesian Theorem: Bayesian Theorem: Pr(Pr(AA//BB) = ) = Pr(Pr(AA&&BB) / ) / Pr(Pr(BB)) Pr(Pr(AA&&BB)) = = Pr(Pr(BB) *) *Pr(Pr(AA//BB))

n take any value even 1n take any value even 1

PrPr*Se*Se

11

PrPr*Se*Se

PrPr*Se + *Se + (1(1--Pr)Pr)*(1*(1--Sp)Sp)

(1(1--Pr)Pr)*Sp*Sp

(1(1--Pr)Pr)*Sp*Sp + + PrPr*(1*(1--Se)Se)

When no a, b, c or d?When no a, b, c or d?but Se, Sp and Prevalencebut Se, Sp and Prevalence

16


1. 1. Prevalence estimate (risk analysis) Prevalence estimate (risk analysis)







Se / SpSe / Sp

DSeDSe

DSpDSp

PrevalencePrevalence





17

in determining if she is infected with Staph. aureus. The PCR test

used assures 99 % accuracy (assumed 99% of DSe and 99% DSp).

The prevalence of cows infected with Staph. aureus and

presenting with clinical signs of mastitis of is 25%.

Define and compute the following values:

PPV =

NPV =

Exercise (3):Predictive value

Clarice is a 3 year-old cow showing

clinical signs of mastitis. The farmer

and the vet are interested

18


1. 1. Prevalence estimate (risk analysis) Prevalence estimate (risk analysis)


3. 3. Surveillance ProgramSurveillance Program






Se / SpSe / Sp

Computation of sample size to declare a population Computation of sample size to declare a population ‘‘free of diseasefree of disease’’

DSeDSe

DSpDSp

PrevalencePrevalence

n / Sen / Se





19

QUESTIONS ?QUESTIONS ?

20

Diagnostic tests evaluation & uses

Documents

Transcript of Diagnostic tests evaluation & uses