Accuracy lecture 131024

ACCURACY (AND OTHER VALIDATION MEASURES)

Adina L. Feldman, M.Sc.

Karolinska Institutet Department of Medical Epidemiology and Biostatistics

e-mail: [email protected] tel. 08 5248 2313

24 October 2013 Adina L. Feldman 1


SYSTEMATIC ERROR

RA

ND

OM

ER

RO

R

Low High

Low

H

igh

Validity

Accuracy is a type of systematic error (potential bias)

(Random error/precision is related to power, e.g. size of study sample)

Validity is what we call the certainty (accuracy) of a proxy measure/test

Why is knowing the validity of a measure important?

Consider these examples: What is the validity of breast cancer screening (mammography)?

What is the validity of home pregnancy tests?

What is the validity of self-reported height? …weight?

What is the validity of register-based Parkinson’s disease diagnoses?


Gold Standard

= The best possible available measure agianst which the measure under study is validated

Discuss: What gold standard was used in these validations? Breast cancer screening (mammography)?

Home pregnancy tests?

Self-reported height? …weight?

Register-based Parkinson’s disease diagnoses?



Gold Standard

Binary Continuous

Test

mea

sure

Bin

ary

Con

tiuou

s

Discuss: Where do these validations fit in?

Breast cancer screening (mammography)?





Gold Standard

Binary Continuous

Test

mea

sure

Bin

ary

Reg PDx X

Con

tiuou

s

Preg test BC screening

Height Weight

Discuss: Where do these validations fit in?

Breast cancer screening (mammography)?





Gold Standard

Binary Continuous

Test

mea

sure

Bin

ary Sensitivity,

Specificity, etc.

X

Con

tiuou

s

ROC-curves Correlations,

Bland-Altman plots

Different validation methods are used for different types of validation studies!

These are covered (or at least mentioned) today


Gold Standard Validity measures for binary outcomes

(Print and pin to your office wall!)

Out

com

e m

easu

re

Positive +

Negative -

Positive +

True Positive (TP)

False Positive (FP)

Positive Predictive Value

(PPV)

=TP/ (TP+FP)

Negative -

False Negative (FN)

True Negative (TN)

Negative Predictive Value

(NPV)

=TN/ (TN+FN)

Sensitivity Specificity

=TP/ (TP+FN)

=TN/ (TN+FP)


Gold Standard

O

utco

me

mea

sure

Positive +

Negative -

Positive +

True Positive (TP)

False Positive (FP)

Positive Predictive Value

(PPV)

=TP/ (TP+FP)

Negative -

False Negative (FN)

True Negative (TN)

Negative Predictive Value

(NPV)

=TN/ (TN+FN)

Sensitivity Specificity

=TP/ (TP+FN)

=TN/ (TN+FP)


Gold Standard These are less commonly used measures, but still good to know

Out

com

e m

easu

re

Positive +

Negative -

Positive +

True Positive (TP)

False Positive (FP)

False Positive Rate (FPR),

cPPV (=1-PPV)

=FP/ (TP+FP)

Negative -

False Negative (FN)

True Negative (TN)

False Negative Rate (FNR),

cNPV (=1-NPV)

=FN/ (TN+FN)

True Positive Rate

FPR (OBS!!) (=1-Spec.) Accuracy

=Sens. =FP/ (FP+TN)

=TP+TN/ (TP+TN+FP+FN)

Misclassification

FN and FP are misclassifications

Consider cause of misclassification FN: Why are some cases not detected?

FP: Why are some noncases given erroneous diagnoses?

Differential misclassification: Non-random distribution of TP and FN (with regards to the exposure)


Misclassification

Discuss: What could be the cause of FP and FN in these validations? What could be the consequences of misclassification here? Breast cancer screening (mammography)?





Fictional Example 1 Cohort study of 10,000 participants (random population-based sample)

Binary proxy measure e.g. self-reported myocardial infarction (”heart attack”) ever/never

Binary Gold Standard e.g. myocardial infarction confirmed according to best clinical practice



Gold Standard Example 1

Out

com

e m

easu

re

Positive +

Negative -

Positive +

90 5 PPV?

Negative -

10 9,895 NPV?

Sens.? Spec.? GS prev.?

OM prev.?


Gold Standard Example 1 ↓prevalence

↑PPV ↑Sens.

Out

com

e m

easu

re

Positive +

Negative -

Positive +

90 5 PPV 94.7%

Negative -

10 9,895 NPV 100%

(99.899%)

Sens. Spec. GS prev. 1.0%

90.0% 100% (99.95%)

OM prev. 0.95%

Fictional Example 2 Cohort study of 10,000 participants (random population-based sample)

Binary proxy measure e.g. self-reported influenza during one winter season yes/no

Binary Gold Standard e.g. laboratory-confirmed infection with influenza virus



Gold Standard Example 2

Out

com

e m

easu

re

Positive +

Negative -

Positive +

1950 1400 PPV?

Negative -

50 6600 NPV?

Sens.? Spec.? GS prev.?

OM prev.?


Gold Standard Example 2 ↑prevalence

↓PPV ↑Sens.

Out

com

e m

easu

re

Positive +

Negative -

Positive +

1950 1400 PPV 58.2%

Negative -

50 6600 NPV 99.2%

Sens. Spec. GS prev. 20.0%

97.5% 82.5% OM prev. 33.5%

Discussion points

Many validation study have only available either: Only Gold Standard positive cases

Only proxy outcome positive cases

What validity measures can be calculated in each instance?

Two-phase screening is a very common approach to diagnosing disease, e.g. Breast cancer (mammography followed by ultrasound, cytology) What type of validity is most important in each phase?



Gold Standard

Binary Continuous

Test

mea

sure

Bin

ary Sensitivity,

Specificity, etc.

X

Con

tiuou

s


Bland-Altman plots



Freq

uenc

y of

cas

es

Measures with discrimination threshold for binary outcomes


E.g. biomarker concentration in blood

GS-

GS+

Freq

uenc

y of

cas

es

Measures with discrimination threshold for binary outcomes


E.g. biomarker concentration in blood

GS-

GS+ TN

FN FP

TP


Gold Standard: Reduced insulin sensitivity based on established clinical index cutoff

Proxy test: Appendicular lean body mass (LBM) index (kg/m2)

The threshold for LBM is varied and for each step the sensitivity and 1-specificity for the GS are calculated and plotted

The goal is to determine the optimal threshold for LBM in predicting reduced insulin sensitivity

AUC = Area Under the Curve (%) (Bigger = Better)


Gold Standard

Binary Continuous

Test

mea

sure

Bin

ary Sensitivity,

Specificity, etc.

X

Con

tiuou

s


Bland-Altman plots




?


Pearson correlation coefficient overall = 0.61

Afternoon group excercise: Ad hoc study of the validity of self-reported height Define Gold Standard

Method of ascertainment of self-reported height

Collect data Proxy

Gold Standard

Using Excel Plot correlation (scatter plot)

Brand-Altman plot

Draw conclusion



Welcome to my PhD dissertation defence 10 Januari 2014, at 9 am in Andreas Vesalius,

Karolinska Institutet Campus Solna

Dissertation title: ”If I Only Had a Brain

– Epidemiological Studies of Parkinson’s Disease”

Thank You! (See you this afternoon)

Accuracy lecture 131024

Health & Medicine

Transcript of Accuracy lecture 131024