Diagnostic accuracy – part 1STUDY DESIGN

DEPARTMENT OF MEDICAL LABORATORY DIAGNOSTIC SUNIVERSITY HOSPITAL SVETI DUHZAGREB, CROATIA

Recommended literatureo Bossuyt PM et al. The STARD Statement for Reporting Studies of Diagnostic

Accuracy: Explanation and Elaboration. Clinical Chemistry 2015;61:1446–1452

o Simundic AM. Diagnostic Accuracy - Part 1: Basic Concepts: Sensitivity and Specificity, ROC Analysis, STARD Statement. Point of Care 2012;11:6-8

o Simundic AM. Diagnostic Accuracy - Part 2: Predictive Value and Likelihood Ratio. Point of Care 2012;11:9-11

o Raslich MA, Markert RJ, Stutes SA. Odabir i tumačenje dijagnostičkih pretraga. / Selecting and interpreting diagnostic tests. Biochemia Medica 2007;17(2):151-161.

o Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut-off value: the case of tests with continuous results. Biochemia Medica2016;26(3):297-307.

basic

appliedunderstanding relationships

application in clinical practice

Study types (aim)

observational

experimental

Study types (role)

• no intervention

• low strenght of evidence

• no causal relationship • intervention

• high strenght of evidence

• causal relationship

Study types (time)

time

• study design• data collection• data analysis

prospectiveretrospective

cross-sectional

• study design• data analysis

• data collection (from the past)

• study design

• data collection (in the future)• data analysis

Diagnostic accuracy study

How accurately a diagnostic test/procedure can discriminate….

• health and disease• two conditions (viral vs. bacterial)• disease stage (acute vs. chronic)

Diagnostic processPatient historyClinical exam

Pre-test probability

Diagnostic test

Post-test probability

Does this patient have AMI?

cTn-I

AMI + AMI -

study participants

index test

reference test

disease positive

symptomatic patients

inclusion + exclusion criteria☺

indeterminate

not included

Study prototype

• cross-sectional study• consecutive patients

gate

disease negative

BiasAny trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically incorrect.McGraw-Hill Concise Dictionary ofModern Medicine. © 2002 by TheMcGraw-Hill Companies, Inc.

bias = random error

fx

X

BiasAny trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically incorrect.McGraw-Hill Concise Dictionary ofModern Medicine. © 2002 by TheMcGraw-Hill Companies, Inc.

bias = random error

fx

BMI (kg/m2)25 35

bias=10 kg/m2

Quality of studyWhy is it important?

Bad results

overestimated

biased

subjective

Clinical practice

medical errors

unnecessary costs

patient harm

Lijmer et al. JAMA, 1999.The effect of bias can be quantified

JG Lijmer, et al. Empirical Evidence of Design-Related Bias in Studies of Diagnostic Tests. JAMA. 1999;282(11):1061-66.

Empirical evidence of design-related bias in studies of diagnostic tests

Those who recruit patients for clinical trials should not be aware of treatment allocation.

blinded, N=57

unblinded, N=45

non-random, N=43

14%

>1 variable not distributed equally

among groups

26%

58%

Difference in case fatality rate between

treatment and control group

5 (8,8%)

11 (24,4%)

25 (58%)

Therapeutic RCT – acute myocardial infarction (N=145)

Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983;309(22):1358-61.

Kohn MA, et al. Understanding the direction of bias in studies of diagnostic test accuracy. Acad EmergMed. 2013;20(11):1194-206.

Possible sources of bias in diagnostic accuracy studies

Possible sources of bias in diagnostic accuracy studies

1/2

study participants

index test

reference test 1

disease positive

symptomatic patients

inclusion + exclusion criteria

Differential verification bias

gate

disease negative

reference test 2

study participants

index test

reference test

disease positive

symptomatic patients

inclusion + exclusion criteria

Partial verification bias

gate

disease negative

+

-

Possible sources of bias in diagnostic accuracy studies

2/2

study participants

index test

reference test

disease positive

symptomatic patients

inclusion + exclusion criteria

Treatment paradox

gate

disease negative

Review bias

study participants

index test

reference test

disease positive

symptomatic patients

inclusion + exclusion criteriagate

disease negative

index test

Subjects in whom reference standard was not done are excluded.

Do we really need such test?

Two gate design(case – control)

gate 1 gate 2

patients

index test

healthy controls

index test

This is just to remind you what kind of test we need…

Patient historyClinical exam

Pre-test probability

Diagnostic test

Post-test probability

Does this patient have AMI?

cTn-I

AMI + AMI -

Evidence based laboratory medicine

Case-control (two gate ) design

Example…

Staack A, Badendieck S, Schnorr D, Loening SA, Jung K. BMC Urol. 2006;6:19

BMC Urology has an Impact Factor of 1.606.

Staack A, Badendieck S, Schnorr D, Loening SA, Jung K. BMC Urol. 2006;6:19

Staack A, Badendieck S, Schnorr D, Loening SA, Jung K. BMC Urol. 2006;6:19

ne uključuju opće smjernice za pisanje znanstvenog rada

poboljšati kvalitetu istraživanja

olakšati čitanje radova

Reporting guidelines

• these guidelines refer to the:• study design,

• study methodology and reporting

• these guidelines ensure study: • comparability,

• transparency,

• reproducibility

www.equator-network.org/reporting-guidelines

STARD Statement

Published in 2003 and updated in 2015:◦ Clinical Chemistry

◦ Annals of Internal Medicine

◦ Radiology

◦ BMJ

◦ Lancet

◦ American Journal of Clinical Pathology

◦ Clinical Biochemistry

◦ CCLM

STARD Statement(Flow chart)

STARD Statement (Checklist) 1/2

STARD Statement (Checklist) 2/2

How to assess the quality of a diagnostic accuracy

study?

Whiting PF et al. Ann Intern Med. (2011)

From: QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies

Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009

Risk of Bias and Applicability Judgments in QUADAS-2

Copyright © American College of Physicians. All rights reserved.

From: QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies

Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009

Risk of Bias and Applicability Judgments in QUADAS-2

Copyright © American College of Physicians. All rights reserved.

From: QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies

Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009

Copyright © American College of Physicians. All rights reserved.

From: QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies

Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009

Risk of Bias and Applicability Judgments in QUADAS-2

This tool allows more transparent rating of bias and applicability of primary diagnostic accuracy studies.

Read carefully!Be skeptical!

Question everything!