Rapid Critical Appraisalof diagnostic accuracy studies Professor Paul GlasziouCentre for Evidence Based MedicineUniversity of Oxfordwww.cebm.net

What are tests used for?Diagnosis what is the problem? Log of reasons by several docs:Monitoring has it changed?Prognosis risk/stage within DxTreatment planning, e.g., location Stalling for time!

Is the test accurate?To be accurate a test should be:ReproducibleWe get the same (wrong?) answer every timeP I-I questionValidWe get the right answerP I O question

Reproducibility:Agreement of histopathologistsKen Fleming, Evidence-based pathology. EBM 1997

Diagnostic Test Accuracy measurementsSensitivity is the probability of a positive test in a diseased personSpecificity is the probability of a negative test in a non-diseased person.

Is the test helpful (valid)?The Youden IndexYouden Index = sensitivity+specificity-1For a test to be useful, thensensitivity + specificity > 1 (Youden Index > 0)Examples:Coin Toss with +ve = "heads" sensitivity = 0.5 specificity = 0.5 Youden = 0

Can a test rule-in or rule-out?SpPlnSpecific test, Positive rules In eg: Rovsing's sign, ST elevation > 2mmSnNoutSensitive test, Negative rules Out eg: Erect abdominal film for obstruction, Elevated WCC in CSF (>5/mm )

Can I trust the accuracy data from the study?RAMMbo Recruitment: Was an appropriate spectrum of patients included? (Spectrum Bias)Maintainence: All patients subjected to a Gold Standard?(Verification Bias)Measurements: Was there an independent, blind or objective comparison with a Gold Standard? Observer Bias; Differential Reference Bias

Appraisal of Tests: RAMMbo was the evaluatioin fair?Outcome Measures(Gold Standard)

Outcome Measures (Gold Standard)Index testComparator TestMaintained?Representation?Population

Blinded or Objective?

QUADAS

The Literary Digest PollLandon versus Roosevelt, 1936 % for RooseveltLiterary Digest: 2.4 Million reader pollPrediction for Roosevelt 43%Gallup's 50,000 random samplePrediction of the election result 56%Gallup's 3,000 Digest readersPrediction of Digest prediction 44%Election result 62%

Good sampling: needs a sample frame & unbiased selectionTarget PopulationSample FrameActual SampleComplete data

Were reference Measurements Blinded or Objective?Index +veIndex -veApparent differenceUse standardised measurement strategy across ALL patients

Smith H, et al BMJ, 2000BNP screen of GP elderly patientsUK GP setting155 patients70-84 yrs oldEchocardiogram12 with CCF

Assessment process systematic reviewsAssessment process original studies

Potentially Eligible Systematic Reviews 1999-2002N=191

157 systematic reviews excluded 34 systematic reviewswith 39 meta analysescontaining678 original studies

6 systematic reviews with 8 meta-analyses excluded28 systematic reviewswith31 meta-analysescontaining545 original studies 31 meta-analysescontaining487 original studies

58 original studies excludedReplication & Extension StudyAWS Rutjes et al. 2005

487 studies; 31 meta-analyses

How well are diagnostic studies reported?112 studies in 4 major journals (1978-1993)

StandardN(%)Spectrum composition 30(27)Avoidance of workup bias51(46)Avoidance of review bias43 (38)Test accuracy precision12(11)Indeterminate test results26(23)Test reproducibility26(23)Accuracy in subgroups9(8)

Reid MC, Lachs MS, Feinstein AR. Use of Methodological Standardin diagnostic test research. JAMA 1995;274:645-651

Using Evidence about TestsAppraise the study - PICO

Always askIs it useful at all? (Youden Index > 0)Usually askCan it rule in or rule out a disease?Often askWhat is the post-test probability in the same situation as the study?Rarely askCalculation of post-test probabilities in a different situation

*********A type of experimental research. Remember it fitted into that second category of epidemiological study the experimental or interventional type.

A randomised controlled trial (RCT) is a research trial in which subjects are randomly assigned to two groups: one (the experimental group) receiving the intervention that is being tested (eg. a drug), and the other (the comparison group or control) receiving a placebo or a conventional treatment The two groups are then followed up to see if any differences between them result. They are thus used to assess effectiveness of interventions (e.g. a drug)

Lets look at the example of the 4S Trial, which again, you know about because David Greaves covered this last week.*********