1 midas command - Boston Collegefm · 2 midas: meta-analysis of diagnostic accuracy studies 1 midas...

2 midas: meta-analysis of diagnostic accuracy studies

1 midas command

1.1 Description

midas is a comprehensive program of statistical and graphical routines for undertak-ing meta-analysis of diagnostic test performance in Stata. The index and referencetests (gold standard) are dichotomous. Primary data synthesis is performed within thebivariate mixed-effects regression framework focused on making inferences about av-erage sensitivity and specificity. The bivariate approach was originally developed fortreatment trial meta-analysis (Houwelingen et al. 1993; van Houwelingen et al. 2002)and modified for synthesis of diagnostic test data using an approximate normal within-study model (Reitsma et al. 2005; Riley et al. 2007a,b, 2008; Arends et al. 2008). Anexact binomial rendition (Chu and Cole 2006; Riley et al. 2007b; Arends et al. 2008)of the bivariate model assumes independent binomial distributions for the true pos-itives and true negatives conditional on the sensitivity and specificity in each study.Likelihood-based estimation of the exact binomial approach may be performed by adap-tive gaussian quadrature using Stata-native xtmelogit command (Stata release 10) orgllamm (Rabe-Hesketh et al. 2004, 2002), user-written command, both with readilyavailable post-estimation procedures for model diagnostics and empirical Bayes pre-dictions. Additionally, midas facilitates exploratory analysis of heterogeneity (unob-served, threshold-related and covariate etc.), publication and other precision-relatedbiases. Bayes’ nomograms, likelihood-ratio matrices, and probability modifying plotsmay be derived and used to guide patient-based diagnostic decision making.

1.2 Syntax

midas varlist(min=4 max=4)[if

] [in

],

[options

]The required varlist is the data from the contingency tables of index and reference testresults. The user provides the data in a rectangular array containing variables for the2x2 elements a, b, c and d. Each data row contains the 2x2 data for one observation(i.e.study). The varlist MUST contain variables for a, b, c and d in that order:

Reference Test Positive Reference Test NegativeTest Positive a = true positives c = false negativesTest Negative b = false positives d = true negatives

1.3 Options

Modeling and Post-estimation Options

B.A. Dwamena, R. Sylvester and R.C. Carlos 3

nip(integer) specifies the number of integration points used for maximum likelihoodestimation based on adaptive gaussian quadrature. Default is set at 1 for midas eventhough the default in xtmelogit is 7. Higher values improve accuracy at the expenseof execution times. Using xtmelogit with nip(1), model will be estimated by Lapla-cian approximation. This decreases substantially computational time and yet providesreasonably valid fixed effects estimates. It may, however, produce biased estimates ofthe variance components.

ebpred(for|roc) generates a forest plot or roc curve of empirical Bayes versus observedestimates of sensitivity and specificity.

modchk(gof|bvn|inf|out|all) provides graphical model checking capabilities: mod-chk(gof) displays quantile plot of residual-based goodness-of fit; modchk(bvn) dis-plays Chi-squared probability plot of squared Mahalanobis distances for assessment ofthe bivariate normality assumption; modchk(inf) spikeplot for checking for particularlyinfluential observations using Cook’s distance; modchk(out) displays a scatter plot forchecking for outliers using standardized predicted random effects (standardized level-2residuals); and modchk(all) provides a composite graphic of all four plots.

Quality Assessment Options

qtab(varlist) creates a table showing frequency of methodological quality items.

qbar(varlist) calculates study-specific quality scores and plots a bargraph of method-ological quality.

qlab may be combined with qtab(varlist) or qbar(varlist) to use variable labels fortable and bargraph of methodological items.

Exploratory Graphics

bivbox implements a bivariate generalization of the box plot for univariate data similarto the bivariate box plot(Rousseeuw et al. 1999). It is used to assess location, spread,correlation, skewness and tails of the data and for identifying possible outliers.

chiplot creates a chiplot(Fisher and Switzer 2001) for judging whether or nor thepaired performance indices are independent by augmenting the scatter plot with anauxiliary display. In the case of independence, the points will be concentrated in thecentral region, in the horizontal band indicated on the plot.


Main Reporting Options

results(all) provide summary statistics for all performance indices, group-specific between-study variances, likelihood ratio test statistics and other global homogeneity tests.

results(het) provide group-specific between-study variances, likelihood ratio test statis-tics and other global homogeneity tests.

results(sum) provides summary statistics for all performance indices i.e. sensitiv-ity/specificity, positive/negative likelihood ratios and diagnostic score/odds ratios.

table(dss|dlor|dlr) will create a table of study specific performance estimates withmeasure-specific summary estimates and results of homogeneity (chi-square) and incon-sistency(I squared)tests. dss, dlr or dlor represent the paired performance measuressensitivity/specificity, positive/negative likelihood ratios and diagnostic score/odds ra-tios.

Forest Plots Options

id(varlist) provides a label for studies allowing up to 4 variables.

bforest(dss|dlr|dlor) creates summary graphs with study-specific(box) and overall(diamond)point estimates and confidence intervals for each performance index pair using graphcombine see [G] graph: graph combine.

uforest(dss|dlr|dlor) creates univariate summary graphs with study-specific(box) andoverall(diamond) point estimates and confidence intervals allowed to extend between 0and 1000 beyond which they are truncated and marked by a leading arrow.

fordata adds study-specific performance estimates and 95% CIs to right y-axis.

forstats adds heterogeneity statistics below summary point estimate.

ROC Curve Options

rocplane plots observed data in receiver operating characteristic space (ROC Plane)for visual assessment of threshold effect, a source of heterogeneity unique to diagnosticmeta-analysis. The higher the cut-off value, the higher will be the specificity and thelower the sensitivity. This interdependence between sensitivity and specificity based


on threshold variability may be tested a priori using a rank correlation test such asSpearman’s rho. In midas, the proportion of variation due to threshold effects is calcu-lated as the squared correlation coefficient estimated from the between-study covarianceparameter.

sroc(none|pred|conf|both) plots observed data points, summary operating sensitivity-specificity, and SROC curve without or with either or both of confidence and predictionregions at default or specified confidence level.

sroc(nnoc|pnoc|cnoc|bnoc) plots observed data points, summary operating sensitivity-specificity without or with either or both of confidence and prediction contours at defaultor specified confidence level. No SROC curve is plotted with these choices.

Heterogeneity Options

galb(tpr|tnr|dlor|lrp|lrn) option produces Galbraith(radial) plots of standardized logittransformed proportion (tpr, tnr ) or log-transformed ratio(lrp, lrn and dlor) againstthe inverse of the its precision(x-axis). A regression line that goes through the origin iscalculated, together with 95% boundaries (starting at +2 and -2 on the y-axis). Studiesoutside these 95% boundaries may be considered as outliers.

regvars(varlist) permits univariable meta-regression analysis of one or multiple di-chotomous or continuous covariables, reporting results in table and forest plot

Publication Bias Options

pubbias When this option is invoked, midas performs linear regression of log oddsratios on inverse root of effective sample sizes as a test for funnel plot asymmetry indiagnostic meta-analysis. A non-zero slope coefficient is suggestive of significant smallstudy bias (p value < 0.10). The regression line is superimposed on a funnel plot (seefigure 8).

Clinical Utility Options

fagan(0-0.99) creates a plot showing the relationship between the prior probabilityspecified by user, the likelihood ratio(combination of sensitivity and specificity), andposterior test probability.

pddam(lbp ubp) produces a line graph of post-test probabilities versus prior prob-abilities between 0 and 1 using summary likelihood ratios. Summary unconditional


predictive values based on sensitivity analysis using uniformly distributed prior proba-bilities between lbp and ubp are estimated and displayed on graph.

lrmatrix creates a scatter plot of positive and negative likelihood ratios with combinedsummary point. Plot is divided into quadrants based on strength-of-evidence thresholdsto determine informativeness of measured test.

Miscellaneous Options

level(#) specifies the significance level for statistical tests, confidence regions, predic-tion regions and confidence intervals.

mscale(#) affects size of markers for point estimates on forest plots.

scheme(string) permits choice of scheme for graphs. The default is s2color.

textscale(#) allows choice of text size for graphs especially regarding labels for forestplots.

zcf(#) defines a fixed continuity correction in the case where a study contains a zerocell during logit or log transformations only to calculate study-specific likelihood ratiosand odds ratios. By default, midas adds 0.5 to each cell of a study where a zero isencountered. However, the zcf(#) option allows the use of other constants between 0and 1.

q Technical note

Although midas has pre-programmed graphical options, with Graph Editor (StataRelease 10 or later), you can change almost anything on your graph. You can add text,lines, arrows, and markers wherever you would like. You can right-click on any object tosee a list of operations specific to the object and tool you are working with. This featureis most useful with the Pointer tool. However, there is no record of what you have donewith the graph editor, so if you need to recreate the graph for some reason, you will haveto redo everything that you have done with the graph editor. See [G] graph editor.

q


1.4 Saved results

midas saves results as scalars in r() such as:r(mtpr) Mean sensitivityr(mtprse) standard error of mean sensitivityr(mtnr) Mean specificityr(mtnrse) Standard error of mean specificityr(mlrp) Mean likelihood ratio of a positive test resultr(mlrpse) Standard error of mean likelihood ratio of a positive test resultr(mlrn) Mean likelihood ratio of a negative test resultr(mlrnse) Standard error of mean likelihood ratio of a negative test resultr(mdor) Mean diagnostic odds ratior(mdorse) Standard error of mean diagnostic odds ratior(AUC) Area under summary ROC curver(AUClo) Lower bound of area under summary ROC curver(AUChi) Upper bound of area under summary ROC curver(rho) Correlation between logits of sensitivity and specificityr(reffs1) Variance of logit of sensitivityr(reffs1se) Standard error of variance of logit of sensitivityr(reffs2) Variance of logit of specificityr(reffs2se) Standard error of variance of logit of specificityr(Islrt) Global inconsistency index from likelihood ratio testr(Islrtlo) Lower bound global inconsistency indexr(Islrthi) Upper bound global inconsistency index

2 Example Dataset

This dataset was obtained as part of a systematic review and meta-analysis of the pub-lished literature on the staging performance of axillary positron emission tomography(FDG-PET) in breast cancer toward identification of the number, quality and scopeof primary studies; quantification of overall classification performance (sensitivity andspecificity), discriminatory power (diagnostic odds ratios) and informational value (di-agnostic likelihood ratios); assessment of the impact of technical characteristics of test,methodological quality of primary studies and publication selection bias on estimatesof diagnostic accuracy; and highlighting of any potential issues that require furtherresearch. We performed a comprehensive computer search of the English Languagemedical literature using primarily the PUBMED (MEDLINE) search engine and cross-citation with other databases to identify original peer-reviewed full-length human sub-ject articles published between January 1, 1990 and January 31, 2008. Search was con-ducted using database-specific Boolean search strategies based on: (”breast neoplasm”OR ”breast cancer” OR ”breast malignancy”) AND (”Positron emission tomography”OR ”FDG-PET”) AND (”axillary staging” OR ”axillary metastases” OR ”axillary nodemetastases” OR ”axillary node staging”). Search strategies were augmented with amanual search of reference lists from identified articles and recent subject-area journalsfor additional articles. Details of the data set and 2 by 2 data for the first 20 studies


as shown below were generated respectively with the describe and list commands ofStata:

. describe

Contains data from f:\breastpet.dtaobs: 39

vars: 33 7 Jan 2009 16:06size: 2,652 (99.9% of memory free)

storage display valuevariable name type format label variable label

author str13 %-15s first authoryear int %8.0g Year of Publicationtp int %8.0g True Positivefp byte %8.0g False Positivefn float %9.0g False Negativetn float %9.0g True Negativeperiod byte %8.0g Period of publication(before or

after 1998)prodesign byte %10.0g Prospective designssize30 byte %7.0g Study size greater than 30fulverif byte %8.0g Full Verification of test resultstestdescr byte %9.0g Satisfactory description of index

testrefdescr byte %9.0g Statisfactory description of ref

testclindescr byte %9.0g Clinical Information Availablereport byte %7.0g Satisfactory reporting of resultsconsel byte %7.0g Consecutive selection of subjectsbrdspect byte %10.0g Broad spectrum of diseaseblindref byte %7.0g blinded reference test

interpretationsampsize int %8.0g Number Data Units Per Studyqscore byte %9.0g Quality scoreage byte %8.0g Mean Agefast byte %8.0g Prior Fasting of at leasst 4-6

hoursres byte %8.0g Resolution of PET Cameradose int %8.0g Appropriate dose usedupttime byte %8.0g Uptake Period specifiedacquitime byte %8.0g Duration of image acqusitiontestcrit byte %8.0g Test criteria describedmultiobs byte %8.0g Multiple Observersattcor byte %9.0g Attenuation correction of PET

imagesaxonly byte %8.0g Dedicated axillary imagingpmet float %9.2g Study-specfic Prevalence of

axillary MetastasesAnalysis str7 %9s Unit of Data Analysis(Patient

versus Nodes)patient byte %10.0g Mode of analysisblindtest byte %8.0g blinded index test interpretation

(Continued on next page)


. list author year pmet sampsize tp fp fn tn in 1/20, sep(1) ab(32) abs noo

author year pmet sampsize tp fp fn tn

Tse 1992 .7 10 4 0 3 3

Adler1 1993 .47 18 8 0 1 10

Hoh 1993 .64 14 6 0 3 5

Crowe 1994 .5 20 9 0 1 10

Avril 1996 .47 51 19 1 5 26

Bassa 1996 .81 16 10 0 3 3

Scheidhauer 1996 .5 18 9 1 0 8

Utech 1996 .35 124 44 20 0 60

Adler2 1997 .38 50 19 11 0 20

Palmedo 1997 .3 20 5 0 1 14

Noh 1998 .54 24 12 0 1 11

Smith 1998 .42 50 19 1 2 28

Rostom 1999 .65 74 42 0 6 26

Yutani1 1999 .38 26 8 0 2 16

Hubner 2000 .27 22 6 0 0 16

Ohta 2000 .59 32 14 0 5 13

Yutani2 2000 .42 38 8 0 8 22

Greco 2001 .43 167 68 13 4 82

Schirrmeister 2001 .3 113 27 6 7 73

Yang 2001 .33 18 3 0 3 12

3 Annotated Worked Examples

3.1 Model prediction and diagnostics

Post-estimation predictions may be obtained from the estimated model, parameter es-timates and empirical Bayes estimates of the random effects with standard errors com-puted with the delta method. The predicted logits may then be transformed to obtainpredictions of sensitivity and specificity. midas generates a forest plot or roc curve


of empirical Bayes versus observed estimates of sensitivity and specificity using theebpred(for|roc) options. Figure 1 was obtained using the syntax below:

. midas tp fp fn tn, eb(for)

Model diagnostics are rarely performed because many non-statisticians and appliedresearchers consider meta-analysis as a data-processing procedure rather than a model-fitting exercise (Sutton and Higgins 2008). However, with the application of complexlikelihood-based meta-analytic models, it is important to evaluate possible model mis-specification, goodness of fit, and to identify outlying and possibly influential datapoints. midas provides graphical model checking capabilities: quantile plot of residual-based goodness-of fit; Chi-squared probability plot of squared Mahalanobis distancesfor assessment of the bivariate normality assumption; spikeplot for checking for par-ticularly influential observations using Cook’s distance; a scatter plot for checking foroutliers using standardized predicted random effects (standardized level-2 residuals);and composite graphic of all four plots such as figure 2 which was obtained using thesyntax below:

. midas tp fp fn tn, modchk(all)

3.2 Quality Assessment

Methodologic quality of a study is the extent to which all aspects of a study’s design andconduct can be shown to protect against systematic bias, nonsystematic bias that mayarise in poorly performed studies, and inferential error. The recently developed qualityassessment tool for diagnostic accuracy studies (QUADAS) (Whiting et al. 2005, 2004,2003, 2006), is a rigorously constructed and validated tool that can be used by investiga-tors undertaking new systematic reviews. The QUADAS tool consists of 14 items thatcover patient spectrum, reference standard, disease progression bias, verification andreview bias, clinical review bias, incorporation bias, test execution, study withdrawals,and intermediate results. Possible methods to address quality differences are sensitivityanalysis, subgroup analysis, or meta-regression analysis. Quality assessment may besummarized by stacked bars for each QUADAS item in a bar graph. midas providesthe ability to represent the results of quality assessment by means of a bar graph. Forexample, figure 3 was obtained with the command syntax:

. midas tp fp fn tn, qbar(prodesign fulverif testdescr refdescr clindescr report> spectrum blindref blindtest) qlab

Alternatively using the qtab(varlist) produces frequency table of quality scores.


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

3939

study

0.00 0.25 0.50 0.75 1.00

Sensitivity

0.00 0.25 0.50 0.75 1.00

Specificity

MLE of mean sensitivity and specificity (solid vertical lines)Empirical Bayes (solid lines and markers)

Observed data (dashed lines and markers)

Figure 1: Paired forest plot depiction of empirical Bayes predicted versus observedsensitivity and specificity


0.00

0.25

0.50

0.75

1.00

Dev

ianc

e R

esid

ual

0.00 0.25 0.50 0.75 1.00Normal Quantile

(a) Goodness−Of−Fit

0.00

0.25

0.50

0.75

1.00

Mah

alan

obis

D−

squa

red

0.00 0.25 0.50 0.75 1.00Chi−squared Quantile

(b) Bivariate Normality

8

9

29

32

0.00

0.50

1.00

1.50

2.00

Coo

k’s

Dis

tanc

e

0 10 20 30 40study

(c) Influence Analysis

66

2626

3337

3755

1

130

30

16161919 10

101414

20

2038

38

2

2

4

4

24

2412

12

17

1722

2211

11

33

33

36

36

15

15

7

7

31

31

13

13

23

2339

39

35

35

25

25

18

1827

27 28

2821 2134

34

32

32

8

8

29

299

9

−3.0

−2.0

−1.0

0.0

1.0

2.0

3.0

Sta

ndar

dize

d R

esid

ual(D

isea

sed)

−3.0 −2.0 −1.0 0.0 1.0 2.0 3.0Standardized Residual(Healthy)

(d) Outlier Detection

Figure 2: Graphical depiction of residual-based goodness-of-fit, bivariate normality,influence and outlier detection analyses

0 20 40 60 80 100percent

Blinded index test interpretation

Blinded reference test interpretation

Clinical Information Available

Full Verification of test results

Prospective design

Representative Spectrum

Satisfactory description of index test

Satisfactory reporting of results

Statisfactory description of ref test

Yes No

Figure 3: Bar graph of quality assessment displaying stacked bars for each quality itemincluding modified QUADAS criteria


34

222436

23

30

39

20

28

31

27

17

38

16

14

3533

13

105

15

1211

42

26

8

187

9

19

376

2129

32

3

125

−2

0

2

4

LOG

IT_S

EN

S

0 1 2 3 4LOGIT_SPEC

Bivariate Boxplot

Figure 4: Bivariate box plot with most studies clustering within the median distributionwith seven outliers suggesting indirectly a lower degree of heterogeneity

3.3 Bivariate Association

midas uses a bivariate random effects modeling of sensitivity and specificity, therefore,it is expected that this pair of performance measures will be interdependent (see Figure3). The bivariate box plot describes the degree of interdependence including the cen-tral location and identification of any outliers. The inner oval represents the mediandistribution of the data points. The outer oval represents the 95% confidence bound.We obtained figure 4 with the syntax:

. midas tp fp fn tn, bivbox

It demonstrates a skewedness of the test performance measures toward a higher speci-ficity with lower sensitivity, providing indirect evidence of some threshold variability.

3.4 Summary Performance Estimates

Summary estimates of sensitivity and specificity and their 95% confidence intervalscan be calculated after anti-logit transformation of of the mean logit sensitivity andlogit specificity and respective standard errors. These intervals take into account theheterogeneity beyond chance between studies (random effects model).

. midas tp fp fn tn, res(sum) nip(1)

(Continued on next page)


SUMMARY PERFORMANCE ESTIMATES

Parameter Estimate 95% CI

Sensitivity 0.73 [ 0.63, 0.80]

Specificity 0.96 [ 0.92, 0.97]

Positive Likelihood Ratio 16.2 [ 9.7, 27.3]

Negative Likelihood Ratio 0.29 [ 0.21, 0.39]

Diagnostic Odds Ratio 57 [ 30, 107]

3.5 Heterogeneity Statistics

SENSITIVITY (95% CI)

Q =273.36, df = 38.00, p = 0.00I2 = 86.10 [82.45 − 89.75]

0.73[0.63 − 0.80]

0.57 [0.18 − 0.90]0.89 [0.52 − 1.00]0.67 [0.30 − 0.93]0.90 [0.55 − 1.00]0.79 [0.58 − 0.93]0.77 [0.46 − 0.95]1.00 [0.66 − 1.00]1.00 [0.92 − 1.00]1.00 [0.82 − 1.00]0.83 [0.36 − 1.00]0.92 [0.64 − 1.00]0.90 [0.70 − 0.99]0.88 [0.75 − 0.95]0.80 [0.44 − 0.97]1.00 [0.54 − 1.00]0.74 [0.49 − 0.91]0.50 [0.25 − 0.75]0.94 [0.86 − 0.98]0.79 [0.62 − 0.91]0.50 [0.12 − 0.88]0.68 [0.43 − 0.87]0.43 [0.18 − 0.71]0.20 [0.01 − 0.72]0.47 [0.21 − 0.73]0.53 [0.27 − 0.79]0.80 [0.56 − 0.94]0.25 [0.11 − 0.43]0.21 [0.05 − 0.51]0.67 [0.09 − 0.99]0.60 [0.42 − 0.76]0.36 [0.18 − 0.57]0.61 [0.51 − 0.70]0.84 [0.76 − 0.90]0.28 [0.10 − 0.53]0.85 [0.77 − 0.90]0.44 [0.28 − 0.62]0.80 [0.28 − 0.99]0.60 [0.43 − 0.74]0.37 [0.28 − 0.47]0.37 [0.28 − 0.47]

StudyId

COMBINED

Tse/1992Adler1/1993

Hoh/1993Crowe/1994

Avril/1996Bassa/1996

Scheidhauer/1996Utech/1996

Adler2/1997Palmedo/1997

Noh/1998Smith/1998

Rostom/1999Yutani1/1999Hubner/2000

Ohta/2000Yutani2/2000

Greco/2001Schirrmeister/2001

Yang/2001Danforth/2002

Guller/2002Kelemen/2002

Nakamoto1/2002Nakamoto2/2002

Rieber/2002Van_Hoeven/2002

Barranger/2003Fehr/2004

Inoue/2004Lovrics/2004

Wahl/2004Zornoza/2004

Weir/2005Gil−Rendo/2006

Kumar/2006Stadnik/2006Chung/2006

Veronesi/2006

0.0 1.0SENSITIVITY

SPECIFICITY (95% CI)

Q =217.71, df = 38.00, p = 0.00I2 = 82.55 [77.65 − 87.44]

0.96[0.92 − 0.97]

1.00 [0.29 − 1.00]1.00 [0.69 − 1.00]1.00 [0.48 − 1.00]1.00 [0.69 − 1.00]0.96 [0.81 − 1.00]1.00 [0.29 − 1.00]0.89 [0.52 − 1.00]0.75 [0.64 − 0.84]0.65 [0.45 − 0.81]1.00 [0.77 − 1.00]1.00 [0.72 − 1.00]0.97 [0.82 − 1.00]1.00 [0.87 − 1.00]1.00 [0.79 − 1.00]1.00 [0.79 − 1.00]1.00 [0.75 − 1.00]1.00 [0.85 − 1.00]0.86 [0.78 − 0.93]0.92 [0.84 − 0.97]1.00 [0.74 − 1.00]0.67 [0.30 − 0.93]0.94 [0.71 − 1.00]1.00 [0.69 − 1.00]0.95 [0.76 − 1.00]0.86 [0.64 − 0.97]0.95 [0.75 − 1.00]0.97 [0.86 − 1.00]1.00 [0.81 − 1.00]0.62 [0.38 − 0.82]0.96 [0.85 − 0.99]0.97 [0.89 − 1.00]0.80 [0.74 − 0.85]0.98 [0.92 − 1.00]0.86 [0.65 − 0.97]0.98 [0.95 − 1.00]0.95 [0.84 − 0.99]1.00 [0.48 − 1.00]1.00 [0.81 − 1.00]0.96 [0.91 − 0.99]0.96 [0.91 − 0.99]

StudyId

COMBINED

Tse/1992Adler1/1993

Hoh/1993Crowe/1994

Avril/1996Bassa/1996

Scheidhauer/1996Utech/1996

Adler2/1997Palmedo/1997

Noh/1998Smith/1998

Rostom/1999Yutani1/1999Hubner/2000

Ohta/2000Yutani2/2000

Greco/2001Schirrmeister/2001

Yang/2001Danforth/2002

Guller/2002Kelemen/2002

Nakamoto1/2002Nakamoto2/2002

Rieber/2002Van_Hoeven/2002

Barranger/2003Fehr/2004

Inoue/2004Lovrics/2004

Wahl/2004Zornoza/2004

Weir/2005Gil−Rendo/2006

Kumar/2006Stadnik/2006Chung/2006

Veronesi/2006

0.3 1.0SPECIFICITY

Figure 5: Forest plot showing study-specific (right-axis) and mean sensitivity and speci-ficity with corresponding heterogeneity statistics

The impact of unobserved heterogeneity is traditionally assessed statistically usingthe quantity I2. It describes the percentage of total variation across studies that isattributable to the heterogeneity rather than chance(Higgins and Thompson 2002).


I2 can be calculated from heterogeneity statistic and the degrees of freedom (Higginsand Thompson 2002). Alternatively, for mixed models, it is the intraclass correlationcoefficient expressed as a percentage with similar interpretation and is implementedin midas separately for sensitivity and specificity. I2 lies between 0% and 100%. Avalue of 0% indicates no observed heterogeneity, and values greater than 50% maybe considered substantial heterogeneity. The main advantage of I2 is that it does notinherently depend on the number of studies in the meta-analysis (Higgins and Thompson2002). The midas output of heterogeneity statistics is shown below and was generatedfrom the syntax:

. midas tp fp fn tn, res(het) nip(1)

HETEROGENEITY STATISTICS

Heterogeneity (Chi-square): LRT_Q = 146.822, df =2.00, LRT_p =0.000

Inconsistency (I-square): LRT_I2 = 99, 95% CI = [ 98- 99]

Proportion of heterogeneity likely due to threshold effect = 0.11

Interstudy variation in Sensitivity: ICC_SEN = 0.31, 95% CI = [ 0.18- 0.45]

Interstudy variation in Sensitivity: MED_SEN = 0.76, 95% CI = [ 0.70- 0.83]

Interstudy variation in Specificity: ICC_SPE = 0.29, 95% CI = [ 0.12- 0.45]

Interstudy variation in Specificity: MED_SPE = 0.75, 95% CI = [ 0.68- 0.84]

3.6 Forest plot to demonstrate study-specific sensitivity and speci-ficity on right y-axis

Within midas, forest plots can be created for each test performance parameter individu-ally or may be displayed as paired plots, for example, sensitivity paired with specificity.The user has the option of displaying heterogeneity statistics within the forest plotsusing the option forstats. Figure 5 was obtained using the command syntax:

. midas tp fp fn tn, texts(0.60) bfor(dss) id(author year) ford fors

3.7 Summary ROC Curve

Based on parameters estimated by the bivariate model, several summary ROC linearregression lines based on either the regression of logit sensitivity on specificity, the re-gression of logit specificity on sensitivity, or an orthogonal regression line by minimizingthe perpendicular distances may be derived. These lines can be transformed back to theoriginal ROC scale to obtain a summary ROC curve. In midas, derived logit estimatesof sensitivity, specificity and respective variances are used to construct a hierarchicalsummary ROC curve. midas has several options for the display of the ROC curve. Forexample, figure 6 was obtained by using the syntax below:

. midas tp fp fn tn, sroc(both)


1

2

3

4

56

7 8 9

10

1112

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

0.0

0.5

1.0

Sen

sitiv

ity

0.00.51.0Specificity

Observed Data

Summary Operating PointSENS = 0.73 [0.63 − 0.80]SPEC = 0.96 [0.92 − 0.97]

SROC CurveAUC = 0.95 [0.92 − 0.96]

95% Confidence Contour

95% Prediction Contour

Figure 6: Summary ROC curve with confidence and prediction regions around meanoperating sensitivity and specificity point


The summary ROC curve is displayed along with the observed study data. The dashedline around the summary point estimate, represents the 95% confidence region. The areaunder the curve (AUROC), serves as a global measure of test performance. The AUROCis the average TPR over the entire range of FPR values. The following guidelines havebeen suggested for interpretation of intermediate AUROC values: low (0.5>= AUC <=0.7), moderate (0.7 >= AUC <= 0.9), or high (0.9 >= AUC <= 1) accuracy (Swets1988).

3.8 Meta-regression

Meta-regression, the use of regression methods to incorporate the effect of covaryingfactors on summary measures of performance, has been used to explore between-studyheterogeneity in therapeutic studies. In diagnostic studies, likewise, heterogeneity insensitivity and specificity can result from many causes related to definitions of the testand reference standards, operating characteristics of the test, methods of data collection,and patient characteristics. Covariates may be introduced into a regression with anytest performance measure as the dependent variable. As with any meta-regression,however, the sample size will correspond to the number of studies in the analysis withsmall number of studies limiting the power of regression to detect significant effects.The syntax below produces both tabular and graphical output (Figure 7):

. midas tp fp fn tn, reg(prodesign age qscore sampsize fulverif testdescr refd> escr clindescr report spectrum blindref blindtest)

(output omitted )

Parameter category LRTChi2 Pvalue I2 I2lo I2hi

prodesign Yes 2.73 0.26 27 0 100No . . . . .

age 7.18 0.03 72 38 100qscore 4.84 0.09 59 7 100sampsize 0.83 0.66 0 0 100fulverif Yes 0.03 0.99 0 0 100

No . . . . .testdescr Yes 1.79 0.41 0 0 100

No . . . . .refdescr Yes 0.61 0.74 0 0 100

No . . . . .clindescr Yes 6.75 0.03 70 34 100

No . . . . .report Yes 2.75 0.25 27 0 100

No . . . . .spectrum Yes 1.15 0.56 0 0 100

No . . . . .blindref Yes 8.35 0.02 76 48 100

No . . . . .blindtest Yes 25.80 0.00 92 85 99

No . . . . .


prodesign Yes

No

age

qscore

sampsize

fulverif Yes

No

testdescr Yes

No

refdescr Yes

No

**clindescr Yes

No

report Yes

No

spectrum Yes

No

**blindref Yes

No

**blindtest Yes

No

0.37 1.00Sensitivity(95% CI)

*p<0.05, **p<0.01, ***p<0.001

**prodesign Yes

No

age

qscore

sampsize

**fulverif Yes

No

testdescr Yes

No

refdescr Yes

No

clindescr Yes

No

***report Yes

No

*spectrum Yes

No

**blindref Yes

No

**blindtest Yes

No

**prodesign Yes

No

age

qscore

sampsize

**fulverif Yes

No

testdescr Yes

No

refdescr Yes

No

clindescr Yes

No

***report Yes

No

*spectrum Yes

No

**blindref Yes

No

**blindtest Yes

No

0.85 1.00Specificity(95% CI)

*p<0.05, **p<0.01, ***p<0.001

Univariable Meta−regression & Subgroup Analyses

Figure 7: Forest plot of multiple univariable meta-regression and subgroup analyses


3.9 Linear regression test of funnel plot asymmetry

The funnel plot is generally considered a good exploratory tool for investigating publica-tion bias (Light and Pillemer 1984), plotting a measure of effect size against a measureof study precision, appearing symmetric if no bias is present. However, assessment ofsuch a plot is very subjective. Thus, non-parametric and parametric linear regressionmethods have been developed to formally test for such funnel plot asymmetry (Beggand Mazumdar 1994; Egger and Smith 1998; Harbord et al. 2006; Macaskill et al. 2001;Peters et al. 2006; Rucker et al. 2008; Schwarzer et al. 2007, 2002). Using these meth-ods for assessing publication bias in diagnostic test studies may produce misleadingresults (Song et al. 2002; Deeks et al. 2005). Formal testing for publication bias may beconducted by a regression of diagnostic log odds ratio against 1/sqrt(effective samplesize), weighting by effective sample size (Deeks et al. 2005), with P < .10 for the slopecoefficient indicating significant asymmetry.

. midas tp fp fn tn, pubbias

yb Coef. Std. Err. t P>|t| [95% Conf. Interval]

Bias -3.206976 4.332084 -0.74 0.464 -11.98461 5.57066Intercept 4.255449 .5420326 7.85 0.000 3.157187 5.353711

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

2425

26

27

28

29

3031

32

33

34

35

36

37

38

39

.05

.1

.15

.2

.25

.3

1/ro

ot(E

SS

)

1 10 100 1000

Diagnostic Odds Ratio

Study

RegressionLine

Deeks’ Funnel Plot Asymmetry Testpvalue = 0.46

Figure 8: Funnel plot with superimposed regression line

Using the syntax above yields funnel plot with superimposed regression line as shown infigure 8. The statistically non-significant p-value (0.89) for the slope coefficient suggestssymmetry in the data and a low likelihood of publication bias. However, the test isknown to have low power (Deeks et al. 2005).


3.10 Fagan plot (Bayes Nomogram)

The clinical or patient-relevant utility of a diagnostic test is evaluated using the likeli-hood ratios to calculate post-test probability(PTP) based on Bayes’ theorem as follows:Pretest Probability=Prevalence of target condition PTP= LR × pretest probability/[(1-pretest probability)× (1-LR)] This concept is depicted visually with Fagan’s nomograms(Fagan 1975). When Bayes theorem is expressed in terms of log-odds, the posterior log-odds are linear functions of the prior log-odds and the log likelihood ratios. A Faganplot (see figure 9) consists of a vertical axis on the left with the prior log-odds, anaxis in the middle representing the log-likelihood ratio and an vertical axis on the rightrepresenting the posterior log-odds. Lines are then drawn from the prior probabilityon the left through the likelihood ratios in the center and extended to the posteriorprobabilities on the right.Figure 9 demonstrates that FDG-PET is very informative raising probability of axil-

0.0010.0020.0050.010.020.050.10.20.51251020501002005001000

Likelihood Ratio

0.1

0.20.30.50.71

235710

20304050607080

9093959798

9999.399.599.799.8

99.9

Pos

t−te

st P

roba

bilit

y (%

)

0.1

0.20.30.50.7

1

2357

10

20304050607080

9093959798

9999.399.599.799.8

99.9

Pre

−te

st P

roba

bilit

y (%

)

Prior Prob (%) = 25

LR_Positive = 16Post_Prob_Pos (%) = 84

LR_Negative = 0.29Post_Prob_Neg (%) = 9

Figure 9: Fagan plot

lary breast metastases over 3-fold when positive from 25% and lowering the probabilityof disease to as low as 9% when negative. It was generated with the syntax:

. midas tp fp fn tn, fagan(0.25)


3.11 Likelihood ratio scattergram

Informativeness may also be represented graphically by a likelihood ratio scattergram ormatrix (Stengel et al. 2003). It defines quadrants of informativeness based on establishedevidence-based thresholds:

1. Left Upper Quadrant, Likelihood Ratio Positive > 10, Likelihood Ratio Negative<0.1: Exclusion & Confirmation

2. Right Upper Quadrant, Likelihood Ratio Positive >10, Likelihood Ratio Negative>0.1: Confirmation Only

3. Left Lower Quadrant, Likelihood Ratio Positive <10, Likelihood Ratio Negative<0.1: Exclusion Only

4. Right Lower Quadrant, Likelihood Ratio Positive <10, Likelihood Ratio Negative>0.1: No Exclusion or Confirmation

The likelihood ratio scattergram (figure 10) shows summary point of likelihood ratiosobtained as functions of mean sensitivity and specificity (Leeflang et al. 2008) in theright upper quadrant suggesting that FDG-PET is useful for confirmation of presenceof axillary metastatic disease (when positive) and not for its exclusion (when negative).This figure was generated with the midas syntax:

. midas tp fp fn tn, lrmat

3.12 Predictive Values and Probability Modifying Plot

The conditional probability of disease given a positive OR negative test, the so-calledpositive (negative) predictive values are critically important to clinical application of adiagnostic procedure. They depend not only on sensitivity and specificity, but also ondisease prevalence (p). The probability modifying plot is a graphical sensitivity analysisof predictive value across a prevalence continuum defining low to high-risk populations.It depicts separate curves for positive and negative tests. The user draws a verticalline from the selected pre-test probability to the appropriate likelihood ratio line andthen reads the post-test probability off the vertical scale. General summary statisticshave also been introduced (Li et al. 2007) for when it may be of interest to evaluate theeffect of p on predictive values: unconditional positive and negative predictive values,which permit prevalence heterogeneity. These measures are obtained by integratingtheir corresponding conditional (on p) versions with respect to a prior distribution forp. The prior posits assumptions about the risk level in a hypothetical population ofinterest, e.g. low, high, moderate risk, as well as the heterogeneity in the population.


1

2

3

4

5

67

8

9

1011

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

2728

29

30

31

32

33

34

35

3637

38

39

1

10

100

Pos

itive

Lik

elih

ood

Rat

io

0.1 1Negative Likelihood Ratio

LUQ: Exclusion & ConfirmationLRP>10, LRN<0.1RUQ: Confirmation OnlyLRP>10, LRN>0.1LLQ: Exclusion OnlyLRP<10, LRN<0.1RLQ: No Exclusion or ConfirmationLRP<10, LRN>0.1Summary LRP & LRN for Index TestWith 95 % Confidence Intervals

Figure 10: Likelihood ratio scattergram

. midas tp fp fn tn, pddam(0.25 0.75)

0.0

0.2

0.4

0.6

0.8

1.0

Pos

terio

r P

roba

bilit

y

0.0 0.2 0.4 0.6 0.8 1.0Prior Probability

Positive Test ResultLR+ =16.24 [9.66 − 27.31]

Negative Test ResultLR− =0.29 [0.21 − 0.39]

Prevalence HeterogeneityUniform Prior Distribution = [0.25 − 0.75]Unconditional NPV =0.76 [0.74 − 0.78]Unconditional PPV =0.93 [0.91 − 0.96]

Figure 11: Probability Modifying Plot

Using the syntax above generates figure 11, which plots the relationship between pre-and post-test probability based on the likelihood of a positive (above diagonal line) ornegative (below diagonal line) test result over the 0-1 range of pre-test probababilities.


Tests with more informative positive results have curves tending toward the (0,1) loca-tion while tests with more informative negative results produce curves toward the (1,0)location.

4 ReferencesArends, L. R., T. H. Hamza, J. C. Houwelingen, M. H. Heijenbrok-

Kal, M. G. M. Hunink, and T. Stijnen. 2008. Bivariate Random Ef-fects Meta-Analysis of ROC Curves. Med Decis Making 28: 621–628.http://dx.doi.org/10.1177/0272989X08319957.

Begg, C. B., and M. Mazumdar. 1994. Operating characteristics of a rank correlationtest for publication bias. Biometrics 50: 1088–1101.

Chu, H., and S. R. Cole. 2006. Bivariate meta-analysis of sensitivity and specificity withsparse data: a generalized linear mixed model approach. J Clin Epidemiol 59(12):1331–2; author reply 1332–3. http://dx.doi.org/10.1016/j.jclinepi.2006.06.011.

Deeks, J. J., P. Macaskill, and L. Irwig. 2005. The performance of testsof publication bias and other sample size effects in systematic reviews ofdiagnostic test accuracy was assessed. J Clin Epidemiol 58(9): 882–893.http://dx.doi.org/10.1016/j.jclinepi.2005.01.016.

Egger, M., and G. D. Smith. 1998. Bias in location and selection of studies. BMJ316(7124): 61–66.

Fagan, T. J. 1975. Letter: Nomogram for Bayes theorem. N Engl J Med 293(5): 257.

Fisher, N., and P. Switzer. 2001. Graphical assessment of dependence: Is a pictureworth a 100 tests? American Statistician 55: 233–239.

Harbord, R. M., M. Egger, and J. A. C. Sterne. 2006. A modified test for small-studyeffects in meta-analyses of controlled trials with binary endpoints. Stat Med 25(20):3443–3457. http://dx.doi.org/10.1002/sim.2380.

Higgins, J. P. T., and S. G. Thompson. 2002. Quantifying heterogeneity in a meta-analysis. Stat Med 21(11): 1539–1558. http://dx.doi.org/10.1002/sim.1186.

van Houwelingen, H. C., L. R. Arends, and T. Stijnen. 2002. Advanced methods inmeta-analysis: multivariate approach and meta-regression. Stat Med 21(4): 589–624.

Houwelingen, H. C. V., K. H. Zwinderman, and T. Stijnen. 1993. A bivariate approachto meta-analysis. Stat Med 12(24): 2273–2284.

Leeflang, M., J. Deeks, C. Gatsonis, and P. Bossuyt. 2008. Systematic Reviews ofDiagnostic Test Accuracy. Ann Intern Med 149: 889–897.

Li, J., J. Fine, and N. Safdar. 2007. Prevalence-dependent diagnostic accuracy measures.Statistics in Medicine 26: 3258–3273.


Light, L., and D. Pillemer. 1984. Summing Up: The science of reviewing reaearch.Cambridge, MA: Harvard University Press.

Macaskill, P., S. D. Walter, and L. Irwig. 2001. A comparison of meth-ods to detect publication bias in meta-analysis. Stat Med 20(4): 641–654.http://dx.doi.org/10.1002/sim.698.

Peters, J. L., A. J. Sutton, D. R. Jones, K. R. Abrams, and L. Rushton. 2006. Com-parison of two methods to detect publication bias in meta-analysis. JAMA 295(6):676–680. http://dx.doi.org/10.1001/jama.295.6.676.

Rabe-Hesketh, S., A. Skrondal, and A. Pickles. 2002. Reliable estimation of generalizedlinear mixed models using adaptive quadrature. Stata Journal 2: 1–21.

———. 2004. GLLAMM Manual. University of California–Berkeley, Division of Bio-statistics, Working Paper Series. Paper No. 160.http://www.bepress.com/ucbbiostat/paper160/.

Reitsma, J. B., A. S. Glas, A. W. S. Rutjes, R. J. P. M. Scholten, P. M. Bossuyt, andA. H. Zwinderman. 2005. Bivariate analysis of sensitivity and specificity producesinformative summary measures in diagnostic reviews. J Clin Epidemiol 58(10): 982–990. http://dx.doi.org/10.1016/j.jclinepi.2005.02.022.

Riley, R. D., K. R. Abrams, P. C. Lambert, A. J. Sutton, and J. R. Thompson. 2007a.An evaluation of bivariate random-effects meta-analysis for the joint synthesis of twocorrelated outcomes. Stat Med 26(1): 78–97. http://dx.doi.org/10.1002/sim.2524.

Riley, R. D., K. R. Abrams, A. J. Sutton, P. C. Lambert, and J. R. Thompson. 2007b.Bivariate random-effects meta-analysis and the estimation of between-study correla-tion. BMC Med Res Methodol 7: 3. http://dx.doi.org/10.1186/1471-2288-7-3.

Riley, R. D., J. R. Thompson, and K. R. Abrams. 2008. An alternative model for bi-variate random-effects meta-analysis when the within-study correlations are unknown.Biostatistics 9(1): 172–186. http://dx.doi.org/10.1093/biostatistics/kxm023.

Rousseeuw, P., I. Ruts, and J. W. Tukey. 1999. The bagplot, a bivariate boxplot.American Statistician 53: 382–387.

Rucker, G., G. Schwarzer, and J. Carpenter. 2008. Arcsine test for publica-tion bias in meta-analyses with binary outcomes. Stat Med 27(5): 746–763.http://dx.doi.org/10.1002/sim.2971.

Schwarzer, G., G. Antes, and M. Schumacher. 2002. Inflation of type I error rate intwo statistical tests for the detection of publication bias in meta-analyses with binaryoutcomes. Stat Med 21(17): 2465–2477. http://dx.doi.org/10.1002/sim.1224.

———. 2007. A test for publication bias in meta-analysis with sparse binary data. StatMed 26(4): 721–733. http://dx.doi.org/10.1002/sim.2588.


Song, F., K. S. Khan, J. Dinnes, and A. J. Sutton. 2002. Asymmetric funnel plotsand publication bias in meta-analyses of diagnostic accuracy. Int J Epidemiol 31(1):88–95.

Stengel, D., K. Bauwens, J. Sehouli, A. Ekkernkamp, and F. Porzsolt. 2003. A likelihoodratio approach to meta-analysis of diagnostic studies. J Med Screen 10(1): 47–51.

Sutton, A. J., and J. P. T. Higgins. 2008. Recent Developments in Meta-Analysis. StatMed 27: 625–650.

Swets, J. A. 1988. Measuring the accuracy of diagnostic systems. Science 240: 1285–1293.

Whiting, P., R. Harbord, and J. Kleijnen. 2005. No role for quality scores in sys-tematic reviews of diagnostic accuracy studies. BMC Med Res Methodol 5: 19.http://dx.doi.org/10.1186/1471-2288-5-19.

Whiting, P., A. W. S. Rutjes, J. Dinnes, J. Reitsma, P. M. M. Bossuyt, and J. Kleijnen.2004. Development and validation of methods for assessing the quality of diagnosticaccuracy studies. Health Technol Assess 8(25): iii, 1–iii234.

Whiting, P., A. W. S. Rutjes, J. B. Reitsma, P. M. M. Bossuyt, and J. Kleijnen.2003. The development of QUADAS: a tool for the quality assessment of studies ofdiagnostic accuracy included in systematic reviews. BMC Med Res Methodol 3: 25.http://dx.doi.org/10.1186/1471-2288-3-25.

Whiting, P. F., M. E. Weswood, A. W. S. Rutjes, J. B. Reitsma, P. N. M. Bossuyt,and J. Kleijnen. 2006. Evaluation of QUADAS, a tool for the quality assessment ofdiagnostic accuracy studies. BMC Med Res Methodol 6: 9. http://dx.doi.org/8-6-9.

1 midas command - Boston Collegefm · 2 midas: meta-analysis of diagnostic accuracy studies 1 midas...

Documents

Transcript of 1 midas command - Boston Collegefm · 2 midas: meta-analysis of diagnostic accuracy studies 1 midas...