The GRACE Checklist for Rating the Quality of Observational ...
12
www.amcp.org Vol. 20, No. 3 March 2014 JMCP Journal of Managed Care & Specialty Pharmacy 301 The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution Nancy A. Dreyer, PhD, MPH; Priscilla Velentgas, PhD; Kimberly Westrich, MA; and Robert Dubois, MD ABSTRACT BACKGROUND: While there is growing demand for information about compar- ative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making. OBJECTIVE: To develop and test an item checklist that can be used to quali- fy those observational CE studies sufficiently rigorous in design and execu- tion to contribute meaningfully to the evidence base for decision support. METHODS: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external qual- ity ratings (N=88 articles). The articles compared treatment effective- ness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review–quality assessment from a published systematic review; (b) Single Expert Review–quality assessment made according to the solicited “expert opinion” of a senior researcher; and (c) Concordant Expert Review–quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare tes- ters’ assessments with those of experts. RESULTS: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better. CONCLUSIONS: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias. The current testing and validation efforts did not achieve clear discrimi- nation between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a spe- cific granular decision for evaluation, or not identifying a single study objec- tive in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question. RESEARCH • While there is growing demand for information about compara- tive effectiveness (CE), there is little understanding about when noninterventional studies are good enough for decision support. • Several expert reports have been issued listing criteria that are believed to be important in determining the quality of observa- tional CE studies, yet there have been no systematic, published evaluations of whether or how such criteria actually perform. What is already known about this subject • We developed the GRACE checklist, an objective 11-item check- list about the key attributes of high-quality noninterventional CE studies, a checklist that evaluates data and methods, but not motives, conflicts of interest, or interpretation. We then con- ducted several validation efforts using a large number of raters with diverse training and experience to determine how those individual elements performed when applied to expert opinions on quality. • This testing revealed that the most consistent predictors of quality relate to the validity of the primary outcomes for the study purpose. • Other relatively consistent predictors of quality were related to use of concurrent comparators, whether important covariates were recorded and accounted for, and whether sensitivity analy- ses were shown to support robustness of the conclusion. What this study adds Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on stud- ies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evalu- ate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity. J Manag Care Pharm. 2014;20(3):301-08 Copyright © 2014, Academy of Managed Care Pharmacy. All rights reserved.
Transcript of The GRACE Checklist for Rating the Quality of Observational ...
www.amcp.org Vol. 20, No. 3 March 2014 JMCP Journal of Managed Care & Specialty Pharmacy 301
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
Nancy A. Dreyer, PhD, MPH; Priscilla Velentgas, PhD; Kimberly Westrich, MA; and Robert Dubois, MD
ABSTRACT
BACKGROUND: While there is growing demand for information about compar-ative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making.
OBJECTIVE: To develop and test an item checklist that can be used to quali-fy those observational CE studies sufficiently rigorous in design and execu-tion to contribute meaningfully to the evidence base for decision support.
METHODS: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external qual-ity ratings (N=88 articles). The articles compared treatment effective-ness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review–quality assessment from a published systematic review; (b) Single Expert Review–quality assessment made according to the solicited “expert opinion” of a senior researcher; and (c) Concordant Expert Review–quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare tes-ters’ assessments with those of experts.
RESULTS: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better.
CONCLUSIONS: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias.
The current testing and validation efforts did not achieve clear discrimi-nation between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a spe-cific granular decision for evaluation, or not identifying a single study objec-tive in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question.
RESEARCH
•Whilethereisgrowingdemandforinformationaboutcompara-tiveeffectiveness (CE), there is littleunderstandingaboutwhennoninterventionalstudiesaregoodenoughfordecisionsupport.
•Several expert reportshavebeen issued listing criteria that arebelievedtobeimportantindeterminingthequalityofobserva-tionalCEstudies,yet therehavebeennosystematic,publishedevaluationsofwhetherorhowsuchcriteriaactuallyperform.
What is already known about this subject
•WedevelopedtheGRACEchecklist,anobjective11-itemcheck-list about the key attributes of high-quality noninterventionalCEstudies,achecklistthatevaluatesdataandmethods,butnotmotives, conflicts of interest, or interpretation. We then con-ducted several validation efforts using a largenumber of raterswith diverse training and experience to determine how thoseindividualelementsperformedwhenappliedtoexpertopinionsonquality.
•Thistestingrevealedthatthemostconsistentpredictorsofqualityrelatetothevalidityoftheprimaryoutcomesforthestudypurpose.
•Other relatively consistent predictors of qualitywere related touse of concurrent comparators, whether important covariateswererecordedandaccountedfor,andwhethersensitivityanaly-seswereshowntosupportrobustnessoftheconclusion.
What this study adds
Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on stud-ies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evalu-ate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity.
J Manag Care Pharm. 2014;20(3):301-08
Copyright © 2014, Academy of Managed Care Pharmacy. All rights reserved.
302 Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
ReseArch for Comparative Effectiveness (GRACE) checklist,whichhasbeentestedforitsclarityandabilitytodistinguishsufficientqualityworkaccordingtostudypurpose.
This article describes the development and approaches tovalidation of an item checklist that can be used to identifyobservational CE studies sufficiently rigorous in design andexecutionfordecisionsupport.Wefocusedonrelativelyobjec-tivecriteriathatcanbeassessedthroughreviewofpublishedstudyreports.
■■ MethodsWedraftedtheinitialchecklistfromtheGRACEprinciplesforobservationalCEstudies,developedincollaborationwiththeInternational Society of Pharmacoepidemiolgy.3 The check-list was fine-tuned for content validity by consultation withexpertsandextensiveliteraturereview,includingreportsfromthe Agency for Healthcare Research and Quality on ratingthestrengthofscientificresearchfindings,16-18theGradingofRecommendations Assessment, Development and Evaluationprocess,19,20reportingguidelines,andothertoolsforassessingclinical and observational study quality.21-26 Senior scientistsfromacademia,industry,andpayerswerealsoconsultedaboutitemselectionandscoring,someofwhomalsoservedasexpertraters.Userinstructionsandresponselevelsfortherefinedlistofquestionsweredevelopedbytheauthors.
Checklist testers were recruited via emails and personalrequestsandalsothroughthewebsitewww.graceprinciples.org.Volunteers (N=113) fromNorthandSouthAmerica,Europe,Asia, andAfrica conducted a total of 280 assessments of 88articles.Testers includedclinicians,academics,andrepresen-tativesfromindustry,healthdepartments,andothernonprofitagencies.They reportedawide rangeof trainingandexperi-encewithepidemiologicandstatisticalmethods.Theconstructvalidityofthechecklistwasassessedusingavariationonthe“Extreme Groups” approach27 by applying the checklist to 3“validationsets”ofobservationalCEresearchstudies.Wecom-paredchecklistitemresponses3differentwayswithexternalqualityratings,usingpublishedarticlesofobservationalCEorsafetystudies:(a)SystematicReview–qualityassessmentfromapublishedsystematicreview;(b)SingleExpertReview–qualityassessmentmadeaccordingtothesolicited“expertopinion”ofaseniorresearcher;and(c)ConcordantExpertReview–qualityassessmentsfrom2expertsforwhichtherewasconcordance.The firstversionof thechecklistwasused for theSystematicReviewvalidation test. Itwas then fine tuned for subsequenttestingintheSingleExpertandConcordantExpertReviews.
Inthe first test,asampleofarticleswasdrawnfrompub-lishedsystematicreviewsthatlistedthearticlesconsideredforinclusion,alongwiththeirqualityassessments(articles listedinAppendix,availableinonlinearticle).28-33Articleswerecon-sidered“good”iftheymetqualitycriteriarequiredforinclusion
Developing a sustainable health system requires healthcarethatisguidedbyreliableinformationaboutwhichmedical diagnostics and treatments work best, for
whom, and inwhat situations.1Tomeet thediverseneedsofclinicians,policymakers,andthosewhodecideaboutformu-laries, the full range of comparative effectiveness (CE) stud-ies—randomizedcontrolledtrials,observationalresearch(alsoreferredtoasnoninterventionalresearchsincetreatmentsarenot assigned by protocol), and meta-analyses—are needed.Observationalstudiesareparticularlyusefulbecausetheyoftenprovide information about diverse populations, practitioners,andsettingsinatimelyandcost-effectivemanner.
Recent calls for using the full range of high-quality CEresearch to inform decisions about medical diagnostics andinterventions have brought forth a spate of consensus offer-ingsaboutrecognizingqualityinobservationalCEstudiesandmeta-analysis.2-15 These papers have face validity and largelyappearreasonable,butthereislittle,ifany,evidencethatanyoftheserecommendationscanactuallydistinguishstudiesofsuf-ficientqualitytomeritseriousevaluationforaparticularclini-calorpaymentdecision.Forexample,someguidelinesaddresspotential conflicts of interest by calling for full disclosure, astandardjournalpracticethatreliesonindividualassessmentofpotentialconflicts.Someinsistthat,likeclinicaltrials,onlyhypotheses that were specified in advance of collecting anydatahavevalidity.Othersomitthecriterionaboutprespecifiedhypotheses,insteadgivingmoreweighttothevalueofdescrip-tivedataforfillinggapsandshapingsubsequentresearch.Onevery practical, high-level description of good practice, pub-lishedinthisjournalbyWillkeandMullinsin2011,9focusedongoodresearchpracticesfortheconductandreportingofCEresearch using real-world data with nonrandom assignmentoftreatments.Theyoffer“TenCommandmentsforimprovingthe systematic use of principles that are aimed at achievingthe goals of developing credible and germane CE researchstudiesusingrealworlddata.”9Wesupportthatgoalandhaveattempted to further it with the development of the Good
•On the whole, GRACE checklist items performed better thanopinions from individual experts and concurrent expert opin-ions.Nonetheless,thechecklistwouldbenefitfromfurthervali-dationefforts,includingdirectingreviewerstoaddressaspecificobjective for each evaluation, finding additional validation testsets toevaluate therobustnessof thechecklist,andconductingmoremultivariateanalyses todeterminewhetheranycombina-tions or sequences of responses can improve the ability of thechecklisttodiscriminatestudiesofreasonablystrongqualityforthepurposeathand.
What this study adds (continued)
www.amcp.org Vol. 20, No. 3 March 2014 JMCP Journal of Managed Care & Specialty Pharmacy 303
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
inthesystematicreviewandwereconsideredtobeofinsuffi-cientqualityiftheywereexcludedfromthereview.Fortesting,authorshipwasblindedby redaction to avoidbiasingqualitydeterminations,andtesterswereaskednottotrytoidentifytheauthorsthroughothermeans.Of48articles,21wereconsid-ered“good”and27“notgoodenough”;172completedassess-mentswerereceivedfrom58testers,witheacharticlereceivinganaverageof4reviews(range,1-9reviews).
In the second set of tests (Expert Reviews), the expertsreceiveddirectionsexplicittotheuseofthearticlesfor“deci-sionsupport”andwereaskedtodecidewhethereachobserva-tionalCEstudywasofsufficientqualitytosupportaformularydecision.Tensenioracademicandindustryexpertswereaskedtorate4ormorepublishedobservationalCEarticlesaseither“sufficientqualitytobeusedtosupportaformularydecision”or“sufficientlyflawedtomakeinterpretationunreliable.”TheSingleExpertReviewconsistedof40articles:23thatexpertsratedassufficientand17thatwereratedas tooflawedtobeusefulforthispurpose.
For the third set of tests (Concordant Expert Reviews), 5expertsreviewed23of the40articles toassessconcordance.ThearticlesusedfortestingarelistedinAppendixB(availableinonlinearticle),andthe14expertsarelistedintheacknowl-edgements (10participated in theSingleExpertReview;1ofthose 10 plus 4 others reviewed articles in the ConcordantExpertReview).Fifty-five additional volunteer testers appliedthechecklistto2articleseachinthisvalidation,completingatotalof108assessments,with eacharticle receiving anaver-ageof2.7reviews(range,2-7reviews).Oneitemwasdroppedafterthefirstroundoftestingwhenwelearnedthatnoneofthearticlesreviewedstatedwhetherthehypotheseshadbeenspec-ifiedbeforethestudybegan.Checklistitemswerealsorevisedbeforesubsequenttestingtoimproveclarity.Inaddition,userinstructionswere clarified after review by 2 authors (DreyerandVelentgas)toaccommodatebetterevaluationofstudiesofmedicaldevicesandproceduresaswellasdrugs.
Question response levels in the checklistweremapped todichotomized categories of “sufficient (good enough for deci-sionsupport)”or “insufficient.”Responses that indicated“notenough information in article” were treated as “insufficient,”since this lack of information could be viewed as a negativeaspectofstudyquality.Responsesof“notapplicable”wereclas-sifiedas“sufficient”sothatanarticlewouldnotberatednega-tivelyifaspecificquestionitemwasnotrelevanttoitsobjective.Blank responseswere treated asmissing values. Positive andnegative predictive values were estimated for each checklistitemtodescribehowwellareviewer’sassessments,usingthechecklist,comparedwithanexpert’sassessmentofstudyqual-ity (in thiscase, thebestavailable “goldstandard” forassess-mentofstudyquality).Foreacharticle,asinglereviewfromatester,randomlyselectedfromthemultiplereviewsperarticle,wascomparedwiththe“goldstandard.”Thiscomparisonwas
donetwicetoensurethatresultswerenothighlydependentontherandomsubsetselected.Resultsfrombothanalysissubsetsare presented in the Results section. All analyses presentedwereconductedusingSAS9.2(Cary,NC).
■■ ResultsTheGRACE checklist, asmodified through this testing pro-cess, is shown in Table 1.Questions are grouped into thoserelatingtodataandmethods,andtheguidetoscoringreflectsclarificationsandrevisionsbasedonfeedbackfromratersandjournalreviewers.Table2presentspredictivevalues,compar-ing testers’ assessments of checklist items to experts’ overallqualityassessments.Thiscomparisonwasdone for2 samplereviews for eachof the3validations (6 samples total), strati-fiedbypositivepredictivevalue(PPV)andnegativepredictivevalue(NPV).
Taken as a whole, the checklist showed better NPV thanPPV, with 31 individual items scoring at least 0.67 for NPVversusonly20itemsforPPV.Asimilartrendwasevidentwhenlookingatbothdataandmethodsquestions;20versus11dataitemsscored≥0.67,and11versus9methodsitemsNPVandPPV,respectively.Eachofthe11itemsshowedsomepotentialforNPV(usingthe≥0.67criterion),and9ofthe11questionsalsoshowedsomepotentialfortheirPPV.ThesinglequestionthatmostconsistentlyshowedstrongNPVandPPVaddressedthe validityof theprimaryoutcomes (D4,Table1).ForPPV,theotherquestionthatmostconsistentlyscoredrelativelyhighwaswhether a sensitivity analysis had been conducted (M5,Table1).The2mostfrequentlyidentifiablepredictorsofnega-tivequalityweretheabsenceofaconcurrentcomparatorgroup(M2,Table1)andthelackofadequatedetailsonoutcomes(D2,Table1), followedbynotusingappropriateclinicaloutcomeswhereapplicable(D3andD4,Table1).
■■ DiscussionThe GRACE checklist was designed as an initial evaluationtooltobroadlyscreenthequalityofobservationalCEstudiestoselectthoseworthin-depthconsideration.Wefocusedon11checklistelements,6relating todataand5relating tometh-ods.Usinganarbitrarilyselectedcut-pointof0.67toindicaterelatively strong predictive value, checklist questions aboutdata generally showed better predictive value than questionsaboutmethods.Twoofthemostconsistentpredictorsofqual-ity appropriate forpurpose related to (1)validoutcomesand(2)useofconcurrentcomparators,bothfactorswithimportantdesign, analytic, andbudgetary ramifications.Our small testof concordance among expert reviewers revealed an unset-tlinglackofagreementaboutwhat“good”lookslikethroughconsensus.Therewasagreementonqualityonlyfor12of23articles(52%)ratedby2experts—hardlyanendorsementforpurerelianceonexpertassessments.
304 Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
Components Scoring as Fit for Purpose: Sufficient (+), Insufficient (-)
DataD1.Weretreatmentand/orimportantdetailsoftreatmentexposureade-quatelyrecordedforthestudypurposeinthedatasources?Note:notalldetailsoftreatmentarerequiredforallresearchquestions.
(+)Yes,reasonablynecessaryinformationtodeterminetreatmentorinterven-tionwasadequatelyrecordedforstudypurposes(e.g.,fordrugs,sufficientdetailondose,dayssupplied,routeorotherimportantdata;forvaccines,batch,dose,route,andsiteofadministration,etc.;fordevices,typeofdevice,placement,surgicalprocedureused,serialnumber,etc.)
(-)No,datasourceclearlydeficientornotenoughinformationinarticle.D2.Weretheprimaryoutcomesadequatelyrecordedforthestudypurpose (e.g.,availableinsufficientdetailthroughdatasources)?
(+)Yes,informationtoascertainoutcomeswasadequatelyrecordedinthedatasources(e.g.,ifclinicaloutcomeswereascertainedusingICD-9-CMdiagnosiscodesinanadministrativedatabase,thelevelofsensitivityandspecificitycapturedbythecodeswassufficientforassessingtheoutcomeofinterest).
(-)No,datasourceclearlydeficient(e.g.,thecodescapturedarangeofcondi-tionsthatwastoobroadornarrow,andsupplementaryinformationsuchasthatfrommedicalchartswasnotavailable,ornotenoughinformationinarticle).
D3.Wastheprimaryclinicaloutcomemeasuredobjectivelyratherthansub-jecttoclinicaljudgment(e.g.,opinionaboutwhetherthepatient’sconditionhasimproved)?
(+)Yes,clinicaloutcomewasmeasuredobjectively(e.g.,hospitalization,mortality).
(+)Notapplicable(primaryoutcomenotclinical,suchasPROs).
(-)No(e.g.,clinicalopinionaboutwhetherpatient’sconditionimproved,or notenoughinformationinarticle).
D4.Wereprimaryoutcomesvalidated,adjudicated,orotherwiseknowntobevalidinasimilarpopulation?
(+)Yes,outcomeswerevalidated,adjudicated,orbasedonmedicalchartabstractionswithcleardefinitions(e.g.,avalidatedinstrumentwasusedtoassessPROs[suchasSF-12HealthSurvey];aclinicaldiagnosisviaICD-9-CMcodewasused,withformalmedicalrecordadjudicationbycommitteetoconfirmdiagnosisorotherprocedurestoachievereasonablesensitivityandspecificity;billingdatawereusedtoassesshealthresourceutilization,etc).
(-)No,ornotenoughinformationinarticle.D5.Wastheprimaryoutcomemeasuredoridentifiedinanequivalentman-nerbetweenthetreatment/interventiongroupandthecomparisongroups?
(+)Yes.
(-)No,ornotenoughinformationinarticle.D6.Wereimportantcovariatesthatmaybeknownconfoundersoreffectmodifiersavailableandrecorded?Importantcovariatesdependonthetreat-mentand/oroutcomeofinterest(e.g.,bodymassindexshouldbeavailableandrecordedforstudiesofdiabetes;raceshouldbeavailableandrecordedforstudiesofhypertensionandglaucoma).
(+)Yes,mostifnotallimportantknownconfoundersandeffectmodifiersavailableandrecorded(e.g.,measuresofmedicationdoseandduration).
(-)No,atleast1probableknownconfounderoreffectmodifiernotavailableandrecorded(asnotedbyauthorsorasdeterminedbyuser’sclinicalknowl-edge),ornotenoughinformationinarticle.
MethodsM1.Wasthestudy(oranalysis)populationrestrictedtonewinitiatorsoftreatmentorthosestartinganewcourseoftreatment?Effortstoincludeonlynewinitiatorsmayincluderestrictingthecohorttothosewhohadawashoutperiod(specifiedperiodofmedicationnonuse)priortothebeginningofstudyfollow-up.
(+)Yes,onlynewinitiatorsofthetreatmentofinterestwereincludedinthecohort,orforsurgicalproceduresanddevices,includingonlypatientswhoneverhadthetreatmentbeforethestartofstudyfollow-up.
(-)No,ornotenoughinformationinarticle.
M2.If1ormorecomparisongroupswereused,weretheyconcurrentcom-parators?Ifnot,didtheauthorsjustifytheuseofhistoricalcomparisongroups?
(+)Yes,datawerecollectedduringthesametimeperiodasthetreatmentgroup(“concurrent”),orhistoricalcomparatorswereusedwithreasonablejustification(e.g.,whenitwasimpossibleforresearcherstoidentifycurrentusersofoldertreatmentsorwhenaconcurrentcomparisongroupwasnotvalid,aswhenuptakeofnewproductissorapidthatconcurrentcompara-torsdiffergreatlyonfactorsrelatedtotheoutcome).
(-)No,historicalcomparatorsusedwithoutbeingscientificallyjustifiable,or notenoughinformationinarticle.
M3.Wereimportantconfoundingandeffectmodifyingvariablestakenintoaccountinthedesignand/oranalysis?Appropriatemethodstotakethesevariablesintoaccountmayincluderestriction,stratification,interactionterms,multivariateanalysis,propensityscorematching,instrumentalvari-ables,orotherapproaches.
(+)Yes,mostifnotallimportantcovariatesthatwouldbelikelytochangetheeffectestimatesubstantiallywereaccountedfor(e.g.,measuresofmedi-cationdoseandduration).
(-)No,someimportantcovariateswereavailableforanalysisbutnotana-lyzedappropriately,oratleast1importantcovariatewasnotmeasured,or notenoughinformationinarticle.
M4.Istheclassificationofexposedandunexposedperson-timefreeof“immortaltimebias”?(Immortaltimeinepidemiologyreferstoaperiodofcohortfollow-uptimeduringwhichdeath,oranoutcomethatdeterminesendoffollow-up,cannotoccur.)
(+)Yes.
(-)No,ornotenoughinformationinthearticle.
M5.Wereanymeaningfulanalysesconductedtotestkeyassumptionsonwhichprimaryresultsarebased?(E.g.,weresomeanalysesreportedtoevaluatethepotentialforabiasedassessmentofexposureoroutcome,suchasanalyseswheretheimpactofvaryingexposureand/oroutcomedefinitionswastestedtoexaminetheimpactonresults?)
(+)Yes,andprimaryresultsdidnotsubstantiallychange.
(-)Yes,andprimaryresultschangedsubstantially.
(-)Nonereported,ornotenoughinformationinarticle.
ICD-9-CM = International Classification of Diseases, Ninth Revision, Clinical Modifications; PRO = patient-reported outcome.
TABLE 1 GRACE Checklist: Components and Response Guide
www.amcp.org Vol. 20, No. 3 March 2014 JMCP Journal of Managed Care & Specialty Pharmacy 305
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
Limitations Although the current testing and validation efforts did notachieve clear discrimination between studies fit for purposeandthosenot,weidentifiedacriticalbutremediablelimitation
inourapproach.Bynotspecifyingaspecificgranulardecisionforevaluation(e.g.,“Isthisstudyofsufficientqualitytocom-paretherelativesafetyof twodrugs?”)or identifyingasinglestudyobjectiveinsituationswherereportsincludedmorethan
Ade
quat
e T
reat
men
t
Ade
quat
e O
utco
mes
Obj
ecti
ve
Out
com
es
Val
id
Out
com
es
Sim
ilar
O
utco
mes
Cov
aria
tes
Rec
orde
d
New
In
itia
tors
Con
curr
ent
Com
para
tors
Cov
aria
tes
Acc
ount
ed
For
Imm
orta
l T
ime
Bia
s
Sen
siti
vity
A
nal
ysis
D1 D2 D3 D4 D5 D6 M1 M2 M3 M4 M5
Positive Predictive ValuesSystematicReview1PPV 0.40 0.49 0.49 0.59 0.47 0.60 0.38 0.46 0.69 —a 0.75N/D 10/25 20/41 21/43 16/27 20/43 12/20 6/16 18/39 11/16 —a 9/12
SystematicReview2PPV 0.48 0.48 0.51 0.52 0.48 0.75 0.33 0.48 0.91 —a 0.70N/D 15/31 19/40 20/39 17/33 20/42 9/12 7/21 20/42 10/11 —a 7/10
SingleReview1PPV 0.57 0.63 0.59 0.70 0.58 0.73 0.56 0.59 0.65 0.59 0.56N/D 16/28 19/40 20/34 16/23 22/38 11/15 14/25 23/39 13/20 19/32 9/16
SingleReview2PPV 0.68 0.62 0.61 0.74 0.58 0.64 0.65 0.58 0.68 0.61 0.69N/D 19/28 16/26 20/33 14/19 19/33 9/14 13/20 22/38 13/19 19/31 11/16
ConcordantReview1PPV 0.67 0.71 0.67 0.83 0.44 1.00 0.44 0.67 0.50 0.63 1.00N/D 4/6 5/7 6/9 5/6 4/9 1/1 4/9 6/9 2/4 5/8 2/2
ConcordantReview2PPV 0.63 0.60 0.55 0.67 0.55 0.50 0.67 0.55 0.50 0.56 0.50N/D 5/8 6/10 6/11 4/6 6/11 2/4 6/9 6/11 2/4 5/9 1/2Numberrated≥0.67 2 1 1 4 0 3 1 1 3 0 4
Negative Predictive ValuesSystematicReview1NPV 0.55 0.86 1.00 0.76 0.80 0.68 0.53 0.71 0.71 —a 0.67N/D 12/22 6/7 5/5 16/21 4/5 19/28 17/32 5/7 22/31 —a 24/36
SystematicReview2NPV 0.65 0.75 0.88 0.71 0.83 0.66 0.50 1.00 0.69 —a 0.62N/D 11/17 6/8 7/8 10/14 5/6 23/35 13/26 4/4 25/36 —a 23/37
SingleReview1NPV 0.42 0.67 0.50 0.59 0.50 0.52 0.40 1.00 0.50 0.50 0.39N/D 5/12 4/6 3/6 10/17 1/2 13/25 6/15 1/1 10/20 4/8 9/23
SingleReview2NPV 0.67 0.50 0.57 0.57 0.43 0.44 0.50 0.50 0.55 0.56 0.48N/D 8/12 6/12 4/7 12/21 3/7 11/25 10/20 1/2 11/20 5/9 11/23
ConcordantReview1NPV 0.67 0.67 1.00 1.00 0.33 0.55 0.33 1.00 0.50 0.75 0.56N/D 2/4 2/3 2/2 2/2 1/3 6/11 1/3 3/3 4/8 3/4 5/9
ConcordantReview2NPV 0.75 1.00 1.00 0.67 1.00 0.50 1.00 1.00 0.50 0.67 0.50N/D 3/4 2/2 1/1 4/6 1/1 4/8 3/3 1/1 4/8 2/3 5/10Numberrated≥0.67 3 5 4 4 3 1 1 5 2 2 1
aQuestion not included in Systematic Review.D = denominator (total number of articles rated on quality by raters); N = numerator (number of articles in which raters and experts agreed on quality); NPV = negative predictive value; PPV = positive predictive value.
TABLE 2 Predictive Values by Item for All Validation Test Sets
306 Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
employed consensus methods, which have face validity, butwithoutevaluationofreliabilityanddiscriminantvalidity,itisuncertainhowtheywouldperforminasimilarexercise.
■■ ConclusionsTakenasawhole,theGRACEchecklistcanhelpasascreeningtooltoeliminatestudiesthatdonotmeetthebaselinequalityrequirementsforobservationalstudiesofcomparativeeffective-ness.Werecommendthat theGRACEchecklistbeusedasa“firstpass”toevaluatehowagivenstudymeasuresagainsteachof the checklist itemswhenapplied to a specific studyques-tion.Thosestudiesthatappeartobefairlysoundindesignandmethodsinthecontextofthestudypurposeshouldbeexam-inedmore closely to evaluate the comparability of the studypopulationtothetargetpopulationofinterest,theappropriate-nessofthespecificmedicalinterventionsandcomparatorsforuseinthetargetpopulation,andthelikelihoodofintractablebiasandrelevanceoftheoutcomestopatientsandhealthcareproviders. Studies should also receive further review in thecontextofavailableevidenceregardingrelativerisksandben-efitsandtherequiredthresholdfordecisionsupport,ideallybythosewithmethodologicalandcontentareaknowledge.
Despite thedrawbacks in theGRACE checklist andothertools, having an agreed upon set of assessment elements,checklists,or score cards is critical for thematurationof thefield. Substantial resources will be expended on studies ofreal-worldeffectiveness,andiftherigoroftheseobservationalassessments cannot be ascertained, then the impact of thosestudies will be suboptimal. Similarly, agreement on key ele-ments of quality will ensure that budgets are appropriatelydirectedtowardthosekeyelementsofquality.Giventhecen-tralityofthistaskandthelessonslearnedfromtheseextensiveeffortsatvalidationanduser testing,weareoptimisticaboutthe potential for improved assessments.We believe that thenecessary tools can be produced, enabling diverse types ofassessments by people with a wide range of experience andtraining.
1objective(e.g.,“Doesthisstudydemonstrategreatercompli-ancewithonceperweekvs.daily therapy?”), reviewerswereleftwithtoobroadanassessment.Webelievethatfutureeffortswill bemore successful if reviewers are asked to focus on aspecificobjectiveorquestion.
TheGRACEchecklistalsodoesnotprovideasinglequan-titativesummaryscoreor“pass/fail”result.Ourexpertscoun-seled thata summaryresult fromthechecklistwouldnotbebroadlyreflectiveofthenumerousconsiderationsthatgointoassessing the quality of a given study andwhether it is suf-ficient for a specific purpose. Related efforts have concludedthatapass/fail scorewouldrequiremuchmore tailoringofachecklist to address specific issues andcontexts, suchas thetypesofdecisionsfacedbypharmacy,payer,andotherhealthcareconstituenciesandspecifictherapeuticareas.Nonetheless,weconductedsomepreliminaryanalysesusingCARTsoftware(Salford Systems, San Diego, CA) to create regression trees.Unfortunately, no consistently high-performing combinationofchecklist itemswas identified thatwouldcorrectlyclassifystudiesasgoodorofinsufficientquality.Sincethen,thecheck-listinstructionsandscoringhavebeenimprovedthroughtest-ing,andadditionalanalysesmaybemorefruitful.Inaddition,by addressing the limitation discussed above and specifyingthepurposeof thereview,anoverallquantitativeassessmentmaybefeasible.
In addition, theGRACEchecklistwouldbenefit from fur-ther development using different validation sets, improvinginstructions to raters, and furtheranalysisof results to see ifanycombinationorsequenceofitemresponseshasparticularlyhigh predictive validity. The articles we selected from sys-tematic reviews, for example, reflected publications that hadbeenexamined thoroughlyandvettedbyagroupofexperts.However, not all of these articles reflected use of modernmethods,particularlyas they relate todesignandanalysisofnoninterventionalCEstudies,becausebythetimeasystematicreview had been conducted and published, the articles usedweredated.Findingwell-acceptedstandardsagainstwhichtotestchecklistitemstofurtherrefinethedistinguishingaspectsofqualityremainsanopenquestion.34-36
When considering theGRACEchecklist’s limitations, it isimportant to keep in mind what alternative tools exist andtheirutilityforthispurpose.Thewell-recognizedSTROBEandCONSORTguidelinesaddresshowtoreportstudyresultsandwerenotdesignedtoassessstudyquality;therefore,theywouldnotbesufficientsubstitutes.37Toolsnotdevelopedspecificallyforpharmacoepidemiologyareunlikelytoincludetherelevantelementscriticalfordescription,assessmentofCE,andlikeli-hood of bias.38 Perhapsmost importantly, to our knowledge,noneoftheotherassessmentguidelinesorstandardshavebeensubjectedtomuch,ifany,testing.Thedevelopersofthosetools
NANCY A. DREYER, PhD, MPH, is Global Chief of Scientific Affairs and Senior Vice President, and PRISCILLA VELENTGAS, PhD, is Senior Director, Epidemiology, Quintiles Real-World & Late Phase Research, Cambridge, Massachusetts. KIMBERLY WESTRICH, MA, is Director, Health Services Research, and ROBERT DUBOIS, MD, is Chief Science Officer, National Pharmaceutical Council, Washington, DC.
AUTHOR CORRESPONDENCE: Nancy A. Dreyer, PhD, MPH, Senior Vice President, Quintiles Real-World & Late Phase Research, 201 Broadway, Cambridge, MA 02139. Tel.: 617.715.6810; Fax: 617.621.1620; E-mail: [email protected].
Authors
www.amcp.org Vol. 20, No. 3 March 2014 JMCP Journal of Managed Care & Specialty Pharmacy 307
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
9.WillkeRJ,MullinsCD.“TenCommandments”forconductingcomparativeeffectivenessresearchusing“real-worlddata.”J Manag Care Pharm. 2011;17 (9SupplA):S10-S15.Availableat:http://amcp.org/WorkArea/DownloadAsset.aspx?id=13714.
10.CoxE,MartinBC,VanStaaT,GarbeE,SiebertU,JohnsonML.Goodresearchpracticesforcomparativeeffectivenessresearch:approachestomitigatebiasandconfoundinginthedesignofnonrandomizedstudiesoftreatmenteffectsusingsecondarydatasources:theInternationalSocietyforPharmacoeconomicsandOutcomesResearchGoodResearchPracticesforRetrospectiveDatabaseAnalysisTaskForceReport—PartII.Value Health. 2009;12(8):1053-61.
11.JohnsonML,CrownW,MartinBC,DormuthCR,SiebertU.Goodresearchpracticesforcomparativeeffectivenessresearch:analyticmethodstoimprovecausalinferencefromnonrandomizedstudiesoftreatmenteffectsusingsecondarydatasources:TheISPORGoodResearchPracticesforRetrospectiveDatabaseAnalysisTaskForceReport—PartIII. Value Health. 2009;12(8):1062-73.
12.BergerML,MamdaniM,AtkinsD,JohnsonML.Goodresearchpracticesforcomparativeeffectivenessresearch:defining,reportingandinterpretingnonrandomizedstudiesoftreatmenteffectsusingsecondarydatasources:theISPORGoodResearchPracticesforRetrospectiveDatabaseAnalysisTaskForceReport—PartI.Value Health.2009;12(8):1044-52.
13.BergerML,DreyerN,AndersonF,TowseA,SedrakyanA,NormandSL.Prospectiveobservationalstudiestoassesscomparativeeffectiveness: theISPORGoodResearchPracticesTaskForceReport. Value Health. 2012;15(2):217-30.
14.AppendixD:researchquestionsandPICO(TS):theeffectivehealthcareprogramstakeholderguide.July2011.AgencyforHealthcareResearchandQuality.Rockville,MD.Availableat:http://www.ahrq.gov/research/find-ings/evidence-based-reports/stakeholderguide/appendixd.html.AccessedNovember14,2013.
15.Patient-CenteredOutcomesResearchInstitute.PCORImethodologystandards.December14,2012.Availableat:http://www.pcori.org/assets/PCORI-Methodology-Standards.pdf.AccessedNovember14,2013.
16.AgencyforHealthcareResearchandQuality.Ratingthestrengthofscientificresearchfindings.AHRQPublicationNo.02-P022.April2002.Availableat:http://archive.ahrq.gov/clinic/epcsums/strenfact.htm.AccessedNovember14,2013.
17.OwensDK,LohrKN,AtkinsD,etal.AHRQseriespaper5:gradingthestrengthofabodyofevidencewhencomparingmedicalinterventions—AgencyforHealthcareResearchandQualityandtheEffectiveHealth-CareProgram.J Clin Epidemiol.2010;63(5):513-23.
18.WestS,KingV,CareyTS,etal.Systemstoratethestrengthofscientificevidence.EvidenceReport/TechnologyAssessmentNo.47(PreparedbytheResearchTriangleInstitute-UniversityofNorthCarolinaEvidence-basedPracticeCenterunderContractNo.290-97-0011).AHRQPublicationNo.02-E016.April2002.Rockville,MD:AgencyforHealthcareResearchandQuality.April2002.Availableat:http://www.thecre.com/pdf/ahrq-system-strength.pdf.AccessedJanuary8,2014.
19.GuyattGH,OxmanAD,VistGE,etal.GRADE:anemergingconsen-susonratingqualityofevidenceandstrengthofrecommendations.BMJ. 2008;336(7650):924-26.Availableat:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2335261/.AccessedNovember14,2013.
20.SharmaT,NyongJ,ShawE.S22—UsingandadaptingGRADEmeth-odologyinanareaoflow-qualityevidence:anexamplefromanationalguidelineonablativetherapiesforthetreatmentofBarrett’sesophagus.Otolaryngol Head Neck Surg.2010;143(1Suppl1):S21.
21.StroupDF,BerlinJA,MortonSC,etal.Meta-analysisofobserva-tionalstudiesinepidemiology:aproposalforreporting.Meta-analysisOfObservationalStudiesinEpidemiology(MOOSE)group.JAMA. 2000;283(15):2008-12.
DISCLOSURES
This research was supported in part by a contract from the NationalPharmaceutical Council. The sponsor has provided scientific collaborationandhas rights tononbinding reviewofmanuscriptsbutdoesnothave theright to choose authors ormanuscript topics, nor does it have the right tofinalapprovalofthewordingofanymanuscripts.Dreyerhadfullaccesstoallofthedatainthestudyandtakesresponsibilityfortheintegrityofthedataandtheaccuracyofthedataanalysis.WestrichandDuboisareemployeesoftheNationalPharmaceuticalCouncil,whichisfundedbythepharmaceuticalindustry.Thisworkwas supported inpartbya contract from theNationalPharmaceuticalCouncil.
DreyerandVelentgaswerejointlyresponsibleforthedesignandconductofthestudy,datacollection,management,analysis,andinterpretationofdata,aswellasdraftingandreviewingthemanuscript.WestrichandDuboispro-videdconsultationateverystageoftheprojectandcontributedtothedesignofthestudy,aswellasdraftingandreviewingofthemanuscript.
ACKNOWLEDGMENTS
Expert raters included Mark Berger (Pfizer), Nick Black (London Schoolof Hygiene and Tropical Medicines), Alan Brookhart (University of NorthCarolina), Sarah Garner and Tarang Sharma (NICE, UK), Tobias Gerhard(Rutgers),KathyLohrandNancyBerkman(RTI),NewellMcElwee(Merck),PeterNeumann(TuftsUniversity),StevePearsonandDanOllendorf(ICER),Brian Sweet (AstraZeneca), and NoelWeiss (University ofWashington). Afulllistofparticipatingexpertconsultantsandraterscanbefoundatwww.graceprinciples.org.
TheauthorsalsowishtoacknowledgethesupportofAprilDuddy,MSc,whocontributedtodatacollectionandanalysis,andJaclynBosco,PhD,andAllisonBryant,MPH,whocontributedtoanalysis.
REFERENCES
1.GreeneJA,PodolskySH.Reform,regulation,andpharmaceuticals—theKefauver-HarrisAmendmentsat50.N Engl J Med.2012;367(16):1481-83. Availableat:http://www.nejm.org/doi/full/10.1056/NEJMp1210007.AccessedNovember14,2013.
2.DreyerNA.TunisSR,BergerM,OllendorfD,MattoxP,GliklichR.Whyobservationalstudiesshouldbeamongthetoolsusedincomparativeeffec-tivenessresearch.Health Aff (Millwood).2010;29(10):1818-25.
3.DreyerNA,SchneeweissS,McNeilB,etal.GRACEprinciples:recogniz-inghigh-qualityobservationalstudiesofcomparativeeffectiveness.Am J Manag Care.2010;16(6):467-71.
4.WellsGA,SheaB,O’ConnellD,etal.TheNewcastle-OttawaScale(NOS)forassessingthequalityofnonrandomisedstudiesinmeta-analyses.Availableat:http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.AccessedNovember14,2013.
5.AgencyforHealthcareResearchandQuality.Assessingconfounding,theriskofbiasandprecisionofobservationalstudiesofinterventionsorexpo-sures:furtherdevelopmentoftheRTIItemBank.Evidence-BasedPracticeCenter.DraftMethodsResearchReport.May2012.Availableat:http://www.effectivehealthcare.ahrq.gov/ehc/products/414/1272/Observational-studies-refinement_DraftReport_20120927.pdf.AccessedJanuary8,2014.
6.ThomasL,PetersonED.Thevalueofstatisticalanalysisplansinobser-vationalresearch:defininghigh-qualityresearchfromthestart.JAMA. 2012;308(8):773-74.
7.LuceBR,DrummondMF,DuboisRW,etal.Principlesforplanningandcon-ductingcomparativeeffectivenessresearch. J Comp Eff Res.2012;1(5):431-40.
8.VelentgasP,DreyerNA,NourjahP,SmithSR,TorchiaMM,eds.Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide.AHRQPublicationNo.12(13)-EHC099.Rockville,MD:AgencyforHealthcareResearchandQuality;January2013.
308 Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
22.WongWC,CheungCS,HartGJ.Developmentofaqualityassessmenttoolforsystematicreviewsofobservationalstudies(QATSO)ofHIVpreva-lenceinmenhavingsexwithmenandassociatedriskbehaviours.Emerg Themes Epidemiol.2008;17(5):23.Availableat:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2603000/.AccessedNovember14,2013.
23.TsengTY,BreauRH,FespermanSF,ViewegJ,DahmP.Evaluatingtheevidence:themethodologicalandreportingqualityofcomparativeobser-vationalstudiesofsurgicalinterventionsinurologicalpublications.BJU Int. 2009;103(8):1026-31.
24.VonElmE,AltmanDG,EggerM,etal.StrengtheningtheReportingofObservationalStudiesinEpidemiology(STROBE)statement:guidelinesforreportingobservationalstudies.J Clin Epidemiol.2008;61(4):344-49.
25.DuboisRW,KindermannSL.Demystifyingcomparativeeffectivenessresearch:acasestudylearningguide.NationalPharmaceuticalCouncil.November2009.Availableat:http://www.npcnow.org/publication/demys-tifying-comparative-effectiveness-research-case-study-learning-guide.AccessedNovember14,2013.
26.JadadAR,MooreRA,CarrollD,etal.Assessingthequalityofreportsofrandomizedclinicaltrials:isblindingnecessary?Control Clin Trials. 1996;17(1):1-12.
27.StreinerDL,NormanGR.Health Measurement Scales: A Practical Guide to Their Development and Use.4thed.NewYork:OxfordUniversityPress;2008.
28.McDonaghMS,PetersonK,CarsonS,FuR,ThakurtaS.Drugclassreview:atypicalantipsychoticdrugs.FinalUpdate3Report[Internet].Portland,OR:OregonHealthandScienceUniversity;July2010.
29.HumphreyLL,ChanBK,SoxHC.Postmenopausalhormonereplacementtherapyandtheprimarypreventionofcardiovasculardisease. Ann Intern Med.2002;137(4):273-84.
30.NelsonHD,HumphreyLL,NygrenP,TeutschSM,AllanJD.Postmenopausalhormonereplacementtherapy:scientificreview.JAMA. 2002;288(7):872-81.
31.IpS,BonisP,TatsioniA,etal.Comparativeeffectivenessofmanagementstrategiesforgastroesophagealrefluxdisease.ComparativeEffectivenessReviewNo.1.(PreparedbyTufts-NewEnglandMedicalCenterEvidence-basedPracticeCenterunderContractNo.290-02-0022).Rockville,MD:AgencyforHealthcareResearchandQuality.AHRQPublicationNo.06-EHC003-1December2005.Availableat:http://effectivehealthcare.ahrq.gov/ehc/products/1/43/GERD%20Final%20Report.pdf.AccessedNovember14, 2013.
32.YankV,TuohyCV,LoganAC,etal.Comparativeeffectivenessofin-hospitaluseofrecombinantfactorVIIaforoff-labelindicationsvs.usualcare.ComparativeEffectivenessReviewNo.21.(PreparedbyStanford-UCSFEvidence-basedPracticeCenterunderContractNo.#290-02-0017).Rockville,MD:AgencyforHealthcareResearchandQuality.AHRQPublicationNo.10-EHC030-EFMay2010.Availableat:http://www.effec-tivehealthcare.ahrq.gov/ehc/products/20/450/Final%20Report_CER21_Factor7.pdf.AccessedNovember14,2013.
33.DonahueKE,GartlehnerG,JonasDE,etal.Comparativeeffectivenessofdrugtherapyforrheumatoidarthritisandpsoriaticarthritisinadults.ComparativeEffectivenessReviewNo.11.(PreparedbyRTI-UniversityofNorthCarolinaEvidence-basedPracticeCenterunderContractNo.290-02-0016).Rockville,MD:AgencyforHealthcareResearchandQuality.November2007.Availableat:http://www.effectivehealthcare.ahrq.gov/ehc/products/14/68/RheumArthritisFinal.pdf.AccessedNovember14,2013.
34.LangS,KleijnenJ.Qualityassessmenttoolsforobservationalstudies:lackofconsensus.Int J Evid Based Healthc.2010;8(4):247.
35.SandersonS,TattID,HigginsJP.Toolsforassessingqualityandsuscep-tibilitytobiasinobservationalstudiesinepidemiology:asystematicreviewandannotatedbibliography.Int J Epidemiol.2007;36(3):666-76.
36.DreyerNA.Usingobservationalstudiesforcomparativeeffectiveness:findingqualitywithGRACE.J Comp Eff Res.2013;2(5):413-18.
37.DaCostaBR,CevallosM,AltmanDG,RutjesAW,EggerM.UsesandmisusesoftheSTROBEstatement:bibliographicstudy.BMJ Open. 2011;1(1):e000048.Availableat:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3191404/.AccessedNovember14,2013.
38.NeyarapallyGA,HammadTA,PinheiroSP,IyasuS.Reviewofqualityassessmenttoolsfortheevaluationofpharmacopidemiologicalsafetystud-ies.BMJ Open.2012;2(5):e001362.Availableat:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3467649/.AccessedNovember14,2013.
308a Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
APPENDix A Articles Tested from Systematic Reviews
Article ReferenceQuality for
Purpose
AdvokatC,DixonD,SchneiderJ,ComatyJE.Comparisonofrisperidoneandolanzapineasusedunder“real-world”conditionsinastatepsychiatrichospital.Prog Neuropsychopharmacol Biol Psychiatry.2004;28(3):487-95.
Poor
BondGR,KimHW,MeyerPS,etal.Responsetovocationalrehabilitationduringtreatmentwithfirst-orsecond-generationantipsychotics.Psychiatr Serv. 2004;55(1):59-66.
Poor
deHaanL,BeukN,HoogenboomB,DingemansP,LinszenD.Obsessive-compulsivesymptomsduringtreatmentwitholanzapineandrisperidone:aprospectivestudyof113patientswithrecent-onsetschizophreniaorrelateddisorders.J Clin Psychiatry.2002;63(2):104-107.
Poor
DolderCR,LacroJP,DunnLB,JesteDV.Antipsychoticmedicationadherence:isthereadifferencebetweentypicalandatypicalagents? Am J Psychiatry.2002;159(1):103-108.
Poor
Garcia-CabezaI,GomezJC,SacristanJA,EdgellE,GonzalezdeChavezM.Subjectiveresponsetoantipsychotictreatmentandcomplianceinschizophrenia.Anaturalisticstudycomparingolanzapine,risperidoneandhaloperidol(EFESOStudy).BMC Psychiatry.2001;1:7.
Poor
JoyceAT,HarrisonDJ,LoebelAD,OllendorfDA.Impactofatypicalantipsychoticsonoutcomesofcareinschizophrenia.Am J Manag Care. 2005;11(8Suppl):S254-61.
Poor
KasperS,JonesM,DuchesneI.Risperidoneolanzapinedrugoutcomesstudiesinschizophrenia(RODOS):healtheconomicresultsofaninternationalnaturalisticstudy. Int Clin Psychopharmacol.2001;16(4):189-96.
Poor
KoroCE,FedderDO,L’ItalienGJ,etal.Anassessmentoftheindependenteffectsofolanzapineandrisperidoneexposureontheriskofhyperlipidemiainschizophrenicpatients.Arch Gen Psychiatry.2002;59(11):1021-26.
Poor
MeyerJM.Aretrospectivecomparisonofweight,lipid,andglucosechangesbetweenrisperidone-andolanzapine-treatedinpatients:metabolicoutcomesafter1year. J Clin Psychiatry.2002;63(5):425-33.
Poor
PelagottiF,SantarlasciB,VaccaF,TrippoliS,MessoriA.Dropoutrateswitholanzapineorrisperidone:amulti-centreobservationalstudy. Eur J Clin Pharmacol. 2004;59(12):905-09.
Poor
SoholmB,LublinH.Long-termeffectivenessofrisperidoneandolanzapineinresistantorintolerantschizophrenicpatients.Amirrorstudy.Acta Psychiatr Scand.2003;107(5):344-50.
Poor
VorugantiL,CorteseL,OwyeumiL,etal.Switchingfromconventionaltonovelantipsychoticdrugs:resultsofaprospectivenaturalisticstudy.Schizophr Res.2002;57(2-3):201-08.
Poor
AsklingJ,ForedCM,BrandtL,etal.RiskandcasecharacteristicsoftuberculosisinrheumatoidarthritisassociatedwithtumornecrosisfactorantagonistsinSweden.Arthritis Rheum.2005;52(7):1986-92.
Good
BrodyDL,AiyagariV,ShacklefordAM,DiringerMN.UseofrecombinantfactorVIIainpatientswithwarfarin-associatedintracranialhemorrhage. Neurocrit Care.2005;2(3):263-67.
Poor
FernandoHC,SchauerPR,RosenblattM,etal.Qualityoflifeafterantirefluxsurgerycomparedwithnonoperativemanagementforseveregastroesophagealrefluxdisease. J Am Coll Surg.2002;194(1):23-27.
Poor
FlendrieM,CreemersMC,WelsingPM,denBroederAA,vanRielPL.Survivalduringtreatmentwithtumournecrosisfactorblockingagentsinrheumatoidarthritis.Ann Rheum Dis.2003;62(Suppl2):ii30-33.
Poor
GelsominoS,LorussoR,RomagnoliS,etal.Treatmentofrefractorybleedingaftercardiacoperationswithlow-doserecombinantactivatedfactorVII(NovoSeven):apropensityscoreanalysis.Eur J Cardiothorac Surg.2008;33(1):64-71.
Good
HalleviH,GonzalesNR,BarretoAD,etal.TheeffectofactivatedfactorVIIforintracerebralhemorrhagebeyond3hoursversuswithin3hours.Stroke.2008;39(2):473-75.
Poor
HarrisonTD,LaskoskyJ,JazaeriO,etal.“Low-dose”recombinantactivatedfactorVIIresultsinlessbloodandbloodproductuseintraumatichemorrhage.J Trauma.2005;59(1):150-54.
Poor
HolzmanMD,MitchelEF,RayWA,SmalleyWE.Useofhealthcareresourcesamongmedicallyandsurgicallytreatedpatientswithgastroesophagealrefluxdisease:apopulation-basedstudy. J Am Coll Surg.2001;192(1):17-24.
Poor
HyrichKL,SymmonsDP,WatsonKD,etal.Comparisonoftheresponsetoinfliximaboretanerceptmonotherapywiththeresponsetocotherapywithmethotrexateoranotherdisease-modifyingantirheumaticdruginpatientswithrheumatoidarthritis:resultsfromtheBritishSocietyforRheumatologyBiologicsRegister.Arthritis Rheum.2006;54(6):1786-94.
Good
IsolauriJ,LuostarinenM,ViljakkaM,etal.Long-termcomparisonofantirefluxsurgeryversusconservativetherapyforrefluxesophagitis. Ann Surg.1997;225(3):295-99.
Poor
KalicinskiP,MarkiewiczM,KaminskiA,etal.SinglepretransplantbolusofrecombinantactivatedfactorVIIamelioratesinfluenceofriskfactorsforbloodlossduringorthotopiclivertransplantation.Pediatr Transplant.2005;9(3):299-304.
Poor
KhaitanL,RayWA,HolzmanMD,etal.Healthcareutilizationaftermedicalandsurgicaltherapyforgastroesophagealrefluxdisease:apopulation-basedstudy,1996to2000.Arch Surg.2003;138(12):1356-61.
Poor
NiemannCU,BehrendsM,QuanD,etal.RecombinantfactorVIIareducestransfusionrequirementsinlivertransplantpatientswithhighMELDscores.Transfus Med.2006;16(2):93-100.
Poor
SetoguchiS,SolomonDH,WeinblattME,etal.Tumornecrosisfactoralphaantagonistuseandcancerinpatientswithrheumatoidarthritis.Arthritis Rheum.2006;54(9):2757-64.
Good
TranT,SpechlerSJ,RichardsonP,etal.Fundoplicationandtheriskofesophagealcanceringastroesophagealrefluxdisease:aVeteransAffairscohortstudy.Am J Gastroenterol.2005;100(5):1002-08.
Poor
WetscherGJ,GadenstaetterM,KlinglerPJ,etal.EfficacyofmedicaltherapyandantirefluxsurgerytopreventBarrett’smetaplasiainpatientswithgastroesophagealrefluxdisease.Ann Surg.2001;234(5):627-32.
Poor
308b Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
Article ReferenceQuality for
Purpose
ZinkA,ListingJ,KaryS,etal.TreatmentcontinuationinpatientsreceivingbiologicalagentsorconventionalDMARDtherapy. Ann Rheum Dis. 2005;64(9):1274-79.
Good
BushTL,Barrett-ConnorE,CowanLD,etal.Cardiovascularmortalityandnoncontraceptiveuseofestrogeninwomen:resultsfromtheLipidResearchClinicsProgramFollow-upStudy.Circulation.1987;75(6):1102-09.
Good
CauleyJA,SeeleyDG,BrownerWS,etal.Estrogenreplacementtherapyandmortalityamongolderwomen.Thestudyofosteoporoticfractures.Arch Intern Med.1997;157(19):2181-87.
Good
EttingerB,FriedmanGD,BushT,etal.Reducedmortalityassociatedwithlong-termpostmenopausalestrogentherapy.Obstet Gynecol. 1996;87(1):6-12.
Good
GrodsteinF,MansonJE,ColditzGA,etal.Aprospective,observationalstudyofpostmenopausalhormonetherapyandprimarypreventionofcardiovasculardisease.Ann Intern Med.2000;133(12):933-41.
Good
GrodsteinF,StampferMJ,ColditzGA,etal.Postmenopausalhormonetherapyandmortality.N Engl J Med.1997;336(25):1769-75. GoodHendersonBE,Paganini-HillA,RossRK.Decreasedmortalityinusersofestrogenreplacementtherapy.Arch Intern Med.1991;151(1):75-78. PoorLaffertyFW,FiskeME.Postmenopausalestrogenreplacement:along-termcohortstudy.Am J Med.1994;97(1):66-77. PoorPerssonI,YuenJ,BergkvistL,etal.Cancerincidenceandmortalityinwomenreceivingestrogenandestrogen-progestinreplacementtherapy—long-termfollow-upofaSwedishcohort. Int J Cancer.1996;67(3):327-32.
Good
SidneyS,PetittiDB,QuesenberryCP,Jr.Myocardialinfarctionandtheuseofestrogenandestrogen-progestogeninpostmenopausalwomen.Ann Intern Med.1997;127(7):501-08.
Good
Varas-LorenzoC,Garcia-RodriguezLA,Perez-GutthannS,etal.Hormonereplacementtherapyandincidenceofacutemyocardialinfarction.Apopulation-basednestedcase-controlstudy.Circulation.2000;101(22):2572-78.
Good
CauleyJA,SeeleyDG,EnsrudK,etal.Estrogenreplacementtherapyandfracturesinolderwomen.StudyofOsteoporoticFracturesResearchGroup.Ann Intern Med.1995;122(1):9-16.
Good
ColditzGA,HankinsonSE,HunterDJ,etal.Theuseofestrogensandprogestinsandtheriskofbreastcancerinpostmenopausalwomen. N Engl J Med.1995;332(24):1589-93.
Good
FolsomAR,MinkPJ,SellersTA,HongCP,ZhengW,PotterJD.Hormonalreplacementtherapyandmorbidityandmortalityinaprospectivestudyofpostmenopausalwomen.Am J Public Health.1995;85(8Pt1):1128-32.
Good
GrodsteinF,StampferMJ,FalkebornM,etal.PostmenopausalhormonetherapyandriskofcardiovasculardiseaseandhipfractureinacohortofSwedishwomen.Epidemiology.1999;10(5):476-80.
Good
Paganini-HillA,HendersonVW.EstrogenreplacementtherapyandriskofAlzheimerdisease.Arch Intern Med.1996;156(19):2213-17. PoorSchairerC,GailM,ByrneC,etal.Estrogenreplacementtherapyandbreastcancersurvivalinalargescreeningstudy.J Natl Cancer Inst. 1999;91(3):264-70.
Good
SellersTA,MinkPJ,CerhanJR,etal.Theroleofhormonereplacementtherapyintheriskforbreastcancerandtotalmortalityinwomenwithafamilyhistoryofbreastcancer.Ann Intern Med.1997;127(11):973-80.
Good
Acutemyocardialinfarctionandcombinedoralcontraceptives:resultsofaninternationalmulticentrecase-controlstudy.WHOCollaborativeStudyofCardiovascularDiseaseandSteroidHormoneContraception.Lancet.1997;349(9060):1202-09.
Good
WillisDB,CalleEE,Miracle-McMahillHL,etal.EstrogenreplacementtherapyandriskoffatalbreastcancerinaprospectivecohortofpostmenopausalwomenintheUnitedStates.Cancer Causes Control.1996;7(4):449-57.
Good
APPENDix A Articles Tested from Systematic Reviews (continued)
308c Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
Article Reference
Quality
Single Expert Review
Concordant Review
SolomonDH,MassarottiE,GargR,etal.Associationbetweendisease-modifyingantirheumaticdrugsanddiabetesriskinpatientswithrheumatoidarthritisandpsoriasis.JAMA.2011;305(24):2525-31.
Poor Good
RayWA,ChungCP,SteinCM,etal.RiskofpepticulcerhospitalizationsinusersofNSAIDswithgastroprotectivecotherapyversuscoxibs.Gastroenterology.2007;133(3):790-98.
Good Good
KimSY,SchneeweissS,KatzJN,etal.Oralbisphosphonatesandriskofsubtrochantericordiaphysealfemurfracturesinapopulation-basedcohort. J Bone Miner Res. 2011;26(5):993-1001.
Good Not reviewed
SrivastavaR,DowneyEC,O’GormanM,etal.Impactoffundoplicationversusgastrojejunalfeedingtubesonmortalityandinpreventingaspirationpneumoniainyoungchildrenwithneurologicimpairmentwhohavegastroesophagealrefluxdisease.Pediatrics.2009;123(1):338-45.
Good Not reviewed
ShishehborMH,HawiR,SinghIM,etal.Drug-elutingversusbare-metalstentsfortreatingsaphenousveingrafts.Am Heart J. 2009;158(4):637-43.
Good Not reviewed
SilvermanSL,WattsNB,DelmasPD,etal.Effectivenessofbisphosphonatesonnonvertebralandhipfracturesinthefirstyearoftherapy:therisedronateandalendronate(REAL)cohortstudy.Osteoporos Int.2007;18(1):25-34.
Good Not reviewed
ApplegateRJ,SacrintyMT,KutcherMA,etal.Comparisonofdrug-elutingversusbaremetalstentsonlaterfrequencyofacutemyocardialinfarctionanddeath.Am J Cardiol.2007;99(3):333-38.
Poor Poor
MayorM,MalikAZ,MinorRJJr,etal.One-yearoutcomesfromtheTAXUSexpressstentversuscypherstent.Am J Cardiol. 2009;103(7):930-36.
Good Good
PatelMR,PfistererME,BetriuA,etal.Comparisonofsix-monthoutcomesforprimarypercutaneousrevascularizationforacutemyocardialinfarctionwithdrug-elutingversusbaremetalstents(fromtheAPEX-AMIstudy).Am J Cardiol.2009;103(2):181-86.
Good Not reviewed
PerdueDG,FreemanML,DiSarioJA,etal;ERCPOutcomeStudyERCOSTGroup.Plasticversusself-expandingmetallicstentsformalignanthilarbiliaryobstruction:aprospectivemulticenterobservationalcohortstudy.J Clin Gastroenterol. 2008;42(9):1040-46.
Good Poor
KamatSA,GandhiSK,DavidsonM.Comparativeeffectivenessofrosuvastatinversusotherstatintherapiesinpatientsatincreasedriskoffailuretoachievelow-densitylipoproteingoals.Curr Med Res Opin.2007;23(5):1121-30.
Poor Not reviewed
WuE,GreenbergPE,YangE,etal.Comparisonofescitalopramversuscitalopramforthetreatmentofmajordepressivedisorderinageriatricpopulation.Curr Med Res Opin.2008;24(9):2587-95.
Poor Good
FriedmanHS,RajagopalanS,BarnesJP,etal.Combinationtherapywithezetimibe/simvastatinversusstatinmonotherapyforlow-densitylipoproteincholesterolreductionandgoalattainmentinareal-worldclinicalsetting. Clin Ther.2011;33(2):212-24.
Poor Good
KubiakDW,BryarJM,McDonnellAM,etal.Evaluationofcaspofunginormicafunginasempiricantifungaltherapyinadultpatientswithpersistentfebrileneutropenia:aretrospective,observational,sequentialcohortanalysis.Clin Ther.2010;32(4):637-48.
Poor Not reviewed
RamanathVS,BrownJR,MalenkaDJ,etal;DartmouthDynamicRegistryInvestigators.Outcomesofdiabeticsreceivingbare-metalstentsversusdrug-elutingstents.Catheter Cardiovasc Interv.2010;76(4):473-81.
Good Good
DeleaTE,TanejaC,MoynahanA,etal.Valsartanversuslisinoprilorextended-releasemetoprololinpreventingcardiovascularandrenaleventsinpatientswithhypertension.Am J Health Syst Pharm.2007;64(11):1187-96.
Poor Poor
Platts-MillsTF,CampagneD,ChinnockB,etal.AcomparisonofGlideScopevideolaryngoscopyversusdirectlaryngoscopyintubationintheemergencydepartment.Acad Emerg Med.2009;16(9):866-71.
Poor Not reviewed
Diaz-GuzmanE,Mireles-CabodevilaE,HeresiGA,etal.Acomparisonofmethohexitalversusetomidateforendotrachealintubationofcriticallyillpatients. Am J Crit Care.2010;19(1):48-54.
Good Poor
KonstanceRP,EisensteinEL,AnstromKJ,etal.Outcomesofsecondrevascularizationproceduresafterstentimplantation. J Med Syst.2008;32(2):177-86.
Good Poor
HannanEL,RaczM,WalfordG,HolmesDR,etal.Drug-elutingversusbare-metalstentsinthetreatmentofpatientswithST-segmentelevationmyocardialinfarction.JACC Cardiovasc Interv.2008;1(2):129-35.
Good Poor
LeeMS,TarantiniG,XhaxhoJ,etal.Sirolimus-versuspaclitaxel-elutingstentsforthetreatmentofcardiacallograftvasculopathy.JACC Cardiovasc Interv.2010;3(4):378-82.
Poor Not reviewed
ShorrAF,SarnesMW,PeeplesPJ,etal.Comparisonofcost,effectiveness,andsafetyofinjectableanticoagulantsusedforthromboprophylaxisafterorthopedicsurgery.Am J Health Syst Pharm.2007;64(22):2349-55.
Good Good
AllenRH,KumarD,FitzmauriceG,etal.Painmanagementoffirst-trimestersurgicalabortion:effectsofselectionoflocalanesthesiawithandwithoutlorazepamorintravenoussedation.Contraception.2006;74(5):407-13.
Good Poor
CarrageeEJ,SpinnickieAO,AlaminTF,etal.Aprospectivecontrolledstudyoflimitedversussubtotalposteriordiscectomy:short-termoutcomesinpatientswithherniatedlumbarintervertebraldiscsandlargeposterioranulardefect. Spine (Phila Pa 1976). 2006;31(6):653-57.
Poor Poor
ChuWW,KuchulakantiPK,TorgusonR,etal.Comparisonofclinicaloutcomesofoverlappingsirolimus-versuspaclitaxel-elutingstentsinpatientsundergoingpercutaneouscoronaryintervention. Am J Cardiol.2006;98(12):1563-66.
Poor Not reviewed
CorderaF,LongKH,NagorneyDM,etal.Openversuslaparoscopicsplenectomyforidiopathicthrombocytopenicpurpura:clinicalandeconomicanalysis.Surgery.200;134(1):45-52.
Poor Poor
APPENDix B Articles Tested from Published Studies with Quality Assessment by Experts
308d Journal of Managed Care & Specialty Pharmacy JMCP March 2014 Vol. 20, No. 3 www.amcp.org
The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution
Article Reference
Quality
Single Expert Review
Concordant Review
HechtHS,HarmanSM.Comparisonoftheeffectsofatorvastatinversussimvastatinonsubclinicalatherosclerosisinprimarypreventionasdeterminedbyelectronbeamtomography.Am J Cardiol. 2003;91(1):42-45.
Poor Poor
KlemmeWR,OwensBD,DhawanA,etal.Lumbarsagittalcontourafterposteriorinterbodyfusion:threadeddevicesaloneversusverticalcagesplusposteriorinstrumentation. Spine (Phila Pa 1976).2001;26(5):534-37.
Poor Not reviewed
KonstantinouK,BaddamK,LankaA,etal.Cefepimeversusceftazidimefortreatmentofpneumonia.J Int Med Res. 2004;32(1):84-93.
Poor Poor
LazarusHM,PérezWS,KleinJP,etal.AutotransplantationversusHLA-matchedunrelateddonortransplantationforacutemyeloidleukaemia:aretrospectiveanalysisfromtheCenterforInternationalBloodandMarrowTransplantResearch. Br J Haematol.2006;132(6):755-69.
Good Not reviewed
LondonMJ,MoritzTE,HendersonWG,etal;ParticipantsoftheVeteransAffairsCooperativeStudyGrouponProcesses,Structures,andOutcomesofCareinCardiacSurgery.StandardversusfiberopticpulmonaryarterycatheterizationforcardiacsurgeryintheDepartmentofVeteransAffairs:aprospective,observational,multicenteranalysis.Anesthesiology.2002;96(4):860-70.
Good Not reviewed
RhaSW,KuchulakantiPK,PakalaR,etal.Bivalirudinversusheparinasanantithromboticagentinpatientswhoundergopercutaneoussaphenousveingraftinterventionwithadistalprotectiondevice.Am J Cardiol.2005;96(1):67-70.
Good Poor
SchneeweissS,SolomonDH,WangPS,etal.Simultaneousassessmentofshort-termgastrointestinalbenefitsandcardiovascularrisksofselectivecyclooxygenase2inhibitorsandnonselectivenonsteroidalantiinflammatorydrugs:aninstrumentalvariableanalysis.Arthritis Rheum.2006;54(11):3390-98.
Good Not reviewed
SolomonDH,AvornJ,KatzJN,etal.Immunosuppressivemedicationsandhospitalizationforcardiovasculareventsinpatientswithrheumatoidarthritis.Arthritis Rheum.2006;54(12):3790-98.
Poor Not reviewed
VillarealRP,LeeVV,ElaydaMA,etal.Coronaryarterybypasssurgeryversuscoronarystenting:risk-adjustedsurvivalratesin5,619patients.Tex Heart Inst J.2002;29(1):3-9.
Good Good
VolicerL,LaneP,PankeJ,etal.Managementofconstipationinresidentswithdementia:sorbitoleffectivenessandcost.J Am Med Dir Assoc.2005;6(3Suppl):S32-34.
Good Not reviewed
WeinsteinJN,LurieJD,TostesonTD,etal.Surgicalvsnonoperativetreatmentforlumbardiskherniation:theSpinePatientOutcomesResearchTrial(SPORT)observationalcohort.JAMA.2006;296(20):2451-59.
Good Poor
WolfeF,MichaudK,StephensonB,etal.Towardadefinitionandmethodofassessmentoftreatmentfailureandtreatmenteffectiveness:thecaseofleflunomideversusmethotrexate.J Rheumatol.2003;30(8):1725-32.
Good Poor
ZimmermanS,HawkesWG,HudsonJI,etal.OutcomesofsurgicalmanagementoftotalHIPreplacementinpatientsaged65yearsandolder:cementedversuscementlessfemoralcomponentsandlateraloranterolateralversusposterioranatomicalapproach. J Orthop Res.2002;20(2):182-91.
Good Good
APPENDix B Articles Tested from Published Studies with Quality Assessment by Experts (continued)
