Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools...
Transcript of Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools...
![Page 1: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/1.jpg)
Qualitativesynthesismethodsandtools&ChallengesofSR&MAofcomplex
interventions
August18,2017
MeiChung,PhD,MPHAssistantProfessor
DepartmentofPublicHealthandCommunityMedicine,SchoolofMedicine,
TuftsUniversity
![Page 2: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/2.jpg)
Evidence‐basedDecisionMaking
0 Evidence‐baseddecision‐makingprocessintegrates:0 Bestavailableresearchevidence
0 Practitionerexpertiseandotheravailableresources
0 Characteristics,needs,values,andpreferencesofthosewhowillbeaffectedbytheintervention
Source:JacobsJAetal.ToolsforImplementinganEvidence‐BasedApproachinPublicHealthPractice.inPublicHealthPractice. PrevChronicDis2012;9:110324.Alsoavailableonline:http://www.cdc.gov/pcd/issues/2012/11_0324.htm [access6/26/2016]
6
![Page 3: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/3.jpg)
Qualitativevs.QuantitativeSynthesis0 Qualitativesynthesis
0 Requiredforallsystematicreviews0 Contentexpertiseisneeded0 Subjectivitycannotbetotallyavoidedsotransparencyoftheprocessisthekeytominimizebiases
0 GRADEisthemostcommonlyusedapproach(http://www.gradeworkinggroup.org/)
0 Quantitativesynthesis(e.g.,meta‐analysis)0 Empiricalandobjective0 Contentexpertiseisnotneededforpooling(butwouldhelpinterpretresults)
0 Optionalforallsystematicreviewbut..Justifications areneededforwhenmeta‐analysisCANNOTorISNOTAPPROPRIATEtobeperformed
![Page 4: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/4.jpg)
Source:Piccoli,G.B.,etal.,Lowproteindietsinpatientswithchronickidneydisease:abridgebetweenmainstreamandcomplementary‐alternativemedicines? BMCNephrol,2016.17(1):p.76.
ThisisNot aQualitativeSynthesis
8
![Page 5: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/5.jpg)
0 Definitionofquality:0 “[T]heextenttowhichallaspectsofastudy’sdesignandconductcanbeshowntoprotectagainstsystematicbias,nonsystematicbias,andinferentialerror.” (Lohr &Carey,1999)
0 Consideredtobesynonymouswithinternalvalidity0 =Whetherresultsofaprimarystudyshouldbebelieved
0 DistinctfromassessmentofQuality(riskofbias)forabodyofevidence0 =Thecertaintylevelintheconclusionsdrawnfromthesynthesisofthebodyofevidence
WhatIsQuality(RiskofBias)AssessmentofIndividualStudies?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
![Page 6: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/6.jpg)
0 SystematicErrors:0 Includeselectionbiasandconfounding,inwhichvaluestendtobeinaccurateinaparticulardirection
0 NonsystematicErrors:0 Areattributabletochance
0 InferentialErrors:0 Resultfromproblemsindataanalysisandinterpretation,suchaschoiceofthewrongstatisticalmeasureorwronglyrejectingthenullhypothesis
WhatAretheComponentsofQualityAssessment?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
![Page 7: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/7.jpg)
ConsidertheContributionofanIndividualStudytotheBodyofEvidence
Quality(systematicerrorandinferentialerror,typeofstudy)
Sizeofstudy(nonsystematicorrandomerror)
Directionandmagnitudeofresults
Relevanceofresultstokeyquestions
RiskofBias
Precision
Directness
Consistency
Applicability
Owens DK, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.
Directvs.indirectcomparisons,healthoutcomesvs.surrogateoutcomes
![Page 8: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/8.jpg)
0 Qualityassessmentisrequiredfor:0 Interpretingresults0 Gradingthestrengthofthebodyofevidence
0 Qualityassessmentmayalsobeusedfor:0 Selectingstudiesforthereview(basedonaprioriassessmentofevidencegaps)
0 Selectingstudiesforqualitativesynthesis0 Selectingstudiesforquantitativesynthesis0 Interpretingheterogeneousfindings
ReasonsforQualityAssessment
![Page 9: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/9.jpg)
1. Classifythestudydesign2. Applypredefinedcriteriaforqualityassessmentofeachoutcomebasedon:0 Studydesign— sourcesofbiasmayvarybydesign0 Studyconduct— poorstudyconductanddiscrepancybetweendesignandconductmayincreaseriskofbias
0 Reporting— qualityassessmentmaybeinfluencedbyadequacyofreportingwheninformationonstudydesignandconductaremissing
3. Arriveatasummaryjudgmentofthestudy’squalitytoratestudiesgood,fair,orpoor0 Requiresresolutionofconflictswhentworeviewersindependentlyevaluatequality
WhatAretheStepsinQualityAssessmentofEachIndividualStudy?
HelfandM,BalshemH.In:Methodsguideforcomparativeeffectivenessreviews.Availableat:http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 10: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/10.jpg)
Tools forRiskofBiasAssessmentofIndividualStudies
0 Manytools,butfew“validated”(rigorouslydevelopedortestedforvalidityandreliability)tools*0 Cochraneriskofbiasassessment:Interventionstudieshttp://bmg.cochrane.org/assessing‐risk‐bias‐included‐studies0 TheNewcastle‐OttawaScale:Cohortorcase‐controlstudieshttp://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
*Hartling etal.ValidityandInter‐RaterReliabilityTestingofQualityAssessmentInstruments[Internet].Rockville(MD):AgencyforHealthcareResearchandQuality(US);2012Mar.Availablefromhttp://www.ncbi.nlm.nih.gov/books/NBK92293/**Salametal.Systematicreviewsonselectednutritioninterventions:descriptiveassessmentofconductandmethodologicalchallenges.BMCNutrition2015;1:9
14
![Page 11: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/11.jpg)
Many“specialized”tools
15
![Page 12: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/12.jpg)
ToolsforStrength(Quality)ofEvidenceAssessment(I)0 GRADEapproach(http://www.gradeworkinggroup.org/)
Hollands etal.Portion,packageortablewaresizeforchangingselectionandconsumptionoffood,alcoholandtobacco.CochraneDatabaseofSystematicReviews2015,Issue9.Art.No.:CD011045.
![Page 13: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/13.jpg)
ToolsforStrength(Quality)ofEvidenceAssessment(II)0 AppliedGRADEapproachwhentherewerenometa‐analysisperformed
Murad et al. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med. 2017 Jun;22(3):85-87.
![Page 14: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/14.jpg)
ToolsforStrengthofEvidenceAssessment(III)0 NutritionEvidenceLibrarygradingrubric:quality,quantity,consistency,impact,andgeneralizability(http://www.nel.gov/topic.cfm?cat=3385)
18
![Page 15: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/15.jpg)
ExampleofaNon‐transparentGradingProcess0 Conclusionstatement
0 Limitedevidencefromasmallnumberofstudiessuggestsalowerriskoflungcancerassociatedwithdietarypatternscontainingmorefrequentservingsofvegetables,fruits,seafood,grainsandcereals,legumesandleanvs.higherfatmeatsandlowerfatornon‐fatdairyproducts.Despitereportedmodestsignificantreductionsinrisk,definitiveconclusionscannotbeestablishedatthistimebecauseofthesmallnumberofarticles,aswellaswidevariationinstudydesign,dietaryassessmentandcaseascertainment.
0 Grade:Limited
19
![Page 16: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/16.jpg)
Howdosystematicreviewsincorporateriskofbiasassessmentsintothesynthesisofevidence?
Numberofreviews %ofcategorytotal
Wereriskofbiasassessmentsincorporatedintosynthesis?
Yes 37 63
No 20 34
Notapplicable 2 3
Howwereriskofbiasassessmentsincorporatedintosynthesis?
Sensitivityanalysis(eg,limitingtostudiesatlowestriskofbiasinasecondaryanalysis) 20 54
Narrative(discussionwithintext) 14 38
Exclusionofstudiesathighriskofbiasfrommainreviewsynthesis 5 14
Otherapproach 4 1120
Reference:Katikireddi SVetal.JEpidemiol CommunityHealth.2015Feb;69(2):189‐95. Availableathttp://jech.bmj.com/content/69/2/189
Table 4 Methods for Incorporating risk of bias assessments into reviews during synthesis
![Page 17: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/17.jpg)
MetaAnalysis0 Nonewdatabutcangeneratenewhypothesis0 Validityofmeta‐analysisresultsreliedonthequalityofthedata
![Page 18: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/18.jpg)
IntroductiontoMeta‐analysis0 Manyreadilyavailable(anduserfriendly)softwaretoallowpeoplewithoutmuchstatisticalbackgroundtoperformmeta‐analysis0 But..Mostdevelopedtocombinedrandomizedtrialresults
0 Doingameta‐analysisis(thus)veryeasybutdoingonewellishard0 Manypre‐analysissteps(e.g.,extractingquantitativedatafromtheliterature)requiresomebiostatisticsknowledge
0 Manymissingdata(intheliterature)mayberequiredtobeimputed/calculated
0 Interpretationsof(study‐level)meta‐analysisresultsaredifferentfromanalysisofpeople‐leveldata
22
![Page 19: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/19.jpg)
0 Isitreasonable?0 Arethecharacteristicsandeffectsofstudiessufficientlysimilartoestimateanaverageeffect?
0 Typesofheterogeneity:0 Clinicaldiversity:Arethestudiesofsimilartreatments,populations,settings,design,etcetera,suchthatanaverageeffectwouldbeclinicallymeaningful?
0 Methodologicaldiversity:Arethestudiesofsimilardesignandconductsuchthatanaverageeffectwouldbeclinicallymeaningful?
0 Statisticalheterogeneity:Istheobservedvariabilityofeffectsgreaterthanthatexpectedbychancealone?
Heterogeneity(Diversity):toPoolornottoPool?
23
![Page 20: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/20.jpg)
HeterogeneityIssuesaretheMainChallengesinConductingSRandMAofComplexIntervention
24Source:https://effectivehealthcare.ahrq.gov/topics/interventions‐tools‐guidance/abstract/
![Page 21: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify](https://reader031.fdocuments.net/reader031/viewer/2022021718/5b7a57227f8b9a483c8bdf8e/html5/thumbnails/21.jpg)
HETEROGENEOUS TREATMENT EFFECTS
IGNORE INCORPORATEESTIMATE(insensitive) EXPLAIN
FIXED EFFECTS
MODEL
DO NOT COMBINE WHEN
HETEROGENEITY IS PRESENT
RANDOM EFFECTS MODEL
SUBGROUP ANALYSES
META-REGRESSION(control rate, covariates)
DealingWithHeterogeneity:Meta‐analyticalTechniques