TheGRADE-CERQual approach:Assessingconfidenceinfindingsfromsynthesesofqualitativeevidence

ConfidenceintheEvidencefromReviewsofQualitativeresearch

WhydidwedevelopGRADED-CERQual?

• Systematicreviewsofqualitativeresearch(alsocalledqualitativeevidencesyntheses)becomingincreasinglycommon

• Alsoincreasinglybeingusedinguidelineorpolicydevelopmentprocesses

• Usersneedmethodstoassesshowmuchconfidencetoplaceinfindingsfromthesereviews

• Userslikelytomakethesejudgementsanyway– itishelpfultoprovideasystematicandtransparent wayfordoingthis

HowwasGRADE-CERQual developed?• Researcherswithbackgroundsinqualitativeresearchandsystematicreviews• Broadconsultationwithwidegroupofstakeholders

Neededanapproachthat:• Couldbeappliedtotypicaltypesofqualitativestudyapproachesanddata• Waseasytouse• Allowedjudgementstobereportedtransparently• Allowedthejudgementstobeunderstood

RelationshiptoGRADE

• CERQualispartoftheGRADEWorkingGroup

• CERQualsharesthesameaimastheGRADEtoolusedtoassessthecertaintyofevidenceofeffectiveness

• However,CERQualisgroundedintheprinciplesofqualitativeresearch

CERQualisnotatoolfor:

•Assessinghowwellanindividualqualitativestudywasconducted

•Assessinghowwellasystematicreviewofqualitativestudieswasconducted

•Assessingquantitativestudiesofqualityofcare

•Assessingconfidencein‘narrative’or‘qualitative’summariesoftheeffectivenessofanintervention,wheremeta-analysisisnotpossible

Whatdoes theCERQualapproachdo?•CERQualaimstotransparentlyassessanddescribehowmuchconfidencetoplaceinfindingsfromqualitativeevidencesyntheses

CERQualisappliedtoindividualsynthesisfindings

• Inthecontextofaqualitativeevidencesynthesis,afindingis…:

• Findingsfromqualitativeevidencesynthesescanbepresentedas:• themes,categoriesortheories• Asbothdescriptiveormoreinterpretivefindings

…ananalyticoutputthatdescribesaphenomenonoranaspectofaphenomenon

Whatdowemeanby’confidenceintheevidence’?

Anassessmentoftheextenttowhichareviewfindingisareasonablerepresentationofthephenomenonofinterest

• i.e.thephenomenonofinterestisunlikelytobesubstantiallydifferentfromthereviewfinding

TheCERQualapproach

What skillsdoyou need toapplyCERQual?

• Anunderstanding of systematic review methodology• Anunderstanding of the principles of qualitative research

CERQualmadeeasy

Scenario:

Decisionmakersareconsideringanewintervention- Buthowarefemalepatientslikelytoexperienceit?

Reviewofqualitativeresearchiscarriedout–findingsdescribewomen’sexperiencesoftheintervention

Afterassessingallfourcomponentsanoverallassessmentismade,expressedaseither:

- Highconfidence- Moderateconfidence- Lowconfidence- Verylowconfidence

Component1:Methodologicallimitations

Theextenttowhichthereareproblemsinthedesignorconductoftheprimarystudiessupportingareviewfinding

Concernsaboutmethodologicallimitations

theprimarystudiesunderlyingareviewfindingareshowntohaveproblemsinthewaytheyweredesignedorconducted

� Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:

� Acritical appraisaltoolforqualitativestudiesshouldbeused tomakethisassessment� Typicallyincludesappraisalsofhowtheparticipants

andsettingswereselected,howdatawascollectedandanalysed,researcherreflexivityetc

� Currentlynowidespreadagreementaboutthebesttool– researchagendainplace

Component2:Relevance

Theextenttowhichthebodyofevidencefromtheprimarystudiessupportingareviewfindingisapplicabletothecontextspecifiedinthereviewquestion

Concernsaboutrelevance

thecontextsoftheprimarystudiesunderlyingareviewfindingaresubstantivelydifferentfromthecontextofthereviewquestion

Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:

Assessing relevance - examples

• Exampleofindirectrelevance:Inasynthesisfocusingonchildren,oneincludedstudyfocusedonchildrenfrom3-5yearswhilethesynthesiswasinterestedintheagegroup10-18years

• Exampleofpartialrelevance:Inthesamesynthesis,severaloftheincludedstudiesfocusedongirlsonly,whilethesynthesiswasinterestedinallchildren

• Exampleofuncertainrelevance:Inthesamesynthesis,theagesofthechildreninsomeofthestudieswasunclear

Component3:Coherence

Anassessmentofhowclearandcogentthefitisbetweenthedatafromtheprimarystudiesandthereviewfinding*

*HasbeenupdatedsincePLOSarticle

Concernsaboutcoherence

Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:- Someofthedatacontradictthefinding- Someofthedataareambiguous

Assessingcoherence–transformationofthedata

Degree of transformation

Component4:Adequacyofdata

Thedegreeofrichness andquantityofdatasupportingareviewfinding

Concernsaboutadequacyofdata

thedataunderlyingareviewfindingarenotsufficientlyrichoronlycomefromasmallnumberofstudiesorparticipants

• Reviewauthorsneedtomakeajudgementonwhatconstitutesdatathatarenotsufficientlyrichortoosmallanumberinthecontextofaspecificreviewfinding

Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:

Making anoverallassessment

METHODO-LOGICAL

LIMITATIONS

COHERENCE

RELEVANCE

ADEQUACY OF DATA

Confidence

Afterassessingeachoftheseparatecomponents,wemakeanoveralljudgementoftheconfidenceineachreviewfinding

Confidencecanbeassessedashigh,moderate,loworverylow

• Highconfidence:Itishighlylikelythatthereviewfindingisareasonablerepresentationofthephenomenonofinterest

•Moderateconfidence:Itislikelythatthereviewfindingisareasonablerepresentationofthephenomenonofinterest

• Lowconfidence:Itispossiblethatthereviewfindingisareasonablerepresentationofthephenomenonofinterest

• Verylowconfidence:Itisnotclearwhetherthereviewfindingisareasonablerepresentationofthephenomenonofinterest

CERQual qualitativeevidenceprofile

� Transparencyoftheprocessofassessingeachcomponentisenhancedthroughtheuseofaqualitativeevidenceprofilelikethisone

� Inthisprofile,reviewauthorsshouldindicate:¡ Whethertheyhaveno,minor,moderateorsubstantialconcernsregardingeachCERQual componentandwhatthese

concernsare¡ TheiroverallCERQual assessmentandtheirexplanationofthis

� TheQualitativeEvidenceProfilecanbeincludedasanappendixtoaqualitativeevidencesynthesis

Summary of Qualitative Findings tables

February17,2017 38

ASoQF tableisthefinaloutputoftheprocessofmakingaCERQualassessment,andincludesfour elements:1. Asummaryofeachreviewfinding2. AnoverallCERQual assessmentforeachreviewfinding3. Anexplanationoftheoverallassessment(couldbeincludedasfootnotes)4. Referencetothestudiescontributingdatatothereviewfinding,including

clarificationofthecontextsinwhichthosestudieswereconducted

Operationalising theCERQual approach• OurrecentlypublishedPLOSMedicinepaperprovidessomeguidanceonoperationalizingeachCERQual component– pleasehavealookatthispaperbeforeyoutrytoapplyCERQual!

•WearenotrecommendinganyspecifictoolstoassesseachCERQual component.Rather,wehaveidentifiedarangeofissuesthatshouldbeconsideredforeachcomponent

•Moredetailedguidanceandpromptsarebeingdevelopedthisyearandwillbepublishedasaseriesofpapers

•MembersoftheCERQual ProjectGroupareavailabletomentornewusers

Insummary:CERQualhasthepotentialto:• Facilitatereflectiononfindingsfromqualitativeevidencesyntheses•Makemoreexplicit:• wheretherearegapsorinsufficientevidenceinrelationtoareviewquestion• whereonlypoorqualitystudiesareavailable

• Promptthegenerationofnewexplanationsorconceptstoexplainpatternsinfindings

ApplyingCERQual involvesjudgements.CERQualattemptstomakethesejudgementstransparentandincreasethecontributionofqualitativeresearchtodecisionmaking

GettinginvolvedinGRADE-CERQual• YouareinvitedtojoinourmailinglistandtheGRADE-CERQualProjectGroupviaourwebsite:www.cerqual.org

• You can makeanimportant contribution by:• Piloting CERQual inyour qualitative evidence synthesis• Recording what worked well andwhat was moredifficult inthe process• Sharing this feedbackwith us,sothat we can use this toimprove ourguidance forundertakingCERQual assessments

• Practical resources (e.g.templates)are available andwe can alsolinknew CERQual users with amentor.Please contact us torequest access tothese

• TheGRADE-CERQual ProjectGroupwill holdmeetings andtrainingsessionsduringthe year – information on these will besentoutviaour mailing list

TolearnmoreaboutCERQual

• Jointhemailinglist• Jointheprojectgroup• Joinawebinarorfurtherface-to-facetraining• Forthcomingseriesofpapers,probablyinImplementationScience

GRADECERQual@gmail.comwww.cerqual.org