Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or...
Transcript of Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or...
![Page 1: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/1.jpg)
Dr. Ashrita SaranEvidence Synthesis Specialist, Campbell Collaboration
Introduction to Systematic Reviews and Evidence and Gap Maps
@campbellreviews @AshritaSaran
![Page 2: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/2.jpg)
Range of Systematic Evidence Products
COntent
Scope
Map of maps:
EGMs
Mega-map:
SRs & EGMs
Evidence and
gap map: SRs &
primary studies
Systematic review:
primary studies
Review of reviews:
systematic reviews
![Page 3: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/3.jpg)
What are Systematic Reviews?
Systematic reviews
• Systematic and transparent approaches to summarizing the findings from all relevant studies on a specific question. Systematic reviews aim to minimize bias and error at each stage of the review process.
Meta-analysis
• A set of statistical techniques for analyzing and synthesizing data, including calculation of averages and sources of variation in results
![Page 4: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/4.jpg)
Why Systematic Reviews Matters?
![Page 5: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/5.jpg)
The seven piece study
But these are observational data, which don’t control for selection bias (people who eat more than five portions a day are wealthy, educated, health fanatics)
The five piece study
This is a systematic review, using data from 16 high-quality studies (observational data but analysis controls for confounders)
Why Systematic Reviews Matters?
![Page 6: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/6.jpg)
Steps in the review process
• Setting the question (the PICO)
• Search strategy
• Screening
• Coding
• Synthesis
• Reporting
• Engagement with policy and practice
Systematic
Systematic
Systematic
Systematic
Systematic
systematic
![Page 7: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/7.jpg)
What the evidence synthesis process should look like
Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015
![Page 8: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/8.jpg)
What the evidence synthesis process actually looks like
Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015
Over-emphasis on
significant findings
Studies with significant findings are 2-3 times
more likely to be published
Selective presentation of results
![Page 9: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/9.jpg)
An example: the treatment of results from a single study of parent training (PT) versus multi-systemic training (MST) (a
branded programme)
RCT assigning 43 abusive or neglectful families to either:
• Parent training: group sessions discussing parenting techniques
• Multi-systemic therapy: individual family treatment tackling multiple issues, e.g expectation re. child behaviour, child management, emotional support, parental behaviour change
Study looked at 30 outcomes
on individual and child
functioning, stress etc.
![Page 10: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/10.jpg)
Outcome reporting and confirmation bias in action
9 out of 14
reviews report
just one outcome
from the paper,
favouring MST
Source: Julia Littell: Campbell Systematic Reviews:
Evidence for Implementation and Impact, GIC Dublin 2015
![Page 11: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/11.jpg)
And how reviewers summarized the Brunk et al. paper…
Source: Julia Littell: Campbell Systematic Reviews:
Evidence for Implementation and Impact, GIC Dublin 2015
No difference
![Page 12: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/12.jpg)
What the systematic review says
Out of home placement: no difference
Delinquency: no difference
Family cohesion: no
difference
![Page 13: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/13.jpg)
Publication Bias
Studies with statistically significant, positive results are 2-3 times more likely to be published than similar studies with null or negative results (Song et al., 2009, 2010)
– likelihood of publication is related to direction and significance of results -- net of influence of other variables
– (Begg, 1994; Cooper et al., 1997; Coursol & Wagner, 1986; Dickersin, 1987, 2005; Dwan et al., 2008; Easterbrook et al., 1991; Hopewell et al., 2007, 2009; Scherer et al., 2007; Song et al., 2000, 2009, 2010; Torgerson, 2006; Vecchi et al., 2009)
![Page 14: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/14.jpg)
Systematic reviews rebalance the evidence pyramid
Narrative
reviews
More than 100 narrative
reviews most stating
MST is more effect than
alternatives
MST is not
consistently
better or worse
than
alternatives
![Page 15: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/15.jpg)
So drop narrative reviews in favour of systematic reviews to rebalance the evidence
pyramid
More than 100 narrative
reviews most stating
MST is more effect than
alternatives
MST is not
consistently
better or worse
than
alternatives
![Page 16: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/16.jpg)
• Correlation is not causation
• Systematic reviews only include high quality evidence of effects (experimental and non-experimental designs which take account of selection bias)
• Systematic reviews minimize bias and error at each stage of the review process.
• Part of the evidence revolution
Why Systematic Reviews Matters?
![Page 17: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/17.jpg)
What are Evidence and Gap Maps?
A systematic presentation of all available, relevant evidence for a particular sector or sub-sector.
A typical map is a matrix of intervention categories (rows) and outcome domain (columns).
![Page 18: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/18.jpg)
EGMs are systematic
• Have a pre-specified protocol
• Have a systematic search strategy
• Have clear inclusion and exclusion criteria which are systematically applied
• Systematically report all eligible studies
Construction of the framework is a critical stage: needs stakeholder buy-in
![Page 19: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/19.jpg)
![Page 20: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/20.jpg)
![Page 21: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/21.jpg)
![Page 22: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/22.jpg)
These are all the same review,
i.e. one review not 12
![Page 23: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/23.jpg)
Empty reviews
![Page 24: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/24.jpg)
Area suitable for evidence synthesis
![Page 25: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/25.jpg)
Primary studies neededNote: maps
include
intermediate
outcomes
![Page 26: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/26.jpg)
![Page 27: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/27.jpg)
Evident Gaps in
systematic review
evidence!
![Page 28: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/28.jpg)
Why do we want evidence and gap maps?
Evidence and gap maps
• Guide users to available high quality evidence to inform strategy and programme development
• Tell users where there is no high quality evidence
• Identify gaps to be filled by evidence synthesis and new studies for researchers and research commissioners –and so more strategic, policy-oriented approach to research agenda
![Page 29: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/29.jpg)
Discussion paper on Evidence and Gap Map
https://www.campbell
collaboration.org/libr
ary/evidence-and-
gap-maps-
comparison-of-
different-
approaches.html
![Page 30: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/30.jpg)
Check-lists
Guidelines
Evidence portals
Evidence platforms
Evidence maps
Databases
Systematic reviews
Primary studies
Data
Supply
Evidence maps
are an important
building block in
the evidence
architecture
![Page 31: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/31.jpg)
Comparison of EGMs and systematic reviews
![Page 32: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/32.jpg)
Systematic review Evidence and gap map Comparison
Question setting Often restricted to a single
intervention, and a limited
range of outcomes. A PICOS is
specified to guide study
inclusion criteria.
Broad scope of
interventions across a
sector or sub-sector, with
full range of outcomes
across causal chain. A PICOS
is specified to guide study
inclusion criteria.
EGMs are broader in
scope than systematic
reviews.
Search strategy A comprehensive and
systematic search for primary
studies meeting the inclusion
criteria (and not exclusion
criteria)
A comprehensive and
systematic search for
systematic reviews and
primary studies meeting the
inclusion criteria (and not
exclusion criteria)
No difference in
approach. EGMs search
for systematic reviews
as well as primary
studies.
Screening Identified studies screened
against inclusion and
exclusion criteria
Identified studies screened
against inclusion and
exclusion criteria
No difference in
approach
Coding and data
extraction
Coding of study and
intervention characteristics,
moderators and data
extraction of effect sized and
related statistics
Coding of a limited number
of study and intervention
characteristics
EGMs require coding of
less data than
systematic reviews
![Page 33: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/33.jpg)
Systematic review Evidence and gap map Comparison
Critical appraisal Assessment of quality of
included studies using a
critical appraisal
instrument
Critical appraisal may not be
done, but is recommended
Critical appraisal is
mandatory for
Campbell EGMs
Evidence
synthesis
Statistical or narrative
synthesis of the evidence
Not done EGMs do not
synthesize the
evidence
Reporting Systematic reporting of
evidence
Graphical representation of map
availability of evidence.
Descriptive overview of map.
Systematic reviews
summarize what the
evidence says. EGMs
only summarize what
evidence is available.
Use To inform policy and
practice
To inform research priorities and
research funding
Systematic reviews are
to inform policy, and
EGMS primarily to
inform both research
priorities.
![Page 34: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/34.jpg)
Key take away points
Systematic reviews seek to collate all evidence that fits pre-specified eligibility criteria in order to address a specific research question.
Systematic reviews aim to minimize bias by using explicit, systematic methods
EGMs are an interactive tool to make evidence discoverable, accessible and usable
Both systematic reviews and EGMs are a building block in the evidence architecture and can be applied to range of research questions
![Page 35: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/35.jpg)
Exercise
• Go to 3ie.org and explore some EGMs
• Go to Campbell and Cochrane library and explore some systematic reviews
![Page 36: Introduction to Systematic Reviews and Evidence and Gap Maps · Evidence synthesis Statistical or narrative synthesis of the evidence Not done EGMs do not synthesize the evidence](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fc1c0b18358164af937d3e5/html5/thumbnails/36.jpg)
Thank You
Join us at What Works Global Summit
Visit www.campbellcollaboration.org