Paul Montgomery Joanne Yaffe David Albright Completing a
Campbell Systematic Review Society for Social Work and Research
Washington, DC January 15, 2012
Slide 2
Mission: The Campbell Collaboration (C2) helps people make
well- informed decisions by preparing, maintaining and
disseminating systematic reviews in education, crime and justice,
and social welfare. Who are we? The Campbell Collaboration is an
international research network that produces systematic reviews of
the effects of social interventions. Campbell is based on voluntary
cooperation among researchers of a variety of backgrounds.
Slide 3
What is a systematic review? The purpose of a systematic review
is to sum up the best available research on a specific question.
This is done by synthesizing the results of several studies. A
systematic review uses transparent procedures to find, evaluate and
synthesize the results of relevant research. Procedures are
explicitly defined in advance, in order to ensure that the exercise
is transparent and can be replicated. This practice is also
designed to minimize bias. Studies included in a review are
screened for quality, so that the findings of a large number of
studies can be combined. Peer review is a key part of the process;
qualified independent researchers control the author's methods and
results.
Slide 4
Evidence-based practice demands high quality evidence on which
to base practice and policy decisions. It is impossible for most
social work practitioners to read all relevant research on a given
topic. Even if they could, many of the available studies are of low
quality. Studies accessible in social work journals are biased in
favor of positive findings. Through critical exploration,
evaluation, and synthesis the systematic review separates the
insignificant, unsound or redundant deadwood from the salient and
critical studies that are worthy of reflection.
Slide 5
Additional Rationale for Systematic Reviews: Decision Making
Practitioners need SRs to keep up with the literature in their
field Policy-makers need them to make sensible decisions
Researchers need them to identify gaps and establish protocols for
future research.
Slide 6
Additional Rationale for Systematic Reviews: Efficiency A
review is usually less costly and time-consuming than embarking on
a new study (although an SR may conclude that new studies are
called for). Reviews can also suggest or even establish that
further research in a given area is not called for or even
unethical. Continuously updated reviews can shorten the time
between research discoveries and real world implementation of
effective diagnostic or treatment strategies.
Slide 7
Additional Rationale for Systematic Reviews: Generalizability
Diversity of multiple reviewed studies provides an interpretive
context not available in any one study. studies addressing similar
questions often use different eligibility criteria for
participants, different definitions of the problem, different
variations of an intervention, different study designs, etc.
Slide 8
Additional Rationale for Systematic Reviews: Examining
Consistency of Relationships Are effects in the same direction, and
of the same general magnitudes, given the variation in study
protocols? Does consistency exist between studies of the same
intervention or different dosages/intensities/classes of the same
intervention?
Slide 9
Additional Rationale for Systematic Reviews: Examining
Inconsistency of Relationships If an intervention strategy is
effective in one setting and not in another, or among some people
and not among others, this is important! Furthermore, whether
findings from a single study stand alone for any reason (uniqueness
of study population, study quality or outcome measure) should be
explored
Slide 10
Additional Rationale for Systematic Reviews: Power Quantitative
SRs have the advantage of increased power. The advantage of
increasing power is particularly relevant to conditions of
relatively low event rates or when small effects are being
assessed.
Slide 11
Additional Rationale for Systematic Reviews: Precision A
measure of the likelihood of random errors in the results of a
study, meta-analysis or measurement. Confidence intervals around
the estimate of effect from each study are a measure of precision.
The weight given to the results of each study in a meta- analysis
(typically the inverse of the variance of the estimate of effect)
is a measure of precision (i.e. the degree to which a study
influences the overall estimate of effect in a meta-analysis is
determined by the precision of its estimate of effect).
Slide 12
Additional Rationale for Systematic Reviews: Accuracy or at
least an improved reflection of reality. Traditional
(non-systematic) reviews can be criticized as biased and haphazard
subject to idiosyncrasies of the reviewer SRs and meta-analyses
using scientific principles which aim to reduce random and
systematic errors are superior.
Slide 13
A Campbell systematic review must have: Clear inclusion/
exclusion criteria An explicit search strategy Systematic coding
and analysis of included studies Meta-analysis (where
possible)
Slide 14
How do Campbell systematic reviews differ from other systematic
reviews? Campbell reviews must include a systematic search for
unpublished reports (to avoid publication bias). Campbell reviews
are usually international in scope. A protocol (project plan) for
the review is developed in advance and undergoes peer review. Study
inclusion and coding decisions are accomplished by at least two
reviewers who work independently and compare results. Campbell
reviews undergo peer review and editorial review.
Slide 15
The Campbell Library of Systematic Reviews Campbell reviews are
published in the Campbell Library of Systematic Reviews, and are
freely accessible online. Campbell systematic reviews can be
updated as further relevant information emerges.
Slide 16
Slide 17
Completing a Campbell Review
Slide 18
Major Steps for a Campbell Systematic Review Title Registration
Protocol Development Protocol Publication Review Development Review
Publication Periodic Updates
Slide 19
Campbell Systematic Review Select a topic and formulate a
focused question Register the title Prepare the protocol (Campbell
Editorial Board & external peer review) Electronic publication
of protocol in The Campbell Library Perform a comprehensive
literature search Prepare the Review (Editorial Board &
external peer review) Electronic publication of completed Review in
the Campbell Library Respond to comments/criticism Keep the review
up-to-date
Slide 20
Title Registration Reserve the topic Do your homework!
Components Title: Intervention for Problem in Population E.g.: Does
Eye Movement Desensitization and Reprocessing (EMDR) for
Posttraumatic Stress Disorder (PTSD) in Combat Veterans Background
and objective of the review Define the population Define the
intervention(s) Outcome(s) Methodology Review team Roles and
responsibilities Potential conflicts of interest Support Funding
Preliminary timeframe
Slide 21
Protocol Roadmap for Review Components Background Objective of
the review Methods Acknowledgements References Appendices
Contribution of authors Declaration of Interest Sources of
Support
Slide 22
Background Description of the condition Description of the
intervention How the intervention might work Why it is important to
do this review
Slide 23
Objective of the Review Examples: The primary objective is to
complete a SR of experimental and quasi-experimental studies of
EMDR for combat veterans with PTSD The secondary objective is to
synthesize the results of these studies to asses the effect of EMDR
on reducing PTSD in combat veterans
Slide 24
Methods Criteria for considering studies for the review Types
of studies (e.g., experimental, quasi-experimental) Types of
participants (e.g., military combat veterans) Types of
interventions (i.e., EMD/EMDR) Types of outcomes (e.g., level of
PTSD symptoms via structured diagnostic interview or self-report
questionnaire)
Slide 25
Methods Search methods for identification of studies Electronic
searches (e.g., EMBASE, MEDLINE, PILOTS) Search terms Searching
other resources (e.g., correspondence, grey literature, hand-
searching)
Slide 26
Methods Data Collection and analysis Selection of studies
(stage 1 and 2) Data extraction and management (stage 3) Assessment
of risk of bias within included studies (e.g., sequence generation,
incomplete outcome data) Measures of treatment effect (how we will
handle dichotomous and continuous data) Unit of analysis issues
Dealing with missing data and incomplete data Assessment of
heterogeneity (e.g., Q-statistic, I statistic) Assessment of
publication bias
Slide 27
Methods Data Synthesis Subgroup analysis, moderator analysis
and investigation of heterogeneity Sensitivity analysis
Dr Paul Montgomery University of Oxford Dr Jane Dennis, School
for Policy Studies University of Bristol
Slide 30
1. Did the review address a clearly focussed question? 2. Were
the right sort of studies selected? 3. Was the search strategy
explicit and comprehensive? 4. Did the reviewers assess the quality
of the identified studies?
Slide 31
Are non-drug CBT interventions effective in resolving the sleep
problems of people aged over 60?
Slide 32
1. Did the review address a clearly focussed question? 2. Were
the right sorts of studies selected? 3. Was the search strategy
explicit and comprehensive? 4. Did the reviewers assess the quality
of the identified studies?
Slide 33
Types of studies: Randomised controlled trials Types of
participants Over 60s diagnosed with sleep problems via
standardised measure screened to exclude dementia, depression and
sleep apnoea or secondary insomnia (sleep disturbance caused by a
psychiatric or medical disorder)
Slide 34
Sleep onset latency (SOL) Wake after sleep onset (WASO) Total
wake time (TWT) Sleep duration (total) Early morning wakening Sleep
efficiency Self-report of sleep satisfaction Scales related to
sleep, eg the Pittsburgh Sleep Quality Index (PSQI); Stanford
Sleepiness Scale Quality of life Outcomes were divided, where
possible, into immediate post-treatment, medium term (3-12 months)
and long-term (more than 12 months).
Slide 35
1. Did the review address a clearly focussed question? 2. Were
the right sort of studies selected? 3. Was the search strategy
explicit and comprehensive? 4. Did the reviewers assess the quality
of the identified studies?
Slide 36
Potential hazards: Publication Bias Tower of Babel Bias
Uncritical use of electronic databases
Slide 37
the tendency of investigators, reviewers and editors to
differentially submit or accept manuscripts for publication on the
direction or strength of the study findings. Cook DJ, Guyatt GH,
Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished
data be included in meta-analyses? Current convictions and
controversies. JAMA 1993; 269: 2749-2753
Slide 38
Prospective Registries of all trials in progress Retrospective
Trial amnesty Negotiation Funnel plot
Slide 39
Controversial Unpublished data may not be a full or
representative sample (Cook 1993) Publication is no guarantee of
scientific quality (Oxman 1991) Cook DJ, Guyatt GH, Ryan G, Clifton
J, Buckingham L, Willan A et al. Should unpublished data be
included in meta-analyses? Current convictions and controversies.
JAMA 1993; 269: 2749-2753 Oxman AD, Guyatt GH, Singer J, Goldsmith
CH, Hutchison BG, Milner RA et al. Agreement among reviewers of
review articles. J.Clin.Epidemiol. 1991;44:91-98.
Slide 40
Studies that find a treatment effect are more likely to be
published in English-language journals. Opposing studies may be
published in non-English-language journals. Gregoire G, Derderan F,
Le Lorier J. Selecting the language of the publications included in
a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol.
1995;48:159-163
Slide 41
Electronic databases Reference checking Hand-searching Personal
communications
Slide 42
sleep problems the older adult psychological, physical,
phototherapeutic, treatments
Slide 43
1. Did the review address a clearly focussed question? 2. Were
the right sort of studies selected? 3. Was the search strategy
explicit and comprehensive? 4. Did the reviewers assess the quality
of the identified studies?
Slide 44
Should be routine Dependent on sort of studies included Use of
rating scales with fixed cut-offs potentially misleading
Slide 45
Slide 46
1. Were the results similar from study to study? 2. What is the
overall result of the review? 3. How precise are the results?
Slide 47
1. Were the results similar from study to study? 2. What is the
overall result of the review? 3. How precise are the results?
Slide 48
Sources of Heterogeneity: Differences in study participants
Different comparisons Different duration of follow-up Different
outcome measures Differences in methodological quality
Slide 49
Look at plots of results Formal tests of homogeneity.
Slide 50
1. Were the results similar from study to study? 2. What is the
overall result of the review? 3. How precise are the results?
Slide 51
Slide 52
Slide 53
Slide 54
Slide 55
Slide 56
Slide 57
Slide 58
Slide 59
Slide 60
Slide 61
Slide 62
Slide 63
Slide 64
Slide 65
Slide 66
Slide 67
Slide 68
Slide 69
Estimates from individual trials vary more than can be
explained by the play of chance alone N.B. Meta-analysis should NOT
overlook important material differences in subgroup response
Slide 70
does it makes sense to average effects from the studies?
Slide 71
Qualitative v. quantitative Qualitative reconsider pooling
Fixed v. random effects
Slide 72
short term improvements in sleep quality, sleep efficiency, and
wake time and to long term improvement in self reported sleep
duration
Slide 73
Yes
Slide 74
Pallesen, Nordhus and Kvale (1998) Non-pharmacological
interventions for insomnia in older adults.psychological
interventions included in this meta-analysis produce significant
and long-lasting changes in the sleep of older insomniacs. However,
they included non-randomised trials that included people without a
diagnosis of a sleep problem and trials where more than 30% of
participants dropped out
Slide 75
However this review includes non-randomised trials with
participants as young as 17 as well as people who were not
insomniacs
Slide 76
Slide 77
Title Registration Protocol Development Protocol Publication
Review Development Review Publication Periodic Updates
Slide 78
Resources Cochrane Handbook for Systematic Reviews of
Interventions http://www.cochrane.org/resources/handbook/
http://www.cochrane.org/resources/handbook/ Cochrane Open Learning
Materials http://www.cochrane- net.org/openlearning/index.htm
http://www.cochrane- net.org/openlearning/index.htm EQUATOR Network
(Enhancing the Quality and Transparency of Health Research)
http://www.equator-network.org/index.aspx?o=1032http://www.equator-network.org/index.aspx?o=1032
Centre for Reviews and Dissemination (CRD), University of York, UK
http://www.york.ac.uk/inst/crd/methods.htm
http://www.york.ac.uk/inst/crd/methods.htm David Wilson's page on
"Meta-Analysis Stuff http://mason.gmu.edu/%7Edwilsonb/ma.html
http://mason.gmu.edu/%7Edwilsonb/ma.html All can be found via:
http://www.campbellcollaboration.org/resources/research/Methods_Links.php
http://www.campbellcollaboration.org/resources/research/Methods_Links.php