How to read a Systematic Review: The FAST tool
-
Upload
harlan-paul -
Category
Documents
-
view
19 -
download
0
description
Transcript of How to read a Systematic Review: The FAST tool
![Page 1: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/1.jpg)
How to read a Systematic Review: The FAST tool
Paul GlasziouCentre for Evidence Based Medicine
University of Oxford
www.cebm.net
Find
Appraise
Synthesise
Transferability
![Page 2: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/2.jpg)
The best evidence for different types of question
Level Treatment Prognosis Diagnosis
I Systematic Review of …
Systematic Review of …
Systematic Review of …
II Randomised trial
Inception Cohort
Cross sectional
III
![Page 3: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/3.jpg)
Are RCTs always needed for treatment questions?
• Some immediate & dramatic effects don’t need RCTs*
• Example:• Child with nasal foreign body
Dislodged with Parent Kiss method
Case series of success 15/19o Botma J Laryngol Otol 2000
* Glasziou, Chalmers, Rawlins, McCulloch BMJ 2007
![Page 4: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/4.jpg)
What do you do?
For an acutely ill patient, you do a search
• You find 17 studies: • some show significant results; most do
not
![Page 5: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/5.jpg)
The 17 studies : Forest Plot/Blobbogram
A. Which is the smallest study?B. Which is the largest study?C. How many are statistically
significant?D. Which studies are “large
enough”?
![Page 6: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/6.jpg)
The 17 studies: of streptokinase for MI
![Page 7: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/7.jpg)
How large should the study be?
![Page 8: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/8.jpg)
What sample size is needed?
For E.H.D.* the usual mortality rate is 0%What sample size is needed to detect a reduction
in mortality?• 100• 1,000• 100,000• 1,000,000
* Excessive happiness disorder
![Page 9: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/9.jpg)
Sample Size: Café Rule 1The 50:50 Rule (proportions)
Control Rate Number Events
Control# (Rule 1)
Control# (Fisher exact)
20% 50 250 215
10% 50 500 463
5% 50 1000 962
50 events are needed in the control group:
(For an 80% chance of finding a 50% reduction)
Glasziou P, Doll H. Was the study big enough? Two cafe rules. Evid Based Med. 2006;11(3):69-70.
![Page 10: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/10.jpg)
What sample size is needed?
• There is usually a 12% mortality rate You think your treatment will lower
mortality by 50%
• What sample size is needed?
![Page 11: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/11.jpg)
What sample size is needed?
• There is usually a 12% mortality rate You think your treatment will lower mortality
by 50%
• What sample size is needed?
• 12% means 12/100 or 24/200 or 48/400 and 50 per 417
• Control + Treatment Groups = 834 in total
![Page 12: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/12.jpg)
Systematic Review or meta-analysis?
• A Systematic Review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.
• Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.
![Page 13: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/13.jpg)
Is the review any good?FAST appraisal
• Question – What is the PICO?
• Finding Did they find most studies?
• Appraisal Did they select good ones?
• Synthesis What to they all mean?
• Transferability of results
![Page 14: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/14.jpg)
Why do I need to check the review?
Most reviews do not pass minimum criteria
A study of 158 reviews* Only 2 met all 10 criteria Median was only 1 of 10 criteria met
* McAlister Annals of Intern Med 1999
FAST tool = 4 criteria
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 15: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/15.jpg)
What it the review question (PICO)?
• Population• Intervention• Comparison• Outcome(s)
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 16: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/16.jpg)
Do pedometers increase activity and improve health?
• Find: what is your search strategy? Databases? Terms? Other methods?
Do yourself thenGet neighbour’s help
![Page 17: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/17.jpg)
FIND: Did they find all Studies?
• Check for existing systematic review?• Good initial search
Terms (text and MeSH) At least 2 Databases: MEDLINE, EMBASE, CINAHL, CCTR,
...
• Plus a Secondary search Check references of relevant papers & reviews and Find terms (words or MeSH terms) you didn’t use Search again! (snowballing)
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 18: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/18.jpg)
Is finding all published studies enough?
• Negative studies less likely to be published than ‘Positive’
• How does this happen?• Follow-up of 737 studies at Johns
Hopkins (Dickersin, JAMA, 1992) Positive SUBMITTED more than negative
(2.5 times)
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 19: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/19.jpg)
Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined
Published Registered
No. studies 16 13
Survival ratio 1.16 1.05
95% CI 1.06-1.27 0.98-1.12
P-Value 0.02 0.25
Simes, J. Clin Oncol, 86, p1529
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 20: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/20.jpg)
Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined
Published Registered
No. studies 16 13
Survival ratio 1.16 1.05
95% CI 1.06-1.27 0.98-1.12
P-Value 0.02 0.25
Simes, J. Clin Oncol, 86, p1529
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 21: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/21.jpg)
Which are biased? Which OK?
1. All studies published in BMJ, Lancet, JAMA or NEJM
2. All publicly funded studies3. All studies with more than 100 patients4. All studies conducted in the Northern
Hemisphere5. All studies registered studies
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 22: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/22.jpg)
Publication Bias: Solution
• All trials registered at inception,oThe National Clinical Trials Registry: Cancer
TrialsoNational Institutes of Health Inventory of
Clinical Trials and Studieso International Registry of Perinatal Trials
• Meta-Registry of trial Registries www.controlled-trials.com
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 23: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/23.jpg)
![Page 24: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/24.jpg)
APPRAISE & select studies
Did they select only the good quality studies?
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 25: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/25.jpg)
Miscalculating Number Needed to Treat (NNT)
1. EFFICACY: Would the death penalty for miscalculating an NNT decrease miscalculation? Yes/No
2. ATTITUDE: Should we introduce the death penalty for miscalculation of NNT? Yes/No
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 26: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/26.jpg)
Selective Criticism of EvidenceBiased appraisal increases polarization
Lord et al, J Pers Soc Psy, 1979, p2098
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 27: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/27.jpg)
Selective Criticism of Evidence
“Positive” “Negative”
Relevance 5.2 4.9
Methods 4.2 2.4
Presentation 4.3 2.6
Summary 3.2 1.8
28 reviewers assessed one “study” results randomly positive or negative
(Cog Ther Res, 1977, p161-75)
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 28: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/28.jpg)
Assessment: How can you avoid biased selection of studies?
• Assessment and selection should be: Standardized “Objective” OR Blinded to Results
* assessment of quality blind to study outcome
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 29: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/29.jpg)
Synthesis: pooling the results
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 30: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/30.jpg)
Meta-analysis (Forest) plot
APPRAISE SYNTHESISE TRANSFERABLEFIND
Graphical and tabular summary of studies?
![Page 31: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/31.jpg)
Transferable? Use in my patients
Is the AVERAGE effect similar across studies?
• If NO, then WHY? Study methods (RAMbo - biases) PICO (Patients, Intervention, …)
• If YES, then 2 questions Effect in different individuals? Which version of treatment?
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 32: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/32.jpg)
Meta-analysis (Forest) plot
Are the results similar across studies? 3 tests1. Eyeball” test – do they look they same?2. Test of “Null hypothesis” of no variation (p-value)3. Proportion of variation not due to chance (I2)
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 33: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/33.jpg)
Epidemiology of Reviews
![Page 34: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/34.jpg)
ConclusionEBM and Systematic Review
• EBM (quick & dirty)• Steps
1. Ask Question2. Search3. Appraise
4. Apply
• Time: 90 seconds• < 20 articles• This patient survives!
• Systematic Review• Steps
1. Ask Question2. Search ++++ x 23. Appraise x 24. Synthesize5. Apply
• Time: 6 months, team
• < 2,000 articles• This patient is dead
Find a systematic review!! (and appraise it FAST)
![Page 35: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/35.jpg)
Pros and cons of systematic reviews
• Advantages Larger numbers &
power Robustness across
PICOs
• Disadvantages May conclude small
biases are real effects
![Page 36: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/36.jpg)
Is the review any good?FAST appraisal
• Question – What is the PICO?
• Finding Did they find most studies?
• Appraisal Did they select good ones?
• Synthesis What to they all mean?
• Transferability of results
![Page 37: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/37.jpg)
Combined results
![Page 38: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/38.jpg)
Using review results: what do I do with my patient?
• STUDY: meta-analysis of behavioural interventions for insomnia adults “.. confirms the efficacy of behavioral interventions
for person with chronic insomnia.“
• PROBLEM: No regimens for ‘behavioural intervention’ described Author asked: “what specific treatment regime (or
regimes) would you recommend based on your review?”
Author response: “It was found that cognitive, behavioral and relaxation therapies all in general lead to similar improvements in sleep outcomes---although cognitive approaches might have been a bit better. The references for these studies are found in the article. “
Rx
“Behavioural Intervention”
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 39: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/39.jpg)
![Page 40: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/40.jpg)
Summary: systematic reviews
• Advantages Larger numbers &
power Robustness across
PICOs
• Disadvantages May conclude small
biases are real effects
![Page 41: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/41.jpg)
The results: Are studies similar?
• What are the overall results?• Similarity of results
Heterogeneity statistic
• Similarity of question (PICO) Your judgement!
![Page 42: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/42.jpg)
Traditional cf systematic reviews
• Traditional Many questions No search methods No inclusion criteria No combining
studies
• Systematic One question Explicit search Explicit inclusion
criteria Combine study
results(meta-analysis)
![Page 43: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/43.jpg)
(a*+) (b+)
(c*) (d)
Minimum clinicalImportant difference
No difference
Which are (i) statistically significant * and (ii) Clinically significant +?
![Page 44: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/44.jpg)
• Which studies (presented as Odds Ratio): Are not statistically significant? Have < 50 patients in the control arm?
![Page 45: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/45.jpg)
Sample Size: Cafe Rule 2 - continuousthe 17/(SD squared) Rule
Differencein mmHg
Difference(in SDs)
Number perarm
16 1 17
8 ½ 68
4 ¼ 272
For continuous outcomes, number per arm is
17 / (“worthwhile difference” measured in SDs)
![Page 46: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/46.jpg)
In a small randomized double-blind trial of a new treatment for acute myocardial infarction, the mortality in the treated group was half that in the control group, but the difference was not significant. We can conclude that:
a) The treatment is uselessb) there is no point in continuing to develop the
treatmentc) the reduction in mortality is so great that we
should introduce the treatment immediatelyd) we should keep adding cases to the trial until the
Normal test for comparison of two proportions is significant
e) we should carry out a new trial of much greater size
![Page 47: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/47.jpg)
The best evidence for different types of question
Level Treatment Prognosis Diagnosis
I Systematic Review of …
Systematic Review of …
Systematic Review of …
II Randomised trial
Inception Cohort
Cross sectional
III
![Page 48: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/48.jpg)
Has the systematic reviewer done a good job?
![Page 49: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/49.jpg)
Are the studies equivalent?
• Are variations in results between studies consistent with chance?(Test of homogeneity: has low power)
• If NO, then WHY? Variation in study methods (RAMbo -
biases) Variation in PICO (Patients, Intervention,
…)
APPRAISE SYNTHESISE TRANSFERABLEFIND
![Page 50: How to read a Systematic Review: The FAST tool](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813277550346895d99101a/html5/thumbnails/50.jpg)
Is bed rest ever helpful?A systematic review of trials*
*Allen, Glasziou, Del Mar. Lancet, 1999
• 10 trials of bed rest after spinal puncture no change in headache with bed rest Increase in back pain
• Protocols in UK neurology units - 80% still recommend bed rest after LP
Serpell M, BMJ 1998;316:1709–10
• …evidence of harm available for 17 years preceding...