Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman,...
-
Upload
calvin-osborne -
Category
Documents
-
view
217 -
download
1
Transcript of Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman,...
![Page 1: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/1.jpg)
Why some/many (all?) published clinical trials are false
John P.A. IoannidisProfessor and Chairman, Department of Hygiene and Epidemiology,
University of Ioannina School of Medicine, Ioannina, GreeceProfessor of Medicine (adjunct), Tufts University School of Medicine,
Boston, USA
![Page 2: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/2.jpg)
Why research findings may not be credible?
• There is bias
• There is random error
• Usually there is plenty of both
![Page 3: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/3.jpg)
Discrepancies over time occur in randomized trials
Myocardial infarction interventions
Cumulative sample size
4000030000
2000010000
50004000
30002000
1000500
400300
200100
Rel
ativ
e ch
ange
in tr
eatm
ent e
ffec
t
3
2
1
.9
.8
.7
.6
Ioannidis and Lau, PNAS 2001
![Page 4: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/4.jpg)
Diminishing effects are common in clinical medicine
• Across 100 meta-analyses of mental health related interventions, when it comes to pharmacotherapies, it was far more likely for effect sizes to diminish rather than increase with the appearance of newer trials
Trikalinos et al. J Clin Epidemiol 2004
![Page 5: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/5.jpg)
![Page 6: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/6.jpg)
Highly-cited contradicted findings in early randomized trials
• Vitamin E and cardiovascular mortality (two large prospective cohorts, but also one large trial of 2,002 subjects claimed large decreases in mortality)
• Hormone replacement therapy and coronary artery disease (major benefits claimed by the Nurses’ Health Study, but also by small trials)
• A well-conducted randomized trial suggested that the monoclonal antibody HA-1A halves mortality from gram(-) sepsis; no effect was seen in a 10-times larger RCT
![Page 7: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/7.jpg)
Overall credibility
• Depends on the pre-evidence odds
• Depends on the data (the study at hand)
• Depends on bias
• Depends on the field
• All of these may depend on each other
![Page 8: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/8.jpg)
Simple model: no bias, one team of researchers
![Page 9: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/9.jpg)
Bias present
![Page 10: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/10.jpg)
Many teams of researchers
![Page 11: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/11.jpg)
![Page 12: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/12.jpg)
Illustrative PPV for clinical research designsIoannidis. Why most published research findings are false? PLoS Medicine 2005
Positive predictive value (PPV) of research findings for various combinations of power (1-β),
ratio of true to no relationships (R) and bias (u)
1-β R u Practical example PPV
0.80 1:1 0.10 Adequately powered RCT with little bias and 1:1 pre-study odds .85
0.95 2:1 0.30 Confirmatory meta-analysis of good quality RCTs .85
0.80 1:3 0.40 Meta-analysis of small inconclusive studies .41
0.20 1:5 0.20 Underpowered, phase I/II well-performed RCT .23
0.20 1:5 0.80 Underpowered, phase I/II poorly performed RCT .17
0.80 1:10 0.30 Adequately powered, exploratory epidemiological study .20
0.20 1:10 0.30 Underpowered, exploratory epidemiological study .12
0.20 1:1000 0.80 Discovery-oriented exploratory research with massive testing .0010
0.20 1:1000 0.20 As above, but with more limited bias (more standardized) .0015
![Page 13: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/13.jpg)
Post-study odds of a true finding are small
• When effect sizes are small
• When studies are small
• When field are “hot” (many furtively competitively teams work on them)
• When there is strong interest in the results
• When databases are large
• When analyses are more flexible Ioannidis JP. PLoS Medicine 2005
![Page 14: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/14.jpg)
A research finding cannot reach credibility over 50% unless
u<R
i.e., bias must be less than the pre-study odds
![Page 15: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/15.jpg)
“Quality” of studies
• Early empirical evaluations suggested that effect sizes may depend on aggregate quality scores; this has been dismissed, since there are so many quality scores, that inferences are widely different
• Other empirical evaluations suggested that specific quality items such as lack of blinding and lack of allocation concealment in RCTs may inflate treatment effects (e.g. Shultz et al. JAMA 1995)
• Now it seems more likely that such quality deficits may be associated either with inflated or with deflated treatment effects (e.g. Balk et al. JAMA 2002)
![Page 16: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/16.jpg)
“Averaging” quality is wrong
• A randomized trial with one major flaw may get the wrong answer
• A randomized trial with two major flaws may get an even more wrong answer or may paradoxically get a somehow more correct answer
• Flaws do not cancel out of course, and they may even have multiplicative detrimental effects
![Page 17: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/17.jpg)
The two kinds of bad quality
• Quality is bad on (evil) purpose = the effect sizes are almost always inflated
• Quality is bad because of stupidity = the effect sizes may be anything; usually, but not always, they are deflated
![Page 18: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/18.jpg)
Potential conflicts
Patsopoulos et al. BMJ 2006
![Page 19: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/19.jpg)
Ioannidis PLoS Clinical Trials 2006 and Clinical Trials 2007
![Page 20: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/20.jpg)
Exploratory test for significance chasing
![Page 21: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/21.jpg)
Spurious claims of subgroups
Rothwell P. Lancet 2005
![Page 22: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/22.jpg)
Month of birth and benefit from endarterectomy
![Page 23: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/23.jpg)
Time lag: bad news take longer to appear
Time from opening to enrollment (years)
1086420
Pro
babi
lity
not b
eing
pub
lishe
d
1.0
.8
.6
.4
.2
0.0
Ioannidis JP. JAMA 1998
![Page 24: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/24.jpg)
… even though they are obtained as fast..
Time from opening to enrollment (years)
76543210
Pro
babi
lity
of c
ontin
ued
follo
w-u
p
1.0
.8
.6
.4
.2
0.0
![Page 25: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/25.jpg)
…but publication is delayed
Time from completing follow-up (years)
76543210
Pro
babi
lity
not b
eing
pub
lishe
d
1.0
.8
.6
.4
.2
0.0
![Page 26: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/26.jpg)
Trial registration
• Upfront study registration has been adopted for randomized clinical trials, as a means for minimizing publication and reporting biases and maximizing transparency
• This is an extremely important step forward.• Still many trials are not registered and also among
those that are registered there is room for eventual selective reporting of outcomes and analyses
• Even with transparent and complete reporting there is room for biases that act before the level of study design
![Page 27: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/27.jpg)
Biases that precede the study design
• Setting the wider research agenda• Poor scientific relevance• Poor clinical utility• Poor consideration of prior evidence• Non-consideration of prior evidence• Biased consideration of prior evidence• Consideration of biased prior evidence• Setting the specific research agenda• Straw man effects• Avoidance of head-to-head comparisons• Head-to-head comparisons bypassing demonstration of effectiveness• Overpowered studies• Unilateral aims• Benefits versus harms • Research as bulk advertisement• Ghost management of the literature
![Page 28: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/28.jpg)
Clinical trials and burden of disease in sub-Saharan Africa
![Page 29: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/29.jpg)
Geometry of treatment networks
![Page 30: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/30.jpg)
Inflated effects with early stopping
Pocock et al. Clinical Trials 1989
![Page 31: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/31.jpg)
Biases after study completion• Interpretation biases for the single study• Bias related to metric selection• OR vs. RR• Absolute versus relative effects• P-values versus effect sizes• Selective discussion of results• Selective invocation of external evidence• Silencing of limitations• Inappropriate generalization• Interpretation biases in the wider scientific field• Publication bias• Time lag bias• Selective outcome and analysis reporting bias• Bias related to metrics of effect• Ghost management of the literature• Scientific citation bias• Skewed public dissemination• Resistance to independent replication
![Page 32: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/32.jpg)
Correct, but unilateral = false evidenceNeglecting harms
• Among 375,143 entries in the Cochrane Central Register of Controlled Trials, the search terms harm OR harms yielded 337 references
• Compare: 55,374 retrieved using efficacy and 23,415 retrieved using safety
• Of the 337, excluding several cases articles on self-harm or harm-reduction (an efficacy-equivalent term): only 3 trial reports and 2 abstracts had these words in their titles
• Of the 3 trial reports, one started with the clause “more good than harm”
• The other two actually focused on the harms of the placebo arm
![Page 33: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/33.jpg)
Harms
• An intervention is usually considered safe unless proven otherwise
• It may be more appropriate to consider an intervention potentially harmful until proven otherwise
![Page 34: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/34.jpg)
Reporting of harms in RCTs is neglected
• The space allocated to harms in the Results section is typically the same or smaller than the space allocated to the author names and affiliations
• Ioannidis and Lau, JAMA 2001
![Page 35: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/35.jpg)
SAFETY PARAMETERS
Withdrawal reasonsLaboratoryClinical
Pe
rce
nta
ge
of t
rials
with
ad
eq
ua
te r
ep
ort
ing
100
80
60
40
20
0
SUBJECT
AMI
Arthritis
H. pylori
HIV
Hypertension
SDGIT
Sinusitis
![Page 36: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/36.jpg)
Emphasis on harms is often further limited…
• When no dose comparison is involved
• When a trial appears in a high-impact factor journal
• When there is a prior indication for the intervention
• When the trial shows significant results for efficacy
![Page 37: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/37.jpg)
Reporting of harms is worse for NP than for pharmacological interventions
![Page 38: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/38.jpg)
Mental health trials- no harms recorded for any NP trials
![Page 39: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/39.jpg)
Large-scale evidence is very sparse
![Page 40: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/40.jpg)
Integration after the fact is not easy
![Page 41: Why some/many (all?) published clinical trials are false John P.A. Ioannidis Professor and Chairman, Department of Hygiene and Epidemiology, University.](https://reader033.fdocuments.net/reader033/viewer/2022042703/56649e205503460f94b0b0a7/html5/thumbnails/41.jpg)
Concluding comments
• Randomized controlled trials are a brilliant, simple design with solid history of successful utilization in clinical research
• They can offer extremely useful evidence and they are a must for documenting the effectiveness of proposed interventions
• This does not mean that they cannot suffer from important major biases.
• Caveat lector