Meta-Analysis Concepts and...
Transcript of Meta-Analysis Concepts and...
Meta-AnalysisConcepts and Applications
Michael BorensteinHannah Rothstein
Table of ContentsIntroduction Slides 3- 15Goals Slides 16-17Effect Sizes Slides 18-67Fixed Effect Computations Slides 68-110Fixed Effects vs. Random
Effects Slides 111-133
Acknowledgments
Development funded by NIH
• National Institute of Mental Health• National Institute on Aging• National Institute of Drug Abuse
What is the goal of a meta-analysis?
When effect is consistent
• Provide precise estimate of the effect• Report if it is robust across range of
populations
When the effect varies
• May be used to qualify the mean effect• May make the mean effect irrelevant• May be of more interest than the
combined effect• What factors may explain the variation
Why perform a meta-analysis?
Streptokinase
• Meta-analysis in 1977 could have been definitive
• Additional 40,000 patients randomized• Additional ???? Patients not treated• Even in 1992, narrative review was not
definitive• Without meta-analysis, studies could have
continued indefinitely
Forest plot
• Transparent• A mechanism for understanding the
statistics• A mechanism for communicating the
statistics
Goals of meta-analysis
Assigning weights
• Compute combined effect• Assess heterogeneity• Use to qualify combined effect• Focus on heterogeneity
Steps
• Show sample• Show effects• Show weights• Show combined effect• How compute effects• How compute weights• How compute combined effect
Steps
• Show heterogeneity• How compute heterogeneity• Statistical implications• Practical implications
Goals of meta-analysis
Assigning weights
• Compute combined effect• Assess heterogeneity• Use to qualify combined effect• Focus on heterogeneity
Computing effect sizeand variance
Effect size
Effect size AND precision
Reporting precision
• Standard error• Confidence interval• Variance
Precision
• In primary study, qualifies the effect size• In meta-analysis, is used to assign weight
to effect size
Continuous data
• Start with means and SD• Raw difference• Standardized mean difference (d)• Bias-corrected standard difference (G)
Means and SD’s
5020100Control
5020110Treated
NSDMean
Raw mean difference
• Natural scale• Well known scale• All studies on same scale
Raw mean difference
1 2MeanDifference Mean Mean= −
2 21 1 2 2
1 2
( 1) * ( 1) *2Pooled
N SD N SDSDN N
− + −=
+ −
1 2
1 1 *MeanDifference PooledSE SDN N
= +
Raw mean difference
110 100 10MeanDifference = − =
2 2(50 1) *10 (50 1) *10 2050 50 2PooledSD − + −
= =+ −
1 1 * 20 4.050 50MeanDifferenceSE = + =
Raw mean difference
• Effect size – difference in means• Precision – SD within group, N
Raw mean difference
Raw mean difference
Standardized mean difference
• Proprietary scales • Multiple scales
Standardized mean difference (d)
Within
MeanDifferencedSD
=
21 2
1 2
1/ 1/2 * ( )dN N dSE
N N+ +
=+
Standardized mean difference (d)
10 0.5020
d = =
21/ 50 1/ 50 0.5 .2032 * (50 50)dSE + +
= =+
Standardized mean difference
• Effect size – Difference in means relative to SD within groups
• Precision – Sample size
Standardized mean difference (d)
Standardized mean difference (d)
Bias-corrected d (Hedges g)
314 * 1
Jdf
= −−
*G d J=
*SE SEG D J=
Bias-corrected d (Hedges g)
31 .9924 * 98 1
J = − =−
0.500 * 0.992 0.496G = =
0.203 * 0.992 0.202SEG = =
Bias-corrected d (Hedges g)
Bias-corrected d (Hedges g)
Multiple indices
Other data types
• Correlation• Survival• Events by person/years• One-armed studies• Generic indices
Study design and precision
• Independent groups vs. matched designs• Effect size is the same• Precision is different• Can combine in analysis
Data format
• Back-compute effect size and variance• Test statistics or p-values• Confidence limits
Compute d from p-value
Compute d from p-value
Compute SEOdds ratio from CI
Compute SEOdds ratio from CI
Multiple data formats
Caveat
• These slides are meant as a general introduction.
• They do not deal with special cases such as empty cells.
• They do not address variations in computational formulas.
Binary data
• Start with 2x2 table• Odds ratio• Risk ratio• Risk difference
2 x 2 Table
N2DCControl
N1BATreated
Non-EventsEvents
2 x 2 Table
1008812Control
100928Treated
Non-EventsEvents
Log odds ratio
( ) ADLog OddsRatio LogBC
⎛ ⎞= ⎜ ⎟⎝ ⎠
( )1 1 1 1
Log OddsRatioSEA B C D
= + + +
Log odds ratio
8 * 88( ) 0.45092 *12
Log OddsRatio Log ⎛ ⎞= = −⎜ ⎟⎝ ⎠
( )1 1 1 1 0.4808 92 12 88Log OddsRatioSE = + + + =
Log odds ratio
Log odds ratio
Log risk ratio
1
2
/( )/
A NLog RiskRatio LogC N⎛ ⎞
= ⎜ ⎟⎝ ⎠
( )/ /
Log RiskRatioB A D CSEB A D C
= ++ +
Log risk ratio
8 /100( ) 0.40512 /100
Log RiskRatio Log ⎛ ⎞= = −⎜ ⎟⎝ ⎠
( )92 / 8 88 /12 0.43492 8 88 12Log RiskRatioSE = + =
+ +
Log risk ratio
Log risk ratio
Risk Difference
A CRDA B C D
= −+ +
1 1 2 2* *RD
P Q P QSEA B C D
⎛ ⎞= +⎜ ⎟+ +⎝ ⎠
//
i i i
i i i
p Events Nq NonEvents N
=
=
Risk Difference
8 12 0.0408 92 12 88
RD = − = −+ +
.08 * .92 .12 * .88 0.4348 92 12 88RDSE ⎛ ⎞= + =⎜ ⎟+ +⎝ ⎠
Risk Difference
Risk Difference
Multiple indices
Fixed effect computations
Assigning weights
• To get most precise effect• To give more weight to the more precise
studies
Assign weight to each study
• Weight by 1/variance, or the “Inverse variance”
1i
i
wv
=wi=Study weight
vi=Study variance
Combined mean
1
1
ˆ
k
i ii
k
ii
w y
wθ =
=
=∑
∑
wi=Study weight
yi=Study mean
Variance of combined mean
1
1ˆ( ) k
ii
Varw
θ
=
=
∑wi=Study weight
Var(θ)=Variance of combined mean
Test of the null
0ˆ( )
ˆ( )Z
Var
θ θ
θ
−=
Example using Excel
Enter the summary data
Compute effect size and variancefor each study
Assign weight to each study
Compute combined effect
Variance of combined effect
Same example in CMA
Enter summary data
Compute effect size
Display formula
Combined effect and variance
Weights
Combined effect and variance
Combined effect and variance
More information leads to greater precision
Increase the N within studies
N=50 per group
N=100 per group
Increase the number of studies
Number studies = 3
Number studies = 6
More precise studies are given more weight
Same N for each study
Same N for each study
Same N for each study
N varies by study
N varies by study
N varies by study
Effect size pulled by larger study
d moved from .48 to .66
Relative weights in forest plot
Study name Std diff in means and 95% CIStd diff Standard
in means error
A 0.400 0.202B 0.250 0.201C 0.800 0.208
0.476 0.117
-2.00 -1.00 0.00 1.00 2.00
Favours A Favours B
Example 01
Meta Analysis
Study name Std diff in means and 95% CIStd diff Standard
in means error
A 0.400 0.202B 0.250 0.201C 0.800 0.093
0.658 0.078
-2.00 -1.00 0.00 1.00 2.00
Favours A Favours B
Example 02
Meta Analysis
References
• Hedges and Olkin• Lipsey and Wilson
Files available by e-mail
• Standardized difference. xls• Standardized difference. cma
Fixed effect vs.Random effects
Fixed vs. Random
• Concept• Definition• How weights affect
– Combined value– Confidence interval width
• Which should we use
Concept
• Fixed effect model– Common population– Effect size varies only because of random
error• Random effects model
– Multiple populations– Effect size will vary because of random error– Effect size will vary because of true variation
Definition of combined effect
• Fixed effect model– There is one true effect.– Combined effect is estimate of this value.
• Random effects model– There are a series of effects.– Combined effect is average of a series of
values.
Fixed vs. Random
i i iT µ ξ ε= + +
i iT µ ε= +
Factors affecting Tau-squared
When Tau2 is zero
• Random effects model reduces to the fixed effect model.
Weights
• Fixed effect– One true effect– All variation is random error– Largely ignore the smaller studies
• Random effects– Range of effects– Each study provides information about a different
population– Cannot ignore small studies, nor give too much
weight to large studies
Fixed effect model
Within-Study Error
Total Variance
+ =
Random effects model
Within-Study Error
Total Variance
+ =Between-study
variance
Extreme effect in large study
Extreme effect in small study
Study name Statistics for each study Odds ratio and 95% CI
Odds Lower Upper ratio limit limit
Morton 0.436 0.038 5.022Rasmussen 0.348 0.154 0.783Smith 0.278 0.057 1.357Abraham 0.957 0.058 15.773Feldstedt 1.250 0.479 3.261Shechter-89 0.090 0.011 0.736Ceremuzynski 0.278 0.027 2.883Berschat 0.304 0.012 7.880Singh 0.499 0.174 1.426Pereira 0.110 0.012 0.967Golf 0.427 0.127 1.436Thogersen 0.452 0.133 1.543LIMIT-2 0.741 0.556 0.988Shechter-95 0.208 0.067 0.640ISIS-4 1.059 0.996 1.127MAGIC 1.003 0.873 1.152
0.712 0.564 0.9000.01 0.1 1 10 100
Favours A Favours B
Magnesium Fixed effect
Meta Analysis
Study name Statistics for each study Odds ratio and 95% CI
Odds Lower Upper ratio limit limit
Morton 0.436 0.038 5.022Rasmussen 0.348 0.154 0.783Smith 0.278 0.057 1.357Abraham 0.957 0.058 15.773Feldstedt 1.250 0.479 3.261Shechter-89 0.090 0.011 0.736Ceremuzynski 0.278 0.027 2.883Berschat 0.304 0.012 7.880Singh 0.499 0.174 1.426Pereira 0.110 0.012 0.967Golf 0.427 0.127 1.436Thogersen 0.452 0.133 1.543LIMIT-2 0.741 0.556 0.988Shechter-95 0.208 0.067 0.640ISIS-4 1.059 0.996 1.127MAGIC 1.003 0.873 1.152
1.016 0.961 1.0730.01 0.1 1 10 100
Favours A Favours B
Magnesium Fixed effect
Meta Analysis
Key idea
• Relative weights assigned under random effects will be more balanced than those assigned under fixed effects.
• As we move from fixed effect to random effects, extreme studies will lose influence if they are large, and will gain influence if they are small.
Confidence interval width
• Both models include within-study variance.• Random effects model includes also
between-study variance.• Therefore, the confidence interval for the
random effects model will always be as wide or wider than for the fixed effect model.
Study name Std diff in means and 95% CI
Std diff Standard in means error
A 0.400 0.001B 0.400 0.001C 0.400 0.001D 0.400 0.001E 0.400 0.001
0.400 0.000
-1.00 -0.50 0.00 0.50 1.00
Favours A Favours B
Fixed effect model with huge N
Meta Analysis
Study name Std diff in means and 95% CI
Std diff Standard in means error
A 0.400 0.001B 0.450 0.001C 0.350 0.001D 0.450 0.001E 0.350 0.001
0.400 0.022
-1.00 -0.50 0.00 0.50 1.00
Favours A Favours B
Random effects model with huge N
Meta Analysis
Which model should we use?
Fixed effect
• If there is reason to believe that all the studies are functionally identical
• Our goal is to compute the common effect size, which would then be generalized to other examples of this same population.
• Example, of drug company has run five studies to assess the effect of a drug.
Random effects
• When not likely that all the studies were functionally equivalent.
• When the goal of this analysis is to generalize to a range of populations.
Choice of model should not be based on significance test
• Practical issue– Type-II error
• Fundamental issue– The difference between fixed and random
effects is really conceptual
Common (incorrect) wisdom about significance tests
• “Significance test for the effect size will always be more significant using the fixed effect model” rather than the random effects model.
• Is not true• In any event, should never be a factor in
selecting a computational model.
Criticisms of meta-analysis
Meta-Analysis – Concepts and Applications
SCT Orlando May 21, 2006Michael Borenstein and Julian Higgins
Meta-AnalysisConcepts and Applications
Michael Borenstein and Julian HigginsWorkshops Chairman Domenic RedaSCT Orlando May 21, 2006Additional materials available at www.Meta-
Analysis.comQuestions to [email protected]