Advanced topics in meta-analysis
description
Transcript of Advanced topics in meta-analysis
![Page 1: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/1.jpg)
1
Wim Van den Noortgate Katholieke Universiteit Leuven, Belgium
Belgian Campbell [email protected]
Workshop systematic reviews Leuven June 4-6, 2012
Advanced topics in meta-analysis
![Page 2: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/2.jpg)
2
1. Modelling heterogeneity2. Publication bias
Content
![Page 3: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/3.jpg)
3
1. Modelling heterogeneity
![Page 4: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/4.jpg)
4
Growing popularity of evidence-based thinking:
Decisions in practice and policy should be based on scientific research about the effects of these decisions/interventions
But: conflicting results (failures to replicate), especially in social sciences!
![Page 5: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/5.jpg)
5
1. The role of chance- in measuring variables- in sampling study participants
2. Study results may be systematically biased due to
- the way variables are measured- the way the study is set up
3. Studies differ from each other (e.g., in the kind of treatment, the duration of treatment, the dependent variable, the characteristics of the investigated population, …)
Explanation for failures to replicate?
![Page 6: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/6.jpg)
6
Differences between observed effect sizes due to chance only
Population effect sizes all equal
Fixed effects model
1 2( ... )k
![Page 7: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/7.jpg)
7
0 :H
Assessing heterogeneity
2
1
ˆ( )k
j jj
Q w g
2( 1)k
1 2 ... k :aH at least one differs from an otherj
0 : ~H Q
![Page 8: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/8.jpg)
8
Measuring heterogeneity( )² *100%Q dfIQ
Rough guidelines:0% to 40%: might not be important 30% to 60%: may represent moderate heterogeneity 50% to 90%: may represent substantial heterogeneity 75% to 100%: considerable heterogeneity
Interpretation based on both I² and heterogeneity test!
= percentage of variability in effect estimates due to heterogeneity rather than chance
![Page 9: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/9.jpg)
9
An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)
StudyWeeks prior
contact gj
1.2.3.4.5.6.7.8.9.
10.11.12.13.14.15.16.17.18.19.
Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)
2330033301001233123
0.030.12
-0.141.180.26
-0.06-0.02-0.320.270.800.540.18
-0.020.23
-0.18-0.060.300.07
-0.07
0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17
( )jg
![Page 10: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/10.jpg)
10Q = 35,83, df = 18, I²= 50 %, p = .007
![Page 11: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/11.jpg)
11
Not always wise: make set of studies more homogeneous!
Can help to say something about ‘fruit’ Can help to make detailed conclusions:
Does the effect depend on the kind of fruit?
Mixing apples and oranges?
![Page 12: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/12.jpg)
12
![Page 13: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/13.jpg)
13
Population effect size possibly depends on study category
Differences between observed effect sizes within the same category due to chance only
Fixed effects model with categorical moderator variable
![Page 14: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/14.jpg)
14
An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)
StudyWeeks prior
contact gj
1.2.3.4.5.6.7.8.9.
10.11.12.13.14.15.16.17.18.19.
Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)
2330033301001233123
0.030.12
-0.141.180.26
-0.06-0.02-0.320.270.800.540.18
-0.020.23
-0.18-0.060.300.07
-0.07
0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17
( )jg
![Page 15: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/15.jpg)
15
![Page 16: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/16.jpg)
16
Total variabilityin observed ES’s
A weighted ANOVA
T B WQ Q Q
Variability within groups
Variabilitybetween groups= +
H0: QT ~²k-1
H0: QB ~²J-1
H0: QW ~²k-J
2
1
ˆ( )k
j jj
Q w g
QT : homogeneity test
QB : moderator test
QW : test for within group homogeneity
![Page 17: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/17.jpg)
17
Q total = Q Between + Q within
² 35.83 20.38 15.45
df 18 3 15
p 0.007 0.0001 0.42
![Page 18: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/18.jpg)
18
A second example(using a sorted caterpillar plot)
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Observed effect sizes for the 3 tasks
Semantic categorization Lexical decision Naming
ES
= Mean ES REM
![Page 19: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/19.jpg)
19
Population effect size possibly depends on continuous study characteristic
e.g.,
After taking into account this study characteristic, differences between observed effect sizes due to chance only
Fixed effects model with continuous moderator variable
0 1 1 ... j j p pjx x
![Page 20: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/20.jpg)
20
Initial effect is moderate (0.41, p < .001), but decreases with increasing prior contact (with -0.16 per week, p <.001)
Conclusions for Raudenbush (1984):
![Page 21: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/21.jpg)
21
Population effect size possibly varies randomly over studies
Differences between observed effect sizes are due to- chance- ‘true’ differences
Random effects model
![Page 22: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/22.jpg)
22
![Page 23: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/23.jpg)
23
Population effect size possibly depends on study category
Differences between observed effect sizes within the same category are due to- chance- ‘true’ differences
Random effects model with categorical moderator variable
![Page 24: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/24.jpg)
24
Population effect size possibly depends on continuous study characteristic
e.g.,
After taking into account this study characteristics, differences between observed effect sizes are due to- chance- ‘true’ differences
Random effects model with continuous moderator variable
0 1 1 ... j j p pj jx x u
![Page 25: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/25.jpg)
25
Random effects model with moderators:
◦ The least restrictive model: allows moderator variables & random variation
◦ Also called a ‘Mixed effects model’
![Page 26: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/26.jpg)
26
FEM REM
Without moderator
Categorical moderator
Continuous moderator
![Page 27: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/27.jpg)
27
1. Is there an overall effect?2. How large is this effect?3. Is the effect the same in all studies?4. How large is the variation over studies?5. Is this variation related to study
characteristics?6. Is there variation that remains unexplained?7. What is the effect in the specific studies?
Basic meta-analytic questions
![Page 28: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/28.jpg)
28
An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)
StudyWeeks prior
contact gj
1.2.3.4.5.6.7.8.9.
10.11.12.13.14.15.16.17.18.19.
Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)
2330033301001233123
0.030.12
-0.141.180.26
-0.06-0.02-0.320.270.800.540.18
-0.020.23
-0.18-0.060.300.07
-0.07
0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17
( )jg
![Page 29: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/29.jpg)
29
Parameter REM
Fixed
Intercept 0.084 (0.052)
Between study variance 0.019 (0.023)
0
2u
![Page 30: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/30.jpg)
30
Parameter REM MEM
Fixed
Intercept 0.084 (0.052) 0.41 (0.087)
Weeks -0.16 (0.036)
Between study variance 0.019 (0.023) 0.00 (-)
0
1
2u
![Page 31: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/31.jpg)
31
1. Models can include multiple moderators2. REM assumes randomly sampled studies3. REM requires enough studies4. Association (over studies) ≠ causation!
Be aware of potential confounding moderators (studies are not ‘RCT participants’!)
Remarks
![Page 32: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/32.jpg)
Dependencies between studies◦ E.g., research group, country, …
Multiple effect sizes per study◦ Several samples◦ Same sample but, e.g., several indicator variables
32
Note: Sources of dependencies
![Page 33: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/33.jpg)
Ignoring dependence? NO! Avoiding dependence
◦ (Randomly choosing one ES for each study)◦ Averaging ES’s within a study◦ Performing separate meta-analyses for each kind
of treatment or indicator Modelling dependence
◦ Performing a multivariate meta-analysis, accounting for sampling covariance.
◦ Performing a three level analysis
33
How to account for dependence?
![Page 34: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/34.jpg)
34
2. Publication bias
![Page 35: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/35.jpg)
35
Example 1: advanced ovarian cancer: monotherapy alkylating agent vs. combination chemotherapy? International Cancer Research Data Bank
(Egger, M. D., & Smith, G. (1998). Meta-analysis. Bias in location and selection of studies. British Medical Journal, 316, 61-66. http://www.bmj.com/cgi/content/full/316/7124/61).
![Page 36: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/36.jpg)
36
Proportion of publication within 5 years after conference:
81 % (of 233 trials) for significant results 68 % (of 287 trials) for nonsignificant results
Example 2: 510 large trials presented at conferences of the American Society of Clinical Oncology (ASCO)
(Kryzanowska, M. K., Pintilie, M., & Tennock, I. F. (2003). Factors associated with failure to publish large randomized trials presented at an oncology meeting. Journal of the American Medical Association, 290, 495-501).
![Page 37: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/37.jpg)
37
-1 -0.5 0 0.5 1 1.50
100
200
300
400
500
Observed effect sizes
Sam
ple
size
The funnel plot
![Page 38: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/38.jpg)
38
-1 -0.5 0 0.5 1 1.50
100
200
300
400
500
Observed effect sizes
Sam
ple
size
Example: Raudenbush, 1984
![Page 39: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/39.jpg)
39
Þ Thorough search for all relevant published and unpublished study results
a) Articlesb) Booksc) Conference papersd) Dissertationse) (Un)finished research reportsf) …
![Page 40: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/40.jpg)
40
- outliers- detection using graphs (or tests)- conduct analysis with and without outliers
- calculation effect sizes : several analyses- publication bias: analysis with and without
unpublished results- design & quality: compare results from studies
with strong design or good quality, with those of all studies
- researcher: literature search, effect size calculation, coding quality, …, done by two researchers
- …
Note: sensitivity analysis: how robust are conclusions?
![Page 41: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/41.jpg)
41
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Observed effect sizes
Experiment
ES
Outliers
![Page 42: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/42.jpg)
42
Spreadsheets (e.g., MS Excel, …) Some general statistical software (note:
often not possible to fix the sampling variance)SAS Proc Mixed, Splus, R Metafor package, …
Software for meta-analysis (note: often not MEM; often only one moderator!)CMA (http://www.meta-analysis.com/), RevMan, …
Software for multilevel/mixed modelsHLM, MLwiN, …
Software for MA
![Page 43: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/43.jpg)
43
Software Excel SAS R CMA RevManCalculation of effect sizes X X √ √√ XNumber of moderators X ∞ ∞ 1 1 (cat.)Funnel X X √ √ √Trim & Fill X X √ √ XForest X X √ √ √Max. nr of levels 2 ∞ 2 2 2Flexibility √ √√ √√ X XPrice Expensive Free Expensive Free
(but student (but limitedversion) trial vers.)
Complexity √ X X √√ √√
![Page 44: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/44.jpg)
44
Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.) (2009). The handbook of research synthesis and meta-analysis. New York: The Russell Sage Foundation.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-
analysis. Thousand Oaks, CA: Sage.
Van den Noortgate, W., & Onghena, P. (2005). Meta-analysis. In B. S. Everittt, & D. C. Howell (Eds), Encyclopedia of Statistics in Behavioral Science (Vol. 3 pp. 1206-1217). Chichester, UK: John Wiley & Sons.
Recommended literature:
![Page 45: Advanced topics in meta-analysis](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681635c550346895dd42533/html5/thumbnails/45.jpg)
45
Site of David Wilsonhttp://mason.gmu.edu/~dwilsonb/ma.html
Site of William Shadishfaculty.ucmerced.edu/wshadish/
Recommended sites