Practical cloud adoption for the health & life sciences industry
Practical Meta-Analysis for the Social Sciences
description
Transcript of Practical Meta-Analysis for the Social Sciences
![Page 1: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/1.jpg)
Practical Meta-Analysis for the Social Sciences
Evan J. RingquistSchool of Public and Environmental Affairs
Indiana UniversityWorkshop in Methods Presentation
January 11, 2013Bloomington, IN 47405
1
![Page 2: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/2.jpg)
2
“Meta-Analysis is not a fad. It is rooted in the fundamental values of the scientific enterprise: replicability, quantification, causal and correlational analysis. Valuable information is needlessly scattered in individual studies. The ability of social scientists to deliver generalizable answers to basic questions of policy is too serious a concern to allow us to treat research integration lightly. The potential benefits of meta-analysis method seem enormous.” (Bangert-Drowns 1986: 398)
![Page 3: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/3.jpg)
Meta-Analysis Defined• Meta-analysis is a systematic, quantitative,
replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies.
• Meta-analysis, then, provides a powerful set of tools for aggregating knowledge.
3
![Page 4: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/4.jpg)
Number of Articles Published Each Year in the “Meta-Analysis” Topic Category as Referenced by
the Social Sciences Citation Index
419801981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011
0
200
400
600
800
1000
1200
1400
artic
les p
er y
ear
![Page 5: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/5.jpg)
Outline of Presentation1. Motivation for Using Meta-Analysis in the
Social Sciences2. Quick Introduction to Meta-Analysis3. Meta-Regression Models: The Basics4. Meta-Regression Analysis for the Social
Sciences– CRVE, GEE, and if we have time, HLM
5
![Page 6: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/6.jpg)
Section I: Motivation
6
![Page 7: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/7.jpg)
Cumulative Knowledge and the Scientific Enterprise
• A Central Goal of Science is the Accumulation and Aggregation of Knowledge. Yet the Social Sciences in General and Researchers in Public Management and Policy in Particular Have Found Cumulative Knowledge to be an Elusive Goal.
7
![Page 8: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/8.jpg)
“Cumulative Knowledge” in Environmental Justice
• “Overwhelming evidence” of significant environmental inequities (Goldman 1993; see also Mohai and Bryant 1994)
• “even a reasonably generous reading of the empirical research alleging environment inequity . . . must leave room for profound skepticism” (Foreman 1998; see also Bowen 2001) 8
![Page 9: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/9.jpg)
“Cumulative Knowledge” in Education Policy
• “Extensive research has been conducted on the academic success of students enrolled in school choice program nationwide. Rigorous studies show strong gains for voucher and scholarship tax credit recipients . . .” (Alliance for School Choice, 2010)
• “The reason that vouchers had subsided as a point of advocacy is because they don’t work.” (Randi Weingarten, President, American Federation of Teachers; Associated Press 2011).
• “Few contemporary questions in American education have produced such wide-spread controversy as that regarding the potential of school vouchers to reduce inequality in student outcomes. . . these efforts notwithstanding, answers to the voucher question still appear uncertain.” (Cowen 2008)
9
![Page 10: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/10.jpg)
Cumulative Knowledge in Public Management
• the more knowledge we create and diffuse, “the less we know, because our research is not aggregated or accumulated into substantial bodies of knowledge” (Van Slyke, O’Leary, and Kim 2010: 290.
• The first step in making Public Administration a stronger and more robust field is the need to aggregate knowledge in the sense of making it cumulative. (Rosenbloom 2010)
10
![Page 11: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/11.jpg)
Cumulative Knowledge in Policy Studies
• “political scientists have spent (almost literally) countless articles and books proposing something like ‘laws’ or theories that, taken collectively . . . have produced infinitely more confusion than clarity” (DeLeon 1998: 150).
11
![Page 12: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/12.jpg)
Cumulative Knowledge in Policy Evaluation
• Social scientists identify new and important questions having policy relevance. Initial studies examining these questions provide clear answers regarding, for example, the effectiveness of social policy interventions. Subsequent studies cast doubt on these initial conclusions, however, and with the proliferation of studies comes a proliferation of conclusions. The best efforts of social and behavioral researchers generate confusion and uncertainty rather than clarity. In the end, researchers conclude that the phenomena being studied are ‘hopelessly complex’ and move on to other questions. After several repetitions of this cycle, social and behavioral scientists themselves become cynical about their own work and express doubts about whether behavioral and social science in general is capable of generating cumulative knowledge or answers to socially important questions (Hunter and Schmidt 1996: 325-6).
12
![Page 13: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/13.jpg)
Why Do Social Scientists Struggle with Knowledge Accumulation?
1. Epistemological Differences: Social Sciences ought not aim to generate generalizable “scientific” knowledge ala Chemistry.
2. Subject Matter Differences: Social scientists study organizations and institutions devised and populated by strategic actors, not molecules or organizms without agency.
3. Social Scientists Use The Wrong Tools for Aggregating Knowledge. 13
![Page 14: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/14.jpg)
Section II: Introduction to Meta-Analysis
14
![Page 15: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/15.jpg)
Meta-Analysis Defined (again)• Meta-analysis is a systematic, quantitative,
replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies.
• Meta-analysis, then, provides a powerful set of tools for aggregating knowledge.
15
![Page 16: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/16.jpg)
The Language of Meta-Analysis• Original Study: A piece of original
research, published or unpublished, that aims to test a hypothesis and/or estimate a quantity of interest.
• Focal Predictor: The independent variable in an original study that measures the key exogenous concept associated with the research question of interest. 16
![Page 17: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/17.jpg)
The Language of Meta-Analysis• Effect Size: A standardized measure of the
relationship between the focal predictor and the dependent variable in an original study. Effect sizes are the unit of analysis and the quantity of interest in a meta-analysis, often designated Θi. Without effect sizes there can be no meta-analysis.
17
![Page 18: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/18.jpg)
The Language of Meta-Analysis• Effect Size Variance: A measure of the
uncertainty associated with a particular effect size.
• Fixed and Random Effects Models: Methods of weighting effect sizes that embody different assumptions about the effect size variance.
18
![Page 19: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/19.jpg)
How Do We Calculate Effect Sizes?
• Three Families of Effect Sizes1. D-based effect sizes (standardized mean
differences, common in education and psychology)
2. Odds-based effect sizes (e.g. log odds of an event, common in medicine)
3. R-based effect sizes (partial correlation coefficients, unusual in the literature but most useful for social scientists) 19
![Page 20: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/20.jpg)
Calculating R-based Effect Sizes from Original Studies
• r = √[t2 / (t2 + df)] [1]• r = √[Z2 / n] [2]• r = √[Χ2
1 / n] [3]
• V[r] = (1-r2)2 / (n-1) [4]
20
![Page 21: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/21.jpg)
Fisher’s Corrections for R-based Effect Sizes and Variances
• Zr = 0.5 ln[(1+r) / (1-r)] [5]
• V[Zr] = 1/(n-3) [6]
21
![Page 22: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/22.jpg)
Example: Are Pollution Emissions Higher in Black Neighborhoods?
• Ringquist 1997: t=3.06, df=29202– r=.02– Zr=.02– V[Zr] = .000034
• Downey 1998: t=3.38, df=112– r=.30– Zr=.31– V[Zr] = .0072
22
![Page 23: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/23.jpg)
What Can We Do With Effect Sizes Statistically?
1. Communicate results from original research in a more meaningful fashion (not meta-analysis).
2. Combine (average) effect sizes to estimate the population effect size
3. Test null hypothesis that population effect size equals zero
4. Explain variation in effect sizes across original studies. We use “Meta-Regression” to account for this variation. 23
![Page 24: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/24.jpg)
Combining Effect Sizes• We calculate average effect sizes, or our
estimate of the population effect size, by calculating a weighted average of all effect sizes from original studies where the weights are inverse variances
• Θbar= ΣwiΘi / Σwi [7]• Wi = 1/vi
24
![Page 25: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/25.jpg)
Combining Effect Sizes: Fixed Effects Models
• Fixed Effects models assume that effect sizes vary across original studies only due to sampling error:– Θi = Θ + ei [8], so– E[Θi]= Θ [9], and– ei ~ N(0,vi) [10]
25
![Page 26: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/26.jpg)
Combining Effect Sizes: Fixed Effects Models
• Fixed effects meta-regression assumes that effect sizes conditional upon moderator variables differ only due to sample size
• Fixed effects estimates apply only to sample of original studies in hand
26
![Page 27: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/27.jpg)
Combining Effect Sizes: Random Effects Models
• Random Effects models assume that effect sizes are normally distributed random variables– Θ ~ N(µΘ, τ2) [11]– Θi = µΘ + ei [12]– E[Θi] = µΘ [13], and– ei ~ N(0,vi + τ2) [14]– Always use random effects in the social
sciences!27
![Page 28: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/28.jpg)
Estimating the Random Effects Variance Component
• Using Restricted Maximum Likelihood– Le(τ2) = -.5 * Σ [ln(vi + τ2) + ((ei
2 / (vi + τ2) - .5 * ln |X’v-1X| [15]
• Using Method of Moments (MOM)– tr(M) = Σvi
-1 – tr [(Σvi-1 XX’)-1 (Σvi
-2 XX’)] [16]
• Using MOM Approximation– τ2 = [SSEols / (m-k-1)] – vbar [17]
28
![Page 29: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/29.jpg)
What Can We Do With Effect Sizes Substantively?
• Estimate severity of problems– E.g., environmental inequities– E.g., effects of climate change
• Measure important quantities of interest– E.g., statistical value of a life– E.g., hedonic pricing of environmental
amenities
29
![Page 30: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/30.jpg)
What Can We Do With Effect Sizes Substantively?
• Program Evaluation– E.g., effectiveness of educational vouchers– E.g., effectiveness of job training programs
• Theory Testing and Development– E.g., Top-down vs. bottom-up implementation– E.g., effects of negative campaign ads– E.g., Ricardian equivalence
30
![Page 31: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/31.jpg)
Section III: Introduction to Meta-Regression
31
![Page 32: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/32.jpg)
“exploration of the pattern of variation in effect sizes among studies is a far more important goal of meta-analysis than the
construction of powerful tests of null hypotheses.” (Osenberg et al. 1999: 1105)
32
![Page 33: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/33.jpg)
Moving Meta-Analysis to the Social Sciences
• Techniques for calculating and combining effect sizes developed for synthesizing the results from experiments. Can these same techniques really be used to synthesize results from the non-experimental multivariate models common in the social sciences?
33
![Page 34: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/34.jpg)
Three Critiques of Meta-Analysis in the Social Sciences
1. Parameter estimates from the general linear model are not comparable.
2. Effect sizes cannot be calculated from multivariate models.
3. Parameter estimates from different models estimate different population parameters, and therefore cannot be combined. 34
![Page 35: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/35.jpg)
Response to Critique #1:• While regression parameters are not
comparable, meta-analysis does not combine parameter estimates. Rather, meta-analysis combines effect sizes which are standardized with respect to scale. This is not a valid critique.
35
![Page 36: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/36.jpg)
Response to Critique #2• In fact, the same formulas that can be
used to calculate r-based effect sizes from experimental studies can be used to calculate partial correlations from regression models, probit models, etc. (see Greene 1993: 180)
36
![Page 37: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/37.jpg)
Response to Critique #2• Example 1: Panel regression model,
Y=continuous, K=33, N=307,538– Pcorr = .0099, Zr = .0096
• Example 2: Probit model, Y=dichotomous, K=8, N=14131– Pcorr = .1081, Zr = .1062
• Example 3: OLS regression model, Y=continuous, K=24, N=285– Pcorr = .1619, Zr = .1565
37
![Page 38: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/38.jpg)
Response to Critique #3• E.g., Study 1
– Y = b0 + b1X1 + b2X2 + b3X3 + e• E.g., Study 2
– Y = b0 + b1X1 + b2X2 + b4X4 + e
• E[b11] ≠ E[b12], β11 ≠ β12, and Θ1 ≠ Θ2 • Therefore dependent variable Θi
measures fundamentally different quantities
38
![Page 39: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/39.jpg)
Response to Critique #3• Critique #3 is a valid critique, and this is
why calculating average effect sizes is of little value in the social sciences – these average effect sizes combine estimates of different population parameters, and therefore are not valid estimates of any useful quantity of interest.
• Thankfully, we can address Critique #3 using Meta-Regression. 39
![Page 40: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/40.jpg)
What Do We Need to Conduct a Meta-Regression?
• Measure of effect size that is comparable across original studies employing different measures, models, and samples
• Measure of (un)certainty or effect size variance
• Moderator variables that account for variability in effect sizes
40
![Page 41: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/41.jpg)
Introduction to Meta-Regression
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– Θi is the effect size– X represents moderator variables accounting for
differences in effect sizes within and across studies• Scientifically Interesting Moderators (Rubin 1992)
– e.g., differences attributable to target characteristics– e.g., differences attributable to program or policy design
• Scientifically Uninteresting Moderators– e.g., differences attributable to estimation technique or
research design 41
![Page 42: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/42.jpg)
Number of Publications Using Phrase “Meta-Regression” in Google Scholar, 1990-2011
42
![Page 43: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/43.jpg)
Example: Synthesizing Research on Educational Vouchers
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– Θi is the estimated point biserial correlation (effect
size Zr) between the use of an educational voucher and student standardized test scores from a particular statistical model in a particular original study.
43
![Page 44: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/44.jpg)
Example: Educational Vouchers
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– X1 moderator variable identifying effect sizes from
models limiting sample to black children– X2 moderator variable identifying effect sizes from
voucher programs available to religious schools– X3 moderator variable identifying effect sizes from
models not using random assignment to treatment and control groups.
44
![Page 45: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/45.jpg)
Example: Educational Vouchers• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)
– b0 average effect of vouchers on student test scores for all students in experimental studies where vouchers use is limited to secular schools
– b1 differential effect of vouchers for black students– b2 differential effect from voucher programs that
include religious schools– b3 differential voucher effect from quasi-experimental
studies45
![Page 46: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/46.jpg)
46
Independent VariablesCRVE
Parameter Estimates (St.Err.)
CRVE 95% CIGEE
Parameter Estimates (St.Err.)
GEE 95% CI
Black Student 0.0296** 0.0030 0.0562 0.0370*** 0.0143 0.0598(0.0130) (0.0116)
Hispanic Student -0.0470*** -0.0680 -0.0259 -0.0413*** -0.0577 -0.0248(0.0103) (0.0084)
Religious Schools -0.0312 -0.0693 0.0069 0.0180 -0.0574 0.0213(0.0187) (0.0201)
Matched Control Groups -0.0097 -0.0226 0.0033 -0.0069* -0.0148 0.0009(0.0064) (0.0040)
Convenience Control Groups -0.0073 -0.0246 0.0101 -0.0082 -0.0243 0.0078(0.0085) (0.0082)
Pretest Included -0.0326*** -0.0560 -0.0092 -0.0228* -0.0464 0.0008(0.0115) (0.0120)
Intercept (Baseline Effect Size) 0.1097*** 0.0573 0.1620 0.0830*** 0.0243 0.1416(0.0257) (0.0299)
Adjusted R2a
Psuedo-R2a
I2a
Sample Size
.44
.68
25.01%
611
na
na
na
611
![Page 47: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/47.jpg)
Obtaining Estimates for Meta-Regression Models
• OLS Estimator of Meta-Regression Model– E[b] = β = (X’X)-1 X’Θ [18]– var[b] = σ2(X’X)-1 [19]
• But OLS is incorrect for two reasons:1. Θi is heteroskedastic (recall assumptions)2. OLS weights all observations equally
(unequal variances means unequal certainty) 47
![Page 48: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/48.jpg)
Generalized Least Squares in Meta-Regression
• In OLS, ee’ = σ2I • In GLS, ee’ = σ2Ω
– where Omega is a diagonal matrix with proportional error variances on main diagonal
• GLS estimates then are:– E[bgls] = βgls = (X’Ω-1X)-1 X’Ω-1Θ [20]– var[bgls] = σ2 (X’Ω-1X)-1 [21]
48
![Page 49: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/49.jpg)
Weighted Least Squares in Meta-Regression
• Find a weight matrix W so that W’W = Ω-1
• Then the WLS meta-regression estimates become:– E[bwls] = βwls = (X’W’WX)-1 X’W’WΘ [22]– var[bwls] = σ2 (X’W’WX)-1 [23]
49
![Page 50: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/50.jpg)
Weighted Least Squares in Meta-Regression (cont.)
• The good news is that we have a handy estimate of the diagonal elements of the weight matrix W: the effect size variances (or more properly, their square roots)
• The bad news is that fixed effects variance estimates are almost certainly wrong, so we need to use random effect variances. 50
![Page 51: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/51.jpg)
RE Variance Component τ2
• Cannot include moderator variables for all independent variables or all design elements in original studies
• Random Effects variance component τ2 can be conceived of as reflecting the impact of unobserved variables on Θi
51
![Page 52: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/52.jpg)
Weights in Random Effects Meta-Regression
(v1 + τ2) 0 0 0 0 (v2 + τ2) 0 0ee’remr = σ2 . . . .
0 0 0 (vn + τ2)• With weights as the square roots of the
diagonal elements of this matrix. 52
![Page 53: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/53.jpg)
Estimating the Random Effects Meta-Regression Model in Practice• Option 1: Use the WLS function in Stata,
SAS, or some other statistical package.– Don’t do this. All WLS algorithms treat weights
as proportional variance weights rather than as absolute variance weights. This means that meta-regression parameter standard errors will be incorrect.
53
![Page 54: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/54.jpg)
Estimating the Random Effects Meta-Regression Model in Practice• Option 2: Use the VWLS command in
Stata.– This will work as long as you import the
proper random effects variance weights (actually, their standard deviations). These must be calculated outside of the VWLS model. This works because VWLS treats the imported variances as known and absolute rather than as estimated and proportional. 54
![Page 55: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/55.jpg)
Estimating the Random Effects Meta-Regression Model in Practice• Option 3: Use the “Metareg” Command in
Stata– While Stata does not include commands for
meta-analysis, the Stata User Group (SUG) has produced a number of good meta-analysis ADO files, including a random effects meta-regression algorithm. These commands can be downloaded from the Stata website.
55
![Page 56: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/56.jpg)
Estimating the Random Effects Meta-Regression Model in Practice• Option 4: Use “Manually Weighted” Least
Squares– Estimate the REMR model by first manually
weighting each observation by the square root of the proper random effects variance, then using OLS on the transformed data. While this is more work than using the VWLS or Metareg commands, it is necessary when estimating more advanced meta-regression models. 56
![Page 57: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/57.jpg)
Comparison of REMR Results in Environmental Justice (n=680)
Metareg VWLS MWLS(MLE) (MOM) (MLE)
Intercept .068 .073 .068(.006) (.008) (.006)
Pollution -.020 -.020 -.020(.008) (.011) (.008)
Census Tracts .011 .009 .011(.006) (.008) (.006)
Select on DV -.078 -.093 -.078(.019) (.025) (.019) 57
![Page 58: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/58.jpg)
Consequences of Inadequately Addressing Effect Size Heterogeneity
• Inadequately addressing heterogeneity in effect sizes means you have a misspecified REMR model.
• Consequences if r(Me,Mk) = 0?• Consequences if r(Me,Mk) ≠ 0?• Consequences of remaining effect size
heterogeneity in REMR model are akin to specification error in OLS regression
13
![Page 59: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/59.jpg)
Validity Check: Can Meta-Regression Really Recover the
Quantity of Interest from Original Studies?
59
![Page 60: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/60.jpg)
Simulation Strategy• Two-level Monte-Carlo simulation
– Assume Population Regression Model Y=β0 + β1 X1 + β2 X2 + β3 X3 + ε.
• First level simulate 10,800 original studies by• Generate X from distribution N(0,1,Cxx), generate
Y using β1 = .2, .7 (X1 is focal predictor), generate ε ~ N (0, 30)
• Allow measurement of X and Y to vary across studies [X, X*1000, and ln(X)]
• Estimate sample regression models that employ combinations of X1, X2, and X3
60
![Page 61: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/61.jpg)
Simulation Strategy (cont.)• Second level simulate 1000 meta-analyses of
sample size 500• Estimate meta-regression model
– Y=β0 + β1 M1 + β2 M2 + ε– M represent moderator variables
• Key Question: Can REMR recover partial correlation parameter from population regression model in original studies?
61
![Page 62: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/62.jpg)
Simulation Results
62
Table 7.10. Ability of REMR Model Recover Population Parameter of Interest in the Presence of Effect Size Heterogeneity and Variation in the Measurement of X and Y in Original Studies Corr
(Xj,Xk)
Population Parameter of Interest
Estimate of b0
Parameter Estimate
for Moderator Variable X2
Parameter Estimate
for Moderator Variable X3
Predicted Value of
Population Parameter of Interest
.60
.3234
.5008
-.1648
-.0253
.3107
.60
.0950
.1644
-.0552
-.0065
.1027
![Page 63: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/63.jpg)
Section IV: Advanced Meta-Regression
63
![Page 64: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/64.jpg)
“Meta-Analytic techniques developed to date have not addressed the problem of
combining information on second-order effects, such as partial regression
coefficients. Consequently, meta-analysis has not been very useful in the synthesis of evidence from causal/explanatory research."
(Becker 1992: 342)
64
![Page 65: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/65.jpg)
Remaining Problems in Meta-Regression
1. Non-Independence of Observations2. Unequal Numbers of Effect Sizes per
Original Study
65
![Page 66: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/66.jpg)
Problem 1:Non-Independence of Observations• Reasons for Correlated Observations
1. Common data sets across studies2. Common research teams across studies3. Single original study produces multiple
effect sizes (generally incorporates both 1 and 2)
• Consequences of correlated observations for REMR: incorrect parameter st. errors 66
![Page 67: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/67.jpg)
Addressing Problem 1:The Approach in Meta-Analysis
• Traditional Solutions in Meta-Analysis1. Select one effect size per study to use in
meta-analysis2. Calculate average effect size per study and
use this in meta-analysis
67
![Page 68: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/68.jpg)
Addressing Problem 1:The Approach in Meta-Analysis
• Why the Traditional Solutions are Really Bad Ideas:1. Assumes each additional effect size from a
particular study contains no information2. Cannot estimate effect of moderators within
studies, only across studies3. Dramatically reduces sample size and
statistical power68
![Page 69: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/69.jpg)
Addressing Problem 1:The Approach in Econometrics
• The Traditional Solution in Econometrics: Cluster Robust Variance Estimation (CRVE)– i.e., Huber-White Standard Errors as adjusted
for clusters by Liang and Zeger (1986) and Arellano (1987).
– Effect sizes Θig clustered by original study G
69
![Page 70: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/70.jpg)
Addressing Problem 1:The Approach in Econometrics
• In Matrix Form:
V[bREMRg] = (M’W’WM)-1 (Σ Mg’W’(egeg’)WMg) (M’W’WM)-1 [24]
70
![Page 71: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/71.jpg)
71
(τ2+v11+σ21)
ρ1 0 . . . . . . . . 0
ρ1 (τ2+v21+σ21)
. .
0 . σ22** ρ2 ρ2 ρ2 .
. ρ2 σ22** ρ2 ρ2 .
. ρ2 ρ2 σ22** ρ2 .
. ρ2 ρ2 ρ2 σ22** .
. . .ee’ = . . . . . 0 . σ2
g** ρg ρg
. ρg σ2g** ρg
0 . . . . . . 0 ρg ρg σ2g**
![Page 72: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/72.jpg)
Addressing Problem 1 Using Example REMR Model
CRVEREMR REMR
Intercept .068 .068(.006) (.011)
Pollution -.020 -.020(.008) (.015)
Census Tracts .011 .011(.006) (.010)
Select on DV -.078 -.078(.019) (.032) 72
![Page 73: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/73.jpg)
Addressing Problem 1 Using CRVE
• CRVE REMR model should be standard practice in the Social Sciences -- BUT
• Only three examples of published CRVE CRVE models. Why?
• Researchers developing statistics of meta-analysis unfamiliar with developments in econometrics, and vice versa.
73
![Page 74: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/74.jpg)
Extensions to the CRVE REMR Model
1. Small number of clusters? – Use the “Wild Cluster Bootstrap”– Cameron, Gelbach, and Miller 2008
2. Two-way clustering? E.g., effect sizes clustered by study and by data set?– Use “multi-level clustering”– Cameron, Gelbach, and Miller (2011)
74
![Page 75: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/75.jpg)
Problem 2: Unequal Numbers of Effect Sizes Per Study
• Reasons for Unequal Numbers of Effect Sizes Per Study– Varying numbers of subgroup analyses– Varying sensitivity tests
• Consequences of Unequal Numbers of Effect Sizes Per Study– Few studies may dominate data set and
REMR results, threatening external validity 75
![Page 76: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/76.jpg)
Addressing Problem 2:The Approach in Meta-Analysis
• Traditional Solutions in Meta-Analysis1. Select one effect size per study to use in
meta-analysis2. Calculate average effect size per study and
use this in meta-analysis• As we have seen, this is really bad advice
76
![Page 77: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/77.jpg)
Addressing Problem 2:The Approach in Econometrics
• The Econometric approach to Problem 3 generally takes one of two forms.1. Relative “importance weights” attached to
observations (e.g., weight = 1/g)2. Generalized Estimating Equations (GEE)
77
![Page 78: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/78.jpg)
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE– E[Θi] = h (Mb) [25]
• Where h is a link function (identity in regression)– V[Θi|Mi] = σ2
i = g([Mbi] / φ [26]• i.e., the variance is a function of the expectation• φ (phi) is a scale parameter that we will ignore
today
78
![Page 79: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/79.jpg)
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE (cont.)– Assume we have three effect sizes from two
studies: [Θ11 , Θ21 , Θ22] • Θ = (1/3) Θ11 + (1/3) Θ21 + (1/3) Θ22 [27]• Θ = (1/2) Θ11 + (1/2) Θ21 [28]• Θ = (1/2) Θ11 + (1/4) Θ21 + (1/4) Θ22 [29]
– All are unbiased estimates of population effect size Θ, but with different variances
79
![Page 80: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/80.jpg)
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE (cont.)– GEE estimates Θ using an optimal weighting
scheme employing the correlation among observations from the same cluster (ρ). In this simple example:
Θ = [((1+ρ) / (3+ρ)) Θ11] + [(1 / (3+ρ)) Θ21] + [(1 / (3+ρ)) Θ22][30]
80
![Page 81: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/81.jpg)
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE (cont.)– We still need an estimate of the conditional
variance of the sample effect size Θi. GEE estimates this variance (σ2) using:• σ2 = (A)1/2 R (A)1/2 [31]• A has E[Θi ] = h Mb on the diagonal• R is the “working correlation matrix”• For REMR, the exchangeable working correlation
matrix is most appropriate 81
![Page 82: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/82.jpg)
Addressing Problem 2 Using GEE
• Obtaining GEE Estimates• Estimate a naïve model for Θ assuming independent
observations to obtain starting values for b and e.• Use e to estimate the elements of the working correlation
matrix R.• Re-estimate the structural model with weights derived
from the estimated values of ρ in R.• Iterate steps 2 and 3 to convergence.
82
![Page 83: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/83.jpg)
Addressing Problem 2 Using GEE
A Bonus from Using GEE to Estimate the REMR Model: If we use the “empirical” matrix ee’ rather than the “model” matrix ee’, GEE also controls for Problem 2
– GEE calculates parameter estimates controlling for clustering
– Empirical GEE calculates parameter variance estimates controlling for clustering
83
![Page 84: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/84.jpg)
Addressing Problem 4 Using Example REMR Model
CRVE GEEREMR REMR REMR
Intercept .068 .068 .067(.006) (.011) (.011)
Pollution -.020 -.020 -.015(.008) (.015) (.014)
Census Tracts .011 .011 .008(.006) (.010) (.009)
Select on DV -.078 -.078 -.072(.019) (.032) (.021) 84
![Page 85: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/85.jpg)
Addressing Problem 2 Using GEE
• GEE is an excellent choice for estimating the REMR model.
• GEE REMR is quite rare – fewer than half a dozen in the published literature
• Preferred over CRVE when number of effect sizes are highly unequal across studies. CRVE preferred in balanced samples. 85
![Page 86: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/86.jpg)
Estimating Advanced REMR Models in Stata
• Stata cannot estimate directly either the CRVE REMR or the GEE REMR model.
• Neither VWLS nor Metareg can estimate these models.
• Models must be estimated using MWLS data in conjunction with “robust cluster” or “xtgee” commands in Stata.
86
![Page 87: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/87.jpg)
The Take Aways1. Meta-regression is a powerful and under-
utilized tool for synthesizing research in the social sciences.
2. When estimating meta-regression models, social scientists should use the CRVE REMR or the GEE REMR model.
3. Social scientists should not use the standard REMR model or the HLM REMR model.
87
![Page 88: Practical Meta-Analysis for the Social Sciences](https://reader036.fdocuments.net/reader036/viewer/2022062222/568164cf550346895dd6f900/html5/thumbnails/88.jpg)
88
“(Meta-analysis) is going to revolutionize how the sciences . . . handle data. And it is going to be the way many arguments will end.”
T. Chalmers, quoted in Mann 1990: 480