Moderator analyses: Categorical models and Meta-regression
-
Upload
barrett-mayer -
Category
Documents
-
view
59 -
download
1
description
Transcript of Moderator analyses: Categorical models and Meta-regression
The Campbell Collaboration www.campbellcollaboration.org
Moderator analyses: Categorical models and Meta-regression
Terri Pigott, C2 Methods Editor & co-ChairProfessor, Loyola University Chicago
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Moderator analyses in meta-analysis
• We often want to test our hypotheses about whether variation among studies in effect size is associated with differences in study methods or participants
• We have these ideas a priori, incorporating these characteristics of studies into our coding forms
• Two major forms of moderator analyses in meta-analysis: categorical models analogous to ANOVA, and meta-regression
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Assumptions for this session• We will focus on random effects models as these are the
most common in Campbell reviews• I will assume that we have computed the random effects
variance component (as you did if you were in my session yesterday - though you may feel like this right now)
• We will use two software packages:– RevMan – available here:
http://ims.cochrane.org/revman/download– Comprehensive Meta-analysis – available for free download
and limited trial here: http://www.meta-analysis.com/pages/demo.html
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Categorical moderators
• When the moderator variable is categorical, we can estimate models analogous to ANOVA
• Typically, we are interested in comparing the group mean effect sizes for 2 or more groups
• For example, we will look at a meta-analysis where we compare the mean effect size for studies published in three different sources: journals, dissertations, and unpublished studies
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Categorical moderator models
• With a one-way random effects ANOVA model, recall that we will compute– A mean effect size and standard error for each group, and then
test whether these means are significantly different from one another
– The mean effect size and standard error require an estimate of the variance component
– QUESTION: Will we assume that each group has the same variance component? Or, will we assume that each group has its own variance component?
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
What are our assumptions if we decide to use separate estimates within subgroups?• We believe that the variation among studies is different
between groups.• For example, if we are testing out an intervention and we
have studies that use either a low-income and a high-income group of students, we might believe that there will be more variation in effectiveness among studies that have mostly low-income participants
• Another example: the effectiveness of an intervention for juvenile delinquents will vary more for the group that had a prior arrest than for those that do not have a prior arrest
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
What are our assumptions when we use a pooled estimate?• We believe that the variation among effect sizes are the
same no matter the group.• For an intervention review, we may assume that the variation
among studies does not differ within the groups of interest• Caveat: We might have to use a pooled estimate if we have
small sample sizes within subgroups. We need at least 5 cases (in general) to be able to estimate a separate variance component for each subgroup
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Flowchart from Borenstein
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Steps for a random effects ANOVA
• Make a decision about the use of a pooled or a separate estimate of the variance component
• Compute the group mean effect sizes, and their standard errors
• Compare the group mean effect sizes to see if they are statistically different from one another
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Eagly, Johannesen-Schmidt & van Engen (2003)
• This synthesis examines the standardized mean difference estimated in primary studies for the difference between men and women in their use of transformational leadership.
• Transformational leadership involves “establishing onself as a role model by gaining the trust and confidence of followers” (Eagly et al. 2003, p. 570).
• The sample data is a subset of the studies in the full meta-analysis, a set of 24 studies that compare men and women in their use of transformational leadership
• Positive effect sizes indicate males score higher than females
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
To follow along:
• Open RevMan• Open a review from a file• Open the file named:
Gender_differences_for_transformational_leadership.rm5• Go to Data and analyses on the left-hand menu• Double-click on 1.1 Transformational leadership• NOTE: RevMan uses the assumption that each group has a
different variance component
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Summary of results – separate variance estimates for each group
Group k Mean 95% CI τ2 p
Journals 13 -0.05 [-0.24, 0.12] 0.09 <0.001
Dissertations 7 -0.47 [-0.69,-0.26] 0.02 0.22
Unpublished 4 -0.16 [-0.30,-0.03] 0.00 0.87
TOTAL 24 -0.16 [-0.29, -0.03] 0.08 <0.001
•Journals have a significant variance component, and the mean is not different from zero•Dissertations and unpublished studies both have a non-significant variance component, but both find that women score higher on transformational leadership
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Summary of results – separate variance estimates for each group (continued)
Group k Mean 95% CI τ2 p
Journals 13 -0.05 [-0.24, 0.12] 0.09 <0.001
Dissertations 7 -0.47 [-0.69,-0.26] 0.02 0.22
Unpublished 4 -0.16 [-0.30,-0.03] 0.00 0.87
TOTAL 24 -0.16 [-0.29, -0.03] 0.08 <0.001
•The test of the variance component as different from zero is exactly the fixed effects test of homogeneity. •To get this test, we compute the test of homogeneity within each group of studies.
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Test of between group differences
• To test between group differences in a random effects model, we test whether the variance component for the variation among the random effects means is equal to zero
• There are several ways to obtain this value• We will use a test of homogeneity of the three means – we
will treat the three group means as a meta-analysis
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Test of between-group differences
• We will compute a test of homogeneity using our three means as if this is a meta-analysis
• We will use the means and their estimated variances to compute the sums we need to compute the homogeneity test
• These computations are all done “behind the scenes” by RevMan
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Computation of Q between groups
Source Mean Var Wt Wt*Mean Wt*Mean2
Journals -0.05 0.008 122.53 -6.13 0.031
Dissertations -0.47 0.012 86.46 -40.64 19.10
Unpublished -0.16 0.005 211.32 -33.82 5.41
SUM 420.31 -80.59 24.54
( 80.59* 80.59)24.54
420.319.09
Q
Compare 9.09 to a chi-square with df=3-1=2. p-value is 0.011
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
What happens if we use the same variance component for all groups?
• We will need to try this in Comprehensive Meta-analysis• Open your trial version of Comprehensive Meta-analysis• Check that you will run the trial• Open the file called: leaderage.cma• Data is here:
https://my.vanderbilt.edu/emilytannersmith/training-materials/
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Run analyses
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Group bypub source
Study name Statistics for each study Hedges's g and 95% CI
Hedges's Standard Lower Upper g error Variance limit limit Z-Value p-Value
1.00 B23 -0.090 0.066 0.004 -0.220 0.040 -1.355 0.1761.00 CA -0.170 0.083 0.007 -0.333 -0.007 -2.045 0.0411.00 CH -0.220 0.118 0.014 -0.451 0.011 -1.868 0.0621.00 CW1 0.610 0.092 0.008 0.430 0.790 6.656 0.0001.00 CW2 0.200 0.150 0.023 -0.095 0.495 1.330 0.1831.00 DF -0.250 0.147 0.022 -0.539 0.039 -1.696 0.0901.00 JB -0.130 0.117 0.014 -0.360 0.100 -1.109 0.2671.00 JL -0.290 0.105 0.011 -0.495 -0.085 -2.768 0.0061.00 KO 0.310 0.078 0.006 0.157 0.463 3.982 0.0001.00 KUU 0.090 0.214 0.046 -0.330 0.510 0.420 0.6751.00 LJ -0.350 0.068 0.005 -0.482 -0.218 -5.185 0.0001.00 RH -0.360 0.087 0.008 -0.531 -0.189 -4.116 0.0001.00 SM 0.000 0.283 0.080 -0.556 0.556 0.000 1.0001.00 -0.055 0.084 0.007 -0.219 0.108 -0.662 0.5083.00 BO -0.620 0.262 0.069 -1.134 -0.106 -2.365 0.0183.00 CU -0.170 0.185 0.034 -0.533 0.193 -0.917 0.3593.00 DA -0.150 0.281 0.079 -0.701 0.401 -0.534 0.5933.00 HI -0.360 0.356 0.127 -1.059 0.339 -1.010 0.3123.00 MCG -0.720 0.242 0.059 -1.194 -0.246 -2.975 0.0033.00 RO -0.440 0.225 0.050 -0.880 0.000 -1.960 0.0503.00 WI -0.870 0.239 0.057 -1.339 -0.401 -3.639 0.0003.00 -0.477 0.141 0.020 -0.752 -0.201 -3.389 0.0014.00 BJ -0.100 0.148 0.022 -0.390 0.190 -0.675 0.5004.00 GM -0.100 0.252 0.064 -0.594 0.394 -0.397 0.6924.00 MA -0.080 0.199 0.040 -0.471 0.311 -0.401 0.6884.00 SP -0.210 0.090 0.008 -0.385 -0.035 -2.346 0.0194.00 -0.130 0.162 0.026 -0.447 0.187 -0.805 0.421Overall -0.159 0.066 0.004 -0.288 -0.031 -2.429 0.015
-1.00 -0.50 0.00 0.50 1.00
Favours A Favours B
Meta Analysis
Meta Analysis
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Notes about the CMA forest plot
• Like RevMan, the confidence intervals around each study are the fixed effects confidence intervals (they use the within-study fixed effects variance)
• The group means are the random effects means computed using random effects weights. Their confidence intervals are also use random effects.
• In this example, we are using the same variance component for all groups
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Summary from CMA
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Notes about CMA results
• We assumed that the variance component was 0.08 for all studies
• Compared to our separate variance estimates, this value is smaller than the separate variance estimate for journals, but larger than the separate estimate for dissertations and unpublished articles
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Reporting results from a random effects categorical analysis
Group RE mean RE CI τ3• The assumption made about the random effects variance: separate estimate for each group, or the same estimate for all groups.
• Rationale for the choice of variance component
• The random effects mean and CI• The value of the variance
components (or variance component)
• The test of the between-group differences, and its significance
for between-group means
df and p-value
Q
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
What is meta-regression?
• Meta-regression is a statistical technique used in a meta –analysis to examine how characteristics of studies are related to variation in effect sizes across studies
• Meta-regression is analogous to regression analysis but using effect sizes as our outcomes, and information extracted from studies as moderators/predictors
• NOTE: We can conduct a meta-regression in any statistical program. Here we will use CMA. BUT, note that using other standard programs may necessitate some adjustments to the results since they don’t produce exactly what we want.
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Meta-regression used to examine heterogeneity
• When we have a heterogeneous set of effect sizes, we can use statistical techniques to examine the association among characteristics of the study and variation among effect sizes
• We have a plan for these analyses a priori – based on our understanding of the literature, and a logic model or framework
• Meta-regression used when we have more than one predictor or moderator (either continuous or categorical)
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Form of the meta-regression model
0 1 1 ...i i p ipES B B x B x
1 2
0 1
is our generic effect size for study i
, ,... are the values for the predictor variables for study i
, ,..., are the unknown regression coefficients
i
i i ip
p
ES
x x x
B B B
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Recall that the variance of the effect size
• Depends on the sample size for all types of effects we have talked about
• Thus, the precision of each study’s effect size depends on sample size
• This is different from our typical application of regression where we assume every person has the same “weight”
• Thus, we need to use weighted least squares regression to account for the fact that the precision of each effect size depends on sample size
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Random effects meta-regression
• As in the categorical analysis discussion, we will need an estimate of the random effects variance for our studies that will be used as our weights in the regression
• There are many ways to compute the variance component in a random effects meta-regression
• For now, let’s assume a single variance component for all studies.
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Test for the fit of the meta-regression model• As in a standard regression model, we can use the regression
ANOVA table for diagnostics about the fit of a meta-regression• Recall that in a standard regression analysis, we would get the
following regression ANOVA table:
ANOVAb,c
556.483 2 278.241 6.927 .004a
923.856 23 40.168
1480.338 25
Regression
Residual
Total
Model1
Sum ofSquares df Mean Square F Sig.
Predictors: (Constant), grade, percmina.
Dependent Variable: ztransb.
Weighted Least Squares Regression - Weighted by wtc.
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Test of Model Fit in Meta-regression
• In meta-regression, we use the ANOVA table to get two different Q statistics:
• QM – model sum of squares, compare to chi-square distribution with p – 1 df (p is number of predictors in the model)
• QR – residual sum of squares, compare to chi-square distribution with k - p – 1 df (k is the number of studies)
• See Lipsey & Wilson. 2001. Practical Meta-analysis. Thousand Oaks, CA: Sage. pp. 122-124
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
QM , the model sum of squares
• Qmodel is the test of whether at least one of the regression coefficients (not including the intercept) is different from zero
• We compare QM to a chi-square distribution with p – 1 degrees of freedom with p = # of predictors in model
• If QM is significant, then at least one of the regression coefficients is different from zero
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
QR , the error or residual sum of squares
• QR is the test of whether there is more residual variation than we would expect IF the model “fits” the data
• We compare QR to a chi-square distribution with k - p – 1 degrees of freedom with k = # of studies/effect sizes, and p = # of predictors in model
• If QR is significant, then we have more error or residual variation to explain, or that is not accounted for by the variables we have in the model
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Testing significance of individual regression coefficients in meta-regression• In a standard regression analysis, we find the t-tests on the printout to
see which regression coefficients are significantly different from zero• Those significant regression coefficients indicate that these predictors
are associated with the outcome• We will use CMA which gives us the z-tests for the regression
coefficients• NOTE: When doing meta-regression in a standard program like
SPSS, we have to make some adjustments since these programs do not compute the weighted regression in the way we need for meta-analysis
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
To conduct a meta-regression in CMA: run an analysis to get to the table below
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Under Analyses, choose meta-regression
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
On the next page, choose the continuous outcome, averageage
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
By default, the analysis will be fixed effects. Choose method of moments under Computational options
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Regression of averageage on Hedges's g
averageage
Hed
ges's
g
24.90 28.62 32.34 36.06 39.78 43.50 47.22 50.94 54.66 58.38 62.10
0.80
0.62
0.44
0.26
0.08
-0.10
-0.28
-0.46
-0.64
-0.82
-1.00
Plot of points and regression line
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Results
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Example for meta-regression
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Objective of the review
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Interventions
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Example from Wilson & Lipsey
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
What to report in a random effects meta-regression?• The software and/or method used to compute the results• The method used to compute the random effects variance
component• The goodness of fit tests: Qmodel , and QResidual
• The regression coefficients and their test of significance
Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org
Final notes
• Software may be a problem in meta-regression as only CMA computes meta-regression. RevMan does not have the capacity for meta-regression
• CMA only allows one predictor in the meta-regression• To conduct the analyses as seen in the Wilson & Lipsey
example, you need to use other general statistical programs like SPSS, or STATA
• There are R programs available to conduct meta-analyses as well
The Campbell Collaboration www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass
0130 Oslo, Norway
E-mail: [email protected]
http://www.campbellcollaboration.org