Meta-Analysis using HLM 6.0

Yaacov Petscher

Florida Center for Reading Research

Why use HLM?

• Nested structure

• Necessity of special models– Variation at both subject and study levels

“Special Case” of HLM

• If ES are based on n ≥ 30, we assume approximate normal distribution with sampling variance assumed to be known

• V-known models run in Interactive Mode– Time to brush up on your DOS command

code!

Standardized Mean Difference

• No raw data for us– Must rely on stats to be converted to single

metric

• Many types of statistics that may be usedZ t M/SD

χ² F p-value

r r²

Level-1 (Within-Studies) Model

jjj ed

jd = any standardized effect measure from study j

j = the corresponding population parameter

je = sample error associated with d

Level-2 (Between-Studies) Model

jsjsj uW 0

0 = grand mean effect size

s = regression coefficients

sjW = study characteristics (moderators)

ju = level 2 random error

Combined Model

s

jjssj euWd 0

Estimation

s

sjsjjjj Wd )ˆˆ)(1( 0*

Empirical Bayes Estimator

)/( jj V where

EB Estimates

• Level 1– May be used as a shrinkage estimator to

identify potential outliers • Shrinkage in the direction of the grand mean

• Level 2– Supplying the grand mean provides an

estimate of the conditional shrinkage • Shrinkage towards a value that is conditional on

the amount of prior contacts (WEEKS)

The following example will be run using data from Raudenbush & Bryk

Chapter 7 data (pg 211)

The Effect of Teacher Expectancy on Pupil IQ

Study Week s ES Std Error Unconditional Model Conditional Model1 2 0.03 0.125 0.05 0.092 3 0.12 0.147 0.10 -0.063 3 -0.14 0.167 0.00 -0.064 0 1.18 0.373 0.22 0.415 0 0.26 0.369 0.11 0.416 3 -0.06 0.103 -0.01 -0.067 3 -0.02 0.103 0.02 -0.068 3 -0.32 0.22 -0.03 -0.069 0 0.27 0.164 0.16 0.4110 1 0.8 0.251 0.25 0.2511 0 0.54 0.302 0.16 0.4112 0 0.18 0.223 0.11 0.4113 1 -0.02 0.289 0.06 0.2514 2 0.23 0.29 0.11 0.0915 3 -0.18 0.159 -0.03 -0.0616 3 -0.06 0.167 0.03 -0.0617 1 0.3 0.139 0.19 0.2518 2 0.07 0.094 0.07 0.0919 3 -0.07 0.174 0.02 -0.06

Empirical Bayes Estimates

Some Calculations

• Need the Conditional Variances– Since d in this model is Fisher’s r to Z transformation,

the formula is

)3(

1

n

vi

Since we’re not given n we need to calculate another way…..ideas?

YAY!

• Simply square your standard errorStudy ES Std Error

1 0.03 0.1252 0.12 0.1473 -0.14 0.1674 1.18 0.3735 0.26 0.3696 -0.06 0.1037 -0.02 0.1038 -0.32 0.2209 0.27 0.16410 0.8 0.25111 0.54 0.30212 0.18 0.22313 -0.02 0.28914 0.23 0.29015 -0.18 0.15916 -0.06 0.16717 0.3 0.13918 0.07 0.09419 -0.07 0.174

Study ES Std Error Vj1 0.03 0.125 0.01562 0.12 0.147 0.02163 -0.14 0.167 0.02794 1.18 0.373 0.13915 0.26 0.369 0.13626 -0.06 0.103 0.01067 -0.02 0.103 0.01068 -0.32 0.220 0.04849 0.27 0.164 0.026910 0.8 0.251 0.063011 0.54 0.302 0.091212 0.18 0.223 0.049713 -0.02 0.289 0.083514 0.23 0.290 0.084115 -0.18 0.159 0.025316 -0.06 0.167 0.027917 0.3 0.139 0.019318 0.07 0.094 0.008819 -0.07 0.174 0.0303

Data File Prep Considerations

• Since meta-analysis in HLM is a V-known model, only one data file is used

Study ES Vj Week s1 0.03 0.0156 2.0002 0.12 0.0216 3.0003 -0.14 0.0279 3.0004 1.18 0.1391 0.0005 0.26 0.1362 0.0006 -0.06 0.0106 3.0007 -0.02 0.0106 3.0008 -0.32 0.0484 3.0009 0.27 0.0269 0.00010 0.8 0.0630 1.00011 0.54 0.0912 0.00012 0.18 0.0497 0.00013 -0.02 0.0835 1.00014 0.23 0.0841 2.00015 -0.18 0.0253 3.00016 -0.06 0.0279 3.00017 0.3 0.0193 1.00018 0.07 0.0088 2.00019 -0.07 0.0303 3.000

Data File Prep Considerations, cont

• Four key features to data prep (assume using SPSS)– Column 1 = ID in character format– Column 2 = ES estimates – Column 3 = Variance estimates– Column 4-n = Potential level-2 predictors

Formatting

• Variable View– Column 1

• String, width = 2, decimal = 0

– Columns 2-n• Numeric, width = 12, decimal = 3

– Save as a Fixed ASCII (.dat) file– Hold onto your output, you’re gonna need it!

• You should now have a list of the variables and associated Format statements

HLM in Batch Mode

• Bring up your computer’s Command Prompt– Typically found in “Accessories”

• By default, you should see C:\>– If not, type c: and hit enter

• At this point you want to locate HLM– Type dir, hit enter

• cd program files, enter• cd HLM6, enter• hlm2, enter

– We’re now ready to begin!

HLM2- MDM File CreationC:\Program Files\HLM6\hlm2

Will you be starting with raw data? yIs the input file a v-known file? yHow many level-1 statistics are there? 1How many level-2 predictors are there? 1 Enter 8 character name for level-1 variable number 1: Zes

Enter 8 character name for level-2 variable number 1: weeks Input format of raw data file (the first field must be the character ID) format: (a2, 3f12.3) What file contains the data: e:\test.dat

Enter name of MDM file: e:\test.mdm 19 groups have been processed

C:\Program Files\HLM6>

Specifying UC HLM ModelC:\HLM6> hlm2 e:\test.mdm

SPECIFYING AN HLM MODEL Level-1 predictor variable specification

Which level-1 predictors do you wish to use? The choices are: For ZES enter 1

Level-1 predictors? (Enter 0 to end) 1

Level-2 predictor variable specification

Which level-2 variables do you wish to use? The choices are: For WEEKS enter 1

Which level-2 predictors to model ZES? Level-2 predictor? (Enter 0 to end) 0

ADDITIONAL PROGRAM FEATURES

Select the level-2 variables that you might consider for Inclusion as predictors in subsequent models. The choices are: For WEEKS enter 1

Which level-2 variables to model ZES? Level-2 variable? (Enter 0 to end) 0Do you wish to use any of the optional hypothesis testing procedures? n

OUTPUT SPECIFICATION

Do you want a level-2 residual file? nHow many iterations do you want to do? 10000Do you want to see OLS estimates for all of the level-2 units? n Enter a problem title: lvl1 Enter name of output file: e:\lvl1.lis

Results for UC Model ----------------------------------------------------------------------------------------- Standard Approx. Fixed Effect Coefficient Error T-ratio d.f. ----------------------------------------------------------------------------------------- For ZES, B1 INTRCPT2, G10 0.084376 0.052039 1.621 18 -----------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------- Random Effect Standard Variance df Chi-square P-value Deviation Component ------------------------------------------------------------------------------------------------ ZES, U1 0.13896 0.01931 18 36.25115 0.007 ------------------------------------------------------------------------------------------------

Significant variability exists in true-effect sizes

EB Estimation Level 1

s

sjsjjjj Wd )ˆˆ)(1( 0*

Since there are no predictors at Level 1, the last term is omitted, leaving us with

0* ˆ)1( jjjj d

Using Excel to Calculate EB

)/( jj V ES Variance

0.03 0.01560.12 0.0216

-0.14 0.02791.18 0.13910.26 0.1362

-0.06 0.0106-0.02 0.0106-0.32 0.04840.27 0.02690.80 0.06300.54 0.09120.18 0.0497

-0.02 0.08350.23 0.0841

-0.18 0.0253-0.06 0.02790.30 0.01930.07 0.0088

-0.07 0.0303

ES Variance lambda0.03 0.0156 0.5487360.12 0.0216 0.467877

-0.14 0.0279 0.4052121.18 0.1391 0.1201550.26 0.1362 0.122453

-0.06 0.0106 0.641697-0.02 0.0106 0.641697-0.32 0.0484 0.2818990.27 0.0269 0.4139790.80 0.0630 0.2317040.54 0.0912 0.1724080.18 0.0497 0.276448

-0.02 0.0835 0.1853280.23 0.0841 0.184287

-0.18 0.0253 0.429078-0.06 0.0279 0.4052120.30 0.0193 0.4958120.07 0.0088 0.682569

-0.07 0.0303 0.385583

Using the variance component fromlvl 1 Model, create Lambda usingthe formula function

0* ˆ)1( jjjj d

ES Variance lambda Ebdelta0.03 0.0156 0.548736 0.050.12 0.0216 0.467877 0.10

-0.14 0.0279 0.405212 -0.011.18 0.1391 0.120155 0.210.26 0.1362 0.122453 0.10

-0.06 0.0106 0.641697 -0.01-0.02 0.0106 0.641697 0.02-0.32 0.0484 0.281899 -0.030.27 0.0269 0.413979 0.160.80 0.0630 0.231704 0.250.54 0.0912 0.172408 0.160.18 0.0497 0.276448 0.11

-0.02 0.0835 0.185328 0.060.23 0.0841 0.184287 0.11

-0.18 0.0253 0.429078 -0.03-0.06 0.0279 0.405212 0.030.30 0.0193 0.495812 0.190.07 0.0088 0.682569 0.07

-0.07 0.0303 0.385583 0.02

Supplying the grand mean ESinto the Excel formula function allows us get the EB estimates

See the difference in results?

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

ES Ebdelta

Specifying CL2 HLM ModelC:\HLM6> hlm2 e:\test.mdm

SPECIFYING AN HLM MODEL Level-1 predictor variable specification

Which level-1 predictors do you wish to use? The choices are: For ZES enter 1

Level-1 predictors? (Enter 0 to end) 1

Level-2 predictor variable specification

Which level-2 variables do you wish to use? The choices are: For WEEKS enter 1

Which level-2 predictors to model ZES? Level-2 predictor? (Enter 0 to end) 1

ADDITIONAL PROGRAM FEATURES

Select the level-2 variables that you might consider for Inclusion as predictors in subsequent models. The choices are: For WEEKS enter 1

Which level-2 variables to model ZES? Level-2 variable? (Enter 0 to end) 0Do you wish to use any of the optional hypothesis testing procedures? n

OUTPUT SPECIFICATION

Do you want a level-2 residual file? nHow many iterations do you want to do? 10000Do you want to see OLS estimates for all of the level-2 units? n Enter a problem title: lvl2 Enter name of output file: e:\lvl2.lis

Results for CL2 Model

------------------------------------------------------------------------------------------------- Standard Approx. Fixed Effect Coefficient Error T-ratio d.f. P-value ------------------------------------------------------------------------------------------------- For ZES, B1 INTRCPT2, G10 0.408572 0.087146 4.688 17 0.000 WEEKS, G11 -0.157963 0.035943 -4.395 17 0.000 -------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------ Random Effect Standard Variance df Chi-square P-value Deviation Component ------------------------------------------------------------------------------------------------ ZES, U1 0.00283 0.00001 17 16.53614 >.500 ------------------------------------------------------------------------------------------------

EB Estimation Level 2

Since then = 0 and we’re left with 0ˆ )/( jj V

jj WEEKS)(ˆˆ 10*

ES Weeks EBLvl20.03 2 0.090.12 3 -0.06

-0.14 3 -0.071.18 0 0.410.26 0 0.41

-0.06 3 -0.07-0.02 3 -0.07-0.32 3 -0.070.27 0 0.410.80 1 0.250.54 0 0.410.18 0 0.41-0.02 1 0.250.23 2 0.09

-0.18 3 -0.07-0.06 3 -0.070.30 1 0.250.07 2 0.09

-0.07 3 -0.07

Using G10 and G11, we can calculatethe EB estimates for Level 2

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

ES EBLvl2

EB Estimation Level 2

• Since our Level-2 predictor takes on one of four different values, the shrinkage is towards one of the four points.

End(for now)