SEM Model Fit:Introduction

David A. Kenny

January 25, 2014

2

DefinitionFit refers to the ability of a model to reproduce the data (i.e., usually the variance-covariance matrix). 

A good-fitting model is one that is reasonably consistent with the data and so does not necessarily require any major respecification. 

A good-fitting measurement model is required before interpreting the causal paths of the structural model.

3

However…

• A good-fitting model does not necessarily mean a good model.

• For instance, a model all of whose parameters are zero is likely a "good-fitting" model.

• Additionally, models with nonsensical results (e.g., paths that are clearly the wrong sign) and models with poor discriminant validity or Heywood cases can be “good-fitting” models.  Parameter estimates must be carefully examined to determine if one has a reasonable model as well as the fit statistics.

4

Additionally• A good fitting model can still be improved.• You can make a good-fitting model a better-

fitting model.• Also too you can make the model simpler

and not appreciably change the fit.• A good fitting model is not necessarily the

best model.

5

Model Fit versus Comparison

• Two very different questions

–Model Fit: Is a given model a good-fitting model?

–Model Comparison: Which of two models is better fitting?

6

How Big a Sample Size?

• Rules of Thumb

–Ratio of N to the Number of Free Parameters

–Absolute N• Power Analysis

         

7

Ratio of N to Number of Free Parameters

• Tanaka (1987): 20 to 1, but that is unrealistically high.

• Bentler & Chou (1987): 5 to 1

• Many published studies fail to meet this goal!

8

Absolute N• 200 is seen as a goal for SEM research• Lower sample sizes can be used for

– Models with no latent variables– Models where all loadings are fixed (usually to

one)– Models with strong correlations– Simpler models – Models for which there is a practical upper limit

on N (e.g., countries or years as the unit)

9

Power Analysis• The best way to determine if you have a

large enough sample is to conduct a power analysis.

• Either use the Sattora and Saris (1985) method or conduct a simulation.

• To determine the power to detect a poor-fitting model, you can use Preacher and Coffman’s web-based calculator.

10

2 Test• For models with about 75 to 200 cases, the chi square

test is a reasonable measure of fit.  But for models with more cases (e.g., 400 or more), the 2 test is likely statistically significant. 

• This is why fit indices were invented. They provide a way to claim that one has a good model, despite the fact that the 2 test is statistically significant.

• Sometimes 2 is more interpretable if it is transformed into a Z value.  The following approximation can be used:

Z =  √(2χ2) - √(2df - 1)

11

More on 2

An old measure of fit is the chi square to df ratio or 2/df.  A problem with this fit index is that there is no universally agreed upon standard as to what is a good- and a bad-fitting model.  Note, however, that two very popular fit indices,  TLI and RMSEA, are largely based on this old-fashioned ratio.

The chi square test is too liberal (i.e., too many Type I) errors when variables have non-normal distributions, especially distributions with kurtosis.  Moreover, with small sample sizes, there are too many Type I errors. Note the 2 test is asymptotic test, and so it works best with large sample sizes.

12

Typology of Fit Indices

• This typology is very different from the mainstream definitions. Please note differences.

• Typology

–Incremental

–Absolute

–Comparative

13

Incremental Fit IndicesAn incremental (sometimes called

relative) fit index is analogous to R2, and so a value of zero indicates having the worst possible model and a value of one indicates having the best possible. 

So my model is placed on a continuum. In terms of a formula, it is

        Worst Possible Model – My Model          Worst Possible Model – Best Possible Model

14

The Best Model• The standard definition of best possible

model is one in which 2 equals its degrees of freedom (the expected value of 2 given the null hypothesis of perfect fit).

• An older definition (Bentler-Bonnet) is to assume the best model has a 2 of zero, but that definition ignores sampling error.

15

The Worst Model• The worst possible model is called the null or

independence model and the usual convention is to allow all the variables in the model to have variation but no correlation. 

• The usual null model is to allow the means to equal their actual value.  However, for growth curve models, the null model should set the means as all equal to each other, i.e., no growth. 

16

df of the Null Model• The degrees of freedom of the null

model are k(k – 1)/2 where k is the number of variables in the model.

• If the null model sets the means equal, as in a growth-curve model, its df are (k + 2)(k – 1)/2.

17

Alternative Null Models• Alternative null models might be considered (but almost

never employed).  – One alternative null model is that all latent variable

correlations are zero– Another is that all exogenous variables are correlated but

the endogenous variables are uncorrelated with each other and the exogenous variables. In fact, this is the null model in Mplus when the exogenous variables are measured. 

• O’Boyle and Williams (2011) suggest two different null models for the measurement and structural models.

18

Absolute Fit Indices

• An absolute measure of fit presumes that the best fitting model has a fit of zero.  The measure of fit determines how far the model is from perfect fit. 

• These measures of fit are typically “badness” measure of fit in that a larger number implies worse fit.

• Common absolute fit indices are SRMR and RMSEA.

19

Comparative Fit Indices• A comparative measure of fit is only interpretable

when comparing two different models and cannot be used to determine whether a given model is good-fitting.  This term is unique to this presentation in that these measures are more commonly called absolute fit indices.  However, it is helpful to distinguish absolute indices that do not require a comparison between two models. 

• One advantage of comparative fit indices is that often they can be computed for models that are just-identified.

• Examples are AIC, BIC, and SABIC.  

20

Controversy about Fit Indices

• There is considerable controversy about fit indices. 

• Some researchers do not believe that fit indices add anything to the analysis (e.g., Barrett, 2007) and only the chi square should be interpreted.  The worry is that fit indices allow researchers to claim that a miss-specified model is not a bad model. 

• Others (e.g., Hayduk, Cummings, Boadu, Pazderka-Robinson, & Boulianne, 2007) argue that cutoffs for a fit index can be misleading and subject to misuse. 

21

Consensus View• Most analysts believe in the value of fit indices,

but caution against strict reliance on cutoffs.  • Particularly, problematic is the “cherry picking”

a fit index.  That is, you compute many fit indices and you pick the one index that allows you to make the point that you want to make.  If you decide not to report a popular index (e.g., the TLI or the RMSEA), you need to give a good reason why you are not.

22

Compute a Fit Index?• Kenny, Kaniskan, and McCoach (2014) have

argued that fit indices should not even be computed for small degrees of freedom models. 

• Rather for these models, the researcher should locate the source of specification error by determining what parameter could be added to the model and then test that parameter.

23

Additional Presentations

• Measures of fit

• Factors affecting measures of fit

• References (pdf)