Testing Model Fit

SOC 681

James G. Anderson, PhD

Limitations of Fit Indices

• Values of fit indices indicate only the average or overall fit of a model. It is thus possible that some parts of the model may poorly fit the data.

• Becauwe a single index reflects only a particul;ar aspect oof model fit, a favorable value of that indesx does not by itself indicate good fit. That is why model fit is assessed based on the values of multiple indices.

Limitations of Fit Statistics

• Fit indices do not indicate whether the results are theoretically meaningful.

• Values of fit indices that suggest adequate fit do not indicate the predictive power of the model is also high.

• The sampling distribution of many fit indices are unknown.

Assessment of Model Fit

• Examine the parameter estimates• Examine the standard errors and

significance of the parameter estimates.• Examine the squared multiple correlation

coefficients for the equations• Examine the fit statistics• Examine the standardized residuals• Examine the modification indices

Measures of Fit

• Measures of fit are provided for three models:– Default Model – this is the model that you specified

– Saturated Model – This is the most general model possible. No constraints are placed on the population moments It is guaranteed to fit any set of data perfectly.

– Independence Model – The observed variables are assumed to be uncorrelated with one another.

GPA

HEIGHT

WEIGHT

RATING

ACADEMIC

ATTRACT

e1

e2

1

1

Default Model

GPA

HEIGHT

WEIGHT

RATING

ACADEMIC

ATTRACT

e1

e2

1

1

Saturated Model

GPA

HEIGHT

WEIGHT

RATING

ACADEMIC

ATTRACT

Independence Model

Overall measures of Fit

• NPAR is the number of parameters being estimated (q)

• CMIN is the minimum value of the discrepancy function between the sample covariance matrix and the estimated covariance matrix.

• DF is the number of degrees of freedom and equals the p-q– p=the number of sample moments– q= the number of parameters estimated

Overall measures of Fit

• CMIN is distributed as chi square with df=p-q

• P is the probability of getting as large a discrepancy with the present sample

• CMIN/DF is the ratio of the minimum discrepancy to degrees of freedom. Values should be close to 1.0 for correct models.

Chi Square: 2

• Best for models with N=75 to N=100

• For N>100, chi square is almost always significant since the magnitude is affected by the sample size

• Chi square is also affected by the size of correlations in the model: the larger the correlations, the poorer the fit

Chi Square to df Ratio: 2/df

• There are no consistent standards for what is considered an acceptable model

• Some authors suggest a ratio of 2 to 1

• In general, a lower chi square to df ratio indicates a better fitting model

Transforming Chi Square to Z

Z = (2*2) - (2*df-1)

CMIN

Model NPAR CMIN DF P CMIN/DF

Default model 22 10.335 14 .737 .738

Saturated model 36 .000 0

Independence model

8 243.768 28 .000 8.706

RMR, GFI

• RMR is the Root Mean Square Residual. It is the square root of the average amount that the sample variances and covariances differ from their estimates. Smaller values are better

• GFI is the Goodness of Fit Index. GFI is between 0 and 1 where 1 indicates a perfect fit. Acceptable values are above 0.90.

GFI and AGFI (LISREL measures)

• The AGFI takes into consideration the df available to test the model.

• Values close to .90 reflect a good fit.• These indices are affected by sample size

and can be large for poorly specified models.• These are usually not the best measures to

use.

RMR, GFI

• AGFI is the Adjusted Goodness of Fit Index. It takes into account the degrees of freedom available for testing the model. Acceptable values are above 0.90.

• PGFI is the Parsimony Goodness of Fit Index. It takes into account the degrees of freedom available for testing the model. Acceptable values are above 0.90.

RMR, GFI

Model RMR GFI AGFI PGFI

Default model .003 .975 .935 .379

Saturated model .000 1.000

Independence model

.023 .570 .447 .443

Comparisons to a Baseline Model

• NFI is the Normed Fit Index. It compares the improvement in the minimum discrepancy for the specified (default) model to the discrepancy for the Independence model. A value of the NFI below 0.90 indicates that the model can be improved.

Bentler-Bonett Index or Normed Fit Index (NFI)

• Define null model in which all correlations are zero:

2 (Null Model) - 2 (Proposed Model)2 (Null Model)

• Value between .90 and .95 is acceptable; above .95 is good

• A disadvantage of this index is that the more parameters, the larger the index.

Comparisons to a Baseline Model

• RFI is the Relative Fit Index This index takes the degrees of freedom for the two models into account.

• IFI is the incremental fit index. Values close to 1.0 indicate a good fit.

• TLI is the Tucker-Lewis Coefficient and also is known as the Bentler-Bonett non-normed fit index (NNFI). Values close to 1.0 indicate a good fit.

• CFI is the Comparative Fit Index and also the Relative Noncentrality Iindex (RNI). Values close to 1.0 indicate a good fit.

Tucker Lewis Index or Non-normed Fit Index (NNFI)

• Value: 2/df(Null Model) - 2/df(Proposed Model)

2/df(Null Model)

• If the index is greater than one, it is set to1.

• Values close to .90 reflects a good model fit.

• For a given model, a lower chi-square to df ratio (as long as it is not less than one) implies a better fit.

Comparative Fit Index (CFI)

• If D= 2 - df, then:D(Null Model) - D(Proposed Model)

D(Null Model)• If index > 1, it is set to 1; if index <0, it is set to 0• A lower value for D implies a better fit• If the CFI < 1, then it is always greater than the

TLI• The CFI pays a penalty of one for every

parameter estimated

Baseline Comparisons

ModelNFI

Delta1RFI

rho1IFI

Delta2TLI

rho2 CFI

Default model .958 .915 1.016 1.034 1.000

Saturated model 1.000 1.000 1.000

Independence model

.000 .000 .000 .000 .000

Parsimony Adjusted Measures

• PRATIO is the Parsimony Ratio. It is the number of constraints in the model being evaluated as a fraction of the number of constraints in the independence model.

• PNFI is the result of applying the PRATIO to the NFI.

• PCFI is the result of applying parsimony adjustments to the CFI.

Parsimony-Adjusted Measures

Model PRATIO PNFI PCFI

Default model .500 .479 .500

Saturated model .000 .000 .000

Independence model

1.000 .000 .000

Measures Based on the Population Discrepancy

• NCP is an estimate of the noncentrality parameter obtained by fitting a model to the population moments rather than to the sample moments.

• The 90% confidence interval is also computed for the NCP.

NCP

Model NCP LO 90 HI 90

Default model .000 .000 7.102

Saturated model .000 .000 .000

Independence model

215.768 169.584 269.424

The Minimum Sample Discrepancy Function

• FMIN is the minimum value of the discrepancy .

FMIN

Model FMIN F0 LO 90 HI 90

Default model .107 .000 .000 .073

Saturated model .000 .000 .000 .000

Independence model

2.513 2.224 1.748 2.778

Root Mean Square Error of Approximation (RMSEA)

• Value: [(F0pop-F0est)/n]/df• F0 is the minimum vale of the discrepancy

function.• If 2 < df for the model, RMSEA is set to 0• Good models have values of < .05; values of

> .10 indicate a poor fit.• It is a parsimony-adjusted measure.• Amos provides upper and lower limits of a

90% confidence interval for the RMSEA

PCLOSE

• PCLOSE is the probability for testing the null hypothesis that the population RMSEA is no greater than 0.05.

RMSEA

Model RMSEA LO 90 HI 90 PCLOSE

Default model .000 .000 .072 .877

Independence model

.282 .250 .315 .000

Information Theoretic Measures

• These indices are composite measures of badness of fit and complexity.

• Simple models that fit well receive low scores. Complicated poorly fitting models get high scores.

• These indices are used for model comparison not to evaluate a single model.

AIC

Model AIC BCC BIC CAIC

Default model 54.335 58.835 111.204 133.204

Saturated model 72.000 79.364 165.059 201.059

Independence model

259.768 261.404 280.447 288.447

Akaike Information Criterion (AIC)

• Value: 2 + k(k-1) - 2(df)

where k= number of variables in the model• A better fit is indicated when AIC is smaller• Not standardized and not interpreted for a

given model.• For two models estimated from the same

data, the model with the smaller AIC is preferred.

Information Theoretic Measures

• BCC is the Browne-Cudeck Criterion

• BIC is Bayes Information Criterion.

• CAIC is the consistent AIC

• ECVI except for a constant scale factor it is the same as AIC.

• MECVI except for a scale factor is the same as the BCC.

ECVI

Model ECVI LO 90 HI 90 MECVI

Default model .560 .598 .671 .607

Saturated model .742 .742 .742 .818

Independence model

2.678 2.202 3.231 2.695

Nonhierarchical Models

Diminished SES

NeurologicalDysfunction

Low Morale

IllnessSymptoms

PoorRelationships

Illness Symptoms

NeurologicalDysfunction

Diminished SES

PoorRelationships

LowMorale

Hierarchical Models

Difference in Chi Square

Value: X2diff = X2 model 1 -X2

model 2

DFdiff = DF model 1 –DFmodel 2

Diminished SES

NeurologicalDysfunction

Low Morale

IllnessSymptoms

PoorRelationships

Diminished SES

NeurologicalDysfunction

Low Morale

IllnessSymptoms

PoorRelationships

Miscellaneous Measures

• HOELTER is the largest sample size for which one would accept the hypothesis that a model is correct.

Hoelter Index

• Value: (N-1)*2(crit) + 1 2

Where 2 (crit) is the critical value for the chi-square statistic

• The index should only be calculated if chi square is statistically significant.

Hoelter Index (2)

• If the critical value is unknown, can approximate: [ (1.645 + (2df-1) ]2 + 1

2 2/ (N-1)• For both formulas, one rounds down to

the nearest integer• The index states the sample size at

which the chi square would not be significant

Hoelter Index (3)

• In other words, how small one’s sample size would have to be for chi square to no longer be significant

• Hoelter Recommends values of at least 200

• Values < 75 indicate poor fit

HOELTER

ModelHOELTER

.05HOELTER

.01

Default model 223 274

Independence model

17 20

Which Fit Indices to Report?/

• Chi Square and df

• RMSEA

• CFI

• AGFI

• Hierarchical Models: Difference in Chi Square

• Nonhierarchical Models: AIC