Presenting results from statistical models
description
Transcript of Presenting results from statistical models
![Page 1: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/1.jpg)
Presenting results from statistical models
Professor Vernon Gayle and Dr Paul Lambert(Stirling University)
Wednesday 1st April 2009
![Page 2: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/2.jpg)
Structure of the Seminar
Should take 1 semester!!!
1. Principals of model construction and interpretation
2. Key variables – measurement and func. Form
3. Presenting results
4. Longitudinal data analysis
5. Individuals in households – multilevel models
![Page 3: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/3.jpg)
“One of the useful things about mathematical and statistical models [of educational realities] is that, so long as one states the assumptions clearly and follows the rules correctly, one can obtain conclusions which are, in their own terms, beyond reproach. The awkward thing about these models is the snares they set for the casual user; the person who needs the conclusions, and perhaps also supplies the data, but is untrained in questioning the assumptions….
![Page 4: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/4.jpg)
…What makes things more difficult is that, in trying to communicate with the casual user, the modeller is obliged to speak his or her language – to use familiar terms in an attempt to capture the essence of the model. It is hardly surprising that such an enterprise is fraught with difficulties, even when the attempt is genuinely one of honest communication rather than compliance with custom or even subtle indoctrination” (Goldstein 1993, p. 141).
![Page 5: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/5.jpg)
Structure of the this session
1. Presenting results
• This talk could also take weeks on end
• Two topics only - not the final word
– Quasi-Variances– Sample Enumeration methods
• Many more topics emerging, – propensity score matching– simulation modelling
![Page 6: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/6.jpg)
Using Quasi-variance to Communicate Sociological Results from Statistical Models
Vernon Gayle & Paul S. LambertUniversity of Stirling
Gayle and Lambert (2007) Sociology, 41(6):1191-1208
![Page 7: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/7.jpg)
A little biography (or narrative)…
• Since being at Centre for Applied Stats in 1998/9 I has been thinking about the issue of model presentation
• Done some work on Sample Enumeration Methods with Richard Davies
• Summer 2004 (with David Steele’s help) began to think about “quasi-variance”
• Summer 2006 began writing a paper with Paul Lambert
![Page 8: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/8.jpg)
The Reference Category Problem
• In standard statistical models the effects of a categorical explanatory variable are assessed by comparison to one category (or level) that is set as a benchmark against which all other categories are compared
• The benchmark category is usually referred to as the ‘reference’ or ‘base’ category
![Page 9: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/9.jpg)
The Reference Category Problem
An example of Some English Government Office Regions
0 = North East of England
----------------------------------------------------------------
1 = North West England
2 = Yorkshire & Humberside
3 = East Midlands
4 = West Midlands
5 = East of England
![Page 10: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/10.jpg)
Government Office Region
![Page 11: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/11.jpg)
1 2 3 4
Beta StandardError
Prob. 95% Confidence Intervals
No Higher qualifications - - - - -
Higher Qualifications 0.65 0.0056 <.001 0.64 0.66
Males - - - - -
Females -0.20 0.0041 <.001 -0.21 -0.20
North East - - - - -
North West 0.09 0.0102 <.001 0.07 0.11
Yorkshire & Humberside 0.12 0.0107 <.001 0.10 0.14
East Midlands 0.15 0.0111 <.001 0.13 0.17
West Midlands 0.13 0.0106 <.001 0.11 0.15
East of England 0.32 0.0107 <.001 0.29 0.34
South East 0.36 0.0101 <.001 0.34 0.38
South West 0.26 0.0109 <.001 0.24 0.28
Inner London 0.17 0.0122 <.001 0.15 0.20
Outer London 0.27 0.0111 <.001 0.25 0.29
Constant 0.48 0.0090 <.001 0.46 0.50
Table 1: Logistic regression prediction that self-rated health is ‘good’ (Parameter estimates for model 1 )
![Page 12: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/12.jpg)
Beta StandardError
Prob. 95% Confidence Intervals
North East - - - - -
North West 0.09 0.07 0.11
Yorkshire & Humberside 0.12 0.10 0.14
![Page 13: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/13.jpg)
![Page 14: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/14.jpg)
Conventional Confidence Intervals
• Since these confidence intervals overlap we might be beguiled into concluding that the two regions are not significantly different to each other
• However, this conclusion represents a common misinterpretation of regression estimates for categorical explanatory variables
• These confidence intervals are not estimates of the difference between the North West and Yorkshire and Humberside, but instead they indicate the difference between each category and the reference category (i.e. the North East)
• Critically, there is no confidence interval for the reference category because it is forced to equal zero
![Page 15: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/15.jpg)
Formally Testing the Difference Between Parameters -
)ˆˆ( s.e.
ˆˆ
3-2
3-2
t
The banana skin is here!
![Page 16: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/16.jpg)
Standard Error of the Difference
))ˆˆ( (cov 2 - )ˆvar( )ˆvar( 3-232
Variance North West (s.e.2 )
Variance Yorkshire & Humberside (s.e.2 )
Only Available in the variance covariance matrix
![Page 17: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/17.jpg)
Table 2: Variance Covariance Matrix of Parameter Estimates for the Govt Office Region variable in Model 1
Column 1 2 3 4 5 6 7 8 9
Row North West
Yorkshire &Humberside
East Midlands
West Midlands
East England
South East South West Inner London
Outer London
1 North West .00010483
2 Yorkshire &Humberside
.00007543 .00011543
3 East Midlands
.00007543 .00007543 .00012312
4 West Midlands
.00007543 .00007543 .00007543 .00011337
5 East England
.00007544 .00007543 .00007543 .00007543 .0001148
6 South East .00007545 .00007544 .00007544 .00007544 .00007545 .00010268
7 South West .00007544 .00007543 .00007544 .00007543 .00007544 .00007546 .00011802
8 Inner London
.00007552 .00007548 .0000755 .00007547 .00007554 .00007572 .00007558 .00015002
9 Outer London
.00007547 .00007545 .00007546 .00007545 .00007548 .00007555 .00007549 .00007598 .00012356
Covariance
![Page 18: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/18.jpg)
Standard Error of the Difference
Variance North West (s.e.2 )
Variance Yorkshire & Humberside (s.e.2 )
Only Available in the variance covariance matrix
)0.00007543 ( 2- 0.00011543 0.000104830.0083 =
![Page 19: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/19.jpg)
Formal Tests
t = -0.03 / 0.0083 = -3.6
Wald 2 = (-0.03 /0.0083)2 = 12.97; p =0.0003
Remember – earlier because the two sets of confidence intervals overlapped we could wrongly conclude that the two regions were not significantly different to each other
![Page 20: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/20.jpg)
Comment
• Only the primary analyst who has the opportunity to make formal comparisons
• Reporting the matrix is seldom, if ever, feasible in paper-based publications
• In a model with q parameters there would, in general, be ½q (q-1) covariances to report
![Page 21: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/21.jpg)
Firth’s Method (made simple)
)ˆvar( )ˆvar( 32 quasiquasi s.e. difference ≈
![Page 22: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/22.jpg)
Table 1: Logistic regression prediction that self-rated health is ‘good’ (Parameter estimates for model 1, featuring conventional regression results, and quasi-variance statistics )
1 2 3 4 5
Beta StandardError
Prob. 95% Confidence Intervals
Quasi-Variance
No Higher qualifications - - - - - -
Higher Qualifications 0.65 0.0056 <.001 0.64 0.66 -
Males - - - - - -
Females -0.20 0.0041 <.001 -0.21 -0.20 -
North East - - - - - 0.0000755
North West 0.09 0.0102 <.001 0.07 0.11 0.0000294
Yorkshire & Humberside 0.12 0.0107 <.001 0.10 0.14 0.0000400
![Page 23: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/23.jpg)
Firth’s Method (made simple)
)ˆvar( )ˆvar( 32 quasiquasi s.e. difference ≈
0.0000400 0.0000294 0.0083 =
t = (0.09-0.12) / 0.0083 = -3.6
Wald 2 = (-.03 / 0.0083)2 = 12.97; p =0.0003
These results are identical to the results calculated by the conventional method
![Page 24: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/24.jpg)
The QV based ‘comparison intervals’ no longer overlap
![Page 25: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/25.jpg)
Firth QV Calculator (on-line)
![Page 26: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/26.jpg)
Table 2: Variance Covariance Matrix of Parameter Estimates for the Govt Office Region variable in Model 1
Column 1 2 3 4 5 6 7 8 9
Row North West Yorkshire &Humberside
East Midlands
West Midlands
East England
South East South West Inner London
Outer London
1 North West .00010483
2 Yorkshire &Humberside
.00007543 .00011543
3 East Midlands
.00007543 .00007543 .00012312
4 West Midlands
.00007543 .00007543 .00007543 .00011337
5 East England .00007544 .00007543 .00007543 .00007543 .0001148
6 South East .00007545 .00007544 .00007544 .00007544 .00007545 .00010268
7 South West .00007544 .00007543 .00007544 .00007543 .00007544 .00007546 .00011802
8 Inner London
.00007552 .00007548 .0000755 .00007547 .00007554 .00007572 .00007558 .00015002
9 Outer London
.00007547 .00007545 .00007546 .00007545 .00007548 .00007555 .00007549 .00007598 .00012356
![Page 27: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/27.jpg)
Information from the Variance-Covariance Matrix Entered into the Data Window (Model 1)
0
0 0.00010483
0 0.00007543 0.00011543
0 0.00007543 0.00007543 0.00012312
0 0.00007543 0.00007543 0.00007543 0.00011337
0 0.00007544 0.00007543 0.00007543 0.00007543 0.00011480
0 0.00007545 0.00007544 0.00007544 0.00007544 0.00007545 0.00010268
0 0.00007544 0.00007543 0.00007544 0.00007543 0.00007544 0.00007546 0.00011802
0 0.00007552 0.00007548 0.00007550 0.00007547 0.00007554 0.00007572 0.00007558 0.00015002
0 0.00007547 0.00007545 0.00007546 0.00007545 0.00007548 0.00007555 0.00007549 0.00007598 0.00012356
![Page 28: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/28.jpg)
![Page 29: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/29.jpg)
-10
12
3P
ara
met
er e
stim
ate
BlackChinese
IndianWhite
BangladeshiPakistani
Ethnicity
Parameter estimate 95% confidence intervalParameter estimate 95% QV compariosn intervals
Source: YCS Cohort 9, n=12789.Model: Logistic regression estimating '5+ GCSE Passes A*-C'.
5+ GCSE Passes Year 11Ethnicity Effects
![Page 30: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/30.jpg)
QV Conclusion – We should start using method
Benefits
• Overcomes the reference category problem when presenting models
• Provides reliable results (even though based on an approximation)
• Easy(ish) to calculate
• Has extensions to other models
Costs
• Extra column in results
• Time convincing colleagues that this is a good thing
![Page 31: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/31.jpg)
Example
Drew, D., Gray, J. and Sime, N. (1992)
Against the odds: The Education and Labour Market Experiences of Black Young People
![Page 32: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/32.jpg)
![Page 33: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/33.jpg)
Comparison of Odds
Greater than 1 “higher odds”
Less than 1 “lower odds”
![Page 34: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/34.jpg)
Naïve Odds• In this model (after controlling for other
factors)
White pupils have an odds of 1.0
Afro Caribbean pupils have an odds of 3.2
• Reporting this in isolation is a naïve presentation of the effect because it ignores other factors in the model
![Page 35: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/35.jpg)
A Comparison
Pupil with
4+ higher passes
White
Professional parents
Male
Graduate parents
Two parent family
Pupil with
0 higher passes
Afro-Caribbean
Manual parents
Male
Non-Graduate parents
One parent family
![Page 36: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/36.jpg)
![Page 37: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/37.jpg)
Odds are multiplicative
4+ Higher Grades 1.0 1.0Ethnic Origin 1.0 3.2Social Class 1.0 0.5Gender 1.0 1.0Parental Education 1.0 0.6No. of Parents 1.0 0.9
Odds 1.0 0.86
![Page 38: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/38.jpg)
Naïve Odds
• Drew, D., Gray, J. and Sime, N. (1992) warn of this danger….
• …Naïvely presenting isolated odds ratios is still widespread (e.g. Connolly 2006 Brit. Ed. Res. Journal 32(1),pp.3-21)
• We should avoid reporting isolated odds ratios where possible!
![Page 39: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/39.jpg)
Logit scale
• Generally, people find it hard to directly interpret results on the logit scale – i.e.
![Page 40: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/40.jpg)
Log Odds, Odds, Probability
• Log odds converted to odds = exp(log odds)
• Probability = odds/(1+odds)
• Odds = probability / (1-probability)
![Page 41: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/41.jpg)
Log Odds, Odds, ProbabilityOdds ln odds p
99.00 4.60 0.99
19.00 2.94 0.95
9.00 2.20 0.9
4.00 1.39 0.8
2.33 0.85 0.7
1.50 0.41 0.6
1.00 0.00 0.5
0.67 -0.41 0.4
0.43 -0.85 0.3
0.25 -1.39 0.2
0.11 -2.20 0.1
0.05 -2.94 0.05
0.01 -4.60 0.01
Odds are asymmetric – beware!
![Page 42: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/42.jpg)
Divide by 4 rule
• Gelman and Hill (2008) suggest dividing coefficients from logit models by 4 as a guide for assessing the effects of the estimated for a given explanatory variable as a probability
• They assert that /4 provides a ‘rule of convenience’ for estimating the upper bound of the predictive difference corresponding to a unit change in the explanatory variable.
• Gelman and Hill (2008) are careful to report that this is an approximation and that it performs best near the midpoint of the logistic curve
• We believe that this has some merit as a rough and ready method of interpreting the effects of estimates and is a useful tool especially when tables of coefficients are rapidly flashed up at a conference presentation
Gelman, A. and J. Hill (2008) Data Analysis Using Regression and Multilevel/Hierarchical Models, Cambridge: Cambridge University Press
![Page 43: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/43.jpg)
Communicating Results (to non-technically informed audiences)
• Davies (1992) Sample Enumeration
• Payne (1998) Labour Party campaign data
• Gayle et al. (2002)
• War against the uninformed use of odds (e.g. on breakfast t.v.)
![Page 44: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/44.jpg)
Sample Enumeration Methods
In a nutshell…
“What if” – what if the gender effect was removed
1. Fit a model (e.g. logit)
2. Focus on a comparison (e.g. boys and girls)
3. Use the fitted model to estimate a fitted value for each individual in the comparison group
4. Sum these fitted values and construct a sample enumerated % for the group
![Page 45: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/45.jpg)
Naïve Odds
• Naïvely presenting odds ratios is widespread (e.g. Connolly 2006)
• In this model naïvely (after controlling for other factors)
Girls have an odds of 1.0Boys have an odds of .58
We should avoid this where possible!
![Page 46: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/46.jpg)
Logit Model
• Example from YCS 11
(these pupils took GCSE in 2001)
y=1 5+ GCSE passes (A* - C)
X vars
gender; family social class (NS-SEC);
ethnicity; housing tenure; parental education; parental employment;
school type; family type
![Page 47: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/47.jpg)
Naïve Odds
• Example from YCS 11(these pupils took GCSE in 2001)
• In this model naïvely (after controlling for other factors)
Girls have an odds of 1.0Boys have an odds of .66
We should avoid this where possible!
![Page 48: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/48.jpg)
Sample Enumeration Results
Percentage with 5+ GCSE (A*-C)
All 52%
Girls 58%
Boys 47%
(Sample enumeration est. boys) (50%)
Observed difference 11%
Difference due ‘directly’ to gender 3%
Difference due to other things 8%
![Page 49: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/49.jpg)
Pseudo Confidence Interval
Sample Enumeration
Male Effect
Upper Bound 50.32%
Estimate 49.81%
Lower Bound 49.30%
Bootstrapping to construct a pseudo confidence interval (1000 Replications)
![Page 50: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/50.jpg)
Reference
• A technical explanation of the issue is given in
Davies, R.B. (1992) ‘Sample Enumeration Methods for Model Interpretation’ in P.G.M. van der Heijden, W. Jansen, B. Francis and G.U.H. Seeber (eds) Statistical Modelling, Elsevier
We have recently written a working paper on logit models
http://www.dames.org.uk/publications.html
![Page 51: Presenting results from statistical models](https://reader036.fdocuments.net/reader036/viewer/2022062722/56813971550346895da106ee/html5/thumbnails/51.jpg)
Conclusion –Why have we told you this…
• Categorical X vars are ubiquitous
• Interpretation of coefficients is critical to sociological analyses– Subtleties / slipperiness– (e.g. in Economics where emphasis is often on
precision rather than communication)