Slide 1

Slide 2

Mosteller & Tukey (1977). Data Analysis and Regression.

Slide 3

Gerry Altmann Encouraging linguists to use linear mixed- effects models is like giving shotguns to toddlers. (see Barr et al., 2013)

Slide 4

A world of subjectivity Sarah Depaoli IF YOU BEAT THE DATA, AT SOME TIME IT WILL SPEAK

Slide 5

A world of subjectivity Sarah Depaoli and then you publish and get tenure.

Slide 6

LMM response ~ intercept + slope * fixed effect + error distinguish between test and control variables

Slide 7

Test vs. Control Variable Example test variablecontrol variable Null Model

Slide 8

Test vs. Control Variable Example test variablecontrol variable Null Model

Slide 9

Test vs. Control Variable Example Critical Effect Control 1 Control 2 Random Effects Response ~ BLACK BOX BLACK BOX

Slide 10

Test vs. Control Variable Example Critical Effect Control 2 Random Effects Response ~ BLACK BOX BLACK BOX Control 3

Slide 11

Test vs. Control Variable Example Critical Effect Control 2 Random Effects Response ~ Control 3

Slide 12

Trade-off #1 Model Simplicity Model Fit

Slide 13

Trade-off #2 Data- driven Theory- driven Exploratory End Confirmatory End Harald Baayen

Slide 14

Trade-off #2 Data- driven Theory- driven Exploratory End Confirmatory End Roger Mundry (and many others)

Slide 15

How much do you allow the data to suggest new hypotheses? How much do you depend on a priori theory? Trade-off #2 Big Question:

Slide 16

Approach 1: more data-driven Approach 1: more data-driven Approach 2: more theory- driven Approach 2: more theory- driven e.g., test whether random slopes are needed (maybe not advisable) e.g., test whether interaction for sth. is necessary or not (o.k. if the interaction is a control variable) e.g., test whether sth. requires a non-linear or a linear effect (maybe o.k.)

Slide 17

THINGS TO WORRY ABOUT: Taken to the extreme, this approach has a very high likelihood of finding any significant result The model selection process is less transparent to outsiders (or, you have to write a LONG LONG stats section) Approach 1: more data-driven Approach 1: more data-driven Approach 2: more theory- driven Approach 2: more theory- driven

Slide 18

Approach 1: more data-driven Approach 1: more data-driven Approach 2: more theory- driven Approach 2: more theory- driven ADVANTAGES: You dont miss important things in your data Your model might thus be more accurate and more true to the data

Slide 19

Approach 1: more data-driven Approach 1: more data-driven Approach 2: more theory- driven Approach 2: more theory- driven You formulate your model before you look at the data The components of your model are guided by: Theory + Published Results General world-knowledge Research experience Taken to the extreme, you cant even make a plot before you formulate your model

Slide 20

Approach 1: more data-driven Approach 1: more data-driven Approach 2: more theory- driven Approach 2: more theory- driven ADVANTAGES: It forces you to think a lot Its fun! It gives you a lot of responsibility, as a scientist Your estimates are going to be more conservative

Slide 21

Approach 1: more data-driven Approach 1: more data-driven Approach 2: more theory- driven Approach 2: more theory- driven Think about model (before you conduct your experiment) Build model, evaluate the models assumptions Build model that better fits the assumptions Test whether control variables interact with test variable, or whether they are needed

Slide 22

Dialogue with your model You need to know that theres multiple responses per subject and item! People might speed up or slow down throughout an experiment. You need to know that each item was repeated two times! Token Researcher ;-)

Slide 23

Keep in mind: You have to resolve non-independencies Your random effects structure should be maximal with respect to your experimental design

Slide 24

Protecting your research from yourself: Whatever you do, your model decision should not be based on the significance of your effect

Slide 25

(JEPS Bulletin)

Slide 26

Important principle CONFIRM FIRST EXPLORE SECOND John McArdle McArdle, J. J. (2011). Some ethical issues in factor analysis. In A.T. Panter & S. K. Sterba (Eds.), Handbook of Ethics in Quantitative Methodology (pp. 313-339). New York, NY: Routledge. McArdle (2011: 335)

Slide 27

The write-up

Slide 28

Important principle BE HONEST NOT PURE John McArdle

Slide 29

Cool guidelines United Nations Economic Commission for Europe (2009a). Making Data Meaningful Part 1: A guide to writing stories about numbers. New York and Geneva: United Nations. United Nations Economic Commission for Europe (2009b). Making Data Meaningful Part 2: A guide to presenting statistics. New York and Geneva: United Nations.

Slide 30

We tested a linear mixed effects model with subjects and items as random effects.

Slide 31

The write-up should reflect (as adequately as possible) the details of your model and your model selection procedure = Reproducible Research

Slide 32

Rule of thumb: One needs to provide sufficient information for the reader to be able to recreate the analyses. Barr et al. (2013) Ask yourself: With the information that I provided, could I, myself, replicate the analysis?

Slide 33

How to write up (1) "Phenomenon-oriented write-up" (2) Appendix / Supplementary Materials

Slide 34

We used generalized linear mixed models to test the effect of Gender and Politeness on pitch. Subjects and items were random effects (random intercepts) (Baayen, Davidson & Bates, 2008), with random slopes for subjects and items for the effect Politeness (Barr, Levy, Scheepers & Tily, 2013). We also included a Gender * Politeness interaction into the model and if this interaction was not significant, only included the main effects. /// Q-Q plots and plots of residuals against fitted values revealed no obvious deviations from normality and homoskedasticity. We report p-values based on Likelihood Ratio Tests of the model with the main fixed effect in question (Politeness) against the model without the main fixed effect (null model, including Gender). Example #1

Slide 35

We used generalized linear mixed models to test the association between voice onset time and pitch. The fixed effects quantify the effect of VOT on politeness, as well as the effect of place of articulation, vowel type and gender on politeness. The random effects quantify the by-subject and by- item variability in pitch (random intercepts), as well as the variation of the effect of VOT on pitch for subjects and items (random slopes). Example #2: "Phenomenon-oriented"

Slide 36

Visual inspection of residual plots revealed no obvious deviation from normality and homoskedasticity of errors. We checked plots of residuals against fitted values and found no indication that the normality and homoskedasticity assumption were violated. indicated a problem with We therefore log-transformed the data. Mentioning assumptions

Slide 37

Results o Provide results of likelihood ratio test (i.e., significance etc.) o Provide estimates and standard errors in the metric of the model o For poisson and logistic regression, additionally provide some exemplary back- transformed values (dont back-transform the standard errors)

Slide 38

Data: mag Models: magmodel.maineffect: linelength ~ condition + city_status + german_side + gender + magmodel.maineffect: trial_order + (1 + condition * city_status | subjects) + magmodel.maineffect: (1 + condition * city_status | items) magmodel: linelength ~ condition * city_status + german_side + gender + magmodel: trial_order + (1 + condition * city_status | subjects) + magmodel: (1 + condition * city_status | items) Df AIC BIC logLik Chisq Chi Df Pr(>Chisq) magmodel.maineffect 27 7984.5 8121.9 -3965.3 magmodel 28 7893.7 8036.2 -3918.8 92.821 1 < 2.2e-16 *** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Likelihood Model Output

Slide 39

Important principle BE HONEST NOT PURE John McArdle

Slide 40

Make your scripts orderly and reproducible

Slide 41

Make your script online available Avoid modifying your data manually... make a script that records your process Reproducibility

Slide 42

Thats it