Moving Away From Aggregate Statistics: Observation Oriented Modeling and

Assessement

Lisa D. Cota, MSStatistical Analyst

Office of University Assessment & TestingOklahoma State University

Why not NHST?• Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.

Retrieved from http://psycnet.apa.org/journals/amp/49/12/997/• Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students

share with their teachers. Methods of Psychological Research, 7(1), 1–20. Retrieved from http://www2.uni-jena.de/svw/metheval/lehre/0405-ws/evaluationuebung/haller.pdf

• Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review. doi:10.3758/s13423-013-0572-3

• Lamiell, J. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure? New Ideas in Psychology, 31(1), 65–71. doi:10.1016/j.newideapsych.2011.02.009

• Lykken, D. What’s wrong with psychology anyway? (1991). In Thinking Clearly About Psychology. University of Minnesota Press.

• Toomela, A. (2010). Quantitative methods in psychology: Inevitable and useless. Frontiers in Psychology, 1(July), 29. doi:10.3389/fpsyg.2010.00029

Why not NHST?

• Assessment should be understandable– NHST: Not easily understood…in fact easily

misunderstood• Oakes, 1986; Gigerenzer, 2004• Hoekstra, Morey, Rouder, & Wagonmaker, 2014

• One problem (of many) with NHST:– Variance Explained ≠ Accuracy

y = 0.5x + 3, R2 = 0.663

y = 0.5x + 3, R2 = 0.663

y = 0.5x + 3, R2 = 0.663

y = 0.5x + 3, R2 = 0.663

Assumptions

• Assumptions for independent samples t-test under NHST:– Null hypothesis is true– Observations are independent between and

within groups– Population variances are exactly equal– Random sampling / random assignment (???)* Continuous DV* DV has a normal population distribution

Observation Oriented Modeling

• No meaningless aggregates• Person-centered • No assumption-laden p-values

Observation Oriented Modeling: Ordinal Pattern Analysis

Results

Randomization Test

Results, with chance-value

VALUE Rubric Data

Results

Significant ≠ Meaningful

Significant ≠ Meaningful

Significant ≠ Meaningful

Transparency, Honesty, and Progress

Raw Numbers

Proportions

Compare w. SPSS….

SPSS Legacy Dialog, k Independent Samples

Proportions

Advantages of OOM

• Assumption-free• Transparent• Honest

OOM website

• “We’ve never actually looked at the data!” …before now!

• www.idiogrid.com/OOM