Maryland Validity ConferenceSlide 1October 10, 2008 Validity from the Perspective of Model-Based...

Maryland Validity Conference Slide 1October 10, 2008

Validity from the Perspective of Model-Based Reasoning

Robert J. MislevyMeasurement, Statistics and Evaluation

University of Maryland, College Park

Presented at the conference “The Concept of Validity: Revisions, New Directions and Applications,” University of Maryland, College Park, MD

October 9-10, 2008.

Supported by a grant from the Spencer Foundation.


Overview of the Talk Sources of unease

Cognition in terms of patterns

Model-based reasoning

Measurement models as model-based reasoning

Implications for validity

Feeling better now


Sources of Unease (1)

Different models fit the same data

Tatsuoka (1983) mixed number subtraction

4 57 1 4

7

3 12 2 3

2

4 13 1 5

3

4 110 2 8

10



Cognitive diagnosis model for instruction

Student characterized by vector of 0/1 variables, say , for which operations she had mastered

Task characterized by which ones the task needed

Probability of correct response via latent class model

2PL IRT model for overall proficiency Student characterized by univariate, continuous , for

proficiency in the domain

Tasks modeled by difficulty & discrimination

Probability of correct response via IRT model

Container metaphor Person B

Person D

Measurement metaphor

Item 1 Item 4 Item 5 Item 3 Item 6 Item 2

Person A Person B Person D



Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. …

An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful.

Snow & Lohman, 1989, p. 317



The evidence from cognitive psychology suggests that test performances are comprised of complex assemblies of component information-processing actions that are adapted to task requirements during performance.




The implication is that sign-trait interpretations of test scores and their intercorrelations are superficial summaries at best. At worst, they have misled scientists, and the public, into thinking of fundamental, fixed entities, measured in amounts.




Whatever their practical value as summaries, for selection, classification, certification, or program evaluation, the cognitive psychological view is that such interpretations no longer suffice as scientific explanations of aptitude and achievement constructs.



Sources of Unease (3) What is the nature of parameters like and

? Where are they?

What is the interpretation of the probabilities that arise from IRT, latent class / cognitive diagnosis models, and the like?

What does this mean about validity of the data / the models / the uses of them?


Cognition in Terms of Patterns

The sociocognitive paradigm

Metaphors as foundation

Formal model-based reasoning


The sociocognitive paradigm

Converging ideas from cog psych, neurology, anthropology, linguistics, science ed, etc.

Knowledge as patterns, at many levels… Assembled to understand, to interact with, and to

create particular situations in the world Developed, strengthened, modified by use Associations of all kinds, including applicability,

affordances, procedures, strategies, affect


Walter Kintsch’s CI Theory of Reading Comprehension

More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.

Text Text base Situation ModelContext

Context1

LTM

Kintsch is focusing here on “experiential” cognition – not conscious, occurring at the scale of milliseconds.We’ll talk about reflective cognition in a couple minutes.

Kintsch is focusing here on “experiential” cognition – not conscious, occurring at the scale of milliseconds.We’ll talk about reflective cognition in a couple minutes.




Text Text base LTM Situation Model ActionContext

Context1

Context2





Context1

Context2





Context2

Context3



Lakoff & Johnson

» Metaphors we live by (1980); Philosophy in the flesh (1999)

Key idea:» Cognitive machinery builds from capabilities for interacting

with the real physical and social world.

» We extend and creatively recombine basic patterns and relationships to think about everything from …

everyday things to

extremely complicated and abstract social, conceptual, philosophical realms

True of both experiential and reflective cognition.



Example: Containers

Free Clip Art Provided by Artclips.com


Metaphors as foundationExample: Containers

Everyday experience Set theory

» Very good, mostly.

Knowledge as collection of discrete things inside our heads

» Usually good and useful, in communication

» Sometimes inapt, as sole basis of instructional practice and assessment design (the Jeopardy model of cognition—Rosie Perez in White men can’t jump)

Example: Containers



Example: Cause & Effect




Newton’s laws; kinematics; quantitative models of force and motion, esp. F=MA

Newton’s laws; kinematics; quantitative models of force and motion, esp. F=MA




xj

IRT & SEM models; quantitative models for response probabilities, esp. Rasch’s P=

IRT & SEM models; quantitative models for response probabilities, esp. Rasch’s P=



Example: Cause & EffectExample: Cause & Effect

Everyday experience F=MA


Teleological theories of history, a la Hegel

» Not so good, mostly.


Everyday experience F=MA



Model-Based Reasoning

Real-World Situation Reconceived Real-World Situation

Entities and relationships

Representational Form A

y=ax+b (y-b)/a=xRepresentational

Form B

Mappings among representational

systems

Mainly semantic

Mainly syntactic


Properties of Models (1) Human way to think about complex unique

situations Abstract structure of entities, relationships,

processes What’s included, what’s omitted Levels of analysis and grainsize

» Newtonian and quantum mechanics » Transmission genetics at level of species,

individuals, cells, or molecules


Properties of Models (2)

Can apply different models to same situation

» Can view selling car to brother-in-law in terms of

economic transaction model vs family

relationships model

Models tuned to uses / problems / purposes

» Mixed number subtraction


Properties of Models (2)The modeling cycle:

Evaluate

Revise

Model

Observe

Predict/Use

» Fit?» Does it work?» What’s left out?» Adequacy of rationale?


Models with probabilistic layers Probability from analogy with physical games

of chance (Shafer)

Probability connects to model representation

» Key in model criticism

Model posits space for patterns; parameter

values characterize them; probability models

can characterize …

» Variation in patterns

» Modeler’s uncertainty about patterns & parameters


Psychometric / Measurement Models E.g., IRT, CTT, FA, SEM, CDM Model posits space for patterns, parameter

values characterize them Semantic layer is cause & effect metaphor

» Q: In what sense does “cause” X? » A: The C&E metaphor grounds productive

connection between observations and inferences

Modeling patterns across people, not explaining item responses (Snow & Lohman)» Could model within-person processes at finer

grainsize


Some answers What is the nature of parameters like and

? Where are they?» These are characterizations of patterns we

observe in real-world situations (ones we in part construct for target uses) through the lens of a simplified model we are (provisionally) using to think about those situations and the use situations in which the patterns are apt to be relevant.

» So they are in our heads, but they aren’t worth much unless they reflect patterns in examinees’ actions in the world.


Some answers What is the interpretation of the probabilities

that arise from IRT, latent class / cognitive diagnosis models, and the like? » These are characterizations of patterns we

observe in situations and our degree of knowledge about them, again through the lens of a simplified model we are (provisionally) using to think about those situations.

» In addition to guiding inference through the model, they provide tools for seeing where the model may be misleading, inadequate.


Some answers

What does this mean about validity of the data / the models / the uses of them?


Validity Evidence

Real-World Situation Reconceived Real-World Situation

Entities and relationships

Representational Form A

y=ax+b (y-b)/a=xRepresentational

Form B

Mappings among representational

systems

Theory and experience supporting the

narrative/scientific frame

Theory and experience supporting the

narrative/scientific frame

Empirical evaluation of predictions / outcomes

Empirical evaluation of predictions / outcomes

Theoretical and empirical grounding of task design

Theoretical and empirical grounding of task design

Theoretical and empirical grounding of task-scoring procedures

Theoretical and empirical grounding of task-scoring procedures


Validity Implications, Sense 1 The currently dominant view:

Validity is an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment. (Messick, 1989)

Focus on situated use of data from test Consistent with MBR perspective; i.e.,

reasoning through psychometric model in particular situations & inferences.


Validity Implications, Sense 2 Alternative (e.g., Wiley, Borsboom, Lissitz):

[A] test is valid for measuring an attribute if and only if (a) the attribute exists and (b) variations in the attribute causally produce variations in the outcomes of the measurement procedure. (Borsboom et al, 2004)

MBR view can omit specific uses, but» must consider range of situations and uses that are

apt to be thought about effectively via the model.» Broader range consistent with scientific program, in

opposition to Snow & Lohman quote.» Is realist but strong correspondence to existence of

traits qua traits in individuals is not required.


I am Feeling Better NowModel-based reasoning provides a way of thinking about validity that …

is consistent with the practical methods that have developed to assure quality of inferences from assessments

is realist, in constructive-realism and L&J’s “embodied realism” sense

is consistent with developments in cognitive psychology, including the nature of scientific reasoning, and the meaning of probability.

Maryland Validity ConferenceSlide 1October 10, 2008 Validity from the Perspective of Model-Based...

Documents

Transcript of Maryland Validity ConferenceSlide 1October 10, 2008 Validity from the Perspective of Model-Based...