A Bayesian Perspective on Structured Mixtures of IRT Models

CILVR 2006 Slide 1May 18, 2006

A Bayesian Perspective on Structured Mixtures of IRT Models

Robert Mislevy, Roy Levy, Marc Kroopnick, and Daisy Wise

University of Maryland

Presented at the Conference “Mixture Models in Latent Variable Research,” May 18-19, 2006, Center for Integrated Latent Variable Research, University of Maryland


Probability is not really about numbers;

it is about the structure of reasoning. Glen Shafer, quoted in Pearl, 1988, p. 77


Where we are going

The structure of assessment arguments Probability-based reasoning in assessment Increasingly complex psychological narratives

entail…» Extended view of “data”» More encompassing probability models,

from Classical test theory to mixtures of structured Item response theory (IRT) models


The structure of assessment arguments

A construct-centered approach would begin by asking what complex of knowledge, skills, or other attribute should be assessed, …

Next, what behaviors or performances should reveal those constructs, and

what tasks or situations should elicit those behaviors?

Messick, 1992, p. 17


An Example: Mental Rotation Tasks

Stimulus


An Example: Mental Rotation Tasks

Target


Total Scores• Counts of multiple observations which all signify

proficiency--More is better. • Everyone takes same tasks. • Comparisons/decisions based on total scores X. • Nature of tasks outside the model.• No distinction between observation (X) and target of

inference (proficiency at mental rotation);• No probability-based model for characterizing evidence.• No notion of “measurement error” (except when your

score is lower than you think it ought to be)


A properly-structured statistical model overlays a substantive model for the situation with a model for our knowledge of the situation,

so that we may characterize and communicate what we come to believe—as to both content and conviction—and why we believe it—as to our assumptions, our conjectures, our evidence, and the structure of our reasoning.

Mislevy & Gitomer, 1996

Enter probability-based reasoning


Defining variables

A frame of discernment is all the distinctions one can make within a particular model (Shafer, 1976). » “To discern” means “to become aware of” and “to

make distinctions among.” In assessment, the variables relate to the claims

we would like to make about students and the observations we need to make.

All are framed and understood in terms appropriate to the purpose, the context, and psychological perspective that ground the application.


Conditional independence In assessment, the statistical concept of conditional

independence formalizes the working assumption that if the values of the student model variables were known, there would have no further information in the details.

We use a model at a given grainsize or with certain kinds of variables not because we think that is somehow ”true”, but rather because it adequately expresses patterns in the data in light of our perspective on knowledge/skill and the purpose of the assessment.


Classical Test Theory (CTT) Still total scores, but with the idea of

replication—multiple “parallel” tests Xj that may differ, but are all “noisy” versions of the same “true score” :

Xij = i + eij ,

where eij ~ N(0, e2).

Details of cognition, observation task by task, and content of tasks lie outside the probability model.


Classical Test Theory (CTT)

Directed graph representation of Bayesian probability model for multiple parallel tests.

Note direction of conditional probability in model; contrast w. inference about once Xs are observed.

X 1 X 2 X 3

p( )

p( X 1 | )

p( X 2 | )

p( X 3 | )


Classical Test Theory (CTT)

2 2 *

* 2 2 2 2

, , ,

| , | , .

e

ij i e i ei j

p

p x p p p p

x

Posterior inference, via Bayes theorem:

2 2

2 2 2 2

, , , ,

| , | , .

e

ij i e i ei j

p

p x p p p p

x

The full probability model:

Note what’s there, conditional independence, relationships, what’s not there.


Classical Test Theory (CTT) For Student i, posterior inference is probability distribution for i; that

is,» Expected value for test score, along with standard deviation of distribution for

i .» People can be seen as differing only as to propensity to make correct

responses. Inference bound to particular test form. Also posterior distribution for mean & variance of s, error variance,

and reliability, or

2 2 2 .e


Item Response Theory (IRT)

Modeling now at level of items» Adaptive testing, matrix sampling, test assembly

Multiple non-parallel item responses Xij that may differ, but all depend on i:

where j is the possibly vector-valued parameter of Item j.

Conditional independence of item responses given and parameter(s) for each item.

1 1Pr ,..., , ,..., Pr , ,i iJ i J ij i jj

x x x


Item Response Theory (IRT)

Same item ordering for all people. Content of tasks still outside the probability model.

easier harder

The Rasch IRT model for 0/1 items:

Prob(Xij=1|i,j) = (i - j), where(x) = exp(x)/[1+exp(x)].

Item 1 Item 4 Item 5 Item 3 Item 6 Item 2

Person A Person B Person D

less able more able


A full Bayesian model for IRT

, , , , | , | |ij i j i ji j

p X p X p p p p The measurement model:Item responses conditionally independentgiven person & item parameters

Distributions for person parameters

Distributions for item parameters

Distribution for parameter(s) of distributions for item parameters

Distribution for parameter(s) of distributions for person parameters


A full Bayesian model for IRT

*

*

, , ,

| , | |ij i j i ji j

p

p x p p p p

x

Posterior inference:

•For people, posterior distributions for s, or propensity to make correct responses.•How/why outside model. •For items, posterior distributions for s. •Some harder, some easier; How/why outside model.•Presumed to be the same for all people.


Summary test scores … have often been though of as “signs” indicating the presence of underlying, latent traits.

An alternative interpretation of test scores as samples of cognitive processes and contents, and of correlations as indicating the similarity or overlap of this sampling, is equally justifiable and could be theoretically more useful.

The evidence from cognitive psychology suggests that test performances are comprised of complex assemblies of component information-processing actions that are adapted to task requirements during performance.

(Snow & Lohman, 1989, p. 317)

The “cognitive revolution”


Research on mental rotation tasks

Roger Shepard’s studies in the early 70’s shows difficulty of mental rotation tasks depends mainly on how much it is rotated.

Shepard, R. N. & Meltzer, J (1971) Mental rotation of three-dimensional objects. Science, 171, 701-703.

Cooper, L. A., & Shepard, R. N. (1973). Chronometric studies of the rotation of mental images. In W. G. Chase (Ed.), Visual Information Processing (pp. 75–176). New York: Academic Press.


A structured IRT model: The LLTM

The linear logistic test model (Fischer, 1973) Rasch model but item parameters conditional on item

features qj:

where k is a contribution to difficulty from feature k.• Now difficulty modeled as function of task features, as

correlated with demands for aspects of knowledge or processing. Conditional independence of item parameters given features: They “explain” item difficulty.

• Task features bring psychological theory into model.• For mental rotation, can use degree of rotation for qj .

,j jk k jk

q q


Content of tasks inside the probability model.

easier harder



less able more able

Less rotation More rotation

A structured IRT model: The LLTM


A full Bayesian model for LLTM

, , , , ,

| , | | ,ij i j i j ji j

p X q

p X p p q p p

The measurement model:Item responses conditionally independentgiven person & item parameters

Distributions for item parametersconditional on item features

Note on extensions:•Rating scale, count, vector-valued observations.•Multivariate student models for multiple ways people differ; conditional independence given vector . •Different item features relevant to different components of


A full Bayesian model for LLTM

*

*

, , , ,

| , | | , .ij i j i j ji j

p q

p x p p q p p

x


•For people, posterior distributions for s, or propensity to make correct responses.•How/why interpreted through psychological model. •For items, posterior distributions for s. •Still resumed to be the same for all people.•Some harder, some easier; How/why inside model.•Hypothesized patterns can be checked statistically.


Structured mixtures of IRT models

What makes items hard may depend on solution strategy. » John French (1965) on mental rotation.» Siegler re balance beam--development. » Gentner & Gentner on electrical circuits.» Tatsuoka on mixed number subtraction

Theory says what the relationship ought to be; the trick is putting it into a probability model.



For mental rotation items: Difficulty depends on angle of rotation if mental

rotation strategy. Difficulty depends on acuteness of angle if analytic

strategy. Can make inferences about group membership using

items that are relatively hard under one strategy, relatively easy under the other.

Mislevy, R.J., Wingersky, M.S., Irvine, S.H., & Dann, P.L. (1991).

Resolving mixtures of strategies in spatial visualization tasks.

British Journal of Mathematical and Statistical Psychology, 44,

265-288.


easier harder



less able rotatemore

able

Less rotation More rotation

A structured IRT mixture

easier harder


Person B Person A Person D

less able analyticmore

able

More acute Less acute

OR



Groups of people distinguished by the way they solve tasks.

ik =1 if Person i is in Group k, 0 if not. People differ as to knowledge, skills, proficiencies

within group, expressed by ik’s. Items differ as to knowledge, skills, demands within

group, expressed by qjk’s. Thus, LLTM models within groups. Conditional independence of responses given

person, item, and group parameters.



Consider M strategies; each person applies one of them to all items, and item difficulty under strategy m depends on features of the task that are relevant under this strategy in accordance with an LLTM structure.

The difficulty of item j under strategy m is The probability of a correct response is

jm jmk mkk

b q

Pr 1 , , , .i

ij i i j im jmk mkkm

X q q



1

, , , , , ,

| , , | | ,ik

ij ik jk i ik k i jk jk k k ki j k

p X q

p X p p p q p p

Item responses conditionally independentgiven person’s group and person & item parameters relevant to that group.

Distributions for item parametersconditional on item features and feature effects relevant to each group



*

*

1

, , , , , ,

| , , | | ,ik

ij ik jk i ik k i jk jk k k ki j k

p q

p x p p p q p p

x


•For people, posterior probs for s, or group memberships.•Posterior distributions for s within groups.•How/why interpreted through psychological model. •For items, posterior distributions for s for each group. •Items differentially difficult for different people, based on theories about what makes items hard under different strategies.


Conclusion

Evolution of psychometric models joint with psychology.

Bayesian probability framework well suited for building models that correspond to “narratives”

Can’t just “throw data over the wall” like was done with CTT;

Need to build coordinated observational model and probability model, from psychological foundation.

A Bayesian Perspective on Structured Mixtures of IRT Models

Documents

Transcript of A Bayesian Perspective on Structured Mixtures of IRT Models