Modelling atypical students response patterns using multidimensional parametric models

Montréal, CSSE / CERA 2010 1.Raîche, Béland, Magis, Blais and Brochu

Modelling atypical students response patterns using multidimensional parametric models

Gilles Raîche, UQAMSébastien Béland, UQAM

David Magis, Université de LiègeJean-Guy Blais, Université de Montréal

Pierre Brochu, CMEC

Large-Scale Assessments: Policy, Research and PracticeCSSE / CERA

Montréal, 2010


• Introduction and Objectives• Unidimensional IRT Models• IRT Person Parameters Models

– Person response Curve– Multidimensional Item Response Models– Estimation– An R Package: irtProb

• Examples• Other Considerations• References and contacts

SUMMARY


• Presentation

• IRT Models of Interest– Unidimensional latent proficiency– Dichotomous response– Monotonic– Logistic Probability Distribution

INTRODUCTION


• Simulation of Inappropriate Response Patterns

• Person Misfit Detection Indices

• Distributional Properties of Person Misfit Indices

• Adjusted Proficiency Level Estimation in Presence of Person Misfit

OBJECTIVES


UNIDIMENSIONAL IRT MODELS

3 Parameter Logistic (3PL) (Birnbaum, 1968)

4 Parameters Logistic (4PL) (McDonald, 1967)

where if ai is considered as a standard deviation

)(1

)(),,,|1(

iji baii

iiiijijije

cccabXPP

)(1

)1(),,,|1(

iji bai

iiiijijije

cccabXPP

2

1

i

is

a


PERSON RESPONSE CURVE

(Trabin and Weiss, 1983)-4 -2 0 2 4

0.0

0.2

0.4

0.6

0.8

1.0

-- Difficulty (b) +

Pro

ba

bili

ty

0-200

3-100

000.30

0300.2


MULTIDIMENSIONAL ITEM RESPONSE MODELS• Personal Variance (σ2) (Ferrando, 2004; Thurstone, 1927)

• Personal Inattention (δ)

• • Personal Pseudo-Guessing (χ) (Strandmark and Linn, 1987)

2

)(

1

),,,,,|1(

i

ij

s

b

ijiijjiiiiij

e

cdcdcbsxP

22

)(

1

),,,,,|1(

ji

ij

s

bii

ijjiiiiij

e

cdcdcbsxP

2

)(

1

),,,,,|1(

i

ij

s

b

jiijijjiiiiij

e

cdcdcbsxP


MULTIDIMENSIONAL ITEM RESPONSE MODELS• Higher Order Models

22

)(

1

),,,,,,|1() and ,(

ji

ij

s

b

jiiijjjiiiiij

e

cdcdcbsxP

22

)(

1

)(),,,,,,|1() and ,(

ji

ij

s

b

ijiijjjiiiiij

e

cdcdcbsxP

22

)(

1

)(),,,,,,,|1() and ,,(

ji

ij

s

b

jijiijjjjiiiiij

e

cdcdcbsxP


• Package: irtProb• MAP Estimators• A Priori Probability Distribution

– σ : U(0,4) – θ: U(-4,4)– X: U(0,1)– δ : U(0,1)

ESTIMATION OF SUBJECT PARAMETERS


• Available on R Cran Site• Functionnalities

– Estimation of Person Parameters (MAP)– Likelihood Curves– Person Characteristic Curves– Probability, Density and Random Functions– Simulation of Response Patterns– Classical <-> IRT Item Parameters– Model Selection

A R PACKAGE: irtProb


EXAMPLES – 01 (X)

Table 1. Person Parameter estimation from 100 simulated subjects attempting to augment their estimated proficiency level to a 40 items test [person parameter (Standard Error)]

Pseudo-Guessing

Model1 0.00 0.10 0.20 0.30 0.40

1 θ = -2 -2.18 (0.62)

-1.28 (0.69)

-0.41 (0.69)

0.03 (0.83)

1.00 (0.72)

2 θ = -2C

-2.18 -2.18 (0.62)(0.62)

0.00 0.00 (0.01)(0.01)

-1.75 -1.75 (0.69)(0.69)

0.06 0.06 (0.07)(0.07)

-1.55 -1.55 (0.70)(0.70)

0.16 0.16 (0.10)(0.10)

-1.96 -1.96 (0.88)(0.88)

0.26 0.26 (0.12)(0.12)

-1.64 -1.64 (1.18)(1.18)

0.35 0.35 (0.13)(0.13)

3 θ = -2CSD

-1.98 (0.64)0.01

(0.02)0.07

(0.23)0.03

(0.08)

-1.61 -1.61 (0.74)(0.74)

0.07 0.07 (0.08)(0.08)

0.17 0.17 (0.44)(0.44)

0.03 0.03 (0.07)(0.07)

-1.46 -1.46 (0.79)(0.79)

0.17 0.17 (0.11)(0.11)

0.23 0.23 (0.51)(0.51)

0.03 0.03 (0.06)(0.06)

-1.62 -1.62 (1.10)(1.10)

0.24 0.24 (0.14)(0.14)

0.44 0.44 (0.92)(0.92)

0.03 0.03 (0.07)(0.07)

-1.22 -1.22 (1.59)(1.59)

0.33 0.33 (0.17)(0.17)

0.43 0.43 (0.87)(0.87)

0.02 0.02 (0.05)(0.05)

1 σ = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated sujects Model 1: θ only Model 2: θ and Pseudo-Guessing Model 3 σ, θ, Pseudo-Guessing and δ


EXAMPLES – 01 (X)

(θ=-2, X=0.2) (θ=-2, X=0.4)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0e+00

2e-11

4e-11

6e-11

8e-11

P(X)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0.0e+00

5.0e-11

1.0e-10

1.5e-10

2.0e-10

2.5e-10

P(X)


EXAMPLES – 02 (X)

Table 2. Person Parameter estimation from 100 simulated subjects attempting to augment their estimated proficiency level to a 40 items test [person parameter (Standard Error)]

Pseudo-Guessing

Model1 0.00 0.10 0.20 0.30 0.40

1 θ = 2 2.19 (0.55) 2.52 (0.61) 3.02 (0.58) 3.31 (0.62) 3.46 (0.56)

2 θ = 2

C

2.09 (0.61)

0.02 (0.06)

2.37 (0.60)2.37 (0.60)

0.04 (0.10)

2.52 (0.85)

0.12 (0.18)

2.74 (0.88)

0.16 (0.21)

2.51 (1.01)

0.27 (0.23)

3 θ = 2

S

C

D

1.95 (0.55)

0.11 (0.27)

0.02 (0.05)

0.01 (0.02)

2.11 (0.61)2.11 (0.61)

0.17 (0.34)0.17 (0.34)

0.08 (0.13)0.08 (0.13)

0.00 (0.01)0.00 (0.01)

2.10 (0.93)2.10 (0.93)

0.12 (0.32)0.12 (0.32)

0.20 (0.19)0.20 (0.19)

0.00 (0.01)0.00 (0.01)

2.42 (0.90)2.42 (0.90)

0.12 (0.40)0.12 (0.40)

0.22 (0.21)0.22 (0.21)

0.00 (0.01)0.00 (0.01)

2.02 (1.08)2.02 (1.08)

0.09 (0.30)0.09 (0.30)

0.35 (0.23)0.35 (0.23)

0.00 (0.01)0.00 (0.01)1 σ = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated subjects Model 1: θ only Model 2: θ and Pseudo-Guessing Model 3 σ, θ, Pseudo-Guessing and δ


EXAMPLES – 02 (X)

(θ=2, X=0.2) (θ=2, X=0.4)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0e+00

2e-05

4e-05

6e-05

8e-05

P(X)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0.0e+00

5.0e-06

1.0e-05

1.5e-05

2.0e-05

2.5e-05

3.0e-05

P(X)


EXAMPLES – 03 (σ)

Table 3. Person Parameter estimation from 100 simulated subjects with fluctuating proficiency level to a 40 items test [person parameter (Standard Error)]

Fluctuation

Model1 0.00 0.50 1.00 2.00 4.00

1 θ = -2 -2.14 (0.59) -2.07 (0.61) -2.00 (0.62) -1.71 (0.70) -1.01 (0.86)

2 θ = -2

S

-2.06 (0.58)

0.33 (0.42)

-2.03 (0.59)

0.51 (0.57)

-2.05 (0.67)

0.90 (0.68)

-2.00 (0.84)

1.87 (0.90)

-1.63 (1.28)

3.45 (0.75)

3 θ = -2

S

C

D

-1.92 (0.66)

0.11 (0.24)

0.01 (0.01)

0.04 (0.09)

-1.75 (0.79)

0.15 (0.29)

0.01 (0.02)

0.06 (0.11)

-1.54 (0.83)

0.28 (0.45)

0.01 (0.03)

0.11 (0.14)

-0.97 (1.42)

0.71 (0.84)

0.03 (0.05)

0.18 (0.18)

-0.68 (2.37)

0.79 (1.24)

0.15 (0.14)

0.23 (0.21)1 X = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated subjects Model 1: θ only Model 2: θ and σ Model 3 σ, θ, Pseudo-Guessing and δ


EXAMPLES – 03 (σ)

(θ=-2, σ=1) (θ=2, σ=4)

-4

-2

0

2

4

0

1

2

3

4

2.0e-12

4.0e-12

6.0e-12

8.0e-12

1.0e-11

1.2e-11

P(X)

-4

-2

0

2

4

0

1

2

3

4

2e-06

4e-06

6e-06

8e-06

P(X)


OTHER CONSIDERATIONS

• Multidimensional EAP Estimation Very Computer Intensive

• Warm Weighted Likelihood Estimator

• Item Parameters Estimation

• Confidence Interval For The Additionnal Person Parameters

• Other Person Fit Indices: Pseudo-Guessing and Inattention


Barton, M. A. and Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model. Research bullelin 81-20. Princeton, NJ: Educational Testing Service.

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord and M. Novick (Eds): Statistical theories of mental test scores. New York, NJ: Addison-Wesley.

Ferrando, P. J. (2004). Person reliability in personality measurement: an item response theory analysis. Applied Psychological Measurement, 28(2), 126-140.

Hulin, C. L., Drasgow, F., and Parsons, C. K. (1983). Item response theory. Homewood, IL: Irwin.

Levine, M. V., and Drasgow, F. (1983). Appropriateness measurement: validating studies and variable ability models. In D. J. Weiss (Ed.): New horizons in testing. New York, NJ: Academic Press.

Magis, D. (2007). Enhanced estimation methods in IRT. In D. Magis (Ed.): Influence, information and item response theory in discrete data analysis. Doctoral dissertation, Liège, Belgium: University de Liège.

REFERENCES / 1


McDonald, R. P. (1967). Nonlinear factor analysis. Psyhometric Monographs, 15.

Raîche, G., and Blais, J.-G. (2003). Efficacité du dépistage des étudiants et des étudiants qui cherchent à obtenir un résultat faible au test de classement en anglais, langue seconde, au collégial. In J.-G. Blais, and G. Raîche (Ed.): Regards sur la modélisation de la mesure en en éducation et en sciences sociales. Ste-Foy, QC: Presses de l’Université Laval.

Strandmark, N. L. and Linn, R. L. (1987). A generalized logistic item response model parameterizing test score inappropriateness. Applied Psychological Measurement, 11(4), 355-370.

Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286.

Trabin, T. E., and Weiss, D. J. (1983). The person response curve : fit of individuals to item response theory models. In D. J. Weiss (Ed.): New horizons in testing. New York, NJ: Academic Press.

REFERENCES / 2


• Gilles Raîche– http://camri.uqam.ca

• Sébastien Béland– [email protected]

• David Magis– [email protected]

• Jean-Guy Blais– http://www.griemetic.ca

• Pierre Brochu– [email protected]

CONTACTS

Modelling atypical students response patterns using multidimensional parametric models

Documents

Transcript of Modelling atypical students response patterns using multidimensional parametric models