Identifying Aberrant Response Patterns as outlier...

Introduction Estimation and ARPs detection Results from a simulation study Classification of ARPs Conclusions References

Identifying Aberrant Response Patternsas outlier functional data

Pedro DelicadoDepartament d’Estadıstica i Investigacio Operativa

Universitat Politecnica de Catalunya, Barcelona

Eduardo DovalDepartament de Psicobiologia i de Metodologia de les Ciencies de la Salut

Universitat Autonoma de Barcelona

SEA, UAB, April 13th 2018

Partially suported by grants EDU2013-41399-P, MTM2013-43992-R, and MTM2017-88142-P from

Direccion General de Investigacion and Plan Nacional de I+D+i, Ministerio de Economıa y Competitividad.

1/65 Pedro Delicado


Context of the talk

• Global context: Psychometry. Item Response Theory.

• Specific context: Evaluation of multiple choice examinations.

• Goals:• Identification of examinees with an Aberrant Response Pattern.• Characterization of different Aberrant Response Patterns.

• Statistical tools:• Generalized Additive Models (GAM).• Functional Data Analysis (FDA).

2/65 Pedro Delicado


1 IntroductionIntroduction to Functional Data Analysis

Examples of functional dataFormal definition of functional dataFunctional Principal Component Analysis (FPCA)

Introduction to Item Response Theory

IPRS: Item-Person Response SurfacePerson Response Functions and Aberrant Response Patterns

2 Estimation and ARPs detectionDetection of ARPs by functional outliers detectionDetection of ARPs by log-likelihood differences

3 Results from a simulation study

4 Classification of ARPs

5 Conclusions

3/65 Pedro Delicado









5 Conclusions

4/65 Pedro Delicado


Introduction to FDA








5 Conclusions

5/65 Pedro Delicado


Introduction to FDA

Introduction to Functional Data Analysis

• Observing and saving complete functions as the result of randomexperiments is possible by the development of real-timemeasurement instruments and data storage resources.

• Example: For patients involved in a clinical trial the blood pressureis monitored in continuous-time during 24 hours.

• Samples where a whole function is observed at each sampling unitare referred to as Functional Data.

• Random functions are the statistical atoms in these cases. (Ramsayand Silverman 2005).

• Evolution of Statistics:univariate −→ multivariate −→ functional (from 1990, approx.).

6/65 Pedro Delicado


Introduction to FDA

• Functional Data Analysis deals with the statistical description andmodeling of samples of random functions.

• A broad outlook to Functional Data Analysis is given in the books ofRamsay and Silverman (1997, Second Edition in 2005), Ramsay andSilverman (2002), Ferraty and Vieu (2006), Horvath and Kokoszka(2012), and Kokoszka and Reimherr (2017).

• See also the review papers Cuevas (2013), Wang, Chiou, and Muller(2016) and Reiss, Goldsmith, Shang, and Ogden (2017).

• R libraries: fda (Ramsay and Silverman 2005),fda.usc (Febrero-Bande and Oviedo de la Fuente 2012).

• R task view devoted to FDA:cran.r-project.org/web/views/FunctionalData.html

7/65 Pedro Delicado

cran.r-project.org/web/views/FunctionalData.html


Introduction to FDA

Examples of functional data: Growth data

Heights of 39 boys and 54 girls measured at a set of 31 ages (from 1 to18 years) in the Berkeley Growth Study. Only 10 are shown.

5 10 15

6080

100

120

140

160

180

200

Height for males

Years

5 10 15

6080

100

120

140

160

180

200

Height for females

Years

> library(fda); data(growth); demo(growth)

8/65 Pedro Delicado


Introduction to FDA

The derivatives of functional data are new functional data, that can be asinformative as the original functions (or even more!).

Height velocities and accelerations of 10 girls in the Berkeley Growth Study.

9/65 Pedro Delicado


Introduction to FDA

Examples of functional data: Canadian weather data

0 100 200 300

−30

−10

010

20

Temperature

Day

Deg

C

0 100 200 3000

24

68

1012

Precipitation

Day

Deg

C

Average (over 30 years) daily temperature and precipitation in 35weather stations, Canada.> library(fda); data(daily); demo(daily)

10/65 Pedro Delicado


Introduction to FDA

Each station has its own geographical location

Arvida Bagottville

Calgary

Charlottvl

Churchill

Dawson

Edmonton

FrederictonHalifax

Inuvik

Iqaluit

Kamloops

London

Montreal Ottawa

Pr.Albert Pr.George Pr.Rupert

Quebec

Regina

Resolute

Scheffervll

Sherbrooke St.Johns

Sydney

ThePas

Thunderbay

Toronto

UraniumCty

Vancouver Victoria

Whitehorse

Winnipeg

Yarmouth

Yellowknife

0 100 200 300

-30

-20

-10

01

02

0

Day

Te

mp

era

ture

(de

gre

es C

)

Spatial dependence: Data corresponding to two stations that are close toeach other are more similar than those corresponding to two stations thatare far away each from the other.



Introduction to FDA

Available statistical techniques in FDA

• Most of them are versions of standard statistical methods adaptedto functional data:• Regression models (lm, glm, non-parametric regression, ...).• Multivariate Analysis (PCA, MDS, Clustering, Depth measures, ...).• Time Series, Spatial Statistics, ...

• Others are specific for this kind of data because they exploit thenature of functions:• Principal Differential Analysis is a kind of principal component

analysis made on the derivatives of the observed functions.• Registration is a pre-process step where a change of variable is done

in each observed function ir order to made them as similar aspossible.



Introduction to FDA

Formal definition of Functional Data

• Ferrati and Vieu (2006) define a functional random variable as arandom variable χ taking values in an infinite dimensional space (orfunctional space).

• An observation x of χ is called a functional data.

• A functional data set x1, . . . , xn is the observation of n functionalvariables χ1, . . . ,χn identically distributed as χ.

• Let T = [a, b] ⊆ R. Usually we work with functional data that areelements of

L2(T ) = f : T → R, such that

∫T

f (t)2dt <∞.

• L2(T ) with the usual inner product 〈f , g〉 =∫T f (t)g(t)dt is a

Hilbert space.



Introduction to FDA

Functional Depth

• Depth measure: How deep a data point is in the sample.

• In univariate data, the median is the deepest point of a dataset.

• Different depth notions have been proposed for multivariate data.

• Functional depth: (Cuevas, Febrero-Bande, and Fraiman 2007).• integration over the argument of univariate depth measures,• extensions of multivariate depths,• modal depth.

• Depth measures are useful:• to define robust location and dispersion measures,• for classification,• for outlier detection.



Introduction to FDA

Functional Modal Depth

• Given a functional data xi , how densely surrounded is it by otherdata in the functional dataset?

• The functional mode is the functional data most densely surroundedby other.

• Given a metric or a semi-metric d(·, ·) between functions, for fixedh > 0 the h-depth is defined as

Dh(xi ) =∑j 6=i

1

hK

(d(xi , xj)

h

)

where K (z) is a kernel function (a unimodal symmetric univariatedensity function, typically the standard normal density).

• Choice of the tuning parameter h: In fda.usc h is selected, bydefault, as the the 15%-quantile of d(xj , xk), j , k = 1, . . . , n, j 6= k .



Introduction to FDA

Example: Modal Depth for Canadian Weather data

0 100 200 300

−30

−20

−10

010

20

mode Depth

t

X(t

)

mode.tr10%Median



Introduction to FDA

Functional Principal Component Analysis (FPCA)• Consider the random function χ : T = [a, b] −→ R with

E(χ(t)) = µ(t) for all t, and E(∫ b

a(χ(t)− µ(t))2dt

)<∞.

• FPCA problem: To find orthonormal non-random functionsu1(t), . . . , uK (t) from T to R minimizing

E(‖χ− µ−K∑j=1

〈χ− µ, uj〉uj‖2).

• Low dimensional representation of the functional random variable χ:

χ ≈ µ+K∑j=1

〈χ− µ, uj〉uj = µ+K∑j=1

Zjuj .

• Once the functions u1(t), . . . , uK (t) have been determined,χ(t), t ∈ [a, b], is represented by (Z1, . . . ,ZK ), Zj = 〈χ− µ, uj〉.



Introduction to FDA

• Let C (t, u) = Cov(χ(t),χ(u)) be the covariance function of χ.

• Associated with C (t, u) is the covariance operator:

Γ : L2(T ) −→ L2(T )f 7→ Γ(f ) :T −→ R

t 7→ Γ(f )(t) =∫T C (t, u)f (u)du.

• Let λj and ψj , j ≥ 1, be the eigenvalues and eigenfunctions,respectively, of the covariance operator Γ:

Γ(ψj) = λjψj , j ≥ 1.



Introduction to FDA

Karhunen-Loeve expansion

• Assuming E(∫ b


)= E

(‖χ− µ‖2

)<∞, the

random function χ(t) can be expressed as

χ(t) = µ(t) +∞∑j=1

Zjψj(t),

known as the Karhunen-Loeve expansion of χ, where

Zj = 〈χ− µ, ψj〉, j ≥ 1,

is a sequence of random variables with E(Zj) = 0 and E(Z 2j ) = λj ,

that are uncorrelated: E(ZjZh) = 0 for j 6= h.

• When the random function χ is Gaussian, then Zj are independentGaussian random variables.

• Moreover E(∫ b


)= E

(‖χ− µ‖2

)=∑∞

j=1 λj .



Introduction to FDA

Karhunen-Loeve expansion and FPCA

• Assuming λj > λj+1 for all j ≥ 1 and fixing an integer K , theorthonormal basis u1, . . . , uK solving the minimization problem

minu1,...,uK

E(‖χ− µ−K∑j=1

〈χ− µ, uj 〉uj‖2),

is formed by the first K eigenfunctions of the covariance operator,ψ1, . . . , ψK , and the minimum is

∑j≥K+1 λj .

• That is, the Karhunen-Loeve expansion of χ− µ, truncated at thefirst K terms, gives the best K -dimensional approximation to χ− µ,in the sense of mean square error:

χ(t) ≈ µ(t) +K∑j=1

Zjψj (t), t ∈ [a, b].

• Eigenfunctions or Principal Functions: ψj , j ≥ 1.

• Scores or Functional Principal Components: Zj = 〈χ− µ, ψj〉, j ≥ 1.



Introduction to FDA

Sampling version of FPCA

• The sampling (or empirical) version of the previous results isobtained when replacing the covariance function of χ by theempirical covariance function.

• Let x1, . . . , xn be n i.i.d. random functions on L2(T ) having thesame distribution as χ. The empirical covariance function is

C (t, u) =1

n

n∑i=1

(xi (t)− x(t))(xi (u)− x(u)),

for all t, u ∈ T = [a, b], where x is the sample mean function:x(t) = (1/n)

∑ni=1 xi (t).

• In practice the eigensystem of the operator with kernel C (t, u) isobtained by matrix diagonalization.

• xi (t) ≈ x(t) +∑K

j=1 Zij ψj(t), t ∈ [a, b], i = 1, . . . , n.



Introduction to FDA

Example: FPCA for Canadian Weather data

0 100 200 300

−30

−20

−10

010

20

Day

Deg

C

0 100 200 300

−0.

050.

000.

05

valu

es

0 100 200 300

−20

−10

010

20

PCA function 1 (Percentage of variability 88.8 )

argvals

Har

mon

ic 1

+++++++++++++

+++++++++

++++++

+++++

++++++

+++++++

+++++++

+++++++++++

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

−−−−−−−−−−−−−

−−−−−−−−−

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

−−−−−−

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

0 100 200 300

−10

010

20

PCA function 2 (Percentage of variability 8.5 )

argvals

Har

mon

ic 2

+++++++++++++

++++++++++

++++++

++++++

+++++++++++++++++

+++++

++++++

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

−−−−−−−−−−−−−

−−−−−−−−−

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

−−−−−−

−−−−−−−−

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

−−−−−

−−−−−−−−−−−



Introduction to IRT








5 Conclusions



Introduction to IRT

A very short introduction to Item Response Theory

• Population: Set of individuals with different abilities for doing a task.

Example: Looking for a new employee• Population: Candidates to be hired for a job.• Ability: Adequation to the job’s requirements.• Task: The set of tasks to be performed at the new job.

Example: Basic skills evaluation• Population: Children in the last year of Primary School (11-12 y.o.).• Ability: Basic skills to perform easy tasks.• Task: Basic tasks in Language, Foreign Language and Mathematics.



Introduction to IRT

Measuring the ability of individuals

• Individuals are identified with their abilities θ ∈ Θ ⊆ R(uni-dimensionality is assumed in the present work).

• Problem: The ability θ of an individual can not be measureddirectly.

• So it is measured indirectly by a questionnaire or exam: A set ofquestions (items) to be answered by the individuals.

• The questionnaire is designed to measure the ability of interest:• Individuals with higher ability are more likely to give right answers to

the questions.• Different questions have different difficulty: A given individual has

larger probability of giving the right answer to questions with lowerdifficulties.

• Questions are identified with their difficulties b ∈ Ω ⊆ R.

• The difficulty b of a question can not be measured directly.



Introduction to IRT

Multiple choice exam (or questionaire)

• A set of questions, each presenting a list of k possible answers(k = 4 in this work).

• One of the answers is correct, and all other are considered to beincorrect.

• The incorrect answers are not particularly ordered.

• The respondent (student, examinee, ...) is required to choose oneanswer.

• Typical dataset from a multiple choice exam:• m questions with (unobservable) difficulties b1, . . . , bm.• n examinees with (unobservable) abilities θ1, . . . , θn.• A n ×m matrix X with entries xij ∈ 0, 1, such that xij = 1 if and

only if the i-th examinee has answered correctly to the j-th question.



Introduction to IRT

IPRS: Item-Person Response Surface

p : Θ× Ω −→ [0, 1](θ, b) 7−→ p(θ, b)

• p(θ, b) is the probability that an individual with ability θ gives theright answer to a question of difficulty b.

• p(θ, b) is increasing in θ and decreasing in b.

• Identifiability problem: Abilities, difficulties and IPRS are not wellidentified.• Let τ : Θ −→ Θ∗ and ν : Ω −→ Ω∗ be two increasing monotonic

functions,• Abilities θ∗ = τ(θ), difficulties b∗ = ν(b) and the IPRS

p∗ : Θ∗ × Ω∗ −→ [0, 1](θ∗, b∗) 7−→ p∗(θ∗, b∗) = p(τ−1(θ∗), ν−1(b∗))

are equivalent to θ, b and p(θ, b).

• Only the order of abilities and difficulties is relevant.



Introduction to IRT

A simple statistical model• Let b and θ be two independent random variables with known

distribution functions Fb and Fθ, respectively.

• Let b1, . . . , bm be m i.i.d. observations from b.

• Let θ1, . . . , θn be n i.i.d. observations from θ.

• Given m questions with difficulties b1, . . . , bm, n examinees withabilities θ1, . . . , θn, and the IPRS p(θ, b), let

Xij ∼ Bern(p(θi , bj)), i = 1, . . . , n, j = 1, . . . ,m,

be n ×m independent binary random variables.

• The entries xij in the data matrix X are assumed to be therealization of random variables Xij .

• Inference goals:• Main: To estimate (or to predict) the abilities θ1, . . . , θn.• To estimate (or to predict) the difficulties b1, . . . , bm.• To estimate the IPRS p(θ, b).



Introduction to IRT

−2 −1 0 1 2

−2

−1

01

2IRPS p(θ, b) and data X

θ

b



Introduction to IRT

Simple estimators• For the i-th examinee, compute its total score in the exam

ti =∑m

j=1 xij .• Compute the ranks of t1, . . . , tn: r1, . . . , rn and define

θi = F−1θ

(ri − 1/2

n

).

• Compute the ranks si , . . . , sm for the number of wrong responses toquestions 1, . . . ,m, and define

bj = F−1b

(si − 1/2

m

).

• Monotonicity of p(θ, b) is a sufficient condition for consistency of θiand bj (see, e.g., Ramsay 1991).

• Fit a parametric or non-parametric regression model to the data

(θi , bj , xij), i = 1, . . . , n, j = 1, . . . ,m

to estimate p(θ, b). See, for instance, Ramsay (2000).



Introduction to IRT

−2 −1 0 1 2

−2

−1

01

2Estimation of θ

θi

θ i

−2 −1 0 1 2

−2

−1

01

2

Estimation of b

bj

b j

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Estimation of p(θ, b)

p(θi, bj)

p(θ i

, bj)

−6 −4 −2 0 2 4 6

−6

−4

−2

02

46

Estimation of logit(p(θ, b))

logit(p(θi, bj))

logi

t(p(θ

i, b j

))



Introduction to IRT

thet

a

b

p(theta, b)th

eta

b

hat(p)(theta, b)

−3 −2 −1 0 1 2 3

−3

−2

−1

01

23

p(θ, b)

θ

b

−3 −2 −1 0 1 2 3

−3

−2

−1

01

23

p(θ, b)

θ

b



Introduction to IRT

PRF: Person Response Function• For the i-th individual, its Person Response Function, defined as

PRFi : Ω −→ [0, 1]b 7−→ PRFi (b),

computes its probability of giving a right answer to a question withdifficulty b.

• Under the previous simple model, if θi is the ability of individual i-th,

PRFi (b) = p(θi , b).

• Definition: Aberrant Response Pattern (ARP).We say that the i-th individual follow an Aberrant Response Patternwhen the functions PRFi (·) and p(θi , ·) are different:∫

Ω

(PRFi (b)− p(θi , b))2f (b)db > 0.



Introduction to IRT

A model including Aberrant Response Patterns (ARPs)

• Consider the difficulties b1, . . . , bm and the abilities θ1, . . . , θngenerated as it has been explained before.

• Let p(θ, b) be the IPRS defined as before.

• Let πA ∈ [0, 1] be the probability that an examinee has an ARP.

• For i = 1, . . . , n define the i-th PRF as

PRFi (b) =

p(θi , b) with probability 1− πA,

gi (b) with probability πA,

where gi (b) is an ARP.

• The random selection of individuals with ARPs is independent fromeach other.

• Some possible types of ARPs (according to Karabatsos 2003):cheaters, lucky-guessers, creative respondents, careless respondents,and random respondents.



Introduction to IRT

Some types of Aberrant Response Patterns

• Cheaters, low ability examinees that unfairly obtain the correctanswers on test items that they are unable to answer correctly. Forinstance, they copy from another examinee.

• Lucky-guessers guess the correct answers to some test items, forwhich they do not know the correct answer.

• Creative respondents examinees with high ability that have incorrectresponses to certain easy items, because they interpret these itemsin a creative manner.

• Careless respondents answers certain items incorrectly, for whichthey are able to answer correctly.

• Random respondents randomly choose the multiple-choice option foreach item on the test.



Introduction to IRT

20 40 60 80 100

−2

−1

01

PRFi(b), n=100, m=50, 10 ARPs

Rank of θi

b



Introduction to IRT

Objective: Detection of ARPs

• Relevance of ARPs: They can cause an examinee’s score on a test tobe spuriously high or spuriously low.

• In both cases, a direct interpretation of the results leads to a wrongevaluation of examinees’ ability.

• Therefore, the identification of ARPs has a great practicalimportance.

• Our first goal: To detect examinees with ARP.

• Idea:• To estimate the i-th individual’s PRF in two different ways,

• Using data from all the individuals, as a profile of the estimated IPRS.• Using only data from the i-th individual.

• To look for individuals for which both estimations are abnormallydifferent.

• Differences between estimations are functional data.

• Functional outliers detection methods will help to uncover ARPs.










5 Conclusions



Estimation of IPRS. An iterative procedure (I).

• Initial step:• k = 0.• Compute the ranks r 0

1 , . . . , r0n of the total score of individuals

1, . . . , n and define

θ0i = F−1

θ

(r 0i − 1/2

n

).

• Compute the ranks s0i , . . . , s

0m for the number of right responses to

questions 1, . . . ,m, and define

b0j = F−1

b

(s0i − 1/2

m

).



Estimation of IPRS. An iterative procedure (II).• Iterate until convergence:

• k = k + 1.• Fit a GAM model with logistic link to the data

(θk−1i , bk−1

j , xij), i = 1, . . . , n, j = 1, . . . ,m

and call pk(θ, b) the estimated IPRS.• Estimate by maximum likelihood the abilities and difficulties:

θML,ki = arg max

θ

m∏j=1

pk (θ, bk−1j )xij (1− pk (θ, bk−1

j ))(1−xij ),

bML,kj = arg max

b

n∏i=1

pk (θk−1i , b)xij (1− pk (θk−1

i , b))(1−xij ).

• Compute the ranks r k1 , . . . , rkn of θML,k

i , i = 1, . . . , n, and define

θki = F−1θ

((rki − 1/2)/n

).

• Compute the ranks ski , . . . , skm for bML,k

j , j = 1, . . . ,m, and define

bkj = F−1b

((ski − 1/2)/m

).



Estimation of PRFi , i = 1, . . . , n

• For j = 1, . . . ,m, let bj be the estimation bj coming from the laststep of the IRPS estimation.

• For i = 1, . . . , n,, fit a nonparametric estimator with logistic link (forinstance, local likelihood estimation, as in Loader 1999, or penalizedspline regression, as in Wood 2017) to the data

(bj , xij), j = 1, . . . ,m,

and call PRFi (b) the estimated function.



20 40 60 80 100

−3

−2

−1

01

23

Estimated IPRS(θi, b)

Rank of θi

b

20 40 60 80 100−

3−

2−

10

12

3

Estimated PRFi(b)

Rank of θi

b



Detection of ARPs by functional outliers detection








5 Conclusions




Detection of ARPs

• For i = 1, . . . , n, define the functional data

Di (b) = logit(

PRFi (b))− logit

(p(θi , b)

), b ∈ Ω.

• If individual i has not an ARP, Di (b) should be close to 0 for all

b ∈ Ω because PRFi (b) and p(θi , b) are both estimating the samefunction of b: p(θi , b)

• When the i-th individual has an ARP, Di (b) is expected to be farfrom 0, because ...• p(θi , b) is estimating p(θi , b), specially when the non-parametric

bivariate estimator is not so flexible, as it is the case for GAMs,• PRFi (b) is estimating gi (p).

• Looking for functional outliers in the functional data setDi (b) : i = 1, . . . , n, results in a way to look for ARPs.




−2 −1 0 1 2

−20

−15

−10

−5

05

10Di(b)= logit(PRFi(b)) − logit(IPRS(θi, b))

b

Di(b

)

chgucrcara



Detection of ARPs by log-likelihood differences








5 Conclusions




Detection of ARPs by log-likelihood differences• We also explore a different approach for detecting ARPs.

• Consider the i-th individual. There are two ways to compute thelog-likelihood of the observed data xij , j = 1, . . . ,m.

• The first one uses the estimated PRF:

lglik.PRFi =n∑

j=1

xij log

(PRFi (bj)

)+ (1− xij) log

(1− PRFi (bj)

).

• The second one uses the estimated IPRS:

lglik.IPRSi =n∑

j=1

xij log

(p(θi , bj)

)+ (1− xij) log

(1− p(θi , bj)

).

• It is expected that lglik.PRFi − lglik.IPRSi takes abnormally highvalues for individuals with ARPs.

• Look for outliers in dataset lglik.PRFi − lglik.IPRSi : i = 1, . . . , n.47/65 Pedro Delicado



−40 −30 −20 −10

−25

−20

−15

−10

−5

ARPs detection by log−likelihood differences

lglk.IPRS

lglk

.PR

F

lglik.

PRF −

lglik.

IPRS =

0

lglik.

PRF −

lglik.

IPRS =

upp

er w

hiske

r

chgucrcara










5 Conclusions



Results from a simulation study

• Number of replicates: 50.

• n = 500 examinees, m = 50 items.

• p(θ, b) = 1/(1 + exp(−D(θ − b)) for D = 1.7.

• Three different proportions of ARPs: 0.05, 0.1, 0.25.

• Six types of ARPs: cheaters, lucky-guessers, creative, careless,random, mixed.

• Five ways to detect ARPs:• log-likelihood differences,• outliers in the scores of the first Functional PC of Di (b)i=1,...,n,• outliers in the scores of the first Functional PC of D ′i (b)i=1,...,n,• functional outliers detection in Di (b)i=1,...,n: we use the functionoutliers.depth.trim with modal depth, from library fda.usc

(Febrero-Bande and Oviedo de la Fuente 2012),• functional outliers detection in D ′i (b)i=1,...,n.



Simulating 6 types of ARPs• Cheaters: θi < z.375,

gi (b) = 1 if b > z.85, otherwise gi (b) = p(θi , b).

• Lucky-guessers: θi < z.375,

gi (b) = .25 if b > z.85, otherwise gi (b) = p(θi , b).

• Creative respondents: θi > z.625,

gi (b) = 0 if b < z.15, otherwise gi (b) = p(θi , b).

• Careless: θi > z.625,

gi (b) = .5 if b < z.15, otherwise gi (b) = p(θi , b).

• Random respondents: gi (b) = .25.

• Mixed: One fifth of the simulated ARP comes from each other types.



Evaluating the ability of a method to detect ARPs

ARP detection

TruthNo ARP ARP

No ARP TN: True Negative FP: False PositiveARP FN: False Negative TP: True Positive

• Sensitivity = TPTP+FN

• Specificity = TNTN+FP



True Positive Proportion

Type of ARP

Met

hod

of o

utlie

r de

tect

ion

by

Pro

port

ion

of A

RP

s

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

0.00 0.05 0.10 0.15

1cheaters0.05

2guessing0.05

0.00 0.05 0.10 0.15

3creative0.05

4careless0.05

0.00 0.05 0.10 0.15

5random0.05

6mixed0.05

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.1

2guessing0.1

3creative0.1

4careless0.1

5random0.1

6mixed0.1

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.25

0.00 0.05 0.10 0.15

2guessing0.25

3creative0.25

0.00 0.05 0.10 0.15

4careless0.25

5random0.25

0.00 0.05 0.10 0.15

6mixed0.25



Proportion of Bad Classified: FP + FN

Type of ARP

Met

hod

of o

utlie

r de

tect

ion

by

Pro

port

ion

pf A

RP

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

0.0 0.1 0.2 0.3 0.4

1cheaters0.05

2guessing0.05

0.0 0.1 0.2 0.3 0.4

3creative0.05

4careless0.05

0.0 0.1 0.2 0.3 0.4

5random0.05

6mixed0.05

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.1

2guessing0.1

3creative0.1

4careless0.1

5random0.1

6mixed0.1

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.25

0.0 0.1 0.2 0.3 0.4

2guessing0.25

3creative0.25

0.0 0.1 0.2 0.3 0.4

4careless0.25

5random0.25

0.0 0.1 0.2 0.3 0.4

6mixed0.25



Sensitivity

Type of ARP

Met

hod

of o

utlie

r de

tect

ion

by

Pro

port

ion

pf A

RP

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

0.0 0.2 0.4 0.6 0.8 1.0

1cheaters0.05

2guessing0.05

0.0 0.2 0.4 0.6 0.8 1.0

3creative0.05

4careless0.05

0.0 0.2 0.4 0.6 0.8 1.0

5random0.05

6mixed0.05

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.1

2guessing0.1

3creative0.1

4careless0.1

5random0.1

6mixed0.1

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.25

0.0 0.2 0.4 0.6 0.8 1.0

2guessing0.25

3creative0.25

0.0 0.2 0.4 0.6 0.8 1.0

4careless0.25

5random0.25

0.0 0.2 0.4 0.6 0.8 1.0

6mixed0.25



Specificity

Type of ARP

Met

hod

of o

utlie

r de

tect

ion

by

Pro

port

ion

pf A

RP

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

0.750.800.850.900.951.00

1cheaters0.05

2guessing0.05

0.750.800.850.900.951.00

3creative0.05

4careless0.05

0.750.800.850.900.951.00

5random0.05

6mixed0.05

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.1

2guessing0.1

3creative0.1

4careless0.1

5random0.1

6mixed0.1

1LikRat

2FPC1

3FPC1.d1

4Foutl

5Foutl.d1

1cheaters0.25

0.750.800.850.900.951.00

2guessing0.25

3creative0.25

0.750.800.850.900.951.00

4careless0.25

5random0.25

0.750.800.850.900.951.00

6mixed0.25










5 Conclusions



Classification of ARPs

• We consider one of the simulated datasets, corresponding to aproportion of 10% of ARPs of mixed type.

• Sample size 500: 50 ARPs (10 of each type).

• ARP identification results:

No ARP Cheaters Guessers Creative Careless Random TOTALTrueARPs

10 10 10 10 10 50

Identifiedas ARPs

9 10 6 10 8 10 53



• Two clustering algorithms for functional data have been applied tothe subset of 53 cases identified as ARP.

• k-means (k = 5) and hierarchical clustering (Ward’s method).

• The working functional dataset is formed by

Gj(b) = logit(PRFj(b)), j = 1, . . . , 53.

• Distance: weighted L2-norm of the logit difference functions:

dj,k = d(Gj ,Gk) =

(∫ 2

−2

(Gj(b)− Gk(b))2φ(b)db

)1/2

,

where φ is the density function of a standard normal.



k-means Clusters by k-meansTruth gray gray red blue magentano ARP 3 4 0 2 0cheaters 0 0 10 0 0guessing 0 0 3 0 3creative 0 0 0 9 1careless 0 0 0 7 1random 0 0 2 0 8

−2 −1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

Clusters by k−means

b

IRF

(b)



Hierarchical clusteringcr

eativ

e

crea

tive

crea

tive

crea

tive

care

less

care

less

crea

tive

crea

tive

good

care

less

care

less

good

care

less

crea

tive

crea

tive

care

less

crea

tive

crea

tive

care

less

care

less

good

good

good

good

gues

sing

rand

om

rand

om

gues

sing

rand

om

rand

om

gues

sing

gues

sing

rand

om

rand

om

rand

om

gues

sing

gues

sing

rand

om

rand

om

chea

ters

chea

ters

rand

om

chea

ters

chea

ters

chea

ters

chea

ters

chea

ters

chea

ters

chea

ters

chea

ters

good

good

good

Cluster Dendrogram



Clusters by hierarchical clusteringgreen-

Truth gray gray gray gray red blue cyan magentano ARP 1 2 3 1 0 0 2 0cheaters 0 0 0 0 10 0 0 0guessing 0 0 0 0 0 0 0 6creative 0 0 0 0 0 6 4 0careless 0 0 0 0 0 2 6 0random 0 0 0 0 1 0 0 9

−2 −1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

Hierarchical clustering

b

IRF

(b)



Clusters by hierarchical clusteringClusters green-by k-means gray gray gray gray red blue cyan magentagray 1 2 0 0 0 0 0 0gray 0 0 3 1 0 0 0 0red 0 0 0 0 11 0 0 4blue 0 0 0 0 0 6 12 0magenta 0 0 0 0 0 2 0 11

−2 −1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

Clusters by k−means

b

IRF

(b)

−2 −1 0 1 2

0.0

0.2

0.4

0.6

0.8

1.0

Hierarchical clustering

b

IRF

(b)










5 Conclusions



Conclusions

• The non-parametric estimation of IPRS and PRF, combined withFunctional Data Analysis, are fruitful in Item Response Theory.

• In particular, ARPs detection can be carried out by combining thenon-parametric estimation of IPRS and PRFs, with functionaloutliers detection.

• Nevertheless, our simulation study indicates that there is a simplerand faster method, based on computing log-likelihood differences,based on the global IPRS and the individuals PRFs.

• We also have seen that clustering techniques for functional data areuseful for characterizing the PRFs of individuals detected as ARPs.

• Further work:• To test these techniques in a dataset from a real exam.• To explore other applications of FDA to IRT.



Cuevas, A. (2013).A partial overview of the theory of statistics with functional data.Journal of Statistical Planning and Inference 147, 1–23.

Cuevas, A., M. Febrero-Bande, and R. Fraiman (2007).Robust estimation and classification for functional data via projection-based depth notions.Computational Statistics 22(3), 481–496.

Febrero-Bande, M. and M. Oviedo de la Fuente (2012).Statistical computing in functional data analysis: the R package fda.usc.Journal of Statistical Software 51(4), 1–28.

Ferraty, F. and P. Vieu (2006).Non parametric functional data analysis. Theory and practice.Springer.

Horvath, L. and P. Kokoszka (2012).Inference for functional data with applications.Springer Science & Business Media.

Karabatsos, G. (2003).Comparing the aberrant response detection performance of thirty-six person-fit statistics.Applied Measurement in Education 16, 277–298.

Kokoszka, P. and M. Reimherr (2017).Introduction to functional data analysis.CRC Press.

Loader, C. (1999).Local regression and likelihood.Springer.

Ramsay, J. and B. Silverman (2002).Applied Functional Data Analysis.USA: Springer.

Ramsay, J. O. (1991).Kernel smoothing approaches to nonparametric item characteristic curve estimation.Psychometrika 56, 611–630.



Ramsay, J. O. (2000).TestGraf. A Program for the Graphical Analysis of Multiple Choice Test and Questionnaire Data.McGill University.

Ramsay, J. O. and B. W. Silverman (2005).Functional Data Analysis (Second ed.).New York: Springer.

Reiss, P. T., J. Goldsmith, H. L. Shang, and R. T. Ogden (2017).Methods for scalar-on-function regression.International Statistical Review 85(2), 228–249.

Wang, J., J. Chiou, and H. Muller (2016).Functional data analysis.Annual Review of Statistics and Its Application 3(1), 257–295.

Wood, S. N. (2017).Generalized Additive Models: An Introduction with R (2 ed.).Chapman and Hall/CRC.


Identifying Aberrant Response Patterns as outlier...

Documents

Transcript of Identifying Aberrant Response Patterns as outlier...