Classical Test Theory vs. Item Response Theory

8/12/2019 Classical Test Theory vs. Item Response Theory

1/30

CLASSICAL TEST THEORY vs. ITEMRESPONSE THEORY

An evaluation of the theory test in theSwedish driving-license test

Marie Wiberg

EM No 50, 2004

ISSN 1103-2685


2/30


3/30

INDEX

INTRODUCTION 1

AIM 2

METHOD: SAMPLE 3

METHOD: CLASSICAL TEST THEORY 3

METHOD: ITEM RESPONSE THEORY 41. Verifying the assumptions of the model 5

A. Unidimensionality 5B. Equal discrimination 5C. Possibility of guessing the correct answer 5

2. Expected model features 63. Model predictions of actual test results 6Estimation methods 7

RESULTS: CLASSICAL TEST THEORY 8Modeling the items 10

RESULTS: ITEM RESPONSE THEORY 101. Verifying the assumptions of the model 10

2. Expected model features 12Invariance of ability estimates 12Invariance of item parameter estimates 13

3. Model predictions of actual test results 16Goodness of fit 16Normal distributed abilities 16Test information functions and standard error 17Rank of the test-takers 18

COMPARING CTT AND IRT 20

DISCUSSION 22

FURTHER RESEARCH 24

REFERENCES 25

APPENDIX


4/30

1

Introduction

The Swedish National Road Administration, abbreviated here as SNRA,is responsible for the Swedish driving-license examination. The driving-license test consists of two parts: a practical road test and a theory test. Inthis paper the emphasis is on the theory test. The theory test is a stan-dardized test that is divided into five content areas and is used to decideif the test-takers have sufficient theoretical knowledge to be a safe driveras stated in the curriculum (VVFS, 1996:168). The theory test is a crite-rion-referenced test (Henriksson, Sundstrm, & Wiberg, 2004) and con-sists of 65 multiple-choice items, where each item has 2-6 options, and

only one is correct. The test-taker receives one point for each correctlyanswered item. If the test-taker has a score higher or equal to the cut-offscore, 52 (80%), the test-taker passes the test. Each test-taker is alsogiven five try-out items together randomly distributed among the regularitems, but the score on those are not included in their final total score.(VVFS, 1999:32).

A test can be studied from different angles and the items in the test canbe evaluated according to different theories. Two such theories will bediscussed here; Classical Test Theory (CTT) and Item Response Theory(irt). CTT was originally the leading framework for analyzing and devel-

oping standardized tests. Since the beginning of the 1970s IRT has moreor less replaced the role CTT had and is now the major theoreticalframework used in this scientific field (Crocker & Algina, 1986; Ham-bleton & Rogers, 1990; Hambleton, Swaminathan, & Rogers, 1991).

CTT has dominated the area of standardized testing and is based on theassumption that a test-taker has an observed score and a true score. Theobserved score of a test-taker is usually seen as an estimate of the truescores of that test-taker plus/minus some unobservable measurementerror (Crocker & Algina, 1986; Hambleton & Swaminathan, 1985). Anadvantage with CTT is that it relies on weak assumptions and is rela-

tively easy to interpret. However, CTT can be criticized since the truescore is not an absolute characteristic of a test-taker since it depends onthe content of the test. If there are test-takers with different ability levelsa simple or more difficult test would result in different scores. Anothercriticism is that the items difficulty could vary depending on the sampleof test-takers that take a specific test. Therefore, it is difficult to comparetest-takers results between different tests. In the end, good techniques


5/30

2

are needed to correct for errors of measurement (Hambleton, Robin, &Xing, 2000).

IRT was originally developed in order to overcome the problems withCTT. A major part concerning the theoretical work was produced in the1960s (Birnbaum, 1968; Lord & Novick, 1968) but the development ofIRT continues (van der Linden & Glas, 2000). One of the basic assump-tions in IRT is that the latent ability of a test-taker is independent of thecontent of a test. The relationship between the probability of answeringan item correctly and the ability of a test-taker can be modeled in differ-ent ways depending on the nature of the test (Hambleton et al., 1991). It

is common to assume unidimensionality, i.e. that the items in a testmeasure one single latent ability. According to IRT, test-taker with highability should have a high probability of answering an item correctly.

Another assumption is that it does not matter which items are used inorder to estimate the test-takers ability. This assumption makes it possi-ble to compare test-takers result despite the fact that they have takendifferent versions of a test (Hambleton & Swaminathan, 1985). IRT hasbeen the preferred method in standardized testing since the developmentof computer programs. The computer programs can now perform thecomplicated calculations that IRT requires (van der Linden & Glas,

2000). There have been studies that compare the indices of CTT andIRT (Bechger, Gunter, Huub, & Bguin, 2003). Other studies haveaimed to compare the indices and the applicability of CTT and IRT, seefor example how it is use in the Swedish Scholastic Aptitude Test inStage (2003) or how it can be used in test development (Hambleton &

Jones, 1993).

Aim

The aim of this study is to examine which IRT model is the most suit-able for use when evaluating the theory test in the Swedish driving-

license test. Further, to use this model to compare the indices generatedby classical test theory and item response theory.


6/30

3

Method: Sample

A sample of 5404 test-takers who took one of the test versions of theSwedish theory driving license test in January 2004 was used to evaluatethe test results. All test-takers answered each of the 65 regular items inthe test. Among the test-takers 43.4% were women and 56.4% weremen. Their average age was 23.6 years (range = 18-72 years, s = 7.9years) and 75% of the test-takers were between 18 and 25 years.

Method: Classical test theory

A descriptive analysis was used initially which contained mean and stan-dard deviation of the test score. The reliability was computed with coef-ficient alpha, defined as

==

2

1

2

11

X

n

i

i

n

n

,

where nis number of items in the test, 2i is the variance on item iand2

X is the variance on the overall test result. Each item was examined

using the proportion who answered the item correctly, p-values, andpoint biserial correlation, rpbis. The point biserial correlation is the correla-tion between the test-takers performance on one item compared to thetest-takers performances on the total test score. Finally, an examinationof the 10% of test-takers with the lowest abilities performed on the mostdifficult items was made in order to give clues on how to model the itemsin the test (Crocker & Algina, 1986). In this study coefficient alpha, p-values and rpbisare used from the CTT.


7/30

4

Method: Item response theory

There are a number of different IRT models. In this study, the threeknown IRT models for binary response were used; the one (1PL), two(2PL) and three (3PL) parameter logistic IRT model. The IRT model(1PL, 2PL, 3PL) can be defined using the 3PL model formula

n1,2,...,i,1

)1()()(

)(

=+

+=

ii

ii

ba

ba

iiie

eccP

where Pi()is the probability that a given test-taker with ability answera random item correctly. aiis the item discrimination, biis the item diffi-culty and ci is the pseudo guessing parameter (Hambleton et al., 1991).The 2PL model is obtained when c= 0. The 1PL model is obtained if c=0 and a= 1.

In order to evaluate which IRT model should be used three criteria,summarized in Hambleton and Swaminathan (1985) are used;

Criterion 1. Verifying the assumptions of the model.Criterion 2. Expected model features

Criterion 3. Model predictions of actual test results

These three criteria are a summary of the criteria that test evaluators canpossibly use. These criteria can be further divided in subcategories.Hambleton, Swaminathan and Rogers (1991) suggest that one should fitmore than one model to the data and then compare the models accord-ing to the third criterion. The authors also suggested methods for exam-ining the third criterion more closely. However, these suggested methodsdemand that the evaluator can manipulate the test situation, control thedistribution of the test to different groups of test-takers and have accessto the test. Since we cannot manipulate the test situation and the driv-

ing-license test is classified, only methods that are connected with theoutcome of the test have been used. There are a number of possiblemethods to examine these criteria (see for example (Hambleton & Swa-minathan, 1985; Hambleton et al., 1991). In this study the methodsdescribed in the following three sections will be used to evaluate themodels.


8/30

5

1: Verifying the assumptions of the model

A. Unidimensionality

Unidimensionality refers to the fact that a test should only measure onelatent ability in a test. This condition applies to most IRT models.Reckase (1979) suggests that unidimensionality can be investigatedthrough the eigenvalues in a factor analysis. A test is concluded to beunidimensional if when plotting the eigenvalues (from the largest to thesmallest) of the inter-item correlation matrix there is one dominant firstfactor. Another possibility to conclude unidimensionality is to calculate

the ratio of the first and second eigenvalues. If the ratio is high, i.e. abovea critical value the test is unidimensional. In this study the first methoddescribed is used, i.e. observing if there is one dominant first factor.

B. Equal discrimination

Equal discrimination can be verified through examining the correlationbetween item iand the total score on the test score, i.e. the point biserialcorrelation or with the biserial correlation. The standard deviationshould be small if there is equal discrimination. If the items are notequally discriminating then it is better to use the 2PL or 3PL model than

the 1PL model (Hambleton & Swaminathan, 1985). In this study thenotation ais used for the item discrimination.

C. Possibility of guessing the correct answer

One way to examining if guessing occurs is to examine how test-takerswith low abilities answer the most difficult items in the test. Guessingcan be disregarded from the model if the test-takers with low ability an-swer the most difficult items wrongly. If the test-takers with low abilityanswer the most difficult items correctly to some extent a guessing pa-rameter should be included in the model, i.e. the 3PL model is more

appropriate than the 1PL or the 2PL model (Hambleton & Swamina-than, 1985).


9/30

6

2. Expected model features

The second criterion expected model features is of interest no matterwhich model is used. First, the invariance of the ability parameter esti-mates needs to be examined. This means that the estimations of the abili-

ties of the examinees = N ,...,, 21 should not depend on whether or

not the items in the test are easy or difficult (Wright, 1968) in Hamble-ton, Swaminathan & Rogers (1991).

Secondly, the invariance of the item parameter estimates needs to beexamined. This means that it should not matter if we estimate the itemparameters using different groups in the sample, i.e. groups with low orhigh abilities. In other words there should be a linear correlation betweenthese estimates and this is most easily examined using scatter plots(Shepard, Camilli, & Williams, 1984).

3. Model predictions of actual test results

The third criterion model prediction of actual test results can be exam-ined by comparing the Item Characteristic Curves (ICC) for each item

with each other (Lord, 1970). The third criterion can also be examinedusing plots of observed and predicted score distributions or chi-square

tests can be used (Hambleton & Traub, 1971).

A chi-squared test can be used to examine to what extent the modelspredict observed data (Hambleton et al., 1991). In this study the likeli-hood chi-squared test provided by BILOG MG 3.0 was used. This testcompares the proportion of correct answers on item iin the ability cate-gory hwith the expected proportion of correct answers according to themodel used. The chi-squared test statistic is defined as follows

( )

( )

( )[ ]=

+

=m

h hih

hihhih

hih

hihii

N

rNrN

N

rrG

1

2

1

lnln2

mh

ni

,...,2,1

,...,2,1

=

=

where m is the number of ability categories, hir is the number of ob-

served correct answers on item ifor ability category h , hN is the num-

ber of test-takers in ability category h , h is the average ability of test-

takers in ability category h , ( )hi is the value of the adjusted responsefunction of item i at h , i.e. the probability that a test-taker with ability


10/30

7

h will answer item i correctly. 2iG is chi squared distributed with m

degrees of freedom. If the observed value of 2iG is higher than a critical

value the null hypothesis is rejected and it is concluded that the ICC fitsthe item (Bock & Mislevy, 1990).

The ability is assumed to be normal distributed with mean 0 and stan-dard deviation 1 and is usually examined with graphics; for example ahistogram of the abilities. The test information function can be used toexamine which of the three IRT models estimates best (Hambleton etal., 1991).

Finally, by examining the rank of the test-takers ability estimated fromeach IRT model and compare it with how their rank of abilities are esti-mated in other IRT models the difference between the models was madeclear (Hambleton et al., 1991).

Estimation methods

The computer program BILOG MG 3.0 was used to estimate the pa-rameters. Therefore, the item parameters and the abilities are estimated

with marginal maximum likelihood estimators. Note that in order to

estimate the c-values. BILOG requires that you enter the highest numberof options in an item and uses this as the initial guess when it iterates inorder to find the c-value.


11/30

8

Results: Classical test theory

Using CTT descriptive statistics were obtained about the test. The meanof the test score was 51.44 points with a standard deviation of 6.9 points(range: 13-65 points). Fifty-six percent of the test-takers passed the the-ory test. The reliability in terms of coefficient alpha was estimated to be0.82. The p-values ranged between 0.13 (item 58) and 0.97 (item 38).The point biserial correlation ranged between 0.03 (item 58) and 0.48(item 36). These two variables are plotted against each other in Figure 1and are formally connected according to the following formula

)1/( ppsr X

Xr

pbis

=

,

where r is the mean for those who answered the item of interest cor-

rectly, X is the overall mean on the test, Xs is the standard deviation

on the test for the entire group, and pis the item difficulty (Crocker &Algina, 1986). In Figure 1 it is obvious that there is quite a large varia-tion in these two variables. In general, most items are close to each otherand are in the upper right corner and without the two outliers item 56and item 58 the range for the p-values are between 0.47 and 0.97.

Point biserial correlation

,5,4,3,2,10,0

p-values

1,0

,8

,6

,4

,2

0,0

Figure 1. Point biserial correlations plotted againstp-values.

The variation of the point biserial correlations indicates that there is avariation in how well the items discriminate. Item 58 is problematic withap-value of 0.03 and together with item 56 with ap-value 0.26 and r

pbis

equal to 0.23. It is important to note that indices other than a high valueon the point biserial correlation can be of interest. There are items with


12/30

9

high point biserial correlation that are quite easy, for example items 55and 56, which have the same point biserial correlation but have a hugedifferent on their item difficulty (0.83 and 0.26, respectively). Also, sincethe theory test is a criterion-referenced test, an item that is valuable forthe content cannot necessarily be excluded from a test because it is tooeasy (Wiberg, 1999). A more detailed information is presented in Table1 on the values of the point biserial correlations and thep-values. How-ever Figure 1 indicates that there is no specific connection between itemsthat are easy and items that are difficult.

Table 1. The p-values and the point biserial correlations (rpbis

) for the 65 items*.

Item p rpbis Item p rpbis Item P rpbis1 0.88 0.26 23 0.82 0.26 45 0.91 0.372 0.79 0.28 24 0.95 0.24 46 0.91 0.373 0.79 0.26 25 0.85 0.23 47 0.63 0.354 0.62 0.33 26 0.82 0.32 48 0.93 0.275 0.81 0.31 27 0.96 0.15 49 0.96 0.276 0.62 0.39 28 0.91 0.35 50 0.70 0.317 0.63 0.44 29 0.87 0.29 51 0.82 0.418 0.95 0.29 30 0.85 0.27 52 0.73 0.329 0.88 0.36 31 0.69 0.41 53 0.53 0.3410 0.82 0.31 32 0.90 0.33 54 0.69 0.1911 0.81 0.22 33 0.84 0.33 55 0.83 0.23

12 0.68 0.27 34 0.95 0.25 56 0.26 0.2313 0.94 0.33 35 0.61 0.24 57 0.84 0.2914 0.81 0.36 36 0.78 0.48 58 0.13 0.0315 0.96 0.24 37 0.95 0.22 59 0.78 0.1316 0.95 0.22 38 0.97 0.18 60 0.84 0.2817 0.79 0.35 39 0.73 0.28 61 0.81 0.1318 0.86 0.26 40 0.67 0.37 62 0.85 0.3219 0.76 0.31 41 0.88 0.41 63 0.59 0.3320 0.97 0.20 42 0.87 0.23 64 0.47 0.2921 0.94 0.23 43 0.94 0.12 65 0.52 0.2322 0.84 0.41 44 0.82 0.33Note: Items of special interest are in bold type.


13/30

10

Modeling the items

Figure 1 shows no distinct relationship between item discrimination anddifficulty. This suggests that both these parameters are important andshould be included in a model. There is also of interest to examine

whether guessing is present or not. One possible way is to choose test-takers with the 10% lowest ability on the overall test and study how theyperform on the more difficult items (Hambleton & Swaminathan,1985). The five most difficult items were items 53, 56, 58, 64 and 65.

Among the test-takers with the 10% lowest ability, 87%, 16%, 11%,28% and 32% managed to answer these items correctly. These items had

each four options except for item 65, which had three options. The val-ues of the observed percentage correct can be compared with the theo-retical values if the test-takers are randomly guessing the correct answer,i.e. 25% for the first four items and 33% for item 65. Even though thesevalues are partly close. The overall conclusion is that a guessing parame-ter should be a part of the model since the observed and the theoreticalvalues are not the same and the test have multiple-choice items wherethere is always a possibility to guess the correct answer.

Results: Item response theory

1. Verifying the assumptions of the model

In order to verify the first criterion that Hambleton and Swaminathan(1985) used, the assumptions of the three possible IRT models; 1PL,2PL, and 3PL need to be examined. These three models all rely on threebasic assumptions;

1. Unidimensionality2. Local independence3. ICC, i.e. each item can be described with an ICC.

The first assumption unidimensionality can be examined using coeffi-cient alpha or most commonly by a factor analysis. The coefficient alpha

was 0.82, i.e. a high internal consistence which indicates unidimensional-ity. The factor analysis gives one distinct factor and many small factors ascan be seen in Figure 2. However, it should be noted that the first factor,

which has an eigenvalue of 5.9 only accounts for 9 percent of the vari-ance. It would definitely be preferable if more variance was accounted forby the first factor. However Hambleton (2004) explained that this is not


14/30

11

uncommon and as long as there is one factor with distinctively largereigenvalues it is possible to assume that there is unidimensionality in thetest. Note that there are 18 factors with relevant eigenvalues above 1 andtogether they account for 40% of the total explained variance.

Component Number

6561575349454137332925211713951

Eigenvalue

7

6

5

4

3

2

1

0

Figure 2. Eigenvalues from the factor analysis.

item discrimination

,50,44,38,31,25,19,13,060,00

30

20

10

0

Figure 3. Item discrimination examined using point biserial correlation.

The second assumption local independence was only examined throughpersonal communication with Mattsson (2004) at SNRA. Mattsson as-sured that no item gives a clue to any other items answer. In the future it


15/30

12

would be interesting to review this criterion more carefully. The thirdassumption simply states that the items can be modeled with an ICC.This is most easily examined by plotting the corresponding ICC for allitems in the test (Lord, 1970). If the 2PL IRT model is used the itemdifficulty is the point where a test-taker has an ability of 0.5 to answer anitem correctly, the item discrimination is the slope of the curve at thatpoint. The 3PL model also has a guessing parameter which is the inter-cept of the curve (Hambleton et al., 1991). Therefore an item with areasonably high item discrimination and item difficulty looks like item 4in Figure 4. In the appendix the ICC of all items in the test are shown.From the figures in the appendix it is clear that we can model the items

with an ICC. However, there are some problematic items. For exampleitem 58, in the appendix, has low item discrimination, low item diffi-culty and one can question its role in the test. However, since it is a crite-rion-referenced test it might be an important item in another way.

ability

3,002,001,00,00-1,00-2,00-3,00

Probability

1,0

,8

,6

,4

,2

0,0

Figure 4. ICC of item 4 modeled with the 3PL model.

Finally, if we add the results from CTT to this information we can con-clude that the items do not have equal discrimination and that the test-takers guess the correct answer of an item sometimes. These results leadus to believe that the 3PL model should be preferred instead of the 1PLor the 2PL model.


16/30

13

2. Expected model features

Invariance of ability estimates

To examine if we have invariance among the ability estimates the itemswere divided into easy and difficult items using the item difficulty. Then,the test-takers abilities were estimated using these items and plottedagainst each other in Figure 5.

Figure 5. Ability estimates from easy and difficult items using the 3PLmodel.

Figure 5 shows that the estimations of the abilities differ whether easy ordifficult items are used. This means that this assumption is not quitefulfilled.

(1PL, difficult)

43210-1

-2

-3

-4

(1PL,easy)

4

3

2

1

0

-1

-2

-3-4

(2PL, difficult)

43210-1-2

-3

-4

(2PL,easy)

4

3

2

1

0

-1

-2

-3

-4

(3PL, difficult)

43210-1-2-3-4

(3PL,easy)

4

3

2

1

0

-1

-2

-3

-4


17/30

14

Invariance of item parameters estimates

To examine whether there are invariance among the parameter estimatesthe test-takers were divided into the categories low ability and high abil-ity depending on how they scored on the test (Hambleton & Swamina-than, 1985). The division between low and high ability was made at thecutoff score, i.e. test-takers with scores below 52 were labeled low abilityand test-takers with scores equal to or higher than 52 were labeled highability. The item difficulties were estimated in each group and plottedagainst each other in Figure 6. If the estimates are invariant the plotsshould show a straight line.

b (1PL, high ability)

151050-5-10-15-20

b(1PL,

low

ability)

15

10

5

0

-5

-10

-15

-20


151050-5-10-15-20

b(2PL,

low

ability)

15

10

5

0

-5

-10

-15

-20


151050-5-10-15-20

b(3PL,

low

ability)

15

10

5

0

-5

-10

-15

-20

Figure 6. Item difficulties estimated with the 1PL, 2PL and the 3PL

models using test-takers with low and high ability, respec-tively.

Figure 6 suggests that the item difficulties are quite invariant. The 1PLmodel has most item parameters which are invariant. But both the 2PL


18/30

15

and the 3PL model have some invariant item difficulties. Figure 7 showsthe item discrimination estimated from test-takers with low and highability, respectively. In the figure one can see that the estimations are notthat invariant since the observations are not on a straight line. There area few more parameters that are invariant for the 2PL than the 3PLmodel.

a (2PL, high ability)

1,0,8,6,4,20,0

a(2PL,

low

ability)

1,0

,8

,6

,4

,2

0,0

a (3PL, high ability)

1,0,8,6,4,20,0

a(3PL,

low

ability)

1,0

,8

,6

,4

,2

0,0

Figure 7. Item discrimination estimated with the 2PL and the 3PL mod-

els using test-takers with low and high ability, respectively.

Figure 8 shows the item guessing estimated from test-takers with low andhigh ability, respectively. In Figure 8 the estimation of cis concentratedaround 0.17 for test-takers with high ability but is spread between 0.08-0.65 for test-takers with low ability. The overall conclusion is that theestimations are not particularly invariant for the 3PL model.

c (3PL, high ability)

,70,60,50,40,30,20,100,00

c(3PL,

low

ability)

,7

,6

,5

,4

,3

,2

,1

0,0

Figure 8. Item guessing estimated with the 3PL models using test-takers

with low and high ability, respectively.


19/30

16

3. Model predictions of actual test results

Goodness of Fit

If large samples are used there is a problem with using Goodness of Fittests since small changes in the empirical data material will lead to rejec-tion of the null hypothesis that the model fits the data ((Hambleton &Swaminathan, 1985); Hambleton & Traub, 1973). However, as an ini-tial study the likelihood ratio goodness of fit test was used in order to test

whether or not it is reasonable to model the items according to the one,two or three parameter logistic model. When the 1PL model was used 53

items rejected the null hypothesis that the model fitted the data. For the2PL and the 3PL models the numbers of items were 25 and 16 respec-tively.

Normal distributed abilities

The ability is assumed to be normal distributed with mean 0 and stan-dard deviation 1 and is demonstrated in Figure 9. The line represents thetheoretical standard normal distribution and the bars represents the ob-served distribution. The conclusion from Figure 9 is that the abilities areapproximately normal distributed and therefore this assumption is ful-

filled.

-3 -2 -1 0 1 2 30

100

200

300

400

500

600

700

800

Abi li ty

Frequency

Figure 9. Histogram of the test-takers abilities with respect to the theo-

retical standard normal distribution function.


20/30

17

Test information functions and standard error

How the models manage to describe the items in the test can be exam-ined using Test Information Functions (TIF), and the standard error(S.E). The ideal TIF contains a lot of information on all the test-takersabilities and has a low standard error. The TIF and the S.E. are relatedaccording to the formula

..

1

ESTIF= .

Hambleton (2004) suggested that a TIF 10 is preferable. The curves inFigures 10, 11 and 12 show that the 3PL model gives more informationand has a lower standard error than both the 1PL and the 2Pl models.Note that the y-scales are different in these three figures.

-4 -3 -2 -1 0 1 2 3 40

1

2

3

4

5

6

7

Scale Score

Information

1-PL

0

0.26

0.52

0.78

1.04

1.30

Standard

Error

Figure 10. Test information function and standard error for the 1PL

model.

-4 -3 -2 -1 0 1 2 3 40

2

4

6

8

10

12

Scale Score

Information

2-PL

0

0.31

0.63

0.94

1.25

1.57

Standard

Error


model.


21/30

18

-4 -3 -2 -1 0 1 2 3 40

1

2

3

4

5

6

7

8

Scale Score

Information

3-PL

0

0.33

0.67

1.00

1.33

1.66

Standard

Error


model.

Since the theory test in the Swedish driving license test is a licensure testit is important to find as much information around the cut-off score aspossible (Birnbaum, 1968; Wiberg, 2003). In this test the cut-off score is

on the same level as the ability level 0.1. The TIF with the highestvalue on the TIF at the cutoff score is the 3PL, which has 5.9, followedby the 2PL which has 5.8 and the 1PL which has 4.5. This result sug-gests that the items should be modeled with the 3PL model.

Rank of the test-takers

There is a difference between how different models estimate the test-takers ability. If we rank the abilities of the test-takers for three differentmodels we get the result shown in Figure 13. In Figure 13 the 2PL andthe 3PL models give quite consistent estimations of the test-takers abili-ties while the 1PL model ranks the test-takers abilities somewhat differ-ently. Note especially that all models rank the low ability test-takers andthe high ability test-takers in the same way but the test-takers in themiddle are ranked differently if the 1PL model is compared with either

the 2PL or the 3PL model.


22/30

19

Rang (1-PL)

6000500040003000200010000-1000

Rang(2-PL)

6000

5000

4000

3000

2000

1000

0

-1000

Rang (1-PL)

6000500040003000200010000-1000

Rang(3-PL)

6000

5000

4000

3000

2000

1000

0

-1000

Rang (2-PL)

6000500040003000200010000-1000

Rang(3-PL)

6000

5000

4000

3000

2000

1000

0

-1000

Figure 13. The rank of the test-takers abilities depending on which

model the abilities are estimated from compared with theother models.


23/30

20

Comparing CTT and IRT

There have been a number of studies where CTT and IRT have beencompared (see for example (Bechger et al., 2003; Nicewander, 1993).The item discrimination parameter in IRT denoted by a, is proportionalto the slope of the item characteristic curve at point bon the ability scale.a can take the values


24/30

21

between these indices. The correlation is 0.861, i.e. highly negative, asexpected in theory.

p-values

1,0,8,6,4,20,0

b-values

6

4

2

0

-2

-4

-6

Figure 15. Estimated b-values from the 3PL model plotted against p-

values.

The c-values from IRT can be compared to the inverted number of the2-6 options for each item. Figure 15 displays this comparison. Note thatif an item has only two options, i.e. the test-takers have a theoretical

chance of 0.5 of guessing the correct answer, the c-value can be as low as0.15. Therefore, IRT adds valuable information about the test-takerschoices. As suspected, the test-takers do not choose the options com-pletely at random.

theoretical guessing

,6,5,4,3,2,10,0

c-values

,6

,5

,4

,3

,2

,1

0,0

Figure 15. c-values from the 3PL model plotted against the inverted

number of options.


25/30

22

Discussion

This study has aimed to describe the theory test in the Swedish drivinglicense test using both classical test theory and item response theory withmost weight on the latter. The most important goal has been to comparethe 1PL, 2PL and the 3PL models in order to find the model which ismost suitable for modeling the items. Further to compare this modelresults from CTT. The evaluation using classical test theory showed thatthe internal reliability was high; 0.82. Further, the p-values of the itemsranged from 0.13 to 0.97 and the point biserial correlation from 0.03 to0.48. One important conclusion was that the item discrimination was

not similar between items. Another important result was that if we wantto model the items we should include a guessing parameter in the modelsince some test-takers with low ability tend to guess the correct answeron the most difficult items.

The evaluation using IRT was performed according to three criteria. Thefirst criterion was verifying the model assumptions. The factor analysis inFigure 2 supports the assumption of unidimensionality. There was onedominant factor that explains more of the variation than the other fac-tors. Even though it would be better if the first factor would account formore explained variance this assumption is considered fulfilled. The item

discrimination was concluded to be unequal among the items which leadto the conclusion that we should have that parameter in our model. Inother words, the 1PL is less suitable than the 2PL or the 3PL models.Guessing should probably also be included in the model since test-takers

with low ability still manage to get some of the most difficult items inthe test correct. This result suggests that the 3PL model is preferable overthe 1PL and the 2PL models.

The second criterion was to what extent the models expected featureswere fulfilled. Figure 5 shows the test-takers abilities depending on howthey performed on easy or difficult items. Figure 5 suggest that the

estimations of are invariant in all three IRT models.

Figure 6 shows the item difficulties estimated from test-takers with lowand high ability, respectively. The conclusion from Figure 6 is that theestimation of bis invariant for all three models. Further, Figure 7 showsthe item discrimination estimated with test-takers with low and highability for the 2PL model. The pattern in Figure 7 is quite spread out

which suggests that there are no invariance. Finally, item guessing for the


26/30

23

3PL model was estimated in Figure 8 using test-takers with low and highability, respectively. This figure suggests that c does not have invariantestimates. Since aand cdoes not have invariant estimates a 1PL model

which only consists the item difficulty b should be preferred instead ofthe 2PL or the 3PL model.

The third criterion was model prediction of test results. Goodness of Fittests were used to examine how many items had an ICC that fitted thethree models. 53 items did not fit the 1PL model, 25 items did not fitthe 2PL model and 16 items did not fit the 3PL model. This result sug-gests that the 3PL model is preferable to the other models. However, as

noted earlier it may be dangerous to use these tests in large samples. Fig-ure 9 shows that the estimations are approximately normal distrib-uted. There are a few more test-takers with low ability than with highability. The histogram has the same appearance no matter which modelis used.

Finally, the three models were compared. The TIF for the three modelswere plotted in Figures 10, 11 and 12. Note, when comparing theseplots, that the 2PL model has the highest maximum information and

that it gives most information between 3.5 and 1 on the ability scale.Between 1 and 2 on the ability scale the three models give approximately

the same information. This suggests that the 2PL model should be used.However, since the 3PL model gives more information about the ability

level that corresponds to the cut-off score, i.e. 0.1 the 3PL model is toprefer. In Figure 13 the estimates of the test-takers abilities from the testresults according to the three different models have been ranked andcompared with each other. The conclusion that can be drawn from Fig-ure 13 is that the 2PL and the 3PL models estimate the abilities of thetest-takers similar but the 1PL model is only consistent with the othermodels at the endpoints.

The comparisons of the models have been performed using a large sam-

ple of test results. Note that an extended analysis would contain controlover the testing situation, reviewing the items in the test etc. None of themodels fits the test results perfectly but no model is ever perfect. The2PL model and the 3PL model are, in general, to be preferred over the1PL model. The test evaluated is a criterion-referenced test with multiplechoice items and the results suggest that guessing should be part of themodel. The overall conclusion is that the 3PL model is most suitable tomodel the items in the theory test in the Swedish driving-license test.


27/30

24

The 3PL model can be used to model the items to a higher level andadds valuable knowledge about the theory test in the Swedish driving-license test.

The last part of this work aimed at comparing the classic test theory anditem response theory indices. From this comparison it can be concludedthat the estimates are valid for both CTT and IRT. The item discrimina-tions in Figure 14 are positive linear related and the item difficulties inFigure 15 are negatively linear related as the theory states. Figure 15shows that the values of the guessing parameters are more spread outthan if inverted numbers of options are used. This suggests that IRT

gives valuable information about a test-takers true knowledge. IRT hasthe advantage that the estimates of the item parameters are independentfrom the sample that has been used. This advantage is especially useful

when reusing a test a number of times. From the ICC it is clear how theitems work, and which ability a test-taker has that perform well on eachitem. The TIF and the standard error give us a measure of the amount ofinformation that is obtained from the test about a test-taker dependingon the test-takers ability level. Finally, if both CTT and IRT are used

when evaluating items, different dimensions of information are obtainedsince both CTT and IRT add valuable information about the test.

Further research

There are, of course, many different scientific fields related to licensuretests using both CTT and IRT that can be studied in the future. Forexample, if the evaluators have access to the test situation it can be ma-nipulated. Further, one can examine the effect of speediness, i.e. if theestimated abilities are the same regarding how much time they have tocomplete the test. If the evaluators have access to the items before the testis given to the test-takers the assumption of local independence can bemore carefully examined and it can help understanding why an item has

a certain difficulty, discrimination or a certain value of guessing. Otherinteresting fields concern what kind of studies that can be made usingIRT with a theory test in a driving-license test. For example, studyingtest equating and differential item functioning would definitely be inter-esting in the theory test in the Swedish driving-license test.


28/30

25

References

Bechger, T., Gunter, M., Huub, H., & Bguin, A. (2003). Using clas-

sical test theory in combination with item response theory.Ap-plied psychological measurement, 27(5), 319-334.

Birnbaum, A. (1968). In F. M. Lord & M. R. Novick (Eds.), Statistical

theories of mental test scores. Reading: Addison-Wesley.

Bock, R., & Mislevy, R. (1990).Item analysis and test scoring with

binary logistic models. Mooresville: Scientific Software.Camilli, G., & Shepard, L. (1994).Methods for identifying biased test

items. Thousand Oaks, CA: Sage publications.Crocker, L., & Algina, J. (1986).Introduction to classical and modern

test theory. New York: Holt, Rinehart and Winston, Inc.

Hambleton, R. K. (2004). Personal communication. Ume.Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical

test theory and item response theory and their applications totest development.Educational Measurement: issues and prac-tice, 12(3), 535-556.

Hambleton, R. K., Robin, F., & Xing, D. (2000). Item response mod-els for the analysis of educational and psychological test data.In H. Tinsley & S. Brown (Eds.),Handbook of applied multi-

variate statistics and modelling. San Diego, CA: AcademicPress.Hambleton, R. K., & Rogers, J. H. (1990). Using item response mod-

els in educational assessments. In W. Schreiber & K. Ingenk-amp (Eds.),International developments in large-scale assess-ment(pp. 155-184). England: NFER-Nelson.

Hambleton, R. K., & Swaminathan, H. (1985).Item response theory:principles and applications. Boston: Kluwer-Nijhoff Publish-

ing.Hambleton, R. K., Swaminathan, H., & Rogers, J. H. (1991).Funda-

mentals of item response theory. New York: Sage publications.

Hambleton, R. K., & Traub, R. E. (1971). Information curves and ef-ficiency of the three parameter logistic test models.British

Journal of Mathematical and Statistical Psychology, 24, 273-281.

Henriksson, W., Sundstrm, A., & Wiberg, M. (2004). The Swedish

driving-license test: A summary of studies from the department

of educational measurement, Ume University. Ume: De-

partment of Educational measurement.


29/30

26

Lord, F. M. (1970). Estimating item characteristic curves withoutknowledge of their mathematical form.Psychometrica, 35, 43-50.

Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental

test scores. Reading, MA: Addison-Wesley.

Mattsson, H. (2004). Personal communication. Ume.

Nicewander, W. A. (1993). Some relationships between the informa-tion function of IRT and the signal/noise ratio and reliability

coefficient of classical test theory.Psychometrika, 58, 139-141.

Reckase, M. (1979). Unifactor latent trait models applied to multi-

factor tests: results and implications.Journal of EducationalStatistics, 4, 207-230.

Shepard, L., Camilli, G., & Williams, D. (1984). Accounting for sta-tistical artefacts in item bias research.Journal of EducationalStatistics, 9, 93-128.

Stage, C. (2003). Classical test theory or item response theory: theSwedish experience(No. 42). Ume: Department of Educa-

tional Measurement.van der Linden, W. J., & Glas, C. A. (2000). Computerized adaptive

testing: theory and practice. Dordrecht: Kluwer Academic

Publisher.Wiberg, M. (1999).Mlrelaterade och normrelaterade prov - En teo-

retisk granskning av vilka statistiska tekniker som kan anvn-

das fr att beskriva uppgifternas kvalitet och provens reliabili-

tet(Pedagogiska Mtningar No. 150): Ume Universitet: En-

heten fr pedagogiska mtningar.Wiberg, M. (2003). Computerized Achievement Tests - sequential and

fixed length tests(Statistical studies No. 29). Ume University,Sweden: Department of Statistics.

Wright, B. (1968). Sample-free test calibration and person measure-

ment.Paper presented at the Invitaional conference on testing

problems, Princeton, NJ.VVFS. (1996:168). Vgverkets frfattningssamling. Vgverkets fre-

skrifter om kursplaner, behrighet B.Borlnge: Vgverket.

VVFS. (1999:32). Vgverkets frfattningssamling. Vgverkets fre-

skrifter om kursplaner, behrighet B.Borlnge: Vgverket.


30/30

Appendix

1 - 9

10 - 18

19 - 27

28 - 36

37 - 45

46 - 54

55 - 63

64 - 65

Matrix Plot of Item Characteristic Curv es

Figure A. The ICC for all 65 multiple-choice items in the theory test of

the Swedish driving license test.

Classical Test Theory vs. Item Response Theory

Documents

Transcript of Classical Test Theory vs. Item Response Theory