An Item Response Theory Analysis for the Test of...
Transcript of An Item Response Theory Analysis for the Test of...
![Page 1: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/1.jpg)
International Conference on Teaching Physics Innovatively
August 17-19, 2015, Budapest, Hungary
Suttida Rakkapao
Department of Physics, Faculty of Science
Prince of Songkla University
THAILAND
An Item Response Theory Analysis
for the Test of Understanding of Vectors (TUV)
![Page 2: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/2.jpg)
OUTLINE
• Test of Understanding of Vectors (TUV)
• Objective
• Item Response Theory (IRT)
• Results and Discussion
• Conclusions
2
![Page 3: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/3.jpg)
3
TUV is a 20-item 5-choice research-based
conceptual test of vectors (Barniol and Zavala, 2014).
![Page 4: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/4.jpg)
4
Ten vector concepts used in the introductory course of a university level
Vector Concept
1. Direction of a vector
2. Magnitude of a vector
3. Component of a vector
4. Unit vector in the Cartesian plane
5. Graphic representation of a vector
6. Vector addition
7. Vector subtraction
8. Scalar multiplication of a vector
9. Dot product
10. Cross product
![Page 5: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/5.jpg)
5
Examples of TUV
![Page 6: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/6.jpg)
6
The TUV is a reliable assessment tool based-on the classical test methods.
![Page 7: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/7.jpg)
7
Item difficulty index (P) and Item discriminatory index (D) for item 1-10 of the TUV
Mexican students (N=423)
Thai students (N=774)
item
0.28 0.11 0.20 0.55 0.61 0.80 0.59 0.24 0.38 0.50
0.59 0.17 0.26 0.53 0.52 0.29 0.51 0.49 0.51 0.61
Desired values [0.3, 0.9]
Desired values [ ≥ 0.3]
One of the most important
shortcomings of Classical Test
Theory (CTT) is the item parameters depend on test-takers.
![Page 8: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/8.jpg)
OBJECTIVE
8
To analyze the test of understanding
of vectors (TUV) using a one-
parameter logistic model (1PL) of
an item response theory (IRT),called Rasch model.
![Page 9: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/9.jpg)
9
Item Response Theory (IRT)
( )
1( ) (1 )
1 a bP c c
e
where thedifficulty parameter
thediscrimination parameter
= the guessing parameter
= theability level
( ) = the probability that an examinee with that
ability willgive a correct answer to theitem
b
a
c
P
c = 0
a = 1
person parameter
Item parameters
![Page 10: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/10.jpg)
10
(http://www.jmetrik.com/index.php)
JMetrik free software for Rasch Model
![Page 11: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/11.jpg)
11
An example of the Item Characteristic Curve (ICC)(created by PARSCALE)
( ) 0.5P
( ) (1 ) / 2P b c
Desired values [ -2, 2]
![Page 12: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/12.jpg)
Difficulty index by CTT
12
Difficulty index by IRT (1PL)
Desired values [ 0.3, 0.9]
Desired values [ -2, 2]
0.62 2.08 1.06 -0.88 -0.60 -2.30 -1.09 0.80 -0.01 -0.59
Item 1-10
Results and Discussion (1/3)
![Page 13: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/13.jpg)
Difficulty index by CTT
13
Difficulty index by IRT (1PL)
Desired values [ 0.3, 0.9]
Desired values [ -2, 2]
0.10 1.64 0.09 -0.91 0.15 0.29 0.80 0.42 0.42 -1.85
Item 11-20
Results and Discussion (2/3)
![Page 14: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/14.jpg)
14
Results and Discussion (3/3)
Item 6 of dot product concept of TUV seem to be an easy question, proved by the IRT (1PL).
![Page 15: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/15.jpg)
CONCLUSIONS
Overall, the TUV contained items with different
levels of difficulty values for monitoring students in
different abilities.
The IRT can be used to develop multiple-choice
tests and to better understand the results, due to the
independent of test-takers.
15
![Page 16: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/16.jpg)
[1] P. Barniol and G. Zavala, Test of understanding of vectors: A
reliable multiple-choice vector concept test, Phys. Rev. ST Phys.
Educ. Res. 10, 010121 (2014).
[2] R. K. Hambleton, H. Swaminathan and H. Rogers, Fundamentals
of Item Response Theory (Sage Publications, Inc., California, 1991).
[3] http://www.jmetrik.com/index.php
16
REFERENCES
![Page 17: An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol](https://reader031.fdocuments.net/reader031/viewer/2022041209/5d67181a88c9931d358b9c67/html5/thumbnails/17.jpg)
ACKNOWLEDGEMENT
Department of Physics
Faculty of SciencePrince of Songkla University
This study was supported by the Research Support from Faculty of Science, Prince of Songkla University (budget year 2015).
17