MCQs - Item Writing & Item analysis
-
Upload
jane-holland -
Category
Education
-
view
85 -
download
4
Transcript of MCQs - Item Writing & Item analysis
![Page 1: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/1.jpg)
Multiple Choice Questions -
Item writing & item analysis
15th April 2014
Jane Holland MD PhD MRCSI
RCSI Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn
including slides from Richard Arnett, PhD
![Page 2: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/2.jpg)
MCQ Item writing & item analysis - Outline
Introduction to item-writing
Response formats
Stimulus formats
Technical item flaws
Introduction to item analysis
Jane Holland
http://www.nbme.org/publications/item-writing-manual.html
![Page 3: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/3.jpg)
MCQ response formats
• True / False
The candidate is required to make a determination
as to whether each option provided is true or false
• One Best Answer
The candidate is required to indicate a single
response i.e. the most likely diagnosis
NBME Manual p 13 – 18
Schuwirth: Med Educ. 2004 Sep;38(9):974-9.Jane Holland
![Page 4: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/4.jpg)
True / false vs. Single Best Answer
False True
(Simple true/false items)
(Complex true false items)
Q. True statements about dogs include:
A. They were first domesticated in New Zealand.
B. 35% of households own at least one dog.
C. They have a field of vision up to 200°.
D. They communicate by barking.
Jane Holland
![Page 5: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/5.jpg)
True / false vs. Single Best Answer
Q. Which Irish rugby union player holds the record
for most international caps?
A. D’Arcy
B. Heaslip
C. Kearney
D. O’Connell
E. O’Driscoll
Jane Holland
![Page 6: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/6.jpg)
Q. Which of the following statements about Ireland
is true?
A. Tayto exports exceed € 1 Bn
B. The climate is sub-tropical
C. The main export is zinc
D. The population is 5 million
E. Rainy weather frequently occursX
True / false vs. Single Best Answer
Trade / exports
Weather
DemographicsD
C
B
A
E
Jane Holland
![Page 7: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/7.jpg)
True / false vs. Single Best Answer
Q. What colour is the sky?
A. A hazy shade of winter
B. Blue
C. Grey
D. Orange
E. Red
Jane Holland
![Page 8: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/8.jpg)
Q. Which of the following calls is a rugby union referee most
likely to make in the case of an intercepted knock-on?
A. Advantage
B. Line out
C. Penalty kick
D. Penalty try
E. Scrum
Q. Which of the following was an original recording by
Meatloaf?
A. Bat out of Hell
B. One out of three ain’t bad
C. Paradise by the Kitchen Light
D. Stand by me
E. Total Eclipse of the Heart
Jane Holland
The “cover test”
![Page 9: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/9.jpg)
MCQs – how many options?
Jane Holland
Swanson et al; Academic Medicine. 2005;80(10):S93-S6.
Tarrant & Ware; Nurse Education Today. 2010;30(6):539-43.
![Page 10: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/10.jpg)
Stem /
vignette
Question
(Lead-in)Options
Single Best Answer
Jane Holland
![Page 11: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/11.jpg)
MCQ stimulus formats
Context -free
Tests factual
knowledge
Context-rich
Tests reasoning
or application
Schuwirth: Med Educ. 2004 Sep;38(9):974-9.
Kennedy, 2007Jane Holland
Creating
Evaluating
Analysing
Applying
Understanding
Remembering
![Page 12: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/12.jpg)
NBME Manual p 35 - 49
Q. Which dermatome is at the umbilicus?
A. T6
B. T8
C. T10
D. T12
E. L2
Q. A 65-year-old librarian presents with skin lesions
suggestive of a Herpes zoster infection affecting
the dermatome which includes the skin of the
umbilicus.
Which segmental spinal nerve is associated with
that dermatome?
Jane Holland
![Page 13: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/13.jpg)
Q. A 6 year old boy is admitted with a history of colicky
abdominal pain which has now localised to his right
iliac fossa.
The initial, visceral pain felt with appendicitis is
referred along sympathetic fibres from which level
of the spinal cord?
A. T6
B. T8
C. T10
D. T12
E. L2
NBME Manual p 35 - 49Jane Holland
![Page 14: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/14.jpg)
Stem
Lead-in
Options
Clinical (or contextual) vignette
• Age / gender
• Presenting Problem / history
• Examination findings / test results
Which of the following is the most:
• Likely diagnosis
• Appropriate treatment
Homogenous
• One correct & 4 distractors
• Alphabetical (or logical) order
• All plausible & same length (short!)
NBME Manual p 35 - 49Jane Holland
![Page 15: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/15.jpg)
Q. Which of the following Moodle activities
allows participants to select a single
option from a list?
A. Checklist
B. Choice
C. Forum
D. Survey
E. Workshop
Jane Holland
![Page 16: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/16.jpg)
Q. Gavin is running a conference, at which there will
be a number of parallel workshops with limited
places. Which of the following Moodle activities
could he use to allow participants to preselect
their preferred workshop?
A. Checklist
B. Choice
C. Forum
D. Survey
E. Workshop
Jane Holland
![Page 17: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/17.jpg)
Q. Which of the following Moodle activities
allows participants to select one choice
from a list of options?
Word repeats
Jane Holland
A. Checklist
B. Choice
C. Forum
D. Survey
E. Workshop
![Page 18: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/18.jpg)
Grammatical
clues
Long
correct
answer
Absolutes
Word
repeats
Logical
cues
Convergence
Technical
Item flaws
NBME Manual p 19 - 27Jane Holland
![Page 19: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/19.jpg)
Q. Which of the following Moodle activities
allows participants to select one choice
from a list of options?
Word repeats
Jane Holland
A. Checklist
B. Choice
C. Forum
D. Survey
E. Workshop
![Page 20: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/20.jpg)
Q. It is extremely important for teachers to
maintain a healthy diet. Therefore,
students should always give them an:
A. Apple
B. Banana
C. Bread roll
D. Chocolate bar
E. Mohito
Grammatical clues
Jane Holland
a / an
plural / singular
![Page 21: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/21.jpg)
Q. A 76 year old diabetic is brought into the accident
and emergency department with suspected
diabetic ketoacidosis. After confirming that she is
hyperglycaemic, the first step in management
should be intravenous administration of:
A. Chest X-ray
B. Electrocardiogram
C. Fasting blood glucose level
D. Insulin
E. Placement of an nasogastric tube
Grammatical clues
Jane Holland
![Page 22: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/22.jpg)
Word repeats
Q. A 58 year old man with a history of heavy alcohol
use and previous psychiatric hospitalisation is
confused and agitated. He speaks of experiencing
the world as unreal. This symptom is called:
A. Depersonalisation
B. Derailment
C. Derealisation
D. Focal memory deficit
E. Signal anxiety
Jane Holland
![Page 23: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/23.jpg)
Logical clues
Q. Crime is:
A. Equally distributed among the social classes
B. Overrepresented among the poor
C. Overrepresented among the middle class &
rich
D. Primarily an indication of psychosocial
maladjustment
E. Reaching a plateau of tolerability for the nation
Jane Holland
Also be wary of overlapping numericals...
![Page 24: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/24.jpg)
Convergence Strategy
Q. Local anaesthetics are most effective in the:
A. Anionic form, acting from inside the
nerve membrane
B. Cationic form, acting from inside the
nerve membrane
C. Cationic form, acting from outside the
nerve membrane
D. Uncharged form, acting from inside the
nerve membrane
E. Uncharged form, acting from outside the
nerve membrane
Jane Holland
![Page 25: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/25.jpg)
Long correct answer
Q. Secondary gain is:
A. Synonymous with malingering
B. A frequent problem in OCD
C. A complication of a variety of illnesses and
tends to prolong many of them
D. Never seen in organic brain damage
Jane Holland
![Page 26: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/26.jpg)
Absolutes...
Q. Which of the following shoes are never
worn by men?
A. Boots
B. Runners
C. Sandals
D. Stilettos
E. Wellies
Jane Holland
![Page 27: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/27.jpg)
Absolutes
Holsgrove G, Medical Education 1998; 32: 343 - 350
Jane Holland
0
10
20
30
40
50
60
0% 1% 5%
“Never”
0
10
20
30
40
50
60
80% 95% 99% 100%
“Always”
![Page 28: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/28.jpg)
Jane Holland
![Page 29: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/29.jpg)
Shape Long vignette & stem short options
Cover test
• Ideally can know the correct answer before seeing the
possible options
Use homogenous distracters
• i.e. all arteries, all medications, all diagnoses
• Grammatically consistent
• Logically compatible
• List in logical or alphabetical order
Don’t use vague frequency terms
Never use absolutes!
Do not write a negative question!!
NBME Manual p 35 - 49
Some useful principles
Jane Holland
![Page 30: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/30.jpg)
Jane Holland
![Page 31: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/31.jpg)
Difficulty Discrimination
Corrected
Point
Biserial
Distractor
Choice
Introduction to Item Analysis
Classical Test Theory
Item Response Theory (IRT)
Generalizability Theory (G-Theory)
Richard ArnettDe Champlain: Med Educ. 2010; 44:109-117.
![Page 32: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/32.jpg)
How did the items / questions perform??
1. What proportion of candidates got the
question correct?
2. Which options were chosen most frequently?
3. Which options were chosen by low, medium
and high performers on the test as a whole?
4. How did success on an item correlate with
overall test score?
Jane Holland
Difficulty Discrimination
Corrected
Point
Biserial
Distractor
Choice
De Champlain: Med Educ. 2010; 44:109-117.
![Page 33: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/33.jpg)
Item difficulty (DIF) – or facility
What proportion of candidates got the question correct?
90% of candidates answered correctly
DIF = 0.90
< 0.40 – hard
> 0.80 – easy
0.40 – 0.80 – good question
0.45 – 0.55 are extremely discriminating
Jane Holland
![Page 34: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/34.jpg)
Which options were chosen most frequently?
Distractor choice
Examine the proportion of candidates choosing
each option
ALL options should be chosen by candidates
Replace any options not chosen
Jane Holland
![Page 35: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/35.jpg)
Discriminating Index (or power)
Which options were chosen by low, medium
and high performers on the test as a whole?
Lower 27% Upper 27%Middle
L
N
U
U – L
N= DI
Ideally, DP should be > 0.40
Jane Holland
![Page 36: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/36.jpg)
Jane Holland
![Page 37: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/37.jpg)
Richard Arnett
![Page 38: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/38.jpg)
Additional measures of Discrimination
Point biserial correlation (RPB)
Equivalent to Pearson correlation co-efficient
Is a measure of association between a binary (nominal)
variable and a continuous variable
Corrected point biserial correlation (CRPB)
The item is removed from the test total prior to calculation
Range = -1 to +1
>0.2 good, <0.2 weak
Negative – inverse relationship!!!!
Jane Holland
![Page 39: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/39.jpg)
Jane Holland
![Page 40: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/40.jpg)
Jane Holland
![Page 41: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/41.jpg)
Jane Holland
![Page 42: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/42.jpg)
Moodle Quiz statistics
Item Difficulty...
Jane Holland
In a quiz with random questions...
Position of items within the test may change
Therefore, analyses must distinguish between positions in
the test, and test items
Students may answer different sets of questions
i.e. Answer 30 out of 50 possible questions from the bank
Np
N
![Page 43: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/43.jpg)
Click on the quiz, then access:
Administration
Quiz administration
Results
Statistics
Jane Holland
![Page 44: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/44.jpg)
Jane Holland
![Page 45: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/45.jpg)
Jane Holland
![Page 46: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/46.jpg)
Jane Holland
![Page 47: MCQs - Item Writing & Item analysis](https://reader034.fdocuments.net/reader034/viewer/2022042604/58a3ebd71a28ab272e8b580d/html5/thumbnails/47.jpg)
Good Stats Good Questions
Good stats don’t necessarily mean good
questions and good questions don’t
always get good stats...
...but it is a good place to start
Richard Arnett