Liz Norman Examination and moderation guidelines
-
Upload
liz-norman -
Category
Education
-
view
56 -
download
4
Transcript of Liz Norman Examination and moderation guidelines
Examination and moderation guidelines
Liz NormanMassey University
Examination validity
• Exams are not valid or invalid – it’s the conclusions we draw from exam results that are valid or invalid.
• The entire conclusion about the correct grade for the student, can be called into question if any step is questionable.
• Just like in research, defensible conclusions require attention to methods and protocols.
6 principles to follow1. The content for examination must be appropriate2. The questions must elicit skills and knowledge that we want
to assess3. The questions must not elicit skills and knowledge that are
not part of the curriculum4. The mark scheme must reward the appropriate level of skill
and knowledge5. The questions must be the right level of difficulty6. The highest performing students should answer questions
well and the lowest performing students should answer them less well
1. The content for examination must be appropriate
• Content (as a whole) of all assessment for the paper should:– Reflect the weighting and importance in the curriculum– Involve skills and knowledge the students have practiced and
received feedback on• Do not reuse questions for more than 10-20% of the exam:
recall exams circulate even for total recovery exams• If using old questions, check the content is still relevant and
the material has been taught• Record procedures:
– blueprint – LOs covered by questions– where and when content was taught/practiced/fedback on
Breadth by learning outcome
Lit review assignment
Group presentation
Written examination
Practical examination
LO1 X X X
LO2 X X
LO3 X X X
LO4 X
Breadth across species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%
Breadth by topic
Pathophysiology Investigation and diagnosis
Treatment and management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
2. The questions must elicit skills and knowledge that we want to assess
• The wording of the questions needs to be explicitly clear, precise, and unambiguous
• Give instructions rather than asking questions• Use tasks that require higher order thinking• Images and other ancillary materials must be high
quality and clearly show what is intended• Moderators cannot check this without seeing the
mark scheme• Moderators from different disciplines may see issues
better than moderators from the same discipline
Don’t write questions; write tasks
What is your diagnosis?
State the most likely diagnosis
State the most likely diagnosis and explain your reasoning
Discuss the differential diagnoses you would consider in this case
or …..
Instructional verb examples
Compare: to find similarities between things, or to look for characteristics and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.
Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.
Make short notes on THREE (3) of the following.a) Pulmonary compliance.b) Bicarbonate as a blood buffer.c) The respiratory centre.d) The respiratory functions of the nose.
Discuss commonly found tumours and tumour-like disorders associated with the oral cavity and dental tissues of the horse.
Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.
A dog is presented to your clinic after a road traffic accident.a) What clinical signs would be consistent with urinary bladder rupture?
Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)
How would you localise the site of the lesion?
Answer provided in the marking scheme:Spinal lesion between T3 and L3
3. The questions must not elicit skills and knowledge that are not part of the curriculum• Common issues
– Timing issues (students can only write 16 wpm)– Reading speed– Vague questions (eg write short notes on…)– Content at the wrong level– Irrelevant content/images– Trivial/unimportant content
• Also remember– Colour blindness– Don’t use choice in examinations
• Sources of construct irrelevant difficulty are multiple and unpredictable: therefore – Keep question wording concise and straightforward– Only provide information required to answer the question and relevant distractors if
appropriate– Use images only when required to answer the question
Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal 50(3): 148-157
Effect of time stress
• Time stress– Increases the use of schemas (sterotyping)– Decreases working memory processing capacity– Decreases the ability to maintain relevant
information and suppress irrelevant information
Question 1A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost?
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy & Practice, 14(2), 201-232.
Question 1A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost?
Question 2A ski pass costs $4.20. How much would it cost for 7 days?
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy & Practice, 14(2), 201-232.
Question 1A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost?
Pass rate 41.5%Question 2A ski pass costs $4.20. How much would it cost for 7 days?
Pass rate 79%
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy & Practice, 14(2), 201-232.
Ahm
ed &
Pol
litt (2
007)
Impr
ovin
g th
e qu
ality
of c
onte
xtua
lized
que
stion
s: A
n ex
perim
enta
l in
vesti
gatio
n of
focu
s. A
sses
smen
t in
Educ
ation
: Prin
cipl
es, P
olic
y &
Pra
ctice
, 14(
2), 2
01-2
32.
4. The mark scheme must reward the appropriate level of skill and knowledge
• Avoid awarding marks for unprioritised volume of facts rather than relevant and structured knowledge
• Think about the quality of thinking
Fact recall vs applied
Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)
Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact
Peitzman et al. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.
Disconnected knowledge Connected knowledge
• No particular order to aspects presented
• Inclusion of irrelevant material• Inconsistencies or incorrect
aspects• Superficial or oversimplified• Replication of material from
sources – rote learned or reproduced without significant transformation
• Aspects explained relative to one another• Logically organised answer• Analysis and or synthesis• Compares similarities and differences• Integrates multiple levels (eg: molecular, biochemical,
systemic)• Evaluates inconsistencies• Expresses reasons• Explains implications or reaches a conclusion• Expresses relative importance, value, significance of
aspects. • Selective answer that addresses the point of the
question and may be shorter than a multistructural answer
• Uses the language of the discipline - terminology and phrasing
• Relates answer to examples or experience• Relates answer to organising principles of the discipline
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press.
4. The mark scheme must reward the appropriate level of skill and knowledge
• Avoid awarding marks for unprioritised volume of facts rather than relevant and structured knowledge
• Think about the quality of thinking• Holistic marking schemes are fine, but must specify
criteria to be used• Marking schemes may need refining during marking• Aiming for the median mark to be above 70%
5. The questions must be the right level of difficulty
• After/during marking check for overly difficult questions– Was there a problem with the question wording that has
meant students misinterpreted the intended task?– Was the answer keyed correctly (for MCQs)?– Was the content actually taught/practiced and given
feedback on?– Was there sufficient time to answer the question?
• May need to– Adjust marking scheme– Delete question – consult Liz or Jenny
6. The highest performing students should answer questions well; the lowest performing students should answer them less well
• After/during marking check all items with low or negative item to total score correlation. – Problems with question wording and interpreted task?– More than one right answer, exceptions that are also
correct?– Run out of time?– Even if you can’t see the problem, construct irrelevant
difficulty is likely• May need to
– Adjust marking scheme– Delete question – consult Liz or Jenny
http://www.slideshare.net/liznorman