Post on 07-Feb-2017
Blueprinting and drafting questions
Liz NormanMassey University
What is a blueprint?
"blueprint, n.". Something which acts as a plan, model, or template
OED Online http://www.oed.comRailway and harbour report, Vancouver, B.C. by City of Vancouver Archives, Attribution License
Why we need to blueprint
Its important that we sample representatively from the content domain (all that it is possible to examine)
This is so we can generalise performance to the whole content domain
Constructing blueprints• Table or grid with at least 2 dimensions• Dimensions could include:
– Learning outcomes– Diseases or presenting problem– Body system– Species– Knowledge domain: anatomy, pathophysiology, diagnosis,
management, prognosis, epidemiology, investigations, ethics & law– Skills: interpretation, reasoning, decision making, evaluation, problem
solving– Cognitive level: factual recall, analysis and interpretation of data,
problem solving– Degree of abstraction: principles (theoretical), applied (concrete)– Location of question in the examination components
Constructing blueprints
• Begin by considering the curriculum• What knowledge, skills, and attitudes are
examinable?• Subject guidelines
Body system by knowledge domain
Written paper 1
Written paper 2
Oral examination
Practical examination
LO1 X X X
LO2 X X
LO3 X X X
LO4 X
Body system by knowledge domain by exam component
Pathophysiology Investigation and diagnosis
Treatment and management
Gastrointestinal P1Q1 P1Q1, P2Q4 OQ3
Cardiovascular P1Q4 P2Q2, OQ1 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 OQ2 P2Q3
Respiratory P2Q5
Proportion of exam by species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%
Body system by knowledge domain by cognitive level
Pathophysiology Investigation and diagnosis Treatment and management
recall analyse, interpret
problem solve recall analyse,
interpretproblem
solve recall analyse, interpret
problem solve
Gastrointestinal P1Q1 P1Q1
P2Q4 OQ3
Cardiovascular P1Q4 P2Q2 OQ1 P2Q2
Nervous P1Q2 P2Q1
Endocrine P1Q3 OQ2 P2Q3
Respiratory P2Q5
Body system by knowledge domain by subject of question
Written paper 2 Pathophysiology Investigation and diagnosis Treatment and management
Gastrointestinal Investigation of exocrine pancreatic insufficiency
Cardiovascular Investigation of atrial fibrillation
Management of cardiomyopathy
Nervous Diagnosis of L2-3 spinal neoplasia
Endocrine Management of diabetic ketoacidosis
Respiratory Diagnosis of feline asthma Management of feline asthma
Weighting questions
Impact Weight Frequency Weight
Less important 1 Rarely seen 1
Essential 2 Relatively common 2
High impact 3 Very common 3
McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547
Drafting questions
Creating appropriate tasks
• Questions are actually tasks• We need the tasks we assess to be things that
Members or Fellows need to be able to do.• This ensures we can validly extrapolate from
performance on the exam, to performance in the Membership or Fellowship domain.
Creating appropriate tasks
• Emphasis should be on assessing the candidate’s – understanding– skills in analysing and interpreting information– skills in problem solving– judgement
• Not recall of information
Recall-higher order classification
Fact recall Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)
Applied (higher order)
Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.
Knowledge-understanding classification
Knowledge knowing about; a body of coherent facts; can be thought of as right or wrong.
Understanding knowing how and why; the meaning of facts; the theory that links facts and provides meaning; how sense is made of facts to enable them to be applied to analysis, synthesis, evaluation; to be able to explain why particular facts or skills are applicable to a particular situation; to know which fact to apply when; to be able to create new knowledge or modify or adapt an idea to a new situation.
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.
SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.
Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.
Extended abstract
Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.
Prakash, et al. (2010). Adv Physiol Educ, 34(3), 145-149.
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
Quantitative change
Qualitative change
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
surface
deep
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
recall
application
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
knowledge
understanding
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise, find, note, seek, sketch, pick
Multistructural Combine, classify, structure, describe, enumerate, list, do algorithm, apply method, account for execute, formulate, solve, conduct, prove, complete, illustrate, express, characterise
Relational Analyse, compare, contrast, integrate, relate, explain causes, apply theory (to its own domain), argue, implement, plan, summarize, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt
Extended abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, reason
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.
Specifying tasks• Examples of instructional verbs
– Compare: to find similarities between things, or to look for characteristics and features that resemble each other.
– Contrast: to find differences or to distinguish between things.– Discuss: to present a detailed argument or account of the subject
matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.
– Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.
– Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.
– Illustrate: to use a picture, diagram or example to clarify a point.
Specifying scope
• Do you want a general answer about this condition that applies to all possibilities or one that specifically applies to this case?
• Do you want an answer to include – all possible options?– those that are currently available in this region of the
world?– those that are available under cost constraints?
• Writing marking scheme helps you get the scope right!
Specify boundaries of the answer
• Speciese.g. “in both dogs and cats…”
• Quantities and amountse.g. “Provide 5 reasons why…”
• With reference toe.g. “ With reference to the published research from ..”
• Timeeg: “in the first 24 hours”
• Part of the questionEg: “for one of your differentials….”
Construct irrelevance
When the skills, abilities or knowledge required to answer examination questions is not part of the domain we are trying to assess.
What are we actually assessing?
Important sources of construct irrelevance
• Unclear tasks – because the candidate may do something other than what we intended them to do.
• Unanticipated and irrelevant difficulty and demand
• Insufficient time to complete the task (fully)
Controlling task difficulty and demand
• Tasks need to be at an appropriate level of difficulty
• They don’t need to be “tricky” in order to be difficult
Some factors that affect demand and difficulty
• Type of operation– Simple steps vs synthesis, interpretation, evaluation
• Degree of novelty
Question 1:Explain the physiological actions of insulin.
Question 2:Explain the physiological actions of ghrelin.
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved
Question 1:Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.
Question 2:Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.
Question 3:Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.
Question 4:Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate
For EACH ECG, make a recommendation for management of the horse and explain your reasoning.
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images, diagrams,
or tables provided
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images, diagrams,
or tables provided• Degree of abstraction
Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence.
Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.
Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images,
diagrams, or tables provided• Degree of abstraction• Response strategy – simple, stepwise, integrated
Describe the skull. (30 marks)
Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)
Question 2a) Describe the advantages and disadvantages of insulin therapy for
diabetes mellitus in cats (10 marks)b) Indicate the dose and frequency of administration of insulin you
would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images, diagrams, or
tables provided• Degree of abstraction• Response strategy – simple, stepwise, integrated• Guidance provided
You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.
Pacing examinations
• Reading time: 40 words per minute• Writing time: 20 words per minute
• Marking schemes help you check the pacing