Advancing Online Assessment in Medical Education

Greg Doyle, Faculty of Health Sciences Stephen Marquard, Centre for Educational Technology

(Advised and enthused by Rachel Weiss, Faculty of Health Sciences)University of Cape Town

June 2010

11th Sakai Conference - June 15-17, 2010 2

Overview

• Context • About Extended Matching Items• EMIs in Tests & Quizzes• Managing MCQs in a Question Bank• Sharing MCQs between institutions

UCT Health Science Initiatives

311th Sakai Conference - June 15-17, 2010

• Improve assessment practices• Introducing the use of Extended Matching

Item (R-type) MCQs• A Question Bank application • Author, manage, reuse and share MCQs

• Collaborating to develop and share MCQs

11th Sakai Conference - June 15-17, 2010 4

What is an EMI?

• Extended Matching Item, aka R-type MCQ• EMI = R-Type MCQ (vs A-Type MCQ)• An EMI has:• A theme• A list of options (text or labelled image)• A lead-in statement• A list of items

11th Sakai Conference - June 15-17, 2010 5

Sample EMI

This example shows options labelled1, 2, 3… but more conventionally should be labelled a, b, c …

11th Sakai Conference - June 15-17, 2010 6

EMI variants

• Items may require choosing:• One option• Multiple options (e.g. a, e, g) independent of

order [in previous example, options 1,5,7]• Options in a specific order (e.g. g, a, e)

• Implications for grading• Limits the ability to use older question types

(e.g. fill-in-the-blank) for EMIs

EMIs

• Advantages• More discriminating• Reproducibility using one-third fewer items• Lower probability of guessing the correct answer• Functional distractors• Reusable by just writing a new stem/item

• Disadvantages• Analysis, cannot test all content• Not much software support yet

EMIs in Tests & Quizzes

• What we used to do

• Work around using fill-in-the-blank

• So… http://jira.sakaiproject.org/browse/SAK-17243 :

Work underway to add support for EMIs as a new question type in Tests & Quizzes, including statistics and analysis.

When complete and reviewed by Stanford, should become available in a future T&Q release.

11th Sakai Conference - June 15-17, 2010 9

Beyond Question Pools

• Lifecycle of a question• Now: authored, lives in a Question Pool,

delivered in assessments• Should be: created, reviewed, tagged

(metadata), selected for assessments, delivered, updated with delivery history (e.g. difficulty, discrimination), updated, shared, reused

11th Sakai Conference - June 15-17, 2010 10

We imagine a Question Bank which…

• Supports A-type (traditional) MCQs, EMIs, F-format questions (progressive sequence)

• Supports embedded rich media (image/video) • Supports authoring/revision/approval workflow• Includes flexible question metadata• Enables blueprinting of assessments by selecting sets of

questions to a target level of difficulty• Records statistics from the delivery history of the

question• Enables internal and external reuse and sharing

11th Sakai Conference - June 15-17, 2010 11

Sharing MCQs

Question BankUCT

Assessment engine

(e.g. Sakai T&Q)

<< Questions

>> Delivery history and statistics

Question Bank

University A

Public OER

Question Bank

Question Bank

University B

11th Sakai Conference - June 15-17, 2010 12

We’d love to hear from you!

• If you are now or anticipate using Extended Matching Items for online assessment

• If you’d like to find ways to increase sharing and reuse of assessment items.

• If the type of Question Bank we’ve envisaged sounds attractive to you.

• If you want to help build a Question Bank tool / application.

11th Sakai Conference - June 15-17, 2010 13

Contact us

stephen.marquard@uct.ac.za

gregory.doyle@uct.ac.za