1

Blueprinting Project:Update and Next Steps

Elaine Rodeck, Ph.D.Examination Strategist, Blueprint Lead

Medical Council of CanadaAnnual General Meeting, September 30, 2012

2

Overview

1. Blueprinting Project - Purpose, Timeline2. Delphi Process3. National Survey4. Next Steps5. Q&A

3

Historical Perspective

• 2009 – Medical Council of Canada

(MCC) Objectives Committee revised objectives using CanMEDS as the framework

3

• October 2011– Report of the Assessment Review Task Force (ARTF)

approved by Council

4

ARTF Recommendation 2• The content of the MCC assessment processes shall

be expanded by:– Defining the knowledge and behaviours in the CanMEDS

roles that demonstrates competence of a physician about to enter independent practice

– Reviewing the adequacy of the content and skills coverage on the blueprints for all MCC examinations

– Revising the examination blueprints and reporting systems– Determining whether any core competencies cannot be

tested by the MCC examinations

4

55

MCC Validity Research Agenda

6

Why do a blueprint exercise?

6

The exercise is designed:…to define and validate the critical

competencies that the MCC examinations should sample

in order:

…to assure Medical Regulatory Authorities (MRAs) and stakeholders that a physician has the requisite knowledge, skills and professional behaviours to enter supervised/unsupervised clinical practice

7

Who is participating in the process?• Blueprinting Core Project Team• Governance Board• Steering Group• Process Participants

– Subject matter experts (SMEs) for the Delphi questionnaire and follow-up review by SMEs

– National Survey participants – a cross section of health providers, teachers, learners, and members of the public

– SMEs blueprint panelists– MCC’s Central Examination Committee

7

8

How is this being done?• Phase 1

– January 2012 to fall 2013– Information gathering

• Delphi Group/Follow-up with SMEs• Incidence and Prevalence paper• National Survey• Pilot Project on resident supervision• Medical Education Assessment Advisory Committee paper

– Establish recommendations for the new MCCQE test specifications

8

9

Blueprinting Project…– Phase I

• Approved blueprints (test specifications) for the revisions to the MCCQE Parts I and II• Identification of requirements not measurable in formats such as MCCQE Part I or Part II

– Phase II - operational transition• Gap analysis of the existing item banks• Content and form development, infrastructure, workflow design• Pilot testing• Communication Plan for all stakeholders• Develop process to consider new types of assessment

– Phase III - the launch• New examinations and content• Monitoring for ongoing content renewal.

9

Consultation with stakeholders throughout the process is key to ensuring outcomes are the result of a collaborative process!

10

What have we done?

• Incidence and Prevalence paper

• Delphi process of CanMEDS roles and MCC objectives

• Follow-up review by experts who have content knowledge

10

11

Blueprinting Project:Delphi Process

12

The Delphi Process…• Tool for consensus measurement• Determines the extent of agreement amongst a group of experts*

• Anonymity– to prevent dominance of the process by one or a few experts

• Iteration– several rounds are conducted to allow individuals to change opinions

• Controlled feedback– provide feedback on individual and group responses

• Statistical group response– synthesis and analysis of the group response

*Jones, J & Hunter D (1995) Consensus methods for medical and health services research. BMJ 311: 376-80.

12

13

The Delphi Process…The Panelists• 22 panelists…pan-Canadian medical school

representation including Quebec, medical disciplines, registrar representation, gender, etc.

The Questionnaire

• Medical expert objectives – Over 200 statements

• Non-medical expert objectives – including…professional, communicator, scholar, manager, collaborator, advocate

• Results Review – subsequent review by SMEs

13

14

The Delphi Process…

Rate objectives for entry into unsupervised practice.1.Unnecessary

2.Not Important

3.Slightly Important

4.Important

5.Very Important

6.Essential

7.Cannot Rate

14

15

Example - Delphi Process Question What is your opinion?

151 2 3 4 5 6 7

0%

25%

0%

25%25%25%

0%

1. Unnecessary

2. Not Important

3. Slightly Important

4. Important

5. Very Important

6. Essential

7. Cannot Rate

Glucose Abnormal, Serum/Diabetes Mellitus/Polydipsia

Diabetes mellitusRationale: Diabetes mellitus is an increasingly common multi-system disease associated with a relative or absolute impairment of insulin secretion together with varying degrees of peripheral resistance to the action of insulin.

Key Objectives: Given a patient with diabetes mellitus, the candidate will diagnose the cause, severity and complications, and will initiate an appropriate management plan. Particularly important are early detection of the disease, and recognition of medical emergencies such as acute hypoglycemia, diabetic ketoacidosis, and hyperosmolar nonketotic coma.

16

Example - Delphi Process QuestionWhat is your opinion?

161 2 3 4 5 6 7

0%

25%

0%

25%25%25%

0%

1. Unnecessary

2. Not Important

3. Slightly Important

4. Important

5. Very Important

6. Essential

7. Cannot Rate

Hair and Nail Complaints

Nail complaints Rationale: Nail disorders are common conditions. Although in themselves nail changes may be innocuous, they may indicate underlying disease.

Key Objectives: Given a patient with nail abnormalities, the candidate will diagnose the cause (local or systemic), severity, and complications, and will initiate an appropriate management plan.

17

Example - Delphi Process QuestionWhat is your opinion?

171 2 3 4 5 6 7

0%

25%

0%

25%25%25%

0%

1. Unnecessary

2. Not Important

3. Slightly Important

4. Important

5. Very Important

6. Essential

7. Cannot Rate

Abdominal Pain

Abdominal pain, Acute Rationale: Acute abdominal pain is a common complaint in adults, leading to frequent physician visits both in the Emergency Department and office setting. Acute abdominal pain may result from serious intra-abdominal, intrathoracic, or retroperitoneal processes.

Key Objectives: Given a patient with acute abdominal pain, the candidate will diagnose the cause, severity, and complications, and will initiate an appropriate management plan. In particular, the candidate will identify those patients requiring emergency medical or surgical treatment.

18

Example - Delphi Process QuestionWhat is your opinion?

181 2 3 4 5 6 7

0%

25%

0%

25%25%25%

0%

1. Unnecessary

2. Not Important

3. Slightly Important

4. Important

5. Very Important

6. Essential

7. Cannot Rate

Hair and Nail Complaints

Alopecia Rationale: Alopecia may be physiological or due to local scalp disease or underlying systemic disease and can result in psychological distress.

Key Objectives: Given a patient with alopecia, the candidate will diagnose the cause, severity, and complications, and will initiate an appropriate management plan.

19

Example - Delphi Process QuestionWhat is your opinion?

191 2 3 4 5 6 7

0%

25%

0%

25%25%25%

0%

1. Unnecessary

2. Not Important

3. Slightly Important

4. Important

5. Very Important

6. Essential

7. Cannot Rate

Population Health

Assessing and measuring health status at the Population LevelRationale: Knowing the health status of the population allows for better planning and evaluation of health programs and tailoring interventions to meet patient/community needs. Physicians are also active participants in disease surveillance programs, encouraging them to address health needs in the population and not merely health demands.Key Objectives: Describe the health status of a defined population. Measure and record the factors that affect the health status of a population with respect to the principles of causation.

20

Blueprinting Project:National Survey

21

National Survey• Sampling plan in process – fall 2012

– Physicians, Residents, Students, Program Directors, Recently certified CFPC/RCPSC, Nurses, Pharmacists, MRAs, public members

• Survey timing – early February 2013• SME panel review – May 2013• CEC review – June 2013

21

22

Blueprinting Project:Next Steps

23

Next Steps• 2012 Fall

– Develop national survey questions – CEC to approve national survey

• 2013– February – National survey – March /April – Analyze data, prepare for SME panel meetings– May – SME panel meetings (two)– June – CEC to review draft blueprints – July – Stakeholder invitation to comment – August/September – Revise blueprints to include stakeholder

comments– AGM – Present revised blueprints to AGM for consultation– Late fall – Blueprints to be revised and approved by CEC – Late 2013/early 2014 – Executive Board to approve blueprints

23

24

Blueprinting Project: Q&A

25

Questions, Comments & Discussion

Elaine RodeckERodeck@mcc.ca

Thank you!

25