Going the Distance: Identifying Gaps & Documenting...

Going the Distance: Identifying Going the Distance: Identifying Going the Distance: Identifying Going the Distance: Identifying

Gaps & Documenting the Gaps & Documenting the Gaps & Documenting the Gaps & Documenting the

Implementation of the Implementation of the Implementation of the Implementation of the ACCMEACCMEACCMEACCME’’’’ssss

Updated CriteriaUpdated CriteriaUpdated CriteriaUpdated Criteria

Thursday, November 19, 2009

2:00 – 3:00 PM ET

Faculty & Disclosure

Robert Robert Robert Robert BartelBartelBartelBartel, , , , MScMScMScMSc, Director of Education, Director of Education, Director of Education, Director of Education

• Nothing to disclose

Wanda Johnson, CMP, CAE, Wanda Johnson, CMP, CAE, Wanda Johnson, CMP, CAE, Wanda Johnson, CMP, CAE, Senior Director of Meetings & EducationSenior Director of Meetings & EducationSenior Director of Meetings & EducationSenior Director of Meetings & Education


Eve Wilson, PhD, Associate Director, EducationEve Wilson, PhD, Associate Director, EducationEve Wilson, PhD, Associate Director, EducationEve Wilson, PhD, Associate Director, Education


Learning Objectives:

Upon completion of this activity, successful learners should be able to:

• Describe and implement one successful strategy for identifying and

measuring gaps in your target learners so that you may design activities to

address these gaps.

• Reach beyond the CME unit in your organization to identify and collaborate

with colleagues who can help you document the many ways in which your

organization is addressing the criteria for commendation.

• Engage the CME unit in review of activity files to best document continuous

improvement and to best educate the whole team.

Reaccreditation Demographics

Please select the one description that best matches your circumstances:

1. Self-Study in progress

2. Self-Study imminent but not yet in progress

3. Self-Study completed under the Updated Criteria

4. Self-Study completed under the old criteria

Part 1:

Directly Measuring Gaps in

Your Own Learners

Robert Bartel, MSc

Director of Education

Fall 2006 ACCME Updated Criteria

Emphasized:

• Design of activities based on identified gaps in

knowledge, competence or performance of your target

learner audience

• Measurement of changes in competence, performance

or patient outcomes

• Panic promptly ensued!

• Many core Society educational activities are curriculum

based• Annual Meeting

• Post-Graduate Course

• Board Review Courses

• Self-Assessment Programs

• Journal CME

• How do you approach translating gaps into needs and

objectives for an entire curriculum?

• Direct measure of learners’ gaps from a curricular level

investigation

Challenge = Opportunity

• 1998 Self Assessment Committee (SAC)

formed and the first ESAP printed

• 160 cases with objectives, questions,

answers and discussions

• Content largely follows ABIM Blueprint

for certification exams in Diabetes,

Endocrinology and Metabolism

• 2006 ESAP goes online

• Data collection possible for

identification of gaps

Endocrine Self-Assessment (ESAP)

Opportunity Knocks…

Outcomes Analysis of Learner

Knowledge & Competence

Knowledge & Competence Gaps:

Diabetes

Drilling Down on Gaps in the Differential

Diagnosis of Diabetes

(Subsection Ave: 72.9%)

A. Recognize the differential effects that members of the atypical

antipsychotic drug family have on glucose homeostasis (85.1%)

B. Recognize the factors that can affect the development and

measurement of hemoglobin A1c (66%)

C. Recognize the presentation and pathophysiology of the somatostatinoma

syndrome (76.6%)

D. Recognize conditions that may affect the validity of assays for

hemoglobin A1c (93.6%)

E. Recognize postprandial hypoglycemia caused by anti-insulin antibodies

(57.4%)

F. Treat glucocorticoid-induced diabetes (55.3%)

G. Recognize the clinical presentation and management of latent

autoimmune diabetes in adults (LADA) (76.6%)

Do incorrect answers = gaps?

• Question validation is an important consideration for

certification exams

• TES is substantiated gaps in our learners by adopting

this method of question validation

― Difficulty (40-90% Correct Response Rate)

― Discrimination (Top Half Participants Answer Correctly More

Than Bottom Half)

― Analysis of Distractors:

• Suboptimal Distractors that are “Too Good” or Not Incorrect

Once our learners’ gaps are identified

how do they inform activity planning?

• Once outcomes are analyzed & “bad questions”

eliminated…

• Data reports disseminated to all educational planning

committees

• Staff also highlight gaps to program directors, faculty

and collaborators as appropriate

• Since data from each question is reported with an

objective, the objective usually becomes part of the new

activity

• ESAP & ESAP MOC

• ESAP In-Training Exam

• Endocrine Self-Assessment Tools (ESATs)

• ARS data from live courses

― Endocrine Board Review Course

― Pediatric Endocrine Board Review Course

― Annual Meeting Sessions

― Satellite Symposia

Many Opportunities for Direct Measures

Integrating Gaps Data into the

Society’s CME Program

Courses DATA = Direct Measure of Gaps in our Learners

(ESAP, ESAP ITE , ESAP MOC & Board Review)

Needs for entire Program/Curriculum-wide Activities

Annual Meeting, Post-Graduate Course, focused activities

Outcomes (Re-assessments): Evaluations, follow-up activities & their evaluations & surveys, etc.

Part 2:

Addressing the Criteria for

Commendation:

Looking Beyond your CME Unit

Wanda Johnson, CMP, CAE,

Senior Director of Meetings & Education

Accreditation with Commendation

Are you planning to pursue accreditation with commendation?

1. Yes

2. No

3. N/A

Accreditation with Commendation

Criteria 16 - 22

• Need to consider with a broad perspective

• May be challenging when considered in context of CME

unit… but…

• Compliance may exist when considered in context of

entire organization

Aah Ha! Moment

• Reach beyond the CME unit to identify and collaborate

with colleagues who can help you document the many

ways in which your organization is addressing the

criteria for commendation

• ASK for input – what are they doing in their

departments that address the criteria?

An Example…

• Using one criteria to demonstrate – What did we do?

• C 21. “The provider participates within an institutional

or system framework for quality improvement.”

― What does this mean?

― Does your organization do it?

― What are some examples?

Criteria 21 – One Approach

• The provider participates within an institutional or system framework for quality improvement.

• Ask the question – do we do this?

― “institutional or system framework” = our organization and/or larger medical system

― “quality improvement” = a goal for our physician members

― So, YES, the Society does this

Criteria 21 – One Approach

• What do we do? What are all of the programs or services

or partnerships or collaborations or opportunities that

demonstrate improving the quality of physician care?

― Based on feedback from our colleagues within Society, we

identified & described two – NEITHERNEITHERNEITHERNEITHER from CME Unit

• Performance Measures Subcommittee – internal

• Physician Consortium for Performance Improvement® - external

(AMA program that Society participates)

Criteria 16 - 22

• Considering the entire organization, many providers can demonstrate compliance with criteria.

• Be broad in your interpretations of the questions.

• Get your colleagues to help! Different perspectives view criteria in different ways.

• Consider providing documentation in self-study. May be worth the effort!!

Part 3:

Preparing Materials for

Performance-in-Practice

Review

Eve Wilson, PhD

Associate Director Education

CME Staffing

How many staff comprise your CME unit?

• >10 employees

• 5 – 9 employees



Keys to Success

• Follow ACCME instructions carefully

• Use a planning document

• Engage all CME staff

• Show off your best work: select documentation examples

judiciously

• When in doubt, label it!

• Be your surveyors’ best friend: organize the files logically

• Make sure contents of the performance-in-practice files are

consistent with what’s written in your Self Study, and vice

versa

Keys to Success –

Activity Planning Checklist

Advantages:

• Guides staff through

planning and

implementation

• Helps to “systematize”

compliance with many of

the ACCME Criteria

• Record of compliance with

C1–C6, C11, C18–C20

• Easy for a surveyor to

interpret


Educational Educational Educational Educational GapGapGapGap

Current knowledge, skills, competence, practice,

clinical/ patient outcomes

Desired/ideal knowledge, competence, performance, clinical/patient outcomes

Only 55% of participants in ESAP 2007 showed proficiency in diagnosing and treating glucocorticoid-induced diabetes

Endocrinologists need to be able to manage glucocorticoid-induced diabetes and other atypical presentations of diabetes

Gaps/Needs (C2, C3)Summarize the educational or professional practice gap(s) that underlies the need(s) that the activity will addres s. The “gap” is defined the difference between the current state of knowledge, skills, competence, practice, or patient outcomes a nd the ideal ordesirable state. Note that the gap should be target-audience specific.


� Planning/steering committee meeting notes

or minutes

� Planning/steering committee or faculty

communications

� Literature search

� Peer-reviewed scientific/clinical publications

� Survey data

� Epidemiologic data

� Clinical practice guidelines

� Evaluation data

� Exam performance analysis (eg, ESAP or

ABIM performance analysis)

� Quality improvement guidelines

� Public health data

� Government mandates/legislation

� Expert input (identify expert[s]):

� Other (specify):

Gaps/Needs (C2, C3)What sources were used to identify the professional practice gaps (gaps in the target audience’s knowledge, competence, per formance) underlying the need for this activity? (Check all that apply/minimum of 2 sources; at least one source should be evidence-based. Copies of the sources used must be included in the activity file.)

Briefly explain the need(s) for this activity based on the gap(s) identified above (1-2 sentences per gap).


Barriers (C18, C19)

What barriers to physician change and/or factors outside What barriers to physician change and/or factors outside What barriers to physician change and/or factors outside What barriers to physician change and/or factors outside

TESTESTESTES’’’’ssss control (control (control (control (egegegeg, costs of care, reimbursement issues, , costs of care, reimbursement issues, , costs of care, reimbursement issues, , costs of care, reimbursement issues,

patient compliance, physician misconceptions about patient compliance, physician misconceptions about patient compliance, physician misconceptions about patient compliance, physician misconceptions about

treatment, patient misconceptions about treatment, treatment, patient misconceptions about treatment, treatment, patient misconceptions about treatment, treatment, patient misconceptions about treatment,

physician resistance to change) were identified in planning physician resistance to change) were identified in planning physician resistance to change) were identified in planning physician resistance to change) were identified in planning

this activity?this activity?this activity?this activity?

Briefly explain how the activity addresses the Briefly explain how the activity addresses the Briefly explain how the activity addresses the Briefly explain how the activity addresses the

barriers/factors identified above.barriers/factors identified above.barriers/factors identified above.barriers/factors identified above.


Educational Objectives (C2, C3)

Based on the gaps/needs identified above, what are the learning Based on the gaps/needs identified above, what are the learning Based on the gaps/needs identified above, what are the learning Based on the gaps/needs identified above, what are the learning objectives for objectives for objectives for objectives for

this activity?this activity?this activity?this activity?

Classify each learning objective according to whether its fulfilClassify each learning objective according to whether its fulfilClassify each learning objective according to whether its fulfilClassify each learning objective according to whether its fulfillment would lment would lment would lment would

represent a change in physician competence, physician performancrepresent a change in physician competence, physician performancrepresent a change in physician competence, physician performancrepresent a change in physician competence, physician performance, or e, or e, or e, or

patient outcomes. patient outcomes. patient outcomes. patient outcomes. (For each objective, check all that apply.)

Learning Objective Outcome Level Addressed1. � Competence � Performance � Patient Outcomes

2. � Competence � Performance � Patient Outcomes

Etc. � Competence � Performance � Patient Outcomes

Need Learning objective1.

2.

Etc.


Content Development (C2, C3)• The gaps/The gaps/The gaps/The gaps/need(sneed(sneed(sneed(s) identified in the needs assessment have been incorporated into) identified in the needs assessment have been incorporated into) identified in the needs assessment have been incorporated into) identified in the needs assessment have been incorporated into

the activity and all learning objectives are addressed in the cothe activity and all learning objectives are addressed in the cothe activity and all learning objectives are addressed in the cothe activity and all learning objectives are addressed in the content. ntent. ntent. ntent. Include sample

materials in the activity file.

This activity has been designed to change competence, performancThis activity has been designed to change competence, performancThis activity has been designed to change competence, performancThis activity has been designed to change competence, performance, or patient e, or patient e, or patient e, or patient

outcomes outcomes outcomes outcomes (check all that apply).

� CompetenceCompetenceCompetenceCompetence

List design elements (eg, case-based discussion, ARS, pre-test/post test focused on

clinical decisions) incorporated to help change competence:

� PerformancePerformancePerformancePerformance

List design elements (eg, hands-on training) incorporated to help change

performance:

� Patient OutcomesPatient OutcomesPatient OutcomesPatient Outcomes

List design elements (eg , performance improvement questionnaire with follow up)

incorporated to help change patient outcomes:


Content Development (C4, C6)

This activity includes content that fits the current or potentiaThis activity includes content that fits the current or potentiaThis activity includes content that fits the current or potentiaThis activity includes content that fits the current or potential scope of l scope of l scope of l scope of

professional activities for the target professional activities for the target professional activities for the target professional activities for the target audience(saudience(saudience(saudience(s). ). ). ). Check the appropriate boxes

below.

� The content covers one or more of the areas of knowledge or clinical judgment in

which an endocrinologist should demonstrate a high level of competence, as

defined by the American Board of Internal Medicine (ABIM) or similar authority.*

� The content is covers one or more areas of knowledge or clinical judgment in which

[other primary target audience] should demonstrate a high level of competence, as

defined by The ABIM, American Board of Family Medicine (ABFM), American Board

of Pediatrics (ABP), or similar authority.*

*Include the ABIM certification exam blueprint in Endocrinology Diabetes and

Metabolism or similar outline, with topic areas highlighted, in the Activity File.

� This activity incorporates one or more desirable physician attriThis activity incorporates one or more desirable physician attriThis activity incorporates one or more desirable physician attriThis activity incorporates one or more desirable physician attributes (butes (butes (butes (egegegeg, ACGME, , ACGME, , ACGME, , ACGME,

IOM/ABMS competencies). IOM/ABMS competencies). IOM/ABMS competencies). IOM/ABMS competencies). A completed Desirable Physician Attributes checklist

must be included in the Activity File.


Teaching Methods/Tools/Formats (C5)

What formats and tools will be used to ensure that learners are What formats and tools will be used to ensure that learners are What formats and tools will be used to ensure that learners are What formats and tools will be used to ensure that learners are fully fully fully fully

engaged and objectives are achieved? engaged and objectives are achieved? engaged and objectives are achieved? engaged and objectives are achieved? (Please try to incorporate at least 2

of the following into planning)

� Lecture

� Panel discussion

� Question & Answer

� Slide presentation

� Audience response system (ARS) or similar audience polling system

� Case-based/Problem-based discussion

� Hands-on practice/workshop

� Small group discussion

� Debate

� Pre-test/post-test

� Other (please describe): __________________________


� Pre-registration survey

� Pre-test

� Post-test

� Post-activity evaluation form

� Audience response system technology

� Follow-up survey/assessment

� CME Reviewer evaluation

� Other:

Adjunct Strategies for Reinforcement (C17)

Check the evaluation method(s) to be used to assess outcomes.

List strategies or tools that will be provided to enhance desired

change/outcomes? (eg, reminders, checklists/templates, treatment algorithm

leave-behinds, patient education materials, access to interactive web tools, case-

based follow-up survey):

Collaborations (C20)

List any List any List any List any collaborator(scollaborator(scollaborator(scollaborator(s) involved in planning, approving, endorsing, or other ) involved in planning, approving, endorsing, or other ) involved in planning, approving, endorsing, or other ) involved in planning, approving, endorsing, or other

roles, if applicableroles, if applicableroles, if applicableroles, if applicable. (Examples include ABIM, APDEM, the Lawson Wilkins

Pediatric Endocrine Society, other specialty societies, medical schools, The

Hormone Foundation).

Evaluations (C11)


Approval (C1)

� This activity is congruent with The Endocrine Society’s CME Mission

(Criterion 1).

� This activity has been reviewed and approved for certification. It is

certified for AMA PRA Category 1 Credit(s)™.

___________________________________________________________

Signature Date

(Director or Associate Director, Education)

Keys to Success –

Engage all CME staff

• Hold an initial training session focused on preparing

performance-in-practice review files

• Delegate collection of file materials to those most

familiar with each activity

• Meet again to go over materials collected, answer

questions, and address any problems

• Apply ACCME performance-in-practice review labels

last—after staff are sure about where to place them

Keys to Success – Applying Labels

• Choose examples/sample materials wisely

• Try to find a match for every label

• Highlight and write notes, eg, “evidence for C2,” “applies to C4”

• Some criteria will not apply to all activities

C21: Participation within an institutional or system framework for quality improvement, if applicable

C20: Collaboration and cooperation with other stakeholders, if applicable

C17: The use of non-education strategies to enhance change as adjuncts to the CME activity, if applicable

CriterionCriterionCriterionCriterion ExampleExampleExampleExample

16 Approval letter from ABIM for ESAP MOC 2008

17 Copy of bilingual patient fact sheets distributed in conjunction with a diabetes symposium

Copy of on-topic clinical practice guideline distributed in conjunction with a symposium on cardiometabolic risk

18 Survey data plus examples showing content that addresses challenges such as patient health literacy (patient fact sheets) and compliance with treatment (slide showing steps to encourage compliance)

19 Survey data plus examples showing content that addresses barriers such as reimbursement difficulty (Current Procedural Terminology [CPT] coding template)

20 Communications between TES and the Juvenile Diabetes Research Foundation identifying gaps to address in CME activity on continuous glucose monitoring

21 Addressed in the Self-Study narrative

22 Annual Meeting Steering Committee meeting minutes showing planning process, including gaps to be addressed in sessions

Examples of Documentation for Criteria 16 – 22

Keys to Success –

Organizing the Files

• Order documents logically

• Create a top sheet that shows what is in the file and

how it is organized, eg,

― Table of contents

― List of ACCME Criteria

• If documentation is missing, be up front about it and

include a brief explanation

Keys to Success –

Consistency with Self Study

• Contents of the performance-in-practice files need to be

consistent with what’s written in your Self Study, and

vice versa

Questions

Please submit questions and comments via the chat box.

Thank you!

Please complete the brief online evaluation.

Your feedback is appreciated!

Join Us for the Next Webinar!

Adding Value with CME: How an Independent CME Adding Value with CME: How an Independent CME Adding Value with CME: How an Independent CME Adding Value with CME: How an Independent CME

Provider Can Support Quality Improvement Provider Can Support Quality Improvement Provider Can Support Quality Improvement Provider Can Support Quality Improvement

December 3, 2009 at 2:00 PM EST

Going the Distance: Identifying Gaps & Documenting...

Documents

Transcript of Going the Distance: Identifying Gaps & Documenting...