Student :Erez Shalom

An Evaluation of a Methodology for Specification of Clinical Guidelines at Multiple Representation Levels

Student :Erez Shalom

Supervisors:

Prof. Yuval Shahar

Dr. Meirav Taieb-Maymon

ISE Dep. Seminar 26/4/06

Background Methods Results Conclusions Future Directions

Talk Roadmap :


Clinical Guidelines Textual documents describing “state of

the art” patient management A powerful method to standardize and

improve the quality of medical care Usually specify diagnostic and/or

therapeutic procedures


Subsections Describing

patient diagnosis and

treatment


The Need for Automation of Clinical Guidelines

Automatic support provides: Visual specification Search and retrieval Application of a GL Retrospective quality assurance

However: Most GLs are text based and electronic inaccessible at the point of care


The Required Infrastructure A machine-comprehensible GL representation

ontology (e.g., Asbru ontology) Runtime GL application and QA tools

A preliminary engine, namely , Spock was already developed in our lab by [Young,2005]

Support for a gradual structuring of the GL (from text to an executable code)


The Structuring ProcessThe Guideline as a tree of plansThe Guideline as a text document

Involves 2 main types of knowledge:

Procedural knowledge – e.g. Regimen A for administer the two medications in parallel

In parallel

D D

Declarative knowledge - e.g. 2 g IV

Regimen A

Cefotetan Doxycline

P

PP


Sample GL Modeling Methods

Method Knowledge Acquisition tool EON and SAGE, Prodigy, GLIF

A Protégé-based interface

PROforma Arrezo

GEM GEM-Cutter

GLARE "CG_AM" graphical interface

GUIDE NEWGUIDE

AsbruAsbru-View, GMT, Stepper


//Check HGBIf(HGB > 12)

{...}

ExpertPhysician

Collaboration KnowledgeEngineer

Semanticmarkup

Adding controlstructure

Formalizing toexecutable code

DeGeL KB

//Check HGBIf(HGB > 12)

{...}

Free textGuidelines

Semi- StructuredLevel

Semi- FormalLevel

FormalLevel

Expert Physician Knowledge Engineer

The Hybrid Representation ModelGradually structuring the GL using increasingly formal representation levels

Implemented as part of the Digital Electronic Guideline Library) DeGeL)

Used within the URUZ GL markup tool


Asbru- the Underlying Guideline-Representation Ontology

Conditions KR-Class (e.g., the filter condition, and the abort condition)

Plan-body KR-Class for the GL’s Control structures (e.g., sequential, concurrent, and repeating combinations of actions or sub-guidelines),

GL’s Goals KR-Class (e.g. process and outcome intentions),

Context KR-Class of the activities in the GL (e.g. actors, clinical-context).

Includes semantic Knowledge Roles (KRs) organized in KR-Classes such as:


URUZ (I): Specification of declarative knowledge

Expert physician Selects “filter condition” knowledge role


URUZ (cont’d) :Specification of Procedural Knowledge

Expert physician decomposing the GL into tree of plans


GL Specification: Core Issues

√ Expert Physicians (EPs) - Knowledge Engineers (KEs) collaboration

√ Incremental Specification√ Treatment of Multiple Ontologies √ Distributed Collaboration and Sharing√ Text Based Source

√ Knowledge Conversion


Several unresolved issues: Definition of the necessary steps for the GL

specification process Use optimally of EPs and KEs in the process Evaluation is crucial for quantify the markups quality

To Achieve high quality of markups there is a Need for:• An overall process of guideline specification • A complete evaluation methodology


√ Background Methods Results Conclusions Future Directions

Talk Roadmap


The Overall Process of Guideline Specification

The activities in the markup process include three main phases :

1) Preparations before the markup activities

2) Actions during the Markup activities

3) After Markup activity


A Methodology Specification of Clinical Guidelines

Evaluatingtogether

the Markups

EP

KE

Af ter Markup

9.

KE

KE

Before Markup

1.

KE ChooseSpecification

Language

KEsInstructing

the EPs

2.

EP

KE

Choosingtogether GL

forSpecification

EP

KE

3.Creating

together anOntology-Specific

Consensus

4.

EP

KE

EPs Trainingin the

Markup Tool

5.

EP

KE

Creatingtogether

GoldStandardMarkup

EP

6.

KE

KE

EPs Classifythe GL

EPs PerformMarkup

During Markup

EPEP

7. 8.

Creating a consensus is a crucial, mandatory step before markup


The Importance of Using an Ontology-Specific Consensus (OSC)

An OSC is a structural document that describes schematically the clinical directives of the GL

Described by the semantic of the specification ontology

Prevent disagreement and a great deal of variability among the EPs


Methodology for Creation of OSC

The OSC is created in a iterative fashion by performing the following steps 1. First, we create a preliminary structure of the clinical

pathway

2. The KE adds procedural, control structure

3. The KE adds declarative concepts for each defined step

4. After some iteration of steps 2 and 3, an OSC is formed


The second stage in forming a consensus

YESNOis PID

severe? (*)

Patient Treatment and evaluation

Hospitalization anddischarged(sequential)

OutpatientTreatment

Outpatient Treatment andevaluation (parallel)

Outpatient Evaluationand follow up

See 1.2See 1.7

See 1.8

Hospitalization

Post DischargedSee 1.5


Evaluation Design

Amount of Expertise The acquired knowledge domain The Ontology Specific Consensuses The Gold Standard markup for each GL The Markups for each GL The Evaluation of markups

Considered some specific Criteria :


Evaluation Design (cont’d)

Three GLs in different domains were used :Pelvic Inflammatory Disease (PID)Chronic Obstructive Pulmonary

Disease (COPD) Hypothyroidism(HypoThyrd)


Evaluation Design (cont’d)

HypoThyrd

COPD

PID

GLSources

EP

KE

HypoThyrd

COPD

PID

One OntologySpecific Consensus

For each GL

EP

MarkupMarkup

EP

MarkupMarkup

MarkupMarkup

HypoThyrd

HypoThyrd

COPD COPD

PIDPID

Two markupsfor each GL

EP

PI DGoldStandardGold

Standard

GoldStandard

HypoThyrd

COPD

One Gold StandardMarkup f or each GL

PI D

EP

KE

Each markup isevaluated compare to

the Gold Standard

EP


Research Questions Is markup feasible by EPs? Is there a difference between the EPs

editing the same GLs , and same EPs editing difference GLs?

Is there a difference between the KRs across all EPs?

Is there a difference in the amount of errors when using different OSC?


Evaluation of markups

Subjective Measures - Questionnaires were administered for finding the EPs attitude regarding the specification process

Objective Measures – in two scales (compared to the GS):

* Completeness of the markup * Correctness of the markup


The Objective Measures - Completeness

A CB

GS Markup


* Clinical Measure (CM) – measure the clinical correctness of the content* Asbru Semantic Measure (ASM) - measure the semantics correctness of the content ( Asbru semantic in our case)

The Objective Measures - Correctness


Resolution of Measure Mean (weighted) Quality Score (MQS) for: GLs - to find common trends in a GL, and in all GLs

EPs - to find trends in between the markups of the

EPs across the same GL and between GLs KRs - to find trends in a specific KR type and

common trends across KRs and KR classes across one markup, GL and in all GLs


General errors classified into two types, and thus into two corresponding scales:

Clinical errors: Clinical content not accurate Clinical semantics not well specified Clinical content not complete.

Asbru semantics errors: Asbru semantics content not accurate Asbru semantics content not well specified The content does not includes mappings to standard terms The necessary knowledge is not defined in the guideline

knowledge when it should be.

The Objective Measures – Types of Errors


Specific errors for each KR Type a specific error, for example :

Conditions /Intentions KRs: There are no And/Or operators between the different criteria.

Simple Action Plan-Body Type: Has no text content describing the plan Has no single atomic action semantics with clear specification

and description for the action to be performed.

Plan Activation Plan-Body Type: Plan name is not defined Defined plan does not exist in DeGeL.

The Objective Measures – Types of Errors


The Markup-Evaluation Tool


The Markup-Evaluation Tool (Cont’d)


√ Background√ Methods Results Conclusions Future Directions

Talk Roadmap


Results – Subjective Measures PurposeStageResult

1Finding the Aspects that most helped the EPs' when creating an OSC

after creating the OSC

Using their medial knowledge and their understanding of the specification ontology (Asbru, in our case) Vs. specification Tools

2Finding the Aspects that most helped the EPs' making a markup

after markupSpecification Tools is considered as more helpful

3Finding how well The EPs Understand Asbru KRs

before markupDeclarative KRs are more easy to understand (such as filter condition)

4Finding what were the difficulties of the EPs' in structuring the Asbru KRs

after markupProcedural KRs are more easy to structure (such periodic plan)

5System Usability Scale (SUS) for URUZ

after markupSUS=47 ; Not Usable!

• Non significant correlation between results 1 and 2

• Significant correlation between results 3 and 4


Results – Objective measures

• Mean Completeness for all markups of EPs of 91%

•All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75)

Markup is feasible by EPs

Number of specified plans:

Measures Summary:


Results –Difference between EPs

•Any EP can perform markup with high completeness

•There is wide variability between the EPs in the correctness measure with a range of [0.13,0.58] on a scale of [-1,1]

√Difference in correctness measure between different GLs editing the same EPs4

Difference in Correlation measure between EPs editing the same GLs in most GLs (except the PID)3

√Difference in correctness measure between EPs editing the same GLs in most GLs (except the Hypo)

2

nonsignificant (P>0.05)

Difference between the proportions of completeness measure between EPs editing the same GLs

1

significant (P<0.05)

Issue

√

√


Results – Difference between KRsThere was significant difference (P<0.05) between homogenous groups of KRs

EPs has difficulty to structure procedural KRs than declarative ones


Results – Types of errorsMarkup Error Rate(MER) per KR, per EP

0.55

0.41

0.12

0.31 0.31

0.09

0.24

0.090.03

0

0.1

0.2

0.3

0.4

0.5

0.6

PID COPD HypoThyrdGLs

ME

R

Total Ontological Clinical

360

199

360

112

360

87

194

79

194

61

194

1898

12

98

9

98

3

2*.

.

i

i

GLinKRsofNo

GLinerrorsofNo= MERi.

The differences in total between the three GLs were highly significant in a proportion test (P<0.001)

The more detailed and structured the OSC was, the lower the total number of errors committed by the EPs for each KR


√ Background√ Methods√ Results Conclusions Future Directions

Talk Roadmap


Four main aspects:

Creation of an Ontology-Specific Consensus (OSC)

The essential aspects needed to learn to support the specification process by EPs

The medical and computational qualifications needed for specification

The characteristics of the KA tool needed for this kind of specification


Creation of an Ontology-Specific Consensus (OSC)

Should be made as detailed as possible, including all relevant procedural and declarative concepts

The OSC is independent of the specification tool

Saving the OSCs in an appropriate digital library for re-using and sharing


The Essential Aspects Needed to Learn to Support the Specification Process by EPs creating an OSC and performing the markups are two

different tasks which require teaching two different aspects

Teaching the “difficult” KRs in particular, the procedural KRs

Short test should be administered before the EPs perform markups

A help manual and a small simulation of marking up a GL should be included in the teaching session

ISE Dep. Seminar 26/4/06The Medical and Computational Qualifications Needed for Specification

Senior EPs and KEs together should work on the tasks of selecting a GL for specification and making the OSC

Any EP (senior, non-senior or a general physician) can structure the GL's knowledge in a semiformal representation completely

To specify it correctly, a more available EP should be selected, perhaps from among residents, interns or even students


The Characteristics of the KA Tool Needed for This Kind of Specification A robust, graphical, highly usable framework is

needed More intuitive, graphic, user friendly interfaces

should be used for acquiring the “difficult” KRs , especially the procedural ones

Need to bridge the gap between the initial structuring of the EP and the full semantics of the specification language

GESHER - A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels


Limitations and Advantages of the research

Small of the number of EPs and GLs, But, in fact, 196 sub-plans and 326 KRs in total were structured by all of the EPs together in all markups

Lack of careful measurement of the required time , but, obtain more realistic results, since the interaction with most of the EPs took place in their own "playground"


√ Background√ Methods√ Results √ Conclusions Future Directions

Talk Roadmap


The GESHER’s Main Features

User friendly graphical client application Support specification at multiple

representation levels Support to multiple specification languages

(GL ontologies) Access centralized resources such as

DeGeL and a knowledge base


GESHER: Semi-Structured LevelThe Hybrid

Ontology Treeshowing KRs

at allrepresentation

levels


GESHER(II) :Semi-Formal Widgets


√ Background√ Methods√ Results√ Conclusions√ Future Directions

Talk Roadmap


Summary The need for gradual GL specification

Making an ontology-specific consensus as first step Use a well defined methodology for the overall process

Markup is feasible by EPs Any EP can perform markup with high completeness We should use methodology for increase quality of markups Use GESHER as the new framework for specification Ongoing new research is being conducted (Pre-Eclampsia GL)

based on this research results


Research Publications 2005סביבת עבודה גראפית במס' רמות ייצוג, הכנס הישראלי למ"מ רפואיות Shalom, E. and Shahar Y. (2005). A Graphical Framework for Specification

of Clinical Guidelines at Multiple Representation Levels. AMIA Annual Fall Symposium, Washington DC, USA

2006האם רופאים מסוגלים להבנות ידע רפואי? הכנס הישראלי למ"מ רפואיות Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A,

B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E. (2006) A Methodology for Evaluation of A Markup-Based Specification of Clinical Guidelines Submitted to AMIA , Washington DC, USA

Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A, B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E.(2006) The Importance of Creating an Ontology-Specific Consensus Before a Markup-Based Specification of Clinical Guidelines , Submitted to ECAI06 ,Tronto, Italy

JAMIA journal paper is in preparation


Acknowledgments Prof. Yuval Shahar and Dr. Meirav Taib-Maymon All Medical informatics research center members Our colleagues at Soroka’s university medical center :

Prof. Eitan Lunenfeld, Dr. Avi Yarkoni, Dr. Guy Bar, Prof. Yair Liel and Dr. Tal Marom

Our colleagues at Stanford and VA hospital: Drs. Mary Goldstein, Susana Martins, Lawrence Basso, Herbert Kaizer, Laszlo Tudor Vaszar

NLM award No LM-06806 Contact info : [email protected] Visit our web site : http://medinfo.ise.bgu.ac.il/medlab/


Questions?


EPs create a Clinical consensus :

EPs and KE adds procedural knowledge :

EP+KE add declarative knowledge :

Ontology- specific consensus

Doxycycline Metronidazole

Order :parallel

Filter condition for drug :Is patient not sensitive to Doxycycline and the drug available?

Give Doxycycline 100 mg orally or IV every 12 hours Plus Metronidazole 500 mg IV every 8 hours

Methodology for creating an Ontology-Specific Consensus


Textual Source Parenteral Regimen A

Cefotetan 2 g IV every 12 hours

OR Cefoxitin 2 g IV every 6 hours PLUS Doxycycline 100 mg orally or IV every 12 hours.

Parenteral Regimen B

Clindamycin 900 mg IV every 8 hours

PLUS Gentamicin loading dose IV or IM (2 mg/kg of body weight) followed by a maintenance dose (1.5 mg/kg) every 8 hours. Single daily dosing may be substituted.

Alternative Parenteral Regimens

Ofloxacin 400 mg IV every 12 hours

OR Levofloxacin 500 mg IV once daily WITH or WITHOUT Metronidazole 500 mg IV every 8 hours OR Ampicillin/Sulbactam 3 g IV every 6 hours PLUS Doxycycline 100 mg orally or IV every 12 hours.


The first stage in forming a consensus

Parenteral Regimen A

Cefotetan 2 g IV every 12hours

ORCefoxitin 2 g IV every 6hours

PLUSDoxycycline 100 mg orallyor IV every 12 hours.

Parenteral Regimen B

Clindamycin 900 mg IV every8 hours

PLUSGentamicin loading dose IV orIM (2 mg/kg of body weight)followed by a maintenancedose (1.5 mg/kg) every 8 hours.Single daily dosing may besubstituted.

Alternative Parenteral Regimens

Ofloxacin 400 mg IV every 12hours

ORLevofloxacin 500 mg IV oncedaily

WITH or WITHOUTMetronidazole 500 mg IV every8 hours

ORAmpicillin/Sulbactam 3 g IVevery 6 hours

PLUSDoxycycline 100 mg orally or IVevery 12 hours.

Inpatient treatment -Give one of the following regimens


The second stage in forming a consensus


The third stage in forming a consensus


Results – Subjective measures


Results – Objective measures

• Mean Completeness for all markups of EPs of 91%

•All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75

Markup is feasible by EPs

Student :Erez Shalom

Documents

Transcript of Student :Erez Shalom