Accessing Tacit Knowledge and Linking it to the Peer-Reviewed Literature

Accessing Tacit Knowledge and Linking it to the Peer-

Reviewed Literature

Michael ShepherdWeb Information Filtering LabFaculty of Computer Science

Dalhousie University

Research Team

• Students– Qiufen Qiu (MD and MCS)– Zhixin Chen (MHI and BSc)

• Computer Science Faculty– Michael Shepherd– Qigang Gao– Syed Sibte Raza Abidi

• Anaesthesia & Psychology– G. Allen Finley

Overview

• Introduction

• Research Program

• Results to Date

• Summary

Pediatric Pain Discussion List

• Clinical discussion on pediatric pain

• Informal email-based discussion among professionals

• Initiated in 1993

• Over 700 subscribers world-wide

• More than 10,000 messages

Date: Wed, 04 Jan 1995 16:54:48 -0500 (EST)From: posterSubject: opioids and meningitis

X is a 13 month (9.8kg) old boy suffering from acute meningitis (pneumocoque) treated with IV cefotaxime; at day three, I have been called as pediatric pain consultant to assess X; I have discovered an extreme painfull state: one could not handle or touch him without producing screaming. The child was unable to move spontaneously he looked paralysed by pain and hypertonia ; he also presented a neurological complication : ptosis at the right side.The pain treatment was IV acetaminophen. The first day I have prescribed IV Nalbuphine (weak opioid u antagonist and agonist) 11mg/24h after a loading dose of 1.4 mg; Pain at rest has been succesfully relieved but not the mobilisation pain; the dose has been increased at 14 mg/day wihout relieving the pain associated with moving; he has moved spontaneously limbs 2 days later; nalbuphine has been stopped 4 days later. Neurological examination and CT scan have been still normal (except ptosis) during this period. No opioid's side effects have been observed.

What do you think of this case ?Have you any experience with opioids and acute meningitis ?

Dr Poster, Pediatric pain unit, Poster Hospital

Date: Wed, 04 Jan 1995 17:27:25 -0500 (EST)From: first replySubject: re: opioids and meningitis

Is there any periosteal involvement? If so an NSAID (ibuprofen or naproxen) may be much more effective than even opioid.

-------------------------

Date: Wed, 04 Jan 1995 19:06:32 -0400From: second replySubject: Re: opioids and meningitis

Poster writes:> X is a 13 month (9.8kg) old boy suffering from acute meningitis...> extreme painfull state: one could not handle or touch him without> producing screaming....> The first day I have prescribed IV Nalbuphine ...> succesfully relieved but not the mobilisation pain;...> has moved spontaneously limbs 2 days later; nalbuphine has been stopped 4> days later. Neurological examination and CT scan have been still normal...

I have used IV morphine for similar severe meningitis pain, with success. I wouldn't hesitate to use a pure opioid agonist (in conjunction with acetaminophen, NSAID, and/or tricyclics). However, it sounds like you have the situation under control.

Second Reply, Associate Professor, Dept and University

-------------------------

Date: Thu, 05 Jan 1995 18:58:32 -0800 (PST)From: Third ReplySubject: Re: opioids and meningitis

I wonder if the problem is not due to severe arachnoiditis that is secondary to the inflammation. I would suggest a trial of steroids in this patient, perhaps in combination with a benzodiazepine to reduce the spasm. Narcotics may reduce the pain but I would not like to keep X on them for too long.

Good luck

Third Reply

-------------------------

Tacit and Explicit Knowledge

• Tacit knowledge is what the knower knows and is derived from experience

• Explicit knowledge is represented by some artifact such as a document or journal article

Tacit Knowledge

Explicit Knowledge

Internalization

Co

mb

inatio

n

Socialization

Externalization

Knowledge Transformation Processes


Socialization– Tacit to tacit– Face to face meetings– Synchronous

Externalizaton– Tacit to explicit– Respond to question– List-server discussion– Asynchronous

Internalization– Explicit to tacit– Access to organized explicit knowledge– Structure helps user internalize knowledge

Combination– Explicit to explicit– Organization into categories– Reflects knowledge of domain

Research Questions

1. ExternalizationHow can we capture the tacit knowledge in such a discussion list and transform it into explicit knowledge?

2. CombinationHow can we organize this explicit knowledge?

3. InternalizationHow do we provide access to this explicit knowledge so that users can internalize this knowledge?

4. Linking Tacit Knowledge to Best EvidenceHow do we map this transformed tacit knowledge to the appropriate best evidence literature?

Mapping Tacit Knowledge to Explicit Knowledge in Medical Literature

PPMLThread

CreationData

CleaningPubMed Articles

Mes

h T

erm

ino

log

y M

ap

Thread Clusters

Externalization Combination LinkingInternalization

Access

Data Cleaning• Remove duplicate messages (subject & time stamp)

• Remove responses that were generated automatically by “vacation” mail programs

• Remove other “junk” e-mails

• Removing unnecessary content of the messages themselves. This unnecessary content included non-textual material such as images that would not be used in the clustering process and included original messages that were more than ten lines long as these would skew the clustering process.

• The initial stage of this cleaning was done manually until patterns were recognized and then programs were written to clean the data based on these patterns.

Externalization: Creating Threads

• Messages were threaded based on time stamps and subject headings.

• Those messages that had a blank subject field were processed based on the included original messages to which they had replied.

Thread Representation

• Each thread is treated as though it were a contiguous document

• The original messages that are embedded in the reply messages are removed.

• Stop words are removed• If not on the stop list, they are matched against a

synonym dictionary manually created by a pediatric pain specialist.

• The remaining terms are stemmed• The stemmed terms are assigned tf.idf weights

Data Set

• An archived sample of 6939 messages from 1993-1999

• After cleaning 4033 messages

• After threading 1289 threads

• Each thread is represented by a vector of 4111 term weights

term1 term2 term3 . . . term4111

thread1 w1,1 w1,2 . . . w1,4111

thread2 w2,1 w2,2 . . . w2,4111

.

.

.

thread1289 w1289,1 w1289,2 . . . w1289,4111

Thread-Term Matrix

Combination: Organizing the Threads

• Text clustering – unsupervised learning process – groups documents into clusters so that the documents

within a cluster have high similarity with one another, but are very dissimilar to the documents in the other clusters

• Text classification or categorization– supervised learning process– Assigns documents to pre-defined classes or

categories

k-means clustering with k=2

12

3

45

6 7


12

34 56 7


12 3

4

5

6

7

Evaluation of Clustering

• Performed a study in which 100 randomly selected threads were presented to two experts for clustering and to our clustering algorithm

• Results of clustering between the experts measured

• Results of clustering between the experts and the system measured

Cluster ID

Label Number of Threads

1 adverse effects of a treatment or medication, monitoring requirements 6

2 advice on medications or treatment technique for a particular CONDITION 33

3 announcement of a publication or event 9

4 assessment methods for a particular condition 2

5 availability and benefits of a nonstandard drug compound 3

6 availability and validation of a particular assessment tool 11

7 contact information or other information about a specific person 3

8 dosage or adverse effects or technique for a medication or other treatment 9

9 information on a condition: description, etiology, prognosis 4

10 job description, posting of job or fellowship 3

11 miscellaneous 3

12 other newsgroups and listservs 2

13 policies, guidelines, protocols, algorithms, quality assurance, supervision, competency

12

Clusters and labels created by expert 1 – a psychologist

Clusters and labels created by expert 2 – a medical doctor

Cluster ID Label Number of Threads

1 Assessment 10

2 Musculoskeletal 2

3 sedation & procedures 4

4 oral drugs 5

5 miscellaneous /irrelevant/out-of-date 21

6 neuropathic pain 9

7 regional analgesia 10

8 postoperative pain 5

9 intravenous opioids 11

10 psychology 5

11 visceral pain, bowel function, etc 2

12 topical analgesia, EMLA 2

13 everyday pain 1

14 NMDA antagonists, ketamine 1

15 resources 8

16 administration 3

17 burns 1

Inter-Rater Reliability

The Redundancy(X, Y) is the proportion of uncertainty about X that is removed by knowing Y

In this instance, X and Y represent the two sets of clusters generated by the experts.

The measure is asymmetrical and the calculated redundancy measures are:

R(Expert-1, Expert-2) = 0.51R(Expert-2, Expert-1) = 0.44

Evaluation of the Automatically Generated Clustering

• Assume each manually created cluster is correct

• Compare the manually created cluster against an automatically created cluster

• Recall – the proportion of those items in the manually created cluster that appear together in the same automatically generated cluster

• Precision – the proportion of those items in an automatically created cluster that appear together in the same manually created cluster

• F–measure = 2PR / (P+R)

Hierarchy – k=2

C-1,1

C-2,1

C-3,1 C-3,2 C-3,3 C-3,4

C-4,1 C-4,2

C-2,2

E-1

E-2

E-n

F-measure for a classification

The overall F-measure is used to reflect the quality of the whole hierarchy. The overall F-measure is the average weighted F-measure for all the clusters in a humanly generated clustering and is defined to be:

Overall F-measure = ∑ ( |T| * F(T)) / ∑ |T|

ST ST

TTFT ||/))(*|(|

ST ST

TTFT ||/))(*|(|

Evaluation of Clustering

• Each expert’s set of clusters was compared to the automatically generated hierarchical clustering. The hierarchy was generated ten times using different seed centroids for each run.

• The results of the paired-samples t tests (p=0.05) show that there was no significant difference between the two sets of manually generated clusters when used to evaluate the automatically generated clustering (k = 6).

E-1 E-2

k-means0.47 0.48

Evaluation of k-means Clustering

• We now have 3 different clusterings with inter-rater reliability of < .50

• k-means generated a large number of term representatives for each cluster with no elegant way of mapping the terms into MeSH.

• Therefore, the k-means clustering algorithm was replaced with a SOM in the expectation that the clustering results would be better and that a smaller set of term representatives for each cluster might be identified.

SOM – Self Organizing Maps

• Invented by Teuvo Kohonen • Provide a way of representing

multidimensional data in much lower dimensional spaces - usually one or two dimensions.

• Create a network that stores information in such a way that any topological relationships within the training set are maintained

Example 2-D Lattice of Nodes

Red = 240 Green = 89 Blue = 48

R G B

240 89 48

37 202 219

Mapping 3 Dimensional Colour Vectors Into 2 Dimensions

Notice that in addition to clustering the colours into distinct regions, regions of similar properties are usually found adjacent to each other.

SOM Neighbourhood Decreases

Mapping 3 Dimensional Colour Vectors Into 2 Dimensions

Notice that in addition to clustering the colours into distinct regions, regions of similar properties are usually found adjacent to each other.

term1 term2 term3 . . . term4111

thread1 w1,1 w1,2 . . . w1,4111

thread2 w2,1 w2,2 . . . w2,4111

.

.

.

thread1289 w1289,1 w1289,2 . . . w1289,4111

Thread-Term Matrix

Principal Component Analysis for Feature Length Reduction

PCA Vectors

Eig

en V

alu

es

SOM – Vector Length 150

Growing Hierarchical SOM

SOM Results

Method Features Map SizeNumber

Of Clusters

Best F-Measure(SOM)

Average F-Measure(k-means)

Expert1

Expert2

Expert1

Expert2

SOM 150 8*6 48 0.2968 0.4043

0.47 0.48GHSOM 5005 layers

2*253 0.4235 0.4466

SOM-k 150 10*5 13 0.2783 0.3896

Problems

PPMLThread

CreationData


Mes

h T

erm

ino

log

y M

ap

Thread Clusters


Mapping Tacit Knowledge to Explicit Knowledge in Medical Literature

PPMLThread

CreationData


Mes

h a

nd

UM

LS

Threads


Access

Combination: Organizing the Threads

• Text clustering – unsupervised learning process – groups documents into clusters so that the documents

within a cluster have high similarity with one another, but are very dissimilar to the documents in the other clusters

• Text classification or categorization– supervised learning process– Assigns documents to pre-defined classes or

categories

MetaMap Transfer (MMTx)• Discovers UMLS Metathesaurus concepts in text

• Text is parsed into components including sentences, paragraphs, phrases, lexical elements and tokens. Produces a shallow syntacitc analysis with part-of-speech tagging.

• Variants are generated from the resulting phrases. Includes acronyms, abbreviations and synonyms.

• Candidate concepts from the UMLS Metathesaurus are retrieved and evaluated against the phrases.

• The best of the candidates are organized into a final mapping in such a way as to best cover the text.

Metathesaurus CandidatesThe word "discharge" returns

Semantic Group: Anatomy Discharge, Body Substance (C0012621) - [Body Substance] Discharge, Body Substance, Sample (C0600083) - [Body

Substance]Semantic Group: Procedures Patient Discharge (C0030685) - [Health Care Activity]

from the UMLS Knowledge Server

Metathesaurus Candidates"He is to be discharged home..."

Phrase: "discharged"Meta Candidates (3) 966 C0030685:Discharge <1> (Patient Discharge) [Health Care Activity] 966 C0600083:Discharge <3> (Discharge, Body Substance, Sample)

[Body Substance] 966 C0012621:Discharge, NOS (Discharge, Body Substance) [Body

Substance]

Phrase: "home"Meta Candidates (3) 1000 C0442517:Home [Manufactured Object] 928 C0237154:homeless <1> (Homelessness) [Finding] 928 C0019863:homeless <2> (Homeless persons) [Population Group]

MMTx Scores

MeSH Concept Number

MeSH Concept Term

UMLS Semantic Type

Using the MMTx Results

• The MMTx results were used in three different ways:– Organize the PPML threads according to the

UMLS Semantic Groups -134 semantic types in 15 semantic groups

– Organize the PPML threads according to the MeSH Hierarchy – 15 MeSH trees

– Select terms that can be used as queries to PubMed

Organization by UMLS Semantic Group

Semantic Groups Semantic Types Terms Retained

Activities & Behaviors ACTI Activity, Behavior, Event, Machine Activity … NO

Anatomy ANAT Anatomical structure, Body location … YES

Chemicals & Drugs CHEM Amino Acid, Antibiotic, Chemical … YES

Concepts & Ideas CONC Classification, Concept Entity … NO

Devices DEVI Medical Device, Research Device … NO

Disorders DISO Acquired Abnormality, Disease … YES

Genes & Molecular Sequence GENEAmino Acid Sequence, Gene or Genome,

Molecular Sequence …NO

Geographic Areas GEOG Geographic Area NO

Living Beings LIVB Age group, Alga, Animal … NO (except age group)

Objects OBJC Entity, Food, Manufactured Object … NO

Occupations OCCU Biomedical Occupation … NO

Organization ORGA Organization, Professional Society … NO

Phenomena PHEN Biologic Function, Test Result … NO

Physiology PHYS Cell Function, Clinical Attribute … NO

Procedures PROC Diagnostic procedure … NO

Organization by MeSH Tree

• There are 15 MeSH trees

• It was determined to keep only two trees:– The C tree (Diseases) as the PPML largely

deals with disorders and diseases– The D tree (Chemicals and Drugs) as the

PPML contains discussions on drugs hence it was deemed important to retain drug-related terminology.

Filtering to Generate PubMed Queries

• Filtering approach operates at the semantic/conceptual level as opposed to the term level

• UMLS semantic types associated with each MeSH term are used as the basis for term filtering

• Working at the semantic level we can – Establish a medical context for the thread which can assist in

subsequent search for corresponding literature; – Characterize the entirety of medical terms into a small number of

medical concepts – Design filtering rules that apply to broad semantic types as opposed to

focused individual terms

Filtering UMLS Concepts Associated with MeSH Terms Found in Subject Line

If mapping score = 1000 then retain the MeSH term.

If semantic type = Age group (T100) then retain the MeSH term.

If semantic group = CHEM | DISO | ANAT AND (mapping score > 800) then retain the MeSH term.

If semantic type = Diagnostic Function (T060) | Therapeutic or Preventive Procedure (T060) | Laboratory or Test Result (T034) AND (mapping score > 800) then retain the MeSH term.

Generating a PubMed Query

Concept Name

ScoreSemantic

GroupSemantic Type Retain

Year 694 CONC Temporal Concept No

Old 861 CONC Temporal Concept No

Feline osteogenesisimperfecta

1000 DISO Disease or Syndrome Yes

Adolescent 694 LIVB Age Group Yes

Osteoporosis 861 DISO Disease or Syndrome Yes

Query Terms: Feline osteogenesis imperfecta, Adolescent, Osteoporosis

Summary

• We have various hierarchical organizations of the PPML threads that can be browsed by the user

• We have linked the PPML to the best-evidence literature via PubMed


Socialization– Tacit to tacit– Face to face meetings– Synchronous

Externalizaton– Tacit to explicit– Respond to question– List-server discussion– Asynchronous

Internalization– Explicit to tacit– Access to organized explicit knowledge– Structure helps user internalize knowledge

Combination– Explicit to explicit– Organization into categories– Reflects knowledge of domain

Future Research

• Improve the filters

• Link from medical literature to PPML

• Evaluate the overall system with respect to the users:– Is it useful?– Is it helpful?– Does it improve outcomes?

Thank You

Web Information Filtering Lab

http://www.cs.dal.ca/wifl/

Closeness of Document Vectors

information science

Doc0 0 0

Doc1 0 1

Doc2 1 0

Doc3 1 1

(0,0)

science

information

θ cos θ

cos 0o = 1

cos 90o = 0Doc2

Doc1

(1,1)

Doc3

Accessing Tacit Knowledge and Linking it to the Peer-Reviewed Literature

Documents

Transcript of Accessing Tacit Knowledge and Linking it to the Peer-Reviewed Literature