1

A Patient Care Digital Library

Personalized Search and Summarization over Multimedia Information

A Patient Care Digital Library

Personalized Search and Summarization over Multimedia Information

Kathleen McKeown, Shih-Fu Chang, James Kathleen McKeown, Shih-Fu Chang, James Cimino, Cimino, Paul ClaytonPaul Clayton, Judith Klavans, Judith Klavans

Columbia UniversityColumbia University

2

VisionVision

Personalized access to distributed resourcesPersonalized access to distributed resources information vetting information vetting information fusioninformation fusion information understandinginformation understanding

Provision of patient-specific informationProvision of patient-specific information at the point of patient careat the point of patient care in terms that can be understoodin terms that can be understood

Benefits: awareness of new procedures, Benefits: awareness of new procedures, increased complianceincreased compliance

3

FeaturesFeatures

Tailor search, presentation, and Tailor search, presentation, and summarization of online medical literaturesummarization of online medical literature

Exploit online patient record as user modelExploit online patient record as user model Patients and health care providersPatients and health care providers Multimodal access and presentation over Multimodal access and presentation over

multimedia resourcesmultimedia resources

4

Scope of ResearchScope of Research

““exciting,” “innovative,”exciting,” “innovative,”“broad,” “ambitious,” “complex”“broad,” “ambitious,” “complex”

Computer Science and Medical InformaticsComputer Science and Medical Informatics Links into libraries: CRIALinks into libraries: CRIA Suite of existing component technologiesSuite of existing component technologies Five year visionFive year vision Active partnersActive partners

5

ContinuumContinuum

Current: existing individual components for Current: existing individual components for multiple applicationsmultiple applications

summarization of terrorist news, image search, summarization of terrorist news, image search, categorization of images, multimedia briefings, PATCIS, categorization of images, multimedia briefings, PATCIS, patient mammography record extraction, video patient mammography record extraction, video summarization of news, query translationsummarization of news, query translation

Future: DLI2 enablesFuture: DLI2 enables unification and transformation of componentsunification and transformation of components new level of capabilitynew level of capability integrated in a single application integrated in a single application

6

OutlineOutline

Desmond JordanDesmond Jordan Why this research is neededWhy this research is needed

Kathy McKeownKathy McKeown An example—integration of components An example—integration of components Scope, organization, managementScope, organization, management Role of partners and new developmentsRole of partners and new developments

Jim CiminoJim Cimino Clinical information infrastructureClinical information infrastructure

Kathy McKeownKathy McKeown BudgetBudget

7

RoundsRounds

Patient-centricPatient-centric Current: Access Current: Access

to clinical datato clinical data Missing: Missing:

Access to Access to literature that literature that fits patient fits patient profileprofile

8

Data follows the patient—why not literature?Data follows the patient—why not literature?

9

Ongoing Collaboration: Generation of Multimedia BriefingsOngoing Collaboration: Generation of Multimedia Briefings

Voice: . . . . Ms. Jones is an 80 year old, diabetic, hypertensive, female patient of Dr. Smith undergoing CABG. . . .

10

User Study GroupUser Study Group

Speech, text, and Speech, text, and graphics output graphics output were usefulwere useful

11

Multimedia VideoMultimedia Video

Physicians: residents learn by comparisonPhysicians: residents learn by comparison

Patients:Patients: Video helps them understand their diagnosisVideo helps them understand their diagnosis They like to investigate prognosisThey like to investigate prognosis

Steve Feiner:

If I recall correctly, I found the image at the bottom in a we page for patients with CHF. May have something to do with fluid buildup.

Steve Feiner:

If I recall correctly, I found the image at the bottom in a we page for patients with CHF. May have something to do with fluid buildup.

Why should I prescribe amiodarone?

Query

History

WebSeek

Lexical Navigation

Steve Feiner:

This slide differs from the previous one in that it has a real graphical history, although not in the medical domain. It may be better, since it means that all of the material on the slide will be real. BTW, this is a “comment”. You can either delete it or make it invisible with the “View” menu “comments” command

Steve Feiner:

This slide differs from the previous one in that it has a real graphical history, although not in the medical domain. It may be better, since it means that all of the material on the slide will be real. BTW, this is a “comment”. You can either delete it or make it invisible with the “View” menu “comments” command

14

Key Issues: User InterfaceKey Issues: User Interface

Query augmentation using the patient Query augmentation using the patient recordrecord

Congestive Heart Failure (CHF), male, Base Heart Congestive Heart Failure (CHF), male, Base Heart Rate (BHR) > 90, history of ischemiaRate (BHR) > 90, history of ischemia

Lucent: speech, IBM: query refinement, Lucent: speech, IBM: query refinement, interface for image/video searchinterface for image/video search

b

Treatme CHFTreatme CHF

Amiodarone

Amiodarone

Amiodarone

Documents

Echocardiogram

Echo 7/15/98

Echo 8/15/98

Echo 9/15/98

Echo 10/15/98

SearchSearchAugmented query

Patientheart

sounds

AmiodaroneCHFmale

BHR > 90

NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section ofthe proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entryof information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS

NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section ofthe proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-en the normal mechanism for project reporting to NS

Antiarrhythmic Drugs

Database of Videos

Distributed Search

profile profile profile

Journal Articles

Categorization&

Filtering

Categorization&

Filtering

Methods:All patients hadadvanced CHF with ...

AmiodaroneCHFmale

BHR > 90

DiagnosisArticle

.

.

.

NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section ofthe proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entryof information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS

NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section ofthe proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entryof information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS

NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section ofthe proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entryof information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS

Amiodarone

Amiodarone

Amiodarone

Documents

NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section ofthe proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entryof information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS

Amiodarone

PrognosisArticle

PrognosisArticle

Target SummariesTarget Summaries

Prognosis articles:Prognosis articles:

amiodarone patient, BHR > 90amiodarone patient, BHR > 90

A significant reduction in sudden death was observed in A significant reduction in sudden death was observed in patients with a BHR > 90 beats/min treated with patients with a BHR > 90 beats/min treated with amiodaroneamiodarone

pacemaker patient, BHR pacemaker patient, BHR 90 90

In patients with a BHR In patients with a BHR 90 beats/min, amiodarone- 90 beats/min, amiodarone-treated patients had a mortality rate similar to those not treated patients had a mortality rate similar to those not treated with amiodarone. One article found that pacing treated with amiodarone. One article found that pacing fails to improve hemodynamic function, while another fails to improve hemodynamic function, while another showed a significantly better quality of life.showed a significantly better quality of life.

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeownK. McKeown

J. Klavans

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

L. Gravano

S. Johnson

J. Cimino

S.-F. Chang

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

S. Johnson

S. Feiner

S.-F. Chang

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

K. McKeown

J. Klavans

V. Hatzivassiloglou

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

J. Cimino

D. Jordan

C. Friedman

S. Johnson

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

J. Cimino

D. Jordan

V. Patel

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

P. Molholt

S. Jacobson

J. Klavans

E. La Rue

PERSIVAL Management Structure

Industry Partners

Klavans

Clinical Infrastructure

Cimino

Content and Library Team Klavans

Evaluation

Patel

Project Leader

McKeown

Technology Search

Gravano

User Interface

Johnson

Summarization

McKeown

32

Role of Technology PartnersRole of Technology Partners

Lucent: Lucent: Spoken Language Dialogue Spoken Language Dialogue Systems (speaker independent)Systems (speaker independent)

IBM:IBM: Query refinement, evaluation, text Query refinement, evaluation, text summarization, interface for image/video summarization, interface for image/video searchsearch

GEGE: Concept based retrieval: Concept based retrieval

33

New Partners and ChangesNew Partners and Changes

Siemens Corporate Research: grant for video Siemens Corporate Research: grant for video summarization integrating text/image, plus researcher timesummarization integrating text/image, plus researcher time

$68,680/year$68,680/year AT&T: grant to build our networking infrastructureAT&T: grant to build our networking infrastructure

$50,000$50,000 Welch Allyn: information and equipment, Medical Welch Allyn: information and equipment, Medical

Director’s time, cashDirector’s time, cash $35,000$35,000

GE Medical Systems: commitment to commercialization GE Medical Systems: commitment to commercialization of resultsof results

34

PartnersPartnersPartner Contribution ValueGeneral Electric Software

Researcher$200,000 $15,000/year

TeamCompanies

IBM SoftwareResearcher

$50,000$110,000/year

Lucent SoftwareResearcherStudent Support

$25,000$125,000/year$201,286 over 5 years

Siemens Grant $25,000/yearResearcher $34,000/yearStudent Support $9,680/year

Libraries &Museums

New York PublicLibrary

Consumer HealthInformation Specialists

MOMA Grant, Image ProcessingTechnology

$10,000

Donating Bell Atlantic Authentication Services $115,000Companies Intel Computer Equipment $209,154 over 5 years

AT&T Grant, NetworkInfrastructure

$50,000

Steve Feiner:

IBM is more than just a software company; ditto Lucent, GE, etc. Why not just group companies together?

Steve Feiner:

IBM is more than just a software company; ditto Lucent, GE, etc. Why not just group companies together?

35

PartnersPartners

Partner Contribution ValueHealthwise Consumer Health &

Software $10,000/year

ContentProviders

Lion Reef Images & Animation ofHeart

$30,000

OVID Technology Scientific Biomedical Data& Software

$15,000/year

Univ. of CA -San Francisco

Medical Images

Welch Allyn Information andequipmentMedical Director’s timeUnrestricted funds

$20,000

$10,000 $5,000

36

Clinical Information SystemsClinical Information Systems

Clinical repository of coded and text dataClinical repository of coded and text data Medical Entities Dictionary serverMedical Entities Dictionary server Web-based access for clinicians (WebCIS)Web-based access for clinicians (WebCIS) Web-based access for patient (PatCIS)Web-based access for patient (PatCIS) Infrastructure handles security, application Infrastructure handles security, application

tracking, and evaluationtracking, and evaluation Architecture handles integration of Architecture handles integration of

componentscomponents

37

Input for PERSIVALInput for PERSIVAL

Coded dataCoded data Natural language processingNatural language processing Medical Entities DictionaryMedical Entities Dictionary UMLSUMLS

42

Web/PatCIS and Digital LibraryWeb/PatCIS and Digital Library

WebCIS/PatCIS

InfobuttonInformation

Resource

CIS MED

43

Web/PatCIS and Digital LibraryWeb/PatCIS and Digital Library

WebCIS/PatCIS

InfobuttonInformation

Resource

QueryInterface

PERSIVAL

CIS MED

44

UsersUsers

Clinical usersClinical users Columbia-Presbyterian Medical CenterColumbia-Presbyterian Medical Center New York HospitalNew York Hospital 37 affiliates37 affiliates

PatientsPatients Heart failureHeart failure Home diabetes telemonitoringHome diabetes telemonitoring High risk for heart attackHigh risk for heart attack Cancer patientsCancer patients

45

Evaluation and User-Centered DesignEvaluation and User-Centered Design

Usability engineeringUsability engineering in situ and lab video recordingsin situ and lab video recordings

Evaluation of system components: Evaluation of system components: cognitive and quantitativecognitive and quantitative

Subjective assessment of search plus recall, Subjective assessment of search plus recall, precisionprecision

SurvivabilitySurvivability end-to-end evaluation, reduction of manual end-to-end evaluation, reduction of manual

intervention, training, university support of intervention, training, university support of applicationsapplications

46

SummarySummary

Tailor search, presentation, and Tailor search, presentation, and summarization of online medical literaturesummarization of online medical literature

Exploit online patient record as user modelExploit online patient record as user model Patients and health care providersPatients and health care providers Multimodal access and presentation over Multimodal access and presentation over

multimedia resourcesmultimedia resources