Slides (.ppt)

53
06/11/22 Automated Retrieval and Automated Retrieval and Generation of Brain CT Radiology Generation of Brain CT Radiology Reports Reports Gong Tianxia SOC NUS

Transcript of Slides (.ppt)

Page 1: Slides (.ppt)

04/12/23

Automated Retrieval and Generation of Automated Retrieval and Generation of Brain CT Radiology ReportsBrain CT Radiology Reports

Gong Tianxia

SOC NUS

Page 2: Slides (.ppt)

04/12/23

Outline

Background Motivation Research Work Conclusion

Page 3: Slides (.ppt)

04/12/23

Background

Computer Tomography (CT) has been used to examine the abnormality of human brain due to various causes

The result of each brain CT examination consists of: A set of CT scan image A report written by a radiologist

Page 4: Slides (.ppt)

04/12/23

Abnormalities

Head traumas epidural hemorrhage(EDH) acute subdural hemorrhage (SDH_Acute) chronic subdural hemorrhage (SDH_Chronic) intracerebral hemorrhage (ICH) intraventricular hemorrhage (IVH) subarachnoid hemorrhage (SAH)

Fractures Edemas Others

Midline shift Etc.

Page 5: Slides (.ppt)

04/12/23

Background

Brain CT Scans Samples

Normal EDH

Page 6: Slides (.ppt)

04/12/23

Background

Brain CT Scans Samples

ICHSDH_Acute, SDH_Chronic,

Midline Shift

Page 7: Slides (.ppt)

04/12/23

Background

Report

Unenhanced axial CT head was obtained. No previous study is available for comparison.

There is acute subdural haemorrhage overlying the left convexity & midline falx, which

measures up to a maximum of 1.4 cm in thickness. Subarachnoid haemorrhage is seen in

the sulci at the left fronto-temporal lobe, bilateral Sylvian fissure & cistern and the

basal cistern. Intraventricular extension of haemorrhage with blood seen in all four

ventricles is noted. There is intraparenchymal haemorrhage in the bilateral frontal lobes

raising the suspicion of haemorrhagic contusion. There is considerable mass effect with

midline shift to the right, generalised effacement of cerebral sulci and compression of

the left lateral ventricle. Prominence of the right temporal horn is suspicious for a

hydrocephalus. No skull vault fracture is seen in the CT scan.

Page 8: Slides (.ppt)

04/12/23

Background

Comments

Acute left fronto-temporal-parietal subdural haematoma with bifrontal parenchymal

haematoma and bilateral subarachnoid haemorrhage with intraventricular extension.

Associated mass effect with midline shift to the right, compression of the left lateral

ventricle and generalised effacement of cerebral sulci. Hydrocephalus with right

ventricle dilated.

Page 9: Slides (.ppt)

04/12/23

Motivation

Radiology reports contain rich information which is not used in many medical database systems

The proposed system is aimed to: Provide convenient search functions for radiology reports

and images Help doctors, radiologists, and medical informaticians to

gather needed information for their research Give references to radiologists to compare results Facilitate education systems for researchers, junior

doctors, and medical students Integrate medical records from various sources Provide platform for medical community to exchange

information and knowledge

Page 10: Slides (.ppt)

04/12/23

Automated Retrieval and Generation of Brain CT Radiology Reports

Content-based Retrieval of CT Scan Brain Images

Two Research Directions

Page 11: Slides (.ppt)

04/12/23

Information Extraction from Radiology Reports

Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval

Related Work

Page 12: Slides (.ppt)

04/12/23

Information Extraction from Radiology Reports

Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval

Research Work

Page 13: Slides (.ppt)

04/12/23

MedLEE: Medical Language Extraction and Encoding System

RADA: RADiology Analysis Tool Statistical Natural Language Processor for

Medical Reports

Related Work

Page 14: Slides (.ppt)

04/12/23

MedLEE: Medical Language Extraction and Encoding System

Page 15: Slides (.ppt)

04/12/23

RADA: Radiology Analysis Tool

Page 16: Slides (.ppt)

04/12/23

Statistical Natural Language Processor for Medical Reports

Page 17: Slides (.ppt)

04/12/23

An example of structured representation output

Statistical Natural Language Processor for Medical Reports

Page 18: Slides (.ppt)

04/12/23

Negations Insufficient understanding of the text Ungrammatical writing styles Large vocabulary Assumed knowledge between writer

and reader

Challenges

Page 19: Slides (.ppt)

04/12/23

Information Extraction from Radiology Reports

Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval

Related Work

Page 20: Slides (.ppt)

04/12/23

Most existing medical report automatic generation systems use the following “template filling” approaches:

Structured Data Entry Mail Merge Canned Text

Automatic Generation of Medical Reports

Page 21: Slides (.ppt)

04/12/23

NLG: NLG is still premature application of medical

document generation There is still no system based on NLG principles in

routine use generates medical reports with fluent, concise and readable text

Challenges of NLG in general domain also exist in medical domain

Systems that automatically generate medical report from medical images are still lacking.

Challenges

Page 22: Slides (.ppt)

04/12/23

Information Extraction from Radiology Reports

Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval

Related Work

Page 23: Slides (.ppt)

04/12/23

NeuRadIR: Web-Based Neuroradiological Information Retrieval System

Information Retrieval on MR Brain Images and Radiology Reports

Free Text Assisted Medical Image Retrieval

Page 24: Slides (.ppt)

04/12/23

NeuRadIR

Page 25: Slides (.ppt)

04/12/23

MRI Brain Image and Report Retrieval

Page 26: Slides (.ppt)

04/12/23

Complexity of the system, as the system Consists of many functional components Needs knowledge from various research

areas

Challenges

Page 27: Slides (.ppt)

04/12/23

Information Extraction from Brain CT Radiology Reports

Automatic Generation of Brain CT Radiology Reports

Radiology Reports Assisted Brain CT Images Retrieval

Research Areas

Page 28: Slides (.ppt)

04/12/23

Information Extraction from Brain CT Radiology Reports

Automatic Generation of Brain CT Radiology Reports

Radiology Reports Assisted Brain CT Images Retrieval

Research Areas

Page 29: Slides (.ppt)

04/12/23

Information Extraction from Brain CT Radiology Information Extraction from Brain CT Radiology ReportsReports

Our major task in this research area is to extract structured medical findings from the free text brain CT radiology reports

Page 30: Slides (.ppt)

04/12/23

Input & OutputInput & Output

Input example:

An extra-dural haematoma overlying the right frontal lobe is seen measuring 1.2 cm in thickness.

Finding haematoma

type extradural

location overlying

brain_part lobe

orientation right

orientation frontal

thickness 1.2 cm

Output example:

Page 31: Slides (.ppt)

04/12/23

System ArchitectureSystem Architecture

The system will have these componentsDocument ChunkerParserTerm MapperFinding ExtractorReport Constructor

Page 32: Slides (.ppt)

04/12/23

Document ChunkerDocument Chunker

Decompose the radiology report into three sections Reasons for examination Detailed description of observations and

findings Comments or conclusion

We will focus on second and third sections, as they contain medical findings

Page 33: Slides (.ppt)

04/12/23

ParserParser

Parse each sentence of a report and outputs a typed dependence tree

Parser output example:null:seen

nsubjpass:hematomadet:Anamod:extra-duralpartmod:overlying

dobj:lobedet:theamod:rightamod:frontal

auxpass:ispartmod:measuring

dobj:cmnum:1.2prep-in:thickness

Grammatical relation to parent word

Page 34: Slides (.ppt)

04/12/23

Term MapperTerm Mapper

Maps words to standard forms specified in our medical knowledge source (Unified Medical Language System UMLS and other radiology thesaurus)

Reduces spelling variations

Page 35: Slides (.ppt)

04/12/23

Finding ExtractorFinding Extractor

Apply semantic rules that are derived from semantic features of the words to translate the typed dependency relationship to logical relationship between findings and modifiers (finding’s attributes)

Merge the same finding from different sentences into one finding

Remove the redundant finding

Page 36: Slides (.ppt)

04/12/23

Report ConstructorReport Constructor

Construct structured report according to findings, modifiers, and their logical relationship extracted from the finding extractor

Page 37: Slides (.ppt)

04/12/23

Research AreasResearch Areas

Information Extraction from Brain CT Radiology Reports

Automatic Generation of Brain CT Radiology Reports

Radiology Reports Assisted Brain CT Images Retrieval

Page 38: Slides (.ppt)

04/12/23

Automatic Generation of Brain CT Automatic Generation of Brain CT Radiology ReportsRadiology Reports

A traditional approach based on typical NLG system Content determination Discourse planning Sentence aggregation Lexicalization Referring expression

generation Linguistic realization

Page 39: Slides (.ppt)

04/12/23

Content DeterminationContent Determination

Creates a set of messages from the features extracted from the new brain CT Images

Doctors use size, shape and location of the potential hemorrhage region to determine head trauma types

The system uses similar features for content determination: area, major axis length, minor axis length, eccentricity, solidity, extent, adjacency to skull, adjacency to background

Page 40: Slides (.ppt)

04/12/23

Content DeterminationContent Determination

Image Segmentation

Features Extraction

Page 41: Slides (.ppt)

04/12/23

Discourse PlanningDiscourse Planning

Uses Rhetorical Structure Theory (RST) to organize the text based on relationships that hold between parts of the text

Page 42: Slides (.ppt)

04/12/23

Sentence AggregationSentence Aggregation

Groups messages together into sentences and paragraphs

Page 43: Slides (.ppt)

04/12/23

Sentence AggregationSentence Aggregation

Groups messages together into sentences and paragraphs

Page 44: Slides (.ppt)

04/12/23

LexicalizationLexicalization

• Decides which specific words and phrases should be chosen to express the domain concepts and relations which appear in the messages

• Uses hardcoded specific word and phrases to standardize the output language radiology reporting

• Uses NLG system to generate radiology reports of various writing styles to cater different user groups (at later stage of our project)

Page 45: Slides (.ppt)

04/12/23

Final StepsFinal Steps

Referring Expression Generation Linguistic Realization

Page 46: Slides (.ppt)

04/12/23

A Machine Learning ApproachA Machine Learning Approach

Based on the concept of statistical machine translation

Image and report are two representations of the same medical condition

In a sense, image and text are two different languages

Page 47: Slides (.ppt)

04/12/23

Statistical Machine TranslationStatistical Machine Translation

Foreign/English

parallel text

Englis

h text

Statistical analysis Statistical analysis

Translation Model Language Model

Decoding Algorithm

Page 48: Slides (.ppt)

04/12/23

Syntax Tree Based SMTSyntax Tree Based SMT

IP

VP

BA

NN VP

W PN

把 钢笔 给 我

Give the pen to me

DT NN TO PRP

NP PPVB

VP

Page 49: Slides (.ppt)

04/12/23

Report Generation based on Report Generation based on SMT conceptsSMT concepts

Page 50: Slides (.ppt)

04/12/23

Research AreasResearch Areas

Information Extraction from Brain CT Radiology Reports

Automatic Generation of Brain CT Radiology Reports

Radiology Reports Assisted Brain CT Images Retrieval

Page 51: Slides (.ppt)

04/12/23

Radiology Reports Assisted Brain CT Radiology Reports Assisted Brain CT Images RetrievalImages Retrieval

Page 52: Slides (.ppt)

04/12/23

Radiology Reports Assisted Brain CT Radiology Reports Assisted Brain CT Images RetrievalImages Retrieval

Page 53: Slides (.ppt)

04/12/23

Project StatusProject Status

Project Funding Sources University Research Grant Ministry of Education Academic Research Grant

Project Collaborators School of Computing, NUS National Neuroscience Institute Institute for Infocomm Research

Project Phases Phase I: Pilot Study (Feb 2007 – April 2008) Phase II: R&D (April 2008 – Mar 2011)