Natural Language Tools and Resources for Biomedical Information Extraction Yoshimasa Tsuruoka Tsujii...
-
Upload
mia-burton -
Category
Documents
-
view
217 -
download
0
Transcript of Natural Language Tools and Resources for Biomedical Information Extraction Yoshimasa Tsuruoka Tsujii...
Natural Language Tools and Resources for Biomedical Information Extraction
Yoshimasa Tsuruoka
Tsujii laboratory
University of Tokyo
Outline
• NLP resources for bioNLP– GENIA corpus
• NLP tools– Machine learning
• Maximum entropy modeling for feature forest• Maximum entropy modeling with inequality constraints
– Part-of-speech tagger– Chunker (shallow parser)– HPSG Parser
• Applications of NLP– Extracting disease-gene relationships from MEDLINE abstracts
Application of NLP to the Biomedical domain
• Plenty of text– MEDLINE database: 12 million abstracts – Needs of effective IE and IR
• Domain knowledge– Gene ontology, KEGG, UMLS, ICD, …
• Other Information sources– Molecular databases
• DNA sequences, motifs, diseases, molecular interactions, etc…
Developing NLP resources
• Resources for NLP research– Domain knowledge– Training data for ML-based techniques– Test data for evaluating the transferability of a system
• GENIA resources– Ontology– Corpus
GENIA corpus
• 4,000 MEDLINE abstracts– Selected by MeSH Terms (Human, Blood cells, Transcription
factors)
• XML format• Contents
– Named-entity (Kim et al 2003)– Part-of-speech (Tateisi et al 2004)– Parse tree– Co-reference (Institute of Infocomm Research, Singapore)
GENIA part-of-speech corpus
• Each token is annotated with its part-of-speech tag.• Size
– 2,000 abstracts
– 20,544 sentences
– 50,1054 words (about half the size of Penn Treebank)
The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes …
DT NN NN NN VBZ JJ NN
NN NN CD NN NN IN NNS
The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes …
GENIA named-entity corpus
• Terms are annotated based on the semantic classes in the GENIA ontology
• Size– 2,000 abstracts– Number of the terms: 92,723– Vocabulary size: 36,568
DNA virus
cell_type
GENIA treebank
• Based on the standard of the Penn TreeBank• Size
– 500 abstracts– (1500 abstracts by the end of this summer)
CD3-episilon expression is controlled by a downstream T lymphocyte-specific enhancer element
NP ADJP
NP
PP
VP
VP
S
Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA).
T cell
IL-2 Interferon-gamma
IL-8 familyIL-2
IL-8
IFN-γ
Ca(2+)-iPMA
Ca(2+)-iPMA
Ca(2+)-iPMA
CyA×
CyA×
CyA×
: Target: Interaction: Agent
: Location
Event AnnotationFew known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA).
Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA).
T cell
IL-2 Interferon-gamma
IL-8 familyIL-2
IL-8
IFN-γ
Ca(2+)-iPMA
Ca(2+)-iPMA
Ca(2+)-iPMA
: Target: Interaction: Agent
: Location
Event annotation
Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA).
T cell
IL-2 Interferon-gamma
IL-8 familyIL-2
IL-8
IFN-γCyA×
CyA×
CyA×
: Target: Interaction: Agent
: Location
Event annotation
GENIA corpus
• Used in more than 240 institutions– Japan (28), Asia (54), North America (63), Europe (62),
etc…• De facto standard for evaluating biomedical named-
entity recognition systems– BioNLP workshop at Coling 2004
• Named-entity recognition shared task– Institute for Infocomm Research (Singapore),– Stanford University (USA),– University of Edinburgh (UK),– University of Wisconsin-Madison (USA),– Pohang University of Science and Technology (Korea),– University of Alberta (Canada),– University Duisburg-Essen (Germany),– Korea University (Korea),– National Taiwan University (Taiwan),
NLP tools
• Biomedical text mining– Huge amount of text.
• Machine learning– Training set can be very large.– Efficient training algorithms.
• Taggers (and parsers)– Decoding should be fast.
Machine learning
• Supervised learning– learns the rules for classifying samples into
predefined classes by seeing a large number of training samples with class labels.
• Algorithms– Naïve Bayes, Decision Tree, SVMs, AdaBoost,
Perceptron, Random forests, Maximum Entropy, etc...
Maximum entropy learning
• Log-linear modeling
• Maximum likelihood estimation– determines the parameters so that they maximizes
the likelihood of the training data
F
iii xf
Zxq
1
exp1
Feature functionFeature weight
Maximum entropy modeling with inequality constraints
(Kazama and Tsujii 2003)
• Advantages over the standard ME modeling.– Good regularization effects (as good as Gaussian prior)– Sparse solution
• C++ implementation– offers fast training.– can be used as a library.– can incorporate the model into your source code.
• The C++ library is used in many NLP programs (e.g. POS tagger, chunkers, IE modules)
Part-of-speech tagging
• A PoS tagger annotates each token with its part-of-speech tag.
The peri-kappa B site mediates human immunodeficiency DT NN NN NN VBZ JJ NNvirus type 2 enhancer activation in monocytes … NN NN CD NN NN IN NNS
Chunking (shallow parsing)
• A chunker (shallow parser) segments a sentence into non-recursive phrases.
He reckons the current account deficit will narrow toNP VP NP VP PPonly # 1.8 billion in September . NP PP NP
Chunking (shallow parsing)
• Chunking tasks can be converted into a standard tagging task.
He reckons the current account deficit will narrow toBNP BVP BNP INP INP INP BVP IVP BPP
only # 1.8 billion in September . BNP INPINP INP BPP BNP
Sequential Classification Approaches
• Sequence tagging tasks– Find the tag sequence that maximizes the following probability given
the observation (e.g. words):
• Left to right decomposition (with the first-order markov assumption)
• Right to left decomposition (with the first-order markov assumption)
ottP n |...1
n
iiin ottPottP
111 ||...
n
iiin ottPottP
111 ||...
classification problem
Bidirectional Inference
• Possible decomposition structures
• Bidirectional inference algorithm (Tsuruoka et al.)– We can find the “best” structure and tag sequences in
polynomial time
t1 t2 t3(a) t1 t2 t3(b)
t1 t2 t3(c) t1 t2 t3(d)
State-of-the-art PoS taggers
• Tagging speed and accuracy on Penn Treebank
Tagging Speed Accuracy
Dependency Net (2003) Very slow 97.24
Perceptron (2002) ? 97.11
SVM (2003) Fast 97.05
HMM (2000) Extremely fast 96.48
Bidirectional MEMM Very fast 97.10
State-of-the-art Chunkers
• Chunking speed an accuracy on Penn Treebank
Tagging Speed Accuracy
Perceptron (2003) ? 93.74
SVM + voting (2003) Slow? 93.91
SVM (2000) Fast 93.48
Bidirectional MEMM Very fast 93.70
The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes …
Named-entity recognition
• Recognizing named-entities in text• Similar to chunking
– IOB tagging• Named entities in the biomedical domain are long.
– Sliding window
DNA virus
cell_type
A sliding window approach to biomedical NE recogition
• We want to use rich features on a “term”.
• Enumerate all sub-word sequences in a sentence.
• Classify them into semantic classes.
W1 W2 W3 W4
Accuracy of biomedical NE recognition
Recall Precision F-score
SVM+HMM (Zho 2004) 76.0 69.4 72.6
Sliding window 71.5 70.2 70.8
MEMM (Fin 2004) 71.6 68.6 70.1
CRF (Set 2004) 70.3 69.3 69.8
• Shared task at Coling 2004 BioNLP workshop
HPSG parsing
• HPSG– A few schema– Many lexical entries– Deep syntactic analysis
• Grammar– Corpus-based grammar
construction (Miyao et al 2004)
• Parser– Beam search (Tsuruoka
et al.)
Lexical entryLexical entry
HEAD: verbSUBJ: <>COMPS: <>
Mary walked slowly
HEAD: nounSUBJ: <>COMPS: <>
HEAD: verbSUBJ: <noun>COMPS: <>
HEAD: advMOD: verb
HEAD: verbSUBJ: <noun>COMPS: <>
Subject-head schema
Head-modifier schema
Phrase structure
The company is run by him
DT NN VBZ VBN IN PRP
dt np vp vp pp np
np pp
vp
vp
s
Predicate-argument structure
The company is run by him
DT NN VBZ VBN IN PRP
dt np vp vp pp np
np pp
vp
vp
s
arg1arg2mod
IR search engine using predicate-argument structures
• A maximum entropy model is defined for the entire tree structure– e.g. HPSG parse trees
• Exponentially-many trees are represented with a packed forest of polynomial size
• A probability of each tree is estimated without unpacking the feature forest
Feature forest model (Miyao and Tsujii 2002)
S
NP1
mn
NP2 VP1VP2
number of trees: size:
feature forest
nm
Automatic Generation of Spelling Variants
• Variant GeneratorNF-Kappa B (1.0)NF Kappa B (0.9)NF kappa B(0.6)NF kappaB (0.5)NFkappaB (0.3)
:
GeneratorNF-Kappa B
Each generated variant is associated with its generation probability
Generation Algorithm
T cell (1.0)
T-cell (0.5) T cells (0.2)
T-cells (0.1)
0.5
0.2
0.2
• Recursive generation
P = P’ x Pop
Learning Operation Rules
• Operations for generating variants– Substitution
– Deletion
– Insertion
• Context– Character-level context: preceding (following) two
characters
• Operation Probability
contextf
operationcontextfcontextoperationP
,
Example of variant generation (1)
Generation Probability
Generated Variants Frequency
1.0 (input) antiinflammatory effect 7
0.462 anti-inflammatory effect 33
0.393 antiinflammatory effects 6
0.356 Antiinflammatory effect 0
0.286 antiinflammatory-effect 0
0.181 anti-inflammatory effects 23
: : :
Example of variant generation (2)
Generation Probabilitiy
Generated Variants Frequency
1.0 (Input) tumour necrosis factor alpha 15
0.492 tumor necrosis factor alpha 126
0.356 tumour necrosis factor-alpha 30
0.235 Tumour necrosis factor alpha 2
0.175 tumor necrosis factor alpha 182
0.115 Tumor necrosis factor alpha 8
: : :
Domain Adaptation
• Newspaper articles are widely used as training data for machine learning-based NLP systems.
• Domain Adaptability– Part-of-speech tagging– HPSG parsing
Tagging errors by TnT tagger (Brants 2000)
… and membrane potential after mitogen binding. CC NN NN IN NN JJ… two factors, which bind to the same kappa B enhancers… CD NNS WDT NN TO DT JJ NN NN NNS … by analysing the Ag amino acid sequence. IN VBG DT VBG JJ NN NN… to contain more T-cell determinants than … TO VB RBR JJ NNS IN Stimulation of interferon beta gene transcription in vitro by NN IN JJ JJ NN NN IN NN IN
Accuracy of TnT tagger on the GENIA corpus
• Ignoring unessential errors
Accuracy
TnT (original) 84.4%
NNP = NN, NNPS = NNS 90.0%
LS = NN 91.3%
JJ = NN 94.9%
About 94% in practice
GENIA tagger
training WSJ GENIA
WSJ 97.0 84.3
GENIA 75.2 98.1
WSJ+GENIA 96.9 98.1
•An MEMM tagger trained on WSJ and GENIA corpus
The tagger works well on both types of texts.
Parsing MEDLINE with the HPSG parser
• Parsing accuracy on the GENIA Treebank
#sentences LP / LR UP / UR
All sentences 1,556 82.8 / 81.5 86.4 / 85.1
Covered sentences 1,104 86.8 / 86.5 88.7 / 88.4
Extracting Disease-Gene Associations from MEDLINE abstracts
These results suggested that targeted disruption of Cyp19 caused anovulation and precocious depletion of ovarian follicles
Furthermore, AML cells with methylated p15(INAK4B) tended to express higher levels of DNMT1 and 3B.
Text
• 1.5 million MEDLINE abstracts– Selected by MeSH Terms
• “Disease Category” AND (“Amino Acids, Peptides, and Proteins” OR “Genetic Structures”)
• Parsing– All the sentences were parsed by the HPSG
parser– Using a PC cluster (100 processors with GXP)– Time: 10 days
Training data
All foals with OLWS were homozygous for the Ile118Lys EDNRB mutation, and adults that were homozygous were not found.
Dominant radial drusen and Arg345Trp EFEMP1 mutation.
The 5 year overall survival (OS) and event-free survival (EFS) were 94 and 90 +/- 8%, respectively, with a median follow-up of 48 months.
These data may indicate that formation of parathyroid adenoma in young patients is related to a mechanism involving EGFR.
:
• All co-occurrences are classified into “relevant” or “irrelevant” by a domain expert.
Predicate-argument features (1)
• Dedifferentiation of adenoid cystic carcinoma: report of a case implicating p53 gene mutation.
X gene/disease
ARG2
Predicate-argument features (2)
• These results suggested that targeted disruption of Cyp19 caused anovulation and precocious depletion of ovarian follicles.
• Furthermore, AML cells with methylated p15(INAK4B) tended to express higher levels of DNMT1 and 3B.
X disease/gene
ARG2ARG1
gene/disease
Extraction accuracy
• Training/test data: 2,253 sentences
• 10-fold cross validation
features recall precision f-score
N/A 1.0 0.351 0.520
+ bag of words 0.733 0.682 0.706
+ local context 0.733 0.695 0.714
+ predicate-argument structures
0.759 0.710 0.733
DGA explorer
Summary
• The GENIA corpus– Part-of-speech: 2000 abstracts– Named-entities: 2000 abstracts– Parse tree: 500 abstracts
• Machine learning– Maximum entropy modeling
• Inequality constraints• Feature forests
– Bidirectional inference for sequence tagging• NLP tools
– Part-of-speech tagger: 97.11%– Chunker: 93.7%– HPSG parser: 87.5%– Term variant generation
• Extracting disease-gene associations from MEDLINE
Software and resources• Machine learning packages
– Maximum entropy with inequality constraints– Maximum entropy for feature forests
• Taggers and Parsers– PoS tagger– Chunker– Named-entity tagger– HPSG parser
• GENIA resource– Named-entity corpus– Part-of-speech corpus– Tree corpus– Co-reference corpus (Singapore Univ.)– HPSG parsed results (100,000 MEDLINE abstracts)