Text-enhanced Representation Learning for Knowledge...
-
Upload
duongtuyen -
Category
Documents
-
view
235 -
download
0
Transcript of Text-enhanced Representation Learning for Knowledge...
![Page 1: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/1.jpg)
1
Reporter: Zhigang WANG
Authors: Zhigang WANG, Juanzi LI, Zhiyuan LIU, Jie TANG
Tsinghua University
To appear in IJCAI 2016
2016-04-17
Text-enhanced Representation Learning for Knowledge Graph
![Page 2: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/2.jpg)
2
Outline
Introduction
Problem Definition
Our Proposed Approach
Experiments and Analysis
Conclusion
![Page 3: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/3.jpg)
3
Representation Learning for KG
Input
a knowledge graph 𝒦𝒢 = {(ℎ, 𝑟, 𝑡)}
Target To learn one embedding (a 𝑘-dimensional vector) for each
entity: ℎ → ℎ and 𝑡 → റ𝑡, where ℎ, റ𝑡 ∈ ℝ 𝑘
( Avatar, /film/film/directed_by, James Cameron )
h r t
![Page 4: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/4.jpg)
4
Translation-based Methods
TransE For each triple (head, relation, tail), treat relation as a
translation from head to tail
Simple, effective, and achieving the state-of-the-art performance
Bordes, et al. (2013). Translating embeddings for modeling multi-relational data. NIPS.
James Cameron
Avatar
_directed_by
Titanic
+≈
=
Idea
Issues when modelling 1-to-N, N-to-1, N-to-N relations
![Page 5: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/5.jpg)
5
Translation-based Methods
TransH and TransR Build relation-specific entity embeddings
TransH TransR
Wang, et al. (2014). Knowledge graph embedding by translating on hyperplanes. AAAI.Lin, et al. (2015). Learning entity and relation embeddings for knowledge graph completion. AAAI.
Motivation 1. low performance on 1-to-N, N-to-1 and N-to-N relations
![Page 6: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/6.jpg)
6
Translation-based Methods
Learn embeddings directly from the graph structure Graph sparseness
In domain-specific and non-English situations
James CameronAvatar _directed_by+ =
Canada
_country
20th Century Fox
_distributed_by
English
_language
Motivation 2. limited performance by the structure sparseness of KG
![Page 7: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/7.jpg)
7
Our Idea
Text-enhanced Representation Learning for KG Go back to traditional relation extraction
Inspired by distant supervision
( Avatar, /film/film/directed_by, James Cameron )
James Francis Cameron, the famous director of the movie Avatar, is an
The fiction film Avatar directed by J. Cameron was nominated by
In 1994 director James Cameron wrote an 80-page treatment for Avatar
{film, movie, directed, ...} {director, ...}{direct}
Triple:
Context:
Text:
Contributions:[Motivation 1]. Enable each relation to own different representations for different head and tail entities.[Motivation 2]. Incorporate the textual contexts to each entity and relation.
![Page 8: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/8.jpg)
8
Outline
Introduction
Problem Definition
Our Proposed Approach
Experiments and Analysis
Conclusion
![Page 9: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/9.jpg)
9
Problem Definition
Input Knowledge Graph
𝒦𝒢 = {(ℎ, 𝑟, 𝑡)}
Text Corpus𝒟 = 𝑤1…𝑤𝑖 … 𝑤𝑚
Text-enhanced Knowledge Embedding (TEKE)
learn the entity embeddings ℎ → ℎ ∈ ℝ 𝑘 and 𝑡 → റ𝑡 ∈ ℝ 𝑘 for each triple (ℎ, 𝑟, 𝑡) by utilizing the rich text information in 𝒟 to deal with
• low performance on 1-to-N, N-to-1, N-to-N relations
• knowledge graph sparseness
learn the relation embedding 𝑟 → റ𝑟 ∈ ℝ 𝑘
![Page 10: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/10.jpg)
10
Outline
Introduction
Problem Definition
Our Proposed Approach
Experiments and Analysis
Conclusion
![Page 11: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/11.jpg)
11
The Proposed Approach
( Avatar, /film/film/directed_by, James Cameron )Triple:
Entity
Annotation
James Francis Cameron, the famous director of the movie Avatar, is an
The fiction film Avatar directed by J. Cameron was nominated by
In 1994 director James Cameron wrote an 80-page treatment for Avatar
{film, movie, directed, ...} {director, ...}{direct}
·A ·B ·AEntity/Relation
Representation
Modelling
Representation
Training
Textual Context
Embedding
①
②
④
③
![Page 12: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/12.jpg)
12
The Proposed Approach
Entity Annotation Given the text corpus 𝒟 = 𝑤1…𝑤𝑖 …𝑤𝑚 , use an entity linking
tool to automatically label the entities in 𝒦𝒢, and get an entity-annotated text corpus:
𝒟′ = 𝑋1…𝑋𝑖 …X𝑚′
Textual Context Embedding co-occurrence network 𝒢 = 𝒳,𝒴
𝑥𝑖 ∈ 𝒳 : denotes to the node (a word or an entity)
𝑦𝑖𝑗 ∈ 𝒴 : co-occurrence frequency between 𝑥𝑖 and 𝑥𝑗
![Page 13: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/13.jpg)
13
The Proposed Approach
Textual Context Embedding Pointwise textual context
n 𝑥𝑖 = 𝑥𝑗 𝑦𝑖𝑗 > 𝜃
n 𝑨𝒗𝒂𝒕𝒂𝒓 = 𝑓𝑖𝑙𝑚,𝑚𝑜𝑣𝑖𝑒, 𝑑𝑖𝑟𝑒𝑐𝑡𝑒𝑑…n 𝑱𝒂𝒎𝒆𝒔_𝑪𝒂𝒎𝒆𝒓𝒐𝒏 = 𝑑𝑖𝑟𝑒𝑐𝑡𝑜𝑟…
Pairwise textual context
n 𝑥𝑖 , 𝑥𝑗 = 𝑥𝑘 𝑥𝑘 ∈ n 𝑥𝑖 ∩ n 𝑥𝑗n 𝑨𝒗𝒂𝒕𝒂𝒓, 𝑱𝒂𝒎𝒆𝒔_𝑪𝒂𝒎𝒆𝒓𝒐𝒏 = 𝑑𝑖𝑟𝑒𝑐𝑡 …
![Page 14: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/14.jpg)
14
The Proposed Approach
Textual Context Embedding Word Embedding Learning 𝑥𝑖 → 𝒙𝑖
Pointwise textual context embedding of 𝑥𝑖:
𝒏 𝑥𝑖 =1
σ𝑥𝑗∈n 𝑥𝑖𝑦𝑖𝑗
𝑥𝑗∈n 𝑥𝑖
𝑦𝑖𝑗 ∙ 𝒙𝑗
Pairwise textual context embedding of 𝑥𝑖 and 𝑥𝑗:
𝒏 𝑥𝑖 , 𝑥𝑗 =1
𝑍
𝑥𝑘∈n 𝑥𝑖,𝑥𝑗
𝑚𝑖𝑛 𝑦𝑖𝑘, 𝑦𝑗𝑘 ∙ 𝒙𝑘
{film, movie, directed, ...} {director, ...}{direct}
Textual Context
Embedding
![Page 15: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/15.jpg)
15
The Proposed Approach
Entity/Relation Representation Modeling Incorporate the textual context information to the representation
learning on knowledge graph
𝒉 = 𝒏 ℎ 𝑨 + 𝒉
ො𝒕 = 𝒏 𝑡 𝑨 + 𝒕
ො𝒓 = 𝒏 ℎ, 𝑡 𝑩 + 𝒓
𝑓 ℎ, 𝑟, 𝑡 = 𝒉 + ො𝒓 − ො𝒕2
2
Linear transformation of textual context information
- given a relation, different textual context embeddingsfor different pairs of (head, tail) entities- to better handle 1-to-N, N-to-1, and N-to-N relations
Incorporate textual context information into the KG
- more background information- to deal with knowledge graph sparseness
![Page 16: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/16.jpg)
16
The Proposed Approach
Representation Training
Margin-based score function
Stochastic gradient descent (SGD)
![Page 17: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/17.jpg)
17
Outline
Introduction
Problem Definition
Our Proposed Approach
Experiments and Analysis
Conclusion
![Page 18: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/18.jpg)
18
Experiments and Analysis
Datasets 4 benchmark knowledge graphs
Entity-annotated Wikipedia corpuses
![Page 19: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/19.jpg)
19
Experiments and Analysis
Evaluation (China, /location/location/adjoin, North_Korea)
Link Prediction• Mean Rank: 11
• Hits@10: 0%
• Raw; Filter: 9; 100%
Triple Classification• a binary classification task
Head China
Relation /location/location/adjoin
1 Japan
2 Taiwan
3 Israel
4 South_Korea
5 Argentina
6 France
7 Philippines
8 Hungary
9 Germany
10 USA
11 North_Korea
![Page 20: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/20.jpg)
20
Link Prediction
TEKE compare with baselines
A lower Mean Rank is better while a higher Hits@10 is better
Mean Rank• TEKE methods perform much better than the baselines on WN18.
• No much improvement is observed on FB15K
Hits@10• TEKE methods outperform other baselines significantly and
consistently
![Page 21: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/21.jpg)
21
Link Prediction
Capability to handle 1-to-N, N-to-1 and N-to-N relations FB15K: 1-1, 1-N, N-1, N-N 24.2%, 22.9%, 28.9%, 24.0%
TEKE methods significantly outperform the baselines when predicting the entity where multiple entities could be correct.
TEKE methods have not shown much advantage for predicting the entity where only one entity is correct.
![Page 22: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/22.jpg)
22
Link Prediction
Capability to handle knowledge graph sparseness
Rank 3,000 entities for 2,238 triples for all three datasets
As the graph density gets higher, both TransE and TEKE_E perform better.
TEKE_E achieves the highest improvement on the sparsest FB3K dataset.
![Page 23: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/23.jpg)
23
Triple Classification
TEKE compare with baselines
TEKE_E and TEKE_H consistently outperform the comparison methods, especially on WN11.
TEKE_R (unif) on WN11 and TEKE_R (bern) on FB13 perform better than TransR, while others perform a bit worse.
![Page 24: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/24.jpg)
24
Outline
Introduction
Problem Definition
Our Proposed Approach
Experiments and Analysis
Conclusion
![Page 25: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/25.jpg)
25
Conclusion and Future Work
A novel text-enhanced knowledge embedding method named TEKE for knowledge graph representation learning to deal with Low performance on 1-to-N, N-to-1 and N-to-N relations
Limited performance by structure sparseness of KG
Future Work Improve performance on 1-to-1 relations
Experimentally analyze the influence of entity annotation
Use different text corpus
Incorporate knowledge reasoning
…
![Page 26: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/26.jpg)
26
Thanks!
Zhigang WANG
http://xlore.org/
![Page 27: Text-enhanced Representation Learning for Knowledge Graphqngw2014.bj.bcebos.com/upload/2016/04/王志刚-Text-enhanced... · 11 The Proposed Approach Triple: ( Avatar , /film/film/directed](https://reader030.fdocuments.net/reader030/viewer/2022021808/5bff573209d3f270278c8c37/html5/thumbnails/27.jpg)
27
TransE
Ideavec('Paris') - vec('France') ≅ vec('Rome') - vec('Italy')
Treat each relation as one unique vector
vec('_has_capital') = vec('_ has_capital')
and it would be reasonable that
vec('Paris') - vec('France')≅ vec('Rome') - vec('Italy') = vec('_capital_of')
_has_capital _has_capital
റ𝑡 ℎ റ𝑡 ℎ റ𝑟
Assumption റ𝒕 - 𝒉 = 𝒓