Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a...
Transcript of Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a...
![Page 1: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/1.jpg)
Trans* and DeepWalk
by Ran Jia (22 Apr 2016) and Lili Mou (additional reading)
![Page 2: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/2.jpg)
Trans*
by Ran Jia
2016.4.22Relational fact: (head, relation, tail)
![Page 3: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/3.jpg)
TransE:Translating embeddings for modeling multi-relational data. Bordes, Antoine, et al. Advances in Neural Information Processing Systems. 2013.
TransH:Knowledge Graph Embedding by Translating on Hyperplanes. Wang, Zhen, et al. AAAI. 2014.
TransR:Learning Entity and Relation Embeddings for Knowledge Graph Completion. Lin, Yankai, et al. AAAI. 2015.
![Page 4: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/4.jpg)
TransE
![Page 5: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/5.jpg)
TransHA triple( h,r,t) ,a hyperplane with normal vector ,
Project h and t to the hyperplane: ,
Score function:
![Page 6: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/6.jpg)
TransR
![Page 7: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/7.jpg)
CTransR: Cluster-based TransRTransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity pairs under this relation, because these relations are usually rather diverse.
For a relation r, all entity pairs (h,t) in the training data are clustered into multiple groups, entity pairs in each group are expected to exhibit similar r relation.
Entity pairs are represented with their vector offsets (h-t), h and t are obtained with TransE.
![Page 8: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/8.jpg)
CTransR: Cluster-based TransR
差别不能太大
![Page 9: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/9.jpg)
Training Objective
![Page 10: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/10.jpg)
Data SetsWordNet and FreebaseFreebase( Steve jobs. Founded, Apple Inc.)FB15K, FB13
WordNetSemantic knowledge of wordsRelations between synsetsWN11, WN18
![Page 11: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/11.jpg)
Experiment SettingsLink PredictionTriple ClassicationRelation Extraction from Text
![Page 12: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/12.jpg)
Link PredictionFor each triple (h,r,t), replace the head/tail entity by all entities in the KG.
Rank these entities in descending (ascending?) order of scores
Evaluation Metric:(1) mean rank of correct entities(2) proportion of correct entities in top-10 ranked
entities(Hits@10)A good link predictor should achieve lower mean rank or higher Hits@10
![Page 13: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/13.jpg)
![Page 14: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/14.jpg)
Triple Classification
![Page 15: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/15.jpg)
![Page 16: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/16.jpg)
Relation Extraction from TextCombine the scores from text-based relation extraction model with the scores from knowledge graph embeddings to rank test triples.
Plot a precision-recall curves.
![Page 17: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/17.jpg)
Deep Walk
![Page 18: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/18.jpg)
● A graph (e.g., social networks) defines vertexes (analogous to word in word2vec) and edges (analogous to word cooccurrences)
● The basic idea: Train a “LM” over graph nodes
KDD'14
![Page 19: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/19.jpg)
Step I: A small random walk generates context as a “sentence.”
Note: The random walk is not actually necessary because we can alternatively define context as adjacent nodes. But as some critics point out, random walk increases randomness and is tunable by its weights.
Step II: Standard word2vec (either hierarchical softmax or SkipGram)
![Page 20: Trans* and DeepWalk - PKUSEIsei.pku.edu.cn/~moull12/resource/KB.pdf · TransE/TransH/TransR learn a unique vector for each relation, which may be under-representative to fit all entity](https://reader031.fdocuments.net/reader031/viewer/2022022716/5c21427009d3f269368c4b0f/html5/thumbnails/20.jpg)
An Interesting Phenomenon
● The connections of nodes in social networks, say, typically follow a power law distribution. So does word cooccurrences.