Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

20
DEEP LEARNING JP [DL Papers] “Language as a Latent Variable: Discrete Generative Models for Sentence Compression” Toru Fujino, Chen Lab http://deeplearning.jp/

Transcript of Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

Page 1: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

DEEP LEARNING JP[DL Papers]

“Language as a Latent Variable: Discrete GenerativeModels for Sentence Compression”

Toru Fujino, Chen Lab

http://deeplearning.jp/

Page 2: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

書誌情報

• Yishu Miao (Oxford), Phil Blunsom (Oxford & Deepmind)– Miaoさん: ICML, EMNLP, IJCAI, CIKMなど多数

• EMNLP (long) 2016

• 概要– VAEの潜在表現に入力の要約文をおいて半教師あり学習した

• 選んだ理由– 教師無しとか半教師有りとかの要約に興味があった.– VAEの潜在表現で要約とするアイデアが面白いと思った.

2

Page 3: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

文(書)要約• 文書 (文) を重要な情報だけ残して短縮する問題

– 抽出型要約• 例) “I am a graduate student at the University of Tokyo.” -> “I am a student at the University.”

– 生成型要約• 例) “I am a graduate student at the University of Tokyo.” -> “I am a grad student at UT.”

• 最近の殆どの研究は教師ありのEncoder-Decoder– “Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond”,

Nallapati et al., CoNLL 2016

3

Page 4: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

自然言語における潜在表現

• 言語生成– “Generating Sentences from a Continuous Space”, Bowman et al., 2015

– “Toward Controllable Generation of Text”, Hu et al., 2017

• 連続的な潜在変数が一般4

Page 5: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

圧縮表現 = 要約• 言語の潜在表現を離散的な変数で表そうという試み• そのアプリケーションとして要約問題に適用

– 要約文は一般に入力文の圧縮された表現になっているから

5

入力文(x) 入力文(x)要約文(z)

Page 6: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

提案手法

• 概要– 潜在変数に自然言語をおくことで教師なし学習 (半教師あり学習) で文要約– Gigawordコーパス (文要約の一般的なデータセット) で (当時の) SotAを出した.

• 貢献– 言語の潜在変数を離散変数で表す生成モデルを提案– 抽出型, 生成型どちらでも学習できるフレームワークを提案– 半教師あり学習の枠組みで (当時の) SotA

オリジナル文 要約文

4

Page 7: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

提案手法

• Auto-Encoding Sentence Compression (ASC)– 教師なし学習– VAEのzを抽出型の要約文として表現

• Forced Attention Sentence Compression (FSC)– 教師あり学習 (元文, 要約文)

7

要約文

入力文

要約文

入力文

Page 8: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

ASC: 文の圧縮• Encoder: bidirectional LSTM

• Compressor: Pointer Network– 入力系列の単語から出力を選ぶ

– 語彙数を抑えることにより, 潜在変数𝑐 ∼ 𝑞$(𝑐|𝑠)のサンプリングの分散を抑える

• 通常の出力層: 語彙数 )

• Pointer Network: 入力系列の長さ|)|

8

Page 9: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

ASC: 文の再生成• Compressor: LSTM

• Decoder: LSTM

– 出力層: アテンションのようなもの

9

Page 10: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

ASC: 最適化• 目的関数: Lower bound + ハイパラlambda

– 𝑝 𝑐 : (圧縮文で?) 事前学習済みの言語モデル– 𝜆: 文の圧縮率をコントロールするパラメータ

• 大きいほど𝑝 𝑐 に近くなる (圧縮される). 実験では0.01

10

Page 11: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

ASC:θの勾配の計算• サンプリングによって近似

11

Page 12: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

ASC: 𝝓の勾配の計算• 目的関数を次のように書き直し

• Learning signal (報酬関数) を次のように設定

• 勾配

• 勾配の分散を抑えるためにBaselineを導入 (baselineも別に学習)

12

Page 13: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

FSC: • Decoder (入力文の復元) がない. 教師あり• 出力層: 入力系列の単語と語彙全体の単語に対するprobability

– ステップjの隠れ層とアテンションベクトルの類似度を計算

– 類似度が高いなら𝛼.を重視, 低いなら𝛽.を重視

• 目的関数: 対数尤度13

(入力系列の単語) (語彙全体)

Page 14: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

誤差関数まとめ

• ASC: 教師なし学習

• FSC: 教師あり学習

• ASC + FSC: 半教師あり学習

14

Page 15: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

実験設定

• データセット– Gigaword: 文要約の標準的なコーパス

• 最適化– Adam with 学習率0.002– REINFORCEのサンプリングは1個

15

Page 16: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

実験

• Extractive 要約– ASC, FSCともにPointer Network使用 (出力単語は入力系列に現れたもののみ)

– ASC (教師なし) を組み合わせることにより精度が向上した

16

Page 17: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

実験

• Abstractive な要約– ASC のみ Pointer Network

– ASC (教師なし) を組み合わせることにより精度が向上した

17

Page 18: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

実験

• 先行研究との比較

– 良くなった

18

Page 19: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

実験

• 生成例

19

入力文

要約文

入力文

要約文

Page 20: Language as a Latent Variable: Discrete Generative Models for Sentence Compression (EMNLP 2016)

感想

• 自然言語を自然言語のままで圧縮させて潜在表現とするのは少し無理があるのではないかと思った

20


Latent Dirichlet Allocation - Stanford AI Labai.stanford.edu/~ang/papers/jair03-lda.pdf · Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic

Latent Dirichlet Allocation - Stanford AI Labai.stanford.edu/~ang/papers/jair03-lda.pdf · Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic

Learning a Probabilistic Latent Space of Object Shapes …people.csail.mit.edu/tfxue/papers/nips2016_shape.pdfLearning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial

Learning a Probabilistic Latent Space of Object Shapes …people.csail.mit.edu/tfxue/papers/nips2016_shape.pdfLearning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial

EMNLP 2014 TreeNode Language Model

EMNLP 2014 TreeNode Language Model

Latent Dirichlet Allocation · Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures

Latent Dirichlet Allocation · Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures

GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING … · GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING WITH OBJECT-CENTRIC LATENT REPRESENTATIONS Martin Engelcke z, Adam R. Kosiorekzr,

GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING … · GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING WITH OBJECT-CENTRIC LATENT REPRESENTATIONS Martin Engelcke z, Adam R. Kosiorekzr,

Learning Structured Latent Space Encoding for Part Based …epxing/Class/10708-19/assets/... · 2019. 9. 11. · Learning Structured Latent Space Encoding for Part Based Generative

Learning Structured Latent Space Encoding for Part Based …epxing/Class/10708-19/assets/... · 2019. 9. 11. · Learning Structured Latent Space Encoding for Part Based Generative

Latent Variable Models...Plan for today 1 Latent Variable Models Learning deep generative models Stochastic optimization: Reparameterization trick Inference Amortization Stefano Ermon,

Latent Variable Models...Plan for today 1 Latent Variable Models Learning deep generative models Stochastic optimization: Reparameterization trick Inference Amortization Stefano Ermon,

Latent Chords: Generative Piano Chord Synthesis with ...

Latent Chords: Generative Piano Chord Synthesis with ...

GeLaTO: Generative Latent Textured Objects€¦ · GeLaTO: Generative Latent Textured Objects Ricardo Martin-Brualla, Rohit Pandey, So en Bouaziz, Matthew Brown, and Dan B Goldman

GeLaTO: Generative Latent Textured Objects€¦ · GeLaTO: Generative Latent Textured Objects Ricardo Martin-Brualla, Rohit Pandey, So en Bouaziz, Matthew Brown, and Dan B Goldman

Learning Deep Generative Models With Discrete Latent …bayesiandeeplearning.org/2017/papers/10.pdf · Learning Deep Generative Models With Discrete Latent Variables Hengyuan Hu Carnegie

Learning Deep Generative Models With Discrete Latent …bayesiandeeplearning.org/2017/papers/10.pdf · Learning Deep Generative Models With Discrete Latent Variables Hengyuan Hu Carnegie

Stochastic WaveNet: A Generative Latent Variable …Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data Guokun Lai 1Bohan Li Guoqing Zheng Yiming Yang1 Abstract

Stochastic WaveNet: A Generative Latent Variable …Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data Guokun Lai 1Bohan Li Guoqing Zheng Yiming Yang1 Abstract

Latent space oddity On the curvature of deep generative models · Latent space oddity: On the curvature of deep generative models. In Proceedings of International Conference on Learning

Latent space oddity On the curvature of deep generative models · Latent space oddity: On the curvature of deep generative models. In Proceedings of International Conference on Learning

Latent Dirichlet Allocation (LDA)1 - Hediberthedibert.org/wp-content/uploads/2018/07/lda.pdf · Latent Dirichlet allocation LDA is a generative probabilistic model of a corpus. Documents

Latent Dirichlet Allocation (LDA)1 - Hediberthedibert.org/wp-content/uploads/2018/07/lda.pdf · Latent Dirichlet allocation LDA is a generative probabilistic model of a corpus. Documents

Latent Dirichlet Allocation (LDA) and Topic modeling ... · 2.3 Latent Dirichlet Allocation LDA is a generative probabilistic model of a corpus. The basic idea is that the documents

Latent Dirichlet Allocation (LDA) and Topic modeling ... · 2.3 Latent Dirichlet Allocation LDA is a generative probabilistic model of a corpus. The basic idea is that the documents

Learning a Probabilistic Latent Space of Object Shapes via 3D … · 2017-01-13 · Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling Jiajun

Learning a Probabilistic Latent Space of Object Shapes via 3D … · 2017-01-13 · Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling Jiajun

EMNLP / NASTEAthedansimonson.com/simonson_DCNLP_nastea_presentation... · 2016-11-17 · EMNLP / NASTEA A recap of some EMNLP talks, and my own research presented at the Second Computing

EMNLP / NASTEAthedansimonson.com/simonson_DCNLP_nastea_presentation... · 2016-11-17 · EMNLP / NASTEA A recap of some EMNLP talks, and my own research presented at the Second Computing

Learning Deep Generative Models With Discrete …...DAP Report Learning Deep Generative Models With Discrete Latent Variables Hengyuan Hu Department of Machine Learning Carnegie Mellon

Learning Deep Generative Models With Discrete …...DAP Report Learning Deep Generative Models With Discrete Latent Variables Hengyuan Hu Department of Machine Learning Carnegie Mellon

Understanding Posterior Collapse in Generative Latent ...

Understanding Posterior Collapse in Generative Latent ...

Geometric Disentanglement for Generative Latent …openaccess.thecvf.com/content_ICCV_2019/papers/Aumentado...Geometric Disentanglement for Generative Latent Shape Models Tristan Aumentado-Armstrong*,

Geometric Disentanglement for Generative Latent …openaccess.thecvf.com/content_ICCV_2019/papers/Aumentado...Geometric Disentanglement for Generative Latent Shape Models Tristan Aumentado-Armstrong*,

PULSE: Self-Supervised Photo Upsampling via Latent Space ...€¦ · PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models Sachit Menon*, Alexandru

PULSE: Self-Supervised Photo Upsampling via Latent Space ...€¦ · PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models Sachit Menon*, Alexandru