Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT...

16
Neural sequence-to-sequence models for machine translation CMSC 470 Marine Carpuat

Transcript of Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT...

Page 1: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Neural sequence-to-sequence models for machine translation

CMSC 470

Marine Carpuat

Page 2: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Machine Translation

• Translation system• Input: source sentence F

• Output: target sentence E

• Can be viewed as a function

• Modern machine translation systems

• 3 problems

• Modeling• how to define P(.)?

• Training/Learning• how to estimate parameters from

parallel corpora?

• Search• How to solve argmax efficiently?

Page 3: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

P(E|F) as an encoder-decoder model

The Encoder models the input/source entence

F=(f1,….f|F|)

The Decoder models the output/target sentence

E=(e1,….e|E|).

The decoder hidden state is initialized with the last

hidden state of the encoder

Page 4: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Neural Machine Translation

• Neural language models review

• Sequence to sequence models for MT• Encoder-Decoder

• Sampling and search (greedy vs beam search)

• How to train?

• Model variants and practical tricks

• Attention mechanism

Page 5: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Training

• Same as for RNN language modeling• Intuition: a good model assigns high probability to training examples

• Loss function• Negative log-likelihood of training data

• Also called cross-entropy loss

• Total loss for one example (sentence pair) = sum of loss at each time step (word)

• Backpropagation• Gradient of loss at time step t is propagated through the network all the way back to 1st

time step

Page 6: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Aside: why don’t we use BLEU as training loss?

Page 7: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Training in practice: online

Page 8: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Training in practice: batch

Page 9: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Training in practice: minibatch

• Compromise between online and batch

• Computational advantages• Can leverage vector processing instructions in modern hardware

• By processing multiple examples simultaneously

Page 10: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Problem with minibatches: examples have varying length

• 3 tricks (same as for language modeling)• Padding

• Add </s> symbol to make all sentences same length

• Masking• Multiply loss function calculated

over padded symbols by zero

• + sort sentences by length

Page 11: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Neural Machine Translation

• Neural language models review

• Sequence to sequence models for MT• Encoder-Decoder

• Sampling and search (greedy vs beam search)

• How to train?

• Model variants and practical tricks

• Attention mechanism

Page 12: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Other encoder structures:RNN variants

• LSTMs• Aim to address vanishing/exploding gradient issue

• Stacked RNNs

• …

Page 13: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Other encoder structures:Bidirectional encoder

•

Motivation:- Help bootstrap learning- By shortening length of

dependencies

Motivation: - Take 2 hidden vectors from source

encoder- Combine them into a vector of size

required by decoder

Page 14: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

A few more tricks: ensembling

• Combine predictions from multiple models

• Methods• Linear or log-linear interpolation

• Parameter averaging

Page 15: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Tricks: addressing length bias

• Default models tend to generate short sentences

• Solutions:• Prior probability on sentence length

• Normalize by sentence length

Page 16: Neural sequence-to-sequence models for machine translation · •Sequence to sequence models for MT •Encoder-Decoder ... - Help bootstrap learning - By shortening length of dependencies

Neural Machine Translation

• Neural language models review

• Sequence to sequence models for MT• Encoder-Decoder

• Sampling and search (greedy vs beam search)

• How to train?

• Model variants and practical tricks

• Attention mechanism


Testing sequence stratigraphic models by drilling Miocene ...geology.rutgers.edu/images/stories/faculty/miller... · 1236 Testing sequence stratigraphic models by drilling Miocene

Testing sequence stratigraphic models by drilling Miocene ...geology.rutgers.edu/images/stories/faculty/miller... · 1236 Testing sequence stratigraphic models by drilling Miocene

TimeSHAP: Explaining Recurrent Models through Sequence ...

TimeSHAP: Explaining Recurrent Models through Sequence ...

Statistical Language Models for Alternative Sequence Selectionpubman.mpdl.mpg.de/pubman/item/escidoc:1222608:4/... · Statistical Language Models for Alternative Sequence Selection

Statistical Language Models for Alternative Sequence Selectionpubman.mpdl.mpg.de/pubman/item/escidoc:1222608:4/... · Statistical Language Models for Alternative Sequence Selection

Interaction Models: Sequence and Collaboration · Interaction Models: Sequence and Collaboration • Consider the family of UML diagrams • Dynamic models (a UML view) • Interaction

Interaction Models: Sequence and Collaboration · Interaction Models: Sequence and Collaboration • Consider the family of UML diagrams • Dynamic models (a UML view) • Interaction

Sequence-to-Sequence Modelskevinduh/notes/1905-kduh-Seq2Seq.pdf · Recurrent models for sequence-to-sequence problems • We can use these models for both input and output • For

Sequence-to-Sequence Modelskevinduh/notes/1905-kduh-Seq2Seq.pdf · Recurrent models for sequence-to-sequence problems • We can use these models for both input and output • For

Sequence Models Scenarios Scenarios Sequence Diagram Sequence Diagram Guidelines for Sequence Models Guidelines for Sequence Models.

Sequence Models Scenarios Scenarios Sequence Diagram Sequence Diagram Guidelines for Sequence Models Guidelines for Sequence Models.

Lecture 3: Markov models of sequence evolution

Lecture 3: Markov models of sequence evolution

Hidden Markov Models for biological sequence …regulatorygenomics.upf.edu/courses/Master_AGB/4_Hidden...Hidden Markov Models for biological sequence analysis II Eduardo Eyras Computational

Hidden Markov Models for biological sequence …regulatorygenomics.upf.edu/courses/Master_AGB/4_Hidden...Hidden Markov Models for biological sequence analysis II Eduardo Eyras Computational

SEQUENCE-TO-SEQUENCE MODELS FOR PUNCTUATED TRANSCRIPTION COMBINING LEXICAL AND ... · 2018-03-28 · SEQUENCE-TO-SEQUENCE MODELS FOR PUNCTUATED TRANSCRIPTION COMBINING LEXICAL AND

SEQUENCE-TO-SEQUENCE MODELS FOR PUNCTUATED TRANSCRIPTION COMBINING LEXICAL AND ... · 2018-03-28 · SEQUENCE-TO-SEQUENCE MODELS FOR PUNCTUATED TRANSCRIPTION COMBINING LEXICAL AND

Lecture 9: Machine Translation and Sequence-to-Sequence Models · Lecture 9: Machine Translation and Sequence-to-Sequence Models Andr e Martins Deep Structured Learning Course, Fall

Lecture 9: Machine Translation and Sequence-to-Sequence Models · Lecture 9: Machine Translation and Sequence-to-Sequence Models Andr e Martins Deep Structured Learning Course, Fall

Experimental models of Basement -controlled salients application … · ify the curvature geometry in the sand models. The maximum shortening we used in our models was 30%. An Acad

Experimental models of Basement -controlled salients application … · ify the curvature geometry in the sand models. The maximum shortening we used in our models was 30%. An Acad

Week 6: Protein sequence models, likelihood, hidden Markov ...evolution.gs.washington.edu/gs570/2014/week6.pdf · Week 6: Protein sequence models, likelihood, hidden Markov models

Week 6: Protein sequence models, likelihood, hidden Markov ...evolution.gs.washington.edu/gs570/2014/week6.pdf · Week 6: Protein sequence models, likelihood, hidden Markov models

Mutation models I: basic nucleotide sequence mutation modelspbeerli/isc5317-notes/pdfs/05-mutation.pdf · Mutation models I: basic nucleotide sequence mutation models Peter Beerli

Mutation models I: basic nucleotide sequence mutation modelspbeerli/isc5317-notes/pdfs/05-mutation.pdf · Mutation models I: basic nucleotide sequence mutation models Peter Beerli

Sequence Stratigraphic Models for - GCSSEPM Home · entitled “Sequence Stratigraphic Models for Exploration and ... a progress report on the evolution of sequence ... final editing,

Sequence Stratigraphic Models for - GCSSEPM Home · entitled “Sequence Stratigraphic Models for Exploration and ... a progress report on the evolution of sequence ... final editing,

Graphical Models for Sequence Labeling in NLP

Graphical Models for Sequence Labeling in NLP

Deep Learning Sequence to Sequence models: Attention Modelsbhiksha/courses/deeplearning/... · Deep Learning Sequence to Sequence models: Attention Models 17 March 2018 1. ... ENCODER

Deep Learning Sequence to Sequence models: Attention Modelsbhiksha/courses/deeplearning/... · Deep Learning Sequence to Sequence models: Attention Models 17 March 2018 1. ... ENCODER

Lecture 5: Sequence Models II

Lecture 5: Sequence Models II

Sequence to Sequence Models · Sequence to Sequence Models Jon Dehdari January 25, 2016 1 / 12. Good Morning! ... enc = LSTM or GRU) Are they any good? For Elman networks (SRNs),

Sequence to Sequence Models · Sequence to Sequence Models Jon Dehdari January 25, 2016 1 / 12. Good Morning! ... enc = LSTM or GRU) Are they any good? For Elman networks (SRNs),

Sequence to Sequence Models - GitHub Pages

Sequence to Sequence Models - GitHub Pages

Dynamic Models Sequence Diagrams Collaboration Diagrams Activity Diagrams.

Dynamic Models Sequence Diagrams Collaboration Diagrams Activity Diagrams.