Deep Learning for Machine Translation
-
Upload
matiss-rikters -
Category
Technology
-
view
1.410 -
download
0
Transcript of Deep Learning for Machine Translation
What makes learning deep?
Schedule• Kevin Duh - Fundamentals for DL4MT I & II• Lab 1 - Prep and setup. Compare logistic regression, MLP,
and stacked auto-encoders on data• Lab 2 - word embedding (SGNS), visualization.
• Hermann Ney - Neural LMs and TMs for SMT I & II• Lab 1 - Rescore n-best lists using RNN LM• Lab 2 - n-best rescoring using uni/bidirectional translation and joint models
• Kyunghyun Cho - Neural MT I & II• Lab 1 - Data Preparation: Basic preprocessing; Encoder-Decoder with Theano
(without attention)• Lab 2 - Attention-based Encoder-Decoder with Theano
Neural Networks (NN) – brief introduction
• An NN consists of:• Multiple layers (an input layer, zero
or more hidden layers, and an output layer) that consist of a set of neurones (xi, hj, and y)
• Interconnections between nodes of different layers that have weights assigned (wij and wj)
• Activation functions for each neurone that convert weighted input of neurons into output values
• Deep neural networks –NNs with one or more hidden layers
Activation functions• Many different
types• Most common
functions in NLP – logistic sigmoid and hyperbolic tangent (because of their non-linearity)
http://blog.sciencenet.cn/blog-457187-878461.html
Training• Neural networks are trained
with the backpropagation algorithm• Aim – reduce cost/error of the
model by iteratively performing a forward pass, calculating the error, and adjusting weights based on the “direction” (read - derivative) of the error. The error is “backpropagated” from the output layer back to the input layer
• A long description with a lot of theory: http://neuralnetworksanddeeplearning.com/chap2.html
Word Embeddings• A continuous space representation of words• Multi-dimensional vectors• Decimal values
• By-product of neural networks where words are vectorised• Skip-gram models• neural network language models• neural machine translation models• etc.
Word Embeddings – Skip-Gram model• Input – word (one of N vector)• Output – context of the word• Embeddings – the trained weight
matrix W between the input layer and the hidden layer• Each row represents the embedding
of a single word
• Implementation: word2vec
http://alexminnaar.com/word2vec-tutorial-part-i-the-skip-gram-model.html
Word embeddings are different• The distributional representation of words has to be trained for specific tasks
(e.g., Skip-gram word embeddings are not good for translation)• Similar words (according to cosine similarity) of the given words using different
word embedding models
http://arxiv.org/pdf/1412.6448.pdf
Feedforward Neural Net Language Model (NNLM)• Y. Bengio, R.
Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
Feedforward Neural Net Language Model (NNLM)• The same,
but with a simpler figure
/ http://www.cs.cmu.edu/~mfaruqui/talks/nn-clab.pdf /
What NNLMs are (supposedly) good at(… what n-gram models never will)?
Language Modelling and Machine Translation using Neural NetworksHermann Neyhttp://ej.uz/NNLM
Language Modelling• Conventional Language Modelling
• Measure the quality of an LM with perplexity
• Problem: most of the events are never seen in training data
Language Modelling Using Neural Networks• non ANN: count models (Markov chain):• limited history of predecessor words• smooth relative frequencies
• feedforward multi-layer perceptron (FF MLP):• limited history too• use predecessor words as input to MLP
• recurrent neural networks (RNN):• advantage: unlimited history
Recurrent Neural Net Language Model (RNNLM)
Experiments Results with JRC-ACQUIS EN-LV
Approach Perplexity CPU time Size
5-gram count language model 48.0376 5 minutes + 2 minutes (binarize) 1118MB
4-gram Feedforward Neural Net Language Model with 2 layers• 1000 wrord classes;• batch-size 64; • learning-rate 5e-3;• 200 nodes per input word, and a
subsequent layer of 200 nodes with a sigmoid activation function.
126.9841 1 week 43MB
Practicalities of ANN LM Training (Implementation and Software)• no regularization, no momentum term, no drop-out (so far!)• no pre-training (so far!)• vocabulary reduction: remove singletons, or keep most frequent words• random initialization of weights: Gaussian of mean 0, variance 0.01• training criterion: cross-entropy (perplexity)• stopping: cross-validation, perplexity on a development text• initial learning rate: typically between 1 · e−3 and 1 · e−2• learning rate: halved when the dev perplexity is worse than the best of previous
epochs• use of mini-batches: 4 to 64• low level implementation in C++• GPUs (typically) not used for the results presented in this talk
Language Modelling Using Neural Networks labExercice Sheet: http://ej.uz/NNLMlabhttps://www-i6.informatik.rwth-aachen.de/web/Software/rwthlm.php• Data preparation• Training on small data• Tuning of hyper-parameters• Modifying the network architecture
Conventional SMT
Translation Model based on FF MLP
Joint Language and Translation Model based on Feedforward MLP
Experiments Results
Experiments Results
Experiments Results Syntax-based Multi-System Hybrid Translator + NNLM on JRC-ACQUIS EN-LV
Approach BLEUMHyT with 5-gram count language model 22.69
SyMHyT with 5-gram count language model 24.72
SyMHyT with 4-gram Feedforward Neural Net Language Model with 2 layers 23.71
Machine Translation using Neural Networks lab• Exercice Sheet: http://ej.uz/NNMTlab• Part 1: N-best Reranking using Neural Network Language Models
• Obtain the new 1-best hypotheses• Measure the translation quality• Do reranking and compare to the results obtained before reranking.
• Part 2: Neural Network Translation Models• Train a unidirectional translation model
• Train a unidirectional joint model• Train a bidirectional translation model
• Train a bidirectional joint model• Try to obtain better perplexity values by changing the batch size and learning rate
• Part 3: N-best Reranking using Neural Network Translation Models• Apply rescoring using each of the unidirectional and bidirectional translation and joint models• Optimize the model weights with MERT to achieve the best BLEU score on the dev dataset• Evaluate the translation hypotheses for each of the rescoring experiments
Neural Machine Translation• Encoder-decoder model
https://github.com/nyu-dl/NLP_DL_Lecture_Note/blob/master/lecture_note.pdf
Neural Machine Translation
Neural Machine Translation• Encoder-decoder model• (a) – encoder• (b) - decoder
https://github.com/nyu-dl/NLP_DL_Lecture_Note/blob/master/lecture_note.pdf
http://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part-2/
Neural Machine Translation• Bi-directional recurrent neural networks for attention-based models
http://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part-3/
Neural Machine Translation• Attention-based models• Due to its sequential nature, a
recurrent neural network tends to remember recent symbols better• The attention mechanism
allows to focus at each time step at the relevant symbols by selecting the appropriate vector that summarises the input sentence
http://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part-3/
Attention-based encoder-decoder NMT• English-Latvian
• Vocabulary – 30K• Embedding dimensions – 400• Hidden layer dimensions – 1,024• Batch size - 14• Training corpus – DGT-TM (2,401,815 unique sentences)• Trainined on an NVIDIA GeForce GTX 960 (2GB) GPU• Training time - ~4 days and 2 hours (crashed due to out of memory exception)
• But ... Luckily it saves models iteratively
• During training uses ~40GB of virtual memory• Translation time of 512 sentence test set
(however, it includes also the model loading time)• NMT: 19 minutes and 3 seconds (translation with CPU on 6 cores)• LetsMT: 1 minute and 39 seconds (translation with CPU on 1 core)
https://github.com/kyunghyuncho/dl4mt-material/tree/master/session2
Attention-based encoder-decoder NMT• Comparison with LetsMT (LetsMT – 13.93 BLEU, NMT – 12.42 BLEU)
• Translations more fluent (even if not always correct «according to reference»)
Attention-based encoder-decoder NMT• Unknown words are a problem
Attention-based encoder-decoder NMT
• Sometimes the context around unknown words is surprisingly good
Attention-based encoder-decoder NMT
• Sometimes the NMT creates a translation that is (probably) equally as good as the reference
Attention-based encoder-decoder NMT
• However, sometimes the translation is also bad (total nonsense)
Attention-based encoder-decoder NMT• Named entities that are listed with commas can cause issues
Attention-based encoder-decoder NMT• Word alignments • What is different from
Giza++?• LetsMT translation:šīs paradigma pamatelements ir jaunā struktūra informācijas apstrādes sistēmu .
Attention-based encoder-decoder NMT• Model trained on 50K vocabulary and a bath size of 12• After 300,000 updates (or 3,600,000 observed sentences)• I.e., model is not yet fully trained• Results:
• LetsMT – 13.93 BLEU, NMT – 12.48 BLEU(+0.06)• Not good, but it may improve since the model has not finished training…
• After 520,000 updates (6,240,000 observed sentences)• LetsMT – 13.93 BLEU, NMT – 11.88 BLEU(-0.54)
Attention-based encoder-decoder NMT
• Lessons learned (from the tiny but long experiments):• You need to have a «good» GPU (6GB GDDR may not be enough for systems of a decent
vocabulary size)• A 2GB card will not allow building models with a vocabulary that is larger than 30-50K• The «good» GPUs are expensive (>1K€)• Only Nvidia GPUs are currently usable (the existing libraries are built/tuned for CUDA), OpenCL is an
alternative that is under-supported• If training with a GPU takes up to a week, training with a CPU is a no go• 30K is miles away from a decent vocabulary• You need to have means to handle unknown words• From the translation quality positive tendencies are evident, but an experiment with a
more decent data set and a larger vocabulary is necessary to make more justified judgements
Only
€ 5.979
TensorFlowhttps://www.tensorflow.org/
• Deep Flexibility• True Portability• Connect Research and Production• Auto-Differentiation• Language Options• Maximize Performance
Theanohttp://deeplearning.net/software/theano/
• Built for python• Tight integration with NumPy – Use numpy.ndarray in Theano-compiled functions.• Transparent use of a GPU – Perform data-intensive calculations up to 140x faster than with CPU.(float32 only)• Efficient symbolic differentiation – Theano does your derivatives for function with one or many inputs.• Speed and stability optimizations – Get the right answer for log(1+x) even when x is really tiny.• Dynamic C code generation – Evaluate expressions faster.• Extensive unit-testing and self-verification – Detect and diagnose many types of mistake.
• The EN-LV NMT model was trained using Theano
• Speed comparison of different NN libraries: https://github.com/soumith/convnet-benchmarks