Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 101–112Brussels, Belgium, October 31 - November 1, 2018. c©2018 Association for Computational Linguistics

https://doi.org/10.18653/v1/W18-6311

Contextual Neural Model for Translating Bilingual Multi-SpeakerConversations

Sameen Maruf1Monash University

VIC, Australia

Andre F. T. Martins2Unbabel &

Instituto de TelecomunicacoesLisbon, Portugal

1{firstname.lastname}@monash.edu2andre.martins@unbabel.com

Gholamreza Haffari1Monash University

VIC, Australia

Abstract

Recent works in neural machine transla-tion have begun to explore document trans-lation. However, translating online multi-speaker conversations is still an open problem.In this work, we propose the task of trans-lating Bilingual Multi-Speaker Conversations,and explore neural architectures which exploitboth source and target-side conversation histo-ries for this task. To initiate an evaluation forthis task, we introduce datasets extracted fromEuroparl v7 and OpenSubtitles2016. Our ex-periments on four language-pairs confirm thesignificance of leveraging conversation his-tory, both in terms of BLEU and manual eval-uation.

1 Introduction

Translating a conversation online is ubiquitous inreal life, e.g. in the European Parliament, UnitedNations, and customer service chats. This sce-nario involves leveraging the conversation historyin multiple languages. The goal of this paper is topropose and explore a simplified version of sucha setting, referred to as Bilingual Multi-SpeakerMachine Translation (Bi-MSMT), where speak-ers’ turns in the conversation switch the source andtarget languages. We investigate neural architec-tures that exploit the bilingual conversation historyfor this scenario, which is a challenging problemas the history consists of utterances in both lan-guages.

The ultimate aim of all machine translationsystems for dialogue is to enable a multi-lingualconversation between multiple speakers. How-ever, translation of such conversations is not well-explored in the literature. Recently, there has beenwork focusing on using the discourse or docu-ment context to improve NMT, in an online set-ting, by using the past context (Jean et al., 2017;Wang et al., 2017; Bawden et al., 2017; Voita

et al., 2018), and in an offline setting, using thepast and future context (Maruf and Haffari, 2018).In this paper, we design and evaluate a conversa-tional Bi-MSMT model, where we incorporate thesource and target-side conversation histories into asentence-based attentional model (Bahdanau et al.,2015). Here, the source history comprises of sen-tences in the original language for both languages,and the target history consists of their correspond-ing translations. We experiment with differentways of computing the source context represen-tation for this task. Furthermore, we present aneffective approach to leverage the target-side con-text, and also present an intuitive approach forincorporating both contexts simultaneously. Toevaluate this task, we introduce datasets extractedfrom Europarl v7 and OpenSubtitles2016, con-taining speaker information. Our experimentson English-French, English-Estonian, English-German and English-Russian language-pairs showimprovements of +1.44, +1.16, +1.75 and +0.30BLEU, respectively, for our best model over thecontext-free baseline. The results show the im-pact of conversation history on translation of bilin-gual multi-speaker conversations and can be usedas benchmark for future work on this task.

2 Related Work

Our research builds upon prior work in the fieldof context-based language modelling and context-based machine translation.

Language Modelling There have been fewworks on leveraging context information for lan-guage modelling. Ji et al. (2015) introduced Doc-ument Context Language Model (DCLM) whichincorporates inter and intra-sentential contexts.Hoang et al. (2016) make use of side informa-tion, e.g. metadata, and Tran et al. (2016) useinter-document context to boost the performance

101

of RNN language models.For conversational language modelling, Ji and

Bilmes (2004) propose a statistical multi-speakerlanguage model (MSLM) that considers wordsfrom other speakers when predicting words fromthe current one. By taking the inter-speaker depen-dency into account using a normal trigram context,they report significant reduction in perplexity.

Statistical Machine Translation The few SMT-based attempts to document MT are either restric-tive or do not lead to significant improvementsupon automatic evaluation. Few of these dealwith specific discourse phenomena, such as re-solving anaphoric pronouns (Hardmeier and Fed-erico, 2010) or lexical consistency of translations(Garcia et al., 2017). Others are based on a two-pass approach i.e., to improve the translations al-ready obtained by a sentence-level model (Hard-meier et al., 2012; Garcia et al., 2014).

Neural Machine Translation Using context-based neural models for improving online and of-fline NMT is a popular trend recently. Jean et al.(2017) extend the vanilla attention-based NMTmodel (Bahdanau et al., 2015) by conditioningthe decoder on the previous source sentence via aseparate encoder and attention component. Wanget al. (2017) generate a summary of three previoussource sentences via a hierarchical RNN, which isthen added as an auxiliary input to the decoder.Bawden et al. (2017) explore various ways to ex-ploit context from the previous sentence on thesource and target-side by extending the modelsproposed by Jean et al. (2017); Wang et al. (2017).Apart from being difficult to scale, they report de-teriorated BLEU scores when using the target-sidecontext.

Tu et al. (2017) augment the vanilla NMTmodel with a continuous cache-like memory,along the same lines as the cache-based systemfor traditional document MT (Gong et al., 2011),which stores hidden representations of recentlygenerated words as translation history. The pro-posed approach shows significant improvementsover all baselines when translating subtitles andcomparable performance for news and TED talks.Along similar lines, Kuang et al. (2018) proposedynamic and topic caches to capture contextualinformation either from recently translated sen-tences or the entire document to model coherencefor NMT. Voita et al. (2018) introduce a context-

aware NMT model in which they control and anal-yse the flow of information from the extended con-text to the translation model. They show that us-ing the previous sentence as context their model isable to implicitly capture anaphora.

For the offline setting, Maruf and Haffari (2018)incorporate the global source and target documentcontexts into the base NMT model via memorynetworks. They report significant improvementsusing BLEU and METEOR for the contextualmodel over the baseline. To the best of our knowl-edge, there has been no work on Multi-SpeakerMT or its variation to date.

3 Preliminaries

3.1 Problem Formulation

We are given a dataset that comprises parallelconversations, and each conversation consists ofturns. Each turn is constituted by sentences spo-ken by a single speaker, denoted by x or y, if thesentence is in English or Foreign language, respec-tively. The goal is to learn a model that is able toleverage the mixed-language conversation historyin order to produce high quality translations.

3.2 Data

Standard machine translation datasets are inappro-priate for Bi-MSMT task since they are not com-posed of conversations or the speaker annotationsare missing. In this section, we describe how weextract data from raw Europarl v7 (Koehn, 2005)and OpenSubtitles20161 (Lison and Tiedemann,2016) for this task2.

Europarl The raw Europarl v7 corpus (Koehn,2005) contains SPEAKER and LANGUAGE tagswhere the latter indicates the language the speakerwas actually using. The individual files are firstsplit into conversations. The data is tokenised (us-ing scripts by Koehn (2005)), and cleaned (head-ings and single token sentences removed). Con-versations are divided into smaller ones if thenumber of speakers is greater than 5.3 The cor-pus is then randomly split into train/dev/test setswith respect to conversations in ratio 100:2:3. TheEnglish side of the corpus is set as reference, and

1http://www.opensubtitles.org/2The data is publicly available at https://github.

com/sameenmaruf/Bi-MSMT.git3Using the conversations as is or setting a higher thresh-

old further reduces the data due to inconsistencies in conver-sation/turn lengths in the source and target side.

102

Europarl SubtitlesEn-Fr En-Et En-De En-Ru

# Conversations 6997 4394 3582 23126# Sentences 246540 174218 109241 291516

Mean Statistics per Conversation# Sentences 36.24 40.65 31.50 13.60# Turns 4.77 4.85 4.79 7.12Turn Length 7.12 7.92 6.16 1.68

Table 1: General statistics for training set.

if the language tag is absent, the source languageis English, otherwise Foreign. The sentences inthe source-side of the corpus are kept or swappedwith those in the target-side based on this tag.

We perform the aforementioned steps forEnglish-French, English-Estonian and English-German, and obtain the bilingual multi-speakercorpora for the three language pairs. Beforesplitting into train/dev/test sets, we remove con-versations with sentences having more than 100tokens for English-French, English-German andmore than 80 tokens for English-Estonian4 respec-tively, to limit the sentence-length for using sub-words with BPE (Sennrich et al., 2016). The datastatistics are given in Table 1 and Appendix A5.

Subtitles There has been recent work to obtainspeaker labels via automatic turn segmentation forthe OpenSubtitles2016 corpus (Lison and Meena,2016; van der Wees et al., 2016; Wang et al.,2016). We obtain the English side of OpenSub-titles2016 corpus annotated with speaker informa-tion by Lison and Meena (2016).6 To obtain theparallel corpus, we use the OpenSubtitles align-ment links to align foreign subtitles to the anno-tated English ones. For each subtitle, we extractindividual conversations with more than 5 sen-tences and at least two turns. Conversations withmore than 30 turns are discarded. Finally, sincesubtitles are in a single language, we assign lan-guage tag such that the same language occurs inalternating turns. We thus obtain the Bi-MSMTcorpus for English-Russian, which is then divided

4Sentence-lengths of 100 tokens result in longer sentencesthan what we get for the other two language-pairs.

5Although the extracted dataset is small but we believeit to be a realistic setting for a real-world conversation task,where reference translations are usually not readily availableand expensive to obtain.

6The majority of sentences still have missing annotations(Lison and Meena, 2016) due to changes between the originalscript and the actual movie or alignment problems betweenscripts and subtitles. As for Wang et al. (2016), their publiclyreleased data is even smaller than our En-De dataset extractedfrom Europarl.

into training, development and test sets.

3.3 Sentence-based attentional model

Our base model consists of two sentence-basedNMT architectures (Bahdanau et al., 2015), onefor each translation direction. Each of them con-tains an encoder to read the source sentence andan attentional decoder to generate the target trans-lation one token at a time.

Encoder It maps each source word xm to adistributed representation hm which is the con-catenation of the corresponding hidden states oftwo RNNs running in opposite directions overthe source sentence. The forward and backwardRNNs are taken to be GRUs (gated-recurrent unit;Cho et al. (2014)) in this work.

Decoder The generation of each target wordyn is conditioned on all the previously generatedwords y<n via the state sn of the decoder, and thesource sentence via a dynamic context vector cn:

yn ∼ softmax(Wy · un + by)

un = tanh(sn +Wuc · cn +Wun ·ET [yn−1])

sn = GRU(sn−1,ET [yn−1], cn)

whereET [yn−1] is the embedding of previous tar-get word yn−1, and {W(·),by} are the parameters.The fixed-length dynamic context representationof the source sentence cn =

∑m αnmhm is gen-

erated by an attention mechanism where α spec-ifies the proportion of relevant information fromeach word in the source sentence.

4 Conversational Bi-MSMT Model

Before we delve into the details of how to lever-age the conversation history, we identify the threetypes of context we may encounter in an ongoingbilingual multi-speaker conversation, as shown inFigure 1. It comprises of: (i) the previously com-pleted English turns, (ii) the previously completedForeign turns, and (iii) the ongoing turn (Englishor Foreign).

We propose a conversational Bi-MSMT modelthat is able to incorporate all three types ofcontext using source, target or dual conversa-tion histories into the base model. The basemodel caters to the speaker’s language transitionby having one sentence-based NMT model (de-scribed previously) for each translation direction,English→Foreign and Foreign→English. We now

103

Figure 1: Overview of an ongoing conversation whiletranslating ith sentence in 2k + 1th turn. Xj

|tj | and

Yj|tj | denote the sentences in previous English and For-

eign turn respectively, and xji denotes the sentence i

in ongoing turn j where i ∈ {1, ..., |tj |}. The shadedturns are observed i.e., source (the speaker utterances),while the rest are unobserved i.e., the target translationsor the unuttered source sentences for current turn.

describe our approach for extracting relevant in-formation from the source and target bilingualconversation history.

4.1 Source-Side HistorySuppose we are translating an ongoing conversa-tion having alternating turns of English and For-eign. We are currently in the 2k + 1th turn (in En-glish) and want to translate its ith sentence usingthe source-side conversation history representedby context vector osrc (dimensions H).

Let’s assume that we already have the represen-tations of previous source sentences in the con-versation. We pass the source sentence represen-tations through Turn-RNNs, which are composedof language-specific bidirectional RNNs irrespec-tive of the speaker, as shown in Figure 2, and con-catenate the last hidden states of the forward andbackward Turn-RNNs to get the final turn repre-sentation rj , where j denotes the turn index. Theindividual turn representations are then combined,based on language7, to obtain context vectors oenand ofr, computed in several possible ways (de-scribed below), which are further amalgamated us-

7For this work, we define the turns based on language anddo not use the speaker information as for real-world chat sce-narios (e.g., agent-client in a customer service chat), we donot have multiple speakers based on language. We leave thisfor future exploration.

Figure 2: Architectural overview when translating ith

sentence in 2k + 1th turn using source history.

ing a gating mechanism so as to give differing im-portance to each element of the context vector:

oen,fr = α� oen + (1−α)� ofr (1)

α = σ(Uen × oen +Ufr × ofr + bg)

where σ is the logistic sigmoid function, U’s arematrices and bg is a vector. Finally, we perform adimensionality reduction to obtain:

osrc = tanh(WT × oen,fr + bT ) (2)

In the remainder of this section, {W,U,b} arelanguage-specific learned parameters. We proposefive ways of computing the language-specific con-text representations, oen and ofr.

Direct Transformation The simplest approachis to combine turn representations using alanguage-specific dimensionality reduction trans-formation:

oen = tanh([Wen; ...;Wen]× [r1; ...; r2k+1] + ben)

ofr = tanh([Wfr; ...;Wfr]× [r2; ...; r2k] + bfr)

Here rj’s are concatenated row-wise.

104

Hierarchical Gating We propose a language-specific exponential decay gating based on the in-tuition that the farther the previous turns are fromthe current one, the lesser their impact may be onthe translation of a sentence in an ongoing turn,similar in spirit to the caching mechanism by Tuet al. (2017):

oen = gen(gen(...gen(gen(r1, r3), r5)...), r2k−1), r2k+1)

wheregen(a,b) = α� a+ (1−α)� b

α = σ(U1,en × a+U2,en × b+ ben)

ofr is computed in a similar way.

Language-Specific Attention The English andForeign turn representations are combined sepa-rately via attention to allow the model to focus onrelevant turns in the English and the Foreign con-text:

pen = softmax([r1; ...; r2k+1]T × hi) (3)

pfr = softmax([r2; ...; r2k]T × tanh(Wen × hi + ben))

oen = tanh(Wen × ([r1; ...; r2k+1]× pen) + ben)

ofr = [r2; ...; r2k]× pfr

Here rj’s are concatenated column-wise, hi is theconcatenation of last hidden state of forward andbackward RNNs in the encoder for current sen-tence i in turn 2k+1 (dimensions 2H) and {Wen,ben} transform the language space to that of thetarget language.

Combined Attention This is a language-independent attention that merges all turnrepresentations into one. The hypothesis hereis to verify if the model actually benefits fromLanguage-Specific attention or not.

pen,fr = softmax([r1,en; r2; ...; r2k+1,en]T ×

tanh(Wen × hi + ben))

oen,fr = [r1,en; r2; ...; r2k+1,en]× pen,fr

Here r2k+1,en = tanh(Wen × r2k+1 + ben).

Language-Specific Sentence-level AttentionAll the previous approaches for computing oenand ofr use a single turn-level representation.We propose to use the sentence informationexplicitly via a sentence-level attention to evaluatethe significance of more fine-grained context incontrast to Language-Specific Attention. Wefirst concatenate the hidden states of forwardand backward Turn-RNNs for each sentence and

get a matrix comprising of representations of allthe previous source sentences, i.e., for Englishturns, we have [r11; ...; r

1|t1|; ...; r

2k+11 ; ...; r2k+1

i−1 ],and similarly we have another matrix for all theprevious Foreign sentences. Here, each rji isthe representation of source sentence i in turn jcomputed by the bidirectional Turn-RNN. Theremaining computations are same as in Eq. 3.

4.2 Target-Side HistoryUsing target-side conversation history is as impor-tant as that of the source-side since it helps in mak-ing the translation more faithful to the target lan-guage. This becomes crucial for translating con-versations where the previous turns are all in thesame language. For incorporating the target-sidecontext, we use a sentence-level attention simi-lar to the one described for the source-side con-text, i.e., for all previous English source sentences,we have a matrix Ren comprising of the corre-sponding target sentence representations in For-eign, and another matrix Rfr of target sentencerepresentations (in English) for previous Foreignturns. Here each target sentence representation hasdimensions H. Then,

pen = softmax(RTen × tanh(Wt,en × hi + bt,en))

pfr = softmax(RTfr × (Wtd,en × hi + btd,en))

oen = Ren × pen

ofr = tanh(Wt,en × (Rfr × pfr) + bt,en)

where {Wt,en,bt,en} are for dimensionality re-duction and changing the language space of thequery vector hi and the context vector, while{Wtd,en,btd,en} are only for dimensionality re-duction. oen and ofr are further combined usinga gating mechanism as in Eq. 1 to obtain the finaltarget context vector otgt (dimensions H).

4.3 Dual Conversation HistoryNow that we have explained how to leverage thesource and target conversation history separately,we explain how they can be utilised simultane-ously. The simplest way to do this is to incorporateboth context vectors osrc and otgt into the basemodel (explained in Sec 4.4), referred as Src-Tgtdual context.

Another intuitive approach, as evident from Fig-ure 2, is to separately model English and For-eign sentences using two separate context vectorsoen,m and ofr,m, where each is constructed froma mixture of the original source or target trans-lations, is language-specific and possibly contain

105

less noise. We refer to this as the Src-Tgt-Mixdual context. Suppose Ren,m contains the mixedsource/target representations for English (the di-mensions for source representations have been re-duced to H) and Rfr,m contains the same for For-eign. Then,

pen,m = softmax(RTen,m × (Wtd,en × hi + btd,en))

pfr,m = softmax(RTfr,m × tanh(Wtt,en × hi + btt,en))

oen,m = tanh(Wtr,en × (Ren,m × pen,m) + btr,en)

ofr,m = Rfr,m × pfr,m

where Wtd,en, Wtr,en and Wtt,en are for dimen-sionality reduction, changing the language spaceand both, respectively.

4.4 Incorporating Context into Base ModelThe final representations osrc and otgt or oen,mand ofr,m, can be incorporated together or indi-vidually in the base model by:

• InitDec Using a non-linear transformation toinitialise the decoder, similar to Wang et al.(2017): si,0 = tanh(V×oi+bs), where i isthe sentence index in current turn 2k+1, {V,bs} are encoder-decoder specific parametersand oi is either a single context vector or aconcatenation (transformed) of the two.

• AddDec As an auxiliary input to the decoder(similar to Jean et al. (2017); Wang et al.(2017); Maruf and Haffari (2018)):

si,n = tanh(Ws · si,n−1 +Wsn ·ET [yi,n] +

Wsc · ci,n +Wss · oi,src +Wst · oi,tgt)

• InitDec+AddDec Combination of previoustwo approaches.

4.5 Training and DecodingThe model parameters are trained end-to-end bymaximising the sum of log-likelihood of the bilin-gual conversations in training set D. For example,for a conversation having alternating turns of En-glish and Foreign language, the log-likelihood is:|T |2−1∑

k=0

(|t2k+1|∑

i=1

logPθ(yi|xi,oi) +

|t2k+2|∑

j=1

logPθ(xj |yj ,oj))

where i, j denote sentences belonging to 2k + 1th

or 2k + 2th turn; o(.) is a representation of theconversation history, and |T | is the total numberof turns (assumed to be even here).

The best output sequence for a given input se-quence for the ith sentence at test time, a.k.a. de-coding, is produced by:

argmaxyi

Pθ(yi|xi,oi)

5 Experiments

Implementation and Hyperparameters Weimplement our conversational Bi-MSMT model inC++ using the DyNet library (Neubig et al., 2017).The base model is built using mantis (Cohnet al., 2016) which is an implementation of thegeneric sentence-level NMT model using DyNet.

The base model has single layer bidirectionalGRUs in the encoder and 2-layer GRU in the de-coder8. The hidden dimensions and word embed-ding sizes are set to 256, and the alignment dimen-sion (for the attention mechanism in the decoder)is set to 128.

Models and Training We do a stage-wisetraining for the base model, i.e., we firsttrain the English→Foreign architecture andthe Foreign→English architecture, using thesentence-level parallel corpus. Both architectureshave the same vocabulary9 but separate parame-ters to avoid biasing the embeddings towards thearchitecture trained last. The contextual model ispre-trained similar to training the base model. Thebest model is chosen based on minimum overallperplexity on the bilingual dev set.

For the source context representations, we usethe sentence representations generated by twosentence-level bidirectional RNNLMs (one eachfor English and Foreign) trained offline. For thetarget sentence representations, we use the lasthidden states of the decoder generated from thepre-trained base model10. At decoding time, how-ever, we use the last hidden state of the decodercomputed by our model (not the base) as the tar-get sentence representations. Further training de-tails are provided in Appendix B.

8We follow Cohn et al. (2016) and Britz et al. (2017) inchoosing hyperparameters for our model.

9For each language-pair, we use BPE (Sennrich et al.,2016) to obtain a joint vocabulary of size ≈30k.

10Even though the paramaters of the base model are up-dated, the target sentence representations are fixed throughouttraining. We experimented with a scheduled updating schemein preliminary experiments but it did not yield significant im-provement.

106

Europarl SubtitlesEn-Fr En-Et En-De En-Ru

OverallEn→FrFr→EnOverallEn→EtEt→EnOverallEn→DeDe→En OverallEn→RuRu→EnBase Model 37.36 38.13 36.03 20.68 18.64 26.65 24.74 21.80 27.74 19.05 14.90 23.04

+Source Context as Lang-Specific Attention viaInitDec 38.40† 39.19† 36.86† 21.79† 19.54† 28.33† 26.34† 23.31† 29.39† 18.88 14.89 22.56AddDec 38.50† 39.35† 36.98† 21.65† 19.66† 27.48† 26.30† 23.09† 29.52† 19.34 15.16 23.12InitDec+AddDec 38.55† 39.34† 37.14† 21.49† 19.43† 27.55† 26.25† 23.18† 29.30† 19.35 15.16 23.14

+Source Context viaDirect Tranformation 38.35† 39.13† 36.96† 21.75† 19.59† 28.07† 26.29† 23.34† 29.22† 19.09 14.89 22.76Hierarchical Gating 38.33† 39.14† 36.89† 21.62† 19.55† 27.64† 26.31† 23.17† 29.45† 19.20 15.10 22.73Lang-Specific Attention 38.40† 39.19† 36.86† 21.79† 19.54† 28.33† 26.34† 23.31† 29.39† 19.35 15.16 23.14Combined Attention 38.50† 39.36† 36.94† 21.66† 19.52† 27.90† 26.38† 23.31† 29.44† 18.96 14.82 22.92Lang-Specific S-Attention 38.46† 39.24† 37.06† 21.84† 19.58† 28.43† 26.49† 23.49† 29.49† 19.09 14.59 22.98

+Lang-Specific S-Attention usingSource Context 38.46† 39.24† 37.06† 21.84† 19.58† 28.43† 26.49† 23.49† 29.49† 19.09 14.59 22.98Target Context 38.76† 39.57† 37.35† 21.77† 19.68† 27.86† 26.21† 23.16† 29.26† 19.23 14.77 23.23Dual Context Src-Tgt 38.80† 39.51† 37.50† 21.74† 19.60† 27.98† 26.39† 23.28† 29.50† 18.89 14.52 23.06Dual Context Src-Tgt-Mix 38.76† 39.52† 37.43† 21.68† 19.63† 27.71† 26.37† 23.26† 29.48† 19.26 14.86 23.01

Table 2: BLEU scores for the bilingual test sets. Here all contexts are incorporated as InitDec for Europarl andInitDec+AddDec for Subtitles unless otherwise specified. bold: Best performance, †: Statistically significantlybetter than the base model, based on bootstrap resampling (Clark et al., 2011) with p < 0.05.

5.1 Results

Firstly, we evaluate the three strategies for in-corporating context: InitDec, AddDec, Init-Dec+AddDec, and report the results for sourcecontext using Language-Specific Attention in Ta-ble 2. For the Europarl data, we see de-cent improvements with InitDec for En-Et (+1.11BLEU) and En-De (+1.60 BLEU), and with Init-Dec+AddDec for En-Fr (+1.19 BLEU). We alsoobserve that, for all language-pairs, both transla-tion directions benefit from context, showing thatour training methodology was indeed effective.On the other hand, for the Subtitles data, we seea maximum improvement of +0.30 BLEU for Init-Dec+AddDec . We narrow down to three majorreasons: (i) the data is noisier when compared toEuroparl, (ii) the sentences are short and genericwith only 1% having more than 27 tokens, and fi-nally (iii) the turns in OpenSubtitles2016 are shortcompared to those in Europarl (see Table 1), andwe show later (Section 5.2) that the context fromcurrent turn is the most important.

The next set of experiments evaluates the fivedifferent approaches for computing the source-side context. It is evident from Table 2that for English-Estonian and English-German,our model indeed benefits from using the fine-grained sentence-level information (Language-Specific Sentence-level Attention) as opposed to

just the turn-level one.Finally, our results with source, target and dual

contexts are reported. Interestingly, just using thesource context is sufficient for English-Estonianand English-German. For English-French, on theother hand, we see significant improvements forthe models using the target-side conversation his-tory over using only the source-side. We attributethis to the base model being more efficient andable to generate better translations for En-Fr as ithad been trained on a larger corpus as opposed tothe other two language-pairs. Unlike Europarl, forSubtitles, we see improvements for our Src-Tgt-Mix dual context variant over the Src-Tgt one forEn→Ru, showing this to be an effective approachwhen the target representations are noisier.

To summarise, for majority of the cases ourLanguage-Specific Sentence-level Attention is awinner or a close second. Using the Target Con-text is useful when the base model generatesreasonable-quality translations; otherwise, usingthe Source Context should suffice.

Local Source Context Model Most of the pre-vious works for online context-based NMT con-sider only a single previous sentence as context(Jean et al., 2017; Bawden et al., 2017; Voita et al.,2018). Drawing inspiration from these works,we evaluate our model (trained with Language-Specific Sentence-Level Attention) on the same

107

Europarl SubtitlesEn-Fr En-Et En-De En-Ru

Prev Sent 38.15 21.70 26.09 19.13Our Model 38.46† 21.84 26.49† 19.09

Table 3: BLEU scores for the bilingual test sets. bold:Best performance, †: Statistically significantly betterthan the contextual baseline.

Type of Context BLEUNo context (Base Model) 24.74Current Turn 26.39Current Language from Previous Turns 26.21Other Language from Previous Turns 26.32Complete Context 26.49

Table 4: BLEU scores for En-De bilingual test set.

I II III24

26

28

30

32

34

24.74

28.23

29.75

26.34

29.35

31.23

BL

EU

Base MT BaseMT+SrcContext

Figure 3: BLEU scores on En-De test set while train-ing (I) smaller base model with smaller corpus (pre-vious experiment), (II) smaller base model with largercorpus, and (III) a larger base model with larger corpus.

test set but using only the previous source sentenceas context. This evaluation allows us to hypothe-sise how much of the gain can be attributed to theprevious sentence. From Table 3, it can be seenthat our model surpasses the local-context base-line for Europarl showing that the wider context isindeed beneficial if the turn lengths are longer. ForEn-Ru, it can be seen that using previous sentenceis sufficient due to short turns (see Table 1).

5.2 Analysis

Ablation Study We conduct an ablation studyto validate our hypothesis of using the completecontext versus using only one of the three typesof contexts in a bilingual multi-speaker conversa-tion: (i) current turn, (ii) previous turns in currentlanguage, and (iii) previous turns in the other lan-guage. The results for En-De are reported in Ta-ble 4. We see decrease in BLEU for all types ofcontexts with significant decrease when consider-ing only current language from previous turns.Theresults show that the current turn has the most in-fluence on translating a sentence, and we conclude

En→Fr les; par; est; a; dans; le; en; j’; un; afin; question;entre; qu’; etre; ces; egalement; y; depuis; c’; ou

Fr→En this; of; we; issue; europe; by; up; make; united;does; what; regard; s; must; however; such; whose;share; like; been

En→Et eest; vahel; ule; nimel; ja; aastal; aasta; neid; ainultseeparast; nagu; kes; komisjoni; tehtud; kusimuses;sisserande; liikmesriigi; mulla; liibanoni; dawit

Et→En for; this; of; is; political; important; culture; also; as;order; are; each; their; only; gender; were; its;economy; one; market

En→Dedaß; auf; und; werden; nicht; mussen; aus; mehr;konnen; einem; rates; eines; insbesondere; wurden;habe; mitgliedstaaten; ist; sondern; europa;gemeinsamen

De→Enthat; its; say; must; some; therefore; more; countries;an; favour; public; will; without; particularly;hankiss; much; increase; eu; them; parliamentary

Table 5: Most frequent tokens correctly generated byour model when compared to the base model.

that since our model is able to capture the com-plete context, it is generalisable to any conversa-tional scenario.

Training base model with more data To anal-yse if the context is beneficial even when usingmore data, we perform an experiment for English-German where we train the base model with addi-tional sentence-pairs from the full WMT’14 cor-pus11 (excluding our dev/test sets and filtering sen-tences with more than 100 tokens). For train-ing the contextual model, we still use the bilin-gual multi-speaker corpus. We observe a sig-nificant improvement of +1.12 for the context-based model (Figure 3 II), showing the signifi-cance of conversation history in this experimentcondition.12

We perform another experiment where we use alarger base model, having almost double the num-ber of parameters than our previous base model(hidden units and word embedding sizes set to512, and alignment dimension set to 256), totest if the model parameters are being overesti-mated due to the additional context. We use thesame WMT’14 corpus to train the base model andachieve significant improvement of +1.48 BLEUfor our context-based model over the larger base-line (Figure 3 III).

11https://nlp.stanford.edu/projects/nmt/12It should be noted that the BLEU score for the base

model trained with WMT does not match the published re-sults exactly as the test set contains both English and Germansentences. It does, however, fall between the scores usuallyobtained on WMT’14 for En→De and De→En.

108

Context nous sommes egalement favorables au principe d’un systeme de collecte des miles commun pour le parlementeuropeen, pour que celui-ci puisse beneficier de billets d’avion moins chers, meme si nous voyons difficilementcomment ce systeme pourrait etre deploye en pratique.enfin, nous ne sommes pas opposes a l’attribution de prix culturels par le parlement europeen.

Source neanmoins, nous sommes particulierement critiques a l’egard du prix pour le journalisme du parlement europeenet nous ne pensons pas que celui-ci puisse decerner des prix aux journalistes ayant pour mission de soumettre leparlement europeen a un regard critique.

Target however, we are highly critical of parliament’s prize for journalism, and do not believe that it is appropriate forparliament to award prizes to journalists whose task it is to critically examine the european parliament.

Base Modelnevertheless, we are particularly critical of the price for the european union’s european alism and we do notbelieve that it would be able to make a price to the journalists who have been made available to the europeanparliament to a critical view.

Our Model however, we are particularly critical of the price for the european union’s democratic alism and we do not believethat it can give rise to the prices for journalists who have been tabled to submit the european parliament to acritical view.

Table 6: Example En-Fr sentence translation showing how the context helps our model in generating the appropri-ate discourse connective.

Context oleks hea, kui reitinguagentuurid vastutaksid tulevikus enda tegevuse eest rohkem....kirjalikult. - (it) kiites heaks wolf klinzi raporti, mille eesmark on reitinguagentuuride tohus reguleerimine,votab parlament jarjekordse sammu finantsturgude suurema labipaistvuse suunas....mul oli selle dokumendi ule hea meel, sest krediidireitingute valdkonnal on palju probleeme, millest koigesuuremad on oligopolidele tupilised struktuurid ning konkurentsi, vastutuse ja labipaistvuse puudumine.

Source selles suhtes tuleb rohutada nende tegevuse suuremal abipaistvuse pohirolli.Target in this respect, it is necessary to highlight the central role of increased transparency in their activities.Base Model in this regard it must be emphasised in the major role of transparency in which these activities are to be

strengthened.Our Model in this regard, it must be stressed in the key role of greater transparency in their activities.

Table 7: Example En-Et translation showing how the wide-range context helps in generating the correct pronoun.The antecedent and correct pronoun are highlighted in blue.

Figure 4: Density of token counts for En→Fr illustrat-ing where our model is better (+ve x-axis) and wherethe base model is better (-ve x-axis).

How is the context helping? The underlyinghypothesis for this work is that discourse phe-nomenon in a conversation may depend on long-range dependency and these may be ignored bythe sentence-based NMT models. To analyse ifour contextual model is able to accurately translatesuch linguistic phenomenon, we come up with ourown evaluation procedure. We aggregate the to-

kens correctly generated by our model and thosecorrectly generated by the baseline over the entiretest set. We then take the difference of these countsand sort them13. Table 5 reports the top 20 tokenswhere our model is better than the baseline for theEuroparl dataset. Figure 4 gives the density ofcounts obtained using our evaluation for En→Fr14.Positive counts correspond to correct translationsby our model while the negative counts correspondto where the base model was better. It can be seenthat for majority of cases our model supersedesthe base model. We observed a similar trend forother translation directions. In general, the cor-rectly generated tokens by our model include pro-nouns (that, this, its, their, them), discourse con-nectives (e.g., ‘however’, ‘therefore’, ‘also’) andprepositions (of, for, by).

Table 6 reports an example where our model isable to generate the correct discourse connective‘however’ using the context. If we look at the con-

13We do not normalise the counts with the background fre-quency as it favours rare words. Thus, obscuring the mainreasons of improving the BLEU score.

14Outliers and tokens with equal counts for our model andthe baseline were removed.

109

Figure 5: Attention map when translating a conversa-tion from the Et-En test set.

text of the source sentence in French, we come tothe conclusion that ‘however’ is indeed a perfectfit in this case, whereas the base model is at a dis-advantage and completely changes the underlyingmeaning of the sentence by generating the inap-propriate connective ‘nevertheless’.

Table 7 gives an instance where our model isable to generate the correct pronoun ‘their’. Itshould be noted that in this case, the current sourcesentence does not contain the antecedent and thusthe context-free baseline is unable to generate theappropriate pronoun. On the other hand, our con-textual model is able to do so by giving the high-est attention weights to sentences containing theantecedent (observed from the attention map inFigure 5)15. Figure 5 also shows that for trans-lating majority of the sentences, the model attendsto wide-range context rather than just the previoussentence, hence strengthening the premise of us-ing the complete context.

6 Conclusion

This work investigates the challenges associatedwith translating multilingual multi-speaker con-versations by exploring a simpler task referredto as Bilingual Multi-Speaker Conversation MT.We process Europarl v7 and OpenSubtitles2016to obtain an introductory dataset for this task.Compared to models developed for similar tasks,our work is different in two aspects: (i) the his-tory captured by our model contains multiple lan-guages, and (ii) our model captures ‘global’ his-tory as opposed to ‘local’ history captured in mostprevious works. Our experiments demonstrate the

15For this particular conversation, all previous turns werein Estonian.

significance of leveraging the bilingual conversa-tion history in such scenarios. Furthermore, theanalysis shows that using wide-range context, ourmodel generates appropriate pronouns and dis-course connectives in some cases. We hope thiswork to be a first step towards translating multilin-gual multi-speaker conversations. Future work onthis task may include optimising the base transla-tion model and approaches that condition on spe-cific discourse information in the conversation his-tory.

Acknowledgments

We thank the reviewers for their useful feed-back that helped us in improving our work.We also thank Pierre Lison for providing uswith the speaker-annotated OpenSubtitles2016dataset. This work is supported by theMulti-modal Australian ScienceS Imaging andVisualisation Environment (MASSIVE) (www.massive.org.au), the European ResearchCouncil (ERC StG DeepSPIN 758969), theFundacao para a Ciencia e Tecnologia throughcontract UID/EEA/50008/2013, a Google FacultyResearch Award to GH and the Australian Re-search Council through DP160102686.

ReferencesParnia Bahar, Tamer Alkhouli, Jan-Thorsten Peter,

Christopher Jan-Steffen Brix, and Hermann Ney.2017. Empirical investigation of optimization algo-rithms in neural machine translation. In Conferenceof the European Association for Machine Transla-tion, pages 13–26, Prague, Czech Republic.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. In Proceedings ofthe International Conference on Learning Represen-tations.

Rachel Bawden, Rico Sennrich, Alexandra Birch, andBarry Haddow. 2017. Evaluating discourse phe-nomena in neural machine translation. In Proceed-ings of NAACL–HLT 2018.

Denny Britz, Anna Goldie, Minh-Thang Luong, andQuoc Le. 2017. Massive exploration of neuralmachine translation architectures. In Proceedingsof the Conference on Empirical Methods in Natu-ral Language Processing. Association for Computa-tional Linguistics.

Kyunghyun Cho, B van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the properties

110

of neural machine translation: Encoder-decoder ap-proaches. In Eighth Workshop on Syntax, Semanticsand Structure in Statistical Translation (SSST-8).

Jonathan H. Clark, Chris Dyer, Alon Lavie, andNoah A. Smith. 2011. Better hypothesis testing forstatistical machine translation: Controlling for op-timizer instability. In Proceedings of the 49th An-nual Meeting of the Association for ComputationalLinguistics: Human Language Technologies (ShortPapers), pages 176–181. Association for Computa-tional Linguistics.

Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vy-molova, Kaisheng Yao, Chris Dyer, and GholamrezaHaffari. 2016. Incorporating structural alignmentbiases into an attentional neural translation model.In Proceedings of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies, pages 876–885. Asso-ciation for Computational Linguistics.

Eva Martınez Garcia, Carles Creus, Cristina Espana-Bonet, and Lluıs Marquez. 2017. Using word em-beddings to enforce document-level lexical consis-tency in machine translation. The Prague Bulletinof Mathematical Linguistics, 108:85–96.

Eva Martınez Garcia, Cristina Espana-Bonet, and LluısMarquez. 2014. Document-level machine transla-tion as a re-translation process. Procesamiento delLenguaje Natural, 53:103–110.

Zhengxian Gong, Min Zhang, and Guodong Zhou.2011. Cache-based document-level statistical ma-chine translation. In Proceedings of the Conferenceon Empirical Methods in Natural Language Pro-cessing, pages 909–919. Association for Computa-tional Linguistics.

Christian Hardmeier and Marcello Federico. 2010.Modelling pronominal anaphora in statistical ma-chine translation. In International Workshop onSpoken Language Translation, pages 283–289.

Christian Hardmeier, Joakim Nivre, and Jorg Tiede-mann. 2012. Document-wide decoding for phrase-based statistical machine translation. In Proceed-ings of the 2012 Joint Conference on Empiri-cal Methods in Natural Language Processing andComputational Natural Language Learning, pages1179–1190. Association for Computational Linguis-tics.

Cong Duy Vu Hoang, Trevor Cohn, and GholamrezaHaffari. 2016. Incorporating side information intorecurrent neural network language models. In Pro-ceedings of NAACL–HLT 2016, pages 1250–1255.

Sebastien Jean, Stanislas Lauly, Orhan Firat, andKyunghyun Cho. 2017. Does neural machinetranslation benefit from larger context? InarXiv:1704.05135.

Gang Ji and Jeff Bilmes. 2004. Multi-speaker languagemodeling. In Proceedings of HLT–NAACL 2004.

Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer,and Jacob Eisenstein. 2015. Document context lan-guage models. In Workshop track - ICLR 2016.

Philipp Koehn. 2005. Europarl: A parallel corpus forstatistical machine translation. In Conference Pro-ceedings: the 10th Machine Translation Summit,pages 79–86. AAMT.

Shaohui Kuang, Deyi Xiong, Weihua Luo, andGuodong Zhou. 2018. Modeling coherence forneural machine translation with dynamic and topiccaches. COLING 2018.

Pierre Lison and Raveesh Meena. 2016. Automaticturn segmentation of movie & tv subtitles. In Pro-ceedings of the 2016 Spoken Language TechnologyWorkshop, pages 245–252, San Diego, CA, USA.IEEE.

Pierre Lison and Jorg Tiedemann. 2016. OpenSub-titles2016: Extracting large parallel corpora fromMovie and TV subtitles. In Proceedings of the 10th

International Conference on Language Resourcesand Evaluation (LREC’16), pages 923–929.

Sameen Maruf and Gholamreza Haffari. 2018. Docu-ment context neural machine translation with mem-ory networks. In Proceedings of the 56th AnnualMeeting of the Association for Computational Lin-guistics. Association for Computational Linguistics.

Graham Neubig, Chris Dyer, Yoav Goldberg, AustinMatthews, Waleed Ammar, Antonios Anastasopou-los, Miguel Ballesteros, David Chiang, DanielClothiaux, Trevor Cohn, Kevin Duh, ManaalFaruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji,Lingpeng Kong, Adhiguna Kuncoro, Gaurav Ku-mar, Chaitanya Malaviya, Paul Michel, YusukeOda, Matthew Richardson, Naomi Saphra, SwabhaSwayamdipta, and Pengcheng Yin. 2017. Dynet:The dynamic neural network toolkit. arXiv preprintarXiv:1701.03980.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Neural machine translation of rare words withsubword units. In Proceedings of the 54th AnnualMeeting of the Association for Computational Lin-guistics, pages 1715–1725.

Quan Hung Tran, Ingrid Zuckerman, and GholamrezaHaffari. 2016. Inter-document contextual languagemodel. In Proceedings of NAACL–HLT 2016, pages762–766.

Zhaopeng Tu, Yang Liu, Shuming Shi, and TongZhang. 2017. Learning to remember translation his-tory with a continuous cache.

Elena Voita, Pavel Serdyukov, Rico Sennrich, and IvanTitov. 2018. Context-aware neural machine trans-lation learns anaphora resolution. In Proceedingsof the 56th Annual Meeting of the Association forComputational Linguistics. Association for Compu-tational Linguistics.

111

Longyue Wang, Zhaopeng Tu, Andy Way, and QunLiu. 2017. Exploiting cross-sentence context forneural machine translation. In Proceedings of theConference on Empirical Methods in Natural Lan-guage Processing, pages 2816–2821. Associationfor Computational Linguistics.

Longyue Wang, Xiaojun Zhang, Zhaopeng Tu, AndyWay, and Qun Liu. 2016. Automatic construction ofdiscourse corpora for dialogue translation. In Pro-ceedings of the Tenth International Conference onLanguage Resources and Evaluation (LREC 2016),Paris, France. European Language Resources Asso-ciation (ELRA).

Marlies van der Wees, Arianna Bisazza, and ChristofMonz. 2016. Measuring the effect of conversationalaspects on machine translation quality. In Proceed-ings of COLING 2016, pages 2571–2581.

A Data Statistics

Europarl SubtitlesEn-Fr En-Et En-De En-Ru

Dev/Test# Conversations 140/209 88/132 70/108 462/694

# Sentences 4.9k/7.8k 3.2k/5.2k 2.1k/3.3k 5.9k/9k

Table 8: General statistics for development and testsets.

B Experiments

Training For the base model, we make use ofstochastic gradient descent (SGD)16 with initiallearning rate of 0.1 and a decay factor of 0.5 af-ter the fifth epoch for a total of 15 epochs. Forthe contextual model, we use SGD with an initiallearning rate of 0.08 and a decay factor of 0.9 afterthe first epoch for a total of 30 epochs. To avoidoverfitting, we employ dropout and set its rate to0.2. To reduce the training time of our contextualmodel, we perform computation of one turn at atime, for instance, when using the source context,we run the Turn-RNNs for previous turns once andre-run the Turn-RNN only for sentences in the cur-rent turn.

16In our preliminary experiments, we tried SGD, Adamand Adagrad as optimisers, and found SGD to achieve betterperplexities in lesser number of epochs (Bahar et al., 2017).

112