Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5759–5771July 5 - 10, 2020. c©2020 Association for Computational Linguistics

5759

Cross-Lingual Unsupervised Sentiment Classification withMulti-View Transfer Learning

Hongliang Fei, Ping LiCognitive Computing Lab

Baidu Research1195 Bordeaux Dr, Sunnyvale, CA 94089, USA

10900 NE 8th St, Bellevue, WA 98004, USA{hongliangfei,liping11}@baidu.com

AbstractRecent neural network models have achievedimpressive performance on sentiment classifi-cation in English as well as other languages.Their success heavily depends on the availabil-ity of a large amount of labeled data or paral-lel corpus. In this paper, we investigate an ex-treme scenario of cross-lingual sentiment clas-sification, in which the low-resource languagedoes not have any labels or parallel corpus.We propose an unsupervised cross-lingual sen-timent classification model named multi-viewencoder-classifier (MVEC) that leverages anunsupervised machine translation (UMT) sys-tem and a language discriminator. Unlike pre-vious language model (LM) based fine-tuningapproaches that adjust parameters solely basedon the classification error on training data, weemploy the encoder-decoder framework of aUMT as a regularization component on theshared network parameters. In particular, thecross-lingual encoder of our model learns ashared representation, which is effective forboth reconstructing input sentences of twolanguages and generating more representativeviews from the input for classification. Exten-sive experiments on five language pairs verifythat our model significantly outperforms othermodels for 8/11 sentiment classification tasks.

1 Introduction

Recent neural network models have achieved re-markable performance on sentiment classificationin English and other languages (Conneau et al.,2017; Chen et al., 2018; He et al., 2019; Chenand Qian, 2019). However, their success heavilydepends on the availability of a large amount oflabeled data or parallel corpus. In reality, somelow-resource languages or applications have lim-ited labeled data or even without any labels or par-allel corpus, which may hinder us from training arobust and accurate sentiment classifier.

To build sentiment classification models forlow-resource languages, recent researchers de-veloped cross-lingual text classification (CLTC)models (Xu and Yang, 2017; Eriguchi et al., 2018),which transfers knowledge from a resource-rich(source) language to a low-resource (target) lan-guage. The core of those models is to learna shared language-invariant feature space that isindicative of classification for both languages.Therefore a model trained from the source lan-guage can be applied to the target language. Basedon how the shared feature space is learned, thereare three categories, namely word-level align-ments (Andrade et al., 2015), sentence-level align-ments (Eriguchi et al., 2018) and document levelalignments (Zhou et al., 2016). Those modelscan well capture the semantic similarity betweentwo languages. They, however, require parallelresources such as a bilingual dictionary, parallelsentences, and parallel Wikipedia articles. Sucha limitation may prevent these models from be-ing applicable in languages without any parallelresources.

Recently, there have been several attempts at de-veloping “zero-resource” models (Ziser and Re-ichart, 2018; Chen et al., 2018; Chen and Qian,2019). Most notably, Ziser and Reichart (2018)proposed a cross-lingual & cross-domain (CLCD)model that builds on pivot based learning andbilingual word embedding. Although CLCDdoes not directly need labeled data or parallelcorpus, it requires bilingual word embeddings(BWEs) (Smith et al., 2017) that requires thou-sands of translated words as a supervised signal.Chen et al. (2018) developed an adversarial deepaveraging network to learn latent sentence rep-resentations for classification, but it had an im-plicit dependency on BWEs (Zou et al., 2013)that requires pretraining on a large bilingual par-allel corpus. Chen and Qian (2019) extended the

5760

cross-lingual model in Chen et al. (2018) to mul-tiple source languages by using the unsupervisedBWEs (Lample et al., 2018b) and adding indi-vidual feature extractor for each source language,which eliminated the dependency on a parallelcorpus. Nevertheless, their model is very sensitiveto the quality of BWEs and performs poorly ondistant language pairs such as English-Japanese,as illustrated in their experimental study.

In parallel, cross-lingual language models(LMs) trained from raw Wikipedia texts, suchas multilingual BERT1 (Devlin et al., 2019) andXLM (Conneau and Lample, 2019), have beenprevalent in solving zero-shot classification prob-lems (Wu and Dredze, 2019). Those models usethe BERT-style Transformer (Vaswani et al., 2017)architecture simultaneously trained from multiplelanguages to construct a sentence encoder, andfine-tune the encoder and a classifier on labeledtraining data from the source language. Thenthe fine-tuned model is applied to the target lan-guage. The whole process does not require anylabeled data or parallel corpus. However, un-der the “zero parallel resource” setting, the en-coder trained from self-supervised masked lan-guage modelling within each language may notwell capture the semantic similarity among lan-guages, which could harm the generalization per-formance of fine-tuned models.

In this paper, we propose a sentiment classifi-cation model called multi-view encoder-classifier(MVEC) in an unsupervised setting, in whichwe only have monolingual corpora from two lan-guages and labels in the source language. Dif-ferent from previous language model (LM) basedfine-tuning approaches (Devlin et al., 2019; Con-neau and Lample, 2019) that adjust parameterssolely based on the classification error of trainingdata, we utilize the encoder-decoder network fromunsupervised machine translation (UMT) (Lampleet al., 2018a) to regularize and refine the sharedlatent space. In particular, the transformer-basedencoder regularized by a language discriminatorlearns shared but more refined language-invariantrepresentations, which are effective for both re-constructing sentences from two languages by thedecoder and generating multi-view feature repre-sentations for classification from input documents.In our model, we construct two views from the en-

1https://github.com/google-research/BERT/blob/master/multilingual.md

coder: (i) the encoded sentences in the source lan-guage; (ii) the encoded translations of the sourcesentences in the target language.

Our proposed MVEC is partially initialized bypretrained LMs (Conneau and Lample, 2019) butfurther fine-tuned to align sentences from two lan-guages better, accurately predict labeled data inthe source language and encourage consensus be-tween the predictions from the two views. The fullmodel is trained in an end-to-end manner to updateparameters for the encoder-decoder, the languagediscriminator, and the classifier at each iteration.

Our contributions in this paper are as follows:

• We present an unsupervised sentiment classifi-cation model without any labels or parallel re-source requirements for the target language. Bydesigning a multi-view classifier and integratingit with pretrained LMs and UMT (Lample et al.,2018a), we build our model (MVEC) on a morerefined latent space that is robust to languageshift with better model interpretation comparedto previous zero-shot classification works (Chenet al., 2018; Conneau and Lample, 2019).

• We extensively evaluate our model in 5 lan-guage pairs involving 11 sentiment classifica-tion tasks. Our full model outperforms state-of-the-art unsupervised fine-tuning approaches andpartially supervised approaches using cross-lingual resources in 8/11 tasks. Therefore, ourresults provide a strong lower bound perfor-mance on what future semi-supervised or super-vised approaches are expected to produce.

2 Related Work

2.1 Cross-Lingual Text Classification (CLTC)CLTC aims to learn a universal classifier thatcan be applied to languages with limited labeleddata (Bel et al., 2003; Dong and de Melo, 2019;Keung et al., 2019), which is naturally appli-cable for sentiment analysis. Traditional super-vised methods utilize cross-lingual tools such asmachine translation systems and train a classifieron the source language (Prettenhofer and Stein,2010). The latest models used parallel corpuseither to learn a bilingual document representa-tion (Zhou et al., 2016) or to conduct cross-lingualmodel distillation (Xu and Yang, 2017).

In the unsupervised setting, Chen et al. (2018)learned language-invariant latent cross-lingualrepresentations with adversarial training. Ziser

5761

and Reichart (2018) used pivot based learningand structure-aware DNN to transfer knowledgeto low-resourced languages. In both papers,however, they have an implicit dependency onBWEs, which requires a bilingual dictionary totrain. Chen and Qian (2019) was the first fullyunsupervised approach using the unsupervisedBWEs (Lample et al., 2018b) and multi-sourcelanguages with adversarial training. In contrast,our model is a multi-view classification model thatis seamlessly integrated pretrained LMs (Conneauand Lample, 2019) and the encoder-decoder fromUMT (Lample et al., 2018a) with adversarial train-ing. Hence we learn a more fine-tuned latent spaceto better capture document-level semantics andgenerate multiple views to represent the input.

2.2 Unsupervised Machine TranslationUMT does not rely on any parallel corpus toperform translation, which lays a foundation forour approach. At the word-level, Lample et al.(2018b) built a bilingual dictionary between twolanguages by aligning monolingual word embed-dings in an unsupervised way. At the sentenceand document level, Lample et al. (2018a) pro-posed a UMT model by learning an autoencoderthat can reconstruct two languages under bothwithin-domain and cross-domain settings. Lam-ple et al. (2018c) extended Lample et al. (2018a)with a phrase-based approach. Since we aim tolearn more refined language-invariant representa-tions for classification, it is natural to employ theencoder from a UMT system to generate multipleviews of the input and enable knowledge transfer.

2.3 Multi-View Transfer LearningThe task of multi-view transfer learning is to si-multaneously learn multiple representations andtransfer the learned knowledge from source do-mains to target domains, which have fewer train-ing samples. Generally, data from different viewscontains complementary information and multi-view learning exploits the consistency from mul-tiple views (Li et al., 2019).

Our work is particularly inspired by Fu et al.(2015) and Zhang et al. (2019), both of whichexploit the complementarity of multiple seman-tic representations with semantic space alignment.The difference is that we use an encoder-decoderframework to generate multiple views for inputfrom the source language and enforce a con-sensus between their predictions. Furthermore,

we introduce a language discriminator (Lampleet al., 2018a) to encourage the encoder to generatelanguage-invariant representations from the input.

3 Methodology

In this section, we will introduce our model’s gen-eral workflow, including the details of each com-ponent and our training algorithm.

3.1 Problem Setup

Given monolingual text data {Dsrc, Dtgt} fromboth the source and target language with a sub-set of labeled samples {DL

src, yLsrc} in the source

language where yLsrc is a vector of class labels andDLsrc ⊂ Dsrc, the task aims to build a universal

classification model f(X;θ) → y parameterizedby θ that can be directly applicable to unlabeleddata in the target language, where X is an inputdocument from any language and y is its class la-bel. Note that in this paper we assume two lan-guages share the same class types.

3.2 Model Architecture

Our proposed approach multi-view encoder clas-sifier (MVEC) is composed of three components:an encoder-decoder, a language discriminator, anda classifier. Motivated by the success of unsu-pervised machine translation (UMT) in Lampleet al. (2018a) and reconstruction regularization byan autoencoder in Sabour et al. (2017), we adoptthe encoder-decoder framework from UMT (Lam-ple et al., 2018a) and introduce self-reconstructionloss within one language and back-translation re-construction loss across languages together withthe normal loss from classification. For simplic-ity, we denote self-reconstruction loss as “within-domain loss” and back-translation reconstructionloss as “cross-domain loss” throughout the paper.

Although the encoder from UMT can gen-erate a latent representation for input sen-tences/documents, there is still a semantic gap be-tween the source and target language. Follow-ing Lample et al. (2018a); Chen et al. (2018), weenrich the encoder-decoder framework with a lan-guage discriminator that can produce fine-tunedlatent representations to align latent representa-tions from two languages better. Such represen-tations are necessary to train a language-invariantclassifier that is robust to the shift in languages.

In particular, as illustrated in Figure 1, the en-coder is used to encode source and target docu-

5762

Textclassifier

LC

Languagediscriminator Encoderadv

DecoderDecoder

Sourcelanguage

Targetlanguage

Encoder

crosswithin domain

domainwithin domain

advtraining training

Text label

Ladv

Lwd_src Lwd_tgtLcd_src Lcd_tgt

LD

Figure 1: Multi-view encoder classifier (MVEC) ar-chitecture. Blue (red) lines indicate the message flowwithin the source/target language (across languages),respectively. Green lines indicate the message flowfrom the encoder to the text classifier. The encoder anddecoder share the same parameters.

ments (a sequence of sentences) into a shared la-tent space, while the decoder is responsible fordecoding the documents from the latent space tothe source or the target language. Following Lam-ple et al. (2018a), the encoder-decoder is sharedfor both languages (domains) and trained within-domain and cross-domain. The language discrim-inator aims to predict the language source for eachdocument, and the classifier is trained to classifyeach document into predefined class labels.

Under the unsupervised setting, MVEC onlyobserves unlabeled monolingual corpora from twolanguages and some labeled documents in thesource language. The unlabeled monolingual datais normally sampled from the application domain,i.e., unlabeled product reviews or social mediaposts, which is used in both adopting pretrainedLMs in the target domain and training UMT. Asshown in Figure 1, unlabeled source and targetdata only pass through encoder-decoder and lan-guage discriminator, while the labeled source datapass all components in the system, including thesentiment classifier. For evaluation purposes, wemay have labeled documents in the target lan-guage. However, they are only used during the testperiod. In the following subsections, we introduceeach component of MVEC in detail.

3.3 Encoder-Decoder

Let x(l) = (x(l)1 , x

(l)2 , · · · , x

(l)n ) denote the in-

put document of n words from a particular lan-

guage l, where l ∈ {src, tgt}. The encoderis a neural network eθenc(x

(l)) parameterized byθenc that produces a sequence of n hidden statesZ(l) = (z

(l)1 , z

(l)2 , · · · , z

(l)n ) by using the corre-

sponding word embedding for x(l)i , where z(l)i is

the latent representation of x(l)i in the shared la-tent space and θenc are parameters of the encodershared between two languages. The encoder couldbe a BiLSTM or a transformer (Vaswani et al.,2017). In this paper, we adopt the transformer,which has achieved enormous success in (e.g.,)recent text representation learning tasks (Devlinet al., 2019; Conneau and Lample, 2019).

Given Z(l) as the input, the decoderdθdec(Z

(l)) generates the output sequencey(l) = (y

(l)1 , y

(l)2 , · · · , y(l)k ). We use the same

transformer based decoder as in Conneau andLample (2019), parameterized by θdec. For sim-plicity, we will denote the encoder and decoderby e(x(l)) and d(Z(l)) respectively instead ofeθenc(x

(l)) and dθdec(Z(l)).

It is more likely for the encoder-decoder tomerely memorize every input word one by one ifthere are no imposed constraints. To improve therobustness of encoder-decoder, we follow Lam-ple et al. (2018a) to adopt the Denoising Auto-encoders (DAE) (Vincent et al., 2008), which re-covers input from its corrupted version.

There are three ways to inject noise into thedocument including shuffle, dropout, and replace-ment by special words. In our model, we drop andreplace every word with probabilities of pd andpb, respectively, and we slightly shuffle the inputdocument by implementing random permutation σon the input document, where pd and pb can beviewed as hyper-parameters for controlling noiselevels. In our design, the permutation σ satisfiesthe condition |σ(i) − i| ≤ k, ∀i ∈ {1, · · · , n},where n is the length of input document and k isanother hyper-parameter.

Note that the noise model is only applied to un-labeled data used for training the encoder-decoderand the discriminator, while labeled data will keepits originality for all components training. We useG(.) to denote a stochastic noise model, whichtakes input document x(l) and generates G(x(l))as a randomly sampled noisy version of x(l).

To incorporate the encoder-decoder as regu-larization components, we follow Lample et al.(2018a) to consider both within-domain and cross-domain objective functions. The first objective

5763

function aims to reconstruct a document from anoisy version of itself within a language, whereasthe second (cross-domain) objective function tar-gets to teach the model to translate an input doc-ument across languages. Specifically, given a lan-guage l ∈ {src, tgt}, the within-domain objectivefunction can be written as:

Rwd(θed, l) = Ex∼Dl,x∼d(e(G(x)))[∆(x, x)] (1)

where θed = [θenc,θdec], x ∼ d(e(G(x))) is areconstruction of the corrupted version of x sam-pled from the monolingual dataset Dl, and ∆ isthe sum of token-level cross-entropy loss to mea-sure discrepancy between two sequences.

Similarly, we consider teaching the encoder-decoder to reconstruct x in one language from atranslation of x in the other language, leading tothe following cross-domain objective function:

Rcd(θed, l1, l2) = Ex∼Dl1,x∼d(e(T (x)))[∆(x, x)]

(2)where (l1, l2) ∈ {(src, tgt), (tgt, src)} and T (.)is the current UMT model applied to input docu-ment x from language l1 to language l2.

3.4 Language DiscriminatorCross-lingual classifiers work well when their in-put produced by the encoder is language-invariant,as studied in Chen et al. (2018). Thus, we pre-fer our encoder to map input documents from bothlanguages into a shared feature space indepen-dent of languages. To achieve this goal, we fol-low Chen et al. (2018); Lample et al. (2018a) andintroduce a language discriminator into our model,which is a feed-forward neural network with twohidden layers and one softmax layer to identify thelanguage source from the encoder’s output. In par-ticular, we minimize the following cross-entropyloss function:

LD(θD|θenc) = −E(l,x(l))[logPD(l|e(x(l))] (3)

where θD denotes parameters of the discrimina-tor, (l, x(l)) corresponds to language and docu-ment pairs uniformly sampled from monolingualdatasets, and PD(.) is the output from the softmaxlayer. Meanwhile, the encoder is trained to “fool”the discriminator:

Ladv(θenc|θD) = −Ex(li)∼Dli[logPD(lj |e(x(li))]

(4)with lj = l1 if li = l2, and vice versa.

3.5 Multi-view ClassifierThus far, we have described how we obtain alanguage-invariant latent space to encode two lan-guages, which may not be sufficient to generalizewell across languages if we simply train a clas-sifier on the encoder’s output for the source lan-guage (Chen et al., 2018). One key difference be-tween Chen et al. (2018) and our work is that weuse UMT (Lample et al., 2018a), which can gener-ate multiple views for the input labeled documentsfrom the source language. We can thereby benefitfrom multi-view learning’s superior generalizationcapability over single-view learning (Zhao et al.,2017).

Particularly, we consider two views of input: (i)the encoded labeled documents from the sourcelanguage; (ii) the encoded back-translations of thesource documents from the target language. Ourlearning objective is to train the classifier to matchpredicted document labels with ground truth fromthe source language and to encourage two predic-tive distributions on the two views to be as similaras possible. We consider the following objectivefunction:

LC(θC ,θed) = E(x,y)[∆(y, Pθc(e(x)))

+DKL(Pθc(e(x)) || Pθc(e(T (x)))]︸ ︷︷ ︸Two views’ consensus

(5)

where (x, y) ∼ {DLsrc, y

Lsrc}, DKL(. || .) is KL

Divergence to measure the difference between twodistributions, y is the class label of input documentx and θc are parameters of classifier. Followingprevious studies in text classification (Devlin et al.,2019), we use the first token’s representation in thelast hidden layer from the transformer encoder asthe document representation vector. The classifieris a feed-forward neural network with two hiddenlayers and a softmax layer.

The final objective function at one iteration ofour learning algorithm is to minimize the follow-ing loss function:

Lall = LC + λwd × (Rwd src +Rwd tgt) (6)

+ λcd × (Rcd src +Rcd tgt) + λadv × Ladv

where λwd, λcd, λadv are hyper-parameters totrade-off among within-domain loss, the cross-domain loss and the adversarial loss, respectively.

3.6 Training AlgorithmOur model relies on an initial translation machineT (0), which provides a translation from one lan-

5764

guage to another for calculating the cross-domainloss in Eq. (2) and classifier loss in Eq. (5).

To accelerate the training, we initialize T (0)

by pretraining a transformer-based UMT (Con-neau and Lample, 2019) for certain steps with thesame encoder-decoder architecture as our modelon monolingual Wikipedia text. After pretraining,we use the pretrained encoder-decoder network toinitialize our model and start training the classifierand the discriminator. Meanwhile, we refine theencoder and the decoder on monolingual data andlabeled data from the source language.

During each training step, the optimization iter-ates from updating θD in Eq. (3) to updating θedand θC in Eq. (6). Note that if a batch of docu-ments drawn from monolingual data are all unla-beled, then we suspend updating classifier param-eters and only update the parameters of the lan-guage discriminator and encoder-decoder. In Al-gorithm 1, we provide a detailed procedure.

Algorithm 1 The proposed MVEC algorithm.

1: procedure TRAINING(Dsrc, Dtgt, yLsrc)Dsrc and Dtgt: monolingual datasets, yLsrc:labels in the source language.

2: T (0) ← pretrain a transformer based UMTusing (Conneau and Lample, 2019);

3: for t = 0, · · · ,max epoch do4: Using T (t) to translate each document

in a batch;5: θD ← argmin LD in Eq. (3) while

fixing θC , θed;6: θC , θed ← argmin Lall in Eq. (6)

while fixing θD;7: Update T (t+1) ← {e(t), d(t)};8: return θC , θenc9: End procedure

4 Experiment

We conduct experiments on cross-lingual multi-class and binary sentiment classification using fivelanguage pairs involving 11 tasks. More specif-ically, English is always the source language,and the target languages are French, German,Japanese, Chinese, and Arabic, respectively.

4.1 Datasets

Amazon Review (French, German, Japanese).This is a multilingual sentiment classificationdataset (Duh et al., 2011) in four languages, in-

cluding English (en), French (fr), German (de),and Japanese (ja), covering three products (book,DVD, and music). For each product in each lan-guage, there are 2000 documents in each of thetraining and test sets. Each document contains atitle, a category label, a review, and a 5-point scalestar rating. Following Xu and Yang (2017); Chenand Qian (2019), we convert multi-class ratings tobinary ratings by thresholding at 3-point. For eachproduct, since the test set in English is not used,we combine the English training and test sets andrandomly sample 20% (800) documents as the val-idation set to tune hyper-parameters, and use therest 3200 samples for training. For each target lan-guage, we use the original 2000 test samples forcomparison with previous methods. Unlike Chenet al. (2018); Chen and Qian (2019) that used la-beled data in the target language for model selec-tion, we only use the labels of reviews in the targetlanguage for testing. There are 105k, 58k, 317k,300k unlabeled reviews for English, French, Ger-man and Japanese, respectively, which can be usedas monolingual data to train the encoder-decoderof our model.

Yelp and Hotel Review (Chinese). This dataset isfrom two sources: (i) 700k Yelp reviews in Englishwith five classes from Zhang et al. (2015), and (ii)170k hotel reviews in Chinese segmented and an-notated with five classes from Lin et al. (2015).Following the same setup in Chen et al. (2018),we split all Yelp reviews into a training set with650k reviews and validation set with 50k reviews.The 650k review contents are also served as themonolingual training data for English. For Chi-nese hotel review data, we sample 150k reviews asthe monolingual training set. The rest 20k reviewsare treated as the test set.

Social Media Posts (Arabic). The BBN Ara-bic Sentiment dataset is from Mohammad et al.(2016). There are 1200 documents from socialmedia posts annotated with three labels (negative,neutral, positive) in the data. The original datasetwas split into half as training and the other half astesting. Since we do not need validation data in thetarget language to tune the model, we randomlysample 1000 documents as test data. For Englishresource, we still use Yelp reviews and follow thesame split as the Chinese case, but convert 5 levelreviews into 3 levels2. Also, we randomly sample

21,2 → negative, 3 → neutral, 4,5 → positive

5765

161k sentences from the United Nations CorpusArab subset (Ziemski et al., 2016) as unlabeledmonolingual data for our model training.

4.2 Experiment Setting

For French, German and Japanese, we perform bi-nary classification. For Chinese and Arabic, weperform multi-class classification.

Data Preprocessing. Following Lample et al.(2018c), we extract and tokenize monolingualdata of each language using Moses (Koehn et al.,2007). Then we use the neural machine trans-lation for rare words with subword units, namedfastBPE (Sennrich et al., 2016) in three steps. Indetail, BPE code is collected from the pretrainedXLM-100 models (Conneau and Lample, 2019),then applied to all tokenized data and used to ex-tract the training vocabulary. To constrain ourmodel size, we only keep the top 60k most fre-quent subword units in our training set. Finally,we binarize monolingual data and labeled data formodel training, validation and testing.

Pretraining Details. As mentioned earlier, ourmodel depends on an initial translation machineto compute reconstruction loss and classifier loss.We leverage pretrained language models (Con-neau and Lample, 2019) to initialize a transformer-based UMT (Lample et al., 2018a) and train it onWikipedia text3. In particular, we sample 10 mil-lion sentences from each language pairs and usethe XLM library4 to train a UMT (Lample et al.,2018a) for 200K steps. The resulting encoder-decoder are used to initialize our model.

Regarding word embedding initialization, weuse the embeddings obtained from the 1st layer ofpretrained language models (Conneau and Lam-ple, 2019), which has demonstrated better cross-lingual performance in a number of evaluationmetrics over MUSE (Lample et al., 2018b).

Training Details. In our experiment, both en-coder and decoder are 6 layer transformers with 8-head self-attention. We set both subword embed-ding and hidden state dimension to 1024 and usegreedy decoding to generate a sequence of tokens.The encoder-decoder and classifier are trained us-ing Adam optimizer (Kingma and Ba, 2015) witha learning rate of 10−5 and a mini-batch size of 32.We set the hidden dimension to 128 for both clas-

3http://dumps.wikimedia.org/4www.github.com/facebookresearch/XLM

sifier and discriminator. For parameters of denois-ing auto-encoder, we set pd = 0.1, pb = 0.2 andk = 3 following Lample et al. (2018a). Finally,we perform a grid search for hyper-parameters on{0.5,1,2,4,8} and set λwd, λcd to 1 and λadv to 4.To prevent gradient explosion, we clip the gradientL2 norm by 5.0. Our approach is implemented inPaddlePaddle5 and all experiments are conductedon an NVIDIA Tesla M40 (24GB) GPU.

Competing Methods. We have compared ourmethod with several recently published results.Due to the space limit, we briefly introduce sev-eral representative baselines: LR+MT translatedthe bag of words from target language to sourcelanguage via machine translation and then built alogistic regression model. BWE baselines rely onBilingual Word Embeddings (BWEs), wherein 1-to-1 indicates that we are only transferring fromEnglish, while 3-to-1 means the training data fromall other three languages. CLDFA (Xu and Yang,2017) was built on model distillation on parallelcorpora with adversarial feature adaptation tech-nique. PBLM (Ziser and Reichart, 2018) usedbilingual word embeddings and pivot-based lan-guage modeling for cross-domain & cross-lingualclassification. MBERT (Devlin et al., 2019) andXLM-FT (Conneau and Lample, 2019) directlyfine-tuned a single layer classifier based on pre-trained LM multilingual BERT and XLM.

4.3 Experiment Results

In Table 1 and Table 2, we compare our methodwith others based on their published results or ourreproduced results from their code. Our results areaveraged based on 5 rounds of experiment with thestandard deviation around 1%-1.5%. Followingprevious baselines, we do not report them here.

Our first observation from Table 1 is thatour model and the fine-tuned multilingual LMMBERT (Devlin et al., 2019) and XLM-FT (Con-neau and Lample, 2019) outperform all previousmethods including the methods with cross-lingualresources for 8/9 tasks by a large margin, whichindicates the huge benefit from pretrained LMsin the zero-shot setting. Compared with MBERTand XLM-FT, our model obtains better perfor-mance when the target language is more similarto the source language, for example, German andFrench, and one task in Japanese.

5http://www.paddlepaddle.org/

5766

German (2) French (2) Japanese (2)Approach books DVD music avg books DVD music avg books DVD music avg

With cross-lingual resourcesLR+MT 79.68 77.92 77.22 78.27 80.76 78.83 75.78 78.46 70.22 71.30 72.02 71.18CR-RL1 79.89 77.14 77.27 78.10 78.25 74.83 78.71 77.26 71.11 73.12 74.38 72.87Bi-PV2 79.51 78.60 82.45 80.19 84.25 79.60 80.09 81.31 71.75 75.40 75.45 74.20CLDFA3 83.95 83.14 79.02 82.04 83.37 82.56 83.31 83.08 77.36 80.52 76.46 78.11

With implicit cross-lingual resourcesUMM4 81.65 81.27 81.32 81.41 80.27 80.27 79.41 79.98 71.23 72.55 75.38 73.05PBLM5 78.65 79.90 80.10 79.50 77.90 75.65 75.95 76.50 - - - -

Without cross-lingual resourcesBWE (1-to-1) 76.00 76.30 73.50 75.27 77.80 78.60 78.10 78.17 55.93 57.55 54.35 55.94BWE (3-to-1) 78.35 77.45 76.70 77.50 77.95 79.25 79.95 79.05 54.78 54.20 51.30 53.43MAN-MoE6 82.40 78.80 77.15 79.45 81.10 84.25 80.90 82.08 62.78 69.10 72.60 68.16MBERT7 84.35 82.85 83.85 83.68 84.55 85.85 83.65 84.68 73.35 74.80 76.10 74.75XLM-FT8 86.85 84.20 85.90 85.65 88.1 86.95 86.20 87.08 80.95 79.20 78.02 79.39MVEC (Ours) 88.41 87.32 89.97 88.61 89.08 88.28 88.50 88.62 79.15 77.15 79.70 78.67

1 Xiao and Guo (2013) 2 Pham et al. (2015) 3 Xu and Yang (2017) 4 Xu and Wan (2017)5 Ziser and Reichart (2018) 6 Chen and Qian (2019) 7 Devlin et al. (2019) 8 Conneau and Lample (2019)Table 1: Prediction accuracy of binary classification in the test set for three language pairs. The highest perfor-mance is in bold, while the highest performance within the method group is underlined.

Approach Chinese (5) Arabic (3)LR+MT 34.01 51.67DAN 29.11 48.00mSDA 31.44 48.33ADAN 42.49 52.54MBERT 38.85 50.40XLM-FT 42.22 49.50MVEC (Ours) 43.36 49.70

Table 2: Prediction accuracy of 5-class and 3-classclassification tasks on the test set.

In Table 2, we show the comparison be-tween our method and a few other published re-sults, including ADAN (Chen et al., 2018) andmSDA (Chen et al., 2012) for Chinese and Ara-bic languages in multi-class setting. Similarly,our model obtains slightly better accuracy in Chi-nese. Overall, built on top of the pretrained LMsand UMT, our full model achieves the state-of-the-art performance on 8/11 sentiment classificationtasks, especially when the target language is moresimilar to the source language.

Moreover, we illustrate the effectiveness ofencoder-decoder based regularization in reducingthe language shift in the shared latent space. Intu-itively, if the fine-tuned latent space is less sensi-tive to the language shift, the performance on val-idation sets and test sets should be highly corre-lated during training. In Figure 2, we report theaverage accuracy of both validation and test setw.r.t. training epochs over five runs on Amazonbook review data in French.

0 2 4 6 8 10 12

Epoch index

70

75

80

85

90

95

Accura

cy (

%)

MVEC

valid-en

test-fr

0 2 4 6 8 10 12

Epoch index

70

75

80

85

90

95

Accura

cy (

%)

XLM-FT

valid-en

test-fr

Figure 2: Validation and test accuracy w.r.t. trainingepochs for Amazon book review in French. Left: ourmethod (MVEC). Right: XLM-FT.

From Figure 2, we observe that even thoughour model’s best validation accuracy is lower thanXLM-FT (Conneau and Lample, 2019) in En-glish, it has more correlated accuracy curves thanXLM-FT across English and French. For exam-ple, the validation accuracy of XLM-FT starts de-creasing after epoch 10, while the test accuracyis still increasing. Such an observation showsthat the latent representation learned solely fromself-supervised objectives (e.g., masked languagemodeling) may not well capture the semantic sim-ilarity among languages. Hence the resulting clas-sifier may work well in the source language butmay not generalize to the target language. In con-trast, our model sacrifices some accuracy in thesource language but can select better models forthe target language in a cross-lingual setting.

4.4 Ablation Study

To understand the effect of different componentsin our model on the overall performance, we con-

5767

German French Japanese Chinese ArabicFull model: 88.61 88.62 78.67 43.36 49.70w/o cross-domain loss: 83.22 82.40 72.05 35.74 42.80w/o within-domain loss: 82.90 82.15 71.27 37.21 41.60w/o adversarial training: 84.85 84.58 73.75 39.36 46.37w/o two-views consensus: 86.21 86.18 75.25 40.95 46.77

Table 3: Ablation study on five language pairs.

duct an ablation study, as reported in Table 3.Clearly, the encoder-decoder trained either by thewithin-domain objective or cross-domain objec-tive is the most critical. For Amazon data in threelanguages (German, French, Japanese), the modelwithout cross-domain loss obtains prediction ac-curacy of 83.22%, 82.40%, and 72.05%, whichgets decreased by 5%−7% compared with the fullmodel. The performance is also significantly de-graded when the adversarial training componentis removed because the distribution of latent doc-ument representations is not similar between twolanguages. The two-views consensus componentalso has a significant effect on the performanceof our model, with a performance drop up to 5points for en-jp. Such a result verifies our claimthat cross-lingual model benefits from training onmultiple views of the input.

4.5 Case StudyTo further explore the effectiveness of our ap-proach, we visualize the encoder’s output and thelast layer before softmax for 10 randomly sampledAmazon reviews in English and their translationsin French using Google Translation, as shown inAppendix A.2.

As seen in the lower-left panel of Figure 3, mostred circles and black squares with the same in-dices are very close for our method but are distantfor XLM-FT in the top-left. Such an observationimplies that our encoder combined UMT and alanguage discriminator adequately maps the inputinto a shared language-invariant latent space whilepreserving semantic similarity. For the last layerbefore softmax, even though XLM-FT also gen-erates reasonable representations to separate posi-tive and negative reviews, the data points are scat-tered randomly. On the contrary, our model’s out-put in the lower right panel of Figure 3 shows twomore obvious clusters with corresponding labelsthat can be easily separated. One cluster in the leftcontains all of the positive documents, while thenegative examples only appear on the right side.

-200 -100 0 100 200

-100

-50

0

50

100

150

200

250

1-

2-

3+

4+

5-

6+

7-

8+

9+

10-

1

2

34

5

6

7

8

9

10

-400 -200 0 200 400

-400

-200

0

200

400

1-

2-

3+

4+

5-

6+

7-

8+

9+

10-

1

2

3

4

5

6

7

8

9

10

-60 -40 -20 0 20 40 60

-60

-40

-20

0

20

40

60

1- 2-3+

4+

5-

6+

7-

8+

9+

10-

1

23

4

5

6

78

9

10

Figure 3: t-SNE visualizations of various layers ofXLM-FT and MVEC for en-fr. Red circles and blacksquares indicate documents from English and their cor-responding translations in the target language, respec-tively. Numbers indicate the document index and havea one-to-one mapping. +/- indicates labels and we onlyannotate English documents for simplicity. Top left:encoder output of XLM-FT. Top right: the last layerbefore softmax of XLM-FT. Lower left: encoder out-put of our method. Lower right: the last layer beforesoftmax of our method.

5 Conclusion

In this paper, we propose a cross-lingual multi-view encoder-classifier (MVEC) that requires nei-ther labeled data in the target language nor cross-lingual resources with the source language. Builtupon pretrained language models, our method uti-lizes the encoder-decoder component with a lan-guage discriminator from an unsupervised ma-chine translation system to learn a language-invariant feature space. Our approach departsfrom previous models that could only make use ofthe shared language-invariant features or dependon parallel resources. By constructing the fine-tuned latent feature space and two views of inputfrom the encoder-decoder of UMT, our model sig-nificantly outperforms previous methods for 8/11zero-shot sentiment classification tasks.

5768

ReferencesDaniel Andrade, Kunihiko Sadamasa, Akihiro Tamura,

and Masaaki Tsuchida. 2015. Cross-lingual textclassification using topic-dependent word probabil-ities. In Proceedings of the 2015 Conference ofthe North American Chapter of the Associationfor Computational Linguistics (NAACL-HLT), pages1466–1471, Denver, CO.

Nuria Bel, Cornelis H. A. Koster, and Marta Villegas.2003. Cross-lingual text categorization. In Pro-ceedings of the 7th European Conference on Re-search and Advanced Technology for Digital Li-braries (ECDL), pages 126–139, Trondheim, Nor-way.

Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Wein-berger, and Fei Sha. 2012. Marginalized denoisingautoencoders for domain adaptation. In Proceed-ings of the 29th International Conference on Ma-chine Learning (ICML), Edinburgh, UK.

Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie,and Kilian Q. Weinberger. 2018. Adversarialdeep averaging networks for cross-lingual sentimentclassification. Trans. Assoc. Comput. Linguistics,6:557–570.

Zhuang Chen and Tieyun Qian. 2019. Transfer capsulenetwork for aspect level sentiment classification. InProceedings of the 57th Conference of the Associ-ation for Computational Linguistics (ACL), pages547–556, Florence, Italy.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances inNeural Information Processing Systems (NeurIPS),pages 7057–7067, Vancouver, Canada.

Alexis Conneau, Holger Schwenk, Loıc Barrault, andYann LeCun. 2017. Very deep convolutional net-works for text classification. In Proceedings of the15th Conference of the European Chapter of theAssociation for Computational Linguistics (EACL),pages 1107–1116, Valencia, Spain.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies (NAACL-HLT), pages 4171–4186, Min-neapolis, MN.

Xin Dong and Gerard de Melo. 2019. A robust self-learning framework for cross-lingual text classifi-cation. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processingand the 9th International Joint Conference on Natu-ral Language Processing (EMNLP-IJCNLP), pages6305–6309, Hong Kong, China.

Kevin Duh, Akinori Fujino, and Masaaki Nagata. 2011.Is machine translation ripe for cross-lingual senti-ment classification? In Proceedings of the 49th An-nual Meeting of the Association for Computational

Linguistics: Human Language Technologies (ACL-HLT), pages 429–433, Portland, OR.

Akiko Eriguchi, Melvin Johnson, Orhan Firat, HidetoKazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingualneural machine translation. Technical report,arXiv:1809.04686.

Yanwei Fu, Timothy M. Hospedales, Tao Xiang, andShaogang Gong. 2015. Transductive multi-viewzero-shot learning. IEEE Trans. Pattern Anal.Mach. Intell., 37(11):2332–2345.

Ruidan He, Wee Sun Lee, Hwee Tou Ng, and DanielDahlmeier. 2019. An interactive multi-task learningnetwork for end-to-end aspect-based sentiment anal-ysis. In Proceedings of the 57th Conference of theAssociation for Computational Linguistics (ACL),Florence, Italy.

Phillip Keung, Yichao Lu, and Vikas Bhardwaj. 2019.Adversarial learning with contextual embeddings forzero-resource cross-lingual classification and NER.In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 1355–1360, Hong Kong, China.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In Proceed-ings of the 3rd International Conference on Learn-ing Representations (ICLR), San Diego, CA.

Philipp Koehn, Hieu Hoang, Alexandra Birch, ChrisCallison-Burch, and et al. 2007. Moses: Opensource toolkit for statistical machine translation.In Proceedings of the 45th Annual Meeting of theAssociation for Computational Linguistics (ACL),Prague, Czech Republic.

Guillaume Lample, Alexis Conneau, Ludovic Denoyer,and Marc’Aurelio Ranzato. 2018a. Unsupervisedmachine translation using monolingual corpora only.In Proceedings of the 6th International Conferenceon Learning Representations (ICLR), Vancouver,Canada.

Guillaume Lample, Alexis Conneau, Marc’AurelioRanzato, Ludovic Denoyer, and Herve Jegou.2018b. Word translation without parallel data. InProceedings of the 6th International Conferenceon Learning Representations (ICLR), Vancouver,Canada.

Guillaume Lample, Myle Ott, Alexis Conneau, Lu-dovic Denoyer, and Marc’Aurelio Ranzato. 2018c.Phrase-based & neural unsupervised machine trans-lation. In Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Processing(EMNLP), pages 5039–5049, Brussels, Belgium.

5769

Yingming Li, Ming Yang, and Zhongfei Zhang. 2019.A survey of multi-view representation learning.IEEE Trans. Knowl. Data Eng., 31(10):1863–1883.

Yiou Lin, Hang Lei, Jia Wu, and Xiaoyu Li. 2015. Anempirical study on sentiment classification of chi-nese review using word embedding. In Proceedingsof the 29th Pacific Asia Conference on Language,Information and Computation (PACLIC), Shanghai,China.

Saif M. Mohammad, Mohammad Salameh, and Svet-lana Kiritchenko. 2016. How translation alters sen-timent. J. Artif. Intell. Res., 55:95–130.

Hieu Pham, Thang Luong, and Christopher D. Man-ning. 2015. Learning distributed representations formultilingual text sequences. In Proceedings of the1st Workshop on Vector Space Modeling for Natu-ral Language Processing (VS@NAACL-HLT), pages88–94, Denver, Co.

Peter Prettenhofer and Benno Stein. 2010. Cross-language text classification using structural corre-spondence learning. In Proceedings of the 48th An-nual Meeting of the Association for ComputationalLinguistics (ACL), pages 1118–1127, Uppsala, Swe-den.

Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton.2017. Dynamic routing between capsules. In Ad-vances in Neural Information Processing Systems(NIPS), pages 3856–3866, Long Beach, CA.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Neural machine translation of rare words withsubword units. In Proceedings of the 54th AnnualMeeting of the Association for Computational Lin-guistics (ACL), Berlin, Germany.

Samuel L. Smith, David H. P. Turban, Steven Ham-blin, and Nils Y. Hammerla. 2017. Offline bilin-gual word vectors, orthogonal transformations andthe inverted softmax. In Proceedings of the 5thInternational Conference on Learning Representa-tions (ICLR), Toulon, France.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in Neural Information Pro-cessing (NIPS), pages 6000–6010, Long Beach, CA.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, andPierre-Antoine Manzagol. 2008. Extracting andcomposing robust features with denoising autoen-coders. In Proceedings of the Twenty-Fifth Inter-national Conference on Machine Learning (ICML),pages 1096–1103, Helsinki, Finland.

Shijie Wu and Mark Dredze. 2019. Beto, bentz, be-cas: The surprising cross-lingual effectiveness ofBERT. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processing

and the 9th International Joint Conference on Natu-ral Language Processing (EMNLP-IJCNLP), pages833–844, Hong Kong, China.

Min Xiao and Yuhong Guo. 2013. Semi-supervisedrepresentation learning for cross-lingual text classi-fication. In Proceedings of the 2013 Conference onEmpirical Methods in Natural Language Processing(EMNLP), pages 1465–1475, Seattle, WA.

Kui Xu and Xiaojun Wan. 2017. Towards a uni-versal sentiment classifier in multiple languages.In Proceedings of the 2017 Conference on Em-pirical Methods in Natural Language Processing(EMNLP), pages 511–520, Copenhagen, Denmark.

Ruochen Xu and Yiming Yang. 2017. Cross-lingualdistillation for text classification. In Proceedingsof the 55th Annual Meeting of the Associationfor Computational Linguistics (ACL), pages 1415–1425, Vancouver, Canada.

Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen,Lingbing Guo, and Yuzhong Qu. 2019. Multi-viewknowledge graph embedding for entity alignment.In Proceedings of the Twenty-Eighth InternationalJoint Conference on Artificial Intelligence (IJCAI),pages 5429–5435, Macao, China.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun.2015. Character-level convolutional networks fortext classification. In Advances in Neural Infor-mation Processing Systems (NIPS), pages 649–657,Montreal, Canada.

Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun.2017. Multi-view learning overview: Recentprogress and new challenges. Inf. Fusion, 38:43–54.

Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016.Cross-lingual sentiment classification with bilingualdocument representation learning. In Proceedingsof the 54th Annual Meeting of the Association forComputational Linguistics (ACL), Berlin, Germany.

Michal Ziemski, Marcin Junczys-Dowmunt, and BrunoPouliquen. 2016. The united nations parallel cor-pus v1.0. In Proceedings of the Tenth InternationalConference on Language Resources and Evaluation(LREC), Portoroz, Slovenia.

Yftah Ziser and Roi Reichart. 2018. Deep pivot-basedmodeling for cross-language cross-domain transferwith minimal guidance. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing (EMNLP), pages 238–249, Brus-sels, Belgium.

Will Y. Zou, Richard Socher, Daniel M. Cer, andChristopher D. Manning. 2013. Bilingual wordembeddings for phrase-based machine translation.In Proceedings of the 2013 Conference on Em-pirical Methods in Natural Language Processing(EMNLP), pages 1393–1398, Seattle, WA.

5770

A Additional Details on Datasets

A.1 Summary Statistics of Labeled DatasetsFor the Amazon Review dataset, we use the sametest set as Duh et al. (2011) but transform theminto binary labels for comparison with previousworks. After transformation, the test set of eachproduct category has equal number of positive andnegative ratings (1000 vs 1000).

For the Yelp and Hotel Review dataset, we fol-low the same split as Chen et al. (2018) and keepthe original rating. The test set contains 10k doc-uments in total with around 2000 documents foreach rating level.

The Arabic social media dataset contains 1000test documents sampled from 1200 social mediaposts with about 400 documents for each ratinglevel. Since Arabic data is not used for tuning pa-rameters as validation set, we use more test sam-ples than Chen et al. (2018).

A.2 Sampled Data for the Case StudyIn Section 4.5, we randomly sample 10 Amazonbook reviews in English, and translate them intoFrench using Google Translation for case study.The sampled reviews and their French translationsare as follows:

1. More than mitigated for this tote album thatmixes some good ideas (the parodies of worksof art) and scenes that only echo the previousalbums lazily.

Plus qu’attenue pour cet album cabas qui melequelques bonnes idees (les parodies d’oeuvresd’art) et des scenes qui ne font que faire echoaux albums precedents paresseusement.

2. What a disappointment, so dear for that. Afterthe Gallic comeback, another album story to re-lease an album. Beautiful pictures, some coolstuff (so the picture with all the characters) ...but

Quelle deception, si chere pour ca. Apres leretour des Gaulois, une autre histoire d’albumpour sortir un album. De belles photos, destrucs sympas (donc la photo avec tous les per-sonnages) ... mais

3. We obviously believe we know everythingabout the unspeakable horror of concentrationcamps. Well no; if it’s a man; literally leaves

no voice! Any comment seems inappropriateand for all

Nous pensons evidemment que nous savonstout sur l’horreur indicible des camps de con-centration. Eh bien non, si c’est un homme nelaisse litteralement aucune voix! Tout commen-taire semble inapproprie et pour tous

4. “We who have survived”, said Primo Levi, “arenot good witnesses, because we belong to thistiny minority who, by prevarication, by skill orluck, have never touched”

“Nous qui avons survecu”, a declare PrimoLevi, “ne sommes pas de bons temoins, carnous appartenons a cette minuscule minoritequi, par tergiversation, par habilete ou parchance, n’avons jamais touche”

5. The questions are targeted and you must havethe financial means to follow the plan. I wouldnot recommend this document unlike the otherbook; I do not know how to lose weight, whichis useful

Les questions sont ciblees et vous devez avoirles moyens financiers de suivre le plan. Jene recommanderais pas ce document contraire-ment a l’autre livre; Je ne sais pas commentperdre du poids, ce qui est utile

6. I read this book in Spanish, in the native lan-guage of the writer. I find the book excellent.Not only because of her passionate story but forher love of books and literature

J’ai lu ce livre en espagnol, dans la languematernelle de l’auteur. Je trouve le livre excel-lent. Pas seulement a cause de son histoire pas-sionnee, mais aussi pour son amour des livreset de la litterature

7. I have been reading Chattam for many years,and this is the first time I have to struggle tofinish one of these novels. The bottom of thestory is not bad, but the finished product wasalmost undrinkable.

Je lis Chattam depuis de nombreuses annees etc’est la premiere fois que je dois lutter pour ter-miner l’un de ces romans. Le fond de l’histoiren’est pas mauvais, mais le produit fini etaitpresque imbuvable.

5771

8. THIS BOOK IS GREAT! I had seen the moviebefore I read the book and I was not disap-pointed!

CE LIVRE EST SUPER! J’avais vu le filmavant de lire le livre et je n’ai pas ete decu!

9. I still love it so much. But I wonder if wewill not go around in circles ... We changethe scenery, we add endearing characters, butthere is already the originality of the originalcreators.

Je l’aime toujours tellement. Mais je me de-mande si nous ne tournerons pas en rond ...On change de decor, on ajoute des personnages

attachants, mais il y a deja l’originalite de lacreation originale.

10. There are many mysteries in life, includingGrange’s! I really do not understand the ex-traordinary opinions about this author: it’swrong! And I hope it is not broadcast too muchabroad.

Il y a beaucoup de mysteres dans la vie, y com-pris ceux de Grange! Je ne comprends vrai-ment pas les opinions extraordinaires sur cetauteur: c’est faux! Et j’espere que ca ne serapas trop diffuse a l’etranger