Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4638–4655July 5 - 10, 2020. c©2020 Association for Computational Linguistics

4638

Similarity Analysis of Contextual Word Representation Models

John M. Wu∗1 Yonatan Belinkov*12

Hassan Sajjad3 Nadir Durrani3 Fahim Dalvi3 James Glass1

1MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA2Harvard John A. Paulson School of Engineering and Applied Sciences, Cambridge, MA, USA

3Qatar Computing Research Institute, HBKU Research Complex, Doha 5825, Qatar{johnmwu,belinkov,glass}@csail.mit.edu{hsajjad,ndurrani,faimaduddin}@qf.org.qa

Abstract

This paper investigates contextual word repre-sentation models from the lens of similarityanalysis. Given a collection of trained mod-els, we measure the similarity of their inter-nal representations and attention. Critically,these models come from vastly different archi-tectures. We use existing and novel similaritymeasures that aim to gauge the level of local-ization of information in the deep models, andfacilitate the investigation of which design fac-tors affect model similarity, without requiringany external linguistic annotation. The analy-sis reveals that models within the same fam-ily are more similar to one another, as may beexpected. Surprisingly, different architectureshave rather similar representations, but differ-ent individual neurons. We also observed dif-ferences in information localization in lowerand higher layers and found that higher lay-ers are more affected by fine-tuning on down-stream tasks.1

1 Introduction

Contextual word representations such as ELMo (Pe-ters et al., 2018a) and BERT (Devlin et al., 2019)have led to impressive improvements in a varietyof tasks. With this progress in breaking the stateof the art, interest in the community has expandedto analyzing such models in an effort to illuminatetheir inner workings. A number of studies haveanalyzed the internal representations in such mod-els and attempted to assess what linguistic proper-ties they capture. A prominent methodology forthis is to train supervised classifiers based on themodels’ learned representations, and predict var-ious linguistic properties. For instance, Liu et al.(2019a) train such classifiers on 16 linguistic tasks,including part-of-speech tagging, chunking, named

∗Equal contribution1The code is available at https://github.com/

johnmwu/contextual-corr-analysis.

entity recognition, and others. Such an approachmay reveal how well representations from differentmodels, and model layers, capture different proper-ties. This approach, known as analysis by probingclassifiers, has been used in numerous other stud-ies (Belinkov and Glass, 2019).

While the above approach yields compelling in-sights, its applicability is constrained by the avail-ability of linguistic annotations. In addition, com-parisons of different models are indirect, via theprobing accuracy, making it difficult to commenton the similarities and differences of different mod-els. In this paper, we develop complementary meth-ods for analyzing contextual word representationsbased on their inter- and intra-similarity. While thissimilarity analysis does not tell us absolute factsabout a model, it allows comparing representationswithout subscribing to one type of information. Weconsider several kinds of similarity measures basedon different levels of localization/distributivity ofinformation: from neuron-level pairwise compar-isons of individual neurons to representation-levelcomparisons of full word representations. We alsoexplore similarity measures based on models’ at-tention weights, in the case of Transformer mod-els (Vaswani et al., 2017). This approach enablesus to ask questions such as: Do different models be-have similarly on the same inputs? Which designchoices determine whether models behave simi-larly or differently? Are certain model componentsmore similar than others across architectures? Isthe information in a given model more or less local-ized (encoded in individual components) comparedto other models?2

2Hinton (1984) defines a localist representation as one us-ing one computing element for each represented entity. Ina language model, this definition would depend on what lin-guistic concepts we deem important, and is thus somewhatarbitrary. We develop a measure that aims to capture thisnotion of localization without recourse to a specific set oflinguistic properties.

4639

We choose a collection of pre-trained modelsthat aim to capture diverse aspects of modelingchoices, including the building blocks (Recur-rent Networks, Transformers), language model-ing objective (unidirectional, bidirectional, masked,permutation-based), and model depth (from 3 to 24layers). More specifically, we experiment with vari-ants of ELMo, BERT, GPT (Radford et al., 2018),GPT2 (Radford et al., 2019), and XLNet (Yanget al., 2019). Notably, we use the same methodsto investigate the effect that fine-tuning on down-stream tasks has on the model similarities.

Our analysis yields the following insights:

• Different architectures may have similar rep-resentations, but different individual neurons.Models within the same family are more simi-lar to one another in terms of both their neu-rons and full representations.

• Lower layers are more similar than higher lay-ers across architectures.

• Higher layers have more localized representa-tions than lower layers.

• Higher layers are more affected by fine-tuningthan lower layers, in terms of their representa-tions and attentions, and thus are less similarto the higher layers of pre-trained models.

• Fine-tuning affects the localization of informa-tion, causing high layers to be less localized.

Finally, we show how the similarity analysiscan motivate a simple technique for efficient fine-tuning, where freezing the bottom layers of mod-els still maintains comparable performance to fine-tuning the full network, while reducing the fine-tuning time.

2 Related Work

The most common approach for analyzing neuralnetwork models in general, and contextual wordrepresentations in particular, is by probing classi-fiers (Ettinger et al., 2016; Belinkov et al., 2017;Adi et al., 2017; Conneau et al., 2018; Hupkeset al., 2018), where a classifier is trained on a cor-pus of linguistic annotations using representationsfrom the model under investigation. For example,Liu et al. (2019a) used this methodology for in-vestigating the representations of contextual wordrepresentations on 16 linguistic tasks. One limita-tion of this approach is that it requires specifying

linguistic tasks of interest and obtaining suitable an-notations. This potentially limits the applicabilityof the approach.

An orthogonal analysis method relies on simi-larities between model representations. Bau et al.(2019) used this approach to analyze the role ofindividual neurons in neural machine translation.They found that individual neurons are importantand interpretable. However, their work was limitedto a certain kind of architecture (specifically, a re-current one). In contrast, we compare models ofvarious architectures and objective functions.

Other work used similarity measures to studylearning dynamics in language models by compar-ing checkpoints of recurrent language models (Mor-cos et al., 2018), or a language model and a part-of-speech tagger (Saphra and Lopez, 2019). Our workadopts a similar approach, but explores a range ofsimilarity measures over different contextual wordrepresentation models.

Questions of localization and distributivity of in-formation have been under investigation for a longtime in the connectionist cognitive science litera-ture (Page, 2000; Bowers, 2002; Gayler and Levy,2011). While neural language representations arethought to be densely distributed, several recentstudies have pointed out the importance of indi-vidual neurons (Qian et al., 2016; Shi et al., 2016;Radford et al., 2017; Lakretz et al., 2019; Bau et al.,2019; Dalvi et al., 2019; Baan et al., 2019). Ourstudy contributes to this line of work by designingmeasures of localization and distributivity of infor-mation in a collection of models. Such measuresmay facilitate incorporating neuron interactions innew training objectives (Li et al., 2020).

3 Similarity Measures

We present five groups of similarity measures, eachcapturing a different similarity notion. Considera collection of M models {f (m)}Mm=1, yieldingword representations h

(m)l and potentially atten-

tion weights α(m)l at each layer l. Let k index neu-

rons h(m)l [k] or attention heads α(m)

l [k]. h(m)l [k],

α(m)l [k] are real (resp. matrix) valued, ranging over

words (resp. sentences) in a corpus. Our similar-ity measures are of the form sim(h

(m)l ,h

(m′)l′ ) or

sim(α(m)l ,α

(m′)l′ ), that is, they find similarities

between layers. We present the full mathematicaldetails in appendix A.

4640

3.1 Neuron-level similarity

A neuron-level similarity measure captures sim-ilarity between pairs of individual neurons. Weconsider one such measure, neuronsim, follow-ing Bau et al. (2019). For every neuron k in layerl, neuronsim finds the maximum correlation be-tween it and another neuron in another layer l′.Then, it averages over neurons in layer l.3 Thismeasure aims to capture localization of information.It is high when two layers have pairs of neuronswith similar behavior. This is far more likely whenthe models have local, rather than distributed repre-sentations, because for distributed representationsto have similar pairs of neurons the informationmust be distributed similarly.

3.2 Mixed neuron–representation similarity

A mixed neuron–representation similarity measurecaptures a similarity between a neuron in onemodel with a layer in another. We consider onesuch measure, mixedsim: for every neuron k inlayer l, regress to it from all neurons in layer l′

and measure the quality of fit. Then, average overneurons in l. It is possible that some informationis localized in one layer but distributed in anotherlayer. mixedsim captures such a phenomenon.

3.3 Representation-level similarity

A representation-level measure finds correlationsbetween a full model (or layer) simultaneously.We consider three such measures: two based oncanonical correlation analysis (CCA), namely sin-gular vector CCA (svsim; Raghu et al. 2017) andprojection weighted CCA (pwsim; Morcos et al.2018), in addition to linear centered kernel align-ment (ckasim; Kornblith et al. 2019).4 Thesemeasures emphasize distributivity of information—if two layers behave similarly over all of their neu-rons, the similarity will be higher, even if no in-dividual neuron has a similar matching pair or isrepresented well by all neurons in the other layer.

Other representation-level similarity measuresmay be useful, such as representation similarityanalysis (RSA; Kriegeskorte et al. 2008), which

3In this and other measures that allowed it, we also exper-imented with averaging just the top k neurons (or canonicalcorrelations, in Section 3.3 measures) in case most of the layeris noise. Heatmaps are in the online repository. We did notnotice major differences.

4We also experimented with the RBF variant, which iscomputationally demanding. We found similar patterns inpreliminary experiments, so we focus on the linear variant.

has been used to analyze neural network represen-tations (Bouchacourt and Baroni, 2018; Chrupałaand Alishahi, 2019; Chrupała, 2019), or other vari-ants of CCA, such as deep CCA (Andrew et al.,2013). We leave the explorations of such measuresto future work.

3.4 Attention-level similarityPrevious work analyzing network similarity hasmostly focused on representation-based similari-ties (Morcos et al., 2018; Saphra and Lopez, 2019;Voita et al., 2019a). Here we consider similaritybased on attention weights in Transformer models.

Analogous to a neuron-level similarity measure,an attention-level similarity measure finds the most“correlated” other attention head. We consider threemethods to correlate heads, based on the norm oftwo attention matrices α

(m)l [k], α

(m′)l′ [k′], their

Pearson correlation, and their Jensen–Shannondivergence.5 We then average over heads k inlayer l, as before. These measures are similar toneuronsim in that they emphasize localizationof information—if two layers have pairs of headsthat are very similar in their behavior, the similaritywill be higher.

3.5 Distributed attention-level similarityWe consider parallels of the representation-levelsimilarity. To compare the entire attention headsin two layers, we concatenate all weights from allheads in one layer to get an attention representa-tion. That is, we obtain attention representationsα

(m)l [h], a random variable ranging over pairs of

words in the same sentence, such that α(m)l,(i,j)[h] is

a scalar value. It is a matrix where the first axis isindexed by word pairs, and the second by heads.We flatten these matrices and use svsim, pwsim,and ckasim as above for comparing these atten-tion representations. These measures should behigh when the entire set of heads in one layer issimilar to the set of heads in another layer.

4 Experimental Setup

Models We choose a collection of pre-trainedmodels that aim to capture diverse aspects of mod-eling choices, including the building blocks (RNNs,Transformers), language modeling objective (uni-directional, bidirectional, masked, permutation-based), and model depth (from 3 to 24 layers).

5Other recent work has used the Jensen–Shannon diver-gence to measure distances between attention heads (Clarket al., 2019; Jain and Wallace, 2019).

4641

(a) neuronsim (b) ckasim

Figure 1: Similarity heatmaps of layers in various models under neuron- and representation-level similarities.

ELMo variants We use the original ELMo (Peterset al., 2018a), a bidirectional RNN model with twohidden layers, as well as two variants – a deeper andlarger 4-layer model and a Transformer-equivalentvariant (Peters et al., 2018b).

GPT variants We use both the original OpenAITransformer (GPT; Radford et al. 2018) and its suc-cessor GPT2 (Radford et al., 2019), in the smalland medium model sizes. These are all unidirec-tional Transformer LMs.

BERT We use BERT-base/large (12/24 layers; De-vlin et al. 2019): Transformer LMs trained with amasked LM objective function.6

XLNet We use XLNet-base/large (12/24 lay-ers; Yang et al. 2019). Both are Transformer LMwith a permutation-based objective function.

Data For analyzing the models, we run themon the Penn Treebank development set (Marcuset al., 1993), following the setup taken by Liu et al.(2019a) in their probing classifier experiments.7

We collect representations and attention weightsfrom each layer in each model for computing thesimilarity measures. We obtain representations formodels used in Liu et al. (2019a) from their imple-mentation and use the transformers library (Wolfet al., 2019) to extract other representations. We ag-gregate sub-word representations by taking the rep-resentation of the last sub-word, following Liu et al.(2019a), and sub-word attentions by summing up at-

6BERT is also trained with a next sentence predictionobjective, although this may be redundant (Liu et al., 2019b).

7As suggested by a reviewer, we verified that the resultsare consistent when using another dataset (Appendix B.1).

tention to sub-words and averaging attention fromsub-words, following Clark et al. (2019), whichguarantees that the attention from each word sumsto one.

5 Similarity of Pre-trained Models

5.1 Neuron and representation levels

Figure 1 shows heatmaps of similarities be-tween layers of different models, according toneuronsim and ckasim. Heatmaps for theother measures are provided in Appendix B. Theheatmaps reveal the following insights.

Different architectures may have similar rep-resentations, but different individual neuronsComparing the heatmaps, the most striking distinc-tion is that neuronsim induces a distinctly block-diagonal heatmap, reflecting high intra-model sim-ilarities and low inter-model similarities. Asneuronsim is computed by finding pairs of verysimilar neurons, this means that within a model, dif-ferent layers have similar individual neurons, butacross models, neurons are very different. In con-trast, ckasim- show fairly significant similaritiesacross models (high values off the main diagonal),indicating that different models generate similarrepresentations. The most similar cross-model sim-ilarities are found by mixedsim (Figure 8d in Ap-pendix B), which suggests that individual neuronsin one model may be well represented by a linearcombination of neurons in another layer. The otherrepresentation-level similarities (ckasim, svsim,and pwsim), also show cross-model similarities,albeit to a lesser extent.

4642

Models within the same family are more simi-lar The heatmaps show greater similarity withina model than across models (bright diagonal). Dif-ferent models sharing the same architecture andobjective function, but different depths, also ex-hibit substantial representation-level similarities– for instance, compare BERT-base and BERT-large or ELMo-original and ELMo-4-layers, un-der ckasim (Figure 1b). The Transformer-ELMopresents an instructive case, as it shares ELMo’sbidirectional objective function but with Transform-ers rather than RNNs. Its layers are mostly similarto themselves and the other ELMo models, but alsoto GPT, more so than to BERT or XLNet, whichuse masked and permutation language modelingobjectives, respectively. Thus it seems that the ob-jective has a considerable impact on representationsimilarity.8

The fact that models within the same family aremore similar to each other supports the choice ofSaphra and Lopez (2019) to use models of similararchitecture when probing models via similaritymeasures across tasks.9 A possible confounder isthat models within the same family are trained onthe same data, but cross-family models are trainedon different data. It is difficult to control for thisgiven the computational demands of training suchmodels and the current practice in the communityof training models on ever increasing sizes of data,rather than a standard fixed dataset. However, Fig-ure 2 shows similarity heatmaps of layers frompre-trained and randomly initialized models usingckasim, exhibiting high intra-model similarities,as before. Interestingly, models within the samefamily (either GPT2 or XLNet) are more similarthan across families, even with random models, in-dicating that intrinsic aspects of models in a givenfamily make them similar, regardless of the train-ing data or process.10 As may be expected, inmost cases, the similarity between random and pre-trained models is small. One exception is the verti-cal bands in the lower triangle, which indicate thatthe bottom layers of trained models are similar tomany layers of random models. This may be dueto random models merely transferring informationfrom bottom to top, without meaningful processing.

8Voita et al. (2019a) found that differences in the trainingobjective result in more different representations (accordingto pwsim) than differences in random initialization.

9We thank a reviewer for pointing out this connection.10Relatedly, Morcos et al. (2018) found similar CCA co-

efficients in representations from recurrent language modelstrained on different datasets.

Figure 2: ckasim similarity heatmap of layers in baseand random models.

Still, it may explain why random models some-times generate useful features (Wieting and Kiela,2019). Meanwhile, as pointed out by a reviewer,lower layers converge faster, leaving them closerto their initial random state (Raghu et al., 2017;Shwartz-Ziv and Tishby, 2017).

Lower layers are more similar across architec-tures The representation-level heatmaps (Figure1) all exhibit horizontal stripes at lower layers, es-pecially with ckasim, indicating that lower layersare more similar than higher layers when compar-ing across models. This pattern can be explainedby lower layers being closer to the input, whichis always the same words. A similar observationhas been made for vision networks (Raghu et al.,2017).11 Voita et al. (2019a) found a similar pat-tern comparing Transformer models with differentobjective functions.

Adjacent layers are more similar All heatmapsin Figure 1 exhibit a very bright diagonal and brightlines slightly off the main diagonal, indicating thatadjacent layers are more similar. This is even truewhen comparing layers of different models (noticethe diagonal nature of BERT-base vs. BERT-largein Figure 1b), indicating that layers at the samerelative depth are more similar than layers at dif-ferent relative depths. A similar pattern was foundin vision networks (Kornblith et al., 2019). Somepatterns are unexpected. For instance, comparing

11Raghu et al. (2017) also used svsim to study recurrentlanguage models, showing that lower layers converge faster.Although they have not looked at cross-model comparisons,faster convergence may be consistent with fewer changes dur-ing training, which can explain why lower layers are moresimilar across architectures.

4643

XLNet with the BERT models, it appears that lowerlayers of XLNet are more similar to higher layersof BERT. We speculate that this is an artifact of thepermutation-based objective in XLNet.

We found corroborating evidence for this obser-vation in ongoing parallel work, where we compareBERT and XLNet at different layers through word-(Liu et al., 2019a) and sentence-level tasks (Wanget al., 2019): while BERT requires mostly featuresfrom higher layers to achieve state-of-the-art re-sults, in XLNet lower and middle layers suffice.

Higher layers are more localized than lowerones The different similarity measures capturedifferent levels of localization vs. distributivity ofinformation. neuronsim captures cases of lo-calized information, where pairs of neurons indifferent layers behave similarly. svsim cap-tures cases of distributed information, where thefull layer representation is similar. To quantifythese differences, we compute the average simi-larity according to each measure when compar-ing each layer to all other layers. In effect, wetake the column-wise mean of each heatmap. Wedo this separately for svsim as the distributedmeasure and neuronsim as the localized mea-sure, and we subtract the svsim means from theneuronsim means. This results in a measure oflocalization per layer. Figure 3 shows the results.

In all models, the localization score mostly in-creases with layers, indicating that informationtends to become more localized at higher layers.12

This pattern is quite consistent, but may be surpris-ing given prior observations on lower layers cap-turing phenomena that operate at a local context(Tenney et al., 2019), which presumably requirefewer neurons. However, this pattern is in line withobservations made by Ethayarajh (2019), who re-ported that upper layers of pre-trained models pro-duce more context-specific representations. Thereappears to be a correspondence between our local-ization score and Ethayarajh’s context-specificityscore, which is based on the cosine similarity ofrepresentations of the same word in different con-texts. Thus, more localized representations arealso more context-specific. A direct comparisonbetween context-specificity and localization maybe fruitful avenue for future work.

Some models seem less localized than others,

12Recurrent models are more monotonous than Transform-ers, echoing results by Liu et al. (2019a) on language modelingperplexity in different layers.

Figure 3: Localization score of various model layers.

especially the ELMo variants, although this maybe confounded by their being shallower models.BERT and XLNet models first decrease in local-ization and then increase. Interestingly, XLNet’slocalization score decreases towards the end, sug-gesting that its top layer representations are lesscontext-specific.

5.2 Attention levelFigure 4 shows similarity heatmaps using two ofthe attention-level similarity measures—Jensen–Shannon and ckasim—for layers from 6 mod-els: BERT-base/large, GPT2-small/medium, andXLNet-base/large. Layers within the same modelor model family exhibit higher similarities (brightblock diagonal), in line with results from therepresentation-level analysis. In particular, underboth measures, GPT2 layers are all very similarto each other, except for the bottom ones. Com-paring the two heatmaps, the localized Jensen–Shannon similarity (Figure 4a) shows higher simi-larities off the main diagonal than the distributedckasim measure (Figure 4b), indicating that dif-ferent models have pairs of attention heads thatbehave similarly, although the collection of headsfrom two different models is different in the aggre-gate. Heatmaps for the other measures are providedin Appendix C, following primarily the same pat-terns.

It is difficult to identify patterns within a givenmodel family. However, under the attention-basedsvsim (Figure 10d in Appendix C), and to a lesserextent pwsim (Figure 10e), we see bright diag-onals when comparing different GPT2 (and to alesser extent XLNet and BERT) models, such thatlayers at the same relative depth are similar in theirattention patterns. We have seen such a result alsoin the representation-based similarities.

4644

(a) Jensen–Shannon (b) ckasim

Figure 4: Similarity heatmaps of layers in various models under two attention-level similarity measures.

Adjacent layers seem more similar in some cases,but these patterns are often swamped by the largeintra-model similarity. This result differs from ourresults for representational similarity.

GPT2 models, at all layers, are similar to thebottom layers of BERT-large, expressed in brightvertical bands. In contrast, GPT2 models do notseem to be especially similar to XLNet. ComparingXLNet and BERT, we find that lower layers of XL-Net are quite similar to higher layers of BERT-baseand middle layers of BERT-large. This parallels thefindings from comparing representations of XLNetand BERT, which we conjecture is the result of thepermutation-based objective in XLNet.

In general, we find the attention-based similar-ities to be mostly in line with the neuron- andrepresentation-level similarities. Nevertheless, theyappear to be harder to interpret, as fine-grained pat-terns are less noticeable. One might mention inthis context concerns regarding the reliability ofattention weights for interpreting the importanceof input words in a model (Jain and Wallace, 2019;Serrano and Smith, 2019; Brunner et al., 2020).However, characterizing the effect of such con-cerns on our attention-based similarity measures isbeyond the current scope.

6 Similarity of Fine-tuned Models

How does fine-tuning on downstream tasks affectmodel similarity? In this section, we compare pre-trained models and their fine-tuned versions. Weuse four of the GLUE tasks (Wang et al., 2019):

MNLI A multi-genre natural language inferencedataset (Williams et al., 2018), where the task is to

predict whether a premise entails a hypothesis.

QNLI A conversion of the Stanford question an-swering dataset (Rajpurkar et al., 2016), where thetask is to determine whether a sentence containsthe answer to a question.

QQP A collection of question pairs from theQuora website, where the task is to determinewhether two questions are semantically equivalent.

SST-2 A binary sentiment analysis task using theStanford sentiment treebank (Socher et al., 2013).

6.1 Results

Top layers are more affected by fine-tuningFigure 5 shows representation-level ckasim simi-larity heatmaps of pre-trained (not fine-tuned) andfine-tuned versions of BERT and XLNet. The moststriking pattern is that the top layers are more af-fected by fine-tuning than the bottom layers, asevidenced by the low similarity of high layers ofthe pre-trained models with their fine-tuned coun-terparts. Hao et al. (2019) also observed that lowerlayers of BERT are less affected by fine-tuningthan top layers, by visualizing the training losssurfaces.13 In Appendix D, we demonstrate thatthis observation can motivate a more efficient fine-tuning process, where some of the layers are frozenwhile others are fine-tuned.

There are some task-specific differences. InBERT, the top layers of the SST-2-fine-tuned model

13A reviewer commented that this pattern seems like a nat-ural consequence of back-propagation, which we concur with,although in on-going work we found that middle layers of XL-Net lead to more gains when fine-tuned. Future work can alsoexplore the effect of optimization on the similarity measures.

4645

(a) BERT (b) XLNet

Figure 5: ckasim similarity heatmaps of layers in base (pre-trained, not fine-tuned) and fine-tuned models.

(a) BERT (b) XLNet

Figure 6: Jensen–Shannon attention similarity heatmaps of layers in base (pre-trained, not fine-tuned) and fine-tuned models.

are affected more than other layers. This may be be-cause SST-2 is a sentence classification task, whilethe other tasks are sentence-pair classification. Apotential implication of this is that non-SST-2 taskscan contribute to one another in a multi-task fine-tuning setup. In contrast, in XLNet, fine-tuningon any task leads to top layers being very differentfrom all layers of models fine-tuned on other tasks.This suggests that XLNet representations becomevery task-specific, and thus multi-task fine-tuningmay be less effective with XLNet than with BERT.

Observing the attnsim similarity based onJensen–Shannon divergence for base and fine-tunedmodels (Figure 6), we again see that top layershave lower similarities, implying that they undergogreater changed during fine-tuning. Other attention-based measures behaved similarly (not shown). Ko-

valeva et al. (2019) made a similar observation bycomparing the cosine similarity of attention matri-ces in BERT, although they did not perform cross-task comparisons. In fact, the diagonals withineach block indicate that bottom layers remain sim-ilar to one another even when fine-tuning on dif-ferent tasks, while top layers diverge after fine-tuning. The vertical bands at layers 0 mean thatmany higher layers have a head that is very similarto a head from the first layer, that is, a form ofredundancy, which can explain why many headscan be pruned (Michel et al., 2019; Voita et al.,2019b; Kovaleva et al., 2019). Comparing BERTand XLNet, the vertical bands at the top layers ofBERT (especially in MNLI, QQI, and SST-2) sug-gest that some top layers are very similar to anyother layer. In XLNet, top MNLI layers are quite

4646

(a) BERT (b) XLNet

Figure 7: Localization scores per layer in base and fine-tuned models.

different from any other layer. Thus different objec-tive functions impact the attention heads differentlyunder fine-tuning.

Fine-tuning affects localization Figure 7 showslocalization scores for different layers in pre-trained and fine-tuned models. In contrast to thepre-trained models, the fine-tuned ones decrease inlocalization at the top layers. This decrease maybe the result of top layers learning high-level tasks,which require multiple neurons to capture properly.

7 Conclusion

In this work, we analyzed various prominent con-textual word representations from the perspectiveof similarity analysis. We compared different lay-ers of pre-trained models using both localized anddistributed measures of similarity, at neuron, rep-resentation, and attention levels. We found thatdifferent architectures often have similar internalrepresentations, but differ at the level of individualneurons. We also observed that higher layers aremore localized than lower ones. Comparing fine-tuned and pre-trained models, we found that higherlayers are more affected by fine-tuning in their rep-resentations and attention weights, and become lesslocalized. These findings motivated experimentingwith layer-selective fine-tuning, where we wereable to obtain good performance while freezing thelower layers and only fine-tuning the top ones.

Our approach is complementary to the linguis-tic analysis of models via probing classifiers. Anexciting direction for future work is to combinethe two approaches in order to identify which lin-guistic properties are captured in model compo-nents that are similar to one another, or explicatehow localization of information contributes to the

learnability of particular properties. It may be in-sightful to compare the results of our analysis tothe loss surfaces of the same models, especiallybefore and after fine-tuning (Hao et al., 2019). Onecould also study whether a high similarity entailthat two models converged to a similar solution.Our localization score can also be compared toother aspects of neural representations, such as gra-dient distributions and their relation to memoriza-tion/generalization (Arpit et al., 2017). Finally, thesimilarity analysis may also help improve modelefficiency, for instance by pointing to componentsthat do not change much during fine-tuning and canthus be pruned.

Acknowledgements

We thank Nelson Liu for providing some of therepresentations analyzed in this work. We alsothank the anonymous reviewers for their many valu-able comments. This research was carried out incollaboration between the HBKU Qatar Comput-ing Research Institute (QCRI) and the MIT Com-puter Science and Artificial Intelligence Laboratory(CSAIL). Y.B. is also supported by the HarvardMind, Brain, and Behavior Initiative (MBB).

ReferencesYossi Adi, Einat Kermany, Yonatan Belinkov, Ofer

Lavi, and Yoav Goldberg. 2017. Fine-grained anal-ysis of sentence embeddings using auxiliary predic-tion tasks. In Proceedings of the Internaltional Con-fernece for Learning Representations (ICLR).

Galen Andrew, Raman Arora, Jeff Bilmes, and KarenLivescu. 2013. Deep canonical correlation analysis.In Proceedings of the 30th International Conferenceon Machine Learning, volume 28 of Proceedings of

4647

Machine Learning Research, pages 1247–1255, At-lanta, Georgia, USA. PMLR.

Devansh Arpit, Stanisław Jastrzebski, Nicolas Bal-las, David Krueger, Emmanuel Bengio, Maxin-der S. Kanwal, Tegan Maharaj, Asja Fischer, AaronCourville, Yoshua Bengio, and Simon Lacoste-Julien. 2017. A closer look at memorization in deepnetworks. In Proceedings of the 34th InternationalConference on Machine Learning, volume 70 of Pro-ceedings of Machine Learning Research, pages 233–242, International Convention Centre, Sydney, Aus-tralia. PMLR.

Joris Baan, Jana Leible, Mitja Nikolaus, David Rau,Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes,and Elia Bruni. 2019. On the realization of compo-sitionality in neural networks. In Proceedings of the2019 ACL Workshop BlackboxNLP: Analyzing andInterpreting Neural Networks for NLP, pages 127–137, Florence, Italy. Association for ComputationalLinguistics.

D. Anthony Bau, Yonatan Belinkov, Hassan Sajjad,Nadir Durrani, Fahim Dalvi, and James Glass. 2019.Identifying and controlling important neurons inneural machine translation. In International Confer-ence on Learning Representations (ICLR).

Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, HassanSajjad, and James Glass. 2017. What do Neural Ma-chine Translation Models Learn about Morphology?In Proceedings of the 55th Annual Meeting of the As-sociation for Computational Linguistics (ACL), Van-couver. Association for Computational Linguistics.

Yonatan Belinkov and James Glass. 2019. Analysismethods in neural language processing: A survey.Transactions of the Association for ComputationalLinguistics, 7:49–72.

Diane Bouchacourt and Marco Baroni. 2018. Howagents see things: On visual representations in anemergent language game. In Proceedings of the2018 Conference on Empirical Methods in NaturalLanguage Processing, pages 981–985, Brussels, Bel-gium. Association for Computational Linguistics.

Jeffrey S Bowers. 2002. Challenging the widespreadassumption that connectionism and distributed repre-sentations go hand-in-hand. Cognitive Psychology,45(3):413 – 445.

Gino Brunner, Yang Liu, Damian Pascual, OliverRichter, Massimiliano Ciaramita, and Roger Watten-hofer. 2020. On identifiability in transformers. InInternational Conference on Learning Representa-tions.

Grzegorz Chrupała. 2019. Symbolic inductive biasfor visually grounded learning of spoken language.In Proceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics, pages6452–6462, Florence, Italy. Association for Compu-tational Linguistics.

Grzegorz Chrupała and Afra Alishahi. 2019. Corre-lating neural and symbolic representations of lan-guage. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguis-tics, pages 2952–2962, Florence, Italy. Associationfor Computational Linguistics.

Kevin Clark, Urvashi Khandelwal, Omer Levy, andChristopher D. Manning. 2019. What does BERTlook at? an analysis of BERT’s attention. In Pro-ceedings of the 2019 ACL Workshop BlackboxNLP:Analyzing and Interpreting Neural Networks forNLP, pages 276–286, Florence, Italy. Associationfor Computational Linguistics.

Alexis Conneau, German Kruszewski, Guillaume Lam-ple, Loïc Barrault, and Marco Baroni. 2018. Whatyou can cram into a single $&!#* vector: Probingsentence embeddings for linguistic properties. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 2126–2136, Melbourne, Aus-tralia. Association for Computational Linguistics.

Fahim Dalvi, Nadir Durrani, Hassan Sajjad, YonatanBelinkov, D Anthony Bau, and James Glass. 2019.What is one grain of sand in the desert? analyzingindividual neurons in deep NLP models. In Pro-ceedings of the AAAI Conference on Artificial Intel-ligence (AAAI).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), Min-neapolis, Minnesota. Association for ComputationalLinguistics.

Kawin Ethayarajh. 2019. How contextual are contex-tualized word representations? comparing the geom-etry of BERT, ELMo, and GPT-2 embeddings. InProceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 55–65,Hong Kong, China. Association for ComputationalLinguistics.

Allyson Ettinger, Ahmed Elgohary, and Philip Resnik.2016. Probing for semantic evidence of compositionby means of simple classification tasks. In Proceed-ings of the 1st Workshop on Evaluating Vector-SpaceRepresentations for NLP, pages 134–139, Berlin,Germany. Association for Computational Linguis-tics.

Bjarke Felbo, Alan Mislove, Anders Søgaard, IyadRahwan, and Sune Lehmann. 2017. Using millionsof emoji occurrences to learn any-domain represen-tations for detecting sentiment, emotion and sarcasm.In Proceedings of the 2017 Conference on Empiri-cal Methods in Natural Language Processing, pages

4648

1615–1625, Copenhagen, Denmark. Association forComputational Linguistics.

Ross W. Gayler and Simon D. Levy. 2011. Compo-sitional connectionism in cognitive science ii: thelocalist/distributed dimension. Connection Science,23(2):85–89.

Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visu-alizing and understanding the effectiveness of BERT.In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 4141–4150, Hong Kong, China. Association for Computa-tional Linguistics.

Geoffrey E Hinton. 1984. Distributed representations.Technical Report CMU-CS-84-157, Carnegie Mel-lon University.

Jeremy Howard and Sebastian Ruder. 2018. Universallanguage model fine-tuning for text classification. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 328–339, Melbourne, Australia.Association for Computational Linguistics.

Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema.2018. Visualisation and ‘diagnostic classifiers’ re-veal how recurrent and recursive neural networksprocess hierarchical structure. Journal of ArtificialIntelligence Research, 61:907–926.

Sarthak Jain and Byron C. Wallace. 2019. Attention isnot Explanation. In Proceedings of the 2019 Con-ference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long and Short Pa-pers), pages 3543–3556, Minneapolis, Minnesota.Association for Computational Linguistics.

Simon Kornblith, Mohammad Norouzi, Honglak Lee,and Geoffrey Hinton. 2019. Similarity of neuralnetwork representations revisited. In Proceedingsof the 36th International Conference on MachineLearning, volume 97 of Proceedings of MachineLearning Research, pages 3519–3529, Long Beach,California, USA. PMLR.

Olga Kovaleva, Alexey Romanov, Anna Rogers, andAnna Rumshisky. 2019. Revealing the dark secretsof BERT. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processingand the 9th International Joint Conference on Natu-ral Language Processing (EMNLP-IJCNLP), pages4364–4373, Hong Kong, China. Association forComputational Linguistics.

Nikolaus Kriegeskorte, Marieke Mur, and Peter Ban-dettini. 2008. Representational similarity analysis- connecting the branches of systems neuroscience.Frontiers in Systems Neuroscience, 2:4.

Yair Lakretz, German Kruszewski, Theo Desbordes,Dieuwke Hupkes, Stanislas Dehaene, and Marco Ba-roni. 2019. The emergence of number and syn-tax units in LSTM language models. In Proceed-ings of the 2019 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume 1(Long and Short Papers), pages 11–20, Minneapolis,Minnesota. Association for Computational Linguis-tics.

Jian Li, Xing Wang, Baosong Yang, Shuming Shi,Michael R Lyu, and Zhaopeng Tu. 2020. Neuron in-teraction based representation composition for neu-ral machine translation. In Proceedings of the AAAIConference on Artificial Intelligence (AAAI).

Nelson F. Liu, Matt Gardner, Yonatan Belinkov,Matthew E. Peters, and Noah A. Smith. 2019a. Lin-guistic knowledge and transferability of contextualrepresentations. In Proceedings of the 2019 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long and Short Pa-pers), pages 1073–1094, Minneapolis, Minnesota.Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019b.Roberta: A robustly optimized bert pretraining ap-proach. arXiv preprint arXiv:1907.11692.

Mitchell P. Marcus, Beatrice Santorini, and Mary AnnMarcinkiewicz. 1993. Building a large annotatedcorpus of English: The Penn Treebank. Computa-tional Linguistics, 19(2):313–330.

Paul Michel, Omer Levy, and Graham Neubig. 2019.Are sixteen heads really better than one? In H. Wal-lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc,E. Fox, and R. Garnett, editors, Advances in NeuralInformation Processing Systems 32, pages 14014–14024. Curran Associates, Inc.

Ari Morcos, Maithra Raghu, and Samy Bengio. 2018.Insights on representational similarity in neural net-works with canonical correlation. In S. Bengio,H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neu-ral Information Processing Systems 31, pages 5727–5736. Curran Associates, Inc.

Mike Page. 2000. Connectionist modelling in psychol-ogy: A localist manifesto. Behavioral and BrainSciences, 23(4):443âAS467.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018a. Deep contextualized word rep-resentations. In Proceedings of the 2018 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long Papers), pages2227–2237, New Orleans, Louisiana. Associationfor Computational Linguistics.

4649

Matthew Peters, Mark Neumann, Luke Zettlemoyer,and Wen-tau Yih. 2018b. Dissecting contextualword embeddings: Architecture and representation.In Proceedings of the 2018 Conference on Em-pirical Methods in Natural Language Processing,pages 1499–1509, Brussels, Belgium. Associationfor Computational Linguistics.

Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016.Analyzing linguistic knowledge in sequential modelof sentence. In Proceedings of the 2016 Conferenceon Empirical Methods in Natural Language Process-ing, pages 826–835, Austin, Texas. Association forComputational Linguistics.

Alec Radford, Rafal Jozefowicz, and Ilya Sutskever.2017. Learning to generate reviews and discoveringsentiment. arXiv preprint arXiv:1704.01444.

Alec Radford, Karthik Narasimhan, Tim Salimans, andIlya Sutskever. 2018. Improving language under-standing by generative pre-training.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners. OpenAIBlog, 1(8).

Maithra Raghu, Justin Gilmer, Jason Yosinski, andJascha Sohl-Dickstein. 2017. SVCCA: Singular vec-tor canonical correlation analysis for deep learningdynamics and interpretability. In I. Guyon, U. V.Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish-wanathan, and R. Garnett, editors, Advances in Neu-ral Information Processing Systems 30, pages 6078–6087. Curran Associates, Inc.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. SQuAD: 100,000+ questions formachine comprehension of text. In Proceedings ofthe 2016 Conference on Empirical Methods in Natu-ral Language Processing, pages 2383–2392, Austin,Texas. Association for Computational Linguistics.

Naomi Saphra and Adam Lopez. 2019. Understand-ing learning dynamics of language models withSVCCA. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 3257–3267, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Sofia Serrano and Noah A. Smith. 2019. Is attentioninterpretable? In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 2931–2951, Florence, Italy. Associa-tion for Computational Linguistics.

Xing Shi, Kevin Knight, and Deniz Yuret. 2016. Whyneural translations are the right length. In Proceed-ings of the 2016 Conference on Empirical Methodsin Natural Language Processing, pages 2278–2282,Austin, Texas. Association for Computational Lin-guistics.

Ravid Shwartz-Ziv and Naftali Tishby. 2017. Openingthe black box of deep neural networks via informa-tion. arXiv preprint arXiv:1703.00810.

Natalia Silveira, Timothy Dozat, Marie-Catherinede Marneffe, Samuel Bowman, Miriam Connor,John Bauer, and Chris Manning. 2014. A gold stan-dard dependency corpus for English. In Proceedingsof the Ninth International Conference on LanguageResources and Evaluation (LREC’14), pages 2897–2904, Reykjavik, Iceland. European Language Re-sources Association (ELRA).

Richard Socher, Alex Perelygin, Jean Wu, JasonChuang, Christopher D Manning, Andrew Ng, andChristopher Potts. 2013. Recursive deep modelsfor semantic compositionality over a sentiment tree-bank. In Proceedings of EMNLP, pages 1631–1642.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.BERT rediscovers the classical NLP pipeline. InProceedings of the 57th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for ComputationalLinguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In I. Guyon, U. V. Luxburg, S. Bengio,H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar-nett, editors, Advances in Neural Information Pro-cessing Systems 30, pages 5998–6008. Curran Asso-ciates, Inc.

Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. Thebottom-up evolution of representations in the trans-former: A study with machine translation and lan-guage modeling objectives. In Proceedings of the2019 Conference on Empirical Methods in Natu-ral Language Processing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 4395–4405, Hong Kong,China. Association for Computational Linguistics.

Elena Voita, David Talbot, Fedor Moiseev, Rico Sen-nrich, and Ivan Titov. 2019b. Analyzing multi-headself-attention: Specialized heads do the heavy lift-ing, the rest can be pruned. In Proceedings of the57th Annual Meeting of the Association for Com-putational Linguistics, pages 5797–5808, Florence,Italy. Association for Computational Linguistics.

Alex Wang, Amanpreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R. Bowman. 2019.GLUE: A multi-task benchmark and analysis plat-form for natural language understanding. In Inter-national Conference on Learning Representations.

John Wieting and Douwe Kiela. 2019. No train-ing required: Exploring random encoders for sen-tence classification. In International Conference onLearning Representations.

4650

Adina Williams, Nikita Nangia, and Samuel Bowman.2018. A broad-coverage challenge corpus for sen-tence understanding through inference. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume1 (Long Papers), pages 1112–1122, New Orleans,Louisiana. Association for Computational Linguis-tics.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. HuggingFace’s Trans-formers: State-of-the-art natural language process-ing. ArXiv, abs/1910.03771.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car-bonell, Russ R Salakhutdinov, and Quoc V Le.2019. XLNet: Generalized autoregressive pre-training for language understanding. In H. Wal-lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc,E. Fox, and R. Garnett, editors, Advances in Neu-ral Information Processing Systems 32, pages 5753–5763. Curran Associates, Inc.

4651

A Mathematical Details of SimilarityMeasures

We assume a fixed corpus with W =∑

iWi totalwords, and W (2) =

∑iW

2i total pairs. Here Wi is

the number of words in sentence i.A representational layer h(m)

l may be seen asa W × Nm matrix, where Nm is the number ofneurons (per layer) in model m. A single neuronh(m)l [k] (really h

(m)l [:, k]) is a W × 1 column vec-

tor.An attention head α

(m)l [k] may be seen as a ran-

dom variable ranging over sentences si and takingmatrix values α(m)

l [k](si) ∈ Rti×ti , ti = len(si).

A.1 Neuron-level similarity

For a given neuron h(m)l [k], we define

neuronsim(h(m)l [k],h

(m′)l′ ) =

maxk′|ρ(h(m′)

l′ [k′],h(m)l [k])|

as the maximum correlation between it and anotherneuron in some layer (Bau et al., 2019). Here ρ isthe Pearson correlation. This naturally gives rise toan aggregate measure at the layer level:

neuronsim(h(m)l ,h

(m′)l′ ) =

1

Nm

∑k

neuronsim(h(m)l [k],h

(m′)l′ )

A.2 Mixed neuron–representation similarityWe define

mixedsim(h(m)l [k],h

(m′)l′ ) :=

lstsq(h(m′)l′ ,h

(m)l [k]).r

where .r is the r-value associated with the regres-sion, the norm of the prediction divided by thenorm of the regressand. As before, this is extendedto the layer level:

mixedsim(h(m)l ,h

(m′)l′ ) =

1

Nm

∑k

mixedsim(h(m)l [k],h

(m′)l′ )

A.3 Representation-level similarityIn the following, let Z denote a column centeringtransformation. For a given matrix A, the sum ofeach column in ZA is zero.

SVCCA Given two layers

X,Y = Zh(mx)lx

,Zh(my)ly

we compute the truncated principal components

X′,Y′ = Ux[:, : lx],Uy[:, : ly]

where Ux are the left singular vectors of X, andlx is the index required to account for 99% of thevariance. Uy and ly are defined analogously. TheSVCCA correlations, ρSV CCA, are defined as:

u, ρSV CCA,v = SVD(X′TY′)

The SVCCA similarity, svsim(h(mx)lx

,h(my)ly

), isthe mean of ρSV CCA.

PWCCA Identical to SVCCA, except the com-putation of similarity is a weighted mean. Usingthe same notation as above, we define canonicalvectors,

HX := X′u

HY := Y′v

We define alignments

AX := abs(HT

XX)

AY := abs(HT

YY)

where abs is the element-wise absolute value. Theweights are

αx := weights(AX1), αy := weights(AY1)

where 1 is the column vector of all ones, andweights normalizes a vector to sum to 1. ThePWCCA similarity is

pwsim(h(mx)lx

,h(my)ly

) := αTx ρSV CCA

pwsim(h(my)ly

,h(mx)lx

) := αTy ρSV CCA

It is asymmetric.

CKA We use the same notation as above. Giventwo layers,

X,Y = Zh(mx)lx

,Zh(my)ly

the CKA similarity is

ckasim(h(mx)lx

,h(my)ly

) :=

∥∥XTY∥∥2

‖XTX‖ ‖YTY‖where ‖·‖ is the Frobenius norm. It is symmetric.

A.4 Attention-level similarity

We define

attnsim(α(m)l [k],α

(m′)l′ ) =

maxk′

[Sim(α

(m′)l′ [k′],α

(m)l [k])

]We consider three such values of Sim.

• Matrix norm: for each sentence si, compute theFrobenius norm

∥∥∥α(m′)l′ [h′](si)−α

(m)l [h](si)

∥∥∥.Then average over sentences in the corpus.

• Pearson correlation: for every word xi, com-pare the attention distributions the two heads

4652

induce from xi to all words under Pearson cor-relation: ρ

(α

(m′)l′,i [h′],α

(m)l,i [h]

). Then average

over words in the corpus.

• Jensen–Shannon divergence: for every wordxi, compare the attention distributions underJensen–Shannon divergence: 1

2 KL(α(m′)l′,i [h′]

∥∥β) + 1

2 KL(α(m)l,i [h]

∥∥ β), where KL is the KL-divergence and β is the average of the two atten-tion distributions. Then average of words in thecorpus.

As before, this gives rise to aggregate measures atthe layer level by averaging over heads h.

B Additional Representation-levelSimilarity Heatmaps

Figure 8 shows additional representation-level sim-ilarity heatmaps.

B.1 Effect of Data Used for SimilarityMeasures

The majority of the experiments reported in thepaper are using the Penn Treebank for calculatingthe similarity measures. Here we show that theresults are consistent when using a different dataset,namely the Universal Dependencies English WebTreebank (Silveira et al., 2014). We repeat theexperiment reported in Section 5.1. The resultingheatmaps, shown in Figure 9, are highly similar tothose generated using the Penn Treebank, shownin Figure 8.

C Additional Attention-level SimilarityHeatmaps

Figure 10 shows additional attention-level similar-ity heatmaps.

D Efficient Fine-tuning

The analysis results showed that lower layers ofthe models go through limited changes during fine-tuning compared to higher layers. We use thisinsight to improve the efficiency of the fine-tuningprocess. In standard fine-tuning, back-propagationis done on the full network. We hypothesize thatwe can reduce the number of these operations byfreezing the lower layers of the model since theyare the least affected during the fine-tuning process.We experiment with freezing top and bottom layersof the network during the fine-tuning process. Dif-ferent from prior work (Raghu et al., 2017; Felbo

Froze SST-2 MNLI QNLI QQP

BE

RT

0 92.43 84.05 91.40 91.00

Top 4 91.86 82.86 91.09 90.97Bot. 4 92.43 84.16 91.85 90.86Top 6 91.97 82.53 90.13 90.61Bot. 6 93.00 84.00 91.80 90.71

XL

Net

0 93.92 85.97 90.35 90.55

Top 4 92.89 85.55 87.96 90.92Bot. 4 93.12 86.04 90.65 89.36Top 6 93.12 84.84 87.88 90.75Bot. 6 93.92 85.64 90.99 89.02

Table 1: Freezing top/bottom 4/6 layers of BERT andXLNet during fine-tuning.

et al., 2017; Howard and Ruder, 2018), we freezethe selected layers for the complete fine-tuning pro-cess in contrast to freezing various layers for afraction of the training time. We use the defaultparameters settings provided in the Transformerlibrary (Wolf et al., 2019): batch size = 8, learningrate = 5e−5, Adam optimizer with epsilon = 1e−8,and number of epochs = 3.

Table 1 presents the results on BERT and XL-Net. On all of the tasks except QQP, freezing thebottom layers resulted in better performance thanfreezing the top layers. One interesting observationis that as we increase the number of bottom lay-ers for freezing to six, the performance marginallydegrades while saving a lot more computation. Sur-prisingly, on SST-2 and QNLI, freezing the bottomsix layers resulted in better or equal performancethan not freezing any layers of both models. Withfreezing the bottom six layers, one can save back-propagation computation by more than 50%.

4653

(a) ckasim (b) svsim

(c) pwsim (d) mixedsim

Figure 8: Similarity heatmaps of layers in various models under different representation-level similarity measures.

4654

(a) neuronsim

(b) ckasim (c) svsim

(d) pwsim (e) mixedsim

Figure 9: Similarity heatmaps of layers in various models under neuron-level and representation-level similaritymeasures, using the English Web Treebank corpus.

4655

(a) Matrix norm (b) Jensen–Shannon

(c) Pearson (d) svsim

(e) pwsim (f) ckasim

Figure 10: Similarity heatmaps of layers in various models under different attention-level similarity measures.