Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing, pages 88–99,

Hong Kong, China, November 3–7, 2019. c©2019 Association for Computational Linguistics

88

Game Theory Meets Embeddings:a Unified Framework for Word Sense Disambiguation

Rocco TripodiCa’ Foscari University of Venicerocco.tripodi@unive.it

Roberto NavigliSapienza University of Rome

navigli@di.uniroma1.it

Abstract

Game-theoretic models, thanks to their intrin-sic ability to exploit contextual information,have shown to be particularly suited for theWord Sense Disambiguation task. They rep-resent ambiguous words as the players of anon-cooperative game and their senses as thestrategies that the players can select in orderto play the games. The interaction among theplayers is modeled with a weighted graph andthe payoff as an embedding similarity func-tion, which the players try to maximize. Theimpact of the word and sense embedding rep-resentations in the framework was tested andanalyzed extensively: experiments on standardbenchmarks show state-of-art performancesand different tests hint at the usefulness ofusing disambiguation to obtain contextualizedword representations.

1 Introduction

Word Sense Disambiguation (WSD), the task oflinking the appropriate meaning from a sense in-ventory to words in a text, is an open problem inNatural Language Processing (NLP). It is particu-larly challenging because it deals with the seman-tics of words and, by their very nature, words areambiguous and can be used with different mean-ings in different situations. Among the key tasksaimed at enabling Natural Language Understand-ing (Navigli, 2018), WSD provides a basic, solidcontribution since it is able to identify the intendedmeaning of the words in a sentence (Kim et al.,2010).

WSD can be seen as a classification task inwhich words are the objects to be classified andsenses are the classes into which the objects haveto be classified (Navigli, 2009); therefore it ispossible to use supervised learning techniques tosolve the WSD problem. One drawback withthis idea is that it requires large amounts of data

that are difficult to obtain. Furthermore, in theWSD context, the production of annotated datais even more complicated and excessively time-consuming compared to other tasks. This arisesbecause of the variability in lexical use. Further-more, the number of different meanings to be con-sidered in a WSD task is in the order of thousands,whereas classical classification tasks in machinelearning have considerably fewer classes.

We decided to adopt a semi-supervised ap-proach to overcome the knowledge acquisitionbottleneck and innovate the strand of research in-troduced by Tripodi and Pelillo (2017). Theseresearchers developed a flexible game-theoreticWSD model that exploits word and sense simi-larity information. This combination of featuresallows the textual coherence to be maintained: infact, in this model the disambiguation process isrelational, and the sense assigned to a word mustalways be compatible with the senses of the wordsin the same text. It can be seen as a constraintsatisfaction model which aims to find the bestconfiguration of senses for the words in context.This is possible because the payoff function ofthe games is modeled in a way in which, when agame is played between two players, they are em-boldened to select the senses that have the highestcompatibility with the senses that the co-player ischoosing. Another appealing feature of this modelis that it offers the possibility to configure manycomponents of the system: it is possible to useany word and sense representation; also, one canmodel the interactions of the players in differentways by exploiting word similarity information,the syntactic structure of the sentence and the im-portance provided by specific relations. Further-more, it is possible to use different priors on thesense distributions and to use different game dy-namics to find the equilibrium state of the model.Traditional WSD methods have only some of these

89

properties.The main difference between our model and the

model proposed by Tripodi and Pelillo (2017) isthat they did not use state-of-the-art models forword and sense representations. They used wordco-occurrence measures for word similarity and tf-idf vectors for sense similarity, resulting in sparsegraphs in which nodes can be disjoint or some se-mantic area is not covered. Instead, we are advo-cating the use of dense vectors, which provide acompletely different perspective not only in termsof representation but also in terms of dynamics.Each player is involved in many more games andthis affects the computation of the payoffs and theconvergence of the system. The interaction amongthe players is defined in a different way and thepriors are modeled with a more realistic distribu-tion to avoid the skewness typical of word sensedistributions. Furthermore, our model is evaluatedon recent standard benchmarks, facilitating com-parison with other models.

The main contributions of this paper are as fol-lows:

1. the release of a general framework for WSD;

2. the evaluation of different word and senseembeddings;

3. state-of-the-art performances on standardbenchmarks (in different cases performingbetter than recent supervised models);

4. the use of disambiguated sense vectors to ob-tain contextualized word representations.

2 Word Sense Disambiguation

WSD approaches can be divided into two maincategories: supervised, which require humanintervention in the creation of sense-annotateddatasets, and the so-called knowledge-based ap-proaches (Navigli, 2009), which require the con-struction of a task-independent lexical-semanticknowledge resource, but which, once that workis available, use models that are completely au-tonomous.

As regards supervised systems, a popular sys-tem is It makes sense (Zhong and Ng, 2010), amodel which takes advantage of standard WSDfeatures such as POS-tags, word co-occurrences,and collocations and creates individual supportvector machine classifiers for each ambiguousword. Newer supervised models exploit deep

neural networks and especially long short-termmemory (LSTM) networks, a type of recurrentneural network particularly suitable for handlingarbitrary-length sequences. Yuan et al. (2016)proposed a deep neural model trained with largeamounts of data obtained in a semi-supervisedfashion. This model was re-implemented by Leet al. (2018), reaching comparable results with asmaller training corpus. Raganato et al. (2017)introduced two approaches for neural WSD us-ing models developed for machine translation andsubstituting translated words with sense-annotatedones. A recent work that combines labeled dataand knowledge-based information has been pro-posed by Luo et al. (2018). Uslu et al. (2018)proposed fastSense, a model inspired by fastText(Joulin et al., 2017) which – rather than predictingcontext words – predicts word senses.

Knowledge-based models, instead, exploit thestructural properties of a lexical-semantic knowl-edge base, and typically use the relational infor-mation between concepts in the semantic graphtogether with the lexical information containedtherein (Navigli and Lapata, 2010). A popular al-gorithm used to select the sense of each word inthis graph is PageRank (Page et al., 1999) that per-forms random walks over the network to identifythe most important nodes (Haveliwala, 2002; Mi-halcea et al., 2004; De Cao et al., 2010). An ex-tension of these models was proposed by Agirreet al. (2014) in which the Personalized PageRankalgorithm is applied. Another knowledge-basedapproach is Babelfy (Moro et al., 2014), which de-fines a semantic signature for a given context andcompares it with all the candidate senses in orderto perform the disambiguation task. Chaplot andSalakhutdinov (2018) proposed a method that usesthe whole document as the context for the words tobe disambiguated, exploiting topical information(Ferret and Grau, 2002). It models word sensesusing a variant of the Latent Dirichlet Allocationframework (Blei et al., 2003), in which the topicdistributions of the words are replaced with sensedistributions modeled with a logistic normal distri-bution according to the frequencies obtained fromWordNet.

3 Word and Sense Embeddings

A good machine-interpretable representation oflexical features is fundamental for every NLP sys-tem. A system’s performance, however, depends

90

on the quality of the input representations. Fur-thermore, the inclusion of semantic features, in ad-dition to lexical ones, has been proven effective inmany NLP approaches (Li and Jurafsky, 2015).

Word embeddings, the current paradigm for lex-ical representation of words, were popularizedwith word2vec (Mikolov et al., 2013). The mainidea is to exploit a neural language model whichlearns to predict a word occurrence given its sur-roundings. Another well-known word embeddingmodel was presented by Pennington et al. (2014),which shares the idea of word2vec, but with thedifference that it uses explicit latent representa-tions obtained from statistical calculation on wordco-occurrences. However, all word embeddingmodels share a common issue: they cannot cap-ture polysemy since they conflate the various wordsenses into a single vector representation. Sev-eral efforts have been presented so far to deal withthis problem. SensEmbed (Iacobacci et al., 2015)uses a knowledge-based disambiguation system tobuild a sense-annotated corpus that, in its turn, isused to train a vector space model for word senseswith word2vec. AutoExtend (Rothe and Schutze,2015), instead, is initialized with a set of pre-trained word embeddings, and induces sense andsynset vectors in the same semantic space usingan autoencoder. The vectors are induced by con-straining their representation given the assumptionthat synsets are sums of their lexemes. Camacho-Collados et al. (2015) presented NASARI, an ap-proach that learns sense vectors by exploiting thehyperlink structure of the English Wikipedia, link-ing their representations to the semantic networkof BabelNet (Navigli and Ponzetto, 2012). Morerecent works, such as ELMo (Peters et al., 2018)and BERT (Devlin et al., 2019), are based on lan-guage models learned using complex neural net-work architectures. The advantage of these mod-els is that they can produce different representa-tions of words according to the context in whichthey appear.

4 Game Theory and Game Dynamics

In this work we take a different approach to WSDby employing a model based on game theory (GT).This discipline was introduced by Neuman andMorgenstern (1944) in order to develop a math-ematical framework able to model the essentialsof decision making in interactive situations. Inits normal-form representation (Weibull, 1997), it

consists of a finite set of players N = (1, .., n),a finite set of pure strategies Si = {1, ...,mi} foreach player i ∈ N , and a payoff (utility) functionui : S → R, that associates a payoff with eachcombination of strategies in S = S1×S2×...×Sn.A fundamental assumption in game theory is thateach player i tries to maximize the value of ui.Furthermore, in non-cooperative games the play-ers choose their strategies independently, consid-ering what choices other players can make and try-ing to find the best response to the strategy of theco-players.

A player i, in addition to playing single (pure)strategies from Si, can also use mixed strategies,that are probability distributions over pure strate-gies. A mixed strategy over Si is defined as a vec-tor xi = (x1, . . . , xmi), such that xj ≥ 0 and∑xj = 1. Each mixed strategy corresponds to

a point in the simplex ∆m, whose corners cor-respond to pure strategies. The intuition is thatplayer i randomises over strategies according tothe probabilities in xi. Each mixed strategy pro-file lives in the mixed strategy space of the game,given by the Cartesian product Θ = ∆m1×∆m2×· · · ×∆mn .

In a two-player game, a strategy profile can bedefined as a pair (xi,xj). The expected payoff forthis strategy profile is computed as:

u(xi,xj) = xTi ·Aijxj

where Aij is the mi ×mj payoff matrix betweenplayers i and j.

In evolutionary game theory (Weibull, 1997),the games are played repeatedly and the playersupdate their mixed strategy distributions over timeuntil no player can improve the payoff obtainedwith the current mixed strategy. This situation cor-responds to the equilibrium of the system.

The payoff corresponding to the h-th pure strat-egy is computed as:

u(xhi ) = xhi ·ni∑j=1

(Aijxj)h (1)

It is important to note here that the payoff in Equa-tion 1 is additively separable, in fact, the summa-tion is over all the ni players with whom i is play-ing the games. The average payoff of player i iscalculated as:

u(xi) =

mi∑h=1

u(xhi ) (2)

91

To find the Nash equilibrium of the game it is com-mon to use the discrete time version of the repli-cator dynamics equation (Weibull, 1997) for eachplayer i ∈ N ,

xhi (t+ 1) = xhi (t)u(xhi )

u(xi)∀ h ∈ xi (3)

This equation allows better than average strategiesto grow at each iteration. It can be considered asan inductive learning process, in which the play-ers learn from past experiences how to play theirbest strategy. We note that each player optimizestheir individual strategy space, but this operationis done according to what other players simulta-neously are doing, so the local optimization is theresult of a global process.

Game-theoretic models are appealing becausethey are versatile, interpretable and have a solidmathematical foundation. Furthermore, it is al-ways possible to find the Nash equilibrium innon-cooperative games in mixed strategies (Nash,1951). In fact, starting from an interior point ofΘ, a point x is a Nash equilibrium only if it is thelimit of a trajectory of Equation 3 (Weibull, 1997).

Figure 1: Generic scheme of the model. ·, × and σrefer to elementwise multiplication, matrix multiplica-tion and normalization, respectively.

5 The Model

The model used in this paper, Word Sense Dis-ambiguation Games (WSDG), was introduced byTripodi and Pelillo (2017). It is based on graph-theoretic principles to model the geometry of thedata and on game theory to model the learning al-gorithm which disambiguates the words in a text.It represents the words as the players of a non-cooperative game and their senses as the strategythat the players can select in order to play thegames. The players are arranged in a graph whoseedges determine the interactions and carry wordsimilarity information. The payoff matrix is en-

coded as a sense similarity function. The play-ers play the games repeatedly and – at each iter-ation – update their strategy preferences accord-ing to what strategy has been effective in previ-ous games. These preferences, as introduced pre-viously, are encoded as a probability distributionover strategies (senses).

Formally, for a text T we select its contentwords W = (1, . . . , n) as the players of the gameI = (1, . . . , n). For each word we use a knowl-edge base to determine its possible senses. Eachsense is represented as a strategy that the playercan select from the set Si = {1, ...,mi}, wheremi

is the number of senses of word wi. The set of alldifferent senses in the text, C = {1, ...,m}, is thestrategy space of the games. The strategy space ismodeled, for each player, as a probability distri-bution, xi, of length m. It takes non-zero valuesonly on the entries corresponding to the elementsof Si. It can be initialized with a normal distribu-tion in the case of unsupervised learning or withinformation obtained from sense-labeled corporain the case of semi-supervised learning.

The payoff of a game depends on a payoff ma-trix Z in which the rows are indexed according tothe strategies of player i and the columns accord-ing to the strategies of player j. Its entries Zr,t arethe payoff obtained when player i selects strategyr and player j selects strategy t. It is important tonote here that the payoff of a game does not de-pend on the single strategy taken individually bya player, but always by the combination of two si-multaneous actions. In WSD this means that thesense selected by a word influences the choices ofthe other words in the text and this allows the tex-tual coherence to be maintained.

The disambiguation games to build a payofffunction require: a word similarity matrix A, asense similarity matrix Z and a sense distributionxi for each player i. A models the players’ inter-actions, so that similar players play together andthe more similar they are the more reciprocal influ-ence they have. It can be interpreted as an attentionmechanism (Vaswani et al., 2017) since it weightsthe payoffs. Z is used to create the payoff matricesof the games so that the more similar the senses ofthe words are the more the corresponding playersare encouraged to select them, since they give ahigh payoff. A and Z are obtained by comput-ing vector representations of word and sense (seeSection 3) and then calculating their pairwise sim-

92

ilarity.The strategy space of each player, i, is repre-

sented as a column vector of lengthm. It is initial-ized with:

xhi =

{|mi|−1 if sense h is in Si,0 otherwise.

(4)

This initialization is used in the case of unsuper-vised WSD, since it does not use information fromsense-tagged corpora. If instead this informationis available, |mi|−1 in Equation 4 is substitutedwith the frequency of the corresponding sense andthen xi is normalized in order to sum up to one.

Once these sources of information are com-puted, the WSDG are run by using the replica-tor dynamic equation (Taylor and Jonker, 1978)in Equation 3, where the payoff of strategy h forplayer i is calculated as:

u(xhi ) = xhi ·ni∑j=1

(AijZxj)h (5)

where ni are the neighbours of player i as in thegraph A. The average payoff is calculated as:

u(xi) =

mi∑h=1

u(xhi ) (6)

The complexity of WSDG scales linearly withthe number of words to be disambiguated. Differ-ently from other models based on PageRank, it ispossible to disambiguate all the words at the sametime. As an example, WSDG can disambiguate200 words (1650 senses) in 7 seconds, on a singleCPU core. A generic representation of the modelis proposed in Figure 1.

Implementation details The cosine similaritywas used as similarity measure for both words andsenses. The A matrix was treated as the adjacencymatrix of an undirected weighted graph and, to re-duce the complexity of the model, the edges withweight lower than 0.1 were removed. The sym-metric normalized Laplacian of this graph was cal-culated as D−1/2AD−1/2, where D is the degreematrix of graph A. Since the algorithm operateson an entire text, local information is added to ma-trix A. The mean value of the matrix is addedto the dlog(n)e cells on the left of the main di-agonal. For BERT, this operation was replacedwith its attention layer, adding to matrix A themean attention distribution of all the heads of the

last layer. The choice of the last layer is moti-vated by the fact that it stores semantic informa-tion and its attention distributions have high en-tropy (Clark et al., 2019). The first singular vec-tor was removed from A in the case of word vec-tors whose length exceeded 500. This was doneto reduce the redundancy of the representations inline with Arora et al. (2017). The distributionsfor each x were computed according to SemCor(Miller et al., 1993) and normalized using the soft-max function. The replicator dynamics were rununtil a maximum number of iterations was reached(100) or the difference between two consecutive it-erations was below a small threshold (10−3), cal-culated as

∑ni=1 ‖xi(t− 1)− xi(t)‖. The code

of the model is available at https://github.com/roccotrip/wsd_games_emb.

6 Evaluation

The evaluation of our model was conducted usingthe framework proposed by Raganato et al. (2017).This consists of five datasets which were unified tothe same WordNet 3.0 inventory: Senseval-2 (S2),Senseval-3 (S3), SemEval-2007 (SE7), SemEval-2013 (SE13) and SemEval-2015 (SE15). Thesedatasets have in total 26 texts and 10, 619 wordsto be disambiguated. Our objective was to test ourgame-theoretic model with different settings andto evaluate its performances. To this end we per-formed experiments comparing 16 different sets ofpretrained word embeddings and 7 sets of senseembeddings.

Word embeddings As word embedding mod-els we included 4 pre-word2vec models: the hi-erarchical log-bilinear model (Mnih and Hinton,2007, HLBL), a probabilistic linear neural modelwhich aims to predict the embedding of a wordgiven the concatenation of the previous words;CW (Collobert and Weston, 2008), an embeddingsmodel with a deep unified architecture for mul-titask NLP; Distributional Memory (Baroni andLenci, 2010, DM), a semantically enriched count-based model; leskBasile (Basile et al., 2014), amodel based on Latent Semantic Analysis reducedvia Singular-Value Decomposition; 3 models ob-tained with word2vec: GoogleNews, a set of300-dimensions vectors trained with the GoogleNews dataset; BNC-*, vectors of different di-mensions trained on the British National Cor-pus including POS information during training;and w2vR, trained with word2vec on the 2014

93

dump of the English Wikipedia, enriched withretrofitting (Faruqui et al., 2015), a technique toenhance pre-trained embeddings with semantic in-formation. The enrichment was performed us-ing retrofitting’s best configuration, based on theParaphrase Database (Ganitkevitch et al., 2013,PPDB). We also tested GloVe (Pennington et al.,2014), trained with the concatenation of the 2014dump of the English Wikipedia and Gigaword 5,and fastText (Bojanowski et al., 2017) trained onWikipedia 2017, UMBC corpus and the statmt.orgnews dataset.

Figure 2: Performances of the model on the union of alldatasets. The results are presented as F1 for all combi-nations of word and sense embeddings. Word vectorsare on the rows and sense vectors on the columns.

Contextualized word embeddings As contex-tualized embeddings we used ELMo (Peters et al.,2018) in three different configurations, namely:ELMo-avg, weighted sum of its three layers;ELMo-avg emb, weighted sum of its three layersand the embeddings it produces; and ELMo-emb,the word embeddings produced by the model1.We also tested three implementations of BERT(Devlin et al., 2019): base cased (b-c); large un-cased (l-u) and large cased (l-c). They offer pre-trained deep bidirectional representations of words

1TensorFlow models available at https://tfhub.dev/google/elmo/2.

in context2. We used seven configurations foreach model: one for each of the last four layers(numbered from 1 to 4), the sum of these layers,their concatenation and the embedding layer. Wefed all these models with the entire texts of thedatasets. Since BERT uses WordPiece tokeniza-tion, we averaged sub-token embeddings to obtaintoken-level representations.

We also included three models which were builttogether with the sense embeddings introduced be-low.

Sense embeddings As sense embeddings, in ad-dition to the three models introduced in Section 3(AutoExtend, NASARI and SensEmbed), we in-cluded four models: Chen et al. (2014), a uni-fied model which learns sense vectors by train-ing a sense-annotated corpus disambiguated with aframework based on semantic similarity of Word-Net sense definitions; meanBNC, created using aweighted combination of the words from WordNetglosses, using, as word vectors, the set of BNC-200 mentioned earlier; DeConf (Pilehvar and Col-lier, 2016), also linked to WordNet, a model wheresense vectors are inferred in the same semanticspace of pre-trained word embeddings by decom-posing the given word representation into its con-stituent senses; and finally SW2V (Mancini et al.,2017), a model linked to BabelNet which uses ashallow disambiguation step and which, by ex-tending the word2vec architecture, learns wordand sense vectors jointly in the same semanticspace as an emerging feature.

Results The results of these models are reportedin Figure 2. One of the most interesting patternsthat emerges from the heat map is that there aresome combinations of word and sense embeddingsthat always work better than others. Sense vectorsdrive the performance of the system, contributingin great part to the accumulation of payoffs duringthe games. The sense vectors that maintain highperformances are SensEmbed, AutoExtended andChen2014. In particular Chen2014 has high per-formances with all the word embedding combina-tions. While these models are specific sense em-bedding techniques, the construction of BNC-200follows a very simple method, which in view ofthese results can be refined using more principledgloss embedding techniques. The performances of

2PyTorch models available at https://github.com/huggingface/pytorch-transformers.

94

model S2 S3 SE07 SE13 SE15 All N V A R

sem

i-sup

.

MFS 64.7∗ 65.4 53.9 62.9 66.6∗ 64.1 68.1 49.5 74.1 80.6Babelfy 67.0 63.5∗ 51.6∗ 66.4† 70.3 65.5∗ 68.6∗ 49.9 73.2 79.8ppr w2w 68.8 66.1 53.0 68.8 70.3 67.3 - - - -WSD-TM 69.0 66.9 55.6 65.3∗ 69.6 66.9 69.7† 51.2 76.0 80.9

WSDGα 68.7 68.3 58.9 66.4 70.7 67.7 71.1 51.9† 75.4 80.9WSDGβ 68.9 65.5 54.5 67.0 72.8 67.2 70.4 51.3 75.7 80.6WSDGγ 69.3 66.4 56.0† 65.9 70.8 67.2 70.4 51.5 75.1 80.6

sup.

IMS (2010) 70.9† 69.3 61.3 65.3 69.5† 68.9† 70.5 55.8 75.6 82.9IMSw2v 72.2 70.4 62.6 65.9 71.5 70.1 71.9 56.6 75.9 84.7YuanLSTM 73.8 71.8 63.5 69.5 72.6 71.5 - - - -RaganatoBLSTM 72.0 69.1 64.8 66.9 71.5 69.9 71.5 57.5 75.0 83.8GAS 72.2 70.5† - 67.2 72.6 - - - - -fastSense 73.5 73.5 62.4 66.2 73.2 - - - - -

Table 1: Comparison with state-of-the-art algorithms: unsupervised or knowledge-based (unsup.), and supervised(sup.). MFS refers to the MFS heuristic computed on SemCor on each dataset. The results are provided as F1 andthe first result of the semi supervised systems with a statistically significant difference from the best of each datasetis marked with ∗ (χ2, p < 0.1). † indicates the same statistics but including also supervised models.

NASARI are lower compared to lexical vectors:this may be due to our choice to use NASARI-embed, whose vectors have low dimensionality.

The word vectors that have consistently highperformances in association with the three sensevectors mentioned above are BERT, Chen2014,SensEmbed and SW2V. This is not surprisingsince they are able to produce contextualised wordrepresentations, performing, in fact, a preliminarydisambiguation of the words. In particular, SW2Vis specifically tailored for WSD. ELMo and fast-Text perform slightly worse. The vectors con-structed using syntactic information and trained onthe BNC corpus have similar performances to thetheir counterparts trained on larger corpora with-out the use of syntactic information. If we focuson BERT, we can see that it is able to maintainhigh performances (F1 ≈ 67) with all its configu-rations, except for the embedding layers of all themodels (*-emb). The contribution of the sum andconcatenation operations is not significant.

Comparison We performed a comparison with3 configurations of our model, one for each of thethree best sense vectors: WSDGα, obtained us-ing Chen2014 as sense vectors and BERT-l-u-4 asword vectors; WSDGβ , obtained using SensEm-bed as sense vectors and BERT-l-c-4 as word vec-tors; and WSDGγ , obtained using AutoExtendedas sense vectors and BERT-l-u-3 as word vectors.

As comparison systems we included three semi-supervised approaches mentioned above, Babelfy(Moro et al., 2014), pprw2w, the best configura-tion of UKB (Agirre et al., 2018), and WSD-TM,

introduced by Chaplot and Salakhutdinov (2018)(for this model we did not have the possibilityto verify the results since its code is not avail-able). In addition, we also report the performancesof relevant supervised models, namely: It MakesSense (Zhong and Ng, 2010, IMS), Iacobacci et al.(2016), Yuan et al. (2016), Raganato et al. (2017),Joulin et al. (2017) and Uslu et al. (2018).

The results of our evaluation are shown in Table1. As we can see our model achieves state-of-the-art performances on four datasets and on S13 andS15 it performs better than many supervised sys-tems. In general the gap between supervised andsemi-supervised systems is reducing. This encour-ages new research in this direction. Our modelfares particularly well on the disambiguation ofnouns and verbs. However, the main gap betweenour models and supervised systems relies upon thedisambiguation of verbs.

7 Analysis

Polysemy As expected, most of the errors madeby WSDGα are on highly polysemous words. Fig-ure 3 shows that the number of wrong answers in-creases as the number of senses grows, and thatthe number of wrong answers starts to be higherthan that of correct answers when the number ofsenses for a target word is in the range of 10-15senses. The words with the highest number of er-rors are polysemous verbs such as: say (34), make(24), find (21), have, (17), take (15), get, (15) anddo (13). These are words that in many NLP ap-plications (especially those based on distributionalmodels) are treated as stopwords.

95

Sense rank Mancini et al. (2017) show thatsenses which are not the most frequent ones areparticularly challenging and most sense-based ap-proaches fail to represent them properly. In Fig-ure 4 we report the results of WSDGα divided persense rank, where it is possible to see how the per-formances of the system deteriorate as the rankof the correct sense increases. It is interesting tosee that, in the first part of the plot, the perfor-mances follow a regular pattern that resembles apower-law distribution. This requires further anal-ysis beyond the scope of this work, along the linesof Ferrer-i Cancho and Vitevitch (2018).

Figure 3: Correct and wrong answers given byWSDGα grouped by number of senses

Figure 4: Correct and wrong answers given byWSDGα per sense rank.

Priors Corroborating the findings of Pilehvarand Navigli (2014), Postma et al. (2016) con-ducted a series of experiments to study the ef-fect that the variation of sense distributions inthe training set has on the performances of Itmakes sense (Zhong and Ng, 2010). Specifi-cally, they increased the volume of training ex-amples (V) by enriching SemCor with senses in-ferred from BabelNet; increased the number ofleast frequent senses (LFS) (V+LFS); and over-fitted the model constructing a training set pro-portional to the correct sense distribution of thetest set (GOLD,GOLD+LFS). We used the sametraining sets to compute the priors for our system.The results of this analysis are reported in Table 2.These experiments show that increasing the num-

System V V+LFS GOLD GOLD+LFS

IMS 68.9 62.0 86.8 85.4WSDGα 66.4 57.5 88.4 90.8

Table 2: Comparison using different priors.

ber of training examples has a small beneficial ef-fect. Increasing the number of LFS examples leadsto worse results because this is a deviation from areal sense distribution. Further, to work with bet-ter semantic representations, this operation shouldalso be accompanied by a similar selection on thetraining set of sense and word embeddings, other-wise LFS remain underrepresented. Finally, mim-icking the distribution of the test set is more bene-ficial for WSDGα than for IMS, especially whenLFS examples are added, suggesting that semi-supervised systems can better adapt to specific do-mains than supervised systems.

8 Exploratory study

We now present three WSD applications in asmany tasks: selection of context-sensitive embed-dings; sentence similarity; paraphrases detection.

Context-sensitive embeddings We used theWord in Context (WiC) dataset (Pilehvar andCamacho-Collados, 2019) for this task. It contains7466 sentence pairs in which a target word appearsin two different contexts. The task consisted ofpredicting if a target word has the same sense inthe two sentences or not. The aim of this experi-ment was twofold: we wanted to show the useful-ness of contextualized word embeddings obtainedfrom WSD systems and to demonstrate that themodel was able to maintain the textual coherence.The experiments on this dataset were conductedon the development set (1400 sentence pairs). Thecomparison was conducted against state-of-the-art models for contextualized word embeddingsand sense embeddings: Context2Vec (Melamudet al., 2016) based on a bidirectional LSTM lan-guage model; ELMo1, the first LSTM hiddenstate; ELMo3, the weighted sum of the 3 LSTMlayers; BERTbase; BERTlarge. The results of thesesystems were taken from Pilehvar and Camacho-Collados (2019). We note here that all these mod-els, including WSDGα, do not use training data.They are based on a simple threshold-based clas-sifier, tuned on the development set (638 sentencepairs). WSDGα disambiguates all the words ineach pair of sentences separately and, if the cosine

96

C2V ELMo1 ELMo3 BERTbase BERTlarge WSDGα

59.7 57.1 56.3 63.6 63.8 66.2

Table 3: Performance on the WiC dataset.

Pearson Spearman MSE

sense 46.5 43.9 7.9word 39.8 39.9 8.6

Table 4: WSDGα results on the SICK dataset.

similarity among the senses assigned to the targetwords is below a threshold (0.9), it classifies thepair as different senses, and as the same sense oth-erwise. As shown in Table 3 the disambiguationstep has a big impact on the results.

Sentence similarity We used the SICK dataset(Marelli et al., 2014) for this task. It consists of9841 sentence pairs that had been annotated withrelatedness scores on a 5-point rating scale. Weused the test split of this dataset that contains 4906sentence pairs. The aim of this experiment was totest if disambiguated sense vectors can provide abetter representation of sentences than word vec-tors. We used a simple method to test the tworepresentations: it consisted of representing a sen-tence as the sum of the disambiguated sense vec-tors in one case and as the sum of word vectorsin the other case. Once the sentence representa-tions had been obtained for both methods the co-sine similarity was used to measure their related-ness. The results of this experiment are reportedin Table 4 as Pearson and Spearman correlationand Mean Squared Error (MSE). We used the αconfiguration of our model with Chen2014 to rep-resent senses and BERT-l-u-4 to represent words.As we can see the simplicity of the method leadsto low performances for both representations, butsense vectors correlate better than word vectors.

Paraphrase detection We used the test set ofthe Microsoft Research Paraphrase Corpus (Dolanet al., 2004, MRPC) for this task. The corpus con-tains 1621 sentence pairs that have been annotatedwith a binary label: 1 if the two sentences consti-tute a paraphrase and 0 otherwise. In this task wealso used the sum of word vectors and the sum ofdisambiguated sense vectors to represent the sen-tences, and used part of the training set (10%) inorder to tune the threshold parameter below whichthe sentences are not considered paraphrase. The

classification accuracy for the word vectors usedby WSDGα was 58.1 whereas the disambiguatedsense vectors obtained 66.9.

9 Conclusion

In this work we have presented WSDG, a flexi-ble game-theoretic model for WSD. It combinesgame dynamics with most successful word andsense embeddings, therefore showing the poten-tial of an effective combination of the two areas ofgame theory and word sense disambiguation.

Our approach achieves state-of-the-art perfor-mances on four datasets performing particularlywell on the disambiguation of nouns and verbs.Beyond the numerical results, in this paper wehave presented a model able to construct and eval-uate word and sense representations. This is par-ticularly useful since it can serve as a test bed fornew word and sense embeddings. In particular,it will be interesting to test new sense embeddingmodels based on contextual embeddings.

Thanks to the flexibility and scalability of ourmodel, as future work we plan to explore in depthits use in different tasks, such as the creation ofsentence (document) embeddings and lexical sub-stitution. In fact, we believe that using disam-biguated sense vectors, as shown in the context-sensitive embeddings and paraphrase detectionstudies, can offer a more accurate representationand improve the quality of downstream applica-tions such as sentiment analysis and text classifi-cation (see, e.g., (Pilehvar et al., 2017)), machinetranslation and topic modelling. Encouraged bythe good results achieved in our exploratory stud-ies, we plan to develop a new model for con-textualised word embeddings based on a game-theoretic framework.

Acknowledgments

The authors gratefully acknowl-edge the support of the ODYC-CEUS project No. 732942 (first au-thor) and of the ERC Consolida-tor Grant MOUSSE No. 726487(second author) under the EuropeanUnion’s Horizon 2020 research andinnovation programme.

The experiments have been run on the SCSCFcluster of Ca’ Foscari University. The authorsthank Ignacio Iacobacci for preliminary work onthis paper.

97

ReferencesEneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa.

2018. The risk of sub-optimal use of open sourceNLP software: UKB is inadvertently state-of-the-artin knowledge-based WSD. In Proceedings of Work-shop for NLP Open Source Software (NLP-OSS),pages 29–33, Melbourne, Australia. ACL.

Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa.2014. Random walks for knowledge-based wordsense disambiguation. Computational Linguistics,40(1):57–84.

Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017.A simple but tough-to-beat baseline for sentence em-beddings. In International Conference on LearningRepresentations.

Marco Baroni and Alessandro Lenci. 2010. Dis-tributional memory: A general framework forcorpus-based semantics. Computational Linguis-tics, 36(4):673–721.

Pierpaolo Basile, Annalina Caputo, and Giovanni Se-meraro. 2014. An enhanced Lesk word sense dis-ambiguation algorithm through a distributional se-mantic model. In Proceedings of COLING 2014,the 25th International Conference on ComputationalLinguistics: Technical Papers, pages 1591–1600,Dublin, Ireland. Dublin City University and ACL.

David M. Blei, Andrew Y. Ng, and Michael I. Jordan.2003. Latent dirichlet allocation. J. Mach. Learn.Res., 3:993–1022.

Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching word vectors withsubword information. Transactions of the Associa-tion for Computational Linguistics, 5:135–146.

Jose Camacho-Collados, Mohammad Taher Pilehvar,and Roberto Navigli. 2015. NASARI: a novel ap-proach to a semantically-aware representation ofitems. In Proceedings of the 2015 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, pages 567–577, Denver, Colorado. ACL.

Ramon Ferrer-i Cancho and Michael S. Vitevitch.2018. The origins of zipf’s meaning-frequency law.Journal of the Association for Information Scienceand Technology, 69(11):1369–1379.

Devendra Singh Chaplot and Ruslan Salakhutdinov.2018. Knowledge-based word sense disambiguationusing topic models. In AAAI Conference on Artifi-cial Intelligence.

Xinxiong Chen, Zhiyuan Liu, and Maosong Sun.2014. A unified model for word sense represen-tation and disambiguation. In Proceedings of the2014 Conference on Empirical Methods in NaturalLanguage Processing (EMNLP), pages 1025–1035,Doha, Qatar. ACL.

Kevin Clark, Urvashi Khandelwal, Omer Levy, andChristopher D. Manning. 2019. What does BERTlook at? an analysis of BERT’s attention. In Pro-ceedings of the 2019 ACL Workshop BlackboxNLP:Analyzing and Interpreting Neural Networks forNLP, pages 276–286, Florence, Italy. ACL.

Ronan Collobert and Jason Weston. 2008. A unifiedarchitecture for natural language processing: Deepneural networks with multitask learning. In Proc.ICML, pages 160–167.

Diego De Cao, Roberto Basili, Matteo Luciani,Francesco Mesiano, and Riccardo Rossi. 2010. Ro-bust and efficient page rank for word sense dis-ambiguation. In Proceedings of TextGraphs-5 -2010 Workshop on Graph-based Methods for Nat-ural Language Processing, pages 24–32, Uppsala,Sweden. ACL.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186, Minneapolis, Minnesota. ACL.

Bill Dolan, Chris Quirk, and Chris Brockett. 2004.Unsupervised construction of large paraphrase cor-pora: Exploiting massively parallel news sources.In COLING 2004: Proceedings of the 20th Inter-national Conference on Computational Linguistics,pages 350–356, Geneva, Switzerland. COLING.

Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar,Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015.Retrofitting word vectors to semantic lexicons. InProceedings of the 2015 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 1606–1615, Denver, Colorado. ACL.

Olivier Ferret and Brigitte Grau. 2002. A bootstrap-ping approach for robust topic analysis. Nat. Lang.Eng., 8(3):209–233.

Juri Ganitkevitch, Benjamin Van Durme, and ChrisCallison-Burch. 2013. PPDB: The paraphrasedatabase. In Proceedings of the 2013 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, pages 758–764, Atlanta, Georgia. ACL.

Taher H. Haveliwala. 2002. Topic-sensitive pagerank.In Proceedings of the 11th International Conferenceon World Wide Web, WWW ’02, pages 517–526,New York, NY, USA. ACM.

Ignacio Iacobacci, Mohammad Taher Pilehvar, andRoberto Navigli. 2015. SensEmbed: Learning senseembeddings for word and relational similarity. InProceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the

98

7th International Joint Conference on Natural Lan-guage Processing (Volume 1: Long Papers), pages95–105, Beijing, China. ACL.

Ignacio Iacobacci, Mohammad Taher Pilehvar, andRoberto Navigli. 2016. Embeddings for word sensedisambiguation: An evaluation study. In Proceed-ings of the 54th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 897–907, Berlin, Germany. ACL.

Armand Joulin, Edouard Grave, Piotr Bojanowski, andTomas Mikolov. 2017. Bag of tricks for efficienttext classification. In Proceedings of the 15th Con-ference of the European Chapter of the Associationfor Computational Linguistics: Volume 2, Short Pa-pers, pages 427–431, Valencia, Spain. ACL.

Doo Soon Kim, Ken Barker, and Bruce Porter. 2010.Improving the quality of text understanding by de-laying ambiguity resolution. In Proceedings of the23rd International Conference on ComputationalLinguistics (Coling 2010), pages 581–589, Beijing,China. Coling 2010 Organizing Committee.

Minh Le, Marten Postma, Jacopo Urbani, and PiekVossen. 2018. A deep dive into word sense dis-ambiguation with LSTM. In Proceedings of the27th International Conference on ComputationalLinguistics, pages 354–365, Santa Fe, New Mexico,USA. ACL.

Jiwei Li and Dan Jurafsky. 2015. Do multi-sense em-beddings improve natural language understanding?In Proceedings of the 2015 Conference on Empiri-cal Methods in Natural Language Processing, pages1722–1732, Lisbon, Portugal. ACL.

Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, andZhifang Sui. 2018. Incorporating glosses into neuralword sense disambiguation. In Proceedings of the56th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), Mel-bourne, Australia. ACL.

Massimiliano Mancini, Jose Camacho-Collados, Igna-cio Iacobacci, and Roberto Navigli. 2017. Embed-ding words and senses together via joint knowledge-enhanced training. In Proceedings of the 21stConference on Computational Natural LanguageLearning (CoNLL 2017), pages 100–111, Vancou-ver, Canada. ACL.

Marco Marelli, Stefano Menini, Marco Baroni, LuisaBentivogli, Raffaella Bernardi, and Roberto Zam-parelli. 2014. A SICK cure for the evaluation ofcompositional distributional semantic models. InProceedings of the Ninth International Conferenceon Language Resources and Evaluation (LREC-2014), pages 216–223, Reykjavik, Iceland. Euro-pean Languages Resources Association (ELRA).

Oren Melamud, Jacob Goldberger, and Ido Dagan.2016. context2vec: Learning generic context em-bedding with bidirectional LSTM. In Proceedings

of The 20th SIGNLL Conference on ComputationalNatural Language Learning, pages 51–61, Berlin,Germany. ACL.

Rada Mihalcea, Paul Tarau, and Elizabeth Figa. 2004.PageRank on semantic networks, with applicationto word sense disambiguation. In COLING 2004:Proceedings of the 20th International Conferenceon Computational Linguistics, pages 1126–1132,Geneva, Switzerland. COLING.

Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013.Exploiting similarities among languages for ma-chine translation. CoRR, abs/1309.4168.

George A Miller, Claudia Leacock, Randee Tengi, andRoss T Bunker. 1993. A semantic concordance. InProceedings of the workshop on Human LanguageTechnology, pages 303–308. ACL.

Andriy Mnih and Geoffrey Hinton. 2007. Three newgraphical models for statistical language modelling.In Proceedings of the 24th International Conferenceon Machine Learning, ICML ’07, pages 641–648,New York, NY, USA. ACM.

Andrea Moro, Alessandro Raganato, and Roberto Nav-igli. 2014. Entity linking meets word sense disam-biguation: a unified approach. Transactions of theAssociation for Computational Linguistics, 2:231–244.

John Nash. 1951. Non-cooperative games. Annals ofMathematics, 54(2):286–295.

Roberto Navigli. 2009. Word sense disambiguation: Asurvey. ACM Comput. Surv., 41(2):10:1–10:69.

Roberto Navigli. 2018. Natural Language Under-standing: Instructions for (present and future) use.In Proceedings of the Twenty-Seventh InternationalJoint Conference on Artificial Intelligence (IJCAI2018), pages 5697–5702.

Roberto Navigli and Mirella Lapata. 2010. An ex-perimental study of graph connectivity for unsuper-vised word sense disambiguation. IEEE Transac-tions on Pattern Analysis and Machine Intelligence,32(4):678–692.

Roberto Navigli and Simone Paolo Ponzetto. 2012.BabelNet: The automatic construction, evaluationand application of a wide-coverage multilingual se-mantic network. Artif. Intell., 193:217–250.

John von Neuman and Oskar Morgenstern. 1944. The-ory of games and economic behavior. PrincetonUniversity Press.

Lawrence Page, Sergey Brin, Rajeev Motwani, andTerry Winograd. 1999. The pagerank citation rank-ing: Bringing order to the web. Technical Re-port 1999-66, Stanford InfoLab. Previous number= SIDL-WP-1999-0120.

99

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. Glove: Global vectors for wordrepresentation. In Proceedings of the 2014 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 1532–1543, Doha,Qatar. ACL.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. In Proceedings of the 2018 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long Papers), pages2227–2237, New Orleans, Louisiana. ACL.

Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context datasetfor evaluating context-sensitive meaning represen-tations. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages1267–1273, Minneapolis, Minnesota. ACL.

Mohammad Taher Pilehvar and Nigel Collier. 2016.De-conflated semantic representations. In Proceed-ings of the 2016 Conference on Empirical Methodsin Natural Language Processing, pages 1680–1690,Austin, Texas. ACL.

Mohammad Taher Pilehvar and Roberto Navigli. 2014.A large-scale pseudoword-based evaluation frame-work for state-of-the-art word sense disambiguation.Computational Linguistics, 40(4):837–881.

Mohammed Taher Pilehvar, Jose Camacho-Collados,Roberto Navigli, and Nigel Collier. 2017. Towardsa Seamless Integration of Word Senses into Down-stream NLP Applications. In Proc. of the 55th An-nual Meeting of the Association for ComputationalLinguistics (ACL 2017), pages 1857–1869, Vancou-ver, Canada. ACL.

Marten Postma, Ruben Izquierdo Bevia, and PiekVossen. 2016. More is not always better: balanc-ing sense distributions for all-words word sense dis-ambiguation. In Proceedings of COLING 2016,the 26th International Conference on ComputationalLinguistics: Technical Papers, pages 3496–3506,Osaka, Japan. The COLING 2016 Organizing Com-mittee.

Alessandro Raganato, Jose Camacho-Collados, andRoberto Navigli. 2017. Word sense disambiguation:A unified evaluation framework and empirical com-parison. In Proceedings of the 15th Conference ofthe European Chapter of the Association for Compu-tational Linguistics: Volume 1, Long Papers, pages99–110, Valencia, Spain. ACL.

Sascha Rothe and Hinrich Schutze. 2015. AutoEx-tend: Extending word embeddings to embeddingsfor synsets and lexemes. In Proceedings of the53rd Annual Meeting of the Association for Compu-tational Linguistics and the 7th International Joint

Conference on Natural Language Processing (Vol-ume 1: Long Papers), pages 1793–1803, Beijing,China. ACL.

Peter D. Taylor and Leo B. Jonker. 1978. Evolutionarystable strategies and game dynamics. MathematicalBiosciences, 40(1):145 – 156.

Rocco Tripodi and Marcello Pelillo. 2017. A game-theoretic approach to word sense disambiguation.Computational Linguistics, 43(1):31–70.

Tolga Uslu, Alexander Mehler, Daniel Baumartz, andWahed Hemati. 2018. FastSense: An efficient wordsense disambiguation classifier. In Proceedings ofthe Eleventh International Conference on LanguageResources and Evaluation (LREC-2018), Miyazaki,Japan. ELRA.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In I. Guyon, U. V. Luxburg, S. Bengio,H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar-nett, editors, Advances in Neural Information Pro-cessing Systems 30, pages 5998–6008. Curran As-sociates, Inc.

Jorgen W. Weibull. 1997. Evolutionary game theory.MIT press.

Dayu Yuan, Julian Richardson, Ryan Doherty, ColinEvans, and Eric Altendorf. 2016. Semi-supervisedword sense disambiguation with neural models. InProceedings of COLING 2016, the 26th Interna-tional Conference on Computational Linguistics:Technical Papers, pages 1374–1385, Osaka, Japan.The COLING 2016 Organizing Committee.

Zhi Zhong and Hwee Tou Ng. 2010. It makes sense:A wide-coverage word sense disambiguation systemfor free text. In Proceedings of the ACL 2010 Sys-tem Demonstrations, pages 78–83, Uppsala, Swe-den. ACL.