Facts as Experts: Adaptable and Interpretable Neural Memoryover Symbolic Knowledge

Pat Verga*, Haitian Sun*, Livio Baldini Soares, William W. CohenGoogle Research

{patverga, haitiansun, liviobs, wcohen}@google.com

Abstract

Massive language models are the core of mod-ern NLP modeling and have been shown toencode impressive amounts of commonsenseand factual information. However, that knowl-edge exists only within the latent parameters ofthe model, inaccessible to inspection and inter-pretation, and even worse, factual informationmemorized from the training corpora is likelyto become stale as the world changes. Knowl-edge stored as parameters will also inevitablyexhibit all of the biases inherent in the sourcematerials. To address these problems, we de-velop a neural language model that includesan explicit interface between symbolically in-terpretable factual information and subsym-bolic neural knowledge. We show that thismodel dramatically improves performance ontwo knowledge-intensive question-answeringtasks. More interestingly, the model can beupdated without re-training by manipulatingits symbolic representations. In particular thismodel allows us to add new facts and overwriteexisting ones in ways that are not possible forearlier models.

1 Introduction

Over the past several years, large pretrained lan-guage models (LMs) (Peters et al., 2018; De-vlin et al., 2019; Raffel et al., 2019) have shiftedthe NLP modeling paradigm from approachesbased on pipelines of task-specific architecturesto those based on pretraining followed by fine-tuning, where a large language model discoversuseful linguistic properties of syntax and seman-tics through massive self-supervised training, andthen small amounts of task specific training dataare used to fine-tune that model (perhaps withsmall architectural modifications). More recently,similar approaches have been explored for knowl-

*Equal contribution

edge representation and reasoning (KRR) with re-searchers asking questions like ‘Language Mod-els as Knowledge Bases?’ (Petroni et al., 2019).Results suggest that (Roberts et al., 2020; Brownet al., 2020) the answer is a resounding ‘sort of’(Poerner et al., 2019): while language models canbe coerced to answer factual queries, they still lackmany of the properties that knowledge bases typ-ically have. In particular, when evaluating LM-as-KRR models there are three explanations forwhy a model outputs a correct answer; 1) Themodel has successfully performed some reasoningor generalization required to make a novel infer-ence, 2) the dataset contains some statistical biasesthat the model is exploiting, or 3) the model hasmemorized the exact answer, potentially from pre-training data that overlaps with the test cases.1. Inshort, knowledge encoded only in a LM’s param-eters is generally opaque.

To address these problems, we propose an inter-face between explicit, symbolically bound memo-ries and sub-symbolic distributed neural models.In addition to making more of a language model’sbehavior interpretable, our approach has severalother important benefits. First, there is a massiveamount of useful information that has been cre-ated and curated in structured databases. Some-times this information either does not occur in textat all (such as a new product that hasn’t comeout yet) or is very difficult to interpret from thetext (such as in scientific, technical, or legal doc-uments). In our framework, new knowledge canbe inserted by updating the symbolically boundmemory. Second, pre-trained language modelsappear to require training on very large corpora

1This is a real possibility: for example, the T5 trainingdata contains a large portion of the sources from which Triv-iaQA was derived, and attempts at avoiding leakage in GPT3by looking at large ngram exact match do not account fortrivial surface form changes.

arX

iv:2

007.

0084

9v1

[cs

.CL

] 2

Jul

202

0

to obtain good factual coverage—and the massiveweb corpora required by these data-hungry modelscontain huge amounts of sexist, racist, and incor-rect assertions (Bolukbasi et al., 2016; Sun et al.,2019b). Our approach makes it possible to obtainbetter factual coverage of assertions chosen fromselected trusted sources, by inserting this trustedfactual content into the symbolic memory.

We propose to incorporate an external factmemory into a neural language model. This modelforms its predictions by integrating contextual em-beddings with retrieved knowledge from an exter-nal memory, where those memories are bound tosymbolic facts which can be added and modified.We evaluate our model’s performance empiricallyon two benchmark question answering datasets;FreebaseQA and WebQuestionsSP (section 4.2).In section 5.2, we show how we can inject newmemories at inference time enabling our model tocorrectly answer questions about pairs of entitiesthat were never observed in the pretraining textcorpus. Finally, in section 5.3 we examine to whatextent our model is capable of iteratively updatingby overwriting prior memories with new facts. Wemodify facts such that they actually contradict theoriginal pretraining data, and show that our modelis capable of answering correspondingly modifiedquestion answer pairs. In these experiments weshow that end users can inject new knowledgeand change existing facts by manipulating only thesymbolically bound memories without retrainingany parameters of the model.

2 Related Work

Knowledge bases (KBs) have been a core compo-nent of AI since the beginning of the field (Newelland Simon, 1956; Newell et al., 1959). Widelyavailable public KBs have been invaluable in re-search and industry (Bollacker et al., 2008; Aueret al., 2007) and many companies have createdmassive KBs as the backbones of their most im-portant products (Google, 2012; Dong, 2017).

While traditional KBs were purely symbolic,recent advances in large language models trainedthrough self supervision (Peters et al., 2018; De-vlin et al., 2019; Raffel et al., 2019; Brown et al.,2020) have been shown to encode an impressiveamount of factual information. This has led to re-search on the extent to which a neural languagemodel can serve as a KB (Roberts et al., 2020;Petroni et al., 2019), and other research on how

to best evaluate the factual knowledge in languagemodels (Poerner et al., 2019).

While large LMs appear to absorb KB-like in-formation as a preproduct of pretraining, therehas also been many prior approaches proposedthat explicitly embed symbolic knowledge rep-resentations into neural embedding space. Var-ious neural-symbolic methods have attempted tounify these two extremes (Pinkas, 1991; de Pen-ning et al., 2011; Besold et al., 2017) includingmany cognitive architectures which used hybridsymbolic and subsymbolic systems (Laird et al.,2017), and more recently, compositional querylanguages for embedding KBs that are similar tosymbolic KB query languages (Cohen et al., 2017;Hamilton et al., 2018; Ren et al., 2020; Cohenet al., 2020). One system especially related to ourproposal is EmQL (Sun et al., 2020), which in-cludes a construct quite similar to the “fact mem-ory” used in our Facts-As-Experts model. Unlikethis work, however, EmQL did not embed its factmemory into a language model, which can be fine-tuned for many NLP tasks: instead EmQL must beused with task-specific query templates and inte-grated into some task-specific architecture.

More recently, the past decade has seen hugeamount of work on knowledge base embeddings(Bordes et al., 2013; Lin et al., 2015; Trouillonet al., 2017; Dettmers et al., 2018) which en-able generalization through similarities betweenlearned embeddings. This idea has also been ex-tended with works looking at ways of incorporat-ing raw text and symbolic KGs into a shared em-bedding space (Riedel et al., 2013; Verga et al.,2016), to be jointly reasoned over (Sun et al.,2018, 2019a), or to treat text as a replacement fora knowledge base (Dhingra et al., 2019).

Large external memories have been incorpo-rated into different types of memory networks op-erating over latent parameters (Weston et al., 2014;Miller et al., 2016), entity memories (Henaff et al.,2016; Fevry et al., 2020), relations (Logan et al.,2019), and embedded text passages (Guu et al.,2020; Lewis et al., 2020). Our work directly ex-tends one of these models, the Entities-as-Experts(EaE) model (Fevry et al., 2020), one of severalmodels that inject knowledge of entities by con-structing a memory containing embedded entityrepresentations. Unlike prior models, EaE learnsentity representations end-to-end, rather than us-ing representations from a separately-trained KB

embedding system (Logan et al., 2019). Our workextends EaE by introducing a symbolic memory oftriples which is constructed from these learned en-tity representations, and as in EaE, the entity rep-resentations are learned end-to-end.

3 Model

3.1 Facts-as-Experts (FaE)

Our Facts-as-Experts (FaE) model (see Figure 1)builds an interface between a neural languagemodel and a symbolic knowledge graph. Thismodel builds on the recently-proposed Entities asExperts (EaE) language model Fevry et al. (2020),which extends the same transformer (Vaswaniet al., 2017) architecture of BERT (Devlin et al.,2019) with an additional external memory for en-tities. After training EaE, the embedding asso-ciated with an entity will (ideally) capture infor-mation about the textual context in which thatentity appears, and by inference, the entity’s se-mantic properties. In FaE, we include an addi-tional memory called a fact memory, which en-codes triples from a symbolic KB. Each triple isconstructed compositionally from the EaE-learnedembeddings of the entities that comprise it. Thisfact memory is represented with a key-value mem-ory, and can be used to retrieve entities given theirproperties in the KB. This combination results in aneural language model which learns to access in-formation in a the symbolic knowledge graph.

3.2 Definitions

We represent a Knowledge Base K as a set oftriples (s, r, o) where s, o ∈ E are the subject andobject entities and r ∈ R is the relation, where EandR are pre-defined vocabularies of entities andrelations in the knowledge baseK. A text corpus Cis a collection of paragraphs2 {p1, . . . , p|C|}. LetM be the set of entity mentions in the corpus C.A mention mi is defined as (em, s

pm, t

pm), i.e. en-

tity em is mentioned in paragraph p starting fromthe token at position spm and ends on tpm. Since wedon’t consider multi-paragraph operations in thispaper, we will usually drop the superscript p anduse sm and tm for brevity.

2We use the term paragraph to describe a text span thatis roughly paragraph length (128 token pieces in our exper-iments). In reality the text spans do not follow paragraphboundaries.

3.3 Input

The input to our model is a piece of text whichcan be either a question in the case of fine tun-ing (see section 3.8) or an arbitrary span as inpre-training (see section 3.7). Our pretraininginput is constructed as cloze-type Question An-swering (QA) task. Formally, given a paragraphp = {w1, . . . , w|p|}with mentions {m1, . . . ,mn},we pick a mention mi and replace all tokens fromsmi to tmi with a special [MASK] token. We con-sider the entity in E named by the masked entity tobe the answer to the cloze question q. Mentions inthe paragraph other than this masked entity are re-ferred as below as context mentions. For example,in the cloze question, {‘Charles’, ‘Darwin’, ‘was’,‘born’, ‘in’, [MASK], [MASK], ‘in’, ‘1809’, ‘.’,‘His’, ‘proposition’, . . . }, “Charles Darwin” is acontext entity in mentionm1 = (‘Charles Darwin’,1, 2), and “United Kingdom” is the answer entityin the masked mentionmans = (‘United Kingdom’,6, 7).

Our model learns to jointly link entities fromcontext mentions mi using entity-aware contex-tual embeddings (§3.4) and predict answer entitiesusing knowledge-enhanced embeddings (§3.6).This process will be introduced in more detail inthe following sections.

3.4 Entity-aware Contextual Embeddings

We follow the Entities-as-Experts (EaE) (Fevryet al., 2020) model to train an external entity mem-ory. The EaE model is illustrated in the left part ofFigure 1. This model interleaves standard Trans-former layers with layers that access an entitymemory (see Vaswani et al. (2017) for details onthe transformer architecture). EaE inputs a para-graph (or question) containing unlinked entitieswith known boundaries3 (i.e., the index of the startand end of each mention is provided, but the iden-tity of the entity mentioned is not.) Given a ques-tion q = {w1, . . . , w|q|} with a list of contextmentions mi = (emi , smi , tmi) and the answereans from the masked mention mans = (eans, sans,tans), the contextual embedding h(l)

i is the output atthe i’th token of the l’th intermediate transformerlayer.

h(l)i , . . . ,h

(l)|q| = Transformer({w1, . . . , w|q|})

3Fevry et al. (2020) also showed the model is capable oflearning to predict these boundaries. For simplicity, in thiswork we assume they are given.

Figure 1: Facts-as-Experts model architecture. The model takes a piece of text (a question during fine-tuning orarbitrary text during pre-training) and first contextually encodes it with an entity enriched transformer. The part ofthe model within the dashed line is exactly the Entities-as-Experts model from Fevry et al. (2020). The model usesthe contextually encoded MASK token as a query to the fact memory. In this case, the contextual query choosesthe fact key (Charles Darwin, born in) which returns the a set of values {United Kingdom} (The value set can bemultiple entity objects such as the case from calling the key [United Kingdom, has city]) . The returned objectrepresentation is incorporated back into the context in order to make the final prediction. Note that the entities infacts (both in keys and values) are shared with the EaE entity memory.

These contextual embeddings are used to computequery vectors that interface with an external en-tity memory E ∈ R|E|×de , which is a large ma-trix containing a vector for each entity in E . Toconstruct a query vector, we concatenate the con-text embeddings for the mentionmi’s start and endtokens, h(l)

smiand h(l)

tmi, and project them into the

entity’s embedding space. We compute the atten-tion weights over the embeddings of the full entityvocabulary, and use this to produce the attention-weighted sum of entity embeddings ul

mi. This re-

sult is then projected back to the dimension of thecontextual token embeddings, and added to whatwould have been the input to the next layer of theTransformer.

h(l)mi

= WTe [h

(l)smi

;h(l)tmi

] (1)

u(l)mi

= softmax(h(l)mi,E)× E (2)

h(l+1)j = h(l)

j + WT2 u(l)

mi, smi < j < tmi (3)

Let h(T )j be the contextual embedding of the j’th

token after the final transformer layer T . Similarto the query construction in the intermediate trans-former layer in Eq. 1, EaE constructs the query

vector h(T )mi for mention mi and use it to predict

the context entities ˆemi . This query vector is calledan entity-aware contextual query in the rest of thispaper and denoted as cmi for brevity. This queryvector is trained with a cross-entopy loss againstIemi

, the one-hot label of entity emi .

emi = argmaxei∈E(cTmi

ei)lossctx = cross entropy(softmax(cmi ,E), Iemi

)

As shown in Fevry et al. (2020), supervisionon the intermediate entity access is beneficial forlearning entity-aware contextual embeddings. Wecompute an entity memory access loss using theintermediate query vector in Eq. 1.

lossent = cross entropy(softmax(h(l)mi,E), Iemi

)

In pretraining the FaE model, we used a slightlydifferent pre-training process than was used inEaE. In EaE, mentions in the same paragraphs areindependently masked with some probability andjointly trained in one example.4 In FaE, in addition

4EaE is also jointly trained on mention detection. Pleaserefer to Fevry et al. (2020) for more information.

to the randomly masked context mentions, FaEpicks exactly one of the mentions and masks it.Predicting this masked entity requires additionalaccess to the fact memory which will be discussedin the next section.

3.5 Fact Memory

FaE inherits the external entity memory E fromthe EaE model and adds another fact memorywhich contains triples from the knowledge baseK (see the right side of Figure 1). The factmemory shares its on entity representations withthe entity memory embeddings in E, but eachelement of the fact memory corresponds to asymbolic substructure, namely a key-value pair((s, r), {o1, . . . , on}). The key (s, r) is a (subjectentity, relation) pair, and the corresponding value{o1, . . . , on} is the list of object entities associatedwith s and r, i.e. (s, r, oi) ∈ K for i = {1, . . . , n}.Hence, conceptually, KB triples with the samesubject entity and relation are grouped into a sin-gle element. We call the subject and relation pairaj = (s, r) ∈ A a head pair and the list of ob-jects bj = {o1, . . . , on} ∈ B a tail set5, and willencode K as a new structure K′ = (A,B), with|A| = |B|. Notice that K′ contains same informa-tion as K, but can be encoded as as a key-valuememory: elements are scored using the keys (s, r)from head pairs A, and values from the tail sets Bare returned.

In more detail, we encode a head pair aj =(s, r) ∈ A in the embedding space as follows. LetE ∈ R|E|×de be the entity embeddings trained inSec 3.4, and R ∈ R|R|×dr be embeddings of re-lations R in the knowledge base K. We encode ahead pair a as:

aj = WTa [s; r] ∈ Rda

where s ∈ E and r ∈ R are the embeddings ofsubject s and relation r, and Wa is a learned lineartransformation matrix. A ∈ R|A|×da is the em-bedding matrix of all head pairs in A.

The query vector to access the fact memory isderived from contextual embeddings and projectedto the same embedding space as the head pairs A.For a masked mentionmans = (eans, sans, tans), de-

5The size of the tail set bj can be large for a popular headpair (s, r). In such case, we randomly select a few tails anddrop the rest of them. The maximum size of the tail set is 32in the experiments in this paper.

fine a query vector

vmans = WTf [h(T )

sans;h(T )

tans] (4)

where h(T )sans and h(T )

tansare the contextual embed-

dings at the start and end tokens for the mentionmans, and Wf is the linear transformation matrixinto the embedding space of head pairs A.

Head pairs in A are scored by the query vectorvmans and the top k head pairs with the largest innerproduct scores are retrieved. This retrieval processon the fact memory is distantly supervised. Wedefine a head pair to be a distantly supervised pos-itive example ads = (s, r) for a passage if its sub-ject entity s is named by a context mentionmi andthe masked entity eans is an element of the corre-sponding tail set, i.e. eans ∈ bds. In cases that nodistantly supervised positive example exists for apassage, we introduce add a special example thatretrieves a “null” fact from the knowledge base,where the “null” fact has a special snull head en-tity and special rnull relatio: i.e. ads = (snull, rnull)and its tail set is empty. This distant supervision isencoded by a loss function

TOPk(vmans ,A) = argmaxk,j∈{1,...,|A|}aTj vmans

lossfact = cross entropy(softmax(vmans ,A), Iads)

Here the tail sets associated with the top k scoredhead pairs, i.e. {bj |j ∈ TOPk(v,A)}, will be re-turned from the fact memory. We will discuss in-tegrating the retrieved tail sets bj’s to the contextin the following section.

3.6 Integrating Knowledge and ContextTail sets retrieved from the fact memory are nextaggregated and integrated with the contextual em-beddings. Recall that a tail set bj returned fromthe fact memory is the set of entities {o1, . . . , on}s.t. (s, r, oi) ∈ K for i ∈ {1, . . . , n}with the asso-ciated aj = (s, r). Let oi ∈ E be the embeddingof entity oi. We encode the returned tail set bj asa weighted centroid of the embeddings of entitiesin the tail set bj .

bj =∑oi∈bj

αioi ∈ Rde

where αi is a context-dependent weight of the ob-ject entity oi. To compute the weights αi, we use aprocess similar to Eq. 4, and we compute a secondquery vector zmans to score the entities inside theetail set bj . The weights αi are the softmax of the

inner products between the query vector zmans andthe embeddings of entities in the tail set bj .

zmans = WTb [h

(T )sans

;h(T )tans

] (5)

αi =exp (oTi zmans)∑

ol∈bj exp (oTl zmans)

(6)

where Wb is yet another transformation matrix dif-ferent from We in Eq. 1 and Wf in Eq. 4.

The top k tail sets bj are further aggregated us-ing weights βj , which are the softmax of the re-trieval (inner product) scores of the top k headpairs aj . This leads to a single vector fmans thatwe call the knowledge embedding for the maskedmention mans.

fmans =∑

j∈TOPk(vmans ,A)

βjbj (7)

βj =exp (aTj vmans)∑

t∈TOPk(vmans ,A) exp (aTt vmans)(8)

Intuitively fmans is the result of retrieving a setof entities from the fact memory. We expect FaEshould learn to jointly use the contextual querycmans and knowledge query fmans to predict themasked entity, i.e. use external knowledge re-trieved from the fact memory if there exists an or-acle head pair aorc = (s, r) s.t. eans ∈ borc, orfall back to contextual query to make predictionsotherwise. We compute the integrated query qmans

with a mixing weight λ. λ is the probability ofpredicting the “null” head anull in the fact memoryaccess step, i.e. whether an oracle head pair aorcexists in the knowledge base.

λ = P (y = anull)

qmans= λ · cmans + (1− λ) · fmans

The query vector qmansis called a knowledge-

enhanced contextual query. This query vector fi-nally is used to predict the masked entity. Again,we optimized it with a cross-entropy loss.

eans = argmaxei∈E(qTmans

ei)lossans = cross entropy(softmax(qmans

,E), Ieans)

3.7 PretrainingFaE is jointly trained to predict context entitiesand the masked entity. Context entities are pre-dicted using the contextual embeddings describedin § 3.4; intermediate supervision with oracle en-tity linking labels is provided in the entity mem-ory access step for context entities; the masked

entity is predicted using the knowledge-enhancedcontextual embeddings (§ 3.6); and distant super-vised fact labels are also provided at training time.The final training loss is the unweighted sum ofthe four losses:

losspretrain = lossent + lossctx + lossfact + lossans

3.8 Finetuning on Question AnsweringIn the Open-domain Question Answering task,questions are posed in natural language, e.g.“Where was Charles Darwin born?”, and an-swered by a sequence of tokens, e.g. “UnitedKingdom”. In this paper, we focus on a subset ofopen-domain questions that are answerable usingentities from a knowledge base. In the exampleabove, the answer “United Kingdom” is an entityin Wikidata whose identity is Q145.

We convert an open-domain question to an inputof FaE by appending the special [MASK] tokento the end of the question, e.g. {‘Where’, ‘was’,‘Charles’, ‘Darwin’, ‘born’, ‘?’, [MASK]}. Thetask is to predict the entity named by mask. Here,“Charles Darwin” is a context entity, which is alsoreferred to as question entity in the finetuning QAtask.

At finetuning time, entity embeddings E and re-lation embeddings R are fixed, and we finetune alltransformer layers and the four transformation ma-trices: Wa, Wb, We, Wf . Parameters are tuned tooptimize unweighted sum of the the fact memoryretrieval loss lossfact and the final answer predic-tion loss lossans. If multiple answers are available,the training label Ieans becomes a k-hot vector uni-formly normalized across the answers.

lossfinetune = lossfact + lossans

4 Experiments

4.1 DatasetsWe evaluate our model on two Open-domainQuestion Answering datasets: FreebaseQA (Jianget al., 2019) and WebQuestionsSP (Yih et al.,2015) (See table 1 for data statistics). Bothdatasets are created from Freebase. To align withour pretraining task, we convert the entity ids fromFreebase to Wikidata.FreebaseQA. FreebaseQA is derived from Trivi-aQA and several other trivia resources (See Jianget al. (2019) for full details). Every answer canbe resolved to at least one entity and each ques-tion contains at least one question entity ei. Ad-ditionally, there exists at least one relational path

Full WikidataDataset Answerable

Train 20358 12535FreebaseQA Dev 2308 2464

Test 3996 2440

Train 2798 1388WebQuestionsSP Dev 300 153

Test 1639 841

Table 1: Dataset stats. Number of examples in train,dev, and test splits for our three different experimentalsetups. Full are the original unaltered datasets. Wiki-data Answerable keeps only examples where at leastone question entity and answer entity are mappable toWikidata and there is at least one fact between them inour set of facts.

in Freebase between the question entity ei and theanswer eans. The path must be either a one-hoppath, or a two-hop path passing through a mediator(CVT) node, and is verified by human raters. 72%of the question entities and 80% of the answer en-tities are mappable to Wikidata, and 91.7% of thequestions are answerable by at least one answerentity that is mappable to Wikidata.WebQuestionsSP. WebQuestionsSP is con-structed from Freebase and contains 4737 naturallanguage questions (3098 training and 1639test). Questions in the dataset are linked tocorresponding Freebase entities and relations. Wemapped question entities and answer entities totheir Wikidata ids. 87.9% of the questions areanswerable by at least one answer entity that ismappable to Wikidata.Subset of questions answerable by KB triples.Both of these datasets were constructed so that thatall questions are answerable using the FreeBaseKB, which was last updated in 2016. Because ourpretraining corpus is derived from larger and morerecent versions of Wikipedia, we elected to use aKB constructed from Wikidata instead. Use of themore recent Wikidata KB means that some ques-tions are no longer answerable using the KB, sowe also created a second reduced version of thedatasets called Wikidata Answerable. These sub-sets only contains questions that are answerable bytriples from our Wikidata-based KB. The modelshould learn to rely on the KB to answer the ques-tions.

4.2 Pretraining

FaE is pretrained on Wikipedia and Wikidata. Textin Wikipedia is chunked into 128 token pieces. To

compute the entity-linking loss lossent, we use astraining data entities linked to the 1 million mostfrequently linked-to Wikidata entities. Text pieceswithout such entities are dropped. This resultsin 30.58 million text pieces from Wikipedia. Asdescribed in § 3.2, we generate n training exam-ples from a piece of text containing n entity men-tions, where each mention serves as the maskedtarget for its corresponding example, and other en-tity mentions in the example are treated as contextentities6. This conversion results in 85.58 mil-lion pre-training examples. The knowledge baseK is a subset of Wikidata that contains all factswith subject and object entity pairs that co-occurat least 10 times on Wikipedia pages.7 This resultsin a KB containing 1.54 million KB triples fromWikidata (or 3.08 million if reverse triples are in-cluded). Below, this is called the full setting ofpretraining—we will also train on subsets of thisexample set, as described below. We pretrain themodel for 500,000 steps with the batch size 2048,and we set k = 1 in the TOPk operation for factmemory access.

4.3 Results

We compare FaE with three baseline models:FOFE (Jiang et al., 2019), EmQL (Sun et al.,2020), and Entity-as-Expert (EaE) (Fevry et al.,2020). FOFE is a feed-forward language modeldesigned to encode long sequences and wasthe previous state-of-the-art model on the Free-baseQA dataset. EmQL was introduced as a queryembedding on knowledge bases and is the pre-vious state-of-the-art model on WebQuestionsSP.EaE has been discussed above, and our EaE mod-els are trained using the same hyperparameters andoptimization settings as FaE in order to make themas comparable as possible.

Table 2 compares the FaE model to the baselinemodel. With the full pre-training and fine-tuningdatasets, we outperform the baseline models onthe FreebaseQA dataset by nearly 10 points. Per-formance on WebQuestionsSP in the Full Datasetsetting is relatively lower, however this is largelyexplained due to the incompleteness of the KB dueto mapping between Freebase and Wikidata—only51.3% of the questions in WebQuestionsSP are

6We additionally mask context entities randomly withprobability .15

7This leads to more KB triples than entity pairs, since apair of subject and object entities can be associated with morethan one relation.

FreebaseQA WebQuestionsSP

Data Full WikiData Full WikidataDataset Answerable Dataset Answerable

FOFE 37.0 - 67.6 -EmQL - - 75.5 74.6EaE 53.4 59.1 46.3 61.4FaE (ours) 63.3 73.9 56.1 78.5EaE no finetune 18.3 24.8 12.8 21.4FaE (ours) no finetune 19.7 26.9 15.9 24.6

Table 2: Conventional Setting Evaluation. Accuracy on FreebaseQA and WebQuestionsSP datasets. We pretrainour models on the full unfiltered Wikipedia text corpus. In the Full Data column, we report scores on the originalunfiltered data splits for each dataset. In the WikiData Answerable column, we filter each split to only containexamples where at least one question and answer entity are mappable to WikiData and our WikiData knowledgegraph contains some fact connecting them. Nearly all FreebaseQA and WebQuestionsSP entity pairs that aremappable to WikiData co-occur in the Wikipedia pretraining text. Models marked “no finetune” were not finetuned.

answerable using facts from our KB. In contrast,both FOFE and EmQL have complete coverage asthey both use the full applicable subset of Free-base.

However, if we instead consider only questionsanswerable using our dataset (the column labeled“Wikidata Answerable”) FaE substantially outper-forms EmQL. In this case, both models have com-plete knowledge base coverage. Additionally, inthe Wikidata Answerable setting in FreebaseQA,the gap between EaE and FaE grows even largerto nearly 14 points.

Interestingly, EaE and FaE even answer manyquestions correctly without any fine-tuning at all(denoted “no finetune” in the tables. Both modelsanswer around a quarter of the answerable ques-tions for both datasets in this zero-shot setting withFaE having a slight advantage.

5 Modifiable Knowledge Base

5.1 Filtering to Avoid Pretrain, Finetune, andTest Overlap

We are interested primarily in the ability of mod-els to use external knowledge to answer questions,rather than learning to recognize paraphrases ofsemantically identical questions. Unfortunately,analysis of the two datasets showed that many ofthe test answers also appear as answers to sometraining-set question: this is the case for 75.0%of answers in the test data for FreebaseQA, and57.5% of the answers in WebQuestionsSP. Thisraises the possibility that some of the performanceof the models can be attributed to simply mem-orizing specific question/answer pairs, perhaps inaddition to recognizing paraphrases of the ques-

tion from its pretraining data.To resolve this issue, we discard questions in

the training data that contain answers which over-lap with answers to questions in the dev andtest data. We end up with 9144/2308/3996 data(train/dev/test) in FreebaseQA and 1348/151/1639data in WebQuestionsSP. This setting is referredto as Fine-tune column in table 3 which showsthe effects of different filterings of the data. Thecolumn denoted None has no filtering and is thesame as the Full Dataset setting in table 2. In thecolumn labeled Pretrain, for every question an-swer entity pair in our finetuning dataset (comingfrom any split), we filter every example from ourWikipedia pretraining corpus where those pair ofentities co-occur. Additionally, we filter every factfrom our fact memory containing any of these en-tity pairs. In this way, the model will be unable tosimple memorize paraphrases of question answerpairs that it observed in the text. Finally, the Allcolumn combines both pretrain and fine tune filter-ing. We see that the models perform substantiallworse when these filterings are applied and theyare forced to rely on the ability to reason acrossmultiple examples, and in the case of FaE, the factmemory.

5.2 Injecting New Facts into Memory

Because our model defines facts symbolically, itcan in principle reason over new facts injected intoits memory, without retraining any parameters ofthe model. To test how well the model is ableto perform this task in practice, we look at howwell the model can perform given full knowledge,filtered knowledge, and injected knowledge. Thegap between the filtered knowledge setting and in-

FreebaseQA WebQuestionsSP

Filter Type None Pretrain Fine-tune All None Pretrain Fine-tune All

EaE 53.4 45.2 45.8 28.6 46.3 45.4 30.9 29.4FaE (ours) 63.3 57.5 56.5 48.0 56.1 55.4 40.7 39.2

Table 3: Effects of Different Data Filtering. The column denoted None has no filtering and is the same as theFull Dataset setting in table 2. Pretrain removes all entity pair overlap between the eval datasets (all splits) and thepretraining text and kb. The Fine-tune column removes all entity pair overlap between the eval train and test splits.The All column combines both pretrain and fine tune filtering.

jected knowledge setting should demonstrate howwell the model is able to incorporate newly intro-duced facts.

The results are shown in Table 4. We always usethe filtered Finetune subset of the data (see §5.1)to avoid overlap between finetuning train and testdata. In the “Full” column, we pretrain and fine-tune the FaE model with the full knowledge baseand corpus. In the “Filter” setting, facts about thefinetuning data are hidden from the model at bothpretraining and finetuning time. In this case, themodel should fall back to the language model topredict the answer. As shown in Table 4, the per-formance of FaE and EaE are close. In the “InjectFacts” setting, Facts are hidden at pretraining time,but are injected at test time. The results show thatFaE can effectively use the newly injected facts tomake prediction, i.e. an absolute improvement of9.3% compared to the “Filter” setting. EaE doesnot have a natural mechanism for integrating thisnew information8.

5.3 Updating Stale MemoriesOne of the main motivations for our model is toaddress the need for knowledge representationsthat can avoid stale data by incrementally updat-ing as the world changes. To probe this ability, wesimulate an extreme version of this scenario whereall answers to QA pairs in the FreebaseQA test setare replaced with plausible alternatives. For eachQA pair, we replace the original answer entityeoriginal with another entity from our vocabularyenew that has 1) been used as an object in at leastone of the same relation types that eoriginal was anobject in, and 2) shares at least three Wikipediacategories with eoriginal. We use the same pre-trained models from section 4.2. We fine-tune the

8There are various heuristics one could apply for finetun-ing a standard language model on this type of data by apply-ing one or a small number of gradient steps on textualizedfacts. We are currently exploring to what extent this is effec-tive and what knowledge is lost during that additional learn-ing.

filtered FreebaseQA train set and perform earlystopping on the unmodified FreebaseQA dev set.Overall, FaE is able to utilize the modified KB tomake the correct prediction for 30% of questions.

While this is an encouraging result, the decreasein performance compared to the unmodified eval-uation set (nearly twice as many incorrect predic-tions) shows that the mixing between conextualrepresentations and knowledge requires further re-search. In section 5.2 FaE was able to easily adaptto using newly injected facts because they wereconsistent with the pretraining corpus. These werefacts that did not appear in the model’s pretrain-ing data but they also did not contradict that data.In the case of updating stale memories, we areinstead giving the model new information that insome cases (such as in this experiment) explicitlycontradict the knowledge stored in its latent pa-rameters, and this inconsistency makes the mix-ing much more difficult. Addressing this issue aswell as the even more difficult problem of deletingknowledge is a main focus of ongoing and futureresearch.

6 Conclusion

In this paper, we presented a method for interfac-ing a neural language model with an interpretablesymbolically bound memory. We used that inter-face to change the output of the language modelby modifying only the non-parametric memoriesand without any additional training. We demon-strated the effectiveness of this method by per-forming comparably or better than a high perform-ing language model on factoid question answeringwhile integrating new facts unseen in pretrainingdata. We even showed that we can modify facts,such that they contradict the initial pre trainingtext, and our model is still largely able to answerthese questions correctly.

FreebaseQA WebQuestionsSP

Full Filter Inject Facts Full Filter Inject Facts

EaE 45.8 28.6 28.6 30.9 29.4 29.4FaE (ours) 56.5 38.7 48.0 40.7 32.3 39.2

Table 4: Injecting New Facts. In the Full setting the model is exposed to full knowledge in the pretraining dataand KB. In the Filter setting, the models have access to no direct knowledge about question answer entity pairsfrom either the pretraining corpus or KB. In the Inject Facts setting, the pretraining corpus and training KB are stillFiltered, but at inference time, new facts are injected into the models memory allowing it to recover most of thedrop from the Full setting. In all cases, we remove the overlap between the finetune train and eval sets.

References

Soren Auer, Christian Bizer, Georgi Kobilarov, JensLehmann, Richard Cyganiak, and Zachary Ives.2007. Dbpedia: A nucleus for a web of open data.In The semantic web, pages 722–735. Springer.

Tarek R Besold, Artur d’Avila Garcez, SebastianBader, Howard Bowman, Pedro Domingos, Pas-cal Hitzler, Kai-Uwe Kuhnberger, Luis C Lamb,Daniel Lowd, Priscila Machado Vieira Lima, et al.2017. Neural-symbolic learning and reason-ing: A survey and interpretation. arXiv preprintarXiv:1711.03902.

Kurt Bollacker, Colin Evans, Praveen Paritosh, TimSturge, and Jamie Taylor. 2008. Freebase: a collab-oratively created graph database for structuring hu-man knowledge. In Proceedings of the 2008 ACMSIGMOD international conference on Managementof data, pages 1247–1250.

Tolga Bolukbasi, Kai-Wei Chang, James Y Zou,Venkatesh Saligrama, and Adam T Kalai. 2016.Man is to computer programmer as woman is tohomemaker? debiasing word embeddings. In Ad-vances in neural information processing systems,pages 4349–4357.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.2013. Translating embeddings for modeling multi-relational data. In Advances in neural informationprocessing systems, pages 2787–2795.

Tom B Brown, Benjamin Mann, Nick Ryder, MelanieSubbiah, Jared Kaplan, Prafulla Dhariwal, ArvindNeelakantan, Pranav Shyam, Girish Sastry, AmandaAskell, et al. 2020. Language models are few-shotlearners. arXiv preprint arXiv:2005.14165.

William W Cohen, Haitian Sun, R Alex Hofer, andMatthew Siegler. 2020. Scalable neural methods forreasoning with a symbolic knowledge base. arXivpreprint arXiv:2002.06115. Appeared in ICLR-2020.

William W Cohen, Fan Yang, and Kathryn RivardMazaitis. 2017. Tensorlog: Deep learning meetsprobabilistic dbs. arXiv preprint arXiv:1707.05390.

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp,and Sebastian Riedel. 2018. Convolutional 2dknowledge graph embeddings. In Thirty-SecondAAAI Conference on Artificial Intelligence.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186.

Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachan-dran, Graham Neubig, Ruslan Salakhutdinov, andWilliam W Cohen. 2019. Differentiable reasoningover a virtual knowledge base. In International Con-ference on Learning Representations.

Luna Dong. 2017. Amazon product graph.

Thibault Fevry, Livio Baldini Soares, NicholasFitzGerald, Eunsol Choi, and Tom Kwiatkowski.2020. Entities as experts: Sparse memory ac-cess with entity supervision. arXiv preprintarXiv:2004.07202.

Google. 2012. Introducing the knowledge graph:things, not strings.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu-pat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXivpreprint arXiv:2002.08909.

Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Juraf-sky, and Jure Leskovec. 2018. Embedding logicalqueries on knowledge graphs. In Advances in Neu-ral Information Processing Systems, pages 2026–2037.

Mikael Henaff, Jason Weston, Arthur Szlam, AntoineBordes, and Yann LeCun. 2016. Tracking the worldstate with recurrent entity networks. arXiv preprintarXiv:1612.03969.

Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019. Free-baseqa: A new factoid qa data set matching trivia-style question-answer pairs with freebase. In Pro-ceedings of the 2019 Conference of the North Amer-ican Chapter of the Association for Computational

Linguistics: Human Language Technologies, Vol-ume 1 (Long and Short Papers), pages 318–323.

John E Laird, Christian Lebiere, and Paul S Rosen-bloom. 2017. A standard model of the mind: To-ward a common computational framework across ar-tificial intelligence, cognitive science, neuroscience,and robotics. Ai Magazine, 38(4):13–26.

Patrick Lewis, Ethan Perez, Aleksandara Piktus,Fabio Petroni, Vladimir Karpukhin, Naman Goyal,Heinrich Kuttler, Mike Lewis, Wen-tau Yih, TimRocktaschel, et al. 2020. Retrieval-augmented gen-eration for knowledge-intensive nlp tasks. arXivpreprint arXiv:2005.11401.

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu,and Xuan Zhu. 2015. Learning entity and relationembeddings for knowledge graph completion. InTwenty-ninth AAAI conference on artificial intelli-gence.

Robert Logan, Nelson F Liu, Matthew E Peters, MattGardner, and Sameer Singh. 2019. Baracks wifehillary: Using knowledge graphs for fact-aware lan-guage modeling. In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 5962–5971.

Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason We-ston. 2016. Key-value memory networks for di-rectly reading documents. In Proceedings of the2016 Conference on Empirical Methods in NaturalLanguage Processing, pages 1400–1409.

Allen Newell, J. C. Shaw, and Herbert A. Simon. 1959.Report on a general problem-solving program. InProceedings of the International Conference on In-formation Processing.

Allen Newell and Herbert Simon. 1956. The logictheory machine–a complex information processingsystem. IRE Transactions on information theory,2(3):61–79.

H Leo H de Penning, Artur S d’Avila Garcez, Luıs CLamb, and John-Jules C Meyer. 2011. A neural-symbolic cognitive agent for online learning and rea-soning. In Twenty-Second International Joint Con-ference on Artificial Intelligence.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. In Proceedings of the 2018 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long Papers), pages2227–2237.

Fabio Petroni, Tim Rocktaschel, Patrick Lewis, AntonBakhtin, Yuxiang Wu, Alexander H Miller, and Se-bastian Riedel. 2019. Language models as knowl-edge bases? arXiv preprint arXiv:1909.01066.

Gadi Pinkas. 1991. Symmetric neural networks andpropositional logic satisfiability. Neural Computa-tion, 3(2):282–291.

Nina Poerner, Ulli Waltinger, and Hinrich Schutze.2019. Bert is not a knowledge base (yet): Fac-tual knowledge vs. name-based reasoning in unsu-pervised qa. arXiv preprint arXiv:1911.03681.

Colin Raffel, Noam Shazeer, Adam Roberts, KatherineLee, Sharan Narang, Michael Matena, Yanqi Zhou,Wei Li, and Peter J Liu. 2019. Exploring the limitsof transfer learning with a unified text-to-text trans-former. arXiv preprint arXiv:1910.10683.

Hongyu Ren, Weihua Hu, and Jure Leskovec. 2020.Query2box: Reasoning over knowledge graphs invector space using box embeddings. arXiv preprintarXiv:2002.05969. Appeared in ICLR-2020.

Sebastian Riedel, Limin Yao, Andrew McCallum, andBenjamin M Marlin. 2013. Relation extraction withmatrix factorization and universal schemas. In Pro-ceedings of the 2013 Conference of the North Amer-ican Chapter of the Association for ComputationalLinguistics: Human Language Technologies, pages74–84.

Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.How much knowledge can you pack into the pa-rameters of a language model? arXiv preprintarXiv:2002.08910.

Haitian Sun, Andrew O Arnold, Tania Bedrax-Weiss,Fernando Pereira, and William W Cohen. 2020.Guessing what’s plausible but remembering what’strue: Accurate neural reasoning for question-answering. arXiv preprint arXiv:2004.03658.

Haitian Sun, Tania Bedrax-Weiss, and William Cohen.2019a. Pullnet: Open domain question answeringwith iterative retrieval on knowledge bases and text.In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 2380–2390.

Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, KathrynMazaitis, Ruslan Salakhutdinov, and William Co-hen. 2018. Open domain question answering usingearly fusion of knowledge bases and text. In Pro-ceedings of the 2018 Conference on Empirical Meth-ods in Natural Language Processing, pages 4231–4242.

Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang,Mai ElSherief, Jieyu Zhao, Diba Mirza, ElizabethBelding, Kai-Wei Chang, and William Yang Wang.2019b. Mitigating gender bias in natural lan-guage processing: Literature review. arXiv preprintarXiv:1906.08976.

Theo Trouillon, Christopher R Dance, Eric Gaussier,Johannes Welbl, Sebastian Riedel, and GuillaumeBouchard. 2017. Knowledge graph completion via

complex tensor factorization. The Journal of Ma-chine Learning Research, 18(1):4735–4772.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in neural information pro-cessing systems, pages 5998–6008.

Patrick Verga, David Belanger, Emma Strubell, Ben-jamin Roth, and Andrew McCallum. 2016. Multi-lingual relation extraction using compositional uni-versal schema. In Proceedings of the 2016 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, pages 886–896.

Jason Weston, Sumit Chopra, and Antoine Bor-des. 2014. Memory networks. arXiv preprintarXiv:1410.3916.

Wen-tau Yih, Ming-Wei Chang, Xiaodong He, andJianfeng Gao. 2015. Semantic parsing via stagedquery graph generation: Question answering withknowledge base. In Proceedings of the 53rd AnnualMeeting of the Association for Computational Lin-guistics and the 7th International Joint Conferenceon Natural Language Processing (Volume 1: LongPapers), pages 1321–1331, Beijing, China. Associ-ation for Computational Linguistics.