Are Natural Language Inference Models IMPPRESsive?Learning IMPlicature and PRESupposition

Paloma Jeretic∗1, Alex Warstadt∗1, Suvrat Bhooshan2, Adina Williams21Department of Linguistics, New York University

2Facebook AI Research{paloma,warstadt}@nyu.edu, {sbh,adinawilliams}@fb.com

Abstract

Natural language inference (NLI) is an in-creasingly important task for natural languageunderstanding, which requires one to inferwhether a sentence entails another. However,the ability of NLI models to make pragmaticinferences remains understudied. We createan IMPlicature and PRESupposition diagnosticdataset (IMPPRES), consisting of >25k semi-automatically generated sentence pairs illus-trating well-studied pragmatic inference types.We use IMPPRES to evaluate whether BERT,InferSent, and BOW NLI models trained onMultiNLI (Williams et al., 2018) learn to makepragmatic inferences. Although MultiNLIappears to contain very few pairs illustrat-ing these inference types, we find that BERTlearns to draw pragmatic inferences. It re-liably treats scalar implicatures triggered by“some” as entailments. For some presuppo-sition triggers like only, BERT reliably recog-nizes the presupposition as an entailment, evenwhen the trigger is embedded under an entail-ment canceling operator like negation. BOWand InferSent show weaker evidence of prag-matic reasoning. We conclude that NLI train-ing encourages models to learn some, but notall, pragmatic inferences.

1 Introduction

One of the most foundational semantic discover-ies is that systematic rules govern the inferentialrelationships between pairs of natural languagesentences (Aristotle, De Interpretatione, Ch. 6).In natural language processing, Natural LanguageInference (NLI)—a task whereby a system de-termines whether a pair of sentences instantiatesin an entailment, a contradiction, or a neutralrelation—has been useful for training and evaluat-ing models on sentential reasoning. However, lin-guists and philosophers now recognize that there

∗Equal Contribution

Figure 1: Illustration of key properties of classical en-tailments, implicatures, and presuppositions. Solid ar-rows indicate valid commonsense entailments, and ar-rows with X’s indicate lack of entailment. Dashed ar-rows indicate follow up statements with the addition ofin fact, which can either be acceptable (marked with‘7’) or unacceptable (marked with ‘3’).

are separate semantic and pragmatic modes of rea-soning (Grice, 1975; Clark, 1996; Beaver, 1997;Horn and Ward, 2004; Potts, 2015), and it is notclear which of these modes, if either, NLI mod-els learn. We investigate two pragmatic inferencetypes that are known to differ from classical en-tailment: scalar implicatures and presuppositions.As shown in Figure 1, implicatures differ from en-tailments in that they can be denied, and presuppo-sitions differ from entailments in that they are notcanceled when placed in entailment-cancelling en-vironments (e.g., negation, questions).

To enable research into the relationship be-tween NLI and pragmatic reasoning, we introduceIMPPRES, a fine-grained NLI-style diagnostic testdataset for probing how well NLI models performimplicature and presupposition. Containing 25.5Ksentence pairs illustrating key properties of thesepragmatic inference types, IMPPRES is automati-cally generated according to linguist-crafted tem-plates, allowing us to create a large, lexically var-ied, and well controlled dataset targeting specific

arX

iv:2

004.

0306

6v2

[cs

.CL

] 1

4 Ju

l 202

0

instances of both types.We first investigate whether presuppositions

and implicatures are present in NLI models’ train-ing data. We take MultiNLI (Williams et al.,2018) as a case study, and find it has few in-stances of pragmatic inference, and almost nonethat arise from specific lexical triggers (see §4).Given this, we ask whether training on MultiNLIis sufficient for models to generalize about theselargely absent commonsense reasoning types. Wefind that generalization is possible: the BERTNLI model shows evidence of pragmatic reason-ing when tested on the implicature from some tonot all, and the presuppositions of certain triggers(only, cleft existence, possessive existence, ques-tions). We obtain some negative results, that sug-gest that models like BERT still lack a sophisti-cated enough understanding of the meanings of thelexical triggers for implicature and presupposition(e.g., BERT treats several word pairs as synonyms,e.g., most notably, or and and).

Our contributions are: (i) we provide a newdiagnostic test set to probe for pragmatic infer-ences, complete with linguistic controls, (ii) to ourknowledge, we present the first work evaluatingdeep NLI models on specific pragmatic inferences,and (iii) we show that BERT models can performsome types of pragmatic reasoning very well, evenwhen trained on NLI data containing very few ex-plicit examples of pragmatic reasoning. We pub-licly release all IMPPRES data, models evaluated,annotations of MultiNLI, and the scripts used toprocess data.1

2 Background: Pragmatic Inference

We take pragmatic inference to be a relation be-tween two sentences relying on the utterance con-text and the conversational goals of interlocu-tors. Pragmatic inference contrasts with seman-tic entailment, which instead captures the logicalrelationship between isolated sentence meanings(Grice, 1975; Stalnaker, 1974). We present impli-cature and presupposition inferences below.

2.1 Implicature

Broadly speaking, implicatures contrast with en-tailments in that they are inferences suggested bythe speaker’s utterance, but not included in its lit-eral (Grice, 1975). Although there are many types

1github.com/facebookresearch/ImpPres

Type Example

Trigger Jo’s cat yawned.Presupposition Jo has a cat.

Negated Trigger Jo’s cat didn’t yawn.Modal Trigger It’s possible that Jo’s cat yawned.Interrog. Trigger Did Jo’s cat yawn?Cond. Trigger If Jo’s cat yawned, it’s OK.

Negated Prsp. Jo doesn’t have a cat.Neutral Prsp. Amy has a cat.

Table 1: Sample generated presupposition paradigm.Examples adapted from the ‘change-of-state’ dataset.

of implicatures we focus here on scalar implica-tures. Scalar implicatures are inferences, oftenoptional,2 which can be drawn when one mem-ber of a memorized lexical scale (e.g., 〈some, all〉)is uttered (see §6.1). For example, when some-one utters Jo ate some of the cake, they suggestthat Jo didn’t eat all of the cake, (see Figure 1for more examples). According to Neo-Griceanpragmatic theory (Horn, 1989; Levinson, 2000),the inference Jo didn’t eat all of the cake arisesbecause some has a more informative lexical al-ternative all that could have been uttered instead.We expect the speaker to make the most informa-tive true statement:3 as a result, the listener shouldinfer that a stronger statement, where some is re-placed by all, is false.

Implicatures differ from entailments (and, as wewill see, presuppositions; see Figure 1) in that theyare deniable, i.e., they can be explicitly negatedwithout resulting in a contradiction. For example,someone can utter Jo ate some of the cake, fol-lowed by In fact, Jo ate all of it. In this case, theimplicature (i.e., Jo didn’t eat all the cake fromabove) has been denied. We thus distinguish im-plicated meaning from literal, or logical, meaning.

2.2 Presupposition

Presuppositions of a sentence are facts that thespeaker takes for granted when uttering a sentence(Stalnaker, 1974; Beaver, 1997). Presuppositionsare generally associated with the presence of cer-tain expressions, known as presupposition trig-gers. For example, in Figure 1, the definite de-

2Implicature computation can depend on the cooperativityof the speakers, or on any aspect of the context of utterance(lexical, syntactic, semantic/pragmatic, discourse). See De-gen (2015) for a study of the high variability of implicaturecomputation, and the factors responsible for it.

3This follows if we assume that speakers are cooperative(Grice, 1975) and knowledgeable (Gazdar, 1979).

scription the cake triggers the presupposition thatthere is a cake (Russell, 1905). Other examples ofpresupposition triggers are shown in Table 1.

Presuppositions differ from other inferencetypes in that they generally project out of opera-tors like questions and negation, meaning that theyremain valid inferences even when embedded un-der these operators (Karttunen, 1973). The infer-ence that there is a cake survives even when thepresupposition trigger is in a question (Did Jor-dan eat some of the cake?), as shown in Figure 1.However, in questions, classical entailments andimplicatures disappear. Table 1 provides exam-ples of triggers projecting out of several entail-ment canceling operators: negation, modals, in-terrogatives, and conditionals.

It is necessary to clarify in what sense presup-position is a pragmatic inference. There is no con-sensus on whether presuppositions should be con-sidered part of the semantic content of expressions(see Stalnaker, 1974; Heim, 1983, for opposingviews). However, presuppositions may come tobe inferred via accommodation, a pragmatic pro-cess by which a listener infers the truth of somenew fact based on its being presupposed by thespeaker (Lewis, 1979). For instance, if Jordan tellsHarper that the King of Sweden wears glasses, andHarper did not previously know that Sweden hasa king, they would learn this fact by accommo-dation. With respect to NLI, any presuppositionin the premise (short of world knowledge) will benew information, and therefore accommodation isnecessary to recognize it as entailed.

3 Related Work

NLI has been framed as a commonsense reason-ing task (Dagan et al., 2006; Manning, 2006). Oneearly formulation of NLI defines “entailment” asholding for sentences p and h whenever, “typi-cally, a human reading p would infer that h ismost likely true. . . [given] common human under-standing of language [and] common backgroundknowledge” (Dagan et al., 2006). Although thissparked debate regarding the terms inference andentailment—and whether an adequate notion of“inference” could be defined (Zaenen et al., 2005;Manning, 2006; Crouch et al., 2006)—in recentwork, a commonsense formulation of “inference”is widely adopted (Bowman et al., 2015; Williamset al., 2018) largely because it facilitates untrainedannotators’ participation in dataset creation.

NLI itself has been steadily gaining in popular-ity; many datasets for training and/or testing sys-tems are now available including: FraCaS (Cooperet al., 1994), RTE (Dagan et al., 2006; Mirkinet al., 2009; Dagan et al., 2013), Sentences In-volving Compositional Knowledge (Marelli et al.,2014, SICK), large scale imaging captioning asNLI (Bowman et al., 2015, SNLI), recastingother datasets into NLI (Glickman, 2006; Whiteet al., 2017; Poliak et al., 2018), ordinal common-sense inference (Zhang et al., 2017, JOCI), Multi-Premise Entailment (Lai et al., 2017, MPE), NLIover multiple genres of written and spoken En-glish (Williams et al., 2018, MultiNLI), adversar-ially filtered common sense reasoning sentences(Zellers et al., 2018, 2019, (Hella)SWAG), ex-plainable annotations for SNLI (Camburu et al.,2018, e-SNLI), cross-lingual NLI (Conneau et al.,2018, XNLI), scientific questioning answering asNLI (Khot et al., 2018, SciTail), NLI recast-question answering (part of Wang et al. 2019,GLUE), NLI for dialog (Welleck et al., 2019),and NLI over narratives that require drawing in-ferences to the most plausible explanation fromtext (Bhagavatula et al., 2020, αNLI). Other NLIdatasets are created to identify where models fail(Glockner et al., 2018; Naik et al., 2018; McCoyet al., 2019; Schmitt and Schutze, 2019), manyof which are also automatically generated (Geigeret al., 2018; Yanaka et al., 2019a,b; Kim et al.,2019; Nie et al., 2019; Richardson et al., 2020).

As datasets for NLI become increasingly nu-merous, one might wonder, do we need yet anotherNLI dataset? In this case, the answer is clearlyyes: despite NLI’s formulation as a commonsensereasoning task, it is still unknown whether thisframing has resulted in models that learn specificmodes of pragmatic reasoning. IMPPRES is thefirst NLI dataset to explicitly probe whether mod-els trained on commonsense reasoning actually dotreat pragmatic inferences like implicatures andpresuppositions as entailments without additionaltraining on these specific inference types.

Beyond NLI, several recent works introduceresources for evaluating sentence understandingmodels for knowledge of pragmatic inferences.On the presupposition side, datasets such asMegaVeridicality (White and Rawlins, 2018) andCommitmentBank (de Marneffe et al., 2019) com-pile gradient crowdsourced judgments regardinghow likely a clause embedding predicate is to trig-

ger a presupposition that its complement clause istrue. White et al. (2018) and Jiang and de Marn-effe (2019) find that LSTMs trained on a gradientevent factuality prediction task on these respectivedatasets make systematic errors. Turning to impli-catures, Degen (2015) introduces a dataset mea-suring the strength of the implicature from someto not all with crowd-sourced judgments. Schus-ter et al. (2020) find that an LSTM with supervi-sion on this dataset can predict human judgmentswell. These resources all differ from IMPPRES intwo respects: First, their empirical scopes are allsomewhat narrower, as all these datasets focus ononly a single class of presupposition or implica-ture triggers. Second, the use of gradient judg-ments makes it non-trivial to use these datasets toevaluate NLI models, which are trained to makecategorical predictions about entailment. Both ap-proaches have advantages, and we leave a directcomparison for future work.

Outside the topic of sentential inference,Rashkin et al. (2018) propose a new task wherea model must label actor intents and reactions forparticular actions described using text. Cianfloneet al. (2018) create sentence-level adverbial pre-supposition datasets and train a binary classifierto detect contexts in which presupposition triggers(e.g., too, again) can be used.

4 Annotating MultiNLI for Pragmatics

In this section, we present results of an annotationeffort that show that MultiNLI contains very lit-tle explicit evidence of pragmatic inferences of thetype tested by IMPPRES. Although Williams et al.(2018) report that 22% of the MultiNLI devel-opment set sentence pairs contain lexical triggers(such as regret or stopped) in the premise and/orhypothesis, the mere presence of presupposition-triggering lexical items in the data does not showthat MultiNLI contains evidence that presupposi-tions are entailments, since the sentential infer-ence may focus on other types of information.To address this, we randomly selected 200 sen-tence pairs from the MultiNLI matched develop-ment set and presented them to three expert anno-tators with a combined total of 17 years of train-ing in formal semantics and pragmatics.4 Anno-tators answered the following questions for eachpair: (1) are the sentences P and H related by apresupposition/implicature relation (entails/is en-

4The full annotations are on the IMPPRES repository.

tailed by, negated or not); (2) what subtype of in-ference (e.g., existence presupposition, 〈some, all〉implicature); (3) is the presupposition trigger em-bedded under an entailment-cancelling operator?

Agreement among annotators was low, suggest-ing that few MultiNLI pairs are paradigmatic casesof implicatures or presuppositions. We found only8 presupposition pairs and 3 implicature pairs onwhich two or more annotators agreed. Moreover,we found only one example illustrating a particu-lar inference type tested in IMPPRES (the presup-position of possessed definites). All others weretagged as existence presuppositions and conversa-tional implicatures (i.e. loose inferences depen-dent on world knowledge). The union of anno-tations was much larger: 42% of examples wereidentified by at least one annotator as a presuppo-sition or implicature (51 presuppositions and 42implicatures, with 10 sentences receiving diver-gent tags). However, of these, only 23 presupposi-tions and 19 implicatures could reliably be used tolearn pragmatic inference (in 14 cases, the giventag did not match the pragmatic inference, and in27 cases, computing the inference did not affectthe relation type). Again, the large majority of im-plicatures were conversational, and most presup-positions were existential, and generally not linkedto particular lexical triggers (e.g., topic marking).

We conclude that the MultiNLI dataset at bestcontains some evidence of loose pragmatic rea-soning based on world knowledge and discoursestructure, but almost no explicit information rel-evant to lexically triggered pragmatic inference,which is of the type tested in this paper.

5 Methods

Data Generation. IMPPRES consists of semi-automatically generated pairs of sentences withNLI labels illustrating key properties of implica-tures and presuppositions. We generate IMPPRES

using a codebase developed by Warstadt et al.(2019a) and significantly expanded for the BLiMPdataset (Warstadt et al., 2019b). The codebase, in-cluding our scripts and documentation, are pub-licly available.5 Each sentence type in IMPPRES

is generated according to a template that specifiesthe linear order of the constituents in the sentence.The constituents are sampled from a vocabularyof over 3000 lexical items annotated with gram-matical features needed to ensure morphological,

5github.com/alexwarstadt/data generation

Premise Hypothesis Relation type Logical label Pragmatic label Item type

some not all implicature (+ to −) neutral entailment targetnot all some implicature (− to +) neutral entailment target

some all negated implicature (+) neutral contradiction targetall some reverse negated implicature (+) entailment contradiction targetnot all none negated implicature (−) neutral contradiction targetnone not all reverse negated implicature (−) entailment contradiction target

all none opposite contradiction contradiction controlnone all opposite contradiction contradiction controlsome none negation contradiction contradiction controlnone some negation contradiction contradiction controlall not all negation contradiction contradiction controlnot all all negation contradiction contradiction control

Table 2: Paradigm for the scalar implicature datasets, with 〈some, all〉 as an example.

syntactic, and semantic well-formedness. All sen-tences generated from a given template are struc-turally analogous up to the specified constituents,but may vary in sub-constituents. For instance, ifthe template calls for a verb phrase, the generatedconstituent may include a direct object or comple-ment clause, depending on the argument structureof the sampled verb. See §6.1 and 7.1 for descrip-tions of the sentence types in the implicature andpresupposition data.

Generating data lets us control the lexical andsyntactic content so that we can guarantee that thesentence pairs in IMPPRES evaluate the desiredphenomenon (see Ettinger et al., 2016, for relateddiscussion). Furthermore, the codebase we use al-lows for greater lexical and syntactic variety thanin many other templatic datasets (see discussionin Warstadt et al., 2019b). One limitation of thismethodology is that generated sentences, whilegenerally grammatical, often describe highly un-likely scenarios, or include low frequency combi-nations of lexical items (e.g., Sabrina only revealsthis pasta). Another limitation is that generateddata is of limited use for training models, since itcontains simple regularities that supervised classi-fiers may learn to exploit. Thus, we create IMP-PRES solely for the purpose of evaluating NLImodels trained on standard datasets like MultiNLI.

Models. Our experiments evaluate NLI modelstrained on MultiNLI and built on top of three sen-tence encoding models: a bag of words (BOW)model, InferSent (Conneau et al., 2017), andBERT-Large (Devlin et al., 2019). The BOWand InferSent models use 300D GloVe embed-dings as word representations (Pennington et al.,2014). For the BOW baseline, word embeddingsfor premise and hypothesis are separately summed

to create sentence representations, which are con-catenated to form a single sentence-pair represen-tation which is fed to a logistic regression softmaxclassifier. For the InferSent model, GloVe em-beddings for the words in premise and hypothesisare respectively fed into a bidirectional LSTM, af-ter which we concatenate the representations forpremise and hypothesis, their difference, and theirelement-wise product (Mou et al., 2016). BERTis a multilayer bidirectional transformer pretrainedwith the masked language modelling and next se-quence prediction objectives, and finetuned on theMultiNLI dataset. We concatenate the premiseand hypothesis after a special [CLS] token andseparated them with the [SEP] token. The BERTrepresentation for the [CLS] token is fed into clas-sifier. We use Huggingface’s pre-trained BERTtrained on Toronto books (Zhu et al., 2015).6

The BOW and InferSent models have develop-ment set accuracies of 49.6% and 67.6%. Thedevelopment set accuracy for BERT-Large onMultiNLI is 86.6%, similar to the results achievedby (Devlin et al., 2019), but somewhat lower thanstate-of-the-art (currently 90.8% on test from theensembled RoBERTa model with long pretrainingoptimization, Liu et al. 2019).

6 Experiment 1: Scalar Implicatures

6.1 Scalar Implicature Datasets

The scalar implicature portion of IMPPRES in-cludes six datasets, each isolating a different scalarimplicature trigger from six types of lexical scales(of the type described in §2): determiners 〈some,all〉, connectives 〈or, and〉, modals 〈can, have to〉,numerals 〈2,3〉, 〈10,100〉, scalar adjectives, and

6github.com/huggingface/pytorch-pretrained-BERT/

Figure 2: Results on Controls (Implicatures)

Figure 3: Results on Target Conditions (Implicatures)

verbs, e.g., 〈good, excellent〉, 〈run, sprint〉. Exam-ples pairs of each implicature trigger can be foundin Table 4 in the Appendix. For each type, we gen-erate 100 paradigms, each consisting of 12 uniquesentence pairs, as shown in Table 2.

The six target sentence pairs comprise two mainrelation types: ‘implicature’ and ‘negated implica-ture’. Pairs tagged as ‘implicature’ have a premisethat implicates the hypothesis (e.g., some and notall). For ‘negated implicature’, the premise im-plicates the negation of the hypothesis (e.g., someand all), or vice versa (e.g., all and some). Sixcontrol pairs are logical contradictions, represent-ing either scalar ‘opposites’ (e.g., all and none), or‘negations’ (e.g., not all and all; some and none),probing the models’ basic grasp of negation.

As mentioned in §2.1, implicature computationis variable and dependent on the context of utter-ance. Thus, we anticipate two possible rationalbehaviors for a MultiNLI-trained model tested onan implicature: (a) be pragmatic, and compute theimplicature, concluding that the premise and hy-pothesis are in an ‘entailment’ relation, (b) be log-ical, i.e., consider only the literal content, and notcompute the implicature, concluding they are in a‘neutral’ relation. Thus, we measure both possibleconclusions, by tagging sentence pairs for scalarimplicature with two sets of NLI labels to reflectthe behavior expected under “logical” and “prag-matic” modes of inference, as shown in Table 2.

6.2 Implicatures Results & Discussion

We first evaluate model performance on the con-trols, shown in Figure 2. Success on these controlsis a necessary condition for us to conclude that amodel has learned the basic function of negation(not, none, neither) and the scalar relationship be-tween terms like some and all. We find that BERTperforms at ceiling on control conditions for allimplicature types, in contrast with InferSent andBOW, whose performance is very variable. Sinceonly BERT passes all controls, its results on thetarget items are most interpretable. Full resultsfor all models and target conditions by implicaturetrigger are in Figures 8–13 in the Appendix.

For connectives, scalar adjectives and verbs, theBERT model results correspond neither to the hy-pothesized pragmatic nor logical behavior. In fact,for each of these subdatasets, the results are con-sistent with a treatment of scalemates (e.g., andand or; good and excellent) as synonyms, e.g. itevaluates the ‘negated implicature’ sentence pairsas ‘entailment’ in both directions. This reveals acoarse-grained knowledge of these meanings thatlacks information about asymmetric informativityrelations between scalemates. Results for modals(can and have to) are split between the three la-bels, not showing any predicted logical or prag-matic pattern. We conclude that BERT has insuf-ficient knowledge of the meaning of these words.

In addition to pragmatic and logical interpreta-tions, numerals can also be interpreted as exactcardinalities. We thus predict three different be-haviors: logical “at least n”, pragmatic “at leastn”, and “exactly n”. We observe that results areinconsistent: neither the “exactly” nor “at least”interpretations hold across the board.

Figure 4: BERT results for scalar implicatures trig-gered by determiners 〈some, all〉, by target condition.

For the determiner dataset (some-all), Figure 4breaks down the results by condition and showsthat BERT behaves as though it performs prag-matic and logical reasoning in different condi-tions. Overall, it predicts a pragmatic relationmore frequently (55% vs. 36%), and only 9% ofresults are consistent with neither mode of rea-soning. Furthermore, the proportion of pragmaticreasoning shows consistent effects of sentence or-der (i.e., whether the implicature trigger is in thepremise or the hypothesis), and the presence ofnegation in one or both sentences. Pragmatic rea-soning is consistently higher when the implicaturetrigger is in the premise, which we can see in theresults for negated implicatures: the some–all con-dition shows more pragmatic behavior comparedto the all–some condition (a similar behavior is ob-served with the not all vs. none conditions).

Generally, the presence of negation lowers ratesof pragmatic reasoning. First, the negated im-plicature conditions can be subdivided into pairswith and without negation. Among the negatedones, pragmatic reasoning is lower than for non-negated ones. Second, having negation in thepremise rather than the hypothesis makes prag-matic reasoning lower: among pairs tagged as di-rect implicatures (some vs. not all), there is higherpragmatic reasoning with non-negated some in thepremise than with negated not all. Finally, weobserve that pragmatic rates are lower for somevs. not all than for some vs. all. In this finalcase, pragmatic reasoning could be facilitated byexplicit presentation of the two items on the scale.

In sum, for the datasets besides determiners, wefind evidence that BERT fails to learn even the log-ical relations between scalemates, ruling out thepossibility of computing scalar implicatures. It re-mains possible that BERT could learn these logicalrelations with explicit supervision (see Richard-

Presuppositions Label ItemPremise Hypothesis Type

*Trigger Prsp entailment target*Trigger Neg. Prsp contradiction target*Trigger Neut. Prsp neutral target

Neg. Trigger Trigger contradiction controlModal Trigger Trigger neutral controlInterrog. Trigger Trigger neutral controlCond. Trigger Trigger neutral control

Table 3: Paradigm for the presupposition target (top)and control datasets (bottom). For space, *Triggerrefers to either plain, Negated, Modal, Interrogative, orConditional Triggers as per Table 1.

son et al., 2020), but it is clear that these are notlearned from training on MultiNLI. Only the deter-miner dataset was informative in showing the ex-tent of the NLI BERT model’s pragmatic reason-ing, since it alone showed a fine-grained enoughunderstanding of the semantic relationship of thescalemates, like some and all. In this setting BERTreturned impressive results showing a high propor-tion of pragmatic reasoning compared to logicalreasoning, which was affected by sentence orderand presence of negation in a predictable way.

7 Experiment 2: Presuppositions

7.1 Presupposition Datasets

The presupposition portion of IMPPRES includeseight datasets, each isolating a different kind ofpresupposition trigger. The full set of triggers isshown in Table 5 in the Appendix. For each type,we generate 100 paradigms, with each paradigmconsisting of 19 unique sentence pairs. (Examplesof the sentence types are in Table 1).

Of the 19 sentence pairs, 15 contain targetitems. The first target item tests whether the modelcorrectly determines that the presupposition trig-ger entails its presupposition. The next two alterthe presupposition, either negating it, or replacinga constituent, leading to contradiction and neutral-ity, respectively. The remaining 12 show that therelation between the trigger and the (altered) pre-supposition is not affected by embedding the trig-ger under various entailment-canceling operators.4 control items are designed to test the basic ef-fect of entailment-canceling operators—negation,modals, interrogatives, and conditionals. In eachcontrol, the premise is a presupposition triggerembedded under an entailment-canceling opera-tor, and the hypothesis is an unembedded sentencecontaining the trigger. These controls are neces-

Figure 5: Results on Controls (Presuppositions).

sary to establish whether models learn that presup-positions behave differently under these operatorsthan do classical semantic entailments.

7.2 Presupposition Results & Discussion

The results from presupposition controls are inFigure 5. BERT performs well above chanceon each control (acc. > 0.33), whereas BOWand InferSent perform at or below chance. Inthe “negated” condition, BERT correctly identi-fies that the trigger is contradicted by its negation100% of the time, e.g., Jo’s cat didn’t go con-tradicts Jo’s cat went. In the other conditions, itcorrectly identifies the neutral relation the major-ity of the time, e.g., Did Jo’s cat go? is neutralwith respect to Jo’s cat went. This indicates thatBERT mostly learns that negation, modals, inter-rogatives, and conditionals cancel classical entail-ments, while BOW and InferSent do not capturethe ordinary behavior of these common operators.

Next, we test whether models identify presup-positions of the premise as entailments, e.g., thatJo’s cat went entails that Jo has a cat. Recall from§2.2 that this is akin to a listener accommodatinga presupposition. The results in Figure 6 showthat each of the three models accommodates somepresuppositions, but this depends on both the na-ture of the presupposition and the model. For in-stance, the BOW and InferSent models accommo-date presuppositions of nearly all trigger types atwell above chance rates (acc. � 33%). For theuniqueness presupposition of clefts, these modelsgenerally correctly predict an entailment (acc. >90%), but for most triggers, performance is lessreliable. By contrast, BERT’s behavior is bimodal.It always accommodates the existence presupposi-tions of clefts and possessed definites, as well asthe presupposition of only, but almost never ac-commodates any presupposition involving numer-acy, e.g. Both flowers that bloomed died entails

Figure 6: Results for the unembedded trigger pairedwith positive presupposition.

There are exactly two flowers that bloomed.7

Finally, we evaluate whether models predict thatpresuppositions project out of entailment cancel-ing operators (e.g., that Did Jo’s cat go? entailsthat Jo has a cat). We can only consider such aprediction as evidence of projection if two condi-tions hold: (a) the model correctly identifies thatthe relevant operator cancels entailments in thecontrol from the same paradigm (e.g., Did Jo’s catgo? is neutral with respect to Jo’s cat went), and(b) the model identifies the presupposition as anentailment when the trigger is unembedded in thesame paradigm (e.g. Jo’s cat went entails Jo has acat). Otherwise, a model might correctly predictentailment essentially by accident if, for instance,it systematically ignores negation. For this reason,we filter out results for the target conditions thatdo not meet these criteria.

Figure 7 shows results for the target conditionsafter filtering. While InferSent rarely predicts thatpresuppositions project, we find strong evidencethat the BERT and BOW models do. Specifi-cally, they correctly identify that the premise en-tails the presupposition (acc. ≥ 80% for BERT,acc. ≥ 90% for BOW). Furthermore, BERT is theonly model to reliably identify (i.e., over 90% ofthe time) that the negation of the presupposition iscontradicted. These results hold irrespective of theentailment canceling operator. No model reliablyperforms above chance when the presupposition isaltered to be neutral (e.g., Did Jo’s cat go? is neu-

7The presence of exactly might contribute to poor perfor-mance on numeracy examples. We suspect MultiNLI annota-tors may have used it disproportionately for neut. hypotheses.

Figure 7: Results for presupposition target conditionsinvolving projection.

tral with respect to Jo has a cat).It is surprising that the simple BOW model can

learn some of the projective behavior of presup-positions. One explanation for this finding is thatmany of the key features of presupposition projec-tion are insensitive to word order. If a lexical pre-supposition trigger is present at all in a sentence, apresupposition will generally arise irrespective ofits position in the sentence. There are some edgecases where this heuristic is insufficient, but IMP-PRES is not designed to test such cases.

To summarize, training on NLI is sufficient forall models we evaluate to learn to accommodatepresuppositions of a wide variety of unembeddedtriggers, though BERT rejects presuppositions in-volving numeracy. Furthermore, BERT and eventhe BOW model appear to learn the characteristicprojective behavior of some presuppositions.

8 General Discussion & Conclusion

We observe some encouraging results in §6–7. Wefind strong evidence that BERT learns scalar im-plicatures associated with determiners some andall. Pragmatic or logical reasoning was not diag-nosable for the other scales, whose meaning wasnot fully understood by our models (as most scalarpairs were treated as synonymous). In the case ofpresuppositions, the BERT NLI models, and BOWto some extent, perform well on a number of oursubdatasets (only, cleft existence, possessive exis-tence, questions). For the other subdatasets, themodels did not perform as expected on the basicunembedded presupposition triggers, again sug-

gesting the model’s lack of knowledge of the basicmeaning of these words. Though their behavioris far from systematic, this is suggestive evidencethat some NLI models can perform in ways thatcorrelate with human-like pragmatic behavior.

Given that MultiNLI contains few examples ofthe type found in IMPPRES (see §4), where mightour positive results come from? There are two po-tential sources of signal for the BERT model: NLItraining, and pretraining (either BERT’s maskedlanguage modeling objective or its input word em-beddings). NLI training provides specific exam-ples of valid (or invalid) inferences constitutingan incomplete characterization of what common-sense inference is in general. Since presupposi-tions and scalar implicatures triggered by specificlexical items are largely absent from the MultiNLIdata used for NLI training, any positive results onIMPPRES would likely use prior knowledge fromthe pretraining stage to make an inductive leap thatpragmatic inferences are valid commonsense in-ferences. The natural language text used for pre-training certainly contains pragmatic information,since, like any natural language data, it is pro-duced with the assumption that readers are capa-ble of pragmatic reasoning. Maybe this inducespatterns in the data that make the nature of thoseassumptions recoverable from the data itself.

This work is an initial step towards rigorouslyinvestigating the extent to which NLI models learnsemantic versus pragmatic inference types. Wehave introduced a new dataset IMPPRES for prob-ing this question, which can be reused to evaluatepragmatic performance of any NLI given model.

Acknowledgments

This material is based upon work supported by theNational Science Foundation (NSF) under GrantNo. 1850208 awarded to A. Warstadt. Any opin-ions, findings, and conclusions or recommenda-tions expressed in this material are those of theauthor(s) and do not necessarily reflect the viewsof the NSF. Thanks to the FAIR NLP & Con-versational AI Group, the Google AI NLP group,and the NYU ML2, including Sam Bowman, HeHe, Phu Mon Htut, Katharina Kann, Haokun Liu,Ethen Perez, Richard Pang, Clara Vania for discus-sions on the topic, and/or feedback on an earlierdraft. Additional thanks to Marco Baroni, HagenBlix, Emmanuel Chemla, Aaron Steven White,and Luke Zettlemoyer for insightful comments.

ReferencesAristotle. De Interpretatione.

David I. Beaver. 1997. Presupposition. In Handbookof logic and language, pages 939–1008. Elsevier.

Chandra Bhagavatula, Ronan Le Bras, ChaitanyaMalaviya, Keisuke Sakaguchi, Ari Holtzman, Han-nah Rashkin, Doug Downey, Scott Wen-tau Yih, andYejin Choi. 2020. Abductive commonsense reason-ing. In Proceedings of the 2020 International Con-ference on Learning Representations (ICLR).

Samuel R. Bowman, Gabor Angeli, Christopher Potts,and Christopher D. Manning. 2015. A large anno-tated corpus for learning natural language inference.In Proceedings of the 2015 Conference on Empiri-cal Methods in Natural Language Processing, pages632–642. Association for Computational Linguis-tics.

Oana-Maria Camburu, Tim Rocktaschel, ThomasLukasiewicz, and Phil Blunsom. 2018. e-SNLI:Natural Language Inference with Natural LanguageExplanations. In Advances in Neural InformationProcessing Systems, pages 9539–9549.

Andre Cianflone, Yulan Feng, Jad Kabbara, and JackieChi Kit Cheung. 2018. Let’s do it “again”: A firstcomputational approach to detecting adverbial pre-supposition triggers. In Proceedings of the 56th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 2747–2755, Melbourne, Australia. Association for Com-putational Linguistics.

Herbert H Clark. 1996. Using language. CambridgeUniversity Press.

Alexis Conneau, Douwe Kiela, Holger Schwenk, LoıcBarrault, and Antoine Bordes. 2017. Supervisedlearning of universal sentence representations fromnatural language inference data. In Proceedings ofthe 2017 Conference on Empirical Methods in Nat-ural Language Processing, pages 670–680. Associ-ation for Computational Linguistics.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad-ina Williams, Samuel Bowman, Holger Schwenk,and Veselin Stoyanov. 2018. XNLI: Evaluatingcross-lingual sentence representations. In Proceed-ings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 2475–2485.Association for Computational Linguistics.

Robin Cooper, Richard Crouch, Jan Van Eijck, ChrisFox, Josef Van Genabith, Jan Jaspers, Hans Kamp,Manfred Pinkal, Massimo Poesio, Stephen Pulman,et al. 1994. FraCaS: A framework for computationalsemantics. Deliverable D6.

Richard Crouch, Lauri Karttunen, and Annie Zaenen.2006. Circumscribing is not excluding: A reply toManning. Ms., Palo Alto Research Center.

Ido Dagan, Oren Glickman, and Bernardo Magnini.2006. The PASCAL recognising textual entailmentchallenge. In Machine learning challenges. evalu-ating predictive uncertainty, visual object classifica-tion, and recognising tectual entailment, pages 177–190. Springer.

Ido Dagan, Dan Roth, Mark Sammons, and Fabio Mas-simo Zanzotto. 2013. Recognizing textual entail-ment: Models and applications. Synthesis Lectureson Human Language Technologies, 6(4):1–220.

Judith Degen. 2015. Investigating the distribution ofsome (but not all) implicatures using corpora andweb-based methods. Semantics and Pragmatics,8:11–1.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Allyson Ettinger, Ahmed Elgohary, and Philip Resnik.2016. Probing for semantic evidence of compositionby means of simple classification tasks. In Proceed-ings of the 1st Workshop on Evaluating Vector-SpaceRepresentations for NLP, pages 134–139, Berlin,Germany. Association for Computational Linguis-tics.

Gerald Gazdar. 1979. Pragmatics, implicature, presu-position and logical form. Academic Press, NY.

Atticus Geiger, Ignacio Cases, Lauri Karttunen, andChristopher Potts. 2018. Stress-testing neural mod-els of natural language inference with multiply-quantified sentences. In CoRR.

Oren Glickman. 2006. Applied textual entailment.Bar-Ilan University.

Max Glockner, Vered Shwartz, and Yoav Goldberg.2018. Breaking NLI systems with sentences that re-quire simple lexical inferences. In Proceedings ofthe 56th Annual Meeting of the Association for Com-putational Linguistics (Volume 2: Short Papers),pages 650–655. Association for Computational Lin-guistics.

H Paul Grice. 1975. Logic and conversation. 1975,pages 41–58.

Irene Heim. 1983. On the projection problem for pre-suppositions. Formal semantics: The essential read-ings, pages 249–260.

Laurence Horn. 1989. A natural history of negation.University of Chicago Press.

Laurence R Horn and Gregory L Ward. 2004. Thehandbook of pragmatics. Wiley Online Library.

Nanjiang Jiang and Marie-Catherine de Marneffe.2019. Do you know that florence is packed with vis-itors? evaluating state-of-the-art models of speakercommitment. In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 4208–4213, Florence, Italy. Associa-tion for Computational Linguistics.

Lauri Karttunen. 1973. Presuppositions of compoundsentences. Linguistic inquiry, 4(2):169–193.

Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.SciTail: A textual entailment dataset from sciencequestion answering. In Proceedings of Associa-tion for the Advancement of Artificial Intelligence(AAAI).

Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia,Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross,Tal Linzen, Benjamin Van Durme, Samuel R. Bow-man, and Ellie Pavlick. 2019. Probing what dif-ferent NLP tasks teach machines about functionword comprehension. In Proceedings of the EighthJoint Conference on Lexical and Computational Se-mantics (*SEM 2019), pages 235–249, Minneapolis,Minnesota. Association for Computational Linguis-tics.

Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017.Natural language inference from multiple premises.In Proceedings of the Eighth International JointConference on Natural Language Processing (Vol-ume 1: Long Papers), pages 100–109, Taipei, Tai-wan. Asian Federation of Natural Language Pro-cessing.

Stephen C Levinson. 2000. Presumptive meanings:The theory of generalized conversational implica-ture. MIT press.

David Lewis. 1979. Scorekeeping in a language game.In Semantics from different points of view, pages172–187. Springer.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019.RoBERTa: A robustly optimized BERT pretrainingapproach. arXiv preprint arXiv:1907.11692.

Christopher D. Manning. 2006. Local textual infer-ence: It’s hard to circumscribe, but you know itwhen you see it – and NLP needs it. Ms., StanfordUniversity.

Marco Marelli, Luisa Bentivogli, Marco Baroni, Raf-faella Bernardi, Stefano Menini, and Roberto Zam-parelli. 2014. SemEval-2014 task 1: Evaluation ofcompositional distributional semantic models on fullsentences through semantic relatedness and textualentailment. In Proceedings of the 8th InternationalWorkshop on Semantic Evaluation (SemEval 2014),pages 1–8, Dublin, Ireland. Association for Compu-tational Linguistics.

Marie-Catherine de Marneffe, Mandy Simons, and Ju-dith Tonhauser. 2019. The CommitmentBank: In-vestigating projection in naturally occurring dis-course. In Proceedings of Sinn und Bedeutung, vol-ume 23, pages 107–124.

Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.Right for the wrong reasons: Diagnosing syntacticheuristics in natural language inference. In Proceed-ings of the 57th Annual Meeting of the Associationfor Computational Linguistics, pages 3428–3448,Florence, Italy. Association for Computational Lin-guistics.

Shachar Mirkin, Roy Bar-Haim, Ido Dagan, EyalShnarch, Asher Stern, Idan Szpektor, and JonathanBerant. 2009. Addressing discourse and documentstructure in the RTE search task. In Textual AnalysisConference.

Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan,and Zhi Jin. 2016. Natural language inference bytree-based convolution and heuristic matching. InProceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics (Volume 2:Short Papers), pages 130–136, Berlin, Germany. As-sociation for Computational Linguistics.

Aakanksha Naik, Abhilasha Ravichander, NormanSadeh, Carolyn Rose, and Graham Neubig. 2018.Stress test evaluation for natural language inference.In Proceedings of the 27th International Conferenceon Computational Linguistics, pages 2340–2353,Santa Fe, New Mexico, USA. Association for Com-putational Linguistics.

Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019.Analyzing compositionality-sensitivity of NLI mod-els. In Proceedings of the AAAI Conference on Arti-ficial Intelligence, volume 33, pages 6867–6874.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. GloVe: Global vectors for wordrepresentation. In Proceedings of the 2014 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 1532–1543, Doha,Qatar. Association for Computational Linguistics.

Adam Poliak, Aparajita Haldar, Rachel Rudinger,J. Edward Hu, Ellie Pavlick, Aaron Steven White,and Benjamin Van Durme. 2018. Collecting di-verse natural language inference problems for sen-tence representation evaluation. In Proceedings ofthe 2018 Conference on Empirical Methods in Nat-ural Language Processing, pages 67–81, Brussels,Belgium. Association for Computational Linguis-tics.

Christopher Potts. 2015. Presupposition and implica-ture. The handbook of contemporary semantic the-ory, 2:168–202.

Hannah Rashkin, Maarten Sap, Emily Allaway,Noah A. Smith, and Yejin Choi. 2018. Event2Mind:

Commonsense inference on events, intents, and re-actions. In Proceedings of the 56th Annual Meet-ing of the Association for Computational Linguis-tics (Volume 1: Long Papers), pages 463–473, Mel-bourne, Australia. Association for ComputationalLinguistics.

Kyle Richardson, Hai Hu, Lawrence S Moss, andAshish Sabharwal. 2020. Probing natural languageinference models through semantic fragments. InProceedings of the Thirty-Fourth AAAI Conferenceon Artificial Intelligence (AAAI-20).

Bertrand Russell. 1905. On denoting. Mind,14(56):479–493.

Martin Schmitt and Hinrich Schutze. 2019. SherLIiC:A typed event-focused lexical inference benchmarkfor evaluating natural language inference. In Pro-ceedings of the 57th Conference of the Associationfor Computational Linguistics, pages 902–914, Flo-rence, Italy. Association for Computational Linguis-tics.

Sebastian Schuster, Yuxing Chen, and Judith Degen.2020. Harnessing the richness of the linguistic sig-nal in predicting pragmatic inferences. In Proceed-ings of the 58th Annual Meeting of the Associationfor Computational Linguistics.

Robert Stalnaker. 1974. Pragmatic presuppositions. InMilton K. Munitz and Peter K. Unger, editors, Se-mantics and Philosophy, pages 135–148. New YorkUniversity Press.

Alex Wang, Amapreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R Bowman. 2019.GLUE: A multi-task benchmark and analysis plat-form for natural language understanding. In Pro-ceedings of the Interantional Conference on Learn-ing Representations (ICLR).

Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Ha-gen Blix, Yining Nie, Anna Alsop, Shikha Bordia,Haokun Liu, Alicia Parrish, Sheng-Fu Wang, JasonPhang, Anhad Mohananey, Phu Mon Htut, PalomaJeretic, and Samuel R. Bowman. 2019a. Investi-gating BERT’s knowledge of language: Five analy-sis methods with NPIs. In Proceedings of EMNLP-IJCNLP, pages 2870–2880.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo-hananey, Wei Peng, Sheng-Fu Wang, and Samuel RBowman. 2019b. BLiMP: The benchmark of lin-guistic minimal pairs for English. arXiv preprintarXiv:1912.00582.

Sean Welleck, Jason Weston, Arthur Szlam, andKyunghyun Cho. 2019. Dialogue natural languageinference. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguis-tics, pages 3731–3741, Florence, Italy. Associationfor Computational Linguistics.

Aaron Steven White, Pushpendre Rastogi, Kevin Duh,and Benjamin Van Durme. 2017. Inference is ev-erything: Recasting semantic resources into a uni-fied evaluation framework. In Proceedings of theEighth International Joint Conference on NaturalLanguage Processing (Volume 1: Long Papers),pages 996–1005, Taipei, Taiwan. Asian Federationof Natural Language Processing.

Aaron Steven White and Kyle Rawlins. 2018. The roleof veridicality and factivity in clause selection. InProceedings of the 48th Annual Meeting of the NorthEast Linguistic Society, Amherst, MA, USA. GLSAPublications.

Aaron Steven White, Rachel Rudinger, Kyle Rawlins,and Benjamin Van Durme. 2018. Lexicosyntacticinference in neural models. In Proceedings of the2018 Conference on Empirical Methods in Natu-ral Language Processing, pages 4717–4724, Brus-sels, Belgium. Association for Computational Lin-guistics.

Adina Williams, Nikita Nangia, and Samuel Bowman.2018. A broad-coverage challenge corpus for sen-tence understanding through inference. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume 1(Long Papers), pages 1112–1122. Association forComputational Linguistics.

Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Ken-taro Inui, Satoshi Sekine, Lasha Abzianidze, and Jo-han Bos. 2019a. Can neural networks understandmonotonicity reasoning? In Proceedings of the2019 ACL Workshop BlackboxNLP: Analyzing andInterpreting Neural Networks for NLP, pages 31–40,Florence, Italy. Association for Computational Lin-guistics.

Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Ken-taro Inui, Satoshi Sekine, Lasha Abzianidze, and Jo-han Bos. 2019b. HELP: A dataset for identifyingshortcomings of neural models in monotonicity rea-soning. In Proceedings of the Eighth Joint Con-ference on Lexical and Computational Semantics(*SEM 2019), pages 250–255, Minneapolis, Min-nesota. Association for Computational Linguistics.

Annie Zaenen, Lauri Karttunen, and Richard Crouch.2005. Local textual inference: Can it be defined orcircumscribed? In Proceedings of the ACL Work-shop on Empirical Modeling of Semantic Equiv-alence and Entailment, pages 31–36, Ann Arbor,Michigan. Association for Computational Linguis-tics.

Rowan Zellers, Yonatan Bisk, Roy Schwartz, andYejin Choi. 2018. SWAG: A large-scale adversar-ial dataset for grounded commonsense inference.In Proceedings of the 2018 Conference on Empiri-cal Methods in Natural Language Processing, pages93–104. Association for Computational Linguistics.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, AliFarhadi, and Yejin Choi. 2019. HellaSwag: Can amachine really finish your sentence? In Proceed-ings of the 57th Annual Meeting of the Associationfor Computational Linguistics, pages 4791–4800,Florence, Italy. Association for Computational Lin-guistics.

Sheng Zhang, Rachel Rudinger, Kevin Duh, and Ben-jamin Van Durme. 2017. Ordinal common-sense in-ference. Transactions of the Association for Com-putational Linguistics, 5:379–395.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut-dinov, Raquel Urtasun, Antonio Torralba, and SanjaFidler. 2015. Aligning books and movies: Towardsstory-like visual explanations by watching moviesand reading books. In Proceedings of the IEEEinternational conference on computer vision, pages19–27.

Appendix

Type Premise Hypothesis

Connectives These cats or those fish appear. These cats and those fish don’t both appear.Determiners Some skateboards tipped over. Not all skateboards tipped over.Numerals Ten bananas were scorching. One hundred bananas weren’t scorching.Modals Jerry could wake up. Jerry didn’t need to wake up.Scalar adjectives Banks are fine. Banks are not great.Scalar verbs Dawn went towards the hills. Dawn did not get to the hills.

Table 4: The scalar implicature triggers in IMPPRES. Examples are automatically generated sentences pairs fromeach of the six datasets for the scalar implicatures experiment. The pairs belong to the “Implicature (+ to −)”condition.

Type Premise (Trigger) Hypothesis (Presupposition)

All N All six roses that bloomed died. Exactly six roses bloomed.Both Both flowers that bloomed died. Exactly two flowers bloomed.Change of State The cat escaped. The cat used to be captive.Cleft Existence It is Sandra who disliked Veronica. Someone disliked Veronica.Cleft Uniqueness It is Sandra who disliked Veronica. Exactly one person disliked Veronica.Only Only Lucille went to Spain. Lucille went to Spain.Possessed Definites Bill’s handyman won. Bill has a handyman.Question Sue learned why Candice testified. Candice testified.

Table 5: The presupposition triggers in IMPPRES. Examples are automatically generated sentences pairs from eachof the eight datasets for the presupposition experiment. The pairs belong to the “Plain Trigger / Presupposition”condition.

Figure 8: Results for the scalar implicatures triggered by adjectives, by target condition.

Figure 9: Results for the scalar implicatures triggered by adjectives, by target condition.

Figure 10: Results for the scalar implicatures triggeredby determiners, by target condition.

Figure 11: Results for the scalar implicatures triggeredby modals, by target condition.

Figure 12: Results for the scalar triggered by numerals,by target condition.

Figure 13: Results for the scalar implicatures triggeredby verbs, by target condition.