Natural Logic and Natural Language Inference Bill MacCartney Stanford University / Google, Inc. 8...

Natural Logic and Natural Language Natural Logic and Natural Language InferenceInference

Bill MacCartneyStanford University / Google, Inc.

8 April 2011

2

Two disclaimersTwo disclaimers

• The work I present today isn’t exactly fresh• Essentially, it’s my dissertation work from 2009• I hope it can usefully provide context for more recent

work

• I’m a computer scientist, not a semanticist or a logician• Consequently, I emphasize pragmatism over rigor

Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion

3

Natural language inference (NLI)Natural language inference (NLI)

• Aka recognizing textual ‘entailment’ (RTE)

• Does premise P justify an inference to hypothesis H?• An informal, intuitive notion of inference: not strict logic• Emphasis on variability of linguistic expression

• Necessary to goal of natural language understanding (NLU)

• Can also enable semantic search, question answering, …

P Every firm polled saw costs grow more than expected,even after adjusting for inflation.

H Every big company in the poll reported cost increases.yes

Some

Some no


4

NLI: a spectrum of approachesNLI: a spectrum of approaches

lexical/semanticoverlap

Jijkoun & de Rijke 2005

patternedrelation

extraction

Romano et al. 2006

semanticgraph

matching

MacCartney et al. 2006Hickl et al. 2006

FOL &theoremproving

Bos & Markert 2006

robust,but shallow

deep,but brittle

naturallogic

(this work)

Problem:imprecise easily confounded by negation, quantifiers, conditionals, factive & implicative verbs, etc.

Problem:hard to translate NL to FOLidioms, anaphora, ellipsis, intensionality, tense, aspect, vagueness, modals, indexicals, reciprocals, propositional attitudes, scope ambiguities, anaphoric adjectives, non-intersective adjectives, temporal & causal relations, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, comparatives, phrasal verbs, …

Solution?


5

What is natural logic?What is natural logic? ( ( natural deduction) natural deduction)

• Characterizes valid patterns of inference via surface forms• precise, yet sidesteps difficulties of translating to FOL

• A long history• traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, …• the term “natural logic” was introduced by Lakoff (1970)• van Benthem & Sánchez Valencia (1986-91): monotonicity

calculus• Nairn et al. (2006): an account of implicatives & factives

• We introduce a new theory of natural logic• extends monotonicity calculus to account for negation &

exclusion• incorporates elements of Nairn et al.’s model of implicatives

• …and implement & evaluate a computational model of it


6

‘‘Entailment’ relations in past Entailment’ relations in past workwork

X is a man

X is a woman

X is a hippo

X is hungry

X is a fish

X is a carp

X is a crow

X is a bird

X is a couch

X is a sofa

Yesentailment

Nonon-entailment

2-wayRTE1,2,3

Yesentailment

Nocontradiction

Unknowncompatibility

3-wayFraCaS,

PARC, RTE4

P = Qequivalence

P < Qforward

entailment

P > Qreverse

entailment

P # Qnon-entailment

containmentSánchez-Valencia


7

16 elementary set relations16 elementary set relations

? ?

? ?

y

x

x

y

Assign sets x, y to one of 16 relations, depending on emptiness or non-emptiness of each of four partitions


empty

non-empty

8

16 elementary set relations16 elementary set relations

x ^ y x ‿ y

x y x ⊐ y

x ⊏ y x | y x # y

But 9 of 16 are degenerate: either x or y is either empty or universal.

I.e., they correspond to semantically vacuous expressions, which are rare outside logic textbooks.

We therefore focus on the remaining seven relations.


9

The set of 7 basic entailment The set of 7 basic entailment relationsrelations

Venn symbol

name example

x y equivalence couch sofa

x ⊏ y forward entailment(strict)

crow ⊏ bird

x ⊐ y reverse entailment(strict)

European ⊐ French

x ^ y negation(exhaustive exclusion)

human ^ nonhuman

x | y alternation(non-exhaustive exclusion)

cat | dog

x ‿ y cover(exhaustive non-exclusion)

animal ‿ nonhuman

x # y independence hungry # hippo

Relations are defined for all semantic types: tiny ⊏ small, hover ⊏ fly, kick ⊏ strike,this morning ⊏ today, in Beijing ⊏ in China, everyone ⊏ someone, all ⊏ most ⊏ some


10

|

x R y

Joining entailment relationsJoining entailment relations

fish human nonhuman^

y zS

?

?

⋈

⊏ ⋈ ⊏ ⊏

⊐ ⋈ ⊐ ⊐

^ ⋈ ^

R ⋈ R

⋈ R R

⊏


11

Some joins yield unions of Some joins yield unions of relations!relations!

x | y y | z x ? z

couch | table table | sofa couch sofa

pistol | knife knife | gun pistol ⊏ gun

dog | cat cat | terrier dog ⊐ terrier

rose | orchid orchid | daisy rose | daisy

woman | frog frog | Eskimo woman # Eskimo

What is | | ?⋈

| | {, ⊏, ⊐, |, #}⋈


12

Of 49 join pairs, 32 yield relations in ; 17 yield unions

Larger unions convey less information — limits power of inference

In practice, any union which contains # can be approximated by #

The complete join tableThe complete join table


13

will depend on:• the lexical entailment relation generated by e:

(e)• other properties of the context x in which e is

applied

( , )

Lexical entailment relationsLexical entailment relations

x e(x)

compound expression

atomic edit: DEL, INS, SUB

entailment relation

Example: suppose x is red car

If e is SUB(car, convertible), then (e) is ⊐If e is DEL(red), then (e) is ⊏

Crucially, (e) depends solely on lexical items in e, independent of context x

But how are lexical entailment relations determined?


14

Lexical entailment relations: Lexical entailment relations: SUBsSUBs

(SUB(x, y)) = (x, y)

For open-class terms, use lexical resource (e.g. WordNet)for synonyms: sofa couch, forbid prohibit

⊏for hypo-/hypernyms: crow ⊏ bird, frigid ⊏ cold, soar ⊏ rise

| for antonyms and coordinate terms: hot | cold, cat | dog

or | for proper nouns: USA United States, JFK | FDR

# for most other pairs: hungry # hippo

Closed-class terms may require special handlingQuantifiers: all ⊏ some, some ^ no, no | all, at least 4 ‿ at most

6

See paper for discussion of pronouns, prepositions, …


15

Lexical entailment relations: DEL Lexical entailment relations: DEL & INS& INS

Generic (default) case: (DEL(•)) = ⊏, (INS(•)) = ⊐• Examples: red car ⊏ car, sing ⊐ sing off-key• Even quite long phrases: car parked outside since last week ⊏ car• Applies to intersective modifiers, conjuncts, independent

clauses, …• This heuristic underlies most approaches to RTE!• Does P subsume H? Deletions OK; insertions penalized.

Special cases• Negation: didn’t sleep ^ did sleep• Implicatives & factives (e.g. refuse to, admit that): discussed

later• Non-intersective adjectives: former spy | spy, alleged spy # spy• Auxiliaries etc.: is sleeping sleeps, did sleep slept


16

The impact of semantic The impact of semantic compositioncomposition

How are entailment relations affected by semantic composition?

f

@

f

@

x y

?

The monotonicity calculus provides a partial answer UP

⊏ ⊏⊐ ⊐# #

DOWN ⊏ ⊐⊐ ⊏# #

NON ⊏ #⊐ ## #

If f has monotonicity…

How is (x, y) projected by f?

But how are other relations (|, ^, ‿) projected?


@ means fn application[ ]

negation ⊏ ⊐⊐ ⊏^ ^| ‿‿ |# #

not happy not glad

isn’t swimming # isn’t hungry

17

A typology of projectivityA typology of projectivityProjectivity signatures: a generalization of monotonicity classes

not French ‿ not Germannot more than 4 | not less than 6

not human ^ not nonhuman

didn’t kiss ⊐ didn’t touchnot ill ⊏ not seasick

In principle, 77 possible signatures, but few actually realized

↦Each projectivity signature is a map


intersectivemodification

⊏ ⊏⊐ ⊐^ || |‿ ## #

18

A typology of projectivityA typology of projectivityProjectivity signatures: a generalization of monotonicity

classesEach projectivity signature is a mapIn principle, 77 possible signatures, but few actually realized

↦

negation

⊏ ⊐⊐ ⊏^ ^| ‿‿ |# #

metallic pipe # nonferrous pipe

live human | live nonhumanFrench wine | Spanish wine

See my disseration for projectivity of various quantifiers, verbs


19

Projecting through multiple Projecting through multiple levelslevels

⊏

⊏

⊐

⊐

⊐

a shirtnobody can without enter

@

@

@

@

clothesnobody can without enter

@

@

@

@

Propagate entailment relation between atoms upward, according to projectivity class of each node on path to root

nobody can enter without a shirt ⊏ nobody can enter without clothes


20

Implicatives & factives Implicatives & factives [Nairn et al. 06][Nairn et al. 06]

signature

example

implicatives

+ / – he managed to escape

+ / o he was forced to sell

o / – he was permitted to live

implicatives

– / + he forgot to pay

– / o he refused to fight

o / + he hesitated to ask

factives + / + he admitted that he knew

– / – he pretended he was sick

o / o he wanted to fly

9 signatures, per implications (+, –, or o) in positive and negative contexts


21

Implicatives & factivesImplicatives & factives

signature

example(DEL

)(INS)

implicatives

+ / – he managed to escape he escaped

+ / o he was forced to sell ⊏ he sold ⊏ ⊐

o / – he was permitted to live ⊐ he lived ⊐ ⊏

implicatives

– / + he forgot to pay ^ he paid ^ ^

– / o he refused to fight | he fought | |

o / + he hesitated to ask ‿ he asked ‿ ‿

factives + / + he admitted that he knew ⊏ he knew ⊏ ⊐

– / – he pretended he was sick | he was sick | |

o / o he wanted to fly # he flew # #

We can specify relation generated by DEL or INS of each signature


Room for variation w.r.t. infinitives, complementizers, passivation, etc.Some more intuitive when negated: he didn’t hesitate to ask | he didn’t askFactives not fully explained: he didn’t admit that he knew | he didn’t know

22

Putting it all togetherPutting it all together

1. Find a sequence of edits e1, …, en which transforms p into h. Define x0 = p, xn = h, and xi = ei(xi–1) for i [1, n].

2. For each atomic edit ei:

1. Determine the lexical entailment relation (ei).

2. Project (ei) upward through the semantic composition tree of expression xi–1 to find the atomic entailment relation (xi–1, xi)

3. Join atomic entailment relations across the sequence of edits:(p, h) = (x0, xn) = (x0, x1) ⋈ … ⋈ (xi–1, xi) ⋈ … ⋈ (xn–1, xn)

Limitations: need to find appropriate edit sequence connecting p and h;tendency of ⋈ operation toward less-informative entailment relations; lack of general mechanism for combining multiple premises

Less deductive power than FOL. Can’t handle e.g. de Morgan’s Laws.


23

An exampleAn example

P The doctor didn’t hesitate to recommend Prozac.

H The doctor recommended medication.yes

i ei xi lex atom join

The doctor didn’t hesitate to recommend Prozac.

1 DEL(hesitate to)The doctor didn’t recommend Prozac.

2 DEL(didn’t)The doctor recommended Prozac.

3 SUB(Prozac, medication)The doctor recommended medication.


‿ ||

^^ ⊏

⊏ ⊏ ⊏ yes

24

Different edit orders?Different edit orders?i ei lex atom join

1 DEL(hesitate to) ‿ | |

2 DEL(didn’t) ^ ^ ⊏

3 SUB(Prozac, medication) ⊏ ⊏ ⊏

i ei lex atom join

1 DEL(didn’t) ^ ^ ^

2 DEL(hesitate to) ‿ ‿ ⊏


i ei lex atom join




i ei lex atom join


2 SUB(Prozac, medication) ⊏ ⊐ |


i ei lex atom join

1 DEL(didn’t) ^ ^ ^

2 SUB(Prozac, medication) ⊏ ⊐ |


i ei lex atom join


2 DEL(didn’t) ^ ^ |


Intermediate steps may vary; final result is typically (though not necessarily) the same


25

Implementation & evaluationImplementation & evaluation

The NatLog system: an implementation of this model in codeFor implementation details, see [MacCartney & Manning 2008]

Evaluation on FraCaS test suite183 NLI problems, nine sections, three-way classificationAccuracy 70% overall; 87% on “relevant” sections (60% coverage)Precision 89% overall: rarely predicts entailment wrongly

Evaluation on RTE3 test suiteLonger, more natural premises; greater diversity of inference typesNatLog alone has mediocre accuracy (59%) but good precisionHybridization with broad-coverage RTE system yields gains of 4%


Natural logic is not a universal solution for NLIMany types of inference not amenable to natural logic approachOur inference method faces many limitations on deductive

power

More work to be done in fleshing out our accountEstablishing projectivity signatures for more quantifiers, verbs,

etc.Better incorporating presuppositions

But, our model of natural logic fills an important nichePrecise reasoning on negation, antonymy, quantifiers,

implicatives, …Sidesteps the myriad difficulties of full semantic interpretationPractical value demonstrated on FraCaS and RTE3 test suites

26

ConclusionsConclusions


:-) Thanks! Questions?

27

Backup slides followBackup slides follow


28

An example involving exclusionAn example involving exclusion

P Stimpy is a cat.

H Stimpy is not a poodle. yes


Stimpy is a cat.

1 SUB(cat, dog)Stimpy is a dog.

2 INS(not)Stimpy is not a dog.

3 SUB(dog, poodle)Stimpy is not a poodle.


| ||

^^ ⊏

⊐ ⊏ ⊏ yes

29

An example involving an An example involving an implicativeimplicative

P We were not permitted to smoke.

H We smoked Cuban cigars. no


We were not permitted to smoke.

1 DEL(permitted to)We did not smoke.

2 DEL(not)We smoked.

3 INS(Cuban cigars)We smoked Cuban cigars.


⊐ ⊏⊏

^^ |

⊐ ⊐ | no

30

de Morgan’s Laws for quantifiersde Morgan’s Laws for quantifiers

P Not all birds fly.

H Some birds do not fly. yes


Not all birds fly.

1 DEL(not)All birds fly.

2 SUB(all, some)Some birds fly.

3 INS(not)Some birds do not fly.


^ ^^

⊏⊏ ‿

^ ‿ ⊏⊐‿#wtf??

31

de Morgan’s Laws for quantifiers de Morgan’s Laws for quantifiers (2)(2)

P Not all birds fly.

H Some birds do not fly. yes


Not all birds fly.

1 DEL(not)All birds fly.

2 INS(not)All birds do not fly.

3 SUB(all, some)Some birds do not fly.


^ ^^

|^ ⊐

⊏ ⊏ ⊏⊐‿#wtf??

32

A more complex exampleA more complex example

P Jimmy Dean refused to move without blue jeans.

H James Dean didn’t dance without pants. yes

i ei lex atom join

1 SUB(Jimmy Dean, James Dean)

2 DEL(refuse to) | | |

3 INS(did) |

4 INS(n’t) ^ ^ ⊏

5 SUB(move, dance) ⊐ ⊏ ⊏

6 DEL(blue) ⊏ ⊏ ⊏

7 SUB(jeans, pants) ⊏ ⊏ ⊏


33

A more complex example (2)A more complex example (2)



i ei lex atom join

1 INS(did)

2 INS(n’t) ^ ^ ^

3 DEL(blue) ⊏ ⊐ |

4 SUB(jeans, pants) ⊏ ⊐ |

5 SUB(move, dance) ⊐ ⊐ |

6 DEL(refuse to) | ‿ ⊏

7 SUB(Jimmy Dean, James Dean) ⊏


34

A more complex example (3)A more complex example (3)



i ei lex atom join

1 INS(did)

2 INS(n’t) ^ | |

6 DEL(refuse to) | | ⊏⊐|#

3 DEL(blue) ⊏ ⊏ •

4 SUB(jeans, pants) ⊏ ⊏ •

5 SUB(move, dance) ⊐ ⊐ •

7 SUB(Jimmy Dean, James Dean) •


Natural Logic and Natural Language Inference Bill MacCartney Stanford University / Google, Inc. 8...

Documents

Transcript of Natural Logic and Natural Language Inference Bill MacCartney Stanford University / Google, Inc. 8...