A Computer Aided System to Identify the Face Shape of a person
Executable Semantic Parsing - Tel Aviv Universityjoberant/teaching/class1_index.pdf · SHRDLU...
Transcript of Executable Semantic Parsing - Tel Aviv Universityjoberant/teaching/class1_index.pdf · SHRDLU...
Executable Semantic Parsing
ESSLLI
Aug 22, 2016
Instructor: Jonathan Berant
Oct 2014
1
1B dollars please
2
Why is this an important language problem?
what is the second biggest city in california that is not near a river?
3
Why is this an important language problem?
what is the second biggest city in california that is not near a river?
• You can not memorize this
3
Why is this an important language problem?
what is the second biggest city in california that is not near a river?
• You can not memorize this
• Understand language atoms
• Understand how to compose them
• Generalize to any utterance
• Collect evidence from multiple sources
3
Executable semantic parsing
Which states’ capitals are also their largest cities by area?
4
Executable semantic parsing
Which states’ capitals are also their largest cities by area?
semantic parsing
µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)
4
Executable semantic parsing
Which states’ capitals are also their largest cities by area?
semantic parsing
µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)
execute logical form
Arizona,Hawaii,Idaho,Indiana,Iowa,Oklahoma,Utah
4
Executable semantic parsing
Which states’ capitals are also their largest cities by area?
semantic parsing
µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)
execute logical form
Arizona,Hawaii,Idaho,Indiana,Iowa,Oklahoma,Utah
Execute query against knowledge-base
4
Executable semantic parsing
Send a reminder to all students who haven’t turned in their homework.
5
Executable semantic parsing
Send a reminder to all students who haven’t turned in their homework.
semantic parsing
∀x ∈ (Student u ¬Submit.Homework) : Remind(x)
5
Executable semantic parsing
Send a reminder to all students who haven’t turned in their homework.
semantic parsing
∀x ∈ (Student u ¬Submit.Homework) : Remind(x)
execute logical form
[50 emails sent]
5
Executable semantic parsing
Send a reminder to all students who haven’t turned in their homework.
semantic parsing
∀x ∈ (Student u ¬Submit.Homework) : Remind(x)
execute logical form
[50 emails sent]
Execute command against personal assistant
5
Executable semantic parsing
[utterance: user input]
semantic parsing
[program]
execute logical form
[behavior: user output]
6
Executable semantic parsing
[utterance: user input]
semantic parsing
[program]
execute logical form
[behavior: user output]
Execution connects language with the world
6
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
7
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
8
Personal assistants everywhere!
Show me photos from Utah last august
Find videos I took at Ive’s birthday party
What movies are playing today?
. . .
9
10
Personal assistants everywhere!
Alexa, Play some Christmas music
Alexa, turn the bedroom lights on
Alexa, what’s my commute?
. . .
11
Personal assistants everywhere!
Can you help me order flowers for my mom’s birthday
Where’s the best place to go hiking in the bay area
Is there a dog friendly beach nearby?
Remind me to congratulate Tanya the next time she calls
. . .
12
Personal assistant everywhere!
13
Question answering
14
Robot control
15
Research assistants
Which country has the highest CO2 emissions?
Which had the highest increase since last year?
What fraction is from the five countries with highest GDP?
16
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
17
semantic
parsing
language logic
learning
18
semantic
parsing
language logic
learning
But executable!
18
Non-executable semantic parsing
The boy wants to go
[Banarescu et al., 2013]
19
Non-executable semantic parsing
The boy wants to go
No environment to execute semantic representation
[Banarescu et al., 2013]
19
Semantics
Sentiment analysis, Semantic Role Labeling, Named Entity Recognition, ...
20
Semantics
Sentiment analysis, Semantic Role Labeling, Named Entity Recognition, ...
Semantic parsing: Full meaning representation
20
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
21
SHRDLU [1971]
Person: Pick up a big red block.
Computer: OK.
Person: Grasp the pyramid.
Computer: I don’t understand which pyramid you mean.
Person (changing their mind): Find a block which is taller than the oneyou are holding and put it into the box.
Computer: By ”it”, I assume you mean the block which is taller thanthe one I am holding.
Computer: OK.
22
SHRDLU [1971]
Person: Pick up a big red block.
Computer: OK.
Person: Grasp the pyramid.
Computer: I don’t understand which pyramid you mean.
Person (changing their mind): Find a block which is taller than the oneyou are holding and put it into the box.
Computer: By ”it”, I assume you mean the block which is taller thanthe one I am holding.
Computer: OK.
Person: What does the box contain?
Computer: The blue pyramid and the blue block.
Person: What is the pyramid supported by?
Computer: The box.
Person: How many blocks are not in the box?
Computer: Four of them.
Person: Is at least one of them narrower than the one which I told youto pick up?
Computer: Yes, the red cube.22
Other grammar-based approaches
Lunar (1972):
List the rocks that contain chromite and ulvospinel
What minerals have been identified in the lunar samples?
What is the average analysis of Ir in rock S10055?
Give me the K / Rb ratios for all lunar samples
Chat-80 (1982):
Does Afghanistan border China?
How many countries does the Danube flow through?
What is the capital of each country bordering the Baltic?
Which country is bordered by two seas?
Woods et al., 1972; Warren and Pereira, 1982
23
Other grammar-based approaches
Lunar (1972):
List the rocks that contain chromite and ulvospinel
What minerals have been identified in the lunar samples?
What is the average analysis of Ir in rock S10055?
Give me the K / Rb ratios for all lunar samples
Chat-80 (1982):
Does Afghanistan border China?
How many countries does the Danube flow through?
What is the capital of each country bordering the Baltic?
Which country is bordered by two seas?
Problem: does not scale!
Woods et al., 1972; Warren and Pereira, 1982
23
Statistical semantic parsing
Learn from data:
Input:
What’s California’s capital? Capital(California)
How long is the Mississippi river? RiverLength(Mississippi)
... ...
[Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; ...]
24
Statistical semantic parsing
Learn from data:
Input:
What’s California’s capital? Capital(California)
How long is the Mississippi river? RiverLength(Mississippi)
... ...
Output:
countries in asia ⇒ Type.Country u ContainedBy.Asia
[Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; ...]
24
Statistical semantic parsing
Learn from data:
Input:
What’s California’s capital? Capital(California)
How long is the Mississippi river? RiverLength(Mississippi)
... ...
Output:
countries in asia ⇒ Type.Country u ContainedBy.Asia
What are the problems with this form of supervision?
[Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; ...]
24
Breadth
Depth
information retrieval
semantic parsing
25
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
26
Goals
Be able to go and build a semantic parser
Learn about modern techniques and challenges in semantic parsing
27
Plan
Class 1: Representation
• Lambda calculus
• Lambda DCS
28
Plan
Class 1: Representation
• Lambda calculus
• Lambda DCS
Class 2: Compositionality
• Grammars
• Lexicons
• Derivations
• Modelling
28
Plan
Class 3: Learning
• Traning from logical forms
• Traning from denotations
• Additional training signals
29
Plan
Class 3: Learning
• Traning from logical forms
• Traning from denotations
• Additional training signals
Class 4: Search
• CKY
• Beam-search
• Agenda-based parsing
29
Plan
Class 5: Advanced topics
• Intermediate representations
• Building semantic parsers overnight
• Deep learning
30
About me
31
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
32
Expressing language with logical form
How to handle various linguistic phenomena
• entities, relations
• Multi-arg events
• Comparatives, superlatives
• Quantifiers, determiners
Goal: Intuition for how to write down logical forms
33
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
34
Worlds
35
Worlds
µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)
∀x ∈ (Student u ¬Submit.Homework) : Remind(x)
35
Worlds
µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)
∀x ∈ (Student u ¬Submit.Homework) : Remind(x)
Which states’ capitals are also their largest cities by area?
Send a reminder to students that did not submit homework
35
Worlds
µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)
∀x ∈ (Student u ¬Submit.Homework) : Remind(x)
Which states’ capitals are also their largest cities by area?
Send a reminder to students that did not submit homework
World determines relevant language
35
Worlds
• Query a DB
Mountains in Italy
36
Worlds
• Query a DB
Mountains in Italy
• Update a DB
Add a meeting from 8am-10am Monday to Friday
36
Worlds
• Query a DB
Mountains in Italy
• Update a DB
Add a meeting from 8am-10am Monday to Friday
• Execute in an environment
Bring me the cup on the table
36
Querying a DB
Type
BarackObama Person
MichelleObama Person
Honolulu CityTown
Hawaii USState
ContainedBy
Honolulu Hawaii
Honolulu UnitedStates
Chicago Illinois
Chicago UnitedStates
Profession
BarackObama Politician
BarackObama Author
MichelleObama Lawyer
PlaceOfBirth
BarackObama Honolulu
MichelleObama Chicago
Marriage
BarackObama Event8
MichelleObama Event8
GreaterThan
2 1
8 3
... ...
startDate
Event8 1992.10.03
Event13 2001.13.02
37
Graph DB
BarackObama
Person
Type
Politician
Profession
1961.08.04
DateOfBirth
HonoluluPlaceOfBirth
Hawaii
ContainedBy
City
Type
UnitedStates
ContainedBy
USState
Type
Event8
Marriage
MichelleObama
Spouse
Type
FemaleGender
1992.10.03
StartDate
Event3
PlacesLived
Chicago
Location
Event21
PlacesLived
Location
ContainedBy
Subset of Freebase (2B facts)
38
Representations
Simply-typed lambda calculus (Carpenter, 1998)
• General formalism for natural language semantics
Lambda-DCS (Liang, 2013)
• Natural language interface for databases
39
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
40
Simply-typed lambda calculus
• Constants
• Variables
• Lambda terms
• Function application
adapted from Yoav Artzi
41
Simply-typed lambda calculus
Each type has a set of constants and variables
Constants
Chicagoe
Event8ev
City〈e,t〉
PlaceOfBirthOf〈e〈e,t〉〉
42
Simply-typed lambda calculus
Each type has a set of constants and variables
Constants
Chicagoe
Event8ev
City〈e,t〉
PlaceOfBirthOf〈e〈e,t〉〉
Variables
xe, ye
aev, bev
f〈e〈e,t〉〉, g〈e〈e,t〉〉
42
Simply-typed lambda calculus
Function application
t: (PlaceOfBirthOf(BarackObama))Chicago)
t: City(Honolulu)
43
Simply-typed lambda calculus
Function application
t: (PlaceOfBirthOf(BarackObama))Chicago)
t: City(Honolulu)
Lambda terms
〈e, t〉: λx.President〈e,t〉(xe)
〈e, t〉: λx.President(x) ∧ Lawyer(x)
〈e〈e, t〉〉: λx.λyPlaceOfBirthOf(x, y)
Logical operators are defined as higher-order constants
43
Simply-typed lambda calculus
Function application and beta reduction
t : λx.City(x)(Honolulu)
beta
t : City(Honolulu)
44
Simply-typed lambda calculus
Function application and beta reduction
t : λx.City(x)(Honolulu)
beta
t : City(Honolulu)
t : λx.(λy.PlaceOfBirthOf(x, y)(BarackObama))(Chicago)
beta
t : PlaceOfBirthOf(BarackObama,Chicago)
44
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
45
Lambda-DCS
• Can be viewed as syntactic sugar for lambda-calculus
• Concise logical forms that mirror natural langauge
46
Logical forms: entity
proper nouns
lambda-dcs Chicago
lambda calculus λx.x = Chicago
denotation {Chicago}graph template Chicago
[Liang, 2013]
47
Logical forms: join
Verbs and prepositions
HasBirthplace.Chicago
λx.HasBirthplace(x,Chicago)
{MichelleObama}o ChicagoHasBirthplace
48
Logical forms: join
Verbs and prepositions
HasBirthplace.Chicago
λx.HasBirthplace(x,Chicago)
{MichelleObama}o ChicagoHasBirthplace
ChildOf.HasBirthplace.Chicago
λx.∃yChildOf(x, y) ∧HasBirthplace(y,Chicago)
{NatashaObama, MaliaObama}o yChildOf ChicagoHasBirthplace
48
Logical forms: join
Verbs and prepositions
HasBirthplace.Chicago
λx.HasBirthplace(x,Chicago)
{MichelleObama}o ChicagoHasBirthplace
ChildOf.HasBirthplace.Chicago
λx.∃yChildOf(x, y) ∧HasBirthplace(y,Chicago)
{NatashaObama, MaliaObama}o yChildOf ChicagoHasBirthplace
BirthDateOf.ChildOf.HasBirthplace.Chicago
λx.∃y∃zBirthDateOf(x, y) ∧ChildOf(y, z) ∧HasBirthplace(z,Chicago)
{06.10.2001, 07.04.1998}o yBirthdateOf zChildOf ChicagoHasBirthplace
48
Logical forms: intersection, union, negation
Profession.Lawyer uHasBirthplace.Chicago
λx.Profession(x,Lawyer) ∧HasBirthplace(x,Chicago)
{MichelleObama}o
Lawyer
Profession
Chicago
HasBirthplace
49
Logical forms: intersection, union, negation
Profession.Lawyer uHasBirthplace.Chicago
λx.Profession(x,Lawyer) ∧HasBirthplace(x,Chicago)
{MichelleObama}o
Lawyer
Profession
Chicago
HasBirthplace
Profession.Lawyer tHasBirthplace.Chicago
λx.Profession(x,Lawyer) ∨HasBirthplace(x,Chicago)
{MichelleObama,BarackObama}o
Lawyer
Profession
o
Chicago
HasBirthplace
49
Logical forms: intersection, union, negation
Profession.Lawyer uHasBirthplace.Chicago
λx.Profession(x,Lawyer) ∧HasBirthplace(x,Chicago)
{MichelleObama}o
Lawyer
Profession
Chicago
HasBirthplace
Profession.Lawyer tHasBirthplace.Chicago
λx.Profession(x,Lawyer) ∨HasBirthplace(x,Chicago)
{MichelleObama,BarackObama}o
Lawyer
Profession
o
Chicago
HasBirthplace
¬Profession.Lawyer
λx.¬Profession(x,Lawyer)
{MaliaObama,NatashaObama}49
Logical forms: n-ary relations
Where did Barack Obama live in 1985?
Where did Barack Obama live?
50
Logical forms: n-ary relations
Where did Barack Obama live in 1985?
Where did Barack Obama live?
Do we need a ternary predicate?
λx.PlaceLivedBy(x,BarackObama)
λx.PlaceLivedBy(x,BarackObama, 1985)
50
Logical forms: n-ary relations
Where did Barack Obama live in 1985?
Where did Barack Obama live?
Do we need a ternary predicate?
λx.PlaceLivedBy(x,BarackObama)
λx.PlaceLivedBy(x,BarackObama, 1985)
Neo-Davidsonian semantics: Event node modified by several arguments
o Event8LivedLocation BarackObamaPersonLiving
1985
StartLiving
LivedLocation.(PersonLiving.BarackObama u StartLiving.1985)
λx.∃e.LivedLocation(x, e) ∧PersonLiving(e,BarackObama) ∧ StartLiving(e, 1985)
{Chicago} 50
Logical forms: higher order functions
Operate on sets of entities (not entities)
count(Type.USState)
count(λx.Type(x,USState))
{50}
Similar for min, max, sum
51
Logical forms: higher order functions
Operate on sets of entities (not entities)
count(Type.USState)
count(λx.Type(x,USState))
{50}
Similar for min, max, sum
argmax(Type.USState,Area)
argmax(λx.Type(x,USState), λx.λy.Area(x, y))
{Alaska}
Similar for argmin
51
Logical forms: anaphora
those whose children influenced Barack Obama
HasChild.HasInfluence.BarackObama
λx.∃y.HasChild(x, y) ∧HasInfluence(y,BarackObama)
52
Logical forms: anaphora
those whose children influenced Barack Obama
HasChild.HasInfluence.BarackObama
λx.∃y.HasChild(x, y) ∧HasInfluence(y,BarackObama)
those whose child influenced them
µx.HasChild.HasInfluence.x
λx.∃y.HasChild(x, y) ∧HasInfluence(y, x)
52
Logical forms: lambda abstraction
How to construct compositional binary relations?
53
Logical forms: lambda abstraction
How to construct compositional binary relations?
Person who has the most children
argmax(Type.Person,Rev[λx.count(Rev[HasChildren].x)])
argmax(λx.Type(x,Person), λx.count(λy.HasChildren(x, y)))
53
Logical forms: lambda abstraction
How to construct compositional binary relations?
Person who has the most children
argmax(Type.Person,Rev[λx.count(Rev[HasChildren].x)])
argmax(λx.Type(x,Person), λx.count(λy.HasChildren(x, y)))
In some cases lambda calculus is more transparent
53
Some things we skip
Determiners and quantification
54
Dynamic environments
Database updates
Add a meeting with Tom from 2 to 4pm
Add(Type.Meeting uAttendee.Tom u StartTime.1400 uEndTime.1600)
[Artzi and Zettlemoyer, 2013]
55
Dynamic environments
Database updates
Add a meeting with Tom from 2 to 4pm
Add(Type.Meeting uAttendee.Tom u StartTime.1400 uEndTime.1600)
Executing instruction
[Artzi and Zettlemoyer, 2013]
55
Outline
• Introduction (30 min.)
– Applications
– Background
– Short history
– Goals and plan
• Representation (30 min.)
– Worlds
– Simply-typed lambda calculus
– Lambda-DCS
• Exercise (30 min.)
56
Exercise
http://esslli2016.unibz.it/?page id=356
or
cd /usr/local/sempre
./run @mode=simple-lambdadcs
(loadgraph esslli 2016/geo880.kg)
57
Exercise
a.b (a b)
a u b (and a b)
rank-m top-n for unary and binary (argmax m n u b)
count (count u)
numbers (number n)
µx.u (mark x u)
Rev(u) (rev u)
Type type.object.type
USState location.us state
City location.citytown
MajorCity location.major citytown
Area location.location.area
Size location.location.size
Population location.location.population
Capital location.us state.capital
Border location.us state.border
Traverse location.river.traverse
58
Exercise
Write logical forms for the following utterances:
state with the largest area
Top 5 cities by area
Major cities with at least 140,000 inhabitants
How many states have capitals with at least 200,000 inhabitants
State bordering Oregon and Washington
Second highest mountain in California
Which states’ capitals are also their largest cities by population
State that is traversed by the most rivers
59
Solution
(argmax 1 1 (type.object.type location.us state) location.location.area)
(argmax 1 5 (type.object.type location.citytown) location.location.size)
(and (type.object.type location.major citytown) (loca-tion.location.population (> (number 140000))))
(count (location.us state.capital (and (type.object.type location.capital)(location.location.population (> (number 200000))))))
(and (type.object.type location.us state) (and (location.us state.borderstate.oregon) (location.us state.border state.washington)))
(argmax 2 1 (and (type.object.type location.mountain) (loca-tion.location.containedby state.california)) location.location.elevation)
(and (type.object.type location.us state) (mark x (loca-tion.us state.capital (argmax 1 1 (location.location.containedby(var x)) location.location.population))))
(argmax 1 1 (type.object.type location.us state) (reverse (lambda x(count (location.river.traverse (var x))))))
60
Summary
• Key technology for conversational interfaces
61
Summary
• Key technology for conversational interfaces
• Expressing language constructs with logical forms
61
Summary
• Key technology for conversational interfaces
• Expressing language constructs with logical forms
• But how do we construct logical forms automatically?
61