Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single...
Transcript of Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single...
![Page 1: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/1.jpg)
M I T - I B MW A T S O NA I L A B
Neurosymbolic AI—David Cox, Ph.D.IBM Director, MIT-IBM Watson AI LabIBM Research
![Page 2: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/2.jpg)
“Artificial Intelligence”
![Page 3: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/3.jpg)
Narrow AIEmerging
Broad AIDisruptive and
Pervasive
General AIRevolutionary
▼ We are here 2050 and beyond 3IBM Research AI © 2018 IBM Corporation
The evolution of AI
![Page 4: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/4.jpg)
Narrow AISingle task, single domainSuperhuman accuracy and
speed for certain tasks
Broad AIMulti-task, multi-domain
Multi-modalDistributed AI
Explainable
General AICross-domain
learning and reasoning
Broad autonomy
4IBM Research AI © 2018 IBM Corporation
The evolution of AI
![Page 5: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/5.jpg)
Narrow AISingle task, single domainSuperhuman accuracy and
speed for certain tasks
Broad AIMulti-task, multi-domain
Multi-modalDistributed AI
Explainable
General AICross-domain
learning and reasoning
Broad autonomy
5IBM Research AI © 2018 IBM Corporation
The evolution of AI
![Page 6: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/6.jpg)
![Page 7: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/7.jpg)
Narrow AISingle task, single domainSuperhuman accuracy and
speed for certain tasks
Broad AIMulti-task, multi-domain
Multi-modalDistributed AI
Explainable
General AICross-domain
learning and reasoning
Broad autonomy
7IBM Research AI © 2018 IBM Corporation
The evolution of AI
![Page 8: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/8.jpg)
The path to a “Broad AI” toolbox
+
restaurant
cook Follow recipe
person
sweet
cheesecake
dessert
satisfy hunger oven
bake survive
swallow
eatcake
Learn more from small data
Reasoning
Explainability Security Ethics
Platform for AI Lifecycle
Infrastructure
Learns to transfer
+Physics of AI
Compute Data & Models Applications Workflow
![Page 9: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/9.jpg)
M I T - I B MW A T S O NA I L A B
![Page 10: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/10.jpg)
Narrow AIEmerging
Broad AIDisruptive and
Pervasive
General AIRevolutionary
▼ We are here 2050 and beyond 14IBM Research AI © 2018 IBM Corporation
The evolution of AI
![Page 11: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/11.jpg)
So what’s “narrow” about today’s AI toolbox?
![Page 12: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/12.jpg)
![Page 13: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/13.jpg)
Karpathy and Li, 2015
![Page 14: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/14.jpg)
![Page 15: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/15.jpg)
Gatys et al. 2015 Brock et al. 2018
![Page 16: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/16.jpg)
“Teddy Bear”
Meret Oppenheim, Le Déjeuner en fourrure
![Page 17: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/17.jpg)
Wang et al. 2018
![Page 18: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/18.jpg)
Karpathy and Li, 2015
![Page 19: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/19.jpg)
Lake, Ullman, Tenenbaum & Gershman, 2016
![Page 20: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/20.jpg)
![Page 21: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/21.jpg)
What’s this?
![Page 22: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/22.jpg)
![Page 23: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/23.jpg)
![Page 24: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/24.jpg)
![Page 25: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/25.jpg)
![Page 26: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/26.jpg)
#MITIBM #AI
ObjectNet
Andrei BarbuMIT
Boris KatzMIT
Dan GutfreundIBM
![Page 27: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/27.jpg)
#MITIBM #AI
ObjectNet
• ~50K images
• ~300 object classes
• 4 different room types
![Page 28: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/28.jpg)
#MITIBM #AI
Testing ImageNet-trained models on ObjectNet
![Page 29: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/29.jpg)
Chen et al. 2018
Pin-yu ChenIBM
![Page 30: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/30.jpg)
Xu et al. 2019
![Page 31: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/31.jpg)
![Page 32: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/32.jpg)
![Page 33: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/33.jpg)
![Page 34: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/34.jpg)
“Apple”
Neural Networks / Deep Learning
![Page 35: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/35.jpg)
apple
origin structurekind
apple tree body stem fruit
shape size color taste
round hand red green apple
Reproduced from Minksy, 1991
Symbolic AI
![Page 36: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/36.jpg)
Disentangling reasoning from vision and language understanding
Neural-symbolic AI
Joshua TenenbaumChuang GanJiajun Wu
Small bluerubbercube
Small greenrubbercylinder
Large redmetalsphere
Question: Are there an equal number of large things and metal spheres?
Program: equal_number(count(filter_size(S
cene, Large)), count(filter_material(filter_shape(Scene, Sphere), Metal)))
Answer: Yes
MIT-IBM Watson AI Lab
![Page 37: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/37.jpg)
End-to-End Visual Reasoning
Visual Question AnsweringQ: What’s the shape of the red object?
End-to-EndNeural Network A: Sphere.
NMN [Andreas et al., 2016]IEP [Johnson et al., 2017]FiLM [Perez et al., 2018],MAC [Hudson & Manning, 2018]Stack-NMN [Hu et al., 2018]TbD [Mascharka et al. 2018]
![Page 38: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/38.jpg)
End-to-EndNeural Network A: Sphere.
Visual Question AnsweringQ: What’s the shape of the red object?
NMN [Andreas et al., 2016]IEP [Johnson et al., 2017]FiLM [Perez et al., 2018],MAC [Hudson & Manning, 2018]Stack-NMN [Hu et al., 2018]TbD [Mascharka et al. 2018]
Concept(e.g., colors, shapes)
Reasoning(e.g., count)
End-to-End Visual Reasoning
![Page 39: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/39.jpg)
End-to-EndNeural Network A: Sphere.
Visual Question AnsweringQ: What’s the shape of the red object?
NMN [Andreas et al., 2016]IEP [Johnson et al., 2017]FiLM [Perez et al., 2018],MAC [Hudson & Manning, 2018]Stack-NMN [Hu et al., 2018]TbD [Mascharka et al. 2018]
Concept(e.g., colors, shapes)
Reasoning(e.g., count)
Entangled
End-to-End Visual Reasoning
![Page 40: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/40.jpg)
End-to-EndNeural Network A: Sphere.
Visual Question AnsweringQ: What’s the shape of the red object?
Image CaptioningInstance Retrieval
NMN [Andreas et al., 2016]IEP [Johnson et al., 2017]FiLM [Perez et al., 2018],MAC [Hudson & Manning, 2018]Stack-NMN [Hu et al., 2018]TbD [Mascharka et al. 2018]
Concept(e.g., colors, shapes)
Reasoning(e.g., count)
Entangled
Hard to transfer
End-to-End Visual Reasoning
![Page 41: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/41.jpg)
Task: Visual Reasoning
Question: Are there an equal number of large things and metal spheres?
![Page 42: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/42.jpg)
Question: Are there an equal number of large things and metal spheres?
Task: Visual Reasoning
![Page 43: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/43.jpg)
Question: Are there an equal number of large things and metal spheres?
3 large things!
Task: Visual Reasoning
![Page 44: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/44.jpg)
Question: Are there an equal number of large things and metal spheres?
3 large things!
3 metal spheres!
Task: Visual Reasoning
![Page 45: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/45.jpg)
Question: Are there an equal number of large things and metal spheres?
3 large things!
3 metal spheres!
Equal? Yes!
Task: Visual Reasoning
![Page 46: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/46.jpg)
Question: Are there an equal number of large things and metal spheres?
Visual Perception
Question Understanding
Logic Reasoning
3 large things!
3 metal spheres!
Equal? Yes!
Task: Visual Reasoning
![Page 47: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/47.jpg)
Vision (CNN)
Language (RNN)
Structured Representation
SymbolicProgram
![Page 48: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/48.jpg)
Incorporate Concepts in Visual ReasoningNS-VQA [Yi et al. 2018]
Vision
SceneParsing
Q: What’s the shape ofthe red object?
Language
![Page 49: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/49.jpg)
Incorporate Concepts in Visual Reasoning
Vision
1ID Color Shape Material1 Green Cube Metal
SceneParsing
Q: What’s the shape ofthe red object?
Language
NS-VQA [Yi et al. 2018]
![Page 50: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/50.jpg)
Incorporate Concepts in Visual Reasoning
Vision
ID Color Shape Material1 Green Cube Metal2 Red Sphere Rubber
1
2
Q: What’s the shape ofthe red object?
Language
SceneParsing
SemanticParsing Filter(Red)
Query(Shape)
Program
NS-VQA [Yi et al. 2018]
![Page 51: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/51.jpg)
Incorporate Concepts in Visual Reasoning
Vision
ID Color Shape Material1 Green Cube Metal2 Red Sphere Rubber
1
2
Q: What’s the shape ofthe red object?
LanguageSemanticParsing Filter(Red)
Query(Shape)
Program
SymbolicReasoning
SceneParsing
NS-VQA [Yi et al. 2018]
![Page 52: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/52.jpg)
Q: What’s the shape ofthe red object?
SceneParsing
Incorporate Concepts in Visual Reasoning
Vision
ID Color Shape Material1 Green Cube Metal2 Red Sphere Rubber
1
2
Language
SymbolicReasoning
SemanticParsing Filter(Red)
Query(Shape)
Program
NS-VQA [Yi et al. 2018]
![Page 53: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/53.jpg)
SceneParsing
Incorporate Concepts in Visual Reasoning
Vision
ID Color Shape Material1 Green Cube Metal2 Red Sphere Rubber
1
2Symbolic
Reasoning
Q: What’s the shape ofthe red object?
LanguageSemanticParsing Filter(Red)
Query(Shape)
Program
NS-VQA [Yi et al. 2018]
![Page 54: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/54.jpg)
SceneParsing
Incorporate Concepts in Visual Reasoning
Vision
1
2Symbolic
Reasoning
ID Color Shape Material1 Green Cube Metal2 Red Sphere Rubber
Q: What’s the shape ofthe red object?
LanguageSemanticParsing Filter(Red)
Query(Shape)
Program
NS-VQA [Yi et al. 2018]
Sphere
![Page 55: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/55.jpg)
Advantage 1: High Accuracy
Method Accuracy (%)Human 92.6
RN 95.5IEP 96.9
FiLM 97.6MAC 98.9TbD 99.1
NS-VQA (Ours) 99.8
[Yi et al. NeurIPS 2018, Johnson et al. ICCV 2017, Santoro et al. NIPS 2017, Perez et al. AAAI 2018, Hudson et al. ICLR 2018, Mascharka et al. CVPR 2018]
Effectively perfect!
![Page 56: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/56.jpg)
High accuracy when trained with just 1% the of the data that other methods require
[Yi et al. NeurIPS 2018]
Advantage 2: Data Efficiency
![Page 57: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/57.jpg)
Question: What number of cylinders are gray objects or tiny brown matte objects?
scenefilter_smallfilter_brownfilter_rubber
scenefilter_gray
unionfilter_cylinder
count
filter_smallfilter_brownfilter_largefilter_cyan
...(25 modules)filter_metal
unionfilter_cylinder
count
Ours
Answer: 1
IEP
Answer: 2
Question: Are there more yellow matte things that are right of the gray ball than cyan metallic objects?
scenefilter_cyanfilter_metal
count...(4 modules)
scenefilter_yellowfilter_rubber
countgreater_than
filter_smallfilter_cyan
unionfilter_brown
...(25 modules)filter_smallfilter_yellowfilter_rubber
countgreater_than
Ours
Answer: no
IEP
Answer: no
Advantage 3: Transparency and Interpretability
[Yi et al. NeurIPS 2018, Johnson et al. ICCV 2017]
Question: What number of cylinders are gray objects or tiny brown matte objects?
scenefilter_smallfilter_brownfilter_rubber
scenefilter_gray
unionfilter_cylinder
count
filter_smallfilter_brownfilter_largefilter_cyan
...(25 modules)filter_metal
unionfilter_cylinder
count
Ours
Answer: 1
IEP
Answer: 2
Question: Are there more yellow matte things that are right of the gray ball than cyan metallic objects?
scenefilter_cyanfilter_metal
count...(4 modules)
scenefilter_yellowfilter_rubber
countgreater_than
filter_smallfilter_cyan
unionfilter_brown
...(25 modules)filter_smallfilter_yellowfilter_rubber
countgreater_than
Ours
Answer: no
IEP
Answer: no
![Page 58: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/58.jpg)
62
NeurIPS 2018: Neurosymbolic VQA: Properties (e.g. “color”) and values (“red”) predefined
ICLR 2019: Neurosymbolic Concept Learner: Properties predefined, can learn new values autonomously
NeurIPS 2019: Neurosymbolic Metaconcept Learner:Autonomously learns new concepts
less predefined, more autonomous →
ICML 2020 (target submission): Real world images
![Page 59: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/59.jpg)
Filter Queryred shape
Neuro-Symbolic Concept Learning
Q: What’s the shapeof the red object?
2
1Visual RepresentationObj 1
Concept Embeddingsred......
Obj 2
![Page 60: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/60.jpg)
Filter Queryred shape
Q: What’s the shapeof the red object?
2
1Visual RepresentationObj 1
Concept Embeddingsred......
Obj 2
Color Space
General Representation SpaceObj 1
Color(Obj 1)
ColorProj.
Neuro-Symbolic Concept Learning
![Page 61: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/61.jpg)
Filter Queryred shape
Q: What’s the shapeof the red object?
2
1Visual RepresentationObj 1
Concept Embeddingsred......
Obj 2
Color Space
General Representation SpaceObj 1
Color(Obj 1)
ColorProj.
Obj 2
Color(Obj 2)
Neuro-Symbolic Concept Learning
![Page 62: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/62.jpg)
Filter Queryred shape
Q: What’s the shapeof the red object?
2
1Visual RepresentationObj 1
Concept Embeddingsred......
Obj 2
Color Space
General Representation SpaceObj 1
Color(Obj 1)
Obj 2
Color(Obj 2)
red
Neuro-Symbolic Concept Learning
![Page 63: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/63.jpg)
apple
origin structurekind
apple tree body stem fruit
shape size color taste
round hand red green apple
Reproduced from Minksy, 1991
Symbolic AI
![Page 64: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/64.jpg)
Visual reasoning questions + Metaconcept questionsQ: Is red a same kind of concept as green?A: Yes.
Q: Is cube a synonym of block?A: Yes.
Q: Is Laridae a hypernym of Ivory gull?A: Yes.
CLEVR(Johnson et al. 2017)
color:red
color:green
Q: Is there any red cube?A: Yes.
Q: Is there any green block?A: Yes
CUB(Wah et al. 2011)
IvoryGull
LaridaeBlackTern
Q: Is there any Ivory Gull?A: Yes.Q: Is there any Laridae?A: Yes.Q: Is there any Black Tern?A: Yes.Q: Is there any Laridae?A: Yes.
Meta-concept Learning Han et al. NeurIPS 2019
![Page 65: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/65.jpg)
Augmenting VQA with MetaconceptsVisual reasoning questions + Metaconcept questions
Q: Is red a same kind of concept as green?A: Yes.
Q: Is cube a synonym of block?A: Yes.
Q: Is Laridae a hypernym of Ivory gull?A: Yes.
CLEVR(Johnson et al. 2017)
color:red
color:green
Q: Is there any red cube?A: Yes.
Q: Is there any green block?A: Yes
CUB(Wah et al. 2011)
IvoryGull
LaridaeBlackTern
Q: Is there any Ivory Gull?A: Yes.Q: Is there any Laridae?A: Yes.Q: Is there any Black Tern?A: Yes.Q: Is there any Laridae?A: Yes.
![Page 66: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/66.jpg)
Program Execution Animated
Visual reasoning questions
ObjectDetection
FeatureExtraction
Q: Is there any red object? P: Exist( Filter( red ) )
SemanticParsing
red
Metaconcept questions
Q: Is red a same kind ofconcept as yellow?
P: MetaVerify(red, yellow, same-kind
)
SemanticParsing
score=0.9
score=0.1Max score=0.9
score=0.9redsame-kindyellow
MetaVerify
Similarity
Similarity
Answer: Yes
Obj. 1
Obj. 2
Answer: Yes
P: Exist( Filter( red ) )P: Exist( Filter( red ) )P: Exist( Filter( red ) )
P: MetaVerify(red, yellow, same-kind
)
P: MetaVerify(red, yellow, same-kind
)
![Page 67: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/67.jpg)
Generalization
Metaconcept GeneralizationQ: Is there any airplane?A: Yes
Q: Is there any plane?A: Yes
Q: Is there any kid?A: Yes
Q: Is there any child?A: Yes
Q: Is airplane a synonym of plane?A: YesQ: Is kid a synonym of child?A: Yes
synonym?synonym
Training Testing: metaconcepts onunseen pairs of concepts
airplane
plane
kid
child
![Page 68: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/68.jpg)
Metaconcept Generalization: ResultsQ: Is there any airplane?A: Yes
Q: Is there any plane?A: Yes
Q: Is there any kid?A: Yes
Q: Is there any child?A: Yes
Q: Is airplane a synonym of plane?A: YesQ: Is kid a synonym of child?A: Yes
Training Testing
Generalization
![Page 69: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/69.jpg)
CLEVERER: CoLlision Events for Video REpresentation and Reasoning• Descriptive
Q: What is the material of the last object to collide with the cyan cylinder?
Chuang Gan w/ Kevin Xi, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba & Josh Tenenbaum
A: Metal
![Page 70: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/70.jpg)
• Explanatory
Q: What is responsible for the collision between the rubber and metal cylinder?
A. The presence of the yellow sphereB. The collision between the rubber cylinder and the red rubber sphere
![Page 71: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/71.jpg)
• Counterfactual
Q: What will happen without the cyan cylinder?
A. The red rubber sphere and the metal sphere collideB. The red rubber sphere and the gray object collide
![Page 72: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/72.jpg)
Why is our database down?
What factors might contribute to better output from Factory A vs. Factory B?
How many employees have over 10 years experience but have moved location in the last year?
Looking Ahead
![Page 73: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/73.jpg)
NeurosymbolicPlanning
NeurosymbolicSafe ML/RL
NeurosymbolicNLU
NeurosymbolicCode Optimization
NeurosymbolicGenerative Models
NeurosymbolicMachine Common Sense
Srivastava et al. 2020 (submitted)
Asai et al. AAAI 2018 Shi et al. ICLR 2019
Wilcox et al. NAACL 2019Fulton et al AAAI 2018
Smith et al. NeurIPS 2019
![Page 74: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/74.jpg)
Inferring flexible behavioral plans/policies from temporal observation data
Inducing Behavioral Insight
MIT-IBM Watson AI Lab
Christian MuiseIBM
Julie ShahMIT
![Page 75: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/75.jpg)
(:action pickup
:parameters (?b1 ?b2 - block)
:precondition (and (on ?b1 ?b2)(hand-clear))
:effect (and (not (hand-clear))(not (on ?b1 ?b2))(holding ?b1))
)
Task: Induce the action theory of anenvironment through observations
![Page 76: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/76.jpg)
Mixing symbolic planning with neural networks
LatPlan
MIT-IBM Watson AI Lab
Masataro AsaiIBM
![Page 77: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/77.jpg)
Mixing symbolic planning with neural networks
LatPlan
MIT-IBM Watson AI Lab
Masataro AsaiIBM
![Page 78: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/78.jpg)
Verifiably Safe Reinforcement Learning
82
{accel,brake,turn}
Observe Reward
Safe?Policy
φ
Use a theorem prover to prove:init→[{{accel∪brake};ODEs}*]safe
is correctly monitored by φ.
Nathan FultonIBM
MIT-IBM Watson AI Lab
![Page 79: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/79.jpg)
apple
origin structurekind
apple tree body stem fruit
shape size color taste
round hand red green apple
+
NEURAL NETWORKS SYMBOLIC AI
![Page 80: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/80.jpg)
Causal InferenceBeyond Correlation—inferring and testing for causal relationships in complex systems
Caroline UhlerMIT
Guy BreslerMIT
Karthikeyan Shanmugam
IBM
http://tylervigen.com/spurious-correlations
MIT-IBM Watson AI Lab
![Page 81: Neurosymbolic AIintrotodeeplearning.com/2020/slides/6S191_MIT_Deep...Narrow AI Single task, single domain Superhuman accuracy and speed for certain tasks Broad AI Multi-task, multi-domain](https://reader033.fdocuments.net/reader033/viewer/2022053119/60a0ff8587810929d61dcdd8/html5/thumbnails/81.jpg)
S2 S7
S1 S3
S4
S6
S5
MIT-IBM Watson AI Lab