15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… ·...
Transcript of 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… ·...
![Page 1: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/1.jpg)
15-780: Graduate AINatural Language Processing
Geoff Gordonwith thanks to Noah Smith, LTI, MLD
![Page 2: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/2.jpg)
Admin
Apologies for the late start to Tuesday’s lecture!HW3 due todayHW4 out (due Tuesday, 11/13)
2
![Page 3: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/3.jpg)
Admin
Project proposals due Thursday, 11/8Work in groups of 2If you’re having trouble finding a partner, email thlin@cs by tomorrow (Friday)
include brief statement of interestsAnyone who emails will get the list of people looking for partners
3
![Page 4: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/4.jpg)
Project proposals
A good proposal answers:What result do you hope to get?Why is it interesting?Why is it related to Grad AI?
Limit 1 page
4
![Page 5: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/5.jpg)
Admin
Midterm approaching fast!In class, Thursday 11/15 (two weeks)Review sessions: 11/12 and 11/13 in evening (time and place TBA)
5
![Page 6: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/6.jpg)
Admin
By request, we are also adding a hands-on practice session
scheduled for next Monday evening, time and place TBA over email
Idea: work through some larger example problems in detailNot necessarily midterm-relatedEmail me requests for problem types
6
![Page 7: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/7.jpg)
Game example
![Page 8: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/8.jpg)
A political game
C W O
C –1, 5 0, 0 –5, –3
W 0, 0 0, 0 –5, –3
O –3, –10 –3, –10 –8, –13
8
![Page 9: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/9.jpg)
A political game
!10 !5 0 5!14
!12
!10
!8
!6
!4
!2
0
2
4
6
Payoff to L
Pa
yo
ff t
o U CC
WC
OC
CWWW
OW
COWO
OO
Figure 1: Equilibria of the repeated cooperation game.
much in an equilibrium as he can guarantee himself by acting selfishly (calledhis safety value), and any possible vector of payo!s that satisfies this restrictioncorresponds to an equilibrium.
Figure 1 illustrates the equilibria of the repeated cooperation game, whoseone-round payo!s are given in Table 1. The pentagon in the figure is the feasibleregion—the set of all possible joint payo!s.2 The corners of the feasible regioncorrespond to the pure joint actions as labeled; e.g., the corner labeled WCshows the payo! to each player when U plays W and L plays C. (Some cornershave multiple labels, since some sets of joint actions have the same payo!s.)The dashed lines indicate the safety values: L will not accept less than !10,and U will not accept less than !5. The stippled area (the intersection of thefeasible and individually rational sets) is the set of payo!s that can result fromequilibrium play.
Figure 1 demonstrates that there is more to cooperation than just equilib-rium computation: the payo! vectors that correspond to equilibria (stippledregion) are a significant fraction of all payo! vectors (pentagon); so, neitherparty can use the set of equilibria to place meaningful limits on the payo!sthat he will receive. Worse, each di!erent equilibrium requires the agents to actdi!erently, and so the set of equilibria does not give the agents any guidanceabout which actions to select.
2Here we show average payo!, but the situation is essentially the same for total discountedfuture payo!.
3
R
C
X
9
![Page 10: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/10.jpg)
What if?
Didn’t know each other’s exact payoffs?Couldn’t observe each other’s exact actions?Actions altered state of world?We’ll talk about some of these in later part of course
10
![Page 11: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/11.jpg)
Another example
11
![Page 12: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/12.jpg)
Let’s play the lottery
($6, .05; $91, .03; $99, .92) ($6, .02; $8, .03; $99, .95)Which would you pick?
0
25
50
75
100
5%, 2% 3%, 3% 92%, 95%
Lottery 1Lottery 2
12
![Page 13: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/13.jpg)
Rationality
People often pick ($6, .05; $91, .03; $99, .92)
over ($6, .02; $8, .03; $99, .95)
But, note stochastic dominance
13
![Page 14: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/14.jpg)
Stochastic dominance
Birnbaum & Navarrete. Testing Descriptive Utility Theories: Violations of Stochastic Dominance and
Cumulative Independence
6
8
91
996
99
14
![Page 15: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/15.jpg)
NLP(thanks to Noah Smith)
(errors are my own)
![Page 16: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/16.jpg)
Overview
Overview of trends and tradeoffs in NLPMajor issues in language processingDiscussion of example applications, problems & solutions
Statistical parsingMachine translation
16
![Page 17: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/17.jpg)
Language is central to intelligence
One of the best ways to communicate with those pesky humansOne of the best ways to represent complex, imperfectly-defined conceptsOne of the most flexible reasoning systems ever invented
![Page 18: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/18.jpg)
Language shapes the way we think, and determines what we can think about.
— Benjamin Lee Whorf
Language is central to intelligence
![Page 19: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/19.jpg)
NLP is Interdisciplinary
mathematics/machine learning/CS theory
linguistics cognitivescience
engineering
goal: perfectly
model human
language
goal: plausibly
model human
language (like
humans)
goal: build stuff that works
goal: understand
formal properties
19
![Page 20: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/20.jpg)
NLP is hard!
Ambiguity at all levelsMany different styles of languageLanguage is productive
20
![Page 21: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/21.jpg)
Levels of Language
characters
words
sentences
documents
collections
sounds
words
utterances
conversations
Boundaries not always
unambiguous!
(speech) (text)
21
![Page 22: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/22.jpg)
Where are the words?世界人权宣言
联合国大会一九四八年十二月十日第217A(III)号决议通过并颁布1948 年 12 月 10 日, 联 合 国 大 会 通 过 并 颁 布《 世 界 人 权 宣
言》。 这 一 具 有 历 史 意 义 的《 宣 言》 颁 布 后, 大 会 要 求 所 有
会 员 国 广 为 宣 传, 并 且“ 不 分 国 家 或 领 土 的 政 治 地 位 , 主 要
在 各 级 学 校 和 其 他 教 育 机 构 加 以 传 播、 展 示、 阅 读 和 阐
述。” 《 宣 言 》 全 文 如 下:序 言
鉴 于 对 人 类 家 庭 所 有 成 员 的 固 有 尊 严 及 其 平 等 的 和 不 移 的
权 利 的 承 认, 乃 是 世 界 自 由、 正 义 与 和 平 的 基 础,
鉴 于 对 人 权 的 无 视 和 侮 蔑 已 发 展 为 野 蛮 暴 行, 这 些 暴 行 玷 污
了 人 类 的 良 心, 而 一 个 人 人 享 有 言 论 和 信 仰 自 由 并 免 予 恐 惧
和 匮 乏 的 世 界 的 来 临, 已 被 宣 布 为 普 通 人 民 的 最 高 愿 望,22
![Page 23: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/23.jpg)
Where are the morphemes?
İnsan hakları evrensel beyannamesiÖnsöz
İnsanlık ailesinin bütün üyelerinde bulunan haysiyetin ve bunların eşit ve devir kabul etmez haklarının tanınması hususunun, hürriyetin, adaletin ve dünya
barışının temeli olmasına,İnsan haklarının tanınmaması ve hor görülmesinin insanlık vicdanını isyana sevkeden vahşiliklere sebep olmuş bulunmasına, dehşetten ve yoksulluktan
kurtulmuş insanların, içinde söz ve inanma hürriyetlerine sahip olacakları bir dünyanın kurulması en yüksek amaçları oralak ilan edilmiş bulunmasına,İnsanin zulüm ve baskıya karşı son çare olarak ayaklanmaya mecbur
kalmaması için insan haklarının bir hukuk rejimi ile korunmasının esaslı bir zaruret olmasına,
Uluslararasında dostça ilişkiler geliştirilmesini teşvik etmenin esaslı bir zaruret olmasına,
23
![Page 24: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/24.jpg)
Which words are these?
24
![Page 25: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/25.jpg)
Word Sense Disambiguation
… plant …… workers at the plant …… plant a garden …… plant meltdown …… graze … plant …… house plant … … CIA plant …… plant firmly on the ground …
pick a dictionary definition for each
25
![Page 26: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/26.jpg)
Language
Ambiguity at all levelshaven’t even gone above words yet—it gets worse!
Diversity of domainsNew York Times v. Wall Street Journalnewspaper v. research papernewspaper v. blogblog v. conversation, chat, …
26
![Page 27: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/27.jpg)
Language is productive
New words appear all the timeVery many rare words, so it’s a common occurrence to see a rare word
27
![Page 28: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/28.jpg)
Zipf’s Law
Type: element of a set (e.g., vocabulary word)Zipf: The most frequent types are extremely frequent, and there’s a long tail.
frequency × rank ≈ constantFirst noticed for words; holds for pretty much everything in NLP.Result: sparse data, generalization hard
28
![Page 29: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/29.jpg)
Zipf’s Law
0
0.05
0.10
0.15
0.20
0.25
0.30
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2029
![Page 30: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/30.jpg)
Word Classes
Useful to abstract from the words themselves.Nouns: { cat, dog, horse, pig, cookie, protest, … }Verbs: { hunt, eats, kill, cook, animated, … } More verbs: { dog, horse, pig, protest, … }More nouns: { hunt, eats, kill, cook, … }Adjectives: { animate, funny, heavy-handed, … }Linguist required: { what, that, up, umm, … }
30
![Page 31: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/31.jpg)
Word Classes
Haven’t even gotten to fancier classes:Animate: { horse, cat, professor, … }Place: { New York, Wean 5409, under the boardwalk, … }Intangible: { blue, filibuster, complexity, … }
31
![Page 32: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/32.jpg)
Goals
Given all of this complexity and ambiguity, we want to:
UnderstandRespondTranslateClassify…
32
![Page 33: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/33.jpg)
Goals
For any of these goals, we need some deeper representation.
Two common levels beyond words:
Words Syntax Semantics
How do we get there?
33
![Page 34: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/34.jpg)
Syntax
First thought: use lex/yacc to build a parser for a natural language just like for a programming language!
Need to know grammar of NL
Tremendous number of possible rules; no spec.
Zipf’s law attacks again.Where is NL in the Chomsky Hierarchy?
34
![Page 35: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/35.jpg)
Chomsky Hierarchy
Unrestricted Turing machine
Context-sensitiveNondeterministic linear-bounded
automaton
Context-free Pushdown automaton
Regular Finite-state machine
Grammar type Machine type
35
![Page 36: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/36.jpg)
Chomsky Hierarchy
Some linguistic phenomena don’t exhibit very long-ranging influences.
Phonetics, phonology.Speech recognition uses mostly finite-state models.
Linguists have used examples to demonstrate that there is
arbitrary center-embedding (i.e., NL is not FS)36
![Page 37: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/37.jpg)
NL is not Finite-State
This is the cat.
This is the dog the cat chased.
This is the man the dog the cat chased bit.
This is the woman the man the dog the cat chased bit kissed. 37
![Page 38: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/38.jpg)
NL is not Finite-State
Si V1 Oj
Sk V2 tj
relative clause;modifies Oj
Sl V3 tk
relative clause;modifies Sk
38
![Page 39: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/39.jpg)
Chomsky Hierarchy
Some linguistic phenomena don’t exhibit very long-ranging influences.
Phonetics, phonology.Speech recognition uses mostly finite-state models.
Linguists have used examples to demonstrate that there is
arbitrary center-embedding (i.e., NL is not FS)cross-serial dependencies (i.e., NL is not CF).
39
![Page 40: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/40.jpg)
Chomsky Hierarchy
Many context-sensitive models of language have been proposed!NLP still uses FS or CF models mostly, for speed and coverage.
40
![Page 41: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/41.jpg)
Syntax
First thought: use lex/yacc to build a parser for a natural language just like for a programming language!Where is NL in the Chomsky Hierarchy?
Context-sensitive, but we’ll pretend context-free.
Problem: ambiguity.
41
![Page 42: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/42.jpg)
Ambiguity in English
• IT KNOWS YOU LIKE YOUR MOTHER
• IRAQI HEAD SEEKS ARMS
• JUVENILE COURT TO TRY SHOOTING DEFENDANT
• KIDS MAKE NUTRITIOUS SNACKS
• BRITISH LEFT WAFFLES ON FALKLAND ISLANDS
• LITTLE HOPE GIVEN BRAIN-DAMAGED CHILD
• NEVER WITHHOLD HERPES INFECTION FROM LOVED ONE
• STOLEN PAINTING FOUND BY TREEThanks to: J. Eisner, L. Lee 42
![Page 43: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/43.jpg)
Syntactic Ambiguity
IT KNOWS YOU LIKE YOUR MOTHER
“that”
subjsubj
obj
p = You like your mother.
It knows p.
43
![Page 44: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/44.jpg)
Syntactic Ambiguity
IT KNOWS YOU LIKE YOUR MOTHER
subj
mod
obj arg
YOUR MOTHERIT KNOWS
subj obj
YOUR MOTHER KNOWS YOU
subjobj
It knows your mother in a way p.
It knows you in way p.
Your mother knows you in a way p.
It knows you in way p.
44
![Page 45: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/45.jpg)
Semantic Ambiguity
IT KNOWS YOU LIKE
subj
mod
obj arg
YOUR MOTHER KNOWS YOU
subjobj
the computer?the same “it” that
rains?your bad habits?your first word? 45
![Page 46: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/46.jpg)
Pragmatics and World Knowledge
IT KNOWS YOU LIKE YOUR MOTHER
This statement isn’t meant literally!Someone is trying to sell you something.They are juxtaposing the product with a competitor’s product that is impervious to the user.
46
![Page 47: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/47.jpg)
Ambiguity
Headlines are more ambiguous than most textBut, with any broad-coverage grammar, almost any sentence of reasonable complexity will be ambiguous, often in ways humans will never notice
47
![Page 48: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/48.jpg)
Tradeoffs
words syntax semantics pragmatics
precis
ion,
expres
sivity
,
plausi
bility
of
model
coverage,
efficiency
48
![Page 49: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/49.jpg)
How can we handle ambiguity?
Probability
49
![Page 50: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/50.jpg)
The Revolution
In 1980s and 1990s, NLP started borrowing from speech recognition, information theory, and machine learning.
Probability models, including weighted grammars
Use of statistics on data (corpora)
50
![Page 51: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/51.jpg)
The Revolution
The new paradigm involves learning to accomplish tasks accurately from data.
How much data? What kind?Same tradeoffs as before, and some new ones!
51
![Page 52: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/52.jpg)
Example: Statistical Parsing
Input: a sentenceOutput: a parse tree (usually labeled constituents)
Evaluation: compare to gold standard tree, count erroneous constituents.*Before 1993: write rules by hand.1993: Penn Treebank
A million words’ worth of Wall Street Journal text, annotated by linguists with a consensus structure
52
![Page 53: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/53.jpg)
How We Do It
Assume a model p(t, w).
Probability distribution over discrete structures (trees and sequences).Starting point: Probabilistic Context-Free Grammar (aka Stochastic CFG)
53
![Page 54: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/54.jpg)
Probabilistic CFG
Just like CFGs, but with probability distribution at each rewrite.
S
54
![Page 55: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/55.jpg)
Probabilistic CFG
Sp(NP VP . | S) = 0.44
p(VP ! | S) = 0.26p(Is NP VP ? | S) = 0.27
p(NP . | S) = 0.01…
VP !
Just like CFGs, but with probability distribution at each rewrite.
55
![Page 56: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/56.jpg)
Probabilistic CFG
Sp(V | VP) = 0.24
p(V NP | VP) = 0.23p(V PP | VP) = 0.21
p(V NP PP | VP) = 0.16…
VP !V PP
Just like CFGs, but with probability distribution at each rewrite.
56
![Page 57: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/57.jpg)
Probabilistic CFG
Sp(eat | V) = 0.03p(buy | V) = 0.03p(sell | V) = 0.03
p(implement | V) = 0.02…
p(listen | V) = 0.01…
VP !V PP
listen
Just like CFGs, but with probability distribution at each rewrite.
57
![Page 58: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/58.jpg)
Probabilistic CFG
Sp(to NP | PP) = 0.80
p(up | PP) = 0.02…VP !
V PPlisten NPto
Just like CFGs, but with probability distribution at each rewrite.
58
![Page 59: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/59.jpg)
Probabilistic CFG
Sp(him | NP) = 0.06p(us | NP) = 0.02
…p(me | NP) = 0.01
VP !V PP
listen NPto
me
Just like CFGs, but with probability distribution at each rewrite.
59
![Page 60: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/60.jpg)
Probabilistic CFG
SVP !
V PPlisten NPto
me
p
= p(VP ! | S) × p(V PP | VP) × p(listen | V)
× p(to NP | PP) × p(me | NP)
= 0.26 × 0.21 × 0.01 × 0.80 × 0.01
= 0.0000004368
Just like CFGs, but with probability distribution at each rewrite.
60
![Page 61: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/61.jpg)
How We Do It
Assume a model p(t, w).
Train the model on the Treebank.To parse infer the best tree: maxt p(t | w)
Discrete optimization problemOr, infer properties of posterior over trees: P(words 5–9 are a VP)
Probabilistic inference problem61
![Page 62: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/62.jpg)
Problem
Problem: possible t is O(exp |w|)
Solution: dynamic programmingSimilar to forward-backward or Viterbi algorithms for HMMsAnalog of forward-backward: inside-outsideAnalog of Viterbi: PCKY (Probabilistic Cocke-Kasami-Younger)—will show here
62
![Page 63: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/63.jpg)
Optimization
For simplicity, assume grammar in Chomsky normal formAll productions are
A → BC
A → word
A → ε (nothing)
63
![Page 64: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/64.jpg)
Chomsky normal form example
S → NP VP
VP → V NP
NP → D N
D → the (0.6) | those (0.4)
N → cat(s) (0.3) | dog(s) (0.7)
V → hear(s) (0.9) | dog(s) (0.1)
the cat dogs the dogs
64
![Page 65: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/65.jpg)
Optimization
Given a string of nonterminals:the cat dogs the dogs
And a probabilistic context free grammar (previous slide)Figure out most likely parse
65
![Page 66: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/66.jpg)
Dynamic programming
String of words XFor nonterminal N, write
Pmax(N, X) = max P(t, X)
Similarly, for production A → B C, write
Pmax(A → B C, X) = max P(t, X)
t = N
t = A
B C66
![Page 67: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/67.jpg)
Dynamic programming
Now, we havePmax(VP, X) =
max Pmax((VP → …), X) P(VP → …)
Pmax((VP → V NP), X) =
max Pmax(V, Y) Pmax(NP, Z)X=YZ
VP → …
67
![Page 68: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/68.jpg)
Dynamic programming
Build a table P(i, j, k) = probability of generating the substring from word i to word j from nonterminal k using best possible tree = Pmax(k, X[i..j]) In our example (5 words, 6 nonterminals), this is double[5, 5, 6]
some elements unused (triangle array)
68
![Page 69: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/69.jpg)
Dynamic programming
ways of generating symbol 2
ways of generating
symbols 2–4
the cat dogs the dogs69
![Page 70: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/70.jpg)
Dynamic programming
D .6 N .3 N .7, V .1 D .6 N .7, V .1
the cat dogs the dogs
D → the (0.6) | those (0.3)
N → cat(s) (0.3) | dog(s) (0.7)
V → hear(s) (0.9) | dog(s) (0.1)
S → NP VP
VP → V NP
NP → D N
70
![Page 71: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/71.jpg)
Dynamic programming
NP .18 NP .42
D .6 N .3 N .7, V .1 D .6 N .7, V .1
the cat dogs the dogs
D → the (0.6) | those (0.3)
N → cat(s) (0.3) | dog(s) (0.7)
V → hear(s) (0.9) | dog(s) (0.1)
S → NP VP
VP → V NP
NP → D N
71
![Page 72: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/72.jpg)
Dynamic programming
NP .18 NP .42
VP .042
D .6 N .3 N .7, V .1 D .6 N .7, V .1
the cat dogs the dogs
D → the (0.6) | those (0.3)
N → cat(s) (0.3) | dog(s) (0.7)
V → hear(s) (0.9) | dog(s) (0.1)
S → NP VP
VP → V NP
NP → D N
72
![Page 73: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/73.jpg)
Dynamic programming
NP .18 NP .42
VP .042
D .6 N .3 N .7, V .1 D .6 N .7, V .1
the cat dogs the dogs
D → the (0.6) | those (0.3)
N → cat(s) (0.3) | dog(s) (0.7)
V → hear(s) (0.9) | dog(s) (0.1)
S → NP VP
VP → V NP
NP → D N
73
![Page 74: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/74.jpg)
Dynamic programming
NP .18 NP .42
VP .042
S .0076
D .6 N .3 N .7, V .1 D .6 N .7, V .1
the cat dogs the dogs
D → the (0.6) | those (0.3)
N → cat(s) (0.3) | dog(s) (0.7)
V → hear(s) (0.9) | dog(s) (0.1)
S → NP VP
VP → V NP
NP → D N
74
![Page 75: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/75.jpg)
More Powerful Models
Link words to their arguments: lexicalization
Smooth model for better generalizationTrain models using more sophisticated machine learning
75
![Page 76: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/76.jpg)
Applications
Machine translation
Speech recognition, synthesis, dialog
Information Retrieval
Question Answering
Sentiment Analysis
Spelling/Grammar Checking
Digitization (Optical Character Recognition)
Natural Language Interfaces
Language Education76
![Page 77: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/77.jpg)
Example: Statistical Translation
Input: Chinese sentenceOutput: English translationEvaluation: how close is output to a reference translation? (controversial how to measure this!)
Before 1990: write rules by hand.
77
![Page 78: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/78.jpg)
Example: Statistical Translation
Predominant approach now: learn to translate from a parallel corpus of examples.
Parliamentary proceedings from bilingual countries (Canada, Hong Kong) or the EU or UN; also lawsNews from agencies that publish in multiple languagesNowadays: tens-to-hundreds of millions of words each side
78
![Page 79: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/79.jpg)
Translation by Modeling
Warren Weaver (1948):This Russian document is actually an encoded English document! That is, the writer was thinking in English, and
somehow the message was garbled into this strange “Russian” stuff. All we have to do is decode!
Modern MT: model the source (English sentences) and the channel (translation):
€
) e c( )← argmax
e
p ec( ) = argmaxe
p ce( ) ⋅p e( )p c( )
= argmaxe
p ce( ) ⋅p e( )79
![Page 80: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/80.jpg)
Three Statistical MT Problems
Build a language model over English sentencesLearn from English data!
Build a translation model that turns English into Chinese
Learn from parallel data!Build a decoder that finds the best English sentence, given Chinese input.
NP hard for many models!Difficult search problem.
80
![Page 81: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/81.jpg)
Translational Structure
Klimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
?
81
![Page 82: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/82.jpg)
Translational Structure
Word-to-word correspondences?
Klimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
82
![Page 83: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/83.jpg)
Translational Structure
Word-to-word correspondences?Phrases?
Klimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
83
![Page 84: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/84.jpg)
Translational Structure
Word-to-word correspondences?Phrases?Target tree?
Klimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
84
![Page 85: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/85.jpg)
Translational Structure
Word-to-word correspondences?Phrases?Target tree?Synchronous tree?
Klimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
85
![Page 86: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/86.jpg)
Translational Structure
Word-to-word correspondences?
Phrases?
Source tree?
Synchronous tree?
Synchronous dependency tree?
Klimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
86
![Page 87: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/87.jpg)
Translational Structure
Word-to-word correspondences?
Phrases?
Source tree?
Synchronous tree?
Synchronous dependency tree?
Czech-prime?
Klimatizovan- jídelna, světl- snídan- místnostKlimatizovaná jídelna, světlá místnost pro snídaně.
Air-conditioned dining room, well-lit breakfast room.
87
![Page 88: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/88.jpg)
Statistical MT
As before:Ambiguity at all levelsTradeoffs
Will not discuss the models or how they are learned, but lots of interesting stuff here…
88
![Page 89: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/89.jpg)
Current Hot Areas
Domain: most text ain’t newstext!Biomedical text
Blogs
Conversation
Multilingual NLP: most languages are not like English!
Models and representations (e.g., features) for deeper understanding (e.g., sentiment) or simply better accuracy
Learning from unannotated data (or less-annotated data, or less annotated data)
How should we evaluate NLP systems?89
![Page 90: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/90.jpg)
Summary
Language is hard because it is productive and ambiguous on all levels.
NLP is about trading off betweenAccuracy and coverage
Speed and expressive power
Human and computational effort
General mathematical formalisms and specific applications
• Optimization, search, and probabilistic inference methods underlie much of modern NLP (e.g., dynamic programming, training and applying models).
90
![Page 91: 15-780: Graduate AI Natural Language Processingggordon/780-fall07/lectures/natural-langua… · 15-780: Graduate AI Natural Language Processing Geoff Gordon with thanks to Noah Smith,](https://reader036.fdocuments.net/reader036/viewer/2022062506/5f07eff67e708231d41f8248/html5/thumbnails/91.jpg)
Courses of Interest (11nnn)
Language and Statistics (I and II!)Algorithms in NLPGrammars and LexiconsInformation ExtractionInformation RetrievalMachine TranslationSpeech Recognition and Understanding
11-762 (Noah Smith)
Also 11-411 (u-grad / masters)
91