Paper PresentationA Pendulum Swung Too Far (2011) by

Kenneth Church

Sagar Ahire [133050073]

Roadmap● Introduction● History of NLP● Objections to Empiricism

○ Chomsky○ Minsky○ Pierce

● Reasons for the Problem and Solutions

Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism

○ Chomsky○ Minsky○ Pierce

● Reasons for the Problem and Solutions

Introduction● The paper deals with the oscillation between

the predominance of theory-driven approaches vs data-driven approaches in the history of NLP and its reasons.

● Specifically, it predicts a surge in rationalism in the 2010s and explains why and how researchers need to be prepared for it.

Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate

language faculty”3. Aims at discovering the

language of the human mind (linguistic competence)

4. Assigns categories to language units

5. Major advocates: Chomsky, Minsky

Empiricism1. Emphasizes on data2. Assumes all knowledge

gathered only via senses3. Aims at analysing

language as it actually occurs (linguistic performance)

4. Assigns probabilities to language units

5. Major advocates: Shannon, Norvig

Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate

language faculty”3. Aims at discovering the

language of the human mind (linguistic competence)

4. Assigns categories to language units

5. Major advocates: Chomsky, Minsky

Empiricism1. Emphasizes on data2. Assumes all knowledge

gathered only via senses3. Aims at analysing

language as it actually occurs (linguistic performance)

4. Assigns probabilities to language units

5. Major advocates: Shannon, Norvig

Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate

language faculty”3. Aims at discovering the

language of the human mind (linguistic competence)

4. Assigns categories to language units

5. Major advocates: Chomsky, Minsky

Empiricism1. Emphasizes on data2. Assumes all knowledge

gathered only via senses3. Aims at analysing

language as it actually occurs (linguistic performance)

4. Assigns probabilities to language units

5. Major advocates: Shannon, Norvig

Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate

language faculty”3. Aims at discovering the

language of the human mind (linguistic competence)

4. Assigns categories to language units

5. Major advocates: Chomsky, Minsky

Empiricism1. Emphasizes on data2. Assumes all knowledge

gathered only via senses3. Aims at analysing

language as it actually occurs (linguistic performance)

4. Assigns probabilities to language units

5. Major advocates: Shannon, Norvig

Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate

language faculty”3. Aims at discovering the

language of the human mind (linguistic competence)

4. Assigns categories to language units

5. Major advocates: Chomsky, Minsky

Empiricism1. Emphasizes on data2. Assumes all knowledge

gathered only via senses3. Aims at analysing

language as it actually occurs (linguistic performance)

4. Assigns probabilities to language units

5. Major advocates: Shannon, Norvig

Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism

○ Chomsky○ Minsky○ Pierce

● Reasons for the Problem and Solutions

History of NLP

1950s: Empiricism● Empiricism dominated across several fields● Words were classified on the basis of their

co-occurrence with other words (“You shall know a word by the company it keeps” - Firth, 1957)

1970s: Rationalism● Several authors such as Chomsky, Minsky,

etc criticized the Empirical approach● Failure of the Empirical approach led to

funding cutbacks (“winters”)○ 1966: Machine Translation Failure○ 1970: The abandonment of connectionism○ 1971-75: Speech Recognition Failure

1990s: Empiricism● Large amounts of data became available● Several specialized problems could be

solved by statistical frameworks, without concentration on the general problems

2010s: Rationalism?● Most of the low-hanging fruit has been

picked up● But the original criticisms of the empirical

approach are still as valid

Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism

○ Chomsky○ Minsky○ Pierce

● Reasons for the Problem and Solutions

Objections to Empiricism● Several common empirical frameworks were

opposed by rationalists in the 70s, including:○ Linear Separators (Machine Learning)○ Vector Space Model (Information Retrieval)○ n-grams (Language Modeling)○ HMMs (Speech Recognition)

● Many of these are mere approximations of complex phenomena

Chomsky’s Objections● n-gram Language Modeling● Finite State Methods

Chomsky’s Objections:n-gram Language Modeling● Chomsky showed that n-grams cannot learn

long-distance dependencies (dependencies spanning more than n words)

● For practical purposes ‘n’ needs to be a small value (3 or 5)

● However, such small values fail to capture several interesting facts

Chomsky’s Objections:Finite State Methods● Examples of Finite State Methods include

○ Hidden Markov Models (HMMs)○ Conditional Random Fields (CRFs)

● Finite State Methods can capture dependencies beyond n words

● However, they may require infinite memory to process certain sentences

Chomsky’s Objections:Center Embedded Grammars● A center embedded grammar is of the form:

○ A -> x A y● Chomsky proved that a center embedded

grammar will require infinite memory and thus cannot be handled by finite state methods

● Center embedding is common in English, for example:○ A man that a woman that a child that a bird that I

heard saw knows loves

Minsky’s Objection● Linear Separators

Minsky’s Objections:Perceptrons● Minsky showed that perceptrons (and linear

separators in general) cannot learn functions that are not linearly separable such as XOR.

Minsky’s Objections:Perceptrons● This has implications for several tasks

including:○ Word Sense Disambiguation○ Information Retrieval○ Author Identification○ Sentiment Analysis

● For instance, this is the reason why sentiment analysis ignores loaded terms

Minsky’s Objections:Sentiment Analysis● Loaded terms can be either positive or

negative depending on whom it is addressed to. This is an XOR dependency:

Loaded Term Addressed to us Sentiment

Positive Y Positive

Positive N Negative

Negative Y Negative

Negative N Positive

Pierce’s Objections● Evaluation by Demos● Pattern Matching

Pierce’s Objections:Evaluation by Demos● According to Pierce, evaluation of projects

should be based on scientific principles rather than laboratory demos.

● Projects give good results in laboratory conditions, but have much higher error rates in real-world conditions.

Pierce’s Objections:Pattern Matching● Pierce stated that pattern matching is “artful

deception”, i.e. it is based on heuristics rather than scientific theory.

● Examples:○ The ELIZA effect○ The Chinese Room argument

Pierce’s Objections:Pattern Matching● While pattern matching produces better

results in the short term, it does so only by ignoring real scientific questions.

● While ambitious approaches may require time to deliver, they are backed by hard science.

Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism

○ Chomsky○ Minsky○ Pierce

● Reasons for the Problem and Solutions

Reason for the Oscillations:Gaps in Teaching● The “losing” side of the debate (currently

Rationalism) is never mentioned in textbooks/courses

● Leads to “reinventing the wheel” by each generation of NLP researchers

Reason for the Oscillations:Gaps in Teaching● Currently most courses concentrate on

Statistical methods, ignoring linguistic and scientific questions

● This prepares students only for “low-hanging fruit” but not the real scientific questions

Solution● Introduce the following in NLP courses:

○ Syntax○ Morphology○ Phonology○ Phonetics○ Historical Linguistics○ Language Universals

● Create parallels between computational linguistics and formal linguistics

Solution● Teach both sides of the rationalism vs.

empiricism debate● Educate students about the challenges

ahead of the “low-hanging fruit”

Major References● A Pendulum Swung Too Far by Kenneth

Church, 2001

Other References● Papers In Linguistics 1934-1951 by JR Firth, 1957● Syntactic Structures by Noam Chomsky, 1957● Whither Speech Recognition by John Pierce, 1969● ELIZA - A Computer Program for the Study of Natural

Language Communication between Man and Machine by Joseph Weizenbaum, 1966

● Minds, Brains and Programs by John Searle, 1980