Language Mind and Brain: The Unification Problem Colin Phillips Cognitive Neuroscience of Language...

Language Mind and Brain:The Unification Problem

Colin PhillipsCognitive Neuroscience of Language Laboratory

Department of LinguisticsUniversity of Maryland

• Objective is to bridge gap between linguistic models, real-time models of mental processes, and brain-level models, i.e., focus is on the Unification Problem

• Cognitive Neuroscience of Language necessarily draws on disparate areas - not just looking at pictures of brains!

Unification Problem

??

Unification Problem

Unification Problem

• No longer concerned with questions of whether linguistics is a natural science

• Language is clearly a remarkable specialization of human neurobiology (e.g., a sophisticated symbolic, recursive system, with fixed and parameterized aspects)

• What is it about human psychology & neurobiology that allows it to support the things that we know about language?

• In addressing this question, are we faced with problems or with mysteries?

Unification Problem

• ...And if there are mismatches between theories at different levels, whose problem is this?

Linguists?

Psychologists?

Neuroscientists?

Unification Problem

On mismatches between cognitive and neural models

“The more we learn about the brain,the greater the disanalogy becomes.”

(A philosophy of (neuro-)science talk, October 2001)

Unification Problem

On mismatches between cognitive and neural models

“If language is unlike anything else in the biological world,…then too bad for biology!”

(linguist, often accompanied by story about chemistry & physics)

Encoding & Computation

• Two main issues– How are linguistic representations encoded?– How are linguistic representations computed?

Sensory Maps

Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area.

Notions of sensory maps may be applicable to human phonetic representations…

…although attempts to find them have had little success to date.

Vowel Space

Encoding of Symbols: Abstraction

• But most areas of linguistics (phonology, morphology, syntax, semantics) are concerned with symbolic, abstract representations,

...which do not involve internal representations of dimensions of the outside world.

…hence, the notion of sensory maps does not get us very far into language

Computation: Discrete Infinity

• In neuroscience there are many findings about long-term storage of object representations (e.g., edges, faces, grandmothers, toothbrushes, …).

• Such representations are always finite in number, which can be retrieved from long-term memory

• BUT, much of what interests us in linguistics is infinite

• If representations are drawn from an infinite set, they cannot be retrieved from long-term memory; they must be constructed on-line, as needed - this poses a different kind of challenge …

Overview of Talks

Overview of Talks

1. The Unification Problem

Overview of Talks


2. Building Syntactic Relations

In-situ

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8

Region

Reading Time

DeclC

QP

どの生徒に…

Overview of Talks



3. Abstraction: Sounds to Symbols

In-situ

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8

Region

Reading Time

DeclC

QP

どの生徒に…

Overview of Talks



3. Abstraction: Sounds to Symbols

4. Linguistics and Learning

In-situ

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8

Region

Reading Time

DeclC

QP

どの生徒に…

with help from ...University of Maryland

Shani AbadaSachiko Aoshima

Daniel Garcia-PedrosaAna Gouvea

Nina KazaninaMoti LiebermanLeticia PablosDavid PoeppelBeth RabbinSilke Urban

Carol Whitney

University of Delaware

Evniki EdgarBowen HuiBaris KabakTom Pellathy

Dave SchneiderKaia Wong

Alec Marantz, MITElron Yellin, MIT

National Science FoundationJames S. McDonnell Foundation

Human Frontiers Science ProgramJapan Science & Technology Program

Kanazawa Institute of Technology

Outline

• The Challenge• Real-time Grammar• Accurate Parsing• Incremental Parsing• Mapping onto the Brain: Electrophysiology• Encoding• Outlook

Discrete Infinity

• Linguistic CreativityAbility to make infinite use of finite means

• The finite and infinite aspects of this system present rather different challenges for explicit linking hypotheses

Discrete Infinity


• Lexical entries, Argument Structure templates• Widespread assumption: same representations

accessed in comprehension, production, acceptability ratings, etc.

• Learner’s task is to construct a single lexical entry that covers all of these tasks

Discrete Infinity


• Sentence structures• Structures must be assembled, cannot simply be

retrieved from memory• Widespread assumption: multiple different systems

responsible for structure assembly in comprehension, production, acceptability, etc.

• Learner must master a number of different systems

Discrete Infinity


• Time-independent vs. Time-dependent systems

Standard View

324697+ ?

217 x 32 = ?

arithmetic

Standard View

324697+ ?

217 x 32 = ?

specialized algorithmspecialized algorithm

arithmetic

Standard View

324697+ ?

217 x 32 = ?


?

something deeper

arithmetic

Standard View


speaking understanding

grammaticalknowledge,competencelanguage

Standard View



grammaticalknowledge,competence

precisebut ill-adapted toreal-time operationlanguage

Standard View



grammaticalknowledge,competence

well-adapted toreal-time operationbut maybe inaccuratelanguage

If speaking and understanding involve different systems, there must be an additional store of knowledge that encodes what is shared between speaking and understanding.

As soon as we assume constructs such as ‘parsing strategies’, we are adopting task-specific mechanisms, and endorsing something like the standard architecture.

Standard View



grammaticalknowledge,competencelanguage

speaking,understanding,grammaticality

Analysis-by-Synthesis

• Sentences are understood by internally generating a representation that matches the input

• No separate time-independent system of knowledge

• We know that humans have a real-time system for linguistic computation - issue is whether that’s all there is

But wait a minute...

• Wasn’t this all shown to be wrong long ago?

(Fodor, Bever & Garrett 1974; Levelt 1974; Fillenbaum 1971; etc., etc.)?

• And a recent commentary:

“In this desert of ignorance there have been attempts to resurrect earlier claims that the grammar and the parser are one and the same thing. … The enterprise is misconceived … probably incoherent.” (Smith, 1999)

Motivation for Standard Architecture

• How to constrain hypothesis generation• Grammars are not incremental real-time systems• Evidence for input/output-specific strategies• Analysis-by-synthesis implies active generation,

ahead of input• Reputation of performance systems as fast but

inaccurate• Debunking of ‘Derivational Theory of Complexity’

• Time-dependent system of computation makes it feasible to generate testable linking hypotheses

Motivation for Alternative Architecture

Outline


Linear Order and Constituency

Linguistic Inquiry, 2003

Incremental Structure Building

• Evidence that sentence structures can only be assembled in a left-to-right derivation.

John

said

that

he

ate the entire pizza

S

NP VP

S’

Comp

V

S

NP VP

V NP

John

said

that

he

ate the entire pizza

S

NP VP

S’

Comp

V

S

NP VP

V NP

Constituents

• Many tools used to diagnose groupings of words:

– coordination

– deletion

– interpretation (coreference)

– movement, focus, topicalization

– etc.

• There are many cases where the tools converge on the same result

• There are also many cases where the tools yield conflicting results


A

(Phillips 2003)


A B

(Phillips 2003)


A

B C

(Phillips 2003)


A

B

C D

(Phillips 2003)


A

B

C

D E

(Phillips 2003)


A B

(Phillips 2003)


A B constituent

(Phillips 2003)


A

B Cconstituent is destroyed by addition of new material

(Phillips 2003)


A

B C

(Phillips 2003)


A

B Cconstituent

(Phillips 2003)


A

B

C Dconstituent is destroyed by addition of new material

(Phillips 2003)


the cat

(Phillips 2003)


the cat sat

(Phillips 2003)


the cat

sat on

(Phillips 2003)


the cat

sat

on the rug

(Phillips 2003)


the cat

sat on

(Phillips 2003)


the cat

sat

on the rug

(Phillips 2003)


the cat

sat

on the rug

[sat on] is a temporary constituent, which is destroyed as soon as the NP [the rug] is added.

(Phillips 2003)


Conflicting Constituency Tests

Verb + Preposition sequences can undergo coordination…

(1) The cat sat on and slept under the rug.

…but cannot undergo pseudogapping (Baltin & Postal, 1996)

(2) *The cat sat on the rug and the dog did the chair.

(Phillips 2003)


the cat

sat on

(Phillips 2003)


the cat

sat on slept under

and

(Phillips 2003)


the cat

sat on slept under

and

coordination applies early, before the V+P constituent is destroyed.

(Phillips 2003)


the cat

sat on

(Phillips 2003)


the cat

sat

on the rug

(Phillips 2003)


the cat

sat

on the rug

and the dog did

(Phillips 2003)


the cat

sat

on the rug

and the dog did

pseudogapping applies too late, after the V+P constituent is destroyed.

(Phillips 2003)


• Constituency ProblemDifferent diagnostics of constituency frequently yield conflicting results

• Incrementality Hypothesis(a) Syntactic processes see a ‘snapshot’ of a derivation - they target constituents that are present when the process applies(b) Conflicts reflect the fact that different processes have different linear properties

start

Ellipsis blocks Scope/Binding

Bill read all the books in a week (ambiguous: collective/distributive scope)

…and Sue did in a month (unambiguous: collective scope only)

Bill read as many books as Sue did in a week. (ambiguous)

Bill read as many books in a week as Sue did in a month. (unambiguous)

Ellipsis blocks Scope/Binding

John gave books to the children on Tuesday

…and Mary did on Thursday

John gave books to the children on each other’s birthdays

*…and Mary did on each other’s first day of school

Japanese

John

gave

books

to the children

on each other’s birthdays

VP

V

VP

V

VP

IP

John

+fin

I’

read

all the books

in a week

VP

V

VP

PP

IP

John

+fin

I’

gave

books

to the children


VP

V

VP

V

VP

IP

IP

John

+fin

I’

and

VP

IP

Bill

did

I’

gave

books

to the children

VP

V

VP

IP

John

+fin

I’

gave

books

to the children

on each other’s birthdaysVP

V

VP

VP

IP

John

+fin

I’

gave

books

to the children


V

VP

VP

IP

John

+fin

I’

IP

and

VP

IP

Bill

did

I’

gave

books

to the children


V

VP

VP

IP

John

+fin

I’

IP

and

VP

IP

Bill

did

I’

gave

books

to the childrenV

VP

gave

books

to the children


V

VP

VP

IP

John

+fin

I’

IP

and

VP

IP

Bill

did

I’

gave

books

to the childrenV

VP

on each other’s first day of schoolVP

Movement & Binding

a. John gave books to them on each other’s birthdays.

Movement & Binding

a. John gave books to them on each other’s birthdays.

gave

books

to them


VP

V

VP

V

VP

(Pesetsky 1995)

Movement & Binding

b. …and [give books to them] he did ___ on each other’s birthdays

(Pesetsky 1995)

give

books

VP

V

VP

to them

give

books

VP

V

VP

to them

IP

IP

he did

give

books

to them

VP

V

VP

give

books

VP

V

VP

to them

IP

IP

he

did

I’

constituentmovement

give

books

to them


VP

V

VP

V

VP

give

books

VP

V

VP

to them

IP

IP

he

did

I’

constituentmovement

give

books

to them


VP

V

VP

V

VP

give

books

VP

V

VP

to them

IP

IP

he

did

I’

constituentmovement

binding underc-command

Interim Conclusion

• By building syntactic structures from left-to-right we can explain a number of otherwise mysterious constituency phenomena (see Phillips 1996, 2003 for more examples; see Richards 1999, 2002 for some applications to Japanese)

• We knew independently that humans have a left-to-right structure-building system (i.e. parser, producer)

• Possibility arises that the incremental left-to-right system is the only structure-building system that humans have

• Other arguments leading to related conclusions about grammar, in widely varying formalisms: Kempson et al. (2001), Steedman (2000), Kempen (1999), Milward (1992, 1994)

Outline


Grammatical Accuracy

• It is not enough to show that syntactic structure-building looks like a real-time process

• If the real-time system is the only system, then it should also show the syntactic sophistication normally associated with the grammar - the parser cannot be ‘dumb’

• Question: does the parser access only grammatically legal structural analyses?

Beyond Ambiguity

• Much of parsing literature focuses on issues of ambiguity,

i.e. when 2 structures are grammatically possible, how to choose the right one?

• More basic question: grammatical search

i.e. how do we figure out if a sequence of words has any grammatical analyses?

Example: Argument Structure

Dative and Double-Object Constructions

Alternator Verb: give

The millionaire gave the painting to the museum.

The millionaire gave the museum the painting.

Non-alternator Verb: donate

The millionaire donated the painting to the museum.

*The millionaire donated the museum the painting.

(Phillips, Edgar & Kabak, 2000)


A Severe Garden-Path Sentence


The man gave the boy the dog bit a cookie




The man gave [the boy [the dog bit]] a cookie





availability of double-object parseleads to difficulty at embedded verb





Non-Alternator Verb: donate

The man donated [the boy [the dog bit …

availability of double-object parseleads to difficulty at embedded verb

difficulty should ariseearlier in the sentence

Results (partial)

G

G GG

G

G

G

G

G

G

GG

G

G

G

G

G

G

B

B

B

B B

B

BB

BB

B

B

B

B

B

B

B

B

B

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

-100

-50

0

50

100

150

Region

G alternators, ambig

B alternators, unambig

NP1 NP2 verb

C

C

C

C

C

C

C

C C

C

C

C

C

C

C

C

C

C

H

H

H

HH

H

H

H

H H

H

H

H

H

H

H

H

H

H

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

-100

-50

0

50

100

150

Region

C datives, ambig

H datives, unambig

NP1 NP2 verb

Relative to unambiguous control sentence, readers get intodifficulty at V with alternator verbs, at NP2 with non-alternators.--> Argument structure constraint immediately active on-line

Alternators (e.g. give) Non-Alternators (e.g. donate)

start

Example: Movement Constraints

Grammatical Accuracy in Parsing

Wh-Questions

Englishmen cook wonderful dinners.


Wh-Questions

Englishmen cook what


Wh-Questions

What do Englishmen cook


Wh-Questions

What do Englishmen cook gap


Long-distance Wh-Questions

Few people think that anybody realizes that Englishmen cook wonderful dinners



Few people think that anybody realizes that Englishmen cook what



What do few people think that anybody realizes that Englishmen cook gap


The plan to remove the equipment ultimately destroyed the building.



Direct Object NP Direct Object NP




Main Clause




Main Clause

Subject NP

Embedded Clause


What did the plan to remove the equipment ultimately destroy


What did the plan to remove the equipment ultimately destroy gap


What did the plan to remove ultimately destroy the building


What did the plan to remove gap ultimately destroy the building


What did the plan to remove gap ultimately destroy the building

Subject

Island Constraint

A wh-phrase cannot be moved out of a subject.

Question…

• Do people respect island constraints on movement immediately on-line?

(tomorrow’s talk)

Outline


Incrementality

• Question: is structure building immediate?

Does it operate on a word-by-word level?

…Even in a language where this may be hard?

Incremental Application of Binding Constraints in Japanese

Sachiko AoshimaColin Phillips

Amy Weinberg

Structure-building in Japanese

• NP-wa NP-ni [NP-ga NP-o V] V

• Head-driven Parsing (e.g. Pritchett 1991, Mulders 2002)Structure-building is delayed until verbs are processed– explains how parsing is possible in Japanese

– accounts for flexibility, limited garden-paths in Japanese

• Incremental ParsingStructure-building occurs immediately– accounts for native-speaker intuition of continuous comprehension

– hard to demonstrate experimentally

John-ga …

(Mazuka & Itoh 1995)

John-ga Mary-ni …


John-ga Mary-ni ringo-o …


John-ga Mary-ni ringo-o tabeta …


John-ga Mary-ni ringo-o tabeta inu-o ageta


Verb Surprise Effects

• Surprise effect appears at verb

Can be explained by both head-driven and incremental theories (Schneider 1999, Mulders 2002)

• Better: evidence of structure-building that precedes the verb

e.g. immediate application of grammatical constraints

English

To which of his children did the man give a gift

(Aoshima, Phillips & Weinberg 2002)

English

To which of his children did the man give a gift

Which of his children gave the man a gift?


Japanese

which of his children (DAT) the man (NOM) …

which of his children (NOM) the man (DAT) …


Japanese pronoun and its antecedent

which of his children (DAT) the man (NOM) … his

which of his children (NOM) the man (DAT) …**??

which of his children (DAT)



Gender Mismatch

the woman

the woman

Gender Mismatch

which of his children (DAT) the man (NOM) … his

which of his children (NOM) the man (DAT) …**??

which of his children (DAT)

the womanGender mismatch

the womanGender mismatch irrelevant

Conditionsa. Scrambled - Gender Mismatch

Adverb / [his / which NP]-dat / Adverb / NP FEMALE-nom / Adverb / NP-acc /

verb-Q / NPMALE-top / verb

b. Scrambled - Gender Match

Adverb / [his / which NP]-dat / Adverb / NP MALE-nom / Adverb / NP-acc /

verb-Q / NPFEMALE-top / verb

c. Non-scrambled - Gender Mismatch

Adverb / [his / which NP]-nom / Adverb / NP FEMALE-dat / Adverb / NP-acc /

verb-Q / NPMALE-top / verb

d. Non-scrambled - Gender Match

Adverb / [his / which NP]-nom / Adverb / NP MALE-dat / Adverb / NP-acc /

verb-Q / NPMALE-top / verb.

Examples

a. 台所で　彼の　どの子供に　朝食後　叔母が　急いで　お弁当を　渡したか　父親は　覚えていた。

b. 台所で　彼の　どの子供に　朝食後　叔父が　急いで　お弁当を　渡したか　叔母は　覚えていた。

c. 台所で　彼の　どの子供が　朝食後　叔母に　急いで　お弁当を　渡したか　父親は　覚えていた。

d. 台所で　彼の　どの子供が　朝食後　叔父に　急いで　お弁当を　渡したか　父親は　覚えていた。

Design & Procedure

• 2 X 2 factorial design• 4 lists were created by distributing 24 items in a

Latin Square design• 56 filler sentences• Comprehension questions: matching a subject

with a predicate• Self-paced reading task -Moving Window -• 40 native speakers of Japanese

Self-paced reading task

----- --- --- ---- ---- --- ------ -------


どの子供に --- --- ---- ---- --- ------ -------


----- 叔母は --- ---- ---- --- ------ -------


----- --- 母親が ---- ---- --- ------ -------


----- --- --- ケーキを ---- --- ------ -------


----- --- --- ---- 焼いたと --- ------ -------


----- --- --- ---- ---- 台所で ------ -------


----- --- --- ---- ---- --- お手伝いさんに -------


----- --- --- ---- ---- --- ------ 知らせましたか。

Results: Scrambled conditions

Slowdown at mismatching NP is observed.

500

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8 9 10

scramble,match

scramble,mismatch

his/her

± Match F1(1, 39) = 8.6, p<.01; F2(1,23)=7.4, p<.01

Results: Non-scrambled conditions

Slowdown at mismatching NP only when NP is possible antecedent.

500

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8 9 10

unscr,match

unscr,mismatch

his/her

± Match FS<1

Summary: Experiment 3

NP-nom

Verb

HIS-WH

gap

Binding constraint application takes place in advance of the verb.

Wh-gap is posited in a simple clause.

start

Experiment 3 (off-line):Grammatical judgment test



**??

Experiment 3 (off-line): Stimuli

a. Non-wh, Non-scrambled

[His children]-nom Adv the man-dat

b. Non-wh, Scrambled

[His children]- dat Adv the man-nom

c. Wh, Non-scrambled

[Which of his children]-nom Adv the man-dat

d. Wh, Scrambled

[Which of his children]-dat Adv the man-nom

Experiment 3 (off-line):Design & Procedure

• 4 lists were created by distributing 32 items in a Latin Square design

• 16 items: same materials from online test, and 16 items: different from those in online test.

• 32 filler sentences

• Anaphoric relation judgment task

• 40 native speakers of Japanese, same individuals as the online test

Experiment 3 (off-line):Results

• Backwards anaphora are more allowed in scrambled conditions.

• It confirms that the binding facts underlying in the online test are correct.

1

1.5

2

2.5

3

3.5

4

4.5

5

Non-wh, Unscr Non-wh, Scr Wh, Unscr Wh, Scr

start

Grammatical Search and Reanalysis

David SchneiderColin Phillips

Journal of Memory & Language, 2001

• Previous study showed incremental application of grammatical constraints,

i.e. derivations operate on a word-by-word time-scale

• Next study: do real-time derivations show consistency,

i.e does structure-building keep to a single derivation?how does is grammar searched to find possible analyses?

Grammatical Search

the man

knows the womanNP

VP

V

NP

S

(Schneider & Phillips, 2001)

Grammatical Search

the man

knows the womanNP

VP

V

S

NP

likesV


Grammatical Search

the man

knows

the womanNP

VP

V

S

NP

likesV

S


Grammatical Search

the man

knows

the womanNP

VP

V

S

NP

likesV

S

It’s clear that this is the right conclusion, but it’s less clear how the system reaches this conclusion.


Grammatical Search

the man

knows the womanNP

VP

V

S

NP

likesV

Option 1: combine with a local NP, ignoring existing status of the NP.


Grammatical Search

the man

knows the womanNP

VP

V

S

NP

likesV

Option 2: search the structure for an NP subject that currently lacks a -role,i.e., focused search.


Grammatical Search

the man

knows the womanNP

VP

V

S

NP

likesV

Option 2: search the structure for an NP subject that currently lacks a -role,i.e., focused search.

This fails, so reanalysis is needed, … but only as a last resort operation.

the man

knows the womanNP

VP

V

S

NP

likesV

Test CaseIf there is a higher NP, currently lacking a -role, a focused search will find it.


the man

knows the womanNP

VP

V

S

NP

likesV

S’

NP

t

who



the man

knows the womanNP

VP

V

S

NP

likesV

S’

NP

t

who

STest CaseIf there is a higher NP, currently lacking a -role, a focused search will find it.


the man

knows the womanNP

VP

V

S

NP


S’

NP

t

who

An unconstrained search will not be affected by the presence of the higher NP.

likesV


the man

knows

the womanNP

VP

V

S

NP


S’

NP

t

who


likesV

S


the man

knows

the womanNP

VP

V

S

NP


S’

NP

t

who


likesV

S

ProbeAntecedents for reflexives.


the man

knows

the womanNP

VP

V

S

NP


S’

NP

t

who


likesV

S


...the recipe herself

...the recipe himself





the man

knows the womanNP

VP

V

S

NP

likesV

S’

NP

t

who

S

...the recipe herself

...the recipe himself


Grammatical Search

Relative to its unambiguous control, high attached reflexives pose no difficulty.


Grammatical Search

Relative to its unambiguous control, high attached reflexives pose no difficulty.

…but low attached reflexives present great difficulty.


Therefore...High attachment is chosen

…despite well-known local-attachment biases.

the man

knows the womanNP

VP

V

S

NP

likesV

S’

NP

t

who

S

Grammatical search is focused, constrained by existing commitments


Therefore...High attachment is chosen

…despite well-known local-attachment biases.

Grammatical search is focused, constrained by existing commitments

knows the womanNP

VP

V

S

likesV

the manNP

This helps us to understand how grammatical search proceeds in simple cases.

Conclusions

• Local attachment is easy• The local reanalysis is easy (cf. Sturt et al. 2000)• So why is local attachment avoided?

Reanalysis is a Last Resort (even if it’s easy)

Reversal of results…

H

H

HH

H

H

H

H

H

H

H

H

B B

BB

B

B

B

B

B

B

B

B

C

C

C

C

C C

C

C

C

CC

CC

G

G

G

G

GG

G

G

G

G

G

GG

1 3 4 5 6 7 8 9 10 11 12 13 14

-100

-50

0

50

100

150

Sentence Region

H High Ambiguous

B High Unambiguous

C Low Ambiguous

G Low Unambiguous

verb reflexive

H

H H H HH

H

H

H

H H

H

B

B

B

B

B B BB

B

B

B

B

C

C

C C

C

C

C

C

C

C

C

C

C

G

G

G G

G

G

G

G

G

G

G

G

G

1 3 4 5 6 7 8 9 10 11 12 13 14

-100

-50

0

50

100

150

Sentence Region

H High Ambiguous

B High Unambiguous

C Low Ambiguous

G Low Unambiguous

verb reflexive

NP-biased

hear, warn,understand

S-biased

claim, believe,suspect

Outline


Time Resolution

• Syntax: phrase-by-phrase time scale

• Reading-time studies: word-by-word time scale

• Brain recordings: millisecond time scale

Event-Related Potentials (ERPs)

s1 s2 s3

John is laughing.

Evolving understanding of ERP components associated with language…

Semantically unexpected input

• She spread the warm bread with socks.(Kutas & Hillyard,

1980)

• She was stung by a fly.(Kutas, et al., 1984;

Federmeier & Kutas, 1999)

(Slide from Kaan (2001)

N400

• Negative polarity• peaking at around 400 ms• central scalp distribution


ERP Sentence Processing

• Developing understanding of N400 is informative

• Response to ‘violations’

N400

I drink my coffee with cream and sugarI drink my coffee with cream and socks

Kutas & Hillyard (1980)

Vervet Monkeys

• Many predators: leopard, lionhyena, jackal, eagle, etc. etc.

• Distinct alarm calls for different predators

ERP Sentence Processing

• Developing understanding of N400 is informative

• Response to normal sentences

N400

Fully CongruentMost new drugs are tested onwhite lab rats.

Van Petten & Kutas (1991)

ERP Sentence ProcessingN400

• N400 to semantic anomalies is a special case of a much more general phenomenon

• All words elicit N400-like response, timing and amplitude proportional to congruency, frequency, etc.

• More detailed understanding is contingent on more detailed models of semantic interpretation

Morpho-Syntactic violations

Every Monday he mows the lawn.

Every Monday he *mow the lawn.

The plane brought us to paradise.

The plane brought *we to paradise.(Coulson et al., 1998)


he mowshe *mow

P600


he mowshe *mow

P600

Left Anterior Negativity (LAN)


ERP Sentence ProcessingLAN, P600

Sie bereist dasneuter Landneuter …Sie bereist denmasculine Landneuter …she travels the land ... Gunter et al. (2000)

Kaan et al. (2000)

ERP Sentence ProcessingP600

Emily wondered who the performer in the concert had imitated for the audience’s amusement.Emily wondered whether the performer in the concert had imitated a pop star for the audience’s amusement.

• P600 reflects normal structure-building processes.

Electrophysiology of Wh-movement

Colin Phillips, Nina Kazanina, Shani Abada, Daniel Garcia-Pedrosa

[revision of work done at UDel in 2000]

Experiment DesignMaterials

a. The actress wished that the producers knew that the witty host would tell the jokes during the party.

b. The actress wished that the producers knew which jokes the witty host would tell __ during the party.

c. The producers knew that the actress wished that the witty host would tell the jokes during the party.

d. The producers knew which jokes the actress wished that the witty host would tell __ during the party.


b. The actress wished that the producers knew which jokes the witty host would tell during the party.


d. The producers knew which jokes the actress wished that the witty host would tell during the party.

Short wh-dependency



b. The actress wished that the producers knew which jokes the witty host would tell during the party.


d. The producers knew which jokes the actress wished that the witty host would tell during the party.

Long wh-dependency


-100 0 100 200 300 400 500 600 700 800 900 1000

-5

-4

-3

-2

-1

0

1

2

3

4

5

+wh

-wh

Electrode PZ Short Conditionsn=20

Effect of wh-movement significant (p<.01) from 300-400ms onwards

Verb

-100 0 100 200 300 400 500 600 700 800 900 1000

-5

-4

-3

-2

-1

0

1

2

3

4

5

+wh

-wh

Electrode PZ Long Conditionsn=20

Effect of wh-movement significant (p<.01) from 300-400ms onwards

Verb

start

How fast is Structural Computation?

Silke UrbanColin Phillips

Background

Early Left Anterior Negativity

(Angela Friederici, Anja Hahne, et al.)

Neville et al., 1991

The scientist criticized a proof of the theorem.

The scientist criticized Max’s of proof the theorem.

500ms/word

Hahne & Friederici, 1999

Das Baby wurde gefüttertThe baby was fed

Das Baby wurde im gefüttertThe baby was in-the fed

Question: are the brain responses to violations automatic?


P600


ELAN

ELAN

• Very fast: 150-250ms• Automatic• Left anterior (= frontal) scalp distribution• Elicited by a subclass of syntactic violations

“Phase 1 (100–300 ms) represents the time window in which the initial syntactic structure is formed on the basis of information about the word category.”

(Friederici 2002)

Questions about ELAN

• How plausible is it that ELAN reflects syntactic structure building?

– Speed: 150ms is faster than lexical access!

– Generality: ELAN is not elicited by most violations; almost all studies on ELAN involve one construction (for each of German, English)

– Localization…

Brodmann Areas

160 SQUIDwhole-headarray

pickup coil & SQUIDassembly

Magnetoencephalography (MEG)

(Friederici et al. 2000)

Two regions of interest


Anterior Temporal Lobe?

• Why is anterior temporal lobe so important in ELAN?

• How does it differ from Broca’s area (BA 44 etc.) that are implicated so often in other studies?

• Friederici: both responsible for ‘structure building’; BA44 also responsible for ‘syntactic working memory’; ‘the inferior portion of BA44 is selectively activated when syntactic processes are in focus.’

• Anterior temporal lobe associated with– lexical information

– activated in fMRI by comparisons of sentences with word lists

Alternative Interpretation

• ELAN reflects violation/suppression of automatic lexical prediction– accounts for localization to anterior temporal lobe

– accounts for very early timing

– might account for automaticity

– accounts for very limited distribution




NP

Max’s N


Das Baby wurde gefüttertThe baby was fed

Das Baby wurde im gefüttertThe baby was in-the fed

NP

the N

in

PP

Prediction

• If ELAN reflects violation of lexical prediction, rather than syntactic structure-building, then…

– change lexical predictions

– keep syntactic violation the same

– should ‘turn off’ ELAN brain response




NP

Max’s N

Possible to block the automatic prediction of an N following a possessor: ellipsis

Ellipsis

• Possessors may appear alone in ellipsis contexts:

Although I like Mary’s theory, I don’t like John’s.

Experimental Conditions

Although Erica kissed Mary’s mother, she did not kiss the daughter of the bride.

Although Erica kissed Mary’s mother, she did not kiss Dana’s of the bride.

Although the bridesmaid kissed Mary, she did not kiss the daughter of the bride.

Although the bridesmaid kissed Mary, she did not kiss Dana’s of the bride.

Experimental Conditions

Although Erica kissed Mary’s mother, she did not kiss the daughter of the bride.

Although Erica kissed Mary’s mother, she did not kiss Dana’s of the bride.

Although the bridesmaid kissed Mary, she did not kiss the daughter of the bride.

Although the bridesmaid kissed Mary, she did not kiss Dana’s of the bride.

ellipsis possible

ellipsis impossible

Experimental Design

• 384 sentences per session– 128 targets (drawn from 128 sets of 4 conditions)

– 64 items designed to elicit ‘agreement violation’ LAN

– 192 filler items, designed to hide violations and promote ellipsis

• Procedure– RSVP (Rapid Serial Visual Presentation), 500ms/word

– Grammaticality judgment task

• Recording: 32-electrode montage

• 22 subjects (so far)

Although

kissed

Mary’s

mother,

Dana’s

bride.

???

good bad

Preliminary Results

a. Although … Mary’s mother … Dana’s of …b. Although … Mary … Dana’s of …

b

a

Interim Conclusion

• Preliminary results lend support to our interpretation of the (E)LAN - the anterior negativity is reduced in an ellipsis context

– structural violation is identical in both conditions

– obligatory lexical prediction of N following possessor (e.g. Mary’s…) is absent in ellipsis context

• Structure-building may begin ~250-300ms after a word is presented

Outline


The Binding Problem

• Discrete infinity

Individual neurons (or groups of neurons) can store finite information about objects, words, etc.

But sentences are infinite in number!

• Representing structure: can’t just activate all words

e.g. THE + MAN + ATE + PIZZA

• Must create & discard structures quickly: 100s of msec

Temporal Binding

the

man

ate

pizza

phaselocked

phaselocked

~25ms, 40Hz

Evidence from Animals & Humans

• Direct recordings (cat) • EEG recordings (human)

(Singer 1999) (Tallon-Baudry & Bertrand 1999)

Limitations…

• No evidence yet of role in human syntax• Limited capacity - ~7 bindings• 1-level of hierarchy only

• Interesting hypothesis (Whitney & Weinberg 2002)– temporal binding is the neural representation used for the

‘syntactic workspace’

– additional neural encoding mechanism used for long-term representation and storage

Outline


Outlook• Overview

– challenge for unification: real-time hypotheses

• Real-time Grammar– syntactic derivations look like real-time derivations

• Accurate Parsing– real-time derivations have the sophistication that is needed

• Incremental Parsing– real-time interpretation is time-locked to incoming words

• Mapping onto the Brain: Electrophysiology– extreme time-precision of EEG/MEG can be linked to detailed

linguistic constructs

• Encoding– plausible models of neural encoding of structure are emerging

• Unification: Problem or Mystery…

www.ling.umd.edu/colin

[email protected]

Language Mind and Brain: The Unification Problem Colin Phillips Cognitive Neuroscience of Language...

Documents

Transcript of Language Mind and Brain: The Unification Problem Colin Phillips Cognitive Neuroscience of Language...