Broca’sarea

Parsopercularis

Motor cortex Somatosensory cortex

Sensory associativecortex

PrimaryAuditory cortex

Wernicke’sarea

Visual associativecortex

Visualcortex

[Adapted from Neural Basis of Thought and Language Jerome Feldman, Spring 2007, feldman@icsi.berkeley.edu

Language categories and concepts

Categories & Prototypes

Three ways of examining the categories we form: relations between categories (e.g. basic-level category) internal category structure (e.g. radial category) instances of category members (e.g. prototypes)

Furniture

Sofa Desk

leathersofa

fabricsofa

L-shapeddesk

Receptiondisk

Basic-Level Category

Superordinate

Subordinate

Basic-level -- Criteria

Perception – overall perceived shapesingle mental image fast identification

Concepts are not categorical

Radial Structure of Mother

The radial structure of this category is defined with respect to the different models

CentralCase

Stepmother

Adoptivemother

Birthmother

NaturalmotherFoster

mother

Biologicalmother

Surrogatemother

Unwedmother

Geneticmother

Action Words- an fMRI study

Somatotopic Representation of Action Words in Human Motor and Premotor CortexOlaf Hauk, Ingrid Johnsrude,and Friedemann

Pulvermuller*Medical Research Council, Cognition and Brain

Sciences Unit Cambridge, United KingdomNeuron, Vol. 41, 1–20, January 22, 2004,

Copyright 2004 by Cell Press

Traditional theory

Unified meaning center in the left temporal lobe.Connected to Wernicke’s areaExperiments on highly imageable words/nouns.

Vocalization and grammar associated with frontal lobeConnected to Broca’s area

Do action words activate the motor cortex Given: Cortical representations of the face, arm, and leg

are discrete and somatotopically organized in the motor and premotor cortex

Hypothesis: Words referring to actions performed with the face, arm, or leg would activate premotor networks. neurons processing the word form and those processing

the referent action should frequently fire together and thus become more strongly linked, resulting in word-related networks overlapping with motor and premotor cortex in a somatotopic fashion.

Experiment: An fMRI study with word stimuli from different effectors (face, arm, or leg). ROI based on movements (face, arm, leg)

Movement vs. Actions

Neural Evidence for category structure Are there specific regions in the brain to

recognize/reason with specific categories? No, but there are specific circuits

distributed over relevant regions of the brain.

What are schemas?

Regularities in our perceptual, motor and cognitive systems

Structure our experiences and interactions with the world.

May be grounded in a specific cognitive system, but are not situation-specific in their application (can apply to many domains of experience)

Basis of Image schemas

Perceptual systems Motor routines Social Cognition Image Schema properties depend

onNeural circuitsInteractions with the world

Image schemas

Trajector / Landmark (asymmetric) The bike is near the house ? The house is near the bike

Boundary / Bounded Region a bounded region has a closed boundary

Topological Relations Separation, Contact, Overlap, Inclusion, Surround

Orientation Vertical (up/down), Horizontal (left/right, front/back) Absolute (E, S, W, N)

LMTR

bounded region

boundary

Spatial schemas

TR/LM relation Boundaries, bounded region Topological relations Orientational Axes Proximal/Distal

TR/LM -- asymmetry

The cup is on the table ?The table is under the cup.

The skateboard is next to the post. ?The post is next to the skateboard.

Topological Relations

Separation Contact Coincidence:

- Overlap- Inclusion

- Encircle/surround

Orientation

Vertical axis -- up/down

up

down

above

belowupright

Relative frame of reference

frontback

left??

right??

Language and Spatial Schemas

People say that they look up to some people, but look down on others because those we deem worthy of respect are somehow “above” us, and those we deem unworthy are somehow “beneath” us.

Much of our language is rich with such spatial talk. Concrete actions such as a push or a lift clearly imply a

vertical or horizontal motion, but so too can more abstract concepts.

Metaphors: Arguments can go “back and forth,” and hopes can get “too high.”

Simulation-based language understanding

Analysis Process

SemanticSpecification

“Harry walked into the cafe.” Utterance

CAFE Simulation

Belief State

General Knowledge

Constructions

construction WALKEDform

selff.phon [wakt]meaning : Walk-Action constraints

selfm.time before Context.speech-time selfm..aspect encapsulated

Simulation specification

A simulation specification consists of:- schemas evoked by constructions- bindings between schemas

Language and Thought We know thought (our

cognitive processes) constrains the way we learn and use language

Does language also influence thought?

Benjamin Whorf argues yes

Psycholinguistics experiments have shown that linguistics categories influence thinking even in non-linguistics task

Language

Thought

cognitive processes

Natural Theory of Language (NTL)

• Basic concepts and words derive their meaning from embodied experience.

• Abstract and theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts.

• Structured Connectionist Models can capture both of these processes nicely.

• Grammar extends this by Constructions: pairings of form with embodied meaning.

Simulation-based language understanding

“Harry walked to the cafe.”

Schema Trajector Goalwalk Harry cafe

Analysis Process

Simulation Specification

Utterance

SimulationCafe

Constructions

General Knowledge

Belief State

The ICSI/BerkeleyNeural Theory of Language Project

Background: Primate Motor Control Relevant requirements (Stromberg, Latash, Kandel, Arbib,

Jeannerod, Rizzolatti) Should model coordinated, distributed, parameterized control programs

required for motor action and perception. Should be an active structure. Should be able to model concurrent actions and interrupts.

Model The NTL project has developed a computational model based on that

satisfies these requirements (x- schemas). Details, papers, etc. can be obtained on the web at http://www.icsi.berkeley.edu/NTL

Active representations Representation based on stochastic Petri nets captures

dynamic, parameterized nature of actions Many inferences about actions derive from what we know

about executing them Generative model: action, planning, recognition, language.

Walking:

bound to a specific walker with a direction or goal

consumes resources (e.g., energy)may have termination condition

(e.g., walker at goal) ongoing, iterative action

walker=Harry

goal=home

energy

walker at goal

Language Development in Children

0-3 mo: prefers sounds in native language 3-6 mo: imitation of vowel sounds only 6-8 mo: babbling in consonant-vowel segments 8-10 mo: word comprehension, starts to lose

sensitivity to consonants outside native language 12-13 mo: word production (naming) 16-20 mo: word combinations, relational words

(verbs, adj.) 24-36 mo: grammaticization, inflectional morphology 3 years – adulthood: vocab. growth, sentence-level

grammar for discourse purposes

Learning Spatial Relation Words Terry Regier

A model of children learning spatial relations.

Assumes child hears one word label of scene.

Program learns well enough to label novel scenes correctly.

Extended to simple motion scenarios, like INTO.

System works across languages.

Mechanisms are neurally plausible.

Learning Verb MeaningsDavid Bailey

A model of children learning their first verbs.Assumes parent labels child’s actions.Child associates action with wordProgram learns well enough to: 1) Label novel actions correctly 2) Obey commands using new words (simulation)

System works across languagesMechanisms are neurally plausible.

Motor Control (X-schema) for SLIDE

Parameters for the SLIDE X-schema

Training ResultsDavid Bailey

English 165 Training Examples, 18 verbs Learns optimal number of word senses (21) 32 Test examples : 78% recognition, 81% action All mistakes were close lift ~ yank, etc. Learned some particle CXN,e.g., pull up

Farsi With identical settings, learned senses not in English

Embodied Construction Grammar(Bergen, Chang & Paskin 2000) Assumptions from Construction Grammar

Constructions are form-meaning pairs(Lakoff 1987, Goldberg 1995)

Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal)

Constructions evoke and bind semantic schemas Additional influences

Cognitive Grammar (Langacker 1987) Frame Semantics (Fillmore 1982) Structured Connectionism (Feldman 1987)

Conclusion Language acquisition and use is a

hallmark of being human Language seems to rely on

fine-grained aspects of embodied (sensory-motor and social cognition) primitives and

brain-like computation (massively parallel, distributed, spreading activation, temporal binding).

Understanding requires imaginative simulation!

Sensory-Motor imagination and simulation is crucial in interpretation!