Connectionist Models of Development

Jeff Elman

Department of Cognitive Science

UCSD

• Today’s class:

– What biology can do

– What learning can do

The issue: Nature vs. Nurture

Two possible ways to control

development. . .

More DNALess DNA

pyramidal cells

mossy cells

muscle cells

sperm cells Purkinje cell

Genetic conservatism

from butterfly host from alderfly host

Trichogramma (wasp)

from butterfly host from alderfly host

Trichogramma (wasp)

• Lesson 1: no “special purpose” genes

• Lesson 2: genetic conservatism

• Lesson 3: change through “tweaks & twiddles”

• Lesson 4: the importance of the environment

Modeling learning with “neural networks”

The first computers. . .

Walter Pitts

e1

en

in

i1

“Every digital computing

device contains certain relay

like elements with discrete

equilibria. Such an element

has two or more distinct states

in which it can exist

indefinitely...It is worth

mentioning that the neurons

of the higher animals are

definitely elements in the

above sense... Following W.

Pitts and W.S. McCulloch...we

ignore the more complicated

aspects of neuron

functioning.”

John von Neumann

The Colossus (one of ten)Bletchley Park, 1944

Is the human brain a computer?

• CPU and memory are separate

• the processor is very fast

• serial operation

• very reliable parts; but the whole computer may break easily

• computers are programmed

• the memory and CPU are the same

• the parts (neurons) are very slow

• parallel; enormous interconnectivity

• unreliable parts; but the whole brain is very reliable

• brains learn

Things digital computers do well

Things human brains do well

• Numeric computation

• Symbolic computation

• Repetitive operations

• Follow instructions

• Find subtle patterns (intuition?)

• Find novel solutions by analogy

• Learn

x = (.5 + 3.9) cos(31.2) - b cos((44.3/41 + 1)7)

ELIZA and the problem of natural language processing

the problem with SCRIPTS

Eliza

Eliza: Hello. I am Eliza. How can I help you?

Me: You can’t help me at all.

Eliza: You like to think I can’t help you, do you not?

Me: I am sure of it.

Eliza: How long have you been sure of it?

Me: Ever since I was 10 years old.

Eliza: Why do you tell me you were 10 years old just now?

Me: You are not a real person

Eliza: Does it please you to believe I am not a real person?

Scripts, Schemata, and Frames

John went to McDonald's with Mary.

They ordered a hamburger and fries.

John gave the woman at the counter $3.49.

They left.

• What is McDonalds?• Why did John and Mary go there?• Why did John give the woman money?• Did they eat anything?

John went to McDonald's restaurant with Mary.

They ordered a hamburger and fries.

Suddenly, Mary's husband Lenny burst in with a shotgun.

John hid under a table.

• Why did John and Mary go to McDonalds?

• Why did John hide under the table?

• Did John get to eat his Big Mac?

• What were they doing before going to McDonalds?

• Where will Mary’s husband spend the rest of his life?

• fragile (vs. graceful degradation)– fixed repertoire– limited to well-defined domains

• how to get the knowledge in the system

What is a neural network?

Flash Player Movie

Learning

vs.

Programming

13 ?

Input Output

2 4

7 49

4 16

fever, runny nose, muscle ache -> flu

fever, runny nose -> ?

Input Output

no fever, runny nose -> allergies

no fever, skin rash -> staphylococcus infection

“ought” ->

“ouch” ->

“tough” ->

“through” ->

“though” ->

“plouty” -> ?

“plough” -> ?

How do you pronounce “ou”?

“aw”

“au”

“uh”

“oo”

“oh”

“The Voringian binx glorphed the Knappoboor.”

1. Learning to read out loud

2. Discovering where the words

are

3. Discovering categories

input

h idden

output

My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.My grandmother lives near us. I like to visit my grandmother.

[first stages of learning]

“I like to go to my grandmother’s house. Well…because

she gives us candy. Well ... and we eat there sometimes.

Sometimes we sleep overnight there. Sometime when

I got to go to my cousin’s ...”

A (surprisingly) hard problem:

Where are the words?

Whereareth es il ens es b et w eew or d sWhereareth es il enc es b et w eenw or d s

“Many years ago, a boy and girl

lived in a castle by the sea.

They played with a dragon….”

manyyearsagoaboyandgirllivedina

castlebytheseatheyplayedwithadr

agon

00011 10111 00010 11011 11011

00100 10111 01111 01000 10111

11000 10010 10111 . . .

INPUT:

OUTPUT:

time

Task: Predict the next input

m

a

a

n

n

y

y

y

y

e

e

a

a

r

r

s

s

a

a

g

. . .

(m)

(a)

(n)

(y)

(y)

(e)

(a)(r)

(s)

(a)

(g)

(o)

(a)

(b)

(o)

(y)

(a)

(n)

(d)

(g)

(i)

(r)

(l)

(l)

(i)

(v)

(e)

(d)

(b)

(y)

(t)

(h)

(e)

(s)

(e)(a)

(t)

(h)

(e)

(y)

(p)

(l)

(a)

(y)

(e)(d)

(h)

(a)

(p)(p)

(i)(l)

(y)

(m)

0

0.5

1

1.5

2

2.5

3

3.5

Err

or

Time

Word learningstatistical learning

Saffran, Aslin, Newport, 1996

• 8 mo. old infants

• Passive exposure to 2 minutes of artificial

nonsense language

• Then present “words” vs. “non-words”

• Infants listened more to novel “non-words”

pabikulatidorepabikutalikulatidopabikulilitalatidotupabiku

0000000000000000000000000000010

0000000000000000000000000010000

0000000000000000000001000000000

0000010000000000000000000000000

0000000000000000000100000000000

0000000000000000100000000000000

0001000000000000000000000000000

0000100000000000000000000000000

0100000000000000000000000000000

0000000000000000000100000000000

0000000000001000000000000000000

0000000000100000000000000000000

0010000000000000000000000000000

0000000010000000000000000000000

0000000000000000000100000000000

0000000000000000001000000000000

0000000000000000001000000000000

0000000000000000000100000000000

1000000000000000000000000000000

0000000000000000000000000010000

0000000000000000000001000000000

0000010000000000000000000000000

0000000000000000000100000000000

0000000000000000100000000000000

0001000000000000000000000000000

0000100000000000000000000000000

0100000000000000000000000000000

0000000000000000000100000000000

0000000000001000000000000000000

0000000000100000000000000000000

0010000000000000000000000000000

0000000010000000000000000000000

0000000000000000000100000000000

0000000000000000001000000000000

0000000000000000001000000000000

0000000000000000000100000000000

1000000000000000000000000000000

time

(woman)

(smash)

(plate)

(cat)

(move)

(man)

(break)

(car)

(boy)

(move)

(girl)

(eat)

(bread)

(dog)

(move)

(mouse)

(mouse)

(move)

(book)

(smash)

(plate)

(cat)

(move)

(man)

(break)

(car)

(boy)

(move)

(girl)

(eat)

(bread)

(dog)

(move)

(mouse)

(mouse)

(move)

(book)

Input Output (prediction)

.

.

bo

yg

irl

cat

dog

boo

kro

ckse

ese

es

hea

rhe

ars

catc

hca

tch

esw

ho

tha

tw

hich .

CURRENT INPUT

PREDICTED NEXT WORD

etc.

etc.

bo

yg

irlca

td

og

bo

okro

ckse

ese

es

hea

rhe

ars

catc

hca

tche

sw

hoth

at

wh

ich

.

CURRENT WORD

PREDICTED NEXT WORD

NOUNS

VERBS

DO absent

DO optional

DO obligatory

small

big

edible

breakable

ANIMATES

INANIMATES

HUMANS

ANIMALS

man

woman

child

girl

dogcat

bird

paper

book

touch

be

drink

grab

seethink

eat

The vocabulary burst

milk

bottle

bed

chair

kitty

doggie

cookie

candyhorse

birdmommy

daddyJane

go

run

see

man

drink

Do you want to eat a MACAROON?

“Critical mass effect”:

A critical number of words must be

learned before categories, concepts,

and relationships will become apparent.

Once that number is learned…things

take off.

So where does language come from?

Nature? Nurture?

A new machine built out of old parts

bonobo macaque human song bird termite

Control over respiration

Control over articulators

Sequencing

Memory

Sociability

Auditoryprocessing

Imitation

Predictive learning

Lang

uage

is

poss

ible