Effect of synchronous vs. non-synchronous recordingsShow movie then say that is an average and show a movie based on activity from a single trial (NOISY)Then say noise could look worse if correlated noise is bad.Compare shifted and unshifted population performanceCompare IC vs. A1

Effect of Neural Correlations on Speech Discrimination

Mike KilgardAssociate Professor

University of Texas at Dallas

Cosyne08 Workshop: Real-Time Processing and the Processing of Time

Manner of Articulation

Stops Fricatives Affricates Nasals Glides Liquids

Place o

f Articu

lation

Lip

sR

oo

fB

ack

30 kHz

20

10

Time (ms)182.5949 282.5949 382.5949 482.5949 582.5949 682.5949 782.5949 882.5949

0

1

2

3

x 104

0 200 400 600 ms

Pad

Tad

Kad Gad Shad Had

Dad

Bad Fad Vad

Sad Zad

Mad Wad

Chad Jad Nad

Yad

Lad

Rad

Manner of Articulation

Stops Fricatives Affricates Nasals Glides Liquids

Place o

f Articu

lation

Lip

sR

oo

fB

ack

30 kHz

20

10

Time (ms)182.5949 282.5949 382.5949 482.5949 582.5949 682.5949 782.5949 882.5949

0

1

2

3

x 104

0 200 400 600 ms

Pad

Tad

Kad Gad Shad Had

Dad

Bad Fad Vad

Sad Zad

Mad Wad

Chad Jad Nad

Yad

Lad

Rad

m/n r/l sh/s sh/ch sh/h sh/j sh/f d/t d/g d/b d/s 0

20

40

60

80

100

Consonant Discrimination Task

Per

cen

t L

ever

Pre

ss*********

Rats can discriminate most human speech sounds.

Data from 5 or 6 rats after ten days of training on a Go/No Go task.

Observation:

Rats can discriminate most speech sounds.

Question:

How are these sounds represented in the central auditory system?

Neurograms of 445 multi-unit recordings from anesthetized A1, 20 repeats

Neurograms of 445 multi-unit recordings from anesthetized A1, 20 repeats

N = 63 A1 multi-unit recording sites, average of 20 repeats

Movie of A1 responses from one rat

Observation:

Most speech sounds evoke unique spatiotemporal activity patterns in A1.

Question:

What is the relationship between neural responses and speech discrimination ability?

Euclidean distancebetween neurogramsis well correlated with behavior only when 1 ms bins are used.

N=445 A1 multi-unit recording sites

Observation:

Sounds that evoke dissimilar spatiotemporal activity patterns are readily discriminable.

Question:

Can A1 neurons discriminate speech sounds in a single trial?

Single Trial

N = 63 A1 multi-unit recording sites

0 10 20 30 401

20

1

20

95% (68%)

bad

dad

bad dad

0 10 20 30 401

20

1

20

100% (63%)

0 10 20 30 401

20

1

20

80% (60%)

0 10 20 30 401

20

1

20

75% (63%)95% correct 100% correct 80% correct 75% correct

PSTH-based Classifier (Foffani & Moxon, 2004; Schnupp et al., 2006)

Single trials matched to mean PSTH templatesusing Euclidean distance.

Triangles indicate classification errors.

DadBad

Ave

rage

PS

TH

’s

Bad

Dad

Sin

gle

Tria

l PS

TH

’s

0 10 20 30 401

20

1

20

95% (68%)

bad

dad

bad dad

0 10 20 30 401

20

1

20

100% (63%)

0 10 20 30 401

20

1

20

80% (60%)

0 10 20 30 401

20

1

20

75% (63%)95% correct 100% correct 80% correct 75% correct

DadBad

Ave

rage

PS

TH

’sS

ingl

e T

rial P

ST

H’s

Bad

Dad

Mean of 445 sites was 90±3% when 1-10 ms bins were used.

Classifier Performance using Spike Timing

0 10 20 30 401

20

1

20

95% (68%)

bad

dad

bad dad

0 10 20 30 401

20

1

20

100% (63%)

0 10 20 30 401

20

1

20

80% (60%)

0 10 20 30 401

20

1

20

75% (63%)

DadBad

Ave

rage

PS

TH

’s

Bad

Dad

Mean of 445 sites was 90±3% when 1-10 ms bins were used.

68% correct 63% correct 60% correct 63% correct Classifier Performance using Spike Number

Sin

gle

Tria

l PS

TH

’s

Neu

ral D

iscr

imin

atio

n

a) Onsetspike timing

b) Onsetmean rate

r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

50%

60%

70%

80%

90%

c) Full responsespike timing

mean rated) Full response

r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

Neural discrimination using one sweep of activity from individual multi-unit clusters recorded in A1

N=445 A1 multi-unit recording sites

Neu

ral D

iscr

imin

atio

n

a) Onsetspike timing

b) Onsetmean rate

r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

50%

60%

70%

80%

90%

c) Full responsespike timing

mean rated) Full response

r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

Observation:

A single trial of onset activity from A1 neurons recorded at a single site can discriminate speech sounds as well as rats.

Question:

Is this true in awake rats?

Awake Auditory Cortex

50 60 70 80 90

50

60

70

80

90

100

Classifier Percent Correct

Be

ha

vio

r P

erc

en

t C

orr

ec

t

R2=0.63, P=0.004

m/n

r/l

sh/ssh/ch

sh/h

sh/j

sh/f

d/t

d/g

d/bd/s

N = 41 A1 multi-unit recording sites

Question:

How good is speech discrimination if more or fewer A1 sites are used?

Observation:

A single trial of onset activity from awake A1 neurons recorded at a single site can discriminate speech sounds as well as rats.

1 4 16 1 4 16 64

50

60

70

80

90

100 P

erce

nt

Co

rrec

t

Number of Number of Single Unit Sites Multi-Unit Sites

d/sd/bd/gd/tsh/fsh/jsh/hsh/chsh/sr/lm/n

Neural Discrimination of English ConsonantsP

erce

nt C

orre

ct

Question:

Can A1 activity distinguish speech sounds from a larger set?

Observation:

A single trial of activity from several hundred A1 neurons can discriminate between pairs of English consonants with 100% accuracy.

1 4 16 1 4 16 64 256 0

10

20

30

40

50

60

70

80

90

100

Number of Number of Single Unit Sites Multi-Unit Sites

Pe

rcen

t C

orr

ect

1 ms bins5 ms bins20 ms bins40 ms bin

Neural Discrimination of 20 English Consonants

Per

cent

Cor

rect

Observation:

A single trial of activity from a set of ~400 A1 neurons can discriminate between 20 English consonants with 100% accuracy.

Question:

Is this an artifact of serial recordings?

10 2030 40501

20

1

20

95% (68%)

bad

dad

bad dad

10 20 30 40 501

20

1

20

100% (63%)

10 2030 40501

20

1

20

80% (60%)

10 2030 40501

20

1

20

75% (63%)95 % correct 100% correct 80% correct 75% correct

DadBad

Ave

rage

PS

TH

’sS

ingl

e T

rial P

ST

H’s

Bad

Dad

Mean of 445 sites was 90±3% when 1-10 ms bins were used.

DadBad

Ave

rage

PS

TH

’sS

ingl

e T

rial P

ST

H’s

Bad

Dad

50 100 150 2001

20

1

20

95 % correct

0 20 40 60 80 1000

20

40

60

80

100Discrimination of 20 Consonants

Pe

rce

nt c

orr

ect

usi

ng

SE

RIA

LL

Y r

eco

rde

d r

esp

on

ses

Percent correct using SIMULTANEOUSLY recorded responses-20 -10 0 10 20 30 400

5

10

15

20

Num

ber

of S

ets

of F

our

A1

Site

s

Improvement in Discrimination Caused by Serial Recordings

Neural Discrimination of 20 English Consonants

Observation:

A single trial of activity from a set of four simultaneously recorded A1 multi-unit clusters can discriminate between 20 English consonants almost as well as serially recorded sites.

Question:

Is this true in awake rats?

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Check duration/bins

-20 -10 0 10 20 30 400

2

4

6

8

10

12

14

16

18

Num

ber

of S

ets

of F

our

A1

Site

s

Improvement in Discrimination Caused by Serial Recordings

Observation:

A single trial of activity from a set of four simultaneously recorded A1 multi-unit clusters can discriminate between 20 English consonants almost as well as serially recorded sites.

Possible Explanation:

The readout mechanisms are optimized for categorization, not identification.

Manner of Articulation

Stops Fricatives Affricates Nasals Glides Liquids

Place o

f Articu

lation

Lip

sR

oo

fB

ack

30 kHz

20

10

Time (ms)182.5949 282.5949 382.5949 482.5949 582.5949 682.5949 782.5949 882.5949

0

1

2

3

x 104

0 200 400 600 ms

Pad

Tad

Kad Gad Shad Had

Dad

Bad Fad Vad

Sad Zad

Mad Wad

Chad Jad Nad

Yad

Lad

Rad

.05%

.5%

1%

5%

50%Van Rullen & Thorpe, 2001

Image reconstructions using only temporal order of the first spike of Retinal Ganglion Cell populations

Speech Conclusions1. Rats are able to accurately discriminate most speech sounds.

2. Consonants appear to be represented by onset firing patterns.

3. A single sweep of activity is sufficient to discriminate most sounds.

4. Responses in A1 are highly correlated with behavioral discrimination.

Tomorrow at ‘Linking Auditory Neurophysiology to Perception’

1. Speech in Noise 2. Non-Primary Auditory Cortex and Inferior Colliculus3. Vowel coding4. Categorization5. Plasticity

Acknowledgements:Crystal Engineer - Speech Training and A1 physiology Claudia Perez - Speech Training and Inferior Colliculus Jai Shetake - Awake A1 Speech Physiology

Figure 7

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Co

rrel

atio

n B

etw

een

Neu

ral a

nd

Beh

avio

ral D

iscr

imin

atio

n (

R2 )

50

60

70

80

90

100

Per

cen

t C

orr

ect

Onset 1 ms bins

Onset 40 ms bin

Duration 1 ms bins

Duration 700 ms bin

SixteenMulti-Units

OneMulti-Unit

OneSingleUnit

SixteenSingleUnits

SixteenMulti-Units

OneMulti-Unit

SixteenSingleUnits

OneSingleUnit

a) b)

Supplementary Figure 2

Neural Activity Patterns Predict Speech Discrimination Ability in Rats

Mike KilgardUniversity of Texas at Dallas

What is the relationship between neural responses and speech discrimination ability?

0 200 400 600 8000

50

100

150

200

250

300

350

Mean IC PSTH

40

30

20

10

Fre

quen

cy

(kH

z)

0 100 200 300 400 500 600Time (msec)

0 100 200 300 400 500 600Time (msec)

Sad

Firi

ng R

ate

(Hz)

N=187 sites, 6 rats

0 100 200 300 400 500 600

High

Medium

Low

40

30

20

10

Fre

quen

cy

(kH

z)

0 100 200 300 400 500 600Time (msec)

0 100 200 300 400 500 600Time (msec)

SadN=187 sites, 6 rats

Mean IC PSTH’s

Inferior Colliculus Neurograms N=187 sites, 6 rats

Spatial Temporal

N=441 sites, 11 rats A1 Neurograms

N=441 sites, 11 rats A1 Neurograms

0 100 200 300 400 0 100 200 300 400

Sad Dad

High

Medium

Low

Time (milliseconds)

vs.

0 100 200 300 400 0 100 200 300 400

Dad

0 100 200 300 400 0 100 200 300 400

Sador

Speech Discrimination by Rats

Easy!

0 100 200 300 400 0 100 200 300 400

Rad Lad

High

Medium

Low

Time (milliseconds)

vs.

Impossible

0 100 200 300 400 0 100 200 300 400

Bad Dadvs.

High

Medium

Low

Impossibleusing mean rateor Easy using spike timing?

Rat Consonant Discrimination

m/n r/l sh/s sh/ch sh/h sh/j sh/f d/t d/g d/b d/s 0

20

40

60

80

100

Consonant Discrimination Task

Per

cen

t L

ever

Pre

ss

*********

N=11 rats

N=441 A1 sites, 6 rats Onset Neurograms

10000 15000 20000 25000

50

60

70

80

90

100

m/nr/l

sh/s

sh/chsh/h

sh/j

sh/f

d/td/g

d/bd/s

R2=0.66, P=0.002

Euclidean Distance Between Neurogram Pairs

Be

ha

vio

r P

erc

en

t C

orr

ec

t

50 60 70 80 90 100

50

60

70

80

90

100

m/nr/l

sh/s

sh/ch

sh/h sh/j

sh/fd/t

d/gd/b

d/s

R2=0.66, P=0.002

Classifier Percent Correct

Be

ha

vio

r P

erc

en

t C

orr

ec

tR= 0.81 P= 0.002

Neural Discrimination Predicts Behavioral Discrimination

Speech Class

SpikeReadout

Inferior Colliculus

N=187

Primary Auditory Cortex

N=441

Awake Primary Auditory Cortex

N=40

Consonants Spike Timing 0.82 0.81 0.82

Mean Rate - - -

Vowels

Temporal Patterns

- - -

MeanRate

0.91 - 0.71

10000 15000 20000 25000

50

60

70

80

90

100

m/nr/l

sh/s

sh/chsh/h

sh/j

sh/f

d/td/g

d/bd/s

R2=0.66, P=0.002

Euclidean Distance Between Neurogram Pairs

Be

ha

vio

r P

erc

en

t C

orr

ec

t

50 60 70 80 90 100

50

60

70

80

90

100

m/nr/l

sh/s

sh/ch

sh/h sh/j

sh/fd/t

d/gd/b

d/s

R2=0.66, P=0.002

Classifier Percent Correct

Be

ha

vio

r P

erc

en

t C

orr

ec

t

Correlation Between Neural and Behavioral Discrimination

D

nVd

S

10

20

30

10

20

30

40

/ a / / ea / / u / / ee / / oo /

500100 200 300 400 6000 500100 200 300 400 6000 500100 200 300 400 6000 500100 200 300 400 6000 500100 200 300 400 6000

Time (ms)

40 kHz

Sad Sead Sud Seed Sood

Dad Dead Dud Deed Dood

10

20

30

40 kHz

Vowel Spectrograms

Sad Sead Sud Seed Sood

Dad Dead Dud Deed Dood

Inferior Colliculus Neurograms N=187 sites, 6 rats

a ea u ee oo

10 20 30 40 50 60 70 80 90

100Vowel Discrimination

Distance from CS+in F1-F2 space

Per

cent

Lev

er

Pre

ss

Be

havi

ora

l P

erf

orm

ance

Vowel Discrimination

N=14 rats

Speech Class

SpikeReadout

Inferior Colliculus

N=187

Primary Auditory Cortex

N=441

Awake Primary Auditory Cortex

N=40

Consonants Spike Timing 0.82 0.81 0.82

Mean Rate - - -

Vowels Spike Timing - - -

Mean Rate 0.91 - 0.71

Correlation Between Neural and Behavioral Speech Discrimination

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.824, P=0.00183

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.915, P=0.000532

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.562, P=0.0721

50 60 70 80 90 100

50

60

70

80

90

100B

ehav

ior

Per

form

ance

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.562, P=0.115

0 150 300 450 6000

100

200

300

400

500

0 150 300 450 6000

50

100

150

200

250

0 150 300 450 6000

50

100

150

200

250

Dad IC Dad A1 Dad A1 AwakeConsonant Spike Timing Consonant Rate Vowel Spike Timing Vowel Rate

Inferior Colliculus

Are the rats learning to categorize speech sounds or simply to associate two sounds

with different responses?

Female

Tad

Dad

Male

Time (ms)100 300 500 700

0

1

2

3

x 104

0 200 400 600 ms

30 kHz

20

10

Vo

icin

gGender

DT Day 1 DT Day 9 MT Day 1 MT Day 9-0.5

0

0.5

1

1.5

2

2.5

3

d'

Pitch Day 1 Pitch Day 9 Gender Day 1 Gender Day 9-0.5

0

0.5

1

1.5

2

2.5

3

d'

Rats generalize rapidly

Dad vs. TadSingle Speaker

Low vs. High Pitch

Day 1 Day 9 Day 1 Day 9

Dad vs. TadSix Speakers

Male vs. Female

Day 1 Day 9 Day 1 Day 9

D-

Pri

me

D-

Pri

me

-20 0 20 40 60 80 10050

60

70

80

90

100

Background Noise (dB SPL)

Per

cent

Cor

rect

D vs. T (n=7)

Female vs. Male (n=2)

-50 0 50 100 150 200 -50 0 50 100 150 200 -50 0 50 100 150 200

Bad

0

Bac

kgro

und

Noi

se

Leve

l (dB

SP

L)

40

60

Dad Tad

Time (ms) Time (ms)Time (ms)

50 100 150 200 250 300 350 400

10

20

30

40

50

60

70

80

100 150 200 250 300 350 400 450 5000

5

10

15

20

25

30

35

40

45

50

A1 PAF

Conclusions1. Rats are able to accurately categorize many speech sounds.

2. Responses in IC and A1 are highly correlated with behavioral

discrimination and generalization of complex sounds.

3. Consonants appear to be represented by firing patterns.

4. Vowels appear to be represented by mean firing rate.

5. Anesthesia alters sustained, but not transient, A1 responses to

speech sounds.

Acknowledgements:Amanda Puckett - Frequency Discrimination Poster Sunday Morning # 174.16Crystal Engineer - Speech Generalization Poster Sunday Morning # 174.15Claudia Perez - Speech Training and Inferior Colliculus Jai Shetake - Awake A1 Speech PhysiologyRob RennakerVikram JakkamsettiRyan CarrawayHelen Chen

Onset NeurogramsFemale

Tad

Dad

Male

-50 0 50 100 150 200 250 300 350 400

-50 0 50 100 150 200 250 300 350 400

-50 0 50 100 150 200 250 300 350 400

-50 0 50 100 150 200 250 300 350 400

/Bad/ - site 1

0

/Bad/ - site 2

/Dad/ - site 1

/Dad/ - site 2

Bac

kgro

und

Noi

se L

evel

(dB

SP

L)

40

60

0

60

40

0

40

60

0

60

40

Time (ms)Time (ms)

25 50 75 100 50

60

70

80

90

100

Beh

avio

r P

erce

nt

Co

rrec

tA) Multi-Unit

1 ms bins

25 50 75 100

Classifier Percent Correct

B) Multi-Unit 40 ms bin

25 50 75 100

C) Single Unit 1 ms bins

25 50 75 100

D) Single Unit 40 ms bin

D/SD/B

D/G

D/T

Sh/F

Sh/JSh/HSh/Ch

Sh/S

R/LM/N

Euclidean

Distance

City block

Distance

Chebychev

Distance

Mean Single Trial MeanSingle Trial

MeanSingle Trial

Temporal OnsetR2=0.75

P=0.0006

R2=0.66

P=0.002

R2=0.59

P=0.005

R2=0.74

P=0.0007

R2=0.39

P=0.04

R2=0.63

P=0.003

Rate OnsetR2=0.046

P=0.5

R2=0.14

P=0.2

R2=0.08

P=0.41

R2=0.15

P=0.24

R2=0.01

P=0.75

R2=0.14

P=0.25

Manner of Articulation

Stops Fricatives Affricatives Nasals Glides Liquids

Place o

f Articu

lation

Lip

sR

oo

fB

ack

0-600 ms0-30 kHz

Pad

Tad

Kad Gad Shad Had

Dad

Bad Fad Vad

Sad Zad

Mad Wad

Chad Jad Nad

Yad

Lad

Rad

Consonant Spectrograms

Natural speech shifted up one octave with the STRAIGHT vocoder (Kawahara, 1997)

Rat Consonant Discrimination

m/n r/l sh/s sh/ch sh/h sh/j sh/f d/t d/g d/b d/s 0

20

40

60

80

100

Consonant Discrimination Task

Per

cen

t L

ever

Pre

ss

*********

N=11 rats

N=441 A1 sites, 6 rats Onset Neurograms

N=441 sites, 11 rats A1 Neurograms

Sensory inputs direct neural plasticity.

Attention regulates plasticity in adults.- Recanzone, Merzenich, Ahissar, Weinberger, Suga, etc.

0 200 400 600 8000

50

100

150

200

250

300

350

Mean IC PSTH

40

30

20

10

Fre

quen

cy

(kH

z)

0 100 200 300 400 500 600Time (msec)

0 100 200 300 400 500 600Time (msec)

Sad

Firi

ng R

ate

(Hz)

N=187 sites, 6 rats

0 100 200 300 400 500 600

High

Medium

Low

40

30

20

10

Fre

quen

cy

(kH

z)

0 100 200 300 400 500 600Time (msec)

0 100 200 300 400 500 600Time (msec)

SadN=187 sites, 6 rats

Mean IC PSTH’s

Manner of Articulation

Stops Fricatives Affricatives Nasals Glides Liquids

Place o

f Articu

lation

Lip

sR

oo

fB

ack

0-600 ms0-30 kHz

Pad

Tad

Kad Gad Shad Had

Dad

Bad Fad Vad

Sad Zad

Mad Wad

Chad Jad Nad

Yad

Lad

Rad

Consonant Spectrograms

Natural speech shifted up one octave with the STRAIGHT vocoder (Kawahara, 1997)

Inferior Colliculus Neurograms N=187 sites, 6 rats

Spatial Temporal

N=441 sites, 11 rats A1 Neurograms

N=441 sites, 11 rats A1 Neurograms

0 100 200 300 400 0 100 200 300 400

Sad Dad

High

Medium

Low

Time (milliseconds)

vs.

0 100 200 300 400 0 100 200 300 400

Dad

0 100 200 300 400 0 100 200 300 400

Sador

Speech Discrimination by Rats

Easy!

0 100 200 300 400 0 100 200 300 400

Rad Lad

High

Medium

Low

Time (milliseconds)

vs.

Impossible

0 100 200 300 400 0 100 200 300 400

Bad Dadvs.

High

Medium

Low

Impossibleusing mean rateor Easy using spike timing?

Rat Consonant Discrimination

m/n r/l sh/s sh/ch sh/h sh/j sh/f d/t d/g d/b d/s 0

20

40

60

80

100

Consonant Discrimination Task

Per

cen

t L

ever

Pre

ss

*********

N=11 rats

N=441 A1 sites, 6 rats Onset Neurograms

N=441 A1 sites, 6 rats Onset Neurograms

10000 15000 20000 25000

50

60

70

80

90

100

m/nr/l

sh/s

sh/chsh/h

sh/j

sh/f

d/td/g

d/bd/s

R2=0.66, P=0.002

Euclidean Distance Between Neurogram Pairs

Be

ha

vio

r P

erc

en

t C

orr

ec

t

50 60 70 80 90 100

50

60

70

80

90

100

m/nr/l

sh/s

sh/ch

sh/h sh/j

sh/fd/t

d/gd/b

d/s

R2=0.66, P=0.002

Classifier Percent Correct

Be

ha

vio

r P

erc

en

t C

orr

ec

tR= 0.81 P= 0.002

Neural Discrimination Predicts Behavioral Discrimination

Speech Class

SpikeReadout

Inferior Colliculus

N=187

Primary Auditory Cortex

N=441

Awake Primary Auditory Cortex

N=40

Consonants Spike Timing 0.82 0.81 0.82

Mean Rate - - -

Vowels

Temporal Patterns

- - -

MeanRate

0.91 - 0.71

10000 15000 20000 25000

50

60

70

80

90

100

m/nr/l

sh/s

sh/chsh/h

sh/j

sh/f

d/td/g

d/bd/s

R2=0.66, P=0.002

Euclidean Distance Between Neurogram Pairs

Be

ha

vio

r P

erc

en

t C

orr

ec

t

50 60 70 80 90 100

50

60

70

80

90

100

m/nr/l

sh/s

sh/ch

sh/h sh/j

sh/fd/t

d/gd/b

d/s

R2=0.66, P=0.002

Classifier Percent Correct

Be

ha

vio

r P

erc

en

t C

orr

ec

t

Correlation Between Neural and Behavioral Discrimination

D

nVd

S

10

20

30

10

20

30

40

/ a / / ea / / u / / ee / / oo /

500100 200 300 400 6000 500100 200 300 400 6000 500100 200 300 400 6000 500100 200 300 400 6000 500100 200 300 400 6000

Time (ms)

40 kHz

Sad Sead Sud Seed Sood

Dad Dead Dud Deed Dood

10

20

30

40 kHz

Vowel Spectrograms

Sad Sead Sud Seed Sood

Dad Dead Dud Deed Dood

Inferior Colliculus Neurograms N=187 sites, 6 rats

a ea u ee oo

10 20 30 40 50 60 70 80 90

100Vowel Discrimination

Distance from CS+in F1-F2 space

Per

cent

Lev

er

Pre

ss

Be

havi

ora

l P

erf

orm

ance

Vowel Discrimination

N=14 rats

Speech Class

SpikeReadout

Inferior Colliculus

N=187

Primary Auditory Cortex

N=441

Awake Primary Auditory Cortex

N=40

Consonants Spike Timing 0.82 0.81 0.82

Mean Rate - - -

Vowels Spike Timing - - -

Mean Rate 0.91 - 0.71

Correlation Between Neural and Behavioral Speech Discrimination

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.824, P=0.00183

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.915, P=0.000532

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.562, P=0.0721

50 60 70 80 90 100

50

60

70

80

90

100B

ehav

ior

Per

form

ance

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.562, P=0.115

0 150 300 450 6000

100

200

300

400

500

0 150 300 450 6000

50

100

150

200

250

0 150 300 450 6000

50

100

150

200

250

Dad IC Dad A1 Dad A1 AwakeConsonant Spike Timing Consonant Rate Vowel Spike Timing Vowel Rate

Inferior Colliculus

Neural Discrimination Using Spike Patterns or Rate

Are the rats learning to categorize speech sounds or simply to associate two sounds

with different responses?

Female

Tad

Dad

Male

Time (ms)100 300 500 700

0

1

2

3

x 104

0 200 400 600 ms

30 kHz

20

10

Vo

icin

gGender

DT Day 1 DT Day 9 MT Day 1 MT Day 9-0.5

0

0.5

1

1.5

2

2.5

3

d'

Pitch Day 1 Pitch Day 9 Gender Day 1 Gender Day 9-0.5

0

0.5

1

1.5

2

2.5

3

d'

Rats generalize rapidly

Dad vs. TadSingle Speaker

Low vs. High Pitch

Day 1 Day 9 Day 1 Day 9

Dad vs. TadSix Speakers

Male vs. Female

Day 1 Day 9 Day 1 Day 9

D-

Pri

me

D-

Pri

me

Physiological Consequencesof Speech Training

Speech training could:

1. Increase response to CS+ but not CS-

2. Increase response to CS+ and CS-

3. Increase response to all speech sounds

-50 0 50 100 150 200 250 300 -50

0

50

100

150

200

250

Driv

en

Ra

te (

Hz)

Target Sound: Dad

NaiveDT TrainedMultiple TrainedStd Error

-50 0 50 100 150 200 250 300 -50

0

50

100

150

200

250

Driv

en

Ra

te (

Hz)

Non-Target Sound: Tad

A)

B)

C)

-50 0 50 100 150 200 250 300 -50

0

50

100

150

200

250

Time (msec)

Driv

en

Ra

te (

Hz)

Untrained Sound: Mad

Enhanced response to

TARGET is consistent with

Hypotheses #1, #2, and #3

9.5±0.7 vs. 5.9±0.5 and 5.6±0.2, p<0.001

Enhanced response to

NON-TARGET is consistent

with Hypotheses #2 and #3

10.1±0.5 vs. 6.4±0.4 and 7.0±0.3, p<0.001

Enhanced response to

NOVEL is consistent with

Hypothesis #3

7.1±0.7 vs. 5.3±0.4 and 4.9±0.2, p<0.0001

Time (ms)

Fre

quen

cy (

kHz)

0 500 1000 15000

2000

4000

6000

8000

10000

Time (ms)

Fre

quen

cy (

kHz)

0 500 1000 15000

2000

4000

6000

8000

10000The dog growled at the neighbors - Original

The dog growled at the neighbors – Four Channels

Neural Correlates of Discriminationof Stimuli on a Continuum

MTBritten, Shadlen, Newsome, & Movshon (1992)

A1Walker, Ahmed, & Schnupp (2007)

S1Luna, Hernandez, Brody & Romo (1994)

Correlation is determined by the: 1) Neurons Included 2) Temporal Precision Allowed

Frequency-Specific Map PlasticityLasts >20 Days

N = 16 rats; 880 A1 sites

2 4 8 16 32

10

30

50

70

A

10

20

30

40

2 4 8 16 32

10

30

50

70

B

-10

0

10

20

2 4 8 16 32

10

30

50

70

C

-10

0

10

20

2 4 8 16 32

10

30

50

70

D

-10

0

10

20

Tone Frequency (kHz)

Inte

nsit

y (d

B)

Inte

nsit

y (d

B)

Inte

nsit

y (d

B)

Inte

nsit

y (d

B)

Percent of A1 Responding

Difference 1 day after 19kHz + NB stim

Difference 20 days after 19kHz + NB stim

Difference 100 days after 19kHz + NB stim20

10

0

-10

20

10

0

-10

20

10

0

-10

60

40

20

0

Onset NeurogramsFemale

Tad

Dad

Male

F1 F2 F3 M1 M2 M3

102030405060708090

100Gender Last 2 Days

Pe

rce

nt

Hit

Female or Male Speaker

F1 F2 F3 M1 M2 M3

102030405060708090

100DT Multi Last 2 Days

Pe

rce

nt

Hit

Female or Male Speaker

N=10 rats N=11 rats

Gender

0 100 200 300 4000

20

40

60

80

100R2=0.9, P=2.2708e-006

Peak Rate

Per

cen

t H

it

FD1FD2FD3

FT1

FT2

FT3

MD1MD2 MD3

MT1

MT2MT3

Peak Firing Rate16-32 kHz sites

Peak Firing Rate1-2 kHz sites

Voicing

0 100 200 300 400 500 6000

200

400

600

0

200

400

600

0

200

400

600

0 100 200 300 400 500 6000

100

200

300

0

100

200

300

0

100

200

300

400

0 100 200 300 400 500 6000

100

200

300

0

100

200

300

0

100

200

300

400

High, Medium, and Low Responses to Dad

IC A1 Awake A1

Time (msec)

Firi

ng R

ate

(Hz)

Plasticitya) Developmentalb) Also works awake but must direction attentional mechanisms

1) Clinical picture – neither preventable or curablea) We have the tools but do not know how to use them.

2) EE – alters anatomy and neurochemistry and aids rehab but the connection between the two is difficult to identify.a) Physiology

I) AnesthetizedII) AwakeIII) In vitro

b) But not very targeted3) NB

a) Arousal and attention activates ACh releaseb) Stimulating NB paired with different sounds alters subcortical, primary and non-primary auditory cortex in a long-lasting and specific manner

I) MapsII) Selectivity III) SensitivityIV) Temporal processingV) SequencesVI) Synchrony

c) What is the relation to behavior? What is plasticity good for?4) NB-improves behavior, just like developmental plasticity does.

a) But patients don’t want deep brain stimulating electrodes5) Pharmacology to open the critical period

a) Rolipramb) May be useful for map based disorders, but lacks temporal specificityc) Aberrant temporal processing has been observed in many (perhaps most) neurological and psychiatric conditions.

• Speech processing • Explain hypothesis multiple articulatory features different more accurate

No hypothesis for single articulatory featureSuggest that sounds which evoke similar neural representations will be confusable

8) How to quantify neural similarity? Rate vs. Temporal codeIC

Vowels = Rate Consonants = Temporal

A1Consonants = Temporal

Awake A1Vowels = Rate (Show Rennaker RRTFs)Consonants = Temporal

Are these patterns learned categorically or memorized?compression, DT speaker, and gender generalization

Show neural correlates 9) Plasticity

Time (ms)100 300 500 700

0

1

2

3

x 104

0 200 400 600 ms

30 kHz

20

10

Tad

Dad

100 90 80 70 60 50 40 30 20 10%

100 90 80 70 60 50 40 30 20 10

102030405060708090

100DT Compression Last 2 Days

Percent of Original Stimulus Length

Per

cen

t H

it

DadTad

DT compression correlation between behavior and peak rate (CF between 1 & 2 kHz)

0 200 400 6000

20

40

60

80

100R2=0.68, P=8.6481e-006

Peak Rate

Per

cen

t H

itD100D90 D80 D70D60D50

D40D30 D20

D10

T100T90

T80T70

T60

T50 T40T30

T20T10

Awake vs. Anesthetized A1 responses

-100 0 100 200 300 400 5000

50

100

0

50

100 sad

Time (ms)

Ave

rage

Firi

ng R

ate

(Hz)

-100 0 100 200 300 400 5000

50

100

0

50

100 dad

Time (ms)

Ave

rage

Firi

ng R

ate

(Hz)

AwakeAnesthetized

Awake vs. Anesthetized A1 responses

-100 0 100 200 300 400 5000

50100150200250300

050

100150200250300

sad

Time (ms)

Ave

rage

Firi

ng R

ate

(Hz)

A1 AwakeA1 AnesthetizedIC Anesthetized

“The isomorphism should be sought --- not in the first-order relation Between (a) an individual object, and (b) its internal representation --- but in the second-order relation between (a) the relations among alternative objects, and (b) the relations among their internal representations.

Thus, although the internal representation for a square need not itself be square, it should (whatever it is) at least have a closer functional relation to the internal representation for a rectangle than to that, say, for a green flash or the taste of a persimmon.” - Shepard and Chipman (1970)

Representation and Generalization

DT Multi correlation between behavior and Euclidean distance (all CFs)

1 1.2 1.4 1.6 1.8 2

x 104

0

20

40

60

80

100R2=0.31, P=0.061044

Euclidean Distance

Per

cen

t H

itDF1

DF2DF3

DM1DM2

DM3

TF1

TF2TF3

TM1

TM2

TM3

500 1000 1500 2000 2500 30003000

3500

4000

4500

5000

5500

6000

6500

7000

DadDead

Dud

Deed

Dood

SadSead

Sud

Seed

Sood

First Peak

Sec

ond

Pea

k

a ea u ee oo

10 20 30 40 50 60 70 80 90

100Vowel Discrimination

Inferior Colliculus Codes Speech Sounds Using Both

Spike Rate and Timing

Inferior Colliculus Codes Speech Sounds using Both Spike Rate and TimingIC uses temporal code for consonants and rate code for vowels

Simultaneous use of Temporal and Rate Code for Speech in Inferior ColliculusTemporal and Rate Coding of Speech Sounds in Inferior Colliculus

1.5

3.2

6.4

7.3

9.9

11.3

14.1

15.7

18.2

22.6

pad

CF

bad fad vad mad wad

1.53.26.47.39.9

11.314.115.718.222.6

tad

CF

dad sad zad chad jad nad rad lad

0 200

1.53.26.47.39.9

11.314.115.718.222.6

kad

CF

Time (ms)0 200

gad

Time (ms)0 200

shad

Time (ms)0 200

had

Time (ms)0 200

yad

Time (ms)

Group A: Started with /DVD/ then moved into /SVD/

Group B: Started with /SVD/ then moved into /DVD/

1)Learning occurs over first 10 days (n=8). h=1/p=0.0035

Performance on day 1 is not above chance. h=0/p=0.0644

2) Learning occurs over second 10 days(n=8). h=1/p=0.0021

Performance on day 11 is above chance (though significantly above chance). h=1/p=0.0024

3)Learning does not occur over last (20) days of training. (n=8) h=0/p=0.0863

Group A

Group B

Behavior

/D/ /S/ /Noise/

a ea u ee oo

10 20 30 40 50 60 70 80 90

100Vowel Discrimination

a ea u ee oo

10 20 30 40 50 60 70 80 90

100Vowel Discrimination

a ea u ee oo

10 20 30 40 50 60 70 80 90

100Vowel Discrimination

500 750 1000 1250 1500 1750 2000 2250 25003000

3500

4000

4500

5000

5500

6000

6500

7000

7500

DadDead

Dud

Deed

Dood

SadSead

Sud

Seed

Sood

First Peak

Se

con

d P

ea

kCut consonant /200 msec vowel

Performance correlated with Peaks

0 0.5 1 1.5 250

60

70

80

90

100

Nead

Nud

Need Nood

Dead

Dud

Deed Dood

Distance from CS+

Be

ha

vio

r P

erf

orm

an

ce

R=0.855, P=0.00682

NVddVdsVd

0 0.5 1 1.5 250

60

70

80

90

100

Nead

Nud

Need Nood

Dead

Dud

Deed Dood Sead

Sud

Seed Sood

Distance from CS+

Be

ha

vio

r P

erf

orm

an

ce

R=0.765, P=0.00377

NVddVdsVd

500 1000 1500 2000 2500 30003000

3500

4000

4500

5000

5500

6000

6500

7000

DadDead

Dud

Deed

Dood

SadSead

Sud

Seed

Sood

First Peak

Sec

ond

Pea

kPerformance correlated with Formants

Cut consonant /200 msec vowel

NEW SLIDE

Performance correlated with Formants

ADD PITCH

NEW SLIDE

MU:Vowel 300 ms

Rate

Rate Class

Temporal Class

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.915, P=0.000532

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.562, P=0.115

0 10 20 30 40 50

50

60

70

80

90

100

Mean Difference in Spike Rate (Hz)

Beh

avio

r P

erfo

rman

ce

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

dad dad low pitch

R=0.913, P=0.000594

NEW SLIDE

MU/Consonant 50ms

Rate

Rate Class

Temporal Class

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.562, P=0.0721

50 60 70 80 90 100

50

60

70

80

90

100

Beh

avio

r P

erfo

rman

ce

Classifier Performance

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.824, P=0.00183

0 50 100 150

50

60

70

80

90

100

Mean Difference in Spike Rate (Hz)

Beh

avio

r P

erfo

rman

ce

mad nad

rad lad

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

dad sad

R=0.327, P=0.326

NEW SLIDE

Figure: CF influences classifier performance based on consonants Temporal:50msec

based on vowel rate:300msec

NEW SLIDE

MU/Consonant 300ms

Rate Class

Temporal Class

Not ShownNEW SLIDE

SU:Vowel 300 msRate

Rate Class

Temporal Class

Not ShownNEW SLIDE

SU Consonant 50 ms

Rate

Rate Class

Temporal ClassNot Shown

NEW SLIDE

SU Consonant 300 ms

Rate Class

Temporal Class

Not ShownNEW SLIDE

1 2 4 8 16 32-35

-30

-25

-20

-15

-10

-5

0

Frequency (kHz)

Rel

ativ

e P

ower

(dB

)

sad

sead

sudsood

seed

1 2 4 8 16 32-35

-30

-25

-20

-15

-10

-5

0

Frequency (kHz)

Rel

ativ

e P

ower

(dB

)

dad

dead

duddood

deed

Powerspectrum of vowel only

200msec

Supplemental

A1

Ba

ckR

oo

fL

ips

Place

of Articu

lation

LiquidsGlidesNasalsAffricatesFricativesStops

Manner of Articulation

Ba

ckR

oo

fL

ips

Place

of Articu

lation

LiquidsGlidesNasalsAffricatesFricativesStops

Manner of Articulation

Time (ms)100 300 500 700

0

1

2

3

x 104

Pad

Tad

Kad Gad Shad Had

Dad

Bad Fad Vad

Sad Zad

Mad Wad

Chad Jad Nad

Yad

Lad

Rad

0 200 400 600 ms

30 kHz

20

10

Figure 1

?Spatial SpatiotemporalTemporal

Spatial

Neu

ral D

iscr

imin

atio

n r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

50%

60%

70%

80%

90%

r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

Figure 6

A) OnsetSpike Timing

B) OnsetMean Rate

C) DurationSpike Timing

D) DurationMean Rate

Figure 7

50

60

70

80

90

100

Per

cen

t C

orr

ect

Onset1 ms bins

Onset40 ms bin

Duration1 ms bins

Duration700 ms bin

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

R2

SixteenMulti-Units

OneMulti-Unit

OneSingleUnit

SixteenSingleUnits

SixteenMulti-Units

OneMulti-Unit

SixteenSingleUnits

OneSingleUnit

A) B)

50

60

70

80

90

100

Per

cen

t C

orr

ect

Onset 1 ms bins

Onset 40 ms bin

Duration 1 ms bins

Duration 700 ms bin

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

R2

50

60

70

80

90

100

Per

cen

t C

orr

ect

Onset 1 ms bins

Onset 40 ms bin

Duration 1 ms bins

Duration 700 ms bin

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

R2

50

60

70

80

90

100

Per

cen

t C

orr

ect

Onset 1 ms bins

Onset 40 ms bin

Duration 1 ms bins

Duration 700 ms bin

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

R2

50

60

70

80

90

100

Per

cen

t C

orr

ect

Onset 1 ms bins

Onset 40 ms bin

Duration 1 ms bins

Duration 700 ms bin

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

R2

Supplementary Figure 2

Supplementary Figure 1

AWAKE A1

40 awake A1 Vowels – Rate

50 60 70 80 90 100

50

60

70

80

90

100B

ehav

ior

Per

form

ance

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

R=0.713, P=0.0471

40 awake A1 Vowels – temporal

50 60 70 80 90 100

50

60

70

80

90

100B

ehav

ior

Per

form

ance

Classifier Performance

dad dead

sad sud

dad dud

sad sood

sad seed

sad sead

dad dood

dad deed

R=0.0336, P=0.937

40 awake A1 Consonants – Temporal

50 60 70 80 90 100

50

60

70

80

90

100B

ehav

ior

Per

form

ance

Classifier Performance

mad nad

rad lad

dad dad low pitch

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

R=0.818, P=0.0021

40 awake A1 Consonants – Rate

50 60 70 80 90 100

50

60

70

80

90

100B

ehav

ior

Per

form

ance

Classifier Performance

mad nad

rad lad

dad dad low pitch

shad sad

shad chad

shad had

shad jad

shad fad

dad tad

dad gad

dadbad

R=0.404, P=0.217

Figure 3

Re

lati

ve

dif

fere

nc

e b

etw

ee

n o

ns

et

res

po

ns

es

Re

lati

ve

dif

fere

nc

e b

etw

ee

n o

ns

et

res

po

ns

es

A) Spike Timing

B) Mean Rate

r l y w n m j ch h z v sh s f g d b k t p

p t k b d g f s

sh v z h

ch j

m n w y l r

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Neural Correlates of Frequency Discrimination

0 0.2 0.4 0.6 0.8 1 1.2 1.440

50

60

70

80

90

100

Octaves from CS+

Per

cent

Cor

rect

LateEarly

1 site

2 sites

5 sites

10 sites30 sites T

emp

ora

l P

reci

sio

nNumber Of Sites

Behavioral Threshold

Sunday Morning Poster #174.16

Peter Heil - Leibniz Institute, Magdeburg

Towards a unifying basis of absolute auditory thresholds

Lutz Wiegrebe – University of Munich

Psychophysical and physiological evidence for fast binaural processing

Tony Zador - Cold Spring Harbor Laboratory

Millisecond spike timing can guide behavior in auditory cortex

Kerry Walker – University of Oxford

A spike pattern based neurometric analysis for the discrimination of natural sounds

Kamal Sen - Boston University

Discrimination of Complex Natural Sounds in Songbirds: Neurons & Behavior

Mike Kilgard - University of Texas

Cortical Activity Patterns Predict Speech Discrimination Ability

Jennifer Bizley – University of Oxford

The Neural Basis of Pitch Perception

Shihab Shamma - University of Maryland

Encoding task rules and performance in auditory and frontal cortex of the ferret

Yale Cohen – Dartmouth University

Auditory attention and auditory categorization in primate ventrolateral prefrontal cortex

Inferior Colliculus

Primary Auditory Cortex

Speech Class

SpikeReadout

MUN=187

SUN=12

MUN=441

AwakeN=40

SUN=16

Consonants

Temporal Patterns

0.82 0.63 0.81 0.82 -

MeanRate

- - - - -

Vowels

Temporal Patterns

- - - - -

MeanRate

0.91 0.74 - 0.71 -

Neural Correlates of Consonant and Vowel Discrimination

0 100 200 300 400 0 100 200 300 4000 100 200 300 400 0 100 200 300 400

Rad Lad ??or

Impossible

Speech Discrimination by Rats