Pattern-information fMRI and representational...
Transcript of Pattern-information fMRI and representational...
![Page 1: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/1.jpg)
Pattern-information fMRI and representational similarity analysis
Mathematics in Brain ImagingIPAM, UCLA, 25 July 2008, Los Angeles
Nikolaus KriegeskorteSection on Functional Imaging Methods, Laboratory of Brain and Cognition
National Institute of Mental Health
![Page 2: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/2.jpg)
response channels
face
sob
ject
s
...
...
stim
uli
response amplitude
functional region
![Page 3: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/3.jpg)
response channels
...
...
stim
uli
response amplitude
functional region
face
sob
ject
s
![Page 4: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/4.jpg)
Activation analysis
response channels
face
sob
ject
s
...
...
stim
uli
response amplitude
functional region
![Page 5: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/5.jpg)
Pattern-information analysis
response channels
face
sob
ject
s
...
...
stim
uli
response amplitude
functional region
![Page 6: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/6.jpg)
Goal: honor all these distinctions
response channels
face
sob
ject
s
...
...
stim
uli
response amplitude
functional region
![Page 7: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/7.jpg)
Talk overview
Specific neuroscientific experiments • inferior temporal object representations
in human and monkey
General methodology• every stimulus is a condition• condition-rich fMRI design• representational similarity analysis
![Page 8: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/8.jpg)
Related literature• multidimensional scaling
Torgerson (1958), Kruskal & Wish (1978), Shepard (1980)
• second-order isomorphism Shepard & Chipman (1970)
• first application of multidimensional scaling to fMRI data Edelman et al. (1998)
• some more recent studies with similarity analyses Laakso and Cottrell (2000) , Op de Beeck et al. (2001), Hanson et al. (2004), O’Toole et al. (2005), Tsao et al. (2006), Sereno & Lehky (2006), Aguirre (2007), Kiani et al. (2007)
![Page 9: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/9.jpg)
Collaborators
Bethesda, MD, USA• Marieke Mur• Douglas Ruff• Jerzy Bodurka• Peter Bandettini
Seattle, WA, USA• Roozbeh Kiani
Tehran, Iran• Hossein Esteky
Wako, Saitama, Japan• Keiji Tanaka
![Page 10: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/10.jpg)
Overview of experiment and analysis
• present images of real-world objects to subjects (human, monkey)
• measure the brain-activity pattern during perception of each particular image (fMRI, cell recording)
• study similarity structure of object representations focusing on inferior temporal (IT) cortex
• compare representations in human and monkey IT by relating representational similarity matrices
![Page 11: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/11.jpg)
Core concept representational similarity matrix
![Page 12: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/12.jpg)
diss
imila
rity
dissimilarity matrix dissimilarity-graph icon
experimental conditions
activity patterns
brain
p<1%5%
![Page 13: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/13.jpg)
Transformation of representational similarity
p<1%5%
right left
Parahippocampalplace area (PPA)
Fusiformface area(FFA)
Early visual areas
Anterior inferotemporal face-exemplar region
Kriegeskorte et al. 2007
![Page 14: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/14.jpg)
rubberband graph• stretched effects: thin• compressed effects: thick• effect [bits] ∝
length ⋅
tickness
Anterior inferotemporal face-exemplar region
p<0.01
Kriegeskorte et al. 2007
![Page 15: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/15.jpg)
96-stimulus experiment
![Page 16: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/16.jpg)
Brain-activity measurementsHuman• fMRI in four subjects
(>12 runs per subject)
• task: fixate, discriminate fixation-point color changes
• occipitotemporal measurement slab (5-cm thick)
• voxels: 1.95×1.95×2mm3
• 3T magnet, 16-channel coil (SENSE, acc. fac. 2)
• stimulus duration: 300ms• quick event-related (SOA=4s)
Monkey (Kiani et al. 2007)• single-cell recordings
in two monkeys• task: fixation• electrodes in anterior IT
(left in monkey 1, right in monkey 2)
• 674 cells total• stimulus duration: 105ms• rapid serial presentation• 140-ms window spike count
starting 71ms after stimulus onset
![Page 17: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/17.jpg)
Single-image hemodynamic response predictors
[fMRI volumes]
fMR
I sig
nal
![Page 18: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/18.jpg)
Design matrix
![Page 19: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/19.jpg)
human monkey
Stimulus quartets (multidimensional scaling, metric stress)
(1) multidimensional scaling (1-r, metric stress)(2) rigid alignment for easier comparison (Procrustes alignment)
![Page 20: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/20.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 21: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/21.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 22: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/22.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 23: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/23.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 24: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/24.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 25: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/25.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 26: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/26.jpg)
Stimulus quartets (multidimensional scaling, metric stress)
human monkey
![Page 27: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/27.jpg)
responsepatterns
stimuli ...
...
brain
dissimilaritymatrix
000000000000
96
96
compute dissimilarity (1-correlation across space)
96
![Page 28: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/28.jpg)
Human IT (1000 most visually responsive voxels within anatomical IT mask)
matrices of4 subjectsaveraged
animate|
inanimate
human face 1|
human face 2
![Page 29: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/29.jpg)
To visualize the representation... Let's arrange the images in a figure
reflecting response-pattern similarity
Multidimensional scaling
• images close similar response patterns
• images far dissimilar response patterns
Torgerson (1958), Kruskal & Wish (1978), Shepard (1980) first application to fMRI: Edelman et al. (1998)
![Page 30: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/30.jpg)
Human IT (1000 visually most responsive voxels)
![Page 31: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/31.jpg)
Human IT (1000 visually most responsive voxels)
![Page 32: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/32.jpg)
Human IT (1000 visually most responsive voxels)
![Page 33: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/33.jpg)
Human IT (1000 visually most responsive voxels)
![Page 34: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/34.jpg)
Human IT (1000 visually most responsive voxels)
![Page 35: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/35.jpg)
Human IT (larger) (3162 visually most responsive voxels)
![Page 36: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/36.jpg)
Human IT (smaller) (316 visually most responsive voxels)
![Page 37: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/37.jpg)
Human early visual cortex (1057 visually most responsive voxels)
![Page 38: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/38.jpg)
Human IT (1000 visually most responsive voxels)
![Page 39: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/39.jpg)
similaritymatrices
activitypatterns
experimentalevents ...
... ...
human brain monkey brain
![Page 40: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/40.jpg)
Monkey IT (674 cells in left and right aIT, windowed spike count,
fixation task, 2 monkeys)
data from Roozbeh Kiani et al. (2007), Journal of Neurophysiology
![Page 41: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/41.jpg)
average of 4 subjectsfixation-color task316 voxels
average of 2 monkeysfixation task
674 cells
Kriegeskorte et al. (submitted)
monkeyhuman
![Page 42: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/42.jpg)
Stimulus arrangements reflecting response-pattern similarity
(1) multidimensional scaling (1-r, metric stress)(2) rigid alignment for easier comparison (Procrustes alignment)
![Page 43: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/43.jpg)
![Page 44: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/44.jpg)
monkey IT dissimilarity (1-r)
hum
an IT
dis
sim
ilarit
y(1
-r)
![Page 45: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/45.jpg)
monkey IT dissimilarity (1-r)
hum
an IT
dis
sim
ilarit
y(1
-r)
![Page 46: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/46.jpg)
average of 4 subjectsfixation-color task316 voxels
average of 2 monkeysfixation task
>600 cells
Kriegeskorte et al. (under revision)
![Page 47: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/47.jpg)
Man-to-monkey correlation of dissimilarites
monkey IT dissimilarity (1-r)
hum
an IT
dis
sim
ilarit
y(1
-r)
![Page 48: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/48.jpg)
monkey IT dissimilarity (1-r)
hum
an IT
dis
sim
ilarit
y(1
-r)
Man-to-monkey correlation of dissimilarities
![Page 49: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/49.jpg)
monkey IT dissimilarity (1-r)
hum
an IT
dis
sim
ilarit
y(1
-r)
Man-to-monkey correlation of dissimilarities
![Page 50: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/50.jpg)
monkey IT dissimilarity (1-r)
hum
an IT
dis
sim
ilarit
y(1
-r)
Man-to-monkey correlation of dissimilarities
![Page 51: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/51.jpg)
significant also...• within faces• within bodies • within humans• within nonhuman animals
not significant...• within human faces• within animal faces
Man-to-monkey correlation of dissimilarites
![Page 52: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/52.jpg)
Representational connectivityEVC LO
![Page 53: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/53.jpg)
EVC LO
bilateral EVC right LO?
?
![Page 54: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/54.jpg)
EVC LO
animateinanimate
(1) multidimensional scaling (1-r, metric stress)
(2) rigid alignment for easier comparison (Procrustes method)
bilateral EVC
right LO
![Page 55: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/55.jpg)
EVC LO
![Page 56: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/56.jpg)
Summary so far...• Natural categorical structure in primate IT
– top-level distinction: animate | inanimate – faces form a separate tight cluster
• Man-monkey match of IT representational similarity– matching major categorical clusters– matching within-category similarity structure
InterpretationHuman and monkey IT may host a common code for continuous and categorical object information.
![Page 57: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/57.jpg)
But questions remain...
• Can the apparent categorical structure be explained by low-level features?
• More generally, what kind of computational model can explain our findings?
![Page 58: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/58.jpg)
From description to explanation...
• use a range of models to predict representational similarity
• models can be computational, conceptual, or based on behavioral data
![Page 59: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/59.jpg)
similaritymatrices
activitypatterns
experimentalevents ...
... ...
brain model
compare
![Page 60: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/60.jpg)
![Page 61: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/61.jpg)
![Page 62: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/62.jpg)
![Page 63: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/63.jpg)
![Page 64: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/64.jpg)
![Page 65: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/65.jpg)
![Page 66: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/66.jpg)
![Page 67: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/67.jpg)
![Page 68: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/68.jpg)
![Page 69: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/69.jpg)
Model dissimilarity matrices
![Page 70: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/70.jpg)
rank (1-correlation)
human IT
HMAX model V1 model
monkey IT animate-inanimate
stimulus image
Riesenhuber& Poggio 2002
![Page 71: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/71.jpg)
perfectly matching dissimilarity matrix
noiseless true model
retest (with different subjects)
Early visual cortexbrain regionnaive modelneuro-model
category model
![Page 72: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/72.jpg)
noiseless true model
retest (with different subjects)
Human ITde
viat
ion
from
hum
an IT
dis
sim
ilarit
y m
atrix
(dissimilarity SNR=1.29) brain regionnaive modelneuro-model
category model
![Page 73: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/73.jpg)
Simultaneously relating multiple representational similarity matrices
multidimensional scaling(metric stress, 1-r(Spearman))
brain regionnaive modelneuro-model
category model
![Page 74: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/74.jpg)
Summary of model-based analyses
• We compared a range of low- and intermediate-level model representations to the IT representation.
• None of them could explain the categorical clustering we saw in human and monkey IT.
InterpretationAn IT model probably needs to include category- specific visual knowledge – as might be acquired by supervised learning.
![Page 75: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/75.jpg)
Methodological concepts
![Page 76: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/76.jpg)
Searchlightmapping
Kriegeskorte et al. 2006
![Page 77: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/77.jpg)
Searchlightmapping
Kriegeskorte et al. 2006
(p2 -p1 )T -1(p2 -p1 )Σ
![Page 78: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/78.jpg)
Representational-similarity searchlight mapping
searchlight local fMRIsimilarity matrix
modelsimilarity matrix
compare
![Page 79: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/79.jpg)
behavioral data• reaction time• errors• explicit judgements
brain-activity data• cell recordings • fMRI• EEG, MEG
computational models• symbolic models• connectionist models• biological neural models
![Page 80: Pattern-information fMRI and representational …helper.ipam.ucla.edu/publications/mbi2008/mbi2008_7524.pdfPattern-information fMRI and representational similarity analysis Mathematics](https://reader033.fdocuments.net/reader033/viewer/2022060419/5f164f43f919667dcf5a23bf/html5/thumbnails/80.jpg)
behavioral data• reaction time• errors• explicit judgements
brain-activity data• cell recordings • fMRI• EEG, MEG
representationalsimilarity matrix
computational models• symbolic models• connectionist models• biological neural models