Following the music to the memories
1
Following the music to the memories Petr Janata Center for Mind and Brain & Dept. of Psychology, Univ. of California, Davis, Davis, CA. 0.25 0.5 1 2 4 1000 2000 N=13 MPFC (peak: x=-20, y=60, z=22) 0.25 0.5 1 2 4 600 1200 N=13 Occipital/Temporal (peak: x=-18, y=-68, z=24) 0.25 0.5 1 2 4 170 340 N=13 left VLPFC (peak: x=-42, y=38, z=6) 0.25 0.5 1 2 4 185 370 N=8 left STS (peak: x=-50, y=-40, z=6) 0.25 0.5 1 2 4 60 120 N=8 left prCG (peak: x=-46, y=-10, z=48) 0.25 0.5 1 2 4 50 100 N=9 right MFG (peak: x=40, y=20, z=38) 0.25 0.5 1 2 4 20 40 N=8 SFG (peak: x=6, y=28, z=54) 0.25 0.5 1 2 4 15 30 N=8 FO/Insula (peak: x=34, y=24, z=-2) 0.25 0.5 1 2 4 Tonality-Tracking Bias Autobiographical salience ratio (log10) Salient Non-salient -60mm -50 -40 -30 -20 -10 -5 +5 +10 +20 +30 +40 +50 +60mm 3 4 5 6 #subjects 60 40 20 0 -20-40-60-80 60 40 20 0 -20-40-60-80 60 40 20 0 -20-40-60-80 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 +40 +25 +15 +5 -10 -40 -70mm L R -60 -40 -20 0 20 40 60 -60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 VLPFC VLPFC MPFC MPFC STS Music Playing Question/Answer Period Autobiographical Salience Familiarity Constant Time (s) Affective Valence Music Playing Question/Answer Period Autobiographical Salience Familiarity Constant Affective Valence Subject 4 Subject 1 0 124 248 372 496 620 744 868 0 120 240 360 480 600 720 840 Run 1 Run 2 Time (s) Run 1 Run 2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Music Playing Question/ Answer Autobio Familiarity Valence A B Correlation Coefficient Figure 1. Design matrices for two subjects illustrate the statistical models that were used. The grayscale codes the regressor values, with black representing the most negative values and white the most positive. The ubiquitous gray represents zero. A) 30s music epochs alter- nated with the question/answer periods. Answers on the multipoint scalse were used to code parametric regressors for Autobiographical Salience, Familiarity and Affective Valence. Black bars represent song excerpts that were unfamiliar or not autobiographically salient, whereas white bars represent songs that were familiar and strongly autobiographically salient. Valence ratings range from very displeasing (black) to very pleasing (white) with neither pleasing nor displeasing represented by zero. B) Correlation matrices of the design matrices shown in A. The Familiarity, Autobiographical Salience, and Valence regressors showed moderate to strong correlations among each other. This was true across all subjects. Introduction The evocation of autobiographical memories and associated emotions by music counts among the most poignant experiences associated with music. Accordingly, excerpts of music can serve as potent retrieval cues with which to study the functional architecture of autobiographical memory. Converging evidence suggests that the rostral aspect of the medial prefrontal cortex (BA10) is a region where music and autobiographical memories might be integrated. This study tested the hypothesis that experiencing autobiographically salient musical excerpts will result in increased activation in the medial prefrontal cortex, and that the medial prefrontal cortex will track the tonal (harmonic) structure of the musical excerpts, particularly the autobiographically salient ones. Methods Prescreening and memory characterization • Thirty 30s excerpts from Billboard Top 100 Pop and R&B charts • Randomly selected from time when subject was between 7 and 19 years of age. • Following each excerpt, a subject reported whether the song was familiar and if so whether it evoked a weakly or strongly autobiographical memory or no clear memory at all. • Subjects were invited to participate in the fMRI version of the experiment if >30% of the excerpts evoked an autobiographical memory. fMRI experiment • 13 subjects (11 female) • 30 stimuli (selected according to the same criteria as above) across two scanning runs. • No repeats of stimuli from prescreen. • Ratings following each excerpt: • Affective valence (5-point scale) • Arousal (5-pt scale) • Familiarity (2 pt scale) • Autobiographical salience (3 pt scale) • Degree of orientation of their attention toward the memories (5 pt scale) • Degree of orientation of their attention toward the music (5 pt scale). Scan parameters: Siemens 3T Trio, 34 slices (4 mm thick, 0 skip), TR=2.0s, in-plane resolution: 3.4x3.4 mm. Analyses: Analyses were performed using SPM5. Data were spatially normalized, and variance associated with estimated movement parameters and linear trends was removed. The residuals were analysed using the models shown in Figures 1 and 4. Modeling music’s movement in tonal space Design matrices for testing general task effects and parametric variation in subjective variables Group-level analyses of each subjective variable contrast Group-level analyses of task effects and combination of Familiarity, Autobiographical Salience, Affect Converging evidence from music theory, cognitive psychology, and self-organizing neural networks indi- cates that the system of major and minor keys in West- ern tonal music can be represented on the surface of the torus. As a piece of music unfolds, the changing harmonies and chord progressions create a changing pattern of activity on the surface of the torus, providing a signa- ture of a piece of music. Subject 4 cc10 cc01 cs01 cs02 cs03 sc10 sc21 sc13 sc21 cc02 cc03 cc11 cc12 cc13 cs11 cs12 cs13 sc20 sc12 sc23 ss12 ss23 constant Time (s) 0 124 248 372 496 620 744 868 Run 1 Run 2 Tonality Regressors (spherical harmonics) Autobiographical Non- Autobiographical Figure 4. Design matrix in which tonality tracking is estimated separately for songs that did and didn’t elicit autobiographical associations. False-positives were controlled for by comparing the residual variance of the veridical model against the distribution of residual variances obtained from 100 models in which the ordering of the songs was permuted. f ( , ) = a mn cc cos(m )cos( n ) + m = 0, n = 0 M, N a mn cs cos(m )sin(n ) + m = 0, n = 0 M, N a mn sc sin( m )cos(n ) + m = 0, n = 0 M, N a mn ss sin(m )sin(n ) m = 0, n = 0 M , N cs01 0.73 Spherical harmonic Cumulative sum sc11 -0.30 sc10 -0.22 cc11 -0.19 cs11 -0.19 Original activation The activation pattern on the torus at any given moment in time can be described as a weighted-sum of a set of spherical harmonics (spatial frequencies) using the equation below. momentary activation pattern θ φ unfolded torus As the activation pattern changes during the course of a song, so do the weights for each spherical harmonic. These changing weights are entered as regressors for each of 34 spherical harmonics. 100 200 300 400 500 600 700 -1.5 -1 -0.5 0 0.5 1 1.5 Volume (TR=3s) Z-value Spherical harmonic regressor 4 Time 1s 3s 5s 7s 9s 11s Subject 1 Subject 2 Subject 3 Tonality Tracking -55mm -40 +0 +40 +55 -55mm -35 +0 +35 +55 -45mm -35 -10 +15 +35 60 40 20 0 -20 -40 -60 -80 -40 -20 0 20 40 60 -40 -20 0 20 40 60 60 40 20 0 -20 -40 -60 -80 -40 -20 0 20 40 60 60 40 20 0 -20 -40 -60 -80 3 2 1 2 3 ratio autobio non- autobio Tonality-tracking responses Figure 5. These tonality tracking images show the proportion of variance explained by the tonality tracking regressors modeling autobiographically salient pieces of music relative to those modeling pieces of music that elicited no autobio- graphical association. The color scale represents the ratio of these variances, with each variance corrected by the number of music excerpts contributing to its estimate. Red values indicate a bias toward tonality tracking associated with an autobiographical memory, and blue denotes tonality tracking without an associated autobiographical memory. Figure 6. A summary of brain areas exhibiting tonality-tracking behavior. The color of each voxel indicates the number of subjects in whom that voxel showed significant tonality-tracking behavior. The cyan contours demarcate the borders of the FAV-related activations shown in Figure 2, and illustrate the co-localization of the FAV and tonality-tracking effects in MPFC. The insets show distributions of tonality tracking biases for all voxels in clusters of voxels that showed tonality tracking for a majority of subjects. A ratio of 1 indicates equal tracking of autobiographically salient and non-salient songs, whereas larger ratios indicate a bias toward tracking of autobiographically salient songs. The box and whisker plots provide information about the distribution of mean tonality-tracking biases across subjects. -60mm -50 -40 -30 -20 -10 -5 +5 +10 +20 +30 +40 60 40 20 0 -20 -40 -60 -80 -40 -20 0 20 40 60 +50 +60mm +40mm +25 +15 +5 -10 -40 -60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -70mm 60 40 20 0 -20-40-60-80 60 40 20 0 -20-40-60-80 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 L R -60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 STS IPS OFC SFG IFS VLPFC VMPFC MPFC DLPFC FO FO Ins STG STG pSMA pSMA ACC PCC pSMA AG AG Cb SPL Music QAP FAV all contrasts p<0.005 Figure 2. Group analyses for 3 basic contrasts: Music Playing (green), the Question/Answer Period (blue) and a contrast in which the Familiarity, Autobiographical Salience, and Valence effects were combined. They were combined because of the strong correlations among these variables for all subjects. Thus, areas in red, yellow, and magenta showed greater activation when the song was familiar, strongly autobiographically salient and evoked a positive emotion. -60mm -50 -40 -30 -20 -10 -5 +5 +10 +20 +30 +40 +50 +60mm 60 40 20 0 -20-40-60-80 -40 -20 0 20 40 60 60 40 20 0 -20-40-60-80 60 40 20 0 -20-40-60-80 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 3.5 4 4.5 5 t-value +40mm +25 +15 +5 -10 -40 -70 L R -60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -40 -20 0 20 40 60 -60 -40 -20 0 20 40 60 PHG DMPFC IFG/FO MTG ITS vltn vltn Cun Cun VLPFC VLPFC PT Ins ins DMPFC VMPFC nAcc nAcc PHG pSMA pSMA all contrasts p<0.005 Figure 3. Individual effects of familiarity (green), autobiographical salience (red), and af- fective (positive) valence (blue). For purposes of comparison, the cyan outlines are the borders of the FAV contrast shown in red in Figure 2. Valence Familiar Autobio Conclusions Dorsal medial prefrontal cortex exhibits properties of an area that binds together music with autobiographically salient memories, in that it responds more strongly during autobiographically salient songs and it also follows the structure of the music, predominantly for songs that are autobiographically salient. Left VLPFC shows similar properties. The effects of familiarity and autobiographical salience (lateralized to the left hemisphere) and tonality tracking were distributed across a number of prefrontal, temporal, and occipital areas, as would be expected given the interactions of frontal memory retrieval mechanisms with content representations in posterior cortices. Acknowledgments: I wish to thank John Ryan, Mike Buonocore, Sonja Rakowski, Susannah Cohen, and Cheryline Mancusi for assistance in data collection. Supported in part by NIH R03 DC05146-01 and a Templeton Advanced Research Program grant from the Metanexus Institute
Transcript of Following the music to the memories
Following the music to the memoriesPetr Janata
Center for Mind and Brain & Dept. of Psychology, Univ. of California, Davis, Davis, CA.
0.25 0.5 1 2 4
1000
2000N=13
MPFC(peak: x=-20, y=60, z=22)
0.25 0.5 1 2 4
600
1200N=13
Occipital/Temporal(peak: x=-18, y=-68, z=24)
0.25 0.5 1 2 4
170
340N=13
left VLPFC(peak: x=-42, y=38, z=6)
0.25 0.5 1 2 4
185
370N=8
left STS(peak: x=-50, y=-40, z=6)
0.25 0.5 1 2 4
60
120N=8
left prCG(peak: x=-46, y=-10, z=48)
0.25 0.5 1 2 4
50
100
N=9
right MFG(peak: x=40, y=20, z=38)
0.25 0.5 1 2 4
20
40
N=8
SFG(peak: x=6, y=28, z=54)
0.25 0.5 1 2 4
15
30N=8
FO/Insula(peak: x=34, y=24, z=-2)
0.25 0.5 1 2 4
Tonality-Tracking Bias
Autobiographical salience ratio(log10)
SalientNon-salient
-60mm -50 -40
-30 -20 -10
-5 +5 +10
+20 +30 +40
+50 +60mm3
4
5
6
#sub
ject
s
60 40 20 0 -20-40-60-8060 40 20 0 -20-40-60-8060 40 20 0 -20-40-60-80-40-20
0204060
-40-20
0204060
-40-20
0204060
-40-20
0204060
-40-20
0204060
+40 +25
+15 +5
-10 -40
-70mm
L R
-60 -40 -20 0 20 40 60-60 -40 -20 0 20 40 60-40
-20
0
20
40
60
-40
-20
0
20
40
60
-40
-20
0
20
40
60
-40
-20
0
20
40
60
VLPFC
VLPFC
MPFC
MPFC
STS
Music Playing
Question/AnswerPeriod
AutobiographicalSalience
Familiarity
Constant
Time (s)
Affective Valence
Music Playing
Question/AnswerPeriod
AutobiographicalSalience
Familiarity
Constant
Affective Valence
Subject 4
Subject 1
0 124 248 372 496 620 744 868
0 120 240 360 480 600 720 840
Run 1 Run 2
Time (s)Run 1 Run 2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Music Playing
Question/
Answ
er
Autobio
Familiarity
Valence
A B
Cor
rela
tion
Coe
ffici
ent
Figure 1. Design matrices for two subjects illustrate the statistical models that were used. The grayscale codes the regressor values, with black representing the most negative values and white the most positive. The ubiquitous gray represents zero. A) 30s music epochs alter-nated with the question/answer periods. Answers on the multipoint scalse were used to code parametric regressors for Autobiographical Salience, Familiarity and Affective Valence. Black bars represent song excerpts that were unfamiliar or not autobiographically salient, whereas white bars represent songs that were familiar and strongly autobiographically salient. Valence ratings range from very displeasing (black) to very pleasing (white) with neither pleasing nor displeasing represented by zero. B) Correlation matrices of the design matrices shown in A. The Familiarity, Autobiographical Salience, and Valence regressors showed moderate to strong correlations among each other. This was true across all subjects.
IntroductionThe evocation of autobiographical memories and associated emotions by music counts among the most poignant experiences associated with music. Accordingly, excerpts of music can serve as potent retrieval cues with which to study the functional architecture of autobiographical memory. Converging evidence suggests that the rostral aspect of the medial prefrontal cortex (BA10) is a region where music and autobiographical memories might be integrated. This study tested the hypothesis that experiencing autobiographically salient musical excerpts will result in increased activation in the medial prefrontal cortex, and that the medial prefrontal cortex will track the tonal (harmonic) structure of the musical excerpts, particularly the autobiographically salient ones.
MethodsPrescreening and memory characterization• Thirty 30s excerpts from Billboard Top 100 Pop and R&B charts • Randomly selected from time when subject was between 7 and 19 years of age. • Following each excerpt, a subject reported whether the song was familiar and if so whether it evoked a weakly or strongly autobiographical memory or no clear memory at all.• Subjects were invited to participate in the fMRI version of the experiment if >30% of the excerpts evoked an autobiographical memory.
fMRI experiment• 13 subjects (11 female) • 30 stimuli (selected according to the same criteria as above) across two scanning runs. • No repeats of stimuli from prescreen. • Ratings following each excerpt: • Affective valence (5-point scale) • Arousal (5-pt scale) • Familiarity (2 pt scale) • Autobiographical salience (3 pt scale) • Degree of orientation of their attention toward the memories (5 pt scale) • Degree of orientation of their attention toward the music (5 pt scale).
Scan parameters: Siemens 3T Trio, 34 slices (4 mm thick, 0 skip), TR=2.0s, in-plane resolution: 3.4x3.4 mm.
Analyses: Analyses were performed using SPM5. Data were spatially normalized, and variance associated with estimated movement parameters and linear trends was removed. The residuals were analysed using the models shown in Figures 1 and 4.
Modeling music’s movement in tonal spaceDesign matrices for testing general task effectsand parametric variation in subjective variables
Group-level analyses of each subjectivevariable contrast
Group-level analyses of task effects and combination of
Familiarity, Autobiographical Salience, Affect
Converging evidence from music theory, cognitive psychology, and self-organizing neural networks indi-cates that the system of major and minor keys in West-ern tonal music can be represented on the surface of the torus.As a piece of music unfolds, the changing harmonies and chord progressions create a changing pattern of activity on the surface of the torus, providing a signa-ture of a piece of music.
Subject 4
cc10
cc01
cs01cs02cs03sc10sc21sc13sc21
cc02cc03
cc11cc12cc13cs11cs12cs13sc20sc12sc23ss12ss23
constant
Time (s)
0 124 248 372 496 620 744 868
Run 1 Run 2
Tona
lity
Reg
ress
ors
(sph
eric
al h
arm
onic
s)
Aut
obio
grap
hica
lN
on-
Aut
obio
grap
hica
l
Figure 4. Design matrix in which tonality tracking is estimated separately for songs that did and didn’t elicit autobiographical associations. False-positives were controlled for by comparing the residual variance of the veridical model against the distribution of residual variances obtained from 100 models in which the ordering of the songs was permuted.
f ( , ) = amncc cos(m )cos(n ) +
m= 0,n= 0
M, N
amncs cos(m )sin(n ) +
m= 0,n= 0
M, N
amnsc sin(m )cos(n ) +
m= 0, n= 0
M, N
amnss sin(m )sin(n )
m=0,n =0
M ,N
cs01 0.73
Spherical harmonic Cumulative sum
sc11 -0.30
sc10 -0.22
cc11 -0.19
cs11 -0.19
Original activation
The activation pattern on the torus at any given moment in time can be described as a weighted-sum of a set of spherical harmonics (spatial frequencies) using the equation below.
momentary activationpattern
θ
φ
unfolded torus
As the activation pattern changes during the course of a song, so do the weights for each spherical harmonic. These changing weights are entered as regressors for each of 34 spherical harmonics.
100 200 300 400 500 600 700
-1.5
-1
-0.5
0
0.5
1
1.5
Volume (TR=3s)
Z-va
lue
Spherical harmonic regressor 4
Time
1s
3s
5s
7s
9s
11s
Subject 1 Subject 2 Subject 3
Tona
lity
Trac
king
-55mm -40
+0 +40
+55
-55mm -35
+0 +35
+55
-45mm -35
-10 +15
+35
60 40 20 0 -20 -40 -60 -80
-40-20
0204060
-40-20
0204060
60 40 20 0 -20 -40 -60 -80
-40-20
0204060
60 40 20 0 -20 -40 -60 -80
32
1
23
ratio
autobio
non-autobio
Tonality-tracking responses
Figure 5. These tonality tracking images show the proportion of variance explained by the tonality tracking regressors modeling autobiographically salient pieces of music relative to those modeling pieces of music that elicited no autobio-graphical association. The color scale represents the ratio of these variances, with each variance corrected by the number of music excerpts contributing to its estimate. Red values indicate a bias toward tonality tracking associated with an autobiographical memory, and blue denotes tonality tracking without an associated autobiographical memory.
Figure 6. A summary of brain areas exhibiting tonality-tracking behavior. The color of each voxel indicates the number of subjects in whom that voxel showed significant tonality-tracking behavior. The cyan contours demarcate the borders of the FAV-related activations shown in Figure 2, and illustrate the co-localization of the FAV and tonality-tracking effects in MPFC. The insets show distributions of tonality tracking biases for all voxels in clusters of voxels that showed tonality tracking for a majority of subjects. A ratio of 1 indicates equal tracking of autobiographically salient and non-salient songs, whereas larger ratios indicate a bias toward tracking of autobiographically salient songs. The box and whisker plots provide information about the distribution of mean tonality-tracking biases across subjects.
-60mm -50 -40
-30 -20 -10
-5 +5 +10
+20 +30 +4060 40 20 0 -20-40-60-80
-40-20
0204060
+50 +60mm
+40mm +25
+15 +5
-10 -40-60 -40 -20 0 20 40 60
-40
-20
0
20
40
60
-70mm
60 40 20 0 -20-40-60-8060 40 20 0 -20-40-60-80
-40-20
0204060
-40-20
0204060
-40-20
0204060
-40-20
0204060
L R
-60 -40 -20 0 20 40 60
-40
-20
0
20
40
60
-40
-20
0
20
40
60
-40
-20
0
20
40
60
STS
IPS
OFC
SFGIFS
VLPFC
VMPFC
MPFC
DLPFC
FO FO
Ins
STG
STG
pSMA
pSMA
ACC
PCC
pSMA
AG
AG
Cb
SPL
Music QAP
FAV
all contrastsp<0.005
Figure 2. Group analyses for 3 basic contrasts: Music Playing (green), the Question/Answer Period (blue) and a contrast in which the Familiarity, Autobiographical Salience, and Valence effects were combined. They were combined because of the strong correlations among these variables for all subjects. Thus, areas in red, yellow, and magenta showed greater activation when the song was familiar, strongly autobiographically salient and evoked a positive emotion.
-60mm -50 -40
-30 -20 -10
-5 +5 +10
+20 +30 +40
+50 +60mm
60 40 20 0 -20-40-60-80-40-20
0204060
60 40 20 0 -20-40-60-8060 40 20 0 -20-40-60-80
-40-20
0204060
-40-20
0204060
-40-20
0204060
-40-20
0204060
3.54
4.55
t-val
ue
+40mm +25
+15 +5
-10 -40
-70
L R
-60 -40 -20 0 20 40 60-40
-20
0
20
40
60
-40
-20
0
20
40
60
-40
-20
0
20
40
60
-40
-20
0
20
40
60
-60 -40 -20 0 20 40 60
PHG
DMPFC
IFG/FO
MTG
ITS
vltn
vltn
Cun
Cun
VLPFCVLPFC
PTIns
ins
DMPFC
VMPFC
nAcc
nAcc
PHG
pSMA
pSMA
all contrastsp<0.005
Figure 3. Individual effects of familiarity (green), autobiographical salience (red), and af-fective (positive) valence (blue). For purposes of comparison, the cyan outlines are the borders of the FAV contrast shown in red in Figure 2.
Vale
nce
Fam
iliar
Aut
obio
ConclusionsDorsal medial prefrontal cortex exhibits properties of an area that binds together music with autobiographically salient memories, in that it responds more strongly during autobiographically salient songs and it also follows the structure of the music, predominantly for songs that are autobiographically salient. Left VLPFC shows similar properties. The effects of familiarity and autobiographical salience (lateralized to the left hemisphere) and tonality tracking were distributed across a number of prefrontal, temporal, and occipital areas, as would be expected given the interactions of frontal memory retrieval mechanisms with content representations in posterior cortices.
Acknowledgments: I wish to thank John Ryan, Mike Buonocore, Sonja Rakowski, Susannah Cohen, and Cheryline Mancusi for assistance in data collection. Supported in part by NIH R03 DC05146-01 and a Templeton Advanced Research Program grant from the Metanexus Institute
