THE EFFECT OF MUSICAL ACUITY ON AUDIOVISUAL …

THE EFFECT OF MUSICAL ACUITY ON AUDIOVISUAL SYNCHRONY PERCEPTION

AND THE MCGURK EFFECT 1

The Effect of Musical Acuity on Audiovisual Synchrony Perception and the McGurk Effect

A Behavioral and ERP study

Lotte E. A. Miegielsen

Tilburg University

Bachelorthesis Psychologie & Gezondheid

Lotte Miegielsen (952099)

Begeleider: Dr. J. J. Stekelenburg



Abstract

Multisensory integration refers to the process in which information from different senses is

combined to an integrated product. This is demonstrated by the McGurk effect, where stimuli

from one modality can influence the perception of another. When auditory stimuli are

combined with incongruent visual stimuli, the perception of the auditory stimulus often is

altered. For this to happen, the stimuli onset must occur inside a temporal window. The

McGurk effect decreases with increasing temporal discrepancies between auditory and visual

stimuli. Personal factors, like musical expertise, can influence the width of the temporal

window of integration. The McGurk effect can evoke mismatch negativity (MMN) in event

related potentials (ERP), even though there is no actual auditory change in stimuli.

In current study, it was examined whether participants’ innate sense of pitch and rhythm

(musical acuity) has an effect on audiovisual synchrony perception and perceived McGurk

effect. It was found that musical acuity makes participants more sensitive to audiovisual

asynchronies, suggesting that not just musical expertise, but also musical acuity can narrow

the temporal window of integration. A tendency toward significance was found for an effect

of musical acuity on the temporal window of the McGurk effect. It further was examined

whether a decrease in MMN could be found for the McGurk effect presented at different

audiovisual asynchronies, and if musical acuity has an effect on this. No significant results

were found in ERP study.

Keywords: McGurk effect; musical acuity; audiovisual; stimulus onset asynchrony; temporal

window of integration; event related potential; mismatch negativity



The Effect of Musical Acuity on Audiovisual Synchrony Perception and the McGurk Effect

In everyday life, we are constantly faced with a multitude of information that reaches

us through multiple senses, such as vision, hearing, smell, taste, or touch. Every situation we

encounter has different characteristics, and for any of these characteristics there is a sense of

optimal usefulness. Together, the senses enhance the likelihood of detecting information that

is useful, interesting, and important to us. Even more important than the information obtained

by the individual senses, is the ability to combine that information. The integrated product of

information from our senses not only provides us with a more accurate representation of

reality than would be predicted from the sum of the individual senses, but also provides it

faster and better. This synergy of information from the different senses is referred to as

multisensory integration (Stein & Stanford, 2008). Receiving information from multiple

senses at a time, may seem inconvenient sometimes. When you’re trying to read a book, for

example, loud noises can be an annoying distraction. Advantages occur when the perceived

multisensory information arises from the same event. Two factors that determine whether

multisensory information can be integrated are the timing and spatial separation of the

information (Calvert, Hansen, Iversen, & Brammer, 2001; Radeau, 1994).

Multisensory processing occurs in many different locations in the brain. It occurs in

association cortex, but also on a more basal level of information processing, for example in

primary auditory cortex. Diverse fMRI studies show influence of multiple modalities on

activation of auditory cortex (Calvert et al., 1997; Kayser, Petkov, & Logothetis, 2009;

Musacchia & Schroeder, 2009; Pekkola et al., 2005). Information from the different senses is

merged and integrated by individual neurons (multisensory neurons). This integration is, for

example, important in the superior colliculus, where sensory stimuli of different modalities

are processed and motor areas are stimulated, to enable orientation. Multisensory integration

provides a significant increase or decrease in cell responses, compared to unisensory



processing (Meredith & Stein, 1985; Meredith & Stein, 1986a; Meredith & Stein 1986b;

Wurtz & Albano, 1980).

The response increase, or decrease not only occurs on neuronal level, but is also

evident on a behavioral level, for example in reaction time. When participants react to

congruent bimodal stimuli (audiovisual), their reaction time is significantly lower than the

reaction time to unimodal stimuli (visual or auditory). Even combined, the unimodal stimuli

cannot account for the difference in reaction time (Diederich & Colonius, 2004; Miller, 1982;

Schröger & Widmann, 1998; Stein, Meredith, Huneycutt, & McDade, 1989).

When processing multisensory information, one modality can influence the perception

of other modalities. This is what happens in the McGurk effect (McGurk & McDonald,

1976). Normally, when we are in conversation, we receive congruent audiovisual stimuli. The

visual information can help us gain a better understanding of the auditory speech. When you

are in a noisy environment, for example, lipreading can improve hearing, and understanding

of spoken language (Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Schwartz,

Berthommier, Savariaux, 2004). When experiencing the McGurk effect, visual speech

information that is incongruent with auditory speech information, alters the perception of the

auditory information. When, for instance, an auditory stimulus /ba/ is linked to a visual

stimulus /ga/, people often report hearing /da/. This effect is called a fusion response, because

two different stimuli are ‘fused’ into a third (McGurk & McDonald, 1976). The McGurk

effect provides a clear example of the role that multisensory processing has in our audiovisual

speech perception, in addition to the unisensory processing of stimuli. The effect visualizes

audiovisual processing, and is therefore studied a lot. The effect is studied on a behavioral,

and also on a neuronal level.

Jones and Munhall (1997) investigated to what extend the McGurk effect is influenced

by spatial separation of the auditory and visual stimuli. They found that the McGurk effect is



not affected by spatial discrepancies up to 90°, suggesting that the McGurk effect is

maintained under spatial incongruent conditions.

When receiving multisensory information, there is a temporal window in which multisensory

stimuli are integrated, called temporal window of integration. Asynchronies in stimulus onset

that fall within that window are not detected (Spence & Squire, 2003). The width of this

temporal window of integration depends on multiple factors, such as the distance between the

person and the visible sound source (Sugita & Suzuki, 2003). Studies show that the McGurk

effect is subjected to temporal discrepancies. When audiovisual stimuli are presented at

different stimulus onset asynchronies (SOAs), the perceived McGurk effect decreases with

increasing asynchrony (Munhall, Gribble, Sacco, & Ward, 1996; van Wassenhove, Grant, &

Poeppel, 2007). Multiple researches have shown that our temporal window of integration of

audiovisual information is asymmetrical. When audio precedes vision, asynchronies are more

easily detected than when vision precedes audio (Dixon & Spitz, 1980; Grant, van

Wassenhove, & Poeppel, 2004; Sugita & Suzuki, 2003). This asymmetry was found for

audiovisual discrepancies in the McGurk effect too (van Wassenhove et al., 2007). A possible

explanation for this phenomena is the physical difference in the natural velocity of light and

sound. In a second, light travels 300.000.000 meters through air, while sound only travels 330

meters (Dixon & Spitz, 1980; Spence & Squire, 2003). Research shows that the brain

accounts for these differences in velocity, making it possible to integrate audiovisual

information and maintaining the perception of synchrony, even when there are temporal

discrepancies (Sugita & Suzuki, 2003).

The McGurk effect in the brain is studied using a part of the auditory event related

potential (ERP) that shows reaction to change in stimuli on a preattentive level: mismatch

negativity (MMN) (Näätänen, Paavilainen, Rinne, & Alho, 2007). MMN can be found when

there is a change (deviant) in a repetitive sound (standard), for example in duration,



frequency, or intensity. Auditory MMN is visible as a negative peak in ERP data, which is

usually most evident in frontocentral and central scalp electrodes. Peak of the MMN mostly

comes around 150-250 ms after deviant stimulus onset (for a review, see Näätanen et al.,

2007). Around 300 ms in ERP signal a positive P3 peak can be found. MMN is a reaction for

which no attention is needed, P3 on the other hand is associated with conscious attention to

the stimulus (Näätänen, Simpson, & Loveless, 1982; Sams, Paavilainen, Alho, & Näätänen,

1985). Interestingly, during the McGurk effect MMN can occur without occurrence of actual

auditory changes. MMN can be evoked using a congruent standard audiovisual stimulus (for

example auditory /ba/ linked to visual /ba/), alternating with an incongruent deviant stimulus

(auditory /ba/ linked to visual /ga/) (Colin et al., 2002; Saint-Amour, De Sanctis, Molholm,

Ritter, & Foxe, 2007; Sams et al., 1991).

Personal factors can affect the width of the temporal window of integration, and

therefore of the perception of synchrony. Petrini et al. (2009) examined whether drummers

with professional training, were more prone to detect asynchronies in audiovisual stimulus

onset than participants with no musical training. Results from their study support this

hypothesis, showing that drummers with musical expertise detected asynchronies more often

than nonmusicians. Petrini et al. (2009) conclude from their study with trained drummers, that

expertise can narrow the temporal window of integration for stimuli. In their study they use

stimuli that are consistent with the subject of expertise: drumming actions. The question

remains whether this effect can be attributed to musical training solely, or that the sensitivity

to temporal asynchronies can be due to participants’ innate sense of pitch and rhythm: their

musical acuity. It would be interesting to examine whether this effect of a narrowed temporal

window of integration can also be found for nonmusicians who scored high on musical acuity,

in comparison to nonmusicians who scored low on musical acuity.



Further, if musical acuity has an influence on the perception of asynchronies, it can be

assessed whether this effect can also be found in the perception of the McGurk effect, a

speech condition, when presented asynchronously. Previous studies have examined the effect

of temporal discrepancies on perceived McGurk effect, but have not yet examined whether

this effect of audiovisual asynchronies can also be found in the brain, using ERP and MMN.

This would be an interesting addition.

The aim of current study is to examine whether musical acuity has an effect on the

perception of audiovisual asynchronies, and if this effect can also be found in the perception

of the McGurk effect on a behavioral and neuronal level.

Two experiments will be conducted. In the first experiment, the effect of thirty different

audiovisual stimulus onset asynchronies (SOAs) on the perceived McGurk effect will be

measured, and this will be linked to participants musical acuity. In the second experiment,

participants MMN will be measured, while they are exposed to standard and deviant

audiovisual stimuli with three different SOAs.

Method

Experiment 1

Participants

Participants were twenty-four students (3 males, 21 females) at Tilburg University

who received course credits for participation. Three were left-handed, twenty-one right

handed, and age ranged from 18 to 27 years (mean age 20, SD 2.38). Participants reported

normal hearing and normal or corrected-to-normal vision. Participants who reported any

neurological or audiovisual abnormality, were excluded. Except one, all subjects reported to

have no musical expertise (training).

Stimuli



The room in which the experiment took place was dimly lit and soundproof.

Participants were monitored via a camera to ensure active participation. Visual stimuli were

presented on a 19 inch Iyama monitor, which was positioned at eye level approximately 70

cm from participant’s head. Sounds came from two speakers, located at both sides of the

monitor.

Stimuli existed of the Dutch auditory pseudo-word /tabi/ and visual word /tagi/, pronounced

by a male speaker, visible from head to shoulders. The visual angle of the video frames was

12º horizontal and 19º vertical. Videos were presented at a rate of 25 frames/second and a

single visual stimulus /tagi/ consisted of 37 frames (1480 ms) and 4 frames to fade-in and

fade-out. Audio was presented at a rate of 44100 bit/s, and the duration of a single auditory

stimulus /tabi/ was 695 ms. Peak intensity of the autitory stimuli was 63 dB.

Stimuli were presented at thirty different stimulus onset asynchronies (SOAs), ranging from -

400 ms (audio precedes video) to +400 ms (video precedes audio). SOA fifteen and sixteen

were approximately synchronous. A catch trial was included, presented at SOA 14, to ensure

that participants focused on the lips of the speaker. In this trial, a white dot was placed on the

lip of the speaker. Participants had to detect this dot, and immediately press a corresponding

button.

Procedure

Participants were asked to sign an informed consent, switch off their phones, and fill

in a musicality questionnaire when entering the experiment. The questionnaire consisted of

five Dutch questions about music, singing and dancing in everyday life, which had to be

answered on a five point Likert scale with categories ‘never’, ‘rarely’, ‘sometimes’, ‘often’,

‘very often’, and twelve propositions about one’s musicality, with response options ‘ yes’ and

‘no’. The questionnaire was based on an online questionnaire, compiled by the International

Laboratory for Brain, Music and Sound Research (2012). Thereafter, a verbal instruction of



the experiment was given and participants read an instruction on screen. Half of the

participants started with a Simultaneous Judgment (SJ) task, in which participants had to

judge whether audio and video were presented synchronously. The response categories were

synchronous (sync), and asynchronous (a-sync), using two buttons. The other half of the

participants started with an Identification (ID) task, in which participants had to judge

whether they heard /tabi/, /tadi/ , or /tagi/. Response categories were /b/, /d/, and /g/, using

three buttons. Both the SJ and the ID task started with a practice session to familiarize

participants with the task, followed by four blocks of 96 stimuli (3 per SOA, and 6 catch

trials), which were randomly assigned. Stimuli were the same in the SJ and ID task.

Participants responded after each stimulus by pressing a corresponding button. After the

response there was an interstimulus interval of 1000 ms. When a catch trial appeared,

participants had to press a button immediately. At the end of a block, a total of missed catch

trials appeared on screen to give participants feedback on their performance. After completing

all blocks of one task, participants continued with the other task.

After the experiment, participants received a link to an online test to measure musicality

(International Laboratory for Brain, Music and Sound Research, 2012). The online test

consisted of three blocks. In the first block, participants heard a series of two successive

melodies that had to be compared and participants had to state whether the two melodies were

the same or different. The second block consisted of melodies that possibly contained an

unusual delay that had to be detected. In the third block, participants judged whether the

melody contained an out-of-tune note.

Data recording and analysis

Data from the SJ and ID task was converted to a proportion ‘synchronous’

answered and a proportion /d/ or /g/ answered at the different SOAs. This resulted in a

psychometric curve which resembles a gaussian distribution. To find the best fit, this curve



was fitted in multiple ways, using Excel and SPSS. The gaussian curve was estimated using

‘gaussian fit’ in excel. The standard deviation (SD) is used as an estimation of the width of

the curve. The width of the curves is used as a measure of accuracy of participants answers,

and acuity. Because of the asymmetry in the temporal window of integration, curves may be

skewed. To deal with this skewing, the width was also estimated using the 70 percent points

of two half curves. The half curves for SOA 1 to 15 and 16 to 30 were estimated using curve

estimation (logistic model) in SPSS. For each half curve, the x-value (SOA) of the estimated

70% point (y=0.7) was calculated. Then, the 70% x-value of SOA 1 to 15 was subtracted

from the 70% x-value of SOA 16 to 30. The obtained value is used as an alternative width

score (WS). The height of the curve was used as a control variable for the presence of the

mcGurk effect.

The music questionnaire resulted in two sum-scores. The questions on 5-point Likert-

scale were scored 1 to 5 and summed. The propositions were scored 0 or 1, 0 for non-musical

and 1 for musical, and summed . The online music test resulted in a percentage of correct

answers per block and an overall score. A correlation will be calculated between the sum

scores of the music questionnaire, the results of the online test, and the outcome of the ID and

SJ tasks.

Experiment 2

Participants

Fourteen of the participants of the behavioral experiment participated in the EEG experiment.

Seventeen were selected, three dropped out of the experiment because they no longer studied

at Tilburg University. Participants were selected if their data from the SJ and ID task of the

behavioral experiment showed a peak that reached at least 80%, and had lowest points of

20% at most.

Stimuli



The experiment took place in the same room as the first experiment, under the same

circumstances. Three types of stimuli were presented in blocks: auditory only (A), audiovisual

(AV), and visual only (V). Auditory stimuli were included to verify that participants produced

MMN when subjected to auditory change. These stimuli consisted of standard and deviant

beeps. Standard beeps were presented at 1000 Hz, deviant at 1100 Hz. Audiovisual stimuli

were the same as in the behavioral experiment, except that there was now a standard and

deviant version. Standard stimulus was the dutch auditory pseudo-word /tabi/ and visual

pseudo-word /tabi/, deviant was the Dutch auditory pseudo-word /tabi/ and visual pseudo-

word /tagi/. Both were pronounced by the same male speaker as in the behavioral experiment.

The AV stimuli were presented at 3 SOAs: the 100, 70, and 20 percent point at the right side

of each person’s curve (where video precedes audio). As a result, each participant had

idiosyncratic combinations of SOAs. To ensure that there would be enough data per SOA,

only one side of the curve was taken into account. Averaged data from the first experiment

showed skewing to the right side of the curve, suggesting that there was more mcGurk effect

at the right side (where V is leading). Visual blocks were the same as the audiovisual blocks,

only with no audio, and therefore with no SOA. Catch trials (same as in the behavioral

experiment) were added in the AV and V condition.

Procedure

The experiment started with an auditory block of 500 stimuli (425 standard, 75 deviant),

presented with an interstimulus interval of 750 ms. Then, three blocks per audiovisual SOA,

and 3 visual blocks were presented in quasi-random order, equally distributing the blocks over

the experiment, and making sure that blocks were never followed by the same block. Every

block, both visual and audiovisual, consisted of 240 stimuli (180 standard, 48 deviant, 12

catch trials), with an interstimulus interval of 1000 ms. In all conditions stimuli were

randomized, and a deviant was preceded by at least two standard stimuli. One block of



audiovisual or visual stimuli took 600 s to complete. Catch trials were added where

participants had to push a button, to make sure that participants kept paying attention. To

reduce fatigue, enough breaks were taken between the blocks.

Data recording and analysis

Sixty-four electrodes were placed on the head in an elastic cap, using the 10-20

system, and two mastoid electrodes. Four electrodes were placed above, under, and at the left

and right side of the eye, to monitor eye movements. The EEG was recorded at a sampling

rate of 512 Hz.

EEG data were referenced offline to an average of the left and right mastoid electrodes. Then,

data were filtered using a high-pass filter of 1 Hz, low-pass filter of 30 Hz, and a notch filter

of 50 Hz. Data from standard and deviant were separately segmented into epochs of 1000 ms,

including a 200-ms prestimulus baseline (A- and V-data), or epochs of 2800 ms, including a

1400 ms prestimulus baseline measured from start of audio (AV-data). This difference in

epochs was necessary because of the different SOAs in AV-condition. EEG was corrected

using EOG correction (Gratton, Coles, & Donchin, 1983), and an artifact rejection was set at

150 µV. An average was calculated per electrode, for standard and deviant. After baseline

correction, difference waves were calculated for every electrode, by subtracting the averaged

standard from the averaged deviant. In AV-condition, a difference wave was computed per

SOA. An average was computed per electrode for each participant, and peaks of the curves

were calculated, to measure MMN and P3 in a condition. To control for MMN evoked by

visual change in the AV-condition, V-difference waves were aligned to AV-difference waves,

and subtracted from AV-difference waves. Latency and height of the peaks were exported to

SPSS for further analysis.

Results

Participants detected 97% of all catch trials during the two experiments.



Experiment 1

Data from the SJ and ID task was converted to a proportion ‘synchronous’ answered

and a proportion /d/ or /g/ answered at the different SOAs. Average results are displayed in

Figure 1 (SJ task) and Figure 2 (ID task).

Figure 1. Simultaneous Judgment (SJ) Figure 2. Identification (ID) task as a

as a function of SOA. function of SOA.

Widths of the curves were estimated using Excel and SPSS. This resulted in two scores: SD

from gaussian fit in Excel and an alternative calculated width score (WS), for SJ and ID tasks.

The alternative width scores (WS) were calculated using participants’ 70% points on two

separated half curves. Not all data showed proper curves, and therefore one curve in SJ task

and four curves in ID task could not be estimated using the 70% points. These were defined

missing in WSSJ and WSID The only significant correlation between the width measures was

found between SDSJ and WSSJ (r(21)=.414, p=.050).

The music questionnaire (MQ) resulted in two sum-scores (MQ1 for Likert-scale

questions, MQ2 for propositions). Three of the twenty-four participants didn’t complete the

online test. The online music test resulted in a percentage of correct answers per block and an

overall score (MT1, MT2, MT3, MTtotal). Descriptive statistics are displayed in Table 1. In both

music questionnaire and music test scores, higher scores represent a better musical acuity.

0

0,2

0,4

0,6

0,8

1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Pro

po

rtio

n S

ynch

ron

y

audio lead SOA video lead

0

0,2

0,4

0,6

0,8

1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Pro

po

rtio

n /

da/

/ga

/

audio lead SOA video lead



Table 1

Descriptive statistics of Music Questionnaire and Music Test scores

variable n M SD Range

MQ1 24 16.50 2.38 11-20

MQ2 24 6.21 2.38 2-10

MT1 21 85.14 8.28 60-100

MT2 21 84.24 9.60 63-100

MT3 21 80.86 15.08 46-100

MTtotal 21 83.43 7.88 64-100

Correlations were found between MQ2 and MTtotal (r(19)=.46, p=.038), and between MQ1 and

MT2 (r(19)=.50, p=.021) (table 2).

Correlations were calculated for the different width measures, the mean SOA score

(M) found in gaussian fit, and different musicality measures (table 2). For SJ task, a

significant correlation was found between the alternative width measure (WSSJ) and the two

sumscores of the music questionnaire (MQ). (MQ1 r(21)=-.432, p=.040; MQ2 r(21)=-.459,

p=.027). Also, a significant correlation was found between WSSJ and block 2 of the music test

(r(18)=-.455, p=.044). The other blocks of the music test showed no significant correlation.

On ID task, the music questionnaire scores did not correlate with the width measures. Block 2

of the music test showed some correlation with WSID, but this was not significant (r(15)=-

.482, p=.050). For correlations, see figure 3, 4, 5, and 6.

Figure 3. Correlation WS SJ and MQ1 Figure 4. Correlation WS SJ and MQ2

THE EFFECT OF MUSICAL ACUITY ON AUDIOVISUAL SYNCHRONY PERCEPTION AND THE MCGURK EFFECT

15

Figure 5. Correlation WS SJ and MT2 Figure 6. Correlation WS ID and MT2

Table 2

Correlations experiment 1

SDSJ WSSJ MSJ SDID WSID MID MQ1 MQ2 MT1 MT2 MT3 MTtotal

SDSJ -

WSSJ .414* -

MSJ .202 -.036 -

SDID -.110 -.224 .202 -

WSID -.001 .159 .172 .243 -

MID .937*** .225 .248 .027 -.085 -

MQ1 -.011 -.432* .131 .258 -.174 .225 -

MQ2 -.204 -.459* .399 -.088 -.014 -.148 .227 -

MT1 -.104 -.197 .001 -.028 .210 -.038 .352 .414 -

MT2 -.156 -.455* .008 .046 -.482 -.012 .501* .378 .353 -

MT3 -.146 .133 .224 .220 -.072 -.125 .189 .260 .381 .191 -

MTtotal -.188 -.166 .134 .131 -.113 -.099 .424 .455* .755*** .612** .809*** -

* p < .05 , ** p < .01, *** p < .001



Experiment 2

The experiment consisted of auditory blocks (A), audiovisual (AV), and visual only (V).

Auditory condition

In A condition, data showed peaks for MMN and P3, of which the voltage and latency was

exported to SPSS. When looking at the averaged scalp activity, you can see that MMN and P3

focuses mainly to the frontal and central electrodes (figure 7). Therefore, a repeated measures

ANOVA was performed with electrodes Fz, Cz and PZ for both MMN and P3 on latency and

amplitude (µV) of peaks.

Figure 7. Auditory MMN (left) and auditory P3 (right)

A repeated measures ANOVA was performed to see if Fz, Cz, and Pz differed significantly in

MMN amplitude. Mauchly’s test showed no violation of spherecity assumption (χ2(2)=.804,

p=.669). For electrodes, a significance difference was found (F(2,10)=3.44, p =.048), and

electrodes together differed from 0 ( F(1,10)=40.98, p < .001). Post hoc analysis showed that

only electrodes Cz and Pz differed significantly (p=.032), with Cz showing more MMN (M=-

5.56) than Pz (M=-4.77). Fz (M=-5.51) did not differ significantly from electrodes Cz

(p=.863) or Pz (p=.074). Three additional one-sample t-tests were conducted to see whether

the three electrodes separately differed from 0. All three t-tests showed significant result,

184 - 186 ms

-3.9 µV 3.9 µV0 µV

293 - 295 ms

-4.3 µV 4.3 µV0 µV



indicating that all three electrodes differed from 0 (tFz(12)=-7.14, p< .001; tCz(12)=-5.85, p<

.001; tPz(12)=-5.90, p< .001).

For MMN latency, sphericity was not assumed (χ2(2)=6.26, p=.044). A repeated measures

ANOVA on latency, with a Greenhouse-Geisser correction, showed no significant differences

between Fz, Cz, and Pz (F(2,10)=.75, p=.441).

For P3, the same repeated measures ANOVA was performed with electrodes Fz, Cz, and Pz.

For P3 amplitude, spherecity assumption was not violated (χ2(2)=.499, p=.779). A significant

difference was found between electrodes (F(2,10)=14.06, p < .001), and electrodes together

differed from 0 ( F(1,10)=43.55, p < .001). Post hoc analysis showed that all three electrodes

differed significantly. Fz (M=4.20) was significantly lower (p=.020) than Cz (M=5.32), and

higher (p=.023) than Pz (M=2.91). Microvolts in Cz were significantly higher than in Pz (p<

.001), indicating that most P3 was found central on the scalp. Three additional one-sample t-

tests were conducted to compare the three electrodes separately with 0. All three t-tests

showed significant result, indicating that all three electrodes differed from 0 (tFz(12)=6,36, p<

.001; tCz(12)=6.55, p< .001; tPz(12)=5.37, p< .001).

For P3 latency, sphericity was not assumed (χ2(2)=12.02, p=.002). A repeated measures

ANOVA with a Greenhouse-Geisser correction showed no significant differences between

Fz, Cz, and Pz on latency (F(2,10)=2.60, p=.125).

To see whether musical acuity has an influence on auditory ERP data, a correlation was

calculated between the musical acuity measures (music questionnaire (MQ) sumscores, and

music test (MT) scores), and the amplitude and latency of MMN and P3 peaks. Most MMN

was found on electrodes Fz and Cz, and therefore these electrodes were included in the

correlation. Most P3 was found on Cz, so this electrode was included. Results are displayed in

tables 3 and 4. Correlations are displayed in figure 8 and 9.



Table 3 .

Correlations of auditory MMN volts and latency and music measures.

Fz V Cz V Fz L Cz L

Fz V -

Cz V .960*** -

Fz L .045 .111 -

Cz L .017 .030 .835*** -

MQ1 .221 .156 -.042 .228

MQ2 -.094 -.091 .057 .055

MT1 -.024 -.016 -.640* -.447

MT2 -.453 -.514 -.071 -.009

MT3 .014 .082 -.289 -.167

MTtotal -.139 -.107 -.453 -.290

* p < .05 , ** p < .01, *** p < .001

Table 4.

Correlations of auditory P3 volts and latency and music measures.

Cz V Cz L

Cz V -

Cz L .081 -

MQ1 -.010 -.021

MQ2 -.490 -.285

MT1 -.304 -.469

MT2 .037 -.209

MT3 -.373 -.561*

MTtotal -.335 -.541

* p < .05

Figure 8. Correlation MMN Fz L and MT1 Figure 9. Correlation P3 Cz L and MT3



Audiovisual condition

Part of the MMN that is found in AV condition, can be accounted for by V MMN. In

figure 10, the overlap in MMN between AV and V condition is visible.

Figure 10. AV MMN for sync condition (left) and V MMN (right)

To control for this, data from V blocks were subtracted from AV data. Analysis were

performed on AV-V data. When looking at the averaged AV-V data (figure 11), MMN seems

to be lateralized around electrodes CP5 (left hemisphere) and CP6 (right hemisphere). MMN

decreases with increasing asynchrony (figure 11 and 12).

424 - 426 ms

-2.4 µV 2.4 µV0 µV

406 - 408 ms

-2.4 µV 2.4 µV0 µV



Figure 11. AV-V MMN in sync condition (left), 70% condition (middle), and 20% condition

(right)

AV data did not contain peaks as clear as the A condition. When looking at the averaged data,

MMN is highest in a window between 350 and 600 ms (figure 12). Therefore, the average of

microvolts in the window of 350 to 600 ms was calculated as a measure of MMN peaks.

Because of this, latency could not be tested.

Figure 12. MMN on electrode CP5

A repeated measures ANOVA was performed, comparing the different SOA

conditions (sync, 70, and 20), the two hemispheres (left, right), and two electrodes (CP3 and



CP5 for left hemisphere, CP4 and CP6 for right hemisphere). This resulted in following

design: SOA (3) x hemisphere (2) x electrode (2). Results are displayed in table 5. No

significant results were found for SOA, hemisphere, electrode, or interactions. This indicates

that there are no, or only very small, differences between the different SOAs, the both

hemispheres, and the electrodes. Across SOAs, hemispheres and electrodes, mean differed

significantly from 0.

Table 5.

AV-V condition, sphericity test and repeated measures ANOVA

Sphericity Repeated Measures

χ2

p F p

SOA 1.92 .384 .83 .449

Hemisphere - - .43 .525

Electrode - - .08 .778

SOA*Hemisphere 8.72 .013 .14 .870

SOA*Electrode 10.51 .005 1.11 .324

Hemisphere*Electrode - - 1.25 .284

SOA*Hemisphere*Electrode 5.19 .075 .34 .716

Intercept - - 8.07 .014

If musical acuity has an effect on the perceived McGurk effect with increasing SOA, it

would be expected that for participants who scored high on musical acuity, MMN decreased

more over SOA, than for participants who scored low on musical acuity. To test this, a

difference variable was computed for electrodes CP5 and CP6 (because these electrodes

showed most MMN (figure 5) ). For this variable, the SOA 20% condition (most

asynchronous) was subtracted from the SOA 0 (synchronous) condition. These variables were

correlated to the music measures. Results are displayed in table 6. A significant correlation

was found for the first block of the music test and the difference variable of CP5 (r(12)=.546,

p=.043), and CP6 (r(12)=-.566, p=.035) (figure 13 and 14).



Table 6.

Correlations AV-V SOA0-20 difference variable for electrodes CP5, CP4 and music

measures.

Dif CP5 Dif CP6

Dif CP5 -

Dif CP4 -.073 -

MQ1 .378 .001

MQ2 .303 -.325

MT1 .546* -.566*

MT2 .275 -.127

MT3 .348 -.473

MTtotal .471 -.510

* p < .05

Figure 13. Correlation Dif CP5 and MT1 Figure 14. Correlation Dif CP6 and MT1

Discussion

Two experiments were conducted to examine whether musical acuity has an effect on

audiovisual synchrony perception and perceived McGurk effect at different SOAs. In the first

experiment this was examined on a behavioral level.

The averaged data from SJ and ID task showed similar curves as found by van

Wassenhove et al. (2007). The width of the curves was the most important measure to link to

musical acuity. The width was estimated using two different measures. These only correlated

for SJ task. When looking at the averaged curves, data from ID task shows more skewing than

SJ task. The first width measure was the SD from a gaussian fitted curve. The gaussian fit



could not account for skewing and therefore an alternative score (WS) was created. It is

therefore not surprising that WS and SD did not correlate in ID task.

The scores of the music questionnaire and music test did show some correlation, suggesting

that the measures were internally consistent and measured the same construct. The online test

was originally designed to measure congenital amusia in nonmusicians. Researchers found a

cut-off score (2 SD under mean score) for amusia of 73.7% for people under 40 years of age

(Peretz et al., 2008). When taking this into account, the variety of scores found in present

study (64-100% on MTtotal) was sufficient to make a distinction in participants’ musical

acuity and correlate this with the other measures.

For SJ task, negative correlations with WS and both music questionnaire scores, and the

second block of the music test were found. This suggests that participants who scored high

on musical acuity, had a smaller temporal window of integration, and thus were more prone to

detect asynchronies, than were participants who scored low on musical acuity. The second

block of the music test, tested for participants sense for rhythm, an ability that is of particular

importance for synchrony perception. For ID task, the correlation found with MT2 showed a

clear tendency towards significance (p= .050). The borderline result, may partly be due to the

very small sample in this correlation (n= 17). The current finding that musical acuity narrows

the temporal window of integration, making participants more prone to detect asynchronies, is

consistent with results from the study of Petrini et al. (2009). Their study showed smaller

curves for musicians, consistent with narrowed temporal windows of integration, when

compared to nonmusicians. Current finding suggest that this may apply to musical acuity in

addition to musical expertise.

In the second experiment it was examined whether the decrease of McGurk effect over

different SOAs could also be found in the brain, in MMN.



First there was an auditory block. Results show that participants had a normal MMN and P3

reaction (Näätanen et al., 2007). A significant correlation was found for Fz latency (MMN)

and MT1. Interestingly, the first block of the music test was about detecting a difference in

two sequential melodies. MMN is a preattentive reaction to a deviant stimulus, and therefore

it is interesting to find that participants who scored high on the detection of differences in

melodies, also had their MMN peak (as a result from a deviant tone) earlier than participants

that scored low on this block of the music test. Research shows that MMN latency on pitch

perception is decreased for musicians, compared to nonmusicians, suggesting that the

auditory system of musicians reacts faster to auditory changes. Research on differences in

MMN amplitude showed mixed results (Brattico, Näätänen, & Tervaniemi, 2002; Koelsch,

Schröger, & Tervaniemi, 1999; Tervaniemi, Just, Koelsch, Widmann, & Schröger, 2005). For

P3, a significant correlation was found between Cz latency and the third block of the music

test. In the third block of the music test, participants had to judge whether the melody

contained an out-of-tune note. The relation with the auditory stimuli is the perception of pitch.

This finding is consistent with researches that show decreased P3 latency in relation to pitch

for musicians, when compared to nonmusicians. Research on P3 amplitude differences

showed mixed results (Crummer, Hantz, & Chuang, 1988; Crummer et al. 1994;Tervaniemi et

al., 2005; Wayman et al. 1992). Current findings in auditory MMN and P3 latency suggest

that the faster reactions to change in auditory system may not only apply to musicians, but

also applies to nonmusicians with higher musical acuity.

In audiovisual data, MMN seemed to be most clear around electrodes CP5 and CP6.

Electrical activation in this electrodes did differ from 0 significantly. However, the different

SOA conditions did not differ significantly. No effect was found for hemisphere or electrodes

either. The limited findings in the AV ERP study can have multiple causes. Participants may

not have experienced McGurk effect. This is not a probable explanation, as only participants



were selected that experienced McGurk effect in the behavioral experiment. In the behavioral

experiment, however, the proportion variable that was used as an indication of perceived

McGurk effect not only consisted of /da/ (fusion) responses, but also contained /ga/ (visual)

responses. Therefore, it could be that the results of the behavioral experiment were found due

to a visual bias of the participants, rather than perceived McGurk effect. If this was true,

however, responses would not have been affected by increasing asynchronies, and behavioral

data showed clear decreases as a function of SOA. Because of the length of the ERP study

(which took approximately 3.5 hours), participants could have been affected by tiredness.

Further, participants may have been affected by habituation due to the large number of

perceived deviant stimuli in the AV conditions. This seems to be a good explanation, although

it would be expected that the habituation would occur in V condition too, and this was not the

case. In the experiment, there were only three V blocks, against nine AV blocks. This could

explain the difference in habituation.

If musical acuity has an effect on decreasing MMN over SOAs, it would be expected

that increasing musical acuity causes MMN to decrease more over SOA. This was hard to

measure, because participants all received stimuli that were adjusted to their own behavioral

ID curve. Participants all got stimuli adjusted to the most McGurk (peak of the curve), 70%,

and 20% point. Therefore, no major differences in MMN decrease would be expected if

behavioral and ERP data were an exact match. It could be however, that behavioral and ERP

data were no exact match, and therefore a difference variable for SOA 0 and 20% was

correlated to the music measures. On CP5, a positive relation was found with the first block of

the music test. A negative correlation was found with CP6, which is against expectations.

When looking at the scatterplot, it seems that this correlation is found mainly due to one

outlier in the data. Synchrony perception is of importance in the difference between SOA 0

and 20%, and therefore it would have made more sense if the second block of the music test



would show a correlation. Because of the very small differences between the SOA 0 and 20%

conditions (no significant differences were found between these AV conditions), it is difficult

to interpret currently found correlations.

Despite limited findings in EEG experiment, current study was thoroughly

constructed. Strengths of the behavioral study included the fact that musical acuity was

measured with an online test, in addition to a self-report. This test was especially designed for

nonmusicians, and is therefore a good measure. Also, multiple measures were used to

estimate the width of the curves, in order to find the best fit. The catch trials included in both

experiments were also a strength. Because of the amount of control in ERP experiment,

alternative explanations of possible findings were ruled out. For example, auditory blocks

served as an indication for found MMN and V blocks controlled for visual MMN. Further, by

choosing to measure only 3 SOAs, many trials per person per SOA were collected, and

together they enhanced the reliability of data.

General conclusion

Previous studies examined the influence of different SOAs on audiovisual speech

perception. Van Wassenhove et al. (2007), for example, examined perceived McGurk effect at

different SOAs. In this study, different temporal windows of integration are described, for

different conditions. Petrini et al. (2009) examined whether expertise can influence the width

of the temporal window of integration. This was the case for expert drummers who were

exposed to audiovisual stimuli of drummers, presented at different SOAs. In addition to this,

current study aimed to examine whether this effect can also be found for participants’ innate

sense of pitch and rhythm: musical acuity, and whether this can be found in audiovisual

integration of speech too, using the McGurk effect. The results of current study, provide

evidence that not just musical expertise, but also musical acuity has an influence on the

perception of synchrony in audiovisual stimuli. Also, a tendency towards significance was



found suggesting that musical acuity may have an influence on perceived McGurk effect,

suggesting that the effect of musical acuity may apply to the perception of speech too. In ERP

experiment, the effect of different asynchronies on MMN could not be found.



References

Brattico, E., Näätänen, R., & Tervaniemi, M. (2002). Context effects on pitch perception in

musicians and nonmusicians: Evidence from event-related-potential recordings.

Music Perception, 19 (2), 199-222.

Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C., McGuire,

P. K., Woodruff, P. W., Iversen, S. D., & David, A. S. (1997). Activation of auditory

cortex during silent lipreading. Science 276 (5312), 593–596.

Calvert, G. A., Hansen, P. C., Iversen, S. D., & Brammer, M. J. (2001). Detection of audio-

visual integration sites in humans by application of electrophysiological criteria to the

BOLD effect. NeuroImage, 14, 427-438.

Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. (2002). Mismatch

negativity evoked by the McGurk-MacDonald effect: a phonetic representation within

short-term memory. Clinical Neurophysiology, 113, 495-506.

Crummer, G. C., Hantz, E., Chuang, S. W. (1988). Neural basis for music cognition: Initial

experimental findings. Psychomusiology, 7 (2), 117-126.

Crummer, G. C., Walton, J. P., Wayman, J. W., Hantz, E. C., & Frisina, R. D. (1994). Neural

processing of musical timbre by musicians, nonmusicians, and musicians possessing

absolute pitch. Journal of the Acoustical Society of America, 95 (5), 2720-2727

Diederich., A. & Colonius, H. (2004). Bimodal and trimodal multisensory enhancement:

effects of stimulus onset and intensity on reaction time. Perception & Psychophysics,

66 (8), 1388–1404.

Dixon, N., & Spitz, L. (1980). The detection of audiovisual desynchrony. Perception, 9, 719–

721.

Grant, K. W., van Wassenhove, V., & Poeppel, D. (2004). Detection of auditory (cross-



spectral) and auditory-visual (cross-modal) synchrony. Speech Communication, 44,

43-53.

Gratton, G., Coles, M. G., & Donchin, E. (1983). A new method for off-line removal of

ocular artifact. Electroencephalography & Clinical Neurophysiology, 55(4), 468–484.

International Laboratory for Brain, Music and Sound Research. (2012). Retrieved from

http://www.brams.umontreal.ca/amusia-general/

Jones, J. A., & Munhall, K. G. (1997). The effects of separating auditory and visual sources

on audiovisual integration of speech. Canadian Acoustics, 25(4), 13-19

Kayser, C., Petkov, C. I., & Logothetis, N. K. (2009). Multisensory interactions in primate

auditory cortex: fMRI and electrophysiology. Hearing Research, 258, 80-88.

Koelsch, S., Schröger, E., & Tervaniemi, M. (1999). Superior pre-attentive auditory

processing in musicians. NeuroReport, 10, 1309-1313.

McGurk, H., & McDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–747.

Meredith, M. A., & Stein, B. E. (1985). Descending efferents from the superior colliculus

relay integrated multisensoryinformation. Science, 227, 657-659.

Meredith, M. A., & Stein, B. E. (1986a). Spatial factors determine the activity of multisensory

neurons in cat superior colliculus. Brain Research, 365, 350–354.

Meredith, M. A., & Stein, B. E. (1986b). Visual, auditory, and somatosensory convergence on

cells in superior colliculus results in multisensory integration. Journal of

Neurophysiology, 56, 640–662.

Miller, J. (1982). Divided attention: Evidence for coactivation with redundant signals.

Cognitive Psychology, 14, 247-279.

Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the

McGurk effect. Perception & Psychophysics, 58(3), 351-362.

Mussacchia, G., & Schroeder, C. E. (2009). Neuronal mechanisms, response dynamics and



perceptual functions of multisensory interactions in auditory cortex. Hearing

Research, 258, 72-79.

Näätänen, R., Simpson, M., & Loveless, N. E. (1982). Stimulus deviance and evoked

potentials. Biological psychology, 14, 53-98.

Näätänen, R., Paavilainen, P., Rinne, T., & Alho, K. (2007). The mismatch negativity (MMN)

in basic research of central auditory processing: A review. Clinical Neurophysiology,

118, 2544-2590.

Petrini, K., Dahl, S., Rocchesso, D., Waadeland, C. H., Avanzini, F., Puce, A., Pollick, F. E.

(2009). Multisensory integration of drumming actions: musical expertise affects

perceived audiovisual asynchrony. Exp Brain Res, 198, 339–352

Pekkola, J., Ojanen, V., Autti, T., Jääskeläinen, I. P., Möttönen, R., Tarkiainen, A., & Sams,

M. (2005). Primary auditory cortex activation by visual speech: an fMRI study at 3T.

Neuroreport, 16 (2), 125-128.

Peretz, I., Gosselin, N., Tillmann, B., Cuddy, L. L., Gagnon, B.,Trimmer, C. G., Paquette, S.,

& Bouchard, B. (2008). On-line identification of congenital amusia. Music Perception,

25(4), 331-343.

Radeau, M. (1994). Auditory-visual spatial interaction and modularity. Current Psychology of

Cognition, 13, 3-51.

Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. (2007). Do you see

what I am saying? Exploring visual enhancement of speech comprehension in noisy

environments. Cerebral Cortex, 17(1), 1147-1153.

Saint-Amour, D., De Sanctis, P., & Molholm, S. (2007). Seeing voices: high-density electrical

mapping and source-analysis of the multisensory mismatch negativity evoked during

the mcGurk illusion. Neuropsychologica, 45, 587-597.

Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S. T., & Simola, J.



(1991). Seeing speech: visual information from lip movements modifies activity in the

human auditory cortex. Neuroscience Letters, 127,141–145.

Sams, M., Paavilainen, P., Alho, K., & Näätänen, R. (1985). Auditory frequency

discrimination and event-related potentials. Electroencephalography and clinical

Neurophysiology, 62, 437-448.

Schwartz, J. L., Berthommier, F., & Savariaux, C. (2004). Seeing to hear better: evidence for

early audio-visual interactions in speech perceptions. Cognition, 93, 69-78.

Schröger, E., & Widmann, A. (1998). Speeded responses to audiovisual signal changes result

from bimodal integration. Psychophysiology, 35, 755-759.

Spence, C., & Squire, S. (2003). Multisensory integration: Maintaining the perception of

synchrony. Current Biology, 13, 519–52.

Stein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade, L. (1989). Behavioral indices of

multisensory integration: orientation to visual cues is affected by auditory stimuli.

Journal of Cognitive Neuroscience, 1(1), 12-24.

Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: current issues from the

perspective of the single neuron. Nature Reviews Neuroscience , 9, 255-266.

Sugita, Y., & Suzuki, Y. (2003). Audiovisual perception: Implicit estimation of sound-arrival

time. Nature, 421, 911.

Tervaniemi, M., Just, V., Koelsch, S., Widmann, A., & Schröger, E. (2005). Pitch

discrimination accuracy in musicians vs nonmusicians: an event-related potential and

behavioral study. Exp Brain Res, 161, 1-10.

van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in

auditory-visual speech perception. Neuropsychologica, 45, 598-607.

Wayman, J. W., Frisina, R. D., Walton, J. P., Hantz, E. C., & Crummer, G. C. (1992). Effects



of musical training and absolute pitch ability on event-related activity in response to

sine tones. Journal of the Acoustical Society of America, 91 (6), 3527–3531.

Wurtz, R. H., & Albano, J. E. (1980). Visual-motor function of the primate superior

colliculus. Ann. Rev. Neurosci., 3, 189-226.

THE EFFECT OF MUSICAL ACUITY ON AUDIOVISUAL …

Documents

Transcript of THE EFFECT OF MUSICAL ACUITY ON AUDIOVISUAL …