Joint Encoding of Auditory Timing and Location in Visual ...

Joint Encoding of Auditory Timingand Location in Visual Cortex

John Plass1, EunSeon Ahn1, Vernon L. Towle2, William C. Stacey1, Vibhangini S. Wasade3,James Tao2, Shasha Wu2, Naoum P. Issa2, and David Brang1

Abstract

■ Co-occurring sounds can facilitate perception of spatiallyand temporally correspondent visual events. Separate lines ofresearch have identified two putatively distinct neural mecha-nisms underlying two types of crossmodal facilitations: Whereascrossmodal phase resetting is thought to underlie enhancementsbased on temporal correspondences, lateralized occipitalevoked potentials (ERPs) are thought to reflect enhancementsbased on spatial correspondences. Here, we sought to clarifythe relationship between these two effects to assess whetherthey reflect two distinct mechanisms or, rather, two facets ofthe same underlying process. To identify the neural generatorsof each effect, we examined crossmodal responses to lateralizedsounds in visually responsive cortex of 22 patients using electro-corticographic recordings. Auditory-driven phase reset andERP responses in visual cortex displayed similar topography,revealing significant activity in pericalcarine, inferior occipital–

temporal, and posterior parietal cortex, with maximal activityin lateral occipitotemporal cortex (potentially V5/hMT+).Laterality effects showed similar but less widespread topogra-phy. To test whether lateralized and nonlateralized componentsof crossmodal ERPs emerged from common or distinct neuralgenerators, we compared responses throughout visual cortex.Visual electrodes responded to both contralateral and ipsilateralsounds with a contralateral bias, suggesting that previouslyobserved laterality effects do not emerge from a distinct neuralgenerator but rather reflect laterality-biased responses in thesame neural populations that produce phase-resetting responses.These results suggest that crossmodal phase reset and ERPresponses previously found to reflect spatial and temporal facili-tation in visual cortex may reflect the same underlying mecha-nism. We propose a new unified model to account for theseand previous results. ■

INTRODUCTION

Co-occurring sounds can facilitate the perception of spa-tially or temporally correspondent visual events. Salientlateralized sounds can enhance the detection and dis-crimination of colocated visual targets (Lu et al., 2009;Driver & Spence, 2004; McDonald, Teder-Sälejärvi, &Hillyard, 2000; Spence & Driver, 1997), whereas spatiallyuninformative sounds can enhance the detection, discrim-ination, and perceived temporal dynamics of co-occurringvisual stimuli (Chen, Huang, Yeh, & Spence, 2011;Fiebelkorn, Foxe, Butler, Mercier, et al., 2011; Jaekl &Soto-Faraco, 2010; Noesselt et al., 2010; Shams, Kamitani,& Shimojo, 2002; Shipley, 1964). However, it is currentlyunclear whether audiovisual interactions based on spa-tial and temporal correspondences are subserved bythe same or distinct neural mechanisms for crossmodalenhancement.

Separate lines of research have previously identifiedtwo putatively distinct mechanisms thought to underliecrossmodal facilitations based on each type of audiovisualcorrespondence. On the one hand, crossmodal phase

resetting (i.e., auditory resetting of oscillations in visualcortex) is thought to facilitate visual perception for tem-porally correspondent stimuli by placing visual cortex in ahigh-excitability state before visual signals arrive (Mercieret al., 2013; Romei, Gross, & Thut, 2012; Naue et al., 2011;Lakatos et al., 2009; for a review, see Thorne & Debener,2014). Because this mechanism produces transient in-creases in visual cortical sensitivity that are time-lockedto auditory events, it can be seen as primarily producingcrossmodal enhancements based on temporal correspon-dences between auditory and visual stimuli.On the other hand, interhemispheric laterality differ-

ences in occipital ERPs produced by spatialized soundsare thought to reflect hemisphere-specific crossmodal ex-citation (or suppression) of activity in visual cortex(Campus, Sandini, Morrone, & Gori, 2017; Matusz, Retsa,& Murray, 2016; Brang et al., 2015; Feng, Störmer,Martinez, McDonald, & Hillyard, 2014; McDonald,Störmer, Martinez, Feng, & Hillyard, 2013; for a review,see Hillyard, Störmer, Feng, Martinez, & McDonald, 2016).By modulating visual activity in a hemisphere-specificmanner, these effects are thought to selectively enhancethe neural encoding of spatially correspondent visual stim-uli, potentially reflecting a mechanism for the crossmodal

1University of Michigan, 2The University of Chicago, 3HenryFord Hospital

© 2019 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 31:7, pp. 1002–1017doi:10.1162/jocn_a_01399

orienting of exogenous visuospatial attention (Hillyardet al., 2016).Differences in the apparent timing and neural topogra-

phy of the effects associated with each putative mechanismhave led researchers to study and interpret them primarilyas independent processes. Whereas activity associated withcrossmodal phase resetting is typically observed 20–150msecafter sound onset (Mercier et al., 2013; Romei et al., 2012;Naue et al., 2011; Lakatos et al., 2009), lateralized ERPdifferences during this period are typically attributed toauditory neural generators, with only later (150–400 msec)laterality effects typically being localized to visual cortex(Matusz et al., 2016; Feng et al., 2014; McDonald et al.,2013). Moreover, crossmodal phase resetting is typicallyobserved in or localized to low-level visual cortex, includingprimary visual cortex (Naue et al., 2011; Lakatos et al.,2009), whereas lateralized ERP differences are typicallylocalized to ventral–lateral occipital regions associated withhigher order visual processing (Matusz et al., 2016; Fenget al., 2014; McDonald et al., 2013).However, additional findings call into question a strict

neural dissociation between these two types of facilita-tion. First, whereas most scalp-recorded EEG studiestend to show the clearest laterality effects at later timepoints, at least one has found robust lateralized occipitalresponses as early as 50–100 msec after sound onset, spe-cifically when the locations of auditory stimuli were taskrelevant (Campus et al., 2017). Early occipital lateralitydifferences have also been observed in other studiesusing scalp-recorded EEG (e.g., McDonald et al., 2013)but have been attributed to lateralized sources in audi-tory cortex on the basis of source localization analyses.However, given the limited spatial resolution of scalp-recorded EEG and potential weakness of EEG source lo-calization procedures (Bradley, Yao, Dewald, & Richter,2016), it is possible that these early laterality differencesare at least partially attributable to lateralized responsesin visual cortex. Second, source localizations of late-lateralized responses have not always unambiguouslyidentified ventral–lateral occipital sources, with somesource estimates including more medial sources poten-tially corresponding to low-level visual regions implicatedin studies of phase resetting (Matusz et al., 2016; Fenget al., 2014). Finally, we have recently observed both early(30–150 msec after sound onset)- and late (300–500 msec)-lateralized ERP responses to sounds in depth electroderecordings from low-level pericalcarine visual cortex (pu-tative V1/V2) in two human patients with epilepsy (Branget al., 2015). Thus, the putative mechanisms subservingcrossmodal facilitations based on spatial versus temporalcorrespondences may not be as clearly dissociable as theliterature exploring these phenomena independentlyappears to suggest.To examine the relationship between these two putative

mechanisms more closely, we used subdural and stereo-tactic electrocorticographic (ECoG) recordings frompatients with epilepsy to examine the topography and

timing of crossmodal phase resetting, bilaterally averagedERPs, and ERP laterality differences evoked by sounds invisual cortex. Consistent with our previous research usingcentrally presented sounds (Brang et al., under review),lateralized noise bursts produced widespread phase reset-ting throughout visually responsive cortex, including peri-calcarine, lateral occipital, inferior occipital–temporal,and posterior parietal cortex, with maximal activity inan occipitotemporal region potentially correspondingto area V5/hMT+. Averaging across contralateral andipsilateral sound conditions, ERP responses showed asimilar topography as the phase-resetting response,suggesting that the bilaterally averaged ERP responseand crossmodal phase resetting likely reflect the samecrossmodal activity.

To evaluate the relationship between this average bilat-eral response and laterality differences in crossmodalresponses to sounds, we compared the cortical topogra-phy, magnitude, and time course of ERPs produced bycontralaterally and ipsilaterally presented sounds in visualcortex. Laterality effects first emerged in pericalcarineand lateral occipitotemporal regions (50–150 msec) andspread to inferior occipitotemporal and posterior parietalregions over longer durations (150–400 msec), mirroringthe topography of the phase reset and bilaterally aver-aged ERP responses. The majority of electrodes exhibit-ing lateralized responses produced significant responsesto sounds in both hemifields, and no distinct region thatconsistently responded exclusively to either hemifieldcould be identified. Additionally, in participants withwidespread coverage over visual cortex, the size of ipsi-lateral and contralateral responses were highly correlatedacross electrodes during both early and late responseperiods, but with a contralateral bias.

Together, these results suggest that crossmodal phaseresetting, bilateral ERPs, and ERP laterality effects ob-served in response to sounds are generated by the sameneural populations in visual cortex and, therefore, that acommon mechanism may facilitate visual perception onthe basis of spatial and temporal correspondences withsounds. Response laterality does not appear to arise froma distinct neural generator that responds exclusively tocontralateral sounds but from a contralateral bias in neu-ral populations that respond to sounds in visual cortexmore generally. This result is consistent with a modelof crossmodal interactions in which the properties of au-ditory cortical responses to sounds, including responselaterality, are inherited by visual cortex, without the aidof an additional mechanism specialized for crossmodalconveyance of spatial information.

METHODS

Participants

Twenty-two patients with epilepsy participated in thisstudy during invasive work-up for medically intractable

Plass et al. 1003

seizures using ECoGmonitoring from chronically implanteddepth electrodes (5 mm center-to-center spacing, 2 mmdiameter) and/or subdural electrodes (10 mm center-to-center spacing, 3 mm diameter). Participants rangedin age from 15 to 56 (mean = 34.2, SD = 12.6) and in-cluded eight females. Electrodes were placed accordingto the clinical needs of the participants. Written consentwas obtained from each participant according to the di-rection of the institutional review boards at the Universityof Michigan, University of Chicago, and Henry FordHospitals.

MRI and CT Acquisition and Processing

A preoperative T1-weighted MRI and a postoperative CTscan were acquired for each participant to aid in localiza-tion of electrodes. Cortical reconstruction and volumetricsegmentation of each participant’s MRI was performedwith the Freesurfer image analysis suite (surfer.nmr.mgh.harvard.edu/; Dale, Fischl, & Sereno, 1999; Fischl,Sereno, & Dale, 1999). Postoperative CT scans were reg-istered to the T1-weighted MRI through SPM, and elec-trodes were localized along the Freesurfer corticalsurface using customized open-source software devel-oped in our laboratories (Brang, Dai, Zheng, & Towle,2016; available for download online https://github.com/towle-lab/electrode-registration-app/). This software seg-ments electrodes from the CT by intensity values andprojects the normal tangent of each electrode to the durasurface, avoiding sulcal placements and correcting forpostimplantation brain deformation present in CT images.

Lateralized Sounds Paradigm

Participants were seated in a hospital bed. Auditorystimuli were delivered via a laptop using PsychToolbox(Brainard, 1997; Pelli, 1997) through a pair of free-fieldspeakers placed approximately 45° to the right and leftof participants’ midline. The laptop and speakers wereplaced on a mobile overbed table, producing a viewingdistance of approximately 80 cm. Two variants of the taskwere utilized across participants. Data were combinedacross tasks because previous studies have demonstratedsimilar crossmodal responses in both tasks (Feng et al.,2014; McDonald et al., 2013) and because we observedhighly similar crossmodal responses in patients whocompleted both tasks. In Task A, participants were pre-sented with one of three sounds on each trial: a 53-msec1000-Hz sinewave tone presented from both speakers si-multaneously and thus localized centrally (15 trials/block)or a 83-msec pink noise burst presented from either theleft (30 trials/block) or right speaker (30 trials/block).Participants completed between two and six blocks.Stimuli were selected for consistency with the paradigmused by McDonald et al. (2013; Experiment 4). A centralfixation cross was displayed on the laptop throughout theexperiment. Participants were instructed to maintain

central fixation and to respond via button press to thecentral 1000-Hz tone while making no response to theperipheral noise bursts (for which ECoG responses wereanalyzed). The ISI varied randomly between 2.0 and2.5 sec (uniform distribution). Task B was based on thetask used by Feng et al. (2014). The same 83-msec pinknoise burst used in Task A was presented from either theleft (60 trials/block) or right speaker (60 trials/block),with no central tone trials. At 400 msec after the presen-tation of a sound, a 50-msec duration visual letter (L or T)was presented on either the left or right side of thescreen, followed by a 100-msec visual masking stimulus;sounds were not statistically predictive of the location ofthe subsequent visual target. Participants were instructedto identify the L or T stimulus via button press. The inter-trial interval varied randomly between 1.65 and 2.25 sec(uniform distribution). Participants completed betweentwo and four blocks. Data from the visual portion ofthe trial were not examined for the purpose of this study,and analyses were restricted to the time period beforetheir onset. Six participants completed Task A only, 11completed Task B only, and 5 completed both.

ECoG Recordings and Analysis

ECoG recordings were acquired at 1024 Hz (14 partici-pants), 4096 Hz (5 participants), or 1000 Hz (3 partici-pants) due to differences in the clinical amplifiers used.Data recorded at 4096 Hz were down-sampled to1024 Hz during the initial stages of processing. The onsetof each trial was denoted online by a voltage-isolated TTLpulse. To ensure that electrodes reflected maximally localand independent activity from one another, we used bi-polar referencing. Noisy channels, defined as those con-taining epileptic spiking (manually identified) or with anoverall variance exceeding 5 SDs, were removed fromanalyses. Similarly, noisy trials, defined as those with anoverall variance exceeding 3 SDs, were removed fromanalyses. These values were selected to match those usedin our previous research using centrally presentedsounds (Brang et al., under review). These values arethe default used by our group and are based on the ap-proach used by other groups as well (e.g., Jiang et al.,2017; Jacques et al., 2016), with values typically rangingfrom 3 to 5 SDs for both channel and trial rejection.Across both tasks, 4.7% (SD = 3.3%) of trials were re-jected on average, resulting in the analysis of between112 and 656 trials across participants (M = 252.0 trialsper participant, SD = 151.9). Following the rejection ofartifactual channels and trials, data were high-pass filteredat 0.01 Hz to remove slow drift artifacts and notch-filteredat 60 Hz and its harmonics to remove line noise. Datawere then segmented into 4-sec duration epochs (−2to 2 sec around the onset of the sound).Specific analyses applied are described in-line through-

out the Results section. In general, data were subjectedto measures of the ERP in which the raw voltage time

1004 Journal of Cognitive Neuroscience Volume 31, Number 7

series from each trial are averaged across trials in a time-locked manner. ERP data were baselined relative to the500 msec before sound onset (prestimulus period rang-ing from −500 to 0 msec). Two-tailed one-sample t testswere used to examine whether single ERP conditions dif-fered from zero at any time point following sound onset.To control for statistical tests conducted at multiple timepoints, multiple comparison corrections were applied ateach channel using maximum statistics (Holmes, Blair,Watson, & Ford, 1996). In specific, a distribution contain-ing 10,000 permuted values from the data (using eithercondition label swapping for two-sample t tests or signswapping for one-sample t tests) were generated for eachelectrode and time point. Next, a t test (either one-sampleor two-sample depending on the comparison) was thenconducted at each time point, and the maximum t valuefor each electrode was taken from each permutation, re-sulting in a null distribution of 10,000 t values for eachelectrode. Finally, the upper and lower 2.5% of this nulldistribution were taken as critical t values; only t valuesin the real data exceeding these thresholds were consid-ered statistically significant. Critically, this test controlsthe family-wise error rate at p= .05, indicating that purelyrandom data would survive this multiple comparison cor-rection at a rate of 5%. To control for statistical tests beingapplied across many channels and participants, the mini-mum multiple-comparison corrected p value from eachelectrode was then false discovery rate (FDR)-corrected(q = .05) across all electrodes and participants (Groppe,Urbach, & Kutas, 2011).Intertrial phase clustering (ITPC) analyses reflect the

consistency of intrinsic oscillatory phase angles across tri-als, providing a general index of phase resetting. ITPCvalues were computed using nine wavelets (center fre-quencies ranging from 4 to 20 Hz at 2-Hz intervals, usinga 750-msec Gaussian temporal window at each frequency).Instantaneous phase angles were calculated at each timepoint, frequency bin, and trial from the resultant waveletconvolutions. ITPC values were calculated at each timepoint and frequency as the magnitude of the complexaverage of the phase angle vectors across trials between0 and 250 msec following sound onset; this restrictedtime period was used to avoid temporal smoothing past400 msec—the time at which a visual stimulus was pre-sented in Task B. Values were then averaged over the250-msec time period and across all frequencies yieldinga single value for each condition/electrode indexing sta-ble phase locking. ITPC values are bound between 0(uniform phase angle distribution) and 1 (identicalphase angles across trials). To identify significant ITPCvalues for each electrode, a null distribution (10,000 per-mutations) was constructed from phase-shuffling the an-gle of filtered data before the calculation of ITPC values.In specific, a random value ranging from −pi to +pi (uni-formly sampled) was added to each trial before calcula-tion of ITPC in each permutation iteration. Critically, oneach permutation, only a single random phase offset was

applied to all time points and frequencies to maintainspectrotemporal dependencies in the data. To evaluatethe statistical reliability of ECoG phase resetting, we com-puted the difference in ITPC values between the realstimulus-related data and data obtained from each per-mutation in the null distribution, counting the numberof permutations with ITPC values exceeding those ofthe real data; electrodes at which <5% of the permutedITPC values exceeded the real ITPC values were consid-ered statistically significant. As in the ERP analyses, thismethod controls the family-wise error rate at 5%. FDRmultiple comparison corrections (q = .05) were thenapplied across all electrodes and participants.

Selection of Visual Electrodes

Electrodes included for analyses are displayed inFigure 1. Visual electrodes were limited to those locatedin occipital, parietal, or temporal areas (excluding thesuperior temporal gyrus) and showing a significant ERP( p < .05, multiple comparison corrected) to visual stim-uli beginning at less than 200 msec using a separate visuallocalizer task that presented participants with complexvisual stimuli (e.g., faces, objects, scenes). All electrodeswere projected onto the left hemisphere of an MNI-152brain for visualization.

RESULTS

Previous research has identified two potential mecha-nisms for auditory facilitation of visual cortical process-ing: Whereas crossmodal phase resetting is thought tounderlie enhancements based on audiovisual temporalcorrespondences (Thorne & Debener, 2014), interhemi-spheric differences in occipital ERPs are thought toreflect enhancements based on audiovisual spatial corre-spondences (Hillyard et al., 2016). However, the relation-ship between these putatively distinct effects is unclear.To clarify whether these effects reflect related or distinctprocesses, we leveraged the excellent spatial resolutionprovided by densely sampled intracranial recordings tocompare the cortical topographies and time courses ofcrossmodal phase resetting, bilaterally averaged ERPs,and ERP laterality differences observed in response tosounds in visual cortex. Averaging across contralateraland ipsilateral sound conditions, we first tested whetherbilaterally averaged ERP responses to sounds exhibited asimilar neural topography as the auditory phase resetresponse in visual cortex. Then, we performed multipletests to evaluate whether this bilaterally averaged re-sponse and lateralized ERP differences were generatedby distinct neural populations that separately encodedspatial and nonspatial stimulus information or reflecteda common laterality-biased response from the sameneural generators.

Plass et al. 1005

Comparison of Crossmodal Phase Reset Responsesand Auditory-Evoked Visual ERPs

Because event-driven phase resetting can generate ERPs(Klimesch, Sauseng, Hanslmayr, Gruber, & Freunberger,2007; Sauseng et al., 2007), we first tested to see whetherthe overall ERP response to sounds, averaged acrosscontralateral and ipsilateral conditions, could plausiblybe attributed to crossmodal phase resetting. To test thehypothesis that the bilaterally averaged ERP response wasproduced by the same neural generators as the cross-modal phase reset response in visual cortex, we com-pared the spatial distributions of visual electrodes thatdisplayed ERPs to the spatial distribution of those thatdisplayed significant phase clustering (ITPC) in responseto lateralized sounds. Both analyses were performed bypooling ipsilateral and contralateral responses and there-fore reflected the nonlateralized component of the cross-modal response to lateralized sounds.

Consistent with our previous research using central-ized sounds (Brang et al., under review), lateralized noisebursts produced widespread phase clustering throughoutvisually responsive cortex, indicating widespread cross-modal phase resetting distributed across multiple visualregions (Figure 2A). Altogether, 153 of 313 (48.9%) visualelectrodes showed significant ITPC during the 250 msecfollowing sound onset. Significant electrodes were ob-served in pericalcarine, lateral occipital, inferior occipital–temporal, and posterior parietal cortex, with maximalactivity in an occipitotemporal region posterior to the

middle temporal gyrus. The anatomical location of thislatter area was consistent with the expected location ofvisual area V5/hMT+, though we could not verify the func-tional specificity of this area using motion-based localizers.The close correspondence between the cortical topog-raphy of this response and that previously observed tocentralized sounds suggests that salient sounds producewidespread phase resetting throughout visual cortexregardless of stimulus location.ERP responses to the same stimuli displayed a highly

similar cortical topography (Figure 2B), suggesting thatthe average ERP response and crossmodal phase resetresponse likely reflect the same neural process. Onehundred ninety-five of 313 (62.3%) of visual electrodesshowed significant ERPs within 400 msec of sound onset.As in the phase clustering analysis, widespread auditory-driven ERPs were observed throughout visually respon-sive cortex, with close correspondences between themost active regions observed in each analysis. These re-sults suggest that at least the nonlateralized aspects ofcrossmodal ERPs elicited by spatialized sounds likely re-flect the same underlying process as the crossmodalphase reset response typically observed in nonspatialauditory tasks. However, this does not rule out the pos-sibility of a separate neural population that responds ex-clusively to sounds in one hemifield, thereby providingan independent mechanism for enhancements basedon spatial correspondence. To assess the possibility ofsuch a mechanism, we compared the time course and

Figure 1. (A) Intracranial electrodes from 22 patients, displayed on an average brain (all electrodes projected into the left hemisphere). Each black orcolored circle reflects a single electrode contact included in analyses, localized to visual areas (313 electrodes included in total). Electrodes wererestricted to those located in occipital, parietal, or inferior/posterior temporal areas (excluding the superior temporal gyrus) and showing a significantERP to visual stimuli beginning at less than 200 msec. Color-coded electrodes correspond to data in panel B and Figures 4 and 5. (B) ERP responsesfrom representative electrodes evoked during visual task. Electrode numbers are located in the top left of each panel and correspond to the samenumbers in Figures 4 and 5. Visual stimulus onset is at 0 sec (vertical dotted line). Shaded error bars reflect 95% confidence intervals.


Figure 2. Colored electrodes reflect significant (multiple comparison corrected for time points and electrodes) activity according to either (A) ITPCor (B) ERP analyses. Sounds activate visual cortex broadly and in a similarly distributed manner across both analyses, suggesting ERP and ITPCmeasures reflect similar mechanisms. The second row of images in each section reflects the same data on partially transparent brains to showstatistics from electrodes not visible at the surface of the brain. The third row shows data from the individual electrodes shown in Figure 1B.

Plass et al. 1007

Figure 3. Colored electrodes reflect significant (multiple comparison corrected for time points and electrodes) differences between contralateraland ipsilateral sounds from either (A) 0–400 msec, (B) 50–150 msec, or (C) 150–400 msec. (B, C) Black electrodes reflect nonsignificant differences inone of the two time periods. The second row of images in each section reflects the same data on partially transparent brains to show statisticsfrom electrodes not visible at the surface of the brain.


topographies of bilaterally averaged ERP responses andERP laterality differences to assess whether they revealedtwo distinct regions that produced nonlateralized (i.e.,spatially uninformative) and lateralized (i.e., spatiallyinformative) responses to sounds.

Identifying the Source of Response Lateralizationin Auditory-Evoked Visual ERPs

If response laterality associated with crossmodal spatialfacilitation and nonlateralized responses associated withtemporal facilitation through phase resetting reflect dis-tinct neural mechanisms, then they would be expectedto arise from distinct neural generators. By contrast, iflateralized and nonlateralized components of visual cor-tex’s response to sounds reflect two aspects of the samespatially biased response, they should arise from a com-mon neural generator displaying a lateralality bias.

To test the hypothesis that these two components arisefrom the same underlying mechanism, we comparedthem in a series of tests evaluating their topographies,magnitudes, and time courses. First, we compared thecortical topography of electrodes exhibiting bilaterallyaveraged ERP responses to the topography of thoseexhibiting laterality differences between contralateraland ipsilateral stimulus conditions. Overall, lateralizedERP differences were less widespread than bilaterallyaveraged responses, with 34 of 313 electrodes (10.9%)showing significantly different responses to contralateralcompared with ipsilateral sounds (Figure 3A). This topo-graphical difference may reflect the smaller effect sizes ofthe laterality differences relative to the average nonlater-alized response, as observed in scalp EEG (e.g., McDonaldet al., 2013), but likely also reflects decreased statisticalpower due to the division of trials into two conditions.Still, sites displaying significant laterality effects clustered

Figure 4. ERPs at visualelectrodes showing significantdifferences betweencontralateral (red) andipsilateral (blue) sounds.Electrode numbers are locatedin the top left of each panel.Sound onset is at 0 sec (verticaldotted line). Shaded color barsreflect 95% confidence intervals.Black bars at bottom of plotsshow time points at which thetwo conditions significantlydiffered from one another(corrected for multiplecomparisons). Asterisksreflect significant one-samplet tests (corrected for multiplecomparisons) for eithercontralateral (red) oripsilateral (blue) sounds.

Plass et al. 1009

closely around the lateral occipitotemporal, posterior pa-rietal, and inferior occipitotemporal regions that showedthe strongest nonlateralized component in our previousanalysis, suggesting the possibility of a common neuralgenerator.

Because previous studies using scalp-recorded EEGhave shown temporally distinct “early” (50–150 msec)and “late” (150–400 msec) laterality effects, we also ana-lyzed the topography of laterality differences duringthese two periods separately. Responses during the earlyperiod clustered around pericalcarine cortex and the lat-eral occipitotemporal region (potentially correspondingto area V5/hMT+) that showed the strongest effects in ouranalysis of the averaged bilateral response (Figure 3B),

suggesting that the visual cortical response to soundsmay originate from these sites. Responses during the lateperiod included this region as well, with additional activityin dorsal (posterior parietal) and ventral (inferior occipito-temporal) sites, suggesting a pattern of spreading activa-tion throughout visual cortex.Although these results suggest a general spatial corre-

spondence between the neural generators of lateralitydifferences and the average bilateral response, theirrelationship can be evaluated more directly by testingwhether electrodes showing a laterality effect respondexclusively to sounds in one hemifield or respond toboth contralateral and ipsilateral sounds with a lateralitybias. If laterality effects are produced by a distinct neural

Figure 5. Locations of electrodes (red) corresponding to ERPs shown in Figure 4, highlighting areas at which significant differences betweenipsilateral and contralateral ERPs were observed.


generator, then most electrodes displaying laterality ef-fects should respond exclusively to sounds presentedto one side. By contrast, if laterality effects result fromlaterality biases in the neural generators that producethe average bilateral response, then most electrodesshowing laterality effects should respond to both contra-lateral and ipsilateral sounds, but with a spatial bias.Consistent with our hypothesis, the majority of elec-trodes displaying laterality effects (26 of 34, 76.5%)responded to both contralateral and ipsilateral sounds,but with a laterality bias (Figure 4); 6 of 34 (17.6%)only showed significant ERP responses to contralateralsounds; and 2 of 34 (5.9%) only showed significantERP responses to ipsilateral sounds. Figure 5 shows thelocation of each individual electrode.Whereas this analysis indicates that most electrodes

displaying a lateralized response responded to both con-tralateral and ipsilateral sounds, it does not rule out thepossibility of there being a small region in visual cortexthat responds selectively to sounds presented only to aparticular side. To evaluate this possibility, we visualizedthe locations of electrodes that displayed significant ERPsto contralateral, ipsilateral, or both types of sounds(Figure 6). Qualitatively, we could not identify any regiondisplaying consistent selectivity for either contralateral oripsilateral sounds. Rather, each major region identified inour previous analyses displayed mostly bilateral re-sponses intermixed with spatially dispersed laterality-selective responses.To evaluate this result in a more quantitative manner,

we compared the effect sizes (Cohen’s d) of ERPs pro-duced by contralateral and ipsilateral sounds across allelectrodes present in four individual participants’ brains.

These participants were selected because they each hadat least 19 implanted electrodes, providing at least 80%power to detect correlations with coefficient r= 0.6, witha Type I error rate of a = 0.05. Electrode locations foreach participant are shown in Figure 7A.

If lateralized and nonlateralized aspects of crossmodalERPs arise from a common neural generator with a later-ality bias, then effect sizes for contralateral and ipsilateralsounds should be positively correlated across electrodes.By contrast, if there are distinct neural populations thatrespond selectively to sounds on a particular side, effectsizes should be negatively correlated or uncorrelatedacross electrodes. Notably, this analysis allows for a morefine-grained comparison of contralateral and ipsilateralresponses than the previous analyses showing a similardistribution of significant electrodes, because it doesnot involve statistical thresholding before comparison.

Cohen’s d values were calculated at each time pointseparately for contralateral and ipsilateral conditions bydividing the absolute value of the mean voltage at eachtime point by the standard deviation (across trials) ofthe voltage at the same time point. Absolute values wereused because the directions of voltage deflections canvary with the position of electrodes relative to recordedactivity, so effect magnitude is of primary interest. Usingsigned effect sizes produced larger correlation values, po-tentially reflecting inflated estimates produced by direc-tional consistency within electrodes (i.e., correlationswere exaggerated by consistency in the sign, rather thanthe magnitude, of effects across conditions).

The largest Cohen’s d value observed between 0 and400 msec post-sound onset was extracted for each condi-tion, yielding two values for each electrode. In all four

Figure 6. Electrodes showing significant ERPs during contralateral sounds (blue), ipsilateral sounds (orange), or both (purple). A majority ofelectrodes were significant in both analyses, suggesting similar neural mechanisms.

Plass et al. 1011

participants, we found strong positive correlations be-tween the effect sizes of contralateral and ipsilateralERPs, indicating predominantly bilateral responses acrosselectrodes (Figure 7B). For all four participants, esti-mated regression slopes were less than one, with 95%confidence intervals excluding one (Participant 8: slope =0.45, 95% CI [0.22, 0.68]; Participant 10: slope = 0.55,95% CI [0.36, 0.74]; Participant 13: slope = 0.45 95%CI [0.37, 0.52]; Participant 15: slope = 0.66, 95% CI[0.54, 0.79]), suggesting a significant contralateral biasacross electrodes. This result remained when interceptterms were fixed at zero. No significant correlations wereobserved when performing the same analysis on the

prestimulus baseline period (Participant 8: r(34) =0.09, p = .61; Participant 10: r(27) = −0.08, p = .68;Participant 13: r(105) = 0.002, p = .98; Participant 15:r(17) = 0.13, p = .59), indicating that correlations weredriven by the poststimulus response, and not generaldifferences in electrode sensitivity. Together, these re-sults strongly suggest that lateralized and nonlateralizedcomponents of the visual cortical response to soundsarise from common neural generators that display acontralateral bias in their responses.Because the timing of peak effect sizes varied consid-

erably across electrodes and participants, including bothearly and late time periods identified in previous

Figure 7. (A) Electrodes present in the four participants with the greatest number of visual electrodes, with distinct colors for each patient.(B) Correlations between ERP effect sizes (Cohen’s d ) for contralateral and ipsilateral sounds, evoked at visual selective electrodes. Data show strongcorrelations across all four participants, suggesting similar mechanisms drive visual activity in response to contralateral and ipsilateral sounds.(C) Correlations between contralateral and ipsilateral effect sizes, calculated at each millisecond following stimulus onset. Black boxes denotesignificant time points (corrected for multiple comparisons). Contralateral and ipsilateral effects were strongly correlated across both early andlate time periods.


research, we examined effect size correlations in a time-resolved manner to assess the time course of observedcorrelations throughout the entire analysis window(0–400 msec). For each participant, we computed corre-lations between contralateral and ipsilateral effect sizesfor each electrode at each millisecond of the 400 msecresponse window (Figure 7C). Statistical significancewas evaluated using FDR correction (q = .05) across allparticipants and time points. For all four participants, weobserved significant correlations during both early (ap-proximately 50–150 msec) and later (approximately150–400 msec) periods emphasized in previous research.Because three of four participants (Participants 10, 13,and 15) exhibited temporally distinct peaks in their cor-relation time courses that corresponded approximatelyto previously identified “early” and “late” response pe-riods, we examined the slopes of regression fits duringthese periods to test for contralaterally biased responses.At time points corresponding to early peaks in the cor-

relation time courses (Participant 8: 99 msec; Participant10: 117 msec; Participant 13: 130 msec; Participant 15:59 msec), each regression yielded a slope significantly(three of four) or nearly significantly (one of four) lessthan 1 (Participant 8: 0.66 [0.33, 1.01]; Participant 10:0.73 [0.58, 0.87]; Participant 13: 0.57 [0.51, 0.64];Participant 15: 0.66 [0.40, 0.93]), suggesting the presenceof contralaterally biased responses during this period. Attime points corresponding to later peaks (Participant 8:295 msec; Participant 10: 319 msec; Participant 13:315 msec; Participant 15: 371 msec), three of four regres-sion slopes were significantly less than 1 (Participant 8:0.60 [0.33, 0.87]; Participant 10: 0.77 [0.33, 1.22];Participant 13: 0.40 [0.24, 0.56]; Participant 15: 0.31[0.07, 0.56]), with all slope estimates being numericallyless than 1, again suggesting contralaterally biased re-sponses during this period. These results further re-inforce the conclusion that the effects observed in separateanalyses of laterality effects and bilaterally averagedresponses do not reflect distinct lateralized and nonlater-alized response but, rather, a single contralaterally biasedresponse.

DISCUSSION

Co-occurring sounds can facilitate the perception of spa-tially (Lu et al., 2009; Driver & Spence, 2004; McDonaldet al., 2000; Spence & Driver, 1997) or temporally (Chenet al., 2011; Fiebelkorn, Foxe, Butler, Mercier, et al., 2011;Jaekl & Soto-Faraco, 2010; Noesselt et al., 2010; Shamset al., 2002; Shipley, 1964) correspondent visual events.Separate lines of research have previously identifiedtwo putatively distinct neural mechanisms thought tounderlie crossmodal facilitations based on each type ofaudiovisual correspondence. On the one hand, auditory-driven phase resetting is thought to facilitate the percep-tion of simultaneous visual stimuli by placing visual cortexin a high-excitability state before visual signals arrive

(Thorne & Debener, 2014). On the other hand, lateralizedERPs elicited by spatialized sounds are thought to facili-tate the perception of spatially correspondent visualstimuli through hemisphere-specific excitation (or sup-pression) of visual cortex (Hillyard et al., 2016). Here,we sought to compare the topography and time courseof these effects to examine the relationship betweencrossmodal enhancements associated with spatial andtemporal correspondences.

Toward this end, we used densely sampled ECoGrecordings from visual cortex to compare the neural gen-erators of the auditory-driven phase reset response tothose of the lateralized and nonlateralized aspects ofthe auditory-driven ERP response in visual cortex. Wefound that the topography of the bilaterally averagedERP response closely matched the topography of thephase-resetting response produced by lateralized soundsin this study and centralized sounds in a previous study(Brang et al., under review), suggesting that both effectsreflect the same underlying process.

To test whether this response reflected a distinct non-lateralized response that is mechanistically independentfrom the lateralized ERP differences produced by soundsin visual cortex, we performed three further analyses.First, we compared the cortical topographies of elec-trodes exhibiting bilaterally averaged ERP responses tothose exhibiting laterality differences in response tocontralateral and ipsilateral sounds. Although lateralitydifferences showed a less widespread topography thanbilaterally averaged effects, areas displaying significantlaterality effects corresponded closely with those thatshowed the strongest bilaterally averaged response,suggesting a common neural generator.

Next, to determine whether response laterality wasattributable to a distinct neural generator that respondedexclusively to sounds in one hemifield, we tested whetherindividual electrodes displaying laterality effects respondedselectively to sounds in either hemifield or responded toboth sounds with a laterality bias. The majority of elec-trodes showed bilateral responses with a laterality bias,suggesting that laterality effects do not arise from a distinctneural generator but actually reflect laterality biases in thesame neural responses that produce the nonlateralizedcomponent of the ERP response. No distinct subregionthat responded exclusively to sounds in either hemifieldcould be identified.

Finally, to broadly characterize the lateralization of au-ditory responses in neural populations throughout visualcortex, we examined the relationship between the effectsizes of ERPs produced by contralateral and ipsilateralsounds across individual visual electrodes. In four pa-tients with widespread visual coverage, we found strongpositive correlations between the effect sizes of contra-lateral and ipsilateral ERPs, indicating predominantlybilateral responses across electrodes. The slopes ofassociated linear regressions revealed a contralateralbias across electrodes, suggesting that lateralized and

Plass et al. 1013

nonlateralized aspects of the visual cortical response tosounds arise from common neural generators that dis-play a contralateral bias.

Taken together, these results suggest that crossmodalphase resetting and lateralized ERP effects, typically stud-ied as distinct responses reflecting separate mechanismsfor crossmodal facilitation, may actually arise from thesame neural generators and, therefore, reflect the sameunderlying process. This result is consistent with a modelof crossmodal facilitation in which lateralized responsesto sounds in visual cortex are not produced by a distinctmechanism specialized for crossmodal conveyance ofspatial information, but a laterality bias inherited from au-ditory cortex and possibly modulated through attentivemechanisms.

On this account, the properties of visual cortical re-sponses to sounds do not arise from multiple specializedmechanisms responsible for the independent processingof various crossmodal correspondences, but rather fromthe passive carryover of the properties of the auditory cor-tical response to sounds—including response laterality—to visual cortex. As auditory cortex shows a contralateralbias in its response to sounds (e.g., right auditory cortexresponds more strongly, but not exclusively, to left later-alized sounds; Kaiser & Lutzenberger, 2001; Celesia,1976) and is thought to possess anatomical connectionsto visual cortex in the same hemisphere (right auditorycortex is connected to right visual cortex; Rockland &Ojima, 2003; Falchier, Clavagnier, Barone, & Kennedy,2002), auditory cortex’s laterality bias may be passivelycarried-over to connected visual areas in the same hemi-sphere, resulting in a laterality bias in visual corticalresponses to sounds.

One primary advantage of this explanation is that it cansimultaneously account for the seemingly contradictoryfindings of context- or task-contingent laterality effectsin visual cortical responses to sounds, which have beentaken to suggest a distinct mechanism for spatial interac-tions across the senses, and our current results, whichsuggest the absence of such a mechanism. Previous stud-ies suggest that laterality effects in occipital responses tosounds may depend on the task relevance (Campus et al.,2017) and unpredictability (Matusz et al., 2016) of thelocations of auditory stimuli. Because auditory spatialattention enhances response lateralization in auditory cor-tex (Alho et al., 1999; Teder-Sälejärvi, Hillyard, Röder, &Neville, 1999), the results of these attentional modula-tions may also be transmitted to visual cortex, producinglarger lateralization effects when auditory spatial infor-mation is task relevant (drawing endogenous auditoryspatial attention; e.g., Campus et al., 2017) or unpredict-able (drawing exogenous auditory spatial attention; e.g.,Matusz et al., 2016). Thus, on this account, previouslyobserved modulations of crossmodal laterality effectscould reflect enhanced lateralization in auditory cortexdue to auditory spatial attention, rather than a distinctmultisensory mechanism that specifically conveys

auditory spatial information to visual cortex. Thus, wepredict that manipulations of auditory spatial attentionor spatial unpredictability would enhance the spatialbiases observed in the current study but would notproduce purely laterality-selective responses.Although our results provide initial evidence consis-

tent with such an account, further research is neededto verify the additional premises underlying the pro-posed model, which presupposes that visual cortex’s re-sponse to sounds is inherited from auditory cortex.However, it is also possible that auditory information istransmitted to visual cortex by subcortical or thalamicpathways that would not display response laterality orattentional modulation (Cappe, Rouiller, & Barone,2012). Although our previous research using amplitude-modulated sounds suggests that visual cortex’s responseto sounds mirrors the temporal dynamics of auditorycortical, but not subcortical, responses (Brang et al., un-der review), additional evidence is needed to confidentlyconclude that crossmodal responses in visual cortex orig-inate from auditory cortex. Additionally, the proposedmodel rests on the assumption that auditory signals aretransmitted to visual cortex via lateralized intrahemi-spheric connections between auditory and visual corti-ces. Although such connections have been observed innonhuman primates (Rockland & Ojima, 2003; Falchieret al., 2002), it is currently unclear whether similar path-ways exist in humans. Indeed, we have successfully re-constructed similar pathways using diffusion-weightedimaging (Plass, Zweig, Brang, Suzuki, & Grabowecky,2015), though ambiguities regarding the accuracy ofestimated cortical terminations in diffusion tractographycurrently preclude us from making conclusive claimsregarding the presence or absence of this pathway inthe human brain (Reveley et al., 2015). Therefore,although our results provide initial evidence for the pro-posed model, additional evidence is necessary to fullycorroborate this account.One specific prediction made by this model is that the

lateralization of crossmodal responses to sounds in visualcortex should reflect the lateralization of auditory cor-tex’s responses to sounds, including variations poten-tially produced by spatial expectation, attention, taskdemands, adaptation, trial-by-trial variability, and theeccentricity-dependent resolution of auditory spatial atten-tion (e.g., Teder-Sälejärvi et al., 1999). Although the moststringent test of these predictions would be to compareactivity in auditory and visual cortices throughout a vari-ety of auditory tasks, electrode placements dictated byclinical needs rarely result in this type of extended cover-age and no participant in the current study had such cov-erage. We are currently exploring alternative methods totest these predictions rigorously. An additional predictionis that the spatial specificity of auditory effects on visualperception should reflect the resolution of spatial infor-mation provided by auditory cortical laterality. For exam-ple, because auditory cortical responses are only partially


lateralized, auditory facilitations of visual perception shouldnot be strictly hemifield specific and should occur even incases of fairly coarse audiovisual spatial alignment.Behavioral and psychophysical studies are largely inagreement with this prediction, with auditory enhance-ments of visual detection and discrimination often provingto be surprisingly impervious to spatial misalignment (e.g.,Spence, 2013; Fiebelkorn, Foxe, Butler, & Molholm, 2011).One potential caveat to our interpretation is that,

although we observed similar topographies across eachof our analyses, it is still possible that distinct but closelycolocated neural circuits or subpopulations separatelygenerate the lateralized and nonlateralized responses ob-served in each area. For example, signals recorded by asingle electrode may reflect the summed activity of neuralpopulations in different cortical layers or adjacent patchesof cortex. Still, given the high spatial precision providedby invasive intracranial electrodes, our results at leastsuggest that the generators of these responses are closelycolocated and are unlikely to occupy distinct regions ofvisual cortex as previous research appears to suggest.One additional concern is that the intracranially re-

corded responses analyzed here may not correspond pre-cisely to crossmodal effects identified previously usingscalp-recorded EEG (Matusz et al., 2016; Feng et al.,2014; McDonald et al., 2013; Naue et al., 2011) or TMS(Romei et al., 2012). Although additional research com-paring source-localized EEG, TMS, and ECoG responsesis needed to clarify the relationship between these effectsmore definitively, the response topography observed inthis study is largely in agreement with previous results,with dominant effects in lateral occipital cortex corre-sponding approximately to EEG source localizations oflaterality effects (Matusz et al., 2016; Feng et al., 2014;McDonald et al., 2013) and pericalcarine effects corre-sponding approximately to sites studied or stimulatedin animal physiology (Lakatos et al., 2009), TMS (Romeiet al., 2012), and human EEG studies of crossmodalphase resetting (Naue et al., 2011) and, arguably, someEEG studies of crossmodal laterality effects (Campuset al., 2017; Matusz et al., 2016; Feng et al., 2014).Thus, the primary discrepancies between our resultsand (some) previous results are that we observe rapid la-teralized responses in low-level visual cortex and laterality-biased responses at both “early” and “late” latencies.Because laterality effects were reliably smaller than theoverall bilaterally averaged response, these discrepanciesmay simply reflect differences in sensitivity betweenECoG and scalp-recorded EEG, potential weaknesses ofEEG source localization (Bradley et al., 2016), and the factthat previous studies of phase resetting have tended not toconsider response laterality. Additional activity in posteriorparietal or occipitoparietal cortex may correspond to previ-ously observed alpha activity associated with audiovisualspatial attention (Frey et al., 2014; Banerjee, Snyder,Molholm, & Foxe, 2011), but further research is neededto confirm this speculation.

Another potential concern is that patients’ inadvertenteye movements could have influenced observed corticalresponses. Although eye movements were not rigorouslymonitored using eye tracking in this study, eye move-ments were monitored by the experimenter throughoutthe tasks and minimized through experimenter feedback.Previous studies using similar task protocols did not ob-serve any eye deviations as measured by the EOG(McDonald et al., 2013), and our experimental observa-tions, though less rigorous, were largely consistent withthis result. Importantly, eye movements would not havebeen advantageous in either task because participants re-sponded only to centralized sounds in Task A and the au-ditory cue location was not predictive of visual targetlocation in Task B. Moreover, the early auditory-evokedresponses observed in this study are unlikely to havebeen contaminated by eye movement-related activity be-cause saccades to lateralized sounds typically take 150–300 msec to initiate (Gabriel, Munoz, & Boehnke, 2010;Yao & Peck, 1997; Frens & Van Opstal, 1995) and sac-cade-induced neural responses in low-level visual cortexand area MT tend to occur 40–100 msec after saccade on-set (e.g., Kagan, Gur, & Snodderly, 2008; Bair & O’keefe,1998). Thus, saccade-related activity is unlikely to havecontaminated the earlier responses observed in thisstudy. Although later responses could have conceivablybeen contaminated by eye movements, these responsesare largely consistent with previous source localizationsof late-lateralized ERPs in scalp-recorded EEG during rig-orous monitoring of eye movements and are thereforeunlikely to reflect eye movement-related activity.

Finally, additional research is necessary to better char-acterize the visual or other functions of the cortical sitesidentified in this study. Although each of our analyses re-peatedly pointed to a lateral occipitotemporal region, po-tentially corresponding to area V5/hMT+, as a critical sitefor audiovisual interactions, we were unable to verify thespecific functional properties of this area using, for exam-ple, motion-based localizers. Further research is neededto verify whether this site indeed corresponds to motion-sensitive area V5/hMT+ or an adjacent site with distinctfunctional properties. Additionally, given the nature ofour functional localizers, we cannot rule out the possibil-ity that some sites identified in this study would be bettercharacterized as multisensory association cortex than asvisual cortex. More detailed characterization of the func-tional properties of each site is necessary to clarify thisquestion; however, this distinction may rest more on se-mantic convention than on actual substance, given thewidespread prevalence of multisensory interactionsthroughout neocortex (Ghazanfar & Schroeder, 2006).Altogether, our results suggest that auditory phase reset-ting of visual activity (responsible for relaying temporalinformation to the visual system) and lateralized ERPeffects (responsible for relaying spatial information tothe visual system) can be accounted for by a single neuralmechanism.

Plass et al. 1015

Acknowledgments

This study was supported by National Institutes of Health (grantR00 DC013828).

Reprint requests should be sent to David Brang, Departmentof Psychology, University of Michigan, 530 Church Street,Ann Arbor, MI 48109, or via e-mail: [email protected].

REFERENCES

Alho, K., Medvedev, S. V., Pakhomov, S. V., Roudas, M. S.,Tervaniemi, M., Reinikainen, K., et al. (1999). Selective tuningof the left and right auditory cortices during spatially directedattention. Cognitive Brain Research, 7, 335–341.

Bair, W., & O’keefe, L. P. (1998). The influence of fixational eyemovements on the response of neurons in area MT of themacaque. Visual Neuroscience, 15, 779–786.

Banerjee, S., Snyder, A. C., Molholm, S., & Foxe, J. J. (2011).Oscillatory alpha-band mechanisms and the deployment ofspatial attention to anticipated auditory and visual targetlocations: Supramodal or sensory-specific controlmechanisms? Journal of Neuroscience, 31, 9923–9932.

Bradley, A., Yao, J., Dewald, J., & Richter, C. P. (2016).Evaluation of electroencephalography source localizationalgorithms with multiple cortical sources. PLoS One,11, e0147266.

Brainard, D. H. (1997). The psychophysics toolbox. SpatialVision, 10, 433–436.

Brang, D., Dai, Z., Zheng, W., & Towle, V. L. (2016). Registeringimaged ECoG electrodes to human cortex: A geometry-basedtechnique. Journal of Neuroscience Methods, 273, 64–73.

Brang, D., Plass, J., Sherman, A., Stacey, W. C., Wasade, V. S.,Grabowecky, M., et al. (Under review). Visual cortex mirrorsauditory cortex’s event-segmented representation of sounds.

Brang, D., Towle, V. L., Suzuki, S., Hillyard, S. A., Tusa, S. D.,Dai, Z., et al. (2015). Peripheral sounds rapidly activate visualcortex: Evidence from electrocorticography. Journal ofNeurophysiology, 114, 3023–3028.

Campus, C., Sandini, G., Morrone, M. C., & Gori, M. (2017).Spatial localization of sound elicits early responses fromoccipital visual cortex in humans. Scientific Reports, 7, 10415.

Cappe, C., Rouiller, E. M., & Barone, P. (2012). Cortical andthalamic pathways for multisensory and sensorimotorinterplay. In The neural bases of multisensory processes.Boca Raton, FL: CRC Press/Taylor & Francis.

Celesia, G. G. (1976). Organization of auditory cortical areasin man. Brain, 99, 403–414.

Chen, Y.-C., Huang, P.-C., Yeh, S.-L., & Spence, C. (2011).Synchronous sounds enhance visual sensitivity withoutreducing target uncertainty. Seeing and Perceiving, 24,623–638.

Dale, A. M., Fischl, B., & Sereno, M. I. (1999). Cortical surface-based analysis: I. Segmentation and surface reconstruction.Neuroimage, 9, 179–194.

Driver, J., & Spence, C. (2004). Crossmodal spatial attention:Evidence from human performance. In Crossmodal spaceand crossmodal attention (pp. 179–220). Oxford: OxfordUniversity Press.

Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002).Anatomical evidence of multimodal integration in primatestriate cortex. Journal of Neuroscience, 22, 5749–5759.

Feng, W., Störmer, V. S., Martinez, A., McDonald, J. J., &Hillyard, S. A. (2014). Sounds activate visual cortex andimprove visual discrimination. Journal of Neuroscience,34, 9817–9824.

Fiebelkorn, I. C., Foxe, J. J., Butler, J. S., Mercier, M. R., Snyder,A. C., & Molholm, S. (2011). Ready, set, reset: Stimulus-

locked periodicity in behavioral performance demonstratesthe consequences of cross-sensory phase reset. Journal ofNeuroscience, 31, 9971–9981.

Fiebelkorn, I. C., Foxe, J. J., Butler, J. S., & Molholm, S. (2011).Auditory facilitation of visual-target detection persistsregardless of retinal eccentricity and despite wide audiovisualmisalignments. Experimental Brain Research, 213, 167–174.

Fischl, B., Sereno, M. I., & Dale, A. M. (1999). Cortical surface-based analysis. II: Inflation, flattening, and a surface-basedcoordinate system. Neuroimage, 9, 195–207.

Frens, M. A., & Van Opstal, A. J. (1995). A quantitative study ofauditory-evoked saccadic eye movements in two dimensions.Experimental Brain Research, 107, 103–117.

Frey, J. N., Mainy, N., Lachaux, J.-P., Müller, N., Bertrand, O., &Weisz, N. (2014). Selective modulation of auditory corticalalpha activity in an audiovisual spatial attention task. Journalof Neuroscience, 34, 6634–6639.

Gabriel, D. N., Munoz, D. P., & Boehnke, S. E. (2010). Theeccentricity effect for auditory saccadic reaction times isindependent of target frequency. Hearing Research, 262,19–25.

Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortexessentially multisensory? Trends in Cognitive Sciences,10, 278–285.

Groppe, D. M., Urbach, T. P., & Kutas, M. (2011). Massunivariate analysis of event-related brain potentials/fields I:A critical tutorial review. Psychophysiology, 48, 1711–1725.

Hillyard, S. A., Störmer, V. S., Feng, W., Martinez, A., &McDonald, J. J. (2016). Cross-modal orienting of visualattention. Neuropsychologia, 83, 170–178.

Holmes, A. P., Blair, R. C., Watson, J. D., & Ford, I. (1996).Nonparametric analysis of statistic images from functionalmapping experiments. Journal of Cerebral Blood Flow &Metabolism, 16, 7–22.

Jacques, C., Witthoft, N., Weiner, K. S., Foster, B. L., Rangarajan,V., Hermes, D., et al. (2016). Corresponding ECoG and fMRIcategory-selective signals in human ventral temporal cortex.Neuropsychologia, 83, 14–28.

Jaekl, P. M., & Soto-Faraco, S. (2010). Audiovisual contrastenhancement is articulated primarily via the M-pathway.Brain Research, 1366, 85–92.

Jiang, H., Schuele, S., Rosenow, J., Zelano, C., Parvizi, J., Tao,J. X., et al. (2017). Theta oscillations rapidly conveyodor-specific content in human piriform cortex. Neuron,94, 207.e4–219.e4.

Kagan, I., Gur, M., & Snodderly, D. M. (2008). Saccades anddrifts differentially modulate neuronal activity in V1: Effects ofretinal image motion, position, and extraretinal influences.Journal of Vision, 8, 19.

Kaiser, J., & Lutzenberger, W. (2001). Location changesenhance hemispheric asymmetry of magnetic fields evokedby lateralized sounds in humans. Neuroscience Letters,314, 17–20.

Klimesch, W., Sauseng, P., Hanslmayr, S., Gruber, W., &Freunberger, R. (2007). Event-related phase reorganizationmay explain evoked neural dynamics. Neuroscience &Biobehavioral Reviews, 31, 1003–1016.

Lakatos, P., O’Connell, M. N., Barczak, A., Mills, A., Javitt, D. C.,& Schroeder, C. E. (2009). The leading sense: Supramodalcontrol of neurophysiological context by attention. Neuron,64, 419–430.

Lu, Z. L., Tse, H. C., Dosher, B. A., Lesmes, L. A., Posner, C., &Chu, W. (2009). Intra- and cross-modal cuing of spatialattention: Time courses and mechanisms. Vision Research,49, 1081–1096.

Matusz, P. J., Retsa, C., & Murray, M. M. (2016). The context-contingent nature of cross-modal activations of the visualcortex. Neuroimage, 125, 996–1004.


McDonald, J. J., Störmer, V. S., Martinez, A., Feng, W., & Hillyard,S. A. (2013). Salient sounds activate human visual cortexautomatically. Journal of Neuroscience, 33, 9194–9201.

McDonald, J. J., Teder-Sälejärvi, W. A., & Hillyard, S. A. (2000).Involuntary orienting to sound improves visual perception.Nature, 407, 906–908.

Mercier, M. R., Foxe, J. J., Fiebelkorn, I. C., Butler, J. S.,Schwartz, T. H., & Molholm, S. (2013). Auditory-driven phasereset in visual cortex: Human electrocorticography revealsmechanisms of early multisensory integration. Neuroimage,79, 19–29.

Naue, N., Rach, S., Strüber, D., Huster, R. J., Zaehle, T., Körner,U., et al. (2011). Auditory event-related response in visualcortex modulates subsequent visual responses in humans.Journal of Neuroscience, 31, 7729–7736.

Noesselt, T., Tyll, S., Boehler, C. N., Budinger, E., Heinze, H.-J.,& Driver, J. (2010). Sound-induced enhancement of low-intensity vision: Multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptualenhancement of visual detection sensitivity. Journal ofNeuroscience, 30, 13609–13623.

Pelli, D. G. (1997). The videotoolbox software for visualpsychophysics: Transforming numbers into movies. SpatialVision, 10, 437–442.

Plass, J., Zweig, L. J., Brang, D., Suzuki, S., & Grabowecky, M.(2015). Connections between primary auditory andprimary visual cortices: Diffusion MRI evidence. Posterpresented at the 2015 International Multisensory ResearchForum.

Reveley, C., Seth, A. K., Pierpaoli, C., Silva, A. C., Yu, D.,Saunders, R. C., et al. (2015). Superficial white matter fibersystems impede detection of long-range cortical connections

in diffusion MR tractography. Proceedings of the NationalAcademy of Sciences, U.S.A., 112, E2820–E2828.

Rockland, K. S., & Ojima, H. (2003). Multisensory convergencein calcarine visual areas in macaque monkey. InternationalJournal of Psychophysiology, 50, 19–26.

Romei, V., Gross, J., & Thut, G. (2012). Sounds reset rhythms ofvisual cortex and corresponding human visual perception.Current Biology, 22, 807–813.

Sauseng, P., Klimesch, W., Gruber, W. R., Hanslmayr, S.,Freunberger, R., & Doppelmayr, M. (2007). Are event-relatedpotential components generated by phase resetting of brainoscillations? A critical discussion. Neuroscience, 146,1435–1444.

Shams, L., Kamitani, Y., & Shimojo, S. (2002). Visual illusioninduced by sound. Cognitive Brain Research, 14, 147–152.

Shipley, T. (1964). Auditory flutter-driving of visual flicker.Science, 145, 1328–1330.

Spence, C. (2013). Just how important is spatial coincidence tomultisensory integration? Evaluating the spatial rule. Annalsof the New York Academy of Sciences, 1296, 31–49.

Spence, C., & Driver, J. (1997). Audiovisual links in exogenouscovert spatial orienting. Perception & Psychophysics, 59, 1–22.

Teder-Sälejärvi, W. A., Hillyard, S. A., Röder, B., & Neville, H. J.(1999). Spatial attention to central and peripheral auditorystimuli as indexed by event-related potentials. CognitiveBrain Research, 8, 213–227.

Thorne, J. D., & Debener, S. (2014). Look now and hear what’scoming: On the functional role of cross-modal phase reset.Hearing Research, 307, 144–152.

Yao, L., & Peck, C. K. (1997). Saccadic eye movements to visualand auditory targets. Experimental Brain Research, 115,25–34.

Plass et al. 1017

Joint Encoding of Auditory Timing and Location in Visual ...

Documents

Transcript of Joint Encoding of Auditory Timing and Location in Visual ...