ANewGenerationofBrain-ComputerInterfacesDrivenbyDiscoveryofLatentEEG-fMRILinkagesUsingTensorDecomposition

HYPOTHESISANDTHEORYARTICLEFrontiers inNeuroscience, 07June 2017

XiaopingHuProfessor ofBiomedical EngineeringUniversity ofCalifornia, Riverside

Andrej CichockiSkolkovo InstituteofScience andTechnology(Skoltech), Moscow, RussiaNicolaus Copernicus University (UMK), Torun,PolandSystemsResearchInstitute,PolishAcademyofScience,Warsaw,Poland

LukeOedingDepartmentofMathematicsandStatisticsAuburn University

Rangaprakash DeshpandeDepartmentofPsychiatry andBiobehavioral SciencesUCLA

Gopikrishna DeshpandeAUMRIResearchCenterAUDepartmentofElectricalandComputer EngineeringAUDepartmentofPsychologyAlabamaAdvanced ImagingConsortium

BrainComputerInterface

https://en.wikipedia.org/wiki/File:Brain-computer_interface_(schematic).jpg

EEGBasedBCI

• +Non-invasive• +Hightemporalresolutionforreal-timeinteraction• +Inexpensive,lightweight,andhighlyportable• - Poorspatialspecificity• - EEGSignalsindifferentchannelsarehighlycorrelated,reducingabilitytodistinguishneurologicalprocesses.• - Longtrainingtimerequired

Real-timefMRIBasedBCI

• +Highspatialspecificity-- moreaccuracy• - Highcost• - Non-portable• - Lowtemporalresolution• - Restrictiveenvironment

SimultaneousEEG– fMRIdataacquisition

http://www.ant-neuro.com/sites/default/files/styles/product_imageshow/public/85500991.jpg?itok=0WPQPi1c

EEGData

Electrodeposition measurements viaPolhemus Fastrak 3DDigitizerSystemEEGdatawereepoched withrespecttoRpeaks ofEKG

signaland averagedovertrials.Theballistocardiogram (BCG)artifactintheEEGsignalobtained inside MRscannerisremoved.

Objective:DiscoveryofLatentLinkagesbetweenEEGandfMRIandimproveBCI• Hypothesis:latentlinkagesbetweenEEGandfMRIcanbeexploitedtoestimatefMRI-likefeaturesfromEEGdata.• ThiscouldallowanindependentlyoperatedEEG-BCItodecodebrainstatesinrealtime,withbetteraccuracyandlowertrainingtime.• Hypothesis:Featuresfromasub-setofsubjectscanbegeneralizedtonewsubjects(forahomogeneoussetofsubjects).

Strategies• ObtainfMRIdatawithhightemporalresolution:

• Usemultibandecho-planarimaging(M-EPI)[Feinberg,etal.2010]toachievewholebraincoveragewithsampling intervals(TR)asshortas200ms.• ViewfMRIasconvolutionofHDF(Hemodynamicresponsefunction)andneuronalstates.UsecubatureKalman filterbasedblinddeconvolutionoffMRI[Havlicek,etal.2011]torecoverdrivingneuronalstatevariableswithhighereffectivetemporalresolution.

• ObtaincleanEEGdata:• EEGsignalsampledat5000Hztoensureaccurategradientartifactremoval,thendownsampled to250Hztomakedatasetmoremanageable.

• UsethecomplexMorlet wavelet[Teolis,1998]togiveatime-frequencyrepresentationofbothEEGandfMRIforeachtrial.

DiscoverlatentlinkagesbetweenEEGandfMRI• SimultaneousEEG/fMRIdatacollectedusingaP300spellerbasedparadigm.• EEGmodalities:trial–time–frequency–channel

• 4msupdates,63+1channels,4trials

• fMRImodalities:trial–time–neuronalstate–voxel• 200msupdateswithwholebraincoverageand3mmvoxels

• ApplyOrthogonalDecompositiontoeachEEGandfMRI.[ZhouandCichocki 2012]• Thefirstdimensionof“trials”isthesameforbothtensors,permittingtheapplicationofHOPLS.ThisimportantpropertyallowsbothEEGandfMRItobesampledatdifferentrates.• ItisnotrequiredtodownsampleEEGtofMRI’stemporalresolution,asdonebymostresearchersintheEEG-fMRIcomparisonliterature(Goldman,etal.2002)(Hinterberger,Veit,etal.2005),whichwillleadtolossofvitaltemporalinformation.

• Assumptions:EEGdataistheindependentvariableX,anddeconvolvedfMRI(neuronalstates)dataisthedependentvariableY.• Reasonableassumptionbecausethehemodynamic/metabolic activityisasecondaryresponsetotheelectricalactivity.

• Goal:GivenX andY overmanytrials,andassumingF(X)=Y,discoverF.• HigherOrderMultilinearSubspaceRegression/HigherOrderPartialLeastSquares(HOPLS)[Q.Zhao,etal.2011]topredictthedependentvariable(deconvolvedfMRI)fromtheindependentvariable(EEG).• HOPLSparameters(latentvariables,coretensorsandtensorloadings)arelikelytoprovideinformationonlatentEEG-fMRIrelationships acrossthedimensionsconsidered.

DiscoverlatentlinkagesbetweenEEGandfMRI

PartialLeastSquares(PLS)

𝑋 = 𝑇𝑃% +𝐸 =( 𝑡*

+

*,-

𝑝*% + 𝐸

𝑌 = 𝑈𝑄% + 𝐹 = ( 𝑢*

+

*,-

𝑞*% + 𝐹

𝑇 = 𝑡-,𝑡7,⋯ , 𝑡+ and𝑈 = 𝑢-,𝑢7,⋯ , 𝑢+ arematricesofR extractedlatentvariables fromX andY,respectively.U will havemaximum covariancewithT column-wise.P andQ arelatentvectorsubspace base loadings.E andF areresiduals.

TherelationbetweenT andU canbeapproximatedas𝑈 ≈ 𝑇𝐷whereD isanR×R diagonalmatrixofregression coefficients.

Partialleastsquares: Predictsasetofdependent variablesY fromasetofindependent variablesX.Attemptstoexplain asmuch aspossible thecovariancebetweenXand Y.

PLSoptimizationobjective istomaximizepairwise covarianceofasetoflatentvariables byprojecting bothXandY ontonewsubspaces.

(3)

LeastSquares(UndergraduateLinearAlgebra)

• Givenalineartransformation𝑃 → 𝑄 wewanttosimultaneouslypredictthesubspacesℝR ⊂ 𝑃 andℝT ⊂ 𝑄 sothattherestrictedmapℝR → ℝTgivesagoodapproximationofthemapping.

• Givenanunderdeterminedmatrixequation𝐴�⃗� = 𝑏,wecanattempttosquarethesystemandsolve:𝐴\𝐴�⃗� = 𝐴\𝑏• PerhapsuseQR.

• Thestandardleast-squaressolutionis𝑥] = 𝐴\𝐴 ^-𝐴\𝑏.• ProjecttoalinearsubspaceandreplaceAwithafullrankmatrixusingSVD.

SingularValueDecomposition• TheSingularValueDecomposition𝐴 = 𝑈Σ𝑉\

• Thequasi-diagonalmatrixofsingularvalues𝜎-, 𝜎7,… canbetruncatedtothelargestr singularvaluestogivethebestrankr approximation.

• TheorthogonalmatricesU andV(calledloadings)givetheembeddings oftherespectivesubspacesonwhichA isbestapproximatedtorankr.

• Thepseudoinverse ofA is𝐴† = 𝑉Σ†𝑈\,whereΣ† = diag(𝜎-^-, 𝜎7^-, …)

• Theminimalnormsolutionto𝐴�⃗� = 𝑏 is𝑥] = 𝐴†𝑏 = Σd,-* ef

ghif𝑣d.

SVDandPLS• Givenm dataobservationsofnparticipantsstoredinadatamatrixX(independentvariables).• Givenk responsesofthen participantsstoredinadatamatrixY(dependentvariables).• FindalinearfunctionF thatexplainsthemaximumcovariancebetweenXandY.

𝑌 = 𝑋𝐹+ 𝐸• CenterandnormalizebothX andY.• ComputetheCovarianceMatrix𝑅 = 𝑌\𝑋• PerformSVD:R = 𝑈Σ𝑉\ (compactform,iterativealgorithm)• ThelatentvariablesofX andY areobtainedbyprojections:

𝐿n = 𝑋𝑉 𝐿o = 𝑌𝑈• U andVgivetheembeddingsofthesubspaces(theloadingsofthevariables)

Structuredvariables• InthesituationofEEG-fMRIdata,evenafterawaveletdecompositionofthedata,westillhaveextrastructureinthedependentvariables(fMRI)YandintheindependentvariablesX.• EEGmodalities:trial–time–frequency–channel

• 4msupdates,63+1channels,4trials• SoXcouldbe4trialsx100waveletsx63channels

• fMRImodalities:trial–time–neuronalstate–voxel• 200msupdateswithwholebraincoverageand3mmvoxels• SoYcouldbe4trialsx200waveletsx36,000voxels

• Don’tthinkof100x63as6,300,don’tthinkof200x36,000as7.2×10u• Unfoldingleadsto“smallp largen”problemandalossofinformation.

ModalProductsforTensors• For𝒜 ∈ ℝxy×xz×⋯×x{ and𝑈 ∈ ℝ|}×x} the𝑛��modetensor-matrixproductis

𝒜×T𝑈 ∈ ℝxy×xz×⋯×x}�y×|}×x}�y×⋯×x{

𝒜×T𝑈 ∶= ( 𝑎dy,dz,…,d},…d{𝑢�},d}d}∈x}

• Thismodalproductgeneralizesthematrixproductandvectorouterproductandreplacestranspose.

If𝐴 ∈ ℝxy×xz and𝐵 ∈ ℝxy×|z then𝐴×-𝐵 = 𝐵\𝐴 ∈ ℝ|z×xz

Ifx ∈ ℝ-×T andy ∈ ℝ-×R then �⃗�\𝑥 ∈ ℝR×T

MatrixSVDusingmodalproductnotation

• SVDTheorem:Everycomplex𝐼-×𝐼7 matrix𝐹 hasanexpression𝐹 = 𝑆×-𝑈(-)×7𝑈(7)

with𝑈(-) aunitary𝐼-×𝐼-matrix𝑈(7) aunitary𝐼7×𝐼7 matrix𝑆 pseudodiagonal𝐼-×𝐼7 matrix,𝑆 = diag(𝜎- , 𝜎7,… , 𝜎��� xy,xz )Thesingularvaluesareordered:𝜎- ≥ 𝜎7 ≥ ⋯ ,≥ 𝜎��� xy,xz ≥ 0

TensorSVD(OrthogonalTuckerDecomposition)

• Every𝐼-×𝐼7×⋯×𝐼� array𝒜 canbewrittenasaproduct:𝒜 = 𝑆×-𝑈(-)×7𝑈(7)⋯×�𝑈(�)

• Each𝑈(T)isaunitary𝐼T×𝐼T matrix.

• 𝑆isa𝐼-×𝐼7×⋯×𝐼� complextensorwithsliceshavingnorm 𝑆d},d = 𝜎d(T),

then-modesingularvaluesof𝒜• Foreachnthesingularvaluesareordered𝜎-

(T) ≥ 𝜎7T ≥ ⋯ ≥ 𝜎x}

T ≥ 0• Theslices𝑆d},d areall-orthogonal:

𝑆d},�,𝑆d},� = 0∀𝛼 ≠ 𝛽∀𝑛

• Computethen-modesingularmatrix𝑈(T) andn-modesingularvaluesbythematrixSVDofthen-th unfoldingofsize𝐼T×𝐼7𝐼�⋯𝐼T^-𝐼T�-𝐼�.• Siscomputedby𝑆 = 𝒜×-𝑈 - ∗

×7𝑈 7 ∗⋯×�𝑈 � ∗

Theorem:[DeLauthawer 2005, Zhao-Cichocki 2013]

TuckerDecompositionforEEG--fMRI• Takean𝑁-×𝑁7×𝑁�×𝑁�tensorandexpressitasa(small)coretensorofsize𝐿-×𝐿7×𝐿�×𝐿�togetherwithchangesofbases(loadings)toputthecorebackintothelargertensorspace.• Let𝑋 and𝑌 betensorsofEEGanddeconvolvedfMRI,respectivelywithmodalities:trials,voxels/channels,timeandfrequency.• ObtainnewtensorsubspacesviatheTuckermodelforeachtrial:

• Approximate𝑋 withasumofmultilinear rank-(1, 𝐿7,𝐿�,𝐿�)terms• Approximate𝑌 withasumofmultilinear rank-(1,𝐾7, 𝐾�,𝐾�) terms

• ThecoretensorsmodeltheunderlyingbiophysicsandaredifferentforEEGandfMRI.• PerformHOSVDonthe𝐿7×𝐿�×𝐿�×𝐾7×𝐾�×𝐾� contraction𝑋×- 𝑌

HigherorderPartialLeastSquares(HOPLS)TheHOPLSisexpressed as

𝑌 = (𝐷*

+

*,-

×-𝑡*×7𝑄*(7)×�𝑄*

(�)×�𝑄*� +𝐹

𝑋 =( 𝐺*

+

*,-

×-𝑡*×7𝑃*(7)×�𝑃*

(�)×�𝑃*� +𝐸

whereR isthenumber oflatentvectors,𝑡* istherth latentvector,𝑃*(T) and𝑄*

(R) areloadingmatricescorrespondingtolatentvector𝑡* onmode-n andmode-m,respectively,𝐺* and𝐷* arecoretensors,and×� istheproduct inthekth mode.

Compute the𝑡* astheleadingleftsingular vectorofanunfolding, deflate,andrepeat.

FeasibilityStudy

• WeperformedthesimultaneousEEG/fMRIexperimentandEEG-onlyBCIusingtheP300spellerparadigmin4right-handedmalesubjects(meanage:21.5years)withnohistoryofneurologicalorotherillness.• fMRIdatawereacquiredusingtheM-EPIsequence(TR=200ms,multibandfactor=8,3mmisotropicvoxels,fullcoverage)anddeconvolvedusingthecubatureKalman filterapproach.• Theanalyseswerecarriedoutona highperformancecomputerwithIntel®Core™i7-3820(QuadCore,10MBCache)overclockedupto4.1GHzprocessorwithatopofthelineNVidiaGPUQuadro Plex 7000.• WeobtainedsignificantlyhighcorrelationusingboththefullandthesignificantHOPLSmodels,withthelatterprovidingbetteraccuracywithruntimesunderasecond.

PreliminaryResultsOffline analysis of

Simultaneous EEG/fMRI ExptFull HOPLS forward model Significant HOPLS forward model

Correlation between deconvolved fMRI data and

that predicted from EEG0.76 ± 0.17 0.84 ± 0.13

Approximate run time for ‘prediction module’ in sec

1.4 0.8

Table.2 Prediction of deconvolved fMRI from simultaneously acquired EEG using offline analysis

Off line analysis of simultaneous

EEG/fMRI Expt

Original fMRI MVPA

fMRI predicted with significant

HOPLS + MVPA

fMRI predicted with full HOPLS

+ MVPA

SVM based on EEG tensors

(from sequential model)

SVM based on fMRI

tensors(from sequential model)

SVM based on ERP amplitude

and latency

Letter decoding accuracy

1 trial block

0.97±0.03 0.94 ± 0.04 0.93±0.05 0.84±0.10 0.86±0.12 0.68 ± 0.17

8 trial blocks

1 1 1 0.98±0.02 0.98±0.02 0.84 ± 0.11

Run time per letter decoded (sec)

0.9 1.8 2.4 0.13 0.24 0.08

Table.3 Letter decoding accuracy from post-hoc analysis of simultaneous EEG/fMRI data

Preliminaryresults

Online analysis of EEG-only BCI dataParameters from same subject’s EEG/fMRI run

Parameters from random prior

subject’s EEG/fMRI run

Parameters learned from all prior subjects’ EEG/fMRI run

Subject-2 Subject-3 Subject-4

Letter decoding accuracy from fMRI predicted with significant

HOPLS + MVPA

1 trial block 0.93 ± 0.04 0.87 ± 0.11 0.86 0.91 0.93

8 trial blocks 1 0.94 ± 0.04 0.93 0.93 0.95

Table.4 Letter decoding accuracy from real-time analysis of EEG data using predicted fMRI (from significant HOPLS) as features for MVPA

In spite of these encouraging results, we stress the fact that they are derived from a small, homogeneous sample of 4 subjects. We need to do more trials to demonstrate more broad generalizability.

(ExtraSlide) HigherOrderPartialLeastSquares• Thesubspacetransformationisoptimizedusingthefollowingobjectivefunction,yieldingthecommonlatentvariable𝑡* insteadof2latentvariables.

• 𝑚𝑖𝑛{¡ } ,¢ } } 𝑋 − [𝐺; 𝑡,𝑃7 ,𝑃 � ,𝑃 � ]

7+ 𝑌 − [𝐷;𝑡,𝑄 7 ,𝑄 � , 𝑄 � ]

7

suchthat 𝑃 T %𝑃 T = 𝐼¨}𝑎𝑛𝑑 𝑄R %𝑄 R = 𝐼ª«

• Simultaneousoptimizationofsubspacerepresentationsandlatentvariable𝑡*.SolutionscanbeobtainedbyMultilinearSingularValueDecomposition(MSVD)(see[Q.Zhao,etal.2011])• Minimizingtheerrorsisequivalenttomaximizingthenorms 𝐺 and 𝐷simultaneously(accountingforthecommonlatentvariable).Todothiswemaximizetheproduct 𝐺 7 ⋅ 𝐷 7.• Computethelatentvariables𝑡* asleadingleftsingularvectorsandthendeflate.Repeatuntilyoureachtheerrorboundsyouwant.