!"#"#$%&&'%()*"#%+,#*-"./%

0.12%!*34"$%!"#$%&%'()*+(,'&+-.(/*01&%-$*"(

2"34'+#3%5(*)(/-.3)*+"3-(6-"(73'8*(

,!9/(6'03"-+(

:-#;3"8%*"(2"34'+#3%5<(6%=(>*&3#(

6'1%'0?'+<(@AB@(

6=(C-D'38(E@AB@F(

G+-3"(1+*H'##'#((

;-4'('4*.4'I(-"I()&"H$*"((

!"#"$%&'()#!*)#!"#$!%&'

"+#!*)#(&)*+,!-(

%;'(?+-3"(*+8-"3J'#(

3"(+'#1*"#'(%*(

.&-$&,+&/'$)*00&12&3'

*1/'!..!-#"1,4&3=((

&-(1+"4+%5$4#.,%(

6)*"#/%-447%784%.8*994#$4%1:%784%-1-4#7;%

6(C-D'38(@ABA(

5+*0"*4!1(

6&-$&.4!1(7$4!1(

Functional Brain Imaging History

6=(C-D'38<(@AAK(0<%!*34"$%=>??(

01-4%8@-*#%()*"#%"-*$"#$%-"94/71#4/(

EEG '+-(((((BK@L(((MB#%((;&0-"(NNO(+'H*+I3"8%

((((BKPQ(((((B#%(E-"-.*8F(NNO(#1'H%+-.(-"-.5#3#(

ERP '+-(((((BKL@(((MB#%(H*01&%'+(NRS(-4'+-83"8(E/9TF(

((((BKU@((MB#%(0-8"'%*'"H'1;-.*8+-0(ECNOF(

fMRI '+-(((((BKKP((( (B#%()CR!(GV>7(+'H*+I3"8#%

((((BKKP(((((B#%(?+*-I?-"I(NR6S(

((((BKKW(((((B#%(0&.$#*&+H'(NNO(X.%'+3"8(?5(!/9(

fEEG / BCI / MoBI '+-(Y(((((@AAK(((MB#%(H*00'+H3-.(I+5Z'.'H%+*I'(NNO(%*5#(

((((@ABB(((MB#%(C*G!(>-?*+-%*+5(

((((@AB@((((MB#%(V".3"'(PZ7(0*?3.'(1;*"'(-11#(

4

16

32

1

256

1

80 14

10000

EEG (scalp surface fields)

ECOG (larger cortical surface fields)

Local Extracellular Fields

Intracellular and peri-cellular fields

Synaptic and other trans-membrane potentials

Brain dynamics are inherently multi-scale

At each spatial recording scale, the signal is produced by active partial coherence of distributed activities at the next smaller scale.

Scott Makeig 2007

Local field dynamics also influence spike

rate, timing, and synchrony.

Scott Makeig 2008

Macro field dynamics are spontaneously emerging

dynamic patterns in complex, nonlinear media.

(S;-#'(H*"'#(E[+''0-"F(

(94-.-"H;'#(EG'88#(\(S.'"JF(

6=(C-D'38(@AAU(

Alan Friedman

The spatiotemporal field dynamics of cortex have not yet been imaged

simultaneously on multiple spatial scales!

A1)B.1.1)B.*9%

A1)B.178*9*-".%

C*9@4%0,/74-%!1+@9*B1#/%

&D8*DB.%!1+@9*B1#/%

E84%$4#4)*B1#%*#+%-1+@9*B1#%1:%&&'%F%GHI%"/%AJ!IG&K%%

0<%!*34"$%=>>L(

M4@)1$9"*9N%

&O."7*71),%P#8"("71),%

/*+%'](

6D&..(

>*H-.(65"H;+*"5(

>*H-.(65"H;+*"5(

6D3"(

0<%!*34"$%=>>L(

&*.8%/.*9D%&&'%+*7*%.8*##49%/@-/%784%D)1Q4.74+%*.BR"B4/%1:%-@9BD94%()*"#%S*#+%#1#T()*"#U%/1@).4%D)1.4//4/<%

NNO(N^'H$4'(6*&+H'#(

@Z7(!"%'+1+'%-$*"(*)(6H-.1(NNO(638"-.#(((

N%

N% N%

N%

N%

N%

N%

The very broad EEG point-spread function

(9D-.3"(9H-+(\(C-D'38(@ABA(

Simulated small parietal source Very broad projected scalp potentials

E84%R4),%()1*+%&&'%D1"#7T/D)4*+%:@#.B1#%

(9D-.3"(9H-+(\(C-D'38(@ABA(

630&.-%'I(H0@Z#H-.'(0&.$Z#*&+H'(-H$43%5<((-"I(3%#(NNO(1+*_'H$*"(

This animation is available for YouTube viewing at http://sccn.ucsd.edu/

and for download at http://sccn.ucsd.edu/eeglab/TwoSourceEEG12.mp4

CSF

NNO (((/*HD%-3.(S-+%5(

G.3"I(NNO(6*&+H'(6'1-+-$*"(?5(

!"I'1'"I'"%(/*01*"'"%(9"-.5#3#(

S-1'+`(6=(C-D'38<('%(-.=<(,!S6(EBKKLF(

Tony Bell, developer of Infomax ICA

ICA can find distinct EEG source activities -- and their ‘simple’ scalp

maps!

/*+%'](

7*0-3"#((*)(>*H-.(65"H;+*"5(

P#+4D4#+4#7%

T;-.-0&#(

5)4%&&'%/1@).4%1@7D@7/%S#4*)U%"#+4D4#+4#7N%

[+''0-"(Z(1;-#'(H*"'#(

G'88#(\(S.'"J(Z(-4-.-"H;'#(

6=(C-D'38(E@AAUF(

Independent EEG Components are Dipolar 7'.*+0'<(S-.0'+<(V"%*"<(\(C-D'38<(@AB@(

J. Onton & S. Makeig 2006

PA5%V#+/%M1#T6)*"#%P#+4D4#+4#7%A1-D1#4#7%SPAU%I)1.4//4/%W%

W%/4D*)*74/%784-%:)1-%784%)4-*"#+4)%1:%784%+*7*%W%

a&.3'(V"%*"(\(6=(C-D'38(E@AALF(

PA5%*9/1%/4D*)*74/%.1)B.*9%()*"#%PA%D)1.4//4/%

7&-.Z#500'%+3H(I31*.'(H*01*"'"%(

63"8.'(I31*.'(H*01*"'"%(

Nb&34-.'"%(I31*.'#(

6(C-D'38<(@AB@(

EEGLAB

SIFT BCILAB

MoBILAB

MPT

0AAM%JD4#%01@).4%01XY*)4%E119/%:1)%!5EG56%

Tools available -- but a two-cultures problem …

Localizing independent component sources

Onton et al., 2005 Z. Akalin Acar, S. Makeig, 2011

IC source domain estimate

Scalp EEG source iEEG sulcal seizure source

FEASIBLE FOR SCALP EEG ICs?

Will need at least: •  Anatomic MR image •  Accurate electrode positions •  Accurate co-registration •  Good skull conductivity estimate

&&'%Z,#*-"./%1:%&-1B1#%P-*$"#*B1#%

6=(C-D'38<(@AAK(

0@$$4/7%784%"-*$"#*BR4%4OD4)"4#.4%1:%?[%4-1B1#/\%

! (-c'+(d'.'"(G*""5(EO!CF(

! (B#+(+'.-]-$*"(3"I&H$*"(

! (-.%'+"-%'(1*#(-"I("'8('0*$*"#(

! (+'.-](?'%e''"('0*$*"('13#*I'#(

! (BZW(03"(1'+3*I#(*)('5'#ZH.*#'I(#1*"%-"'*&#(NNO(](BW('0*$*"#(

! (PP(#&?_'H%#(

V"%*"(\(C-D'38<(,-".%)-/#'.#01&2.#3)1-"/4').4)#‘AK(

!"I'1'"I'"%(C*I&.-%*+#(

V"%*"(\(C-D'38<(,-".%)-/#'.#01&2.#3)1-"/4').4)#‘AK(

P#+4D4#+4#7%-1+4/%1:%/D4.7)*9%-1+@9*B1#%

V"%*"(\(C-D'38<(,-".%)-/#'.#01&2.#3)1-"/4').4)#‘AK(

a&.3'(V"%*"(\(6H*f(C-D'38<(,-".%)-/#'.#01&2.#3)1-"/4').4)<(@AAK(

/;-"8'#(3"(I3#%+3?&$*"(*)(()1*+(*#+%8"$8T:)4]@4#.,%NNO%1*e'+(e3%;(30-83"'I('0*$*"#(

208 IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 2, JUNE 2000

[9] J. Hollander Grill and P. H. Peckham, “Functional neuromuscular stim-ulation for combined control of elbow extension and hand grasp in C5and C6 quadriplegics,” IEEE Trans. Rehab. Eng., vol. 6, pp. 190–199,June 1998.

[10] A. Welford, Fundamentals of Skill. London, U.K.: Butler and Tanner,1971, pp. 11–26.

[11] P. Kennedy and R. Bakay, “Restoration of neural output from a par-alyzed patient by a direct brain connection,” NeuroReport, vol. 9, pp.1707–1711, 1998.

[12] J. Chapin and M. Nicolelis, “Neural network mechanisms of oscillatorybrain states: Characterization using simultaneous multi-single neuronrecordings,”Electroencephalog. Clini. Neurophysiol. Suppl., vol. 45, pp.113–122, 1996.

[13] D. Humphrey and J. Tanji, “What features of voluntary motor controlare encoded in the neuronal discharge of different cortical motor areas,”in Motor Control: Concepts and Issues, D. Humphrey and H. Fruend,Eds. New York: Wiley, 1991, pp. 413–444.

[14] A. Schwartz, “Motor cortical activity during drawingmovements: Popu-lation representation during sinusoid tracing,” J. Neurophysiol., vol. 70,pp. 28–36, 1992.

[15] J. Wolpaw, D. McFarland, G. Neat, and C. Forneris, “An EEG-basedbrain-computer interface for cursor control,” Electroencephalogr. Clin.lNeurophysiol., vol. 78, pp. 252–259, 1991.

[16] J. Wolpaw and D. McFarland, “Multichannel EEG-based brain-com-puter communication,” Electroencephalogr. Clin. Neurophysiol., vol.71, pp. 444–447, 1994.

[17] C. Guger, A. Schlogl, D. Walterspacher, and G. Pfurtscheller, “Designof an EEG-based brain-computer interface (BCI) from standard compo-nents running in real-time under windows,” Biomed. Tech., vol. 44, no.1–2, pp. 6–12, 1999.

[18] A. Kuebler, B. Kotchoubey, T. Hinterberger, N. Ghanayim, J. Perel-mouter,M. Schauer, C. Fritsch, E. Taub, andN. Birbaumer, “The thoughttranslation device: A neurophysiological approach to communication intotal motor paralysis,” Exper. Brain Res., vol. 124, no. 4, pp. 223–232,1999.

[19] R. Lauer, P. H. Peckham, and K. Kilgore, “EEG-based control of a handgrasp neuroprosthesis,” NeuroReport, vol. 10, pp. 1767–1771, 1999.

[20] D. McFarland, W. Sarnacki, and J. Wolpaw, “EEG-based brain-com-puter interface (BCI): Multiple selections with one dimensional con-trol,” Soc. Neurosci. Abstr., vol. 23, p. 656, 1998.

[21] P. Peckham, E. Marsolais, and J. Mortimer, “Restoration of key grip andrelease in the C6 quadriplegic through functional electrical stimulation,”J. Hand Surg., vol. 5, pp. 464–469, 1980.

[22] C. Tallon Baudry, O. Bertrand, C. Delpuech, and J. Pernier, “Oscillatorygamma-band (30–70 Hz) activity induced by a visual search task in hu-mans,” J. Neurosci., vol. 17, no. 2, pp. 722–734, 1997.

[23] E. Basar, C. Basar-Eroglu, S. Karakas, and M. Schumann, “Are cogni-tive processes manifested in event-related gamma, alpha, theta and deltaoscillations in EEG?,” Neurosci. Lett., vol. 259, pp. 165–168, 1999.

[24] T. Fernandez, T. Harmony, M. Rodriquez, J. Bernal, J. Silva, A. Reyes,and E. Marosi, “EEG activation patterns during the performance oftasks involving different components of mental calculation,” Electroen-cephalogr. Clin. Neurophysiol., vol. 94, pp. 175–182, 1995.

[25] B. Brouwer and D. Hopkins-Rosseel, “Motor cortical mapping of prox-imal upper extremitymuscles following spinal cord injury,” Spinal Cord,vol. 35, pp. 205–212, 1997.

[26] P. H. Peckham, J. T. Mortimer, and J. P. Van der Meulen, “Physiologicand metabolic changes in white muscle of cat following induced exer-cise,” Brain Res., vol. 50, pp. 424–429, 1973.

[27] J. Huggins, S. Levine, R. Kushwaha, S. BeMent, L. Schuh, and D. Ross,“Identification of cortical signal patterns related to human tongue pro-trusion,” in Proc. RESNA’95 Annu. Conf., vol. 15, 1995, pp. 670–672.

[28] A. Hoogerwerf and K.Wise, “A three-dimensional microelectrode arrayfor chronic neural recording,” IEEE Trans. Biomed. Eng., vol. 38, pp.75–81, 1994.

[29] R. Normann, E. Maynard, P. Rousche, and D. Warren, “A neural inter-face for a cortical vision prosthesis,”Vision Res., vol. 39, pp. 2577–2587,1999.

A Natural Basis for Efficient Brain-Actuated Control

Scott Makeig, Sigurd Enghoff, Tzyy-Ping Jung, andTerrence J. Sejnowski

Abstract—The prospect of noninvasive brain-actuated control ofcomputerized screen displays or locomotive devices is of interest to manyand of crucial importance to a few ‘locked-in’ subjects who experiencenear total motor paralysis while retaining sensory and mental faculties.Currently several groups are attempting to achieve brain-actuated controlof screen displays using operant conditioning of particular features ofthe spontaneous scalp electroencephalogram (EEG) including central-rhythms (9–12 Hz). A new EEG decomposition technique, independent

component analysis (ICA), appears to be a foundation for new research inthe design of systems for detection and operant control of endogenous EEGrhythms to achieve flexible EEG-based communication. ICA separatesmultichannel EEG data into spatially static and temporally independentcomponents including separate components accounting for posterioralpha rhythms and central activities. We demonstrate using data froma visual selective attention task that ICA-derived -components can showmuch stronger spectral reactivity to motor events than activity measuresfor single scalp channels. ICA decompositions of spontaneous EEG wouldthus appear to form a natural basis for operant conditioning to achieveefficient and multidimensional brain-actuated control in motor-limitedand locked-in subjects.

I. INTRODUCTION

Recent work in several laboratories has demonstrated that noninva-sively recorded electric brain activity can be used to voluntarily con-trol switches and communication channels, allowing a few so-calledlocked-in near-totally paralyzed subjects the ability to communicate,however slowly, with their families and aides ([4]; [14]; [2]). Com-munication rates achieved to date are in the range of several bits aminute, far from rates that would allow locked-in persons access tonormal social interaction. This communication briefly describes a tech-nique for blind decomposition of electroencephalogram (EEG) datainto temporally and often functionally independent components thatwould appear to provide a natural basis for optimizing brain-actuatedcontrol ([7]; [9]). An example is given of a decomposition of sponta-neous EEG in one subject into four components accounting for spatiallydistinguishable though widely overlapping posterior alpha and central-rhythmic activities. Learned control of the amplitude of motor-re-lated central -rhythms in the alpha frequency range (8–12 Hz) ([5]) isbeing used for brain-actuated control by at least two groups ([13]; [15]).We demonstrate that the motor-response related spectral perturbationsdemonstrated by the independent component analysis (ICA)-defined

Manuscript received August 2, 1999; revisedFebruary 28, 2000. The workof S. Makeig was supported by the Office of Naval Research, Department ofthe Navy (ONR.reimb.6429), the work of T. Sejnowski was supported by theHoward Hughes Medical Institute and the work of T.-P. Jung and T. Sejnowskiwas supported by the Swartz Foundation.S. Makeig is with the Naval Health Research Center, San Diego, CA 92186

USA. He is also with Computational Neurobiology Laboratory, Salk Institutefor Biological Studies, La Jolla, CA 92037 USA (e-mail: scott@salk.edu).S. Enghoff is with the Department of Physics, Technical University of Den-

mark, Copenhagen, Denmark. He is also with the Computational NeurobiologyLaboratory, Salk Institute for Biological Studies, La Jolla, CA 92037 USA.T.-P. Jung is with the Institute for Neural Computation, University of Cal-

ifornia, San Diego, La Jolla, CA 92093 USA. He is also with ComputationalNeurobiology Laboratory, Salk Institute for Biological Studies, La Jolla, CA92037 USA.T. J. Sejnowski is with the Howard Hughes Medical Institute and Computa-

tional Neurobiology Laboratory, Salk Institute for Biological Studies, La Jolla,CA 92037 USA. He is also with the Institute for Neural Computation, Univer-sity of California, San Diego, La Jolla, CA 92093 USA.Publisher Item Identifier S 1063-6528(00)04112-4.

1063–6528/00$10.00 © 2000 IEEE

ICA for BCI ?

C-D'38('%(-.=<(5666#7-2./#8)*29:#6.;'.:<(@AAA(

6APG56%

/(g*%;'(\(,(G38I'.5Z6;-0.*(@ABB(

#HH"=&H#I='I&hY"3"h6APG56(

MPT

5@+"1R"/@*9%524#B1#%08"X%&OD4)"-4#7%

7-%-`(a(T*e"#'"I('%(-.=<(@AAP(

^@4/B1#\%:;-%(3#(%;'(?+-3"(-H$43%5(#38"-%&+'(*)(#e3%H;3"8(?'%e''"(-&I3%*+5(-"I(43#&-.(-f'"$*"i(

Challenge: Given EEG epochs each time-locked to a HEAR or LOOK cue, train a model to estimate, from the EEG in each epoch, in which direction attention switched (HEAR"LOOK or LOOK"HEAR).

BA(

@W(

WA(

(P((W(

(ZB( A( B( @(#(

(9f'"I(("(9f'"I(9&I3*(((((((((j3#&-.( k(

Z(

A(G9(BQ<(BU<(6'H*"I-+5(Ej@F(-"I(S+30-+5(EjBF(43#&-.(

G9(BK<(PA<(BQ<(9##*H3-$4'(EjPF(\(6'H*"I-+5(Ej@F(43#&-.(

G9(BQ<(BK<(PA<(6'H*"I-+5(Ej@F(-"I(9##*H3-$4'(EjPF(43#&-.( G9(PK<BK<(PU<(9##*H3-$4'(43#&-.(EjPF(

G9(PB<BQ<(@P<(43#&-.( G9(K<Q<(3"H.&I'#()+*"%-.('5'(X'.I(

G9(P<(lP<(l<(lA<(S+30-+5(6*0-%*#'"#*+5(-"I(C*%*+(

5#%&&'%524#B1#T08"X%M47Y1)3%5.+"-&2%<)#,)2!1-)#=.2>?/'/#

/(g*%;'<(,(G38I'.5Z6;-0.*<(6(C-D'38(<(3"(1+'1-+-$*"(@ABB(

>-%'"H5(+'(6;3c((

[+'b

&'"H5(EdJF(

HEAR " LOOK

T Mullen, 2011

E)*#/"4#7%&_M%E847*%M47Y1)3%A1##4.BR"7,(

Tim Mullen, S. Makeig et al. unpublished

MICRO

MACRO RT

SPIKES

LFP

ECOG

EEG Recorded !?

~1,000,000 GHz

~1 Hz

BEHAVIOR BRAIN ?

~1 MHz

S. Makeig 2007

?

Average MoBI Mobile Brain/Body Imaging

Record what the brain does, What the brain experiences, And what the brain organizes.

@-2'.#'&2;'.;#A1-'.;#&"!"-#9)*2<'"-B(•  ,'-+.5(-..(?+-3"(30-83"8(#%&I3'#(ECNO<(SNT<()CR!<(-"I(

NNOF((-+'(H*"I&H%'I(3"(+383I.5(#%-$H(#'-%'I(*+(1+*"'(1*#3$*"#(e3%;(*".5(%;'(0*#%(03"30-.(X"8'+(0*4'0'"%#(-..*e'I=((

`8,N((•  !"(-..(0*I-.3$'#(?&%(NNO<(%;'(#'"#*+#(-+'(84*R,=((

•  C&#H.'(-"I(0*4'0'"%#(H*"%+3?&%'(E‘"*3#'’F(#38"-.#=(

•  6@7%78"/%9"-"7*B1#%"/%8"$89,%*)BV."*9<(,'-+.5(-..(*&+(.3)'(3"4*.4'#(24%<)#&"<)&).!/#-"I('.!)-24%"./#e3%;3"(-(PZ7('"43+*"0'"%=(

m  !%6)*"#%*.BR"7,%+@)"#$%:)44%-1R4-4#7%"#%aTZ%/D*.4%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%8*/%#4R4)%(44#%1(/4)R4+%1)%-1+494+n(

)CR!(

CNO(

SNT(

NNO(

6H*f(C-D'38(@AAQ(

!1("94%6)*"#F61+,%P-*$"#$%S!16PU%A1#.4D7%

B=(R'H*+I(#30&.%-"'*&#.5<(I&+3"8("-%&+-..5(0*$4-%'I(?';-43*+<(

(`8*7%784%()*"#%+14/%%%%% (E;38;ZI'"#3%5(NNOF(

( ((`8*7%784%()*"#%4OD4)"4#.4/%%%E#'"#*+5(#H'"'(+'H*+I3"8F(

( ( (`8*7%784%()*"#%1)$*#"b4/% ((E?*I5(\('5'(0*4'0'"%#<(

( ( ( ( ( ((((((((((((((( ( ( ( ( (((((1#5H;*1;5#3*.*85F(

@=(T;'"(o((

(c/4%4R19R"#$%-*.8"#4%94*)#"#$%-4781+/%%

%%%%71%V#+d%-1+49d%*#+%-4*/@)4%%

%%%%%%%#1#T/7*B1#*),%S.1#74O7T%*#+%"#74#B1#T)49*74+U%%

%%%%%%%%%%:@#.B1#*9%)49*B1#/8"D/%*-1#$%784/4%+*7*%-1+*9"B4/<%

Scott Makeig, 2011

!16P\%!1("94%6)*"#F61+,%P-*$"#$(

6=(C-D'38(('%(-.=<(5.!#C#D/?4*"$*?/'">";?##@AAK(

MoBI Lab at SCCN, UCSD

See http://thesciencenetwork.org/programs/inc-sccn-open-house/inc-sccn-open-house-hi-lite-reel

LSL

Lab Streaming Layer software for synchronous multi-stream, multi-platform recording and feedback – freely available on Google Code. Also, SNAP – Python-based scripting for complex experiment control.

C*G!(']1'+30'"%(3"(1+*8+'##`(6(C-D'38(\(C(C35-D*#;3<(@ABB@(

MoBI Lab: Two-Person Mirroring Experiment

Photo: T Bel Bahar & E Tumer, 2011

MoBI Lab: Two-Person Mirroring Experiment

Gedeon Deak et al., 2011

Development of Shared Attention – A Mother and Child MoBI Experiment

3-yr old child – Reward Observation

blank screen (baseline) mu

T Mullen, Yu Liao, ,S Makeig, G. Deak 2011

Mother Pops the Bubble!

S. Makeig, J Grethe, N Bigdely-Shamlo, 2012

10th Anniversary SCCN Impromptu celebration 1/2/12

Swartz Center for Computational Neuroscience (SCCN) UCSD, La Jolla CA

SCCN currently has over 50 researchers and students working on electrophysiological brain dynamics via high-density EEG, ECoG, MoBI, and other data – some 26 of us shown here ...