Recent Advances in an SSVEP-based BCIdesa/SSVEP-Masaki.pdf · 2020. 2. 27. · Outline. UCSD, COGS...

Recent Advances in an SSVEP- based BCI

Masaki Nakanishi, PhDSwartz Center for Computational Neuroscience,

Institute for Neural Computation, University of California San Diego

COGS 189; February 28, 2020

UCSD, COGS 189, 02-28-2020

2

Masaki Nakanishi [email protected]

1. Introduction- Steady-state visual evoked potentials (SSVEPs)- An SSVEP-based BCI- Current status

2. Our contributions to the field- Effective visual stimulus presentation- Advanced signal processing

3. Clinical applications- Communication for ALS patients- Glaucomatous visual field assessment

4. Summary

Outline

UCSD, COGS 189, 02-28-2020

3





4. Summary

Outline

UCSD, COGS 189, 02-28-2020

4


§ The brain’s electrical responses to repetitive visual stimulation

§ Sinusoidal-like waveforms at stimulus frequency and its harmonics

Steady-state VEP (SSVEP)

Vialatte et al., Prog. Neurobiol., 90(4): 418-438, 2010

UCSD, COGS 189, 02-28-2020

5


An SSVEP-based BCI

Wang et al., IEEE Eng. Med. Biol. Mag, 27(5): 64-71, 2008

UCSD, COGS 189, 02-28-2020

6


Comparison of EEG features for BCIs

Nicolas-Alonso et al., Sensors, 12: 1211-1279, 2012

UCSD, COGS 189, 02-28-2020

7


Comparison of EEG features for BCIs

Nicolas-Alonso et al., Sensors, 12: 1211-1279, 2012

→ 300~ bits/min→ Even higher

The performance of an SSVEP-based BCI has been dramaticallyimproved in the past decade.

UCSD, COGS 189, 02-28-2020

8


SSVEP-based BCI speller ten years ago

UCSD, COGS 189, 02-28-2020

9


- 20 healthy adults- 40 commands- 800 ms / input- 89.83 ± 6.07 %- 325.33 ± 38.17 bits/min

Chen et al., Proc. Nat. Acad. Sci. USA, 2015; Nakanishi et al., IEEE Trans. Biomed. Eng., 2018

High-speed BCI speller today

UCSD, COGS 189, 02-28-2020

10


Information transfer rate (ITR) [bits/min]

Accuracy of target identification

The number of targets Average time for a selection [s]

Cheng et al., IEEE Biomed. Eng., 49(10): 1181-1186, 2002

UCSD, COGS 189, 02-28-2020

11


BCI Performance improvement

Nakanishi et al., 2014

Chen et al., 2015

Nakanishi et al., 2018

UCSD, COGS 189, 02-28-2020

12





4. Summary

Outline

UCSD, COGS 189, 02-28-2020

13


§ Designing effective stimulus presentation- Display-based stimulation method- Challenges in designing visual stimulation

§ Proposing advanced signal processing- Preprocessing – Spatial filtering- Target identification algorithm – Model-based method, Template-based method

Our achievements

UCSD, COGS 189, 02-28-2020

14


Display-based stimulus presentation§ Display-based stimulation is better than LED-based one because parameters (e.g.,

color, size, frequency) can be flexibly configured.§ Stimulus frequency can be produced by reversing the stimulus pattern between

white and black (e.g., ‘000111000111’).

UCSD, COGS 189, 02-28-2020

15


§ Stability of the stimulation MUST be tested before experiments to make sure if the stimulation is precise.

§ Our laboratory uses a phototransistor to measure luminance changes.

Stability test of stimulation

UCSD, COGS 189, 02-28-2020

16




Our achievements

UCSD, COGS 189, 02-28-2020

17


§ Frequency which can be presented on a computer monitor was limited byits refresh rate- Impossible to present the frequencies by which the refresh rate cannot be divided.- e.g., 11 Hz cannot be presented under 60 Hz refresh rate

=> We solved it by proposing a frequency approximation method (Nakanishi et al., 2014a)

§ Stimulus frequencies still need to be selected from a narrow range- SSVEPs have different amplitudes in three different frequency ranges (Wang et al., 2006)- Higher frequency resolution required longer SSVEP data epochs to reliably classify- e.g., 0.5 Hz resolution requires 2-s data; 0.2 Hz resolution requires 5-s data.

=> We solved it by proposed hybrid frequency and phase tagging techniques(Nakanishi et al., 2014b; Chen et al., 2015)

Challenges in designing visual stimulation

UCSD, COGS 189, 02-28-2020

18


§ Frequency which can be presented on a computer monitor was limited byits refresh rate- Impossible to present the frequencies by which the refresh rate cannot be divided.

- e.g., 11 Hz cannot be presented under 60 Hz refresh rate


§ Stimulus frequencies still need to be selected from a narrow range- SSVEPs have different amplitudes in three different frequency ranges (Wang et al., 2006)

- Higher frequency resolution required longer SSVEP data epochs to reliably classify

- e.g., 0.5 Hz resolution requires 2-s data; 0.2 Hz resolution requires 5-s data.



UCSD, COGS 189, 02-28-2020

19


Frequency approximation approach

Nakanishi et al., PLoS One, 9(6): e99235, 2014

f：Stimulus frequencyi：Frame index

UCSD, COGS 189, 02-28-2020

20










UCSD, COGS 189, 02-28-2020

21


Frequencies from differentfrequency ranges shouldnot co-exist in a system.

Amplitude responses in SSVEPs

Wang et al., IEEE Trans. Neural. Syst. Rehabil. Eng., 14(2): 234-239, 2006

UCSD, COGS 189, 02-28-2020

22










UCSD, COGS 189, 02-28-2020

23


Mixed frequency-phase modulation

Nakanishi et al., Int. J. Neural Syst., 24(6): 1450019, 2014

Φ：Initial phase

UCSD, COGS 189, 02-28-2020

24


§ Stimulus design for a BCI speller

- 26 English alphabets, 10 digits, 4 symbols

- Frequency range : 8 – 15.8 Hz with an interval of 0.2 Hz

- Phase range : 0 – 2 ! with an interval of 0.35 !

Stimulus design for our speller application

Freq. (Hz)

Phase( )

>> HIGH SPEED BCI 8.0

0.00

8.2

0.35

8.4

0.70

8.6

1.05

8.8

1.40

9.0

1.75

9.2

0.10

9.4

0.45

9.6

0.80

9.8

1.15

10.0

1.50

10.2

1.85

10.4

0.20

10.6

0.55

10.8

0.90

11.0

1.25

11.2

1.60

11.4

1.95

11.6

0.30

11.8

0.65

12.0

1.00

12.2

1.35

12.4

1.70

12.6

0.05

12.8

0.40

13.0

0.75

13.2

1.10

13.4

1.45

13.6

1.80

13.8

0.15

14.0

0.50

14.2

0.85

14.4

1.20

14.6

1.55

14.8

1.90

15.0

0.25

15.2

0.60

15.4

0.95

15.6

1.30

15.8

1.65

Freq. (Hz)

Phase( )

>> HIGH SPEED BCI 8.0

0.00

8.2

0.35

8.4

0.70

8.6

1.05

8.8

1.40

9.0

1.75

9.2

0.10

9.4

0.45

9.6

0.80

9.8

1.15

10.0

1.50

10.2

1.85

10.4

0.20

10.6

0.55

10.8

0.90

11.0

1.25

11.2

1.60

11.4

1.95

11.6

0.30

11.8

0.65

12.0

1.00

12.2

1.35

12.4

1.70

12.6

0.05

12.8

0.40

13.0

0.75

13.2

1.10

13.4

1.45

13.6

1.80

13.8

0.15

14.0

0.50

14.2

0.85

14.4

1.20

14.6

1.55

14.8

1.90

15.0

0.25

15.2

0.60

15.4

0.95

15.6

1.30

15.8

1.65

>> HIGH SPEED BCI>> HIGH SPEED BCI

AA B C D E F G H

I J K L M N O P

Q R S T U V W X

Y Z 0 1 2 3 4 5

6 7 8 9 , . <

Nakanishi et al., IEEE Trans. Biomed. Eng., 65(1): 104-112, 2018

UCSD, COGS 189, 02-28-2020

25



§ Proposing advanced signal processing- Preprocessing – Spatial filtering- Target identification algorithm – Model-based / Template-based method

Our achievements

UCSD, COGS 189, 02-28-2020

26


§ Scalp EEG recordings can be modeled as an instantaneous linear combination ofcortical source signals.

EEG mixture model and spatial filtering

! = #$ = #%& = & ((ℎ*+ #% = 1)§ Source signals can be estimated as:

Spatial filter

UCSD, COGS 189, 02-28-2020

27


Comparison of spatial filtering techniquesApproach Hypothesis

Average combination (AVG) The SSVEP manifests globally over the scalp without phase variations.

Principal component analysis (PCA) The SSVEP signal is uncorrelated from the background EEG.

Independent component analysis (ICA) The SSVEP signal is statistically independent from the background EEG.

Minimum energy combination (MEC) The optimal spatial filter results from minimizing an estimate of the noises.

Canonical correlation analysis (CCA) The optimal spatial filter can maximize correlation between SSVEPs and computer-generated SSVEP models.

Task-related component analysis (TRCA) The optimal spatial filter can maximize inter-trial correlation

Garcia-Molina et al., IEEE EMBS Conf. Neural Eng., 2011

UCSD, COGS 189, 02-28-2020

28


§ Task-related component analysis (TRCA) finds a linear coefficient which maximizes the reproducibility across trials

TRCA-based spatial filtering

UCSD, COGS 189, 02-28-2020

29


§ Assume there are two source signals: 1) task-related signal ! " ∈ ℝ; 2) task-unrelated signal %(") ∈ ℝ.

§ A linear generative model of observed multi-channel signal ((") ∈ ℝ)* is assumed as:

Problem setting for TRCA

+, " = ./,,! " + .2,,% " , 3 = 1,2, … , 78

§ The problem is to recover the task-related signal ! " from a linear sum of observed signals ((") as:

9 " =:,;/

)*<,+, " =:

,;/

)*<,./,,! " + <,.2,,%(")

§ Ideally, the problem has a solution of ∑,;/)* <,./,, = 1 and ∑,;/)* <,.2,, = 0, leading to

the final solution 9 " = !(")

UCSD, COGS 189, 02-28-2020

30


§ The problem can be solved by inter-trial covariance maximization§ The h-th trial of EEG signal and the estimated task-related component are

described as ! " and # " , ℎ = 1,2, … , *+.§ The covariance between ℎ,-th and ℎ--th trials of # is described as:

Problem solution using TRCA (1/2)

."/,"0 = Cov # "/ , # "0 = 45/,506,

7895/950Cov :5/

"/ , :50"0

§ All possible combination of trials are summed as:

4"/,"06,"/;"0

7<."/,"0 = 4

"/,"06,"/;"0

7<4

5/,506,

7895/950Cov :5/

"/ , :50"0 = =>?=

UCSD, COGS 189, 02-28-2020

31


§ To obtain a finite solution, the variance of ! " is constrained as:

Problem solution using TRCA (2/2)

Var !(") = )*+,*-./

012*+2*-Cov 6*+, 6*- = 7897 = 1

§ The constrained optimization problem can be solved using the method of Lagrange multiplier as:

; 7, < = 78=7 − < 7897− 1?@ 7,A?7 = =7 − < 97 = 0

§ The optimal coefficient vector is obtained as the eigenvector of the matrix 9C/=

UCSD, COGS 189, 02-28-2020

32




Our achievements

UCSD, COGS 189, 02-28-2020

33


Model-based methods§ Canonical correlation analysis (CCA)-based method- CCA takes two sets of multi-dimensional variables as an input

- CCA finds a pair of linear coefficients that maximize the correlation between two variablesprojected onto the coefficients.

- Target stimulus frequency can be identified by finding the model maximizing correlation.

Multi-channel EEG

Computer-generated SSVEP models

Lin et al., IEEE. Trans. Biomed. Eng., 54(6): 1172-1176, 2007

UCSD, COGS 189, 02-28-2020

34


§ Correlation between scalp EEG and individual templates after spatial filtering.§ Individual template can be obtained by averaging training data across trials

Template-based method

Nakanishi et al., Int. J. Neural Syst., 24(6): 1450019, 2014

UCSD, COGS 189, 02-28-2020

35


Performance comparison (Template-based)§ 40 visual stimuli were presented on a

23.6-inch LCD monitor.§ EEG data were recorded from 35

subjects with 9 electrodes placed over parietal and occipital areas.

§ The experiment consisted of 6 trials,in which the subjects gazed at one of the stimuli for 5 s.

UCSD, COGS 189, 02-28-2020

36





4. Summary

Outline

UCSD, COGS 189, 02-28-2020

37


UCSD, COGS 189, 02-28-2020

38


Glaucoma -緑内障 / 青光眼

Normal optic nerve

Glaucomatous optic nerve

Retinal ganglion cells

Weinreb, et al., JAMA, 2014

UCSD, COGS 189, 02-28-2020

39


§ Early detection- Glaucomatous visual field losses progress without

noticeable initial symptoms, resulting frequently in latediagnosis or late detection of progressive damage.

Challenges in glaucoma assessment

§ Lack of objectivity and portability- Conventional assessment methods have significant

drawbacks such as large test-retest variability,cumbersome clinic-based setting.

UCSD, COGS 189, 02-28-2020

40


§ Previous studies showed a good correspondence between the results of conventional visualfield assessment and the amplitude of SSVEPs

§ Current data recordings are time consuming and uncomfortable for patients due to skinpreparation and gel application

Glaucoma assessment using VEPs

Hood et al., Vis. Neurosci., 2000

UCSD, COGS 189, 02-28-2020

41


Glaucoma assessment using VEPs

Hood et al., Vis. Neurosci., 2000

§ Previous studies showed a good correspondence between the results of conventional visualfield assessment and the amplitude of SSVEPs

§ Current data recordings are time consuming and uncomfortable for patients due to skinpreparation and gel application

UCSD, COGS 189, 02-28-2020

42


Glaucoma diagnosis with a portable BCINeuromonitoring unit

- High-precision data acquisition unit- Bluetooth + Wi-Fi modules

Electrodes- 6 EEG + 2 EOG sensors- Sampling rate at 500 Hz

VR Display- Visual stimuli were programed by Unity

Pz

PO4PO3

OzO2

O1

Dry electrodes

Visual stimulus

UCSD, COGS 189, 02-28-2020

43


§ Visual stimuli eliciting multi-focal SSVEPs in 20 sectors over the 35-degree field of vision were presented on the nGoggle’s display

§ Stimulus frequencies: 8 - 11.8 Hz with an interval of 0.2 Hz

Stimulus design

UCSD, COGS 189, 02-28-2020

44


Demographical characteristicsGlaucoma

(n = 62 eyes of33 subjects)

Control (n = 30 eyes of

17 subjects)

P-Value

Age, years 68.2 ± 11.0 66.1 ± 9.9 0.57

Gender, female, n (%) 8 (47) 16 (48) 0.92

Race, n (%) 0.50

White 19 (58) 9 (53)

Black 12 (36) 8 (47)

Asian 2 (6) 0 (0)

SAP 24-2 MD, dB -4.0 (-12.7 to -1.8) -0.6 (-2.4 to 1.0) < 0.001

SAP 24-2 PSD, dB 4.7 (2.2 to 9.9) 1.9 (1.4 to 3.0) < 0.001

SSVEP CCA ! 0.289 ± 0.020 0.334 ± 0.024 < 0.001

* SAP: Standard automated perimetry; MD: Mean deviation; PSD: Pattern standard deviation;CCA: Canonical correlation analysis

Nakanishi et al., JAMA Ophthalmol. 2017

UCSD, COGS 189, 02-28-2020

45


Demographical characteristicsGlaucoma

(n = 62 eyes of33 subjects)

Control (n = 30 eyes of

17 subjects)

P-Value

Age, years 68.2 ± 11.0 66.1 ± 9.9 0.57

Gender, female, n (%) 8 (47) 16 (48) 0.92

Race, n (%) 0.50

White 19 (58) 9 (53)

Black 12 (36) 8 (47)

Asian 2 (6) 0 (0)

SAP 24-2 MD, dB -4.0 (-12.7 to -1.8) -0.6 (-2.4 to 1.0) < 0.001

SAP 24-2 PSD, dB 4.7 (2.2 to 9.9) 1.9 (1.4 to 3.0) < 0.001

SSVEP CCA ! 0.289 ± 0.020 0.334 ± 0.024 < 0.001

* SAP: Standard automated perimetry; MD: Mean deviation; PSD: Pattern standard deviation;CCA: Canonical correlation analysis


UCSD, COGS 189, 02-28-2020

46


Assessment of visual field deficits


UCSD, COGS 189, 02-28-2020

47


Diagnostic ability

AUC: 0.92 (95%CI: 0.88 - 0.96)

AUC: 0.81 (95%CI: 0.72 - 0.90)


UCSD, COGS 189, 02-28-2020

48





4. Summary

Outline

UCSD, COGS 189, 02-28-2020

49


§ The performance of an SSVEP-based BCI has been significantly improved in the past decade.

§ SCCN has contributed to the improvement.- Designing effective visual stimulation (frequency approximation, Mixed freq/phase tagging)- Proposing advanced signal processing (TRCA, Template-based target identification)- Clinical applications (Communication support for ALS patients, Glaucoma detection)

Summary

We are looking for volunteers that would like to participate in our experiments. Each participant will be paid at a rate of $15/hour. Please contact Nicole Wells ([email protected]) for more info.

mailto:[email protected]

UCSD, COGS 189, 02-28-2020

50


Thank you for your kind attention

Recent Advances in an SSVEP-based BCIdesa/SSVEP-Masaki.pdf · 2020. 2. 27. · Outline. UCSD, COGS...

Documents

Transcript of Recent Advances in an SSVEP-based BCIdesa/SSVEP-Masaki.pdf · 2020. 2. 27. · Outline. UCSD, COGS...