Multimodal human-machine interface based

on a brain-computer Interface and an

electrooculography Interface

CHELSIA GEORGEECE ARoll No:13123028

2

CONTENTSOBJECTIVEINTRODUCTIONWHAT IS BRAIN-COMPUTER INTERFACE?WHAT IS ELECTROOCULOGRAPHY?COMBINED INTERFACEEXPERIMENTAL RESULTSADVANTAGES AND DISADVANTAGESAPPLICATIONS CONCLUSION AND FUTURESCOPE REFERENCES

3

OBJECTIVEThis paper describes a basic understanding of BCI and EOG and multimodal interface that combines both of these.

To illustrate an experiment to move a dot in a graphic user interface in support of the avove concept.

4

INTRODUCTIONØ Nowadays there are numerous assistive technologies for physically impaired users.

Ø In this sense there are great a number of interfaces that enhance the quality of life of disabled people, increasing their interdependence and granting greater social inclusion.

Ø. Brain-computer interface (BCI) technology is now being incorporated into their treatment, offering the promise of a greatly enhanced quality of life

5

ØSome patients still have voluntary control of the muscles controlling eye movement, the monitoring of eye movement can help these patients communicate with their environment and control devices.

ØThere is a huge opportunity for advancement in this field by combining EEG and EOG data.

6

What is Brain-Computer Interface?

ØBrain-computer interface (BCI) is a fast-growing emergent technology, in which researchers aim to build a direct channel between the human brain and the computer

ØA Brain Computer Interface (BCI) is a collaboration in which a brain accepts and controls a mechanical device as a natural part of its representation of the body.

ØIn this experiment the users wer asked to imagine small circular movements of their right snd left arm and then their bioelectrical activity will be measured.

7

ØMain principle behind this is the bio-electrical activity of the nerves and muscles

ØBrain is composed of millions of neurons, and when these neurons fires there is a voltage change across the cells generating signals across the surface of the brain.ØEEG(ElectroEncephalography)is a non -ivasive method to record this brain activity directly from the scalp.

ØNo EEG pattern is duplicated for any 2 distinct actions.

ØIt is recorded by placing small electrodes on the scalp in certain positions

ØBy monitoring and analysing these signals by externally placed electrodes when can understand the working of the brain.

8

9

The common structure of BCI is the following:

ØSignal Acquisition: The EEG are obtained from the brain through invasive or non invasive techniques.

ØSignal Preprocessing: Once the signals are acquired it is necessary to clean them.

ØSignal Classification: Once they are cleaned they will be processed and classified to find out which kind of mental task the subject is performing.

ØComputer Interaction :;Once they are classified they will be used for the development of certain application.

10

what is electrooculography?ØElectrooculography (EOG) is a technology of placing electrodes on user's forehead around the eyes to record eye movements .

ØThis technology is based on the principle of recording the corneal-retinal potential (CRP), which is the resting potential between the cornea and the retina commonl y known as electrooculogram.

Ø.(EOG) is a very small electrical potential that can be detected using electrodes and  islinearly proportional to eye displacement

ØBasically EOG is a bio-electrical skin potential measured around the eyes

11

.

ØIn the process of detection, the electrodes act as a transducer converting the ion current obtained at the skin to electrode current

Ø. For detection of horizontal motion there is generally one electrode in the outer edge of each. Eye

ØFor detection of vertical motion electrodes are kept above and below the eyes

12

13

ØTo detect the eyes movement an algorithm has been developed which is able to detect the gaze direction of the eyes (right, left, up and down) as well as the blink of the users.

Øthe user must perform fast movements of their eyes in the desired direction and then returning to their centre positions.

ØTo obtain the output, i.e. the direction of the gaze or the blink of the user, the output is analyzed. In every 0.5 seconds .First, a moving average filter is applied to get a clearer signal.

ØNext, the derivative is done to detect the abrupt changes of the signal when the user performs a fast movement of his/her eyes.

ØThe maximums and minimums of the waveform are searched as well as the time when they are produced ,and the values Iower than the noise threshold are removed and the rest of maximums and minimums values are added to detect the eye movement.

14

ØFinally, this is evaluated in order to obtain the gaze direction or the blink.

ØTo detect a blink it is checked if the value of the vector is higher than the blink threshold.

ØOtherwise, if a sequence of two values of the vector (maximum-minimum or mini mum-maximum) was produced with a difference lower than 0.6 seconds, a right/up or left/down movement has been pro duced.

15

Combined Interface Of brain-computer interface and electrooculography

16

ØThe BCI and the EOG interface have been combined in order to be able to perform more complex tasks.

Ø Once the EEG and EOG signals are processed, the mental task that the user is performing and the gaze direction are obtained each 0.5 seconds.

ØThe combination of both commands has been used to interact, in this case with a GUI.

ØThe commands of the EOG interface (up, down, right and left) have been used to move a dot in a plane while the commands of the BCI will control the height of the dot.

17

ØIn this first approach the control has been simplified using a non simultaneous control. When the user performs an eye movement in the desired direction, the dot starts moving in this direction.

ØWhen the user wants to stop it, he/she must perform an eye movement on the opposite direction.

ØWhile the dot is moving, it is not allowed to change the direction of movement without stopping the dot first. This has been done to force the user to return to a rest state.

18

ØWhen the dot is stopped, the BCI starts working, changing the height of the dot.

ØIf the user thinks of the mental task corresponding to right, the dot increases its height, and if the user thinks of the left mental task the dot decreases its height.

ØTo give time to the user to start thinking in the correct mental task, when the dot stops, the height of the dot does not change until 1.5 seconds later

19

EXPERIMENTAL RESULTSØThree male healthy volunteers with ages

between 25 and 35 participate on the experiments. and learned to control both interfaces independently.

ØFirst, the thresholds of the EOG interface were searched for each user and then the users could train performing several movements.

ØAfter that, the users started their training with the BCI.

ØThe users trained with several graphic interfaces that shows them a visual feedback of BCI output, so the users knows if they are doing well or not.

ØOnce both interfaces are controlled properly, the user starts working with the multi modal interface.

20

ØThe GUI shows a workspace where a dot can be moved in X, Y and Z. Two targets with a specific position have been placed on the GUI.

ØThe objective of the user is to perform several eyes movements using the EOG interface to get closer to them in the X and Y plane, and then, with the assistance of the BCI, to think of the corresponding mental task in order to change the height of the dot to match it with the target.

ØThe dot starts in the position X=0, Y=0 and Z=0. First, the user will go to the target 1 (placed in X=-200, Y=120 and Z=150) and next to the target 2 (placed in X=280, Y=-210 and Z=50).

ØTime used to arrive from the stating position to the first target, and to go from the first to the second target are measured.

ØThe dot is moved in a each time a command is received (each 0.5 seconds).

21

ØThe EOG algorithm was tested obtaining satisfactory results and the users obtained a success percentage around 94% (the rest corresponded to no detections).

ØThe BCI was tested in where the users obtained a initial good enough results (success rate of around a 60% with a 10% of uncertainty) that improved during the tests ØThe BCI requires thinking constantly in the corresponding mental task to increase or decrease the height of the dot, while with the EOG interface only a movement must be performed to start moving the dot or to stop it. The time used by the users to arrive to the targets is quite low.

22

ADVANTAGES AND DISADVANTAGES

23

ADVANTAGES

Ø allow paralyzed people to control prosthetic limbs with their mind

Ø Transmit visual images to the mind of a blind person, allowing them to see

ØTransmit auditory data to the mind of a def person, allowing them to hear

Øallow gamers to control video games with their minds

Øallow a mute person to have their thoughts displayed and spoken by a computer

24

ØResearch is still in beginning stages

ØThe current technology is crude

ØEthical issues may prevent its development

ØElectrodes outside of the skull can detect very few electric signals from the brain

DISADVANTAGES

25

APPLICATIONS

26

.

In a hospital you have to lie still, away from all sources of electrical interference, to have your brainwaves read.,Ttis chip captures your very weak brainwave as you go about normal life.”

NEUROSKY

27

The sensor, which is implanted into the brain, monitors brain activity in the patient and converts the intention of the user into computer commands.

BRAIN GATE

28

Researchers have developed an interface for Honda's Asimo robot that allows individuals to control it simply by thinking.

HONDA ASIMO

29

A visual prosthesis, often referred to as a bionic eye, is an experimental visual device intended to restore functional vision in those suffering from partial or total blindness

BIONIC EYE

30

Gaming control using a wearable and wireless EEG and EOG interface device with novel dry foam-based sensors.

And many more....!!!

GAME CONTROL

31

 

ØA multimodal interface composed by a BCI and an EOG interface has been designed. This interface allows increasing the number of commands that can be generated to interact with a device.

ØIt has been tested in a graphic user interface where several trajectories have been performed by three users to get closer to some targets.

ØThe results have shown that the users were able to use the multimodal interface to control the 3D movement of the dot.

ØUltimately, multimodal interfaces are just one part of the larger movement to establish richer communications interfaces, ones that can expand existing computational functionality and also improve support for human cognition and performance.

CONCLUSION AND FUTURESCOPE

32

ØIn conclusion, multimodal interfaces are just beginning to model human-like sensory perception and communication patterns.

ØThey are recognizing and identifying actions, language and people that have been seen, heard, or in other ways experienced in the past.

ØThey literally reflect and acknowledge the existence of human users, empower them in many new ways.

Ø.In all of these ways novel multimodal interfaces, as primitive as their early stages, represent a new multidisciplinary science, a new art form, and a socio-political statement about our collective desire to humanize the technology we create.

33

Øbiodegradable electrode, - dissolve away safely. - -longer lasting versions --better signal quality

Ødirect control over the activities of all individual nerrons n anobrobots

ØMemory Upload/Download and have memories that they themselves had not had.

Ø Dream Capture

Ø Brain as a Computer

Ø Google Search” through brain

34

.

E3 2009- Project Natal MiloEyePet – PlayStation Xbox 360- Kinect - E3 2010- All Up Montage -

35

REFERENCESØ. Chen and W.S. Newman, "A Human-robot Interface Based on El elecctrooculography" , IEEE International Conference on Robotics and Automation, vol. 1, pp. 243-248, 2004.

ØE. lanez, J. M. Azorin, E. Fernandez and A. ubeda "Interface Based on Electrooculograhy for Velocity Control of a Robot Arm". Applied Bionics and Biomechanics, vol. 7, no 3, pp. 199-207, 2010

ØG. Dornhege, J.R. Milian, T. Hinterberger, D. McFarland and K. Muller. "Towa-ds Brain-Computer Interfacing". MIT Press. Cambridge, Massachusetts, 2007.

ØE. IMez, M. C. Furi6, J. M. Azorin, J. A. Huizzi and E. Fernandez, "Bran-Robot Interface for Controlling a Remote Robot Arm", Lecture Notes on Computer Sciences. Ill International Vbrk-Conference on the Interplay between Natural and Artificial Corruption (IVNANAC), vol. 5602, pp. 353-361. 2009

36

ØR. Leeb, H. Sagha. R. Chaariaga and J.R. Milln, "Multi modal Fusion of Muscle and Bran Signals for a Hybrid-BCI", In Proceedings of the 32nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 4343-4346. 2010.

ØE.C. Lee, J C. OADob, J.H Kim, M. Whaig and K.R. Pak, "A Bran-Computer Interface Method Combined with Eye Tracking for 3D Interaction", Journal of Neuroscience Methods. vol. 190, pp 289-298, 2010

Ø E. lariez, J. M. Azorin. A Ubecla J. M. Fernandez and E Fernandez, " Mental Tasks-Based Bran—Robot Interface" , Robotics and Au tonomous Sisters, vol 58, no 12, pp. 1238-1245, 2010

37

Thank You!!!