An Investigation i nto Application of Eye Tracking for Displacement of a

Wheelchair for Quadriplegic Disabled

Aldo Villanueva Marcocchio

Department of Automatic Control & Systems Engineering

Sheffield, UK

09/2013

Supervisor: Dr. Ian Lilley

A dissertation submitted in partial fulfilment of the requirements for

the degree of Master of Science in Control

Systems Engineering

ii

EXECUTIVE SUMMARY

INTRODUCTION/BACKGROUND

This project explores the potential of using eye-tracking as an alternative control

method for an assistive vehicle such an electric wheelchair. The extraction of pupil

centre and its position is analysed through the aid of a low-cost digital camera which

constantly captures images of the eye. During image processing, different variables

are analysed such as features, eye properties and coordinates extraction to determine

the eye centre position on real time.

AIMS AND OBJECTIVES

The aim of this project is to investigate, design and implement software for control of

an assistive mobility vehicle using LABVIEW; developing a protocol for eye-based

command and communication with a simulated vehicle. The project objectives are:

conduct background research into assistive mobility vehicles and define the project

scope, familiarization with LABVIEW Vision and Robotics Simulator, design and

develop a simulation environment, implement camera hardware and software for

image capture, design and develop software for eye-position data extraction and eye-

tracking, design and develop a protocol for eye-based command using the simulation

test environment, integrate the component parts to achieve eye-based control of a

simulation environment, test and optimize the application.

ACHIEVEMENTS

During the project, the achievements were the following:

• Eye tracking based on Pupil detection and data extraction on multiple users in

real time from image acquisition.

• Design and implementation in LABVIEW from an own CAD designed

environment in an external program.

• Integration of Eye Tracking and simulation environment, allowing Eye-based

control of a simulated vehicle in a low-cost computer.

• Proposal, evaluation and implementation of novel strategies to reduce Eye

Tracking computational cost.

• Final design includes a proposed algorithm to reduce calibration time, the

implemented algorithm also adjusts the system automatically to ideal

parameters respect to the user and lighting in indoor environments.

iii

CONCLUSIONS / RECOMMENDATIONS

The results of this project showed that this application is viable for future

implementation in an Electric-Powered Wheelchair, the image processing algorithms

used showed that it is possible to achieve eye-tracking using a low-cost computer

running at 1.8 Ghz and positioning the camera from the user’s eye at a distance

between 5 to 10 centimetres where the maximum accuracy was found. The algorithms

were proposed considering the pupil shape. In some people the pupil shape is round

and in others is slightly elliptical, from that concept a set of algorithms during image

processing were used to track the pupil centre position. Based on results, it can be

said that it is possible to use a single piece of software for several tasks, including eye-

tracking, a simulation environment (it was achieved to import the design from a 3D

modelling program) and control outputs. The modification in the camera to remove the

IR filter was successful, therefore when using IR light (not visible for the human)

significantly improved the contrast in the images, allowing a significant improvement

in image processing in order to extract the pupil location. This system could be

improved by adding an external frontal camera for obstacle avoidance, providing a

more reliable control alternative for people with limited mobility.

iv

ABSTRACT

Eye tracking is the process of electronically locating the point of gaze from the user.

Following and recording the movements from the eye. The potential applications for

eye tracking technology are unlimited, some of the most popular include consumer

behaviour research, website usability testing, driving distraction, human computer

interface devices and assistive technologies.

The dissertation project focuses on assistive technologies, where eye tracking

technology can be applied to assistive mobility devices such an Electric

Wheelchair(EW) for quadriplegic disabled and to people with severe limitations to

make controlled movements, where due to those limitations they cannot make use of

a joystick to control the EW.

The dissertation project involves design and development of software for eye-position

data extraction and a designed simulation environment where the simulated vehicle is

controlled in response to eye-based commands. An ellipse-circular edge combination

algorithm is proposed to ensure pupil centre extraction during calibration and

execution, additionally a novel strategy is suggested to reduce calibration time.

v

ACKNOWLEDGEMENTS

I would like to thank my parents and siblings who have always been in my mind and

my heart, without whom this achievement would not have been possible. To my father,

for his wise guidance and motivation towards me to improve my education and every

aspect of my life, showing me that there are no limits in our lives, limits are only in our

minds. To my mother, for her constant support, affection and attention towards me,

always seeking my welfare. To my friends, who have always been ready to listen and

support. This space is not enough to show how grateful I am to life for being close to

such wonderful human beings.

I would like to express my deepest gratitude to Dr. Ian Lilley for his excellent guidance,

patience and for creating a pleasant atmosphere for carrying out my dissertation

project.

Finally, I would like to thank the National Council on Science and Technology

(CONACYT) for his financial support during this MSc Programme.

Agradecimientos

Quiero agradecer a mis padres y hermanos que siempre han estado en mi mente y

mi corazón, sin ellos este logro no hubiera sido posible. A mi padre, por siempre

guiarme y motivarme a ser una persona más preparada en la vida, demostrándome

que los límites de una persona llegan hasta donde uno mismo los establece. A mi

madre, por su constante apoyo, cariño y atención hacia mí, siempre procurando mi

bienestar. A mis amigos, quienes siempre han estado listos para escuchar y

apoyarme. Este espacio no es suficiente para demostrar lo agradecido que estoy con

la vida por brindarme tan maravillosos seres humanos.

Quiero expresar mi profunda gratitud hacia el Dr. Ian Lilley por ser un excelente guía,

por su paciencia y por crear un ambiente placentero para llevar a cabo mi proyecto

de disertación.

Finalmente, quiero agradecer al Consejo Nacional de Ciencia y Tecnología

(CONACYT) por su apoyo de financiamiento durante este curso de Posgrado.

vi

TABLE OF CONTENTS

Executive Summary...............................................................................................ii

Abstract.................................................................................................................iv

Acknowledgements................................................................................................v

Table of Contents……………………………………………………………………….vi

Chapter 1 - Introduction .......................................................................................1

1.1 Background/Motivation……………………………………………………..1

1.2 Aim and Objectives................................................................................2

1.3 Project Management..............................................................................3

Chapter 2 - Literature Review ..............................................................................8

2.1 Eye-Tracking ........................................................................................8

2.1.1 Definition of Eye-Tracking........................................................8

2.1.2 Eye-Tracking methods…….……..............................................8

2.1.3 Popular applications……….…………………………………….10

2.2 Wheelchair mobility ...........................................................................14

2.2.1 Introduction……………………………………………………….14

2.2.2 Electric Wheelchair Control Interfaces...................................14

2.3 Image Processing ...............................................................................17

2.3.1 Digital Image Processing........................................................17

2.3.2 Colour, Grey scale and Binary Images...................................18

2.3.3 Image Processing in LABVIEW..............................................20

2.3.4 Spatial Filtering.......................................................................21

Chapter 3 - Methodology ...................................................................................23

3.1 Hardware required to implement Eye-Tracking …………………….23

3.2 Reference Points ………….................................................................23

3.3 Evaluation of cameras .......................................................................24

3.4 Evaluation of light source ……………………………………………....26

3.5 Final hardware set up ........................................................................28

3.5.1 Choice of camera………………………………………………..28

3.5.2 Experimentation Settings…………………………………….....29

vii

Chapter 4 - Development ...................................................................................30

4.1 Eye Detection ……………...................................................................30

4.1.1 Pattern Matching…............................................................... 30

4.1.2 Edge Detection……...............................................................34

4.2 Pupil Extraction ………………...........................................................35

4.2.1 Introduction.............................................................................35

4.2.2 Filter Application.....................................................................35

4.2.3 Circular- Elliptical edge detection.……………………………...36

4.3 Simulation Environment ………………………………………………....39

4.3.1 Export a CAD design to LABVIEW…………..……..………....39

4.4 Pupil Centre Approach ......................................................................46

4.4.1 Results...................................................................................51

4.5 Eye tracking and simulation environment Integration ………..…...52

Chapter 5 - Discussion .......................................................................................53

5.1 Future Work…………………………………………………………………53

Chapter 6 - Conclusions ....................................................................................54

REFERENCES.....................................................................................................55

1

Chapter 1 - INTRODUCTION

1.1 Background/ Motivation

Disabled people (DP) with quadriplegia or with severe limitations to make controlled

movements-for example, cerebral palsy, traumatic brain injury or Parkinson’s disease-

often have great difficulties to communicate their desires, thoughts and needs even if

they have perfect cognitive functions, such as perfect memory or attention span. They

use their limited voluntary motions to communicate with family, friends and medical

providers. Assistive technology devices have been developed to help them use their

voluntary movements to control computers and other devices such as wheelchairs.

Traditional electric-powered wheelchairs are normally controlled by users using a

joystick, which cannot satisfy the needs of DP mentioned previously. Many alternative

control methods have been developed to satisfy the needs of those patients, some of

them make use of biological signals such as Electromyography,

Electroencephalography, Electrocardiography and Electro-Oculogram (EOG). Other

alternative methods include scleral coil searching, infrared oculography and eye

tracking. Most of the people who suffer neurological and muscular disorders still have

the ability to move their eyes[1]. Some contact eye tracking methods such as EOG

detect eye gaze from the user by analysing and measuring the electrical signals

generated by eye movements. Electrodes placed on specific areas near the eyes can

detect those electrical signals. One of the disadvantages of the EOG method is that

sweat may affect the electrical signal read out and the user need to use electrodes on

the face at all times. In coil search method, a coil is inserted into the eye and a

magnetic field is generated by two coils placed on either side of the head. When a

coil moves in a magnetic field, the field induces a voltage in the coil, hence, eye

movements can be recorded and detected after signal processing. One inconvenient

of scleral search coil method is that it is invasive and uncomfortable for the user.

Eye-tracking is an alternative option to overcome invasive methods, by providing non-

contact operation using a camera for image acquisition. Eye tracking based on corneal

reflection has the disadvantage that requires careful calibration, and require the user

2

to keep the head in a steady position, additionally the available eye-trackers in the

market are expensive due to high computational cost. For example, In year 2000, first

commercial eye-trackers such as the Permobil Eye Tracker, which used googles

including infrared light emitters and diodes for eye-tracking detection cost between

$5000-$29900 dollars (dls) [2]. Nowadays another reduced price eye trackers can be

found in the market, for example the binocular myGaze eye tracker costs between

$2490-$7490 dls. One of the disadvantages is that their products are only available

for business customers and institutes, and the central processing unit (CPU) should

have a minimum speed of 2.8 Ghz which is a high requirement for a low cost

computer[3]. In this dissertation, two main challenges predominate, eye tracking where

computational cost is faced, and achieve eye-tracking control over a vehicle in a

simulation environment. By overcoming those challenges, future work related to this

project can include the implementation in an electric wheelchair. An important

improvement in the life quality of people with physical disabilities can be achieved by

proposing an alternate solution to their mobility issues.

1.2 Aims and Objectives

AIM

The aim of this project is to investigate, design and implement software for control of

an assistive mobility vehicle using LABVIEW. Its focus is on image capture and eye

position data extraction, tracking eye movements and developing a protocol for eye-

based command and communication with a simulated vehicle.

OBJECTIVES

To achieve the aim of the project, the following objectives should be pursued:

1. Conduct background research into assistive mobility vehicles and define the

scope of this project.

2. Gain familiarization with LABVIEW for visual image processing and simulation.

3. Design and develop a simulation environment.

4. Implement camera hardware and software for image capture

5. Design and develop software for eye-position data extraction and eye-tracking

6. Design and develop a protocol for eye-based command and communication

with the simulation test environment

3

7. Integrate the component parts to achieve eye-based control of a simulation

environment.

8. Test and optimize the application.

1.3 Project Management

The project was self-managed by the student, therefore a proper organization was

required to complete the project within the given deadlines. A breakdown of the tasks

and subtasks are outlined below.

a) Develop project definition

b) Develop aims and objectives

c) Project Planning

• Project scope definition

• Identification of required resources

• Task scheduling

d) Background research

• Existent assistive mobility vehicles

• Previous applications using Eye-Tracking

• Principal image processing methods used in computer vision

• Identification of effective computer vision algorithms

• Familiarization with LABVIEW Vision Module and Robotics Environment

Simulator (RES)

e) Execution

• Design and development of Robotics Environment Simulator

• Implement camera software and hardware

• Algorithms execution for image data extraction

• Software programming to perform Eye-Tracking

f) Monitoring and Control

• Model testing on several users and under different lighting conditions

• Represent data by summarizing results in tables and graphs

• Compare data against expected results

• Integrate Eye-Tracking model and Robotics simulation environment

• Optimize application

g) Documentation

4

• Conduct conclusions and future work

• Final report writing

• Oral presentation

Identification of Resources

o People Table 1.1

Resource Availability Additional Notes

Student Yes Responsible for project management and

execution

Supervisor Yes Responsible for guidance and advice during the

project period

Second

Reader

Yes Responsible for project evaluation

o Computer/Software Table 1.2

Resource Availability Additional Notes

Computer Yes Allocated in Bio-Incubator Building

Webcam Yes Purchased as part of the project

Software-

Sketch Up 8

Yes Free software, used for 3D modelling

Software-

LABVIEW

Yes Downloaded from the Internet under Campus

licence agreement

o Others Table 1.3

Resource Availability Additional Notes

Books Yes Available at University Libraries

Journals and

Publications

Yes Available online- Access provided by the University

Internet Yes Provided by University Network

5

Resource management Table 1.4

Cost (Risk) Availability Impact Drivers

Student Time

(High Risk)

600 Hours Time available to

research, plan, design,

develop, test and deliver

the project.

Management: Self-

managed, Project Plan

(see Gantt chart), log

book used for guidance

• Effective

planning

• Tasks

prioritization

• Motivation

• Efficient time

utilization

Supervisor

Time (Medium

Risk)

1 Hour

meeting, 1

session per

week

Discussion regarding

project advance, advice

on viable solutions and

points to consider to

ensure successful project

completion.

• Quality and

improvements

undertaken by

the student

between each

meeting.

Access to

Expert

Knowledge

(High Risk)

None Access to expert advice,

by benefiting from their

extensive knowledge on

Eye-Tracking and RES.

Management: Student on

the need to become a

Subject Matter Expert.

• Conduct

efficient

research to

achieve

proficiency on

the subject.

• Read

manuals,

tutorials and

previous

approaches to

support

project

evolution.

6

Webcam

(Low Risk)

£70 A webcam with the best

meeting criteria was

required to conduct the

project.

Management: Device within

project budget and

immediately available from a

large range of suppliers.

Research was required to

select appropriate camera

for the project.

• Research on

available web

cameras with

auto-focus

capability.

• High

resolution

camera

requirements

for accurate

eye-tracking.

Infrared Light

Source and

Power Adapter

(Low Risk)

£20 Infrared light source was

required for eye-tracking

experimentation under

different lighting conditions.

Management: Infrared light

kit instantly available and at

a very low cost. Research

required to select viable kit.

• Amount of

light,

brightness

and shape of

the kit, they

are all factors

to consider as

the kit will be

used in

conjunction

with the

camera.

7

8

Chapter 2 – LITERATURE REVIEW

2.1 Eye-Tracking

2.1.1 Definition of Eye Tracking

The term Eye-Tracking refers to a set of technologies including software and hardware

that allows the monitoring and recording of the manner in which the point of gaze is

directed to a particular scene or image.

The device that is usually used to measure the eye movements and eye position is

commonly known as eye tracker. Generally, there are two different types of eye

movement monitoring techniques: the first one measures the position of the eye

relative to the head, and the second technique, which measures the orientation of the

eye in space, or the point of regard [4]. Probably the most widely used device to

measure the point of regard and to extract the eye position is the video-based corneal

reflection eye tracker.

2.1.2 Eye-Tracking Methods

Most of the popular eye movement measurement techniques principally fall into

three categories, below is listed the list of categories and its most representative

example:

1. Object movement measurement – Search coil/ Scleral contact lens

2. Measurement of electric potential – Electro-Oculography (EOG)

3. Optical tracking – video based corneal and pupil reflection

Search coil is a method that relies on a voltage induced by a coil. This technique

involves attaching a mechanical or optical reference object directly on the eye, such a

special contact lens that extends over the cornea and sclera, a magnetic field is

generated by placing two extra coils on either side of the head. The eye movements

can be recorded when the coil moves in the generated magnetic field, inducing a

voltage in the coil that can be measured and then processed using specialised

equipment.

9

Figure 1. Search coil used for eye-tracking [5].

Electro-oculography (EOG) is an eye movement recording method that relies on

measurement of the skin’s electrical potential differences. On EOG, small metal discs

called electrodes are placed on the skin around the eye, this technique measures

electrical activity related to eye movements.

Figure 2. EOG measurement [6]. While the above techniques are suitable for eye movement measurements, they need

direct contact with the user, using either contact lenses or electrodes on the skin.

Video-based corneal and pupil reflection is a contactless alternative that uses features

such as corneal reflection (from a light source, usually infrared light) and the pupil

centre for eye-tracking. The infrared light is reflected from the eye and sensed by a

video camera or other designed optical sensor. The data is then processed to extract

10

the rotation on the eye from changes in reflections. Commonly, the video based eye-

trackers use corneal reflection (called first Purkinje image) and the pupil centre, as

features to track during execution. Nowadays, this non-invasive video technique is

widely used for gaze tracking for several applications.

Figure 3. Example of Video-based corneal and pupil reflection. 2.1.3 Popular Applications

The continuous advances in computer technology and image processing, allows eye-

tracking techniques to be used in a wide variety of disciplines, including: cognitive

science, psychology, human-computer interaction (HCI), marketing and medical

research among others.

The human eye is a complex organ with several functions, movement patterns and

behaviours. In our daily life we are highly dependent on our eyes for acquiring details

from our environment, taking information constantly and stimuli that are then

processed by our brain. Eye tracking is a technology that can be applied in countless

ways by exploiting the potential of visual activities. The most relevant applications that

use eye tracking technology are described below.

o Human Computer Interface Devices

A field of application for eye-trackers extends to people with extreme disabilities. A

revolution in human computer interaction (HCI) offers an alternative to those disabled

people to interact with the computer. An example of a commercial device using this

11

technology is the remote Eye-Tracker called PCEye, which allows the user to work

with an application that is normally controlled by a standard computer mouse. Some

examples include: surfing the web, connect with friends online and even make

spreadsheets and documents by using the eyes [7]. Moreover, it has been shown that

using peripherals such as the mouse and keyboard cause wrist injury and negatively

affect joint health in the knuckles. An eye tracker is a wise alternative to avoid those

health risks, allowing the user to look naturally at the screen and select icons by blink

clicks or intentional prolonged gaze on a desired area.

Figure 4. Example of an Eye Motion controlled virtual mouse [8].

o Consumer Behaviour Research

Perhaps this application is the biggest research field in terms of money profit that make

use of eye tracking technology. Eye trackers allow the examination of the customer’s

visual behaviour while interacting with a target product, usually this procedure is

utilized while the product is in the prototype stage. In order to conduct this research,

different people are sent to a supermarket wearing a portable eye tracker. The

procedure is relatively simple; two cameras are aimed at the users’ eyes to record the

eye gaze while interacting with different products in the store, an additional camera is

used to record the external environment, by collecting data from both sources, it is

possible to detect and extract the position in the image that mostly attracted the

consumer’s attention. The attractiveness and the tendency of the package to be

chosen for the purchase may be based on many factors such as product location,

container size or product presentation. The summary of the user interaction is usually

represented by heat maps.

12

a) b)

Figure 5 . a) Innovative marketing research using eye tracking glasses. b) Viewing

frequency, less attention is detected on the products down right [9].

o Driver distraction / Drowsiness

With the extensive mobile phones proliferation, drivers are now more distracted than

ever, exposing them to have car accidents. Additionally, high levels of stress

generated in a society that demands more working hours and fewer hours of sleep

than in the past, produces a risky situation among drivers. Drowsiness and driver

distraction is a dangerous combination that results in thousands of accidents every

year. Research is currently underway to incorporate eye tracking cameras into

automobiles. The objective is to provide the vehicle with the capability to assess in

real-time the visual behaviour of the driver. Lexus in 2006, incorporated the first

monitor system in the model LS460, a warning is provided if the driver takes his eyes

off the road to prevent car accidents [10].

o Website Usability Testing

Nowadays, websites have become a key component to any business model.

Essentially if the company is not available in the World Wide Web (www), it does not

exist for the customer. The user interface provided by web pages are very often the

only way to contact the company, therefore it is essential for the company to catch the

user’s attention to improve sales, which can be achieved through web page

attractiveness, great usability and competitive prices. Eye tracking technology is

frequently used to evaluate website usability as the tracking devices made for this

application are becoming accessible and inexpensive. One of the tools to represent

13

the user’s experience and provide feedback to the website owner is the use of graphic

fixation and saccade representation that shows gaze patterns and visual attention

while the user navigates in the site.

Figure 6. Heat map of Google homepage [11].

o Assistive Technologies

The quality of life of the people is enhanced by technological advances, this being

even more so for people with limited mental and physical capabilities. The use of eye

tracking technology offers the benefit to be applied to both assistive communication

and assistive mobility vehicles. As mentioned previously, some software programs are

controlled through gaze tracking as a HCI method. The term assistive technology

according to the Assistive Technology Industry Association, is “any item, piece of

equipment, product or system, modified or customized, that is used to increase,

maintain, or improve the functional capabilities of individuals with disabilities” [12]. The

objective of this project is centralized at this point, where an important approach is

presented in a simulated vehicle using eye-tracking technology, it is of great

importance that the continuation of this project includes the implementation of eye

tracking with assistive devices so that it will give patients with paralysis the ability to

become mobile, improving the quality of life of many disabled people.

14

2.2 Wheelchair mobility (check feedback from here up to page 36)

2.2.1 Introduction

The number of disabled and elderly people who experience difficulties in walking is

increasing. An explanation of this event is that the life expectancy has gradually

increased due to the great advances in the medical field and the great effort to

eradicate existing harmful substances in daily life consumables, resulting in a fast

growing population of senior citizens in Europe and developed countries [13]. It is

estimated that by the middle of the century, the age of the 20% of elderly citizens will

be 80 years or even more [14]. Due to lack of balance or strength, many elderly need

help from nurses to perform their regular daily activities. At this point, assistance robots

or Electric-Powered Wheelchairs (EPW) can reduce nurse’s workload and help the

elderly people to do their daily activities with minimum support from care providers. It

is important to consider that the use of a joystick or other standard input device may

not be enough to fulfil the needs of some people to navigate within a home or other

environment. Due to this situation, through research and development, new ideas

have raised to use different types of control interfaces according to the needs and

abilities of the people. Additionally, independent mobility is crucial for the development

of cognitive, physical, communicative and social skills [15].

2.2.2 Electric Wheelchair Control Interfaces

Electric-powered wheelchairs can be defined in function of the interface device and its

forms. Below are described electric wheelchairs that use different control interfaces.

o Joystick controlled wheelchair (JCW)

Perhaps the most known EPW is the one that uses a joystick as a manual input

controller on which the human operator can manually give driving commands such as

direction and velocity [16], [17]. In summary, the JCW is a modified version of the

industry standard wheelchair, it is composed of two motor integrated wheels, a chassis

that holds the battery and a joystick controller mounted on the arm rest, which provide

15

the input to pulse width modulation (PWM) circuits, resulting in a pulse train (PT). The

PT deliver driving power to the motors. Generally the electric motors are powered by

12-80 ampere-hour rechargeable deep cycle batteries at 12 Volts [18].

Recent smart wheelchairs integrate the capability to navigate avoiding obstacles to

reach the desired destination in a safe and efficient way [19]. Safety and efficiency

should be considered for future implementation of this project in a real EPW.

o Voice Controlled Wheelchair

Great advances in technology and cost reduction in electronic equipment has allowed

many researchers to investigate and develop possible solutions to fulfil the needs of

people that cannot use a conventional electric wheelchair for autonomous mobility.

The application of voice control has long been pursued as an alternative control

mechanism for wheelchairs and many approaches have been made in relation to this

topic [20].Voice control has yet to become a viable control method for electric

wheelchairs due to the challenges that need to be faced such as the limited bandwidth

of the voice, which makes it very hard to make frequent fine adjustments to the

wheelchair’s velocity. Other difficulties include external noise from the environment

where the wheelchair is moving, accuracy of the digital signal processor to avoid

misinterpretation, and the ability from the user to speak clearly in the language

programmed, using the pre-established words to be used as control commands.

o Brain Controlled Wheelchair

Other approach to move an EPW include the use of Brain Computer Interface (BCI)

techniques. BCI is used as a direct interface between the human brain and a computer.

These techniques are classified into invasive and non-invasive. Important research is

being done in relation to non-invasive techniques since they are becoming increasingly

popular. Examples of non-invasive BCI techniques include Electro-Encephalography

(EEG) and Electro-Oculography (EOG). In the case of EEG technique, it uses a set of

electrodes placed on the scalp of the user to acquire EEG signals. Depending on the

scalp potential differences detected by the electrodes, allows the classification of the

16

potential differences into different bands such as alpha, beta, theta and delta. Each

band has different frequency ranges and those bands are correlated to human

emotional states such as relaxation, concentration and deep sleep. Some researchers

had proposed the use of thought controlled wheelchair systems, where the captured

signals from the brain are processed to control an EPW. The main disadvantage of

using this interface is that the speed of the brain signal processing is not fast enough

to control the wheelchair on real time, other disadvantages include the user’s

concentration level. If the level does not exceed a given threshold, the system is

unable to detect the user’s intention. Additionally, the use of electrodes on the scalp

is not comfortable for the user. In some cases the use of inductive gel on the electrodes

is needed before contact with the user's scalp to improve signal processing [21].

o Eye-tracking controlled wheelchair (JCW)

Eye-tracking as an alternative control interface is faced on this project, the viability to

implement this technology and at a low-cost is discussed, with the objective to fulfil the

needs of people with limited mobility and affected speech, the main challenges are

identified and possible solutions to each case are proposed.

Figure 7. An electric-powered wheelchair with possible control interfaces.

17

2.3 Image Processing Given that this project is focused on the detection and extraction of important data

derived from eye movements to then give control commands to displace a vehicle in

relation to the information extracted, it is important to explore the theory and

techniques used in image processing. Through image capture and image processing,

it will allow to analyse important features of the eye during the program execution.

2.3.1 Digital Image Processing

A digital image can be defined as a two dimensional array composed of small-squared

regions called pixels. Images can be produced by several physical devices, such as

still and video-based cameras, ultrasound, microscopes, telescopes and x-ray devices

among others. The acquired images can be used for a variety of purposes including

medical (e.g. x-ray), business (e.g. documents), security (e.g. authorized personnel),

military (e.g. reconnaissance), entertainment (e.g. familiar photos), civil (e.g. traffic)

and scientific (e.g. research). The aim in each case is for a human or a machine, to

extract relevant information from the image that is examined. Figure 8 shows an

example of an industrial application that uses image processing to make a process

more robust and efficient.

Figure 8. Digital image processing in GM is used to verify that the correct tire is installed [22]. A) Light to allow visibility B) Camera for image acquisition C) Inspected tyre Digital image processing (DIP) can be defined as a set of operations made over an

image, usually made by computer software. DIP allows the enhancement of Region(s)

18

of Interest (ROI) while irrelevant details are attenuated or removed in a given

application, which allows to extract relevant data from the enhanced image. In order

to acquire a digital image from a physical device, some internal processes are involved

such as image capture, sampling, quantification and codification. An image can be

defined as a two dimensional function that quantifies the light intensity, the image is

generally described as I (x, y), on which the intensity value known as pixel, is the value

obtained from the indexed process of the coordinates x and y. The most representative

model of an image is given by a matrix, shown in Equation 1.

I (x, y) = I (1, 1) I (2, 1) I (N, 1)

I (1, M) I (2, M) I (N, M)

Equation 1 Fig. 9 Two dimensional distribution Where, N and M = Image size representation of image points (pixels). As mentioned in the previous paragraph, digital images can be modified by software

programs for different purposes. LABVIEW is one of those programs, it can access

the matrices that correspond to the image and perform different mathematical

operations, which results in a modified output image.

2.3.2 Colour, Grey scale and Binary Images Images than can be manipulated using image processing can be divided into three

types:

• Colour images

• Grey-scale images

• Binary images

o Colour images

Humans detect colour through wavelength-sensitive photoreceptor cells called cones.

Three types of cones are present in the retina of the eye, each cone has a different

19

sensitivity to electromagnetic radiation (light) of different wavelength. The important

point to consider is that one type of cone is principally sensitive to red light (R), the

second type to green light (G) and the third one to blue light (B). People are able to

perceive almost any colour when a controlled combination of these basic colours

(RGB) is emitted and the three types of cones are stimulated [23].That is the reason

why colour images are usually stored as three separate image matrices, the first matrix

stores the amount of red in each pixel, the second matrix stores the amount of green

and the last one the amount of blue. Thus, those colour images are called RGB due

to the format they are stored in.

Typically each colour pixel is stored using three 8-bit bytes; where each byte

represents a colour (Red, Blue or Green). Each byte ranges from 0 to 255, where 0

indicates the absence of that primary colour in the pixel and 255 indicates that the

maximum value of that primary colour is present in the pixel. Since each byte has 256

different states, this leads to a total of 16777216 possible colours per pixel in a 24 bit

RGB colour image.

Fig. 10 Information inside a pixel in a 24 bit RGB colour image.

o Grey-scale images

A Grey-scale image is composed by a matrix of size M X N as shown in equation 1,

each element in the matrix (pixel) has an assigned intensity that ranges from 0 to 255

in the case of an 8-bit greyscale image. A grey-scale image is composed of many

shades of grey. As shown in Fig. 11, each pixel has an integer value that ranges from

0 (black) to 255 (white), and grey levels are the numbers in between, for example, in

a linear scale 127 corresponds to 50% of grey level. The range of pixel values depends

20

on the colour depth of the image, in 8 bit case results in 256 different greyscales or

tones.

Fig. 11 Possible pixel values in a grey-scale image.

o Binary images

A binary image is composed by a logical array containing a value in each pixel of 0 or

1 in the matrix. The process to create a binary image is done using a threshold

technique in function of the pixel intensity, the pixel can take two logical values. If the

pixel intensity exceeds a given threshold value, the pixel takes a value of 1 (white) and

0 (black) if the pixel intensity is below the threshold, displaying an output image known

as monochrome or black and white.

A) B) C)

Fig. 12 Same image containing different information per pixel. A) Colour image B) Gray-scale image C) Binary image.

2.3.3 Image Processing in LABVIEW

LABVIEW (Laboratory Virtual Instrumentation Engineering Workbench) is a

graphically based programming language. Because of the graphical nature of this

platform makes it ideal for electrical and control engineering applications such as test

and measurement, automation, instrument control, data analysis and data acquisition.

LABVIEW has been chosen for this project because it allows the creation of

21

complicated applications in a reduced time, process images with a complete suite of

algorithms and integrate with motion drives and automation devices. Programming in

this robust graphical language allows the use of a single software package for image

acquisition, image processing and robotics environment simulation. Additionally, it is

relatively easy to implement serial communication to control a real electric wheelchair

for future work related to this project. LABVIEW can be expanded by adding a range

of modules, the relevant modules for this project are the LABVIEW Robotics Simulator

(LRS) and the NI Vision Development Module (VDM).

2.3.4 Spatial Filtering

During image analysis, after gathering information from the image, the user may want

to improve the image quality for inspection. Spatial Filters (SF) are used for that

purpose, filters can smooth, transform, sharpen and remove noise from an image,

allowing to extract the desired information.

SF can alter pixel values in function to variations in the light intensity in their

neighbourhood, where the neighbourhood of the pixel is defined by a matrix size or

mask, where the centre is on the pixel itself. The filters mentioned can be sensitive to

light intensity variations such as the presence or absence of light.

There are two categories of spatial filters:

o High-pass filters

o Low-pass filters

High-pass filters (HPF) are used to emphasize variations caused by light intensity,

these variations are generally found at the objects boundaries. HPF help isolate

sudden changes in light intensity in an image, hence these filters are used to enhance

details in an image, the disadvantage is that HPF tend to also increase high frequency

noise along with the region of interest in the image.

Low-pass filters (LPF) on the other hand, can help to remove insignificant details by

removing sharp edges, smoothing the image and emphasizing patterns such as the

22

background and objects. Hence, these filters have the tendency to smooth images by

removing noise and blurring edges.

Spatial Filters are relevant to image processing, as mentioned previously they can

help to manipulate images by improving or reducing image sharpness and they can

help to reduce noise. The filter used for this project is the low-pass filter, which helps

to smooth the image attenuating high frequencies signals present in the image which

allows to reduce noise such as light reflections on the eye of the user.

A) B) C)

Fig. 13. Examples of high-pass and low-pass filters applied to an image to smooth or

highlight high frequencies signals such as corneal reflections. A) Original Image

B) Low-Pass Filter C) High-Pass filter.

23

Chapter 3 – METHODOLOGY

3.1 Hardware required to implement Eye-Tracking

To conduct the experimentation, the fundamental resources to implement eye tracking

should be identified, the hardware needed includes:

o High sensitivity camera- webcam

o Light source- infrared illumination

o Power source for infrared light- 12 Volts 1000 milliamps (mA) AC/DC adapter

Eye-tracking using a webcam does not require any specialized equipment, hence, the

test users do not have to come to a lab, therefore all experimentation was carried out

indoors.

3.2 Reference Points

Two main reference points exist in order to identify the eye centre position, the limbus

and the pupillary zone. Two methods can be identified to do eye-tracking relating the

reference points previously mentioned.

1. Basic Method: This method involves comparing the position of the limbus in

relation to the sclera. Example: If a human’s iris and pupil are showing to the

right of the eye, with the sclera on the left, the person must be looking to his or

her right. This method of gaze estimation is used by humans on a daily basis,

it is used to know where other people are looking.

2. Advanced Method: The second method is used to find the centre of the pupil.

This can be calculated by finding the limbus or the pupillary zone, which is the

zone located between the iris and the pupil edge. Both of these edges are said

to be circular, and share a common centre which is the pupil centre.

Fig. 14. Representation of the visible regions of the eye[24].

Eye tracking methods: 1) Basic Method 2) Advanced Method

24

3.3 Evaluation of Cameras

The previous evaluation of suitable cameras for the project was fundamental. In terms

of eye-tracking, the higher the resolution of the camera, the more accurately the user’s

eye can be detected.

Standard video camera and image resolution are shown in Table 2 below.

Video

Standard

Full name

Display

Resolution(pixels)

Width x Height

Aspect ratio

Width: Height

VGA Video Graphics Array 640 x 480 4:3

SVGA

Super Video Graphics Array 800 x 600 4:3

XGA Extended Graphics Array 1024 x 768 4:3

HD High Definition (720p) 1280 x 720 16:9

SXGA Super Extended Graphics

Array

1280 x 1024 4:3

UXGA Ultra Extended Graphics

Array

1600 x 1200 4:3

FULL-HD Full High Definition (1080p) 1920 x 1080 16:9

Table 2. Table of computer display standards

25

A research was conducted to investigate the five best existing web cameras in relation

to image quality and features relevant to the project. In Table 3 the actual top five web-

cameras are shown, comparing characteristics of each one of them.

Table 3. Relevant features for the project are highlighted in yellow [25].

26

3.4 Evaluation of Light Source The majority of existing eye trackers use infrared (IR) light from light emitting diodes

(LED’s) in order to illuminate the eye. The use of IR light also allows measurements

to be taken from relative locations of the pupil and use the LED’s reflection off the

cornea to calculate the eye orientation and position. This approximation is known as

pupil centre corneal reflection method. The use of infrared light in conjunction with an

infrared light-sensitive camera improves the effectiveness in an eye-tracking system,

due to the fact that it is infrequent to pick up other IR light which can cause

disturbances or noise for the system, an example is shown in Figure 15. Finally, the

use of IR was selected for this project since the lack of a controlled light source

severely reduces the system accuracy and negatively affects the results.

Figure 15 - Eye showing corneal reflections from two IR light emitting diodes.

As mentioned previously, the use of infrared light together with an infrared light-

sensitive camera is fundamental in any eye-tracking system. One important point to

consider is that most of the web cameras do not detect IR light, the reason is that most

of them are built with an IR filter to filter out any IR light. It is significant to know that

many light sources emit infrared light including the sun. A colour camera exposed to

daylight without an IR filter will catch an important amount of IR light, resulting in

strange colours present in the image. Due to that situation, it was required to conduct

a modification in the camera to remove the IR filter, taking special care in the exact

lens position, Failure to follow this procedure meticulously may result in the camera

being focused to infinity all the time. The IR filter was removed using a heat gun in

order to not damage the original filter. The steps to achieve this modification can be

followed through an online tutorial [26]. The complete procedure followed to modify

27

the hardware and remove the infrared filter can be found in the Compact Disc attached

to this dissertation project in the folder named “Webcam IR filter removal”.

A twelve IR Led board plate was chosen for this application. After experimentation it

was discovered that illuminating the eye with many IR LED’s produces a lot of

disturbances during image processing when the pupil wants to be extracted from the

image acquired. After testing, the use of two IR LEDs as the lighting source

significantly reduced the noise. Moreover, using two LED’s is sufficient to allow good

visibility to the eye. No modification on the printed circuit board (PCB) is needed. This

can be achieved by isolating the light coming out from the rest of the LED’s on the

electronic board. An illustration of this experimentation is shown in Figure 16.

Figure 16. Corneal Reflections using different amount of infrared light. 1. Illumination

on the eye given by twelve IR LED’s. 2. Illumination on the eye given by two IR LED’s.

1 2

28

3.5 Final hardware set up

3.5.1 Choice of camera

Relevant characteristics for the project were considered in order to choose the most

appropriate camera for this application, considering the following aspects:

o Image quality

o Features

o Design – suitable to incorporate the IR light electronic board.

The most important aspect of a webcam is the image quality. The maximum resolution

that an actual webcam can offer is 1920 x 1080 and a video capture speed (VCS) of

30 frames per second (FPS). Both aspects are correlated in creating full HD video,

even if a camera is capable of providing a resolution of 1080p, if the VCS is less than

30 FPS it may cause inconsistent video stream due to reduced speed. On the other

hand, the best still image quality that a webcam can offer at the present time is 15

megapixels. On this basis, the Logitech c920 HD Pro Webcam was chosen since it

has the best photo quality and the best video recording that a webcam can offer at the

present day. Additional to image quality, this camera offers features that are relevant

to this project which are: 20-step autofocus which allows to take high quality images

at short distances. Also, this camera is tripod ready and the minimum system

requirements are not high for a low-cost computer. The minimum Central Processing

Unit (CPU) required is 1 Gigahertz (GHZ) and a minimum memory RAM of 256

Megabytes [27] .Figure 17 shows the incorporation of the selected camera and the IR

Led module used for conducting the present project.

Figure 17. Implementation of selected Webcam and lighting source.

29

3.5.2 Experimentation Settings

The testing and development was performed on an i3 dual-core laptop computer with

a 1.8 GHz processor, 64-bit operating system and 4 Gigabytes memory RAM. The

operating system installed is windows 8. This dissertation requires image processing

pattern recognition algorithms to achieve eye-tracking. As described previously in

chapter 2.3.3, LABVIEW, a high-level graphical programming environment was

chosen to perform all tasks involved in the project.

The implemented system uses the black pupil corneal reflection (refer to chapter

4.2.1). The camera delivers the image of one eye with a selected resolution of 640 x

480 pixels at 30 frames per second. All measurements are done at 60 Hz sampling

rate which is about one frame every 17 milliseconds. A distance of 5-10 centimetres

exists between the eye and the camera as shown in Figure 18.

A) B)

Figure 18. Final hardware set up. A) Distance between the user’s eye and the

camera=5cms. B) Distance between the user’s eye and the camera=10cms.

30

Chapter 4 - DEVELOPMENT

4.1 Eye detection

Detecting the eye is fundamental in eye-tracking and iris recognition, basically if the

position of the eye is located, the point of gaze can be determined. Usually in remote

eye trackers, the eye detection follows two steps: in the first step, the face is located

in order to extract the eye region. In the second step, the eye is detected from an eye

window. It is important to consider that if an eye tracker is at a considerable distance

from the eye, more image processing is required in order to detect the user’s face and

then process the eye’s location within a relatively small region. The reduction of high

computational costs faced in this project implies the minimal use of image processing

algorithms, while considering that the system needs to be reliable, which implicates

an accurate location of the eye position. For that reason, the experimentation for this

project was conducted at a close distance between the user’s eye and the camera (5-

10 centimetres) which is the case on wearable eye-tracking systems.

4.1.1 Pattern Matching

Pattern matching (PM) is a common and important inspection task in a machine vision

application. Usually PM provides information in relation to the presence or absence of

an object, the number of instances and the location of the defined template in the

image (pattern).

From a simple perspective, assuming that the eye-tracker is kept at a fixed position

and at a set distance from the user’s eye. It can be determined that if the initial eye

centre position is known with the corresponding coordinates on the plane xy, the

position of the eye centre in the next frame can be compared with the initial position in

order to determine if the user is looking to the left, right, up or down. The final step

would be to generate a command to the simulated vehicle to displace it forward,

backward, left or right, in function of the eye centre position.

In relation to the previous paragraph, several methods are tested to prove eye-tracking

accuracy with the minimum use of image processing algorithms to reduce high

computational cost and achieve real-time eye-tracking at 30 frames per second in a

low-cost computer.

31

The first approach to identify the position of the eye is through recognition of different

patterns (templates) within an image. For simplicity, an image containing arrows

pointing at different directions (left, right, up, down) and a circle at the centre as shown

in figure 19. That symbolizes the five different states that the simulated vehicle can

receive in order to produce a movement in the simulation environment, hence, five

different templates are created. Each image acquired by the camera is analysed in

order to check the presence of the pre-set templates during image processing. An

input called “minimum match score” is given by the user to set the detection sensitivity,

it is used as a threshold so that if the template exceed that value, the output of the

image processing algorithm provides data related to the object (see figure 21),

indicating presence of the ROI (template) in the image. Depending on the similarity

level of the pattern found in the image, a different match score ranging from 0 to 1000

is given (a perfect match would score 1000), an example of the outgoing information

given by the pattern matching algorithm is shown in figure 21 .

Figure 19. Image used to differentiate between different templates in

“pattern matching”.

Note: The VI programmed in LABVIEW for this experimentation can be found in the

Compact Disc attached to this dissertation project in the following route: Dissertation

Project\LABVIEW VIs\Eye-Tracking VIs\Pattern Matching\Arrow-Tracking.vi

32

The simplified structure of the VI programmed is the following:

Figure 20. First Approach- Steps for performing pattern matching.

33

Internally during image processing, LABVIEW perform the pattern matching search,

the data shown below in figure 21 is available for the user, the user needs to extract

the information from the vision processing VI, the output is a 1 dimensional array, and

inside that array, a cluster of 5 elements exists, the information needs to be unbundled

in order to use it as an indicator or as an input control for another process.

Figure 21. Location of the defined template (plane x-y coordinates), angle and score

Figure 22. Execution of the VI programmed. 1. Image acquisition 2. Template

searching. 3. Indicator of the template presence in the image acquired.

34

4.1.2 Edge Detection

The algorithms used in image processing traditionally sort the type of information that

exists in an image such as edges, textures, surfaces, patterns. Some machine vision

algorithms can extract one or more information types.

An edge detector in LABVIEW use edges that are present in an image. This type of

algorithm and related techniques can locate the edge position of an object with high

accuracy. One of the great advantages of this technique is that multiple edge locations

can be combined in order to compute circles, intersection points, projections or

ellipses. Additionally, the edge detection algorithm is able to take measurements such

as the width of a section in the image called clamping. On the other hand, the reliability

and robustness of the measurements depends on the stability conditions of the image

acquisition. Components such as lighting, vibration control and sensor resolution

among others, are all factors of the image acquisition process that directly affect the

accuracy of the measurements. Due to the impact of those factors in image

acquisition, an adequate selection of the camera with high image-video resolution was

fundamental, along with the selection of a controlled light source such as infrared light,

both factors lead to an improvement in the accuracy of the image measurements.

35

4.2 PUPIL EXTRACTION

4.2.1 Introduction

The measurement of the pupil-centre location is a critical task of the algorithm used in

an eye-tracking application. A good contrast between the pupil and the iris is needed

in order to extract accurate pupil measurements, this contrast is generally generated

through the use of controlled infrared light reflected by the eye. When the eye centre

location wants to be extracted, the pupil or the limbus (boundary between sclera and

the iris, as shown in Fig. 14) can be tracked, due to the high contrast between their

neighbour sections. The advantages of pupil tracking over limbus tracking are the

following:

o The limbus is more covered by the eyelids than the pupil.

o The pupil’s border is regularly sharper than the limbus, resulting in a higher

resolution.

4.2.2 Filter Application

As mentioned in section 2.3.4, the use of filters during image processing is one of the

basic tasks applied right after image acquisition. It allows the user to smooth,

attenuate, reduce or improve sharpness in order to compensate the noise caused by

external factors. A low-pass filter is used for the project and more specifically during

pupil extraction. The filter used called “smoothing-median”, helps to smooth the image,

allowing to attenuate the high frequencies in the image (e.g. high-light reflections).

Smoothing median (SM) is a type of linear low-pass filter, available during image

processing in LABVIEW for grayscale images. The SM attenuates the variations in the

light intensity in the neighbourhood of a pixel. In summary, this filter smooths the shape

of existing objects, remove details and blur the edges. Depending on the kernel size

selected, the smooth effect changes. A larger kernel size represents a stronger

smoothing effect, as shown in figure 23 where the image on the left is the original

image, the image on the centre is the processed image with the minimum kernel size

x=1, y=1 (minimum smooth), image on the right corresponds to the processed image

with a selected kernel size x=10, y=10, the effect is noticeable in the corneal reflections

and the eyelashes.

36

A) B) C)

Figure 23. Smoothing median filter applied to an image. A) Original Image

B) Minimum kernel size C) Selected kernel size

4.2.3 Circular- Elliptical Edge detection

In order to conduct an appropriate pupil tracking system and take measurements

based on characteristic features of the object within an image. It is important to know

every aspect of the object. In this case, the pupil shape. In most people, the pupils are

roughly circular and slightly elliptical pupils can be normal. When the pupil is more

elliptical than normal, could be a symptom of astigmatism. Astigmatism is when your

cornea is not a round shape. Instead it is an oval shape. This situation is common

since most people have a form of astigmatism, but it is generally mild and doesn’t need

special treatment [28]. Therefore, due to the characteristics described previously, a

novel strategy is proposed to use a combination of circular edge detection together

with elliptical shape detection so that the algorithm proposed can be more robust, and

can be applied to more people, regardless of the pupil shape.

The machine vision function in LABVIEW called find circular edge, locates a circular

edge in a region of interest defined by the user. Fig 24 shows an example of this

function. A binary object (circular shaped) exists in the image, although the object is

not perfectly round, with the information extracted from the existing edges, the

algorithm produces a circular object. This is one of the main advantages of using this

algorithm, since it is highly relevant for the project. An example of a situation where

this algorithm can be used, is when the pupil is occluded by the eyelid. The algorithm

makes a search and if it finds an object with circular characteristics (adjustments made

by the user via the settings tab, see figure 24), even with a few points (data) extracted

from the circular edges, a complete circular object is produced. The output of this

37

function provides important information respect to the circular object detected, such

as: position (x-y), radius and deviation as shown in Fig. 25. Which is data that is used

in this project to extract the pupil centre position.

The complete algorithm used for pupil extraction that includes: circular detection,

elliptical shape detection and other processing functions, is described in detail in

Section 4.4.1

Figure 24. Processing function “find circular edge” with its parameters.

Figure 25. Information extracted from the circular object.

On the other hand, the processing function called shape detection, is capable of

detecting ellipses within an image, previous adjustments given by the user as shown

in figure 26 below. The difference and advantage respect to the circular edge detector,

is that the elliptical edge detector, only provides information about the object if the

conditions set by the user are met. This point is relevant to the project, instead of

interpolate the data like in the circular edge detector, the elliptical edge detector is

based on a minimum match score, which is an input given by the user along with the

curve parameters. A second novel strategy is proposed at this point, the purpose of

38

the suggested strategy is to do execute the calibration process automatically during

the first seconds of operation (simple instructions for the user). The algorithm

automatically change a threshold in each frame (described in detail in Chapter 4.4.1) ,

at the same time, it scans for similarities on the output of both the elliptical and circular

edge detectors, when the radius and the position output from both algorithms is

minimal, it is likely that the pupil centre position was detected.

The output of this function provides important information respect to the elliptical object

detected, such as: position (x-y), major radius and minor radius (due to the shape of

the ellipse, two different radius exist), score, and angle as shown in Fig. 27 Which is

data that is used in this project to extract the pupil centre position.

Figure 26. Ellipse Detection setup.

Figure 27. Information extracted from the elliptical object.

39

4.3 SIMULATION ENVIRONMENT

The use of a simulation environment is fundamental in any robot development, the

reason is that developers can validate their designs and algorithms in an effective and

efficient way. The LABVIEW Robotics Simulator (LRS) is a simulator used for the

emulation of a robotics design. The LRS include useful tools to make an application

more robust, some of them includes: sensor and actuator interfaces, vision tools,

communication protocols, simulation and control design. The advantage of using LRS

together with an eye-tracking application is that it allows the user to use a custom 3D

CAD design to simulate an environment or a robot that can be very similar or with the

same characteristics as one existing in the real life. For example, this design can

represent the open environment or the hospital where the EPW could be implemented.

4.3.1 Export a CAD design to LABVIEW

The procedure to export a 3D Cad design to LABVIEW from 3D modelling programs

such as: SolidWorks, AutoCAD or Google Sketch Up is not completely straight

forward. When a simulated mobile robot is chosen to be used in a Simulated

Environment (SE) within LABVIEW, existing parts of the SE are defined in *.ive and

*.stl files. The LRS use these files together with other data in order to render the

environment. After becoming familiar with the basic functions and existing tools in the

3D CAD modelling programs mentioned previously, it was discovered that the 3D CAD

model should be saved in two different formats: *. stl (stereolithography) and *.dae

(collada file). After that, the *dae format needs to be converted to *ive format (via the

VI mentioned in Step 4 in the following tutorial). This procedure is required so that

LABVIEW can import the designed model and render it. The only program that allowed

the program to be saved in both formats was Sketch Up version 8 (SKP). By default,

SKP allows the model to be exported in *dae format, but the model should be

converted to *ive format as mentioned previously. Finally, in order to export the model

to *stl, an extra plugin needs to be downloaded, available online [29]. A detailed tutorial

is shown below, describing each step to achieve the exportation from a 3D CAD design

to LABVIEW.

40

The following steps should be followed to take a custom 3D CAD model design into

the LRS.

1. Design the 3D CAD model in Google Sketch up 8.

Figure 28. 3D CAD model designed in Google Sketch Up.

2. Once the model is completed, from the tab menu select File -> Export -> 3D

model. Save the model as ProjectName.dae (collada file).

Figure 29. Step 2 Illustration. 1. Export 3D Model 2 Save file (dae format).

41

3. Download and install the plugin mentioned in the previous paragraph in order

to save the file in *stl format. Select the complete 3D model and from the tab

menu select: File -> Export to DXF or STL -> from the drop down menu select

export units: Millimetres -> From the Export to DXF options select “stl” -> save

the file as: ProjectName.stl.

Figure 30. Step 3 Illustration.

42

4. In LABVIEW, open and run the VI called "CAD Model Converter", usually found

in the following location: “C:\Program Files (x86)\National

Instruments\LabVIEW2012\vi.lib\robotics\Simulator\Interface\Utility\Converter\

CADModel Converter.vi")

4.1 In the “input path” text box, select the file saved as “ProjectName.dae” from

step 2, and click load. A list of objects derived from the CAD model should

appear on the left side of the window under the name “Node List”.

4.2 In the “export path” text box, choose a name with extension *.ive for

example: “ProjectName.ive” and save it in the following directory:

“C:\Program Files (X86)\National Instruments\Labview

2012\resource\robotics\simulator\CADModels\environment” and click

export.

4.3 A confirmation pop-up screen is shown.

4.4 Close the VI.

Figure 31. Step 4 Illustration

43

5. Open the VI "XMLConverterEyeTracking.vi" located in the Compact Disc

attached to this dissertation project in the folder named “Robotics Simulator

VIs”. The instructions were taken from the National Instruments Support Web

Page [30].The function of the VI is to create an .xml file which defines a

simulation environment in relation to the input files saved as .ive (described in

step 4.2) and .stl (described in step 3). Additionally, another string input is used

to make reference to the environment (ID in the block diagram). The block

diagram shown below is the VI used to generate the .xml file which can then be

loaded in the Robotics Environment Simulator Wizard.

Figure 32. LABVIEW VI used to define a simulation environment from a cad file.

44

6. - Open LABVIEW Robotics, from the tab menu select File-> New -> Project ->

Project from Wizard -> Robotics Project and then click ok.

6.1 Select the project type: Robotics Environment Simulator, click next.

Figure 33. Step 6 Illustration.

7. On the next screen, on the overview tab, select SD6 Simulation Example (it

refers to the simulated vehicle that simulates the movement of the EPW for the

project), then switch to the Environment tab and select the environment

reference name “OwnEnvironment” in this case (shown in Figure 32). An

overview of the 3D CAD model imported will be displayed on the right side of

the window (as shown in Figure 34 under the label “simulation scene”). Finally,

select the desired project name, the project folder and the simulation instance

name (name of the VI that will be generated).

Figure 34. Step 7 Illustration.

45

8. On the new screen, the project folder and the generated VI are shown, select

the VI and when the front panel is displayed, run the program.

Figure 35. Robotics Simulation Environment in LABVIEW, 3D CAD model imported

from Google Sketch Up.

46

4.4 Pupil Centre Approach

After analysing common-used algorithms in eye-tracking systems such as corneal

reflection (localization and removal), Purkinje images extraction, Kalman Filters,

neural networks, etc. It was observed that most of those studies try to use the same

methods or approximations. Due to the extensive computational cost generated when

the image is analysed recursively using several algorithms. It results in consumption

of many resources from the processor. It is rare to find an eye-tracker that can execute

and perform control actions in a low-cost computer. Additionally, no previous studies

were found regarding a similar application using eye-tracking and performing control

actions in LABVIEW. All the experimentations for this project were taken in the same

low-cost personal computer (specifications in section 3.5.2) using as few resources as

possible to perform eye-tracking and face the challenge. The image processing

algorithms used to extract the pupil centre are described below, those 12 tasks are

applied to each frame acquired from the webcam. The VI can be found in the Compact

Disc attached to this dissertation project.

The following list describes the tasks and show the corresponding input and output

image (processed image) in order to allow a full understanding of every function

involved to achieve eye-tracking based on pupil centre extraction.

� Step 1: An image is acquired in real-time using a webcam (distance=5-10 cms.)

Fig 36. Step 1

� Step 2: The colour image acquired is converted to a grayscale image (8-bit)

Fig 37. Step 2

47

� Step 3: A mask is applied to the surrounding area where the pupil centre was

detected. Note: The mask is applied until the pupil location is extracted, if no

detection exists, the complete image is analysed on the subsequent steps.

Fig. 38. Mask application

� Step 4: A filter is used to smooth the image (kernel size x=10, y=10).

Fig. 39.Illustration step 4. A) Filter parameters B) Original Image C) Processed Image.

� Step 5: A threshold is selected*.

Fig 40. Step 5. A) Look for: Dark objects B) Threshold=82 C) Processed image

48

� Step 6: Objects touching the borders are removed.

Fig 41. Border objects removal.

� Step 7: Small objects are eliminated.

Fig 42. Small objects removal.

� Step 8: Convex hull application, where convex hull of a group of points

corresponds to the smallest convex set that links all points and solidify all the

particles inside. In the following icon, the line that connects all the point is the

convex hull, highlighted in yellow for easy understanding.

49

Fig 43. Step 8. Effect of Convex hull application. A) Morphology selection B) Input

image C) output image, the hole inside the object is filled using this function.

� Step 9: Image equalization.

Fig 44. Effect of image equalization

� Step 10: Image reverse.

Fig. 45. Image reverse effect.

A B

C

50

� Step 11: Elliptical detection.

Figure 46. Ellipse detection setup and Information extracted from the elliptical object

Center X and Center Y= pupil centre position (x-y coordinates).

� Step 12: Circular edge detection.

Figure 47. Processing function “find circular edge”, settings and relevant information

of the object detected.

Center X and Center Y= pupil centre position (x-y coordinates).

51

4.4.1 Results

The procedure described above was tested in 20 images, 17 of them accurately

detected the pupil centre position = 85% accuracy, those images where taken at a

distance between 5-10 cms from 4 different users, pointing the eye gaze in different

directions (up, down, left, right, center). The pupil centre was accurately extracted by

processing the image offline and manually changing the threshold value. The reason

for doing the process offline is that depending on the lighting conditions, less or greater

contrast appear in the image, therefore the threshold needs to be adjusted to

compensate for those situations. In Chapter 5.2 an alternative strategy is suggested

in order to do all the image processing online. With the algorithm suggested, an optimal

threshold value is automatically set during calibration. During image processing it was

observed that the threshold value ranges from 20 to 90 depending on the contrast of

the image. Therefore with the automatic threshold algorithm, the program check for

the best minimum threshold value that drops the detection of both elliptical and circular

detection. The calibration process finishes when the difference in the radius and the

position in the x-y plane (coordinates) from both edge detections is +/- 5 pixels,

calibration process stops and the threshold value is assigned for post processing.

Additionally, this calibration process extracts the initial position of the pupil centre,

therefore a mask is applied on the surroundings to reduce computational cost. If the

camera is at a fixed position from the user’s eye, there is no reason to scan the

complete image, if the pupil will lie within a certain region.

52

4.5 - Eye tracking and simulation environment Integ ration

The following tasks are followed in order to integrate both VIs (Eye-Tracking and the

simulation environment).

1 Load simulation environment.

2 Run calibration VI (Outputs: Ideal Threshold value and pupil centre coordinates)

3 Initialization of simulated vehicle components (motors, sensors, camera)

4 While loop:

o Vision Acquisition

o Image Processing. - Including algorithms for spatial filtering, particle

removal, remove border objects, Elliptical-Circular Edge detection,

histogram, equalization, etc.)

o Image Mask.- defines a region of 160 x 120 pixels if the pupil is detected,

otherwise the complete image is scanned to find the pupil centre, in this

case the image size is 640 x 480

o Comparison of new pupil position in relation to initial values, if the new

coordinates exceed horizontal and vertical limits, a command is generated

to displace a simulated vehicle

5 Close software connections to sensors and stop the simulation instance.

Fig. 48. Integration of eye-tracking to control a simulated vehicle, the crosshair gives

feedback in real-time of the position detected as the pupil centre.

53

Chapter 5 – DISCUSSION

An algorithm was developed for eye tracking that combines elliptical and circular

features in an object. The corneal reflections on the pupillary zone are ignored during

image processing, resulting in a significant time reduction. Then the novel strategy

suggested to automatically find an ideal threshold, is used to allow eye-tracking in real-

time, Avoiding the need for the user to modify the parameters to achieve pupil location

extraction. A set of steps are described to import a 3D CAD model from an external

program in order to incorporate a simulation instance in LABVIEW and execute all

tasks in a single piece of software. A conducted validation indicated that the algorithm

performs well on video acquired from a low-cost web camera. The average detection

rate is calculated with the following formula: total number of frames where the pupil

centre is extracted, divided by the total number of frames.

5.1 Future work

Possible improvements related to this project can include:

o Implementation of a frontal camera. – An obstacle avoidance system could be

implemented if the environment where the EPW is processed, in order to have

a more robust, secure and reliable system.

o Sensors implementation. - A set of infrared or ultrasonic sensors could be

implemented in order to detect risky situations for the user. For example a steep

hill or stairs could be detected and provide feedback to the user via an alarm or

a pre-set recorded message voice, in order to inform the user about the

situation detected.

o Design of the wearable tracker. - Due to the limited time for the dissertation

project it was not possible to design a wearable tracker, the objective of future

work is to keep testing the algorithms with the camera at a fixed distance and

in a steady position.

o Speed Control. - Future work on this project could include the implementation

of speed control on the wheelchair in relation to the new position of the eye

gaze in comparison with the initial eye centre position.

54

CHAPTER 6 - Conclusions

The results of this study showed that after the assessment of different image

processing algorithms for Eye Tracking, and more specifically for pupil centre

extraction, the proposed algorithm combining circular edge and ellipse detection, in

addition to automatic threshold adjustment achieved an accuracy of about 91% based

on 250 frames taken from 3 different users at a distance of 5-10 centimetres between

the camera and the user. The average detection rate is calculated with the following

formula: total number of frames where the pupil centre is extracted, divided by the total

number of frames.

The implementation of infrared light as a lighting source (to illuminate the eye during

image processing) and the web camera modification where the IR filter was removed

(using a head gun). Substantially improved the pupil extraction process, by diminishing

the noise caused by surrounding light conditions and increasing contrast around the

pupil, substantially improved the accuracy of the system. Moreover, by implementing

an image mask to scan only the ROI where the eye is located, allowed execution in

real-time (image acquisition at 30 fps and sampling frequency at 60Hz) in a low-cost

computer with the advantage that it was achieved that all processes run in a single

program - LABVIEW. Finally, it was achieved to import a 3D CAD design from an

external program (Google SketchUp), so that the vehicle can be tested in different

simulated environments to prove the accuracy of the system. E.g. the vehicle can be

tested in places where the space is reduced. Finally concluding that it is possible to

implement in a near future, a Low-Cost Eye Tracking System together with an electric

wheelchair to help quadriplegic disabled and people with motion limitations.

Note: The VI programmed in LABVIEW for eye-tracking and all the SUB VIs developed

during the project can be found in the Compact Disc attached to this dissertation.

55

REFERENCES

[1] T. Yagi, Y. Kuno, K. Koga, and T. Mukai, “Drifting and Blinking Compensation in Electro-oculography (EOG) Eye-gaze Interface,” 2006 IEEE International Conference on Systems, Man and Cybernetics, pp. 3222–3226, Oct. 2006.

[2] “EYE-MOVEMENT Archives.” [Online]. Available: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=eye-movement;a6bdf544.9908. [Accessed: 19-Aug-2013].

[3] “myGaze.” [Online]. Available: http://or.mygaze.com/developer-solutions/index.html. [Accessed: 15-Aug-2013].

[4] L. R. Young and D. Sheena, “METHODS & DESIGNS Survey of eye movement recording methods,” vol. 7, no. 5, pp. 397–429, 1975.

[5] “Chronos Vision.” [Online]. Available: http://www.chronos-vision.de/en/company_news_archiv.html. [Accessed: 15-Aug-2013].

[6] “Driver drowsiness detection with eyelid related parameters by Support Vector Machine.” [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0957417408006714. [Accessed: 15-Aug-2013].

[7] “Access your computer through eye gaze.” [Online]. Available: http://www.tobii.com/en/assistive-technology/global/products/hardware/pceye/. [Accessed: 17-Aug-2013].

[8] “EyeCan-Eye operated mouse from Samsung.” [Online]. Available: http://tokyotek.com/eyecan-eye-operated-mouse-from-samsung-video/. [Accessed: 16-Aug-2013].

[9] “Deciding At First Glance.” [Online]. Available: http://www.esn-network.com/eye_movements.html. [Accessed: 17-Aug-2013].

[10] “LS460 achieves a world-first in preventative safety.” [Online]. Available: http://www.newcarnet.co.uk/Lexus_news.html?id=5787.

[11] “Website Usability Testing: Guide To The Best Professional Usability Testing Tools And Services.” [Online]. Available: http://www.masternewmedia.org/website-usability-testing-guide-to-the-best-professional-usability-testing-tools-and-services. [Accessed: 20-Aug-2013].

[12] “Assistive Technology and how is it funded.” [Online]. Available: http://www.atia.org/i4a/pages/index.cfm?pageid=3859#What_is_AT_. [Accessed: 20-Aug-2013].

[13] L. M. Camarinha-Matos and and H. Afsarmanesh, “Design of a virtual community infrastructure for elderly care,” no. May, pp. 279–284, 2002.

56

[14] “Social research takes on challenges of population ageing.” [Online]. Available: http://www.un.org/swaa2002/prkit/socialresearch.htm. [Accessed: 28-Aug-2013].

[15] R. A. Cooper, “Intelligent Control of Power Wheelchairs,” no. August, 1995.

[16] R. a. Cooper, L. M. Widman, D. K. Jones, R. N. Robertson, and J. F. Ster, “Force sensing control for electric powered wheelchairs,” IEEE Transactions on Control Systems Technology, vol. 8, no. 1, pp. 112–117, 2000.

[17] H. Seki and A. Kiso, “Disturbance road adaptive driving control of power-assisted wheelchair using fuzzy inference.,” Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference, vol. 2011, pp. 1594–9, Jan. 2011.

[18] “Battery Source Store-Wheelchair Batteries.” [Online]. Available: http://www.batterysource.com/browse.cfm/wheelchair-batteries/2,9034.html. [Accessed: 20-Aug-2013].

[19] L. Montesano, M. Díaz, S. Bhaskar, and J. Minguez, “Towards an intelligent wheelchair system for users with cerebral palsy.,” IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society, vol. 18, no. 2, pp. 193–202, Apr. 2010.

[20] R. C. Simpson and S. P. Levine, “Voice control of a powered wheelchair.,” IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society, vol. 10, no. 2, pp. 122–5, Jun. 2002.

[21] R. K. Megalingam, A. A. Thulasi, R. R. Krishna, M. K. Venkata, A. G. B. V, and T. U. Dutt, “Thought Controlled Wheelchair Using EEG Acquisition Device,” pp. 207–212, 2013.

[22] “Image Processing.” [Online]. Available: http://www.photonics.com/Article.aspx?AID=25135. [Accessed: 24-Aug-2013].

[23] E. E. Sutter and D. Tran, “The field topography of ERG components in man--I. The photopic luminance response.,” Vision research, vol. 32, no. 3, pp. 433–46, Mar. 1992.

[24] Saunders, “Dorland’s Medical Dictionary for Health Consumers,” 2007. [Online]. Available: http://medical-dictionary.thefreedictionary.com/corneal+limbus. [Accessed: 28-Aug-2013].

[25] “2013 Best Webcam Reviews and Comparisons,” 2013. [Online]. Available: http://webcam-review.toptenreviews.com/. [Accessed: 28-Aug-2013].

[26] “Logitech C910-C920 IR conversion for nightvision,” 2012. [Online]. Available: http://www.alcs.ch/logitech-c910-infrared-conversion-for-nightvision.html. [Accessed: 29-Aug-2013].

57

[27] “C920 Technical Specifications,” 2013. [Online]. Available: http://logitech-en-amr.custhelp.com/app/answers/detail/a_id/28927. [Accessed: 30-Aug-2013].

[28] “Health topics- Astigmatism.” [Online]. Available: http://www.bupa.co.uk/individuals/health-information/directory/a/astigmatism. [Accessed: 31-Aug-2013].

[29] “Convert Sketchup SKP files to DXF or STL,” 2008. [Online]. Available: http://www.guitar-list.com/download-software/convert-sketchup-skp-files-dxf-or-stl. [Accessed: 20-Aug-2013].

[30] “Using CAD files to Define a Simulated Environment (Robotics Module),” 2011. [Online]. Available: http://zone.ni.com/reference/en-XX/help/372983B-01/lvrobogsm/robo_sim_environment/. [Accessed: 31-Aug-2013].

Note: The complete VIs and Sub VIs programmed in LABVIEW for this project can be

found in the Compact Disc attached to this dissertation.