Seminar 8th Eye Blink

8/9/2019 Seminar 8th Eye Blink

1/20

Blink Detection and Eye Tracking for Eye Localization

1. INTRODUCTION

A method of using frame differencing coupled with optical flow computation for

eye blink detection is proposed. Frame differencing allows quick determination of

possible motion regions. If they are detected, optical flow is computed within these

regions. The direction and magnitude of the flow field are then used to determine

whether a blink has occurred. The eyes are then tracked using the Kanade Lucas

Tomasi (KLT) tracker. We obtained a success rate of 97.0% in blink detection usingthe proposed method, and localised the eyes automatically at an average rate of 22

frames per second.

Eye localization is useful in applications such as face normalization, gaze-based

human-computer interfaces, and security systems using the human iris for identification.

Blink detection also allows measurement of blinking rate and the rate of closures of

eyes. These cues are often used as indicators of driver awareness while operating

vehicles [7], [10].

Methods of blink detection:

A few methods have been proposed for blink detection. In

Yano et al. [11], frame differencing is used for blink

detection. Al-Qayedi and Clark [1] track features about the

eyes and infer blinks through detection of changes in the

eye shape. Smith et al. [8] try to differentiate between

occlusion of the eyes (due to rotation of the head) and

blinking. The subjects sclera is detected using intensity

information to indicate whether the eyes are open or closed


2/20

(i.e., a blink is occurring). Black et al. [4] detect blink using

optical flow but the system restricts motion of the subject

and needs near frontal views in order to be effective. The

reported 65% success rate in detecting blinks seems to be

too low for applications such as communicating via blinks.

Grauman, et al. [5] report a success rate of 96.5% in almost

real-time. Here, frame differencing is initially performed to

obtain motion regions. Then a pre-trained open-eye

template is selected for the computation of the Mahalanobis

distance measure which is used to retain the best pair of

blobs as candidates for eyes. The eyes are tracked and

correlation scores between the actual eye and the

corresponding closed-eye template are used to detect

blinks. However, it requires offline training for different

depths from the camera for the computation of the distance.

In addition, the system requires initialization in which a

stored template for a particular depth is chosen. Once the

template is chosen and the system is in operation, the

subject will be restricted to be at the specified distance.

Another disadvantage of the system is that changing camera

positions requires the whole system to be retrained.

Furthermore, the same system may not be as effective if it

were used on people of different races with disparate eye

sizes and distance between the eyes.

The algorithm proposed here reduces computational costs

by first performing frame differencing to localize motion

regions.

However, the detected regions may arise due

various causes such as eye blinks, eyeball motion, head


3/20

motion and possibly background motion. To differentiate

between blinks and other types of motions, we use optical

flow and the fact that during an involuntary blink, the eyes

are first covered and then uncovered within a window of a

few consecutive few frames (typically 4-5). Our goal is to

create a system which allows

(i) the subject relative freedom of motion

(as long as the eye sizes are between 500 and

5000 pixels and there are no drastic movements in

consecutive frames)

(ii) the operator the flexibility to place

the camera at any suitable location and yet achieve a high

success rate.

2. PROPOSED ALGORITHM

Blink detection allows localization of the eyes, by using

frame differencing followed by optical flow. Subsequently,

the localized eyes are tracked by the Kanade-Tomasi-Lucas

(KLT) [6, 9, 2] method. These are described below.

2.1 Blink Detection

The steps in the blink detection algorithm are describedbelow:

(i) Obtain locations of possible motion using frame

differencing.


4/20

(ii) Suitably threshold the motion regions and obtain blobs

using morphological operations and connected components.

(iii) Remove unsuitable blobs i.e. blobs which are either too

big or too small or have incorrect width to height ratios to

be considered as eyes.

(iv) If no possible candidate blobs are found, repeat (i) to

(iii) on subsequent frames until at least 2 suitable blobs are

found, and mark their positions.

(v) Compute optical flow field in the blob regions.

(vi) Ascertain dominant direction of motion in the blobs. If

the dominant motion is downwards in a pair of blobs, their

positions are noted. These would represent eye closure

during a blink. If the motion is not downwards then steps (i)

to (vi) are repeated. The remaining steps are used for

detecting subsequent eye opening phase of blink, followed

by bounding the eye regions.

(vii) Repeat steps (i) to (iii).

(viii) Discard blobs that are not situated near the location of

the blobs found with downward motion.

(ix) Compute optical flow to ascertain if the dominant

motion is upwards if there are at least 2 blobs remaining,

Otherwise repeat from step (vii) on a new frame.


5/20

(x) If the dominant motion is upwards, then classify the

frames beginning from the frame where downward motion

was detected to the frame where upward motion was

detected as blink frames. If after the downward motion, no

upward motion is detected within the next 3 frames, no

blink is deemed to have occurred (this is because

involuntary blinks in general do not last more than 5

frames) and the process of blink detection starts from step

(i) on new frames.

(xi) The bounding boxes of the blobs where blink is deemed

to have occurred are taken as eye locations.

The computation of optical flow allows for the

differentiation between vertical eyelid movements during

blinks and movement of the eyeball (which is

predominantly horizontal) and horizontal head movements

(Figure 2). Vertical head movements result in blobs that

can be eliminated based on size, ratio and the average

magnitude of the velocity vectors (which is much lower in

the case of head motion. See Figure 3).

2.2Tracking of the Eyes

After having located the eyes, we track them using the KLT

tracker [9], [6], [2]. The tracker is initialized on the frame

immediately after the blink is detected. We found that

twenty feature points gave the best results for tracking


6/20


7/20

accuracy. These twenty feature points are selected in the

area of the eyes where there are edges and then they are

tracked in subsequent frames.

It was found that when there is drastic motion in the eyes

(which can be caused by a blink or motion of the eye ball),

several feature points are lost. If this number is greater than

20% of the total feature points, then the accuracy of the

tracker would be compromised; hence it is disabled and re-

initialized from a newly detected blink frame.

However,

occasionally less than 20% of the feature points were

reported as lost even during a blink. Thus, additional

checks such as height to width ratio of the tracked region

are used to ensure that the tracker will indeed be disabled if

necessary. Reinitialization at every blink enables the tracker

to track the eye accurately even in long sequences.

an eye localizerand face matcher:

Ouranalysis of systems composed of an eye localizerand

face matcher in terms of performance and computational

complexityrevealed that:

1. Worst-case eye localization errors have a big impact

on face matching performance, making eye localizers

that, at identical eye localizer accuracy, have a long-

tail distribution of errors perform worse to ones that

have compactly distributed errors.


8/20

2. Given an eye localizer performance better than our

Priors method and in respect to the previous obser-

vation, eye localization can boost the performance of

naive face matching approaches such as PCA.

3. Given a suitable eye localizer (such as known in liter-

ature), eye localization allows for more ecient face

matching without degrading perfomance. This e-

ciency gain will improve when the number of images

to match increases regardless of the complexity of the

eye localizer.


9/20

3.3 Overview of the visual attention model

The model used in this work [32] employs an algorithm

that assigns high VA scores to pixels where

neighbouring pixel configurations do not match

identical positional arrangements in other randomly

selected neighbourhoods in the image.

This means, for example, that high scores will be associated with

anomalous objects, or edges and boundaries, if those

features do not predominate in the image. A flowchart

describing this process is given in Fig


10/20

Fig Visual attention model.

For display purposes the VA scores for each pixel are

displayed as a map using a continuous spectrum of false

colours with the scores being marked with a distinctive color of gray level.


11/20

2.3 Speed Considerations

To speed up localization and tracking of the eyes, the

parameters for optical flow computations and KLT are

tuned. The time taken for optical flow is the major

component in the time needed for the detection of a blink.

Considerable time is used in the gradient descent iterations

for accurate flow computations. By limiting these to 10

iterations, flow computation required an average of 0.8 sc

per frame; assuming that an average blink lasts 4 frames,

optical flow computations needs 3.2 sc to detect a blink.

For the 13 sequences we tested, the accuracy of blink

detection was unchanged due to this speed modification. As

for increasing the speed of the tracker, the number of points

tracked around the eye region was set to 10. However, the

limitation is that the eyes are not localized as perfectly as

with 20 feature points. The optimal window size about each

point was found to be 11 x 11 in terms of speed and

accuracy.

In our simulations we considered 13 stored video sequences

of 700 frames (768 x 576 PAL frames at 25 fps), obtained

from different people, with each sequence containing an

average of 5 blinks. On a 1.6 GHz Pentium IV computerwe obtained the following average times for each sequence

as below:


12/20

1. Optical flow: 20 secs.

2. Tracking of the eyes: 10 secs.

3. Reading the images from disk: 13 secs.

We postulate if the system is run online, then by avoiding

the disc accesses, a processing rate of about 22 fps can be

obtained.

3. RESULTS ANDDISCUSSION

Figure 1 shows a typical result obtained with optical flow

computation at the start and end of a blink. In our

experiments, we found that optical flow reliably

distinguished between motions such as blinks, sideways

motion of the eyeballs and the head movements. During

activities such as reading and playing games on a computer

monitor, the optical flow vectors arising from vertical head

movements were found to be typically smaller in

magnitude, compared to flow vectors arising from blinks. In

Figures 2 and 3, we see that performing only frame-

differencing can lead to the wrong regions being identified

as eyes. Using only frame differencing fails to distinguish

between motion of the eyeball and a blink as seen in Figure

3. Thus optical flow is necessary to ensure the accuracy of

the localization of the eyes. By comparing the optical flow

results in Figures 1 and 3, it is clear that the magnitude of

the velocity vectors in the case of the blink is much larger


13/20

than in the case of vertical head movement.

( Graph for visualizing blink length/frames )

It was found that the KLT tracker performs well and is able

to track the eyes in the interval between blinks. In our

experiments, we found that for eyes of size about 30x50,

whenever there was a movement of about 9 pixels or more

in the subjects eyeball, the tracker was disabled. This is

undesirable as the tracker should only be disabled by the


14/20

occurrence of a blink. To avoid this, we use the fact that the

dominant direction of the optical flow for eye-ball

movements is horizontal.

If the direction of the flow

vectors is sideways in the motion regions, then eye-ball

movement is deemed to have taken place and the tracker is

reinitialized in the next frame using eye regions from the

previous frame. At every blink, to prevent drifting, the

tracker is reinitialized.

Figure 4 shows the eyes being

reliably tracked in the interval between two blinks.

The algorithm was tested on 13 video sequences of different

people, each of about 700 frames (resolution of the frames

being 768 x 576, captured at 25 fps). Of the 65 blinks in the

sequences two were missed and there were no false

detections, giving an accuracy rate of 97%.


15/20

Figure 1: Results of frame differencing (a,c) and opticalflow computation (b,d). Motion regions obtained by frame

differencing are bounded by black boxes. In (b) majority of

the optical flow vectors from the first 2 frames of a blink,

are pointing downwards. In (d) majority of flow vectors are

pointing upwards in the frames where the eye first reopens.


16/20

Figure 2: (a,b) Motion regions (rectangular regions)

resulting from frame differencing and (c) optical flow

computations when there is horizontal motion of the head

as well as motion of the eye ball. Majority of the optical

flow vectors are pointing sideways.


17/20

Figure 3: (a) Motion regions (rectangles) from frame

differencing and (b) optical flow vectors when there is

vertical head movement. The majority of the flow vectors

are pointing upwards and their magnitude is much smaller

than that seen in Figure 1 (for a blink).

Applications:

Once blinking is detected, measures like blinking rate,

number of eye closures and eye positions are easily

obtained. Possible areas of application for such a system

include gaze-based human-computer interfaces and security

systems.

ACCURACY:


18/20

accuracy rate of 97%

4. CONCLUSIONS

In this paper we have proposed an accurate and fast method

for locating and tracking the eyes of a computer user

situated in front of the monitor. By computing optical flow

and using both the magnitude and direction of the flow

vectors,

we can differentiate blinking from the other

motions. The located eyes are tracked by the KLT tracker.

Based on testing of 13 sequences from different individuals,

a blink detection accuracy of 97% was obtained. We

estimate that the algorithm can run at approximately 22 fps

with accurate tracking.

Once blinking is detected, measures like blinking rate,

number of eye closures and eye positions are easily

obtained. Possible areas of application for such a system

include gaze-based human-computer interfaces and security

systems.

Future work involves development of a faster and robust

optical flow algorithm for this application to reduce the

amount of time required for blink detection. A limitation ofthe proposed scheme is that a blink is necessary to detect

the eyes. However this should not be a problem in practi


19/20

REFERENCES

[1] A. M. Al-Qayedi. and A. F. Clark, Constant-rate eye

tracking and animation for model-based-coded video,Proc. ICASSP2000, 2000.

[2] S. Birchfield, Derivation of Kanade-Lucas-Tomasi

Tracking Equation, unpublished, May 1996. Available on

the website: http://vision.stanford.edu/~birch

[3] M. J. Black and P.Anandan, The robust estimation of

multiple smooth motions: parametric and piecewise

smooth flow fields, Computer Vision and Image

Understanding, pp 75-104, January 1996.

[4] M.J. Black, D.J. Fleet, Y. Yacoob, A framework for

modeling appearance change in image sequences, Proc.

Intl. Conf. Computer Vision, 1998.

[5] K. Grauman, M. Betke, J. Gips and G. Bradski,

Communication via Eye Blinks - Detection and Duration

Analysis in Real Time, Proc CVPR 2001, 2001.

[6] B. D. Lucas and T. Kanade, An iterative image

registration technique with an application to stereo vision,

International Joint Conference on Artificial Intelligence,

1981.

[7] T. Nakano, K Sugiyama, M. Mizuno and S. Yamamoto,

Blink measurement by image processing and application to

warning of drivers drowsiness in automobiles, IEEE

Intelligent Vehicles, 1998.


20/20

[8] P.Smith, M.Shah, N. daVitoria, Monitoring head-eye

motion for driver alertness with one camera, Pattern

Recognition, 2000.

[9] C. Tomasi and T. Kanade, Detection and Tracking of

Point Features, Carnegie Mellon University Technical

Report CMU-CS-91-132, April 1991.

[10] H. Ueno, M. Kaneda and M. Tsukino, Development

of Drowsiness Detection System, Proc. Conf. Vehicle

Navigation & Information Systems Conference, 1994.

[11] K.Yano, K.Ishihara, M. Makikawa, H. Kusuoka,

Detection of eye blinking from video camera with dynamic

ROI fixation, IEEE Trans. SMC, Vol 6, pp 335-339,

1999.

Seminar 8th Eye Blink

Documents

Transcript of Seminar 8th Eye Blink