Eyes Alive

Eyes AliveEyes AliveSooha Park - Lee Jeremy B. Badler - Norman I. BadlerSooha Park - Lee Jeremy B. Badler - Norman I. Badler

University of Pennsylvania - The Smith-Kettlewell Eye Research University of Pennsylvania - The Smith-Kettlewell Eye Research InstituteInstitute

Presentation Prepared By:Presentation Prepared By:Chris WidmerChris Widmer

CSE 4280CSE 4280

OutlineOutline

IntroductionIntroduction MotivationMotivation BackgroundBackground Overview of SystemOverview of System DescriptionsDescriptions ResultsResults ConclusionsConclusions

IntroductionIntroduction

Eye movement an important Eye movement an important expression techniqueexpression technique

Statistical eye movement model Statistical eye movement model based on empirical databased on empirical data

MotivationMotivation

Natural look eye movement for Natural look eye movement for animations of close-up face viewsanimations of close-up face views

Traditionally difficult to attain accurate Traditionally difficult to attain accurate eye movement in animationseye movement in animations

No proposals for saccadic eye No proposals for saccadic eye movement for easy use in movement for easy use in speaking/expressive facesspeaking/expressive faces

Recent interest in construction of Recent interest in construction of human facial modelshuman facial models

BackgroundBackground

To build a realistic face model:To build a realistic face model:– Geometry modelingGeometry modeling– Muscle behaviorMuscle behavior– Lip synchronizationLip synchronization– Text synthesisText synthesis

Research has traditionally not Research has traditionally not focused on eye movement.focused on eye movement.


Eyes are essential for non-verbal Eyes are essential for non-verbal communicationcommunication

– Regulate flow of conversationRegulate flow of conversation– Search for feedbackSearch for feedback– Express emotionExpress emotion– Influence of behaviorInfluence of behavior

New approach based on statistical data New approach based on statistical data and empirical studiesand empirical studies

SaccadesSaccades Rapid movements of both eyes from one gaze Rapid movements of both eyes from one gaze

position to another.position to another.– Only Eye Movement Executed ConsciouslyOnly Eye Movement Executed Consciously– Balance conflicting demands of speed and accuracyBalance conflicting demands of speed and accuracy

MagnitudeMagnitude – angle the eyeball rotates to – angle the eyeball rotates to change positionchange position

DirectionDirection – 2D axis of rotation, 0 degrees to – 2D axis of rotation, 0 degrees to the rightthe right

DurationDuration – Time of movement – Time of movement Inter-saccadic IntervalInter-saccadic Interval – time between – time between

saccadessaccades

SaccadesSaccades Example: Magnitude 10, 45 degreesExample: Magnitude 10, 45 degrees

– Rotating 10 degrees, right-upwardRotating 10 degrees, right-upward

Initial/Final Acceleration: 30,000 deg/secInitial/Final Acceleration: 30,000 deg/sec Peak Velocity – 400 – 600 deg/secPeak Velocity – 400 – 600 deg/sec Reaction Time: 180 – 220 msecReaction Time: 180 – 220 msec Duration and Velocity Functions of MagnitudeDuration and Velocity Functions of Magnitude

Magnitude ApproximationMagnitude Approximation D = DD = D0 + d * A + d * A D = Duration, A = Amplitude, d = increment in D = Duration, A = Amplitude, d = increment in

duration per degree (2-2.7 msec/deg), Dduration per degree (2-2.7 msec/deg), D0 = = Intercept (20-30 ms)Intercept (20-30 ms)

Often accompanied by head rotationOften accompanied by head rotation


Three Functions of GazeThree Functions of Gaze– Sending Social SignalsSending Social Signals– Open Channel to Receive InformationOpen Channel to Receive Information– Regulate Flow of ConversationRegulate Flow of Conversation

Overview of SystemOverview of System

1.1. Eye-tracking images analyzed, Eye-tracking images analyzed, statistically based model generated statistically based model generated using Matlabusing Matlab

2.2. Lip movements/Eye Blinks/Head Lip movements/Eye Blinks/Head Rotation analyzed by Rotation analyzed by alterEGOalterEGO face face motion analysis systemmotion analysis system

Overview of SystemOverview of System

Face Animation Parameter (FAP) FileFace Animation Parameter (FAP) File Eye Movement Synthesis System Eye Movement Synthesis System

(EMSS)(EMSS)– Adds eye movement data to FAP fileAdds eye movement data to FAP file

Modified from face2face’s animator Modified from face2face’s animator plug-in for 3D Studio Maxplug-in for 3D Studio Max

Analysis of DataAnalysis of Data

Eye movements recorded with eye-Eye movements recorded with eye-tracking visor (ISCAN) – monocle and two tracking visor (ISCAN) – monocle and two miniature camerasminiature cameras

One views environment from left eye One views environment from left eye perspective, other is close-up of left eyeperspective, other is close-up of left eye

Eye image recordedEye image recorded Device tracks by comparing corneal Device tracks by comparing corneal

reflection of the light source relative to the reflection of the light source relative to the location of the pupil centerlocation of the pupil center– Reflection acts as reference point while pupil Reflection acts as reference point while pupil

changes during movementchanges during movement


Pupil position found using pattern Pupil position found using pattern mappingmapping

Default threshold grey level using Default threshold grey level using Canny Edge Detection operatorCanny Edge Detection operator

Positional histograms along X and Y Positional histograms along X and Y axis calculatedaxis calculated

Two center points with maximum Two center points with maximum correlation chosencorrelation chosen


Saccade MagnitudeSaccade Magnitude

Frequency of a specific magnitude Frequency of a specific magnitude (least mean squares distribution)(least mean squares distribution)

• d = Distance traversed by pupil center

• r = radius of eyeball (1/2 of xmax

• P = % chance to occur

• A = Saccade Magnitude (Degrees)


Saccade Duration measured with 40 Saccade Duration measured with 40 deg/sec thresholddeg/sec threshold

Used to derive instantaneous velocity Used to derive instantaneous velocity curve for every saccadecurve for every saccade

Duration of each movement Duration of each movement normalized to 6 framesnormalized to 6 frames

Two classes of Gaze: Two classes of Gaze: – MutualMutual– AwayAway

Talking vs. ListeningTalking vs. Listening

Synthesis of Eye MovementSynthesis of Eye Movement

Attention Attention Monitor (Monitor (AttMonAttMon))

Parameter Parameter Generator Generator ((ParGenParGen))

Saccade Saccade Synthesizer Synthesizer ((SacSynSacSyn))

Head Rotation MonitoringHead Rotation Monitoring

Synthesis of Natural Eye Synthesis of Natural Eye MovementMovement

AttMonAttMon determines mode, changes in determines mode, changes in head rotation, gaze statehead rotation, gaze state

ParGenParGen determines saccade determines saccade magnitude, direction, duration, and magnitude, direction, duration, and instantaneous velocityinstantaneous velocity

SacSynSacSyn synthesizes and codes synthesizes and codes movements into FAP valuesmovements into FAP values


Magnitude determined by inverse of fitting Magnitude determined by inverse of fitting function shown earlier (Slide 16)function shown earlier (Slide 16)

Mapping guarantees same probability Mapping guarantees same probability distribution as empirical datadistribution as empirical data

Direction determined by head rotation Direction determined by head rotation (threshold), and distribution table(threshold), and distribution table– Uniform Distribution, 0 to 100Uniform Distribution, 0 to 100– 8 non-uniform intervals assigned to respective 8 non-uniform intervals assigned to respective

directionsdirections


Duration determined by first equation, Duration determined by first equation, respective values for d and Drespective values for d and D0

Velocity determined by using fitted Velocity determined by using fitted instantaneous velocity curveinstantaneous velocity curve

SacSyn system calculates sequence of SacSyn system calculates sequence of coordinates for sys centerscoordinates for sys centers

Translated into FAP values, rendered in 3D Translated into FAP values, rendered in 3D Studio MAXStudio MAX

Face2face animation plug-in to render Face2face animation plug-in to render animations with correct parametersanimations with correct parameters

ResultsResults

3 Different Methods Tested3 Different Methods Tested Type 1 -> No Saccadic MovementsType 1 -> No Saccadic Movements Type 2 -> Random Eye MovementType 2 -> Random Eye Movement Type 3 -> Sampled from Estimated Type 3 -> Sampled from Estimated

Distributions (synchronized with Distributions (synchronized with head movements)head movements)

Tests were subjectiveTests were subjective

ResultsResults

Q1: Did the character on the screen Q1: Did the character on the screen appear interested in (5) or indifferent (1) appear interested in (5) or indifferent (1) to you?to you?

Q2: Did the character appear engaged (5) Q2: Did the character appear engaged (5) or (1) distracted during the conversation?or (1) distracted during the conversation?

Q3: Did the personality of the character Q3: Did the personality of the character look friendly (5) or not (1)?look friendly (5) or not (1)?

Q4: Did the face of the character look Q4: Did the face of the character look lively (5) or deadpan (1)?lively (5) or deadpan (1)?

Q5: In general, how would you describe Q5: In general, how would you describe the character?the character?

ResultsResults

ConclusionsConclusions

Saccade Model for Talking and Listening Saccade Model for Talking and Listening ModesModes

3 Different Eye Movements: Stationary, 3 Different Eye Movements: Stationary, Random, Model-basedRandom, Model-based

Model-based scored significantly higherModel-based scored significantly higher Eye Tracking Data recorded from a subjectEye Tracking Data recorded from a subject

– New recorded data for every characterNew recorded data for every character

This model allows any number of unique eye This model allows any number of unique eye movement sequences.movement sequences.

Drawbacks and Drawbacks and ImprovementsImprovements

Aliasing with small movementsAliasing with small movements Sensing of eye movement vs. head Sensing of eye movement vs. head

movement during data gatheringmovement during data gathering

Future EnhancementsFuture Enhancements– Eye/Eyelid dataEye/Eyelid data– More model gaze patternsMore model gaze patterns– More subjects for dataMore subjects for data– Scan-path model for close-up imagesScan-path model for close-up images

DevelopmentsDevelopments

J. Badler, Director, J. Badler, Director, Center for Human Center for Human Modeling and SimulationModeling and Simulation– Digital Human Modeling/BehaviorDigital Human Modeling/Behavior– ““Jack” SoftwareJack” Software– Simulation of workflow using virtual Simulation of workflow using virtual

peoplepeople

ReferencesReferences

Badler, Jeremy B., Badler, Norman I, Lee, Badler, Jeremy B., Badler, Norman I, Lee, Sooha Park, “Eyes AliveSooha Park, “Eyes Alive

http://www.cis.upenn.edu/~sooha/http://www.cis.upenn.edu/~sooha/home.htmlhome.html

http://www.cis.upenn.edu/~badler/http://www.cis.upenn.edu/~badler/ http://cg.cis.upenn.edu/hms/research.htmlhttp://cg.cis.upenn.edu/hms/research.html

Eyes Alive

Documents

Transcript of Eyes Alive