Eye-Hand coordination Stan Gielen Radboud University Nijmegen.

Eye-Hand coordination

Stan GielenRadboud University Nijmegen

Why eye-hand coordination ?

Visual feedbackguidanceExploratory behaviour

Some scientific problems

1. The dynamics of eye and hand movements are very different2. Dynamics for the eye are constant; inertia of the arm varies3. Different neuronal structures are involved how to synchronize ?4. Different frames of reference:

• retinal coordinate system for visual perception• Body-reference system for coordination of arm movements• World coordinate system for planning actions

5. Synchronisation of visual (afferent) and motor (efferent) activity due to time delays in efferent and afferent pathways

6. ……..

Questions

1. Is there a common drive for eye- and hand in aiming movements ?2. How does the CNS deal with visuo-motor interaction if the

dynamics of the eye and hand are very different ?3. How is information about self-initiated movements used in gaze

control ?4. How does the CNS deal with the fact that dynamics of the arm are

different due to variable inertia (arm posture, objects) ?5. Which coordinate system is used in visuo-motor control ?6. How does the CNS deal with synchronisation of visual (afferent)

and motor (efferent) activity due to time delays in efferent and afferent pathways ?

7. If gaze is used to guide the hand along a path (tracking), does gaze lead the hand by a constant time or a constant distance ?

Dynamics of eye and arm movements

Introduction in saccadic system

van Gisbergen, Robinson, S.Gielen. J. Neurophysiol 45, 417-442, 1981.

Position and velocity dependent force components

EEEkF )( 0

Burster cel

EEEkF )( 0

Motoneuron behavior

EEEkF )( 0

Neuronal wiring diagram

Saccadic RT

Dynamics for limb movements

+β+k(Θ-Θ0)

Agonist EMG

AntagonistEMG

Smooth pursuit : driven by retinal slip

Carpenter, Movement of the Eyes, PION, 1977

Solid : predictable target

Open: unpredictable target

Eye-Hand coordinationin aiming movements

Eye-hand coordination in aiming movements

time

posi

tion

Coordination of eye and hand movements

Gielen et al. Exp Brain Res, 56, 154-161, 1984.

eye

hand

Hand and eye positions as a function of RT

Conclusion

• When the eye and hand have to track an object, they are driven by a common drive.

• The common drive changes continuously as a function of time

Gielen et al. Exp Brain Res, 56, 154-161, 1984.

Action and perception are coupled:Action-perception cycle

Active versus Passive perception

Smooth pursuit : anticipation/prediction

Carpenter, Movement of the Eyes, PION, 1977

Solid : predictable target

Open: unpredictable target

Tracking of passively moving target and of target motion due to active head movements

Gaze in passive and active conditions

Gaze in passive conditions

Mean gain for the passive condition for 3 distances(25, 50 and 75 cm), 3 frequencies (0.5, 1 and 1.5 Hz)and 3 stimulus diameters

Frequency-dependent gain for active (closed) and passive (open) conditions

Gielen et al. Exp Brain Res (2004) 155: 211–219

Summary

Gaze control is significantly better in active conditions than in passive conditions due to anticipation and/or efference copies

Frames of reference in eye-hand coordination

Frames of reference viewer centered: relative to the eye, head, or shoulder ?

Frames of reference for pointing

Desmurget et al., J.Neurophysiol

In darkness In light environment

Potential error sources in pointing

• Error in sensory signal (visual, auditory)• Error in storing information in memory• Error in coordinate transformation from sensory to motor• Error in motor command

Constant error

Variable error

Pointing to a remembered target

Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

Subjects were standing,in a completely dark room

Targets were removed during pointing

FINGER: no frame, LED on finger

DARK: no frame, no LED on finger

FRAME: frame and LED on finger

0 1 2 3 4 5 6 7s

targetcuepointingframe

delay

DarkFinger Frame

Pointing results

1. Errors in perception small for azimuth and elevation, but large for depth direction

2. Errors in motor programming relatively small !

Gaze and pointing to remembered visual targets

Fixation is on target after target presentation in all conditions


Fixation drifts away from the subject in radial direction in the delay period.

The effect is larger in the FRAME condition.


Fixation and pointing do not overlap,

but orientation tuning is similar

Variable error reflects eye-centered fram of reference

Have a cup of coffee and relax !

Pointing to a remembered target after egomotion

gaze


DARK FINGERPointing position

FRAME



FRAME

Fixation during pointing



FRAME

Fixation at the end of step

Fixation during pointing

DARK FINGER

Average over all subjects


For discrete changes in relative target position:

Covariation between gaze and hand position:

Hand position is biased to gaze position

Conclusions

1. Errors in visual perception are small in azimuth and elevation, but large depth

2. An external frame of reference reduces pointing errors3. Errors in motor programming are relatively small; main contribution to

variable error in pointing is due to errors in visual perception.4. Updating for ego-motion is incomplete

Pointing accuracy in pathology: Parkinson Disease

Pointing in Parkinsons Disease

Parkinson Disease is primarily a sensory-motor disorder

Keijsers, Gielen et al., European Journal of Neuroscience, 21, 239–248, 2005

Reference frames for eye-hand coordination

For a retinal frame of reference pointing errors should depend on gaze position relative to target position

?

Pointing errors in retinal frame of reference !

Beurze, van Pelt en Medendorp J Neurophysiol 96: 352–362, 2006.

H, initial hand position; F, fixation point; T, target for movement. B: schematic representation of the paradigm. Horizontal position of the eyes (- - -) and the hand’s pointing direction (—). Boxes indicate location and duration of initial hand position (H, 0–2500 ms), fixation (F, 2500–4500 ms), and target (T, 3500–4500 ms) . Visual feedback about index finger position was available until start of movement.

Spatial updating operates in gaze-centered coordinates.

Medendorp et al. J Neurophysiol 93: 954–962, 2005.

• Occipital, posterior parietal, and premotor cortex are active for saccades and during pointing movements,

• retIPS is activated more strongly for saccades than for pointing. • The activation associated with pointing was significantly greater when pointing with the unseen hand

to targets ipsilateral to the hand. Although there was activation in the left retIPS when pointing to targets on the right with the left hand, the activation was significantly greater when using the right hand (mirror symmetric effect for right retIPS).

• Each hand is more effective in directing movements to targets in ipsilateral visual space.

Hypothetical flow of eye- and hand-related signals to Posterior Parietal Cortex (IPS)

A posteriorly directed stream flows to the retIPS, where relative joint angles are converted into a map with the position of possible effectors coded with respect to the eye. In retIPS, this map, with the effectors initially coded in the contralateral cortex having the weaker representation, is integrated with the map of target location.

Visual information about target location, travels along an anteriorly directed stream into retIPS, which encodes a map of target location in the contralateral visual field.

Proprioceptive information of each effector resides within the primary somatosensory cortex, in the postcentral sulcus, which somatotopically represents the contralateral body parts.

Eye-Hand coordination in tracking and tracing

Eye and hand tracking in 3D

azimuth

ele

vati

on

depth

Eye and hand in 3D

azimuth

ele

vati

on

depth azimuth

ele

vati

on

depth

tracking tracing

Interpretation of crosscorrelation : optimal overlap of eye and hand traces

T

T

xy dttytxT

R )()(2

1)(

Obviously gaze leads hand position.How to quantify the delay between saccadic eye position and hand position ?

How to quantify the delay between saccadic eye position and hand position

Tramper & Gielen The Journal of Neuroscience 31 (21): 7857-7866, 2011

Tracking

Gielen et al. Cortex 45: 340 – 355, 2009

Tracing


Lead time of gaze relative to finger is not constant !


After correction for saccadic lead time

Gaze leads the finger by a constant distance, not a constant lead time !

Δs : 2.2 – 3.7 cm


Mean correlation between hand velocity and saccade interval -0.45 (SD 0.10)*

* p<0.01

Eye fixations lead hand position by a constant displacement

Gielen et al., Cortex 45: 340 – 355, 2009

Simplified model for trajectory generation for discrete aiming movements

Simplified model for trajectory generation for discrete aiming movements

Taken from Wolpert, Kawato et al.

Time delays should be incorporated for moving targets !

Δt1

Δt2

Comparing signals at different positions in time !

Sensory prediction

Courtesy Daniel Wolpert

Courtesy Daniel Wolpert

Virtual reality

Thanks for your attention !

Eye-Hand coordination Stan Gielen Radboud University Nijmegen.

Documents

Transcript of Eye-Hand coordination Stan Gielen Radboud University Nijmegen.