Control of Attention and Gaze in Natural Environments

Control of Attention and Gaze in Natural Environments1

Selecting information from visual scenesWhat controls the selection process?2What happens when were in a visual scene like this? Natural scenes contain much more information than we can perceive in a brief exposure. If we view this scene for a sec or two, well move our gaze around the image, perhaps looking at the bicycle in the center, or large objects like the building. This process of selecting particular information in the scene isnt random - but we really dont know what determines where we look & what we attend to.Humans must select a limited subset of the available information in the environment. Fundamental Constraints Acuity is spatially restricted.Attention is limited. Visual Working Memory is limited.Only a limited amount of information can be retained.What controls these processes?3So when viewing scenes like this, we must deal with fundamental constraints>. Humans select a limited amt of the info from the vis env, and can retain only a limited amt. What controls these processes. 4Now this is not a question we usually ask. Typically we ask slightly different questions - heres an example. See a sequence of two brief images of simple shapes - one object is changed in the second view.Your job is to identify the changed item.567Anyone see the one that changed? If you happened to be looking at the right spot, your may have seen it change - or you might have looked at a couple of the objects, but then forgotten what they were like between the two presentations - When people do expts like this they find that you can remember about 4 items.Gives us a visceral sense of limitations

Image properties eg contrast, edges, chromatic saliency can account for some fixations when viewing images of scenes.Saliency - bottom-up8One approach to the problem is to try to predict where you look from an examination of the properties of the image. (Exogenous atten) gaze typically goes wit attn Limitations of SaliencyImportant information may not be salient eg Stop signs in a cluttered environment.

Salient information may not be important - eg retinal image transients from eye/body movements.

Doesnt account for many observed fixations, especially in natural behavior (eg Land etc). 9However, there are important ways in which saliency models arent adequate to explain distribution of gaze in a scene.Natural vision is not the same as viewing pictures.

Behavioral goals determine what information is needed.

Task structure (often) allows interpretation of role of fixations.Need to Study Natural Behavior10Inclined to think of vision as viewing a picture, but more often were acting in the environment - also, if viewing a 2D image, dont really know what obs is doing - maybe remembering objects, maybe judging image quality - dont really know - if tasl requiring overt actions, have a good idea of what the obs is doing form moment to moment Need for action means diff info is requiredNot only is stim different (2 vs 3 D, fov etc) the info you need is different

Viewing pictures of scenes is different from acting within scenes.Top-down factors11Other problem with trying to explain fix patterns or dsn of attn by looking at fixations of images is that real vision really is differnet form looking at an image of a scene. If youre in a scene you need different kinds of info from when youre looking at an image.For example.. Can think of natural vision as being composed of a set of mini-tasks like this, and gaze needs to be doled out in the service of each of these tasks.Whwen looking at an image, not clear what obs is doing - recog, mem?

Foot placementObstacle avoidanceHeadingViewing pictures of scenes is different from acting within scenes.Top-down factors12Other problem with trying to explain fix patterns or dsn of attn by looking at fixations of images is that real vision really is differnet form looking at an image of a scene. If youre in a scene you need different kinds of info from when youre looking at an image.For example.. Can think of natural vision as being composed of a set of mini-tasks like this, and gaze needs to be doled out in the service of each of these tasks.Whwen looking at an image, not clear what obs is doing - recog, mem?To what extent is the selection of information from scenes determined by cognitive goals (ie top-down) and how much by the stimulus itself (ie salient regions - bottom-up effects)?

1314Modeling Top Down Control Virtual Humanoid has a small library of simple visual behaviors:Sidewalk FollowingPicking Up BlocksAvoiding Obstacles

Each behavior uses a limited, task-relevant selection of visual information from scene.This is computationally efficient.Walter the Virtual HumanoidSprague & Ballard (2003)

14COULD A PURELY TD SYS WORK? This idea is behind the work of Sprague & Ballard, who developed a model of gaze behavior in a walking context. This is Walter, a virtual agent, Walters task is to walk through this virtual env. Walter uses vision to do 3 things.The agent has a small library of behavioral routines that need visual input. Through reinforcement learning the humanoid learns the appropriate policy by which to schedule extraction of visual information In this model a top down scheduler to acquire visual information is adequate for obstacle avoidance and Walters other tasks

Walter learns where/when to direct gaze using reinforcementlearning algorithm.Walters sequence of fixationsobstaclessidewalklitter15Model suggests that such a system is feasible - subject has a set of sub-tasks to perform, and gaze reflects performance of sub-tasks.

Walter the Virtual HumanoidSprague & Ballard (VSS 2004)

What about unexpected events?

16However what Walter would be able to handle is an unexpected salient event, such as appearance of another pedestrian in the field of viewWalter would be in trouble because he doesnt have looking for other pedestrians in his behavioral repertoireDynamic Environments

17However what Walter would be able to handle is an unexpected salient event, such as appearance of another pedestrian in the field of viewWalter would be in trouble because he doesnt have looking for other pedestrians in his behavioral repertoireComputational load

Unexpected eventsBottom-up ExpensiveCan handle unexpected salient eventsTop-down EfficientHow to deal with unexpected events?18Top down systems are more efficient because they select limited, task-specific inf from the image, but will miss things not on the agenda. Bottom up systems that do a bunch of pre-processing of the image can catch a wider variety of information, but are comp expensive.How would a top down system deal with unexpected events? Through learning or frequent checking?

Driving Simulator

19Gaze distribution depends on tasks

Time fixatingIntersection.Follow

Obey TrafficRules2021The Problem Any selective perceptual system must choose the right visual computations, and when to carry them out.How do we deal with the unpredictability of the natural world?Answer - its not all that unpredictable and were really good at learning it.21Answer - its not all that unpredictble and were really good at learnig it.So this is the essential problem for top-down systems - How do you know what to look for, and when to look for it?

This tight link between vision and task demands brings up the problem of scheduling behaviors. The visual system has limited capacity and computational ability.How does the visual system manage between the current task goals and dealing with new stimuli that may change task demands?. How does this selection occur? Learning, Frequent checkingHuman Gaze Distribution when WalkingExperimental Question: How sensitive are subjects to unexpected salient events?

General Design: Subjects walked along a footpath in a virtual environment while avoiding pedestrians. Do subjects detect unexpected potential collisions?

22To examine these tradeoffs we designed a walking experiment in virtual reality in which we could manipulate the bottom up signal, What happens if ped suddenly starts to come at you - looming stim.Virtual Walking EnvironmentVirtual Research V8 Head Mounted Display with 3rd Tech HiBall Wide Area motion trackerV8 optics with ASL501 Video Based Eye Tracker (Left) and ASL 210 Limbus Tracker (Right)D&c emily

Video Based TrackerLimbus Tracker23Our lab has several systems integrated to allow such a virtual reality experiment. We have a head mounted display with two eyetrackers installed. In the picture on the right hand side you can see the video based tracker for POG recording which is complemented by an limbus tracker used for saccade contingent updates. To allow the subjects to walk a sufficient length - a wide area motion tracking system is used to update the view inside the display while allowing the subject to walk the ~27 meter perimeter of rectangular path in the lab.

Virtual EnvironmentBirds Eye view of the virtual walking environment.

Monument

24Bird-eye view of the foot path that the subjects walked 6 subjects each performed 6 trials of walking. 3 in the no-following onditon and 3 in the following. Each trial consisted of walking around six times about 3-4minutes and

Short clip of normal speed1 - Normal Walking: Avoid the pedestrians while walking at a normal pace and staying on the sidewalk.

2 - Added Task: Identical to condition 1. However, the additional instruction of following a yellow pedestrian was given

Normal walkingFollow leaderExperimental Protocol25Side-by-side pictures of the two conditions (explanation of them) 3 blocks of 6 circuits of each.Pedestrians paths

Colliding pedestrian pathWhat Happens to Gaze in Response to an Unexpected Salient Event?The Unexpected Event: Pedestrians on a non-colliding path changed onto a collision course for 1 second (10% frequency). Change occurs during a saccade.Does a potential collision evoke a fixation?26Pedestrian must be 3-5 meters away and the angular delta could be no greater than 30degs.Contingent on saccade.

Fixation on Collider27Purple ped turn corner fixate ped look to path maintain fix during collision period and as it passes

No Fixation During Collider Period28In this clip a purple pedestrian appears in the visual field shortly after which the pedestrian starts on a collision path. The subject does not fixate the collider pedestrian during its collision course.

Probability of Fixation During Collision PeriodPedestrians paths

Colliding pedestrian pathMore fixations on colliders in normal walking.

No effect in Leader condition

Controls CollidersNormal WalkingFollow Leader29So collision event does seem to attract gaze, but only to a limited extent, and not if you have the added task of following a leader..Small increase in probability of fixating the collider.

Failure of collider to attract attention with an added task (following) suggests that detections result from top-down monitoring.Why are colliders fixated?30

Detecting a Collider Changes Fixation StrategyLonger fixation on pedestrians following a detection of a collider MissHitTime fixating normal pedestrians following detection of a colliderNormal Walking

Follow Leader

31TD systems rely on estimating likelihood of environmental events, so detection of an unlikely or signif event like a potential collision might lead subjects to spend more time monitoring peds.This indicates that subjects can quickly modify their fixation strategy in response to information that indicates a need to change policy.To make a top-down system work, Subjects need to learn statistics of environmental events and distribute gaze/attention based on these expectations.Subjects rely on active search to detect potentially hazardous events like collisions, rather than reacting to bottom-up, looming signals.32Possible reservations

Perhaps looming robots not similar enough to real pedestrians to evoke a bottom-up response.33Walking -Real WorldExperimental question:

Do subjects learn to deploy gaze in response to the probability of environmental events?

General design: Subjects walked on an oval path and avoided pedestrians

34To examine these tradeoffs we designed a walking experiment in virtual reality in which we could manipulate the general task demands as well as a salient bottom up signal, used to probe the questions we have framed. Explain block heads

Experimental Setup

System components: Head mounted optics (76g), Color scene camera, Modified DVCR recorder, Eye Vision Software, PC Pentium 4, 2.8GHz processor A subject wearing the ASL Mobile Eye 35Occasionally some pedestrians veered on a collision course with the subject (for approx. 1 sec)

3 types of pedestrians:

Trial 1: Rogue pedestrian - always collides Safe pedestrian - never collides Unpredictable pedestrian - collides 50% of time

Trail 2: Rogue Safe Safe Rogue Unpredictable - remains same

Experimental Design (ctd)36

Fixation on Collider37Effect of Collision ProbabilityProbability of fixating increased with higher collision probability.

38

Detecting Collisions: pro-active or reactive?Probability of fixating risky pedestrian similar, whether or not he/she actually collides on that trial.

39Note this may seem obvious, but in contrast, lot of work trying to predict fix locs by analyzing properties of image.Not clear what role of saliency might be in normal visionBody motion generates image motion over whole retina

Learning to Adjust GazeChanges in fixation behavior fairly fast, happen over 4-5 encounters (Fixations on Rogue get longer, on Safe shorter)

40

Shorter Latencies for Rogue FixationsRogues are fixated earlier after they appear in the field of view. This change is also rapid.

41

Effect of Behavioral RelevanceFixations on all pedestrians go down when pedestrians STOP instead of COLLIDING.STOPPING and COLLIDING should have comparable salience. Note the the Safe pedestrians behave identically in both conditions - only the Rogue changes behavior.

42Fixation probability increases with probability of a collision.Fixation probability similar whether or not the pedestrian collides on that encounter.Changes in fixation behavior fairly rapid (fixations on Rogue get longer, and earlier, and on Safe shorter, and later)43Our Experiment:

Allocation of gaze when driving.

Effect of task on gaze allocation. Does task affect ability to detect unexpected events?

Drive along street with other cars and pedestrians. 2 instructions - drive normally or follow a lead car.

Measure fixation patterns in the two conditions.

44A competing task of following a leader diminished fixations on colliders and this is consistent with a top down strategy (and reprioritizing resources).45Note this may seem obvious, but in contrast, lot of work trying to predict fix locs by analyzing properties of image.Not clear what role of saliency might be in normal visionBody motion generates image motion over whole retina

46

Reward weights estimated from human behavior using InverseReinforcement Learning - Rothkopf 2008.Human path Avatar path 47Subjects must learn the probabilistic structure of theworld and allocate gaze accordingly. That is, gaze control is model-based.

Subjects behave very similarly despite unconstrained environment and absence of instructions.

Control of gaze is proactive, not reactive, and thus is model based.

Anticipatory use of gaze is probably necessary for much visually guided behavior.Conclusions48Behaviors Compete for Gaze/ Attentional ResourcesThe probability of fixation is lower for both Safe and Rogue pedestrians in both the Leader conditions than in the baseline condition .Note that all pedestrians are allocated fewer fixations, even the Safe ones.

49Competes for gaze resources, and we are inferring attentional resources

ConclusionsData consistent with task-driven sampling of visual informationrather than bottom up capture of attention- No effect of increased salience of collision event. - Colliders fail to attract gaze in the leader condition, suggesting the extra task interferes with detection.Observers rapidly learn to deploy visual attention based on environmental probabilities.

Such learning is necessary in order to deploy gaze and attention effectively.

50Competing taskCertain stimuli thought to capture attention bottom-up (eg Theeuwes et al, 2001 etc )

Looming stimuli seem like good candidates for bottom-upattentional capture (Regan & Gray, 200; Franceroni & Simons,2003).51. All have the intuition that attention is attracted by certain stimuli - Eg something about to hit you.extensive literature on what does and doesnt capture attention exogenously - considerable debate.

No LeaderNormal Walking

Follow LeaderGreater saliency of the unexpected event does not increase fixations. No effect of increased collider speed.52To get more evidence on thi isue we increased the saliency of the collisding ped by incr speed at same time as ped turns onto a collison course.

Other evidence for detection of colliders?Do subjects slow down during collider period? Subjects slow down, but only when they fixate collider. Implies fixation measures detection.Slowing is greater if not previously fixated. Consistent with peripheral monitoring of previously fixated pedestrians.53ConclusionsSubjects learn the probabilities of events in the environment and distribute gaze accordinglyThe findings from the Leader manipulation support the claim that different tasks compete for attention54Effect of Context

Probability of fixating Safe pedestrian higher in a context of a riskier environment5556SummaryDirect comparison between real and virtual collisions is difficult, but colliders are still not reliably fixated.Subjects appear to be sensitive to several parameters of the environment:ExperienceExperience with the Rogue pedestrian elevated fixation probabilities of the Safe pedestrian to 70% (50% wto. exp.)Experience with the Safe lead to 80% fixation probability of the Rogue (89% wto. exp.)Experience of Safe carries less weight than the experience of Rogue

57Presumably, withsuch a highly salient stimulus, one would expect a high detection rate of these colliders.Our preliminary results show only a marginal increase in fixations on colliders in realenvironment (0-20% depending on the condition) when compared to those from theexperiment described in Chapter 2. This result would favor active search as source ofinformation (colliders missed if they dont coincide with an active search episode), ratherthan bottom-up interpretation.If we compare fixations on Risky (goes from 62-70%) to colliders in virtual (40-60%) In this experiment there are many more collisions, so theres the overall context effect. Fixations on the Safe are higher with collisions, than with the stops

Time fixatingIntersection.

Follow the car.orFollow the car and obeytraffic rules.CarRoadsideRoadIntersectionShinoda et al. (2001)Detection of signs at intersection results from frequent looks.5821What do we know? Previous work on dsn of attn in natural environments:IntersectionP = 1.0Mid-blockP = 0.3Greater probability of detection in probable locationsSuggests Ss learn where to attend/look.

How well do human subjects detect unexpected events? Shinoda et al. (2001)Detection of briefly presented Stop signs. 59What do Humans Do? Shinoda et al. (2001) found better detection of unexpected stop signs in a virtual driving task.

60To try and answer this question its worth looking at human behavioral data.Shinoda et al subjects in a driving simulation - subjects strategically deployed fixationsAt key moments during the based on learningWhat are the capabilities and limitations of a top-down schedulerWe would like to examine a more demanding situation and does shinodas result generalize?A cartoon of a stop sign in the intersection, and in the middle of the block.

61

62

Chart40.390.040.040.50.030.030.70.0550.055

Safe ped./ Safe environmentSafe ped./ No prior experienceSafe ped./ Conflicting experienceSafe pedestrianProbability of fixationFixations of Safe pedestrian in different contexts

Condition 1Condition 1 (Colliders+ regular leader)Trial #1 (No Leader trial)100% coll50% collnever collpercent of fixated colliders89730.90.5percent of fixated non-colliders65500.70.8Condition 1 (Colliders+ regular leader)Trial #2 (No Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders6780percent of fixated non-colliders706273650.50.050.03467620.630.70.390.50.70.05Condition 1 (Colliders+ regular leader)Trial #3 (Leader trial)100% coll50% collnever coll7063.51.54.24264068710.040.030.05percent of fixated colliders73501.06066017183percent of fixated non-colliders30420.7020.50.01060.030.0410.90.630.02Condition 1 (Colliders+ regular leader)Trial #4 (Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders5868percent of fixated non-colliders553559298950734254458070685556.53784.56070.548.50.570.370.850.60.70.4810677086545435256073477808302.584.5102.56.50.590.670.85142134122971021046774871.4433756734.61880215352.59807621145.77350269191.4433756733.75277674970.320.680.0150.0460.0260.0570.0150.0380.630.050.0320.7020.6660.040.030.055

Condition 1

No experienceConflicting experienceSafe RogueProbability of fixation

Condition 20.630.5

Unpredictable pedestrians when not colliding Safe pedestriansProbability of fixationFixations on non-colliding pedestrians

Condition 30.7020.9

Upredictable pedestrians Rogue pedestriansProbability of fixationFixations on colliding pedestrians

Condition 40.60.480.370.0260.0260.0150.0150.0150.0150.850.70.570.0570.0570.0380.0380.0460.046

No leaderRegual leaderIrregular leaderSafe pedestrians Rogue pedestriansProbability of fixationEffect of an increased perceptual load on probability of fixation

Condition 50.020.020.030.03

Unpredictable pedestrians when not collidingUnpredictable pedestrians when collidingfixations on unpredictable pedestriansProbability of fixationFixations on unpredictable pedestrians

Condition 60.030.030.0410.041

Safe RogueProbability of fixationPedestrian fixations with no prior experience (Trial 1)

0.050.050.0340.034

Fixations on pedestrians after conflicting experineceSafe (previously Rogue) Rogue (previously Safe)Probability of fixationPedestrian fixations after conflicting experience (Trial 2)

0.040.040.030.030.0550.055

Safe ped./ Safe environmentSafe ped./ No prior experienceSafe ped./ Conflicting experienceSafe pedestrianProbability of fixationFixations of Safe pedestrian in different contexts

586.4516.8333333333570.2158.4874127494158.4874127494104.9674288106104.967428810652.001346136452.00134613641083.3076923077537.8181818182260145.2331628714145.2331628714106.5094804301106.509480430149.171129740949.17112974091007.0714285714641.6268.5133.3356692953133.335669295393.875236351293.875236351239.308759474639.3087594746

RogueUnpredictableSafeEarly Middle LateFixation durationsChange in fixation behavior accross a single trial (Trial 1- no prior experience)

3476767142142979778074741341341021028308787122122104104

Rogue (previously Safe)Safe (previously Rogue)UpredictableEarly Middle LateFixation durationsChange in fixation behavior accross a single trial (Trial2- after conflicting experience)

0.420.0320.0320.450.040.04

Conflicting experienceSafe (non-stopping) peds Rogue (stopping) pedsProbability of fixationPedestrian fixations with no prior experience (Trial 1)

0.320.68

No experienceSafe (non-stopping) peds (previously Rogue) Rogue (stopping) peds (previously Safe)Probability of fixationPedestrian fixations after conflicting experience (Trial 2)

0.020.020.030.030.0410.041

Safe Unpredictable RogueProbability of fixation

Condition 2 (Colliders+ irregular leader)Trial #1 (No Leader trial)100% coll50% collnever collpercent of fixated colliders9070percent of fixated non-colliders7452Condition 2 (Colliders+ irregular leader)Trial #2 (No Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders7585percent of fixated non-colliders7167Condition 2 (Colliders+ irregular leader)Trial #3 (Leader trial)100% coll50% collnever collpercent of fixated colliders5937percent of fixated non-colliders4229Condition 2 (Colliders+ irregular leader)Trial #4 (Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders4554percent of fixated non-colliders4525

Condition 3 (Stop+ regular leader)Trial #1 (No Leader trial)100% coll50% collnever coll0.32percent of fixated colliders68430.680.15percent of fixated non-colliders25150.420.45Condition 3 (Stop+ regular leader)Trial #2 (No Leader trial)100% coll never coll50% collnever coll100% coll0.050.0320.04percent of fixated colliders2742percent of fixated non-colliders452343256815481345152723424521353331Condition 3 (Stop+ regular leader)Trial #3 (Leader trial)100% coll50% collnever coll3524553034.5243923percent of fixated colliders48300.350.240.550.30.350.240.390.23percent of fixated non-colliders2113Condition 3 (Stop+ regular leader)Trial #4 (Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders2021percent of fixated non-colliders3514483021132021351425.25

0.050.050.0320.0320.0320.0320.040.04

No experienceConflicting experienceSafe (non-stopping) peds Rogue (stopping) pedsProbability of fixation

Non colliding pedestriansUnpredictable pedestrians when not colliding Safe pedestriansProbability of fixationFixations on non-colliding pedestrians

Unpredictable pedestrians Rogue pedestriansProbability of fixationFixations on colliding pedestrians

No leaderRegular leaderIrregular leaderSafe pedestrians Rogue pedestriansProbability of fixationEffect of an increased perceptual load on fixations

Condition 4 (No Coll+ regular leader)Trial #1 (No Leader trial)100% coll50% collnever collpercent of fixated non-colliders483046383046Condition 4 (No Coll+ regular leader)Trial #2 (No Leader trial)100% coll never coll50% collnever coll100% coll41percent of fixated non-colliders504047403738.66666666670.38Condition 4 (No Coll+ regular leader)Trial #3 (Leader trial)100% coll50% collnever coll5.2788887720.044percent of fixated non-colliders3322322.1550973167average safge0.43Stand. Err.0.03Condition 4 (No Coll+ regular leader)Trial #4 (Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated non-colliders3630273313484522133033321721313614131530153531271320253014.166666666714220.30.142133average no coll + reg leaderno coll+irr leader25.2529.3750.260.29stop+rlstop+irr.l

Regular leaderIrregular leaderNon-colliding pedestrians Stopping pedestriansProbability of fixation

Condition 5 (No Coll+ irregular leader)Trial #1 (No Leader trial)100% coll50% collnever coll100% coll50% collnever collpercent of fixated non-colliders464543483046Condition 5 (No Coll+ irregular leader)Trial #2 (No Leader trial)100% coll never coll50% collnever coll100% coll100% coll never coll50% collnever coll100% collpercent of fixated non-colliders413748504047Condition 5 (No Coll+ irregular leader)Trial #3 (Leader trial)100% coll50% collnever coll100% coll50% collnever collpercent of fixated non-colliders131317332232Condition 5 (No Coll+ irregular leader)Trial #4 (Leader trial)100% coll never coll50% collnever coll100% coll100% coll never coll50% collnever coll100% collpercent of fixated non-colliders14151336302746134833451330224317463241145036371540304813472743.333333333314.166666666743.5300.430.140.440.3

Regular leaderIrregular leaderNo colliders+no leader No colliders+leaderProbability of fixation

Condition 6 (Stop+ irregular leader)Trial #1 (No Leader trial)100% coll50% collnever collpercent of fixated colliders6954percent of fixated non-colliders3124Condition 6 (Stop+ irregular leader)Trial #2 (No Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders4048percent of fixated non-colliders3929Condition 6 (Stop+ irregular leader)Trial #3 (Leader trial)100% coll50% collnever collpercent of fixated colliders45334515percent of fixated non-colliders311533313923Condition 6 (Stop+ irregular leader)Trial #4 (Leader trial)100% coll never coll50% collnever coll100% collpercent of fixated colliders2533percent of fixated non-colliders3122453331152533312229.375