Action as a Window to Perception: Measuring Attention with Mouse Movements

A Validation Study of the MetrixLab FocusTracker

Prof. dr. A. Johnson and dr. ir. L.J.M. Mulder

©2006 1

Action as a Window to Perception: Measuring Attention with Mouse Movements

Introduction Attention has been described as the interface between memory and events in the world. We have to attend to information if we are to encode it, and retrieval of past experiences depends on attention to appropriate cues in the environment (Logan & Compton, 1998). In visual processing, such as when shopping for a product on the internet or scanning supermarket shelves, attention is needed to locate relevant information and to guide action. Although it is possible to move the focus of attention at least a few degrees of visual angle away from the focus of the eyes (von Helmholtz, 1894), we almost always attend where we look (Johnson & Proctor, 2004). Therefore, if we want to know whether someone has attended to information, we will want to know if they have looked at it. Eye movement tracking is one means of measuring attention to scenes (Duchowski, 2002). In practice, however, the inconvenience and cost of collecting and analyzing eye-movement data limit the effectiveness of the technique for evaluating visual information displays. Recently, market research institute MetrixLab has developed a computer-based tool for tracking visual attention. This tool, the FocusTracker™, is based on the assumption that mouse movements provide a reliable indication of when and where attention is allocated on a computer screen. Because it is internet based, the tool can be used with hundreds of participants from any location in the world, allowing maximal freedom in targeting specific groups. It has the advantage that participants stay in a natural setting and that no laboratory or specialized equipment is necessary. Participants are first trained to point the mouse at high speeds, moving over images or texts on the computer screen. After this short training period, the displays of interest are presented with the instruction to the participant to “point to whatever catches your eye.” “Scan path” data from the FocusTracker can then be replayed using MetrixLab’s online FT Replay™ program to determine how attention is allocated to objects in the scene. The FocusTracker is based on the assumption that there is a one-to-one relationship between where we fix our gaze, where we point via a mouse, and what we are attending to. The question addressed in this research is thus whether the hand can be trained to follow visual spatial attention and whether attentional processing can be measured by tracking pointing movements with a handheld computer mouse. A related question is whether viewers can be adequately instructed to perform the task from their own homes, without the direct intervention of a researcher. In brief, the results of the research are very promising, showing high correlations between the scan paths for the eye and the mouse as well as high correlations between the percentage of time spent in designated regions of interest for the mouse and the eye. An additional comparison of the data with that of a group who did not use a mouse while viewing the experimental stimuli showed that viewing patterns were not disrupted by mouse use.

©2006 2

The Experiment: Tracking the Focus of Attention In order to determine whether the hand can effectively follow the eye while viewing visual information, an experiment was conducted in which 21 advertisements were viewed for 5 seconds each. Participants were instructed to view the advertisements and to attempt to move the mouse in the same way as the eyes. Eye movements were registered with an eye tracker and hand movements were registered by logging the position of a handheld computer mouse. Two conditions were compared: one in which the participants received a short, verbal instruction and demonstration of how the mouse should be moved, and one in which participants followed the FocusTracker instruction program.

Method

Participants. Each group included 15 participants. The FocusTracker instruction group included 5 men and 10 women, 11 of which were university students. The mean age in this group was 25 years old (sd = 7.15). The verbal instruction group included 3 men and 12 women; 11 of which were university students. The mean age in this group was 25 years old (sd = 9.58). All but one participant had completed either a VWO or HBO program.

Stimuli. The stimuli were relatively unknown advertisements1 taken from relatively expensive magazines. Advertisements were 25.5 cm high and 16-20.5 cm wide and were presented on a 17-inch computer screen. Each advertisement was made up of four regions of interest (ROIs): A headline, an illustration, text, and a trademark (see Figure 1). The position of each ROI varied across the advertisements and in some cases they overlapped.

Apparatus. Eye movements were recorded with an Applied Science Laboratories model 504 eye tracker equipped with a pan/tilt camera.2 Eye position was determined 50 times per s. A Logitech infrared mouse was used to record hand position. Mouse position was also sampled 50 times per s. Participants were tested individually in a dimly lit room. Procedure. Participants were randomly assigned to either the verbal instruction or FocusTracker training groups. Participants in both groups were told that they should move the mouse to follow their eye movements. Participants in the verbal instruction group were also given a demonstration of how the mouse should be moved along with the eyes. Participants in the FocusTracker training group performed the tasks in the FocusTracker training: following a moving butterfly with the mouse, moving the mouse to each of a series of sequentially presented objects of the same type, moving the mouse to each of a series of sequentially presented objects of different types and moving the

1 The familiarity of the advertisements was tested in a pilot study in which 15 people (aged 17-40) were asked whether they had ever seen each of the advertisements. Only advertisements that were recognized by no more than 3 of the 15 participants in the pilot study were used as stimuli in the experiment. 2 Eye position is determined by comparing the pupil and the corneal reflection of infrared light emitted from the camera.

©2006 3

mouse along with the eyes while viewing each of two advertisements. The training was followed independently by each participant, without the intervention of the researcher, and lasted approximately 1.5 – 2 min.

The session began with the calibration of the eye tracker. The training was then given and was immediately followed by the experimental trials. Trials were separated with the presentation of the mouse cursor centered in a light gray screen. This screen was shown for 6 s before the first trial and for 2 s between subsequent trials. Participants were instructed to look at the mouse cursor until the advertisement appeared and were told that it would be impossible to move the mouse during these 2 s. The 21 advertisements were presented for 5 s each in the same order for all participants. After participants had viewed all of the advertisements, a surprise memory test was given. In Part 1 of the memory test, participants were shown either the headline, illustration or trademark from one of the advertisements and were asked to recall the other two attributes of the advertisement (e.g., if the trademark was shown, participants should report the illustration and the headline; recall of the text was not tested). Each cue was used seven times. Part 2 of the memory test was a recognition test in which 42

(a)

(b)

(c)

(d)

Figure 1. A sample advertisement showing the four regions of interest (ROIs): the headline (a), illustration (b), text (c) and trademark (d). ROIs were defined for analysis by enclosing them in rectangles. In some cases (see ROI d), two rectangles were used to define the region. In case of overlap, the smaller ROI was subtracted from the larger ROI.

©2006 4

trademarks were presented, 21 from the advertisements used in the experiment and 21 from related products. The participant’s task was to classify each trademark as having been presented or not. Finally, participants were asked how familiar they were with each of the advertisements. At the end of the experimental session personal data (e.g., age, level of education, familiarity with the computer) was collected and participants were debriefed. The entire experiment lasted approximately 1 hour.

Data analysis. Not all trials could be included in the analysis because of missing or unreliable eye movement data. If more than 0.5 s data was not usable the trial was not analyzed. Missing data resulted from the camera losing the eye position, extreme eye position readings as a result of the camera misreading the reflection point and as a result of correcting for eye blinks. Approximately 33 trials per group (9.6% of the data) were excluded from analysis. Additionally, occasional extreme values (outliers) were removed and replaced by the average of the two values before and after the outliers. The same procedure was applied to brief eye blinks. The relation between eye and hand movements was tested by (1) comparing the scan paths for the eye and hand using bidimensional regression techniques and (2) by comparing the percentages of time spent by the eye and hand, respectively in each of the four ROIs. Because mouse movements lagged behind eye movements, it was necessary to compensate for the lag on each trial. This was done by determining the best fit (bidimensional r) between the mouse and eye data. On average, 0.63 s of the mouse data at the beginning of the trial was discarded, as was a corresponding amount of the eye movement data at the end of the trial. The FocusTracker training group took on average 0.58 s to move the mouse whereas the verbal instruction group needed 0.69 s (F(1, 25) =

Eye scan path Mouse scan path

Figure 2. Eye and mouse scan paths from a single participant.

©2006 5

4.37, p = .047). Both groups were slower to move the mouse on the first advertisement (m = 1.17 s) as compared to all other advertisements (m = 0.61 s; (F(1, 25) = 22.63, p < .001).

Results

The relation of mouse and eye scan paths. Sample eye and mouse scan paths are shown in Figure 2. The degree to which the mouse followed the position of the eyes was assessed with bidimensional regression. The overall correlation between the scan paths (r), and the rotation (θ), expansion (φ), and translation (to the right or left or up or down; α) of the mouse scan path relative to that of the eye were determined individually for each participant and each advertisement (see Table 1). Correlations were Fisher transformed for analysis. Large correlations (ranging from .83 - .92 across advertisements) were found between the mouse and eye scan paths. These correlations did not significantly differ as a function of group. Differences between the advertisements were also minimal.3 Analysis of the rotation parameter, θ, revealed that the mouse scan path showed a slight (m = 11.5°) rotation to the right relative to the eye scan path. The expansion parameter, φ, with an average value of 0.8, reflected that the mouse scan path covered a somewhat smaller area than that covered by the eye. Finally, the mouse scan path was shifted, on average, about 4 cm above and to the right of the eye scan path.

3 One advertisement significantly differed from two others; no other differences between advertisements were found.

Table 1. Average Values from the Bimendimensional Regression as a Function of Group (standard error in parentheses)

Group

Value

FocusTracker training Verbal Instruction

Correlation (r) 0.87 (0.04) 0.89 (0.04)

Rotation (θ) 0.14 (0.06) 0.09 (0.05)

Expansion (φ) 0.80 (0.02) 0.79 (0.02)

Left-right translation (α1) 90.2 (13.3) 87.0 (12.0)

Up-down translation (α2) 79.8 (9.2) 70.9 (6.1)

©2006 6

Percentage time in each region of interest. The overall percent time spent in each of the ROIs by the eye and mouse, respectively, is shown in Table 2. Two analyses were carried out: A correlational analysis in which the correlation between the time spent in each region by the eye and the mouse was computed across all advertisements for each participant, and an ANOVA with ROI (illustration, headline, text or trademark), effector (eye or mouse) and group as factors. Correlations were Fisher transformed for analysis. Overall, the correlation between time spent in each of the ROIs for the eye and mouse was high (r = .88). No significant difference in the correlation as a function of group was found (r = .88 and r = .87 for the FocusTracker and verbal instruction groups, respectively). Additional analysis carried out per advertisement showed average correlation coefficients ranging from .68 to .97. Figure 3 shows the time spent in each ROI by the mouse as a function of the time spent in each ROI by the eye. An ANOVA with ROI (illustration, headline, text or trademark) and effector (eye or mouse) as within subject factors and group (FocusTracker training or verbal instruction) as a between subject factor showed a main effect of ROI (F(3, 84) = 102.81, p < .001). Follow-up tests showed that significantly more time was spent on the illustration than the headline, and on these two ROIs than on the text or trademark. Furthermore, there was a significant Effector x ROI interaction (F(3, 84) = 14.14, p < .001). As can be seen in Table 2, the tendency to spend the most time on the illustration was more pronounced for the mouse than for the eye. No differences between groups were found. To investigate in more detail the differences in percentage time spent in the ROIs, the number of cases in which the eye visited a ROI not visited by the mouse, and vice versa, was computed. These percentages are shown in Table 3. The eye was more likely to visit an area not visited by the mouse than vice versa (F(1, 28) = 31.98, p < .001). This was especially the case for the trademark.

Table 2 Percent Time Spent in Each Region of Interest by Eye and Mouse as a Function of Group (standard error in parentheses) Group

FocusTracker Training Verbal Instruction

Region of Interest % time eye % time mouse % time eye % time mouse

Illustration 41.2 (3.12) 47.1 (3.51) 46.3 (2.32) 50.1 (2.16)

Headline 30.1 (1.83) 28.0 (2.15) 27.7 (1.61) 26.9 (1.75)

Text 14.8 (1.89) 14.8 (2.10) 11.6 (1.76) 10.9 (1.69)

Trademark 11.3 (1.27) 9.5 (1.27) 11.0 (1.22) 10.9 (1.44)

©2006 7

Effects of Mouse Use on Viewing Behavior The results of the experiment comparing mouse and eye scan paths suggest that “mouse tracking” can be an excellent substitute for eye tracking. High correlations were found between the forms of the scan paths for mouse and eye and for the amount of time spent in each ROI by the mouse and eye. Before using mouse tracking to evaluate observer behavior, however, it is important to know whether the use of the mouse leads to a different way of looking. That is, it is important to know that mouse use does not result in viewing behavior that is different than looking under normal conditions. In order to

100,0080,0060,0040,0020,000,00

percentage tijd oog per aandachtsgebied

100,00

80,00

60,00

40,00

20,00

0,00

per

cen

tag

e ti

jd m

uis

per

aan

dac

hts

geb

ied

merk

afbeelding

tekst

headlineaandachtsgebieden

0 20 40 60 80 100

Percent time eye in ROI

Per

cent

tim

e m

ouse

in R

OI

0

20

4

0

60

80

10

0 Region of InterestHeadlineTextIllustrationTrademark

100,0080,0060,0040,0020,000,00

percentage tijd oog per aandachtsgebied

100,00

80,00

60,00

40,00

20,00

0,00

per

cen

tag

e ti

jd m

uis

per

aan

dac

hts

geb

ied

merk

afbeelding

tekst

headlineaandachtsgebieden

0 20 40 60 80 100

Percent time eye in ROI

Per

cent

tim

e m

ouse

in R

OI

0

20

4

0

60

80

10

0 Region of InterestHeadlineTextIllustrationTrademark

Figure 3. Percent time spent in each region of interest (ROI) by the mouse as a function of percent time spent in each region of interest by the eye

Table 3 Average Percentage of Trials in Which Either the Eye Visited a Region of Interest Not Visited by the Mouse or the Mouse Visited a Region of Interest Not Visited by the Eye

Group

FocusTracker Training Verbal Instruction

Region of Interest Eye only Mouse only Eye only Mouse only

Illustration 1.47 (0.65) 0.0 (0.0) 0.73 (0.50) 0.33 (0.33)

Headline 6.27 (1.78) 0.0 (0.0) 7.73 (1.73) 1.07 (0.57)

Text 9.47 (2.58) 5.53 (1.66) 10.40 (2.30) 8.27 (1.30)

Trademark 13.00 (2.01) 1.73 (1.03) 7.53 (1.62) 0.33 (0.33)

©2006 8

examine this issue, a group of observers comparable to the experimental groups was tested under the experimental conditions but without use of the mouse. The number and length of fixations in each ROI was then computed.

Table 4 shows the average number and duration of fixations within each ROI as a function of whether or not observers used the mouse while viewing the advertisements. Separate ANOVAs with ROI (illustration, headline, text or trademark) as a within-subjects factor and group (FocusTracker training, verbal instruction or eye tracking only) as a between-subjects factor were conducted on the percentage of time spent in each ROI, the number of fixations in each ROI, and the average duration of the fixations. No differences between groups were found, nor did group interact with ROI. In short, effects of using the mouse on looking behavior are minimal.

Effects of Mouse Use on Recognition and Recall If the mouse is used to track attention, and attention is subsequently tested with recognition and recall questions, it is important to know whether mouse use has an effect on memory for the viewed material. In order to test this we measured recall of the illustration, headline and trademark, and recognition of trademarks, for each of the three groups. For the recall test, one element of the advertisement was shown (e.g., the headline), and observers were asked to recall the other ROIs, excluding the text. Table 5 shows recall performance as a function of group, ROI and cue (e.g., either the headline or the trademark could be shown as a cue for the illustration). An ANOVA with ROI (illustration, headline or trademark) as a within-subject factor and group (FocusTracker training, verbal instruction or eye tracking only) as a between-subject factor showed significant effects of ROI (F(2, 88) = 87.44, p < .001) and group (F(2, 44) = 6.75, p =

Group

FocusTracker Training Verbal Instruction Eye Tracker Only

Region of Interest

Number of Fixations

Duration of Fixations

Number of Fixations

Duration of Fixations

Number of Fixations

Duration of Fixations

Illustration 8.5 (0.57) 204 (9.50) 9.7 (0.65) 203 (11.15) 8.3 (0.43) 196 (11.33)

Headline 6.6 (0.38) 191 (9.25) 6.0 (0.40) 189 (3.41) 6.3 (0.43) 171 (8.27)

Text 3.7 (0.49) 168 (5.92) 2.9 (0.43) 181 (7.68) 3.7 (0.41) 163 (8.39)

Trademark 2.6 (0.13) 168 (6.14) 2.3 (0.25) 184 (6.94) 2.8 (0.19) 157 (5.59)

Total 21.3 (0.41) 188 (8.15) 20.8 (0.57) 190 (8.17) 21.1 (0.41) 176 (7.90)

Table 4 Number and Duration of Fixations as a Function of Group and Region of Interest (standard error in parentheses)

©2006 9

.003), as well as a significant ROI x Group interaction (F(4, 88) = 6.17, p < .001). Both groups who used the mouse remembered fewer elements of the advertisements than the group who performed the task without the mouse. This effect was significant for the ROIs “illustration” and “headline”, but not for the ROI “trademark”. For the recognition test, participants were presented with a list of the 21 trademarks seen in the advertisement, combined with a list of 21 similar trademarks. On average 33% of the presented trademarks were recognized. Of the similar trademarks, 13% were incorrectly classified as having been seen. Recognition performance did not differ between groups.

Summary High correlations between eye and mouse scan paths and between percentage of time spent in each ROI by the eye and the mouse indicate that the mouse is a viable alternative to the eye tracker for measuring attention under natural viewing conditions. More than 75% of the variability in eye movements is captured by the mouse. Moreover, the lack of differences between the FocusTracker and verbal instruction groups suggest that instructions can be given remotely without any decrement to the accuracy of the technique. Use of the mouse had little influence on the way in which observers viewed the advertisements. Thus, the results would seem to be generalizable to other viewing

Table 5 Percent Correct Recall as a Function of Group, Region of Interest and Cue (standard error in parentheses) Group

Region of Interest FocusTracker training Verbal Instruction Eye Tracker Only

Illustration

Headline cue 43% (4.83) 23% (5.88) 47% (3.83)

Trademark cue 41% (3.91) 38% (5.34) 61% (4.98)

Headline

Illustration cue 14% (3.94) 9% (3.36) 42% (6.87)

Trademark cue 14% (4.18) 14% (4.62) 29% (5.05)

Trademark

Illustration cue 21% (3.91) 24% (5.34) 36% (5.63)

Headline cue 18% (4.06) 17% (4.67) 18% (3.19)

©2006 10

situations. Use of the mouse did influence how much could be remembered of what was seen. This suggests that using the mouse does make demands on mental resources.

Recommendations Several aspects of the data should be taken into account when using the mouse to track attention. First, both the FocusTracker training and verbal instruction groups showed a lower correlation between eye and mouse scan paths on the first advertisement than on subsequent advertisements. This suggests that at least one “practice trial” should be used before the stimuli of interest are shown. Second, consideration should be taken of the fact that the scan path of the mouse covers a smaller area than that of the eye, and is shifted somewhat to the right and to the top of the display. The smaller area is due to the fact that the eye sometimes made movements that were not followed with the mouse. The shift to the right and to the top of the display may be a result of using the right hand to move the mouse. Another factor that may have played a role in this shift is that in many of the advertisements, the trademark was at the bottom of the advertisement. In at least some cases, the eye moved to the trademark while the mouse lagged behind, perhaps because the cursor would get in the way of reading the text or because the trial ended before the mouse movement was completed. Specifically, the following recommendations are made:

• Present a “practice” stimulus after training and before beginning the evaluation of the stimuli of interest

• Define regions of interest to take into account the tendency of the mouse to be shifted to the right and top of the display

• Do not analyze the first 0.63 s of the trial data, or measure the time taken to move the mouse

• Do not rely on memory for viewed advertisements as an indication of where observers allocated their attention.

Mouse tracking is a viable alternative to eye tracking for determining which elements of advertisements receive attention during a short viewing period. Moreover, the FocusTracker training is a viable, on-line method of instructing observers to perform the task.

©2006 11

References Duchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behavior

Research Methods Instruments & Computers, 34, 455-470. Johnson, A., & Proctor, R. W. (2004). Attention: Theory and practice. Thousand Oaks,

CA: Sage Publications. Logan, G. D., & Compton, B. J. (1998). Attention and automaticity. In R. D. Wright

(Ed.), Visual attention. Vancouver studies in cognitive science (Vol. 8, pp. 108-131). New York: Oxford University Press.

von Helmholtz, H. (1894). Über den Ursprung der richtigen Deutung unserer Sinneseindrücke (The origin of the correct interpretation of our sensory impressions). Translated in R. M. Warren & R. P. Warren (1968). Helmholtz on perception, its physiology, and development (pp. 249-260). New York: Wiley.