Eye Tracking Analysis of User Behavior in WWW

Search

Laura GrankaThorsten Joachims

Geri Gay

why use eye-tracking for information retrieval?

• Understand how searchers evaluate online search results

• Enhanced interface design

• More accurate interpretation of implicit feedback (eg, clickthrough data)

• More targeted metrics for evaluating retrieval performance

Figure: popular regions are highlighted through shadow-intensity

key research questions• How long does it take searchers to select a

document?• How many abstracts do searchers look at

before making a selection?• Do searchers look at abstracts ranked lower

than the selected document?• Do searchers view abstracts linearly?• Which parts of the abstract are most likely to

be viewed?

what is eye-tracking?

• Device to detect and record where and what people look at

• Multiple applications: reading, usability, visual search, in both physical and virtual contextsEye tracking device

Figure: Cornell HCI eye-tracking configuration

View of subject’s pupil on monitor; used for calibration

ocular indices for www tracking

• Fixations: ~200-300ms; information is acquired• Saccades: extremely rapid movements between fixations • Pupil dilation: size of pupil indicates interest, arousal

Aggregate eye-tracking graphs depict viewing intensity in key regions

“Scanpath” output depicts pattern of movement throughout screen. Black markers represent fixations.

experimental search tasks • Ten search tasks

given to all participants

• Search topics included travel, science, movies, local, television, college, and trivia

• Searches evenly split between informational and navigational tasks

experimental procedures

• Users conducted live Google searches

• Users allowed to search freely, with any queries

• Script removed all ad content

• Proxy stored all pages and log files

Figure: Specific “zones” were created around each result, enabling eye-movements to be analyzed specific to the rankings

sample eye-tracking output

sample eye-tracking output

sample eye-tracking output

sample eye-tracking output

sample eye-tracking output

overall searching behaviorHow long does it take users to select a document?

Overall mean: 5.7 seconds, St.D: 5.4

Time spent before a result is clicked

less difficultmore difficult task

overall viewing behavior

Total number of abstracts viewed per page

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10Total number of abstracts viewed

fre

qu

en

cy

Mean: 3.07 Median/Mode: 2.00

Most likely to view only two documents per results set

How many abstracts do we view, and in what order?How many abstracts do we view, and in what order?

Notice dip after page

break

overall viewing behaviorHow many abstracts do we view, and in what order?How many abstracts do we view, and in what order?

Instance of arrival to each result

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Rank of result

mea

n f

ixat

ion

val

ue

of

arri

val

Results viewed linearly

overall viewing behavior

Time spent viewing each abstract compared with the frequency that each rank is selected. Error bars are 1 SE

How much time do we spend viewing each abstract?How much time do we spend viewing each abstract?

overall viewing behavior

Number of abstracts viewed above and below selected link

How thoroughly do we view the results set?How thoroughly do we view the results set?

overall viewing behaviorWhat information in the abstract is most useful?

Title: 30%Snippet: 43%Category: 0.3%URL: 21%

Other: 5% (includes, cached, similar pages, description)

Percentage of time spent viewing each part of abstract

Other: 5.3%

Title: 30.5%

Category: 0.3%Snippet: 42.8%

URL: 21.1%

overall searching behavior

*Difficulty and satisfaction are ranked on a 1-10 scale; 10 meaning very difficult and very satisfied, respectively

Search task difficulty and satisfaction with Google Search task difficulty and satisfaction with Google

less difficultmore difficult

overall searching behavior

Mean rank of selected doc: 2.66 Median/ Mode: 1.00

Task difficultyinfluences rank of selected document and number of abstracts viewed

less difficultmore difficult

overall searching behavior

1. Michael Jordan statistician 20

2. Thousand acres dude ranch 11

2. One thousand acres dude ranch 11

3. 1000 acres dude ranch 9

4. Time machine movie 7

4. Carnegie mellon university graduate housing 7

5. Imdb 6

5. Emeril lagasse 6

5. First modern antibiotic 6

5. Greyhound bus 6

5. Carnegie mellon graduate housing 6

Top Query Terms Frequency

conclusions Searching Trends: Popularity of specialized,

vertical portals

Several students preferred conducting a Google search from the cmu.edu homepage

Majority of students preferred an internal imdb.com search over a general Google search

conclusions

• Document selected in under 5 seconds• Users click on the first promising link they see• Results viewed linearly• Top 2 results most likely to be viewed• Users rather reformulate query than scroll• Task type and difficulty affect viewing

behavior• Presentation of results affects selection

future research Impact on advertising

With such fast selections being made, will searchers even view ads?

?

Ads most likely to be seen:• Difficult task• Ambiguous info need• Informational query• Low searcher expertise

future research

Relevance judgments

• Do we spent more time viewing relevant abstracts?• Do we click the first relevant abstract viewed?• Does pupil dilation increase for more relevant

documents

If results were re-ranked, wouldviewing behavior differ?

If results were re-ranked, wouldviewing behavior differ?

Cornell University Computer Science &

Human-Computer Interaction

Thorsten Joachimstj@cs.cornell.edu

Laura Grankagranka@cornell.edu

Matthew Feusnerfeusner@cornell.edu

Geri Gaygkg1@cornell.edu

Helene Hembrookehah4@cornell.edu