Individual Differences, User Perceptions and EyeGaze in Biomedical Search Interfaces

7 September 2015

Ying-Hsang Liu 1,2

1School of Information Studies

Charles Sturt University

2Research School of Computer Science

The Australian National University

1Outline

Introduction

Research Questions

Interfaces

Research Design

Results

Summary and Discussion

2Introduction

Interactive InformationRetrieval (IIR)

▶ Current IR systems designed forspecified search (Belkin, 2008)

▶ Natural search userinterfaces (Hearst, 2011)

▶ Usefulness of controlled indexinglanguages (Salton, 1972; Svenonius, 1986)

Medical Subject Headings(MeSH) terms

3Research Questions

Research questions

▶ What elements of searchinterfaces do searchers look atwhen searching for documentsto answer complex questions?

▶ What is the relation betweenuser perceptions of aninterface and the interfaceelements they look at?

▶ What is the relation betweenindividual differences and theinterface elements which arelooked at?

User experiment in alaboratory setting

4Interface A: Google style

5Interface B: Per query, ProQuest

6Interface C: Per query, ProQuest+EBSCOhost

7Interface D: Per document, EBSCOhost

8Test Collection

Selection of search topics

▶ Document test collection fromOHSUMED (Hersh, Buckley, Leone, & Hickam, 1994)

▶ MEDLINE from 1987 to 1991;348,566 records

▶ Randomly select 8 topics basedon proportion of judged relevantdocuments

▶ 2 topics from each of thequartiles (4 search topic pairs)

Sample search topic

▶ ID: 78▶ Imagine that you are 42-year-oldblack man with hypertension.

▶ You would like to findinformation about beta blockersand blacks with hypertension,utility.

9Experimental Design

Factorial design

▶ 4× 2× 2 Factorial design; 4interfaces, controlled searchtopic pairs and cognitive styles

▶ 4× 4 Graeco-Latin Square toarrange experimental conditions

▶ Power Analysis for ANOVADesign; medium effect size of.25, α < .05 and N = 256,statistical power of .93 (Cohen, 1988;

Faul, Erdfelder, Lang, & Buchner, 2007)

4× 4 Graeco-Latin Square

10Software and Hardware

Experimental system setup

▶ Experimental search systembased on Solr

▶ Gaze tracking uses FaceLabsoftware and hardware

▶ EyeWorks for data recordingand analysis

▶ Emotiv headset for EEG data▶ Search logs and mouse clicksrecorded

Gaze tracking by FaceLab

11Experimental Procedure

Experimental procedure Data collection▶ User characteristics (backgroundquestionnaire and cognitive styletest)

▶ User perceptions (exitquestionnaire)

▶ Search behaviours (search logs,mouse clicks and documentssaved)

▶ Physiological signals (eye gazeand EEG)

12Searcher Characteristics

▶ 32 subjects; male (50.0%), female(50.0%)

▶ Student: postgraduate (46.9%),undergraudate (40.6%)

▶ Age: 18–25 (59.4%), 25–35(28.1%)

▶ Online database experience: < 5years (62.5%), 5–10 years(21.9%)

▶ Search engine: every day(50.0%), several times a day ormore (37.5%)

▶ Pilot study (Liu, Thomas, Schmakeit, & Gedeon, 2012)

Biology background

13Searcher Characteristics (cont’d)

▶ Cognitive style: Individual’spreference or tendency toprocess information

▶ E-CSA-WA (Extended CognitiveStyle Analysis–WholisticAnalytic) test (Peterson, Deary, & Austin, 2003)

▶ Wholistic Analytic Ratio▶ WA ratio (M = 1.31, SD = .24);cut-off = 1.32 (Clewley, Chen, & Liu, 2010; Chen,

Magoulas, & Macredie, 2004; Yuan, Zhang, Chen, & Avery, 2011)

E-CSA-WA Test

14Data Analysis

▶ Where do people look? Area ofinterest (AOI)

▶ Logarithmic cross ratio analysisbetween individualdifferences/user perceptions andAOI (Fleiss, Levin, & Paik, 2003; Saracevic, Kantor, Chamis, &

Trivison, 1988)

▶ ANOVA between interface andsearcher characteristics, such ascognitive style and searchexperience

Heat map and AOI

15Search Interfaces and AOI

Title Author Abstract MeSH

●

●

●●

●

●●●

●● ●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

0

25

50

75

A B C D A B C D A B C D A B C DTypes of Interface

Prop

ortio

n of

fixat

ions i

n AO

I

16User Perceptions and AOI

Table: Summary of the relation between user perceptions and AOI

Difficulty Usefulness Notice of Keywords Use of Keywords

B C D B C D

Title m m l l l l l —Author m — m m m m m m

Abstract m l — — — — — l

MeSH m — — — — — — —

Note. The relation is not statistically significant (—), positively significant (l), ornegatively significant (m) at 95%).

17Individual Differences and AOI

Table: Summary of the relation between individual differences and AOI

Domain Knowledge Search Experience Cognitive Style

UG PG Search Engine OnlineDatabase

Title m m — — —Author — — l — —Abstract — — m — —MeSH — — l — —

Note. The relation is not statistically significant (—), positively significant (l), ornegatively significant (m) at 95%).

18Interface and Search Experience Interaction

19Interface and Cognitive Style Interaction

20Summary and Discussion

Research findings

▶ Searchers look at abstract moreoften than other interfaceelements

▶ Interfaces and user perception ofsearch task difficulty significantlyaffects elements look at

▶ Significant interaction effectbetween cognitive style/searchexperience and interface forMeSH AOI

Discussion▶ Design of Search Engine ResultsPage (SERP)

▶ Detection of search taskdifficulty

▶ Individual differences for searchuser interface design

21

Thank You!

Questions orComments?

This study is partially funded by 2014 ALIA Research GrantAward, led by Dr Ying-Hsang Liu with Marijana Bacic (Monash

Health), Dr Paul Thomas (CSIRO) and Professor TomGedeon (ANU).