Neuro-Physiological Evidence as a Basis for Studying Search

Neuro-‐Physiological Evidence as a Basis for Studying Search

Dr. Jacek Gwizdka

Information eXperience (IX) Lab, Co-Director School of Information, University of Texas at Austin

[email protected]

http://gwizdka.com/research www.neuroinfoscience.org | www.neuroir.org

December 8, 2015

http://bit.ly/ix_lab http://www.ischool.utexas.edu

Understanding Search

2

Psycho–physiological (PP) States

3

�  Two-way interaction ◦ PP states affect human information interaction (HII) ◦ HII induces changes in PP states

Search intent Relevance

Engagement

Peeking Inside a Searcher’s Brain

© Jacek Gwizdka

Brain activation Word vs. Concept search

Brain activation responding to relevant vs. partially relevant document

fMRI

Eye gaze patterns •  reading vs. scanning

à task type •  cognitive load

à task difficulty •  “mindless” reading …

EEG Brain activity measured during information-interaction •  cognitive load •  stress •  engagement …

Eye tracking

Peeking Inside a Searcher’s Brain Using Neuro-Physiological Methods

© Jacek Gwizdka

Outline

© Jacek Gwizdka 6

Neuro-Physiological (NP) Methods Three modalities: eye-tracking, EEG, and fMRI

Correlating NP data with Information Relevance

1.  fMRI on static text documents 2.  Eye-tracking -- ,, -- 3.  Eye-tracking + EEG -- ,, -- 4.  Eye-tracking on web pages

Challenges and opportunities in using NP

Experimental Design � Mixed-design with 2 blocks (types of tasks, balanced)

� Document corpus – small subset from AQUAINT ◦  a corpus of English-language news, international topics, text only

�  Task type 1: WS – word search: ◦ Find target word in a short news story – press yes/no ◦  21 trials composed of a target word 1 documents ◦  Order of trials randomized

7

xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek kdkddk dkdkdk dkdkdkd kkdkd d d dd d djdj djdjdj rjrjr rjr jweje ejejej ejej kekekek ekeke wej e eej eje j

+ + target: word

21 x

WS task instruc-

tions +

30s 6s 6s 6s 20s max 6s

WS

Experimental design with Dr. Michael Cole © Jacek Gwizdka

Experimental Design �  Task type 2: IS – information search ◦ Find information that answers a given question – press yes/no. ◦  21 trials composed of a question (task) and 3 documents at different

levels of relevance: Relevant (R), Topical (T), Irrelevant (I) à next slide

(pseudo-randomized – subset of possible combinations) ◦  Order of trials randomized ◦  To minimize memory load, the question was repeated before each stimulus

xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek dkdkdkkd kdkddk dkdkdk dkdkdkd kkdkd d d dd d djdj djdjdj rjrjr rjjweje ejejej ejej kek ekeke wej e ejej eje j

xmx ssms nsns snsns jsdjsd ke ekek dkdkdkk kdkddk dkdkdkdkdkdkd kkdkd d rjr jweje ejeje ekeke wej e ejej fjfjf fjfjfjfjf fjfjrjr rreje j

xmx ssms nsns snsns jsdjsd ke ekek dkdkdkkd

kdkddk dkdkdk dkdkdkd kkdkd d rjr jweje ejejej ejej kekekek ekeke wee ejej fjfjf fjfjfjfjf fjfjrjr rreje j

+ target: infoQ

target: infoQ

target: infoQ

21 x

IS task instruc-

tions

+ + + + + + 30s 6s 8s 6s 20s max 6s 4s 6s 20smax 6s 4s 6s 20smax

8

© Jacek Gwizdka

IS

© Jacek Gwizdka

Example IS Question and Text Docs

© Jacek Gwizdka 9

Does the next news story contain the following information: Russian submarine Kursk sinks: Which Russian fleet was the submarine part of?

Relevant News Story (R)

Irrelevant Story (I)

Partially Relevant Story (T)

© Jacek Gwizdka 10

Are brain areas involved in processing relevant and irrelevant texts different?

Experiment 1: fMRI

TYPE controlled lab study PARTICIPANTS 8 students (out of 18 total) (19-26 years); 1 fMRI session each SEARCH TASKS fact-finding task (Q&A); 21 topics STUDY DESIGN within-subjects DOC CORPUS AQUAINT - a corpus of English-language news (TREC Q&A) DATA COLLECTION binary relevance judgments

fMRI Siemens 3T and eye-tracking (EyeLink 1000)

11 © Jacek Gwizdka

NeuroIS’2013 “Looking for information relevance in the Brain” Most visionary paper award! Experimental design and data collection with Dr. Michael Cole

fMRI – functional magnetic resonance imaging

© Jacek Gwizdka 12

�  fMRI uses BOLD – Brain Oxygen Level Dependent signal ◦  an indirect measure of neural activity – disputed ◦  oxygenated and de-oxygenated blood differ in magnetic properties ◦  increased neural activity à increased blood flow à more freshly oxygenated

blood à increased BOLD signal � Allows for precise location of activated brain regions, but: ◦ Poor temporal resolution (> 6 seconds)

◦  “Harsh” conditions for participants (noise, no movement, tiny space) ◦ Very expensive and difficult to use ◦ Difficult data analysis

� Danger of “reverse” inference ◦ For example �  executive function à medial frontal gyrus �  medial frontal gyrus à executive function

X

Pilot Study: fMRI + Eye-tracking � RUBIC = RUtgers Brain Imaging Center

Prof. Steve Hanson lab director �  Lab Equipment: ◦  fMRI: 3T Siemens TRIO ◦ Eye-tracker: Eyelink-1000 �  non-ferromagnetic optimized design; up to 2000 Hz sampling rate

13 © Jacek Gwizdka

fMRI + Eye-tracking Setup

Eye-tracking imposes additional constraints on projection (geometry)

projected screen

mirror

eye-tracker

14

Infrared cam

projected screen

Hypotheses

© Jacek Gwizdka 15

1.  Brain activations associated with fact-finding (IS task) relevance judgments are different from activations associated with visual word search (WS task)

2.  Brain activations associated with judging relevant, partially relevant and irrelevant information are different

◦  no hypothesis about specific ROIs (i.e. where the brain activity is located)

Model

Text stimulus

Text stimulus

Text stimulus

Response Response Response

6s Fixation

+

6s Fixation

+

6s Fixation

+

xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek dkdkdkkd fsdf sdfsdf kdkddk dkdkdk dkdkdkdff fsdf sdf kkdkd d d dd d djdj djdjdj rjrjr rjjweje ejejej ejej jddddd hfisdfh h osifh oishd foosh foih

xmx ssms nsns snsns ddjsdjsd ke ekek dkdkdkk kdkd sss

xmx ssms nsns snsns jsdjsd ke ekek dkdkdkkd

kdkddk dkdkdk dkdkdkd kkdkd d rjr jweje ejejej e kekekek ekeke wee ejej fjfjf fjfjfjfjf fjfjrjr rreje j

target: infoQ Q

8s 6s 20s max 6s 4s 4s 20smax 6s 4s 4s 20smax 6s 4s 4s IS

++ + ++ Q ++ Q

© Jacek Gwizdka

fMRI Data Pre-Processing

© Jacek Gwizdka 17

� Anatomical (structural) scans manually extracted brains (FSL/BET)

� motion correction �  spatial smoothing 5mm �  temporal filtering

Analysis – 1st Level (individual)

© Jacek Gwizdka 18

� Analysis of individual sessions using GLM (FSL/FEAT) � Activations for WS, IS (R,T,I) � Voxel thresholding (p=.05) ◦  select voxels activated at a given significance level ◦  input square waveform was convoluted using Gaussian kernel ◦  slice timing corrected by shifting the model achieved by adding temporal

derivative

Text stimulus

Text stimulus

Text stimulus

Response Response Response

6s Fixation

6s Fixation

6s Fixation

Analysis – 2nd Level (group)

© Jacek Gwizdka 19

� Group analysis using individual analysis results � GLM: t-tests and ANOVAs with repeated measures ◦ Mixed-effects analysis

�  2 factors : IS and WS ◦ WS – 1 level ◦  IS – 3 levels (R,T,I)

�  IS vs. WS �  IS R vs. T, R vs. I, T vs. I

� Clusters of voxels thresholded at two levels: ◦  clusters1 (C1): less restrictive Z=1.65, p=.05 ◦  clusters 2 (C2): more restrictive: Z=2.33, p=.001

Results: IS vs. WS � Simple repeated measures ANOVA, 2 factors ◦  1-fixed (the two conditions: IS, WS), and 1 random-effect (subject)

�  IS>WS significant, but WS>IS not significant ◦ C1: 1 cluster p<10-9, Zmax=3.89; ◦ C2: 2 clusters p<10-4, Zmax=3.89, �  Left Occipital lobe/pole, middle occipital lobe, (near visual cortex) �  Right Occipital lobe/pole, lingual gyrus ◦  (X,Y,Z)= (-25,-95,6) & (17,-93,-6) [mm] @ voxel: (115, 31, 78) & (73, 33,66)

© Jacek Gwizdka

Results IS-R, T, I

© Jacek Gwizdka 21

� Repeated measures ANOVA, 1-factor with 3-levels ◦ mixed- effects: 1 factor: fixed-effect; subject: random-effect

�  F-test ◦ C1: p=.008; Zmax=3.62; Left Frontal lobe, (Middle Frontal Gyrus) �  lower probability of: Precentral G, Superior FG, BA: 6 �  (X,Y,Z)= (-39, 4, 53) [mm]; @ voxel (129,130,125) à Region 1 ◦ C2: not sig

Region 1

Results IS-R, T, I

� ANOVA w. repeated measures - individual contrasts: ◦  IS R-T (Relevant-Topical) sig ◦  IS R-I (Relevant-Irrelevant) sig ◦  IS-T-I (Topical-Irrelevant) not-sig

© Jacek Gwizdka

IS Relevant-Topical

© Jacek Gwizdka 23

� C1: 3 clusters ◦  p<.001 Zmax=3.58 �  Left Parietal lobe �  Lateral Occipital Cortex, AG, BA39 �  (X,Y,Z)=(-42,51,9) [mm] ◦  p=.007 Zmax=3.18 �  Right Frontal Lobe/Pole �  MFG ◦  p=.02 Zmax=2.99 �  Left Frontal Lobe/Pole �  MFG, BA46

� C2 – none left

Region 2 Left

IS Relevant- Irrelevant

© Jacek Gwizdka 24

� C1: 3 clusters ◦  p =10-12 Zmax=4.13 �  Left Frontal lobe (MFG)

◦  p=.0013, Zmax=3.25 �  Right Temporal lobe �  Middle & Inferior TG, BA20 ◦  p=.04, , Zmax=3.31 �  Right Parietal lobe �  Lateral OC, Angular Gyrus, BA39

� C2: first cluster left ◦  p=10-6 Zmax=4.13 ◦  Left Frontal lobe ◦ X,Y,Z= -39, 4, 53 [mm] ◦ Middle Frontal Gyrus �  PG, SFG, BA6

Region 1

Region 2 Right

Results Summary: IS-WS; IS R–T R-I T– I

� Significant difference in activation between IS-WS

� Pattern of significant and insignificant differences in activation between IS R-T, R-I, T-I

�  Brain activations similar to fMRI study with images by Moshfeghi et al. Moshfeghi, Y., Pinto, L. R., Pollick, F. E., & Jose, J. M. (2013). Understanding Relevance: An fMRI Study. ECIR 2013.

C1. Less restrictive C2. More restrictive R-T yes

(L parietal–R2L, R&L frontal) no

R-I yes (L frontal-R1, R temporal, R parietal-R2R )

yes (L frontal-R1)

T-I no no

© Jacek Gwizdka

Experiment 2: Eye-tracking + EEG + static text stimuli

TYPE controlled lab study PARTICIPANTS 24 students (9 females); 1 search session each SEARCH TASKS fact-finding task (Q&A); 21 topics STUDY DESIGN within-subjects, more in the next slides DOC CORPUS AQUAINT - a corpus of English-language news (TREC Q&A) DATA COLLECTION binary relevance judgments, eye-tracking (Tobii T60)

26 © Jacek Gwizdka

IIIX’2014 “Characterizing Relevance with Eye-tracking Measures” ETRA 2014: “News Stories Relevance Effects on Eye-Movement” Experimental design and data collection with Dr. Michael Cole

Eye-tracking History – A Digression

© Jacek Gwizdka 27

�  Mechanical registration: Delabarre (1898) and Huey (1898) �  Writing lever secured to eyeball using ring of plaster of Paris, later

using suction cups �  Cornea was cocainized to prevent pain and winking.

writing lever light rod

eye

cornea cocainized

ring of plaster, suction cup

Eye-tracking

28

�  Eye-mind link hypothesis: attention is where eyes are focused* (Just & Carpenter, 1980; 1987)

�  Modern eye-trackers computerized and “easy to use” ◦  infrared light sources and cameras (low-accuracy possible using web cams)

◦  eye-trackers use relationship between pupil and corneal light reflection to calculate where a person is looking – eye-gaze location

Example Tobii eye-‐trackers © Jacek Gwizdka

mobile /wearable eye-tracker stationary (“remote”)

Research Questions

© Jacek Gwizdka 29

RQ1. Does the degree of relevance of a text document affect how it is read?

RQ2. Does the degree of relevance affect cognitive effort invested in reading it?

RQ3. Could the degree of relevance be plausibly inferred in an non-intrusive way from eye-tracking data?

Independent and Dependent Variables

© Jacek Gwizdka 30

�  Independent: ◦  document relevance ◦  perceived relevance

� Dependent: ◦  time on a document (reaction time) ◦  reading state probability transitions, ◦  eye-tracking-based cognitive effort measures, ◦  pupil dilation (relative change in pupil diameter)

Two-State Reading Model

31

◦  Filter fixations < 150ms (min time required for lexical processing) ◦  Model states characterized by: �  probability of transitions; number of lexical fixations; duration �  length of eye-movement trajectory, amount of text covered

Scan Read

1-q

p

1-p

q isolated fixations fixation

sequences in one line of text

© Jacek Gwizdka

Cole, M. J., Gwizdka, J., Liu, C., Bierig, R., Belkin, N. J., & Zhang, X. (2011). Task and user effects on reading patterns in information search. Interacting with Computers, 23(4), 346–362

Results: Document relevance vs. perceived relevance

© Jacek Gwizdka 32

Document Relevance

I T R Total

Perceived Relevance

I 258 178 36 470

R 2 29 229 260

Total 260 205 265 730

Mean participant accuracy

99.2% 85.9% 86.4% Overall

accuracy: 91.1%

Results: Example Eye-Movement Patterns

© Jacek Gwizdka 33

What kind of document is read: Relevant, Topical, Irrelevant? 1.


© Jacek Gwizdka 36

1. 2. 3.

R Relevant document T Topical document I Irrelevant document


© Jacek Gwizdka 37

1. Irrelevant document 2. Topical document 3. Relevant document

Results: Reading Transition Probabilities

© Jacek Gwizdka 38

Degree of text’s relevance affects:

◦  how the text is read

Scan Read

1-q

p

1-p

q

R relevant T part.relevant I irrelevant

Results: Cognitive Effort & Relevance

© Jacek Gwizdka 39

Eye-tracking derived cognitive effort measures (per document)

Document relevance

Perceived relevance

Variable I T R I R Reaction time (RT) [s] L H M not. sig.

Reading speed [pixels] L H H L H Reading speed [words] not. sig. not. sig.

Duration of longest reading seq [s] L M H L H Max fixation duration in a reading seq [ms] L M H L H

Number of words fixated upon L H M not. sig. Number fixations on words L H M not. sig.

Proportions of words fixated on [%] L H M not. sig.

All significant at p<.001 H-high; M-medium; L-low

Results: Cognitive Effort & Relevance

© Jacek Gwizdka 40

Degree of text’s relevance affects:

◦ Cognitive effort

◦ Maximum cognitive effort

(not to scale)

I irrelevant T part.relevant R relevant

(not to scale)

Results: Pupilometry

© Jacek Gwizdka 41

� Pupil dilation is controlled by the Autonomic Nervous System � Dilation associated with cognitive functions: ◦ mental effort, interest, making a decision à attention

� Relative change in pupil diameter:

Relative pupil dilation measured during the whole text stimulus exposure

I T R ANOVA

Document relevance - 1.2% (.02%) - 0.4% (.02%) +0.8%(.02%) F(2, 505183)=3439, p<.001

Perceived relevance - 0.9% (.02%) +1.1% (.02%) F(1,505186)=8689, p<.001

Relative pupil dilation during the last 1000 ms before a participant’s response

Document relevance - 0.3% (.05%) +1% (.05%) +2.9% (.05%) F(2, 47498)=1211, p<.001

Perceived relevance +0.1% (.03%) +3% (.05%) F(1, 47499)=2696, p<.001

prit = (pt-pi

avg)/piavg

Results: Classification

© Jacek Gwizdka 42

� Perceived relevance: two classes: R & I ◦ Decision tress C4.5 algorithm with 10-fold cross-validation �  as implemented in Weka 3.7 under the name J48 ◦ Chance probability for two classes: 50%

Variables Accuracy Reading transition probabilities 72%

Eye-tracking cognitive effort 72% Reaction time 64%

Eye-tracking + reaction time 74% Pupil diameter – whole text stimulus exposure 65%

Pupil diameter – 1000 ms before response 67%

Summary of Experiment Results at Text Stimulus Level

© Jacek Gwizdka 43

� RQ1. Does the degree of relevance of a text document affect how it is read? ◦  yes… Reading patterns differ between R, T, I text documents

� RQ2. Does it affect cognitive effort invested in reading it? ◦  yes…

� RQ3. Could the degree of relevance be plausibly inferred in an non-intrusive way from eye-tracking data? ◦  yes… possibility of inferring degree of relevance from eye-movement

patterns and from pupil dilation

�  Limitations ◦  one type of texts (news stories) and only one type of a search task (fact

finding)

Experiment 2 continued Eye-tracking + EEG + static text stimuli

44 © Jacek Gwizdka

Results – within text stimulus

Unpublished. Collaboration on data analysis with Dr. Shouyi Wang, UT Arlington

EEG – Electro-encephalography

© Jacek Gwizdka 45

� Captures electrical signals at the skull – neural activity in cortex ◦  very good temporal resolution (1ms) ◦  not good at identifying location ◦  need to clean signal �  electricity; facial muscles

�  Two types of analysis ◦  signal properties �  spectral analysis (α θ β δ) �  signal power ◦  event-related potentials ERP

� Correlated with: ◦  cognitive load ◦  attention ◦  engagement ◦  meditation…

Research Questions

© Jacek Gwizdka 46

RQ1. Could the text document relevance be plausibly inferred from EEG signals obtained from a low-cost device alone and in combination with eye-tracking data?

RQ2. Does the text document relevance affect EEG signals and eye-tracking data differently at early, middle, late stages of reading and during reading relevant words?

Classification: EYE features

© Jacek Gwizdka 47

Classification: Algorithms

© Jacek Gwizdka 49

�  Feature Selection ◦ Minimum redundancy maximum relevance (mRMR)

� Classification Method ◦ Binary classification model Proximal Support Vector Machine (PSVM)

Classification: Data Segmentation - Epochs

© Jacek Gwizdka 50

t 200ms fixations on relevant words

1s-2s 1s-2s 1s-2s

BEGINNING MIDDLE END RELEVANT

0

Classification Results

© Jacek Gwizdka 52

Feature Set

Epochs (pairwise for perceived relevance)

AUC Accuracy

EEG beg; mid; end 0.57; 0.60; 0.59 0.55; 0.60; 0.65

EYE beg; mid; end 0.56; 0.70; 0.80 0.40; 0.66; 0.72

EEG Fixations on relevant vs. other (beg/mid/end)

0.76-0.78 0.74-0.76

EYE Fixations on relevant vs. beg; mid; end 0.88; 0.79; 0.77 0.79 0.70; 0.71

EYE* Fixations on relevant vs. beg for 1000ms epoch 0.95 0.86

EYE+EEG* Fixations on relevant vs. beg for 1000ms epoch 0.96 0.87

Classification improvement from adding EEG features •  to best EYE classification: 0.01-0.02 •  to comparable EYE classification: 0.05-0.07

Classification Results Interpretation

© Jacek Gwizdka 53

� Reading relevant and not-relevant texts ◦ Eye-tracking data reflects how texts are read ◦ Relevant and irrelevant texts are initially read similarly, but then reading

style diverges

t 200ms 1s-2s 1s-2s 1s-2s

BEGINNING MIDDLE END reading relevant text

reading irrelevant text

Eye-tracking features

Classification Results Interpretation

© Jacek Gwizdka 54

� Making relevance judgment and reading other parts of texts ◦ EEG signals reflect (mainly) cognitive processes involved in relevance

judgment – plausibly decision making ◦ Eye-tracking data also shows differences in reading relevant passages

vs. all other parts

t 200ms 1s-2s 1s-2s 1s-2s

BEGINNING MIDDLE END

reading relevant text

reading irrelevant text

fixations on relevant words

RELEVANT

EEG features

© Jacek Gwizdka 55

So far: static text stimuli

Can we find similar results on web search?

Can we infer degree of relevance from eye-tracking data collected on web search?

Experiment 3: Eye-tracking + web search

TYPE controlled lab study PARTICIPANTS 32 participants (15 females); 1 search session each SEARCH TASKS 4 tasks at two levels of complexity STUDY DESIGN within-subjects DOC CORPUS Search on Wikipedia using searchtechnologies.com

search engine DATA COLLECTION binary relevance judgments, screen cam, interaction

logs, eye-tracking (Tobii T60) collected using iMotions software and our own YASFIIRE IIR framework

56 © Jacek Gwizdka

SIGIR’2015 “Differences in Eye-tracking Measures Between Visits and Revisits to Relevant and Irrelevant Web Pages“ with masters student Yinglong Zhang

Hypotheses

© Jacek Gwizdka 57

1.  Pupil dilation and eye-tracking measures will differ between relevant and irrelevant pages.

2.  Pupil dilation and eye-tracking measures will differ between first and subsequent visits to Web pages.

3.  Pupil dilation and eye-tracking measures will differ between visits to relevant pages when a page relevance was decided compared to other visits to the same relevant pages, when the pages were not judged as relevant yet.

Eye-tracking Measures

© Jacek Gwizdka 58

Variable DescriptionFixation duration Duration of an eye fixation, in milliseconds

Saccade duration Duration of a saccade, that is of a fast eye movement between eye fixations, in milliseconds

Saccade length Length of a saccade, in pixels

Saccade angle Angle of a saccade relative to the horizontal axis, in degrees

Relative pupil dilation

The relative change in pupil diameter: A difference between pupil size at a time t and the average pupil size for a participant, normalized by that average

Results: Web Page Visit Types

© Jacek Gwizdka 59

# Page and visit types Count

1Irrelevant page

First visit306

215

2 Revisit 91

3Relevant page

First visit697

323

4 Revisit 374

5 Relevant page visit with relevance judgment 68

Comparisons Between 5 Visit Types

© Jacek Gwizdka 60

Variable 1-2 1-3 1-4 1-5 2-3 2-4 2-5 3-4 3-5 4-5

Fixation duration + +

Saccade duration + + + + + +

Saccade length + + + +

Saccade angle

Relative pupil

dilation+ + + + + + + +

Results: pupilometry

© Jacek Gwizdka 61

� Pupils dilated more on visits to relevant pages ◦  this suggests higher mental effort and attention paid to relevant pages

� Pupil dilation did not differentiate between revisits and relevance judgment visits to relevant pages. ◦ However, an increased pupil dilation in the last 2 seconds before the

relevance judgment was made allows to differentiate visits to web pages when relevance judgment was made ◊ increased attention during relevance judgment.

Classification

© Jacek Gwizdka 62

�  Two binary classification models ◦  first visits to relevant pages and visits to relevant pages during which

participants judged relevance ◦  visits to irrelevant pages and visits to relevant pages during which

participants judged relevance ◦ Flexible discriminant analysis (FDA) ◦ Maximize ROC

Classification Results

© Jacek Gwizdka 63

Classification accuracy in experiment 2 with static text stimuli was 0.65-0.72

Variable Model 1 Model 2Fixation duration 29.38 29.66Saccade duration 81.27 59.94Saccade length 46.70 0.00Saccade angle 0.00 54.94

Relative pupil dilation 100.00 100.00

Model Accuracy Sensitivity SpecificityModel 1 0.57 0.57 0.57

Model 2 0.61 0.57 0.62

On-going Projects

64

� EEG and Eye-tracking to infer relevant words (PCDE) ◦  PCDE: Personalized Complex Data ExploraKon (with Lockheed MarKn and U Maryland) ◦ Hypothesis: reading of relevant vs. irrelevant words can be detected by EEG in combinaKon with eye-‐tracking: EFRPs – Eye-‐fixaKon related potenKals

�  Learning about Search as Learning (LaSaL) � Detecting Periods of Mindless Information Seeking (DeMIS) ◦  employing eye-tracking methodology

© Jacek Gwizdka

Challenges in using NP methods

© Jacek Gwizdka 66

�  Lack of standardized metrics �  Lack of standardized tasks �  Lack of baseline NP data

� Some NP methods intrusive ◦  unnatural way to perform tasks, if possible at all (e.g., fMRI)

� Highly specialized expertise required � Equipment settings, algorithms

� But … it is still worth doing !

Mapping Search Variables on NP Responses

© Jacek Gwizdka 67

Mostafa J., & Gwizdka J., (in press) “Deepening the role of the User: Neuro-Physiological Evidence as a basis for Studying and Improving Search” to appear in CHIIR 2016

68

Acknowledgements Funding: IMLS National Leadership Research CAREER Award

Google Faculty Award Rutgers University (fMRI) Lockheed Martin Corporation iSchool at UT Austin - fellowship

Collaboration: Profs. Paul Kantor and Steve Hanson with Dr. Catherine Hanson (fMRI – RUBIC/RU) Dr. Shouyi Wang (UT Arlington) (EEG data analysis) PhD candidate Michael Cole (now Dr.) Masters student Yinglong Zhang (now PhD student)

© Jacek Gwizdka

Thank You

Questions?

More info & contact: [email protected] www.gwizdka.com/research

www.neuroinfoscience.org | www.neuroir.org

Neuro-Physiological Evidence as a Basis for Studying Search

Education

Transcript of Neuro-Physiological Evidence as a Basis for Studying Search