Slide 1people.csail.mit.edu/teevan/work/publications/talks/sigird… · PPT file · Web viewThe...
37
Jaime Teevan MIT, CSAIL The Re:Search Engine
Transcript of Slide 1people.csail.mit.edu/teevan/work/publications/talks/sigird… · PPT file · Web viewThe...
Jaime TeevanMIT, CSAIL
The
Re:Search Engine
“Pick a card, any card.”
Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
Your Card is GONE!
People Forget a Lot
Change Blindness
http://www.usd.edu/psyc301/ChangeBlindness.htm
Change Blindness
http://www.usd.edu/psyc301/ChangeBlindness.htm
Re:Search Engine
?
Merge Old and New ResultsOld
New
Merged
We still need magic!
Overview
♠ Memorability study♠ Recognition study♠ Assumptions♠ Implementation issues♠ Evaluation issues♠ Choose your own adventure
Memorability Study♠ Participants issued
self-selected query♠ After an hour, asked
to fill out a survey♠ 129 people
remembered something
Data Analysis
Probability of being remembered♥ Anything? # of words? # of fields?♥ Features
♣Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc.
♣Query features: query type, query length, # of search in session, elapsed time, etc.
♠ Remembered rank v. real rank♥ Map remembered rank to real rank
“Memorability”
00.10.20.30.40.50.60.70.8
1 2 3 4 5 6 7 8 9 10
Rank - R
P(R
emem
|R,C
)
Clicked - C Not clicked
Remembered Results Ranked High
-2
0
2
4
6
8
10
12
-2 0 2 4 6 8 10 12
Remembered Rank
Act
ual R
ank
Recognition Study
♠ Same set-up as Memorability Study♠ Follow-up survey: Results the same?
♥ Case 1: Old results♥ Case 2: New results♥ Case 3: Clicked to top♥ Case 4: Intelligent merging
♠ 92 people have completed both steps
16%
74%
65%
17%
Assumptions♠ Re-search v. search♠ Memorable v. relevant♠ Results change v. stay the same♠ Hide change v. show change♠ Forget v. remember as forgettable♠ Merge v. identify old or new
Why? How to test? What if I’m wrong?
Implementation Issues
♠ Page of cached result may disappear♠ Multiple result pages♠ Identifying repeat queries
♥ User identified♥ Search sessions are not repeat queries♥ Exact query may be forgotten
Evaluation Issues
♠ Various goals to test♥ Does a merged list look like the original?♥ Does merging make re-finding easier?♥ Is search improved overall?
♠ Lab study♥ How to set up re-finding task?♥ Timing differences significant enough?
♠ Longitudinal study – What to measure?♠ What are good baselines?
Choose Your Own Adventure♠ Re-search v. search♠ Memorable v. relevant♠ Results change v. stay the same♠ Hide change v. show change♠ Forget v. remember as forgettable♠ Merge v. identify old or new
♠ Implementation issues♠ Evaluation issues
Choose Your Own Adventure♠ Re-search v. search♠ Memorable v. relevant♠ Results change v. stay the same ♠ Hide change v. show change♠ Forget v. remember as forgettable♠ Merge v. identify old or new
♠ Implementation issues♠ Evaluation issues (Done)
Hide Change v. Show Change
♠ Why I think change should be hidden♥ Example: dynamic menus
♠ How to prove♥ New results better, called the same or worse♥ Baseline for testing – 2 lists, change explicit
♠ What if we should show change?♥ Memorability suggests changes to highlight♥ Other applications where want to hide change
(Done)
Memorable v. Relevant
♠ Why I think memorability is important♥ Relevance at a future date is what matters♥ Necessary to hide change
♠ How to prove♥ Baseline for lab study with target first
♠ What if relevance is what’s important?♥ Mapping between memorable and relevant♥ Useful related work on implicit feedback
(Done)
Re-search v. Search
♠ Why I think people repeat searches♥ Information seeking literature♥ Re-finding consistently reported as a problem
♠ How to prove♥ Study shows prefer to follow known paths♥ Search log analysis
♠ What if people just want to search?♥ Memorable results ranked first♥ Other domains where list consistency matters
(Done)
Merge v. Identify Old and New
♠ Why I think results should be merged♥ Information need not necessarily one or other♥ People don’t like to do extra work
♠ How to prove♥ Search log analysis♥ Look at what people do in longitudinal study♥ Lab study – timing becomes an issue
♠ What if people want to identify query type?♥ Other applications where merging is useful
(Done)
Results Change v. Stay the Same
♠ Why I think results change♥ How search engines work♥ Personalization and dynamic content
♠ How to prove♥ Track query results
♠ What if results don’t change?♥ Probably will in future applications♥ Existing applications where lists change
(Done)
Forget v. Remember as Forgettable
♠ Why I think people forget♥ Visual analogy
♠ How to prove♥ Lab study – Do people find new information?♥ Longitudinal study – Ever click on new result?
♠ What if remember as forgettable?♥ Build better model of memorability♥ Highlight important changes
(Done)
Implementation Issues
♠ Page of cached result may disappear♠ Multiple result pages♠ Identifying repeat queries
♥ User identified♥ Search sessions are not repeat queries♥ Exact query may be forgotten
(Done)
Evaluation Issues
♠ Various goals to test♥ Does a merged list look like the original?♥ Does merging make re-finding easier?♥ Is search improved overall?
♠ Lab study♥ How to set up re-finding task?♥ Timing differences significant enough?
♠ Longitudinal study – What to measure?♠ What are good baselines?
(Done)
Jaime [email protected]
Teleporting
Orienteering
Strategies for Finding
Why Do People Orienteer?
♠ Easier than saying what you want♠ You know where you are♠ You know what you find
♠ The tools don’t work
All must be the same to re-find the information!
Structural Consistency Important
New name
Absolute Consistency Unnecessary
New name
Focus on search result lists
Query Changes
♠ Most changes are simple♥ Capitalization♥ Phrasing♥ Word ordering♥ Word form♥ New queries shorter
♠ What about longer time horizons?♠ Recognition v. recall
Result List Changes
♠ Tracked 10 queries on Google for a year+♠ 1.18 of top 10 disappear each week♠ Rate of change likely to increase, e.g.:
♥ Raw personalization♥ Relevance feedback
♠ People forget their queries♥ 28% of queries forgotten within an hour
Example: “neon signs”
0
2
4
6
8
10
12
14
