ECIR – a Lightweight Approach for Entity-centric Information Retrieval

ECIR – a Lightweight Approach for Entity-centric Information Retrieval

TREC 2010

HPI Potsdam, SAP Research Center Dresden

Barczynski, Brauer, Emde, Hold, Leben, Thiele, Naumann

SAP ResearchHPI ECIR seminar

Who we are

Benjamin Michael Christoph Alexander Wojciech Falk

Emde Leben Thiele Hold Barczynski Brauer

Prof. Dr. Felix

Naumann

Information Systems chair

at Hasso-Plattner-Institut

2

Motivation:Entity-centric Search in Enterprises

■ Companies cannot index the whole web:

□ have to leverage resources on the web (search engines)

□ extract/ rank entities at runtime (limited hardware resources)

■ But there is business value in discovering related entities, e.g., to determine competing companies:

□ Source: SAP

□ Target: organization

□ Narrative: Find competitors developing ERP software.

■ Our system evaluates the capabilities of three different search engines to answer queries for the Related Entity Finding (REF)- task.

3

TREC2010-ECIR HPI/SAP | HPI Information Systems | SAP Research | Nov 19, 2010

Processing Pipeline

4

TopicsTopics QueryRewriting

QueryRewriting

FreebasePOS Tagger

DocumentRetrieval

DocumentRetrieval

Search Engine

Rule Generator

Rule Generator

Target Entitiy Extraction


Deduplication & Ranking


HomepageRetrieval

HomepageRetrievalResultsResults Clueweb

MappingCluewebMapping


Processing Pipeline

5


QueryRewriting

FreebasePOS Tagger

DocumentRetrieval

DocumentRetrieval

Search Engine

Rule Generator

Rule Generator





HomepageRetrieval




Query Rewriting -Key word query generation

6<entity_name>Costco</entity_name><entity_URL>cw09 − en0006 − 60 − 20817</entity_URL><target_entity>organization</target_entity><narrative>Find homepages of manufacturers of LCD televisions sold by Costco.</narrative>

~sold ~homepages ~manufacturers ~LCD ~television (“Costco” OR “Costco Wholesale” OR … )~sold ~homepages ~manufacturers ~LCD ~television (“Costco” OR “Costco Wholesale” OR … )


Query Rewriting-Synonym Retrieval

7


get alternative names from Freebase and take the most popular (rank using search engines hit count)


<entity_name>Costco</entity_name><entity_URL>cw09 − en0006 − 60 − 20817</entity_URL><target_entity>organization</target_entity><narrative>Find homepages of manufacturers of LCD televisions sold by Costco.</narrative>

Query Rewriting -Filtering using POS-tags

8


Find/VB homepages/NNS of/IN manufacturers/NNS of/IN LCD/NNP televisions/NNS sold/VBN by/IN Costco/NNP


<entity_name>Costco</entity_name><entity_URL>cw09 − en0006 − 60 − 20817</entity_URL><target_entity>organization</target_entity><narrative>Find homepages of manufacturers of LCD televisions sold by Costco.</narrative>

Processing Pipeline

9


QueryRewriting

FreebasePOS Tagger

DocumentRetrieval

DocumentRetrieval

Search Engine

Rule Generator

Rule Generator





HomepageRetrieval




■ Source entitiy and its alternative names:

■ Mapping a topic’s target type to predefined ThingFinder-types

■ Stemmed noun and verb tokens from a topic’s narrative

■ Combined rule to extract candidates

Extraction Rule Constructionfor SAP Business Objects ThingFinder

10

#group Candidate (scope="Paragraph"): [UL]%(TargetType),%(SourceName),(%(Context))[/UL]

#group Candidate (scope="Paragraph"): [UL]%(TargetType),%(SourceName),(%(Context))[/UL]

#group SourceName (scope="Sentence"):(<Costco>|<Costco><travel>|<Costco><Wholesale><Corp.>|..)#group SourceName (scope="Sentence"):(<Costco>|<Costco><travel>|<Costco><Wholesale><Corp.>|..)

#group TargetType (scope="Sentence"): [TE ORGANIZATION]<>+[/TE]

#group TargetType (scope="Sentence"): [TE ORGANIZATION]<>+[/TE]

#group Context (scope="Sentence"): (<STEM:sold>|<STEM:LCD>|<STEM:manufacturer>|..)

#group Context (scope="Sentence"): (<STEM:sold>|<STEM:LCD>|<STEM:manufacturer>|..)


Mischka

Vielleicht ein kurzer Überblick zu BOBJ-TA.Custom setup vs. spezialisierte Regeln- alternative names: ist klar- Thingfinder type: was heißt "TE" - der thingfindertype ist "Organization"- N/V tokens: wer macht das stemming ? POS-Tagger output?? TF macht das.Kann man die Regeln mit RegEx vergleichen?

Mischka

Prof Naumann wünscht sich die Bulletpoints hervorgehobener und die Originalsyntax eher im Hintergrund

EXTRACTED PARAGRAPH:EXTRACTED PARAGRAPH:

Example Candidate Rule Firing

SENTENCE#1: Update, October 2006: Prices have dropped everywhere in the flat panel TV market, even among top-tier manufacturers like Panasonic, Sharp and Sony.

SENTENCE#2: While Costco offers a 50" Vizio HDTV LCD for around $2,000, top-notch sets from Panasonic are selling for as low as $2,400 at authorized internet dealers—that factors out to a difference of only $40 a year if you consider the 10-year lifespan of the Panasonic Plasma TV.

SENTENCE#3: From what we've seen of the build quality in the "bargain-basement" models by Vizio, Maxent, and Envision, you'll be lucky if your discount 50" televisions lasts half that long.

11

Source Entity Context Token Potential Target Entitiy


michael.leben

Mischka

Naumann: kürzer.War sehr ausführlich erklärt, obwohl auf die Einzelnen Probleme (duplicates, entity ranking) später noch ausführlich eingegangen wird

Processing Pipeline

12


QueryRewriting

FreebasePOS Tagger

DocumentRetrieval

DocumentRetrieval

Search Engine

Rule Generator

Rule Generator





HomepageRetrieval




Entity Deduplication and Ranking per document

■ Deduplication combines Jaro-Winkler and Jaccard Similarity.

■ local ranking (duplicate group per document):

□ distance in sentences between Source and Target Entity

□ normalized such that score for distance = 0 is score = 1

□ per duplicate group consider only maximum score per document

■ rank 1: Vizio, Panasonic (sentence distance 0)

■ rank 2: Sharp, Sony, Maxent, Envision (sentence dinstance 1)

13

…even among top-tier manufacturers like Panasonic, Sharp and Sony.

While Costco offers a 50" Vizio HDTV LCD … from Panasonic are selling … consider the 10-year

lifespan of the Panasonic Plasma TV.

From what we've seen … by Vizio, Maxent, and Envision, you'll be lucky …





Entity Deduplication and Ranking across documents

■ global ranking (for duplicate group in document corpus)

□ aggregated local scores for duplicate groups

□ considers rank position of documents

□ Target Entities extracted from higher ranked documents are preferred

14

… stores Costco Wholesale and Sam's Clubs.

In addition, the quality of Vizio LCDs looks very similar to the Sonys and Samsungs on …

… stores Costco Wholesale and Sam's Clubs.

In addition, the quality of Vizio LCDs looks very similar to the Sonys and Samsungs on …

document b; search engine rank 1

document a; search engine rank 5









Processing Pipeline

15


QueryRewriting

FreebasePOS Tagger

DocumentRetrieval

DocumentRetrieval

Search Engine

Rule Generator

Rule Generator





HomepageRetrieval




Homepage Candidate Retrieval

16

expanded query:~sold ~homepages ~manufactures ~LCD ~television (“Costco” OR “Costco Wholesale” OR … )

expanded query:~sold ~homepages ~manufactures ~LCD ~television (“Costco” OR “Costco Wholesale” OR … )

Topic:<entity_name>Costco</entity_name><entity_URL>cw09 − en0006 − 60 − 20817</entity_URL><target_entity>organization</target_entity><narrative>Find homepages of manufacturers of LCD televisions sold by Costco.</narrative>

Target Entities:Panasonic, Sharp, Sony, Vizio, Maxent, EnvisionTarget Entities:Panasonic, Sharp, Sony, Vizio, Maxent, Envision

queries to the selected search enginequeries to the selected search engine


Homepage Candidate Retrieval

17

homepage candidates set:search engine resultsWikipedia outgoing linksshortened URLs

homepage candidates set:search engine resultsWikipedia outgoing linksshortened URLs


queries to the selected search engine:http://www.google.com/search?q=viziohttp://www.google.com/search?q=feature:homepage+viziohttp://www.google.com/search?q=allintitle:viziohttp://www.google.com/search?q=allinanchor:Find+homepages+of+manufacturers...

queries to the selected search engine:http://www.google.com/search?q=viziohttp://www.google.com/search?q=feature:homepage+viziohttp://www.google.com/search?q=allintitle:viziohttp://www.google.com/search?q=allinanchor:Find+homepages+of+manufacturers...

Homepage Ranking | source-specific

aggregate scores of candidates from multiple sources

extract vector of 17 features for each candidate:

5 source-specific flags (f1 to f5)

12 text-based page features (f6 to f17)

18homepage candidates sethomepage candidates set

search engine resultshttp://www.vizio.com/warranties-installation/http://www.vizio.com/discover/via/

search engine resultshttp://www.vizio.com/warranties-installation/http://www.vizio.com/discover/via/

Wikipedia outgoing Linkshttp://www.youreviewelectronics.com/vizio-reviews/http://www.vizio.com/

Wikipedia outgoing Linkshttp://www.youreviewelectronics.com/vizio-reviews/http://www.vizio.com/

shortened URLshttp://www.vizio.com/http://www.vizio.com/discover/http://wiki.answers.com/Qhttp://www.youreviewelectronics.com/vizio-reviews

shortened URLshttp://www.vizio.com/http://www.vizio.com/discover/http://wiki.answers.com/Qhttp://www.youreviewelectronics.com/vizio-reviews

allintitleoperatorallintitleoperator

related to Wikipedia pagehttp://www.vizio.com/productshttp://wiki.answers.com/Q/Who_manufactures_Vizio_TV

related to Wikipedia pagehttp://www.vizio.com/productshttp://wiki.answers.com/Q/Who_manufactures_Vizio_TV

allinanchoroperatorallinanchoroperator

f1 f2

f3feature:homepage operator

http://www.vizio.com/products

feature:homepage operator

http://www.vizio.com/products

f4

f5


Mischka

feature:homepage nur für Yahoo: sites mit "~"

Homepage Ranking | configurable

features:19


feature values are multiplied with configurable weights

finding best weight configuration:

depends on search engine

genetic algorithm for training

Evaluation | TREC2010 nDCG values

B = Bing, G = Google, Y = Yahoo; 64/16: number of documents retrieved for entity extraction

TREC2010-ECIR HPI/SAP | HPI information systems | SAP research | Nov 09, 2010

20

Evaluation | TREC2010 nDCG values

B = Bing, G = Google, Y = Yahoo; 64/16: number of documents retrieved for entity extraction

nDCG aggregated over all topics

TREC2010-ECIR HPI/SAP | HPI information systems | SAP research | Nov 09, 2010

21

Evaluation 2009 | Target Entity Types


22 all runs averaged score for TREC2009 topics

performance for Organization and Persons better than for Products

Results from TREC 2010


23 avg nDCG around 0.8

Results from TREC 2010 | bugfixed


24 avg nDCG around 1.15

ECIR – a Lightweight Approach for Entity-centric Information Retrieval

Documents

Transcript of ECIR – a Lightweight Approach for Entity-centric Information Retrieval