Enhancing Search with Predictive Analytics

Text Analytics World – Boston 2013

Andrew Fast Chief Scientist

Elder Research, Inc. fast@datamininglab.com

•  “It is difficult to describe, but you know it when you see it.” –  Lord Justice Stuart Smith,

Cadogan Estates Limited v. Morris (1998)

•  Likewise, most textual concepts cannot be easily defined with a single keyword query

The Elephant Test

Quote  from:  h,p://www.bailii.org/ew/cases/EWCA/Civ/1998/1671.html  

•  Search and Predictive modeling each provides a different trade-off between power and generality.

Combining Search and Predictive Models

Keyword  queries  can  answer  any  query,  but  with  limited  depth  for  complex  queries.  

Document Classification

Generality

Pow

er

Keyword Search

A  predicIve  model  can  answer  one  query  well,  especially  a  complex  query  

Our Approach •  A “search ensemble” ranking function that

“boosts” keyword relevance based on a predictive model

High  Keyword  Relevance,  High  Model  Ranking  

Model  Ranking  

Keyw

ord  Re

levance  

High  Keyword  Relevance,  Low  Model  Ranking  

Low  Keyword  Relevance,  Low  Model  Ranking  

Low  Keyword  Relevance,  High  Model  Ranking  

The Problem •  The Goal: Explore NEW interesting ideas using

OLD social entrepreneurship contest entries

•  The Data: A collection of contest entries from 19 different contests sponsored by our client –  Contests cover a range of topics such as health,

education, literacy, finance, technology, and geo-tourism.

•  The Challenge: Emphasize high-quality entries in the results as entry quality varies widely

Combining Search and Predictive Models •  Keyword ranking does not help you find high-

quality entries … •  … but Model Ranking is not topic centric.

•  Complimentary strengths –  Search for exploration and discovery –  Predictive Models for trends and correlations

THE MODEL

Target Variable •  Identify characteristics of past entries that are

correlated with that proposal being ‘Shortlisted’ by the Contest Judges

•  Rankings: 1 – Likely Finalist 2 – Top Tier 3 – Honorable Mention 4 – Passed Screening 5 – No

•  Note: Not every contest used all 5 rankings

‘Shortlisted’  

The Inputs •  Learn a logistic regression model to fit the feature

weights

•  Inputs:

Structured  Data  

Taxonomy   Textual  Features  

•  Budget  Size  •  Maturity  •  Impact  

•  Auto-­‐tagging  taxonomy  terms  

•  Length  •  Lexical  

Diversity  

•  Joint work with Beth Maser and Richard Iams at PPC

•  Non-traditional, general approach –  Broad, flexible taxonomy

•  Focus on the range of interests of the organization

The Taxonomy

Using the Taxonomy •  Each contest emphasizes different branches of

the taxonomy –  Taxonomy features need to be contest specific

•  Step 1: Use the “Wisdom of Crowds” to find the center of each contest

•  Step 2: Rate each entry based on the distance from the center

Evaluation: Area Under the ROC •  Evaluate the overall ranking provided by the model.

–  Higher means more ‘Shortlisted’ entries at the top of the list

Evaluation: Lift •  Evaluates the improvement using the model at a

fixed amount of work –  How much more efficient are the judges using our

model alone?

•  Every contest showed positive lift. –  Maximum lift of 3.3 –  Average lift of 1.67

THE SEARCH APPROACH

Our Approach •  A new search ranking function that “boosts”

keyword relevance for probable shortlisted entries

High  Keyword  Relevance,  High  Model  Ranking  

Model  Ranking  

Keyw

ord  Re

levance  

High  Keyword  Relevance,  Low  Model  Ranking  

Low  Keyword  Relevance,  Low  Model  Ranking  

Low  Keyword  Relevance,  High  Model  Ranking  

The Prototype Platform

ERI  Text  Mining  

Model    (PredicIve  +  Taxonomy)  

Search  Index  

Custom  Search  Interface  

Data  

Faceted Search with Solr

Apache Solr is an open-source faceted search engine (http://lucene.apache.org/solr)

•  Text mining can be viewed from many different perspectives

•  No single view provides a complete solution

•  Must consider the

entire “beast” to get the best solution

“Blind Men and the Elephant”

19

Contact Information

Andrew Fast, Ph.D. Chief Scientist

fast@datamininglab.com

(434) 973-7673 www.datamininglab.com

Practical Text Mining •  Winner of the 2012

PROSE award for Computing and Information Science

•  Written for a technical audience seeking more text experience

•  Includes trial versions of major software tools

21  

Andrew Fast"Chief Scientist, Elder Research, Inc.

Dr. Fast graduated Magna Cum Laude from Bethel University and earned Master’s and Ph.D. degrees in Computer Science from the University of Massachusetts Amherst. There, his research focused on causal data mining and mining complex relational data such as social networks. At ERI, Andrew leads the development of new tools and algorithms for data and text mining for applications of capabilities assessment, fraud detection, and national security. Dr. Fast has published on an array of applications including detecting securities fraud using the social network among brokers, and understanding the structure of criminal and violent groups. Other publications cover modeling peer-to-peer music file sharing networks, understanding how collective classification works, and predicting playoff success of NFL head coaches (work featured on ESPN.com). With John Elder and other co-authors, Andrew has written a book on Practical Text Mining, that was awarded the prose Award for Computing and Information Science in 2012.

Dr. Andrew Fast leads research in Text Mining and Social Network Analysis at Elder Research, the nation’s leading data mining consultancy. ERI was founded in 1995 and has offices in Charlottesville VA and Washington DC,(www.datamininglab.com). ERI focuses on Federal, commercial, investment, and security applications of advanced analytics, including stock selection, image recognition, biometrics, process optimization, cross-selling, drug efficacy, credit scoring, risk management, and fraud detection.