Managed Search Jacob Graves, Principal Engineer at Getty Images jacob.graves@gettyimages.com

Introduction Getty Images is the global leader in visual communications with over 170 million assets available through its premium content site www.gettyimages.com and its leading stock content site www.istock.com. With its advanced search and image recognition technology, Getty Images serves business customers in more than 100 countries and is the first place creative and media professionals turn to discover, purchase and manage images and other digital content. Its award-winning photographers and content creators help customers produce inspiring work which appears every day in the world’s most influential newspapers, magazines, advertising campaigns, films, television programs, books and online media.

Getty Search Obviously, in order to buy images you have to be able to find them. Search Process: •  Receive search containing words. •  Tokenize and map the words onto our controlled vocabulary keywords •  Find all the images associated with the correct keywords. •  Score all the images and then sort them by the score.

The scoring determines which images the users see.

Managed Search The details of how the scoring takes place is a technical concern, but the end result is a business concern. Goal – make business users self sufficient. So the problem is to create a framework for business users that will: •  Hide technical complexity. •  Allows Control over scoring components and result ordering. •  Allows Balancing of these scoring components against each other. •  Provides Feedback. •  Allows Visualization of the results of their changes.

We call this Managed Search.

Managed Search – Our Implementation 1.  We created a SOLR search ecosystem containing all our images, keywords and

associated metadata, and added plugins using Java. 2.  We used a C# middle tier to wrap around our SOLR ecosystem. 3.  We built a web application called SAW – Search Administration Workbench, using

the Java Play framework and lots of javascript.

Managed Search Architecture Diagram SOLR

Index

Custom  functions  

(valuesources)

Price  Tier  Shuffle  

(RankQuery)

Search  Middle  Tier

SAW

Site

Business  User

Customer

Algorithm  DB

Search  and  algorithm  

Search  results

Load  algorithm  settings

search

Search  results

Save  algorithm  settingsSOLR  select  url

Search  results(for  site  searches)

with  debug  scores

settings

SAW SAW has 5 main areas: •  Algorithm – control sort scoring. •  Preview – see search results. •  Single Page Charts – single search score component charts. •  Scale report charts – all searches score component charts. •  Live tests – expose test algorithms to live users to gather and view KPI data.

Scoring Breakdown To help the business control the scoring we break it down into 3 different scoring components: •  Relevancy – image attributes that are relative to the search (i.e. keywords). •  Recency •  Image Source – image attributes that are not related to the specific search.

Then we provide 2 types of parameter the user can control: •  Internal parameters - to control how the component is calculated. •  External boosts - to control how the components are weighted against each other

SAW Scoring Boosts Screenshot

Scoring Architecture •  In order to allow immediate feedback we have to implement scoring using query

time boosting. •  Use boost functions as they are cleaner. •  Favor Query time over Index time, to prioritize control over small performance

gains. •  Define minimum performance metrics and ensure that we stay within them. Initially we had concentrated on performance above all else and had ended up with inflexible scoring in return for fairly minor performance gains. We used the Valuesource plugin to create our own boost functions.

Relevancy •  The most important component, how confident are we that this image is correct? •  We measure relevancy at the image/keyword level by tracking user interactions. •  After experimenting we settled on a form of the standard tf-idf VSM (Vector Space

Model) and expose a normalization parameter. •  We also expose a boost so they can control the strength of relevancy relative to

other factors

Recency •  Recency is the age of the images. •  Newer images get a higher score to prevent staleness. •  Aging curve – the way an images recency score changes with age. •  We expose 3 different aging curves (reciprocal, linear and reversed reciprocal) and

appropriate parameters to control the shape of the curve. •  We also expose a boost so they can control the strength of recency relative to

other factors

SAW Recency Screenshot

Image Source •  We have a variety of image level attribute data that should affect the sort order,

mostly to do with how likely we think the image is to be of high quality. •  We separate our images into groups based on these attributes, called the source. •  We expose a boost that allows the users to increase the score of images with a

given source. •  Unlike relevancy, this is an image level, not image/keyword level property, so it

doesn’t vary from one search to the next. •  Because it isn’t context specific it is dangerous to make this boost too large.

Custom Shuffle As well as influencing the scoring, the business wants to have control over the order the images where displayed in, so that instead of just appearing in score order certain slots on the page can be allocated to particular classes of image. This is to ensure that we always show a diverse range of images. To accommodate this we need to be able to apply a custom shuffle, similar to a sort but with more control. To accomplish this we take advantage of a new SOLR plugin (new in 4.9) called the RankQuery plugin.

Image Tier Shuffle We classify our images into separate groups or image tiers based on various image level attributes, e.g. •  Licensing Structure •  Image partner •  Exclusivity •  Etc. We distill these factors into a single image property that we assigned at index time. We generate a mapping of result slots to image tiers, e.g. •  slot 1 => image tier 2 •  slot 2 => image tier 4 •  etc. We pass in the mapping at query time and used the RankQuery implementation to shuffle the query results.

Preview page •  Search and get results scored using algorithm settings. •  Display in pages of 100 images. •  Show image score breakdown by component. •  Show image tier. To calculate the score for each component we run the SOLR query in debug mode, and parse the results with regex expressions to get the score for each component. This is the least stable piece of the whole application, as debug syntax can change quite frequently between SOLR releases. However, it’s also pretty easy to fix.

Preview Screenshot

Single Page Charts This allows the users to verify what they think they are seeing visually with numbers. •  Aggregate the component scoring data across all the 100 images on a page. •  Create interactive charts from the data. •  Charts that display the distribution of each score component. •  Chart that displays the comparative score from each component. •  Chart that shows the custom shuffle distribution. We use the javascript D3.js library to generate the graphs.

Single Page Charts Screenshot

Scale Reports This allows the users to validate their settings across the full spectrum of searches that users execute at Getty. •  Execute 1000 different searches (throttled). •  Use the first 100 images from each search by default, number can be increased up

to 10000 (slower). •  Aggregate the component scoring data across all the results. •  Create and display charts similar to the ones used in the single page charts view. To generate the list of 1000 searches we use proportional sampling, from search log data.

Scale Reports Screenshot

Live tests Once the users are happy with an algorithm the next stage is to test it for real. To do this we have a page that controls: •  The algorithm settings for the various live and test sorts. •  Saving these settings to a database where they are used to generate production

SOLR queries. •  The percentage of users for a given live sort that will be allocated to a test sort.

KPI monitoring We also have a page that displays the user interaction data. •  Displays actions against our KPI’s (Key Performance Indicators). •  Primarily we use click-through (i.e. user clicks on an image in the search results). •  Broken out by time and by sort so we can compare the test algorithms against the

live ones. •  We get this data in a feed from our existing analytics framework.

Conclusion Self sufficient business user, path to changing sort order: 1.  Change algorithm settings. 2.  Execute searches and evaluate sort order visually. 3.  Use single page charts to confirm visual impressions. 4.  Use scale report to confirm behavior across proportional set of searches. 5.  Set a test algorithm to have the settings you want. 6.  Set a percentage of users to experience the test. 7.  Monitor KPI’s over time to see if settings work as intended. 8.  Set the live algorithm to have the settings you want.

ValueSource Plugins This is a well-established SOLR plugin for adding custom query functions.

http://wiki.apache.org/solr/SolrPlugins#ValueSourceParser There are 3 parts: •  Implement ValueSource. This is where the actual logic is implemented. It can take in

either simple datatypes (like Strings or floats) or other ValueSource objects (e.g. an asset field value or another query function).

•  Implement ValueSourceParser. This creates the ValueSource object with appropriate inputs.

•  Solrconfig.xml. Add a line to enable the new ValueSource plugin. You can look at any of the existing Query function implementations to see how they should work. e.g. – for the “Map” query function see: •  org.apache.solr.search.ValueSourceParser •  org.apache.lucene.queries.function.valuesource.RangeMapFloatFunction You can also change the debug output so that we can see the results of each custom function in debug mode, this allows us to display the individual score components to the users.

RankQuery Implementation This is a new plugin in SOLR 4.9, created by Joel Bernstein.

https://issues.apache.org/jira/browse/SOLR-5973 There is a test in the SOLR 4.9 tests that shows a good example implementation:

org.apache.solr.search.TestRankQueryPlugin Very briefly, you have to implement: •  QParserPlugin, it creates and returns the QParser implementation. •  QParser, it creates and returns the RankQuery implementation. •  RankQuery, it creates and returns the TopDocsCollector and MergeStrategy implementations. •  TopDocsCollector, this returns the top documents from each shard that you wish to include in your final

results. In our case we separate the documents into separate priority queues by image tier, and order by score within each image tier. Then we go through a pre-determined list of which image tier should occupy each slot, and pull the next item from the appropriate image tier priority queue to generate the top documents List.

•  MergeStrategy, this combines the top documents generated by the TopDocsCollectors on each shard. In our case we followed the same logic as we had for each individual shard, assigning documents to priority queues by image tier in score order, and then assigning queues to pre-determined slots.

Lastly you reference the new QParserPlugin in your solrconfig.xml. The pre-determined list of image tier slots could either be a user configurable parameter that is passed in or it could just included in the solrconfig.xml, or even hard coded.

Q & A Please contact me if you have any questions or thoughts. I will be attending till the end of the conference. Email – jacob.graves@gettyimages.com