CSE 8337 SPRING 2011 PROJECT 3

Richa Arora

Agenda Tool Identified and Overview Schema.xml Tokenization, Stop words, and Synonym Handling Indexing Data Import Handler Query format and Matching documents to query Function Queries Bibliography

TOOL IDENTIFIED & OVERVIEW

Tool Identified & Overview SOLR - Open Source enterprise search platform from

Apache Lucene project Purpose

◦ To implement a full text search functionality in a web application

Commercial Websites using SOLR◦ www.digg.com◦ http://www.whitehouse.gov/ - Uses SOLR via Drupal for site

search w/highlighting & faceting◦ http://beta.fcc.gov/ ◦ http://www.netflix.com/

SOLR ApplicationWeb server Database server

Web Applicati

on

SOLR

Document

Database

Features and Technology Features

◦ Full text search◦ Rich document handling (including MS Word, PDF, RTF etc.)◦ HTML administration interface◦ Scalable

Technology◦ Java programming language◦ Lucene Java search library◦ Runs as a search server within a servlet container such as

Tomcat or Jetty

Functioning of SOLR

DocumentsBrowser based web

interface

Solr Server

Documents for indexing

Search Queries Search Results

Indexing Searching

Index

schema.xml

solrconfig.xml

Operations in SOLR Documents form the basic unit of SOLR Documents are composed of fields Examples:

◦ Document for Person: Fields – name, height, age, etc.◦ Document for Recipes: Fields – origin, ingredients, etc.

Documents are fed to SOLR SOLR extracts the information from the fields in the documents and

makes it searchable Steps:

◦ Field Analysis◦ Tokenization◦ Filter application◦ Indexing

SCHEMA.XML

schema.xml Governs how should SOLR build indexes from input

documents Defines field types and specific fields that the

documents can contain Describes how SOLR should handle the fields when

adding documents to the index or when querying those fields

Elements of schema.xml<schema>

<types><fields><uniqueKey><defaultSearchField><solrQueryParser defaultOperator><copyField>

</schema>

Analyzers These are used for examining the text of fields and to generate a token

stream Indexing Analyzers: The results of the analysis are added to an index and

a set of terms like positions, sizes, etc for a field are defined Querying Analyzers: The values being searched for are analyzed and the

terms that result are matched against those that are stored in the field's index

<fieldType name=“nametext” class=“solr.TextField”><analyzer type=“index”><tokenizer class=“solr.StandardTokenizerFactory”/><filter class=“solr.LowerCaseFilterFactory”/><filter class=“solr.KeepWordFilterFactory” words=“keepwords.txt”/><filter class=“solr.SynonymFilterFactory” synonyms=“syns.txt”/></analyzer><analyzer type=“query”><tokenizer class=“solr.StandardTokenizerFactory”/><filter class=“solr.LowerCaseFilterFactory”/></analyzer>

</fieldType>

TOKENIZATION, STOP WORDS, AND SYNONYM

HANDLING

Tokenization To splits a stream of text into tokens Tokens are subsequences of the characters A token contains various metadata in addition to its text value, such as the

location at which the token occurs in the field<fieldType name="text" class="solr.TextField">

<analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> </analyzer>

</fieldType> Example

◦ Standard Tokenizer: Treats whitespace and punctuation as delimiters Input: “Email: johndoe@xyz.com” Output: “Email:”, “johndoe@xyz.com”

◦ N-Gram Tokenizer: Reads the field text and generates n-gram tokens of sizes in the given range (default minimum is 1 and maximum is 2) Input: “hello world” Output: “h”, “e”, “l”, “l”, “o”, “ “, “w”, “o”, “r”, “l”, “d”, “he ”, “el”, “ll”, “lo”, “o “, “wo”, “or”, “rl”, “ld”

Filters Filters take tokens as input from the Tokenizers and

produce another stream of tokens as output Multiple filters can be used one after the other Example:<fieldType name="text" class="solr.TextField">

<analyzer> <tokenizer class="solr.StandardTokenizerFactory"/><filter class="solr.StandardFilterFactory"/> <filter class="solr.LowerCaseFilterFactory"/><filter class="solr.EnglishPorterFilterFactory"/></analyzer>

</fieldType>

Types of Filters

Result of Filter application

Stop Words Handling Stop Filter: This filter is used to discard tokens that are on the

given stop words list. A standard stop words list is included in the SOLR config directory, named stopwords.txt, for English language text

Example: Using the standard stopwords.txt<analyzer>

<tokenizer class="solr.StandardTokenizerFactory"/><filter class="solr.StopFilterFactory" words="stopwords.txt"/>

</analyzer> Tokenizer Input : “welcome to the world of Solr”Tokenizer Output/Filter Input: “welcome”(1), “to”(2),

“the”(3), “world”(4), “of”(5), “Solr”(6)Filter Output: “welcome”(1), “world”(2), “Solr”(3)

stopwords.txt

Synonym Handling Synonym Filter: This is used for finding synonyms at the time of indexing as

well as while querying. Tokens are looked up in the list of synonyms and if a match is found, then the synonyms are put in place of the token

Example: We can define the synonyms in a file (test_synonyms.txt) and use it for comparing the tokens ◦ home, dwelling, house◦ shop => workshop, store◦ teh => the<analyzer> <tokenizer class="solr.StandardTokenizerFactory"/>

<filter class="solr.SynonymFilterFactory" synonyms=“test_synonyms.txt"/>

</analyzer> Tokenizer Input : “teh home shop”Tokenizer Output/Filter Input: “teh”(1), “home”(2), “shop”(3)Filter Output: “the”(1), “workshop”(2), “shop”(2), “home”(2),

“dwelling”(3), “house”(3)

INDEXING

Indexing Refers to adding the content to a SOLR index To make the content searchable Sources of data for indexing:

◦ XML◦ CSV◦ Rich text formats (PDF, MS Word, MS Excel, text etc.)◦ Data extracted from tables in a database

Posting Data to SOLR Uploading Data with SOLR Cell

◦ Using ExtractingRequestHandler◦ With a POST◦ With SOLR Cell and SOLRJ

Uploading Data with Index Handlers◦ XMLUpdateRequestHandler for XML-formatted Data◦ Using the CSVRequestHandler for CSV Content◦ Indexing Using SOLRJ

Uploading Structure Data Store Data with the Data Import Handler

Content Streams

cURL Utility curl posts and retrieves data over HTTP, FTP, and many other protocols In the example below, the Extraction Request Handler is called, uploads the file

tutorial.html and assigns it the unique ID doc1 curl “http://localhost:8983/solr/update/extract?

literal.id=doc1&uprefix=attr_&fmap.content=attr_content&commit=true” -F "myfile=@tutorial.html"

literal.id provides a unique ID to the document uploaded to SOLR

commit=true makes the document searchable after indexing

The -F flag instructs curl to POST data using the Content-Type multipart/form-data and supports the uploading of binary files

The @ symbol instructs curl to upload the attached file

The argument myfile=@tutorial.html needs a valid file path

Example – XMLUpdateRequestHandler

Order of operation:1. Modify the schema.xml file to add the fields which may not be already existing in the schema.xml file, example:

authors, dd, isbn, yearpub, publisher2. Modify the schema.xml file to copy the newly created fields to text field to make the search results viewable3. Run the curl utility with the command for adding XML document:

curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --data-binary "<add><doc><field name='id'>doc26</field><field name='authors'>Patrick Eagar</field><field name='subject'>Sports</field><field name='dd'>796.35</field><field name='isbn'>0002166313</field><field name='yearpub'>1982</field><field name='publisher'>Collins</field></doc><commit waitFlush='false' waitSearcher='false'/></add>"

Uploading Structure Data Store Data with the Data Import Handler Often data is stored in relational databases Data Import Handler (DIH) provides a mechanism to

import data from database and to index it DIH can also index content from RSS and ATOM feeds,

e-mail repositories and structured XML

Configuration Handler to be registered in the solrconfig.xml file<requestHandler name="/dataimport"

class="org.apache.solr.handler.dataimport.DataImportHandler"> <lst name="defaults"> <str

name="config">${solr.config.dir:./solr/conf}/dataimporthandler/data-config.xml</str>

</lst> </requestHandler> There can be multiple configuration files

DIH Example1. Create a database in SQL Server 20052. The tables and the relationships in the database are shown below

DIH Example3. Create an XML file called DIH_Test.xml for importing into SOLR

4. Modify solrconfig.xml file to instruct SOLR to import data as per the file DIH_Test.xml

DIH Example5. Do a full-import of the DIH from the browser

using:

http://localhost:8983/solr/dataimport?command=full-import

DIH Example7. Run queries on the newly

indexed data from the database

8. Example:http://localhost:8983/solr/select?q=ipad2

The above query returns the result. Executing queries on the original database returns similar results

DIH Example – Multiple Datasources

QUERY FORMAT AND MATCHING DOCUMENT TO A

QUERY

Searching in SOLR

Request Handler

Query Parser

IndexResponse Writer

qt: selects a Request Handler for a query using /select

defType: selects a query parser for the query

qf: selects which field to query in the index

start: specifies an offset into the query results where the returned response should begin

rows: specifies the number of rows to be displayed at run time

fq: flters the query by applying an additional query to the initial query’s results; caches the results

wt: selects a response writer for formatting the query response

Advantage - Enables the user to specify very precise queries Disadvantage – Is less tolerant of syntax errors than the DisMax query

parser Parameters Supported

◦ Terms – Use of wild card characters, Fuzzy Searches, Boosts and Ranges◦ Fields – Identified by name followed by a colon◦ Boolean Operators – AND, OR, NOT, &&, !, ||◦ Common query parameters – debugQuery, defType, explainOther, fl, fq,

omitHeader, rows, sort, start, timeAllowed◦ Functions – abs, constant, div, fieldValue, log, linear, max, etc.◦ Faceting ◦ Highlighting ◦ MoreLikeThis (mlt)

Query Syntax and Parsing - The Standard Query Parser

q – Defines a query using standard query syntax. This parameter is mandatory

q.op – Specifies the default operator for query expressions (this parameter’s value is defined in schema.xml). Possible values are “AND” or “OR”

df – Specifies a default field, overriding the definition of a default field in schema.xml

Default parameter values are specified in solrconfig.xml

Standard Query Parser Parameters

Queryhttp://localhost:8983/solr/select?

q=id:6H500F0&popularity=6

Sample Responses - Example

Fuzzy Searches - based on the Levenshtein Distance or Edit Distance

E.g. tight~ will match terms like flight, slight etc. Additional parameter to specify degree of similarity –

tight~0.8 will match sight. When set closer to 1, optional parameter causes only terms with higher similarity to be matched

If numerical parameter is omitted, the default value taken is 0.5

Term Modifiers – To add flexibility and precision

Range Searches◦ Specifies a range(with an

upper and lower bound) of values for a field

◦ Can be inclusive or exclusive of the lower and upper bounds

Query:http://localhost:8983/solr/

select?q=popularity:{5 TO 7}

Term Modifiers – To add flexibility and precision

Parameter DescriptiondefType Query parser to be used (DisMax or Standard

Query Parser)Sort Sorts the response to a query in asc or desc

order based on response’s score or other characteristic

Start Offset into the responses at which solr should begin displaying content

Rows Number of rows of responses displayed at a time

fq Filter query for search results

fl Limits responses to a listed set of fields

Common Query Parameters

Parameter Description

debugQuery Include debugging information

timeAllowed Time allowed for a query to be processed. If time elapses before response is complete are returned, partial information returned

omitHeader Excludes header information from returned results

wt Specifies the response writer

Common Query Parameters

Used to generate a relevancy score using the actual value of one or more numeric fields

Functions available for function queries◦ abs – abs(x); abs(-5)◦ constant - 1.5; _val_:1.5◦ div – div(1,y); div(sum(x,100), max(y,1))◦ linear – linear(x, m, c); linear(x, 2, 4) returns 2*x+4◦ log – log(x); log(sum(x,100))◦ …

Include function query in a SOLR query◦ With a _val_keyword – e.g. _val_:myNumericField◦ Parameter with an explicit type of FunctionQuery (DisMax query parser’s

bf parameter)

Function Queries

http://localhost:8983/solr/select/?q=cat:electronics+_val_:”div(price,weight)”&fl=*,score

Function Query - Example

Generated a formatted response of a search wt parameter sets the response writer Response writers supported

◦ Json◦ Php◦ Phps◦ Python◦ Ruby◦ Xml◦ xslt

Response Writers

http://wiki.apache.org/solr/FrontPage(link last accessed on 04/25/2011)

Lucid Works SOLR Reference Guide 1.4 http://www.lucidimagination.com/user_download/certified/cdrg/lucidworks-solr-refguide-1.4.pdf

(link last accessed on 04/25/2011)

Bibliography