Introduction to Full-Text Searching in SQL Server

2012Adolfo J. Socorro, Ph.D.

IT Impact, Inc.asocorro@itimpact.com

Outline

What can we do with FTS?How to install FTSFTS componentsCreating FTS indexesHow to query with FTSFILESTREAM and FileTable

FTS Basics

charvarcharncharnvarchartext

ntextimagexmlvarbinaryvarbinary(max) 

FTS allows searching against character-based data

Search Functionality

• “hotel” => “hotel”

Specific words or phrases

• “fan” => “fantastic”, “fantasy”• “local store” => “locally stored”

Prefixes

• “minimized” => “minimizing”, “minimise”

Inflectional forms

Search Functionality

• “search,query” => “query to perform search”

Proximity

• “folder” => “directory”

Synonyms

Weighted Values

A First Look

Let’s run some simple examples to get a feel for FTS!

LIKE vs FTS

LIKE works on character patterns only

Cannot use the LIKE predicate to query formatted binary data

FTS is much faster against large amounts of unstructured text data

Supported SQL Server Editions

EnterpriseBusiness IntelligenceStandardWebExpress with Advanced Services

Available since at least SQL Server 2000

FTS Components

Word Breaker

Stemmer Stoplists

Thesaurus Filters Property

Lists

Language Support

50+ languages

Language-specific componentsWord breakers and stemmersStoplistsThesaurus files

How to Install

Default FTS Language

FTS Indexes

One index per table or indexed view

Must have a unique, single-column, non-nullable index on the table

Grouped within the same database into one or more full-text catalogs (“containers”)

Full-Text Catalogs

A logical construct

A way to manage FT indexes together

Index Population

Population: the addition of data to full-text indexes

Automatic Manual• On Request• Scheduled

Steps to Setup an Index on a Table

Create Full-Text Catalog

For Each Column to Index• Indicate language• Indicate document type *

Choose Change-Tracking Mechanism

Full-Text Index Wizard

Example: Create Catalog and Index

CONTAINS

Precise or prefix matches to single words and phrases

Proximity matchesLogical operations between conditions:

AND, OR, AND NOTOptional use of inflectional forms and

thesaurus

FREETEXT

Matching the meaning, but not the exact wording, of specified words or phrases

Always uses inflectional forms and thesaurus

CONTAINSTABLE AND FREETEXTTABLE

Return a relevance ranking value (RANK) and full-text key (KEY) for each row

The actual RANK values are unimportant and typically differ each time the query is run

ISABOUT/WEIGHT influence the rankingin CONTAINSTABLE

Example: Queries

Stoplists

A mechanism to discard commonly occurring strings that do not help the search

a is the

by and …

Thesaurus

Nicknames: Robert/BobCommon misspellings:

calendar/calenderHomophones: Geoff/JeffTechnical terms: proc/procedure

Very powerful if you log searches and learn what users are commonly

searching for

Thesaurus

One file per language

Expansions

“bike”in addition to

“bicycle”

Replacements

“calendar” instead of “calender”

Filters

Extract textual information from the document (removing the formatting)

Send the text to the word-breaker component for the language associated with the column

Need to manually install Office 2010 and PDF filters

Example: FTS Components

Where to Store Large Objects?

Database File System

security

manageability,

recoverability

transactional

consistency

performance

Why Store in the Database?

Integrating unstructured data into the relational database provides significant benefits: Integrated storage and data

management capabilities (e.g., backup)

Ease of administration and policy management

Full-text search

FILESTREAM

A database/file system hybrid

FILESTREAM is an attribute that can be assigned to a varbinary(max) column

Allows storing BLOB data in the file system

Not restricted to the 2 GB limitSQL Server imposes on BLOBs

FILESTREAM

SQL Server buffer pool is not used

Isolation semantics are governed byDatabase Engine transaction isolation levels

Steps to FILESTREAM

Enable at OS level

Configure at instance level

Create a filegroup

Add a file to the filegroup• Indicate root folder

OS-level Configuration ofFILESTREAM

Instance-level Configuration of FILESTREAM

Example: FILESTREAM

FILESTREAM

All data access must be transactional

Must use specific APIs for file I/O

Do not edit the files directly!

When to Use FILESTREAM

Objects that are being stored are, on average, larger than 1 MBStore smaller objects in the

database

Fast read access is important

You are using a middle tier for application logic

FileTables

A special, fixed-schema kind of table

Builds on top of existing FILESTREAM capabilities

Store files and documents in in the database, but access them from Windows applications as if they were stored in the file system (WIN32 API)

FileTables

Hierarchical namespace

Includes file system properties as columns

Preserves full file names

Non-transactional access through the FS

FileTables

Calls to create or change a file or directory through the Windows share are intercepted by a SQL Server component and reflected in the corresponding relational data in the FileTable

Example: FTS over FileTables

FileTables vs FILESTREAM

File and directory hierarchy maintained in the database

Windows application compatibility

Relational access to file attributes

Both are available in all editions

Wrap Up

Advanced searching on character-based data, including documents

FTS setup, components, and queries

FILESTREAM

FileTables

Other Topics

Document-property searchSemantic searchOptimizationsQuery plans and execution traces

References

Posts and presentations by Bob Beauchemin http://www.sqlskills.com/blogs/bobb/

Blog: SQL Server FTS Team Blog http://blogs.msdn.com/b/sqlfts

SQL Server 2012 Books Online http://msdn.microsoft.com/en-us/library/

cc645577(SQL.110).aspx

Filter Packs

Adobe PDF Filter http://www.adobe.com/support/downloads/

thankyou.jsp?ftpID=4025&fileID=3941

Office 2010 Filters http://www.microsoft.com/en-us/download/

details.aspx?id=17062