Microsoft SQL Server 7.0 Full-Text Search

What is full-text search

and how do I troubleshoot it?

John Kane, MCSEPSS SQL Server Support

Microsoft Corporation

2

SQL Server 7.0 Full-Text Search

Full-text search (FTS) is a new component of Microsoft® SQL Server™ 7.0 that allows for faster and more flexible searching of character-based columns in SQL Server tables and file-based documents.

Full-text search is integrated with the new Microsoft Search Service for Microsoft® Windows NT® Server 4.0.

3

SQL Server 7.0 Full-Text Search

Full-text search is installed under the custom installation type and is available on the following SQL Server versions: SQL Server 7.0 Enterprise Edition SQL Server 7.0 Standard SQL Server 7.0 Small Business Server

Both Windows NT Workstation and Microsoft® Windows® 9.x clients can access Full-Text Search.

4

SQL Server 7.0 Full-Text Search

Real-world applications for full-text search: Internet/intranet search interface Product catalog searching Document management Data Warehouse / Business Intelligence (DW/BI)

search interface Any application requiring a fast and flexible search

interface, possibly combining SQL data and file-based documents

5

SQL Server 7.0 Full-Text Search

How is full-text search different than using the LIKE operator? Full-text search can search on words or phrases,

including the use of wildcards such as asterisk (*). Full-text search can search for words in close

proximity. Full-text search can search on inflectional forms

(different tenses of words) such as drive, drove, or driving.

A full-text search query is much faster than a LIKE query.

6

SQL Server 7.0 Full-Text Search

What do I need to run full-text search queries? Transact-SQL predicates:

CONTAINS

or FREETEXT

For rowset and ranked results use: CONTAINSTABLE

or FREETEXTTABLE.

7

SQL Server 7.0 Full-Text Search

How do I install full-text search?

Full-text search is not set up by default in the standard installation of SQL Server 7.0.

You must choose Custom Install during setup and select Full-Text under Server Components.

8

SQL Server 7.0 Full-Text Search

9

SQL Server 7.0 Full-Text Search

How do I install full-text search? (cont.)

After setup is complete, a new service (Microsoft Search Service) has been added.

This service can be managed under the Enterprise Manager Support Services folder.

10

SQL Server 7.0 Full-Text Search

11

SQL Server 7.0 Full-Text Search

What tools do I need to set up a full-text index? Use the Full-Text Indexing Wizard in SQL Server

Enterprise Manager. Use system stored procedures:

sp_fulltext_catalog sp_fulltext_table sp_fulltext_column

Note: The table must have a one-column, non-nullable unique index to be considered for full-text indexing.

12

SQL Server 7.0 Full-Text Search

13

SQL Server 7.0 Full-Text Search

What else do I need to do to use a full-text What else do I need to do to use a full-text index?index? After you have established the full-text index, you After you have established the full-text index, you

must must populatepopulate it using one of the following methods: it using one of the following methods: Enterprise Manager Database-specific Full-Text Enterprise Manager Database-specific Full-Text

Catalogs’ Catalog Content menu.Catalogs’ Catalog Content menu. System Stored Procedure: System Stored Procedure: sp_fulltext_catalogsp_fulltext_catalog

with with start_fullstart_full or or start_incrementalstart_incremental actions. actions.

NoteNote: You should also set up a SQL Server Agent job : You should also set up a SQL Server Agent job for scheduled full-text indexing.for scheduled full-text indexing.

14

SQL Server 7.0 Full-Text Search

15

SQL Server 7.0 Full-Text Search

New full-text search functions and objects:New full-text search functions and objects: Metadata functions:Metadata functions:

FULLTEXTCATALOGPROPERTYFULLTEXTCATALOGPROPERTY FULLTEXTSERVICEPROPERTYFULLTEXTSERVICEPROPERTY

SQL-DMO Objects:SQL-DMO Objects: FullTextCatalogFullTextCatalog FullTextService and more…FullTextService and more…

16

SQL Server 7.0 Full-Text Search

Are the full-text indexes stored within SQL Are the full-text indexes stored within SQL Server?Server? No. The full-text indexes are stored externally in No. The full-text indexes are stored externally in

system folders and files called catalogs.system folders and files called catalogs. By default, these full-text catalogs are located under By default, these full-text catalogs are located under

the parent folder the parent folder Mssql7Mssql7\Ftdata.\Ftdata. Full-text catalogs are maintained as a collection of Full-text catalogs are maintained as a collection of

folders and files, such as Sql0000500005.folders and files, such as Sql0000500005.

17

SQL Server 7.0 Full-Text Search

Simple full-text search full-text search SQL code examples:

Obtain a list of articles where the description mentions both “Picabo Street” and “downhill racing.”

SELECT article_id FROM Hockey_NewsWHERE CONTAINS (description, ' “Picabo Street“ AND

“downhill racing“ ')

This matches the following ...

18

SQL Server 7.0 Full-Text Search

Results of a simple full-text search Results of a simple full-text search CONTAINS query:CONTAINS query:

““Picabo StreetPicabo Street is scheduled to return to is scheduled to return to downhill racingdownhill racing.”.”

““Picabo Street'sPicabo Street's super G win during super G win during downhill racing...downhill racing...””

““DownhillDownhill Champions: Lindh and Champions: Lindh and Picabo StreetPicabo Street excel in excel in downhill downhill skiingskiing. Racing . Racing is their passion…”is their passion…”

19

SQL Server 7.0 Full-Text Search

Simple full-text search full-text search SQL code examples: (cont.) Why not just use LIKE?

A: Performance !! Results for searching on “dropdown” in DB (~

150,000 rows):

Search Field Full-Text Search (secs) SQL Server LIKE (secs)Title 2 15

Description 4 195All FT Cols. 5 Very Long

20

SQL Server 7.0 Full-Text Search

Simple full-text search full-text search SQL code examples: (cont.) Proximity search: words or phases close to one

another (approximately 50 words):SELECT Company, Price

FROM Travel_Loc

WHERE CONTAINS(Description,

'skiing NEAR Aspen'

21

SQL Server 7.0 Full-Text Search

Simple full-text search SQL code examples Simple full-text search SQL code examples (cont.):(cont.): Word generation or inflectional forms of words, such Word generation or inflectional forms of words, such

as swim, swam, swimming…as swim, swam, swimming…

SELECT colActivity, colClass FROM HERC_Club SELECT colActivity, colClass FROM HERC_Club

WHERE CONTAINS(colExercise,WHERE CONTAINS(colExercise,

'FORMSOF(INFLECTIONAL,'FORMSOF(INFLECTIONAL, swimswim)'))')

22

SQL Server 7.0 Full-Text Search

Simple full-text search SQL code examples (cont.): The FREETEXT predicate supports a simple form

of natural language query, for example, searches for similar meanings, not exact words:

SELECT help_text_col FROM on_line_help_tbl WHERE FREETEXT ( help_text_col,'Download Service Packs' )

23

SQL Server 7.0 Full-Text Search

Relevance ranking of results: Rank is a derived column value between 1 and 1,000. Indicates how well the row matches the selection

criteria. Applies to all FTS queries and is useful with both the

ISABOUT and NEAR functions. Must use CONTAINSTABLE and FREETEXTTABLE

rowset-valued functions.

24

SQL Server 7.0 Full-Text Search

Relevance ranking SQL code examples: Return the abstract column for articles on text processing

where those articles that are most relevant are listed first:

SELECT M.article_number, M.abstract, FT.RANK FROM magazines, CONTAINSTABLE(magazines, article,'ISABOUT ( ("DB2" NEAR "text extender") WEIGHT(0.9), ("SQL Server" NEAR "text") WEIGHT(0.3), ("SQL Server" NEAR ”fulltext") WEIGHT(0.9))')

AS FT WHERE M.article_number = FT.[KEY] ORDER BY FT.RANK DESC

25

SQL Server 7.0 Full-Text Search

Full-text search stored procedure calling sequence: sp_fulltext_database 'enable' sp_fulltext_catalog 'PubInfo', 'create' sp_fulltext_table 'pub_info', 'create', 'PubInfo',

'UPKCL_pubinfo' sp_fulltext_column 'pub_info', 'pr_info', 'add' sp_fulltext_table 'pub_info', 'activate' sp_fulltext_catalog 'PubInfo', 'start_full' ... wait ... Use sp_help_fulltext_* for full-text metadata queries...

26

SQL Server 7.0 Full-Text Search

Removing FTS from a SQL database stored procedure calling sequence: sp_fulltext_table 'pub_info', 'drop' sp_fulltext_catalog 'PubInfo', 'drop' sp_fulltext_database 'disable' sp_fulltext_service 'clean_up'

27

SQL Server 7.0 Full-Text Search

Full versus incremental full-text population Full full-text index population

Every row of every table associated with a full-text catalog is re-indexed.

Full-text catalog is discarded and rebuilt from scratch. Most useful when there are many changes or inserts.

28

SQL Server 7.0 Full-Text Search

Full versus incremental full-text population (cont.):

Incremental full-text index population: Only changed rows of tables within an full-text catalog are

re-indexed. Row will be reindexed even if the other columns are

changed. Requires presence of a timestamp column on a table. If the table schema version of table is altered, may be a

full population. Most useful on large tables with smaller turnover of rows.

29

SQL Server 7.0 Full-Text Search

Full-text search and distributed queries: Issue SQL queries against SQL data and system

file documents and properties. OLE DB Provider for Index Server: MSIDXS OPENQUERY OPENROWSET

30

SQL Server 7.0 Full-Text Search

Full-text search and distributed query examples find: Titles, filenames, file sizes, and URLs of the Excel

files under the IIS virtual root of /Excel/Revenue such that: The file names start with ‘OBOS’. The file sizes are less than 10,000 bytes. Include the words “Index Server” near

“SQLServer”

31

SQL Server 7.0 Full-Text Search

FTS and distributed query examples (cont.):

SELECT DocTitle, FileName, size, VpathFROM SCOPE(' "/Excel/Revenue" ')WHERE FileName LIKE ‘OBOS%.xls’ AND size < 10000 AND CONTAINS (' "Index Server" NEAR "SQL Server” ')

32

SQL Server 7.0 Full-Text Search

Full-text performance tuning from practical experience: Full-text queries are faster than queries using

LIKE “%test%. For SQL tables with fewer than 1 million rows,

both full-text indexing and search have acceptable performance and no special performance tuning is usually required.

33

SQL Server 7.0 Full-Text Search

Full-text performance tuning from practical Full-text performance tuning from practical experience (cont.):experience (cont.): For SQL tables with more than 1 million rows:

Hardware considerations Windows NT system configuration

considerations SQL Server 7.0 configuration considerations Full-text indexing and searching

considerations

34

SQL Server 7.0 Full-Text Search

Full-text performance for more than 1 million rows: Hardware considerations:

Fast and multiple CPUs: one to four 500-MHz Xeon III processors

Lots of memory: 1 GB to 4 GB of physical RAM

Multiple disk controllers with several channels

Fast disk I/O subsystems, RAID0, and RAID5

FTS is not currently supported on clusters

35

SQL Server 7.0 Full-Text Search

Full-text performance for more than 1 million rows (cont.): Windows NT system configuration considerations:

Pagefile.sys needs to be sized one and one-half to two times the amount of available physical RAM.

Pagefile.sys files need to be placed on their own drives (RAID0), preferably on a separate controller, or least on a separate channel off of a shared controller.

Microsoft Search Service can be set to dedicated during full-text index periods and then reset to background or normal when not full-text indexing.

36

SQL Server 7.0 Full-Text Search

Full-text performance for more than 1 million rows (cont.): SQL Server configuration considerations:

While full-text indexing or populating the full-text catalogs is ongoing, SQL Server’s maximum memory may need to be limited to half of the available physical RAM.

Support for language neutral full-text can be configured using the sp_configure system stored procedure.

37

SQL Server 7.0 Full-Text Search

Full-text performance for more than 1 million rows (cont.): Full-text indexing and search considerations:

Full-text Indexing or populating the full-text catalogs should be done during periods of low system activity, typically during database maintenance windows.

The full-text indexing of SQL tables with more than 4 million rows can take many hours or days to complete. Consider the options offered in KB article Q240867, “INF: How to Move, Copy & Backup SQL Server 7.0 Full-Text Catalog Folders & Files.”

38

SQL Server 7.0 Full-Text Search

Full-text performance for more than 1 million rows (cont.): Full-text indexing and search considerations:

Apply the latest service pack (SP), currently SP1.

When designing full-text search queries, consider using the CONTAINSTABLE and FREETEXTTABLE functions and their new Top_N_Rank parameter.

See KB Article Q240833, “FIX: Full-Text Search Support for TOP via CONTAINSTABLE and FREETEXTTABLE Clauses.”

Currently available as a hotfix and is included in SP2.

39

SQL Server 7.0 Full-Text Search

Full-text search basic troubleshooting techniques: Identify the trouble area(s):

Full-text indexing (populating full-text catalogs) and/or

Full-text searching Gather information about:

Hardware, Windows NT, and SQL Server configurations.

Review the Windows NT application log using Event Viewer.

SQL tables and full-text catalogs involved.

40

SQL Server 7.0 Full-Text Search

Full-text search basic troubleshooting techniques (cont.): Review the SQL Server Full-Text Search

Troubleshooter on the Microsoft Support Web site at:

http://support.microsoft.com/support/tshoot

/sql7fts.asp Are you getting some type of FTI or FTS error?

Most are recorded in the Windows NT application log as a “Microsoft Search” source.

Are you having FTI or FTS performance problems?Review the previous slides on full-text performance issues.

41