Setup and Configuration Guide -...

Avid® iNEWS® Fast Text SearchSetup and Configuration Guide

Purpose of This Document

This document provides the iNEWS system administrator with an overview of Fast Text Search for the iNEWS environment and recommended best practices. The document also provides an analysis of the IPC application against the most common security flaws for Web-based applications.

Intended Audience

This document is intended for anyone responsible for system maintenance, including iNEWS system managers and IT administrators.

2

Contents

Overview of Fast Text Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Installing FTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Configuring FTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Starting and/or Stopping FTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Building the iNEWS Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Ftsdebug Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Monitoring Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Viewing Index Requests in SYSTEM.INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Adjusting Indexes for Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Changing Read Groups on Indexed Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Backup Index File Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Backing up FTS Index Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Moving FTS Indexes to New FTS Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Indexing Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Where to Find More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Overview of Fast Text Search

3


Fast Text Search (FTS) is software for indexing material so it can be searched faster. An FTS server installation supports up to 50 iNEWS systems.

FTS is comprised of two major software components:

• FTS software which runs on the iNEWS Servers.

- the ftsseek server

- the ftsindex server

• FTS software which runs on a Windows-based computer.

- ftsindex.exe (for indexing)

- ftssearch.exe (for searching)

n As of the date of this publication, version 4 of Fast Text Search (FTS 4) is required for iNEWS Server version 3.5 and later. FTS 4 is not supported on any iNEWS Server versions earlier than 3.5. FTS 4 is supported on Windows Server 2008 R2, Windows XP SP3 (32-bit), and Windows Server 2003 SP2 (32-bit). The recommended operating system is Windows Server 2008 R2. A full re-index is required with an upgrade to FTS 4.

FTS is qualified in a virtualized environment when the guest OS instance meets the application’s minimum system requirements.

When configured and running, system managers can assign the Indexed database trait to queues and directories. The iNEWS side FTS software works in conjunction with the Windows-based side FTS software to create indexes of those queues and directories. The FTS index files are stored on the Windows-based computer.

When a user performs a search in an indexed queue, the results are looked up in the index on the Windows-based side and passed back to the user. Search results are very fast compared to the old SEEK style searches which search every single word of every single story trying to make a match.


4

How Does Searching Work

Additional training should not be required for the users. The search dialog boxes that are presented when doing an FTS search versus a seek search are nearly identical, so implementing FTS is fairly transparent to the users.

The type of search executed depends on the status of the indexed trait at the level where the search initiates. If a user launches a search at a directory or queue level that is not indexed, the search request will be placed in SYSTEM.SEEK and passed off to the slower seek server. If the user launches a search at a directory or queue level that has the indexed trait assigned, then the search is placed in SYSTEM.FTS and passed off to the ftsseek server.

For example, if ARCHIVE.2013, ARCHIVE.2014, and ARCHIVE have the indexed trait turned off, then searches launched at that level will go to SYSTEM.SEEK and execute slowly. Even if the subsidiary monthly queues are indexed.

If you turn the indexed trait ON for those directories, then the searches at the directory level will be handed off to FTS.

n If the subsidiary queues are not yet completely indexed, this FTS search will not return results for any non-indexed subsidiary queues because there is no information in the indexes about the contents of those queues.

How Material Gets Indexed

When the indexed trait is applied to a queue, an index request for that queue is placed in the SYSTEM.INDEX queue. The ftsindex server on the iNEWS side retrieves the indexing request and hands it to the ftsindex.exe process on the Windows-based computer. The ftsindex.exe process adds the story to the index files on the Windows-based side. When the index request is completed, the index request is removed from the SYSTEM.INDEX queue.

If the indexed trait is applied to a folder, an index request is placed in SYSTEM.INDEX for each queue beneath that folder and each queue is indexed as outlined above.

If material is added to an indexed queue or a story modified in an indexed queue, an index request for that individual story is placed in the SYSTEM.INDEX queue for processing.

If a story is killed or is purged from an indexed queue, an un-index request is placed in the SYSTEM.INDEX queue for that individual story.

If the Indexed trait is removed from a queue or folder, un-index requests are placed in SYSTEM.INDEX for processing.


5

Index/un-index requests in the SYSTEM.INDEX queue are processed in sequential order from oldest request to newest request. If there are very many requests pending in the SYSTEM.INDEX queue, it may take FTS a little while to catch up and process them all.

You can find out how many index and un-index requests are pending by using the following version of the ftsdebug command, as shown in this sample command line and output:

ftsdebug index countrequests

Q remove Q purge Q add Q replace

1 13167 1 0

S remove S purge S add S replace

0 0 0 0

The Q in the sample output stands for Queue; S stands for story.

An alternative to the ftsdebug command is by piping a queue listing of SYSTEM.INDEX to word count. This can give you an idea of how many total index and/or un-index requests are pending in SYSTEM.INDEX. The following command must be done from a Unix shell:

WAVD-A# sh

# list q system.index | wc -l

29598

#

The list q command produces a blank line and a two line header so subtract 3 from the number returned for the exact number of pending requests.

If you instruct the system to do a search adjust to an already indexed queue, using the command, ftsdebug search adjust <queuename>, then index requests are placed in the SYSTEM.INDEX queue.

If you were to assign the indexed trait to WIRES.ALL, change your mind and remove the indexed trait, and then later reassign it, that would put an index request for WIRES.ALL in SYSTEM.INDEX, followed by an un-index request, followed by an index request. The system would index all of WIRES.ALL, then un-index it, then index it all over again.

Installing FTS

6

Installing FTS

An FTS server installation supports up to 50 iNEWS systems.

To install FTS:

1. Navigate to the FTS folder on the iNEWS installation DVD and click Setup.exe.

The InstallShield Wizard starts and displays a Welcome dialog box.

2. Click Next.

The License Agreement dialog box appears.

Installing FTS

7

3. Read the Avid Software License Agreement, accept the terms, and click Next.

The Destination Folder dialog box appears.

Installing FTS

8

4. (Optional) Click Change if you want to alter the default installation location. Otherwise, you can accept the default and click Next.

5. Click Install when you are ready to begin the installation.

When the InstallShield Wizard completes the installation successfully, the following dialog box appears.

Installing FTS

9

6. Select the Launch Avid FTS Configuration check box and click Finish.

The Fast Text Search configuration window appears. You are now ready to configure the system.

Configuring FTS

10

Configuring FTS

After installing Avid Fast Text Search, some configuration is required.

To configure FTS:

1. Do one of the following:

t When the FTS installation is completed, select the Launch Avid FTS Configuration check box and click Finish.

t Select Start >Avid > FTS Configuration.

n If the User Account Control dialog box appears, click Yes to continue.

The Fast Text Search configuration window appears. By default, no services are active.

Configuring FTS

11

The default drop-down option, FTS_1, will select from any of the 50 potential FTS service pairs. The status Not Active means a service has not been installed or configured.

2. Click Configure to configure the FTS_1 services.

Configuring FTS

12

The default configuration values will appear, as shown in the following illustration.

3. Set the Active field to ACTIVE.

When you save your changes, if the Active field is set to ACTIVE, then:

- The service pair is installed and started.

- The Windows firewall is updated with the port information.

If the Active field is set to DISABLED, then:

- The service pair is uninstalled.

- The Windows firewall port information is removed.

4. Set the System Name to the name of the iNEWS system to be associated with this service.

5. (Optional) Modify the ports and index path. The index and search ports must be unique across all configured FTS services.

n Additionally, the W_BINDFTSI and W_BINDFTSS must correspond to the port numbers for the iNEWS system.

6. Click Save.

When you save changes, the INI file for the service pair is created (or updated). Each service pair will have its own INI file, FTS(n).ini.

Configuring FTS

13

The location of the INI files will vary:

- C:\Documents and Settings\AllUsers\Avid\FTS for computers with WinXP or Windows 2003 operating systems

- C:\ProgramData\Avid\FTS for computers with Windows Vista, Windows 7, or Windows 2008 operating systems

7. After you click Save, a warning might appear if the Index Path Folder, indicated in the Index Path field, does not exist. Click Yes to create the folder at the configured path.

The firewall exceptions, service configurations, and index path are created at this time.

8. Click Back to return to the main confirmation window and start the services. The configuration window will now display the active services with the configured system name.

When changes are made, clicking Back may put up a confirmation dialog box.

Configuring FTS

14

9. Configure additional systems by selecting unused FTS_# entries from the list.

After you have configured the services, you can start or stop them from the FTS configuration and management user interface (UI). For more information, see “Starting and/or Stopping FTS” on page 15.

Installation locations vary depending on the computer’s Windows operating system.

n If FTS needs to be uninstalled, all FTS service pairs should be set to “DISABLED” prior to the uninstall.

Operating System Location

Win XP C:\Program Files\Avid\FTS

Windows 2003 C:\Program Files\Avid\FTS

Windows Vista C:\Program Files (x86)\Avid\FTS

Windows 7 C:\Program Files (x86)\Avid\FTS

Windows 2008 C:\Programs Files (x86)\Avid\FTS

Starting and/or Stopping FTS

15


Stopping and starting services should be performed from the Fast Text Search configuration window, using the Pause and Play buttons next to the relevant system name. Even though the services will appear in the Windows Services list, all stopping and starting should be performed from the FTS configuration and management UI.

To start or stop FTS services:

1. Select Start > Avid > FTS Configuration.

If started from the Start menu, you may see the User Account Control prompt. If so, answer Yes.

2. Click the drop-down menu to select any one of 50 potential service pairs.


16

3. Click the Play (purple triangle) button to start services or the Pause (double lines) button to stop services.

n You can also use the options on the bottom of the Fast Text Search window to stop or start all services.

Building the iNEWS Index

17


An index rebuild is required when upgrading to FTS v.4.x from earlier versions.

n Avid recommended to index all static, unchanging queues (such a ARCHIVES) first and then make a backup of the index files before assigning the indexed trait to a dynamic queue, such as a wires queue. Having a backup ensures that you can easily start again indexing a dynamic queue like wires without having to first re-index all the archive material. For more information, see “Strategy for Indexing Wires” on page 18.

For sites utilizing the multiple index feature, and if your ARCHIVES are indexed in a different index base than WIRES.ALL, then there is no need to perform the suggested backup of the index files. For more information, see “FTS Multiple Indexes” on page 19.

To build the iNEWS index:

t Perform the following commands:

- ftsdebug index statistics — This command confirms communication with the FTS Index service.

- ftsdebug search statistics — This command confirms communication with the FTS Search service.

- list c fts — This command displays the ftsseek and ftsindex device numbers (ftsseek# and ftsindex#).

- stop <ftsindex#> <ftsseek#> — This command stops the FTS devices.

- dbpurge system.index 0 — This command clears any unhandled index requests.

- dbpurge system.fts 0 — This clears any unhandled search requests.

- dbtraits . reindex > /dev/null & — This command creates an index request for each indexed queue. The output is sent to /dev/null because of output volume.

n Alternatively, more important wires and recent archives can be added to the index before adding the full database during the re-indexing.

- start <ftsindex#> <ftsseek#> — This command restarts the FTS devices.


18

Strategy for Indexing Wires

WIRES are highly dynamic, changing by the minute. In addition, each hour stories reach their purge limit and are purged out of the WIRES queue.

For each wire story that comes into an indexed wire queue, an index request is placed in SYSTEM.INDEX. Unindex requests are also placed in the queue as stories are purged.

The best strategy would be to utilize the FTS multiple indexing capabilities, which supports up to 50 separate index bases, allowing for indexing of more documents while avoiding index file size limits. For more information, see “FTS Multiple Indexes” on page 19.

Another strategy would be to create a "shadow" copy of a wires queue to be used for indexing only. With this in place, users would still be able to search in the normal WIRES queue if there was a problem with the indexes.

The easiest way to send copies of all wire stories to a shadow queue is to adjust the SYSTEM.WIRES.DISTRIBUTION story. For each block of distribution codings in SYSTEM.WIRES.DISTRIBUTION, there is likely to be a line similar to the following that always makes a copy of any story coming in that wire port to a queue such as WIRES.ALL:

AP######## WIRES.ALL ALWAYS URGENT

Add another line just below that to always send copies to a wires queue that will be used for indexed searches:

AP######## WIRES.ALL ALWAYS URGENT

AP######## WIRES.ALL-INDEXED ALWAYS SILENT

You can then assign the indexed trait to WIRES.ALL-INDEXED and users can run indexed FTS searches in there.

If the index becomes corrupted, users will still be able to perform regular seek searches in WIRES.ALL.


19

FTS Multiple Indexes

Avid iNEWS supports up to 50 separate index bases. This separation allows indexing of more documents while avoiding the index file size limit of 2GB. And if one becomes corrupted, only it must be reindexed; it will not affect others.

The index base is assigned using the dbtraits command. All directories default to index 0. For example, the following commands will create separate index bases:

dbtraits wires.all ftsindex 49

dbtraits archive.2012 ftsindex 2

dbtraits archive.2013 ftsindex 3

The command list d-i displays the index base used by a directory. For example:

WAVD-A# list d-i wires.all

SRPlo-LIsUGQSXWFiTM interplay fts directory

Q-------I------WF--- 0 49 WIRES.ALL

The command list fts=index d-i displays the directories assigned to an index base. For example:

WAVD-A# list fts=49 d-i

SRPlo-LIsUGQSXWFiTM interplay fts directory

Q-------I------WF--- 0 49 WIRES.ALL

FTS handles the separation of the bases. There is still only one ftsindex device on the iNEWS server, and there is still only one ftsindex.exe process on the FTS server.

An ftsindex attribute was added to iNEWS that supports any value from 0 to 49. The ftsindex attribute is independent of the INDEXED attribute and determines which FTS index base is used for the data in the queue with that ftsindex value.

The location of the index files is determined in the FTS configuration as the index path.

The FTS index path will contain a subdirectory for each index. These subdirectories will have 3 digit names with leading zeros. So, index base 0 is in the directory C:\FTS\000, and index base 32 is in C:\FTS\032.

The index base files are always named ftsindex with the distinction coming in the tag. For example, four files would be: ftsindex.in1, ftsindex.in2, ftsindex.in3, and ftsindex.in4.


20

FTS search results will always be returned in descending date order; therefore, W_AGEWEIGHT, W_HITSWEIGHT, and W_LOCWEIGHT are removed from /site/dict/words.

In using the multiple index functionality, it is best to leave index 0 empty as that’s the default index. This means if a queue is assigned index 0, there is a minimal chance of it reaching the 2GB file limit.

One index can hold an estimated four years of archives. In terms of usage, indexes 1-5 have been successfully used for 22 years of archives (1990-2012). For example, configure FTS accordingly:

• Index bases 1-20 for archives (approximately four years of archives fit in one index)

• Index base 48 for assignments

• Index base 49 for WIRES.ALL

n Avid recommends, at the time of configuring FTS, you configure hidden, top level, empty Archive folders for the next four years or so; that way, as a system administrator, you can configure the usage today and not have to do further work as each new year commences.

If there are a large number of wires coming into a system and all are indexed, there may be performance issues. Avid recommends that only the WIRES.ALL queue be indexed as this will be sufficient to index all wire stories on the system without encountering performance issues.

Ftsdebug Command

21

Ftsdebug Command

The ftsdebug command may be used for various purposes, such as adjusting indexes for queues or to manually verify communication from the iNEWS servers to the FTS processes running on the Windows-based computer. Here is the usage message for ftsdebug.

WAVD-A# ftsdebug ?

Usage: ftsdebug (index | search) <command>

Some commands may be used with either index or search, but some are limited to one or the other. The following table provides a list of commands that can be used with ftsdebug.

Command Used with... Description

viewlog index and search Print contents of the log file.

openlog index and search Open/reopen the log. Truncates existing log.

closelog index and search Close the log, leaving it intact.

rotatelog index and search Rename current log and reopen it.

limitlog index and search Set log file size limit in MiB (log rotated when limit reached).

statistics (or stat) index and search Show program version, start time, and some statistics.

debug index and search Set debug log level for the program.

Level 0: no logging done

Level 1: each action logged

Level 2: request parameters logged

Level 3: do both levels 1 and 2

countrequests index only Tabulate pending index requests (SYSTEM.INDEX)

showrequests index only List pending index requests (SYSTEM.INDEX)

tune on|off index only Turn tuning on or off or force a major tune.

tune <type> index only Options include: minor, major, or absolute tune immediately.

list <path> search only List FTS index entries for <path>

Ftsdebug Command

22

compare [-v] <path> search only Compare FTS index for <path> against database content.

adjust [-v] <path> search only Same as the compare command, plus the discrepancies will be fixed.

search [-v] <query> search only Search FTS index for query terms.

Command Used with... Description

Diagnostics

23

Diagnostics

The iNEWS Server now logs a diagnostic report when any queue is added or removed from indexing. This feature allows tracking index rebuild progress without scheduling ftsdebug index statistics commands. For more information about ftsdebug commands, see “Ftsdebug

Diagnostics

24

Command” on page 21.

The first diagnostic reports the start of indexing, and the second diagnostic reports total number of documents indexed.

Dec 22 10:20:03 WAVD-B S120: Queue WIRES.ALL::add:0:-1:-1:0

Dec 22 10:23:24 WAVD-B S120: Queue WIRES.ALL 11226 adds

You can still manually verify communication from the iNEWS servers to the FTS processes running on the Windows-based computer with the following ftsdebug commands:

• Type: ftsdebug index stat

• Type: ftsdebug search stat

These commands gather statistics from their respective executable processes on the Windows-based side. For instance:

• The command ftsdebug index stat gathers stats from ftsindex.exe and tells you how many documents (stories) are indexed.

• The command ftsdebug search stat gathers stats from ftssearch.exe.

WAVD-B: ftsdebug index stat

Service name: ftsindex1

Version: 4.0.2.4

Process id: 238

Started: 08/26/13 09:00:12

Debug value: 0

Tuning: on

Index Base: C:\ProgramData\Avid\FTS\WAVD

iNEWS System: wavd

Port: 6100

Current Index: 000

Total Documents: 39736

Requests Serviced: 95081

000 File sizes (KB): 25702/25660 16376/1 15823 1

Diagnostics

25

Documents: 34076 Added: 39724 Removed: 5651 Replaced: 3 Purged: 0

001 File sizes (KB): 592/543 572/1 1215 1


003 File sizes (KB): 2540/2528 2547/1 1086 1


010 File sizes (KB): 4/0 4/1 3 1


WAVD-B:

WAVD-B: ftsdebug search stat

Service name: ftssearch2

Version: 4.0.2.4

Process id: 239

Started: 08/26/13 09:01:52

Debug value: 0

Tuning: on

Index Base: C:\ProgramData\Avid\FTS\NRCS

iNEWS System: nrcs

Port: 6100


WAVD-B:

Diagnostics

26

If the normal statistics display (shown above) is not returned in response to the command and the ftsdebug command hangs and eventually times out with a “Cannot connect to FTS...” message, then there is no communication to that process on the Windows-based side. Here are examples of the failed connection messages:

WAVD-B: ftsdebug search stat

Cannot connect to FTS index server ‘csmas:6101’

WAVD-B:

WAVD-B: ftsdebug index stat

Cannot connect to FTS index server ‘csmas:6100’

WAVD-B:

If the iNEWS side ftsdebug program cannot communicate with the Windows-based side, check the condition of the computer and whether the ftsindex.exe and ftssearch.exe programs are still running. The computer may have rebooted or could be experiencing some other difficulty.

When a communication problem exists, all of the FTS components should be stopped and then restarted. If your ftsdebug command is still hanging, you can break out of it with a CTRL+\ (backslash).

By default, logs and index bases are stored in C:\ProgramData\Avid\FTS in Windows Server 2008 R2 and in C:\Documents and Settings\All Users\Avid\FTS in Windows XP and 2003.

Below is sample output from ftsdebug:

WAVD-A$ ftsdebug index statistics

Service name: ftsindex1

Version: 4.0.2.4

Process id: 1628

Started: 08/26/13 07:30:47

Debug value: 0

Tuning: on

Index Base: C:\ProgramData\Avid\FTS\WAVD

iNEWS System: wavd

Diagnostics

27

Port: 6102

Current Index: 001

Total Documents: 39736


000 File sizes (KB): 749/380 653/1 621 1


001 File sizes (KB): 592/543 572/1 1215 1


003 File sizes (KB): 2540/2528 2547/1 1086 1


010 File sizes (KB): 4/0 4/1 3 1


n At sites with multiple index bases, each numbered index is listed separately (as shown in the previous example). Notice, for instance, the line 000 File sizes (KB): 749/380 653/1 621 1. The .in1, .in2, .in3, and .in4 file sizes are shown in Kilobytes. The numbers after the ‘/’ represent the optimal or packed size of the file if that index were tuned. In this case the .in1 and in2 file sizes could be reduced. This is probably only worthwhile for the more static indexes such as for ARCHIVE stories.

Monitoring Indexing

28

Monitoring Indexing

There are various ways to monitor the indexing process. One way is to use the ftsdebug index stat command as shown in “Diagnostics” on page 23. The ftsdebug index stat command produces statistics about indexing. By doing successive ftsdebug index stat commands, you can monitor and track the number of indexed documents rising. Each time you run the command, you can notice the number of indexed documents climbing when the system was processing index requests from SYSTEM.INDEX. One could infer that indexing was completed when the number of indexed documents stops climbing and/or the SYSTEM.INDEX queue is empty. The Requests Serviced number displayed in the results from running the command includes index requests, purge requests, and ftsdebug status requests; therefore, it is likely to be the same as the number of indexed documents.

Another alternative lets you find out exactly how many index and un-index requests are pending by using the following version of the ftsdebug command: ftsdebug index countrequests

Also, there is the more informative showrequests option of the ftsdebug command, which lists any pending requests in SYSTEM.INDEX. For more information on these variations to the ftsdebug command, see “Viewing Index Requests in SYSTEM.INDEX” on page 28 and “Ftsdebug Command” on page 21.

Another way is to perform successive word counts of the SYSTEM.INDEX queue listings (outlined in “Viewing Index Requests in SYSTEM.INDEX” on page 28) to see how the number of index requests decreases or increases over time.

Viewing Index Requests in SYSTEM.INDEX

29


It is not possible to see index requests in SYSTEM.INDEX from a client login session. They can only been viewed at the console.

To tabulate pending index requests in SYSTEM.INDEX, using ftsdebug:

t Type ftsdebug index countrequests

Here is sample output from this command:

Q remove Q purge Q add Q replace

1 13167 1 0

S remove S purge S add S replace

0 0 0 0

The Q in the sample output stands for Queue; S stands for story.

To list pending index requests in SYSTEM.INDEX, using ftsdebug:

t Type ftsdebug index showrequests

Here is a truncated sample of output from this command:

SHOWS.10PM:election-recap:0:replace:1906225454:39008:0

SHOWS.10PM:candidates:0:replace:1956557102:39008:1

SHOWS.10PM:placeholder:0:remove:1957081000:39008:2

WIRES.ALL:ap-wi-mad-urgent:1:add:1637789999:39018:0

WIRES.ALL:ap-ia-oax-ialocalsto:1:add:1637979894:39018:1

ARCHIVES.2013:2:add:1906225454:39008:0

SYSTEM.PROPERTIES.WIRES:20130517001503:purge:1

[...]

In the sample, there are various requests, such as: replace, remove, add, and purge. The sample lines also show queue names or project IDs, story titles (if any). Index file IDs are provided as single numerical values, such as 0 (zero), 1 (one), or 2 (two). And the long numerical strings at the end of each line are record IDs.

To view index requests in SYSTEM.INDEX, using the list q command:

t Type list q system.index.

n The list q command will display one line for each index request (story) in the queue (SYSTEM.INDEX). Since there might be hundreds or thousands of stories in the queue, it is wise to specify a record limit, such as list q system.index 20, which would display up to the first 20 stories in the queue.


30

t You can determine exactly how many stories are in a queue by going into the Unix shell, performing a full listing of the queue, and piping that information to word count. This will calculate the total number of resulting lines from which you can get an idea of how many stories are in the queue. For instance:

WAVD-A# sh

list q system.index | wc -1

2

#

An empty or purged queue will have a line count of 2 because a list q of an empty queue still produces a two line header, which includes the queue ID number:

WAVD-A# list q system.index

SYSTEM.INDEX id=322792

WAVD-A#

Because of this 2 line header, the actual number of stories in any queue will be exactly two less than the number returned by the list q <queue name> | wc -1 command.

For instance:

WAVD-A# sh

list q wires.all | wc -1

20950

#

The WIRES.ALL queue in the example has 20948 stories in it.

Adjusting Indexes for Queues

31

Adjusting Indexes for Queues

If you suspect a queue is not properly indexed, it can be brought back up to date using the ftsdebug command. See “Ftsdebug Command” on page 21 for more information on ftsdebug and its usage.

By using the ftsdebug search adjust <pathname> command, you can instruct FTS to compare the contents of a queue to what's actually in the index files, and adjust the indexes to bring indexing up to date for that queue.

When you ftsdebug search adjust a queue, index requests are placed in the SYSTEM.INDEX queue for any stories that are found in the queue that do not exist in the indexes. These index requests are processed by the ftsindex server.

The ftsdebug search adjust command can be used at any time to verify and automatically adjust discrepancies between what is in the index files and what is actually in the indexed queues. Once a quarter or every couple of months, you could launch an ftsdebug search adjust on your latest archives before leaving for the evening to ensure that the indexes are correct. For example, type:

ftsdebug search adjust archives.2012

Changing Read Groups on Indexed Queues

While setting or changing the read group for ARCHIVES queues, you should be aware of the consequences of such action. Changing the read group on existing FTS indexed material generates index requests to SYSTEM.INDEX for each indexed story. Such requests could number in the hundreds of thousands or more. Additionally, index requests for any new material added to the ARCHIVES queues will be placed beneath all of the requests generated by the read group change. Until this indexing backlog is taken care of, newly archived shows will not be added to the FTS indexes and will not be searchable with FTS. Also, if the WIRES queues are indexed, then index requests for newly added or purged stories will also be placed way down beneath the re-index requests generated by the read group change, effectively rendering new wires stories unsearchable.

If such a change must be done, Avid recommends doing it at a moment preceding a period of downtime, such as Friday night to allow FTS to process the requests over the weekend. Depending on the number of requests generated by the read group change, the system may or may not have time to complete the process by the time system usage picks up on Monday. Also, Avid recommends de-indexing the queue by turning off the index attribute, applying the read group change, and then turning the index attribute back on for that queue to make the process as fast as possible.

Backup Index File Management

32


A crucial part in managing the Fast Text Search capabilities of an iNEWS system is backing up the index files so that if something happens, a searchable index of material is protected. This section includes procedures for how to back up FTS index files, how to restore the system from backup files, and if necessary, how to delete an old index and start over.

Backing up FTS Index Files

Before backing up the index files on the FTS Windows-based server, you must first shutdown FTS so the files are quiescent and not in use.

To back up FTS index files:

1. Stop the ftsindex and ftsseek servers on the console. Their numbers can be found with the command: list c fts

2. After they have been stopped, use the FTS configuration and management UI to shutdown the FTS processes running on the Windows-based side. See “Starting and/or Stopping FTS” on page 15 for more information.

This should cause the ftsindex.exe and ftssearch.exe processes to exit on the Windows-based server. You might want to use Windows Task Manager to verify that both processes are no longer in the process list.

3. Drag and drop the index files to a backup folder, using Windows Explorer, or back them up using your normal backup programs and procedures.

4. Connect to the console from a client, run the following command, and then disconnect from the console:

list flags=i d-i

This will produce a listing of every indexed queue on the system, including which index a queue uses. By connecting to the console, running the command, then disconnecting (CTRL-D), that listing should be captured to SYSTEM.CAPTURE. You should copy or print out this listing for future reference in case you need to restore and fallback to a set of backed up FTS indexes. That listing of indexed queues matches the contents of the index files and it may be necessary to set the indexed flags back to this listing if the files are restored.

5. Restart the Windows-based side FTS processes via the FTS configuration and management UI, then restart the FTS servers on the iNEWS side when done.


33

Restoring from Backup FTS Index Files

If you need to restore your system from backed up copies of FTS index files, you must first stop all FTS processes and delete existing indexes before restoring the system from the backup files.

To restore from backup FTS index files:

1. Stop the ftsseek and ftsindex servers on the news server.

n If you do not know the servers numbers, use the list c fts command to reveal them.

2. Stop the ftssearch.exe and ftsindex.exe servers on the Windows-based system by using the FTS configuration and management UI. See “Starting and/or Stopping FTS” on page 15 for more information.

3. Open the folder where the index base files reside. You can get the location of this folder by using the FTS configuration and management UI. It is provided as part of the pathname found in the Index Path field.

4. Select and delete all index base files plus any other temporary files in the folder.

5. Copy the backed up versions of the FTS index files into the folder.

6. Remove any stale edit locks from the SYSTEM.INDEX queue by entering the following command: dbclean system.index

7. Since the previous index removal command might have placed a large number of un-index requests in SYSTEM.INDEX, purge the contents of the SYSTEM.INDEX queue by using the following command: dbpurge system.index 0

If the database is busy with another db process, try again later. For instance, the automatic dbpurge running at quarter past the hour might prevent you from purging the queue until it has finished.

8. Restart the Windows-based servers via the FTS configuration and management UI.

9. Restart the ftsseek and ftsindex services on the iNEWS server. See “Starting and/or Stopping FTS” on page 15 for more information on how to start and stop FTS services.

The indexes now match the state of indexing at the time when the indexes were backed up. However, it is likely that additional material was indexed in the intervening interval since the backup, it will now be necessary to reindex just that additional material.

10. Type list flags=i d-i to get a listing of the currently indexed queues. Compare this list to one made at the time the index files were backed up to determine which queues were indexed after the backups were made.

11. Use the ftsdebug command—in the following syntax—to check the indexes against the queues that were indexed after the backup and adjust the indexes to reindex those queues.

ftsdebug search adjust <pathname>


34

For example, if the backup of the index files was for the following indexed queues: ARCHIVE.2009, ARCHIVE.2010, and ARCHIVE 2011, but results for a list flags=i d command now shows the following queues have been indexed: ARCHIVE.2009, ARCHIVE.2010, ARCHIVE.2011, ARCHIVE.2012, DESK.ARCHIVE, you should use the ftsdebug command twice to put index requests in SYSTEM.INDEX for each queue that was indexed after the backups were made, namely:

ftsdebug search adjust archive.2012

ftsdebug search adjust desk.archive

n You can only run one ftsdebug search adjust command at a time. Wait for the first one to finish before entering the second command.

Deleting Old Indexes and Starting Over

If, for some reason, an index base becomes unusable, Avid recommends that you simply remove the index base on the Windows-based search system and re-index from scratch.

To delete an old index and start over:

1. Stop the ftsseek and ftsindex servers on the news server.

n If you do not know the servers numbers, use the list c fts command to reveal them.

2. Stop the ftssearch.exe and ftsindex.exe servers on the Windows-based system using the FTS configuration and management UI. See “Starting and/or Stopping FTS” on page 15 for more information.

3. Open the folder where the index base files reside. You can get the location of this folder by using the FTS configuration and management UI. It is provided as part of the pathname found in the Index Path field.

4. Select and delete all index base files plus any other temporary files in the folder.

5. (Optional) Get a listing of queues that currently have the index database trait, by entering the following command: list flags=i d-i

6. Remove the index flag from any and all indexed queues by using the dbtraits command with the following syntax: dbtraits <queuename> -index

n As a superuser, you can globally remove the index flag from all directories and queues in the database by entering dbtraits . -index from the console. Also, the properties of a queue—including the index trait (flag)—can be adjusted directly from the Queue Properties by right-clicking on the queue in the directory panel.

7. Remove any stale edit locks from the SYSTEM.INDEX queue by entering the following command: dbclean system.index


35

8. Since the previous index removal command might have placed a large number of un-index requests in SYSTEM.INDEX, purge the contents of the SYSTEM.INDEX queue by using the following command: dbpurge system.index 0

If the database is busy with another db process, try again later. For instance, the automatic dbpurge running at quarter past the hour might prevent you from purging the queue until it has finished.

9. Restart the Windows-based servers. This is accomplished by using the FTS configuration and management UI. See “Starting and/or Stopping FTS” on page 15 for more information.

10. After the server programs have restarted, verify communications to them from the console by using the following commands: ftsdebug index stat and ftsdebug search stat. See “Diagnostics” on page 23 for more about these commands and verifying communications.

11. Restart the ftsseek and ftsindex services on the iNEWS server. See “Starting and/or Stopping FTS” on page 15 for more information on how to start and stop FTS services.

12. Reassign the index flag to your queues and trigger the rebuilding of the Windows-based index files by using the dbtraits command in the following format:

dbtraits <queuename> +index

n Avid suggests it might be better to assign the indexed trait to a single queue at a time and wait for that queue to complete indexing before assigning the flag to another queue. The reason for this is because assigning traits to higher level directories (such as dbtraits archives +index) assigns the indexed trait to every queue beneath that level. It might take several days or even weeks to fully index all of the material, and during that time any searches in the sub-queues would be handed off to FTS and search requests placed in SYSTEM.FTS. This could result in incomplete search results for users, since results can only come from fully indexed material. For instance, if 10% of the queues have been indexed, searches in 90% of the queues would return zero hits while the indexing is still underway.

Searches launched at a level that does not have the indexed trait are passed off to the seek servers and a search request placed in SYSTEM.SEEK.

13. (Optional) Monitor the status of indexing by using the ftsdebug index statistics command. This command can be abbreviated to ftsdebug index stat. When the number of indexed documents stops climbing, you may assume indexing of assigned queues is complete.

Moving FTS Indexes to New FTS Computer

36


On occasion, sites need to upgrade their computers. This section provides information for when that upgrade is needed on the computer hosting the FTS index files.

To transition FTS index fils from one computer to another:

1. Install the FTS software on the new computer.

2. Ensure the new computer is on your network with an IP address that is different from the old FTS computer.

3. Select Start > Avid > FTS Configuration to open the FTS configuration and management user interface (UI).

4. Determine what folder holds the FTS index files on the old FTS computer. Find where the index files reside via the FTS configuration and management UI on the old computer. The location of this folder is provided as part of the pathname found in the Index Path field.

For example, the Index Path field in the previous graphic shows the folder’s pathname for the index files to be C:\ProgramFiles\Avid\FTS\NRCS.

5. Using the FTS configuration and management UI, create the same folder(s) on the new FTS PC to hold the base index files, so that they match what was on the old system. This location is found and/or created in the Index Path field.


37

n If you want to move the FTS folder to a different location, the /site/dict/words file on the iNEWS Servers will need to be edited and adjusted.

6. Using Microsoft Windows filesharing, share out the fts folder on the new FTS PC to your network so that you will be able to map a drive to it from the OLD FTS PC. It will be necessary to get the old FTS index files from the old FTS PC over to the new FTS PC. The easiest way would be to drag and drop them through shared folders on your network.

n Consult your corporate Windows network administrators if you need assistance establishing these shares.

7. Go to the old FTS PC and map a drive to the fts folder you shared out on the new FTS PC.

n Consult your corporate Windows network administrators if you need assistance mapping a drive to this share on the new FTS PC.

8. Determine the numbers of the FTS servers with the list c fts command, such as:

WAVD-A: list c fts

DEV DEVICE_TYPE COMPUTER CCU PRINTER SPEED OPTIONS DEVNAME

S239 ftsindex A N239

S240 ftsseek B N240

WAVD-A:

In the example above, the ftsindex server is number 239 and runs on the A computer. The ftsseek server is number 240 and runs on the B computer.

n Your FTS server numbers and which computer they run on may differ from the example.

9. When you are ready to move to the new FTS computer, go to the iNEWS console and stop the iNEWS side FTS servers. For instance, using the same numbers given in the previous example, the commands would be:

WAVD-A: stop 239

WAVD-B: stop 240

10. Shutdown the ftsindex.exe and ftssearch.exe processes using the FTS configuration and management UI. See “Starting and/or Stopping FTS” on page 15 for more information.

You might want to use Windows Task Manager to verify that both processes are no longer in the process list.

11. Using Windows Explorer, copy the fts index files from the old FTS PC over to the network share on the new FTS PC.

12. Write down the current IP address settings of the old FTS PC, and then change the IP address of the old FTS PC and reboot it to be sure it comes up on a new IP address.


38

13. Set the IP address of the new FTS PC to the IP address of the old FTS PC so iNEWS will know how to find it.

n If you must set the new FTS PC to a new IP address, it would be necessary to edit the /etc/hosts file on the iNEWS servers to match the new IP address of the new FTS PC.

14. Reboot the new FTS PC for it to come up on its new IP address (the IP address of the old PC), then login to the PC to start the ftsindex.exe and ftssearch.exe programs, using the FTS configuration and management UI. See “Starting and/or Stopping FTS” on page 15 for more information. Use this program to configure the FTS services to start automatically after reboot. Check the Processes tab of Task Manager; you should see the ftsindex.exe and ftssearch.exe programs running since you replicated the old computer’s setup earlier.

15. Go to the iNEWS console and restart the iNEWS side fts servers. This example uses the same numbers used in examples for previous steps of this procedure; yours may differ.

WAVD-A: restart 239

WAVD-B: restart 240

Indexing Scenarios

39

Indexing Scenarios

This section will outline some theoretical situations involving indexing, raise questions about these indexing scenarios, and point out what FTS is doing and why it is behaving the way it does when presented with these theoretical situations.

n These scenarios apply to sites not utilizing multiple index bases. For more information on multiple indexes, see “FTS Multiple Indexes” on page 19.

Scenario #1

Let's say Sammy the system manager decides he wants to index some queues in the Wires and Archives directories:

If the system administrator decides to place the number of index requests in the SYSTEM.INDEX queue using the order shown in the table—with archive queues indexed after wires—any new wire stories that come into the system will result in index requests for these new wire stories being placed at the bottom of the SYSTEM.INDEX queue. Additionally, old wire stories that are purged off the system will have an un-index request placed at the bottom of SYSTEM.INDEX, like this:

Name of Directory.Queue Number of Stories in Queue

WIRES.ALL 7,500

WIRES.SPORTS 2,400

ARCHIVES.2009 165,000

ARCHIVES.2010 172,000

ARCHIVES.2011 204,000

ARCHIVES.2012 267,000

7,500 WIRES.ALL

2,400 WIRES.SPORTS

165,000 ARCHIVES.2009

172,000 ARCHIVES.2010

204,000 ARCHIVES.2011

267,000 ARCHIVES.2012

Indexing Scenarios

40

In this scenario, after the 7,500 index requests for WIRES.ALL are processed, users will be able to run FTS searches in WIRES.ALL and get accurate hits...but only for a while. The system will then need to index 840,000 other stories before it comes around to the new wire stories and begins indexing them. Until these new stories are indexed, they will not appear in any FTS search results, which means FTS searches of WIRES.ALL will become inaccurate and incomplete.

Also, if the system administrator set up a 7-day purge interval on WIRES queues, and it takes the system longer than that period to index the hundreds of thousands of stories between WIRES.ALL and the new wire stories, it is possible that some stories get purged before the ftsindex server gets back around to indexing new wire stories. This means some index requests might become invalid by the time all other pending requests have been processed. Consequently, indexing might never get caught up to date for wire stories on the system.

Scenario #2

Let’s say Sammy the system administrator realizes his mistake (in scenario #1) and to resolve the issue, he removes the indexed trait from all of the ARCHIVES queues.

Unfortunately, this compounds the problem. By removing the indexed trait from the four ARCHIVES queues, un-index requests have been placed in SYSTEM.INDEX for these queues, like this:

100 New wire stories

100 Un-index requests for purged wire stories

7,500 WIRES.ALL

2,400 WIRES.SPORTS

165,000 ARCHIVES.2009

172,000 ARCHIVES.2010

204,000 ARCHIVES.2011

267,000 ARCHIVES.2012

100 New wire stories

100 Un-index requests for purged wire stories

165,000 Un-index requests for ARCHIVES.2009



Indexing Scenarios

41

New wire stories are still flowing into the system and old wire stories are still purging off the system.

Scenario #3

Let’s say Sammy the system administrator decides he must resolve the issues now troubling the iNEWS system’s Fast Text Search capabilities—caused from his earlier efforts in scenarios #1 and #2— and to do this, he decides to delete the indexes and start over.

In this scenario, the indexes are deleted, and Sammy is ready to begin assigning the indexed trait to queues all over again. He logs in to iNEWS and applies the indexed trait to the following queues in the following order:

A few minutes after assigning the indexed trait, users are reporting problems with searches; they are getting no results when searching the WIRES.ALL queue, even when they search for words they know appear in the stories.

The problem in this scenario is that with the reorder of queues, it’s unlikely that the WIRES.ALL queue has been indexed yet. The ftsindex server has to process more than 800,000 stories (from the archives) before it will begin indexing the WIRES.ALL queue. This could take several days or weeks. Until then, there will be no entries in the indexes, so no search results will be obtained via FTS.


100 More new wire stories

100 More un-index requests for purged wire stories

Name of Directory.Queue Number of Stories in Queue

ARCHIVES.2009 165,000

ARCHIVES.2010 172,000

ARCHIVES.2011 204,000

ARCHIVES.2012 267,000

WIRES.ALL 7,500

WIRES.SPORTS 2,400

Indexing Scenarios

42

Scenario #4

To fix the problem presented in Scenario #3, let’s say Sammy the system administrator removes the indexed trait from the WIRES.ALL and WIRES.SPORTS queues. By doing so, Sammy has seemingly resolved the problem users were having with not getting search results for WIRES.ALL. Now, when users search WIRES.ALL, which no longer has the indexed trait, the search request is placed in SYSTEM.SEEK so the seek server processes the search request instead of FTS. However, a new problem appears; users begin reporting that searching all of the years of ARCHIVES from the folder level is very, very slow.

n There will still be some index and un-index requests for the wires queues in SYSTEM.INDEX, but FTS will attempt to index stories and then immediately un-index them.

The slow down at the ARCHIVES folder level occurs when the index trait is not set at that level, so any searches at that level would be non-indexed search requests in SYSTEM.SEEK and processed by the seek server. Seek servers search every single word of every single story instead of looking up results in an index, so the process takes longer. While the system manager can assign the indexed trait to ARCHIVES, and that will cause the search requests to be placed in SYSTEM.FTS and processed by FTS, doing so at this point could cause users to receive partial results. For instance, users might only receive results from ARCHIVES.2009. Why? Because there are hundreds of thousands of index requests in SYSTEM.INDEX, and it will take FTS a while to process all of those requests and fully index all stories in ARCHIVES. When users receive results limited to one archive queue (such as ARCHIVES.2009), that most likely means the ftsindex server is still working its way through that archived queue and has not yet gotten to the requests for later years. It could take a day or two, or longer, for all archived queues to become fully indexed.

Scenario #5

Understanding how FTS works is key to making decisions about what gets indexed, when, and in what order. For this final scenario, the best generalized strategy for how to index ARCHIVES and WIRES queues would be:

• Index all static queues, like ARCHIVES, before assigning the indexed trait to dynamic queues like WIRES.

• Assign the indexed trait to one or two static queues at a time, wait for indexing to complete, and then index the next one or two until all are completed. Assigning the indexed trait to dozens of queues at a time will result in inaccurate search results until FTS has had enough time to index all of the assigned queues.

• Index all subsidiary queues beneath a folder first, then assign the indexed trait to the parent folder so that parent level searches become FTS searches.

Where to Find More Information

43

• Make a backup of the Windows-based side index files after the static indexing is done and before assigning the indexed trait to dynamic queues.

• If indexing highly dynamic queues like WIRES, consider setting up a parallel, non-indexed structure of the same material so users have the ability to search with seek instead of FTS. For instance, set up:

- WIRES.ALL — indexed

- WIRES.ALL2 — non-indexed

Where to Find More Information

Avid iNEWS documentation can be found on the Avid Customer Support Knowledge Base. Version 4.x documentation is located here:

http://avid.force.com/pkb/articles/en_US/readme/Avid-iNEWS-v4-5-Documentation

Legal NoticesCopyright © 2013 Avid Technology, Inc. and its licensors. All rights reserved.

Attn. Government User(s). Restricted Rights LegendU.S. GOVERNMENT RESTRICTED RIGHTS. This Software and its documentation are “commercial computer software” or “commercial computer software documentation.” In the event that such Software or documentation is acquired by or on behalf of a unit or agency of the U.S. Government, all rights with respect to this Software and documentation are subject to the terms of the License Agreement, pursuant to FAR §12.212(a) and/or DFARS §227.7202-1(a), as applicable.

All trademarks contained herein are the property of their respective owners.

Avid iNEWS Fast Text Search • 9329-65319-00 Rev A • August 2013 • Created 8/19/13 • This document is distributed by Avid in online (electronic) form only, and is not available for purchase in printed form.

Setup and Configuration Guide -...

Documents

Transcript of Setup and Configuration Guide -...