Document: Application Note for Witness Systems

Product: Witness Systems IP Recording Suite

Call Manager V 3.3 CallManager

Revision: 2

Date: October 15, 2003

Author: Maxine Dye

Application Note for Witness Systems

Contact Store IP

Table of Contents

1 General Information.....................................................................................................32 Main Contact Information...........................................................................................33 Witness Contacts.........................................................................................................34 Witness Support Contact Information.........................................................................45 Witness Support Detail................................................................................................4

5.1 General Information.............................................................................................45.2 Severity Description and Response Times..........................................................55.3 Support Level Definitions....................................................................................55.4 Call Flow Description..........................................................................................6

6 Witness Support Escalation Procedures......................................................................77 Witness Product Information.......................................................................................88 Witness Product Overview..........................................................................................89 Witness Product Features............................................................................................810 Witness Product Performance (as tested)................................................................811 Witness Product Security Considerations..............................................................1012 Packaging, Bundling, Upgrading...........................................................................1013 Overview of Integrated Solution (as tested)..........................................................1014 Network Designs/Topologies (as tested)...............................................................11

14.1 Sample network designs/topologies..................................................................1114.1.1 Simple Topologies.....................................................................................1114.1.2 Single Site, Single Gateway and E1/T1 Trunk..........................................1214.1.3 Single Site, Dual E1/T1 Trunks.................................................................1214.1.4 Hub and Satellite Sites...............................................................................1314.1.5 More Complex Scenarios..........................................................................14

15 Witness Product Requirements..............................................................................1516 Witness Product Installation Procedures (as tested)..............................................1617 Witness Product Isolation Procedures...................................................................1618 General Witness Product Considerations..............................................................16

18.1 Limitations of this release..................................................................................1618.2 If you alter your server......................................................................................1718.3 If you alter your network...................................................................................1718.4 If you alter your IP telephony system................................................................18

19 Troubleshooting.....................................................................................................1919.1 Debugging and Diagnostics...............................................................................1919.2 Error Messages..................................................................................................2019.3 Tools / Aids........................................................................................................2219.4 List of Commands..............................................................................................24

20 Redundancy and Recovery....................................................................................25

20.1 Redundancy.......................................................................................................2520.2 Recovery............................................................................................................2520.3 Dual Spanning...................................................................................................25

21 Interoperability Test Results..................................................................................2522 Cisco Product Information (as tested)...................................................................2523 Cisco Product Interfaces (as tested).......................................................................2624 Cisco Product Requirements (as tested)................................................................2625 CallManager Device Table....................................................................................26Appendix A........................................................................................................................27

Glossary of Terms:

API Application Program InterfaceA set of routines used by an application program to direct the performance of procedures by the computer's operating system.

AVVID Architecture for Voice, Video and Integrated Data.

CRM Customer Relationship Management

CTI The practice of using a computer to control one or more telephone and communications functions.

IVR Interactive Voice ResponseA computer that operates through the telephone system, in which input commands and data are transmitted to the computer as spoken words and numbers, or tones and dial pulses generated by a telephone instrument; and output instructions and data are received from the computer as prerecorded or synthesized speech.

IIS Internet Information ServerMicrosoft's brand of Web server software, utilizing HTTP (Hypertext Transfer Protocol) to deliver World Wide Web documents. It incorporates various functions for security, allows for CGI (Common Gateway Interfaces) programs, and also provides for Gopher and FTP (File Transfer Protocol) services.

IP Internet Protocol

LAN Local Area Network

PSTN Public Switched Telephone NetworkPublic telephone system.

RAID Redundant Array of Independent DisksA data storage method in which data, along with information used for error correction, such as parity bits or Hamming codes, is distributed among two or more hard disk drives in order to improve performance and reliability. The hard disk array is governed by array management software and a disk controller, which handles the error correction.

RTP Real-Time ProtocolAn Internet-standard network transport protocol used in delivering real-time data, including audio and video.

SCCP Skinny Client Control ProtocolA Cisco protocol used for communication between a gateway and Cisco CallManager.

SPAN Switch Port Analyser.

1

VLAN Virtual Local Area NetworkA local area network consisting of groups of hosts that are on physically different segments but that communicate as though they were on the same wire.

VoIP Voice over Internet ProtocolThe use of the Internet Protocol (IP) for transmitting voice communications. VoIP delivers digitised audio in packet form and can be used for transmitting over intranets, extranets, and the Internet.

2

1 General Information

Witness Systems provides a business-driven multimedia recording, performance analysis and e-learning management solution that enables companies to:

Enhance the quality of their customer interactions Achieve greater customer retention and loyalty Develop staff Generate additional revenue

2 Main Contact Information

Main Location Additional Location Additional LocationAddress Corporate Headquarters

300 Colonial Center ParkwayRoswell, GA 30076USA

European HeadquartersKings Court, Kingston RoadLeatherheadSurrey, KT22 7SL

Asia Pacific HeadquartersRoom 1809-1818, 18FShui On Centre6-8 Harbour RoadWan Chai Hong Kong

Main Telephone

1-800-494-8637 +44 (0) 1372 869 000 +852 3198 6000

Main Fax 7707541888 +44 (0) 1372 869 000 +852 3198 6100General E-mail support@witness.com

3 Witness ContactsEMEAName Phone EmailUSA 1 800 494 8637 support@witness.com

Leatherhead 0800 496 1111 cic-emea@witness.com

Asia Pacific: support@witness.com

ANZ (within Australia)

1 300 367 394 support@witness.com

(elsewhere) 61 2 9232 5833 support@witness.com

RoAP (rest of APAC)

+852 8103 0104 support@witness.com

3

4 Witness Support Contact Information

Team Main Location (USA) Europe – UK Asia PacificPhone 1 800 494 8637 0800 093 1835 (out of

hours – for customers with out of hours contracts)0800 496 1111 (7:00 – 19:00)

1 300 367 394 (Aust)61 2 9232 5833 (ANZ)+852 8103 0104 (rest of Asia Pacific)During business hours Out of hours pager service available

Fax 770 754 1888 01372 869 005 N/AE-mail support@witness.com cic-emea@witness.com N/AHours Phone support is 24

hours a day from 3:00pm Sunday – 8:00 pm Friday (excluding holidays); Off hours (emergency based support) is 8:00 pm Friday to 3:00 Sunday.

N/A

The IPTelephony category requires 24x7 support contact. List all applicable after hours numbers.

5 Witness Support Detail

5.1 General InformationWitness Systems will respond to customer telephone requests for assistance in the order in which calls are received. For Standard Support telephone requests, Witness Systems’ intent is to answer its customer phone inquiries directly. If all Witness Systems CSR are working with other customers at the time of a request, the customer calling for assistance can expect to receive a return call from a Witness Systems CSR within 2 hours of the original call to the company. A call directed to the Hotline number outside standard support hours for critical issues will receive immediate attention and a return phone call within 30 minutes.

In addition, Witness Systems also offer on-line support where customers can log in using unique log in identification details to report and monitor progress of issue resolution.

5.2 Severity Description and Response Times

4

Priority/SeverityDescription Response TimeCritical Current WITNESS® performance prohibits the

customer from conducting a critical business function, thereby creating an “emergency” situation

Immediate assignment of service resources to research and correct the problem as soon as possible.With progress updates every 4 hours

High Current WITNESS® performance has significant impact on a business function, but the customer's business is still functioning.

5 business days.With progress updates every day.

Medium Current WITNESS® performance has a moderate impact on a business function, but the customer’s business is still functioning.

Workaround in 15 business days and / or Next Service PackWith progress update every 5 business days.

Low Current WITNESS® performance has a minor impact on the customer’s business operations.

Incorporated into next release activities.With What’s New in release documentation.

5.3 Support Level Definitions

Level DescriptionBronze 09:00 – 17:00 Monday to Friday inclusive but excluding all Bank and Public

Holidays. Response time = 4 hours; On Site Support = 8 hours.Silver 09:00 - 17:00 Monday to Friday inclusive but excluding all Bank and Public

Holidays. Response time = 2 hours; On Site Support = 4 hours. Gold 07:00 - 21:00 Monday to Friday inclusive but excluding all Bank and Public

Holidays. Response time = 2 hours; On Site Support = 4 hours.Platinum 00:00 – 24:00 Monday to Sunday inclusive of all Bank and Public Holidays

Betw 07:00 –21:00 5 days Response time = 2 hours; On Site Support = 4 hours.Betw 21:00 – 07:00 and 24 hr weekends Response time = 2 hours; On Site Support = 6 hours

5

5.4 Call Flow Description

6

6 Witness Support Escalation Procedures

Elapsed Time Priority 1Urgent

Priority 2High

Priority 3Medium

Priority 4Low

1-Hour CIC Manager CIC Manager CIC Manager CIC Manager

4-Hour CIC Director/ES Manager CIC Manager CIC Manager CIC Manager

24-Hours SVP Global services CIC Director/ES Manager

CIC Director/ES Manager

CIC Manager

48-Hours SVP SVP Global services

CIC Director/ES Manager

CIC Director/ES Manager

72-Hours SVP SVP Global services

CIC Director/ES Manager

CIC Director/ES Manager

96-Hours SVP SVP SVP Global services CIC Director/ES Manager

For USA Escalation Contacts

Title NamePhone NumberCustomer Interaction Centre N/A 1 800 494 8637Customer Interaction Centre ManagerN/A 0770 7548644Director Customer Interaction Centre N/A 678 254 5363

SVP Global Services N/A+44 (0) 1372 869 401- or -+44 (0) 7767 791 599

For EMEAEscalation Contacts

Title NamePhone Number

Customer Interaction Centre ManagerN/A+44 (0) 1372 869518 - or -+44 (0) 7879 607 123

Engineering Support Manger N/A +44 (0) 7767 791 581Director EMEA CIC N/A +44 (0) 7770 422 788

SVP Global Services N/A+44 (0) 1372 869 401- or -+44 (0) 7767 791 599

SVP EMEA N/A+44 (0) 1372 869 445- or -+44 (0) 7968 486 882

7

For Asia PacificEscalation Contacts

Title Name Phone NumberANZ ProfSManager Terry Halvey +61 2 8907 0336RoAP ProfS ManagerRisk Israel +852 3198 6226

ProfS Director Ian Millward +61 2 8907 0330SVP Nigel Hewett +852 3198 6000

President Nick Discombe+44 1372 869 468

7 Witness Product Information

8 Witness Product OverviewThe Witness Systems Contact Store IP architecture provides 3 levels of products to suit the different needs of Cisco customers to record and live monitor their IP telephony conversations. These products are differentiated by functionality and the number of simultaneous recordings they can handle. They are aimed at meeting different business requirements of different user/customer groups. The following table lists the products and related customer groups.

PRODUCT CUSTOMEReQuality ContactStore Exec

CEO’s, VPs – record an important call only.

eQuality ContactStore IP Micro

As above, aimed at small call centre and bank level.

eQuality ContactStore IP

As above, with CTI and CRM integration with multi-site support and application integration.

eQuality eWare and eQuality Unify are elements to the product suite that are embedded in the product that manage the database and external CTI interfaces respectively. They are not exposed to the customer unless a large enterprise solution is implemented.

9 Witness Product FeaturesSee table below

10 Witness Product Performance (as tested)The Witness Systems’ recording suite of software combines scalable platform architecture to provide functionality to customers to suit their needs and budget. The aim of the suite of software is to allow a customer an entry level solution that can be upgraded without the need to replace any element of previously purchased software. The following table shows the different performance capabilities of each level of the suite.

8

Product CapabilitiesContact Store Exec Base System has server licence and bundled 2 channels of on-

demand recording – 'Contact Store Exec' extension licenses. The system has a maximum capacity of 30 Extensions. Bundled is a single concurrent Eyretel Insight Viewer client license,

which cannot be expanded. Multiple user accounts for the single license is possible. No live monitoring is available; this requires an upgrade to Eyretel

Observer. No full-time recording is available; this requires an upgrade to

Contact Store IP Micro

Contact Store IP Micro Capabilities as Contact Store Exec, but also including: Base System has server licence, database application and

bundled 1 Eyretel Insight Viewer client license. Support for 1-30 channels of full time recording or Contact Store

Exec Multiple Contact Store IP Micro recorders cannot not exist in the

same network, upgrade to Contact Store IP is needed Single Eyretel Insight Viewer client license is expandable to 5

concurrent licenses Any of the recorded channels can be live monitored with the

purchase of an Observer client license. The maximum live monitoring of conversations with Observer is 5.

Contact Store IP As Contact Store IP Micro, but also including: Base System has server licence, database application and

bundled 5 concurrent Viewer client licenses. Support for SQL Server 2000 Enterprise Database Support for Multiple Recorders forming an enterprise Recording

network Support for 1-60 channels of full time recording per server Support for Replay Studio cal analysis solutions Support for QualityCall Call Evaluation suite Support for Performance Manager Support for 3600 Busy Hour Call Attempts (BHCA) Each recorded channel can be configured as a full-time recording

or Contact Store Exec channel Any of the recorded channels can be live monitored with the

purchase of an Eyretel Observer client license. The maximum live monitoring of conversations with Observer is 5.

Minimum machine specification is 2.4 Ghz pentium 4 processor, 1 Gb RAM; 250 Gb HD.

Consolidation time of 60 calls a minute. Recording size of 8 Kb per second.

9

11 Witness Product Security ConsiderationsWitness Systems’ recorder system utilizes Windows 2000 Server and as such there is the scope for an organization to undertake their ‘usual lockdown’ procedure for servers. The scope and the level to which an organization secures their server(s) is varied and at the discretion of that organization.

The main general area for securing the system lies in secure access to replay calls.

The users associated with particular extensions are set in the Administrator console and the usernames themselves are derived from one of 2 scenarios: a) The Windows 2000 Local User accounts of the IP Recorder or b) The Domain/Active Directory users of the Domain the system is hosted on.

Whether the system uses local users or domain users is normally decided by the size of the system and organization. With 1 or 2 users needing to replay in an entire system it is preferable to use standalone users to simplify the install, whereas more than 5 users would mandate integration to existing domain users.

Achieving the different levels of security is purely an Internet Information Server setting.

12 Packaging, Bundling, UpgradingThe 3 levels of product are each licensed for specific capabilities and also maximum number of concurrent recordings achievable.

Additionally the product is capable of being licensed on a timed-trial basis, i.e. 5-days, where at the end of the trial the customer has the option of buying a full license key to continue the use of the system in a non-destructive upgrade.

13 Overview of Integrated Solution (as tested)At the highest level, the Witness Systems’ recording system consists of three conceptual sub-systems: The Capture subsystem, which captures information in real time. The Storage subsystem, which compresses and retains these recordings for use later. The Exploitation subsystem, which lets you exploit the recordings in your business.These are shown in Figure 2-1 along with the major data flows through the system. In many cases, these three components are built into a single PC often just referred to as a ‘recorder’ but in larger systems, the three components may be split across your entire enterprise.

10

Figure 2-1 Witness Systems Generic Recording System

The most effective way to ensure normal operation of each of these subcomponents is to test the ‘exploit’ section, as for this to work, the other two must by definition be working.

To do this, simply record a call and attempt to replay that call and check the audio quality.

Contact Store IP performs replay via a web-browser based application (Viewer) that can be used from Internet Explorer V5.5 (Media Player 7.x).

14 Network Designs/Topologies (as tested)14.1 Sample network designs/topologies

If the Cisco CallManager topology and recording requirements fall within any of the standard scenarios described in Section14.2.1, configuration is straightforward and the options are detailed here.

If the required system does not match one of these simple configurations, you will need to review the information described below to ensure that an appropriate topology and system sizing is determined. This is generally isolated to larger telephony sites where the permutations and combinations are too varied for the scope of this document.

Contact Store IP is licensed according to the maximum number of concurrent extension recordings that the recorder will support so, in all cases, you must determine how many concurrent extension recordings are required.

14.1.1Simple TopologiesThe following commonly met scenarios are easily met by the configurations and topologies described in the following paragraphs:

11

14.1.2Single Site, Single Gateway and E1/T1 Trunk

Figure 1-1 Single Site, Single Gateway and E1/T1 Trunk

The system matches this category if: Only calls originating or terminating in the PSTN are recorded.

There is a single Gateway with a single T1 or E1 trunk over which all such calls are routed.

The CallManager is connected to the same Catalyst switch as the Gateway.

IP phones may be connected to this or other switches via routers.

The Contact Store IP recorder is connected to a port on the same Catalyst switch as the Gateway and CallManager.

All desks to be recorded use IP phones as opposed to SoftPhones.

If the system meets these criteria: Configure the port to which the recorder is connected to SPAN both the CallManager’s port

and the Gateway’s port.

Select a PC of specification at least equal to the ‘Minimum recommended’ entry in Witness Systems’ IP Pre-requisites document.

14.1.3Single Site, Dual E1/T1 Trunks

Figure 1-2 Single Site, Dual E1/T1 Trunks

If the required system meets all the criteria of paragraph 14.1.2, except that there are 2 E1 or T1 trunks connecting the Gateway to the PSTN, adopt the same configuration as above

12

but ensure the PC specification is at least equal to the ‘Preferred’ entry in IP Pre-requisites document.

14.1.4Hub and Satellite Sites

Figure 1-3 Hub and Satellite Sites

The system matches this category if: Only calls originating or terminating in the PSTN are recorded.

A single Router links the Satellite site to the central hub site.

IP phones are to be recorded at one or more satellite sites.

The Contact Store IP recorder is connected to a port on the same Catalyst switch as the Router to the Satellite site.

All desks to be recorded use IP phones as opposed to SoftPhones.

The link to the remote site is 8Mbps or less.

No more than 60 concurrent calls will be carried over the link between central and satellite site.

If your system meets these criteria: Configure the port to which the recorder is connected, to SPAN the port through which all

traffic to the satellite site travels.

Select a PC of specification at least equal to the Preferred entry in the Witness Systems IP pre-requisites document.

Size the disk storage accordingly.Where there are multiple small satellite sites, a single recorder may be set to SPAN multiple sites provided the total bandwidth does not exceed that stated above.

In sizing this topology you need to determine the maximum concurrent number of external calls that can be placed from the remote site. This is often determined by the

13

bandwidth of the link to that site and is typically very much lower than the total number of IP phones at that site unless the site is being operated as a Call Centre, in which case up to 100% load must be catered for.

If a single router is used both to route traffic to the remote site and as a PSTN gateway, then calls made from that site that go via the gateway will not be visible to the central switch. In such cases, you must reconfigure your network to ensure that all calls are transmitted via the port being SPANned.

14.1.5More Complex Scenarios If the topology and requirements do not meet one of the more common scenarios described, you will need to determine the system sizing. You should determine the information detailed in the paragraph below:

RecordingFirst determine whether you need to record: Internal calls – within the CallManager system.

External PSTN calls – between one party on the CallManager system and a distant party on the far side of a gateway.

External IP call – between one party on the CallManager system and a distant party on the far side of a firewall and ISP up-link.

For each of the above call types determine which transmission types (G.711 or G.729A) are to be used and in what proportion. Moving over to G.729A entirely will require much less processor power on the recorder.

You should also determine how many concurrent calls of each of the above types will be occurring and, of these, how many must be recorded.

Call Type %G.711

%G.729A

Max concurrent Calls

Max concurrent Recordings

InternalExternal PSTNExternal IP

15 Witness Product RequirementsTo install eQuality Contact Store IP you will need the following:

A working Cisco CallManager system (Version 3.3) with at least one pair of IP phones.

A non-switching Ethernet hub or a switched hub or router that supports Switch Port Analyzer (SPAN).

14

Ethernet drop lead for the computer that is to be used as a recorder. A high speed computer with Windows 2000 Server installed.

A single processor is required, due to limitations of Windows 2000, whereby the packet driver cannot run with more than one processor.

Windows 2000 Service Pack 3. Java Development Kit (version 1.2) installed on the pc where you are installing

eQuality ContactStore IP. If eQuality Viewer is to be installed, you will need to install Microsoft Message

Queue and IIS. A PCI Ethernet Network Interface Card (NIC). Windows 2000 drivers for the Ethernet Card if not pre-installed. SQL Server 2000 Installation kit. eQuality Contact Store Installation CD. Witness Systems licence code for the required channel count. At least 6 gigabyte (Gbyte) free disk space for installation of the Calls folder.

You must be logged in as a local system administrator on the computer on which you are installing this software.

15

LAN configurationFor VoIP recording to work, it is essential that the recorder can intercept the call control packets passing between the IP phones and the call managers, and the voice streams passing directly between the two parties engaged in each call.For a Cisco implementation, you are recommended to use the SPAN algorithm to set up a port on the Ethernet switch to receive copies of all the packets needed. Refer to the System Administration Guide, for guidance on topologies.Refer to the Ethernet switch documentation to set this up.

Platform InstallationTo prepare the computer ready for the installation of the Witness Systems components, the following steps are required:

Disable all power-saving features.These could interfere with the recorder’s ability to start recording at any time.

Install correct Operating SystemWindows 2000 Server must be installed and Service Pack 3 must be applied. Windows 2000 upgrades are available as an option on the installation CD.

Install and set up Internet Information Server (IIS)Ensure that IIS has been installed as part of your Windows 2000 Server installation and that all necessary IP configuration has been undertaken to allow the PC to act as a web-server.

Install Message Queuing ServiceThese services can be installed by the Message Queuing Service Windows Component

Install Ethernet Card and DriverInstall the Ethernet card, checking for resource conflicts. Prove connectivity over TCP/IP to the outside world. Ensure the PC has a valid IP address (can be statically assigned or via DHCP).

16 Witness Product Installation Procedures (as tested)Please refer to eQuality ContactStore IP Installation Guide document

17 Witness Product Isolation Procedures

18 General Witness Product Considerations18.1 Limitations of this releasePlease read the following limitations before submitting problem reports or enhancement requests.

Confirm that the PC you are using is capable of sustaining recording under peak load conditions. Guidelines are given earlier but PCs can vary considerably in performance and Witness Systems recommend that you confirm loading figures on your system.

Liability of Witness Systems for failure to record any calls is limited under the terms of supply.

16

A limited range of PC configurations have been tested to date. Configurations recommended at this time will include significant contingency - which may prove over cautious in the medium-term.

A limited range of compression formats are supported. This list will grow and if an as yet unsupported format is to be received or is required as the long term storage format, please contact Witness Systems for the latest position.

Configurations above 90 concurrent recordings, that is, requiring multiple recorders or separate database/disk servers must be designed and verified by Witness Systems.

It should be noted that only calls between extensions marked for recording will be recorded. Therefore, if a call is transferred from a recording extension to one which is not recorded, recording will cease.For example, when A calls B, who forwards to C, if B only is marked for recording, then the forwarded call (from A to C) is not recorded.

If Executive Record is enabled, even though there is no active call, the phone displays the message:Call is being recordedYou can clear this message by selecting the Exit button.

If you have the options Record all calls and Executive Record enabled, please note that Record all calls will override Executive Record.For example, a call between a marked extension and an executive extension will be recorded.

If Executive Record is enabled and a call is being recorded, if another incoming call arrives, causing an interrupt, then the recording is stopped and Executive Record has to be manually enabled again when the original call is retrieved.

18.2 If you alter your serverIf you make any changes to the setup of your server you should take a ghost image of it to allow it to be reconstructed quickly in the event of major hardware failure.

18.3 If you alter your networkContact Store IP can only record calls whose voice packets are presented to the port(s) connecting it to the network.

The recorder also has a finite capacity, both for filtering required calls out of the incoming packet stream and for the number of calls that it can record concurrently.Any changes you make to your network topology can potentially impact the recorder’s performance and you should follow these guidelines whenever you change your network.

Changes to IP addressesIf you change the IP addresses of any components of the system, you should perform a full commissioning test to confirm correct recording of all required calls.

Changes to network topologyIf you change the topology of your network you should check all SPAN port settings and then perform a full commissioning test to confirm correct recording of all required calls.

17

18.4 If you alter your IP telephony system

Adding new telephonesIf you add additional telephones you should check SPAN port settings and the appropriateness of existing tap points, adding additional taps as required.

You should then confirm that calls to and from these new phones are recorded correctly.Where fall-back IP-PBXs or clusters are used, you should force the new phones to use each server in turn and confirm that calls are still recorded. It is possible, for example, to have a backup server that is not correctly configured in the required VLAN. Such errors can go unnoticed for long periods until the time when the main server fails and operation switch to the standby server.

You should also confirm that the total loading on the server is still within its capability.

Adding additional gateway capacityFollow the same processes as for additional telephones:

Check SPAN settings Confirm recording of calls via these ports Confirm recording under failover scenarios Confirm load is acceptable

In this case, traffic tends to increase gradually after additional capacity is installed, hence you should pay special attention to loading over the following few weeks until it stabilises.

Configuring your hard diskYour hard disk must be configured to have a minimum space of 30,000. You use the System Administration tool to configure this.

P R O C E D U R E

1 To do this, from the System Administration tool, expand the tree, locate and click on the ContactStore Storage Processing node, then in the right pane, check the setting against the Minimum space to maintain entry.

2 Ensure the value is set to 30,000.

There is no deletion setting. When your disk becomes full, it will automatically delete.

19 Troubleshooting

18

19.1 Debugging and DiagnosticsShould you experience problems with your Contact Store IP recorder, there is a wide range of diagnostic tools available to help you determine the cause of and ultimately resolve the problem. In summary you have at your disposal:

Several components of the Windows 2000 system management tool set are of use. Most can be accessed via the Microsoft Management Console (MMC) snap-in provided. These include:

Windows Task Manager. To see if the appropriate tasks are running. (The following tools have been included in the default console settings of the

Administrator tool for your convenience). Windows Service Manager. To allow you to stop and start services manually. Windows Performance Monitor. To observe a wide range of performance and

diagnostic counters – this can directly accessed from within the Administrator console under the system monitor branch.

Windows Event Log Viewer. To observe any service start/stop/error messages generated by the Contact Store services.

In addition, each Contact Store service outputs more detailed diagnostic information into log files accessible via the Administrator console.

You may also choose to run each service from a command-line prompt, giving more immediate, real-time visibility of any major events occurring.

Witness Systems’ CTI Studio can be used to observe and modify the behaviour of the Unify service as it communicates with the other components in the system.

You can also use all of the SQL Server administration tools to examine the contents of the database as recordings progress.

19

19.2 Error MessagesThe following are a list of known error messages and issues.

eQuality Contact Store service(s) failCheck the Application Event log, see example below, for error messages using the Windows Event Viewer or the MMC snap-in provided.

Shut the service down and run as console application with the –c command-line option. Also check for the log files Capture.log, Compressor.log, Consolidator.log and DiskManager.log in <installation dir>\log files (The default directory is C:\Program Files\Eyretel\Contact Store IP\Log files).If you run one of the executables from the command line, the log file will be located in the path from which you ran the executable.If the Contact Store services will not start, check the PC’s node name. Problems have been observed when the computer's nodename (as recognised by SQL server for example) differs from that shown in the properties of the LAN connection and in the Network Identification tab of My Computer properties.

Solution:Only solution found to date has been to reset the Network node name to match the underlying name.

Calls observed but no audio

Solution:Check the network topology carefully. If using a dumb, non-switching repeater type hub, beware of mixing 10 and 100Mbps ports as this tends to stop RTP packets from being received reliably.If in doubt, plug the recorder PC into the back of an IP phone from which calls are being made. If the recorder is functioning correctly, it will record calls to/from that phone.

Service functions but no performance counters visible

Solution:Confirm that the service is running under the user account EYR_Contact7k. This account should have administrator privileges.

20

You may also find that the SNMP service was running when the installation program tried to install the performance monitor variables. The current installation kit tests for and warns if this service is running but as it relies on the user to stop the service, it may not succeed in installing the performance monitor variables. The only option here is to remove eQuality ContactStore IP and reinstall with SNMP service stopped.The performance counter can also equal zero if you are viewing them through terminal services.

Configuration under Tools/Options failsWhen setting the eWare details a dialog box stating ‘Error in setting the Eware2 Directory’ is displayed.

Solution:P R O C E D U R E

1 Double check the password entered for User EWCODE in SQL Server.

2 Create the directory C:\Program Files\Eyretel\Eware2\Log Files and try again.

3 Examine the log file produced in this directory.

Cannot search for calls

Solution:Confirm that the databases ‘EwareCalls’ and ’EwareConfig’ are present within the SQL Server.

If call details are visible within the SQL server’s EwareCalls Database and call files are appearing in the correct call path, but cannot replay calls successfully in Vision, try the following:

Search Manager setupConfirm the registry key:HKEY_LOCAL_MACHINE\SOFTWARE\EYRETEL\Search Manager\Config\Path is set to the node name of the PC running the SQL server as the example shown below. Do NOT include ‘\\’ at the start.

Ignore other settings shown above.

21

Missed call, audio out of sync, missing call party identification information

Cause:These types of issues can be caused by the use of anti-virus software. Use of any anti-virus software will cause server performance issues, in particular packet loss.

Solution:We recommend if you need to run anti-virus software, that you use Norton AntiVirus version 7.5 or greater and make the following changes:

Run with File System Realtime Protection scanning disabled. Schedule scanning to occur outside of recording hours and avoid full system scans. Exclude .DAT, .TMP, .XML and .WAV files from scans. Exclude the following paths

C:\Calls C:\Program Files\Witness Systems\ContactStore\PersistQ C:\Program Files\Witness Systems\ContactStore\ArchiverTemp

C:\Program Files\Witness Systems\Unify\PersistQ These 4 paths assume the default installation path, if you set custom paths during the

install, you will need to exclude these.

19.3 Tools / Aids

Performance logs and alertsThis branch gives you access to the standard Windows Performance logs and events utility. You can use these branches to set up regular monitoring and logging and to alert you to system events and exceptions such as lack of disk space, overload etc.

Counter logsYou can use this branch to log periodic samples of performance monitor variables.

Trace logsYou can use this branch to log system events such as process creations and deletions.

AlertsUsing the Administrator’s System Monitor node, you can observe, via Microsoft’s Performance Monitor, the behaviour of the Contact Store IP system. A default view is provided which shows all the variables that are available. To alert you to incorrect operation or system problems, however, the system uses a range of system alerts set through this branch of the Administrator.A number of triggers are also defined which indicate abnormal performance. You may wish to customise these to suit your needs and/or alter the actions that will be taken should the trigger fire. This can include generating SNMP traps, pager messages, emails etc. Refer to the help function on Microsoft Management Console for instructions on how

22

to achieve these. Most properties can be set by selecting one of the alerts in the right-hand pane and right-clicking it. Select Properties to adjust the threshold and/or actions to be taken when the alert is triggered. For example, you can select to have alerts pushed to a different PC from the server if you wish to track problems from your desktop PC.

A L A R M M E S S A G E P R O B L E M A C T I O N

Failed to Compress Compressor unable to compress audio files

Check .wav files are being produced, disk is not full.

Failed to Consolidate Consolidator unable to upload call details into database.

Check .xml files are being produced; check database is running and accessible

Disk Space Low Disk Manager cannot free disk space

Check for other applications using excessive disk space.

Failed to Delete Disk Manager unable to delete files

Reboot to clear any erroneous file locks. Check for read-only files in recording path.

Excessive Memory Buffer Usage

Overload causing memory buffer to backlog.

Increase CPU power or memory or decrease traffic loading.

Losing Audio Packets More than a few packets per second being lost

Check network loss rates, check for overload.

Excessive Packet Rate Too many packets per second being received

Reduce packet load through better SPAN settings.

Concurrent Usage Nearing Limit

Within 10% of allowed concurrent loading.

Increase server capacity to full Contact Store IP.

Archive Media Full DVD-RAM disk is full Change disk

Archive Drive Empty No disk in DVD-RAM drive

Insert and format clean disk in DVD-RAM drive.

23

ServicesThis branch gives you access to the standard Windows Service control utility. You can use this to start, stop and observe the status of the various services running on the computer. Of particular interest are the following services:

Contact Store IPCapture Contact Store Compressor Contact Store Consolidator Contact Store Archiver Contact Store Disk Manager Unify

All of these should be running permanently. On installation, all of these services are set to restart automatically on failure. You may wish to have the system alert you should these services fail. Select the service required, right-click and select Properties on the shortcut menu to change these recovery options.

19.4 List of CommandsInstead of running MediaStore.exe as a service, it can be run from the command line. This is particularly helpful when first configuring the system as it gives immediate feedback of calls being recorded in the command window,When run from the command line, IPCapture.exe supports the following options:

-c run as console application rather than a service-i install as a service (will fail if already installed).-r de-install as a service-f accept input from a test file rather than the network

When run with the -c option, each service will display text indicating when the service is fully initialised and ready. Each of the services will then display the following service specific information when operating correctly:

ContactStore Service

Shows… When…

Consolidator Processing <filename>

Unify instructs it to upload call details to the SQL Server

Consolidator DONE Database record inserted successfully

Compressor Processing <filename>

Unify instructs it to compress .WAV file specified

Compressor DONE .WAV file compressed and original deleted

Disk Manager Entering X:\yyy\zzz\aa\bb

Files in the specified directory are next candidates for deletion

24

20 Redundancy and Recovery20.1 RedundancyUtilizing standard Hardware platforms the resilience of the server itself can be dictated by the customer and any Cost/Benefit analysis they derive. Witness Systems strongly recommends that a minimum of RAID1 configured mirrored hard disks be deployed.However for further redundancy it is possible to implement a distributed architecture to the system. Following is a summary of some of the elements that can be added for redundancy:

1. Multiple storage locations – replicate the audio and XML file created for each call to a secondary device – i.e. Off site, SAN etc.

2. Utilize a centralized SQL Server Database.

3. Replicate & confirm backup of the Call details in the SQL 2000 Database – remember that the underlying data for each call is still stored as an XML file and hence can be recovered by Witness Systems’ Professional Service in the event of a total failure.

4. Connect to multiple switches each with the same SPAN algorithm allowing for redundant feeds from the switches (note this would require the use of min. Adaptec DualPort Ethernet card).

20.2 RecoveryWitness Systems’ IP Recording suite of products is totally passive to the AVVID environment. Hence a failure of the system and subsequent automatic or manual recovery has NO impact on the AVVID system.

20.3 Dual SpanningTo achieve maximum resilience and redundancy in the system, you can use double spanning.

21 Interoperability Test Results

22 Cisco Product Information (as tested)Cisco CallManager 3.3.Cisco 7960 phones.

23 Cisco Product Interfaces (as tested)Witness Systems’ Contact Store IP does not interface directly with to the Cisco Product Interfaces. It passively decodes the SCCP messages.

24 Cisco Product Requirements (as tested)The minimum requirements for the system are those needed for a basic Cisco CallManager demo system, providing basic internal call functionality:

25

a working Cisco CallManager system V2.4 or higher at least one pair of IP phones, 3 phones provide the ability to transfer calls etc.

7960, 7940 & 7910 phones are supported Currently Cisco SoftPhone 1.1 is NOT supported. Although Softphone 1.2 is

supported. a non-switching Ethernet hub or a switched hub that supports the “RMON SPAN”

algorithm. Optionally the Visual Switch Manager web administration tool for configuring

Cisco switches 2 Ethernet drop leads for the PC to be used as a recorder – one for SPAN connection of the

Voice VLAN and one for Data LAN connection to service Client replay requests.

25 CallManager Device TableNO CallManager Devices need to be configured.

26

APPENDIX APatch Files Changed Description

Number Filename VersionCSIP0009 This is a rollup patch for

IP please read attached readme.

See readme.txt

CVWR0016 EyrInfQueuedCache.dll 7.1.0.134 Replay can be very slow when a UNC calls path is used instead of accessing the Calls share/folder directly for an IP capture platform.

This patch does not affect HTTP retrieval. If HTTP interfaces are used they will not be improved

CVWR0017 player.htcactionflow.htcreplaymanager.htc itemsetview.htc

The top (or near top) line of the player.htc, actionflow.htc, replaymanager.htc, and itemsetview.htcfiles will read as below:<!-- Patch CVWR0017 -->

If a user attempts to replay multiple calls quickly and repeatedly the Contact Viewerbrowser application can get into a state in which it does not replay subsequent calls.Sometimes it recovers but the user can be in the situation where clicking a call doesnot result in a replay.

CSIP0012 IPCapture.exe

RTPHandler.dll

NGAComponent

Unify.spt

3.2.7.151

1.0.0.28

7.1.0.3

7.1.0.12

This patch improves the performance of the system, and adds extra diagnosticvariables.

CSIP0014 Observer.exe 7.2.0.26 This patch fixes a number of issues in observer:- When an extension was monitored it would be highlighted in green in the calls view, but when that call was terminated and a new call started on the same extension the call would be monitored but the call would not be highlighted in green.- The stop button would not work and you would continue to hear audio even after pressing stop!- On a machine that has a full eventlog Observer would crash with an error message.

27