Download - WVR Troubleshooting

IBM Software Group

®

Presented by Yuchin [email protected]

WebSphere® Support Technical Exchange

Troubleshooting on WebSphere Voice Response 4.2 on AIX

IBM Software Group

WebSphere® Support Technical Exchange 2

Agenda

ObjectivesCommon problems / solutionsWVR tracesuseful PD toolsJava / VXML specificsReferences / ResourcesQ & A

IBM Software Group


Session Objectives

Resolve / avoid some common problemsIdentify possible problem areasKnow how to use common PD toolsKnow how to run different WVR tracesLocate more problem solving informationGather useful data for PMR to work with IBM Support

IBM Software Group


Product componentsWVR is built up with the components:

• base runtime• Java / VXML application environment• State Table application environment• Telephony interfaces

IBM Software Group


Installation – FAQ / common issuesNo adapter / trunk available after installation

AIX 32-bit kernel for systems with voice adapters

What is the supported environmentAIX 5.2 with ML 2 or later or AIX 5.3 with ML2 or later

Java 1.4.1 is the pre-req – also support JDK 1.4.2 or 1.5

Communications Server (sna) 6.1 or 6.3

DTXA / DTTA / DTEA placement – details available at http://www-306.ibm.com/software/pervasive/products/voice/universalaccess_2_125.shtml

IBM Software Group


Migration – common issues / solutionssaveDT / restoreDT or DTdatabase –m

Follow the steps in Installation (GC34-6380)Install the latest WVR PTF (AIX 5.3 requires WVR 4.2.0.181 +)

reboot after all installation before DTdatabase -mCheck correct hostname in db2nodes.cfgCheck TechNote (1235594 ) for DB2 LIBPATH Do NOT backup & restore /home/dtdb23inWVR user (dtuser) and dtdb23in in same group (staff)hostname defined in /etc/hosts

• collect /tmp/DTdatabase.out.[pid]

IBM Software Group


Cloning – checklist to avoid problems(mksysb) can be used to build additional WVR systems.After loading mksysb image and change the hostname --

check /etc/hosts for correct hostname / IP addressrun WVR fsupdate script check $DTJ_DIR/default.cff on hostname and IP addresscheck /home/dtdb23in/sqllib/db2nodes.cfgsna and 3270 session configuration need fresh setupCheck TechNote (1228218) for LUM configuration reset set adapater ownership:

• dt_setowner –q[adapter ID] to query current setup• dt_setowner –s[adapter ID] for DTXA• dt_setowner –x –s[adapter ID] for DTTA

IBM Software Group


Start with the basics

identify the date / time of the events

Check WVR System Monitor

check the logs from $OAM_LOG_PATH• errorlog• DTstatus.out

check the logs from $DTJ_HOME/native/aix for DTJxxxx• [VR node].out• [app node].out• dtjflog –o [output] log.[n].log

AIX error report: errpt -a

Developing Java Application (GC34-6377)Problem Determination (GC34-6382)

IBM Software Group


Start with the basics (cont)If VRBE related, query system status with

• dtjqhost• dtjqnode• dtjqapps

check Javacore / Javadump in native/aix if JVM node disappeared

IBM Software Group


WVR base runtime trace$ trace –a –1 –L [trace file size] –T [buffer size]$ trcstop –1$ print_trace > [ASCII trace file]

setup and limitationsAIX trace for the whole WVR system pre-defined trace file and trace buffer events could be wrapped out on busy system in short time frame

Information providedMost of the WVR background proc events (DBHEALTH, DBIM, etc)Signal events on voice channels (ring, answer, transfer, call terminate)State table application actions and returned edges

Pub:Problem Determination (GC34-6382)

IBM Software Group


WVR base runtime trace (cont)Would be helpful for issues like

call events from CAS protocol (answer, transfer)application flow from state table applicationsCalls answered with wrong applicationsevents related to the specific error_id in WVR errorlog

Use with the DTalarm daemon update $SYS_DIR/dtalarmd.update with trace stop command like

alarm_id <ID> command sleep 5; trcstop –1refresh DTalarm definition $ DTalarm –ustart trace $ trace –a –1 –L 8000000 –T 1000000run print_trace after the error_id event

pub: Managing and Monitoring the System (GC34-6384)

IBM Software Group


WVR base runtime trace (cont)

Verify trace is valid for the eventCheck the top lines in the trace output --

Fri Sep 1 16:07:02 2006 date/time when trace is turned onSystem: AIX 5.2 Node: beech1 system OS and hostname informationMachine: 000C751F4C00Internet Address: 092A7926 9.42.121.38

Verify if the specific WVR error_id was logged$ grep ERROR_ENROLL <ASCII trace file>

Collect dtProblem output soon after the trace

IBM Software Group


WVR VRBE traceset up and limitations

setup from WVR VRBE trace daemon

binary trace files $DTJ_HOME/dtj_logs/wvrtrace.*

trace option reset after stop / start VRBE environment

Trace events can be wrapped out quickly on busy system

NO WVR base runtime events

NO signal interface events

Would be helpful for issues likeunexpected flow or event from VRBE VXML applications

DTJXXXX errors in <node>.out

problems to start or stop VRBE environment

Other issues from the Java / VXML environment

IBM Software Group


WVR VRBE trace (cont)How to turn on VRBE trace

Stop VRBE environment: dtjstop; dtjshost –exit; trcserv -kEdit $DTJ_HOME/dtj.ini to have trace.option=9Edit $DTJ_HOME/native/aix/trcserv.ini to have trace.mode=3Start VRBE environment again: dtjshost, dtjstart

How to collect trace data Collect dtbeProblem output immediately after the “event”

tar up all files under dtj_logs as backup

turn off the VRBE tracetrace events will be logged while trace.mode=3 is setremove the trace.option and trace.mode to avoid extra system load restart VRBE environment after above changes

IBM Software Group


Combined WVR runtime and VRBE trace set up and limitations

redirect VRBE trace into WVR runtime trace file

trace file can be wrapped out event sooner with combined events

may need to start runtime trace with larger trace file / buffer

stop the trace from command line or from DTalarm daemon

Would be helpful for issues likecall flow issues involve state table and Java / VXML

Issues on the interface of VRBE and base runtime

Call flow issues involve Java / VXML and signalling

IBM Software Group


Combined WVR runtime and VRBE trace (cont)How to turn on the combined trace

stop VRBE environment: dtjstop; dtjshost –exit; trcserv -kedit $DTJ_HOME/dtj.ini to have trace.option=9edit $DTJ_HOME/native/aix/trcserv.ini to have trace.mode=5start VRBE environment again: dtjshost, dtjstartstart runtime trace with larger trace file size and buffer size

$ trace –a -1 –L 32000000 –T 1000000

Turn off trace AS SOON AS POSSIBLE after the particular event$ trcstop -1$ print_trace | dtjtrcfmt > [output file]$ grep DTBeans [output file] VRBE trace events

IBM Software Group


Some issues in VRBE environmentJava OOM exception in [node].out file

Each node defined in default.cff is one JVMEach node can have its own JavaCommand= lineA node with large number of instances, may need larger heap sizeSetup in default.cff for the node with. For example --

-Xms32M –Xmx448Mwill have the initial heap size of 32MB with the maximun of 448MB

To debug memory usage, add the option in JavaCommand line-verbosegc

VRBE environment fails to startAll JVM communication is TCP/IP basedNon-responding DNS can cause dtjshost dtjstart failure

IBM Software Group


Some issues in VRBE environment (cont)All application instances are in use

Instance could stay in use even after caller hang up

Application should set timeout check for all backend query

Calls are answered, but caller hear silencePlay audio from local system before any backend query

IBM Software Group


Must-GatherdtProblem output

dtbeProblem output if system has VRBE environment

WVR traces

http://www-1.ibm.com/support/docview.wss?uid=swg21240832

IBM Software Group


Find out more help informationWVR product page:

http://www-306.ibm.com/software/pervasive/voice_response_aix/

TechNote search under Support page:http://www-306.ibm.com/software/pervasive/voice_response_aix/support/

ALL WVR publication are available from this link:http://www-128.ibm.com/developerworks/websphere/zones/voice/vresponse_aix.html

Latest WVR PTF information:http://www-306.ibm.com/software/pervasive/products/voice/universalaccess_2_124.shtml

IBM Software Group


Additional WebSphere Product Resources

Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.htmlJoin the Global WebSphere User Group Community: www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistantLearn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf.htmlSign up to receive weekly technical My support emails: www.ibm.com/software/support/einfo.html

IBM Software Group


Questions and Answers