WVR Troubleshooting
Transcript of WVR Troubleshooting
IBM Software Group
®
Presented by Yuchin [email protected]
WebSphere® Support Technical Exchange
Troubleshooting on WebSphere Voice Response 4.2 on AIX
IBM Software Group
WebSphere® Support Technical Exchange 2
Agenda
ObjectivesCommon problems / solutionsWVR tracesuseful PD toolsJava / VXML specificsReferences / ResourcesQ & A
IBM Software Group
WebSphere® Support Technical Exchange 3
Session Objectives
Resolve / avoid some common problemsIdentify possible problem areasKnow how to use common PD toolsKnow how to run different WVR tracesLocate more problem solving informationGather useful data for PMR to work with IBM Support
IBM Software Group
WebSphere® Support Technical Exchange 4
Product componentsWVR is built up with the components:
• base runtime• Java / VXML application environment• State Table application environment• Telephony interfaces
IBM Software Group
WebSphere® Support Technical Exchange 5
Installation – FAQ / common issuesNo adapter / trunk available after installation
AIX 32-bit kernel for systems with voice adapters
What is the supported environmentAIX 5.2 with ML 2 or later or AIX 5.3 with ML2 or later
Java 1.4.1 is the pre-req – also support JDK 1.4.2 or 1.5
Communications Server (sna) 6.1 or 6.3
DTXA / DTTA / DTEA placement – details available at http://www-306.ibm.com/software/pervasive/products/voice/universalaccess_2_125.shtml
IBM Software Group
WebSphere® Support Technical Exchange 6
Migration – common issues / solutionssaveDT / restoreDT or DTdatabase –m
Follow the steps in Installation (GC34-6380)Install the latest WVR PTF (AIX 5.3 requires WVR 4.2.0.181 +)
reboot after all installation before DTdatabase -mCheck correct hostname in db2nodes.cfgCheck TechNote (1235594 ) for DB2 LIBPATH Do NOT backup & restore /home/dtdb23inWVR user (dtuser) and dtdb23in in same group (staff)hostname defined in /etc/hosts
• collect /tmp/DTdatabase.out.[pid]
IBM Software Group
WebSphere® Support Technical Exchange 7
Cloning – checklist to avoid problems(mksysb) can be used to build additional WVR systems.After loading mksysb image and change the hostname --
check /etc/hosts for correct hostname / IP addressrun WVR fsupdate script check $DTJ_DIR/default.cff on hostname and IP addresscheck /home/dtdb23in/sqllib/db2nodes.cfgsna and 3270 session configuration need fresh setupCheck TechNote (1228218) for LUM configuration reset set adapater ownership:
• dt_setowner –q[adapter ID] to query current setup• dt_setowner –s[adapter ID] for DTXA• dt_setowner –x –s[adapter ID] for DTTA
IBM Software Group
WebSphere® Support Technical Exchange 8
Start with the basics
identify the date / time of the events
Check WVR System Monitor
check the logs from $OAM_LOG_PATH• errorlog• DTstatus.out
check the logs from $DTJ_HOME/native/aix for DTJxxxx• [VR node].out• [app node].out• dtjflog –o [output] log.[n].log
AIX error report: errpt -a
Developing Java Application (GC34-6377)Problem Determination (GC34-6382)
IBM Software Group
WebSphere® Support Technical Exchange 9
Start with the basics (cont)If VRBE related, query system status with
• dtjqhost• dtjqnode• dtjqapps
check Javacore / Javadump in native/aix if JVM node disappeared
IBM Software Group
WebSphere® Support Technical Exchange 10
WVR base runtime trace$ trace –a –1 –L [trace file size] –T [buffer size]$ trcstop –1$ print_trace > [ASCII trace file]
setup and limitationsAIX trace for the whole WVR system pre-defined trace file and trace buffer events could be wrapped out on busy system in short time frame
Information providedMost of the WVR background proc events (DBHEALTH, DBIM, etc)Signal events on voice channels (ring, answer, transfer, call terminate)State table application actions and returned edges
Pub:Problem Determination (GC34-6382)
IBM Software Group
WebSphere® Support Technical Exchange 11
WVR base runtime trace (cont)Would be helpful for issues like
call events from CAS protocol (answer, transfer)application flow from state table applicationsCalls answered with wrong applicationsevents related to the specific error_id in WVR errorlog
Use with the DTalarm daemon update $SYS_DIR/dtalarmd.update with trace stop command like
alarm_id <ID> command sleep 5; trcstop –1refresh DTalarm definition $ DTalarm –ustart trace $ trace –a –1 –L 8000000 –T 1000000run print_trace after the error_id event
pub: Managing and Monitoring the System (GC34-6384)
IBM Software Group
WebSphere® Support Technical Exchange 12
WVR base runtime trace (cont)
Verify trace is valid for the eventCheck the top lines in the trace output --
Fri Sep 1 16:07:02 2006 date/time when trace is turned onSystem: AIX 5.2 Node: beech1 system OS and hostname informationMachine: 000C751F4C00Internet Address: 092A7926 9.42.121.38
Verify if the specific WVR error_id was logged$ grep ERROR_ENROLL <ASCII trace file>
Collect dtProblem output soon after the trace
IBM Software Group
WebSphere® Support Technical Exchange 13
WVR VRBE traceset up and limitations
setup from WVR VRBE trace daemon
binary trace files $DTJ_HOME/dtj_logs/wvrtrace.*
trace option reset after stop / start VRBE environment
Trace events can be wrapped out quickly on busy system
NO WVR base runtime events
NO signal interface events
Would be helpful for issues likeunexpected flow or event from VRBE VXML applications
DTJXXXX errors in <node>.out
problems to start or stop VRBE environment
Other issues from the Java / VXML environment
IBM Software Group
WebSphere® Support Technical Exchange 14
WVR VRBE trace (cont)How to turn on VRBE trace
Stop VRBE environment: dtjstop; dtjshost –exit; trcserv -kEdit $DTJ_HOME/dtj.ini to have trace.option=9Edit $DTJ_HOME/native/aix/trcserv.ini to have trace.mode=3Start VRBE environment again: dtjshost, dtjstart
How to collect trace data Collect dtbeProblem output immediately after the “event”
tar up all files under dtj_logs as backup
turn off the VRBE tracetrace events will be logged while trace.mode=3 is setremove the trace.option and trace.mode to avoid extra system load restart VRBE environment after above changes
IBM Software Group
WebSphere® Support Technical Exchange 15
Combined WVR runtime and VRBE trace set up and limitations
redirect VRBE trace into WVR runtime trace file
trace file can be wrapped out event sooner with combined events
may need to start runtime trace with larger trace file / buffer
stop the trace from command line or from DTalarm daemon
Would be helpful for issues likecall flow issues involve state table and Java / VXML
Issues on the interface of VRBE and base runtime
Call flow issues involve Java / VXML and signalling
IBM Software Group
WebSphere® Support Technical Exchange 16
Combined WVR runtime and VRBE trace (cont)How to turn on the combined trace
stop VRBE environment: dtjstop; dtjshost –exit; trcserv -kedit $DTJ_HOME/dtj.ini to have trace.option=9edit $DTJ_HOME/native/aix/trcserv.ini to have trace.mode=5start VRBE environment again: dtjshost, dtjstartstart runtime trace with larger trace file size and buffer size
$ trace –a -1 –L 32000000 –T 1000000
Turn off trace AS SOON AS POSSIBLE after the particular event$ trcstop -1$ print_trace | dtjtrcfmt > [output file]$ grep DTBeans [output file] VRBE trace events
IBM Software Group
WebSphere® Support Technical Exchange 17
Some issues in VRBE environmentJava OOM exception in [node].out file
Each node defined in default.cff is one JVMEach node can have its own JavaCommand= lineA node with large number of instances, may need larger heap sizeSetup in default.cff for the node with. For example --
-Xms32M –Xmx448Mwill have the initial heap size of 32MB with the maximun of 448MB
To debug memory usage, add the option in JavaCommand line-verbosegc
VRBE environment fails to startAll JVM communication is TCP/IP basedNon-responding DNS can cause dtjshost dtjstart failure
IBM Software Group
WebSphere® Support Technical Exchange 18
Some issues in VRBE environment (cont)All application instances are in use
Instance could stay in use even after caller hang up
Application should set timeout check for all backend query
Calls are answered, but caller hear silencePlay audio from local system before any backend query
IBM Software Group
WebSphere® Support Technical Exchange 19
Must-GatherdtProblem output
dtbeProblem output if system has VRBE environment
WVR traces
http://www-1.ibm.com/support/docview.wss?uid=swg21240832
IBM Software Group
WebSphere® Support Technical Exchange 20
Find out more help informationWVR product page:
http://www-306.ibm.com/software/pervasive/voice_response_aix/
TechNote search under Support page:http://www-306.ibm.com/software/pervasive/voice_response_aix/support/
ALL WVR publication are available from this link:http://www-128.ibm.com/developerworks/websphere/zones/voice/vresponse_aix.html
Latest WVR PTF information:http://www-306.ibm.com/software/pervasive/products/voice/universalaccess_2_124.shtml
IBM Software Group
WebSphere® Support Technical Exchange 21
Additional WebSphere Product Resources
Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.htmlJoin the Global WebSphere User Group Community: www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistantLearn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf.htmlSign up to receive weekly technical My support emails: www.ibm.com/software/support/einfo.html
IBM Software Group
WebSphere® Support Technical Exchange 22
Questions and Answers