Depository IBM Business Process Management … · 7. Created Cluster with BPM Configure job –...
-
Upload
trinhnguyet -
Category
Documents
-
view
217 -
download
0
Transcript of Depository IBM Business Process Management … · 7. Created Cluster with BPM Configure job –...
© 2012 IBM Corporation
DFK
Data Warehouse
GeneralLedger
AP
SalesCorrections
POReceiving
Return toVendor
WarehouseManagement
Credit App
EmployeeChange Notice
OTHER APPS - PC
ACCTS REC APPS - PCINVENT ORY CONTROL APPS - PC INVENT ORY CONTROL APPS - PC
Journal Entry Tool Kit
Scorecard
ResourceScheduling
P09 - P17Cyb.
Mill.
Millennuim 3.0
Banks - ACH and Pos toPay
Cobra
StockStatus
Polling
On-line NewHire Entry
CTS
Plan Administrators(401K, PCS, Life)
Post LoadBilling
HomeDeliveries
-Transfers
Planning
PurchaseOrder
SolutionSoftware
Inventory Info
Interface
Sales Posting
Price ManagementSystem
Cycle PhysicalInventory
SKUInformation
Customer RepairTracking I35 Early Warning
System
MerchandiseAnalysis
AutoReplenishment
CTO
Intercept
Counts
EmployeePurchase
Tex A
ACH
Stock Options
Customer PerceivedIn-Stock
Tx
SS
CapitalProjects
FixedAssets
ReconFile
Repair
EDICoordinator
Mesa D ataNEW Soundscan
Resumix
Op.
Store BudgetReporting
Tally Sheet
Cash Receipts/Credit
HouseCharges
Ad Expense
-PromoAnalysis
PriceMarketingSupport
BMP - Busperformance Mngt
StoreScorecard
PriceTesting
Media
Bonus/HR
Hand ScanApps
Shows
POS
SalesTax
A04 - CustRefund Chks
Equifax
Credit
CellularRollover
SatelliteSystem
Scanning
VAN
SKU Rep
Host to AS400Communication
Layaways
Bus Systems
V04-SignSystem
Count CorrectionsN.
P01-EmployeeMasterfile
CustomerOrder
ABCCo
Universal AccountReconcilliation
DepositoryBanks
CellPhones
- ISPTracking
AAS
PO
Cash Over/Short
Coop SKU SelectionTool
SKUPerformance
SupplierCompliance
1
DRK
ABBX
Misc Accounting/Finance Apps - PC/NT
AIMSMngr Approval
Batch ForcastingAd Measurement
AIMSReportingAd
Launcher
MktReactions
SpecSource
website
RebateTransfer
SignSystem
WriterWorkspace
PowerSuite
StoreMonitor
Calendar
Stores & Mrkts
Due Dates
Smart Plus
InsertionsOrders
BudgetAnalysis Tool
Print CostingInvoice App
Reports
BroadcastFilter
Smart PlusLauncher
GeneralMaintenance
Printer PO
PrinterMaintenance
VendorMaintenance
Vendor Setup
Connect 3
Connect 3Reports
Connect 3PDF Transfe
Spec SourceSKU Tracking
S20-SalesPolling
Prodigy
PSP
In-HomeRepair
WarrantyBillingSystem
Process Servers(Imaging)
DFK
Data Warehouse
GeneralLedger
AP
SalesCorrections
POReceiving
Return toVendor
WarehouseManagement
Credit App
EmployeeChange Notice
OTHER APPS - PC
ACCTS REC APPS - PCINVENT ORY CONTROL APPS - PC INVENT ORY CONTROL APPS - PC
Journal Entry Tool Kit
Scorecard
ResourceScheduling
P09 - P17Cyb.
Mill.
Millennuim 3.0
Banks - ACH and Pos toPay
Cobra
StockStatus
Polling
On-line NewHire Entry
CTS
Plan Administrators(401K, PCS, Life)
Post LoadBilling
HomeDeliveries
-Transfers
Planning
PurchaseOrder
SolutionSoftware
Inventory Info
Interface
Sales Posting
Price ManagementSystem
Cycle PhysicalInventory
SKUInformation
Customer RepairTracking I35 Early Warning
System
MerchandiseAnalysis
AutoReplenishment
CTO
Intercept
Counts
EmployeePurchase
Tex A
ACH
Stock Options
Customer PerceivedIn-Stock
Tx
SS
CapitalProjects
FixedAssets
ReconFile
Repair
EDICoordinator
Mesa D ataNEW Soundscan
Resumix
Op.
Store BudgetReporting
Tally Sheet
Cash Receipts/Credit
HouseCharges
Ad Expense
-PromoAnalysis
PriceMarketingSupport
BMP - Busperformance Mngt
StoreScorecard
PriceTesting
Media
Bonus/HR
Hand ScanApps
Shows
POS
SalesTax
A04 - CustRefund Chks
Equifax
Credit
CellularRollover
SatelliteSystem
Scanning
VAN
Ad Expense
-PromoAnalysis
PriceMarketingSupport
BMP - Busperformance Mngt
StoreScorecard
PriceTesting
Media
Bonus/HR
Hand ScanApps
Shows
POS
SalesTax
A04 - CustRefund Chks
Equifax
Credit
CellularRollover
SatelliteSystem
Scanning
VAN
SKU Rep
Host to AS400Communication
Layaways
Bus Systems
V04-SignSystem
Count CorrectionsN.
P01-EmployeeMasterfile
CustomerOrder
ABCCo
Universal AccountReconcilliation
DepositoryBanks
CellPhones
- ISPTracking
AAS
PO
Cash Over/Short
Coop SKU SelectionTool
SKUPerformance
SupplierCompliance
1
DRK
ABBX
Cash Over/Short
Coop SKU SelectionTool
SKUPerformance
SupplierCompliance
1
DRK
ABBX
Misc Accounting/Finance Apps - PC/NT
AIMSMngr Approval
Batch ForcastingAd Measurement
AIMSReportingAd
Launcher
MktReactions
SpecSource
website
RebateTransfer
SignSystem
WriterWorkspace
PowerSuite
StoreMonitor
Calendar
Stores & Mrkts
Due Dates
Smart Plus
InsertionsOrders
BudgetAnalysis Tool
Print CostingInvoice App
Reports
BroadcastFilter
Smart PlusLauncher
GeneralMaintenance
Printer PO
PrinterMaintenance
VendorMaintenance
Vendor Setup
Connect 3
Connect 3Reports
Connect 3PDF Transfe
Spec SourceSKU Tracking
S20-SalesPolling
Prodigy
PSP
In-HomeRepair
WarrantyBillingSystem
Process Servers(Imaging)IBM Business Process Management Advanced: Process Server V8.5 for z/OS
Problem Determination, Advanced Topics and Wrap-up
Problem Determination, Administration, Operations, Toplogies, Migration, Performance
(Updated Nov. 22, 2013)
© 2012 IBM Corporation
Topics
� Problem Avoidance & Trouble-Shooting➔ Prevention
➔ Diagnosis➔ Tools & Techniques
� Administration, Operations & Recovery
� Performance
� Migration
� Information Resources
� Summary & Feedback
3© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
PD is not a Science or Structured Methodology � Learned through (unsuccessful) experiences
� Skills & Teamwork (Sysprog, DBA, Sec.Admin...)
� Get good tools & learn how to use them.➨ UNIX: vi, viascii, awk, grep, a2e,
➨ java: jar, jad, debug
➨ TSO: ISPF, SDSF, MXI
➨ MVS: Operator Commands
➨ DB2: SPUFI, DB2Admin (ADBL)
➨ MQ: ISPF Panels
➨ WAS: dumpNameSpace, wsadmin, adminconsole
➨ Monitors, Sniffers, Drivers, ...
➨ Traces: WAS errorlog, IPCS, TraceJava, JDBC, JVM, JRAS, log4j, wsadmin
➨ Java tools
� Your Brain is your best tool . . .
� Resources & References: InfoCenter, admin Help, WebClient Help, IBMLink,
e-Support, TechDocs, RedBooks, Google, . . .
think
outside
the box
think
outside
the box
Problem Determination
4© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Problem Prevention (Avoidance)
� Adequate Hardware resources (CPU, Real Storage, DASD)
� Good Naming Standards & Planning
– Use the Spreadsheet!
� Software pre-reqs
– See the Program Directory.
� Tools (z/OS, Workstation, Web, ...)
� Skills (z/OS, UNIX, DB2, Security, TCP/IP, Java, ...)
� Adequate Testing Environments & Procedures
6© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Configuration – Problem Avoidance:(Here's what we set up before the Lab to avoid problems)
� Hardware:
– z/OS: 4Gb Real Memory, Six 3390-9 Public/Storage or SMS DASD volumes
– Linuz (for ProcessCenter):
� Software: z/OS, WAS, BPM, DB2, ...
– Installation Manager & Plug-ins
� Tailored Spreadsheet to produce: (from Techdoc WP102261)
– Response Files for WCT DMgr & EmptyNode + Augment Jobs
– BPM Configuration properties file
– DB2 Database Definitions
� Sample Scripts provided by ATS & BPM:PS Development to …
– Create Security profiles in RACF – Techdoc WP101427
– Create DB2 Tables
– Set up BPM Security
– Fix WAS Variables (port #s, Short names - based on techdoc TD105447)
Not much can go wrong if you avoid typing & spelling errors & follow directions.
7© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
What can possibly go wrong? (Common Problems)
Configuration
> Deployment
> Runtime
� Reading, Typing, Spellling
� Authorization problems
� Following directions? Poor directions?
� Real men don't need directions! ;-)
� Scripts not run in the correct order
� Space exhausted in config HFS, Joblogs, or /tmp
� Files or directories owned by another user
� Scripts interrupted
� ...
Logs and Traces
➔ Java, DB2, TCP/IP, ...
Check for messages ...
... in a Browser window
... in the Joblog, SYSLOG, & SYSPRINT
... in SYSLOG
... in log files (in the HFS)
If the problem scenario is reproducible ...
... capture relevant info (time, system, input, etc.)
... turn on more/other traces
... write down the steps how to reproduce
Search for these messages/symptoms ...
... in the WebSphere and Compute Grid InfoCenters
... on the WebSphere eSupport site ibm.com/software/webservers/appserv/zos_os390/support.html
... in the IBM knowledge database (IBMLink)
... in Google
If you still can't figure out your problem ...
� ... contact local experts or IBM Support for asst.
When first encountering a problem...
8© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
BPM:PS z/OS Configuration – Reviewing the steps (1st Lab)
1. Used the Spreadsheet to create response, BPM property and Sql files
2. Used WCT/PMT to create “Stacked” Jobstreams
3. Configured Security – scripts supplied
4. Configured the Deployment Manager – run the 6 jobs in order
5. Created DB2 Tables – scripts supplied
6. Created an Empty Node and Federation – run the 6 jobs in order
7. Created Cluster with BPM Configure job – automated by spreadsheet
8. Updated Variables & Bootstrap PS Database - scripts supplied
9. Installed sample applications & Tested them
Pretty Simple compared to Earlier Versions!
9© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Setting up Security
● The standard BBO%BRAK jobs do not include the following:
● Administrator's userID password not set
● STARTED profiles for servers not set if you are creating Empty Nodes
● The DORAC700 Exec copied from Techdoc TD101427 does both.
● Do NOT run any of the BBO%BRA% jobs except for the BBODBRAM job which creates the Home directories.
● Run it right after the DORAC700 Exec.
● Review the jobs you submitted from USER1.WAS.CNTL:
– RUNRC700 – runs the DORAC700 exec in USER1.WAS.DATA
– B#RACUID – Defined user Ids for BPM
– B#EJBROL – Defined EJBROLE profiles for BPM
10© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Misc. Configuration Problems – Ownership Errors
Ownerships can be corrupted: wasv8config/b6cell/b6dmnode/DeploymentManager/profiles/default ->ls -alt
drwxrwx--- 5 OMVSKERN B6CFG 416 Dec 23 14:50 configuration
Fix: (See FIXPERMS in USER1.WAS.CNTL)
cd /shared/zWebSphere/V8R0/bin; +
./bbowhfsb.sh /wasv8config/b6cell/b6dmnode/DeploymentManager; +
./bbowhfsc.sh /wasv8config/b6cell/b6dmnode/DeploymentManager b6admin b6cfg; +
./bbowhfsb.sh /wasv8config/b6cell/b6nodea/AppServer; +
./bbowhfsc.sh /wasv8config/b6cell/b6nodea/AppServer b6admin b6cfg
Or run these commands from the BBODHFSA job:
chmod -Rh 775 /wasv8config/b6cell/b6dmnode/DeploymentManager
chown -Rh B6ADMIN /wasv8config/b6cell/b6dmnode/DeploymentManager
chgrp -Rh B6CFG /wasv8config/b6cell/b6dmnode/DeploymentManager
11© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Problems Avoided when Creating the DB2 Tables
The following tools will save you time
and eliminate problems by automating the process:
– Spreadsheet generates the DDL
statements to define the Storage Group and Databases, and properties file used
by the BPMConfig.sh script
– BPMConfig.sh script run by the
BPZCNFGD job generates the necessary .sql files.
– createDatabase.sh script reads the .sql
files and creates the DB2 tables and other artifacts.
PDSTargetJobs
DB2Tables
WASConfig
xml
ResponseFiles
Spreadsheet
createDatabase.sh
zPMT
BPZCNFGD
Job
BPMConfig.sh
SQL tabledefinitions
BPMproperty File
1 2
34
1
12© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Common SQL Error Codes
� -104 Illegal Symbol [caused by extra blanks after '(' or ')' in col. 1 w/DB2 V9.]
� -204 Usually Table not defined (or wrong Schema prefix)
� -443 DB2 maintenance applied (Pacakages need to be re-bound)
� -551, -552, -553 Authorization problems (e.g., Missing GRANTs)
� -601 Duplicate name (e.g., already defined)
� -805 Packages need to be re-bound
� -904 Resource unavailable (e.g., DASD space)
� -913 Deadlock or Timeout
� -922 J2C Authentication errors (RACF APPL classprofiles missing)
� -4475 Missing DB2JccProperties file
13© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Federation Problems
1. If you don't change the UserID on the BBOWMNAM job to the admin's
● Change the userID to the administrator's
● Remove any files in /tmp created by the BBOWMNAN job
● Resubmit the job
2. If you Federate the Empty Node before augmenting it, …
� Error federating the empty node.
– ADMU0027E: An error occurred during federation The dmgr is not WPS enabled
– while the node is, we doesn't support such federation scenario!;
– rolling back to original configuration.
– Turns out there were errors in the Augment job
� Go to the /wasv8config/b6cell/b6nodea/AppServer/bin:
>./removeNode.sh -user b6admin -password b6admin -force
� Then go to the ISC and do a 'Force Delete' of the node.
� Then run the Augment job and then the Federate job....
14© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
DB2 Tools & Techniques
DASD space used by your table spaces
DB2 Table spaces are backed by VSAM datasets whose names have the following format:
Vcatname.DSNDBD.dbname.tsname.y0001.znnn
In the data set:
● Vcatname: The integrated catalog name (high level index), or VCAT
● DSNDBD: The data component of the VSAM cluster
● dbname: The database name
● tsname: The table space name or index name
Here is an example, sorted with the largest ones on the top.
Command ===> sort tracks Tracks
---------------------------------------------------------
WSLDB2.DSNDBD.B6SR01.WLP1194.I0001.A001 1185
WSLDB2.DSNDBD.B6SR01.BSPRTP1.I0001.A001 1059
WSLDB2.DSNDBD.B6SR01.BSPLTP3.I0001.A001 555
WSLDB2.DSNDBD.GLDDB.DESCTS.I0001.A001 300
WSLDB2.DSNDBD.GLDDB.DIRRSEAR.I0001.A001 300
15© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
DB2 Administrator's Tool - ISPF: Option 6 (TSO Command line) – 'ADBL' > 1 > 'D'
DB2 Admin ------------------- DSN9 Databases ----
Storage Buffer Created
Select Name Owner Group Pool DBID By
B6* * * * * *
------ -------- -------- -------- -------- ------ --------
B6CELLDB SYSADM1 B6DBSTO BP1 308 SYSADM1
T B6SR01DB SYSADM1 B6DBSTO BP1 312 SYSADM1
=========================================================
DB2 Admin --------------- DSN9 Tables, Views, and Alias
Sel Name Schema T DB Name TS Name
----- -----------------> -------- - -------- -------- -
BR LSW_TIME_PERIOD B8C1PS T B8SR01DB WLPT113
==========================================================
Admin -- DSN9 BROWSE B8C1PS.LSW_TIME_PE > -------------
****************************************************** Top of Data
ID NAME START_TIME END_TIME EFFECTIVE_DAYS
-- ----------------- ---------- -------- ---------------------------
0 24X7 0000 2359 mon,tue,wed,thu,fri,sat,sun
1 7AM-7PM M-F 0700 1900 mon,tue,wed,thu,fri
2 7AM-7PM Every Day 0700 1900 mon,tue,wed,thu,fri,sat,sun
3 9AM-5PM M-F 0900 1700 mon,tue,wed,thu,fri
4 9AM-1PM Sat 0900 1300 sat
***************************************************** Bottom of Data
18© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
jdbc Tracing
Specify in the jdbc properties file, normally named db2.jcc.propertiesFile:
db2.jcc.ssid=DSN9
db2.jcc.override.traceFile=/tmp/B6SR_tracejdbc.txt
See "Problem diagnosis with the IBM Data Server Driver for
JDBC and SQLJ”
http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp
19© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Runtime Tracing
Java Traces
Use the MVS modify command to set java tracing dynamically:
➨➨➨➨ Turn on Java Tracing for specified components
F <server_name>,TRACEJAVA='com.ibm.bpe.*=all=enabled'
➨➨➨➨ Reset to trace settings in configuration (was.env)
F <server_name>,TRACEINIT
Logs
- Messages in servants' & adjuncts' SYSPRINT & SYSOUT
- ffdc - "first failure data capture"
➨➨➨➨ Messages in servers' SYSPRINT shows the name & location:
FFDC closed incident stream file
/wasv6config/b6cell/nodeb/AppServer/profiles/default/logs/ffdc/
b6cell_b6nodeb_b6sr01b_STC32711_B6SR01BS_12.09.30_20.07.03_0.txt
<cell> <node> <server> <STC#> <STCname> <DateYMD> <TimeHMS>.txt
Strip_trailing_spaces.sh
iconv -f ISO8859-1 -t IBM-1047 $files | sed -e 's/'"$(printf '\015')"'$//' | sed -e's/[ \ ]*$//' | iconv -f IBM-1047 -t ISO8859-1 >.work_file_clean && mv .work_file_clean $files
Or use Mike's script:
/u/user1/wpswork/strip_trailing_spaces.sh . '*.sql'
25© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
wsadmin.sh tracing
Edit {profile_root}/properties/wsadmin.properties file:
Uncomment the line: #com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled
#-------------------------------------------------------------------------
# The traceFile property determines where trace and logging
# output are directed. If more than one user will be using
# wsadmin simultaneously, different traceFile properties should
# be set in user properties files.
# The default is that all tracing and logging go to the console;
# it is recommended that a value be specified here.
# If the file name contains DBCS characters, use unicode format such as \uxxxx, ..
#-------------------------------------------------------------------------
com.ibm.ws.scripting.traceFile=/wasv6config/b6cell/nodea/AppServer/profiles/default/logs/wsadmin.traceout
#-------------------------------------------------------------------------
# The traceString property governs the trace in effect for
# the scripting client process.
# The default is no tracing.
#-------------------------------------------------------------------------
com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled...
Then run the script and go look in the
{profile_root}/logs/wsadmin.traceout file.
29© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
ProcessCenter - Administrator
Launch from
http://<host>:<port>/ProcessCenter
UserID = tw_admin
Process Designer – Accessing the Process Center
Launch Windows Appl. From:
C:\IBM\ProcessDesigner\V8.0\eclipse.exe
31© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Connecting to the Process Center from the P.D.
● In the PD directory (C:\IBM\ProcessDesigner\v8.0) there is a file eclipse.ini
● Change the line
-Dcom.ibm.bpm.processcenter.url=http://192.168.17.232:9080
● to point to your ProcessCenter ip address... the port should be defaulted to 9080.
See traces in the same directory (orbtrc.11112011.1247.20.txt) or in C:\IBM\ProcessDesigner\v8.0\workspace\metadata\.log
● 12:47:20.448 com.ibm.ws.orbimpl.transport.WSTransport getHostIPAddress:137 Thread-10 ORBRas[default] java.net.UnknownHostException: wpspctr
● Add wpspctr to your /etc/hosts file so the PD can find your Process Center.
34© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Connecting to the Process Center
� Servers with the same TCPIP HostName & ServerName appear as the same Server.
– even if they have unique dotted IP addresses, and Port #s.
� That's why we defined each AppServer Node with the unique TCPIP
Host IP Address:
37© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Other tools & Techniques
� Tracing the Connection to the PC Repository:
F b1sr01b,tracejava='com.ibm.ws.repository.*=all=enabled'
� Clearing the osgi cache:/wasv8config/b1cell/b1nodeb/AppServer/bin-> ./osgiCfgInit.sh
OSGi cache successfully cleaned for /wasv8config/b1cell/b1nodeb/AppServer/profiles/default.
/wasv8config/b1cell/b1dmnode/DeploymentManager/bin-> ./osgiCfgInit.sh
OSGi cache successfully cleaned for /wasv8config/b1cell/b1dmnode/DeploymentManager/profiles/default.
38© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Security Errors – LTPA Tokens
Don't worry about these messages:
SECJ0371W: Validation of the LTPA token failed because the token
expired with the following info:
Token expiration Date: Thu Jan 26 13:03:00 EST 2012,
current Date: Thu Jan 26 15:00:33 EST 2012.
● The LTPA tokens expire 2 hours after they are created.
● They are created for users when they login to an app or the admin console.
● Most of the LTPA token expired messages are due to the servers, which use
them as credentials when they talk JMX.
● Time stamps on the server userid related LTPA token expired messages are
120 minutes apart.
● They are not serious and there's nothing you can do about them.
43© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Tracing in the Process Center
� In the ProcessCenter ICS, go to: Troubleshooting > Logs and Traces > pcsr01a > Change log detail levels
– Set it to something like: *=info: WLE.*=all
� To view the Traces & Logs: In the ISC: Troubleshooting > Logs and Traces
● Diagnostic Trace – Change trace settings
● JVM Logs – SystemOut.log,
SystemErr.log
● Process Logs – native_stdout.log,
native_stderr.log
● IBM Service Logs - activity.log
● Change Log Level Details
Server logs:
AppServer: /opt/IBM/WebSphere/BPM/profiles/PCEMPTYA/logs/pcsr01a/SystemOut.log
Dmgr: /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/logs/dmgr/SystemOut.log
51© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
IBM Support Assistant (ISA)• Contains many Diagnostic tools for debugging (mostly Java)
– ibm.com/software/support/isa/
– ibm.com/developerworks/websphere/techjournal/0906_supauth/0906_supauth.html
� Health Center
� Dump Analyzer, Heap Analyzer, Log Analyzer
� Garbage Collection and Memory Visualizer (GCMV)
� Memory Dump Diagnostic for Java (MDD4J)
� IBM Pattern Modeling and Analysis Tool for Java Garbage Collector (PMAT)
� IBM Thread and Monitor Dump Analyzer for Java (TMDA)
� IBM Trace and Request Analyzer for WebSphere Application Server (TRA)
� Log Analyzer
� Visual Configuration Explorer
See these Techdocs:
� WP101575 - “Introducing the ISA on z/OS”
� WP101612 - “Getting started - analysis GC, Heapdumps & Javacores”
52© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Which Tools for which Artifacts?
Artifact Tool
GCverbose xml (SYSOUT) GCMV
PMAT
Javacore TMDA
Heapdump Memory Analyzer
MDD4J
HeapAnalyzer
System Dump (jextracted) Memory Analyzer
Dump Analyzer
Logs & Traces TRA
Log Analyzer
Real Time Running Server Health Center
Configuration XML files VCE
53© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
PD DocumentationInfoCenter: Troubleshooting and support� Overview and New features
� Troubleshooting checklist for Business Process Manager
� Debugging applications
� Failed to log in to Process Center
� Troubleshooting administration tasks and tools
� Tools for troubleshooting your applications
� Recovering from a failure
� Disaster recovery
� Getting fixes
� Contacting IBM Software Support
Redbooks & Redpapers:� Problem Determination for WebSphere for z/OS SG24-6880� Problem Determination Methodology for WebSphere REDP6001� Problem Symptoms in WebSphere for z/OS REDP6002 � Problem Avoidance for WebSphere App. Server REDP6003� WAS for z/OS Problem Determination Means & Tools REDP6880
55© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
ND Topologies(see IEA foils)
Clustering basics: ● High availability
– The ability to always have the applications available for client applications
� Scalability
– The ability to grow the capacity based on business needs
• Workload management
• Improved resource utilization
• Easier administration
Horizontal
Ve
rtic
al
58© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Administration and Operations
� Installing Applications
� Starting & Stopping Servers
� Backing up HFSes & DB2 tables
� Recovering from outages
� The administrative console
� Command-line tools, scripts, and programming interfaces
� BPC Explorer and Business Process Archive Explorer
� Administering processes applications
� Deploying modules directly to IBM Business Process Manager
� Administering the IT infrastructure
� Troubleshooting administration
62© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Backing up and Restoring zFS with DFDSS
Backup -//STEP1 EXEC PGM=ADRDSSU,REGION=4M
//SYSPRINT DD SYSOUT=*
//OUTDD1 DD DSN=USER1.Z8CELL.BKUP.CONFIG,DISP=(NEW,CATLG),
// SPACE=(CYL,(180,50),RLSE),UNIT=(3390)
//SYSIN DD *
DUMP DATASET(INCLUDE(OMVS.WAS700.Z8CELL.Z8NODEA.CONFIG.ZFS)) -
OUTDDNAME(OUTDD1) CANCELERROR TOL(ENQF) OPTIMIZE(1) WAIT(2,2)
Restoring - (First unmount your zFSes)
//UNMNT EXEC PGM=IKJEFT01,DYNAMNBR=20,REGION=2M
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
UNMOUNT FILESYSTEM('OMVS.WAS700.Z8CELL.Z8NODEA.CONFIG.ZFS') IMMEDIATE
//RESTORE EXEC PGM=ADRDSSU
//SYSPRINT DD SYSOUT=*
//INODEA DD DISP=SHR,DSN=USER1.Z8CELL.BKUP.CONFIG
//SYSIN DD *
RESTORE DATASET(INCLUDE( OMVS.WAS700.Z8CELL.Z8NODEA.CONFIG.ZFS )) -
INDD(INODEA) REPLACE TGTALLOC(SOURCE) TOL(ENQF)
//MOUNT EXEC PGM=IKJEFT01,DYNAMNBR=20,REGION=2M
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('OMVS.WAS700.Z8CELL.Z8NODEA.CONFIG.ZFS') TYPE(HFS) -
MOUNTPOINT('/wasv6config/Z8cell/z8nodea')
Don't forget RACF changes, Procs, DB2 Tables for WPS, etc.
63© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Performance
� “Prescriptive Use Cases” - Co-Location Update Comparison
� z/OS Measurement & Tuning Tools:
– RMF, SDSF
– SMF Records
– WLM Tools
� Java Diagnostics
� Recommendations specific to BPM
67© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Tuning Checklist� Tracing and logging overhead - Minimize� DB2 Database tuning� Java tuning (Heap Size & Garbage Collection Policy)� Workload Profile� Messaging and Message Bindings � Workload Manager Service Class� MDB ActivationSpec� Thread & JMS Connection Pool Sizes� DataSource Connection Pool Size & Prepared Statement Cache Size� Messaging Engine Properties� Minimize security overhead� Disable automatic synchronization for ND� Run production servers in production mode� Large Objects� Maximum Concurrency� Messaging Tuning� Web Services Tuning� WebSphere MQ Tuning� Business Process Choreographer Tuning� Macro vs. Micro-flows
73© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Migration Termiology
• Migration vs. Upgrade
– Migration is 'side by side' and refers to movement of a group of objects
from one installation environment to another
• Required when delivering major new features, and profile updates
• Applies to 1st and 2nd digit release number changes, e.g. 6.1->6.2
• The new version must be installed side-by-side the old version of the
product
– Upgrade is 'in-place' and refers to replacement of a product with a newer
release of that same product
• Delivers updates or fixes to the existing components
• Applies to 3rd and 4th digit release number changes (interim fixes )
• The new release is installed on top of the existing installation.
74© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Migration Patterns - Three approaches:
● Runtime migration
➔ Install new version
➔ Migrate runtime configuration, applications and data using migration tools
● Manual migration
➔ Install new version
➔ Manually configure new deployment infrastructure and databases
➔ Manually re-install applications on new infrastructure
● Artifact migration
➔ Migrate applications to new version using authoring tools (WID, etc.)
➔ Install new version
➔ Manually configure new deployment infrastructure and databases
➔ Manually re-install applications on new infrastructure
● IBM Education Assistant:
➔ http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?
topic=/com.ibm.iea.ibpmgr/ibpmgr/7.5/Migration/BPMv75_WPSMigration/player.html
75© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Migration Method Considerations
Several different issues to consider when determining the
right migration method:● Production data
● Downtime
● Long-running processes and human tasks
● Application enhancements
● Target environment configuration
● Risk mitigation
● Selective or phased application migration
76© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Migration Tools
BPM migration command-line utilities
– BPMConfigureProcessServer
– BPMCreateRemoteMigrationUtilities
– BPMCreateTargetProfile
– BPMGenerateUpgradeSchemaScripts
– BPMMigrateProfile
– BPMMigrationStatus
– BPMQueryDeploymentConfiguration
– BPMSnapshotSourceProfile
– upgradeSchema (Common Database)
– BackupConfig
– MigrationDisablementReversal . . . . .
78© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Tools & Other Resources
� Tools
� Resources
� Other . . .
79© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Running shell scripts
� Running scripts in batch jobs with BPXBATCH or JZOS
� Running scripts without interruption in telnet
– nohup sh -c './xxx.sh ' >/tmp.xxx.out' 2>&1
� Halting scripts started with 'nohup':
– kill -9 <pid>
� Tracing shell scripts
– Insert 'set -x” at the beginning, or use this:
– sh -x './xxxsh' >/tmp.xxx.out' 2>&1
� wsadmin Tracing
– Uncomment this line in the
{profile_root}/properties/wsadmin.properties file
#com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled
80© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
Using Wide 3270 ScreensHelpful in looking at JES Logs, SDSF, RMF Reports, and anything wide:
� Fix PCOM profiles for good keyboard and char-set mapping to display brackets & braces:
� Wide WG31.WS - Large screen 50 x 132
– TCPIP profile for large screen – in DSN=SYS1.TCPPARMS(PROFILEx)
– TELNETDEVICE DYNAMIC ,D4C32XX3
– (See Techdoc TD102151 “Creating dynamic 3270 screen size definitions for increased productivity”)
81© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
BPM V8.5 Library and other Resourceshttp://www.ibm.com/software/integration/business-process-manager/library/documentation/
● InfoCenter: http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r5m0/index.jsp
● Quick Start Guides
● Product Documentation (PDFs)
● Tutorials (Hiring Sample)
● Announcements
84© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
ISC, PC, PD, IID Help
� Click on the “Help” or “?” button at the top of the console:
This brings up Specific Help Panels for the tool you are working with.
85© 2012 IBM CorporationIBM Americas Advanced Technical SkillsWashington Systems Center, Gaithersburg, MD
BPM:PS V8 z/OS Workshop Review
You've had an opportunity to work with the following areas in BPM: Process Server on z/OS:
1. Intro to IBM BPM
2. Review WAS ND on z/OS environment
3. Implementing BPM:PS for z/OS in a WAS ND env. + Lab
4. Configuring Process Center
5. Using Process Center, Process Designer & IBM Integration Designer
6. Overview of New functions in BPM Version 8
7. Problem Determination, Administration, Performance
Which ones were worth-while for you?
Which ones were not?What else would you like added?