INSPIRING VALUE CHAIN ADVANTAGE

2013 OTM SIG CONFERENCE Performance Tuning/Monitoring

Alex Chang

alex.chang@inspirage.com

July 30, 2013

Agenda

Page 2 INSPIRING VALUE CHAIN ADVANTAGE

• General guidelines – Effective tuning goal – Tuning session – Tuning life cycle

• Common tools • Case studies & Common tuning problems • Q&A

Effective Tuning Goal • Specific • Measurable • Achievable • Cost effective

3 INSPIRING VALUE CHAIN ADVANTAGE

Tuning Session • Define the problem and state the goal • Collect current performance statistics • Consider common performance errors • Build a trial solution • Implement and measure the change • Decide: ” Did the solution meet the goal? ”

– No. Then go back to step 3 and repeat – Yes. Then create a new baseline

4 INSPIRING VALUE CHAIN ADVANTAGE

Tuning Life Cycle • An OTM life cycle can be divided into different phases:

– Functional and technical design and implementation – Testing – Deployment – Production: Troubleshooting and tuning

• Use a test instance to determine whether the solution eliminated the bottleneck – Migration, upgrade, and environment changes

• Any change of OS, database and OTM requires thorough testing to avoid performance issues.

5 INSPIRING VALUE CHAIN ADVANTAGE

Tuning Life Cycle • Functional and technical design and implementation

– Proactive (make it better, so it will not break) • Test scenarios • Measure results • Find the problem areas • Resolve the problem

– Whenever possible, you should start tuning at this level. With a good design, many tuning problems do not occur

– It is less expensive to correct bugs and performance problems early in the life cycle than later

6 INSPIRING VALUE CHAIN ADVANTAGE

Tuning Life Cycle • Testing:

– More realistic test data – The test case should exercise the application functions, expected loads,

and stress tests of improbable loads – Check physical layout – Monitor for resource contention – Test for resource exhaustion – Incorporate the business requirements for recovery time and availability as

much as possible

7 INSPIRING VALUE CHAIN ADVANTAGE

Tuning During the Life Cycle • Deployment:

– When OTM is initially deployed, the performance expectations are often different than reality

– Generate regular performance reports and save them as baselines – Monitor growth and performance

• Compare new performance reports to the previous reports • Tune before the performance degrades to an unacceptable level

8 INSPIRING VALUE CHAIN ADVANTAGE

Design and Implementation • Production

– You need to know: • What has changed? • Were is the baseline?

9 INSPIRING VALUE CHAIN ADVANTAGE

Design and Implementation • Migration, Upgrade, and Environment Changes

– Create a test system. – Capture a representative workload – Execute the workload against the test system. – Collect statistics. – Reset the test system, make a change. – Execute the workload. – Capture statistics and compare.

10 INSPIRING VALUE CHAIN ADVANTAGE

Common Tools • OTM diagnostics servlets • Database monitoring tools

– AWR – ADDM

• OS monitoring tools – SAR/TOP

• JVM tools – Console logs – Thread dumps – Heap dumps

• Network monitoring

11 INSPIRING VALUE CHAIN ADVANTAGE

OTM Diagnostic Servlets • Event diagnostics

– http://localhost/GC3/glog.webserver.event.EventDiagServlet

• Cache diagnostics – http://localhost/GC3/glog.webserver.cache.CacheDiagServlet

• Object lock diagnostics – http://localhost/GC3/glog.webserver.synch.object.ObjectLockDiagServlet

• Mediator diagnostics – http://localhost/GC3/glog.webserver.mediator.MediatorDiagServlet

• Threads – http://localhost/GC3/glog.webserver.thread.ThreadDiagServlet

12 INSPIRING VALUE CHAIN ADVANTAGE

Event Diagnostics

13 INSPIRING VALUE CHAIN ADVANTAGE

Event Diagnostics

14 INSPIRING VALUE CHAIN ADVANTAGE

Cache Diagnostics

15 INSPIRING VALUE CHAIN ADVANTAGE

Object Lock Diagnostics

16 INSPIRING VALUE CHAIN ADVANTAGE

Oracle Enterprise Manager

17 INSPIRING VALUE CHAIN ADVANTAGE

Oracle Enterprise Manager

18 INSPIRING VALUE CHAIN ADVANTAGE

Oracle Enterprise Manager

19 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 1: Design Phase • Business requirement: The client wants to manage its fleet resources

for its operational units in OTM as follows. – Driver: Available, Intransit, Unavailable – Power unit: Available, Intransit, Unavailable – Equipment: Intransit Empty, Intransit Loaded, Total intransit, Assigned, Out

of Service, Loaded Available, Empty Available…etc.

20 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 1 : Design Phase • Potential approach – Location Asset Bucket Inventory

– Standard built-in inventory control and use SKU tables – Buckets are setup by resource type (i.e. 48’ Trailer,48’ Flatbed…etc.) – 41 buckets proposed

21 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 1: Design Phase

22 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 1: Design Phase • Location asset bucket inventory

– Pre-setup buckets for all operational units would result in the number of bucket records equal to # locations x bucket counts ( i.e. 100,000 locations x 41 buckets = 4.1 Million records)

– Daily shipments : 500,000 shipments – 1,500,000 insert and update per day – Serious performance bottleneck

23 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 1 : Design Phase • Pros, Cons and Data Use Analysis

24 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Logging

– The most frequent cause of performance problems – Single thread on the log file – each thread must wait its turn to write to the

log file – Create user logs for debugging and disable these when not in use

25 INSPIRING VALUE CHAIN ADVANTAGE

Bulk Plan – No Logging • Bulk plan 500 orders without logging

– Start time : 23:38:58 – End time: 23:39:45 – Duration: 47 seconds

26 INSPIRING VALUE CHAIN ADVANTAGE

Bulk Plan – Logging • Bulk plan 500 orders with logging

– Start time : 00:11:04 – End time: 00:12:22 – Duration: 78 seconds

27 INSPIRING VALUE CHAIN ADVANTAGE

Bulk Plan – No Logging • Performance improved by 65%!!! • These are simple test scenarios with very small data volume in my

database. In reality, performance improve much more. • Minimize logging!! • NO Detail logs!! • Disable logs for all domains!!

28 INSPIRING VALUE CHAIN ADVANTAGE

Bulk Plan – No Logging • Use Logging Overview to check

29 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Business Number Generator (BNG) contention

– Business number generator assigns serialized business numbers to relevant business objects.

– Single thread – Eliminate unnecessary use of BNG

30 INSPIRING VALUE CHAIN ADVANTAGE

BNG Impact • Created 500 order releases without BNG

– Used Integration layer to control order release keys

31 INSPIRING VALUE CHAIN ADVANTAGE

BNG Impact • Created 500 order releases without BNG

– Start time: 01:17:40 – End time: 01:18:19 – Duration: 29 seconds

32 INSPIRING VALUE CHAIN ADVANTAGE

BNG Impact • Created 500 order releases with BNG

– BNG assigned unique keys to each order

33 INSPIRING VALUE CHAIN ADVANTAGE

BNG Impact • Created 500 order releases with BNG

– Start time: 01:15:44 – End time: 01:16:27 – Duration: 43 seconds – Spent 48% more time!!

34 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Inefficient or high-load SQL statements

– Long parse time – Excessive I/O (physical reads and writes) – Excessive buffer gets – Excessive CPU time – Excessive waits

35 INSPIRING VALUE CHAIN ADVANTAGE

Top SQL Reports

36 INSPIRING VALUE CHAIN ADVANTAGE

Explain Plan

37 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Scalability

– Minimize the size of the object_lock table – Clear as frequently as possible – No need to keep data for more than one day even shorter

38 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Scalability – Object Lock Cleanup

39 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Bulk Plan

– Query phase • Spend extra time to fine tune SQL queries • Keep data volume in mind

– Planning phase • Itinerary, rate offering and rate record need to be as specific as possible • Design them with zero or minimal overlap with each other • Try not use “global” accessorial codes

40 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Shipment Created Phase

– Bulk Plan 500 orders/2 shipment built – No agent listens to shipment created event – Duration: 58 seconds

41 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Bulk Plan 500 orders/2 shipment built

– An agent listens to shipment created event – Duration: 74 seconds – 27% more time

42 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Bulk Plan 500 orders/2 shipment built

– Created a new process – Duration: 65 seconds – 12% more time

43 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Bulk Plan

– Shipment created phase • Agents that are triggered on shipment creation can prolong the bulk plan

significantly!! • Create new process if possible • Potential data lock or contention issue if an order are in multiple shipments or

multi-leg shipments

44 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 2 - Migration • Bulk plan performance issue after OTM upgraded to 6.3

– Before upgrade • 10 minutes for a couple hundred orders

– After upgrade • 3 hrs

45 INSPIRING VALUE CHAIN ADVANTAGE

Case Study 2 - Migration • Cause: high volume of gets and low hit ratios

– glog.cache.REAccessorialCache – glog.cache.RERateGeoCache – glog.cache.TPackagedItemCache – glog.cache.TRateGeoSeqCache

• Cache size issue

46 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Agent

– Avoid one event triggers multiple agents and each agent checks saved conditions

• Data contention: For example, one agent set status and another agent sends integration.

• Use parent-child agent approach

47 INSPIRING VALUE CHAIN ADVANTAGE

Common Tuning Problems • Agent

– Avoid sequential agents • Agent A triggers agent b and agent b triggers agent c. • Consolidate into one agent

– Avoid overuse direct sql update, assign variable and saved query • Consider using PL/SQL to consolidate them

48 INSPIRING VALUE CHAIN ADVANTAGE

Q&A

49 INSPIRING VALUE CHAIN ADVANTAGE