INSPIRING VALUE CHAIN...

49
INSPIRING VALUE CHAIN ADVANTAGE 2013 OTM SIG CONFERENCE Performance Tuning/Monitoring Alex Chang [email protected] July 30, 2013

Transcript of INSPIRING VALUE CHAIN...

Page 1: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

INSPIRING VALUE CHAIN ADVANTAGE

2013 OTM SIG CONFERENCE Performance Tuning/Monitoring

Alex Chang

[email protected]

July 30, 2013

Page 2: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Agenda

Page 2 INSPIRING VALUE CHAIN ADVANTAGE

• General guidelines – Effective tuning goal – Tuning session – Tuning life cycle

• Common tools • Case studies & Common tuning problems • Q&A

Page 3: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Effective Tuning Goal • Specific • Measurable • Achievable • Cost effective

3 INSPIRING VALUE CHAIN ADVANTAGE

Page 4: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Tuning Session • Define the problem and state the goal • Collect current performance statistics • Consider common performance errors • Build a trial solution • Implement and measure the change • Decide: ” Did the solution meet the goal? ”

– No. Then go back to step 3 and repeat – Yes. Then create a new baseline

4 INSPIRING VALUE CHAIN ADVANTAGE

Page 5: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Tuning Life Cycle • An OTM life cycle can be divided into different phases:

– Functional and technical design and implementation – Testing – Deployment – Production: Troubleshooting and tuning

• Use a test instance to determine whether the solution eliminated the bottleneck – Migration, upgrade, and environment changes

• Any change of OS, database and OTM requires thorough testing to avoid performance issues.

5 INSPIRING VALUE CHAIN ADVANTAGE

Page 6: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Tuning Life Cycle • Functional and technical design and implementation

– Proactive (make it better, so it will not break) • Test scenarios • Measure results • Find the problem areas • Resolve the problem

– Whenever possible, you should start tuning at this level. With a good design, many tuning problems do not occur

– It is less expensive to correct bugs and performance problems early in the life cycle than later

6 INSPIRING VALUE CHAIN ADVANTAGE

Page 7: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Tuning Life Cycle • Testing:

– More realistic test data – The test case should exercise the application functions, expected loads,

and stress tests of improbable loads – Check physical layout – Monitor for resource contention – Test for resource exhaustion – Incorporate the business requirements for recovery time and availability as

much as possible

7 INSPIRING VALUE CHAIN ADVANTAGE

Page 8: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Tuning During the Life Cycle • Deployment:

– When OTM is initially deployed, the performance expectations are often different than reality

– Generate regular performance reports and save them as baselines – Monitor growth and performance

• Compare new performance reports to the previous reports • Tune before the performance degrades to an unacceptable level

8 INSPIRING VALUE CHAIN ADVANTAGE

Page 9: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Design and Implementation • Production

– You need to know: • What has changed? • Were is the baseline?

9 INSPIRING VALUE CHAIN ADVANTAGE

Page 10: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Design and Implementation • Migration, Upgrade, and Environment Changes

– Create a test system. – Capture a representative workload – Execute the workload against the test system. – Collect statistics. – Reset the test system, make a change. – Execute the workload. – Capture statistics and compare.

10 INSPIRING VALUE CHAIN ADVANTAGE

Page 11: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tools • OTM diagnostics servlets • Database monitoring tools

– AWR – ADDM

• OS monitoring tools – SAR/TOP

• JVM tools – Console logs – Thread dumps – Heap dumps

• Network monitoring

11 INSPIRING VALUE CHAIN ADVANTAGE

Page 12: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

OTM Diagnostic Servlets • Event diagnostics

– http://localhost/GC3/glog.webserver.event.EventDiagServlet

• Cache diagnostics – http://localhost/GC3/glog.webserver.cache.CacheDiagServlet

• Object lock diagnostics – http://localhost/GC3/glog.webserver.synch.object.ObjectLockDiagServlet

• Mediator diagnostics – http://localhost/GC3/glog.webserver.mediator.MediatorDiagServlet

• Threads – http://localhost/GC3/glog.webserver.thread.ThreadDiagServlet

12 INSPIRING VALUE CHAIN ADVANTAGE

Page 13: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Event Diagnostics

13 INSPIRING VALUE CHAIN ADVANTAGE

Page 14: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Event Diagnostics

14 INSPIRING VALUE CHAIN ADVANTAGE

Page 15: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Cache Diagnostics

15 INSPIRING VALUE CHAIN ADVANTAGE

Page 16: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Object Lock Diagnostics

16 INSPIRING VALUE CHAIN ADVANTAGE

Page 17: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Oracle Enterprise Manager

17 INSPIRING VALUE CHAIN ADVANTAGE

Page 18: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Oracle Enterprise Manager

18 INSPIRING VALUE CHAIN ADVANTAGE

Page 19: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Oracle Enterprise Manager

19 INSPIRING VALUE CHAIN ADVANTAGE

Page 20: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 1: Design Phase • Business requirement: The client wants to manage its fleet resources

for its operational units in OTM as follows. – Driver: Available, Intransit, Unavailable – Power unit: Available, Intransit, Unavailable – Equipment: Intransit Empty, Intransit Loaded, Total intransit, Assigned, Out

of Service, Loaded Available, Empty Available…etc.

20 INSPIRING VALUE CHAIN ADVANTAGE

Page 21: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 1 : Design Phase • Potential approach – Location Asset Bucket Inventory

– Standard built-in inventory control and use SKU tables – Buckets are setup by resource type (i.e. 48’ Trailer,48’ Flatbed…etc.) – 41 buckets proposed

21 INSPIRING VALUE CHAIN ADVANTAGE

Page 22: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 1: Design Phase

22 INSPIRING VALUE CHAIN ADVANTAGE

Page 23: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 1: Design Phase • Location asset bucket inventory

– Pre-setup buckets for all operational units would result in the number of bucket records equal to # locations x bucket counts ( i.e. 100,000 locations x 41 buckets = 4.1 Million records)

– Daily shipments : 500,000 shipments – 1,500,000 insert and update per day – Serious performance bottleneck

23 INSPIRING VALUE CHAIN ADVANTAGE

Page 24: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 1 : Design Phase • Pros, Cons and Data Use Analysis

24 INSPIRING VALUE CHAIN ADVANTAGE

Page 25: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Logging

– The most frequent cause of performance problems – Single thread on the log file – each thread must wait its turn to write to the

log file – Create user logs for debugging and disable these when not in use

25 INSPIRING VALUE CHAIN ADVANTAGE

Page 26: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Bulk Plan – No Logging • Bulk plan 500 orders without logging

– Start time : 23:38:58 – End time: 23:39:45 – Duration: 47 seconds

26 INSPIRING VALUE CHAIN ADVANTAGE

Page 27: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Bulk Plan – Logging • Bulk plan 500 orders with logging

– Start time : 00:11:04 – End time: 00:12:22 – Duration: 78 seconds

27 INSPIRING VALUE CHAIN ADVANTAGE

Page 28: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Bulk Plan – No Logging • Performance improved by 65%!!! • These are simple test scenarios with very small data volume in my

database. In reality, performance improve much more. • Minimize logging!! • NO Detail logs!! • Disable logs for all domains!!

28 INSPIRING VALUE CHAIN ADVANTAGE

Page 29: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Bulk Plan – No Logging • Use Logging Overview to check

29 INSPIRING VALUE CHAIN ADVANTAGE

Page 30: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Business Number Generator (BNG) contention

– Business number generator assigns serialized business numbers to relevant business objects.

– Single thread – Eliminate unnecessary use of BNG

30 INSPIRING VALUE CHAIN ADVANTAGE

Page 31: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

BNG Impact • Created 500 order releases without BNG

– Used Integration layer to control order release keys

31 INSPIRING VALUE CHAIN ADVANTAGE

Page 32: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

BNG Impact • Created 500 order releases without BNG

– Start time: 01:17:40 – End time: 01:18:19 – Duration: 29 seconds

32 INSPIRING VALUE CHAIN ADVANTAGE

Page 33: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

BNG Impact • Created 500 order releases with BNG

– BNG assigned unique keys to each order

33 INSPIRING VALUE CHAIN ADVANTAGE

Page 34: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

BNG Impact • Created 500 order releases with BNG

– Start time: 01:15:44 – End time: 01:16:27 – Duration: 43 seconds – Spent 48% more time!!

34 INSPIRING VALUE CHAIN ADVANTAGE

Page 35: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Inefficient or high-load SQL statements

– Long parse time – Excessive I/O (physical reads and writes) – Excessive buffer gets – Excessive CPU time – Excessive waits

35 INSPIRING VALUE CHAIN ADVANTAGE

Page 36: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Top SQL Reports

36 INSPIRING VALUE CHAIN ADVANTAGE

Page 37: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Explain Plan

37 INSPIRING VALUE CHAIN ADVANTAGE

Page 38: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Scalability

– Minimize the size of the object_lock table – Clear as frequently as possible – No need to keep data for more than one day even shorter

38 INSPIRING VALUE CHAIN ADVANTAGE

Page 39: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Scalability – Object Lock Cleanup

39 INSPIRING VALUE CHAIN ADVANTAGE

Page 40: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Bulk Plan

– Query phase • Spend extra time to fine tune SQL queries • Keep data volume in mind

– Planning phase • Itinerary, rate offering and rate record need to be as specific as possible • Design them with zero or minimal overlap with each other • Try not use “global” accessorial codes

40 INSPIRING VALUE CHAIN ADVANTAGE

Page 41: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Shipment Created Phase

– Bulk Plan 500 orders/2 shipment built – No agent listens to shipment created event – Duration: 58 seconds

41 INSPIRING VALUE CHAIN ADVANTAGE

Page 42: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Bulk Plan 500 orders/2 shipment built

– An agent listens to shipment created event – Duration: 74 seconds – 27% more time

42 INSPIRING VALUE CHAIN ADVANTAGE

Page 43: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Bulk Plan 500 orders/2 shipment built

– Created a new process – Duration: 65 seconds – 12% more time

43 INSPIRING VALUE CHAIN ADVANTAGE

Page 44: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Bulk Plan

– Shipment created phase • Agents that are triggered on shipment creation can prolong the bulk plan

significantly!! • Create new process if possible • Potential data lock or contention issue if an order are in multiple shipments or

multi-leg shipments

44 INSPIRING VALUE CHAIN ADVANTAGE

Page 45: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 2 - Migration • Bulk plan performance issue after OTM upgraded to 6.3

– Before upgrade • 10 minutes for a couple hundred orders

– After upgrade • 3 hrs

45 INSPIRING VALUE CHAIN ADVANTAGE

Page 46: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Case Study 2 - Migration • Cause: high volume of gets and low hit ratios

– glog.cache.REAccessorialCache – glog.cache.RERateGeoCache – glog.cache.TPackagedItemCache – glog.cache.TRateGeoSeqCache

• Cache size issue

46 INSPIRING VALUE CHAIN ADVANTAGE

Page 47: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Agent

– Avoid one event triggers multiple agents and each agent checks saved conditions

• Data contention: For example, one agent set status and another agent sends integration.

• Use parent-child agent approach

47 INSPIRING VALUE CHAIN ADVANTAGE

Page 48: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Common Tuning Problems • Agent

– Avoid sequential agents • Agent A triggers agent b and agent b triggers agent c. • Consolidate into one agent

– Avoid overuse direct sql update, assign variable and saved query • Consider using PL/SQL to consolidate them

48 INSPIRING VALUE CHAIN ADVANTAGE

Page 49: INSPIRING VALUE CHAIN ADVANTAGEotmsig.communities.oaug.org/multisites/otm/media/2013/US/C13U-25C... · INSPIRING VALUE CHAIN ADVANTAGE ... – ... Top SQL Reports INSPIRING VALUE

Q&A

49 INSPIRING VALUE CHAIN ADVANTAGE