Application Performance Tuning @ KBC Dirk Beauson.
-
Upload
iris-griffith -
Category
Documents
-
view
221 -
download
0
Transcript of Application Performance Tuning @ KBC Dirk Beauson.
Application Performance Tuning @ KBC
Click icon to add picture
Dirk Beauson
2
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC Application Performance Tuning Organization @ KBC New Challenges Current optimizations Conclusions
3
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
4
Why Application Performance Tuning @ KBC
RECOGNIZE ?
DB2 online - Poor Response Time- Timeouts/Deadlocks
Long Running DB2 jobs
Locking problems
5
Why Application Performance Tuning @ KBC
80% of the performance problems are related to inefficient application and/or database implementation !!
Building
Cost
Design Implementation
Problems can cost up to 50 times more to fix during implementation than in design !!!
6
Why Application Performance Tuning @ KBC
Performance Players Applications (Design/Build)
Databases
DB2 subsystem
z/OS
Network
Performance Factors Workload
Throughput
Resources
Contention
7
Why Application Performance Tuning @ KBC Using more CPU
results in using more MSU Can result in higher 4 hour rolling average
Common Performance Issues @ KBC Index design/usage SQL (please know your tables/indexes and explain) Application Design Lock waits Database design
So we want to run applications fast and as cheap as possible
8
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
9
EXPLAIN Tools Used
Simply Explain and PLAN_TABLE Need to look @ extra explain tables !!! (Study DB2 11)
Data Studio (visual explain) SQL Explorer Catalog Manager (to view Current Access Path)
COST*RATE QB PL MIX QTYPE METH ACC MTCH IX TBNAME IXNAME 11.083908 1 1 0 SELECT 0 N 2 N CBTXXX22 CBX7XXX 1.118052 1 2 0 SELECT 1 I 1 N CBTXXX03 CBX1XXX 1.118566 1 3 0 SELECT 1 I 2 N CBTXXX02 CBX3XXX 1.293676 1 4 0 SELECT 3 0 N
Access table CBTXXX22
using index CBX4XXX22 (1 COL)
Join using nested loop join with
Access table CBTXXX03
using index CBX1XXX03 (1 COL)
Join using nested loop join with
Access table CBTXXX02
using index CBX1XXX02 (1 COL)
Composite table is sorted for ORDER BY clause,
10
EVAL reporting tool
Everyday we report bad BMP’s & MPP’s @ DEV List under construction :
ABEND :ABENDs per BMP / DAY > x (dependent on USER/U0240/pseudo/system abends)
ABENDs per MPP / DAY > y (dependent on USER/U0240/pseudo/system abends)
LOCKING :DEADLOCKS & TIMEOUTS per BMP / DAY > x
DEADLOCKS & TIMEOUTS per MPP / DAY > x
LOCK ESCALATIONS
COMMIT FREQ :COMMIT FREQ in BMP > x sec and updates
ELP time MPP > x SEC and updates
Online Transaction running under IMS
Batch running under IMS
11
EVAL reporting tool
Number of SQL CALL : Number of SQL CALLS in BMP > x
Number of SQL CALLS in MPP > x
Number of GETPAGES per PLAN :Number of GETPAGES per BMP > x
Number of GETPAGES per MPP > x
Number of GETPAGES per PACKAGE/function :Number of GETPAGES per PACKAGE in BMP > x
Number of GETPAGES per PACKAGE in MPP > x
Number of GETPAGES per STMT :Number of GETPAGES per PLAN per STMT in BMP > x
Number of GETPAGES per PLAN per STMT in MPP > x
12
EVAL reporting tool
Long running SQL STMT in MPP :SQL STMT > x SEC DB2 CPU
13
Performance Monitor Tools
Mainview Locking problems
Date / Time Lockout Victim Victim Victim Victim Victim ----------------- Type Plan AuthID Connect CType Corr ID 22MAY-11:36:52.54 DEADLOCK PLAN1 USER001 IKC1 MPP 0037PLAN122MAY-11:29:13.49 TIMEOUT PLAN2 USER002 ITC2 MPP 0012PLAN2
Blocker Blocker Blocker Holder Plan Connect Corr ID AuthID PLAN1 IKC2 0111PLAN1 USER011DISTSERV SERVER db2jcc_appli USER012
14
Performance Analyse Tools
ITSV &EXTA-DB2-tables (SMF info in files & DB2-tables) day-1 : Performance information about every execution (MPP, BMP)
PLANNAME CORRID BEGINTIME DB2AUTH ELPTIME--------++------------++--------------------------++--------++------------PLAN1 0012PLAN1 2015-05-21-09.22.55.309341 USER1 0.077PLAN1 0008PLAN1 2015-05-21-09.23.01.482621 USER1 0.227
CPUTIME CPUTIMEDB2 EWAITIO EWAITLAL-------++------------++------------++------------ 0.011 0.002 0.060 0.000 0.023 0.005 0.167 0.000
BPGETPAGE BPSYNCRD BPPREFET BPDPF---------++-----------++-----------++----------- 37 17 0 0 93 44 0 0
SELECT OPEN FETCH UPDATE INSERT DELETE ------++-----------++-----------++-----------++-----------++----------- 8 1 3 0 0 0 18 2 2 0 1 0
…
15
Performance Analyse Tools
RapitWeb Interface on ITSV data & extra info
- KBC made Reporting on performance information- Graphical presentation- Also known as
16
Performance Analyse Tools
ITSV (files and derived DB2 tables) : RapitWeb
- A lot of system information for the system guys- Also lots of information to help DEV
17
Performance Analyse Tools
ITSV (files and derived DB2 tables) : RapitWeb
- IMS info
- DB2 info
TRX1
18
Performance Analyse Tools
ITSV (files and derived DB2 tables) : RapitWeb TOPPERS (overall, domain, subsystem, …)
- Toppers Online
- Toppers Batch
MPP1
MPP2
MPP3
BMP1
BMP2
BMP3
19
Performance Analyse Tools
Apptune Performance problems
- PLAN- PACKAGE- STMT- OBJECT
20
Performance Analyse Tools
DWH from Apptune PLAN PACKAGE STMT OBJECT STMT/OBJECT Daily and monthly base
- Daily, 2 hour interval (16 days)- Daily (15 months)- Monthly (15 months)
21
Compare Access paths & UOC
Changing access paths Test -> Acceptance :
- A list is published of changing access pathsLIMITATION : only if queryno doesn’t change
Acceptance -> Production :- Same list
No follow up because list is not complete- Search for better (100 %) solution.
TS SCAN instead of IX access
Wrong registration
22
DB2 Object Follow Up Tools
Daily reporting : Object not RW state 70% or more full
- Only TS/PT Not recoverable TS …
Weekly process : Runstats of selected objects
- Own selection <> DSNACCOX IC REORG of selected objects
- Own selection <> DSNACCOX
23
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
24
Reactive Application Performance Tuning Triggers :
25
Reactive : INCIDENTS/PROBLEMS
CASE : REBIND query in function XXXXX is performing badly in acceptance QUERY
26
Reactive : INCIDENTS/PROBLEMS
XXXXX – C200 op ACC: BAD
Access table CBTXXX22
using index CBX4XXX22 (1 COL)
Join using nested loop join with
Access table CBTXXX03
using index CBX1XXX03 (1 COL)
Join using nested loop join with
Access table CBTXXX02
using index CBX1XXX02 (1 COL)
Composite table is sorted for ORDER BY clause,
XXXXX – C200 op PRO: GOOD
Access table CBTXXX22
scan index CBX7XXX22 (2 COLS)
Join using nested loop join with
Access table CBTXXX03
using index CBX1XXX03 (1 COL)
Join using nested loop join with
Access table CBTXXX02
using index CBX3XXX02 (2 COLS)
Composite table is sorted for ORDER BY clause,
27
Reactive : INCIDENTS/PROBLEMS
IX information QUALIFIER: TABLE=CBTXXX22 CmIndex Name Column Seq Num O Lth DatTyp ----v----1----v----2----v----3----v----4----v----5----v----6----v----7----v---- CBX1XXX22 OVK_NR 1 8 A 12 CHAR CBX2XXX22 LOG_BSD_REG_DS 1 1 A 10 TIMESTMP CBX2XXX22 VLG_NR 2 2 A 2 SMALLINT CBX3XXX22 OSP_REK_NR 1 10 A 34 CHAR CBX4XXX22 OLOG_BSD_REG_DS 1 3 A 10 TIMESTMP CBX5XXX22 OVK_NR 1 8 A 12 CHAR CBX5XXX22 LOG_BSD_REG_DS 2 1 A 10 TIMESTMP CBX5XXX22 OSP_REK_NR 3 10 A 34 CHAR CBX6XXX22 OSP_REK_NR 1 10 A 34 CHAR CBX6XXX22 LOG_BSD_REG_DS 2 1 A 10 TIMESTMP CBX6XXX22 OVK_NR 3 8 A 12 CHAR CBX7XXX22 OVK_NR 1 8 A 12 CHAR CBX7XXX22 OLOG_BSD_REG_DS 2 3 A 10 TIMESTMP
QUALIFIER: TABLE=CBTXXX22 C Index Name Table Name UT Cl Col Keys ----v----1----v----2----v----3----v----4----v----5----v----6----v----7----v--- CBX1XXX22 CBTXXX22 D2 NN 1 79K CBX2XXX22 CBTXXX22 P2 YY 2 6573K CBX3XXX22 CBTXXX22 D2 NN 1 95K CBX4XXX22 CBTXXX22 D2 NN 1 4104K CBX5XXX22 CBTXXX22 D2 NN 3 5544K CBX6XXX22 CBTXXX22 D2 NN 3 5544K CBX7XXX22 CBTXXX22 D2 NN 2 4310K
28
Reactive : INCIDENTS/PROBLEMS
EXTA : after BIND Function used in online transaction YYYYY YYYYY info 19/05 DB2 related ? CPU -> GP ? ELP -> IO, wait ?
29
Reactive : INCIDENTS/PROBLEMS
APPTUNE or DWH APPTUNE : after BIND PLAN YYYYY 12743 = open YYYYY info 19/05 08:00 hour interval PLAN ? PACKAGE ? STMT ?
30
Reactive : INCIDENTS/PROBLEMS
EXPLAIN Tune query
Changed Query
COST*RATE QB PL MIX QTYPE METH ACC MTCH IX TBNAME IXNAME 11.083908 1 1 0 SELECT 0 N 2 N CBTXXX22 CBX7XXX 1.118052 1 2 0 SELECT 1 I 1 N CBTXXX03 CBX1XXX 1.118566 1 3 0 SELECT 1 I 2 N CBTXXX02 CBX3XXX 1.293676 1 4 0 SELECT 3 0 N
OK
31
Reactive : INCIDENTS/PROBLEMS
EXTA after fix: Function used in online transaction YYYYY YYYYY info 20/05
32
Reactive : INCIDENTS/PROBLEMS
APPTUNE or DWH APPTUNE after fix PLAN YYYYY 12741 = open YYYYY info 20/05 08:00 hour interval
33
Reactive : INCIDENTS/PROBLEMS
CONCLUSION Change Query :
FROM CBVXXX02 B, CBVXXX03 C, CBVXXX22 D
WHERE C.LOG_BSD_REG_DS = D.LOG_BSD_REG_DS
AND B.OLOG_BSD_REG_DS = C.OLOG_BSD_REG_DS
AND D.OLOG_BSD_REG_DS BETWEEN :WS-WV.WWV-END-DT-LZN AND :WS-WV.
WWV-END-DT-ITV
AND D.OLOG_BSD_REG_DS <= :WS-WV.WWV-HPOS-KEY-REG-DS
AND D.OVK_NR IN(:WTA-OVK-NR-STR-ZN.WTA-VMT-OVK-NR-1, :
In
AND D.OLOG_BSD_REG_DS >= :WS-WV.WWV-END-DT-LZN AND D.OLOG_BSD_REG_DS <=:WS-WV.WWV-END-DT-ITV
I also changed cardinality of PRD_TYPE_KD from 15 to 1005, to force DB2 to use the correct index on CBTXXX02.
34
Reactive : INCIDENTS/PROBLEMS
Next slides hidden : Describes steps to investigate :
- Batch incidents- Online incidents
42
Reactive : INCIDENTS/PROBLEMS
Conclusion Access path problems Much more difficult to force access path in DB2 10 and DB2 11
- We (THINK) know what DB2 has to do- So we try to let DB2 choose that access path- Let DB2 know all you know !!! (extra predicates can help)
Statistics more and more important to DB2- Cardinality columns in indexes -> very important (1e column)- Index Levels
Query Rewrite process !!! (DB2 11 study) Even order of tables in stmt can matter >= and <= better than BETWEEN …
43
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
44
Proactive Application Performance Tuning Sometimes we (applicative DBAs) are really asked to participate
in a new projects Not that often
Reason : Very important project where performance is an issue by design Large Workload Large Throughput needed Locking can kill system Large DB system Very fast respons expected
45
Proactive Application Performance Tuning We give advise to the DEV team how to design the application,
taking into account the previous issues We design the database model together with the DEV team
Lots of rows (> 1.000.000.000) High volume insert Active/Passive data (archive) Complex overview queries Index design (larger index pages, include columns, …)
We co-write SQL, or at least screen the access paths We monitor the application’s behaviour from test over
acceptance to production And are co-responsible for the performance of the application
46
Proactive Application Performance Tuning We set up performance test together Follow up these tests
Test Acceptance Production
47
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
48
Ad Hoc Initiatives Applicative DBA-group
Performance days : On regular base lookup bad processes via EVAL Reporting
- Analyse - Solution- Fix
If possible without touching application -> budget !!!
But sometimes redesign is really needed -> negotiate with DEV
DB2 10 optimizations : Look at index Usage/Design
- Lots of random access via indexes with number of levels > 3Check benefit of larger index pages
Fix
49
Ad Hoc Initiatives Applicative DBA-group
Performance days Look at index Usage/Design
- Tables with lots of indexesIndexes used ?
Possible to delete indexes ?
Use include columns in existing indexes
100 Cases done :- Conclusion
Saved a lot of CPU/ELP TIME
Difficult to predict what the effective benefit will be
Predicted value always > the real value
In some cases good estimated values (look per STMT instead of PACKAGE or PLAN)
50
Ad Hoc Initiatives Applicative DBA-group
Performance days What’s in for ourselves (Applicative DBA’s) ?
- We share knowledge- Train each other- Try to get everyone on a higher DB2 knowledge level- …
51
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
52
Application Performance Tuning Organization @ KBC
Only DBA’s (both system & applicative) + system guys
1
1
53
Application Performance Tuning Organization @ KBC
KIT Quality Program Performance becomes very important LOOKUP & SOLVE current performance problems Training DEV-people by applicative DBA’s
- Performance Course :Why Performance
Working of DB2
Explain/STAGE 1/STAGE 2/Access Paths/Joins
Performance Tools
1
2
2
54
Application Performance Tuning Organization @ KBC
KIT Quality Program Birth of :
- EVAL Reporting & Incident Proces- RapitWeb- PFA (Performance Aanspreekpunt) (Performance Contact)
Performance specialists @DEV
1
2
2
55
Application Performance Tuning Organization @ KBC
Role PFA : Has to become a performance specialist @ DEV
- Education by applicative DBA’s- 1 per department
Knowlegde -> DEV Proactive Perf role in every new project (obliged !!!) Follow up EVAL reporting They do lots of very good work
ONION model
DBA (S&A)SYST
PFA
DEV
56
Application Performance Tuning Organization @ KBC
PFA become PFM Also reporting role about Performance per department Less problems to solve No fulltime PFM need anymore So have to do other DEV tasks Performance less important
1 32
3 anagerM
57
Application Performance Tuning Organization @ KBC
Less PFM New PFM PFM not a real role anymore
1 32 4
4
58
Application Performance Tuning Organization @ KBC
Performance back to DBA Reactive Triggered (no continu follow up) EVAL Reporting (no incidents process)
More performance issues
1 324
5
5
59
Application Performance Tuning Organization @ KBC
Performance becomes more and more important Agile Mobile More need for PFA
1 324 6
5
6
60
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
61
New Performance Challenges
AGILE New way of Development
- Not always started with Global Technical Design- Build on build on build on … repetitive
Not always first Database Design in mind
- If needed first Database Design not corrected- Misuse DB2 objects (columns, number of rows,…)- Less communication with DBA’s
Only little things applied at a time
No big changes
- Performance not always in mind during repetitive DEV process
62
New Performance Challenges
Mobile/Online workload Much more sessions Much more executions Workload Throughput Locking
63
New Performance Challenges
More Data More tables More rows, lots of rows !!! Data has to be active much longer Sizing objects much more difficult Follow up usage % object Problems table-PT full
- Acute increase in the number of rows- Other hotspots
64
New Performance Challenges
More Data Problems index-PT full
- Usage larger columns in IX (CHAR 50 and > 50)- In combination with no compression- IX larger than TS/PT
Only follw up TS/PT space usage !!!
65
New Performance Challenges
BIG DATA DWH-environment Link IDAA Data Analists
- Play in DB2- Own tables- Keep all data
Forever ???
66
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
67
Current Optimizations
Redesign EVAL process : Production problems
- Visable in EVAL since acceptance- Not followed up
DEV wants better EVAL process :- INCIDENTs ?
NO : Applicatieve DBA group cannot afford 1 person fulltime being busy with that
- Better EVAL process, designed together with DEV : OK ;o)- Exception reporting :
Problem parked till date xx/xx/xxxx
- Only reporting per SUBSYSTEMNot all problems to everyone
68
Current Optimizations
Quality Assurance DB2 objects : Better follow up process
- Not only follow up TS/PT- Also IX
Usage of large columns 50 and > 50
Better weekly process :- Runstats
Better follow up exceptions
- Reorgs- Evaluate DSNACCOX rules versus our own rules based on RTS
Lots of Differences !!!
69
Agenda
Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC
Reactive Proactive Ad Hoc initiative Appl DBA’s
Application Performance Tuning Organization @ KBC History
New Challenges Current optimizations Conclusions
70
Conclusions
We are going back in the right direction. Performance becomes important again
Is Jan (System Engineer) satisfied with our current Performance Setup ? Jan ?
Is our current Performance Setup Bulletproof ? Jan ?
71
Questions ???