Response Time Management for Oracle Databases · Author: Gaja Krishna Vaidyanatha Subject: Oracle...

Copyright © 2004 - 2013 DBPerfMan LLC, All rights reserved 1

Response Time Management for Oracle Databases

Gaja Krishna Vaidyanatha CEO & Founder, DBPerfMan LLC [email protected] http://www.dbperfman.com LinkedIn - gvaidyanatha Twitter – dbperfman Facebook - dbperfman

mailto:[email protected]

http://www.dbperfman.com

Copyright © 2004 - 2013 DBPerfMan LLC, All rights reserved

Due to certain unavoidable circumstances there will be no white paper associated with

this presentation. The inconvenience is regretted!


Welcome!


Who am I and what have I done in life?

1992 - First database - Oracle 6.0.31… 2000 - Compulsive Tuning Disorder (CTD) 2000 - Product Management at Quest Software 2002 – Product Management of OEM 10g at Oracle Corp. 2004 - DBPerfMan LLC


It’s all about the numbers ;)

Source – Dilbert by Scott Adams – http://www.dilbert.com


Execution Plan for the next hour!

Response Time Management The Methodology 10046 – The Holy Grail of Oracle Performance Diagnostics 10053 – The Optimizer Trace When OS Diagnostics is the ONLY way! Conclusion


Response Time Management


What is Compulsive Tuning Disorder (CTD)?

State of mind where tuning never ceases…. Branch of Obsessive Compulsive Disorder in the DBA world Carved its niche in every relevant performance optimization discussion Desperate attempt to reveal the psychological malady Do you suffer from CTD?


What is Response Time Management?

Systematic and deliberate effort to achieve a system runtime goal The “goal” needs to be clearly defined

Query X runs in 40 minutes and needs to run in less than 4 minutes Requires a simple diagnostic method that is repeatable

No world expert methods please… Eliminate bottlenecks SQL Optimization - Two key things to remember

1. Reducing logical I/O (LIO) 2. Reducing elapsed time


The Response Time Mantra!

Response Time = Service Time + Wait Time Service Time = fn (Hardware specs, Laws of Physics) Wait Time = fn (Wasteful Processing, Resource Hogging, Resource Contention)


The Oracle Wait Interface (OWI) Instrumentation within the Oracle Kernel that provides execution/wait times for various code paths

Timestamp – t1 Code X Timestamp – t2 Time spent = t2 – t1

Externalized via a set of V$ views (Oracle Wait Interface – OWI) V$System_Event, V$Session_Event, V$Session_Wait Trace files generated via event# 10046


The Methodology


Methodology – Take I 1. Set a reasonable tuning goal 2. DB Health report - AWR report 3. Trace the most offending SQL using 10046

(DBMS_MONITOR) – Prong I 4. Run tkprof on the generated trace file 5. Identify the most expensive indented step in the execution

plan 6. Make one change at a time to the SQL 7. If problem is not revealed by 10046, collect the relevant OS

diagnostic data (CPU, Memory, Process Trace) – Prong II 8. Rinse and repeat until tuning goal is achieved


Methodology – Take II 1. Set a reasonable tuning goal 2. Identify processes with the most CPU consumption – top 3. Map the PID to the session (SPID SID and SERIAL#) 4. Trace the most “offending session” using 10046

(DBMS_MONITOR) – Prong I 5. Run tkprof on the generated trace file 6. Identify the most expensive indented step in the execution

plan 7. Make one change at a time to the SQL 8. If problem is not revealed by 10046, collect the relevant OS

diagnostic data (CPU, Memory, Process Trace) – Prong II 9. Rinse and repeat until tuning goal is achieved


Methodology – Why is the OS Prong required?

CPU or memory bottlenecks are not revealed within the OWI Some OS Events and Oracle Waits are related

Paging and Latch Free System-level I/O Bottlenecks Helps identify bottlenecks external to Oracle


Methodology – Prong II : CPU Bottlenecks

Cross-reference high CPU numbers in the AWR reports with the nature of CPU Utilization

sar –u 5 1000, top %usr, %sys, %wio, %idle %wio and %sys typically should be < 10-15% 0% idle is OK, so long as %wio and %sys is not consistently above 15% Comparable data available in Windows Performance Monitor


Methodology – Prong II : CPU Bottlenecks

Cross-reference high CPU numbers in the AWR reports with the CPU Run Queue

Run queue is the only deterministic CPU bottleneck metric

Average run queue <= ((2-3) * (# of CPUs/Cores) vmstat - column “r” – Reveals system run queue

Excessive logical I/O is almost always the cause for high CPU utilization


Methodology – Prong II : Memory Bottlenecks

Cross-reference high “free memory” numbers in the AWR reports with the nature of memory utilization

vmstat –S or vmstat –a sr (scanrate) should be in and around 0 swapins and swapouts should be 0 Level of paging (pageins/sec. and pageouts/sec.)

Paging and swapping culprits Oversized SGAs (in an attempt to attain high cache-hit ratios) Not using direct I/O

Even if “vmstat” lies to you, high I/O activity on swapfile devices will reveal virtual-memory management issues


Methodology – Prong II : I/O Bottlenecks

Cross-reference high “file I/O” times in the AWR report with the I/O statistics

sar –d, iostat –xnz, Response times ideally should be < 5ms Response Time = Service Time (AvServ) + Wait Time (AvWait) Queuing is an indication of high response time and high wait times High disk queue numbers + high response times I/O contention

I/O requests should be evenly balanced across all devices Fewer the number of devices, better the overall utilization and I/O

spread


Methodology – What if the Optimizer is not making sense?

Use the 10053 trace facility to determine the workings of the Oracle Optimizer for the specific SQL:

1. Trace the offending SQL using 10053 at Level 1 2. View the trace file generated for the various execution plan

permutations that the Optimizer generated 3. Understand why the Optimizer chose the plan it did - Is it an issue with object-level statistics? - Is it a set of defaults? - Is it a bug?


Methodology - Summary Set tuning goals Benchmark Identify bottlenecks Make one change at a time Rinse and repeat until tuning goal is achieved

Oracle Diagnostics – Prong I Extended SQL Trace (Event 10046) Optimizer Trace (Event 10053) AWR Reports

Two-pronged Approach OS Diagnostics – Prong II CPU Memory Process Trace (When Prong I does not reveal the problem)


10046 - The Holy Grail of Oracle Performance

Diagnostics


Enabling 10046 – The Recipe 1. Identify the session’s process ID (SPID)

- select S.Username, P.Spid, S.Sid, S.Serial# from V$SESSION S, V$PROCESS P where S.PADDR = P.ADDR and P.SPID = NNNN; -- Get from “top”

2. Turn on trace

- exec dbms_monitor.session_trace_enable(session_id, serial#,TRUE,TRUE) 3. Turn off trace after the problem has been captured

- exec dbms_monitor.session_trace_disable(session_id, serial#) 3. Locate trace file in USER_DUMP_DEST using SPID

4. Run TKPROF on trace file

- tkprof input_trace_file_name.trc output1.txt sys=no sort=prsela,exeela,fchela


Reading the output of TKPROF SELECT t0.ITEM_ID, t1.M_ID FROM t0, t1 WHERE (((t0.ITEM_ID IN (:1, :2, :3, :4, :5, :6, :7, :8,

:9, :10, :11,:12, :13, :14, :15, :16)) AND (t1.P_ID = :17)) AND (t1.PP_ID = t0.P_ID) call count cpu elapsed disk query current rows ------- ------- -------- ---------- ---------- ---------- ---------- ---------- Parse 1 0.03 0.02 0 0 0 0 Execute 1 0.21 0.20 0 0 0 0 Fetch 1 13.29 15.85 353 6449761 0 1 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 3 13.53 16.08 353 6449761 0 1 Misses in library cache during parse: 1 Optimizer goal: CHOOSE Parsing user id: 23 (XXXXX) (recursive depth: 1)

Elapsed = CPU + (Wait for I/O and/or time spent on context switches) Disk = Physical I/O Query + Current = Logical I/O Analyzing Waits in the TKPROF Output is essential to determine where the core performance problem exists


Getting Oracle’s True Plan of Action, Resource Consumption and Rowstats

Rows Row Source Operation ------- --------------------------------------------------- 1 SORT AGGREGATE (cr=6449761 r=353 w=0 time=15856109 us) 999 NESTED LOOPS (cr=6449761 r=353 w=0 time=15855025 us) 1429 INDEX RANGE SCAN INDX_POP_ORDID_ID (cr=11 r=9 w=0 time=75665 us)(object id 89425) 999 INLIST ITERATOR (cr=6449750 r=344 w=0 time=15773066 us) 999 INDEX RANGE SCAN INDX_PO_ITEM_ITEMID_POPKGID (cr=6449750 r=344 w=0 time=12583443

us)(object id 61254)


Waits in TKPROF Output Elapsed times include waiting on following events:

Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited -------------- ----------------- db file sequential read 353 0.02 2.79 ********************************************************************************


10053 - The Optimizer Trace


CBO – Means different things to different people


Can you get better statistics and better plans with anything

other than a COMPUTE?


COMPUTE on CAT_STUFF_ONLINE - 10046 SELECT /* With COMPUTE on CAT_STUFF_ONLINE */

FROM CAT_STUFF_ONLINE t0 , CMS_COMMON_ACTIVE t2 , CMS_STUFF t1 WHERE….

ORDER BY t0.AVG_RATING DESC , LOWER ( t0.NAME ) ASC , t0.REL_DATE DESC

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------

Parse 1 0.01 0.00 0 0 0 0

Execute 1 0.00 0.00 0 0 0 0

Fetch 58 8.91 19.33 42294 61490 0 57

------- ------ -------- ---------- ---------- ---------- ---------- ----------

total 60 8.92 19.34 42294 61490 0 57

Misses in library cache during parse: 1

Optimizer goal: CHOOSE

Parsing user id: 25 (SSONLINE)

Rows Row Source Operation

------- ---------------------------------------------------

57 SORT ORDER BY (cr=61490 r=42294 w=0 time=19332601 us)

57 HASH JOIN (cr=61490 r=42294 w=0 time=19331528 us)

57 HASH JOIN (cr=47203 r=42294 w=0 time=13025076 us)

830 TABLE ACCESS FULL CAT_STUFF_ONLINE (cr=544 r=0 w=0 time=22956 us)

457333 TABLE ACCESS FULL CMS_STUFF (cr=46659 r=42294 w=0 time=12595961 us)

2041718 TABLE ACCESS BY INDEX ROWID CMS_COMMON (cr=14287 r=0 w=0 time=4533405 us)

2042063 INDEX RANGE SCAN INDX_CMS_COMMON_ACTIVE_FLAG (cr=3705 r=0 w=0 time=1241017 us)(object id 24769)


SELECT /* Without STATS on CAT_STUFF_ONLINE */

FROM CAT_STUFF_ONLINE t0 , CMS_COMMON_ACTIVE t2 , CMS_STUFF t1 WHERE….

ORDER BY t0.AVG_RATING DESC , LOWER ( t0.NAME ) ASC , t0.REL_DATE DESC

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------

Parse 1 0.01 0.00 0 0 0 0

Execute 1 0.00 0.00 0 0 0 0

Fetch 58 0.01 0.01 0 2998 0 57

------- ------ -------- ---------- ---------- ---------- ---------- ----------

total 60 0.02 0.02 0 2998 0 57

Misses in library cache during parse: 1

Optimizer goal: CHOOSE

Parsing user id: 25 (SSONLINE)

Rows Row Source Operation

------- ---------------------------------------------------

57 SORT ORDER BY (cr=2998 r=0 w=0 time=17502 us)

57 NESTED LOOPS (cr=2998 r=0 w=0 time=16594 us)

57 NESTED LOOPS (cr=2825 r=0 w=0 time=15637 us)

830 TABLE ACCESS BY INDEX ROWID CAT_STUFF_ONLINE (cr=309 r=0 w=0 time=3241 us)

830 INDEX RANGE SCAN OF INDX_CAT_STUFF_ONL_CAT_ID

(NON-UNIQUE) (cr=4 r=0 w=0 time=697 us)(object id 7333)

57 TABLE ACCESS BY INDEX ROWID CMS_STUFF (cr=2516 r=0 w=0 time=10379 us)

830 INDEX UNIQUE SCAN OF CMS_STUFF_PK (cr=1662 r=0 w=0 time=4521 us)(object id 24536)

57 TABLE ACCESS BY INDEX ROWID OF CMS_COMMON (cr=173 r=0 w=0 time=801 us)

57 INDEX UNIQUE SCAN OF INDX_CMS_COMMON_FOREIGN_ID (cr=116 r=0 w=0 time=415 us)(object id 24768)

No Stats on CAT_STUFF_ONLINE - 10046


/* WITH COMPUTE on CAT_STUFF_ONLINE */

Table stats Table: CAT_STUFF_ONLINE Alias: T0

TOTAL :: CDN: 71918 NBLKS: 541 AVG_ROW_LEN: 49

Column: P_ID Col#: 1 Table: CAT_STUFF_ONLINE Alias: T0

NDV: 35012 NULLS: 0 DENS: 2.8562e-05 LO: 100002 HI: 998260270

NO HISTOGRAM: #BKT: 1 #VAL: 2

-- Index stats

INDEX NAME: CAT_STUFF_ONLINE_PK COL#: 1 5

TOTAL :: LVLS: 1 #LB: 213 #DK: 71918 LB/K: 1 DB/K: 1 CLUF: 1130

INDEX NAME: INDX_CAT_STUFF_ONL_CAT_ID COL#: 5


COMPUTE on CAT_STUFF_ONLINE - 10053


/* WITHOUT STATS on CAT_STUFF_ONLINE */

Table stats Table: CAT_STUFF_ONLINE Alias: T0

TOTAL :: (NOT ANALYZED) CDN: 44189 NBLKS: 541 AVG_ROW_LEN: 100

Column: P_ID Col#: 1 Table: CAT_STUFF_ONLINE Alias: T0

NO STATISTICS (using defaults)

NDV: 1381 NULLS: 0 DENS: 7.2416e-04


-- Index stats

INDEX NAME: CAT_STUFF_ONLINE_PK COL#: 1 5


INDEX NAME: INDX_CAT_STUFF_ONL_CAT_ID COL#: 5


No stats on CAT_STUFF_ONLINE - 10053



SINGLE TABLE ACCESS PATH

Column: CATEGORY_I Col#: 5 Table: CAT_STUFF_ONLINE Alias: T0

NDV: 23 NULLS: 0 DENS: 4.3478e-02 LO: 1082905 HI: 1097616


TABLE: CAT_STUFF_ONLINE ORIG CDN: 71918 ROUNDED CDN: 3127 CMPTD CDN: 3127

Access path: tsc Resc: 54 Resp: 54

Skip scan: ss-sel 0 andv 35012

ss cost 35012

table io scan cost 54

Access path: index (no sta/stp keys)

Index: CAT_STUFF_ONLINE_PK

TABLE: CAT_STUFF_ONLINE

RSC_CPU: 0 RSC_IO: 264

IX_SEL: 1.0000e+00 TB_SEL: 4.3478e-02

Access path: index (equal)

Index: INDX_CAT_STUFF_ONL_CAT_ID



IX_SEL: 0.0000e+00 TB_SEL: 4.3478e-02

BEST_CST: 54.00 PATH: 2 Degree: 1

Stats on CAT_STUFF_ONLINE - 10053




Column: CATEGORY_I Col#: 5 Table: CAT_STUFF_ONLINE Alias: T0

NO STATISTICS (using defaults)

NDV: 1381 NULLS: 0 DENS: 7.2416e-04


TABLE: CAT_STUFF_ONLINE ORIG CDN: 44189 ROUNDED CDN: 442 CMPTD CDN: 442



Index: CAT_STUFF_ONLINE_PK



IX_SEL: 1.0000e+00 TB_SEL: 1.0000e-02

Access path: index (stp-guess)

Index: INDX_CAT_STUFF_ONL_CAT_ID



IX_SEL: 0.0000e+00 TB_SEL: 4.0000e-03


Single Table Access Path for CAT_STUFF_ONLINE - 10053



Column: PRIMARY_PR Col#: 38 Table: CMS_STUFF Alias: T1

NDV: 2 NULLS: 446860 DENS: 5.0000e-01


TABLE: CMS_STUFF ORIG CDN: 516400 ROUNDED CDN: 511718 CMPTD CDN: 511718



Index: CMS_STUFF_PK

TABLE: CMS_STUFF


IX_SEL: 1.0000e+00 TB_SEL: 1.0000e+00


Single Table Access Path for CMS_STUFF - 10053



Join cardinality: 3127 = outer (3127) * inner (518190) * sel (1.9298e-06) [flag=0]

Best NL cost: 7881 resp: 7881

. . .

HA Join

Outer table:

resc: 4754 cdn: 3127 rcz: 338 deg: 1 resp: 4754

. . .

Join result: cost: 5259 cdn: 3127 rcz: 355

Final:

CST: 5259 CDN: 3127 RSC: 5259 RSP: 5259 BYTES: 1110085

IO-RSC: 5259 IO-RSP: 5259 CPU-RSC: 0 CPU-RSP: 0

Join Selection - 10053



Join cardinality: 442 = outer (442) * inner (518190) * sel (1.9298e-06) [flag=0]

Best NL cost: 1328 resp: 1328

HA Join

Outer table:

resc: 207 cdn: 518190 rcz: 17 deg: 1 resp: 207

. . .

Hash join one ptn Resc: 2569 Deg: 1

hash_area: 256 (max=13107) buildfrag: 1835 probefrag: 18803 ppasses: 1

Hash join Resc: 7255 Resp: 7255

Final:

CST: 1363 CDN: 442 RSC: 1362 RSP: 1362 BYTES: 190502

IO-RSC: 1362 IO-RSP: 1362 CPU-RSC: 0 CPU-RSP: 0

Join Selection - 10053


Is OS Diagnostics the ONLY way?


Performance Case Study

Problem Definition Customer decommissions old hardware and migrates to

new hardware (Sun 880 to Sun T2000) Upgrades from Solaris 9 to Solaris 10 Simple and straightforward queries are much slower

(171.5 seconds vs. 73 seconds) after upgrade to new Sun T2000

Tuning Goal Identify source of performance degradation


Performance Case Study - OLD select… call count cpu elapsed disk query current rows ----------- ------------ ----------- ----------- -------- --------------- ------------ ---------- Parse 1 0.02 0.02 0 1 0 0 Execute 1 0.00 0.00 0 3 0 0 Fetch 8 75.59 74.17 145 35194 0 92 ------- ------- --------- ---------- ---------- ---------- ---------- total 10 75.61 74.19 145 35198 0 92 ------- ---------- ------- -------- ---------- ---------- ---------- ---------- Misses in library cache during parse: 1 Optimizer goal: CHOOSE Parsing user id: 21 (XXXXX) Rows Row Source Operation ------- --------------------------------------------------- 92 SORT GROUP BY (cr=35194 r=145 w=0 time=74171837 us) 68122 NESTED LOOPS (cr=35194 r=145 w=0 time=73633565 us) 73.6 seconds for this step 3519 NESTED LOOPS (cr=17599 r=142 w=0 time=337865 us) 3519 NESTED LOOPS (cr=10559 r=142 w=0 time=265304 us) 3519 TABLE ACCESS BY INDEX ROWID OBJ#(8351) (cr=3519 r=142 w=0 time=164903 us) 3548 INDEX RANGE SCAN OBJ#(13475) (cr=15 r=15 w=0 time=29265 us)(object id 13475) 3519 TABLE ACCESS BY INDEX ROWID OBJ#(8249) (cr=7040 r=0 w=0 time=78796 us) 3519 INDEX RANGE SCAN OBJ#(8255) (cr=3521 r=0 w=0 time=48717 us)(object id 8255) 3519 TABLE ACCESS BY INDEX ROWID OBJ#(8288) (cr=7040 r=0 w=0 time=56896 us) 3519 INDEX UNIQUE SCAN OBJ#(8291) (cr=3521 r=0 w=0 time=28746 us)(object id 8291) 68122 TABLE ACCESS FULL OBJ#(13570) (cr=17595 r=3 w=0 time=73258253 us)

The most expensive step!


Performance Case Study - OLD Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 8 0.00 0.00 db file sequential read 142 0.01 0.06 db file scattered read 1 0.01 0.01 SQL*Net message from client 8 0.00 0.01 ********************************************************************************


Performance Case Study - NEW select… call count cpu elapsed disk query current rows ----------- ------------ ----------- ----------- -------- --------------- ------------ ---------- Parse 1 0.08 0.05 0 0 0 0 Execute 1 0.00 0.00 0 3 0 0 Fetch 8 177.89 173.74 0 31554 0 92 ------- ------- ------ -------- ---------- ---------- ---------- ---------- total 10 177.97 173.80 0 31557 0 92 ------- ------- ------ -------- ---------- ---------- ---------- ---------- Misses in library cache during parse: 1 Optimizer goal: CHOOSE Parsing user id: 21 (XXXXX) Rows Row Source Operation ------- --------------------------------------------------- 92 SORT GROUP BY (cr=31554 r=0 w=0 time=29952790 us) 68122 NESTED LOOPS (cr=31554 r=0 w=0 time=172170616 us) 172.1 seconds for this step 3519 NESTED LOOPS (cr=17478 r=0 w=0 time=465352 us) 3519 NESTED LOOPS (cr=10438 r=0 w=0 time=313726 us) 3519 TABLE ACCESS BY INDEX ROWID OBJ#(19467) (cr=3398 r=0 w=0 time=101746 us) 3548 INDEX RANGE SCAN OBJ#(39204) PARTITION: KEY KEY (cr=8 r=0 w=0 time=15111 us)(object id 39204) 3519 TABLE ACCESS BY INDEX ROWID OBJ#(17864) (cr=7040 r=0 w=0 time=158839 us) 3519 INDEX RANGE SCAN OBJ#(18530) (cr=3521 r=0 w=0 time=91792 us)(object id 18530) 3519 TABLE ACCESS BY INDEX ROWID OBJ#(17881) (cr=7040 r=0 w=0 time=111884 us) 3519 INDEX UNIQUE SCAN OBJ#(18549) (cr=3521 r=0 w=0 time=52940 us)(object id 18549) 68122 TABLE ACCESS FULL OBJ#(17896) (cr=14076 r=0 w=0 time=171580852 us) 171.5 seconds of the marked “NESTED LOOP” is occurring in this “one step”

The most expensive step!


Performance Case Study - NEW Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 8 0.00 0.00 SQL*Net message from client 8 0.00 0.01 ********************************************************************************


Performance Case Study – OLD (truss –c)

syscall seconds calls errors _exit .000 2 read .006 245 write .006 212 open .013 94 2 close .000 101 chdir .000 1 brk .014 126 stat .002 199 111 lseek .001 82 getpid .000 21 getuid .000 8 fstat .000 56 access .001 28 22 setsid .000 2 1 dup .000 2 pipe .000 2 times .010 667 shmat .000 2 shmctl .000 1 shmdt .000 2 shmget .000 4 3 semctl .000 1 semtimedop .000 1 ioctl .000 13 3 execve .000 2 umask .000 2 fcntl .000 28 getdents .000 6 lstat .000 2


Performance Case Study – OLD (truss –c)

syscall seconds calls errors sigprocmask .001 31 sigaction .000 105 sigfillset .000 2 getcontext .000 2 getustack .000 2 setustack .000 2 fstatvfs .000 5 mmap .006 134 mprotect .000 2 munmap .000 30 readv .000 1 setrlimit .000 4 getrlimit .000 17 memcntl .001 26 uname .000 3 sysconfig .000 19 fork1 .001 1 pread .040 567 door_info .000 5 door_call .000 5 lwp_schedctl .000 2 resolvepath .001 34 -------- ------ ---- ---- sys totals: .114 2911 142 -------- ------ ---- ---- usr time: 75.568


Performance Case Study – NEW (truss –c)

syscall seconds calls errors _exit .000 2 read .009 247 write .008 212 open .003 79 2 close .001 80 chdir .000 1 brk .002 124 stat .005 139 88 lseek .001 83 getpid .000 22 getuid .000 8 fstat .000 20 access .001 36 28 setsid .000 2 1 dup .000 2 pipe .000 2 times .011 761 shmat .001 2 shmctl .000 1 shmdt .000 2 shmget .000 4 3 semctl .000 1 semtimedop .000 1 ioctl .001 16 5 execve .002 2 umask .000 2 fcntl .000 20 sigaction .000 55 sigfillset .000 2


Performance Case Study – NEW (truss –c)

syscall seconds calls errors getcontext .000 2 setustack .000 2 mmap .004 119 mprotect .000 2 munmap .001 27 setrlimit .000 4 getrlimit .000 12 memcntl .005 23 uname .000 3 sysconfig .000 17 fork1 .001 1 lwp_self .000 1 lwp_sigmask .000 18 pollsys .066 3441 door_info .000 5 door_call .000 5 schedctl .000 2 resolvepath .001 32 getcwd .000 2 -------- ------ ---- sys totals: .133 5646 127 -------- ------ ---- usr time: 178.964

The pollsys calls are not the core of the problem! Elapsed time ~ .06 seconds!!


Performance Case Study – OLD vs. NEW (OS Tracing)

Traced the Solaris 10 Oracle server process and compared to the Solaris 9 Last resort – Solaris 10 DTRACE Facility Ran dappprof to get more information from the operating system The results were very revealing!!!


Performance Case Study – OLD vs. NEW (OS Tracing)

Ran DTRACE (using dappprof) on Solaris 10 $ ./dappprof -c -e -o -T -p 10711 SPID of the Oracle process CALL CPU evaddi 15397897 evadid 15987167 lnxsub 16884455 evadrn 20201062 evadiv 30267626 evadan 55163571 evaval 56127612 ldxftd 160104654 evadnj 179183554 expeal 549975042 qertb 2483019003 ldxjtc 114572802099 qctginf 114574653567 ldxite 114578553305 ldxeti 114584457590 lnxmul 114586658306 xtyginf 114595565455 ldxads 114605059802 evaopn2 114705155175 expepr 115097827088 lnxcpn 229161554108 lnxsni 343781573330 TOTAL: 1608541753964

These 2 functions are utilized for datamapping and character conversion in the NLS!!!


Metalink search revealed fixing of some bugs related to lnxcpn and lnxsni in 9.2.0.6/9.2.0.7 Reproduced the problem on a 9.2.0.7 database Problem was not reproducible on Oracle 10gR2 Bottom line – It was NOT the OS…it is how Oracle was ported on this platform for the version of the OS! Response Time is King…!

Performance Case Study : Post-Fix!!!


Q & A/Wrap Up


Gratitude!!! My customers My peers at the Oaktable Network (http://www.oaktable.net) For life’s daily lessons

http://www.oaktable.net


References 1. Vaidyanatha, Gaja : Oracle Performance Tuning 101, Oracle

Press, 2000 2. Vaidyanatha et. al. : Oracle Insights - Tales of the Oaktable,

Apress, 2004


Conclusion Oracle Performance Tuning is not wizardry or witchcraft Response time management - Key to performance optimization Laundry list tuning (cache-hit ratios) is not productive use of tuning time


Thank You!

Response Time Management for Oracle Databases · Author: Gaja Krishna Vaidyanatha Subject: Oracle...

Documents

Transcript of Response Time Management for Oracle Databases · Author: Gaja Krishna Vaidyanatha Subject: Oracle...