4/22/2008 © 2008 IBM Corporation
IBM Systems & Technology Group
Technical Conference
14 – 18 April, 2008, Sevilla, Spain
Oracle/AIX Performance and Tuning
Session ID : pS014
Fabienne Lepetit – Oracle/IBM Joint Solutions CenterSébastien Chabrolles – IBM – Power Benchmark PSSC Montpellier
[email protected] / [email protected]
Version 2.0
21.04.2008
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation2 4/22/2008
This presentation…
� … is a Oracle/IBM joint work : IBM PSSC Power benchmark center & IBM/Oracle Joint Solutions Center.
� … is based on our experience and feedback of benchmarks
� … is a checklist which can be used in most production environment
� … can be widely distributed to BPs and customers
� … has been built with many written informations in order to be easily read and presented – don’t hesitate to reuse it!
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation3 4/22/2008
Learning objectives
� Configure AIX to support Oracle with the best performance
� Setup Oracle to take advantage of AIX features
� Understand the joint effort to tune both Oracle and AIX and how they interact
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation4 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation5 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation6 4/22/2008
Oracle Server Architecture
DBWR
System Global Area
Buffer Cache Log BufferSharedSQL
DB Files
Program Global AreaPrivate SQLArea
Oracle
database
Oracle
instance
LGWR SMON…
DIA0Background processes
Memory structures
Redologs Control files
Database files
how to manage Oracle memory structures
how to implement thedata on disk
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation7 4/22/2008
The components of an Oracle server are : � database = set of datafiles (data + redologs + control files)� instance = memory structures allocated at startup + background processes (automatically forked at startup) associated to a database
� Database files
� datafiles – contains data (tables, indexes…)� redologs – contains redo entries� control files – record the physical structure of the database
� Oracle memory structures = System Global Area (SGA) + Process Global Area (PGA)
� SGA � shared memory region that contains data and control information for one Oracle instance� allocated at instance startup and deallocated at instance shutdown � each instance has its own SGA
� PGA � memory buffer that contains data and control information for a server process� PGA is created by Oracle when a server process is started� contains private SQL area
Oracle Server Architecture (cont’d)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation8 4/22/2008
� The three main components of the SGA are :
• buffer cache:- store the most recently used blocks of data- improves performance by reducing disk I/O activity
• shared pool contains : - shared SQL area (parse tree and execution plan for SQL statements) - dictionnary cache- library cache
• redolog bufferstores redo entries (log of changes made to the database) written to the online redo log (used during database recovery)
Oracle Server Architecture (cont’d) – Focus on the SGA
System Global AreaBuffer Cache Log BufferSharedSQL
Program Global AreaPrivate SQL Area
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation9 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation10 4/22/2008
� Objective :
Tune the VMM to protect computational pages
(Programs, SGA, PGA) from being paged out and force the LRUD to steal pages from FS-CACHE only.
1 - AIX is started, applications load some computational pages into the memory.
As a UNIX system, AIX will try to take advantage of the free memory by using it as a cache file to reduce the IO on the physical drives.
2 - The activity is increasing, the DB needs more memory but there is no free pages available. LRUD (AIX page stealer) is starting to free some pages into the memory.
3 - With the default setting, LRUD will page out some computational pages instead of removing some pages from the File System Cache.
Virtual Memory Management
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation11 4/22/2008
Memory : VMM Tuning for Filesystems (lru_file_repage)
This VMM tuning tip is applicable for AIX 5.2 ML4+ or AIX 5.3 ML1+
► lru_file_repage :
● This new parametter prevents the computational pages to be paged out.
● By setting lru_file_repage=0 (1 is the default value) you’re telling the VMM that your preference is to steal pages from FS Cache only.
► “New” vmo tuning approach : Recommendations based on VMM developer experience.
Use “vmo –p –o” to set the following parametters :
lru_file_repage = 0
minfree = 120 x # lcpus Increase if page stealing activity, vmstat “po” column
maxfree = minfree +(maxpaghead x # lcpus)
minperm% = from 3 to 10
maxperm% = maxclient% = from 70 to 90
strict_maxclient = 1
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation12 4/22/2008
Memory : VMM Tuning for Filesystems (“legacy”)
Recommended VMM tuning for AIX version prior to 5.2 ML4 or AIX 5.3 ML1.
The following VMM tuning as the same objective : protect the computational pages to be page out.
► vmo “legacy” tuning approach :
Use “vmo –p –o” to set the following parametters :
minfree = 120 x # lcpus Increase if page stealing activity, vmstat “po” column
maxfree = minfree +(maxpaghead x # lcpus)
minperm% = from 3 to 10
maxperm% = maxclient% = from 5 to 15 but < minperm%
strict_maxclient = 0
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation13 4/22/2008
Memory : AIX Memory Management for OracleSystem-wide contention scope (user thread/kernel thread one-to-one mapping) – reduces memory usage
of Oracle process & makes AIX post/wait mechanism most effective
► In Oracle9i, set AIX environment variable AIXTHREAD_SCOPE=S
► Since Oracle10g, AIXTHREAD_SCOPE=S is Internally set
Pinning shared memory – prevents paging out SGA (only if App + DB on same AIX and >80%computationnal pages)
► On AIX part
● vmo –p –o v_pinshm = 1● Leave maxpin% at the default of 80% unless the SGA exceeds 77% of real memory :
vmo –p –o maxpin%=[(total mem-SGA size)*100/total mem] + 3
► On Oracle part (9i/10g)
● LOCK_SGA = TRUE
Large Page Support – improves performance of prefetching
► On AIX part
● vmo –r –o lgpg_size = 16777216 –o lgpg_regions=(SGA size / 16 MB)● chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle (allow Oracle user ID to use
Large Pages)● ldedit –b lpdata oracle (to allow oracle binary to use large page data)● export LDR_CNTRL=LARGE_PAGE_TEXT=Y@LARGE_PAGE_DATA=M (prior Oracle user ID to
starting Oracle instance and listener to allow both large page text and large page data)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation14 4/22/2008
Memory : Oracle Memory Structures Allocation
8i : Manual management
► All Oracle memory structures were manually managed and were mostly static (db_block_buffers…)
9i : Dynamic memory resizing
db_cache_size (dynamic parameter)
►sga_max_size (static parameter) – maximum size of the SGA for the lifetime of the instance.
►pga_aggregate_target (dynamic parameter) – specifies the target aggregate PGA memoryavailable to all server processes attached to the instance
►additional parameter : db_cache_advice (dynamic parameter) – enables or disables statistics gathering used for predicting behavior with different cache sizes.
10g : Automatic Shared Memory Management (ASMM)
►sga_target (dynamic) – if set the db_cache_size is automatically sized (shared_pool_size, large_pool_size, java_pool_size as well).
Can be increased up to sga_max_size.
To use Automatic Shared Memory Management, sga_target must be <>0.
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation15 4/22/2008
Memory : Oracle Memory Structures Allocation NEW!11g
11g : Automatic Memory Management (AMM)
►memory_target (dynamic parameter) – specifies the total memory size to be used by the instance – SGA and PGA. Exchanges between SGA and PGA are done according to the needs.
If sga_target and pga_aggregate_target are not set the policy is to give 60% of memory_target to the SGA and 40% to the PGA.
►memory_max_target (static parameter) – specifies the maximum memory size for the instance.
To use Automatic Memory Management, memory_target must be <>0.
See Metalink notes 443746.1 and 452512.1 explaining AMM and these new parameters.
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation16 4/22/2008
Memory : Use jointly AIX dynamic LPAR and Oracle dynamic allocation of memory + AMM
Scenario :Memory_max_size= 18 GB
Real memory= 12 GB
Memory_target= 8 GB
SGA
+
PGA
Oracle tuning advisor indicates that SGA+PGA need to be increased to 11GB:
memory_target can be increased dynamically to 11GB but real memory is only 12GB, so it needs to be increased as well.
► Step 1 - Increase physical memory allocated to the LPAR to 15 GB
► Step 2 - Increase SGA+PGA allocated to the instance of the database to 11GB
AIX
+ free
Memory_max_size= 18 GB
Initial configuration Final configuration
� Memory allocated to the system has been increased dynamically, using AIX DLPAR
� Memory allocated to Oracle (SGA and PGA) has been increased on the fly
ssh hscroot@hmc "chhwres -r mem
-m <system> -o a -p <LPAR name> -q 3072"
alter system set memory_target=11GB;
Real memory= 15 GB
AIX
+ free
Memory_target= 11 GB
SGA
+
PGA
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation17 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation18 4/22/2008
IO : Database Layout
Stripe and mirror everything (S.A.M.E) approach:
�Goal is to balance I/O activity across all disks, loops, adapters, etc...
�Avoid/Eliminate I/O hotspots
�Manual file-by-file data placement is time consuming, resource intensive and iterative
�Additional advices to implement SAME :• apply the SAME strategy to data, indexes
• if possible separate redologs (+archivelogs)
�Having a good Storage configuration is a key point :
� Because disk is the slowest part of an infrastructure
� Reconfiguration can be difficult and time consuming
Oracle
recommendation
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation19 4/22/2008
� RAID-5 vs. RAID-10 Performance Comparison
ExcellentExcellentRandom Read
ExcellentFairRandom Write
GoodExcellentSequential Write
ExcellentExcellentSequential Read
RAID-10RAID-5I/O Profile
IO : RAID Policy with ESS, DS6/8K
� With Enterprise class storage (with huge
cache), RAID-5 performances are comparable to
RAID-10 one (for most customer workloads)
� Consider RAID-10 for workloads with a high percentage of random write activity (> 25%) and high I/O access densities (peak > 50%)
Use RAID-5 or RAID-10 to create striped LUNs
If possible try to minimize the number of LUNs per RAID array to avoid contention on physical disk.
HW Striping
LUN 1
LUN 2
LUN 3
LUN 4
Storage
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation20 4/22/2008
IO : 2nd Striping (LVM)
1. Luns are striped across physical disks (stripe-size : ~ 64k, 128k, 256k)
2. LUNs are seen as hdisk device on AIX server.
3. Create AIX Volume Group(s) (VG) with LUNs from multiple arrays
4. Logical Volume striped across hdisks (stripe-size : 8M, 16M, 32M, 64M)
=> each read/write access to the LV are well balanced accross LUNs and use the maximum number of physical disks for best performance.
HW Striping
LUN 1
LUN 2
LUN 3
LUN 4
StorageAIX
hdisk1
hdisk2
hdisk3
hdisk4
LV
M S
trip
ing
Volume Group
LV
(ra
wo
r w
ith
jfs2
)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation21 4/22/2008
IO : LVM Spreading / Striping
2 different ways to stripe on the LVM layer :
1. Stripe using Logical Volume (LV)
► Create Logical Volume with the striping option : mklv –S <stripe-size> ...
Valid LV Stripe sizes:
● AIX 5.2: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1M
1. AIX 5.3+: AIX 5.2 Stripe sizes + 2M, 4M, 16M, 32M, 64M, 128M
2. PP striping (AKA spreading)
► Create a Volume Group with a 8M,16M or 32M PPsize. (PPsize will be the “Stripe size”)
● AIX 5.2 : Choose a Big VG and change the “t factor” : # mkvg –B –t <factor> –s <PPsize>...
● AIX 5.3 +: Choose a Scalable Volume Group : # mkvg –S –s <PPsize> ...
► Create LV with “Maximum range of physical volume” option to spread PP on different hdisk in a
Round Robin fashion : # mklv –e x ...
jfs2 filesystem creation advice :
If you create a jfs2 filesystem on a striped (or PP spreaded) LV, use the INLINE logging option. It willavoid « hot spot » by creating the log inside the filesystem (which is striped) instead of using a uniq PP stored on 1 hdisk.
# crfs –a logname=INLINE …
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation22 4/22/2008
IO : Disk Subsystem Advices
� Check if the definition of your disk subsystem is present in the ODM.
� If the description shown in the output of “lsdev –Cc disk” the word “Other”, then it means that AIX doesn’t have a correct definition of your disk device in the ODM and use a generic device definition.
# lsdev –Cc disk
� In general, a generic device definition provides far from optimal performance since it doesn’t properly customize the hdisk device :
exemple : hdisk are created with a queue_depth=1
1. Contact your vendor or go to their web site to download the correct ODM definition for your storage subsystem. It will setup properly the “hdisk” accordingly to your hardware for optimal performance.
2. If AIX is connected to the storage subsystem with several Fiber Channel Cards for performance, don’t forget to install a multipath device driver or path control module.
- sdd or sddpcm for IBM DS6000/DS8000
- powerpath for EMC disk subsystem
- hdlm for Hitachi etc....
Generic device definitionbad performance
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation23 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation24 4/22/2008
IO : Filesystems Mount Options (DIO, CIO)If Oracle data are stored in a Filesystem, some mount option can improve performance :
�Direct IO (DIO) – introduced in AIX 4.3.
• Data is transfered directly from the disk to the application buffer, bypassing the file buffer cache hence avoiding double caching (filesystem cache + Oracle SGA). • Emulates a raw-device implementation.
�To mount a filesystem in DIO$ mount –o dio /data
�Concurrent IO (CIO) – introduced with jfs2 in AIX 5.2 ML1
• Implicit use of DIO. • No Inode locking : Multiple threads can perform reads and writes on the same file at the same time. • Performance achieved using CIO is comparable to raw-devices.
�To mount a filesystem in CIO:$ mount –o cio /data
Bench throughput over run duration – higher
tps indicates better performance.
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation25 4/22/2008
� Benefits :
1. Avoid double caching : Some data are already cache in the Application layer (SGA)
2. Give a faster access to the backend disk and reduce the CPU utilization
3. Disable the inode-lock to allow several threads to read and write the same file (CIO only)
IO : Benefits of CIO for Oracle
� Restrictions :
1. Because data transfer is bypassing AIX buffer cache, jfs2 prefetching and write-behind can’t be used. These functionnalities can be handled by Oracle.
⇒ (Oracle parameter) db_file_multiblock_read_count = 8, 16, 32, ... , 128 according to workload
2. When using DIO/CIO, IO requests made by Oracle must by aligned with the jfs2 blocksize to avoid a demoted IO (Return to normal IO after a Direct IO Failure)
=> When you create a JFS2, use the “mkfs –o agblksize=XXX” Option to adapt the FSblocksize with the application needs.
Rule : IO request = n x agblksize
Exemples: if DB blocksize > 4k ; then jfs2 agblksize=4096
Redolog are always written in 512B block; So jfs2 agblksize must be 512
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation26 4/22/2008
IO : Direct IO demoted
� How to detect demoted IO :
Trace command to check demoted io :# trace –aj 59B,59C ; sleep 2 ; trcstop ; trcrpt –o directio.trcrpt# grep demoted directio.trcrpt
Waits on redolog (with demoted IO, FS blk=4k)
Waits% Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn log file sync 2,229,324 0.00 62,628 28 1.53
Waits on redolog (without demoted IO, FS blk=512)
Waits% Time -outs Total Wait Time (s) Avg wait (ms) Waits /txnlog file sync 494,905 0.00 1,073 2 1.00
� Extract from Oracle AWR (test made in Montpellier with Oracle 10g)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation27 4/22/2008
CIO/DIO implementation Advices
mount -o rw mount -o rw
mount -o rw mount -o cio *(1)
Cached by Oracle
Cached by AIX
mount -o rw mount -o cio (jfs2 + agblksize=512)
Cached by Oracle
Cached by AIX
mount -o rw mount -o rbrw
Use JFS2 write-behind …
but are not kept in AIX Cache.
mount -o rw mount -o rw
Oracle bin and shared lib.
with Standard mount options with optimized mount options
Cached by AIX Cached by AIX
Oracle Datafiles
Oracle Redolog
Cached by Oracle
Cached by Oracle
Cached by AIXCached by AIX
Cached by AIXOracle Archivelog
Oracle Control files
*(1) : to avoid demoted IO : jfs2 agblksize = Oracle DB block size / n
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation28 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation29 4/22/2008
IO : Oracle Automatic Storage Management (ASM)
• ASM is a database file system which provide cluster file system and volume manager capabilities. Alternative to conventional file-system and LVM functions. For Oracle datafiles.
• Integrated to Oracle DB with no additional cost, for single or RAC databases
• With ASM, the management of Oracle datafiles is the same for the DBA on all platforms (Unix, Linux, Windows)
• Datafiles are striped across all ASM disks - I/O is spread evenly to prevent hot spots and maximize performance
• Online add/drop of disk devices with automatic online redistribution of data
LUN 1
LUN 2
AIX
hdisk1
ASM DISKGROUP
StorageOracle
ASM striping
LUN 4
LUN 3
hdisk2
hdisk3
hdisk4
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation30 4/22/2008
IO : Oracle Automatic Storage Management (ASM)
• ASM is implemented as a special kind of Oracle instance, with its own SGA and backgroundprocesses (by default, less than 100M of SGA are allocated for the ASM instance) and generate very little activity except during rebalance operations.
• A single ASM instance can service one or more instance databases.
• Can maintain redundant copies of data
• An ASM-managed database will have approximately the same performance as a database implemented in raw-devices.
ASM Instance
ASM DISKGROUP
Database on
ASM
create tablespace DATA
datafile ‘+DG/data01.dbf’ size 15G;
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation31 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation32 4/22/2008
• Allows multiple requests to be sent without to have to wait until the disk subsystem has completed the physical IO.
• Utilization of asynchronous IO is strongly advised whatever the type of file-system and mount option implemented (JFS, JFS2, CIO, DIO).
IO : Asynchronous IO (AIO)
�Posix vs Legacy
Since AIX5L V5.3, two types of AIO are now available : Legacy and Posix. For the moment, theOracle code is using the Legacy AIO servers.
aioQ
Application aioserversDisk
1
2
3 4
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation33 4/22/2008
IO : Asynchronous IO (AIO) tuning
� Rule of thumb :
maxservers should be = (10 * <# of disk accessed concurrently>) / # cpu
maxreqs (= a multiple of 4096) should be > 4 * #disks * queue_depth
� but only tests allow to set correctly minservers and maxservers
• important tuning parameters :
check AIO configuration with : AIX 5.X : lsattr –El aio0 AIX 6.1 : ioo –L | grep aio
• maxreqs : size of the Asynchronous queue.
• minserver : number of kernel proc. Aioservers to start (AIX 5L system wide).
• maxserver : maximum number of aioserver that can be running per logical CPU
� Monitoring :In Oracle’s alert.log file, if maxservers set to low : “Warning: lio_listio returned EAGAIN” “Performance degradation may be seen”
#aio servers used can be monitored via “ps –k | grep aio | wc –l” , “iostat –A” or nmon (option A)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation34 4/22/2008
IO : Asynchronous IO (AIO) fastpath
• FS with CIO/DIO and AIX 5.3 TL5+ :� Activate fsfast_path (comparable to fast_path but for FS + CIO/DIO)
AIX 5L : adding the following line in /etc/inittab: aioo:2:once:aioo –o fsfast_path=1
AIX 6.1 : ioo –p –o aio_fsfastpath=1 (default setting)
• Raw Devices / ASM :� check AIO configuration with : lsattr –El aio0
enable asynchronous IO fast_path. : AIX 5L : chdev -a fastpath=enable -l aio0 (default since AIX 5.3)AIX 6.1 : ioo –p –o aio_fastpath=1 (default setting)
Application Disk
1
2
3
With fast_path, IO are queued directly from the application into the LVM layer without any “aioservers kproc” operation.
� Better performance compare to non-fast_path� No need to tune the min and max aioservers� No ioservers proc. => “ps –k | grep aio | wc –l” is not relevent, use “iostat –A” instead
AIX Kernel
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation35 4/22/2008
� How to set filesystemio_options parameter
Possible values
ASYNCH : enables asynchronous I/O on file system files (default)DIRECTIO : enables direct I/O on file system files (disables AIO)SETALL SETALL SETALL SETALL : enables both asynchronous and direct I/O on file system filesNONE : disables both asynchronous and direct I/O on file system files
Since the version 10g, Oracle will open data files located on the JFS2 file system
with the O_CIO option if the filesystemio_options initialization parameter is set to
either directIO or setall.
Advice : set this parameter to ‘setall’…
Note : set the disk_asynch_io parameter to ‘true’ as well
IO : AIO,DIO/CIO & Oracle Parameters
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation36 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation37 4/22/2008
CPU : Simultaneous Multi-Threading (SMT)
P6 Core
Registerthread1
Registerthread2
• Feature available on POWER5 and POWER6 system running AIX
5.3 or 6.1
• allows 2 instruction path to share access to the POWER execution units on every clock cycle
• instruction path = “virtual processor” = logical POWER processor (i.e. a server with 4 physical processors will have 8 logical processors with SMT enabled)
• SMT allows to use resources that would otherwise usually be
unexploited and can achieve significant throughput gains – up to
40% greater performance
• No source code modification needed to use SMT
How to control SMT:
� smtcl [–m on | off]
� “-m on” enables the SMT mode, “-m off” disables it
� “-w boot” makes SMT change effective on reboot
� “-w now” makes SMT change effective immediately but not persistent across reboots
thread2thread1
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation38 4/22/2008
CPU: Micro-Partitioning
An example given based on a PeopleSoft/Oracle10g RAC benchmark (2,400 Virtual Users) ran at
PSSC MOP (DLPAR performance vs. SPLPAR one)
SPLPAR ConfigurationTotal Throughput (bytes):2,715,373,065Action_Transaction: 53.256 secs
DLPAR ConfigurationTotal Throughput (bytes):2,037,677,193Action_Transaction: 59.07 secs
Micro-Partitioning give the possibility to share processors between several LPARs. It improve CPU utilisation by
reducing the number of idle cycle. (available on POWER5 and POWER6 system running AIX 5.3+)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation39 4/22/2008
CPU: Compatibility Matrix
DL
PA
R
SM
T
Mic
ro-
Pa
rtitio
n
AIX 5.2
AIX 5.3
AIX 6.1
Oracle 9i
Oracle 10g
Oracle 11g
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation40 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation41 4/22/2008
Monitoring & Tuning : Methodology
� tuning is an iterative process
� a minimun of changes should be performed at each iteration
� tuning is a joint effort between the system administrator and the DBA
� don’t hesitate to keep a log to track changes and the impact of these changes
OS monitoring
Oracle monitoring
design/parameters changes
DBA System administrator
impact ?
log of changes
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation42 4/22/2008
Monitoring & Tuning (cont’d) : System Monitoring Tools
� vmstat – useful for obtaining an overall picture of CPU, paging, and memory usage
� lvmstat – useful to get logical volume I/O statistics
� iostat – allows to get statistics on disk activity (and also on terminals, CPU, adapters)
� lparstat – reports logical partition related information and statistics
� nmon – complete tool which gives information on all the components of the system
� filemon – uses the trace facility to report on the I/O activity of physical volumes, logical volumes, individual files, and the Virtual Memory Manager
� xmperf – allows to define monitoring environments to supervise the performance of the local and remote systems
� netstat – allows to monitor network activity
....
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation43 4/22/2008
Monitoring & Tuning (cont’d) : Oracle Monitoring Tools
� AWR is a standard toolwhich allow the collection, automation, storage andviewing of performance data. Replaces utlbstat/utlestat and Statspack.
� Grid Control (10g) provide a graphical, web-based, centralized management of databases.
�Tkprof allows to take and exploit SQL traces, gives explain plans etc…
�Toad, SpotLight …graphical tools (Quest)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation44 4/22/2008
IO
storage monitoring tool
vmstat (wa)
iostat, nmon,filemon
accesses well spread? hot spots?
jfs mount options?
AIO?
Monitoring & Tuning (cont’d) : Methodology SummaryPerformance issue?
Monitor the system, diagnose the problem.Tools to use, questions to ask…
Metalink note 282036.1
alert.log
bdump,cdump,udump…
init.ora/spfile
AWR
Grid Control
strategy of analyze?
memory allocated to instances (SGA, PGA)
tkprof (parsing, explain plans, bind variables…)
( and check ulimits + AIXTHREAD_SCOPE=S...)
CPU
topas, nmon
vmstat (us, sys, id, r)
sar –P ALL
CPU bound?
runqueue?
workload ?
parallelization of workload?
MEMORY
topas, nmon
vmstat (pi, po)
pagingout activity?
memory allocated to Oracle?
other applications running?
other Oracle
instances?
Network
Topas, nmon
netstat,netpmon
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation45 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation46 4/22/2008
Automatic Workload Repository (AWR)
… is the tool to use to collect information about the behavior of a database!
• Collects, automates, stores performance statistics for problem determination.
• AWR is setuped during the creation of the database is automaticgenerates reports in html formatreports more information than Statspack
�prefer AWR to statspack reports!
• Snapshots (set of statistics gathered at a single time) are taken automatically each 60 minutes or can be run manually:
execute dbms_workload_repository.create_snapshot();
• AWR reports are generated by running $ORACLE_HOME/rdbms/admin/awrrpt.sql and providing 2 snapshot Ids. Can be also generated using the Grid Control.
• To enable most of the new statistical gathering and advisors, ensure that the parameterSTATISTICS_LEVEL is set to TYPICAL (recommended).
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation47 4/22/2008
Automatic Workload Repository (cont’d)
• Compare reports to mesure the impact of tuning and to check the behavior of the database
•Use Metalink and/or the Oracle documentation to get the definition of events, statistics etc…
• Advised to run manually additional snapshots during performance issue phases / optimization campaign
• No shutdown between two snapshots!
• Choose the most representative snapshots IDs to generate the report.
snapshot
period analyzed in AWR report
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation48 4/22/2008
Performance tuning advisors
Automatic Database Diagnostic Monitor (ADDM)
• Based on AWR, invoked automatically every time that a new AWR snapshot is generated
• Powerful self-diagnostic engine which :
� analyze the system
� indentify major problems
� recommend corrective actions!!!
• ADDM reports are generated by running $ORACLE_HOME/rdbms/admin/addmrpt.sql and providing 2 snapshot Ids – Can also be generated using the Grid Control.
• In 11g, for RAC clusters, ADDM analyze the system at the cluster level.
SQL Tuning Advisor (STA)
• Detects missing or stale statistics, missing indexes…
• Suggests new execution plan which can be applied to the system and used without any change in the application
• Advised to use STA through Grid Control – but also possible to invoke it using DBMS_SQLTUNE package.
NEW!11g
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation49 4/22/2008
AWR Report : Header & Hit Ratios
…
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation50 4/22/2008
AWR Report : Top 5 timed events
This “Top 5 Timed Events” table is very important – it lists the events to focus on and will lead the performance analysis.
According the event listed in the table, refer to the corresponding section in the report.
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation51 4/22/2008
AWR Report : SQL ordered by…
Sections « SQL statement ordered by… »present SQL statements ordered by differentscriterias.
They are useful to detect a problem of parsing, a response time of a particular statement etc…
SQL is clickable and the whole statement ispresented later in the report.
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation52 4/22/2008
AWR Report : Cache sizes and advisors
Buffer Pool Advisory help to tune the bufferpool, givingestimation of the impact of changes of the size of the buffer pool.
Advisors exist for:- PGA memory - Shared Pool - Sga_target - …
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation53 4/22/2008
Oracle Grid Control Targets : databases
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation54 4/22/2008
Oracle Grid Control Focus on one database
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation55 4/22/2008
Oracle Grid ControlPerformance graphs
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation56 4/22/2008
Oracle Grid Control Advisor Central
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation57 4/22/2008
Oracle Grid Control ADDM (Automatic Database Diagnostic Monitor)
Focus needed on
these statements!
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation58 4/22/2008
Oracle Grid Control ADDM (Automatic Database Diagnostic Monitor)
Statement with most impact on performances
Run the advisor to get a tuning hint!
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation59 4/22/2008
Oracle Grid Control SQL Tuning Advisor
You can implement the advice (or not!)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation60 4/22/2008
Oracle Grid Control Focus on one query
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation61 4/22/2008
Oracle Grid Control Focus on one query
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation62 4/22/2008
Oracle Grid Control
• Grid is the most powerful tool to identify issues, get advices, tune your database
• It facilitates DB administration tasks and allows to perform the maintenance of the database (backups etc…)
• Targets monitored by the Grid are the hosts, databases, OracleApplications Servers, Oracle eBusiness Suite etc…
• Allows to have a central point for the management of the targets
• Helps the DBA, but does not replace him/her! ;-)
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation63 4/22/2008
Agenda
�Oracle Server Architecture
�Memory
� AIX VMM tuning
� Oracle Memory Structure Allocation
� IO
� AIX LVM Striping
� AIX mount option
� Oracle ASM
� Asynchronous IO
�CPU
�Monitoring & Tuning
� Methodology & tools
� Focus on Two Oracle Tools: AWR & Grid Control
�Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation64 4/22/2008
• Implement and get the best of Oracle on AIX is the result of a joint effort between the OS administrator and the Oracle DBA.
• The tuning process is made of a first guess parameter setting and then, iterations to change progressively the configuration to get the best results…
• Prefer JFS2 + Concurrent IO or ASM.
• Always use Asynchronous IO servers and, if available, use FastPath.
• Implement SMT if available on your system.
• Use AIX, Oracle and storage monitoring tools (AIX : Nmon, Topas – Oracle : AWR & Grid Control – Storage : TPC…)
• … and feel free to contact us for additional questions!!!
Conclusion
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation65 4/22/2008
Addendum - Useful note for Oracle running on AIX platforms
•Before installing an Oracle product on AIX, it is strongly advised to have a look at the
Metalink note 282036.1 : Minimum Software Versions and Patches Required to Support Oracle Products on IBM pSeries
It contains all mandatory AIX PTFs & APARs and Oracle patches to install on a system is updated on a regular basis and
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation66 4/22/2008
Addendum - Certification matrix : Oracle database Enterprise Edition on IBM AIX based Systems
When upgrading to Oracle 10GR2 on AIX 6.1, download and run an updated version of rootpre script before proceeding with the Oracle Database install (refer to bug 6613550 for more details).
10th of april 2008
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation67 4/22/2008
Oracle/AIX Performance and Tuning
Session ID : pS014
[email protected]@fr.ibm.com
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation68 4/22/2008
Reference Materials – DRAFT -
Thanks to the authors of the following documents :
• Improving Database Performance with AIX Concurrent I/O – S. Kashyap, B. Olszewski, R Hendrickson
• Performance Management Guide – AIX5L version 5.3 documentation
• Oracle 9i & 10g on IBM AIX5L : Tips & Considerations – IBM/Oracle ICC
• Oracle Architecture and Performance Tuning on AIX – R. Ballough
• Configuring IBM TotalStorage for Oracle OLTP Applications – D. Martin
• Tuning IBM AIX5L for an Oracle Database – D. Massanari, M. Riviere
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation69 4/22/2008
EMEA Oracle/IBM Joint Solutions Center
C/O Compagnie IBM France
Rue de la Vieille Poste
34006 Montpellier Cedex - FRANCE
[email protected] : +33 4 67 34 67 49
IBM Systems & Technology Group – Technical Conference 2008
© 2008 IBM Corporation70 4/22/2008
TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
The following are trademarks or registered trademarks of other companies.
* All other products may be trademarks or registered trademarks of their respective companies.
Notes:
Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:
*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not
actively marketed or is not significant within its relevant market.
Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.
Top Related