Oracle Database 11g RAC Performance Tuning
-
Upload
yelena-bytenskaya -
Category
Documents
-
view
285 -
download
1
Transcript of Oracle Database 11g RAC Performance Tuning
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
1/52
Instance Recovery in RAC
Learning Objective
After completing this topic, you should be able to
identify how instance recovery works in a RAC environment
1. Elements of RAC instance recovery
When tuning your system, it is important that you compare the CPU time with the wait
time of your system. Comparing CPU time with wait time helps to determine how much of
the response time is spent on useful work and how much on waiting for resources
potentially held by other processes.
As a general rule, the systems where CPU time is dominant usually need less tuning than
the ones where wait time is dominant. On the other hand, heavy CPU usage can be
caused by badly written !" statements.
Graphic
In this example graph, the X-axis is Wait time and the Y-axis is CPU time. The
application is scalable when the CPU and Wait time are eqal in proportion. When
the CPU time is more than the wait time, the application might need !"# tning.
When the wait time is more than the CPU time, it needs instance or $%C tning.In sch a case, no gain is achie&ed b' adding CPUs or nodes.
Spplement
!electing the lin( title opens the resorce in a new browser window.
Style consi!erations
#iew more information on the style considerations for Oracle $$g%atabase used
in this course.
Lanch "in!o"
Although the proportion of CPU time to wait time always tends to decrease as load on the
system increases, steep increases in wait time are a sign of contention and must be
addressed for good scalability.
Adding more CPUs to a node, or nodes to a cluster, would provide very limited benefit
http://dowindow%28%27../t2/misc/lwod_orac_a03_it_enust201.html')http://dowindow%28%27../t2/misc/lwod_orac_a03_it_enust201.html') -
8/10/2019 Oracle Database 11g RAC Performance Tuning
2/52
under contention. Conversely, a system where the proportion of CPU time does not
decrease significantly as load increases can scale better, and would most likely benefit
from adding CPUs or &eal Application Clusters or &AC instances if needed.
#ote
%tomatic Wor(load $epositor' or %W$ reports displa' CPU time together with
wait time in the Top ) Timed *&ents section, i+ the CPU time portion is among the
top +i&e e&ents.
Although there are specific tuning areas for &AC such as instance recovery and
interconnect traffic, you get most benefits by tuning your system like a single'instance
system. At least this must be your starting point. Obviously, if you have seriali(ation
issues in a single'instance environment, these may be e)acerbated with &AC.
*ou have basically the same tuning tools with &AC as with a single'instance system.
+owever, certain combinations of specific wait events and statistics are well'known &ACtuning cases.
*ou see some of those specific combinations, as well as the &AC'specific information
that you can get from the nterprise -anager performance pages and tatspack and
AW& reports. inally, you see the &AC'specific information that you can get from the
Automatic %atabase %iagnostic -onitor or A%%-.
When an instance fails and the failure is detected by another instance, the second
instance performs these recovery steps.
Graphic
$eco&er' time process is di&ided into two phases. irst phase contains steps
and o+ the process, sch as $emaster enqee resorces and $emaster cache
resorces. In this phase, 'o se in+ormation +or other caches and the #/!
reco&ers 0$1. The second phase contains steps 2, 3, and ), which are 4ild
reco&er' set, $esorce claim, and $oll +orward reco&er' set. In this phase, !/56
reco&ers the database and merges +ailed redo threads.
Remaster en$ee resorces
%uring the first phase of recovery, /lobal n0ueue ervices or / remasters the
en0ueues.
Remaster cache resorces
1he /lobal Cache ervices or /C remasters its resources. 1he /C processes
remaster only those resources that lose their masters. %uring this time, all /C resource
re0uests and write re0uests are temporarily suspended. +owever, transactions can
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
3/52
continue to modify data blocks as long as these transactions have already ac0uired the
necessary resources.
%il! recovery set
After en0ueues are reconfigured, one of the surviving instances can grab the 2nstance
&ecovery en0ueue. 1herefore, at the same time as /C resources are remastered,-O3 determines the set of blocks that need recovery. 1his set is called the recovery set.
4ecause, with Cache usion, an instance ships the contents of its blocks to the re0uesting
instance without writing the blocks to the disk, the on'disk version of the blocks may not
contain the changes that are made by either instance.
1his implies that -O3 needs to merge the content of all the online redo logs of each
failed instance to determine the recovery set. 1his is because one failed thread might
contain a hole in the redo that needs to be applied to a particular block. o redo threads of
failed instances cannot be applied serially. Also, redo threads of surviving instances are
not needed for recovery because -O3 could use past or current images of their
corresponding buffer caches.
Resorce claim
4uffer space for recovery is allocated and the resources that were identified in the previous
reading of the redo logs are claimed as recovery resources. 1his is done to avoid other
instances to access those resources.
Roll for"ar! recovery set
All resources re0uired for subse0uent processing have been ac0uired and the /lobal
&esource %irectory or /&% is now unfro(en. Any data blocks that are not in recovery can
now be accessed. 3ote that the system is already partially available. 1hen, assuming that
there are past images or current images of blocks to be recovered in other caches in thecluster database, the most recent image is the starting point of recovery for these
particular blocks.
2f neither the past image buffers nor the current buffer for a data block is in any of the
surviving instances5 caches, then -O3 performs a log merge of the failed instances.
-O3 recovers and writes each block identified in step 6, releasing the recovery
resources immediately after block recovery so that more blocks become available as
recovery proceeds.
After all blocks have been recovered and the recovery resources have been released, the
system is again fully available.
2n summary, the recovered database or the recovered portions of the database becomes
available earlier, and before the completion of the entire recovery se0uence. 1his makes
the system available sooner and it makes recovery more scalable.
#ote
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
4/52
The per+ormance o&erhead o+ a log merge is proportional to the nmber o+ +ailed
instances and to the si7e o+ the amont o+ redo written in the redo logs +or each
instance.
&estion
When an instance fails and the failure is detected by another instance, the second
instance performs a series of recovery steps. What is the first step performed by
the second instance7
Options'
$. 1he /&% is unfro(en
8. 1he / remasters the en0ueues
6. 4uffer space for recovery is allocated
9. 1he /lobal Cache ervices :/C; remasters its resources
Ans"er
Option 1:Incorrect. 1ring the +inal step in instance reco&er', all resorces
reqired +or sbseqent processing ha&e been acqired and the 0lobal $esorce
1irector' 80$19 is n+ro7en. %n' data bloc(s that are not in reco&er' can then be
accessed.
Option 2:Correct. 1ring the +irst step in instance reco&er', the 0lobal *nqee
!er&ices 80*!9 remasters the enqees. 5nce this step has been per+ormed, one
o+ the sr&i&ing instances can grab the Instance $eco&er' enqee, 0C!resorces are remastered, and !/56 determines the set o+ bloc(s that need
reco&er'.
Option 3:Incorrect. 1ring the +orth step per+ormed dring instance reco&er',
b++er space +or reco&er' is allocated and the resorces that were identi+ied in the
pre&ios reading o+ the redo logs are claimed as reco&er' resorces. This is done
to a&oid other instances accessing those resorces.
Option 4:Incorrect. 1ring the second step in instance reco&er', the 0lobal
Cache !er&ices 80C!9 remasters its resorces. The 0C! processes remaster
onl' those resorces that lose their masters. 1ring this time, all 0C! resorcereqests and write reqests are temporaril' sspended. :owe&er, transactions
can contine to modi+' data bloc(s as long as these transactions ha&e alread'
acqired the necessar' resorces.
Correct ans"er(s)'
8. 1he / remasters the en0ueues
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
5/52
1he e)ample illustrates the degree of database availability during each step of Oracle
instance recovery.
Graphic
The example +or 5racle instance reco&er' contains a database a&ailabilit' ranging
+rom 6one, Partial, and ll with increasing *lapsed time.
A
&eal Application Clusters is running on multiple nodes.
%
3ode failure is detected.
C
1he en0ueue part of the /&% is reconfigured< resource management is redistributed to the
surviving nodes. 1his operation occurs relatively 0uickly.
*
1he cache part of the /&% is reconfigured and -O3 reads the redo log of the failed
instance to identify the database blocks that it needs to recover.
E
-O3 issues the /&% re0uests to obtain all the database blocks it needs for recovery.
After the re0uests are complete, all other blocks are accessible.
+
1he Oracle server performs roll forward recovery. &edo logs of the failed threads areapplied to the database, and blocks are available right after their recovery is completed.
G
1he Oracle server performs rollback recovery. Undo blocks are applied to the database for
all uncommitted transactions.
,
2nstance recovery is complete and all data is accessible.
1he dotted steps represent the ones identified in the &AC and 2nstance or Crash
&ecovery process. 1he dashed line represents the blocks identified while remastering
cache resources in this process.
2n a single'instance environment, the instance startup combined with the crash recovery
time is controlled by the setting of the FAST_START_MTTR_TARGETinitiali(ation
parameter. *ou can set its value if you want incremental checkpointing to be more
aggressive than autotune checkpointing. +owever, this is at the e)pense of a much
higher 2=O overhead.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
6/52
2n a &AC environment, including the startup time of the instance in this calculation is
useless because one of the surviving instances is doing the recovery.
2n a &AC environment, it is possible to monitor the estimated target :in seconds; for the
duration from the start of instance recovery to the time when /C% is open for lock
re0uests for blocks that are not needed for recovery. 1his estimation is published in the
V$INSTANCE_RECOVERYview through the ESTD_CLUSTER_AVAILABLE_TIMEcolumn.
4asically, you can monitor the time your cluster is fro(en during instance recovery
situations.
2n a &AC environment, the FAST_START_MTTR_TARGETinitiali(ation parameter is used
to bind the entire instance recovery time, assuming that it is instance recovery for single'
instance death.
#ote
I+ 'o reall' want to ha&e small instance reco&er' time b' setting
FAST_START_MTTR_TARGET, 'o can sa+el' ignore the alert log messages abot
raising its &ale.
+ere are some guidelines you can use to make sure that instance recovery in your &AC
environment is faster>
se parallel instance recovery
Use parallel instance recovery by setting RECOVERY_PARALLISM.
increase PARALLEL_EXECUTION_MESSAGE_SIZE
2ncrease PARALLEL_EXECUTION_MESSAGE_SIZE from its default of 8,$9? bytes to 9 @4
or ? @4. 1his should provide better recovery slave performance.
set PARALLEL_MIN_SERVERS
et PARALLEL_MIN_SERVERStoCPU_COUNT-1. 1his will prespawn recovery slaves at
startup time.
se asynchronos I-O an!
Using asynchronous 2=O is one of the most crucial factors in recovery time. 1he first'pass
log read uses asynchronous 2=O.
increase !efalt bffer cache si/e2nstance recovery uses B percent of the default buffer cache for recovery buffers. 2f this is
not enough, some of the steps of instance recovery will be done in several passes. *ou
should be able to identify such situations by looking at your alertlo!file. 2n that case,
you should increase the si(e of your default buffer cache.
Smmary
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
7/52
When the wait time of a system is more than the CPU time, it needs instance or &AC
tuning. pecific wait events, system and en0ueue statistics, nterprise -anager
performance pages, tatspack and AW& reports, and A%%- reports help in &AC specific
tuning.
1he recovery time in &AC and instance or crash recovery processes is divided into two
phases. 1he first phase includes remastering en0ueue and cache resources. 1he second
phase includes building recovery set, resource claim, and roll forward of recovery set.
%atabase availability ranges from none to full during the Oracle instance recovery
process. ollow simple but important guidelines to ensure fast instance recovery in your
&AC environment.
RAC 0ait Events an! Latencies
Learning Objective
After completing this topic, you should be able to
recognize typical wait events and latencies for RAC
1. Cache fsion latencies an! "ait events
1he effect of accessing blocks in the global cache and maintaining cache coherency are
represented as follows>
the /lobal Cache ervices statistics for current and cr blocks such as gc current blocks received
and gc cr blocks received and
the /lobal Cache ervices wait events for gc current block 6'way, gc cr grant 8'way, and so on
1he response time for Cache usion transfers is determined by the messaging time and
processing time imposed by the physical interconnect components, the 2PC protocol, and
the /C protocol.
2t is not affected by disk input=output or 2=O factors other than occasional log writes. 1he
Cache usion protocol does not re0uire 2=O to data files in order to guarantee cache
coherency, and &AC inherently does not cause any more 2=O to disk than a nonclusteredinstance.
2n a &AC AW& report, the /lobal Cache and n0ueue ervices ' Workload
Characteristics table in the &AC tatistics section contains average times :latencies; for
some /lobal Cache ervices and /lobal n0ueue ervices operations.
1hose latencies should be monitored over time, and significant increases in their values
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
8/52
should be investigated. 1he table presents some typical values, based on empirical
observations. 1he factors that may cause variations to those latencies are as follows>
Graphic
The 0lobal Cache and *nqee !er&ices - Wor(load Characteristics table has
two colmns. The +irst colmn incldes ser&ices and the second colmn
incldes a &ale +or each ser&ice. The ser&ices in the +irst colmn are %&g
global enqee get time 8ms9, %&g global cache cr bloc( recei&e time 8ms9, %&g
global cache crrent bloc( recei&e time 8ms9, %&g global cache cr bloc( bild time
8ms9, %&g global cache cr bloc( send time 8ms9, 0lobal cache log +lshes +or cr
bloc(s ser&ed ;, %&g global cache cr bloc( +lsh time 8ms9, %&g global cache
crrent bloc( pin time 8ms9, %&g global cache crrent bloc( send time 8ms9, 0lobal
cache log +lshes +or crrent bloc(s ser&ed ;, and %&g global cache crrent bloc(
+lsh time 8ms9.
utili(ation of the 2PC protocol, user'mode 2PC protocols are faster, but only 1ru9Ds &%/ is
recommended for use
scheduling delays, when the system is under high CPU utili(ation, and
log flushes for current blocks served
Other &AC latencies in AW& reports are mostly derived from V$GES_STATISTICSand
may be useful for debugging purposes, but do not re0uire fre0uent monitoring.
Graphic
The table incldes +or colmns < %W$ $eport #atenc' 6ame, #ower 4ond,
T'pical, and Upper 4ond.The latenc' names are %&erage time to process cr
bloc( reqest, %&g global cache cr bloc( recei&e time 8ms9, %&erage time to
process crrent bloc( reqest, and %&g global cache crrent bloc( recei&e time
8ms9. The &ales +or #ower 4ond, T'pical, and Upper 4ond are pro&ided +or
each latenc'.
#ote
The time to process consistent read or C$ bloc( reqest in the cache corresponds
to
8"#l%t&e'(l#)*t&e')e+%t&e9, and the time to process crrent bloc(
reqest in the cache corresponds to 8,+t&e'(l#)*t&e')e+%t&e.
Analy(ing what sessions are waiting for is an important method to determine where time
is spent. 2n &AC, the wait time is attributed to an event that reflects the e)act outcome of
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
9/52
a re0uest.
or e)ample, when a session on an instance is looking for a block in the global cache, it
does not know whether it will receive the data cached by another instance or whether it
will receive a message to read from disk.
1he wait events for the global cache convey precise information and wait for global cache
blocks or messages. 1hey are mainly categori(ed by the following>
summari(ed in a broader category called Cluster Wait Class
temporarily represented by a placeholder event that is active while waiting for a block, and
attributed to precise events when the outcome of the re0uest is known
When it takes some time to ac0uire resources because of the total path length and
latency for re0uests, processes sleep to avoid spinning for indeterminate periods of time.
When the process decides to wait, it wakes up either after a specific timer value e)pires
:timeout; or when the event it is waiting for occurs and the process is posted. 1he wait
events are recorded and aggregated in the views.
1he first three are aggregations of wait times, timeouts, and the number of times waited
for a particular event, whereas the rest enable the monitoring of waiting sessions in real
time, including a history of recent events waited for.
Graphic
There are se&en &iews.
The =>!Y!T*/?*=*6T &iew displa's total waits +or an e&ent. The
=>!*!!I56?W%IT?C#%!! &iew waits +or a wait e&ent class b' a session. The
=>!*!!I56?*=*6T &iew waits +or an e&ent b' a session. The
=>%CTI=*?!*!!I56?:I!T5$Y &iew displa's acti&it' o+ recent acti&e sessions.
The =>!*!!I56?W%IT?:I!T5$Y &iew displa's last @ wait e&ents +or each
acti&e session. The =>!*!!I56?W%IT &iew displa's e&ents +or which acti&e
sessions are waiting. The =>!"#!T%T! &iews identi+' !"# statements impacted
b' interconnect latencies.
or most of the global cache wait events, the parameters include file number, block
number, the block class, and access mode dispositions, such as mode held and
re0uested.
1he wait times for events presented and aggregated in these views are very useful when
debugging response time performance issues. 3ote that the time waited is cumulative,
and that the event with the highest score is not necessarily a problem.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
10/52
+owever, if the available CPU power cannot be ma)imi(ed, or response times for an
application are too high, the top wait events provide valuable performance diagnostics.
#ote
Use the CLUSTER_.AIT_TIMEcolmn in V$S/LSTATSto identi+' !"#
statements impacted b' interconnect latencies, or rn an %11/ report on the
corresponding %W$ snapshot.
&estion
Which wait event view contains a CLUSTER_.AIT_TIMEcolumn that can be used
to identify !" statements impacted by interconnect latencies7
Options'
$. V$S/LSTATS
8. V$SESSION_.AIT
6. V$SYSTEM_EVENT
9. V$SESSION_.AIT_0ISTORY
Ans"er
Option 1:Correct. Use the CLUSTER_.AIT_TIMEcolmn in V$S/LSTATSto
identi+' !"# statements impacted b' interconnect latencies, or rn an %11/
report on the corresponding %W$ snapshot.
Option 2:Incorrect. This wait e&ent &iew shows e&ents +or which acti&e sessions
are waiting.
Option 3:Incorrect. This wait e&ent &iew shows total waits +or an e&ent.
Option 4:Incorrect. This wait e&ent &iew shows last @ wait e&ents +or each acti&e
session.
Correct ans"er(s)'
$. V$S/LSTATS
. Cache "ait events an! en$ee "aits
1he main global cache wait events for Oracle %atabase $Bgare briefly described.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
11/52
gc[current/cr][multil!c"]re#ue$t
1hese wait events are relevant only while a gc re0uest for a cr or current buffer is in
progress. 1hey act as placeholders until the re0uest completes.
gc[current/cr][%/&]'()*
A current or cr block is re0uested and received after two or three network hops. 1here0uest is processed immediately> the block is not busy or congested.
gc[current/cr]l!c"u$*
A current or cr block is re0uested and received, but is not sent immediately by "-
because some special condition that delayed the sending was found.
gc[current/cr]gr)nt%'()*
A current or cr block is re0uested and a grant message received. 1he grant is given
without any significant delays. 2f the block is not in its local cache, a current or cr grant is
followed by a disk read on the re0uesting instance.
gccurrentgr)ntu$*
A current block is re0uested and a grant message received. 1he busy hint implies that the
re0uest is blocked because others are ahead of it or cannot be handled immediately.
gc[current/cr][l!c"/gr)nt]c!nge$te,
A current or cr block is re0uested and a block or grant message received. 1he congested
hint implies that the re0uest spent more than $ ms in internal 0ueues.
gc[current/cr][-)ilure/retr*]
A block is re0uested and a failure status received or some other e)ceptional event has
occurred.
gcu--eru$*
2f the time between buffer accesses becomes less than the time the buffer is pinned in
memory, the buffer containing a block is said to become busy and as a result interested
users may have to wait for it to be unpinned.
or dynamic remastering, two events are of the most practical importance> !
re&a)terand ! 3#e)e. 1hey can be symptoms of the effect of remastering on the
running processes.
1he 4-5a6block re0uest shows what typically happens when the master instance
re0uests a block that is not cached locally. +ere it is supposed that the master instance iscalled SGA1, and SGA4contains the re0uested block. 1he scenario contains these steps>
step 1
SGA1sends a direct re0uest to SGA4. o SGA1waits on the ! #rre+t "lo7
re3#e)tevent.
step an!
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
12/52
When SGA4receives the re0uest, its local "/W& process may need to flush some
recovery information to its local redo log files. or e)ample, if the cached block is
fre0uently changed and the changes have not been logged yet, "- would have to ask
"/W& to flush the log before it can ship the block. 1his may add a delay to the serving of
the block and may show up in the re0uesting node as a busy wait.
step 2
1hen, SGA4sends the re0uested block to SGA1. When the block arrives in SGA1, the wait
event is complete and is reflected as ! #rre+t "lo7 4-5a6.
Using the notation RE time at re0uestor, .E wire time and transfer delay, and SE time at
server, the total time for a round'trip would be> R:send; F .:small msg; F S:process msg,
process block, send; F .:block; F R:receive block;.
1his is a modified scenario for a cluster with more than two nodes. 2t is very similar to the
4-5a6 4lock &e0uest. +owever, the master for this block is on a node that is different
from that of the re0uestor, and where the block is cached. 1hus, the re0uest must beforwarded.
1here is an additional delay for one message and the processing at the master node>
R:send; F .:small msg; F S:process msg, send; F .:small msg; F S:process msg,
process block, send; F .:block; F R:receive block;.
While a remote read is pending, any process on the re0uesting instance that is trying to
write or read the data cached in the buffer has to wait for a gc buffer busy. 1he buffer
remains globally busy until the block arrives.
2n this scenario, a grant message is sent by the master because the re0uested block isnot cached in any instance. 2f the local instance is the resource master, the grant happens
immediately. 2f not, the grant is always 8'way, regardless of the number of instances in
the cluster.
1he grant messages are small. or every block read from the disk, a grant has to be
received before the 2=O is initiated, which adds the latency of the grant round'trip to the
disk latency> R:send; F .:small msg; F S:process msg, send; F .:small msg; F R:receive
block;.
1he round'trip looks similar to a 8'way block round'trip, with the difference that the wire
time is determined by a small message, and the processing does not involve the buffercache.
An en0ueue wait is not &AC specific, but involves a global lock operation when &AC is
enabled.
-ost of the global re0uests for en0ueues are synchronous, and foreground processes
wait for them. 1herefore, contention on en0ueues in &AC is more visible than in single'
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
13/52
instance environments. 1he following are the en0ueue types for which most waits for
en0ueues occur.
34
1ransaction en0ueue< used for transaction demarcation and tracking.
35
1able or partition en0ueue< used to protect table definitions during %-" operations.
,0
+igh'water mark en0ueue< ac0uired to synchroni(e a new block operation.
S&
e0uence en0ueue< used to seriali(e incrementing of an Oracle se0uence number.
3A
n0ueue used mainly for transaction recovery as part of instance recovery.
6S
Undo segment en0ueue< mainly used by the Automatic Undo -anagement :AU-; feature.
2n the case of all the en0ueue types, the waits are synchronous and may constitute
serious seriali(ation points that can be e)acerbated in a &AC environment.
#ote
In 5racle 1atabase @g, the enqee wait e&ents speci+' the resorce name and
a reason +or the wait A +or example, TX *nqee index bloc( split. This ma(es
diagnostics o+ enqee waits easier.
Using system statistics based on V$SYSSTATenables characteri(ation of the database
activity based on averages. 2t is the basis for many metrics and ratios used in various
tools and methods, such as AW&, tatspack, and %atabase Control.
2n order to drill down to individual sessions or groups of sessions, V$SESSTATis useful
when the important session identifiers to monitor are known. 2ts usefulness is enhanced if
an application fills in the MODULEand ACTIONcolumns in V$SESSION.
V$SEGMENT_STATISTICSis useful for &AC because it also tracks the number of C&
and current blocks received by the obGect.
1he &AC'relevant statistics can be grouped into the following categories>
/lobal Cache ervice statistics such as gccrbloc(srecei&ed,gccrbloc(recei&etime, and so on
/lobal n0ueue ervice statistics such as globalenqeegetsand so on, and
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
14/52
tatistics for messages sent such as gcsmessagessentand gesmessagessent
V$EN/UEUE_STATISTICScan be 0ueried to determine which en0ueue has the highest
impact on database service times and eventually response times.
V$INSTANCE_CAC0E_TRANSFERindicates how many current and C& blocks per blockclass are received from each instance, including how many transfers incurred a delay.
Smmary
2n a &AC AW& report you monitor typical latencies of some /lobal Cache ervices and
/lobal n0ueue ervices operations for significant increases in their values. actors
causing variations include utili(ation of the 2PC protocol, scheduling delays, and log
flushes. Other &AC latencies in AW& reports are mostly derived from
V$GES_STATISTICSand may be useful for debugging purposes, but do not re0uire
fre0uent monitoring.
1here are many global cache wait events but for dynamic remastering, two events are of
the most practical importance> gc remaster and gc 0uiesce. 1o manage and monitor
database activities and sessions, you can use session and system statistics such as,
V$SYSSTAT, V$SEGMENT_STATISTICS, V$EN/UEUE_STATISTICS, and
V$INSTANCE_CAC0E_TRANSFER.
RAC 3ning 3ips an! Consi!erations
Learning Objectives
After completing this topic, you should be able to
identify the most common RAC tuning tips
identify factors that affect tuning in a RAC environment
1. +actors affecting tning in RAC
2n any database system, &AC or single instance, the most significant performance gains
are usually obtained from traditional application'tuning techni0ues. 1he benefits of those
techni0ues are even more remarkable in a &AC database. 2n addition to traditional
application tuning, these are some of the techni0ues that are particularly important for
&AC.
Avoi! long fll7table scans
1ry to avoid long full'table scans to minimi(e /C re0uests. 1he overhead caused by the
global C& re0uests in this scenario is because of the fact that when 0ueries result in local
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
15/52
cache misses, an attempt is first made to find the data in another cache, based on the
assumption that the chance that another instance has cached the block is high.
Resi/e an! tne the bffer cache
+ash partitioning may help to reduce buffer busy contention by making buffer access
distribution patterns sparser, enabling more buffers to be available for concurrent access.6se ASS5
Automatic egment pace -anagement can provide instance affinity to table blocks.
Increase se$ence caches
2ncreasing se0uence caches improves instance affinity to inde) keys deriving their values
from se0uences. 1hat techni0ue may result in significant performance gains for multi'
instance insert'intensive applications.
Re!ce interinstance traffic "ith partitioning
&ange or list partitioning may be very effective in conGunction with data'dependent routing,
if the workload can be directed to modify a particular range of values from a particularinstance.
Avoi! nnecessary parsing
2n &AC, library cache and row cache operations are globally coordinated. o e)cessive
parsing means additional interconnect traffic. "ibrary cache locks are heavily used, in
particular by applications using P"=!" or Advanced !ueuing. "ibrary cache locks are
ac0uired in e)clusive mode whenever a package or procedure has to be recompiled.
5inimi/e loc8ing sage
4ecause transaction locks are globally coordinated, they also deserve special attention in
&AC. or e)ample, using tables instead of Oracle se0uences to generate uni0ue numbersis not recommended because it may cause severe contention even for a single instance
system.
Remove nselective in!e9es
2nde)es that are not selective do not improve 0uery performance, but can degrade %-"
performance. 2n &AC, unselective inde) blocks may be subGect to interinstance contention,
increasing the fre0uency of cache transfers for inde)es belonging to INSERT'intensive
tables.
Configre interconnect properly
Always verify that you use a private network for your interconnect, and that your private
network is configured properly. nsure that a network link is operating in full duple) mode.
nsure that your network interface and thernet switches support -1U si(e of H @4. 3ote
that a single /4 can scale up to ten thousand ?'@4 blocks per second before saturation.
&estion
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
16/52
Which common &AC tuning techni0ues may result in significant performance
gains for multi'instance, insert'intensive applications7
Options'
$. using hash partitioning
8. increasing se0uence caches
6. using range or list partitioning
9. avoiding long full'table scans
Ans"er
Option 1:Incorrect. :ash partitioning ma' help to redce b++er bs' contention
b' ma(ing b++er access distribtion patterns sparser, enabling more b++ers to be
a&ailable +or concrrent access.
Option 2:Correct. Increasing seqence caches impro&es instance a++init' to index
(e's deri&ing their &ales +rom seqences. That techniqe ma' reslt in signi+icant
per+ormance gains +or mlti-instance, insert-intensi&e applications.
Option 3:Incorrect. $ange or list partitioning ma' be &er' e++ecti&e in conBnction
with data-dependent roting, i+ the wor(load can be directed to modi+' a particlar
range o+ &ales +rom a particlar instance.
Option 4:Incorrect. Tr' to a&oid long +ll-table scans to minimi7e 0C! reqests.
The o&erhead cased b' the global C$ reqests in this scenario is de to the +act
that when qeries reslt in local cache misses, an attempt is +irst made to +ind the
data in another cache, based on the assmption that the chance that another
instance has cached the bloc( is high.
Correct ans"er(s)'
8. increasing se0uence caches
2n application systems where the loading or batch processing of data is a dominant
business function, there may be performance issues affecting response times because of
the high volume of data inserted into inde)es. %epending on the access fre0uency and
the number of processes concurrently inserting data, inde)es can become hot spots and
contention can be e)acerbated by these three factors.
irst, ordered, monotonically increasing key values in the inde) :right'growing trees;.
econd, fre0uent leaf block splits, and third, low tree depth which means all leaf block
access go through the root block. A leaf or branch block split can become an important
seriali(ation point if the particular leaf block or branch of the tree is concurrently
accessed.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
17/52
1he tables sum up the most common symptoms associated with the splitting of inde)
blocks, listing wait events and statistics that are commonly elevated when inde) block
splits are prevalent. As a general recommendation, to alleviate the performance impact of
globally hot inde) blocks and leaf block splits, a more uniform, less skewed distribution of
the concurrency in the inde) tree should be the primary obGective.
1he primary obGective can be achieved by the following actions>
Graphic
There are two tables. The +irst table contains onl' one colmn named Wait e&ents.
The wait e&ents listed in this colmn are enq TX - indexcontention, gc b++er bs',
gc crrent bloc( bs', and gc crrent split.
The second table also contains onl' one colmn named as !'stem statistics. The
s'stem statistics listed in this colmn are #ea+ node splits, 4ranch node splits,
*xchange deadloc(s, gcs re+se xid, gcs ast xid, and !er&ice IT# waits.
global inde) hash partitioning
increasing the se0uence cache, if the key value is derived from a se0uence
use natural keys as opposed to surrogate keys, and
use reverse key inde)es
2nde)es with key values generated by se0uences tend to be subGect to leaf block
contention when the insert rate is high. 1hat is because the inde) leaf block holding the
highest key value is changed for every row inserted, as the values are monotonically
ascending. 2n &AC, this may lead to a high rate of current and C& blocks transferred
between nodes.
One of the simplest techni0ues that can be used to limit this overhead is to increase the
se0uence cache, if you are using Oracle se0uences. As the difference between se0uence
values generated by different instances increases, successive inde) block splits tend to
create instance affinity to inde) leaf blocks.
Graphic
%n index lea+ bloc( consists o+ )@@ rows. The bloc( has instance and instance .
Instance inserts &ales ...)@@@@ in $%C@. Instance inserts &ales
)@@@...@@@@@ in $%C@.
or e)ample, suppose that an inde) key value is generated by a CAC0E NOORDER
se0uence and each inde) leaf block can hold BB rows. 2f the se0uence cache is set to
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
18/52
BBBB, while instance $ inserts values $, 8, 6, and so on, instance 8 concurrently inserts
BBB$, BBB8, and so on. After some block splits, each instance writes to a different part
of the inde) tree.
1he ideal value for a se0uence cache has a dual purpose. 2t avoids interinstance leaf
inde) block contention and minimi(es possible gaps. One of the main variables to
consider is the insert rate> the higher it is, the higher the se0uence cache must be.
+owever, creating a simulation to evaluate the gains for a specific configuration is
recommended.
#ote
4' de+alt, the cache &ale is @. T'picall', @ is too small +or this example.
)cessive undo block shipment and contention for undo buffers usually happens when
inde) blocks containing active transactions from multiple instances are read fre0uently.
When a SELECTstatement needs to read a block with active transactions, it has to undo
the changes to create a C& version.
2f the active transactions in the block belong to more than one instance, there is a need to
combine local and remote undo information for the consistent read. %epending on the
amount of inde) blocks changed by multiple instances and the duration of the
transactions, undo block shipment may become a bottleneck.
Graphic
In this example, there is an Index with Instance and Instance . It also contains$%C@, $%C@, !0%, and !0%. $%C@ sends changes to Instance and
!0%. There are constant reads between Instance and !0% and between
!0% and $%C@. This reslts in %dditional interconnect tra++ic. The Undo bloc(s
in !0% and !0% compare Instance with $%C@ and $%C@ respecti&el' to
chec( i+ the C$ &ersion exists.
Usually this happens in applications that read recently inserted data very fre0uently, but
commit infre0uently. 1hese are the techni0ues that can be used to reduce such
situations.
horter transactions reduce the likelihood that any given inde) block in the cachecontains uncommitted data, thereby reducing the need to access undo information for
consistent read.
2ncreasing se0uence cache si(es can reduce interinstance concurrent access to inde)
leaf blocks. C& versions of inde) blocks modified by only one instance can be fabricated
without the need of remote undo information.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
19/52
#ote
In $%C, the problem is exacerbated b' the +act that a sbset o+ the ndo
in+ormation has to be obtained +rom remote instances.
A certain combination of wait events and statistics presents itself in applications where
the insertion of data is a dominant business function and new blocks have to be allocated
fre0uently to a segment. 2f data is inserted at a high rate, new blocks may have to be
made available after unfruitful searches for free space.
1his has to happen while holding the +igh'Water -ark or +W- en0ueue. 1herefore, the
most common symptoms for this scenario include a high percentage of wait time for e+38
0. 9 o+te+to+and a high percentage of wait time for !#rre+t!ra+tevents.
Graphic
$%C@ and $%C@ ha&e hea&' inserts dring changes and reads with :W/.
:W/ enqee ma(es new bloc(s a&ailable +or +ree space.
1he former is a conse0uence of the seriali(ation of the +W- en0ueue, and the latter is
because of the fact that current access to the new data blocks that need formatting is
re0uired for the new block operation.
2n a &AC environment, the length of this space management operation is proportional to
the time it takes to ac0uire the +W- en0ueue and the time it takes to ac0uire global locks
for all the new blocks that need formatting. 1his time is small under normal circumstances
because there is never any access conflict for the new blocks.
1herefore, this scenario may be observed in applications with business functions
re0uiring a lot of data loading, and the main recommendation to alleviate the symptoms is
to define uniform and large e)tent si(es for the locally managed and automatic space
managed segments that are subGect to high'volume inserts.
2n data warehouse and data mart environments, it is not uncommon to see a lot of
TRUNCATEoperations.
1hese essentially happen on tables containing temporary data. 2n a &AC environment,
truncating tables concurrently from different instances does not scale well, especially if,you are also using direct read operations such as parallel 0ueries.
A truncate operation re0uires a cross'instance call to flush dirty blocks of the table that
may be spread across instances. 1his constitutes a point of seriali(ation. o, while the
first TRUNCATEcommand is processing, the second has to wait until the first one
completes.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
20/52
Graphic
The example contains !0%and !0%. !0% consists o+ Table and Table.
!0% consists o+ Table with a 1irt' bloc( and Table. Trncate Table is
processing with !0% and Trncate Table with !0%. There is a Cross-instance
call between the two trncate commands. Chec(pointing CDPT connects !0%,!0%, and the Cross-instance call.
1here are different types of cross'instance calls. +owever, all use the same seriali(ation
mechanism.
or e)ample, the cache flush for a partitioned table with many partitions may add latency
to a corresponding parallel 0uery. 1his is because each cross'instance call is seriali(ed at
the cluster level, and one cross'instance call is needed for each partition at the start of
the parallel 0uery for direct read purposes.
&estion
2n a &AC environment, truncating tables concurrently from different instances does
not scale well, especially if, you are also using direct read operations such as
parallel 0ueries. What causes this issue7
Options'
$. undo information has to be obtained from remote instances
8. concurrent cross'instance calls are a point of seriali(ation
6. all leaf block access going through the root block leads to contention9. key values generated by se0uences are subGect to leaf block contention
Ans"er
Option 1:Incorrect. *xcessi&e ndo bloc( shipment and contention +or ndo
b++ers sall' happens when index bloc(s containing acti&e transactions +rom
mltiple instances are read +reqentl'. In $%C, the problem is exacerbated b' the
+act that a sbset o+ the ndo in+ormation has to be obtained +rom remote
instances.
Option 2:Correct. % trncate operation reqires a cross-instance call to +lsh dirt'
bloc(s o+ the table that ma' be spread across instances. This constittes a point
o+ seriali7ation. !o, while the +irst TRUNCATEcommand is processing, the second
has to wait ntil the +irst one completes. There are di++erent t'pes o+ cross-
instance calls. :owe&er, all se the same seriali7ation mechanism.
Option 3:Incorrect. 1epending on the access +reqenc' and the nmber o+
processes concrrentl' inserting data, indexes can become hot spots and
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
21/52
contention can be exacerbated b' low tree depth. That is all lea+ bloc( access
going throgh the root bloc(.
Option 4:Incorrect. Indexes with (e' &ales generated b' seqences tend to be
sbBect to lea+ bloc( contention when the insert rate is high. That is becase the
index lea+ bloc( holding the highest (e' &ale is changed +or e&er' row inserted,
as the &ales are monotonicall' ascending. In $%C, this ma' lead to a high rate o+
crrent and C$ bloc(s trans+erred between nodes.
Correct ans"er(s)'
8. concurrent cross'instance calls are a point of seriali(ation
Smmary
*ou should avoid long full'table scans, use A-, and increase se0uence caches during
&AC tuning. 2nde) 4lock Contention is worsened by increasing key values in the inde),
fre0uent leaf block splits and low tree depth.
*ou can use global inde) hash partitioning for a less skewed distribution of the
concurrency in the inde) tree. 2ncreasing se0uence cache, replacing surrogate with
natural keys, and using reverse key inde)es will also facilitate this concurrency. 4y
increasing the se0uence cache, you can limit the overhead caused by high rates of block
transfer in &AC.
1o handle new data that is fre0uently read but rarely committed, you can use shorter
transactions and increase se0uence cache si(e. 4e sure to define uniform and large
e)tent si(es while dealing with +igh'Water -ark en0ueues.
5onitoring RAC *atabase an! Clster :erformance
Learning Objective
After completing this topic, you should be able to
recognize how to use the Cluster Database Performance pages to monitor RAC database and
cluster performance
1. 3he Clster *atabase :erformance page
4oth Oracle nterprise -anager %atabase Control and /rid Control are cluster'aware
and provide a central console to manage your cluster database.
rom the Cluster %atabase +ome page, you can perform the following functions>
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
22/52
vie" overall system stats
#iew the overall system status, such as the number of nodes in the cluster and their
current status so you do not have to access each individual database instance for details.
vie" alert messages
#iew alert messages aggregated across all the instances with lists for the source of eachalert message.
revie" isses
&eview issues that are affecting the entire cluster as well as those that are affecting
individual instances.
monitor clster cache coherency statistics
-onitor cluster cache coherency statistics to help you identify processing trends and
optimi(e performance for your Oracle &AC environment. Cache coherency statistics
measure how well the data in caches on multiple instances is synchroni(ed.
!etermine availability problems an!
%etermine if any of the services for the cluster database are having availability problems. A
service is determined to be a problem service if it is not running on all preferred instances,
if its response time thresholds are not met, and so on.
revie" alerts
&eview any outstanding Clusterware interconnect alerts.
&estion
Which two tasks can be performed directly from the Cluster %atabase +omepage7
Options'
$. #iew the overall system status
8. -onitor cluster cache coherency statistics
6. -onitor summary charts for cache coherency metrics for the cluster
9. #iew how much work the database is performing on behalf of the users or
applications
Ans"er
Option 1:Correct. rom the Clster 1atabase :ome page, 'o can &iew the
o&erall s'stem stats, sch as the nmber o+ nodes in the clster and their crrent
stats so that 'o do not ha&e to access each indi&idal database instance +or
details.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
23/52
Option 2:Correct. rom the Clster 1atabase :ome page, 'o can monitor
clster cache coherenc' statistics to help 'o identi+' processing trends and
optimi7e per+ormance +or 'or 5racle $%C en&ironment. Cache coherenc'
statistics measre how well the data in caches on mltiple instances is
s'nchroni7ed.
Option 3:Incorrect. The Clster Cache Coherenc' page contains smmar' charts
+or cache coherenc' metrics +or the clster.
Option 4:Incorrect. The 1atabase Throghpt charts, on the Clster 1atabase
Per+ormance page, smmari7e an' resorce contention that appears in the
%&erage %cti&e !essions chart, and also show how mch wor( the database is
per+orming on behal+ o+ the sers or applications.
Correct ans"er(s)'
$. #iew the overall system status
8. -onitor cluster cache coherency statistics
1he Cluster %atabase Performance page provides a 0uick glimpse of the performance
statistics for a database. nterprise -anager accumulates data from each instance over
specified periods of time :known as collection-based data;. nterprise -anager also
provides current data from each instance :known as real-time data;.
tatistics are rolled up across all the instances in the cluster database. Using the links
ne)t to the charts, you can get more specific information and perform any of the following
tasks>
identify the causes of performance issues
decide whether resources need to be added or redistributed
tune your !" plan and schema for better optimi(ation, and
resolve performance issues
1his a partial view of the Cluster %atabase Performance page. *ou access this page by
clicking the :erformancetab on the Cluster %atabase +ome page.
Graphic
This page displa's graphs that pro&ide in+ormation on Clster :ost #oad %&erage,
0lobal Cache 4loc( %ccess #atenc', and %&erage %cti&e !essions.
1he Cluster +ost "oad Average chart on the Cluster %atabase Performance page shows
potential problems that are outside the database. 1he chart shows ma)imum, average,
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
24/52
and minimum load values for available nodes in the cluster for the previous hour.
2f the load average is higher than the average of the total number of CPUs across all the
hosts in the cluster, then too many processes are waiting for CPU resources. !"
statements that are not tuned often cause high CPU usage.
Compare the load average values with the values displayed for CPU Used in the Average
Active essions chart. 2f the session5s value is low and the load average value is high,
something else on the host I other than your database I is consuming the CPU.
*ou can click any of the load value labels for the Cluster +ost "oad Average chart to view
more detailed information about that load value.
or e)ample I if you click the label Average, the +osts> "oad Average page appears,
displaying charts that depict the average host load for up to four nodes in the cluster.
*ou can select whether the data is displayed in a summary chart :combining the data for
each node in one display; or in tile charts :where the data for each node is displayed in its
own chart;. *ou can click Cstomi/eto change the number of tile charts displayed in
each row or the method of ordering the tile charts.
1he /lobal Cache 4lock Access "atency chart shows the latency for each different type
of data block re0uests I current and consistent'read or C& blocks. 1hat is the elapsed
time it takes to locate and transfer consistent'read and current blocks between the buffer
caches.
*ou can click metric for the Global Cache %loc8 Access Latencychart to view more
detailed information about that type of cached block.
2f the /lobal Cache 4lock Access "atency chart shows high latencies :high elapsed
times;, the cause can be any of the following>
a high number of re0uests caused by !" statements that are not tuned
a large number of processes in the 0ueue waiting for the CPU, or scheduling delays, or
slow, busy, or faulty interconnects
When you e)perience slow, busy, or faulty interconnects, check your network connection
for dropped packets, retransmittals, or cyclic redundancy check or C&C errors.
Concurrent read and write activity on shared data in a cluster is a fre0uently occurring
activity. %epending on the service re0uirements, this activity does not usually cause
performance problems.
+owever, when global cache re0uests cause a performance problem, optimi(ing !"
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
25/52
plans and the schema to improve the rate at which data blocks are located in the local
buffer cache, and minimi(ing 2=O is a successful strategy for performance tuning.
2f the latency for consistent'read and current block re0uests reaches $B milliseconds, the
first step in resolving the problem should be to go to the Cluster Cache Coherency page
for more detailed information.
1he Average Active essions chart in the Cluster %atabase Performance page shows
potential problems inside the database.
Categories I called wait classes I show how much of the database is using a resource,
such as CPU or disk 2=O. Comparing CPU time to wait time helps determine how much of
the response time is consumed with useful work rather than waiting for resources that are
potentially held by other processes.
At the cluster database level, this chart shows the aggregate wait class statistics across
all the instances.
1he steps to be followed for a more detailed analysis are as follows>
click the clipboar! icon at the bottom of the chart to view the A%%- analysis for the database for
that time period
The clipboard icon is a chec( mar( present in the %&erage %cti&e !essions graph.
click the wait class legends beside the Average Active essions chart I to view instance'level
information stored in Active essions 4y 2nstance pages
There are 2 legends next to the %&erage %cti&e !essions chart < 5ther, Clster, "eeing,
6etwor(, %dministrati&e, Con+igration, Commit, %pplication, Concrrenc', !'stem IE5, User IE5,
!chedler, and CPU Used.
use the Wait Class action list on the Active essions 4y 2nstance page to view the different wait
classes
use the Cstomi/ebutton I and select the instances that are displayed because Active essions
4y 2nstance pages show the service times for up to four instances, and
use tile charts or combine the data into a single summary chart to view the data for the instances
separately
1he 1hroughput chart on the Performance page monitors the usage of various database
resources. 4y clicking the 3hroghpttab at the top of this chart, you can view the
%atabase 1hroughput chart.
Compare the peaks on the Average Active essions chart with those on the %atabase
1hroughput charts. 2f internal contention is high and throughput is low, consider tuning the
database.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
26/52
1he %atabase 1hroughput charts summari(e any resource contention that appears in the
Average Active essions chart, and also show how much work the database is
performing on behalf of the users or applications. 1he Per econd view shows the
number of transactions compared to the number of logons, and :not shown here; the
amount of physical reads compared to the redo si(e per second.
1he Per 1ransaction view shows the amount of physical reads compared to the redo si(e
per transaction. "ogons is the number of users that are logged on to the database.
*ou can also obtain information at the instance level by clicking one of the legends to the
right of the charts to access the %atabase 1hroughput by 2nstance page. 1his page
shows the breakdown of the aggregated %atabase 1hroughput chart for up to four
instances. *ou can select the instances that are displayed.
*ou can drill down further on the %atabase 1hroughput by 2nstance page to see the
sessions of an instance that is consuming the greatest resources. Click an instance name
legend Gust under the chart to go to the 1op essions subpage of the 1op Consumers
page for that instance.
1he 1hroughput chart on the Performance page monitors the usage of various database
resources. 4y clicking the Instancestab at the top of this chart, you can view the Active
essions by 2nstance chart, which summari(es any resource contention that appears in
the Average Active essions chart. *ou can thus 0uickly determine how much of the
database work is being performed on each instance.
*ou can also obtain information at the instance level by clicking one of the legends to the
right of the chart to access the 1op essions page, where you can view real'time data
showing the sessions that consume the greatest system resources. 2n the graph, the
ora4instance after ?>8B P- consistently shows more active sessions than the ora1
instance.
. Other RAC !atabase an! performance pages
1he steps to access the Cluster Cache Coherency page are as follows>
click the :erformancetab on the Cluster %atabase +ome page
click Clster Cache Coherencyin the Additional -onitoring "inks section, and
click either of the legends of the /lobal Cache 4lock Access "atency chart
There are two legends next to the 0lobal Cache 4loc( %ccess #atenc' chart < %&erage Crrent
4loc( $ecei&e Time and %&erage C$ 4loc( $ecei&e Time.
1he Cluster Cache Coherency page contains summary charts for cache coherency
metrics for the cluster.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
27/52
/lobal Cache 4lock Access "atency shows the total elapsed time, or latency, for a block
re0uest. Click one of the legends to the right of the chart to view the average time it takes
to receive data blocks for each block type :current or C&; by instance.
On the Average Current 4lock &eceive 1ime 4y 2nstance page, you can click an instance
legend under the chart to go to the 4lock 1ransfer for "ocal 2nstance page, where you can
identify which block classes :such as undo blocks, data blocks, and so on; are subGect to
intense global cache activity. 1his page displays the block classes that are being
transferred and the instances that are transferring most of the blocks.
Cache transfer indicates how many current and C& blocks for each block class were
received from remote instances, including how many transfers incurred a delay :busy; or
an une)pected longer delay :congested.;
1he /lobal Cache 4lock 1ransfer &ate section shows the total aggregated number of
blocks received by all instances in the cluster by way of an interconnect. Click one of the
legends to the right of the chart to go to the /C Current 4locks &eceived 4y 2nstance
page for that type of block.
Click an instance legend under the chart to go to the egment tatistics by 2nstance page
I where you can see which segments are causing cache contention.
1he /lobal Cache 4lock 1ransfers and Physical &eads :vs. "ogical &eads; section
shows the percentage of logical read operations that retrieved data from the buffer cache
of other instances by way of %irect -emory Access and from disk. 2t is essentially a
profile of how much work is performed in the local buffer cache rather than the portion of
remote references and physical reads :which both have higher latencies.;
Click one of the legends to the right of the chart to go to the J/lobal Cache 4lock
1ransfers and Physical &eads :vs. "ogical &eads; and KPhysical &eads vs. "ogical
&eads 4y 2nstanceJ pages. rom there, you can click an instance legend under the chart
to go to the Jegment tatistics by 2nstanceJ page, where you can see which segments
are causing cache contention.
&estion
Which summary chart on the Cluster Cache Coherency page shows the
percentage of logical read operations that retrieved data from the buffer cache of
other instances by way of %irect -emory Access and from disk7
Options'
$. /lobal Cache 4lock 1ransfer &ate
8. /lobal Cache 4lock Access "atency
6. /lobal Cache 4lock 1ransfers and Physical &eads
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
28/52
Ans"er
Option 1:Incorrect. This smmar' chart shows the total aggregated nmber o+
bloc(s recei&ed b' all instances in the clster b' wa' o+ an interconnect.
Option 2:Incorrect. This smmar' chart shows the total elapsed time, or latenc',
+or a bloc( reqest.
Option 3:Correct. This smmar' chart shows the percentage o+ logical read
operations that retrie&ed data +rom the b++er cache o+ other instances b' wa' o+
1irect /emor' %ccess and +rom dis(. It is essentiall' a pro+ile o+ how mch wor(
is per+ormed in the local b++er cache rather than the portion o+ remote re+erences
and ph'sical reads. 4oth o+ which ha&e higher latencies.
Correct ans"er(s)'
6. /lobal Cache 4lock 1ransfers and Physical &eads
1he Cluster 2nterconnects page is useful for monitoring the interconnect interfaces,
determining configuration issues, and identifying transfer rate'related issues, including
e)cess traffic. 1his page helps determine the load added by individual instances and
databases on the interconnect.
ometimes you can immediately identify interconnect delays that are due to applications
outside Oracle.
Graphic
Interconnects is one o+ the +i&e tabs o+ the Clster page. 5ther tabs are :ome,
Per+ormance, Targets, and Topolog'.
*ou can use the Cluster 2nterconnects page to perform the following tasks>
view all interfaces that are configured across the cluster
view statistics for the interfaces, such as absolute transfer rates and errors
determine the type of interfaces, such as private or public
determine whether the instance is using a public or private network
determine which database instance is currently using which interface, and
determine how much the instance is contributing to the transfer rate on the interface
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
29/52
1he Private 2nterconnect 1ransfer &ate :-4=ec; value shows a global view of the private
interconnect traffic, which is the estimated traffic on all the private networks in the cluster.
1he traffic is calculated as the summary of the input rate of all private interfaces that are
known to the cluster.
On the Cluster 2nterconnects page, you can access the +ardware %etails page, where
you obtain more information about all the network interfaces defined on each node of
your cluster.
imilarly, you can access the 1ransfer &ate metric page, which collects the internode
communication traffic of a cluster database instance. 1he critical and warning thresholds
of this metric are not set by default. *ou can set them according to the speed of your
cluster interconnects.
#ote
Yo can qer' the V$CLUSTER_INTERCONNECTS &iew +or in+ormation abot thepri&ate interconnect.
Use the %atabase "ocks page to determine if multiple instances are holding locks for the
same obGect. 1he page shows user locks, all database locks, or locks that are blocking
other users or applications. *ou can use this information to stop a session that is
unnecessarily locking an obGect.
1o access the %atabase "ocks page, you perform the following steps>
select :erformanceon the Cluster %atabase +ome page, and
Per+ormance is one o+ the eight a&ailable tabs on the Clster 1atabase :ome page.
click *atabase Loc8sin the Additional -onitoring "inks section at the bottom of the Performance
subpage
2. Snapshots of A0R in RAC
2n &AC environments, each AW& snapshot captures data from all active instances within
the cluster. 1he data for each snapshot set that is captured for all active instances is from
roughly the same point in time. 2n addition, the data for each instance is stored separately
and is identified with an instance identifier.
or e)ample, the "#((er_"#)6_5atstatistic shows the number of buffer waits on
each instance. 1he AW& does not store data that is aggregated from across the entire
cluster. 1hat is, the data is stored for each individual instance.
Graphic
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
30/52
The In-memor' statistics is connected to %W$ tables in !Y!%UX.
1he statistics snapshots generated by the AW& can be evaluated by producing reports
displaying summary data such as load and cluster profiles based on regular statistics and
wait events gathered on each instance.
1he AW& functions in a similar way as tatspack 1he difference is that the AW&
automatically collects and maintains performance statistics for problem detection and
self'tuning purposes. Unlike in tatspack, in the AW& there is only one )+a,)*ot_%
per snapshot across instances.
1he &AC'related statistics in an AW& report are organi(ed in different sections. A &AC
statistics section appears after the 1op 1imed vents containing
nmber of instances
1he number of instances open at the time of the begin snapshot and the end snapshot to
indicate instances Goined or left between the two snapshots.
Global Cache Loa! :rofile
/lobal Cache "oad Profile lists the number of blocks and messages that are sent and
received, as well as the number of fusion writes.
Global Cache Efficiency :ercentages
/lobal Cache fficiency Percentages indicates the percentage of buffer divided into
buffers received from the disk, local cache, and remote caches. 2deally, the percentage of
disk buffer access should be close to (ero.
GCS an! GES 0or8loa! Characteristics an!
/C and / Workload Characteristics give you an overview of the more important
numbers first. 4ecause the global en0ueue convert statistics have been consolidated with
the global en0ueue get statistics, the report prints only the average global en0ueue get
time. 1he round'trip times for C& and current block transfers follow, as well as the
individual sender'side statistics for C& and current blocks. 1he average log flush times are
computed by dividing the total log flush time by the number of actual log flushes. Also, the
report prints the percentage of blocks served that actually incurred a log flush.
GCS an! GES 5essaging Statistics
2n /C and / -essaging tatistics, the most important statistic is the average
message sent 0ueue time on ks)p, which gives a good indicator of how well the 2PCworks. Average numbers should be less than $ ms.
Additional &AC statistics are organi(ed in these sections>
the /lobal n0ueue tatistics section contains data e)tracted from V$GES_STATISTICS
the /lobal C& erved tats section contains data from V$CR_BLOC:_SERVER
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
31/52
the /lobal CU&&31 erved tats section contains data from V$CURRENT_BLOC:_SERVER,
and
the /lobal Cache 1ransfer tats section contains data from V$INSTANCE_CAC0E_TRANSFER
1he egment tatistics section also includes the /C 4uffer 4usy Waits, C& 4locks&eceived, and
CU& 4locks &eceived information for relevant segments.
&estion
Which &AC'related statistic section in an AW& report shows round'trip times for
C& and current block transfers7
Options'
$. /lobal Cache "oad Profile
8. /lobal Cache fficiency Percentages
6. /C and / -essaging tatistics
9. /C and / Workload Characteristics
Ans"er
Option 1:Incorrect. This section essentiall' lists the nmber o+ bloc(s and
messages that are sent and recei&ed, as well as the nmber o+ +sion writes.
Option 2:Incorrect. This section indicates the percentage o+ b++er di&ided intob++ers recei&ed +rom the dis(, local cache, and remote caches. Ideall', the
percentage o+ dis( b++er access shold be close to 7ero.
Option 3:Incorrect. The most important statistic in this section is the a;era!e
&e))a!e)e+t3#e#et&eo+7)
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
32/52
9. /C and / Workload Characteristics
Smmary
Oracle nterprise -anager %atabase Control and /rid Control are cluster'aware and
help to manage your cluster database. Using the Cluster %atabase Performance page,
you can identify the causes of performance issues, decide whether resources need to be
added or redistributed, tune !" plans, and resolve performance issues. 1his page
provides various charts that provide information on database performance. or e)ample,
the Average Active essions chart shows potential problems inside the database and the
1hroughput chart displays the usage of various database resources.
1he Cluster Cache Coherency page contains summary charts for cache coherency
metrics for the cluster. 1he Cluster 2nterconnects page monitors the interconnect
interfaces, determines configuration issues, and identifies transfer rate'related issues.
1he Private 2nterconnect 1ransfer &ate value contains a global view of the private
interconnect traffic. 1he %atabase "ocks page helps determine if multiple instances areholding locks for the same obGect.
2n &AC environments, each AW& snapshot captures data from all active instances within
the cluster for problem detection and self'tuning purposes. &AC'related statistics are
organi(ed in different sections in the AW& report.
Atomatic *atabase *iagnostic 5onitor for RAC
Learning Objectives
After completing this topic, you should be able to
recognize how to discover RAC performance problems using ADDM and nterprise Manager
recognize how to fi! performance problems in RAC
1. +in!ing RAC performance isses "ith A**5
Oracle %atabase $$goffers an e)tension to the set of functionality that increases the
database5s manageability by offering clusterwide analysis of performance.
A special mode of Automatic %atabase %iagnostic -onitor or A%%- analy(es a &AC
database cluster and reports on issues that affect the entire cluster as well as on those
that affect individual instances. 1his mode is called database A%%- :as opposed to
instance A%%-, which already e)isted with Oracle %atabase $Bg;.
%atabase A%%- for &AC is not simply a report of reports. &ather, it has independent
analysis that is appropriate for &AC.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
33/52
#ote
The 1atabase %11/ report is generated on the %W$ snapshot coordinator.
2n Oracle %atabase $$g, you can create a period analysis mode for A%%- that analy(es
the throughput performance for an entire cluster. When the advisor runs in this mode, it is
called database %11/. *ou can run the advisor for a single instance, which is e0uivalent
to Oracle %atabase $BgA%%- and is now called instance%11/.
%atabase A%%- has access to AW& data generated by all instances, thereby making the
analysis of global resources more accurate. 4oth database and instance A%%- run on
continuous time periods that can contain instance startup and shutdown.
2n the case of database A%%-, there may be several instances that are shut down or
started during the analysis period. *ou must, however, maintain the same database
version throughout the entire time period.
%atabase A%%- runs automatically after each snapshot is taken. 1he automatic instance
A%%- runs are the same as in Oracle %atabase $Bg. *ou can also perform analysis on a
subset of instances in the cluster. 1his is calledpartial anal'sis %11/.
2=O capacity finding :the 2=O system is overused; is a global finding because it concerns a
global resource affecting multiple instances. A local finding concerns a local resource or
issue that affects a single instance. or e)ample, a CPU'bound instance results in a local
finding about the CPU.
Although A%%- can be used during application development to test changes to either the
application, the database system, or the hosting machines, database A%%- is targeted at%4As.
&estion
Which mode of Automatic %atabase %iagnostic -onitor or A%%- analy(es a &AC
database cluster and reports on issues that affect the entire cluster as well as on
those that affect individual instances7
Options'
$. 2nstance A%%-
8. %atabase A%%-
6. Partial analysis A%%-
Ans"er
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
34/52
Option 1:Incorrect. Yo can rn the ad&isor +or a single instance, which is
eqi&alent to 5racle 1atabase @g %11/ and is now called instance %11/.
Option 2:Correct. % special mode o+ %11/ anal'7es a $%C database clster
and reports on isses that a++ect the entire clster as well as on those that a++ect
indi&idal instances. This mode is called database %11/ 8as opposed to instance
%11/, which alread' existed with 5racle 1atabase @g9. 1atabase %11/ +or
$%C is not simpl' a report o+ reports. $ather, it has independent anal'sis that is
appropriate +or $%C. The 1atabase %11/ report is generated on the %W$
snapshot coordinator.
Option 3:Incorrect. Yo can per+orm anal'sis on a sbset o+ instances in the
clster. This is called partial anal'sis %11/.
Correct ans"er(s)'
8. %atabase A%%-
&estion
Which are true statements about the database A%%- mode7
Options'
$. %atabase A%%- became available in Oracle %atabase $Bg
8. %atabase A%%- runs automatically after each snapshot is taken
6. %atabase A%%- has access to AW& data generated by all instances
9. %atabase A%%- performs analysis on a subset of instances in the cluster
Ans"er
Option 1:Incorrect. In 5racle 1atabase g, 'o can create a period anal'sis
mode +or %11/ that anal'7es the throghpt per+ormance +or an entire clster.
When the ad&isor rns in this mode, it is called database %11/.
Option 2:Correct. 1atabase %11/ rns atomaticall' a+ter each snapshot is
ta(en. The atomatic instance %11/ rns are the same as in 5racle 1atabase
@g.
Option 3:Correct. 1atabase %11/ has access to %W$ data generated b' all
instances, thereb' ma(ing the anal'sis o+ global resorces more accrate. 4oth
database and instance %11/ rn on continos time periods that can contain
instance startp and shtdown. In the case o+ database %11/, there ma' be
se&eral instances that are sht down or started dring the anal'sis period. Yo
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
35/52
mst, howe&er, maintain the same database &ersion throghot the entire time
period.
Option 4:Incorrect. Yo can per+orm anal'sis on a sbset o+ instances in the
clster. This is called partial anal'sis %11/.
Correct ans"er(s)'
8. %atabase A%%- runs automatically after each snapshot is taken
6. %atabase A%%- has access to AW& data generated by all instances
A%%- diagnoses the following in &AC>
latency problems in interconnect
congestion :identifying top instances affecting the entire cluster;
contention :buffer busy, top obGects, and more;
top consumers of multiblock re0uests
lost blocks
information about interconnect devices :warns about using PU4"2C interfaces;, and
throughput of devices I how much of it is used by Oracle and for what purpose :/C, locks, and
P!;
1he data sources of &AC that A%%- diagnoses are as follows>
wait events :especially Cluster class and buffer busy;
A+
instance cache transfer data, and
interconnect statistics :throughput, usage by component, pings;
A%%- analy(es the effects of &AC for both the entire database :%A1A4A analysismode; and for each instance :231A3C analysis mode.;
Oracle %atabase $$gnterprise -anager displays the A%%- analysis on the Cluster
%atabase home page.
On the Automatic %atabase %iagnostic -onitor or A%%- page, the %atabase Activity
chart plots the database activity during the A%%- analysis period. %atabase activity types
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
36/52
are defined in the legend based on its corresponding color in the chart. ach icon below
the chart represents a different A%%- task, which in turn corresponds to a pair of
individual Oracle %atabase snapshots saved in the Workload &epository.
2n the A%%- Performance Analysis section, the A%%- findings are listed in descending
order, from highest impact to least impact. or each finding, the Affected 2nstances
column displays the number :m of n; of instances affected. %rilling down further on the
findings takes you to the Performance indings %etail page.
Graphic
The table in the %11/ Per+ormance %nal'sis section is di&ided into +or colmns
named Impact 8;9, inding, %++ected Instances, and 5ccrrences 8last 3 hrs9.
1he 2nformational indings section lists the areas that do not have a performance impact
and are for informational purpose only.
1he Affected 2nstances chart shows how much each instance is impacted by these
findings. 1he display indicates the percentage impact for each instance.
Graphic
The %++ected Instances chart in the In+ormational indings section contains three
colmns < 6ame, Impact 8;9, and !tats.
. +i9ing RAC performance problems
1his demonstration shows you how to discover performance problems in your &AC
environment. 2n this demonstration, you identify performance issues using nterprise
-anagement performance pages as well as A%%- and you fi) those issues in three
different steps. At each step, you generate the same workload to make sure you are
making progress in your resolution.
*ou first generate a workload that uses a bad &AC application design. Using %atabase
Control, and connected as user *, navigate to the Performance page of your Cluster
%atabase.
1he steps to navigate to the Performance page are as follows>
click the :erformancetab on the Cluster %atabase +ome page and
on the Cluster %atabase Performance page, make sure Real 3ime' 1; Secon! Refreshis
selected from the #iew %ata drop'down list
Use P"=!" to create a new AW& snapshot.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
37/52
Co!e
6=>at ?*o&e?orale?+o%e+(o @ )e% -+ 11,>
=>at ?*o&e?orale?+o%e+(o @ )e% -+ 44,>
DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%
)?ora_%"5_??>
I1NAME=$DBNAMEH1H
I4NAME=$DBNAMEH4H
e
on the Performance page, click the *atabase Loc8slink in the Additional -onitoring "inks
section of the page
on the %atabase "ocks page, make sure that %loc8ing Loc8sis selected from the #iew drop'
down list, and
The 1atabase #oc(s page has a table with se&eral colmns. !ome o+ the colmns inclde !elect,
Username, !essions 4loc(ed, Instance 6ame, !ession I1, !erial 6mber, Process I1, !"# :ash
=ale, #oc( T'pe, /ode :eld, /ode $eqested, 5bBect T'pe, 5bBect 5wner, 5bBect 6ame, and
$5WI1. %ll colmns except the Username colmn are blan(. The Username colmn contains the
text < 6o loc(s o+ this t'pe crrentl' exist.
in this case no blocking locks currently e)ist in the database
While the scripts are still e)ecuting, look at the Average Active essions graph. 1hen drill
down to the Cluster wait class for the first node. *ou can see the following>
Graphic
The %&erage %cti&e !essions graph has its Y-axis contains &ales sch as @.@,
@.3, and @.F and its X-axis contains &ales sch as 3, @, and ).
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
38/52
"ook at the Average Active essions graph. 1hen drill down to the Other wait class.
Click the Clster *atabase locator link at the top of the page to return to the Cluster %atabase
Performance page.
rom there you can now see the Average Active essions graph. -ake sure that the #iew %atafield is set to Real 3ime' 1; Secon! Refresh.
Using the 1hroughput tabbed page graph underneath the Average Active essions graph, you
can see the transaction rate per second.
2n the Average Active essions graph, click the Otherlink on the right. 1his takes you to the
Active essions 4y 2nstance> Other page.
On the Active essions 4y 2nstance> Other page, you can see the number of active sessions for
the Other wait class.
The %cti&e !essions 4' Instance 5ther page has the !mmar' Chart graph that is similar to the
%&erage %cti&e !essions graph.
After the workload finishes, use P"=!" to create a new AW& snapshot.
Co!e
6=>at ?*o&e?orale?+o%e+(o @ )e% -+ 11,>
=>at ?*o&e?orale?+o%e+(o @ )e% -+ 44,>
DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;
allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%)?ora_%"5_??>
I1NAME=$DBNAMEH1H
I4NAME=$DBNAMEH4H
e
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
39/52
On the Cluster %atabase +ome page, click the A!visor Centrallink.
The %d&isor Central lin( is located in the $elated #in(s section o+ the page.
On the Advisor Central page, make sure that the Advisory 1ype field is set to All 3ypes, and that
the Advisor &uns field is set to Last Rn. Click Go.
The %d&isor Central page also contains the Tas( 6ame and !tats +ields.
2n the &esults table, select the latest A**5run corresponding to 2nstance All. 1hen click
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
40/52
DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;
allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%
)?ora_%"5_??>
I1NAME=$DBNAMEH1H
I4NAME=$DBNAMEH4H
e
"ook at the Average Active essions graph. %rill down to the System I-Owait class to view
ystem 2=O wait class details.
Click the Clster *atabaselocator link at the top of the page to return to the Cluster %atabase
Performance page.
On the 2=O tabbed page underneath the Average Active essions graph, you can see graphs
based on 2=O functions.
This example shows the IE5 /egab'tes per !econd b' IE5 nction graph.
Click the LG0Rlink of the 2=O &e0uests per econd by 2=O unction graph to see specific 2=O
&e0uests per econd 4y 2nstance or 2=O unction> "/W& graphs.
Click the IOdrop down list and select %ffer Cache Rea!sto view the graphs specific to
4uffer Cache &eads on the 2=O -egabytes per econd 4y 2nstance or 2=O unction>
4uffer Cache &eads page.
*ou can drill down on any of the functions for the 2=O -egabytes per econd by 2=O
unction and 2=O &e0uests per econd by 2=O unction pages, which are e)amples of Gust
some of the drill down functionality available in the nterprise -anager performance
pages.
After the workload finishes, use P"=!" to create a new AW& snapshot.
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
41/52
Co!e
6=>at ?*o&e?orale?+o%e+(o @ )e% -+ 11,>
=>at ?*o&e?orale?+o%e+(o @ )e% -+ 44,>
DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%
)?ora_%"5_??>
I1NAME=$DBNAMEH1H
I4NAME=$DBNAMEH4H
e
On the Cluster %atabase +ome page, click the A!visor Centrallink.
The %d&isor Central lin( is located in the $elated #in(s section o+ the Clster 1atabase :ome
page.
On the Advisor Central page, make sure that the Advisory 1ype field is set to All 3ypesand that
the Advisor &uns field is set to Last Rn. Click Go.
The %d&isor' T'pe and %d&isor $ns +ields are located in the !earch section o+ the %d&isor Central
page.
2n the &esults table, select the latest A**5 run corresponding to 2nstance All. 1hen click
-
8/10/2019 Oracle Database 11g RAC Performance Tuning
42/52
%atabase A%%- for &AC reports on issues that
affect the entire cluster and individual instances. When an advisor is run for a single
instance, it is called instance A%%-. %atabase A%%- runs automatically when taking
AW& snapshots. When analysis is performed on a subset of instances in the cluster, it is
called partial analysis A%%-. A%%- diagnoses data sources for &AC, which includes
Wait events and A+.
Performance problems in a &AC environment can be manually discovered by using the
nterprise -anager performance pages as well as A%%-.
Atomatic *atabase *iagnostic 5onitor for RAC
Learning Objectives
After completing this topic, you should be able to
recognize how to discover RAC performance problems usi