Wait Events in Rac

8/11/2019 Wait Events in Rac

1/25

Wait Events in RAC

Session 362

Arup Nanda

Longtime Oracle DBAarup.blogspot.com


2/25

What is This?

RAC Performance Tuning

I teach a coursePerformance Tuning in RAC

Wait events are useful for understanding the bottlenecks

All single instance wait events are applicable to RAC

RAC has some special cases This session talks about those RAC-specific wait events

This is just a subset of the events; not a comprehensive

list

2Wait Events in RAC


3/25

Whats a Wait?

A process in Oracle can only be in three states

Doing something Useful (consuming CPU) ..U

Idle, waiting for some work to be assigned .. I

Waiting for something, e.g. .. W

a block from disk

a lock

a latch (could be waiting on CPU)

Response time = U + I + W

We must accurately measure each component time

before we decide what and how to tune

3Wait Events in RAC


4/25

Node 1

Cluster Coordination

RAC for Beginners4

Buffer Cache Buffer Cache

DBWR DBWR

LMS LMS

SCN1

DBWR must get a lock on the database block before writing to

the disk. This is called a Block Lock.

Node 2

Database

SCN2

Checkpoint!

Checkpoint!


5/25

Cache Fusion

5Wait Events in RAC

Instance 1 Instance 2

5

session1 session2

5

Has

modifiedit

Wants to

modify it

Will get it via

interconnect


6/25

Checking for Buffers

How exactly is this checkperformed?

By checking for a lock on the

block

The request comes to the GrantQueue of the block

Someone checks that no other

instance has any lock

Instance 1 can read from the

disk

i.e. Instance 1 is grantedthe

lock

6

Block

SID1

SID2

SID3

Grant

Queue

Convert

Queue

SID5

SID6

SID7

Wait Events in RAC


7/25

Master Instance

Only one instance holds the grant and

convert queues of a specific block This instance is called Master Instance

of that block

Master instance varies for each block

The memory structure that shows themaster instance of a buffer is calledGlobal Resource Directory (GRD)

That is replicated across all instances

The requesting instance must check theGRD to find the master instance

Then make a request to the masterinstance for the lock

7Wait Events in RAC

Block

SID1

SID2

SID3

Grant

Queue

Convert

Queue

SID5

SID6

SID7


8/25

Scenario 1

Session connected to Instance 1 wants to select a blockfrom the table

Activities by Instance 1

1. Check its own buffer cache to see if the block exists1. If it is found, can it just use it?

2. If it not found, can it select from the disk?

2. If not, then check the other instances

How will it know which copy of the block is the bestsource?

8Wait Events in RAC

Instance 1 Instance 2Session


9/25

Node 2Node 1

Cache Fusion

9Wait Events in RAC

Buffer Cache Buffer Cache

SMON SMON

LMS LMS

When node 2 wants a buffer, it sends a message to the

other instance. The message is sent to the LMS (Lock

Management Server) of the other instance. LMS then

sends the buffer to the other instance. LMS is also called

Global Cache Server (GCS).

message

buffer


10/25

Grant Scenario 2

1. Check its buffer cache to see if the block exists

2. And the buffer is found. Can Instance1 use it?

Not really. The buffer may be old; it may have been changed

3. LMS of node1 sends message to master of the buffer

3. Master checks the GES and doesnt sees any lock

4. Instance 1 is grantedthe global block lock

5. No buffer actually gets transferred

10Wait Events in RAC


11/25

Grant Scenario 3

Instance 1 is the master

Then it doesnt have to make a request for the grant

In summary, here are the possible scenarios when

Instance1 requests a buffer

Instance1 is the master; so no more processing is

required

No one has the lock on the buffer, the grant is made

by the master immediately

Another instance has the buffer in an incompatible

mode. It has to be changed.



12/25

Buffer States and Locks

Buffers can be gotten in two states

Currentwhen the intention is to modify Shared Currentmost recent copy. One copy per

instance. Same as disk

Exclusive Currentonly one copy in the entire

cluster. No shared current present

CRwhen the intention is to only select

Locks facilitate the state enforcement

XCUR for Exclusive Current

SCUR for Shared Current

No locking for CR



13/25

Placeholder Event

When the buffer is first requested, the session does not

know which of the three paths it will go on to

Therefore it is assigned aplaceholderevent

This event is known as gc cr block request (for Consistent

Read requests)

If the request is made for the buffer in Current mode, the

event is gc current block request

When one of the three options is chosen, the appropriate

event replaces the placeholder event



14/25

Grant Event

If the session merely requests a grant from the master:

It waits with the gc cr|current grant 2-way event

gc cr grant 2-way, for requests for buffers in Consistent

Read mode

gc current grant 2-way, for requests for buffers in

Currentmode



15/25

15

gc current|cr grant 2-way

Instance 1

Instance 2

Session

Database

LMSLMS

gc current block requestgc current grant 2-way

GRD GES

db file scattered read

Wait Events in RAC

Block

SID1

SID2

SID3

Grant

Queue

Conver

t Queue

SID5

SID6

SID7

request

granted


16/25

Block Event

After the request is made

and assuming that the buffer is not in the local cache

the buffer may be found in another instance

Requestor requests the buffer from the holding

instance

Possibilities:

Holder is also the master of that buffer

Holder is not the master

If the holder is the master, the requesting session waitswith the event gc cr|current block 2-way



17/25

17

gc current|cr block 2-way

Wait Events in RAC

Instance 1

Instance 2

Session

Database

LMSLMS

gc current block requestgc current block 2-way

GRD GES

Block

SID1

SID2

SID3

Grant

Queue

Conver

t Queue

SID5

SID6

SID7


18/25

18

gc current|cr block 3-way

Instance 1

Instance 2

Instance 3

Session

Requestor

Master

Holder

gc current block 3-way

Wait Events in RAC

Block

SID1

SID2

SID3

GrantQueue

Convert Queue

SID5

SID6

SID7


19/25

Grant vs- Get Events

Action Wait Event is

Block is requested by a session Placeholder eventgc cr|current block

request

Buffer lock is requested from the master gc cr|current grant 2-way

Buffer lock is granted, get from the disk db file scattered|sequential read

Buffer is requested from the holder which

is the same as the master

gc cr|current block 2-way

Buffer is requested from the holder which

is not the master

gc cr|current block 3-way

19

1. There is no 3-way grant event, since the request is made to the master

2. There is no 4-way block event; since there will a maximum of 2 hops:

requestormasterholder

Wait Events in RAC


20/25

Interpretation

Scenario Interpretation Tuning Implication

The grant waits are very high

compared to block waits

The requested blocks are in

the current instance; but the

master is another instance

Manually remaster the

object?

The 2-way block waits are

very high compared to 3-way

The cache fusion is taking

place as expected, and most

of the blocks are held and

mastered at the same

instance. But most of the

blocks are not found in the

local instance.

Faster interconnect

Shorten the run queue

The 3-way block waits arehigh compared to 2-way

Cache fusion is taking place;but the master and the

holder are usually different.

Manually remaster theobject

Application partitioning



21/25

gc current/cr block lost

Lost blocks while being transferred to

the remote instance [in theinterconnect]

Cause: may not be the network itself

Cause #1 Network

Network traffic drops the packets

Confirm from the ifconfig -a output If packets are dropping, this couldbe a cause

Why?

Bad network configuration

CPU used for networkprocessing

Cause #2 CPU

LMS process is CPU starved


Instance 1

Instance 2

Session

Database

LMSLMS

gc current block requestgc current block 2-way

GRD GES

Block

S

I

D

1S

I

D

2S

I

D

3

G

r

a

n

t

Q

ue

u

e

C

o

n

v

e

r

t

Qu

e

u

eS

I

D

5S

I

D

6S

I

D

7


22/25

gc current/cr block busy

What it Means

Session wants a block from the remote instance

Remote instance delays preparing the block to

fulfill the request [CR or current] read

Cause:

Local delay on the remote instance

Most likely: an I/O bottleneck on the remote

instance

block is being accessed by some session which is CPU

starved

LGWR has not written a buffer to redo yet

Less likely: CPU starvation



23/25

gc current/cr block congested

Meaning

The instance has requested block from remote instance.

Remote instance has prepared the block and shipped it

but it has not reached the requesting instance within 1 ms.

What could be causenetwork bottleneck?

Not necessarily. Causes:

Long run queues, causing the LMS process to be

delayed in processing the incoming block

Heavy paging due to memory deficiency. This causes

the blocks to be paged in before being processed.


ll h


24/25

Putting it All Together

Every Oracle process is either

Doing some productive work Waiting for some work to be given (idle)

Waiting for some resource

Understand the reason for the wait

Devise a plan accordingly RAC related wait events are manifestations of these

issues mostly

Caused by

Network issues LMS being overloaded

Blocks busy



25/25

Thank You!Session 362

More Information:

Blog: arup.blogspot.com

100 Things You Probably Didn't Know About OracleDatabase http://bit.ly/evr05e

Twitter: arupnanda

http://bit.ly/evr05ehttp://bit.ly/evr05e

Wait Events in Rac

Documents

Transcript of Wait Events in Rac