1 © 2008 Julian Dyke juliandyke.com Advanced Queuing Internals Julian Dyke Independent Consultant...

1 © 2008 Julian Dyke juliandyke.com

Advanced QueuingInternals

Julian Dyke

Independent Consultant

Web Version - November 2008


Agenda

Introduction Single Consumer Queues Multiple Consumer Queues

Recipients Subscribers

Exception Queues Array Payloads Buffered Messages

Spilled Messages Performance


IntroductionAdvanced Queuing Advanced Queuing

Introduced in Oracle 8.0 Extended and enhanced in most subsequent versions Supports Oracle Streams in Oracle 9.2 and above Supports buffered messages in Oracle 10.2 and above

Allows messages to be enqueued and dequeued from queues that are managed by the database

Each queue is associated with a queue table Properties of queue table specify behaviour of associated queues

Each queue has a payload which can be: RAW - only messages of type RAW can be enqueued Object type- only messages of the specified type can be enqueued ANYDATA - messages with any object type can be enqueued


IntroductionAdvanced Queuing By default messages are dequeued in the order they are enqueued

Default behaviour can be overridden in several ways

Messages can be persistent or buffered Persistent messages

Stored in queue table Survive an instance restart

Buffered messages Stored in SGA Can be spilled to queue table Lost during instance restart

Messages can be immediate or on-commit Immediate messages are committed immediately when they are

enqueued/dequeued On-commit messages are committed with the enqueuing transaction Buffered messages can only be immediate.


IntroductionAdvanced Queuing Queue tables can be created for single or multiple consumers

Messages in single consumer queue tables can only be dequeued once Messages in multiple consumer queue tables can be dequeued multiple

times by multiple consumers

Multiple consumer queue tables can be associated with Multiple recipients Multiple subcribers

Subscribers can: Specify rules to control which messages they dequene Specify transformations to be performed against dequeued data


Advanced QueuingIntroduction Sessions can listen for messages on multiple queues

Session is notified when a message arrives on any of the target queues

Queue messages can be propagated from one queue to another In the same database In different databases Propagation can be immediate or at specified intervals

Transformation converts payload from one object type to another Queue data can be transformed when messages are:

Enqueued Propagated Dequeued

Messages must be transformed using DBMS_TRANSFORM API


IntroductionPayloads Queue payloads can be

RAW Abstract data types ANYDATA

Abstract data types Maximum number of attributes is limited to 900 For example:

CREATE TYPE type1 AS OBJECT(

c1 NUMBER,c2 NUMBER,c3 NUMBER

);/


Single Consumer QueuesQueue Tables Every queue must be associated with a queue table Queue table defines properties of queue Reported in DBA_QUEUE_TABLES Managed using DBMS_AQADM For example:

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');

By default single consumer queues will be created Each queue table can contain multiple queues

Queues inherit properties of queue table Each queue table block contains blocks for one queue


Single Consumer QueuesQueue Table Columns (RAW Payload)


Col# Intcol# Column Name Data Type

1 1 Q_NAME VARCHAR2(30)

2 2 MSG_ID RAW(16)

3 3 CORRID VARCHAR2(128)

4 4 PRIORITY NUMBER

5 5 STATE NUMBER

6 6 DELAY TIMESTAMP(6)

7 7 EXPIRATION NUMBER

8 8 TIME_MANAGER_INFO TIMESTAMP(6)

9 9 LOCAL_ORDER_NO NUMBER

10 10 CHAIN_NO NUMBER

11 11 CSCN NUMBER

12 12 DSCN NUMBER

13 13 ENQ_TIME TIMESTAMP(6)

14 14 ENQ_UID VARCHAR2(30)

15 15 ENQ_TID VARCHAR2(30)


16 16 DEQ_TIME TIMESTAMP(6)

17 17 DEQ_UID VARCHAR2(30)

18 18 DEQ_TID VARCHAR2(30)

19 19 RETRY_COUNT NUMBER

20 20 EXCEPTION_QSCHEMA VARCHAR2(30)

21 21 EXCEPTION_QUEUE VARCHAR2(30)

22 22 STEP_NO NUMBER

23 23 RECIPIENT_KEY NUMBER

24 24 DEQUEUE_MSG_ID RAW(16)

25 25 SENDER_NAME VARCHAR2(30)

26 26 SENDER_ADDRESS VARCHAR2(1024)

27 27 SENDER_PROTOCOL NUMBER

28 28 USER_DATA BLOB

29 29 USER_PROP SYS.ANYDATA

Queue table QT1 contains the following columns:

10

© 2008 Julian Dyke juliandyke.com

Single Consumer QueuesQueue Table Columns (Object Payload)



1 1 Q_NAME VARCHAR2(30)

2 2 MSG_ID RAW(16)

3 3 CORRID VARCHAR2(128)

4 4 PRIORITY NUMBER

5 5 STATE NUMBER

6 6 DELAY TIMESTAMP(6)

7 7 EXPIRATION NUMBER

8 8 TIME_MANAGER_INFO TIMESTAMP(6)

9 9 LOCAL_ORDER_NO NUMBER

10 10 CHAIN_NO NUMBER

11 11 CSCN NUMBER

12 12 DSCN NUMBER

13 13 ENQ_TIME TIMESTAMP(6)

14 14 ENQ_UID VARCHAR2(30)

15 15 ENQ_TID VARCHAR2(30)

16 16 DEQ_TIME TIMESTAMP(6)


17 17 DEQ_UID VARCHAR2(30)

18 18 DEQ_TID VARCHAR2(30)

19 19 RETRY_COUNT NUMBER

20 20 EXCEPTION_QSCHEMA VARCHAR2(30)

21 21 EXCEPTION_QUEUE VARCHAR2(30)

22 22 STEP_NO NUMBER

23 23 RECIPIENT_KEY NUMBER

24 24 DEQUEUE_MSG_ID RAW(16)

25 25 SENDER_NAME VARCHAR2(30)

26 26 SENDER_ADDRESS VARCHAR2(1024)

27 27 SENDER_PROTOCOL NUMBER

28 28 USER_DATA TYPE1

28 29 SYS_NC00029$ NUMBER



29 32 USER_PROP SYS.ANYDATA

Queue table QT1 contains the following columns:


Single Consumer QueuesDatabase Objects (RAW payload)


The following objects will be created (object IDs and constraint IDs will vary):

Object ID

Object Name Object Type

70581 QT1 TABLE

70582 SYS_LOB0000070581C00028$$

LOB

70583 SYS_IL0000070581C00028$$ LOB INDEX

70584 SYS_LOB0000070581C00029$$ LOB

70585 SYS_IL0000070581C00029$$ LOB INDEX

70586 SYS_C009433 INDEX

70587 AQ$_QT1_T INDEX

70588 AQ$_QT1_I INDEX

70589 QT70581_BUFFER VIEW

70590 AQ$QT1 VIEW

70591 AQ$_QT1_F VIEW

70592 AQ$_QT1_E QUEUE LOB columns are used for USER_DATA and USER_PROP columns

12


Single Consumer QueuesDatabase Objects (Object Payload)



Object ID

Object Name Object Type

70581 QT1 TABLE

70582 SYS_LOB0000070581C00032$$

LOB

70583 SYS_IL0000070581C00032$$ LOB INDEX

70584 SYS_C009433 INDEX

70585 AQ$_QT1_T INDEX

70586 AQ$_QT1_I INDEX


70589 AQ$QT1 VIEW


70591 AQ$_QT1_E QUEUE LOB column is used for USER_PROP column

13


Single Consumer QueuesIndex Columns


The following indexes will be created by default (constraint IDs will vary):

Index Name Column #

Column Name

SYS_C009436

1 MSGID

AQ$_QT1_I 1 Q_NAME

2 STATE

3 ENQ_TIME

4 STEP_NO

5 CHAIN_NO

6 LOCAL_ORDER_NO

AQ$_QT1_T 1 TIME_MANAGER_INFO

14


Single Consumer QueuesSort Lists

The columns indexed by AQ$_QTI are determined by the SORT_LIST parameter.

Possible values are enq_time (default) priority priority,enq_time enq_time,priority

Must be defined when queue table is created Cannot be subsequently altered

Column #

enq_time priority priority,enq_time enq_time,priority

1 Q_NAME Q_NAME Q_NAME Q_NAME

2 STATE STATE STATE STATE

3 ENQ_TIME PRIORITY PRIORITY ENQ_TIME

4 STEP_NO CHAIN_NO ENQ_TIME STEP_NO

5 CHAIN_NO LOCAL_ORDER_NO STEP_NO PRIORITY

6 LOCAL_ORDER_NO

CHAIN_NO CHAIN_NO

7 LOCAL_ORDER_NO LOCAL_ORDER_NO

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1',SORT_LIST=>"priority,enq_time");

15


Single Consumer QueuesViews Two views are created for each queue table

For example for QT1 (object ID =70581):

QT<object_id>_BUFFER e.g. QT70581_BUFFER based on X$BUFFER2

AQ$_<queue_table_name>_F e,g. AQ$_QT1_F based on QT1 and ALL_DEQUEUE_QUEUES

16


Single Consumer QueuesViews Two views are created for each queue table

For example for QT1 (object ID =70581) QT70581_BUFFER AQ$_QT1_F

SELECT "ADDR", "INDX", "INST_ID", "OBJNO", "QUEUE_ID", "MSGID", "CORRID",

"SEQUENCE_NUM", "MSG_NUM", "STATE", "PRIORITY", "EXPIRATION","ENQ_TIME", "ENQ_UID", "ENQ_USER_NAME", "RETRY_COUNT","SENDER_NAME", "SENDER_ADDRESS", "SENDER_PROTOCOL","DEQUEUE_MSGID", "SRCSEQUENCE_NUM", "SUBSCRIBER_ID","EXCEPTIONQ_SCHEMA", "EXCEPTIONQ_NAME"

FROM X$BUFFER2 WHERE objno = 70581;

QT70581_BUFFER is defined as follows:

17


Single Consumer QueuesViews AQ$_QT1_F is defined as follows:

SELECT qt.q_name Q_NAME, qt.rowid ROW_ID, qt.msgid MSGID, qt.corrid CORRID,qt.priority PRIORITY, qt.state STATE, qt.delay DELAY, qt.expiration EXPIRATION,qt.enq_time ENQ_TIME, qt.enq_uid ENQ_UID, qt.enq_tid ENQ_TID,qt.deq_time DEQ_TIME, qt.deq_uid DEQ_UID, qt.deq_tid DEQ_TID,qt.retry_count RETRY_COUNT, qt.exception_qschema EXCEPTION_QSCHEMA,qt.exception_queue EXCEPTION_QUEUE, qt.cscn CSCN, qt.dscn DSCN,qt.chain_no CHAIN_NO, qt.local_order_no LOCAL_ORDER_NO,qt.time_manager_info TIME_MANAGER_INFO, qt.step_no STEP_NO,qt.user_data USER_DATA , qt.sender_name SENDER_NAME,qt.sender_address SENDER_ADDRESS, qt.sender_protocol SENDER_PROTOCOL,qt.dequeue_msgid DEQUEUE_MSGID, 'PERSISTENT' DELIVERY_MODE,0 SEQUENCE_NUM, 0 MSG_NUM, qo.qid QUEUE_ID,qt.user_prop USER_PROP

FROM "QT1" qt, ALL_DEQUEUE_QUEUES qo

WHERE qt.q_name = qo.name AND qo.owner = 'US01' WITH READ ONLY;

18


Single Consumer QueuesQueues Every queue must be associated with a queue table

Queue table must exist before queue can be created

Every queue has a type which can be: NORMAL (default) EXCEPTION NON PERSISTENT

Non persistent queues are deprecated in Oracle 10.2 Use buffered messages instead

DBMS_AQADM.CREATE_QUEUE (queue_name => 'Q1',queue_table => 'QT1');

This statement might create the following object:

Object ID Object Name Object Type

70793 Q1 QUEUE

19


Single Consumer QueuesEnqueue The following code enqueues a message of TYPE1 on a single consumer

queue:

DECLAREmessage TYPE1;msgprop dbms_aq.message_properties_t;enqopt dbms_aq.enqueue_options_t;enq_msgid RAW(16);

BEGINmessage := new TYPE1 (10001,20001,30001);msgprop.expiration :=DBMS_AQ.NEVER

dbms_aq.enqueue(

queue_name => 'Q1',enqueue_options => enqopt,message_properties => msgprop,payload => message,msgid => enq_msgid

);END;

20


Single Consumer QueuesEnqueue The enqueue processes executes the following recursive statement:

insert into "US01"."QT1" (q_name, msgid, corrid, priority, state, delay, expiration,time_manager_info, local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid,retry_count, exception_qschema, exception_queue, recipient_key, dequeue_msgid,user_data, sender_name, sender_address, sender_protocol, user_prop, cscn, dscn)values (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, 0, :15,:16, :17, :18, :19, :20, :21,:22, :23, :24, :25)

In Oracle 11.1 this statement uses the LOAD TABLE CONVENTIONAL operation

STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL (cr=1 pr=5 pw=5 time=0 us)')

For this statement the following objects are modified


70581 QT1 TABLE

70586 SYS_C009433 INDEX

70588 AQ$_QT1_1 INDEX

21


Single Consumer QueuesDequeue The following code dequeues a message of TYPE1 from a single consumer

queue:

DECLAREmessage TYPE1;msgprop dbms_aq.message_properties_t;deqopt dbms_aq.dequeue_options_t;deq_msgid RAW(16);

BEGINdbms_aq.dequeue(

queue_name => 'Q1',dequeue_options => deqopt,message_properties => msgprop,payload => message,msgid => deq_msgid

);END;

22


Single Consumer QueuesDequeue The dequeue processes executes the following recursive statement:

select /*+ FIRST_ROWS(1) */ tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay, tab.expiration, tab.retry_count, tab.exception_qschema, tab.exception_queue, tab.chain_no, tab.local_order_no, tab.enq_time, tab.time_manager_info, tab.state, tab.enq_tid, tab.step_no, tab.sender_name, tab.sender_address, tab.sender_protocol, tab.dequeue_msgid, tab.user_prop, tab.user_data from "US01"."QT1" tab where q_name = :1 and (state = :2 ) order by q_name, state, enq_time, step_no, chain_no, local_order_no for update skip locked

The statement selects all rows in the queue specified by :1 with a state of :2 The FIRST_ROWS(1) hint is used to optimize the plan The statement locks any rows to be deleted

This will generate undo/redo The statement uses the FOR UPDATE SKIP LOCKED clause to skip any rows

still locked by ongoing transactions

23


Single Consumer QueuesDequeue Execution plan for SELECT FOR UPDATE statement is:

STAT #3 id=1 cnt=1 pid=0 pos=1 obj=0 op='FOR UPDATE (cr=7 pr=2 pw=2 time=0 us)'STAT #3 id=2 cnt=1 pid=1 pos=1 obj=0 op='SORT ORDER BY (cr=7 pr=0 pw=0 time=0 us cost=4 size=2759 card=1)'STAT #3 id=3 cnt=1 pid=2 pos=1 obj=70581 op='TABLE ACCESS FULL QT1 (cr=7 pr=0 pw=0 time=0 us cost=3 size=2759 card=1)'

As queue grows, object statistics must be gathered to ensure AQ$_QT1_I index is used prevent full table scans on QT1

Rows identified by SELECT FOR UPDATE are deleted using:

delete /*+ CACHE_CB("QT9") */ from "US01"."QT1" where rowid = :1

Execution plan for DELETE statement is:

STAT #7 id=1 cnt=1 pid=0 pos=1 obj=0 op='DELETE QT1 (cr=1 pr=2 pw=2 time=0 us)'STAT #7 id=2 cnt=1 pid=1 pos=1 obj=70581 op='TABLE ACCESS BY USER ROWID QT1 (cr=1 pr=0)'

24


QueuesException Queues A default exception queue is created for each queue table

Exception messages will be moved to default queue unless a user-defined exception queue has been specified when the message is enqueued

For example to create a user-defined exception queue

DBMS_AQADM.CREATE_QUEUE (queue_name => 'Q1',queue_table => 'QT1');

DBMS_AQADM.CREATE_QUEUE (queue_name => 'Q1E',queue_table => 'QT1'queue_type => DBMS_AQADM.EXCEPTION_QUEUE);


Queue Type

70793 Q1 QUEUE NORMAL

70794 Q1E QUEUE EXCEPTION

To check number of rows in each queue:

SELECT q_name, COUNT(*)FROM qt1GROUP BY q_name;

25


QueuesException Queues Exceptions will be written to user-defined exception queue if it is specified

during enqueue operation

DECLAREl_payload TYPE1;l_msgprop dbms_aq.message_properties_t;l_enqopt dbms_aq.enqueue_options_t;l_enq_msgid RAW(16);

BEGINl_payload := new TYPE1 (10001,20001,30001);l_msgprop.expiration := 60; l_msgprop.exception_queue := 'Q1E';

dbms_aq.enqueue(

queue_name => 'Q1',enqueue_options => l_enqopt,message_properties => l_msgprop,payload => l_payload,msgid => l_enq_msgid

);END;

Message will expire after 60 seconds Expired message will be move to exception queue Q1E by queue monitor

26


Multiple Consumer QueuesIntroduction There are two ways to use multiple consumer queues

Multiple Recipients Multiple Subscribers

BEGIN dbms_aqadm.create_queue_table('QT3','TYPE1',multiple_consumers=>TRUE)

dbms_aqadm.create_queue ('Q3','QT3');

dbms_aqadm.start_queue ('Q3');END;/

The same queue definitions are used for both examples:

27


Multiple Consumer QueuesDatabase Objects

DBMS_AQADM.CREATE_QUEUE_TABLE('QT1','TYPE1',MULTIPLE_CONSUMERS=>TRUE);


70756 QT1 TABLE

70757 SYS_LOB0000070581C00032$$

LOB

70758 SYS_IL0000070581C00032$$ LOB INDEX

70759 SYS_C009457 INDEX

70760 AQ$_QT1_S TABLE

70761 SYS_C009460 INDEX

70762 AQ$_QT1_N SEQUENCE

70763 AQ$QT1_S VIEW

70764 AQ$_QT1_V EVAL CTXT

70765 AQ$_QT1_T TABLE

70766 SYS_IOT_TOP_70765 INDEX


70767 AQ$_QT1_H TABLE


70769 AQ$_QT1_G TABLE

70770 SYS_IOT_OVER_70769 TABLE


70772 AQ$_QT1_I TABLE



70775 AQ$QT1 VIEW


70777 AQ$_QT1_E QUEUE


28


Multiple Consumer QueuesTables AQ$_<queue_table_name>_T e.g AQ$_QT3_T

IOT used queue monitor to manage timed operations Single consumer queues use TIME_MANAGER_INFO column only

AQ$_<queue_table_name>_I IOT that maintains state for dequeue operations One row per message per recipient/subscriber

AQ$_<queue_table_name>_S Heap table containing information about subscribers

AQ$_<queue_table_name>_H IOT used to store dequeue history One row per message per recipient/subscriber

AQ$_<queue_table_name>_G IOT correlating messages to subscriber signatures

29


Multiple Consumer QueuesTables AQ$_<queue_table_name>_T

IOT used queue monitor to manage timed operations e.g. AQ$_QT3_T

Column Name

Data Type

NEXT_DATE TIMESTAMP

TXN_ID VARCHAR2(30)

MSGID RAW(16)

ACTION NUMBER

First 3 columns form primary key

Values for the ACTION column include: 0 - delay 1 - expiration 2 - delay

Single consumer queues use TIME_MANAGER_INFO column only

30


Multiple Consumer QueuesTables AQ$_<queue_table_name>_I

IOT that maintains state for dequeue operations

Column Name Data Type

SUBSCRIBER NUMBER

NAME VARCHAR2(30)

QUEUE# NUMBER

MSG_ENQ_TIME TIMESTAMP

MSG_STEP_NO NUMBER

MSG_CHAIN_NO NUMBER

MSG_LOCAL_ORDER_NO

NUMBER

MSG_ID RAW(16)

HINT ROWID

SPARE RAW(16)

First eight columns form primary key HINT and SPARE columns are stored in IOT overflow segment

31


Multiple Consumer QueuesTables AQ$_<queue_table_name>_S

Heap table containing information about subscribers


SUBSCRIBER_ID NUMBER

QUEUE_NAME VARCHAR2(30)

NAME VARCHAR2(30)

ADDRESS VARCHAR2(1024)

PROTOCOL NUMBER

SUBSCRIBER_TYPE NUMBER

RULE_NAME VARCHAR2(30)

TRANS_NAME VARCHAR2(65)

RULESET_NAME VARCHAR2(65)

NEGATIVE_RULESET_NAME VARCHAR2(65)

CREATION_TIME TIMESTAMP(6)

MODIFICATION_TIME TIMESTAMP(6)

DELETION_TIME TIMESTAMP(6)

SCN_AT_REMOVE NUMBER

32


Multiple Consumer QueuesTables AQ$_<queue_table_name>_H

IOT used to store dequeue history


MSGID RAW(16)

SUBSCRIBER# NUMBER

NAME VARCHAR2(30)

ADDRESS# NUMBER

DEQUEUE_TIME TIMESTAMP

TRANSACTION_ID VARCHAR2(30)

DEQUEUE_USER VARCHAR2(30)

PROPAGATED_MSGID RAW(16)

RETRY_COUNT NUMBER

HINT ROWID

SPARE RAW(16)

First four columns form primary key No IOT overflow segment

33


Multiple Consumer QueuesTables AQ$_<queue_table_name>_G

IOT correlating messages to subscriber signatures

Column Name

Data Type

NAME VARCHAR2(30)

ADDRESS# NUMBER

SIGN SYS.AQ$_SIG_PROP

DBS_SIGN SYS.AQ$_SIG_PROP

All columns form primary key

34


Multiple Consumer QueuesIndexes By default six indexes are created for each queue table. For example:

Index Name Index Type # Columns

Table Name

SYS_C009457 NORMAL 1 QT3

SYS_C009460 NORMAL 1 AQ$_QT3_S

SYS_IOT_TOP_70765

IOT 3 AQ$_QT3_T

SYS_IOT_TOP_70767

IOT 4 AQ$_QT3_H

SYS_IOT_TOP_70769

IOT 4 AQ$_QT3_G

SYS_IOT_TOP_70772

IOT 8 AQ$_QT3_IIndex Name Column

#Column Name

SYS_C009457

1 MSGID

SYS_C009460

1 SUBSCRIBER_ID

Index columns for NORMAL indexes are:

Index columns for IOT indexes are shown on previous slides

35


Multiple Consumer QueuesViews Three views are created for each queue table For example for QT3 (object ID = 70756)

QT70756_BUFFER AQ$_QT3_F AQ$QT3

<queue_object_Id>_BUFFER e.g QT70756_BUFFER Similar for single and multiple consumers

AQ$_<queue_table_name>_F e.g AQ$_QT3_F Similar for single and multiple consumers

AQ$<queue_table_name> views e.g. AQ$QT3 Based on:

Queue table (QT3) History IOT (AQ$_QT3_H) Subscriber table (AQ$_QT3_S)

36


Multiple Consumer QueuesViews AQ$<queue_table_name> views (AQ$QT3) are based on:

Queue table (QT3) History IOT (AQ$_QT3_H) Subscriber table (AQ$_QT3_S)

Abbreviated definition is as follows:

SELECT<column_list>

FROM "QT8" qt, "AQ$_QT8_H" h, "AQ$_QT8_S" s

WHERE qt.msgid = h.msgid AND ((h.subscriber# != 0 AND h.subscriber# = s.subscriber_id) OR (h.subscriber# = 0 AND h.address# = s.subscriber_id)) AND (qt.state != 7 OR qt.state != 9) WITH READ ONLY;

Best view to understand current state of queue for all subscribers

37


Multiple Consumer QueuesViews AQ$QT3 contains the following columns


QUEUE VARCHAR2(30)

MSG_ID RAW(16)

CORR_ID VARCHAR2(128)

MSG_PRIORITY NUMBER

MSG_STATE VARCHAR2(16)

DELAY DATE

DELAY_TIMESTAMP

TIMESTAMP(6)

EXPIRATION NUMBER

ENQ_TIME DATE

ENQ_TIMESTAMP TIMESTAMP(6)

ENQ_USER_ID VARCHAR2(30)

ENQ_TXN_ID VARCHAR2(30)

DEQ_TIME DATE

DEQ_TIMESTAMP TIMESTAMP(6)

DEQ_USER_ID VARCHAR2(30)

DEQ_TXN_ID VARCHAR2(30)


RETRY_COUNT NUMBER

EXCEPTION_QUEUE_OWNER VARCHAR2(30)

EXCEPTION_QUEUE VARCHAR2(30)

USER_DATA TYPE1

PROPAGATED_MSGID RAW(16)

SENDER_NAME VARCHAR2(30)

SENDER_ADDRESS VARCHAR2(1024)

SENDER_PROTOCOL NUMBER

ORIGINAL_MSGID RAW(16)

ORIGINAL_QUEUE_NAME VARCHAR2(30)

ORIGINAL_QUEUE_OWNER VARCHAR2(30)

EXPIRATION_REASON VARCHAR2(31)

CONSUMER_NAME VARCHAR2(30)

ADDRESS VARCHAR2(1024)

PROTOCOL NUMBER

38


Multiple Consumer QueuesRecipients The following code enqueues a message for three named recipients

DECLAREl_payload type1;l_msgprop dbms_aq.message_properties_t;l_enqopt dbms_aq.enqueue_options_t;l_enq_msgid RAW(16);l_recipient_list dbms_aq.aq$_recipient_list_t;

BEGINl_recipient_list(1) := sys.aq$_agent ('CONSUMER1',NULL,NULL);l_recipient_list(2) := sys.aq$_agent ('CONSUMER2',NULL,NULL);l_recipient_list(3) := sys.aq$_agent ('CONSUMER3',NULL,NULL);l_msgprop.recipient_list := l_recipient_list;l_msgprop.expiration := DBMS_AQ.NEVER;

l_payload := new TYPE1 (10001,20001,30001);

dbms_aq.enqueue(


);END;

39


Multiple Consumer QueuesRecipients A recipient list is constructed using AQ$_AGENT objects

In the example all recipients are in the local database

The enqueue operation performs the following actions: Inserts one row in the queue table (QT3) Inserts three rows in the queue status table (AQ$_QT3_I) Inserts three rows in the queue history table (AQ$_QT3_H)

40


Multiple Consumer QueuesRecipients The following code dequeues a message for one of the named recipients

DECLAREl_payload TYPE1;l_msgprop dbms_aq.message_properties_t;l_deqopt dbms_aq.dequeue_options_t;l_deq_msgid RAW(16);

BEGINl_deqopt.consumer_name := 'CONSUMER2';

dbms_aq.dequeue(

queue_name => 'Q3',dequeue_options => l_deqopt,message_properties => l_msgprop,payload => l_payload,msgid => l_deq_msgid

);END;

Notes A consumer name MUST be specified The message must have been enqueued specifically for that consumer

41


Multiple Consumer QueuesRecipients The dequeue operation performs the following actions

Deletes one row from the queue status IOT (AQ$_QT3_I) Updates the following columns in one row of the queue history IOT

(AQ$_QT3_H) DEQUEUE_TIME TRANSACTION_ID DEQUEUE_USER

Inserts one row into the queue timer table (AQ$_QT3_T)

The queue monitor (QMNC) process asynchronously checks the timer table (AQ$_QT3_T) for actions

If any actions are found these are sent to the queue monitor slaves (Q001, Q002 etc)

When last recipient has dequeued message, queue monitor slaves perform the following actions

Delete all rows for message in queue history table (AQ$_QT3_H) Delete row in queue table (QT3) for message

42


Multiple Consumer QueuesSubscribers Subscribers must exist for the queue before messages can be enqueued The following code creates two subscribers for queue Q3

DECLAREl_subscriber sys.aq$_agent;

BEGINl_subscriber := sys.aq$_agent ('SUBSCRIBER1',NULL,NULL);DBMS_AQADM.ADD_SUBSCRIBER(

queue_name => 'Q3',subscriber => l_subscriber

);

l_subscriber := sys.aq$_agent ('SUBSCRIBER2',NULL,NULL);

DBMS_AQADM.ADD_SUBSCRIBER(


);END;

Creating a subscriber inserts one row in the AQ$_QT3_S table

43


Multiple Consumer QueuesSubscribers The following code enqueues ten messages on Q3


BEGINFOR f IN 1..10LOOP

l_payload := new TYPE1 (10000 + f,20000 + f,30000 + f);

l_msgprop.expiration := DBMS_AQ.NEVER;

dbms_aq.enqueue(


);END LOOP;

END;

44


Multiple Consumer QueuesSubscribers The following code dequeues a message from Q3 for SUBSCRIBER1

SET SERVEROUTPUT ON


BEGINl_deqopt.consumer_name := 'SUBSCRIBER1';

dbms_aq.dequeue(


);

DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1));DBMS_OUTPUT.PUT_LINE ('C2 = '||TO_CHAR (l_payload.c2));DBMS_OUTPUT.PUT_LINE ('C3 = '||TO_CHAR (l_payload.c3));

END;

45


Multiple Consumer QueuesSubscribers Subscribers can subsequently be added and deleted dynamically

DECLARE l_subscriber sys.aq$_agent;

BEGINl_subscriber := sys.aq$_agent ('SUBSCRIBER3',NULL,NULL);

DBMS_AQADM.ADD_SUBSCRIBER(


);

DBMS_AQADM.REMOVE_SUBSCRIBER(


);END;

New subscribers will only be allowed to dequeue messages that have been enqueued after the subscriber was added

46


Array PayloadsIntroduction Payload of a queue can optionally be a VARRAY of object types For example:

CREATE OR REPLACE TYPE type2 AS VARRAY (10) OF type1;/

CREATE OR REPLACE TYPE type3 AS OBJECT (c1 type2);/

Queue table can be created with a TYPE3 payload It is not possible to create a queue table with a TYPE2 payload

For example:


DBMS_AQADM.CREATE_QUEUE ('Q3','QT3');

DBMS_AQADM.START_QUEUE ('Q3');

47


Array PayloadsEnqueue

DECLAREl_payload TYPE3;msgprop dbms_aq.message_properties_t;enqopt dbms_aq.enqueue_options_t;enq_msgid RAW(16);

BEGINl_payload := new TYPE3 (TYPE2 (

TYPE1 (10001,20001,30001),TYPE1 (10002,20002,30002),TYPE1 (10003,20003,30003),TYPE1 (10004,20004,30004)

));

msgprop.expiration := DBMS_AQ.NEVER;

dbms_aq.enqueue(

queue_name => 'Q4',enqueue_options => enqopt,message_properties => msgprop,payload => l_payload,msgid => enq_msgid

);END;

48


Array PayloadsDequeue

SET SERVEROUTPUT ON

DECLAREl_payload TYPE3;msgprop dbms_aq.message_properties_t;deqopt dbms_aq.dequeue_options_t;deq_msgid RAW(16);

BEGINdbms_aq.dequeue(

queue_name => 'Q4',dequeue_options => deqopt,message_properties => msgprop,payload => l_payload,msgid => deq_msgid

);

FOR i IN 1..message.c1.COUNTLOOP

DBMS_OUTPUT.PUT ('C1 = '||TO_CHAR (l_payload.c1(i).c1)||' ');DBMS_OUTPUT.PUT ('C2 = '||TO_CHAR (l_payload.c1(i).c2)||' ');DBMS_OUTPUT.PUT ('C3 = '||TO_CHAR (l_payload.c1(i).c3));DBMS_OUTPUT.NEW_LINE ();

END LOOP;END;

49


Buffered MessagesIntroduction In Oracle 10.2 and above messages can be buffered in the SGA

Messages will not be written to database immediately Messages are spillled to database if:

Number of messages exceeds threshold value Messages not dequeued within 10 minutes

Buffered messages Are much faster than persistent queues Do not guarantee reliability Cannot form part of a transaction Do not support (Oracle 11.1)

Message retention / delay Transaction grouping Array enqueue / dequeue Message export / import

50


Buffered MessagesIntroduction Buffering is specified at message level

Queues can contain both persistent and buffered messages Payload can be ADT, XML, ANYDATA or RAW Support for LOB payloads is restricted

BEGIN dbms_aqadm.create_queue_table ('QT1','TYPE1')

dbms_aqadm.create_queue ('Q1','QT1');

dbms_aqadm.start_queue ('Q1');END;

Note that all queue tables support buffered messages No additional attributes are specified for the queue or the queue table

The following definitions are used with the examples in this section

51


Buffered MessagesEnqueue The following code enqueues a buffered message


BEGINl_payload := new TYPE1 (10001,20001,30001);

l_msgprop.expiration := DBMS_AQ.NEVER;l_enqopt.visibility := DBMS_AQ.IMMEDIATE;l_enqopt.delivery_mode := DBMS_AQ.BUFFERED;

dbms_aq.enqueue(


);END;

52


Buffered MessagesDequeue The following code dequeues a buffered message:

SET SERVEROUTPUT ON


BEGINl_msgprop.expiration := DBMS_AQ.NEVER;l_deqopt.visibility := DBMS_AQ.IMMEDIATE;l_deqopt.delivery_mode := DBMS_AQ.BUFFERED;

dbms_aq.dequeue(


);

DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1));DBMS_OUTPUT.PUT_LINE ('C2 = '||TO_CHAR (l_payload.c2));DBMS_OUTPUT.PUT_LINE ('C3 = '||TO_CHAR (l_payload.c3));

END;

53


Buffered MessagesMemory Usage Memory is allocated from the Streams Pool The following table shows the amount of streams pool memory required to

enqueue 5101 messages with the TYPE1 payload:

Before

After

kodpaih3 0 10,324,448

kwqbsinfy:mpr 480 2,448,480

image handles 84 428,512

kwqbsinfy:bms 72 387,692

kggmem_fl_1 44 224,444

kggbt_alloc_block 2,072 88,060

Sender info 14,140 19,796

recov_kgqbtctx 12,288 16,384

kwqbcqini:spilledovermsgs 2,952 3,936

kwqbsinfy:bqg 1,236 1,648

recov_kggmctx 924 1,232

Before

After

recov_kgqmsub 336 504

kwqbsinfy:cco 332 332

kwqbsinfy:sta 208 312

spilled:kwqbl 216 288

fixed allocation callback 256 256

kgqmsub 144 216

deqtree_kgqmctx 136 192

substree_kgqmctx 120 160

time manager index 120 160

msgtree_kgqmctx 120 160

name_kgqmsub 32 48

54


Buffered MessagesDatabase Objects Additional database objects are created the first time a buffered message is

enqueued on a queue table This will cause elapsed time of first enqueue operation to be high

For example the following objects might be created


72638 AQ$_QT3_P TABLE

72639 SYS_LOB0000072638C00032$$

LOB

72640 SYS_IL0000072638C00032$$ LOB INDEX

72641 SYS_C0010003 INDEX The enqueuing session also creates a service for the queue For example SYS$US01.Q3.TEST where

US01 is the queue owner Q3 is the queue name TEST is the database name

55


Buffered MessagesDatabase Objects AQ$_<table_queue_name>_P contains the following columns


Q_NAME VARCHAR2(30)

MSGID RAW(16)

CORRID VARCHAR2(128)

PRIORITY NUMBER

STATE VARCHAR2(16)

DELAY DATE

EXPIRATION NUMBER

TIME_MANAGER_INFO

TIMESTAMP(6)

LOCAL_ORDER_NO NUMBER

CHAIN_NO NUMBER

CSCN NUMBER

DSCN NUMBER

ENQ_TIME DATE

ENQ_UID VARCHAR2(30)

ENQ_TID VARCHAR2(30)


DEQ_TIME DATE

DEQ_UID VARCHAR2(30)

DEQ_TID VARCHAR2(30)

RETRY_COUNT NUMBER

EXCEPTION_QSCHEMA VARCHAR2(30)

EXCEPTION_QUEUE VARCHAR2(30)

STEP_NO NUMBER

RECIPIENT_KEY NUMBER

DEQUEUE_MSGID RAW(16)

SENDER_NAME VARCHAR2(30)

SENDER_ADDRESS VARCHAR2(1024)

SENDER_PROTOCOL NUMBER

USER_DATA TYPE1

USER_PROP SYS.ANYDATA

56


Buffered MessagesDatabase Objects The AQ$_<queue_table_name>_P table has one primary key index on

Q_NAME MSGID

Two view definitions are also updated when the first buffered message is enqueued:

AQ$<queue_table_name> e.g. AQ$QT3 reports all messages in persistent and buffered queues

AQ$_<queue_table_name>_F e.g. AQ$_QT3_F reports all messages that have not yet been dequeued in both

persistent and buffered queues

57


Buffered MessagesDatabase Objects The queue monitor slaves write spilled messages to

AQ$_<queue_table_name>_P Rows are inserted individually; no array operation is used

For example

INSERT INTO "us01"."aq$_qt3_p" (

q_name, msgid, corrid, priority,state, delay, expiration, time_manager_info,local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid, retry_count,

exception_qschema, exception_queue, recipient_key, dequeue_msgid,user_data, sender_name, sender_address, sender_protocol, dscn, cscn

) VALUES (:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,0,:15,:16,:17,:18,:19,:20,:21,:22,:23,:24)

Messages are asynchronously deleted from AQ$_<queue_table_name>_P by queue monitor slaves

Messages are deleted using an array size of 32 For example

DELETE FROM us01.aq$_qt24_p WHERE q_name = :1 AND msgid = :2

58


Buffered MessagesDatabase Objects AQ$_<table_queue_name>_D contains the following columns

Column Name

Data Type

OID NUMBER

MSGNUM NUMBER

MSGID RAW(16)

SUB NUMBER

SEQNUM NUMBER

RSUBS SYS.AQ$_RECIPIENTS The RSUBS column is stored as a LOB

59


Buffered MessagesSpillage If flow control is enabled then number of buffered messages that can be

enqueued on any queue is limited Subsequent attempts to enqueue messages will be rejected

Set _BUFQ_STOP_FLOW_CONTROL parameter to TRUE to disable flow control completely

Limited to 5000 buffered messages 15000 captured messages

Can be overridden in 10.2.0.3 by applying Patch 5093060 and setting Event 10867 for buffered messages (level is # messages) Event 10868 for captured messages (level is # messages)

Can be fixed in 10.2.0.4 onwards by setting: _BUFFERED_PUBLISHER_FLOW_CONTROL_THRESHOLD _CAPTURED_PUBLISHER_FLOW_CONTROL_THRESHOLD

60


Buffered MessagesDatabase Objects For a multiple consumer queue the following objects will be created when the

first buffered message is enqueued:


72638 AQ$_QT3_P TABLE

72639 SYS_LOB0000072638C00032$$ LOB

72640 SYS_IL0000072638C00032$$ LOB INDEX

72641 SYS_C0010003 INDEX

72642 AQ$_QT3_D TABLE

72643 SYS_IOT_OVER_72642 TABLE

72644 SYS_LOB0000072642C00006$$ LOB

72645 SYS_IL0000072642C00006$$ INDEX


61


PerformanceElapsed Times

Enqueue Dequeue

No Commit Commit No Commit Commit

Single Consumer PERSISTENTON COMMIT

4.77 10.99 5.75 9.62

Single ConsumerPERSISTENTIMMEDIATE

10.80 11.41 8.77 9.78

Single ConsumerBUFFEREDIMMEDIATE

2.32 2.60 1.53 2.13

Single ConsumerPERSISTENTON COMMIT

VARRAY(10) OF TYPE1

0.66 1.38 1.00 1.35

Multi Consumer

PERSISTENT

ON COMMIT2 recipients

6.40 14.45 6.36 11.20

Multi Consumer

PERSISTENT

ON COMMIT2 subscribers

6.02 14.59 6.54 11.40

10000 TYPE1 messages enqueued then 10000 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5 x86

62


PerformanceRedo Generation

Enqueue Dequeue

No Commit Commit No Commit Commit

Single Consumer PERSISTENTON COMMIT

9223 15491 10806 15521

Single ConsumerPERSISTENTIMMEDIATE

15165 15485 14936 14904

Single ConsumerBUFFEREDIMMEDIATE

0 0 0 0

Single ConsumerPERSISTENTON COMMIT

VARRAY(10) OF TYPE1

1211 1831 1381 1832

Multi Consumer

PERSISTENT

ON COMMIT2 recipients

16459 23324 8102 12521

Multi Consumer

PERSISTENT

ON COMMIT2 subscribers

15832 23404 7934 12953

10 TYPE1 messages enqueued then 10 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5 x86

63


Conclusion Several single queues may be more efficient than

Multiple recipients Multiple subscribers

Use ON_COMMIT visibility where possible No transaction overhead for queuing operations Reduces undo / redo generation IMMEDIATE is much more expensive

Buffered messages give best performance Provided they do not spill regularly

Array payloads are very efficient Message overhead is reduced

64


Thank you for your interest

[email protected]

1 © 2008 Julian Dyke juliandyke.com Advanced Queuing Internals Julian Dyke Independent Consultant...

Documents

Transcript of 1 © 2008 Julian Dyke juliandyke.com Advanced Queuing Internals Julian Dyke Independent Consultant...