WMQ and DB2 – MQ Verbs in DB2 & Q-Replication The DB2 MQListener † The DB2 MQListener is a...

1

WMQ and DB2 – Working together DB2/WMQ Functions and QReplication

Lyn [email protected]

2

Agenda

• The DB2 provided MQ Listener and Functions

• QREP – aka InfoSphere Replication Server• What it is and how it works• Issues we have seen• Options and Requirements

3

DB2 provided MQ Listener and Functions

• MQListener• DB2 MQ Services• DB2 MQ Policies• MQSEND• MQREAD and MQREADCLOB• MQRECEIVE and MQRECEIVECLOB• Using WebSphere MQ pub/sub with DB2

4

The DB2 MQListener Process

MQ Listener

MQ Queue

5

The DB2 MQListener

• The DB2 MQListener is a combination of WMQ and DB2 functionality• DB2 Process will listen for messages on a queue• Invoke a stored procedure to act on the message• Possibly send a reply

• Often used to send updates asynchronously, when the data is ‘preformatted’• ‘Near realtime’ updates with all the advantages of queuing! • Growth in usage for event and audit trail collections

6

DB2 MQListener - continued

• The DB2 MQListener on z/OS:• Provides both 1 and 2 phase commit for resources

• db2mqln1 - single phase • Db2mqln2 – two phase

• Installs and runs in USS• Configured via database updates

• SYSMQL.LISTENERS table

• From an MQ perspective:• Standard WMQ administration• Good recommendations for DLQ, Backout queue and

threshold

7

The DB2 MQListener - continued

• The Listeners table entries include:• Configuration name• Queue manager• Input queue• Node – unused• Schema – the schema name for the stored procedure to be invoked• Name – the name of the stored procedure• Type – unused• Instances – number of instances of the procedure that will be

invoked • Wait – the number of milliseconds that the listener should wait

before checking for a new message

8

DB2 WMQ Functions

• MQListener• DB2 MQ Services• DB2 MQ Policies• MQSEND• MQREAD, including

• MQREADALL• MQREADALLCLOB • MQREADCLOB

• MQRECEIVE, including • MQRECEIVEALL • MQRECEIVEALLCLOB• MQRECEIVECLOB

• Using MQ pub/sub with DB2 WMQ functions

9

The DB2 Service Definitions for MQ Functions

• Defines the MQ resources that the DB2 MQ functions use• The connection between DB2 and WMQ• Typically maintained by the DB2 admin

• SYSIBM.MQSERVICE_TABLE - Holds the following information:• Service Name - DB2.DEFAULT.SERVICE• Queue manager - !QUEUEMANAGER!• Queue name - DB2MQ_DEFAULT_Q

• This field is called ‘Input queue’• CCSID• Encoding• Description

10

DB2 Policy Definitions for MQ Functions

• Typically maintained by the DB2 admin• Defines the MQ quality of service for the DB2 MQ functions

• The send and receive values correspond to MQ message settings

• SYSIBM.MQPOLICY_TABLE - Holds the following information:• Policy Name - DB2.DEFAULT.POLICY• Send Values:

• Priority – 0-9, and -1 which is ‘as queue defined’• Persistence

• Q – As queue defined• Y - Persistent - messages are logged and recoverable• N - Nonpersistent – messages are not logged

11

DB2 Policy Definitions for MQ Functions - Continued

• Send Values Continued• Expiry

• Messages can be set to expire if not processed • Retry Parameters

• Retry Count• If the put of the message fails, the put can be retried up

to this limit • Retry Interval

• The wait time in milliseconds between retries of the message put

12


• Send Values Continued• Correlid Settings

• New Correlid• The correlation ID can be set to new or not

• Response Message ID• Sets whether the report message will have the same message

ID as the original • Response Correlation ID

• Whether the processing application should set the correlid on the reply message from the request message ID or request correlid

13


• Send Values Continued• Exception Actions

• A problem message can be discarded or put to the queue manager’s dead letter queue

• Report Options• Exception reports

• Send or do not send an exception message • Confirm on Arrival

• Send or do not send an confirmation report when the message is put to the target queue

• Confirm on Delivery• Send of do not send a confirmation report when the message

is retrieved from the target queue• Message expiration

• Send or do not send a report when the message expires.

14


• Send Values Continued• Message Type• Reply to Queue• Reply to Queue Manager

• Receive Values:• Wait interval• Accept Truncated Message• Open Shared

• Syncpoint

15

DB2 Policy Definitions for MQ

•The foils that follow•Give the DB2/MQ cross reference

•Most are hidden for the presentation

•Highlighted (and not hidden) are where MQ recommendations differ from the DB2 default

Who lives in the details???

16

DB2 Policy Definitions for MQ Functions – Send columns

Confirmation of delivery reply is sent when the message is retrieved from the queue, and the retreivalremoves the message (aka ‘destructive MQGET’).

MQRO-COD BINARY 2048

MQMD-REPORTCSEND_REPORT_COD

Confirmation of arrival reply is sent when the message arrives on the specified queue.

MQRO-COA BINARY 256

MQMD-REPORTCSEND_REPORT_COA

COD reply is sent with the first 100 bytes of the message body

MQRO-COD-WITH-DATA BINARY 6144

MQMD-REPORTDSEND_REPORT_COD

COD reply is sent with the entire message body

MQRO-COD-WITH-FULL-DATA BINARY 14336

MQMD-REPORTFSEND_REPORT_COD

COA reply is sent with the first 100 bytes of the message body

MQRO-COA-WITH-DATA BINARY 768

MQMD-REPORTDSEND_REPORT_COA

COA reply is sent with the entire message body

MQRO-COA-WITH-FULL-DATA BINARY 1792

MQMD-REPORTFSEND_REPORT_COA

MQMD-REPORT

MQMD-REPORT

MQMD Field

N -Default

N -Default

Value

No COA is sent. MQRO-NONE BINARY 0

SEND_REPORT_COA

No COD is sentMQRO-NONE BINARY 0

SEND_REPORT_COD

MeaningMQ ValueDB2 Policy Name

17


• Each report option has four values:• ‘N’ – do not report • ‘C’ – send a report message, but do not include any of the

message data• ‘D’ – send a report message, include the first 100 bytes of

message data• ‘F’ – send a report message and include the entire message

body

18


The reply to queue manager for request and report messages.

SpacesMQMD-REPLYTOQMGR

SAME AS INPUT_QMGRDefault

REPLY_TO_QMGR

The reply to queue for request and report messages. This SHOULD NEVER BE ALLOWED TO DEFAULT LIKE THIS! Infinite loops will occur if left like this.

SpacesMQMD-REPLYTOQSAME AS INPUT_QDefaultREPLY_TO_Q

MQMD FieldValue MeaningMQ ValueDB2 Policy Name

19

DB2 Policy Definitions for MQ Functions – Receive columns

The message body is not converted by WebSphere MQ.

zeroMQGMO-OPTIONS NRCV_CONVERT

The message body is to be converted from the CCSID and encoding from the sender to the CCSID and encoding used by the receiving queue queue manager. Typically the data is in string format.

MQGMO-CONVERT BINARY 16384

MQGMO-OPTIONS YDefault

RCV_CONVERT

The time in milliseconds that the MQGET request will wait on an inbound message. Note that in MQ, waiting is implemented using a value on the MQGET options and an interval.

ZeroMQGMO-WAITINTERVAL

10Default

RCV_WAIT_INTERVAL

MQMD Field(s)Value MeaningMQ ValueDB2 Policy Name

20

DB2 Policy Definitions for MQ Functions – Receive columns

Queue is open in shared mode, so other applications can retrieve messages from the queue concurrently.

MQOO-INPUT-SHARED BINARY 2

MQ Open optionsSDefault

RCV_OPEN_SHARED

Queue is open for input to this application only. Used when data serialization is required.

MQOO-INPUT-EXCLUSIVE BINARY 4

MQ Open optionsERCV_OPEN_SHARED

If the message is larger than the buffer provided, the message is not retrieved. The function will get a reason code of MQRC_TRUNCATED_MSG_FAILED.

Zero.MQGMO-OPTIONS NRCV_ACCEPT_TRUNC_MSG

As much of the message as can be is retrieved into the message buffer available. The remainder is lost when the transaction completes. Note that DB2 recommends this behavior, but those of us in WMQ do not. Some of the message body is lost.

MQGMO-ACCEPT-TRUNCATED-MSG BINARY 64.

MQGMO-OPTIONS YDefault

RCV_ACCEPT_TRUNC_MSG

Queue is opened with the values as defined on the queue.

MQOO-INPUT-AS-Q-DEF BINARY 1

MQ Open optionsDRCV_OPEN_SHARED

MQMD Field(s)Value MeaningMQ ValueDB2 Policy Name

21

The Functions: MQSEND

• MQSEND is a SQL-centric way of putting messages on a queue

• The parameters include:• The Send Service • The Service Policy • Message Data• Correlid

Test Send Service Definition

INSERT INTO SYSIBM.MQSERVICE_TABLE (SERVICENAME, QUEUEMANAGER,

INPUTQUEUE, DESCRIPTION) VALUES('LYNMQSEND','QML1','LYNE.TESTDB2.MQSEND',

'LYN ELKINS MQSEND TEST');

22

Test MQSEND Policy DefinitionINSERT INTO SYSIBM.MQPOLICY_TABLE

(POLICYNAME, SEND_PRIORITY, SEND_PERSISTENCE, SEND_RETRY_COUNT,

SEND_RETRY_INTERVAL,SEND_NEW_CORRELID, SEND_RESPONSE_MSGID,

SEND_RESPONSE_CORRELID, SEND_EXCEPTION_ACTION, SEND_REPORT_EXCEPTION,

SEND_REPORT_COA, SEND_REPORT_COD, SEND_REPORT_EXPIRY,

SEND_REPORT_ACTION, SEND_MSG_TYPE, REPLY_TO_Q, REPLY_TO_QMGR,

RCV_WAIT_INTERVAL, RCV_CONVERT, RCV_ACCEPT_TRUNC_MSG,

RCV_OPEN_SHARED, SYNCPOINT, DESC)

VALUES('LYNE_MQSEND_POLICY','-1','Q', '4',

'1000', 'N', 'N',

'C', 'Q', 'N',

'N','N', 'N',

'N', 'DTG', 'LYNE.TESTDB2.BADQ', 'QML1',

'10', 'Y', 'N',

'S', 'Y', 'LYNS TEST MQSEND POLICY');

23

Test MQSEND results

24

MQREAD and MQRECEIVE

• MQREAD and MQRECEIVE are SQL-centric way of retrieving messages from a queue

• The parameters include:• The Send Service • The Service Policy• CORRELID for MQRECEIVE

• MQREAD and MQREADCLOB are non-destructive MQGET operations

• MQRECEIVE and MQRECEIVECLOB are destructive MQGET operations

25

MQREAD & MQREADALL Tests

Service Definition:

INSERT INTO SYSIBM.MQSERVICE_TABLE (SERVICENAME, QUEUEMANAGER, INPUTQUEUE, DESCRIPTION) VALUES('LYNMQREAD','QML1','LYNE.TESTDB2.MQREAD', 'LYN ELKINS MQREAD TEST');

26

MQREAD Test

27

MQREADALL Test

28

MQRECEIVE & MQRECEIVEALL Tests

Service Definition:

INSERT INTO SYSIBM.MQSERVICE_TABLE (SERVICENAME, QUEUEMANAGER, INPUTQUEUE, DESCRIPTION)

VALUES('LYNMQRECEIVE','QML1','LYNE.TESTDB2.MQRECEIVE','LYN ELKINS MQRECEIVE TEST');

29

MQRECEIVE Test

30

MQRECEIVEALL Test

31

Using MQ Pub/Sub Features from DB2

• When you require more control over the messages that a particular service should receive, you can use the MQPUBLISH function, in conjunction with the WebSphere MQSeries Integrator facility. This facility provides a publish-and-subscribe system, which provides a scalable, secure environment in which many subscribers can register to receive messages from multiple publishers. Subscribers are defined by queues, which are represented by service names.

• MQPUBLISH enables you to specify a list of topics that are associated with a message. Topics enable subscribers to more clearly specify the messages they receive. The following sequence illustrates how the publish-and-subscribe capabilities are used:

• An MQSeries administrator configures the publish-and-subscribe capability of the WebSphere MQSeries Integrator facility.

• Interested applications subscribe to subscriber services that are defined in the WebSphere MQSeriesIntegrator configuration. Each subscriber selects relevant topics and can also use the content-based subscription techniques that are provided by Version 2 of the WebSphere MQSeries Integrator facility.

• A DB2® application publishes a message to a specified publisher service. The message indicates the topic it concerns.

• The MQSeries functions provided by DB2 for z/OS® handle the mechanics of publishing the message. The message is sent to the WebSphere MQSeries Integrator facility by using the specified service policy.

• The WebSphere MQSeries Integrator facility accepts the message from the specified service, performs any processing defined by the WebSphere MQSeries Integrator configuration, and determines which subscriptions the message satisfies. It then forwards the message to the subscriber queues that match the subscriber service and topic of the message.

• Applications that subscribe to the specific service, and register an interest in the specific topic, will receive the message in their receiving service.

32

Publishing a message from DB2

• In WebSphere MQ set up the following definitions:• A WMQ topic

33


• A WMQ alias queue

34

• In DB2 set up the following definition:• A new service using the ALIAS queue name


35

Publishing a message from DB2 -Testing

• In WMQ, define the following:• Two local queues to

serve as the subscription destination

• LYNE.TESTDB2.Q1• LYNE.TESTDB2.Q2

• Two subscriptions using the topic defined earlier and the local queue as the destination for the publications

36


• Use the MQSEND function to publish

37


• Both queues that are targets of the subscriptions have a copy of the message

38

Using native MQ Subscriptions and DB2

• This is basically an administrative task• Define the subscriptions to MQ pointing• Define MQ policies in DB2 that point to those queues• Use the MQREAD/MQRECEIVE functions to retrieve the

publications (messages)

39

Now for something completely different

• QREP• Aka InfoSphere Replication Server

40

Q Replication Technology• Part of the InfoSphere Data Replication product• A software-based asynchronous replication solution

• For Relational Databases• For selected tables/columns/transactions/operations• Changes are captured from the database recovery log; transmitted as (compact) binary data;

and then applied to the remote database(s) using SQL statements. Technology is log-capture/transaction-replay

• Each DB2 is ACTIVE and can be used for READ/WRITE operations

DB2

Control Tables

Site ADB2

Control tables

Q CaptureQ Apply

agentagentagentUser

tablesdatabase recoverylog

User tables

Unlimited Distance

Site B

Configuration & Monitoring

logrdr publishData CenterWebSphere MQ

DB2 Transaction Parallel Replay

AsynchronousLOG change data capture

Active DB2Active DB2 Persistent Staging Capability

SQL statements

41

Q Replication Process

WebSphere MQ

queue manager

DB2

Site 1

DB2

Q Capture

Q Applyagentagentagent

User tables

database recoverylog

User tables

WebSphere MQ

queue manager

Network

Site 2

Applications

Remote Admin QAdmin queue

Send queue and XMITQ

Restart queue

Receive queue(s)

Captured DB2 Transactions from the LOG

logrdr

publish

Q Apply agents execute DB2 SQL transactions

DB2 transactions

SQL statements

SQL statements

• Capture and Apply are DB2 applications• Parallel Capture program publishes captured database transactions as compact MQSeries

messages, only the data is sent• One MQ message = One captured DB2 transaction

• Apply program replays transactions by rebuilding SQL statements• Each component runs independently and can be individually stopped/restarted at any time, in any

order

42

WebSphere MQ Queue Manager (or client on LUW)

DB2

Active/Active Configurations

Control Tables

Site A

DB2

Control tables

Q Capture



User tables


User tables

Q Capture

WebSphere MQ Queue Manager (or client on LUW)

Network1000s of

kilometers

Site B

Applications

Remote Admin QAdmin queue

Receive queue(s)

Send queue(s)

Restart queue Restart queue

Remote Admin Q

Send queue(s)

Receive queue(s)

Admin queue

Configuration & Monitoring

• Both sites are active• Sites can be asymmetric; with different capacities, and at different HW/OS/SW levels• A workload can be immediately redirected for recovery or planned maintenance (RTO of sub-second)

• Replication is defined as bi-directional• Bi-directional, even if one side is used for read-only or standby for all or a subset of the workloads• Replication is ready for fallback after a failover, from any direction

• Routing decision determines the role. e.g., West-Coast customers to site A and East-Coast customers to site B, until failover. Objective is to avoid conflicts.

West-Coast OLTP workload East-Coast OLTP

workload

sub-secondlatency

Query Reporting workload


43

Conflict Detection and Resolution with Q Replication -Bidirectional configurations

‘Bidirectional’ mode – detection by value - resolution Force/Ignore; Ignore/Ignore

• 2 or up to tens of sites connected though a central site; a hub and spokes model (or tree topology)

• Conflict detection: Compare current values at the target with the before values carried over from the source. Check key, changed cols only, or all columns

• Possibility of merging updates to different columns of the same row made from different sites

• Conflict action: Force or Ignore (merging updates is not a conflict) - Designated site wins in case of conflict, master/master is OK

• Recapture avoidance: Apply inserts a row into the SIGNAL table with each transaction it applies. If there are only 2 sites, can use Capture IGNORE TRANSACTION for less overhead.

Resolving conflicts at the row level minimize the impact of conflicts by “merging” transactions for non conflicting changes. One conflict in a transaction doesn’t invalidate the whole transaction, which could result in more conflicts later on. Resolved conflicts are reported into the EXCEPTIONS table. Each conflict is reported only once.

A: Master

C: Master or Slave

B

44

Q Apply includes the before values in the where clause for conflict detection (bi-di protocol)In ‘bidi’, Q Apply augments the SQL with a where clause to check old values, e.g.:

• table T1(ID int primary key, C1 int, C2 varchar);• Row1: (100, 5, ‘Jim’)

• SQL by source application:• update T1 set C1=6 where C2=‘Jim’;

• DB2 log record:• Old row: ID = 100, C1 = 5; C2 = ‘Jim’• New row: ID = 100, C1 = 6; C2 = ‘Jim’

• SQL by Q Apply:• update T1 set C1=6 where ID=100 and C1=5 and C2=‘Jim’;

•The SQL where clause was augmented to verify that the before value was not modified and that the correct row is updated.

• If row not found (sqlcode -803) --> CONFLICT!• The same technique is used for uni-directional replication, but only the replication key before value is included in the where clause.

45

The EXCEPTIONS table• All conflicts and errors are reported in EXCEPTIONS table• Complete SQL statement is logged for each conflict• Sample entry:

EXCEPTION_TIME RECVQ SUBNAME OPERATION REASON-------------------------- ----- ---- --------- ---------2009-06-17-12.35.41.387373 Q1 S1 INSERT DUPLICATE

SRC_COMMIT_LSN SRC_TRANS_TIME IS_APLLIED----------------------- -------------------------- -----------x'00000000000002F1F0A5' 2009-06-17-19.35.39.000001 Y

SRC_TRANS_ID CONFLICT_RULE----------------------------------- --------------------x'000000000000000002E3' K

SQLCODE SQLSTATE SQLERRMC------- -------- ---------------------------------803 23505 x’31FF0000...0000000000000000000'

TEXT-------------------------------------------------------------------INSERT INTO "SOMILK"."TARG_T1“( "KEYP“, “PAY1”, “PAY2”) VALUES (1, ‘’, NULL)

46

Q Replication Potential Bottlenecks

WebSphere MQ

queue manager

DB2

Site A

DB2

Q CaptureQ Apply

agent

agent

agentUser

tables


User tables

WebSphere MQ

queue manager

Channel (TCP/IP)

Site BApplications

Send queue and XMITQ Receive queue(s)

Captured DB2 Transactions from the LOG

(1 per MQ msg)

logrdrpublish

Q Apply agents execute SQL transactions

SQL statements

SQL statements

1. MQ channel (single TCP/IP connection) for OLTP transactions (small messages) limited at 7MB/second for 2K messages2. MQ bufferpool filling up. Reading or writing to the pageset can reduce replication throughput by 4 times.3. Capture publish thread might become CPU bound, this thread includes MQ PUT and COMMIT elapsed times. 4. Workload contention at the target (locking and or dependency of transactions updating the same rows) can be caused by

applications running at the target or by multiple apply agents applying transactions in parallel.5. Apply program limited to one data sharing member.6. CPU and I/O Speed

DB2 IFI 306(read log I/F) MQ pageset

MQGET (browse)browse

prune

MQGET (delete)

MQ Bufferpool

MQPUT, MQCMT

MQ pageset

MQ Bufferpool12

3

4

5

Applications

47

Q Replication Parameter SettingsQ Capture:• MAXTRANS(new parm)=500. At one customer, an effective MQ batch size of 42 was the best

that cold be acheived. Tests have established that an MQ batch size of 200 is optimal, with good values ranging from 50 to 200. COMMIT_INTERVAL=500 (milliseconds). Q Capture commits MQ after a maximum of 500ms. However, at high throughput, the maxtrans limit is reached much sooner, and Capture commits much more frequently, about every 50 ms.

• SLEEP_INTERVAL= 50 (milliseconds) - This was established as being optimal during the customer and internal tests.

• TRANS_BATCH_SZ=3 and TRANS_BATCH_SZ=1. This value may vary, at one high volume customer a value of 3 was established as yielding the best performance results. With value 3, Q Replication replicate 3 DB2 transactions as 1.

• MEMORY_LIMIT=500 (MB) – This setting had no significant impact in our testing, equally good results were achieved with 200MB.

Q Apply:• NUM_APPLY_AGENTS=48 - The optimal number of agents depends on cpu resources, the

characteristics of the replicated workload, and possibly contention for locks. Q Replication provides monitor counters to help tune this parameter (apply_sleep_time).

• MEMORY_LIMIT= 500MB

48

Queue Replication – Ongoing work and suggestions

• The QREP development team are currently evaluating• Using log compression

• Very data dependant

• Some suggestions that have been made: • More flexibility in Message Persistence requirements

• Right now at a ‘universal’ setting• Perhaps using a technique like FTE, where most messages are

nonpersistent, only status (‘blocking’) messages are persistent• Multiple capture and apply queues

• Message groups processed in a round-robin fashion

49

CICS and WMQ - The Resurrection of Useful

Getting the best availability from MQ on z/OS by using Shared Queues

Clustering - The Easier Way to Connect Your Queue Managers

06:00

WebSphere MQ Channel Authentication Records

MQ & DB2 – MQ Verbs in DB2 & Q-Replication

What’s New in the WebSphere MQ Product Family

WebSphere MQ application design, the good, the bad and the ugly

04:30

Using IBM WebSphere Application Server and IBM WebSphere MQ Together

WebSphere MQ CHINIT Internals

BIG Connectivity with WebSphere MQ and WebSphere Message Broker

First Steps With WebSphere Message Broker: Application Integration for the Messy

03:00

The Dark Side of Monitoring MQ - SMF 115 and 116 Record Reading and Interpretation

Migration and maintenance, the necessary evil

MQ on z/OS – VivisectionIntroduction to MQ01:30

12:15

BIG Data Sharing with the cloud - WebSphere eXtreme Scale and WebSphere Message Broker integration

WMQ - Introduction to Dump Reading and SMF Analysis -Hands-on Lab

Extending IBM WebSphere MQ and WebSphere Message Broker to the Cloud

11:00

Diagnosing Problems for MQ

What's New in WebSphere Message Broker

09:30

Are you running too many queue managers or brokers?

08:00

FridayThursdayWednesdayTuesdayMonday

This was session ???? - The rest of the week ……

50

Lyn [email protected]

MQ & DB2 – MQ Verbs in DB2 & Q-Replication [z/OS]Wednesday, February 6, 2013: 4:30 PM-5:30 PM

WMQ and DB2 – MQ Verbs in DB2 & Q-Replication The DB2 MQListener † The DB2 MQListener is a...

Documents

Transcript of WMQ and DB2 – MQ Verbs in DB2 & Q-Replication The DB2 MQListener † The DB2 MQListener is a...