3452 - Managing your applications

3452: Managing your applications

Tim McCormick

IBM MQ Service Architect

[email protected]

Who is this presentation aimed at?

• Application developers who want to understand how their application is

really interacting with MQ

– Does the reality match the design and expectation?

• MQ administrators who want to understand how applications are

interacting with their queue manager

– Is the application behaving as described?

– Has the application been coded to required standards?

– Is the application exhibiting best practice?

Agenda

• Accounting messages

• API Trace

• Application Activity trace

Accounting messages

What are accounting messages?

• Information about the MQI operations performed by an application

• PCF messages

– Delivered to SYSTEM.ADMIN.ACCOUNTING.QUEUE

• When the application disconnects and…

• When the specified interval has been reached (ACCTINT)

– Not persistent

What are accounting messages?

• Two types

– MQI accounting

• MQI accounting messages contain information relating to the number of MQI calls

made using a connection to a queue manager.

– Queue accounting

• Queue accounting messages contain information relating to the number of MQI calls

made using connections to a queue manager, grouped by queue.

How do I enable accounting messages?

• Queue manager attributes

– ACCTMQI

– ACCTQ

• Queue attributes

– ACCTQ

• MQCONNX connection options

– MQCNO_ACCOUNTING_MQI_ENABLED

– MQCNO_ACCOUNTING_MQI_DISABLED

– MQCNO_ACCOUNTING_Q_ENABLED

– MQCNO_ACCOUNTING_Q_DISABLED

How do I read accounting messages?

• Read PCF messages from SYSTEM.ADMIN.ACCOUNTING.QUEUE

• amqsmon sample

– Some output is in cryptic list form, example:

• OpenCount: [0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

• This means 2 objects of type MQOT_Q (queue) were opened

• Demystify via documentation:

– Search ‘MQI accounting message data’

• Vendors

• Roll your own

amqsmon example output (MQI accounting)

MonitoringType: MQIAccounting

QueueManager: 'MQ80'

IntervalStartDate: '2016-02-16'

IntervalStartTime: '18.16.56'

IntervalEndDate: '2016-02-16'

IntervalEndTime: '18.17.00'

CommandLevel: 800

ConnectionId: x'41…02e01'

SeqNumber: 0

ApplicationName: 'C:\WMQ\8.0\bin64\amqsput.exe'

ApplicationPid: 35388

ApplicationTid: 1

UserId: 'timmccor'

ConnDate: '2016-02-16'

ConnTime: '18.16.56'

DiscDate: '2016-02-16'

DiscTime: '18.17.00'

DiscType: Normal

OpenCount: [0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

OpenFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

CloseCount: [0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

CloseFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

PutCount: [2, 0]

– Much more data is logged than appears

on this slide

– Interval start/end matches connection

life, means no intermediates generated

– See useful information about the

application connected

– Most interesting fields are

[-,Q,-,-,-,QMGR,-,-,Topic,-,-,-,-,-]

We opened 2 queues.

– PutCount: [Nonpersistent,Persistent].

We put 2 non-persistent messages

amqsmon example output (queue accounting)

MonitoringType: QueueAccounting



UserId: 'timmccor'

QueueAccounting: 0

QueueName: 'Q1'

CreateDate: '2015-02-05'

CreateTime: '13.45.58‘

QueueDefinitionType: Local

OpenCount: 1

OpenDate: '2016-02-18'

OpenTime: '14.04.20'

CloseCount: 1

CloseDate: '2016-02-18'

CloseTime: '14.04.22'

PutCount: [1, 0]

PutFailCount: 0

Put1Count: [0, 0]

Put1FailCount: 0

PutBytes: [4, 0]

PutMinBytes: [4, 0]

PutMaxBytes: [4, 0]

– Much more data is logged than appears

on this slide

– See useful information about the

application connected and when it

opened and closed the queue

– See useful information about how the

queue was defined

– PutCount: [Nonpersistent,Persistent].

We put 1 non-persistent messages

When should I use accounting messages?

• Low impact on system

• Scoped to an individual connection or queue

– Connection level information, won’t provide much detail on the objects accessed

– Queue level information may be too broad

API Trace

What is API trace?

• API trace is a part of the diagnostic trace IBM support will request in

PMRs

– Cut down to show only the API calls and less of the internal working of the system

– Intended for IBM support, but can be useful for customers

How do I enable API trace?

• strmqtrc –t api –p myapplicationname

– Start API trace for applications with named ‘myapplicationname’

– Don’t forget the .exe on Windows

• endmqtrc -a

– Stop trace

How do I read API trace?

• MQ_DATA_DIR/trace/AMQPID.0.TRC

– Where PID is the process ID

• Trace is output in a binary non-readable format

– Run dspmqtrc to format the trace into human readable format

– Use dspmqtrc from the same installation which generated the trace

– Except on Windows where trace is output in human readable format

API trace example (header)

• Header at top of every file

• A 64-bit version of amqsput generated

this file

• Application was run on a Windows 7

system named MYHOST

• Application was at MQ version 8.0.0.4

• MQ install and data paths

• This install is MQ80, there are 3 others

on the system

Process : C:\WMQ\8.0\bin64\amqsput.exe (64-bit)

Arguments :

Host : MYHOST

Operating System : Windows 7 Professional x64 Edition, Build

7601: SP1

Product Long Name : WebSphere MQ for Windows (x64 platform)

Version : 8.0.0.4 Level : p800-004-151017

O/S Registered : 1

Data Path : C:\WMQ\data

Installation Path : C:\WMQ\8.0

Installation Name : MQ80 (3)

License Type : Production

UTC Date : 2016/02/18: Time : 11:52:44.759

LCL Date : 2016/02/18: Time : 11:52:44.759 GMT Standard Time

API trace example (trace records)

• The counter is a hexadecimal field which

increments by 1 for every trace record

• The timestamp is the time in the

timezone from the header

• PID.TID – Process ID and Thread ID

logging the trace record

• Ident provides additional identity

information depending on context

• Data is the ‘interesting’ data which was

traced

Counter TimeStamp PID.TID Ident Data

==================================================

000001DE 12:09:21.687697 25528.1 : !! - __________

000001DF 12:09:21.687713 25528.1 : !! - MQCONN >>

000001E9 12:09:21.702901 25528.1 CONN:525958 !! - MQCONN <<

API trace example (request)

• Only trace data shown on this

slide

• Requests in the form

MQIVERB >>

• Bytes displayed at their

memory address, ASCII

conversion to the right

• MQ compresses duplicate trace

lines

• Name: MQ80

4D = M 51 = Q 38 = 8 30 = 0

Padded by spaces (20)

!! - MQCONN >>

!! - Name:

Data:-

0x00000000 002AEAB0 4D 51 38 30 20 20 20 20 20 20 20 20 20 20 20 20 : MQ80

0x00000000 002AEAC0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 :

0x00000000 002AEAD0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 :

!! - ConnectOpts:

Data:-

0x00000000 002AF6E0 43 4E 4F 20 01 00 00 00 00 00 00 00 : CNO ........

!! - Hconn : Output Parm

!! - Compcode : Output Parm

!! - Reason : Output Parm

Data:-

0x00000000 002AF010 4C 43 4E 4F 04 00 00 00 00 40 00 00 00 00 00 00 : LCNO.....@......

0x00000000 002AF020 FF FF FF FF 00 00 00 00 00 00 00 00 00 00 00 00 : ................

0x00000000 002AF030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................

7 lines suppressed, same as above

0x00000000 002AF0B0 00 00 00 00 00 00 00 00 FF FF FF FF 00 00 00 00 : ................

0x00000000 002AF0C0 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 : ................

0x00000000 002AF0D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................

API trace example (response)

• Only trace data shown on this

slide

• Responses in the form

MQIVERB <<

• Little endian platforms need to

be byte swapped

• MQOPEN returned Hobj 2

02 00 00 00 byte swaps to

00 00 00 02 = 2

• MQOPEN was a queue

• API call successful

!! - MQOPEN <<

!! - Hconn : Input Parm

!! - Objdesc:

Data:-

0x00000000 002AEDD0 4F 44 20 20 01 00 00 00 01 00 00 00 51 31 00 00 : OD ........Q1..

0x00000000 002AEDE0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................

4 lines suppressed, same as above

0x00000000 002AEE30 00 00 00 00 00 00 00 00 00 00 00 00 41 4D 51 2E : ............AMQ.

0x00000000 002AEE40 2A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : *...............

0x00000000 002AEE50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................

0x00000000 002AEE60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................

0x00000000 002AEE70 00 00 00 00 00 00 00 00 : ........

!! - Options : Input Parm

!! - Hobj:

Data:-

0x00000000 002AF7B0 02 00 00 00 : ....

!! - ObjHdl:2 ObjType:QUEUE ObjName:Q1 ResObjName:Q1

!! - Compcode:

Data:-

0x00000000 002AF6D4 00 00 00 00 : ....

!! - Reason:

Data:-

0x00000000 002BFD60 00 00 00 00 : ....

!! - MQI:MQOPEN HConn=03400006 HObj=00000002 rc=00000000 ObjType=00000001

ObjName=Q1

When should I use API trace?

• Easy to enable

• Impact on disk I/O

• Difficult to read

– Intended as diagnostic information for IBM support teams

• Scoped to process

– All Java applications will be seen as ‘java’ as the JVM is the process

Application Activity Trace

What is application activity trace?

• Similar to accounting messages, but…

– Much more detail on what the application was doing

– Can be controlled at the application level

– Could be used to audit application behavior

• PCF messages

– Delivered to SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE when…

– Application disconnects, or…

– ActivityCount or ActivityInterval thresholds are reached, or…

– When the size of the message which will be written reaches the maximum message length for the

queue

– Non-persistent messages

How do I enable application activity trace?

• Queue manager attribute

– ACTVTRC

• MQCONNX connection option

– ACTVCONO queue manager attribute must be enabled

– MQCNO_ACTIVITY_TRACE_DISABLED

– MQCNO_ACTIVITY_TRACE_ENABLED

• mqat.ini configuration file

– Found in the queue manager data directory

– Does not require queue manager restart, reconnect or

alter qmgr object in any way

– Can specify individual settings by application name Overrides

Application activity trace on the MQ Appliance

• Subscribe to application activity trace system topic

– $SYS/MQ/INFO/QMGR/my_qmgr/ActivityTrace/…

• mqat.ini configuration file

– Only global settings available on the MQ Appliance via setmqini

mqat.ini example, global settings

AllActivityTrace:

ActivityInterval=1

ActivityCount=100

TraceLevel=MEDIUM

TraceMessageData=0

StopOnGetTraceMsg=ON

– Application activity trace won’t be logged without at least one ApplicationTrace stanza, or the queue

manager attribute ACTVTRC set to on

– Log an application activity trace message every one second, or one hundred MQI or XA operations

– Log a medium amount of detail, documentation lists which operations are logged for each level

– Do not trace any message data for get or put operations

– Don’t generate application activity trace when handling application activity trace messages (to avoid

looping)

mqat.ini example, application settings

ApplicationTrace:

ApplClass=ALL

ApplName=amqs*

Trace=ON

TraceLevel=HIGH

– You need at least one ApplicationTrace stanza to log application activity trace, or ACTVTRC set to on

– User applications only or include the MCA application?

– Application name, can be wildcarded. By default the application name is the process name. From MQ

7.5 JMS clients can specify application name via the connection factory.

– Turn application activity trace on

– Override global settings as needed

How do I read application activity trace?

• Read PCF messages from

SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE

• amqsact

• MS0P

• Other vendors

• Roll your own

amqsact exampleMonitoringType: MQI Activity Trace


Host Name: ‘MYHOST'

IntervalStartDate: '2016-02-17'

IntervalStartTime: '11:59:20'

IntervalEndDate: '2016-02-17'

IntervalEndTime: '11:59:22'

CommandLevel: 800

SeqNumber: 0


UserId: 'timmccor'

Trace Detail Level: 2

Platform: MQPL_WINDOWS_NT==========================================================Tid Date Time Operation CompCode MQRC HObj (ObjName)

001 2016-02-17 11:59:20 MQXF_CONNX MQCC_OK 0000 -

001 2016-02-17 11:59:20 MQXF_OPEN MQCC_OK 0000 2 (Q1)

001 2016-02-17 11:59:22 MQXF_PUT MQCC_OK 0000 2 (Q1)

001 2016-02-17 11:59:22 MQXF_CLOSE MQCC_OK 0000 2 (Q1)

001 2016-02-17 11:59:22 MQXF_DISC MQCC_OK 0000 -

==========================================================

– amqsput

– Running on Windows

– Connected to MQ80

– On MYHOST

– With user ID timmccor

– This activity trace was logged with

medium detail

• 1 = Low, 2 = Medium, 3 = High

– Summary of API calls made

amqsact verbose example (MQOPEN)

– 2nd MQI call

– MQOPEN

– Of a queue

– Named Q1

– Which resolved to a queue named Q1

on MQ80

– Opened on object handle 2

– With options 8208

• MQOO_FAIL_IF_QUIESCING &

MQOO_OUTPUT

– Successfully

– And the API call took 41ms to complete

on the queue manager

MQI Operation: 1

Operation Id: MQXF_OPEN

ApplicationTid: 1

OperationDate: '2016-02-17'

OperationTime: '16:31:42'

Object_type: MQOT_Q

Object_name: 'Q1'

Object_Q_mgr_name: ''

Hobj: 2

QMgr Operation Duration: 41

Completion Code: MQCC_OK

Reason Code: 0

Open_options: 8208

Resolved_Q_Name: 'Q1'

Resolved_Q_mgr: 'MQ80'

Resolved_local_Q_name: 'Q1'

Resolved_local_Q_mgr: 'MQ80'

Resolved_type: MQOT_Q

Dynamic_Q_name: 'AMQ.*'

amqsact verbose example (MQPUT extract)

– Much more data than can be displayed on this slide.

– 3rd MQI call

– MQPUT

– In addition to usual timings, you get the epoch value when the

operation started

– The object was Q1 which resolved to a local queue named Q1.

– The message was put under object handle 2


• MQPMO_NO_SYNCPOINT & MQPMO_NEW_MSG_ID &

MQPMO_FAIL_IF_QUIESCING

– Successfully

– And the API call took 115ms to complete on the queue manager

– Did the application use best practices?

MQI Operation: 2

Operation Id: MQXF_PUT

High Res Time: 1455726703762959


Reason Code: 0

Hobj: 2

Put Options: 8260

Object_name: 'Q1‘

Resolved_Q_Name: 'Q1'

Resolved_local_Q_name: 'Q1‘


Report Options: 0

Expiry: -1

Format_name: 'MQSTR'

Priority: -1

Persistence: 2

Msg_id: …

Correl_id: …

Reply_to_Q : ''

Reply_to_Q_Mgr: ''

Coded_char_set_id: 0

Encoding: 546

Put_date: '20160217'

Put_time: '16314375'

amqsact verbose example (MQGET extract)

– Much more data than can be displayed on this slide.

– 3rd MQI call

– MQGET

– In addition to usual timings, you get the epoch value when the

operation started

– The object was Q1 which resolved to a local queue named Q1.

– The MQGET was under object handle 2


• MQGMO_BROWSE_FIRST & MQGMO_CONVERT

– But it failed, MQRC_NO_MSG_AVAILABLE

– And the API call took 17ms to complete on the queue manager

MQI Operation: 2

Operation Id: MQXF_GET

ApplicationTid: 1

OperationDate: '2016-02-18'

OperationTime: '11:18:50'


Completion Code: MQCC_FAILED

Reason Code: 2033

Hobj: 2

Get Options: 16400

Msg length: 0

High Res Time: 1455794330745687

Object_type: MQOT_Q

Object_name: ‘Q1'

Object_Q_mgr_name: ''

Resolved_Q_Name: ‘Q1'

Resolved_Q_mgr: 'MQ80'

Resolved_local_Q_name: ‘Q1'

Resolved_local_Q_mgr: 'MQ80'


Buffer Length: 102400

Performance Impact

• Only trace the applications you’re interested in

• Set the TraceLevel at the lowest level which provides the information

you require

• Tune ActivityInterval and ActivityCount, experiment

– Less frequent messages will use more memory

– More frequent messages will use more IO

• Frequently consume messages from the queue to avoid queue full

When should I use application activity trace?

• Scoped to application

• Application audit trail

• Performance analysis

– How long are my API calls taking to complete on the queue manager?

• Resource usage

– Which queues or topics are actually being used

• Problem Determination

– Which Queue / Queue Manager is the application

actually opening

• Application Coding Standards

– Does everyone use the MQI in the recommended way

• And more …

Summary

Summary

Accounting

messages

API Trace Application Activity

Trace

Admin can enable X X X

Developer can enable X X

Documented format X X

Useful for performance X X

Scoped to applications X X

Questions?

Notices and Disclaimers

37

Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission

from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial

publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED

"AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS

INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and

services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers

have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in

which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and

discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their

specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and

interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such

laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

Notices and Disclaimers Con’t.

38

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not

tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.

Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the

ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT

NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual

property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,

FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,

Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,

PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,

StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business

Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM

trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

http://www.ibm.com/legal/copytrade.shtml

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee

Portal to complete your session surveys from your

smartphone,

laptop or conference kiosk.

Monday

10:30-11:30 3592 New MQ features

3452 Managing applications

12:00-13:00 2835 MQ on z/OS and Distributed

15:00-16:00 3470 Latest MQ z/OS features

2833 Where is my message?

3544 MQ Light in an MQ infrastructure

16:30-17:30 3573 Hybrid cloud messaging

2941 MQ Advanced

Tuesday

08:30-09:30 3540 The MQ Light API

12:00-13:00 3456 The IBM MQ Appliance

13:15-14:15 3499 Introducing Message Hub

3458 MQ Appliance administration

14:30-15:30 6432 MQ updates and futures (InnerCircle)

2849 Messaging feedback roundtable

16:00-17:00 3544 MQ Light in an MQ infrastructure

3513 MQ hands on lab

Wednesday

08:30-09:30 3602 Managing your MQ environment

12:00-13:00 3613 Designing MQ self service

6408 Hybrid messaging roadmap (InnerCircle)

13:15-14:00 3416 HA and DR with MQ

3433 Why secure your messaging?

15:45-16:30 3429 Securing MQ

2847 Meet the messaging experts

16:00-17:00 3508 MQ Light hands on lab

16:45-17:30 2275 Migrating to the IBM MQ Appliance

Thursday

08:30-09:15 3420 MQ Clustering

2931 Business agility with self service MQ

09:30-10:15 3479 MQ z/OS clusters and shared queue

3450 Optimising MQ applications

2849 Messaging feedback roundtable

10:30-11:15 3465 MQ Appliance high availability

3481 MQ z/OS messaging connectivity

11:30-12:15 3474 Active-active messaging

3537 Monitoring and managing MQ

3425 MQ publish/subscribe

Find us at the EXPO:

Hybrid Integration peds 65-68

Check out the Hybrid Messaging sub topic under the

Hybrid Integration topic for further customer and business

partner sessions

Hybrid Messaging from the IBM experts at InterConnect 2016

Sunday

14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)

3452 - Managing your applications

Software

Transcript of 3452 - Managing your applications