2005 Metastorm, Inc. Powering Enterprise Process Advantage ® Metastorm PM4Data Managed File...

2005 Metastorm, Inc.

Powering Enterprise Process Advantage®

MetastormPM4Data

Managed File Transfer

Chris Howarth [email protected]



2

WebSphere Business IntegrationReference Architecture

Infrastructure Services

Application and Data Access Services

Business Application and Data Services

Business Performance Management Services

Business Application

Services

ProcessServices

Information Services

Development Platform

Enterprise Applications and Data

Interaction Services

Partner Services

ESBWMQWBIMB PM4DATA and PM4CICSWMQWBIC



3

B2B Interactions

PM4Data is the File and Database Adapter for the Enterprise Service Bus

Data

ServiceFlow

ExistingApplications

NewService Logic

Portal Service

SOAPService Request(e.g. J2EE, .NET)

Utilize MQ Messaging to access and

provide managed

transfer facilities for Files and Databases

XML enable Files,

Databases WMQ & XML

Onramp



4

What is PM4Data?

• PM4Data is:– A high-performance, highly scalable solution for “managed”

high volume data transfer of almost any type of data between two or more distributed heterogeneous systems (Managed File Transfer Solution).

– A solution that leverages the award winning IBM transactional MOM (WebSphere MQ ®) for assured, high performance, reliable and secured asynchronous delivery of data.

– A extendible, customizable, reliable and scalable foundation for a modern visible process-centric WMQ based data transfer & integration solution.

– A powerful WBI companion and WMQ implementation accelerant tool

– A WebSphere MQ and XML Power Tool.



5

Supported Platforms• Mainframe z/OS (running OS/390 or z/OS)• Unix

– AIX– HP-UX– Sun Solaris– Linux

• OS/400• Windows

– NT, 2000, XP, 2003• Tandem NSK• IBM 4690 Point of Sale System

– Optional “Continuous Extract Facility” for TLOG Trickle Feed• Both WebSphere MQ Client or Server supported



6

Features and Functions SummaryHigh performance, secured, bulk data transfer solution that delivers real-time and batch File

or Database Integration capabilities.

Usability

• Eclipse® based Workbench IDE for developing and testing multiplatform integration Scripts & XML Policies

• Centralized management

• Built-in Status and Statistics Tracking

• Common multi-platform scripting language

• Web based Command/Control Center and Status

• User profile and access control

• File and Database Integration Service

• File Transfer request repository

• Interactive retry / fail selection of failed transfers

• Performance analysis reports and graphs

• Native multi-platform Interface

•TSO, JCL, Command, Message, Shell script, bat/cmd

• Directory Monitoring: Sense creation/arrival of file to initiate a file transfer

Features

•Customizable process flows for end to end status tracking.

• Accelerates file and message based integration.

•XML-enabler (Files and Databases).

•XML Control interface.

•Distribution list for one-to-many transfers.

• Transfer list support for many-to-many transfers.

• Service-oriented architecture.

• Accelerates integration with IBM WMQ Broker

• Logical record processing

•SSL based data encryption (WMQ SSL)

•Policy driven I/O:

• File or Database to Message

• Message to File or Database

• Simple file extract, transfer, and load capabilities.

• Highly scalable, both in volume and platform capabilities

• Automated exception handling facility



7

– Central Registry• Management and configuration artifacts

– MQ Dashboard• Easily Monitor Enterprise MQ load and Usage

• Daily Summary reports

– Process Flow Editor to create customized Process Flows

– JMX based Systems Management infrastructure

• JMX (Java Management Extensions) infrastructure and support

– Web Browser Component Launcher Console

• LOG4J (STDOUT / STDERR) log management • Email notifications for exception• Alert Management integration to Tivoli Console

PM4Data new features List in V8



8

Managed Data Movement

Simplified Data Movement Scenario

Target platformTarget

platform

Far more sophistication, reliability and performance than putting a data in a single message

Platform file differences handled automatically during transfer Extended functionality with customizable exits End-to-end data transfer visibility

Far more sophistication, reliability and performance than putting a data in a single message

Platform file differences handled automatically during transfer Extended functionality with customizable exits End-to-end data transfer visibility

SourcePlatform

Data broken into messages

End-to-end detailed transfer status

Messages received andassembled

Message Message Message



9

PM4Data Architecture

PM4Data Node

ApacheWeb Server

MQSeries Bus

Status Query Server

ProcessMonitor

Status Logger

Status Database

File Transfer Service

File Transfer ServiceFile Transfer

ServiceFile Transfer

Service

Exit Processor

User Database

User Service

Coordinator

Registry File Store

Registry Service

Directory Monitor Service



10

PM4DATAFiles, Databases and Queue Integration Simple Example

• Example– Source Data needs to be moved to one or more targets

– This can be caused by application updates and sensed by PM4DATA

– End-to-end visibility with Status and Notification

oror

DBDB

FileFile

Queue

Data transferrequest

Data transferrequest

PM4DataPM4Data

PM4DataPM4DataNew or AppendfileNew or Appendfile

Databaseupdate

Databaseupdate

oror

oror

Queue

PM4DataPM4DataNew fileNew file

Databaseupdate

Databaseupdate

oror

oror

Queue



13

Process Flow Editor

Exception HandlerBusiness Provides the

“What needs to be done”

RoutingIT provides the “How it

gets done”

File Transferas a sub Flow

Activity



14

– Central HFS based Artifact Registries• Eclipse Workbench IDE

– Web Browser access for transfer request / policies

• Version and Change Control Support (CVS or similar product)

• Customized or template based Node configuration support• Support for “Application” or “Project” level Registry• Support for MQ / TCP and File based registry drivers• Support for Registry chaining and search order control

<RegistryList> <registry location="C:\CQDemos\data\Registry">Emergency</registry> <registry location="C:\CQDemos\data\Registry">Application1</registry> <registry location="C:\CQDemos\data\Registry"> Application2 </registry> <registry location="C:\CQDemos\data\Registry"> Application3 </registry> <registry location="C:\CQ\data\Registry">CommerceQuest</registry> </RegistryList>

Process

XML

Application1

Request

Group2

Policies

Programs

XMScript

Messages

Group1

Search Order

Physical Location

“Registry Name”Typically used to group artifacts for

a project or “Application”

Registry

Contents

PM4Data Registry



16

Transfer Process Monitor• Web Server support

for:• WebSphere,

JBOSS (included) – Status– Component

Configuration / Management (JMX)

– Component Logs (LOG4J)

• Status display– End-to-End

transfer status• Access Control

– User management, with roles & rights

• Transfer request management

– Ping nodes– Create, save, and

submit transfer requests

– Transfer grouping– Distribution List

Management– Directory

monitoring



19

Conclusion

• Satisfies Enterprise Managed File Transfer requirements• Strategic Component of the Enterprise Service Bus

Architecture• Accelerates the On Demand Strategy



20

Mapping XML to XMObject’sand

Vise Versa <order><order>

<customer><customer>

<name>I. Wannabuy</name><name>I. Wannabuy</name>

<address> … </address><address> … </address>

</customer></customer>

<item><item>

<description>Small Rodent, <description>Small Rodent, Generic</description>Generic</description>

<quantity>35</quantity><quantity>35</quantity>

<price>29.99</price><price>29.99</price>

</item></item>

<item><item>

<description>Catapult</description><description>Catapult</description>

<quantity>1</quantity><quantity>1</quantity>

<price>149.95</price><price>149.95</price>

</item></item>

</order></order>

name

customer

item

item[1]

order

item[0]

…

address

description

description

quantity

quantity

price

price

• XMOBJECT



21

Things that are important in XMLwhich XMObject takes care off• Well defined order semantics

– What is property order for new object?– Where are new properties added?– What is impact of deleting properties?

• Operators for controlling order– Specify property order– Modify property order– Preserve property order

• Operators for creating and manipulating additional XML artifacts

– Attribute– Namespaces, Mixed content

• Specialized methods for managing Objects.– Creating, Serializing, Cursor-ring, Searching, Printing– Renaming, Copying, Allocating and de-allocating

memory– Encoding, Compressing, Code Page conversions

etc.

name

customer

item

order

item[1]

…

address

description

quantity

price

Type = “This order is …

id = “342”

custid = “007”

item[2]



22

XMObject’s Objective

• Provide a simple, familiar, general purpose programming model for XML that:– Leverages existing skills and knowledge– Reuses familiar concepts, operators and syntax– Flattens the learning curve– Minimizes need for specialized skills and knowledge– Enables developers immediately with little or no training

• Ultimately, provide a simple object abstraction for creating, navigating and manipulating XML



Metastorm

DB XMObjects



24

<?xml version='1.0'?> <Account> <CustomerID>123451</CustomerID> <ContactFirstName>John</ContactFirstName> <ContactLastName>Smith</ContactLastName> <BillingAddress>123 Front Street</BillingAddress> <City>Toronto</City> <StateOrProvince>Ontario</StateOrProvince></Account>

Supplier

Account

Product

PaymentOrder

Customer

SQLServer

Supplier

Account

Product

Payment

OrderCustomer

IMSDB2

The Data format dilemma:The Fix = DB Objects

DBObject

PaymentProductAccount

PaymentProduct

Account

Customer

DBObject

Z/OS

SupplierProduct

Customer

DBObject

NCR

DBObject

Warehouse

IDMS

DBObject

ODBC

Oracle

DBObject

AS/400

DBObject

DBObject



25

DB Objects Database Wrapper• DB Objects is a database independent XMObject Interface

• Language and Platform independent API support, including XMScript• Supports any ODBC accessible database• XMObjects or XML formatted messages for input and output

• Native database access support included for:• DB2 ( OS/400, Windows,Unix & OS/390)• All Popular RDBMS via Data Direct Drivers On Unix Systems• SQL Server (Windows)

• Allows for Database access via MQSeries network• Reduces or eliminates ODBC client connections

• Instantly XML and Message enables almost any database• Serves up data as XML from any database• Turns Stored Procedures into “Message Enabled Components”

• Full read, update, create, merge, commit, backout & insert support



MetastormProcess Manager for Data Architecture Overview



27

PM4Data Architecture

PM4Data Node

ApacheWeb Server

MQSeries Bus

Status Query Server

ProcessMonitor

Status Logger

Status Database

File Transfer Service

File Transfer ServiceFile Transfer

ServiceFile Transfer

Service

Exit Processor

User Database

User Service

Coordinator

Registry File Store

Registry Service

Directory Monitor Service



28

Process Flow Manager (Coordinator)

Request Message

1

Process Flow Definition

2

InitiatingApplication

Process FlowManager

BusinessProcess

DefinitionRepository

AsynchronousWorkflowEnabled

Component(response required)

AsynchronousWorkflowEnabled

Component(NO response req)

Synchronous In-lineWorkflow EnabledService ProgramOr Shared Library

Message

3

Response

4 Message

5Message

3

Response

4

Synchronous External Service, including WS, EJB

Message

Response

5

6

WBIMessageBroker

Status TrackingAudit Sub System



29

Dispatch Manger for the Coordinator

UNIX / Windows / AS400 / ZOS DispatchManager

ForCoordinator

CoordinatorInstance

Q DepthMonitorThread

WSD

L

XML

CoordinatorInstance

CoordinatorManagement

Thread


Thread

CoordinatorInstance


Thread

ConsoleManagement

ThreadConsole

WorkflowApplicationsWorkflow

ApplicationsWorkflowApplicationsPM

Applications

Incoming“XML” WMQ

Messages



30

PM4Data

Auditor Sub System

Process Flow(Coordinator)

Server

MQXML

StatusDatagram

MQXML

StatusDatagram

MQXML

StatusQuery Request

MQXML

StatusQuery Response

Control CenterJSP Application

ApacheApacheWeb ServerWeb Server

DB2

Status QueryStatus QueryServerServer

Status LoaderStatus Loader

UserComponents

UserComponents



Metastorm PM4Data Components



32

Coordinator Service

• Directs all file-transfer activity– Receives transfer request in XML format from the PM4Data

interface• Process Coordinator

• Command Line

• XML Interface

– Manages file transfers from end to end• Sends transfer request to sending File Transfer Service (FTS)

• Uses XML to direct file transfer processing

• Receives status information back from the receiving FTS

• At least one must exist in the enterprise– More than one can exist in the enterprise– More than one can exist on a host



33

File Transfer Service (FTS)

• Can send and receive files, based on direction from the Coordinator– A single instance of the FTS can both send and receive files

• Each host that performs transfers must have at least one FTS– The FTS must have access to the source and destination file

system

• Multiple instances of the FTS can run on one host– Typically, two instances of the FTS are placed on a host at

installation– More instances of FTS can be added to provide additional

throughput



34

Exit Service• Responsible for running PM4Data exits• Can be accessed from any exit point in the transfer process

– Receives XML message directing execution– Contains name of script and entry point in script– Returns result element to PM4Data

• Uses XMScript modules– User can easily add additional methods

• Exits run at any node– Sender, Receiver, Process Coordinator– Pre-process, Post-process

• Multiple exits at any exit point– Exits can be chained



35

Auditor Service – Status Query Server• Provides interface between the GUI interface and the status

database– Process Monitor – web-based– Explorer – MMC-based

• Obtains status information for display



36

Auditor Service – Status Logger

• Provides interface between each component that reports status and the status database– Receives status information from the following components:

• Process Coordinator

• Sending FTS

• Receiving FTS

• Exit Service

– Puts header and detail status information in the status database



37

User Service

• Provides interface between the Process Monitor web interface and the user database

• Manages user accounts for the web interface– Govern access to create and submit transfers and view status

• Typically only used in planning and administration



38

Registry Service

• Stores transfer-related artifacts created in the Process Monitor– Transfer requests– Destination entries– Destination lists– Directory Monitor entries– Policies Object– Scripts– Process Flows– Message Object– Meta Data



Metastorm Policy based I/O



40

Why use policy based I/O?

Issues Solutions BenefitsIntegration requires custom “glue” code at the end points.

Describe end points or data stores with a policy.

Improve time to market by replacing coding with configuration.

Files are “familiar” and non-transactional, while databases, queues or other data targets require transactional properties for integration.

Use file metaphor for all data targets including databases, queues, TCP/IP sockets, HTTP protocol.

Bridge file based “legacy” applications with queue, database, or other data stores applications.

File based integration requires costly operational support.

A managed file transfer with flexibility for non-file based data stores.

Simplifies development, without burdening operations.



41

2005 Metastorm, Inc. Powering Enterprise Process Advantage ® Metastorm PM4Data Managed File...

Documents

Transcript of 2005 Metastorm, Inc. Powering Enterprise Process Advantage ® Metastorm PM4Data Managed File...