Data Transformation Engine Intelligent Business Integration Reference Guide

HP NonStop Data Transformation Engine

Intelligent Business Integration Reference Guide Abstract

This document details the components of the HP NonStop™ Data Transformation Engine (NonStop DTE) and Mercator product architecture.

Product Version

NonStop Data Transformation Engine 6.7.1

Version Updates (RVUs)

N/A

Part Number Published

528258-002 October 2004

Document History

Part Number

Product Version Published

528258-001 NonStop Data Transformation Engine 6.7.1

June 2004

528258-002 NonStop Data Transformation Engine 6.7.1

October 2004

I n t e l l i g e n t B u s i n e s s I n t e g r a t i o n R e f e r e n c e G u i d e

3

Contents About This Document

Related References........................................................................................... 5

Chapter 1 - Introduction

Mercator Technology......................................................................................... 7 Design Environment – Design Studio....................................................................... 8

Type Designer ............................................................................................ 9 Map Designer ............................................................................................ 13 Integration Flow Designer............................................................................. 17 Database Interface Designer.......................................................................... 20 Source Control .......................................................................................... 22

Chapter 2 - Integration Maps

Anatomy of a Mercator Map ...............................................................................24 Input and Output Cards................................................................................ 25 Transformation, Routing, and Business Logic Rules .............................................. 25 Adapter Configuration for Input and Output....................................................... 26

Map Behavior During Execution ...........................................................................28 Input and Output Card Settings ...................................................................... 29 Map Settings ............................................................................................. 33 Resource Registry....................................................................................... 35

Chapter 3 - Methods of Execution

Servlet Execution Model....................................................................................37 Servlet Integrator....................................................................................... 38

Enterprise JavaBean Execution Model ...................................................................39 Command-Driven Execution Model .......................................................................40

Execution Command Overriding a Database Target Example................................... 41 Event-Driven Execution Model.............................................................................42

Event Server Architecture............................................................................. 42 Event Server System Example ........................................................................ 44 Audit and Transaction Logging Example............................................................ 46

Application-Embedded Execution Model ................................................................51

Chapter 4 - Developing Custom Adapters

XML Descriptor File..........................................................................................53

Chapter 5 - Operation Management & Monitoring Capabilities

Management Console........................................................................................56 Event Server Administration.......................................................................... 59

SNMP Support.................................................................................................60

Chapter 6 - High-Availability and Load-Balancing Architecture

Server Clustering ............................................................................................61 Enterprise JavaBeans .......................................................................................63

Contents


4

Chapter 7 - Securing HTTP Communications

What is SSL?...................................................................................................64 Encryption ............................................................................................... 64 Digital Signatures and Certificates .................................................................. 65

Implementing an SSL Environment .......................................................................65 Event Agent.............................................................................................. 65

Chapter 8 - Web Services

What is a Web Service? .....................................................................................67 Why Use Mercator Products to Implement Web Services?....................................... 67

Technologies Used in Web Services ......................................................................68 SOAP ...................................................................................................... 68 WSDL...................................................................................................... 68 UDDI....................................................................................................... 69

Web Services Architecture.................................................................................70 Consumer Architecture ................................................................................ 71 Provider Architecture.................................................................................. 72

Web Services Example ......................................................................................73

Index


5

About This Document This document details the components of Mercator’s product architecture.

This document also details Mercator’s transformation, connectivity, flexibility, and scalability capabilities, demonstrating how Mercator products can be the robust backbone of your critical business operations.

Related References The Mercator Online Library contains information about all products. Refer to the following references for more information about topics frequently discussed in this document.

Related Reference Description

Command Server Reference Guide Using the Command Server to execute maps on various operating system platforms.

Event Agent Reference Guide Using the Event Agent to securely send external HTTP requests to the Event Server.

Event Server Reference Guide Using the Event Server to help automate the execution of your systems of maps and the Management Console to remotely monitor systems running.

Integration Flow Designer Reference Guide

Using the Integration Flow Designer as a Design Studio companion and graphical facility to manage collections of related maps. Also graphically organizing these maps, based upon your requirements, into logical collections called systems.

Introduction to the Design Studio Introduction of the concepts, terminology, interfaces, and functionality of the Design Studio client components. A companion book to the Design Studio Tutorial.

About This Document Related References


6

Related Reference Description

Map Designer Reference Guide The Map Designer user interface and instruction on specifying mapping rules, configuring map sources and targets, defining map-level settings, and executing a compiled map.

Mercator Integration for Java Reference Guide

How to use the components included with the Mercator Integration for Java product, which allow for server-independent map execution. The guide includes detailed instructions for running the Servlet Integrator. (SDK Online Library only)

Mercator Programming Interface The Mercator Programming Interface offers an object-oriented approach that enables applications to invoke Mercator maps, thus removing reliance on command lines. (SDK only)

Platform API Reference Guide Using a platform-specific API with your application to tightly integrate the Enterprise Broker with your application. (SDK Online Library only)

Resource Adapters Reference Guide Using adapters as map sources and map targets in general. More specific information can be found in the respective reference guides for each adapter.

Resource Registry Reference Guide Using name aliases for source and target resources that are specified in map cards.

SNMP Agent Reference Guide Provides standardized network management framework for controlling and monitoring the internetwork, an interconnected system of computer networks.

Type Designer Reference Guide Using the Type Designer to create and edit type trees that describe your data.


7

Chapter 1 - Introduction

Mercator Technology Mercator products combine leading-edge technology for both external (B2B) and internal (A2A) application integration, powerful graphical technology for integrating application content, and management facilities that provide run-time control of both real-time and event-driven integration activities. They support the entire integration solution life cycle, beginning with graphical definition of integration workflows. They provide a feature-rich graphical environment for creating all integration interfaces without the need to resort to programming. A fully functional Software Development Kit (SDK) is available, providing the capability to create tightly coupled application interfaces and custom resource adapters as required.

Whatever type of integration model is being employed, the Mercator difference is that a single technology base is used for all integration scenarios.

Mercator products

Chapter 1 - Introduction Design Environment – Design Studio


8

Design Environment – Design Studio The Mercator Design Studio is the client interface into an extensive array of functionality that is available to create application integration solutions. Application integration solutions created with the Design Studio are executed on transformation servers specific to the Mercator product being used. Separation of design and execution provides exceptional user flexibility in implementing enterprise-wide integration solutions. A single Design Studio can deploy integration solutions to multiple servers, or multiple Design Studios can be used to create integration solutions for a single production environment.

The Mercator Design Studio consists of multiple user-friendly, GUI-based applications that run on the Microsoft Windows platforms. These applications include designers for integration flows, data object definitions, data transformation maps, intelligent routing rules, and database interface definitions.

The major components of the Mercator Design Studio include:

♦ Type Designer

♦ Type Importers

♦ Type Tree Maker

♦ Map Designer

♦ Integration Flow Designer

♦ Database Interface Designer



9

Type Designer

The Type Designer is a facility for creating, maintaining, and viewing data object definitions in a graphical form called a type tree. The type tree provides visual cues for understanding the structure of an object and supports point-and-click techniques enabling you to “drill down” to uncover an object’s complete definition. The following is an example of a type tree:

A type tree is a repository of one or more data object definitions. A data object definition, called a type, describes data object structures in terms of their syntax, composition, existence, and validity. A composite type, called a group, describes sequences and choices of other types. An elementary type, called an item, represents a basic building block such as time, date, numbers, and text. The trees are arranged in a classification hierarchy to facilitate reuse of type definitions in a subtype.



10

Type trees can be created in a number of ways:

♦ The Type Designer includes an importer tool for automatically generating type trees for data described in XML, COBOL, SAP R/3, Tuxedo, and PeopleSoft formats or application data structures.

♦ The Type Designer enables creation of type trees “from scratch” as well as modification and customization of existing type trees; including those generated by importers, DTD, or provided by Mercator. For example, the EDI type trees provided with the Mercator Integration Broker can be customized to reflect a trading partner agreement.

♦ Mercator provides type trees for industry-standard formats such as S.W.I.F.T., FIX, X12 EDI, EDIFACT, ACORD, Tradacoms, and Odette.

♦ The Database Interface Designer provides a feature that establishes a session with any relational database management system and automatically imports definitions from the database catalogs.

Importers define a transformation between well-known metadata formats such as:

! Extensible Markup Language (XML) document type definitions (DTDs)

! Extensible Markup Language (XML) schemas

! Web Services Description Language (WSDL) web service definitions

! OAG Business Object Documents (BODs)

! COM type libraries

! COBOL Copybooks

! Java classes

! Interface definitions for packaged applications such as SAP, PeopleSoft, and Siebel.



11

The type definitions in a type tree can be exported to a flat file in XML format for manipulation by XML and text-based tools. The modified trees can then be imported back into the Type Designer. Mercator provides the DTDs for the importing and exporting of type trees. Below is a portion of a document file generated by exporting a type tree:

Note Refer to the Type Designer Reference Guide for more information.



12

Type Tree Differences

The type tree differences feature enables the user to compare two type tree files. When the analysis is complete, both trees display. Source compare differences will appear with either blue or red text. When a type is present in one tree but not present in the other tree, the text will be blue. When a type is present in both trees but have different properties, components, or restrictions, the text will be red. There is a “step-through” facility that cycles through each of the differences in turn. In extremely large, complex type trees, this can be a real time saver.



13

Map Designer

The Map Designer is used to create maps, which consist of rules for data content, transformation, and intelligent routing. This process is facilitated by convenient “from” and “to” windows, drag-and-drop techniques, and spreadsheet-like rules. Mapping rules are added using logical statements. The Map Designer provides a rich set of over 100 pre-defined functions for operations such as conditional testing, table lookups, mathematical functions, character string parsing, and data extraction. Rules and functions can be combined and nested arbitrarily.

The capability for mapping from multiple inputs to multiple outputs gives Mercator products unique transformation power. Each input to and output from a map is associated with a specific data object definition and a resource adapter. Data object definitions are reusable; the same definition can apply to one or more inputs, to both input and output for the same map, or to different maps. When a data object is used as input, data is validated to ensure the data conforms to the data object definition. When used as output, the data object is constructed automatically, in its entirety.



14

Input objects are not consumed when used in a transformation rule. This allows for a one-to-many relationship between an input and its outputs. An output can be constructed from any number of zero or more inputs. This creates a one-to-many relationship between an output and its inputs. Taken together, Mercator products provide many-to-many transformation support between inputs and outputs.

Data object definitions, data maps, and resource adapters are all separately managed objects. Each object is reusable in any number of integration scenarios. The result is flexibility, ease of use, and reduced maintenance as integration requirements change.

The map definition is completely portable between all server platforms. Simply select the target platform from a list when building the map to generate a compiled map file that is optimized for the specified platform.

Note Refer to Chapter 2 - Integration Maps for more information on the anatomy and execution characteristics of maps.



15

As with the Type Designer, the Map Designer is able to export its rules and execution settings to a flat file in XML format.

Note Refer to the Map Designer Reference Guide for more information.



16

Map Source File Differences

As with the Type Designer, the Map Designer also has a graphical source compare feature that enables you to compare two map source files.



17

Integration Flow Designer

The Integration Flow Designer (IFD) is a graphical manager for data flows. It is used during development to define a set of logically related integration solutions called a system. The IFD defines interactions among maps and systems of maps, where maps are objects that serve one or more purposes such as routing or transformation.

The IFD provides clarity and control for managing the more complex systems built with Mercator products. It also reduces the administrative burden and simplifies the non-development activities associated with interface development. Some of the key features of the IFD are:

♦ Graphical data flow design

♦ Analysis of integration for logical consistency



18

♦ Build and port entire systems with simple mouse-clicks

Note Refer to the Integration Flow Designer Reference Guide for more information.

Graphical Design of Data Flow

Visual data flows are created to aid in the design of integration solutions. A graphical palette enables you to produce system definition diagrams and navigate easily among the Mercator Design Studio tools. Point-and-click techniques are used to select which map and subsystem components to place in a specific integration flow diagram.

Using the IFD, you can also define specific map settings that take effect at runtime. Simply select the desired maps and the data flow between them is automatically determined from the inputs and outputs. The Override tool in the IFD toolbox can be used to modify the inputs and outputs to achieve the desired data flows. These modifications do not affect the source map, which allows the same map to be reused in multiple data flows.

Logical Analysis

The IFD includes an Analyzer that checks your system definitions for logical consistency to ensure they can be executed. The Analyzer detects any inconsistencies you may have introduced in the definition while experimenting with system models.

Configuration Management & Automated System Deployment

Once a system is designed, the IFD offers another benefit—saving time in preparing a system for operation. One mouse-click builds and deploys all of the integration objects and supporting data (such as lookup files and scripts).



19

Parameters for different servers representing different operating environments can be defined within the IFD.

A server can then be assigned to a particular system of maps, therefore becoming the “active” server for this system.



20

After the appropriate server definitions have been created, it is then possible to produce, using the wizard, automated deployment definitions that are specific to each server.

It is possible to specify within a script, that all maps are built and transferred, or only a specific subset of the maps in a system. Additional files required to support the system’s operation can also be included in the transfer.

Database Interface Designer

Databases can be both sources of input data and destinations for output data for a map. The Database Interface Designer is used to import metadata about queries, tables, and stored procedures for data stored in relational databases. It is also used to identify characteristics of those objects to meet mapping and execution requirements, such as update keys and database triggers.



21

The Database Interface Designer makes it possible to:

♦ Specify databases, user IDs, passwords to connect to as sources and destinations

♦ Define query statements

♦ Generate type trees automatically for queries, stored procedure input and output parameters or tables

♦ Define specific columns to update for update transactions

♦ Define tables for which inserts, updates, and deletions will trigger maps via the Mercator Event Server

Note Refer to the Database Interface Designer Reference Guide for more information.



22

Source Control

All of the Mercator Design Studio components provide a direct interface to any source control package that supports the Microsoft Source Code Control Interface API. This means that source components in all of the designers can be directly managed within a source control package (such as Merant PVCS, Rational ClearCase, and Microsoft Visual SourceSafe).

This feature enables you to save and retrieve multiple revisions of the source components developed with the Mercator Design Studio to or from a source control system, avoiding manual backup of original source files.

Note Refer to the Source Control Reference Guide for more information.


23

Chapter 2 - Integration Maps The core components for any Mercator product-based application integration solution are the maps. Maps are at work whether implementing:

♦ An asynchronous, event driven architecture

♦ A component or Enterprise JavaBean (EJB) architecture

♦ A synchronous, web-based architecture

♦ An application-embedded architecture

♦ A batch interface architecture

Maps provide a non-programmatic approach to the development of interfaces, with pre-built functionality for connectivity, transformation, and routing. The benefits of the same components prevailing for all integration models are that:

♦ Components can be created once and re-used many times for different types of integration scenarios.

♦ The skills required to deploy these different models are consistent and re-usable.

Chapter 2 - Integration Maps Anatomy of a Mercator Map


24

Anatomy of a Mercator Map

A map is comprised of the following components:

♦ Input and output map cards that represent the sources and targets.

♦ Map rules that define how the output data is built based on business rules.

♦ Specification of resource adapter used to retrieve and send data.



25

Input and Output Cards

Each input and output is represented in a separate map card. The data structure for each of the inputs and outputs is represented in the form of a type tree created using the Type Designer.

Input cards 1 and 2 Output card 1

Transformation, Routing, and Business Logic Rules

Type trees are completely interchangeable between inputs and outputs, and the same type tree can be reused multiple times for as many different maps as required. After the output card(s) for a map has been associated with a type in a type tree in the Map Designer, a rule cell for each object within the output card object is available. It is in these cells that rules are constructed, defining how the output data will be built. The components of map rules are color-coded to assist in reading them. The following is an example of a map rule:



26

Adapter Configuration for Input and Output

Mercator resource adapters provide the connectivity needed to GET data from sources and PUT data to targets. However, adapters do not provide transformation; that is the purpose of map rules. Map rules describe the nature of the transformation while the transformation server performs the actual transformation of the data.

Mercator provides adapters for common enterprise-level resources. In its simplest form, specifying an adapter for an input or output card is merely a case of selecting the required adapter from a list:

Select the required adapter from the drop down list.

When Database is selected as the adapter, select the .MDQ file created using the Database Interface Designer that contains the appropriate database connectivity and authentication.



27

Note Adapter settings defined in the map can be overridden with different adapter settings at map runtime.

The resource adapters are loosely coupled with the transformation process. An adapter retrieves data to be processed by the map as a byte stream. This means that to the extent that the type definitions and byte streams are compatible, type trees and adapters can be used interchangeably.

For example, when testing, it may be quicker to debug problems with the map’s logic using a file. However, for production, the data source may be from a database. As long as the format of the data retrieved from the database is consistent with the format of the data in the file that was used for development and debugging, the only change that has to be made to the map is to change the relevant input(s) from using the File adapter to using the Database adapter.

Furthermore, the map source file need not be changed at all, as the map’s execution settings, including the adapter configuration settings, can be overridden at runtime.

Where possible, resource adapters utilize the native client libraries of the resources concerned for connecting to those resources. It is through this mechanism that Mercator products can achieve “direct” connectivity to those resources. For example, the Mercator Oracle adapter uses the Oracle Net8 client to achieve connectivity to Oracle databases.

Mercator provides an open API to allow for development of a user-defined adapter, should the need arise. This allows connectivity to unforeseen, possibly proprietary systems (such as a “home-grown” HR system), in addition to the standards-based adapters that Mercator provides.

Note For more information, see Application-Embedded Execution Model.

Chapter 2 - Integration Maps Map Behavior During Execution


28

Map Behavior During Execution The execution characteristics of a map are determined by the execution parameters that are configured at design time. However, these map execution characteristics or map settings can also be overridden at run time.

The behavior of a single map invocation is as follows:

1 The adapters configured for their respective input cards retrieve some of the input data.

The input data is validated against the respective input cards’ type tree definition. This ensures that only valid data is passed to the output building process. It is for this reason that any piece of any of the input data can be used to build any of the output data. All of the input data is validated and stored in the map’s workspace before any output data is built. Because the entire workspace is available to the map when it comes time to build the output, any portion of the input data can be used, counted, or manipulated to generate any part of the output data.

2 After the input data has been validated, output objects can be used as inputs for a subsequent output.

It is possible to create temporary outputs to signify map logic in certain situations. The Sink adapter can be employed to ensure that the data is not retained.

By default, the rollback behavior is applied at the entire map thread level. Therefore, nothing will be committed to output or deleted from input until all inputs have been successfully validated and all outputs have been written successfully. A variety of commit and rollback options are available within the configuration settings of each input and output card. By manipulating these settings, it is possible to change the map’s default commit and rollback behavior significantly.



29

Input and Output Card Settings

Card settings define the data object the card represents, and how the data is retrieved or routed, and compiled into the map. The behavior of input and output cards configured in the compiled map can be overridden.

Schema Setting

This setting defines which type tree definition is to be used to represent the data source for an input card and the output data structure for an output card.



30

SourceRule Settings

SourceRule settings in the input card specify how and what data to retrieve, where to get it, how much to get, and what to do when an error occurs.

The Source card setting specifies the adapter source of the input data. The Command card setting enables additional adapter-specific parameters to be entered (such as the queue manager name as when using the WebSphere MQ adapter, queue name, and other parameters used to control adapter behavior).

The OnSuccess and OnFailure card settings, which work in conjunction with the Scope setting, specify what the map does with the input data when a map, burst, or card succeeds or fails.

The Retry card setting specifies action to take if adapter errors occur. For example, if Retry > Switch = On and the source connection fails, the adapter process is retried at the interval specified with the Interval value up to as many times as defined with the MaxAttempts setting.



31

FetchAs Setting

This setting applies to input cards only. It dictates how the adapter retrieves the input data. By default, this setting is Integral. This means that all of the input data is retrieved once from the source.

Setting this parameter to Burst means that the adapter will retrieve the input data in increments specified in the FetchUnit parameter. The units of data correspond to the highest-level repeating object in the type tree. For example, a payroll file contains 105 records. Using Burst mode with the FetchUnit parameter set to 10, data will be sent in 11 “chunks”. The first ten chunks will contain ten records each with the eleventh chunk containing the remaining five records.

Despite the fact that bursts process some of the map data and multiple sets of outputs are built, transactional integrity is not sacrificed. If desired, the Scope card setting can be set to Map to prevent any changes from being committed until all bursts have finished successfully.



32

There are a number of potential benefits in using Burst mode:

♦ A single map consumes a large quantity of input data, but only a small portion of that data is needed to build an individual output. Burst mode can be used to minimize the size of the workspace (an internal index of types present in the data stream). In some cases, a smaller workspace can increase map performance.

♦ Coordinating multiple inputs together. For example, the first unit of data for Input Card 1 must be coordinated with the first unit of data for Input Card 2, and so forth. If Burst mode is not enabled, one input must then act as a “driver” and matching outputs must be searched for.

♦ Processing a single logical unit of work at a time. For example, one row from a database table at a time or one message from a message queue at a time. This would equate to the Burst mode being enabled with the FetchUnit parameter set to 1.

TargetRule Settings

TargetRule settings in the output card specify what data is output, how the output data is routed, and what to do when an error occurs.

The Target card setting specifies the adapter target for the data produced by the map. The Command card setting enables additional adapter-specific parameters to be entered (such as the queue manager name as when using the WebSphere MQ adapter, queue name, and other parameters used to control adapter behavior).

The OnSuccess and OnFailure card settings, which work in conjunction with the Scope setting, specify what the map does with the output data when a map, burst or card succeeds or fails.

The Retry card setting specifies action to take if adapter errors occur. For example, if Retry > Switch = On and the source connection fails, the adapter process is retried at the interval specified with the Interval value up to as many times as defined with the MaxAttempts setting.



33

Map Settings

The previous section outlined the settings that apply to individual input and output cards. However, there are also map settings that dictate the behavior of the entire map, across all input and output cards.

The following map settings provide the ability to control aspects of the map’s execution such as:

♦ MapAudit: specifies options for creating an audit of the map’s execution, data, and execution settings, and to what level of detail

♦ MapTrace: specifies options for creating a trace, which can assist in troubleshooting during development, and to what level of detail

♦ WorkSpace: specifies the location (file or memory) of the workspace and the amount of memory to allocate for the map’s execution

♦ Retry: specifies whether to try again, and the interval between each retry, if compiled map files, work files, audit log files, or trace files that are needed during map execution cannot be opened



34

The MapAudit setting used in combination with the WorkSpace setting determines whether different threads of the same map component will run concurrently (multi-threaded) or sequentially (single-threaded).

It is suggested to create a trace only during development as performance can be adversely affected.

The WorkSpace settings specify memory page settings enabling you to optimize the speed of map processing.

The Retry settings specify whether to try again if resources that are needed during map execution cannot be

d



35

Note As mentioned in the Adapter Configuration for Input and Output section, virtually all of the source, target, and map setting parameters can be overridden with different values at runtime. This capability is available for all modes of map execution. This enables the ability to change map execution parameters for different deployment environments, sources and destinations (such as development, user acceptance testing (UAT), and production environments) without having to modify and recompile the maps.

Resource Registry

The Resource Registry is an application that provides a resource name alias repository, which can be used to abstract various parameter settings using aliases that resolve to specific resources within the enterprise. The premise behind the Resource Registry is that it permits generic resource variables to be used within Mercator components. These resource variables are then automatically resolved to the actual resources (such as directory paths, database names, message queue names, and server names). Resource variables make it possible to use the same map and system definitions in different deployment environments.



36

The Resource Registry provides a Java-based graphical user interface (GUI) for creating and managing these resource definitions.

Step 1: A resource name is created.

Step 3: The resource’s parameters aredefined for multiple deployment

environments. In this example, they arethe different locations of the map files

for each environment.

Step 2: A resource name is used to identify the deployment environment’s resources. The different values of the resource name are resolved at run-time by the map server.


37

Chapter 3 - Methods of Execution The core execution engine in Mercator products is developed in ANSI C for platform portability and high performance. However, there are a variety of mechanisms that can be used to invoke integration components, including a Java API that enables Java-based interfaces to maps. Other Java-based mechanisms for map execution are servlets and Enterprise JavaBeans (EJB).

The main methods for map execution are as follows:

♦ A Web-based execution model, where maps can be called and executed as a synchronous transaction by a servlet running on a Web server and servlet engine or an application server that has a servlet engine available. No programming is required (Java or otherwise) in this execution model.

♦ An EJB execution model, where maps can be “wrapped” inside an EJB and deployed to any EJB 1.0, EJB 1.1, or J2EE-compliant application server. Some Java programming is required to be able to call the EJB-wrapped maps running on the application server.

♦ Command-driven execution, primarily geared towards batch processing, using the Mercator Command Server.

♦ Event-driven execution, where systems of maps can be coordinated together by the Mercator Event Server to provide a scalable, distributed, multi-threaded execution environment.

♦ An application embedded model, where maps can be embedded in third-party applications using the Mercator Platform API, Mercator Programming Interface, or the Java API.

Servlet Execution Model Mercator offers a servlet-based execution model that enables a map to be invoked by a Java servlet. This is particularly useful for Web browser-based interfaces requiring a request and response mode of operation—the map is able to function in a stateful fashion due to the servlet’s ability to maintain state on the HTTP session. Using the Mercator’s Servlet Integrator, you can automatically generate the necessary servlet code that will interact with the corresponding map. The Servlet Integrator provides the ability to create browser-based request and response interfaces without having to write any procedural code at all.

Chapter 3 - Methods of Execution Servlet Execution Model


38

This approach has been successfully employed in a number of implementations, in particular, to provide the capability to Web-enable legacy applications. The powerful transformation and connectivity capabilities of Mercator products provide the application connectivity. Utilization of the automatically generated servlet exposes the power of this application integration capability to the Web.

The servlet generated by the Servlet Integrator utilizes the Mercator Java API, which in turn instantiates the maps via the native Mercator Platform API function libraries. In essence, the mechanism that is used is the same as for providing the runtime instructions for a map’s execution from the command prompt—overrides are submitted to control the map’s execution. The difference is that these overrides are supplied programmatically, in memory, rather than on the command line. However, even under Command Server execution, the overrides specified on the command line are passed by the Command Server, in memory, to the core execution code, which is embodied in the Platform API. Note that whichever execution model is employed, the same underlying Mercator technology prevails.

Servlet Integrator

The Servlet Integrator is a stand-alone, Java-based graphical wizard tool for generating Java servlets that enable you to expose maps to the Web using a third-party Web or application server that has a servlet engine available (such as Internet Information Server (IIS) or IBM WebSphere).

The Servlet Integrator enables you to:

♦ Select one or more maps to expose to the Internet (HTTP)

♦ Configure map-related options for the selected map(s); for example, retry, trace, and audit settings

♦ Generate a servlet to one or more target directories within the Web or application server and servlet space

♦ Execute the servlet using standard HTTP POST or GET syntax to invoke the corresponding map

Note Refer to the Mercator Integration for Java Reference Guide for more information.

Using the Servlet Integrator

The Servlet Integrator guides the user through the following process of tailoring and generating servlets to execute maps:

♦ Associating servlets with compiled map files

♦ Specifying details for compiling Java servlet class files

Chapter 3 - Methods of Execution Enterprise JavaBean Execution Model


39

♦ Overriding input and output map cards and sending and returning data using HTTP

♦ Enabling and overriding map runtime settings

♦ Generating and compiling servlets

The Servlet Integrator wizard provides you with step-by-step instructions on how to create, set up, and execute a Java servlet.

Enterprise JavaBean Execution Model Mercator provides an EJB API that enables maps to be executed by an Enterprise JavaBean containing business logic and deployed to an application server. Providing this capability enables maps to be tightly integrated into a Java component infrastructure, thereby leveraging all of the business logic (middle-tier) consolidation, flexibility, and load-balancing capabilities of EJB or J2EE application servers. While some Java programming is required to create the necessary Enterprise JavaBean (EJB) interfaces, Mercator provides packaged EJB API components that are “ready-to-go”, enabling maps to be deployed and run as EJBs.

The benefit of this approach is that all of the power of Mercator product’s standardized transformation and connectivity capabilities can be fully leveraged by anyone seeking to deploy an EJB-based architecture. Rather than needing to write Java code to perform complex transformation, connectivity, and routing, maps can be fully leveraged to perform these tasks. Again, as with other execution models, the same inherent Mercator technology components are used—maps. This means that a more unified skill set is applicable across all integration models, thereby reducing to a minimum, the overhead in writing the Java code to perform transformation, routing, and connectivity. This frees up more time to concentrate efforts on developing the business logic in the application layer code, resulting in a much quicker implementation. In addition, because the source components (type trees, map source files, and so on) are not procedural code listings, but rather visual, object-based components, the ongoing maintenance of these components is more straightforward; utilizing the extensive capabilities provided by the Mercator Design Studio.

EJB-wrapped maps can be invoked in a variety of ways. The most common method is by using another piece of coded EJB logic (often referred to as an “executive” EJB). It is also possible to invoke maps using servlets.

The EJB API consists of a stateful and stateless Session bean that wraps a map.

Note Refer to the Mercator Integration for Java Reference Guide for more information.

Chapter 3 - Methods of Execution Command-Driven Execution Model


40

Command-Driven Execution Model The Mercator Command Server provides a command-driven capability for the execution of integration maps. This model is normally associated with batch-driven, one-off integration processes. A variety of methods exist for executing maps in command mode such as:

♦ Execution and utility commands typed at the command prompt

♦ Execution commands invoked at the command prompt via a command parameter file

♦ From the Mercator Command Server, which provides a GUI enabling you to load and run maps, and also override map settings (Windows platform only)

♦ Execution commands defined in a batch file or shell script

♦ Execution commands defined in a scheduling program

As mentioned earlier, by default, the map’s execution settings (such as MapAudit and WorkSpace) are compiled into the runtime object. In all modes of execution, it is possible to override almost all of these settings at runtime. In the case of Command Server invocation, the overrides are supplied as parameters on the command line. For example:

/mercator/bin/mercator /mercator/maps/mymap.hp

Platform specific compiled map component

Command Server executable

Chapter 3 - Methods of Execution Command-Driven Execution Model


41

Execution Command Overriding a Database Target Example

Suppose you want to run the map multitran.hp, overriding some of the settings for the fourth output card, which is a database target.

Use the following command:

/mercator/bin/mercator /Mercator/maps/multitran.hp –G –OD4R5:3B ‘–TRACE ERROR’

Option Description

multitran.hp Specifies the name of the map to be executed.

-G All item properties (including size, presentation, and restrictions) are fully validated.

Overrides the default settings compiled into the map using the database target for the data of output card #4 (-OD4) as follows:

R5:3 If the database resource is unavailable, attempt to make the connection five more times at three-second intervals.

B If the map does not complete successfully, roll back any changes made to the database table.

-OD4R5:3B ‘…’

‘–TRACE ERROR’ Generates an adapter trace file containing error information only.

Note Refer to the Execution Commands Reference Guide for more information on execution commands.

Chapter 3 - Methods of Execution Event-Driven Execution Model


42

Event-Driven Execution Model The event-driven execution model relies upon the Mercator Event Server to coordinate the triggering of map threads based upon either input event or time event triggers.

Event Server Architecture

The Event Server architecture is an event-driven software model executing maps. Systems contain maps supporting a system data flow that is specified by using the Integration Flow Designer.

An Event Server runs on a server platform and can manage one or more systems concurrently. Event Server functionality is based on the following premises:

♦ The event manager sub-component of the Event Server controls the initiation of maps based on sources that a map subscribes to.

♦ The resource manager sub-component of the Event Server synchronizes shared data and timing interfaces among heterogeneous sources and targets.

♦ Maps publish data that result from the content transformation of source data.

♦ Maps are transactional processes with each map having its own error detection and recovery procedures.

♦ The Event Server runs as a single process with multiple threads. An associated process monitors input source resource changes.

Event Server Watches and Triggers

The set of events that initiate a map within a system is called a watch. Each event that contributes to the initiation of a map is called a trigger. When the resource (file, database, or message queue) that is an input to a map changes state (created or modified), this input event can cause the Event Server to initiate a map.

The Event Server starts maps based on triggers that you identify using the Integration Flow Designer. In some cases, there may be multiple triggers that must collectively exist before a map is initiated. The Event Server manages the coordination of these events to ensure that the correct set of circumstances have occurred before a map is initiated.

Data sources using database adapters can serve as input event triggers that are defined in the Database Interface Designer, enabled in the Integration Flow Designer, and executed by an Event Server.



43

A watch may have multiple map instances executing simultaneously. The Event Server uses its resource manager to insure that map inputs and outputs remain in a consistent state.

Event Server Methodology

Whenever the Event Server detects a new event trigger, the watches associated with that trigger are either:

♦ Initiated, thereby causing the map to execute

♦ Placed in an initiation pending state, awaiting the completion of all specified triggers

♦ Maintained in an initiation pending state until operating system resources (such as threads) are available

The Event Server initiates an individual map thread for each successful combination of specified triggers. Multiple instances of the same map component can be invoked and executed simultaneously if the Event Server is configured to take advantage of concurrent multi-threading of map execution for performance. Alternatively, if strictly sequential execution architecture is required (FIFO); configuring the Event Server accordingly can accommodate this. This concurrent versus sequential capability can be configured at the individual map component level. So maps contained in a system could be concurrent, while others could be set to execute sequentially.

Business Advantages

Some of the main business advantages to using the Event Server are:

♦ Automatic system execution

♦ Multi-threaded environment for multiple transactions. Linear scaling across multiple CPUs

♦ Process prioritization



44

Event Server System Example

The following is an example of a system defined in the Integration Flow Designer and executable by the Event Server:



45

The data flow for this system is as follows:

1 The VBInput source to the VBtoSQL map component is a file. The arrival or update of this file is the trigger for the automatic invocation of an instance of the VBToSQL map component by the Event Server. This trigger, represented by the glasses symbol ( ), displays on the VBtoSQL map component icon and on the VBInput card.

If the map component has been configured accordingly, multiple instances of the same map component are invoked if the file is updated again before the first map instance completes.

The transformation logic in the map component is executed, and the resulting output is delivered to a SQL Server database table (SQLInsert output).

2 The output SQLInsert from the VBtoSQL map component is the same table as the input to the second map, SQL to R/3 (VendorMaster). Inserts to this table are also defined as triggers to the SQL to R/3 map component. When inserts to the VendorMaster table occur, the Event Server triggers the SQL to R/3 map automatically to retrieve the rows inserted.

The data from the input rows in the VendorMaster table are converted into SAP R/3 IDocs. The Mercator R/3 ALE adapter delivers the resulting IDocs as output to the R/3 system.

3 SQL to R/3 (VendorMaster) is configured to generate an audit log. Based on the map settings configuration, the audit log can contain various aspects of each map thread’s execution, settings, and data. These audit logs can themselves be mapped as input data to provide transaction history and error and exception logging to any output resource supported by a Mercator resource adapter (such as the Database or E-mail adapter).

Because maps are able to handle multiple inputs and multiple outputs in a single execution thread, it is easy to create additional output cards for the maps so that transaction logging data can be delivered to any enterprise-level database, or update transaction data, in real-time, to data warehouse databases.

For example, the AuditLogProcessing subsystem (a subsystem is an object that represents another system that you have already defined) is a reusable Event Server-controlled system that provides the transaction logging and exception handling service. This system can be used to process the audit logs for multiple Event Server-based systems. Execution history logging can be achieved by mapping each map thread’s audit log to an external database.



46

Audit and Transaction Logging Example

The totally flexible architecture of Mercator products allows the design and development of sophisticated transaction and audit logging using standard functionality. This is supported by the complete range of resource adapters and the many tools available in the complete line of Mercator products.

Maps are transactional processes and each map can be configured to have its own error detection and recovery procedures, as well as auditing capability. The resulting audit logs themselves can be mapped, as input data, to dedicated execution logging, error logging, and exception handling routines.

Typically, audit and transaction logging processing is achieved by designing re-usable map systems to process audit logs resulting from map execution. The following is an example of this type of functionality:

♦ The map component, Read Audit is configured to automatically be triggered by the arrival of audit log files generated by the main map execution activity.

♦ Outputs 1 and 2 (TransmissionLog and ExecutionHistory) are intermediate output formats inside the map, and are not sent to an external destination. Their output is re-used in outputs 3 and 4 (ErrorLog and ExecutionHistoryDB).



47

♦ The output from ErrorLog is delivered to a file. Each time a map fails, an entry is appended to this file. An alternative to this file-based method is to use the Mercator SNMP Agent to issue traps, which trigger alerts in SNMP-enabled network management software (such as HP OpenView, Tivoli NetView, and BMC Patrol).

♦ Every single map thread execution is logged to an execution history database through the ExecutionHistoryDB output. Finally, output 5 (PutToEmail), sends the audit log data to another map for reformatting and dispatch to an operator’s email account with information on exactly which map failed in its execution and detailed reasons for the map failure.

It should be stressed that this is only an example of the type of audit logging, execution logging, and error and exception handling that you can develop with Mercator products. It is possible to construct a logging and management system as simple or as sophisticated as is required for the purposes of the enterprise concerned.

Event Agent

The Event Agent, available for both the Windows and UNIX platforms, is a light-footprint solution designed to receive external Web requests (high-volume capable) through the Event Server. The Event Agent is a utility that listens for incoming HTTP requests and passes them onto the Mercator Event Server to trigger coordinated systems of maps. The difference between handling HTTP requests using the Event Agent as compared to a servlet is that the Event Server can process an entire asynchronous system of maps by a single HTTP request. Unlike the servlet, which is essentially one map invocation per servlet (although there are ways to call additional maps in the same session), the Event Agent can trigger many maps in a coordinated fashion via the Event Server. Also, unlike the servlet where the map that received the initial HTTP request has to be the map that sends the resulting HTTP response, the Event Agent and Event Server-based invocation permits any map in the triggered system, or associated systems, to send the HTTP response to the Event Agent and forwarded to the point of origin.

This capability not only allows conventional HTTP processing scenarios to be implemented (such as browser-based access to enterprise resources via the Internet or Intranet), but also supports a growing trend for using HTTP as the primary transport for intra-enterprise application-to-application (A2A) and inter-enterprise business-to-business (B2B) integration. The Event Agent works in conjunction with the HTTP adapter enabling the Event Server to directly process HTTP input events without compromising the integrity of the firewall. The Event Agent can be deployed outside the corporate firewall in the demilitarized zone (DMZ) for business-to-consumer (B2C) or B2B integration, or inside the firewall for A2A integration, or in both places.



48

Because an application or Web server is not required, complete A2A, B2B, and B2C capabilities are provided without the need to resort to programming.

The Event Agent supports HTTP/1.1, and with the installation of the Security Option, supports the following protocols:

♦ Secure Sockets Layer 2.0 (SSLv2)

♦ Secure Sockets Layer 3.0 (SSLv3)

♦ Transport Layer Security 1.0 (TLSv1)

The following depicts a possible Event Agent configuration:



49

Event Agent Operation

The following is an example of how the Event Agent, HTTP adapter, and the Event Server work together to process HTTP requests:

The processing sequence in this example is as follows:

1 When the Event Server is started, Map A, which is expecting an HTTP request as input, sends an HTTP request to the Event Agent via the HTTP adapter. The HTTP adapter registers the URL that identifies the requests it can receive from the Event Agent.

2 An HTTP request is received by the Event Agent, which then looks for an adapter that is registered to handle the request type.

3 The request is then forwarded to the HTTP adapter, which previously registered the URL. The HTTP forwards the request to the map, assigning a unique session ID for that session.



50

4 The map triggered by the HTTP request generates its output, which is passed onto another map whose output may be passed onto another map, and so forth. In all cases, the session ID assigned by the Event Agent is passed along to each subsequent map in the form of a wildcard parameter. Once the required output has been generated, Map N sends an HTTP request (including the session ID originally assigned by the Event Agent) to the HTTP adapter and passes, in that request, its output data to be returned in response to the original HTTP request in step 2. The HTTP adapter forwards the HTTP request to the Event Agent.

5 The Event Agent sends the data sent in the HTTP request in step 4 to the client that made the original HTTP request in step 2. During the time the Event Server has been executing the maps, the Event Agent has maintained state on the original HTTP request.

6 The response returned by the output data for the original HTTP request sent in step 5, is passed back as a response to the Event Server map that made the HTTP request to the Event Agent in step 4. This in turn can be mapped and updated as an additional output to a transaction-logging database.

When to use the Event Agent ♦ When the HTTP request requires a complex system of maps to generate the

response.

♦ If the integration is required to use asynchronous messaging.

♦ If a light-footprint solution is required to process internal and external HTTP requests.

Business Advantages ♦ Single architecture for A2A, B2C, and B2B integration

! Greater re-usability and portability of skills

! No procedural programming

! Ease of ongoing maintenance

! Quicker implementation times

♦ Major platform support

! Leverage existing IT investments—technology & skills

! Reduced cost—no new skill sets needed

♦ Firewall integrity is maintained—no requirement to “punch a hole” into the firewall from outside in the DMZ

! Threats to business security are minimized

Chapter 3 - Methods of Execution Application-Embedded Execution Model


51

Application-Embedded Execution Model The Mercator API provides the ability to execute maps programmatically as “embedded” processes and is included in the Mercator Software Development Kit (SDK). The API, available for both the C and Java languages, interfaces directly with the core execution engine in Mercator products running on the appropriate target platform.

The Mercator API is an object-orientated interface used for invoking, executing, and controlling maps. The API abstracts map components and settings as objects, and is based on map, card, adapter, and stream objects. Objects are used for source and target overrides for map execution.

If no card overrides are required, only the map object is referenced. All other objects are created internally. If card overrides are required, the card and adapter objects are referenced.


52

Chapter 4 - Developing Custom Adapters The interface for developing custom adapters makes it possible to develop adapters with as much functionality as those provided with the product. The primary benefits that this interface provides are:

♦ The ability to integrate adapters into Mercator products using the XML registration file

♦ The ability to participate in connection pooling

♦ The ability to provide an event listener

♦ An object-orientated interface

The Adapter Toolkit provides the complete set of components to create adapters that can be seamlessly embedded into the Mercator product execution environment. Custom developed adapters can even be added as selections in the Map Designer and Integration Flow Designer GUI when configuring map card settings. With this capability, custom developed adapters can have the same status within Mercator products as those that are shipped with the products.

The process for integrating a new adapter into the list of available adapters is very straightforward; add your custom developed adapter to the adapters.xml file. This file provides information about all adapters—Mercator’s and user-defined.

Chapter 4 - Developing Custom Adapters XML Descriptor File


53

XML Descriptor File The XML descriptor file adapters.xml, which is located in the Mercator installation directory, contains a single tag per adapter, including a number of attributes to provide information about the adapter. The following is an excerpt from the file:

<Adapters> <M4Adapters> <M4Adapter name="File" alias="FILE" id="0" type="file" library="m4file"/> <M4Adapter name="Buffer" alias="BUF" id="4" type="buff" /> <M4Adapter name="Database" alias="DB" id="5" type="db" library="dbutil"/> <M4Adapter name="Application" alias="APP" id="6" type="app" />  <M4Adapter name="Batch File" alias="BAT" id="110" type="app" library="m4batch"/> <M4Adapter name="Shell Script" alias="SHL" id="111" type="app" library="m4shell"/> <M4Adapter name="OracleAQ" alias="AQ" id="112" type="msg" library="m4aq" /> <M4Adapter name="Sink" alias="SINK" id="113" type="app" library="m4sink"/> <M4Adapter name="Tuxedo" alias="TUX" id="114" type="msg" library="m4tux" version="6.x"/>  <UserAdapters>  </UserAdapters> </Adapters>

A description of the various tags for each of the adapter entries is as follows:

Attribute Description

name Name of the adapter used in the adapter list in the Map Designer and Integration Flow Designer (such as Database or WebSphere MQ (server)).

alias Abbreviated name of the adapter that is used in a GET or PUT function inside map rules, as well as command line overrides (such as DB for database or MQS for WebSphere MQ).

id Identification type of the adapter. For adapters developed by Mercator, these numbers are between 0 and 200. For external, custom developed adapters, the ID numbers should be >= 200.

type Class type of the adapter. For example, database (db) or messaging (msg).

54

Attribute Description

library Name of the DLL or shared library. The DLL, SO, or SL file extension should not be provided. In addition, for UNIX adapters, the name should not be prefixed with lib. This will be added when accessing the adapter.

vendor Name of the adapter manufacturer. The vendor name is displayed along with the adapter name in the Map Designer and Integration Flow Designer.

The following is an example of adding an adapter to the available adapter pool:

1 Create the adapter using the Adapter Toolkit.

2 Add the entry for the new adapter to the adapters.xml file:  <UserAdapters> <UserAdapter name="FIX API (Cstm)" alias="FIX" id="201" type="app" library="fixapi" vendor="CLSA"/> <UserAdapter name="OMS API (Cstm)" alias="OMS" id="202" type="app" library="omsapi" vendor="CLSA"/> </UserAdapters> </Adapters>



55

3 The adapter is now a selection in the adapter list in the Map Designer and Integration Flow Designer.


56

Chapter 5 - Operation Management & Monitoring Capabilities Mercator products provide a number of capabilities and components for managing and monitoring the integration execution environment such as:

♦ A Management Console that provides detailed monitoring information as well as dynamic configuration features for systems under control of the Event Server.

♦ Ability through SNMP to provide the same detailed monitoring information to third-party management software.

Management Console The Mercator Management Console is a cross-platform application that provides in-depth, real-time monitoring information on:

♦ System performance

♦ Errors

♦ Heartbeats

Chapter 5 - Operation Management & Monitoring Capabilities Management Console


57

It also permits centralized, remote management of all Event Server installations.

Separate connections are defined for each Event Server installation. Each Event Server then appears in the Management Console’s main window, and each can be managed from the single, remote interface:



58

From the Management Console, it is possible to remotely stop, start, and pause systems. Event Server debugging can also be enabled to assist in operational troubleshooting of Event Server operations.

The Management Console provides a variety of statistics about the maps being run by the Event Server, such as how long the Event Server has been running, how many maps have been processed, and so on.

♦ Summary: Provides a general view of performance and status information

♦ Status: Contains active and pending system status information

♦ History: Contains a rollup of the total number of maps run, number of successes, number of failures, and so on

♦ Configuration: Contains Event Server configuration parameters

Note The historical map execution information provided by the Management Console is similar to information contained in audit logs as outlined in the Audit and Transaction Logging Example section. However, the Management Console-based logging information relates to Event Server-executed systems of maps, whereas audit logs apply to maps configured to generate an audit log and executed with any transformation server including the Command Server, Platform API, and so on.



59

Event Server Administration

An Event Server Administration service is available to configure the systems that can be controlled by the Management Console. The Event Server Administration enables the configuration of secure access rights to monitor and control individual systems running under one or more Event Servers. Configurations can be defined for:

♦ Automatically starting systems when the Event Server service is started

♦ Management Console connection ports

♦ Resource resolution directives as defined in the Resource Registry

Specify resource configuration file to resolve resource values.

Chapter 5 - Operation Management & Monitoring Capabilities SNMP Support


60

SNMP Support With the Mercator SNMP Agent, you can monitor and manage the status of the Event Server and associated map events. The following functionality is provided using the Mercator SNMP Agent:

♦ Configure thresholds

♦ Detect errors

♦ Monitor Event Server performance and help identify failure situations

♦ Notify SNMP-enabled enterprise management software of changes in the state of:

! Connections

! Listeners

! Event Server

SNMP support is provided with the Event Server on all platforms.

The Mercator SNMP supports the following traps:

Trap Warnings and Errors

Map failure Signals when a map fails

State and Listener change Signals a change in state of the Event Server or Listeners

Event Server/resource connection failure

Notifies SNMP management software that connection to the Event Server and the resource could not be established

Thresholds Signals when a map or connection pending state threshold has been met

Note Refer to the SNMP Agent Reference Guide for more information.


61

Chapter 6 - High-Availability and Load-Balancing Architecture The following are the high-availability and load-balancing options that Mercator products currently offer:

♦ Server clustering

♦ Enterprise JavaBeans

Server Clustering A server clustering approach caters for high-availability and failover but does not address the load-balancing requirement. The Mercator products support failover in the Microsoft Cluster Service and Sun Cluster 3 server environments.

Typically, in clustering approaches, there are two server nodes, both of which have access to a shared disk array. One server node is the active, primary node, and the second server is online, but passive. The two servers send heartbeat information, which is monitored by cluster management software. In the event that the primary server fails and does not send its heartbeat, the monitoring software begins to fail over to the secondary server, which then becomes the active node.

The cluster management software permits the selection of processes and services to be restarted on the secondary node in the event that the primary node fails. Because the Event Server is installed as a service on Windows or a daemon on UNIX, it is an ideal candidate for restarting on the secondary node. The key to not losing any data in the failover is in the fact that both nodes point to a shared disk and that each map thread is a single atomic transaction.

The map components and any import and export directories, local message queues, and so forth are installed on the shared disk farm. Because each map thread is transactional, if any map threads are running on the primary node when it fails, the map rolls back the transaction it was processing at the time of the failure, leaving all input sources (files, databases, message queues, and so on) completely intact. When the systems fails over to the secondary node and the Mercator Event Server is started, the map picks up precisely from the point the primary Event Server rolled back to, working with the components left intact on the shared disk array.

Chapter 6 - High-Availability and Load-Balancing Architecture Server Clustering


62

The following diagram illustrates a Windows cluster environment for the Event Server.

Only one node can accessthe shared drive at at time.

Microsoft Cluster Service (MSCS)core and Resource Monitor

Event Server

MSCS Resource DLLfor the Event Server

Microsoft Cluster Service (MSCS)core and Resource Monitor

Event Server

Shared Drive

Contains maps,data, and system(.msl) files.

MSCS Resource DLLfor the Event Server

Clustering Support for the Event Server

MSCS Node

Computer 2

MSCS Node

Computer 1

ActiveNode

YesFailure?

InactiveNode

No

" Each computermust have MSCSrunning onWindows 2000Advanced Server.

" The MSCSResource DLL thenrelays a shutdownmessage to theEvent Server.

" Only one node isactive at a time.When a faiureoccurs, theResource Monitornotifies the MSCSResource DLL.

" The IntegrationBroker is installedon each node thatis part of thecluster network.

Becomes activeupon failure ofNode 1.

Chapter 6 - High-Availability and Load-Balancing Architecture Enterprise JavaBeans


63

Enterprise JavaBeans By utilizing an EJB integration approach, maps can be wrapped inside both stateful and stateless Session beans, and deployed to EJB 1.0, EJB 1.1, and J2EE-compliant application servers. The application server environments themselves provide extensive load-balancing and failover between redundant instances of EJBs running on different physical application servers.

Because map threads are atomic transactions, in the event that a map bean fails, either because the application server has crashed, or for some other reason, the map thread’s transaction will be rolled back in its entirety; leaving the sources and destinations in their original state. Assuming identical map beans are running on a parallel application server, the source data that was being processed by the original map bean when it failed will be consumed by the duplicate map bean running on a different application server.

However, the downside to this approach is that Java programming is required to create the application layer that interfaces with the map bean.


64

Chapter 7 - Securing HTTP Communications The two main requirements of a secure environment are the Secure Socket Layer (SSL) protocol and digital certificates. To activate the SSL capabilities within your Web server, all you need to do is install a digital certificate on your server. An SSL server secures an HTTP session by encrypting the information exchanged between a client and the server.

What is SSL? SSL is a protocol developed by Netscape Communications Corporation for ensuring security and privacy in Internet communications by implementing public-key encryption of data. Because this technology is already incorporated into most Web servers (such as Microsoft Internet Information Server (IIS) and IBM WebSphere) and commonly used browsers (such as Internet Explorer and Netscape), they are ready for SSL implementation.

Note Mercator products support the Privacy Enhanced Mail (PEM) format for digital certificates and private keys as the Internet standard used for encryption techniques to ensure the privacy and security of messages.

Encryption

A key system is used to safeguard data sent through the Internet. The sender of a message encrypts it with a key and in order to read this message, the receiver must decrypt it with another key. Public and private keys are used together to encrypt a message. Public keys and corresponding private keys are generated using encryption software installed on the server hosting the HTTPS URL you are connecting to. Once a session is initiated between both machines, the server sends a public key. The public key is given to everyone and therefore, can also be seen by everyone. When a message is sent to the server, the message is encrypted with the public key. A private key, installed on the server, is used to decrypt the message received by the server. And because nobody else has access to this private key, privacy is ensured and the identity of the sender is authenticated.

Chapter 7 - Securing HTTP Communications Implementing an SSL Environment


65

Digital Signatures and Certificates

A digital signature is used to sign or authenticate a message. A digital signature uses a private key to encrypt a portion of the message. The corresponding public key, which is given to everyone, is used to decrypt the message. This process ensures the recipient that the message and its digital signature could have come only from the owner of the private key corresponding to the public key that was used to decrypt the message.

A digital certificate verifies the integrity of the signed message. A digital certificate is a way to authenticate a person or information on a computer and is used to implement public key encryption on Web servers. It is like a driver’s license; you are provided assurance that the person or entity is truly who or what they say they are. A Certificate Authority (CA) that acts as a third-party that both sides can trust issues these “electronic credentials”.

Digital certificates are sent with the encrypted message in order to identify the sender of the message and also to verify that the message was not altered after it was sent.

Implementing an SSL Environment A Secure Socket Layer (SSL) Security Option is available for Mercator products. This option installs the necessary SSL libraries enabling you to connect to an HTTPS URL and process secure HTTP transactions. The SSL Security Option enables 40-bit and 128-bit SSL sessions, meaning that the HTTP transactions you process can employ either 40-bit or 128-bit encryption.

To exchange encrypted data during an HTTP session using the Event Agent, HTTP adapter, or FTP adapter, the following must be in place on your Web or application server:

♦ SSL capabilities

♦ Public-key encryption infrastructure

♦ Digital certificate

Note The Security Option includes HTTPS, FTPS, OpenPGP, and S/MIME adapters. Refer to the Resource Adapters Library for documentation about each adapter.

Event Agent

If you are using the Mercator Event Agent to receive external Web requests through the Event Server, a digital certificate must be installed on the Event Agent in order to process secure HTTP transactions.

Chapter 7 - Securing HTTP Communications Implementing an SSL Environment


66

The utility RSA BSAFE SSL-C from RSA Security Inc. and included in the Security Option install, enables you to generate a self-signed digital certificate installable on the Event Agent. A self-signed certificate is a digital certificate that is not issued and signed by a CA. This type of certificate can be used when testing the SSL capabilities of the Event Agent.

The RSA BSAFE SSL-C utility also enables you to generate a certificate signing request (CSR). A CSR is submitted to a certificate authority in order to receive a digital certificate for the Event Agent.

To access the RSA BSAFE SSL-C utility

1 From the command line, change to the Mercator installation directory.

2 Enter the following:

sslc.exe

3 At the sslc prompt, enter the following to access help:

help

Note More information on the RSA BSAFE SSL-C utility can be found on the following website: http://www.rsasecurity.com/.


67

Chapter 8 - Web Services Mercator software provides integration solutions that accelerate the adoption of web services, eliminating the expensive, time-consuming, and risky overhaul of critical business systems. By combining the powerful integration capabilities of the Mercator Integration Broker with the emerging web services standards, Mercator products make it possible for organizations to provide and employ web services at both the business process and data process levels.

Mercator products now support the implementation and invocation of web services. In the simplest terms, web services represent a new standards-based and simplified model for creating and connecting distributed applications across the Web in the form of services. Web services are built on top of existing and widely-adopted Internet protocols. Web services are XML depictions of objects, messages and documents designed to interact over the Web to enable application-to-application integration. Web service applications can be published, found, or invoked as atom-like services anywhere on the Internet, thus creating a network of dynamic business components.

What is a Web Service? A web service is a method exposed by a company or software program that is both discoverable and accessible by other programs or organizations that are in need of a particular service, such as purchasing a product, reserving a flight, or calculating tariffs. These are discrete business services that have value to many organizations.

Why Use Mercator Products to Implement Web Services?

The following is an analysis for using Mercator products as both a provider and consumer of web services.

♦ Does the web service require a complex system of maps to generate the response?

♦ Does the web service, or overall integration solution, require asynchronous messaging?

♦ Is a light footprint solution desired to receive external web requests?

♦ Is integration needed to support internal application system integration using XML transported over HTTP, where the application system currently may not be implemented as a web service?

Chapter 8 - Web Services Technologies Used in Web Services


68

If the answer to any of the above questions is yes, then using Mercator Integration Broker products such as our SOAP and HTTP adapters, the Event Server, and the Event Agent, which acts as a listener for web service requests, is the way to go.

Technologies Used in Web Services Web services are built around widely-adopted Internet standards such as TCP/IP, HTTP, Java, HTML, and XML plus an emerging set of new standard technologies such as Simple Object Access Protocol (SOAP), Web Services Description Language (WSDL), and Universal Description Discovery and Integration (UDDI).

SOAP

SOAP defines the XML-based message format that applications use to communicate and inter-operate with each other over the Web. The heterogeneous environment of the Web demands that applications support a common data encoding protocol and message format. SOAP is a standard for encoding messages in XML that invoke functions in other applications. It is analogous to Remote Procedure Calls (RPC) used in many technologies such as DCOM and CORBA, but eliminates some of the complexities of using these interfaces. SOAP enables applications to call functions from other applications, running on any hardware platform regardless of different operating systems or programming languages.

SOAP Adapter

The SOAP adapter is used to aid in building a web service request and decode the response. The SOAP adapter encodes and decodes messages in SOAP format. The HTTP adapter acts as a transporter, enabling the adapter to transport SOAP messages over HTTP. The SOAP adapter can also work with any other transport adapter.

Note Refer to the Resource Adapters Reference Guide for information on the adapter commands that enable the transporting, and encoding and decoding of messages.

WSDL

WSDL is a collection of metadata about XML-based services used for describing what businesses do and how to access their services electronically. WSDL specifies the procedures to discover functional and technical information about web services over the Internet.

Chapter 8 - Web Services Technologies Used in Web Services


69

A WSDL document is comprised of a number of elements:

♦ Type definitions for data elements (normally using XML Schema)

♦ Message definitions that comprise one or more typed data elements

♦ Operation definitions that are abstract descriptions of actions supported by the service, and which define what are the input and output messages

♦ Port Type definition lists the set of operations supported by the Web Service

♦ Binding definition describes the binding between the port types and protocols (such as SOAP)

♦ Service definition lists the set of bindings

WSDL Importer

Type trees can now be created based on a WSDL web service definition. A map using this type tree can be created to call a web service or can be published as a web service.

Note Refer to the Type Designer Reference Guide for information on the WSDL Importer.

UDDI

The UDDI Business Registry is a standard for cataloging and publishing WSDL descriptions of web services that are available over the Internet. In much the same way as people peruse the common yellow pages for a particular service, commerce systems can search the UDDI registry and find web services, download the parameters for interaction, and effectively interact with the discovered web service (using SOAP).

UDDI Business Registry is an implementation of the UDDI specification. A UDDI business registration consists of three components:

♦ white pages – address, contact and known identifiers

♦ yellow pages – industrial categorizations based on standard taxonomies

♦ green pages – the technical information about exposed services (such as WSDL documents)

UDDI specifications consist of an XML schema for SOAP messages and a description of the UDDI API specification – the latter actually is a SOAP specification. The API defines methods for both inquiry and publishing services.

Chapter 8 - Web Services Web Services Architecture


70

The UDDI invocation model is as follows:

1 You use the UDDI business registry (either via a Web interface or using the inquiry API) to locate the business entity information for the business partner advertising the web service.

2 You drill down to find the advertised web services and obtain the binding templates (WSDL documents).

3 You prepare the program based on the WSDL specifications. Create a type tree based on the WSDL document using the WSDL Importer in the Type Designer. Then use the Map Designer to create a map that incorporates this type tree.

4 At runtime, the program invokes the web service as planned, using the binding obtained from the WSDL document.

Web Services Architecture There are three collaborators in the architecture of a web service: the provider, consumer, and broker. These collaborators interact using the following three basic operations:

♦ Publish: The provider publishes a web service (described by a WSDL document) to the broker, which is typically a UDDI registry.

♦ Request: The consumer requests information from the broker about the web service.

♦ Bind: The consumer binds its application to the web service. Now the application can interact with the web service, invoking its methods via SOAP requests.

By providing support for the importing of WSDL (WSDL Importer) and sending and receiving of SOAP messages (SOAP and HTTP adapters), Mercator products fit into this architecture as both a consumer and provider of web services.



71

Consumer Architecture

For design time purposes, the WSDL Importer generates type trees from WSDL files.

Type Designer

WSDL Importer WSDL file

Type tree

For run time purposes, the SOAP adapter is used to invoke the web service by sending a request message and processing the response. Note that two adapters are used in communicating between the map and the web service: the SOAP adapter and the HTTP adapter. The following diagram demonstrates the flow of the request from the map, through the SOAP and HTTP adapters to the web service, and the flow of the response back to the map from the web service.

Map SOAP Adapter

HTTP Adapter

Web Services

Because the map sends and receives SOAP messages over the Web, it is considered a consumer of a Web service.



72

Provider Architecture

Mercator Integration Broker, acting as a provider of web services, receives SOAP requests over HTTP, executes a map system, and then sends a SOAP response.

Consumer

Event Agent

HTTP Adapter

SOAP Adapter

Map SOAP Adapter

HTTP Adapter

SOAP Request

SOAP Response

Event Server

Because the map or system performs a process over the Web, it is considered a provider of a Web service.

Chapter 8 - Web Services Web Services Example


73

Web Services Example The install_dir\integration_package\WebServices_v1\examples directory contains sample files to be used with Mercator Web Services components, including the WSDL Importer, SOAP and HTTP adapters, Event Agent, and the Event Server, to implement web services strategies.

There are three examples:

1 Provider - This example demonstrates how to provide a web service using an existing map and exposing it as a web service through a wrapper map.

2 Secure Provider - This example is the same as the Provider example except that it uses the Mercator SSL Security Option to connect to a secure URL (HTTPS).

3 Consumer - This example demonstrates how to consume a web service defined in WSDL using a map to invoke the service.

Note Refer to the readme_ws_examples.txt file located in the install_dir\integration_package\WebServices_v1\examples directory for more information regarding these examples.


74

Index A

adapters customizing, 52 FTP, 65 HTTP, 47 overview, 26 SOAP, 68

adapters.xml, 53 aliases

creating for resources, 35 API

overview, 51 audit

map, 33

B business rules

implementing in maps, 25

C clustering, 61 Command Server, 40 custom adapters, 52

XML descriptor file, 53, 56

D data

retrieving in increments, 31 transforming using maps, 13 using adapters to send and receive, 26

Database Interface Designer overview, 20

databases importing metadata from, 20

Design Studio Command Server, 40 Database Interface Designer, 20 Integration Flow Designer, 17 interfacing to version control software, 22 Map Designer, 13 overview, 8 Resource Registry, 35 Type Designer, 9 using adapters to send and recieve data, 26

digital certificates, 65 digital signatures, 65

E encryption, 64 Enterprise JavaBean API

overview, 39 Enterprise JavaBeans

load-balancing, 63 Event Agent

implementing SSL, 65 overview, 47

Event Server, 42, 60 audit and transaction logging example, 46 clustering, 61 monitoring remotely, 56 processing HTTP messages, 47 system example, 44 watches and triggers, 42

Event Server Administration, 59 examples

audit and transaction logging, 46 Event Server system, 44 overriding a database target, 41 processing HTTP messages, 49 web services, 73

exporting maps, 15 type trees, 11

F failover

monitoring using SNMP Agent, 61

H HTTP adapter, 47 HTTP messages, 47

security, 64

I illustrations, 62 importers

WSDL, 69 input

maps, 25 settings, 30

integration creating solutions with Design Studio, 8 using custom adapters, 52 using Mercator software, 7

Integration Flow Designer

Index L


75

building and deploying systems, 18 creating a data flow, 18 logic analysis, 18 overview, 17

L load-balancing

EJBs, 63 server clustering, 61

logic analyzing in the Integration Flow Designer, 18

M Management Console, 56 Map Designer

comparing differences in maps, 16 exporting maps, 15 map execution behavior, 28 overview, 13 settings, 33 using maps to transform data, 13

maps anatomy of, 24 comparing differences, 16 executing using the API, 51 executing using the Command Server, 40 executing using the Enterprise JavaBean API,

39 executing using the Event Agent, 47 executing using the Event Server, 42 executing using the Servlet Integrator, 37 execution behavior, 28 exporting, 15 implementing business rules, 25 initiating, 42 input and output, 25 methods of map execution, 37 overview, 23 retrieving data in increments, 31 settings, 33 source settings, 30, 35 target settings, 32 triggering, 42 using to invoke web services, 70 using to transform data, 13 using type trees in, 25

methods of map execution API, 51 Command Server, 40 Enterprise JavaBean API, 39 Event Agent, 47 Event Server, 42 overview, 37 Servlet Integrator, 37

O output

maps, 25 settings, 32

R related references, 5 resource aliases, 35 Resource Registry, 35

S security

for HTTP messages, 64 server clustering, 61 Servlet Integrator

overview, 37 SNMP Agent, 60 SOAP, 68

adapter, 68 source control, 22 SSL, 64

implementing with Event Agent, 65 implementing with Mercator software, 65

systems audit and transaction logging example, 46 building and deploying, 18 configuring for monitoring, 59 definition, 17 Event Server example, 44 monitoring, 56 monitoring using SNMP Agent, 60

T troubleshooting

maps, 33 Type Designer

comparing differences in type trees, 12 creating type trees, 10 exporting type trees, 11 overview, 9

type trees comparing differences, 12 creating, 10 definition, 9 exporting, 11 importers, 10 using in maps, 25

U UDDI, 69

V version control, 22

Index W


76

W web services, 67

consuming, 71 definition, 67 examples, 73 providing, 72 SOAP, 68 technologies used in, 68

UDDI, 69 using maps to invoke, 70 WSDL, 68

Web-based execution Enterprise JavaBean, 39 HTTP processing requests from, 47 Servlet Integrator, 37

WSDL, 68 WSDL importer, 69

Data Transformation Engine Intelligent Business Integration Reference Guide

Documents

Transcript of Data Transformation Engine Intelligent Business Integration Reference Guide