WebSphere Application Server for z/OS V6 - IBM WWW Page · WebSphere Application Server for z/OS...

WebSphere Application Server for z/OS

IBM Compute Grid and WebSphere Java Batch

Version Date: November 19, 2012

See "Document Change History" on page 83 for a descriptionof the changes in this version of the document

IBM Advanced Technical SkillsGaithersburg, MD

WP101783 at ibm.com/support/techdocs

© IBM Corporation 2012

WP101783 - IBM Compute Grid and WebSphere Java Batch

Many thanks to Sajan Sankaran, Chris Vignola, Rajesh Ramachandran and Tim Fanelli. Each works at IBM and

each provided valuable input and feedback on this document.

The WAS z/OS team in ATS consists of John Hutchinson, Mike Kearney, Mike Loos, Louis Wilen, Lee-Win Tai and Don

Bagwell. Glenn Fisher manages the team.

A special "thank you" to Lee-Win Tai for his help with Java as well as understanding the sample applications.

Mike Cox, Distinguished Engineer, serves as technical consultant to all our activities.

© 2012, IBM CorporationAmericas Advanced Technical Skills - 2 -

WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012


Table of Contents

Executive Overview...............................................................................................................................5Introduction to Java Batch Processing...............................................................................................6

What is 'batch' processing............................................................................................................................6IBM batch processing solutions on the mainframe.......................................................................................6Java as a batch processing language...........................................................................................................6Invocation of Java batch...............................................................................................................................7The shrinking and disappearing 'batch window'...........................................................................................7Java batch execution within the WAS Java EE environment.......................................................................8Java batch programming and execution framework.....................................................................................8Integration of existing (and valuable) non-Java assets.................................................................................9The parallel nature of some batch processing..............................................................................................9Integration with critical enterprise scheduling tools......................................................................................9Summary.....................................................................................................................................................10

Technology Overview.........................................................................................................................11WebSphere Application Server Version 8.5...............................................................................................11

Functional services made available to applications by WAS...........................................................................11Consistent support of specification and APIs across platforms.......................................................................11WAS z/OS and System z platform exploitation...............................................................................................12Servers, nodes, clusters and cells..................................................................................................................13

The z/OS application server design.................................................................................................................................13Accessing applications in WAS.......................................................................................................................14Accessing data from application in WAS.........................................................................................................15Transactionality............................................................................................................................................... 15High availability............................................................................................................................................... 16Compute Grid incorporated as function in WAS V8.5.....................................................................................17Other sources of information on WAS z/OS....................................................................................................17

WebSphere Compute Grid..........................................................................................................................18Components of the Compute Grid model........................................................................................................18Jobs, steps, the Java batch application code, and xJCL.................................................................................20Submitting Jobs to Compute Grid...................................................................................................................21

Job state -- submitted, executing, ended ... and more.....................................................................................................21Transactional batch and Compute Intensive batch jobs..................................................................................22Functional services provided by Compute Grid...............................................................................................23

Application Support Functions.........................................................................................................................................23Administrator Support Functions......................................................................................................................................26z/OS Platform Exploitation...............................................................................................................................................26

WSGRID and integration with existing job schedulers....................................................................................27The Parallel Job Manager function.................................................................................................................28Developing Java batch applications for Compute Grid...................................................................................30

Technology Focus...............................................................................................................................33The COBOL Container................................................................................................................................33

Architectural overview..................................................................................................................................... 33WAS z/OS application server setup requirements..........................................................................................34COBOL program compilation requirements, program restrictions...................................................................35COBOL stub generator................................................................................................................................... 36

Overview of the COBOL Call Stub Generator..................................................................................................................36Installing the COBOL Call Stub Generator on your workstation......................................................................................37Requirements for IBM Rational Application Developer....................................................................................................38What the COBOL Stub Generator generates...................................................................................................................39Invoking from the Command Line....................................................................................................................................41Invoking as ANT task in RAD...........................................................................................................................................45Invoking as ANT and processing multiple COBOL files...................................................................................................47

Invoking the COBOL call stub from the Java batch application.......................................................................48Sharing DB2 Type 2 connection between Java and COBOL..........................................................................49

Programming enhancements in CG 8 (and WAS 8.5)................................................................................49Compute Grid Listeners.................................................................................................................................. 49




Job listener...................................................................................................................................................... 50Overview of the job listener..............................................................................................................................................50Implementing a job listener..............................................................................................................................................50Registering a job listener..................................................................................................................................................51

Skip-record processing................................................................................................................................... 52Background context on need for skip-record processing.................................................................................................52Configuring and activating a skip-record policy................................................................................................................52What happens when a record is skipped?.......................................................................................................................54Sample application for illustrating skip-record processing...............................................................................................54Registering and using a skip listener...............................................................................................................................55Skip-record metrics.......................................................................................................................................................... 56

Retry-step processing..................................................................................................................................... 57Overview of retry-step processing....................................................................................................................................57Sample application used for illustration............................................................................................................................58Configuring and activating a retry-step policy..................................................................................................................58How the skip-record sample application implements retry-step processing....................................................................61

Configurable transaction mode.......................................................................................................................64Overview of the function................................................................................................................................................... 64Configuration in xJCL....................................................................................................................................................... 64Getting the shared connection from the Job Step Context Manager...............................................................................65

Parallel Job Manager and enhancements in CG 8 (and WAS 8.5)............................................................65General enhancements in PJM function.........................................................................................................65Control over where parallel subjobs are dispatched -- threads or JVMs.........................................................65Mixing of job-step types within a multi-step job...............................................................................................67A look inside the Mailer sample parallel program............................................................................................67

Overview of the Mailer sample application.......................................................................................................................68Mailer sample application -- xJCL overview.....................................................................................................................69Parallel processing for the data in the Mailer sample application....................................................................................69The Parameterizer is what determines how to partition the data.....................................................................................70The Mailer sample parameterizer at a high-level.............................................................................................................71The Mailer sample parameterizer in detail.......................................................................................................................71Other Parallel Job Manager properties and what they do................................................................................................73

Submitting the Mailer job -- one top job and three sub-jobs............................................................................75Backup Documentation.......................................................................................................................77

Information Centers....................................................................................................................................77InfoCenter URLs............................................................................................................................................. 77Search Scope................................................................................................................................................. 77Samples.......................................................................................................................................................... 78

Code generated by the Call Stub Generator..............................................................................................78ADDER.java.................................................................................................................................................... 78Data Bindings.................................................................................................................................................. 79

xJCL for the Mailer Sample.........................................................................................................................79Document Change History..................................................................................................................83




Executive OverviewThis document started out as a means to focus attention on four elements of the IBM Compute Grid product that are new in Version 8, or relatively new and deserving of a fresh look. These are the items found under the "Technology Focus" section starting on page 33:

COBOL Container Provides a means of executing compiled COBOL modules inside the address space of the WAS z/OS application server. The COBOL container function provides the infrastructure to call COBOL from Java and exchange information in a very efficient manner

33

Programming Enhancements

Job Listener The Job Listener provides a means of registering application code to get control at key points in the processing of a Java batch job. That allows the batch programmer to do setup and cleanup processing as needed, with the Compute Grid batch container invoking the application code at those points in the processing.

50

Skip-Record Skip-record processing provides a way to define how the Compute Grid batch container may skip records for which read or write exceptions have occurred. The function provides control over the number of records that may be skipped and the exceptions that may be included or excluded. A skip-record listener function, similar in concept to the Job Listener, is also provided.

52

Retry-Step Retry-step processing provides a way to define how the Compute Grid batch container may retry job steps for which an exception was thrown. The function provides control over the number of retries, the delay between retries and he exceptions to be included or excluded. A retry-step listener is provided.

57

Transaction Mode Provides a way to define local mode transactions rather than global, which provides increased efficiency of operations.

64

Parallel Job Manager Enhancements

The Parallel Job Manager function provides the ability to process batch jobs in parallel, with facilities to programmatically set data range parameters, and with facilities to manage the sub-jobs as part of the overall job. This function has been enhanced to provide mixed batch step types within a job as well as specifying the parallel processing to be done at the job step level.

65

In writing the technical specifics it became apparent that discussing enhancements to a functional offering like Compute Grid required context. Some readers might not be familiar with Compute Grid. Or, for that matter, familiar with the underlying Java runtime platform provided by WebSphere Application Server. So we added a "Technology Overview" section starting on page 11.

One final piece of context was required -- setting the very basic context about batch processing and what Compute Grid provides that addresses some of the issues surrounding batch processing using Java. That is provided under "Introduction to Java Batch Processing" starting on page 6.

So the document proceeds from relatively high-level concepts to more and more technical specifics about Compute Grid and what it provides.




Introduction to Java Batch ProcessingBatch processing has been a part of business information technology since the very early days. It remains a large and important part of today's business information processing workloads.

IBM technology, and IBM mainframe technology in particular, has played a key role in the industry's approach to batch processing. That was true in the early days; it is true now.

What is 'batch' processing

Various definitions exist, and in essence they all have at its core the idea of processing data with little or no human interaction. Contrast that with Online Transaction Processing (OLTP) where human interaction is involved repeatedly.

An example of batch processing would be a program that calculated the sales tax on every customer account in a database1. The batch program would read in the first customer record, calculate the tax, insert the tax into the database and total the amount owed, then move to the next customer record. If the customer database consisted of one million records, then the batch program would loop through its processing one million times.

IBM batch processing solutions on the mainframe

The following table summarizes a few solutions:

Language compilers such as COBOL Allow the development and execution of batch programs run in the traditional manner; that is, as submitted jobs using JCL that calls the compiled program.

BPXBATCH A facility of Unix Systems Services (USS) that allows launching of a UNIX shell environment from JCL. This in turn allows execution of shell scripts or the launching of a Java Virtual Machine (JVM) to run Java programs in batch.

JZOS A set of functions and programming libraries that make execution of Java in a batch JVM easier. IBM acquired the technology2 and incorporated it into the z/OS operating system.

z/OS 1.13 Batch Runtime A framework to better facilitate Java to COBOL interoperability with transactional updates to DB2. This batch runtime environment uses a subset of the WebSphere Compute Grid function, which is discussed next.

WebSphere Compute Grid A Java batch execution runtime and programming framework that operates within the WebSphere Application Server Java EE environment. The most current version of Version 8. See "WebSphere Compute Grid" on page 18 for an overview.

WebSphere Application Server V8.5 The Java batch function of WebSphere Compute Grid V8 was incorporated into the WebSphere Application Server V8.5 offering. Therefore, with WAS V8.5 you have access to all the function found in Compute Grid V8. See "WebSphere Application Server Version 8.5" on page 11 for an overview.

The focus of this document will be on Compute Grid and WAS V8.5, which are highlighted in yellow here.

Java as a batch processing language

There is a growing interest in using Java for batch processing. The trend towards using Java has been underway for several years now. The reasons for considering Java for batch processing include:

1 This is an example of what we call a "transactional" batch program. Another form is a "compute intensive" batch program. You'll see both phrases used later in this document.

2 IBM acquired the technology from Dovetail Technologies.




• Skills -- Java is a programming language for which there is a good supply of skills in the market. Skills for other languages, such as COBOL or Assembler, are less readily available. Further, Java is a programming language many companies are standardizing on as the core development language for applications. It makes sense to extend this standardization to batch processing as well.

• Development Tooling -- there is a rich set of development tooling available for Java. Many companies have standardized on tooling such as IBM's Rational Application Developer for Java development. Developing batch processing programs in Java allows companies to leverage existing tooling3.

• Specialty Engines -- on the IBM System z mainframes specialty engines4 provide a means of offloading certain classes of work (Java being one class of work) to a processing engine other than the General Processors (GP engines). This provides meaningful cost reduction in acquisition as well as ongoing software licensing charges.

Other motivations to use Java for batch processing exist. The point is that there are valid reasons to consider Java for batch processing. Many have already chosen Java for batch processing requirements.

Invocation of Java batch

JVM launcher approaches such as BPXBATCH or JZOS work on the principle of initiating the JVM, running the named Java program, then tearing down the JVM to complete the process. That works, but it carries with it the overhead of JVM creation and destruction. When Java batch programs are run hundreds (or thousands) of times a day, that overhead becomes excessive.

That is why the Compute Grid and WAS V8.5 Java batch function operates on the principle of a JVM environment that is maintained across invocations of batch programs. The overhead of JVM creation and destruction is incurred far less frequently.

How Java batch programs are invoked and managed within the Compute Grid environment is discussed under "WebSphere Compute Grid" on page 18.

The shrinking and disappearing 'batch window'

Years ago companies would operate with two windows of processing -- online processing during the day, and batch processing at night. Often there was no overlap of the two; no batch was run during online processing, and no online was run during the batch window.

Many factors have led to the reduction or elimination of the dedicated batch window. The most obvious factor is the requirement to make online processing available 24 hours a day.

In addition, there is a growing need to execute certain batch processes throughout the day, not just at night. What results is a blend of online processing with batch, and with batch increasingly executing in smaller segments throughout the day:

This suggests the need for coordinated intermixing of online and batch. That is one of the

3 IBM Rational Application Developer 8.0.1 and later has tooling support for batch processing in Compute Grid.4 zIIPs, zAAPs, and the newer "zAAP on zIIP" approach




attributes of IBM's Compute Grid because it operates within the same Java EE environment as OLTP. The runtime (WebSphere Application Server) can coordinate the execution.

Java batch execution within the WAS Java EE environment

WebSphere Application Server is IBM's Java runtime environment that implements the Java EE standards. WAS is a fully compliant Java EE server runtime.

Java EE is in many ways an online processing runtime model, but it is not limited to just that. It has the ability to execute asynchronous threads. This allows long-running tasks to operate inside the JVM but outside the timeout constraints of OLTP processing. That is the basis for Java batch execution in WAS provided by Compute Grid.

The IBM WebSphere Compute Grid product provides a set of functions that are installed into a WAS runtime environment. Those functions provide a Java batch execution platform:

Notes: • WebSphere Compute Grid is a product offering supported on all operating system platforms supported by WebSphere Application Server itself. That includes Windows, AIX, Linux, z/OS and others.

• With the release of WebSphere Application Server V8.5 the Compute Grid product function has been incorporated into the WAS product itself. The Java batch function provided by Compute Grid V8 is identical to the batch function included with WAS V8.55.

WebSphere Application Server and Compute Grid are explored at an overview level under the heading "Technology Overview" starting on page 11.

Java batch programming and execution framework

Compute Grid provides a batch programming framework that provides a set of valuable services to Java developers. For example:

• A means of using common data access mechanisms to read or write data, iterate through the data records, and map the records to data objects6. This eliminates the need for developers to "roll their own" function, which in turn allows the developers to focus on business logic. It also avoids the build-up of custom code which is costly to support in the future.

• A means of specifying transactional checkpoint intervals. The Batch Data Stream framework maintains awareness of the checkpoint status and handles commits and rollbacks, as well as job pausing and job restarting at these checkpoints.

• A means of declaring job properties separate from the actual Java application code. This is done with an XML file called "xJCL" which Compute Grid reads to understand such things as the number of job steps to process, the Java classes to call for each step, and many other operational parameters related to the job.

5 The Compute Grid product is still available for customers not yet at WAS V8.5 and not wishing to migrate their environment to that level at this time. Compute Grid V8 is compatible with WAS V7 and V8.

6 This is the "Batch Data Stream" framework. See "Batch Data Stream Framework" on page 23.




• A means of defining job execution behavior when specific data records are seen as invalid. This allows otherwise good data to operate when occasional bad records are encountered.

Writing Java batch programs that provide all these functions will prove to be a time-consuming process and expensive to maintain in the long run. Compute Grid provides a Java batch programming and execution framework which your Java batch programs use for such functions. Compute Grid is IBM-supported Java batch middleware. You focus on the Java batch logic and IBM focuses on the Java batch runtime.

Integration of existing (and valuable) non-Java assets

The trend towards batch processing written in Java does not imply the elimination of existing batch assets. COBOL is the most commonly cited form of existing batch asset, though others exist (such as assembler).

While some COBOL batch assets are being reengineered to Java, most continue to be of value to the business in their present form. The desire is to not replace the COBOL in all cases but rather to integrate existing COBOL assets with newer Java batch processes.

To address this, the Compute Grid product (as well as the WAS V8.5 product) provide a "COBOL Container" technology. This allows compiled COBOL modules to be run within the WAS z/OS server address space, which the Java program invoking the COBOL across an IBM-written JNI interface. This allows existing COBOL assets to be used in concert with Java batch processes, with invocation of the COBOL accomplished with a minimum of latency and code path.

This topic is explored more fully under "The COBOL Container" starting on page 33.

The parallel nature of some batch processing

Some batch processing designs are natural candidates for breaking the process up into smaller jobs and running them in parallel. This is particularly true when the source data is arranged in relational data stores in such a way where separate parallel jobs may access the data without data access contention.

Go back to the earlier example of a transactional batch process that calculated the tax owed for each of a million customers on record. Imagine the customer billing records were indexed on customer number. The nature of that batch processing suggests potential for benefit if the customer records were partitioned into ten ranges with ten separate batch jobs each operating on its own range of customer billing data. A job that used to take 10 hours could, in theory, take 1 hour7.

Compute Grid has a facility called the Parallel Job Manager which provides a means of programmatically indicating how large jobs may be partitioned and executed in separate batch execution end points. The Parallel Job Manager handles the partitioning and dispatching and monitors all sub-jobs as part of its overall submitted job management.

In Compute Grid Version 8 (and WAS V8.5) several enhancements to the Parallel Job Manager have been provided. See "The Parallel Job Manager function" on page 28 for an overview of this function. See "Parallel Job Manager and enhancements in CG 8 (and WAS 8.5)" on page 65 for specifics on the enhancements.

Integration with critical enterprise scheduling tools

Enterprise scheduling tools, such as IBM's Tivoli Workload Scheduler, are a critical component in the overall job execution management function in many companies. The emergence of Java batch has not eliminated the need for comprehensive job scheduling and coordination. The reverse is the case: the requirement is to insure Java batch execution can be integrated into existing scheduling functions.

7 Assuming the processing capacity was present to execute all ten jobs concurrently, and assuming no other contention issues were to arise. The point here is that where parallelization suggests itself there is the potential for benefit.




Compute Grid provides an enterprise scheduler integration function called "WSGRID." It allows enterprise job schedulers to submit JCL which runs a program that makes contact with Compute Grid and requests the submission of the Java batch job within Compute Grid.

Throwing a request over the wall is not enough, however. The enterprise job schedulers are also interested in when the Java batch job completes and the success or failure of the submitted Java batch job. The enterprise schedule may have other jobs whose submission is conditional upon successful completion of the Java batch job.

Further still, collection and retention of the Java batch job log is important, which means the Java batch output from Compute Grid must find its way back to JES spool.

The WSGRID function takes all this into account -- it maintains contact with Compute Grid for the duration of the job execution within Compute Grid; it streams the Java batch job log records back to JES spool; and it reports back to the enterprise scheduler the return code seen from execution in Compute Grid.

See "WSGRID and integration with existing job schedulers" on page 27 for more on this function of Compute Grid.

Summary

The objective of this introductory section was to establish the context for Java batch processing and to place that within a framework of understanding of IBM's Compute Grid function.

The next section of this document will provide a more detailed overview of WebSphere Application Server and WebSphere Compute Grid.

The final section will provide even greater detail on specific elements of Compute Grid V8, which is identical to the Java batch function that has been incorporated into the WebSphere Application Server V8.5 product.

Most of the functions of Compute Grid and WebSphere Application Server are in fact common across all supported operating system platforms. But some functions are exclusive to z/OS. For example, the COBOL Container technology is a z/OS exclusive. Other z/OS platform exploitation performed by Compute Grid z/OS includes z/OS WLM workload classification and the writing of SMF usage records.




Technology OverviewWebSphere Application Server Version 8.5

WebSphere Application Server (WAS) is IBM's implementation of the Java EE server runtime model. As such, WAS implements a wide range of industry standards for Java programs such as standards for servlets and JSPs, EJBs, data access such as JDBC, and a many others.

IBM provides WebSphere Application Server all its server platforms -- z, p, i and x. WAS is also supported on HP-UX and Solaris.

WAS Version 8.5 is IBM's latest release of WebSphere Application Server. It was made available in June of 2012.

Functional services made available to applications by WAS

The purpose of an application server is to provide a set of functions and services to applications. Applications stay focused on the business logic and let the server platform provide common functions and services:

Those services are exposed as documented application programming interfaces (APIs). Further, the functions and APIs in WebSphere Application Server are all industry standards8.

The following URL is the WAS V8.5 Information Center:http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp

If you search on the keyword rovr_specs you will get a page that summarizes all the APIs and specifications supported by WebSphere Application Server by version and release.

You will see that WebSphere Application Server supports a long list of standards9.

Consistent support of specification and APIs across platforms

As mentioned earlier, IBM supports WebSphere Application Server on many different operating system platforms -- z/OS, i/OS, AIX, Linux, Windows, HP-UX and Solaris.

Key Point The specifications and APIs supported by WebSphere Application Server are consistent across all supported operating system platforms. That means for any given version or release, if one platform supports a specification, all other platforms will as well.

This provides applications with platform mobility -- that is, an application written to the supported specifications will run on any WAS platform at that same version and release level10.

Does that mean there is no difference between platforms? No. Each platform provides unique attributes that may be exploited by WAS below the standard specification line. That is the key to understanding WAS on z/OS. The z/OS operating system provides a long list of exploitable functionality which WAS on z/OS takes advantage of.

8 Meaning people from different companies in the industry met and agreed to the specifications. Standard specifications allow for broader acceptance of the technology. The specifications do not dictate how specifications are to be implemented, just what the programming interfaces must be. Vendors then compete on the quality and completeness of their implementations.

9 It is not necessary to understand what each of those standards provides. It is better to focus on the fact that a given standard is or is not supported, and the specification level supported for a given release of WAS.

10 An perhaps different version and release levels, but for that case you would need to inspect the application to make sure there are no issues with function removal or deprecation.



http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp

http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-zos&topic=rovr_specs


WAS z/OS and System z platform exploitation

As mentioned, WAS on z/OS takes advantage of specific functions of the operating system. This is done below the standard specification line. The specifications and APIs supported by WAS on z/OS is the same as that support on any other platform for that release.

The following picture illustrates this concept:

The following table summarizes some of the key z/OS exploitation that takes place:

WLM The multiple-JVM model of WAS z/OS (the controller and servant model) has at its heart z/OS WLM acting to classify and route work. WLM is used to route certain work within WAS z/OS clusters.

SAF WAS z/OS uses the Security Access Facility (SAF) of z/OS for a wide array of security profiles, as well as a place to store digital certificates.

SMF WAS z/OS uses Systems Management Facility (SMF) to log detailed records of request activity processed by WAS. This provides a great deal of valuable information that may be used for analysis, capacity planning and billing purposes.

JES The z/OS Job Entry Subsystem (JES) is used to start and stop WAS z/OS servers as well as hold server logging output routed to JES spool.

Cross Memory z/OS permits authorized processes to copy memory from one area of virtual memory to another. WAS z/OS takes advantage of this in a variety of ways to provide very high speed transfer of data with a minimum of latency.

Specialty Engines

System z specialty engines (zIIP, zAAP) provide a means of offloading certain classes of work to lower-cost processors. Work executed on the specialty engines do not count towards other software license charges. The Java provided with WAS z/OS works with the dispatcher of z/OS to perform this offload function.

There is a great deal of detail left out of that summary, of course. If you are interested in more detail the following Techdoc articles will be of interest:http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4848

The presentation handouts from a technical hands-on workshop dedicated to the topic of WAS on z/OS and the platform exploitation that takes place.

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101532

A presentation (with speaker notes) on the question "Why WAS z/OS?" The presentation details the value attributes of the System z and z/OS platform and explains how WAS z/OS benefits from them.

Key Point Applications that run on WAS z/OS benefit from this exploitation that takes place under the open specification line. That includes functions such as Compute Grid. Compute Grid on WAS z/OS allows Java batch to execute with the benefits of z/OS.




http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4848


See "Other sources of information on WAS z/OS" on page 17 for a more comprehensive list of WAS z/OS technical documentation.

Servers, nodes, clusters and cells

WebSphere Application Server across all platforms has the concept of "servers," "nodes," "clusters" and "cells." Those terms represent ways in which the JVMs are logically organized to form the WAS runtime topology. The following picture illustrates this, with notes that correspond to the numbered blocks:

1. An application server contain a JVM11 in which the applications run. An application server may host a single application (for isolation purposes) or multiple applications.

2. A node is a logical collection of application servers on a given server platform. For example, a given Windows server, or a z/OS logical partition (LPAR).

3. A cluster is a logical grouping of application servers, often on different server platforms, for the purpose of providing additional availability.

4. A cell is a logical grouping of nodes for the purpose of defining the extent of control for the administrative server, called the "Deployment Manager." Cells are often used to provide administrative isolation between, for example, test and production environments.

The general concepts of servers, nodes, clusters and cells apply to WAS on all platforms. Where WAS on z/OS differentiates is in the design of the application server.

The z/OS application server design

WebSphere Application Server on z/OS has a unique application server model. Where WAS on every other platform has a single JVM for an application server, WAS on z/OS has multiple JVMs. The following picture illustrates this at a very high level:

11 On z/OS the "multi-JVM" server model implies at least two JVMs (controller and servant) and potentially more if multiple servants are configured.




The controller region JVM hosts IBM-written code that does all the request handling. User application code does not run here.

The controller queues work request with WLM, which then dispatches the work to a servant region. Earlier we spoke of WLM as being an element of the z/OS system exploited by WAS z/OS ... this is a summary view of that exploitation.

The picture illustrates stacked servant regions because it is possible to configure multiple servants to exist behind any given controller. Each servant would have its own physically separate address space and physically separate JVM.

Multiple servant regions provide many benefits, including increased availability of applications and dynamic scaling of application resources.

Note: There is a great deal of sophistication to this application server model. The WP101740 document at ibm.com/support/techdocs provides much more information.

For the purposes of setting context for the Java batch function of Compute Grid or that provided with WAS V8.5, the high-level overview offered in this document is sufficient.

Accessing applications in WAS

Applications provide their intended function to those people or processes that make a request of the application. There are several essential ways in which requests may flow into an application hosted in an application server:

HTTP The request flows on a TCP connection to the HTTP listener port of the application server. A common form of this would be web browser requests, where browser HTML flows over the HTTP connection. But this request type also includes web services, where SOAP or other HTTP verbs are used to invoke an action and receive a result.

Message Queueing With message queueing a program outside of WAS puts a message on a queue, and the application in WAS retrieves the message from the queue. The application may either perform a GET operation to retrieve the message or be triggered to retrieve the message if the application is a Message Driven Bean (MDB).

A common message queueing product used is IBM's WebSphere MQ. Another messaging queueing function is the Service Integration Bus (SIBus), which is a Java-based queueing mechanism built into WAS.

IIOP Internet Inter-ORB Protocol is a means of one Java program invoking another Java program remotely over a TCP network. A very common form of this request type is internal to an application server where a servlet invokes an EJB.

WOLA WebSphere Optimized Local Adapters (WOLA) is a WAS z/OS exclusive function for cross-memory transfer of requests into an application.

Other Other request types exists, such as Session Initiation Protocol (SIP) and various internal work requests.

Again, considerable detail is left out of that summary chart. The point here is that WebSphere Application Server supports many different means of inbound requests to initiate action and provide results.

Java Batch When you get to the WebSphere Compute Grid section (page 18) you will see that Compute Grid -- which may be thought of as a very sophisticated application that runs inside of WebSphere Application Server -- makes use of many of those request types to process Java batch jobs.




Accessing data from application in WAS

For an application to performance meaningful value it requires data to act upon. Some of the data may come from the requester, but in most cases the request triggers the application to go get data from some other data store, such as a database.

The following table summarizes some of the ways in which applications in WAS may access data outside of WAS:

JDBC JDBC provides an industry standard means of accessing relational databases, such as IBM's DB2. Vendors of relational databases write and supply "provider code" that performs the actual connectivity to the database. Applications make use of the JDBC interface to request the data, and the underlying provider code does the actual work to connect and retrieve the requested data.

JCA JCA provides an industry standard means of accessing non-relational data systems, such as IBM's CICS or IMS products. Vendors of these non-relational systems write and supply "resource adapters" that install into WAS. The resource adapters provide the code that does the actual connectivity function. Applications make use of the JCA interface to request the data, and the underlying resource adapter code does the actual work to connect and retrieve the requested data.

JMS JMS provides an industry standard means of accessing message queueing systems, such as IBM's MQ, or the built-in Service Integration Bus (SIBus) function. Applications seeking data can place a message on a queue, which is picked up by another program and the data requested is made ready. That program then returns via the message queue the requested data. The application in WAS pulls the message from the queue and processes the data.

Web Services An application in WAS may form up and send a web services request to a service provider elsewhere in WAS, or anywhere else on the Internet.

IIOP An application in WAS may form up and send an IIOP request to an EJB residing in the same server, the same cell, or another Java EE environment on the Internet.

WOLA WOLA is a z/OS-exclusive cross-memory request function. It can be used to request data of CICS, IMS or batch programs running outside of WAS z/OS. Requests outbound from WAS z/OS make use a WOLA JCA adapter.

Other Such as reading from a UNIX file, MVS data set or VSAM file; opening a TCP socket to another process; or processing FTP to retrieve data.

In short, there is considerable flexibility in the means by which an application may reach out of WAS to retrieve data.

Java Batch Java batch applications running within the Compute Grid function have access to all these forms of data retrieval. That illustrates an important point about Compute Grid: it is built upon the full function Java EE WebSphere Application Server platform. Therefore, services provided by WAS are available to the Java batch application12.

Transactionality

Applications that make updates to data need to be aware of the need to maintain transactional consistency across all the data affected. That means, for example, that changes made to separate data stores be committed against both or rolled back against both. But never a case where data inconsistency is introduced by updating one but failing to update the other.

WebSphere Application Server provides a transaction manager service to applications that run on WAS. That means that applications themselves do not need to write the transaction manager code, the application simply invokes the transaction service of the platform.

12 The Batch Data Stream framework provides built-in patterns for JDBC read and write, JPA read and write, file read and write, and z/OS data set read and write. Other data access methods would require the Java batch application to implement the data access logic.




WAS is not the only application server with a transaction manager ... CICS and IMS provide that service as well. This is where transaction management requires some care, which is why having that function as a service of the platform is so valuable -- when more than one transaction participant is involved, all the participants must coordinate to make sure the data commits are done together or rolled back together. That degree of coordination is part of the design of WebSphere Application Server as well as CICS and IMS.

When multiple participants are involved in a transaction something must act as the repository of information of where the participants are in any given transaction. On z/OS that function is provided by Resource Recovery Services (RRS). For non-z/OS platforms that function is provided by in distributed XA processing in each manager.

Java Batch Batch processing very often requires that transactional integrity be maintained. The Compute Grid function provides checkpoint services to Java batch applications so effective and efficient transactional commits may be performed. The Compute Grid function maps those requests down upon the transaction management function of the WAS platform.

Once again, the message is this -- the Java batch application developer does not need to code this often-complex functionality. The Java batch application developer needs merely to indicate the checkpoint requirements in xJCL.

High availability

When application up-time is important to the business, the question of making the application "highly available" comes to the surface. The topic involves far more than simply duplicating server hardware resources.

Earlier we mentioned "clusters" as a logical element of a WAS runtime topology. Clusters are a means of duplicating application and application servers across multiple server platforms as a means of reducing single point of failure. WAS clusters perform other availability services such as replicating HTTP session objects, or failing over server resources to recover transactions.

On z/OS the availability is enhanced further by the Parallel Sysplex function of System z. At a very high level, Parallel Sysplex is a means of clustering z/OS operating system images with a data sharing facility in the middle called the Coupling Facility (CF). What's more, many IBM middleware products such as CICS, DB2, IMS and MQ are "Sysplex-aware" -- that is, they take direct advantage of Parallel Sysplex to augment availability.

WebSphere Application Server for z/OS supports clustering across z/OS images and benefits from the Sysplex-aware middleware. When people speak of System z being "highly available," they are alluding to all this and much more.

Java Batch Because the Compute Grid function exists within WebSphere Application Server, highly-available batch processing can be built upon what WAS itself offers. When WAS is on z/OS then the availability features of Parallel Sysplex may be employed to enhance the availability of the Java batch processing.

For example, Compute Grid operating in a cluster on WAS z/OS may process a batch job on one logical partition (LPAR). If an outage is experienced on that LPAR, the batch job may be restarted on the cluster member of the other LPAR. Compute Grid will pick up at the last checkpoint. The shared data in the middle of Parallel Sysplex means Compute Grid components on one LPAR have access to data created and stored on another LPAR.




Compute Grid incorporated as function in WAS V8.5

Starting with WebSphere Application Server Version 8.5 (made available in June of 2012), the Java batch function equivalent to that offered in IBM WebSphere Compute Grid V8 was made part of WAS itself.

IBM WebSphere Compute Grid V8 is still offered for those customers who are at WAS V7 or V8 (but not yet V8.5) that wish to use the Compute Grid function, but do not wish at this time to migrate their application server environment to WAS V8.5.

The following picture represents this13:

Other sources of information on WAS z/OS

There is a wealth of information on WebSphere Application Server for z/OS. The following list attempts to organize some of those sources of information:http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp

The InfoCenter is the primary source of reference information for WebSphere Application Server14.


The presentation handouts from a technical hands-on workshop dedicated to the topic of WAS on z/OS and the platform exploitation that takes place.


Provides a step-by-step cookbook for the use of IBM Installation Manager used to install WebSphere Application Server for z/OS V8 or V8.5

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4686http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4944

Spreadsheets used to format the input to the WebSphere Customization Tool (WCT) that generates the jobs to create a runtime on z/OS. PRS4686 is for V8, PRS4944 is for V8.5.


A presentation (with speaker notes) on the question "Why WAS z/OS?" The presentation details the value attributes of the System z and z/OS platform and explains how WAS z/OS benefits from them.

13 The picture uses the word "augment" to suggest the adding of the Compute Grid function to a WAS runtime. Augmentation involves running an IBM-supplied process (or on z/OS, a job) to update the WAS configuration to recongize and use the Compute Grid function.

14 See "Search Scope" starting on page 77 for information about setting search scope when using the InfoCenters.








http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp



Provides a very detailed look at the architecture and operations of the multi-JVM application server model of WAS z/OS and the role of WLM as part of that design.


Provides a detailed look at the WebSphere Optimized Local Adapters (WOLA) function of WAS z/OS.

In addition to those shown here, the IBM Advanced Technical Skills (ATS) organization "Techdocs" website provides the search entry point to other documents on WAS z/OS as well as a wealth of other topics as well:

http://www.ibm.com/support/techdocs

WebSphere Compute Grid

In this section we'll take a technical overview look at the Compute Grid function. The topics presented in this section apply to both the IBM WebSphere Compute Grid product offering, or the equivalent function that has been incorporated into the WebSphere Application Server V8.5 product offering15.

Components of the Compute Grid model

The following picture and the notes that accompany it explain at a high level the components of the Compute Grid model:

Note: That picture may look a bit complex at first. It will become more clear as you read the notes that follow and gain more understanding of the Compute Grid function.

The message this picture sends is that Compute Grid is a very structured and managed Java batch environment16. Compute Grid provides many of the same kinds of management and control for Java batch as you enjoy today with traditional z/OS batch.

1. Any discussion of Compute Grid starts with the understanding that the Java batch function is built upon the foundation of WebSphere Application Server17. This is why the overview of WAS was offered starting on page 11. All of the functional services offered

15 See "Compute Grid incorporated as function in WAS V8.5" on page 17 for a graphical representation of the relationship between the two.16 Implied in this picture is another message -- if you attempted to develop your own custom Java batch execution environment you would,

eventually, end up creating many or all of these components. That would imply a great deal of custom middleware to support and maintain. With Compute Grid that middleware is not custom, it is vendor-supported.

17 It may help to think of Compute Grid as a sophisticated IBM-written Java application deployed into WAS.



http://www.ibm.com/support/techdocs




by WAS itself are functional services available to Compute Grid and, by extension, Java batch applications running in the Compute Grid environment.

2. Compute Grid relies on an XML document submitted by the user to understand the nature and elements of the Java batch job to process. This document is called the "xJCL" because it provides the same conceptual elements JCL has for decades -- information about the job, job steps, source of input and output, substitution variables and many other piece of information18.

3. The xJCL document is submitted to the Job Dispatching Function19, which interrogates the submitted xJCL, determines where the requested Java batch class files reside, and prepares the environment to dispatch and launch the batch program.

4. The Job Dispatching Function accepts the xJCL through one of many programmatic interfaces -- HTTP, web services, command line, RMI and MDB. This set of interfaces makes integration of the Compute Grid function with other processes easier.

5. The Dispatching Function determines where the Java batch program class files reside and signals to the Batch Execution Endpoint Function to execute the class file. Multiple endpoints may exist, and they may be clustered for availability or scalability.

6. The Dispatching Function and the Endpoint Functions maintain information about jobs and the job states in a set of relational tables.

7. The Java batch logic is developed and deployed as WAS applications deployed into the endpoint server. These applications are written as POJOs20 using Eclipse-based tooling such as IBM's Rational Application Developer.

8. The development tooling makes use of a set of development class libraries supplied with the Compute Grid function. Or, in the case of IBM's Rational Application Developer at Version 8.0.1 or above, the tooling support is included.

9. On z/OS, if the batch application needs to make use existing COBOL modules for its execution, it may invoke those COBOL modules by using the COBOL Container function21. This container consists of IBM-written JNI and native code that creates a separate z/OS LE environment, which operates inside the WAS z/OS server servant region address space.

10. If the xJCL indicates the job is to be processed in parallel, then the Parallel Job Manager is called to partition the top-job into a set of sub-jobs and dispatch those sub-jobs to endpoint servers. The Parallel Job Manager monitors execution of all the sub-jobs and reports on the status of the top-job22.

11. Finally, the WSGRID function23 is a means of integrating enterprise scheduler functions with the Compute Grid function. WSGRID is a messaging program, invoked as a job using normal JCL, that places a job submission message on a queue which is picked up by an MDB function in the Dispatcher Function. The WSGRID program stays active for the duration of the batch job within Compute Grid. Job log output from the batch job executing in Compute Grid is sent back to WSGRID in a series of messages placed on an output queue. To the enterprise scheduler function, WSGRID is the batch job, but behind WSGRID the Compute Grid function is executing the Java batch job.

That's a somewhat complex picture, and with time you will become more familiar with it. It serves as a conceptual framework for the details that will follow.

18 IBM Rational Application Developer 8.0.1 and later has tooling support for developing Java batch applications for Compute Grid, including an "xJCL editor" function.

19 You will see this referred to in the Compute Grid documentation as the "Job Scheduler." We prefer "Dispatcher." The term "Scheduler" tends to confuse this function with enterprise job scheduler functions such as IBM Tivoli Workload Scheduler.

20 Plain Old Java Objects ... in other words, they are not developed as EJBs.21 See "The COBOL Container" on page 33 for more detail on the COBOL Container function.22 See "The Parallel Job Manager function" on page 28.23 See "WSGRID and integration with existing job schedulers" on page 27.




Jobs, steps, the Java batch application code, and xJCL

Compute Grid operates with the same general concepts as batch programs on the mainframe have for decades. If you are familiar with traditional batch processing using JCL and compiled module libraries on the mainframe, you will see many similarities in what we discuss here.

The following picture illustrates at a high-level the key concepts of "job," "step," "application," and "xJCL."

• Job -- this is what is submitted to Compute Grid and what Compute Grid reports on when showing status. A job may be relatively simple or more complex. If more complex it will likely consist of multiple steps ...

• Step -- a step represents a discrete set of processing function to be executed as part of the submitted job. A simple job may consist of only one step. More complex jobs may consist of many steps. The code that executes when a step executes is held as a Java class file, which is part of a deployed Java batch application ...

• Application -- the code that performs the job step is contained within a Java class file, and the class files that make up all the steps in a job are packaged within a WebSphere Application Server EAR24 file. WAS and Compute Grid are aware the deployed application is a batch application, and the class files inside the EAR are known to WAS and Compute Grid as well.

• xJCL -- this XML file25 tells the story of the batch job and the steps contained within. Each step specifies the class file to run along with other declarations about how to run the batch program step. The xJCL is submitted to the Job Dispatching function, which reads and understand the xJCL and then signals to the batch endpoint (the server where the batch application is deployed) to begin execution of the job and its steps.

24 EAR = Enterprise ARchive file ... the standard packaging and deployment file type for WAS.25 It's XML, but the general principles are very similar to regular Job Control Language (JCL).




Submitting Jobs to Compute Grid

In traditional batch processing on z/OS, jobs are submitted using Job Entry Subsystem, or JES. JES reads the submitted JCL and from that determines what programs to run, and how to run them.

Compute Grid operates in a similar way, except rather than JES being the function that initiates the job execution, it's the Job Dispatching Function of Compute Grid that does that. And rather than traditional JCL26 (with all those lines starting with // ), the job declaration is done with XML in a file called xJCL.

The Job Dispatching Function runs inside an application server you designate. It understands what batch applications are deployed and where they run; it understands the Compute Grid database tables; it understands xJCL; and it accepts job submissions across several different interfaces:

• HTTP -- a browser interface27 allows you to submit jobs and monitor the status of those jobs. This is the easiest interface to start with.

• Command Line -- a shell script is provided that accepts command line arguments to submit and monitor jobs. The shell script opens up an HTTP connection to the Job Dispatching server and uses web services to pass in the commands you enter.

• Web Services -- the same web services interface used by the command line shell script may be used by a custom program you write that uses web services to submit jobs.

• RMI -- allows Java programs to interact with the Job Dispatcher function using RMI

• MDB -- this is how enterprise scheduler products integrate with Compute Grid using a supplied program called WSGRID. For more details on this, see page 27.

Job state -- submitted, executing, ended ... and more

When you submit a job to Compute Grid it goes through three states:

That's the normal cycle for a batch job. But the picture is actually more sophisticated than that because Compute Grid allows you to act upon excuting jobs to cancel, stop or suspend execution. So the picture of the possible states looks more like this28:

26 Traditional JCL/JES is used to integrate Compute Grid with enterprise scheduler functions, such as IBM's Tivoli Workload Scheduler. See "WSGRID and integration with existing job schedulers" on page 27.

27 Known as the "Job Management Console," or JMC for short.28 In reality there's even more to this. For example, when jobs go through a "pending" state when they are canceled, stopped or suspended.

The InfoCenter has a good article on job states. Search on the string: cgrid_xdbgstate




For transactional batch jobs the restartable and suspended states are entered at a defined checkpoint, and Compute Grid maintains that job's checkpoint in one of the database tables. When you restart or resume the job, Compute Grid will pick up where it left off -- starting from the last checkpoint.

Note: What's more, if you seek highly available batch execution you may define clusters across LPARs with a shared database. In the event the original cluster member is lost, the resumed or restarted job picks up execution at the checkpoint on a surviving cluster member.

Compute Grid provides all this supporting function so you may focus on the batch logic.

Transactional batch and Compute Intensive batch jobs

The Compute Grid function supports two kinds of Java batch jobs:

• Transactional -- this is what most think of when they think of batch ... the looping through a set of records with processing performed on each record until the last record is processed.

The Compute Grid function provides a rich set of functions to the transactional batch jobs that run. For instance, managing checkpoint intervals is a function of the batch container provided by Compute Grid. Record input and output functions make handing input and output easier. For more, see "Functional services provided by Compute Grid" on page 23.

• Compute Intensive -- the name implies a batch job that is launched and then performs a series of internal computations with relatively little I/O. An example of this would be numerical analysis of geological seismic information.

One use of the Compute Intensive model is as a means of deploying into Compute Grid "Java main" applications. Rather than refactor those Java programs to the Compute Grid framework, you may simply wrapper them as Compute Intensive batch applications and deploy them as is.

This distinction between Transactional and Compute Intensive batch applications is offered here because (a) the terminology comes up in other Compute Grid documentation, and (b)




when developing Compute Grid applications certain function services are available to transactional batch but not to compute intensive29.

Key Point: As you think about how you might use Compute Grid for your Java batch programs, keep in mind these two categories of jobs. If your program is a traditional loop process then it's mostly likely a transactional batch model. Otherwise, consider mapping it to the compute intensive model.

Functional services provided by Compute Grid

The primary purpose of the Compute Grid function is to provide a set of Java batch execution functions to the batch applications that run and to the administrators responsible for batch processing.

The objective is to enhance the productivity of batch developers and batch administrators. The functional services are provided in IBM-supported middleware. That makes coding the batch applications simpler, and it makes administering the environment easier.

The following picture provides a way to organize these functions into three categories:

Application Support Functions

Development class libraries and tooling support

Compute Grid comes with a set of development class libraries that facilitate the coding of Java batch applications to run in Compute Grid. If you have IBM Rational Application Developer 8.0.1 or later the tool has an optional feature for Java batch, including the development class libraries and artifact creation wizards30.

xJCL, substitution properties and external job and step declarations

Job and step properties do not need to be coded within the application. They may be specified in the xJCL job control file as properties, and those properties are available to the Java batch application during execution.

Batch Data Stream Framework

The Batch Data Stream (BDS) framework helps abstract data input and output, making development of the I/O function of Java batch jobs much easier. xJCL properties define to the BDS the input and output streams for a job step. BDS iterates through the result set, invoking your job step batch logic for each record.

29 For example, the "Skip-Record Processing" function (see page 52) is for transactional batch only.30 Including a wizard to assist with the creation of xJCL.




BDS comes with a set of packaged patterns for common read and write operations. The following table summarizes the current patterns available:

Data Read Data Write

JDBC JDBCReaderPattern JDBCWriterPattern

JPA JPAReaderPattern JPAWriterPattern

File Byte Data ByteReaderPattern ByteWriterPattern

File Text Data TextReaderPattern TextWriterPattern

z/OS Data Set RecordOrientedDatasetReaderPattern RecordOrientedDatasetWriterPattern

Java batch developers do not need to write their own Java stream handlers for these popular patterns. Compute Grid provides them in the form of Batch Data Stream patterns. Application code simply uses the methods of the pattern to read and write the data as needed.

Checkpoint algorithm

Transaction batch processes that update data records must specify when data udpates are to be committed. It's a balance between performance (fewer commits) and recoverability (more commits).

Compute Grid provides an interface and algorithm so batch applications do not need to do this checkpoint processing themselves. Properties in the xJCL for a Java batch job specify the checkpoint algorithm -- either number of records or amount of time that has passed -- and Compute Grid (along with WAS under it) manages the commit and rollbacks as needed.

If a batch job is stopped or paused, Compute Grid will maintain the checkpoint state information in its relational table. When the job is restarted or resumed the processing will pick up at the last committed checkpoint.

This kind of checkpoint processing could be written within each Java batch program, but that would not be efficient. Better to make the checkpoint processing part of the Java batch execution platform and allow applications to simply request the service.

Configurable transaction mode

In earlier versions of Compute Grid, the transaction mode was fixed at "global." With Compute Grid Version 8 (as well as the Compute Grid function that is part of WAS V8.5) this is configurable by step as either "local" or "global."

Where applicable, local transaction mode improves performance since no global transactions are started. The connection object is shared between the batch container and the application code.

This is one of the items in the Technology Focus section. See "Configurable transaction mode" on page 64.

Results algorithm

This provides an interface for the setting of a step's return code. This information is maintained in a "JobStepContext" object which is available for the life of the job.

Step return codes may be used to influence the running of subsequent steps in the batch job. Condition declarations in the xJCL will determine, based on previous step return codes, whether a step will run or not.

The results algorithm also allows developers to set the overall job return code based on the return codes of all the steps. When using the WSGRID mechanism to integrate enterprise schedulers with Compute Grid, a job's overall return code is what gets fed back to the enterprise scheduler. That's important because the




enterprise scheduler often determines subsequent processing based on return code it sees for a processed job. With the results algorithm the developer has control over step and overall job return codes that are produced.

Skip record processing

Transactional batch applications31 occasionally encounter exceptions when reading and writing records. Unless those errors are handled, the processing will be rolled back to the last checkpoint and the job thrown into a restartable state. For the occasional error it makes better sense to provide a way to skip the record, log the skip, and continue on to the next record.

This type of function could be custom-coded in the application. But that would not be consistent with the objective of container managed services for applications. So a better place to provide this would be a built-in function of the batch execution framework. That is what the Compute Grid skip record processing is -- a mechanism built into the Batch Data Stream Framework to handle record skipping when exceptions are seen.

The skip record processing feature allows you to define how many such read or write errors may occur, and what types of exceptions the batch container should include or exclude. In addition, it provides a way to register your own "skip listener" code when a record is skipped. That would be useful in logging information about the skipped record so auditors may review and take corrective action on that record.

The properties that control this behavior -- the skip count and the exceptions to include or exclude -- are coded in the xJCL for the Batch Data Stream.

This is one of the items in the Technology Focus section. See "Skip-record processing" on page 52 for more.

Retry step processing

The execution of a transactional batch job32 step may occasionally encounter errors. At that point there are two courses of action -- set the return code and move on to the next step33, or retry the job step.

This function provides a mechanism to specify how many retries to attempt, and what types of exceptions to include or exclude for job step retries. The properties that control this behavior are coded in the xJCL for the job step.

This is one of the items in the Technology Focus section. See "Retry-step processing" on page 57 for more.

Job Listener

The Job Listener is a function of the container that enables developers to invoke custom setup or cleanup code at the beginning and end of the job and the steps within the job. This applies to both transactional batch and compute intensive batch jobs.

This is one of the items in the Technology Focus section. See "Job listener" on page 50 for more.

31 This function applies to transaction batch only, and not compute intensive batch jobs.32 This function applies to transaction batch only, and not compute intensive batch jobs.33 And based on conditional processing, the next step may not run based on the return code of the previous step, which would mean the

whole job would cease processing.




Administrator Support Functions

Job Dispatcher (and its role in overall Compute Grid)

The Job Dispatcher function34 is the primary interface into the Compute Grid function. Through it administrators submit jobs, review the results, control the jobs (start, stop, suspend, restart, resume), and purge jobs.

The Dispatcher understands the whole Compute Grid environment. It controls the dispatching of submitted jobs to batch endpoints where the batch application code resides.

Key Point: The Job Dispatcher function is integral to the overall functioning of Compute Grid. It makes available many administrator functions that would not otherwise be available in less powerful Java batch approaches such as JVM launchers.

A custom solution to do everything the Dispatcher Function does would involve a great deal of custom code that would then require custom support. Hence the value of the Job Dispatcher in the overall Compute Grid scheme.

Security: authentication and user roles

When security is enabled for the WAS runtime on which the Compute Grid function runs, then access to the "Job Management Console" (the dispatcher function, or "JMC") is controlled35. On z/OS SAF may be used as the repository for userids and passwords as well as the EJBROLE definitions that allow access.

Further, there are three classes of roles that define the degree of authority the ID has once in the Job Management Console. They are:

administrator Full authority -- submit jobs, control jobs, view and download any job logs.

submitter Partial authority -- may submit jobs, but may only control jobs that ID submitted. Able to download any job log.

monitor Limited authority -- unable to submit jobs, but may view jobs and download job logs for any job.

Integration with enterprise scheduler tools

Enteprise scheduler tools (such as IBM Tivoli Workload Scheduler) are critical components in many customer batch processes. Those tools may integrate with Compute Grid so Java batch may be part of larger scheduled batch processes.

How this integration is accomplished is covered under "WSGRID and integration withexisting job schedulers" starting on page 27.

z/OS Platform Exploitation

SMF job usage recording

The Compute Grid function on z/OS has the capability to write SMF records that record the Java activity processed in Compute Grid36. The InfoCenter article found by keyword search cgrid_zosjobusage has details.

There are two record types available -- 120.9 and 120.2037. The 120.9 record contains a superset of information contained in the 120.20 record. Further, the 120.9 records are dynamically started and stopped with the MODIFY command. We recommend those interested in job accounting information for Compute Grid focus on the 120.9 record. Consult the InfoCenter article for more details.

34 Also known as the "Job Scheduler," but again, that name is often confused with enterprise schedulers. So we prefer "dispatcher."35 Access is controlled for other interfaces as well, such as the command line or web services. JMC is the browser interface.36 Compute Grid on all platforms has the ability to write job usage information to a database. On z/OS the capability to write to SMF was

added because SMF is a key part of many customer usage, reporting and analysis processes.37 The story behind why there are two records is somewhat involved and not really central to this document.




WLM classification

WLM is a z/OS facility for the classification of work and the management of resources to complete work based on defined performance goals. WAS z/OS has the ability to identify work by request and have that work classified by WLM.

This ability to identify and classify work per request allows WAS z/OS to differentiate work running in the same server. From that WAS z/OS is able to segment work into separate servant regions so WLM may better manage resources to goals38.

Compute Grid on WAS z/OS has the ability to utilize this capability and have WLM classify batch jobs per request. That means a server running Java batch jobs and defined with multiple servants has the ability to run jobs in separate servant regions, with WLM managing system resources to the goals of each.

WSGRID and integration with existing job schedulers

The use of enterprise job schedulers39 to manage batch execution processes is common in the business world. Therefore, it's important that some means be available of integrating Java batch jobs in Compute Grid with these enterprise job schedulers. For Compute Grid that integration is provided with a function called WSGRID.

The following picture illustrates the key points of this WSGRID integration mechanism:

1. WSGRID is a program40 supplied with Compute Grid that puts a job submission message on a queue, which is then picked up by the Compute Grid dispatcher function. WSGRID is submitted using normal JCL, so to the Enterprise Scheduler Function it looks like any other submitted batch job. Here's the key -- the WSGRID program stays up for the life of the Java batch job that executes in Compute Grid. Why that's important becomes evident a bit later as we discuss how the job output flows back to JES spool.

2. As mentioned, the Enterprise Scheduler Function submits JCL to begin the execution of the WSGRID program. The SYSIN DD of the JCL has the information for Compute Grid Java batch job41 to be submitted (including the xJCL to use), as well as information about the input and output queue to use when communicating with Compute Grid.

38 The document at ibm.com/support/techdocs under search argument WP101740 is the definitive document on how WAS z/OS interacts with WLM for the purposes of work classification. It's a somewhat complex topic. If you have some knowledge of WLM and wish to understand better how WAS z/OS works with WLM, that document is the key.

39 For example: IBM Tivoli Workload Scheduler, CA Workload Automation CA 7, or BMC Control-M.40 WSGRID comes in two forms -- native z/OS and Java. On z/OS JCL is used to submit either form. For Compute Grid on the other

platforms the submission of WSGRID is done using shell scripts or BAT files.41 Use separate JCL for each Compute Grid job you wish to submit, each JCL containing the specifics about the xJCL to submit.




3. The WSGRID program places the Java batch job submission message on a queue, either MQ or the SIBus function of WebSphere Application Server. The Compute Grid Job Dispatching Function has an MDB interface which picks the message off the queue.

4. At this point the job dispatching within Compute Grid functions as it would had you submitted the job using the browser interface. Compute Grid determines where the requested Java batch applications reside and dispatches the job to that endpoint.

5. As the Java batch job executes in Compute Grid the dispatcher function puts job output messages on an output queue, which the WSGRID program fetches back.

6. WSGRID routes the Java batch job output messages to JES spool on z/OS. This is valuable because audit practices and archival on z/OS often starts with job output in JES spool.

7. The return code of the Java batch job in Compute Grid flows back to WSGRID. This is the return code WSGRID ends with, so that is the return code seen by the Enterprise Scheduler function. WSGRID does not end until the job in Compute Grid ends.

The essential message here is that Enterprise Scheduler functions see this process as submitting a job called WSGRID. All the mechanics after that -- the queues, the MDB interface, the job dispatching within Compute Grid -- all are unknown to the Enterprise Scheduler. The Enterprise Scheduler sees the submitted WSGRID job.

The WSGRID job stays up during the execution of the Java batch job in Compute Grid. The job output messages from Compute Grid stream back to JES. The WSGRID job does not end until the job in Compute Grid ends. The return code reported back to WSGRID from Compute Grid is what the Scheduler sees as the return code for WSGRID.

Key Point: Java batch jobs that execute in Compute Grid may be part of a larger batch process managed by the Enterprise Scheduler function.

The Parallel Job Manager function

Transactional batch jobs often have data requirements that allow for the job to be run in parallel sub-jobs. The benefit of this a shortened time to run the job as the execution is split out and run in parallel.

Example: A sample application is supplied with Compute Grid that is called "Mailer," which simulates the mailing (postal or e-mail) of promotional offers to customers in the input database42. The output of this sample is an update in the database for when the mailing took place.

This sample parallelizes processing by dividing the customers based on the U.S. state abbreviation code. There are 50 possible abbreviations, from AK (Alaska) through WY (Wyoming). Two parallel jobs would imply 25 states processed for each; five parallel jobs would imply 10 states processed for each.

At a very high level, here's how this function works:

42 See "A look inside the Mailer sample parallel program" on page 67.




1. The xJCL submitted to the Compute Grid job dispatching function carries with it an indication of how the Parallel Job Manager (PJM) is to handle parallel processing. Here we're illustrating the following:

instances="multiple" A value of "multiple" indicates this is a parallel job.

jvm="multiple" A value of "multiple" indictes this parallel job may run in separate Compute Grid endpoint servers43.

parallel.jobcount value="2" This indicates how many Compute Grid endpoint server instances this is to run across. This value is used to determine the data range parameters.

This initial job submission is referred to as the "top job."

2. The Job Dispatching function communicates with an instance of the PJM function to analyze and determine how the parallel sub-jobs will be constituted. The PJM function resides in every endpoint server; the dispatcher picks one in an endpoint server where the requested application resides.

3. One of the things the PJM function does is invoke the "parameterizer" function to determine the data ranges to supply to each sub-job.

Key Point: The picture is a bit misleading ... the parameterizer code is often user-written44, since the user is best able to understand how the data behind the application is structured and how best to slice the data into parallel chunks.

The picture is a bit misleading because the parameterizer class file will be packaged with the deployed batch application, not outside the endpoint.

This parameterizer function will slice the data into chunks based on the job count value specified in the xJCL45.

43 The alternative is a value of "single" which tells Compute Grid the parallel job may run in one endpoint server, but multiple instances may be dispatched on separate threads within the single JVM.

44 There is a generic parameterizer provided, but it has limitations. Hence this will often be user-written code. See "The Mailer sample parameterizer in detail" on page 71 for more on how that implementation works.

45 That's what the "Mailer" sample does. It creates an array of the 50 state abbreviations, then divides that array into sub-arrays based on the parallel.jobcount value. The SQL query is then based on the portion of the array for that sub-job. The first sub-job query pulls a result set for states AK through KS and works on that sub-set of the total data.




4. The PJM function ships the sub-job information back to the Job Dispatcher

5. The Job Dispatcher function then dispatches the subjobs to the endpoints based on the information in the xJCL. In this example we're showing a hypothetical data range of customer names starting with A through M for one server, and N through Z for the other.

6. Each Compute Grid endpoint function server then executes the sub-job, working on the data range assigned to it.

Not shown in the picture is the collecting up of all the job completion states for the sub-jobs and the rolling up into a completion state for the top job. The following table illustrates the state of the top job based on the results of the sub-jobs46:

Status of Sub-Jobs Status of Top-Job

All sub-jobs complete successfully Top-job is marked as completed successfully

Any sub-job ends restartable with no sub-jobs ended failed

Top-job is marked restartable

Any sub-job ends failed Top-job is marked failed

If the top-job ends "restartable" then you restart the job by selecting the top-job and restarting it. You never act upon the sub-jobs directly.

This is one of the items in the Technology Focus section. See "Parallel Job Manager and enhancements in CG 8 (and WAS 8.5)" on page 65.

Developing Java batch applications for Compute Grid

What follows is a high-level flow of how to approach this topic:

• Settle on development tooling

Either Eclipse or IBM Rational Application Developer will work. RAD at Version 8.0.1 or higher supports the "WebSphere Batch" developer tools47, which "provides batch project support, including job creation wizards, job launcher and xJCL editors supporting the batch programming model for WebSphere Application Server."

Using Eclipse implies more manual creation of Java batch artifacts.

• Establish test runtime

You will need a Compute Grid runtime in which to run the Java batch applications you develop. It does not need to be complex ... a single application server topology will work for simple development and validation. It does not need to be on z/OS ... a validation runtime on another platform will work for testing and validation48.

If your tooling environment is IBM Rational Application Developer, then you have the ability to configure a WAS runtime as part of the RAD installation.

If separate from RAD, and if the level of WebSphere Application Server is at V7 or V8, then you will need to augment Compute Grid on top the WAS profile. Once augmented, then you must configure the environment. The following InfoCenter49 search strings will get you to the articles for each:

Topic InfoCenter Search String

Augmenting Profile tgrid_pro_augment_cmd (command line)tgrid_pro_augment_gui (graphical interface)tgrid_ins_installation_zos (z/OS) See Note

Configuring Job Scheduler tgrid_cgconf

Configuring Endpoints tgrid_cgendpoint

46 InfoCenter search: cgrid_cgparallel47 This is a feature of RAD that is not installed by default. You must explicitly select that feature for installation.48 But not if you are testing the COBOL container function. That requires z/OS.49 http://pic.dhe.ibm.com/infocenter/wasinfo/cgwas80/index.jsp




Note: Augmentation of WAS z/OS requires several steps. The first is to install the Compute Grid "extension" into the WebSphere Customization Tool. This gives the WCT knowledge of how to create the augmentation jobs. The next step is to create and generate the augmentation jobs based on your knowledge of the z/OS runtime you wish to augment. The final step is to upload the generated jobs and run them to augment the WAS z/OS runtime.

If the level of WebSphere Application Server is at V8.5, then the Compute Grid function is provided and all you need do is configure the Job Scheduler and the Endpoints. The process is the same as if you had augmented an existing WAS runtime.

• Run sample applications and understand their structure

Compute Grid comes with a set of samples that you may use to validate the runtime. They also serve as helpful guides to how an actual Java batch job may be constructed.

The samples are avaiable for download. See "Samples" on page 78 for information on locating and downloading them.

You'll find a list of six samples, the two highlighted being the ones we recommend you start with:

Sample Description

Java Batch ILC sample Sample to demonstrate the COBOL Container function. This is only valid on z/OS, and only valid if you've configured the COBOL Container. See "The COBOL Container" on page 33.

Java Batch Mailer sample Sample to demonstrate Parallal Job Manager function. We go into that in some details starting on page 68.

Java Batch Overdraft sample Sample transactional batch program that uses the JDBC and Text Batch Data Stream (BDS) patterns.

Java Batch Postingv2 sample Similar to the Overdraft sample.

Java Batch SimpleCI sample A very simple Compute Intensive job that waits the specified period of time, then ends.

Java Batch IVT sample Sample transactional batch program that uses either the JDBC or Text Batch Data Stream (BDS) patterns.

Experiment with these samples to validate your runtime and to see how simple batch programs run inside Compute Grid.

• Review Compute Grid InfoCenter articles on application development

The Compute Grid InfoCenter50 has a number of articles related to Java batch application development:

Description InfoCenter Search String

Initial topic page with links container_computegrid_developing

Brief description of transactional batch versus compute intensive

cgrid_cgpojo

Using sample applications tgrid_samples

Batch programming model cgrid_schbatch

Batch Data Stream Framework cgrid_btchpttrnover

Developing a simple CI application tgrid_bgridtutorialci

Developing a simple transactional batch application

tgrid_bgridtutorial2

Elements of xJCL cgrid_xdbatchjcl

50 http://pic.dhe.ibm.com/infocenter/wasinfo/cgwas80/index.jsp




• Develop first simple application and validate

At some point you must roll up your sleeves and develop your first application. You can take one of the existing samples and modify it, or you can try to start from the very beginning of job creation51.

The key is to start relatively simple and to get experience with the model and how things work together. With time and practice you'll learn enough to proceed to developing real-world applications.

• Other Compute Grid application development resources

An excellent technical review developed by Chris Vignola of IBM development:

http://www.ibm.com/developerworks/forums/thread.jspa?threadID=417633&tstart=45

An IBM DeveloperWorks article written by Chris Vignola:

http://www.ibm.com/developerworks/websphere/techjournal/0801_vignola/0801_vignola.html

51 RAD with the Java Batch feature makes this much easier. It has job creation wizards to assist.








Technology FocusThe COBOL Container

The COBOL Container function, which is exclusive to Compute Grid on z/OS, provides a means of executing compiled COBOL DLLs in the same z/OS address space as the Compute Grid container. Your Java batch applications may then call across a native interface to invoke the COBOL and receive the results.

Benefits: • Allows re-use of existing COBOL assets within the context of a Java batch job executing Compute Grid z/OS Version 8

• Provides an extremely efficient call mechanism to call COBOL modules as compared to other approaches

The COBOL Container function was first made available as part of the service stream for Compute Grid z/OS Version 6.1.3. It is now incorporated into the Compute Grid z/OS Version 8 product as well as the WebSphere Application Server z/OS Version 8.5 product.

Architectural overview

The following picture illustrates the components of the COBOL Container function. Notes follow and correspond to the numbered blocks in the picture:

1. Java batch applications that execute in a Compute Grid z/OS endpoint server do so in the servant region address space of the application server52.

2. The COBOL Container, when instantiated by the calling batch program (more coming up on that), manifests as a separate LE environment from that for the servant region, and a fresh LE environment is created every time the COBOL container is created.

Note: That's of value because it (a) assures no cross-contamination between WAS and COBOL, and (b) assures a clean LE environment for every instance of the batch container.

52 See "The z/OS application server design" on page 13 for more detail.




3. The COBOL DLL to execute comes from whatever location you specify as STEPLIB to the servant region53 or the WAS LIBPATH variable. This may be either a LIBRARY data set or a UNIX Systems Services (USS) directory location, respectively.

4. The COBOL Container code provides the infrastructure to create the LE environment as well as the "glue code" that allows calls in Java to traverse over to the native COBOL environment. This comes with Compute Grid z/OS V8 (or WAS z/OS V8.5).

Note: This is some fairly sophisticated JNI54 code that facilitates call and response between the two environments. This is yet another example of code you could write yourself, but why create custom middleware when you can have it as part of a supported product?

5. This is your Java batch application that calls the COBOL module. How that's done will be explained under "Invoking the COBOL call stub from the Java batch application" on page 48.

6. Calling the COBOL module and passing data in and receiving data back requires some Java code. That's the call stubs and data bindings code. Coding that manually is possible but challenging. Compute Grid comes with a "Call Stub Generator" to generate this code based on the COBOL source ...

7. The Call Stub Generator is a tool that reads in the COBOL source55 and creates the call stub and data bindings. The Call Stub Generator relies on an IBM Rational Application Developer (RAD) to do the generation. How the Call Stub Generator is used and what it produces is discussed under "COBOL stub generator" on page 36.

With the COBOL container function your Java batch program may call COBOL DLLs and receive the result back using a code path that is considerably shorter than alternative methods.

And by the way, it's also possible to create a JDBC Type 2 connection (z/OS cross memory) in Java and share that connection with the COBOL code that runs in the COBOL Container. That's discussed on page 49.

WAS z/OS application server setup requirements

With Compute Grid z/OS Version 8 or WAS z/OS Version 8.5 the COBOL Container libraries are included with the product and ready to use. However, there are a few things you must do to make use of this function:

• The application server in which you plan to use the COBOL Container function must run in 31-bit mode. That is because COBOL itself is a 31-bit execution language. Changing an application server from the default 64-bit mode to 31-bit mode is relatively easy56.

In the Administrative Console for the WAS z/OS cell, go to the "General Properties" for an application server. You'll see this:

Uncheck that box, save and synchronize, then stop and restart the application server.

53 Or link list.54 JNI = Java Native Interface55 Specifically, the LINKAGE SECTION.56 You should make sure no applications already deployed to that server depend upon 64-bit addressing.




• If the COBOL code to be called issues DISPLAY commands, then to insure those DISPLAY messages come back to the Java batch application you need to set the "workload profile" for the server to ISOLATE.

Note: If your COBOL code does not use DISPLAY then this is not required.

If your COBOL code uses DISPLAY and you don't set this to ISOLATE, the OUTDD(WCGILOUT) display message may come back to application, or may not. Setting to ISOLATE insures that they do57.

This setting's location in the Admin Console is a bit more obscure:

Set to ISOLATE, then click "OK," save and synchronize. Then stop and restart the application server.

• If the COBOL modules are compiled into a LIBRARY data set, then add a STEPLIB DD statement to the servant region's JCL start procedure, pointing to the data set58.

If the COBOL modules are compiled into a USS directory, then add that directory to the application server's LIBPATH.

COBOL program compilation requirements, program restrictions

The COBOL modules to be invoked in the COBOL Container have the following requirements59:

dll COBOL modules must be in DLL format

rent COBOL modules must contain reentrant code

thread COBOL modules must be thread safe

outdd(WCGILOUT) Required for COBOL DISPLAY output to appear in WebSphere Extended Deployment Compute Grid job logs

SQL('ATTACH(RRSAF)') Required for DB2 access

57 Setting the workload profile to ISOLATE limits the worker threads in the WAS servant to 1, which then aligns with the single-threaded nature of COBOL, and the DISPLAY messages are guaranteed to come back to the application.

58 Or add the LIBRARY to the link list concatenation.59 InfoCenter search: cgrid_cobol_container_overview




The InfoCenter60 lists the following restrictions on the COBOL program that will run in the COBOL container:

• User-defined error handlers are not allowed.• Explicit transaction control is not allowed: for example, no COMMIT or ROLLBACK.• DB2 special registers such as SOLID or SCHEMA, cannot be set.• DD access requires user code to dynamically allocate or deallocate the DD.• Java invocation of ENTRY labels is not supported.• COBOL invocation of Java code is not supported

COBOL stub generator

In the section "Architectural overview" on page 33, we indicated that the "call stubs" provided the method of calling the COBOL module and to pass the data the COBOL program requires, and receive back the data the COBOL program produces.

Those call stubs and data binds could be written manually, but that implies a fair amount of effort61. The Call Stub Generator is designed to do the work for you, based on the information it sees in the COBOL source you provide it.

Overview of the COBOL Call Stub Generator

The following picture illustrates at a high-level the Call Stub Generator. The numbered blocks correspond to notes that follow.

1. The COBOL Call Stub Generator ("CSG") operates on the workstation where your RAD V7.5 or higher development tool exists.

Important: The COBOL Call Stub Generator requires IBM Rational Application Developer. Generic Eclipse will not suffice.

The Call Stub Generator makes use of the Java Data Binder tool, which is part of RAD when the J2C tools feature is installed62.

2. The Call Stub Generator is a Java program supplied with Compute Grid that you download and unzip to your workstation63.

3. You invoke the Call Stub Generator either from the workstation command line (see page 41) or from within RAD as an ANT task (see page 45).

4. The csg.properties file is one you customize to provide information on how the program should run. Specifically, three key pieces of information64 are supplied here: where RAD is installed on the workstation, where the RAD workspace is located on

60 InfoCenter search: cgrid_cobol_container_overview61 Particularly the data bindings.62 This is the same tool that generates the COMMAREA marshaling code for JCA connectors. The same format is used.63 Provided at: /<mount>/AppServer/lib/COBOLCallStubGenerator.zip ... details on what's in that zip provided next.64 The csg.properties file has more properties that may be set, but those three are required.




the workstation, and the Java batch program project in the workspace to generate the call stubs and data bindings into.

5. You make the COBOL source program available to the Call Stub Generator as a file downloaded to your workstation. The Call Stub Generator reads this file to determine specifics about the call stub and data bindings it will generate.

6. The Call Stub Generator reaches into your installed RAD environment and workspace and invokes the Java data binder tools, passing in parameters based on the COBOL source deck you provided. RAD then builds the call stub and data bindings files and includes them in the Java batch program project you specified.

In the next few sections we'll show the details behind installing and using the Call Stub Generator.

Installing the COBOL Call Stub Generator on your workstation

The COBOLCallStubGenerator.zip file is what Compute Grid (or WAS z/OS V8.5) uses to supply the CSG to you for installation. It is located at the following location in the configuration of the WAS z/OS runtime:

/<mount>/AppServer/lib/COBOLCallStubGenerator.zip

Simply download that file in binary to your workstation and extract the files to a location of your choosing65.

What you will then see is the following:

1. Contents of the root COBOLCallStubGenerator folder

csg.properties Properties file used when invoking the Call Stub Generator from the command line. See "Invoking from the Command Line" on page 41.

CSG.xml XML file used when invoking the Call Stub Generator as an ANT task from within the Eclipse tool. See "Invoking as ANT task in RAD" on page 45.

CSGBatch.xml XML file used when invoking as an ANT task and processing multiple COBOL source files in one invocation. See "Invoking as ANT and processing multiple COBOL files" on page 47.

README Brief set of instructions on invoking the Call Stub Generator.

65 To install that's all that's needed. To use the Call Stub Generator involves a little customization as you'll see.




2. Contents of the lib folder

COBOLCallStubGenerator.jar This is the JAR that contains all the Java components that make up the Call Stub Generator.

ilcontainer.jar These two files are not needed to invoke the Call Stub Generator. But they are needed on the build path of the Java batch program that uses the COBOL container function. These files will allow a clean compile of the Java program.

marshal.jar

3. Contents of the testcases folder

Contains nine sample COBOL source files illustrating various data types.

Requirements for IBM Rational Application Developer

As mentioned earlier, the COBOL Call Stub Generator tool requires the use of IBM Rational Application Developer (RAD) Version 7.5 or higher. That is because the Call Stub Generator makes use of the Java Data Binder tool, which is part of RAD if the Java J2C tool is installed in RAD.

The Call Stub Generator has a few requirements of the target Eclipse environment and the workspace into which the artifacts will be generated66. They are:

Java EE Connection - J2C tools

The "Java EE Connection - J2C" tools must be installed. Here's what it looks like when viewed in the Installation Manager:

Or you can check in RAD itself by selecting "File" "New" "Other" and searching for "J2C":

If the J2C wizard is available, then the J2C tools are installed. If not, then go back to Installation Manager and install the tools.

66 InfoCenter search: tgrid_cobol_csg_generate




New "source folder" in the target Java batch application project

The Call Stub Generator expects to see a "source folder" named /src in the RAD project. Using the "SimpleCI" sample application as the starting point for this illustration, the /src folder, once created, would look like this:

In this example that was created by selecting the project "SimpleCIEarEJBs", then right-clicking and selecting "New" and then "Source Folder:"

By default the name you give it must be src.

What the COBOL Stub Generator generates

We've mentioned the output of the Call Stub Generator is a COBOL module call stub and a set of data bindings. To see what that looks like, let's use a real example of the output produced by the Call Stub Generator.

Input source COBOL

The source is a very simple67 COBOL program that takes two integer values, adds them, and returns the result: CBL LIB,RENT,THREAD,EXPORTALL****************************************************************** MODULE NAME = ADDER ****************************************************************** IDENTIFICATION DIVISION. PROGRAM-ID. ADDER RECURSIVE. ENVIRONMENT DIVISION. CONFIGURATION SECTION. DATA DIVISION. LINKAGE SECTION. 01 adder-input. 05 int-a PIC 9(8). 05 int-b PIC 9(8). 01 adder-return. 05 int-c PIC 9(8).

PROCEDURE DIVISION USING adder-input RETURNING adder-return.

MAINLINE SECTION. DISPLAY "COBOL -- In the COBOL routine". DISPLAY "COBOL -- int-a = " int-a.

67 Far more complex COBOL data structures are possible. This one is simple for illustration purposes.




DISPLAY "COBOL -- int-b = " int-b. COMPUTE int-c = int-a + int-b. DISPLAY "COBOL -- int-c = " int-c. GO TO END-PROGRAM. END-PROGRAM. DISPLAY "COBOL -- ADDER done". GOBACK.

Command line invocation68 of Call Stub Generator

From the C:\COBOLCallStubGenerator folder, which is where we unzipped the files (shown here split across two lines, but this is really a one-line command):

java -jar lib/COBOLCallStubGenerator.jar testcases/ADDER.cbl

-configFile csg.properties -callStubPackage com.ibm.cobolstub

Note the -callStubPackage parameter ... the call stub and data bindings will be generated under that package name in RAD, as you'll see next.

Result in RAD

Here's what shows up in the RAD workspace:

Call Stub

This is the object used for calling the COBOL module. See "Code generated by the Call Stub Generator" starting on page 78. Look for the ADDER.java heading.

Data Bindings

These are the objects used to marshal the data back and forth. See "Code generated by the Call Stub Generator" starting on page 78.

Data Input -- look for adder_input.java heading

Data Return -- look for adder_return.java heading

Note: You'll see that the data bindings are long and not trivial. This is the value of the Call Stub Generator: it relieves you of the burden of coding these data marshalling routines. And keep in mind that ADDER.cbl was an extremely simple program.

68 See "Invoking from the Command Line" on page 41 for more details.




Invoking from the Command Line

This is a relatively simple means of generating the call stubs. In involves a few simple edits to the csg.properties file69 and then the submission of a relatively simple command.

cgs.properties file

This file is found in the root folder at the location where you unzipped the COBOLCallStubGenerator.zip file. When you first open the file you will see the following. The settings highlighted in yellow are the ones that require update70.

A description of what to code for those required fields follows this listing.# Configuration file for COBOLCallStubGenerator###### EclipseHome specifies the install location of RAD 7.5 or higher#EclipseHome=<full-path-to-eclipse-directory> ###### Workspace specifies the location of the RAD workspace. If it # does not exist, the COBOLCallStubGenerator will create the workspace.#WorkSpace=<full-path-to-workspace> ####### EclipseProjectName specifies the project in the RAD workspace# that will receive the generated call stubs and bindings.#EclipseProjectName=<eclipse-project-name>

###### The target operating system. The permitted options are: Win32, AIX and# z/OS.## PARAMETER - Required# Platform=Win32Platform=z/OS

###### The target codepage.## PARAMETER - RequiredCodePage=IBM1047

###### The floating point format has only two possible values: # IEEE 754# IBM Hexadecimal# The default is IEEE 54## PARAMETER - RequiredFloatingPointFormat=IBM Hexadecimal

# PARAMETER - RequiredExternalDecimalSign=EBCDIC

# PARAMETER - RequiredEndian=Big

# PARAMETER - RequiredRemoteEndian=Little

# PARAMETER - RequiredQuote=DOUBLE

# PARAMETER - RequiredTrunc=STD

69 See "Installing the COBOL Call Stub Generator on your workstation" on page 37 for an illustration of the folders and files produced when unzipping the Call Stub Generator zip.

70 InfoCenter search: tgrid_cobol_csg_create_config




# PARAMETER - RequiredNsymbol=DBCS

###### Possible values for overwriteExistingClass are true or false.## PARAMETER - RequiredOverwriteExistingClass=true

###### Possible values for GenerationStyle are:# Default, "Preserve case of names" or "Shorten names"# Be sure to use quotes for values with space characters in them.## PARAMETERGenerationStyle=Default

###### Verbose is used to set the trace level to "debug"# Values for Verbose are either true or false. ## PARAMETER - OptionalVerbose=false

The values you supply for the higlighted values are:EclipseHome=<full-path-to-eclipse-directory>

The path to where RAD is installed. By default it will be:

C:/Program Files/IBM/SDP

Important! Use forward slashes, not the Windows backslash.

WorkSpace=<full-path-to-workspace>

The path to the RAD workspace where the target Java batch application project resides. For example:

WorkSpace=C:/RAD/workspaces

Again, note the forward slashes.

EclipseProjectName=<eclipse-project-name>

The Java EE project within the RAD workspace where the Call Stub Generator will place its output. For the SimpleCI sample program that would be:

And thus:

EclipseProjectName=SimpleCIEarEJBs




Structure of the command

The structure of the command71 is as follows:

Important! RAD must not be started when this command is issued.

1 2 3java -jar lib/COBOLCallStubGenerator.jar testcases/ADDER.cbl

-configFile csg.properties -callStubPackage com.ibm.cobolstub 4 5

Notes:

1. The command invokes Java, which means java needs to be on the PATH for your workstation.

2. The command shown assumes you are invoking it from the root COBOLCallStubGenerator folder. The subfolder lib is where the JAR file is actually located.

3. The next parameter tells the Call Stub Generator where to find the input COBOL file. Here we're illustrating the named file -- ADDER.cbl -- is one we added to the testcases folder of the unzipped COBOLCallSubGenerator.

4. The -configFile value tells the Call Stub Generator where to find the properties file. By default that is in the COBOLCallStubGenerator folder. If you invoke the command from that folder (which we illustrate here), then no pathing is needed.

5. The -callStubPackage value tells the Call Stub Generator what package name to give the generated stub and data bindings. You may use whatever package name you wish.

Output from a successful generation

If your generation is successful72 you will see output that looks something like this:Buildfile: C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\CSG.temp.xmlCSG: [csg] [csg] Config File Name: C:\COBOLCallStubGenerator\csg.properties [csg] COBOL Source: C:\COBOLCallStubGenerator\testcases\ADDER.cbl [csg] Call Stub Package Name: com.ibm.cobolstub [csg] Call Stub Class Name: ADDER [csg] Build String: Unknown Build [csg] Build Date: 2012/08/13 13:19:50 [csg] Eclipse Workspace Name: C:/COBOLCallSTubGenerator/RAD/csg [csg] Eclipse Project Name: SimpleCIEarEJBs [csg] [csg] Processed 1 input files.

Generate.com.ibm.cobolstub.parameters.adder_input:

Generate.com.ibm.cobolstub.parameters.adder_return:

GenAllBindings: [echo] GenAllBindings complete [delete] Deleting: C:\COBOLCallSTubGenerator\RAD\csg\SimpleCIEarEJBs\src\GenAllBindings.xmlBUILD SUCCESSFUL

BUILD SUCCESSFULTotal time: 9 seconds

71 The command is issued as one line. It is split here simply to show the whole command without making the typeface too small.72 If unsuccessful, see "Common CSG generation problems and their symptoms" on page 44.




Note: The call stub and data bindings are generated, but not yet referenced in the target Java batch application. To see how to set up the COBOL container and call the stub, see "Invoking the COBOL call stub from the Java batch application" on page 48.

Updating the application build path

The generated call stubs and data bindings are going to reference class libraries that will not by default be found in your RAD project. Therefore, you must update the build path for the project so the class files will compile cleanly.

The Call Stub Generator supplies the following in the lib folder under where you unzipped the COBOLCallStubGenerator.zip file:

ilcontainer.jar marshal.jar

Add those two as external JARs to the build path of your Java batch application project:

Common CSG generation problems and their symptoms

Symptom:CSG:BUILD FAILEDjava.lang.NoClassDefFoundError: com.ibm.etools.cobol.importer.CobolException

Cause:The target RAD environment does not have the J2C tools installed.

Symptom:Trace: 2012/08/13 13:46:59.562 ThreadId: 1 FunctionName: CallStubGenerator SourceId : com.ibm.ws.batch.cobol.csgen.COBOLCallStubGenerator Category: ERROR ExtendedMessage: Exception: DataBinder validation failed: EclipseHome C:Program FilesIBMSDP_1 is not a path to an existing directory.com.ibm.ws.batch.cobol.csgen.exceptions.DataBinderException: DataBinder validation failed: EclipseHome C:Program FilesIBMSDP_1 is not a path to an existing directory.

Cause:Several potential causes of this problem:• Use of " \ " rather than " / " in EclipseHome= path in csg.properties file

• Typo in path name of EclipseHome= path in csg.properties file




Symptom:CSG:[csg] com.ibm.ws.batch.cobol.csgen.exceptions.CallStubGeneratorException:The RAD project does not exist: C:COBOLCallSTubGeneratorRADcsg\SimpleCIEarEJBs\src

Cause:Several potential causes of this problem:• Use of " \ " rather than " / " in WorkSpace= path in csg.properties file

• Typo in path name of WorkSpace= path in csg.properties file• Incorrect EclipseProjectName= in csg.properties file

Symptom:CSG:

[csg] com.ibm.ws.batch.cobol.csgen.exceptions.CallStubGeneratorException:

The RAD project does not exist: C:/COBOLCallSTubGenerator/RAD/csg\SimpleCIEarEJBs\src

Cause:The src source folder was not created in the target Java EE project in RAD

Symptom:CSG: [csg] Processed 0 input files.

BUILD FAILEDC:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\CSG.temp.xml:22: The following error occurred while executing this line:java.io.FileNotFoundException: C:\COBOL Call STub Generator\RAD\csg\SimpleCIEarEJBs\src\GenAllBindings.xml (The system cannot find the file specified.)

Cause:The source COBOL file named on the command is either incorrect or there is a case mismatch in the file name.

Invoking as ANT task in RAD

This is a way to invoke the Call Stub Generator from within your RAD environment73.

Note: The end result is exactly the same as produced by invoking from the command line. The exact same tools are used within RAD. The exact same results are produced. The means of invoking the tools is slightly different.

The operative configuration file is the CSG.xml file74, which is found in the COBOLCallStubGenerator folder where you unzipped the ZIP file. You don't need to modify that ... you'll provide RAD information to prompt you for information it needs to complete the task.

The steps are as follows:

• Open RAD and have a Java batch application project created or imported.

• Make sure the "source folder" src is created under the Java EE batch application project that will serve as the target for the generated stub and bindings.

• Import your COBOL source into the src folder -- select the src folder, right click and select "Import," then expand the "General" folder and select "File System." Click "Next" and navigate through the wizard to import the COBOL source. For this example we'll illustrate using the simple ADDER.cbl used for the command line illustration.

• On the menu bar, click on "Run" "External Tools" "External Tools Configurations."

73 Nice instructions in the InfoCenter on this. Search: tgrid_cobol_csg_gui74 Reference to the CSG.xml elements: InfoCenter search -- rgrid_csg_xml_file




• Double-click the "Ant Build" option:

• Then do the following:

1. Name the configuration

2. Use "Browse File System" (or simply type in the value) of the full path and file name of the CSG.xml file located where you unzipped the Call Stub Generator.

3. Use "Browse File System" (or simply type in the value) of the full path to the location where you unzipped the Call Stub Generator.

4. Provide the following arguments:

-DcobolSource=${resource_loc}-DworkSpace=${workspace_loc}-DeclipseProjectName=${project_name}-DcallStubPackage=${string_prompt:callStubPackage}

• Click "Apply" then click on the "JRE" tab. Select "Run in the same JRE as the workspace:"

• Click "Apply" and "Close."

• Now select the source COBOL program you imported into src. Then on the menu bar, click on "Run" "External Tools" "External Tools Configurations."

• Select your new configuration, then click "Run."




• It should prompt you for the call stub package name:

Provide a package name, such as com.ibm.cobolstub. Then click "OK."

• It should then process the request and generate the call stub and data bindings. The format and location of that stub and data bindings will be identical to those created by command line invocation.

Invoking as ANT and processing multiple COBOL files

This is very similar to the single file invocation75 except the XML file is changed from CSG.xml to CSGBatch.xml, and one of the arguments is modified.

Assuming you've configured the single file ANT task from the previous section, the steps to modify it to process multiple files is:

• On the menu bar, click on "Run" "External Tools" "External Tools Configurations."

• Then do the following:

1. Change the "BuildFile" value from CSG.xml to CSGBatch.xml

2. Change the first argument from -DcobolSource=${resource_loc} to -DcobolSourceDir=${resource_loc}

Note: That simply tells RAD to process every source COBOL file in the selected src directory rather than process the one selected file as before.

• Then click "Apply" and "Close"

• Import more COBOL source files into the src folder

• Now select the src folder (not just an individual file). Then on the menu bar, click on "Run" "External Tools" "External Tools Configurations."

• Select your new configuration, then click "Run."

75 See "Invoking as ANT task in RAD" on page 45.




• It should prompt you for the call stub package name:

• Provide a package name, such as com.ibm.cobolstub. Then click "OK."

• It should then process the request and generate the call stub and data bindings for all the COBOL source files seen in that selected src folder.

Invoking the COBOL call stub from the Java batch application

With the call stub generated and the data bindings in place, your application may now call the COBOL module and pass the input parameters.

To illustrate this, we'll use the ADDER.cbl source (shown on page 39) and the call stub and data bindings generated from this. Listings of the Java generated for the call stub is provided under "Code generated by the Call Stub Generator" on page 7876.

Note: Be sure to add ilcontainer.jar and marshal.jar to the build path for the project. See "Updating the application build path" on page 44 for more.

try { ILContainer ilc = ILContainerFactory.getFactory().create(); 1 ADDER myadder = new ADDER(); 2 adder_input myadder_input = myadder.getadder_input(); 3 myadder_input.setInt__a(100); 4 myadder_input.setInt__b(200); ilc.invokeProcedure(myadder); 5 adder_return myAdder_return = myadder.getReturnValue(); 6 System.out.println("Integer addition value returned from COBOL = "

+ myAdder_return.getInt__c()); }

catch (ILContainerException e) { 7 e.printStackTrace();}

catch (ILProcedureException e) { 8 e.printStackTrace();}

Notes:

1. A container object is created77

2. A call stub object is instantiated based on the code produced by the call stub generator

3. The data input object is created based on the code produced by the call stub generator

4. The data value are set

76 The data bindings generated are very long and are not included in this document.77 The container, once created, persists until the end of the job step, where it is torn down by Compute Grid. During a batch step the created

container may be used again and again, by any Java class that's part of the batch application being managed by Compute Grid.




5. The procedure is invoked

6. The return values are retrieved

7. If the container throws an exception, this catches it

8. If the procedure throws an exception, this catches it

Sharing DB2 Type 2 connection between Java and COBOL

Imagine a Java batch application that performs a normal JNDI lookup of a JDBC data source, then invokes a COBOL module using the COBOL Container function:ds = (javax.sql.DataSource) context.lookup(datasourceName);con = ds.getConnection();ILContainer ilc = ILContainerFactory.getFactory().create();ilc.setDB2Connection(con);ilc.invokeProcedure(mycobol);

The InfoCenter78 shows a method summary for ILContainer:

The snippet of code above shows the setDB2Connection() method being used to pass in the created JDBC Type 2 connection to the COBOL Container for use by the COBOL program.

This only works for JDBC Type 2. It does not work for JDBC Type 4 connections.

Keep in mind the issues outlined under "COBOL program compilation requirements, program restrictions" on page 35.

Programming enhancements in CG 8 (and WAS 8.5)

Compute Grid Listeners

Compute Grid "listeners" provide a means by which the batch container may call user code at key points during the processing of the batch job. You write the listener code and the batch container invokes your listener methods at key points during job processing.

In this document we will illustrate three listeners:

Job Listener Invoked at the beginning of the job, and the start and end of each step, and again at the end of the job. Discussed in this section.

Skip-Record Listener Invoked during skip-record processing. Skip-record processing is discussed starting on page 52, and the listener on page 55.

Retry-Step Listener Invoked during retry-step processing. Retry-step processing is discussed starting on page 57, and the listener starting on page 55.

78 InfoCenter search: com.ibm.websphere.batch.ilc.ILContainer




Job listener

Overview of the job listener

A Job Listener is code that gets invoked by the batch container at specific points during the execution of a batch job.

It must implement the com.ibm.websphere.batch.listener.JobListener interface. The InfoCenter79 provides this method summary of the interface:

Those methods on your Job Listener are called at start of the job, at the start and end of each job step, and again at the job end. Graphically80, that looks like this:

At each of those points during the job's execution your Job Listener is called using the methods shown. Your Job Listener implementation may take whatever setup or cleanup steps you deem necessary.

Implementing a job listener

A class file that serves as a Job Listener must implement the JobListener interface81. The following is what IBM Rational Application Developer generates when a Job Listener class is created. You then code the function for the methods you wish to make use of.

79 InfoCenter search: com.ibm.websphere.batch.listener.JobListener80 The InfoCenter has a nice reference of when certain interfaces are called. Search: cgrid_batchjob_flow 81 InfoCenter search: com.ibm.websphere.batch.JobListener




package <your package name>;import java.util.Properties;import com.ibm.websphere.batch.listener.JobListener;public class MyListener implements JobListener {

@Overridepublic void afterJob() {

// TODO Auto-generated method stub}

@Overridepublic void afterStep() {


@Overridepublic void beforeJob() {


@Overridepublic void beforeStep() {


@Overridepublic Properties getProperties() {

// TODO Auto-generated method stubreturn null;

}

@Overridepublic void setProperties(Properties arg0) {


}

Registering a job listener

Compute Grid needs to know what class implements your Job Listener. It needs this so it may invoke the methods shown above. You do this by including a <listener> in the xJCL for the job82. For example:<job name="SkipRetrySample" ... > : <step-scheduling-criteria> <scheduling-mode>sequential</scheduling-mode> </step-scheduling-criteria>

<substitution-props> <prop name="inputEmpFile" value="..." /> : </substitution-props>

<listener> <classname>com.mycomp.MyListener</classname> </listener>

:

<job-step name="SkipRetryStep1"> : </job-step>

</job>

The key is that it's a job-level definition, not a step-level definition.

82 This is different from how the skip-record and retry-step listeners are registered. Those are registered with a small bit of Java code in the batch job. But because the Job Listener is invoked at the very start of the job, Compute Grid needs to know the Job Listener class at that time. It learns the Job Listener class file by reading the xJCL.




Skip-record processing

As noted earlier, there are times during a transactional batch process when the occasional exception may surface during the processing of records. The question is where those exceptions may be best handled -- by the batch container, or by the application?

The new skip-record processing83 found in Compute Grid V8 (the function of which is incorporated into the WebSphere Application Server V8.5 product as well) provides an exception handling mechanism in the container.

Note: This section borrows heavily from an excellent article written by Rob Alderman of the WAS development lab:http://www.ibm.com/developerworks/websphere/techjournal/1109_alderman/1109_alderman.html

This function has two essential components:

1. Skip-record policy definition -- configuration elements in the xJCL for a batch data stream that determines the skip record limit and what exceptions to include or exclude

2. Skip listener -- provides a means to have the container turn control to the batch application so it may log information about the records that have been skipped

Background context on need for skip-record processing

Batch applications will occasionally encounter exceptions reading or writing records. If the exception is not handled somewhere, the error will come back to the batch container, which will roll back to the current checkpoint and end the job, placing it in a restartable state.

Simply restarting the job will likely result in the same exception unless the root cause of the exception has been corrected.

Exception handling could be part of the batch application. But that suggests additional coding in the application for function that is better handled by the batch container84 infrastructure. It does this by consuming the exception within the container and automatically moving on to the next record85.

The skip-record policy defines the behavior of the skip-record processing, and it is configured on a per-Batch Data Stream basis in the xJCL for a job. We'll see that next.

Configuring and activating a skip-record policy

If you take no action to code a skip-record policy, then no policy will be in effect. In that case no skipping will take place. Any read or write exceptions that occur will flow up from the Batch Data Stream to the application.

A skip-record policy is enabled for a Batch Data Stream by coding a BDS property in the xJCL for the job step86:<batch-data-streams> <bds> <logical-name>myInputBDS</logical-name> <props> : <prop name="com.ibm.batch.bds.skip.count" value="5" /> : </props> </bds></batch-data-streams>

83 InfoCenter search: rgrid_xdbatchskiprp84 In the case of Compute Grid, this would be the Batch Data Stream framework. See "Batch Data Stream Framework" on page 23.85 The Skip Listener gives you the ability to log skipped records for audit purposes. See "Registering and using a skip listener" on page 55.86 More precisely, a non-zero value for the property.



http://www.ibm.com/developerworks/websphere/techjournal/1109_alderman/1109_alderman.html

http://www.ibm.com/developerworks/websphere/techjournal/1109_alderman/1109_alderman.html


In this example the "myInputBDS" batch data stream is provided a skip.count value of 5. That is, up to five records may be skipped. After that limit is reached, no further record skipping takes place. That Batch Data Stream87 would no longer consume the exceptions.

This would indicate that all exceptions would be eligible for skipping by this skip-record processing policy.

But you may wish to have greater control over what exceptions are considered. For that two other properties are provided -- one to include specific exceptions, and one to exclude specific exceptions88:

com.ibm.batch.bds.skip.include.exception.class.n

Specifies a Java exception class to be included in the skip-policy. You may code as many exceptions to be included as you wish. The n at the end of that is used to differentiate one from the other -- you increment the value of n for each, starting with 1, then 2, 3, etc. For example:<batch-data-streams> <bds> <logical-name>myInputBDS</logical-name> <props> : <prop name="com.ibm.batch.bds.skip.count" value="5" /> <prop name="com.ibm.batch.bds.skip.include.exception.class.1" value="java.lang.NumberFormatException" /> <prop name="com.ibm.batch.bds.skip.include.exception.class.2" value="java.io.IOException" /> : </props> </bds></batch-data-streams>

Key Point: In this example two includes89 are specified. All other exceptions are therefore excluded; they are not processed by skip-record processing.

If the number of exceptions you wish to include in skip-record processing is relatively large, then perhaps a better approach for you would be to exclude those exceptions you do not want to skip. That's what the next property is for:

com.ibm.batch.bds.skip.exclude.exception.class.n

Specifies a Java exception class to be excluded in the skip-policy. You may code as many exceptions to be excluded as you wish. Any exception not specifically excluded is by default included.<batch-data-streams> <bds> <logical-name>myInputBDS</logical-name> <props> : <prop name="com.ibm.batch.bds.skip.count" value="5" /> <prop name="com.ibm.batch.bds.skip.exclude.exception.class.1" value="java.io.IOException" /> : </props> </bds></batch-data-streams>

87 Other batch data streams in the job step, or other job steps, would be controlled by the skip-record policy defined for each.88 The two are mutually exclusive. You could code both, but there's no reason to. Why that is so is explained in the description of each

property.89 The NumberFormatException exception is what's illustrated in the sample application supplied with the DeveloperWorks article written

by Rob Alderman cited earlier. A field in the input file is defined as integer, and the "bad record" has alpha text in that field. Java throws the NumberFormatException error because of the mismatch. Skip-record processing skips the error due to the include statement.




In this example all exceptions are eligible for skipping except I/O exceptions. The container will not consume the Java I/O exceptions ... they will bubble up to the application where they should then be handled in some manner.

Key Point: This is why we say the two are mutually exclusive, and that you code one or other but not both. A single include property includes only that exception and automatically excludes all exceptions not named. A single exclude property excludes only that exception and automatically includes all properties not named. Choose one or the other but not both.

What happens when a record is skipped?

From Rob Alderman's document90:

1. The batch data stream consumes the exception and increments the skip count91

2. The skip is logged with this message:

CWLRB5852I: Record skipped by batch data stream inputEmp in job SkipRetrySample:00035 step SkipRetryStep1 due to error java.lang.NumberFormatException: For input string: "BAD"

Note: The fields of that message highlighted in blue are job-specific. That particular message is what gets thrown if you run the sample application that is provided with Rob Alderman's DeveloperWorks article.

3. The SkipListener callback is invoked, if one is registered.

4. When an input record is skipped, the batch data stream immediately attempts to read the next record. Control does not return to the caller until either:• A record is read successfully,• A non-skippable exception occurs, or• The skip limit is reached.

5. When an output record is skipped, the batch data stream returns to the caller normally.

To summarize: if no skip-record policy is in place, none of what's written above occurs. If a skip-record policy is in place, what's written above occurs up to and including the skip limit. After the skip limit has been reached then no further skipping occurs.

Sample application for illustrating skip-record processing

The Rob Alderman article92 at IBM DeveloperWorks provides an excellent sample application that illustrates the use of skip record processing. The sample may be downloaded from the bottom of the article:

The application is relatively simple in design: it reads employee records from a supplied file (input-emp.txt) and writes the records back out into a temporary file called output-emp.txt.

90 http://www.ibm.com/developerworks/websphere/techjournal/1109_alderman/1109_alderman.html91 Some deeper detail -- the skip record count is persisted to the Compute Grid database at every checkpoint interval. If skips occur

between checkpoints and a rollback occurs, the restart will pick up at the last checkpoint and the skip count will be what was persisted at that last checkpoint as well.

92 http://www.ibm.com/developerworks/websphere/techjournal/1109_alderman/1109_alderman.html




The data object (file EmpRecord.java) defines one of the data fields as:

private short _edlevel;

which means the data read in for that field must be integer format.

The error that produces the skip record processing is introduced in the input-emp.txt file. Two of the sixteen records in that file have a text string BAD for that field:

EMPNO|FIRSTNME|LASTNAME|EDLEVEL|SALARY1|JERRY|SEINFELD|10|1500.552|GEORGE|COSTANZA|20|750.853|COSMO|KRAMER|30|10.004|ELAINE|BENES|BAD|22000.905|UNCLE|LEO|15|100.556|POSTMAN|NEWMAN|25|500.557|SEINFELD|MAURI|30|100.108|KENNY|BANIA|35|15.559|LLOYD|BRAUN|45|15000.5510|JACKIE|CHILES|22|5000.8511|DAVID|PUDDY|32|100.8512|SUSAN|ROSS|42|11100.8513|FRANK|COSTANZA|BAD|900.8514|ESTELLE|COSTANZA|10|901.8515|JACOPO|PETERMAN|120|15901.85

That will result in an exception thrown -- java.lang.NumberFormatException -- and that is the exception specified as an "include" on the properties for the batch data stream. As long as the skip.count value is 2 or more, the exceptions will not surface.

The sample also includes a "skip listener" to log information about the records that are skipped. We will explore that next.

Registering and using a skip listener

As noted above, when a record is skipped one of the steps is to invoke a SkipListener callback. This will allow code you write to record detailed information about record that has been skipped.

There are two pieces to this:

1. Code that registers your skip listener to the job step, and

2. Code that implements the skip listener itself.

This must implement the com.ibm.websphere.batch.SkipListener interface, which consists of these methods:

public void onSkippedRead(Throwable t);

Invoked when an input (read) record is skipped. The exception is passed as an argument.

public void onSkippedWrite(Object record, Throwable t);

Invoked when an output (write) record is skipped. The record that failed and the exception are passed as arguments.

The sample application provided with Rob Alderman's document provides a skip listener. We'll use that to see how to register a skip listener and what code resides in the listener. We'll look at two files:

EmpProcessor.java -- contains the code that registers the skip listener

MySkipListener.java -- contains the skip listener implementation

Let's look at some code snippets in those files:




File: EmpProcessor.java

Method: public void createJobStep()

Code (starting on line 67):// Add SkipListener_mySkipListener = new MySkipListener();((AbstractBatchDataStreamRecordMetrics)_inputEmpBDS)

.addSkipListener(_mySkipListener);

Notes: This is what registers the skip listener to the job step. With a skip listener registered, the skip record processing will call the listener when a record is skipped.


Method: public int processJobStep()

Code (starting on line 125):_mySkipListener.setLastCleanRecord(empRecord);

Notes: This code is part of good record processing ... it invokes a method in the skip listener to keep track of the last known good record read. We'll see that this value is used when logging information about skipped records.

File: MySkipListener.java

Method: public void setLastCleanRecord(EmpRecord empRecord)

Code (starting on line 31):_lastCleanRecord = empRecord;

Notes: This code is part of good record processing ... every time a good record is processed, this method is called from EmpProcessor and information about the good record is maintained. This is used when logging information about skipped records, which is next ...

File: MySkipListener.java

Method: public void onSkippedRead(Throwable t)

Code (starting on line 21):logger.log(Level.FINE,"onSkippedRead called for failure", t);logger.log(Level.FINE,"last clean record", _lastCleanRecord);

Notes: When a record is skipped this method is invoked. It writes two lines: the first logs the exception; the second information about the last good record read.

You could code more elaborate record keeping in the onSkippedRead() or onSkippedWrite() methods. This sample is relatively simple, but it nicely illustrates how the function operates.

Skip-record metrics

The batch data stream of Compute Grid keeps track of record metrics, including the number of skipped records and the records-per-second processed.

There are two ways you can get this information:

1. By looking in the job log for record metrics messagesCWLRB5844I: [06/11/11 11:37:47:906 EDT] Job Step Batch Data Stream [SkipRetrySample:00035,SkipRetryStep1,outputEmp]: Metric = skip Value = 0CWLRB5844I: [06/11/11 11:37:47:906 EDT] Job Step Batch Data Stream [SkipRetrySample:00035,SkipRetryStep1,outputEmp]: Metric = rps Value = 82,074CWLRB5844I: [06/11/11 11:37:47:906 EDT] Job Step Batch Data Stream




[SkipRetrySample:00035,SkipRetryStep1,inputEmp]: Metric = skip Value = 2CWLRB5844I: [06/11/11 11:37:47:906 EDT] Job Step Batch Data Stream [SkipRetrySample:00035,SkipRetryStep1,inputEmp]: Metric = rps Value = 78,265

That is showing two batch data streams -- outputEmp and inputEmp -- and the skip metric and record-per-second metric for each. As we noted, the sample data file has two records with intentional errors on read operations. Hence the skip value of 2. There were no write errors, hence the skip value of 0 for the output BDS.

The sample data file only had 16 records, which took far less than a second to read in. The record-per-second values are extrapolations of the rate seen for the actual records read and written.

2. By retrieving the record metrics in the batch application

Compute Grid maintains a Job Context Manager93 object that contains a great deal of information about a job.

The sample application illustrates the use of this to get the skip count and the records-per-second count. This is seen in the EmpProcess.java file, under the method checkMetrics(). First it gets the record metrics for the input batch stream, then it collects the skip and rps values:

RecordMetrics inputEmpRM = JobStepContextMgr.getContext().getRecordMetrics("inputEmp");

long skips = inputEmpRM.getMetric(RecordMetrics.MetricName.skip);long rps = inputEmpRM.getMetric(RecordMetrics.MetricName.rps);

Then it uses that information in a log message:

logger.fine("RecordMetric data for inputEmp BDS. Number ofskipped records = " + skips + ". Records/second = " + rps);

Retry-step processing

The retry-step processing function94 is provides a means of re-running a transactional batch job step when the processJobStep() method throws an exception. The function has two essential components:

1. Retry-step policy definition -- configuration elements in the xJCL for a batch job step that determines the step retry limit and what exceptions to include or exclude

2. Retry listener -- provides a means to have the container turn control to the batch application so it may log information about the retries that have occurred

Overview of retry-step processing

The following picture illustrates some essential points about this function:

93 In the InfoCenter, search on string com.ibm.websphere.batch.context.JobStepContextMgr94 InfoCenter search: rgrid_xdbatchretrysp




1. After a job step commences an exception occurs in processing

2. If a retry step listener is registered then the onError() method of that listener is invoked. If no listener is registered then processing continues to the destroyJobStep() method, the batch data streams are closed and transactions are rolled back to the last checkpoint.

3. If retry-step processing is configured and the limit has not yet been reached, then retry processing takes place. The retry step policy definitions that apply to this decision are described under "Configuring and activating a retry-step policy" on page 58.

4. If there is no retry policy in effect, or the limit has been reached, then the job falls into a restartable state.

5. A delay for step retry is configurable. If configured, it takes place before the call to onRetry() for the registered listener.

6. The step retry listener is called using onRetry().

7. The createJobStep() method and processJobStep() methods are processed. All batch data streams are reopened. The step is tried again.

Sample application used for illustration

The sample application we referred to back on page 54 has Java code to illustrate retry-step processing as well. The companion xJCL elements for retry-step processing are not there by default, however. So in this section we will illustrate how to add them.

Configuring and activating a retry-step policy

There are four xJCL properties that set the policy for retry-step processing:

• com.ibm.batch.step.retry.count

Sets the limit for retries attempted for a job step




• com.ibm.batch.step.retry.delay.time

Sets the a delay time, in milliseconds, the retry step processing will wait after an exception is thrown and before the step is retried. See note 5 from the previous illustration.

• com.ibm.batch.step.retry.include.exception.class.n

Provides a means of specifying exceptions to include for invoking retry step processing. You may specify multiple include properties to specify different exceptions. The n on the end is an integer you increment, starting with 1, for each include property you specify for a given job step.

If you don't specify any includes (and no excludes either, see next), then all exceptions are eligible to invoke the retry-step processing.

If you specify one include, then only that exception is eligible to invoke retry processing and all other exceptions are excluded.

• com.ibm.batch.step.retry.exclude.exception.class.n

Provides a means of specifying exceptions to exclude from invoking retry step processing. You may specify multiple exclude properties to specify different exceptions. The n on the end is an integer you increment, starting with 1, for each exclude property you specify for a given job step.

If you specify one exclude, then only that exception is excluded and all others are included.

Note: The include and exclude policy definitions should be considered mutually exclusive. Include policy definitions exclude all others; exclude policy definitions exclude only those specified and include all others. So you pick one or the other, but never code both.

Sample application xJCL and the coding of these properties

As noted, the sample application supplied with Rob Alderman's DeveloperWorks article95 on skip-record processing has retry-step processing as well. But the xJCL that comes with the sample application does not include any properties for retry-step.

But it's really easy to add them. Here's a portion of the xJCL supplied with the sample and where the properties for retry-step would be coded: : <job-step name="SkipRetryStep1"> <classname>com.ibm.ws.batch.srs.EmpProcessor</classname> <checkpoint-algorithm-ref name="chkpt"/> <results-ref name="jobsum"/> <props> <prop name="max_record_count" value="${numberRecords}" /> <prop name="com.ibm.batch.step.retry.count" value="2" /> <prop name="com.ibm.batch.step.retry.delay.time" value="5000" /> <prop name="debug" value="${debugEnabled}" /> </props>

<batch-data-streams> <bds> <logical-name>inputEmp</logical-name> <props> <prop name="PATTERN_IMPL_CLASS" value="com.ibm.ws.batch.srs.bds.EmpFileReader"/> <prop name="file.encoding" value="${fileEncoding}"/> <prop name="FILENAME" value="${inputEmpFile}" /> [ What follows are properties for skip-record processing, which is different from retry-step ] <prop name="com.ibm.batch.bds.skip.count" value="3" /> <prop name="com.ibm.batch.bds.skip.include.exception.class.1"

95 See "Sample application for illustrating skip-record processing" on page 54.




value="java.lang.NumberFormatException" /> </props> : </bds>

In this example the job step will be retried a maximum of two times, and a delay of five seconds will be introduced before proceeding on to invoking a listener and invoking createJobStep() to try again96.

Since there are no include or exclude statements, all exceptions are eligible and retry-step processing will be done (up to the limit) for any exception seen.

Key Point: The xJCL properties for retry-step processing are coded for the job step.

Earlier we looked at the skip-record processing function. Those properties are coded for the batch data stream.

If you look in the xJCL above you'll see the skip-record properties under in the <bds></bds> stanza. The retry-step properties are under <job-step>.

Subtle point about the running counter for retry limits

As noted, the com.ibm.batch.step.retry.count property defines how many times the step will be retried. After the limit is reached exceptions thrown by processJobStep() result in the job going into a restartable state.

Here's the subtle point: the running counter Compute Grid uses to keep track of the retries is set back to zero every time normal processing passes a checkpoint interval.

Here's an illustration of what that means:• Imagine you have a job step that loops through 100 records. • Let's say the checkpoint interval is 10 records97

• The retry.count property is set at 3

• Everything is working fine up through record 13, then an exception is thrown.• The destroyJobStep() method is called and the batch container rolls back to the

last checkpoint, which would have been record 10.• Retry-step processing is invoked and is successful ... the step picks up after the last

checkpoint (record 10) and processes from record 11. The running counter of retries is now at "1".

• Processing continues ... records 11 through 19 are processed without incident and then record 20 is processed. The checkpoint interval is 10 records, so at record 20 Compute Grid does a checkpoint. Updates to records 11 through 20 are committed.

• Because a checkpoint interval has been passed, the running counter of retries is now set back to 0.

So in reality the retry.count property is the limit for retries between checkpoint intervals, not for the job step overall.

Let's look at another scenario and see how the limit would be invoked:• Same job: 100 records, checkpoint of 10• Assume the retry.count is 2 (just to keep this explanation a little shorter)

• Records processed successfully up through 13 then exception thrown.• Rollback to 10, retry the step and set running retry counter to 1.• Record 11, 12 and 13 are processed, then an exception on record 14.• Rollback to 10, retry the step and set running retry counter to 2. Limit now reached.• Record 11 through 17 processed, then an exception on record 18.• The limit has been reached. Therefore retry step processing will not be performed.

Rollback to 10. The job goes into a restartable state.

96 See the picture under "Overview of retry-step processing" on page 57.97 Compute Grid supports checkpoint intervals expressed in number of records or elapsed time.




How the skip-record sample application implements retry-step processing

The skipsample.zip file98 contains a sample application to illustrate skip-record processing. But it also contains code to illustrate retry-step processing.

There are four pieces of this to explore:

1. The code that registers the retry listener

2. The code that implements the retry listener

3. The mechanism used to generate an exception

4. The xJCL updates needed to make this work

Retry listener registration


Method: public void createJobStep()

Code (starting on line 71):// Add RetryListener_myRetryListener = new MyRetryListener();JobStepContextMgr.getContext().addRetryListener(_myRetryListener);

Notes: Registers MyRetryListener to the job step.

Retry listener implementation

A retry-step listener has a few requirements:

• It must implement com.ibm.websphere.batch.RetryListener

• It must implement a public void onError(Throwable t) method

• It must implement a public void onRetry(Throwable t) method

Here's what the sample application provides for onError():

File: MyRetryListener.java

Method: public void onError(Throwable t)

Code (starting on line 22):public void onError(Throwable t){ logger.log(Level.FINE,"onError called for failure", t); logger.log(Level.FINE,"current record", _currentRecord);}

Notes: The method onError() is called after an exception is thrown, but only if (a) the retry.count property is greater than zero99, (b) the retry limit has not yet been reached, and (c) a listener is registered.

This code simply logs two messages: one indicates the exception that was thrown, the other indicates the current record processed100.

98 See "Sample application for illustrating skip-record processing" on page 54.99 A retry.count value greater than zero is what enables the retry-step processing. Without that, no retry processing is done.100 This is similar to what we explored for the skip record code (see page 56). For every employee record processed successfully in

EmpProcess the setCurrentRecord() method of the retry listener is invoked. Then when an actual retry onError() is called the log message can use that information to provide some audit information about where processing was at the time of the exception and retry.




Here's what the sample application provides for onRetry():

File: MyRetryListener.java

Method: public void onRetry(Throwable t)

Code (starting on line 28):public void onRetry(Throwable t){ logger.log(Level.FINE,"onRetry called for failure", t);}

Notes: The method onRetry() is called after destroyJobStep() is invoked and rollback has occured, and before createJobStep(). But on if (a) the retry.count property is greater than zero101, (b) the retry limit has not yet been reached, and (c) a listener is registered. If a delay.time property is set, then the delay processes before onRetry() is called102.

This code simply logs one message with the exception that was thrown that resulted in the retry.

Mechanism used to generate an exception

The EmpProcess.java code has a method called simulateRandomFailure() that will throw an exception at "random" intervals. The frequency is controlled by a setting that by default implies no exceptions. But if you change the value then you may see the exception thrown more frequently.

That method is called after every record is processed. The code looks like what is shown here. The numbered blocks are not part of the code; they relate to the notes that follow: public int processJobStep() : processEmp(empRecord); : simulateRandomFailure(); Line 132 of "EmpProcess.java" : private void simulateRandomFailure() Line 190 of "EmpProcess.java" throws RandomFailureException { if (_randomFailureMod > 0) 1 { Random rand = new Random(System.nanoTime()); 2 int x = rand.nextInt(); 3 if (x % _randomFailureMod == 0) 4 { String msg = "Generating random failure with random number " + x + " and modulo " + _randomFailureMod; 5 logger.fine(msg); throw new RandomFailureException(msg); 6 } } }

1. Checks to see if the value for _randomFailureMod (the setting that controls the frequency of the exceptions) is 0. If zero then no failure is attempted; the method simply returns having done nothing.

By default the value of _randomFailureMod will be 0. The value is initialized to zero on line 37:

101 A retry.count value greater than zero is what enables the retry-step processing. Without that, no retry processing is done.102 See "Overview of retry-step processing" on page 57.




private int _randomFailureMod = 0;

A little further down (line 56) you'll see this:_randomFailureMod = Integer.parseInt(_props.getProperty(RANDOM_FAILURE_MOD,"0"));

That is code that allows properties in the xJCL to override the initialized value. We'll explore how to update the xJCL to accomodate that next.

2. The random number object is created and seeded.

3. The integer variable x is populated with a random integer variable.

4. A modulo (or remainder) operation is performed on x using _randomFailureMod as the denominator. If the result is equal to 0 then an exception will be thrown. If not equal to 0 then no exception is thrown.

Here's where the value of _randomFailureMod influences the frequency:

• If set to 1 then an exception will always be thrown because the remainder of any division by 1 will always be 0

• If set to 2 then an exception will be thrown 50% of the time because any even number will result in the remainder being 0

• If set to 10 then an exception will be thrown 10% of the time. The percentage chance diminishes the higher the number gets.

5. A string is constructed that will serve as the message of the exception

6. The exception is thrown

The exception will trigger the retry-step processing, provided the retry.count property is greater than zero, the limit has not been met, and the exception is eligible, which it would be if no include or exclude properties are coded.

xJCL updates needed

The EmpProcess.java file contains a few different lines of code that set up the ability to use _randomFailureMod and to pass that in from xJCL at runtime. : public static final String RANDOM_FAILURE_MOD = "random_failure_mod"; : private int _randomFailureMod = 0; : _randomFailureMod = Integer.parseInt(_props.getProperty(RANDOM_FAILURE_MOD,"0")); :

It's the third line shown that does the getProperty() to pull the value passed in from the xJCL when the job was submitted for execution. The sample xJCL as provided does not have the properties to pass in the value. But it's really easy to add:<job name="SkipRetrySample" ... > : <substitution-props> <prop name="inputEmpFile" value="e:/rad/workspace/SkipRetrySample/etc/input-emp.txt" /> <prop name="outputEmpFile" value="f:/tmp/output-emp.txt" /> <prop name="fileEncoding" value="8859_1"/> <prop name="numberRecords" value="20" /> <prop name="checkPoint" value="5" /> <prop name="debugEnabled" value="false" /> 2 <prop name="random_failure_mod" value="2" /> </substitution-props> : <job-step name="SkipRetryStep1"> <classname>com.ibm.ws.batch.srs.EmpProcessor</classname>




<checkpoint-algorithm-ref name="chkpt"/> <results-ref name="jobsum"/> <props> <prop name="max_record_count" value="${numberRecords}" /> <prop name="debug" value="${debugEnabled}" /> 1 <prop name="random_failure_mod" value="${random_failure_mod}" /> </props>

1. This is what sets the property that's read by getProperties() in the code. The value coded is ${..}, which means it's a substitution value set higher in the xJCL. That's what the next line illustrated does ...

2. This is where the substitution property is set. The value set here is populated in the ${random_failure_mod} variable lower. By coding it as a substitution property103 you tell the Job Management Console104 of the Compute Grid dispatcher function to surface that property on the browser panel. You may than change it on the fly as you submit the job.

Here we show a value 2, which means the exception will be thrown 50% of the time.

Configurable transaction mode

As an introduction to this enhancement, the following paragraph105 provides a good overview of what "Configurable transaction mode" is and the benefit it provides:

"In this mode, no global transaction is started; the connection object is shared between the container and the user code via the JobStepContext object. Local transaction mode significantly improves performance by reducing the overhead required to manage global transactions. This mode is especially useful if your data is located on the same database system as the Compute Grid tables. In addition, large cursors are less likely to be closed when transactions are committed."

Overview of the function106

This setting is like a switch that affects all the batch artifacts for a job. That includes all listeners, all batch data streams and all checkpoint algorithms.

It is a job-level setting in xJCL that has two options: global and local.

The default is global.

Configuration in xJCL

Using a snippet of the xJCL from the sample we used when discussing the skip-record and retry-step functions, you can see how this transaction mode xJCL property is coded. The key is that it's a job-level property, not a step-level property.<job name="SkipRetrySample" ... > : <step-scheduling-criteria> <scheduling-mode>sequential</scheduling-mode> </step-scheduling-criteria>

<props> <prop name="com.ibm.websphere.batch.transaction.policy" value="LOCAL"/> </props> : <job-step name="SkipRetryStep1"> : </job-step></job>

103 As opposed to just hard-coding the value in the <prop> of the job step.104 Or "JMC" ... the browser interface to the dispatcher function. Substitution properties may be used with the other interfaces of the job

dispatcher function as well.105 http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.wxd_v6/wxd/8.0/ComputeGrid

/XD80_ComputeGrid_Overview.pdf ... hyperlink here ... Page 7 of 23.106 InfoCenter search: rgrid_xdbatchtransm



http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.wxd_v6/wxd/8.0/ComputeGrid/XD80_ComputeGrid_Overview.pdf


Getting the shared connection from the Job Step Context Manager

Word-for-word from the InfoCenter107:

Because the local transaction mode is specified, all user code must share the Java Database Connectivity (JDBC) connection with the batch container. The batch container creates and saves a JDBC connection in the job step context.

You can fetch the job step context with the following code:

java.sql.Connection conn =

JobStepContextMgr.getContext().getSharedSQLConnection();

Do not attempt to create a JDBC connection or close a connection obtained with the job step context. Local transaction support is built into the batch data stream framework JDBC reader or JDBC writer patterns.

Setting the com.ibm.websphere.batch.transaction.policy property to local forces the batch data stream framework to use the shared JDBC connection.

Parallel Job Manager and enhancements in CG 8 (and WAS 8.5)

We provided an overview of the Parallel Job Manager back on page 28. There we painted a picture of the Parallel Job Manager providing the framework to process batch jobs in parallel based on segmentation of the data ranges. That segmentation is done by the "parameterizer" based on the parallel.jobcount number set in the xJCL.

In this section we'll take a look at the enhancements made in V8 that affect parallel job processing, and we'll finish up by taking a tour inside the "Mailer" parallel job sample application.

General enhancements in PJM function

For those who have some experience with the Parallel Job Manager function of XD Compute Grid 6.1.1.3, some improvements will be noticed when moving to Compute Grid V8 (or the equivalent function provided as part of WAS V8.5). From the InfoCenter108:

• The parallel job manager is in the batch container instead of in a separate system application as it had been in previous versions.

• Only a single xJCL file is required. The xJCL combines the contents of the top-level job xJCL with the contents of the subordinate job xJCLs.

• You do not need to create a separate database as you had in previous versions.• Because the PJM is part of the batch container, you do not need to install and

configure the PJM.• You package the PJM APIs in the batch application as a utility Java archive (JAR).

No shared library is required.• The contents of the xd.spi.properties file are now part of the xJCL. No

xd.spi.properties file is required.

The essential message here is that in the past the PJM function had to be manually installed and configured. With Compute Grid V8 all the items listed above suggest PJM is now more integrated the product.

Control over where parallel subjobs are dispatched -- threads or JVMs

A new xJCL element is provided to indicate to Compute Grid whether to run in sequential or parallel, and if parallel, then where to run the parallel sub-jobs. The xJCL element takes this form:<run instances="<single|multiple>" jvm="<single|multiple>">

107 InfoCenter search: rgrid_xdbatchtransm108 InfoCenter search: cgrid_cgparallel




The following table summarizes what the combinations of parameters provides:

instances="single" jvm="single" The job is to be run sequentially (not parallel). The jvm= parameter is ignored. A single JVM is implied in running a job sequentially.

instances="single" jvm="multiple" The job is to be run sequentially (not parallel). The jvm= parameter is ignored109. A single JVM is implied in running a job sequentially.

instances="multiple" jvm="single" The job is to be run in parallel, but within a single JVM. In that case the sub-jobs will be run on separate execution threads within the single JVM.

instances="multiple" jvm="multiple" The job is to be run in parallel, and across the available batch endpoint servers hosting the named batch application.

Using the xJCL from the Mailer sample110, the use of this new property at the job-level is illustrated:<job name="MailerSample" ... > <jndi-name>ejb/com/ibm/ws/batch/MailerSampleBatchController</jndi-name> <run instances="multiple" jvm="multiple"> <props> This is where PJM properties would go </props> </run> <job-step name="IdentifyRecipientsStep"> <classname>com.ibm.websphere... </classname> <checkpoint-algorithm-ref name="recordbased" /> <results-ref name="jobsum" /> : </job-step> :</job>

The property is also supported down at the step level, which is what would allow you to mix sequential and parallel in the same job:<job name="Someotherjob" ... > <jndi-name>...</jndi-name> <job-step name="Step1"> <run instances="multiple" jvm="multiple"> <props> This is where PJM properties would go </props> </run> : </job-step> : <job-step name="Step2"> <run instances="single" jvm="single" /> : </job-step></job>

109 Though ignored, you should not code this invalid combination.110 See "A look inside the Mailer sample parallel program" on page 67 for the start of a closer look inside this sample program.




Mixing of job-step types within a multi-step job

In past releases of Compute Grid the steps within a multi-step job had to be of the same type -- either all transactional batch, or all compute intensive, but not a mix of both.

With Compute Grid V8 (and the function included in WAS V8.5), that restriction has been lifted. Now a multi-step job may consist of steps of type transactional, compute intensive or native execution mode.

The InfoCenter111 says this:

You can mix transactional batch, compute intensive, and native execution job steps. The run time uses a controller that is the same for every job, regardless of the type of steps that the job contains. The controller runs appropriate logic for the step, whether the step is a batch, compute-intensive, or native execution step.

Those job steps that are transactional in nature may also be run in parallel with the <run> option set.

A look inside the Mailer sample parallel program

The "Mailer" sample program is one supplied in the InfoCenter112. There you will see a tab bar with "Downloads:"

You'll find a section labeled "Java Batch," under which will be the Mailer sample available for download.

It is a Java batch application that utilizes the Parallel Job Manager to execute in parallel. It's a nice sample because it's not overly-complicated. It serves as a background for illustrating the key points about the PJM.

Note: At the time this document was being written, the ZIP file at that location was lacking a few pieces of the sample solution, specifically the DDL for creation of the database table. It is a known issue and will be corrected.

111 InfoCenter search: cgrid_cgpojo112 See "Samples" on page 78 for information locating and downloading samples.




Overview of the Mailer sample application

At its heart the Mailer application is a two step job that reads from a database, determines if a promotional mailing (e-mail or postal) should be sent, writes intermediate results to a file, then in the second step reads from that file and updates the original database table with the date the promotion was sent:

1. The input to the first step is a relational database table. The step uses the Batch Data Stream "JDBCReader" pattern113 to read from the table. The table has 10,000 customer records in it, with the data record layout being:

NAME VARCHAR(250),ADDRESS VARCHAR(250),CITY VARCHAR(250),STATE VARCHAR(2),ZIPCODE VARCHAR(12),EMAIL VARCHAR(128),PHONE VARCHAR(20),ANNUALINCOME INTEGER,CUSTOMERID INTEGER NOT NULL,LASTOFFERDATE DATE

2. The processing logic in the first step is shown on the chart -- it checks to see if the annual income field is above a given threshold value, and if so then it determines whether to e-mail promotion or postal mail based on the presence of an e-mail address in that field.

The annual income threshold value is passed in as a property in the xJCL<prop name="PROMOTIONAL_MAILER_THRESHOLD" value="5000" />

3. The output from the first step is a text file called promotionalMailings.txt. The step uses the Batch Data Stream "TextFileWriter" pattern to write to the file.

The file name is passed in as a property in the xJCL114:<prop name="FILENAME" value="c:\temp\PJM-TEST-DATA.txt" />

113 See "Batch Data Stream Framework" on page 23.114 If you run this on z/OS then you would have to change that property to be a UNIX-format file pathing.




4. The input to the second step is the output text file from the first step. The step uses the Batch Data Stream "TextFileReader" pattern to read from the file.

5. The processing logic in the second step is very simple -- read the record and then prepare to update the database table.

6. The output from the second step is to write to the same database table as was read from in the first step. Specifically it updates the LASTOFFERDATE field. The Batch Data Stream uses the "JDBCWriter" pattern to write to the table.

That's what would happen if we did not parallelize the job. But the sample is written to be run as a parallel job. So let's see how that's done.

Mailer sample application -- xJCL overview

To better understand the processing of the Mailer parallel job application it is best to have an understanding of what's in the xJCL. Some of the snippets from that xJCL we'll offer inline with the discussion of the function. A more complete listing of the xJCL with notes that explain key components is offered under "xJCL for the Mailer Sample" on page 79.

Parallel processing for the data in the Mailer sample application

The 10,000 records of customer data that are created when the table is created has customers in all 50 of the states. So one method of partitioning up the data for parallel processing is by state. That is what the Mailer sample does.

In the xJCL file ( MailerV8xJCL.xml ) you will see these two job-level properties:

<run instances="multiple" jvm="multiple"> :<prop name="parallel.jobcount" value="3" />

The first is what tells Compute Grid the job may be run in parallel, and to run the parallel sub-jobs in separate batch endpoint servers if available115. The second is what's used to indicate how many parallel sub-jobs to run. In this example the number of parallel sub-jobs will be 3 ... the 50 states will be divided by 3 and each sub-job will process approximately one-third of the states each:

115 See "Control over where parallel subjobs are dispatched -- threads or JVMs" on page 65. If the number specified here is more than the number of endpoint servers, Compute Grid will work with what's available.




The function that does that data range division is called the parameterizer.

The Parameterizer is what determines how to partition the data

The Parallel Job Manager of Compute Grid is a framework of infrastructure code that helps with parallel job processing. One element of this is what's called the parameterizer, which can be thought of as two components:

1. Java code that implements the logic to partition the data into the number of data ranges needed. This is code you provide116. You name your parameterizer class file in the xJCL, as you'll see next.

2. A container-managed call to the parameterizer code specified in the xJCL. This is done automatically by the Parallel Job Manager after the top-job is dispatched but before the sub-jobs are dispatched.

If you look in the xJCL for the Mailer sample, you see the following properties at the job level of the xJCL:<job name="MailerSample" ... > : <run instances="multiple" jvm="multiple"> <props> 1 <prop name="com.ibm.websphere.batch.parallel.parameterizer"

value="com.ibm.websphere.samples.spi.MailerParameterizer"/> 2 <prop name="com.ibm.websphere.batch.parallel.synchronization"

value="com.ibm.websphere.samples.spi.MailerTXSynchronization"/> <prop name="com.ibm.websphere.batch.parallel.subjobanalyzer"

value="com.ibm.websphere.samples.spi.MailerSubJobAnalyzer"/> <prop name="com.ibm.websphere.batch.parallel.subjobcollector"

value="com.ibm.websphere.samples.spi.MailerSubJobCollector"/> <prop name="com.ibm.wsspi.batch.parallel.subjob.name"

value="MailerSampleSubJob" />

116 There is a generic parameterizer provided with Compute Grid, but its function is limited. Better to provided your own parameterizer class that implements data partitioning based on your knowledge of the data structure.




3 <prop name="parallel.jobcount" value="3" /> </props> </run>

1. The parallel.parameterizer property tells the PJM what Java class to use to set the parallel job parameters. We'll take how the Mailer sample implements this class in just a bit.

2. These other properties we'll cover after we discuss the parameterizer.

3. The parallel.jobcount property indicates how many sub-jobs to spawn. The parameterizer code uses this to divide up the data range.

Key Point: By specifying a class file on the parameterizer property, you tell the PJM what class to invoke to segment the data for the sub-jobs to act upon.

If you intend to run a job with <run instances="multiple"> then you must implement a parameterizer.

The Mailer sample parameterizer at a high-level

The Mailer sample parameterizer:

• Defines the initial array of the 50 state abbreviation codes

• Loops up to the number defined by parallel.jobcount

• For each loop it:• Creates a sub-job array of state abbreviation codes

• Creates a property that contains the sub-job array of state abbreviations

• Creates a property that contains the sub-job file name that will be used as output for the first step and input for the second

The result is a set of properties each sub-job uses. The properties allow each sub-job to act of a separate range of data, and for each sub-job to have a separate file to use for ouput of the first step and input for the second step.

The Mailer sample parameterizer in detail

If we look at the source we see code that splits the list of state abbreviations into shorter arrays based on the parallel.jobcount property:package com.ibm.websphere.samples.spi;

import java.util.Properties;import java.util.StringTokenizer;import com.ibm.wsspi.batch.parallel.Parameterizer;import com.ibm.wsspi.batch.parallel.Parameters;

public class MailerParameterizer extends Parameterizer {

public MailerParameterizer() {} 1 private String states[] = {

"AL","AK","AZ","AR","CA","CO","CT","DE","FL","GA","HI","ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI","MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC","ND","OH","OK","OR","PA","RI","SC","SD","TN","TX","UT","VT","VA","WA","WV","WI","WY"};

@Overridepublic Parameters parameterize(String logicalJobName,

String logicalTXID, Properties props) { :

// get job count from properties2 int jobcount = Integer.valueOf(props.getProperty("parallel.jobcount","1"));3 String unqualifiedFilename =

props.getProperty("FILENAME","C:\\TEMP\\DATAFILE");

StringTokenizer tok = new StringTokenizer(logicalJobName,":");4 String jobnameWithoutColon = tok.nextToken()+"."+tok.nextToken();




// Establish properties that will be delegated to all without modifications

Parameters parms = new Parameters();parms.setSubJobCount(jobcount);

//Populate a Properties object for each subjob.Properties newprops [] = new Properties[jobcount];

5 for ( int i=0; i<jobcount; i++) {newprops[i] = new Properties();

// calculate slice of states array and build // state listfor subjob

6 int slice_size = states.length / jobcount;int start_index = i * slice_size;int end_index = (i == jobcount-1) ?

states.length : (i+1)*slice_size;String stateList = new String();for ( int j = start_index; j < end_index; j++ ) {

if ( j == start_index ) { stateList += states[j];

} else {stateList += ","+states[j];

}}

// Create a unique file extension for each job7 newprops[i].put("EXCHANGED_FILENAME",

unqualifiedFilename+"."+jobnameWithoutColon+"."+i);8 newprops[i].put("STATES_LIST", stateList);

}

parms.setSubJobProperties(newprops);return parms;

}}

1. An array with the list of all 50 states is established117.

2. The parallel.jobcount property is derived from what was set in the xJCL at job submission. The xJCL supplied with the sample has that property set to 3. This is what's used to divide the array of states into the sub-job ranges.

3. The FILENAME property is derived from the what was set in the xJCL at job submission. This is used as the foundation for the output file for the first step, which is also the input file for the second step.

4. The Compute Grid job number is derived

5. The loop up to parallel.jobcount is set up

6. The initial array of 50 state abbreviations is divided into the sub-job arrays. The length of the initial array (50) is dividing by parallel.jobcount (3) to come up with an integer (16). Sub-job 1 array will contain 16 states; sub-job 2 array will contain 16 states; and sub-job 3 array will contain the remaining 18 states.

7. A unique sub-job file name property is created that will be used as the output file for the first step and the input file for the second step.

If the xJCL FILENAME property118 is set to, for example, PJM, then the resulting file names will be:PJM.MailerSample.xxxxx.0PJM.MailerSample.xxxxx.1PJM.MailerSample.xxxxx.2

117 They're not quite in proper alphabetical order J. But that doesn't affect the parameterizer.118 See "xJCL for the Mailer Sample" on page 79.




Where xxxxx is the job number assigned the top job by Compute Grid.

8. A unique sub-job state array property is created. This is used in the first step of each sub-job when the query is made against the relational database table.

Key Point: When each subjob processes its first step JDBC read operation, the SQL query it will make will be based on the sub-job array created here. You can see this in the CustomerJDBCReader.java source file.

The SQL query generated will be:SELECT name,address,city,state,zipcode,email,customerID,phone,annualIncome,lastOfferDate FROM MAILERSCHEMA.CUSTOMER WHEREstate in ("+statesList+")

+statesList+ is the sub-job array of states generated by the parameterizer.

Therefore, sub-job one will work on a slice of states; sub-job two a different slice of states; and sub-job three the final slice of states.

Other Parallel Job Manager properties and what they do

The xJCL for the Mailer sample shows three other PJM-related properties119 in addition to the parameterizer:<props> <prop name="com.ibm.websphere.batch.parallel.parameterizer"

value="com.ibm.websphere.samples.spi.MailerParameterizer"/> <prop name="com.ibm.websphere.batch.parallel.synchronization"



value="com.ibm.websphere.samples.spi.MailerSubJobCollector"/> :</props>

The very first point to make is that these three -- synchronization, subjobanalyzer and subjobcollector -- are optional. The Mailer sample implements them for illustration.

Only the parameterizer is required for parallel jobs.

Like the parameterizer, these are Java class files you implement that are called at various points in the execution of the parallel job. For example, we noted that the parameterizer is called after the top-job is dispatched but before the sub-jobs are dispatched. It is called just that once in the processing of a parallel job.

Let's review what the others provide, and what the Mailer sample implementation of them offers:

com.ibm.websphere.batch.parallel.synchronization

This interface is used to help you coordinate the overall job in a logical transaction.

Important: This is different from transactions and checkpointing performed down at the Batch Data Stream level. Those are real transactions that are committed and rolled back by the BDS.

That BDS-level transaction processing still takes place when a job runs in parallel. Each sub-job will operate with its own batch data streams and will do their own commits and rollbacks as if those sub-jobs were running separately.

No actual resources are enlisted in this logical transaction. The logical transaction is only a set of call backs to allow user code to be invoked by the PJM at key points in the lifecycle of a parallel job.

If you implement a synchronization class120 it must provide four methods:

119 InfoCenter search: rgrid_cgpjmspi120 Remember, this is optional.




Method Description

begin() This method is called at the very beginning -- when the top-job xJCL is submitted but before any of the sub-jobs are dispatched.

beforeCompletion() This method is called before the last sub-job completes.

afterCompletion() This method is called after the last sub-job completes.

rollBack() This method is called under two conditions121:

1. When any SPI in the com.ibm.wsspi.batch.parallel family throws declared exception RollbackLogicalTXException.

2. When any subjob completes in the Restartable or Failed state.

When rollback is triggered, the following actions are taken:

• In-flight subjobs are cancelled. This includes subjobs still executing on endpoints, as well as subjobs that may be awaiting dispatch.

• The rollBack method on this SPI is invoked. It is the responsibility of the rollBack method to undo any work done by subjobs that have successfully completed one or more checkpoints. Note this includes subjobs that have completed successfully; that is, are in the "ended" state.

The Mailer sample implements this, but the implementation class does nothing other than process System.out.println() to indicate the methods have been called. It is useful to see what calls are made when within the parallel job.

com.ibm.websphere.batch.parallel.subjobcollector

This interface122 is used to collect information about a sub-job and return it to the analyzer, which is discussed next. The batch container calls this, and it is called just after each sub-job processes a checkpoint, as well at the end of each sub-job.

This is a relatively simple interface that's easy to implement:

Method Description

collect() The InfoCenter indicates this returns: "Externalizable containing data to be delivered by ParallelJobManager to SubJobAnalyzer SPI. If return value is null, SubJobAnalyzer will not be called.

getName() Returns the name of this SPI instance. Should return a unique identifier.

The Mailer sample implementation of this is:public class MailerSubJobCollector extends SubJobCollector {

public MailerSubJobCollector() {}@Overridepublic Externalizable collect(String parallelJobName,

String logicalTXID,String subJobID) { CollectedInfoForAnalysis ret = CollectedInfoForAnalysis.getCollectedInfo(); ret.markCheckpoint(); return ret;}

}

com.ibm.websphere.batch.parallel.subjobanalyzer

This interface123 is used to evaluate information about each sub-job. The information it receives is based on what is returned from the call to the SubJobCollector. In that sense the SubJobAnalyzer is best implemented along with the SubJobCollector.

121 InfoCenter search: com.ibm.wsspi.batch.parallel.logicalTX.Synchronization122 InfoCenter search: com.ibm.wsspi.batch.parallel.status.SubJobCollector123 InfoCenter search: com.ibm.wsspi.batch.parallel.status.SubJobAnalyzer




The InfoCenter has the following information about the methods of this interface:

The implementation of this as provided by the Mailer sample, which is provided in the sample ZIP as MailerSubJobAnalyzer.java, issues System.out.println() messages to indicate the information gathered at that point.

Submitting the Mailer job -- one top job and three sub-jobs

The Mailer job is submitted like any Compute Grid job. The fact that it's a parallel job does not change how it is submitted.

In the Job Management Console the normal path is taken:

The sequence of activities that takes place then will be:

• Job is dispatched to an endpoint where the Parallel Job Manager invokes the parameterizer and the sub-job parameters are created




• The job is returned to the Job Dispatching function, which dispatches the sub-jobs to the endpoints

• The jobs -- top-job and the sub-jobs -- go from "submitted" to "executing" to "ended" states124.

• When the last sub-job completes then the top-job is marked "ended" and the overall job is complete.

If you "View jobs" from the Job Management Console, you will see something like this:

Note: These jobs complete fairly quickly. To catch all four in "executing" state requires a degree of luck to "View Jobs" at just the right time. More likely you'll see some "Ended" and some still "Executing."

When all the jobs are done you will see something like this:

If you wish to download the complete job log then click on the top-job and then the download button. Compute Grid will combine the job logs of each sub-job into the job log of the top-job.

You may, if you wish, view or download the job-log of a given sub-job. That may be useful to see the specific issues related to a sub-job that did not end successfully.

124 Provided things work as intended. If something doesn't work as intended you may see "Restartable" or "Failed."




Backup DocumentationInformation Centers

InfoCenter URLs

WebSphere Compute Grid V8http://publib.boulder.ibm.com/infocenter/wasinfo/cgwas80/index.jsp

WebSphere Application Server V8.5http://publib.boulder.ibm.com/infocenter/wasinfo/v8r5/index.jsp

Search Scope

The InfoCenter has a function to limit the search scope of what you see, from "All Topics" to just those you specify.

By default it will be "All Topics." You may verify that by looking at the top of your browser screen:

Searching with a scope of "All Topics" will work, but you may see multiple articles returned. That's because the topic applies to several scopes.

You may limit the scope by clicking on the "Scope" link and then:



http://publib.boulder.ibm.com/infocenter/wasinfo/v8r5/index.jsp

http://publib.boulder.ibm.com/infocenter/wasinfo/cgwas80/index.jsp


1. Click on the "New" button

2. Give your custom scope a name

3. Select those topics or sub-topics you wish to have included in your scope

4. Click on the "OK" button

5. You should see your new scope

6. Click "OK"

7. You should see your new scope shown as in effect

8. You should see only the topics you selected showing in the list

Now when you search on a string you will receive back only those hits within the topics of your specified scope.

Samples

The downloadable Java batch samples are found in the "Samples, V8.5" topic.

If you go to the WAS V8.5 InfoCenter and your scope is "All Topics," or your custom search scope125 includes "Samples, V8.5," then search on welcome_samples. You will get one article, which looks like this:

Click on the "Downloads" tab as shown, then scroll down to find the "Java Batch" samples:

Each sample may be downloaded with either FTP or HTTP:

Code generated by the Call Stub Generator

ADDER.javapackage com.ibm.cobolstub;

import com.ibm.ws.batch.ilc.impl.ILProcedureImpl;import com.ibm.cobolstub.parameters.adder_input;import com.ibm.cobolstub.parameters.adder_return;

public class ADDER extends ILProcedureImpl {

private adder_input _adder_input; private adder_return _adder_return;

private static int RETURNVALUE = 0; private static int ADDER_INPUT = 1;

public ADDER() { super("ADDER", // Cobol Load Module "ADDER", // Procedure entry point 2, // the parameter count (01 level), including return value

125 See "Search Scope" on page 77.




"Unknown Build", // user-defined build identifier "2012/08/13 09:52:56", // stub generation date null); // optional, user defined test stub

_adder_return = new adder_return();

_adder_input = new adder_input();

}

public byte[][] getParmList() { byte[][] parmList = super.getParmList(); parmList[RETURNVALUE] = _adder_return.getBytes(); parmList[ADDER_INPUT] = _adder_input.getBytes(); return parmList; }

/** * <pre>LINKAGE SECTION. * 01 adder-return. * 05 int-c PIC 9(8). * </pre> */ public adder_return getReturnValue() { return _adder_return; }

/** * <pre>LINKAGE SECTION. * 01 adder-input. * 05 int-a PIC 9(8). * 05 int-b PIC 9(8). * </pre> */ public adder_input getadder_input() { return _adder_input; }

}

Data Bindings

We are not showing the data bindings generated since they are quite long. For example, the adder_input.java bindings was 475 lines long, and that was for two simple integer input values. The adder_return.java source was of comparable length.

xJCL for the Mailer Sample

The following is a listing of the xJCL for the Mailer sample parallel application. What's shown here is not exactly like what comes with the sample ... some XML not central to the discussion of the application have been removed. Also, some of the indentation and formatting has been adjusted to better illustrate the sections of the XML.

Start of Job <job name="MailerSample" ... > Start of Job-Level Properties 1 <run instances="multiple" jvm="multiple"> <props>2 <prop name="com.ibm.websphere.batch.parallel.parameterizer"

value="com.ibm.websphere.samples.spi.MailerParameterizer"/>3 <prop name="com.ibm.websphere.batch.parallel.synchronization"



value="com.ibm.websphere.samples.spi.MailerSubJobCollector"/> <prop name="parallel.jobcount" value="3" />4 <prop name="FILENAME" value="/tmp/PJM" /> </props> </run>




<checkpoint-algorithm name="recordbased"> <classname>com.ibm.wsspi.batch.checkpointalgorithms.recordbased</classname> <props>5 <prop name="recordcount" value="100" /> </props> </checkpoint-algorithm> : <substitution-props>6 <prop name="EXCHANGED_FILENAME" value="/tmp/promotionalMailings.txt" /> </substitution-props> End of Job-Level Properties

Start of the first Job Step <job-step name="IdentifyRecipientsStep"> <classname>com.ibm...GenericXDBatchStep</classname>7 <checkpoint-algorithm-ref name="recordbased" /> <batch-data-streams>8 <bds> <logical-name>inputStream</logical-name> <impl-class>9 com.ibm.websphere.batch.devframework.datastreams.patterns.JDBCReader </impl-class> <props> <prop name="IMPLCLASS"10 value="com.ibm.websphere.samples.CustomerJDBCReader" />11 <prop name="ds_jndi_name" value="jdbc/mailing" />12 <prop name="STATES_LIST" value="${STATES_LIST}" /> </props> </bds>13 <bds> <logical-name>outputStream</logical-name> <impl-class>14 com.ibm.websphere.batch.devframework.datastreams.patterns.TextFileWriter </impl-class> <props> <prop name="IMPLCLASS"15 value="com.ibm.websphere.samples.PromotionalMailingFile" />16 <prop name="FILENAME" value="${EXCHANGED_FILENAME}" />17 <prop name="AppendJobIdToFileName" value="false" /> </props> </bds> </batch-data-streams> 18 <props> <prop name="BATCHRECORDPROCESSOR"19 value="com.ibm.websphere.samples.IdentifyRecipientsStep" /> <prop name="AD_CAMPAIGN_CODE" value="A1025" />20 <prop name="PROMOTIONAL_MAILER_THRESHOLD" value="5000" />21 <prop name="com.ibm.wsspi.batch.parallel.jobname"

value="${parallel.jobname}" /> <prop name="com.ibm.wsspi.batch.parallel.logicalTXID"

value="${logicalTXID}" /> <prop name="com.ibm.wsspi.batch.parallel.jobmanager"

value="${parallel.jobmanager}" /> </props> </job-step> End of the first Job Step

Start of Second Job Step <job-step name="SendPromotionsStep"> <classname>com.ibm...GenericXDBatchStep</classname> <checkpoint-algorithm-ref name="recordbased" /> <batch-data-streams> <bds>




<logical-name>inputStream</logical-name> <impl-class>22 com.ibm.websphere.batch.devframework.datastreams.patterns.TextFileReader </impl-class> <props> <prop name="IMPLCLASS"23 value="com.ibm.websphere.samples.PromotionalMailingFile" />24 <prop name="FILENAME" value="${EXCHANGED_FILENAME}" /> </props> </bds> <bds> <logical-name>outputStream</logical-name> <impl-class>25 com.ibm.websphere.batch.devframework.datastreams.patterns.JDBCWriter </impl-class> <props> <prop name="IMPLCLASS"26 value="com.ibm.websphere.samples.CustomerPromotionDateWriter" />27 <prop name="ds_jndi_name" value="jdbc/mailing" /> </props> </bds> </batch-data-streams>

<props> <prop name="BATCHRECORDPROCESSOR"28 value="com.ibm.websphere.samples.SendPromotionsStep" />29 <prop name="com.ibm.wsspi.batch.parallel.jobname"

value="${parallel.jobname}" /> <prop name="com.ibm.wsspi.batch.parallel.logicalTXID"

value="${logicalTXID}" /> <prop name="com.ibm.wsspi.batch.parallel.jobmanager"

value="${parallel.jobmanager}" /> </props> </job-step> End of the second Job Step </job> End of Job

1. The <run> property which indicates multiple instances (parallel job) and multiple JVMs (dispatch to multiple job endpoints if available).

2. This defines the class that implements the "parameterizer" for this parallel job. See "The Mailer sample parameterizer in detail" on page 71.

3. The other three batch.parallel properties are discussed under "Other Parallel Job Manager properties and what they do" on page 73.

4. This property is used by the parameterizer to establish the file name to be used by each sub-job as it writes out from the first step and reads from the second step.

5. Definition of a checkpoint algorithm that is record-based with the record count value set to 100. That means a checkpoint will be taken every 100 records for those job steps that reference this checkpoint algorithm. You will see that both job steps reference this checkpoint algorithm, so both steps will operate with a record-based checkpoint policy with a record count of 100 in effect.

6. This section of xJCL should be removed (or commented out). The parameterizer has code to use the FILENAME property above and to populate the value of EXCHANGED_FILENAME for each sub-job. This substitution property alters the expected behavior. The job still runs, it's just the file name created is not what is expected from the code in the parameterizer.

7. The checkpoint algorithm defined earlier is referenced for the first job step.

8. This marks the start of the first of two Batch Data Stream (BDS) definitions for this job step.

9. This defines the BDS pattern to use ... in this case, "JDBCReader" since the first job step reads from the database.




10. This defines the customer implementation that will be invoked when reading from this input BDS. The Mailer sample implementation -- CustomerJDBCReader -- sets up the SQL query to be issued and the mapping of the query results to a customer data object.

11. The JNDI name that will be looked-up and used to access the relational database.

12. Establishment of the property STATES_LIST, which resolves to a substitution property. The value for this is created by the parameterizer for each sub-job. See "The Mailer sample parameterizer in detail" on page 71 for more detail on how that is done.

13. The start of the second of two BDS definitions for this job step.

14. The BDS pattern to use ... TextFileWriter since the output is written to a file.

15. The customer implementation that will be invoked when writing to this output BDS.

16. The output filename to which the customer implementation of the output BDS will write. Each sub-job will write to a uniquely named output file. That file name is created in the parameterizer. See "The Mailer sample parameterizer in detail" on page 71.

17. This is a property we found you should add to the xJCL so the file name created by each such-job is as expected, based on the code in the parameterizer.

18. The start of the property definitions for the batch job step.

19. The customer code that implements the batch step. For each record read the logic in this file determines if the income is greater than the specified threshold, and if so, then it prepares to write to the output file. See "Overview of the Mailer sample application" on page 68.

20. The income threshold value used by the logic in the IdentifyRecipientsStep class.

21. The Parallel Job Manager settings to be used by this job step. All are set by the PJM as substitution variables to be resolved by each subjob at the time of execution.

22. In the second job step the first of two BDS defines the use of the TextFileReader pattern since the input will be the file written in the first step.

23. The customer implementation of this BDS -- PromotionalMailingFile. The logic in this file determines if the record is for an e-mail promotion or postal mail promotion and then prepares to write the output.

24. The file read by the input BDS is determined by this property, which is a substitution property set by the parameterizer. Each sub-job writes to its own unique file and reads back from that same unique file.

25. The output BDS names the JDBCWriter pattern since the output will be to update the database table with information about the promotional mailing.

26. The customer implementation of the output BDS. That class sets up the SQL to update the lastOfferDate field in the database table for the customer ID.

27. The JNDI name used to access the relational data system where the table resides.

28. The customer implementation of the job step function.

29. The PJM properties for the job step, again resolved to variables set by the PJM earlier in the execution of the job.




Document Change HistoryCheck the date in the footer of the document for the version of the document.

September 11, 2012 Original document at time of Techdoc creation

November 19, 2012 Small correction of "LE enclave" to "LE environment" for COBOL Container.

End of WP101783



WebSphere Application Server for z/OS V6 - IBM WWW Page · WebSphere Application Server for z/OS...

Documents

Transcript of WebSphere Application Server for z/OS V6 - IBM WWW Page · WebSphere Application Server for z/OS...