WebSphere Application Server for z/OS V6 - IBM WWW Page · WebSphere Application Server for z/OS...

of 83 /83
WebSphere Application Server for z/OS IBM Compute Grid and WebSphere Java Batch Version Date: November 19, 2012 See "Document Change History" on page 83 for a description of the changes in this version of the document IBM Advanced Technical Skills Gaithersburg, MD WP101783 at ibm.com/support/techdocs © IBM Corporation 2012

Embed Size (px)

Transcript of WebSphere Application Server for z/OS V6 - IBM WWW Page · WebSphere Application Server for z/OS...

  • WebSphere Application Server for z/OS

    IBM Compute Grid and WebSphere Java Batch

    Version Date: November 19, 2012

    See "Document Change History" on page 83 for a descriptionof the changes in this version of the document

    IBM Advanced Technical SkillsGaithersburg, MD

    WP101783 at ibm.com/support/techdocs

    © IBM Corporation 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Many thanks to Sajan Sankaran, Chris Vignola, Rajesh Ramachandran and Tim Fanelli. Each works at IBM and

    each provided valuable input and feedback on this document.

    The WAS z/OS team in ATS consists of John Hutchinson, Mike Kearney, Mike Loos, Louis Wilen, Lee-Win Tai and Don

    Bagwell. Glenn Fisher manages the team.

    A special "thank you" to Lee-Win Tai for his help with Java as well as understanding the sample applications.

    Mike Cox, Distinguished Engineer, serves as technical consultant to all our activities.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 2 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Table of ContentsExecutive Overview...............................................................................................................................5Introduction to Java Batch Processing...............................................................................................6

    What is 'batch' processing............................................................................................................................6IBM batch processing solutions on the mainframe.......................................................................................6Java as a batch processing language...........................................................................................................6Invocation of Java batch...............................................................................................................................7The shrinking and disappearing 'batch window'...........................................................................................7Java batch execution within the WAS Java EE environment.......................................................................8Java batch programming and execution framework.....................................................................................8Integration of existing (and valuable) non-Java assets.................................................................................9The parallel nature of some batch processing..............................................................................................9Integration with critical enterprise scheduling tools......................................................................................9Summary.....................................................................................................................................................10

    Technology Overview.........................................................................................................................11WebSphere Application Server Version 8.5...............................................................................................11

    Functional services made available to applications by WAS...........................................................................11Consistent support of specification and APIs across platforms.......................................................................11WAS z/OS and System z platform exploitation...............................................................................................12Servers, nodes, clusters and cells..................................................................................................................13

    The z/OS application server design.................................................................................................................................13Accessing applications in WAS.......................................................................................................................14Accessing data from application in WAS.........................................................................................................15Transactionality............................................................................................................................................... 15High availability............................................................................................................................................... 16Compute Grid incorporated as function in WAS V8.5.....................................................................................17Other sources of information on WAS z/OS....................................................................................................17

    WebSphere Compute Grid..........................................................................................................................18Components of the Compute Grid model........................................................................................................18Jobs, steps, the Java batch application code, and xJCL.................................................................................20Submitting Jobs to Compute Grid...................................................................................................................21

    Job state -- submitted, executing, ended ... and more.....................................................................................................21Transactional batch and Compute Intensive batch jobs..................................................................................22Functional services provided by Compute Grid...............................................................................................23

    Application Support Functions.........................................................................................................................................23Administrator Support Functions......................................................................................................................................26z/OS Platform Exploitation...............................................................................................................................................26

    WSGRID and integration with existing job schedulers....................................................................................27The Parallel Job Manager function.................................................................................................................28Developing Java batch applications for Compute Grid...................................................................................30

    Technology Focus...............................................................................................................................33The COBOL Container................................................................................................................................33

    Architectural overview..................................................................................................................................... 33WAS z/OS application server setup requirements..........................................................................................34COBOL program compilation requirements, program restrictions...................................................................35COBOL stub generator................................................................................................................................... 36

    Overview of the COBOL Call Stub Generator..................................................................................................................36Installing the COBOL Call Stub Generator on your workstation......................................................................................37Requirements for IBM Rational Application Developer....................................................................................................38What the COBOL Stub Generator generates...................................................................................................................39Invoking from the Command Line....................................................................................................................................41Invoking as ANT task in RAD...........................................................................................................................................45Invoking as ANT and processing multiple COBOL files...................................................................................................47

    Invoking the COBOL call stub from the Java batch application.......................................................................48Sharing DB2 Type 2 connection between Java and COBOL..........................................................................49

    Programming enhancements in CG 8 (and WAS 8.5)................................................................................49Compute Grid Listeners.................................................................................................................................. 49

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 3 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Job listener...................................................................................................................................................... 50Overview of the job listener..............................................................................................................................................50Implementing a job listener..............................................................................................................................................50Registering a job listener..................................................................................................................................................51

    Skip-record processing................................................................................................................................... 52Background context on need for skip-record processing.................................................................................................52Configuring and activating a skip-record policy................................................................................................................52What happens when a record is skipped?.......................................................................................................................54Sample application for illustrating skip-record processing...............................................................................................54Registering and using a skip listener...............................................................................................................................55Skip-record metrics.......................................................................................................................................................... 56

    Retry-step processing..................................................................................................................................... 57Overview of retry-step processing....................................................................................................................................57Sample application used for illustration............................................................................................................................58Configuring and activating a retry-step policy..................................................................................................................58How the skip-record sample application implements retry-step processing....................................................................61

    Configurable transaction mode.......................................................................................................................64Overview of the function................................................................................................................................................... 64Configuration in xJCL....................................................................................................................................................... 64Getting the shared connection from the Job Step Context Manager...............................................................................65

    Parallel Job Manager and enhancements in CG 8 (and WAS 8.5)............................................................65General enhancements in PJM function.........................................................................................................65Control over where parallel subjobs are dispatched -- threads or JVMs.........................................................65Mixing of job-step types within a multi-step job...............................................................................................67A look inside the Mailer sample parallel program............................................................................................67

    Overview of the Mailer sample application.......................................................................................................................68Mailer sample application -- xJCL overview.....................................................................................................................69Parallel processing for the data in the Mailer sample application....................................................................................69The Parameterizer is what determines how to partition the data.....................................................................................70The Mailer sample parameterizer at a high-level.............................................................................................................71The Mailer sample parameterizer in detail.......................................................................................................................71Other Parallel Job Manager properties and what they do................................................................................................73

    Submitting the Mailer job -- one top job and three sub-jobs............................................................................75Backup Documentation.......................................................................................................................77

    Information Centers....................................................................................................................................77InfoCenter URLs............................................................................................................................................. 77Search Scope................................................................................................................................................. 77Samples.......................................................................................................................................................... 78

    Code generated by the Call Stub Generator..............................................................................................78ADDER.java.................................................................................................................................................... 78Data Bindings.................................................................................................................................................. 79

    xJCL for the Mailer Sample.........................................................................................................................79Document Change History..................................................................................................................83

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 4 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Executive OverviewThis document started out as a means to focus attention on four elements of the IBM Compute Grid product that are new in Version 8, or relatively new and deserving of a fresh look. These are the items found under the "Technology Focus" section starting on page 33:

    COBOL Container Provides a means of executing compiled COBOL modules inside the address space of the WAS z/OS application server. The COBOL container function provides the infrastructure to call COBOL from Java and exchange information in a very efficient manner

    33

    Programming Enhancements

    Job Listener The Job Listener provides a means of registering application code to get control at key points in the processing of a Java batch job. That allows the batch programmer to do setup and cleanup processing as needed, with the Compute Grid batch container invoking the application code at those points in the processing.

    50

    Skip-Record Skip-record processing provides a way to define how the Compute Grid batch container may skip records for which read or write exceptions have occurred. The function provides control over the number of records that may be skipped and the exceptions that may be included or excluded. A skip-record listener function, similar in concept to the Job Listener, is also provided.

    52

    Retry-Step Retry-step processing provides a way to define how the Compute Grid batch container may retry job steps for which an exception was thrown. The function provides control over the number of retries, the delay between retries and he exceptions to be included or excluded. A retry-step listener is provided.

    57

    Transaction Mode Provides a way to define local mode transactions rather than global, which provides increased efficiency of operations.

    64

    Parallel Job Manager Enhancements

    The Parallel Job Manager function provides the ability to process batch jobs in parallel, with facilities to programmatically set data range parameters, and with facilities to manage the sub-jobs as part of the overall job. This function has been enhanced to provide mixed batch step types within a job as well as specifying the parallel processing to be done at the job step level.

    65

    In writing the technical specifics it became apparent that discussing enhancements to a functional offering like Compute Grid required context. Some readers might not be familiar with Compute Grid. Or, for that matter, familiar with the underlying Java runtime platform provided by WebSphere Application Server. So we added a "Technology Overview" section starting on page 11.

    One final piece of context was required -- setting the very basic context about batch processing and what Compute Grid provides that addresses some of the issues surrounding batch processing using Java. That is provided under "Introduction to Java Batch Processing" starting on page 6.

    So the document proceeds from relatively high-level concepts to more and more technical specifics about Compute Grid and what it provides.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 5 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Introduction to Java Batch ProcessingBatch processing has been a part of business information technology since the very early days. It remains a large and important part of today's business information processing workloads.IBM technology, and IBM mainframe technology in particular, has played a key role in the industry's approach to batch processing. That was true in the early days; it is true now.What is 'batch' processing

    Various definitions exist, and in essence they all have at its core the idea of processing data with little or no human interaction. Contrast that with Online Transaction Processing (OLTP) where human interaction is involved repeatedly.An example of batch processing would be a program that calculated the sales tax on every customer account in a database1. The batch program would read in the first customer record, calculate the tax, insert the tax into the database and total the amount owed, then move to the next customer record. If the customer database consisted of one million records, then the batch program would loop through its processing one million times.

    IBM batch processing solutions on the mainframeThe following table summarizes a few solutions:

    Language compilers such as COBOL Allow the development and execution of batch programs run in the traditional manner; that is, as submitted jobs using JCL that calls the compiled program.

    BPXBATCH A facility of Unix Systems Services (USS) that allows launching of a UNIX shell environment from JCL. This in turn allows execution of shell scripts or the launching of a Java Virtual Machine (JVM) to run Java programs in batch.

    JZOS A set of functions and programming libraries that make execution of Java in a batch JVM easier. IBM acquired the technology2 and incorporated it into the z/OS operating system.

    z/OS 1.13 Batch Runtime A framework to better facilitate Java to COBOL interoperability with transactional updates to DB2. This batch runtime environment uses a subset of the WebSphere Compute Grid function, which is discussed next.

    WebSphere Compute Grid A Java batch execution runtime and programming framework that operates within the WebSphere Application Server Java EE environment. The most current version of Version 8. See "WebSphere Compute Grid" on page 18 for an overview.

    WebSphere Application Server V8.5 The Java batch function of WebSphere Compute Grid V8 was incorporated into the WebSphere Application Server V8.5 offering. Therefore, with WAS V8.5 you have access to all the function found in Compute Grid V8. See "WebSphere Application Server Version 8.5" on page 11 for an overview.

    The focus of this document will be on Compute Grid and WAS V8.5, which are highlighted in yellow here.

    Java as a batch processing languageThere is a growing interest in using Java for batch processing. The trend towards using Java has been underway for several years now. The reasons for considering Java for batch processing include:

    1 This is an example of what we call a "transactional" batch program. Another form is a "compute intensive" batch program. You'll see both phrases used later in this document.

    2 IBM acquired the technology from Dovetail Technologies.© 2012, IBM CorporationAmericas Advanced Technical Skills - 6 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    • Skills -- Java is a programming language for which there is a good supply of skills in the market. Skills for other languages, such as COBOL or Assembler, are less readily available. Further, Java is a programming language many companies are standardizing on as the core development language for applications. It makes sense to extend this standardization to batch processing as well.

    • Development Tooling -- there is a rich set of development tooling available for Java. Many companies have standardized on tooling such as IBM's Rational Application Developer for Java development. Developing batch processing programs in Java allows companies to leverage existing tooling3.

    • Specialty Engines -- on the IBM System z mainframes specialty engines4 provide a means of offloading certain classes of work (Java being one class of work) to a processing engine other than the General Processors (GP engines). This provides meaningful cost reduction in acquisition as well as ongoing software licensing charges.

    Other motivations to use Java for batch processing exist. The point is that there are valid reasons to consider Java for batch processing. Many have already chosen Java for batch processing requirements.

    Invocation of Java batchJVM launcher approaches such as BPXBATCH or JZOS work on the principle of initiating the JVM, running the named Java program, then tearing down the JVM to complete the process. That works, but it carries with it the overhead of JVM creation and destruction. When Java batch programs are run hundreds (or thousands) of times a day, that overhead becomes excessive.That is why the Compute Grid and WAS V8.5 Java batch function operates on the principle of a JVM environment that is maintained across invocations of batch programs. The overhead of JVM creation and destruction is incurred far less frequently.How Java batch programs are invoked and managed within the Compute Grid environment is discussed under "WebSphere Compute Grid" on page 18.

    The shrinking and disappearing 'batch window'Years ago companies would operate with two windows of processing -- online processing during the day, and batch processing at night. Often there was no overlap of the two; no batch was run during online processing, and no online was run during the batch window.Many factors have led to the reduction or elimination of the dedicated batch window. The most obvious factor is the requirement to make online processing available 24 hours a day.In addition, there is a growing need to execute certain batch processes throughout the day, not just at night. What results is a blend of online processing with batch, and with batch increasingly executing in smaller segments throughout the day:

    This suggests the need for coordinated intermixing of online and batch. That is one of the

    3 IBM Rational Application Developer 8.0.1 and later has tooling support for batch processing in Compute Grid.4 zIIPs, zAAPs, and the newer "zAAP on zIIP" approach© 2012, IBM CorporationAmericas Advanced Technical Skills - 7 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    attributes of IBM's Compute Grid because it operates within the same Java EE environment as OLTP. The runtime (WebSphere Application Server) can coordinate the execution.

    Java batch execution within the WAS Java EE environmentWebSphere Application Server is IBM's Java runtime environment that implements the Java EE standards. WAS is a fully compliant Java EE server runtime.Java EE is in many ways an online processing runtime model, but it is not limited to just that. It has the ability to execute asynchronous threads. This allows long-running tasks to operate inside the JVM but outside the timeout constraints of OLTP processing. That is the basis for Java batch execution in WAS provided by Compute Grid.The IBM WebSphere Compute Grid product provides a set of functions that are installed into a WAS runtime environment. Those functions provide a Java batch execution platform:

    Notes: • WebSphere Compute Grid is a product offering supported on all operating system platforms supported by WebSphere Application Server itself. That includes Windows, AIX, Linux, z/OS and others.

    • With the release of WebSphere Application Server V8.5 the Compute Grid product function has been incorporated into the WAS product itself. The Java batch function provided by Compute Grid V8 is identical to the batch function included with WAS V8.55.

    WebSphere Application Server and Compute Grid are explored at an overview level under the heading "Technology Overview" starting on page 11.

    Java batch programming and execution frameworkCompute Grid provides a batch programming framework that provides a set of valuable services to Java developers. For example:

    • A means of using common data access mechanisms to read or write data, iterate through the data records, and map the records to data objects6. This eliminates the need for developers to "roll their own" function, which in turn allows the developers to focus on business logic. It also avoids the build-up of custom code which is costly to support in the future.

    • A means of specifying transactional checkpoint intervals. The Batch Data Stream framework maintains awareness of the checkpoint status and handles commits and rollbacks, as well as job pausing and job restarting at these checkpoints.

    • A means of declaring job properties separate from the actual Java application code. This is done with an XML file called "xJCL" which Compute Grid reads to understand such things as the number of job steps to process, the Java classes to call for each step, and many other operational parameters related to the job.

    5 The Compute Grid product is still available for customers not yet at WAS V8.5 and not wishing to migrate their environment to that level at this time. Compute Grid V8 is compatible with WAS V7 and V8.

    6 This is the "Batch Data Stream" framework. See "Batch Data Stream Framework" on page 23.© 2012, IBM CorporationAmericas Advanced Technical Skills - 8 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    • A means of defining job execution behavior when specific data records are seen as invalid. This allows otherwise good data to operate when occasional bad records are encountered.

    Writing Java batch programs that provide all these functions will prove to be a time-consuming process and expensive to maintain in the long run. Compute Grid provides a Java batch programming and execution framework which your Java batch programs use for such functions. Compute Grid is IBM-supported Java batch middleware. You focus on the Java batch logic and IBM focuses on the Java batch runtime.

    Integration of existing (and valuable) non-Java assetsThe trend towards batch processing written in Java does not imply the elimination of existing batch assets. COBOL is the most commonly cited form of existing batch asset, though others exist (such as assembler).While some COBOL batch assets are being reengineered to Java, most continue to be of value to the business in their present form. The desire is to not replace the COBOL in all cases but rather to integrate existing COBOL assets with newer Java batch processes.To address this, the Compute Grid product (as well as the WAS V8.5 product) provide a "COBOL Container" technology. This allows compiled COBOL modules to be run within the WAS z/OS server address space, which the Java program invoking the COBOL across an IBM-written JNI interface. This allows existing COBOL assets to be used in concert with Java batch processes, with invocation of the COBOL accomplished with a minimum of latency and code path.This topic is explored more fully under "The COBOL Container" starting on page 33.

    The parallel nature of some batch processingSome batch processing designs are natural candidates for breaking the process up into smaller jobs and running them in parallel. This is particularly true when the source data is arranged in relational data stores in such a way where separate parallel jobs may access the data without data access contention.Go back to the earlier example of a transactional batch process that calculated the tax owed for each of a million customers on record. Imagine the customer billing records were indexed on customer number. The nature of that batch processing suggests potential for benefit if the customer records were partitioned into ten ranges with ten separate batch jobs each operating on its own range of customer billing data. A job that used to take 10 hours could, in theory, take 1 hour7.Compute Grid has a facility called the Parallel Job Manager which provides a means of programmatically indicating how large jobs may be partitioned and executed in separate batch execution end points. The Parallel Job Manager handles the partitioning and dispatching and monitors all sub-jobs as part of its overall submitted job management.In Compute Grid Version 8 (and WAS V8.5) several enhancements to the Parallel Job Manager have been provided. See "The Parallel Job Manager function" on page 28 for an overview of this function. See "Parallel Job Manager and enhancements in CG 8 (and WAS 8.5)" on page 65 for specifics on the enhancements.

    Integration with critical enterprise scheduling toolsEnterprise scheduling tools, such as IBM's Tivoli Workload Scheduler, are a critical component in the overall job execution management function in many companies. The emergence of Java batch has not eliminated the need for comprehensive job scheduling and coordination. The reverse is the case: the requirement is to insure Java batch execution can be integrated into existing scheduling functions.

    7 Assuming the processing capacity was present to execute all ten jobs concurrently, and assuming no other contention issues were to arise. The point here is that where parallelization suggests itself there is the potential for benefit.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 9 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Compute Grid provides an enterprise scheduler integration function called "WSGRID." It allows enterprise job schedulers to submit JCL which runs a program that makes contact with Compute Grid and requests the submission of the Java batch job within Compute Grid. Throwing a request over the wall is not enough, however. The enterprise job schedulers are also interested in when the Java batch job completes and the success or failure of the submitted Java batch job. The enterprise schedule may have other jobs whose submission is conditional upon successful completion of the Java batch job.Further still, collection and retention of the Java batch job log is important, which means the Java batch output from Compute Grid must find its way back to JES spool.The WSGRID function takes all this into account -- it maintains contact with Compute Grid for the duration of the job execution within Compute Grid; it streams the Java batch job log records back to JES spool; and it reports back to the enterprise scheduler the return code seen from execution in Compute Grid.See "WSGRID and integration with existing job schedulers" on page 27 for more on this function of Compute Grid.

    SummaryThe objective of this introductory section was to establish the context for Java batch processing and to place that within a framework of understanding of IBM's Compute Grid function.The next section of this document will provide a more detailed overview of WebSphere Application Server and WebSphere Compute Grid.The final section will provide even greater detail on specific elements of Compute Grid V8, which is identical to the Java batch function that has been incorporated into the WebSphere Application Server V8.5 product. Most of the functions of Compute Grid and WebSphere Application Server are in fact common across all supported operating system platforms. But some functions are exclusive to z/OS. For example, the COBOL Container technology is a z/OS exclusive. Other z/OS platform exploitation performed by Compute Grid z/OS includes z/OS WLM workload classification and the writing of SMF usage records.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 10 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Technology OverviewWebSphere Application Server Version 8.5

    WebSphere Application Server (WAS) is IBM's implementation of the Java EE server runtime model. As such, WAS implements a wide range of industry standards for Java programs such as standards for servlets and JSPs, EJBs, data access such as JDBC, and a many others.IBM provides WebSphere Application Server all its server platforms -- z, p, i and x. WAS is also supported on HP-UX and Solaris. WAS Version 8.5 is IBM's latest release of WebSphere Application Server. It was made available in June of 2012.Functional services made available to applications by WAS

    The purpose of an application server is to provide a set of functions and services to applications. Applications stay focused on the business logic and let the server platform provide common functions and services:

    Those services are exposed as documented application programming interfaces (APIs). Further, the functions and APIs in WebSphere Application Server are all industry standards8.The following URL is the WAS V8.5 Information Center:http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jspIf you search on the keyword rovr_specs you will get a page that summarizes all the APIs and specifications supported by WebSphere Application Server by version and release.You will see that WebSphere Application Server supports a long list of standards9.

    Consistent support of specification and APIs across platformsAs mentioned earlier, IBM supports WebSphere Application Server on many different operating system platforms -- z/OS, i/OS, AIX, Linux, Windows, HP-UX and Solaris.Key Point The specifications and APIs supported by WebSphere Application Server are

    consistent across all supported operating system platforms. That means for any given version or release, if one platform supports a specification, all other platforms will as well.

    This provides applications with platform mobility -- that is, an application written to the supported specifications will run on any WAS platform at that same version and release level10.Does that mean there is no difference between platforms? No. Each platform provides unique attributes that may be exploited by WAS below the standard specification line. That is the key to understanding WAS on z/OS. The z/OS operating system provides a long list of exploitable functionality which WAS on z/OS takes advantage of.

    8 Meaning people from different companies in the industry met and agreed to the specifications. Standard specifications allow for broader acceptance of the technology. The specifications do not dictate how specifications are to be implemented, just what the programming interfaces must be. Vendors then compete on the quality and completeness of their implementations.

    9 It is not necessary to understand what each of those standards provides. It is better to focus on the fact that a given standard is or is not supported, and the specification level supported for a given release of WAS.

    10 An perhaps different version and release levels, but for that case you would need to inspect the application to make sure there are no issues with function removal or deprecation.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 11 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

    http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsphttp://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-zos&topic=rovr_specs

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    WAS z/OS and System z platform exploitationAs mentioned, WAS on z/OS takes advantage of specific functions of the operating system. This is done below the standard specification line. The specifications and APIs supported by WAS on z/OS is the same as that support on any other platform for that release.The following picture illustrates this concept:

    The following table summarizes some of the key z/OS exploitation that takes place:WLM The multiple-JVM model of WAS z/OS (the controller and servant model) has at

    its heart z/OS WLM acting to classify and route work. WLM is used to route certain work within WAS z/OS clusters.

    SAF WAS z/OS uses the Security Access Facility (SAF) of z/OS for a wide array of security profiles, as well as a place to store digital certificates.

    SMF WAS z/OS uses Systems Management Facility (SMF) to log detailed records of request activity processed by WAS. This provides a great deal of valuable information that may be used for analysis, capacity planning and billing purposes.

    JES The z/OS Job Entry Subsystem (JES) is used to start and stop WAS z/OS servers as well as hold server logging output routed to JES spool.

    Cross Memory z/OS permits authorized processes to copy memory from one area of virtual memory to another. WAS z/OS takes advantage of this in a variety of ways to provide very high speed transfer of data with a minimum of latency.

    Specialty Engines

    System z specialty engines (zIIP, zAAP) provide a means of offloading certain classes of work to lower-cost processors. Work executed on the specialty engines do not count towards other software license charges. The Java provided with WAS z/OS works with the dispatcher of z/OS to perform this offload function.

    There is a great deal of detail left out of that summary, of course. If you are interested in more detail the following Techdoc articles will be of interest:http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4848

    The presentation handouts from a technical hands-on workshop dedicated to the topic of WAS on z/OS and the platform exploitation that takes place.

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101532A presentation (with speaker notes) on the question "Why WAS z/OS?" The presentation details the value attributes of the System z and z/OS platform and explains how WAS z/OS benefits from them.

    Key Point Applications that run on WAS z/OS benefit from this exploitation that takes place under the open specification line. That includes functions such as Compute Grid. Compute Grid on WAS z/OS allows Java batch to execute with the benefits of z/OS.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 12 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101532http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4848

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    See "Other sources of information on WAS z/OS" on page 17 for a more comprehensive list of WAS z/OS technical documentation.

    Servers, nodes, clusters and cellsWebSphere Application Server across all platforms has the concept of "servers," "nodes," "clusters" and "cells." Those terms represent ways in which the JVMs are logically organized to form the WAS runtime topology. The following picture illustrates this, with notes that correspond to the numbered blocks:

    1. An application server contain a JVM11 in which the applications run. An application server may host a single application (for isolation purposes) or multiple applications.

    2. A node is a logical collection of application servers on a given server platform. For example, a given Windows server, or a z/OS logical partition (LPAR).

    3. A cluster is a logical grouping of application servers, often on different server platforms, for the purpose of providing additional availability.

    4. A cell is a logical grouping of nodes for the purpose of defining the extent of control for the administrative server, called the "Deployment Manager." Cells are often used to provide administrative isolation between, for example, test and production environments.

    The general concepts of servers, nodes, clusters and cells apply to WAS on all platforms. Where WAS on z/OS differentiates is in the design of the application server.The z/OS application server design

    WebSphere Application Server on z/OS has a unique application server model. Where WAS on every other platform has a single JVM for an application server, WAS on z/OS has multiple JVMs. The following picture illustrates this at a very high level:

    11 On z/OS the "multi-JVM" server model implies at least two JVMs (controller and servant) and potentially more if multiple servants are configured.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 13 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    The controller region JVM hosts IBM-written code that does all the request handling. User application code does not run here.The controller queues work request with WLM, which then dispatches the work to a servant region. Earlier we spoke of WLM as being an element of the z/OS system exploited by WAS z/OS ... this is a summary view of that exploitation.The picture illustrates stacked servant regions because it is possible to configure multiple servants to exist behind any given controller. Each servant would have its own physically separate address space and physically separate JVM.Multiple servant regions provide many benefits, including increased availability of applications and dynamic scaling of application resources.Note: There is a great deal of sophistication to this application server model. The WP101740

    document at ibm.com/support/techdocs provides much more information.For the purposes of setting context for the Java batch function of Compute Grid or that provided with WAS V8.5, the high-level overview offered in this document is sufficient.

    Accessing applications in WASApplications provide their intended function to those people or processes that make a request of the application. There are several essential ways in which requests may flow into an application hosted in an application server:

    HTTP The request flows on a TCP connection to the HTTP listener port of the application server. A common form of this would be web browser requests, where browser HTML flows over the HTTP connection. But this request type also includes web services, where SOAP or other HTTP verbs are used to invoke an action and receive a result.

    Message Queueing With message queueing a program outside of WAS puts a message on a queue, and the application in WAS retrieves the message from the queue. The application may either perform a GET operation to retrieve the message or be triggered to retrieve the message if the application is a Message Driven Bean (MDB).A common message queueing product used is IBM's WebSphere MQ. Another messaging queueing function is the Service Integration Bus (SIBus), which is a Java-based queueing mechanism built into WAS.

    IIOP Internet Inter-ORB Protocol is a means of one Java program invoking another Java program remotely over a TCP network. A very common form of this request type is internal to an application server where a servlet invokes an EJB.

    WOLA WebSphere Optimized Local Adapters (WOLA) is a WAS z/OS exclusive function for cross-memory transfer of requests into an application.

    Other Other request types exists, such as Session Initiation Protocol (SIP) and various internal work requests.

    Again, considerable detail is left out of that summary chart. The point here is that WebSphere Application Server supports many different means of inbound requests to initiate action and provide results.

    Java Batch When you get to the WebSphere Compute Grid section (page 18) you will see that Compute Grid -- which may be thought of as a very sophisticated application that runs inside of WebSphere Application Server -- makes use of many of those request types to process Java batch jobs.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 14 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Accessing data from application in WASFor an application to performance meaningful value it requires data to act upon. Some of the data may come from the requester, but in most cases the request triggers the application to go get data from some other data store, such as a database.The following table summarizes some of the ways in which applications in WAS may access data outside of WAS:

    JDBC JDBC provides an industry standard means of accessing relational databases, such as IBM's DB2. Vendors of relational databases write and supply "provider code" that performs the actual connectivity to the database. Applications make use of the JDBC interface to request the data, and the underlying provider code does the actual work to connect and retrieve the requested data.

    JCA JCA provides an industry standard means of accessing non-relational data systems, such as IBM's CICS or IMS products. Vendors of these non-relational systems write and supply "resource adapters" that install into WAS. The resource adapters provide the code that does the actual connectivity function. Applications make use of the JCA interface to request the data, and the underlying resource adapter code does the actual work to connect and retrieve the requested data.

    JMS JMS provides an industry standard means of accessing message queueing systems, such as IBM's MQ, or the built-in Service Integration Bus (SIBus) function. Applications seeking data can place a message on a queue, which is picked up by another program and the data requested is made ready. That program then returns via the message queue the requested data. The application in WAS pulls the message from the queue and processes the data.

    Web Services An application in WAS may form up and send a web services request to a service provider elsewhere in WAS, or anywhere else on the Internet.

    IIOP An application in WAS may form up and send an IIOP request to an EJB residing in the same server, the same cell, or another Java EE environment on the Internet.

    WOLA WOLA is a z/OS-exclusive cross-memory request function. It can be used to request data of CICS, IMS or batch programs running outside of WAS z/OS. Requests outbound from WAS z/OS make use a WOLA JCA adapter.

    Other Such as reading from a UNIX file, MVS data set or VSAM file; opening a TCP socket to another process; or processing FTP to retrieve data.

    In short, there is considerable flexibility in the means by which an application may reach out of WAS to retrieve data.

    Java Batch Java batch applications running within the Compute Grid function have access to all these forms of data retrieval. That illustrates an important point about Compute Grid: it is built upon the full function Java EE WebSphere Application Server platform. Therefore, services provided by WAS are available to the Java batch application12.

    TransactionalityApplications that make updates to data need to be aware of the need to maintain transactional consistency across all the data affected. That means, for example, that changes made to separate data stores be committed against both or rolled back against both. But never a case where data inconsistency is introduced by updating one but failing to update the other.WebSphere Application Server provides a transaction manager service to applications that run on WAS. That means that applications themselves do not need to write the transaction manager code, the application simply invokes the transaction service of the platform.

    12 The Batch Data Stream framework provides built-in patterns for JDBC read and write, JPA read and write, file read and write, and z/OS data set read and write. Other data access methods would require the Java batch application to implement the data access logic.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 15 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    WAS is not the only application server with a transaction manager ... CICS and IMS provide that service as well. This is where transaction management requires some care, which is why having that function as a service of the platform is so valuable -- when more than one transaction participant is involved, all the participants must coordinate to make sure the data commits are done together or rolled back together. That degree of coordination is part of the design of WebSphere Application Server as well as CICS and IMS.When multiple participants are involved in a transaction something must act as the repository of information of where the participants are in any given transaction. On z/OS that function is provided by Resource Recovery Services (RRS). For non-z/OS platforms that function is provided by in distributed XA processing in each manager.

    Java Batch Batch processing very often requires that transactional integrity be maintained. The Compute Grid function provides checkpoint services to Java batch applications so effective and efficient transactional commits may be performed. The Compute Grid function maps those requests down upon the transaction management function of the WAS platform. Once again, the message is this -- the Java batch application developer does not need to code this often-complex functionality. The Java batch application developer needs merely to indicate the checkpoint requirements in xJCL.

    High availabilityWhen application up-time is important to the business, the question of making the application "highly available" comes to the surface. The topic involves far more than simply duplicating server hardware resources.Earlier we mentioned "clusters" as a logical element of a WAS runtime topology. Clusters are a means of duplicating application and application servers across multiple server platforms as a means of reducing single point of failure. WAS clusters perform other availability services such as replicating HTTP session objects, or failing over server resources to recover transactions.On z/OS the availability is enhanced further by the Parallel Sysplex function of System z. At a very high level, Parallel Sysplex is a means of clustering z/OS operating system images with a data sharing facility in the middle called the Coupling Facility (CF). What's more, many IBM middleware products such as CICS, DB2, IMS and MQ are "Sysplex-aware" -- that is, they take direct advantage of Parallel Sysplex to augment availability.WebSphere Application Server for z/OS supports clustering across z/OS images and benefits from the Sysplex-aware middleware. When people speak of System z being "highly available," they are alluding to all this and much more.

    Java Batch Because the Compute Grid function exists within WebSphere Application Server, highly-available batch processing can be built upon what WAS itself offers. When WAS is on z/OS then the availability features of Parallel Sysplex may be employed to enhance the availability of the Java batch processing.For example, Compute Grid operating in a cluster on WAS z/OS may process a batch job on one logical partition (LPAR). If an outage is experienced on that LPAR, the batch job may be restarted on the cluster member of the other LPAR. Compute Grid will pick up at the last checkpoint. The shared data in the middle of Parallel Sysplex means Compute Grid components on one LPAR have access to data created and stored on another LPAR.

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 16 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Compute Grid incorporated as function in WAS V8.5Starting with WebSphere Application Server Version 8.5 (made available in June of 2012), the Java batch function equivalent to that offered in IBM WebSphere Compute Grid V8 was made part of WAS itself. IBM WebSphere Compute Grid V8 is still offered for those customers who are at WAS V7 or V8 (but not yet V8.5) that wish to use the Compute Grid function, but do not wish at this time to migrate their application server environment to WAS V8.5.The following picture represents this13:

    Other sources of information on WAS z/OSThere is a wealth of information on WebSphere Application Server for z/OS. The following list attempts to organize some of those sources of information:http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp

    The InfoCenter is the primary source of reference information for WebSphere Application Server14.

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4848The presentation handouts from a technical hands-on workshop dedicated to the topic of WAS on z/OS and the platform exploitation that takes place.

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102014Provides a step-by-step cookbook for the use of IBM Installation Manager used to install WebSphere Application Server for z/OS V8 or V8.5

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4686http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4944

    Spreadsheets used to format the input to the WebSphere Customization Tool (WCT) that generates the jobs to create a runtime on z/OS. PRS4686 is for V8, PRS4944 is for V8.5.

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101532A presentation (with speaker notes) on the question "Why WAS z/OS?" The presentation details the value attributes of the System z and z/OS platform and explains how WAS z/OS benefits from them.

    13 The picture uses the word "augment" to suggest the adding of the Compute Grid function to a WAS runtime. Augmentation involves running an IBM-supplied process (or on z/OS, a job) to update the WAS configuration to recongize and use the Compute Grid function.

    14 See "Search Scope" starting on page 77 for information about setting search scope when using the InfoCenters.© 2012, IBM CorporationAmericas Advanced Technical Skills - 17 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101532http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4944http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4686http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102014http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4848http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101740Provides a very detailed look at the architecture and operations of the multi-JVM application server model of WAS z/OS and the role of WLM as part of that design.

    http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101490Provides a detailed look at the WebSphere Optimized Local Adapters (WOLA) function of WAS z/OS.

    In addition to those shown here, the IBM Advanced Technical Skills (ATS) organization "Techdocs" website provides the search entry point to other documents on WAS z/OS as well as a wealth of other topics as well:http://www.ibm.com/support/techdocs

    WebSphere Compute GridIn this section we'll take a technical overview look at the Compute Grid function. The topics presented in this section apply to both the IBM WebSphere Compute Grid product offering, or the equivalent function that has been incorporated into the WebSphere Application Server V8.5 product offering15.Components of the Compute Grid model

    The following picture and the notes that accompany it explain at a high level the components of the Compute Grid model:

    Note: That picture may look a bit complex at first. It will become more clear as you read the notes that follow and gain more understanding of the Compute Grid function. The message this picture sends is that Compute Grid is a very structured and managed Java batch environment16. Compute Grid provides many of the same kinds of management and control for Java batch as you enjoy today with traditional z/OS batch.

    1. Any discussion of Compute Grid starts with the understanding that the Java batch function is built upon the foundation of WebSphere Application Server17. This is why the overview of WAS was offered starting on page 11. All of the functional services offered

    15 See "Compute Grid incorporated as function in WAS V8.5" on page 17 for a graphical representation of the relationship between the two.16 Implied in this picture is another message -- if you attempted to develop your own custom Java batch execution environment you would,

    eventually, end up creating many or all of these components. That would imply a great deal of custom middleware to support and maintain. With Compute Grid that middleware is not custom, it is vendor-supported.

    17 It may help to think of Compute Grid as a sophisticated IBM-written Java application deployed into WAS. © 2012, IBM CorporationAmericas Advanced Technical Skills - 18 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

    http://www.ibm.com/support/techdocshttp://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101490http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101740

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    by WAS itself are functional services available to Compute Grid and, by extension, Java batch applications running in the Compute Grid environment.

    2. Compute Grid relies on an XML document submitted by the user to understand the nature and elements of the Java batch job to process. This document is called the "xJCL" because it provides the same conceptual elements JCL has for decades -- information about the job, job steps, source of input and output, substitution variables and many other piece of information18.

    3. The xJCL document is submitted to the Job Dispatching Function19, which interrogates the submitted xJCL, determines where the requested Java batch class files reside, and prepares the environment to dispatch and launch the batch program.

    4. The Job Dispatching Function accepts the xJCL through one of many programmatic interfaces -- HTTP, web services, command line, RMI and MDB. This set of interfaces makes integration of the Compute Grid function with other processes easier.

    5. The Dispatching Function determines where the Java batch program class files reside and signals to the Batch Execution Endpoint Function to execute the class file. Multiple endpoints may exist, and they may be clustered for availability or scalability.

    6. The Dispatching Function and the Endpoint Functions maintain information about jobs and the job states in a set of relational tables.

    7. The Java batch logic is developed and deployed as WAS applications deployed into the endpoint server. These applications are written as POJOs20 using Eclipse-based tooling such as IBM's Rational Application Developer.

    8. The development tooling makes use of a set of development class libraries supplied with the Compute Grid function. Or, in the case of IBM's Rational Application Developer at Version 8.0.1 or above, the tooling support is included.

    9. On z/OS, if the batch application needs to make use existing COBOL modules for its execution, it may invoke those COBOL modules by using the COBOL Container function21. This container consists of IBM-written JNI and native code that creates a separate z/OS LE environment, which operates inside the WAS z/OS server servant region address space.

    10. If the xJCL indicates the job is to be processed in parallel, then the Parallel Job Manager is called to partition the top-job into a set of sub-jobs and dispatch those sub-jobs to endpoint servers. The Parallel Job Manager monitors execution of all the sub-jobs and reports on the status of the top-job22.

    11. Finally, the WSGRID function23 is a means of integrating enterprise scheduler functions with the Compute Grid function. WSGRID is a messaging program, invoked as a job using normal JCL, that places a job submission message on a queue which is picked up by an MDB function in the Dispatcher Function. The WSGRID program stays active for the duration of the batch job within Compute Grid. Job log output from the batch job executing in Compute Grid is sent back to WSGRID in a series of messages placed on an output queue. To the enterprise scheduler function, WSGRID is the batch job, but behind WSGRID the Compute Grid function is executing the Java batch job.

    That's a somewhat complex picture, and with time you will become more familiar with it. It serves as a conceptual framework for the details that will follow.

    18 IBM Rational Application Developer 8.0.1 and later has tooling support for developing Java batch applications for Compute Grid, including an "xJCL editor" function.

    19 You will see this referred to in the Compute Grid documentation as the "Job Scheduler." We prefer "Dispatcher." The term "Scheduler" tends to confuse this function with enterprise job scheduler functions such as IBM Tivoli Workload Scheduler.

    20 Plain Old Java Objects ... in other words, they are not developed as EJBs.21 See "The COBOL Container" on page 33 for more detail on the COBOL Container function.22 See "The Parallel Job Manager function" on page 28.23 See "WSGRID and integration with existing job schedulers" on page 27.© 2012, IBM CorporationAmericas Advanced Technical Skills - 19 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Jobs, steps, the Java batch application code, and xJCLCompute Grid operates with the same general concepts as batch programs on the mainframe have for decades. If you are familiar with traditional batch processing using JCL and compiled module libraries on the mainframe, you will see many similarities in what we discuss here.

    The following picture illustrates at a high-level the key concepts of "job," "step," "application," and "xJCL."

    • Job -- this is what is submitted to Compute Grid and what Compute Grid reports on when showing status. A job may be relatively simple or more complex. If more complex it will likely consist of multiple steps ...

    • Step -- a step represents a discrete set of processing function to be executed as part of the submitted job. A simple job may consist of only one step. More complex jobs may consist of many steps. The code that executes when a step executes is held as a Java class file, which is part of a deployed Java batch application ...

    • Application -- the code that performs the job step is contained within a Java class file, and the class files that make up all the steps in a job are packaged within a WebSphere Application Server EAR24 file. WAS and Compute Grid are aware the deployed application is a batch application, and the class files inside the EAR are known to WAS and Compute Grid as well.

    • xJCL -- this XML file25 tells the story of the batch job and the steps contained within. Each step specifies the class file to run along with other declarations about how to run the batch program step. The xJCL is submitted to the Job Dispatching function, which reads and understand the xJCL and then signals to the batch endpoint (the server where the batch application is deployed) to begin execution of the job and its steps.

    24 EAR = Enterprise ARchive file ... the standard packaging and deployment file type for WAS.25 It's XML, but the general principles are very similar to regular Job Control Language (JCL).© 2012, IBM CorporationAmericas Advanced Technical Skills - 20 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Submitting Jobs to Compute GridIn traditional batch processing on z/OS, jobs are submitted using Job Entry Subsystem, or JES. JES reads the submitted JCL and from that determines what programs to run, and how to run them.Compute Grid operates in a similar way, except rather than JES being the function that initiates the job execution, it's the Job Dispatching Function of Compute Grid that does that. And rather than traditional JCL26 (with all those lines starting with // ), the job declaration is done with XML in a file called xJCL.The Job Dispatching Function runs inside an application server you designate. It understands what batch applications are deployed and where they run; it understands the Compute Grid database tables; it understands xJCL; and it accepts job submissions across several different interfaces:• HTTP -- a browser interface27 allows you to submit jobs and monitor the status of those

    jobs. This is the easiest interface to start with.• Command Line -- a shell script is provided that accepts command line arguments to

    submit and monitor jobs. The shell script opens up an HTTP connection to the Job Dispatching server and uses web services to pass in the commands you enter.

    • Web Services -- the same web services interface used by the command line shell script may be used by a custom program you write that uses web services to submit jobs.

    • RMI -- allows Java programs to interact with the Job Dispatcher function using RMI• MDB -- this is how enterprise scheduler products integrate with Compute Grid using a

    supplied program called WSGRID. For more details on this, see page 27.Job state -- submitted, executing, ended ... and more

    When you submit a job to Compute Grid it goes through three states:

    That's the normal cycle for a batch job. But the picture is actually more sophisticated than that because Compute Grid allows you to act upon excuting jobs to cancel, stop or suspend execution. So the picture of the possible states looks more like this28:

    26 Traditional JCL/JES is used to integrate Compute Grid with enterprise scheduler functions, such as IBM's Tivoli Workload Scheduler. See "WSGRID and integration with existing job schedulers" on page 27.

    27 Known as the "Job Management Console," or JMC for short.28 In reality there's even more to this. For example, when jobs go through a "pending" state when they are canceled, stopped or suspended.

    The InfoCenter has a good article on job states. Search on the string: cgrid_xdbgstate© 2012, IBM CorporationAmericas Advanced Technical Skills - 21 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    For transactional batch jobs the restartable and suspended states are entered at a defined checkpoint, and Compute Grid maintains that job's checkpoint in one of the database tables. When you restart or resume the job, Compute Grid will pick up where it left off -- starting from the last checkpoint.

    Note: What's more, if you seek highly available batch execution you may define clusters across LPARs with a shared database. In the event the original cluster member is lost, the resumed or restarted job picks up execution at the checkpoint on a surviving cluster member.

    Compute Grid provides all this supporting function so you may focus on the batch logic.Transactional batch and Compute Intensive batch jobs

    The Compute Grid function supports two kinds of Java batch jobs:• Transactional -- this is what most think of when they think of batch ... the looping

    through a set of records with processing performed on each record until the last record is processed.The Compute Grid function provides a rich set of functions to the transactional batch jobs that run. For instance, managing checkpoint intervals is a function of the batch container provided by Compute Grid. Record input and output functions make handing input and output easier. For more, see "Functional services provided by Compute Grid" on page 23.

    • Compute Intensive -- the name implies a batch job that is launched and then performs a series of internal computations with relatively little I/O. An example of this would be numerical analysis of geological seismic information. One use of the Compute Intensive model is as a means of deploying into Compute Grid "Java main" applications. Rather than refactor those Java programs to the Compute Grid framework, you may simply wrapper them as Compute Intensive batch applications and deploy them as is.

    This distinction between Transactional and Compute Intensive batch applications is offered here because (a) the terminology comes up in other Compute Grid documentation, and (b)

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 22 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    when developing Compute Grid applications certain function services are available to transactional batch but not to compute intensive29.

    Key Point: As you think about how you might use Compute Grid for your Java batch programs, keep in mind these two categories of jobs. If your program is a traditional loop process then it's mostly likely a transactional batch model. Otherwise, consider mapping it to the compute intensive model.

    Functional services provided by Compute GridThe primary purpose of the Compute Grid function is to provide a set of Java batch execution functions to the batch applications that run and to the administrators responsible for batch processing.The objective is to enhance the productivity of batch developers and batch administrators. The functional services are provided in IBM-supported middleware. That makes coding the batch applications simpler, and it makes administering the environment easier. The following picture provides a way to organize these functions into three categories:

    Application Support FunctionsDevelopment class libraries and tooling support

    Compute Grid comes with a set of development class libraries that facilitate the coding of Java batch applications to run in Compute Grid. If you have IBM Rational Application Developer 8.0.1 or later the tool has an optional feature for Java batch, including the development class libraries and artifact creation wizards30.

    xJCL, substitution properties and external job and step declarationsJob and step properties do not need to be coded within the application. They may be specified in the xJCL job control file as properties, and those properties are available to the Java batch application during execution.

    Batch Data Stream FrameworkThe Batch Data Stream (BDS) framework helps abstract data input and output, making development of the I/O function of Java batch jobs much easier. xJCL properties define to the BDS the input and output streams for a job step. BDS iterates through the result set, invoking your job step batch logic for each record.

    29 For example, the "Skip-Record Processing" function (see page 52) is for transactional batch only.30 Including a wizard to assist with the creation of xJCL.© 2012, IBM CorporationAmericas Advanced Technical Skills - 23 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    BDS comes with a set of packaged patterns for common read and write operations. The following table summarizes the current patterns available:

    Data Read Data Write

    JDBC JDBCReaderPattern JDBCWriterPattern

    JPA JPAReaderPattern JPAWriterPattern

    File Byte Data ByteReaderPattern ByteWriterPattern

    File Text Data TextReaderPattern TextWriterPattern

    z/OS Data Set RecordOrientedDatasetReaderPattern RecordOrientedDatasetWriterPattern

    Java batch developers do not need to write their own Java stream handlers for these popular patterns. Compute Grid provides them in the form of Batch Data Stream patterns. Application code simply uses the methods of the pattern to read and write the data as needed.

    Checkpoint algorithmTransaction batch processes that update data records must specify when data udpates are to be committed. It's a balance between performance (fewer commits) and recoverability (more commits). Compute Grid provides an interface and algorithm so batch applications do not need to do this checkpoint processing themselves. Properties in the xJCL for a Java batch job specify the checkpoint algorithm -- either number of records or amount of time that has passed -- and Compute Grid (along with WAS under it) manages the commit and rollbacks as needed.If a batch job is stopped or paused, Compute Grid will maintain the checkpoint state information in its relational table. When the job is restarted or resumed the processing will pick up at the last committed checkpoint.This kind of checkpoint processing could be written within each Java batch program, but that would not be efficient. Better to make the checkpoint processing part of the Java batch execution platform and allow applications to simply request the service.

    Configurable transaction modeIn earlier versions of Compute Grid, the transaction mode was fixed at "global." With Compute Grid Version 8 (as well as the Compute Grid function that is part of WAS V8.5) this is configurable by step as either "local" or "global."Where applicable, local transaction mode improves performance since no global transactions are started. The connection object is shared between the batch container and the application code.This is one of the items in the Technology Focus section. See "Configurable transaction mode" on page 64.

    Results algorithmThis provides an interface for the setting of a step's return code. This information is maintained in a "JobStepContext" object which is available for the life of the job.Step return codes may be used to influence the running of subsequent steps in the batch job. Condition declarations in the xJCL will determine, based on previous step return codes, whether a step will run or not.The results algorithm also allows developers to set the overall job return code based on the return codes of all the steps. When using the WSGRID mechanism to integrate enterprise schedulers with Compute Grid, a job's overall return code is what gets fed back to the enterprise scheduler. That's important because the

    © 2012, IBM CorporationAmericas Advanced Technical Skills - 24 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    enterprise scheduler often determines subsequent processing based on return code it sees for a processed job. With the results algorithm the developer has control over step and overall job return codes that are produced.

    Skip record processingTransactional batch applications31 occasionally encounter exceptions when reading and writing records. Unless those errors are handled, the processing will be rolled back to the last checkpoint and the job thrown into a restartable state. For the occasional error it makes better sense to provide a way to skip the record, log the skip, and continue on to the next record.This type of function could be custom-coded in the application. But that would not be consistent with the objective of container managed services for applications. So a better place to provide this would be a built-in function of the batch execution framework. That is what the Compute Grid skip record processing is -- a mechanism built into the Batch Data Stream Framework to handle record skipping when exceptions are seen.The skip record processing feature allows you to define how many such read or write errors may occur, and what types of exceptions the batch container should include or exclude. In addition, it provides a way to register your own "skip listener" code when a record is skipped. That would be useful in logging information about the skipped record so auditors may review and take corrective action on that record.

    The properties that control this behavior -- the skip count and the exceptions to include or exclude -- are coded in the xJCL for the Batch Data Stream.

    This is one of the items in the Technology Focus section. See "Skip-record processing" on page 52 for more.

    Retry step processingThe execution of a transactional batch job32 step may occasionally encounter errors. At that point there are two courses of action -- set the return code and move on to the next step33, or retry the job step.This function provides a mechanism to specify how many retries to attempt, and what types of exceptions to include or exclude for job step retries. The properties that control this behavior are coded in the xJCL for the job step.This is one of the items in the Technology Focus section. See "Retry-step processing" on page 57 for more.

    Job ListenerThe Job Listener is a function of the container that enables developers to invoke custom setup or cleanup code at the beginning and end of the job and the steps within the job. This applies to both transactional batch and compute intensive batch jobs.This is one of the items in the Technology Focus section. See "Job listener" on page 50 for more.

    31 This function applies to transaction batch only, and not compute intensive batch jobs.32 This function applies to transaction batch only, and not compute intensive batch jobs.33 And based on conditional processing, the next step may not run based on the return code of the previous step, which would mean the

    whole job would cease processing.© 2012, IBM CorporationAmericas Advanced Technical Skills - 25 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    Administrator Support FunctionsJob Dispatcher (and its role in overall Compute Grid)

    The Job Dispatcher function34 is the primary interface into the Compute Grid function. Through it administrators submit jobs, review the results, control the jobs (start, stop, suspend, restart, resume), and purge jobs.The Dispatcher understands the whole Compute Grid environment. It controls the dispatching of submitted jobs to batch endpoints where the batch application code resides.

    Key Point: The Job Dispatcher function is integral to the overall functioning of Compute Grid. It makes available many administrator functions that would not otherwise be available in less powerful Java batch approaches such as JVM launchers. A custom solution to do everything the Dispatcher Function does would involve a great deal of custom code that would then require custom support. Hence the value of the Job Dispatcher in the overall Compute Grid scheme.

    Security: authentication and user rolesWhen security is enabled for the WAS runtime on which the Compute Grid function runs, then access to the "Job Management Console" (the dispatcher function, or "JMC") is controlled35. On z/OS SAF may be used as the repository for userids and passwords as well as the EJBROLE definitions that allow access.Further, there are three classes of roles that define the degree of authority the ID has once in the Job Management Console. They are:

    administrator Full authority -- submit jobs, control jobs, view and download any job logs.submitter Partial authority -- may submit jobs, but may only control jobs that ID

    submitted. Able to download any job log.

    monitor Limited authority -- unable to submit jobs, but may view jobs and download job logs for any job.

    Integration with enterprise scheduler toolsEnteprise scheduler tools (such as IBM Tivoli Workload Scheduler) are critical components in many customer batch processes. Those tools may integrate with Compute Grid so Java batch may be part of larger scheduled batch processes. How this integration is accomplished is covered under "WSGRID and integration withexisting job schedulers" starting on page 27.

    z/OS Platform ExploitationSMF job usage recording

    The Compute Grid function on z/OS has the capability to write SMF records that record the Java activity processed in Compute Grid36. The InfoCenter article found by keyword search cgrid_zosjobusage has details.There are two record types available -- 120.9 and 120.2037. The 120.9 record contains a superset of information contained in the 120.20 record. Further, the 120.9 records are dynamically started and stopped with the MODIFY command. We recommend those interested in job accounting information for Compute Grid focus on the 120.9 record. Consult the InfoCenter article for more details.

    34 Also known as the "Job Scheduler," but again, that name is often confused with enterprise schedulers. So we prefer "dispatcher."35 Access is controlled for other interfaces as well, such as the command line or web services. JMC is the browser interface.36 Compute Grid on all platforms has the ability to write job usage information to a database. On z/OS the capability to write to SMF was

    added because SMF is a key part of many customer usage, reporting and analysis processes.37 The story behind why there are two records is somewhat involved and not really central to this document.© 2012, IBM CorporationAmericas Advanced Technical Skills - 26 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    WLM classificationWLM is a z/OS facility for the classification of work and the management of resources to complete work based on defined performance goals. WAS z/OS has the ability to identify work by request and have that work classified by WLM.This ability to identify and classify work per request allows WAS z/OS to differentiate work running in the same server. From that WAS z/OS is able to segment work into separate servant regions so WLM may better manage resources to goals38.Compute Grid on WAS z/OS has the ability to utilize this capability and have WLM classify batch jobs per request. That means a server running Java batch jobs and defined with multiple servants has the ability to run jobs in separate servant regions, with WLM managing system resources to the goals of each.

    WSGRID and integration with existing job schedulersThe use of enterprise job schedulers39 to manage batch execution processes is common in the business world. Therefore, it's important that some means be available of integrating Java batch jobs in Compute Grid with these enterprise job schedulers. For Compute Grid that integration is provided with a function called WSGRID.The following picture illustrates the key points of this WSGRID integration mechanism:

    1. WSGRID is a program40 supplied with Compute Grid that puts a job submission message on a queue, which is then picked up by the Compute Grid dispatcher function. WSGRID is submitted using normal JCL, so to the Enterprise Scheduler Function it looks like any other submitted batch job. Here's the key -- the WSGRID program stays up for the life of the Java batch job that executes in Compute Grid. Why that's important becomes evident a bit later as we discuss how the job output flows back to JES spool.

    2. As mentioned, the Enterprise Scheduler Function submits JCL to begin the execution of the WSGRID program. The SYSIN DD of the JCL has the information for Compute Grid Java batch job41 to be submitted (including the xJCL to use), as well as information about the input and output queue to use when communicating with Compute Grid.

    38 The document at ibm.com/support/techdocs under search argument WP101740 is the definitive document on how WAS z/OS interacts with WLM for the purposes of work classification. It's a somewhat complex topic. If you have some knowledge of WLM and wish to understand better how WAS z/OS works with WLM, that document is the key.

    39 For example: IBM Tivoli Workload Scheduler, CA Workload Automation CA 7, or BMC Control-M.40 WSGRID comes in two forms -- native z/OS and Java. On z/OS JCL is used to submit either form. For Compute Grid on the other

    platforms the submission of WSGRID is done using shell scripts or BAT files.41 Use separate JCL for each Compute Grid job you wish to submit, each JCL containing the specifics about the xJCL to submit.© 2012, IBM CorporationAmericas Advanced Technical Skills - 27 -

    WP101783 at ibm.com/support/techdocsVersion Date: Monday, November 19, 2012

  • WP101783 - IBM Compute Grid and WebSphere Java Batch

    3. The WSGRID program places the Java batch job submission message on a queue, either MQ or the SIBus function of WebSphere Application Server. The Compute Grid Job Dispatching Function