IMS Replication with InfoSphere

Replicating IMS Data with InfoSphere Classic Replication

Gregory Meimers NA IIDR Specialist

IMS Replication

InfoSphere IMS Replication for z/OS delivers a native IMS-to-IMS software replication solution that supports high-availability IMS data environments. This solution synchronizes the contents of IMS databases on a single site, or across geographically dispersed locations, in

near real time with full recovery.

3

Heterogeneous:Synchronize Operational Data Stores with Reporting data bases, Information Server (ETL/Quality), MDM, Data Warehouses, and Datamarts for Business Intelligence, Analytics, Data Consolidation & Migrations

Homogeneous:Synchronize data between two or more data centers for Business Continuity & Disaster Recovery & Query Offload

Heterogeneous:Synchronize data across multiple systems.Data distribution and synchronization for Billing, Inventory, Financials, Customer data, etc.

The Role of Data ReplicationHomogeneous and Heterogeneous Data Synchronization

Changes

Changes

Changes

Consider the New Ways to Use Copied DataMobile Applications, Business Analytics and Big Datawill all drive dramatic increases in query workloads

A dedicated query data environment maintained by data replication

is a quick and easy way to handle the growth!

− Offload the query workload to a real time copy of the data on a secondary system

� No new programming

� No visible impact on users

� No impact on the existing transaction environment

− “Update” workload continues on the primary system

� Optimizes update environment

− Using dual sites also introduces continuous availability benefits !

� Secondary environment ready and able to carry the load during both planned and unplanned outages

5

legacy

apps

dbs

xls, xml, flat

warehouse

z/OS

custom

Reducing Cost (Extracting, Moving & Loading Data)• CPU usage/cost/time for extracting batch data?• Is more hardware required to support growing batch extraction process?• What are the development costs for supporting custom batch extraction?• What is the impact of intraday queries on performance / processing costs?

Extending Application Availability (Batch Windows)• Are business applications unavailable during extract or load processes? • How many hours are these applications unavailable?• What is their data volume growth rate? If batch is not a problem today, will

it be tomorrow?

+ Revenue / - Risk (Decisions Based on Current Data)• How is revenue generation impacted by lack of timely access to data (cross-

sell, up-sell, service)?• What are the risks of making important decisions based on outdated or

unavailable data?

The Business Value

legacy

apps

dbs

xls, xml, flat

warehouse

z/OS

custom

Reducing Cost (Extracting, Moving & Loading Data)• CPU usage/cost/time for extracting batch data?• Is more hardware required to support growing batch extraction process?• What are the development costs for supporting custom batch extraction?• What is the impact of intraday queries on performance / processing costs?

Extending Application Availability (Batch Windows)• Are business applications unavailable during extract or load processes? • How many hours are these applications unavailable?• What is their data volume growth rate? If batch is not a problem today, will

it be tomorrow?

+ Revenue / - Risk (Decisions Based on Current Data)• How is revenue generation impacted by lack of timely access to data (cross-

sell, up-sell, service)?• What are the risks of making important decisions based on outdated or

unavailable data?

DBMS

ETL

Message Queue

SOA

Flat files

Log

SourceData

ApplyCapture Push

The IBM Data Replication ModelEfficient Capture & Delivery of Changes to Enterprise Data

� Log Based Captures independent of source applications

− Minimizes impact on source platform resources

− Easy to restart and recover via spill or archive logs

� Native Apply APIs to optimize throughput

− Efficient I/O to targets

� Broadest Source, Target, OS and Platform support

− No other vendor comes close to IBM’s coverage

6

B. InfoSphere Data ReplicationExpansive support that is continuously expanding

SOURCES TARGETS O/S PLATFORM

DB2 (z/OS, i, LUW) All Sources §§§§ z/OS System z

Informix Pure Data for Analytics Red Hat / SuSE System z

Oracle Information Server AIX System p

MS SQL Server Cognos Now! IBM i OS System i

Sybase Solid DB Red Hat / SuSE Intel / AMD

IMS Teradata MS Windows Intel / AMD

MQ Series / JMS HP-UX HP- Itanium

WebMethods / BEA / TIBCO HP-UX HP PA-RISC

MY SQL / Greenplum ÷÷÷÷ Solaris Sun Sparc

§§§§IMS Target ONLY for an IMS SourceVSAM Target ONLY for a VSAM Source

÷÷÷÷ Customized solution, some restrictions

IMS Replication

B. Heterogeneous IMS to non-IMS replication*** (when used with InfoSphere Data Replication’s CDC Target Engines)

A. High speed, low latency IMS to IMS data

replication spanning unlimited distances

InfoSphere Data Replication for IMS for z/OSTwo models in one product

IMSdatabases

InfoSphereIMS Replication

IMSdatabases

InfoSphere IMS Replication

Messagequeues

ETL

e.g. DataStage Applications

IMSlogs

IMS databases

IBM InfoSphere Data Replication (CDC)

databases

*** Also offered stand-alone, without the IMS to IMS capability, as IBM InfoSphere Classic Change Data Capture for z/OS

Subscriptions

� Subscription

A programming object consisting of multiple replication mappings that identify the source tables that you want to replicate as a consistent group to the target tables.

Subscription methods:

Continuous

Net change

Refresh

Persistent subscriptions start replicating automatically:

When you start the data server

After transient failures, such as communication outages

The data server disables persistency:

When you explicitly stop replication

After a non-recoverable error

A. High speed, low latency unidirectional IMS to IMS data replication

spanning unlimited distances

� Support for updates made via DB/TM, DBCTL and Batch DL/I programs

� Requires DBRC to “chase” logs for scheduled & recovery modes

� Replication monitoring is built in

� Independent initial load of target DB is required

� Integration with Tivoli for GDPS Active-Active Continuous Availability


IMSdatabases

InfoSphereIMS Replication

IMSdatabases

IMS CCA – DBD Example

DBD NAME=DI21PART,ACCESS=(HISAM,VSAM)

DATASET DD1=DI21PART,DEVICE=3380,OVFLW=DI21PARO,

SIZE=(2048,2048),RECORD=(678,678)

SEGM NAME=PARTROOT,PARENT=0,BYTES=50,FREQ=250, x

EXIT=(*,NOKEY,DATA,NOPATH,(NOCASCADE),LOG)

FIELD NAME=(PARTKEY,SEQ),TYPE=C,BYTES=17,START=1

SEGM NAME=STANINFO,PARENT=PARTROOT,BYTES=85,FREQ=1

FIELD NAME=(STANKEY,SEQ),TYPE=C,BYTES=2,START=1

SEGM NAME=STOKSTAT,PARENT=PARTROOT,BYTES=160,FREQ=2, x

EXIT=(*,KEY,DATA,NOPATH,(CASCADE,KEY),LOG)

FIELD NAME=(STOCKEY,SEQ),TYPE=C,BYTES=16,START=1

SEGM NAME=CYCCOUNT,PARENT=STOKSTAT,BYTES=25,FREQ=1

FIELD NAME=(CYCLKEY,SEQ),TYPE=C,BYTES=2,START=1

SEGM NAME=BACKORDR,PARENT=STOKSTAT,BYTES=75,FREQ=0

FIELD NAME=(BACKKEY,SEQ),TYPE=C,BYTES=10,START=1

DBDGEN

FINISH

END

TARGET SERVER

IMS

Target IMS DBsReplication

Metadata ACBLIB

BookmarkDB

Admin. Services

IMS DRA Interface

Unitof

Recovery Analysis

Apply

SOURCE SERVER

ReplicationMetadata

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

ClassicData

Architect

TCP/IP

A. IMS to IMS Data Replication

Read source IMS logs - Send committed changes - Apply changes in parallel


TARGET SERVER

IMS


Metadata ACBLIB

BookmarkDB

Admin. Services

IMS DRA Interface

Unitof

Recovery Analysis

Apply

SOURCE SERVER

ReplicationMetadata

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

ClassicData

Architect

TCP/IP

14

Capture– Capture Engine reads source IMS log records

– Records from distinct logs are merged and properly sequenced

– Committed changes are pushed based on unit-of-recovery criteria

– All changes for a unit-of-work are sent together

Capture Services

A. IMS to IMS Data ReplicationDetails of IMS Source Capture

ChangeStreamOrdering

SOURCE SERVER

ReplicationMetadata

RECON

Batch Start-StopExit Routine

BATCHDL/I

IMSTM / DB*

Start NotificationExit Routine

Log Info

IMS Databases

TCP/IP NotificationTCP/IP Notification

* includes BMP and DBCTL

TCP/IP NotificationTCP/IP Notification

Log ReaderService

IMSLogs

DBRCAPI

WHAT: Classic routine associated with IMS’s Partner Program Exit.

ACTION: Notify Classic Server when an IMS system starts.

WHAT: Classic routine associated with IMS’s Partner Program Exit.

ACTION: Notify Classic Server when a Batch DL/I job starts or stops.

Push– TCP/IP Conversations with the Target Engine

• Two threads per conversation

• One “control” conversation for source engine

• One “data” conversation per subscription


TARGET SERVER

IMS


Metadata ACBLIB

BookmarkDB

Admin. Services

IMS DRA Interface

Unitof

Recovery Analysis

Apply

SOURCE SERVER

ReplicationMetadata

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

ClassicData

Architect

16

TCP/IP

Apply– Receive committed transactions

• Serialization based on resources updated by unit of recovery

– Apply change to the target in parallel when possible

– Replication Bookmark Database required to manage recovery & restarts


TARGET SERVER

IMS


Metadata ACBLIB

BookmarkDB

Admin. Services

IMS DRA Interface

Unitof

Recovery Analysis

Apply

SOURCE SERVER

ReplicationMetadata

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

ClassicData

Architect

TCP/IP

17

A. IMS to IMS Data ReplicationTarget Engine Details

WriterServices

TARGET SERVER

StagedUnit-of-Recovery

Data

IMS

DRAthread

Dependency Analysis

WriterServices

ApplyService

CHANGEMessages

CHANGE

Messages

WHAT: Patented Dependency Analysis

ACTION: Parallelizes writes to the target when possible to increase endto end throughput.

A. IMS to IMS Data ReplicationSample Replication Performance

� Source System

− 8-way IMSPLEX

− Peak OLTP workload: 2,200 – 2,600 transactions/second

− Peak Batch workload: 100,000 – 120,000 updates/second

� Target System

− 2-way IMSPLEX

− OLTP Latency: Sub-second end-to-end

− Batch Latency: Maximum 10 minute delay during peak processing only


B. Heterogeneous IMS to non-IMS replication � Continuous mirroring: Apply changes at target as made at source

� Refresh: Apply a snapshot version of source system

− Shares IMS to IMS Capture Engine but :

� Requires “Classic” IMS meta data to map to a relational target model

− Requires InfoSphere Data Replication’s CDC Target engine for:

1. Broad targeting and target engine transformations

2. Map source to target replication subscriptions

3. View and report on data flow characteristics

InfoSphere IMS Replication

Messagequeues

ETL

e.g. DataStage Applications

IMSlogs

IMS databases

IBM InfoSphere Data Replication (CDC)

databases

AccessServer

TARGET

Target Engine

Comm Layer

Admin APIMetaData

Admin Agent

Apply Agent

SOURCE SERVER

ReplicationMetadata

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

B. IMS to Non-IMS Data Replication

ClassicServer

TCP/IP

IBM InfoSphere Data Replication(CDC)

IBM InfoSphere Data Replication for IMS(Classic CDC)

ClassicData

ArchitectManagementConsole

AccessServer

ManagementConsole

B. IMS to Non-IMS Data ReplicationMetadata Management, Configuration and Monitoring

ReplicationMetadata

ClassicData

Architect

IIDR

– Define and Manage Subscriptions:

• Main configuration object for replication

• Link one or more Classic “tables” or “views” to targets

• Define source to target communication connection

• Start and stop data movement

Classic

‒ Map IMS segments to logical tables using imported copybooks & DBDs

• Automates translation of legacy data types

• Handles legacy constructs like recurring data, redefines, variable lengths, etc.

• Metadata-driven WHERE filtering clauses

– Configure/Manage Classic address spacee.g. Tracelevel, heartbeat interval, logging, K

AccessServer

B. IMS to Non-IMS Data ReplicationSource IMS Capture with “Classic” Data Reformatting

SOURCE SERVER

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

ClassicServer Capture

– IMS Replication Source Server with additional components for:

• Transforming captured IMS data into the relational table and view model based on the metadata mapping

ReplicationMetadata


ClassicData


24

AccessServer

B. IMS to Non-IMS Data ReplicationLeverage InfoSphere Data Replication’s Targeting

SOURCE SERVER

SourceIMSDBs

IMS

DBRC API

ACBLIB

Admin. Services

IMS

Log Read/Merge

Unitof

Recovery Capture

RECON

IMSLogs

ClassicServer

ReplicationMetadata

TCP/IP

TARGET

Target Engine

Comm Layer

Admin APIMetaData

Admin Agent

Apply Agent

ApplyReceive committed transactions

• Apply any target transformations • Apply changes to the target

IBM InfoSphere Data Replication(CDC)


ClassicData


The Value of Data ReplicationReduced down time, MIPS savings, Faster solution delivery

� Optimize resource utilization by eliminating massive batch movements

− Lower costs by saving CPU and network resources

� Shorten batch windows by streaming changes as they happen

− Extend application availability

� Reduce data latency with “right time” updating

− Improve the bottom line with accurate information when and where needed

Leading petroleum refiner Cuts batch window by 60%

Major Chinese bank Shortens weekly planned outage by 90+%

Insurance provider Saves $100K / month

Large U.S. Telco $500K savings by eliminating home-grown

IMS Replication with InfoSphere

Technology

Transcript of IMS Replication with InfoSphere