Info Sphere CDC - Real Time Data Integration UC 17-09-08

8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08

1/18

2008 IBM Corporation1

CDC Transformation and Delivery

Data at the speed of business


2/18

2

What is CDC

Change Data Capture

Capture data events in source database and move onlythe changes to the target

Many different ways of doing CDC

Timestamps

Triggers

API

Log-based


3/18

3

What fuels the IBM CDC Roadmap? The widest breadth of functionality:

Batch/pull and real-time push processing

Guaranteed delivery/transactional integrity

Multiple topologies (peer to peer, 1 to many, many to 1, uni-directional, bi-directional)

Homogeneous & heterogeneous data synchronization

Broadest range of sources and targets

Log-based capture agents for DB2 (on all platforms), Oracle, SQL Server, Sybase,IMS, VSAM, IDMS, ADABAS

Native/parallel applies for all RDBMS and JMS

Multiple data delivery protocols (TCP/IP, JMS)

Industry leading performance and scalability

End to end throughput and low latency

Parallel Apply to target system

Low impact on source database systems


4/18

4

What fuels the IBM CDC Roadmap .

3000+ customers using the existing CDC products for;

HA/DR (DB back-up, fault tolerance)

Real-time reporting/off-load querying

Application Co-existence (migrations, upgrades, modernization)

eCommerce (web apps, portals, data distribution)

Dynamic Data Warehousing, Master Data Management

700+ people in engineering focused on Information Integration including 170+focused on CDC technologies

The most comprehensive suite of data integration products

BoB transform / cleanse / discovery, metadata management, scalableperformance, services enabled for SOA architectures

5000+ customers using Information Server components


5/18

5

The IBM Solution: IBM Information ServerDelivering information you can trust

IBM Information Server

Discover, model, andgovern information

structure and content

Standardize, merge,and correct information

Combine andrestructure information

for new uses

Capture, virtualize andmove information for in-

line delivery

Unified Deployment

Unified Metadata Management


6/18

6

InfoSphere CDC Solution

Provides real-time change datacapture and delivery for

Dynamic warehousing and real-timereporting

Synchronization and replication Event detection

Minimal impact on productionsystems

High scalability and end-to-endperformance

Guaranteed data integrity

Proven Heterogeneous support

Delivers real time changed data toInformation Server, applications and

targets or message queues

ArchitectsDevelopers

DataMirror

Without impacting performance ofproduction systems


7/18


8/188

Architecture

DatabasesOracle, DB2, DB2 UDB, SQL Server, Sybase, Teradata, Netezza, PointBase

IMS, VSAM, IDMS, Adabas, DataCom - Classic

Platformsz/OS, System i5, Red Hat and SUSE Linux, AIX, HP/UX (PA-RISC and Itanium), Solaris SPARC,Tru64 UNIX, Windows

Messaging Middleware

MQSeries, Sun Open Message Queue (JMS), TIBCO, BEA AquaLogic, Oracle Fusion Middleware

Journal LogRedo/Archive Logs

Source EngineAnd Metadata

Target EngineAnd Metadata

TCP/IP

Java-based GUIfor admin & monitoring

Database

ODS

JMS

BusinessProcess

Publisher Subscriber

Flat files

Audit

Direct to existing ETL


9/18 2008 IBM Corporation9

Customer examples

Use Cases


10/1810

1. Building A Low Latency ODS for Operational Reporting and Auditing

ODSManufacturing

ERP

OLTP

Native

Log

DB

Manufacturing

Finance

OLTP

Native

Log

DB

Production Server

Production Server

All OLTP insert, update and delete operations canbe stored as inserts to maintain completetransaction history.

Add relevant information such as timestamp,transaction type, source system id, and id of user

who changed the transaction.

Operational Data Store

Solution deployed to improve visibility into lines of business for organizations withOperational BI and Data Auditing requirements

Each OLTP insert, update and delete operationcan be stored as an insert, update and delete tomaintain synchronized copy of data.


11/1811

2. Complementing An Existing ETL Technology

Continuous

Retail

Point Of Sale

OLTP

Native

Log

DB

EDWStageETL

ETL ServerProduction Server

Scheduled Batch

2. Business Objects Data Integrator

1. Informatica Power Center

3. Ab Initio

4. IBM DataStage (has native integration)

Complementary ETL Technologies:

Data Warehouse

2. Flat File

1. Relational Table

3. Message Queue

4. Direct to ETL

Stage can be:

Solution deployed to improve visibility into lines of business (i.e. Dynamic Warehousing)

and help manage impact concerns caused by ETL on mission critical systems


12/1812

3. Continuous Feed Of A Business Intelligence Appliance

Manufacturing

ERP

OLTPNative

Log

DB

CDC

Stage

Flat File Appliance

Continuous (to Appliance)

CDC

Staging ServerProduction Server Appliance Nodes/Cluster

Flat file containing transaction changes viewed asan external file to the appliance.

Load threshold based on # of Transactions or timeinterval.

Once threshold reached, call appliance load API

to bulk load transactions into appliance.

Solution deployed to improve visibility into lines of business by combining the

cost/performance benefits of a BI Appliance with real-time data feeds.

Appliance

Load API

2. Netezza

1. Teradata

Supported Appliances

3. GreenPlum

4. Paraccel

5. IBM Balanced Warehouse


13/1813

Data Event Synchronization via an Enterprise Service Bus

CDC/Replication License

CDC/Replication Process

Other Technology

E

S

B

Queue 1Queue 1

Billing

Continuous

CDC

OLTP

Telco

Native

Log

DB

Production Server

Telco

OLTP

CRM

Production Server

2. TIBCO Business Works

1. IBM MQ Series

3. BEA Aqualogic

4. WebMethods Fabric

Complimentary ESB Technologies:

Solution deployed to provide real time data feeds for SOA and application

integration business requirements.

A license would reside on the server thathosts the message oriented middleware.

ContinuousCDC

ETL

Solution deployed to provide real time data feeds for SOA and applicationintegration business requirements.


14/1814

5. e-Commerce Application Synchronization

Retail

Inventory

OLTP

Native

Log

DB

Native

Log

DB

Corporate

Native

LogDB

Downtown Store

OLTP

OLTP

Production Server

Website Orders

Point Of Sale

Solution deployed to provide continuous customer, sales and inventory visibilityin web base e-commerce applications.

Provides continuous bi-directional synchronizationbetween web based applications and missioncritical business applications.

Helps organizations improve customer online shoppingexperience with improved visibility into inventory and customershopping activities.


15/1815

6. Data Synchronization for Upgrades, Migrations and WorkloadBalancing

Manufacturing

OLTP

Native

Log

DB OLTPNative

Log

DB

ERP

Upgrades, Migrations

ERP

Manufacturing

Production Server Testing Server

Workload Balancing

Solution deployed to help IT support application, database and platform migrations.

Keep data synchronized between currentproduction server and a server deployed to test anew application upgrade/version, or ahardware/OS upgrade.

Workload balancing capability (i.e. master to master support)allows database instances to remain synchronized where dual ordouble data entry is a requirement (i.e. data entry occurring onboth systems at the same time).


16/18

16

7. Offloading Production Query & Reporting Cycles

Services

OLTP

Report

Services

Finance 3

OLTP

Native

Log

DB

Native

Log

DB

Finance 1

Services

Finance 2

OLTP

Table Copy

Query

Production Server(s)

Reporting Server

Reporting server can also be used forconsolidation requirements i.e. consolidatingfinancials from multiple branches into a singlecorporate instance.

Solution deployed to allow organizations to offload the impact of query andreporting to a non mission critical system.

Replication frequency generally varies fromcontinuous (near real-time) to periodic. Table levelrefresh or copy can be used in addition to logbased change data capture.


17/18

17

8. Data Backup And Availability

Finance

Backup

Backup

Native

Log

DB

Partition 1

Partition 2

Continuous (to backup instance)

CDC

OLTP

Production ServerAvailabilityServer

Availability of data only, does not support DDLreplication.

Exact image replication to produce a backup copyon a separate server or in a different partition onthe same server.

A separate license is not required for each partitionused on the production server.

Solution deployed to allow organizations to backup copies of critical data for

recovery where a full disaster solution is not a requirement.


18/18

2008 IBM Corporation18

Thank You

Info Sphere CDC - Real Time Data Integration UC 17-09-08

Documents

Transcript of Info Sphere CDC - Real Time Data Integration UC 17-09-08