JCC LogMiner Loader: Today and Beyond - download.oracle.com · 14/11/2006 · Replicate data from...

11/14/06Copyright 2004 JCC Consulting, Inc. All

rights reserved 1

JCC LogMinerJCC LogMinerLoader: Today andLoader: Today and

BeyondBeyondJeffrey S. Jalbert Jeffrey S. Jalbert __ Thomas H. Musson Thomas H. Musson __ Cheryl P. Cheryl P.

JalbertJalbert

JCC Consulting, Inc.JCC Consulting, Inc.

11/14/06 2Copyright 2006, JCC Consulting, Inc.

All rights reserved

Presentation AgendaPresentation Agenda

Basics of the Loader, Basics of the Loader, brieflybriefly Clever Customer ConfigurationsClever Customer Configurations Recent releasesRecent releases FuturesFutures Your questionsYour questions


All rights reserved

JCC LogMiner LoaderJCC LogMiner Loader The Loader publishes database changes to aThe Loader publishes database changes to a

target or targets.target or targets. The Loader is used in mission-criticalThe Loader is used in mission-critical

architectures.architectures. The source database is, generally, a productionThe source database is, generally, a production

database that must not experience a significantdatabase that must not experience a significantimpact.impact.

Targets can diverge significantly from theTargets can diverge significantly from thesource.source.

Some applications are extremely intensive,Some applications are extremely intensive,involving millions of customers or thousands ofinvolving millions of customers or thousands oftransactions per second.transactions per second.


All rights reserved

Sources and TargetsSources and Targets SourceSource

Oracle Rdb (any version that supports the LogMiner)Oracle Rdb (any version that supports the LogMiner) TargetTarget

Oracle Rdb Oracle Rdb (any version that supports multi-statement procedures)(any version that supports multi-statement procedures) Oracle (requires Oracle SQL*net on the system running theOracle (requires Oracle SQL*net on the system running the

Loader)Loader) 10g10g 9.2 (linked with 9.2.0.4)9.2 (linked with 9.2.0.4) 9.09.0 8.1.78.1.7

TuxedoTuxedo XML (to your own API)XML (to your own API) FileFile JDBC Class 4 driver (Sun lists 141 platforms with a Class 4JDBC Class 4 driver (Sun lists 141 platforms with a Class 4

driver.)driver.)


All rights reserved

Sources and Targets (cont.)Sources and Targets (cont.) The source and target can be different.The source and target can be different.

Format of the target can be completely different than the formatFormat of the target can be completely different than the formatof the source database.of the source database.

LogicallyLogically PhysicallyPhysically

Tuning of the target can be different.Tuning of the target can be different. IndicesIndices PlacementPlacement BufferingBuffering CachingCaching


All rights reserved

Source to Target, PlusSource to Target, Plus Static or Continuous (Static or Continuous (““near near realtimerealtime””)) Low impact on Mission Critical systemsLow impact on Mission Critical systems Resilient and resistant to environmental issuesResilient and resistant to environmental issues RestartableRestartable, no data loss, no data loss Materialized information, such as commit timeMaterialized information, such as commit time Single or multi-threaded, fastSingle or multi-threaded, fast TunableTunable Extensively equipped for monitoring and performanceExtensively equipped for monitoring and performance

analysisanalysis Data PumpData Pump

Initial populationInitial population Correction of downstream difficultiesCorrection of downstream difficulties

Data transforms, Filtering, Data mappingData transforms, Filtering, Data mapping


All rights reserved

How The Loader Does ItHow The Loader Does It

UpdateProcesses

RMUCLM

JCC CLMLControlProcess

CLML

RMUBackup

Target

JCCLML

ActiveAIJ

BackupAIJs

Source RdbDatabase

RestartContext

ControlFile


All rights reserved

Getting StartedGetting Started Keeping it simpleKeeping it simple

The JCC LogMiner Loader may be fairly easily established for simpleThe JCC LogMiner Loader may be fairly easily established for simplereplication.replication.

The defaults for many of the keywords and other choices will be usefulThe defaults for many of the keywords and other choices will be usefulfor many applications.for many applications.

The most important things to know are that the Loader cannot processThe most important things to know are that the Loader cannot processAIJ files that have been removed from the system and cannot find rowsAIJ files that have been removed from the system and cannot find rowsin the target (to update) without a key.in the target (to update) without a key.

Adding SanityAdding Sanity The Loader has extensive aids to the DBA and extensive monitoring.The Loader has extensive aids to the DBA and extensive monitoring. New features support testing your application in ways that have seldomNew features support testing your application in ways that have seldom

been possible before.been possible before. Making it complexMaking it complex

The Loader is often used in very complex architectures demanding aThe Loader is often used in very complex architectures demanding agreat many great many ““knobsknobs”” for setting its behavior. for setting its behavior.

The Loader is often used in demanding applications in which theThe Loader is often used in demanding applications in which theperformance tuning options are critical.performance tuning options are critical.

The Loader is, increasingly, asked to translate between different dataThe Loader is, increasingly, asked to translate between different datamodels.models.


All rights reserved

ArchitecturesArchitectures ReplicationReplication RollupRollup ArchivalArchival AuditAudit DataData

WarehousingWarehousing Operational DataOperational Data

Stores (ODS)Stores (ODS) Web SiteWeb Site Extract TransformExtract Transform

Store (ETS)Store (ETS) Publish changesPublish changes

to trigger actionsto trigger actions

Database ReorganizationDatabase Reorganization Disaster RecoveryDisaster Recovery Conversion from one databaseConversion from one database

platform to anotherplatform to another Capture of real world scenarios forCapture of real world scenarios for

regression testingregression testing Test environments for tuningTest environments for tuning

alternativesalternatives Specialization for remote orSpecialization for remote or

departmental data subsets,departmental data subsets,whether for convenience,whether for convenience,conservation of resources, orconservation of resources, orsecuritysecurity


All rights reserved

Clever CustomerClever CustomerConfigurationsConfigurations

Flexible architecture allows for varied usesFlexible architecture allows for varied uses No two Loader installations are the sameNo two Loader installations are the same

Obviously, source databases are differentObviously, source databases are different Requirements for operation differ, mostlyRequirements for operation differ, mostly

based on the targetbased on the target


All rights reserved

VX Company at the DutchVX Company at the DutchRailway CompanyRailway Company

Initial pilotInitial pilot Proof of conceptProof of concept Validate that feature requirements for Validate that feature requirements for RdbRdb

LogMinerLogMiner are low profile are low profile Validate that the JCC Validate that the JCC LogMinerLogMiner Loader is also Loader is also

low profilelow profile 2 source databases to a standalone database2 source databases to a standalone database RdbRdb RdbRdb Replication-style, roll-up operationReplication-style, roll-up operation


All rights reserved

VX @ DR (cont.)VX @ DR (cont.) 22ndnd project project –– Regional to Roll-up Regional to Roll-up

A different source database to a newA different source database to a newdatabasedatabase

RdbRdb RdbRdb Replication-style operationReplication-style operation

33rdrd project project –– Regional to Roll-up (cont.) Regional to Roll-up (cont.) Expand 2Expand 2ndnd project to include 13 more source project to include 13 more source

databases, same target databasedatabases, same target database RdbRdb RdbRdb Replication-style, roll-up operationReplication-style, roll-up operation


All rights reserved

VX @ DR (cont.)VX @ DR (cont.) 44thth project project –– Roll-up to Application Roll-up to Application

Replicate data from Replicate data from targettarget database from the 3 database from the 3rdrd

project directly into the applicationproject directly into the application RdbRdb XML/API XML/API Audit-style operationAudit-style operation

55thth project project –– Roll-up to Consumable Roll-up to Consumable Replicate data from Replicate data from targettarget database from the 3 database from the 3rdrd

project to a messaging databaseproject to a messaging database RdbRdb Oracle Oracle Replication-style operationReplication-style operation


All rights reserved

Minnesota State College andMinnesota State College andUniversity SystemUniversity System

11stst use use –– Central to Subset Central to Subset Replicate centrally controlled data to multipleReplicate centrally controlled data to multiple

subset databasessubset databases RdbRdb RdbRdb Replication-style, partitioning operationReplication-style, partitioning operation

22ndnd use use –– Central to Query Central to Query Replicate databases to multiple subsetReplicate databases to multiple subset

databasesdatabases RdbRdb RdbRdb Replication-style, partitioning operationReplication-style, partitioning operation


All rights reserved

MnSCUMnSCU (cont.) (cont.) 33rdrd use use –– Distributed to Central Distributed to Central

Roll-up multiple production databases into a singleRoll-up multiple production databases into a singletargettarget

RdbRdb Oracle Oracle Replication-style, roll-up operationReplication-style, roll-up operation

44thth use use –– Central to Development Central to Development Replicate production data into non-Replicate production data into non-RdbRdb databases to databases to

support development of database-independentsupport development of database-independentapplicationapplication

RdbRdb Oracle Oracle Replication-style operationReplication-style operation


All rights reserved

An Ohio-based utilityAn Ohio-based utility 11stst project project –– CIS to GIS CIS to GIS

Replicate data from source database to a geographic-Replicate data from source database to a geographic-based databasebased database

RdbRdb Oracle Oracle Replication-style, transformation operationReplication-style, transformation operation

22ndnd project project –– CIS to WWW CIS to WWW Replicate data from source database to a websiteReplicate data from source database to a website

databasedatabase RdbRdb SQL Server SQL Server Replication-style operationReplication-style operation


All rights reserved

‘‘He-Who-Must-Not-Be-NamedHe-Who-Must-Not-Be-Named’’ 11stst project project –– Federated to CRM Federated to CRM

Replicate data from source databases to a CRMReplicate data from source databases to a CRMdatabasedatabase

RdbRdb XML/API XML/API Replication-style operationReplication-style operation

22ndnd project project –– Federated to ODS Federated to ODS Replicate data from source databases to an ODSReplicate data from source databases to an ODS

databasedatabase RdbRdb Tuxedo Tuxedo Performance requirements exceed 2,000 Performance requirements exceed 2,000 rpsrps Replication-style, roll-up operationReplication-style, roll-up operation


All rights reserved

‘‘He-Who-Must-Not-Be-NamedHe-Who-Must-Not-Be-Named’’ ––book 2book 2

11stst project project –– TP to Parallel TP to Parallel Replicate data from source database to a parallel databaseReplicate data from source database to a parallel database RdbRdb Oracle Oracle Performance requirements: 4,000+ Performance requirements: 4,000+ tpstps during the day and during the day and

360,000+ row transactions at night360,000+ row transactions at night Replication-style operationReplication-style operation

22ndnd project project –– TP to DR TP to DR Replicate data from source databases to a secondary DRReplicate data from source databases to a secondary DR

databasedatabase RdbRdb RdbRdb Performance requirements: 4,000+ Performance requirements: 4,000+ tpstps during the day and during the day and

360,000+ row transactions at night360,000+ row transactions at night Replication-style operationReplication-style operation


All rights reserved

‘‘He-Who-Must-Not-Be-NamedHe-Who-Must-Not-Be-Named’’ ––book 3book 3

CIS to Reporting/Look-upCIS to Reporting/Look-up Replicate data from source database to aReplicate data from source database to a

reporting/look-up databasereporting/look-up database RdbRdb Oracle Oracle Initial population of targetInitial population of target

32,000,000 rows32,000,000 rows 50 - 60ms ping time50 - 60ms ping time

Replication-style operationReplication-style operation


All rights reserved

Triple-C AwardTriple-C AwardThe most The most ““Clever Customer ConfigurationClever Customer Configuration””award goes toaward goes to……

Jack Broer of the VX Company for theJack Broer of the VX Company for theDutch Railway Company implementationDutch Railway Company implementationof the Loader to send XML messages toof the Loader to send XML messages tothe application to monitor the currency ofthe application to monitor the currency ofthe Loader synchronization.the Loader synchronization.


All rights reserved

Recent ReleasesRecent Releases Version 3.0.1, June, 2006Version 3.0.1, June, 2006

Additional tuning support for Java Virtual Machine (JDBC target)Additional tuning support for Java Virtual Machine (JDBC target) New DBA tools (New DBA tools (DBKeyDBKey -> -> bigintbigint, , bigintbigint -> -> DBKeyDBKey)) Corrects a timing issue on restartCorrects a timing issue on restart

Version 3.0, January, 2006Version 3.0, January, 2006 JDBC (class IV) driver target supportJDBC (class IV) driver target support Support for Itanium, OpenVMS 8.2, Support for Itanium, OpenVMS 8.2, RdbRdb 7.2, Oracle 10g client 7.2, Oracle 10g client Performance: lock management optimization to reduce deadlock searchesPerformance: lock management optimization to reduce deadlock searches

Version 2.2.8, July, 2005Version 2.2.8, July, 2005 Provides a significant performance enhancement for parallel constrained modeProvides a significant performance enhancement for parallel constrained mode Provides new procedure names to simplify specificationProvides new procedure names to simplify specification

Version 2.2.7, June, 2005Version 2.2.7, June, 2005 Support date conversion for Oracle targets in some international contextsSupport date conversion for Oracle targets in some international contexts Addresses floating point accuracy when using Oracle 10g targetsAddresses floating point accuracy when using Oracle 10g targets

Version 2.2.6, May, 2005Version 2.2.6, May, 2005 Correct Correct datatypedatatype issue with TIMESTAMP and Oracle targets issue with TIMESTAMP and Oracle targets

Version 2.2.5, April, 2005Version 2.2.5, April, 2005 Corrected a rarely seen stall associated with constrained synchronization locksCorrected a rarely seen stall associated with constrained synchronization locks Provided a workaround for a memory leak in TuxedoProvided a workaround for a memory leak in Tuxedo Added robustness to the installation process and the IVPAdded robustness to the installation process and the IVP


All rights reserved

Implications of that ListImplications of that List JCC responds promptly when an issue isJCC responds promptly when an issue is

discovered. Example: Date internationalizationdiscovered. Example: Date internationalization The Loader is well enough tested that there areThe Loader is well enough tested that there are

few corrections required.few corrections required. Performance enhancements are always beingPerformance enhancements are always being

sought. Example: sought. Example: Extremely Extremely large transactionslarge transactions DBA assists are added whenever possible.DBA assists are added whenever possible.

Example: Logical names and Example: Logical names and searchlistsearchlist for for AIJsAIJs JCC resolves issues with companion products,JCC resolves issues with companion products,

whether in the Loader or requiring a workaround.whether in the Loader or requiring a workaround.Example: Tuxedo memory leakExample: Tuxedo memory leak


All rights reserved

Futures Futures –– Disappointment Disappointment TuxedoTuxedo

BEA Corporation reports that it will not support futureBEA Corporation reports that it will not support futuredevelopment on OpenVMS and will not support adevelopment on OpenVMS and will not support aversion on Itanium.version on Itanium.

LogMiner Loader Version 3 will be the last release ofLogMiner Loader Version 3 will be the last release ofthe Loader to support a Tuxedo target.the Loader to support a Tuxedo target.

JCC will continue to support any installation that isJCC will continue to support any installation that isusing the Loader Version 3 with a Tuxedo target, forusing the Loader Version 3 with a Tuxedo target, forso long as that use continues and the LML supportso long as that use continues and the LML supportcontract continues.contract continues.

JCC has dropped our support contract with Tuxedo,JCC has dropped our support contract with Tuxedo,following notice that the only Loader target that wefollowing notice that the only Loader target that weknow of will be phased out.know of will be phased out.


All rights reserved

Futures Futures –– Possibilities Possibilities Improved performance for JDBC targetsImproved performance for JDBC targets Oracle as a sourceOracle as a source Additional database products as direct targetsAdditional database products as direct targets

with the attention to product specific knowledgewith the attention to product specific knowledgethat has made Loader Oracle and Rdb targetsthat has made Loader Oracle and Rdb targetsso successfulso successful

Configuration GUIConfiguration GUI Automation improvements in responding toAutomation improvements in responding to

changes in the source database metadatachanges in the source database metadata Expanded ETL (schema change) support forExpanded ETL (schema change) support for

weaving back together enterprises with fracturedweaving back together enterprises with fracturedinformation architecturesinformation architectures


All rights reserved

FYIFYIThe JCC LogMiner Loader is now at Version 3.0.1. Thenumber of targets supported include Rdb, Oracle, XML,platforms with JDBC Class IV drivers and Tuxedo.The Loader is fully tested and certified for versionupgrades in all the companion products.Other recent enhancements are based on extensiveexperience at multiple installations. The Loader isincreasingly feature-rich. Aids for the DBA and fortesting your own architectures are outstanding.


All rights reserved

AvailabilityAvailability Kit is available at Kit is available at FTP.JCC.COMFTP.JCC.COM

DocumentationDocumentation KitKit

Evaluation license available on requestEvaluation license available on request Send mail to Send mail to [email protected]@jcc.com

Find descriptions of the LogMiner LoaderFind descriptions of the LogMiner Loaderand other information atand other information athttp://http://www.jcc.com/LML.htmwww.jcc.com/LML.htm


All rights reserved

AcknowledgementsAcknowledgements Thanks to Rdb engineering for theirThanks to Rdb engineering for their

support and counselsupport and counsel For sharing their experiences with theFor sharing their experiences with the

LoaderLoader VX Company and the Dutch RailwayVX Company and the Dutch Railway

CompanyCompany Minnesota State Colleges and UniversitiesMinnesota State Colleges and Universities An Ohio-based utilityAn Ohio-based utility

JCC LogMiner Loader: Today and Beyond - download.oracle.com · 14/11/2006 · Replicate data from...

Documents

Transcript of JCC LogMiner Loader: Today and Beyond - download.oracle.com · 14/11/2006 · Replicate data from...