- Oracle · Oracle Data Quality Data Profiling Data Parsing Data Cleansing ... Data Availability...
Transcript of - Oracle · Oracle Data Quality Data Profiling Data Parsing Data Cleansing ... Data Availability...
1
<Insert Picture Here>
Oracle GoldenGate Product Strategy and Roadmap
Chris McAllister
Senior Director Product Management – Oracle GoldenGate
3
This Time Last Year at Oracle Open World…
• Officially GoldenGate was
purchased September 3, 2009
• Things were pretty hectic at
the last OOW
• This year….
4
Fast Forward to Today…
• Things are still a bit
hectic!
• This year we are
excited about our
first release since
the acquisition
• It is a joint Oracle
Data Integration
release with
integration between
Oracle GoldenGate
and Oracle Data
Integration
5
Oracle GoldenGate
and the Red Stack
6
Oracle Data Integration SolutionBest-in-class Heterogeneous Platform for Data Integration
MDMApplications
SOAPlatforms
OracleApplications
BusinessIntelligence
Activity Monitoring
Custom Applications
Oracle GoldenGate
Log-based CDC
Bi-directional Replication
Real-time Data
SOA Abstraction Layer
Service BusProcess Manager Data Services
Oracle Data Integrator
ELT/ETL
Data Transformation
Bulk Data Movement
OLTPSystem
Flat FilesData Warehouse/Data Mart
OLAP Cube Web 2.0 Web and Event Services, SOA
Storage
Data Verification
Oracle Data Quality
Data Profiling
Data Parsing
Data Cleansing
Data Federation
Data Lineage Match and Merge
Comprehensive Data Integration Solution
77
Active Data Guard
Direct
Memory Access
Read
On-Disk LogsFast SQL
Direct
Write to Logs
Oracle Products for Data MovementComparing How They Work
Redo in
Memory
Buffer
GoldenGate
Data
Integrator
DE
CR
EA
SIN
G L
AT
EN
CY
IN
CR
EA
SIN
G T
RA
NS
FO
RM
AT
IO
N
Retrieve Data Load Data
Disaster Recovery & Data Protection
Real Time Data Integration & High Availability
SQL Query Set-based,
Complex SQL
Data Integration for Data Warehouse & SOA
HE
TE
RO
GE
NE
OU
S
8
Any Data Warehouse
Any Bulk Data Transformation
Complex Migrations
ODIAny Real Time Data Needs
Zero-Downtime Migrations (OS, HW, DB)
Active-Active High-Availability
OGG
Any Oracle DR
Read-only Reporting
ADG
Oracle Products for Data MovementWhen to Use
9
Comparing GoldenGate & Streams
GoldenGate is Oracle’s strategic
replication solution
• Highly customizable and flexible
• Optimized for Oracle Database
• Few large references
Oracle Streams
• Out-of-the-box solutions
• Broad heterogeneous support
• Excellent reference base
Oracle GoldenGate
Streams continues to be a
supported Oracle Database feature
10
• Oracle GoldenGate
Architecture
11
• Broad, heterogeneous connectivity
• Maximum uptime for sources and targets
• Wide range of use cases
Continuous, Pervasive Access
• Low-impact, non-intrusive, scalable
• Ability to select latency and transformation points
Real-Time
• High-performance, low-TCO architecture
• Rapid development and deployment
• Reliability and easy maintenanceEfficient
Oracle Data Integration’s Advantages
12
How Oracle GoldenGate Works
LAN/WANInternet
TCP/IP
CaptureTrail
Pump DeliveryTrail
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
SourceOracle & Non-Oracle
Database(s)
TargetOracle & Non-Oracle
Database(s)
13
How Oracle GoldenGate Works
LAN/WANInternet
TCP/IP
Bi-directional
CaptureTrail
Pump DeliveryTrail
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
SourceOracle & Non-Oracle
Database(s)
TargetOracle & Non-Oracle
Database(s)
15
EM
P
Oracle Real Time Data IntegrationOracle‟s Unique Offering for Real Time Business Intelligence
Oracle
GoldenGate
Oracle Data
Integrator
EM
P
DEP
T
DI
MFAC
T
DI
M
DI
M
DI
MODS Schema DW Schema
On-Disk
Logs
Source 1
Source 2
Oracle
GoldenGate
Oracle GoldenGate
• Real time extracts from
transactional systems• Non-invasive on sources
• Continuous streaming load
into ODS Schema of target• Latency in seconds
Oracle Data Integrator EE
• High Performance „E-LT‟ on
target data warehouse• Periodic mini-batches (15 min)
• Transform in the Database
• Never go back to sources
EM
P
DEP
T
DEP
T
On-Disk Logs
16
Traditional ETL + CDC
• Invasive Capture on OLTP systems
using complex Adapters
• Transformations in ETL engine on
expensive middle tier servers
• Bulk load to the data warehouse with
large nightly/daily batch
• Continuous feeds from operational
systems
• Non-invasive data capture
• Thin middle tier with transformations
on the database platform (target)
• Mini-batches throughout the day or
bulk processing nightly
ODI + Oracle GoldenGate
Staging
Trickle
LookupData
Load
Extract
LookupData
Xform XformBulk
GG
+ O
DI
GG
+ O
DI
Heterogeneous
Real-time CDC Integration Best-in-class solution for real-time with Oracle GoldenGate
17
Oracle GoldenGate
Solutions
18
Mission-Critical Systems and Batch ProcessingToo Much Data, Not Enough Time
What time of your day is your business
NOT at it‟s peak?
operations
operations
Available
19
Business event
Action Time
Data
latency
Analysis
latency
Decision
latency
Busin
ess V
alu
e
Data captured
Information delivered
Action taken
From TDWI The Business Case for Real-Time BI
Based on concept developed by Richard Hackathorn, Bolder Technology
Why Real Time?Increasing the Value of Enterprise Information
How late is your business reacting?
20
Data Integration Solutions
Business Intelligence Consolidation / Migration
Data Availability Data-based App Integration
• Real Time Data Warehouse• Need fresh data 24X7?
• Mission-critical OLTP?
• Want to get double duty from DW
investment?
• Operational Reporting• Need operational visibility in real
time?
• Can‟t impact performance of OLTP?
• Zero Downtime Migration• Want to get on lower cost standard
infrastructure?
• Can‟t afford downtime?
• Data Move & Improve • Does your data cost $$ or make $$?
• Why not fix data assets as well as
HW/SW assets during a move?
• Data Integration for SOA• Large flat files require transformation?
• Direct to database connections for
source and/or target?
• Batch processes?
• Data Synchronization• Require very low latencies with very
high transaction throughput?
• Require active-active data sync?
• DR / Data Protection• Will your data be available when…?
• Is your standby database sitting idle?
• Zero Downtime Operations• Is planned downtime at least as costly
as unplanned downtime?
• Need continual online maintenance?
• Ever expect to do another HW, OS,
DB upgrade?
ODI &
GoldenGate
GoldenGate
Active
Data Guard
GoldenGate
ODI, Data
Quality &
Profiling
GoldenGate
ODI Suite
GoldenGate
21
Announcing
22
The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle‟s
products remains at the sole discretion of Oracle.
23
Designed for Enterprise Usage & Fast Time to ValueGoldenGate 11g New Features
Enhanced
performance
with LOBs
Enhanced
Delivery
Performance
Certified and
configured for
Operational Reporting
with EBS, PSFT, JDE
Capture from JMS,
MySQL
Expanded
heterogeneity and
datatype coverage
Simpler, faster
recovery for capture
from Oracle DB
Consistent and
enhanced tracing for
all platforms
Oracle DB
Enhancements:
Direct Load
operations
Enhanced
Manageability
– timeouts for
long-running
transactions
Optional trigger
suppression and
relaxed constraint
checking
24
Oracle GoldenGate 11gR1 Goals
• Differentiators
• New databases
• New versions
• JMS capture
• New data types
• Increased performance
• Simplified Operations
• Simplify recovery
• Enhanced manageability
• Focused tracing
25
Oracle Enhancements
Bounded recovery
Reduce fetching of LOBs
Support direct loads (Capture)
DDL password replication
High priority DDL enhancements
Suppress triggers on apply
Optionally defer constraint checking
26
Other DB Enhancements
MySQL Log based capture (v5.1.16
and above)
Support InnoDB for Capture
Support InnoDB for Delivery
Statement Caching
Dynamic SQL Support
Support TEXT and BLOB data
types
TimesTen Delivery
27
Other DB Enhancements
DB2 LUW 9.7
Added support of DB2CONNECT –
LUW delivery
DB2 for i delivery
Base24 Full Refresh enhancement
TCP streaming
Temporary file management
SOURCEISFILE - restart by key
UNI[VAR]CHAR support
Dynamic SQL support
Array fetching for initial load
28
Databases O/S and Platforms
Oracle GoldenGate Capture:
Oracle
DB2 for v 9.7
Microsoft SQL Server
Sybase ASE
Teradata
Enscribe
SQL/MP
SQL/MX
MySQL
JMS message queues
Oracle GoldenGate Delivery:
All listed above, plus:
TimesTen, IBM System i
Netezza, Greenplum, HP Neoview and any
ODBC compatible databases
ETL products
Linux
Sun Solaris
Windows 2000, 2003, XP
HP NonStop
HP-UX
HP OpenVMS
IBM AIX
IBM z Series
zLinux
28
Expanded Heterogeneity
NEW
NEW
NEW
NEW
29
• Oracle Future Ideas
30
Oracle GoldenGate Future Ideas• Oracle Community
• Tighter hardware and DB integration
• Increased application support
• Expanded use of FMW stack
• Integration with OEM
• Competition
• Support more 3rd party software
• Improve scalability without sacrificing transactionality
• Leverage Oracle distribution channels
• Market Expansion
• Groundwork for heterogeneous international data support
• Increase DB sources and targets
31
Oracle GoldenGate
Management Pack
32
Oracle GoldenGate Director
What is GoldenGate Director?
• Simplifies overall deployment and management of
GoldenGate TDM products
• A centralized server-based graphical application that
offers an intuitive way to define, configure, manage, and
report on GoldenGate TDM processes.
• An add-on product for new or existing GoldenGate
customers.
33
GoldenGate Director: Architecture Components
Director Server
• Live connections to each
GoldenGate instance
• Database central repository
for user and GoldenGate
information
Director Client
• GUI for creating custom network
views and accessing the
services provided by the Director
Server
Director Web
• Browser-based UI that enables
remote monitoring and control of
GoldenGate
Director Administrator
• For configuring the Director Server
34
Director Client
• Rich GUI Client
• Customizable Diagrams
• GoldenGate Networks
• View
• Configure
• Manage
• Monitor
• GGSCI Execution
• Consolidated Logs
• Alert Functionality
• Custom
• Wizard Driven
36
• Oracle GoldenGate
Veridata
37
What is GoldenGate Veridata?
“GoldenGate Veridata is a high-speed, low-impact data comparison
solution that identifies and reports data discrepancies between two
databases, without interrupting those systems or the business processes
they support.”
38
Why Do You Need Veridata?
Data discrepancies are a reality of life
User errors
Input errors
Unintended use
Malicious intent
Infrastructure errors
System failure
Disk corruption
Network outage
Migration Confirmation
Database upgrades
Database migrations
Operating system upgrades
Application upgrades
Real Time Data Integration
Verification
Data warehouse
Query offloading
Marketing databases
Application errors
Faulty logic
Failed upgrades
Latent bugs
Configuration errors
Applications
Replication
Network
39
Veridata 3.0 Component Overview
• Java agent is one executable for all supported databases (except NSK).
Supports heterogeneous cross database comparisons.
• C agent is required for NSK and can be used for Oracle like-to-like comparisons.
Veridata Server Machine
Veridata Web
Server
Command Line
Veridata Web Client
Veridata
Server
web
services
web service
client cli
veridata
repository
db
Target Machine
Java Veridata
Agent
Trg DB
Source Machine
C Veridata
Agent
Src DB 2Possible
On Going
Replication
Compressed
Data
Source Machine
Java Veridata
Agent
Src DB 1
40
Oracle GoldenGate Veridata Platforms
Component Database O/S and Platforms
GoldenGate Veridata
Java Agent
Oracle 9i, 10g, 11g
SQL Server 2000, 2005, 2008
Teradata 2vR6, 12, 13
DB2 LUW v8.x, v9.5
DB2 z/OS v8.x, v9.5
Windows
Red Hat
Sun Solaris
SUSE
HP-UX
HP Tru64
IBM AIX
GoldenGate Veridata
C Agent
Oracle 9i, 10g, 11g
Enscribe
NonStop SQL/MP
HP NonStop Server
(Guardian environment)
GoldenGate Veridata
Server
Repository:
Oracle 9i, 10g, 11g
MySQL 5
SQL Server 2005 and 2008
Windows
Red Hat
Sun Solaris
HP-UX
IBM AIX
41
Veridata Future Ideas
• Resync functionality
• XML population functionality
• Sybase support
• On database hash calculation
• Globalization
• Light transformation
• Delta processing
• Populating an XML input file from
a GoldenGate replication
environment
• Compare pair / column mapping
• Promoting a Veridata setup to
another environment
42
Oracle GoldenGate Future Release Estimates
Q3 Q4 Q1 Q2
2010 2011
Q1
Oracle
GoldenGate
11gR1
Q3 Q4
OGG Veridata
11gR2?
Oracle
GoldenGate
11gR2 ?
OGG Director
11gR1
OGG 11gR1
PSU
OGG 11gR1
PSU ?
43
Tuesday 14:00 Dell Success Story: Harnessing the Power of Oracle GoldenGate and Oracle Streams
Moscone West L3 3020
Wednesday 11:30 Achieving Continuous Availability: Retail Decisions Case Study Moscone West L3 3020
Wednesday 13:00 Customer Panel: Innovations in Data Integration Architecture
(Research In Motion, Ross, Cerner, Herbalife)
Moscone West L3 3020
Thursday 9:00 The Emerging Data Integration Platform: Real-Time CDC and Continuous E-LT
Moscone West L3 3020
Thursday 12:00 Real-Time Operational Reporting for Oracle Applications with Oracle GoldenGate
Moscone West L3 3020
Thursday 13:30 Zero-Downtime Database Migration and Upgrade at RIM Moscone West L3 3020
Thursday 14:30 Oracle GoldenGate Architecture Deep Dive Hotel Nikko- Peninsula
Other Related SessionsOracle GoldenGate and Oracle Data Integration
44
Follow us on:
Join the Oracle DI CommunityDiscuss, Share, Learn
• Oracle Data Integration: www.oracle.com/goto/odi
Oracle GoldenGate : www.oracle.com/goto/goldengate
Technical information and downloads: www.oracle.com/technetwork/middleware/data-integration/
Data Integration Events http://www.oracle.com/events
• www.twitter.com/ORCLGoldenGate
• http://blogs.oracle.com/dataintegration
• www.linkedin.com “Oracle Data Integration”
Visit to learn more:
45
46
47
• Case Studies
• Migration
48
Business Intelligence Real Time Data Warehouse
• Solution• Log-based extract of database transactions
from source
• Load to target with sub-second latency
• Transformation performed on the database
using E-LT in mini-batches
• Benefits• No resource / performance impact to OLTP
• Live data available for better decision making
• Get double-duty from database investment by
using it for transformations
• No batch windows necessary – key for global
businesses
• Products• ODI & GoldenGate
Ministry of Finance,
Financial Data Center
Create a Real Time DW
49
GoldenGate Usage – Zero Downtime Migration
ManagerManager
- Capture / Extract - Delivery / Replicat - Trail
Oracle 9i Oracle 10gInitial Instantiation
OLTP UsersTesters
OLTP users are connected to primary server. The new
database is empty.
Start GoldenGate capture process & store transactions in trail. Eliminates need for point-
of-consistency.
Instantiate new database with primary data while OLTP users
are active. Can use any method preferred.
Once instantiation is complete, deliver the real-time
transactions to new database handling collisions.
Now testers can confirm that the application will work
correctly with live data and not affect production.
Once testing is complete, start replication back to primary
from new database. Enables a fallback option.
Switch OLTP users to new database. If problems occur,
can immediate fallback to “old” database.
50
Consolidation / MigrationZero Downtime Migration & Data Move and Improve
• Solution• Use a tool to move data onto new system for
a major system migration: replace error-
prone, expensive hand coding
• Migrate HW, OS, DB with zero downtime
using GoldenGate
• Consolidate / migrate complex applications
while improving the quality of your data
• Benefits• Minimize application downtime while
migrating to new system
• Mitigate risk during migration or consolidation
by enabling fallback
• Reduce costs and risk by automating and
removing human error
• Improve data during consolidation or
migration; avoid garbage in / garbage out
• Products• Oracle GoldenGate, Oracle Data Integrator,
Oracle Data Profiling, Oracle Data Quality
51
High AvailabilityData Availability to Avoid Unplanned and Planned Downtime
Solution1. Physical replication with Active Data Guard
• Best for Disaster Recovery for Oracle Applications
• Best for Disaster Recovery for Oracle Database 11g
2. Logical, heterogeneous replication with
GoldenGate• Only solution for real-time replication from Teradata
• Best for regular maintenance in heterogeneous environment
3. Active-Active bi-directional or multi-master
replication with GoldenGate• Best for custom and specialized applications
Benefits• Ensure business continuity in any situation
• Reduce planned downtime for maintenance
• Use standby database so it is not sitting idle
Products• Oracle GoldenGate and/or
Oracle Active Data Guard