Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
-
Upload
continuent -
Category
Technology
-
view
1.221 -
download
2
description
Transcript of Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
![Page 1: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/1.jpg)
©Continuent 2014
Replicate Oracle to Oracle, Oracle to MySQL ,
and Oracle to Analytics
Linas Virbalas, Senior Software Engineer Robert Hodges, CEO
![Page 2: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/2.jpg)
©Continuent 2014
Introducing Continuent
���2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
![Page 3: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/3.jpg)
©Continuent 2014
What is Tungsten Replicator?
���3
Tungsten Replicator: a high-performance
open source database replication engine
!GPL V2 license - 100% open source
Download from https://code.google.com/p/tungsten-replicator/ Annual support subscription available from Continuent
“GoldenGate without the Price Tag”®
![Page 4: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/4.jpg)
©Continuent 2014
Tungsten Replicator Overview
���4
Master
(Transactions + Metadata)
Slave
THL
DBMS Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
![Page 5: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/5.jpg)
©Continuent 2014 ���5
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
Master DBMS
Transaction History Log
QueueSlave
DBMS
Tungsten Replication Service
![Page 6: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/6.jpg)
©Continuent 2014
Filters and Parallel Apply
���6
Extract Filter Apply
StageExtract Filter Apply
StageStage
Pipeline
Master DBMS
Transaction History Log
In-Memory Queue
Slave DBMS
Extract Filter ApplyExtract Filter ApplyExtract Filter Apply
![Page 7: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/7.jpg)
©Continuent 2014
Multiple Services per Replicator
Service frommysql
Service fromoracle
frommysql
fromoracle
Replicator
Replicator
Replicator
Replicator
Aggregated
![Page 8: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/8.jpg)
©Continuent 2014
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL
![Page 9: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/9.jpg)
©Continuent 2014©Continuent 2014
Heterogeneous Replication
![Page 10: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/10.jpg)
©Continuent 2014©Continuent 2014
Steps to Heterogeneous Replication
1. Prepare (translate) schema for the slave DBMS
2. Set up replication
3. Provision initial data
4. Continue real-time replication
ddlscan
tpm
Parallel ApplyMySQL Blackhole
Tungsten Replicator
Parallel Extract
![Page 11: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/11.jpg)
©Continuent 2014©Continuent 2014
Replicating from MySQL to Oracle
![Page 12: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/12.jpg)
©Continuent 2014©Continuent 2014
Use Case: Real-Time Sales/Order Fulfillment
Real-Time Replication
Web-based Sales Purchase Ordering
3M Order Items
~1M Xacts/ Day
![Page 13: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/13.jpg)
©Continuent 2014©Continuent 2014
1. Translating schema for the slave
![Page 14: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/14.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���14
• Goal - set up heterogeneous replication
Replication
![Page 15: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/15.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���15
• Beginning - how to convert tables?
empty
Tables
![Page 16: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/16.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���16
• Beginning - how to convert tables?
empty
Tables
•Data types? •Column lengths? •Naming conventions? •Reserved words?
![Page 17: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/17.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���17
• ddlscan automates these conversions
empty
Tables
ddlscan
![Page 18: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/18.jpg)
©Continuent 2014©Continuent 2014
ddlscan
���18
• Part of Tungsten Replicator, GPL v2
• Translates schema with replication in mind
• Provides errors and warnings
• Can rename schema/tables/columns
![Page 19: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/19.jpg)
©Continuent 2014©Continuent 2014
Usage (MySQL to Oracle Example)
���19
$ cd tungsten-replicator/bin !
$ ./ddlscan \ -db test \ -template ddl-mysql-oracle.vm \ -user tungsten \ -pass secret
![Page 20: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/20.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���20
• ddlscan looks into source schema
empty
Tables
![Page 21: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/21.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���21
• ddlscan translates and renders DDL commands
empty
Tables
oracle-ddl.sql
![Page 22: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/22.jpg)
©Continuent 2014©Continuent 2014
Result of ddlscan
���22
CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */
( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0)
/* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */
! /* ERROR: table must have a primary key! */ ); !CREATE TABLE talks ...
![Page 23: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/23.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���23
• You run resulting SQL file on Oracle
empty
Tables
oracle-ddl.sql
![Page 24: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/24.jpg)
©Continuent 2014©Continuent 2014
Translating Schema
���24
• Tables are ready!
Tables Translated tables (empty)
![Page 25: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/25.jpg)
©Continuent 2014©Continuent 2014
2. Set Up Replication
![Page 26: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/26.jpg)
©Continuent 2014©Continuent 2014
Replicator Installation
���26
./tools/tpm configure my_ora \ --enable-heterogenous-service=true \ --install-directory=/opt/mysql_to_ora/continuent \ --members=alpha,bravo \ --master=alpha !./tools/tpm configure my_ora --hosts=alpha \ --replication-user=tungsten \ --replication-password=secret !./tools/tpm configure my_ora --hosts=bravo \ --replication-user=tungsten_frommysql \ --replication-password=secret \ --datasource-type=oracle \ --datasource-oracle-service=ORCL \ --svc-applier-filters=dropstatementdata !./tools/tpm install my_ora !./tools/tpm start my_ora
!
![Page 27: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/27.jpg)
©Continuent 2014©Continuent 2014
Replication Under the Hood
���27
MySQL Binlog
binlog_format=row
Service my_ora
Tungsten Master Replicator
MySQLExtractor Special Filters •Transform
ENUM to string •Transform SET
to string •Column names
and signed flag
Service my_ora
Tungsten Slave Replicator
Special Filters •Drop DDL •Map names to upper case •Rename too long or reserved
objects •Optimize updates to remove
unchanged columns •Ignore extra tables
OracleApplierMySQLExtractor
![Page 28: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/28.jpg)
©Continuent 2014©Continuent 2014
3. Provisioning an Oracle slave
![Page 29: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/29.jpg)
©Continuent 2014©Continuent 2014
table structure
table data
table structure
Provisioning (2)
���29
table data
table structuretable data
table structure table structuretable structureddlscan
![Page 30: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/30.jpg)
©Continuent 2014©Continuent 2014
table structure
Provisioning (3)
���30
table structuretable structure
MySQL Sandbox
CONVERT to BLACKHOLE
table data
table structuretable data
table structuretable data
table structure
![Page 31: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/31.jpg)
©Continuent 2014©Continuent 2014
Provisioning (4)
���31
MySQL Sandbox
table data
table structuretable data
table structuretable data
table structure
not replicating
backup
![Page 32: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/32.jpg)
©Continuent 2014©Continuent 2014
Provisioning (5)
���32
MySQL Sandbox
backup
bin logs
THL
1
(ROW)
Master Tungsten
Replicator
Slave Tungsten
Replicator
2 3 4
![Page 33: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/33.jpg)
©Continuent 2014©Continuent 2014
4. Continue Real-Time Replication
![Page 34: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/34.jpg)
©Continuent 2014©Continuent 2014
Provisioning (6)
���34
MySQL Sandbox
table data
table structuretable data
table structuretable data
table structure
bin logs
THL
Master Tungsten
Replicator
Slave Tungsten
Replicator
![Page 35: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/35.jpg)
©Continuent 2014
Replicating from Oracle to MySQL
![Page 36: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/36.jpg)
©Continuent 2014
Use Case: Web Content Publishing
Real-Time Publication
Backend Office Web-Based Catalog
![Page 37: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/37.jpg)
©Continuent 2014
How Does Tungsten Do That?
���37
Service ora_my
Tungsten Master Replicator
OracleCDCExtractor No Special Filters
Service ora_my
Tungsten Slave Replicator
Special Filters •Map names to lower case •Ignore extra tables •Heartbeat table renaming
demo schema
demo_pub schema
Oracle CDC (Synchronous or
Async Hotlog)
MySQLApplierOracleCDCExtractor
![Page 38: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/38.jpg)
©Continuent 2014
Replicating from Oracle to Oracle
![Page 39: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/39.jpg)
©Continuent 2014
How Does Tungsten Do That?
���39
Service ora_ora
Tungsten Master Replicator
OracleCDCExtractor No Special Filters
Service ora_ora
Tungsten Slave Replicator
Special Filters No Special Filters
demo schema
demo_pub schema
Oracle CDC (Synchronous or
Async Hotlog)
Tungsten Slave Replicator
OracleApplierOracleCDCExtractor
![Page 40: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/40.jpg)
©Continuent 2014©Continuent 2014
DEMO
DEMO.* demo.*
DEMO2.* demo2.*
![Page 41: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/41.jpg)
©Continuent 2014
Replicating from Oracle to Vertica
![Page 42: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/42.jpg)
©Continuent 2014
Use Case: Web Content Publishing
Near Real-Time Publication
Backend Office Analytic Reports
Feeds from other data sources
![Page 43: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/43.jpg)
©Continuent 2014
How Does Tungsten Do That?
���43
Service ora_ver
Tungsten Master Replicator
OracleCDCExtractor No Special Filters
Service ora_ver
Tungsten Slave Replicator
Special Filters •Map names to lower case •Ignore extra tables •Rename data tables (if required)
demo schema
demo_pub schema
Oracle CDC (Synchronous or
Async Hotlog)
SimpleBatchApplierOracleCDCExtractor
![Page 44: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/44.jpg)
©Continuent 2014
Batch Loading--The Gory Details
���44
Replicator
Transactions from master
CSV FilesCSV FilesCSV Files
Staging TablesStaging TablesStaging Tables
Base Tables
Base Tables
Base Tables
Merge Script
(or) COPY
directly to base tables
COPY to stage tables SELECT to
base tables
Service ora_ver
![Page 45: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/45.jpg)
©Continuent 2014
DEMO
![Page 46: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/46.jpg)
©Continuent 2014
Coming Attractions
![Page 47: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/47.jpg)
©Continuent 2014
New Replication Features in 2014
• Parallel extraction for fast provisioning
• Select data as of specific SCN
• Pull data across in multiple streams
• Matches parallel apply on slave side
!
• Hadoop data loading
• Direct loading from MySQL/Oracle to HDFS
• View data using Hive
• Generation of materialized views
���47
![Page 48: Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics](https://reader035.fdocuments.net/reader035/viewer/2022081420/5589e45ad8b42a76558b46bd/html5/thumbnails/48.jpg)
©Continuent 2014 ���48
Continuent Web Page: http://www.continuent.com
!Documentation:
https://docs.continuent.com !
Tungsten Replicator: http://code.google.com/p/tungsten-replicator
Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected]