Oracle Data Integration - Overview

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

Oracle Data IntegrationINTRODUCTION, CORE CAPABILITIES AND INVESTMENT ROADMAP

Jeff PollockVice President of ProductsOracle Data Integration September, 2016

#OpenWorld 2016


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

#OpenWorld 2016 2

Five CoreCapabilities

1. Business ContinuityDATA ALWAYS AVAILABLE

2. Data MovementDATA ANYWHERE IT’S NEEDED

3. Data TransformationDATA ACCESSIBLE IN ANY FORMAT

4. Data GovernanceDATA THAT CAN BE TRUSTED

5. Streaming DataDATA IN MOTION OR AT RES

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 4

Eight Core Products

Cloud or On-Premise

MostInnovativeTechnology

#1#1

Realtime / StreamingData Integration Tool

Pushdown / E-LTData Integration Tool

1st to certify replication withStreaming Big Data

1st to certify E-LT tool withApache Spark/Python

1st to power Data Preparationw/ML + NLP + Graph Data

1st to offer Self-Service &

Hybrid Cloud solution

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016

Data Integration ProductsGOLDENGATE – DATA INTEGRATOR – DATA PREPARATION –STREAM ANALYTICS – DATAFLOW ML – METADATA

MANAGEMENT – DATA QUALITY


Oracle GoldenGate

Realtime Performance

Extensible & Flexible

Proven & Reliable

Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of database transactions across homogeneous and heterogeneous environments in real-time with no distance limitations.

MostDatabases

DataEvents

Transaction Streams

Cloud

DBs

Big Data

Supports Databases, Big Data and NoSQL:

* The most popular enterprise integration tool in history

7


Oracle Data Integrator

Bulk Data Performance

Non Invasive Footprint

Future Proof IT Skills

Oracle Data Integrator provides high performance bulk data movement, massively parallel data transformation using database or big data technologies, and block-level data loading that leverages native data utilities

Bulk DataTransformation

Most Apps,Databases

& Cloud Bulk Data Movement

Cloud

DBs

Big Data

1000’s of customers –

more than other ETL tools

Flexible ELT workloads run

anywhere: DBs, Big Data, Cloud

Up to 2x faster batch processes

and 3x more efficient tooling

8


Self-Service

Better Recommendations

Built-in Data Graph Zero software to install, easy to use browser based interface

Better automation and less grunt work

for humans

Graph database of real-world facts used for enrichment

Oracle Data Preparation

ReportingApps

FilesETL

Oracle Data Preparation is a self-service tool that makes it simple to transform, prepare, enrich and standardize business data – it can help IT accelerate solutions for the Business by giving control of data formatting directly to data analysts.

9


Business Friendly

Extreme Performance

Spatial Awareness

Oracle Stream Analytics

DB

Web / Devices

DataEvent

Data & Transaction Streams

Downstream(eg; Hadoop)

DataEvent

Oracle Stream Analytics is a powerful analytic toolkit designed to work directly on data in motion – simple data correlations, complex event processing, geo-fencing, and advanced dashboards run on millions of events per second.

Innovative dual model for

Apache Spark or Coherence grid

Simple to use spatial and geo-fencing features an industry first

Includes Oracle GoldenGate for

streaming transactions

10


Stream or Batch Data

Spark based Pipelines

ML-powered Profiling

Oracle Dataflow MLOracle Dataflow ML is big data solution for stream and batch processing in a single environment – Lambda based applications that can run streaming ETL for cloud based analytic solutions.

Batch and stream

processing at the same time

Machine learning guides users for data

profiling

Data movement across Oracle PaaS services

Most Apps,Databases

& Cloud

Bulk Data Movement

Streaming Data Cloud

DBs

Big Data

Big DataPipeline

11


Business Glossary

End-to-End Lineage

100+ Supported Systems

Oracle Metadata ManagementOracle Metadata Management provides an integrated toolkit that combines business glossary, workflow, metadata harvesting and rich data steward collaboration features.

Supports Databases, Big Data, ETL Tools, BI Tools etc:

BI Report Lineage

Taxonomy Lineage

Data Model Lineage

12


Simple to Use

High Performance Matching

Powerful DQ Rules

Oracle Enterprise Data QualityOracle Enterprise Data Quality establishes trusted business data by providing a foundation for data profiling, data standardization, match and merge capabilities and data cleansing.

Profile, Standardize, Match, Merge and Cleanse your data

DWMDM

Apps ETL

Health check for your data; quick & easy profiling and cleansing

Intuitive business user

friendly toolkit for quality rules

Suitable for high performance

Applications or Master Data

13


Powerful Cloud SolutionsFor Data Integration


Self-Service, Streaming or Batch Data Integration

Data Integration Cloud

Easy Self Service Data Preparation

Machine Learning and Semantic Technology

Streaming or Batch Integrations

• Modern Lambda and Kappa style streaming applications for real-time data pipelines

• ETL integrations for classical Data Warehousing, Data Marts and Operational Databases

• Real-time Replication with Changed Data Capture

• Point and click deployment of Replication services, ETL, and Dataflow pipeline services

• Self-Service Data Preparation designed for non-programmers to prepare rich data transformations

• Spark-based ML platform for Dataflow deployments

• High value Natural Language Processing (NLP) and built-in data graph for Recommendation Service

DataPreparation

StreamingApplications

ETL DataProcessing

Ingest from Sources Automate Publish


Comprehensive and Differentiated Data Integration in the Public CloudData Integration Cloud – Core Capabilities

Ground to Cloud Data Streams Pushdown ELT/ETL

• GoldenGate realtime Replication from on-premise to Cloud

• Data Integrator bulk data movement into Cloud

• From on-premise to pipelines to analytics in sub-seconds

• Ingest, move, transform and analyze data 100% in memory

• Ultra-high performance ELT processing in Hadoop or DBaaS

• Simple to choose which PaaS you want the workload to run

streaming

batch

Pipelines Analytics

batchdata transformation

Five CloudSolutions

1. Data MigrationsREPEATABLE PROVEN DATA MIGRATION TOOLS

2. DW IntegrationENABLE CLOUD DATA WAREHOUSE STRATEGIES

3. Dev & Test CloudOPERATE DEV-TEST DB’S IN CLOUD

4. Data High AvailabilityDATA RELIABILITY AT 99.999% SERVICE LEVELS

5. Heterogeneous CloudBEST OF BREED FOR AMAZON AWS ETC.


Key Benefits – Online/Offline Migrations

3rd Party Cloud Support

Database Migrations

• Seamless data migration from older databases or poorly optimized Cloud services

• Take advantage of newest Oracle DB versions in a simple subscription model

• No data loss for phased / online migrations

• Strong automation around provisioning and billing

• Reliable and proven core GoldenGate technology

• Migrate older Amazon RDS instances to Oracle

• Data delivery from on-premise to 3rd party Clouds

#OpenWorld 2016 18

Solution 1: Database Migrations in to Cloud

On Premise

RDS

Oracle DBaaS

I need to migrate my DBs to newer and less

expensive systems


Key Capabilities: Comprehensive Solution

3rd Party Cloud Support

Data Warehouse Integration

• Operational or Advanced Analytics in the Cloud requires reliable data feeds

• Work with data coming from on-premise data sources or other cloud sources

• Prepare and Ingest data in realtime; built in support for replication and streaming ingestion to DW/Marts

• Transform and Cleanse data; innovative approach allows workloads to run on different PaaS resources

• Supports Amazon RDS databases and Redshift

• Big data support with Kinesis and EMR

#OpenWorld 2016 19

Solution 2: Data Warehouse Integration in the Cloud

DW on ExadataCloud Service

UN

STR

UC

TUR

EDST

RU

CTU

RED

BI & DataVisualization

RDS & REDSHIFT

Help me run my analytics

from the Cloud


Key Benefits: Automate Data Movement

Oracle Database as a Service

Database Dev-Test Environments

• Keep development and test environments continuously in sync with minimum infrastructure or process overhead

• Leverage automation rather than manual syncs

• Low Overhead reduces manual and complex scripting

• Repeatable approach standardizes DBA processes

• Simple framework can be used across a variety of DBs

• All Supported Oracle Versions regardless of what you have running on premise

• Mix and match versions or patch levels

#OpenWorld 2016 20

Solution 3: Dev & Test Systems in the Cloud

On Premise

Oracle DBaaS12.1 & 12.2

10g & 11g

Make my DBA job easier and more

productive


Key Benefits for DBaaS

Powerful Cloud Solution

Regional High Availability

• When you can’t afford to have your mission critical databases take downtime

• Particularly necessary when disaster recover is necessary across regional datacenters

• Standby Databases can be always on with Active Data Guard in the Cloud

• Active-Active can enable you to take read/write transactions on all databases.

• Strong Automation of service provisioning and billing

• Oracle or Amazon Cloud operate active-active Oracle Database in any Cloud across regional data centers

#OpenWorld 2016 21

Solution 4: High Availability for DBaaS in the Cloud

Oracle DBaaS

Datacenter Region 1

Active / ActiveOracle DBaaSor Amazon RDS

Datacenter Region 2

On Premise

My business applications can not go

down!


Heterogeneous and Complete Solution

Bring Enterprise-strength to AWS

• Advanced data integration and availability capabilities for Kinesis, EMR, Redshift and RDS

• Proven data transfer rates of 10’s of thousands of DB transactions per second

• Security controls work correctly across Amazon secure data centers

• Flexible solution can leverage DB or Big Data from within the AWS services

• Most Trusted data replication engine over all others

• Top most popular ETL/ELT tool in worldwide market

• Fastest growing data integration for big data

• Oracle Data Integration is a heterogeneous capability that works across different data platforms and different cloud platforms.

#OpenWorld 2016 22

Solution 5: Make Amazon AWS Enterprise Grade

Oracle DatabaseActive-Active

Realtime Big Data

GoldenGate

Datacenter A Datacenter B

GoldenGate

Data Integrator


Big Data SolutionsFor Data Integration & Governance

PragmaticBig DataSolutions

1. Data IngestionINDUSTRIAL STRENGTH DATA INGESTION LAYER

2. TransformationsFUTURE-PROOF TOOLING FOR CODE GENERATION

3. GovernanceMETADATA LINEAGE AND BUSINESS GLOSSARY

4. ConnectorsHIGH SPEED PARALLEL LOADERS FOR BULK DATA

5. Streaming Big DataINGEST & ANALYTICS 100% IN-MEMORY


BusinessData

ServingLayer

Apps

Analytics

EDWsRawData

Staging

Data Discovery

ETL Offload

Batch Layer

Data Streams

Social and Logs

Enterprise Data

Highly AvailableDatabases

Databus(Pub/Sub)

Streaming Analytics

Streaming Data Pipelines

...Speed Layer data broadcasting & streaming data processing

...Batch Layer data processing for huge data volumes

...Serving Layer for consumption

Pub / Sub

REST APIs

NoSQL

Bulk Data

Vision = Lambda/Kappa

Databus(Pub/Sub)

Speed Layer


Hybrid Open-Source...Open Source at the core of speed & batch processing engines

...Enterprise Vendor tools for connecting to existing IT system and

...Cloud Platforms for data fabric

BusinessData

ServingLayer

Apps

Analytics

Batch Layer

Data Streams

Social and Logs

Enterprise Data


Pub / Sub

REST APIs

NoSQL

Bulk Data

Speed LayerRaw Data Stream Processing

Batch Processing

Prepared Data


Examples

#OpenWorld 2016 27

Reference Architecture

BusinessData

ServingLayer

Apps

Analytics

Batch Layer

Data Streams

Social and Logs

Enterprise Data


Pub / Sub

REST APIs

NoSQL

Bulk Data

Speed Layer

GoldenGate

Data Preparation

Data Quality, Metadata Management & Business Glossary


Active DataGuard

Comprehensive architecture covers key areas – #1. Data Ingestion, #2. Data Preparation & Transformation, #3. Streaming Big Data, #4. Parallel Connectivity, and #5. Data Governance –and Oracle Data Integration has it covered.

Dataflow ML

Stream Analytics

Connectors


ApplicationsApplications DatabusApplications Speed Layer

Batch Layer

Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp

#OpenWorld 2016 28

Streaming Analytics

Application

ServingLayer

RESTServices

VisualizationTools

ReportingTools

Data Marts

UserUpdates

DBMSUpdates

GoldenGate for Ingest

GG GG

Applications ServingLayer

Speed Layer

Batch Layer

Platforms


ODI for Transformations

ETL Engines

Big Data Frameworks

Speed Layer

Batch Layer

ServingLayer

ApplicationsApplications DatabusApplications

Application

RESTServices

VisualizationTools

ReportingTools

Data Marts

UserUpdates

DBMSUpdates


Speed Layer

Batch Layer


Spark Streaming

Spark SQLSqoop

ERP

Oozie

Pig

Hive

Loaders

Kafka

NoSQL

OGG

SQL


Data CatalogSpeed Layer

Batch Layer

ServingLayer

#OpenWorld 2016 30

OEMM for Data GovernanceApplicationsApplications DatabusApplications

Application

RESTServices

VisualizationTools

ReportingTools

Data Marts

UserUpdates

DBMSUpdates


Speed Layer

Batch Layer

KafkaGenerated Streaming

Generated ETL CodeSqoopOLTP Databases

HDFS Files

HCatalog

Hive

NoSQL

ETLTools

Data Warehouses

BI Models

ER Models

Oracle Enterprise Metadata Management

140+ Supported Tools


from Devices

Batch Layer

#OpenWorld 2016 31

Streaming Data

ApplicationsApplications

Databus

Applications

Speed Layer ServingLayer

RESTServices

VisualizationTools

ReportingTools

Data Marts

Applications

ServingLayer

Speed Layer

Batch Layer

Oracle Stream Analytics

Oracle Dataflow ML

Oracle GoldenGate

Application

ApplicationsApplicationsDevices

from Databases


World Class SolutionsFor Data Integration & Governance


140+ Open Source, Standards and Third Party Integrations

Adaptive Altova Apache HCatalog Apache Hive/HQL Borland CA ERwin Cloudera Impala COBOL Copybook DataStax Embarcadero EMC ProActivity GentleWare Google BigQuery Grandite Hadapt Hive Hortonworks Hive IBM Cognos IBM DB2 IBM DataStage IBM Discovery IBM Federation Server IBM Lotus Notes IBM Netezza IBM Rational Rose IBM Rational Architect Informatica Metadata Manager

Informatica PowerCenter CoSORT ISO SQL Standard (DDL) MapR Hadoop Hive MicroFocus Microsoft Access Microsoft Office Excel Microsoft Visio Microsoft SQL Server Microsoft SSIS Microsoft Visual Studio Microstrategy Magic Draw OMG CWM Standard OMG UML Standard Oracle BI Answers Oracle BI Enterprise Edition Oracle BI Server Oracle DAC Oracle Data Integrator Oracle Data Modeler Oracle Database Oracle Designer Oracle Hyperion Applications Oracle Hyperion Essbase Oracle Warehouse Builder

Pivotal Greenplum PostgreSQL QlikView SAP BO Crystal Reports SAP BO Designer SAP BO Desktop Intelligence SAP BO Repository SAP BO Data Integrator SAP BO Data Steward SAP Master Data Management SAP Sybase PowerDesigner SAP Sybase ASE Database SAS Data Integration Studio SAS BI Server SAS Information Map SAS Metadata Management SAS OLAP Server Select Sparx Architect Syncsort Tableau Talend Teradata Tigris Visible W3C DTD & XSD Schema

Metadata Harvesting (Glossary, Lineage & Impact Analysis)Key Standards


Data Integration Cloud

Data Integration

SaaS Data Quality

DaaS Data Quality

ODI on Big DataCloud Service

Data Preparation

ODI CloudService

Dataflow MLCloud Service

EDQ CloudService

GoldenGateCloud Service


• Use Starbucks loyalty card to buy a Latte

• Pay for a purchase with PayPal

• Withdraw cash at an ATM from Bank of America, or Citibank, or Wells Fargo, or CapitalOne, or American Express, or JP Morgan Chase, or… most other banks in the world

• Click on an Google AdWords link

• Select a recommended item from EBay.com

• Buy something from Amazon

• Use a store coupon from Safeway

• Browse the catalog from Overstock.com

• Check flight availability on American Airlines

• Use the Cartwheel App at Target stores

• Take your General Motors car in for service

• Go shopping at Macy’s online store on Black Friday

• Deposit your payroll check issued by Paychex

• Take care of your small business accounting on Quickbooks.com

• Buy a song on Apple iTunes

• Change your profile or add a skill on LinkedIn.com

Everyday Moments


Presen-tationson:

#OpenWorld 2016 36

Data Integration Solutions Program - tinyurl.com/DISOOW16

DemoStations:

Hands-on labs:

OracleEnterprise Metadata

Management

OracleEnterprise

Data Quality

Oracle GoldenGate

OracleData

Integrator

OracleBig Data

PreparationCloud Service

OracleEnterprise

Data QualityHOL7466

Oracle GoldenGateDeep DiveHOL7528

ODI and OGGfor Big Data

HOL7434

Oracle Big DataPreparation

Cloud ServiceHOL7432

MiddlewareDemoground

- Moscone South

Big Data Showcase

- Moscone South

DatabaseDemoground

- Moscone South


Data Integration Solutions Program - tinyurl.com/DISOOW16

Monday, Sept 19• Oracle Data Integration Solutions – Platform Overview and Roadmap

[CON6619 ]• Oracle Data Integration: the Foundation for Cloud Integration [CON6620 ]• A Practical Path to Enterprise Data Governance with Cummins [CON6621]• Oracle Data Integrator Product Update and Strategy [CON6622]• Deep Dive into Oracle GoldenGate 12.3 New Features for the Oracle 12.2

Database [CON6555]

Tuesday, Sept 20• Oracle Big Data Integration in the Cloud [CON7472] • Oracle Data Integration Platform: a Cornerstone for Big Data [CON6624]• Oracle Data Integrator and Oracle GoldenGate for Big Data [HOL7434]• Oracle Enterprise Data Quality – Product Overview and Roadmap

[CON6627] • Self Service Data Preparation for Domain Experts – No Programming

Required [CON6630] • Oracle Big Data Preparation Cloud Service: Self-Service Data Prep for

Business Users [HOL7432] • Oracle GoldenGate 12.3 Product Update and Strategy [CON6631] • New GoldenGate 12.3 Services Architecture [CON6551] • Meet the Experts: Oracle GoldenGate Cloud Service [MTE7119]

Wednesday, Sept 21• Data Quality for the Cloud: Enabling Cloud Applications with Trusted Data

[CON6629] • Transforming Streaming Analytical Business Intelligence to Business

Advantage [CON7352]• Oracle Enterprise Data Quality for All Types of Data [HOL7466] • Oracle GoldenGate for Big Data [CON6632] • Accelerate Cloud On-Boarding using Oracle GoldenGate Cloud Service

[CON6633] • Oracle GoldenGate Deep Dive and Oracle GoldenGate Cloud Service for Cloud

Onboarding [HOL7528]

Thursday, Sept 22• Best Practices for Migrating to Oracle Data Integrator [CON6623] • Best Practices for Oracle Data Integrator: Hear from the Experts [CON6625]• Dataflow, Machine Learning and Streaming Big Data Preparation [CON6626] • Data Governance with Oracle Enterprise Data Quality and Metadata

Management [CON6628] • Faster Design, Development and Deployment with Oracle GoldenGate Studio

[CON6634] • Getting started with Oracle GoldenGate [CON7318] • Best Practice for High Availability and Performance Tuning for Oracle

GoldenGate [CON6558]


Connect with Oracle Data Integration

@OracleDI

Blogs.oracle.com/DataIntegration/

Oracle Data Integration

Oracle Data Integration

#OpenWorld 2016 38


Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

#OpenWorld 2016 40

Oracle Data Integration - Overview

Technology

Transcript of Oracle Data Integration - Overview