Talk IT_ Oracle_김태완_110831

33
<Insert Picture Here> Maximizing Return for your Data Warehouse

description

 

Transcript of Talk IT_ Oracle_김태완_110831

Page 1: Talk IT_ Oracle_김태완_110831

<Insert Picture Here>

Maximizing Return for your Data Warehouse

Page 2: Talk IT_ Oracle_김태완_110831

<Insert Picture Here>

Agenda

• Top Business Imperatives and Data

Requirements to Succeed

• Data Warehouse Basics and Challenges

• The Purpose of Data Warehouse

• Why Real-Time Data Warehouse for BI?

• Real-Time Data Integration Considerations

• Traditional Vs Real-time Data Warehouse

• Data Quality & Profiling

• Oracle Data Integration Solution

• Customer Case Studies

2

Page 3: Talk IT_ Oracle_김태완_110831

Top Business Imperatives and Data

Requirements to SucceedAccess to Timely, Trusted, and Consistent data

Operational & Analytical Business Applications

DWH / BIDataMart /

BI

Improve Decisions &RegulatoryCompliance

Mergers &

Acquisitions

Data Quality

MDM /

BI

Real-Time Data Integration

Risk

Management

CRM / Direct

Marketing

Data

Migration &

Consolidation

IT Projects

3

Page 4: Talk IT_ Oracle_김태완_110831

Why Real-Time DataWarehouse for BI?

Real-Time= Relevant and actionable information

Improved insights for operational decision making

Better customer service and cost savings via operational efficiencies

Increased profitability, customer retention, and competitive advantage

1. Business Driver:

To compete more effectively

by using better insights in

day to day operations and

strategic decisions

2. Technical Driver

To move away from batch

ETL jobs and eliminate

the impact it has on the IT

infrastructure

4

Page 5: Talk IT_ Oracle_김태완_110831

OLTP & ODS

Systems Data Warehouse, Data Mart

Oracle, PeopleSoft,

Siebel, SAP,

Custom Apps

Files, Excel

XML

Integration ChallengesFragmented Approach

Enterprise Performance

Custom Reporting

PackagedApplications

BusinessIntelligence

Analytics

Data Federation

Data Warehousing

Custom

Data Marts

Data Access

Data Silos

SQL

Java

Batch Scripts

Data Hubs

DataMigration

DataReplication

OLAP

Accessibility

High Cost of

Custom Coding

Lack of clean,

consistent data

Multiple standards,

disciplines

Data Warehouse,

Data Mart

Data Warehouse,

Data Mart

Up-To-Date

information

Trusted

Information

5

Page 6: Talk IT_ Oracle_김태완_110831

Data Warehouse and It’s Process

ER

P/

CR

M

Extract

Integrate

Data Cleansing

Data

Enrichment

Transform

Load

DM1 DM2

DM3 DM4

DATA WAREHOUSE

Analytical Reporting

Ad-hoc Reporting

Dashboards

Scorecards

Sources

RD

BM

S

A centralized repository containing comprehensive detailed andsummary data that provides a complete view of customers,suppliers, business processes, and transactions, from ahistorical perspective with little volatility.

Target

Typically loaded on a nightly basis with batch extracts from sourcetransaction processing systems such as CRM, ERP, etc to support reportingand analysis.

6

Page 7: Talk IT_ Oracle_김태완_110831

Real-Time Data

for Intelligence & Operations

• Up-to-the second data for operations

• Access to timely information for analysis

• Data distribution across regions

Real-Time

Enterprise

Business Drivers For a Modern Data Integration

Platform & Real-Time DWH

Continuous Availability

for 24/7 Global Operations

• Continuous uptime in event of disaster

• No downtime during planned outage

• Low-impact data capture for integration

Trusted Information

• Consistent with other systems

•High data integrity

Demand for Continuous, Real-Time,

Trusted Information

7

Page 8: Talk IT_ Oracle_김태완_110831

Traditional Vs Real-Time Data Warehouse

• Day+ old data

• Batch data extracts within specified

“off business hours”

• A middle-tier server for

transformations

• Process interruptions impact data

recoverability

Legacy, Packaged Apps, OLTP Databases

Enterprise Data Warehouse

Traditional Data Warehouse

ETL + CDC

Real-Time Data Warehouse

Enterprise Data

Warehouse/MDM

Transformation & Quality

(ETL / ELT &

Data Cleansing)

Legacy, Packaged Apps, OLTP

Real-Time

Continuous Feeds

• Timely, relevant data are continuous

feed from operational systems

• No batch windows on OLTP

• Sub-second latency

• No impact on source systems

• Read-consistent changed data with

referential integrity

• Transformations at capture, delivery or

within the database

8

Page 9: Talk IT_ Oracle_김태완_110831

Oracle Complete Data Integration

Solution

Page 10: Talk IT_ Oracle_김태완_110831

Oracle Data Quality

& Profiling

Trusted Information

• Discover data

problems

• Global data cleansing

• Data standardization

• Fuzzy matching

• Heterogeneous

Oracle GoldenGate

Up-to-Date Information

• Real-Time data

capture and delivery

• Low impact and non-

invasive

• Transactional integrity

• Guaranteed delivery

• Delivers continuous

availability

• Heterogeneous

Oracle Data Integrator

• E-LT

• Bulk data movement

• Complex

transformation

• Easy to use GUI

design

• Data lineage & impact

analysis

• Heterogeneous

Accessibility

Oracle Data Integration: 3 Key ProductsAddressing operational and strategic analysis

10

Page 11: Talk IT_ Oracle_김태완_110831

Declarative Set-based

design

Change Data Capture for

Dynamic Updates

E-LT Transformation

vs. E-T-L

Hot-pluggable Architecture

Any Data Warehouse

Any Planning System

OLTP DB Sources

Application Sources

Legacy Sources

Pluggable Knowledge

Modules

Oracle Data Integrator Enterprise EditionOptimized E-LT for Improved Performance

11

Page 12: Talk IT_ Oracle_김태완_110831

Data Warehouse Bulk Loading w/E-LTFastest ELT Solution for your Data Warehouse

Solution

• ODI for bulk loading Data Warehouse

• Run ODI Agent within Data

Warehouse JVM

• Fastest possible data

transformations

• Heterogeneous and loads any 3rd

party data warehouse.

Benefits

• No extra ETL servers

• RDBMS Specific Knowledge

Modules

• Exploit DW RDBMS Optimizer

• Easier to deploy than conventional

ETL tools

• Faster time to market

• Enforce DW Best Practices

Extract

Load

Transform

12

Page 13: Talk IT_ Oracle_김태완_110831

• Massively parallel high volume hardware to quickly process vast amounts of data

• Exadata runs data intensive processing directly in storage

• Most complete analytic capabilities• OLAP, Statistics, Spatial, Data Mining, Real-time

transactional ETL, Efficient point queries

• Powerful warehouse specific optimizations• Flexible Partitioning, Bitmap Indexing, Join indexing,

Materialized Views, Result Cache

• E-LT runs 20X faster only with OracleData Mining

OLAP

ELT

New

Optimized for ExadataFastest E-L-T Processing

13

Page 14: Talk IT_ Oracle_김태완_110831

ODI is FasterUp to 7TB per hour of real world data loading and complex transformations

ODI ELT (on Exadata)

ODI scales with Exadata

ODI runs on Exadata – no ETL hardware required

Common administration, monitoring and

management

All the benefits of rapid tools-based ETL

development

Conventional ETL

As data sets grow additional hardware ($$) needed

ETL parallel optimization and design ($$$) is heavily

dependent on resources available to the ETL system

Poor performance – transformations take place

outside of database, require staging tables

Lack of light-weight architecture for rapid data

loading

ETL engine hardware resources only used for ETL

Hardware not co located, multiple vendors

Different management, monitoring and administration

from database and BI infrastructure ($$)

14

Page 15: Talk IT_ Oracle_김태완_110831

Differentiator: E-LT ArchitectureHigh Performance

Conventional ETL Architecture

Extract LoadTransform

Next Generation Architecture

“E-LT”LoadExtract

Transform Transform

Conventional: Separate ETL Server• Proprietary ETL Engine

• Poor Performance

• High Costs for Separate Standalone Server

Oracle: No New Servers

• Lower Cost: Leverage Compute Resources & Partition Workload efficiently

• Efficient: Exploits Database Optimizer

• Fast: Exploits Native Bulk Load & Other Database Interfaces

• Scalable: Scales as you add Processors to Source or Target

BenefitsOptimal Performance & Scalability

Better Hardware Leverage

Easier to Manage & Lower Cost

15

Page 16: Talk IT_ Oracle_김태완_110831

Metadata Profiling of Source & Target

Duplicate Detection, Matching & Merging

Global Address Cleansing

Data Control and Visibility

Optimized for Customer/Party Data

Oracle Data Profiling and Quality Integrated Data Profiling and Quality for Customer Data

Any Data

Warehouse

&

MDM

Any Planning System

Oracle Data Integrator Enterprise Edition

Oracle Data Profiling, Oracle Data Quality

Any Source System

16

Page 17: Talk IT_ Oracle_김태완_110831

Oracle Data QualityEnsure Data Quality as Part of the Integration Process

ODI EE

Oracle Data Profiling Oracle Data Quality

Better Data Visibility

Tighter Data Control

Greater Data Accuracy

Metadata Profiling of Source & Target

Duplicate Detection, Matching & Merging

Visual Data Quality Tools

Best-in-class data quality and profiling for

integration processes

17

Page 18: Talk IT_ Oracle_김태완_110831

Sample Data Quality Issues

COMPLETENESS

CONFORMITY

CONSISTENCY

DUPLICATION

ACCURACYAccuracy

What data is

incorrect or out of

date?

Duplication

What data records

or attributes are

repeated?

Conformity

What data is stored

in a non-standard

format?

Completeness

What data is

missing or

unusable?

Consistency

What data values

give conflicting

information?

18

Page 19: Talk IT_ Oracle_김태완_110831

Oracle GoldenGate Enterprise-wide Solution for Real Time Data Needs

Log Based, Real-

Time Change Data

Capture

Heterogeneous

Source Systems

EDWODS

EDW

Active-Active High

Availability

Zero Downtime

Migration and

Upgrades

Real-time BI

Fully Active

Distributed Database

Reporting

Database

ETL

ETL

Query Offloading

Data Distribution

New DB/

OS/HW/App

Global Data Centers

SOA/EDA

Oracle

GoldenGate

• Reduce Costs

• Lower Risks

• Achieve Operational Excellence

19

Page 20: Talk IT_ Oracle_김태완_110831

Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in

real time

Key Differentiators:

Non-intrusive, low-impact, sub-second latency

Open, modular architecture - Supports heterogeneous sources and targets

Maintains transactional integrity - Resilient against interruptions and failures

Data Movement plus Real-Time

Performance

Flexible and Extensible

Reliable

20

Page 21: Talk IT_ Oracle_김태완_110831

Oracle GoldenGate ArchitectureDesigned for Speed, Flexibility and Reliability

LAN/WAN

Internet

TCP/IP

Bi-directional

CaptureTrail

DeliveryTrail

Source

Oracle & Non-Oracle

Database(s)

Target

Oracle & Non-Oracle

Database(s)

• Real-time change data capture, routing and delivery across

heterogeneous systems

• Non-invasive , log-based change data capture for minimal impact on

infrastructure

• Transactional integrity and guaranteed data delivery across regions

• Bi-directional replication and support for different replication topologies

21

Page 22: Talk IT_ Oracle_김태완_110831

Oracle Complete Data Integration Solution UsageComplete, Flexible, Integrated

CRM Apps

SCM Apps

Operations

ERP Apps

BI Reporting Tool

Oracle GoldenGate

Oracle Exadataor Teradata

or Netezza, etc

Source OLTP

Database

Real-Time Feed

Historic

Analytics

Real-time

Analytics

Batch or

Real-Time Feed

ODS

Staging

ODI/ODP/ODQ

EDW

Transform

& Cleanse

DR, Query, Reporting

Oracle GoldenGate

Financials

22

Page 23: Talk IT_ Oracle_김태완_110831

Customer Case Studies

Page 24: Talk IT_ Oracle_김태완_110831

Customer Example

Challenges & Objectives Solution & Benefits

Need to enable sales, finance, marketing and

merchandising teams with near real-time data

Oracle GoldenGate captures real-time change data

from ecommerce and auction systems. Oracle Data

Integrator is used for highly complex transformations

and data loading to user tables

Required to have business insight on company

performance meeting target metrics

Resulted in updated, current Teradata data warehouse providing critical business intelligence for decision making

Need to be able to handle high-volume data

loading and transformation requirements like

1.2M+ SKUs, 5M+ daily transactions, 300+

users

Complete, accurate data to give LOBs a trusted view of

business progress, etc.

Leading online retailer offering a wide variety of high-quality, brand-

name merchandise at discount prices and manufacturers, distributors

and other retailers an alternative sales channel.

24

Page 25: Talk IT_ Oracle_김태완_110831

350,000 sq. ft.

Fulfillment

Warehouse

User

Friendly

Web Front End

DemandSupply

Consumers

Small

Businesses

Distributors

Manufacturers

Overstock.com

• Information-sensitive business

• Demanding Business Community

• Pricing, Forecasting

• Rapid Growth

25

Page 26: Talk IT_ Oracle_김태완_110831

Overstock.comInnovations in Data Warehousing

Traditional Framework

• Batch extracts/feeds from

operational systems

• Transformations in ETL engine on

the middle tier

• Bulk load to the data warehouse

• Large nightly batch, user online day

Emerging Framework

• Near real-time feeds from

operational systems

• Thin middle tier with E-LT

• Transformations on the database

platform

• Small mini-batches throughout the

day

26

Page 27: Talk IT_ Oracle_김태완_110831

Overstock.comEnterprise Solution for Enhanced Intelligence

27

Page 28: Talk IT_ Oracle_김태완_110831

Overstock.com: Innovations in Data IntegrationSpeed translates to Improved Opportunities

• Batch windows nearly eliminated

• Low-latency data provides new opportunities

• Trigger/event campaigns

• Personalization, scoring, ranking

• Marketing and merchandising improvements

• Improved Customer Service

• Now ranked #2 in the nation

• Operational improvements in scoring and

logistics led to amazing results

• Upgraded Oracle 9i Database without business

interruption

28

Page 29: Talk IT_ Oracle_김태완_110831

Lifetouch Portrait Studios Inc• Serve the portraiture needs of 2M Guests annually

• 730 Photographic studios across the country

We preserve memories

and help our guest

tell their unique story…

FLASH! Digital Portraits

The Studio at Target

JCPenney Portraits

29

Page 30: Talk IT_ Oracle_김태완_110831

Oracle Solution

Implemented Oracle Data Quality for

Data Integrator to cleanse customer

data.

Combined Oracle solution with

Oracle Database, MySQL and Oracle

Data Integrator for every data

movement and the overall

orchestration of the process

Return on Investment

Improved customer data quality by a

minimum of 25%

Studio workflow performance

improved by 10% resulting in higher

customer satisfaction and retention

Accurate and complete customer data

into data warehouse enabled better

customer segmentation and targeted

marketing campaign

Reduced the time and budget required

to implement data quality processes.

Business Challenges

Implement a single, high-quality and consistent view of each

customer to be available throughout the organization

Streamline the sharing of customer data across all the

customer facing applications

Improve the accuracy of customer data to provide a better

visibility and to elevate the customer relationships.

Better Customer Insight with Oracle Data

Quality

30

Page 31: Talk IT_ Oracle_김태완_110831

Benefits

• Improved customer service with real time guest and appointment

data enabling improved Studio efficiencies

• Studio workflow performance improvement of 10%

• Higher guest satisfaction and retention

• Accurate and complete guest data in the Data Warehouse

enabling

• Improved guest segmentation

• Improved targeted marketing campaigns

• Cleansed and unique guest profiles shared across primary

applications resulting in a minimum of 25% improved data quality.

• Expected reduction of technical resource hours in implementing

custom code, and performing manual data quality implementation

and audits.

• Time and budget saved with implementing ODI/ODQ solution vs.

building functionality internally.

31

Page 32: Talk IT_ Oracle_김태완_110831

Benefits of Real-Time DW with OracleCut Costs, Reduce Risk, and Revolutionize Business Insight

• Move only the changed data from redo logs and reduce source and network overhead

• Shorten implementation times from months to weeks using pre-packaged integrations to well-known applications, sources and targets.

Cut Costs and Improve Efficiencies.

• Eliminate performance impact on source systems

• Reduce the risk of missed orders, poor customer interactions, missed opportunities through improved recoverability, data quality

Reduce Risk, Ensure Continuity

• Enable near real-time decision making with real-time data flows

• Combine real-time data with historical context for better insights

Improve Business Insight

32

Page 33: Talk IT_ Oracle_김태완_110831

Questions