Talk IT_ Oracle_김태완_110831
description
Transcript of Talk IT_ Oracle_김태완_110831
<Insert Picture Here>
Maximizing Return for your Data Warehouse
<Insert Picture Here>
Agenda
• Top Business Imperatives and Data
Requirements to Succeed
• Data Warehouse Basics and Challenges
• The Purpose of Data Warehouse
• Why Real-Time Data Warehouse for BI?
• Real-Time Data Integration Considerations
• Traditional Vs Real-time Data Warehouse
• Data Quality & Profiling
• Oracle Data Integration Solution
• Customer Case Studies
2
Top Business Imperatives and Data
Requirements to SucceedAccess to Timely, Trusted, and Consistent data
Operational & Analytical Business Applications
DWH / BIDataMart /
BI
Improve Decisions &RegulatoryCompliance
Mergers &
Acquisitions
Data Quality
MDM /
BI
Real-Time Data Integration
Risk
Management
CRM / Direct
Marketing
Data
Migration &
Consolidation
IT Projects
3
Why Real-Time DataWarehouse for BI?
Real-Time= Relevant and actionable information
Improved insights for operational decision making
Better customer service and cost savings via operational efficiencies
Increased profitability, customer retention, and competitive advantage
1. Business Driver:
To compete more effectively
by using better insights in
day to day operations and
strategic decisions
2. Technical Driver
To move away from batch
ETL jobs and eliminate
the impact it has on the IT
infrastructure
4
OLTP & ODS
Systems Data Warehouse, Data Mart
Oracle, PeopleSoft,
Siebel, SAP,
Custom Apps
Files, Excel
XML
Integration ChallengesFragmented Approach
Enterprise Performance
Custom Reporting
PackagedApplications
BusinessIntelligence
Analytics
Data Federation
Data Warehousing
Custom
Data Marts
Data Access
Data Silos
SQL
Java
Batch Scripts
Data Hubs
DataMigration
DataReplication
OLAP
Accessibility
High Cost of
Custom Coding
Lack of clean,
consistent data
Multiple standards,
disciplines
Data Warehouse,
Data Mart
Data Warehouse,
Data Mart
Up-To-Date
information
Trusted
Information
5
Data Warehouse and It’s Process
ER
P/
CR
M
Extract
Integrate
Data Cleansing
Data
Enrichment
Transform
Load
DM1 DM2
DM3 DM4
DATA WAREHOUSE
Analytical Reporting
Ad-hoc Reporting
Dashboards
Scorecards
Sources
RD
BM
S
A centralized repository containing comprehensive detailed andsummary data that provides a complete view of customers,suppliers, business processes, and transactions, from ahistorical perspective with little volatility.
Target
Typically loaded on a nightly basis with batch extracts from sourcetransaction processing systems such as CRM, ERP, etc to support reportingand analysis.
6
Real-Time Data
for Intelligence & Operations
• Up-to-the second data for operations
• Access to timely information for analysis
• Data distribution across regions
Real-Time
Enterprise
Business Drivers For a Modern Data Integration
Platform & Real-Time DWH
Continuous Availability
for 24/7 Global Operations
• Continuous uptime in event of disaster
• No downtime during planned outage
• Low-impact data capture for integration
Trusted Information
• Consistent with other systems
•High data integrity
Demand for Continuous, Real-Time,
Trusted Information
7
Traditional Vs Real-Time Data Warehouse
• Day+ old data
• Batch data extracts within specified
“off business hours”
• A middle-tier server for
transformations
• Process interruptions impact data
recoverability
Legacy, Packaged Apps, OLTP Databases
Enterprise Data Warehouse
Traditional Data Warehouse
ETL + CDC
Real-Time Data Warehouse
Enterprise Data
Warehouse/MDM
Transformation & Quality
(ETL / ELT &
Data Cleansing)
Legacy, Packaged Apps, OLTP
Real-Time
Continuous Feeds
• Timely, relevant data are continuous
feed from operational systems
• No batch windows on OLTP
• Sub-second latency
• No impact on source systems
• Read-consistent changed data with
referential integrity
• Transformations at capture, delivery or
within the database
8
Oracle Complete Data Integration
Solution
Oracle Data Quality
& Profiling
Trusted Information
• Discover data
problems
• Global data cleansing
• Data standardization
• Fuzzy matching
• Heterogeneous
Oracle GoldenGate
Up-to-Date Information
• Real-Time data
capture and delivery
• Low impact and non-
invasive
• Transactional integrity
• Guaranteed delivery
• Delivers continuous
availability
• Heterogeneous
Oracle Data Integrator
• E-LT
• Bulk data movement
• Complex
transformation
• Easy to use GUI
design
• Data lineage & impact
analysis
• Heterogeneous
Accessibility
Oracle Data Integration: 3 Key ProductsAddressing operational and strategic analysis
10
Declarative Set-based
design
Change Data Capture for
Dynamic Updates
E-LT Transformation
vs. E-T-L
Hot-pluggable Architecture
Any Data Warehouse
Any Planning System
OLTP DB Sources
Application Sources
Legacy Sources
Pluggable Knowledge
Modules
Oracle Data Integrator Enterprise EditionOptimized E-LT for Improved Performance
11
Data Warehouse Bulk Loading w/E-LTFastest ELT Solution for your Data Warehouse
Solution
• ODI for bulk loading Data Warehouse
• Run ODI Agent within Data
Warehouse JVM
• Fastest possible data
transformations
• Heterogeneous and loads any 3rd
party data warehouse.
Benefits
• No extra ETL servers
• RDBMS Specific Knowledge
Modules
• Exploit DW RDBMS Optimizer
• Easier to deploy than conventional
ETL tools
• Faster time to market
• Enforce DW Best Practices
Extract
Load
Transform
12
• Massively parallel high volume hardware to quickly process vast amounts of data
• Exadata runs data intensive processing directly in storage
• Most complete analytic capabilities• OLAP, Statistics, Spatial, Data Mining, Real-time
transactional ETL, Efficient point queries
• Powerful warehouse specific optimizations• Flexible Partitioning, Bitmap Indexing, Join indexing,
Materialized Views, Result Cache
• E-LT runs 20X faster only with OracleData Mining
OLAP
ELT
New
Optimized for ExadataFastest E-L-T Processing
13
ODI is FasterUp to 7TB per hour of real world data loading and complex transformations
ODI ELT (on Exadata)
ODI scales with Exadata
ODI runs on Exadata – no ETL hardware required
Common administration, monitoring and
management
All the benefits of rapid tools-based ETL
development
Conventional ETL
As data sets grow additional hardware ($$) needed
ETL parallel optimization and design ($$$) is heavily
dependent on resources available to the ETL system
Poor performance – transformations take place
outside of database, require staging tables
Lack of light-weight architecture for rapid data
loading
ETL engine hardware resources only used for ETL
Hardware not co located, multiple vendors
Different management, monitoring and administration
from database and BI infrastructure ($$)
14
Differentiator: E-LT ArchitectureHigh Performance
Conventional ETL Architecture
Extract LoadTransform
Next Generation Architecture
“E-LT”LoadExtract
Transform Transform
Conventional: Separate ETL Server• Proprietary ETL Engine
• Poor Performance
• High Costs for Separate Standalone Server
Oracle: No New Servers
• Lower Cost: Leverage Compute Resources & Partition Workload efficiently
• Efficient: Exploits Database Optimizer
• Fast: Exploits Native Bulk Load & Other Database Interfaces
• Scalable: Scales as you add Processors to Source or Target
BenefitsOptimal Performance & Scalability
Better Hardware Leverage
Easier to Manage & Lower Cost
15
Metadata Profiling of Source & Target
Duplicate Detection, Matching & Merging
Global Address Cleansing
Data Control and Visibility
Optimized for Customer/Party Data
Oracle Data Profiling and Quality Integrated Data Profiling and Quality for Customer Data
Any Data
Warehouse
&
MDM
Any Planning System
Oracle Data Integrator Enterprise Edition
Oracle Data Profiling, Oracle Data Quality
Any Source System
16
Oracle Data QualityEnsure Data Quality as Part of the Integration Process
ODI EE
Oracle Data Profiling Oracle Data Quality
Better Data Visibility
Tighter Data Control
Greater Data Accuracy
Metadata Profiling of Source & Target
Duplicate Detection, Matching & Merging
Visual Data Quality Tools
Best-in-class data quality and profiling for
integration processes
17
Sample Data Quality Issues
COMPLETENESS
CONFORMITY
CONSISTENCY
DUPLICATION
ACCURACYAccuracy
What data is
incorrect or out of
date?
Duplication
What data records
or attributes are
repeated?
Conformity
What data is stored
in a non-standard
format?
Completeness
What data is
missing or
unusable?
Consistency
What data values
give conflicting
information?
18
Oracle GoldenGate Enterprise-wide Solution for Real Time Data Needs
Log Based, Real-
Time Change Data
Capture
Heterogeneous
Source Systems
EDWODS
EDW
Active-Active High
Availability
Zero Downtime
Migration and
Upgrades
Real-time BI
Fully Active
Distributed Database
Reporting
Database
ETL
ETL
Query Offloading
Data Distribution
New DB/
OS/HW/App
Global Data Centers
SOA/EDA
Oracle
GoldenGate
• Reduce Costs
• Lower Risks
• Achieve Operational Excellence
19
Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in
real time
Key Differentiators:
Non-intrusive, low-impact, sub-second latency
Open, modular architecture - Supports heterogeneous sources and targets
Maintains transactional integrity - Resilient against interruptions and failures
Data Movement plus Real-Time
Performance
Flexible and Extensible
Reliable
20
Oracle GoldenGate ArchitectureDesigned for Speed, Flexibility and Reliability
LAN/WAN
Internet
TCP/IP
Bi-directional
CaptureTrail
DeliveryTrail
Source
Oracle & Non-Oracle
Database(s)
Target
Oracle & Non-Oracle
Database(s)
• Real-time change data capture, routing and delivery across
heterogeneous systems
• Non-invasive , log-based change data capture for minimal impact on
infrastructure
• Transactional integrity and guaranteed data delivery across regions
• Bi-directional replication and support for different replication topologies
21
Oracle Complete Data Integration Solution UsageComplete, Flexible, Integrated
CRM Apps
SCM Apps
Operations
ERP Apps
BI Reporting Tool
Oracle GoldenGate
Oracle Exadataor Teradata
or Netezza, etc
Source OLTP
Database
Real-Time Feed
Historic
Analytics
Real-time
Analytics
Batch or
Real-Time Feed
ODS
Staging
ODI/ODP/ODQ
EDW
Transform
& Cleanse
DR, Query, Reporting
Oracle GoldenGate
Financials
22
Customer Case Studies
Customer Example
Challenges & Objectives Solution & Benefits
Need to enable sales, finance, marketing and
merchandising teams with near real-time data
Oracle GoldenGate captures real-time change data
from ecommerce and auction systems. Oracle Data
Integrator is used for highly complex transformations
and data loading to user tables
Required to have business insight on company
performance meeting target metrics
Resulted in updated, current Teradata data warehouse providing critical business intelligence for decision making
Need to be able to handle high-volume data
loading and transformation requirements like
1.2M+ SKUs, 5M+ daily transactions, 300+
users
Complete, accurate data to give LOBs a trusted view of
business progress, etc.
Leading online retailer offering a wide variety of high-quality, brand-
name merchandise at discount prices and manufacturers, distributors
and other retailers an alternative sales channel.
24
350,000 sq. ft.
Fulfillment
Warehouse
User
Friendly
Web Front End
DemandSupply
Consumers
Small
Businesses
Distributors
Manufacturers
Overstock.com
• Information-sensitive business
• Demanding Business Community
• Pricing, Forecasting
• Rapid Growth
25
Overstock.comInnovations in Data Warehousing
Traditional Framework
• Batch extracts/feeds from
operational systems
• Transformations in ETL engine on
the middle tier
• Bulk load to the data warehouse
• Large nightly batch, user online day
Emerging Framework
• Near real-time feeds from
operational systems
• Thin middle tier with E-LT
• Transformations on the database
platform
• Small mini-batches throughout the
day
26
Overstock.comEnterprise Solution for Enhanced Intelligence
27
Overstock.com: Innovations in Data IntegrationSpeed translates to Improved Opportunities
• Batch windows nearly eliminated
• Low-latency data provides new opportunities
• Trigger/event campaigns
• Personalization, scoring, ranking
• Marketing and merchandising improvements
• Improved Customer Service
• Now ranked #2 in the nation
• Operational improvements in scoring and
logistics led to amazing results
• Upgraded Oracle 9i Database without business
interruption
28
Lifetouch Portrait Studios Inc• Serve the portraiture needs of 2M Guests annually
• 730 Photographic studios across the country
We preserve memories
and help our guest
tell their unique story…
FLASH! Digital Portraits
The Studio at Target
JCPenney Portraits
29
Oracle Solution
Implemented Oracle Data Quality for
Data Integrator to cleanse customer
data.
Combined Oracle solution with
Oracle Database, MySQL and Oracle
Data Integrator for every data
movement and the overall
orchestration of the process
Return on Investment
Improved customer data quality by a
minimum of 25%
Studio workflow performance
improved by 10% resulting in higher
customer satisfaction and retention
Accurate and complete customer data
into data warehouse enabled better
customer segmentation and targeted
marketing campaign
Reduced the time and budget required
to implement data quality processes.
Business Challenges
Implement a single, high-quality and consistent view of each
customer to be available throughout the organization
Streamline the sharing of customer data across all the
customer facing applications
Improve the accuracy of customer data to provide a better
visibility and to elevate the customer relationships.
Better Customer Insight with Oracle Data
Quality
30
Benefits
• Improved customer service with real time guest and appointment
data enabling improved Studio efficiencies
• Studio workflow performance improvement of 10%
• Higher guest satisfaction and retention
• Accurate and complete guest data in the Data Warehouse
enabling
• Improved guest segmentation
• Improved targeted marketing campaigns
• Cleansed and unique guest profiles shared across primary
applications resulting in a minimum of 25% improved data quality.
• Expected reduction of technical resource hours in implementing
custom code, and performing manual data quality implementation
and audits.
• Time and budget saved with implementing ODI/ODQ solution vs.
building functionality internally.
31
Benefits of Real-Time DW with OracleCut Costs, Reduce Risk, and Revolutionize Business Insight
• Move only the changed data from redo logs and reduce source and network overhead
• Shorten implementation times from months to weeks using pre-packaged integrations to well-known applications, sources and targets.
Cut Costs and Improve Efficiencies.
• Eliminate performance impact on source systems
• Reduce the risk of missed orders, poor customer interactions, missed opportunities through improved recoverability, data quality
Reduce Risk, Ensure Continuity
• Enable near real-time decision making with real-time data flows
• Combine real-time data with historical context for better insights
Improve Business Insight
32
Questions