Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing...
Transcript of Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing...
![Page 1: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/1.jpg)
1 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Evolving To The Big Data Warehouse
Kevin Lancaster
Director, Engineered Systems, Oracle EMEA
![Page 2: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/2.jpg)
2 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Big Data In Action
ANALYZE
DECIDE ACQUIRE
ORGANIZE
Make Better Decisions Using Big Data
![Page 3: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/3.jpg)
Oracle Integrated Solution Stack for All Data
ACQUIRE
Oracle NoSQL
Database
HDFS
Enterprise
Applications
ORGANIZE
Hadoop (MapReduce)
Oracle Loader for Hadoop
Oracle Data Integrator
DECIDE
Analytic
Applications
ANALYZE
In-D
ata
base
An
aly
tics
Data
Warehouse
![Page 4: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/4.jpg)
Oracle Integrated Solution Stack Oracle Engineered Systems for All Data Analytics
ACQUIRE ORGANIZE ANALYZE DECIDE
![Page 5: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/5.jpg)
5
EXALYTICS IN-MEMORY BI MACHINE
![Page 6: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/6.jpg)
6
BIG DATA APPLIANCE
![Page 7: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/7.jpg)
Oracle NoSQL Database
Key value pair database Dynamic data model Highly scalable, available Transparent load balancing Built using BerkeleyDB
Nodes East
Nodes West
Nodes Central
Nodes
NoSQL Driver
Application
NoSQL Driver
Application
… Nodes
…
Rea
d
Dele
te
Rea
d
Upd
ate
![Page 8: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/8.jpg)
Hadoop Architecture
Management/Monitoring
Hadoop Distributed File System (HDFS)
MapReduce
Distributed file system with redundant storage Map/Reduce programming paradigm Highly scalable data processing Cost-effective model for high volume, low density data
![Page 9: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/9.jpg)
A Map/Reduce Pipeline
SHUFFLE
/SORT
SHUFFLE
/SORT
MAP
MAP
MAP
MAP
SHUFFLE
/SORT
REDUCE
REDUCE
SHUFFLE
/SORT
SHUFFLE
/SORT
REDUCE
REDUCE
REDUCE
INPUT 2
INPUT 1
OUTPUT 2
OUTPUT 1
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
![Page 10: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/10.jpg)
•Oracle Data Integrator Application Adapter for Hadoop
•Oracle Loader for Hadoop
•Oracle Direct Connector for Hadoop Distributed File System
•Oracle R Connector for Hadoop
Oracle Big Data Connectors
![Page 11: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/11.jpg)
Oracle Data Integrator
Reduces Hadoop complexities through graphical tooling
![Page 12: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/12.jpg)
Oracle Loader for Hadoop
SHUFFLE
/SORT
SHUFFLE
/SORT
MAP
MAP
MAP
MAP
SHUFFLE
/SORT
REDUCE
REDUCE
SHUFFLE
/SORT
SHUFFLE
/SORT
REDUCE
REDUCE
REDUCE
INPUT 2
INPUT 1
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
![Page 13: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/13.jpg)
13
EXADATA DATABASE MACHINE
![Page 14: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/14.jpg)
14
The World’s Best Platform for Data Warehousing
• Oracle Database: #1 for DW, for decades
• Oracle Exadata: #1 Platform for the #1 Database
![Page 15: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/15.jpg)
15 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
• World record performance for fast access to information
• World-class security
• The most compete in-database analytics capability
Real Application Clusters
Advanced Compression
Partitioning
Advance Security & Firewall
In-Database Advanced Analytics (Data Mining + R Enterprise)
In-Database Multidimensional OLAP
In-Database Semantics, Text Mining, Spatial, Statistics...
Oracle Database 11g The Best Database for Data Warehousing
![Page 16: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/16.jpg)
16 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
ETL
OLAP Data Mining
OLAP Data Mining
ETL
Data Marts
Data Marts
Challenge: No Single Source of Truth Expensive Data Warehouse Architecture
![Page 17: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/17.jpg)
17 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Database 11g
Oracle Exadata Database Machine
Data
Marts
Data Mining
Online
Analytics ETL
Consolidate Onto a Single Platform Faster Performance, Single Source of Truth
![Page 18: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/18.jpg)
18 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
• Complex and predictive analytics embedded into Oracle Database 11g
• Reduce cost of additional hardware, management resources
• Even more performance by eliminating data movement and duplication, specialist data structures
Oracle Data Mining & Enterprise R Uncover and predict
Oracle OLAP Analyze and summarize
Built-In Analytics Secure, Scalable Platform for Advanced Analytics
![Page 19: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/19.jpg)
19 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Partition for Performance Partition Pruning
What was the total sales amount for May 20 and May 21 2010?
Select sum(sales_amount)
From SALES
Where sales_date between
to_date(‘05/20/2011’,’MM/DD/YYYY’)
And
to_date(‘05/22/2011’,’MM/DD/YYYY’);
5/20
5/21
5/22
5/19
Sales Table
• Performs operations only on relevant partitions
• Dramatically reduces amount of data retrieved from disk
• Improves query performance and optimizes resource utilization
![Page 20: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/20.jpg)
20 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
• Improve query performance by 10x
– Better insight into customer requirements
– Expand revenue opportunities
• Consolidate OLTP and analytic workloads
– Lower admin and maintenance costs
– Reduce points of failure
• Integrate analytics and data mining
– Complex and predictive analytics
• Lower risk
– Streamline deployment
– One support contact
Oracle Exadata Database Machine For OLTP, Data Warehousing & Consolidated Workloads
![Page 21: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/21.jpg)
21 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Database Server Pool
• 8 2-processor Database Servers
– 96 CPU Cores
– 768 GB Memory
– Oracle Linux or Solaris 11 Express
Exadata Storage Server Pool
• 14 Storage Servers
– 5 TB Smart Flash Cache
– 504 TB Disk Storage
Unified Server/Storage Network
• 40 Gb/sec Infiniband Links
Available in full, half, quarter racks
Oracle Exadata Database Machine Family Oracle Exadata Database Machine X2-2
![Page 22: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/22.jpg)
22 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Database Server Pool
• 2 8-processor Database Servers
– 160 CPU Cores
– 4 TB Memory
– Oracle Linux or Solaris 11 Express
Exadata Storage Server Pool
• 14 Storage Servers
– 5 TB Smart Flash Cache
– 504 TB Disk Storage
Unified Server/Storage Network
• 40 Gb/sec Infiniband Links
Full and multi-rack configuration
Oracle Exadata Database Machine Family Oracle Exadata Database Machine X2-8
![Page 23: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/23.jpg)
23 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Select sum(sales) where salesdate= ‘22-Jan-2011’…
Sum
Return Sales for Jan 22 2011
Exadata Smart Scan Improve Query Performance by 10x or More
What Were Yesterday’s
Sales?
• Data intensive processing runs in Exadata Storage Servers
• Rows and columns filtered as data streams from disks
• Complex operations also run in storage
• Parallelize query execution and removes bottlenecks
![Page 24: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/24.jpg)
24 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
• Full rack has 5 TB of Smart Flash Cache
• Can process over 1.5 million IOs per second
• 50 GB/sec query throughput on uncompressed data
• 5x more I/Os than 1000 Disk Enterprise Storage Array
Infrequently Used Data
Frequently Used Data
Exadata Smart Flash Cache Extreme Performance for Random Access & Database OLAP Cubes
![Page 25: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/25.jpg)
25 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Exadata Hybrid Columnar Compression Reduce Disk Space Requirements
3x
10x 15x
1.4x
2.5 x
Uncompressed Data
Data Warehouse Appliances
OLTP Data
DW Data
Archive Data
Oracle
![Page 26: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/26.jpg)
26 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Exadata Momentum Rapid Adoption in All Geographies and Industries
![Page 27: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/27.jpg)
27 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Exadata for Data Warehousing
![Page 28: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/28.jpg)
28 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
45%
43%
35%
31%
25%
16%
14%
13%
12%
11%
11%
9%
8%
3%
Data Warehouse Performance
Maintenance and Admin Costs
Data Integration
Maintaining/Updating Operational/Admin Skills
Purchasing and Installation Costs
Inability and Cost to Scale to Support Growing …
Can't Support Real-Time Data Warehousing
Inadequate High Availability
Can't Support Advanced Analytics
Inadequate Security
No Major Issues Encountered
Can't Support Mixed Workloads
Don't Know/Unsure
Other
Source: A New Dimension to Data Warehousing, 2011 IOUG Data Warehousing Survey
Challenge: User Requirements Leading Data Warehouse Challenges
![Page 29: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/29.jpg)
29 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
![Page 30: Evolving To The Big Data Warehouse - Citia BTC · The World’s Best Platform for Data Warehousing •Oracle Database: #1 for DW, for decades •Oracle Exadata: #1 Platform for the](https://reader034.fdocuments.net/reader034/viewer/2022051801/5ae4820b7f8b9a90138f1125/html5/thumbnails/30.jpg)