Hi - Kerry Osborne’s Oracle Blogkerryosborne.oracle-guy.com/papers/Hadoop Meets Exadata 2.pdf ·...
Transcript of Hi - Kerry Osborne’s Oracle Blogkerryosborne.oracle-guy.com/papers/Hadoop Meets Exadata 2.pdf ·...
Hadoop Meets Exadata
DW Global Leaders Program
Presented by: Kerry Osborne
Hi
Decemeber, 2012
3
whoami –
Never Worked for Oracle Worked with Oracle DB Since 1982 (V2) Working with Exadata since early 2010 Work for Enkitec (www.enkitec.com) (Enkitec owns an Exadata Half Rack – V2/X2) (Enkitec owns an Oracle Big Data Appliance) Exadata Book (recently translated to Chinese)
Blog: kerryosborne.oracle-guy.com Twitter: @KerryOracleGuy
Hadoop Aficionado
4
What’s the Point?
Data Volumes are Increasing Rapidly Cost of Processing / Storing is High Scalability is Big Concern
And …
5
Hadoop Is A Virus
* Stolen from Orbitz
6
Google Trends
7
Google Trends
8
Google Trends
9
Disjointed Presentation ???
Architectures Integration Approaches Oracle Products Exadoop Case Study
10
Traditional RDBMS Architecture
DB Server
Storage
Compute work
Storage
Plumbing
11
Traditional Oracle Architecture
Cache
Storage
dbwr lgwr etc…
workers
RAC
Block Mapper (ASM)
(SGA) work
tasktracker tasktracker
12
HDFS/Hadoop Architecture
Name Node Job Tracker work
Storage
workers
datanode
Storage
workers
datanode
HA ?
tasktracker tasktracker
13
HDFS/Hadoop Architecture HA ?
Block Mapper (namenode)
Job Tracker work
Storage
workers
datanode
Storage
workers
datanode
14
Exadata Architecture RAC
Block Mapper (ASM)
Cache work
Storage
workers
Storage Node
Storage
workers
Storage Node
workers
tasktracker tasktracker
15
HDFS/Hadoop Architecture HA ?
Block Mapper (namenode)
Job Tracker work
Storage
workers
datanode
Storage
workers
datanode
16
Oracle + Hadoop Integration
17
Obligatory Marketing Slide
18
Oracle Big Data Appliance
Prebuilt Hadoop Stack in a Rack Engineered System Open Source Software Includes Cloudera Distribution
19
Oracle Big Data Appliance
20
BDA Software
21
Top Secret Feature of BDA
22
Integration Options
Many Ways to Skin the Cat
• Fuse • Sqoop • Oracle Big Data Connectors
23
Fuse – External Tables
24
Sqoop (SQL-to-Hadoop)
• Graduated from Incubator Status in March 2012 • Slower (no direct path?) • Quest has a plug-in (oraoop) • Bi-Directional
25
Oracle Big Data Connectors
Oracle Loader for Hadoop - OLH
Oracle Direct Connector for HDFS - ODCH
Oracle R Connector for Hadoop – ORHC
Oracle Data Integrator Application Adapter for Hadoop
Note:
All Connectors are One Way
26
Oracle Data Integrator Application Adapter for Hadoop
ODIAAH ?
27
Oracle R Connector for Hadoop (ORHC)
• Provides ability to pull data from Oracle RDBMS • Provides ability to pull data from HDFS • Provides access to local file system • Not really a loader tool • Most useful for analysts
28
Oracle Loader for Hadoop (OLH) • Implemented as a MapReduce job (oraloader.jar) • Saves CPU on DB Server • Can convert to Oracle datatypes • Can partition data and optionally sort it • Online – direct into Oracle tables
• Can load into Oracle via JDBC or OCI Direct Path • Offline – generate preprocessed files in HDFS (DP format)
29
Oracle Direct Connector for HDFS (ODCH)
• Uses External Tables • Fastest - 12T per hour • Can load DP files preprocessed by OLH • Allows Oracle SQL to query HDFS data • Doesn’t require loading into Oracle • Downside – uses DB CPU’s
Exadoop
30
* Mad Scientist Project
Exadoop
31
Unusual Situation! Exadata Half Rack with 4 Spare Storage Servers Company Playing with “Big Data” Technology Exadata Cells Very Similar to BDA Servers 4 Cells ≈ Mini BDA! (happy face)
Exadoop Layout
32
- Exa Compute Nodes
- Exa Storage Nodes (108TB raw)
- Hadoop Cluster (144TB raw)
Big
Fat
Pip
e
- 4 Compute Nodes
- 7 Storage Nodes (252TB)
Exa Half Rack
X X X X X X X X X X X X
Exadoop
Exadoop Applications
33
Telecom Company Call Detail Records Dumped by Switches Loaded into HDFS via Flume
Exadoop – Proposed Architecture
34
- Exa Compute
- Exa Storage
- Hadoop Cluster
SIP Server
Flume Agent
CDR HDFS
Packet Sniffer
Hbase
Error Codes
Apex App
Java App
Exadoop Applications
35
Wrap Up
36
Is Hadoop the right tool for the job?
Maybe
All the Cool Kids Are Doing It!
37
Questions? Contact Information : Kerry Osborne
[email protected] kerryosborne.oracle-guy.com
www.enkitec.com