Deeper into the elephant: a whirlwind tour of Hadoop ecosystem

Roman Shaposhnik Director of Open Source @Pivotal

(Twitter: @rhatr)

Who’s this guy?

•  Director of Open Source, building a team of OS contributors

•  Apache Software Foundation guy (VP of Apache Incubator, ASF member, committer on Hadoop, Giraph, Sqoop, etc)

•  Used to be root@Cloudera

•  Used to be PHB@Yahoo! (original Hadoop team)

•  Used to be a hacker at Sun microsystems (Sun Studio compilers and tools)

Agenda

&

Agenda

Long, long time ago…

HDFS

ASF Projects FLOSS Projects Pivotal Products

MapReduce

In a blink of an eye:

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

GemFire XD

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

Spark

Shark

Streaming

MLib

GraphX

Impala

HAW

Q

SpringXD

MADlib

Ham

ster

PivotalR

YARN

Genesis of Hadoop

• Google papers on GFS and MapReduce

• A subproject of Apache Nutch

• A bet by Yahoo!

Data brings value

• What features to add to the product

• Data analysis must enable decisions

• V3: volume, velocity, variety

Big Data brings big value

Entering: Industrial Data

Hadoop’s childhood

• HDFS: Hadoop Distributed Filesystem

• MapReduce: computational framework

HDFS: not a POSIXfs • Huge blocks: 64Mb (128Mb)

• Mostly immutable files (append, truncate)

• Streaming data access

• Block replication

How do I use it?

$ hadoop fs –lsr / # hadoop-fuse-dfs dfs://hadoop-hdfs /mnt $ ls /mnt # mount –t nfs –o vers=3,proto=tcp,nolock host:/ /mnt $ ls /mnt

Principle #1

HDFS is the datalake

Pivotal’s Focus on Data Lakes

Existing EDW / Datamarts

Raw “untouched” Data

In-Mem

ory Parallel Ingest

Data Management���

(Search Engine)

Processed Data

In-Memory Services BI / A

nalytical Tools

Data Lake

ERP

HR

SFDC

New Data Sources/Formats

Machine

Traditional Data Sources

Finally! I now have full

transparency on the data

with amazing speed!

All data��� is now

accessible!

I can now afford ���“Big Data”

Business Users

ELT Processing with Hadoop

HDFS MapReduce/SQL/Pig/Hive

Analytical Data Marts/

Sandboxes

Security and Control

HDFS enables the stack

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

GemFire XD

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

Spark

Shark

Streaming

MLib

GraphX

Impala

HAW

Q

SpringXD

MADlib

Ham

ster

PivotalR

YARN

Principle #2

Apps share their internal state

MapReduce

• Batch oriented (long jobs; final results)

• Brings the computation to the data

• Very constrained programming model

• Embarrassingly parallel programming model

• Used to be the only game in town for compute

MapReduce Overview

• Record = (Key, Value)

• Key : Comparable, Serializable

• Value: Serializable

• Logical Phases: Input, Map, Shuffle, Reduce, Output

Map

• Input: (Key1, Value1)

• Output: List(Key2, Value2)

• Projections, Filtering, Transformation

Shuffle

• Input: List(Key2, Value2)

• Output

• Sort(Partition(List(Key2, List(Value2))))

• Provided by Hadoop : Several Customizations Possible

Reduce

• Input: List(Key2, List(Value2))

• Output: List(Key3, Value3)

• Aggregations

Anatomy of MapReduce

d a c

a b c

a 3 b 1 c 2

a 1 b 1 c 1

a 1 c 1 a 1

a 1 1 1 b 1 c 1 1

HDFS mappers reducers HDFS

MapReduce DataFlow

How do I use it? public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { public void map(Object key, Text value, Context context) { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } }

How do I use it? public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } }

How do I run it?

$ hadoop jar hadoop-examples.jar wordcount \ input \ output

Principle #3

MapReduce is assembly language of Hadoop

Hadoop’s childhood

• Compact (pretty much a single jar)

• Challenged in scalability and SPOFs

• Extremely batch oriented

• Hard for non-Java programmers

Then, something happened

Hadoop 1.0

HDFS

ASF Projects FLOSS Projects Pivotal Products

MapReduce

Hadoop 2.0

HDFS

ASF Projects FLOSS Projects Pivotal Products

MapReduce Tez

YARN

Ham

ster

YARN

Hadoop 2.0

• HDFS 2.0

• Yet Another Resource Negotiator (YARN)

• MapReduce is just an “application” now

• Tez is another “application”

• Pivotal’s Hamster (OpenMPI) yet another one

MapReduce 1.0

Job Tracker

Task Tracker���(HDFS)

Task Tracker���(HDFS)

task1 task1 task1 task1 task1

task1 task1 task1 task1 taskN

YARN (AKA MR2.0)

Resource���Manager

Job Tracker

task1 task1 task1 task1 task1 Task Tracker

YARN (AKA MR2.0)

Resource���Manager

Job Tracker

task1 task1 task1 task1 task1 Task Tracker

YARN • Yet Another Resource Negotiator

• Resource Manager

• Node Managers

• Application Masters

• Specific to paradigm, e.g. MR Application master (aka JobTracker)

YARN: beyond MR

Resource���Manager

MPI

MPI

Hamster

•  Hadoop and MPI on the same cluster

•  OpenMPI Runtime on Hadoop YARN

•  Hadoop Provides: Resource Scheduling, ���Process monitoring, Distributed File System

•  Open MPI Provides: Process launching, ���Communication, I/O forwarding

Hamster Components • Hamster Application Master

• Gang Scheduler, YARN Application Preemption

• Resource Isolation (lxc Containers)

• ORTE: Hamster Runtime

• Process launching, Wireup, Interconnect

Hamster Architecture

Hadoop 2.0

HDFS

ASF Projects FLOSS Projects Pivotal Products

MapReduce Tez

YARN

Ham

ster

YARN

Hadoop ecosystem

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

YARN

Ham

ster

YARN

There’s way too much stuff

• Tracking dependencies

• Integration testing

• Optimizing the defaults

• Rationalizing the behaviour

Wait! We’ve seen this!

GNU Software Linux kernel

Apache Bigtop Hadoop ecosystem (Hbase, Pig, Hive)

Hadoop���(HDFS, YARN, MR)

Principle #4

Apache Bigtop is how the Hadoop distros get

defined

The ecosystem • Apache HBase

• Apache Crunch, Pig, Hive and Phoenix

• Apache Giraph

• Apache Oozie

• Apache Mahout

• Apache Sqoop and Flume

Apache HBase • Small mutable records vs. HDFS files

• HFiles kept in HDFS

• Memcached for HDFS

• Built on HDFS and Zookeeper

• Google’s Bigtable

Hbase datamodel

• Driven by the original Webtable usecase:

com.cnn.www <html>...

content:

CNN CNN.co

anchor:a.com anchor:b.com

How do I use it? HTable table = new HTable(config, “table”);

Put p = new Put(Bytes.toBytes(“row”));

p.add(Bytes.toBytes(“family”),

Bytes.toBytes(“qualifier”),

Bytes.toBytes(“data”));

table.put(p);

Dataflow model

HBase

HDFS

Producer Consumer

When do I use it?

• Serving up large amounts of data

• Fast random access

• Scan operations

Principle #5

HBase: when you need OLAP + OLTP

What if its OLTP?

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

YARN

Ham

ster

YARN

GemFire XD

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

YARN

GemFire XD

Ham

ster

YARN

GemFire XD: a better HBase? • Close sourced but extremely mature

• SQL/Objects/JSON data model

• High concurrency, high update load

• Mostly selective point queries (no scans)

• Tiered storage architecture

YCSB Benchmark; Throughput is 2-12X

0

100000

200000

300000

400000

500000

600000

700000

800000

AU BU CU D FU LOAD

Th

rou

ghp

ut

(op

s/se

c)

HBase

4

8

12

16

0

100000

200000

300000

400000

500000

600000

700000

800000

AU BU CU D FU LOAD

Th

rou

ghp

ut

(op

s/se

c)

GemFire XD

4

8

12

16

YCSB Benchmark; Latency is 2X – 20X better

0

2000

4000

6000

8000

10000

12000

14000

Lat

en

cy (μ

sec)

HBase

4

8

12

16

0

2000

4000

6000

8000

10000

12000

14000

Lat

en

cy (μ

sec)

GemFire XD

4

8

12

16

Principle #6

There are always 3 implementations

Querying data

• MapReduce: “an assembly language”

• Apache Pig: a data manipulation DSL (now Turing complete!)

• Apache Hive: a batch-oriented SQL on top of Hadoop

How do I use Pig?

grunt> A = load ‘./input.txt’;

grunt> B = foreach A generate ��� flatten(TOKENIZE((chararray)$0)) as��� words;

grunt> C = group B by word;

grunt> D = foreach C generate COUNT(B), ��� group;

How do I use Hive? CREATE TABLE docs (line STRING);

LOAD DATA INPATH 'text' OVERWRITE INTO TABLE docs;

CREATE TABLE word_counts AS

SELECT word, count(1) AS count FROM

(SELECT explode(split(line, '\s')) AS word FROM docs)

GROUP BY word

ORDER BY word;

Can we short Oracle now?

• No indexing

• Batch oriented scheduling

• Optimization for long running queries

• Metadata management is still in flux

[Close to] real-time SQL

• Impala (inspired by Google’s F1)

• Hive/Tez (AKA Stinger)

• Facebook’s Presto (Hive’s lineage)

• Pivotal’s HAWQ

HAWQ

• GreenPlum MPP database core

• True ANSI SQL support

• HDFS storage backend

• Parquet support

Principle #7

SQL on Hadoop

Feeding the elephant

Getting data in: Flume • Designed for collecting log data

• Flexible deployment topology

Sqoop: RDBMs connection • Sqoop 1

• A MapReduce tool

• Must use Oozie for workflows

• Sqoop 2

• Well, 0.99.x really

• A standalone service

Spring XD

• Unified, distributed, extensible system for data ingestions, real time analytics and data exports

• Apache Licensed, not ASF

• A runtime service, not a library

• AKA “Oozie + Flume + Sqoop + Morphlines”

How do I use it?

# deployment: ./xd-singlenode

$ ./xd-shell

xd:> hadoop config fs –namenode hdfs://nn:8020

xd:> stream create –definition “time | hdfs” ��� –name ticktock

xd:> stream destroy –name ticktock

Feeding the Elephant

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

YARN

GemFire XD

SpringXD

Ham

ster

YARN

Spark the disruptor

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

GemFireXD

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

Spark

Shark

Streaming

MLib

GraphX

SpringXD

YARN

Ham

ster

YARN

What’s wrong with MR?

Source: UC Berkeley Spark project (just the image)

Spark innovations • Resilient Distribtued Datasets (RDDs)

• Distributed on a cluster

• Manipulated via parallel operators (map, etc.)

• Automatically rebuilt on failure

• A parallel ecosystem

• A solution to iterative and multi-stage apps

RDDs

warnings = textFile(…).filter(_.contains(“warning”)) .map(_.split(‘ ‘)(1))

HadoopRDD���path = hdfs://

FilteredRDD���contains…

MappedRDD split…

Parallel operators

• map, reduce

• sample, filter

• groupBy, reduceByKey

• join, leftOuterJoin, rightOuterJoin

• union, cross

An alternative backend

• Shark: a Hive on Spark (now Spark SQL)

• Spork: a Pig on Spark

• Mlib: machine learning on Spark

• GraphX: Graph processing on Spark

• Also featuring its own streaming engine

How do I use it?

val file = spark.textFile("hdfs://...")

val counts = file.flatMap(line => line.split(" "))

.map(word => (word, 1))

.reduceByKey(_ + _)

counts.saveAsTextFile("hdfs://...")

Principle #8

Spark is the technology of 2014

Where’s the cloud?

What’s new?

• True elasticity

• Resource partitioning

• Security

• Data marketplace

• Data leaks/breaches

Hadoop Maturity

ETL Offload Accommodate massive ���

data growth with existing EDW investments

Data Lakes Unify Unstructured and Structured Data Access

Big Data Apps

Build analytic-led applications impacting ���

top line revenue

Data-Driven Enterprise

App Dev and Operational Management on HDFS

Data Architecture

Pivotal HD on Pivotal CF

� Enterprise PaaS Management System

� Flexible multi-language ‘buildpack’ architecture

� Deployed applications enjoy built-in services

� On-Premise Hadoop as a Service

� Single cluster deployment of Pivotal HD

� Developers instantly bind to shared Hadoop Clusters

� Speeds up time-to-value

Pivotal Data Fabric Evolution

Analytic���Data Marts

SQL Services

Operational ���Intelligence

In-Memory Database

Run-Time���Applications

Data Staging���Platform

Data Mgmt. Services

Pivotal Data Platform

Stream ���Ingestion

Streaming Services

Software-Defined Datacenter

New Data-fabrics

In-Memory Grid

...ETC

Principle #9

Hadoop in the Cloud is one of many

distributed frameworks

2014 is the year of Hadoop

HDFS

Pig

Sqoop Flume

Coordination and workflow

management

Zookeeper

Command Center

ASF Projects FLOSS Projects Pivotal Products

GemFire XD

Oozie

MapReduce

Hive

Tez

Giraph

Hadoop UI

Hue

SolrCloud

Phoenix

HBase

Crunch Mahout

Spark

Shark

Streaming

MLib

GraphX

Impala

HAW

Q

SpringXD

MADlib

Ham

ster

PivotalR

YARN

A NEW PLATFORM FOR A NEW ERA

Additional Line 18 Point Verdana

Credits

• Apache Software Foundation

• Milind Bhandarkar

• Konstantin Boudnik

• Robert Geiger

• Susheel Kaushik

• Mak Gokhale

Questions ?