[Rakuten TechConf2014] [C-6] Leveraging Spark for Cluster Computing

Apache Spark, Spark, Apache, and the Spark logo are trademarks of The Apache Software Foundation.

All Spark images and code are from http://spark.apache.org/

Leveraging For Cluster Computing

Robin M. E. SwezeyRakuten Institute of Technology, Tokyo

Intelligence Domain Group

[email protected]

mailto:[email protected]


All Spark images and code are from http://spark.apache.org/2

What is Spark?



In short, Spark is the future of

open-source MapReduce



Current Hadoop stack is heterogeneous

Spark = Fully integrated analytics suite and cluster

computing framework

Berkeley AMP lab + Apache Software Foundation

Why Spark?

Apache Hadoop, Hive, Mahout, Pig and their respective logos are trademarks of The Apache Software Foundation.



On the surface, very similar to Hadoop

• Relies on HDFS

• Runs on Yarn, Mesos, or standalone

• MapReduce + General cluster computing

Platform



1. Resilient Distributed Dataset (RDD)Central to Spark (R dataframe-ish)

Platform

RDD

RDDRDD

Key differences with usual stack



1. Resilient Distributed Dataset (RDD)Central to Spark (R dataframe-ish)

Platform


RDD<String>RDD<Tuple>RDD<Tuple>



Platform

2. Better resource utilizationDisk is slow. Memory is fast. Several levels of persistence.


Read blocks

from disk

Cache aggregates

in memory



Platform


2. Better resource utilizationMore cores > more machines. Resource locality.

Each node x each core

/ each local block



Platform

3. Easier development & operationsScala, Java, Python API

(Logistic Regression)


8 Lines



Platform

3. Easier AnalyticsInteractive Shells in Scala, Python

Easy to connect with SparkContext (e.g. iPython Notebook)




4. Integrated Solution

Easy MapReduce

DBMS-like Functionality

Streaming

Machine Learning

Platform

Apache Hadoop, Hive, Mahout, Pig and their respective logos are trademarks of The Apache Software Foundation.




Applications

Thanks to RDD Distributed Operators

map()

reduce()

reduceByKey()

groupBy()

sample()

pipe()

foreach()

fold()

histogram()

…

Easy MapReduce



Applications

Cf.

Integrated Unified Data Access

Hive Compatible Standard Connectivity

Sped-up Analytics with DBMS-like SQL Functionality



Applications

Cf.

Streaming



Applications

Cf.

Statistics

Classification / Regression

Collaborative Filtering

Clustering

Dimensionality Reduction

Feature Extraction

Image: http://en.wikipedia.org/wiki/Machine_learning#mediaviewer/File:Linear-svm-scatterplot.svg

Machine Learning



Applications

Flexible

FastPageRank

Connected components

Label propagation

SVD++

Triangle count

Graph Processing



More

There are deployed clusters

of 1,000+ nodes

How does it scale?



More

Spark 1.1.0

had 171 contributors!

There’s open-source, and there’s highly supported open-source



In Conclusion

Hadoop Cluster Computing Hype Cycle

Image: http://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Gartner_Hype_Cycle.svg/2000px-Gartner_Hype_Cycle.svg.png



Thank you!

http://spark.apache.org

[Rakuten TechConf2014] [C-6] Leveraging Spark for Cluster Computing

Technology

Transcript of [Rakuten TechConf2014] [C-6] Leveraging Spark for Cluster Computing