using Apache Spark and Zeppelin -...
Transcript of using Apache Spark and Zeppelin -...
Big Data Visualizationusing
Apache Spark and Zeppelin
Prajod Vettiyattil, Software Architect, Wipro
Agenda
• Big Data and Ecosystem tools
• Apache Spark
• Apache Zeppelin
• Data Visualization
• Combining Spark and Zeppelin
2
BIG DATA AND ECOSYSTEM TOOLS
Big Data
• Data size beyond systems capability
– Terabyte, Petabyte, Exabyte
• Storage
– Commodity servers, RAID, SAN
• Processing
– In reasonable response time
– A challenge here
4
Server
Tradition processing tools
• Move what ?
– the data to the code or
– the code to the data
5
Data
Server
move data to code
move code to data
Code
Traditional processing tools
• Traditional tools
– RDBMS, DWH, BI
– High cost
– Difficult to scale beyond certain data size
• price/performance skew
• data variety not supported
6
Map-Reduce and NoSQL
• Hadoop toolset
– Free and open source
– Commodity hardware
– Scales to exabytes(1018), maybe even more
• Not only SQL
– Storage and query processing only
– Complements Hadoop toolset
– Volume, velocity and variety
7
All is well ?
• Hadoop was designed for batch processing
• Disk based processing: slow
• Many tools to enhance Hadoop’s capabilities
– Distributed cache, Haloop, Hive, HBase
• Not for interactive and iterative
8
TOWARDS SINGULARITY
What is singularity ?
10
0
1000
2000
3000
4000
5000
6000
7000
8000
1 2 3 4 5 6 7
AI
cap
acit
y
Decade
Decade vs AI capacity
Point of singularity
Technological singularity
• When AI capability exceeds Human capacity
• AI or non-AI singularity
• 2045: http://en.wikipedia.org/wiki/Ray_Kurzweil
– The predicted year
11
APACHE SPARK
History of Spark
13
Spark 1.3.0 released
2015
MarchSpark 1.0.0
released. 100TB sort achieved in
23 mins
2014
Spark donated to Apache Software
Foundation
2013
Spark is made open source. Available on
github
2010
Spark created by PhD student
at UC Berkeley, Matei
Zaharia
2009
Contributors in Spark
• Yahoo
• Intel
• UC Berkeley
• …
• 50+ organizations
14
Hadoop and Spark
• Spark complements the Hadoop ecosystem
• Replaces: Hadoop MR
• Spark integrates with
– HDFS
– Hive
– HBase
– YARN
15
Other big data tools
• Spark also integrates with
– Kafka
– ZeroMQ
– Cassandra
– Mesos
16
Programming Spark
• Java
• Scala
• Python
• R
17
Spark toolset
18
Apache Spark
Spark
SQL
Spark
Streamin
g
MLlib GraphX
Spark R
Blink DB
Spark
Cassandra
Tachyon
What is Spark for ?
19
Batch
Interactive Streaming
The main difference: speed
• RAM access vs Disk access
– RAM access is 100,000 times faster !
20
https://gist.github.com/hellerbarde/2843375
Lambda Architecture pattern
• Used for Lambda architecture implementation
– Batch layer
– Speed layer
– Serving layer
21
Batch
Layer
Speed
Layer
Serving Layer
Data Input
Data
consumers
Worker Node
Executor
Deployment Architecture
22
Master Node
Executor
Task CacheTaskTask
Worker Node
ExecutorExecutorExecutor
HDFS
Data
Node
HDFS
Data
Node
TaskTaskTaskCache
Spark’s Cluster
Manager
Spark Driver
HDFS Name
Node
APACHE ZEPPELIN
Interactive data analytics
• For Spark and Flink
• Web front end
• At the back, it connects to
– SQL systems(Eg: Hive)
– Spark
– Flink
24
Deployment Architecture
25
Spark / Flink /
Hive
Zeppelin daemon
Web
browser 1
Web
browser 2
Web
browser 3
Web Server
Local
Interpreters
Optional
Remote
Interpreters
Notebook
• Is where you do your data analysis
• Web UI REPL with pluggable interpreters
• Interpreters
– Scala, Python, Angular, SparkSQL, Markdown and Shell
26
User Interface features
• Markdown
• Dynamic HTML generation
• Dynamic chart generation
• Screen sharing via websockets
27
SQL Interpreter
• SQL shell
– Query spark data using SQL queries
– Return normal text, HTML or chart type results
28
Scala interpreter for Spark
• Similar to the Spark shell
• Upload your data into Spark
• Query the data sets(RDDs) in your Spark server
• Execute map-reduce tasks
• Actions on RDD
• Transformations on RDD
29
DATA VISUALIZATION
Visualization tools
31Source: http://selection.datavisualization.ch/
D3 Visualizations
32Source: https://github.com/mbostock/d3/wiki/Gallery
The need for visualization
33
Big DataDo
something to data
User gets
comprehensible
data
Tools for Data Presentation Architecture
34
1.Identify
2.Locate
3.Manipulate
4.Format
5.Present
A data analysis tool/toolset would support:
COMBINING SPARK AND ZEPPELIN
Spark and Zeppelin
36
Spark
Worker
Node
Spark
Master
Node
Spark
Worker
Node
Zeppelin daemon
Web
browser 1
Web
browser 2
Web
browser 3
Web Server
Local
Interpreters
Remote
Interpreters
Zeppelin views: Table from SQL
37
Zeppelin views: Table from SQL
38
%sql select age, count(1) from bank where
marital="${marital=single,single|divorced|married}"
group by age order by age
Zeppelin views: Pie chart from SQL
39
Zeppelin views: Bar chart from SQL
40
Zeppelin views: Angular
41
Share variables: MVVM
• Between Scala/Python/Spark and Angular
• Observe scala variables from angular
42
Scala-Spark Angular
x = “foo” x = “bar”
Zeppelin
Screen sharing using Zeppelin
• Share your graphical reports
– Live sharing
– Get the share URL from zeppelin and share with others
– Uses websockets
• Embed live reports in web pages
43
FUTURE
Spark and Zeppelin
• Spark
– Machine Learning using Spark
– GraphX and MLlib
• Zeppelin
– Additional interpreters
– Better graphics
– Report persistence
– More report templates
– Better angular integration45
SUMMARY
Summary
• Spark and tools
• The need for visualization
• The role of Zeppelin
• Zeppelin – Spark integration
47