Real Time Data Processing With Spark Streaming, Node.js and Redis with Visualization
-
Upload
brandon-obrien -
Category
Data & Analytics
-
view
294 -
download
2
Transcript of Real Time Data Processing With Spark Streaming, Node.js and Redis with Visualization
Real Time Data Processing with
Spark Streaming
Brandon O’BrienOct 26th, 2016
Spark Streaming: Intro
1. Intro2. Demo + high level walkthrough3. Spark in detail4. Detailed demo walkthrough and/or
workshop
Spark Streaming
Spark experience level?
Select one: Beginner Intermediate Expert
Spark Streaming: Demo
DEMO
Spark Streaming: Demo Info
• Data Source:• Data Producer Thread• Redis
• Data Consumer• Spark as Stream Consumer• Redis Publish
• Dashboard:• Node.js/Redis Integration• Socket.io Publish• AngularJS + JavaScript
Spark Streaming: Spark in detail
SPARK IN DETAIL
Spark Streaming: Concepts
Application:• Driver program• RDD• Partition• Elements• DStream• InputReceiver• 1 JVM for driver
program• 1 JVM per executor
Cluster:• Master• Executors• Resources• Cores• Gigs RAM
• Cluster Types:• Standalone• Mesos• YARN
Spark Streaming: Lazy execution
//Allocate resources on clusterval conf = new SparkConf().setAppName(appName).setMaster(master) val sc = new SparkContext(conf)
//Lazy definition of logical processing (transformations)val textFile = sc.textFile("README.md")
.filter(line=> {line.length> 10})
//foreachPartition() triggers execution (actions)textFile.foreachPartition(partition=> {
partition.foreach(line => {println(line)
})})• Use rdd.persist() when multiple actions are called on the same RDD
Spark Streaming: Execution Env
• Distributed data, distributed code• RDD partitions are distributed across executors• Actions trigger execution and return results to the driver program• Code is executed on either the driver or executors• Be careful of function closures!
//Function arguments to transformations executed on executorsval textFile = sc.textFile("README.md")
.filter(line=> {line.length> 10})
//collect() triggers execution (actions)//executed on driver. foreachPartition executed on executorstextFile.collect().foreach(line => {
println(line)})
Spark Streaming: Execution Env
Spark Streaming: Parallelism
• RDD partitions are processed in parallel• Elements in a single partition are processed serially• You control the number of partitions in an RDD• If you need to guarantee any particular ordering of processing, use
groupByKey() to force all elements with the same key onto the same partitions
• Be careful of shuffles
val textFile = sc.textFile("README.md”)val singlePartitionRDD = textFile.repartition(1)
val linesByKey = shopResultsEnriched.map(line => (getPartitionKey(line), line)).groupByKey()
Spark Streaming: DStreams
• Receiver Types• Kafka (Receiver + Direct)• Flume• Kinesis• TCP Socket• Custom (Ex: redis.receiver.RedisReceiver.scala)
• Note: Kafka receiver will consume an entire core (no context switch)
Real Time Data Processing with
Spark Streaming
Brandon O’BrienOct 26th, 2016