Post on 11-Jan-2017
Leverage Mesos for running Spark Streaming production jobs
Iulian DragoșLuc Bourlier
Agenda
● Intro to Mesos
● Spark deployment modes (coarse-fine cluster client)
● Fault-tolerance○ driver failures
○ --supervise○ WAL
○ Kafka direct
● Streaming backpressure
● profit !2Leverage Mesos for running Spark Streaming production jobs
Mesos
3Leverage Mesos for running Spark Streaming production jobs
General (a “distributed kernel”)
Efficient resource management
Proven technology (in production at Apple and Twitter)
Typesafe & Mesosphere maintain the Spark/Mesos framework
Why Apache Mesos?
4
“Program against your datacenter as a single pool of resources”
Frameworks running on Mesos
HDFS
Marathon
Cassandra
ElasticSearch
Yarn (Myriad)
and of course, Spark5
Resource Scheduling with Mesos
6
2-level scheduling
Mesos offers resources to frameworks
Frameworks accept or reject offers
Frameworks are free to schedule tasks on those resources as they see fit
Offers include CPU cores, memory, ports, disk
Spark Deployment modes
7
Deployment modes
8
Driver placementClient mode:
on the machine starting the job
(developer machine)
quick and easy
if the machine stops, the job is lost
Cluster mode:
inside the cluster
managed by the framework
no need to keep the developer machine online
less convenient
Spark UI may be hard to find/unavailable
spark-shell and stdout/stderr are not available 9
Fine-grained:
Each Spark task is a Mesos task. Resources are acquired and freed as needed.
better resource sharing
slower startup time
good for batch and static streaming
Coarse-grained:
Mesos executors are acquired at the beginning of a job.Spark tasks are dispatched immediately.
fast startup for tasks
better for interactive sessions
resources are locked for the duration of the app
10
Executor policy
Spark Streaming
11Leverage Mesos for running Spark Streaming production jobs
12
Fault tolerance
13
Node dies
14
Write ahead log (WAL)
15
Driver failure
16
Driver restart
17
on restart, the driver can recover everything from checkpoint dir
can: you need to code it that way
Standalone and Yarn have --supervise for this purpose
On Mesos, use Marathon
Driver restarts
18
Recommended setup
19
use cluster-mode deployment
enable WAL or use a reliable source (Kafka)
driver restarts à la --supervise using Marathon
enable back-pressure (see next)
Back pressure inSpark Streaming
20Leverage Mesos for running Spark Streaming production jobs
Resilient system ≠ stable system
Data may arrive faster than Spark can process
⇒ scheduling delay
⇒ memory spill
⇒ driver stopped
21Leverage Mesos for running Spark Streaming production jobs
Congestion support in Spark 1.4
Static rate limit
● conservative
● difficult to find the right limit (depends on cluster size)
● one limit to rule all stream
22Leverage Mesos for running Spark Streaming production jobs
Back pressure
● (also, flow control)
● a slow consumer should slow down the producer
○ classic example: TCP
● Reactive Streams
23Leverage Mesos for running Spark Streaming production jobs
Back pressure in Spark 1.5
● dynamic rate limit
● rate estimator
○ estimates the number of element that can be safely processed by system
during the batch interval
● rate sent to the receivers
● rate limiter
○ relies on TCP to slow down producers
24Leverage Mesos for running Spark Streaming production jobs
Rate estimator
● each BatchCompleted event contains
○ processing delay, scheduling delay
○ number of element in mini-batch
● the rate is (roughly) elements/processingDelay
● but what about accumulated delay?
25Leverage Mesos for running Spark Streaming production jobs
Rate estimator
Propotional-Integrale-Derivative
● P, I, D constants can change
convergence, overshooting,
oscillations
26Leverage Mesos for running Spark Streaming production jobs
https://en.wikipedia.org/wiki/PID_controller
27Leverage Mesos for running Spark Streaming production jobs
https://en.wikipedia.org/wiki/PID_controller
Back pressure in Spark 1.5
● each input has its own estimator
● work with all stream receivers,
including KafkaDirectInputStream
● configuration:
○ spark.streaming.backpressure.enabled true
○ spark.streaming.backpressure.minRate R
28Leverage Mesos for running Spark Streaming production jobs
Limitations
● linearity assumption
● records with similar execution times
● TCP back pressure accumulates in the TCP channel
29Leverage Mesos for running Spark Streaming production jobs
Back pressure withReactive Streams
30Leverage Mesos for running Spark Streaming production jobs
Reactive Streams
● standard for asynchronous stream processing
● consumer controls rate by asking for elements from producers
with Spark Streaming
● more direct back pressure control
31Leverage Mesos for running Spark Streaming production jobs
32Leverage Mesos for running Spark Streaming production jobs
Reactive Streams
● support expected with Spark 1.6
● SPARK-10420 (Spark Streaming Receiver)
33Leverage Mesos for running Spark Streaming production jobs
Thank you
34