Leverage Mesos for running Spark Streaming production jobs by Iulian Dragos and Luc Bourlier
Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier
-
Upload
spark-summit -
Category
Data & Analytics
-
view
1.385 -
download
0
Transcript of Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier
![Page 1: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/1.jpg)
REACTIVE STREAMS, linking REACTIVE APPLICATIONS to SPARK STREAMING
Luc BourlierTypesafe Inc.
![Page 2: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/2.jpg)
Agenda• Back pressure• Back pressure in Spark Streaming• Reactive Application• Reactive Streams• Demo
![Page 3: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/3.jpg)
Back Pressure
![Page 4: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/4.jpg)
Back Pressure• a slow consumer should slow down the
producer– the produce applies pressure– the consumer applies back pressure
• the classic example: TCP
![Page 5: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/5.jpg)
Spark Streaming
![Page 6: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/6.jpg)
Spark Streaming
![Page 7: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/7.jpg)
Spark StreamingCongestion support in Spark 1.4Static rate limit
• spark.streaming.receiver.maxRate
• conservative• difficult to find the right limit (depends on cluster size)• one limit to all streams
![Page 8: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/8.jpg)
Spark StreamingBack pressure in Spark 1.5Dynamic rate limit
• rate estimator– estimates the number of element that can be safely processed by
system during the batch interval
• rate sent to receivers• rate limiter
– relies on TCP to slow down producers
![Page 9: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/9.jpg)
Spark StreamingRate estimator• each BatchCompleted event contains
– processing delay, scheduling delay– number of element in mini-batch
• the rate is (roughly) elements / processingDelay• but what about accumulated delay?
![Page 10: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/10.jpg)
Spark StreamingRate estimator
Proportional-Integral-Derivative• P, I, D constants change
convergence, overshooting
and oscillations
https://en.wikipedia.org/wiki/PID_controller
![Page 11: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/11.jpg)
Spark StreamingBack pressure in Spark 1.5
• each input has its own estimator• work with all stream receivers
including KafkaDirectInputStream• configuration
– spark.streaming.backpressure.enable true– spark.streaming.backpressure.minRate R
![Page 12: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/12.jpg)
![Page 13: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/13.jpg)
Spark StreamingLimitations
• linearity assumption• records with similar execution times• TCP back pressure accumulates in the TCP channel
![Page 14: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/14.jpg)
Reactive Application
![Page 15: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/15.jpg)
Reactive Application
http://www.reactivemanifesto.org
![Page 16: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/16.jpg)
responds in a timely manner
stays responsive in the face of failure
stays responsive under varying workload
relies on asynchronousmessage-passing
Reactive ApplicationResponsive ResilientElastic Message Driven
![Page 17: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/17.jpg)
Reactive Streams
![Page 18: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/18.jpg)
Reactive Streams• one tool to create reactive applications• specification for back pressure interface to connect
systems supporting back pressure in the JVM– small: 3 interfaces, 7 methods total
• subscriber controls rate by requesting elements from producers
http://www.reactive-streams.org
![Page 19: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/19.jpg)
End to end back pressure
![Page 20: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/20.jpg)
End to end back pressure• Reactive application with reactive streams
connector⇒ back pressure enabled
• Spark Streaming 1.5+⇒ back pressure enabled
• Reactive streams Spark Streaming receiver⇒ end to end back pressure
![Page 21: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/21.jpg)
![Page 22: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/22.jpg)
Demo
![Page 23: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/23.jpg)
Demo
Image, Wikipedia. By CaitlinJo - Own workThis mathematical image was created with Mathematica, CC BY 3.0
RNG pair mergeInt
(Int, Int)sliced
in/out
over all values
per slices
per windows
Boolean
π
in/out
over all values
per slices
per windows
Boolean(Int, Int)windowed
ReactiveApplication
SparkStreaming
receiver StatisticalAnalysis
RDDs
![Page 24: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/24.jpg)
What’s Next?• Last bit of API change (in Spark 2.0)
SPARK-10420
• Publish the Reactive Streams Receiver
![Page 25: Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bourlier](https://reader034.fdocuments.net/reader034/viewer/2022042723/587155961a28ab8e5b8b512d/html5/thumbnails/25.jpg)
THANK [email protected]