How to manage large amounts of data with akka streams
-
Upload
igor-mielientiev -
Category
Software
-
view
289 -
download
0
Transcript of How to manage large amounts of data with akka streams
How to manage large amounts of data with
Akka Streamsby Ihor
Mielientiev
Agenda1. Reactive Streams2. Backpressure3. Handling big amount of data (Demos)
Reactive Streamshttp://www.reactive-streams.org/
Reactive Streams: Goals
1. Backpressured asynchronous stream processing2. Standard interfaces
Reactive StreamsReactive Streams is a standard and specification for Stream-oriented libraries for the JVM thatprocess a potentially unbounded number of elementsasynchronously passing elements between componentswith mandatory non-blocking backpressure.
Reactive Streams: ImplementationsAkka Streams @ Lightbend (Typesafe)
Reactor @ Pivotal (Spring 5 will adopt “reactive” programming)
RxJava @ Netflix
Vert.x @ RedHat
Ratpack
Slick 3.0
MongoDB
Reactive Rabbit (driver)
Open JDK 9 - Doug Lea @ Oracle
etc.
Reactive Streams (interfaces)
Backpressure
What is backpressure?
What is backpressure?
What is backpressure?
Solutions1.Just Increase Buffer Size :)
2.Bounded buffer (drop messages + require resending)
3.Negative ACK (tell Publisher to slow down)4.speed(publisher) < speed(subscriber) (Dynamic Push/Pull)
Handling big files (HTTP)In many cases the request stored in memory
(gather/parse http request and then analyzing)It’s a bad, bad Idea to put the body in memory. (Try
to upload 10 GB file and store it)
Classic Java Stream Management
ProblemsLow Performance (if no buffer)
Buffer Issues (buffer management/overflow/...)
Error Handling
Mixing IO management with business logic
Not Modular
Synchronous (Blocking)
Ugly Code
Better Way: Akka Streams
Data Creator (Producer)
Data ConsumerPipe and transfor data (Connector)
Akka StreamsPure FunctionalComposableType Safe
Let’s manage HTTP body stream with Akka Stream/HTTP
Process (Upload)1. File is post by client (browser etc.)2. Normalize stream (chunks)3. Async save each chunk in K/V storage (Amazon S3)4. Store metadata
Process (Download)1. Get a fileId from client2. Stream each chunk to client :)
Questions?