Scala usergroup stockholm - reactive integrations with akka streams
-
Upload
johan-andren -
Category
Data & Analytics
-
view
447 -
download
2
Transcript of Scala usergroup stockholm - reactive integrations with akka streams
akka streamsReactive Integrations with
that just work™
Johan Andrén Scala Usergroup Stockholm 2016-10-18
Johan AndrénAkka Team Stockholm Scala User Group
Make building powerful concurrent & distributed applications simple.Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community
Complete Java & Scala APIs for all features
What’s in the toolkit?
“Stream”has many meanings…
akka streamsAsynchronous back pressured stream processing
Source Sink
Flow
akka streamsAsynchronous back pressured stream processing
Source Sink
(possible) asynchronous
boundaries
Flow
akka streamsAsynchronous back pressured stream processing
Source Sink
10 msg/s 1 msg/s
OutOfMemoryError!!
Flow
akka streamsAsynchronous back pressured stream processing
Source Sink
10 msg/s 1 msg/s
hand me 3 morehand me 3 more
1 msg/s Flow
akka streamsNot only linear streams
Source
SinkFlow
SourceSink
FlowFlow
Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols
http://www.reactive-streams.org
Part of JDK 9java.util.concurrent.Flow
Reactive Streams
RS Library A RS library B
async boundary
Reactive Streams
RS Library A RS library B
async boundary
Make building powerful concurrent & distributed applications simple.
The APIAkka Streams
Complete and awesome Java and Scala APIs (Just like everything in Akka)
Akka Streams in 20 seconds:
val source: Source[Int, NotUsed] = Source(0 to 200000) val flow: Flow[Int, String, NotUsed] = Flow[Int].map(_.toString)val sink: Sink[String, Future[Done]] = Sink.foreach(println) val runnableGraph = source.via(flow).to(sink) runnableGraph.run()
Akka Streams in 20 seconds:
source.via(flow).to(sink) Source[Int, NotUsed]Flow[Int, String, NotUsed]
Sink[String, Future[Done]]
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
AlpakkaA community for Streams connectorshttp://blog.akka.io/integrations/2016/08/23/intro-alpakka
Alpakka – a community for Stream connectors
Threading & Concurrency in Akka Streams Explained (part I)
Mastering GraphStages (part I, Introduction)
Akka Streams Integration, codename Alpakka
A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II)
Writing Akka Streams Connectors for existing APIs
Flow control at the boundary of Akka Streams and a data provider
Akka Streams Kafka 0.11
Alpakka – a community for Stream connectors
Existing examples:MQTT AMQP
Streaming HTTPStreaming TCP
Streaming FileIOCassandra Queries
“Reactive Kafka” (akka-stream-kafka)S3, SQS & other Amazon APIs
Streaming JSONStreaming XML
…
Alpakka – a community for Stream connectors
Demo
KafkaSink
FileTailSource
Kafka
Bytes =>
Lines
Alpakka – a community for Stream connectors
Demo
Kafka
parse linewindow
last 10
Find hacker
KafkaSource
Sound the alarm!
Alpakka – a community for Stream connectors
Kafka Stream Stream
Stream
Stream
cluster
Akka Streams & HTTP
streams& HTTP
Akka Streams / HTTP
Quiz time!
Akka Streams / HTTP
recv buffer
send buffer
🚚
🚚🚚
🚚
🚚
🚚
🚚
Akka Streams / HTTP
recv buffer
send buffer
🚚🚚🚚🚚
🚚
🚚🚚🚚🚚
🚚
🚑
Streaming in Akka HTTP
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
HttpServer as a: Flow[HttpRequest, HttpResponse]
Streaming in Akka HTTP
HttpServer as a: Flow[HttpRequest, HttpResponse]
HTTP Entity as a: Source[ByteString, _]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
Streaming in Akka HTTP
HttpServer as a: Flow[HttpRequest, HttpResponse]
HTTP Entity as a: Source[ByteString, _]
Websocket connection as a: Flow[ws.Message, ws.Message]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
Streaming from Akka HTTP
send buffer
🚚🚚
🚚
OSAppEntrySource
Json =>
Bytes
Entry =>
Json
TcpSink
Streaming from Akka HTTP
send buffer
🚚🚚🚚
OSAppEntrySource
Json =>
Bytes
Entry =>
Json
TcpSink🚚🚚
Streaming from Akka HTTP
send buffer
🚚🚚🚚
OSAppEntrySource
Json =>
Bytes
Entry =>
Json
TcpSink🚚🚚
No demand from TCP =
No demand upstream =
Source won’t read log
=> Bounded memory stream processing!
Demo
Akka ❤ contributionsEasy to contribute tickets:
https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22
Akka Stream Contrib https://github.com/akka/akka-stream-contrib
Mailing list: https://groups.google.com/group/akka-user
Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka
Thanks for listening!
@apnylle [email protected]
Sample sources https://github.com/johanandren/akka-stream-samples/tree/scala-stockholm-2016-10