Post on 16-Apr-2017
Konrad `@ktosopl` Malawski // ScalaDays Berlin pic: Armored Core for Answer, White Gilt
streamsand Reactive Streams in action
Konrad `ktoso` Malawski
Akka Team, Reactive Streams TCK,
Persistence, HTTP
Konrad `@ktosopl` Malawski
akka.iotypesafe.comgeecon.org
Java.pl / KrakowScala.plsckrk.com
GDGKrakow.pl lambdakrk.pl
… in Action
Yes, we’ll build stuff.
… in Action
Yes, we’ll build stuff.
With pre-release stuff, because ScalaDays.
“Stream”
“Stream” What does it mean?!
Akka Streams
Akka Streams && Reactive Streams
Reactive Streams - story: early FRP
http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspxhttp://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky
https://github.com/ReactiveX/RxJava/graphs/contributorshttps://github.com/reactor/reactor/graphs/contributors
https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy
- .NETs’ Reactive Extensions
.NET 3.5
Reactive Streams - story: 2013’s impls
~2013:
Reactive Programming becoming widely adopted on JVM.
- Play introduced “Iteratees”- Akka (2009) had Akka-IO (TCP etc.)- Ben starts work on RxJava
http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspxhttp://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky
https://github.com/ReactiveX/RxJava/graphs/contributorshttps://github.com/reactor/reactor/graphs/contributors
https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy
Teams discuss need for back-pressure in simple user API.Play’s Iteratee / Akka’s NACK in IO.
}
Reactive Streams - story: 2013’s impls
Play Iteratees – pull back-pressure, difficult API
http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspxhttp://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky
https://github.com/ReactiveX/RxJava/graphs/contributorshttps://github.com/reactor/reactor/graphs/contributors
https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy
Akka-IO – NACK back-pressure; low-level IO (Bytes); messaging API
RxJava – no back-pressure, nice API
Reactive Streams - Play’s Iteratees
def fold[B]( done: (A, Input[E]) => Promise[B], cont: (Input[E] => Iteratee[E, A]) => Promise[B], error: (String, Input[E]) => Promise[B]): Promise[B]
// an iteratee that consumes chunkes of String and produces an IntIteratee[String,Int]
https://www.playframework.com/documentation/2.0/Iteratees
Feb 2013 Iteratees solved the back-pressure problem,but were hard to use.
Iteratee & Enumeratee – Haskell inspired.
Play / Akka teams looking for common concept.
Reactive Streams - expert group founded
October 2013
Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,while recording “Principles of Reactive Programming” Coursera Course.
Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava) and Marius Eriksen (Twitter) meet at Twitter HQ.
The term “reactive non-blocking asynchronous back-pressure” gets coined.
Reactive Streams - expert group founded
October 2013
Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,while recording “Principles of Reactive Programming” Coursera Course.
Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava) and Marius Eriksen (Twitter) meet at Twitter HQ.
The term “reactive non-blocking asynchronous back-pressure” gets coined.
Goals:- asynchronous- never block (waste)- safe (back-threads pressured)- purely local abstraction- allow synchronous impls.
Also, for our examples today:- compatible with TCP
Reactive Streams - expert group founded
October 2013
Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,while recording “Principles of Reactive Programming” Coursera Course.
Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava) and Marius Eriksen (Twitter) meet at Twitter HQ.
The term “reactive non-blocking asynchronous back-pressure” gets coined.
December 2013Stephane Maldini & Jon Brisbin (Pivotal Reactor) contacted by Viktor.
Reactive Streams - expert group founded
October 2013
Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,while recording “Principles of Reactive Programming” Coursera Course.
Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava) and Marius Eriksen (Twitter) meet at Twitter HQ.
The term “reactive non-blocking asynchronous back-pressure” gets coined.
December 2013Stephane Maldini & Jon Brisbin (Pivotal Reactor) contacted by Viktor.
Soon after, the “Reactive Streams” expert group is formed.
Also joining the efforts: Doug Lea (Oracle), Endre Varga (Akka), Johannes Rudolph & Mathias Doenitz (Spray), and many others, including myself join the effort soon after.
October 2013
Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,while recording “Principles of Reactive Programming” Coursera Course.
Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava) and Marius Eriksen (Twitter) meet at Twitter HQ.
The term “reactive non-blocking asynchronous back-pressure” gets coined.
December 2013Stephane Maldini & Jon Brisbin (Pivotal Reactor) contacted by Viktor.
Soon after, the “Reactive Streams” expert group is formed.
Also joining the efforts: Doug Lea (Oracle), Endre Varga (Akka), Johannes Rudolph & Mathias Doenitz (Spray), and many others, including myself join the effort soon after.
Reactive Streams - expert group founded
I ended up implementing much of the TCK.Please use it, let me know if it needs improvements :-)
Reactive Streams - story: 2013’s impls
2014–2015:
Reactive Streams Spec & TCK development, and implementations.
1.0 released on April 28th 2015, with 5+ accompanying implementations.
2015Proposed to be included with JDK9 by Doug Leavia JEP-266 “More Concurrency Updates”
http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/6e50b992bef4/src/java.base/share/classes/java/util/concurrent/Flow.java
2014–2015:
Reactive Streams Spec & TCK development, and implementations.
1.0 released on April 28th 2015, with 5+ accompanying implementations.
2015Proposed to be included with JDK9 by Doug Leavia JEP-266 “More Concurrency Updates”
http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/6e50b992bef4/src/java.base/share/classes/java/util/concurrent/Flow.java
Reactive Streams - story: 2013’s impls
JEP-266 – soon…!public final class Flow { private Flow() {} // uninstantiable
@FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); }
public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); }
public static interface Subscription { public void request(long n); public void cancel(); }
public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { }}
Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries2) Inter-op interfaces between various libraries
Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries
2) Inter-op interfaces between various libraries
Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries
2) Inter-op interfaces between various libraries
Argh, implementing a correct RS Publisher or Subscriber is so hard!
Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries
2) Inter-op interfaces between various libraries
Argh, implementing a correct RS Publisher or Subscriber is so hard!
Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries
2) Inter-op interfaces between various libraries
Argh, implementing a correct RS Publisher or Subscriber is so hard!
Don’t implement RS interfaces directly, use GraphStage instead!
Akka Streams
Streams complement Actors, they do not replace them.
Actors – distribution (location transparency) Streams – back-pressured + more rigid-blueprint
13番: Akka is a Toolkit, pick the right tools for the job.
Runar’s excellent talk @ Scala.World 2015
Asynchronous processing toolbox:
Power
Constraints
13番: Akka is a Toolkit, pick the right tools for the job.
Asynchronous processing toolbox:
Constraints
Power
13番: Akka is a Toolkit, pick the right tools for the job.
Single value, no streaming by definition.Local abstraction. Execution contexts.
Power
Constraints
13番: Akka is a Toolkit, pick the right tools for the job.
Mostly static processing layouts.Well typed and Back-pressured!
Constraints
Power
13番: Akka is a Toolkit, pick the right tools for the job.
Plain Actor’s younger brother, experimental.Location transparent, well typed.Technically unconstrained in actions performed
Constraints
Power
13番: Akka is a Toolkit, pick the right tools for the job.
Runar’s excellent talk @ Scala.World 2015
Location transparent. Various resilience mechanisms.
(watching, persistent recovering, migration, pools)
Untyped and unconstrained in actions performed.
Constraints
Power
Lightbend monitoring
Measure. Don’t guess.
http://monitoring.lightbend.com/docs/latest/home.html
Lightbend monitoring
Measure. Don’t guess.
Let’s find if & where we have a bottleneck…
http://monitoring.lightbend.com/docs/latest/home.html
DEMO
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
Custom Stages &
Understanding the Types
Custom stages, made simple (!)
Custom stages, made simple (!)
Custom stages, made simple (!)
Custom stages, made simple (!)
Understanding the Types
Source.fromGraph(Graph[SourceShape[Status], M]) .via(Graph[FlowShape[Status, T], M]) .to(Graph[SinkShape[Status], _])
Understanding the Types
Source.fromGraph(Graph[SourceShape[Status], M]) .via(Graph[FlowShape[Status, T], M]) .to(Graph[SinkShape[Status], _])
Custom != hard; custom == extensible
Your Stages “feel” native.
DEMOLet’s try “Hundred-ify”.
http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala
See also mapConcat / statefulMapConcat.
Akka HTTP ~ streaming all the way
Streaming all the way with Akka HTTP
Streaming in Akka HTTPDEMO
http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778
HttpServer as a: Flow[HttpRequest, HttpResponse]
Streaming in Akka HTTPDEMO
http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778
HttpServer as a: Flow[HttpRequest, HttpResponse]
HTTP Entity as a: Source[ByteString, _]
Streaming in Akka HTTPDEMO
http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778
HttpServer as a: Flow[HttpRequest, HttpResponse]
HTTP Entity as a: Source[ByteString, _]
Websocket connection as a: Flow[ws.Message, ws.Message]
Persistence Query (experimental)
“The Query Side” of Akka Persistence
Persistence Query (experimental)
Persistence Query Journals
akka/akka-persistence-cassandra 0.16 akka/akka @ leveldb-journal 2.4.8
dnvriend/akka-persistence-jdbc 2.3.3 scullxbones/akka-persistence-mongo 1.2.5
…and more, that I likely forgot about.
DEMO
Implementation of “data-pump” pattern.
Akka + Kafka = BFF
Reactive Kafka
+
Started by Krzysiek Ciesielski & Adam Warski @ SofwareMill.com
“ACKnowladged streams”
happy ACKing!
Kafka + Akka = BFF
Akka is Arbitrary processing.
Kafka is somewhat more than a message queue, but very focused on “the log”.
Spark shines with it’s data-science focus.
Kafka + Akka = BFF
Kafka + Akka = BFF
Streams talking to Actors &&
Actors talking to Streams
Streams <=> Actors inter-op
Source.actorRef (no back-pressure) Source.queue (safe)
Sink.actorRef (no back-pressure) Sink.actorRefWithAck (safe)
Exciting times ahead!
Next steps for Akka
Completely new Akka Remoting (goal: 1M+ msg/s (!)),(it is built using Akka Streams).
More integrations for Akka Streams stages, also dynamic fan-in/out A.K.A. “the Hub”.
Reactive Kafka polishing and stable release with SoftwareMill.
“Confirmed Streams” work from Reactive Kafka generalised.
Akka Typed likely to progress again.
Of course, continued maintenance of Cluster and others.
Happy hAkking!
Happy hAkking!
Announcing that there will be an announcement.
hAkknowledge (pilot name?)
“initiative” / “competition”
Akka <3 contributions• Easy to contribute:
• https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute • https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22
• Akka Stream Contrib • https://github.com/akka/akka-stream-contrib
• Mailing list:• https://groups.google.com/group/akka-user
• Public chat rooms: • http://gitter.im/akka/dev developing Akka • http://gitter.im/akka/akka using Akka
Thanks!
ktoso @ typesafe.com twitter: ktosopl
github: ktosoteam blog: letitcrash.com
home: akka.io
Thus spake the Master Programmer:
“After three days without programming,life becomes meaningless.”
Q/A (Now’s the time to ask things!)
ktoso @ typesafe.com twitter: ktosopl
github: ktosoteam blog: letitcrash.com
home: akka.io
©Typesafe 2016 – All Rights Reserved