2014 akka-streams-tokyo-japanese

Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL

Konrad `@ktosopl` Malawski

Akka Streams

Tokyo Scala User Group @ Tokyo 2014


hAkker @ 「アッカ」チームのメンバー。


typesafe.com geecon.org

Java.pl / KrakowScala.pl sckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl meetup.com/Lambda-Lounge-Krakow

hAkker @

http://typesafe.com

http://geecon.org

http://java.pl

http://krakowscala.pl

http://sckrk.com

http://www.meetup.com/Paper-Cup

http://gdgkrakow.pl

http://www.meetup.com/Lambda-Lounge-Krakow/

ありがとうございました, Benedict!

For translating the slides!

You?

?

You?

?

z ?

You?

?

z ?

?

You?

?

z ?

?

?

Streams

Streams“You cannot enter the same river twice” ~ Heraclitus

http://en.wikiquote.org/wiki/Heraclitus


Streamsライルタイムストリーム処理 Real Time Stream Processing

!

パブリッシャーにつけることが遅ければ、データは川のように流れるため、最初の要素を逃がしてしまう可能性がある When you attach “late” to a Publisher, you may miss initial elements – it’s a river of data.



Reactive Streams

Reactive Streams

!

!

Stream processing

Reactive Streams

Back-pressured !

Stream processing

Reactive Streams

Back-pressured Asynchronous

Stream processing

Reactive Streams

Back-pressured Asynchronous

Stream processing Standardised (!)

Reactive Streams: Goals

1. Back-pressured Asynchronous Stream processing !

2. Standard implemented by many libraries

Reactive Streams - Specification & TCK

http://reactive-streams.org


Reactive Streams - Who?


Kaazing Corp. rxJava @ Netflix,

reactor @ Pivotal (SpringSource), vert.x @ Red Hat,

Twitter, akka-streams @ Typesafe,

spray @ Spray.io, Oracle,

java (?) – Doug Lea - SUNY Oswego …


Reactive Streams - Inter-op


システム達を協力させたい. We want to make different implementations

co-operate with each other.




リアクティブストリームのプロトコールを

使ってシステム達を話し合わせる The different implementations “talk to each other”

using the Reactive Streams protocol.




リアクティブストリームSPIはユーザーAPIではない。

対象システムのライブラリを使うべき。 The Reactive Streams SPI is NOT meant to be user-api.

You should use one of the implementing libraries.


Back-pressure, なにですか？

Back-pressure? Example Without

Publisher[T] Subscriber[T]

Back-pressure? Example Without

速度が早いパブリッシャー Fast Publisher 速度が遅いサブスクライバー

Slow Subscriber

Back-pressure? Push + NACK model

Back-pressure? Push + NACK model通常の場合、サブスクライバーはバッファーがある.

Subscriber usually has some kind of buffer.

Back-pressure? Push + NACK model

Back-pressure? Push + NACK modelバッファーが溢れてしまった場合、どうなる？ What if the buffer overflows?

Back-pressure? Push + NACK model (a)

有界バッファーを利用して、溢れてしまったメッセージを落とし、再送を求める.

!Use bounded buffer, drop messages + require re-sending

有界バッファーを利用して、溢れてしまったメッセージを落とし、再送を求める.

!Use bounded buffer, drop messages + require re-sending

Back-pressure? Push + NACK model (a)

Kernel does this! Routers do this!

(TCP)

Back-pressure? Push + NACK model (b)バッファーの容量を増やす・・・まぁ、

メモリ容量がある限り！ Increase buffer size… Well, while you have memory available!

Back-pressure? Push + NACK model (b)

Back-pressure? Why NACKing is NOT enough

Back-pressure? Example NACKingたいへんですよ！

バッファーが溢れるまで間もなく！ Buffer overflow is imminent!

Back-pressure? Example NACKingデータを送る速力を落とす、または送信を停止する、

ようにパブリッシャーに告げる. Telling the Publisher to slow down / stop sending…

Back-pressure? Example NACKingメッセージが送信されている最中のため、

NACKが間に合わなかった！ NACK did not make it in time,

because M was in-flight!

Back-pressure? !

パブリッシャーの速度 < サブスクライバーの速度 speed(publisher) < speed(subscriber)

Back-pressure? Fast Subscriber, No Problem

速度が速いサブスクライバーなら、問題ない！No problem!

Back-pressure? Reactive-Streams

= “Dynamic Push/Pull”

プッシュ一方 - 速度が遅いサブスクライバーの場合、安全じゃない

Just push – not safe when Slow Subscriber !!

プル一方 - 速度早いサブスクライバーの場合、遅すぎる

Just pull – too slow when Fast Subscriber

Back-pressure? RS: Dynamic Push/Pull

プッシュ一方 - 速度が遅いサブスクライバーの場合、安全じゃない

Just push – not safe when Slow Subscriber !!

プル一方 - 速度早いサブスクライバーの場合、遅すぎる

Just pull – too slow when Fast Subscriber !

Solution: 動的な調整（リアクティブストリーム）

Dynamic adjustment (Reactive Streams)


Back-pressure? RS: Dynamic Push/Pull速度が遅いサブスクライバーは自分のバッファーが3つの要素まで受け取られる。パブリッシャーはサブスクライバーのバッファーが溢れるほどデータを送らない。Slow Subscriber sees it’s buffer can

take 3 elements. Publisher will never blow up it’s buffer.

Back-pressure? RS: Dynamic Push/Pull速度が速いパブリッシャーは最大でも３つの要素を送る。これは、プル型のback-pressure. Fast Publisher will send at-most 3 elements. This is pull-based-backpressure.


速度が速いサブスクライバーは、実際のデータが届く前に、たくさんリクエストを送ることができる！Fast Subscriber can issue more Request(n), before more data arrives!

Back-pressure? RS: Dynamic Push/Pullパブリッシャーはサブスクライバーの全てのリクエストを溜める. Fast Subscriber can issue more Request(n), before more data arrives. Publisher can accumulate demand.

Back-pressure? RS: Accumulate demand

パブリッシャーはサブスクライバーの全ての

リクエストを溜める. Publisher accumulates total demand per subscriber.

Back-pressure? RS: Accumulate demand溜まった要素をパブリッシュすることは安全だ。サブスクライバーのバッファーは溢れない。 Total demand of elements is safe to publish. Subscriber’s buffer will not overflow.

Back-pressure? RS: Requesting “a lot”速度が速いサブスクライバーは、パブリッシャーに対し要求が多くても良い。これは「パブリッシャー・プッシュ」だ。バッファー容量は知られているので、安全だ。

Back-pressure? RS: Requesting “a lot”

Fast Subscriber, can request “a lot” from Publisher. This is effectively “publisher push”, and is really fast. Buffer size is known and this is safe.


MAX速度


安全！溢れること絶対ない！

MAX速度

わなにですか？

Akka = アッカAkka is a high-performance concurrency library for Scala and Java. !

At it’s core it focuses on the Actor Model:

アクターができること:

• メッセージを送信・受信する (Send / receive messages)

• アクターを作る (Create Actors)

• 自分の動作を変える (Change it’s behaviour)

Akka = アッカAkka is a high-performance concurrency library for Scala and Java. !

At it’s core it focuses on the Actor Model:

AkkaAkka has multiple modules: !

Akka-camel: integration Akka-remote: remote actors Akka-cluster: clustering Akka-persistence: CQRS / Event Sourcing Akka-streams: stream processing …

Akka Streams 0.7 early preview

Akka Streams – Linear Flow


FlowFrom[Double].map(_.toInt). [...]

ソースはまだ付けられていない.

パイプはダブルを処理する準備を整いた. No Source attached yet.

“Pipe ready to work with Doubles”.


implicit val sys = ActorSystem("tokyo-sys")!!

アクターが住んでいる世界.

AkkaStreamsはアクターを使うため、

アクターシステムは必要.

!ActorSystem is the world in which Actors live in.

AkkaStreams uses Actors, so it needs ActorSystem.


implicit val sys = ActorSystem("tokyo-sys")!implicit val mat = FlowMaterializer()!

ストリームをどうやって具体化するかの

ロジックが含まれている.

!Contains logic on HOW to materialise the stream.



単純にアクターか、もしくは実装されていれば、

Apache Spark (?!) !

A materialiser can choose HOW to materialise, it could even use Apache Spark (?!) if someone would implement that… :-)



バッファーの容量を設定できるなど !

You can configure it’s buffer sizes etc.



val foreachSink = ForeachSink[Int](println)!val mf = FlowFrom(1 to 3).withSink(foreachSink).run()

Uses the implicit FlowMaterializer



val foreachSink = ForeachSink[Int](println)!val mf = FlowFrom(1 to 3).withSink(foreachSink).run()(mat)


val mf = FlowFrom[Int].! map(_ * 2).! withSink(ForeachSink(println)) // needs source,! // can NOT run

走らせるためにソースが必要！


val f = FlowFrom[Int].! map(_ * 2).!! ! ! withSink(ForeachSink(i => println(s"i = $i”))).! ! ! // needs Source to run!

インプットが必要

走らせるためにソースが必要！



!! ! ! f.withSource(IterableSource(1 to 10)).run()

準備完了！

Akka Streams – Flows are reusable

!! ! ! f.withSource(IterableSource(1 to 10)).run()! ! ! ! f.withSource(IterableSource(1 to 100)).run()! ! ! ! f.withSource(IterableSource(1 to 1000)).run()

Akka Streams <-> Actors – Advanced

val subscriber = system.actorOf(Props[SubStreamParent], ”parent")!!FlowFrom(1 to 100).! map(_.toString).! filter(_.length == 2).! drop(2).! groupBy(_.last).! publishTo(ActorSubscriber(subscriber))!



各グループもストリームだよ！ Each “group” is a stream too! It’s a “Stream of Streams”.

Akka Streams <-> Actors – Advanced! groupBy(_.last).

「１１」をグループ「１」に、「１２」をグループ「２」になど GroupBy groups “11” to group “1”, “12” to group “2” etc.


サブスクライバーに[グループキー、サブストリームフロー]を提供する

It offers (groupKey, subStreamFlow) to Subscriber


子供を起動させ、サブーフローを扱わせる It can then start children, to handle the sub-flows!


例えば、グループ毎に1人の子供 For example, one child for each group.



普通 Akka Actor, will consume SubStream offers.


class SubStreamParent extends ActorSubscriber ! with ImplicitFlowMaterializer ! with ActorLogging {!! override def requestStrategy = OneByOneRequestStrategy!! override def receive = {! case OnNext((groupId: String, subStream: FlowWithSource[_, _])) =>!! val subSub = context.actorOf(Props[SubStreamSubscriber], ! s"sub-$groupId")! subStream.publishTo(ActorSubscriber(subSub))! }!}!


class SubStreamParent extends ActorSubscriber {!! override def requestStrategy = OneByOneRequestStrategy!! override def receive = {! case OnNext(n: String) => println(s”n = $n”) ! }!}!

Akka Streams – GraphFlow

GraphFlow


Linear Flows or

non-akka pipelines

Could be another RS implementation!


Fan-out elements and

Fan-in elements


Fan-out elements and

Fan-in elements

Now you need a FlowGraph


// first define some pipeline pieces!val f1 = FlowFrom[Input].map(_.toIntermediate)!val f2 = FlowFrom[Intermediate].map(_.enrich)!val f3 = FlowFrom[Enriched].filter(_.isImportant)!val f4 = FlowFrom[Intermediate].mapFuture(_.enrichAsync)!!// then add input and output placeholders!val in = SubscriberSource[Input]!val out = PublisherSink[Enriched]!

Akka Streams – GraphFlowval b3 = Broadcast[Int]("b3")!val b7 = Broadcast[Int]("b7")!val b11 = Broadcast[Int]("b11")!val m8 = Merge[Int]("m8")!val m9 = Merge[Int]("m9")!val m10 = Merge[Int]("m10")!val m11 = Merge[Int]("m11")!val in3 = IterableSource(List(3))!val in5 = IterableSource(List(5))!val in7 = IterableSource(List(7))!


// First layer!in7 ~> b7!b7 ~> m11!b7 ~> m8!!in5 ~> m11!!in3 ~> b3!b3 ~> m8!b3 ~> m10!


!// Second layer!m11 ~> b11!b11 ~> FlowFrom[Int].grouped(1000) ~> resultFuture2 !b11 ~> m9!b11 ~> m10!!m8 ~> m9!


!// Third layer!m9 ~> FlowFrom[Int].grouped(1000) ~> resultFuture9!m10 ~> FlowFrom[Int].grouped(1000) ~> resultFuture10!


val resultFuture2 = FutureSink[Seq[Int]]!val resultFuture9 = FutureSink[Seq[Int]]!val resultFuture10 = FutureSink[Seq[Int]]!!val g = FlowGraph { implicit b =>! // ...! m10 ~> FlowFrom[Int].grouped(1000) ~> resultFuture10! // ...!}.run()!!Await.result(g.getSinkFor(resultFuture2), 3.seconds).sorted! should be(List(5, 7))

Sinks and Sources are “keys” which can be addressed within the graph


!val g = FlowGraph {}!

FlowGraphは不変で、安全に共有でき、

何度も使いまわせる！

!FlowGraph is immutable and safe to share and re-use! Think of it as “the description” which then gets “run”.

Available Elements 0.7 early preview

Available Sources• FutureSource • IterableSource • IteratorSource • PublisherSource • SubscriberSource • ThunkSource • TickSource (timer based) • 簡単に自分のものを追加できる！

… easy to add your own!

0.7 early preview

Available operations• buffer • collect • concat • conflate • drop / dropWithin • take / takeWithin • filter • fold • foreach • groupBy • grouped • map • onComplete • prefixAndTail

• broadcast • merge / “generalised merge” • zip • 自分のオペレーションを追加する事は可能！

… possible to add your own!

0.7 early preview

Available Sinks• BlackHoleSink • FoldSink • ForeachSink • FutureSink • OnCompleteSink • PublisherSink / FanoutPublisherSink • SubscriberSink • 簡単に自分のものを追加できる！

… easy to add your own!

0.7 early preview

Spray => Akka-Http && ReactiveStreams

Spray is now merged into Akka, as Akka-Http Works on Reactive Streams

Streaming end-to-end!

Links• http://akka.io • http://reactive-streams.org • https://groups.google.com/group/akka-user <- ask questions here! • http://akka.io/news/2014/09/12/akka-streams-0.7-released.html

http://akka.io


https://groups.google.com/group/akka-user

http://akka.io/news/2014/09/12/akka-streams-0.7-released.html

ありがとうございました！ Ask questions, get Stickers!

http://akka.io

ktoso @ typesafe.com twitter: ktosopl github: ktoso team blog: letitcrash.com

http://akka.io

http://twitter.com/ktosopl

http://github.com/ktoso

http://letitcrash.com

2014 akka-streams-tokyo-japanese

Technology

Transcript of 2014 akka-streams-tokyo-japanese