AKKA STREAMSFROM ZERO TO KAFKACreated by / Mark Harrison @markglh

HOW IT ALL BEGAN“Reactive Streams is an initiative to provide astandard for asynchronous stream processing

with non-blocking back pressure. Thisencompasses efforts aimed at runtime

environments (JVM and JavaScript) as well asnetwork protocols.”

WHYEf�ciently processing large indeterminate streams is hardAvoiding blocking is essential to maximise performanceEvery stage in the stream needs to be able to push and pullWe don't want to overload (or starve!) downstreamconsumers...

HOWTreat data as a stream of elementsAsynchronous non-blocking data and demand �owsDemand �ows upstream, causing data to �ow downstreamData �ow is therefore restricted by demand

Back Pressure!!Demand happens on a separate �ow!

WHATThe Reactive Streams speci�cation is just that

A collection of interfaces methods and protocolsProvides example implementations and a TCK forveri�cationAimed at providing a way to build commonimplementations

INTRODUCING AKKA STREAMS!!AKKA'S IMPLEMENTATION OF REACTIVE STREAMS

DESIGN PRINCIPLESExplicitness over magic (I'm looking at you Shapeless!)Fully composable

Each component, or set of componenents can be combinedEach building block is immutableFully compatible with other Reactive Stream implementations

BUILDING BLOCKS

BUILDING BLOCKS CONT...Source

Traditionally known as a producerSupplies messages that will �ow downstreamExactly one output stream

SinkTraditionally known as a consumerEnd point of the stream, this is where messages end up

BUILDING BLOCKS CONT...Flow

A processing stage in the StreamUsed to compose StreamsExactly one input and one output streamSee also BidirectionalFlow (two in -> two out)

BUILDING BLOCKS CONT...RunnableGraphs

A pre-assembled set of Stream components, packaged intoa Graph.All exposed ports are connected (between a Source andSink)This can then be Materialized

BUILDING BLOCKS CONT...Composite Flows

It is possible to wrap several components into morecomplex onesThis composition can then be treated as one block

Partial Flow GraphsAn incomplete Flow (Graph)Can be used to construct more complex Graphs easily

BUILDING BLOCKS CONT...Materializer

Once complete, the �ow is Materialized in order to startstream processingSupports fully distributed stream processing

Each step must be either serializable immutable valuesor ActorRefs

Fails immediately at runtime if the Graph isn't complete

ERRORS VS FAILURESErrors handlied within the stream as normal data elements

Passed using the onNext functionFailure means that the stream itself has failed and is collapsing

Raises the onError signal... (???)Each block in the �ow can choose to absorb or propagate theerrors

Possibly resulting the the complete collapse of the �ow

FIRST THINGS FIRSTWe need to create an ActorSystem and Materializer

implicit val system = ActorSystem("actors") implicit val materializer = ActorMaterializer()

SIMPLE STREAMWe need to create an ActorSystem and Materializer

Source(1 to 5) .filter(_ < 3) // 1, 2 .map(_ * 2) // 2, 4 .to(Sink.foreach(println)) .run() //prints 2 4

COMPOSING ELEMENTS TOGETHERWe can combine multiple components together

Composing elements together val nestedSource = Source(1 to 5) .map(_ * 2) val nestedFlow = Flow[Int] .filter(_ <= .map(_ + 2) val sink = Sink.foreach(println) //link up the Flow to a Sink val nestedSink = nestedFlow.to(Sink.foreach(println)) // Create a RunnableGraph - and run it! Prints 4 6 nestedSource.to(nestedSink).run()

COMPOSING ELEMENTS TOGETHER CONT...Alternatively we could do this, linking them in one step

nestedSource .via(nestedFlow) .to(Sink.foreach(println(_)))

COMPOSING ELEMENTS TOGETHER CONT...

GRAPH PROCESSING STAGESFan OutBroadcast[T] – (1 input, N outputs)Balance[T] – (1 input, N outputs)...

Fan InMerge[In] – (N inputs , 1 output)...

Timer DrivengroupedWithin(Int, Duration)

Groups elements when either the number or duration isreached (whichever is �rst). Very useful for batchingmessages.

See the Akka Stream docs for more!

GRAPH PROCESSING STAGES CONT...

THE GRAPH DSLWhenever you want to perform multiple operations tocontrol the Flow of a Graph, manually constructing them asabove can become very clumbersome and tedius, not tomentioned hard to maintain.For this reason the Akka team have written a DSL to helpwrite complex Graphs.

THE GRAPH DSLval g = FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => //This provides the DSL import FlowGraph.Implicits._ val in = Source(1 to 3) val out = Sink.foreach(println) //2 outputs, 2 inputs val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val f1, f2, f3, f4 = Flow[Int].map(_ + 10) in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge } g.run() //Prints 31 31 32 32 33 33

THE GRAPH DSL CONT...

EXAMPLE - REACTIVE KAFKAThe guys at SoftwareMill have implemented a wrapper forApache Kafka

Tried and tested by yours trulyhttps://github.com/softwaremill/reactive-kafka

EXAMPLE - REACTIVE KAFKA CONT...Source is a Kafka ConsumerSink is a Kafka Publisher

val kafka = new ReactiveKafka() val publisher: Publisher[StringKafkaMessage] = kafka.consume( ConsumerProperties(...) ) val subscriber: Subscriber[String] = kafka.publish( ProducerProperties(...) ) Source(publisher).map(_.message().toUpperCase) .to(Sink(subscriber)).run()

A REAL WORLD EXAMPLE

A REAL WORLD EXAMPLE CONT...FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => import FlowGraph.Implicits._ val in = Source(kafkaConsumer) val out = Sink.foreach(println) val bcast = builder .add(Broadcast[StringKafkaMessage](2)) val merge = builder .add(Merge[StringKafkaMessage](2)) val parser1, parser2 = Flow[StringKafkaMessage] .map(...) val group = Flow[StringKafkaMessage].grouped(4) in ~> bcast ~> parser1 ~> merge ~> group ~> out bcast ~> parser2 ~> merge }.run()

IT'S BEEN EMOTIONAL...Slides at

Follow me

http://markglh.github.io/AkkaStreams-Madlab-Slides

@markglh