Akka Streams - From Zero to Kafka

Click here to load reader

download Akka Streams - From Zero to Kafka

of 31

  • date post

    16-Apr-2017
  • Category

    Software

  • view

    93
  • download

    3

Embed Size (px)

Transcript of Akka Streams - From Zero to Kafka

  • AKKA STREAMSFROM ZERO TO KAFKACreated by / Mark Harrison @markglh

    https://github.com/markglhhttp://twitter.com/markglh

  • HOW IT ALL BEGANReactive Streams is an initiative to provide astandard for asynchronous stream processing

    with non-blocking back pressure. Thisencompasses efforts aimed at runtime

    environments (JVM and JavaScript) as well asnetwork protocols.

  • WHYEfciently processing large indeterminate streams is hardAvoiding blocking is essential to maximise performanceEvery stage in the stream needs to be able to push and pullWe don't want to overload (or starve!) downstreamconsumers...

  • HOWTreat data as a stream of elementsAsynchronous non-blocking data and demand owsDemand ows upstream, causing data to ow downstreamData ow is therefore restricted by demand

    Back Pressure!!Demand happens on a separate ow!

  • WHATThe Reactive Streams specication is just that

    A collection of interfaces methods and protocolsProvides example implementations and a TCK forvericationAimed at providing a way to build commonimplementations

  • INTRODUCING AKKA STREAMS!!AKKA'S IMPLEMENTATION OF REACTIVE STREAMS

  • DESIGN PRINCIPLESExplicitness over magic (I'm looking at you Shapeless!)Fully composable

    Each component, or set of componenents can be combinedEach building block is immutableFully compatible with other Reactive Stream implementations

  • BUILDING BLOCKS

  • BUILDING BLOCKS CONT...Source

    Traditionally known as a producerSupplies messages that will ow downstreamExactly one output stream

    SinkTraditionally known as a consumerEnd point of the stream, this is where messages end up

  • BUILDING BLOCKS CONT...Flow

    A processing stage in the StreamUsed to compose StreamsExactly one input and one output streamSee also BidirectionalFlow (two in -> two out)

  • BUILDING BLOCKS CONT...RunnableGraphs

    A pre-assembled set of Stream components, packaged intoa Graph.All exposed ports are connected (between a Source andSink)This can then be Materialized

  • BUILDING BLOCKS CONT...Composite Flows

    It is possible to wrap several components into morecomplex onesThis composition can then be treated as one block

    Partial Flow GraphsAn incomplete Flow (Graph)Can be used to construct more complex Graphs easily

  • BUILDING BLOCKS CONT...Materializer

    Once complete, the ow is Materialized in order to startstream processingSupports fully distributed stream processing

    Each step must be either serializable immutable valuesor ActorRefs

    Fails immediately at runtime if the Graph isn't complete

  • ERRORS VS FAILURESErrors handlied within the stream as normal data elements

    Passed using the onNext functionFailure means that the stream itself has failed and is collapsing

    Raises the onError signal... (???)Each block in the ow can choose to absorb or propagate theerrors

    Possibly resulting the the complete collapse of the ow

  • FIRST THINGS FIRSTWe need to create an ActorSystem and Materializer

    implicit val system = ActorSystem("actors") implicit val materializer = ActorMaterializer()

  • SIMPLE STREAMWe need to create an ActorSystem and Materializer

    Source(1 to 5) .filter(_ < 3) // 1, 2 .map(_ * 2) // 2, 4 .to(Sink.foreach(println)) .run() //prints 2 4

  • COMPOSING ELEMENTS TOGETHERWe can combine multiple components together

    Composing elements together val nestedSource = Source(1 to 5) .map(_ * 2) val nestedFlow = Flow[Int] .filter(_

  • COMPOSING ELEMENTS TOGETHER CONT...Alternatively we could do this, linking them in one step

    nestedSource .via(nestedFlow) .to(Sink.foreach(println(_)))

  • COMPOSING ELEMENTS TOGETHER CONT...

  • GRAPH PROCESSING STAGESFan OutBroadcast[T] (1 input, N outputs)Balance[T] (1 input, N outputs)...

    Fan InMerge[In] (N inputs , 1 output)...

    Timer DrivengroupedWithin(Int, Duration)

    Groups elements when either the number or duration isreached (whichever is rst). Very useful for batchingmessages.

    See the Akka Stream docs for more!

    http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/stages-overview.html

  • GRAPH PROCESSING STAGES CONT...

  • THE GRAPH DSLWhenever you want to perform multiple operations tocontrol the Flow of a Graph, manually constructing them asabove can become very clumbersome and tedius, not tomentioned hard to maintain.For this reason the Akka team have written a DSL to helpwrite complex Graphs.

  • THE GRAPH DSLval g = FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => //This provides the DSL import FlowGraph.Implicits._ val in = Source(1 to 3) val out = Sink.foreach(println) //2 outputs, 2 inputs val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val f1, f2, f3, f4 = Flow[Int].map(_ + 10) in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge } g.run() //Prints 31 31 32 32 33 33

  • THE GRAPH DSL CONT...

  • EXAMPLE - REACTIVE KAFKAThe guys at SoftwareMill have implemented a wrapper forApache Kafka

    Tried and tested by yours trulyhttps://github.com/softwaremill/reactive-kafka

    https://github.com/softwaremill/reactive-kafka

  • EXAMPLE - REACTIVE KAFKA CONT...Source is a Kafka ConsumerSink is a Kafka Publisher

    val kafka = new ReactiveKafka() val publisher: Publisher[StringKafkaMessage] = kafka.consume( ConsumerProperties(...) ) val subscriber: Subscriber[String] = kafka.publish( ProducerProperties(...) ) Source(publisher).map(_.message().toUpperCase) .to(Sink(subscriber)).run()

  • A REAL WORLD EXAMPLE

  • A REAL WORLD EXAMPLE CONT...FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => import FlowGraph.Implicits._ val in = Source(kafkaConsumer) val out = Sink.foreach(println) val bcast = builder .add(Broadcast[StringKafkaMessage](2)) val merge = builder .add(Merge[StringKafkaMessage](2)) val parser1, parser2 = Flow[StringKafkaMessage] .map(...) val group = Flow[StringKafkaMessage].grouped(4) in ~> bcast ~> parser1 ~> merge ~> group ~> out bcast ~> parser2 ~> merge }.run()

  • IT'S BEEN EMOTIONAL...Slides at

    Follow me

    http://markglh.github.io/AkkaStreams-Madlab-Slides

    @markglh

    http://markglh.github.io/AkkaStreams-Madlab-Slideshttps://twitter.com/markglh