Reactive Kafka with Akka Streams

Click here to load reader

  • date post

    16-Apr-2017
  • Category

    Software

  • view

    4.035
  • download

    4

Embed Size (px)

Transcript of Reactive Kafka with Akka Streams

  • Reactive Kafka with Akka Streams

    Krzysztof CiesielskiReactive Summit 2016, Austin, TX

  • about.me

    Scala developer @

    Editor of ScalaTimes

    Open Source (akka-stream-kafka, scala-clippy and other)

    Premier Consulting Partner

  • akka-stream-kafka

    Formely reactive-kafka

    A library for accessing Kafka as stages of Akka Streams

  • Apache Kafkaa message log

    8 7 6 5 4 3 2 1

    Producer

    Producer

    Consumer

    Consumer

    topic topic1

  • Apache Kafkaa distributed message log

    4 3 2 1

    Producer

    4 3 2 1

    partition 1

    partition 2

    topic1

    topic1

  • Apache Kafkaa distributed message log

    Consumer

    balancing4 3 2 1

    Producer

    4 3 2 1

    partition 1

    partition 2

    topic1

    topic1

  • Apache Kafkaa distributed message log

    Consumer1

    Consumer2

    4 3 2 1

    Producer

    4 3 2 1

    partition 1

    partition 2

    topic1

    topic1

  • Apache Kafkaa distributed message log

    4 3 2 1

    4 3 2 1

    partition 1

    partition 3

    topic1

    topic1

    Consumer1

    Consumer2

    4 3 2 1

    partition 2topic1manual assignment

  • Apache KafkaUses Zookeeper

    Leader election

    Cluster coordination

    Shared configuration

  • Apache Kafka

    Decoupling of producers and consumers

  • Apache KafkaCommit

    8 7 6 5 4 3 2 1 Consumer

    commit(p1, o3)

    msg1, msg2

  • 8 7 6 5 4 3 2 1 Consumer

    Apache KafkaCommit

  • Apache KafkaCommit

    8 7 6 5 4 3 2 1 New consumermsg3

  • Akka Streams

    DSL for describing data transformation pipelines.

    Uses Actor Model.

    Implements the Reactive Streams specification.

  • Akka Streams

    Focused on backpressure and async processing.

    Comes with a powerful test kit.

    Extensible.

  • AlpakkaInitiative to build a rich set of connectors

    HTTP

    Streaming TCP

    Streaming File IO

    Reactive Streams (MongoDB Reactive Streams Java Driver)

  • Akka Streams + Kafka

  • Akka Streams + Kafka

  • Akka Streams + Kafka

  • Akka Streams + Kafka

  • akka-stream-kafka

    Started as reactive-kafka @ SoftwareMill

    First users

    Alexey Romanchuk has rebuilt the library

    Akka Team joined (Patrik Nordwall, Endre Varga)

  • Example: plain consumer

  • Example: plain consumer

  • Example: plain consumer

    +async processing

    +error handling

    +backpressure

  • Consumer Actor

    SourceStage

    ConsumerActorconsumer (Kafka API)

    RequestMessages

    demand messages

    Messages

    Scheduled Poll

    poll()

    dispatcher

  • Plain consumer performance

    1,202,081944,746

  • Apache KafkaCommit

    8 7 6 5 4 3 2 1 Consumer

    commit(p1, o3)

    msg1, msg2

  • Consumer Actor and commit

    SourceStage

    ConsumerActorconsumer (Kafka API)

    ? Commit

    commit()

    Scheduled Poll

    poll()

    dispatcher

    msg

    msg

    commitAsync()

    callback

    Request Messages

  • Committable Source

  • Committable Source

  • Batched Committable Source

  • Batched Committable Source

  • At-least-once delivery

    456,695

    1,194,331

  • External commit source

  • Producer

  • Producer Vanilla API

  • Producer as a Flow

  • Producer as a Flow

  • From Kafka to Kafka

  • From Kafka to KafkaSourceStage

    Producer Stage

    msg send()

    callback()msg

    msg mapAsync

    ConsumerActor

  • From Kafka to Kafka

  • Its streaming streams

    backpressure per partition

  • Source[(TopicPartition, Source[CommittableMessage[Array[Byte], String], NotUsed]), Control]

    Its streaming streams

  • Error handling

  • Java API

  • Kafka Streams

  • Kafka Streamsakka-stream-kafka

    Kafka Streams

    Scala + Java, backpressure, async processing, interop with Akka Streams

    Java, stateful processors, windowing, joining, aggregation operations

  • Thank you!

    Join us!https://gitter.im/akka/reactive-kafka

    @kpciesielski

    https://gitter.im/akka/reactive-kafka