Learning Kafka Streams with Scala

Click here to load reader

  • date post

    22-Jan-2018
  • Category

    Software

  • view

    1.207
  • download

    0

Embed Size (px)

Transcript of Learning Kafka Streams with Scala

  1. 1. Learning Kafka Streams with Scala Himanshu Gupta Lead Consultant Knoldus Software LLP Img src - http://danlebrero.com/2017/01/05/proof-of-concept-using-kafkastreams-and-ktables/
  2. 2. Agenda What is Kafka Streams? Why do we need to implement it in Scala? Challenge Solution Examples of Kafka Streams with Scala Demo
  3. 3. What is Kafka Streams? The easiest way to write mission-critical real- time applications and microservices with all the benefits of Kafka's server-side cluster technology
  4. 4. Write standard Java applications. No separate processing cluster required Elastic, highly scalable, fault-tolerant Equally viable for small, medium, & large use cases Exactly-once processing semantics Develop on Mac, Linux, Windows Deploy to containers, VM(s), bare metal, cloud Fully integrated with Kafka security Key Features
  5. 5. Where we can use it? Img src - https://kafka.apache.org/documentation/streams/
  6. 6. Example objectWordCountApplication{ defmain(args:Array[String]){ valtextLines:KStream[String,String]= builder.stream("TextLinesTopic") valwordCounts:KTable[String,Long]=textLines .flatMapValues(textLine=> textLine.toLowerCase.split("W+").toIterable.asJava) .groupBy((_,word)=>word) .count("Counts") wordCounts.to(Serdes.String(),Serdes.Long(), "WordsWithCountsTopic") valstreams:KafkaStreams=newKafkaStreams(builder,config) streams.start() } }
  7. 7. Why do we need to implement it in Scala? To provide a functional solution to a problem. To get better concurrency model. To write succinct and concise code. To have type safety or static typing.
  8. 8. Challenge No Scala API
  9. 9. Challenge No Scala API Is it really a challenge?
  10. 10. Challenge No Scala API Is it really a challenge? Yes
  11. 11. Challenge No Scala API Is it really a challenge? Yes Because Scala does not accept a function literal as a valid expression for any Single Abstract Method (SAM) type
  12. 12. Example Scala functions dont implement the Runnable and Callable interfaces even though theres a simply mapping. valr:Runnable=()=>println("Run!") error:typemismatch; found:()=>Unit required:Runnable valr:Runnable=()=> println("Run!") Compile
  13. 13. Solution???
  14. 14. Solution valr:Runnable= newRunnable(){ defrun()=println("Run!") } Explicitly implement Runnable interface anonymously in Scala
  15. 15. Solution valr:Runnable= newRunnable(){ defrun()=println("Run!") } Explicitly implement Runnable interface anonymously in Scala Whats the problem with this approach?
  16. 16. Solution valr:Runnable= newRunnable(){ defrun()=println("Run!") } Explicitly implement Runnable interface anonymously in Scala Its too verbose!
  17. 17. What can be other solution?
  18. 18. Alternative Solution Use Scala 2.12
  19. 19. Alternative Solution Use Scala 2.12 But why?
  20. 20. Alternative Solution Use Scala 2.12 But why? Because Scala 2.12 accepts a function literal as a valid expression for any Single Abstract Method (SAM) type
  21. 21. Example It works! It doesnt work!
  22. 22. Key Features of Scala 2.12 A trait compiles directly to an interface with default methods. This improves binary compatibility and Java interoperability. Scala and Java 8 interop is also improved for functional code, as methods that take functions can easily be called in both directions using lambda syntax. The FunctionN classes in Scalas standard library are now Single Abstract Method (SAM) types, and all SAM types are treated uniformly from type checking through code generation. No class file is generated for a lambda, invokedynamic is used instead.
  23. 23. Examples of Kafka Streams with Scala
  24. 24. Example #1 Simple Pipeline
  25. 25. Example #2 Mapped Pipeline (w/o SAM)
  26. 26. Example #2 Mapped Pipeline (with SAM)
  27. 27. Example #2 Aggregation Pipeline (w/o SAM)
  28. 28. Example #2 Aggregation Pipeline (with SAM)
  29. 29. There are other Examples too - https://github.com/knoldus/kafka-streams-scala-examples
  30. 30. DEMO
  31. 31. References https://kafka.apache.org/documentation/streams/ http://www.scala-lang.org/news/2.12.0/#lambda-syn
  32. 32. Thank You