Apache kafka

Click here to load reader

  • date post

    22-Jan-2017
  • Category

    Software

  • view

    514
  • download

    0

Embed Size (px)

Transcript of Apache kafka

  • Apache KafkaYour step 1 to Big Data

  • Who am I

    Saurabh Minni also on the web as @the100rabh

    Yet another developer in Bangalore

    I just love tinkering with different technologies

    Currently working as Technical Architect at Near

    Been part of planning for Barcamp Bangalore since 2007

    Author of Apache Kafka Cookbook - https://www.

    packtpub.com/big-data-and-business-intelligence/apache-

    kafka-cookbook

    https://twitter.com/the100rabhhttps://www.packtpub.com/big-data-and-business-intelligence/apache-kafka-cookbookhttps://www.packtpub.com/big-data-and-business-intelligence/apache-kafka-cookbookhttps://www.packtpub.com/big-data-and-business-intelligence/apache-kafka-cookbookhttps://www.packtpub.com/big-data-and-business-intelligence/apache-kafka-cookbook

  • Features of Kafka

    Fast

    Scalable

    Durable

    Distributed by Design

  • Architecture of an era

    Node Processor

  • Architecture of an scaling era - II

    Front End Node

    Hadoop

    Front End Node

    Front End Node

  • Things got a little more complicated

    Front End Node

    Hadoop

    Front End Node

    Front End Node

    Realtime Processing

    System

  • Apache Kafka to the rescue

    Front End Node

    Apache Kafka

    Front End Node

    Front End Node

    Hadoop

    Apache Storm

  • We need more

    Front End Node

    Apache Kafka Cluster

    Front End Node

    Front End Node

    Hadoop

    Apache Storm

    Apache Kafka Node

    Apache Kafka Node

    Apache Kafka Node

    Apache Kafka Node

  • Some more details

    Kafka Cluster

    Apache Kafka Node

    Apache Kafka Node

    Apache Kafka Node

    Apache Kafka Node

    Producer Kafka Topic

    Producer Kafka Topic

    Producer

    Consumer (Hadoop)

    Consumer (Apache Storm)

  • What is Apache Kafka

    Apache Kafka is publish-subscribe

    messaging rethought as a distributed

    commit log.

  • Monitoring Kafka

  • Which frameworks think Kafka is good

  • Which companies use Kafka

    Probably a better question would be

    which significantly large companies

    dont use Kafka

  • Where Apache Kafka gives trouble?

    Requires ZooKeeper

    Complexity in making a consumer

    A lot of manual processes

    Management tools leave a lot to be desired