Apache KafkaYour step 1 to Big Data

Who am I

Saurabh Minni also on the web as @the100rabh

Yet another developer in Bangalore

I just love tinkering with different technologies

Currently working as Technical Architect at Near

Been part of planning for Barcamp Bangalore since 2007

Author of Apache Kafka Cookbook - https://www.

packtpub.com/big-data-and-business-intelligence/apache-

kafka-cookbook

Features of Kafka

➤ Fast

➤ Scalable

➤ Durable

➤ Distributed by Design

Architecture of an era

Node Processor

Architecture of an scaling era - II

Front End Node

Hadoop

Front End Node

Front End Node

Things got a little more complicated

Front End Node

Hadoop

Front End Node

Front End Node

Realtime Processing

System

Apache Kafka to the rescue

Front End Node

Apache Kafka

Front End Node

Front End Node

Hadoop

Apache Storm

We need more

Front End Node

Apache Kafka Cluster

Front End Node

Front End Node

Hadoop

Apache Storm

Apache Kafka Node

Apache Kafka Node

Apache Kafka Node

Apache Kafka Node

Some more details

Kafka Cluster

Apache Kafka Node

Apache Kafka Node

Apache Kafka Node

Apache Kafka Node

Producer Kafka Topic

Producer Kafka Topic

Producer

Consumer (Hadoop)

Consumer (Apache Storm)

What is Apache Kafka

Apache Kafka is publish-subscribe

messaging rethought as a distributed

commit log.

Monitoring Kafka

Which frameworks think Kafka is good

Which companies use Kafka

Probably a better question would be

which significantly large companies

don’t use Kafka

Where Apache Kafka gives trouble?

➤ Requires ZooKeeper

➤ Complexity in making a consumer

➤ A lot of manual processes

➤ Management tools leave a lot to be desired