Apache KafkaA high throughput distributed messaging system

What is Kakfa?

Kafka is a distributed publish-subscribe messaging system rethought as a distributed commit log.

It’s designed to be Fast Scalable Durable

When used in the right way and for the right use case, Kafka has unique attributes that make it a highly attractive option for data integration.

Publish subscribe messaging system Kafka maintains feeds of messages in categories called topics Producers are processes that publish messages to one or more topics Consumers are processes that subscribe to topics and process the feed

of published messages

Subscriber

Subscriber

Subscriber

Publisher

Message

Message

MessageMessage Topic

Kafka cluster Since Kafka is distributed in nature, Kafka is run as a cluster. A cluster is typically comprised multiple servers; each of which is called a

broker. Communication between the clients and the servers takes place over TCP protocol

Kafka clusterConsumer

Consumer

Consumer

Producer

Producer

Producer

Broker 1Topic 1 Topic 2

Broker 2Topic 1 Topic 2

Zookeeper

Deep dive into high level abstractions

Topic To balance load, a topic is

divided into multiple partitions and replicated across brokers.

Partitions are ordered, immutable sequences of messages that’s continually appended i.e. a commit log.

The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition.

Partitions allow a topic’s log to scale beyond a size that will fit on a single server (i.e. a broker) and act as the unit of parallelism

The partitions of a topic are distributed over the brokers in the Kafka cluster where each broker handles data and requests for a share of the partitions.

For fault tolerance, each partition is replicated across a configurable number of brokers.

Distribution and partitions

Distribution and fault tolerance

Each partition has one server which acts as the "leader" and zero or more servers which act as "followers".

The leader handles all read and write requests for the partition while the followers passively replicate the leader.

If the leader fails, one of the followers will automatically become the new leader.

Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.

Retention

The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time; after which it will be discarded to free up space.

Metadata retained on a per-consumer basis is the position of the consumer in the log, called the offset; which is controlled by consumer.

Normally a consumer will advance its offset linearly as it reads messages, but it can consume messages in any order it likes.

Kafka consumers can come and go without much impact on the cluster or on other consumers.

Producers Producers publish data to the topics by assigning messages to a

partition within the topic either in a round-robin fashion or according to some semantic partition function (say based on some key in the message).

Consumers Kafka offers a single consumer abstraction called consumer group that

generalises both queue and topic. Consumers label themselves with a consumer group name. Each message published to a topic is delivered to one consumer instance

within each subscribing consumer group. If all the consumer instances have the same consumer group, then

this works just like a traditional queue balancing load over the consumers.

If all the consumer instances have different consumer groups, then this works like publish-subscribe and all messages are broadcast to all consumers.

Consumer groups Topics have a small number of consumer groups, one for each logical

subscriber. Each group is composed of many consumer instances for scalability and fault

tolerance.

Ordering guarantees Kafka assigns partitions in a topic to consumers in a consumer group

so, each partition is consumed by exactly one consumer in the group. Limitation: there cannot be more consumer instances in a consumer group

than partitions. Provides a total order over messages within a partition, not between different

partitions in a topic.

Comaprison

Kafka JMS message broker; Rabbit MQA fire hose of events arriving at rate of approximately 100k+/sec

Messages arriving at a rate of 20k+/sec

‘At least once‘ processed as data is read with an offset within a partition.

Exactly once processed by consumers

Producer-centric. Doesn't have message acknowledgements as consumers track messages consumed.

Broker-centric. Uses the broker itself to maintain state of what's consumed (via message acknowledgements)

Supports both online and batch consumers that may be online or offline. It also supports producer message batching - it's designed for holding and distributing large volumes of messages at a very low latency.

Consumers are mostly online, and any messages "in wait" (persistent or not) are held opaquely.

Comaprison

Kafka JMS message broker; Rabbit MQProvides a rudimentary routing. It uses topic for exchanges.

Provides rich routing capabilities with Advanced Message Queuing Protocol’s (AMQP) exchange, binding and queuing model.

Makes distributed cluster explicit, by forcing the producer to know it is partitioning a topic's messages across several nodes.

Makes the distributed cluster transparent, as if it were a virtual broker

Preserves ordered delivery within a partition

Almost always unordered delivery. AMQP model says "one producer channel, one exchange, one queue, one consumer channel" is required for in-order delivery

Throttling is un-necessary

The whole job of Kafka is to provide a "shock absorber" between the flood of events and those who want to consume them in their own way.

Performance benchmark

500,000 messages published per second 22,000 messages consumed per second on a 2-node cluster with 6-disk RAID 10. See

research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

Key benefits

Horizontally scalable It’s a distributed system can be elastically and transparently expanded

with no downtime High throughput

High throughput is provided for both publishing and subscribing, due to disk structures that provide constant performance even with many terabytes of stored messages

Reliable delivery Persists messages on disk, and provides intra-cluster replication Supports large number of subscribers and automatically balances consumers

in case of failure.

Use cases

Common use cases include1. Stream processing, Event sourcing or a replacement for a more traditional

message broker2. Website activity tracking - original use case for Kafka3. Metrics collection and monitoring - centralized feeds of operational data4. Log aggregation

Getting practical

Download and extract Kafka

Download the archive from kafka.apache.org/downloads.html and extract it

Kafka uses ZooKeeper for cluster coordination Kafka uses ZooKeeper; which enables

highly reliable distributed coordination so, one needs to first start a ZooKeeper server.

Kafka bundles a single-node ZooKeeper instance. Single node zookeeper cluster does NOT run a leader and a follower.

Typically exchanged metadata include Kafka broker addresses Consumed messages offset

Common Challenges faced by distributed system Outages Co-ordination of tasks Reduction of operational complexity Consistency and ordering guarantees

Zookeeper to rescue

Apache Zookeeper: Definition

Centralised service for Maintaining configuration information Naming Distributed synchronisation and providing group services.

Apache Zookeeper: Features

Distributed consistent data store which favours consistency over everything else.

High availability - Tolerates a minority of an ensemble members being unavailable and continues to function correctly. In an ensemble of n members where n is an odd number, the loss of (n-1)/2 members can

be tolerated. High performance - All the data is stored in memory and benchmarked at 50k

ops/sec but the numbers really depend on your servers and network Tuned for read heavy write light work load. Reads are served from the node to which a

client a connected. Provides strictly ordered access for data.

Atomic write guarantees in the order they're sent to zookeeper. Writes are acknowledged and changes are also seen in the order they occurred.

Apache Zookeeper: Operation basics A cluster is an ensemble with a leaders and several followers Read requests are serviced from each server using its local replica but write

request are forwarded to a leader. When the leader receives a write request, it calculates what the state of the system is when the write is to be applied and transforms this into a transaction that captures this new state.

When zookeeper starts, it goes through a loading algorithm where by one node of the cluster is elected to act as the leader. At any given point in time, only one node acts as a leader.

Apache Zookeeper: Operation basics Clients create a state full session (i.e. with heartbeats through an open socket)

when they connect to a node of an ensemble. Number of open sockets available on a zookeeper node will limit the number of clients that connect to it.

When a cluster member dies, clients notice a disconnect event and thus reconnect themselves to another member of the quorum.

Session (i.e. state of the client connected to node of an ensemble) stay alive when the client goes down, as the session events go through the leader and gets replicated in a cluster onto another node.

When the leader goes down, remaining members of the cluster will re-elect a new leader using a atomic broadcast consensus algorithm. Cluster remains unavailable only when it re-elects a new leader.

1. Start ZooKeeper

2. Set-up a cluster with 3 brokers

2. Adjust broker configuration files

3. Start kafka server 1

3. Start kafka server 2

3. Start kafka server 3

3. Servers 1, 2 and 3 are started and running

4. Create a kakfa topic, list topics and describe one

5. Start a producer and publish some messages

6. Start a consumer and process messages

Log directories for each of the broker instances