Yogesh..BG27th Nov 2015

• Introduction• Features• Architecture• Storage• Topic• Producer and consumer• Broker• Zookeeper• Use Cases• Comparison with RMQ & AMQ

05/01/2023 Confidential 2

Agenda

Introduction

05/01/2023 Confidential 3

Open Source messaging Q written in scala.LinkedIn Corp -> Apache

Producer1

Producer2Queue

Consumer1

Consumer2

Consumer3

Features

05/01/2023 Confidential 4

• Brokered

• Distributed

• high throughput for publish and subscribe

• easy scalable

• fast

• replicated commit log service

• partitioned

• stores messages on disk

• In order delivery, per partition.

Architecture

05/01/2023 Confidential 5

Elements are:• Messages• Producer• Consumer• Topic• Broker [Cluster]• Zookeeper

Broker

05/01/2023 Confidential 6

• Stateless brokers.

• Transient System – deletes data

• Time-based SLA for the retention policy.

• Zero copy

Storage

05/01/2023 Confidential 7

Topic

05/01/2023 Confidential 8

• Unique identification for messages – offset• Consumer can change the offset to re consume or skip the message• Replicated among the configurable number of servers.• Retention policy• Parallelism at partitions level

Producers and Consumers

05/01/2023 Confidential 9

Producers:• Decides which message to which partition - LB.• Batch the messages• Async Send

Consumers:• Pull• Queuing and pub-sub• Consumer groups - cluster of consumers• Ordered per partition

Zookeeper

05/01/2023 Confidential 10

Use Cases

05/01/2023 Confidential 11

• Messaging

• Website activity tracking

• Metrics

• Log Aggregation

• Stream Processing

• Event Publishing

Comparison

05/01/2023 Confidential 12

Sooo fast… How come?

05/01/2023 Confidential 13

1. No wait.

2. efficient storage format. Header -> 9bytes (144 bytes in AMQ(As per JMS)).

3. Zero copy 4. Compressing multiple messages – message set

Thank YouYogesh..BG

05/01/2023 Confidential 14