Reactive Kafka

Unless o therwise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion -NonCommerc ia l l i cense : h t tp : / /c reat ivecommons .org / l i censes /by -nc /3 .0 /

Reactive Kafka Rajini Sivaram


Safe Harbor Statement

•  The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.

2


About me

•  Principal Software Engineer at Pivotal UK •  Previously at IBM

•  Message Hub developer – Kafka-as-a-Service on Bluemix

•  Contributor to Apache Kafka

3


Agenda

Ø  Introduction to Apache Kafka Ø  Clients and tools for Kafka

Ø  Non-reactive Java clients

Ø  Introduction to Reactive Streams

Ø  Reactive Kafka

Ø  Reactive API for Kafka

Ø  Performance evaluation

Ø  Summary

4

Reactive Kafka


What is Kafka?

•  Message Bus •  Publish/Subscribe

•  Fast

•  Scalable

•  Durable

•  Highly available

•  Commit Log Service

•  Distributed

•  Replicated •  Partitioned

5


Kafka as a Message Bus

6

Ø  Decouples producers and consumers Ø  Messages are stored for a configurable retention interval

Ø  Consumers can replay message stream


Kafka as a Commit Log Service

7

Broker 1 Broker 2 Broker 3 Broker 4 Broker 5

Partition 0

Partition 1

topicA Follower

Leader

Ø  Distributed Ø  Replicated

Ø  Partition Ø  Immutable, Ordered Stream Ø  Enables scalability and parallelism


Message Format

8

Message (v1) => Crc MagicByte Attributes Key Value o  Crc => int32

o  MagicByte => int8

o  Attributes => int8

o  Timestamp => int64

o  Key => bytes

o  Value => bytes

Header

Typically used to choose partition

Opaque array of bytes

CRC Magic Attr Timestamp Key Length

Key Bytes

Value Length Value Bytes


Topics and Logs

9

On-disk: Ø  /kafka-logs

Ø  topicA-0 Ø  00000000000000000000.index

Ø  00000000000000000000.log Ø  topicA-1

Ø  00000000000000000000.index

Ø  00000000000000000000.log Ø  00000000000008012974.index Ø  00000000000008012974.log

Ø  recovery-point-offset-checkpoint

Ø  replication-offset-checkpoint Ø  cleaner-offset-checkpoint

Ø  Topics Ø  Partitions Ø  Offsets

Ø  Logs Ø  Log segments Ø  Log compaction


Kafka Broker Kafka Broker

Kafka Cluster

10

Kafka Broker

Kafka Cluster

Kafka Broker Kafka Broker Zookeeper Server

Zookeeper Cluster

Monitoring


Message transmission

11

Broker 1 Broker 2 Broker 3 Broker 4 Broker 5

Partition 0

Partition 1

topicA Follower

Leader

Producer Consumer

Ø  Replication Ø  Async persistence to disk Ø  In-sync replicas

Ø  Zero-copy writes

þ


Broker 5 Broker 4

High Availability

12

Broker 1 Broker 2 Broker 3

Partition 0

Partition 1

topicA Follower

Leader

Controller

Zookeeper

Ø  Rolling upgrades


Messaging guarantees

•  Message ordering •  Ordered partitions

•  Messages sent by a producer to a partition are appended to log in that order

•  Consumer instances see messages in the order stored in the log •  Delivery semantics

•  At least once

•  At most once

•  Exactly once – not currently supported within Kafka

13


Broker configuration

14

Configuration Description

advertised.listeners Listeners published to ZK for clients to use

auto.leader.rebalance.enable Background thread periodically checks and triggers rebalance if required

replica.lag.time.max.ms If follower hasn’t fetched or consumed up to log end for this period, follower is removed from ISR

unclean.leader.election.enable Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss

broker.rack Used for rack-aware replica assignment for improved fault tolerance

etc. etc.


Topic configuration

15


cleanup.policy Compact or delete

retention.bytes Maximum log size

retention.ms Maximum time logs are retained (7 days by default)

max.message.bytes Largest message size allowed for topic

segment.bytes Maximum log segment size

min.insync.replicas For producer acks=all, minimum number of replicas for a successful write

etc. etc.


Security

•  TLS •  Encryption

•  Client authentication

•  SASL Authentication

•  GSSAPI/Kerberos

•  PLAIN

•  Access Control

•  Operations: CRUD, ClusterAction, All

•  ACL Resources: Cluster, Topic, Consumer groups •  Quotas

16

Kafka

Zoo

keep

er Kafka

Kafka

Client

Proxy

Proxy

Proxy

TLS/Plaintext

TLS


Kafka Use cases

•  Messaging •  Micro-service architecture •  Commit Log

•  Stream processing

•  Event sourcing

•  See http://kafka.apache.org/documentation.html#uses for more examples

17


Agenda







Ø  Summary

18


Kafka clients

19

Kafka Broker Kafka Broker Kafka Broker

Kafka Connect

REST proxy

Ø  Large ecosystem of clients, tools, connectors

Kafka Streams

Schema registry

C/C++ Consumer

Go Consumer

Java Consumer Java Producer

C/C++ Producer

Go Producer


Kafka producer: non-reactive Java example

20

producer = new KafkaProducer<>(producerProps); for (int i = 0; i < count; i++) { String message = "Message_" + i; producer.send(new ProducerRecord<>(topic, i, message), new Callback() { public void onCompletion(RecordMetadata metadata, Exception exception) { if (exception == null) System.out.println("Message sent successfully, metadata=" + metadata); else log.error("Send failed", exception); } }); }


Kafka producer: non-reactive Java 8

21

producer = new KafkaProducer<>(producerProps); for (int i = 0; i < count; i++) { String message = "Message_" + i; producer.send(new ProducerRecord<>(topic, i, message), (metadata, exception) -> { if (exception == null) System.out.println("Message sent successfully, metadata=" + metadata); else log.error("Send failed", exception); });}


Producer sequence diagram

22

Producer Broker 1 Broker 2

Find leader

Send message to leader

Determine partition

Add to local queue

Producer Network Thread

send() Wait for metadata

Broker 3

Fetch

Send response Callback

Message sent successfully, metadata=demo-topic-2@62

Bootstrap Leader ISR


Producer configuration

23


acks 0, 1, all

buffer.memory Total memory for local buffer

max.block.ms Max block time to wait for metadata or buffer space

retries Automatic retries

max.in.flight.requests.per.connection Max number of unacked requests on a connection

batch.size Max size for batch of records sent to broker

linger.ms Time to wait for batch

etc. etc.


Kafka Consumer

Ø  Messaging models Ø  Publish-subscribe

Ø  Point-to-point

24


Broker

Consuming from Kafka

•  Push vs Pull •  Push

•  Pull

•  Kafka

•  Producers push messages, consumers pull messages

•  Long poll

•  Offset management

•  Auto-commit

•  Manual commit •  External offset management

25

__consumer_offsets

commit()

Compacted topic


Kafka consumer: non-reactive Java example

26

consumer.subscribe(topics, new ConsumerRebalanceListener() { public void onPartitionsAssigned(Collection<TopicPartition> partitions) {} public void onPartitionsRevoked(Collection<TopicPartition> partitions) {} }); while (received < count) { ConsumerRecords<Integer, String> records = consumer.poll(1000); // 1 second for (ConsumerRecord<Integer, String> record : records) { System.out.println("Received message: " + record); received++; } }


Consumer sequence diagram

27

Broker 1 Broker 2 Consumer Broker 3

subscribe()

poll()

Create Subscription

Find group coordinator

Consumer

Client-side coordinator Bootstrap Leader Group coordinator

Rebalance

Join group

Sync Group

onPartitionsRevoked()

Fetch topic metadata

onPartitionsAssigned() Fetch offsets


Consumer sequence diagram

28

Broker 1 Broker 2 Consumer Broker 3 Consumer

Heartbeat poll()

Prefetch

Client-side coordinator Bootstrap Leader Group coordinator

Received message: ConsumerRecord(topic = demo-topic, partition = 2, offset = 62…, , key = 5, value = Message_5

Fetch messages


Consumer configuration

29


group.id Consumer group

fetch.min.bytes Min bytes to wait for per fetch

max.partition.fetch.bytes Max bytes per partition per fetch

fetch.max.wait.ms Max time server blocks before sending response

max.poll.records Max messages per poll

heartbeat.interval.ms Keeps session active and facilitates group rebalancing

session.timeout.ms Timeout to detect consumer instance failure

etc. etc.


Kafka Connect

30


Kafk

a C

onne

ct K

afka C

onnect

External Data Sources

External Data Sinks


Kafka Streams

31

Kafka Broker

Processor Topology

Kafka Streams

Ø  Simple threading model Ø  Non-reactive Ø  Back-pressure not required


Agenda







Ø  Summary

32


Reactive Streams

•  Fundamental shift from imperative-style to asynchronous, non-blocking functional-style code

•  Serve more heterogeneous requests concurrently

•  Handle remote operations more efficiently

•  Composable •  Back-pressure

•  Consumer requests data from emitter

•  Start receiving data only when consumer is ready to process data

•  Control the amount of inflight data

33


Project Reactor

34

Flux

Mono

Unless o therwise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion -NonCommerc ia l l i cense : h t tp : / /c reat ivecommons .org / l i censes /by -nc /3 .0 / 35

GATEWAY

Service B

Message Broker

Datastore

Service A

End-to-End Flow Control


Agenda







Ø  Summary

36

Reactive Kafka


Reactive Kafka

•  Bringing reactive streams and Kafka together •  Functional-style reactive API for Kafka

•  Back-pressure

•  Composable API

•  Implementation

•  As a shim over non-reactive Java API for Kafka

37

Reactive Kafka


Reactive Kafka API

38

public static <K, V> KafkaSender<K, V> create(SenderConfig<K, V> config) {} public Mono<RecordMetadata> send(ProducerRecord<K, V> record) {}

public static <K, V> KafkaFlux<K, V> listenOn(FluxConfig<K, V> config, Collection<String> topics) {} public enum AckMode { AUTO_ACK, ATMOST_ONCE, MANUAL_ACK, MANUAL_COMMIT }

KafkaSender:

KafkaFlux:


Reactive Kafka Producer

39

KafkaSender<Integer, String> sender = KafkaSender.create(senderConfig); Flux<RecordMetadata> flux = Flux.range(1, count) .flatMap(i -> sender.send(new ProducerRecord<>(topic, i, "Message_" + i))); .doOnNext(metadata -> { log.info("Message sent successfully, metadata=" + metadata); latch.countDown(); }) .doOnError(e-> log.error("Send failed", e)) … flux.subscribe();


Reactive KafkaSender sequence

40

1

2

2 3

3

flatMap({ }, maxConcurrency=256, prefetch=32)

2 3 1

requ

est(2

56)

requ

est(3

2)

Kafka Broker

Kafka Broker

Kafka Broker

1


Producer sample log (edited for readability)

[main] onSubscribe(r.c.publisher.FluxRange$RangeSubscription)[main] onSubscribe(r.c.publisher.FluxFlatMap$FlatMapMain)[main] request(unbounded)[main] request(256)[main] onNext(1)[main] ProducerConfig values: [main] onNext(2) ....[single-1] onNext(demo-topic-2@64)[single-1] Message sent successfully, metadata=demo-topic-2@64

41

Kafka producer created


Reactive Kafka Consumer

42

KafkaFlux<Integer, String> kafkaFlux = KafkaFlux.listenOn(fluxConfig, Collections.singleton(topic)) .doOnPartitionsAssigned(partitions -> log.debug(assigned {}", partitions)) .doOnPartitionsRevoked(partitions -> log.debug(”revoked {}", partitions)); kafkaFlux.subscribe(message -> { ConsumerOffset offset = message.consumerOffset(); ConsumerRecord<Integer, String> record = message.consumerRecord(); log.info("Received message: {} commitOffset: {}”, record.value(), offset); latch.countDown(); });


Reactive KafkaFlux sequence

43

concatMap({ })

Kafka Broker

Kafka Broker

Kafka Broker

H

merge

H

P

P

P P

P H

P

C

H

P C

Poll

Commit

Heartbeat


Consumer sample log (edited for readability) [main] onSubscribe(r.c.publisher.EmitterProcessor$EmitterSubscriber)[main] onSubscribe(r.c.publisher.FluxFlatMap$FlatMapMain)[main] request(256)[main] onNext(reactor.kafka.internals.FluxManager$InitEvent)[main] request(unbounded)[main] onNext(r.k.internals.FluxManager$PollEvent)[sample-group-1] ConsumerConfig values:[sample-group-1] Discovered coordinator … group sample-group.[sample-group-1] (Re-)joining group sample-group[sample-group-1] onPartitionsAssigned [demo-topic-2, demo-topic-1][parallel-1] onNext(org.apache.kafka.clients.consumer.ConsumerRecords)[parallel-1] ConsumerRecord(topic = demo-topic, partition = 2, offset = 64, …, key = 5, value = Message_5)[parallel-1] Received message: Message_5 commitOffset: demo-topic-2@65

44

Kafka consumer created


Reactive consumer configuration

45


pollTimeout Timeout specified in each consumer.poll()

commitInterval Duration between auto commits

commitBatchSize Maximum batch size before auto commit is triggered

maxAutoCommitAttempts Number of consecutive attempts to auto commit before raising an error

closeTimeout Timeout for graceful close (for both producer and consumer)

Ø  All non-reactive producer and properties configurable for reactive clients


Agenda







Ø  Summary

46

Reactive Kafka


Performance: Test configuration (Rackspace)

47

Test Driver

Kafka Cluster

ZK Cluster

Kafka Broker

Zookeeper

Kafka Broker

Zookeeper

Kafka Broker

Zookeeper


Rackspace machine configuration

•  OnMetal I/O v2 instances for Kafka •  CPU: Dual 2.6 GHz, 10 core Intel® Xeon® E5-2660 v3

•  RAM: 128 GB

•  Storage: Dual 1.6 TB PCIe flash cards

•  Network: Redundant 10 Gb / s connections in a high availability bond

•  Bare Metal Compute instances for Test driver and HTTP proxy

•  CPU: Dual 2.4 Ghz, 6 core Intel® Xeon® E5-2620 v3

•  RAM: 64 GB

•  Network: Redundant 10 Gb / s connections in a high availability bond

48


Producer performance

49

0

0.5

1

1.5

2

Mill

ion

mes

sage

s/se

cond

Message Size

Throughput(million msgs/sec)

Non-reactive Reactive

1

10

100

1000

MB

/sec

ond

Message Size

Throughput (MB/second)

Non-reactive Reactive


Consumer Performance

50

012345

Millionmessages/second

MessageSize

Throughput(millionmsgs/sec)

Non-reac0ve Reac0ve

1

10

100

1000

MB/second

MessageSize

Throughput(MB/second)

Non-reac0ve Reac0ve


End-to-End Latency

51

0

1

2

3

Latency(m

s)

MessageSize

Latency50%(millis)

Non-reac0ve Reac0ve

0

1

2

3

Latency(m

s)

MessageSize

Latency75%(millis)

Non-reac0ve Reac0ve


Performance: Kafka-REST vs Reactive HTTP

52

HTTP

Test Driver

Kafka-REST Proxy orSpring-Reactive-Web Proxy

Kafka Cluster

ZK Cluster

Kafka Broker

Zookeeper

Kafka Broker

Zookeeper

Kafka Broker

Zookeeper


Spring Reactive Web Framework HTTP->Kafka

53

@Profile("kafkahttp") @RestController public class KafkaHttpController { private final ReactiveKafkaHttpSender sender; public KafkaHttpController(ReactiveKafkaHttpSender sender) { this.sender = sender; } @RequestMapping(path= "/kafkahttp/{topic}", method = RequestMethod.POST) public Flux<SendResponse> sendToKafka(@PathVariable String topic, @RequestBody Flux<Records> binaryStream) { return this.sender.sendToKafka(topic, binaryStream) .map(metadata -> new SendResponse(metadata)); }


HTTP->Kafka

54

@Profile("kafkahttp”) public class ReactiveKafkaHttpSender { private final KafkaSender<String, byte[]> kafkaSender; public ReactiveKafkaHttpSender(KafkaSender<String, byte[]> kafkaSender) { this.kafkaSender = kafkaSender; } public Flux<RecordMetadata> sendToKafka(String topic, Publisher<Records> recordStream) { return Flux.from(recordStream) .concatMap(records -> Flux.fromArray(records.getRecords())) .concatMap(r -> kafkaSender.send(new ProducerRecord<>(topic, r.getValue()))); }


HTTP->Kafka: Kafka-REST vs Reactive

55

01000020000300004000050000

1 10

50

100

1000

2000

4000

6000

8000

Messages/second

Concurrency

Throughput(messages/second)

Non-reac0ve Reac0ve

020406080

1 10

50

100

1000

2000

4000

6000

8000

CPU(%

)

Concurrency

CPUusage

Non-reac0ve Reac0veØ  Higher throughput, better CPU utilization with reactive


HTTP->Kafka: Kafka-REST vs Reactive

56

0200400600800

1 10

50

100

1000

2000

4000

6000

8000Latency(m

s)

Concurrency

Latency50%(millis)

Non-reac0ve Reac0ve

Ø  Lower latency at high concurrency with reactive

0200400600800

1 10

50

100

1000

2000

4000

6000

8000Latency(m

s)

Concurrency

Latency75%(millis)

Non-reac0ve Reac0ve


ETL: MongoDB => Kafka => ElasticSearch

57


Ø  ETL pipeline: Extract, Transform, Load Ø  Can be done with Kafka Connect

Ø  Reactive API useful when pipeline includes multiple remote components


MongoDB => Kafka => ElasticSearch

58

kafkaConnector.createFlux() .doOnSubscribe(s -> mongoSource.findAll() // Extract from MongoDB source .flatMap(person -> kafkaConnector.store(person)) .subscribe() ) .map(k-> new PersonRecord(k).transform()) // Transform .window(sinkBufferSize, sinkBufferTimeout) .flatMap(list -> elasticSearchSink.put(list)); // Load into ElasticSearch


Composition

59

mongoSource .findAll() // Extract .flatMap(person -> kafkaConnector.store(person)) // Store in Kafka .map(person -> personalDataSource.fetch(person)) // Remote operation .window(sinkBufferSize, sinkBufferTimeout) .flatMap(list -> elasticSearchSink.put(list)); // Load


Summary •  Apache Kafka

•  Distributed, highly available, scalable, high throughput, low-latency •  Vast ecosystem of clients and tools

o  Works well without the need for back-pressure for Kafka interactions •  Reactive Streams

•  Functional-style, Back-pressure •  Reactive Kafka

•  Full functionality of non-reactive clients •  May see performance drop in some scenarios •  Reactive pipeline with Kafka as well as non-Kafka components

o  Benefits from non-blocking back-pressure o  Better concurrency, more efficient use of resources, better CPU utilization

60

Reactive Kafka


Kafka clients - revisited

61


Kafka Connect

REST proxy

Ø  Large ecosystem of clients, tools, connectors

Kafka Streams

Schema registry

C/C++ Consumer

Go Consumer

Java Consumer Java Producer

C/C++ Producer

Go Producer

ReactiveProducer ReactiveConsumer


Want to find out more?

•  Apache Kafka •  http://kafka.apache.org/documentation.html

•  https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

•  https://cwiki.apache.org/confluence/display/KAFKA/Clients •  Project Reactor

•  https://projectreactor.io/docs/

•  Reactive Kafka

•  https://github.com/reactor/reactor-kafka

62


Learn More. Stay Connected. [email protected]

Related Sessions:

Reactor 3.0, a JVM Foundation for Java 8 and Reactive Streams Designing, Implementing, and Using Reactive APIs

From Imperative To Reactive Web Apps Spring for Apache Kafka

@springcentral spring.io/blog

@pivotal pivotal.io/blog

@pivotalcf http://engineering.pivotal.io


Thank you for listening.

@springcentral spring.io/blog

@pivotal pivotal.io/blog

@pivotalcf http://engineering.pivotal.io

Questions?

Reactive Kafka

Technology

Transcript of Reactive Kafka