101 ways to configure kafka - badly

101* ways to configure Kafka - badly

Hva FINN.no har lært av å kjøre Kafka

Audun Fauchald StrandLead Developer Infrastructure

@audunstrandbio: gof, mq, ejb, wli, bpel eda, soa esb, ddd, k8s

Henning Spjelkavik Architect

@spjelkavikbio: Skiinfo (Vail Resorts),

FINN.noenjoys reading jstacks

agenda

Introduction to kafka

kafka @ finn.no

101* mistakes

questions

“From a certain point onward

there is no longer any turning

back. That is the point that must

be reached.”

― Franz Kafka, The Trial

https://www.goodreads.com/author/show/5223.Franz_Kafka

https://www.goodreads.com/work/quotes/2965832

introduction to kafka

why use kafka

#notAnESB

what is a log

terminology

components

giant leap“A First Sign of the Beginning of Understanding is the Wish to Die.”― Franz Kafka

https://commons.wikimedia.org/wiki/File:Kafka.jpg


Why use Kafka?

“Apache Kafka is publish-subscribe messaging

rethought as a distributed commit log.”

Fast

Scalable

Durable

Distributed by design

Sweet spot: High volume, low latency

Quora:“Use Kafka if you have a fire hose of events (100k+/sec)

you need delivered in partitioned order 'at least once' with a mix of online and batch consumers, you want to be able to re-read messages”

“Use Rabbit if you have messages (20k+/sec) that need to

be routed in complex ways to consumers, you want per-message delivery guarantees, you don't care about ordered delivery”

What is a (data) log / journal?

A log is perhaps the simplest possible storage abstraction.

It is an append-only, totally-ordered sequence of records ordered by time.

Appended to the end of the log, reads proceed left-to-right.

Each entry is assigned a unique sequential log entry number.

The ordering of records defines a notion of "time" since entries to the left are

defined to be older then entries to the right.

This is a data log, not an application log (i.e not log4j)

Changelog 101: Tables and Events are Dual

Duality: a log of changes and a table.

Accounting

log: credit and debit (events pr key)

table: all current balances (i.e state pr key)

In a sense the log is the more fundamental data structure: in addition to creating the

original table you can also transform it to create all kinds of derived tables.

producers writes to brokers

consumers reads from brokers

everything is distributed

data is stored in topics

topics are split into partitions

which are replicated

kafka cluster

consumer

producerproducerproducer producer

consumerconsumerconsumer

consumerconsumer

producerproducer

terminology

producer“ad.new”

consumergroup.1

“ad.new”

Broker1

Broker2

Broker3

P0R1

P0R2

P1R3

P2R1

P1R2

P2R3

zookeeper

components consumergroup.id

“ad.new”

consumergroup.1

“ad.new”

1:data 2:... 3:... 4:... 5:... 6:... 7:...

old messages newer messages

...

consumer.group.1

consumer.group.2

producer a1

Giant leap?

In fact, persistent replicated messaging is such a giant leap in messaging architecture that it may be worthwhile to point out a few side

effects:

a. Per-message acknowledgments have disappeared

b. ordered delivery

c. The problem of mismatched consumer speed has disappeared. A slow consumer can peacefully co-exist with a fast

consumer now

d. Need for difficult messaging semantics like delayed delivery, re-delivery etc. has disappeared. Now it is all up to the

consumer to read whatever message whenever - onus has shifted from broker to consumer

e. The holy grail of message delivery guarantee: at-least-once is the new reality - both Kafka and Azure Event Hub

provides this guarantee. You still have to make your consumers and downstream systems idempotent so that recovering

from a failure and processing the same message twice does not upset it too much, but hey - that has always been the

case

http://blogs.msdn.com/b/opensourcemsft/archive/2015/08/08/choose-between-azure-event-hub-and-kafka-_2d00_-what-

you-need-to-know.aspx

Confluent platform

Top 5

1. no consideration of data on the

inside vs outside

2. schema not externally defined

3. same config for every

client/topic

4. 128 partitions as default config

5. running on 8 overloaded nodes

reklame

FINN.nohttps://finn.no/apply-here

https://twitter.com/@FINN_techhttp://tech.finn.no

https://github.com/finn.no

60 millions pageviews a day

250-300 microservices

130 developers

1479 deployed changes into

production last week

6 minutes from commit to deploy

(median)

https://finn.no/apply-here

https://twitter.com/@FINN_tech

http://tech.finn.no


#javazone @spjelkavik @audunstrand

Schibsted Media Group6800 people in 30 countries

FINN.no is a part of

6900

employees30 countries

200 million

users

jobs.schibsted.com

https://jobs.schibsted.com/

kafka @ finn.no

kafka @finn.no architecture

use cases

tools


in the beginning ...

Architecture governance board decided to use RabbitMQ as message queue.

Kafka was installed for a proof of concept, after developers spotted it januar 2013.


2013 - POC

“High” volume

Stream of classified ads

Ad matching

Ad indexed

mod05

zk

kafka

mod07

zk

kafka

mod01

zk

kafka

mod03

zk

kafka

mod06

zk

kafka

mod08

zk

kafka

mod02

zk

kafka

mod04

zk

kafka

dc 1

dc 2

Version 0.8.1

4 partitions

common client

java library

thrift


2014 - Adoption and complaininglow volume/ high

reliability

Ad Insert

Product Orchestration

Payment

Build Pipeline

click streams

mod05

zk

kafka

mod07

zk

kafka

mod01

zk

kafka

mod03

zk

kafka

mod06

zk

kafka

mod08

zk

kafka

mod02

zk

kafka

mod04

zk

kafka

dc 1

dc 2

Version 0.8.1

4 partitions

experimenting

with

configuration

common java

library


toolingalerting


2015 - Migration and consolidation“reliable messaging”

asynchronous

communication

between services

store and forward

zipkin

slack notifications

dc 1

dc 2

Version 0.8.2

5-20 partitions

multiple

configurations

broker05

zk

kafka

broker01

zk

kafka

broker03

zk

kafka

broker04

zk

kafka

broker02

zk

kafka


toolingGrafana dashboard visualizing jmx stats

via Prometheus

Kafka-manager from Yahoo

Kafka-cat cli


2016 - Confluent

broker01

broker05

Schema

Registry

kafka

kafka

broker03

kafka

broker04

kafka

broker02

kafka

zk02 zk zk03 zk

Rest Proxy

zk01 zk

platform

schema registry

data replication

kafka connect

kafka streams

101* mistakes

“God gives the nuts, but he does not crack them.”― Franz Kafka


Pattern Language

why is it a mistake

what is the consequence

what is the correct solution

what has finn.no done

Top 5

1. no consideration of data on the

inside vs outside

2. schema not externally defined

3. same config for every

client/topic

4. 128 partitions as default config

5. running on 8 overloaded nodes


mistake: no consideration of data on the inside vs outside

https://flic.kr/p/6MjhUR


why is it a mistakeeverything published on Kafka (0.8.2) is visible to any client that can access


what is the consequencedirect reads across services/domains is quite normal in legacy and/or enterprise

systems

coupling makes it hard to make changes

unknown and unwanted coupling has a cost

Kafka had no security per topic - you must add that yourself


what is the correct solutionConsider what is data on the inside, versus data on the outside

Convention for what is private data and what is public data

If you want to change your internal representation often, map it before publishing it

publicly (Anti corruption layer)


what has finn.no doneDecided on a naming convention (i.e Public.xyzzy) for public topics

Communicates the intention (contract)


mistake: schema not externally defined


why is it a mistakedata and code needs separate versioning strategies

version should be part of the data

defining schema in a java library makes it more difficult to access data from non-

jvm languages

very little discoverability of data, people chose other means to get their data

difficult to create tools


what is the consequencedevelopment speed outside jvm has been slow

change of data needs coordinated deployment

no process for data versioning, like backwards compatibility checks

difficult to create tooling that needs to know data format, like data lake

and database sinks


what is the correct solutionconfluent.io platform has a separate schema registry

apache avro

multiple compatibility settings and evolutions strategies

connect

more automatic tooling


what has finn.no donestill using java library, with schemas in builders

confluent platform 2.0 is planned for the next step, not (just) kafka 0.9


mistake: running mixed load with a single, default configuration

https://flic.kr/p/qbarDR


why is it a mistakeHistorically - One Big Database with Expensive License

Database world - OLTP and OLAP

Changed with Open Source software and Cloud

Tried to simplify the developer's day with a single config

Kafka supports very high throughput and highly reliable


what is the consequenceTrade off between throughput and degree of reliability

With a single configuration - the last commit wins

Either high throughput, and risk of loss - or potentially too slow


what is the correct solutionUnderstand your use cases and their needs!

Use proper pr topic configuration

Consider splitting / isolation


Defaults that are quite reliable

Exposing configuration variables in the client

Ask the questions;

at least once delivery

ordering - if you partition, what must have strict ordering

99% delivery - is that good enough?

what level of throughput is needed



ConfigurationConfiguration for production

Partitions

Replicas (default.replication.factor)

Minimum ISR (min.insync.replicas)

Wait for acknowledge when producing messages (request.required.acks, block.on.buffer.full)

Retries

Leader election

Configuration for consumer

Number of threads


Gwen Shapira recommends...akcs = all

block.on.buffer.full = true

retries = MAX_INT

max.inflight.requests.per.connect = 1

Producer.close()

replication-factor >= 3

min.insync.replicas = 2

unclean.leader.election = false

auto.offset.commit = false


mistake: default configuration of 128 partitions for each topic

https://flic.kr/p/6KxPgZ

https://flic.kr/p/6KxPgZ


why is it a mistakepartitions are kafkas way of scaling consumers, 128 partitions can handle 128

consumer processes

in 0.8; clusters could not reduce the number of partitions without deleting data

highest number of consumers today is 20


what is the consequenceour 0.8 cluster was configured with 128 partitions as default, for all topics.

many partitions and many topics creates many datapoints that must be coordinated

zookeeper must coordinate all this

rebalance must balance all clients on all partitions

zookeeper and kafka went down (may 2015)

(500 topics * 128 partitions)


what is the correct solutionsmall number of partitions as default

increase number of partitions for selected topics

understand your use case

reduce length of transactions on consumer side

partitions per topic: max(t/p, t/c)

Max partitions on a broker 100 x brokers x replication factor => 1500 in our case

http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/

http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/


what has finn.no done5 partitions as default

4 topics have more than 5 partitions, topics with lots of traffic


mistake: deploy a proof of concept hack - in production ; i.e why we had 8 zk nodes

https://flic.kr/p/6eoSgT


why is it a mistakeKafka was set up by Ops for a proof of concept - not for hardened production use

By coincidence we had 8 nodes for kafka, the same 8 nodes for zookeeper

Zookeeper is dependent on a majority quorum, low latency between nodes

The 8 nodes were NOT dedicated - in fact - they were overloaded already


what is the consequenceZookeeper recommends 3 nodes for normal usage, 5 for high, and any more is

questionable

More nodes leads to longer time for finding consensus, more communication

If we get a split between data centers, there will be 4 in each

You should not run Zk between data centers, due to latency and outage

possibilities


what is the correct solutionHave an odd number of Zookeeper nodes - preferrably 3, at most 5

Don’t cross data centers

Check the documentation before deploying serious production load

Don’t run a sensitive service (Zookeeper) on a server with 50 jvm-based services,

300% over committed on RAM

Watch GC times



dc 1

dc 2

broker05

zk

kafka

broker01

zk

kafka

broker03

zk

kafka

broker04

zk

kafka

broker02

zk

kafka

Version 0.8.2

5-20 partitions

multiple

configurations


“They say ignorance is

bliss.... they're wrong ”

― Franz Kafka


“It's only because of their stupidity that they're able to be so sure of themselves.” ― Franz Kafka, The

Trial

Audun Fauchald Strand

@audunstrand

Henning Spjelkavik

@spjelkavik

Q?

https://www.finn.no/apply-here

https://tech.finn.no

https://twitter.com/@FINN_tech


http://www.schibsted.com/en/Career/


https://www.goodreads.com/work/quotes/2965832

http://www.finn.no/apply-here

http://tech.finn.no

http://twitter.com/@FINN_tech


http://www.schibsted.com/en/Career/


Runner upUsing pre-1.0 software

Have control of topic creation

Kafka is storage - treat it like one also ops-wise

Client side rebalancing

Commiting on all consumer threads, believing that you only commited on one


References / Further readingDesigning data intensive systems, Martin Kleppmann

Data on the inside - data on the outside, Pat Helland

The Confluent Blog, http://confluent.io/

Kafka - The definitive guide

This presentation, in English: http://www.confluent.io/blog/the-top-sessions-from-

kafka-summit-2016

www.finn.no/apply-here

jobs.schibsted.com/

tech.finn.no

twitter.com/@FINN_tech

github.com/finn.no

http://confluent.io/

http://www.confluent.io/blog/the-top-sessions-from-kafka-summit-2016

http://www.finn.no/apply-here

http://jobs.schibsted.com/

http://tech.finn.no

http://twitter.com/@FINN_tech


101 ways to configure kafka - badly

Technology

Transcript of 101 ways to configure kafka - badly