Apache Kafka Lightning Talk

Click here to load reader

  • date post

    19-Jan-2015
  • Category

    Technology

  • view

    663
  • download

    3

Embed Size (px)

description

This short deck provides a high-level overview of how Apache Kafka works under the covers. It covers logs, topics partitions, consumer groups, and replication from a conceptual perspective.

Transcript of Apache Kafka Lightning Talk

  • 1. QQQQQQQTuesday, December 3, 13QQQQQQQQQQQQQQQ QJeff Kunkle Nov 22, 2013 QQ QQQQ QQQApache KafkaQQQQQQQQQQ QQQ Q

2. QQQQQQQTuesday, December 3, 13QQQQQQQQQQQQQQQ QQ QQ QQQQ QQQWhat is Kafka?QQQQQQQQQQ QQQ Q 3. Kafka is a distributed, partitioned, replicated commit log service. Tuesday, December 3, 13 4. Tuesday, December 3, 13 5. ? Tuesday, December 3, 13 6. ... a non-JMS messaging system with a unique design.Tuesday, December 3, 13 7. Key Concepts 1 2Consumer Groups3Tuesday, December 3, 13Topics, Logs, and Partitions Replication 8. Topics and LogsTuesday, December 3, 13 9. Topics and Logs partition 0partition 1partition 2Tuesday, December 3, 13 10. Topics and Logs logpartition 01234567logpartition 1123456logpartition 21old Tuesday, December 3, 132345new 11. Topics and Logs logpartition 01234567logpartition 112345writes6logpartition 21old Tuesday, December 3, 132345new 12. partition 0partition 1 P2 5 2 1 3 4partition 2Tuesday, December 3, 13ProducersP1 2 6 5 4 3 1 7 13. partition 012partition 11 P2 5 2 3 4partition 2Tuesday, December 3, 131ProducersP1 2 6 5 4 3 7 14. bulk publishing partition 012partition 11 P2 5 2 3 4partition 2Tuesday, December 3, 131ProducersP1 2 6 5 4 3 7 15. bulk publishing partition 012partition 11 P2 5partition 2Tuesday, December 3, 131234ProducersP1 2 6 5 4 3 7 16. parallel publishing partition 012partition 11 P2 5partition 2Tuesday, December 3, 131234ProducersP1 2 6 5 4 3 7 17. parallel publishing partition 01234partition 112 P2 5partition 2Tuesday, December 3, 131234ProducersP1 4 3 6 5 7 18. partition 01234567partition 1123456 P2partition 2Tuesday, December 3, 1312345ProducersP1 19. consumer group 1ConsumersC123456123456123457partition 0C2 C3C4 consumer group 2Tuesday, December 3, 131partition 1partition 2 20. consumer group 1Consumers1 C13456123456123457partition 0C2 C3C4 1 consumer group 2Tuesday, December 3, 132partition 1partition 2 21. consumer group 1Consumers1 C134562345623457partition 0C2 1 C3C4 1 consumer group 2Tuesday, December 3, 1321partition 1partition 2 22. consumer group 1Consumers1 C134562345623457partition 01 C2 1 C3C4 1 consumer group 2Tuesday, December 3, 132partition 1partition 2 23. consumer group 1Consumers1 3 C1456456457partition 01 2 C2 1 2 C3C4 3 2 1 consumer group 2Tuesday, December 3, 1333partition 1partition 2 24. consumer group 1Consumers1 3 C13456456457partition 01 2 C2 1 2 C3C4 3 2 1 consumer group 2Tuesday, December 3, 13bulk consumption3partition 1partition 2 25. consumer group 1bulk consumptionConsumers1 3 C1partition 01 2 C2 6 4 3 5 41 2 C3C4 6 5 4 3 2 1 consumer group 2Tuesday, December 3, 13735456partition 1partition 2 26. consumer group 1parallel consumptionConsumers1 3 C1partition 01 2 C2 6 4 3 5 41 2 C3C4 6 5 4 3 2 1 consumer group 2Tuesday, December 3, 13735456partition 1partition 2 27. consumer group 1parallel consumptionConsumers1 3 5 4 C1partition 01 2 C2 6 4 3 5 61 2 C3 3C4 6 5 4 3 2 1 consumer group 2Tuesday, December 3, 13745partition 1partition 2 28. consumer group 1Consumers1 3 5 4 C1 1 2 C2 6 4 3 5 1 2 C3 3 7 4 5C4 7 6 5 4 3 2 1 consumer group 2Tuesday, December 3, 13partition 0partition 1partition 2 29. Replication (all reads and writes)server AA masterP1 C5 A7 A6 A4 A3 A2 A1 B4 B3 B2 B1 (all reads and writes)(all reads and writes)Tuesday, December 3, 13server CC masterserver BB masterP2 C1 A5 C2 C3 C4 B6 B5 30. Replication (all reads and writes)A1server AA masterP1 C5 A7 A6 A4 A3 A2 B4 B3 B2 B1 (all reads and writes)(all reads and writes)Tuesday, December 3, 13server CC masterserver BB masterP2 C1 A5 C2 C3 C4 B6 B5 31. Replication (all reads and writes)A1server AA masterP1 C5 A7 A6 A4 A3 A2 B4 B3 B2 B1 (all reads and writes)A1(all reads and writes)Tuesday, December 3, 13server CC masterserver BB masterP2 C1 A5 C2 C3 C4 B6 B5 32. Replication (all reads and writes)A1 A2server AA masterP1 C5 A7 A6 A4 A3 B4 B3 B2 B1 (all reads and writes)A1 A2(all reads and writes)Tuesday, December 3, 13server CC masterserver BB masterP2 C1 A5 C2 C3 C4 B6 B5 33. Replication (all reads and writes)A1 A2 C1server AA masterP1 C5 A7 A6 A4 A3 B4 B3 B2 B1 (all reads and writes)A1 A2(all reads and writes)Tuesday, December 3, 13C1server CC masterserver BB masterP2 A5 C2 C3 C4 B6 B5 34. Replication (all reads and writes)server AA masterA1 A2 A3 A4 C1 C2 C3 C4 C5P1 A7 A6 (all reads and writes)B1 B2 B3 B4 B5 B6 A1 A2 A3 A4(all reads and writes)C1 C2 C3 C4 C5 B1 B2 B3 B4 B5Tuesday, December 3, 13B6server CC masterserver BB masterP2 A5 35. Replication (all reads and writes)A1 A2 A3 A4 A5 A6 A7 C1 C2 C3 C4 C5server AA masterP1 (all reads and writes)B1 B2 B3 B4 B5 B6 A1 A2 A3 A4(all reads and writes)C1 C2 C3 C4 C5 B1 B2 B3 B4 B5Tuesday, December 3, 13B6server CC masterserver BB masterP2 36. Replication (all reads and writes)A1 A2 A3 A4 A5 A6 A7 C1 C2 C3 C4 C5server AA masterP1 (all reads and writes)B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 A5 A6 A7(all reads and writes)C1 C2 C3 C4 C5 B1 B2 B3 B4 B5Tuesday, December 3, 13B6server CC masterserver BB masterP2 37. 123Messages in a partition are assigned a sequential id number, referred to as the offset.Consumers keep track of their consumed offset in the partition. Kafka doesnt maintain any metadata.Kafka maintains all messages for a congurable time, regardless of whether theyve been consumed.Tuesday, December 3, 13 38. Alternatives http://queues.ioTuesday, December 3, 13 39. Questions?Tuesday, December 3, 13