Claudius MbembaCascadiaJS / Seattle, WA

Scalable Distributed Messaging

w/ Kafka on Containers

Who am I?CTO, Neu Inc.

@mbembashipsclaudiusmbemba.com

github.com/user1mgithub.com/neucleans

What we’ll cover...● The Problem● The Solution● The Demo● The End

The Problemwe had to solve

at Neu

We needed to write our

own solution

What is Kafka?

Image from Kafka.org

Image from Kafka.org

“Kafka® is used for building real-time data pipelines and streaming apps.

It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies”

Image from USPS.com

A Simple Analogy...● Kafka = Post Office● Producer = Sender● Consumer = Receiver● Topic = P.O. Box Wall

○ Partitions = P.O. Boxes○ Messages = Mail

Image from Google

Image from Google Images

Image from cloudkarafka.com

Consumer Group

Node+Kafka

Image from medium.com

Kafka+Docker

Image from medium.com

Bring in the Real Time!

Image fromdev.to

The Solution

Kafka-Pub-Sub

Github.com/NeuCleans/kafka-pub-sub

Code Walkthrough

Github.com/NeuCleans/kafka-pub-sub

Commented out code hidden

Github.com/NeuCleans/kafka-pub-sub

The End

What we covered...● The Problem● The Solution● The Demo● The End

Github.com/NeuCleans/kafka-pub-sub

Takeaway Resources

● http://podplayer.net/?id=82304318 - Podcast w/ Kafka co-creator, Neha Narkhede

● https://kafka-tutorials.confluent.io/ - Kafka tutorials

● https://www.youtube.com/watch?v=udnX21__SuU - Kafka Tutorial - Core Concepts

● https://www.youtube.com/watch?v=UEg40Te8pnE - Introduction to Apache Kafka

● https://kafka.apache.org/ - Kafka Homepage

Thank You!- claudiusmbemba.com/public/slides/cjs2019.pdf

- github.com/NeuCleans/kafka-pub-sub

- github.com/User1m/socketio-kafka-pubsub-sample

- github.com/User1m/kafka-pub-sub-investigation