Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming...

53
1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne @s1p

Transcript of Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming...

Page 1: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

1

Apache Kafka Event-Streaming Platform for .NET DevelopersOctober, 2019

@gamussa | #SpringOne | @ConfluentINc

#SpringOne @s1p

Page 2: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

2

Page 3: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #springone | @ConfluentINc

3

I build highly scalable

Hello World

apps

Page 4: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

4

A company is build on DATA FLOWS

but All we have is DATA STORES

Page 5: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

5

Pre-Streaming

Page 6: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

6

Page 7: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne
Page 8: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

8

New World Streaming first• DB/DWH + Many more

distributed data systems

• Monolith -> Microservices

• Batch -> Real-time

Page 9: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

9

Origins in Stream Processing

Serving Layer

(Microservices, Elastic, etc.)

Java Apps with Kafka Streams or KSQL

Continuous Computation

High Throughput Streaming platform

API based clustering

Page 10: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

10

Streaming Platform

Storage

Pub / Sub

Processing

Page 11: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

11

Storage

Page 12: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

12

● DB - table ● Hadoop - file ● Kafka - ?

Core

Abstraction

Page 13: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne
Page 14: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

14

LOG

Page 15: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

15

The log is a simple idea

Messages are added at the end of the log

Old New

Page 16: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

16

Messages are added at the end of the log

Old New

The log is a simple idea

Page 17: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

17

Pub / Sub

Page 18: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

18

Time

Page 19: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

19

C2 C3C1

Time

Page 20: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

20

TimeA

B

C

D

hash(key) % numPartitions = N

Page 21: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

21

Messages will be produced in a round robin fashion

Time

Page 22: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

22

Consumers have a position all of their own

Old New

Robin is here

Scan Viktor is here

Scan

Ricardo is here

Scan

Page 23: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

23

Old New

Robin is here

Scan Viktor is here

Scan

Ricardo is here

Scan

Consumers have a position all of their own

Page 24: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

24

Old New

Robin is here

Scan Viktor is here

Scan

Ricardo is here

Scan

Consumers have a position all of their own

Page 25: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

25

Only Sequential Access

Old NewRead to offset & scan

Page 26: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

CONSUMER GROUP COORDINATORCONSUMERS

CONSUMER GROUP

Page 27: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

27

C

Page 28: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

28

CCC1

CCC2

Page 29: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

29

C C

C C

Page 30: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

30

0 1

2 3

Page 31: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

31

0 1

2 3

Page 32: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

32

0, 3 1

2 3

Page 33: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

33

Linearly Scalable Architecture

Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!!

Producers

Consumers

Page 34: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

34

Replicate to get fault

replicate

msg

msg

leader

Machine A

Machine B

Page 35: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

35

Partition Leadership and Replication

Broker 1

Topic1 partition1

Broker 2 Broker 3 Broker 4

Topic1 partition1

Topic1 partition1

Leader Follower

Topic1 partition2

Topic1 partition2

Topic1 partition2

Topic1 partition3

Topic1 partition4

Topic1 partition3

Topic1 partition3

Topic1 partition4

Topic1 partition4

Page 36: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

36

Replication provides resiliency

A replica takes over on machine failure

Page 37: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

37

Partition Leadership and Replication - node failure

Broker 1

Topic1 partition1

Broker 2 Broker 3 Broker 4

Topic1 partition1

Topic1 partition1

Leader Follower

Topic1 partition2

Topic1 partition2

Topic1 partition2

Topic1 partition3

Topic1 partition4

Topic1 partition3

Topic1 partition3

Topic1 partition4

Topic1 partition4

Page 38: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

38

Similar to a traditional messaging system (ActiveMQ, Rabbit etc) but with: (a) Far better scalability (b) Built in fault tolerance / HA (c) Storage

The log is a type of durable messaging system

Page 39: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

Stop! Demo time!

Page 40: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

40

Processing

Page 41: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

41

Streaming

is the toolset for dealing with events

as they move!

Page 42: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

42

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 43: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

43

CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 44: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

44

CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 45: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

45

CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 46: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

46

CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 47: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

47

CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 48: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

48

CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;

authorization_attempts possible_fraud

What exactly is Stream Processing?

Page 49: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

49

Lower the bar to enter the world of streaming

User Population

Codi

ng S

ophi

stic

atio

n Core developers who use Java/Scala

Core developers who don’t use Java/Scala

Data engineers, architects, DevOps/SRE

BI analysts

streams

Page 50: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

50

KSQL #FTW

4 Headless1 UI 2 CLI

ksql>

3 REST

POST /query

Page 51: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

51

Interaction with Kafka

Kafka(data)

KSQL (processing)

Application (processing) Jva/KStreams, .NET

Does not run on Kafka brokers

Does not run on Kafka brokers

Page 52: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@gamussa | #SpringOne | @ConfluentINc

52

Find your local Meetup Group https://cnfl.io/kafka-meetups

Join us in Slackhttp://cnfl.io/slack

Grab Stream Processing books https://cnfl.io/book-bundle

Page 53: Apache Kafka Event-Streaming Platform for .NET Developers · 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc #SpringOne

@@gamussa | #SpringOne | @ConfluentINc

Thanks!@gamussa [email protected]