Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

99
1 Building Event Driven Services with Apache Kafka and Kafka Streams Ben Stopford @benstopford

Transcript of Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

Page 1: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

1

Building Event Driven Services with Apache Kafka and Kafka StreamsBen Stopford@benstopford

Page 2: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

2

Build Features

Build for the Future

Page 3: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

3

Evolution!

Page 4: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

4

KAFKA

ServingLayer

(Cassandra etc.)

Kafka Streams / KSQL

Streaming Platforms

Data is embedded in each engine

High Throughput Messaging

Clustered Java App

Page 5: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

5

authorization_attempts possible_fraud

Streaming Example

Page 6: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

6

CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

authorization_attempts possible_fraud

Page 7: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

7

CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

authorization_attempts possible_fraud

Page 8: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

8

CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

authorization_attempts possible_fraud

Page 9: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

9

CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

authorization_attempts possible_fraud

Page 10: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

10

CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

authorization_attempts possible_fraud

Page 11: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

11

CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

authorization_attempts possible_fraud

Page 12: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

12

Streaming == Manipulating Data in Flight

Page 13: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

13

Business Applications

Page 14: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

14

EcosystemsApp

Increasingly we build ecosystems

Page 15: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

15

SOA / Microservices / EDA

CustomerService

ShippingService

Page 16: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

16

The Problem is DATA

Page 17: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

17

Most services share the same core facts.

Catalog

Most services live in here

Page 18: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

18

So how do we share data between services?

OrdersService

ShippingService

CustomerService

Webserver

Page 19: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

19

Buying an iPad (with REST)

SubmitOrder

shipOrder() getCustomer()

OrdersService

ShippingService

CustomerService

Webserver

Page 20: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

20

Buying an iPad with Events

Message Broker (Kafka)

Notification Data is replicated

(incrementally)

SubmitOrder

Order Created

Customer Updated

OrdersService

ShippingService

CustomerService

Webserver

KAFKA

Page 21: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

21

Events for Notification Only

Message Broker (Kafka)

SubmitOrder

Order Created

getCustomer()REST

Notification

OrdersService

ShippingService

CustomerService

Webserver

KAFKA

Page 22: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

22

Events for Data Locality

Customer Updated

SubmitOrder

Order Created

Data is replicated

OrdersService

ShippingService

CustomerService

Webserver

KAFKA

Page 23: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

23

Events have two hats

Notification Data replication

Page 24: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

24

Events are the key to scalable service ecosystems

Page 25: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

25

Streaming is the toolset for dealing with events as they move!

Page 26: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

26

Kafka: a Streaming Platform

The Log ConnectorsConnectors

Producer Consumer

Streaming Engine

Page 27: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

27

Kafka: a Streaming Platform

The Log ConnectorsConnectors

Producer Consumer

Streaming Engine

Page 28: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

28

What is a Distributed Log?

Page 29: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

29

Shard on the way in

ProducingServices

Kafka

ConsumingServices

Page 30: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

30

Each shard is a queue

ProducingServices

Kafka

ConsumingServices

Page 31: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

31

Consumers share load

ProducingServices

Kafka

ConsumingServices

Page 32: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

32

Reduces to a globally ordered queue

Page 33: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

33

Load Balanced Services

Page 34: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

34

Fault Tolerant Services

Page 35: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

35

Build ‘Always On’ Services

Rely on Fault Tolerant Broker

Page 36: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

36

Services can “Rewind & Replay” the log

Rewind & Replay

Page 37: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

37

Compacted Log(retains only latest version)

Version 3

Version 2

Version 1

Version 2

Version 1

Version 5

Version 4

Version 3

Version 2

Version 1

Page 38: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

38

Kafka: a Streaming Platform

The Log ConnectorsConnectors

Producer Consumer

Streaming Engine

Page 39: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

39

Kafka Connect

KafkaConnect

KafkaConnect

Kafka

Page 40: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

40

Kafka: a Streaming Platform

The Log ConnectorsConnectors

Producer Consumer

Streaming Engine

Page 41: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

41

A database engine for data-in-flight

Page 42: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

42

SELECT card_number, count(*)FROM authorization_attemptsWINDOW (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;

Continuously Running Queries

Page 43: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

43

Features: similar to database query engine

JoinFilterAggr-egate

View

Window

Page 44: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

44

KAFKA

Buffer 5 mins

Windows

Page 45: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

45

Lookup tables

KAFKA

Page 46: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

46

Stateful Stream Processing

stream

Compactedstream

Join

Stream

Table

KafkaKafka Streams

Page 47: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

47

Scales Out

Page 48: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

48

Embeddable

Orders Service

Kafka

Orders Service

Orders Service

Page 49: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

49

Streaming is about

1. Processing data incrementally

2. Moving data to where it needs to be processed (quickly and efficiently)

On Notification

Data Replication

Page 50: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

50

Kafka: a Streaming Platform

The Log ConnectorsConnectors

Producer Consumer

Streaming Engine

Page 51: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

51

Steps to Streaming Services

Page 52: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

52

1. Take Responsibility for the past and evolve

Page 53: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

53

Stay Simple. Take Responsibility for the past

Browser

Webserver

Page 54: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

54

Evolve Forwards

Browser

WebserverOrdersService

Page 55: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

55

2. Raise events. Don’t talk to services.

Page 56: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

56

Events have two hats

Notification Data replication

Page 57: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

57

Raise events. Don’t talk to services

Browser

WebserverOrdersService

Page 58: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

58

KAFKA

Order RequestedOrder

Received

Browser

Webserver

OrdersService

Raise events. Don’t talk to services

Page 59: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

59

KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

Raise events. Don’t talk to services

Page 60: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

60

KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

Use Kafka as a Backbone for Events

Page 61: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

61

3. Use Connect (& CDC) to evolve away from legacy

Page 62: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

62KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

Evolve away from Legacy

Page 63: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

63KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

Use the Database as a ‘Seam’

Connect

Products

Page 64: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

64

4. Make use of Schemas

Page 65: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

65KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

Use Schemas to validate Backwards Compatibility

Connect

ProductsSchema Registry

Page 66: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

66

5. Use the Single Writer Principal

Page 67: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

67KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

Apply the single writer principal

Connect

ProductsSchema Registry

Order Completed

Page 68: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

68

Orders Service

EmailService

T1 T2

T3

T4

RESTService

T5

Single Writer Principal

Page 69: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

69

Single Writer Principal

- Creates local consistency points in the absence of Global Consistency

- Makes schema upgrades easier to manage.

Page 70: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

70

6. Store Datasets in the Log

Page 71: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

71

Messaging that Remembers

Orders Customers

PaymentsStock

Page 72: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

72KAFKA

Order Requested

Order Validated

Order Received

Browser

Webserver

OrdersService

New Service, No Problem!

Connect

Products

Schema Registry

Order Completed Repricing

Page 73: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

73

Orders Customers

PaymentsStock

Single, Shared Source of Truth

Page 74: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

74

But how do you query a log?

Page 75: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

75

7. Move Data to Code

Page 76: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

76

Page 77: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

77

Connect

Order Requested

Order Validated

Order Completed

Order Received

Products

Browser

Webserver

Schema Registry

OrdersService Stock

Stock

Materialize Stock ‘View’ Inside Service

KAFKA

Page 78: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

78

Connect

Order Requested

Order Validated

Order Completed

Order Received

Products

Browser

Webserver

Schema Registry

OrdersService Stock

Stock

Take only the data we need

KAFKA

Page 79: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

79

Data Movement

Be realistic:• Network is no longer the bottleneck• Indexing is:

• In memory indexes help• Keep datasets focused

Page 80: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

80

8. Use the log instead as a ‘database’

Page 81: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

81

Connect

Order Requested

Order Validated

Order Completed

Order Received

Products

Browser

Webserver

Schema Registry

OrdersService

Reserved Stocks

Stock

Stock

Reserved Stocks

Apply Event Sourcing

KAFKA

Table

Page 82: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

82

Connect

Order Requested

Order Validated

Order Completed

Order Received

Products

Browser

Webserver

Schema Registry

OrdersService

Reserved Stocks

Stock

Stock

Reserved Stocks

Order Service Loads Reserved Stocks on Startup

KAFKA

Page 83: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

83

Kafka has several features for reducing the need to move data on startup

- Standby Replicas- Disk Checkpoints- Compacted topics

Page 84: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

84

9. Use Transactions to tie All Interactions Together

Page 85: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

85

OrderRequested(IPad)

2a. Order Validated

2c. Offset Commit2b. IPad Reserved

Internal State:Stock = 17Reservations = 2

Tie Events & State with Transactions

Page 86: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

86

Connect

TRANSACTION

Order Requested

Order Validated

Order Completed

Order Received

Products

Browser

Webserver

Schema Registry

OrdersService

Reserved Stocks

Stock

Stock

Reserved Stocks

Transactions

KAFKA

Page 87: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

87

10. Bridge the Sync/Async Divide with a Streaming Ecosystem

Page 88: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

88

POST

GET

Load

Ba

lance

r

ORDE

RSOR

DERS

OV T

OPIC

Order ValidationsKAFKA

INVENTORY

Orders

Inventory

Fraud Service

Order DetailsService

InventoryService

(see previous figure)

Order Created

Order Validated

Orders View

Q in CQRS

Orders ServiceC is CQRS

Services in the Micro: Orders ServiceFind the code online!

Page 89: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

89

Orders Customers

Payments Stock

Query Engine (Kstreams/KSQL)

Larger Ecosystems

HISTORICAL EVENT STREAMS

Page 90: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

90

Kafka

KAFKA

New York

Tokyo

London

Global / Disconnected Ecosystems

Page 91: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

91

So…

Page 92: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

92

Good architectures have little to do with this:

Page 93: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

93

It’s about how systems evolves over time

Page 94: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

94

Request driven isn’t enough

• High coupling• Hard to handle

async flows• Hard to move and

join datasets.

Page 95: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

95

Leverage the Duality of Events

Notification Data replication

Page 96: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

96

With a toolset built for data in flight

Page 97: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

97

• Broadcast events• Retain them in the log• Evolve the event-stream with

streaming functions• Recasting the event stream into

views when you need to query.

Event Driven Services

Page 98: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

98

Confluent Streaming Platform

Page 99: Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

99

Thank You@benstopford

Blog Series: https://www.confluent.io/blog/tag/microservices/