Post on 21-Jan-2018
1
Building Event Driven Services with Apache Kafka and Kafka StreamsBen Stopford@benstopford
2
Build Features
Build for the Future
3
Evolution!
4
KAFKA
ServingLayer
(Cassandra etc.)
Kafka Streams / KSQL
Streaming Platforms
Data is embedded in each engine
High Throughput Messaging
Clustered Java App
5
authorization_attempts possible_fraud
Streaming Example
6
CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
authorization_attempts possible_fraud
7
CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
authorization_attempts possible_fraud
8
CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
authorization_attempts possible_fraud
9
CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
authorization_attempts possible_fraud
10
CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
authorization_attempts possible_fraud
11
CREATE STREAM possible_fraud ASSELECT card_number, count(*)FROM authorization_attemptsWINDOW TUMBLING (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
authorization_attempts possible_fraud
12
Streaming == Manipulating Data in Flight
13
Business Applications
14
EcosystemsApp
Increasingly we build ecosystems
15
SOA / Microservices / EDA
CustomerService
ShippingService
16
The Problem is DATA
17
Most services share the same core facts.
Catalog
Most services live in here
18
So how do we share data between services?
OrdersService
ShippingService
CustomerService
Webserver
19
Buying an iPad (with REST)
SubmitOrder
shipOrder() getCustomer()
OrdersService
ShippingService
CustomerService
Webserver
20
Buying an iPad with Events
Message Broker (Kafka)
Notification Data is replicated
(incrementally)
SubmitOrder
Order Created
Customer Updated
OrdersService
ShippingService
CustomerService
Webserver
KAFKA
21
Events for Notification Only
Message Broker (Kafka)
SubmitOrder
Order Created
getCustomer()REST
Notification
OrdersService
ShippingService
CustomerService
Webserver
KAFKA
22
Events for Data Locality
Customer Updated
SubmitOrder
Order Created
Data is replicated
OrdersService
ShippingService
CustomerService
Webserver
KAFKA
23
Events have two hats
Notification Data replication
24
Events are the key to scalable service ecosystems
25
Streaming is the toolset for dealing with events as they move!
26
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
27
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
28
What is a Distributed Log?
29
Shard on the way in
ProducingServices
Kafka
ConsumingServices
30
Each shard is a queue
ProducingServices
Kafka
ConsumingServices
31
Consumers share load
ProducingServices
Kafka
ConsumingServices
32
Reduces to a globally ordered queue
33
Load Balanced Services
34
Fault Tolerant Services
35
Build ‘Always On’ Services
Rely on Fault Tolerant Broker
36
Services can “Rewind & Replay” the log
Rewind & Replay
37
Compacted Log(retains only latest version)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1
38
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
39
Kafka Connect
KafkaConnect
KafkaConnect
Kafka
40
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
41
A database engine for data-in-flight
42
SELECT card_number, count(*)FROM authorization_attemptsWINDOW (SIZE 5 MINUTE)GROUP BY card_numberHAVING count(*) > 3;
Continuously Running Queries
43
Features: similar to database query engine
JoinFilterAggr-egate
View
Window
44
KAFKA
Buffer 5 mins
Windows
45
Lookup tables
KAFKA
46
Stateful Stream Processing
stream
Compactedstream
Join
Stream
Table
KafkaKafka Streams
47
Scales Out
48
Embeddable
Orders Service
Kafka
Orders Service
Orders Service
49
Streaming is about
1. Processing data incrementally
2. Moving data to where it needs to be processed (quickly and efficiently)
On Notification
Data Replication
50
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
51
Steps to Streaming Services
52
1. Take Responsibility for the past and evolve
53
Stay Simple. Take Responsibility for the past
Browser
Webserver
54
Evolve Forwards
Browser
WebserverOrdersService
55
2. Raise events. Don’t talk to services.
56
Events have two hats
Notification Data replication
57
Raise events. Don’t talk to services
Browser
WebserverOrdersService
58
KAFKA
Order RequestedOrder
Received
Browser
Webserver
OrdersService
Raise events. Don’t talk to services
59
KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
Raise events. Don’t talk to services
60
KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
Use Kafka as a Backbone for Events
61
3. Use Connect (& CDC) to evolve away from legacy
62KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
Evolve away from Legacy
63KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
Use the Database as a ‘Seam’
Connect
Products
64
4. Make use of Schemas
65KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
Use Schemas to validate Backwards Compatibility
Connect
ProductsSchema Registry
66
5. Use the Single Writer Principal
67KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
Apply the single writer principal
Connect
ProductsSchema Registry
Order Completed
68
Orders Service
EmailService
T1 T2
T3
T4
RESTService
T5
Single Writer Principal
69
Single Writer Principal
- Creates local consistency points in the absence of Global Consistency
- Makes schema upgrades easier to manage.
70
6. Store Datasets in the Log
71
Messaging that Remembers
Orders Customers
PaymentsStock
72KAFKA
Order Requested
Order Validated
Order Received
Browser
Webserver
OrdersService
New Service, No Problem!
Connect
Products
Schema Registry
Order Completed Repricing
73
Orders Customers
PaymentsStock
Single, Shared Source of Truth
74
But how do you query a log?
75
7. Move Data to Code
76
77
Connect
Order Requested
Order Validated
Order Completed
Order Received
Products
Browser
Webserver
Schema Registry
OrdersService Stock
Stock
Materialize Stock ‘View’ Inside Service
KAFKA
78
Connect
Order Requested
Order Validated
Order Completed
Order Received
Products
Browser
Webserver
Schema Registry
OrdersService Stock
Stock
Take only the data we need
KAFKA
79
Data Movement
Be realistic:• Network is no longer the bottleneck• Indexing is:
• In memory indexes help• Keep datasets focused
80
8. Use the log instead as a ‘database’
81
Connect
Order Requested
Order Validated
Order Completed
Order Received
Products
Browser
Webserver
Schema Registry
OrdersService
Reserved Stocks
Stock
Stock
Reserved Stocks
Apply Event Sourcing
KAFKA
Table
82
Connect
Order Requested
Order Validated
Order Completed
Order Received
Products
Browser
Webserver
Schema Registry
OrdersService
Reserved Stocks
Stock
Stock
Reserved Stocks
Order Service Loads Reserved Stocks on Startup
KAFKA
83
Kafka has several features for reducing the need to move data on startup
- Standby Replicas- Disk Checkpoints- Compacted topics
84
9. Use Transactions to tie All Interactions Together
85
OrderRequested(IPad)
2a. Order Validated
2c. Offset Commit2b. IPad Reserved
Internal State:Stock = 17Reservations = 2
Tie Events & State with Transactions
86
Connect
TRANSACTION
Order Requested
Order Validated
Order Completed
Order Received
Products
Browser
Webserver
Schema Registry
OrdersService
Reserved Stocks
Stock
Stock
Reserved Stocks
Transactions
KAFKA
87
10. Bridge the Sync/Async Divide with a Streaming Ecosystem
88
POST
GET
Load
Ba
lance
r
ORDE
RSOR
DERS
OV T
OPIC
Order ValidationsKAFKA
INVENTORY
Orders
Inventory
Fraud Service
Order DetailsService
InventoryService
(see previous figure)
Order Created
Order Validated
Orders View
Q in CQRS
Orders ServiceC is CQRS
Services in the Micro: Orders ServiceFind the code online!
89
Orders Customers
Payments Stock
Query Engine (Kstreams/KSQL)
Larger Ecosystems
HISTORICAL EVENT STREAMS
90
Kafka
KAFKA
New York
Tokyo
London
Global / Disconnected Ecosystems
91
So…
92
Good architectures have little to do with this:
93
It’s about how systems evolves over time
94
Request driven isn’t enough
• High coupling• Hard to handle
async flows• Hard to move and
join datasets.
95
Leverage the Duality of Events
Notification Data replication
96
With a toolset built for data in flight
97
• Broadcast events• Retain them in the log• Evolve the event-stream with
streaming functions• Recasting the event stream into
views when you need to query.
Event Driven Services
98
Confluent Streaming Platform
99
Thank You@benstopford
Blog Series: https://www.confluent.io/blog/tag/microservices/