Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

download Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

of 99

  • date post

    21-Jan-2018
  • Category

    Technology

  • view

    145
  • download

    2

Embed Size (px)

Transcript of Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

  1. 1. 1 Building Event Driven Services with Apache Kafka and Kafka Streams Ben Stopford @benstopford
  2. 2. 2 Build Features Build for the Future
  3. 3. 3 Evolution!
  4. 4. 4 KAFKA Serving Layer (Cassandra etc.) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
  5. 5. 5 authorization_attempts possible_fraud Streaming Example
  6. 6. 6 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  7. 7. 7 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  8. 8. 8 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  9. 9. 9 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  10. 10. 10 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  11. 11. 11 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  12. 12. 12 Streaming == Manipulating Data in Flight
  13. 13. 13 Business Applications
  14. 14. 14 EcosystemsApp Increasingly we build ecosystems
  15. 15. 15 SOA / Microservices / EDA Customer Service Shipping Service
  16. 16. 16 The Problem is DATA
  17. 17. 17 Most services share the same core facts. Catalog Most services live in here
  18. 18. 18 So how do we share data between services? Orders Service Shipping Service Customer Service Webserver
  19. 19. 19 Buying an iPad (with REST) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
  20. 20. 20 Buying an iPad with Events Message Broker (Kafka) Notification Data is replicated (incrementally) Submit Order Order Created Customer Updated Orders Service Shipping Service Customer Service Webserver KAFKA
  21. 21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
  22. 22. 22 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
  23. 23. 23 Events have two hats Notification Data replication
  24. 24. 24 Events are the key to scalable service ecosystems
  25. 25. 25 Streaming is the toolset for dealing with events as they move!
  26. 26. 26 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  27. 27. 27 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  28. 28. 28 What is a Distributed Log?
  29. 29. 29 Shard on the way in Producing Services Kafka Consuming Services
  30. 30. 30 Each shard is a queue Producing Services Kafka Consuming Services
  31. 31. 31 Consumers share load Producing Services Kafka Consuming Services
  32. 32. 32 Reduces to a globally ordered queue
  33. 33. 33 Load Balanced Services
  34. 34. 34 Fault Tolerant Services
  35. 35. 35 Build Always On Services Rely on Fault Tolerant Broker
  36. 36. 36 Services can Rewind & Replay the log Rewind & Replay
  37. 37. 37 Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
  38. 38. 38 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  39. 39. 39 Kafka Connect Kafka Connect Kafka Connect Kafka
  40. 40. 40 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  41. 41. 41 A database engine for data-in-flight
  42. 42. 42 SELECT card_number, count(*) FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; Continuously Running Queries
  43. 43. 43 Features: similar to database query engine JoinFilter Aggr- egate View Window
  44. 44. 44 KAFKA Buffer 5 mins Windows
  45. 45. 45 Lookup tables KAFKA
  46. 46. 46 Stateful Stream Processing stream Compacted stream Join Stream Table KafkaKafka Streams
  47. 47. 47 Scales Out
  48. 48. 48 Embeddable Orders Service Kafka Orders Service Orders Service
  49. 49. 49 Streaming is about 1. Processing data incrementally 2. Moving data to where it needs to be processed (quickly and efficiently) On Notification Data Replication
  50. 50. 50 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  51. 51. 51 Steps to Streaming Services
  52. 52. 52 1. Take Responsibility for the past and evolve
  53. 53. 53 Stay Simple. Take Responsibility for the past Browser Webserver
  54. 54. 54 Evolve Forwards Browser Webserver Orders Service
  55. 55. 55 2. Raise events. Dont talk to services.
  56. 56. 56 Events have two hats Notification Data replication
  57. 57. 57 Raise events. Dont talk to services Browser Webserver Orders Service
  58. 58. 58 KAFKA Order Requested Order Received Browser Webserver Orders Service Raise events. Dont talk to services
  59. 59. 59 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Raise events. Dont talk to services
  60. 60. 60 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Kafka as a Backbone for Events
  61. 61. 61 3. Use Connect (& CDC) to evolve away from legacy
  62. 62. 62 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Evolve away from Legacy
  63. 63. 63KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use the Database as a Seam Connect Products
  64. 64. 64 4. Make use of Schemas
  65. 65. 65KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Schemas to validate Backwards Compatibility Connect Products Schema Registry
  66. 66. 66 5. Use the Single Writer Principal
  67. 67. 67KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Apply the single writer principal Connect Products Schema Registry Order Completed
  68. 68. 68 Orders Service Email Service T1 T2 T3 T4 REST Service T5 Single Writer Principal
  69. 69. 69 Single Writer Principal - Creates local consistency points in the absence of Global Consistency - Makes schema upgrades easier to manage.
  70. 70. 70 6. Store Datasets in the Log
  71. 71. 71 Messaging that Remembers Orders Customers Payments Stock
  72. 72. 72 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service New Service, No Problem! Connect Products Schema Registry Order Completed Repricing
  73. 73. 73 Orders Customers Payments Stock Single, Shared Source of Truth
  74. 74. 74 But how do you query a log?
  75. 75. 75 7. Move Data to Code
  76. 76. 76
  77. 77. 77 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Materialize Stock View Inside Service KAFKA
  78. 78. 78 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Take only the data we need KAFKA
  79. 79. 79 Data Movement Be realistic: Network is no longer the bottleneck Indexing is: In memory indexes help Keep datasets focused
  80. 80. 80 8. Use the log instead as a database
  81. 81. 81 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Apply Event Sourcing KAFKA Table
  82. 82. 82 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Order Service Loads Reserved Stocks on Startup KAFKA
  83. 83. 83 Kafka has several features for reducing the need to move data on startup - Standby Replicas - Disk Checkpoints - Compacted topics
  84. 84. 84 9. Use Transactions to tie All Interactions Together
  85. 85. 85 OrderRequested (IPad) 2a. Order Validated 2c. Offset Commit 2b. IPad Reserved Internal State: Stock = 17 Reservations = 2 Tie Events & State with Transactions
  86. 86. 86 Connect TRANSACTION Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Transactions KAFKA
  87. 87. 87 10. Bridge the Sync/Async Divide with a Streaming Ecosystem
  88. 88. 88 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service (see previous figure) Order Created Order Validated Orders View Q in CQRS Orders Service C is CQRS Ser