Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS...

19
Data Streaming with Kafka November 11, 2020

Transcript of Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS...

Page 1: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Data Streaming

with Kafka

November 11, 2020

Page 2: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Three initiatives to turn your Siebel into the best non-SaaS CRM

TACTICAL STRATEGICAL

TEC

HN

ICA

LBU

SIN

ESS

KAFKA

Adopt

Real-Time Data Streaming

with Kafka

SIEBEL UX

Improve

your Siebel UX with

Nexus Face and Nexus Apps

CLOUD NATIVE

Modernize

your Siebel Platform with

Cloud Native Architecture

SIEBEL UX

Transform

your Siebel UX with

Nexus Face and Nexus Apps

Page 3: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Adopt Real-Time

Data Streaming

with Kafka

KAFKA

Page 4: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Siebel CRM Cloud Native Sessions: https://bit.ly/SiebelCloudNative

Page 5: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Kafka comes with the Siebel Cloud Native Architecture

Page 6: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Siebel Kafka integration: background and demo

http://bit.ly/SiebelKafka 35'20'' - 45'50''

Page 7: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

TECHNICAL CHARACTERISTICS

• Delivers messages with very low latency

• Scales to a thousand brokers and trillions of messages per day

• Stores streams of data safely in fault-tolerant cluster

• Processes streams of events in real time

01

SIEBEL RELATED NOTES

• Expected in 2021; update to the latest version is required

• Publish and subscribe to events declaratively and using scripting

• Siebel Cloud Native Architecture uses Kafka internally

• Now Siebel can work with Kafka using CDC or custom

Connectors

02

USE CASES

• Audit of the customers’ data access events

• Speed up queries: replace read calls with data syncs

• Google-like search: push Siebel data into Elasticsearch

• Identify and act on risks and opportunities in Siebel

data updates using Streams API

03

KAFKA

Page 8: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Consumers/

Target Systems

Producers/

Source Systems

Data streaming with Kafka: Real-time performance and reliability

Mobile Apps

Data Lakes / DWH

Search

Audit

Microservices

SaaS Apps

Enterprise Apps

Sensor Data

Clicks Streams

Social Data

Mobile Apps

Microservices

SaaS Apps

Enterprise Apps

Kafka

Topics

Stream Processors

(Kafka Streams API,

ksqlDB, Spark)

Source

Connectors

Sink

Connectors

Page 9: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Common Kafka Usage Scenario for Siebel applications

Audit: record

user actions

and customers’

data access

events

Query

performance:

replace read

calls with data

syncs

Google-like

search: push

Siebel data

into Elastic-

search

Streams API:

identify and act

on risks &

opportunities in

Siebel data

updates

Page 10: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Speeding up queries with CQRS pattern

main

storage

command

handler

consumer

read

storage

query

handler

commands

queries

projection

Page 11: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Speeding up queries in the Siebel CRM ecosystem

Siebel DBSiebel REST

API

API Gatewaymobile app

On-premAzure

Page 12: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Speeding up queries in the Siebel CRM ecosystem

Siebel DBSiebel REST

API

API Gateway

CosmosDBREST API

commands

queries

kafka

mobile app

On-prem

Azure

Read Performance Gain

• Record count in DB, BC, CosmosDB

• Accounts 600k > 550k > 500k

• Assets: 85m > 2.5m > 1m

• Flat by System ID – 1.2-1.7x

• Hierachy by System ID – 2.5-3.8x

• Large Response Size – 5.5-13.5x

When to apply

• Pushing Siebel data into

«Digital» applications

• Replacing Siebel VBC by data

streaming from source systems

Current Kafka integration

options for Siebel

• Source connector – CDC (Debezium,

GoldenGate, Striim, etc)

• Sync connector – REST API based

Page 13: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Implementing Google-like search in the Siebel UI

Siebel DBSiebel PM via

Nexus Bridge

Nexus Face

built UI

Siebel Config

Elastic

search

kafka

REST APIsearch

visibility

replication

Data Access

Contol

Page 14: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Kafka is emerging as a middleware platform

SUPPORT OF VARIOS

PATTERNS

While Kafka’s core

integration pattern is event-

based, it also supports Fire-

and-Forget, Publish /

Subscribe, Request-

Response / RPC, Batch and

other patterns.

SINGLE SET OF

TOOLS

Kafka provides all required

middleware components,

e.g., messaging, storage,

connectors, processing.

How many products do you

currently run in your

middleware stack?

RELIABLE & SCALABLE

INFRASTRUCTURE

Kafka offers extreme scale

and throughput while being

highly available. With the

decoupling of clients, it

solved the problems of

backpressure or unavailable

consumers.

https://www.kai-waehner.de/blog/2019/03/07/apache-kafka-middleware-mq-etl-esb-comparison/

1 2 3

Page 15: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Organizational implications of Kafka’s “Dumb pipes smart endpoints” approach

Source

Connector

(if needed)

Topics

Sink

Connector

(if needed)

Event Consumer

or

Target System

Event Producer

or

Source System

Source

Connector

(if needed)

Topics

Sink

Connector

(if needed)

Event Consumer

or

Target System

Event Producer

or

Source System

1st System Dev Team ESB Dev Team 2nd System Dev Team

1st System Dev Team Operations team 2nd System Dev Team

ESB

KAFKA

Page 16: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Stream processing

Producers/

Source

Systems

Consumers/

Target

Systems

Consumers/

Target

Systems

Stream Processors

(Kafka Streams API,

ksqlDB, Spark)

Two Kafka’s technologies

• ksqlDB (SQL like)

• Streaming API (Java, Scala)

Capabilities

• Transformation of messages

• Filtering

• Aggregation

• Joining topics

• Time windowing

Present Kafka topics as

• Streams

• Tables

Main use-cases: real time action on

• Detected anomalies

• Prediction / Recommendation

• Analyticshttps://kafka-tutorials.confluent.io/

Page 17: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Adoption of Kafka in the enterprise

https://www.confluent.io/blog/event-streaming-benefits-increase-with-greater-maturity/

Page 18: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,

Your next steps

Watch recordings from Siebel CRM Virtual Summit

Determine your use-cases for Kafka

Adopt Event Driven Framework and Cloud Native Architecture

Find out if your organization is already

running Kafka

Deploy and run Kafka in Kubernetes cluster

1 2 3 4 5

Page 19: Data Streaming with Kafka...Kafka is emerging as a middleware platform SUPPORT OF VARIOS PATTERNS While Kafka’s core integration pattern is event-based, it also supports Fire-and-Forget,