Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

1Apache Kafka meetup, Munich, Germany, Jan 25, 2017

Introducing Kafka’s Streams APITaking real-time processing to the mainstream

Michael Noll <michael@confluent.io>Product manager, Confluent

Our Dream Our Reality

6Confidential

Kafka’s Streams APITaking real-time processing to the mainstream

Taking real-time processing to the mainstreamKafka’s Streams API• Powerful yet easy-to-use library to build stream

processing apps• Apps are standard Java applications that run on client

machines• Part of open source Apache Kafka, introduced in 0.10+• https://github.com/apache/kafka/tree/trunk/streams

Streams API

Your App

KafkaCluster

<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>0.10.1.1</version></dependency>

Build Applications, not Clusters!

“Cluster to go”: elastic, scalable, distributed, fault-tolerant, secure apps

”Database to go”: tables, state management, interactive queries

Equally viable for S / M / L / XL / XXL use cases

Ok. Ok. Ok.

Runs everywhere: from containers to cloud

When to use Kafka’s Streams API

Use case examples• Customer 360-degree view• Fleet or inventory management• Fraud detection• Real-time monitoring &

intelligence• Location-based marketing• Virtual Reality (avatar replication)• <and more>

To build real-time applications for your core business

Scenarios• Microservices• Fast Data apps for small and big

data• Reactive applications• Continuous queries and

transformations• Event-triggered processes• The “T” in ETL• <and more>

Some public use cases in the wild & external articles• Why Kafka Streams: towards a real-time streaming architecture, by Sky

Betting and Gaming• http://engineering.skybettingandgaming.com/2017/01/23/streaming-architectures/

• Applying Kafka’s Streams API for social messaging at LINE Corp.• http://developers.linecorp.com/blog/?p=3960 • Production pipeline at LINE, a social platform based in Japan with 220+ million users

• Microservices and Reactive Applications at Capital One• https://

speakerdeck.com/bobbycalderwood/commander-decoupled-immutable-rest-apis-with-kafka-streams

• Containerized Kafka Streams applications in Scala, by Hive Streaming• https://www.madewithtea.com/processing-tweets-with-kafka-streams.html

• Geo-spatial data analysis• http://www.infolace.com/blog/2016/07/14/simple-spatial-windowing-with-kafka-streams/

• Language classification with machine learning• https://dzone.com/articles/machine-learning-with-kafka-streams

Kafka Summit NYC, May 09

Here, the community will sharelatest Kafka Streams use cases.

http://kafka-summit.org/

Do more with less

17Confidential

Architecture comparison: use case exampleReal-time dashboard for security monitoring

“Which of my data centers are under attack?”

With Streams API

Organizational benefits: decouple teams and roadmaps, scale people

Available APIs

• API option 1: DSL (declarative)

KStream<Integer, Integer> input = builder.stream("numbers-topic");

// Stateless computationKStream<Integer, Integer> doubled = input.mapValues(v -> v * 2);

// Stateful computationKTable<Integer, Integer> sumOfOdds = input .filter((k,v) -> v % 2 != 0) .selectKey((k, v) -> 1) .groupByKey() .reduce((v1, v2) -> v1 + v2, "sum-of-odds");

The preferred API for most use cases.

9 out of 10 users pick the DSL.

Particularly appeals to:• Fans of Scala, functional

programming• Users familiar with e.g. Spark

• API option 2: Processor API (imperative)

class PrintToConsoleProcessor implements Processor<K, V> {

@Override public void init(ProcessorContext context) {}

@Override void process(K key, V value) { System.out.println("Got value " + value); }

@Override void punctuate(long timestamp) {}

@Override void close() {}}

Full flexibility but more manual work

Appeals to:• Users who require functionality

that isnot yet available in the DSL

• Users familiar with e.g. Storm, Samza• Still, check out the DSL!

Writing and running your first application• Preparation: Ensure Kafka cluster is accessible, has data to process

• Step 1: Write the application code in Java or Scala, see next slide• Great starting point: https://github.com/confluentinc/examples • Documentation: http://docs.confluent.io/current/streams/

• Step 2: Run the application• During development: from your IDE, from CLI … (pro tip: Application Reset Tool is great for

playing around)• In production: e.g. bundle as fat jar, then `java -cp my-fatjar.jar

com.example.MyStreamsApp`• http://

docs.confluent.io/current/streams/developer-guide.html#running-a-kafka-streams-application

Example: complete app, ready for production at large-scaleWordCoun

App configuration

Define processing(here: WordCount)

Start processing

Key concepts

Kafka’s data model Kafka’s Streams API

30Confidential

Streams and TablesStream Processing meets Databases

Key observation: close relationship between Streams and Tables

http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables

Key features

Key features in 0.10• Native, 100%-compatible Kafka integration

Native, 100% compatible Kafka integration

Read from Kafka

Write to Kafka

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features

Secure stream processing with the Streams API• Your applications can leverage all client-side security features in Apache Kafka• Security features include:

• Encrypting data-in-transit between applications and Kafka clusters• Authenticating applications against Kafka clusters (“only some apps may talk to the

production cluster”)• Authorizing application against Kafka clusters (“only some apps may read data from

sensitive topics”)

Streams API

Your AppKafkaCluster

”I’m the Payments app!” “Ok, you may read the Purchases topic.”

Data encryption

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant• Stateful and stateless computations

Stateful computations• Stateful computations include aggregations (e.g. counting), joins, and windowing• State stores are the backbone of state management

• … are local for best performance• … are continuously backed up to Kafka to enable elasticity and fault-tolerance• ... are per stream task for isolation, think: share-nothing

• Pluggable storage engines• Default: RocksDB (a key-value store) to allow for local state that is larger than available

RAM• You can also use your own storage engine

• From the user perspective• DSL: no need to worry about anything, state management is automatically being done for

you• Processor API: direct access to state stores – very flexible but more manual work

Use case: real-time, distributed joins at large scale

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant• Stateful and stateless computations• Interactive queries

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant• Stateful and stateless computations• Interactive queries• Time model

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant• Stateful and stateless computations• Interactive queries• Time model• Supports late-arriving and out-of-order data

Out-of-order and late-arriving data: example

Users with mobile phones enterairplane, lose Internet connectivity

Emails are being writtenduring the 8h flight

Internet connectivity is restored,phones will send queued emails now,

though with an 8h delay

Bob writes Alice an email at 2 P.M.

Bob’s email is finally being sent at 10 P.M.

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant• Stateful and stateless computations• Interactive queries• Time model• Supports late-arriving and out-of-order data• Windowing

Windowing• To group related events in a stream• Use case examples:

• Time-based analysis of ad impressions (”number of ads clicked in the past hour”)

• Monitoring statistics of telemetry data (“1min/5min/15min averages”)• Analyzing user browsing sessions on a news site

Input data, wherecolors represent

different users events

Rectangles denotedifferent event-time

windows

processing-time

event-time

windowing

Key features in 0.10• Native, 100%-compatible Kafka integration• Secure stream processing using Kafka’s security features• Elastic and highly scalable• Fault-tolerant• Stateful and stateless computations• Interactive queries• Time model• Supports late-arriving and out-of-order data• Windowing• Millisecond processing latency, no micro-batching• At-least-once processing guarantees (exactly-once is in the works as we

speak)

Roadmap Outlook

Roadmap outlook for Kafka StreamsUpcoming in Confluent 3.2 & Apache Kafka 0.10.2• Sessionization aka “session windows” -- e.g. for analyzing user browsing behavior• Global KTables (vs. today’s partitioned KTables) – e.g. for convenient facts-to-

dimensions joins• Now you can use newer versions of the Streams API against older clusters, too• Further operational metrics to improve monitoring and 24x7 operations of apps

Feature highlight for 2017• Exactly-Once processing semantics• But much more to come!

Wrapping Up

Where to go from here• Kafka’s Streams API is available in Confluent Platform 3.1 and in Apache

Kafka 0.10.1• http://www.confluent.io/download

• Demo applications: https://github.com/confluentinc/examples • Interactive Queries, Joins, Security, Windowing, Avro integration, …

• Confluent documentation: http://docs.confluent.io/current/streams/• Quickstart, Concepts, Architecture, Developer Guide, FAQ

• Recorded talks• Introduction to Kafka’s Streams API:

http://www.youtube.com/watch?v=o7zSLNiTZbA• Application Development and Data in the Emerging World of Stream Processing (higher

level talk): https://www.youtube.com/watch?v=JQnNHO5506w

72Confidential

Thank You

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Data & Analytics

Transcript of Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Franz Kafka's Outsider Jurisprudence. - Research at UVU

Paris Kafka Meetup - Concepts & Architecture

Kafka Utrecht Meetup

Kafka's ''Schloß'' and the Subversion of Plot

Barcelona Spain Apache Spark Meetup Oct 20, 2015: Spark Streaming, Kafka, MLlib, SQL, Project Tungsten, Text Analytics, Natural Language Processing

Kafka's Antitheatrical Gestures Registered office ...puchner/Kafka's antitheatrical gestures... · THE GERMANIC REVIEW Kafka’ s Antitheatrical Gestures MARTIN PUCHNER N o writer

APACHE KAFKA Cluster and Zookeeper Setup/Administration in … · To understand how Kafka does these things, let's dive in and explore Kafka's capabilities from the bottom up. First

Dublin Apache Kafka Meetup, 30 August 2017 …...© 2017 Mesosphere, Inc. All Rights Reserved. 1 Dublin Apache Kafka Meetup, 30 August 2017The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*,

Kafka's BPR Slides!

'Interpreting and Translating Gestures for Power Play in Kafka s In … · 2017-02-12 · Interpreting and Translating Gestures for Power Play in Kafka's 11In the Penal Colony" Annette

Kierkegaard - static-curis.ku.dk file · Web viewSøren Kierkegaard's Influence on Franz Kafka. Isak Winkel Holm [uredigeret kapitel; ... Nothing indicates that Kafka's use of the

Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

Ruhleder - Franz Kafka's 'Das Urteil'~ an Interpretation

Net flix kafka seattle meetup

Kafka & Hadoop - for NYC Kafka Meetup

Nyc kafka meetup 2015 - when bad things happen to good kafka clusters

Kafka's concept of humour.pdf

An Interpretation of Paradox Literary Technique in Kafka's ...

Gregor Samsa's Insect Transformation in Franz Kafka's the Metamorphosis

Kafka Audit - Kafka Meetup - January 27th, 2015

Dublin Apache Kafka Meetup, 30 August 2017 …...© 2017 Mesosphere, Inc. All Rights Reserved. 1 Dublin Apache Kafka Meetup, 30 August 2017The SMACK Stack: Spark, Mesos, Akka, Cassandra*,