A Practical Guide to Selecting a Stream Processing Technology

Michael � G. � NollProduct � Manager, � Confluent

Kafka Talk SeriesDate Title

Sep 27 Introduction To Streaming Data and Stream Processing with Apache Kafka

Oct 06 Deep Dive into Apache Kafka

Oct 27 Data Integration with Apache Kafka

Nov 17 Demystifying Stream Processing with Apache Kafka

Dec 01 A Practical Guide to Selecting a Stream Processing Technology

Dec 15 Streaming in Practice: Putting Apache Kafka in Production

https://www.confluent.io/apache-‐kafka-‐talk-‐series

Agenda

• Recap: � What � is � Stream � Processing?• The � Three � Pillars � of � Stream � Processing � in � Practice• Key � Selection � Criteria• Organizational/Non-Technical � Dimensions• Technical � Dimensions

• Summary

Agenda

• Summary

Agenda

• Summary

Powered by Kafka (﴾thousands more)﴿

Spark Streaming API (﴾2.0)﴿

Kafka’s Streams API (﴾0.10)﴿

Example: Streams and Tables in Kafka

Word Count

hello 2

kafka 1

world 1

… …

Streams & Databases

• A � stream � processing � technology � must � have � first-class � support � for Streams � and Tables• With � scalability, � fault � tolerance, � …

• Why? � Because � most � use � cases � require � not � just � one, � but � both!• Support � – or � lack � thereof � – strongly � impacts � the � resulting �

technical � architecture � and � development � efforts• No � support � means:• Painful � Do-It-Yourself• Increased � complexity, � more � moving � pieces � to � juggle

Agenda

• Summary

Agenda

• Summary

Organizational/Non-‐Tech Dimensions

• Can � your � org � understand � and � leverage � the � technology?• Familiarity � with � languages; � intuitive � concepts � and � APIs; � trainings

• Are � you � permitted � to � use � it � in � your � organization?• Security � features, � licensing, � open � source � vs. � proprietary

• Can � you � continue � to � use � it � in � the � future?• Longevity � of � technology, � licensing, � vendor � strength

• Do � you � believe � in � the � long-term � vision?• Switching � technologies � in � an � organization � is � often � expensive/slow: �

legacy � migration, � re-training, � resistance � to � change, � etc.

• What � is � the � path � and � time � to � success?• Can � you � move � smoothly � and � quickly � from � proof-of-concept � to �

production?

• Areas � and � range � of � applicability in � your � organization• General-purpose � vs. � niche � technology• Viable � for � S/M/L/XL � use � cases � vs. � for � XL � use � cases � only• Building � core � business � apps � vs. � doing � backend � analytics

Licensing Vision/Roadmap ROI

Impact onOrganization

Broad vs. NicheApplicability

Time to Market

ProfessionalServices

Documentation Examples User CommunityLearning Curve

Impact on Tools,Infrastructure, …

Agenda

• Summary

Technical Dimensions

Reprocessing Scalability &Elasticity

Fault Tolerance

API Dev/OpsLifecycle

Security ProcessingModel

Out of OrderData

Abstractions Time Model WindowingState

• Stateful � processing � of � any � kind � requires…state• Many � (most?) � use � cases � for � stream � processing � are � stateful• Joins, � aggregations, � windowing, � counting, � ...

• Is � state � performant? � Local � vs. � remote � state?

• Is � state � performant? � Local � vs. � remote � state?• Is � state � fault-tolerant? � How � fast � is � recovery/failover?

• Is � state � performant? � Local � vs. � remote � state?• Is � state � fault-tolerant? � How � fast � is � recovery/failover?• Is � state � interactively � queryable?• Kafka: � ready � for � use � (GA)• Spark, � Flink: � under � development � (alpha)• Storm, � Samza, � and � others: � not � available

Fault Tolerance

Out of OrderData

Abstractions

• What � are � the � data � model � and � the � available � abstractions?• Most � common � abstraction: � stream of � records, � events• Kafka, � Spark, � Storm, � Samza, � Flink, � Apex, � ...

• New, � very � powerful: � table � of � records• Currently � unique � to � Kafka• Represents � latest � state and � materialized � views• State � must � have � a � first-class � abstraction � because, � as � we � just � saw � in �

the � previous � section, � state � is � crucial � for � stream � processing!

Fault Tolerance

Out of OrderData

Time model

• Different � use � cases � require � different � time � semantics• Great � majority � of � use � cases � require � event-time semantics• Other � use � cases � may � require � processing-time (e.g. � real-

time � monitoring) � or � special � variants � like � ingestion-time• A � stream � processing � technology � should, � at � a � minimum, �

support � event-time � to � cover � most � use � cases � in � practice• Examples: � Kafka, � Beam, � Flink

Time Model

Fault Tolerance

Out of OrderData

Windowing• Windowing � is � an � operation � that � groups events

Windowing

Input data, wherecolors represent

different users events

Rectangles denotedifferent event-‐time

windows

processing-‐time

event-‐time

windowing

alicebob

Windowing• Windowing � is � an � operation � that � groups events• Most � commonly � needed: � time � windows, � session � windows• Examples:• Real-time � monitoring: � 5-minute � averages• Reader � behavior � on � a � website: � user � browsing � sessions

Windowing

Fault Tolerance

Out of OrderData

Out-‐of-‐order and late-‐arriving data

• Is � very � common in � practice, � not � a � rare � corner � case• Related � to � time � model � discussion

Users with mobile phones enterairplane, lose Internet connectivity

Emails are being writtenduring the 10h flight

Internet connectivity is restored,phones will send queued emails now

• Is � very � common in � practice, � not � a � rare � corner � case• Related � to � time � model � discussion

• We � want � control over � how � out-of-order � data � is � handled• Example:• We � process � data � in � 5-minute � windows, � e.g. � compute � statistics• When � event � arrives � 1 � minute � late: � update the � original � result!• When � event � arrives � 2 � hours � late: � discard it!

• Handling � must � be � efficient because � it � happens � so � often

Fault Tolerance

Out of OrderData

Reprocessing

• Re-process � data � by � rewinding � a � stream � back � in � time• Use � cases � in � practice � include• Correcting � output � data � after � fixing � a � bug• Facilitate � iterative � and � explorative � development• A/B � testing• Processing � historical � data• Walking � through � "What � If?" � scenarios

• Also: � often � used � behind-the-scenes � for � fault � tolerance

Fault Tolerance

Out of OrderData

Scalability, Elasticity, Fault Tolerance

• Can � the � technology � scale according � to � your � needs?• Desired � latency, � throughput?• Able � to � process � millions � of � messages � per � second?

• What � is � the � minimum � footprint?• Expand/shrink � capacity � dynamically � during � operations?

• Helps � with � resource � utilization � because � most � stream � apps � run � continuously• Resilience and � fault � tolerance

• Which � guarantees � for � data � delivery � and � for � state? � "At-least-once", � "exactly-once", � "effectively-once", � etc.

• Failover � behavior � and � recovery � time? � Automated � or � manual?• Any � negative � impact � of � fault � tolerance � features � on � performance?

Fault Tolerance

Out of OrderData

Security

• To � meet � internal � security � policies, � legal � compliance, � etc.• Typical � base � requirements � for � stream � processing � applications:• Encrypt � data-in-transit � (e.g. � from/to � Kafka)• Authentication: � "only � some � applications � may � talk � to � production"• Authorization: � "access � to � sensitive � data � such � as � PII � is � restricted”

• The � easier � it � is � to � use � security � features, � the � more � likely � they � are � actually � being � used � in � practice

Fault Tolerance

Out of OrderData

Processing Model• True � stream � processing � is � record-at-a-time processing

• Benefits � include � low � latency (millisecs), � dealing � efficiently � with � out-of-order � data• Can � provide � both � latency � and � high � throughput � via � internal � optimizations• Examples: � Kafka, � Storm, � Samza, � Flink, � Beam

• Some � processing � technologies � opt � for � (micro)batching• Micro-batching � has � no � true � benefits: � consider � it � a � technical � workaround � to �

shoehorn � stream-like � functionality � into � a � tool• Suffers � from � significant � overhead � when � dealing � with � e.g. � out-of-order/late-arriving �

data, � when � performing � windowed � analyses � (e.g. � session � windows)• Typically � a � strong � blocker � for � use � cases � such � as � fraud � detection � or � anything � where �

"a � few � seconds" � of � latency � is � prohibitive• Examples: � Spark, � Storm � (Trident), � Hadoop*

Fault Tolerance

Out of OrderData

• Choice � of � API � is � a � subjective � matter � – skills, � preference, � …• Typical � options• Declarative, � expressive � API: � operations � like � map(), � filter()• Imperative, � lower-level � API: � callbacks � like � process(event)• Streaming � SQL: � STREAM SELECT … FROM … WHERE … • In � the � best � case � you � get � not � just � one, � but � all � three

• "Abstractions � are � great!"• "Abstractions � considered � harmful!"

Fault Tolerance

Out of OrderData

Developer/Operations Lifecycle

• How � should � your � daily � work � look � and � feel � like?• "I � like � to � do � quick, � iterative � development" � (modify/test/repeat)• "I � want � to � decouple � team � roadmaps, � project � schedules"

• Big � difference � between � App � Model � <-> � Cluster � Model• Testing, � packaging, � deployment, � monitoring, � operations• "Do � I � need � to � know � Java � (app) � or � YARN � (cluster) � for � this?”• "I � want � reactive � processing � in � containers � that � run � on � Mesos!"

• Rolling, � no-downtime � upgrades?• Integration � with � existing � Ops � infra, � tools, � processes?

Agenda

• Summary

Summary

• What � we � covered � is � a � good � starting � point• But, � no � free � lunch!• Understand � what � you � need, � and � weigh � criteria � appropriately• Think � end-to-end: � idea, � development, � operations, � troubleshooting• Think � big-picture: � future � use � cases, � architecture, � security, � training, � …• Do � your � own � internal � hackathons, � proof-of-concepts• Do � your � own � benchmarks

• If � in � doubt: � simplicity � beats � complexity• Faster � to � learn, � easier � to � understand, � less � likely � to � fail, � …

Q&A Session

Coming Up NextDate Title Speaker

Dec 15 Streaming in Practice: Putting Apache Kafka in Production

Roger Hoover

https://www.confluent.io/apache-‐kafka-‐talk-‐series

A Practical Guide to Selecting a Stream Processing Technology

Technology

Transcript of A Practical Guide to Selecting a Stream Processing Technology

Practical Guide to Selecting a Web CMS for State and Local Governments

END-TO-END LIVE STREAMING/media/25DFBBD8EE334...You can test that your stream works as expected by selecting your live stream within Wowza Streaming Cloud, clicking Start Stream at

The Incinerator Guidebook: A Practical Guide for Selecting ...

A Practical Guide to Selecting Data ... - helpIT systems · systems CLEANER DATA. BETTER DECISIONS. A Practical Guide to Selecting Data Quality Software. Table of Contents SECTION

Merino Sheep Breeding - Woolwise...MERINO SHEEP BREEDING TRAINER GUIDE PAGE 5 4.3.1 Practical Exercise 2 – Selecting a stud 56 Worksheet 2 – Practical Exercise 2 59 4.3.2 Practical

Heat Exchanger Design Guide: A Practical Guide for Planning, Selecting and Designing of Shell

Total Maximum Daily Loads of Carbonaceous Biochemical ... · The stream flow runoff rate is determined by selecting a representative reference stream gaging station near the study

The Practical Guide to Selecting a Web CMS for Media and Entertainment Firms

Value Stream Mapping Accounts Payable - Servicespracticalprocessimprovementct.com/uploads/Web_-Accounts_Payabl… · Value Stream Mapping Accounts Payable. 860-638-9874 Practical

Stream ciphers: A Practical Solution for E cient Homomorphic … · 2018. 3. 16. · Stream ciphers: A Practical Solution for E cient Homomorphic-Ciphertext Compression? Anne Canteaut1,

Starting with vsm and kanban; A practical workshop on value stream mapping & WIP

The Power of Both Choices: Practical Load Balancing for Distributed Stream Processing Engines

Cold-Shock Test Is a Practical Method for Selecting Boar ...

Selecting the Number of States in Hidden Markov Models ...koskinen/Teaching... · Selecting the Number of States in Hidden Markov Models — Pitfalls, Practical Challenges and Pragmatic

Practical Reasons for Algonkian Indian Stream and Place Names

Value Stream Mapping, Prioritizing Projects and Selecting ... · Mapping a Value Stream 9 Current State Follow a process path from end to beginning and draw a visual representation

Practical Algebraic Attacks on the HITAG2 TM Stream Cipher · 2015-07-28 · Practical Algebraic Attacks on the HITAG2 TM Stream Cipher Nicolas T. Courtois 1 Sean O ’Neil2 Jean-Jacques

Selecting An Electronic Health Record A Practical Guide.

A PRACTICAL GUIDE TO SELECTING THE RIGHT LIMS · Key Considerations When Selecting a LIMS –Technical Considerations / IT Considerations •Premise-based vs. Cloud-based (aka SaaS)

A Practical The Streamkeepers Handbook Guide To Stream And ... · The Stewardship Series The Streamkeepers Handbook A Practical Guide To Stream And Wetland Care. The Stewardship Series