Streaming Analytics with Spark, Kafka, Cassandra and Akka

download Streaming Analytics with Spark, Kafka, Cassandra and Akka

of 84

  • date post

    15-Apr-2017
  • Category

    Technology

  • view

    33.234
  • download

    2

Embed Size (px)

Transcript of Streaming Analytics with Spark, Kafka, Cassandra and Akka

  • Streaming Analytics with Spark, Kafka, Cassandra, and Akka

    Helena Edelson VP of Product Engineering @Tuplejump

    https://twitter.com/tuplejump

  • Committer / Contributor: Akka, FiloDB, Spark Cassandra Connector, Spring Integration

    VP of Product Engineering @Tuplejump

    Previously: Sr Cloud Engineer / Architect at VMware, CrowdStrike, DataStax and SpringSource

    Who@helenaedelsongithub.com/helena

    https://twitter.com/tuplejumphttps://twitter.com/helenaedelsonhttp://github.com/helena

  • TuplejumpTuplejump Data Blender combines sophisticated data collection with machine learning and analytics, to understand the intention of the analyst, without disrupting workflow.

    Ingest streaming and static data from disparate data sources

    Combine them into a unified, holistic view

    Easily enable fast, flexible and advanced data analysis

    3

  • Tuplejump Open Source github.com/tuplejump

    FiloDB - distributed, versioned, columnar analytical db for modern streaming workloads

    Calliope - the first Spark-Cassandra integration Stargate - Lucene indexer for Cassandra SnackFS - HDFS-compatible file system for Cassandra

    4

    http://github.com/tuplejump

  • What Will We Talk About The Problem Domain Example Use Case Rethinking Architecture

    We don't have to look far to look back Streaming Revisiting the goal and the stack Simplification

  • THE PROBLEM DOMAINDelivering Meaning From A Flood Of Data

    6

  • The Problem DomainNeed to build scalable, fault tolerant, distributed data processing systems that can handle massive amounts of data from disparate sources, with different data structures.

    7

  • TranslationHow to build adaptable, elegant systems for complex analytics and learning tasks to run as large-scale clustered dataflows

    8

  • How Much Data

    Yottabyte = quadrillion gigabytes or septillion bytes

    9

    We all have a lot of data Terabytes Petabytes...

    https://en.wikipedia.org/wiki/Yottabyte

  • Delivering Meaning Deliver meaning in sec/sub-sec latency Disparate data sources & schemas Billions of events per second High-latency batch processing Low-latency stream processing Aggregation of historical from the stream

  • While We Monitor, Predict & Proactively Handle

    Massive event spikes Bursty traffic Fast producers / slow consumers Network partitioning & Out of sync systems DC down Wait, we've DDOS'd ourselves from fast streams? Autoscale issues

    When we scale down VMs how do we not lose data?

  • And stay within our AWS / Rackspace budget

  • EXAMPLE CASE: CYBER SECURITY

    Hunting The Hunter

    13

  • 14

    Track activities of international threat actor groups, nation-state, criminal or hactivist Intrusion attempts Actual breaches

    Profile adversary activity Analysis to understand their motives, anticipate actions

    and prevent damage

    Adversary Profiling & Hunting

  • 15

    Machine events Endpoint intrusion detection Anomalies/indicators of attack or compromise

    Machine learning Training models based on patterns from historical data Predict potential threats profiling for adversary Identification

    Stream Processing

  • Data Requirements & Description Streaming event data

    Log messages User activity records System ops & metrics data

    Disparate data sources Wildly differing data structures

    16

  • Massive Amounts Of Data

    17

    One machine can generate 2+ TB per day Tracking millions of devices 1 million writes per second - bursty High % writes, lower % reads TTL

  • RETHINKING ARCHITECTURE

    18

  • WE DON'T HAVE TO LOOK FAR TO LOOK BACK

    19

    Rethinking Architecture

  • 20

    Most batch analytics flow from several years ago looked like...

  • STREAMING & DATA SCIENCE

    21

    Rethinking Architecture

  • StreamingI need fast access to historical data on the fly for predictive modeling with real time data from the stream.

    22

  • Not A Stream, A Flood Data emitters

    Netflix: 1 - 2 million events per second at peak 750 billion events per day

    LinkedIn: > 500 billion events per day Data ingesters

    Netflix: 50 - 100 billion events per day LinkedIn: 2.5 trillion events per day

    1 Petabyte of streaming data23

  • Which Translates To Do it fast Do it cheap Do it at scale

    24

  • Challenges Code changes at runtime Distributed Data Consistency Ordering guarantees Complex compute algorithms

    25

  • Oh, and don't lose data

    26

  • Strategies Partition For Scale & Data Locality Replicate For Resiliency Share Nothing Fault Tolerance Asynchrony Async Message Passing Memory Management

    27

    Data lineage and reprocessing in runtime Parallelism Elastically Scale Isolation Location Transparency

  • AND THEN WE GREEKED OUT

    28

    Rethinking Architecture

  • Lambda ArchitectureA data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods.

    29

  • Lambda ArchitectureA data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods.

    An approach Coined by Nathan Marz This was a huge stride forward

    30

  • 31https://www.mapr.com/developercentral/lambda-architecture

    https://www.mapr.com/developercentral/lambda-architecture

  • Implementing Is Hard

    33

    Real-time pipeline backed by KV store for updates Many moving parts - KV store, real time, batch Running similar code in two places Still ingesting data to Parquet/HDFS Reconcile queries against two different places

  • Performance Tuning & Monitoring: on so many systems

    34

    Also hard

  • Lambda ArchitectureAn immutable sequence of records is captured and fed into a batch system and a stream processing system in parallel.

    35

  • WAIT, DUAL SYSTEMS?

    36

    Challenge Assumptions

  • Which Translates To Performing analytical computations & queries in dual

    systems Implementing transformation logic twice Duplicate Code Spaghetti Architecture for Data Flows One Busy Network

    37

  • Why Dual Systems? Why is a separate batch system needed? Why support code, machines and running services of

    two analytics systems?

    38

    Counter productive on some level?

  • YES

    39

    A unified system for streaming and batch Real-time processing and reprocessing

    Code changes Fault tolerance

    http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html - Jay Kreps

    http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html

  • ANOTHER ASSUMPTION: ETL

    40

    Challenge Assumptions

  • Extract, Transform, Load (ETL)

    41

    "Designing and maintaining the ETL process is often considered one of the most difficult and resource-intensive portions of a data warehouse project."

    http://docs.oracle.com/cd/B19306_01/server.102/b14223/ettover.htm

    http://docs.oracle.com/cd/B19306_01/server.102/b14223/ettover.htm

  • Extract, Transform, Load (ETL)

    42

    ETL involves Extraction of data from one system into another Transforming it Loading it into another system

  • Extract, Transform, Load (ETL)"Designing and maintaining the ETL process is often

    considered one of the most difficult and resource-intensive portions of a data warehouse project."

    http://docs.oracle.com/cd/B19306_01/server.102/b14223/ettover.htm

    43

    Also unnecessarily redundant and often typeless

    http://docs.oracle.com/cd/B19306_01/server.102/b14223/ettover.htm

  • ETL

    44

    Each ETL step can introduce errors and risk Can duplicate data after failover Tools can cost millions of dollars Decreases throughput Increased complexity

  • ETL

    Writing intermediary files Parsing and re-parsing plain text

    45

  • And let's duplicate the pattern over all our DataCenters

    46

  • 47

    These are not the solutions you're looking for

  • REVISITING THE GOAL & THE STACK

    48

  • Removing The 'E' in ETLThanks to technologies like Avro and Protobuf we dont need the E in ETL. Instead of text dumps that you need to parse over multiple systems:

    Scala & Avro (e.g.)

    Can work with binary data that remains strongly typed

    A return to strong typing in the big data ecosystem

    49

  • Removing The 'L' in ETLIf data collection is backed by a distributed messaging system (e.g. Kafka) you can do real-time fanout of the ingested data to all consumers. No need to batch "load".

    From there each consumer can do their own transformations

    50

  • #NoMoreGreekLetterArchitectures

    51

  • NoETL

    52

  • Strategy Technologies

    Scalable Infrastructure / Elastic Spark, Cassandra, Kafka

    Partition For Scale, Network Topology Aware Cassandra, Spark, Kafka, Akka Cluster

    Replicate For Resiliency Spark,Cassandra, Akka Cluster all hash the node ring

    Share Nothing, Masterless Cassandra, Akka Cluster both Dynamo style

    Fault Tolerance / No Single Point of Failure Spark, Cassandra, Kafka

    Replay From Any Point Of Failure Spark, Cassandra, Kafka, Akka + Akka P