Apache Apex: Stream Processing Architecture and Applications

of 31/31
Apache Apex: Stream Processing Architecture and Applications Thomas Weise <[email protected]> @thweise Spark & Hadoop User Group Munich, July 19 th 2016
  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Apache Apex: Stream Processing Architecture and Applications

Apache Apex: Stream Processing Architecture and ApplicationsThomas Weise @thweiseSpark & Hadoop User Group Munich, July 19th 2016

Stream Data Processing


Data SourcesEventsLogsSensor DataSocialDatabasesCDC



Oper3Real-time analytics on data in motion

Data Visualization


Industries & Use Cases


Financial ServicesAd-TechTelecomManufacturingEnergyIoTFraud and risk monitoringReal-time customer facing dashboards on key performance indicatorsCall detail record (CDR) & extended data record (XDR) analysisSupply chain planning & optimizationSmart meter analyticsData ingestion and processingCredit risk assessmentClick fraud detectionUnderstanding customer behavior AND contextPreventive maintenanceReduce outages & improve resource utilizationPredictive analyticsImprove turn around time of trade settlement processesBilling optimizationPackaging and selling anonymous customer dataProduct quality & defect trackingAsset & workforce managementData governanceLarge scale ingest and distributionReal-time ELTA (Extract Load Transform Analyze) Dimensional computation & aggregationEnforcing data quality and data governance requirementsReal-time data enrichment with reference dataReal-time machine learning model scoring



Apache Apex4In-memory, distributed stream processingApplication logic broken into components (operators) that execute distributed in a clusterUnobtrusive Java API to express (custom) logicMaintain state and metrics in member variablesWindowing, event-time processingScalable, high throughput, low latencyOperators can be scaled up or down at runtime according to the load and SLADynamic scaling (elasticity), compute localityFault tolerance & correctnessAutomatically recover from node outages without having to reprocess from beginningState is preserved, checkpointing, incremental recoveryEnd-to-end exactly-onceOperabilitySystem and application metrics, record/visualize dataDynamic changes, elasticity


Native Hadoop Integration5

YARN is the resource managerHDFS for storing persistent state

Application Development Model6

A Stream is a sequence of data tuplesA typical Operator takes one or more input streams, performs computations & emits one or more output streamsEach Operator is YOUR custom business logic in java, or built-in operator from our open source libraryOperator has many instances that run in parallel and each instance is single-threadedDirected Acyclic Graph (DAG) is made up of operators and streamsDirected Acyclic Graph (DAG)








Output Stream

Filtered StreamEnrichedStreamFiltered StreamEnrichedStream

Application Specification (Java)7

Java Stream API (declarative)DAG API (compositional)


Java Streams API + Windowing8

Next Release (3.5): Support for Windowing la Apache Beam (incubating):

@ApplicationAnnotation(name = "WordCountStreamingApiDemo")public class ApplicationWithStreamAPI implements StreamingApplication{ @Override public void populateDAG(DAG dag, Configuration configuration) { String localFolder = "./src/test/resources/data"; ApexStream stream = StreamFactory .fromFolder(localFolder) .flatMap(new Split()) .window(new WindowOption.GlobalWindow(), new TriggerOption().withEarlyFiringsAtEvery(Duration.millis(1000)).accumulatingFiredPanes()) .countByKey(new ConvertToKeyVal()).print(); stream.populateDag(dag); }}


Writing an Operator9


Operator Library 10


Stateful Processing11

(All) : 5t=4:00 : 2t=5:00 : 3k=A, t=4:00 : 2k=A, t=5:00 : 1k=B, t=5:00 : 2(All) : 4t=4:00 : 2t=5:00 : 2k=A, t=4:00 : 2

K=B, t=5:00 : 2k=At=5:00(All) : 1t=4:00 : 1

k=A, t=4:00 : 1



State Checkpointing12Distributed, asynchronousPeriodic callbacksNo artificial latencyPluggable storage


Fault Tolerance13Operator state is checkpointed to persistent storeAutomatically performed by engine, no additional coding neededAsynchronous and distributed In case of failure operators are restarted from checkpoint stateAutomatic detection and recovery of failed containersHeartbeat mechanismYARN process status notificationBuffering to enable replay of data from recovered pointFast, incremental recovery, spike handlingApplication master state checkpointedSnapshot of physical (and logical) planExecution layer change log

In-memory PubSubStores results emitted by operator until committedHandles backpressure / spillover to local diskOrdering, idempotency

Operator 1

Container 1

BufferServerNode 1

Operator 2

Container 2Node 2Buffer Server


Recovery Scenario EW2, 1, 3, BW2, EW1, 4, 2, 1, BW1sum0 EW2, 1, 3, BW2, EW1, 4, 2, 1, BW1sum7 EW2, 1, 3, BW2, EW1, 4, 2, 1, BW1sum10 EW2, 1, 3, BW2, EW1, 4, 2, 1, BW1sum715

Processing Guarantees16At-least-onceOn recovery data will be replayed from a previous checkpointNo messages lostDefault, suitable for most applicationsCan be used to ensure data is written once to storeTransactions with meta information, Rewinding output, Feedback from external entity, Idempotent operationsAt-most-onceOn recovery the latest data is made available to operatorUseful in use cases where some data loss is acceptable and latest data is sufficientExactly-onceAt-least-once + idempotency + transactional mechanisms (operator logic) to achieve end-to-end exactly once behavior

End-to-End Exactly Once17Important when writing to external systemsData should not be duplicated or lost in the external system in case of application failuresCommon external systemsDatabasesFilesMessage queuesExactly-once = at-least-once + idempotency + consistent stateData duplication must be avoided when data is replayed from checkpointOperators implement the logic dependent on the external systemPlatform provides checkpointing and repeatable windowing

Scalability18NxM PartitionsUnifier0123Logical DAG01211Unifier120Logical DiagramPhysical Diagram with operator 1 with 3 partitions0Unifier1a1b1c2a2bUnifier3Physical DAG with (1a, 1b, 1c) and (2a, 2b): No bottleneckUnifierUnifier01a1b1c2a2bUnifier3Physical DAG with (1a, 1b, 1c) and (2a, 2b): Bottleneck on intermediate Unifier


Advanced Partitioning1901a1b234UnifierPhysical DAG043a2a1a1b2b3bUnifierPhysical DAG with Parallel PartitionParallel PartitionContaineruopruopr1uopr2uopr3uopr4uopr1uopr2uopr3uopr4doprdoprdoprunifierunifierunifierunifierContainerContainerNICNICNICNICNICContainerNICLogical PlanExecution Plan, for N = 4; M = 1Execution Plan, for N = 4; M = 1, K = 2 with cascading unifiersCascading Unifiers01234Logical DAG

Dynamic Partitioning20Partitioning change while application is runningChange number of partitions at runtime based on statsDetermine initial number of partitions dynamicallyKafka operators scale according to number of kafka partitionsSupports re-distribution of state when number of partitions changeAPI for custom scaler or partitioner


Unifiers not shown

How dynamic partitioning works21Partitioning decision (yes/no) by trigger (StatsListener)Pluggable component, can use any system or custom metricExternally driven partitioning example: KafkaInputOperatorStateful! Uses checkpointed stateAbility to transfer state from old to new partitions (partitioner, customizable)Steps:Call partitionerModify physical plan, rewrite checkpoints as neededUndeploy old partitions from execution layerRelease/request container resourcesDeploy new partitions (from rewritten checkpoint)No loss of data (buffered)Incremental operation, partitions that dont change continue processingAPI: Partitioner interface

Compute Locality22By default operators are deployed in containers (processes) on different nodes across the Hadoop clusterLocality options for streamsRACK_LOCAL: Data does not traverse network switchesNODE_LOCAL: Data transfer via loopback interface, frees up network bandwidthCONTAINER_LOCAL: Data transfer via in memory queues between operators, does not require serializationTHREAD_LOCAL: Data passed through call stack, operators share threadHost LocalityOperators can be deployed on specific hostsNew in 3.4.0: (Anti-)Affinity (APEXCORE-10)Ability to express relative deployment without specifying a host

Enterprise Tools23

Monitoring ConsoleLogical View24

Physical View


Real-Time Dashboards25


Maximize Revenue w/ real-time insights26

PubMatic is the leading marketing automation software company for publishers.Through real-time analytics, yield management, and workflow automation, PubMatic enables publishers to make smarter inventory decisions and improve revenue performance

Business NeedApex based SolutionClient OutcomeIngest and analyze high volume clicks & views in real-time to help customers improve revenue200K events/second data flowReport critical metrics for campaign monetization from auction and client logs 22 TB/day data generatedHandle ever increasing traffic with efficient resource utilizationAlways-on ad networkDataTorrent Enterprise platform, powered by Apache ApexIn-memory stream processingComprehensive library of pre-built operators including connectorsBuilt-in fault toleranceDynamically scalableManagement UI & Data Visualization console

Helps PubMatic deliver ad performance insights to publishers and advertisers in real-time instead of 5+ hoursHelps Publishers visualize campaign performance and adjust ad inventory in real-time to maximize their revenueEnables PubMatic reduce OPEX with efficient compute resource utilizationBuilt-in fault tolerance ensures customers can always access ad network


Industrial IoT applications27

GE is dedicated to providing advanced IoT analytics solutions to thousands of customers who are using their devices and sensors across different verticals. GE has built a sophisticated analytics platform, Predix, to help its customers develop and execute Industrial IoT applications and gain real-time insights as well as actions.Business NeedApex based SolutionClient OutcomeIngest and analyze high-volume, high speed data from thousands of devices, sensors per customer in real-time without data loss Predictive analytics to reduce costly maintenance and improve customer serviceUnified monitoring of all connected sensors and devices to minimize disruptionsFast application development cycleHigh scalability to meet changing business and application workloadsIngestion application using DataTorrent Enterprise platformPowered by Apache ApexIn-memory stream processingBuilt-in fault toleranceDynamic scalabilityComprehensive library of pre-built operatorsManagement UI console

Helps GE improve performance and lower cost by enabling real-time Big Data analytics Helps GE detect possible failures and minimize unplanned downtimes with centralized management & monitoring of devices Enables faster innovation with short application development cycleNo data loss and 24x7 availability of applicationsHelps GE adjust to scalability needs with auto-scaling


Smart energy applications28

Silver Spring Networks helps global utilities and cities connect, optimize, and manage smart energy and smart city infrastructure. Silver Spring Networks receives data from over 22 million connected devices, conducts 2 million remote operations per year

Business NeedApex based SolutionClient OutcomeIngest high-volume, high speed data from millions of devices & sensors in real-time without data loss Make data accessible to applications without delay to improve customer service Capture & analyze historical data to understand & improve grid operationsReduce the cost, time, and pain of integrating with 3rd party apps Centralized management of software & operations

DataTorrent Enterprise platform, powered by Apache ApexIn-memory stream processingPre-built operator Built-in fault toleranceDynamically scalableManagement UI console

Helps Silver Spring Networks ingest & analyze data in real-time for effective load management & customer service Helps Silver Spring Networks detect possible failures and reduce outages with centralized management & monitoring of devices Enables fast application development for faster time to marketHelps Silver Spring Networks scale with easy to partition operatorsAutomatic recovery from failures


More about the use cases29Pubmatichttps://www.youtube.com/watch?v=JSXpgfQFcU8


SilverSpring Networkshttps://www.youtube.com/watch?v=8VORISKeSjIhttp://www.slideshare.net/ApacheApex/iot-big-data-ingestion-and-processing-in-hadoop-by-silver-spring-networks


Resources31http://apex.apache.org/Learn more: http://apex.apache.org/docs.html Subscribe - http://apex.apache.org/community.htmlDownload - http://apex.apache.org/downloads.htmlFollow @ApacheApex - https://twitter.com/apacheapexMeetups http://www.meetup.com/pro/apacheapex/More examples: https://github.com/DataTorrent/examplesSlideshare: http://www.slideshare.net/ApacheApex/presentationshttps://www.youtube.com/results?search_query=apache+apexFree Enterprise License for Startups - https://www.datatorrent.com/product/startup-accelerator/