Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
Apache Flink: Streaming Done Right @ FOSDEM 2016
-
Upload
till-rohrmann -
Category
Technology
-
view
1.207 -
download
0
Transcript of Apache Flink: Streaming Done Right @ FOSDEM 2016
What Is Apache Flink? § Apache TLP since December
2014
§ Parallel streaming data flow runtime
§ Low latency & high throughput § Exactly once semantics § Stateful operations
2
Why Stream Processing? § Most problems have streaming nature § Stream processing gives lower latency § Data volumes more easily tamed § More predictable resource consumption
3
Eventstream
How Would I Do It with Flink?
5
caseclassTweet(id:Long,timestamp:Long,count:Long)valenv=StreamExecutionEnvironment.getEnvironment()valtweets:DataStream[Tweet]=env.addSource(newMyTwitterSource())valresult:DataStream[Tweet]=tweets.keyBy("id").timeWindow(Time.minutes(10)).sum("count")result.print()env.execute()
Wehavetodefinea6meframeforthe
aggrega6on
Stateful Operators § What if a window grows too large? § Solution: Stateful mapper with counter
7
Count tweet impressions
#42
Operator
User function
Operator state
#13 #37
Read Write
Intuitive API
8
caseclassTweet(id:Long,timestamp:Long,count:Long)valenv=StreamExecutionEnvironment.getEnvironment()valtweets:DataStream[Tweet]=env.addSource(newMyTwitterSource())valresult:DataStream[Tweet]=tweets.keyBy("id").mapWithState{(tweet,state:Option[Long])=>statematch{caseSome(counter)=>(tweet.copy(count=counter+1L),Some(counter+1L))caseNone=>(tweet,Some(1L))}}
What If a Failure Occurs?
Loss of data and incorrect count! 9
Count tweet impressions
#42
Operator
User function
Operator state
#?? #??
Read Write
Processing Guarantees § At most once • No guarantees at all
§ At least once • Ensure that all operators
see all events § Exactly once
12
Flink gives you all guarantees
Checkpointed Operators § State is automatically checkpointed § State backend is configurable
13
Barriers
Count tweet impressions
#42
Operator
User function
Operator state
#13 #37
Read Write
What About Performance?
14
Continuous streaming
Latency-bound buffering
Distributed Snapshots
High Throughput & Low Latency
With configurable throughput/latency tradeoff
Asynchronous Snapshots § Taking snapshots stalls the operator § Solution: Out of core & asynchronous snapshots
16 Async/incremental snapshots
Spill to disk
Barriers
Count tweet impressions
#42
User function
Operator state
#13 #37
Read Write
Streams with Varying Data Rate
17 time
even
ts/s
econ
d
With static resources: Provision for max. rate
Idle capacity
(2) Dynamic Worker Pool
19
JobManager
Resource Manager
Pool of Cluster Resources YARN/Mesos/…
Allocate
TaskManager
TaskManager Release
Declarative Queries § StreamSQL
§ Complex event processing
20
valtabEnv=newTableEnvironment(env)tabEnv.registerStream(stream,“myStream”,(“ID”,“MEASURE”,“COUNT”))valsqlQuery=tabEnv.sql(“SELECTID,MEASUREFROMmyStreamWHERECOUNT>17”)
Sober Drunk
Car
Afoot drink
drink
enters car
walks
Raisealert