Flink4 jug

Big Data series : Apache Flink
Jrme BlachonLaurent TardifStphane Thiers
Juin 2015 : Jug Grenoble Septembre 2015 : Jug Lausanne

Qui sommes nous
Jrme Blachon Laurent Tardif Stphane Thiers

Un peu dhistoireLa stackFlink
Demo Comment ca marche Les plus
Roadmap
La soire

Histoire

BigData success story
Map / ReduceOSDI 04
Map / ReduceOSDI 04
Hadoop1
Dryad EuroSys
07
Dryad EuroSys
07 TEZ
RDDs HotCloud10,
NSDI12
RDDs HotCloud10,
NSDI12Spark
PACTsSOCC10, VLDB12
PACTsSOCC10, VLDB12 Flink
Map/Reduce extended to DAGBacktracking recovery
Map/Reduce extended to DAGBacktracking recovery
Small recoverable tasksSequencial code
Small recoverable tasksSequencial code
Functional implementation of Dryad
recovery
Functional implementation of Dryad
recovery
Cyclic Graph (and incremental construction)Query Processing runtime embed in DAG
engine
Cyclic Graph (and incremental construction)Query Processing runtime embed in DAG
engine
Stonebraker/ Cetintemel /
Zdonik2005
Stonebraker/ Cetintemel /
Zdonik2005

Keep data moving Low latency on critical path
Query on stream High level language
Handle stream imperfection Timeout (ex: avg of last 25 securities) Out of order (must leave window open)
Generate predictable outcomes Time ordered
Criteria for stream processing (1/2)

Integrate stored / streaming data Uniform language for both stored and streamed data Combine streamed and stored data
Data safety / availability Resistant to failure
Partition and scale automatically Process and respond instantaneously
100 000 msg / s
Criteria for stream processing (2/2)

Big data stack

The stack
Data Processing engine Data Processing engine
User requirementUser requirement
App and ressource managementApp and ressource management
Storage / streamStorage / stream

Eco system
Applications
Data processing engines
App and resourcemanagement
Storage/Stream

Une autre vue
http://practicalanalytics.wordpress.com

Word count
The hello world
// read test file or in Memory, and generate a set of StringDataSet text = getTextDataSet(env);DataSet counts = // split up the lines in pairs (2-tuples) containing: (word,1)text.flatMap(new Tokenizer())
// group by the tuple field "0" and sum up tuple field "1.groupBy(0).sum(1);

Word countTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",
To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",
To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",
"The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",
(to,1)(to,1)
(be,1)(be,1)
(or,1)(or,1)
(to,1)(to,1)(to,1)(to,1)
(be,1)(be,1)(be,1)(be,1)
(or,1)(or,1)
(to,2)(to,2)
(be,2)(be,2)
(or,1)(or,1)
Flatmap(tojenizer)
groupby
sum

Data in memory
public static final String[] WORDS = new String[] {"To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune","Or to take arms against a sea of troubles,","And by opposing end them?--To die,--to sleep,--","No more; and by a sleep to say we end","The heartache, and the thousand natural shocks","That flesh is heir to,--'tis a consummation","Devoutly to be wish'd. To die,--to sleep;--",.

File
private static DataSet getTextDataSet(ExecutionEnvironment env) {return env.readTextFile(textPath);}

With POJO
public static class Word {// fieldsprivate String word;private Integer frequency;// constructorspublic Word() { }public Word(String word, int i) {
this.word = word;this.frequency = i; }
// getters setters// to String@Overridepublic String toString() {
return "Word="+word+" freq="+frequency; }

PojoTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",
Word 1 {to,1}Word 1 {to,1}
Word 2 {be,1}Word 2 {be,1}
Word 3 {or,1}Word 3 {or,1}
Word 1 {to,1}Word 5 {to,1}Word 1 {to,1}Word 5 {to,1}
Word 2 {be,2}Word 6 {be,1}Word 2 {be,2}Word 6 {be,1}
Word 3 {be,1}Word 3 {be,1}
Word7 {to,2}Word7 {to,2}
Word8 {be,2}Word8 {be,2}
Word9 {or,1}Word9 {or,1}
Flatmap(tokenizer)
groupby
sum

JDBC
(To be, or not to be,--that is the question:--")(To be, or not to be,--that is the question:--")
("Whether 'tis nobler in the mind to suffer")("Whether 'tis nobler in the mind to suffer")
(to,1)(to,1)
(be,1)(be,1)
(or,1)(or,1)
(to,1)(to,1)(to,1)(to,1)
(be,1)(be,1)(be,1)(be,1)
(or,1)(or,1)
(to,2)(to,2)
(be,2)(be,2)
(or,1)(or,1)
Map +Flatmap(tokenizer)
groupby
sum
hamlet
To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",

StreamTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",
(to,1)(to,1)
(be,1)(be,1)
(or,1)(or,1)
(to,1)(to,1)(to,1)(to,1)
(be,1)(be,1)(be,1)(be,1)
(or,1)(or,1)
(to,2)(to,2)
(be,2)(be,2)
Flatmap(tokenizer)
groupby
sum
(or,1)(or,1)

(to,1)(to,1)
(be,1)(be,1)
(or,1)(or,1)
(to,1)(to,1)(to,1)(to,1)
(be,1)(be,1)(be,1)(be,1)
(or,1)(or,1)
(to,2)(to,2)
(be,2)(be,2)
Flatmap(tokenizer)
groupby
sum
"Or to take arms against a sea of troubles,","Or to take arms against a sea of troubles,",
(or,1)(or,1)

(to,1)(to,1)
(be,1)(be,1)
(or,1)(or,1)
(to,1)(to,1)(to,1)(to,1)
(be,1)(be,1)(be,1)(be,1)
(or,1)
(to,2)(to,2)
(be,2)(be,2)
Flatmap(tokenizer)
groupby
sum
(or,1)(or,1)

(to,1)(to,1)
(be,1)(be,1)
(or,1)(or,1)
(to,1)(to,1)(to,1)(to,1)
(be,1)(be,1)(be,1)(be,1)
(or,1)(or,1)
(to,2)(to,2)
(be,2)(be,2)
Flatmap(tokenizer)
groupby
sum
"Or to take arms against a sea of troubles,","Or to take arms against a sea of trouble

Flink4 jug

Engineering

Transcript of Flink4 jug

Terracotta JUG Milano

Java EE 7 (Lyon JUG & Alpes JUG - March 2014)

Sjever i Jug

Jug Intro 10

Jug Jug Maro (Kavya Comics) - Freelance Talents

JUG â€“ Ancona Italy BIRT Business Intelligence Reporting Tools Francesco Paladino Jug coordinator Jug Ancona Meeting del 18/04/2007

Jug Zurich Slides

Flash Catalyst Jug

Dart JUG 2013

The Closed Jug System - Micro Matic ... The Closed Jug System The quick & easy Closed Jug System is designed to extract liquid pesticides from a jug into a transfer/mixing system and

JUG Intro 16

Reunion JUG

Project Avatar (Lyon JUG & Alpes JUG - March 2014)

Terracotta jug

Jug Fishing Tips

3 Gallon Jug

Mockito Chti JUG

Jclouds jug vale

Israeli JUG - IL JUG presentation

Watering Jug