Flink4 jug

Click here to load reader

  • date post

    20-Mar-2017
  • Category

    Engineering

  • view

    597
  • download

    0

Embed Size (px)

Transcript of Flink4 jug

  • Big Data series : Apache Flink

    Jrme BlachonLaurent TardifStphane Thiers

    Juin 2015 : Jug Grenoble Septembre 2015 : Jug Lausanne

  • Qui sommes nous

    Jrme Blachon Laurent Tardif Stphane Thiers

  • Un peu dhistoireLa stackFlink

    Demo Comment ca marche Les plus

    Roadmap

    La soire

  • Histoire

  • BigData success story

    Map / ReduceOSDI 04

    Map / ReduceOSDI 04

    Hadoop1

    Dryad EuroSys

    07

    Dryad EuroSys

    07 TEZ

    RDDs HotCloud10,

    NSDI12

    RDDs HotCloud10,

    NSDI12Spark

    PACTsSOCC10, VLDB12

    PACTsSOCC10, VLDB12 Flink

    Map/Reduce extended to DAGBacktracking recovery

    Map/Reduce extended to DAGBacktracking recovery

    Small recoverable tasksSequencial code

    Small recoverable tasksSequencial code

    Functional implementation of Dryad

    recovery

    Functional implementation of Dryad

    recovery

    Cyclic Graph (and incremental construction)Query Processing runtime embed in DAG

    engine

    Cyclic Graph (and incremental construction)Query Processing runtime embed in DAG

    engine

    Stonebraker/ Cetintemel /

    Zdonik2005

    Stonebraker/ Cetintemel /

    Zdonik2005

  • Keep data moving Low latency on critical path

    Query on stream High level language

    Handle stream imperfection Timeout (ex: avg of last 25 securities) Out of order (must leave window open)

    Generate predictable outcomes Time ordered

    Criteria for stream processing (1/2)

  • Integrate stored / streaming data Uniform language for both stored and streamed data Combine streamed and stored data

    Data safety / availability Resistant to failure

    Partition and scale automatically Process and respond instantaneously

    100 000 msg / s

    Criteria for stream processing (2/2)

  • Big data stack

  • The stack

    Data Processing engine Data Processing engine

    User requirementUser requirement

    App and ressource managementApp and ressource management

    Storage / streamStorage / stream

  • Eco system

    Applications

    Data processing engines

    App and resourcemanagement

    Storage/Stream

  • Une autre vue

    http://practicalanalytics.wordpress.com

  • Demo

  • Word count

    The hello world

    // read test file or in Memory, and generate a set of StringDataSet text = getTextDataSet(env);DataSet counts = // split up the lines in pairs (2-tuples) containing: (word,1)text.flatMap(new Tokenizer())

    // group by the tuple field "0" and sum up tuple field "1.groupBy(0).sum(1);

  • Word countTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",

    "The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",

    (to,1)(to,1)

    (be,1)(be,1)

    (or,1)(or,1)

    (to,1)(to,1)(to,1)(to,1)

    (be,1)(be,1)(be,1)(be,1)

    (or,1)(or,1)

    (to,2)(to,2)

    (be,2)(be,2)

    (or,1)(or,1)

    Flatmap(tojenizer)

    groupby

    sum

  • Data in memory

    public static final String[] WORDS = new String[] {"To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune","Or to take arms against a sea of troubles,","And by opposing end them?--To die,--to sleep,--","No more; and by a sleep to say we end","The heartache, and the thousand natural shocks","That flesh is heir to,--'tis a consummation","Devoutly to be wish'd. To die,--to sleep;--",.

  • File

    private static DataSet getTextDataSet(ExecutionEnvironment env) {return env.readTextFile(textPath);}

  • With POJO

    public static class Word {// fieldsprivate String word;private Integer frequency;// constructorspublic Word() { }public Word(String word, int i) {

    this.word = word;this.frequency = i; }

    // getters setters// to String@Overridepublic String toString() {

    return "Word="+word+" freq="+frequency; }

  • PojoTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",

    "The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",

    Word 1 {to,1}Word 1 {to,1}

    Word 2 {be,1}Word 2 {be,1}

    Word 3 {or,1}Word 3 {or,1}

    Word 1 {to,1}Word 5 {to,1}Word 1 {to,1}Word 5 {to,1}

    Word 2 {be,2}Word 6 {be,1}Word 2 {be,2}Word 6 {be,1}

    Word 3 {be,1}Word 3 {be,1}

    Word7 {to,2}Word7 {to,2}

    Word8 {be,2}Word8 {be,2}

    Word9 {or,1}Word9 {or,1}

    Flatmap(tokenizer)

    groupby

    sum

  • JDBC

    (To be, or not to be,--that is the question:--")(To be, or not to be,--that is the question:--")

    ("Whether 'tis nobler in the mind to suffer")("Whether 'tis nobler in the mind to suffer")

    (to,1)(to,1)

    (be,1)(be,1)

    (or,1)(or,1)

    (to,1)(to,1)(to,1)(to,1)

    (be,1)(be,1)(be,1)(be,1)

    (or,1)(or,1)

    (to,2)(to,2)

    (be,2)(be,2)

    (or,1)(or,1)

    Map +Flatmap(tokenizer)

    groupby

    sum

    hamlet

    To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer",

  • StreamTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",

    "The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",

    (to,1)(to,1)

    (be,1)(be,1)

    (or,1)(or,1)

    (to,1)(to,1)(to,1)(to,1)

    (be,1)(be,1)(be,1)(be,1)

    (or,1)(or,1)

    (to,2)(to,2)

    (be,2)(be,2)

    Flatmap(tokenizer)

    groupby

    sum

    (or,1)(or,1)

  • StreamTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",

    "The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",

    (to,1)(to,1)

    (be,1)(be,1)

    (or,1)(or,1)

    (to,1)(to,1)(to,1)(to,1)

    (be,1)(be,1)(be,1)(be,1)

    (or,1)(or,1)

    (to,2)(to,2)

    (be,2)(be,2)

    Flatmap(tokenizer)

    groupby

    sum

    "Or to take arms against a sea of troubles,","Or to take arms against a sea of troubles,",

    (or,1)(or,1)

  • StreamTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",

    "The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",

    (to,1)(to,1)

    (be,1)(be,1)

    (or,1)(or,1)

    (to,1)(to,1)(to,1)(to,1)

    (be,1)(be,1)(be,1)(be,1)

    (or,1)

    (to,2)(to,2)

    (be,2)(be,2)

    Flatmap(tokenizer)

    groupby

    sum

    "Or to take arms against a sea of troubles,","Or to take arms against a sea of troubles,",

    "Or to take arms against a sea of troubles,","Or to take arms against a sea of troubles,",

    (or,1)(or,1)

  • StreamTo be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--","Whether 'tis nobler in the mind to suffer","The slings and arrows of outrageous fortune",

    To be, or not to be,--that is the question:--",To be, or not to be,--that is the question:--",

    "Whether 'tis nobler in the mind to suffer","Whether 'tis nobler in the mind to suffer",

    "The slings and arrows of outrageous fortune","The slings and arrows of outrageous fortune",

    (to,1)(to,1)

    (be,1)(be,1)

    (or,1)(or,1)

    (to,1)(to,1)(to,1)(to,1)

    (be,1)(be,1)(be,1)(be,1)

    (or,1)(or,1)

    (to,2)(to,2)

    (be,2)(be,2)

    Flatmap(tokenizer)

    groupby

    sum

    "Or to take arms against a sea of troubles,","Or to take arms against a sea of troubles,",

    "Or to take arms against a sea of troubles,","Or to take arms against a sea of trouble