Interactive Apache Spark in Your Browser

Click here to load reader

download Interactive Apache Spark in Your Browser

of 22

  • date post

    17-Jan-2017
  • Category

    Software

  • view

    2.148
  • download

    2

Embed Size (px)

Transcript of Interactive Apache Spark in Your Browser

  • INTERACTIVE SPARK IN YOUR BROWSER

    Spark Lightning Night - Hadoop World

    September 2015

  • GOALOF HUE

    WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP

    SIMPLIFY AND INTEGRATEFREE AND OPEN SOURCE

    > BIG NOTEBOOK FOR HADOOP

  • SPARKARCHITECTURE

    Livy

    Spark

    Spark

    Spark

    YARN

    Backend partWeb part

  • SPARKARCHITECTURE

    Livy

    Spark

    Spark

    Spark

    YARN

    Spark ServerUI

  • HADOOP NOTEBOOK

    UI

  • Married with full ecosystem File, Job browsers Create table wizards Any language (Hive, Spark...) Graphing Export/Import/Sharing

    MulT users ImpersonaTon

    HADOOP WITH SPARKNOTEBOOK

  • LIVY SPARK SERVER

    Backend

  • REST Web server in Scala

    InteracTve Spark Sessions and Batch Jobs

    Type IntrospecTon for VisualizaTon

    Running sessions in YARN local

    Backends: Scala, Python, R

    Open Source: h[ps://github.com/cloudera/hue/tree/master/apps/spark/java

    Play with Curl h[p://gethue.com/how-to-use-the-

    LIVYSPARK SERVER

    https://github.com/cloudera/hue/tree/master/apps/spark/javahttp://gethue.com/how-to-use-the-livy-spark-rest-job-server-for-interactive-spark/

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    Livy Server

    Scalatra

    Session Manager

    Session

  • LIVY WEB SERVERARCHITECTURE

    Livy Server

    YARN Master

    Scalatra

    Spark Client

    Session Manager

    Session

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1

    2

    Livy Server

    Scalatra

    Session Manager

    Session

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1

    2

    3

    Livy Server

    Scalatra

    Session Manager

    Session

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1

    2

    3

    4Livy Server

    Scalatra

    Session Manager

    Session

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1

    2

    3

    4

    5

    Livy Server

    Scalatra

    Session Manager

    Session

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1

    2

    3

    4

    5

    6Livy Server

    Scalatra

    Session Manager

    Session

  • LIVY WEB SERVERARCHITECTURE

    YARN Master

    Spark Client

    YARNNode

    SparkInterpreter

    SparkContext

    YARNNode

    SparkWorker

    YARNNode

    SparkWorker

    1 7

    2

    3

    4

    5

    6Livy Server

    Scalatra

    Session Manager

    Session

  • SESSION CREATIONAND EXECUTION

    % curl -XPOST localhost:8998/sessions \ -d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" }

    % curl -XPOST localhost:8998/sessions/0/statements -d '{"code": "1+1"}' { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }

  • DEMO TIME

  • Stability and Scaling Security iPython/Jupyter interpreter and file format

    COMING SOON

  • TWITTER

    @gethue

    USER GROUP

    hue-user@

    WEBSITE

    h[p://gethue.com

    LEARN

    h[p://learn.gethue.com

    THANKS!

    http://twitter.com/gethuehttp://groups.google.com/a/cloudera.org/group/hue-userhttp://gethue.comhttp://learn.gethue.com