INTERACTIVE SPARK IN YOUR BROWSER

Spark  Lightning  Night  -­‐  Hadoop  World

September  2015

GOALOF HUE

WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP  

SIMPLIFY AND INTEGRATEFREE AND OPEN SOURCE

—> “BIG NOTEBOOK” FOR HADOOP

SPARKARCHITECTURE

Livy

Spark

Spark

Spark

YARN

Backend  partWeb  part

SPARKARCHITECTURE

Livy

Spark

Spark

Spark

YARN

Spark  ServerUI

HADOOP NOTEBOOK

UI

•Married  with  full  ecosystem  • File,  Job  browsers  • Create  table  wizards  • Any  language  (Hive,  Spark...)  • Graphing  • Export/Import/Sharing  

•MulT  users  • ImpersonaTon  

HADOOP WITH SPARKNOTEBOOK

LIVY SPARK SERVER

Backend

• REST  Web  server  in  Scala  

• InteracTve  Spark  Sessions  and  Batch  Jobs  

• Type  IntrospecTon  for  VisualizaTon  

• Running  sessions  in  YARN    local  

• Backends:  Scala,  Python,  R  

• Open  Source:  h[ps://github.com/cloudera/hue/tree/master/apps/spark/java  

• Play  with  Curl  h[p://gethue.com/how-­‐to-­‐use-­‐the-­‐

LIVYSPARK SERVER

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

Livy  Server

Scalatra

Session  Manager

Session

LIVY WEB SERVERARCHITECTURE

Livy  Server

YARN  Master

Scalatra

Spark  Client

Session  Manager

Session

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1

2

Livy  Server

Scalatra

Session  Manager

Session

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1

2

3

Livy  Server

Scalatra

Session  Manager

Session

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1

2

3

4Livy  Server

Scalatra

Session  Manager

Session

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1

2

3

4

5

Livy  Server

Scalatra

Session  Manager

Session

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1

2

3

4

5

6Livy  Server

Scalatra

Session  Manager

Session

LIVY WEB SERVERARCHITECTURE

YARN  Master

Spark  Client

YARNNode

SparkInterpreter

SparkContext

YARNNode

SparkWorker

YARNNode

SparkWorker

1 7

2

3

4

5

6Livy  Server

Scalatra

Session  Manager

Session

SESSION CREATIONAND EXECUTION

%  curl  -­‐XPOST  localhost:8998/sessions  \      -­‐d  '{"kind":  "spark"}'  {      "id":  0,      "kind":  "spark",      "log":  [...],      "state":  "idle"  }  

%  curl  -­‐XPOST  localhost:8998/sessions/0/statements  -­‐d  '{"code":  "1+1"}'  {      "id":  0,      "output":  {          "data":  {  "text/plain":  "res0:  Int  =  2"  },          "execution_count":  0,          "status":  "ok"      },      "state":  "available"  }  

DEMO TIME

• Stability  and  Scaling  • Security  • iPython/Jupyter  interpreter  and  file  format  

COMING SOON

TWITTER

@gethue

USER GROUP

hue-­‐user@

WEBSITE

h[p://gethue.com

LEARN

h[p://learn.gethue.com

THANKS!