Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

Spark Summit 2016 recap

クリエーションライン ( 株 )木内

2016/7/26 Spark Summit2016 報告会＆データ分析勉強会 講演資料

1

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

自己紹介 木内 満歳（きうち みつとし）

クリエーションライン株式会社 シニアコンサルタント

Slideshare: http://www.slideshare.net/mkiuchi4

各種寄稿a. gihyo.jp: “ Mesosphere DCOSでつくるクラウドアプリケーション”

b. 日経クラウドファースト 2016 年 6 “月 Azure IoT Suiteの評価”

c. Codezine: “ Bluemix Apache Spark機械学習をクラウドで手軽に体験！ の で異常なセ

ンサーデータを洗い出す”

専門分野： Apache Mesos, Apache Spark, 分散コンピューティング , クラウドコンピューティング , NoSQL DB, グラフ DB

O’reilley Certified Developer on Apache Spark

Docker Certified Technical Trainer2

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

Spark 2.0 is (still) comming...

• Easier– SparkSQL SQL:2003 compliant– DataFrame– Machine learning pipeline persistence

• Faster– 2nd-gen Tungsten Engine

Vitual function call -> “Whole-stage code generation” (=JIT bytecode generation)

•Smarter– Structured Streaming

3

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

Breakout sessions よりいくつかピックアップ ...

DATA SCIENCE18

DEVELOPER12

RESEARCH18

ECOSYSTEM18

USECASE15

ENTERPRISE9

4

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

DATA SCIENCE

http://www.slideshare.net/SparkSummit/graphframes-graph-queries-in-spark-sql

5

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

DEVELOPER

http://www.slideshare.net/JenAman/highperformance-python-on-spark

6

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

RESEARCH

http://www.slideshare.net/SparkSummit/gpu-support-in-spark-and-gpucpu-mixed-resource-scheduling-at-production-scale-63065895

7

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

ECOSYSTEM

http://www.slideshare.net/JenAman/efficient-state-management-with-spark-20-and-scaleout-databases http://www.slideshare.net/JenAman/elasticsearch-and-apache-lucene-for-apache-spark-and-mllib

http://www.slideshare.net/JenAman/gpu-computing-with-apache-spark-and-python-63064880

http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark http://www.slideshare.net/JenAman/building-realtime-data-pipelines-with-kafka-connect-and-spark-streaming

http://www.slideshare.net/JenAman/netflix-productionizing-spark-on-yarn-for-etl-at-petabyte-scale

USECASE ENTERPRISE

8

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ

まとめ

• Spark2.0 以降も継続的な進歩が行われる

– 性能 (Tungsten, GPU)– 利便性・多様なユースケース (GraphFrame,DataFrame)

• エコシステムは継続的に拡充している

• おおよそ考えうる限りのユースケースが出始めている

9

Copyright 2016 CREATIONLINE, INC. All Rights Reservedⓒ 10