Podling Hivemall in the Apache Incubator

13
Podling Hivemall in the Apache Incubator Research Engineer Makoto YUI @myui <[email protected]> 1 2016/11/08 Apache Hadoop Meetup at CWT 2016

Transcript of Podling Hivemall in the Apache Incubator

Page 1: Podling Hivemall in the Apache Incubator

Podling HivemallintheApacheIncubator

ResearchEngineerMakotoYUI@myui

<[email protected]>

12016/11/08ApacheHadoopMeetupatCWT2016

Page 2: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 2

HivemallenteredApacheIncubatoronSept13,2016🎉

hivemall.incubator.apache.org

@ApacheHivemall

Page 3: Podling Hivemall in the Apache Incubator

•MakotoYui<TreasureData>• TakeshiYamamuro <NTT>Ø HivemallonApacheSpark• DanielDai<Hortonworks>Ø HivemallonApachePigØ ApachePigPMCmember• TsuyoshiOzawa<NTT>ØApacheHadoopPMCmember• KaiSasaki<TreasureData>

3

Initialcommitters

2016/11/08ApacheHadoopMeetupatCWT2016

Page 4: Podling Hivemall in the Apache Incubator

Champion

NominatedMentors

4

Projectmentors

• ReynoldXin<Databricks,ASFmember>ApacheSparkPMCmember• MarkusWeimer<Microsoft,ASFmember>ApacheREEFPMCmember• Xiangrui Meng <Databricks,ASFmember>ApacheSparkPMCmember

• RomanShaposhnik <Pivotal,ASFmember>ApacheBigtop/IncubatorPMCmember

2016/11/08ApacheHadoopMeetupatCWT2016

Page 5: Podling Hivemall in the Apache Incubator

WhatisApacheHivemall

ScalablemachinelearninglibrarybuiltasacollectionofHiveUDFs

52016/11/08ApacheHadoopMeetupatCWT2016

Multi/Crossplatform Versatile Scalable Ease-of-use

Page 6: Podling Hivemall in the Apache Incubator

Hivemalliseasyandscalable…

ClassificationwithMahout

CREATETABLElr_model ASSELECTfeature,-- reducersperformmodelaveraginginparallelavg(weight)asweightFROM(SELECTlogress(features,label,..)as(feature,weight)FROMtrain)t-- map-onlytaskGROUPBYfeature;-- shuffledtoreducers

MLmadeeasyforSQLdevelopers

Borntobeparallelandscalable

ThisSQLqueryautomaticallyrunsinparallelonHadoopcluster

62016/11/08ApacheHadoopMeetupatCWT2016

Ease-of-use

Scalable

Page 7: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 7

Hivemallisamulti/cross-platformMLlibrary

HiveQL SparkSQL/Dataframe API PigLatin

HivemallisMulti/Crossplatform..

Multi/Crossplatform

predictionmodelsbuiltbyHivecanbeusedfromSpark,andconversely,predictionmodelsbuildbySparkcanbeusedfromHive

Page 8: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 8

HivemallonApacheHive

Page 9: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 9

HivemallonApacheSparkDataframe

Page 10: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 10

HivemallonSparkSQL

Page 11: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 11

HivemallonApachePig

Page 12: Podling Hivemall in the Apache Incubator

2016/11/08ApacheHadoopMeetupatCWT2016 12

Versatile

HivemallisaVersatilelibrary..

ü HivemallisnotonlyforMachineLearning

ü Hivemallprovidesbunchofgenericutilityfunctions(e.g.,top-k,NLP)

EachorganizationhasownsetsofUDFsfordatapreprocessing!

Don’tRepeatYourself!Don’tRepeatYourself!

Page 13: Podling Hivemall in the Apache Incubator

ConclusionandTakeaway

Hivemallisamachinelearninglibrarythatis…

2016/11/08ApacheHadoopMeetupatCWT2016 13

WewelcomeyourcontributionstoApacheHivemallJ

Multi/Crossplatform Versatile Scalable Ease-of-use

hivemall.incubator.apache.org


UAM Azcapotzalco mcbc@correo.azc.uamaisii.azc.uam.mx/mcbc/Cursos/POS/Sesion 8.pdfApache Wink, proyecto de Apache Software Foundation Incubator, el módulo del servidor implementa JAX-RS.

UAM Azcapotzalco [email protected]/mcbc/Cursos/POS/Sesion 8.pdfApache Wink, proyecto de Apache Software Foundation Incubator, el módulo del servidor implementa JAX-RS.

Yonik Seeley yonik@apache.org 29 June 2006 Dublin, Irelandpeople.apache.org/~yonik/presentations/Solr.pdf• CNET grants code to Apache, Solr enters Incubator 17 Jan 2006 • Solr

Yonik Seeley [email protected] 29 June 2006 Dublin, Irelandpeople.apache.org/~yonik/presentations/Solr.pdf• CNET grants code to Apache, Solr enters Incubator 17 Jan 2006 • Solr

A pac he Explained - servicecomb.incubator.apache.orgservicecomb.incubator.apache.org/assets/slides/...Sep 20, 2019  · Member, Incubator Project Management Committee Chairman, Apache

A pac he Explained - servicecomb.incubator.apache.orgservicecomb.incubator.apache.org/assets/slides/...Sep 20, 2019  · Member, Incubator Project Management Committee Chairman, Apache

Hivemall: Scalable machine learning library for Apache Hive/Spark/Pig

Hivemall: Scalable machine learning library for Apache Hive/Spark/Pig

Content Storage with Apache Jackrabbitarchive.apachecon.com/c/aceu2009/sessions/materials/...Introducing Apache Jackrabbit Entered Apache Incubator in 2004, graduated in 2006, now

Content Storage with Apache Jackrabbitarchive.apachecon.com/c/aceu2009/sessions/materials/...Introducing Apache Jackrabbit Entered Apache Incubator in 2004, graduated in 2006, now

Apache Incubator: where it is coming from and where it is ... · • Started from Apache Group in 1999 • 8 folks hacking abandoned httpd code from NCSA • Formed a non-profit,

Apache Incubator: where it is coming from and where it is ... · • Started from Apache Group in 1999 • 8 folks hacking abandoned httpd code from NCSA • Formed a non-profit,

airflow ozgur yazilim · 2018-06-05 · apache / incubator-airflow Watch 547 Unstar 8,040 Fork Apache-2.0 2,571 O Code Pull requests 210 Projects O p 7 branches Insights 0 90 releases

airflow ozgur yazilim · 2018-06-05 · apache / incubator-airflow Watch 547 Unstar 8,040 Fork Apache-2.0 2,571 O Code Pull requests 210 Projects O p 7 branches Insights 0 90 releases

Apache Hivemall Meets PySpark...Apache Hivemall Meets PySpark Scalable Machine Learning with Hive, Spark, and Python Takuya Kitazawa @ takuti Apache Hivemall PPMC EUROPEScalability

Apache Hivemall Meets PySpark...Apache Hivemall Meets PySpark Scalable Machine Learning with Hive, Spark, and Python Takuya Kitazawa @ takuti Apache Hivemall PPMC EUROPEScalability

INTRODUCING APACHE IGNITE An Apache Incubator Project

INTRODUCING APACHE IGNITE An Apache Incubator Project

LTP0067-Incubator REV C - Scientific Industries · INCUBATOR-GENIE ® Environmentally Controlled Benchtop 3-in-1 Incubator / Rotator / Rocker The Incubator-Genie is an Incubator

LTP0067-Incubator REV C - Scientific Industries · INCUBATOR-GENIE ® Environmentally Controlled Benchtop 3-in-1 Incubator / Rotator / Rocker The Incubator-Genie is an Incubator

Recommendation 101 using Hivemall

Recommendation 101 using Hivemall

Using Apache Spark Pat McDonough - Databricks. Apache Spark spark.incubator.apache.org github.com/apache/incubator- spark user@spark.incubator.apache.or.

Using Apache Spark Pat McDonough - Databricks. Apache Spark spark.incubator.apache.org github.com/apache/incubator- spark [email protected].

2nd Hivemall meetup 20151020

2nd Hivemall meetup 20151020

Should I bring my project to the Apache Incubator?

Should I bring my project to the Apache Incubator?

2014-10-30 Taverna as an Apache Incubator project

2014-10-30 Taverna as an Apache Incubator project

Apache Kafka - docs.huihoo.comdocs.huihoo.com/apache/kafka/Introduction-and... · Project info Written in Scala Open sourced by LinkedIn SNA in 2011 Soon after entered Apache Incubator

Apache Kafka - docs.huihoo.comdocs.huihoo.com/apache/kafka/Introduction-and... · Project info Written in Scala Open sourced by LinkedIn SNA in 2011 Soon after entered Apache Incubator

Java Server Performance Monitoring - …people.apache.org/.../jsug-2015/jee-performance-monitoring.pdf• Displayed on multiple Flat Screen TVs ... Apache Sirona • Apache Incubator

Java Server Performance Monitoring - …people.apache.org/.../jsug-2015/jee-performance-monitoring.pdf• Displayed on multiple Flat Screen TVs ... Apache Sirona • Apache Incubator

Hivemall Talk at TD tech talk #3

Hivemall Talk at TD tech talk #3

Happy Farm Business Incubator. Happy Farm Business Incubator:

Happy Farm Business Incubator. Happy Farm Business Incubator:

Hivemall v0.3の機能紹介＠1st Hivemall meetup

Hivemall v0.3の機能紹介＠1st Hivemall meetup