2nd Hivemall meetup 20151020

IntroductiontoHivemallandit’snewfeaturesinv0.4

ResearchEngineerMakotoYUI@myui

2015/10/20Hivemallmeetup#2 1

Tweetw/#hivemallmtup

http://eventdots.jp/event/571107

Ø 2015.04JoinedTreasureData,Inc.1st ResearchEngineerinTreasureDataMymissioninTDisdevelopingML-as-a-Service

Ø 2010.04-2015.03SeniorResearcheratNationalInstituteofAdvancedIndustrialScienceandTechnology,Japan.Workedonalarge-scaleMachineLearningprojectandParallelDatabases

Ø 2009.03Ph.D.inComputerSciencefromNAISTØ SuperprogrammerawardfromtheMITOU

Foundation

WhoamI?


Agenda

1. WhatisHivemall

2. HowtouseHivemall

3. NewFeaturesinHivemallv0.41. RandomForest2. FactorizationMachine

4. DevelopmentRoadmapofHivemall


WhatisHivemallScalablemachinelearninglibrarybuiltasacollectionofHiveUDFs,licensedundertheApacheLicensev2


https://github.com/myui/hivemall

WhatisHivemall

HadoopHDFS

MapReduce(MR v1)

Hive /PIG

Hivemall

ApacheYARN

ApacheTezDAGprocessing MRv2

MachineLearning

QueryProcessing

ParallelDataProcessingFramework

ResourceManagement

DistributedFileSystem


ScalablemachinelearninglibrarybuiltasacollectionofHiveUDFs,licensedundertheApacheLicensev2

Hivemall’s Vision:MLonSQL

ClassificationwithMahout

CREATETABLElr_model ASSELECTfeature,-- reducersperformmodelaveraginginparallelavg(weight)asweightFROM(SELECTlogress(features,label,..)as(feature,weight)FROMtrain)t-- map-onlytaskGROUPBYfeature;-- shuffledtoreducers

✓MachineLearningmadeeasyforSQLdevelopers(MLfortherestofus)✓InteractiveandStableAPIsw/ SQLabstraction

ThisSQLqueryautomaticallyrunsinparallelonHadoop


ListofFeaturesinHivemallv0.3.2Classification(bothbinary- andmulti-class)✓ Perceptron✓ PassiveAggressive(PA)✓ ConfidenceWeighted(CW)✓ AdaptiveRegularizationofWeightVectors(AROW)✓ SoftConfidenceWeighted(SCW)✓ AdaGrad+RDA

Regression✓LogisticRegression(SGD)✓PARegression✓AROWRegression✓AdaGrad✓AdaDELTA

kNN andRecommendation✓Minhash andb-BitMinhash(LSHvariant)✓ Similarity SearchusingK-NN

(Euclid/Cosine/Jaccard/Angular)✓MatrixFactorization

Featureengineering✓ FeatureHashing✓ FeatureScaling(normalization, z-score)✓ TF-IDFvectorizer✓ Polynomial Expansion

AnomalyDetection✓ LocalOutlierFactor

TreasureDatasupportsHivemallv0.3.2-3


Ø CTRpredictionofAdclicklogs• Algorithm:Logisticregression• Freakout Inc.andmore

Ø GenderpredictionofAdclicklogs• Algorithm:Classification• Scaleout Inc.

Ø ChurnDetection• Algorithm:Regression• OISIXandmore

Ø Item/Userrecommendation• Algorithm:Recommendation(MatrixFactorization/kNN)• Adtech Companies,ISPportal,andmore

Ø ValuepredictionofRealestates• Algorithm:Regression• Livesense

IndustryusecasesofHivemall

82015/10/20Hivemallmeetup#2

HowtouseHivemall

MachineLearning

Training

Prediction

PredictionModel Label

FeatureVector

FeatureVector

Label

Datapreparation2015/10/20Hivemallmeetup#2 9

CREATE EXTERNAL TABLE e2006tfidf_train (rowid int,label float,features ARRAY<STRING>

) ROW FORMAT DELIMITED

FIELDS TERMINATED BY '¥t' COLLECTION ITEMS TERMINATED BY ",“

STORED AS TEXTFILE LOCATION '/dataset/E2006-tfidf/train';

HowtouseHivemall- Datapreparation

DefineaHivetablefortraining/testingdata


HowtouseHivemall

MachineLearning

Training

Prediction


FeatureVector

FeatureVector

Label

FeatureEngineering


create view e2006tfidf_train_scaled asselect

rowid,rescale(target,${min_label},${max_label}) as label,

featuresfrom

e2006tfidf_train;

ApplyingaMin-MaxFeatureNormalization

HowtouseHivemall- FeatureEngineering

Transformingalabelvaluetoavaluebetween0.0and1.0


HowtouseHivemall

MachineLearning

Training

Prediction


FeatureVector

FeatureVector

Label

Training


HowtouseHivemall- Training

CREATE TABLE lr_model ASSELECTfeature,avg(weight) as weight

FROM (SELECT logress(features,label,..)

as (feature,weight)FROM train

) tGROUP BY feature

Trainingbylogisticregression

map-onlytasktolearnapredictionmodel

Shufflemap-outputstoreducesbyfeature

Reducersperformmodelaveraginginparallel


HowtouseHivemall- Training

CREATE TABLE news20b_cw_model1 ASSELECT

feature,voted_avg(weight) as weight

FROM(SELECT

train_cw(features,label) as (feature,weight)

FROMnews20b_train

) t GROUP BY feature

TrainingofConfidenceWeightedClassifier

Votetousenegativeorpositiveweightsforavg

+0.7,+0.3,+0.2,-0.1,+0.7

TrainingfortheCWclassifier


create table news20mc_ensemble_model1 asselect label, cast(feature as int) as feature,cast(voted_avg(weight) as float) as weightfrom (select

train_multiclass_cw(addBias(features),label) as (label,feature,weight)

from news20mc_train_x3

union allselect

train_multiclass_arow(addBias(features),label) as (label,feature,weight)


union allselect

train_multiclass_scw(addBias(features),label)as (label,feature,weight)


) t group by label, feature;

Ensemblelearningforstablepredictionperformance

Juststackpredictionmodelsbyunionall

26 / 43162015/10/20Hivemallmeetup#2

HowtouseHivemall

MachineLearning

Training

Prediction


FeatureVector

FeatureVector

Label

Prediction


HowtouseHivemall- Prediction

CREATETABLElr_predictasSELECTt.rowid,sigmoid(sum(m.weight)) asprobFROMtesting_exploded tLEFTOUTERJOINlr_model mON(t.feature =m.feature)GROUPBYt.rowid

PredictionisdonebyLEFTOUTERJOINbetweentestdataandpredictionmodel

Noneedtoloadtheentiremodelintomemory


HowtouseHivemall

MachineLearning

Batch Training on Hadoop

Online Prediction on RDBMS


FeatureVector

FeatureVector

Label

Exportpredictionmodels



OnlinePredictiononMySQL(RDBMS)

Quick(msec)responseonaRDBMSbyaddinganindextofeaturecolumn

bit.ly/hivemall-mysql

Agenda

1. WhatisHivemall

2. HowtouseHivemall




Features tobesupportedinHivemallv0.4


1.RandomForest• classification,regression• BasedonSmilegithub.com/haifengl/smile

2.FactorizationMachine• classification,regression (factorization)

Plannedtoreleasev0.4inOct.

FactorizationMachineareoftenusedbydatasciencecompetitionwinners(Criteo/Avazu CTRprediction)


RandomForestinHivemallv0.4

EnsembleofDecisionTrees

Alreadyavailableonadevelopment(smile)branchandit’susageisexplainedintheprojectwiki

Bagging


TrainingofRandomForest

Out-of-bagtestsandVariableImportance



PredictionofRandomForest


RandomForest

DEMO

http://bit.ly/hivemall-rf


FactorizationMachine

MatrixFactorization



Contextinformation(e.g.,time)canbeconsidered

Source:http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf



FactorizationModelwithdegress=2(2-wayinteraction)

Global BiasRegression coefficience

of j-th variable

Pairwise Interaction

Factorization



FactorizationMachine≈ PolynomialRegression+Factorization

Forafeature[a,b],thedegree-2polynomialfeaturesare[1,a,b,a^2,ab,b^2].

bit.ly/hivemall-poly



DEMO

Agenda

1. WhatisHivemall

2. HowtouseHivemall




Features tobesupportedinHivemallv0.4.1


1.GradientTreeBoosting• classifier,regression

2.Field-awareFactorizationMachine• classification,regression (factorization)• Existingimplementation, i.e.,LibFFM,onlycanbeappliedforclassification

Plannedtoreleasev0.4.1inNov/Dec.


GradientTreeBoosting(orGradientBoostingTrees)

RF≈Bagging+DecisionTreesparallel execution ofdecision trees

GBT≈Boosting+DecisionTreesSequential execution ofdecision trees


GradientTreeBoosting

Features tobesupportedinHivemallv0.4.2


1. OnlineLDA• topicmodeling,clustering

2. MixserveronApacheYARN• Serviceforparametersharingamongworkers• workingw/@maropu

Plannedtoreleasev0.4.2inDec/Jan.

Externalservicetoshareparametersbydistributedtrainingprocessesinthemiddleoftraining


What’sMixServer?

・・・・・・

Modelupdates

Async addPiggybackif…

AVG/Argmin KLDaccumulator

hash(feature)%N

Non-blockingChannel(singlesharedTCPconnectionw/TCPkeepalive)

classifiers

Mixserv.Mixserv.

Computation/trainingisnotbeingblocked

Takingbenefitsofasynchronousnon-blockingI/OisthecoreideabehindHivemall’s MIXprotocol


createtablekdd10a_pa1_model1asselectfeature,cast(voted_avg(weight)asfloat)asweightfrom(selecttrain_pa1(addBias(features),label,"-mixhost01,host02,host03")

as(feature,weight)fromkdd10a_train_x3

)tgroupbyfeature;

HowtouseMixServer

ConclusionandTakeaway

Newfeaturesinv0.4


• RandomForest• FactorizationMachine

Morewillfollowinv0.4.1

NextActions• ProposeHivemalltoApacheIncubator

• NewHivemallLogo

HivemallprovidesacollectionofmachinelearningalgorithmsasHiveUDFs/UDTFsThelatestversionofHivemallisavailableonTreasureDataandusedbyseveralcompaniesIncludingOISIX,Livesense,Scaleout,andFreakout.


BeyondQuery-as-a-Service!

WeOpen-source!Weinvented..

Wearehiringmachinelearningengineer!


Additionalslides

Recommendation

RatingpredictionofaMatrix

Canbeappliedforuser/ItemRecommendation


44

MatrixFactorization

Factorizeamatrixintoaproductofmatriceshavingk-latentfactor


45

MeanRating

MatrixFactorization

Regularization

Biasforeachuser/item

CriteriaofBiasedMF


Factorization

46

TrainingofMatrixFactorization

Support iterative training using local disk cache2015/10/20Hivemallmeetup#2

47

PredictionofMatrixFactorization


ØAlgorithmisdifferentSpark:ALS-WR(considersregularization)Hivemall:Biased-MF(considersregularizationandbiases)

ØUsabilitySpark:100+lineScalacodingHivemall:SQL(wouldbemoreeasytouse)

ØPredictionAccuracyAlmostsameforMovieLens 10Mdatasets


ComparisontoSparkMLlib

rowid features

1 ["reflectance:0.5252967","specific_heat:0.19863537","weight:0.0"]



UnsupervisedLearning:AnomalyDetection

Sensordataetc.

AnomalydetectionrunsonaseriesofSQLqueries



AnomaliesinaSensorData

Source:https://codeiq.jp/q/207

ImageSource:https://en.wikipedia.org/wiki/Local_outlier_factor2015/10/20Hivemallmeetup#2 51

LocalOutlierFactor(LoF)

BasicideaofLOF:comparingthelocaldensityofapointwiththedensities ofitsneighbors


DEMO:LocalOutlierFactor

rowid features




2nd Hivemall meetup 20151020

Engineering

Transcript of 2nd Hivemall meetup 20151020