Introduction to Mahout

1©MapR Technologies 2013- Confidential

Introduction to MahoutAnd How To Build a Recommender


Me, Us

Ted Dunning, Chief Application Architect, MapRCommitter PMC member, Mahout, Zookeeper, DrillBought the beer at the first HUG

MapRDistributes more open source components for HadoopAdds major technology for performance, HA, industry standard API’s

TonightHash tag - #tchugSee also - @ApacheMahout @ApacheDrill

@ted_dunning and @mapR


Sidebar on Drill

Apache Drill– SQL on Hadoop (and other things)– Intended to solve problems for 1-5 years from now

Not the problems from 1-10 years ago– Multiple levels of API supported• SQL-2003• Logical plan language (DAG in JSON)• Physical plan language (DAG with push-down, exchange markers)• Execution plan language (many DAG’s)

Current state– SQL 2003 support in place– Logical plan interpreter useful for testing– Value vectors near completion– High performance RPC working


More on Drill

Just completed OSCON workshop

Workshop materials available shortly– Extracted technology demonstrators– Sample queries

Send me email or tweet for more info


What’s Up?

What is Mahout?– Math library– Clustering, classifiers, other stuff

Recommendation– Generalities– Algorithm Specifics– System Design– Important things never mentioned

Final thoughts


What is Mahout?

“Scalable machine learning”– not just Hadoop-oriented machine learning– not entirely, that is. Just mostly.

Components– math library– clustering– classification– decompositions– recommendations


Mahout Math


Mahout Math

Goals are– basic linear algebra,– and statistical sampling,– and good clustering,– decent speed,– extensibility,– especially for sparse data

But not – totally badass speed– comprehensive set of algorithms– optimization, root finders, quadrature


Matrices and Vectors

At the core:– DenseVector, RandomAccessSparseVector– DenseMatrix, SparseRowMatrix

Highly composable API

Important ideas: – view*, assign and aggregate– iteration

m.viewDiagonal().assign(v)


Assign? View?

Why assign?– Copying is the major cost for naïve matrix packages– In-place operations critical to reasonable performance– Many kinds of updates required, so functional style very helpful

Why view?– In-place operations often required for blocks, rows, columns or diagonals– With views, we need #assign + #views methods– Without views, we need #assign x #views methods

Synergies– With both views and assign, many loops become single line


Assign

Matrices

Vectors

Matrix assign(double value);Matrix assign(double[][] values);Matrix assign(Matrix other);Matrix assign(DoubleFunction f);Matrix assign(Matrix other, DoubleDoubleFunction f);

Vector assign(double value);Vector assign(double[] values);Vector assign(Vector other);Vector assign(DoubleFunction f);Vector assign(Vector other, DoubleDoubleFunction f);Vector assign(DoubleDoubleFunction f, double y);


Views

Matrices

Vectors

Matrix viewPart(int[] offset, int[] size);Matrix viewPart(int row, int rlen, int col, int clen);Vector viewRow(int row);Vector viewColumn(int column);Vector viewDiagonal();

Vector viewPart(int offset, int length);


Aggregates

Matrices

Vectors

double zSum();double aggregate( DoubleDoubleFunction reduce, DoubleFunction map);double aggregate(Vector other, DoubleDoubleFunction aggregator, DoubleDoubleFunction combiner);

double zSum();Vector aggregateRows(VectorFunction f);Vector aggregateColumns(VectorFunction f);double aggregate(DoubleDoubleFunction combiner, DoubleFunction mapper);


Predefined Functions

Many handy functions

ABS LOG2 ACOS NEGATE ASIN RINT ATAN SIGN CEIL SIN COS SQRT EXP SQUARE FLOOR SIGMOID IDENTITY SIGMOIDGRADIENT INV TAN LOGARITHM


Examples

double alpha; a.assign(alpha);

a.assign(b, Functions.chain( Functions.plus(beta), Functions.times(alpha));


Sparse Optimizations

DoubleDoubleFunction abstract properties

And Vector properties

public boolean isLikeRightPlus();public boolean isLikeLeftMult();public boolean isLikeRightMult();public boolean isLikeMult();public boolean isCommutative();public boolean isAssociative();public boolean isAssociativeAndCommutative();public boolean isDensifying();

public boolean isDense();public boolean isSequentialAccess();public double getLookupCost();public double getIteratorAdvanceCost();public boolean isAddConstantTime();


More Examples

The trace of a matrix

Set diagonal to zero

Set diagonal to negative of row sums


Examples




m.viewDiagonal().zSum()


Examples





m.viewDiagonal().assign(0)


Examples



Set diagonal to negative of row sums excluding the diagonal


m.viewDiagonal().assign(0)

Vector diag = m.viewDiagonal().assign(0);diag.assign(m.rowSums().assign(Functions.MINUS));


Iteration

Matrices are Iterable in Mahout

Vectors are densely or sparsely iterable

// compute both row and columns sums in one passfor (MatrixSlice row: m) { rSums.set(row.index(), row.zSum()); cSums.assign(row, Functions.PLUS);}

double entropy = 0;for (Vector.Element e: v.nonZeroes()) { entropy += e.get() * Math.log(e.get());}


Random Sampling

Samples from some type

Lots of kinds

ChineseRestaurant Missing Normal Empirical Multinomial PoissonSampler IndianBuffet MultiNormal Sampler

public interface Sampler<T> { T sample();}

public abstract class AbstractSamplerFunction extends DoubleFunction implements Sampler<Double>


Clustering and Such

Streaming k-means and ball k-means– streaming reduces very large data to a cluster sketch– ball k-means is a high quality k-means implementation– the cluster sketch is also usable for other applications– single machine threaded and map-reduce versions available

SVD and friends– stochastic SVD has in-memory, single machine out-of-core and map-reduce

versions– good for reducing very large sparse matrices to tall skinny dense ones

Spectral clustering– based on SVD, allows massive dimensional clustering


Mahout Math Summary

Matrices, Vectors– views– in-place assignment– aggregations– iterations

Functions– lots built-in– cooperate with sparse vector optimizations

Sampling– abstract samplers– samplers as functions

Other stuff … clustering, SVD


Recommenders


Recommendations

Often known as collaborative filtering Actors interact with items– observe successful interaction

We want to suggest additional successful interactions Observations inherently very sparse


The Big Ideas

Cooccurrence is the core operation (and it is pretty simple)

Cooccurrence can be extended to handle important new capabilities

Recommendation systems can be deployed ideally using search technology


Examples of Recommendations

Customers buying books (Linden et al) Web visitors rating music (Shardanand and Maes) or movies (Riedl,

et al), (Netflix) Internet radio listeners not skipping songs (Musicmatch) Internet video watchers watching >30 s (Veoh) Visibility in a map UI (new Google maps)


A simple recommendation architecture

Look at the history of interactions

Find significant item cooccurrence in user histories

Use these cooccurring items as “indicators”

For all indicators in user history, accumulate scores for related items


Recommendation Basics

History:

User Thing1 3

2 4

3 4

2 3

3 2

1 1

2 1



History as matrix:

(t1, t3) cooccur 2 times, (t1, t4) once, (t2, t4) once, (t3, t4) once

t1 t2 t3 t4

u1 1 0 1 0

u2 1 0 1 1

u3 0 1 0 1


A Quick Simplification

Users who do h

Also do r

User-centric recommendations

Item-centric recommendations



Coocurrence

t1 t2 t3 t4

t1 2 0 2 1

t2 0 1 0 1

t3 2 0 1 1

t4 1 1 1 2


Problems with Raw Cooccurrence

Very popular items co-occur with everything– Welcome document– Elevator music

That isn’t interesting– We want anomalous cooccurrence



Coocurrence

t1 t2 t3 t4

t1 2 0 2 1

t2 0 1 0 1

t3 2 0 1 1

t4 1 1 1 2t3 not t3

t1 2 1

not t1 1 1


Spot the Anomaly

Root LLR is roughly like standard deviations

A not A

B 13 1000

not B 1000 100,000

A not A

B 1 0

not B 0 2

A not A

B 1 0

not B 0 10,000

A not A

B 10 0

not B 0 100,000

0.44 0.98

2.26 7.15


Threshold by Score

Coocurrence

t1 t2 t3 t4

t1 2 0 2 1

t2 0 1 0 1

t3 2 0 1 1

t4 1 1 1 2


Threshold by Score

Significant cooccurrence => Indicators

t1 t2 t3 t4

t1 1 0 0 1t2 0 1 0 1t3 0 0 1 1t4 1 0 0 1


So Far, So Good

Classic recommendation systems based on these approaches– Musicmatch (ca 2000)– Veoh Networks (ca 2005)

Currently available in Mahout– See RowSimilarityJob

Very simple to deploy– Compute indicators– Store in search engine– Works very well with enough data


What’s right about this?


Virtues of Current State of the Art

Lots of well publicized history– Musicmatch, Veoh, Netflix, Amazon, Overstock

Lots of support– Mahout, commercial offerings like Myrrix

Lots of existing code– Mahout, commercial codes

Proven track record Well socialized solution


What’s wrong about this?


Problems for Recommenders

Cold start Disjoint populations Long tail Multiple kinds of evidence (multi-modal recommendations)– unstructured add-on data– other transaction streams– textual descriptions


What is this multi-modal stuff?

But people don’t just do one thing

One kind of behavior is useful for predicting other kinds

Having a complete picture is important for accuracy

What has the user said, viewed, clicked, closed, bought lately?


Example Multi-modal Inputs

Overlap in restaurant visits is useful Big spender cues Cuisine as an indicator Review text as an indicator


Too Limited

People do more than one kind of thing Different kinds of behaviors give different quality, quantity and

kind of information

We don’t have to do co-occurrence We can do cross-occurrence

Result is cross-recommendation


Heh?


For example

Users enter queries (A)– (actor = user, item=query)

Users view videos (B)– (actor = user, item=video)

ATA gives query recommendation– “did you mean to ask for”

BTB gives video recommendation– “you might like these videos”


The punch-line

BTA recommends videos in response to a query– (isn’t that a search engine?)– (not quite, it doesn’t look at content or meta-data)


Real-life example

Query: “Paco de Lucia” Conventional meta-data search results:– “hombres del paco” times 400– not much else

Recommendation based search:– Flamenco guitar and dancers– Spanish and classical guitar– Van Halen doing a classical/flamenco riff


Real-life example


Hypothetical Example

Want a navigational ontology? Just put labels on a web page with traffic– This gives A = users x label clicks

Remember viewing history– This gives B = users x items

Cross recommend– B’A = label to item mapping

After several users click, results are whatever users think they should be


Nice. But we can do better?


users

things


users

thingtype 1

thingtype 2


Summary

Input: Multiple kinds of behavior on one set of things

Output: Recommendations for one kind of behavior with a different set of things

Cross recommendation is a special case


Now again, without the scary math


Input Data User transactions– user id, merchant id– SIC code, amount– Descriptions, cuisine, …

Offer transactions– user id, offer id– vendor id, merchant id’s, – offers, views, accepts


Input Data User transactions– user id, merchant id– SIC code, amount– Descriptions, cuisine, …

Offer transactions– user id, offer id– vendor id, merchant id’s, – offers, views, accepts

Derived user data– merchant id’s– anomalous descriptor terms– offer & vendor id’s

Derived merchant data– local top40– SIC code– vendor code– amount distribution


Cross-recommendation

Per merchant indicators– merchant id’s– chain id’s– SIC codes– indicator terms from text– offer vendor id’s

Computed by finding anomalous (indicator => merchant) rates


How can we deploy this?


Search-based Recommendations

Sample document– Merchant Id– Field for text description– Phone– Address– Location




– Indicator merchant id’s– Indicator industry (SIC) id’s– Indicator offers– Indicator text– Local top40





Sample query– Current location– Recent merchant descriptions– Recent merchant id’s– Recent SIC codes– Recent accepted offers– Local top40





Sample query– Current location– Recent merchant descriptions– Recent merchant id’s– Recent SIC codes– Recent accepted offers– Local top40

Original data and meta-data

Derived from cooccurrence and cross-occurrence analysis

Recommendation query


SolRIndexerSolR

IndexerSolrindexing

Cooccurrence(Mahout)

Item meta-data

Indexshards

Complete history

Analyze with Map-Reduce


SolRIndexerSolR

IndexerSolrsearchWeb tier

Item meta-data

Indexshards

User history

Deploy with Conventional Search System


Objective Results

At a very large credit card company

History is all transactions

Development time to minimal viable product about 4 months

General release 2-3 months later

Search-based recs at or equal in quality to other techniques


Contact:– [email protected]– @ted_dunning– @apachemahout– @[email protected]

Slides and suchhttp://www.slideshare.net/tdunning

Hash tags: #mapr #apachemahout #recommendations

mailto:[email protected]

http://www.slideshare.net/tdunning



Introduction to Mahout

Technology

Transcript of Introduction to Mahout