An Infinite Factor Model Hierarchy Via a Noisy-Or Mechanism

Aaron C. Courville, Douglas Eck and Yoshua BengioNIPS 2009

Presented by Lingbo LiECE, Duke University

May 21, 2010

Note: all tables and figures taken from the original paper

Outline

• Motivations• Latent Factor Modeling• A Hierarchy of Latent Features Via a Noisy-

Or Mechanism• Inference • Experiments• Conclusions

Motivations• Indian Buffet Process (IBP): factorial representation of data.• Music tag data (Last.fm): to organize playlists. e.g. RADIOHEAD: alternative, rock, alternative rock, indie, el

ectronic, britpop, british, and indie rock.• In IBP, latent features are independent across object instances.• Dependency between latent factors: co-occurrence of some fea

tures. e.g. ‘alternative’ + ’indie’ > ‘alternative’ + ‘classical’• Extend infinite latent factor models to two unbounded layers o

f factors.• ‘Upper-layer factors express correlations between lower-layer

factors via a noisy-or mechanism.’

Latent Factor Modeling• objects model parameters binary feature variables • Features: active inactive • Model is summarized as

• are mutually independent.

Latent Factor Modeling

• As , IBP gets the distribution of an unbounded binary feature matrix by marginalizing out

• Stick-breaking construction for the IBP

• Factor probabilities are expressed as:

A Hierarchy of Latent Features Via a Noisy-OR Mechanism

Extent to two layers of binary latent features: an upper-layer binary latent feature matrix with elements

an lower-layer binary latent feature matrix with elements

• The weight matrix connect every element of to every element of , where

• The active can be interpreted as ‘the possible causes of the activation of the individual

A Hierarchy of Latent Features Via a Noisy-OR Mechanism

A Hierarchy of Latent Features Via a Noisy-OR Mechanism

• Define an additional random matrix with inactive upper-layer features are failure

s active upper-layer features are failures• For each

trial

SuccessNo further trials

FailureMove on to Trial

if all trials with

A Hierarchy of Latent Features Via a Noisy-OR Mechanism

• Posterior distributions for the model parameters and

: number of times is active

: number of times that the j-th trial was a

success for : number of times that the j-th trial was a failure for despite being active

• Integrate out

Inference• Based on the blocked Gibbs sampling and the IBP

semi-ordered slice sampler

• Semi-ordered slice sampling of the upper-layer IBP

• Semi-ordered slice sampling of the lower-layer factor model

• Efficient extended blocked Gibbs sampler over the entire model without approximation

Experiments (I)• MNIST dataset• 1000 examples of images of the digit 3, preprocessed

by projecting onto the first 64 PCA components• Set 500 examples as training and the left 500 as

testing• Each data object is modeled as

• Add random noise (std = 0.5) on the post-processed test set

• Recover the noise-free version

Experiments (II)• Music Tags • Tags and tag frequencies are extracted from the social music

website (http://www.last.fm/) using the Audioscrobbler web service

• Dataset: 1000 artists with a vocabulary size of 100 tags representing a total 312134 counts.

• Goal: to reduce the noisy collection of tags to a sparse representation for each artist;

• Model the data as

where C is the limit on the number of possible counts achievable, C=100

Experiments (II)

• Both layers are sparse• Most features at the upper layer use one to three tags• Most features at the lower layer cover a broader range of tags

Tags with the two most probable factors at the upper layer:

Experiments (II)• Comparison among Generalized linear model, IBP and two-

layer Noisy-Or IFM • Test data contains 600 artist-tag collections, and 90% of the tags

are missing; To impute the missing data from the left 10%.• For generalized linear model

• Both IBP and noisy-or models perform better than the generalized latent linear model

Conclusions• Bayesian nonparametric version of the noisy-or mechanism

• Extend infinite latent factor models to two or more unbounded layers of factors

• Efficient inference via Gibbs sampling procedure

• Compare performance with the standard IBP construction