Non-negative Matrix Factorization with Sparseness Constraints
description
Transcript of Non-negative Matrix Factorization with Sparseness Constraints
NON-NEGATIVE MATRIX FACTORIZATION WITH SPARSENESS CONSTRAINTS
JOURNAL OF MACHINE LEARNING RESEARCH,2004Patrik O. Hoyer
Jain-De,Lee
2
OUTLINE Introduction
Adding Sparseness Constraints to NMF
Experiments with Sparseness Constraints
Conclusions
3
INTRODUCTION• Non-negative matrix factorization (NMF)• A useful representation typically makes latent
structure in the data explicit• Reduces the dimensionality of the data
• The non-negativity constraints make the representation purely additive
4
INTRODUCTION• Sparse representation• Representation encodes much of the data using few
‘active’ components
• The sparseness given by NMF is somewhat of a side-effect rather than a goal
• Include the option to control sparseness explicitly
5
ADDING SPARSENESS CONSTRAINTS TO NMF• The concept of sparse coding• Only a few units are effectively used to represent
typical data vectors
Illustration of various degrees of sparseness
6
ADDING SPARSENESS CONSTRAINTS TO NMF• Sparseness measure
• Based on the relationship between the L1 norm and the L2 norm
1
/)()(
2
n
xxinxsparseness i
where n is the dimensionality of x
7
ADDING SPARSENESS CONSTRAINTS TO NMF• To constrain NMF to find solutions with desired
degrees of sparseness• What exactly should be sparse?• Under optional constraints minimized
iSwsparseness wi ,)(
iShsparseness hi ,)(
Where wi is the ith column of W and hi is the ith row of H
Sw and Sh are the desired sparsenesses of W and H (respectively)
8
ADDING SPARSENESS CONSTRAINTS TO NMF• Projected gradient descent algorithm for NMF
with sparseness constraints
Initialize
Project
Iterate
9
ADDING SPARSENESS CONSTRAINTS TO NMF
Project
If sparseness constraints on W:
Project each column of W to be non-negative
Have unchanged L2 norm, L1 norm set to achieve desired sparseness
If sparseness constraints on H:
Project each row of H to be non-negative
Have unchanged L2 norm, L1 norm set to achieve desired sparseness
10
ADDING SPARSENESS CONSTRAINTS TO NMF• Iterate• If sparseness constraints on W(or H) apply
• Set or
• Project• else take standard multiplicative step
TW HVWHWW )(
)( VWHWHH TH
Where μW and μH are small positive constants
T
T
HWHVHWW
)( or
)(WHWVWHH T
T
11
ADDING SPARSENESS CONSTRAINTS TO NMF
• Projection operator• Problem
Given any vector x, find the closest non-negative vector s with a given L1 norm and a given L2 norm
12
ADDING SPARSENESS CONSTRAINTS TO NMF• Algorithm• Set• Set Z={}• Iterate
1. Set2. Set ,where α ≥ 03. If all components of s are non-negative, return
s, end 4. Set Z=Z {i ; s∪ i<0}5. Set si=0 , 6. Calculate 7. Set si= si –c ,8. Go to 1
ixxLxs iii ),dim(/)( 1
0
))()/(dim(1 ZsizexLmi
)( msms
Zi
Zi))()/(dim()( 1 ZsizexLsc i
ZiifZiif
13
EXPERIMENTS WITH SPARSENESS CONSTRAINTS
NMF applied to various image data sets(a) Basis images given by NMF applied to face image data from the CBCL database(b) Basis images derived from the ORL face image database
(c) Basis vectors from NMF applied to ON/OFF-contrast filtered natural image data
14
EXPERIMENTS WITH SPARSENESS CONSTRAINTS
Features learned from the CBCL face image database using NMF with sparseness constraints
15
EXPERIMENTS WITH SPARSENESS CONSTRAINTS
Features learned from the ORL face image database using NMF with three levels sparseness constraints (a) 0.5 (b) 0.6 (c) 0.75
16
EXPERIMENTS WITH SPARSENESS CONSTRAINTS
Standard NMF (Figure 1c)
The sparseness of the coefficients was fixed at 0.85
17
EXPERIMENTS WITH SPARSENESS CONSTRAINTS
Number of iterations required for the projectionalgorithm to converge
18
CONCLUSIONS• Useful to control the degree of sparseness explicitly
• Describe a projection operator capable of simultaneously enforcing both L1 and L2 norms
• To show its use in the NMF framework for learning representations that could not be obtained by regular NMF