Extensions of Non-negative Matrix Factorization to Higher Order data

Informatics and Mathematical Modelling / Intelligent Signal Processing

1Morten Mørup

Extensions of Non-negative Matrix Factorization to Higher Order dataMorten Mørup Informatics and Mathematical ModelingIntelligent Signal ProcessingTechnical University of Denmark


2Morten Mørup

Parts of the work done in collaboration with

Sæby, May 22-2006

Lars Kai Hansen, ProfessorDepartment of Signal Processing

Informatics and Mathematical Modeling,Technical University of Denmark

Mikkel N. Schmidt, Stud. PhDDepartment of Signal Processing

Informatics and Mathematical Modeling,Technical University of Denmark

Sidse M. Arnfred, Dr. Med. PhDCognitive Research Unit

Hvidovre HospitalUniversity Hospital of Copenhagen


3Morten Mørup

Outline Non-negativity Matrix Factorization

(NMF) Sparse coding

(SNMF)

Convolutive PARAFAC models (cPARAFAC) Higher Order Non-negative Matrix Factorization

(an extension of NMF to the Tucker model)


4Morten Mørup

NMF is based on Gradient Descent

NMF: VWH s.t. Wi,d,Hd,j0

Let C be a given cost function, then update the parameters according to:


5Morten Mørup

The idea behind multiplicative updates

Positive termNegative term


6Morten Mørup

Non-negative matrix factorization (NMF)

(Lee & Seung - 2001)

NMF gives Part based representation(Lee & Seung – Nature 1999)


7Morten Mørup

The NMF decomposition is not unique

Simplical ConeHWH)(WP)(PWHV -1 ~~

NMF only unique when data adequately spans the positive orthant (Donoho & Stodden - 2004)


8Morten Mørup

Sparse Coding NMF (SNMF)

(Eggert & Körner, 2004)


9Morten Mørup

Illustration (the swimmer problem)

True Expressions

Sw

imm

er A

rtic

ulat

ions

NMF Expressions SNMF Expressions

)()()( pixelExpressionExpressiononArticulatipixelonArticulati HWV


10Morten Mørup

Why sparseness? Ensures uniqueness Eases interpretability

(sparse representation factor effects pertain to fewer dimensions)

Can work as model selection(Sparseness can turn off excess factors by letting them become zero)

Resolves over complete representations (when model has many more free variables than data points)


11Morten Mørup

PART I: Convolutive PARAFAC (cPARAFAC)


12Morten Mørup

By cPARAFAC means PARAFAC convolutive in at least one modality

c2PARAFAC

,,,,,

cPARAFAC

,,,,,

PARAFACRegular

,,,,,

kjikji

kjikji

kjikji

HWDV

HWDV

HWDV

Convolution can be in any combination of modalities-Single convolutive, double convolutive etc.

,,, jiji SAX

Convolution: The process of generating Xby convolving (sending) the sources S

through the filter A

Deconvolution: The process of estimating

the filter A from X and S


13Morten Mørup

Relation to other models PARAFAC2 (Harshman, Kiers, Bro) Shifted PARAFAC (Hong and Harshman, 2003)

,,,,, kjikji HWDV

cPARAFAC can account for echo effects cPARAFAC becomes shifted PARAFACwhen convolutive filter is sparse

3

3


14Morten Mørup

Application example of cPARAFAC

Transcription and separation of music


15Morten Mørup

The ‘ideal’ Log-frequency Magnitude Spectrogram of an instrument Different notes played by an

instrument corresponds on a logarithmic frequency scale to a translation of the same harmonicstructure of a fixed temporal pattern

Time [s]

Fre

qu

en

cy

[H

z]

0 0.5 1 1.5 2 2.5 3 3.5

200

400

800

1600

3200

Tchaikovsky: Violin Concert in D Major

Mozart Sonate no,. 16 in C Major


16Morten Mørup

NMF 2D deconvolution (NMF2D1): The Basic Idea

Model a log-spectrogram of polyphonic music by an extended type of non-negative matrix factorization:– The frequency signature of a specific note played by an

instrument has a fixed temporal pattern (echo) model convolutive in time

– Different notes of same instrument has same time-log-frequency signature but varying in fundamental frequency (shift) model convolutive in the log-frequency axis.

(1Mørup & Scmidt, 2006)


17Morten Mørup

NMF2D Model

NMF2D Model – extension of NMFD1:

(1Smaragdis, 2004, Eggert et al. 2004, Fitzgerald et al. 2005)


18Morten Mørup

048

0 2 4 6

Time [s]

Fre

qu

ency

[H

z]

0 0.2 0.4 0.6 0.8200

400

800

1600

3200

Understanding the NMF2D Model


19Morten Mørup

The NMF2D has inherent ambiguity between the structure in W and H

To resolve this ambiguity sparsity is imposed on H to force ambiguous structure onto W


20Morten Mørup

NMF2D SNMF2D

Real music example of how imposing sparseness resolves the ambiguity between W and H


21Morten Mørup

Unique!!

Not unique

PARAFAC(Harshman & Carrol and Chang 1970)

Factor analysis(Charles Spearman ~1900)

Extension to multi channel analysis by the PARAFAC model

λb


22Morten Mørup

cPARAFAC: Sparse Non-negative Tensor Factor 2D deconvolution (SNTF2D)

(Extension of Fitzgerald et al. 2005, 2006 to form a sparse double deconvolution)


23Morten Mørup

SNTF2D algorithms


24Morten Mørup

Tchaikovsky: Violin Concert in D Major Mozart Sonate no. 16 in C Major


25Morten Mørup

Stereo recording of ”Fog is Lifting” by Carl Nielsen

6850

0.72

86

0.4209

0.90

71

Estimated Harp Estimated Flute

Stereo Channel 1 Stereo Channel 2

25.9 ms

50 Hz

22 kHz

25.9 ms

22 kHz

50 Hz

25.9 ms25.9 ms

50 Hz

22 kHz 22 kHz

50 Hz

Log-Spectrogram Channel 1 Log-Spectrogram Channel 2


26Morten Mørup

Applications Applications

– Source separation.– Music information retrieval. – Automatic music transcription (MIDI compression).– Source localization (beam forming)


27Morten Mørup

PART II: Higher Order NMF (HONMF)


28Morten Mørup

Higher Order Non-negative Matrix Factorization (HONMF)Motivation:Many of the data sets previously explored by the Tucker model are non-negative and could with good reason be decomposed under constraints of non-negativity on all modalities including the core.

Spectroscopy data(Smilde et al. 1999,2004, Andersson & Bro 1998, Nørgard & Ridder 1994)

Web mining(Sun et al., 2004)

Image Analysis(Vasilescu and Terzopoulos, 2002, Wang and Ahuja, 2003, Jian and Gong, 2005)

Semantic Differential Data(Murakami and Kroonenberg, 2003)

And many more……

TimeSpectreBatchStrength

X

pagesWebQueriesUserscountsClick

X

PixelssExpressiononsIlluminatiViewsPeopleIntensityImage

X

ScalesPiecesMusicJudgesGrade

X


29Morten Mørup

However, non-negative Tucker decompositions are notin general unique!

But - Imposing sparseness overcomes this problem!


30Morten Mørup

The Tucker Model


31Morten Mørup

Algorithms for HONMF


32Morten Mørup

Results

HONMF with sparseness, above imposed on the core canbe used for model selection -here indicating the PARAFACmodel is the appropriate model to the data.Furthermore, the HONMF gives a more part based hence easy interpretable solution than the HOSVD.


33Morten Mørup

Evaluation of uniqueness


34Morten Mørup

Data of a Flow Injection Analysis (Nørrgaard, 1994)

HONMF with sparse core and mixing captures unsupervisedthe true mixing and model order!


35Morten Mørup

Conclusion HONMF not in general unique, however when

imposing sparseness uniqueness can be achieved. Algorithms devised for LS and KL able to impose

sparseness on any combination of modalities The HONMF decompositions more part based hence

easier to interpret than other Tucker decompositions such as the HOSVD.

Imposing sparseness can work as model selection turning of excess components


36Morten Mørup

Coming soon in a MATLAB implementation near You


37Morten Mørup

ReferencesCarroll, J. D. and Chang, J. J. Analysis of individual differences in multidimensional scaling via an N-way generalization of "Eckart-Young" decomposition, Psychometrika 35 1970 283--319Eggert, J. and Korner, E. Sparse coding and NMF. In Neural Networks volume 4, pages 2529-2533, 2004Eggert, J et al Transformation-invariant representation and nmf. In Neural Networks, volume 4 , pages 535-2539, 2004Fiitzgerald, D. et al. Non-negative tensor factorization for sound source separation. In proceedings of Irish Signals and Systems Conference, 2005FitzGerald, D. and Coyle, E. C Sound source separation using shifted non.-negative tensor factorization. In ICASSP2006, 2006Fitzgerald, D et al. Shifted non-negative matrix factorization for sound source separation. In Proceedings of the IEEE conference on Statistics in Signal Processing. 2005Harshman, R. A. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modal factor analysis},UCLA Working Papers in Phonetics 16 1970 1—84

Harshman, Richard A.Harshman and Hong, Sungjin Lundy, Margaret E. Shifted factor analysis—Part I: Models and properties J. Chemometrics (17) pages 379–388, 2003Kiers, Henk A. L. and Berge, Jos M. F. ten and Bro, Rasmus PARAFAC2 - Part I. A direct fitting algorithm for the PARAFAC2 model, Journal of Chemometrics (13) nr.3-4 pages 275-294, 1999Lathauwer, Lieven De and Moor, Bart De and Vandewalle, Joos MULTILINEAR SINGULAR VALUE DECOMPOSITION.SIAM J. MATRIX ANAL. APPL.2000 (21)1253–1278Lee, D.D. and Seung, H.S. Algorithms for non-negative matrix factorization. In NIPS, pages 556-462, 2000Lee, D.D and Seung, H.S. Learning the parts of objects by non-negative matrix factorization, NATURE 1999

Murakami, Takashi and Kroonenberg, Pieter M. Three-Mode Models and Individual Differences in Semantic Differential Data, Multivariate Behavioral Research(38) no. 2 pages 247-283, 2003

Mørup, M. and Hansen, L.K.and Arnfred, S.M.Decomposing the time-frequency representation of EEG using nonnegative matrix and multi-way factorization Technical report, Institute for Mathematical Modeling, Technical University of Denmark, 2006a

Mørup, M. and Schmidt, M.N. Sparse non-negative matrix factor 2-D deconvolution. Technical report, Institute for Mathematical Modeling, Tehcnical University of Denmark, 2006b

Mørup, M and Schmidt, M.N. Non-negative Tensor Factor 2D Deconvolution for multi-channel time-frequency analysis. Technical report, Institute for Mathematical Modeling, Technical University of Denmark, 2006c

Schmidt, M.N. and Mørup, M. Non-negative matrix factor 2D deconvolution for blind single channel source separation. In ICA2006, pages 700-707, 2006d

Mørup, M. and Hansen, L.K.and Arnfred, S.M. Algorithms for Sparse Higher Order Non-negative Matrix Factorization (HONMF), Technical report, Institute for Mathematical Modeling, Technical University of Denmark, 2006e

Nørgaard, L and Ridder, C.Rank annihilation factor analysis applied to flow injection analysis with photodiode-array detection Chemometrics and Intelligent Laboratory Systems 1994 (23) 107-114

Schmidt, M.N. and Mørup, M. Sparse Non-negative Matrix Factor 2-D Deconvolution for Automatic Transcription of Polyphonic Music, Technical report, Institute for Mathematical Modelling, Tehcnical University of Denmark, 2005

Smaragdis, P. Non-negative Matrix Factor deconvolution; Extraction of multiple sound sources from monophonic inputs. International Symposium on independent Component Analysis and Blind Source Separation (ICA)W

Smilde, Age K. Smilde and Tauller, Roma and Saurina, Javier and Bro, Rasmus, Calibration methods for complex second-order data Analytica Chimica Acta 1999 237-251

Sun, Jian-Tao and Zeng, Hua-Jun and Liu, Huanand Lu Yuchang and Chen Zheng CubeSVD: a novel approach to personalized Web search WWW '05: Proceedings of the 14th international conference on World Wide Web pages 382—390, 2005

Tamara G. Kolda Multilinear operators for higher-order decompositions technical report Sandia national laboratory 2006 SAND2006-2081.

Tucker, L. R. Some mathematical notes on three-mode factor analysis Psychometrika 31 1966 279—311

Welling, M. and Weber, M. Positive tensor factorization. Pattern Recogn. Lett. 2001

Vasilescu , M. A. O. and Terzopoulos , Demetri Multilinear Analysis of Image Ensembles: TensorFaces, ECCV '02: Proceedings of the 7th European Conference on Computer Vision-Part I, 2002

Extensions of Non-negative Matrix Factorization to Higher Order data

Documents

Transcript of Extensions of Non-negative Matrix Factorization to Higher Order data