744 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND …

744 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 19, NO. 4, MAY 2011

NMF With Time–Frequency Activations toModel Nonstationary Audio Events

Romain Hennequin, Student Member, IEEE, Roland Badeau, Member, IEEE, and Bertrand David, Member, IEEE

Abstract—Real-world sounds often exhibit time-varying spec-tral shapes, as observed in the spectrogram of a harpsichord toneor that of a transition between two pronounced vowels. Whereasthe standard non-negative matrix factorization (NMF) assumesfixed spectral atoms, an extension is proposed where the temporalactivations (coefficients of the decomposition on the spectral atombasis) become frequency dependent and follow a time-varyingautoregressive moving average (ARMA) modeling. This extensioncan thus be interpreted with the help of a source/filter paradigmand is referred to as source/filter factorization. This factorizationleads to an efficient single-atom decomposition for a single audioevent with strong spectral variation (but with constant pitch). Thenew algorithm is tested on real audio data and shows promisingresults.

Index Terms—Music information retrieval (MIR), non-negativematrix factorization (NMF), unsupervised machine learning.

I. INTRODUCTION

T HE decomposition of audio signals in terms of elemen-tary atoms has been a large field of research for years.

As we usually encounter very dissimilar audio events (both intheir spectral and temporal characteristics), the decompositionon a single basis (such as the Fourier basis) is generally notsufficient to accurately explain the content of a large class ofsignals. Sparse decomposition techniques [1] use a redundantdictionary of vectors (called atoms) and try to decompose asignal using few of them (much fewer than the dimension ofthe space): thus, the signal can be accurately decomposed withfew elements. When atoms are designed to fit the signal (for in-stance harmonic atoms for musical signals [2]), these elementsbecome more meaningful, and then a supervised classificationcan be performed to cluster atoms corresponding to a real eventin the signal [3]. These methods are quite powerful and givegood results. However, since the dictionary is fixed, it must bedesigned to fit all possible signals, which is not achievable inpractice. Recently methods of data factorization were proposedto simultaneously extract atoms from the signal and provide a

Manuscript received December 08, 2009; revised April 15, 2010; acceptedJuly 16, 2010. Date of publication August 03, 2010; date of current version Feb-ruary 14, 2011. This work was supported in part by the French GIP ANR underContract ANR-06-JCJC-0027-01, DESAM, and in part by the Quaero Program,funded by OSEO, French State agency for innovation. The associate editor co-ordinating the review of this manuscript and approving it for publication wasDr. James Johnston.

The authors are with the Institut Telecom, Telecom ParisTech, CNRS LTCI,75634 Paris Cedex 13, France (e-mail: [email protected]; [email protected]; [email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TASL.2010.2062506

decomposition on these atoms. These techniques that we callfactorization make use of the natural redundancy of the signal,mimicking human cognition which utilizes this redundancy tounderstand visual and audio signals: principal component anal-ysis, independent component analysis [4], [5], sparse coding [6],or NMF [7] have been introduced both to reduce the dimension-ality and to explain the whole data set by a few meaningful ele-mentary objects. Thanks to the non-negativity constraint, NMFis able to provide a meaningful representation of the data: ap-plied to musical spectrograms it will hopefully decompose theminto elementary notes or impulses. The technique is widely usedin audio signal processing, with a number of applications suchas automatic music transcription [8]–[10] and sound source sep-aration [11]–[13].

However, the standard NMF is shown to be efficient whenthe elementary components (notes) of the analyzed sound arenearly stationary, i.e., when the envelope of the spectra of thesecomponents does not change over time. Nevertheless, in severalsituations, elementary components can be strongly nonsta-tionary. In this paper, we will focus on timbral variability, i.e.,variability of the spectral shape that we can find in plucked stringssounds or singing voice (sounds of different vowels presentgreatly dissimilar spectral shapes). However, we will not addresspitch variability that is encountered in vibrato or prosody. In caseof a noticeable spectral variability, the standard NMF will likelyneed several non-meaningful atoms to decompose a single event,whichoften leads toanecessarypostprocessing(tocluster thedif-ferent parts of a single source [14]). To overcome this drawback,Smaragdis [15] proposes a shift-invariant extension of NMF inwhich time/frequency templates are factorized from the originaldata: each atom then corresponds to a time–frequency musicalevent able to include spectral variations over time. This methodgives good results, but does not permit any variation betweendifferent occurrences of the same event (atom), its duration andspectral content evolution being fixed.

In this paper, an extension of NMF is proposed in which tem-poral activation becomes frequency dependent: it thus can be in-terpreted with the help of the classical source/filter paradigm asa source/filter factorization. Our method includes autoregressivemoving average (ARMA) filters estimated from the data, asso-ciatesa time-varyingfilterwitheachsourceand learns thesources(atoms) in a totally unsupervised way. this method presents somesimilarity with Durrieu’s work [16], [17] in which a source/filtermodel is used in a NMF framework to extract the main melody ofmusical pieces. This model permits to efficiently take the strongspectral variations of the human voice into account.

However, our approach is quite different since, in oppositionto Durrieu’s work, sources are learned, a time-varying filter is

1558-7916/$26.00 © 2010 IEEE

HENNEQUIN et al.: NMF WITH TIME–FREQUENCY ACTIVATIONS TO MODEL NONSTATIONARY AUDIO EVENTS 745

associated with each source and the class of filter we use is morestandard.

In Section II, we introduce the source/filter decomposition asan extension of NMF. In Section III, we derive an iterative al-gorithm similar to those used for NMF to compute this decom-position. In Section IV, we present experiments of source/filterdecomposition of the spectrogram of three different sounds, andcompare this decomposition to the standard NMF. Conclusionsare drawn in Section V.

II. MODEL

A. NMF and Extension

Given an non-negative matrix and an integer suchthat , NMF approximates by the product ofan non-negative matrix , and an non-negativematrix :

i.e.

(1)

This approximation is generally quantified by a cost functionto be minimized with respect to (w.r.t.) and .

A common class of cost functions is designed element-wise

where is a scalar divergence (i.e., a function such thatand if and only if ). Sev-

eral classes of such divergences have been proposed, for in-stance Bregman divergences [18] and -divergences [19], [20].In this paper, we will focus on the -divergence which includesusual measures (Euclidean distance, Kullback–Liebler diver-gence and Itakura–Saito divergence). The -divergence is de-fined for by

(2)

For and , the -divergence is defined by continuity

For these values, the -divergence, respectively, corresponds toItakura–Saito divergence and Kullback–Leibler divergence, andfor , it corresponds to the Euclidean distance.

One could notice that the singularities in the definition of the-divergence for and no longer appear in the

partial derivative w.r.t. : this partial derivative is useful for de-signing descent methods in order to minimize the cost function

. The first-order partial derivative of the -divergence w.r.t.is for all

When applied to power (squared magnitude) spectrograms,NMF factorizes data into a matrix (or basis) of frequency tem-plates which are the columns of and a matrix whose

rows are the temporal vectors of activations correspondingto each template. For a musical signal made of several notesplayed by the same instrument, it is hoped that the decomposi-tion leads to spectral templates corresponding to single notes orpercussive sounds. will then display a representation similarto a “piano-roll” (cf. [8]).

This factorization however does not yield an effective rep-resentation of a sound presenting a noticeable spectral evolu-tion. For instance, a single note of a plucked string instrumentmost of the time shows high-frequency components which de-crease faster than low-frequency components. This character-istic is not well modeled with a single frequency template. Sev-eral templates are needed which results in a less meaningful de-composition: roughly one for low-frequency partials and one forhigh-frequency partials. The meaning of each template is lost(a template no longer corresponds to a musical event such as anote).

To address this issue, we propose an extension of NMF wheretemporal activations become time/frequency activations. Thefactorization (1) becomes

(3)

where the activation coefficients are now frequency dependent.To avoid an increase of the problem dimensionality thecoefficients are further parameterized by means of ARMAmodels (Section II-B).

Equation (3) can be interpreted with the help of thesource/filter paradigm: the spectrum of each frame of thesignal results from the combination of filtered templates(sources). corresponds to the time-varying filter asso-ciated to the source . The decomposition thus benefits fromthe versatility of the source/filter model proved well suited fornumerous sound objects.

B. AutoRegressive Moving Average (ARMA) Modeling

is parameterized following the general ARMA model:

where is the normalized frequencyassociated to frequency index (as audio signalare real valued, we only consider frequencies between 0 and theNyquist frequency). are the coefficients of the MA part ofthe filter and those of the AR part. is the global gainof the filter: in order to avoid identifiability problems, the firstcoefficient of all filters is imposed to be equal to 1. For

no longer depends on and the decompositioncorresponds to a standard NMF with temporal activations .


Defining and ,time/frequency activations can be rewritten as

where is the Toeplitz matrix withand is similar to but of

dimension . MA only and AR only models areincluded by respectively taking and . It is worthnoting that is always non-negative while there existsno non-negativity constraint on or on .

The parameterized power spectrogram given in (3) thenbecomes

(4)

III. ALGORITHM

We choose a general -divergence cost function

with andand the expression of is given in (2).

The partial derivative of the cost function w.r.t. any variable( being any coefficient of or ) is

(5)

The expression of the gradient of w.r.t. a vector of severalcoefficients of or is the same, replacing the partial deriva-tive by a gradient in (5).

This leads to update rules for a multiplicative gradient descentalgorithm similar to those used in [7], [21], [15]. In such an it-erative algorithm, the update rule associated to one of the pa-rameters is obtained from the partial derivative of the cost func-tion w.r.t. this parameter, written as a difference of two positiveterms:

The update rule for is then

(6)

This rule ensures that remains non-negative, becomes constantif the partial derivative is zero and evolves in the opposite direc-tion of the partial derivative (thus in the descent direction).

A. Update of Frequency Templates

We derive multiplicative update rules for from the ex-pression of the partial derivative of the cost function with re-spect to .

The partial derivative of the parameterized spectrogram de-fined in (4) with respect to is

where is a Kronecker delta.Then, by replacing this expression in (5) with , we

obtain the partial derivative of the cost function with respect to

This derivative is written as a difference of two positive terms

and

Then the update rule of is:

(7)

B. Update of Temporal Activation Gain

In the same way as for , we derive multiplicative updaterules for from the expression of the partial derivative of thecost function with respect to .

The partial derivative of the parameterized spectrogram de-fined in (4) with respect to is

where is a Kronecker delta.Then, by substituting this expression into (5) with ,

we obtain the partial derivative of the cost function with respectto

This derivative is written as a difference of two positive terms

and


Then the update rule of is

(8)

We can notice that when and (i.e., there is nofilter), the update rules of and are the same as the onesgiven in [21] which are the standard NMF rules for a -diver-gence cost function where corresponds to frequency tem-plates and to temporal activations.

C. Update of Filters

The update rules of the coefficients of the filters are derived ina similar way, but the updates are not element-wise, but rather“vector-wise”: we derive an update rule for each and foreach .

Update of : The gradient of the parameterized spectro-gram w.r.t. is

Then, by substituting this expression into (5) with ,we obtain the gradient of the cost function w.r.t.

where

Both matrices and are positive definite undermild assumptions: these matrices are clearly positive semi-defi-nite and it can easily be shown that if there are at least dif-ferent indexes such that then is non-sin-gular (for , the assumption is very similar). This assump-tion is always true in practice as long as the frame with index

is not a null vector (i.e., with all samples equal to 0): in thisparticular case, the decomposition is trivial, and the global gains

should be equal to 0.Then, we follow the approach given in [22] and derive the

following update rule for the MA part of the filter:

(9)

As and are both nonsingular, is well de-fined and is ensured to never be zero.

Update of : The update rules of are derived in thesame way as for . The partial gradient of the parameterizedspectrogram with respect to is

Then, by substituting this expression into (5) with ,we obtain the partial gradient of the cost function with respectto

where

and

Both matrices and are positive definite under mildassumptions.

Thus, we derive the following update rule for the AR part ofthe filter:

(10)

D. Description of the Algorithm

The update rules (7)–(10) are applied successively to all thecoefficients of , all the coefficients of , all the coefficients of

, and all the coefficients of . Between the updates of each ofthese matrices (and tensors), the parameterized spectrogramis recomputed: as for the standard NMF algorithm, this recom-putation between each update is necessary to ensure the conver-gence.

Identifiability: As for the standard NMF, the decomposition(4) which minimizes the cost function is not unique. To copewith identifiability issues, we impose constraints onand :

• for all and , we impose that and (considered aspolynomials) have all their roots inside the unit circle;

• for all , we impose for some norm ;• for all and , we impose and .Thus, at the end of each iteration of our algorithm, we trans-

form and by replacing roots outside the unit circle bythe conjugate of their inverse and accordingly adapting the gain,normalize each column of , divide and by their firstcoefficient and update in order not to change by thesemodifications. All these transformations have no influence onthe values of the parameterized spectrogram.

Another choice of filters normalization has been tested: ratherthan imposing and , we can impose for all

and . Itcorresponds to a power normalization and then it is more mean-ingful.


Our algorithm is detailed in Algorithm 1. In the remainderof the paper, we will refer to this algorithm by the expression“source/filter factorization.”

Algorithm 1 Source/filter spectrogram factorization

Input:Output:

Initialize with non-negative values

Initialize with flat filters

for to do

compute

for all and docompute and

end for

compute


end for

compute


end forbring back roots of all filters inside the unit circledivide the coefficients of all filters by the first onenormalize

update appropriately

end for

E. Dimensionality

Since our algorithm is a dimensionality reduction techniquelike NMF, one should take care of the dimension of the decom-position provided. The dimension of the original data is . Inthe standard NMF with atoms, the dimension of the parame-terized spectrogram is (where

stands for the number of coefficients of matrix or tensor). With our algorithm, the dimension of the parameters is:

.Thus, one should have , i.e.,

and , so and must besmall. As in practice , the condition to be respected is

.One should notice that our decomposition allows a significant

reduction of the number of atoms needed to accurately fitthe data when the parts of the sounds present strong spectralvariations. Then the total dimension of the parameters obtained

with our decomposition remains comparable to the one obtainedwith standard NMF as will be shown in Section IV.

Besides, one should notice that a large number of the coef-ficients of the filters are useless and therefore do not need tobe retained: when the global gain of one of the filter be-comes close to zero, these coefficients ( and ) becomemeaningless and then are useless in the decomposition and canbe removed without affecting the values of the parameterizedspectrogram.

Finally, in the decomposition (4), all atoms are associated tofilters of the same order, but it is also possible to implement alarger model where filters do not have the same characteristicsfor all atoms. This larger model is not presented in this paper forreadability reasons.

F. Computational Complexity

The computational complexity of one iteration of source/filterfactorization depends on and . The computationalcomplexity of each step of the algorithm is given here.

• Computation of : operations.• Update of and operations each.• Update of operations.• Update of operations.• Normalization/stabilization: oper-

ations.The total complexity of a single iteration of the algorithm isthen operations. With our current implemen-tation in Matlab, 100 iterations of our algorithm applied to a1025 550 spectrogram (corresponding to a 6.5 s signal sam-pled at Hz with 2048-sample-long windows and75% overlap) with atoms, and lastabout 300 s (on an Intel® Core™ 2 Duo E8400 @3.00 GHz, withMatlab’s multithreaded math libraries). In comparison, 100 it-erations of standard NMF with the same spectrogram, the sameparameters ( , but and ) on the same com-puter last about 9 s: our algorithm appears to be slower thanstandard NMF. However, this comparison puts our algorithm ata disadvantage, which is designed to work with fewer atoms thanstandard NMF: in this case our algorithm deals with many moreparameters than NMF. If we compare execution times with thesame number of parameters, the difference is smaller: for thesame spectrogram, 100 iterations of our algorithm withatoms, and (i.e., with the dimensionality of astandard NMF with ) last about 60 s.

About a third of the computation time is due to the computa-tion of the roots of all filters (considered as polynomials) duringthe stabilization process. Some improvements could be made byconsidering another regularization method. The inversion of thematrices and (a bit more than 10% of the total compu-tation time) and the computation the frequency response of allfilters (slightly less than 10% of the total computation time) arealso very costly.

G. Practical Implementation and Choice of

We empirically observed the monotonic decrease of the costfunction and the convergence of the algorithm for


over a large set of tests: this decrease and this convergence areillustrated in particular cases in Section IV-D.

However, the algorithm is unstable for : some nu-merical instabilities appear while poles of the filters come closeto the unit circle. These instabilities are very common whenbecomes close to 2 (Euclidean distance); however, the strongdynamics of audio data is better fitted when becomes close to0 as stated in [21]. In order to avoid these instabilities, we limitthe modulus of poles: the monotonic decrease of the cost func-tion is no longer observed but it permits to avoid non desirablebehavior of the decomposition (very resonant order 2 filters onlypermit to fit one partial).

For the examples in the next section, we chose sincethe numerical instabilities were almost negligible and the resultswere more accurate than with (Itakura–Saito divergence).

IV. EXAMPLES

In this section, several experiments are presented to showthat our algorithm is well adapted to decompose sounds havingstrong spectral variations. All the spectrograms used in these ex-periments are power spectrograms obtained from recorded sig-nals by means of a short-time Fourier transform (STFT).

Algorithms (standard NMF and source/filter factorization)were initialized with random values (except for the filters whichwere initially flat) and were run until apparent convergence. Thealgorithms have been rerun with 100 different initializationsin order to maximize the chances to come close to a “good”minimum. Despite these different starting points, the reachedsolutions were similar in terms of qualitative aspect of thereconstructed spectrograms.

A. Didgeridoo

1) Description of the Excerpt: In this section, our algorithmis applied to a short didgeridoo excerpt. The didgeridoo is anethnic wind instrument from northern Australia. It makes a con-tinuous modulated sound produced by the vibrations of the lips.The modulations result from the mouth and throat configura-tion with the help of which the player is able to control severalresonances. Fig. 1(a) represents the spectrogram of the excerpt:the sound produced is almost harmonic (with some noise) anda strong moving resonance appears in the spectrogram. We canthus consider that this signal is composed of a single event en-compassing spectral variations, and try to decompose it witha single atom . The sampling rate of the excerpt is

Hz. We chose a 1024-sample-long Hann windowwith 75% overlap for the STFT.

2) Experiment and Results: The spectrogram of the excerptis decomposed with a standard NMF algorithm foratom and atoms, and with source/filter factorizationfor atom, with an order 3 AR modeling ( and

). Reconstructed spectrograms are respectively repre-sented in Figs. 1(b), (c), and (d).

Although the didgeridoo is played alone in the analyzed spec-trogram, the standard NMF needs many atoms to accurately de-compose the power spectrogram. With 1 atom, NMF does notaccurately represent the moving resonance [Fig. 1(b)]. With five

Fig. 1. Original power spectrogram of the extract of didgeridoo (a) and recon-structed spectrograms (b), (c), and (d). (a) Original spectrogram, (b) StandardNMF,� � �, (c) Standard NMF,� � �, (d) Source/filter factorization,� � �.

atoms, some spectral variations appear [Fig. 1(c)], but the res-onance trajectory remains a bit unclear. Besides, the signal isnot decomposed in a meaningful way (each atom is a part of thesound which has no perceptual meaning) and the dimensionalityof the parameters is large .

In opposition to the standard NMF, source/filter factorizationpermits to accurately represent the spectral variability of thesound [Fig. 1(d)] with a single atom, keeping the dimensionalitylow : the moving resonance of theoriginal sound is well tracked, and the total error is smallerthan that of the standard NMF with . In this case, thedecomposition is more efficient and relevant than the standardNMF.

B. Harpsichord

1) Description of the Excerpt: In this section, our algorithmis applied to a short harpsichord excerpt, composed of two dif-ferent notes ( and ): first, the is played alone, thenthe , and lastly, both notes are played simultaneously. Thespectrogram of the extract is represented in Fig. 2(a). As formost of plucked string instruments, high-frequency partials of aharpsichord tone decay faster than low-frequency partials. Thisphenomenon clearly occurs in the L-shaped spectrograms ofFig. 2(a). The sampling rate of the excerpt is Hz.We chose a 2048-sample-long Hann window with 75% overlapfor the STFT.

2) Experiment and Results: The spectrogram of the excerptwas decomposed with a standard NMF algorithm foratoms (1 atom per note) and atoms, and with source/filterfactorization for atoms, with an ARMA modeling (

and ). Reconstructed spectrograms are respectivelyrepresented in Figs. 2(b), (c), and (d).

The standard NMF needs several atoms per note to accuratelydecompose the L-shaped power spectrograms: with only twoatoms (one per note played), the faster decay of high frequencycontent does not appear at all [Fig. 2(b)]. With six atoms, theattenuation of high frequency partials appears [Fig. 2(c)], but


Fig. 2. Original power spectrogram of the extract of harpsichord (a) and recon-structed spectrograms (b), (c), and (d). (a) Original spectrogram, (b) StandardNMF,� � �, (c) Standard NMF,� � �, (d) Source/filter factorization,� � �.

each atom is a part of a note spectrum and has no real perceptualmeaning.

The ARMA modeling included in our algorithm leads to agood description of the overall spectrogram shape. Two atoms(one per note) are enough to accurately fit the original short timespectrum: each atom is harmonic [Fig. 3(a)] and corresponds toone note while the decay of high frequency partials is clearlywell described by the ARMA modeling [see time/frequency ac-tivations in Fig. 3(b)]. The dimensionality of thedata provided by our algorithm

remains lower than with a standard NMF with six atomsand the global error between the original

and the reconstructed spectrogram is approximately the same asthe one obtained with the standard NMF with .

Thus, the decomposition provided by source/filter factoriza-tion seems to give a more meaningful representation of the givenspectrogram than the one obtained with the standard NMF.

C. Guitar With Wah Pedal

1) Description of the Excerpt: In this section, our algorithmis used to decompose a short extract of electric guitar processedby a wah pedal. The wah pedal (or wah-wah pedal) is a popularguitar effect which consists of a resonant filter, the resonant fre-quency of which is controlled by means of a foot pedal. This ef-fect is named by emphasizing the resemblance with the humanonomatopoeia “Wah.” A single note of electric guitar processedby a moving wah pedal presents strong spectral variations andtherefore cannot be well represented by a single atom in a stan-dard NMF.

As a wah pedal is well modeled by an AR filter with two com-plex conjugates poles, we chose to decompose the extract with

and . The chosen extract represented in Fig. 4(a) iscomposed of three different notes played successively (the firstnote is played a second time at the end of the extract). Each notecan be viewed as a harmonic pattern which is filtered by a reso-nant filter, the resonant frequency of which varies between 400

Fig. 3. Source/filter decomposition (� � �� and � � �) of the powerspectrogram of the harpsichord excerpt. (a) Frequency templates. (b) Time/fre-quency activations.

and 1200 Hz: this resonance clearly appears in the power spec-trogram. The sampling rate of the excerpt is Hz.We chose a 1024-sample-long Hann window with 75% overlapfor the STFT.

2) Experiment and Results: As the analyzed sound presentsstrong spectral variations, the standard NMF needs many atomsto accurately decompose the power spectrogram. Thus oneatom no longer corresponds to one note, and the decompositiondoes not correspond to the analysis that could be performed bya human listener. Fig. 4(b) represents the power spectrogramreconstructed from the NMF of the original spectrogramwith three atoms and Fig. 4(c) with ten atoms. With threeatoms, NMF is not able to track the resonance of the wahpedal. With ten atoms, the resonance appears, but the signalis not well explained (each atom is a part of a note andthen has no perceptual meaning) and the dimensionality ishigher .

With source/filter factorization, the strong spectral variationsof each note can be accurately represented in the filter activa-


Fig. 4. Original power spectrogram of the extract of electric guitar processedby a wah pedal (a) and reconstructed spectrograms (b), (c), and (d). (a) Orig-inal spectrogram, (b) Standard NMF, � � �, (c) Standard NMF, � � ��,(d) Source/filter factorization,� � �.

Fig. 5. Source/filter decomposition (� � � and � � �) of the power spectro-gram of the wah processed guitar excerpt. (a) Frequency templates. (b) Time/fre-quency activations.

tions taking an order 2 AR modeling ( and )as shown in Fig. 4(d). Then three atoms (one for each note)

Fig. 6. Evolution of the cost function over iterations (decomposition of ex-cerpts in Sections IV-B and IV-C). (a) Harpsichord excerpt. (b) Wah guitar ex-cerpt.

are enough to correctly fit the original spectrogram. Indeed, theglobal -divergence error between the original and the recon-structed spectrogram obtained with source/filter factorization isapproximately the same as the one obtained with standard NMFwith ten atoms; this -divergence (obtained with source/filterfactorization) is also approximately half that obtained with stan-dard NMF with three atoms. Each atom is harmonic and corre-sponds to one note, and the resonance of the wah pedal clearlyappears. The dimensionality of the representation obtained withsource/filter factorization remains about half that of NMF withten atoms: . The decompositionprovided by our algorithm distinguishes a stationary spectrumrepresenting “average” guitar sounds (contained in ) from thenonstationary effect due to the wah pedal (described by time/fre-quency activations). The three frequency templates (columns of

) obtained are represented in Fig. 5(a): each template is har-monic with its own fundamental frequency; thus, it correspondsto a note (standard NMF with three atoms provides similar tem-plates). The time/frequency activations are rep-resented in Fig. 5(b): the resonance of the wah pedal appearsclearly where the notes are played. Thus, the decomposition pro-vided by our algorithm seems to give a more meaningful repre-sentation of the given spectrogram.


D. Convergence of the Algorithm

The evolution of the cost function over iterations for sourcefilter/factorization is represented in Fig. 6 with eight differentrandom initializations, for the decomposition of excerpts pre-sented in Sections IV-B (harpsichord excerpt) and IV-C (wahguitar excerpt). The value of the -divergence is representedafter each iteration. Figures show a monotonic decrease of thecost function and an apparent convergence. In Fig. 6(a), all ini-tializations lead to the same final value of the cost function andthe shape of the evolution is very similar for all initializations.On the other hand, in Fig. 6(b), all initializations do not lead tothe same value of the cost function, showing that multi-initial-ization is useful.

V. CONCLUSION AND FUTURE WORK

In this paper, we proposed a new iterative algorithm whichis an extended version of the non-negative matrix factorizationbased on a source/filter representation. We showed that this rep-resentation is particularly suitable to efficiently and meaning-fully decompose nonstationary sound objects including notice-able spectral variations.

In the future, this extended decomposition should be furtherdeveloped to deal with small pitch variations (like vibrato), forinstance using a constant-Q spectrogram like in [23], [24]. Be-sides, we plan to introduce harmonic constraints in the basisspectra following [25]–[27]. Finally, we plan to investigate theintroduction of continuity constraints between filters from oneframe to another following the approach given in [12], [26],[21].

REFERENCES

[1] S. Mallat and Z. Zhang, “Matching pursuit with time-frequency dictio-naries,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397–3415,Dec. 1993.

[2] R. Gribonval and E. Bacry, “Harmonic decomposition of audio signalswith matching pursuit,” IEEE Trans. Signal Process., vol. 51, no. 1, pp.101–111, Jan. 2003.

[3] P. Leveau, E. Vincent, G. Richard, and L. Daudet, “Instrument-spe-cific harmonic atoms for mid-level music representation,” IEEE Trans.Audio, Speech, Lang. Process., vol. 16, no. 1, pp. 116–128, Jan. 2008.

[4] P. Comon, “Independent component analysis, a new concept,” SignalProcess. Special Iss. Higher-Order Statist., vol. 36, pp. 287–314, Apr.1994.

[5] S. Makino, S. Araki, R. Mukai, and H. Sawada, “Audio source sepa-ration based on independent component analysis,” in Proc. Int. Symp.Circuits Syst., Vancouver, BC, Canada, May 2004, pp. V-668–V-671.

[6] S. A. Abdallah and M. D. Plumbley, “Unsupervised analysis of poly-phonic music by sparse coding,” IEEE Trans. Neural Netw., vol. 17,no. 1, pp. 179–196, Jan. 2006.

[7] D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791,Oct. 1999.

[8] P. Smaragdis and J. C. Brown, “Non-negative matrix factorization forpolyphonic music transcription,” in Proc. Workshop Applicat. SignalProcess. Audio Acoust., New Paltz, NY, Oct. 2003, pp. 177–180.

[9] J. Paulus and T. Virtanen, “Drum transcription with non-negativespectrogram factorization,” in Proc. Eur. Signal Process. Conf. (EU-SIPCO), Antalya, Turkey, Sep. 2005.

[10] N. Bertin, R. Badeau, and G. Richard, “Blind signal decompositionsfor automatic transcription of polyphonic music: NMF and K-SVD onthe benchmark,” in Proc. Int. Conf. Acoust., Speech, Signal Process.(ICASSP), Honolulu, HI, Apr. 2007, vol. 1, pp. I-65–I-68.

[11] A. Cichocki, R. Zdunek, and S. I. Amari, “New algorithms for nonneg-ative matrix factorization in applications to blind source separation,” inProc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Toulouse,France, May 2006, vol. 5, pp. 621–625.

[12] T. Virtanen, “Monaural sound source separation by nonnegative matrixfactorization with temporal continuity,” IEEE Trans. Audio, Speech,Lang. Process., vol. 15, no. 3, pp. 1066–1074, Mar. 2007.

[13] A. Ozerov and C. Févotte, “Multichannel nonnegative matrix factoriza-tion in convolutive mixtures for audio source separation,” IEEE Trans.Audio, Speech, Lang. Process., vol. 18, no. 3, pp. 550–563, Mar. 2010.

[14] D. FitzGerald, M. Cranitch, and E. Coyle, “Non-negative tensor factor-ization for sound source separation,” in Proc. Irish Signals Syst. Conf.,Dublin, Ireland, Sep. 2005.

[15] P. Smaragdis, “Non-negative matrix factor deconvolution; extraction ofmultiple sound sources from monophonic inputs,” in Proc. Conf. Ind.Compon. Anal. Blind Source Separation (ICA), Grenada, Spain, Sep.2004, pp. 494–499.

[16] J.-L. Durrieu, G. Richard, and B. David, “An iterative approach tomonaural musical mixture de-soloing,” in Proc. Int. Conf. Acoust.,Speech, Signal Process. (ICASSP), Taipei, Taiwan, Apr. 2009, pp.105–108.

[17] J.-L. Durrieu, G. Richard, and B. David, “Singer melody extractionin polyphonic signals using source separation methods,” in Proc. Int.Conf. Acoust., Speech, Signal Process. (ICASSP), Las Vegas, NV, Aug.2008, pp. 169–172.

[18] I. Dhillon and S. Sra, “Generalized nonnegative matrix approximationswith Bregman divergences,” in Proc. Neural Inf. Process. Syst. Conf.(NIPS), Y. Weiss, B. Schölkopf, and J. Platt, Eds., Cambridge, MA,2006, pp. 283–290, MIT Press.

[19] R. Kompass, “A generalized divergence measure for nonnegative ma-trix factorization,” Neural Comput., vol. 19, no. 3, pp. 780–791, Mar.2007.

[20] A. Cichocki, R. Zdunek, and S. Amari, “Csiszars divergences fornon-negative matrix factorization: Family of new algorithms,” in Proc.Conf. Ind. Compon. Anal. Blind Source Separation (ICA), Charleston,SC, Mar. 2006, pp. 32–39.

[21] C. Févotte, N. Bertin, and J.-L. Durrieu, “Nonnegative matrix factor-ization with the Itakura–Saito divergence. With application to musicanalysis,” Neural Comput., vol. 11, no. 3, pp. 793–830, Mar. 2009.

[22] R. Badeau and B. David, “Weighted maximum likelihood autore-gressive and moving average spectrum modeling,” in Proc. Int. Conf.Acoust., Speech, Signal Process. (ICASSP), Las Vegas, NV, Mar. 2008,pp. 3761–3764.

[23] P. Smaragdis, B. Raj, and M. Shashanka, “Sparse and shift-invariantfeature extraction from non-negative data,” in Proc. Int. Conf. Acoust.,Speech, Signal Process. (ICASSP), Las Vegas, NV, Mar. 2008, pp.2069–2072.

[24] M. Schmidt and M. Mørup, “Nonnegative matrix factor 2-D decon-volution for blind single channel source separation,” in Proc. Conf.Ind. Compon. Anal. Blind Source Separation (ICA), Paris, France,Apr. 2006, 3889 of Lecture Notes in Computer Science (LNCS), pp.700–707, Springer.

[25] E. Vincent, N. Bertin, and R. Badeau, “Harmonic and inharmonic non-negative matrix factorization for polyphonic pitch transcription,” inProc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Las Vegas,NV, Mar. 2008, pp. 109–112.

[26] N. Bertin, R. Badeau, and E. Vincent, “Enforcing harmonicity andsmoothness in Bayesian non-negative matrix factorization applied topolyphonic music transcription,” IEEE Trans. Audio, Speech, Lang.Process., vol. 18, no. 3, pp. 538–549, Mar. 2010.

[27] E. Vincent, N. Bertin, and R. Badeau, “Adaptive harmonic spectral de-composition for multiple pitch estimation,” IEEE Trans. Audio, Speech,Lang. Process., vol. 18, no. 3, pp. 528–537, Mar. 2010.

Romain Hennequin (S’10) was born in ChatenayMalabry, France, on February 15, 1984. He receivedthe State Engineering degree from the École Poly-technique, Palaiseau, France, in 2008, the StateEngineering degree from Telecom ParisTech, Paris,France, in 2008, and the M.Sc. degree in acoustics,signal processing, and computer science applied tomusic from the Université Pierre et Marie Curie ParisVI, Paris, France, in 2008. He is currently pursuingthe Ph.D. degree in the Department of Signal andImage Processing, Telecom ParisTech.

His research interests include music information retrieval and musical signalprocessing.


Roland Badeau (M’02) was born in Marseilles,France, on August 28, 1976. He received the StateEngineering degree from the École Polytechnique,Palaiseau, France, in 1999, the State Engineeringdegree from the École Nationale Supérieure desTélécommunications (ENST), Paris, France, in2001, the M.Sc. degree in applied mathematicsfrom the École Normale Supérieure (ENS), Cachan,France, in 2001, and the Ph.D. degree from ENST in2005, in the field of signal processing.

In 2001, he joined the Department of Signal andImage Processing, Telecom ParisTech (ENST), as an Assistant Professor, wherehe became Associate Professor in 2005. His research interests include high res-olution methods, adaptive subspace algorithms, audio signal processing, andmusic information retrieval.

Bertrand David (M’06) was born in Paris, France,on March 12, 1967. He received the M.Sc. degreefrom the University of Paris-Sud, Paris, France, in1991, the Agrégation, a competitive French exami-nation for the recruitment of teachers, in the field ofapplied physics, from the École Normale Supérieure(ENS), Cachan, France, and the Ph.D. degree fromthe University of Paris 6, Paris, France, in 1999, inthe fields of musical acoustics and signal processingof musical signals.

He formerly taught in a graduate school in elec-trical engineering, computer science, and communication. He also carried outindustrial projects aiming at embarking a low-complexity sound synthesizer.Since September 2001, he has worked as an Associate Professor with the Signaland Image Processing Department, Telecom ParisTech (ENST). His researchinterests include parametric methods for the analysis/synthesis of musical andmechanical signals, music information retrieval, and musical acoustics.

744 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND …

Documents

Transcript of 744 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND …