1

Integer Matrix Factorization and ItsApplication

Matthew M. Lin, Bo Dong, Moody T. Chu

Abstract—Matrix factorization has been of fundamental importance in modern sciences and technology. This workinvestigates the notion of factorization with entries restricted to integers or binaries, , where the “integer” couldbe either the regular ordinal integers or just some nominal labels. Being discrete in nature, such a factorizationor approximation cannot be accomplished by conventional techniques. Built upon a basic scheme of rank oneapproximation, an approach that recursively splits (approximates) the underlying matrix into a sum of rank onematrices with discrete entries is proposed. Various computational issues involved in this kind of factorization, whichmust take into account the metric being used for measurement, are addressed. The mechanism presented in thispaper can handle multiple types of data. For application purposes, the discussion emphasizes mainly on binary-integer factorizations. But the notion is readily generalizable with slight modifications to, for example, integer-integerfactorizations. Applications to cluster analysis and pattern discovery are demonstrated through some real-worlddata. Of particular interest is the result on the ordering of rank one approximations which, in a remote sense, isthe analogy for discrete data to the ordering of singular values for continuous data. As such, a truncated low rankfactorization (for discrete data) analogous to the truncated singular value decomposition is also obtained.

Index Terms—matrix factorization, clustering, classification, pattern discovery

F

1 INTRODUCTION

MATRIX factorization has been of funda-mental importance in modern sciences

and technology. In the past decades, theoryand numerical methods for matrix factorizationhave been extensively studied. In an interest-ing appraisal of the two purposes of matrixfactorization, Hubert, Meulman, and Heiser [1]commented that, on one hand,

“Matrix factorization in the context ofnumerical linear algebra (NLA) gener-ally serves the purpose of rephrasingthrough a series of easier subproblemsa task that may be relatively difficult tosolve in its original form.”

Thus, the factorization of a square matrix A asthe product of a lower triangular matrix L and

• Matthew M. Lin is with Department of Mathematics, NorthCarolina State University, Raleigh, NC 27695-8205.Email: mlin@ncsu.edu.This research was supported in part by the National ScienceFoundation under grant DMS-0505880.

• Bo Dong is with Department of mathematics, Dalian Univer-sity of Technology, Dalian, Liaoning, 116024.Email: dongbodlut@gmail.comThis work is partially supportedby Chinese Scholarship Council and DLUT(Dalian Universityof Technology) under grands 3004-893327 and 3004-842321.

• Moody T. Chu is with Department of Mathematics, NorthCarolina State University, Raleigh, NC 27695-8205.Email: chu@math.ncsu.edu.This research was supported in part by the National Sci-ence Foundation under grants CCR-0204157 and DMS-0505880, and NIH Roadmap for Medical Research grant 1 P20HG003900-01.

an upper triangular matrix U , for example, “hasno real importance in and of itself other than asa computationally convenient means for obtain-ing a solution to the original linear system.” Onthe other hand, regarding a matrix A ∈ Rm×n

as a data matrix containing numerical observa-tions on m objects (subjects) over n attributes(variables), or possibly B ∈ Rn×n representing,say, the correlation between columns of A, theymade the point that

“The major purpose of a matrix fac-torization in this context is to obtainsome form of lower-rank (and there-fore simplified) approximation to A (orpossibly to B) for understanding thestructure of the data matrix, particu-larly the relationship within the objectsand within the attributes, and how theobjects relate to the attributes.”

Thus, in the context of applied statis-tics/psychometrics (AS/P), “matrix factoriza-tions are again directed toward the issue of sim-plicity, but now the actual components makingup a factorization are of prime concern andnot solely as a mechanism for solving anotherproblem.” We prefer to be able to “interpret thematrix factorization geometrically and actuallypresent the results spatially through coordinatesobtained from the components of the factoriza-tion”. In such a factorization can be achieved,we might better understand the structure orretrieve the information hidden in the original

2

data matrix.There are at least three different ways to for-

mulate matrix factorization, depending on thetypes of constraints imposed on the factors. Themost conventional notion is that the matrix Ais equal to the product of two or more factors.In this case, the factorization can be consideredas the sum of rank-one matrices. Indeed, aremarkably simple rank-one reduction formuladue to Wedderburn is able to unify severalfundamental factorization processes under oneframework. It was shown in the review paperby Chu, Funderlic and Golub [2] that almost allmatrix decompositions known in NLA could beobtained via this mechanism. Perhaps not notedto the NLA community, the conception of rankreduction, including the now well recognizedsingular value decomposition, has been preva-lent for many decades in the AS/P community.See, for example, discussions in [3], [4], [5], [6],[7], [8] and also the time line for the appearanceof the rank reduction results in [1]. The datainvolved in this type of calculation are mostlyfrom the real or complex field of numbers.

There are situations in applications, includingtext mining [9], [10], image articulation [11], [12],bioinformatics [13],micro-array analysis [14] ,cheminformatics [15], [16], air pollution research[16], and spectral data analysis [17], where thedata to be analyzed is nonnegative. As such, thelow rank data are also required to be comprisedof nonnegative values only in order to avoidcontradicting physical realities. This demandbrings forth the notion of the so called nonneg-ative matrix factorization (NMF). That is, for agiven nonnegative matrix A ∈ Rm×n and a fixedinteger k, we are interested in the minimizationproblem [15]

minU∈Rm×k

+ ,V ∈Rk×n+

‖A− UV ‖F , (1)

where R+ stands for the half-arry of nonnega-tive real numbers. Note that the equality A =UV may never be true, but the two matricesU, V are still termed as low rank “factors” of Ain the literature. Classical tools cannot guaran-tee to maintain the nonnegativity. Considerableresearch efforts have benn devoted to developspecial techniques for NMF. See, for example,the multiplicative update algorithm by Lee andSeung [11], [18], the gradient methods [19], [20],and alternating least square methods [21], [22],[23], [24], [16]. Additional discussions can befound in [25], [26], [27], [28], [29], [30], and themany references contained therein.

The two types of factorizations mentionedabove and their approaches are of distinct na-ture. The data they intend to handle, however,

share a common feature in that they are froma continuum domain. To motivate the discretetype of data considered in this paper, envisagethe scenario of switch manufacturing in thetelecommunications industry [31]. Suppose thateach switch cabinet consists of n slots whichcan be fitted with only one type of specifiedboard options. Assume that there are ti differentboard types for each slot. In all, there are

∏ni=1 ti

different models. It might be desirable to firstbuild a few basic models so that, correspondingto different custom orders, we might meet a spe-cific configuration more efficiently. The questionis how many and what basic models shouldbe built. One plausible approach is to “learn”from past experiences, that is, we try to obtainthe basic model information from the matrixof past sales. Labeling each board type in slotby an integer (or any token) while the sameinteger for different slots (columns) may referto different attributes, the data representing pastm customer orders form an integer matrix ofsize m× n. Let it be stressed again that, in thissetting, entries in the data matrix are of discretevalues and that these values may or may notreflect any ordinal scale. A factorization of sucha data matrix therefore must be subject to certainspecifically defined arithmetic rules, which isthe main thrust of this paper.

Without causing ambiguity, let Z denote eitherthe conventional set (or subset) of integers whenordinal scale matters, or the set of all possibletokens which are not totally preordered, for thesystem under consideration. For the simplicityof discussion, we shall assume that the entriesof the given matrix A are from the same setZ. In practice, different columns of A may becomposed of elements from different subsets ofZ. Even the same element in Z may have differ-ent meaning in different columns. If a compari-son between two observations (rows) is needed,entries at different locations in the rows mightneed to be measured differently. Before we canperform the factorization, a properly definedmetric for the row vectors, such as the metricfor the product space (of individual entries),therefore must be in place, which then inducesa metric d to measure the “nearness” betweentwo matrices. With this in mind, we define ourinteger matrix factorization (IMF) as follows.

(IMF) Given an integer matrix A ∈Zm×n, an induced metric d, and a pos-itive integer1 k < min{m,n}, find abinary matrix U of size m× k withmutually orthogonal columns and V ∈

1. The determination of such a rank k is itself an importantquestion which thus far has no satisfactory answer yet.

3

Zk×n so that the functional

f(U, V ) := d(A,UV ), (2)

is minimized.Each entry aij in A denotes, in a broad sense,

the score obtained by entity i on variable j;the value of ui`, being dichotomous, indicateswhether entity i is influenced by “factor” `; andv`j marks the attribute of variable j to the factor`. Take the sale record of a certain electronicdevice as the matrix A, for example. Then rowsof A denote the specifications by m customerswith respect to the n different slots in the device;rows of V represent the basis models to bebuilt; and the mutual exclusion of columns ofU implies that each customer order correspondsto exactly one possible basic model. The factorsin this case represent some abstract concepts bywhich the customer orders are divided into kclusters. Clearly, this is a typical classificationproblem written in factorization form. We alsosee that the number k plays an important rolein the low rank approximation. In practice, weprefer to have as few factors as possible whilekeeping the objective value f(U, V ) low.

In the special case when the set Z is madeof Z2 := {0, 1}, i.e., binaries, a nonorthogonalbinary decomposition by recursive partitioninghas recently been proposed in [32]. The recur-sion continues until either the Hamming dis-tance within a cluster is less than a preset thresh-old or all members within a cluster have reachedhomogeneity. The code PROXIMUS, written inthe C language, takes into account efficientdata structure, storage, and movement withinthe system. In this article, we generalize thatmethod to handle integer matrices in general.

Needless, the metric d for measuring nearnessor similarity plays critical roles. We need tocarefully discern whether variables are ordinal(totally ordered) or nominal (unordered) to thecategories. For the former, we also need to de-cide whether the spacing between the (catego-rial) values is the same across all levels of thevariables or not. If the values are not equallyspaced, using integers to represent these val-ues has the advantage of numerically scalingin the spacing disparities. The commonly usedHamming distance or Euclidean distance canbe refined to reflect the different situations. Welimit our discussion in this paper to these twometrics, although our concept can easily be gen-eralized to other means of measurement.

This presentation contains several interestingapplications of IMF, but it is the details on thecomputational aspects of IMF that deserve ourattention. To make our approach effectual, weneed to analyze several essential components

separately first before putting them to work inconjunction. We thus organize this paper as fol-lows: We begin in Section 2 with a basic schemethat does alternating direction search for rank-one approximation. We show that in each alter-nating direction an optimal solution is attainablein closed form. This scheme serves as a buildingblock by which we construct in Section 3 adivide-and-conquer strategy that ultimately fac-torize the given matrix A. The method graduallybreaks down the data matrix to submatrices ina recursive way. Although it seems that we aredealing with numerals, our method actually canhandle categorical variables. A suitable crite-rion for assessing the validity of the proposedsplitting, therefore, must also be addressed. Aprototype code is sketched in Algorithm 3 todemonstrate how these components should beassembled together. In practice, the code needsto be modified to reflect the different natureof the underlying data and the metric, but thegeneral concept is the same. We have experi-mented our method with quite a few real-worlddata sets. We select to report only a few inSection 4. Our numerical experiences seem tosuggest that our approach is easy to program,converges considerably fast even with a largedata matrix, and can handle multiple types ofdata.

2 BASIC RANK-ONE APPROXIMATION

Our IMF algorithm is built upon a series ofrank-one approximations. In this section we de-scribe how such an approximation is achieved.Suppose that, after taking into account the datatype, a metric d is already given. The generalidea for computing the rank-one approximationis to employ the notion of alternating directioniterations (ADI) as follows:

Algorithm 1 Rank-one approximation with gen-eral metric d

Given matrix A ∈ Zm×n, and initial vector v ∈Z1×n.return Vectors u ∈ Zm×1

2 and v ∈ Z1×n thatminimize d(A,uv).repeat

For the fixed v, find u ∈ Zm×12 to minimize

d(A,uv)For the fixed u, find v ∈ Z1×n to minimized(A,uv)

until convergence

Obviously, the iterations can also start with aninitial vector u ∈ Zm×1

2 . Note that Algorithm 1 isa descent method and that an optimal solutionat each step can be found since there are only

4

finitely many choices for u and v. Thus the realissue at stake is to effectively find the minimizerin each step. We analyze the computation withregard to two commonly used metrics.

2.1 Approximation with Hamming metric

The Hamming distance between two arrays of thesame length is defined as the minimum numberof substitutions required to transform one arrayinto the other. Here, we apply the same notion tomeasure the number of different entries betweentwo matrices.

With respect to the Hamming metric, the fol-lowing two results show that for each givenvector v (or u), the optimal solution u (or v)for minimizing d(A,uv) can be found in closedform. For convenience, we introduce three op-erators, match(x,y), zeros(x) and mode(x), tocount the number of matching elements be-tween vectors x and y, the number of zeroentries in the vector x, and the most frequententry of the vector x, respectively. Also, weadopt the notation A(i, :) for the ith row of amatrix A.

Theorem 1: Given A ∈ Zm×n and v ∈ Z1×n,then among all possible u ∈ Zm×1

2 the Hammingmetric d(A,uv) is minimized at u∗ whose ithentry u∗i , i = 1, . . . ,m, is defined by

u∗i :={

1, if match(A(i, :),v) ≥ zeros(A(i, :)),0, otherwise.

(3)

Proof: For any u ∈ Zm×12 , observe that

d(A,uv) =m∑

i=1

d(A(i, :), uiv). (4)

Thus minimizing d(A,uv) is equivalent to min-imizing each individual term d(A(i, :), uiv) fori = 1, 2, · · · ,m. From the relationships thatmatch(A(i, :),v) = n − d(A(i, :),v) whereaszeros(A(i, :)) = n − d(A(i, :),0), it is clear thatthe choice in (3) is optimal.

Theorem 2: Given A ∈ Zm×n and u ∈ Zm×12

with u 6= 0, then among all v ∈ Z1×n theHamming metric d(A,uv) is minimized at v∗

whose jth entry v∗j , j = 1, . . . , n, is defined by

v∗j := mode(A2(:, j)), (5)

where A2 is the submatrix composed of allA(i, :) whose corresponding ui is 1.

Proof: With permutations if necessary, wemay assume without loss of generality that

u =[

u1

u2

]

with u1 = [0, · · · , 0]>1×s and u2 =[1, · · · , 1]>1×(m−s). Divide A accordingly intotwo parts

A =[A1

A2

]with A1 ∈ Zs×n and A2 ∈ Z(m−s)×n. For anyvector v ∈ Z1×n, observe that

d(A,uv)= d(A1,u1v) + d(A2,u2v)= d(A1,0v) +

∑nj=1

∑m−si=1 d(A2(i, j), vj),

Minimizing d(A,uv) is equivalent to minimiz-ing

∑m−si=1 d(A2(i, j), vj) for each j = 1, . . . , n.

The optimal choice for vj is certainly the mostfrequently occuring entry in the jth column ofA2.

It is worth noting that in both theorems above,the optimal solutions may not be unique if thereis a tie. For instance, we may assign u∗i = 0 in-stead of 1 when match(A(i, :),v) = zeros(A(i, :)). It will not affect the optimal objective valued(A,u∗v) in Theorem 1. Likewise, there will bemultiple choices for v∗j in Theorem 2 when thereare more than one most frequent entries in thejth column of A2.

Equipped with the formulas described in The-orems 1 and 2, the rank-one approximation withthe Hamming metric as the measurement ofnearness is now taking shape in the form ofAlgorithm 2. We name this algorithm VOTEbecause the choice of u (and v if A is a binarymatrix) is a matter of majority rule between thedichotomy of zeros and matches. A functionsimilar to the command find in Matlab, whereind = find(x) locates all nonzero elements ofarray x and returns the linear indices of thoseelements in the vector ind, proves handy in thecoding.

In the event that A is a binary matrix, we nowargue that the definition (5) for v is equivalentto the formula (3), if the roles of u and v andof rows and columns are interchanged, respec-tively. More precisely, we make the followingobservation whose proof is essentially the sameas that for Theorem 1.

Lemma 1: Suppose A ∈ Zm×n2 . Given u ∈

Zm×12 , Then among all v ∈ Z1×n

2 the Hammingmetric d(A,uv) is minimized at v∗ whose jthentry v∗j is given by

v∗j ={

1, if match(A(:, j),u) ≥ zeros(A(:, j)),0, otherwise.

(6)

Another view of the equivalence betweenTheorem 2 and Lemma 1 is through the rela-

5

Algorithm 2 Rank-one factorization with Ham-ming metric: [u,v] = VOTE(A)

Given matrix A ∈ Zm×n.return Vectors u ∈ Zm×1

2 and v ∈ Z1×n suchthat d(A,uv) is minimized.v ← randomly selected from one row of ma-trix Az← numbers of zeros in A per rowrepeat

vold← vm ← numbers of matches between v andeach row of Aif mi ≥ zi thenui ← 1

elseui ← 0

end ifif u = 0 then

v← 1else

ind← find(u)vi ← mode(A(ind, i))

end ifuntil vold = v

tionships

match(A(:, j),u)= match(A1(:, j),0) + match(A2(:, j),1),zeros(A(:, j))= zeros(A1(:, j)) + zeros(A2(:, j)),

whereas it is obviously that match(A1(:, j),0) =zeros(A1(:, j). Thus the choice of mode(A2(:, j))in Theorem 2 is again a majority rule betweenmatch(A2(:, j),1) and zeros(A2(:, j)).

Based on Lemma 1, we claim that the methodVOTE applied to binary data is theoreticallyequivalent to the existing code PROXIMUS whichhas already been demonstrated to have a widerange of important applications [32]. To seethe equivalence, recall that the basic rank-oneapproximation in PROXIMUS is based on themechanism of minimizing the Euclidean dis-tance ‖A− uv‖2F . Upon rewriting

‖A− uv‖2F = ‖A‖2F − 2u>Av> + ‖u‖22‖v‖22,

we see that the functional

f(u,v) = 2u>Av> − ‖u‖22‖v‖22 (7)

needs to be maximized. It is not difficult toconclude (see the proof for Theorem 4) that,given a binary vector v, the optimal entries umust be defined by

ui :={

1, if 2(Av>)i − ‖v‖22 ≥ 0,0, otherwise , (8)

whereas, given a binary vector u, the optimalentries v is given by

vi :={

1, if 2(u>A)i − ‖u‖22 ≥ 0,0, otherwise . (9)

Both (8) and (9) happen to be the rules adoptedby PROXIMUS. On the other hand, observe that

match(A(i, :),v)− zeros(A(i, :))= {n− (v −A(i, :))(v −A(i, :))>}−{n−A(i, :)A(i, :)>}= −vv> + 2A(i, :)v>

= 2(Av>)i − ‖v‖22,

(10)

andmatch(A(:, i),u)− zeros(A(:, i))= {m− (u−A(:, i))>(u−A(:, i))}−{m−A(:, i)>A(:, i)}= −u>u + 2u>A(i, :)= 2(u>A)i − ‖u‖22,

(11)

implying that VOTE and PROXIMUS are em-ploying exactly the same basic rank-one ap-proximation in each sweep when dealing withbinary data. However, note that our methodis computationally simpler than PROXIMUS be-cause VOTE avoids matrix-vector multiplica-tions needed in (8) and (9). Furthermore, ourapproach VOTE can handle more general poly-chotomous data.

2.2 Approximation with Euclidean metricIn many applications, it is essential to differ-entiate the true discrepancies among variablevalues. That is, the values that a variable takessignifies levels of priority, weight, or worth. Ifthe data are somehow represented in the Eu-clidean space, then the real distance betweentwo points is meaningful and makes a differencein the interpretation. Under such a setting, wediscuss in this section the rank-one approxima-tion when the Euclidean metric is used as themeasurement for nearness. For demonstrate, Zdenotes all regular integers in this section.

Given v, even if it is not binary, the definition(8) for u remains to be the optimizer. For com-pletion, we restate the following theorem, butprovide a slightly different proof.

Theorem 3: Given A ∈ Zm×n and v ∈ Z1×n,then among all u ∈ Zm×1

2 the minimal value of‖A − uv‖2F is attained at u∗ whose ith entry isdefined by

ui ={

1, if 2(Av>)i − ‖v‖22 ≥ 0,0, otherwise. (12)

Proof: Since

‖A− uv‖2F =m∑

i=1

‖A(i, :)− uiv‖22,

6

it is clear that in order to minimize each individ-ual term in the summation we should choose

ui :={

1, if ‖A(i, :)‖22 ≥ ‖A(i, :)− v‖22,0, otherwise ,

which is equivalent to (12).Let round(x) denote the operator that rounds

every entry of x to its nearest integer. Given u,the next result nicely generalizes the selectioncriterion in PROXIMUS for general integer vectorv.

Theorem 4: Given A ∈ Zm×n and u ∈ Zm×12

with u 6= 0, then among all v ∈ Z1×n, theminimal value of ‖A − uv‖2F is attained at v∗

defined by

v∗ := round(

u>A‖u‖2

). (13)

Proof: We rewrite (7) as

2u>Av> − ‖u‖22‖v‖22=∑

i

[−‖u‖22

(vi − (u>A)i

‖u‖2

)2

+(

(u>A)i

‖u‖2

)2].

(14)

It is now clear that the only option for v tominimize ‖A−uv‖2F while keeping v an integervector is the definition (13).

It takes only some slight modifications in thecode VOTE to reflect the Euclidean metric, so weshall skip the particulars here. Also, in the eventthat Z is restricted to only a subset of integers,the operator round might return a value outsideZ. Instead, we can use (14) to define a propervalue vi ∈ Z that is nearest to (u>A)i

‖u‖2 .We conclude this section with two remarks.

Firstly, in the event that u = 0, we can simplyassign v arbitrarily, say, v = 1, without affectingthe iteration. Secondly and most importantly,the reason we insist on restricting u to Zm×1

2

is only for the purpose of establishing mutu-ally exclusive clusters, as we shall see in thenext section. Our emphasis in this paper is onthe binary-integer matrix factorization. How-ever, the proof of Theorem 4 certainly can beapplied to u if u is to be an integer vector.In this case, the optimal integer vector u forminimizing ‖A − uv‖2F with a fixed v is givenby

u∗ := round(Av>

‖v‖2

). (15)

If we continue building a sequence of integerrank-one approximations to A in the same wayas the IMF algorithm to be described in the nextsection, the term “integer matrix factorization”is then justified.

3 ALGORITHMS OF IMFWe now describe the procedure for constructingthe IMF of a given matrix A. The main ideais to compose A via a sequence of rank-oneapproximations, but not in the traditional senseof additive manner as we shall explain in thissection. Three essential issues must be resolvedbefore we can move this idea forward. Theseare — how to assess the quality of an approxi-mation, how to recursively deflating the matrix,and how to determine the optimal rank. Weaddress each issue separately in the sequel.

For the convenience of reference, after a rank-one approximation uv for A is established, wesay collectively that those rows A(i, :) of Acorresponding to ui = 1 are active and call vthe pattern vector or representative of these activerows.

3.1 Communal ruleAlthough in each sweep of our Algorithm 1we are able to find the “global” minimizer ineach direction, the product of the one-sidedminimizers does not necessarily give rise to theglobal minimizer for two-sided objective func-tion. The quality of a rank-one approximationA ≈ uv depends highly on the initial value. Thisdependence limits the ADI to find only localinterpretable patterns. It is therefore reasonableto set up a checking criterion to decide whetheractive rows are adequately represented by thepattern vector v.

To put it differently, the assignment of ui foreach i = 1, . . . ,m thus far is based solely on acomparison of A(i, :) individually with v. Theactive rows themselves have not been juxta-posed with each other. Qualifying A(i, :) into theactive group by self-justification has the dangerthat the differences between the pattern vectorand corresponding active rows of A might varyimmensely and, thus, the homogeneity withinthe collective of all active rows cannot be war-ranted. To exclude outliers, the conventional ap-proach by fixing a neighborhood around v is noteffective because v is not necessarily the meanand every single active row, though preliminar-ily counted as active, could be far away from v.Instead, we propose to check the communalityof the active rows more dynamically by using asimple statistical rule.

Firstly, let the vector r denote the collectionof “distances” between active rows of A and v,that is,

rj = d(A(ij , :),v), (16)

whenever uij= 1. Secondly, we calculate the

mean µ and standard deviation σ of r. For a

7

fixed β > 0, we adopt a communal rule thatwhenever

|rj − µ| > βσ, (17)

the membership of A(ij , :) is rejected from thecluster by resetting uij

= 0. It is possible thatall active rows will be rejected based on (17),leading to the so called cluster death and causingthe algorithm to break down. To avoid such asituation, we keep ui`

= 1 whenever r` = min r.The parameter β in (17) serves as a threshold

which can affect the ultimate partitions of datamatrix. The larger the threshold β is, the moreinclusive the cluster become, and the less thenumber of partitions of A will be. For data withlarge noises, for instance, increasing β lead tofewer representatives for the rows of A which,in turn, might rid of some of the unwantedsubstances in the data. On the other hand, ifpairwise matching is more important than pair-wise distance, we might decrease the β valueto increase the uniformity. In our algorithm, theuser can adjust the parameter value β accordingto the need.

3.2 Recursive decompositionThe purpose of the rank-one approximation is toseparate rows of A into two mutually exclusiveclusters according to innate attributes of eachrow while trying to find a representative forthe active rows. The motive of a filtering bythe communal rule is to further refine the activerows to form a tighter cluster. These objectivesare repeatedly checked through the followingdivide-and-conquer procedure:

Step 1. Given a matrix A, assess a possible clusteru of active rows and its representative v.

Step 2. Based on available u, deflate the matrixA into two submatrices composed of allactive and inactive rows, respectively.

Step 3. Recursively apply Step 1 to each submatrixin Step 2 until no more splitting is possibleor a predesignated number of iteration isreached.

Step 4. Whenever an action in Step 3 is termi-nated, record the corresponding u and vas an extra column and row in the matrixU and V , respectively.

Algorithm 3 sketches how a simple integerfactorization A ≈ UV can be executed throughVOTE without controlling the size p in the finaloutput matrices U ∈ Zm×p

2 and V ∈ Zp×n.Strictly speaking, this is not the IMF we havedefined in (2) because the ultimate p obtained byAlgorithm 3 could be exceedingly large, but itdoes represent an exhaustive search for factors.In the extreme case, each cluster contains only

one member, that is, we have U = I and V = A,which of course is of little interest.

Algorithm 3 Integer matrix factorization byVOTE: [U, V ] = IMFVOTE(A, active, β)

GivenA = matrix in Zm×n to be decomposed,active = array of indices identifying

submatrices of A being analyzed,β = threshold for communal rule.

return Matrices U ∈ Zm×p2 and V ∈ Zp×n for

some integer p such that A ≈ UV .A← A(active, :)u,v ← VOTE(A)% Check the communalityuactive← find(u)for each row A(i, :) of A(uactive, :) do

if A(i, :) does not meet communal rule thenuuactivei ← 0

end ifend for% Keep decomposing matrixZeroCheck ← find(u = 0)if ZeroCheck is not empty thenactive1← active(find(u = 1))if actove1 is not empty then

IMFVOTE(A, active1, β)end ifactive0← active(ZeroCheck)IMFVOTE(A, active0, β)

elseAugment u and v in U and V as a columnand a row, respectively

end if

A few remarks about Algorithm 3 are worthnoting. Observe that the splitting of A accordingto u automatically guarantees that the result-ing matrix U has mutually orthogonal columns,which also means that each row of A is assignedto one and only one group. Observe also that thecode IMFVOTE invokes itself recursively. Sucha feature, allowable in most modern program-ming languages, makes the code particularlyefficient. Finally, be aware of the selection mech-anism embedded in the code that determinesthe final membership u and representative vthrough multiple levels of screening.

3.3 Optimal low rank approximation

In the application of low rank approximations,one of the most challenging issues even tothis date is the predetermination of the lowrank k. Algorithm 3 calculates an approximationA ≈ UV without any restriction on the sizes of

8

U ∈ Zm×p2 and V ∈ Zp×n. Suppose that such a

factorization is now at hand. It is natural to askwhether there is a way to pick up two subma-trices Uk ∈ Zm×k

2 and Vk ∈ Zk×n from U and Vsuch that A ≈ UkVk and k < p. In this section,we make an interesting observation on how tochoose the best pair of submatrices (Uk, Vk) sothat d(A,UkVk) is minimal among all possiblesubmatrices of compatible sizes. Although thisis a postscript to the main computation alreadydone by Algorithm 3, it sheds a remarkable in-sight into the optimal low rank approximation.

Denote the columns and rows of U and V by

U = [u1,u2, · · · ,up] , V =

v1

v2

...vp

,respectively. Note that these rank-one approx-imations uivi are found, say, by Algorithm 3,successively without any specific ordering. For` = 1, . . . , p, define S` to be the collection

S` :=

(i1, . . . , i`)∣∣∣ dA, [ui1 , · · · ,ui`

]

vi1...

vi`

= min(j1,··· ,`) d

A, [uj1 , · · · ,uj`]

vj1...

vj`

,

(18)

where d is either the Hamming or the Euclideanmetric and the `-tuple (j1, . . . , j`) is made ofdistinct indices from {1, . . . , p}. The folliwngnesting effect among S`’s is rather surprising.

Theorem 5: Suppose that the matrix A ∈ Zm×n

has been factorized into A ≈ UV with U ∈ Zm×p2

and V ∈ Zp×n by Algorithm 3. Then everyelement in Ss must appear as a segment in someelement of St, if s < t.

Proof: It suffices to prove the assertion forthe case s = 1 and t = 2. The followingargument can be generalized to other cases.Suppose that there exists an integer i1 ∈ S1 but{i1, i2} 6∈ S2, for any i2 ∈ {1, 2, · · · , p}. Givenany {j1, j2} ∈ S2, then we have

d

(A, [uj1 ,uj2 ]

[vj1

vj2

])< d

(A, [ui1 ,uj2 ]

[vi1

vj2

]).

(19)

We want to prove that a contradiction arises.Note that the numeral “1” appears at mu-

tually disjoint positions within the vectors ui1 ,uj1 and uj2 . Simultaneously permuting rowsif necessary, we may assume without loss of

generality that rows of A have been divided intofour blocks

A1

A2

A3

A4

(20)

where rows of A1, A2 and A3 correspond to1’s in ui1 , uj1 and uj2 , respectively, and A4

corresponds to the common zeros of all ui1 , uj1

and uj2 . Then it becomes clear that

d(A,ui1vi1 + uj2vj2) = d(A1,vi1) + d(A2, 0)+ d(A3,vj2) + d(A4, 0),

d(A,uj1vj1 + uj2vj2) = d(A1, 0) + d(A2,vj1)+ d(A3,vj2) + d(A4, 0),

d(A,ui1vi1) = d(A1,vi1) + d(A2, 0)+ d(A3, 0) + d(A4, 0),

d(A,uj1vj1) = d(A1, 0) + d(A2,vj1)+ d(A3, 0) + d(A4, 0).

Upon substitution, it follows from (19) that

d(A,uj1vj1) < d(A,ui1vi1),

which contradicts with respect to the assump-tion that i1 ∈ S1.

An application of the preceding theorem en-ables us to generate a reduced version of the UVapproximation of A. The procedure is outlinedin the following theorem.

Theorem 6: Suppose that the matrix A ∈ Zm×n

has been factorized into A ≈ UV with U ∈ Zm×p2

and V ∈ Zp×n by Algorithm 3. Sort throughthe rank-one approximations and rearrange therows if necessary, assume that

d(A,u1v1) ≤ d(A,u2v2) ≤ · · · ≤ d(A,upvp).(21)

Then, for k = 1, 2, · · · , p, the product UkVk

where Uk and Vk are submatrices of U and Vgiven by

Uk := [u1,u2, · · · ,uk] , Vk =

v1

v2

...vk

, (22)

is the best possible approximation to A in thesense that the k-tuple (1, 2, · · · , k) is in Sk.

Proof: We prove the assertion by inductionon k.

Obviously, the case k = 1 is already done dueto the rearrangement specified in (21). Assumetherefore that (1, 2, · · · , k − 1) ∈ Sk−1. We wantto show that (1, 2, · · · , k) ∈ Sk. The followingargument is essentially parallel to that in Theo-rem 5, except that we work on blocks.

9

By Theorem 5, there exists an integer q ∈{k, k + 1, · · · , p} such that (1, 2, · · · , k − 1, q) ∈Sk. If q = k, then we are done. Assume, bycontradiction, that q 6= k. Consider the pair ofsubmatrices

Uk := [Uk−1,uq] , Vk :=[Vk−1

vq

].

Still, columns of Uk are mutually exclusive. Be-cause (1, 2, · · · , k − 1, k) 6∈ Sk, we know

d(A, UkVk) < d(A,UkVk),

which is equivalent to

d(A,Uk−1Vk−1 + uqvq) < d(A,Uk−1Vk−1 + ukvk).(23)

Again, without loss of generality, we may par-tition A into blocks such as that in (20) whererows of A1, A2 and A3 correspond to 1’s in Uk−1,uq and uk, respectively, and A4 corresponds tothe common zeros of all Uk−1, uq and uk. Then,clearly we have the following expressions:

d(A,Uk−1Vk−1 + uqvq)= d(A1, Vk−1) + d(A2,vq) + d(A3, 0) + d(A4, 0),d(A,Uk−1Vk−1 + ukvk)= d(A1, Vk−1) + d(A2, 0) + d(A3,vk) + d(A4, 0),d(A,uqvq)= d(A1, 0) + d(A2,vq) + d(A3, 0) + d(A4, 0),d(A,ukvk)= d(A1, 0) + d(A2, 0) + d(A3,vk) + d(A4, 0).

It follows from (23) that

d(A,uqvq) < d(A,ukvk),

which contradicts the assumption that

d(A,uqvq) ≥ d(A,ukvk),

for q ∈ {k + 1, · · · , p}.In short, even though we still do not know

how to find the overall optimal rank, the abovediscussion suggests that something meaningfulcan be done after we have “completely” factor-ize A as A ≈ UV . That is, by ranking the rank-one approximations as in (21), we can buildsome lower rank approximations in a controlledway. This byproduct is interesting enough thatwe summarize it in Algorithm 4.

4 NUMERICAL EXPERIMENTS

In this section, we report some interesting ap-plications of our IMF techniques. We carry outexperiments on five data sets, two of whichrelated to cluster analysis are well documentedin the literature, another two demonstrate theability of our algorithms in discovering latentpatterns, and the last one is randomly generatedto assess the overall performance.

Algorithm 4 Low Rank Optimization: [U, V ] =LOWRANKAPPROX(A, k)

Given matrix A ∈ Zm×n2 and integer k.

return Low rank factors U ∈ Zm×k2 and V ∈

Zk×n.U, V ← IMFVOTE(A, active, β)for all i from 1 to p do

r← [r; match(A,U(:, i)V (i, :))]end forindex rankings← sort(r)U ← U(:, index rankings(1 : k))V ← V (index rankings(1 : k), :)

4.1 Cluster analysisThe general purpose of cluster analysis is to par-tition a set of observations into subsets, calledclusters, so that observations in the same clustershare some common features. The most difficultpart in cluster analysis is the interpretation ofthe resulting clusters. Our purpose here is notto compare the performance of our method withthe many cluster analysis algorithm alreadydeveloped in the literature. Rather, we merelywant to demonstrate that IMFVOTE naturallyproduces clusters and their corresponding in-teger representatives. If needed, we can alsoemploy LOWRANKAPPROX to pick up the mostrelevant low rank representation of original dataset.

We report experiments on two real data sets,mushroom and Wisconsin breast cancer (origi-nal), from the Machine Learning Repository main-tained by the Computer Science Department atthe University of California at Irvine [33]. Togauge the effectiveness of our algorithm, we em-ploy four parameters, RowErrorRate, Compres-sionRatio, Precision, and Recall whose mean-ings are outline below [34].

RowErrorRate, defined by

RowErrorRate :=d(A,UV )

m, (24)

refers to the average difference per row betweenthe retrieved data UV and the original data A.This value reflects how effective row vectors inV are representing the original matrix A.

CompressionRatio [35], defined by

CompressionRatio:= ] of entries in matrices U and V

] of entries in matrices A

= (m+n)kmn

(25)

measures how efficiently the original data A hasbeen compressed by the low rank representationUV . It is hoped that through the compression,less relevant data or redundant information isremoved.

10

Precision and Recall, defined by

Precision := relevant data∩retrieved dataretrieved data ,

Recall := relevant data∩retrieved datarelevant data ,

compute the percentages of the retrieved datathat are relevant and relevant data that areretrieved, respectively. In the context of infor-mation retrieval, “relevant data” usually referto documents that are relevant to a specifiedinquiry. Since our task at present is simply todivide observations into disjoint clusters basedon innate attributes, we do not know the ex-act meaning of each cluster. In other words,we do have at hand a representative for eachcluster, but we do not have an interpretationof the representative. Still, considering pointsin a cluster as retrieved data relative to theirown representative, it might be interesting tocompare the corresponding Precision and Recallwith some known training data. In this way, ahigh correlation between a particular cluster andthe training data might suggest an interpretationfor the cluster.

Mushroom Data Set. This Agaricus and Le-piota Family data set consists of biological for8124 hypothetical species of gilled mushroom.Each species is characterized by 23 attributessuch as its cap shape, cap surface, odor, andso on. Each attribute has different nominal val-ues. For example, the cap shape may be belled,conical, convex, flat, knobbed, or sunken, whilethe cap surface may be fibrous, grooved, scaly,or smooth. To fit into our scheme, we convertthe attribute information into a list of integervalues. These integers should not be regardedas numerals, but only labels. Since attributes areindependent of each other, the same integer fordifferent attributes has different meanings. Ourinput data A is an 8124 × 23 integer matrix.Of particularly noticeable is its first columnwhich is dichotomous and indicates whether themushroom is edible or poisonous. The Ham-ming metric is more suitable than the Euclideanmetric for this problem.

In our first experiment, we simply want toinvestigate how different β values affect thequantities of approximation. As is expected, asmaller β value would lead to a larger numberp of rows in the matrix V after a completefactorization. For mushroom data set, we findthat corresponding to β = 1, 2, 3 the numbersof rows in V are p = 1418, 15, 1, respectively.For each β, construct Vk according to Theorem 6after sorting the corresponding V . We plot inFigure 1 the value RowErrorRate versus k. Notethe rapidly decreasing behavior, especially inthe cases β = 1, 2, suggesting that the sorting

in LOWRANKAPPROX is capable of identifyingthe first few most important clusters and theirrepresentatives. A comparison with the Com-pressionRatio≈ 0.0436k in this case is also worthnoting. To achieve RowErrorRate= 9.6128, wejust need one pattern vector with β = 3, butwe will need many more representatives withβ = 1 or 2. On the other hand, with β = 3 wecannot possibly improve the RowErrorRate, butwith more restrictive β values there is a chanceto improve the RowErrorRate.

0 500 1000 14180.00

7.50

12.50

17.50

22.00

2.39

k

Row

Err

orR

ate

! = 1

0 5 10 15

9.50

10.00

10.50

11.00

8.98

k

Row

Err

orR

ate

! = 2

0 1 2

9.61

k

Rw

oErr

orR

ate

! = 3

Fig. 1. Performance of algorithms on the mush-room data set

In our second experiment, we applyLOWRANKAPPROX to the 8124 × 22 submatrixafter removing the first column from A.Rather surprisingly, by deleting merely onecolumn of A, the resulting IMF behavesvery differently. With β = 2, for example,the complete factorization returns only threeclusters with RowErrorRate= 9.4623. In anattempt to provide some meaning to theseclusters, we construct the so called confusionmatrix in Table 1 with respect to the attribute ofwhether the mushroom is edible or poisonous.As is seen, the first cluster represented by v1

contains large amounts of mushrooms in bothkinds, resulting in relative high Recall rates.But the Precision values by v1 are about thesame as the distribution rates in the originaldata, indicating that the cluster by v1 does notdifferentiate edible mushrooms from poisonousones. The exact meaning of these clusters is yetto be understood.

Wisconsin Breast Cancer Data Set. In a sim-ilar way, we experiment our method the BreastCancer Wisconsin Data. This data set A contains699 samples, with 458 (65.5%) benign and 241(34.5%) malignant cases. Each sample is char-acterized by 11 attributes. The first attributeis dichotomous with labels 2 or 4, indicatingbenign and malignant tumors. The remainingten attributes are of integer values ranging from1 to 10, but some entries in this part of A havemissing values for which we assign the value0 in our computation. It is commonly knownthat aspects such as the thickness of clumps or

11

Retrieved data setv1 v2 v3

Edible 4128 80 0Poisonous 3789 119 8

Cardinality of vi 7917 199 8

Precision of edible mushrooms 0.5214 0.4020 0Recall of edible mushrooms 0.9810 0.0190 0

Precision of poisonous mushrooms 0.4786 0.5980 1Recall of poisonous mushrooms 0.9676 0.0304 0.0020

TABLE 1Precision and Recall of edible or poisonous

mushrooms

the uniformity of cell shape affect the prognosis.Thus the ordinal of values in attributes matters.It is more appropriate to use the Euclideanmetric.

Again, we try out three different communalrules by varying β. With CompressionRatio≈0.1125k, meaning a reduction of 11% memoryspace per one less cluster, we plot RowErrorRateversus k in Figure 2. It is seen for this breastcancer data that increasing k would improveRowErrorRate only modestly. Since the Row-ErrorRate is already low to begin with, a lowrank IMF should serve well in approximatingthe original A.

0 111 2220.11

0.20

0.30

0.40

0.50

k

Row

Err

orR

ate

! = 1

0 6 120.18

0.20

0.22

0.24

0.26

0.28

0.30

0.32

k

Row

Err

orR

ate

! = 2

0 6 120.20

0.21

0.22

0.23

0.24

0.25

0.26

0.27

k

Row

Err

orR

ate

! = 3

Fig. 2. Performance of algorithms on Wisconsinbreast cancer data set

We then apply LOWRANKAPPROX with β = 3to the 699× 10 submatrix by removing the firstcolumn of A. A complete factorization returns11 pattern vectors with most samples beingincluded in the first two clusters. We constructthe confusion matrix with respect to the firstattribute of A in Table 2. Judging from theRecall and Precision values, we have high levelof confidence that the two patterns v1 and v2

should be good indicators of whether the tu-mor is malignant or benigh, respectively. Sucha classification of the original 699 samples into2 major indicators will be extremely useful inmedical science.

4.2 Pattern discovery

Most modern image processing techniquestreats images as a two-dimensional array com-posed of pixels. A digital image is an instructionof how to color each pixel. In a gray scale image,for example, the instruction for every element isan integer between 0 and 255 (or a nonnegativedecimal number between [0, 1] which requiresmore storage and often is converted to integers)indicating the intensity of brightness at the cor-responding pixel. In this section, let A ∈ Zm×n

denote a collection of n images each of whichis represented by a column of m pixels. Con-sider the scenario that images in this library arecomposite objects of many basic parts which arelatent at present. The factorization A = UV thenmight be suggested as a way to identify andclassify those “intrinsic” parts that make up theobject being imaged by multiple observations.More specifically, columns of U are the basiselements while each row of V can be thoughtof as an identification sequence representingthe corresponding image in A in terms of thebasis elements. This idea has been extensivelyexploited by NMF techniques. See the manyreferences mentioned earlier. The point to make,nonetheless, is that the NMF techniques cannotguarantee the resulting images to have integer-valued pixels whereas our IMF can.

Needless to say, the same idea can be appliedto an extended field of applications, such asthe quantitative structure-activity relationship(QSAR) discovery in chemoinformatics. Due tospace limitation, we shall illustrate the patterndiscovery ability of IMF by using images.

Swimmer Data Set. The ”Swimmer” data setcharacterized in [26] contains a set of black-and-white stick figures satisfying the so calledseparable factorial articulation criteria. Each figure,placed in a frame of 32 × 32 pixels, is madeof ”torso” of 12 pixels in the center and four”limbs” of six pixels. Each limb points to oneof four directions at its articulation position, sototally there are 256 figures in the collection.Sample images from the Swimmer data set aredepicted in Figure 3.

After vectorizing each image into a column,our target matrix A is of size 1024 × 256. Theoriginal data matrix contains only two integervalues, 0 and 255, which we convert withoutloss of generality to binaries. The question iswhether we are able to recover the 17 basicparts that make up these swimmers. Takingeverything into account, we should also expectone additional part for the background. Suchan expectation of U ∈ Z1024×18

2 appears to beproblematic because the original matrix A has

12

Fig. 3. 80 sample images from the swimmerdatabase.

numerical rank of only 13. In this context, thenotion of “low” rank approximation to A reallyis not appropriate.

After using the Hamming metric and β = 0.5in our code IMFVote to carry out a completefactorization, we reshape the 18 columns ofthe resulting binary U into 32 × 32 matrices.We recover all 16 limbs, one torso, plus thebackground as depicted in Figure 4. Note thatthese 17 recovered “body” parts are completelydisjointed from each other and suffer from noblurring at all — a result cannot be accom-plished by NMF techniques. In fact, the factorsU and V returned from our IMF satisfy A = UVexactly.

Fig. 4. Basic elements recovered from the swim-mer data set.

Block Matrix Data Set. To demonstrate thatour method can recognize patterns more com-plicated than one-dimensional sticks, considera 5 × 5 block matrix with each block of size5 × 5. Randomly select 2 out of the 25 blocksand assign the value 1 to their entries whilekeeping all other entries 0. Border this 25 × 25matrix with 4 pixels on each side and call theresulting 33× 33 matrix an image. Totally thereare 300 images. Collect these images into the1089 × 300 binary matrix A with each columnrepresenting a vectorized 33× 33 image. Apply

Fig. 5. Bases elements recovered from blockmatrix data set.

IMFVOTE with Euclidean metric and β = 1 toA. The resulting complete factorization returns27 clusters shown in Figure 5. It is interestingto note that 25 basic patterns containing exactlyone 5× 5 block are completely discovered. Ad-ditionally, one pattern representing almost theentire border except the point on the upper leftcorner is found.

4.3 Random performance test

To further test the capability of the IMF inrecovering random clusters or patterns, for agiven triplet (m,n, k) of integers we randomlygenerate W ∈ Zm×k

2 and H ∈ Zk×n14 , where

columns of W are kept mutually exclusive andZ14 = {0, 1, . . . , 13}. Define A = WH and applyIMFVOTE with Euclidean metric and β = 1 toA.

Let (U, V ) denote the pair of factors returnedby our calculation. We wonder how likely (U, V )would be the same as the original (W,H) af-ter some permutations. When this happens, wesay that our method has reached its optimalperformance. Recall that the basic rank oneapproximation in our scheme is only a localminimizer. We expect that pushing our algo-rithm, foredoomed just like most optimizationtechniques for nonlinear problems, to reach itsoptimal performance would be an extremelychallenging task.

For each given (m,n, k), we repeat the above-described experiment 1000 times and tally therate of success, denoted by OptRate, in reach-ing the optimal performance. We also mea-sure the CPU time needed for each experi-ment on a PC running Windows XP and Mat-lab R2009a with Intel(R)Core(TM)2 Duo CPUT8300@2.4GHz and 3.5GB of RAM. The averageCPU time, denoted by AvgTime, then serves asan across-the-board reference for the computa-tional overhead. Test results for a few selectedtriplets are summarized in Table 3. It seems

13

possible to draw a few general rules from thistable. For problems of the same size (m,n),larger k means more complexity and deeperrecursion which, in turn, reduce OptRate andcost more AvgTime. For problems of the same(m, k), increasing n costs only AvgTime, but hasmodest effect on OptRate. For problem of thesame (n, k), increasing m also costs only Avg-Time and effects little on OptRate. The overallspeed of our method seems reasonable, eventhough our code is yet to be further polished forefficiency by taking into account data structure,storage, and movement within the system.

5 CONCLUSIONSMatrix factorization has many important appli-cations. In this paper, we investigate the no-tion of factorization with entries restricted tointegers or binaries. Being discrete in nature,such a factorization or approximation cannot beaccomplished by conventional techniques. Builtupon a basic scheme of rank one approximation,we propose an approach that recursively splits(or more correctly, approximates) the underlyingmatrix into a sum of rank one matrices withdiscrete entries. We carry out a systematic dis-cussion to address the various computationalissues involved in this kind of factorization. Theideas are implemented into an algorithm usingeither the Hamming or the Frobenius metric, butusing other types of metrics is possible.

If the underlying data are binary, our idea isin line with the existing code PROXIMUS. Butour formulation is readily generalizable to othertypes of data. For example, for application pur-poses we have mainly concentrated on binary-integer factorizations, where the “integer” couldbe either the regular ordinal integers or justsome nominal labels. If the underlying data areregular integers, we have developed the mech-anism to perform integer-integer factorizations.

Five different testing data are used to demon-strate the working of our IMF algorithm. Ofparticular interest is the result in Theorem 6where we show how an optimal lower rank canbe selected after a simple sorting. We think, ina remote sense, the ordering in (21) for discretedata is analogous to the ordering of singular val-ues for continuous data. Similarly, the truncatedproduct UkVk defined in (22) is analogous to thetruncated singular value decomposition.

We hope that our discussion in this paperoffers a unified and effectual avenue of attackon more general factorization problem. There isplenty of room for future research, including arefinement of our algorithm for more efficientdata management and a generalization to morecomplex data types.

ACKNOWLEDGMENT

The authors wish to thank Professor Robert EHartwig and Mr. Yen-Wei Li for helpful discus-sions.

REFERENCES

[1] Lawrence Hubert, Jacqueline Meulman, and WillemHeiser. Two purposes for matrix factorization: a his-torical appraisal. SIAM Rev., 42(1):68–82 (electronic),2000.

[2] Moody T. Chu, Robert E. Funderlic, and Gene H.Golub. A rank-one reduction formula and its applica-tions to matrix factorizations. SIAM Rev., 37(4):512–530,1995.

[3] Louis Guttman. General theory and methods for matricfactoring. Psychometrika, 9(1):1–16, 1944.

[4] Louis Guttman. Multiple group methods for common-factor analysis: Their basis, computation, and interpre-tation. Psychometrika, 17(2):209–222, 1952.

[5] Louis Guttman. A necessary and sufficient formula formatrix factoring. Psychometrika, 22(1):79–81, 1957.

[6] Paul Horst. Factor Analysis of Data Matrices. Holt,Rinehart and Winston, New York, 1965.

[7] Harry H. Harman. Modern Factor Analysis. Universityof Chicago Press, Chicago, 1976.

[8] Andrew L. Comrey and Howard B. Lee. A FirstCourse in Factor Analysis. Lawrence Erlbaum Associates,Hillsdale, NJ, 1992.

[9] C. Ding, X. He, and H.D. Simon. On the equivalenceof nonnegative matrix factorization and spectral clus-tering. In In: Proceedings of the Fifth SIAM InternationalConference on Data Mining, Newport Beach, CA, 2005.

[10] W. Xu, X. Liu, and Y. Gong. Document-clustering basedon non-negative matrix factorization. In In: Proceedingsof SIGIR03, pages 267–273, Toronto, CA, July 28–August1 2003.

[11] D. D. Lee and H. S. Seung. Learning the parts of objectsby non-negative matrix factorization. Nature, 401:788–791, 1999.

[12] A. Pascual-Montano, J. M. Carzzo, K. Lochi,D. Lehmann, and R. D. Pascual-Marqui. Nonsmoothnonnegative matrix factorization (nsnmf). IEEETransactions on, Pattern Analysis and Machine Intelligence,28:403–415, 2006.

[13] Z. Chen, A. Cichocki, and T. M. Rutkowski. Con-strained non-negative matrix factorization method foreeg analysis in early detection of alzheimers disease. InIn IEEE International Conference on Acoustics, Speech, andSignal Processing, ICASSP2006, Toulouse, France, 2006.

[14] M. Dugas, S. Merk, S. Breit, and P. Dirschedl. Mdclust-exploratory microarray analysis by multidimensionalclustering. Bioinformatics, 20:931–936, 2004.

[15] P. Paatero and U. Tapper. Positive matrix factorization:A non-negative factor model with optimal utilizationof error. Environmetrics, 5:111–126, 1994.

[16] P. Paatero. The multilinear engine–a table-driven, leastsquares program for solving multilinear problems, in-cluding the n-way parallel factor analysis model. J.of Computational and Graphical Statistics, 8(4):854–888,1999.

[17] Michael W. Berry, Murray Browne, Amy N. Langville,Paul V. Pauca, and Robert J. Plemmons. Algorithmsand applications for approximate nonnegative matrixfactorization. Computational Statistics and Data Analysis,52(1):155–173, 2007.

[18] D. D. Lee and H. S. Seung. Algorithms for non-negativematrix factorization. In Advances in Neural InformationProcessing Systems, volume 13, pages 556–562, 2001.

[19] M. Chu, F. Diele, R. Plemmons, and S. Ragni.Optimality, computation and interpretation of non-negative matrix factorizations. Available onlineat http://www4.ncsu.edu/ mtchu/Research/Papers/nnmf.ps,2005.

14

[20] Patrik O. Hoyer. Non-negative matrix factorizationwith sparseness constraints. The Journal of MachineLearning Research, 5:1457–1469, 2004.

[21] D. P. Bertsekas. Nonlinear programming (2nd). Belmont,MA: Athena Scientific., 1999.

[22] Rasmus Bro and Sijmen De Jong. A fast non-negativity-constrained least squares algorithm. Journal of Chemo-metrics, 11(5):393–401, 1997.

[23] Hyunsoo Kim and Haesun Park. Nonnegative matrixfactorization based on alternating nonnegativity con-strained least squares and active set method. SIAM J.Matrix Anal. Appl., 30(2):713–730, 2008.

[24] Charles L. Lawson and Richard J. Hanson. Solving leastsquares problems, volume 15 of Classics in Applied Mathe-matics. Society for Industrial and Applied Mathematics(SIAM), Philadelphia, PA, 1995. Revised reprint of the1974 original.

[25] Moody T. Chu and Matthew M. Lin. Low-dimensionalpolytope approximation and its applications to non-negative matrix factorization. SIAM J. Sci. Comput.,30(3):1131–1155, 2008.

[26] D. Donoho and V. Stodden. When does nonnegativematrix factorization give a correct decomposition intoparts? In Proc. 17th Ann. Conf. Neural InformationProcessing Systems, NIPS, Stanford University, Stanford,CA, 2003, 2003.

[27] P. K. Hopke. Receptor Modeling for Air Quality Manage-ment. Elsevier, Amsterdam, 1991.

[28] Patrik O. Hoyer. Nonnegative sparse coding. In Proc.IEEE Workshop Neural Networks for Signal Processing,Martigny, 2002.

[29] T. Kawamoto, K. Hotta, T. Mishima, J. Fujiki,M. Tanaka, and T Kurita. Estimation of single tonesfrom chord sounds using non-negative matrix factor-ization. Neural Network World, 3:429–436, 2000.

[30] Suvrit Sra and Inderjit S. Dhillon. Nonnegative matrixapproximation: Algorithms and applications. Technicalreport, Deptartment of Computer Sciences, Universityof Texas at Austin, 2006.

[31] Shona D. Morgan. Cluster analysis in electronic man-ufacturing. Ph.D. dissertation, North Carolina State Uni-versity, Raleigh, NC 27695., 2001.

[32] Mehmet Koyuturk, Ananth Grama, and Naren Ra-makrishnan. Nonorthogonal decomposition of binarymatrices for bounded-error data compression and anal-ysis. ACM Trans. Math. Software, 32(1):33–69, 2006.

[33] A. Asuncion and D.J. Newman. UCI machine learningrepository, 2007.

[34] G. G Chowdhury. Introduction to modern informationretrieval. 2nd ed. facet publishing, 2004.

[35] Thomas M. Cover and Joy A. Thomas. Elements ofinformation theory. Wiley-Interscience [John Wiley &Sons], Hoboken, NJ, second edition, 2006.

15

Retrieved data set

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11

Benign 33 411 1 5 3 0 1 1 1 1 1

Malignant 225 13 0 0 1 1 1 0 0 0 0

Cardinality of vi 258 424 1 5 4 1 2 1 1 1 1

Precision of benign cancer 0.1279 0.9693 1 1 0.75 0 0.5 1 1 1 1

Recall of benign cancer 0.0721 0.8974 0.0022 0.0109 0.0066 0 0.0022 0.0022 0.0022 0.0022 0.0022

Precision of malignant cancer 0.8721 0.0307 0 0 0.25 1 0.5 0 0 0 0

Recall of malignant cancer 0.9336 0.0539 0 0 0.0041 0.0041 0.0041 0 0 0 0

TABLE 2Precision and Recall of benign or recall patients

m 500 500 500 500 500 500 500 500 500 1000 1000 1000 1000 1000 1000

n 30 40 50 30 40 50 60 80 100 30 40 50 60 80 100

k 5 6 7 10 12 14 5 6 7 10 12 14 5 6 7

OptRate .5530 0.3200 .1820 0.0480 .0110 .0010 .5280 .3130 .1450 .0470 .0100 .0020 .5440 .3520 .1740

AvgTime .0133 .0158 .0178 .0167 .0196 .0224 .0177 .0213 .0252 .0468 .0524 .0575 .0529 .0627 .0724

TABLE 3Global convergence rate and average time per factorzation (in seconds) on randomly generated

data sets.