40

Learning Scale Free Networks by Reweighted `1 regularization

Qiang Liu Alexander IhlerUniversity of California, Irvine University of California, Irvine

Abstract

Methods for `1-type regularization have beenwidely used in Gaussian graphical model se-lection tasks to encourage sparse structures.However, often we would like to include morestructural information than mere sparsity.In this work, we focus on learning so-called“scale-free” models, a common feature thatappears in many real-work networks. Wereplace the `1 regularization with a powerlaw regularization and optimize the objec-tive function by a sequence of iterativelyreweighted `1 regularization problems, wherethe regularization coefficients of nodes withhigh degree are reduced, encouraging the ap-pearance of hubs with high degree. Ourmethod can be easily adapted to improve anyexisting `1-based methods, such as graphi-cal lasso, neighborhood selection, and JSRMwhen the underlying networks are believedto be scale free or have dominating hubs. Wedemonstrate in simulation that our methodsignificantly outperforms the a baseline `1method at learning scale-free networks andhub networks, and also illustrate its behavioron gene expression data.

1 Introduction

A scale-free network is a network whose degree distri-bution follows a power law. Scale-free networks havebeen empirically observed in a wide variety of systems,including protein and gene networks, publication cita-tion networks, and many social networks; for exam-ples, see Barabasi and Albert (2002) and referencestherein. It has been shown that scale free networks canbe generated by preferential attachment mechanisms,

Appearing in Proceedings of the 14th International Con-ference on Artificial Intelligence and Statistics (AISTATS)2011, Fort Lauderdale, FL, USA. Volume 15 of JMLR:W&CP 15. Copyright 2011 by the authors.

in which new nodes prefer to connect to nodes withhigh degree (Barabasi and Albert, 1999). Perhaps themost notable characteristic of a scale-free network isthe relative frequency of “hubs” – vertices with a de-gree that greatly exceeds the average. Identifying hubsis often a primary step of understanding the networkstructure, as hubs are thought to serve specific, per-haps critical purposes in their network. For example,one important task in bioinformatics is to identify thehub proteins and hub genes.

On the other hand, Gaussian Markov random fields(GMRFs) have been widely used to infer the struc-ture of networks from data. In particular, supposethat x = [x1, · · · , xp]′ follows a p-dimensional Gaus-sian distribution with zero mean and covariance Σ. LetΩ = Σ−1 be the precision matrix. It is well known thatthe precision matrix reflects the dependency structureof the GMRFs, since the (i, j) element of Ω is non-zeroif and only if xi and xj are connected in the Markovrandom field, i.e., are conditionally dependent giventhe values of all other elements of x. The task of infer-ring network structure can be thus recast as the sta-tistical problem of estimating the precision matrix Ω.However, a major difficulty in this task derives fromthe fact that the number of observed data points n isusually small compared with the dimensionality p, inwhich case the empirical covariance matrix will havesignificant noise. The naıve method of simply takingthe empirical covariance matrix usually results in afully connected graph, and thus does not indicate anystructural independence within the network. Worse, inthe increasingly common case that n < p the empiricalcovariance matrix becomes singular.

Various methods have been developed to estimatethe structure of graphical models through the use of`1 regularizations (e.g., Meinshausen and Buhlmann,2006; Yuan and Lin, 2007; Peng et al., 2009; Banerjeeet al., 2008). Among them, the neighborhood selectionmethod of Meinshausen and Buhlmann (2006) is per-haps the simplest: by noting that the (i, j) element ofΩ is, up to a positive scalar, the regression coefficientof variable j in the regression of variable i against therest, they estimate a sparse graphical model by fitting

41

Learning Scale Free Networks by Reweighted `1 regularization

a collection of lasso regression models for each xi us-ing the other variables x¬i = xj |j 6= i in turn. Theyshowed that this method provides an asymptoticallyconsistent estimator of the set of non-zero elements ofΩ. Building upon this approach, Peng et al. (2009)proposed a joint sparse regression method to addressthe asymmetry of Meinshausen and Buhlmann (2006).This method is related to the recent symmetric lassomethod (Friedman et al., 2010), which is formulated asa maximum pseduolikelihood estimator (Besag, 1974).A more systematic approach is to maximize the `1 pe-nalized log-likelihood (e.g., Yuan and Lin, 2007; Baner-jee et al., 2008). Friedman et al. (2007) proposed anefficient blockwise coordinate descend method, calledgraphical lasso, which maximizes the `1 penalized log-likelihood by iteratively solving a series of modifiedlasso regressions.

However, it has been observed in previous work that`1-based methods work poorly for scale-free networks,in which some nodes (called hubs) have extremely highdegrees. Schafer and Strimmer (2005) showed an ex-periment in which Meinshausen and Buhlmann (2006)failed to identify the hub-type structures in gene net-works. A possible reason for this is that the `1 regular-ization imposes sparsity uniformly and independentlyon each node, without any preference for identifyinghub-like structures. Theoretically, the consistency re-sult in Meinshausen and Buhlmann (2006) relies onan sparsity assumption that restricts the maximal sizeof the neighborhoods of variables (see Assumption 3in Meinshausen and Buhlmann (2006)), which is in-consistent with the existence of hubs. For graphicallasso, Ravikumar et al. (2008) shows that to ensure

an elementwise bound ||Ω − Ωtrue||∞ = O(√

log pn )

holds with high probability, one need a sample sizeof n = Ω(d2 log p), where d is the maximum degreeof the graph. In other words, by this scaling analysisthe sample size needs to be polynomial in the graphdegree, while only logarithmic in its size (dimensional-ity). This suggests that the “curse of degree” is evenworse than the “curse of dimensionality”. Ravikumaret al. (2010) shows that a similar difficulty occurs inhigh dimensional Ising model selection when using `1-regularized logistic regression.

In fact, it has been widely realized that a simple `1sparsity prior often does not take full advantage of theprior information that we may hold about the underly-ing model, especially when the true network has somespecific structural features. There has been consider-able recent work on improving the lasso by incorporat-ing various prior information of structures in additionto simple sparsity, including the group lasso (Yuanand Lin, 2006), simultaneous lasso (Turlach et al.,2005) and fused lasso (Tibshirani et al., 2005) in vari-

able selection problems. Analogously, there are manyworks incorporating different structured prior informa-tion in Gaussian graphical model selection in variousways, for example Duchi et al. (2008); Schmidt et al.(2009); Marlin et al. (2009); Friedman et al. (2010);Peng et al. (2010); Marlin and Murphy (2009); andJacob et al. (2009). In this work, we develop an effi-cient method to incorporate a preference for scale-freebehavior into our prior for network inference. We showthat our method can be recast as a sequence of iter-atively reweighted `1 regularization problems, wherethe regularization coefficients of the nodes with higherdegree are decreased, encouraging the creation of hubs.We demonstrate the performance of our algorithm onboth simulated and gene expression data.

2 Learning sparse networks

Suppose x = [x1, x2, · · · , xp]′ is drawn from a multi-variate normal distribution N (0,Σ), where Σ is thep× p covariance matrix. Let Ω = Σ−1 = ωij be theprecision matrix. X = [x1, x2, · · · , xn] is a collectionof n observed data. The task is to estimate the setof non-zero elements of Ω, which corresponds to theedges of the Gaussian graphical model. Let us denoteρij = corr(xi, xj |x¬ij) to be the partial correlation,where x¬ij are the other elements of x besides xi andxj . A well know fact is that the multivariate Gaus-sian model can be presented in a self-regression form,xi =

∑i 6=j βijxj + δi, where βij = −ωij

ωii= ρij

√ωjj

ωii,

and δi are Gaussian noise that are independent of x¬i:δi ∼ N (0, 1/ωii). We will use B to represent a matrixwith zero diagonal and βij as the off-diagonal entries,and Φ a matrix with zero diagonal and ρij in the off-diagonals. Note that the off-diagonals of Ω, B and Φonly differ up to a nonzero constant, and hence sharethe same non-zero pattern; one can predict the modelstructure by estimating the non-zero pattern of anyone of them.

A body of research has been developed to estimate thenon-zero pattern by applying `1 regularization to theprecision matrix or partial correlation matrix. Mein-shausen and Buhlmann (2006) proposed a simple negh-borhood selection method (henceforth referred to as“MB”) by regressing each variable w.r.t. to all theother variables with a `1 regularization:

minβ

12||xi −

∑j 6=i

βijxj ||2 + λ∑j 6=i

|βij |,

in which the non-zero elements of βij |j 6= i decidethe neighborhood of xi. This leads to n independentstandard lasso problems, which can be solved by effi-cient algorithms such as LARS or coordinate descent(Friedman et al., 2007). It was shown that it gives

42

Qiang Liu, Alexander Ihler

asymptotical consistent estimator of the set of non-zero elements of precision matrix. Its disadvantage,however, is that since βij and βji are estimated inde-pendently, it cannot guarantee that the non-zero pat-tern of βij is symmetric and must use post-processingto create a symmetric estimate.

Peng et al. (2009) proposed a joint regression sparsemethod (“JSRM”) to address this asymmetry issue,by minizing the following joint loss function

12

p∑i

µi||xi −∑j 6=i

ρij

√ωjjωii

xj ||2 + λ∑j 6=i

|ρij |. (1)

with the symmetry constraint that ρij = ρji. For fixedωii, (1) also forms a joint lasso regression of variable[ρ12, ρ13, · · · , ρp−1,p]T ; for fixed ρij , ωii can be opti-mized analytically. Therefore, Peng et al. (2009) pro-posed to optimize ρij and ωii iteratively. Note thatMB is equivalent to a simplified version of (1) in whichthe symmetry constraint is relaxed and ωii are set tobe a uniform constant. The weights µi in (1) areparameters that can be manipulated to increase theflexibility of the method. If µi = ωii, (1) resemblesa maximum pseudo likelihood estimator of the mul-tivariate Gaussian model, which was made precise bythe recent “symmetric lasso” method (Friedman et al.,2010). Peng et al. (2009) also explored various heuris-tics for choosing µi, and suggested that by taking µito be the estimated degree of node i in the previous it-eration, the algorithm (referred as “JSRM.dew”) willprefer scale free networks. This relates to our basictask of learning scale free networks, and we return todiscuss and compare this method further in the sequel.

A more systematic solution is to minimize the neg-ative `1 penalized log-likelihood (e.g. Yuan and Lin,2007; Banerjee et al., 2008; d’Aspremont et al., 2008;Friedman et al., 2008):

− log p(x|Ω) + λ∑i,j

|ωij |, (2)

where the log-likelihood is

log p(x|Ω) = log det(Ω)− tr(SΩ),

with S = 1nXX

T being the empirical covariance ma-trix. The matrix Ω is constrained to be symmetric andpositive definite during the optimization.

Optimizing (2) can be recognized as a non-differentiable convex problem. Various optimizationmethods have been developed to solve it. See e.g.,Banerjee et al. (2008); Friedman et al. (2008); Duchiet al. (2008). One of the most efficient is the graphi-cal lasso (“glasso”), which uses a blockwise coordinate

descent strategy, updating each row/column of the co-variance matrix by solving a modified lasso subprob-lem in each descent step. This method is guaranteedto maintain the property of positive definiteness dur-ing each update, so that the constraint does not needto be considered explicitly.

Across these methods there is a debate as to whether|ωij | or |ρij | should be regularized. (Note that ρij =− ωij√

ωiiωjj.) In principle, JSRM can be modified to reg-

ularize |ωij |, or an optimizer of the likelihood (2) suchas graphical lasso could instead regularize |ρij |. Theyare all based on the idea of `1 regularization for en-couraging sparse patterns, but have minor differencesin their mathematical details. Since the focus of thecurrent work is to show how existing methods can beimproved by incorporating scale-free priors, we will re-spect the different traditions of the original methods.

In general, all the described `1 based methods have asimilar basic form, maximizing a score function with a`1 penality term

L`1(Θ) = S(X,Θ)− λ||Θ||1, (3)

where ||Θ||1 =∑ij |θij | is the `1 norm. Θ may be

the precision matrix Ω or the partial correlation ma-trix Φ, and S(X,Θ) takes on different forms depend-ing on the implementation details of different methods.The term ||Θ||1 can be thought of as the continuousconvex surrogate of the `0 norm of Θ, which equalsthe number of non-zero elements and thus encouragessparsity; there are also arguments that `1 is in fact theoptimal convex surrogate in certain senses (see e.g.,d’Aspremont et al., 2008). From a Bayesian point ofview, the `1 term can be interpreted as a Laplacianprior, i.e., as assuming that the parameters θij aredistributed i.i.d. according to prior probability distri-bution p(θij) ∝ exp(−λ|θij |).

3 Power Law Regularization

In scale-free network, the degree distribution of thevertices follows a power law: p(d) ∝ d−α. The degreed of vertex i in a GMRF can be understood as the`0 norm of θ¬i = θij |j 6= i. We propose to use||θ¬i||1 + εi as a continuous surrogate of the degreed, where ||θ¬i||1 =

∑i 6=j |θij | is the `1 norm. The

small positive term εi is used to make the surrogatepositive, and hence ensure that its value in the powerlaw distribution will be well defined. Then, we canoptimize the following score function:

Lsf (Θ) = S(X,Θ)− α∑i

log(||θ¬i||1 + εi)− β∑i

|θii|.

(4)

43

Learning Scale Free Networks by Reweighted `1 regularization

The positive term εi represents the sensitivity thresh-old for the parameters and is used to “smooth out” thescale-free property. In practice, we take εi to have thesame order as θij . If εi is very large, i.e., ||θ¬i||1 εi,then log(||θ¬i||1 + εi) ≈ 1

εi||θ¬i||1 + log(εi), and the

off-diagonal regularization term becomes α∑i 6=j(

1εi

+1εj

)|θij |, which is equivalent to the standard `1 reg-ularization in (2), but with a different regularizationcoefficient on each element (i, j). In this sense, (4)generalizes the `1 regularization method. The diago-nal elements are separated out, since we do not wantto apply the power law to the diagonal. In practice, wetake β = 2 αεi , which is the `1 regularization coefficientin the limiting case of ||θ¬i||1 εi. Note that when Θis the partial correlation, the diagonal terms are zero,and need not to be not considered.

The scale-free regularization is no longer convex as instandard `1 regularization because of the use of the logfunction. Intuitively, nonconvexity arises naturally inthe problem of learning a scale-free network, since thelocations of the hubs must be identified, creating thepotential for multiple local modes.

As an alternative interpretation, the regularization in(4) can be also thought of as an approximate form ofthe log-normal distribution, which has been proposedas an alternative to the power law for characterizingthe degree distribution of scale-free networks (Pennocket al., 2002; Mitzenmacher, 2003; Arita, 2005). A ran-dom variable z is said to be log-normal if log(z) followsa normal distribution. Formally, its log probability is

log p(z) = − log(z)− 12

(log(z)− µ

σ

)2

+ const.

If σ is very large, the second term is small and thelog-log plot of z versus p(z) becomes approximatelylinear (Arita, 2005). Therefore, we can interpret (4)as the approximate posterior distribution after addinga log-normal prior on (||θ¬i||1 +εi)1/α. Analogously tothe central limit theorem, which states that the sum ofmany i.i.d. random variables approaches a Gaussiandistribution, the product of many i.i.d. positive ran-dom variables approaches a log-normal distribution.Therefore, log-normal distributions arise quite natu-rally throughout the physical world, making it a rea-sonable prior on the parameters from a more formalBayesian perspective.

4 Reweighted Methods

In this section, we derive an algorithm to maximize(4). We show that (4) can be solved by a sequenceof `1 regularization problems, whose regularization co-efficients are updated iteratively in a way that mim-

ics the preferential attachment in scale free networks.Our method is derived as an instance of the minorize-maximize (MM) algorithm (Hunter and Lange, 2004),which is an extension of the expectation-maximization(EM) algorithm (Bilmes, 1998; Dempster et al., 1977).The MM algorithm maximizes the objective functionby iteratively maximizing its minorizing functions,which are lower bounds of the objective function thatare tangent to it at the current estimation point. Inthis way, one can always guarantee that the objectivefunction increases monotonically and converges if it isupper bounded (Dempster et al., 1977; Wu, 1983)).More specifically, suppose Θn is the estimate found inthe last iteration; noting that

α∑i

log(||θ¬i||1 + εi)− α∑i

log(||θn¬i||1 + εi)

≤ α∑i

(||θ¬i||1 + εi||θn¬i||1 + εi

− 1)

=∑i 6=j

λij |θij |+ const.

where

λij = α

(1

||θn¬i||1 + εi+

1||θn¬j ||1 + εj

). (5)

then we have

Lsf (Θ)− Lsf (Θn)

≥ S(X,Θ)−∑i 6=j

λij |θij | − β∑i

|θii|+ const. (6)

The equality in (6) holds if and only if Θ = Θn.

Therefore, Lsf (Θ) can be improved by iteratively max-imizing the lower bound:

Θn+1 = arg maxΘ

Q(Θ|Θn)

= arg maxΘS(X,Θ)−

∑i 6=j

λij |θij | − β∑i

|θii|,

(7)

which can be implemented using any of the previouslydiscussed methods (MB, JSRM, graphical lasso, orothers). It is easy to see that (6) and (7) establishesa MM algorithm, and hence the objective function in-creases monotonically, i.e., Lsf (Θn+1) ≥ Lsf (Θn).

This process (and the MM algorithm in general) isclosely analogous to the EM algorithm, with (5) cor-responding to the E-step and (7) corresponding to theM-step. The properties of EM can thus be directlyapplied to our algorithm. For example, the results ofWu (1983) demonstrate that Θn convergences to astationary point of Lsf(Θ), under some mild regular-ity conditions. We also note that it is not necessaryto exactly maximize Q(Θ|Θn); as with the philoso-phy of generalized EM (GEM) (Dempster et al., 1977),

44

Qiang Liu, Alexander Ihler

the desired properties can be maintained so long asQ(Θ|Θn) is improved in each iteration. In algorithmssuch as graphical lasso in which the exact optimiza-tion of (7) is iterative and costly, one can update theweights more quickly, for example after one sweep ofall the rows.

The weight update in (5) has an appealing intuition. Itdecreases the regularization strength if ||θ¬i||1 is large,and hence encourages the appearance of high degreenodes (network hubs) in a rich-get-richer fashion. Theprocess mirrors the mechanism of preferential attach-ment (Barabasi and Albert, 1999) that has been pro-posed as an underlying generative model to explainpower law degree distributions in scale-free networks.

5 Related Work

There is some existing work on learning scale free net-works, but little in a systematic manner. For exam-ple, as Peng et al. (2009) suggested, JSRM can beencouraged to learn scale free network by setting theweights µi in (1) to be proportional to the estimateddegrees in the previous iteration. This heuristic ap-proach, however, cannot be adapted to improve othermethods such as the graphical lasso. A strength ofour approach is that it can be applied to improve any`1 based methods that have the form (3) when theunderlying network is believed to be scale free. Ourexperimental results also suggest that our method canprovide more accurate estimates than by adjusting µi.Another work related to constructing scale free net-works is Chen et al. (2008), which is based on a simpleheuristic of ranking the empirical correlation matrix.

Very recent work such as Friedman et al. (2010) andPeng et al. (2010) use a similar idea based on grouplasso, where a `2 regularization on the columns of Θ isapplied. These works are similar to our method in thesense that edges incident on the same node share infor-mation with each other, but induce different behaviors.Friedman et al. (2010) encourages “node sparsity”, inwhich a node with weak correlations tends to be to-tally disconnected, while our method tends to reshapethe network towards a power degree distribution.

An interesting connection can be drawn between ourtechnique and the reweighted `1 minimization methodfor compressive sensing (Candes et al., 2008), whichgives a different perspective for (5) . To illustrate,we re-write the `0 norm of some vector z as ||z||0 =∑i|zi||zi|+εi , where εi is positive constant that is much

smaller than the non-zero elements of z. The `1 norm||z||1 =

∑i |zi| has been widely used as a convex surro-

gate of ||z||0. However, it introduces large bias whenthe magnitude of the non-zero entries of z are very

different from one another. A more accurate approx-imation is ||z||w =

∑i|zi||z0i |+εi

, where z0 is a previousestimate of z. Candes et al. (2008) showed that aniteratively reweighted `1 minimization is closer to theground truth `0 in this sense, and presented exper-iments demonstrating that the reweighted algorithmleads to remarkably improved performance in appli-cations such as sparse signal recovery. Candes et al.(2008) further noticed that the reweighted algorithm isequivalent to using the log surrogate

∑i log(|zi|+ εi),

which also appears in our regularization. An analogousidea for lasso was explored in Shimamura et al. (2009),and was demonstrated to decrease the false positiverate when inferring a gene network. Also related inspirit is the two-pass adaptive lasso (Zou, 2006; Huanget al., 2006; Shimamura et al., 2007), in which z0 is es-timated using a predefined rough estimator such asshrinkage regression, but no further steps are taken torefine the weighting. Zou and Li (2008) and Fan et al.(2009) proposed a nonconcave penality function calledthe SCAD penality, to attenuate the bias problem ofthe `1 penality, and showed that the objective functioncan be optimized by solving a series of reweighted `1problems. Although all these methods relate to theidea of reweighting coefficients in various ways, theyare distinct from our method in that they do not ex-plore using the weights to encourage a scale free priorof the network, but focus on attenuating the statisti-cal bias from `1 regularization. Our preferential at-tachment updating is tailored to reconstruct scale freenetworks, which appear widely in real-world systemsacross many disciplines.

6 Experimental Results

We show simulation results in two types of simulateddata: scale free networks simulated using a preferen-tial attachment mechanism, and a sparse network witha few dominant hubs. Finally, we apply our methodto infer a network from gene expression data, illus-trating its ability to find and prefer attachments tohubs. For all the experiments in this paper, we im-plemented our scale free regularization on neighbor-hood selection (“MB”), JSRM with uniform weights(“JSRM”), and graphical lasso(“glasso”). We referto the scale free versions of MB, JSRM, and glassoas MB-SF, JSRM-SF and glasso-SF. For comparison,we also implemented the degree-reweighted version ofJSRM in Peng et al. (2009) (“JSRM.dew”). For MB-SF and JSRM-SF, the variables θij are the partial cor-relations ρij ∈ [−1, 1], and we take εi = 1. For graph-ical lasso, θij = ωij are the elements of the precisionmatrix, we take εi equal to θii estimated in the lastiteration, to make sure εi is on the same magnitudeof ||θ¬i||1. For all the scale free methods, the initial

45

Learning Scale Free Networks by Reweighted `1 regularization

100 101 102100

101

102True

100 101 102100

101

102JSRM

100 101 102100

101

102JSRM.dew

100 101 102100

101

102JSRM−SF

(a) True (b) JSRM (c) JSRM.dew (d) JSRM-SF

Figure 1: Top: The true scale free network and the networks estimating using JSRM (Peng et al., 2009),JSRM.dew (Peng et al., 2009), and JSRM-SF (this work). The four hubs of the original graph are shown in redin all four networks. Bottom: log-log plots of the degree distributions for each estimated network. JSRM.dewand JSRM-SF encourage scale-free degree distributions, evidenced by their linear appearance.

values of Θ are all taken to be the identity matrix sothat initially ||θ¬i||1 = 0. This makes the first itera-tion of each of the reweighting methods equivalent toits original `1 counterpart. For each method we stopafter the 5th reweighting iteration, although as we willshow later the most significant improvement has usu-ally been obtained even by the 2nd iteration (i.e., thefirst reweighted iteration). The JSRM related meth-ods are implemented in the “SPACE” R package (Penget al., 2009).

6.1 Scale Free Network

We tested our method in random simulated scale-free networks. First, a scale-free network is simu-lated via the Barabasi -Albert (BA) model (Barabasiand Albert, 1999), which generates random scale-freenetworks using a preferential attachment mechanism.More specifically, the network begins with an initial,4-node cycle. New nodes are added to the network oneat a time, and each new node is connected to one of theexisting nodes with a probability that is proportionalto the current degree of the existing node. Formally,the probability pi that the new node is connected tonode i is pi = di/

∑j dj , where di is the degree of i-th

node. Our resulting network consisted of 100 nodesand 99 edges, with 4 hubs having degrees larger than9; see Figure 1a.

We define L = ηD − G, where G is the adjacencymatrix, D is a diagonal matrix with i-th diagonal entryequal to the degree of the node i, and η is a constant

larger than 1. If η = 1, L is the Laplacian matrix.We take η to be strictly larger than one (e.g., η =1.1) to force L to be positive definite. The precisionmatrix Θ is then defined by Θ = Λ

12LΛ

12 , where Λ

is the diagonal matrix of L−1, which scales Θ suchthat the covariance matrix Σ = Θ−1 has unit diagonal,meaning that each dimension of the random vector xhas equal, unit variance.

We simulated a dataset X from the Gaussian Markovmodel N (0,Θ−1) with size n = 100. We tested MB,JSRM and glasso and their counterpart with scale freeregularization on this dataset. The off-diagonal regu-larization coefficients α are varied to control the falsepositive and true positive rate for edge prediction,which yields an ROC curve. We repeat the experi-ment 20 times, and plot the averaged ROC curves inFigure 2. The fraction of estimated edges connectingto the hubs are shown in Figure 2 as well.

As can be seen in Figure 2 (best viewed in color), theROC curves of the scale-free regularization methods(solid lines) are consistently above their original coun-terparts (dashed), and encourage a greater number ofedges connecting to the hubs. Comparing the results ofJSRM.dew and our JSRM-SF, it shows that the powerlaw regularization is a more effective way of learningscale free networks.

In Figure 2c, we show the ROC curve of JSRM-SFafter performing different numbers of iterations. It isevident that the greatest gain in accuracy is obtainedat the 2nd iteration, i.e., with only one extra iteration

46

Qiang Liu, Alexander Ihler

0 0.05 0.1 0.15 0.2 0.250.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

glassoglasso−SFMBMB−SFJSRMJSRM−SFJSRM.dew

(a)

0 0.05 0.1 0.15 0.2 0.250

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

False positive rate

Perc

enta

ge o

f edg

es c

onne

ctin

g to

hub

s

glassoglasso−SFMBMB−SFJSRMJSRM−SFJSRM.dew

(b)

0 0.05 0.1 0.15 0.2 0.250.7

0.75

0.8

0.85

0.9

0.95

1

JSRM−SF(iter=5)JSRM−SF(iter=4)JSRM−SF(iter=3)JSRM−SF(iter=2)JSRM.dewJSRM

(c)

Figure 2: Experimental results on estimating the edges of a scale free network (see also Figure 1). (a) TheROC curves of each of the methods. Standard methods are shown as dashed, while their scale-free versions areshown as solid. In each case, the scale-free version significantly dominates its original; in the case of JSRM,it also dominates the scale-free JSRM.dew variant. (b) The fraction of edges connecting to the hubs vs. thefalse positive rate. Note that the true fraction is 0.6. Scale-free variants are more likely to find hub-connectededges “early”, when few edges have been included in the graph. (c) The ROC curves of JSRM-SF after differentnumbers of iterations. Note that the major improvement of accuracy is obtained at the 2nd iteration (i.e., oneextra iteration compared to the original `1 counterpart). This suggests that our method is not significantly morecomputationally expensive than the `1 counterpart.

(a)

0 0.05 0.1 0.15 0.2 0.250.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

glassoglasso−SFMBMB−SFJSRMJSRM−SFJSRM.dew

(b)

0 0.05 0.1 0.15 0.2 0.250.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

False positive rate

Perc

enta

ge o

f edg

es c

onne

ctin

g to

hub

s

glassoglasso−SFMBMB−SFJSRMJSRM−SFJSRM.dew

(c)

Figure 3: Experiment results on estimating the edges of a hub network. (a) The true hub network from whichdata are simulated. (b) The ROC curves of different methods. Standard methods are shown as dashed, whiletheir scale-free versions are shown as solid. Again in each case, the scale-free version significantly dominates itsoriginal, as well as the scale-free JSRM.dew variant. (c) The fraction of edges connecting to the hubs vs. thefalse positive rate. Note that the true fraction is 0.5053. Again, we see that the scale-free variants find edgesincident to the hubs earlier than their original counterparts.

compared with the original `1 method. Therefore, ourscale-free iterative reweighting does not entail muchadditional computation over the original methods. In-terestingly, similar behaviors have been found in otherreweighting methods (e.g. Candes et al., 2008; Zou andLi, 2008), and it is recommended by many authorsto stop after the 2nd iteration, i.e., perform only onereweighting step (Fan et al., 2009; Candes et al., 2008;Zou and Li, 2008). In case when the underlying net-work is not scale free, using fewer iterations may alsoavoid introducing too large of a bias.

We also show examples of the estimated networksfound by JSRM, JSRM.dew and JSRM-SF in Figure 1.All the estimated networks have been selected to havethe same number of edges (≈ 160) and thus the samefalse positive rate (≈ 0.025). The bottom row of Fig-ure 1 shows log-log plots of the true degree distributionand the degree distributions of the estimated networks.Visually, the network estimated by scale free regular-ization most closely follows a power law; the networkestimated by JSRM.dew is also similar. The resultsof MB vs. MB-SF and glasso vs. glasso-SF are alsosimilar, but are not shown due to space limitations.

47

Learning Scale Free Networks by Reweighted `1 regularization

(a) glasso (b) glasso-SF

100 101 102100

101glasso

100 101 102100

101

102glasso−SF

(c) Degree distributions

Figure 4: Networks estimated on gene expression data. (a) The network with 120 edges estimated using glasso.(b) The network with 120 edges estimated using glasso-SF. The top 4 hubs are colored red. Both algorithmsidentify the same four hubs, but they are more “hub-like” in the network estiamted with glasso-SF. (c) Thelog-log plot of the degree distributions of the networks in (a) and (b). The network estimated using glasso-SF isnoticeably more scale-free.

6.2 Hub Network

We also tested our algorithm on a sparse network witha few dominating hubs. This graph consists of fourk-star subgraphs of k = 25 nodes, in which a hub con-nects to all the other nodes in its subgroup. We thenadd random edges between the non-hub nodes. Ournetwork (shown in Figure 3a) results in 100 nodes and190 edges with 4 hubs each of degree of 24. We simu-lated the values of the precision matrix using the samemethod described in Section 6.1. We draw sample datasets of size 200, and repeat the experiment 20 times tofind average performance. The ROC curves of each ofthe different methods are shown in Figure 3b, and thefraction of edges connected to the true hubs are plot-ted in Figure 3c. The result strengthens the argumentthat our method improves accuracy compared to eachof its `1 counterparts, and prefers hub structures.

6.3 Network for Gene Expression Data

We tested our algorithm on a time course profile for aset of 102 genes selected from Saccharomyces cerevisiae(Spellman et al., 1998; Chen et al., 2008). These mi-croarray experiments were designed to identify yeastgenes that are periodically expressed during the cellcycle. The gene expressions were collected over 18 timepoints, which are treated as independent samples froma GMRF in our setting. Figure 4 shows two networkswith 120 edges estimated using glasso and glasso-SF,along with a log-log plot of their degree distributions(the other methods are implemented but bear similarresults and are omitted for space). Visually, the net-work estimated using our algorithm appears closer toscale-free behavior and exhibits more clustering to thehubs. It is difficult to assess the accuracy of any of the

algorithms for this problem, since the true underly-ing network is unknown, and existing side informationis not very consistent with the data set (Chen et al.,2008; Zou and Conzen, 2005). However, we note thatour methods identify the same set of highest-degreenodes (colored in red) with their `1 counterparts, butallocate more edges on the hubs (exhibiting a “pref-erential attachment” mechanism). This suggests thatour method is consistent in the sense that it does notdeviate greatly from the original methods, but imposesa slight bias toward the scale-free behavior believed toexist in the true network.

7 Conclusions and Future Directions

The study of complex networks is an active area of sci-entific research that examines common topological fea-tures of real-world networks. While scale-free behavioris widely acknowledged to be common, it is only oneof many possible examples. Other features, such asassortativity or disassortativity among vertices, com-munity, and hierarchical structure may also provideimportant information for network inference. We ex-pect to see considerable additional cross fertilizationbetween these two areas. There are also a number oftheoretical issues of our algorithm which remain unex-plored; for example, the asymptotic consistency, andthe method of selecting the regularization coefficient.

Acknowledgements

This work was supported in part by NIH grants P50-GM076516 and NIAMS AR 44882 BIRT revision. Theauthors thank Prof. Yang Dai for sharing the microar-ray dataset.

48

Qiang Liu, Alexander Ihler

References

M. Arita. Scale-freeness and biological networks. JBiochem, 138(1):1–4, July 2005.

O. Banerjee, L. El Ghaoui, and A. d’Aspremont. Modelselection through sparse maximum likelihood estimationfor multivariate Gaussian or binary data. JMLR, 9:485–516, 2008.

A.-L. Barabasi and R. Albert. Emergence of scaling inrandom networks. Science, 286(5439):509–512, 1999.

A.-L. Barabasi and R. Albert. Statistical mechanics of com-plex networks. Rev. Mod. Phys., 74:47–97, June 2002.

J. Besag. Spatial interaction and the statistical analysis oflattice systems. J. Roy. Stat. Soc. B Stat. Meth., 36(2):192–236, 1974.

J. Bilmes. A gentle tutorial of the EM algorithm and its ap-plication to parameter estimation for Gaussian mixtureand hidden markov models. Technical report, Universityof Berkeley, 1998.

E. J. Candes, M. Wakin, and S. Boyd. Enhancing sparsityby reweighted L1 minimization. J. Fourier Anal. Appl.,14(5), December 2008.

G. Chen, P. Larsen, E. Almasri, and Y. Dai. Rank-basededge reconstruction for scale-free genetic regulatory net-works. BMC Bioinformatics, 9(1):75, January 2008.

A. d’Aspremont, O. Banerjee, and L. El Ghaoui. First-order methods for sparse covariance selection. SIAM J.Matrix Anal. Appl., 30(1):56–66, 2008.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximumlikelihood from incomplete data via the EM algorithm.J. Royal Statist. Soc. B, 39(1):1–38, 1977.

J. Duchi, S. Gould, and D. Koller. Projected subgradientmethods for learning sparse Gaussians. In UAI, 2008.

J. Fan, Y. Feng, and Y. Wu. Network exploration via theadaptive lasso and scad penalties. Ann. Statist., 3(2):521–541, June 2009.

J. Friedman, T. Hastie, H. Hofling, and R. Tibshirani.Pathwise coordinate optimization. Ann. Appl. Stat., 1(2):302–332, 2007.

J. Friedman, T. Hastie, and R. Tibshirani. Sparse inversecovariance estimation with the graphical lasso. Biostat,9(3):432–441, July 2008.

J. Friedman, T. Hastie, and R. Tibshirani. Applications ofthe lasso and grouped lasso to the estimation of sparsegraphical models. 2010.

J. Huang, S. Ma, and C.-H. Zhang. Adaptive lasso forsparse highdimensional regression. Technical report,University of Iowa, 2006.

D. R. Hunter and K. Lange. A tutorial on MM algorithms.The American Statistician, 1(58), February 2004.

L. Jacob, G. Obozinski, and J.-P. Vert. Group lasso withoverlap and graph lasso. In ICML, 2009.

B. Marlin, M. Schmidt, and K. Murphy. Group sparsepriors for covariance estimation. In UAI, 2009.

B. M. Marlin and K. P. Murphy. Sparse gaussian graphicalmodels with unknown block structure. In ICML, 2009.

N. Meinshausen and P. Buhlmann. High-dimensionalgraphs and variable selection with the lasso. Ann.Statist., 34(3):1436–1462, 2006.

M. Mitzenmacher. A brief history of generative models for

power law and lognormal distributions. Internet Math.,1(2), 2003.

J. Peng, P. Wang, N. Zhou, and J. Zhu. Partial correlationestimation by joint sparse regression models. JASA, 104(486):735–746, 2009.

J. Peng, J. Zhu, A. Bergamaschi, W. Han, D.-Y. Noh, J. R.Pollack, and P. Wang. Regularized multivariate regres-sion for identifying master predictors with application tointegrative genomics study of breast cancer. Ann. Appl.Stat., 4(1), 2010.

D. M. Pennock, G. W. Flake, S. Lawrence, E. J. Glover,and C. L. Giles. Winners don’t take all: Characterizingthe competition for links on the web. PNAS, 99(8):5207–5211, April 2002.

P. Ravikumar, G. Raskutti, M. J. Wainwright, and B. Yu.High-dimensional covariance estimation by minimizingL1-penalized log-determinant. In NIPS, 2008.

P. Ravikumar, M. J. Wainwright, and J. D. Lafferty. High-dimensional ising model selection using L1-regularizedlogistic regression. Ann. Statist., 38(3), 2010.

J. Schafer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications forfunctional genomics. Stat. App. Gen. Mol. Biol., 4, 2005.

M. Schmidt, E. van den Berg, M. P. Friedlander, andK. Murphy. Optimizing costly functions with simple con-straints:a limited-memory projected quasi-newton algo-rithm. In AISTATS, volume 5, 2009.

T. Shimamura, S. Imoto, R. Yamaguchi, and S. Miyano.Weighted lasso in graphical gaussian modeling for largegene network estimation based on microarray data.Genome Inform., 19, 2007.

T. Shimamura, S. Imoto, R. Yamaguchi, A. Fujita, M. Na-gasaki, and S. Miyano. Recursive regularization for in-ferring gene networks from time-course gene expressionprofiles. BMC Systems Biology, 3(1):41+, 2009.

P. T. Spellman, G. Sherlock, M. Q. Zhang, V. R. Iyer,K. Anders, M. B. Eisen, P. O. Brown, D. Botstein, andB. Futcher. Comprehensive identification of cell cycle-regulated genes of the yeast saccharomyces cerevisiae bymicroarray hybridization. Mol. Biol. Cell, 9(12):3273–3297, 1998.

R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, andK. Knight. Sparsity and smoothness via the fused lasso.J. Royal Statist. Soc. B, 67(1):91–108, 2005.

B. A. Turlach, W. N. Venables, and S. J. Wright. Simul-taneous variable selection. Technometrics, 47(3), 2005.

C. F. J. Wu. On the convergence properties of the EMalgorithm. Ann. Statist., 11(1), 1983.

M. Yuan and Y. Lin. Model selection and estimation inregression with grouped variables. J. Royal Stat. Soc.B, 68:49–67, 2006.

M. Yuan and Y. Lin. Model selection and estimation inthe gaussian graphical model. Biometrika, 1, 2007.

H. Zou. The adaptive lasso and its oracle properties. JASA,101:1418–1429, December 2006.

H. Zou and R. Li. One-step sparse estimates in nonconcavepenalized likelihood models. Ann. Statist., 36(4), 2008.

M. Zou and S. D. Conzen. A new dynamic Bayesian net-work (DBN) approach for identifying gene regulatorynetworks from time course microarray data. Bioinfor-matics, 21(1):71–79, January 2005.