Modeling Sparse Deviations for Compressed Sensing using Generative Models

Manik Dhar 1 Aditya Grover 1 Stefano Ermon 1

AbstractIn compressed sensing, a small number of lin-ear measurements can be used to reconstruct anunknown signal. Existing approaches leverageassumptions on the structure of these signals,such as sparsity or the availability of a genera-tive model. A domain-specific generative modelcan provide a stronger prior and thus allow forrecovery with far fewer measurements. However,unlike sparsity-based approaches, existing meth-ods based on generative models guarantee exactrecovery only over their support, which is typi-cally only a small subset of the space on whichthe signals are defined. We propose Sparse-Gen, aframework that allows for sparse deviations fromthe support set, thereby achieving the best of bothworlds by using a domain specific prior and allow-ing reconstruction over the full space of signals.Theoretically, our framework provides a new classof signals that can be acquired using compressedsensing, reducing classic sparse vector recovery toa special case and avoiding the restrictive supportdue to a generative model prior. Empirically, weobserve consistent improvements in reconstruc-tion accuracy over competing approaches, espe-cially in the more practical setting of transfercompressed sensing where a generative modelfor a data-rich, source domain aids sensing on adata-scarce, target domain.

1. IntroductionIn many real-world domains, data acquisition is costly. Forinstance, magnetic resonance imaging (MRI) requires scantimes proportional to the number of measurements, whichcan be significant for patients (Lustig et al., 2008). Geo-physical applications like oil drilling require expensive sim-ulation of seismic waves (Qaisar et al., 2013). Such appli-

1Computer Science Department, Stanford University, CA, USA.Correspondence to: Manik Dhar ,Aditya Grover .

Proceedings of the 35 th International Conference on MachineLearning, Stockholm, Sweden, PMLR 80, 2018. Copyright 2018by the author(s).

cations, among many others, can benefit significantly fromcompressed sensing techniques to acquire signals efficiently(Candès & Tao, 2005; Donoho, 2006; Candès et al., 2006).

In compressed sensing, we wish to acquire an n-dimensionalsignal x ∈ Rn using only m� n measurements linear in x.The measurements could potentially be noisy, but even inthe absence of any noise we need to impose additional struc-ture on the signal to guarantee unique recovery. Classicalresults on compressed sensing impose structure by assum-ing the underlying signal to be approximately l-sparse insome known basis, i.e., the l-largest entries dominate therest. For instance, images and audio signals are typicallysparse in the wavelet and Fourier basis respectively (Mallat,2008). If the matrix of linear vectors relating the signal andmeasurements satisfies certain mild conditions, then onecan provably recover x with only m = O(l log nl ) measure-ments using LASSO (Tibshirani, 1996; Candès & Tao, 2005;Donoho, 2006; Candès et al., 2006; Bickel et al., 2009).

Alternatively, structural assumptions on the signals beingsensed can be learned from data, e.g., using a dataset oftypical signals (Baraniuk et al., 2010; Peyre, 2010; Chenet al., 2010; Yu & Sapiro, 2011). Particularly relevant to thiswork, Bora et al. (2017) proposed an approach where struc-ture is provided by a deep generative model learned fromdata. Specifically, the underlying signal x being sensed is as-sumed to be close to the range of a deterministic function ex-pressed by a pretrained, latent variable modelG : Rk → Rnsuch that x ≈ G(z) where z ∈ Rk denote the latent vari-ables. Consequently, the signal x is recovered by optimizingfor a latent vector z that minimizes the `2 distance betweenthe measurements corresponding to G(z) and the actualones. Even though the objective being optimized in thiscase is non-convex, empirical results suggest that the recon-struction error decreases much faster than LASSO-basedrecovery as we increase the number of measurements.

A limitation of the above approach is that the recoveredsignal is constrained to be in the range of the generator func-tion G. Hence, if the true signal being sensed is not in therange of G, the algorithm cannot drive the reconstructionerror to zero even when m ≥ n (even if we ignore errordue to measurement noise and non-convex optimization).This is also observed empirically, as the reconstruction errorof generative model-based recovery saturates as we keep

Modeling Sparse Deviations for Compressed Sensing using Generative Models

increasing the number of measurements m. On the otherhand, LASSO-based recovery continues to shrink the er-ror with increasing number of measurements, eventuallyoutperforming the generative model-based recovery.

To overcome this limitation, we propose a framework thatallows recovery of signals with sparse deviations from theset defined by the range of the generator function. The re-covered signals have the general form of G(ẑ) + ν̂, whereν̂ ∈ Rn is a sparse vector. This allows the recovery al-gorithm to consider signals away from the range of thegenerator function. Similar to LASSO, we relax the hard-ness in optimizing for sparse vectors by minimizing the`1 norm of the deviations. Unlike LASSO-based recov-ery, we can exploit the rich structure imposed by a (deep)generative model (at the expense of solving a hard optimiza-tion problem if G is non-convex). In fact, we show thatLASSO-based recovery is a special case of our frameworkif the generator function G maps all z to the origin. Unlikegenerative model-based recovery, the signals recovered byour algorithm are not constrained to be in the range of thegenerator function.

Our proposed algorithm, referred to as Sparse-Gen, hasdesirable theoretical properties and empirical performance.Theoretically, we derive upper bounds on the reconstructionerror for an optimal decoder with respect to the proposedmodel and show that this error vanishes with m = n mea-surements. We confirm our theory empirically, whereinwe find that recovery using Sparse-Gen with variational au-toencoders (Kingma & Welling, 2014) as the underlyinggenerative model outperforms both LASSO-based and gen-erative model-based recovery in terms of the reconstructionerrors for the same number of measurements for MNISTand Omniglot datasets. Additionally, we observe signifi-cant improvements in the more practical and novel task oftransfer compressed sensing where a generative model ona data-rich, source domain provides a prior for sensing adata-scarce, target domain.

2. PreliminariesIn this section, we review the necessary background andprior work in modeling domain specific structure in com-pressed sensing. We are interested in solving the followingsystem of equations,

y = Ax (1)

where x ∈ Rn is the signal of interest being sensed throughmeasurements y ∈ Rm, and A ∈ Rm×n is a measurementmatrix. For efficient acquisition of signals, we will designmeasurement matrices such that m � n. However, thesystem is under-determined whenever rank(A) < n. Hence,unique recovery requires additional assumptions on x. Wenow discuss two ways to model the structure of x.

Sparsity. Sparsity in a well-chosen basis is natural in manydomains. For instance, natural images are sparse in thewavelet basis whereas audio signals exhibit sparsity in theFourier basis (Mallat, 2008). Hence, it is natural to assumethe domain of signals x we are interested in recovering is

Sl(0) = {x : ‖x− 0‖0 ≤ l}. (2)

This is the set of l-sparse vectors with the `0 distance mea-sured from the origin. Such assumptions dominate the priorliterature in compressed sensing and can be further relaxedto recover approximately sparse signals (Candès & Tao,2005; Donoho, 2006; Candès et al., 2006).

Latent variable generative models. A latent variablemodel specifies a joint distribution Pθ(x, z) over the ob-served data x (e.g., images) and a set of latent variablesz ∈ Rk (e.g., features). Given a training set of signals{x1, · · · , xM}, we can learn the parameters θ of such amodel, e.g., via maximum likelihood. When Pθ(x, z) isparameterized using deep neural networks, such generativemodels can effectively model complex, high-dimensionalsignal distributions for modalities such as images and au-dio (Kingma & Welling, 2014; Goodfellow et al., 2014).

Given a pretrained latent variable generative model withparameters θ, we can associate a generative model functionG : Rk → Rn mapping a latent vector z to the mean of theconditional distribution Pθ(x|z). Thereafter, the space ofsignals that can be recovered with such a model is given bythe range of the generator function,

SG = {G(z) : z ∈ Rk}. (3)

Note that the set is defined with respect to the latent vectorsz, and we omit the dependence of G on the parameters θ(which are fixed for a pretrained model) for brevity.

2.1. Recovery algorithms

Signal recovery in compressed sensing algorithm typicallyinvolves solving an optimization problem consistent withthe modeling assumptions on the domain of the signalsbeing sensed.

Sparse vector recovery using LASSO. Under the assump-tions of sparsity, the signal x can be recovered by solvingan `0 minimization problem (Candès & Tao, 2005; Donoho,2006; Candès et al., 2006).

minx‖x‖0

s.t. Ax = y. (4)

The objective above is however NP-hard to optimize, andhence, it is standard to consider a convex relaxation,

minx‖x‖1

s.t. Ax = y. (5)

Modeling Sparse Deviations for Compressed Sensing using Generative Models

In practice, it is common to solve the Lagrangian of theabove problem. We refer to this method as LASSO-basedrecovery due to similarities of the objective in Eq. (5) tothe LASSO regularization used broadly in machine learn-ing (Tibshirani, 1996). LASSO-based recovery is the pre-dominant technique for recovering sparse signals since itinvolves solving a tractable convex optimization problem.

In order to guarantee unique recovery to the underdeter-mined system in Eq. (1), the measurement matrix A is de-signed to satisfy the Restricted Isometry Property (RIP) orthe Restricted Eigenvalue Condition (REC) for l-sparse ma-trices with high probability (Candès & Tao, 2005; Bickelet al., 2009). We define these conditions below.Definition 1. Let Sl(0) ⊂ Rn be the set of l-sparse vectors.For some parameter α ∈ (0, 1), a matrix A ∈ Rm×n is saidto satisfy RIP(l, α) if ∀ x ∈ Sl(0),

(1− α)‖x‖2 ≤ ‖Ax‖2 ≤ (1 + α)‖x‖2.

Definition 2. Let Sl(0) ⊂ Rn be the set of l-sparse vectors.For some parameter γ > 0, a matrix A ∈ Rm×n is said tosatisfy REC(l, γ) if ∀ x ∈ Sl(0),

‖Ax‖2 ≥ γ‖x‖2.

Intuitively, RIP implies that A approximately preservesEuclidean norms for sparse vectors and REC implies thatsparse vectors are far from the nullspace of A. Many classesof matrices satisfy these conditions with high probability,including random Gaussian and Bernoulli matrices whereevery entry of the matrix is sampled from a standard normaland uniform Bernoulli distribution respectively (Baraniuket al., 2008).

Generative model vector recovery using gradient de-scent. If the signals being sensed are assumed to lie closeto the range SG of a generative model function G as definedin Eq. (3) , then we can recover the best approximation tothe true signal by `2-minimization over z,

minz‖AG(z)− y‖22. (6)

The function G is typically expressed as a deep neural net-work which makes the overall objective non-convex, butdifferentiable almost everywhere w.r.t z. In practice, goodreconstructions can be recovered by gradient-based opti-mization methods. We refer to this method proposed byBora et al. (2017) as generative model-based recovery.

To guarantee unique recovery, generative model-based re-covery makes two key assumptions. First, the generatorfunctionG is assumed to be L-Lipschitz, i.e., ∀ z1, z2 ∈ Rk,

‖G(z1)−G(z2)‖2 ≤ L‖z1 − z2‖2.

Secondly, the measurement matrix A is designed to satisfythe Set-Restricted Eigenvalue Condition (S-REC) with highprobability (Bora et al., 2017).

Definition 3. Let S ⊆ Rn. For some parameters γ >0, δ ≥ 0, a matrix A ∈ Rm×n is said to satisfy the S-REC(S, γ, δ) if ∀ x1, x2 ∈ S,

‖A(x1 − x2)‖2 ≥ γ‖x1 − x2‖2 − δ.

S-REC generalizes REC to an arbitrary set of vectors S asopposed to just considering the set of approximately sparsevectors Sl(0) and allowing an additional slack term δ. Inparticular, S is chosen to be the range of the generatorfunction G for generative model-based recovery.

3. The Sparse-Gen frameworkThe modeling assumptions based on sparsity and gener-ative modeling discussed in the previous section can belimiting in many cases. On one hand, sparsity assumes arelatively weak prior over the signals being sensed. Empiri-cally, we observe that the recovered signals xL have largereconstruction error ‖xL− x‖22 especially when the numberof measurements m is small. On the other hand, generativemodels imposes a very strong, but rigid prior which workswell when the number of measurements is small. However,the performance of the corresponding recovery methodssaturates with increasing measurements since the recoveredsignal xG = G(zG) is constrained to lie in the range of thegenerator function G. If zG ∈ Rk is the optimum valuereturned by an optimization procedure for Eq. (6), then thereconstruction error ‖xG − x‖22 is limited by the dimen-sionality of the latent space and the quality of the generatorfunction.

To sidestep the above limitations, we consider a strictly moreexpressive class of signals by allowing sparse deviationsfrom the range of a generator function. Formally, the domainof the recovered signals is given by,

Sl,G = ∪z∈Dom(G)Sl(G(z)) (7)

where Sl(G(z)) denotes the set of sparse vectors centeredon G(z) and z varies over the domain of G (typically Rk).We refer to this modeling assumption and the consequentalgorithmic framework for recovery as Sparse-Gen.

Based on this modeling assumption, we will recover signalsof the form G(z) + ν for some ν ∈ Rn that is preferablysparse. Specifically, we consider the optimization of a hy-brid objective,

minz,ν‖ν‖0

s.t. A (G(z) + ν) = y. (8)

In the above optimization problem the objective is non-convex and non-differentiable, while the constraint is non-convex (for general G), making the above optimization

Modeling Sparse Deviations for Compressed Sensing using Generative Models

(a) LASSO (b) Generative model (c) Sparse-Gen

Figure 1. LASSO vs. Generative model vs. Sparse-Gen recovery. Unlike LASSO, Sparse-Gen imposes a stronger prior on the signalsbeing sensed (shaded grey regions). Unlike generative model, the recovered signals are not constrained to lie on the range of thegenerator function (red points). Aspects of visualization due to eigenvalue conditions not shown for simplicity.

problem hard to solve. To ease the optimization problem,we propose two modifications. First, we relax the `0 mini-mization to an `1 minimization similar to LASSO.

minz,ν‖ν‖1

s.t. A (G(z) + ν) = y. (9)

Next, we square the non-convex constraint on both sidesand consider the Lagrangian of the above problem to get thefinal unconstrained optimization problem for Sparse-Gen,

minz,ν‖ν‖1 + λ‖A (G(z) + ν)− y‖22 (10)

where λ is the Lagrange multiplier.

The above optimization problem is non-differentiable w.r.t.ν and non-convex w.r.t. z (if G is non-convex). In practice,it can be solved in practice using gradient descent (since thenon-differentiability is only at a finite number of points) orusing sequential convex programming (SCP). SCP is an ef-fective heuristic for non-convex problems where the convexportions of the problem are solved using a standard convexoptimization technique (Boyd & Vandenberghe, 2004). Inthe case of Eq. (10), the optimization w.r.t. ν (for fixed z) isa convex optimization problem whereas the non-convexitytypically involves differentiable terms (w.r.t. z) if G is adeep neural network. Empirically, we find excellent recov-ery by standard first order gradient-based methods (Duchiet al., 2011; Tieleman & Hinton, 2012; Kingma & Ba, 2015).

Unlike LASSO-based recovery which recovers only sparsesignals, Sparse-Gen can impose a stronger domain-specificprior using a generative model. If we fix the generator func-tion to map all z to the origin, we recover LASSO-basedrecovery as a special case of Sparse-Gen. Additionally,Sparse-Gen is not constrained to recover signals over therange of G, as in the case of generative model-based recov-ery. In fact, it can recover signals with sparse deviationsfrom the range of G. Note that the sparse deviations can be

defined in a basis different from the canonical basis. In suchcases, we consider the following optimization problem,

minz,ν‖Bν‖1 + λ‖A (G(z) + ν)− y‖22 (11)

where B is a change of basis matrix that promotes spar-sity of the vector Bν. Figure 1 illustrates the differencesin modeling assumptions between Sparse-Gen and otherframeworks.

4. Theoretical AnalysisThe proofs for all results in this section are given in theAppendix. Our analysis and experiments account for mea-surement noise � in compressed sensing, i.e.,

y = Ax+ �. (12)

Let ∆ : Rm → Rn denote an arbitrary decoding functionused to recover the true signal x from the measurementsy ∈ Rm. Our analysis will upper bound the `2-error inrecovery incurred by our proposed framework using mixednorm guarantees (in particular, `2/`1). To this end, we firststate some key definitions. Define the least possible `1 errorfor recovering x under the Sparse-Gen modeling as,

σSl,G(x) = infx̂∈Sl,G

‖x− x̂‖1

where the optimal x̂ is the closest point to x in the alloweddomain Sl,G. We now state the main lemma guiding thetheoretical analysis.Lemma 1. Given a function G : Rk → Rn and mea-surement noise � with ‖�‖2 ≤ �max, let A be any matrixthat satisfies S-REC(S1.5l,G, (1− α), δ) and RIP(2l, α) forsome α ∈ (0, 1), l > 0. Then, there exists a decoder∆ : Rm → Rn such that,

‖x−∆(Ax+ �)‖2 ≤ (2l)−1/2C0σl,G(x) + C1�max + δ′

for all x ∈ Rn, where C0 = 2((1+α)(1−α)−1 +1), C1 =2(1− α)−1, and δ′ = δ(1− α)−1.

Modeling Sparse Deviations for Compressed Sensing using Generative Models

The above lemma shows that there exists a decoder such thatthe error in recovery can be upper bounded for measurementmatrices satisfying S-REC and RIP. Note that Lemma 1 onlyguarantees the existence of such a decoder and does not pre-scribe an optimization algorithm for recovery. Apart fromthe errors due to the bounded measurement noise �max anda scaled slack term appearing in the S-REC condition δ′,the major term in the upper bound corresponds to (up toconstants) the minimum possible error incurred by the bestpossible recovery vector in Sl,G given by σl,G(x). Similarterms appear invariably in the compressed sensing literatureand are directly related to the modeling assumptions regard-ing x (for example, Theorem 8.3 in Cohen et al. (2009)).

Our next lemma shows that random Gaussian matrices sat-isfy the S-REC (over the range of Lipschitz generativemodel functions) and RIP conditions with high probabil-ity for G with bounded domain, both of which together aresufficient conditions for Lemma 1 to hold.

Lemma 2. LetG : Bk(r)→ Rn be anL-Lipschitz functionwhere Bk(r) = {z | z ∈ Rk, ‖z‖2 ≤ r} is the `2-norm ballin Rk. For α ∈ (0, 1), if

m = O

(1

α2

(k log

(Lr

δ

)+ l log(n/l)

))then a random matrix A ∈ Rm×n with i.i.d. entries suchthat Aij ∼ N

(0, 1m

)satisfies the S-REC(S1.5l,G, 1− α, δ)

and RIP(2l, α) with 1− e−Ω(α2m) probability.

Using Lemma 1 and Lemma 2, we can bound the error dueto decoding with generative models and random Gaussianmeasurement matrices in the following result.

Theorem 1. Let G : Bk(r)→ Rn be an L-Lipschitz func-tion. For any α ∈ (0, 1), l > 0, let A ∈ Rm×n be a randomGaussian matrix with

m = O

(1

α2

(k log

(Lr

δ

)+ l log(n/l)

))rows of i.i.d. entries scaled such that Ai,j ∼ N(0, 1/m).Let ∆ be the decoder satisfying Lemma 1. Then, we havewith 1− e−Ω(α2m) probability,

‖x−∆(Ax+ �)‖2 ≤ (2l)−1/2C0σl,G(x) + C1�max + δ′

for all x ∈ Rn, ‖�‖2 ≤ �max, where C0, C1, γ, δ′ are con-stants defined in Lemma 1.

From the above lemma, we see that the number of measure-ments needed to guarantee upper bounds on the reconstruc-tion error of any signal with high probability depends on twoterms. The first term includes dependence on the Lipschitzconstant L of the generative model function G. A high Lips-chitz constant makes recovery harder (by requiring a larger

number of measurements), but only contributes logarithmi-cally. The second term, typical of results in sparse vectorrecovery, shows a logarithmic growth on the dimensionalityn of the signals. Ignoring logarithmic dependences and con-stants, recovery using Sparse-Gen requires about O(k + l)measurements for recovery. Note that Theorem 1 assumesaccess to an optimization oracle for decoding. In practice,we consider the solutions returned by gradient-based op-timization methods to a non-convex objective defined inEq. (11) that are not guaranteed to correspond to the opti-mal decoding in general.

Finally, we obtain tighter bounds for the special case whenG is expressed using a neural network with only ReLU acti-vations. These bounds do not rely explicitly on the Lipschitzconstant L or require the domain of G to be bounded.

Theorem 2. If G : Rk → Rn is a neural network of depthd with only ReLU activations and at most c nodes in eachlayer, then the guarantees of Theorem 1 hold for

m = O

(1

α2

((k + l)d log c+ (k + l) log(n/l)

)).

Our theoretical analysis formalizes the key properties of re-covering signals using Sparse-Gen. As shown in Lemma 1,there exists a decoder for recovery based on such model-ing assumptions that extends recovery guarantees based onvanilla sparse vector recovery and generative model-basedrecovery. Such recovery requires measurement matrices thatsatisfy both the RIP and S-REC conditions over the set ofvectors that deviate in sparse directions from the range of agenerative model function. In Theorems 1-2, we observedthat the number of measurements required to guarantee re-covery with high probability grow almost linearly (withsome logarithmic terms) with the latent space dimensional-ity k of the generative model and the permissible sparsity lfor deviating from the range of the generative model.

5. Experimental EvaluationWe evaluated Sparse-Gen for compressed sensing of high-dimensional signals from the domain of benchmark imagedatasets. Specifically, we considered the MNIST datasetof handwritten digits (LeCun et al., 2010) and the OM-NIGLOT dataset of handwritten characters (Lake et al.,2015). Both these datasets have the same data dimensional-ity (28× 28), but significantly different characteristics. TheMNIST dataset has fewer classes (10 digits from 0-9) asopposed to Omniglot which shows greater diversity (1623characters across 50 alphabets). Additional experimentswith generative adversarial networks on the CelebA datasetare reported in the Appendix.

Baselines. We considered methods based on sparse vectorrecovery using LASSO (Tibshirani, 1996; Candès & Tao,

Modeling Sparse Deviations for Compressed Sensing using Generative Models

50 100

200

300

400

500

750

Number of measurements

0.000

0.025

0.050

0.075

0.100

0.125

ℓ 1 e

rror (

per p

ixel

)

LASSOVAESparse-VAE

(a) MNIST - `1

50 100

200

300

400

500

750

Number of measurements

0.000.050.100.150.200.250.300.35

ℓ 2 e

rror (

per p

ixel

)

(b) MNIST - `2

50 100

200

300

400

500

750

Number of measurements

0.0

0.2

0.4

0.6

0.8

1.0

ℓ ∞ e

rror

(c) MNIST - `∞

50 100

200

300

400

500

750

Number of measurements

0.00

0.02

0.04

0.06

0.08

0.10

ℓ 1 e

rror (

per p

ixel

)

(d) Omniglot - `1

50 100

200

300

400

500

750

Number of measurements

0.00

0.05

0.10

0.15

0.20

0.25ℓ 2

erro

r (pe

r pix

el)

(e) Omniglot - `2

50 100

200

300

400

500

750

Number of measurements

0.0

0.2

0.4

0.6

0.8

1.0

ℓ ∞ e

rror

(f) Omniglot - `∞

Figure 2. Reconstruction error using `1 (left), `2 (center), and `∞ (right) norms for MNIST (top) and Omniglot (bottom). Theperformance of Sparse-Gen is better or competitive with both generative model-based methods and sparse vector recovery methods.

2005) and generative model based recovery using variationalautoencoders (VAE) (Kingma & Welling, 2014; Bora et al.,2017). For VAE training, we used the standard train/held-outsplits of both datasets. Compressed sensing experiments thatwe report were performed on the entire test set of images.The architecture and other hyperparameter details are givenin the Appendix.

Experimental setup. For the held-out set of instances, weartificially generated measurements y through a random ma-trix A ∈ Rm×n with entries sampled i.i.d. from a Gaussianwith zero mean and standard deviation of 1/m. Measure-ment noise is sampled from zero mean and diagonal scalarcovariance matrix with entries as 0.01. For evaluation, wereport the reconstruction error measured as ‖x̂− x‖p wherex̂ is the recovered signal and p is a norm of interest, varyingthe number of measurementsm from 50 to the highest valueof 750. We report results for the p = {1, 2,∞} norms.

We evaluated sensing of both continuous signals (MNIST)with pixel values in range [0, 1] and discrete signals (Om-niglot) with binary pixel values {0, 1}. For all algorithmsconsidered, recovery was performed by optimizing over acontinuous space. In the case of sparse recovery methods(including Sparse-Gen) it is possible that unconstrained op-timization returns signals outside the domain of interest, inwhich case they are projected to the required domain bysimple clipping, i.e., any signal less than zero is clipped to0 and similarly any signal greater than one is clipped to 1.

Results and Discussion. The reconstruction errors for vary-ing number of measurements are given in Figure 2. Consis-tent with the theory, the strong prior in generative model-based recovery methods outperforms the LASSO-basedmethods for sparse vector recovery. In the regime of lowmeasurements, the performance of algorithms that can incor-porate the generative model prior dominates over methodsmodeling sparsity using LASSO. The performance of plaingenerative model-based methods however saturates withincreasing measurements, unlike Sparse-Gen and LASSOwhich continue to shrink the error. The trends are consistentfor both MNIST and Omniglot, although we observe therelative magnitudes of errors in the case of Omniglot aremuch higher than that of MNIST. This is expected due tothe increased diversity and variations of the structure ofthe signals being sensed in the case of Omniglot. We alsoobserve the trends to be consistent across the various normsconsidered.

5.1. Transfer compressed sensing

One of the primary motivations for compressive sensing is todirectly acquire the signals using few measurements. On thecontrary, learning a deep generative model requires accessto large amounts of training data. In several applications,getting the data for training a generative model might notbe feasible. Hence, we test the generative model-basedrecovery on the novel task of transfer compressed sensing.

Modeling Sparse Deviations for Compressed Sensing using Generative Models

50 100

200

300

400

500

750

Number of measurements

0.00

0.02

0.04

0.06

0.08

0.10

ℓ 1 e

rror (

per p

ixel

)

(a) mnist2omni - `1

50 100

200

300

400

500

750

Number of measurements

0.00

0.05

0.10

0.15

0.20

0.25

ℓ 2 e

rror (

per p

ixel

)

(b) mnist2omni - `2

50 100

200

300

400

500

750

Number of measurements

0.0

0.2

0.4

0.6

0.8

1.0

ℓ ∞ e

rror

(c) mnist2omni - `∞

50 100

200

300

400

500

750

Number of measurements

0.000

0.025

0.050

0.075

0.100

0.125

ℓ 1 e

rror (

per p

ixel

)

LASSOVAESparse-VAE

(d) omni2mnist - `1

50 100

200

300

400

500

750

Number of measurements

0.000.050.100.150.200.250.300.35

ℓ 2 e

rror (

per p

ixel

)

(e) omni2mnist - `2

50 100

200

300

400

500

750

Number of measurements

0.0

0.2

0.4

0.6

0.8

1.0

ℓ ∞ e

rror

(f) omni2mnist - `∞

Figure 3. Reconstruction error in terms of `1 (left), `2 (center), and `∞ (right) norms for transfer compressed sensing from MNIST assource to Omniglot as target (top) and Omniglot as source and MNIST as target (bottom). Under the `2 and `∞ metric, Sparse-VAEbeats both VAE and LASSO. Under the `1 metric, performance is slightly poorer until 100 measurements.

Experimental setup. We train the generative model on asource domain (assumed to be data-rich) and related to adata-hungry target domain we wish to sense. Given thematching dimensions of MNIST and Omniglot, we conductexperiments transferring from MNIST (source) to Omniglot(target) and vice versa.

Results and Discussion. The reconstruction errors for thenorms considered are given in Figure 3. For both the source-target pairs, we observe that the Sparse-Gen consistently per-forms well. Vanilla generative model-based recovery showshardly an improvements with increasing measurements. Wecan qualitatively see this phenomena for transferring fromMNIST (source) to Omniglot (target) in Figure 4. With onlym = 100 measurements, all models perform poorly andgenerative model based methods particularly continue tosense images similar to MNIST. On the other hand, there is anoticeable transition at m = 200 measurements for Sparse-VAE where it adapts better to the domain being sensed thanplain generative model-based recovery and achieves lowerreconstruction error.

6. Related WorkSince the introduction of compressed sensing over a decadeago, there has been a vast body of research studying variousextensions and applications (Candès & Tao, 2005; Donoho,

2006; Candès et al., 2006). This work explores the effectof modeling different structural assumptions on signals intheory and practice.

Themes around sparsity in a well-chosen basis has drivenmuch of the research in this direction. For instance, theparadigm of model-based compressed sensing accounts forthe interdependencies between the dimensions of a sparsedata signal (Baraniuk et al., 2010; Duarte & Eldar, 2011;Gilbert et al., 2017). Alternatively, adaptive selection ofbasis vectors from a dictionary that best capture the struc-ture of the particular signal being sensed has also beenexplored (Peyre, 2010; Tang et al., 2013). Many of thesemethods have been extended to recovery of structured ten-sors (Zhang et al., 2013; 2014). In another prominent lineof research involving Bayesian compressed sensing, thesparseness assumption is formalized by placing sparseness-promoting priors on the signals (Ji et al., 2008; He & Carin,2009; Babacan et al., 2010; Baron et al., 2010).

Research exploring structure beyond sparsity is relativelyscarce. Early works in this direction can be traced to Bara-niuk & Wakin (2009) who proposed algorithms for recov-ering signals lying on a smooth manifold. The generativemodel-based recovery methods consider functions that donot necessarily define manifolds since the range of a gener-ator function could intersect with itself. Yu & Sapiro (2011)coined the term statistical compressed sensing and proposed

Modeling Sparse Deviations for Compressed Sensing using Generative Models

Orig

inal

(a) Original

Lass

oV

AE

Spar

se-V

AE

(b) Reconstructions with m = 100 measurements.

Lass

oV

AE

Spar

se-V

AE

(c) Reconstructions with m = 200 measurements.

Figure 4. The image matrix above shows the recovery of Omniglot images (top) using a VAE trained over MNIST using 100 measurements(middle) and 200 measurements (bottom). We note that with 100 measurements recovery is very poor across the board. With 200measurements, all three methods show significant improvement and Sparse-VAE shows greater sensitivity to the domain being sensed.

algorithms for efficient sensing of signals from a mixture ofGaussians. The recent work in deep generative model-basedrecovery differs in key theoretical aspects as well in the useof a more expressive family of models based on neural net-works. A related recent work by Hand & Voroninski (2017)provides theoretical guarantees on the solution recovered forsolving non-convex linear inverse problems with deep gen-erative priors. Empirical advances based on well-designeddeep neural network architectures that sacrifice many of thetheoretical guarantees have been proposed for applicationssuch as MRI (Mardani et al., 2017; 2018). Many recentmethods propose to learn mappings of signals to measure-ments using neural networks, instead of restricting them tobe linear, random matrices (Mousavi et al., 2015; Kulkarniet al., 2016; Chang et al., 2017; Lu et al., 2018).

Our proposed framework bridges the gap between algo-rithms that model structure using sparsity and enjoy goodtheoretical properties with advances in deep generative mod-els, in particular their use for compressed sensing.

7. Conclusion and Future WorkThe use of deep generative models as priors for compressedsensing presents a new outlook on algorithms for inexpen-

sive data acquisition. In this work, we showed that thesepriors can be used in conjunction with classical modelingassumptions based on sparsity. Our proposed framework,Sparse-Gen, generalizes both sparse vector recovery andrecovery using generative models by allowing for sparse de-viations from the range of a generative model function. Thebenefits of using such modeling assumptions are observedboth theoretically and empirically.

In the future, we would like to design algorithms that canbetter model the structure within sparse deviations. Follow-up work in this direction can benefit from the vast bodyof prior work in structured sparse vector recovery (Duarte& Eldar, 2011). From a theoretical perspective, a betterunderstanding of the non-convexity resulting from genera-tive model-based recovery can lead to stronger guaranteesand consequently better optimization algorithms for recov-ery. Finally, it would be interesting to extend Sparse-Genfor compressed sensing of other data modalities such asgraphs for applications in network tomography and recon-struction (Xu et al., 2011). Real-world graph networks aretypically sparse in the canonical basis and can be modeled ef-fectively using deep generative models (Grover et al., 2018),which is consistent with the modeling assumptions of theSparse-Gen framework.

Modeling Sparse Deviations for Compressed Sensing using Generative Models

Acknowledgements

We are thankful to Tri Dao, Jonathan Kuck, Daniel Levy,Aditi Raghunathan, and Yang Song for helpful commentson early drafts. This research was supported by Intel Cor-poration, TRI, a Hellman Faculty Fellowship, ONR, NSF(#1651565, #1522054, #1733686 ) and FLI (#2017-158687).AG is supported by a Microsoft Research PhD Fellowship.

ReferencesAbadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z.,

Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin,M., et al. Tensorflow: Large-scale machine learningon heterogeneous distributed systems. arXiv preprintarXiv:1603.04467, 2016.

Andersen, M. S., Dahl, J., and Vandenberghe, L. Cvxopt: Apython package for convex optimization, version 1.1. 6.Available at cvxopt. org, 54, 2013.

Babacan, S. D., Molina, R., and Katsaggelos, A. K.Bayesian compressive sensing using Laplace priors. IEEETransactions on Image Processing, 19(1):53–63, 2010.

Baraniuk, R., Davenport, M., DeVore, R., and Wakin, M.A simple proof of the restricted isometry property forrandom matrices. Constructive Approximation, 28(3):253–263, 2008.

Baraniuk, R. G. and Wakin, M. B. Random projections ofsmooth manifolds. Foundations of computational mathe-matics, 9(1):51–77, 2009.

Baraniuk, R. G., Cevher, V., Duarte, M. F., and Hegde, C.Model-based compressive sensing. IEEE Transactionson Information Theory, 56(4):1982–2001, 2010.

Baron, D., Sarvotham, S., and Baraniuk, R. G. Bayesiancompressive sensing via belief propagation. IEEE Trans-actions on Signal Processing, 58(1):269–280, 2010.

Bickel, P. J., Ritov, Y., and Tsybakov, A. B. Simultaneousanalysis of lasso and Dantzig selector. The Annals ofStatistics, pp. 1705–1732, 2009.

Bora, A., Jalal, A., Price, E., and Dimakis, A. G. Com-pressed sensing using generative models. In InternationalConference on Machine Learning, 2017.

Boyd, S. and Vandenberghe, L. Convex optimization. Cam-bridge university press, 2004.

Candès, E. J. and Tao, T. Decoding by linear programming.IEEE Transactions on Information Theory, 51(12):4203–4215, 2005.

Candès, E. J., Romberg, J., and Tao, T. Robust uncertaintyprinciples: Exact signal reconstruction from highly in-complete frequency information. IEEE Transactions onInformation Theory, 52(2):489–509, 2006.

Chang, J. R., Li, C.-L., Poczos, B., Kumar, B. V., andSankaranarayanan, A. C. One network to solve themallsolving linear inverse problems using deep projectionmodels. arXiv preprint, 2017.

Chen, M., Silva, J., Paisley, J., Wang, C., Dunson, D., andCarin, L. Compressive sensing on manifolds using anonparametric mixture of factor analyzers: Algorithmand performance bounds. IEEE Transactions on SignalProcessing, 58(12):6140–6155, 2010.

Cohen, A., Dahmen, W., and DeVore, R. Compressedsensing and best k-term approximation. Journal of theAmerican mathematical society, 22(1):211–231, 2009.

Donoho, D. L. Compressed sensing. IEEE Transactions onInformation Theory, 52(4):1289–1306, 2006.

Duarte, M. F. and Eldar, Y. C. Structured compressed sens-ing: From theory to applications. IEEE Transactions onSignal Processing, 59(9):4053–4085, 2011.

Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradientmethods for online learning and stochastic optimization.Journal of Machine Learning Research, 12(Jul):2121–2159, 2011.

Gilbert, A. C., Zhang, Y., Lee, K., Zhang, Y., and Lee, H.Towards understanding the invertibility of convolutionalneural networks. arXiv preprint arXiv:1705.08664, 2017.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,Warde-Farley, D., Ozair, S., Courville, A., and Bengio,Y. Generative adversarial nets. In Advances in NeuralInformation Processing Systems, 2014.

Grover, A., Zweig, A., and Ermon, S. Graphite: Iter-ative generative modeling of graphs. arXiv preprintarXiv:1803.10459, 2018.

Hand, P. and Voroninski, V. Global guarantees for enforcingdeep generative priors by empirical risk. arXiv preprintarXiv:1705.07576, 2017.

He, L. and Carin, L. Exploiting structure in wavelet-basedbayesian compressive sensing. IEEE Transactions onSignal Processing, 57(9):3488–3497, 2009.

Ji, S., Xue, Y., and Carin, L. Bayesian compressive sensing.IEEE Transactions on Signal Processing, 56(6):2346–2356, 2008.

Modeling Sparse Deviations for Compressed Sensing using Generative Models

Kingma, D. and Ba, J. Adam: A method for stochasticoptimization. In International Conference on LearningRepresentations, 2015.

Kingma, D. and Welling, M. Auto-encoding variationalBayes. In International Conference on Learning Repre-sentations, 2014.

Kulkarni, K., Lohit, S., Turaga, P., Kerviche, R., and Ashok,A. Reconnet: Non-iterative reconstruction of imagesfrom compressively sensed measurements. In IEEE Con-ference on Computer Vision and Pattern Recognition,2016.

Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.Human-level concept learning through probabilistic pro-gram induction. Science, 350(6266):1332–1338, 2015.

LeCun, Y., Cortes, C., and Burges, C. J. MNIST handwrittendigit database. http://yann. lecun. com/exdb/mnist, 2010.

Lu, X., Dong, W., Wang, P., Shi, G., and Xie, X. Convcsnet:A convolutional compressive sensing framework based ondeep learning. arXiv preprint arXiv:1801.10342, 2018.

Lustig, M., Donoho, D. L., Santos, J. M., and Pauly, J. M.Compressed sensing MRI. IEEE signal processing maga-zine, 25(2):72–82, 2008.

Mallat, S. A wavelet tour of signal processing: the sparseway. Academic press, 2008.

Mardani, M., Gong, E., Cheng, J. Y., Vasanawala, S., Za-harchuk, G., Alley, M., Thakur, N., Han, S., Dally, W.,Pauly, J. M., et al. Deep generative adversarial networksfor compressed sensing automates MRI. arXiv preprintarXiv:1706.00051, 2017.

Mardani, M., Monajemi, H., Papyan, V., Vasanawala, S.,Donoho, D., and Pauly, J. Recurrent generative adversar-ial networks for proximal learning and automated com-pressive image recovery. In IEEE Conference on Com-puter Vision and Pattern Recognition, 2018.

Matouek, J. Lectures on Discrete Geometry - 2002. SpringerScience and Business Media, 2002.

Mousavi, A., Patel, A. B., and Baraniuk, R. G. A deeplearning approach to structured signal recovery. In AnnualAllerton Conference on Communication, Control, andComputing, 2015.

Peyre, G. Best basis compressed sensing. IEEE Transac-tions on Signal Processing, 58(5):2613–2622, 2010.

Qaisar, S., Bilal, R. M., Iqbal, W., Naureen, M., and Lee,S. Compressive sensing: From theory to applications, asurvey. Journal of Communications and networks, 15(5):443–456, 2013.

Radford, A., Metz, L., and Chintala, S. Unsupervised rep-resentation learning with deep convolutional generativeadversarial networks. arXiv preprint arXiv:1511.06434,2015.

Tang, G., Bhaskar, B. N., Shah, P., and Recht, B. Com-pressed sensing off the grid. IEEE Transactions on Infor-mation Theory, 59(11):7465–7490, 2013.

Tibshirani, R. Regression shrinkage and selection via thelasso. Journal of the Royal Statistical Society. Series B(Methodological), pp. 267–288, 1996.

Tieleman, T. and Hinton, G. Lecture 6.5-RMSProp: Dividethe gradient by a running average of its recent magnitude.Coursera: Neural networks for machine learning, 4(2):26–31, 2012.

Xu, W., Mallada, E., and Tang, A. Compressive sensingover graphs. In IEEE INFOCOM, 2011.

Yu, G. and Sapiro, G. Statistical compressed sensing ofGaussian mixture models. IEEE Transactions on SignalProcessing, 59(12):5842–5858, 2011.

Zhang, X., Wang, D., Zhou, Z., and Ma, Y. Simultane-ous rectification and alignment via robust recovery oflow-rank tensors. In Advances in Neural InformationProcessing Systems, 2013.

Zhang, X., Zhou, Z., Wang, D., and Ma, Y. Hybrid singu-lar value thresholding for tensor completion. In AAAIConference on Artificial Intelligence, 2014.