Joint segmentation and reconstruction of hyperspectral data with compressed measurements

Joint segmentation and reconstruction ofhyperspectral data with compressed

measurements

Qiang Zhang,1,* Robert Plemmons,2 David Kittle,3 David Brady,3 and Sudhakar Prasad4

1Biostatistical Sciences, Wake Forest School of Medicine, Winston-Salem, North Carolina 27157, USA2Computer Science and Mathematics, Wake Forest University, Winston-Salem, North Carolina 27106, USA

3Electrical and Computer Engineering, Duke University, Durham, North Carolina 27708, USA4Physics and Astronomy, University of New Mexico, Albuquerque, New Mexico 87131, USA

*Corresponding author: [email protected]

Received 19 April 2011; revised 8 July 2011; accepted 10 July 2011;posted 12 July 2011 (Doc. ID 146042); published 27 July 2011

This work describes numerical methods for the joint reconstruction and segmentation of spectral imagestaken by compressive sensing coded aperture snapshot spectral imagers (CASSI). In a snapshot, a CASSIcaptures a two-dimensional (2D) array of measurements that is an encoded representation of bothspectral information and 2D spatial information of a scene, resulting in significant savings in acquisitiontime and data storage. The reconstruction process decodes the 2D measurements to render a three-dimensional spatio-spectral estimate of the scene and is therefore an indispensable component of thespectral imager. In this study, we seek a particular form of the compressed sensing solution that as-sumes spectrally homogeneous segments in the two spatial dimensions, and greatly reduces the numberof unknowns, often turning the underdetermined reconstruction problem into one that is over-determined. Numerical tests are reported on both simulated and real data representing compressedmeasurements. © 2011 Optical Society of AmericaOCIS codes: 100.3010, 110.3010, 110.4234.

1. Introduction

Hyperspectral remote sensing technology allows oneto capture images using a range of spectra from UVto visible to IR. Multiple images of a scene or objectare created using light from different parts of thespectrum. These hyperspectral images, forming adata cube, can be used, e.g., for ground or space objectidentification [1], astrophysics [2], and biomedicaloptics [3].

Since a single digital image can typically have asize of 12Mbytes or more, the size of hyperspectraldata cubes could easily move to the gigabyte level.Such high dimensional data pose challenges in both

data acquisition and reconstruction. Technologiessuch as tunable filters [4] or computed tomography[5] measure at least as many elements as thereare in a hyperspectral data cube, and require longacquisition time and vast data storage and transfer,but relatively little effort in reconstruction. Recentlyproposed compressive imagers, such as a coded aper-ture snapshot spectral imager (CASSI) [6,7], needonly take a single snapshot fromwhich to reconstructa hyperspectral data cube if the latter is assumed tobe sparse in some basis. Clearly compressed mea-surements need much less acquisition time and datastorage, although they demand powerful algorithmsfor data reconstruction, which is usually a highly un-derdetermined problem. For example, if the size of ahyperspectral cube f is n1 × n2 × n3, the double dis-perser CASSI (DD-CASSI, to be discussed later)

0003-6935/11/224417-19$15.00/0© 2011 Optical Society of America

1 August 2011 / Vol. 50, No. 22 / APPLIED OPTICS 4417

[6] measures only a two-dimensional (2D) image gwith size n1 × n2, from which we need to reconstructthe original cube. Here we consider first a simpleleast squares approach to estimating f , i.e., f ¼argminf ‖Hf − g‖2

2, where H is the DD-CASSI sys-tem matrix with a size n1n2 × n1n2n3. Since thereare fewer rows than columns in H, the problemcan possess an infinite number of solutions, andhence additional constraints are needed to obtaina meaningful, practical solution.

Mathematical analysis of compressive sensing hasdrawn a great deal of attention after importantresults were obtained by Donoho [8,9] and Candeset al. [10]. The problem is generally posed as oneof finding the sparsest solution to f , with sparsitymeasured by the l1 norm or l0 pseudonorm, i.e.,

min‖f‖p; subject to Hf ¼ g; ð1Þ

where p ¼ 0 or 1. Sometimes though the signalsthemselves are not sparse, an appropriate or even op-timal basis Φ can be found on which the projectionsof the signals are sparse, i.e.,

minf ;Φ

‖Φf‖p; subject to Hf ¼ g: ð2Þ

Though it remains important to develop methodsto determine the optimal basis for given classes ofsignals andmeasurement systems, in many practicalsituations the signals to be reconstructed are com-posed of relatively homogeneous segments or clus-ters, e.g., hyperspectral images in remote sensingproblems [11] and in computed tomography [12].The optimal basis here is the set of segment member-ship functions on the two spatial dimensions, whichtake on values of either 0 or 1 for hard segmentationor within the interval [0, 1] for fuzzy segmentation[13]. Variational segmentation algorithms are a pop-ular area of research, see e.g., [12,14–16], but theyare often applied either directly to the original mea-surements or after the reconstruction. By contrast,work is just beginning on combining reconstructionand segmentation in a more general linear inverseproblem setting. For instance, Li et al. [15] havecoupled the segmentation and deblurring/denoisingmodels in order to simultaneously segment anddeblur/denoise degraded hyperspectral images.Ramlau and Ring [17] jointly reconstructed and seg-mented Radon transformed tomography data. Xinget al. [18] divided the hyperspectral imagery into con-tiguous blocks and used a Bayesian dictionary learn-ing method to estimate both the dictionary and thespectra in each block from limited noisy observa-tions. However, neither example considered compres-sive measurements directly from a sensor.

We now formalize our approach in the following.Let H ∈ RN×n1n2n3 represent the hyperspectralimager system matrix, f ∈ Rn1n2n3×1 the vectorizedhyperspectral cube, and g ∈ RN×1 the vectorizedmeasurement image(s). Consider the following linear

least squares problem for reconstructing compres-sive measurements:

minf

‖Hf − g‖22: ð3Þ

By compressive measurements, we mean N <n1n2n3. Here we assume the solution f is composedof a limited number of segments or materials, each ofwhich essentially has a homogeneous value at eachspectral channel. Thus we seek a decomposed solu-tion described in a continuous form as

f ðx; y; λÞ ¼XLi¼1

uiðx; yÞsiðλÞ; ð4Þ

where uiðx; yÞ is the ith membership function, whosevalues can be either 0 or 1 for a hard segmentation orin the interval [0,1] for a fuzzy segmentation, andsatisfies the constraint

PLi¼1 ui ¼ 1. Here, siðλÞ repre-

sents the spectral signature function of the ith seg-ment or material. The support of ui lies only onthe two spatial dimensions represented by x and y,and is thus independent of the spectral dimensionrepresented by λ. The spectral signatures, s ¼ fsiðλÞ;i ¼ 1;…;Lg, vary only along the spectral dimension.The discrete version of f can thus be written as

f ¼XLi¼1

uisTi ; ð5Þ

where f ∈ Rn1n2×n3 is the folded hyperspectral cube,ui ∈ Rn1n2×1 is the vectorized membership function,and si ∈ Rn3×1. We identify f with f in the remainderof the paper.

With this decomposed form of f , the savings in thenumber of signals to reconstruct is significant,Lðn1n2 þ n3Þ unknowns compared to the originaln1n2n3, since we can usually expect n3 ≫ L. Asshown later, due to this reduction the reconstructionproblem to be solved can be turned from being highlyunderdetermined to being overdetermined. Also, foreach i the membership function ui is expected to besparse in terms of its gradient, since only near theboundaries is there significant mixing of membersin our applications. Thus a total variation (TV) reg-ularization is particularly suitable here.

This manuscript is organized as follows. InSection 2, we present an alternating least squares(ALS) approach to separate the original problem[Eq. (3)] into two subproblems and to solve for uiand si in an alternating fashion. For the first subpro-blem, namely to solve for ui given si, when the systemmatrix H preserves boundaries, e.g., the DD-CASSIsystem, we present a generalized segmentation algo-rithm based on the Chan–Vese model [14] and thevariational model [15,16] within an inverse problemsetting. This enables us to directly segment a hyper-spectral data cube from a single observed image,given known spectral signatures. For more general

4418 APPLIED OPTICS / Vol. 50, No. 22 / 1 August 2011

cases ofH, we present and compare three algorithmswith increasing computation complexity to solve thefirst subproblem. The second subproblem, namely tosolve for si, given ui, is typically a highly overdeter-mined problem, and simple regularized pseudoin-verse (PI) methods suffice here, as we shall see. InSection 3, we present both simulated and real com-pressed hyperspectral images to be reconstructedwith the proposed method. By increasing the numberof measurements N, we also study the relationshipbetween N and the probability of successful recon-struction from random initial values, and how thisprobability depends on the nature of the reconstruc-tion algorithms, thus shedding light on certainthresholds on the minimum number of measure-ments required for a finite probability of success. Bycomparing the three algorithms, we were pleasantlysurprised to find that even a simple PI approach canachieve quite satisfying results when N is suffi-ciently large. We present our main conclusions andcomments in Section 4.

2. Joint Segmentation and Reconstruction

Using the decomposed form of f ≡ f described inEq. (5), we have essentially turned the original linearproblem into a nonlinear problem. Nevertheless, thereduction in the number of unknowns enables usto take advantage of compressed measurements toachieve satisfactory reconstruction results. Ourapproach to solve this problem is to optimize for uiand si by alternating iterations, given initial valuesof ui or si. But before we go into the details, somesimple results and notation on the matrix-vectormultiplications are provided.

We denote the concatenation of all membershipvectors ui as u ∈ RLn1n2×1, i.e., u ¼ ðuT

1 ;uT2 ;…;uT

LÞT ,and the concatenation of all spectral signatures sias s ∈ RLn3×1, i.e., s ¼ ðsT1 ; sT2 ;…; sTLÞT .

Proposition 1. With the decomposed form of f inEq. (5), we have the following equality:

Hf ¼ HSu ¼ HUs: ð6ÞHere S ¼ ~S ⊗ I1, the column vectors of ~S are thespectral signature vectors si, and I1 ∈ Rn1n2×n1n2 isthe identity matrix. U ¼ ~U ⊗ I2, ~U ¼ ð~uijÞn1n2×L,where each column corresponds to the vectorizedui, and I2 ∈ Rn3×n3 is another identity matrix. ⊗ de-notes the Kronecker product.

Proposition 2. With the decomposed form of f inEq. (5), we have the following equality:

Hf ¼XLi¼1

HSiui ¼XLi¼1

HUisi: ð7Þ

Here Si ¼ si ⊗ I1, Ui ¼ ~ui ⊗ I2, and ~ui is the ithcolumn of ~U.

The proofs of both propositions are basically book-keeping by noting the vectorization of f is done firstby the spectral dimension and then by two spatialdimensions, i.e.,

f ¼ ðf 111;…; f 11n3; f 121;…; f 12n3

;…; f n1n21;…; f n1n2n3ÞT :ð8Þ

We denote HS by Hs and HU by Hu to representthe system matrices used for solving for u and s,respectively.

A. Alternating Least Squares

The ALS approach turns the original problem[Eq. (3)] into two subproblems, i.e., given sðnÞ at stepn, we solve

uðnþ1Þ ¼ argminu

‖HðnÞs u − g‖2

2; subject to Eu ¼ 1;

ð9Þ

and given uðnþ1Þ at step n, we solve

sðnþ1Þ ¼ argmins

‖Hðnþ1Þu s − g‖2

2; subject to s > 0;

ð10Þ

where HðnÞs ¼ HSðnÞ and Hðnþ1Þ

u ¼ HUðnÞ by Proposi-tion 1. The two constraints on u and s, respectively,are the sum-to-one constraint for the membershipfunctions expressed as Eu ¼ P

Li¼1 ui ¼ 1, with E ¼

ðI1; I1;…; I1Þn1n2×n1n2L, and the nonnegativity con-straint for the spectral signatures.

Since the second subproblem [Eq. (10)] has onlyLn3 unknowns, it is often overdetermined, i.e.,N ≫ Ln3. Hence, simple approaches such as the PImethod with Tikhonov regularization for noise canbe sufficient, and the nonnegativity constraint canbe satisfied with a projection function onto thenonnegative orthant. But because N ≫ Ln3, the PIsolution could be well within the nonnegativeorthant, and this would render the projection stepunnecessary.

The first subproblem [Eq. (9)] has Ln1n2 unknownsand hence could be underdetermined, but becauseoften times n3 ≫ L, and when Ln1n2 ≤ N < n1n2n3,we can expect Eq. (9) to be exact or even overdeter-mined. Even better, if we consider a hard segmenta-tion, the number of nonzeros in u would not exceedn1n2, because at each pixel there is exactly one uithat is nonzero. In cases when Eq. (9) is underdeter-mined, we seek a sparse solution in its null spacewith the sparsity defined as the l1 norm of the bound-aries of segments, and rewrite Eq. (9) as

minu

XLi¼1

‖Gui‖1; subject to Hsu ¼ g and Eu ¼ 1;

ð11Þ

where G is the gradient matrix, the discrete versionof the more familiar operator ∇, and thus theproblem above can also be regarded as a TV minimi-zation problem, e.g., [19], with the functional


Fu ¼ ‖Hsu − g‖22 þ αu

XLi¼1

‖Gui‖1; ð12Þ

where αu is the TV regularization parameter.As a necessary condition for convergence, consider

a simpler situation when both Eqs. (9) and (10) areexact or overdetermined and we only use L2 func-tionals (informally, square integrable functionals) foroptimizing u and s. We then have the followingtheorem that guarantees a nonincreasing sequenceof the L2 functional values through the iterationsof the ALS approach.

Theorem 1. When only using the L2 functionals,the ALS approach results in a nonincreasingsequence of functional values, i.e.,

‖HðnÞs uðnþ1Þ − g‖2

2 ≥ ‖Hðnþ1Þu sðnþ1Þ − g‖2

2

≥ ‖Hðnþ1Þs uðnþ2Þ − g‖2

2; ð13Þ

for n ¼ 0; 1; 2;….Proof. The proof becomes obvious after noting that

by Proposition 1,

HðnÞs uðnþ1Þ ¼ HSðnÞuðnþ1Þ ¼ HUðnþ1ÞsðnÞ ¼ Hðnþ1Þ

u sðnÞ:

Thus, by definition of sðnþ1Þ, we have the firstinequality

‖HðnÞs uðnþ1Þ − g‖2

2 ¼ ‖Hðnþ1Þu sðnÞ − g‖2

2

≥ ‖Hðnþ1Þu sðnþ1Þ − g‖2

2: ð14Þ

And by Hðnþ1Þu sðnþ1Þ ¼ Hðnþ1Þ

s uðnþ1Þ and the definitionof uðnþ2Þ, we have the second inequality

‖Hðnþ1Þu sðnþ1Þ − g‖2

2 ¼ ‖Hðnþ1Þs uðnþ1Þ − g‖2

2

≥ ‖Hðnþ1Þs uðnþ2Þ − g‖2

2: ð15Þ

In the more general case, when Eq. (9) is underde-termined and when noise is present, we seek to mini-mize a combined functional of Eqs. (10) and (12) withadded Tikhonov regularization for si:

F ¼ ‖Hf − g‖22 þ αu

XLi¼1

‖Gui‖1 þ αsXLi¼1

‖si‖22; ð16Þ

where αs is the Tikhonov regularization parameter.For Eq. (16), we have a similar theorem to guaranteea nonincreasing sequence of F ðnÞ with the ALSapproach.

Theorem 2. Using the functional defined inEq. (16), the ALS approach results in a nonincreasingsequence of functional values.

Proof. First, we define the solutions of two subpro-blems as

uðnþ1Þ ¼ argminu

‖HðnÞs u − g‖2

2 þ αuXLi¼1

‖Gui‖1;

subject to Euðnþ1Þ ¼ 1;

ð17Þ

sðnþ1Þ ¼ argmins

‖Hðnþ1Þu s − g‖2

2 þ αsXLi¼1

‖si‖22;

subject to s > 0: ð18Þ

Then the functional value at step n, after optimizingfor u, becomes

F ðnÞu ¼ ‖HðnÞ

s uðnþ1Þ − g‖22 þ αu

XLi¼1

‖Guðnþ1Þi ‖1

þ αsXLi¼1

‖sðnÞi ‖22; ð19Þ

and after optimizing for s at step n, it becomes

F ðnÞs ¼ ‖Hðnþ1Þ

u sðnþ1Þ − g‖22 þ αu

XLi¼1

‖Guðnþ1Þi ‖1

þ αsXLi¼1

‖sðnþ1Þi ‖2

2: ð20Þ

By the inequality in Eq. (14) and by the definition ofsðnþ1Þ, we have

F ðnÞu ≥ F ðnÞ

s : ð21Þ

Similarly, by the inequality in Eq. (15) and thedefinition of uðnþ2Þ, we can see

F ðnÞs ≥ F ðnþ1Þ

u : ð22Þ

Next we prove that the nonincreasing sequenceleads to a minimizer of Eq. (16) in the space Adefined as

A ¼�ðu; sÞjui ∈ BVðΩÞ;ui ≥ 0;

XLi¼1

ui ¼ 1; s ≥ 0

�;

ð23Þwhere BVðΩÞ is a bounded variation space. Werewrite Eq. (16) in its continuous form,

F ¼ZΩ½Hðf Þ − g�2dxdyþ αu

XLi¼1

ZΩj∇uijdxdy

þ αsXLi¼1

ZΛs2i ðλÞdλ; ð24Þ

where H is the continuous version of the systemoperator H.


Theorem 3. In the space A, a minimizer exists ofthe functional defined in Eq. (24).

Proof. If we take ui ¼ 1=L and si ¼ 1, thenF ¼ R

Ω½Hð1Þ − g�2dxdyþ αsLjΛj < þ∞. Since F ≥ 0 inA, we know the infimum of the functional would befinite. Let ðuðnÞ; sðnÞÞ⊆A be a minimizing sequence ofthe ALS approach, with uðnÞ and sðnÞ defined inEqs. (17) and (18). Then a constant M > 0 exists,such that

F ðuðnÞ; sðnÞÞ ≤ M: ð25Þ

Hence each term in F ðuðnÞ; sðnÞÞ is also bounded, i.e.,

αuXLi¼1

ZΩj∇uðnÞ

i jdxdy ≤ M: ð26Þ

It is also easy to see that uðnÞi is bounded in L1 since

‖ui‖L1ðΩÞ ¼RΩ uðnÞ

i dxdy < jΩj, and by the compact-ness of BV space, up to a subsequence also denotedby fuðnÞ

i g after relabeling, a function u�i ∈ BVðΩÞ

exists such that

uðnÞi → u�

i strongly in L1ðΩÞ;uðnÞi → u�

i a:e:ðx; yÞ ∈ Ω;

∇uðnÞi → ∇u�

i in the sense of measure: ð27Þ

Also, by the lower semicontinuity of TV,

ZΩj∇u�

i jdxdy ≤ lim infn→∞

ZΩj∇uðnÞ

i jdxdy: ð28Þ

Since uðnÞ satisfies two constraints, by convergence,so would u�

i . For the convergence of sðnÞ, we haveat each iteration

sðnÞ ¼ Pþ½ðHðnÞTu HðnÞ

u þ αsIÞ−1ðHðnÞTu gÞ�: ð29Þ

Here we use the discrete form of the objective func-tional in Eq. (18) to avoid introducing more notationswhile maintaining the spirit of the proof. Since theeigenvalues of HðnÞT

u HðnÞu þ αsI have a lower bound

αs, and both HðnÞu and g are bounded, we know sðnÞ

must have an upper bound. Furthermore, becauseuðnÞ

→ u�i , we know HðnÞ

u → H�u, where H�

u ¼ HU� asdefined in Proposition 1. Hence we can find an upperbound ofHðnÞ

u for all n and thus an upper bound of sðnÞ

for all n. By the boundedness of the sequence fsðnÞg,we can extract a subsequence also denoted by fsðnÞgand a limit s� such that

sðnÞ → s�: ð30Þ

Since uðnÞi → u�

i a:e:ðx; yÞ ∈ Ω and sðnÞ → s�, by Fatou’slemma, we know

ZΩ½Hðf �Þ − g�2dxdyþ

Xi

Zs�i ðλÞdλ

≤ lim infn→∞

ZΩ½Hðf ðnÞÞ − g�2dxdyþ

Xi

ZsðnÞi ðλÞdλ; ð31Þ

where f �¼Piu

�i ðx;yÞs�i ðλÞ and f ðnÞ¼P

iuðnÞi ðx;yÞsðnÞi ðλÞ.

By Eqs. (28) and (31), the functional also satisfies theinequality

F ðu�; s�Þ ≤ lim infn→∞

F ðuðnÞ; sðnÞÞ; ð32Þ

and we can conclude ðu�; s�Þ must be a minimizer.In the next two sections we focus our attention on

solving the first subproblem [Eq. (12)], since thesecond subproblem is highly overdetermined andwell-posed, and thus a simple PI approach wouldbe sufficient. For the first subproblem, we start froma simpler case, i.e., when the operator Hs preservesthe boundaries. This effectively renders ui indepen-dent from uj, when i ≠ j, and this also makes indivi-dual entries within ui independent from each other.For solving this problem, we generalize two existingsegmentation approaches, the Chan–Vese model [14]and a variational model [15,16]. We then move to themore general Hs, i.e., where boundaries are not pre-served and where all ui are coupled together by Hs.

B. Boundary Preserving Operator H

Clearly any operations on a hyperspectral data cubeonly along the spectral dimension will preserve theboundaries in the two spatial dimensions. One sim-ple example is the summation operator along thespectral dimension which turns a hyperspectral cubeinto a 2D image. Another example, the DD-CASSIsystem, is similar to the summation operator exceptthat by using a coded aperture, the system first effec-tively multiplies the hyperspectral cube with a ran-dom aperture code, and then sums along the spectraldimension, as shown in the following equation:

gij ¼Xk

f ijkci;j−k; ð33Þ

where cij is the calibrated 2D aperture code withoutspectral content, f is the original hyperspectral cube,and g is the observed 2D image. A third example isthe correlation operator, e.g., the moving averagemethod [16], which computes the correlations be-tween the spectral signatures. Here we formalizethe definition as the following.

Definition 1. We define a system matrix H asboundary preserving if it satisfies

HSiui ¼ Λiui; ð34Þ

where H ∈ Rn1n2×n1n2n3 , Si ∈ Rn1n2n3×n1n2 is defined inProposition 2, and Λi ∈ Rn1n2×n1n2 is a diagonalmatrix.


Notice that the boundary preserving operatorshave a fixed number of rows, i.e., n1n2, for the appar-ent reason that the boundary of an object could beexactly the boundary of the scene. Again using thesummation operator as an example, we have H ¼eT ⊗ I1, where e ∈ Rn3×1 is a constant vector withall entries 1. It is not hard to verify that HSi is adiagonal matrix.

Theorem 4. The boundary preserving operatoreffectively renders each ui independent from eachother, that is to say, we can optimize for each uiseparately.

Proof. BecauseP

iui ¼ 1, we have g ¼ Pig⊙ui ¼P

iDiagðgÞui, where ⊙ represents the element wiseproduct and DiagðgÞ is a diagonal matrix withelements of g on the diagonal. Hence by Proposition2 and by the definition of the boundary preservingoperator, we have

‖Hsu − g‖22 ¼ ‖

XLi¼1

HSiui −XLi¼1

DiagðgÞui‖22

¼ ‖XLi¼1

ðΛi −DiagðgÞÞui‖22

¼XLi¼1

‖Λi −DiagðgÞ‖22u

2i : ð35Þ

Clearly ui is decoupled from uj when i ≠ j. Addition-ally, uij1 is also independent from uij2 for j1 ≠ j2. Notethat the last equality in Eq. (35) is due to the hardsegmentation assumption, i.e., the binary ui, andhence the cross terms disappear after expandingðPL

i¼1 aijuijÞ2, where aij is the jth element on thediagonal of Λi −DiagðgÞ.

Because of the independence of the ui, we can solvefor each ui separately through, for example, by usingthe popular active contour partial differential equa-tion model, also called the Chan–Vese (C-V) model[14], which can be described using the followingfunctional:

Fðϕ; c1; c2Þ ¼ μZΩδðϕÞj∇ϕj þ ν

ZΩHðϕÞ

þ α1ZΩðu0 − c1Þ2HðϕÞ

þ α2ZΩðu0 − c2Þ2ð1 −HðϕÞÞ; ð36Þ

where ϕðx; y; tÞ is the function whose zero level setrepresents the evolving curve C, and is chosen tobe positive inside C and negative outside C. HðϕÞis the Heavyside function of ϕ, which is defined as

HðϕÞ ¼�1 ϕ ≥ 0;0 ϕ < 0:

ð37Þ

δðϕÞ is the derivative of HðϕÞ, u0 is the observedimage, c1 is the mean intensity within C, and c2 is

the mean intensity outside C. Also, μ, ν, α1, α2 arethe weighting parameters of the model. The lasttwo terms on the right are the force terms that eitherexpand or shrink the initial contour C0. We referreaders to [14] for further details of the model.

For a hard segmentation, the membership functionuiðx; yÞ is equivalent to the Heavyside function of ϕ,and gðx; yÞ is the observed image. The modificationonly involves slightly changing two force terms ofthe original model, i.e.,

Fðϕ; c1; c2Þ ¼ μZΩδðϕÞj∇ϕj þ ν

ZΩui þ α1

ZΩð~gi − gÞ2ui

þ α2ZΩð~gi − g − c2Þ2ð1 − uiÞ; ð38Þ

where ~giðx; yÞ is the image spectrally coded by the ithspectral signature. The discrete form of ~giðx; yÞ isderived by taking the diagonal of HSi.

We can see the only difference from the originalC-V model are the force terms. The modified C-Vmodel in Eq. (38) is a generalization of the originalmodel in the sense that if ~giðx; yÞ ¼ c1, Eq. (38) isthe same as the original C-V model. It is equivalentto say that we are segmenting the zero valuesegment of the image ~gi − g, rather than c1 − g. Thisbecomes clear if we replace f with its decomposedform and apply the boundary preserving assumptionin the following equation:

Hðf Þ ¼ H�X

i

uiðx; yÞsiðλÞ�

¼Xi

HðsiðλÞÞuiðx; yÞ;

ð39Þand the difference between Hðf Þ and g now becomes

Hðf Þ − g ¼Xi

½HðsiðλÞÞ − gðx; yÞ�uiðx; yÞ: ð40Þ

Here ~g would simply be HðsiðλÞÞ, and its differencefrom g is the driving force for ui. If HðsiðλÞÞ ¼ ci,the equation above becomes the regular C-V model.

This modification is crucial since a spectrallyhomogeneous segment in f could result in inhomoge-neous intensities in the same segment of Hðf Þ or g.One example would be to multiply f with a randomcube and then sum along the spectral dimension. Thevariation of intensities in the original segmentswould give us wrong segmentation results if wedirectly applied the C-V model. An example of suchis provided in Fig. 1(a).

The active contour model evolves each individualinitial contour according to the given force terms,and thus it depends on the initial contours andhas to be implemented separately for each segment.The variational model proposed in [15,16] is able tosegment all L segments at the same time and isalso formulated for the fuzzy segmentation, i.e.,ui ∈ ½0; 1�. The model also accounts for the sum-to-one constraint ui and the nonnegativity constraints


on si. Briefly, the original formulation includes a TVregularization term for ui and the intensity differ-ence between values in the segment and the meanof the area, i.e.,

Xi

ZΩj∇uijdxdyþ αu

Xi

ZΩðg − ciÞ2u2

i dxdy: ð41Þ

The modification again comes by replacing the forceterm with the difference between images ~g and g.

Xi

ZΩj∇uijdxdyþ αu

Xi

ZΩ½~gi − g�2u2

i dxdy: ð42Þ

Again, Eq. (42) can be seen as a generalization ofEq. (41) because when ~gi ¼ ci, we have exactly thesame model. From Theorem 4, we know that afterdiscretization, Eq. (42) is the same as the functionalFu in Eq. (12). One example is shown in Fig. 1(b),where we were able to correctly segment and re-construct a simulated Hubble Satellite Telescope(HST) hyperspectral cube while only using a singleDD-CASSI image.

C. General Operator H

In many applications, such as in using the Radontransform [20] in computed tomography or in usingthe single disperser CASSI (SD-CASSI) system [7],the boundaries cannot be preserved by the operator.Hence, we cannot always rely on modifying existingsegmentation algorithms. We solve the constrainedsubproblem [Eq. (9)] with approaches such as thePI methods, methods with TV minimization, or sub-optimal methods for sparse signal recovery, e.g., thematching pursuit (MP) [21] or the orthogonal MP[22]. In this section, we will discuss these threeoptions.

As stated before, due to the reduction in the num-ber of unknowns, the subproblem [Eq. (9)] can be

exact or even overdetermined when Ln1n2 ≤ N, whilemeasurements are still compressed when N <n1n2n3. The PI solution in this case would simply be

u ¼ PZ2½ðHT

s �HsÞ−1ðHTs gÞ�; ð43Þ

where PZ2is the projection operator onto the space

Z2 ¼ f0; 1g for a hard segmentation. The projectionoperator onto the space [0,1] for a fuzzy segmenta-tion is

u ¼ minfmaxfðHTs �HsÞ−1ðHT

s gÞ; 0g; 1g: ð44ÞThe sum-to-one constraint can be satisfied or closelysatisfied by adding a regularization term to the leastsquare functional, i.e.,

α‖Eu − 1‖22: ð45Þ

The PI solution often suffers from noise, but thiscan be effectively reduced with TV regularization.Li et al. [15] proposed the following functional withan auxiliary variable v to jointly segment anddeblur/denoise hyperspectral images.

FTV ¼ 12‖Hsu − g‖2

2 þαu2‖v − u‖2

2 þ ‖Gv‖1; ð46Þ

where v is the auxiliary variable for smoothing u.The following equations, similarly derived as in[15,16,19], can be used to alternatively solve for uand v through the iterations

pðnþ1Þ ¼ pðnÞ þ ϕ∇ðdivpðnÞ − αuuðnÞÞ1þ ϕ∇ðdivpðnÞ − αuuðnÞÞ ;

vðnþ1Þ ¼ uðnÞ −1αu

divpðnþ1Þ;

uðnþ1Þ ¼ ðHTs Hs þ αuIÞ−1ðHT

s gþ αuvðnþ1ÞÞ; ð47Þ

1500 Iterations

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

1500 Iterations

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

Fig. 1. (Color online) (a) Segmented bottom cylinder area using the original C-V model. (b) Segmented same area using the generalizedC-V model.


where pðnþ1Þ serves as an intermediate variable. See[15,19] for details such as the satisfaction of con-straints. We call this method the TV regularizationmethod.

The MP algorithm finds the “best matching” pro-jections of multidimensional data onto an overcom-plete dictionary. It iteratively generates for anysignal u and any dictionary Hs a sorted list of indicesand scalars which constitute a suboptimal solution tothe problem of sparse signal representation. Theorthogonal matching pursuit algorithm (OMP) is amodification to the MP that maintains full backwardorthogonality of the residual at every step. In eachiteration, the OMP calculates a new signal approxi-mation uðnÞ. The approximation error rðnÞ ¼ u − uðnÞ isthen used in the next iteration to determine whichnew element is to be selected. In particular, the selec-tion is based on the inner products between the cur-rent residual rðnÞ and the column vectors of Hs. Thecomplete algorithm is described in [23].

Here we modify the OMP slightly by introducing agradient operator, because though the membershipfunction uiðx; yÞ is not necessarily sparse, its gradi-ent, being supported over the segment boundaries,often is. Let u ¼ Gu. We first solve

minu

‖u‖1; subject to HsG−1u ¼ g; ð48Þ

through the OMP and then set u ¼ G−1u. We call thisapproach the gradient orthogonal matching pursuit(GOMP). A similar approach is given in [24] to re-store images from the subsets of Fourier transforms.With the GOMP, we only consider the hard segmen-tation, i.e., after solving for all ui, we search for themaximum ui for each x and y:

uiðx; yÞ ¼�1 uiðx; yÞ ≥ ujðx; yÞ; ∀ j ≠ i0 otherwise: : ð49Þ

3. Numerical Examples

Our experiments used to illustrate the effectivenessof the proposed methods are divided into two parts:one for the boundary preserving system operator andthe other for more general system operators. Thoughthe methods can also be applied to other compressivehyperspectral sensing systems, the systems we con-sider here are the DD-CASSI and the SD-CASSI.

A. Boundary Preserving Operator

The operator of interest here is the DD-CASSI sys-tem, whose forward model has been described byEq. (33). The details on the optics can be found in[6]. In the three examples presented in this section,we move progressively from completely simulateddata to completely real data. In the first example,through the forward model [Eq. (33)], we simulatea DD-CASSI image from a simulated hyperspectralcube of the HST [25] at size 177 × 193 × 33, thenwe simulate a DD-CASSI image from a recently ac-quired hyperspectral dataset on a urban setting [26]at size 320 × 360 × 31, and finally we reconstruct ahyperspectral cube from a real DD-CASSI image offluorescent beads [27] at size 600 × 800 × 59.

We start by testing the generalized C-V model[Eq. (38)] and the generalized variation model[Eq. (42)] with known spectral signatures in the HSTscene. Figures 1 and 2 compare the two modifiedmodels with the original ones, and we clearly seethe advantages offered by the two modified models.In Fig. 1, an initial square contour is selected in thebottom cylinder of the satellite, while the modified C-V model progressed to the correct bound and stabi-lized, the original C-V model easily broke out ofthe cylinder boundary and moved to quite arbitraryplaces, though it does stabilize in the end. This is dueto the highly varying intensities within each segmentin the observed image g. Figure 2 is a similar compar-ison between the two TV models, and clearly the ori-ginal TV model (41) cannot segment out the correctareas, while the modified model [Eq. (42)] can.

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

Fig. 2. (Color online) Segmentation of a DD-CASSI image (a) using the original TValgorithm and (b) using the modified TV with knownsiðλÞ.


Next we jointly segmented and reconstructed theoriginal hyperspectral cube from a single DD-CASSIimage. The ALS approach starts from random values

of s, i.e., without any prior knowledge of spectral sig-natures, and uses the modified variation model[Eq. (42)] to estimate uðnþ1Þ and the PI method with

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

Fig. 3. (Color online) First 25 iterations of the estimation of membership functions from left to right and top to bottom.

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Aluminum

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Glue

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Solar Cell

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Black Rubber Edge

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb side/Bolt

Reconstructed

Honeycomb side

Bolt

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Copper Stripping

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb Top

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Background

Fig. 4. (Color online) Estimated spectral signatures in blue are compared with the true ones in red.


the Tikhonov regularization to estimate sðnþ1Þ. Thereconstructed tensor is compared with the originaltensor, using the l2 norm error

ϵ ¼ ‖f − f 0‖22

‖f 0‖22

; ð50Þwhere f and f 0 are the reconstructed and originalhyperspectral cubes, respectively.

Figure 3 shows the first 25 iterations of the esti-mated membership functions u shown in a false colormap, with one color assigned to each segment. Evenwithout any prior knowledge on the spectral signa-

tures, we can correctly reconstruct and segmentthe original HST cube, and the solution u apparentlyconverges after 20 iterations. Figure 4 compares theestimated spectral signatures with the true ones.Only the fifth spectral signature in the middle differsfrom the true one, due to the rather small prevalenceof that particular material in the scene. We alsodirectly ran the general purpose two-step iterativeshrinkage/thresholding (TwIST) algorithm [28] with-out assuming the decomposed form of solution f , forwhich the norm error was 0.205 as compared to0.012 for the new approach. The new approach has

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Aluminum

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Glue

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Solar Cell

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Black Rubber Edge

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb side/Bolt

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Copper Stripping

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb Top

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Background

Fig. 5. (Color online) Joint estimation of membership functions ui in (a) and spectral signatures, si in (b) in the presence of noise(SNR ¼ 21dB).


dramatically reduced the norm error by taking ad-vantage on the solution form of f .

To test the robustness of the proposed methodagainst noise, we polluted the simulated DD-CASSIimage with white noise having a standard deviationof 0.3, which effectively results in an signal-to-noiseratio (SNR) of 21dB, with the SNR in decibelsdefined as

SNR ¼ 20 log10

�σsignalσnoise

�: ð51Þ

Figure 5 shows the estimated u and s, both of whichclosely resemble the true ones, though noisier. Thenorm error is 0.05.

In the second example, we considered a real hyper-spectral dataset of size 320 × 360 × 29, with bandsfrom 0.453 to 0:719 μm, taken by Optech (OptechInternational, Inc. Kiln, Mississippi) on the campus

of University of Southern Mississippi, in Gulfport,Mississippi. The data were collected as part of a pro-ject led by Professor Paul Gader at the University ofFlorida Department of Computer and InformationScience and Engineering [26]. The original datasethad 72 bands ranging from 0.4 to 1:0 μm, but becausethe system cube of DD-CASSI, hðx; y; λÞ, are only ca-librated from 0.45 to 0:72 μm, and after matching thecalibrated wavelengths of DD-CASSI with those ac-tually measured bands by Optech, we chose 31 of the72 bands and ran it through a DD-CASSI forwardmodel for a simulated DD-CASSI image. The leftimage in Fig. 6 shows the Google map of the area,and the right image shows the simulated DD-CASSIimage. Here we ran the segmentation algorithmdirectly on the simulated DD-CASSI image withseven known spectra taken from the original hyper-spectral cube, shown in Fig. 7, and the result isshown in Fig. 8. The algorithm clearly separatedout the areas of trees, water/shadow, grass, and

100 200 300 400

50

100

150

200

250

300

350

40050 100 150 200 250 300 350

50

100

150

200

250

300

Fig. 6. (Color online) Left image shows a Google map of the area while the right image shows one band of the simulated DD-CASSI image.

400 600 8000

0.2

0.4

0.6

0.8

1Dirt

Ref

lect

ance

Wavelength (nm)400 600 8000

0.2

0.4

0.6

0.8

1Pavement

Ref

lect

ance


0.2

0.4

0.6

0.8

1Grass

Ref

lect

ance


0.2

0.4

0.6

0.8

1Tree

Ref

lect

ance

Wavelength (nm)

400 600 8000

0.2

0.4

0.6

0.8

1Sand

Ref

lect

ance


0.2

0.4

0.6

0.8

1Rooftop

Ref

lect

ance


0.2

0.4

0.6

0.8

1Water/Shadow

Ref

lect

ance

Wavelength (nm)

Fig. 7. (Color online) Estimated spectra of the segmented areas.


pavement with a relatively high resolution. For ex-ample, we observe sharp boundaries between treesand grass, and thin lines of dirt splitting the grassarea into four parts in the middle slightly to the left.Unfortunately due to the chosen bands, the two roadstrips at the bottom were recognized as grass, butthis can be fixed once we have a system cube coveringlonger wavelengths.

In terms of target identification, the reconstruc-tion/segmentation clearly identifies three targetspurposely placed on the ground near the center of thescene. These consist of colored cloths, and we markthese targets by the yellow circle in Fig. 8. This isquite encouraging considering we are only usingone snapshot, or 3.45% of the original data. The onlymissed target has a similar spectra as grass from0.453 to 0:719 μm, but if our DD-CASSI system cube

100 200 300 400 500 600 700 800

100

200

300

400

500

600100 200 300 400 500 600 700 800

100

200

300

400

500

600

R1R4

R5

RO1R3

CA2

Y2

Y7

YG1

G1

G2

BG1

Y3

O1Y1

Y5

Y4

Y6

R2

CA3

CA4

CA1

Fig. 9. (Color online) (a) DD-CASSI image of the beads. (b) The estimated membership function map. (c) The beads spectral signaturesfrom [27]. (d) The pseudocolored reconstructed image using TV minimization [27] with the spectral signatures of the marked beads in (c).

50 100 150 200 250 300 350

50

100

150

200

250

300

Dirt

Pavement

Grass

Rooftop

Sand

Tree

Water/Shadow

Fig. 8. (Color online) Segmentation map estimated from theDD-CASSI image.


is calibrated to longer wavelengths, we will be able toidentify that target as well. The norm error betweenthe reconstructed hyperspectral data cube and theoriginal cube is 0:033.

In the third example, we used a real DD-CASSIimage of a biomedical scene with fluorescent beadsof different colors [27]. The size of the image,600 × 800, plus the number of spectral channels, 59,would result in more than 28 million unknowns if we

were not using the decomposed form, and that mightrender the reconstruction impossible by general pur-pose algorithms such as the TwIST. However, withour decomposition approach, we can estimate thespectral signatures of those beads quite close tothe estimates given in [27]. Figure 9(a) shows theoriginal DD-CASSI image and Fig. 9(b) shows thereconstructed membership functions, where we tryto match the bead colors with false colors as close as

20 40 60 80 100 120

20

40

60

80

100

120

50 100 150

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

20 40 60 80 100 120

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Fig. 10. (Color online) Top left image is a slice of the simulated hyperspectral cube, the top middle is the simulated SD-CASSI image, thetop right is the spectral signature applied to all pixels in the square to the left, the bottom left is a slice of the reconstructed cube, and thebottom middle is the reconstructed spectral signature.

20 40 60 80 100 120

20

40

60

80

100

120

50 100 150

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

20 40 60 80 100 120

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Fig. 11. (Color online) Top left image is a slice of the simulated hyperspectral cube, the top middle is the simulated SD-CASSI image, thetop right are two spectral signatures applied to pixels in two rectangles to the left, the bottom left is a slice of the reconstructed cube, andthe bottom middle are the reconstructed spectral signatures.


possible, and they are labeled in the same way as inFig. 9(d). Figures 9(c) and 9(d) are directly takenfrom [27] for comparison purposes. We can see thatthe colors of most beads are identified correctly. Ourreconstruction also shows that the boundaries ofbeads appear to have different colors from the centerparts due to the geometry, while the reconstructionin [27] treats the whole area of each bead regionas uniform in color. However, notice that becausethe signatures of CA and R are quite close, we cannotquite differentiate between them in Fig. 9(b). Also,several long wavelength beads identified in [27],namely C1, S1, S2, and S3, are missing here becausepixels of these beads in the observed DD-CASSI im-age have weak intensities in the order of 0.01, ashardly seen in Fig. 9(a), while intensities at otherbeads are between 0.15 to 1. Hence the algorithmrecognizes them as background.

B. General Operator

The system operator of interest here is the SD-CASSI system [7], which can be characterized usingsubscript notation as follows:

Pseudo−Inverse

20 40 60 80 100 120

10

20

30

40

Total Variation

20 40 60 80 100 120

10

20

30

40

Orthogonal Matching Pursuit

20 40 60 80 100 120

10

20

30

40

Fig. 12. (Color online) Comparison of three reconstruction meth-ods using a single frame observation and known siðλÞ. Only themiddle 40 rows of the original scene is used here.

Fig. 13. (Color online) (a) Relationship between the probability of a successful reconstruction and the number of observed frames for thethree reconstruction methods. (b) Computation time of each method per iteration of ALS.


gij ¼Xn3

k¼1

ci;jþkf i;jþk;k; ð52Þ

where c is the 2D calibrated aperture code, and theonly difference from the DD-CASSI system is themultiplexing in both the second spatial dimensionand the spectral dimension, and hence the imagetaken has a size of n1 × ðn2 þ n3 − 1Þ. Because of themultiplexing in the second spatial dimension, thisoperator does not preserve boundaries. The detailsof the optical setup can be found in [7]. Three hyper-spectral cubes were used to simulate the SD-CASSI

images, the first having only an isolated squareobject in the image with a size of 128 × 128 × 33,the second having two side-by-side rectangularobjects in the image to test the method’s ability toidentify sharp boundaries, at a size of 128 × 128 × 33,and the third being the HST cube at a size of177 × 193 × 33. The forward model [Eq. (52)] wasused to generate three SD-CASSI images.

We jointly segmented and reconstructed the origi-nal hyperspectral cube from one or more frames ofthe SD-CASSI images. The ALS approach againstarts from random values of s, i.e., without any prior

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500

0.2

0.4

0.6

0.8

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 650

0.7

0.8

0.9

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500

0.5

1

1.5

Wavelength (nm)

Ref

lect

ance

Fig. 14. (Color online) (a) Estimatedmembership functionmap in 20 iterations. (b) The estimated spectral signatures in blue compared tothe original in red.


knowledge, and uses three different algorithms toestimate uðnþ1Þ, the simple regularized PI method,the TV regularization model, and the GOMP model.The PI method with Tikhonov regularization wasused to estimate sðnþ1Þ.

For the first scene with an isolated object in thescene, we were able to reconstruct exactly as shownin Fig. 10, with only one frame of the SD-CASSIimage. This tells us that one highly compressiveSD-CASSI imagemay be sufficient for reconstructinghyperspectral cubes with only isolated objects.

For the second scene with two rectangular objectsside-by-side using random starting values of s, wewere able to reconstruct exactly from eight simulatedframes of the SD-CASSI images using the simple PIapproach, as shown in Fig. 11. We also compared thethree algorithms for estimating u with prior knownspectral signatures from a single SD-CASSI image.Figure 12 shows the false color images of the esti-mated membership functions by all three algorithms,where we see an increasing reconstruction qualityfrom top to bottom. Here, the GOMP results on thethird row match almost exactly with the true ones,while the TV results are smoother and closer to truththan those of PI. Clearly, the GOMP may be the bestcandidate among the three when measurements aremost compressed.

Our experience indicates that testing all threealgorithms without any prior knowledge of s, wefound a zero probability to reconstruct successfullywhen the number of frames is below 4. With the PIapproach and with eight frames of measurements,we were able to reconstruct exactly to a high prob-ability, and the results are shown in Fig. 11. Wefurther expanded this experiment by running allthree algorithms 100 times with different randomstarting values, and with different numbers offrames of measurements. A parallel computationwas set up on an eight-core machine andwas finishedin a week. The probabilities of successful reconstruc-tions are shown in Fig. 13(a), and the computationtime per iteration of ALS is shown in Fig. 13(b). First,we observe that when the number of frames of obser-vations is below 4, there is no chance to reconstructsuccessfully for any algorithm. With four frames, thePI approach still could not reconstruct successfullywhile the TV and GOMP algorithms have a finiteprobability, 7% and 8%, respectively. The GOMPhas a 40% success rate at five frames compared to27% for PI and 28% for TV. This informs us thatthe GOMP could do reasonably well with more com-pressive measurements, but given a certain scenewith a fixed number of nonzero projections, thereshould be a threshold on the minimum number offrames for a successful reconstruction. The TV

200 400 600 800 1000 1200 1400

100

200

300

400

500

600

700

800

900

200 400 600 800 1000 1200 1400

10

20

30

40

50

Fig. 15. (Color online) (a) Imaged color chart object. (b) The original SD-CASSI image with the reconstructed area in the red rectanglebox. (c) The reconstructed segments.


approach turns out to be the least successful beyonda sufficiently high number of frames, while we ob-serve an almost equivalent success rate by the PIand GOMP, which is quite encouraging since thePI is much faster, as shown in Fig. 13(b).

Next, we tried to reconstruct the complex HSTscene with a number of frames of SD-CASSI images.Here we only tested the PI approach, since the other

two would take a prohibitively long computationtime. With random starting values of s, when thenumber of frames reach 20, we had satisfactoryreconstruction results, as shown in Fig. 14, thoughin the beginning, the membership function map looksquite messy. Figure 14(b) compares the estimated sixmost dominant spectral signatures with the corre-sponding true ones, and they all agree quite well.

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000.2

0.4

0.6

0.8

1Purplish Blue

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Moderate Red

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Yellow Green

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange Yellow

Wavelength (nm)

Ref

lect

ance

Ocean OpticsPI

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000.2

0.4

0.6

0.8

1Purplish Blue

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Moderate Red

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Purple

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Yellow Green

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange Yellow

Wavelength (nm)

Ref

lect

ance

TwISTPI

Fig. 16. (Color online) (a) Reconstructed spectra compared with the reference by an Ocean Optics spectrometer. (b) The reconstructedspectra compared with reconstruction using the TwIST.


Finally, the real SD-CASSI images of a color chartimage shown in Fig. 15(a) were used for reconstruct-ing both the color segments and their spectra. Sincethe spectral signatures within each segment arefairly uniform, the scene served as a good candidatefor testing the algorithm. For brevity, we only showthe reconstructed colors in the second row as indi-cated by the red box in Fig. 15(b). The six colors onthe second row from left to right are orange, purplishblue, moderate red, purple, yellow green, and orangeyellow, respectively, and the spectra of five colorsexcluding the purple were measured by an OceanOptics (Dunedin, Florida) spectrometer. A total of12 frames of the SD-CASSI images were used forreconstructing a hyperspectral cube in 44 wave-length channels. Figure 15(c) shows the identifiedsix segments. Clearly we are able to identify thesharp boundaries of the segments. For the accuracyof the reconstructed spectra, we compared them withboth references by the spectrometer in Fig. 16(a) andthe reconstruction by the TwIST in Fig. 16(b). Ourresults match almost perfectly with those obtainedby the TwIST, and also show good agreement withthe spectrometer references.

4. Conclusions

By assuming a low-rank representation, we wereable to jointly segment and reconstruct hyperspec-tral data cubes from compressed measurementsusing the algorithms proposed in this study. Theassumed form of the solution dramatically reducedthe number of unknowns in the reconstruction, thusallowing better accuracy and faster computation. Inour tests, we obtained promising results after apply-ing the algorithms to a sequence of simulated andreal DD-CASSI and SD-CASSI images.

For boundary preserving measurement operators,such as the DD-CASSI system, we generalized twoexisting segmentation algorithms, the C-V modeland a TV model, to directly segment the compressedmeasurements along the spatial and spectral dimen-sions without first reconstructing the data into thehyperspectral cubes. In our studies, we show poorsegmentation results after applying the originaltwo models directly on simulated DD-CASSI images,but had very good results after applying the general-ized two models. For more general operators, such asthat of SD-CASSI, we proposed three methods, i.e.,the PI method, the TV method, and the GOMPmeth-od, for estimating the membership functions with gi-ven spectral signatures. A more extensive simulationstudy shows that several frames of measurementsare often needed for a successful reconstruction,though the theoretical threshold on the minimumnumber of frames needs further analysis. Withenough observations, the probabilities of successfulreconstruction by different algorithms tend to con-verge, and even the simple PI approach can providesuccessful reconstruction. But when the number ofobservations is limited, the GOMP method performsbest. For real SD-CASSI data taken on a color chart

scene, we have shown good reconstruction resultswith the GOMP method.

In using a low-rank representation, the algorithmsproposed here would better suit situations where dis-tinct spectra lie in some spatially homogeneous areas(segments), e.g., cartoon-like scenes. As the size ofsegments gets smaller and the number of spectrabecomes greater, we would expect performancereduction given the same number of compressedmeasurements. In these situations, we recommendadding measurements as in [29] but when the scenein study becomes too complex, more general algo-rithms such as the TwIST may be more reliable.However, we would caution that in these nonsparsesituations, the compressive sensing approach maynot be that appealing after all.

This work was sponsored in part by the UnitedStates Air Force Office of Scientific Research(AFOSR) under grant FA9550-08-1-0151.

References1. K. Jorgensen, J. Africano, K. Hamada, E. Stansbery, P. Sydney,

and P. Kervin, “Physical properties of orbital debris from spec-troscopic observations,” Adv. Space Res. 34, 1021–1025 (2004).

2. R. P. Lin, B. R. Dennis, G. J. Hurford, D. M. Smith, A. Zehnder,P. R. Harvey, D. W. Curtis, D. Pankow, P. Turin, M. Bester, A.Csillaghy, M. Lewis, N. Madden, H. F. Van Beek, M. Appleby,T. Raudorf, J. McTiernan, R. Ramaty, E. Schmahl, R.Schwartz, S. Krucker, R. Abiad, T. Quinn,P. Berg, M. Hashii,R. Sterling, R. Jackson, R. Pratt, R. D. Campbell, D. Malone,D. Landis, C. P. Barrington-Leigh, S. Slassi-Sennou, C. Cork,D. Clark, D. Amato, L. Orwig, R. Boyle, I. S. Banks, K. Shirey,A. K, Tolbert, D. Zarro, F. Snow, K. Thomsen, R. Henneck, A.McHedlishvili, P. Ming, F. Fivian, J. Jordan, R. Wanner, J.Crubb, J. Preble, M. Matranga, A. Benz, H. Hudson, R. C.Canfield, G. D. Holman, C. Crannell, T. Kosugi, A. G. Emslie,N. Vilmer, J. C. Brown, C. Johns-Krull, M. Aswchwanden, T.Metcalf, and A. Conway, “The Reuven Ramaty high-energysolar spectroscopic imager (RHESSI),” Sol. Phys. 210, 3–32(2002).

3. T. Pham, F. Bevilacqua, T. Spott, J. Dam, B. Tromberg, and S.Andersson-Engels, “Quantifying the absorption and reducedscattering coefficients of tissuelike turbid media over a broadspectral range with noncontact Fourier-transform hyperspec-tral imaging,” Appl. Opt. 39, 6487–6497 (2000).

4. H. Morris, C. Hoyt, and P. Treado, “Imaging spectrometers forfluorescence and Raman microscopy: acousto-optic and liquidcrystal tunable filters,” Appl. Spectrosc. 48, 857–866 (1994).

5. J. Mooney, V. Vickers, M. An, and A. Brodzik, “High-throughput hyperspectral infrared camera,” J. Opt. Soc.Am. A 14, 2951–2961 (1997).

6. M. Gehm, R. John, D. Brady, R. Willett, and T. Schulz, “Single-shot compressive spectral imaging with a dual-disperserarchitecture,” Opt. Express 15, 14013–14027 (2007).

7. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Singledisperser design for coded aperture snapshot spectralimaging,” Appl. Opt. 47, B44–B51 (2008).

8. D. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory52, 1289–1306 (2006).

9. D. Donoho and M. Elad, “Optimally sparse representation ingeneral (nonorthogonal) dictionaries via l1 minimization,”Proc. Natl. Acad. Sci. USA 100, 2197–2202 (2003).

10. E. Candès, J. Romberg, and T. Tao, “Robust uncertaintyprinciples: exact signal reconstruction from highly incomplete


frequency information,” IEEE Trans. Inf. Theory 52, 489–509(2006).

11. J. Ball and L. Bruce, “Level set segmentation of remotelysensed hyperspectral images,” in Proceedings of the 2005IEEE International Geoscience and Remote Sensing Sympo-sium (IEEE, 2005), pp. 5638–5642.

12. R. Maksimovic, S. Stankovic, and D. Milovanovic, “Computedtomography image analyzer: 3D reconstruction and segmen-tation applying active contour models–′snakes′,” Int. J. Med.Inform. 58, 29–37 (2000).

13. J. Kent and K. Mardia, “Spatial classification using fuzzymembership models,” IEEE Trans. Pattern Anal. MachineIntell. 10, 659–671 (1988).

14. T. Chan and L. Vese, “Active contours without edges,” IEEETrans. Image Process. 10, 266–277 (2001).

15. F. Li, M. Ng, and R. Plemmons, “Coupled segmentation anddenoising/deblurring models for hyperspectral material iden-tification,” in Numerical Linear Algebra with Applications(Wiley, 2010).

16. F. Li, M. Ng, R. Plemmons, S. Prasad, and Q. Zhang, “Hyper-spectral image segmentation, deblurring, and spectral analy-sis for material identification,” Proc. SPIE 7701, 770103(2010).

17. R. Ramlau and W. Ring, “A Mumford-Shah level-set approachfor the inversion and segmentation of x-ray tomography data,”J. Comput. Phys. 221, 539–557 (2007).

18. Z. Xing, Department of Electrical and Computer Engineering,Duke University, Durham, NC, M. Zhou, A. Castrodad, G.Sapiro, and L. Carin, are preparing a manuscript to be called“Dictionary learning for noisy and incomplete hyperspectralimages.”

19. A. Chambolle, “An algorithm for total variation minimizationand applications,” J. Mathematical Imaging and Vision 20,89–97 (2004).

20. S. Deans, The Radon Transform and Some of Its Applications(Wiley, 1983).

21. S. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process. 41,3397–3415 (1993).

22. J. Tropp, “Greed is good: algorithmic results for sparse approx-imation,” IEEE Trans. Inf. Theory 50, 2231–2242 (2004).

23. Y. Pati, R. Rezaiifar, and P. Krishnaprasad, “Orthogonalmatching pursuit: recursive function approximation withapplications to wavelet decomposition,” in 1993 ConferenceRecord of The Twenty-Seventh Asilomar Conference onSignals, Systems and Computers (IEEE, 1993), pp. 40–44.

24. R. Maleh, A. Gilbert, and M. Strauss, “Sparse gradientimage reconstruction done faster,” in IEEE InternationalConference on Image Processing, 2007. ICIP 2007 (IEEE,2007), Vol. 2. pp. 77–80.

25. Q. Zhang, H.Wang, R. Plemmons, and V. Pauca, “Tensormeth-ods for hyperspectral data analysis: a space object materialidentification study,” J. Opt. Soc. Am. A 25, 3001–3012(2008).

26. P. Gader, A. Zare, R. Close, and G. Tuell, “Co-registered hyper-spectral and LiDAR Long Beach, Mississippi data collection,”University of Florida, University of Missouri, and OptechInternational (2010).

27. C. Cull, K. Choi, D. Brady, and T. Oliver, “Identification offluorescent beads using a coded aperture snapshot spectralimager,” Appl. Opt. 49, B59–B70 (2010).

28. J. Bioucas-Dias and M. Figueiredo, “A new TwIST: two-step iterative shrinkage/thresholding algorithms for imagerestoration,” IEEE Trans. Image Process. 16, 2992–3004(2007).

29. D. Kittle, K. Choi, A. Wagadarikar, and D. Brady, “Multiframeimage estimation for coded aperture snapshot spectralimagers,” Appl. Opt. 49, 6824–6833 (2010).


Joint segmentation and reconstruction of hyperspectral data with compressed measurements

Documents

Transcript of Joint segmentation and reconstruction of hyperspectral data with compressed measurements