Joint Segmentation and Reconstruction of

Hyperspectral Data with Compressed Measurements

Qiang Zhang1,∗, Robert Plemmons2, David Kittle3, David Brady3,

and Sudhakar Prasad4

1Biostatistical Sciences, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA

2Computer Science and Mathematics, Wake Forest University, Winston-Salem, NC 27106,

USA

3Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA

4Physics and Astronomy, University of New Mexico, Albuquerque, NM 87131, USA

∗Corresponding author: qizhang@wakehealth.edu

This work describes numerical methods for the joint reconstruction and

segmentation of spectral images taken by compressive sensing coded aperture

snapshot spectral imagers (CASSI). In a snapshot, a CASSI captures a two-

dimensional (2D) array of measurements that is an encoded representation of

both spectral information and 2D spatial information of a scene, resulting in

significant savings in acquisition time and data storage. The reconstruction

process decodes the 2D measurements to render a three-dimensional spatio-

spectral estimate of the scene, and is therefore an indispensable component

of the spectral imager. In this study, we seek a particular form of the

compressed sensing solution that assumes spectrally homogeneous segments

in the two spatial dimensions, and greatly reduces the number of unknowns,

often turning the under-determined reconstruction problem into one that is

over-determined. Numerical tests are reported on both simulated and real

data representing compressed measurements. c⃝ 2011 Optical Society of

America

OCIS codes: 100.3010, 110.3010, 110.4234

1

1. Introduction

Hyperspectral remote sensing technology allows one to capture images using a range of

spectra from ultraviolet to visible to infrared. Multiple images of a scene or object are created

using light from different parts of the spectrum. These hyperspectral images, forming a data

cube, can be used, e.g. for ground or space object identification [1], astrophysics [2], and

biomedical optics [3].

Since a single digital image can typically have a size of 12 megabytes or more, the size of

hyperspectral data cubes could easily move to the gigabyte level. Such high dimensional data

pose challenges in both data acquisition and reconstruction. Technologies such as tunable

filters [4] or computed tomography [5] measure at least as many elements as there are in a

hyperspectral data cube, and require long acquisition time and vast data storage and transfer,

but relatively little effort in reconstruction. Recently proposed compressive imagers such as

a Coded Aperture Snapshot Spectral Imager (CASSI) [6,7] need only take a single snapshot

from which to reconstruct a hyperspectral data cube if the latter is assumed to be sparse

in some basis. Clearly compressed measurements need much less acquisition time and data

storage, although they demand powerful algorithms for data reconstruction, which is usually

a highly under-determined problem. For example, if the size of a vectorized hyperspectral

cube f is n1 × n2 × n3, the double disperser CASSI (DD-CASSI, to be discussed later) [6]

measures only a 2D vectorized image g with size n1×n2, from which we need to reconstruct

the original cube. Here we consider first a simple least squares approach to estimating f ,

i.e. f = argminf ∥Hf − g∥22, where H is the DD-CASSI system matrix having a size n1n2 ×n1n2n3. Since there are fewer rows than columns in H, the problem can possess an infinite

number of solutions, and hence additional constraints are needed to obtain a meaningful,

practical solution.

Mathematical analysis of compressive sensing has drawn a great deal of attention after

important results were obtained by Donoho [8, 9] and Candes, Romberg and Tao [10]. The

problem is generally posed as one of finding the sparsest solution to f , with sparsity measured

by the l1 norm or l0 pseudo-norm, i.e.

min ∥f∥p, subject to Hf = g, (1)

where p = 0 or 1. Sometimes though the signals themselves are not sparse, an appropriate

or even optimal basis Φ can be found on which the projections of the signals are sparse, i.e.

minf,Φ

∥Φf∥p, subject to Hf = g. (2)

Though it remains important to develop methods to determine the optimal basis for given

2

classes of signals and measurement systems, in many practical situations the signals to be

reconstructed are composed of relatively homogeneous segments or clusters, e.g. hyperspec-

tral images in remote sensing problems [11], and in computed tomography [12]. The optimal

basis here is the set of segment membership functions on the two spatial dimensions, which

take on values of either 0 or 1 for hard segmentation or within the interval [0, 1] for fuzzy

segmentation [13]. Variational segmentation algorithms are a popular area of research, see

e.g. [12, 14–16], but they are often applied either directly to the original measurements or

after the reconstruction. By contrast, work is just beginning on combining reconstruction

and segmentation in a more general linear inverse problem setting. For instance, Li, Ng and

Plemmons [15] have coupled the segmentation and deblurring/denoising models in order

to simultaneously segment and deblur/denoise degraded hyperspectral images. Ramlau and

Ring [17] jointly reconstructed and segmented Radon transformed tomography data. Xing,

Zhou, Castrodad, Guillermo Sapiro, and Carin [18] divided the hyperspectral imagery into

contiguous blocks and used Bayesian dictionary learning method to estimate both the dictio-

nry and the spectra in each block from limited noisy observations. However, neither example

considered compressive measurements directly from a sensor.

We now formalize our approach in the following. Let H ∈ RN×n1n2n3 represent the hy-

perspectral imager system matrix, f ∈ Rn1n2n3×1 the vectorized hyperspectral cube, and

g ∈ RN×1 the vectorized measurement image(s). Consider the following linear least squares

problem for reconstructing compressive measurements,

minf

∥Hf − g∥22. (3)

By compressive measurements, we mean N < n1n2n3. Here we assume the solution f is

composed of a limited number of segments or materials, each of which essentially has a

homogeneous value at each spectral channel. Thus we seek a decomposed solution described

in a continuous form as

f(x, y, λ) =L∑i=1

ui(x, y)si(λ), (4)

where ui(x, y) is the ith membership function, whose values can be either 0 or 1 for a hard

segmentation or in the interval [0, 1] for a fuzzy segmentation, and satisfies the constraint∑Li=1 ui = 1. Here, si(λ) represents the spectral signature function of the ith segment or

material. The support of ui lies only on the two spatial dimensions represented by x and y,

and is thus independent of the spectral dimension represented by λ. The spectral signatures,

s = si(λ), i = 1, . . . , L, vary only along the spectral dimension. The discrete version of f

3

can thus be written as,

f =L∑i=1

uisTi , (5)

where f ∈ Rn1n2×n3 is the folded hyperspectral cube, ui ∈ Rn1n2×1 is the vectorized member-

ship function and si ∈ Rn3×1. We identify f with f in the remainder of the paper.

With this decomposed form of f , the savings in the number of signals to reconstruct is

significant, L(n1n2 + n3) unknowns compared to the original n1n2n3, since we can usually

expect n3 ≫ L. As shown later, due to this reduction the reconstruction problem to be

solved can be turned from being highly under-determined to being over-determined. Also,

for each i the membership function ui is expected to be sparse in terms of its gradient, since

only near the boundaries is there significant mixing of members in our applications. Thus a

total variation (TV) regularization is particularly suitable here.

This manuscript is organized as follows. In Section 2, we present an alternating least

squares (ALS) approach to separate the original problem (3) into two subproblems and to

solve for ui and si in an alternating fashion. For the first subproblem, namely to solve for

ui given si, when the system matrix H preserves boundaries, e.g. the DD-CASSI system,

we present a generalized segmentation algorithm based on the Chan-Vese model [14] and

the variational model [15, 16] within an inverse problem setting. This enables us to directly

segment a hyperspectral data cube from a single observed image, given known spectral signa-

tures. For more general cases of H, we present and compare three algorithms with increasing

computation complexity to solve the first sub-problem. The second sub-problem, namely to

solve for si, given ui, is typically a highly overdetermined problem, and simple regularized

pseudoinverse methods suffice here, as we shall see. In Section 3, we present both simulated

and real compressed hyperspectral images to be reconstructed with the proposed method.

By increasing the number of measurements N , we also study the relationship between N

and the probability of successful reconstruction from random initial values, and how this

probability depends on the nature of the reconstruction algorithms, thus shedding light on

certain thresholds on the minimum number of measurements required for a finite probability

of success. By comparing the three algorithms, we were pleasantly surprised to find that even

a simple pseudo-inverse approach can achieve quite satisfying results when N is sufficiently

large. We present our main conclusions and comments in Section 4.

2. Joint Segmentation and Reconstruction

Using the decomposed form of f ≡ f described in (5), we have essentially turned the origi-

nal linear problem into a non-linear problem. Nevertheless, the reduction in the number of

unknowns enables us to take advantage of compressed measurements to achieve satisfactory

reconstruction results. Our approach to solve this problem is to optimize for ui and si by

4

alternating iterations, given initial values of ui or si. But before we go into details, some

simple results and notation on matrix-vector multiplications are provided.

We denote the concatenation of all membership vectors ui as u ∈ RLn1n2×1, i.e. u =

(uT1 , u

T2 , . . . , u

TL)

T , and the concatenation of all spectral signatures si as s ∈ RLn3×1, i.e.

s = (sT1 , sT2 , . . . , s

TL)

T .

Proposition 1. With the decomposed form of f in (5), we have the following equality

Hf = HSu = HUs. (6)

Here S = S ⊗ I1, the column vectors of S are the spectral signature vectors si, and I1 ∈Rn1n2×n1n2 is the identity matrix. U = U⊗I2, U = (uij)n1n2×L, where each column corresponds

to the vectorized ui, and I2 ∈ Rn3×n3 is another identity matrix. ⊗ denotes the Kronecker

product.

Proposition 2. With the decomposed form of f in (5), we have the following equality

Hf =L∑i=1

HSiui =L∑i=1

HUisi. (7)

Here Si = si ⊗ I1, Ui = ui ⊗ I2, and ui is the ith column of U .

The proofs of both propositions are basically bookkeeping by noting the vectorization of

f is done first by the spectral dimension and then by two spatial dimensions, i.e.

f = (f111, . . . , f11n3 , f121, . . . , f12n3 , . . . , fn1n21, . . . , fn1n2n3)T . (8)

We denote HS by Hs and HU by Hu to represent the system matrices used for solving for

u and s, respectively.

2.A. Alternating Least Squares (ALS)

The ALS approach turns the original problem (3) into two subproblems, i.e. given s(n) at

step n, we solve,

u(n+1) = argminu

∥H(n)s u− g∥22, subject to Eu = 1 (9)

and given u(n+1) at step n, we solve

s(n+1) = argmins

∥H(n+1)u s− g∥22, subject to s > 0, (10)

5

where H(n)s = HS(n) and H

(n+1)u = HU (n) by Proposition 1. The two constraints on u

and s respectively are the sum-to-one constraint for the membership functions expressed as

Eu =∑L

i=1 ui = 1, with E = (I1, I1, . . . , I1)n1n2×n1n2L, and the nonnegativity constraint for

the spectral signatures.

Since the second sub-problem (10) has only Ln3 unknowns, it is often over-determined, i.e.

N ≫ Ln3. Hence, simple approaches such as the pseudo-inverse (PI) method with Tikhonov

regularization for noise can be sufficient, and the nonnegativity constraint can be satisfied

with a projection function onto the nonnegative orthant. But because N ≫ Ln3, the PI

solution could be well within the nonnegative orthant and this would render the projection

step unnecessary.

The first sub-problem (9) has Ln1n2 unknowns and hence could be under-determined, but

because often times n3 ≫ L, and when Ln1n2 ≤ N < n1n2n3, we can expect (9) to be exact

or even over-determined. Even better, if we consider a hard segmentation, the number of

nonzeros in u would not exceed n1n2, because at each pixel there is exactly only one ui that

is nonzero. In cases when (9) is under-determined, we seek a sparse solution in its null space

with the sparsity defined as the l1 norm of the boundaries of segments, and rewrite (9) as

minu

L∑i=1

∥Gui∥1, subject to Hsu = g and Eu = 1, (11)

where G is the gradient matrix, the discrete version of the more familiar operator ∇, and

thus the problem above can also be regarded as a total variation (TV) minimization problem,

e.g. [19], with the functional

Fu = ∥Hsu− g∥22 + αu

L∑i=1

∥Gui∥1, (12)

where αu is the TV regularization parameter.

As a necessary condition for convergence, consider a simpler situation when both (9)

and (10) are exact or over-determined and we only use L2 functionals (informally, square

integrable functionals) for optimizing u and s. We then have the following theorem that

guarantees a nonincreasing sequence of the L2 functional values through the iterations of the

ALS approach.

Theorem 1. When only using the L2 functionals, the alternating least square approach

results in a nonincreasing sequence of functional values, i.e.

∥H(n)s u(n+1) − g∥22 ≥ ∥H(n+1)

u s(n+1) − g∥22 ≥ ∥H(n+1)s u(n+2) − g∥22, (13)

6

for n = 0, 1, 2, . . ..

Proof. The proof becomes obvious after noting that by Proposition 1,

H(n)s u(n+1) = HS(n)u(n+1) = HU (n+1)s(n) = H(n+1)

u s(n).

Thus by definition of s(n+1), we have the first inequality,

∥H(n)s u(n+1) − g∥22 = ∥H(n+1)

u s(n) − g∥22 ≥ ∥H(n+1)u s(n+1) − g∥22. (14)

And by H(n+1)u s(n+1) = H

(n+1)s u(n+1) and the definition of u(n+2), we have the second inequal-

ity,

∥H(n+1)u s(n+1) − g∥22 = ∥H(n+1)

s u(n+1) − g∥22 ≥ ∥H(n+1)s u(n+2) − g∥22. (15)

In the more general case, when (9) is under-determined and when noise is present, we seek

to minimize a combined functional of (10) and (12) with added Tikhonov regularization for

si,

F = ∥Hf − g∥22 + αu

L∑i=1

∥Gui∥1 + αs

L∑i=1

∥si∥22, (16)

where αs is the Tikhonov regularization parameter. For (16), we have a similar theorem to

guarantee a nonincreasing sequence of F (n) with the ALS approach.

Theorem 2. Using the functional defined in (16), the alternating least square approach

results in a nonincreasing sequence of functional values.

Proof. First we define the solutions of two subproblems as,

u(n+1) = argminu

∥H(n)s u− g∥22 + αu

L∑i=1

∥Gui∥1, subject to Eu(n+1) = 1, (17)

and

s(n+1) = argmins

∥H(n+1)u s− g∥22 + αs

L∑i=1

∥si∥22, subject to s > 0. (18)

Then the functional value at step n after optimizing for u becomes

F (n)u = ∥H(n)

s u(n+1) − g∥22 + αu

L∑i=1

∥Gu(n+1)i ∥1 + αs

L∑i=1

∥s(n)i ∥22, (19)

7

and after optimizing for s at step n, it becomes

F (n)s = ∥H(n+1)

u s(n+1) − g∥22 + αu

L∑i=1

∥Gu(n+1)i ∥1 + αs

L∑i=1

∥s(n+1)i ∥22. (20)

By the inequality in (14) and by the definition of s(n+1), we have

F (n)u ≥ F (n)

s . (21)

Similarly, by the inequality in (15) and the definition of u(n+2), we can see

F (n)s ≥ F (n+1)

u . (22)

Next we prove that the nonincreasing sequence leads to a minimizer of (16) in the space

A defined as,

A = (u, s)|ui ∈ BV (Ω), ui ≥ 0,L∑i=1

ui = 1, s ≥ 0, (23)

where BV (Ω) is a bounded variation space. We rewrite (16) in its continuous form,

F =

∫Ω

[H(f)− g]2dxdy + αu

L∑i=1

∫Ω

|∇ui|dxdy + αs

L∑i=1

∫Λ

s2i (λ)dλ, (24)

where H is the continuous version of the system operator H.

Theorem 3. In the space A, there exists a minimizer of the functional defined in (24).

Proof. If we take ui = 1/L and si = 1, then F =∫Ω[H(1)− g]2dxdy + αsL|Λ| < +∞. Since

F ≥ 0 in A, we know the infimum of the functional would be finite. Let (u(n), s(n)) ⊆ A be a

minimizing sequence of the ALS approach, with u(n) and s(n) defined in (17) and (18). Then

there exists a constant M > 0, such that

F(u(n), s(n)) ≤ M. (25)

Hence each term in F(u(n), s(n)) is also bounded, i.e.,

αu

L∑i=1

∫Ω

|∇u(n)i |dxdy ≤ M. (26)

It is also easy to see that u(n)i is bounded in L1 since ∥ui∥L1(Ω) =

∫Ωu(n)i dxdy < |Ω|, and by

8

the compactness of BV space, up to a subsequence also denoted by u(n)i after relabeling,

there exists a function u∗i ∈ BV (Ω) such that

u(n)i → u∗

i strongly in L1(Ω),

u(n)i → u∗

i a.e. (x, y) ∈ Ω,

∇u(n)i → ∇u∗

i in the sense of measure. (27)

Also, by the lower semi-continuity of total variation,∫Ω

|∇u∗i |dxdy ≤ lim inf

n→∞

∫Ω

|∇u(n)i |dxdy. (28)

Since u(n) satisfies two constraints, by convergence, so would u∗i . For the convergence of s

(n),

we have at each iteration

s(n) = P+

[(H(n)T

u H(n)u + αsI

)−1 (H(n)T

u g)]

. (29)

Here we use the discrete form of the objective functional in (18) for to avoid introducing more

notations while maintaining the spirit of the proof. Since the eigenvalues of H(n)Tu H

(n)u +αsI

have a lower bound αs, and both H(n)u and g are bounded, we know s(n) must have an upper

bound. Furthermore, because u(n) → u∗i , we knowH

(n)u → H∗

u, whereH∗u = HU∗ as defined in

Proposition 1. Hence we can find an upper bound of H(n)u for all n and thus an upper bound

of s(n) for all n. By the boundedness of the sequence s(n), we can extract a subsequence

also denoted by s(n) and a limit s∗ such that

s(n) → s∗. (30)

Since u(n)i → u∗

i a.e. (x, y) ∈ Ω and s(n) → s∗, by Fatou’s Lemma we know∫Ω

[H(f ∗)−g]2dxdy+∑i

∫s∗i (λ)dλ ≤ lim inf

n→∞

∫Ω

[H(f (n))−g]2dxdy+∑i

∫s(n)i (λ)dλ, (31)

where f ∗ =∑

i u∗i (x, y)s

∗i (λ) and f (n) =

∑i u

(n)i (x, y)s

(n)i (λ). By (28) and (31), the functional

also satisfies the inequality

F(u∗, s∗) ≤ lim infn→∞

F(u(n), s(n)), (32)

and we can conclude (u∗, s∗) must be a minimizer.

In the next two sections we focus our attention on solving the first subproblem (12), since

9

the second subproblem is highly over-determined and well-posed, and thus a simple pseudo-

inverse approach would be sufficient. For the first subproblem, we start from a simpler case,

i.e. when the operator Hs preserves the boundaries. This effectively renders ui independent

from uj, when i = j, and this also makes individual entries within ui independent from each

other. For solving this problem, we generalize two existing segmentation approaches, the

Chan-Vese model [14] and a variational model [15, 16]. We then move to the more general

Hs, i.e. where boundaries are not preserved and where all ui are coupled together by Hs.

2.B. Boundary Preserving Operator H

Clearly any operations on a hyperspectral data cube only along the spectral dimension will

preserve the boundaries in the two spatial dimensions. One simple example is the summation

operator along the spectral dimension to turn a hyperspectral cube into a 2D image. Another

example, the DD-CASSI system, is similar to the summation operator except that by using

a coded aperture, the system effectively first multiplies the hyperspectral cube with a ran-

dom aperture code, and then sums along the spectral dimension, as shown in the following

equation.

gij =∑k

fijkci,j−k, (33)

where cij is the calibrated 2D aperture code without spectral content, f is the original

hyperspectral cube and g is the observed 2D image. A third example is the correlation

operator, e.g. the moving average method [16], which computes the correlations between

spectral signatures. Here we formalize the definition as following.

Definition 1. We define a system matrix H as boundary preserving if it satisfies

HSiui = Λiui, (34)

where H ∈ Rn1n2×n1n2n3, Si ∈ Rn1n2n3×n1n2 is defined in Proposition 2, and Λi ∈ Rn1n2×n1n2

is a diagonal matrix.

Notice that the boundary preserving operators have a fixed number of rows, i.e. n1n2, for

the apparent reason that the boundary of an object could be exactly the boundary of the

scene. Again using the summation operator as an example, we have H = eT ⊗ I1, where

e ∈ Rn3×1 is a constant vector with all entries 1. It is not hard to verify that HSi is a

diagonal matrix.

Theorem 4. The boundary preserving operator effectively renders each ui independent from

each other, that is to say, we can optimize for each ui separately.

Proof. Because∑

i ui = 1, we have g =∑

i g ⊙ ui =∑

i Diag(g)ui, where ⊙ represents the

element wise product and Diag(g) is a diagonal matrix with elements of g on the diagonal.

10

Hence by Proposition 2 and by the definition of the boundary preserving operator, we have

∥Hsu−g∥22 = ∥L∑i=1

HSiui−L∑i=1

Diag(g)ui∥22 = ∥L∑i=1

(Λi−Diag(g))ui∥22 =L∑i=1

∥Λi−Diag(g)∥22u2i .

(35)

Clearly ui is decoupled from uj when i = j. Additionally, uij1 is also independent from uij2

for j1 = j2. Note that the last equality in (35) is due to the hard segmentation assumption,

i.e. the binary ui, and hence the cross terms disappear after expanding (∑L

i=1 aijuij)2, where

aij is the jth element on the diagonal of Λi −Diag(g).

Due to the independence of the ui, we can solve for each ui separately through, for example,

by using the popular active contour PDE model, also called the Chan-Vese (C-V) model [14],

which can be described using the following functional,

F (ϕ, c1, c2) = µ

∫Ω

δ(ϕ)|∇ϕ|+ ν

∫Ω

H(ϕ) +α1

∫Ω

(u0 − c1)2H(ϕ) +α2

∫Ω

(u0 − c2)2(1−H(ϕ)),

(36)

where ϕ(x, y, t) is the function whose zero level set represents the evolving curve C, and is

chosen to be positive inside C and negative outside C. H(ϕ) is the Heavyside function of ϕ,

which is defined as

H(ϕ) =

1 ϕ ≥ 0,

0 ϕ < 0.(37)

δ(ϕ) is the derivative of H(ϕ), u0 is the observed image, c1 is the mean intensity within C,

and c2 is the mean intensity outside C. Also, µ, ν, α1, α2 are weighting parameters of the

model. The last two terms on the right are the force terms that either expand or shrink the

initial contour C0. We refer readers to [14] for further details of the model.

For a hard segmentation, the membership function ui(x, y) is equivalent to the Heavyside

function of ϕ, and g(x, y) is the observed image. The modification only involves slightly

changing two force terms of the original model, i.e.

F (ϕ, c1, c2) = µ

∫Ω

δ(ϕ)|∇ϕ|+ ν

∫Ω

ui + α1

∫Ω

(gi − g)2ui + α2

∫Ω

(gi − g − c2)2(1− ui), (38)

where gi(x, y) is the image spectrally coded by the ith spectral signature. The discrete form

of gi(x, y) is derived by taking the diagonal of HSi.

We can see the only difference from the original C-V model are the force terms. The

modified C-V model in (38) is a generalization of the original model in the sense that if

gi(x, y) = c1, (38) is the same as the original C-V model. It is equivalent to say that we are

segmenting the zero value segment of the image gi−g, rather than c1−g. This becomes clear

if we replace f with its decomposed form and apply the boundary preserving assumption in

11

the following equation,

H(f) = H

(∑i

ui(x, y)si(λ)

)=∑i

H(si(λ))ui(x, y), (39)

and the difference between H(f) and g now becomes,

H(f)− g =∑i

[H(si(λ))− g(x, y)]ui(x, y). (40)

Here g would simply be H(si(λ)) and its difference from g is the driving force for ui. If

H(si(λ)) = ci, the equation above becomes the regular C-V model.

This modification is crucial since a spectrally homogeneous segment in f could result in

inhomogeneous intensities in the same segment of H(f) or g. One example would be to

multiply f with a random cube and then sum along the spectral dimension. The variation of

intensities in the original segments would give us wrong segmentation results if we directly

applied the C-V model. An example of such is provided in Figure 1 (a) in Section 3.

The active contour model evolves each individual initial contour according to the given

force terms, and thus it depends on the initial contours and has to be implemented separately

for each segment. The variational model proposed in [15,16] is able to segment all L segments

at the same time and is also formulated for the fuzzy segmentation, i.e. ui ∈ [0, 1]. The model

also accounts for the sum-to-one constraint ui and the nonnegativity constraints on si. Briefly,

the original formulation includes a total variation (TV) regularization term for ui and the

intensity difference between values in the segment and the mean of the area, i.e.

∑i

∫Ω

|∇ui|dxdy + αu

∑i

∫Ω

(g − ci)2u2

i dxdy. (41)

The modification again comes by replacing the force term with the difference between images

g and g. ∑i

∫Ω

|∇ui|dxdy + αu

∑i

∫Ω

[gi − g]2u2i dxdy. (42)

Again, (42) can be seen as a generalization of (41) because when gi = ci, we have exactly

the same model. From Theorem 4, we know that after discretization (42) is the same as the

functional Fu in (12). One example is shown by Figure 1 (b) in Section 3, where we were

able to correctly segment and reconstruct a simulated Hubble Satellite Telescope (HST)

hyperspectral cube while only using a single DD-CASSI image.

12

2.C. General Operator H

In many applications, such as in using the Radon Rransform [20] in computed tomography

or in using the single disperser CASSI (SD-CASSI) system [7], the boundaries cannot be

preserved by the operator. Hence, we cannot always rely on modifying existing segmentation

algorithms. We solve the constrained subproblem (9) with approaches such as pseudo-inverse

methods, methods with total variation minimization, or suboptimal methods for sparse signal

recovery, e.g. the matching pursuit [21] or the orthogonal matching pursuit [22]. In this

section, we will discuss these three options.

As stated before, due to the reduction in the number of unknowns, the subproblem (9) can

be exact or even over-determined when Ln1n2 ≤ N , while measurements are still compressed

when N < n1n2n3. The pseudo-inverse solution in this case would simply be:

u = PZ2

[(HT

s ∗Hs

)−1 (HT

s g)]

, (43)

where PZ2 is the projection operator onto the space Z2 = 0, 1 for a hard segmentation.

The projection operator onto the space [0, 1] for a fuzzy segmentation is

u = minmax

(HT

s ∗Hs

)−1 (HT

s g), 0, 1. (44)

The sum-to-one constraint can be satisfied or closely satisfied by adding a regularization

term to the least square functional, i.e.

α∥Eu− 1∥22. (45)

The pseudo-inverse solution often suffers from noise, but this can be effectively reduced

with total variation regularization. Li, Ng and Plemmons [15] proposed the following func-

tional with an auxiliary variable v to jointly segment and deblur/denoise hyperspectral im-

ages.

FTV =1

2∥Hsu− g∥22 +

αu

2∥v − u∥22 + ∥Gv∥1, (46)

where v is the auxiliary variable for smoothing u. The following equations, similarly derived

as in [15,16,19], can be used to alternatively solve for u and v through the iterations,

p(n+1) =p(n) + ϕ∇(div p(n) − αuu

(n))

1 + ϕ∇(div p(n) − αuu(n)),

v(n+1) = u(n) − 1

αu

div p(n+1),

u(n+1) = (HTs Hs + αuI)

−1(HTs g + αuv

(n+1)), (47)

13

where p(n+1) serves as an intermediate variable. See [15,19] for details such as the satisfaction

of constraints. We call this method the TV regularization method.

The matching pursuit (MP) algorithm finds the “best matching” projections of multidi-

mensional data onto an over-complete dictionary. It iteratively generates for any signal u and

any dictionary Hs a sorted list of indices and scalars which constitute a sub-optimal solution

to the problem of sparse signal representation. The orthogonal matching pursuit algorithm

(OMP) is a modification to MP that maintains full backward orthogonality of the residual

at every step. In each iteration, OMP calculates a new signal approximation u(n). The ap-

proximation error r(n) = u− u(n) is then used in the next iteration to determine which new

element is to be selected. In particular, the selection is based on the inner products between

the current residual r(n) and the column vectors of Hs. The complete algorithm is described

in [23].

Here we modify OMP slightly by introducing a gradient operator, because though the

membership function ui(x, y) is not necessarily sparse, its gradient, being supported over the

segment boundaries, often is. Let u = Gu. We first solve

minu

∥u∥1, subject to HsG−1u = g, (48)

through OMP and then set u = G−1u. We call this approach the gradient orthogonal match-

ing pursuit (GOMP). A similar approach is given in [24] to restore images from subsets of

Fourier transforms. With GOMP, we only consider the hard segmentation, i.e. after solving

for all ui, we search for the maximum ui for each x and y,

ui(x, y) =

1 ui(x, y) ≥ uj(x, y),∀j = i

0 otherwise.(49)

3. Numerical Examples

Our experiments used to illustrate the effectiveness of the proposed methods are divided

into two parts, one for the boundary preserving system operator and the other for more

general system operators. Though the methods can also be applied to other compressive

hyperspectral sensing systems, the systems we consider here are the DD-CASSI and SD-

CASSI.

3.A. Boundary Preserving Operator

The operator of interest here is the DD-CASSI system, whose forward model has been

described by (33). The details on the optics can be found in [6]. In the three examples

presented in this section, we move progressively from completely simulated data to completely

real data. In the first example, through the forward model (33), we simualte a DD-CASSI

14

image from from a simulated hyperspectral cube of the Hubble Space Telescope (HST)

[25] at size 177 × 193 × 33, then we simulate a DD-CASSI image from a recently acquired

hyperspectral dataset on a urban setting [26] at size 320×360×31, and finally we reconstruct

a hyperspectral cube from a real DD-CASSI image of fluorescent beads [27] at size 600 ×800× 59.

We start by testing the generalized C-V model (38) and the generalized variation model

(42) with known spectral signatures in the HST scene. Figures 1 and 2 compare the two

modified models with the original ones and we clearly see the advantages offered by the two

modified models. In Figure 1, an initial square contour is selected in the bottom cylinder of

the satellite, while the modified C-V model progressed to the correct bound and stabilized,

the original C-V model easily broke out of the cylinder boundary and moved to quite arbitrary

places, though it does stabilize in the end. This is due to the highly varying intensities within

each segment in the observed image g. Figure 2 is a similar comparision between the two

total variation (TV) models and clearly the original TV model (41) cannot segment out the

correct areas, while the modified model (42) can.

Next we jointly segmented and reconstructed the original hyperspectral cube from a single

DD-CASSI image. The ALS approach starts from random values of s, i.e. without any prior

knowledge of spectral signatures, and uses the modified variation model (42) to estimate

u(n+1) and the pseudo-inverse method with the Tikhonov regularization to estimate s(n+1).

The reconstructed tensor is compared with the original tensor, using the l2 norm error,

ϵ =∥f − f0∥22∥f0∥22

, (50)

where f and f0 are the reconstructed and original hyperspectral cubes, respectively.

Figure 3 shows the first 25 iterations of the estimated membership functions u shown in a

false color map, with one color assigned to each segment. Even without any prior knowledge

on the spectral signatures, we can correctly reconstruct and segment the original HST cube

and the solution u apparently converges after 20 iterations. Figure 4 compares the estimated

spectral signatures with the true ones. Only the fifth spectral signature in the middle differs

from the true one, due to the rather small prevalence of that particular material in the

scene. We also directly ran the general purpose TwIST algorithm [28] without assuming the

decomposed form of solution f , for which the norm error was 0.205 as compared to .012 for

the new approach. The new approach has dramatically reduced the norm error by taking

advantage on the solution form of f .

To test the robustness of the proposed method against noise, we polluted the simulated

DD-CASSI image with white noise having a standard deviation of .3, which effectively results

15

in an SNR of 21 dB, with SNR in dB defined as,

SNR = 20 log10

(σsignal

σnoise

). (51)

Figure 5 shows the estimated u and s, both of which closely resemble the true ones, though

noisier. The norm error is .05.

In the second example, we considered a real hyperspectral dataset of size 320× 360× 29,

with bands from .453µm to .719µm, taken by OpTech (OpTech International, Inc. Kiln,

MS) on the campus of University of Southern Mississippi, in Gulfport, Mississippi. The

data were collected as part of a project led by Professor Paul Gader at the University

of Florida Department of Computer and Information Science and Engineering [26]. The

original dataset had 72 bands ranging from .4µm to 1.0µm, but because the data cube of

DD-CASSI, h(x, y, λ), are only calibrated from .45µm to .72µm, and after matching the

calibrated wavelengths of DD-CASSI with those actually measured bands by OpTech, we

chose 31 of the 72 bands and ran it through a DD-CASSI forward model for a simulated

DD-CASSI image. The left image in Figure 6 shows the Google map of the area and the

right image shows the simulated DD-CASSI image. Here we ran the segmentation algorithm

directly on the simulated DD-CASSI image with seven known spectra taken from the original

hyperspectral cube, shown in Figure 7, and the result is shown in Figure 8. The algorithm

clearly separated out the areas of trees, water/shadow, grass and pavement with a relatively

high resolution. For example, we observe sharp boundaries between trees and grass, and

thin lines of dirt splitting the grass area into four parts in the middle slightly to the left.

Unfortunately due to the chosen bands, the two road strips at the bottom were recognized

as grass, but this can be fixed once we have a system cube covering more long wavelengths.

In terms of target identification, the reconstruction/segmentation clearly identifies three

targets purposely placed on tables just above the ground near the center of the scene. These

consist of colored cloths placed on tables, and we mark these targets by the yellow circle in

Figure 8. This is quite encouraging considering we are only using one snapshot, or 3.45%

of the original data. The only missed target has a similar spectra as grass from .453µm to

.719µm, but if our DD-CASSI system cube is calibrated to longer wavelengths, we will be

able to identify that target as well. The norm error between the reconstructed hyperspectral

data cube and the original cube is .033.

In the third example, we used a real DD-CASSI image of a biomedical scene with fluo-

rescent beads of different colors [27]. The size of the image, 600 × 800, plus the number of

spectral channels, 59, would result in more than 28 million unknowns if we were not using

the decomposed form, and that might render the reconstruction impossible by general pur-

pose algorithms such as TwIST. However, with our decomposition approach we can estimate

16

the spectral signatures of those beads quite close to the estimates given in [27]. Figure 9(a)

shows the original DD-CASSI image and Figure 9(b) shows the reconstructed membership

functions, where we try to match the bead colors with false colors as close as possible, and

they are labeled in the same way as in Figure 9(d). Figure 9(c) and 9(d) are directly taken

from [27] for comparison purposes. Notice that because the signatures of CA and R are quite

close, we cannot quite differentiate between them in Figure 9(b). Also, several long wave-

length beads identified in [27], namely C1,S1,S2 and S3, are missing here because pixels of

these beads in the observed DD-CASSI image have weak intensities in the order of 0.01, as

hardly seen in Figure 9(a), while intensities at other beads are between .15 to 1. Hence the

algorithm recognizes them as background. Our reconstruction does show that the boundaries

of beads appear to have different colors from the center parts due to the geometry, while the

reconstruction in [27] treats the whole area of each bead region as uniform in color.

3.B. General Operator

The system operator of interest here is the SD-CASSI system [7], which can be characterized

using subscript notation as follows:

gij =

n3∑k=1

ci,j+kfi,j+k,k, (52)

where c is the 2D calibrated aperture code, and the only difference from the DD-CASSI

system is the multiplexing in both the second spatial dimension and the spectral dimension,

and hence the image taken has size n1 × (n2 + n3 − 1). Because of the multiplexing in

the second spatial dimension, this operator does not preserve boundaries. The details of the

optical setup can be found in [7]. Three hyperspectral cubes were used to simulate SD-CASSI

images, the first having only an isolated square object in the image of size 128×128×33, the

second having two side-by-side rectangular objects in the image to test the method’s ability

to identify sharp boundaries, at size 128 × 128 × 33, and the third being the HST cube at

size 177× 193× 33. The forward model (52) was used to generate three SD-CASSI images.

We jointly segmented and reconstructed the original hyperspectral cube from one or more

frames of SD-CASSI images. The ALS approach again starts from random values of s, i.e.

without any prior knowledge, and uses three different algorithms to estimate u(n+1), the

simple regularized pseudo-inverse (PI) method, the TV regularization model (TV) and the

gradient orthogonal matching pursuit (GOMP) model. The pseudo-inverse method with

Tikhonov regularization was used to estimate s(n+1).

For the first scene with an isolated object in the scene, we were able to reconstruct exactly

as shown in Figure 10, with only one frame of SD-CASSI image. This tells us that one highly

compressive SD-CASSI image may be sufficient for reconstructing hyperspectral cubes with

17

only isolated objects.

For the second scene with two rectangular objects side by side, using random starting

values of s, we were able to reconstruct exactly from 8 simulated frames of SD-CASSI images

using the simple PI approach, as shown in Figure 11. We also compared the three algorithms

for estimating u with prior known spectral signatures from a single SD-CASSI image. Figure

12 shows the false color images of estimated membership functions by all three algorithms,

where we see an increasing reconstruction quality from top to bottom. Here, GOMP results

on the third row match almost exactly with the true ones, while the TV results are smoother

and closer to truth than those of PI. Clearly, GOMP may be the best candidate among the

three when measurements are most compressed.

Our experience indicates that testing all three algorithms without any prior knowledge

of s, we found a zero probability to reconstruct successfully when the number of frames is

below 4. With the pseudo-inverse approach and with 8 frames of measurements, we were

able to reconstruct exactly to a high probability, and the results are shown in Figure 11. We

further expanded this experiment by running all three algorithms 100 times with different

random starting values, and with different numbers of frames of measurements. A parallel

computation was set up on an eight-core machine and was finished in a week. The probabil-

ities of successful reconstructions are shown in Figure 13(a) and the computation time per

iteration of ALS is shown in Figure 13(b). First we observe that when the number of frames

of observations is below 4, there is no chance to reconstruct successfully for any algorithm.

With 4 frames, the PI approach still could not reconstruct successfully while the TV and

GOMP algorithms have a finite probability, 7% and 8%, respectively. The GOMP has a 40%

success rate at 5 frames compared to 27% for PI and 28% for TV. This informs us that the

GOMP could do reasonably well with more compressive measurements, but given a certain

scene with a fixed number of nonzero projections, there should be a threshold on the min-

imum number of frames for a successful reconstruction. The TV approach turns out to be

the least successful beyond a sufficiently high number of frames, while we observe an almost

equivalent success rate by PI and GOMP, which is quite encouraging since PI is much faster

as shown in Figure 13(b).

Next, we tried to reconstruct the complex HST scene with a number of frames of SD-

CASSI images. Here we only tested the PI approach, since the other two would take a

prohibitively long computation time. With random starting values of s, when the number of

frames reach 20, we had satisfactory reconstruction results, as shown in Figure 14, though

in the beginning, the membership function map looks quite messy. Figure 14(b) compares

the estimated six most dominant spectral signatures with the corresponding true ones, and

they all agree quite well.

Finally, the real SD-CASSI images of a color chart image shown in Figure 15(a) were used

18

for reconstructing both the color segments and their spectra. Since the spectral signatures

within each segment are fairly uniform, the scene served as a good candidate for testing

the algorithm. For brevity, we only show the reconstructed colors in the second row as

indicated by the red box in Figure 15(b). The six colors on the second row from left to

right are respectively orange, purplish blue, moderate red, purple, yellow green and orange

yellow, and the spectra of five colors excluding the purple were measured by an Ocean

Optics (Dunedin, Florida) spectrometer. A total of 12 frames of SD-CASSI images were used

for reconstructing a hyperspectral cube in 44 wavelength channels. Figure 15(c) shows the

identified six segments. Clearly we are able to identify the sharp boundaries of the segments.

For the accuracy of reconstructed spectra, we compared them with both references by the

spectrometer in Figure 16(a) and the reconstruction by TwIST in Figure 16(b). Our results

match almost perfectly those obtained by TwIST, and also show good agreements with the

spectrometer references.

4. Conclusions

By assuming a low-rank representation, we were able to jointly segment and reconstruct

hyperspectral data cubes from compressed measurements using the algorithms proposed in

this study. The assumed form of the solution dramatically reduced the number of unknowns

in the reconstruction, thus allowing better accuracy and faster computation. In our tests, we

obtained promising results after applying the algorithms to a sequence of simulated and real

DD-CASSI and SD-CASSI images.

For boundary preserving measurement operators, such as the DD-CASSI system, we gen-

eralized two existing segmentation algorithms, the Chan-Vese model and a total variation

model, to directly segment the compressed measurements along the spatial and spectral di-

mensions without first reconstructing the data into hyperspectral cubes. In our studies, we

show poor segmentation results after applying the original two models directly on simulated

DD-CASSI images, but very good results after applying the generalized two models. For more

general operators, such as that of SD-CASSI, we proposed three methods, i.e. the pseudo-

inverse method, the total variation method and the gradient orthogonal matching pursuit

(GOMP) method for estimating the membership functions with given spectral signatures. A

more extensive simulation study shows that several frames of measurements are often needed

for a successful reconstruction, though the theoretical threshold on the minimum number

of frames needs further analysis. With enough observations, the probabilities of successful

reconstruction by different algorithms tend to converge, and even the simple pseudo-inverse

approach can provide successful reconstruction. But when the number of observations is

limited, the GOMP method performs best. For real SD-CASSI data taken on a color chart

scene, we have shown good reconstruction results with the GOMP method.

19

In using a low-rank representation, the algorithms proposed here would better suit sit-

uations where distinct spectra lie in some spatially homogeneous areas (segments), e.g.,

cartoon-like scenes. As the size of segments gets smaller and the number of spectra becomes

greater, we would expect a performance reduction given the same number of compressed

measurements. In these situations, we recommend adding measurements as in [29] but when

the scene in study becomes too complex, more general algorithms such as TwIST may be

more reliable. However, we would caution that in these non-sparse situations, the compressive

sensing approach may not be that appealing after all.

5. Acknowledgement

This work was sponsored in part by the AFOSR under grant number FA9550-08-1-0151.

References

1. K. Jorgensen, J. Africano, K. Hamada, E. Stansbery, P. Sydney, and P. Kervin, “Physical

properties of orbital debris from spectroscopic observations,” Advances in Space Research

34, 1021–1025 (2004).

2. R. Lin, B. Dennis, G. Hurford, D. Smith, A. Zehnder, P. Harvey, D. Curtis, D. Pankow,

P. Turin, M. Bester, and Others, “The Reuven Ramaty high-energy solar spectroscopic

imager (RHESSI),” Solar Physics 210, 3–32 (2002).

3. T. Pham, F. Bevilacqua, T. Spott, J. Dam, B. Tromberg, and S. Andersson-Engels,

“Quantifying the absorption and reduced scattering coefficients of tissuelike turbid media

over a broad spectral range with noncontact Fourier-transform hyperspectral imaging,”

Appl. Optics 39, 6487–6497 (2000).

4. H. Morris, C. Hoyt, and P. Treado, “Imaging spectrometers for fluorescence and Raman

microscopy: acousto-optic and liquid crystal tunable filters,” Applied spectroscopy 48,

857–866 (1994).

5. J. Mooney, V. Vickers, M. An, and A. Brodzik, “High-throughput hyperspectral infrared

camera,” Journal of the Optical Society of America A 14, 2951–2961 (1997).

6. M. Gehm, R. John, D. Brady, R. Willett, and T. Schulz, “Single-shot compressive spectral

imaging with a dual-disperser architecture,” Optics Express 15, 14013–14027 (2007).

7. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded

aperture snapshot spectral imaging,” Applied Optics 47, B44–B51 (2008).

8. D. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory 52, 1289–

1306 (2006).

9. D. Donoho and M. Elad, “Optimally sparse representation in general (nonorthogonal)

dictionaries via l1 minimization,” Proceedings of the National Academy of Sciences of

the United States of America 100, 2197 (2003).

20

10. E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal recon-

struction from highly incomplete frequency information,” IEEE Transactions on infor-

mation theory 52, 489 (2006).

11. J. Ball and L. Bruce, “Level set segmentation of remotely sensed hyperspectral im-

ages,” in “2005 IEEE International Geoscience and Remote Sensing Symposium, 2005.

IGARSS’05. Proceedings,” (2005), pp. 5638–5642.

12. R. Maksimovic, S. Stankovic, and D. Milovanovic, “Computed tomography image ana-

lyzer: 3D reconstruction and segmentation applying active contour models–[] snakes’,”

International journal of medical informatics 58, 29–37 (2000).

13. J. Kent and K. Mardia, “Spatial classification using fuzzy membership models,” IEEE

Transactions on Pattern Analysis and Machine Intelligence pp. 659–671 (1988).

14. T. Chan and L. Vese, “Active contours without edges,” IEEE Transactions on image

processing 10, 266–277 (2001).

15. F. Li, M. Ng, and R. Plemmons, “Coupled Segmentation and Denoising/Deblurring

Models for Hyperspectral Material Identification,” Numerical Linear Algebra with Ap-

plications (2010).

16. F. Li, M. Ng, R. Plemmons, S. Prasad, and Q. Zhang, “Hyperspectral image segmenta-

tion, deblurring, and spectral analysis for material identification,” in “SPIE Symposium

on Defense, Security, and Sensing,” , vol. 7701 (2010), vol. 7701.

17. R. Ramlau and W. Ring, “A Mumford-Shah level-set approach for the inversion and

segmentation of X-ray tomography data,” Journal of Computational Physics 221, 539–

557 (2007).

18. Z. Xing, M. Zhou, A. Castrodad, G. Sapiro, and L. Carin, “Dictionary learning for noisy

and incomplete hyperspectral images,” IEEE Transactions on Geoscience and Remote

Sensing (submitted).

19. A. Chambolle, “An algorithm for total variation minimization and applications,” Journal

of Mathematical Imaging and Vision 20, 89–97 (2004).

20. S. Deans, The Radon transform and some of its applications (Wiley New York, 1983).

21. S. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE

Transactions on signal processing 41, 3397–3415 (1993).

22. J. Tropp, “Greed is good: Algorithmic results for sparse approximation,” IEEE Trans-

actions on Information Theory 50, 2231–2242 (2004).

23. Y. Pati, R. Rezaiifar, and P. Krishnaprasad, “Orthogonal matching pursuit: recursive

function approximation withapplications to wavelet decomposition,” in “1993 Conference

Record of The Twenty-Seventh Asilomar Conference on Signals, Systems and Computers,

1993,” (1993), pp. 40–44.

24. R. Maleh, A. Gilbert, and M. Strauss, “Sparse gradient image reconstruction done

21

faster,” in “IEEE International Conference on Image Processing, 2007. ICIP 2007,” ,

vol. 2 (2007), vol. 2.

25. Q. Zhang, H. Wang, R. Plemmons, and V. Pauca, “Tensor methods for hyperspectral

data analysis: a space object material identification study,” Journal of the Optical Society

of America A 25, 3001–3012 (2008).

26. P. Gader, A. Zare, R. Close, and G. Tuell, “Co-registered hyperspectral and LiDAR Long

Beach, Mississippi data collection,” (2010). University of Florida, University of Missouri,

and Optech International.

27. C. Cull, K. Choi, D. Brady, and T. Oliver, “Identification of fluorescent beads using a

coded aperture snapshot spectral imager,” Applied Optics 49, 59–70 (2010).

28. J. Bioucas-Dias and M. Figueiredo, “A new TwIST: two-step iterative shrink-

age/thresholding algorithms for image restoration,” IEEE Transactions on Image pro-

cessing 16, 2992–3004 (2007).

29. D. Kittle, K. Choi, A. Wagadarikar, and D. Brady, “Multiframe image estimation for

coded aperture snapshot spectral imagers,” Applied Optics 49, 6824–6833 (2010).

22

List of Figure Captions

Fig. 1. (a) Segmented bottom cylinder area using the original Chan-Vese model; (b)

Segmented same area using the generalized C-V.

Fig. 2. Segmentation of a DD-CASSI image (a) using the original TV algorithm and (b)

using the modified TV with known si(λ).

Fig. 3. The first 25 iterations of the estimation of membership functions from left to right

and top to bottom.

Fig. 4. The estimated spectral signatures in blue are compared with the true ones in red.

Fig. 5. The joint estimation of membership functions ui in (a) and spectral signatures, si in

(b) in the presence of noise (SNR = 21 dB).

Fig. 6. The left image shows a Google map of the area while the right image shows one band

of the simulated DD-CASSI image.

Fig. 7. The estimated spectra of the segmented areas.

Fig. 8. The segmentation map estimated from the DD-CASSI image.

Fig. 9. (a) The DD-CASSI image of the beads. (b) The estimated membership function

map. (c) The beads spectral signatures from [27]. (d) The pseudo-colored reconstructed

image using TV minimization [27], with the spectral signatures of the marked beads in (c).

Fig. 10. The top left image is a slice of the simulated hyperspectral cube; the top middle is

the simulated SD-CASSI image; the top right is the spectral signature applied to all pixels

in the square to the left; the bottom left is a slice of the reconstructed cube and the bottom

middle is the reconstructed spectral signature.

Fig. 11. The top left image is a slice of the simulated hyperspectral cube; the top middle is

the simulated SD-CASSI image; the top right are two spectral signatures applied to pixels

in two rectangles to the left; the bottom left is a slice of the reconstructed cube and the

bottom middle are the reconstructed spectral signatures.

Fig. 12. Comparison of three reconstruction methods using a single frame observation and

known si(λ). Only the middle 40 rows of the original scene is used here.

Fig. 13. (a) Relationship between the probability of a successful reconstruction and the

number of observed frames for the three reconstruction methods. (b) Computation time of

each method per iteration of ALS.

Fig. 14. (a) The estimated membership function map in 20 iterations. (b) The estimated

spectral signatures in blue, compared to the original in red.

Fig. 15. (a) The imaged color chart object. (b) The original SD-CASSI image with the

reconstructed area in the red rectangle box. (c) The reconstructed segments.

Fig. 16. (a) The reconstructed spectra compared with reference by an Ocean Optics

spectrometer. (b) The reconstructed spectra compared with reconstruction using TwIST.

1500 Iterations

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

(a)

1500 Iterations

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

(b)

Fig. 1. (a) Segmented bottom cylinder area using the original Chan-Vesemodel; (b) Segmented same area using the generalized C-V.

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

(a)

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

(b)

Fig. 2. Segmentation of a DD-CASSI image (a) using the original TV algorithmand (b) using the modified TV with known si(λ).

100 150 50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

100 150 50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

100 150 50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

100 150 50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

100 150 50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

50 100 150

50

100

150

Fig. 3. The first 25 iterations of the estimation of membership functions fromleft to right and top to bottom.

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Aluminum

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Glue

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Solar Cell

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Black Rubber Edge

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb side/Bolt

Reconstructed

Honeycomb side

Bolt

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Copper Stripping

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb Top

400 600 8000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Background

Fig. 4. The estimated spectral signatures in blue are compared with the trueones in red.

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

(a)

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Aluminum

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Glue

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Solar Cell

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Black Rubber Edge

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb side/Bolt

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Copper Stripping

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Honeycomb Top

400 500 600 7000

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Background

(b)

Fig. 5. The joint estimation of membership functions ui in (a) and spectralsignatures, si in (b) in the presence of noise (SNR = 21 dB).

100 200 300 400

50

100

150

200

250

300

350

40050 100 150 200 250 300 350

50

100

150

200

250

300

Fig. 6. The left image shows a Google map of the area while the right imageshows one band of the simulated DD-CASSI image.

400 600 8000

0.2

0.4

0.6

0.8

1Dirt

Ref

lect

ance

Wavelength (nm)400 600 8000

0.2

0.4

0.6

0.8

1Pavement

Ref

lect

ance

Wavelength (nm)400 600 8000

0.2

0.4

0.6

0.8

1Grass

Ref

lect

ance

Wavelength (nm)400 600 8000

0.2

0.4

0.6

0.8

1Tree

Ref

lect

ance

Wavelength (nm)

400 600 8000

0.2

0.4

0.6

0.8

1Sand

Ref

lect

ance

Wavelength (nm)400 600 8000

0.2

0.4

0.6

0.8

1Rooftop

Ref

lect

ance

Wavelength (nm)400 600 8000

0.2

0.4

0.6

0.8

1Water/Shadow

Ref

lect

ance

Wavelength (nm)

Fig. 7. The estimated spectra of the segmented areas.

50 100 150 200 250 300 350

50

100

150

200

250

300

Dirt

Pavement

Grass

Rooftop

Sand

Tree

Water/Shadow

Fig. 8. The segmentation map estimated from the DD-CASSI image.

100 200 300 400 500 600 700 800

100

200

300

400

500

600

(a)

100 200 300 400 500 600 700 800

100

200

300

400

500

600

R1R4

R5

RO1R3

CA2

Y2

Y7

YG1

G1

G2

BG1

Y3

O1Y1

Y5

Y4

Y6

R2

CA3

CA4

CA1

(b)

(c) (d)

Fig. 9. (a) The DD-CASSI image of the beads. (b) The estimated membershipfunction map. (c) The beads spectral signatures from [27]. (d) The pseudo-colored reconstructed image using TV minimization [27], with the spectralsignatures of the marked beads in (c).

20 40 60 80 100 120

20

40

60

80

100

120

50 100 150

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

20 40 60 80 100 120

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Fig. 10. The top left image is a slice of the simulated hyperspectral cube; thetop middle is the simulated SD-CASSI image; the top right is the spectralsignature applied to all pixels in the square to the left; the bottom left is aslice of the reconstructed cube and the bottom middle is the reconstructedspectral signature.

20 40 60 80 100120

20

40

60

80

100

120

50 100 150

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

20 40 60 80 100120

20

40

60

80

100

120

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

Fig. 11. The top left image is a slice of the simulated hyperspectral cube; thetop middle is the simulated SD-CASSI image; the top right are two spectralsignatures applied to pixels in two rectangles to the left; the bottom left is aslice of the reconstructed cube and the bottom middle are the reconstructedspectral signatures.

Pseudo−Inverse

20 40 60 80 100 120

10

20

30

40

Total Variation

20 40 60 80 100 120

10

20

30

40

Orthogonal Matching Pursuit

20 40 60 80 100 120

10

20

30

40

Fig. 12. Comparison of three reconstruction methods using a single frameobservation and known si(λ). Only the middle 40 rows of the original scene isused here.

(a)

(b)

Fig. 13. (a) Relationship between the probability of a successful reconstructionand the number of observed frames for the three reconstruction methods. (b)Computation time of each method per iteration of ALS.

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

20 40 60 80

20

40

60

80

(a)

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500

0.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500

0.2

0.4

0.6

0.8

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500.2

0.4

0.6

0.8

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 650

0.7

0.8

0.9

1

Wavelength (nm)

Ref

lect

ance

450 500 550 600 6500

0.5

1

1.5

Wavelength (nm)

Ref

lect

ance

(b)

Fig. 14. (a) The estimated membership function map in 20 iterations. (b) Theestimated spectral signatures in blue, compared to the original in red.

(a)

200 400 600 800 1000 1200 1400

100

200

300

400

500

600

700

800

900

(b)

200 400 600 800 1000 1200 1400

10

20

30

40

50

(c)

Fig. 15. (a) The imaged color chart object. (b) The original SD-CASSI imagewith the reconstructed area in the red rectangle box. (c) The reconstructedsegments.

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000.2

0.4

0.6

0.8

1Purplish Blue

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Moderate Red

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Yellow Green

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange Yellow

Wavelength (nm)

Ref

lect

ance

Ocean OpticsPI

(a)

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000.2

0.4

0.6

0.8

1Purplish Blue

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Moderate Red

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Purple

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Yellow Green

Wavelength (nm)

Ref

lect

ance

400 500 600 700 8000

0.2

0.4

0.6

0.8

1Orange Yellow

Wavelength (nm)

Ref

lect

ance

TwISTPI

(b)

Fig. 16. (a) The reconstructed spectra compared with reference by an OceanOptics spectrometer. (b) The reconstructed spectra compared with reconstruc-tion using TwIST.