Compressive Blind Source Separation

BackgroundsBayesian Probabilistic Inference

Numerical ResultsConclusions and Future Work

Compressive Blind Source Separation

Yiyue Wu, Yuejie Chi, Robert Calderbank

Dept. of Electrical EngineeringPrinceton University

Sept. 27, 2010

Yuejie Chi ICIP 2010: Compressive BSS



Outline

1 BackgroundsMotivationsBackground

2 Bayesian Probabilistic InferenceProblem FormulationPrior DistributionMarkov Chain Monte Carlo

3 Numerical ResultsOne-dimensional BSSTwo-dimensional BSS

4 Conclusions and Future Work




MotivationsBackground

Bridging Compressive Sensing and Machine Learning

There are growing interests in applying sparse techniques tomachine learning and image processing:

SVM can be done in compressed domain [CJS09];Multi-label prediction via CS [HKLZ09];Bayesian inference for reconstruction [HC09, DWB08] [IMD06];Bayesian inference for image denoising, inpainting...

This work and take-away message: pick an interestingproblem to showcase compressed measurements are as goodas complete measurements as long as you have enoughmeasurements.





Motivations

Blind Source Separation (BSS) from conventional mixtures

Important in many areas: speech recognition, MIMOcommunications, etc.

In many cases, measurements are expensive:

Body Area Networks (BAN): sensors are power hungry andneed to last a few days to a few weeks.Recent developments in Compressive Sensing (CS) provide anintriguing solution.

Our work: recover mixtures from small measurements:

Conventional methods like PCA and ICA may fail due toreduced dimensionality.





Background: Compressive Sensing

Recovery of a sparse or compressible signal from a smallnumber of linear measurements [Don06, CT05].

y = Φx + n, Φ ∈ CM×N , M � N

Many classes of reconstruction algorithms available now:

”Old-fashioned” `1 minimization.Greedy algorithms: OMP, CoSaMP, GPSR...Bayesian inference.

Theoretical performance gaurantee is usually given byRestricted Isometry Property (RIP) of measurement matrixsatisfied by random matrices with high probability.





Background: Blind Source Separation

Goal: to recover T independent sources from L observedmixture of sources, possibly corrupted by noise.

X = ΘA + ε

where

X ∈ CN×L is the matrix of observations;Θ ∈ CN×T is the matrix of sources;ε ∈ CN×L is the noise;A ∈ CT×L is the mixing matrix.

Without loss of generality, we let all representations lie in thewavelet domain.

particularly fit for image applications.





Approaches

Separate procedures: Mixture Recovery + BSS

We address a Baysian answer to this problem.

Simplified proceduresBetter performance




Problem FormulationPrior DistributionMarkov Chain Monte Carlo

Problem Formulation

Compressed measurements of mixtures of the sources:

Y = ΦΘA + N,

sop(Yk |Φ,Θ,A, αN

k ) ∼ N (ΦΘAk , (αNk )−1I),

where Yk and Ak are the kth columns of Y and A..

To maximize the posterior distribution:

p(Θ,A,N|Y,Φ)

∝ p(Y|Φ,Θ,A, αN)π(N|αN)π(A|αA)π(Θ|αΘ)

where α = [αN, αA, αΘ] is the set of the hyper parameters.





Hidden Markov Tree Model

Model the statistical dependencies between wavelet-domaincoefficients. [CNB98]

Persistence:Parent node large/small

h.p.−→ Child node large/small

A node large/smallh.p.−→ Adjacent nodes large/small

Mixed Gaussian Model:

θs,i ∼ (1− πs,i )δ0

+πs,iN (0, (αs)−1)





Prior Distributions

Prior distribution of noise variance:

αNk ∼ Gamma(a0, b0)

Prior distribution of A = {aij}:

aij ∼ N (µij , α−1ij ), 1 ≤ i ≤ T , 1 ≤ j ≤ L

Prior distributions of Θ:

θs,i ∼ (1− πs,i )δ0 + πs,iN (0, (αs)−1)

with πs,i =

πr ∼ Beta(er0, f

r0 ), if s = 1

πs0 ∼ Beta(es00 , f

s00 ), if 2 ≤ s ≤ S , θp(s,i) = 0

πs1 ∼ Beta(es10 , f

s10 ), if 2 ≤ s ≤ S , θp(s,i) 6= 0

αs ∼ Gamma(c0, d0)





MCMC

The Gibbs sampler samples from the following conditionaldistributions at iteration t,

θs,ik (t) ∼ p(θs,ik |Y,Φ,A(t − 1), αN(t − 1), αsk(t − 1), πs,ik (t − 1)),

A(t) ∼ p(A|Y,Φ,A(t − 1),Θ(t − 1), αN(t − 1)),

αNk (t) ∼ p(αN

k |Yk ,Φ,Ak(t − 1),Θ(t − 1)),

αsk(t) ∼ p(αs

k |θs,ik (t − 1)),

πs,ik (t) ∼ p(πs,ik |θs,ik (t − 1)),

where {θs,ik (t)} is the set of wavelet coefficients associatedwith the kth source in the tth iteration.

Note: all distributions belong to the exponential family!




One-dimensional BSSTwo-dimensional BSS

One-dimensional BSS

100 200 300 400 500−1

−0.50

0.5

Original Signal 0

100 200 300 400 500

−1

−0.5

0Original Signal 1

100 200 300 400 500−1

−0.50

0.5Mixture Signal 0

100 200 300 400 500

−1−0.5

0

Mixture Signal 1

100 200 300 400 500

−1−0.5

00.5

Sep−Recovered Signal 0

100 200 300 400 500

−1−0.5

0


100 200 300 400 500−1

−0.50

0.5Bayes−Recovered Signal 0

100 200 300 400 500

−1

−0.5

0Bayes−Recovered Signal 1

Figure: Bayesian compressive blind separation of one dimensional signals

Our proposed method out performs the separate procedureYuejie Chi ICIP 2010: Compressive BSS




One-dimensional BSS

100 200 300 400 500

−2

0

2

4

Original Signal 0

100 200 300 400 500−2

−1

0

1

Original Signal 1

100 200 300 400 500

−2

0

2

4

Bayes−Recovered Signal 0

100 200 300 400 500

−2

0

2Bayes−Recovered Signal 1

100 200 300 400 500

−2

0

2

4


100 200 300 400 500

−2

0

2Sep−Recovered Signal 1

Figure: Comparisons of recovered wavelet coefficients

Original signals are sparse

Our proposed method out performs the separate procedure





Two-dimensional BSS

Original Image 0

10 20 30

10

20

30

Mixed Image 0

10 20 30

10

20

30

Bayes−Recovered Image 0

10 20 30

10

20

30

Sep−Recovered Image 0

10 20 30

10

20

30

Original Image 1

10 20 30

10

20

30

Mixed Image 1

10 20 30

10

20

30

Bayes−Recovered Image 1

10 20 30

10

20

30

Sep−Recovered Image 1

10 20 30

10

20

30

Figure: Bayesian compressive blind separation of two images





Two-dimensional BSS

50 100 150 200 250

0

500

1000

Original Signal 0

50 100 150 200 250

−500

0

500

1000

Original Signal 1

50 100 150 200 250−400−200

0200400600


50 100 150 200 250

−500

0

500


50 100 150 200 250−1000

0

1000


50 100 150 200 250

−1000

0

1000


Figure: Comparisons of the first 256 recovered wavelet coefficients

Original signals are not well sparse




Conclusions and Future Work

Conclusions:

We have addressed the blind source separation problemdirectly from the compressed mixtures obtained fromcompressive sensing measurements.

Our approach out performs the existing separate procedure.

Future work:

Improving our proposed method in separation and recovery ofnearly sparse signals.

Incorporating dictionary learning in the inference procedure in2-D case to obtain better performance.




Main References

Robert Calderbank, Sina Jafarpour, and Robert Schapire.

Compressed learning: Universal sparse dimensionality reduction and learning in the measurement domain.Preprint, 2009.

M. S. Crouse, R. D. Nowak, and R. G. Baraniuk.

Wavelet-based statistical signal processing using hidden markov models.IEEE Trans. Signal Processing, 46(4):882–902, 1998.

E. J. Candes and T. Tao.

Decoding by linear programming.IEEE Trans. Info. Theory, 51:4203– 4215, 2005.

D. L. Donoho.

Compressed sensing.IEEE Trans. Info. Theory, 52(4):1289–1306, 2006.

Marco Duarte, Michael Wakin, and Richard Baraniuk.

Wavelet-domain compressive signal reconstruction using a hidden markov tree model.2008.

Lihan He and Lawrence Carin.

Exploiting structure in wavelet-based bayesian compressive sensing.IEEE Trans. Signal Processing, 57(9):3488–3497, 2009.

Mahieddine M. Ichir and Ali Mohammad-Djafari.

Hidden markov models for wavelet-based blind source separation.IEEE Transactions on Image Processing, 15(7):1887–1899, 2006.




THANK YOU!


Compressive Blind Source Separation

Documents

Transcript of Compressive Blind Source Separation