Fast Compressive Sampling UsingFast Compressive Sampling Using Structurally Random Matrices

Presented by:Thong Do (thongdo@jhu.edu)g ( g @j )The Johns Hopkins University

A joint work with

P f T T Th J h H ki U i it1

Prof. Trac Tran, The Johns Hopkins University

Dr. Lu Gan, Brunel University, UK

Compressive Sampling FrameworkCompressive Sampling Framework• Main assumption: K-sparse representation of an input signal

Ψ: sparsifying transform;

• Compressive sampling:Φ

1 1N N N Nx α× × ×= Ψ Ψ: sparsifying transform;α: transform coefficients;

– : random matrices, random row subset of an orthogonal matrix (partial Fourier),etc.

1 1M M N Ny x× × ×= ΦM N×Φ

Sensing matrix

M N×Φ N N×Ψ1My × 1Nα ×

has K nonzero entries

1Nα ×

Sensing matrix=

C d

e t es

• Reconstruction: L1-minimization (Basis Pursuit)

Compressed measurements

21

arg minα

α α= s.t. y α= ΦΨ

x α= Ψ

A Wish-list of the Sampling OperatorA Wish list of the Sampling Operator

• Optimal Performance:Optimal Performance:– require the minimal number of compressed

measurementsmeasurements

• Universality:incoherent with various families of signals– incoherent with various families of signals

• Practicality:f i– fast computation

– memory efficiency– hardware friendly– streaming capability

3

Current Sensing MatricesCurrent Sensing Matrices• Random matrices[Candes, Tao, Donoho]

Optimal performanceHuge memory and computational complexityNot appropriate in large scale applications

• Partial Fourier[Candes et. al.]Fast computationNon-universality

O l i h t ith i l i ti• Only incoherent with signals sparse in time• Not incoherent with smooth signals such as natural images.

• Other methods (Scrambled FFT RandomOther methods (Scrambled FFT, Random Filters,…)

Either lack of universality or no theoretical guarantee4

y g

MotivationMotivation• Significant performance improvement of

bl d i l iscrambled FFT over partial FourierA well-known fact [Baraniuk, Candès] Original 512x512 Lena image[ ]

But no theoretical justificationCompressed

FFTRandom downsampler

pmeasurements

Reconstruction from 25% of measurements: 16.5 dB

ReconstructionReconstructionBasis Pursuit

5

MotivationMotivation• Significant performance improvement of

bl d i l iscrambled FFT over partial FourierA well-known fact [Baraniuk, Candès] Original 512x512 Lena image[ ]

But no theoretical justificationCompressed

ScrambledFFT

Random downsampler

pmeasurements

Reconstruction from 25% of measurements: 29.4 dB

ReconstructionReconstructionBasis Pursuit

6

Our ContributionsOur Contributions• Propose the concept of Structurally random• Propose the concept of Structurally random

ensemblesE t i f S bl d F i E bl– Extension of Scrambled Fourier Ensemble

• Provide theoretical guarantee for this novel sensing framework

• Design sensing ensembles with practical g g pfeatures– Fast computable memory efficient hardwareFast computable, memory efficient, hardware

friendly, streaming capability etc.

7

Proposed CS SystemProposed CS System• Pre-randomizer:

• Global randomizer: random permutation of sample indicesp

• Local randomizer: random sign reversal of sample valuessample values

FFT,WHT,DCT Pre-randomizerInput signalRandom

downsampler

Compressed measurements

downsampler

Reconstruction signal recovery

8Basis Pursuit

Proposed CS SystemProposed CS System• Compressive Sampling

• Pre-randomize an input signal (P)• Apply fast transform (T)

( ( ( ))) ( )y D T P x A x= =

pp y ( )• Pick up a random subset of transform coefficients (D)

• ReconstructionB i P it ith i t d it dj i t• Basis Pursuit with sensing operator and its adjoint:

( ( ( ( ))))A D T P= Ψ • * * * * *( ( ( ( )))))A P T D= Ψ •

TFFT WHT DCT

PPre-randomizer

Input signalDRandom downsampler

Compressed measurements

x α= ΨFFT,WHT,DCT,… Pre randomizerRandom downsampler

Reconstruction signal recovery

x α= Ψ

9Basis Pursuit

Proposed CS SystemProposed CS System• Compressive Sampling

• Pre-randomize an input signal (P)• Apply fast transform (T)

( ( ( ))) ( )y D T P x A x= =

pp y ( )• Pick up a random subset of transform coefficients (D)

• ReconstructionB i P it ith i t d it dj i t

Structurally random matrix

• Basis Pursuit with sensing operator and its adjoint:

( ( ( ( ))))A D T P= Ψ • * * * * *( ( ( ( )))))A P T D= Ψ •

TFFT WHT DCT

PPre-randomizer

Input signalDRandom downsampler

Compressed measurements

x α= ΨFFT,WHT,DCT,… Pre randomizerRandom downsampler

Reconstruction signal recovery

x α= Ψ

10Basis Pursuit

Structurally Random MatricesStructurally Random Matrices

• Structurally random matrices with local• Structurally random matrices with local randomizer: a product of 3 matrices

Random downsamplerPr( 0) 1d M N Fast transformPr( 1)iid M N= =Pr( 0) 1iid M N= = −

Local randomizerPr( 1) 1/ 2iid ′ = ± =

Fast transformFFT, WHT, DCT,…

( )ii

11

Structurally Random MatricesStructurally Random Matrices• Structurally random matrices with global y g

randomizer: a product of 3 matrices

Global randomizerRandom downsamplerPr( 0) 1d M N Fast transform Global randomizer

Uniformly random permutation matrix

Pr( 1)iid M N= =Pr( 0) 1iid M N= = − FFT, WHT, DCT,…

p

12Partial Fourier

Sparse Structurally Random MatricesSparse Structurally Random Matrices• With local randomizer:

• Fast computation • Memory efficiency• Hardware friendly• Streaming capabilityg p y

Random downsampler Local randomizerBlock-diagonal WHT, DCT FFT etc Pr( 1) 1/ 2d ′ ±p

Pr( 1)iid M N= =Pr( 0) 1iid M N= = −

DCT, FFT, etc. Pr( 1) 1/ 2iid = ± =

13

Sparse Structurally Random MatricesSparse Structurally Random Matrices• With global randomizer

• Fast computation • Memory efficiency• Hardware friendly• Nearly streaming capabilityy g p y

Global randomizerUniformly random Random downsampler

Block-diagonal WHT, DCT FFT etc f y

permutation matrixp

Pr( 1)iid M N= =Pr( 0) 1iid M N= = −

DCT, FFT,etc.

14

Theoretical AnalysisTheoretical Analysis• Theorem 1: Assume that the maximum absolute

entries of a structurally random matrix and an orthonormal matrix is not larger than Wi h hi h b bili h f d iΨΦ

M N×ΦN N×Ψ 1 log N

With high probability, coherence of and is not larger than

th b f t i f( log / )N sΟ

Φ

N N×ΨM N×Φ

– s: the average number of nonzero entries per row of

• Proof: M N×Φ

– Bernstein’s concentration inequality of sum of independent random variables

Th ti l h (G i /B lli d• The optimal coherence (Gaussian/Bernoulli random matrices): ( log / )N NΟ

15

Theoretical Analysisy• Theorem 2: With the previous assumption, sampling

a signal using a structurally random matrix guaranteesa signal using a structurally random matrix guarantees exact reconstruction (by Basis Pursuit) with high probability, provided thatp y, p

– s: the average number of nonzero entries per row of the

2~ ( / ) logM KN s N

sampling matrix – N: length of the signal, – K: sparsity of the signalK: sparsity of the signal

• Proof: – follow the proof framework of [Candès2007] and previous p [ ] p

theorem of coherence of the structurally random matrices• The optimal number of measurements required by

Ga ssian/Berno lli d random matrices: lK N16

Gaussian/Bernoulli dense random matrices: logK N[Candès2007] E. Candès and J. Romberg, “Sparsity and incoherence in compressive sampling”, Inverse Problems, 23(3) pp. 969-985, 2007

Simulation Results: Sparse 1D Signalsp g• Input signal sparse in DCT p g p

domain– N=256, K = 30

R t ti• Reconstruction: – Orthogonal Matching Pursuit(OMP)

• WHT256 + global randomizerWHT256 + global randomizer– Random permutation of samples indices

+ Walsh-Hadamard

WHT256 + l l d i• WHT256 + local randomizer– Random sign reversal of sample values

+ Walsh-Hadamardh f i f• WHT8 + global randomizer

– Random permutation of samples indices + block diagonal Walsh-8 8×

The fraction of nonzero entries:1/32

32 times sparser than S bl d FFT i i d

17Hadamard Scrambled FFT, i.i.d

Gaussian ensemble,…

Simulation Results: Compressible 2D Signalsp g• Experiment set-up:

Test images: 512×512 Lena and Boat;– Test images: 512×512 Lena and Boat;– Sparsifying transform ψ: Daubechies 9/7 wavelet transform;

L1 minimi ation sol er: GPSR [Mario Robert Stephen]– L1-minimization solver: GPSR [Mario, Robert, Stephen]– Structually random sensing matrices Φ:

• WHT512 & local randomizer– Random sign reversal of sample values + 512×512 block diagonal

Hadamard transform;– Full streaming capability

• WHT32 & global randomizer

– Random permutation of sample indices + 32×32 block diagonal Hadamard transform;

– Highly sparse: The fraction of nonzero entries is only 1/213

18

– Highly sparse: The fraction of nonzero entries is only 1/2

Rate-Distortion Performance: Lena

P i l FFT i lFull

Partial FFT in wavelet domain:– transform the image into wavelet coeffs

streaming capability

8000 times sparser than

into wavelet coeffs– sense these coeffs (rather than sense directly image pixels) using partial FFTsparser than

the Scrambled FFT

FFTNo universalityMore computational

complexityp yServes as a benchmark

19R-D performance of 512x512 Lena

Reconstructed Images: Lena• Reconstruction from 25% of measurements using GPSR

Original 512x512 Lena image Partial FFT:16 dB Partial FFT in wavelet domain: 30.1 dBOriginal 512x512 Lena image Partial FFT:16 dB

WHT512 + local WHT32 + globalScrambled FFT:29 3 dB

20

WHT512 + local randomizer: 28.4 dB

WHT32 + global randomizer: 29 dB

Scrambled FFT:29.3 dB

Rate-Distortion Performance: BoatRate Distortion Performance: Boat

21R-D performance of 512x512 Boat image

Future ResearchFuture Research• Develop theoretical analysis of structurally random p y y

matrices with greedy, iterative reconstruction algorithms such as OMP

• Application of structurally random matrices to high dimensionality reduction– Fast Johnson-Lindenstrauss transform using structurally

random matricesCl t d t i i ti i li• Closer to deterministic compressive sampling– Replace a random downsampler by a deterministic

downsampler or a deterministic lattice of measurements*downsampler or a deterministic lattice of measurements– Develop theoretical analysis for this nearly deterministic

framework

22* Basarab Matei and Yves Meyer, “A variant of the compressed sensing of Emmanuel Candès ”, Preprint, 2008.

ConclusionsConclusions• Structurally random matrices

– Fast computable; – Memory efficient:

Hi hl l i d i f bl k• Highly sparse solution: random permutation ⇒ fast block diagonal transform ⇒ random sampling;

– Streaming capability: g p y• Solution with full streaming capability: random sign flipping

⇒ fast block diagonal transform ⇒ random sampling; Hardware friendly;– Hardware friendly;

– Performance:• Nearly optimal theoretical bounds;Nearly optimal theoretical bounds; • Numerical simulations: comparable with completely random

matrices

23

ReferencesReferences• E. Candès and J. Romberg, “Sparsity and incoherence in compressive sampling”, Inverse Problems,

23(3) pp. 969-985, 2007E C dè J R b d T T “R b i i i l E i l i f• E. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles:Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. on Information Theory, vol. 52, pp. 489 – 509, Feb. 2006.

• E. Candès, J. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Communications on Pure and Applied Mathematics, vol. 59, pp. 1207–1223, Aug. 2006.

• M. F. Duarte, M. B. Wakin, and R. G. Baraniuk, “Fast reconstruction of piecewise smooth signals from incoherent projections,” SPARS’05, Rennes, France, Nov 2005.

• Basarab Matei and Yves Meyer “A variant of the compressed sensing of Emmanuel Candès ” PreprintBasarab Matei and Yves Meyer, A variant of the compressed sensing of Emmanuel Candès , Preprint, 2008.

• Mário A. T. Figueiredo, Robert D. Nowak, and Stephen J. Wright, “Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems”, IEEE Journal of Selected Topics in Signal Processing: Special Issue on Convex Optimization Methods for Signal Processing 1(4)Topics in Signal Processing: Special Issue on Convex Optimization Methods for Signal Processing, 1(4), pp. 586-598, 2007.

24

• thanglong ece jhu edu/CS/fast cs SRM rar• thanglong.ece.jhu.edu/CS/fast_cs_SRM.rar

25