Compressive Sensing - Examples in Image Compressionw3.impa.br/~aschulz/CS/Lecture4.pdf · (a) DCT 0...
Transcript of Compressive Sensing - Examples in Image Compressionw3.impa.br/~aschulz/CS/Lecture4.pdf · (a) DCT 0...
Compressive Sensing
1
Compressive SensingExamples in Image Compression
Lecture 4 , July 30, 2009
Luiz VelhoEduardo A. B. da SilvaAdriana Schulz
Compressive Sensing
2
Today’s Lecture
• Discuss applications of CS in image compression• Evaluate CS efficiency• Review important definitions and theorems
Compressive Sensing
3
Software
• Matlab functions for CS imaging• Optimization algorithms• Data• Demos
http://www.impa.br/∼aschulz/CS/course.html
Compressive Sensing
4
Experimental Setup• 4 images of size 256× 256
(a) Phantom (b) Lena
(c) Camera man (d) Text
Compressive Sensing
5
Frequency Distributions
50 100 150 200 250
50
100
150
200
250
(a) Phantom
50 100 150 200 250
50
100
150
200
250
(b) Lena
50 100 150 200 250
50
100
150
200
250
(c) Camera man
50 100 150 200 250
50
100
150
200
250
(d) Text
Compressive Sensing
6
Acquisition
• Images are stored as a matrix of pixels
Compressive Sensing
7
Noiselets
• RIP• orthogonal• self-adjoint
Φ =12·
1 −1 1 1
−1 1 1 11 1 −1 11 1 1 −1
Compressive Sensing
8
Sparsity
50
100
150
200
250
Compressive Sensing
9
Reconstruction
Acquisition:y = ΦΩx
Reconstruction:y = ΘΩs
s = mins ‖s‖l1 s. t.
y = Θs‖y −Θs‖l2 ≤ ε
x = Ψ∗s
Compressive Sensing
9
Reconstruction
Acquisition:y = ΦΩx
Reconstruction:y = ΘΩs
s = mins ‖s‖l1 s. t.
y = Θs‖y −Θs‖l2 ≤ ε
x = Ψ∗s
Compressive Sensing
9
Reconstruction
Acquisition:y = ΦΩx
Reconstruction:y = ΘΩs
s = mins ‖s‖l1 s. t.
y = Θs‖y −Θs‖l2 ≤ ε
x = Ψ∗s
Compressive Sensing
9
Reconstruction
Acquisition:y = ΦΩx
Reconstruction:y = ΘΩs
s = mins ‖s‖l1 s. t.
y = Θs‖y −Θs‖l2 ≤ ε
x = Ψ∗s
Compressive Sensing
10
Basic CS• Images are sparse in the Ψ domain• No noise is added
Ordinary images are only approximately sparse :
50
100
150
200
250
(a)
50
100
150
200
250
(b)
Figure: Different visualizations of Lena’s DCT.
Compressive Sensing
10
Basic CS• Images are sparse in the Ψ domain• No noise is added
Ordinary images are only approximately sparse :
50
100
150
200
250
(a)
50
100
150
200
250
(b)
Figure: Different visualizations of Lena’s DCT.
Compressive Sensing
11
Forcing SparsityDCT Example:
(a) Original Image (b) Original DCT (c) Original DCT
(d) Result Image (e) Result DCT (f) Result DCT
Compressive Sensing
12
First Experiment
• x Ψ−→ sforce sparsity−−−−−−−→ sS
Ψ∗
−−→ xS
• Acquisition: y = ΦΩxS
• Reconstruction of x via l1 minimization
• Compare x and xS using PSNR (Peak Signal to Noise Ratio)
Compressive Sensing
13
First Result
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
Figure: Basic CS experiment: 10k-sparse Lena.
Theorem
Reconstruction is exact if
M & S · µ2(Φ,Ψ) · log N
Compressive Sensing
14
Varying S
10 20 30 40 50 6020
25
30
35
40
45
50
55
60
65
70
Measurements( × 103)
PS
NR
3.5k−Sparse6k−Sparse10k−Sparse14k−Sparse
Compressive Sensing
15
DCT Linear Compression
We measure M N coefficients on the upper-left corner
Compressive Sensing
15
DCT Linear Compression
We measure M N coefficients on the upper-left corner
Compressive Sensing
15
DCT Linear Compression
We measure M N coefficients on the upper-left corner
Compressive Sensing
15
DCT Linear Compression
We measure M N coefficients on the upper-left corner
Compressive Sensing
16
DCT Linear Compression
Reconstruction: x = Ψ∗s
Compressive Sensing
17
DCT Linear Compression
Problem: we don’t measure all the significant coefficients andmeasure some zero coefficients!
Compressive Sensing
17
DCT Linear Compression
Problem: we don’t measure all the significant coefficients andmeasure some zero coefficients!
Compressive Sensing
18
CS: Linear + Noiselets Measurements
Reconstruction: s = mins ‖s‖l1 s. t. Φ′
ΩΨ∗s = yx = Ψ∗s
Compressive Sensing
19
Results: CS × Linear Compression10 20 30 40 50 60
[DCT−l1−N] [DCT−l
2−L] [DCT−l
1−LN]
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(a) 3.5K-Sparse
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(b) 6K-Sparse
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(c) 10K-Sparse
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(d) 14K-Sparse
Compressive Sensing
20
Coherence
Before Boundary: best to use linear compressionAfter Boundary: best to use CS
10 20 30 40 50 6020
30
40
50
60
70
80
90
100
110
120
Measurements( × 103)
PS
NR
DCT−l1−N
DCT−l1−LN
Compressive Sensing
21
Different ImagesWhat happens when we consider the other images?
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Linear Compression
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(f) Compressive Sensing
(a) Phantom (b) Lena (c) Camera man (d) Text
Compressive Sensing
21
Different ImagesWhat happens when we consider the other images?
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Linear Compression
10 20 30 40 50 6020
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(f) Compressive Sensing
0 50 100 150 200 250
0
50
100
150
200
250
(a) Phantom
0 50 100 150 200 250
0
50
100
150
200
250
(b) Lena
0 50 100 150 200 250
0
50
100
150
200
250
(c) Camera man
0 50 100 150 200 250
0
50
100
150
200
250
(d) Text
Compressive Sensing
21
Different ImagesWhat happens when we consider the other images?
20 40 60 phantom lena cameraman text
10 20 30 40 50 6020
30
40
50
60
70
80P
SN
R
Measurements( × 103)
(a) Linear Compression
10 20 30 40 50 6010
20
30
40
50
60
70
80
PS
NR
Measurements( × 103)
(b) Compressive Sensing
0 50 100 150 200 250
0
50
100
150
200
250
(a) Phantom
0 50 100 150 200 250
0
50
100
150
200
250
(b) Lena
0 50 100 150 200 250
0
50
100
150
200
250
(c) Camera man
0 50 100 150 200 250
0
50
100
150
200
250
(d) Text
Compressive Sensing
22
Different Sparsity DomainsWhat happens when we consider different sparsity domains?
0 50 100 150 200 250
0
50
100
150
200
250
nz = 10000
(a) DCT
0 50 100 150 200 250
0
50
100
150
200
250
nz = 10000
(b) Block DCT
0 50 100 150 200 250
0
50
100
150
200
250
nz = 10000
(c) Wavelets
10 20 30 40 50 6020
25
30
35
40
45
50
55
60
65
70
PS
NR
Measurements( × 103)
DCT−l1−N
B_DCT−l1−N
DWT−l1−N
Compressive Sensing
22
Different Sparsity DomainsWhat happens when we consider different sparsity domains?
0 50 100 150 200 250
0
50
100
150
200
250
nz = 10000
(a) DCT
0 50 100 150 200 250
0
50
100
150
200
250
nz = 10000
(b) Block DCT
0 50 100 150 200 250
0
50
100
150
200
250
nz = 10000
(c) Wavelets
10 20 30 40 50 6020
25
30
35
40
45
50
55
60
65
70
PS
NR
Measurements( × 103)
DCT−l1−N
B_DCT−l1−N
DWT−l1−N
Compressive Sensing
23
But this is not for real...
All this happens because we FORCE sparsity!
Theorem
Reconstruction is exact if
M & S · µ2(Φ,Ψ) · log N
Important Parameters:• µ - Noiselets are highly incoherent with all considered domains• S - We are tempering with it! - Lets stop and see what happens...
Compressive Sensing
24
Sparsity Errors
When Φ are Noiselet measurements the RIP holds and we can use:
Theorem
Let sS be the best S-sparse approximation of s.If the RIP holds, the solution s to
s = mins‖s‖l1 subject to ΘΩs = y
obeys
‖s − s‖l2 .1√S· ‖s − sS‖l1
Compressive Sensing
25
Let’s verify this theorem empirically
Compressive Sensing
26
Let’s verify this theorem empirically
10 20 30 40 50 6020
22
24
26
28
30
32
34
36
38
40
Measurements( × 103)
PS
NR
3.5k−Sparse
6k−Sparse
10k−Sparse
14k−Sparse
10 20 30 40 50 6020
25
30
35
40
45
50
PS
NR
Measurements( × 103)
10 20 30 40 50 6020
25
30
35
40
45
50
55
60
65
Measurements( × 103)
PS
NR
Test image LenaMeasurements Sparsity is forced Sparsity is not forced
M = 20k S = 3.5k PSNR = 28.8 PSNR = 26.6M = 25k S = 6k PSNR = 30.7 PSNR = 27.8M = 35k S = 10k PSNR = 33.0 PSNR = 30.2M = 40k S = 14k PSNR = 34.9 PSNR = 31.5
Compressive Sensing
26
Let’s verify this theorem empirically
10 20 30 40 50 6020
22
24
26
28
30
32
34
36
38
40
Measurements( × 103)
PS
NR
3.5k−Sparse
6k−Sparse
10k−Sparse
14k−Sparse
10 20 30 40 50 6020
25
30
35
40
45
50
PS
NR
Measurements( × 103)
Test image LenaMeasurements Sparsity is forced Sparsity is not forced
M = 20k S = 3.5k PSNR = 28.8 PSNR = 26.6M = 25k S = 6k PSNR = 30.7 PSNR = 27.8M = 35k S = 10k PSNR = 33.0 PSNR = 30.2M = 40k S = 14k PSNR = 34.9 PSNR = 31.5
Compressive Sensing
27
Different Sparsity Domains
10 20 30 40 50 60 [DCT−l
2−L] [B_DCT−l
2−L] [DCT−l
1−N] [B_DCT−l
1−N] [DWT−l
1−N] [SVD−l
1−N] [TV−l
1−N]
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(b) Phantom
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(c) Lena
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(d) Camera man
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Text
Compressive Sensing
27
Different Sparsity Domains
10 20 30 40 50 60 [DCT−l
2−L] [B_DCT−l
2−L] [DCT−l
1−N] [B_DCT−l
1−N] [DWT−l
1−N] [SVD−l
1−N] [TV−l
1−N]
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(b) Phantom
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(c) Lena
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(d) Camera man
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Text
Compressive Sensing
27
Different Sparsity Domains
10 20 30 40 50 60 [DCT−l
2−L] [B_DCT−l
2−L] [DCT−l
1−N] [B_DCT−l
1−N] [DWT−l
1−N] [SVD−l
1−N] [TV−l
1−N]
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(b) Phantom
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(c) Lena
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(d) Camera man
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Text
Compressive Sensing
27
Different Sparsity Domains
Total Variation (TV)
minx‖x‖TV subject to y = ΦΩx
• l1-norm of the (appropriately discretized) gradient.
Compressive Sensing
27
Different Sparsity Domains
10 20 30 40 50 60 [DCT−l
2−L] [B_DCT−l
2−L] [DCT−l
1−N] [B_DCT−l
1−N] [DWT−l
1−N] [SVD−l
1−N] [TV−l
1−N]
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(b) Phantom
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(c) Lena
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(d) Camera man
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Text
Compressive Sensing
27
Different Sparsity Domains
Singular Value Decomposition (SVD)
• Calculated for each image!• Upper bounds - insights into performance limitations.
Compressive Sensing
27
Different Sparsity Domains
10 20 30 40 50 60 [DCT−l
2−L] [B_DCT−l
2−L] [DCT−l
1−N] [B_DCT−l
1−N] [DWT−l
1−N] [SVD−l
1−N] [TV−l
1−N]
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(b) Phantom
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(c) Lena
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(d) Camera man
10 20 30 40 50 6010
20
30
40
50
60
70
80
Measurements( × 103)
PS
NR
(e) Text
Compressive Sensing
28
Measurement Errors
Measurements: y = ΦΩx + n
• n is a random variable with normal distribution
Theorem
If the RIP holds and ‖n‖l2 ≤ ε, the solution s to
s = mins‖s‖l1 s.t. ‖ΘΩs − y‖l2 ≤ ε
obeys
‖s − s‖l2 .1√S· ‖s − sS‖l1 + ε
Compressive Sensing
29
Gaussian Errors
10 20 30 40 50 6015
20
25
30
35
40
45
50
55
60
65
70
Measurements( × 103)
PS
NR
var = 0.1var = 0.5var = 1var = 2var = 3var = 4var = 5var = 10
Compressive Sensing
30
Compression
Number of Measurements × Number of Bits
• To compress we have to quantize the acquired data!
Compressive Sensing
31
Quantization
−300 −200 −100 0 100 200 3000
5
10
15
−300 −200 −100 0 100 200 3000
200
400
600
800
Each measurement may assume K values.
Compressive Sensing
32
Quantization
• Larger quatization steps: less bits• Smaller quatization steps: less errors
Compressive Sensing
33
Rate
Rate = number of bits per pixel
Original Image:• N pixels• 8 bits/pixel
Acquired Measurements:• M N coefficients• Hy bits/coefficients
Rate =MN
Hy bits/pixel
If Rate < 8 data is being compressed!
Compressive Sensing
34
The Rate × Distortion Criteria
=⇒ which is the best compression scheme?
Compressive Sensing
34
The Rate × Distortion Criteria
=⇒ which is the best compression scheme?
Compressive Sensing
35
Rate × Distortion
step = 0.01step = 0.02step = 0.05step = 0.1step = 0.2step = 0.5
step = 100
step = 1step = 2step = 3step = 4step = 5step = 10step = 20step = 50
10 20 30 40 50 6010
20
30
40
50
60
70
80
90
100
Measurements x 103
PS
NR
Compressive Sensing
35
Rate × Distortion
0 2 4 6 8 10 12 14 16 180
10
20
30
40
50
60
70
80
90
100
step = 0.01step = 0.02step = 0.05step = 0.1step = 0.2step = 0.5step = 1step = 2step = 3step = 4step = 5step = 10step = 20step = 50step = 100
Rate
PS
NR
Compressive Sensing
35
Rate × Distortion
0 2 4 6 8 10 12 14 16 180
10
20
30
40
50
60
70
80
90
100
step = 0.01step = 0.02step = 0.05step = 0.1step = 0.2step = 0.5step = 1step = 2step = 3step = 4step = 5step = 10step = 20step = 50step = 100
Rate
PS
NR
Compressive Sensing
35
Rate × Distortion
step = 0.01step = 0.02step = 0.05step = 0.1step = 0.2step = 0.5step = 1step = 2step = 3step = 4step = 5step = 10step = 20step = 50step = 100
0 2 4 6 8 10 12 14 16 180
10
80
90
100
Compressive Sensing
35
Rate × Distortion
0 2 4 6 8 10 12 14 16 180
10
20
30
40
50
60
70
80
90
100
step = 0.01step = 0.02step = 0.05step = 0.1step = 0.2step = 0.5step = 1step = 2step = 3step = 4step = 5step = 10step = 20step = 50step = 100
Rate
PS
NR
Compressive Sensing
36
Both Sparsity and Measurement Errors
A real example: real image and quantized measurements.
Recovery Strategies:• Ψ = DCT• Ψ = block DCT• Ψ = DWT• TV- minimization• Ψ = SVD - of each image!
Compressive Sensing
37
Both Sparsity and Measurement Errors
0 1 2 3 4 5 6 7 8 Jpeg2000 DCT−l
1−N B_DCT−l
1−N DWT−l
1−N TV−N SVD−l
1−N
0 1 2 3 4 5 6 7 80
10
20
30
40
50
60
70
80
90
100
Rate
PS
NR
(b) Phantom
0 1 2 3 4 5 6 7 80
10
20
30
40
50
60
70
80
90
100
Rate
PS
NR
(c) Lena
0 1 2 3 4 5 6 7 80
10
20
30
40
50
60
70
80
90
100
Rate
PS
NR
(d) Camera man
0 1 2 3 4 5 6 7 80
10
20
30
40
50
60
70
80
90
100
Rate
PS
NR
(e) Text
Compressive Sensing
38
Sparsity × Quantization Errors
‖x − x‖l2 .
εq +1√S‖xS − x‖l1︸ ︷︷ ︸
εs
,
The reconstruction error in CS is of the order of the maximum of thequantization (εq) and sparsity errors (εs).
Compressive Sensing
39
(a) DCT (b) DWT
(c) TV (d) SVD
Figure: Rate × PSNR of image Lena
Compressive Sensing
40
(a) DCT (b) DWT
(c) TV (d) SVD
Figure: Measurements × PSNR of image Lena
Compressive Sensing
41
Lets wrap it up!
CS ApplicationsImage acquisition and compression• Interesting, but• Limitations ...
TomorowApplications in graphics and vision• One Pixel Camera• Dual Photography