Signal Processing Course : Approximation

Approximation and Coding

with Orthogonal Decompositions

Gabriel Peyréhttp://www.ceremade.dauphine.fr/~peyre/numerical-tour/

http://www.ceremade.dauphine.fr/~peyre/wavelet-tour/

http://www.ceremade.dauphine.fr/~peyre/wavelet-tour/

Overview

• Approximation and Compression

• Decay of Approximation Error

• Fourier for Smooth Functions

• Wavelet for Piecewise Smooth Functions

• Curvelets and Finite Elements for Cartoons

Sparse Approximation in a Basis

Hard Thresholding

(usually polynomial)

Approximation Speed

Approximation error decay:

(usually polynomial)

Approximation Speed

Approximation error decay:

log 1

0(||

f�

f M||)

Log/Log plot: approx. a�ne curve

log10(M/N)

Efficiency of Transforms

Fourier DCT

Local DCT Wavelets

log 1

0(||

f�

f M||)

log10(M/N)

Overview






fforward

Compression by Transform-coding

a[m] = ⇥f, �m⇤ � R

Image f Zoom on f

transform

fforward


a[m] = ⇥f, �m⇤ � R

Quantization: q[m] = sign(a[m])�

|a[m]|T

⇥� Z

Image f Zoom on f

transform

a[m]

�T T 2T�2T a[m]

Quantized q[m]

bin Tq[m] � Z

fforward coding


a[m] = ⇥f, �m⇤ � R


|a[m]|T

⇥� Z

Image f Zoom on f

transform

Entropic coding: use statistical redundancy (many 0’s).a[m]

�T T 2T�2T a[m]

Quantized q[m]

bin Tq[m] � Z

fforward coding


a[m] = ⇥f, �m⇤ � R


|a[m]|T

⇥� Z

Image f Zoom on f

decoding

q[m] � Za[m] dequantization

transform


�T T 2T�2T a[m]

Quantized q[m]

bin Tq[m] � Z

Dequantization:

fforward coding


a[m] = ⇥f, �m⇤ � R


|a[m]|T

⇥� Z

Image f Zoom on f fR, R =0.2 bit/pixel

decoding

q[m] � Za[m] dequantizationtransform

backwardfR =

�

m�IT

a[m]�m

transform


�T T 2T�2T a[m]

Quantized q[m]

bin Tq[m] � Z

Dequantization:

Thresholding vs. Quantizing

Non-linear Approximation and Compression


a[m]

�T T 2T�2T a[m]


|a[m]|T

⇥� Z

=⇥ |a[m]� a[m]| � T

2Dequantization: a[m] = sign(q[m])

�|q[m]| +

12

⇥T


||f � fM ||2 # = M

Theorem: ||f � fR||2 � ||f � fM ||2 + MT 2/4

where M = # {m \ a[m] �= 0}.

||f � fR||2 =⇤

m

(a[m]� a[m])2 �⇤

|a[m]|<T

|a[m]|2 +⇤

|a[m]|�T

�T

2

⇥2

a[m]

�T T 2T�2T a[m]


|a[m]|T

⇥� Z

=⇥ |a[m]� a[m]| � T

2Dequantization: a[m] = sign(q[m])

�|q[m]| +

12

⇥T

A Naive Support Coding ApproachCoding: relate # bits R to # coe�cients M .

(H1) Ordered coe�cients |�f, �m⇥| decays like m��+12 .

=⇤ ||f � fM ||2 ⇥M��.

A Naive Support Coding ApproachCoding: relate # bits R to # coe�cients M .

(H1) Ordered coe�cients |�f, �m⇥| decays like m��+12 .

f � RN sampled from f0, error:

(H2) To ensure ||f � f0||2 ⇥ ||f � fM ||2: M ⇥ N⇥/�.

||f � f0||2 ⇥ N��

=⇤ ||f � fM ||2 ⇥M��.

A Naive Support Coding Approach

Simple coding strategy: R = Rval + Rind

=� Rind � log2

�N

M

⇥= O(M log2 (N/M)) = O(M log2(M)).

Coding: relate # bits R to # coe�cients M .(H1) Ordered coe�cients |�f, �m⇥| decays like m��+1

2 .



||f � f0||2 ⇥ N��

=⇤ ||f � fM ||2 ⇥M��.



=� Rind � log2

�N

M

⇥= O(M log2 (N/M)) = O(M log2(M)).

=� Rval = O(M | log2(T )|) = O(M log2(M))


2 .



||f � f0||2 ⇥ N��

=⇤ ||f � fM ||2 ⇥M��.



=� Rind � log2

�N

M

⇥= O(M log2 (N/M)) = O(M log2(M)).

Theorem: Under hypotheses (H1) and (H2), ||f � fR||2 = O(R�� log�(R)).

=� Rval = O(M | log2(T )|) = O(M log2(M))


2 .



||f � f0||2 ⇥ N��

=⇤ ||f � fM ||2 ⇥M��.

Entropic Coders

JPEG-2000 Overview

Contextual Coding

code block width

3 ! 3 context window

JPEG-2000 vs. JPEG, 0.2bit/pixel

Overview






1D Fourier Approximation

Sobolev and Fourier

Singularities and Fourier

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Sobolev for Images

Overview






Vanishing moments:

p = 3

p = 4

Magnitude of Wavelet Coefficients

f(x)

−1 0 1 2−2

−1

0

1

2

−2 −1 0 1 2

−1

0

1

2

−2 0 2 4−1

−0.5

0

0.5

1

1.5

� k < p,

��(x)xkdx = 0

p = 2

Vanishing moments:

p = 3

p = 4


f(x)

−1 0 1 2−2

−1

0

1

2

−2 −1 0 1 2

−1

0

1

2

−2 0 2 4−1

−0.5

0

0.5

1

1.5

� k < p,

��(x)xkdx = 0

p = 2

|�f, �j,n⇥| � Cf ||�||12j(�+d/2)

t =x� 2jn

2j

⇥f, �j,n⇤ =1

2j d2

⇤f(x)�

�x � 2jn

2j

⇥dx = 2j d

2

⇤R(2jt)�(t)dt

If f is C� on supp(⇥j,n), p � �:

f(x) = P (x� 2jn) + R(x� 2jn) = P (2jt) + R(2jt)

Vanishing moments:

p = 3

p = 4


f(x)

−1 0 1 2−2

−1

0

1

2

−2 −1 0 1 2

−1

0

1

2

−2 0 2 4−1

−0.5

0

0.5

1

1.5

� k < p,

��(x)xkdx = 0

|�f, �j,n⇥| � ||f ||�||�||12j d2

p = 2

|�f, �j,n⇥| � Cf ||�||12j(�+d/2)

t =x� 2jn

2j

⇥f, �j,n⇤ =1

2j d2

⇤f(x)�

�x � 2jn

2j

⇥dx = 2j d

2

⇤R(2jt)�(t)dt

If f is C� on supp(⇥j,n), p � �:

f(x) = P (x� 2jn) + R(x� 2jn) = P (2jt) + R(2jt)

1D Wavelet Coefficient Behavior

−0.2

−0.1

0

0.1

0.2

−0.2

−0.1

0

0.1

0.2

−0.5

0

0.5

−0.5

0

0.5

0

0.2

0.4

0.6

0.8

1


If f is C� in supp(�j,n), then

|�f, �j,n⇥| � 2j(�+1/2)||f ||C� ||�||1

−0.2

−0.1

0

0.1

0.2

−0.2

−0.1

0

0.1

0.2

−0.5

0

0.5

−0.5

0

0.5

0

0.2

0.4

0.6

0.8

1



|�f, �j,n⇥| � 2j(�+1/2)||f ||C� ||�||1

|�f, �j,n⇥| � 2j/2||f ||�||�||1

If f is bounded (e.g. arounda singularity), then

−0.2

−0.1

0

0.1

0.2

−0.2

−0.1

0

0.1

0.2

−0.5

0

0.5

−0.5

0

0.5

0

0.2

0.4

0.6

0.8

1

For Fourier, linear�non-linear, sub-optimal.

For wavelets, linear�non-linear, optimal.

Piecewise Regular Functions in 1D

Theorem: If f is C� outside a finite set of discontinuities:

�n[M ] = ||f � fnM ||2 =

�O(M�1) (Fourier),O(M�2�) (wavelets).

Examples of 1D Approximations

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

−2 −1.8 −1.6 −1.4 −1.2 −1 −0.8−6

−5.5

−5

−4.5

−4

−3.5

−3

−2.5

−2

−1.5

−1

Large coe�cient

|�f, �jn⇥| < Tx1 x2

S = {x1, x2}

Localizing the Singular Support

f(x)

j1

j2

Small coe�cient

Singular support:

|Cj | � K|S| = constant

Large coe�cient

|�f, �jn⇥| < Tx1 x2

S = {x1, x2}


f(x)

j1

j2

Small coe�cient

Singular support:

Coe�cient behavior:Regular, n � Cc

j : |⇥f, �j,n⇤| � C2j(�+1/2)

Singular, n � Cj : |⇥f, �j,n⇤| � C2j/2


Large coe�cient

|�f, �jn⇥| < Tx1 x2

S = {x1, x2}


f(x)

j1

j2

Small coe�cient

Singular support:


j : |⇥f, �j,n⇤| � C2j(�+1/2)


Cut-o� scales (depends on T ):Singular: 2j1 = (T/C)

1�+1/2

Regular: 2j2 = (T/C)2


Large coe�cient

|�f, �jn⇥| < T

Hand-made approximate: fM =�

j�j2

�

n�Cj

�f, �j,n⇥�j,n +�

j�j1

�

n�Ccj

�f, �j,n⇥�j,n

x1 x2

S = {x1, x2}


f(x)

j1

j2

Small coe�cient

Singular support:


j : |⇥f, �j,n⇤| � C2j(�+1/2)



1�+1/2

Regular: 2j2 = (T/C)2


||f � fM ||2 � ||f � fM ||2 ��

j<j2,n�Cj

|⇥f, �j,n⇤|2 +�

j<j1,n�Ccj

|⇥f, �j,n⇤|2

Computing Error and #Coefficients

f(x)

j1

j2n � Cc

j : |⇥f, �j,n⇤| � C2j(�+1/2)

n � Cj : |⇥f, �j,n⇤| � C2j/2

2j1 = (T/C)1

�+1/2

2j2 = (T/C)2

��

j<j2

(K|S|)� C22j +�

j<j1

2�j � C22j(2�+1)

= O(2j2 + 22�j1) = O(T 2 + T2�

�+1/2 ) = O(T2�

�+1/2 )


Singulatities

�

regular part

||f � fM ||2 � ||f � fM ||2 ��

j<j2,n�Cj

|⇥f, �j,n⇤|2 +�

j<j1,n�Ccj

|⇥f, �j,n⇤|2


f(x)

j1

j2n � Cc

j : |⇥f, �j,n⇤| � C2j(�+1/2)

n � Cj : |⇥f, �j,n⇤| � C2j/2

2j1 = (T/C)1

�+1/2

2j2 = (T/C)2

��

j<j2

(K|S|)� C22j +�

j<j1

2�j � C22j(2�+1)

M ��

j�j2

|Cj | +�

j�j1

|Ccj | �

�

j�j2

K|S| +�

j�j1

2�j

= O(| log(T )| + T�1

�+1/2 ) = O(T�1

�+1/2 )

= O(2j2 + 22�j1) = O(T 2 + T2�

�+1/2 ) = O(T2�

�+1/2 )


Singulatities

�

regular part

||f � fM ||2 � ||f � fM ||2 ��

j<j2,n�Cj

|⇥f, �j,n⇤|2 +�

j<j1,n�Ccj

|⇥f, �j,n⇤|2


f(x)

j1

j2n � Cc

j : |⇥f, �j,n⇤| � C2j(�+1/2)

n � Cj : |⇥f, �j,n⇤| � C2j/2

2j1 = (T/C)1

�+1/2

2j2 = (T/C)2

��

j<j2

(K|S|)� C22j +�

j<j1

2�j � C22j(2�+1)

M ��

j�j2

|Cj | +�

j�j1

|Ccj | �

�

j�j2

K|S| +�

j�j1

2�j

= O(| log(T )| + T�1

�+1/2 ) = O(T�1

�+1/2 ) ||f � fM ||2 = O(M�2�)

= O(2j2 + 22�j1) = O(T 2 + T2�

�+1/2 ) = O(T2�

�+1/2 )


Singulatities

�

regular part

2D Wavelet Approximation


|�f, �j,n⇥| � 2j(�+1)||f ||C� ||�||1

|�f, �j,n⇥| � 2j ||f ||�||�||1

If f is bounded (e.g. around a singularity), then

Fourier � Wavelet, both sub-optimal.

Wavelets: same result for BV functions (optimal).

Piecewise Regular Functions in 2D

�n[M ] = ||f � fnM ||2 =

�O(M�1/2) (Fourier),O(M�1) (wavelets).

Theorem: If f is C� outside a set of finite length edge curves,

Example of 2D Approximations

Length(S) = L

Localizing the Singular Support in 2D

j1

j2

f(x, y)

|Cj | � LK2�j �= constant

Length(S) = L


j1

j2

f(x, y)


j : |⇥f, �j,n⇤| � C2j(�+1)

Singular, n � Cj : |⇥f, �j,n⇤| � C2j


Length(S) = L


j1

j2

f(x, y)


j : |⇥f, �j,n⇤| � C2j(�+1)



1�+1

Regular: 2j2 = T/C


Hand-made approximate: fM =�

j�j2

�

n�Cj

�f, �j,n⇥�j,n +�

j�j1

�

n�Ccj

�f, �j,n⇥�j,n

Length(S) = L


j1

j2

f(x, y)


j : |⇥f, �j,n⇤| � C2j(�+1)



1�+1

Regular: 2j2 = T/C


||f � fM ||2 � ||f � fM ||2 ��

j<j2,n�Cj

|⇥f, �j,n⇤|2 +�

j<j1,n�Ccj

|⇥f, �j,n⇤|2

Singulatities�

regular part


j1

j2

f(x, y)

n � Ccj : |⇥f, �j,n⇤| � C2j(�+1)

n � Cj : |⇥f, �j,n⇤| � C2j

2j2 = T/C

2j1 = (T/C)1

�+1

��

j<j2

LK2�j � C222j +�

j<j1

2�2j � C222j(�+1)


= O(2j2 + 22�j1) = O(T + T2�

�+1 ) = O(T )

||f � fM ||2 � ||f � fM ||2 ��

j<j2,n�Cj

|⇥f, �j,n⇤|2 +�

j<j1,n�Ccj

|⇥f, �j,n⇤|2

Singulatities�

regular part


j1

j2

f(x, y)

n � Ccj : |⇥f, �j,n⇤| � C2j(�+1)

n � Cj : |⇥f, �j,n⇤| � C2j

2j2 = T/C

2j1 = (T/C)1

�+1

��

j<j2

LK2�j � C222j +�

j<j1

2�2j � C222j(�+1)


= O(2j2 + 22�j1) = O(T + T2�

�+1 ) = O(T )

M ��

j�j2

|Cj | +�

j�j1

|Ccj | �

�

j�j2

LK2�j +�

j�j1

2�2j

= O(T�1 + T�1

�+1 ) = O(T�1)

||f � fM ||2 = O(M�1)

||f � fM ||2 � ||f � fM ||2 ��

j<j2,n�Cj

|⇥f, �j,n⇤|2 +�

j<j1,n�Ccj

|⇥f, �j,n⇤|2

Singulatities�

regular part


j1

j2

f(x, y)

n � Ccj : |⇥f, �j,n⇤| � C2j(�+1)

n � Cj : |⇥f, �j,n⇤| � C2j

2j2 = T/C

2j1 = (T/C)1

�+1

��

j<j2

LK2�j � C222j +�

j<j1

2�2j � C222j(�+1)


= O(2j2 + 22�j1) = O(T + T2�

�+1 ) = O(T )

M ��

j�j2

|Cj | +�

j�j1

|Ccj | �

�

j�j2

LK2�j +�

j�j1

2�2j

= O(T�1 + T�1

�+1 ) = O(T�1)

Overview






Geometric image model: f is C� outside a set of C� edge curves.

BV image: level sets have finite lengths.Geometric image: level sets are regular.

Geometry = cartoon image Sharp edges Smoothed edges

Geometrically Regular Images

�|�f |

Approximation of f , C2 outside C2 edges.

Piecewise linear approximation on M triangles: fM .

Geometic Construction : Finite Elements




Regular areas:�M/2 equilateral triangles.

M�1/2

M�1/2





M�1/2

M�1/2

�M/2 anisotropic triangles.Singular areas:



Di�culties to build e�cient approximations.No optimal strategies (greedy solutions).

Theorem: If f is C2 outside a set of C2 contours, then one has for an adaptedtriangulation ||f � fM ||2 = O(M�2).



M�1/2

M�1/2

�M/2 anisotropic triangles.Singular areas:

Greedy Triangulation OptimizationBougleux, Peyre, Cohen, ECCV’08

Curvelet Atoms

Parabolic dyadic scaling:

Rotation:

[Candes, Donoho] [Candes, Demanet, Ying, Donoho]

“width � length2”

Curvelet Tight Frame

Spacial sampling:

Tight frame of L2(R2):

Angular sampling:

Discrete curvelets: O(N log(N)) algorithm.

Redundancy � 5 =⇥ not e�cient for compression.

M -term curvelet approximation:

Curvelet Approximation

Theorem: If f is C2 outside a set of C2 edges, ||f � fM ||2 = O(M�2(log M)3).

Works on elongated edges.

Works also on locally parallel textures !

Curvelets Denoising

Conclusion

Conclusion

0

0.2

0.4

0.6

0.8

1

Signal Processing Course : Approximation

Documents

Transcript of Signal Processing Course : Approximation