Brice Lecture - 1

Brice Lecture, Rice Sept. 19 2002,

Signal Representations: from Fourier to Waveletsand Beyond

Martin Vetterl iEPFL & UC Berkeley

1. The Problem and its History

2. Mathematical Representation of Signals

3. Information Theory, Signal Processing and Wavelets

4. Wavelets and Approximation Theory

5. Approximation and Applications in Denoising andCompression

6. Going to Two Dimensions: Nonseparable Bases

7. Conclusions

Brice Lecture - 2

Acknowledgements

Swiss National Science Foundation

Collaborators• T. Blu (EPFL)• M. Do (UIUC)• P.L. Dragott i ( Imper ial Col lege)• P. Marzi l l iano (Genimedia)• P. Roud (EPFL)

Brice Lecture - 3

1. The Problem and its History

Henry the 8th looks for a new spouse

Image communication is an old problem.. .

source

representation

transmission

reception

evaluationAnne de Clève, Holbein, 1539

Brice Lecture - 4

<=> {0,1}

How many bits for Mona Lisa?

Brice Lecture - 5

2. Mathematical Representation of Signals

Joseph Fourier (1768-1830)

Studies the heat equation ( in Egypt. . . )

Brice Lecture - 6

1807: Fourier upsets the French Academy.. . .

Fourier Series:• Harmonic ser ies• Frequency changes, f0, 2f0, 3f0, . . .

++=

t

f(t)

Brice Lecture - 7

and: successive appr oximat ion

“What is the ma gic t r ic k?’ ’

or , wi th Euc l id:

or thogonal i ty coor dinate system

Brice Lecture - 8

But

1898: Gibbs’ paper 1899: Gibbs’ correct ion

and i t wi l l take almost another 60 years to set t le the con-vergence quest ion (Car leson 1964).

Brice Lecture - 9

1910: Alfred Haar discovers the Haar waveletdual to the Fourier construction

Haar ser ies:• Scale change, scales S0, 2S0, 4S0, 8S0• Time shi f t

+++=

temps

f(t)

Brice Lecture - 10

The Haar system

Again a set of or thonormal vectors!

Size: length propor t ional to 2m

‘ ’ f requency’’ : f0, 2f0, 4f0, 8f0, . . . octaves!

t

t

t

m = 0

m = 1

m = -1

Brice Lecture - 11

1945: Gabor localizes the Fourier transform ⇒ STFT

1980: Morlet proposes the continuous wavelettransform

shor t- t ime Four ier t ransform wavelet t ransform

time

frequency

t ime

frequency

Brice Lecture - 12

Analogy with the musical scoreBach knew about wavelets!

Brice Lecture - 13

1983: Lena discovers pyramids(actually, Burt and Adelson)

+-

x x

MR encoder MR decoder

D II

+

coarse

residual

Brice Lecture - 14

1984: Lena gets crit ical(subband coding)

f1

f2π

−π

−π

πLL LH HL HH

H1 2

H0 2

H1 2

H0 2

HL 2

HΗ 2

H1

H0

HH

HL

LH

LL

x

horizontal

vertical

Brice Lecture - 15

1986: Lena gets formal. . .(mult iresolution theory by Mallat, Meyer. . . )

= +

Brice Lecture - 16

1988: Ingr id disco ver s Daubec hies’ wa velets!

• New fami l ies of or thonormal bases,(general iz ing Haar)

• Bior thogonal fami l ies, f rames

• many new appl icat ions

0.5 1.0 1.5 2.0 2.5 3.0

Time

-1

-0.5

0

0.5

1

1.5

Amplitude

Brice Lecture - 17

3. Information Theory, Signal Processingand Wavelets

Claude Shannon: The founding genius

1. Source coding

2. Channel coding

3. Separat ion of source and channel coding

Brice Lecture - 18

Source Coding

exchanging descr ipt ion complexi ty for qual i ty

Again, successive approximat ion is key

distorsion

D(R)

N 0 σ2,( ) D R( ) σ2 2 2R–⋅=

complexity

Brice Lecture - 19

256:1

128:1

32:1

Brice Lecture - 20

Signal Processing

Subband coding

2

2

2

2x

analysis synthesis

H1

H0

G1

G0

y1

y0 x0

x1

+ x

beam of

white light

Brice Lecture - 21

I terated f i l ter banks

H1 2

H0 2 H1 2

H0 2 H1 2

H0 2

x

stage J

stage 2

stage 1

analysis

f

f

f

t

t

t

Hi

ππ2---π

4---π

8---π

16------

ω…

Frequency div is ion

Brice Lecture - 22

Separable application in 2D

An image and its wavelet decomposit ion

Important:• audi tory system works in octaves• v isual system works in f requency bands

Brice Lecture - 23

The iterated f i l ter bank leads to wavelets

The Daubechies iterative wavelet construction

Scaling function and Wavelet

0 1 2

Time

0

0.2

0.4

0.6

0.8

Amplitude

0 1 2

Time

-0.2

0

0.2

0.4

0.6

0.8

Amplitude

0 1 2

Time

-0.2

0

0.2

0.4

0.6

0.8

Amplitude

0 1 2

Time

-0.2

0

0.2

0.4

0.6

0.8

Amplitude

i = 1

i = 4i = 3

i = 2

Brice Lecture - 24

Fini te length, cont inuous and , based on L=4i terated f i l ter

0.5 1.0 1.5 2.0 2.5 3.0

Time

-0.25

0

0.25

0.5

0.75

1

1.25

Amplitude

8.0 16.0 24.0 32.0 40.0 48.0

Frequency [radians]

0

0.2

0.4

0.6

0.8

1

Magnitude response

0.5 1.0 1.5 2.0 2.5 3.0

Time

-1

-0.5

0

0.5

1

1.5

Amplitude

8.0 16.0 24.0 32.0 40.0 48.0

Frequency [radians]

0

0.2

0.4

0.6

0.8

1

Magnitude response

scalingfunction

wavelet

lowpass

bandpass

ϕ t( ) ψ t( )

Brice Lecture - 25

I terated f i l ter banks lead to two-scale equations

Hat funct ion

Daubechies’ scal ing funct ion

Relat ion to sel f -s imi lar i ty useful for analysis andcharater izat ion of f ractal processes

ϕ t( ) cnϕ 2t n–( )n∑=

=

0.5 1.0 1.5 2.0 2.5 3.0

Time

0

0.5

1

1.5

Amplitude

0.5 1.0 1.5 2.0 2.5 3.0

Time

-0.25

0

0.25

0.5

0.75

1

1.25

Amplitude =

Brice Lecture - 26

4. Wavelets and Approximation Theory

Consider piecewise smooth s ignals• Wavelet act as s ingular i ty detectors• Scal ing funct ions catch smooth par ts• “Noise” is c i rcular ly symetr ic

Wavelets

Noise

time

f(t) Scaling functions

Brice Lecture - 27

How does this work? Proper choice of f i l ters!

I terated f i l ter bank ( )

• polynomials are ‘‘eaten’’ in the highpass• polynomials are reproduced by the lowpass channel• d iscont inui t ies are detected by the wavelets

~wavelet

~scal ingfunct ion

H1 2

H0 2 H1 2

H0 2 H1 2

H0 2

x

stage J

stage 2

stage 1

analysis

Hj z( ) Gj z 1–( )=

Brice Lecture - 28

Example: S4 reproduces linear fcts

0 5 10 15 20 25 30−1

−0.5

0

0.5

1

1.5

2

2.5

3

3.5

Brice Lecture - 29

How about singularit ies?I f we have a s ingular i ty of order n at the or ig in

(-1 Dirac, 0: Heavis ide, . . . ) , the CWT transform behavesas

In the or thogonal wavelet ser ies: same behavior, but onlyL=2N-1 coeff ic ients inf luenced at each scale!

• e.g. Dirac/Heavis ide: behavior as and

X a 0,( ) cn an 2⁄⋅=

Time

Scale

Time

1 2 3 4

Time

0

1

2

3

4

5

6

Amplitude

2 m– 2⁄ 2m 2⁄

Brice Lecture - 30

Example:

• phase changes randomize signs, but not decay• a s ingular i ty inf luences only L wavelets at each scale

(L=2N-1=3)

1024

512

256

128

64

32

16

16

f(t)

t

L=3

Brice Lecture - 31

Approximation: l inear versus non-l inear

Given an or thonormal basis {g n} for a space S and a s ignal

,

• the best l inear approximat ion is given by the project ion onto af ixed subspace of s ize M ( independent of f ! )

• the best nonl inear approximat ion is given by the project ion ontoan adapted subspace of s ize M (dependent on f ! )

or : take the f i rst M coeffs ( l inear) or take the largest Mcoeffs (non- l inear)

f f gn,⟨ ⟩ gn⋅n∑=

fM f gn,⟨ ⟩M 1=

M

∑ gn⋅=

fM˜ f gn,⟨ ⟩

n IM∈∑ gn⋅=

IM: set of largestM coeffs

Brice Lecture - 32

Nonlinear approximation

Nonl inear approximat ion power depends on basisExample:

Two di f ferent bases for [0,1] :

• Four ier ser ies• Wavelet ser ies: Haar wavelets

Linear approximat ion in Four ier or wavelet bases

Nonl inear approximat ion in a Four ier basis

Nonl inear approximat ion in a wavelet basis

11/sqrt2t

f ( t )

cst

ej2πkt{ }k ℑ∈

εM 1 M⁄∼

εM 1 M⁄∼

εM 1 2M⁄∼

Brice Lecture - 33

Fourier Basis: N=1024, M= 64, l inear versus nonlinear

• nonl inear approximat ion is not necessar i ly much better!

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

D= 2.7

D= 2.4

Brice Lecture - 34

Wavelet basis: N=1024, M= 64, J=6, l inear versus non-l inear

• nonl inear approximat ion is vast ly super ior !

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

D=3.5

D=0.01

Brice Lecture - 35

5. Approximation and Applications in Denoisingand Compression

Wavelets approximate piecewise smooth s ignals wi th fewnon-zero coeff ic ients

This is good for• Compression• Denois ing• Classi f icat ion• Inverse problems

Thus: sparsi ty is good!

Brice Lecture - 36

DenoisingIdea:

• Dominant features are caught by large wavelet coeff ic ients• Noise is spread uni formly over al l coeff ic ients• Thresholding smal l coeff ic ients to 0 keeps the signal but removes

the noise

Schematical ly:

Now:

Note:• very s imple• works wel l for p iecewise smooth s ignals• for jo int ly gaussian, standard l inear methods (Wiener f i l ter) are

f ine

WT IWTx n[ ] w n[ ]+ x n[ ] w n[ ]+

x n[ ] x n[ ]∼w n[ ] w n[ ]«

Brice Lecture - 37

Example: 1D Signal

WT

IWT

Threshold

Brice Lecture - 38

Example: 2D signal

WT

Threshold

IWT

Brice Lecture - 39

Compression

Idea• sparse representat ion should be good for compression• t ransform, keep large coeff ic ients through quant izat ion• reconstruct ion gives good qual i ty

Note• s imple• at the hear t of JPEG 2000• for jo int ly Gaussian, standard l inear approach (KLT) is opt imal

WT Ex n[ ] 0110001Q

quantization entropycoding

IWTE-10110001 Q-1 x n[ ]

Brice Lecture - 40

Example: 1D

WT

IWT

Quantization

50 100 150 200 250−3

−2

−1

0

1

2

3

4

5

6

Brice Lecture - 41

Example: 2D

WT

IWT

Quantization

Brice Lecture - 42

Old Versus New JPEG: D(R) on log scale

Notes• improvement by a few dB’s• lot more funct ional i t ies (e.g. progressive download on internet)• low rate behavior• is th is the l imi t?

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2−50

−45

−40

−35

−30

−25

−20

−15

-6 dB/bit

old JPEG (DCT)

new JPEG(wavelets)

Brice Lecture - 43

From the compar ison, JPEG fai ls above 40:1 compressionwhi le JPEG2000 survives

Images cour tesy of www.dspworx.com

Original Lena Image (256 x 256 Pixels,24-Bit RGB)

JPEG Compressed (Compression Ratio43:1)

JPEG2000 Compressed (CompressionRatio 43:1)

Brice Lecture - 44

So, are wa velets c losing the“Ho w man y bi ts f or Mona Lisa” quest ion?

(un) for tunately: No!

Reason:

Shannon tel ls us

but wavelets give

for cer tain c lasses of s imple s ignals

D R( ) α12β1R–∼

DW R( ) α2 R2β2 R–∼

Brice Lecture - 45

Reason: independent coding of dependent information

Al l these wavelets coeff ic ients correspond to a s ingle de-gree of f reedom!

1024

512

256

128

64

32

16

16

f(t)

t

L=3

Brice Lecture - 46

Solution: model dependencies betweenwavelets coeff icients

Var ious proposals• Markov models (Baraniuk)• Zero t rees• Footpr ints

Brice Lecture - 47

Example: An optimal algorithm

This uses dynamic programming [Prandoni :00]

0 50 100 150 200 2500

50

100

150

200

250

0 50 100 150 200 2500

50

100

150

200

250

0 50 100 150 200 2500

50

100

150

200

250

351 bits297 bits146 bits

Brice Lecture - 48

Wavelet Footprints [Dragotti:01]

Can we ‘ ’ f ix ’’ the wavelet scenar io?

That is, achieve the same rate-distor t ion performance asan oracle or a dynamic programming methodbut wi th the s impl ic i ty of wavelet methods?

The structure of wavelet representat ion ofs ingular i t ies is s imple:

• locat ion: random• structure accross scales: determinist ic!

Data structure to capture discont inui t iesin wavelet domain

• in or thogonal expansion• in f rame

This leads to a s imple and intui t ive data structureWavelet Footprint

Brice Lecture - 49

The wavelet footprint

• th is is the s ignature of the discont inui ty• behaviour wel l understood (c lassic wavelet analysis)

Brice Lecture - 50

Denoising

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

Original signal Noisy Signal (SNR=15.62dB)

Hard-Thresholding (SNR=21.3dB) Cycle-Spinning (SNR=25.4dB)

Denoising with Footprints (SNR=27.2dB)

Brice Lecture - 51

Compression

Brice Lecture - 52

6. Going to Two Dimensions: Nonseparable Bases

Objects in two dimensions we are interested in

• textures: per pixel• smooth surfaces: per object !

smooth surface

smooth boundary

texture

D R( ) C0 2 2R–⋅=D R( ) C1 2 2R–⋅=

Brice Lecture - 53

Current approaches to two dimensions.. . .

Mostly separable, direct products

Wavelets: good for point singularit iesbut what is needed are sparse coding of edge singularit ies!

DWT

Brice Lecture - 54

Two dimensonal wavelet bases

Ex: Tensor products of Haar funct ions

That is

or in 3D

That is very l i t t le direct ional i ty!

yx

� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �

� � � � � � �� � � � � � �� � � � � � �� � � � � � �

� � � � � �� � � � � �� � � � � �� � � � � �� � �� � �� � �� � �� � �� � �� � �� � �

� � �� � �� � �� � �� � �� � �

etc;

Brice Lecture - 55

What is needed are direct ional bases• Local Radon transform• Ridgelets• Curvelets• Contour lets• etc

That is:

a zoo of t rue two dimensional animals

Brice Lecture - 56

Example: a directional block transform [Do:01]

Brice Lecture - 57

Multiresolution Contour Approximation

Consider object c2 boundary between two cst• # of wavelet coeffs: 2 j

• # of curvelet coeffs: 2 j /2

Rate fo approximat ion, M-term NLA• Four ier :• Wavelets:• Curvlets:

XletWavelet

O 1 M⁄( )O 1 M⁄( )

O 1 M2⁄( )

Brice Lecture - 58

Operational Solution

Direct ional Analysis (as in Radon transform)+

Mult i resolut ion as in wavelets

Directional Fi l ter Banks• d iv is ion of 2-D spectrum into f ine s l ices using i terated tree

structured f i l ter banks

6

3210

5

6

7

3 2

5

74

1

4

w1

(-pi,-pi)

(pi,pi)w2

0

Brice Lecture - 59

Pyramidal Directional Fi l ter Banks (PDFB)

Motivat ion: + add mult iscale into the direct ional f i l ter bank+ improve i ts non- l inear approximat ion power

Proper t ies: + Flexible mult iscale and direct ional represen-tat ion for images (can have di f ferent number of d i rect ionat each scale!)

(2,2)

multiscale dec. directional dec.

(pi,pi)

(-pi,-pi)

w2

w1

Brice Lecture - 60

Example: A p yramidal directional filter bank

Compression, denois ing, inverse problems: most ly open!

Brice Lecture - 61

7. Conclusions

Mult i resolut ion is good for you!• Percept ion and mathematics (most ly) agree.. .

Non- l inear can buy a lot . . .• in approximat ion, the di f ference can be huge!

Compression is hard but gener ic• understanding complexi ty is fundamental

Mult ip le dimension is ( inf in i te ly) harder than one.. .

The search for the ul t imate basis is a fascinat ing andt imeless topic

Brice Lecture - 62

References

For a tutor ia l :

M. Vetter l i , Wawelets, Approximat ion and Compression,Signal Processing, May 2001

For more detai ls

P. L. Dragott i , M. Vetter l i , Wavelet Footpr ints, IEEE Trans-act ions on Signal Processing, submit ted

M. Do, M. Vetter l i , Pyramidal Direct ional Fi l ter Banks,to be submit ted, 2002

Brice Lecture - 63

Appendix:

1930: Heisenberg discovers thatyou cannot have your cake and eat i t too!

Uncer tainty pr inciple• lower bound on TF product

Time-frequency t i l ing for a

t ω

ω

ω

t

t

perfect ly localin t ime

perfect ly localin f requency

trade-of f

f t( )

f t( )

f t( )

F ω( )

F ω( )

F ω( )

ω

t

Brice Lecture - 64

sine + Delta

t

f

t

f

t

f

t

f

t

t0 T

t0 T t0 T

t0 T t0 T

f

FT

STFT WT

f 0

f 0f 0

f 0

so.. . .

what is a good basis?

x n[ ] 2πf0cos Aδ n t0–[ ]+=

Brice Lecture - 65