Variational Image Restoration Leah Bar PhD. thesis supervised by: Prof. Nahum Kiryati and Dr. Nir...

Variational Image Variational Image RestorationRestoration

Leah BarLeah BarPhD. thesis supervised by:

Prof. Nahum Kiryati and Dr. Nir Sochen*

School of Electrical Engineering

*Department of Applied Mathematics

Tel-Aviv University, ISRAEL

2

Inverse problem which has been investigated for more than 40 years.

Given the image g and the blur kernel h, restore the original image f .

What is image Restoration?

nfhg *• Camera out of focus

• Motion blur

• Atmospheric turbulence

• Sensor noise

• Quantization

Image is degraded by deterministic (blur) and random (noise) processes.

Blur is assumed as linear shift invariant process with additive noise.

3

Image Restoration - Applications

Microscopy

4

Image Restoration - Applications

Astronomy

5

Image Restoration - ApplicationsMedical Imaging

6

ffff NFHG nfhg *

Frequency domain:

Spatial domain:

Assuming Gaussian distribution of the noise

22 2/

)( nLn

Aen

Bayesian and Variational Viewpoints

22 2/*

maxarg)|(maxargˆ nLfhg

ffML efgf

Maximum Likelihood

min*)( 2 L

fhgfF

Variational

Noise amplification

In high frequencies

fffff GHHHF *1* )( Ill-Posed Solution

f

f

ff G

constH

HF

2

*

Pseudo inverse Filter

7

Bayesian and Variational Regularization

Maximum a posteriori prob. MAP)(

)()|()|(

g

ffggf

2

2

2

2*

maxarg)|(maxargˆ d

L

n

Lffhg

ffMAP eAegff

min/*)( 22

22 LdnL

ffhgfF Variational (Tikhonov , 1977)

][

][

][][

22

22

*

kG

kkH

kHkF

d

n

Solution – Wiener Filter (over smoothing)

smoothness prior

8

Edge Preservation

Edges are very important features in image processing, and therefore have to be preserved.

Image Deconvolution

Image Denoising

Preserve Edges

observed image - g

recovered image - f

9

Total Variation Regularization

dxffhgfFL

)(2*)(

Rudin, Osher, Fetami 1992

Wiener Total variation

10

Mumford-Shah Segmentation (Mumford and Shah, 1985)

KK

ddxfdxfgKfF \

22)(),(

gradients within segments

total edge lengthdata fidelity

Ω: image domain K: edge set f: recovered image g: observed image

Canny edges M-S edgesOriginal

Image is modeled as piecewise smooth function separated by edges

11

Deconvolution with Mumford-Shah Regularization

KK

ddxfdxfhgKfF \

22)*(),(



M-S functional: difficult to minimize (free-discontinuity problem).

Solution is via the -convergence framework (Ambrosio and Tortorelli 1990)

Strategy: approximate the solution by approximation of the problem

),(minarg),(minarg 0 KfFvfF

),(),( 0 KfFvfF

L. Bar, N. Sochen, N. Kiryati, ECCV 2004

12

Fj(u)=sin (ju)

uj=1.5n/j

Example:

-convergence

A sequence -converges to if:

],[: XFj],[: XF

)(inflim)(: jjjj uFuFuu

)(suplim)(: jjjj uFuFuu

1. liminf inequality

2. existence of recovery sequence

-lim(Fj)=-1

)De Giorgi, 1979(

13

Fundamental theorem of -convergence:

Suppose that

and let a compact set exist such that for all j,

then .

Moreover if uj is a converging sequence such that

then its limit is a minimum point for F.

XK

jX

jX

FF inflimmin

-convergence

jj FF lim

jk

jx

FF infinf

jxj

jjj

FuF inflim)(lim

)(minarg)(minarg fFfF jj

Proof: Let satisfyKu j .infinflim)(inflim jX

jjjj FuF

There exists a subsequence converging to

some u, such that

kj

u

.infinflimlim jX

jjjk FuFkk

,infinfliminflim)(inf)1

jX

jjjkX

FuFuFFkk

)()(supliminfsuplim)2

uFuFF jjjjX

j

*

This is satisfied for every u and in particular

).(infinfsuplim uFFX

jX

j

(*)infinfliminf

infsupliminf

jX

jX

jX

jX

FF

FF

jX

jX

FF infliminf

jX

jX

Finflimmin

15


KK

ddxfdxfhgKfF \

22)*(),(

dx

vvdxfvdxfhgvfF

4

1||)*(),(

22222



v(x): smooth function v(x)~0 at edges v(x)~1 otherwise (in segments)

16

Iterate

Minimize with respect to v by Euler equation (edge detection)

Minimize with respect to f by Euler equation (image restoration)

02),(** 2 fvyxhgfhf

F

022

12 22

v

vfv

v

F


17

Zero padding

Zero paddingZero padding

Convolution Implementation

Neumann boundary conditions

FFT multiplications

18


blurred suggested restoration

suggested edges (v)

19

Semi-blind Deconvolution via Mumford-Shah Regularization

L. Bar, N. Sochen, N. Kiryati, IEEE Trans. Image Processing, 2006

Blind deconvolution: the blur kernel is unknown

Chan and Wong 1998:

dxhdxfdxfhghfF 2*),(

dxhdx

vvdxfvdxfhgvfF

22

2222

4

1||)*(),,(

Suggested: Gaussian kernel parameterized by .

-The restored image is very sensitive to the recovered kernel.- The recovered kernel depends on the contents of the image.

20

Semi-blind Deconvolution via Mumford-Shah Regularization

blurred suggested methodChan-Wong

21

Image Deblurring in the Presence pf Salt-and-Pepper noise

L. Bar, N. Sochen, N. Kiryati, Scale Space, 2005 (best student paper)

Special care should be taken in the case of salt-and-pepper noise

L2 fidelity term in not adequate anymore

Total Variation

22


L. Bar, N. Sochen, N. Kiryati, Scale Space, 2005 (best student paper)

Special care should be taken in the case of salt-and-pepper noise

L2 fidelity term in not adequate anymore

Sequential approach: Deblurring following median-type filtering-poor

1. Median filter 3x3 window 2. TV restoration3. Noise remains!

1. Median filter 5x5 window2. TV restoration3. Nonlinear distortion!

23


Suggested approach: robust L1 fidelity and Mumford-Shah

regularization

dx

vvdxfvdxfhgvfF

4

1||)*(),(

22222



Iterate

Minimize with respect to v by Euler equation (edge detection)

Minimize with respect to f by Euler equation (image restoration)

02),(**

* 2

2

fvyxh

fhg

gfh

f

F

022

12 22

v

vfv

v

F

24

02),(**

*:Calc

22

22:Calc

121

21

11

1221

nn

n

nn

nnn

fvyxhghf

ghff

vfv

Linearization via fixed point scheme:

coefficients in nonlinear terms are lagged by one iteration → linear equation

2,1

1,121

2,1

1,1

*2),(*

*

*

ghf

gfvyxh

ghf

hfln

lnn

ln

ln


)(),( ,1,, lnlnln fGffvH Linear operator

25

Results - pill-box kernel (9x9), radius 4, 10% noise

suggested5x5 median + TV3x3 median + TV

blurred blurred and noisy

26


blurred and noisy recovered

27



28



29

What is the theoretical explanation to the simultaneous deblurring and denoising?

Is Mumford-Shah regularization better than Total Variation?

L. Bar, N. Sochen, N. Kiryati, International Journal of Computer Vision

There is discrimination between image and noise edges.

Image edges are preserved while impulse noise is removed

Theoretical Questions

30

Edge Preservation

|)(|Fidelity Data)( fRfF Relations between:

robust statistics

anisotropic diffusion

line process (half-quadratic)

were shown by

Black and Rangarajan, IJCV, 1996 Black, Sapiro, Marimont and Heeger, IEEE T-IP, 1998

robust statisticsanisotropic diffusionline process

(half quadratic)

Perona & Malik, 1987

Geman & Yang, 1993

Charbonnier et al., 1997

Hampel et al., 1986

31

Edge Preservation

1. Robust smoothness dxffR

)( Gradient Descent:

f

ffdivft '

Influence function-

fft2

f

fft

s ’(s)=(s)

32

Edge Preservation

2. Diffusion Isotropic diffusion (heat equation) fdivft

ffgdivft g is “edge stopping” function 0lim

fgf

Anisotropic diffusion (Perona and Malik, 1987)

From robust smoothness point of view f

ffg

'

2

22 5.01log)(

x

x22 /2

2)(

x

xx

22 2/1

1)(

xxg

Lorentzian

33

Diffusion Illustration

Original Isotropic DiffusionAnisotropic Diffusion

34

3. Line-process (Half-Quadratic) (Geman and Yang, 1993)

Edge Preservation

)(inf||

2bfbfR

MbL

Dual function b represents edges

Penalty function enforces sparse edges

0b1b

across edges

otherwise

)(inf2

bfbfMbL

From robust smoothness point of view

35

Example: Geman-McClure Function

2

2

1)(

s

ss

dxffR

)(

Robust Smoothing

robust -function

Geman McClure

edge stopping function

22 )1(

2)(

ss

Anisotropic Diffusion

ffdivt

f

edge penalty

21)( bb

Line Process (Half-Quadratic)

dxbfbfRb

)(||inf)( 2

otherwise1

edgeson 0

b

b

36

Relation to M-S Terms

The Geman-McClure function in half-quadratic form

22

2

2

11

bfb

f

f

Appears in M-S terms with b = v2

dx

vvdxfvdxfhgvfF

4

)1(*),(

222222

M-S: extended line process = extended Geman-McClure

Edges are forced to be smooth and continuous

image edges are preserved

37

Color Deblurring in the Presence of Impulsive Noise

L. Bar, A. Brook, N. Sochen, N. Kiryati, VLSM’05

),()*(),( 2 vfdxfhgvfF MS

RGBa

aa

222222yxyxyx BBGGRRf

Channels have to be coupled

One edge map for all channels

38

Image Restoration in 3D

blurred

recovered edges

39

Future Work: Space Variant Image

Restoration

preliminary results

40

Conclusions

Novel unified approach to variational segmentation, deblurring and denoising.

Mumford-shah regularization reflects the piecewise-smooth model of natural images.

Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector.

Restoration outcome is superior to state-of-the-art methods

41

Conclusions





42

Conclusions





43

Conclusions





Thank you for your attention

Variational Image Restoration Leah Bar PhD. thesis supervised by: Prof. Nahum Kiryati and Dr. Nir...

Documents

Transcript of Variational Image Restoration Leah Bar PhD. thesis supervised by: Prof. Nahum Kiryati and Dr. Nir...