Introduction to nonlinear image processing Introduction to...

43
1 Introduction to nonlinear image processing IPAM Summer School on Computer Vision July 22, 2013 Introduction to Nonlinear Image Processing Iasonas Kokkinos Center for Visual Computing Ecole Centrale Paris / INRIA Saclay

Transcript of Introduction to nonlinear image processing Introduction to...

Page 1: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

1 Introduction to nonlinear image processing

IPAM Summer School on Computer Vision

July 22, 2013

Introduction to Nonlinear Image Processing

Iasonas Kokkinos

Center for Visual Computing Ecole Centrale Paris / INRIA Saclay

Page 2: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

2 Introduction to nonlinear image processing

Mean and median

Observations in a 3x3 window:

Mean:

Median:

1) Sort:

2) Pick mid-point

robust to outliers

non-linear

outlier

Page 3: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

3 Introduction to nonlinear image processing

Mean

Page 4: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

4 Introduction to nonlinear image processing

Median

Page 5: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

5 Introduction to nonlinear image processing

Gaussian blur

Page 6: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

6 Introduction to nonlinear image processing

Image Processing

image image filter

Previous lecture:

This lecture: remove this constraint

freedom! at the expense of control

Page 7: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

7 Introduction to nonlinear image processing

7

Multi-scale Gaussian smoothing

Page 8: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

8 Introduction to nonlinear image processing

Rewrite Gaussian:

Scale space:

Gaussian scale space

‘time’

A. Witkin, Scale-space filtering, IJCAI, 1983.

J. Koenderink, The structure of images, Biological Cybernetics, 1984

J. Babaud, A. P. Witkin, M. Baudin, and R. O. Duda, ‘Uniqueness of the Gaussian kernel for

scale-space filtering’, PAMI, 1986.

A. Yuille, T.A. Poggio: Scaling theorems for zero crossings. PAMI, 1986.

T. Lindeberg, Scale-Space Theory in Computer Vision, Kluwer, 1994

L. Florack, Image Structure, Kluwer, 1997

B. Romeny, Front-End Vision and Multi-Scale Image Analysis, Kluwer, 2003.

J. Weickert, Linear scale space has first been proposed in Japan. JMIV, 1999.

Page 9: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

9 Introduction to nonlinear image processing

Heat diffusion and image processing

Gaussian satisfies:

Scale-space satisfies:

Associative property:

Scale-space satisfies:

Heat diffusion PDE (Partial Differential Equation)

Page 10: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

10 Introduction to nonlinear image processing

Heat diffusion and image processing

Page 11: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

11 Introduction to nonlinear image processing

Heat diffusion in 1D

Page 12: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

12 Introduction to nonlinear image processing

Heat diffusion in 1D – inhomogeneous material

12

conductivity

Page 13: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

13 Introduction to nonlinear image processing

Heat diffusion in 2D

Homogeneous

material

Inhomogeneous material

Page 14: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

14 Introduction to nonlinear image processing

Perona-Malik Diffusion

P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, PAMI 1990

F. Catte, P.L. Lions, J.M. Morel, T. Coll, Image selective smoothing and edge detection by

nonlinear diffusion, SIAM J. Numer. Analysis, 1992

Image-dependent conductivity

CLMC formulation:

Diffusion stops at strong image gradients (structure-preserving)

Page 15: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

15 Introduction to nonlinear image processing

(a) Linear diffusion at t=2 (b) Nonlinear diffusion at t=4.4

Nonlinear vs. linear diffusion

Page 16: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

16 Introduction to nonlinear image processing

Extension to vectorial images

• Extension of nonlinear diffusion to vectorial images:

generalization

where:

Page 17: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

17 Introduction to nonlinear image processing

(a) Color Image with Noise (b) Perona-Malik diffusion

Nonlinear diffusion for color image denoising

Page 18: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

18 Introduction to nonlinear image processing

• Cost functional:

• Euler-Lagrange:

• Heat diffusion: modifies temperature to decrease E quickly

Variational interpretation of heat diffusion

Page 19: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

19 Introduction to nonlinear image processing

Variational techniques

• Denoising as functional minimization – Functional: encodes undesirable properties

• Total Variation:

Minimization flow

u1(x) u2(x)

TV[u1] = TV[u2]

TV[f] = TV[g] = TV[h]

w1(x) w2(x)

TV[w1] > TV[w2]

Rudin, L. I.; Osher, S.; Fatemi, E.

"Nonlinear total variation based noise

removal algorithms". Physica D 60, 1992

Page 20: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

20 Introduction to nonlinear image processing

(a) Noisy image (b) Total Variation diffusion

Total Variation diffusion

Page 21: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

21 Introduction to nonlinear image processing

(a) Perona-Malik diffusion (b) Total-Variation diffusion

Perona-Malik versus Total Variation

Page 22: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

22 Introduction to nonlinear image processing

What is the ‘right’ cost functional?

Natural images: natural image statistics

Can we learn the cost function?

S.C. Zhu, Y. N. Wu and D. Mumford, ‘FRAME’, IJCV 1997

• Filters: Use Gabors, Difference-of-Gaussians, Gaussian filters,

• Random Fields: Construct distribution that reproduces their histograms

• And Maximum Entropy: while being as random as possible

Sample: Training:

Page 23: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

23 Introduction to nonlinear image processing

GRADE: Gibbs Reaction & Diffusion Equation

GRADE: maximize image probability using Euler-Largange PDEs

From FRAME to GRADE

S.C. Zhu, Y. N. Wu, D. Mumford, ‘Filters, Random Fields and Max. Ent.’, IJCV 1997.

S.C. Zhu and D. Mumford, ‘Gibbs Reaction and Diffusion Equation’, PAMI 1998.

S. Roth and M. Black, ‘Fields of Experts’, IJCV 2009

Page 24: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

24 Introduction to nonlinear image processing

Second Moment Matrix

Distribution of gradients:

Page 25: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

25 Introduction to nonlinear image processing

Second Moment Matrix

Distribution of gradients:

Page 26: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

26 Introduction to nonlinear image processing

Second Moment Matrix

Distribution of gradients:

Page 27: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

27 Introduction to nonlinear image processing

Second Moment Matrix

Distribution of gradients:

Page 28: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

28 Introduction to nonlinear image processing

• Eigenvectors : directions of maximal and minimal variation of u

• Eigenvalues: amounts of minimal and maximal variation u

Second Moment Matrix

Page 29: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

29 Introduction to nonlinear image processing

J. Weickert, Coherence-Enhancing Diffusion Filtering, Image and Vision Computing, 1999.

R. Kimmel, R. Malladi, and N. Sochen. Images as Embedded Maps and Minimal Surfaces, IJCV, 2001.

D. Tschumperlé, and R. Deriche (2005), Vector-valued image regularization with PDE’s. PAMI, 2005.

A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009

M. Lindenbaum, M. Fischer and A. Bruckstein, "On Gabor's contribution to image enhancement”, 1994.

D. Gabor, ‘Information theory in electron microscopy’, 1965

Anisotropic diffusion

Nonlinear diffusion

diffusion tensor

structure tensor

Nonlinear Anisotropic diffusion

Slide credit: A. Roussos

Page 30: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

30 Introduction to nonlinear image processing

Anisotropic Diffusion example

Slide credits: A. Roussos

Page 31: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

31 Introduction to nonlinear image processing

Anisotropic Diffusion example

Slide credits: A. Roussos

Page 32: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

32 Introduction to nonlinear image processing

Nonlinear anisotropic diffusion

Extension to vectorial images

Structure tensor for vectorial image:

Slide credits: A. Roussos

Page 33: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

33 Introduction to nonlinear image processing

Nonlinear anisotropic diffusion

Nonlinear Anisotropic Diffusion Noisy input

Slide credits: A. Roussos

Page 34: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

34 Introduction to nonlinear image processing

Nonlinear vs. Nonlinear and anisotropic diffusion

Nonlinear Anisotropic Diffusion Nonlinear Diffusion

Slide credits: A. Roussos

Page 35: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

35 Introduction to nonlinear image processing

Nonlinear vs. Nonlinear and anisotropic diffusion

Nonlinear Anisotropic Diffusion Nonlinear Diffusion

Slide credits: A. Roussos

Page 36: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

36 Introduction to nonlinear image processing

(a) Image with missing region

(c1) 300 iterations

(b) Initial prediction (constant)

(c2) 600 iterations (c3) 1500 iterations (converged)

Inpainting region D

Inpainting problem

M. Bertalmío, G. Sapiro, V. Caselles and C. Ballester., "Image Inpainting", SIGGRAPH 2000

T. F. Chan and J. Shen, "Mathematical Models for Local Nontexture Inpainting", SIAM, 2001

Page 37: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

37 Introduction to nonlinear image processing

(a) Input (b) Total Variation Inainting

Application: text removal

Slide credits: A. Roussos

Page 38: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

38 Introduction to nonlinear image processing

Application: fake bravado

Slide credits: A. Roussos

Page 39: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

39 Introduction to nonlinear image processing

(a) Low resolution input image (b) Zero-Padding initialization (c) 4x4 PDE-based magnification

PDE-based interpolation Interpolation problem

A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009

Page 40: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

40 Introduction to nonlinear image processing

(a) Input Signal (b) Bicubic interpolation (4 x 4)

(d) PDE-based interpolation(4 x 4)

PDE-based interpolation

A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009

Page 41: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

41 Introduction to nonlinear image processing

(a) Low-resolution Input (b) Initialization: Zero-Padding (c) 4x4 PDE-based interpolation

PDE-based interpolation

A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009

Page 42: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

42 Introduction to nonlinear image processing

PDE-based interpolation

(a) Input

(c) TV based Interpolation (d) Structure-tensor based interpolation

(b) Bilinear Interpolation

A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV

2009

Page 43: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further

43 Introduction to nonlinear image processing

Further study

G. Papandreou and P. Maragos, Multigrid Geometric Active Contour Models,TIP, 2007.

J. Weickert and B. H. Romeny, 'Efficient Schemes for Nonlinear Diffusion Filtering', TIP '98

A. Chambolle, 'An Algorithm for Total Variation Minimization and Applications', JMIV 2004

T. Goldstein, S. Osher The Split Bregman method for L1-regularized problems, SIAM 2009

Fast numerical solutions

Online software

http://www.ipol.im/

Not covered in this talk

Bilteral Filter, Non-Local Means

Buades, B. Coll, and J. Morel, “A non-local algorithm for image denoising,” CVPR, 2005

C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” ICCV 1998

P Milanfar, " A Tour of Modern Image Filtering ", IEEE Signal Processing Magazine, 2013