1 Computer Vision Seminar, HUJI, December 2002 PDE-based Image Processing and the Triple Well...
-
Upload
jailyn-heggs -
Category
Documents
-
view
217 -
download
0
Transcript of 1 Computer Vision Seminar, HUJI, December 2002 PDE-based Image Processing and the Triple Well...
1
Computer Vision Seminar, HUJI, December 2002
PDE-based Image Processing and the Triple Well Potential for Image
Sharpening
Faculty of Electrical Engineering, Technion, Haifa, Israel
Guy Gilboa
A joint work with Nir Sochen & Yehoshua Y. Zeevi.
2
Objectives
• A review on PDE-based methods for image processing.
• Show relation to energy minimization. • Present a new well-shaped potential for
image sharpening.• Introduce hyper-diffusion for regularization.• Examples and conclusion.
3
• Linear scale-space.• Perona-Malik scheme.• The intuition behind nonlinear-
diffusion filtering.• Numerical schemes
Basic Linear and Nonlinear Diffusion
4
Uncommitted front end vision“We know nothing, we have no
preference whatsoever”The mathematical formulation for that is:• Linearity (no knowledge, no model)• Spatial shift invariance (no preferred
location)• Isotropy (no preferred orientation)• Scale invariance (no preferred size, or
scale)
5
Connecting PDE’s to Image processing, introducing: The Linear Scale-Space
• Scale Space is represented by the linear diffusion equation:
• We add a scale dimension to the original image – using a single scale parameter t.
• As shown by Koendrink: The diffusion equation is the unique scheme that incorporates all the above requirements (isotropy, homogeneity, causality).
image" original" |)( ; 0
2 t
xuu u t
6
Linear Scale-SpaceApplying the diffusion equation to the original
image – creating a 3rd dimension t
Adopted from [Romeny ‘96]
t
7
Pyramid
representation
Scale-Space
representation
8
Edge and corner detection of images
Adopted from:[Lindeberg-’94]
9
Application example: edge detection
Edge detection at many scales simultaneously, in the 1D case, by zero-crossing of Laplacian.
Signal at different scales edgesAdopted from:[Witkin-’83]
10
Linear Diffusion as a LPFThe Gaussian is the Green’s function of the
diffusion equation. In the 1D case we get:
)2 std.ith Gaussian w(
,0
4
2
exp4
1),(
,*),(),( :Solution
)0, ( :Equation
t
xt
t
x
ttxg
ftgtu
ft , u uuxx
t
11
Diffusion Processes• Linear diffusion
• Non-linear (inhomogeneous diffusion)
gradient theoffunction decreasing a is c
I)|)Idiv(c(| I t
constant positive a is c , Ic I)div(c I 2 t
12
Nonlinear diffusion example – Perona Malik:
2/K)|I(|1
1 |)Ic(|
0 5 10 15 20 25 30 35 40 45 500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
C(|�
U|)
� U
• Smoothing low gradients (mainly noise)• Preserving high gradients (singularities
and edges).
13
Linear diffusion example
14
Nonlinear diffusion example
15
Intuition for adaptive denoising
• Diffuse (low-pass-filter) only within the same region \ object.
– Therefore -> Slow the diffusion near edges.
“Do not diffuse the leaves of the tree with the sky at the background” [P-M]
16
Gaussian averaging along the curve of the signal
Adopted from [Sochen et al ‘01]
• The distance is measured not only spatially, the values of the signal are also considered.
•Related to the logic of bilateral filters.
201
2)0110 ))()((( pfpfpp),p(pd f
17
Relation to robust statistics
• Reducing the effect of outliers – pixels at the other side of an edge are treated as outliers and should not be considered in the estimation (see Black et al `98).
18
Numerics – how do we actually do that ?
• Reminder from first year Infi course:
• For images we usually take h=1and simply compute the difference between neighboring pixels: 1st order: Forward: Ii+1,j - Ii,j , Backward: Ii,j – Ii_1,j
Central: (Ii+1,j – Ii-1,j )/2
2nd order: Central: Ii+1,j – 2Ii,j + Ii-1,j
h
)h,y)-I(x,yI(xyxIh
lim
x
),(0
19
Explicit Schemes• Computation is done on each pixel
separately – using the values of the previous iteration. Example of linear diffusion:
ni
ni
ni
ni
ni
xxt
IIIhtII
h
thxItxIthxIttxIttxI
h
thxItxIthxI
t
txIttxI
II
1121
2
2
2)(
),(),(2),(),(),(
),(),(2),(),(),(
20
Explicit scheme – cont’
• Main advantage – simplicity, very easy to implement.
• Main disadvantage - time constraints (CFL bound): the scheme is stable only when
(for 2D)
• Summary: a very popular scheme (esp. when the process does not need a lot of iterations).
2 .25h0 t
21
Perona-Malik example in Matlab
function J=diffusion(J,K,N)for i=1:N,
% calculate gradient in all directions (N,S,E,W)In=[J(1,:); J(1:Ny-1,:)]-J;Is=[J(2:Ny,:); J(Ny,:)]-J;Ie=[J(:,2:Nx) J(:,Nx)]-J;
Iw=[J(:,1) J(:,1:Nx-1)]-J;% calculate diffusion coefficients
Cn=exp(-(abs(In)/K).^2);Cs=exp(-(abs(Is)/K).^2);Ce=exp(-(abs(Ie)/K).^2);Cw=exp(-(abs(Iw)/K).^2);
J=J+0.2*(Cn.*In + Cs.*Is + Ce.*Ie + Cw.*Iw);end; % for i
22
Advanced Schemes
• Implicit schemes – need to solve for all the pixels together, no time constraints, produces a very large set of equations. Iterative methods are often used (Jacobi, Gauss-Seidel, Multigrid).
• Level-sets – used for curve evolution (like snakes). Representing a curve as a level set of a higher dimensional function. Scheme is stable, non-parametric, able to change topologies.
23
Other PDE-based Processes
• Coherence-enhanceing diffusion.
• Total-Variation denoising
• Beltrami flow color processing
• Segmentation – Mumford-Shah functional and active contours (snakes).
24
Anisotropic diffusion• Cottet and Germain, Weickert -
coherence enhancing flow: strong diffusion along the edge, weak diffusion across the edge (tensor diffusion coef.) .
25
Total Variation denoising (Osher-Rudin-Fatemi)
• Denoising by minimizing the total variation yet staying close to the input image. Reduces the oscillatory part of the signal that contains mostly noise (but also texture and some small details).
• Energy to be minimized:
dxdyufdxdyuuETV
2)(||)( f – original image
26
TV - does not penalize large gradients (edges)
Same energy for any monotone part of the signal, unlike linear diffusion.
For L1 norm all the lines on the right has the same energy, whereas for L2 norm the blue line has the highest energy and the red line has the lowest.
27
TV vs. PM - a sketchy comparisonTop: original+noise(SNR=11.9dB)
Bottom:
left – TV (SNR=17.6dB)
right – PM (SNR=16.9dB)
TV PM
28
Color processing by Beltrami flow (Sochen, Kimmel, Malladi)
• Representing color image as a 2D surface in a 5D Riemannian manifold. A surface minimizing process that denoises and preserves edges
• Evolving each color channel via the Beltrami flow:
B.G,R,i
),det(
manifold theof metric
matrix symmetric definite, positive 2x2 a is where
1
I I 1t
G
G
G
g
,IgDivg
Δ iig
i
29
Beltrami flow – examples
30
Beltrami flow (cont’): denoising JPEG lossy effect – surface rendering of RGB channels.
31
Beltrami flow - movies
CENSOREDCENSORED
see www.cs.technion.ac.il/~ron
32
Image Segmentation
• Mumford shah functional
• Active contours (snakes)
33
Mumford-Shah Functional
• A variational approach for image segmentation.
• Minimizing the following energy functional:
f – original image, u – piece-wise smooth approx. of f separated by
the contour – C.
CMS CxduxdufCuE ||||)(),( 22222
34
Active contours (snakes)• Evolving a curve like a rubber-band, with the aim to
“close” on the object to be segmented, creating a continuous, smooth curve.
• Motivation is drawn from the active contour model of Kass et al (’87) but rely on level set techniques introduced by Osher and Sethian to handle topological changes in a seamless fashion (introduced independently by Caselles et al. and Malladi et al. in
’95-`97, “Geodesic Active Contours”).
Emin (C) “smooth”+”elastic”+”on edges”
L ppp dppCgpCpCCE ))((|)(||)(|)( 22
35
Segmentation examples
Adpoted from Chan & Vese, UCLA site
36
Video Segmentation
Adpoted from Julian Jerome ©
37
More processes
• Texture segmentation • Smoothing of vector fields• Image inpainting (filling missing information)• Movies – smoothing, filling frames etc.• Knowledge-based segmentation• Stereo vision
and more..
38
Some References1. Black, M., G. Sapiro, D. Marimont, and D. Heeger, Robust anisotropic
diffusion, IEEE Transactions on Image Processing Volume 7, PP 421-432, 1998.
2. V. Caselles, R. Kimmel, and G. Sapiro. Geodesic active contours. International Journal of Computer Vision, 22(1):61-79, 1997.
3. Chan T, Vese L.A., Image Segmentation Using Level Sets and the Mumford-Shah Model , CAM 00-14, April 2000
4. Chan T, Vese L.A. , Active Contours Without Edges, IEEE Image Proc. Feb 2001.
5. Chan T, Shen J., Vese L.A., Variational PDE models in image processing, Amer. Math. Soc. Notice, 50, pp. 14-26, January 2003.
6. G.H. Cottet and L. Germain, “Image processing through reaction combined with nonlinear diffusion", Math. Comp., 61 (1993) 659--673.
7. M. Kass, A. Witkin and D. Terzopoulos, "Snakes: Active contour models," International Journal of Computer Vision, pp. 321-331, 1987.
8. R Kimmel, R Malladi and N Sochen, ``Images as Embedding Maps and Minimal Surfaces: Movies, Color, Texture, and Volumetric Medical Images", Int. J. of Computer Vision, 39(2):111-129, Sept. 2000.
39
9. R. Malladi, J. A. Sethian and B.C. Vemuri. Shape modeling with front propagation : A level set approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(2):158-175, February 1995.
10. D. Mumford and J. Shah, Optimal approximations by piece-wise smooth functions and assosiated variational problems, Comm. Pure and Appl. Math., LII (1989), 577-685.
11. P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion", IEEE Trans. PAMI vol. 12,no. 7, pp. 629-639, 1990.
12. T. Lindeberg, “Scale-space theory: a basic tool..”, J. App. Statistics, 21(2):223-261, 1994.
13. Rudin L, Osher S and Fatemi C 1992 Nonlinear total variation based noise removal algorithm, Physica D 60, 259-268 (1992).
14. N Sochen, R Kimmel and R Malladi , “`A general framework for low level vision", IEEE Trans. on Image Processing, 7, (1998) 310-318.
15. N. Sochen, R. Kimmel, and A.M. Bruckstein. Diffusions and confusions in signal and image processing, Journal of Mathematical Imaging and Vision, 14(3):195-209, 2001.
16. ter Haar Romeney B.M., “An Intorduction to Scale-Space Theory”, VBC-’96, Hamburg, Germany.
40
• Books– ter Haar Romeny, Geometry-driven diffusion in computer vision.
– Weickert, Anisotropic diffusion in image processing .
– Sapiro, Geometric partial differential equations and image analysis .
• Sites– Ron Kimmel’s course “Numerical Geometry of Images”:
http://www.cs.technion.ac.il/~cs236861/index.html
– My web site: http://tiger.technion.ac.il/~gilboa/
17. J. Weickert,``Coherence-enhancing diffusion of colour images", Image and Vision Comp., 17 (1999) 199-210.
18. J. Weickert, A review on nonlinear diffusion filtering, LNCS 1252, Scale-Space Theory in Computer Vision, Springer-Verlag, 1997, 3-28.
19. A. P. Witkin, ``Scale space filtering", Proc. Int. Joint Conf. On Artificial Intelligence, pp. 1019-1023, 1983.
41
Nonlinear diffusion as an energy minimizing process
• A general nonlinear diffusion process can be viewed as a steepest descent sequence that minimizes the signal’s energy.
• The energy functional E is defined as the cumulative potential (energy density) Ψ of the signal in the domain Ω.
42
)||
|)(|'div()( I
I
IIEI t
dxIIE |)(|)( We define an energy functional E:
where Ψ is a potential which is a function of the gradient magnitude.
The steepest descent process is:
43
)|)(|div( IIcI t
Assigning:
we get the nonlinear diffusion equation(“Perona-Malik” style):
||
|)(|'|)(|
I
IIc
see- You, Xu, Tannenbaum, Kaveh, IEEE Trans. IP, 5(11), 1996.- Weickert, LNCS 1252, pp.3-28, 1997.
44
The nature of the diffusion depends on the potential function ψ (or the corresponding diffusion
coefficient c)
45
Potentials of several processes
0 0.5 1 1.5 2 2.5 3 3.5 4-8
-6
-4
-2
0
2
4
6
8
(a) Lin
(b) TV
(c) Char
(d) PM
(e) Inv
46
Diffusion coefficients of several processes
0 0.5 1 1.5 2 2.5 3 3.5 4-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
(a) Lin
(b) TV
(c) Char(d) PM
(e) Inv
47
• Reach a global minimum
• Well-posed processes
• Strong denoising
• Edge preservation is weaker
Convex Potentials(e.g. linear diffusion, Charbonnier
et al., Beltrami)
48
Nonconvex Potentials(e.g. Perona-Malik)
• In general can have many minima (Hollig, You et al.). produce staircasing.
• Need some sort of regularization to be well-posed (Catte et al).
• Weaker denoising
• Strong edge preservation
• Performs well for images-processing
49
forward
backward
Sharpening by going back in time ?
50
• Basic sharpening property: gradients should increase (at least in some range).
• High gradients should “cost “ less energy than medium gradients.
• Minimum-maximum principle is not kept.
Potential requirement for image sharpening
blur sharpen
51
• A classical ill posed sharpening process, attempting to reverse forward diffusion (Gaussian blur).
• Drawbacks:– Oscillatory– Amplifies noise exponentially– Causes the explosion of the signal
Inverse diffusionII t
2
52
1D inverse diffusion example: trying to restore a blurred step.
0 10 20 30 40 50 60-1012
Original
0 10 20 30 40 50 60-1012
Blurred
0 10 20 30 40 50 60-1012
inverse dif. t = 1
0 10 20 30 40 50 60-1012
inverse dif. t = 4
0 10 20 30 40 50 60-1012
inverse dif. t = 10
53
Inverse diffusionof a blurred image
54
We would like to find a potential function that has a sharpening ability and yet
avoids the inverse diffusion drawbacks.
55
Rule 1:
Low gradients should not be enhanced
• Avoid amplification of noise
• Specifically, the zero gradient should be stable –> have minimum energy.
Restrictions on the potential - 1
56
Rule 2:
Very high gradients should not be enhanced
• Avoid explosion of the signal• To reduce staricasing – very high
gradients should contribute some positive energy.
Restrictions on the potential - 2
57
Rule 3:
There should be minimal oscilations between low
energy statesWe assume the original image is with
little oscillations.
Restrictions on the potential - 3
58
The triple-well potential
Ix
Denoising
SharpeningSharpening
Slow smoothing
Slow smoothing
W(Ix)
Sharpening potential W(Ix) in one dimension.
Forms the shape of three wells.
59
From inverse diffusion to a Forward-and-Backward (FAB) diffusion
|grad(U)|
C(|grad(U)|)
0
-α
1
0 10 20 30 40 50 60-1
-0.5
0
0.5
1
1.5
2
0 10 20 30 40 50 60-1
-0.5
0
0.5
1
1.5
2
0 10 20 30 40 50 60-1
-0.5
0
0.5
1
1.5
2
60
Stability of smooth regionsGiven Mf > Mb then for every x0: |Ix (x0;0)|<rf we
satisfy |Ix (x0;t)|<rf for any t>0.
-50 -40 -30 -20 -10 0 10 20 30 40 50-6
-4
-2
0
2
4
6
J=C
(|Ix|)I
x
Ix
rf
kf
rb
kb-w k
bk
b+w-r
f-k
f-r
b
-kb+w-k
b-k
b-w
Mf
Mb
-Mb
-Mf
61
dxIRIFIWIE |)(|)(|)(|)( 2
The proposed energy functional for sharpening:
where
• W is a gradient dependent well-shaped potential.
• F is a fidelity term.
• R is a high order regularization term.
62
20 )(
2
1)( IIIF
Next:
We discuss the need for a high order regularization term R .
We assign a standard convex fidelity term to the input image I0:
63
• We search for the “smoothest” energy minimizer with minimal oscillations between the low energy states (similar to the viscosity solutions reasoning).
• For that we add a second order term to the energy functional. We use a convex rotationally invariant term:
The steepest descent flow is of hyper-diffusion.
Higher order regularization
22 ||2
1IR
64
Initial and boundary conditions:
Hyper-diffusion
II t4
xxIxI
II
nnn
t
,0)( ,0)(
;| 00
n is a unit vector normal to the boundary
65
• In the Cahn-Hilliard and Kuramoto-Sivashinsky equations a hyper diffusion term is used to stabilize linear inverse diffusion. Kuramoto, Dynamics of interacting particles, Springer 1984, Sivashinsky Ann. Rev. Mech. 15, 1983, J.W Cahn, J.E. Hilliard, J. Chem. Phys. 28,2, 1958.
• Physical processes modeled by forward-and-backward diffusion and hyper-diffusion are shown to have a unique solution (no proof in our case yet). see Witelski, Studies in Applied Mathematics, 96, pp. 277-300, 1996.
Hyper-diffusion as a stabilizer of inverse diffusion (from the literature)
66
Hyper-diffusion vs. diffusion 1D(Step and noise processed after times 0.1,1,10)
Hyper-diffusion Diffusion
10 20 30 40 50 60 70 80 90 100
0
0.5
1
10 20 30 40 50 60 70 80 90 100
0
0.5
1
10 20 30 40 50 60 70 80 90 100
0
0.5
1
10 20 30 40 50 60 70 80 90 100
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
20 40 60 80 100 120 140 160 180 200
0
0.5
1
67
Hyper-diffusion 2D(Cameraman processed after times 0.1,1,10)
68
Summary: Energy minimization process
for selective sharpening
IIIIIcI Wt4
0 )()|)(|div(
xxIxI
II
nnn
t
,0)( ,0)(
;| 00
Initial and boundary conditions:
69
Processing a 1D step with blur and noise
0 50 100 150-1012
0 50 100 150-1012
0 50 100 150-1012
0 50 100 150-1012
0 50 100 150-1012
70
1D line-edge with blur and noise
0 50 100 150-1012
0 50 100 150-1012
0 50 100 150-1012
0 50 100 150-1012
0 50 100 150-1012
71a
d
b
c
Processing a 2D step with different blur and noise
(a) Isotropic Gaussian(b) Anisotropic exponential(c) 5x5 box averaging(d) Jagginess+ additive Gaussian and uniform white noise.
72
Original Well-potential Regularized shock(ours) (Alvarez Mazorra)
Enhancement of a toy car by our scheme and a regularized shock filter (Alvarez-Mazorra).
73
Shrinked
Application: super-resolution from a single image
Low
resolution
High
resolution
74
Triple well modification (anisotropic with texture preserving)
Original Processed
75
Original
Processed
76
Conclusion• PDE-based techniques were shown to be effective
in a variety of image-processing applications with concise and well defined formulations.
• A well-shaped potential was presented for image sharpening.
• The process rewards the increase of gradients in some range, while being able to operate in a noisy environment and avoid oscillations and the explosion of the signal.
• Hyper-diffusion was introduced as a means to stabilize inverse-diffusion type processes.