Fourier, filtering, smoothing, and noise
Transcript of Fourier, filtering, smoothing, and noise
Fourier, filtering, smoothing, and noise
Nuno Vasconcelos ECE Department, UCSD
(with thanks to David Forsyth)
2
Plan for todaywe have talked about some theoretic of 2D DSPtoday we will see examples with real imageswe will:• talk about the discrete-space Fourier transform• sampling in 2D• visualize concepts such as frequency spectra and aliasing• see that very important vision operations are really just filtering,
with some post processing• introduce some important filters that are widely used in practice • talk about the impact of noise and how to deal with it
3
The Discrete-Space Fourier Transformis, once again, a straightforward extension of the 1D Discrete-Time Fourier Transform
properties:• basically the same as in 1D (see table in Lim, page 25)• only novelty is separability (homework)
2121221
2121
2211
2211
1 2
),()2(
1],[
],[),(
ωωωωπ
ωω
ωω
ωω
ddeeXnnx
eennxX
njnj
njnj
n n
∫∫
∑∑
=
= −−
)()(),(][][],[ 221121221121 ωωωω XXXnxnxnnx =↔=
4
Image spectrumtwo images, the magnitude, and phase of their FTs
5
Phase and Magnitudecurious fact• all natural images have about the same magnitude transform• monotonically decaying with frequency
hence, phase seems to matter, but magnitude largely doesn’twe have seen this a little bit in the first problem settook two pictures, swap the phase transforms, compute the inversehere is another example
22
21
211),(ωω
ωω+
∝X
6
The importance of phaseReconstruction with zebra
phase, cheetah magnitudeReconstruction with cheetah
phase, zebra magnitude
7
Sampling in 2Dconsider an analog signal xc(t1,t2) and let its analog Fourier transform be Xc(Ω1,Ω2)• we use capital Ω to emphasize that this is analog frequency
sample with period (T1,T2) to obtain a discrete-space signal
222111 ;2121 ),(],[TntTntc ttxnnx
===
8
Sampling in 2Drelationship between the Discrete-Space FT of x[n1,n2]and the FT of xc(t1,t2) is simple extension of 1D result
DSFT of x[n1,n2] FT of xc(ω1,ω2) “discrete spectrum” “analog spectrum”
Discrete Space spectrum is sum of replicas of analog spectrum• in the “base replica” the analog frequency Ω1 (Ω2) is mapped into
the digital frequency Ω1T1 (Ω2T2)• discrete spectrum has periodicity (2π,2π)
∑ ∑∞
−∞=
∞
−∞=⎟⎟⎠
⎞⎜⎜⎝
⎛ −−=
1 2 1
11
1
11
2121
2,21),(r r
c Tr
TrX
TTX πωπωωω
9
For exampleω2
ω1
Ω’T1
Ω”T2
ω1
ω1
Ω’
2π−Ω”T2
2π2π
2π
2π
2π−Ω’T1
Ω2
Ω1
Ω’
Ω”
2
1
''''''
TT
Ω=→ΩΩ=→Ω
βα
no aliasing if
⇔⎩⎨⎧
Ω−≤ΩΩ−≤Ω
22
11
'2'''2'TTTT
ππ
⎩⎨⎧
Ω≤Ω≤
⇔''/'/
2
1
ππ
TT
10
Aliasingthe frequency (Ω’/π,Ω’’/π)is the critical sampling frequencybelow it we havealiasing
this is just like the 1D case, but now there are more possibilities for overlap
ω2
ω1
11
Reconstructionif there is no aliasing we can recover the signal in a way similar to the 1D case
note: in 2D there are many more possibilities than in 1D• e.g. the sampling grid does not have to be rectangular, e.g.
hexagonal sampling when T2 = T1/sqrt(3) and
• in practice, however, one usually adopts the rectangular grid
( )
( )
( )
( )∑ ∑∞
−∞=
∞
−∞= −
−
−
−=
1 2222
2
2222
1111
1111
2121
sinsin],[),(
n nc
TntT
TntT
TntT
TntTnnxtty π
π
π
π
⎩⎨⎧
= ==
otherwise0
odd or even both21;2121
,),(],[ 222111
nnttxnnx TntTntc
12
Sampling and aliasing in 2Din summary, sampling is not very different from the 1D case
but aliasing is a lot more fun to look at in imagesit shows up in video too (the wagon wheel effect)
13
a sequence of images obtained by down-sampling without any filteringaliasing: the low-frequency parts are replicated throughout the low-res image
14
The role of smoothing
none
some
a lot
too little leads to aliasingtoo much leads to loss of information
15
Linear Filteringsmoothing is implemented with linear filtersgiven an image x(n1,n2), filtering is the process ofconvolving it with a kernel h(n1,n2)
some very common operations in image processing are nothing but filtering, e.g.• smoothing an image by low-pass filtering• contrast enhancement by high pass filtering• finding image derivatives• noise reduction
),(),(),( 2211212121
knknhkkxnnykk
−−= ∑
16
Popular filtersbox function
Fourier transform of a box is the sinc, low-pass filter
side-lobes produce artifacts, smoothed image does not look like the result of defocusing
⎩⎨⎧ −≤≤−≤≤
=otherwise
NnNnnnR xNN 0
10,10,1),( 2211
2121
F
17
Example: Smoothing by Averaging
18
Camera defocusingif you point an out-of-focus camera at a very small white light (e.g. a light-bulb) at night, you get something like thisthe light can be thought of as an impulsethis must be the impulse responsewell approximated by a Gaussianmore natural filter for image blur than the box
⎟⎟⎠
⎞⎜⎜⎝
⎛ +−= 2
22
2 2exp
21),(
σπσyxyxh
19
The Gaussianthe discrete space version is
obviously separable
h(n1,n2) has Fourier transform
⎟⎟⎠
⎞⎜⎜⎝
⎛ +−= 2
22
21
221 2exp
21),(
σπσnnnnh
44344214434421)(
2
)(
221
2
2
22
1
2
21
21
21),(
nh
n
nh
n
eennh σσ
σπσπ
−−×=
⎟⎟⎠
⎞⎜⎜⎝
⎛ +−=
2)(exp),(
22
21
2
21ϖϖσϖϖH
20
The Gaussian filterthe Fourier transform of a Gaussian is a Gaussian(σx,σy) ∝ (1/σw1,1/σw2)
F
note that there are no annoying side-lobes
21
Smoothing with a Gaussian
22
Role of the variancethe variance controls the amount of smoothingeach column shows different realizations of an image of gaussian noiseeach row shows smoothing with gaussians of different σ
23
Gradients and edgesfor image understanding, one of the problems is that there is too much information in an image just smoothing is not good enoughhow to detect important (most informative) image points?note that derivatives are large at points of great change• changes in reflectance (e.g. checkerboard pattern)• change in object (an object boundary is different from background)• change in illumination (the boundary of a shadow)
these are usually called edge pointsdetecting them could be useful for various problems• segmentation: we want to know what are object boundaries• recognition: cartoons are easy to recognize and terribly efficient to
transmit
24
The importance of edgeslight
reflectance
boundary
25
Gradientsfor a 2D function, f(x,y) the gradient at a point (x0,y0)
is the direction of greatest increase at that pointthe gradient magnitude
measures the rate of changeit is large at edges!
T
yxyfyx
xfyxf ⎟⎟
⎠
⎞⎜⎜⎝
⎛∂∂
∂∂
=∇ ),(),,(),( 000000
( )Tyx yxfyxf ),(),,( 0000=
f(x,y)
),( 00 yxf∇
),( 11 yxf∇
– large gradient magnitude
– small gradient magnitude
2
)⎟⎟⎠
⎞00
2
002
00 ,(),(),( ⎜⎜⎝
⎛∂∂
+⎟⎠⎞
⎜⎝⎛∂∂
=∇ yxyfyx
xfyxf
26
Derivatives and convolutionrecall that a derivative is defined as
linear and shift invariant, so must be the result of a convolution.we could approximate as
where the derivative kernel is
( ) ( ) ( )x
xfxxfxxf
x ∆−∆+
=∂
∂→∆ 0
lim
( ) ( ) ( ) ( ) ( ) )(*11
1 nhfnfnfnfnfnnf
=−+=−+
=∂
∂
)()1()( nnnh δδ −+=
27
Finite difference kernelsin two dimensions we have various possible kernelse.g. , N1=2, N2=3, derivative along n1, (line n2=k)(horizontal)
derivative along n2, (line n1=k) (vertical)
derivative along line n1=n2 (diagonal)
n1
n21 -11 -11 -1
0 01 -10 0
n1
n2
0 -1 00 1 0
-1 -1 -11 1 1
n1
n20 0 -10 0 01 0 0
1 -1 00 1 -10 0 1
28
Finite difference kernelsnote that, when
h(n1,n2)=
we have
derivative is a high-pass filterhw: check that this holds for all othersintuitive, because a derivative is a measure of the rate of change of a function
222111 ϖϖϖ jjj
eee ⎟⎟⎠
⎞⎜⎜⎝
⎛−=
−
0 01 -10 0
211
2sin2
ϖϖ je⎟⎠⎞
⎜⎝⎛=
( ) 1, 121 −= ϖϖϖ jeH
29
Finite differencesQ: which one do we have here? (gray=0,white=+,dark=-)
30
Finite differences and noisebecause they perform high-pass filtering, finite difference filters respond strongly to noisegenerally, the larger the noise the stronger the responsefor noisy images it is usually best to apply some smoothing before computing derivativeswhat do mean by noise? we only consider the simplest model• independent stationary additive Gaussian noise• the noise value at each pixel is given by an independent draw
from the same normal probability distribution
),0(~),,(),(),( 2212121 σεε Nnnnnxnny +=
31
sigma=1
32
sigma=16
33
Finite differences responding to noise
increasing noise variance
note that as the noise variance increases the estimates of the image derivative are also very noisyQ: would a larger filter do better?
34
The response of a linear filter to noisesuppose we apply filter h(n1,n2) to
by linearity
consider the noise term
by linearity of expectation
),0(~),,(),(),( 2212121 σεε Nnnnnxnny +=
noiseoutputsignaloutput
hxhyh
ε*** +=
),(),(),( 2211212121
knknhkknnkk
−−= ∑εν
[ ] [ ] 0),(),(),( 2211212121
=−−= ∑ knknhkkEnnEkk
εν
35
The response of a linear filter to noiseand since the ε(n1,n2) are independent
output error has• zero mean• variance that increases with the size of the filter and the input
variance
increasing the filter size would definitely not help!
[ ] [ ] ),(),(var),(var 22112
212121
knknhkknnkk
−−=∑ εν
),( 221122
21
knknhkk
−−= ∑σ
36
Smoothing reduces noisenoise has a lot of high-frequenciesstrategy:1. start by low-pass filtering, to suppress
noise2. compute derivative on smoothed image
i.e. for a smoothing filter g(n1,n2)compute
note that, by associativity of convolution, this is equal to
i.e. filter the image with the filter h*g
g(n1,n2)
h(n1,n2)
( )xgh **
( ) xgh **smoothed derivatives
37
The derivative of a GaussianDoG along n1let’s consider, for example,
in which case
is a difference of two Gaussiansthis is the derivative of a Gaussian (DoG) filterfor other definitions of h we have a similar result
),(),1(),( 212121 nnnnnnh δδ −+=
),(),1(),(* 212121 nngnngnngh −+=DoG along n2
38
Smoothed derivativesno smoothing
with smoothing
39
Choosing the right scalethe scale of the smoothing filter affects • derivative estimates, • the semantics of the derivative image
trade-off between noise and ability to detect detail
1 pixel 3 pixels 7 pixels
40
41
Gradients and edgesby now we have a good idea of how to differentiate imagesremember that edges are points of large gradient magnitudeedge detection strategy1. determine magnitude of image gradient
2. mark points where gradient magnitude is particularly large wrt neighbours(ideally, curves of such points)
– large gradient magnitude
– small gradient magnitude
2
00
2
002
00 ),(),(),( ⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
+⎟⎠⎞
⎜⎝⎛∂∂
=∇ yxyfyx
xfyxf
42
Looks easy butthree major issues (to discuss next class):• 1) gradient magnitude at different scales is different (see below);
which should we choose?• 2) gradient magnitude is large along thick trail; what are the
significant points?• 3) how do we link the relevant points up into curves?