2D DSP2D DSP - svcl.ucsd.edu · 2D-DSP • in summary: – image is a N x M array of pixels –...
Transcript of 2D DSP2D DSP - svcl.ucsd.edu · 2D-DSP • in summary: – image is a N x M array of pixels –...
2D DSP2D DSP
Nuno VasconcelosUCSD
Images• the incident light is collected by an image sensor
– that transforms it into a 2D signal E
θPo
V
θ2Pi
V
2
2D-DSP• in summary:
– image is a N x M array of pixels– each pixel contains three colors– overall, the image is a 2D discrete-space
signal– each entry is a 3D vector
}0{},...,0{),,,(],[ 121
MnNnbgrnnx ∈=
– for simplicity, we consider only singlechannel images
},...,0{2 Mn ∈
b hi d l i i h f d
},...,0{},...,0{],,[
2
121
MnNnnnx
∈∈
3
– but everything extends to color in a straightforward manner
Separable sequences• a trivial concept,
– but probably the only real novelty in this lecture– very important in practice, because it reduces 2D problem to
collection on 1D problems
• Definition: a sequence is separable if and only if
][][][ ngnfnnx ×=
where f[.] and g[.] are 1D functions
][][],[ 2121 ngnfnnx ×=
• note: there are many examples of separable sequences• but most sequences are not separable
4
but most sequences are not separable
Linear Shift Invariant (LSI) systems• straightforward extension of LTI systems
• Definition: a system T that maps x[n1,n2] into y[n1,n2] isLSI if and only if– it is linearit is linear
{ }{ } { }],[],[
],[],[
212211
212211
nnxbTnnxaTnnbxnnaxT
+==+
– it is shift invariant
{ } { }],[],[
],[],[
212211
212211
nnbynnay +=
{ } ],[],[ 22112211 mnmnymnmnxT −−=−−
5
2D convolution• the operation
][][][ knknhkkxnny ∑ ∑∞ ∞
is the 2D convolution of x and h
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑−∞= −∞=
– we will denote it by
],[],[],[ 212121 nnhnnxnny ∗=
• this is of great practical importance:– for an LSI system the response to any input can be obtained by– for an LSI system the response to any input can be obtained by
the convolution with this impulse response– the IR fully characterizes the system
it is all that I need to measure
6
– it is all that I need to measure
2D convolution• has various properties of interest• but these are the ones that you have already seen in 1D
(check handout)• some of the more important:
commutative:– commutative:
– associative:
xyyx ∗=∗
( ) ( )zyxzyx ∗∗=∗∗
– distributive:
l ti ith i l
( ) zxyxzyx ∗+∗=+∗
– convolution with impulse:
],[],[],[ 2211221121 mnmnxmnmnnnx −−=−−∗ δ
7
2D convolution• as in 1D, it is most easily done in graphical form• e.g. how do we convolve these two sequences?
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑∞
−∞=
∞
−∞=
n2 n2
(3) (4)
n1 n1(1) (2)*
(3) (4)
],[ 21 nnx ],[ 21 nnh
8• we need four steps
2D convolution• step 1): express sequences in terms of (k1, k2)
][][][ kkhkk∑ ∑∞ ∞
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑−∞= −∞=
k2 k2
(3) (4)
k1 k1(1) (2)
],[ 21 kkx ],[ 21 kkh
9
2D convolution• step 2): invert h(k1, k2)
][][][ kkhkk∑ ∑∞ ∞
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑−∞= −∞=
2k2
(1)(2)
k2
(3) (4)
k2
(1) (2)
k1
( )( )
(3)(4)
k1(1) (2) k1
(1) (2)
(3) (4)1
],[],[ 2121 kkhkkg −−=],[ 21 kkh ],[ 21 kkh −
10
2D convolution• step 3): shift g(k1, k2) by (n1, n2)
][][][ kkhkk∑ ∑∞ ∞
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑−∞= −∞=
k2
(1)(2)
k2
(1)(2)
(3)(4)
n2
this sendswhatever is at (0,0) to (n1,n2)
k1
( )( )
(3)(4)
k1n1
],[],[ 2121 kkhkkg −−=][
],[ 2211
knknhnknkg =−−
11
],[ 2211 knknh −−
2D convolution• step 4): point-wise multiply the two signals and sum
][][][ kkhkk∑ ∑∞ ∞
– e.g. for (n1,n2) = (1,0)
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑−∞= −∞=
1 2
k2k2 k2
k1
(1)(2)
(3)(4)
n2
k1
x =k1
n2
][ knknh −−
n1
][ kkx ][ nny
n1
12
],[ 2211 knknh],[ 21 kkx ],[ 21 nny
2D convolution• step 4): point-wise multiply the two signals and sum
][][][ kkhkk∑ ∑∞ ∞
– e.g. for (n1,n2) = (2,0)
],[],[],[ 221121211 2
knknhkkxnnyk k
−−= ∑ ∑−∞= −∞=
1 2
k2k2 k2
k1
(1)(2)
(3)(4)
n2
k1
x =k1
n2
(3)
][ knknh −−
n1
][ kkx ][ nny
n1
13
],[ 2211 knknh],[ 21 kkx ],[ 21 nny
etc.
2D convolution• is this the only way to look at convolution?
– any signal can be written as
e g
],[],[],[ 221121211 2
knknkkxnnxk k
−−= ∑ ∑∞
−∞=
∞
−∞=
δ
– e.g.n2 n2 n2
n1
(a) (b)
n1
(a)
n1
(b)= +
],1[],[ 2121 nnbnna −+ δδ
14]0,0[x ]0,1[x
2D convolution• we combine this
][][][ knknkkxnnx ∑ ∑∞ ∞
δ
with the properties of the convolution
],[],[],[ 221121211 2
knknkkxnnxk k
−−= ∑ ∑−∞= −∞=
δ
– commutative:
– associative:
xyyx ∗=∗
( ) ( )zyxzyx ∗∗=∗∗associative:
– distributive:
( ) ( )zyxzyx ∗∗=∗∗
( ) zxyxzyx ∗+∗=+∗
– convolution with impulse:
],[],[],[ 2211221121 mnmnxmnmnnnx −−=−−∗ δ
15• to obtain another interpretation
2D convolution• it is done like this
],[*],[],[ 212121 nnhnnxnny =
],[*],[],[
][][][
21221121
212121
1 2
nnhknknkkx
y
k k⎟⎟⎠
⎞⎜⎜⎝
⎛−−= ∑ ∑
∞
−∞=
∞
−∞=
δ
( )],[*],[],[],[ 21221121211 2
1 2
nnhknknkkxnnyk k
k k
−−=
⎠⎝
∑ ∑∞
−∞=
∞
−∞=
∞ ∞
δ
• note that this is just our definition of convolution1 2k k∞ ∞
][][][ knknkkxnnx ∑ ∑∞ ∞
δ
(no surprises here)
],[],[],[ 221121211 2
knknkkxnnxk k
−−= ∑ ∑−∞= −∞=
δ
16
( p )• but we want to think about it like this, not like this
2D convolution• we can think of
( )][*][][][ hkkkk∑ ∑∞ ∞
δ
as the following sequence of operations
( )],[*],[],[],[ 21221121211 2
nnhknknkkxnnyk k
−−= ∑ ∑−∞= −∞=
δ
1. set y[n1, n2] = 0, for all (n1,n2)2. for each (k1,k2) such that x[k1,k2] is not zero
• set α = x[k k ]• set α = x[k1,k2]• shift h[n1,n2] by (k1,k2)• multiply the entire sequence by α
• add the entire sequence to y[n1,n2]
( )],[*],[],[ 21221121 nnhknknnnz −−= δα
17],[],[],[ 212121 nnznnynny +=
2D convolutionn2 n2
• examplen2 n2
*(3) (4)
n1 n1(1) (2)
• (k1,k2) = (0,0)],[ 21 kkx ],[ 21 nnh
n2 n2
(3) (4)
n2
(3) (4)
n1= n1(1) (2)+ 1xn1(1) (2)
18],[ 21 nny ],[ 2211 knknh −−],[ 21 nny
2D convolutionn2 n2
• examplen2 n2
*(3) (4)
n1 n1(1) (2)
• (k1,k2) = (1,0)],[ 21 kkx ],[ 21 nnh
n2n2
(3) (7)
n2
(3) (4) (3) (4)(4)
= n1+ 1xn1(1) (3) n1(1) (2) (1) (2)(2)
19],[ 2211 knknh −−],[ 21 nny ],[ 21 nny
2D convolutionn2 n2
• examplen2 n2
*(3) (4)
n1 n1(1) (2)
• (k1,k2) = (0,1)],[ 21 kkx ],[ 21 nnh
n2
(4) (9) (4)
n2
(3) (7) (4)
n2
(1) (2)
(3) (4)(3) (4)
=n1(1) (3) (2) n1(1) (3) (2) n1+ 1x
(1) (2)
20],[ 21 nny ],[ 21 nny ],[ 2211 knknh −−
2D convolutionn2 n2
• examplen2 n2
*(3) (4)
n1 n1(1) (2)
• (k1,k2) = (1,1)],[ 21 kkx ],[ 21 nnh
n2
(4) (10) (6)
n2
(3) (7)
n2
(4) (9) (4)
(3) (4)
(1) (2)
(3) (4)(4)
=n1(1) (3) (2) n1+ 1xn1(1) (3) (2)
(1) (2)
21],[ 21 nny ],[ 2211 knknh −−],[ 21 nny
2D convolution• note the differences with the previous convolution
– before we were computing one y[n1,n2] at a time
k1
k2
(1)(2)n2
k1
k2
x = k1
k2
n2(3)
k1
(3)(4)
],[ 2211 knknh −−
n1
k1
],[ 21 kkx
k1
],[ 21 nny
n1
– now we update the entire sequence at a time],[ 2211 knknh],[ 21 kkx ],[ 21 nny
n2 n2
(3) (7)
n2
(3) (4) (3) (4)(4)
=n1(1) (3)
(4) (10)
(2)
(6)
n1+ 1x
(3) ( )
n1(1) (3)
(4) (9)
(2)
(4)
(3) ( )
(1) (2)
(3) (4)( )
22],[ 21 nny ],[ 2211 knknh −−],[ 21 nny
Separable systems• Definition: a system is separable if and only if its
impulse response is a separable sequence
• note that, in this case the convolution simplifies
][][],[ 221121 nhnhnnh ×=
note that, in this case the convolution simplifies
∑ ∑∞
−∞=
∞
−∞=
−−=1 2
],[],[],[ 22112121k k
knknhkkxnny
∑ ∑∞
−∞=
∞
−∞=
−−=1 2
][][],[ 22211121k k
knhknhkkx
∑ ∑∞
∞
−∞=
∞
−∞=
−−=1 2
][],[][ 22221111k k
knhkkxknh
23∑∞
−∞=
−=1
],[][ 21111k
nkfknh
Separable systems• the convolution simplifies to
∑∞
][][][ nkfknhnny
with
∑−∞=
−=1
],[][],[ 2111121k
nkfknhnny
∞
• note that:
∑∞
−∞=
−=2
][],[],[ 2222121k
knhkkxnkf
• note that:– for a fixed k1, f[k1,n2] is 1D convolution of x[k1,n2] and h2[n2]
][][][ nhnkxnkf ∗
– for a fixed n2, y[n1,n2] is 1D convolution of f[n1,n2] and h1[n1]
][],[],[ 222121 nhnkxnkf ∗=
24][],[],[ 112121 nhnnfnny ∗=
Separable systems• the convolution simplifies to a sequence of 1D steps• step1) for every k1,
– f[k1,n2] is 1D convolution of x[k1,n2] and h2[n2]
][],[],[ 222121 nhnkxnkf ∗=
– which means: “convolve the columns of x with h2 to obtain columns of f”
n22n2
*
n1h2
25
n1
],[ 21 nnx ],[ 21 nnf
Separable systems• step2) for every n2,
– y[n1,n2] is 1D convolution of f[n1,n2] and h1[n1]
][],[],[ 112121 nhnnfnny ∗=
– which means: “convolve the rows of f with h1 to obtain rows of y”
n2n2
*
n1n1 h1
26
],[ 21 nny],[ 21 nnf
Separable systems• in summary, if we have a separable system
][][],[ 221121 nhnhnnh ×=
to convolve with x[n1,n2] we:
][][],[ 221121
– 1) “convolve the columns of x with h2 to create f”
][],[],[ 222121 nhnkxnkf ∗=
– 2) “convolve the rows of f with h1 to obtain y”
][][][ nhnnfnny ∗ ][],[],[ 112121 nhnnfnny ∗=
27
The Discrete-Space Fourier Transform• as in 1D, an important concept in linear system analysis
is that of the Fourier transform• the Discrete-Space Fourier Transform is the 2D
extension of the Discrete-Time Fourier Transform2211][)( ωω ωω eennxX njnj∑∑ −−
2121
2211
2211
1 2
)(1][
],[),(
ωωωω
ωω
ωω ddeeXnnx
eennxX
njnj
jj
n n
∫∫
∑∑
=
=
• note that this is a continuous function of frequency
2121221 ),()2(
],[ ωωωωπ
ddeeXnnx ∫∫=
q y• the nomenclature distinguishes it from the 2D Discrete
Fourier transform (we will get back to this)h d h DSFT f i l k lik ?
28
• what does the DSFT of an image look like?
Image spectrum• two images, the magnitude, and phase of their FTs
29
Phase and Magnitude
• curious fact– all natural images have about the same magnitude transformall natural images have about the same magnitude transform– monotonically decaying with frequency
1)( ωω ∝X
• hence, phase seems to matter, but magnitude largely d ’t
22
21
21 ),(ωω
ωω+
∝X
doesn’t• we can see this through the following experiment• take two pictures swap the phase transforms compute• take two pictures, swap the phase transforms, compute
the inverse• here is what you get
30
The importance of phaseReconstruction with zebra
phase, cheetah magnitudeReconstruction with cheetah
phase, zebra magnitude
31
LSI systems• why do we care so much about Fourier transforms?• note that when we convolve a sequence with a complex
exponential,
2211],[ 21njnj eennx ϖϖ=
we get
][][][ knknxkkhnny −−=∞ ∞
∑ ∑
e e ],[
],[],[],[
)(j)(j21
22112121
222111
1 2
ϖϖkkh
knknxkkhnny
knkn
k k
=
=
−−∞ ∞
−∞= −∞=
∑ ∑
∑ ∑
)(][),(e e 21
jj
21
2211
1 2
ϖϖϖϖϖϖ
HnnxHnn
k k
==
−∞= −∞=∑ ∑
32
),(],[ 2121 ϖϖHnnx=
LSI systems• but we have seen that, for an LSI system
– the output in response to x[n1,n2]– is the convolution with the impulse response h[n1,n2]– hence, the response to
2211][ njnj eennx ϖϖ
– is
2211],[ 21jj eennx =
),(],[],[ 212121 ϖϖHnnxnny =
– this means that complex exponentials are the eigenfunctions of LSI systems
)(][][ 212121y
– when we input an eigenfunction, we get back the same function– but scaled by H(ω1,ω2)– this is called the frequency response of the system
33
q y p y
LSI systems• this is remarkable, since
– we know that any signal can be represented as a weighted sum of comple e ponentialsof complex exponentials
21212211
1 2
],[),( ωω ωω eennxX njnj
n n∑∑= −−
21212212211
1 2
),()2(
1],[ ωωωωπ
ωω ddeeXnnx njnj∫∫=
– when the signal is fed to an LSI system, each exponential isscaled by H(ω1,ω2)
– hence the frequency response completely characterizes the– hence, the frequency response completely characterizes the system
– and the DSFT of the output is just the product of the two
34
),(),(),( 212121 ωωωωωω XHY =
Example• the system with impulse response on the left• has frequency response
n2
⎟⎠⎞
⎜⎝⎛
61
⎟⎠⎞
⎜⎝⎛
31
n1
⎠⎝ 6
⎟⎠⎞
⎜⎝⎛
61
⎟⎠⎞
⎜⎝⎛
61
⎟⎠⎞
⎜⎝⎛
61
⎠⎝ 3
2121
11111
],[),( 2211
1 2
ωω ωω= −−∑∑ njnj
n neennhH
11161
61
61
61
31 2121 ϖϖϖϖ ++++= −− jjjj eeee
• note that:
21 cos3
cos33
ϖϖ ++=
– the response is 1 at DC– lower for high frequencies
this system is a low pass filter
35
– this system is a low-pass filter
WARNING• WARNING, WARNING, WARNING!• the equivalence
11 ϖϖ jnjn−
1cos2
11
ϖϖϖ
nee jnjn
=+
• is the oldest trick in the DSP book!
2n2
• please do not fall for it– you can “read” the sequence that has this DSFT
111
2
⎟⎠⎞
⎜⎝⎛
61
⎞⎛
⎟⎠⎞
⎜⎝⎛
31
by applying this trick and the definition of DSFT!
2121 cos31cos
31
31 ),( ϖϖωω ++=H n1⎟
⎠⎞
⎜⎝⎛
61
⎟⎠⎞
⎜⎝⎛
61
⎟⎠⎞
⎜⎝⎛
61
36
WARNING• Quizz: which on the left is the DSFT of
this image?
?
37
WARNING• the way to think about this is the
following:
– this image has low frequencies– this image has low frequencieshorizontally (ω1)
– high frequencies vertically (ω2)the spectrum is a delta function along– the spectrum is a delta function along(ω1) and harmonics along (ω2)
– the spectrum is this“b i i h i l b ”
38
– wrong way: “because image is horizontal spectrum must be too”
57