Multi View Geometry (Spring '08)
Transcript of Multi View Geometry (Spring '08)
1
Camera Models 1
2008-1
Multi View Geometry (Spring '08)
Prof. Kyoung Mu LeeSoEECS, Seoul National University
Camera Models
Camera Models 2
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Camera Models – Finite Cameras• Pinhole camera model:
In homogeneous coordinates
ZYfy
ZY
fy
=⇒=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
10100
ZYX
ff
ZfYfX
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
1010101
1ZYX
ff
ZfYfX
PXx = [ ]0|I)1,,(diagP ff=
2
Camera Models 3
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Finite cameras
• Principal point offset model:Set the coordinates of the principal point (image center) beThen the mapping becomes
Or in homogeneous representation
Tyx pp ),(
calibration matrix
Tyx
T pZfYpZfXZYX )/,/(),,( ++a
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛++
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
10100
1ZYX
pfpf
ZZpfYZpfX
ZYX
y
x
x
x
a
[ ] camX0|IKx =⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1y
x
pfpf
K
Camera Models 4
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Finite cameras
• Camera Rotation and Translation:
center camera the of scoordinate ousinhomogene :~
scoordiante camera ofmatrix rotation :scoordinate camera ousinhomogene:~
scoordinate worldousinhomogene:~)~~~
cam
cam
C
RX
X
CXR(X −=
world coordinatescamera coordinates
X10RCR
110
C~RRXcam ⎥⎦
⎤⎢⎣
⎡ −=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎦
⎤⎢⎣
⎡ −=
ZYX
[ ] camX0|IKx = [ ]XC~|IKRx −= PXx =[ ]t|RKP =
C~Rt −=
3
Camera Models 5
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
[ ]C~|IKRP −=
Finite cameras
• CCD Cameras:Non-square pixels
10 DOF
• Finite projective camera:Including skew parameter s (zero for most normal cameras)
11 DOF (5+3+3)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1yyy
xxx
pmfmpmfm
M: 3x3 nonsingular
[ ]44 ][ pM|IMp|MP 1−==
{finite cameras}={P4x3 =[M|p4]|det M≠0}
If mx and my pixels per unit distancein x and y directions
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
10
0
yx
x
x
αα
K⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
11y
x
x
x
pfpf
mm
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1yx
xx
pp
αα
K
Camera Models 6
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Camera Models - The projective camera
• General projective cameras: [ ]4p|MP =
C~
4
Camera Models 7
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• Camera center:is the null-space of the camera projection matrix
For all A, all points on the line AC project on image of A, therefore C is the camera centerImage of camera center is (0,0,0)T, i.e. undefined
Finite Camera case:
Infinite Camera case:
0PC =CAX λ)(1λ −+=
PAPCPAPXx λλ)(1λ =−+==
⎟⎟⎠
⎞⎜⎜⎝
⎛−=
−
14
1pMC
0,0
=⎟⎟⎠
⎞⎜⎜⎝
⎛= Md
dC
C~
Camera Models 8
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• Column vectors of P:[ ]
[ ] origin world1000 , While,
. and , toSimilarilyon point axis,-X ofdirection 0001 ,
4
3
1
⇒==
⇒== ∞
Too
TXX
DPDp
ppDPDp
2
π
5
Camera Models 9
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• Row vectors of P:
The principal plane: qthe plane passing C parallel to the image plane, set of X, s.t.,
Axis plane:qPoints on P1,
qSimilarly, P2 is a plane defined by C and the line y=0
plane principal thengrepresenti vector theis 0)0,,(
3
3
PXPPX
⇒
=⇒= TTyx
0 line theand by defined plane a is Thus,on lies 0
axis-yon points),,0(0
1
11
1
=
⇒=
⇒=⇒=
x
yT
TT
CPPCCP
PXXP ω
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=T
T
T
3
2
1
ppp
P
Camera Models 10
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• C is the intersection of P1, P2 and P3: 0CPPP
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
T
T
T
2
2
1
Principal axis
Camera center C
Principal point
6
Camera Models 11
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• The principal point:
P3
Normal direction of P3
= (p31,p32,p33)T
x0
[ ] 334
30
ˆ|ˆ MmPPMPPx ===
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
0
ˆ33
32
31
3
ppp
P
∞π
Point on π∞
Camera Models 12
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• Principal axis vector:There is an ambiguity between m3 and –m3
[ ] [ ]
cameratheoffront thetowarddirection theis on t independen isdirection
)det( )det(ˆ)ˆdet(ˆ
ˆ|ˆ~|for
then,)det(Let
434
343
4
3
v
vmMmKRmMv
pMCRRKP
mMv
⇒⇒
==
==
=−=
=
kkk
k
k 3m
7
Camera Models 13
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Action of projective camera on point
• Forward projection:
Projection of vanishing point:
• Back-projection:
Finite camera case:
D=(dT,0)T
1)( −+ = TT PPPP
A ray passing through C and P+x
41~ pMC −−=
CBack projected point on π∞
xPX +=
IPP =+
Camera Models 14
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• Depth of points:Consider the projection of a space point X onto an image point x
. ofdirection (positive) in the from ofdepth the
represent ,1 and 0 det if ,Thus
)~~()(
Then,
1
~
1
3
3
333
3
2
1
mCX
mM
CXmCXPXP
X
PPP
PXx
ω
ω
ω
=>
−=−==
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
==⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
TTT
T
T
T
yx
Since PC=0
dot product of the ray from C to X, with the principal ray direction
8
Camera Models 15
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective camera
• In general, for
( )3
)sign(det;depthm
MPXT
ω=
( )TX X,Y,Z,T=
Camera Models 16
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The projective cameras - Decomposition of P
• Finding the camera center:
• Finding the camera orientation and internal parameters:
Decompose M as M = KR using RQ factorization methodK: Upper triangular matrix R: rotation matrix (orthogonal) Keep K to have positive diagonals to resolve ambiguity
SVD using of space null),,,( , PC0PC →== TTZYX
p. 67
9
Camera Models 17
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Decomposition of P
• When is skew non-zero?
for CCD/CMOS, always s=0Image from image, s≠0 possible (non coinciding principal axis)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1yx
xx
pps
αα
K1 γ
arctan(1/s)
Camera Models 18
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Decomposition of P
• Euclidean vs. projectivegeneral projective interpretation
Meaningfull decomposition in K,R,t requires Euclidean image and spaceCamera center is still valid in projective spacePrincipal plane requires affine image and spacePrincipal ray requires affine image and Euclidean space
[ ] [ ]homography 44010000100001
homography 33 ×⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡×=P
10
Camera Models 19
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Cameras at infinity – Affine cameras
• Camera at infinity Cameras center lying on the plane at infinityaffine and non-affine cameras
• Affine cameras: the last row of P, P3T is the form of (0,0,0,1)Thus, points at infinity are mapped to points at infinityConsider moving back a finite projection camera while zooming tokeep the size of object be the same
[ ]
0singular is
4
=
=
PCM
p|MP
ray principal theofdirection in thecenter camera thefromorigin world theof distance the:~
ray principal ofdirection the:3
0
3
Cr
rTd −=
Camera Models 20
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras
• Moving back at unit speed for a time t, so that
• Next, by zooming with a zooming factor so that the image size remains fixed. Then,
3~~ rCC t−→
ttd T
t
at timeray principal theofdirection in thecenter camera thefromorigin world theof distance the:~3 +−= Cr
0/ ddk t=
11
Camera Models 21
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras
Camera Models 22
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras
• As dt tends to infinity,
Affine camera
12
Camera Models 23
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras-Error
• For point on plane parallel with principal plane and through the world origin,
• For general points
⎟⎟⎠
⎞⎜⎜⎝
⎛ +=
1βα 21 rr
X XPXPXP ∞== t0
⎟⎟⎠
⎞⎜⎜⎝
⎛ Δ++=
1βα 321 rrr
X
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
+==
Δ
~~
0
0proj
dyx
KXPx
projxaffinex
0x
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛== ∞
0
affine~~
dyx
KXPx
( )0proj0
00affine
~-~~-~ xxxxd
d Δ+=
Camera Models 24
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras-Error
• Thus, the error in employing an affine camera:
• Approximation error is small ifΔ is small compared to the average depth d0
The distance of the point from the principal ray is small (i.e. small field of view)
( )0proj0
projaffine~-~~-~ xxxx
dΔ
=
13
Camera Models 25
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras-Decomposition
• Decomposition of
or
Thus,
∞P
0x0 =~for
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡= ×
1100000100001
1ˆˆ
22TT 0
tR0
0K
Camera Models 26
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Affine cameras
• Summary of parallel projection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=∞
100000100001
P canonical representation
⎥⎦
⎤⎢⎣
⎡= ×
100K 22K calibration matrix
principal point is not defined
14
Camera Models 27
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Hierarchy of affine cameras
• Orthographic projection:
• Incorporating rotation and translation
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
11
100000100001
1YX
ZYX
yx
[ ]tM0
rr
0tR
P =⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
11
100000100001
22
11
tt
T
T
T
unit norm
5 DOF
Camera Models 28
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Hierarchy of affine cameras
• Scaled orthographic projection:
• Weak perspective projection:
6 DOF
7 DOF
15
Camera Models 29
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Hierarchy of affine cameras
• Affine camera:
• In inhomogeneous coordinates
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡= ×
1100000100001
1ˆˆ
22TT 0
tR0
0K
3x3 affine 4x4 affine
3x4 affine
linear mapping
8 DOF
Camera Models 30
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Hierarchy of affine cameras
• Properties of the affine camera:
Any projective camera with the principal plane is the plane at infinity is an affine cameraThe camera center is also lies on the plane at infinityParallel lines in space are mapped to parallel image linesThe vector d satisfying M2x3d=0 is the direction of parallel projection, and (dT,0)T is the camera center.
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
00
YX
ZYX
APplane at infinity
= principal plane
point at infinity
0d
P =⎥⎦
⎤⎢⎣
⎡0A
16
Camera Models 31
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
General cameras at infinity
• Non-affine camera at infinity
[ ]
0zeronot is rowlast but thesingular is
4
=
=
PCM
p|MP
Camera Models 32
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Other camera models
• Pushbroom cameras:• Linear Pushbroom (LP) camera – satellite sensor, SPOT
(11dof)
17
Camera Models 33
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Other camera models
• straight lines are not mapped to straight lines
⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡→
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
ωω
/~~
1y
xyx
ZYX
yx
P
hyperbola ~~~~ ⇒=+++⇒ oyxyx δγβα
Camera Models 34
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Other camera models
• Line cameras:2D to 1D projection
5 DOF
0PC =
camera center