Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical...
Transcript of Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical...
Camera Calibration
Machine Vision (CIS 580)
[ ] =
x XK R tL
1 1
Camera body configuration (extrinsic parameter)
Spatial relationship between sensor and pinhole (internal parameter)
Lens configuration (internal parameter)
Why Do We Calibrate?
• To find an intrinsic bias produced by lens configuration and manufacture imprecision. • The intrinsic bias is often fixed as long as the focal length
of the camera does not change. • Pre-calibration of the intrinsic bias reduces the number of
unknowns in estimation of camera parameters, e.g., structure from motion.
• The intrinsic bias encoded in an image can serve as a fingerprint of the camera.
Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e.,
f (mm) fx (pixel).
• Position of image center (principal point), i.e., px, py
• A skew factor between x and y axis of the image, i.e., uimg=fx x+sy+px.
• Lens distortion
f (mm)
fx (pixel)
Single View Calibration
[ ] =
x XK R tL
1 1
Camera body configuration (extrinsic parameter)
Spatial relationship between sensor and pinhole (internal parameter)
Lens configuration (internal parameter)
Ideal Perspective Projection
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
x X×∈R 3 3 tK
• Estimating K from 2D measurements, x, without assumption is an ill-posed problem because the number of unknowns are greater than the number of equations, i.e.,
• # of unknowns: 5 (K) + 3 (R) + 3 (t) + 3 (X) • # of equations: 2
Single View
[ ]×
=
= I 0x x x x
y y y y
Xu f s p f s p X
Yz v f p f p Y
ZZ
camimg cam
camimg 3 3 cam
camcam1 1 1
1
• Without loss of generality, we can reduce the number of unknowns by setting . If Xcam is known, the intrinsic parameters can be estimated using multiple Xcam.
• # of unknowns: 5 (K) • # of equations: 2n
[ ]×P = K I 03 3
Lines of Projections
u
v x
X−K x1
[ ]×
=
Xx = K I 0 KXz 3 3 1
−K x = Xz 1
−K x1 defines a ray emitting from the origin (the camera center) towards X.
[ ]×P = K I 03 3
Lines of Projections
u
v x1
[ ]×P = K I 03 3
X−K x1
x2θ
2
− −
− − − −
− −
− − − −=
K x K x
K x K x K x K x
x K K x
x K K x x K K x
1 T 11 2
1 T 1 1 T 11 1 2 2
T T 11 2
T T 1 T T 11 1 2
( ) ( )cos
( ) ( ) ( ) ( )
( )
( ) ( )
θ =
[ ]×
=
Xx = K I 0 KXz 3 3 1
−K x = Xz 1
−K x1 defines a ray emitting from the origin (the camera center) towards X.
Angle between two rays:
Angle Between Vanishing Points
[ ] ∞
∞
=
=
x K 0 X
KX1 1
1
∞X2
∞=x KX2 2θ
′θ
∞X1
′θ = θKey observation: the angle between two lines in 3D is the same as the angle between two rays pointing at the vanishing points of the two lines, i.e., .
2
− −
− − − −=
x K K x
x K K x x K K x
T T 11 2
T T 1 T T 11 1 2
( )cos
( ) ( )θ =
Result: if we know angle between two lines in 3D, we can estimate K via
Projection of ∞X1
X is inhomogeneous coordinate of . X
2D Case
∞X2
∞=x KX2 2
θ′θ
∞X1
′θ = θ
Camera center at O
f
∞=x KX1 1
p
Focal plane
L1 ′L1
L2
′L2
L1 ′L1 L2′L2because and
2D Case Wrong Focal Length
∞X2
∞=x KX2 2
θ
′θ
∞X1
′≠θ θ
Camera center at O
f
∞=x KX1 1
p
2D Case Wrong Principal Point
∞X2
∞=x KX2 2
θ
′θ
∞X1
Camera center at O
f
∞=x KX1 1
p
′≠θ θ in general.
2D Case
∞X2
∞=x KX2 2
θ′θ
∞X1
′θ = θ
Camera center at O
f
∞=x KX1 1
p
Focal plane
L1 ′L1
L2
′L2
L1 ′L1 L2′L2because and
Given two vanishing points, the focal length and principal point are uniquely defined. For the 3D case, another vanishing point is needed to uniquely define f, px, and py.
Calibration via Vanishing Points
Calibration via Vanishing Points
Calibration via 3 Vanishing Points
− − =x K K xT T 11 2( ) 0
x1 x2
x3
When vanishing points of lines are perpendicular,
x1
x2
O
2
− −
− − − −=
x K K x
x K K x x K K x
T T 11 2
T T 1 T T 11 1 2
( )cos 0
( ) ( )θ =
−K x1 2
−K x1 1
Calibration via 3 Vanishing Points − − =x K K xT T 1
1 2( ) 0− − =x K K xT T 1
3 2( ) 0− − =x K K xT T 1
1 3( ) 0x1 x2
x3
Assume zero skew and square pixel:
K =x
y
f p
f p
0
13 unknowns and 3 equations.
− −
= =
K K B
B B
B B
B B B
1 2T 1
1 3
2 3 4
=x BxT3 2 0
=x BxT1 2 0
=x BxT1 3 0
Substitution to formulate linear equations:
+ + +
+ + + = = + + +
A 0b
Bu u v v u u v v
Bu u v v u u v v
Bu u v v u u v v
B
11 2 1 2 1 2 1 2
23 2 3 2 3 2 3 2
31 3 1 3 1 3 1 3
4
1
1
1
Calibration via 3 Vanishing Points
x1 x2
x3
+ + +
+ + + = = + + +
A 0b
Bu u v v u u v v
Bu u v v u u v v
Bu u v v u u v v
B
11 2 1 2 1 2 1 2
23 2 3 2 3 2 3 2
31 3 1 3 1 3 1 3
4
1
1
1
= Ab null( )
b has to satisfy Ab=0 while avoiding a trivial solution, i.e., b=0.
Such b is the nullspace of A,i.e, .
The nullspace of A is can be computed by Singular Value Decomposition (SVD):
A = UDV T
b = V:,4 (the fourth or last column of V) where
Once B is estimated, it can be decomposed by Cholesky decomposition that decomposes a positive definite matrix to lower triangular matrix, i.e., where . B = LLT −=L KT 1
Calibration via 3 Vanishing Points
−
= − = − +
K K
x
y
y yx x
pf f
B Bp
B Bf f
B B Bp p-p p
f f f f
2 2
1 2-T -1
1 32 2
2 3 422
2 2 2 2
1
1
+1
( )
/
/
= −
= −
= − +
y
x
x y
p B B
p B B
Bf p p
B
3 1
2 1
2 24
1
Given B, solve for intrinsic parameters:
Or
Multiple View Calibration
Counting Argument
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
x X×∈R 3 3 tK
u
v
: knowns
# of unknowns: 5 (K) + 3 (R) + 3 (t) # of equations: 2m where m is the number of known points.
Multiple View Calibration
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
# of unknowns: 5 (K) + 3n (R) + 3n (t) # of equations: 2nm where m is the number of known points and n is the number of images.
Known 3D Points
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
x X×∈R 3 3 tK
: knowns
: unknowns
Known 3D Points
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
X Y( , ,0)
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
r r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 3
01 1
1
x X×∈R 3 3 tK
: knowns
: unknowns
Known 3D Points
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
u f s p r r t X
z v f p r r t Y
r r t
img 11 12 1
img 21 22 2
31 32 31 1 1
u
v
: knowns
X Y( , ,0)
: unknowns
Homography btw Two Planes
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
X Y( , ,0)
=
u h h h X
z v h h h Y
h h h
11 12 13
21 22 23
31 32 331 1H Xx =: estimate
: knowns
: homography between 3D plane and 2D image plane. H
Homography and Projection
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
′=
x x
y y
h h h f s p r r t
h h h z f p r r t
h h h r r t
11 12 13 11 12 1
21 22 23 21 22 2
31 32 33 31 32 31
h1
: estimate
K r1 r2 t1h2 h3
: unknowns
Orthogonality
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
′=
x x
y y
h h h f s p r r t
h h h z f p r r t
h h h r r t
11 12 13 11 12 1
21 22 23 21 22 2
31 32 33 31 32 31
h1
: estimate
K r1 r2 t1
Recall orthogonality:
h2 h3
: unknowns
r rT1 2 = 0r rT2 2 = 1r rT1 1 = 1
Orthogonality
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
′=
x x
y y
h h h f s p r r t
h h h z f p r r t
h h h r r t
11 12 13 11 12 1
21 22 23 21 22 2
31 32 33 31 32 31
h1
: estimate
K r1 r2 t1
Recall orthogonality: r rT1 1 = 1 r rT2 2 = 1 r rT1 2 = 0
h2 h3
: unknowns
where and ′=h rK z11
-1 ′=h rK z1
2-
2
=h h hK K K K hT T1 1 2
-T -1 -T -12 =K Kh h-T -1T
1 2 0 2 equations and 5 unknowns
Substitution
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
−
= − + − − =
− − +
K K
y x y
x x y x y
y x y y
x y x y y x y y
y x y y x y y y x y y
x y x y y x y y
p s - p fsf f f f f
B B Bs p s - p f ps s
B B Bf f f f f f f f
B B Bp s - p f s p s - p f p p s - p f p
f f f f f f f f
2 2 2
11 12 132-T -1
21 22 232 2 2 2 2 2 2
31 32 332 2
2 2 2 2 2 2 2
1
( )1
( ) ( )+1
=h h hK K K K hT T1 1 2
-T -1 -T -12 =K Kh h-T -1T
1 2 0
=K K-T -1 : symmetric matrix
=h hBT1 2 0=h h h hB BT T1 1 2 2 2 equations and linear in B.
Linear Equations
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
[ ]=b B B B B B BT
11 12 22 13 23 33
=h h vB bi j ijT T
where
= + + + v ij i j i j i j i j i j i j i j i j i jh h h h h h h h h h h h h h h h h hT
1 1 1 2 2 1 2 2 3 1 1 3 3 2 2 3 3 3
=h h h hB BT T1 1 2 2 =h hBT1 2 0
Rewrite equation by vectorizing B.
=
B
B B B
B B B
B B B
11 12 13
21 22 23
31 32 33
Linear equation in B
−
v v= 0
vb
T T11 22
T12v b =T
12 0=h hBT1 2 0
=h h h hB BT T1 1 2 2 v = vb bT T
11 22: 2 equations per view
Linear Equations
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
[ ]=b B B B B B BT
11 12 22 13 23 33
=h h vB bi j ijT T
where
= + + + v ij i j i j i j i j i j i j i j i j i jh h h h h h h h h h h h h h h h h hT
1 1 1 2 2 1 2 2 3 1 1 3 3 2 2 3 3 3
=h h h hB BT T1 1 2 2 =h hBT1 2 0
Rewrite equation by vectorizing B.
=
B
B B B
B B B
B B B
11 12 13
21 22 23
31 32 33
Linear equation in B
v b =T12 0=h hBT1 2 0
=h h h hB BT T1 1 2 2 v = vb bT T
11 22 −
−
v v
v= 0
v v
b
v
n n
n
1 T 1 T11 221 T
12
T T11 22
T12
Minimum configuration: • 3 images for general K. • 2images when w=0.
Solving for Intrinsic Parameters
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
−
= − + − − =
− − +
K K
y x y
x x y x y
y x y y
x y x y y x y y
y x y y x y y y x y y
x y x y y x y y
p s - p fsf f f f f
B B Bs p s - p f ps s
B B Bf f f f f f f f
B B Bp s - p f s p s - p f p p s - p f p
f f f f f f f f
2 2 2
11 12 132-T -1
21 22 232 2 2 2 2 2 2
31 32 332 2
2 2 2 2 2 2 2
1
( )1
( ) ( )+1
Recall
/
/
= − −
= − + −
=
y
y
x
p B B B B B B B
c B B p B B B B B
f c B
212 13 11 23 11 22 12
233 13 12 13 11 23 11
11
( ) ( )
[ ( )] /
/
/
/ /
= −
= −
= −
y
x y
x y y x
f cB B B B
s B f f c
p sp f B f c
211 11 22 12
212
213
( )
Given B, solve for intrinsic parameters:
Solving for Extrinsic Parameters
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
−
−
=′
=′
r
r
K h
K h
z
z
11
12
1
2
1
1
−′ = K hz 11
where
[ ] [ ]′ r r rh h = th Kz 13 22 31
−
=
=
×
′
r r
K
r
t hz
3 1 2
13
1
Summary of Linear Parameter Estimation
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
X Y( , ,0)
• Reduce dimensionality of a 3D point to 2D assuming the point lies on a plane.
• Estimate a homography between 3D plane and the 2D projected image.
• Constrain the estimated homography using orthogonality of rotation matrix
• Substitute where B is a symmetric matrix.
• Formulate a system of linear equations in B.
• Compute the intrinsic and extrinsic parameters.
r r r rT T1 1 2 2= =h h hK K K K hT T
1 1 2-T -1 -T -1
2
=K Kh h-T -1T1 2 0r rT1 2 = 0
B = K K-T -1
−
v v= 0
vb
T T11 22
T12
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
x ij
x̂ K R t Xij i i j( , , , )
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
, ,ˆ
= =
−∑∑K R t
x x K R t XI J
ij ij i i ji j
2
1 1
minimize ( , , , )reprojection
measurement
x ij
x̂ K R t Xij i i j( , , , )
Lens Distortion
[ ] =
x XK R tL
1 1
Camera body configuration (extrinsic parameter)
Spatial relationship between sensor and pinhole (internal parameter)
Lens configuration (internal parameter)
Lens Distortion Radial Distortion
Lens Distortion Radial Distortion
Straight lines are not straight in the image due to lens distortion.
Lens Distortion Radial Distortion
Lens distortion correction
Radial Lens Distortion
=x xL rimg ideal( )
Image undistortion
r
Radial lens distortion model assumes that the distortion is a function of distance between image point to the principal point.
x imgx ideal
: a point measured in the image (distorted)
: a point that follows perspective projection (undistorted) ,x yp p= xr ideal
Radial Lens Distortion
rx img
x ideal
= xr ideal
where = + + +L r
2 41 2( ) 1 k r k r
Polynomial fitting of distortion
=x xL rimg ideal( )
Radial Lens Distortion
= + +
= + +x
y
u u u - p r r
v v v - p r
k
r
k
k k
2 4img ideal ideal
2 4img idea
1 2
1l id l 2ea
( )[ ]
( )[ ]
where and are measurements and and are the projected point.
u img
− = −
x x
y y
u uku - p r u - p rv vkv - p r v - p r
2 4img ideal1ideal ideal
2 4img ideal2ideal ideal
( ) ( )
( ) ( )
Mapping between an ideal point and image point (2nd order):
v img u ideal v ideal
r x imgx ideal
Linear equations in k1 and k2:
: unknowns
Radial Lens Distortion
= + +
= + +x
y
u u u - p r r
v v v - p r
k
r
k
k k
2 4img ideal ideal
2 4img idea
1 2
1l id l 2ea
( )[ ]
( )[ ]
where and are measurements and and are the projected point.
u img
− − = − −
x x
y y
m mm mx m x m
m mm my m y m
u uu - p r u - p r
v vv - p r v - p rk
ku uu - p r u - p r
v vv - p r v - p r
1 11 2 1 4img idealideal 1 ideal 1
1 11 2 1 4img idealideal 1 ideal 1
1
22 4img idealideal ideal
2 4img ideideal ideal
( ) ( )
( ) ( )
( ) ( )
( ) ( )
al
Mapping between an ideal point and image point (2nd order):
v img u ideal v ideal
r x imgx ideal
: unknowns
Linear equations in k1 and k2:
for m points
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
, ,ˆ
= =
−∑∑K R t
x x K R t XI J
ij ij i i ji j
2
1 1
minimize ( , , , )reprojection
measurement
x ij
x̂ K R t Xij i i j( , , , )
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
x ij
x̂ K R t Xij i i j( , , , )
, ,= =
−∑∑K R t,
x K R t Xm n
ij i i jk ki j
L k k1 2
2
1 2,
1 1
minimize ( , , , , , )reprojection
measurement
Geometric Error Minimization
Demo with MATLAB
http://www.vision.caltech.edu/bouguetj/calib_doc/
• Download the MATLAB calibration toolbox: http://www.vision.caltech.edu/bouguetj/calib_doc/download/index.html
• Print out a calibration pattern: http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/pattern.pdf
• Take images from multiple angles and save them in the calibration toolbox folder. • Type calib and load images.
• Type pattern dimensions and annotate corners.
• Calibrate the camera.
Lens Large Distortion
Fisheye Lens Distortion
Fisheye Lens Distortion
Fisheye Lens Model Logarithmic Distortion
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
=x xL rimg ideal ideal( )
=r c krimg ideallog(1+ )
rideal x img
x ideal
rimg
rL r
rimg
ideal
( ) =
Fisheye have a high resolution at the fovea, and non linearly decreasing resolution towards the periphery, i.e.,
∆rideal
∆rimg
= =
∆ ∆>
∆ ∆r r r r
r rr r
ideal 1 ideal 2
ideal ideal
img img
>r r1 2for
=x xc kr
rideal
img idealideal
log(1+ )
where
Fisheye Lens Model Field of View (FOV) Distortion
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
∝ =r r kimg ideal( ) θ θ
=
=
_
_
r kr kw /
r
r w
img
img max
img
img max
2
tan
tan( / 2)
θ
θ
z
C
X
x ideal x img
ridealrimg
,x yp p
θ=
wmax2θ
_rimg max
=
=
wr r
wr w
rw
img ideal
imgideal
1arctan(2 tan )
2tan( )
2tan( / 2)
rideal x img
x ideal
rimgSpatial resolution is proportional to angular resolution, i.e.,
=x xL rimg ideal ideal( )r
L rrimg
ideal
( ) =
=_
kwrimg max 2
Spatial resolution:
where w is angular field of view.
Fisheye Lens Model Field of View (FOV) Distortion
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
z
C
X
x ideal x img
ridealrimg
,x yp p
θ=
wmax2θ
_rimg max
=x xw
rwrimg ideal ideal
ideal
1arctan(2 tan )
2
rideal x img
x ideal
rimg=x xL rimg ideal ideal( )
rL r
rimg
ideal
( ) =
where w is angular field of view.
Distortion Correction Procedure
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
1. Detect edges from an image.
Distortion Correction Procedure
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
1. Detect edges from an image.
2. Link edges that are likely to belong to the same edge.
3. Select distortion model and initialize the distortion parameters with a reasonable value.
4. Fit lines with the linked edges with least squares.
5. Optimize the distortion parameters to fit the edges to the ideal lines.
Detected edge segment
Ideal edge