(Geometric) Camera Calibration - Purdue University(Geometric) Camera Calibration CS635 Spring 2010...
Transcript of (Geometric) Camera Calibration - Purdue University(Geometric) Camera Calibration CS635 Spring 2010...
(Geometric) Camera Calibration(Geometric) Camera Calibration
CS635 Spring 2010
Daniel G AliagaDaniel G. AliagaDepartment of Computer Science
Purdue University
Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration
• Digital Cameras and CCDsDigital Cameras and CCDs
• Aberrations• Aberrations
P ti P j ti• Perspective Projection
• Calibration
Digital Camera vs. “Film” CameraDigital Camera vs. “Film” Camera
• Charge‐Coupled Device (CCD)
Digital Camera vs. Film CameraDigital Camera vs. Film Camera
– Image plane is a CCD array instead of film– CCD arrays are typically ¼ or ½ inch in size– CCD arrays have a pixel resolution (e.g., 640x480, 1024x1024)y p ( g , , )– CCD Cameras have a maximum “frame rate”, usually determined by
the hardware and bandwidth
• Number of CCDs– 3: each CCD captures only R, G, or B wavelengths– 1: the single CCD captures RGB simultaneously, reducing the
resolution by 1/3 (kinda)y / ( )
• Video– Interlaced: only “half” of the horizontal lines of pixels are present in
each frameeach frame– Progressive scan: each frame has a full‐set of pixels
The simplest 1The simplest 1‐‐CCD CCD camera in towncamera in town
image plane
objectCCD array
object
ExposuresExposuresExposuresExposures
apertureimage plane
objectCCD array
object
“effective” shutter ( ft j t t(e.g., often just turn
CCD on/off)
ExposuresExposuresExposuresExposures
• An “exposure” is when the CCD is exposed to theAn exposure is when the CCD is exposed to the scene, typically for a brief amount of time and with a particular set of camera parameters
• The characteristics of an “exposure” are determined pby multiple factors, in particular:– Camera aperture
• Determines amount of light that shines onto CCD
– Camera shutter speed• Determines time during which aperture is “open” and light shinesDetermines time during which aperture is open and light shines on CCD
Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration
• Digital Cameras and CCDsDigital Cameras and CCDs
• Aberrations• Aberrations
P ti P j ti• Perspective Projection
• Calibration
AberrationsAberrationsAberrationsAberrations
• A “real” lens system does not produce a perfectA real lens system does not produce a perfect image
• Aberrations are caused by imperfect manufacturing and by our approximate modelsy pp– Lenses typically have a spherical surface
• Aspherical lenses would better compensate for refraction but are more difficult to manufacturemore difficult to manufacture
– Typically 1st order approximations are used• Remember sin Ω = Ω ‐Ω3/3! + Ω5/5! ‐ …
• Thus, thin‐lens equations only valid iff sin Ω ≈ Ω
AberrationsAberrationsAberrationsAberrations
• Most common aberrations:Most common aberrations:– Spherical aberration
Coma– Coma
– Astigmatism
C t f fi ld– Curvature of field
– Chromatic aberration
Di i– Distortion
Spherical AberrationSpherical Aberration
D t i t i l i
Spherical AberrationSpherical Aberration
• Deteriorates axial image
ComaComaComaComa
• Deteriorates off‐axial bundles of raysDeteriorates off axial bundles of rays
Astigmatism and Curvature of FieldAstigmatism and Curvature of Field
• Produces multiple (two) images of a single
Astigmatism and Curvature of FieldAstigmatism and Curvature of Field
Produces multiple (two) images of a single object point
Chromatic AberrationChromatic AberrationChromatic AberrationChromatic Aberration
• Caused by wavelength dependent refractionCaused by wavelength dependent refraction– Apochromatic lenses (e.g., RGB) can help
DistortionDistortion
• Radial (and tangential) image distortions
Example radial distortions
Radial DistortionRadial DistortionRadial DistortionRadial Distortion
• (x, y) pixel before distortion correction( , y) p
• (x’, y’) pixel after distortion correction
• Let r = (x2 + y2) ‐1
• Then– x’ = x(1 ‐ Δr/r)x x(1 Δr/r)– y’ = y(1 ‐ Δr/r)– where Δr = k0r + k1r3 + k2r5 + …
• Finally,– x’ = x(1 – k0 – k1r2 – k2r4 ‐ …)x x(1 k0 k1r k2r …)
– y’ = y(1 – k0 – k1r2 – k2r4 ‐ …)
Radial DistortionRadial DistortionRadial DistortionRadial Distortion
before after
Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration
• Digital Cameras and CCDsDigital Cameras and CCDs
• Aberrations• Aberrations
P ti P j ti• Perspective Projection
• Calibration
Perspective ProjectionPerspective ProjectionPerspective ProjectionPerspective Projection
zZ
(X, Y, Z)fzZ
/ i i t
(x, y)
optical axis
Yy?
image planeeye/viewpoint optical axis
f Zx = f X
Zy = f Y
Zy = Y &
Perspective ProjectionPerspective ProjectionPerspective ProjectionPerspective Projection
Z
image plane
f
Z
(X, Y, Z)plane
bj t(x, y) object(x, y)
x = f X y = f Yx = f XZ
y = f YZ
Camera CalibrationCamera CalibrationCamera CalibrationCamera Calibration
• Digital Cameras and CCDsDigital Cameras and CCDs
• Aberrations• Aberrations
P ti P j ti• Perspective Projection
• Calibration
Tsai Camera Model and CalibrationTsai Camera Model and CalibrationTsai Camera Model and CalibrationTsai Camera Model and Calibration
• A widely used camera model to calibrateA widely used camera model to calibrate conventional cameras based on a pinhole cameracamera
• Reference• Reference– “A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology UsingHigh‐Accuracy 3D Machine Vision Metrology Using Off‐the‐Shelf TV Cameras and Lenses”, Roger Y. Tsai, IEEE Journal of Robotics and Automation,Tsai, IEEE Journal of Robotics and Automation, Vol. 3, No. 4, August 1987
Calibration GoalCalibration GoalCalibration GoalCalibration Goal
• Determine the intrinsic (and extrinsic)Determine the intrinsic (and extrinsic) parameters of a camera (with lens)
22
Camera ParametersCamera ParametersCamera ParametersCamera Parameters
• Intrinsic/InternalIntrinsic/Internal– Focal length
l ( )
f– Principal point (center)
– Pixel size yx pp ,
yx ss ,
– (Distortion coefficients)
• Extrinsic/External
,...1k
Extrinsic/External– Rotation
Translation
ψϕφ ,,ttt
23
– Translation zyx ttt ,,
Focal LengthFocal LengthFocal LengthFocal Length
⎟⎞
⎜⎛⎤⎡
⎟⎞
⎜⎛ 000
XffX
⎞⎛⎞⎛
⎟⎟⎟⎟⎟
⎠⎜⎜⎜⎜⎜
⎝⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
10100000000
ZY
ff
ZfYfX
⎟⎟⎠
⎞⎜⎜⎝
⎛ZfYZfX
//
=⎟⎟⎠
⎞⎜⎜⎝
⎛yx
⎟⎠
⎜⎝⎦⎣⎠⎝ 1
Principal PointPrincipal PointPrincipal PointPrincipal Point
⎟⎟⎞
⎜⎜⎛
⎥⎤
⎢⎡
⎟⎞
⎜⎛ + 00
YX
pfZpfX xx⎞⎛
⎟⎟⎟⎟
⎠⎜⎜⎜⎜
⎝⎥⎥⎥
⎦⎢⎢⎢
⎣
=⎟⎟⎟
⎠⎜⎜⎜
⎝
+
1010000
ZY
pfZ
ZpfY yy⎟⎟⎠
⎞⎜⎜⎝
⎛
0
0
yx
⎠⎝
CCD Camera: Pixel SizeCCD Camera: Pixel SizeCCD Camera: Pixel SizeCCD Camera: Pixel Size
⎤⎡⎤⎡ 0000 pfs
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
01000000
1000000
y
x
y
x
pfpf
ss
K
⎦⎣⎦⎣
⎥⎥⎤
⎢⎢⎡
= 0000 xx
pp
K αα
(intrinsic) calibration matrix⎥⎥⎦⎢
⎢⎣ 0100
00 yy pK α (intrinsic) calibration matrix
Translation & RotationTranslation & RotationTranslation & RotationTranslation & Rotation
⎟⎟⎞
⎜⎜⎛⎤⎡ tR Y
X)~(~ CXRxcam −= ψϕφ RRRR=
⎟⎟⎟⎟
⎠⎜⎜⎜⎜
⎝
⎥⎦
⎤⎢⎣
⎡=
110tR~
ZY
xcamRCXRxcam −=~~
t−
3x3 rotation matrices
[ ]Tzyx tttt=⎠⎝ 1 [ ]zyxtranslation vector
(extrinsic) calibration matrix P
Calibration TaskCalibration TaskCalibration TaskCalibration Task
points in space
observation
physical arrangement (calibration pad)
(camera with initial parameters)
calibration result(camera with calibrated
? ?
( p ) (parameters)
~~ KG PWhat is ? ?ii x~X ↔ KGiven P
A Linear FormulationA Linear FormulationA Linear FormulationA Linear Formulation
KPM =Let KPM =
XMxcam~~ =
⎟⎞
⎜⎛⎤⎡⎞⎛ '
X
Let
⎟⎟⎟⎟⎟⎞
⎜⎜⎜⎜⎜⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
'''
34333231
24232221
14131211
ZY
mmmmmmmmmmmm
wyx
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛'/''/'
wywx
yx
⎟⎟⎠
⎜⎜⎝⎥⎦⎢⎣⎠⎝ 134333231 mmmmw
)~/()~( 31 XmXmx ⋅⋅= )()( 31
)~/()~( 32 XmXmy ⋅⋅=
A Linear FormulationA Linear FormulationA Linear FormulationA Linear Formulation
for observationsni ...1=)~/()~( 31 XmXmx ⋅⋅=o obse a o s
0~)( X
)~/()~( 32 XmXmy ⋅⋅=
0)( 31 =⋅− ii Xmxm 2n homogeneous linear equations and 12 unknowns (coefficients of M)0~)( 32 =⋅− ii Xmym
Thus, given n≥6 can solve for M; namely 0=Qm
⎥⎤
⎢⎡ − TTT XxX ~0~
111
⎥⎥⎥⎥⎥
⎢⎢⎢⎢⎢
−=
TTT
TTT XyXQ
~~......
~~0 111
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛= 2
1
mm
m
⎥⎥
⎦⎢⎢
⎣ −−
Tnn
TT
Tnn
TTn
XynXXxX~~0
0⎟⎠
⎜⎝ 3m
Decomposing M into Camera Decomposing M into Camera ParametersParameters
[ ] [ ]tRKbAM == ρ
⎥⎥⎤
⎢⎢⎡
++−
=⎥⎥⎤
⎢⎢⎡
= Ty
Ty
Tx
TTx
T
T
rprrprr
aa
A 32
321
2
1
)sin/()cot(
θαθαα
ρ⎥⎥
⎦⎢⎢
⎣⎥⎥
⎦⎢⎢
⎣T
yyT r
pa 3
32
3
2 )(ρ
…function of yxyx pp ,,,αα and skew θyy
(often we assume skew is non-existent, which means )2/πθ =
Decomposing M into Camera Decomposing M into Camera ParametersParameters
2
233
3
)(
/ar
a=
=
ρ
ερ
aa
aa
y
x
322
312
sin
sin
×=
×=
θρα
θρα
322
312
)()(
)(
)(
aap
aap
y
x
⋅=
⋅=
ρ
ρ
aaaar
32
321
×
××
=
3231
3231 )()(cosaaaaaaaa
×××⋅×
−=θbKt
rrr1
132−=
×=
ρ
⎥⎤
⎢⎡
=⎥⎥⎤
⎢⎢⎡
= ,0000
tRPp
pK yy
xx
αα
⎥⎦
⎢⎣⎥
⎥⎦⎢
⎢⎣
10,
0100pyy
A Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationA Nonlinear Formulation
• Bundle AdjustmentBundle Adjustment– Given initial guesses, use nonlinear least squares to refine/compute the calibration parametersto refine/compute the calibration parameters
– Simple but good convergence depends on accuracy of initial guessaccuracy of initial guess
A Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationA Nonlinear Formulation
)~/()~( XXRecall
)/()( 31 XmXmx ⋅⋅=
)~/()~( 32 XmXmy ⋅⋅=
∑⎥⎥⎦
⎤
⎢⎢⎣
⎡
⋅
⋅−+
⋅
⋅−=
ij ji
jiij
ji
jiij Xm
Xmy
XmXm
xmn
E 2
3
22
3
1 )~~
()~~
(1
Goal is 0→E
A Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationA Nonlinear FormulationOption A:
Define as a matrix of 11 unknowns (i.e., ) M 134 =mAnd solve for ijm
Can be made very efficient, especially for sparse matrices
Option B:
Define as function of intrinsic and extrinsic parameters so that it is “recomputed” during each loop
Mparameters so that it is recomputed during each loop of the optimization