Camera Calibration - Inspiring InnovationCamera Calibration ! Goal: build the geometric relations...
Transcript of Camera Calibration - Inspiring InnovationCamera Calibration ! Goal: build the geometric relations...
Jesus J Caban
Camera Calibration
Schedule ! Today:
! Camera calibration
! Wednesday: ! Lecture: Motion & Optical Flow
! Monday: ! Lecture: Medical Imaging
! Final presentations: ! Nov 29th: W. Griffin ! Dec 1st: F. Zafar, Y. Wang, and N. Chhaya ! Dec 6th: J. Dandois ! Dec 8th: ! Dec 13th: ! Dec 20th:
+5 bonus points +3 bonus points
+2 bonus points
+1 bonus points
+0 bonus points +0 bonus points
Note: You have until next Monday to let me know…
Camera Calibration: Motivation ! Images are obtained from 3D scenes
! The exact position and orientation of the camera sensing device are often unknown
! Camera needs to be related to some global frame / reference to be able to: ! Get accurate measurements ! Inspect materials ! Etc…
Camera Model ! Camera model:
! from world to camera (coordinate systems) ! from camera (3D) to image frame (2D)
Zw
Xw
Yw
Pw
O
P
x y
p
xim
yim
(xim,yim)
Camera Calibration ! Goal: build the geometric relations between a 3D scene and
its 2D images. ! 3D transformation that represents the viewpoints and viewing
directions of the camera ! The perspective projection that maps 3D points into 2D images
Camera Model: Parameters
! Camera Parameters ! Intrinsic Parameters (of the camera): link the frame coordinates of an image point
with its corresponding camera coordinates
! Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system
Zw
Xw
Yw
Pw
O
P
x y
p
xim
yim
(xim,yim)
2D Rotation
! Rotation by angle ! (about the origin)
!"#"$% !"#"$%
&'()%*+%)',%*-.,/+,0%12/%/2)(32-+4%
!%
Scaling
! Scaling operation:
! Or, in matrix form:
!
x 'y '"
# $ %
& ' =
a 00 b"
# $
%
& ' xy"
# $ %
& '
scaling matrix S
Scaling ! Scaling a coordinate means multiplying each of its components by a
scalar ! Uniform scaling means this scalar is the same for all components:
" 2
Scaling
! Uniform scaling by s:
!"#"$% !"#"$%
! Non-uniform scaling: different scalars per component:
Non-uniform Scaling
X " 2, Y " 0.5
2x2 Matrices
! What types of transformations can be represented with a 2x2 matrix?
2D Mirror about Y axis?
2D Mirror over (0,0)?
All 2D Linear Transformations ! Linear transformations are combinations of …
! Scale, ! Rotation, ! Shear, and ! Mirror
! Properties of linear transformations: ! Origin maps to origin ! Lines map to lines ! Parallel lines remain parallel ! Ratios are preserved ! Closed under composition
! Translation is not a liner operation on 2D coordinates ! Homogeneous coordinates
56%7/(-+8(32-0%
What about 2D Translation?
Homogeneous coordinates
7/*9:4%%(;;%2-,%<2/,%922/;*-(),4%
'2<2=,-,2>+%*<(=,%%922/;*-(),+%
?2-.,/3-=%!"#$%'2<2=,-,2>+%922/;*-(),+%
@%
A%
B%
!%#%&#%'$%
'%C%D% !%E'#%&E'#%($%
'2<2=,-,2>+%F8(-,%
Homogeneous Coordinates ! Add a 3rd coordinate to every 2D point
! (x, y, w) represents a point at location (x/w, y/w) ! (x, y, 0) represents a point at infinity ! (0, 0, 0) is not allowed
Convenient coordinate system to represent many useful transformations 1 2
1
2 (2,1,1) or (4,2,2) or (6,3,3)
x
y
2D Translation?
! Solution: homogeneous coordinates to the rescue
Translation ! Example of translation
!
x 'y '1
"
#
$ $ $
%
&
' ' '
=
1 0 tx0 1 ty0 0 1
"
#
$ $ $
%
&
' ' '
xy1
"
#
$ $ $
%
&
' ' '
tx = 2 ty = 1
Homogeneous Coordinates
!
=
x + txy + ty1
"
#
$ $ $
%
&
' ' '
Affine transformations
(-A%)/(-+G2/<(32-%B*)'%%8(+)%/2B%H%"%"%D%I%B,%9(88%(-%%)*+,%)/(-+G2/<(32--
Basic affine transformations
Translate
2D in-plane rotation Shear
Scale
3D Rotations
!
Z" =
cos" #sin" 0sin" cos" 00 0 1
$
%
& & &
'
(
) ) )
!
X" =
1 0 00 cos" #sin"0 sin" cos"
$
%
& & &
'
(
) ) )
!
Y" =
cos" 0 sin"0 1 0
#sin" 0 cos"
$
%
& & &
'
(
) ) )
Where do we go from here?
(J-,%)/(-+G2/<(32-%
B'()%'(FF,-+%B',-%B,%9'(-=,%)'*+%/2B0-
Projective Transformations
?(88,;%(%.#$#/")0.&%%!2/%01)+)"-0,"20,345,-$)0$-
Image warping with homographies
image plane in front image plane below black area where no pixel maps to
Camera Calibration
Camera Model: Parameters
! Camera Parameters ! Intrinsic Parameters (of the camera): link the frame coordinates of an image point
with its corresponding camera coordinates
! Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system
Zw
Xw
Yw
Pw
O
P
x y
p
xim
yim
(xim,yim)
Camera coordinate system
• Principal axis: line from the camera center perpendicular to the image plane
• Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system)
Perspective Projection Equations (X-Z Plane) X
x
Z f
!
"xf
=XZ
!
"x = fXZ
Notice: Position on the image plane is related to depth. (u,v) are scaled equally based on focal length and depth
Equivalent Image Geometries ! Consider case with object on optical axis
! More convenient geometry, with an upright image
! Both are equivalent mathematically
!
x = fXZ
!
(X,Y,Z)! ( f XZ, f YZ)
!
XYZ1
"
#
$ $ $ $
%
&
' ' ' '
!f Xf YZ
"
#
$ $ $
%
&
' ' '
Pinhole camera model
!
XYZ1
"
#
$ $ $ $
%
&
' ' ' '
!
f 0f 01 0
"
#
$ $ $
%
&
' ' '
=
Principal point offset
• Camera coordinate system: origin is at the principal point • Image coordinate system: origin is in the corner
principal point:
!
(X,Y,Z)! ( f XZ
+ px, fYZ
+ py )
Principal point offset
principal point:
Pixel size
! mx pixels per meter in horizontal direction, my pixels per meter in vertical direction
Pixel size:
pixels/m m pixels
Skew
The camera coordinate system may be skewed due to some manufacturing error.
Lens distortion ! Two main lens distortion:
! Radial distortion: arise as a result of the shape of lens (e.g. fish-eye) ! Tangential distortion: arise from the assembly process of the camera
as a whole.
Radial Distortion ! Zero at the optical center of the image ! Increases as we move toward the periphery
! Rays farther from the center of the lens are bent more than those closer in
! Terms: k1, k2, k3
Camera parameters
• Intrinsic parameters ! Principal point coordinates ! Focal length ! Pixel magnification factors ! Skew (non-rectangular pixels) ! Radial distortion
• Extrinsic parameters ! Rotation and translation relative to world coordinate system
!
˜ X cam = R ˜ X - ˜ C ( )
Camera rotation and translation
• In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation
coords. of point in camera frame
coords. of camera center in world frame
coords. of a point in world frame (nonhomogeneous)
Camera parameters
• Intrinsic parameters ! Principal point coordinates ! Focal length ! Pixel magnification factors ! Skew (non-rectangular pixels) ! Radial distortion
• Extrinsic parameters ! Rotation and translation relative to world coordinate system
Camera Calibration
!
G = PLRT =
1 0 0 00 1 0 00 0 1 00 0 1/ f 0
"
#
$ $ $ $
%
&
' ' ' '
1 0 0 t10 1 0 t20 0 1 t30 0 0 1
"
#
$ $ $ $
%
&
' ' ' '
R11 R12 R13 0R21 R22 R23 0R31 R32 R33 00 0 0 1
"
#
$ $ $ $
%
&
' ' ' '
1 0 0 T10 1 0 T20 0 1 T30 0 0 1
"
#
$ $ $ $
%
&
' ' ' '
Intrinsic Parameters Extrinsic Parameters
Basic matrix:
How to estimate those parameters?
Camera calibration • Given n points with known 3D coordinates Xi and known
image projections xi, estimate the camera parameters
Xi
xi
44
Example Calibration Pattern
From Sebastian Thrun
Harris Corner Detector
Camera calibration
46
Calibration: 2 steps ! Step 1: Transform into camera coordinates
! Step 2: Transform into image coordinates
!
˜ X C
˜ Y C
˜ Z C
"
#
$ $ $
%
&
' ' '
= f (XW
Y W
ZW
"
#
$ $ $
%
&
' ' ' ,(,),*,T)
Camera Calibration ! Algorithm
1. Measure N 3D coordinates (Xi, Yi,Zi) 2. Locate their corresponding image points (xi,yi) -
Edge, Corner, Hough 3. Build matrix A of a homogeneous system Av = 0 4. Compute SVD of A , solution v 5. Determine aspect ratio # and scale |$| 6. Recover the first two rows of R and the first two
components of T up to a sign 7. Determine sign s of $ by checking the projection
equation 8. Compute the 3rd row of R by vector product, and
enforce orthogonality constraint by SVD 9. Solve Tz and fx using Least Square and SVD, then fy
= fx / #%
Yw
Xw
Zw
50
How Many Images Do We Need? ! Assumption: K images with M
corners each ! 4+6K parameters ! 2KM constraints ! 2KM & 4+6K ' M>3 and K &2/(M-3)
! 2 images with 4 points, but will 1 images with 5 points work? ! No, since points cannot be co-planar!
Two-view geometry • Scene geometry (structure): Given corresponding
points in two or more images, where is the pre-image of these points in 3D?
• Correspondence (stereo matching): Given a point in just one image, how does it constrain the position of the corresponding point x’ in another image?
• Camera geometry (motion): Given a set of corresponding points in two images, what are the cameras for the two views?
Bahadir K. Gunturk
Parameters of a Stereo System
! Intrinsic Parameters ! Characterize the transformation
from camera to pixel coordinate systems of each camera
! Focal length, image center, aspect ratio
! Extrinsic parameters ! Describe the relative position and
orientation of the two cameras ! Rotation matrix R and translation
vector T
p l p
r
P
Ol Or
Xl
Xr
Pl Pr
fl fr
Zl
Yl
Zr
Yr
R, T
Triangulation • Given projections of a 3D point in two or more images (with
known camera matrices), find the coordinates of the point
O1 O2
x1 x2
X?
Triangulation • We want to intersect the two visual rays corresponding to x1
and x2, but because of noise and numerical errors, they don’t meet exactly
O1 O2
x1 x2
X?
R1 R2
Triangulation: Geometric approach • Find shortest segment connecting the two viewing rays and
let X be the midpoint of that segment
O1 O2
x1 x2
X
• Epipolar Plane – plane containing baseline (1D family)
• Epipoles = intersections of baseline with image planes = projections of the other camera center = vanishing points of camera motion direction
• Epipolar Lines - intersections of epipolar plane with image planes (always come in corresponding pairs)
• Baseline – line connecting the two camera centers
Epipolar geometry X
x x’
Epipolar constraint
• If we observe a point x in one image, where can the corresponding point x’ be in the other image?
x x’
X
• Potential matches for x have to lie on the corresponding epipolar line l’.
• Potential matches for x’ have to lie on the corresponding epipolar line l.
Epipolar constraint
x x’
X
x’
X
x’
X
Rectification ! Problem: Epipolar lines not parallel to scan lines
p l p r
P
Ol Or el er
Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles
From Sebastian Thrun/Jana Kosecka
Rectification ! Problem: Epipolar lines not parallel to scan lines
p l p r
P
Ol Or
Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles at infinity
Rectified Images
From Sebastian Thrun/Jana Kosecka
Stereo results
Ground truth Scene
(from Seitz)
Results with window correlation
Window-based matching (best window size)
Ground truth
(from Seitz)
Results with better method
Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.
(from Seitz)
Ground truth State of the art
How can We Improve Stereo?
Space-time stereo scanner uses unstructured light to aid in correspondence
Result: Dense 3D mesh (noisy)
From Sebastian Thrun/Jana Kosecka
Unstructured Light 3D Scanner
OpenCV ! cvFindChessboardCorners()
! Calibrate: ! cvCalibrateCamera2() ! cvStereoCalibrate()
! Rectify / Epipolar ! cvInitUndistortMap() ! cvInitUndistortRectifyMap() ! cvComputeCorrespondEpilines() ! cvFindFundamentalMatrix() ! cvStereoRectify()
! Stereo ! cvFindStereoCorrespondenceBM()
! Depth maps ! cvReprojectImageTo3D()
Assignment #4
! Download the Matlab Camera Calibration Toolbox
! Read Documentation ! First calibration example
! Download data
! Run camera calibration software
! Submit results
Conclusion ! There’s a mathematical model to describe cameras
! Parameters can be estimated using a camera calibration approach
! Camera calibration is key for accurate stereo reconstruction