Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both...

19
Camera calibration Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/4/6 with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes Announcements Project #1 artifacts voting. Project #2 camera. Outline Nonlinear least square methods Camera projection models Camera calibration Bundle adjustment Nonlinear least square methods

Transcript of Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both...

Page 1: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Camera calibration

Digital Visual Effects, Spring 2005

Yung-Yu Chuang

2005/4/6

with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes

Announcements

• Project #1 artifacts voting.

• Project #2 camera.

Outline

• Nonlinear least square methods

• Camera projection models

• Camera calibration

• Bundle adjustment

Nonlinear least square methods

Page 2: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Least square

It is widely seen in data fitting.

Linear least square

y

t

txxtxMty 10),()(

),()( iii txMyxf

3

210),( txtxxtxM is linear, too.

residualprediction

Nonlinear least square Function minimization

It is very hard to solve in general. Here, we only consider

a simpler problem of finding local minimum.

Least square is related to function minimization.

Page 3: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Function minimization Quadratic functions

Quadratic functions

isocontour gradient

Quadratic functions

Page 4: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Descent methods Descent direction

Steepest descent method

It has good performance in the initial stage of the

iterative process.

Steepest descent method

Page 5: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Newton’s method

It has good performance in the final stage of the iterative

process.

Hybrid method

This needs to calculate second-order derivative which

might not be available.

Line search Levenberg-Marquardt method

• LM can be thought of as a combination of

steepest descent and the Newton method.

When the current solution is far from the

correct one, the algorithm behaves like a

steepest descent method: slow, but guaranteed

to converge. When the current solution is close

to the correct solution, it becomes a Newton

method.

Page 6: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Nonlinear least square

).(ˆ with ,ˆ

Here,minimal.isdistancesquared

that thesovectorparameterbestthe

find try to,tsmeasuremenofsetaGiven

pxxx

p

x

f

T

Levenberg-Marquardt method

Levenberg-Marquardt method

Page 7: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Camera projection models

Pinhole camera

Pinhole camera model

(optical

center)

origin

principal point

Pinhole camera model

10100

000

000

~

1Z

Y

X

f

f

Z

fY

fX

y

x

Z

fYy

Z

fXx

Page 8: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Pinhole camera model

10100

0010

0001

100

00

00

~

1Z

Y

X

f

f

Z

fY

fX

y

x

Principal point offset

10100

0010

0001

100

0

0

~

1

0

0

Z

Y

X

yf

xf

Z

fY

fX

y

x

XIKx 0~

intrinsic matrix

Intrinsic matrix

• non-square pixels (digital video)

• skew

• radial distortion

100

0

0

0

0

yf

xf

K

100

0 0

0

yf

xsfa

K

Is this form of K good enough?

Camera rotation and translation

tR

Z

Y

X

Z

Y

X

33

'

'

'

1100

0

0

~

1

0

0

Z

Y

X

yf

xf

y

x

tR

XtRKx ~

extrinsic matrix

Page 9: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Two kinds of parameters

• internal or intrinsic parameters such as focal

length, optical center, aspect ratio:

what kind of camera?

• external or extrinsic (pose) parameters

including rotation and translation:

where is the camera?

Other projection models

Orthographic projection

• Special case of perspective projection

– Distance from the COP to the PP is infinite

– Also called “parallel projection”: (x, y, z) (x, y)

Image World

Other types of projection

• Scaled orthographic

– Also called “weak perspective”

• Affine projection

– Also called “paraperspective”

Page 10: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Fun with perspective Perspective cues

Perspective cues Fun with perspective

Ames room

Page 11: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Forced perspective in LOTR

Camera calibration

Camera calibration

• Estimate both intrinsic and extrinsic parameters

• Mainly, two categories:

1. Photometric calibration: use reference objects

with known geometry

2. Self calibration: only assume static scene, e.g.

structure from motion

Camera calibration approaches

1. linear regression (least squares)

2. nonlinear optinization

3. multiple planar patterns

Page 12: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Chromaglyphs (HP research) Linear regression

MXXtRKx ~

Linear regression

• Directly estimate 11 unknowns in the M matrix

using known 3D points (Xi,Yi,Zi) and measured

feature positions (ui,vi)

Linear regression

Solve for Projection Matrix M using least-square

techniques

Page 13: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Normal equation

Given an overdetermined system

bAx

bAAxATT

the normal equation is that which minimizes the

sum of the square differences between left and

right sides

Linear regression

• Advantages:

– All specifics of the camera summarized in one matrix

– Can predict where any world point will map to in the

image

• Disadvantages:

– Doesn’t tell us about particular parameters

– Mixes up internal and external parameters

• pose specific: move the camera and everything breaks

Nonlinear optimization

• Feature measurement equations

• Likelihood of M given {(ui,vi)}

Optimal estimation

• Log likelihood of M given {(ui,vi)}

• How do we minimize C?

• Non-linear regression (least squares), because

ûi and vi are non-linear functions of M

• We can use Levenberg-Marquardt method to

minimize it

Page 14: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

Advantage• Only requires a plane

• Don’t have to know positions/orientations

• Good code available online!

– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/

– Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html

– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/

Step 1: data acquisition

Step 2: specify corner order Step 3: corner extraction

Page 15: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Step 3: corner extraction Step 4: minimize projection error

Step 4: camera calibration Step 4: camera calibration

Page 16: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Step 5: refinement

Bundle adjustment

Bundle adjustment

• Bundle adjustment (BA) is a technique for

simultaneously refining the 3D structure and

camera parameters

• It is capable of obtaining an optimal

reconstruction under certain assumptions on

image error models. For zero-mean Gaussian

image errors, BA is the maximum likelihood

estimator.

Bundle adjustment

• n 3D points are seen in m views

• xij is the projection of the i-th point on image j

• aj is the parameters for the j-th camera

• bi is the parameters for the i-th point

• BA attempts to minimize the projection error

Euclidean distance

predicted projection

Page 17: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Bundle adjustment

Bundle adjustment

3 views and 4 points

Typical Jacobian

Page 18: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Bundle adjustment Block structure of normal equation

Bundle adjustment

Multiplied by • Parameterise each camera by rotation and

focal length

• This gives pairwise homographies

Recognising panoramas

Page 19: Camera calibration - 國立臺灣大學 · Camera calibration Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration:

Error function

• Sum of squared projection errors

– n = #images

– I(i) = set of image matches to image i

– F(i, j) = set of feature matches between images i,j

– rijk = residual of kth feature match between images

i,j

• Robust error function

A sparse BA software using LM

• sba is a generic C implementation for bundle

adjustment using Levenberg-Marquardt method.

It is available at

http://www.ics.forth.gr/~lourakis/sba.

• You can use this library for your project #2.

MatchMove Reference

• Manolis Lourakis and Antonis Argyros, The Design and

Implementation of a Generic Sparse Bundle Adjustment Software

Package Based on the Levenberg-Marquardt Algorithm, FORTH-

ICS/TR-320 2004.

• K. Madsen, H.B. Nielsen, O. Timgleff, Methods for Non-Linear Least

Squares Problems, 2004.

• Zhengyou Zhang, A Flexible New Techniques for Camera

Calibration, MSR-TR-98-71, 1998.

• Bill Triggs, Philip McLauchlan, Richard Hartley and Andrew

Fitzgibbon, Bundle Adjustment - A Modern Symthesis, Proceedings

of the International Workshop on Vision Algorithms: Theory and

Practice, pp298-372, 1999.