Lecture 3 - Silvio Savarese 13-Jan-15

Professor Silvio Savarese Computational Vision and Geometry

Lab

Lecture 3 Camera Models 2 & Camera Calibration

Lecture 3 - Silvio Savarese 14-Jan-15

• Recap of camera models• Camera calibration problem• Camera calibration with radial distortion • Example

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Reading: [FP] Chapter 1 “Geometric Camera Calibration” [HZ] Chapter 7 “Computation of Camera Matrix P”

Lecture 3 Camera Models 2 & Camera Calibration

Pinhole perspective projectionProjective camera

f = focal length

f

Pinhole perspective projection

x

y

xc

C’=[uo, vo]

Projective camera

f = focal lengthuo, vo = offset (note a different convention w.r.t. lecture 2)

f

yc

Units: k,l [pixel/m]f [m]

[pixel],α βNon-square pixels

Projective camera

f = focal lengthuo, vo = offset

→ non-square pixels,α β

f

f

x

y

xc

yc

C’ = [uo, vo]θ

Projective camera

f = focal lengthuo, vo = offset

→ non-square pixels,α βθ = skew angle

Projective camera

!P =

α −α cotθ uo 0

0 β

sinθvo 0

0 0 1 0

&

'

(((((

)

*

+++++

xyz1

&

'

((((

)

*

++++

f = focal lengthuo, vo = offset

→ non-square pixels,α βθ = skew angle

K has 5 degrees of freedom!

f

f

Projective camera

Ow

kw

jwR,T

f = focal lengthuo, vo = offset

→ non-square pixels,α βθ = skew angleR,T = rotation, translation

wPTR

P4410×

"#

$%&

'=

f

f = focal lengthuo, vo = offset

→ non-square pixels,α β

Projective camera

wPMP =!

[ ] wPTRK=Internal parameters

External parameters

θ = skew angleR,T = rotation, translation

Ow

kw

jwR,T

P'3×1 =M Pw = K3×3 R T"#

$% 3×4

Pw4×1

=

m1

m2

m3

!

"

####

$

%

&&&&

PW =m1PWm2PWm3PW

!

"

####

$

%

&&&&

→ P'E = (m1Pwm3Pw

, m2Pwm3Pw

)E

f

Projective camera

Ow

kw

jwR,T

!!!

"

#

$$$

%

&

=

3

2

1

mmm

M

Theorem (Faugeras, 1993)

!!!

"

#

$$$

%

&

=

3

2

1

aaa

A[ ] [ ] ][ bATKRKTRKM ===

[Eq.13; lect. 2]

Exercise!

Ow

iw

kw

jwR,T

M = K R T!"

#$

= (m1Pwm3Pw

, m2Pwm3Pw

)

P

P’

f

P'E = ( f xwzw, f yw

zw)

=

f 0 0 00 f 0 00 0 1 0

!

"

###

$

%

&&&

Pw =

xwywzw1

!

"

#####

$

%

&&&&&

!

= K I 0!"

#$

Projective camera

p’q’r’ O

f’R

Q

P

Weak perspective projection

R

Q

P

O

f’

When the relative scene depth is small compared to its distance from the camera

zo

p’q’r’ Q_

P_

R_

π

Weak perspective projection

O

f’ zo

p’q’r’ Q_

P_

R_

x' = f 'zx

y' = f 'zy

!

"

###

$

###

→

x ' = f 'z0x

y' = f 'z0y

!

"

##

$

##

Magnification m

R

Q

Pπ

!"

#$%

&=

10bA

M[ ]TRKM = !"

#$%

&=

1vbA

O

f’ zo

p’q’r’ Q_

P_

R_R

Q

P

π

!

Projective (perspective) Weak perspective

Weak perspective projection

!!!

"

#

$$$

%

&

=

!!!

"

#

$$$

%

&

=

10002

1

3

2

1

mm

mmm!

!!

"

#

$$$

%

&

=

!!!

"

#

$$$

%

&

=

1PP

P W2

W1

W

3

2

1

mm

mmm

)P,P( 21 ww mm→E

wPMP =!!"

#$%

&=

10bA

M

magnification

!!!

"

#

$$$

%

&

=

3

2

1

mmm

wPMP =! !"

#$%

&=

1vbA

M!!!

"

#

$$$

%

&

=

!!!

"

#

$$$

%

&

=

W3

W2

W1

W

3

2

1

PPP

Pmmm

mmm

)PP,

PP(

3

2

3

1

w

w

w

w

mm

mm

→E

Perspective

Weak perspective

Orthographic (affine) projectionDistance from center of projection to image plane is infinite

x' = f 'zx

y' = f 'zy

!

"

###

$

###

→x ' = xy' = y

!"#

$#

Pros and Cons of These Models

• Weak perspective much simpler math. – Accurate when object is small and distant.– Most useful for recognition.

• Pinhole perspective much more accurate for scenes. – Used in structure from motion or SLAM.

The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang Hui

Weak perspective projection

Weak perspective projection

The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang Hui

Lecture 3 - Silvio Savarese 14-Jan-15

• Recap of camera models• Camera calibration problem• Camera calibration with radial distortion • Example

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Lecture 3 Camera Calibration

Reading: [FP] Chapter 1 “Geometric Camera Calibration” [HZ] Chapter 7 “Computation of Camera Matrix P”

Projective camera

!!!

"

#

$$$

%

& −

=

100

v0

ucot

K o

o

sinθβ

θαα

!!!

"

#

$$$

%

&

=T3

T2

T1

Rrrr

!!!

"

#

$$$

%

&

=

z

y

x

ttt

T

43×

wPMP =! [ ] wPTRK=Internal parameters

External parameters

Goal of calibration

wPMP =! [ ] wPTRK=

Estimate intrinsic and extrinsic parameters from 1 or multiple images

Change notation:P = Pw

p = P’!!!

"

#

$$$

%

& −

=

100

v0

ucot

K o

o

sinθβ

θαα

!!!

"

#

$$$

%

&

=T3

T2

T1

Rrrr

!!!

"

#

$$$

%

&

=

z

y

x

ttt

T

43×

Calibration Problem

•P1… Pn with known positions in [Ow,iw,jw,kw]

jC

Calibration rig

Calibration Problem

•P1… Pn with known positions in [Ow,iw,jw,kw]•p1, … pn known positions in the image

jC

Calibration rig

image

pi

Goal: compute intrinsic and extrinsic parameters

Calibration Problem

jC

Calibration rig

How many correspondences do we need?• M has 11 unknown • We need 11 equations • 6 correspondences would do it

image

pi

Calibration Problem

imagejC

Calibration rig

In practice, using more than 6 correspondences enables more robust results

pi

Calibration Problem

jC

ii PMP → !"

#$%

&=→

i

ii v

up

!!!

"

#

$$$

%

&

=

3

2

1

Mmmm

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

mmmm

in pixels [Eq. 1]

Calibration Problem

i3

i1i P

Pumm

=

i2i3i P)P(v mm =→

i1i3i P)P(u mm =→

i3

i2i P

Pvmm

=

!"

#$%

&

i

i

vu

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

mmmm

0P)P(v 23 =−→ iii mm

→ ui (m3 Pi )−m1 Pi = 0

[Eq. 1]

[Eqs. 2]

Calibration Problem

……

0)( 12131 =− PPv mm

0)( 11131 =− PPu mm

0)( 23 =− iii PPv mm

0)( 13 =− iii PPu mm

0)( 23 =− nnn PPv mm

0)( 13 =− nnn PPu mm

[Eqs. 3]

Block Matrix Multiplication

!"

#$%

&=!

"

#$%

&=

2221

1211

2221

1211

BBBB

BAAAA

A

What is AB ?

!"

#$%

&

++

++=

2222122121221121

2212121121121111

BABABABABABABABA

AB

!!!

"

#

$$$

%

&

=T3

T2

T1def

mmm

m

1 11

1 1 1def

nn

n n

PPP P

PP

P P

TT T

T TT

TT Tn

T TTn

uv

uu

! "−$ %

−$ %$ %=$ %$ %−$ %$ %−& '

00

P0

0

!

Calibration Problem

2n x 1212x1

1x44x1

…

Homogenous linear system

knownunknown

0)( 12131 =+− PPv mm

0)( 11131 =+− PPu mm

0)( 23 =+− nnn PPv mm

0)( 13 =+− nnn PPu mm

0=P m [Eq. 4]

Homogeneous M x N Linear Systems

P m 0=

Rectangular system (M>N)• 0 is always a solution

Minimize |P m|2 under the constraint |m|2 =1

M=number of equations = 2nN=number of unknown = 11

• To find non-zero solution

N

M

• How do we solve this homogenous linear system?

Calibration Problem

• Via SVD decomposition!

0=P m

Calibration Problem

1212T

121212n2 VDU ×××

Last column of V gives m

M

SVD decomposition of P

Why? See pag 592 of HZ

!!!

"

#

$$$

%

&

=T3

T2

T1def

mmm

m

0=P m

Extracting camera parameters

M = ρ

Extracting camera parameters

A

!!!

"

#

$$$

%

&

=T3

T2

T1

Aaaa

[ ]TRK=

3

1a±

=ρ

!!!

"

#

$$$

%

&

=

3

2

1

bbb

b

Estimated values

)( 312 aa ⋅= ρou

)(v 322

o aa ⋅= ρ

( ) ( )3231

3231cosaaaaaaaa

×⋅×

×⋅×=θ

Intrinsic

b !!!

"

#

$$$

%

& −

=

100

v0

ucot

K o

o

sinθβ

θααρ

M =

Box 1

Theorem (Faugeras, 1993)

Extracting camera parameters

!!!

"

#

$$$

%

&

=T3

T2

T1

Aaaa

[ ]TRK=

!!!

"

#

$$$

%

&

=

3

2

1

bbb

b

Estimated values

Intrinsic

θρα sin312 aa ×=

θρβ sin322 aa ×=

ρ

A b

Extracting camera parameters

A

!!!

"

#

$$$

%

&

=T3

T2

T1

Aaaa

b

[ ]TRK=

!!!

"

#

$$$

%

&

=

3

2

1

bbb

b

Estimated values

Extrinsic( )

32

321 aa

aar×

×=

3

33 a

ar ±=

132 rrr ×= b1KT −= ρ

ρ

Degenerate cases

•Pi’s cannot lie on the same plane!• Points cannot lie on the intersection curve of two quadric surfaces

Lecture 3 - Silvio Savarese 14-Jan-15

• Recap of projective cameras • Camera calibration problem• Camera calibration with radial distortion• Example

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Lecture 3 Camera Calibration

Reading: [FP] Chapter 1 “Geometric Camera Calibration” [HZ] Chapter 7 “Computation of Camera Matrix P”

No distortion

Pin cushion

Barrel

Radial Distortion– Image magnification (in)decreases with distance from the optical axis– Caused by imperfect lenses– Deviations are most noticeable for rays that pass through the edge of

the lens

Radial Distortion

ii

ii p

vu

PM

100

00

00

1

1

=!"

#$%

&→

!!!!!!

"

#

$$$$$$

%

&

λ

λd

v

vucvbuad 222 ++=

u

To model radial behavior

∑±==

3

1p

2ppdκ1λ

Polynomial function

Distortion coefficient

[Eq. 5] [Eq. 6]

Sλ

Image magnification decreases with distance from the optical center

!"

#$%

&=

i

ii v

up

!!!

"

#

$$$

%

&

=

3

2

1

Qqqq

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

qqqq

ii

ii p

vu

PM

100

00

00

1

1

=!"

#$%

&→

!!!!!!

"

#

$$$$$$

%

&

λ

λ

Q

!"#

=

=

PPvPPu

2i3i

i1i3i

qqqq

Is this a linear system of equations?

Radial Distortion

No! why? [Eqs.7]

X = f (Q)

measurements parameters

f( ) is the nonlinear mapping

-Newton Method-Levenberg-Marquardt Algorithm

• Iterative, starts from initial solution • May be slow if initial solution far from real solution • Estimated solution may be function of the initial solution• Newton requires the computation of J, H• Levenberg-Marquardt doesn’t require the computation of H

General Calibration Problem

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

qqqq

!"

#$%

&

i

i

vu

i=1…n

[Eq .8]

A possible algorithm

1. Solve linear part of the system to find approximated solution 2. Use this solution as initial condition for the full system3. Solve full system using Newton or L.M.

General Calibration Problem

X = f (Q)

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

qqqq

!"

#$%

&

i

i

vu

i=1…n

measurements parameters

[Eq .8]

f( ) is the nonlinear mapping

Typical assumptions:

- zero-skew, square pixel- uo, vo = known center of the image

General Calibration Problem

X = f (Q)

measurement parameter

f( ) is nonlinear!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

qqqq

!"

#$%

&

i

i

vu

Can we estimate m1 and m2 and ignore the radial distortion?

Radial Distortion

!"

#$%

&=

i

ii v

up

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

1

mmmm

λ

d

v

uHint: slopevu

i

i =

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

qqqq

Estimating m1 and m2…

Radial Distortion

!"

#$%

&=

i

ii v

up

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

1

mmmm

λ

0)()( 121111 =− PuPv mm0)()( 21 =− iiii PuPv mm

0)()( 21 =− nnnn PuPv mm…

L n= 0 !"

#$%

&=

2

1

mm

n

Tsai [87]

i

i

i

i

i

i

i

i

PP

PPPP

vu

2

1

3

2

3

1

)()()()(

mm

mmmm

==

Get m1 and m2 by SVD

[Eq .9]

[Eq .10][Eq .11]

Once that m1 and m2 are estimated…

Radial Distortion

!"

#$%

&=

i

ii v

up

!!!!

"

#

$$$$

%

&

=

i3

i2

i3

i1

PPPP

1

mmmm

λ

3m is non linear function of 1m 2m λ

There are some degenerate configurations for which m1 and m2 cannot be computed

, ,

Lecture 3 - Silvio Savarese 14-Jan-15

Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li

Lecture 3 Camera Calibration • Recap of projective cameras • Camera calibration problem• Camera calibration with radial distortion• ExampleReading: [FP] Chapter 1 “Geometric Camera Calibration”

[HZ] Chapter 7 “Computation of Camera Matrix P”

Calibration ProcedureCamera Calibration Toolbox for MatlabJ. Bouguet – [1998-2000] http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Calibration Procedure

Next lecture

• Single view reconstruction