Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical...

66
Camera Calibration Machine Vision (CIS 580)

Transcript of Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical...

Page 1: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Camera Calibration

Machine Vision (CIS 580)

Page 2: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center
Page 3: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

[ ] =

x XK R tL

1 1

Camera body configuration (extrinsic parameter)

Spatial relationship between sensor and pinhole (internal parameter)

Lens configuration (internal parameter)

Page 4: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Why Do We Calibrate?

• To find an intrinsic bias produced by lens configuration and manufacture imprecision. • The intrinsic bias is often fixed as long as the focal length

of the camera does not change. • Pre-calibration of the intrinsic bias reduces the number of

unknowns in estimation of camera parameters, e.g., structure from motion.

• The intrinsic bias encoded in an image can serve as a fingerprint of the camera.

Page 5: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e.,

f (mm) fx (pixel).

• Position of image center (principal point), i.e., px, py

• A skew factor between x and y axis of the image, i.e., uimg=fx x+sy+px.

• Lens distortion

f (mm)

fx (pixel)

Page 6: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Single View Calibration

Page 7: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

[ ] =

x XK R tL

1 1

Camera body configuration (extrinsic parameter)

Spatial relationship between sensor and pinhole (internal parameter)

Lens configuration (internal parameter)

Page 8: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Ideal Perspective Projection

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

img 11 12 13 1

img 21 22 23 2

31 32 33 31 11

x X×∈R 3 3 tK

• Estimating K from 2D measurements, x, without assumption is an ill-posed problem because the number of unknowns are greater than the number of equations, i.e.,

• # of unknowns: 5 (K) + 3 (R) + 3 (t) + 3 (X) • # of equations: 2

Page 9: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Single View

[ ]×

=

= I 0x x x x

y y y y

Xu f s p f s p X

Yz v f p f p Y

ZZ

camimg cam

camimg 3 3 cam

camcam1 1 1

1

• Without loss of generality, we can reduce the number of unknowns by setting . If Xcam is known, the intrinsic parameters can be estimated using multiple Xcam.

• # of unknowns: 5 (K) • # of equations: 2n

[ ]×P = K I 03 3

Page 10: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lines of Projections

u

v x

X−K x1

[ ]×

=

Xx = K I 0 KXz 3 3 1

−K x = Xz 1

−K x1 defines a ray emitting from the origin (the camera center) towards X.

[ ]×P = K I 03 3

Page 11: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lines of Projections

u

v x1

[ ]×P = K I 03 3

X−K x1

x2θ

2

− −

− − − −

− −

− − − −=

K x K x

K x K x K x K x

x K K x

x K K x x K K x

1 T 11 2

1 T 1 1 T 11 1 2 2

T T 11 2

T T 1 T T 11 1 2

( ) ( )cos

( ) ( ) ( ) ( )

( )

( ) ( )

θ =

[ ]×

=

Xx = K I 0 KXz 3 3 1

−K x = Xz 1

−K x1 defines a ray emitting from the origin (the camera center) towards X.

Angle between two rays:

Page 12: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Angle Between Vanishing Points

[ ] ∞

=

=

x K 0 X

KX1 1

1

∞X2

∞=x KX2 2θ

′θ

∞X1

′θ = θKey observation: the angle between two lines in 3D is the same as the angle between two rays pointing at the vanishing points of the two lines, i.e., .

2

− −

− − − −=

x K K x

x K K x x K K x

T T 11 2

T T 1 T T 11 1 2

( )cos

( ) ( )θ =

Result: if we know angle between two lines in 3D, we can estimate K via

Projection of ∞X1

X is inhomogeneous coordinate of . X

Page 13: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

2D Case

∞X2

∞=x KX2 2

θ′θ

∞X1

′θ = θ

Camera center at O

f

∞=x KX1 1

p

Focal plane

L1 ′L1

L2

′L2

L1 ′L1 L2′L2because and

Page 14: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

2D Case Wrong Focal Length

∞X2

∞=x KX2 2

θ

′θ

∞X1

′≠θ θ

Camera center at O

f

∞=x KX1 1

p

Page 15: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

2D Case Wrong Principal Point

∞X2

∞=x KX2 2

θ

′θ

∞X1

Camera center at O

f

∞=x KX1 1

p

′≠θ θ in general.

Page 16: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

2D Case

∞X2

∞=x KX2 2

θ′θ

∞X1

′θ = θ

Camera center at O

f

∞=x KX1 1

p

Focal plane

L1 ′L1

L2

′L2

L1 ′L1 L2′L2because and

Given two vanishing points, the focal length and principal point are uniquely defined. For the 3D case, another vanishing point is needed to uniquely define f, px, and py.

Page 17: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Calibration via Vanishing Points

Page 18: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Calibration via Vanishing Points

Page 19: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Calibration via 3 Vanishing Points

− − =x K K xT T 11 2( ) 0

x1 x2

x3

When vanishing points of lines are perpendicular,

x1

x2

O

2

− −

− − − −=

x K K x

x K K x x K K x

T T 11 2

T T 1 T T 11 1 2

( )cos 0

( ) ( )θ =

−K x1 2

−K x1 1

Page 20: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Calibration via 3 Vanishing Points − − =x K K xT T 1

1 2( ) 0− − =x K K xT T 1

3 2( ) 0− − =x K K xT T 1

1 3( ) 0x1 x2

x3

Assume zero skew and square pixel:

K =x

y

f p

f p

0

13 unknowns and 3 equations.

− −

= =

K K B

B B

B B

B B B

1 2T 1

1 3

2 3 4

=x BxT3 2 0

=x BxT1 2 0

=x BxT1 3 0

Substitution to formulate linear equations:

+ + +

+ + + = = + + +

A 0b

Bu u v v u u v v

Bu u v v u u v v

Bu u v v u u v v

B

11 2 1 2 1 2 1 2

23 2 3 2 3 2 3 2

31 3 1 3 1 3 1 3

4

1

1

1

Page 21: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Calibration via 3 Vanishing Points

x1 x2

x3

+ + +

+ + + = = + + +

A 0b

Bu u v v u u v v

Bu u v v u u v v

Bu u v v u u v v

B

11 2 1 2 1 2 1 2

23 2 3 2 3 2 3 2

31 3 1 3 1 3 1 3

4

1

1

1

= Ab null( )

b has to satisfy Ab=0 while avoiding a trivial solution, i.e., b=0.

Such b is the nullspace of A,i.e, .

The nullspace of A is can be computed by Singular Value Decomposition (SVD):

A = UDV T

b = V:,4 (the fourth or last column of V) where

Once B is estimated, it can be decomposed by Cholesky decomposition that decomposes a positive definite matrix to lower triangular matrix, i.e., where . B = LLT −=L KT 1

Page 22: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Calibration via 3 Vanishing Points

= − = − +

K K

x

y

y yx x

pf f

B Bp

B Bf f

B B Bp p-p p

f f f f

2 2

1 2-T -1

1 32 2

2 3 422

2 2 2 2

1

1

+1

( )

/

/

= −

= −

= − +

y

x

x y

p B B

p B B

Bf p p

B

3 1

2 1

2 24

1

Given B, solve for intrinsic parameters:

Or

Page 23: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Multiple View Calibration

Page 24: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Counting Argument

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

img 11 12 13 1

img 21 22 23 2

31 32 33 31 11

x X×∈R 3 3 tK

u

v

: knowns

# of unknowns: 5 (K) + 3 (R) + 3 (t) # of equations: 2m where m is the number of known points.

Page 25: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Multiple View Calibration

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

u

v

# of unknowns: 5 (K) + 3n (R) + 3n (t) # of equations: 2nm where m is the number of known points and n is the number of images.

Page 26: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Known 3D Points

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

u

v

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

img 11 12 13 1

img 21 22 23 2

31 32 33 31 11

x X×∈R 3 3 tK

: knowns

: unknowns

Page 27: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Known 3D Points

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

u

v

X Y( , ,0)

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

r r r t

img 11 12 13 1

img 21 22 23 2

31 32 33 3

01 1

1

x X×∈R 3 3 tK

: knowns

: unknowns

Page 28: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Known 3D Points

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=x x

y y

u f s p r r t X

z v f p r r t Y

r r t

img 11 12 1

img 21 22 2

31 32 31 1 1

u

v

: knowns

X Y( , ,0)

: unknowns

Page 29: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Homography btw Two Planes

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

u

v

X Y( , ,0)

=

u h h h X

z v h h h Y

h h h

11 12 13

21 22 23

31 32 331 1H Xx =: estimate

: knowns

: homography between 3D plane and 2D image plane. H

Page 30: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Homography and Projection

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

′=

x x

y y

h h h f s p r r t

h h h z f p r r t

h h h r r t

11 12 13 11 12 1

21 22 23 21 22 2

31 32 33 31 32 31

h1

: estimate

K r1 r2 t1h2 h3

: unknowns

Page 31: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Orthogonality

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

′=

x x

y y

h h h f s p r r t

h h h z f p r r t

h h h r r t

11 12 13 11 12 1

21 22 23 21 22 2

31 32 33 31 32 31

h1

: estimate

K r1 r2 t1

Recall orthogonality:

h2 h3

: unknowns

r rT1 2 = 0r rT2 2 = 1r rT1 1 = 1

Page 32: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Orthogonality

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

′=

x x

y y

h h h f s p r r t

h h h z f p r r t

h h h r r t

11 12 13 11 12 1

21 22 23 21 22 2

31 32 33 31 32 31

h1

: estimate

K r1 r2 t1

Recall orthogonality: r rT1 1 = 1 r rT2 2 = 1 r rT1 2 = 0

h2 h3

: unknowns

where and ′=h rK z11

-1 ′=h rK z1

2-

2

=h h hK K K K hT T1 1 2

-T -1 -T -12 =K Kh h-T -1T

1 2 0 2 equations and 5 unknowns

Page 33: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Substitution

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

= − + − − =

− − +

K K

y x y

x x y x y

y x y y

x y x y y x y y

y x y y x y y y x y y

x y x y y x y y

p s - p fsf f f f f

B B Bs p s - p f ps s

B B Bf f f f f f f f

B B Bp s - p f s p s - p f p p s - p f p

f f f f f f f f

2 2 2

11 12 132-T -1

21 22 232 2 2 2 2 2 2

31 32 332 2

2 2 2 2 2 2 2

1

( )1

( ) ( )+1

=h h hK K K K hT T1 1 2

-T -1 -T -12 =K Kh h-T -1T

1 2 0

=K K-T -1 : symmetric matrix

=h hBT1 2 0=h h h hB BT T1 1 2 2 2 equations and linear in B.

Page 34: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Linear Equations

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

[ ]=b B B B B B BT

11 12 22 13 23 33

=h h vB bi j ijT T

where

= + + + v ij i j i j i j i j i j i j i j i j i jh h h h h h h h h h h h h h h h h hT

1 1 1 2 2 1 2 2 3 1 1 3 3 2 2 3 3 3

=h h h hB BT T1 1 2 2 =h hBT1 2 0

Rewrite equation by vectorizing B.

=

B

B B B

B B B

B B B

11 12 13

21 22 23

31 32 33

Linear equation in B

v v= 0

vb

T T11 22

T12v b =T

12 0=h hBT1 2 0

=h h h hB BT T1 1 2 2 v = vb bT T

11 22: 2 equations per view

Page 35: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Linear Equations

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

[ ]=b B B B B B BT

11 12 22 13 23 33

=h h vB bi j ijT T

where

= + + + v ij i j i j i j i j i j i j i j i j i jh h h h h h h h h h h h h h h h h hT

1 1 1 2 2 1 2 2 3 1 1 3 3 2 2 3 3 3

=h h h hB BT T1 1 2 2 =h hBT1 2 0

Rewrite equation by vectorizing B.

=

B

B B B

B B B

B B B

11 12 13

21 22 23

31 32 33

Linear equation in B

v b =T12 0=h hBT1 2 0

=h h h hB BT T1 1 2 2 v = vb bT T

11 22 −

v v

v= 0

v v

b

v

n n

n

1 T 1 T11 221 T

12

T T11 22

T12

Minimum configuration: • 3 images for general K. • 2images when w=0.

Page 36: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Solving for Intrinsic Parameters

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

= − + − − =

− − +

K K

y x y

x x y x y

y x y y

x y x y y x y y

y x y y x y y y x y y

x y x y y x y y

p s - p fsf f f f f

B B Bs p s - p f ps s

B B Bf f f f f f f f

B B Bp s - p f s p s - p f p p s - p f p

f f f f f f f f

2 2 2

11 12 132-T -1

21 22 232 2 2 2 2 2 2

31 32 332 2

2 2 2 2 2 2 2

1

( )1

( ) ( )+1

Recall

/

/

= − −

= − + −

=

y

y

x

p B B B B B B B

c B B p B B B B B

f c B

212 13 11 23 11 22 12

233 13 12 13 11 23 11

11

( ) ( )

[ ( )] /

/

/

/ /

= −

= −

= −

y

x y

x y y x

f cB B B B

s B f f c

p sp f B f c

211 11 22 12

212

213

( )

Given B, solve for intrinsic parameters:

Page 37: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Solving for Extrinsic Parameters

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=′

=′

r

r

K h

K h

z

z

11

12

1

2

1

1

−′ = K hz 11

where

[ ] [ ]′ r r rh h = th Kz 13 22 31

=

=

×

r r

K

r

t hz

3 1 2

13

1

Page 38: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Summary of Linear Parameter Estimation

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

u

v

X Y( , ,0)

• Reduce dimensionality of a 3D point to 2D assuming the point lies on a plane.

• Estimate a homography between 3D plane and the 2D projected image.

• Constrain the estimated homography using orthogonality of rotation matrix

• Substitute where B is a symmetric matrix.

• Formulate a system of linear equations in B.

• Compute the intrinsic and extrinsic parameters.

r r r rT T1 1 2 2= =h h hK K K K hT T

1 1 2-T -1 -T -1

2

=K Kh h-T -1T1 2 0r rT1 2 = 0

B = K K-T -1

v v= 0

vb

T T11 22

T12

Page 39: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Geometric Error Minimization

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

11 12 13 1

21 22 23 2

31 32 33 31 11

x XR tKu

v

x ij

x̂ K R t Xij i i j( , , , )

Page 40: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Geometric Error Minimization

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

11 12 13 1

21 22 23 2

31 32 33 31 11

x XR tKu

v

, ,ˆ

= =

−∑∑K R t

x x K R t XI J

ij ij i i ji j

2

1 1

minimize ( , , , )reprojection

measurement

x ij

x̂ K R t Xij i i j( , , , )

Page 41: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lens Distortion

Page 42: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

[ ] =

x XK R tL

1 1

Camera body configuration (extrinsic parameter)

Spatial relationship between sensor and pinhole (internal parameter)

Lens configuration (internal parameter)

Page 43: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lens Distortion Radial Distortion

Page 44: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lens Distortion Radial Distortion

Straight lines are not straight in the image due to lens distortion.

Page 45: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lens Distortion Radial Distortion

Lens distortion correction

Page 46: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Radial Lens Distortion

=x xL rimg ideal( )

Image undistortion

r

Radial lens distortion model assumes that the distortion is a function of distance between image point to the principal point.

x imgx ideal

: a point measured in the image (distorted)

: a point that follows perspective projection (undistorted) ,x yp p= xr ideal

Page 47: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Radial Lens Distortion

rx img

x ideal

= xr ideal

where = + + +L r

2 41 2( ) 1 k r k r

Polynomial fitting of distortion

=x xL rimg ideal( )

Page 48: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Radial Lens Distortion

= + +

= + +x

y

u u u - p r r

v v v - p r

k

r

k

k k

2 4img ideal ideal

2 4img idea

1 2

1l id l 2ea

( )[ ]

( )[ ]

where and are measurements and and are the projected point.

u img

− = −

x x

y y

u uku - p r u - p rv vkv - p r v - p r

2 4img ideal1ideal ideal

2 4img ideal2ideal ideal

( ) ( )

( ) ( )

Mapping between an ideal point and image point (2nd order):

v img u ideal v ideal

r x imgx ideal

Linear equations in k1 and k2:

: unknowns

Page 49: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Radial Lens Distortion

= + +

= + +x

y

u u u - p r r

v v v - p r

k

r

k

k k

2 4img ideal ideal

2 4img idea

1 2

1l id l 2ea

( )[ ]

( )[ ]

where and are measurements and and are the projected point.

u img

− − = − −

x x

y y

m mm mx m x m

m mm my m y m

u uu - p r u - p r

v vv - p r v - p rk

ku uu - p r u - p r

v vv - p r v - p r

1 11 2 1 4img idealideal 1 ideal 1

1 11 2 1 4img idealideal 1 ideal 1

1

22 4img idealideal ideal

2 4img ideideal ideal

( ) ( )

( ) ( )

( ) ( )

( ) ( )

al

Mapping between an ideal point and image point (2nd order):

v img u ideal v ideal

r x imgx ideal

: unknowns

Linear equations in k1 and k2:

for m points

Page 50: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Geometric Error Minimization

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

11 12 13 1

21 22 23 2

31 32 33 31 11

x XR tKu

v

, ,ˆ

= =

−∑∑K R t

x x K R t XI J

ij ij i i ji j

2

1 1

minimize ( , , , )reprojection

measurement

x ij

x̂ K R t Xij i i j( , , , )

Page 51: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Geometric Error Minimization

Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.

=x x

y y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

11 12 13 1

21 22 23 2

31 32 33 31 11

x XR tKu

v

x ij

x̂ K R t Xij i i j( , , , )

, ,= =

−∑∑K R t,

x K R t Xm n

ij i i jk ki j

L k k1 2

2

1 2,

1 1

minimize ( , , , , , )reprojection

measurement

Page 52: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Geometric Error Minimization

Page 53: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center
Page 54: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center
Page 55: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center
Page 56: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Demo with MATLAB

http://www.vision.caltech.edu/bouguetj/calib_doc/

Page 57: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

• Download the MATLAB calibration toolbox: http://www.vision.caltech.edu/bouguetj/calib_doc/download/index.html

• Print out a calibration pattern: http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/pattern.pdf

• Take images from multiple angles and save them in the calibration toolbox folder. • Type calib and load images.

• Type pattern dimensions and annotate corners.

• Calibrate the camera.

Page 58: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Lens Large Distortion

Page 59: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center
Page 60: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Fisheye Lens Distortion

Page 61: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Fisheye Lens Distortion

Page 62: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Fisheye Lens Model Logarithmic Distortion

F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.

=x xL rimg ideal ideal( )

=r c krimg ideallog(1+ )

rideal x img

x ideal

rimg

rL r

rimg

ideal

( ) =

Fisheye have a high resolution at the fovea, and non linearly decreasing resolution towards the periphery, i.e.,

∆rideal

∆rimg

= =

∆ ∆>

∆ ∆r r r r

r rr r

ideal 1 ideal 2

ideal ideal

img img

>r r1 2for

=x xc kr

rideal

img idealideal

log(1+ )

where

Page 63: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Fisheye Lens Model Field of View (FOV) Distortion

F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.

∝ =r r kimg ideal( ) θ θ

=

=

_

_

r kr kw /

r

r w

img

img max

img

img max

2

tan

tan( / 2)

θ

θ

z

C

X

x ideal x img

ridealrimg

,x yp p

θ=

wmax2θ

_rimg max

=

=

wr r

wr w

rw

img ideal

imgideal

1arctan(2 tan )

2tan( )

2tan( / 2)

rideal x img

x ideal

rimgSpatial resolution is proportional to angular resolution, i.e.,

=x xL rimg ideal ideal( )r

L rrimg

ideal

( ) =

=_

kwrimg max 2

Spatial resolution:

where w is angular field of view.

Page 64: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Fisheye Lens Model Field of View (FOV) Distortion

F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.

z

C

X

x ideal x img

ridealrimg

,x yp p

θ=

wmax2θ

_rimg max

=x xw

rwrimg ideal ideal

ideal

1arctan(2 tan )

2

rideal x img

x ideal

rimg=x xL rimg ideal ideal( )

rL r

rimg

ideal

( ) =

where w is angular field of view.

Page 65: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Distortion Correction Procedure

F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.

1. Detect edges from an image.

Page 66: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center

Distortion Correction Procedure

F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.

1. Detect edges from an image.

2. Link edges that are likely to belong to the same edge.

3. Select distortion model and initialize the distortion parameters with a reasonable value.

4. Fit lines with the linked edges with least squares.

5. Optimize the distortion parameters to fit the edges to the ideal lines.

Detected edge segment

Ideal edge