Camera models and single view geometry

Camera model

Camera: optical system

d2 1

thin lens

small angles:

Y

Z

11

Y

21

curvature radius

22

Y

Y

Z

incident light beam

deviated beam

deviation angle ? ’’

lens refraction index: n

)11

()1(21

Yn

n

)'sin(

)sin(

' 1

1

1

1

n

)'sin(

)''sin(

'

''

2

2

2

2

Thin lens rulesa) Y=0 = 0

)11

)(1(

1

21

n

f

f

Y

parallel rays converge onto a focal plane

b) f = Y

beams through lens center: undeviated

independent of y

r

f

Y

h

Where do all rays starting from a scene point P converge ?

Z

r

Y

h

Z

f

fr

Y

h

frZ

111Fresnel law

P

Obs. For Z ∞, r f

O

p?

d

f

a

Z

if d ≠ r …

focussed image:blurring circle) <image resolution

depth of field: range [Z1, Z2] where image is focussed

image plane P

p

O

r (blurring circle)=a (d-r)/r

image of a point = blurring circle

the image of a point P belongs to the line (P,O)

p

P

O

p = image of P = image plane ∩ line(O,P)

interpretation line of p: line(O,p) =locus of the scene points projecting onto image point p

image plane

r fHp: Z >> a

the image of a point P belongs to the line (P,O)

p

P

O

p = image of P = image plane ∩ line(O,P)

interpretation line of p: line(O,p) =locus of the scene points projecting onto image point p

image plane

r fHp: Z >> a

p

P

O Z

Y

X

c

y

x

Z

Xfx

Z

Yfy

perspective projection

f

-nonlinear-not shape-preserving-not length-ratio preserving

•Point [x,y]T expanded to [u,v,w]T

•Any two sets of points [u1,v1,w1]T and [u2,v2,w2]T

represent the same point if one is multiple of the other

•[u,v,w]T [x,y] with x=u/w, and y=v/w

•[u,v,0]T is the point at the infinite along direction (u,v)

• In 2D: add a third coordinate, w

Homogeneous coordinates

Transformations

translation by vector [dx,dy]T

scaling (by different factors in x and y)

rotation by angle

Homogeneous coordinates

In 3D: add a fourth coordinate, t

•Point [X,Y,Z]T expanded to [x,y,z,t]T

•Any two sets of points [x1,y1,z1,t1]T and [x2,y2,z2,t2]T

represent the same point if one is multiple of the other

•[x,y,z,t]T [X,Y,Z] with X=x/t, Y=y/t, and Z=z/t

•[x,y,z,0]T is the point at the infinite along direction (x,y,z)

Transformations

scaling

translation

rotation

Obs: rotation matrix is an orthogonal matrix

i.e.: R-1 = RT

Pinhole camera model

TT ZfYZfXZYX )/,/(),,(

101

0

0

1

Z

Y

X

f

f

Z

fY

fX

Z

Y

X

10100

000

000

Z

Y

X

f

f

w

v

u

with

w

ux

w

vy

Scene->Image mapping: perspective transformation

With “ad hoc” reference frames, for both image and scene

Let us recall them

O Z

Y

X

c

y

x

f

scene reference- centered on lens center- Z-axis orthogonal to image plane- X- and Y-axes opposite to image x- and y-axes

image reference- centered on principal point- x- and y-axes parallel to the sensor rows and columns- Euclidean reference

O

Z

YX

c

yx

f

scene reference- not attached to the camera

image reference- centered on upper left corner- nonsquare pixels (aspect ratio) noneuclidean reference

Actual references are generic

principal axis

principal point

Principal point offset

Too

T vZfYuZfXZYX )/,/(),,(

principal pointT

oo vu ),(

101

0

0

1

Z

Y

X

vf

uf

Z

ZvfY

ZufX

Z

Y

X

o

o

o

o

CCD camera

1oy

ox

vf

uf

K

11

1

o

o

vf

uf

aK

Scene-image relationship wrt actual reference frames

w

v

u

va

uas

w

v

u

o

o

100

0

)(1

Auimage

scene XtR

10

1

Z

Y

X

normally, s=0

XtR

XPu 10

0100

000

000

100

0

01

f

f

va

u

o

o

tKRKtR

IP 10

0

100

0

0

o

o

vaf

uf

K upper triangular: intrinsic camera parameters

scene-camera tranformation

extrinsic camera parameters

orthogonal (3D rotation) matrix

P: 10-11 degrees of freedom (10 if s=0)

1

z

y

x

oIMXoIMu

)(1

oxMx

oIMu

i.e., defining x = [x, y, z]T

oIMoMMmMtKRKP

with RKM and mMo 1

The locus of the points x whose image is u is a straight line through o having direction

mMo 1 is independent of u

o is the camera viewpoint (perspective projection center)

uMd 1

line(o, d) = Interpretation line of image point u

Interpretation of o:

u is image of x if uMox 1)(

i.e., if uMox 1

Intrinsic and extrinsic parameters from P

RKM

M K and R

1111 KRKRM T

RQ-decomposition of a matrix: as the product betweenan orthogonal matrix and an upper triangular matrix

M and m t

tRKtKRKtKR)mMo 1T1 T (1

Rot

Camera anatomy

Camera centerColumn pointsPrincipal planeAxis planePrincipal pointPrincipal ray

Camera center

0P O

null-space camera projection matrix

Oλ)(1λAX

Oλ)P(1λPAPXx

For all A all points on AO project on image of A,

therefore O is camera center

Image of camera center is (0,0,0)T, i.e. undefined

Finite cameras:

1

M

1

1moO

Infinite cameras: 0Md,0

d

o

Column vectors

0

0

1

0

mpppp 3212

Image points corresponding to X,Y,Z directions and origin

Row vectors

1p

p

p

0 3

2

1

Z

Y

X

y

x

T

T

T

Image of a point on the principal plane (the plane of the thin lens) is at the infinity

w = 0

is the principal plane

0

1

p3

Z

Y

X

T

T3p

1p

p

p0

3

2

1

Z

Y

X

w

yT

T

T

note: p1,p2 dependent on image reparametrization

0

1

p2

Z

Y

X

T T2p is the plane through the u-axis

T1p is the plane through the v-axis similarly,

The principal point

principal point

0,,,p̂ 3332313 ppp

33o Mmp̂Pc

The principal axis vector

3m

vector defining front side of camera

the direction of the normal to the principal plane

3m

Action of projective camera on point

PXx

MdDm|MPDx

Forward projection

Back-projection

xPX 1PPPP

TT IPP

(pseudo-inverse)

0P O

OλxPλX

1

m-μxM

1

mM-

0

xMμλX

-1-1-1

xMd -1

OD

Depth of points

oO X~

mXPXPT3T3T3 w

(dot product)(PO=0)

1m;0det 3 MIf , then m3 unit vector in positive direction

3m

)sign(detMPX;depth

T

w TX X,Y,Z,T

Camera matrix decomposition

Finding the camera center

0P O (use SVD to find null-space)

m,p,pdet 32X m,p,pdet 31Y

m,p,pdet 21Z m,p,pdet 21T

Finding the camera orientation and internal parameters

KRM (use RQ decomposition ~QR)

Q R=( )-1= -1 -1QR

(if only QR, invert)

When is skew non-zero?

1yx

xx

p

ps

K

1

arctan(1/s)

for CCD/CMOS, always s=0

Image from image, s≠0 possible(non coinciding principal axes)

HPresulting camera:

Euclidean vs. projective

homography 44

0100

0010

0001

homography 33P

general projective interpretation

Meaningfull decomposition in K,R,t requires Euclidean image and space

Camera center is still valid in projective space

Principal plane requires affine image and space

Principal ray requires affine image and Euclidean space

Camera calibration

11

v

u

Z

Y

X

ii uX

?

3

2

1

P

P

P

P

from scene-point to image point correspondence…

…to projection matrix

Basic equations

ii PXu 0u ii PX

0

0

01

10

3

2

1

i

ii

i

i

P

P

P

uv

u

v

X

0

0

01

10

0

01

10

3

2

1

3

2

1

TT

TT

TT

P

P

P

uv

u

v

P

P

P

uv

u

v

i

i

i

ii

i

i

i

i

i

ii

i

i

X

X

X

X

X

X

Basic equations ctd.

0Ap

0

0

01

10

0

01

10

3

2

1

3

2

1

TT

TT

TT

P

P

P

uv

u

v

P

P

P

uv

u

v

i

i

i

ii

i

i

i

i

i

ii

i

i

X

X

X

X

X

X

00

0

3

2

1

T

T

T

TT

TT

P

P

P

u

v

iii

iii

XX

XX

withT

321 PPP p (12x1)

singular matrix

minimal solution

over-determined solution

5½ correspondences needed (say 6)

P has 11 dof, 2 independent eq./points

n 6 points

Apminimize subject to constraint 1p

0Ap

p: eigenvector of ATA associated to its smallest eigenvalue

Degenerate configurations

More complicate than 2D case (see Ch.21)

(i) Camera and points on a twisted cubic

(ii) Points lie on plane or single line passing through projection center

Data normalization

32ii UXX

~

ii Tuu~

(i) translate origin to gravity center(ii) (an)isotropic scaling

from line correspondences

Extend DLT to lines

ilPT

0PXl 1T ii

(back-project image line)

0PXl 2T ii

(2 independent eq.)

Geometric error

Gold Standard algorithmObjective

Given n≥6 2D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P

Algorithm

(i) Linear solution:

(a) Normalization:

(b) DLT:

(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:

(iii) Denormalization:

ii UXX~ ii Txx~

UP~

TP -1

~ ~~

Calibration example

(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners

typically precision <1/10

(HZ rule of thumb: 5n constraints for n unknowns

Exterior orientation

Calibrated camera, position and orientation unkown

Pose estimation

6 dof 3 points minimal (4 solutions in general)

short and long focal length

Radial distortion

Radial distortion

Correction of distortion

Choice of the distortion function and center

Computing the parameters of the distortion function(i) Minimize with additional unknowns(ii) Straighten lines(iii) …

Properties of perspective transformations

1) vanishing points

V image of the unproper point along direction d

dMd

mMuV 0

VuMd 1

the interpretation line of V is parallel to d

O

V

Pd

The images of parallel lines are concurrent lines

2) cross ratio invariance

Given four colinear points 4321 ,,, pppp

42

32

41

31

4321 ,,,

xx

xxxx

xx

CR

pppp

let 4321 ,,, xxxx be their abscissae

Properties of perspective transformations ctd.

Cross ratio invariance under perspective transformation

TT ],0,0,[],,,[ txtzyx Xa point on the line y=0=z

xPu T 144

1],[t

xwu

p

p

its image belongs to a line

its coordinate u

XPu T],,[ wvu

4231

43214321 ,det,det

,det,det,,,

uuuu

uuuuuuuu CR

42143114

43142114

,detdet,detdet

,detdet,detdet

xxPxxP

xxPxxP

4231

4321

,det,det

,det,det

xxxx

xxxx 4321 ,,, xxxxCR

Object localization 1: three colinear points

geometric model of an objecta perspective image of the object

position and orientation

of the object ?

A

B

C

C’

A’

B’O

calibrated camera: mMP A

B

C

known

known interpretation lines

A

B

C

C’

A’

B’O

∞

V

a) orientation

cb

caCBACRVCBACR

),,,(,',','

Cross ratio invariance:

solve for V (image of ∞)

V: vanishing point of the direction of (A,B,C)

interpretation line of V parallel to (A,B,C) VuM 1direction

b) position (e.g., distance(O,A))

A

B

C

C’

A’

B’O

V

VuM 1

CuM 1

AuM 1

interpretation lines

angles and

sin

sinACOA

Object localization 2: four coplanar points

O

(i) orientation of (A,E,C)(ii) orientation of (B,E,D)(iii) distance (O,A)

E

A

B

CD

a’

b’

b”

a”

Find vanishing point of the field-bottom direction

Off-side

images of symmetric segments

a and b: abscissae of the endpoints of a segment

c=(a+b)/2: abscissa of segment midpoint,

d=∞: point at the infinite along the segment direction

a c b d

1,,,

cb

ca

dbdacbca

dcbaCR

(a’,b’) and (a”, b”) are image of symmetric segments

same image of the midpoint c’, same vanishing point d’

Harmonic 4-tuple (a,b,c,d)

solve

1

''''

''

''

',',','

dbda

cb

ca

dcbaCR

1

''''''

''''''

',','',''

dbda

cbca

dcbaCR

{ for c’, d’

system of two linear equations in (c’d’) and (c’+d’)

two degree equation, whose solutions are c’ and d’

among the two solutions, the one for d’ is the value external to the range [a’,b’]

Action of projective camera on planes

1ppp

10ppppPXx 4214321 Y

XYX

The most general transformation that can occur between a scene plane and an image plane under perspective imaging is a plane projective transformation

Action of projective camera on lines

forward projection

μbaμPBPAμB)P(AμX

back-projection

PXx

lPT

0xlT

with

X0PXl TT

Interpretation plane of line l

PxxY

X

T

Y

X

4214214321 ppp

1

ppp0

ppppPXx

Image of a conic

xCx0C 1-TTT PPxx

therefore

1-TCC' PP

Action of projective camera on conics

back-projection of a conic C to cone

C

coQ

coQ

CPPQ Tco

XQX0CPXPXCxx coTTTT

000CKK0|KC

0KQ

TT

T

co

example:

0CxxT

PXx with

Interpretation cone of a conic C

back-projection of a conic C to cone coQ

Images of smooth surfaces

The contour generator is the set of points X on S at which rays are tangent to the surface. The corresponding apparent contour is the set of points x which are the image of X, i.e. is the image of

The contour generator depends only on position of projection center, depends also on rest of P

Action of projective camera on quadrics

apparent contour of a quadric Q

TPPQC **

0lPPQlQ T*T*T

dual quadric1* QQ is a plane quadric:

0Q*T the set of planes tangent to Q

Let us consider only those planes that are backprojection of image lines

lPT

with its dual is1*CC

The plane containing the apparent contour of a quadric Q from a camera center O follows from pole-polar relationship

The cone with vertex V and tangent to the quadric Q is

TTCO (QV)(QV)-QV)QV(Q 0VQCO

back-projection to cone

=QO

What does calibration give?

xKd 1

0d0]|K[Ix

21-T-T

211-T-T

1

2-1-TT

1

2T

21T

1

2T

1

)xK(Kx)xK(Kx

)xK(Kx

dddd

ddcos

An image line l defines a plane through the camera center

with normal n=KTl measured in the camera’s Euclidean

frame. In fact the backprojection of l is PTl=KTl

The image of the absolute conic

KRd0

dO]|KR[IPXx

mapping between ∞ to an image is given by the planar

homogaphy x=Hd, with H=KR

absolute conic (IAC), represented by I3 within ∞ (w=0)

1-T-1T KKKKω 1TCHHC

(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT

(iv) K (Cholesky factorization)(v) image of circular points belong

to (image of absolute conic)

2T

21T

1

2T

1

ωxxωxx

ωxxcos

its image (IAC)

A simple calibration device

(i) compute Hi for each square

(corners (0,0),(1,0),(0,1),(1,1))

(ii) compute the imaged circular points Hi [1,±i,0]T

(iii) fit a conic to 6 imaged circular points

(iv) compute K from K-T K-1 through Cholesky factorization (= Zhang’s calibration method)

Orthogonality = pole-polar w.r.t. IAC

The calibrating conic

1T K1

11

KC

Vanishing points

λKdaλPDPAλPXλx

KdλKda limλ xlimvλλ

KdPXv

Vanishing lines

Orthogonality relation

2T

21T

1

2T

1

ωvvωvv

ωvvcos

0ωvv 2T

1

0lωl 2*T

1

Calibration from vanishing points and lines

Calibration from vanishing points and lines