Camera Projection Model

3D Point Projection (Pixel Space)

O

Projection plane ( , , )X Y Z

Z

img x

Xu f p

Z

img y

Yv f p

Z

CCD sensor (mm) Image (pixel)

u vccd ccd( , )

u vimg img( , )

ccdw

ccdh

(0,0)

(0,0) imgw

imgh

x yp p( , )

img imgu v( , )

f

img xZu fX p Z

img yZv fY p Z

img

img

11

x

y

u f p X

v f p Y

Z

K

+

Camera intrinsic parameter : metric space to pixel space

Metric space Pixel space

2D Inverse Projection

O

Projection plane

img

img

11

x

y

u f p X

v f p Y

Z

K

Metric space

f

img imgu v( , )

Pixel space

u X

v Y

Z

K

img

-1img

1

u

vK

img

-1img

1

( , , )X Y Z

2D point == 3D ray

3D ray

The 3D point must lie in the 3D ray passing through the origin and 2D image point.

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

Degree of freedom?

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

rX : camera x axis seen from the world coord.

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

rX : camera x axis seen from the world coord.

Yr : camera y axis seen from the world coord. Zr

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

1r

2r

3 1 2= r r r1r : world x axis seen from

the camera coord.

r1 2r 3r

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RCW SO(3)

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C CW W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

1r

2r

3 1 2= r r r1r : world x axis seen from

the camera coord.

r1 2r 3r

2r : world y axis seen from the camera coord.

3r

x1 x2 x3

y1 y2 y3

z1 z2

C

z3

C

C1 1

1

x

y

x

y

u f p

v f p

f p r r r

f p r r r

r

X

Y

Zr r

X

Y

Z

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Pure Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

1r : world x axis seen from the camera coord.

r1 2r 3r

2r : world y axis seen from the camera coord.

K

Camera projection of world point:

3r

x1 x2 x3

y1 y2 y3

z1 z2

C

z3

C

C1 1

1

x

y

x

y

u f p

v f p

f p r r r

f p r r r

r

X

Y

Zr r

X

Y

Z

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Pure Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

x1 x2 x3

Cy1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

1r : world x axis seen from the camera coord.

r1 2r 3r

2r : world y axis seen from the camera coord.

K

Camera projection of world point:

K

3r

Ground plane

Camera

3D world

Origin at world coordinate

Euclidean Transform=Rotation+Translation

X

C CW ,R t

World

= ( , , )X x y z

Ground plane

Camera

3D world

Origin at world coordinate

X

Camera

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

Euclidean Transform=Rotation+Translation

Ground plane

Camera

3D world

Origin at world coordinate

X

Camera

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

Euclidean Transform=Rotation+Translation

Ground plane

Camera

3D world

Origin at world coordinate

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

Euclidean Transform=Rotation+Translation

C t

C twhere is the world orgin seen from camera.

Ground plane

Camera

3D world

Origin at world coordinate

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

Euclidean Transform=Rotation+Translation

C t

C twhere is the world orgin seen from camera.

Ground plane

Camera

3D world

Origin at world coordinate

Geometric Interpretation

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

C t

Rotate and then, translate.

C twhere is the world orgin seen from camera.

Ground plane

Camera

3D world

Origin at world coordinate

Geometric Interpretation

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C CW y1 y2 y3 y

z1 z2 z3 z

C = =1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C= ( , , )X x y z

C CW ,R t

Rotate and then, translate.

x1 x2 x3 x

CW y1 y2 y3 y

z1 z2 z3 z

C

1

= = 11

1

XXR CX

r r r -C

r r r -C

r r r -C

cf) Translate and then, rotate.

where is the camera location seen from world. C

C

C twhere is the world orgin seen from camera.

Projective Line

Point-Point in Image

1 1 0 au bv c

1 1( , )u v

1xx2

2 2( , )u v

2 2 0 au bv c

x lT1 0 x lT

2 0

1

1 1

1

x

u

v2

2 2

1

x

u

v

a

b

c

lwhere

A 2D line passing through two 2D points:

xl 0

x

T1

T2

0 0 A

null A l x x1 2=or

2x3

Two 2D lines in an image intersect at a 2D point:

Line-Line in Image

l xT1 0 l xT

2 0

lx 0

l

T1

T2

1 2= x l l

1

1 1

1

l

a

b

c

( , )u vx

2

2 2

2

l

a

b

c

1 1 1 0 a u b v c2 2 2 0 a u b v c

1

x

u

v1

1 1

1

l

a

b

c

where

2

2 2

2

l

a

b

c

2x3

0 0 A

null A

or

(1804,934)

Vanishing Point

11l

1l 2

Ground plane

1u

2u3u

4u

Parallel lines:

11 4 3 l u u 12 1 2 l u u

Vanishing Point

11l

Ground plane

4u

Parallel lines:

11 4 3 l u u 12 1 2 l u u

21 4 1 l u u 22 3 4 l u u

2l 1

l221l 2

1u

2u3u

Vanishing Point

11l

x1

Ground plane

4u

Parallel lines:

11 4 3 l u u 12 1 2 l u u

21 4 1 l u u 22 3 4 l u u

2l 1

l221l 2

1u

2u3u

x l l1 11 12

Vanishing points:

x l l2 21 22

x2

Vanishing Point

11l

1v

Ground plane

4u

Parallel lines:

11 4 3 l u u 12 1 2 l u u

21 4 1 l u u 22 3 4 l u u

2l 1

l221l 2

1u

2u3u

1 11 12 v l l

Vanishing points:

2 21 22 v l l

2v Vanishing line:

1 2 l v vl

Where was I (how high)?

Taken from my hotel room (6th floor) Taken from beach

Vanishing point

Vanishing point

Multiple vanishing point

Vanishing point

Vanishing point

Vanishing line: Horizon

Vanishing line for horizon

Keller Hall

Vanishing point

Vanishing point

What can vanishing line tell us about me? • Horizon • Camera pitch angle (looking down) • Camera roll angle (tilted toward right)

Vanishing line for horizon

Ground plane

Single Vanishing Point Camera

3D world

r1

r2

r3

v

0 0

0 0

1

1 0

Z

C1 2 3 W

0

0

1

0

v K r r r t

-13

K v r

3v Kr

Ground plane

Single Vanishing Point Camera

3D world

r1

r2

r3

v

0 0

0 0

1

1 0

Z

C1 2 3 W

0

0

1

0

v K r r r t

-13

K v r

3v Kr

K v

rK v

-1

3 -1 because r3 is a unit vector.

Z vanishing point tells us about the surface normal of the ground plane

Ground plane

Single Vanishing Point Camera

3D world

r1

r2

r3

v

0 0

0 0

1

1 0

Z

C1 2 3 W

0

0

1

0

v K r r r t

-13

K v r

3v Kr

K v

rK v

-1

3 -1 because r3 is a unit vector.

Z vanishing point tells us about the surface normal of the ground plane

Ground plane

Rotation ambiguity

Ground plane

Two Vanishing Points Camera

X

1

0

0

0

v K r r r t X

Kr

CX 1 2 3 W

1

0

1

0

0

Y

3D world

r1

r2

r3

Yv

vX

3D world

Ground plane

Two Vanishing Points Camera

X

1

0

0

0

CY 1 2 3 W

2

v K r r r t Y

Kr

v K r r r t X

Kr

CX 1 2 3 W

1

0

1

0

0

Y

3D world

r1

r2

r3

Yv

vX

3D world

Ground plane

Two Vanishing Points Camera

X

1

0

0

0

-1 -1

X Y1 2-1 -1

X Y

,

K v K v

r rK v K v

3 1 2 r r r : Orthogonality constraint

CY 1 2 3 W

2

v K r r r t Y

Kr0

1

0

0

Y

3D world

r1

r2

r3

v K r r r t X

Kr

C

X 1 2 3 W

1

Yv

vX

3D world

Single View Metrology

Cross Ratio

Ground plane

A B

C D

Reference points

Target points

AC ac a c

BC bc

BD bd b d

AD ad a db c

a b

c

d a' b' c'

d'

Cross ratio (perspective transformation invariant)

v

sh_f

jp_f

jp_h p2

p3

p1

p4

h_prime = norm(p1-p2); h_R = norm(p1-p3); h_prime_inf = norm(p4-p2); h_inf = norm(p4-p3); H = H_R * h_prime * h_prime_inf / h_R / h_inf

ComputeHeightFromCrossRatio.m

h

Rh

vZ

h

h

R

R

h

hH

h

hH

H = 1.6779 Ground truth: 1.7m

Homography

Perspective Transform (Homography)

I1 I2

x 11 12 13 x x

y 21 22 23 y y

31 32 11 1 1

H

v h h h u u

v h h h u u

h h

: General form of plane to plane linear mapping

Ground plane

Camera plane

u

Perspective Transform (Homography)

u K R t X

0

1

X =

X

Y

uGround plane Camera plane

1 2 30

1

u K r r r t

X

Y

1 2

1

u K r r t

X

Y

x

y

11

H

u X

u Y

Perspective Transform (Homography)

I1 I2Invariant properties

• Cross ratio

• Concurrency

• Colinearity

x 11 12 13 x x

y 21 22 23 y y

31 32 11 1 1

H

v h h h u u

v h h h u u

h h

Degree of freedom

8 (9 variables 1 scale)

Hierarchy of Transformations

Euclidean (3 dof) Similarity (4 dof) Affine (6 dof) Projective (8 dof)

• Length • Angle • Area

• Length ratio • Angle

• Parallelism • Ratio of area • Ratio of length

• Cross ratio • Concurrency • Colinearity

x

y

cos sin

sin cos

1

t

t

θ θ

θ θ

x

y

cos sin

sin cos

1

t

t

θ θ

θ θ

11 12 13

21 22 23

0 0 1

a a a

a a a11 12 13

21 22 23

31 32 1

h h h

h h h

h h

Image Transform via Plane

Keller entrance left Keller entrance right

Plane

Image Transform via 3D Plane

X

YX =

0

1

1 2

1 2

=

1

=

1

u K r r t

v K r r t

X

Y

X

Y

u

u K R t X

v

How are two image coordinates (u,v) related?

-1-1 -1 -1

1 2 1 2

-1 -11 2 1 2

= =

1

=

=

r r t K u r r t K v

v K r r t r r t K u

v Hu

X

Y

Camera

Image Transform by Pure 3D Rotation

X

Camera

Image Transform by Pure 3D Rotation

X

2 = v KRX

1 = u KX

-1 T -11 2= = X K u R K v

-1=v KRK u

-1=H KRK-1=R K HK

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

11 12 13

31 32 33

21 22 23

31 3

x y

x

x y

x y

y

x y2 33

+

+ +

+

+ +

u uv

u u

u uv

h h h

h h h

h

h u

h

h hu

h

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

11 12 13

31 32 33

21 22 23

31 3

x y

x

x y

x y

y

x y2 33

+

+ +

+

+ +

u uv

u u

u uv

h h h

h h h

h

h u

h

h hu

h

x y x x y x x

x y

11 12 13 31 32 33

21 22 23 31 32 3x y y y y3

+ + + 0

+ + + + 0

h h h h h h

h

u u u v u v v

u u u v u vh h vh h h

Unknowns: 11 33, ,h h

Equations: 2 per correspondence

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

11 12 13

31 32 33

21 22 23

31 3

x y

x

x y

x y

y

x y2 33

+

+ +

+

+ +

u uv

u u

u uv

h h h

h h h

h

h u

h

h hu

h

11 12 13 31 3x y x x y x x

x y x

2 33

21 22 23 31 32 33y y y y

+ 0

+ 0

h h h h h h

h

u u u v u v v

u u u v u vh h vh h h

11

12

13

21

22

x y x x y x x

x y x y y y y23

31

32

33

1 0

1 0

h

h

h

h

h

h

u u -u v -u v -v

u u -u v -u v

h

-v

h

h

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

11 12 13

31 32 33

21 22 23

31 3

x y

x

x y

x y

y

x y2 33

+

+ +

+

+ +

u uv

u u

u uv

h h h

h h h

h

h u

h

h hu

h

11 12 13 31 3x y x x y x x

x y x

2 33

21 22 23 31 32 33y y y y

+ 0

+ 0

h h h h h h

h

u u u v u v v

u u u v u vh h vh h h

11

12

13

21

22

x y x x y x x

x y x y y y y23

31

32

33

1 0

1 0

h

h

h

h

h

h

u u -u v -u v -v

u u -u v -u v

h

-v

h

h

A x 02x9

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

11

12

13

21

22

x y x x y x x

x y x y y y y23

31

32

33

1 0

1 0

h

h

h

h

h

h

u u -u v -u v -v

u u -u v -u v

h

-v

h

h

A x 02x9

How many correspondences are needed?

Homography Computation

x 11 12 13 x

y 21 22 23 y

31 32 33

=

1 1

v h h h u

v h h h u

h h h

How many correspondences are needed? 4

11

12

13

21

22

x y x x y x x

x y x y y y y23

31

32

33

1 0

1 0

h

h

h

h

h

h

u u -u v -u v -v

u u -u v -u v

h

-v

h

h

A x 08x9

:end= nullx V A

Epipolar Geometry

X

u

Bob Alice

v

u v

Properties of Fundamental Matrix

Fundamental Matrix

P K I 0bob 3 3 1

P K R talice

• Transpose: if F is for Pbob, Palice, then FT is for Palice, Pbob.

• Epipolar line:

• Epipole:

• rank(F)=2: degree of freedom 9 (3x3 matrix)-1 (scale)-1 (rank)=7

ealiceebob

ul Fu vl F vT

ealiceebob

-T -1

F K t RK

rank 2 matrix

Fe 0bob = F e 0Talice =

u v

Bob Alice

Essential Matrix

Essential Matrix:

-T -1 -T -1

=

=

F F(R,t)

K t RK K EK

T=E K FKCalibrated fundamental matrix

=

E t Rwhere

Property of essential matrix:

T T

1

= 1

0

E UDV = U V

HW #4 RANSAC Fundamental matrix

v

Bob Alice

X

bobP aliceP

u

Baseline

Triangulation General camera pose

1 bob=1 1

u XP

Two 3D vectors are parallel.

bob1 1

u XP 0

bob1 1

u XP 0 : Knowns

: Unknowns

alice1

vP

mikeP

mike1

wP

PnP

u

X

3D-2D correspondence: u X

= P K R t (3)R SOwhere

11

12

13

14x x x x1 1 1 1 1 1 1

21y y y y1 1 1 1 1 1 1

22

23x x x x

24y y y y

31

32

33

34

1

1

1

1

m m m m m m m

m m m m m m m

p

p

p

pX Y Z -u X -u Y -u Z -u

pX Y Z -u X -u Y -u Z -u

p

pX Y Z -u X -u Y -u Z -u

pX Y Z -u X -u Y -u Z -u

p

p

p

p

0

0

0

0

x 11 12 13 14

31 32 33 34

+ +

+ +

p X p Y p Z pu

p X p Y p Z py 21 22 23 24

31 32 33 34

+ +

+ +

p X p Y p Z pu

p X p Y p Z p

2mx12

Bundle Adjustment

Nonlinear System

E

x

T T( ) ( )( )

x xx b

x x

f ff

Find x such that the following equation is satisfied:

How?

Strategy: Given x, move such that x

x x x

( ) ( ) x x xE E

Taylor expansion:

T T( ) ( ) ( )( )

x x xx x b

x x x

f f ff

T T( ) ( ) ( )

( )

x x xx b x

x x x

f f ff

= x TA bTA A-1

Cf.)

( )( ) ( ) H.O.T.

xx x x x

x

ff f

X

bobP

uu

Point Jacobian

-1

T T( ) ( ) ( )( )

x x xx b x

x x x

f f ff

X

Black: given variables Red: unknowns

XKR -C

u

v

w

where

( )

Xf

u

w

v

w

2 2

geom

Ev

wx y

u

w

2

2

( )

X XX

X X

X X

u ww u

u

w w

v u wv v

ww

f

KR

X

u

v

w

-1

T T( ) ( ) ( )( )

x x xx b x

x x x

f f ff

2

2

( )

X XX

X X

X X

u ww u

u

w w

v u wv v

ww

f

Damping factor (Levenberg-Marquardt algorithm)

-1

T T( ) ( ) ( )( )

x x xx I b x

x x x

f f ff

X

bobP

uu

Camera Jacobian

p

-1

T T( ) ( ) ( )( )

p p pp b p

p p p

f f ff

Black: given variables Red: unknowns

X CK -R

u

v

w

where

2 2

geom

Ev

wx y

u

w

-KC

R

u

v

w

11 1 3 13

1 3 22 23

1 3 1 3

- -

- -

-

K X C 0 K X C

0 K X C K X CR

0 0 X C

u

v

w

1

2

3

=

C

rp

r

r

3+9 parameters

X

bobP

uu

Camera Jacobian

p

Black: given variables Red: unknowns

X CK -R

u

v

w

where

2 2

geom

Ev

wx y

u

w

-KC

R

u

v

w

11 1 3 13

1 3 22 23

1 3 1 3

- -

- -

-

K X C 0 K X C

0 K X C K X CR

0 0 X C

u

v

w

=

Cp

q

3+4 parameters

R

q R q

u u

v v

w w

: Chain rule

Quaternion jacobian

v

Bob Alice

1X

bobP aliceP

u

pp

X

Black: given variables Red: unknowns

Camera & Point Jacobian ( ) ( )

p X

p ,X p ,XJ J J

p Xj i j i

ij ij ij

j i

f f

J1,bobpJ 2 70 1,bobXJ

1,aliceXJ1,alicepJ2 70

v

Bob Alice

1X

bobP aliceP

u

pp

X

Black: given variables Red: unknowns

Camera & Point Jacobian ( ) ( )

p X

p ,X p ,XJ J J

p Xj i j i

ij ij ij

j i

f f

J1,bobpJ 2 70 1,bobXJ

1,aliceXJ1,alicepJ2 70

2X

2,bobpJ 2 70 2,bobXJ

2,aliceXJ2,alicepJ2 70

2 30

2 30

2 30

2 30

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 3x4

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

J JTsize of : 24x24

x J J J b f(x)-1T T

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

J JTsize of : 24x24

x J J J b f(x)-1T T

Main computational bottle neck

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3