Camera Projection Matrix

Slides by HyunSoo Park

x XR 3 3 tK

=

x x

x y

Xu f s p r r r t

Yz v f p r r r t

Zr r r t

img 11 12 13 1

img 21 22 23 2

31 32 33 31 11

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-

to-find-their-vanishing-point.html

First person coordinate system World coordinate system

Camera Model

Ground plane

Camera Model (1st Person Perspective)Camera

Ground plane

Camera Model (1st Person Perspective)Camera

1 1

x

y

u f p X

v f p Y

Z

Recall camera projection matrix:

+

Camera intrinsic parameter

: metric space to pixel space

Ground plane

Camera Model (1st Person Perspective)Camera

1 1

x

y

u f p X

v f p Y

Z

K

Recall camera projection matrix:

+

Camera intrinsic parameter

: metric space to pixel space

Ground plane

Camera Model (1st Person Perspective)Camera

1 1

x

y

u f p X

v f p Y

Z

K

Recall camera projection matrix:

3D world (metric)2D image (pix)

Ground plane

Camera Model (1st Person Perspective)Camera

1 1

x

y

u f p X

v f p Y

Z

K

Recall camera projection matrix:

3D world (metric)2D image (pix)

Origin at camera

-1

1

K XC

u X

v Y

Z

CX

Ground plane

Camera Model (multiple 1st Person Perspective)Camera

1 1

x

y

u f p X

v f p Y

Z

K

Recall camera projection matrix:

3D world (metric)2D image (pix)

Origin at camera

1

1 1

-1

1 1 C

11

K X

u X

v Y

Z

Where is origin?1C

X

2CX

2

2 2

-1

2 2 C

21

K X

u X

v Y

Z

1 1

x

y

u f p X

v f p Y

Z

K

Recall camera projection matrix:

3D world (metric)2D image (pix)

X

Ground plane

Camera

X3D world

Origin at world coordinate

Camera Model (3rd Person Perspective)

1 1

x

y

u f p X

v f p Y

Z

K

Recall camera projection matrix:

3D world (metric)2D image (pix)

X

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation matrix)

X

1C

WR

2C

WR

1

x1 x2 x3

C y1 y2 y3

z1 z2 z3

=

X X

r r r

r r r

r r r

Coordinate transformation from world to camera:

X

3D world Camera

1CX

Coordinate Transform (Rotation)

2D coordinate transform:

World

x

y

= ( , )X x y

Coordinate Transform (Rotation)

= ( , )X x y

2D coordinate transform:

World

Camera

C C C= ( , )X x y

C

C

cos sin=

sin cos

x

y

x

y

?

Coordinate Transform (Rotation)

= ( , )X x y

2D coordinate transform:

World

Camera

C C C= ( , )X x y

C

C

cos sin=

sin cos

x

y

x

y

Coordinate Transform (Rotation)

= ( , )X x y

2D coordinate transform:

World

Camera

C C C= ( , )X x y

C

C

cos sin=

sin cos

x

y

x

y

2 2cos sindet = cos sin 1

sin cos

Coordinate Transform (Rotation)

= ( , )X x y

2D coordinate transform:

World

Camera

C C C= ( , )X x y

C

C

cos sin=

sin cos

x

y

x

y

rX

rX

: x axis of camera seen from the worldrX

Coordinate Transform (Rotation)

= ( , )X x y

2D coordinate transform:

World

Camera

C C C= ( , )X x y

C

C

cos sin=

sin cos

x

y

x

y

rX

rX

: x axis of camera seen from the worldrX

: y axis of camera seen from the worldYr

Yr

rY

Coordinate Transform (Rotation)

= ( , )X x y

2D coordinate transform:

World

Camera

C C C= ( , )X x y

C

C

cos sin=

sin cos

x

y

x

y

: x axis of world seen from the camera1r

: y axis of world seen from the camera2r

r1

r1

2r 2

r

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

= ( , , )X x y z

World

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

?

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

?

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

Degree of freedom?

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

World

RC

WSO(3)

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C C

W W 3 W, det 1 R R I R• Orthogonal matrix

Degree of freedom?

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

= ( , , )X x y z

Camera

C C C C= ( , , )X x y z

World

RC

WSO(3)

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C C

W W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

rX

Yr

Zr

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RC

WSO(3)

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C C

W W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

rX

Yr

Zr

rX

Yr

Z X Y= r r r

rX

: camera x axis seen

from the world coord.Zr

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RC

WSO(3)

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C C

W W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

Yr

Zr

rX

Yr

Z X Y= r r r

: camera y axis seen

from the world coord.

rX

Yr

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RC

WSO(3)

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C C

W W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

1r

2r

3 1 2= r r r

1r : world x axis seen

from the camera

coord.

r1 2

r3r

Ground plane

Camera

3D world

Origin at world coordinate

Coordinate Transform (Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

RC

WSO(3)

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

T

C C C

W W 3 W, det 1 R R I R• Orthogonal matrix

• Right hand rule

1r

2r

3 1 2= r r r

1r : world x axis seen

from the camera

coord.

r1 2

r3r

2r : world y axis seen

from the camera

coord.

3r

x1 x2 x3

y1 y2 y3

z1 z2

C

z3

C

C1 1

1

x

y

x

y

u f p

v f p

f p r r r

f p r r r

r

X

Y

Zr r

X

Y

Z

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Pure Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

1r : world x axis seen

from the camera

coord.

r1 2

r3r

2r : world y axis seen

from the camera

coord.

3r

K

Camera projection of world point:

x1 x2 x3

y1 y2 y3

z1 z2

C

z3

C

C1 1

1

x

y

x

y

u f p

v f p

f p r r r

f p r r r

r

X

Y

Zr r

X

Y

Z

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Pure Rotation)

X

1C

WR

Coordinate transformation from world to camera:

Camera

World

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

1r : world x axis seen

from the camera

coord.

r1 2

r3r

2r : world y axis seen

from the camera

coord.

3r

K

Camera projection of world point:

K

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Euclidean Transform)

X

Coordinate transformation from world to camera:

x1 x2 x3

C

y1 y2 y3 W

z1 z

C

2 z3

= =

X XRX

r r r

r r r

r r r

C C

W,R t

World

= ( , , )X x y z

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Euclidean Transform)

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C C

W y1 y2 y3 y

z1 z2 z3 z

C= =

1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C

= ( , , )X x y z

C C

W,R t

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection (Euclidean Transform)

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C C

W y1 y2 y3 y

z1 z2 z3 z

C= =

1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C

= ( , , )X x y z

C C

W,R t

Ct

CtWhere is translation from world to camera

seen from camera.

Ground plane

Camera

3D world

Origin at world coordinate

Geometric Interpretation

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C C

W y1 y2 y3 y

z1 z2 z3 z

C= =

1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C

= ( , , )X x y z

C C

W,R t

Ct

Ctwhere is translation from world to camera

seen from camera.

Rotate and then, translate.

Ground plane

Camera

3D world

Origin at world coordinate

Geometric Interpretation

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C C

W y1 y2 y3 y

z1 z2 z3 z

C= =

1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C

= ( , , )X x y z

C C

W,R t

Ctwhere is translation from world to camera

seen from camera.

Rotate and then, translate.

x1 x2 x3 x

C

W y1 y2 y3 y

z1 z2 z3 z

C

1

= = 11

1

XXR CX

r r r -C

r r r -C

r r r -C

cf) Translate and then, rotate.

where is translation from world to camera

seen from world.C

C

Ground plane

Camera

3D world

Origin at world coordinate

Camera Projection Matrix

X

Coordinate transformation from world to camera:

Camera

x1 x2 x3 x

C C

W y1 y2 y3 y

z1 z2 z3 z

C= =

1

RX

XX t

r r r t

r r r t

r r r t

World

= ( , , )X x y zC C C C

= ( , , )X x y z

C C

W,R t

C

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z

C

C

C1

1

1

1

x

y

x

y

u f p

v f p

f p r r r t

f p r r r t

r r r t

X

Y

Z

X

Y

Z

K

Camera projection of world point:

K C

WR

Ct

Image Projection: Sanity Check

r

= 0

0

C

-r 0 -1 0

= 0 0 -1

1 0 0

R

Image Projection

r

r

= 0

0

C

-r 0 -1 0

= 0 0 -1

1 0 0

R

Image Projection

r

r

= 0

0

C

-r 0 -1 0

= 0 0 -1

1 0 0

R

cos

= sin

0

C

r

r

-sin cos 0

= 0 0 -1

-cos -sin 0

R

Image Projection K = [200 0 100;0 200 100;0 0 1];

radius = 5;

theta = 0:0.02:2*pi;

for i = 1 : length(theta)camera_offset = [radius*cos(theta(i)); radius*sin(theta(i)); 0];camera_center = camera_offset + center_of_mass';

rz = camera_center-center_of_mass';rz = rz / norm(rz);ry = [0 0 1]';rx = Vec2Skew(ry)*rz; % cross productR = [rx'; ry'; rz'];C = camera_center;P = K * R * [ eye(3) -C];

proj = [];for j = 1 : size(sqaure_point,1)

u = P * [sqaure_point(j,:)';1];proj(j,:) = u'/u(3);

end end

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

11 12 13 14

21 22 23 24

31 32 33 341

1

Xu p p p p

Yv p p p p

Zp p p p

What does each number mean?

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

What is point at infinity in world x direction?

X

0

0

1

X

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

What is point at infinity in world x direction?

X

0

0

1

X

This point is at infinite but finite in image.

Xu 11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Point at infinity

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

What is point at infinity in world x direction?

X

0

0

1

X

This point is at infinite but finite in image.

Xu 11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

X

0

0

1

X

Xu

11 14 11

31 34 31

21 24 21

31 34 31

+lim

+

+lim

+

X

X

p X p pu

p X p p

p X p pv

p X p p

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

X

0

0

1

X

Xu

11 14 11

31 34 31

21 24 21

31 34 31

+lim

+

+lim

+

X

X

p X p pu

p X p p

p X p pv

p X p p

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

11

X 21

311

u

u p

v p

p

11

X 21

31

u

p

p

p

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

Xu

12 14 12

32 34 32

22 24 22

32 34 32

+lim

+

+lim

+

X

X

p Y p pu

p Y p p

p Y p pv

p Y p p

11 12 13 14

21 22 23 24

31 32 33 34

0

01

1

u p p p p

v p p p p

p p p p

12

Y 22

321

u

u p

v p

p

12

Y 22

32

u

p

p

p

Yu

Ground plane

Camera

3D world

Geometric Interpretation

C C

W,R t

x1 x2 x3 x

y1 y2 y3 y

z1 z2 z3 z 11 1

x

y

u f p r r r t

v f p r r r t

X

Y

Zr r r t

Camera projection of world point:

K C

WR

Ct1

=

XP

13 14 13

33 34 33

23 24 23

33 34 33

+lim

+

+lim

+

X

X

p Z p pu

p Z p p

p Z p pv

p Z p p

11 12 13 14

21 22 23 24

31 32 33 34

0

0

11

u p p p p

v p p p p

p p p p

13

Z 23

331

u

u p

v p

p

13

Z 23

33

u

p

p

p

Celestial Navigation

Far far away: point at infinity

Earth My location

Practiceimg

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp

img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

Practiceimg

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp

img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P= KR I C

img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp

img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P= KR I C

R = [-0.8496 0.0498 0.5731-0.3216 -0.8203 -0.40670.4180 -0.5299 0.6835];

C = [0.00700.7520-0.2738];

P = K * R * [eye(3) -C]u_x = P(1:2,1)/P(3,1)u_y = P(1:2,2)/P(3,2)u_z = P(1:2,3)/P(3,3)

[-0.2374 0.0578]/ 0.0004

=

[-593.5 144.5]

Practiceimg

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp

img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P= KR I C

R = [-0.8496 0.0498 0.5731-0.3216 -0.8203 -0.40670.4180 -0.5299 0.6835];

C = [0.00700.7520-0.2738];

P = K * R * [eye(3) -C]u_x = P(1:2,1)/P(3,1)u_y = P(1:2,2)/P(3,2)u_z = P(1:2,3)/P(3,3)

[-0.9565 -1.5763] / -0.0005

=

[1,913 3,152]

img

m

ccd

3840= = 0.00153 1224pix

0.0048

Wf f

W

img

x

3840= = 1920pix

2 2

Wp

img

y

2160= = 1080pix

2 2

Hp

0.0070

0.7520

-0.2738

C

-0.8496 0.0498 0.5731

-0.3216 -0.8203 -0.4067

0.4180 -0.5299 0.6835

R

3

-0.2374 -0.9565 2.0138 1.2723

0.0578 -1.5763 0.2404 1.2508

0.0004 -0.0005 0.0007 0.0006

P= KR I C

R = [-0.8496 0.0498 0.5731-0.3216 -0.8203 -0.40670.4180 -0.5299 0.6835];

C = [0.00700.7520-0.2738];

P = K * R * [eye(3) -C]u_x = P(1:2,1)/P(3,1)u_y = P(1:2,2)/P(3,2)u_z = P(1:2,3)/P(3,3)

[2.0138 0.2404]/0.0007

=

[2877 343]

Z direction in the world coordinate system

Z direction in the world coordinate system

uv

Z direction in the world coordinate system

z point at infinity

uv

: z vanishing pointvz

z =T

0 0 1 0

Calibration via Vanishing Points

v1 v3

v2

v1

v3

K v1 3

K v1 1

Strong perspectivenessWeak perspectiveness

Orthographic CameraAffine camera:

x1 x2 x1 x2

A y1 y2 y1 y2

0 0 0 0

0 0 0 0

1 0 0 0 1 0 0 0 1

P

x x

y y

f p r r f / d p r r

f p r r d f / d p r r

d

1f

Orthographic camera:

X Y0 p p

Orthographic CameraAffine camera:

x1 x2 x1 x2

A y1 y2 y1 y2

0 0 0 0

0 0 0 0

1 0 0 0 1 0 0 0 1

P

x x

y y

f p r r f / d p r r

f p r r d f / d p r r

d

1f

Orthographic camera:

X Y0 p p

x1 x2 x3

O y1 y2 y3

0

0

0 0 0 1

P

r r r

r r r

Lens Radial Distortion