Geometry of Images

Pinhole camera, projection A taste of projective geometry Two view geometry:

Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix

Stereo vision: 3D shape reconstruction from two views

Factorization: reconstruction from many views

Geometry of Images

Cameras

Camera obscura dates from 15th century First photograph on record shown in the book -

1822 Basic abstraction is the pinhole camera Current cameras contain a lens and a recording

device (film, CCD) The human eye functions very much like a

camera

Camera Obscura

"Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle

Why Not UsePinhole Camera

If pinhole is too big - many directions are averaged, blurring the image

Pinhole too small diffraction effects blur the image

Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen.

Lenses

Lenses

Lenses collect light from a large hole and direct it to a single point

Overcome the darkness of pinhole cameras But there is a price

Focus Radial distortions Chromatic abberations …

Pinhole is useful as a model

Pinhole Camera

Single View Geometry

f

X

P Y

Z

x

p y

f

∏x

p y

f

Notation

O – Focal center π – Image plane Z – Optical axis f – Focal length

Projection

x y f

X Y Z

f

x

y

Z

X

Y

Perspective Projection

f Xx

Zf Y

yZ

x Xf

y YZ

f Z

Homogeneous Coordinates

Orthographic Projection

•Projection rays are parallel•Image plane is fronto-parallel(orthogonal to rays)

•Focal center at infinity

x X

y Y

Scaled Orthographic ProjectionAlso called “weak perspective”

x sX

y sY

0

fs

Z

Pros and Cons of Projection Models Weak perspective has simpler math.

Accurate when object is small and distant. Most useful for recognition of objects.

Pinhole perspective much more accurate for scenes. Used in structure from motion.

When accuracy really matters, we must model the real camera Use perspective projection with other calibration

parameters (e.g., radial lens distortion)

World Cup 66: England-Germany

World Cup 66: Second View

World Cup 66: England-Germany

Conclusion: no goal (missing 3 inches)

(Reid and Zisserman, “Goal-directed video metrology”)

Euclidean Geometry

Answers the question what objects have the same shape (= congruent)

Same shapes are related by rotation and translation

Projective Geometry

Answers the question what appearances (projections) represent the same shape

Same shapes are related by a projective transformation

Perspective Distortion

Where do parallel lines meet?

Parallel lines meetat the horizon(“vanishing line”)

Line Perspective

1P

Pencil of rays

Perspective mapping

Plane Perspective

2P

Ideal points

Projective transformation can map ∞ to a real point

Coordinates in Euclidean Space

0 1 2 3 ∞

Not in space

Coordinates in Projective Line

-1 0 1 2 ∞

k(0,1)

k(1,0)

k(2,1)k(1,1)k(-1,1)

Points on a line P1 are represented as rays from origin in 2D,Origin is excluded from space

“Ideal point”

Coordinates in Projective Plane

k(0,0,1)

k(x,y,0)

k(1,1,1)

k(1,0,1)

k(0,1,1)

“Ideal point”

2D Projective Geometry: Basics A point:

A line:

we denote a line with a 3-vector

Points and lines are dual: p is on l if

Intersection of two lines:

A line through two points:

2 2( , , ) ( , )T Tx yx y z P

z z

0 ( ) ( ) 0x y

ax by cz a b cz z

0Tl p

1 2 ,l l

1 2p p

( , , )Ta b c

Cross Product

1 2 1 2 1 2

1 2 1 2 1 2

1 2 1 2 1 2

x x y z z y

y y z x x z

z z x y y x

0T Tw u v w u w v

Every entry is a determinant of the two other entries

w Area of parallelogram bounded by u and v

Ideal points

1 1 2 2

1 2 1 2

( , , ) , ( , , )

( )( , ,0) ( , ,0)

T Tl a b c l a b c

l l c c b a b a

Q: How many ideal points are there in P2?A: 1 degree of freedom family – the line at infinity

Projective Transformation (Homography) Any finite sequence of perspectivities is a

projective transformation Projective transformations map lines to lines Represented by an invertible 3x3 linear

transformation (up to scale), denote by H

, or Given homography H, how does it operate on

lines?

q Hp , 0q Hp

1: 0 ( )( ) 0T T Tl H l l p l H Hp

cos sin, , det 1

sin cosTR R R I R

Rotation:

Translation:x

y

tt

t

2 2, 1, (2)a b

R a b R SOb a

Euclidean Transformations (Isometries)

q Rp t

Hierarchy of Transformations

Isometry (Euclidean),

Similarity,

Affine, general linear

Projective,

0 1

R t

,0 1

a bsR tsR

b a

, (2)0 1

A tA GL

(3) : , 0H GL q Hp

Invariants

Length Area Angles Parallelism

Isometry √ √ √ √

Similarity ××

(Scale)√ √

Affine × × × √

Projective × × × ×

Perspective Projection

x Xf

y YZ

f Z

3

( )

P p

fp P p P

Z

Note: P and p are related by a scale factor, but it is a differentfactor for each point (depends on Z)

Two View Geometry

When a camera changes position and orientation, the scene moves rigidly relative to the camera

In two cases this results in homography: Camera rotates around its focal point The scene is planarIn this case the mapping from one image to the

second is one to one and depth cannot be recovered

In the general case the induced motion is more complex and is captured by what is termed “epipolar geometry”

Camera Rotation

' , 0

( )

'' ' ( ' ')

' ( ' )'

P RP t

Zp P P p

f

Zp P P p

f

Zp Rp p Rp

Z

(R is 3x3 non-singular)

IntuitivelyA sequence of two perspectivities

Algebraically

( )

1'

1, '

' ,'

T

TT

T

n P d aX bY cZ d

n PP RP t RP t R tn P

d d

H R tn P HPd

Zp Hp

Z

Planar ScenesScene

Camera 1

Camera 2

Two Views Related by HomographyTwo images are related by homography:

One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines

2 equations 4 points are required to recover H Depth cannot be recovered

' ,'

Zp Hp

Z

General Case: Epipolar Lines

epipolar lineepipolar line

Epipolar Plane

epipolar plane

epipolar lineepipolar lineepipolar lineepipolar line

BaselineBaseline

PP

OO O’O’

Epipole

Every plane through the baseline is an epipolar plane, and determines a pair of epipolar lines in the two images

Two systems of epipolar lines are obtained, each system intersects in a point, the epipole

The epipole is the projection of the center of the other camera

epipolar planeepipolar linesepipolar linesepipolar linesepipolar lines

BaselineBaselineOO O’O’

Epipolar Lines

epipolar plane

epipolar lineepipolar lineepipolar lineepipolar line

BaselineBaseline

PP

OO O’O’

To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some worldcoordinates as follows:

' ' 0T

OP OO O P

Epipolar Lines

Set world coordinates around the first camera

What to do with O’P? Every rotation changes the observed coordinate in the second image

We need to de-rotate to make the second image plane parallel to the first

Replacing by image points

' ' 0T

OP OO O P

' 0TP t RP

, 'P OP t OO

' 0Tp t Rp

Essential Matrix

Denote this by:

Then

Define , then

E is called the “essential matrix”

0

0

0

x y z z y

y z x z x

z x y y x

t x t z t y t t x

t y t x t z t t y

t z t y t x t t z

t p t p

' ' 0T Tp t Rp p t Rp

E t R

' 0Tp Ep

Essential Matrix

E is rank 2. Its (right and left) null spaces are the two epipoles

is linear and homogeneous in E, E can be recovered up to scale using 8 points

The additional constraint detE=0 reduces the needed points to 7

In fact, there are only 5 degrees of freedom in E, 3 for rotation 2 for translation (up to scale), determined by

epipole

' 0Tp Ep

Internal Calibration

Camera parameters may be unknown:

(cx,cy) camera center, (ax,ay) pixel dimensions, b skew

Radial distortions are not accounted for

0

0 0 1

x x

y y

a b c

K a c

q Kp

Fundamental Matrix

F, the fundamental matrix, too is rank 2 F has 7 d.o.f. (9 entries, homogeneous, and

detF=0)

1 1

1

1

' 0 ( ) ( ') 0

( ) ' 0

T T

T T

T

p Ep K q E K q

q K EK q

F K EK

Summary

Homography Perspective Orthographic

Form

Shape One-to-one (Group)

Concentric epipolar lines

Parallel epipolar lines

D.o.f. 8(5) 8(5) 5

Eqs/pnt 2 1 1

Minimal configuration

4 5+ (8, linear) 4

Depth No Yes, up to scale

No, third view required

' 0Tp Ep ' 0Tp Ep 'p Hp