ROBOT VISION

Lecture Notes

로봇 비젼

강의 노트

Course No. 15678

Credits Hours 3

Instructor:

Professor

Choi, Tae-Sun

Course outline:

The principles of the machine/robot vision are introduced. It covers image formation,

pattern classification, motion and optical effect for object recognition. In addition, the

design technology of the robot vision system with optical device is studied.

Prerequisite: Digital Signal Processing, Image Processing.

Textbook and References: 1. Robot Vision, B. K. P. Horn, MIT Press.

2. Computer Vision, Dana Ballard and Christopher

Brown, Prentice Hall.

Course Schedule

1. Image Formation & Image Sensing

2. Binary Images: Geometrical Properties

3. Binary Images: Topological Properties

4. Regions & Image Segmentation

5. Image Processing: Continuous Images

6. Image Processing: Discrete Images

7. Edge & Edge Finding

8. Lightness & Color

9. Reflectance Map: Photometric Stereo

10. Reflectance: Shape from Shading

11. Motion Field & Optical Flow

12. Photogrammetry & Stereo

13. Pattern Classification

14. Polyhedral Objects

15. Extended Gaussian Images

16. Passive Navigation & Structure from Motion

1

1. Introduction

I. Human vision system

Figure 1—1

• Rod : more sensitive to light

Scotopic vision

Slender receptor

• Cone : shorter & thicker in structure

Photopic vision : R.G.B

6.5 million

• Fovea : Density of cones is greatest

The region of sharpest photopic vision Camera visual system

2

Figure 1—2

i. Photometric information

brightness : (ex: bright green, dark green)

color (ex, R. G. B)

ii. Geometric information

shape

Figure 1—3

2

NTSC : US standard for TV picture and audio coding and transmission

(timing calculation)

For one frame : 512×512 = 256k Byte

For RGB : 3

For real time : 30 frames/sec

256k × 3 × 30 = 22.5M Byte/sec

180 M bit/sec

180 Mbps T1 1.544 Mbps

Compression ratio : 180

1201.544

≈

Wavelet compression

MPEG I, II, IV

Compression JPEG

Fractal

H.261

II. Robot vision

RV

Computer graphics

Image processing

compressio

3

Figure 1—4

Figure 1—5

4

“Scene description”

Under constrained problems : Inverse problem

for example 2 equations

3 unknown variables are given

1. Image analysis :a detailed but undigested description

2. Scene analysis : more parsimonious, structured descriptions suitable for

decision marking

The apparent brightness of a surface depends on three factors

1. Microstructure

2. Distribution of incident light

3. Orientation of the surface w. r. t light source & observer

5

Figure 1—6

“Lambertian Surface”

One that appears equally bright from all viewing directions and reflects all incident light,

absorbing none

6

2. Image Formation & Image sensing

What determines where(Shape-Geometric) the image of some point will appear?

What determines how(Brightness, Color – Photonetiz) bright the image of some surface

will be?

I. Two Aspects of Image Formation

i. Perspective projection

Figure 2—1

7

ii. Orthographic Projection

2 2

2 20

' '' (magnification)

x yfm

z x y

δ δ

δ δ

+= =

+

In case the depth range of a scene is small relative to the distance z0

Figure 2—2

⇒

=

=

myy

mxx

'

'

if m=1

=

=

yy

xx

'

'

=> This orthographic projection can be modeled by rays parallel to the optical axis

(rather than one passing through the origin)

(a) (b)

Figure 2—3

I

pE

δ

δ= [watt/m]

Where δP is the power of the radiant energy falling on the

8

II. Brightness

1. Image Brightness : Irradiance 2 [ / ]

PE watt m

I

δ

δ=

2. Scene Brightness :

L=2

2 [ / ]

PL watt m

I W

δ

δ δ= where δ2

P is the power emitted by the infinitesimal surface patch

of area (δI) into an infinitesimal solid angle (δw)

2, where is area, and is Distance

ASolidAngle A D

D≡

Ex 1 Hemisphere

wδ

R

2

2

4 1Solid angle = = 2

2

Whole sphere = 4

R

R

ππ

π

⋅

Figure 2—4

Ex 2 Small patch

9

Figure 2—5

Lens

Figure 2—6

cos

( / cos )2

Image Irradiance

PE

I

ISolid angle

f

δ

δδ α

α

=

=

0

2

= '

cos

( / cos )

Scene Radiance

PL

O w

Solid anglez

δ

δ δδ θ

α=

Transfer Function: ( )L

fE

δ

δ=i

10

f(·) : BRDF(Bidirectional Reflectance Distribution Function) in Ch 10

Ex3) Sphere/Radius 2m

Figure 2—7

III. Lenses

Figure 2—8

i. Lens formula

1 1 1

'f z z= + 2–1

' '

2

s z z

R D

−= 2–2

Solve (2-1) & (2-2) to calculate R – blur circle radius

11

1 1 1( )

2

DSR

f s z= − −

R = 0 at focused point P′

ii. Depth of field (DOF)

The range of distances over which objects are focused “sufficiently well” in the sense

that thediameter of the blur circle is less than the resolution of the imaging device.

The larger the lens aperture => the less the DOF

Figure 2—9

iii. Vignetting

In a simple lens, all the rays that enter the front surface of the lens end up being focused in the image

Figure 2—10

12

In a compound lens, some of the rays that pass through the first lens may be occluded

by portions of the second lens, and so on … Vignetting is a reduction in light-gathering

power with increasing inclination of light rays with respect to the optical axis

Figure 2—11

iv. Aberration

Points on the optical axis may be quite well focused, which those in a corner of the

image are smeared out

13

IV. Image sensing

i. Color sensing

a) Human eye

Figure 2—12

b) CCD sense

R, G, B filter

R(λ) : 1 ( at Red color)

14

0 (otherwise)

G(λ) : 1 (at Green color)

0 (otherwise)

B(λ) : 1 (at Blue color)

0 (otherwise)

→ UV filter filtered out UV ray

ii. Randomness and Noise

a) Random variable [R.V]

Consider an experiment H with sample description space Ω the elements on points of

Ω,δ, are the random out comes of H

If to every δ, we assign a real number X(δ) The real establish a correspondence rule

between δ and R the real time, such a rule subject to certain constraints is called a

random variable

15

Figure 2—13

R.V.

Probability density function (p.d.f) p(x)

Where p(x) ≥ 0 for all x, therefore: ( ) 1p x dx

∞

−∞

=∫

mean :

( )xp x dxµ∞

−∞

= ∫ : first moment

variance :

2 2 2( ) ( ) ( ) : second momentx p x dx x p x dxδ µ∞ ∞

−∞ −∞

= − =∫ ∫

Cumulative probability distribution (CDF) P(x)

( ) ( )

x

P x p x dt−∞

= ∫

Two R.V.

SVR

XXX

..

21 +=

P(X1) & P(X2) what P(x) = ?

Solution

16

Figure 2—14

Given x2 , x1 : x-x2

1 1 1 1 2( ) ( )P x x P x x xδ δ⋅ = −

Now, x2 can take on a range of values P2(x2).δ(x2)

To find the prob. that x lies between x and x+δx,

1 2 2 2 2

1 2 2 2 2

1 2

1 2

( ) ( ) ( )

( ) ( ) ( )

( ) ( )

P x x P x x xP x dx

P x P x x P x dx

P x t P t dt

P P

δ δ∞

−∞

∞

−∞

∞

−∞

⋅ = −

∴ = −

= −

= ∗

∫

∫

∫ ∫

For multiple RV's : ∑=

=N

i

ixN

x1

1

Mean value : µ

µ=

N

N

Variance : 2 2

2

N

N N

σ σ=

Standard Deviation : N

σ⇒

b) Gaussian (Normal)

p.d.f :

21

21( )

2

x

p x e

µ

σ

πσ

− −

=

17

c) Poisson (m>0)

p.d.f : ( )!

n

m mp k e

n

−= ⋅

where n arrivals in a time interval T for some m

mean = m variance = σ2

Ex Let X and Y be independent r.v.'s with ( ) ( )x

xp x e u x

−= and

1( ) [ ( 1) ( 1)]

2y

p y u y u y= + − − and Let Z X Y≡ + , what is the p.d.f of Z?

(a) (b)

Figure 2—15

18

( )

1

( ) ( )* ( ) ( ) ( )

(a) 1 ; ( ) 0

1(b) 1 1 ; ( )

2

(c) z 1 ; (

z x y x y

z

z

z y

z

z

p z p x p y p z y p y dy

z p z

z p z e dy

p z

∞

−∞

− −

−

= = −

< − =

− ≤ < =

≥

∫

∫1

( )

1

1)

2

z ye dy− −

−

= ∫

Figure 2—16

19

3. Binary images : Geometric properties

I. Binary Images

Figure 3—1

1. Area of the object 1 sin cos 0

sin cos

x yx yθ θ ρ

ρ ρ

θ θ

+ = ⇒ − + =

−

( , )I

A b x y dxdy= ∫ ∫ ; zero-th Moment

2. Position of the object : center of area

1( , )x xb x y dxdy

A= ∫ ∫ : First moment

1( , )y b x y dxdy

A= ∫ ∫

3. Orientation of the object

(ρ,θ)

2 ( , )I

E r b x y dxdy= ∫ ∫ : second moment

20

II. Simple Geometrical Properties

Figure 3—2

2 2 2

0 0

2 2

2 2 2 2

( ) ( )

( ( sin cos )) ( ( cos sin ))

( ) 2 ( sin cos ) 2 ( cos sin )

r x x y y

x s y s

x y x y s x y s

ρ θ θ ρ θ θ

ρ ρ θ θ θ θ

= − + −

= − − + + − +

= + + + − − + +

To find the shortest distance from (x, y) to (x0, y0) differentiate with respect to S and set

equal to zero

2( )

0d r

ds=

θθ sincos yxS +=

2 2

0

2

(sin cos ) sin ( cos sin )cos

sin sin sin cos

sin (sin cos )

x x x x y

x y

y

θ θ ρ θ θ θ θ

θ ρ θ θ θ

θ θ θ ρ

− = + + − +

= + −

= − +

)cossin(cos0 ρθθθ +−−=− yxyy

2

0

2

0

2 )()( yyxxr −+−=∴

2)cossin( ρθθ +−= yx

21

If, r=0 , ;0cossin =+− ρθθ yx

;0cossin),,( =+− ρθθ yxyxAt

Finally, 2 ( , )E r b x y dxdy= ∫ ∫

2( sin cos ) ( , )I

x y b x y dxdyθ θ ρ= − +∫ ∫

Differentiating w.r.t ρ and setting the result to zero lead to

2( sin cos ) ( , ) 0I

dEx y b x y dxdy

dθ θ ρ

ρ= − + =∫ ∫

0)cossin( =+− ρθθ yxA

Let '' xxxxxx −=⇒−=

yyy −='

θθρθθ

ρθθ

cos'sin'cossin

0)cos)'(sin)'(

yxyx

yyxx

−=+−

=+−−−

and so

2

2 2

( 'sin 'cos ) ( , )

' sin ( , ) ( 2 ' ') cos sin

I

I I

x y b x y dxdy

x b x y dxdy x y

θ θ

θ θ θ

−

= + −

∫ ∫

∫ ∫ ∫ ∫

2 2

2 2

( , ) ' cos ( , )

sin sin cos cos

Ib x y dxdy y b x y dxdy

a b c

θ

θ θ θ θ

+

= − +

∫ ∫

where,

22

2

2

( ') ( , ) ' '

2 ( ' ') ( , ) ' '

( ') ( , ) ' '

I

I

I

a x b x y dx dy

b x y b x y dx dy

c y b x y dx dy

=

= ⋅

=

∫ ∫

∫ ∫

∫ ∫

Now

θθ 2sin2

12cos)(

2

1)(

2

1bcacaE −−−+=

Differentiating w.r.t θ and setting the result to zero, we have

ca

b

−=θ2tan

unless 0 ; b a c= =

+−=

−=

∴

−

θθρ

θ

cossin

tan2

1 1

yx

ca

b

III. Projections

Figure 3—3

23

∫ ∫== dxyxbxVdyyxbxh ),()(,),()(

Area :

( , ) ( )

( )

A b x y dxdy h x dx

v y dy

= =

=

∫ ∫ ∫

∫

Position :

∫

∫ ∫ ∫

=

==

dyyyvA

y

dxxxhA

dxdyyxxbA

x

)(1

)(1

),(1

Orientation : ca

b

−= −1tan

2

1θ

( , )ρ θ θθρ cossin yx +−=

2 2( , ) ( )Ix b x y dxdy x h x dx=∫ ∫ ∫

2 2( , ) ( )I

y b x y dxdy y v x dy=∫ ∫ ∫

∫ ∫∫∫ ∫ −−= dyyhyxvxdttdtdxdyyxxybI )(2

1)(

2

1)(),( 222

Diagonal projection (θ=45°)

Figure 3—4

24

Suppose that θ is 45°

∫ ∫ +−== dsststbdxdyyxbtd )(2

1),(

2

1(),()(

Now consider that

2 2

2 2

1( ) ( , ) ( )

2

1 1( ) ( , )2 2

I I

I

x y b x y dxdy t d t dt

x xy y b x y dxdy

+ =

= + +

∫ ∫ ∫ ∫

∫ ∫

so, 2 2 21 1( , ) ( ) ( ) ( )

2 2Ixyb x y dxdy t d t dt x h x dx y v y dy= − −∫ ∫ ∫ ∫ ∫

IV. Discrete Binary Images

(reference of book figure 3-8)

area ∑∑= =

=N

i

M

j

ijbA1 1

position ),( ji

∑∑

∑∑

=

=

i j

ij

i j

ij

jbA

j

ibA

i

1

1

orientation ( , )ρ θ

∑∑

∑∑

∑∑

i j

ij

i j

ij

i j

ij

ijb

bj

bi

2

2

25

V. Run-Length coding

Figure 3—5

Where rik is the k-th run of the i-th line and the first run in each row is a run of zeros

2

.........., 6

1

2

1

4

1

22

i

ii

n

i

m

k

i

n

i

iki

mrrrrrA

i

+++==∑∑ ∑= = =

position (center of area) : ),( ji

26

∑∑

∑

==

=

=⋅=

=

n

i

j

n

i

i

k

m

k

ii

jvA

jhiA

i

rh

i

11

2

2

1

1..........

1

,

jv

1. Find the first horizontal differences of the image data

Figure 3—6

2. The first difference of the vertical projection can be computed from the

projection of the first

27

horizontal differences

⇒ count the number of circles subtracted by the number of triangles.

3. The vertical projection vj can be found by summing the result from left to right

Orientation ( , )ρ θ

j

ji

jt

ti j

ij

j

j

i

i

vjhidtijb

vj

hi

∑∑∑∑∑

∑

∑

−−= 222

2

2

2

1

2

1

Where, )(2

1jit +=

dt can be obtained in a way similar to that used to obtain the vertical projection

28

4. Binary Image : Topological properties

Jordan curve theorem

A simple closed curve separate the image into two simply connected regions

Figure 4—1

4 Connectedness – only edge-adjacent cells are considered neighbors.

Figure 4—2

29

Ex 1

Figure 4—3

4 objects

` 2 backgrounds

No closed curve ⇒ contradiction(by Jordan curve theorem)

8 connectedness – Corner-adjacent cells are considered neighbors, too.

Figure 4—4

6 connectedness

30

Figure 4—5

Figure 4—6

Figure 4—7

i. A sequential Labeling Algorithm

Recursive Labeling

1. Choose a point where bij=1 and assign a label to this point and into neighbors

31

Figure 4—8

2. Next, label all the neighbors of these neighbors ⇒ one component will have

been labeled completely

3. Find new places to start a labeling operation whenever an unlabeled point is

found where

bij=1

4. Try every cell in this scan

Sequential Labeling

32

Figure 4—9

1. if A is zero, there is nothing to do.

Figure 4—10

2. if A is one, if D has been labeled, simply copy that label and move on if not [if

one B orc is labeled, copy that label else, choose a new label for A]

(go to step 2)

33

II. Local counting and iterative modification

i. Local counting

Figure 4—11

Horizontal N

Vertical N Total Length = 2N

34

Figure 4—12

The overestimated rate : 1:22:2 =NN

Considering all slopes overestimated average ratio : 4

π

Overestimated average rate : π

4

-Euler number-

1. No. of Bodies – No. of Holes

2. No. of upstream convexities – No. of upstream concavities

convexities(+1) concavities(-1)

Figure 4—13

35

Figure 4—14

Body hole

1. body - hole : E = 1- 2 = -1 2 - 0 = 2 1 - 0 = 1 1 - 1 = 0 1 - 0 = 1 1 - 0 = 1

2. convexity - concavity : 1- 2 = -1 2 – 0 = 2 1 – 0 = 1 1 – 1 = 0 1 – 0 = 1 1 - 0 = 1

ii. The additive set property

Figure 4—15

36

⇒This permits to split an image up into smaller pieces and obtain an overall answer by

combining the

results of operations performed on the pieces

Figure 4—16

1 2 2 2 1

)()()()()()( eEdEcEbEaEedcbaE ++++=∪∪∪∪

[ ( ) ( ) ( ) ( )]E a b E b c E c d E d e− + + +∩ ∩ ∩ ∩

1-0 = 1 2-0 =2 3-0 =3 2-0 =2

(1 2 2 2 1) (1 2 3 2) = 0= + + + + − + + +

right (body = 1, hole 1) 1-1 =0

convexities =2

1(body) –

37

concavities =2

Ex1) hand

Figure 4—17

Body : 1 Hole : 0 1-0 =1

Convexities : 5 , Concavities : 4 5-4 =1

Euler number = 1

iii. Iterative modification

Instead of adding up the outputs of Local operators in the previous section, the new

binary image determined by the iterative method can be used as input to another cycle

of computation this process, called iterative modification is useful because it allows us

convexit

concavit

38

to incrementally changes an image that is difficult to process into one that might

succumb to the methods already discussed

Euler Differential E*

2. E* = - ( 1 – 1 = 0 ) + ( 1- 0 ) = 1

Part pattern Current pattern

2

3

39

4

5

**EEnew −=

40

5. Regions & image segmentation

I. 5.1 Thresholding Methods

•Image segmentation : An image segmentation is the partition of an image into a set of

non-overlapping regions whose union is the entire image

Figure 5—1

Detection of discontinuities : points, lines, and edges

Figure 5—2

41

Figure 5—3

∑=

=+⋅⋅⋅⋅⋅++=q

i

iiqq zwzwzwzwR1

2211

A mark used for detecting isolated points

Figure 5—4

i. Point detection

The detection of isolated points in an image is straightforward

If |R| > T when T is threshold, it’s an isolated point

ii. Line detection

The next level of complexity involves the detection of lines in an image. Consider

the four masks, as shown below

42

Figure 5—5

• If at a certain point in the image, |Ri|>|Rj| for all j≠i that point is said to be

more likely associated with a line in the direction of mask i, for example, if

|R1|>|Rj| , for j=2,3,4 the line is more likely associated with a horizontal line.

iii. Edge detection

The idea underlying most edge detection technologies is the computation of a local

derivative operator

Figure 5—6

43

Gradient operator

∂

∂

∂

∂

=∇

y

f

x

f

f

22

∂

∂+

∂

∂=∇

y

f

x

ff

Figure 5—7

Laplacian Operator

2

2

2

22

y

f

x

ff

∂

∂+

∂

∂=∇

)(4 86425

2zzzzzf +++−=∇ in discrete form

Figure 5—8

44

2

2

5 2 4 6 8

Cross Section of f

f = 4 (z )z z z z

∇

∇ − + + +

II. Thresholding

• Thresholding is one of the most important approaches to image segmentation

Figure 5—9

1 if ( , )( , )

0 otherwise

f x y Tg x y

<=

III. Region-oriented segmentation

i. Basic Formulation

Figure 5—10

45

Let R represent the entire image region and Ri the partitioned region for n=1,2,………n

(a) RRin

i

==

∪1

(entire image region)

(b) i

R is a connected region, i=1,2,……n

(c) i j

R R∩ = φ for all i & j , i ≠j

≠

=

=

φ

φ

φ

11

21

51

RR

RR

RR

∩

∩

∩

(d) )( iRP = True for i=1,2,…….n

Figure 5—11

ii. Region Growing by Pixel Aggregation

“Region Growing” is a procedure that groups pixels or sub-regions into larger regions.

The simplest of these approaches is pixel aggregation, which starts with a set of “seed”

points and from these grows regions by appending to each seed

Point those neighboring pixels that have similar properties

46

Ex 1

Figure 5—12

iii. Region splitting and Merging

Spitting

Let R represent the entire image region and select a predicate P as discussed in the

previous section For a square image, one approach for segmenting R is to subdivide it

successively into smaller and smaller quadrant regions so that, for any region Ri,

P(Ri) = TRUE that is, if P(Ri or R) = FALSE, divide the image into quadrants

Figure 5—13

Merging

If only splitting were used, the final partition likely would contain adjacent regions

With identical properties, this draw back may be remedied by allowing merging as well

as splitting two adjacent regions Rj &Rk are merged only if P(Rj∪Rk)=TRUE

Figure 5—14

47

• The preceding discussion may be summarized by the following procedure in

which, at any step, we split into four disjointed quadrants any region Ri where

P(Ri)= FALSE

Figure 5—15

a) merge any adjacent regions Rj & Rk for which P(Rj∪Rk) =TRUE ;

b) stop when no further merging and splitting is possible

Ex 2

Figure 5—16

MergingFALSERP

TRUERRRPFALSERP

FALSERPFALSERP

FALSERPTRUERP

TRUERPFALSERP

→+

++

=+

==

==

)(

)()(

)()(

)()(

)()(

4

4132233

432

341

21

∪∪

Figure 5—17

48

6. Image Processing : Continuous images

I. Linear, Shift-Invariant Systems

Spatial domain → Point Spread Function (PSF)

Frequency domain → Modulation Transfer Function (MTF)

Figure 6—1

• Defocused image ( g ) is a processed version of the focused image ( f )

Double the brightness of ideal image and the defocused image. If the image system

moves, f & g moves the same amount.

Figure 6—2

Linearity : 1 2 1 2( , ) ( , ) ( , ) ( , )f x y f x y g x y g x yα β α β+ → +

Shift : ( , ) ( , )f x a y b g x a y b− − → − −

49

II. Convolution

( , ) ( , ) ( , )f x y h x y g x y→ →

( , ) ( , ) ( , )g x y f x y h d d

g f h

ε η ε η ε η∞ ∞

−∞ −∞

= − −

→ = ⊗

∫ ∫

Linear ( ) ( )1( , ) 2 ( , ) ?f x y f h g x yα β+ → → =

[ ]1 2( , ) ( , ) ( , ) ( , )g x y f x y f x y h d dα ε η β ε η ε η ε η∞ ∞

−∞ −∞

= − − + − − ⋅∫ ∫

1 2( , ) ( , )g x y g x yα β= +

Shift invariant

( , ) ( , ) ( , )f x a y b h x y g x y− − → →

( , ) ( , ) ( , )g x y f x a y b h d dε η ε η∞ ∞

−∞ −∞

= − −∫ ∫

Impulse response

( , ) ( , ) ( , )f x y h x y h x y→ →

( , ) ( , ) ( , )g x y f x y h d dε η ε η ε η∞ ∞

−∞ −∞

= − −∫ ∫

and ( , ) ( , )g x y h x y=

Only possible if Origin

( , )0 otherwise

f x y∞

=

This function is called unit impulse ( , )x yδ or Dirac delta function

( , ) 1x y dxdyδ∞ ∞

−∞ −∞

=∫ ∫

50

Shifting Property : ( , ) ( , ) (0,0)x y h x y dxdy hδ∞ ∞

−∞ −∞

=∫ ∫

and ( , ) ( , ) ( , )x y h d d h x yδ ε η ε η ε η∞ ∞

−∞ −∞

− − ⋅ =∫ ∫

=> ( , )h x y : Impulse Response of the system

Figure 6—3

( , ) ( , )k x yε η δ ε η− − : scaled delta function

( , ) ( , ) ( , )f x y f x y d dε η δ ε η ε η∞ ∞

−∞ −∞

= ⋅ − −∫ ∫

( , ) ( , ) ( , )g x y f x y h x y d dε η ε η∞ ∞

−∞ −∞

= − − ⋅∫ ∫

b a⊗ = ( , ) ( , )a x y b x y d dε η ε η∞ ∞

−∞ −∞

− − ⋅∫ ∫ <= Let ,x ε α− = y η β− =

= ( , ) ( , )a b x y d d b aα β α β α β∞ ∞

−∞ −∞

⋅ − − = ⊗∫ ∫

Similarly, ( ) ( )a b c a b c⊗ ⊗ = ⊗ ⊗

51

Cascade

1 1 2( )1 2

f h f h hf h h

⊗ ⊗ ⊗→ → → => 1 2( )f h h⊗ ⊗

1 2f h h g→ ⊗ →

III. MTF

An Eigen function of a system is a function that is reproduced with at most a change in

amplitude.

( )jwt jwte A w e→ →

2 ( )( , ) i ux vyf x y e π +→ =

2 ( ) ( )( , ) ( , )

i u x v yg x y e h d d

π ε η ε η ε η∞ ∞

− + −

−∞ −∞

= ⋅∫ ∫

= 2 ( ) 2 ( ) ( , )i ux vy i u ve e h d dπ π ε η ε η ε η∞ ∞

+ − +

−∞ −∞

⋅∫ ∫

= 2 ( ) ( , )i ux vye A u vπ + ⋅

IV. Fourier Transform

Represent the signal as an infinite weighted sum of an infinite number of sinusoids (u: angular frequency)

( ) ( ) iuxF u f x e dx∞

−

−∞= ∫

Note:

cos sin 1ike k i k i= + = −

52

Arbitrary function => Single Analytic Expression

Spatial Domain(x)=> Frequency Domain(u) (Frequency Spectrum F(u))

Inverse Fourier Transform (IFT)

( ) ( )1

2

iuxf x F u e du

π

∞

−∞= ∫

( )

2

1( , ) ( , )

4

i ux vyf x y F u v e dudvπ

∞ ∞+ +

−∞ −∞

= ∫ ∫

V. The Fourier Transform of Convolution

Let g f h= ⊗

Now,

( ) ( )

( ) ( )

( ) ( ) ( )

( ) ( )

( ) ( )

2

2

22

2 2 '

' '

i ux

i ux

i u xi u

i u i ux

G u g x e dx

f h x e d dx

f e d h x e dx

f e d h x e dx

F u H u

π

π

π τπ τ

π τ π

τ τ τ

τ τ τ

τ τ

∞ ∞−

−∞ −∞

∞ ∞−

−∞ −∞

∞ ∞− −−

−∞ −∞

∞ ∞− −

−∞ −∞

=

= −

= −

=

=

∫ ∫

∫ ∫

∫ ∫

∫ ∫

Convolution in spatial domain Multiplication in frequency domain

Spatial Domain (x) Frequency Domain(u)

g f h G FH

g fh G F H

= ⊗ ↔ =

= ↔ = ⊗

53

So, we can find g(x) by Fourier transform

Figure 6—4

Spatial Domain (x) Frequency Domain (u)

Linearity ( ) ( )1 2c f x c g x+ ( ) ( )1 2c F u c G u+

Scaling ( )f ax 1 uF

a a

Shifting ( )0f x x− ( )02i uxe F u

π−

Symmetry ( )F x ( )f u−

Conjugation ( )f x∗ ( )F u

∗ −

Convolution ( ) ( )f x g x⊗ ( ) ( )F u G u

Differentiation ( )n

n

d f x

dx

( ) ( )2n

i u F uπ

54

VI. Generalized Functions and Unit Impulses

• The integral of the one-dimensional unit impulse is the unit step function,

( ) ( )x

t dt u xδ−∞

=∫ , where ( )u x = 1, for 0;x ≻

1

2= , for 0;x =

0= , for 0x ≺ .

What is the Fourier transform of the unit impulse?

We have ( )( , ) 1i ux vyx y e dxdyδ∞ ∞

− +

−∞ −∞

=∫ ∫ , as can be seen by substituting 0x = and 0y =

into ( )i ux vye− + , using the sifting property of the unit impulse. Alternatively, we can use

( )

0lim ( , ) i ux vyx y e dxdyεε

δ∞ ∞

− +

→−∞ −∞

∫ ∫ , or 0

1 1lim

2 2

iux ivye dx e dy

ε ε

εε εε ε

− −

→− −

∫ ∫ ,

That is 0

sin sinlim 1

u v

u vε

ε ε

ε ε→= .

VII. Convergence Factors and the Unit Impulse

We want a smoothed function of f(x)

( ) ( ) ( )g x f x h x= ∗

Let us use a Gaussian kernel

Figure 6—5

55

( )2

2

1 1exp

22

xh x

σπσ

= −

Then

( ) ( )2 21

exp 22

H u uπ σ

= −

( ) ( ) ( )G u F u H u=

H(u) attenuates high frequencies in F(u) (Low-pass Filter)!

VIII. Partial Derivatives

What is the F.T. of ( , )x yf

x

∂

∂and

f

x

∂

∂ ?

2 ( )i ux vyfe dxdy

x

π∞ ∞

− +

−∞ −∞

∂

∂∫ ∫ = 2 2ux vyfe dx e dy

x

π π∞ ∞

− −

−∞ −∞

∂⋅

∂ ∫ ∫

2 uxfe dx

x

π∞

−

−∞

∂

∂∫= 2 2( , ) ( 2 ) ( , )iux iux

f x y e iu f x y e dxπ ππ

∞∞− −

−∞−∞

− − ⋅∫

= 2 2( , ) (2 ) ( , )iux iuxf x y e iu f x y e dx

π ππ∞

∞− −

−∞−∞

+ ⋅∫

We can’t proceed unless, lim ( , ) 0x

f x y→±∞

=

In that case,

2 ( )2 ( , ) i ux vyiu f x y e dxdy

ππ∞ ∞

− +

−∞ −∞

∫ ∫ =(2 ) ( , )iu F u vπ ⋅

( , )

( , )

2 ( , )

2 ( , )

x y

x y

fF iu F u v

x

fF iv F u v

x

π

π

∂ ∴ = ⋅

∂

∂ = ⋅

∂

56

IX. Rotational Symmetry and Isotropic operators

• Rotationally symmetric operators are particularly attractive because the treat

image features in the same way, no matter what their orientation is.

• Circular symmetry

Figure 6—6

Let us introduce polar coordinates in both spatial and frequency domain

Figure 6—7

cosx r φ= and siny r φ= cos( )ux vy r ρ φ α+ = ⋅ −

cosu ρ α= ⋅ and sinv ρ α=

57

2 ( )( , ) ( , ) i ux vyF f x y f x y e dxdy

π∞ ∞

− +

−∞ −∞

= ∫ ∫

2 cos( )

0 0

( ) i rrf r e rdrd

π φ α φ∞ ∞

− ⋅ −= ⋅∫ ∫

Zeroth order Vessel function

(Hankel Transform) => 0( ) 2 ( ) (2 )rF f r J r rdrρ ρ π π ρ∞

−∞

= ∫

0

0

( ) 2 ( ) (2 )fr r f J r dρπ ρ π ρ ρ ρ∞

= ∫

where 2

cos( )0

0

1( )

2

i x wJ x e d

πθ θ

π− ⋅ −= ∫

X. Blurring

2 2

2

1

2

2

1( , )

2

x y

h x y e σ

πσ

+−

=

. .F T⇓

2 2

2

1

2 ( )2

2

1( , )

2

x y

i ux vyH u v e e dxdy

πσ

πσ

+∞ ∞−

− +

−∞ −∞

= ⋅∫ ∫

2 21 1

2 22 21 1

2 2

x y

iux ivye e dx e e dy

π πσ σ

πσ πσ

∞ ∞− − − −

−∞ −∞

= ⋅ ⋅∫ ∫

21

221

2

x

iuxe e dx

πσ

πσ

∞− −

−∞

∫

2 21 1

2 2cos(2 ) sin(2 )

x x

e ux dx i e ux dxσ σπ π ∞ ∞

− −

−∞ −∞

= − ⋅ ⋅∫ ∫

58

The second integral is odd so, integral over symmetric region is zero.

And,

2 2 21

2

2

1cos(2 ) ,

2

x u

ae ux dx e aa

πσ π

πσ

∞− −

−∞

= ⋅ =∫

2

2 2 21 1

(2 )2 2cos(2 ) 2

xu

e ux dx eπ σ

σ π π σ ∞

− − ⋅ ⋅

−∞

= ⋅ ⋅∫

2 2 2 2 2 21 1(2 ) (2 )

2 21 1

( , ) 2 22 2

u v

H u v e eπ σ π σ

πσ πσπ σ π σ

− ⋅ ⋅ − ⋅ ⋅

= ⋅ ⋅ ⋅ ⋅ ⋅⋅ ⋅

2 2 2 21

(2 ) ( )2

u v

eπ σ− ⋅ + ⋅

=

Which is also rotationally symmetric. F G a u s s ia n G a u s s ia n=

Figure 6—8: Low Pass Filtering

XI. Restoration and Enhancement

( , )f x y → ( , ) '( , ) ( , )h x y h x y f x y→ →

The cascade of the two systems is the identity system.

1'( , ) min ,

( , )H u v A

H u v

=

,

Where, A is the maximum gain. Or more elegantly, we can use something like

59

2 2

( , )'( , )

( , )

H u vH u v

H u v B=

+,

where 1

(2 )Bis the maximum gain, if ( , )H u v is real.

XII. Correlation

Figure 6—9

Cross correlation of ( , )a x y and ( , )b x y is defined as

( , ) ( , )* ( , )

( , ) ( , )

ab x y a x y b x y

a x y b d d

φ

ε η ε η ε η∞ ∞

−∞ −∞

=

= − − ⋅∫ ∫

If ( , )a x y = ( , )b x y , the result is called autocorrelation

( , ) .( ( , ) ( , ))ab ab abx y Symmetric x y x yφ φ φ→ − − =

(0,0) ( , )aa aa x yφ φ≥ for all ( , )x y

2(0,0) ( , ) ( , ) ( , )aa a a d d a d dφ ε η ε η ε η ε η ε η∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞

= ⋅ =∫ ∫ ∫ ∫

60

XIII. 6.13 Optimal Filtering

Figure 6—10

E = 2

( , ) ( , )o x y d x y dxdy

∞ ∞

−∞ −∞

−∫ ∫

( , ) ( , ) ( , )i x y b x y n x y= +

( , ) ( , ) ( , )o x y i x y h x y= ⊗

So, 2 2( , ) 2 ( , ) ( , ) ( , )E o x y o x y d x y d x y dxdy

∞ ∞

−∞ −∞

= − ⋅ +∫ ∫

22 ( , ) ( , ) ( , )o x y i x y h x y= ⊗

( , ) ( , ) ( , ) ( , )i x y h d d i x y h d dε η ε η ε η α β α β α β∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞

= − − ⋅ × − − ⋅∫ ∫ ∫ ∫

Therefore,

2 ( , ) ( , ) ( , ) ( , )o dxdy h h d d d d i x y i x y dxdyε η α β ε η α β ε η α β∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞ −∞ −∞ −∞ −∞

= ⋅ − − ⋅ − −∫ ∫ ∫ ∫ ∫ ∫ ∫ ∫

x A x A

y B y B

α α

β β

− = ⇒ = +

− = ⇒ = +

61

⇒ ( ( ), ( ) ( , )i A B i A B dAdBε α η β∞ ∞

−∞ −∞

= − − − − ⋅∫ ∫

( , )iiφ ε α η β= − − (Autocorrelation)

62

7. Image Processing – Discrete Image

I. Sampling Theorem

To convert a continuous distribution of brightness to a digital image, it necessary to

extract the brightness at each point arranged at a period. Thus is called sampling.

Assume an image as a one-dimensional function for simplicity.

Let the brightness at position x i.e. pixel value of pixel at x, be fix.

Figure 7—1

Figure 7—2

63

Suppose a function that is composed by infinite numbers of Dirac’s delta functions

assigned at an interval T. This is called comb function defined as follows.

A simple digital image from ( )f x , denoted ( )Tf x , is expressed as ( )f x multiplied by

the comb function ( )Tcomb x , i.e.

( ) ( ) ( )T Tf x f x comb x= ⋅

( ) ( ) ( ) ( ) ( ) ( )T TFT f x v FT f x v FT comb x v= ⊗

and, 11( ) ( ) ( )T

TFT comb x v comb V

T=

1

( ) ( ) ( ) ( )TFT f x v FT f x vT

= ⊗ 1( )

Tcomb V

That’s a convolution of comb function?

We start from “convolution with delta function”

( ) ( ) ( ) ( ) ( ) ( ) ( )f t t f y t y dy f t t t dy f tδ δ δ∞ ∞

−∞ −∞∗ = ⋅ − = ⋅ − =∫ ∫

i.e., convolution of a function and delta function is equal to original function itself.

Since a comb function is a sequence of the delta function arranged at constant period,

convolution of a function and comb function is an arrangement of duplication of the

original function at a constant period.

64

The F.T. of ( )Tf x , which is the sampled version of ( )f x at period T is an infinite

sequence of duplicating of ( )TFT f x which is the F.T. of the original function ( )f x

at period 1

T.

Figure 7—3

If the period of comb function in the frequency domain is sufficiently large, adjacent

( )FT f x ’s do not overlap. In this case, the F.T. of the original function ( )FT f x ,

can be separated and extracted i.e. no information of the brightness distribution of the

original image is lost by sampling. However, if the internal of the comb function is the

frequency domain is small, adjacent ( )FT f x (v)’s overlap. In this case, the original

( )FT f x can’t be separated and a faulty function will be extracted. This effect is

called aliasing.

65

( ) ( ) ( ) ( ) ( ) ( )T TFT f x v FT f x v FT comb x v⇒ = ⊗

Figure 7—4

Since the support of ( )FT f x is in the range between ~c cV V− , the period has to be at

least 2 cV to avoid overlapping of ( )FT f x ’s since, T is the sampling period. 1

T

denote the number of sampling per unit length, i.e. sampling rate consequently, the

original brightness distribution can be reconstructed by a sampled digital image of the

sampling rate is more than twice the maximum frequency contained in the original

distribution. This theorem is called sampling theorem.

66

Discrete Fourier Transform (DFT)

1 1 2 ( )2 ( )

0 00 0

1( , ) ( , ) ( , )

m nM N i u vi ux vy M N

m n

F u v f x y e dxdy f m n eMN

ππ

∞ ∞ − − − +− +

= =

= = ⋅∑∑∫∫

(u=0~M-1, v=0~N-1)

Inv. D.F.T.1 1 2 ( )

0 0

( , )m nM N i u vM N

m n

F u v eπ− − + +

= =

= ⋅∑∑ (m=0~M-1, n=0~N-1)

• f(m,n) ≜ Input Signal Amplitude (real or complex) at sample (m, n)

T ≜ Sampling interval

sf ≜ Sampling rate 1

( )T

= , samples/second

M,N = Numbers of Spatial samples = Number of frequency samples (integer)

• Magnitude 1

2 2 2( , ) ( , ) ( , )F u v R u v I u v = +

Phase Angle 1 ( , )( , ) tan

( , )

I u vu v

R u vφ −=

Power Spectrum 2 2 2( , ) ( , ) ( , ) ( , )P u v F u v R u v I u v= = +

• 1 1

0 0

1(0,0) ( , )

M N

m n

F f m nMN

− −

= =

= ∑∑ ⇒ Average of ( , )f m n or D.C. component.

If ( , )f x y is real, its F.T. is conjugate complex

( , ) ( , )

( , ) ( , )

F u v F u v

F u v F u v

∗

∗

= − −

= − − ⇒ Spectrum of D.F.T. is symmetric.

67

• Relationship between samples in the spatial and frequency domains.

1u

M x=

⋅ ∆

1v

N y=

⋅∆

• Translation

0 0

0 0

2 ( )

( , ) ( , )u v

i x yM Nf x y e F u u v v

π− ⋅ ⋅ + ⋅

⋅ ⇔ − −

0 0

0 0

2 ( )

( , ) ( , )ux vy

iM Nf x x y y F u v e

π− ⋅ +

− − ⇔

Case : when 0 0,2 2

M Nu v= =

2 ( )

( )2 2 cos ( ) ( 1)x y

ii x y x y

e e x yπ π π

⋅ ++ += = + = −

( , )( 1) ( , )2 2

x y M Nf x y F u v

+⇒ − ⇔ − −

And similarly,

( , ) ( , )( 1)2 2

u vM Nf x y F u v

+− − ⇒ −

Rotation cos

sin

x r

y r

θ

θ

= ⋅

= ⋅

cos

sin

u w

v w

φ

φ

= ⋅

= ⋅

Then, ( , )f x y and ( , )F u v become ( , )f r θ and ( , )F w φ

68

0 0( , ) ( , )f r F wθ θ φ θ+ ⇔ +

⇒ Rotating ( , )f x y by an angle 0θ rotates ( , )F u v by the same angle, and vice versa.

Periodicity ( , ) ( , ) ( , ) ( , )F u v F u M v F u v N F u M v N= + = + = + +

The inverse transform is also periodic.

( , ) ( , ) ( , ) ( , )f x y f x M y f x y N f x M y N= + = + = + +

Separability 1 1 22

0 0

1 1( , )

yxM M j vj uNM

x y

F u v e eM N

ππ− − −−

= =

= ⋅∑ ∑

1 2

0

1( , )

xM j uM

x

F u v eM

π− −

=

= ∑ , (where 1 2

0

1( , )

yM j vN

y

F u v eN

π− −

=

= ∑ )

( , ) ( , ) ( , )f x y F x v F u v→ →

• Convolution

1 1

0 0

1( , ) ( , ) ( , ) ( , )

M N

m n

f x y h x y f m n h x m y nMN

− −

= =

⊗ = ⋅ − −∑∑

f h F H

f h F H

⊗ ⇒ ⋅

⋅ ⇒ ⊗ Pr ( . .)ove H W→

1-D

Row transform

1-D

column transform

69

8. Edge & Edge detection

I. Edges in Images

• An Edge is a boundary between two figures with relatively distinct grey-level

properties.

• Edges are curves in the image where rapid changes occur in brightness or in the

spatial derivatives of brightness.

• Change of brightness can be occurred at

• Change of surface orientation

• Different surfaces

• One object occludes another

• A boundary between light and shadow falling on a single surface

II. Differential Operators

Figure 8—1

70

Figure 8—2

sin cos 0x yθ θ ρ− + =

( ) ( )1 2 1E , B B -B u( sin cos )x y x yθ θ ρ= + − +

If t > 0 B1

t < 0 B2

t = 0 ½(B1 + B2)

( ) ( )

( ) 1

t

u t x dx

x dx

δ

δ

−∞

∞

−∞

=

=

∫

∫

Figure 8—3

)cossin()(cos

)cossin()(sin

21

21

ρθθδθ

ρθθδθ

+−−−=∂

∂

+−−=∂

∂

yxBBy

E

yxBBx

E

71

squared gradient

2

21

22

))cossin()(( ρθθδ +−−=

∂

∂+

∂

∂yxBB

y

E

x

E

Laplacian of E(x,y)

2 22

2 2

22

1 22

22

1 22

2 '

1 2

sin ( ) '( sin cos )

cos ( ) '( sin cos )

( ) ( sin cos )

xx

yy

E EE

x y

EE B B x y

x

EE B B a y

y

E B B x y

θ δ θ θ ρ

θ δ θ θ ρ

δ θ θ ρ

∂ ∂∇ = +

∂ ∂

∂= = − − +

∂

∂= = − − +

∂

∇ = − − +

III. Discrete Approximations

• Discrete form of Laplacian

Ex = E( i+1 , j ) – E( i , j )

Ey = E( i , j+1 ) – E( i , j )

Exx = E( i – 1 , j ) – 2E( i , j ) + E( i + 1 , j )

Exx + Eyy = (Ei-1 j + Eij-1 + Ei-1 j + E i j+1 ) – 4Eij

72

IV. 8.4 Local operators and noise

bb

nn

H

φφ

ρ+

⋅−=1

12' -------------

ρ2 = u

2 + v

2

suppose that 2

2

2

& NS

nnbb == φρ

φ

Then 222

22'

NS

SH

ρ

ρ

+−= -

at low frequency -ρ2 : Laplacian operation

at high frequency -2

2

N

S << ∞ small gain Noise can be reduced !

Derivation of

Figure 8—4

''

' '' 2

1 1 ( )

1

( )

i bb nb

ii bb bn nb nn

bb

bb nn

nn

bb

H

SNR

H H

αϕ ϕ ϕ

ϕ ϕ ϕ ϕ ϕ

ϕ

ϕ ϕ

ϕϕ

+= =

+ + +

=+

= =+

= ⋅ ℑ ∇

73

For example, if the optimal filter is a gaussian

Figure 8—5

2

22

2

22

1),( σ

πσ

yx

eyxh

+−

=

)(2

1 222

)0,(vu

euH+−

=σ

74

9. Lightness & color

Figure 9—1

b´(x,y) = e´(x,y) - r´(x,y)

• Note that r´(x,y) is constant within a patch and sharp discontinuities at edges

between patches and e´(x,y) varies smoothly

Figure 9—2

Task : separate r´(x,y) & e´(x,y)

• Take the logarithm of image brightness b´(x,y)

log b´(x,y) = log r´(x,y) +log e´(x,y)

b(x,y) = r(x,y) +e(x,y)

∇2b(x,y) =∇2

r(x,y) +∇2e(x,y)

Figure 9—3

75

• cut slow slope by a threshold

Figure 9—4

T(∇2b(x,y) = ∇2

r(x,y)

|∇2b(x,y)| > T then t(x,y) = ∇2

b(x,y) else t(x,y) = 0

• recover r(x,y)

Figure 9—5

∫ ∫∇=

=∇

dxdyyxryxl

yxtyxl

),(),((

),(),(

2

2

Fourier Transform :

)(1

)(

)()(

)],([)],([

2

2

2

eTL

TL

yxtyxl

ρρ

ρρρ

−=⇒

=−⇒

ℑ=∇ℑ

2

1)(..

)()(

ρ

ρρ

−=

⋅=

pGwhere

TG

ρπ

+= )log(2

1)( rrg

IFT : I(x,y) = g(r) * t(x,y)

76

Figure 9—6

I(x,y) = r(x,y) = reflectance

e(x,y) = b(x,y) – I(x,y)

Figure 9—7

77

...

),(),(),(

),(),(),(

555

444

333

222

111

erb

erb

erb

yxeyxryxb

yxeyxryxb

⋅=

⋅=

⋅=

=

=

)(74

37: 21

22

11

2

11 ee

er

er

b

bR ≅== - equation 9.1

52

26

2

1 =r

r - equation 9.2

3

2

3

2

5

2

5

22

95

38

112

28:

r

r

b

b

r

r

b

bR

==

== 　

Least square error 11 variables 16 equations

0 ≤ reflectance ≤ 1

16

equations

78

10. Reflectance Map : photometric stereo

(using one camera)

I. Radiometry

irradiance A

pE

δ

δ= geometric stereo (using one camera)

radiance WA

pL

δδ

δ 2

=

solid angle = Area of a spherical surface patch / (radius of the sphere)2

Figure 10—1

2

2

14

2Solid angle 2

R

R

ππ

⋅= =

Figure 10—2

79

Figure 10—3

221

)1,,(ˆ

qp

qpn

++

−−=

II. Image Formation

Figure 10—4

80

solid angle = 2)cos/(

cos

α

θδ

z

O

solid angle = 2)cos/(

cos

α

α

S

SI

2

cos

cos

=

S

z

I

O

θ

α

δ

δ --------- equation 10.2.1

The solid angle subtended by the lens

ααπ

α

αα

π3

2

2

2

cos4)cos/(

cos2

=

=Ωzz

Thus the power of the light originating on the patch and passing through the lens is

θδδ cosΩ⋅= OLP

where L is the radiance of the surface in the direction toward the Lens

θααπ

δ

δ

δ

δ

θααπ

δδ

coscos4

coscos4

3

2

3

2

==

=

zI

OL

I

PE

zOLp

where E is the irradiance of the image at the patch under consideration

substituting for IO δδ , we finally obtain

81

ααπ

ααπ

4

2

4

2

cos4

cos4

⋅=

∝∴

=

sLE

LE

sLE

III. Bidirectional reflectance distribution function (BRDF)

Figure 10—5

Figure 10—6

82

),( iiE φθδ ; irradiance

The amount of light falling on the surface from the direction ),( ii φθ

),( eeL φθδ : radiance

The brightness of the surface as seen from the direction

),( ee φθ

BRDF :

1

cossin),;,(

),(

),(),;,(

2

0

=

=

∫ ∫−

π

π

π

φθθθφθφθ

φθδ

φθδφθφθ

eeeeeeii

ii

ee

eeii

dd

E

Lf

IV. Extended Light sources

Figure 10—7

solid angle wδ 2)tan(

.

cedis

areaThe = The area = iiδφθδθsin

83

(proof : 2

0

sin 2i i i

sw

ππ

π

θ δθ δϕ π−

= =∫ ∫ ∫ ∫

),( iiE φθ : Radiance per unit solid angle coming from the direction ),( ii φθ

The radiance from the patch under consideration

iiiiiii EE δφδθθφθδωφθ sin),(),( =

Total irradiance of the surface is

2

0

0

( , ) sin cosi i i i i i

E E d d

ππ

π

θ ϕ θ θ θ ϕ−

= ∫ ∫

where iθcos accounts for the foreshortening of the surface as seen from the direction

),( ii φθ

The radiance of the surface,

∫ ∫= ),(),,(),( iieeiiee EjfL φθφθφθφθ

iiii dd φθθθ cossin

V. Surface Reflectance properties

• Ideal Lambertian surface

One that appears equally bright from all viewing directions and reflects all

incident light, absorbing none

84

BRDF ),,( eeii jf φθφθ constant

2

0

2

0

( , , , ) sin cos 1

2 sin cos 1

1

1

i i e e e e e e

e e e

f j d d

f d

f

f

ππ

π

π

θ φ θ ϕ θ θ θ ϕ

π θ θ θ

π

π

−

=

=

=

∴ =

∫ ∫

∫

sine BRDF is constant for a Lambertian surface

)1

(1

cossin),(

ππ

φθθθφθ

=⋅=⋅=

⋅= ∫ ∫

fEEf

ddEfL

oo

iiiiii

∵

where the irradiance is Eo

• Ideal specular Reflector

Figure 10—8

85

2

0

( , ; , ) sin cos 1

sin cos 1

1

sin cos

i i e e e e e e

i i

i i

f d d

k

k

ππ

π

θ ϕ θ ϕ θ θ θ ϕ

θ θ

θ θ

−

=

⇒ =

⇒ =

∫ ∫

In this case, ii

ieieeeii jf

θθ

πφθδθθδφθφθ

cossin

)()(),,( −−−=

Determine the radiance of a specular reflecting surface under an extended source where

the irradiance is Eo

2

0

( ) ( )( , )

sin cos

( , ) sin cos

( , )

( , )

e i e ie e

i i

i i i i i i

e e

i i

L

E d d

E

E

ππ

π

δ θ θ δ ϕ ϕ πθ ϕ

θ θ

θ ϕ θ θ θ ϕ

θ ϕ π

θ ϕ

−

− − −=

⋅ ⋅

= −

=

∫ ∫

VI. Surface Brightness

Figure 10—9

86

iEL θπ

cos1

=

•Light source : a “sky” of uniform radiance E

2

0

2

0

2

0

1sin cos

sin cos

cos 2[1 1]

2 2

i i i i

i i i i

i

L E d d

Ed d

EE E

ππ

π

ππ

π

π

θ θ θ ϕπ

θ θ θ ϕπ

θ

−

−

= ⋅

=

− = = + =

∫ ∫

∫ ∫

The radiance of the patch ≡The radiance of the source

VII. Surface orientation

Figure 10—10

87

T

yyy

T

xxx

yx

qr

pr

qprrn

),,0(

),0,(

)1,,(

δδ

δδ

=

=

−−=×=

The unit surface normal n

221

)1,,(ˆ

qp

qp

n

nn

T

++

−−==

Figure 10—11

Tv )1,0,0(ˆ ≡

assuming α is very small

2222

22

1

1)1,0,0(

1

)1,,(ˆˆcos

1

)1,,(ˆ

qpqp

qpvn

qp

qpn

T

++=⋅

++

−−==

++

−−=

θ

88

VIII. The reflectance Map

Figure 10—12

),(ˆ

),(ˆ

22

11

qpRs

qpRs

→

→

iEL θπ

cos1

= for 0≥iθ

22222222 11

1

1

)1,,(

1

)1,,(ˆˆcos

sx

ss

ss

T

ss

i

qpqp

qqpp

qp

qp

qp

qpsn

++++

++=

++

−−⋅

++

−−=⋅=θ

= R(p,q) ; Reflectance Map

cqpqp

qqppqpR

ss

ss =++++

++=

2222 11

1),( - equation 10.9.1

Figure 10—13

89

222

1

2

1

222

222

1

2

1

11

1

11

1),(

11

1),(

ss

ss

ss

ss

qpqp

qqppqpR

qpqp

qqppqpR

++++

++=

++++

++=

IX. Shading in image

Consider a smoothly curved object the image of such an object will have spatial

variations in brightness due to the fact that surface patches with different orientations

appear with different brightness the variation of brightness is called shading

),(cos

cos4

4

2

qpRLE

f

dLE

i ≡

⋅=

θαα

απ

after normalizing E(x,y) = R(p,q)

ρ : reflectance factor where 10 ≤≤ ρ

nsyxEwhere

nspyxE

ˆˆ),(,1

)ˆˆ(),(

⋅==

⋅⋅=

ρ

Ex1) Consider a sphere with a Lambertian surface illuminated by a point source at

essentially the same place as the viewer in the case of cylinder

90

Figure 10—14

In this case,

221

)1,,(ˆ

)1,0,0(ˆ

)1,0,0(ˆ

qp

qpn

v

s

++

−−=

=

=

From equation 10.9.1 - snqpR ˆˆ),( ⋅=

22

2222

1

1

11

1

qp

qPqp

qqpp

ss

ss

++=

++++

++=

0

2

0

0

2

1

222

)(

)2())((2

1

zz

zzy

d

dq

zz

xxyxr

d

dp

y

z

x

z

−

−−==

−−=−+−==

−

Finally, E(x,y) = R(p,q)

91

2

22

2

0

22

22

1

)(1

1

1

1

r

yx

zz

yx

qp

+−=

−

++

=

++=

( x , y ) = ( 0 , 0 )

E( x, y) maximum = 0

x2+y

2 =r

2

E(x,y) minimum = 0

X. Photometric stereo

Figure 10—15

92

2

2

2

2

22

2222

2

1

2

1

22

1111

11

1),(

11

1),(

ss

ss

ss

ss

qPqp

qqppqpRE

qPqp

qqppqpRE

++++

++==

++++

++==

Ex2) R(p, q) : linear and independent

2

222

1

111

1),(

1),(

r

qqppqpR

r

qqppqpR

++=

++=

then 2121

12

2

221

2

1 )1()1(

pqqp

qrEqrEp

−

−−−=

2121

21

2

111

2

2 )1()1(

pqqp

prEprEq

−

−−−=

Ex3) R(p,q) : nonlinear

Figure 10—16

93

XI. Albedo : reflectance factor :

)ˆˆ( nsE ii ⋅= ρ

where 221

)1,,(ˆ

ii

T

ii

i

qp

qps

++

−−= for i=1, 2, 3, …………

221

)1,,(ˆ

qp

qpn

T

++

−−=

Figure 10—17

Unknown variables p, q, ρ

At least 3 equations

94

11. Reflectance Map : shape from shading

I. Recovering Shape from Shading

Figure 11—1

i. Picture

One more information => strong constraint smoothness neighboring patches of the

surface => similar orientation assumption of smoothness : strong constraint

ii. Linear reflectance map

Figure 11—2

95

220

2200 sincostan

ba

b

ba

a

a

b

+=

+== θθθ

Figure 11—3

:sincos)( θθθ qpm += directional derivative

)),((1

sincos)(

1

2222

000

yxEfbaba

bqap

qm

−

+=

+

+=

+=⇒ θθθ

S can be recovered from E(x,y) using above curve

Figure 11—4

96

δξδ mz = (small change a long the characteristic curve)

)),((1 1

22yxEf

ba

zm −

+==

δξ

δ

θξξ

θξξ

sin)(

cos)(

0

0

+=

+=

yy

xxwhere

0

0

1

02 2

0

( ) ( )

1( ( , ))

z z m d

z f E x y da b

ξ

ξ

ξ θ ξ

ξ−

= +

= ++

∫

∫

iii. Rotationally symmetric reflectance maps

Figure 11—5

97

22

22

sin

cos

tan

qp

q

qp

p

p

q

s

s

s

+=

+=

=

θ

θ

θ

The slope sss qpm θθθ sincos)( +=

22 qp +=

)),((1 yxEf −=

The change in Z

δξδξθδ

δξδξθδ

δξδξδ

22

22

122

sin

cos

)),((

qp

qy

qp

px

yxEfqpmz

s

s

+==

+==

⋅+== −

To simplify δz , we could take a step of length δξ22

1

qp + rather than δξ

δξ

δξ

δ

)),((

)(

1

22

2222

yxEf

qp

qpqpz

−=

+=

+×+=

δx = pδξ - equation 11.1.1

δy = qδξ

98

We need to determine p&q at the new point in order to continue the solution

yxtrwyxEateDifferenti

qpfyxE

&,,),(

)(),( 22 +=

22

' )(

qpSwhere

x

SSf

x

S

S

E

x

EEx

+=

∂

∂=

∂

∂⋅

∂

∂=

∂

∂=

y

zq

x

zp

x

qq

x

pp

x

s

∂

∂=

∂

∂=

∂

∂+

∂

∂=

∂

∂,,22

yx

zq

x

zp

∂∂

∂+

∂

∂=

2

2

2

22

2 2

2

2( ) '( )

2( ) '( )

x

y

E pr qs f s

E ps qt f s

p p z zp x x y

q y x x yδ δ δ δ

= +

= +

∂ ∂ ∂ ∂= + = +

∂ ∂ ∂ ∂ ∂

- equation 11.1.2

ysxr δδ +=

ytxsq δδδ +=

in our case δξδδξδ qypx =&

δξδ

δξδ

⋅+=

⋅+=

)(

)(

qtpsq

qsprp

From (2)

δξδ

δξδ

'2

'2

f

Eq

f

Ep

y

x

=

=

As δξ → 0 , we obtain the differential equations

99

==

=+=== −

'2,

'2

)),((,, 122

f

Eq

f

Ep

yxEfqpzqypx

yx ɺɺ

ɺɺɺ

(x, y, z, p & q) solvable

5 Differential Eqs

one more time by differentiating

)),((

'2,

'2

..,,

1yxEfz

f

Eqy

f

Epx

trwqypx

yx

−=

====

==

ɺ

ɺɺɺɺɺɺ

ɺɺ ξ

iv. General Case

R(p,q) = f(ap + bq)

R(p,q) = f(P2 + q2)

R(p,q) = arbitarary

ysxr

yyx

zx

x

zp

yqxpz

δδ

δδδ

δδδ

+=

∂∂

∂+

∂

∂=

+=2

2

2

- equation 11.1.3

ytxs

yy

zx

yx

zq

δδ

δδδ

+=

∂

∂+

∂∂

∂=

2

22

100

⋅=

=

y

xH

y

x

ts

sr

q

p

δ

δ

δ

δ

δ

δ

where H is the Hessian Matrix

2

2

2

2

y

z

x

ztr

∂

∂+

∂

∂=+ : Laplacian

Differentiating E(x,y) w.r.t x & y

x

q

q

E

x

p

p

E

x

EEx

∂

∂⋅

∂

∂+

∂

∂⋅

∂

∂=

∂

∂=

SRrR

Syx

zR

x

zR

qp

qp

⋅+⋅=

∂∂

∂⋅+

∂

∂⋅=

2

2

2

tRSRE qpy ⋅+⋅=

⋅=

q

p

y

x

R

RH

E

E - equation 11.1.4

from equation 11.1.3 & equation 11.1.4

Let δξδ

δ⋅

=

q

p

R

R

y

x

q

p

Ry

Rx

=

=

ɺ

ɺ

Then δξδ

δ

δ

δ

=

⋅=

⋅=

y

x

q

p

E

E

R

RH

y

xH

q

p

101

==

+=+=⋅+⋅=

+==

=

yx

qp

qpq

p

EqEp

qRpRyqxpzy

qx

pz

qRpRzRy

Rx

ɺɺ

ɺɺɺ

ɺɺ

ɺ

,

,

δξ

δ

δξ

δ

δξ

δ

5unknown variables : x,y,z, p & q

Figure 11—6

II. 11.3 Singular points

Figure 11—7

),(),( 00 qpRqpR < for all ),(),( 00 qpqp ≠

Figure 11—8

102

0,0

0,0

====

==

qp

qp

RyRx

RR

ɺɺ

)(2

1),( 22

qpqpR += - equation 11.3.1

(p,q) = (0,0) – singular point

)2(2

1 22

0 cybxyaxzz +++=

Thus byaxdx

dzp +==

& cybxdy

dzq +==

substituting p & q in equation 11.3.1

])()[(2

1)(

2

1),( 2222

cybxbyacqpqpR +++=+=

222222 )(2

1)()(

2

1ycbbxycaxba +++++=

The brightness gradient is

ycbbxcaE

bycaxbaE

y

x

)()(

)()(

22

22

+++=

+++=

Differentiating again,

+=

+=

+=

22

22

)(

cbE

bcaE

baE

yy

xy

xx

3 equations, 3 unknowns

103

III. Stereo graphic projection

Figure 11—9

=+

+=

+=

+

222

22

22

)(

2

Rba

R

qp

a

b

R

qf

aR

b

solve the above equations

2222 11

2,

11

2

qp

qg

qp

pf

+++=

+++=

conversely

2222 4

4,

4

4

gf

gq

gf

fp

−−=

−−=

104

IV. 11.7 Relaxation methods

characteristic strip expansion

E(x , y) = R(p , q)

= Rs(f,g)

Minimize

∫ ∫ +++= dxdygygxfyfes )()( 2222

A low, Minimize

∫ ∫ −= dxdygfRyxEe si

2)),(),((

overall, minimize is ee λ+

if brightness measurement are very accurate,

λis large

if brightness measurement are very noisy,

λis small

105

minimize ∫ ∫ dxdyggffgfF yxyxI ),,,,,(

= It’s a problem in the calculus of variations.(see appendix)

The corresponding Euler equations are

0

0

=∂

∂−

∂

∂−

=∂

∂−

∂

∂−

yx

yx

ggg

fff

Fy

Fx

F

Fy

Fx

F

where Ff is the partial derivative of F with respect to F

22222)),(),(()()( gfRyxEggffF syxyx ⋅++++= λ

The Euler equations for this problem yield

yyf

xxxf

s

sf

fFy

ffx

Fx

f

RgfRyxEF

y

x

2

2)2(

)),(),((2

=∂

∂

=⋅∂

∂=

∂

∂

∂

∂−−= λ

fffyyxx FFy

Fx

ffyx

=∂

∂+

∂

∂=+ )(2

f

RgfRyxE s

s∂

∂−−= )),(),((λ

g

RgfRyxEg s

s∂

∂−−=∇ )),(),((2 λ (2 equations, 2 unknowns(f,g))

Application to photometric stereo consider n images,

106

∫ ∫ ∑ ∫ ∫=

−+++−=n

i

iiIiyxyI dxdygfRyxEdxdyggffxe1

22222 )),(),(())()(( λ

where Ei(x,y) is the brightness measured in the i th image and Ri is the corresponding

reflectance map

The Euler equations for this problem yield

∑

∑

=

=

∂

∂−−=∇

∂

∂−−=∇

n

i

i

iii

i

ii

n

i

i

g

RgfRyxEg

f

RgfRyxEf

1

2

1

2

)),(),((

)),(),((

λ

λ

V. 11.8 Recovering Depth from a needle diagram

Figure 11—10

107

Figure 11—11

0 0

( , )

0 0

( , )

( , ) ( , ) ( )

x y

x y

z x y z x y pdx qdy= + +∫

Minimize the error

dxdypzpzI

px ))()(( 22 −+−∫ ∫

where p and q are the given estimates of the components of the gradient while yx zz &

are the partial derivatives of the best-fit surface

Minimize ∫ ∫ dxdyzzzF yxI ),,(

The Euler equations is

0=∂

∂−

∂

∂−

yx zzz Fy

Fx

F

108

so that from 22 )()( qzpzF yx −+−=

0

)(2)(2

=

−∂

∂⋅+−

∂

∂⋅=

∂

∂+

∂

∂=

qzy

pzx

Fy

Fx

F

yx

zzz yx

)(

0)(2)(2

2

yx

yxyyxx

yyyxxx

qpz

qpzz

qzpz

+=∇

+=+

=−+−

109

12. Motion field & Optical flow

Figure 12—1

• Motion field : a purely geometric concept

• Optical flow : Motion of brightness patterns

observed when a camera is moving relative to the objects being imaged

I. Motion Field

Figure 12—2

110

0

0

0

0

,&

ˆ'

ir r

i

t t

i

v v

r r

f r z

δ δ

δ δ= =

=⋅

- equation 12.1.1

Differentiate - equation 12.1.1

2

0

0000

2

0

000

0

)ˆ(

)ˆ()ˆ(

)ˆ(

)ˆ()ˆ(

'

1

zr

rzvvzr

zr

rzdt

dr

dt

drzr

dt

rd

f

i

⋅

⋅−⋅=

⋅

⋅−⋅=⋅

2

0

00

)ˆ(

ˆ)(

'

1

zr

zvrv

fi

⋅

××=⋅ - equation 12.1.2

II. Optical flow

Figure 12—3

111

Ex1)

Figure 12—4

E(x, y, t) ≡ E(x + uδt , y + vδt , t + δt) where u(x, y) & v(x, y) are the x & y components

of the optical flow vector

For a small time interval tδ , if brightness varies smoothly with x, y, &t,

),,(),,(),,( tyxEt

Et

y

Ey

x

ExtyxEtttvytuxE ≡++++=+++ ρ

α

αδ

α

αδ

α

αδδδδ

where ρ contains second and higher order terms in δx , δy , and δt

0=⋅∂

∂+⋅

∂

∂+⋅

∂

∂

t

t

t

E

t

y

y

E

t

x

x

E

δ

δ

δ

δ

δ

δ

0=++ tyx EvEuE (Optical flow constraint equation)

u, v : two variables

one equation :

⇒ one more constraint needed

III. Smoothness of the optical flow

Usually, the motion field varies smoothly minimize a measure of departure from

smoothness

112

∫ ∫ +++= dxdyvyvxuyuxes ))()(( 2222

Also, minimize the error in the optical flow constraint eq.

∫ ∫ ∫∫ ++=−= ++ ydxdEvEuEdxdytyxEtyxEe tyxatttatc

22 )(]),,(),,([ δ

Hence, minimize cs ee λ+ ,

Minimize cs ee λ+

Where ∫ ∫ +++= dxdyvyvxuyuxes ))()(( 2222

∫ ∫ ++= dxdyEvEuEe tyxc

2)(

λ: a parameter that weights the error in the image motion equation relative to the

departure

from smoothness

λ=> Large

=> small

Minimize ∫ ∫ dxdyvvuuvuF yxyx ),,,,,(

22222 )()()( tyxyxYX EvEuEvvuuF ++++++= λ

Euler equations are

0

0

=∂

∂−

∂

∂−

=∂

∂−

∂

∂−

yx

yx

vvv

uuu

Fy

Fx

F

Fy

Fx

F

113

yyyu

xxxu

xtyxu

uux

Fy

uux

Fx

EEvEuEF

y

x

2)2(

2)2(

)(2

=∂

∂=

∂

∂

=∂

∂=

∂

∂

++= λ

uuuEEvEuE yyxxxtyx

2)( ∇=+=++λ

Similarly,

0

)( 2

=++

∇=++

tyx

ytyx

EvEuE

vEEvEuEλ

IV. Boundary conditions

*Natural boundary conditions (see appendix)

ds

dxF

ds

dyF

ds

dxF

ds

dyF

yx

yx

vv

uu

=

=

where S denotes arc length along the boundary curve

T

ds

dx

ds

dyn

−= ,ˆ is a unit vector

perpendicular to the boundary

Rewriting the above conditions.

0ˆ),(

0ˆ),(

=⋅

=⋅

nFF

nFF

yx

yx

vv

uu

114

in our case, 22222 )()()( tyxyxyx EvEuEvvuuF ++++++= λ

0ˆ),(

0ˆ),(

=⋅

=⋅

nvv

nuu

yx

yx

V. Discrete case

From cs eee λ+=

dxdyEvEuEdxdyvvuu tyxyxyx

22222 )())()(( ++++++= ∫ ∫∫ ∫ λ

)( ij

i j

ij cse λ+=∑∑

))()()()((4

1 2

,1,

2

,,1

2

,1,

2

,,1, jijijijijijijijiji vvvvuuuus −+−+−+−= ++++

2)( tijyijxij EvEuEc ++=

Differentiating w.r.t klkl vu & yields

,)(2)(2

,)(2)(2

ytklyklxklkl

kl

xtklyklxklkl

kl

EEvEuEvvv

e

EEvEuEuuu

e

+++−=∂

∂

+++−=∂

∂

λ

λ

where vu & are average of u & v

115

0

0

=∂

∂

=∂

∂

kl

kl

v

e

u

e

solve this

tyklxklxyklyx

txklyxklyklyx

EEvEuEEvEE

EEvEEuEuEE

λλλλ

λλλλ

−++−=++

−−++=++

)1())(1(

,)1())(1(

222

222

Iterative scheme is

.)(1

,)(1

22

1

22

1

y

yx

t

n

kly

n

klxn

kl

n

kl

x

yx

t

n

kly

n

klxn

kl

n

kl

EEE

EvEuEvv

EEE

EvEuEuu

++

++−=

++

++−=

+

+

λ

λ

Figure 12—5

116

13. Photogrammetry & Stereo

I. Disparity between the two images.

Figure 13—1

Disparity : Dxx rl =− ''

)3(

)2()(2

)1()(2

''

'

'

−==

−←−

=

−←+

=

z

y

f

y

f

y

lineredz

bx

f

x

linebluez

bx

f

x

rl

r

l

solve (1) (2) & (3)

fz

bxx rl =− )( '' : disparity

''''

''

''

''

,2/)(

,2/)(

rlrl

rl

rl

rl

xx

bfz

xx

yyby

xx

xxbx

−=

−

−=

−

−=

117

Dxxz

rl

11''

=−

∝

II. Photogrammetry

Figure 13—2

• absolute orientation

),,( knownrr rl

r

r

r

z

y

x

0rrRr lr +=

333231

232221

131211

rrr

rrr

rrr

component

rotaional

34

24

14

r

r

r

component

naltranslatio

xr

118

where R is a 3*3 orthonormal matrix

IRRT =⋅

rlll

rlll

rlll

zrzryrxr

yrzryrxr

xrzryrxr

=+++

=+++

=+++

34333231

24232221

14131211

3 equations => 12 unknowns

• Orthonormality

IRRT =⋅

1

1

1

2

33

2

32

2

31

2

23

2

22

2

21

2

13

2

12

2

11

=++

=++

=++

rrr

rrr

rrr

0

0

0

133312321131

332332223121

231322122111

=++

=++

=++

rrrrrr

rrrrrr

rrrrrr

Figure 13—3

119

1212 rrll rrrr −=−

3+7 = 10 equations

12 unknown variables

P&Q : 10+3 = 13 equations

Figure 13—4

0rrRr lr +⋅= - equation 13.3.1

=

=

34

24

14

0

333231

232221

131211

r

r

r

r

rrr

rrr

rrr

R

III. Relative orientation

rl rr , ; not known rl zz , unknowns

The projection points ),(&),( ''''

rrll yxyx known determine the ratios of x and y to z

using

120

r

rr

r

rr

l

ll

l

ll

z

y

f

y

z

x

f

x

z

y

f

y

z

x

f

x

==

==

''

''

&

&

- - equation 13.4.1

12 + 2*(n)

substitute B into A

l

r

rl

ll

l

l

rlll

z

f

f

zxrzr

f

zyr

f

zxr

z

f

xrzryrxr

)()'( '

141312

'

11

14131211

=+++

=+++

⇒

==+++

==+++

==+++

l

r

l

ll

r

l

r

r

l

ll

r

l

r

r

l

ll

z

zf

z

frfryrxr

s

y

z

zy

z

frfryrxr

s

x

z

zx

z

frfryrxr

3433

'

32

'

31

'

2423

'

22

'

21

'

1413

'

12

'

11

- equation 13.3.2

12 + 2n ; # of unknowns for n points

7 + 3n ; # of equations

7 from orthonormality & distance length

121

IRRT =⋅

100

010

001

jiifrr

r

ji

i

≠=

=

0

12

∴12 + 2n = 7 + 3n

n = 5

n ≥5

IV. 13.5 Using a known relative orientation

sf

z l = ; constant sxf

zxx l

l

ll ⋅== /' syy ll ⋅= ' sfz l ⋅=

where s is a scaling factor

from equation 13.3.2

wcsrsfryrxrz

vbsrsfryrxry

uasrsfryrxrx

llr

llr

llr

+=+++=

+=+++=

+=+++=

3433

'

32

'

31

2423

'

22

'

21

1413

'

12

'

11

)(

)(

)(

wcs

vbs

z

y

f

y

wcs

uas

z

x

f

x

r

rr

r

rr

+

+==

+

+==

'

'

122

V. Computing depth

Once IR , ),(),,(, ''''

0 rrll yxyxr known, The depth ),( rl zz can be computed

as follows

Figure 13—5

From equation 13.3.2, multiply by f

z l

rl

ll

r

r

l

ll

r

r

l

ll

zrzrf

yr

f

xr

zf

yrzr

f

yr

f

xr

zf

xrzr

f

yr

f

xr

=+++

=+++

=+++

3433

'

32

'

31

'

2423

'

22

'

21

'

1413

'

12

'

11

)(

)(

)(

From any two equations

Compute rl zz &

Then compute

r

rrT

rrrl

l

llT

llll

zf

y

f

xzyxr

zf

y

f

xzyxr

==

==

1,,),,(

1,,),,(

''

''

123

VI. 13.7 exterior orientation

Figure 13—6

0rrRr ac +⋅=

caaa

caaa

caaa

zrzryrxr

yrzryrxr

xrzryrxr

=+++

=+++

=+++

34333231

24232221

14131211

IRRT =⋅ (orthonormality)

c

c

c

c

z

y

f

y

z

x

f

x==

'&

'

34333231

24232221

34333231

14131211

'

'

rzryrxr

rzryrxr

z

y

f

y

rzryrxr

rzryrxr

z

x

f

x

aaa

aaa

c

c

aaa

aaa

c

c

+++

+++==

+++

+++==

- equation 13.7.1

# of equations = 2n +6

# of unknowns = 12

2n+6 = 12 => n=3 (at least we need three equations)

124

VII. Interior Orientation

• scaling error

• translation error

• skewing error

• shearing error

An affine transformation

232221

131211

)()('

)()('

az

ya

z

xay

az

ya

z

xax

c

c

c

c

c

c

c

c

++=

++=

from equation 13.7.1

34333231

24232221

34333231

14131211

'

'

szsysxs

szsysxs

f

y

szsysxs

szsysxs

f

x

aaa

aaa

aaa

aaa

+++

+++=

+++

+++=

ijsR : not orthonormal

IRRijij s

T

s ≠⋅ 12 unknowns

#of equations = 2n

∴ 2n=12 ; n=6

125

VIII. Finding conjugate points

i. Gray-Level Matching

• (reference figure 3-1)

z

bx

f

x

z

bx

f

x rl 2&2'' −

=+

=

at a matched point

),(),( ''''

rrrlll yxEyxE =

)(

)',2()',2(~

''yyy

yz

bx

fEyz

bx

fEE

rl

rl

=≅

−=

+

∵

Let z

bfyxd

z

x

f

x=≅ )','(&

'

)'),','(2

1'()'),','(

2

1'( yyxdxEyyxdxE rl −=+

criterion : minimize is eee λ+=

where ∫ ∫ ∇⋅∇= '')( 2dydxdes

∫ ∫ −= '')( 2dydxEEe rli

(solution) using Euler equation

0''

'' =∂

∂−

∂

∂−

yx dd Fy

Fx

Fd

where

126

( )

]''

[2

1()]()[2

]'

())

2

1()

2

1(

'

()()][()[2

)]'),','(2

1'()'),','(

2

1'([)( 22

x

E

x

EEE

x

E

x

EEEEFd

yyxdxEyyxdxEdF

rl

rl

rl

rl

rl

∂

∂+

∂

∂−=

∂

∂−−

∂

∂−−=

−−++∇⋅∇=

λ

λ

λ

]'

()

'

()[

2

1()]()[)(

)(2''

22

22

''

x

E

x

EEEd

dFy

Fx

rl

rl

dydx

∂

∂+

∂

∂−=∇∇

∇∇=∂

∂+

∂

∂

λ∵

127

14. Pattern classification

Reference – “ Pattern Classification & Scene Analysis”

Dudda & Hart (Wiley Interscience pub. C)

1. Bayes Decision Theory

Fundamental statistical approach to the problem of pattern classification

• Maximum likelihood estimation : parameters fixed but unknown

• Baysian estimation : parameters, random variables having some known a prior

distribution

(A) Maximum likelihood estimation

Figure 14—1

Given ; 1. Model of system

2. pata : outcome of some type of probabilistic experiment.

Goal : Estimate unknown model parameters from data

Let nii .,.........2,1, =λ be n I.I.D. observations on a random variable X drawn from

):( θxf x

128

The joint lpdf of nXXX ,........, 21 is 1xf

Where );..,.........( 1 θnxxL is called the likelihood function (LF)

The maximum likelihood estimation (MLE) θ is that value that maximizes the LF, that

is

);..,.........()ˆ;..,.........( 11 θθ nn xxLxxL ≥

Ex1) Assume ),(; 2σµNX where σis known. Compute the MLE of the mean μ

<solution> The LF for h realizations of X is on a

−−

= ∑

=

h

i

i

n

xXL0

2

22)(

2

1exp

2

1);( µ

σπσµ

since the log function is monotonic, the maximum of L(x ; μ) is also that of log L(x ; μ)

Hence ∑=

−−−=n

i

ixn

xL1

2

2

2 )(2

1)2log(

2);(log µ

σπσµ

Set 0);(log

=∂

∂

u

xL µ

This yields ∑=

=−∂

n

i

ix0

20)(

1µ ∑

=

=n

i

ixn 1

1µ

Ex2) Consider the normal p.d.f.,

129

∑=

−−=n

i

x xxf1

2

22

2 ))(2

1exp(

2

1),,( µ

σπσσµ

The log likelihood function is

∑=

−−−−=n

i

ixnn

jxxL1

2

2

2

21 )(2

1log2log

2),,.......(log µ

σσπσµ

Now set 0,0 =∂

∂=

∂

∂

σµ

LL

Obtain the simultaneous equations

∑

∑

=

=

=−+−

=−

n

i

i

n

i

i

xn

x

1

2

3

1

0)(1

0)(

µσσ

µ

Hence ∑=

=n

i

iXn 1

1µ

∑=

−=n

i

ixn 1

22 )ˆ(1

ˆ µσ

(B) Baysian Estimation

Suppose that an classifier to separate two kinds of lumber, ash and birch

Figure 14—2

130

A prior : • probability p(w1) : the next piece is ash

• probability p(w2) : the next piece is birch

Decision Rule : Decide w1 if p(w1) > p(w2)

Decide w2 otherwise

Bay’s Rule ;

)(

)()|()|(

xp

wpwxpXwp

jj

j

⋅=

where ∑=

⋅=r

j

jj wpwxpxp1

)()|()(

Bayes Decision Rule for minimizing the probability of error ;

Decide w1 if p(w1|x) > p(w2|x)

Decide w2 otherwise

Decide w1 if P(x| w1)p(w1) > p(x| w2)p(w2)

Decide w2 otherwise

If equally likely, p(w1)=p(w2), then

131

Decide w1 if p(x| w1) > p(x| w2)

Decide w2 if otherwise

Figure 14—3

Probability error : p(error)

( ) ( , )

( | ) ( )

p error p error x dx

p error x p x dx

∞

−∞

∞

−∞

=

=

∫

∫

where p(error|x)

12

21

)|(

)|(

wdecideweifxwp

wdecideweifxwp

Figure 14—4

132

(B-1) Generalized Form (for Bayes classifier)

Let .......... 1 sww=Ω be the finite set of S states of nature and )........ 1 aA αα= be the

finite set of possible actions.

Let )|( ji wαλ be the loss incurred for taking action iα when the state of nature is jw

Let the feature vector X be a d-component vector-valued r.v

Then )(

)()|()|(

xp

wpwxpxwp

jj

j =

Where ∑=

=s

j

jj wpwxpXp1

)()|()(

The expected loss associated with taking action iα is

)|()|()|(1

∑=

⋅=s

j

jjii XwpwXR αλα

Target: to find a Bayes decision rule against p(wj) that minimizes the overall risk

The overall risk is given by

∫ ⋅= xdxpxxRR )()|)((α

clearly, if )(xα is chosen so that )|)(( xxR α is as small as possible, for every X , then

the overall risk will be minimized Bayes decision rule for the general form; to minimize

the overall risk.

133

1.Compute the conditional risk

∑=

⋅=s

j

jjii XwpwXR1

)|()|()|( αλα

for i=1,………..a

2.Select the action iα for which )|( XR iα is minimum

3.Obtain the resulting minimum overall risk called the Bayes risk

B.2 Two category classification

Let )|( jiij wαλλ = the loss incurred for deciding wi when the true state of nature is wj.

)|()|()|(

)|()|()|(

2221212

2121111

XwpXwpXR

XwpXwpXR

λλα

λλα

+=

+=- equation 14.1

The decision rule :

Decide w1 if )|()|( 21 XRXR αα < - equation 14.2

Decide w2 otherwise

If we substitute 1 into 2

We have

)|()()|()(

)|()|()|()|(

1112122212

222121212111

XwpXwp

XwpXwpXwpXwp

λλλλ

λλλλ

−<−⇒

+<+

Hence likelyhood ratio

134

Figure 14—5

<Minimum – error – rate classification>

Symmetrical on zero-one loss function

The loss )|( ji wαλ = 0 i=j for i,j=1,……c

1 i≠j

all errors are equally costly

The conditional risk is

∑=

⋅=c

j

jjii XwpwXR1

)|()|()|( αλα

)|(1

)|(

Xwp

Xwp

i

c

ij

j

−=

=∑≠

To minimize the average prob of error, we should select the i that maximizes the a

posterior prob

)|( Xwp i In other words, for minimum error rate j

Decide iw if )|()|( XwpXwp ji > for all ,ij ≠

B-3 The multicategory classification

135

A pattern classifier

Figure 14—6

),(Xg i for i=1,………c ; discriminant function

The classification is said to assign a feature vector X to class jw

If )()( XgXg ji > for all ij ≠

)|()( XwpXg ii = ; a posterior prob

∑=

=c

j

jj

ji

wpwXp

wpwXp

1

)()|(

)()|( by Bayes rule

)(log)|(log

)()|(

ii

ii

wpwXp

wpwXp

+⇒

⇒

Decision Rule

1. Divide the feature space into C division regions, R1,………..Rc

2. If )()( XgXg ji > for all j≠i then X is in RI

136

In other words, assign X to wI

Ex) three category classification

Probability error calculation

P(error Xo) )()( 00 XPXerrorP ⋅=

)()()()( 330220 wPwxPwPwXP +=

Total error

P(error) ∫∞

∞−= dxxerrorP ),(