Image Formation

N.A.Thacker, ISBE, University of Manchester.

Intensity reflectivityObject interactionLens modelsSpectral sensitivityHuman eyeCCD arrayCamera modelPoint spread functionImages from ImagesError propagationMonte-CarloSummary

Figures c.o. Tim Ellis.

1

Intensity Reflectivity

Source Illumination Material Reflectivity(E) (S)

sunshine 10,000 black velvet 0.01cloudy day 1,000 stainless steel 0.65office 100 white wall 0.80moonlight 0.01 chrome 0.90

snow 0.93

Table 1: Typical values for relative levels of reflected light = E × S.

The brightest/best lit objects are 107× brighter than thedarkest/poorest lit.

2

Object Interaction with Light

The image of an object is subject to several phys-ical processes;

• spectral reflectance

• specularities

• shading

•mutual illumination

• shadows

A diffuse body reflects equally in all directions(Lambertian)A specular surface has the property that incidentlight reflects directly from the surface.Torrence-Sparrow , dichromatic reflection model.

3

Shading

Let z = z(x, y) be a depth map with illuminant directionL = (Lx, Ly, Lz) = (cosτ sinσ, sinτ sinσ, cosσ) ie: asingle distant light source with σ slope and τ tilt.Putting; p = ∂z(x, y)/∂x q = ∂z(x, y)/∂y

I(x, y) =p cosτ sinσ + q sinτ sinσ + cosσ

(p2 + q2 + 1)1/2

≈ const + p cosτ sinσ + q sinτ sinσ

Taking Fourier transforms;

Fz = mz(f, θ) exp[i(φz(f, θ) + π/2)]

Fp = 2π cosθ f mz(f, θ) exp[i(φz(f, θ) + π/2)]

Fq = 2π sinθ f mz(f, θ) exp[i(φz(f, θ) + π/2)]

and ignoring the constant term

FI(f, θ) ≈ 2π sinσ f mz(f, θ)exp[i(φz(f, θ) + π/2)]

×(cosθ cosτ + sinθ sinτ )

ie: The image is related to the depth map by a convolu-

tion.

FI(f, θ) ≈ Fz(f, θ)2π sinσf (cosθ cosτ + sinθ sinτ )

Extended light sources are linear combinations so shapefrom shading can be solved by deconvolution.[A. Pentland]

4

Optical Lens Characteristics

• f - focal length

•D - diameter of aperture (light flux ∝ D2)

• f- number = f/D

The ‘speed’ of the lens (luminous flux throughthe aperture) is halved for each ‘stop’: -

1 :1.4 : 2 : 2.8 : 4: 5.6 : 8: 11 : 16 : 22

• depth of field

• field of view

• geometric distortion (eg: radial)

• sharpness (e.g. PSF)

5

Imaging with a Lens

6

Thin Lens and Pin Hole models

1

f=

1

u+

1

v

As the aperture decreases (f → ∞) the modelapproximates a pin hole camera.

7

Pin-Hole Camera Model.

The simplest camera model (pinhole) is written as

u =uv1

=fku 0 u00 −fkv v00 0 1

xyf

= Cxyf

The vector (x, y, f ) is obtained from perspective geom-etry

xyf

= f/ZXYZ

=f/Z 0 0 00 f/Z 0 00 0 f/Z 0

XYZ1

= P

XYZ1

8

Finally a point in the external world co-ordinate systemXw transforms to the camera frame using

X = RXw + T

which can be written in homogenous coordinates as

XYZ1

=R T0T 1

Xw

Yw

Zw

1

= H

Xw

Yw

Zw

1

Thus the full process of projecting a point in 3D onto theimage plane can be written as a single matrix product

u =uv1

= CPH

Xw

Yw

Zw

1

where CPH is the 3x4 projection matrix.

This allows camera calibration to be achieved via closedform optimisation of a quadratic Likelihood function.

9

Radial Distortion

Conventional optics can generate images which are sharpbut geometrically distorted.The mathematical form is best represented as a modifica-tion to radius r

r′ = r(1 + k1r2 + k2r

4 + k3r6 + ...)

10

Point Spread Functions

The physical properties of the optics and the sensor oftenlead to spatial blurring (Point Spread Function) h(x, y).

i′(x, y) = i(x, y)⊗ h(x, y)

which can be considered (via the convolution theorem) asan Optical Transfer Function (OTF) in Fourier space.

FFT (i′(x, y)) = FFT (i(x, y))×H(θ, φ)

11

Optical Spectral Sensitivity

i(x, y) =∫

∞

0R(λ)S(x, y, λ)E(λ)dλ

where; E(λ) - illuminant spectral distributionS(x, y, λ) - surface reflectance distributionR(λ) - sensor spectral sensitivity distribution

12

CCD Array

• Photosensitive detector: planar silicon

• CCD: mega pixel (1,000 x 1,000 array).

• brightness range (at fixed exposure time) - 105,

actually encoded 102

• uniform spatial distribution

• colour images constructed from mosaic sensors

CCD images have linear sensitivity ( ≈ 100 levels) andGaussian noise.

I(x, y) = i(x, y)± σ

13

Human Eye

• Photosensitive detector: hemispherical retina

• rods: 75- 150 million high sensitivity monochromaticreceptors

• cones: 6-7 million low sensitivity colour receptors (infovea).

• non-uniform spatial distribution

• brightness range - 107, actually encoded 105.

The human eye has sensitivity to proportional differences( ≈ 100 levels), i.e.:

S(x, y) = i(x1, y1)− i(x2, y2) ± γS(x, y)

These properties seem to be useful for exploiting invari-ances (both of geometry and illumination), during recogni-tion.

[Thacker 2007]

14

Making Images from Images.

In simple image processing the requirements of an imageprocessing algorithm may be purely to enhance the imagefor viewing.

But; the aim of advanced image processing to producean image that makes certain information explicit in theresulting image values for automated data extraction.

eg: edge strength maps.

Generally, high values located over features of interest. Theprocess which determines a good algorithm is its behaviourin the presence of noise, in particular does the resultingimage give results which really can be interpreted purelyon the basis of output value.

ie: is a high value genuine or just a product of the prop-agated noise.

There are two complementary ways of assessing algorithms:Error Propagation and Monte-Carlo techniques.

15

Error Propagation.

General Approach for Error Propagation.

∆f 2(X) = ∇fTCX∇f

where CX is the parameter covariance and ∇f is a vectorof derivatives

∇f = (∂f

∂X1

,∂f

∂X2

,∂f

∂X3

, ....)

and ∆f (X) is the standard deviation on the computedmeasure

If we apply this to image processing assuming that im-ages have uniform random noise then we can simplify thisexpression to

∆f 2xy(I) =

∑

nmσ2nm(

∂fxy∂Inm

)2

ie: the contribution to the output from each independentvariance involved in the calculation is added in quadrature.

[Haralick 1994]

16

Common Statistical Models.

Gaussian sensor (CCD, MRI, CT)

I(x, y) = i(x, y) ± σ

Simple non-linear transformations can be used to transformsome statistical models to approximate Gaussian.

Logarithmic sensor (Retina)

S(x, y) = i(x, y) ± γi(x, y)

→ I(x, y) = log[i(x, y)] ± σ

Poisson sensor (Low light, Histogram, SAR)

P (x, y) = i(x, y) ± γ√

i(x, y)

→ I(x, y) =√

i(x, y) ± σ

All are of the form

I(x, y) = f (scene) ± σ

17

Analysis Chains.

F(I)

J(I) K(I) L(I)

H(I)G(I)

uniform errors uniform errors uniform errors

non−uniform errors really non−uniform maggots

Propagation through stable algorithm.

Propagation through un−stable algorithm.

Alternatively, every algorithm should provide detailed errorestimates and make appropriate use of them in subsequentstages.

[Courtney 2001]

18

Image Arithmetic.

We can drop the xy subscript as it is not needed.

Addition:O = I1 + I2

∆O2 = σ21 + σ2

2

Division:O = I1 / I2

∆O2 =σ21

I22+

I21σ22

I42

Multiplication:O = I1 . I2

∆O2 = I22σ21 + I21σ

22

Square-root:O =

√

(I1)

∆O2 =σ21

I1

19

Logarithm:

O = log(I1)

∆O2 =σ21

I21

Polynomial Term:

O = In1

∆O2 = (nIn−11 )2σ2

1

Square-root of Sum of Squares:

O =√

I21 + I22

∆O2 =I21σ

21 + I22σ

22

I21 + I22

Notice that some of these results are independent of the im-age data. Thus these algorithms preserve uniform randomnoise in the output image.

Such techniques form the basis of the most useful buildingblocks for image processing algorithms.

Some however, (most notably multiplication and division)produce a result which is data dependent, thus each outputpixel will have different noise characteristics. This compli-cates the process of algorithmic design.

20

Linear Filters.

For Linear Filters we initially have to re-introduce the spa-tial subscript for the input and output images I and O.

Oxy =∑

nmhnmIx+n,y+m

where hnm are the linear co-efficient.

Error propagation gives:

∆O2xy =

∑

nm(hnmσx+n,y+m)

2

for uniform errors this can be rewritten as

∆O2xy = σ2 ∑

nm(hnm)

2 = K σ2

Thus linear filters produce outputs that have uniform er-rors.

Unlike image arithmetic, although the errors are uniformthey are no-longer independent because the same data isused in the calculation of the output image pixels. Thuscare has to be taken when applying further processing.

21

Histogram Equalisation.

For this algorithm we have a small problem as the differ-ential of the processing process is not well defined.

If however we take the limiting case of the algorithm fora continuous signal then the output image can be definedas:

Oxy =∫ Ixy0

fdI/∫

∞

0fdI

where f is the frequency distribution of the grey levels (iethe histogram.

This can now be differentiated giving

∂Oxy

∂Ixy= K fIxy

ie: the derivative is proportional to the frequency of oc-currence of grey level value Ixy and the expected varianceis:

∆O2xy = K σ2

xyf2Ixy

Clearly this will not be uniform across the image, nor wouldit be in the quantised definition of the algorithm.

Thus although histogram equalisation is a popular pro-cess for displaying results (to make better use of the dy-namic range available in the display) it should generally beavoided as part of a Machine Vision algorithm.

22

Monte-Carlo Techniques.

Differential propagation techniques are inappropriate when:

• Input errors are large compared to the range of linearityof the function.

• Input distribution is non-Gaussian.

The most general technique for algorithm analysis which isstill applicable under these circumstances is known as theMonte-Carlo technique.

This techniques takes values from the expected input dis-tribution and accumulates the statistical response of theoutput distribution.

Non−Linear Function. Non−Gaussian Errors.

Algorithm

f(x,y)

x

y

Monte−Carlo Technique.

23

Generating Distributions.

Conventional random number generators produce uniformrandom variables x. The method for generating a variatefrom the distribution f (x) using x is to solve for y in

x =∫ y0−∞

f (y)dy/∫

∞

−∞f (y)dy

ie: x is used to locate a variable some fraction of the waythrough the integrated distribution (c.w. histogram equal-isation).

eg: a Gaussian distribution leads to the BOX MULLERmethod

y1 =√

(−2ln(x1))cos(2πx2)

y2 =√

(−2ln(x1))sin(2πx2)

which generates two Gaussian random deviates y1 and y2for every two input deviates x1 and x2.

Armed with distribution generators we can generate manyalternative images for statistical testing from only a few ex-amples of image data. Examples:

• drop out noise in images.

• bit inaccuracy and thermal noise.

• feature location and orientation accuracy.

24

Summary.

Conventional image formation combines many aspects (op-tics, geometry and solid state physics).

The human eye and sensor equipment are characteristicallydifferent. This seems to be a consequence of jointly evolvingthe processor and the sensor.

Many imaging systems generate images which can be mod-elled as

I = f (scene) ± σ

Though other models (e.g. Rician) exist, this is the con-ventional basis for statistical algorithm design.

Pre-processing alters statistical properties of an image, thesecan be assesed via error propagation or Monte-Carlo.

Image based matching metrics should be constructed usinga valid noise model (e.g. Least-Squares for homogenousGaussian errors).

25

References.

www.tina-vision.net

A. Pentland, Neural Computation: 1,2, 208-217, 1989.

R.M. Haralick, Performance Characterisation in Com-

puter Vision, CVGIP-IE, 60, 1994, pp.245-249.

P. Courtney and N.A.Thacker, Performance Character-

isation in Computer Vision: The Role of Statistics in

Testing and Design, ch. “Imaging and Vision Systems:Theory Assessment and Applications”, J. Blanc-Talon andD. Popescu, NOVA Science Books, 2001.

N.A.Thacker and E.C. Leek, Retinal Sampling, Feature

Detection and Saccades; A Statistical Perspective. proc.BMVC 2007, vol 2, pp 990-1000, 2007..

26