ME5286 – Lecture 2 (Theory)

#1

Lecture 2: Image Formation and Cameras

Saad J Bedrossbedros@umn.edu

ME5286 – Lecture 2 (Theory)

#2

Last Lecture

• What is Computer vision: deals with the formation, analysis and interpretation of Images

• Evolving field in Artificial Intelligence• Enabler in Robotics as a smart sensor for

better decision making.

ME5286 – Lecture 2 (Theory)

#3

Outline for this Lecture

• Image Formation• Cameras and Lenses• Human Visual System• Digital Cameras• Digital Image representation

ME5286 – Lecture 2 (Theory)

ME5286 – Lecture 2 (Theory)

The Electromagnetic Spectrum

• Radio Waves - communication • Microwaves - used to cook• Infrared - “heat waves”• Visible Light - detected by your eyes• Ultraviolet - causes sunburns• X-rays - penetrates tissue• Gamma Rays - most energetic

ME5286 – Lecture 2 (Theory)

The Multi-Wavelength Sun

X-Ray UV Visible

Infrared RadioComposite

ME5286 – Lecture 2 (Theory)

#7

Visible Spectrum

Light waves extend in wavelength from about 400 to 700 nanometers

ME5286 – Lecture 2 (Theory)

#8

Image Formation

ME5286 – Lecture 2 (Theory)

#9

Quantum Theory of Light

• Newton proposed that light is a stream of particles traveling in a straight line. Each particle is called a quantum and each quantum of light is a photon. Thus the intensity of light is measured in number of photons. – the visible spectrum is from 380 nm (violet) to 760 nm (red)

• refraction occurs when light enters a different medium causing the velocity of the light to change, this change bends the direction of the light

• Short wavelengths (violet) of light are refracted more than longer wavelengths (red). This is why a spectrum is formed from white light passing through a prism and it also causes the problem of chromatic aberration

ME5286 – Lecture 2 (Theory)

#10

How do we Capture an Image ?

ME5286 – Lecture 2 (Theory)

Image formation• There are two parts to the image formation

process:

(1)The geometry, which determines where in the image plane the projection of a point in the scene will be located.

(2) The physics of light, which determines the brightness of a point in the image plane.

f(x,y) = i(x,y) r(x,y)Simple model:i: illumination, r: reflectance

ME5286 – Lecture 2 (Theory)

Image formation

• Let’s design a camera– Idea 1: put a piece of film in front of an object– Do we get a reasonable image? Blurring …

FilmObject

ME5286 – Lecture 2 (Theory)

Pinhole camera

• Add a barrier to block off most of the rays– This reduces blurring– The opening known as the aperture– How does this transform the image?

FilmObject Barrier

ME5286 – Lecture 2 (Theory)

Image Formation: Simple Model

Digital Camera

The Eye

Film

ME5286 – Lecture 2 (Theory)

History of Imaging: Camera Obscura

Device that led to Photography and the Camera"When images of illuminated objects ... penetrate through a small hole into a very dark room ... you will see [on the opposite wall] these objects in their proper form and color, reduced in size ... in a reversed position, owing to the intersection of the rays". Leonardo da Vinci

Slide credit: David Jacobs

History 1544

ME5286 – Lecture 2 (Theory)

Camera Obscura

• The first camera– How does the aperture size affect the image?– How does the size of the box affect the image?

ME5286 – Lecture 2 (Theory)

“Pinhole” camera model• The simplest device to form an image of a

3D scene on a 2D surface.• Rays of light pass through a "pinhole" and

form an inverted image of the object on the image plane.

perspective projection:center of projection

(x,y)

fXxZ

=fYyZ

=(X,Y,Z)

f: focal length, distance from pinhole to image plane

ME5286 – Lecture 2 (Theory)

Pinhole and the Perspective Projection

(x,y)

screen scene

Is an image being formedon the screen?

YES! But, not a “clear” one.

image plane

effective focal length, f’opticalaxis

y

x

zpinhole

),,( zyx=r

zy

fy

zx

fx

==''

''

zfrr

=''

)',','(' fyx=r

ME5286 – Lecture 2 (Theory)

What is the effect of aperture size?

Large aperture: light from the source spreads across the image (i.e., not properly focused), making it blurry!

Small aperture: reducesblurring but (i) it limits the amount of light entering the camera and (ii) causes light diffraction.

ME5286 – Lecture 2 (Theory)

#20

Shrinking the aperture

• Why not make the aperture as small as possible?– Less light gets through

ME5286 – Lecture 2 (Theory)

Shrinking more the aperture

• What happens if we keepdecreasing aperture size?

• When light passes through a small hole, it does not travel in a straight line and is scattered in many directions (i.e., diffraction)

ME5286 – Lecture 2 (Theory)

#22

Shrinking the aperture

•Pinhole too big - many directions are averaged, blurring the image

•Pinhole too small -diffraction effects blur the image

•Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen.

ME5286 – Lecture 2 (Theory)

Problems with Pinholes

• Pinhole size (aperture) must be “very small” to obtain a clear image.

• However, as pinhole size is made smaller, less light is received by image plane.

• If pinhole is comparable to wavelength of incoming light, DIFFRACTION effects blur the image!

• Sharpest image is obtained when:

pinhole diameter

Example: If f’ = 50mm,

= 600nm (red),

d = 0.36mm

λ'2 fd =

λ

ME5286 – Lecture 2 (Theory) 24

Traditional Photography

• A chemical process, little changed from 1826

• Taken in France on a pewter plate

• … with 8-hour exposure

The world's first photograph

ME5286 – Lecture 2 (Theory)

Lens Based Camera Obscura, 1568

History of Imaging: Adding a Lens

ME5286 – Lecture 2 (Theory)

#26

First Camera Design

ME5286 – Lecture 2 (Theory)

#27

The Reason for Lenses

Gather more light from each scene point

ME5286 – Lecture 2 (Theory)

Adding a lens

• Pinhole replaced by a Lens

• Lens redirect light rays emanating from the object

• Lens improve image quality, leading to sharper images.

ME5286 – Lecture 2 (Theory)

#29

Lenses

• A lens focuses parallel rays onto a single focal point– focal point at a distance f beyond the plane of the lens

• f is a function of the shape and index of refraction of the lens

– Aperture of diameter D restricts the range of rays• aperture may be on either side of the lens

– Lenses are typically spherical (easier to produce)

focal point

F

optical center(Center Of Projection)

ME5286 – Lecture 2 (Theory)

Properties of “thin” lens (i.e., ideal lens)

• Light rays passing through the center are not deviated.• Light rays passing through a point far away from the center

are deviated more.

focal point

f

ME5286 – Lecture 2 (Theory)

Properties of “thin” lens (i.e., ideal lens)

• All parallel rays converge to a single point.• When rays are perpendicular to the lens, it is

called focal point.

focal point

f

ME5286 – Lecture 2 (Theory)

Properties of “thin” lens

• The plane parallel to the lens at the focal point is called the focal plane.

• The distance between the lens and the focal plane is called the focal length (i.e., f) of the lens.

focal point

f

ME5286 – Lecture 2 (Theory)

Thin lenses

• Thin lens equation:

– Any object point satisfying this equation is in focus– What is the shape of the focus region?– How can we change the focus region?– Thin lens applet: http://www.phy.ntnu.edu.tw/ntnujava/index.php (by Fu-Kwun Hwang )

FilmObject Lens

Focal point

Not quite right…

ME5286 – Lecture 2 (Theory)

object

fuv

image

Thin lens equationAssume an object at distance u from the lens plane:

ME5286 – Lecture 2 (Theory)

Thin lens equation (cont’d)

Using similar triangles:

y’/y = v/u

fuv

y’y

image

ME5286 – Lecture 2 (Theory)

Thin lens equation (cont’d)

fuv

y’y

y’/y = (v-f)/f

Using similar triangles:

image

ME5286 – Lecture 2 (Theory)

Thin lens equation (cont’d)

fuv

1 1 1u v f+ =

image

Combining the equations:

ME5286 – Lecture 2 (Theory)

#38

Lens Aperture

ME5286 – Lecture 2 (Theory)

Adding a lens

• A lens focuses light onto the film– There is a specific distance at which objects are “in

focus”• other points project to a “circle of confusion” in the image

– Changing the shape of the lens changes this distance

“circle of confusion”

FilmObject Lens

ME5286 – Lecture 2 (Theory)

depth of field

• The size of blur circle is proportional to aperture size.

ME5286 – Lecture 2 (Theory)

Controlling depth of field

http://en.wikipedia.org/wiki/File:Depth_of_field_illustration.svg

• Changing the aperture size affects depth of field– A smaller aperture increases the range in which the object is

approximately in focus– But small aperture reduces amount of light – need to increase

exposure

ME5286 – Lecture 2 (Theory)

Depth of Field

• Changing the aperture of a camera also changes the amount of the image that is in focus – this amount is called the depth of field

ME5286 – Lecture 2 (Theory)

depth of field trade off

• Changing aperture size (controlled by diaphragm) affects depth of field.– A smaller aperture increases

the range in which an object is approximately in focus (but need to increase exposure time).

– A larger aperture decreases the depth of field (but need to decrease exposure time).

ME5286 – Lecture 2 (Theory)

Depth of field trade off

• Changing the aperture size affects depth of field– A smaller aperture increases the range in which the object is

approximately in focus

f / 5.6

f / 32

http://en.wikipedia.org/wiki/Depth_of_field

FilmAperture

ME5286 – Lecture 2 (Theory)

Depth of Field

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm

The range of depths over which the world is approximately sharp (i.e., in focus).

ME5286 – Lecture 2 (Theory)

Another Example

Large aperture = small DOF

ME5286 – Lecture 2 (Theory)

Varying aperture size

Large aperture = small DOF Small aperture = large DOFDOF is Depth of Field

ME5286 – Lecture 2 (Theory)

Aperture• Small Apertures

(e.g. f11, f16, f22) only let a small amount of light through

• Large Apertures(e.g. f4, f5.6, f8) let through a lot of light

• So for a sunny day you might need to use a small aperture to get the correct exposure

ME5286 – Lecture 2 (Theory)

Thin lens assumption

FilmObject Lens

Focal point

The thin lens assumption assumes the lens has no thickness, but this isn’t true…

By adding more elements to the lens, the distance at which a scene is in focus can be made roughly planar.

ME5286 – Lecture 2 (Theory)

Field of View (Zoom)

ff

• The cone of viewing directions of the camera.• Inversely proportional to focal length.

ME5286 – Lecture 2 (Theory)

Field of View (Zoom)

ME5286 – Lecture 2 (Theory)

Lens Flaws: Chromatic Aberration

• Lens has different refractive indices for different wavelengths.• Could cause color fringing:

– i.e., lens cannot focus all the colors at the same point.

ME5286 – Lecture 2 (Theory)

Chromatic Aberration - Example

ME5286 – Lecture 2 (Theory)

Lens Flaws: Radial Distortion

• Straight lines become distorted as we move further away from the center of the image.

• Deviations are most noticeable for rays that pass through the edge of the lens.

ME5286 – Lecture 2 (Theory)

Lens Flaws: Radial Distortion

No distortion Pin cushion Barrel

ME5286 – Lecture 2 (Theory)

Lens Flaws: Tangential Distortion

• Lens is not exactly parallel to the imaging plane!

ME5286 – Lecture 2 (Theory)

Now the test!• Under or over exposure?

ME5286 – Lecture 2 (Theory)

Now the test!

• Which is the largest aperture?

ME5286 – Lecture 2 (Theory)

Now the test!• Under, or over exposed?

ME5286 – Lecture 2 (Theory)

Now the test!

• Small or large aperture?

ME5286 – Lecture 2 (Theory)

#62

• Cameras are a Copy of the Human Eye !

Human Eye

ME5286 – Lecture 2 (Theory)

Human Eye vs. the Camera

• We make cameras that act “similar” to the human eye

ME5286 – Lecture 2 (Theory)

64

Image formation in the eye

Light receptor

radiant energy

electrical impulses

Brain

ME5286 – Lecture 2 (Theory)

http://www.cas.vanderbilt.edu/bsci111b/eye/human-eye.jpg

Human Eye

• The eye has an iris like a camera

• Focusing is done by changing shape of lens

• Retina contains cones (mostly used) and rods (for low light)

• The fovea is small region of high resolution containing mostly cones

• Optic nerve: 1 million flexible fibers

Slide credit: David Jacobs

ME5286 – Lecture 2 (Theory)

66

Human Eye: Pupil• Hole or opening where light enters

– Or, the diameter of that hole or opening• Pupil of the human eye

– Bright light: 1.5 mm diameter– Average light: 3-4 mm diameter– Dim light: 8 mm diameter

• Camera– Wider aperture admits more light– Though leads to blurriness in the

objects away from point of focus

ME5286 – Lecture 2 (Theory)

Human Eye : Lens

• Focusing is achieved by varying the shape of the lens (i.e., flattening of thickening).

ME5286 – Lecture 2 (Theory)

Human Eye: Retina• Retina contains light sensitive cells that convert light

energy into electrical impulses that travel through nerves tothe brain.

• Brain interprets the electrical signals to form images.

ME5286 – Lecture 2 (Theory)

Retina up-close

Light

ME5286 – Lecture 2 (Theory) © Stephen E. Palmer, 2002

Conescone-shaped less sensitiveoperate in high lightcolor vision

Two types of light-sensitive receptors

Rodsrod-shapedhighly sensitiveoperate at nightgray-scale vision

ME5286 – Lecture 2 (Theory)

Rod / Cone sensitivity

The famous sock-matching problem…

ME5286 – Lecture 2 (Theory)

Human Eye - Color

• Three different types of cones; each type has a special pigment that is sensitive to wavelengths of light in a certain range:– Short (S) corresponds to blue– Medium (M) corresponds to green– Long (L) corresponds to red

• Ratio of L to M to S cones: – approx. 10:5:1

• Almost no S cones in the center of the fovea

400 450 500 550 600 650R

ELA

TIV

E A

BS

OR

BA

NC

E (%

)

W AVELENGTH (nm.)

100

50

440

S

530 560 nm.

M L

ME5286 – Lecture 2 (Theory)

#73

Visual Cortex

ME5286 – Lecture 2 (Theory)

#74

Visual Perception

• Modern view is that visual transformation is a creative process– Vision transforms light stimuli on the retina

into mental constructs of a stable 3D world– Visual perception is a 3D perception of the

world that is invariant to a wide range of changes in illumination, size, shape, and brightness of the image

ME5286 – Lecture 2 (Theory)

Digital Image Formation

ME5286 – Lecture 2 (Theory)

First digitally scanned photograph

• 1957, 176x176 pixels

http://listverse.com/history/top-10-incredible-early-firsts-in-photography/

ME5286 – Lecture 2 (Theory)

Digital cameras• A digital camera replaces

film with a sensor array.

– Each cell in the array is light-sensitive diode that converts photons to electrons

– Two common types• Charge Coupled Device (CCD) • Complementary metal oxide

semiconductor (CMOS)

ME5286 – Lecture 2 (Theory)

What is a digital image?

8 bits/pixel

0

255

ME5286 – Lecture 2 (Theory)

#79

Spatial Sampling

ME5286 – Lecture 2 (Theory)

#80

Quantization

ME5286 – Lecture 2 (Theory)

sceneradiance

(W/sr/m )

∫sensorirradiance

sensorexposure

Lens Shutter

2

∆t

analogvoltages

digitalvalues

pixelvalues

CCD ADC Remapping

Image Acquisition Pipeline

ME5286 – Lecture 2 (Theory)

Digital Camera: Properties• Focus – Shifts the depth that is in focus.

• Focal length – Adjusts the zoom, i.e., wide angle or telephoto

lens.

• Aperture – Adjusts the depth of field and amount of light let into

the sensor.

• Exposure time – How long an image is exposed. The longer an

image is exposed the more light, but could result in motion blur.

• ISO – Adjusts the sensitivity of the “film”. Basically a gain

function for digital cameras. Increasing ISO also increases noise.

ME5286 – Lecture 2 (Theory) 83

Camera exposure• ISO number

– Sensitivity of the film or the sensor – Can go as high as 1,600 and 3,200

• Shutter speed– How fast the shutter is opened and closed

• f/stop– The size of aperture– 1.0 ~ 32

ME5286 – Lecture 2 (Theory)

#84

Illumination

• 1 Lux = 1 Lumen/m2 = 0.093 ft-candles• Office = 100-1,000 Lux• Sunlight = 50,000 – 200,000 Lux• Cloudy day = 1,1000 Lux• Twilight = 1-10 Lux• Full moon = 0.1 – 1 Lux• Night sky = 10-9 – 10-8

ME5286 – Lecture 2 (Theory)

#85

Exposure

ME5286 – Lecture 2 (Theory)

#86

Exposure Time

ME5286 – Lecture 2 (Theory)

#87

Aperture & Shutter control Exposure

ME5286 – Lecture 2 (Theory)

#88

Aperture vs. Shutter

ME5286 – Lecture 2 (Theory)

Long Exposure

10-6 106

10-6 106

Real world

Picture

0 to 255

High dynamic range

ME5286 – Lecture 2 (Theory)

Short Exposure

10-6 106

10-6 106

Real world

Picture

0 to 255

High dynamic range

ME5286 – Lecture 2 (Theory)

Varying Exposure

ME5286 – Lecture 2 (Theory)

Image file formats• Many image formats adhere to the simple model shown below

(line by line, no breaks between lines).• The header contains at least the width and height of the image.• Most headers begin with a signature or “magic number”

(i.e., a short sequence of bytes for identifying the file format)

ME5286 – Lecture 2 (Theory)

Common image file formats

• GIF (Graphic Interchange Format) -• PNG (Portable Network Graphics)• JPEG (Joint Photographic Experts Group)• TIFF (Tagged Image File Format)• PGM (Portable Gray Map)• FITS (Flexible Image Transport System)

ME5286 – Lecture 2 (Theory)

PBM/PGM/PPM format• A popular format for grayscale images (8 bits/pixel)• Closely-related formats are:

– PBM (Portable Bitmap), for binary images (1 bit/pixel)– PPM (Portable Pixelmap), for color images (24 bits/pixel)

» ASCII or binary (raw) storage

ASCI

Binary

ME5286 – Lecture 2 (Theory)

#95

Chromatic Aberration

ME5286 – Lecture 2 (Theory)

#96

Spherical Aberration