The Camera Computational Photography - IDC · 1 1 Computational Photography Yacov Hel-Or and Yossi...

1

1

Computational PhotographyComputational Photography

Yacov Hel-Or

and

Yossi Rubner

2

The CameraThe Camera

• A camera is a device that takes photos of images

• Camera Obscura (Latin = "dark chamber")

19th century camera Sony’s smile recognition camera

3

Camera DevelopmentsCamera Developments

1825 19901913

permanent capturing

(wet plates)

1850

exposuretime and

motion capture(dry plates)

digital cameras

quality and size (35 mm)

1885

portability(film-Kodak)

1933

optics (SLR)

1950

instancy(polaroid)

2000

computationalphotography

1826 - Earliest surviving photograph. This image required an eight-hour exposure.

4

Computational PhotographyComputational Photography

Computational photography refers

broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera.

Steve Mann: The Cyberman

Generated by Foxit PDF Creator © Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

2

5

Goal: Record a richer, multiGoal: Record a richer, multi--layered visual experiencelayered visual experience

1.1. Overcome limitations of todayOvercome limitations of today’’s camerass cameras

2.2. Support better postSupport better post--capture processingcapture processing

3.3. Enables new classes of recording the visual signal Enables new classes of recording the visual signal

4.4. Synthesize Synthesize ““impossibleimpossible”” photosphotos

6

AdministrationAdministration

• Pre-requisites / prior knowledge

• Course Home Page:

– http://www1.idc.ac.il/toky/CompPhoto-09/

– “What’s new”

– Lecture slides and handouts

– Matlab guides

– Homework, grades

• Exercises:

– Programming in Matlab, ~3 Assignments

– Final project

7

Administration (Cont.)Administration (Cont.)

• Matlab software:– Available in PC labs

– Student version

– For next week: Run Matlab “demo” and read Matlab primer until section 13.

• Grading policy: – Final Grade will be based on: Exercises (60%) , Final project

(40%)

– Exercises will be weighted

– Exercises can be submitted in pairs

• Office Hours: by email appointment to [email protected]

8

Project presentation29.01.09

Light Field22.01.09

Single View Modeling15.01.09

Segmentation and Matting08.01.09

Data Driven Synthesis01.01.09

Multi exposure enhancement25.12.08

Multi exposure enhancement18.12.08

Appearance-based registration11.12.08

Blending and Composition04.12.08

Panoramas and feature-based registration27.11.08

Single exposure enhancement20.11.08

Acquisition and camera model13.11.08

Intro and image formation06.11.08

TopicDate

ScheduleSchedule


3

9

ReadingsReadings

• Related papers

• New book: Computational Photography by R. Raskarand J. Tumblin

10

SyllabusSyllabus• Image Formation

Image formation HVS pathwayColor models

• Acquisition and camera modelCamera model + perspective projectionsSensorsNoise models & DistortionsSampling (spatial+temporal) and quantizationCamera parametersCamera Parameters trade-offs.

• Single exposure enhancementWhite BalancingDe-mosaicingDe-noisingDe-blurringGeometrical distortion correction

• Panoramas and feature based registrationImage featuresSIFTPanoramasFeature based registrationPanoramasHomographyRANSAC Image stitching

11

Syllabus Syllabus –– cont.cont.• Blending and Composition

Pyramid blendingOptimal cutSeam CarvingGraph-cutGradient domain editing

• Appearance based registrationSimilarity measures Lucas Kanade optical flowMulti-modal registrationApplications

• Multi exposure enhancement (2 weeks)HDRSuper-resolutionmulti-exposure fusion

• Data Driven SynthesisTexture synthesisVideo textureQuiltingImage analogiesSuper-ResolutionImage Completion

12

Syllabus Syllabus –– cont.cont.• Segmentation and Matting

Segmentation using Graph cut.mean-shiftSpectral clusteringInteractive and semi-automatic Matting

• Single View ModelingCamera CalibrationMeasurements in affine camera3D reconstruction

• Light FieldPlenoptic function and the Lumiograph

• Re-sampling the plenoptic function


4

13

1. Image Formation1. Image Formation

• Taking a picture

• HVS pathway

• Color models

Optic NerveFovea

Vitreous

Optic Disc

Lens

Pupil

Cornea

Ocular Muscle

Retina

Humor

Iris

14

2. Camera Model and Acquisition2. Camera Model and Acquisition

• Perspective projections

• Camera pipeline and parameters

• Sensors

• Sampling and quantization

• Noise models & Distortions

• Camera Parameters trade-offs.

15

3. Single Exposure Enhancement3. Single Exposure Enhancement

• White Balancing

• De-mosaicing

• De-noising

• De-blurring

• Geometrical distortion correction

Difference in white point

16

4. Panoramas and Feature Based Registration4. Panoramas and Feature Based Registration

• Image features

• SIFT

• Feature based registration

• Panoramas

• Homography

• RANSAC

• Image stitching


5

17

5. Blending and Composition5. Blending and Composition

• Pyramid blending

• Gradient domain editing

• Optimal cut

• Graph-cut

18

6. Appearance Based Registration (warping?)6. Appearance Based Registration (warping?)

• Similarity measures

• Lucas Kanade optical flow

• Multi-modal registration

• Applications

19

7. Multi Exposure Enhancement 7. Multi Exposure Enhancement

• HDR

• Super-resolution

• Different-exposures fusion

20

8. Data Driven Synthesis 8. Data Driven Synthesis

• Texture synthesis

• Video texture

• Quilting

• Image analogies

• Super-Resolution

• Image Completion


6

21

9. Segmentation and Matting 9. Segmentation and Matting

• Segmentation using Graph cut.

• mean-shift

• Spectral clustering

• Interactive and semi-automatic Matting

22

10. Single View Modeling10. Single View Modeling

• Camera Calibration

• 3D reconstruction

• Metrology

Flagellation by Pietro della Francesca (1416-92, Italian Renaissance period)Animation by Criminisi et al., ICCV 99

23

11. Light Field11. Light Field

• Plenoptic function and the Lumiograph

• Re-sampling the plenoptic function

24

TodayToday’’s Topic s Topic -- Image FormationImage Formation

• What is an image ?

• What is a color ?


7

25

Computer Vision

Rendering

Image/video Processing

Model3D Object

Geometric Modeling

2D Images

The Visual SciencesThe Visual Sciences

26

What is an Image ?What is an Image ?

• An image is a projection of a 3D scene into a 2D projection plane.

• An image can be defined as a 2 variable function I(x,y) , where for each position (x,y) in the projection plane, I(x,y) defines the light intensity at this point.

27

The Pinhole Camera ModelThe Pinhole Camera Model

• Pinhole model:– Captures pencil of rays – all rays through a single point

– The point is called Center of Projection (COP)

– The image is formed on the Image Plane

– Effective focal length f is distance from COP to Image Plane

Slide by Steve Seitz

COP

Image plane

Focal length

28

Funny things happenFunny things happen……


8

29

Parallel lines arenParallel lines aren’’tt……

30

Lengths canLengths can’’t be trusted...t be trusted...

31

Projection Model (where)Projection Model (where)

• The coordinate system– We will use the pin-hole model as an approximation

– Put the optical center (Center Of Projection) at the origin

– Put the image plane (Projection Plane) in front of the COP

– The camera looks down the negative z axis

Slide by Steve Seitz 32

The Shading Model (what)The Shading Model (what)

Shading Model: Given the illumination incident at a point on a surface, what is reflected?


9

33

ShadingShading Model ParametersModel Parameters

• The factors determining the shading effects are:

– The light source properties:

• Positions, Electromagnetic Spectrum, Shape.

– The surface properties:

• Position, orientation, Reflectance properties.

– The eye (camera) properties:

• Position, orientation, Sensor spectrum sensitivities.

34

Newton’s Experiment, 1665 Cambridge.Discovering the fundamental spectral components of light.

Light and the Visible SpectrumLight and the Visible Spectrum

35

The light SpectrumThe light Spectrum

Electromagnetic Radiation - Spectrum

Gamma X rays Infrared Radar FM TV AMUltra-violet

10-12

10-8

10-4

104

1 108

electricityACShort-

wave

400 nm 500 nm 600 nm 700 nmWavelength in nanometers (nm)

Wavelength in meters (m)

Visible light

36

MonochromatorsMonochromators

Monochromators measure the power or energy at different wavelengths


10

37

The Spectral Power Distribution (SPD) of a light is a

function e(λ) which defines the energy at each wavelength.

Wavelength (λ)

400 500 600 7000

0.5

1

Rela

tive

Pow

er

Light ParametersLight Parameters

38

ExamplesExamples of Spectral Power Distributionsof Spectral Power Distributions

Blue Skylight Tungsten bulb

Red monitor phosphor Monochromatic light

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

39

Specular reflectionmirror like reflection at the surface

Diffuse (lambertian) reflectionreflected randomly between color particlesreflection is equal in all directions

Incident light Specular reflection

Diffuse reflection

normal

Surface ParametersSurface Parameters

40

Different Types of Surfaces


11

41

400 500 600 700

0.2

0.4

0.6

0.8

1

400 500 600 700

0.2

0.4

0.6

0.8

1

400 500 600 700

0.2

0.4

0.6

0.8

1

400 500 600 700

0.2

0.4

0.6

0.8

1

Surface Body Reflectances (albedo)

Yellow Red

Blue Gray

Wavelength (nm)

Spectral Property of Spectral Property of LambertianLambertian SurfacesSurfaces

42

θ

NL

R

V

Ambient reflection: Iamb= K(λ) ea(λ)

Diffuse reflection: Idiff= K(λ) ep(λ) (N⋅L)

Specular reflection: Ispec= Ks(λ)ep (λ) (R⋅V)n

• ep ea - the ambient and point light intensities.

• K , Ks ∈ [0,1] - the surface ambient / diffuse / specular reflectivity.

• N - the surface normal, L - the light direction, V – viewing direction

Surface propertiesLight properties

geometry

43

θ

NL

R

V

Ambient reflection: Iamb= K(λ) ea(λ)

Diffuse reflection: Idiff= K(λ) ep(λ) (N⋅L)

Specular reflection: Ispec= Ks(λ)ep (λ) (R⋅V)n

• ep ea - the ambient and point light intensities.

• K , Ks ∈ [0,1] - the surface ambient / diffuse / specular reflectivity.

• N - the surface normal, L - the light direction, V – viewing direction

44

Diffusesurface

Ambientsurface

Diffuse +

Specular


12

45

I(λ) = Iamb+Idiff+Ispec

• The final illumination equation:

• If several light sources are placed in the scene:

I(λ)= Iamb+Σk (Ikdiff+Ik

spec)

46

Composition of Light SourcesComposition of Light Sources

47

Optic NerveFovea

Vitreous

Optic Disc

Lens

Pupil

Cornea

Ocular Muscle

Retina

Humor

Iris

The Human Visual SystemThe Human Visual System

Ø Cornea - קרנית Ø Pupil - איש ון Ø Iris - קשתית Ø Retina - רשתית

48


13

49

The Visual PathwayThe Visual Pathway

Retina

Optic Nerve

Optic Chiasm

LateralGeniculateNucleus (LGN)

Visual Cortex

50

Eye Eye v.sv.s. Camera. Camera

Yaho Wang’s slides

51

light

rods cones

horizontal

amacrine

bipolar

ganglion

The Human RetinaThe Human Retina

52

• Retina contains 2 types of photo-receptors

– Cones:• Day vision, can perceive color tone

– Rods: • Night vision, perceive brightness only


14

53

Cones:• High illumination levels (Photopic vision)

• Sensitive to color (there are three cone types: L,M,S)

• Produces high-resolution vision

• 6-7 million cone receptors, located primarily in the central

portion of the retina

Wavelength (nm)

Rela

tive s

en

sitiv

ity

Cone Spectral Sensitivity

400 500 600 7000

0.25

0.5

0.75

1ML

SM

A side note:• Humans and some monkeys have three types of cones (trichromatic vision); most other mammals have two types of cones (dichromatic vision).• Marine mammals have one type of cone.• Most birds and fish have four types. •Lacking one or more type of cones result in color blindness.

54

Rods:• Low illumination levels (Scotopic vision).• Highly sensitive (respond to a single photon).• Produces lower-resolution vision• 100 million rods in each eye.• No rods in fovea.

Wavelength (nm)

Rela

tive

sen

sitiv

ity

400 500 600 7000

0.25

0.5

0.75

1

Rod Spectral Sensitivity

55

rodsS - Cones

L/M - Cones

Foveal Periphery photoreceptors

Photoreceptor Distribution Photoreceptor Distribution

56

Cone Receptor Mosaic(Roorda and Williams, 1999)

L-cones M-cones S-cones


15

57

Distribution of rod and cone photoreceptors

Degrees of Visual Angle

Recep

tors

per

squ

are

mm

-60 -40 -20 0 20 40 60

2

6

10

14

18x 10

4

rodscones

Cone’s Distribution:

• L-cones (Red) occur at about ~65% of the cones throughout the retina .

• M-cones (green) occur at about ~30% of the cones.

• S-cones (blue) occur at about ~2-5% of the cones (Why so few?).

fovea58

The Cone ResponsesThe Cone Responses

Assuming Lambertian Surfaces

IlluminantSensors

I(λ) – Fixed, point source illuminantl(λ),m(λ),s(λ) – Cone responsivities

Output

∫= )()( λλ IlL

∫= )()( λλ ImM

∫= )()( λλ IsS

59

Metamer - two lights that appear the same visually. They might have different SPDs(spectral power distributions).

400 500 600 7000

400

800

400 500 600 7000

100

200

Wavelength (nm)

Po

wer

The phosphors of the monitor were set to match the tungsten light.

Tungsten light Monitor emission

60

The The TrichromaticTrichromatic Color TheoryColor Theory

Thomas Young (1773-1829) -A few different retinal receptors operating with different wavelength sensitivities will allow humans to perceivethe number of colors that they do.Suggested 3 receptors.

Helmholtz & Maxwell (1850) -Color matching with 3 primaries.

Trichromatic: “tri”=three “chroma”=colorcolor vision is based on three primaries (i.e., it is 3D).


16

61

Color Matching ExperimentColor Matching Experiment

+ -

+ -

+ -

test match

Primaries

• Given a set of 3 primaries, one can determine for every spectral distribution, the intensity of the guns required to match the color of that spectral distribution.

• The 3 numbers can serve as a color representation.

( ) ( ) ( ) ( )λλλλ bBgGrRT ++≡

R(λ)

G(λ)

B(λ)

T(λ)

62

Color matching experiment for Monochromatic lights

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

Primary Intensities

63

r(λ)

g(λ)b(λ)

400 500 600 700

0

1

2

3

Wavelength (nm)

Prim

ary

Inte

nsi

ty

Stiles & Burch (1959) Color matching functions. Primaries are: 444.4 525.3 and 645.2

Problems: Some perceived colors cannot be generated. This is true for any choice of visible primaries.

64

• Observation - Color matching is linear:

– if (S≡P) then (S+N≡P+N)

– if (S≡P) then (α S≡ α P)

• Outcome 1: Any T(λ) can be matched:

• Outcome 2: CMF can be calculated for any chosen primaries U(λ), V(λ), W(λ):

( ) ( ) ( ) ( ) ( ) ( ) λλλλλλλλλ dbTbdgTgdrTr ∫∫∫ === ;;

=

b

g

r

ccc

ccc

ccc

w

v

u

bwgwrw

bvgvrv

buguru


17

65

• The CIE (Commission Internationale d’Eclairage) defined in 1931 three hypothetical lights X, Y, and Z whose matching functions are positive everywhere:

The CIE Color StandardThe CIE Color Standard

66

TristimulusTristimulus

• Let X, Y, and Z be the tristimulus values.

• A color can be specified by its trichromatic coefficients, defined as

Xx

X Y Z=

+ +

Yy

X Y Z=

+ +

Zz

X Y Z=

+ +

X ratio

Y ratio

Z ratio

Two trichromatic coefficients are enough to specify a color. (x + y + z = 1)

From: Bahadir Gunturk

67

CIE Chromaticity DiagramCIE Chromaticity DiagramInput light spectrum

x

y

From: Bahadir Gunturk68


x

y



18

69


x

y



Boundary

x

y

380nm

700nm


71


Boundary


CIE Chromaticity DiagramCIE Chromaticity Diagram

Light composition



19

73


Light composition

Light composition



• The CIE chromaticity diagram is helpful to determine the range of colors that can be obtained from any given colors in the diagram.

Source: http://hyperphysics.phy-astr.gsu.edu/hbase/vision/visioncon.html#c1

Gamut: The range of colors that can be produced by the given primaries.

http://www.brucelindbloom.com/index.html?Eqn_ChromAdapt.html

75

• The sRGB is a device-independent color space. It was created in

1996 by HP and Microsoft for use on monitors and printers.

• It is the most commonly used color space.

• It is defined by a transformation from the xyz color space.

The The sRGBsRGB Color StandardColor Standard

76

Color matching predicts matches, not appearanceColor matching predicts matches, not appearance


20

77

Color Appearance

78

Color Appearance

79

Color Appearance

80

Color SpacesColor Spaces


21

81

RGB Color Space (additive)RGB Color Space (additive)

• Define colors with (r, g, b) ; amounts of red, green, and blue

82

CMY Color Space (subtractive)CMY Color Space (subtractive)

• Cyan, magenta, and yellow are the complements of red, green, and blue– We can use them as filters to subtract from white

– The space is the same as RGB except the origin is white instead of black

83

HSV color spaceHSV color space

• Hue - the color we see (red, green, purple).

• Saturation - how pure is the color (how far the color from gray ).

• Value (brightness) - how bright is the color.

84

HSV HSV -- a more intuitive color spacea more intuitive color space

Value

Saturation

Hue


22

85

Opponent Color SpaceOpponent Color Space

• Observation: Color bands are highly correlated in high spatial frequencies

∗),( yxh

86

A joint Histogram of rx v.s. gx

Red derivative

Gre

en

de

riva

tive

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

87

A joint Histogram of gx v.s. bx

Green derivative

Blu

e d

erivativ

e

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

88

A joint Histogram of rx v.s. bx

Red derivative

Blu

e d

erivativ

e

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500


23

89 90

Joint histograms of R v.s. G for a low pass images.

91

• Define a new color basis (l,c1,c2):

−

−=

=

211

011

111

2

1 nTwhere

B

G

R

T

c

c

l

l – luminanceC1- red/greenC2 – blue/yellowA joint Histogram of r

x v.s. g

x

Red derivative

Gre

en

de

riva

tive

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

L

c1

l – luminance valueC1 – Red-GreenC2 – Blue-Yellow

92

Comments:

– l channel encodes the color luminance.

– C1 and C2 encodes the chrominance.

– In the chrominance channels high freq. are attenuated.

– It the luminance channel high freq. are maintained.

– The 3 opponent channels are uncorrelated in the high freq.

– Efficient for encoding


24

93

High freq. details Low freq. details Low freq. details

Claim: The HVS’ high spatial sensitivity in the luminance domain and low spatial sensitivity in the chrominance domains is a direct outcome of the statistical properties of color images!

94

Original Image

95

After blurring C1 and C2 bands

96

After blurring l band as well


25

97

Opponent Color SpacesOpponent Color Spaces

• The standard representation used in TV broadcasting

• Backwards compatibility with B/W TV

• Low bit rate is needed in the chrominance channels

• There are various opponent representations:

– YIQ - used for NTSC color TV

– YUV (also called YCbCr) - used for PAL TV and video

• Question: why S cones are sparsely populated?

98

T H E E N DT H E E N D


The Camera Computational Photography - IDC · 1 1 Computational Photography Yacov Hel-Or and Yossi...

Documents

Transcript of The Camera Computational Photography - IDC · 1 1 Computational Photography Yacov Hel-Or and Yossi...