Computational Optical Imaging - Optique Numerique --...

Ivo Ihrke / Winter 2013

Computational Optical Imaging -Optique Numerique

-- Noise, Dynamic Range and Color --

Winter 2013

Ivo Ihrke


Organizational Issues

I received your email addresses

Course announcements will be send via email

Course webpage at http://giana.mmci.uni-saarland.de/

Teaching -> Computational Optical Imaging

http://giana.mmci.uni-saarland.de/


Noise


Sources of noise

Photon shot noise

Dark current shot noise

Fixed pattern noise

Readout noise

…

[Janesick97]


Noise Sources

[Reibel2003]

read noisephoton shot noise

dark current noise


Photon shot noise

Variance in number of photons that are counted

they arrive in a Poisson random process

Standard deviation is square root of signal

relative noise decreases with signal

Fundamental limit on photodetector precision!

Can be reduced by averaging multiple exposures.


Fixed pattern noise

Caused by variations in component values

Big problem for CMOS sensors

An amp at every pixel, and one for every column

Gain variation (proportional to signal PRNU)

Bias variation (independent of signal – dark current)

Can be partially canceled by correlated double sampling (CDS)

CCD’s transfer all charge to a single output amplifier


Dark current

Things besides photons can knock electrons loose in the silicon. These are collected, too.

Highly temperature dependent

doubles every 5-8 degrees C

May be reduced by cooling the sensor.

Proportional to exposure time

Limits exposure durations—eventually, the dark current fills your well capacity.


Dark Current Noise

Dark current has fixed pattern noise.

Dark current varies because of irregularities in the silicon.

Dark current has shot noise, too!

dominates in dark areas for long exposures

Mean dark current may be subtracted

but subtracting frames increases shot noise

subtract the average dark current

Dark current is why astronomers chill their image sensors.


Peltier Cooling of CMOS chip

[Gary Honis]


Thermal Noise

Generated by thermally induced motion of electrons in resistive regions (resistors, transistor channels in strong inversion…)

What does it mean?

Independent of the signal.

Zero mean, white (flat, wide bandwidth)

Another problem for CMOS, not CCD imagers

Dominates at low signal levels

Can limit dynamic range


Dark Current Noise – Removal

cooling the chip

noise removal techniques to separate image data from noise

e.g. median filtering

25 s exposure time

uncooled cooled


Noise, noise, noise…

Reset (kTC) noise

thermal noise when “resetting” the CMOS photodetector—a big deal, actually.

can be corrected with CDS

Amplifier noise

thermal

spatially non-uniform

1/f noise

non-linearities

Quantization noise

“truncate” analog value to N bits


Analog/Digital Conversion


Correlated Double Sampling

reduce noise by comparing against a reference charge


Combined Noise Model [Reibel2003]

- fixed pattern noise

- readout noise

- thermal dark current shot noise

- photon shot noise

- photo response non-uniformity

- non-linear effects

NLPRNUPSNPSNDSNRFPNN CTOT

2222222

FPN2

R2

DSN2

PSN2

PRNU2

NLC


Combined Noise Model [Reibel2003]

- fixed pattern noise (can be calibrated)

- readout noise (CDS)

- thermal dark current shot noise (cooling)

- photon shot noise (multiple exposures)

- photo response non-uniformity (per-pixel gain)

- non-linear effects (can also be calibrated for)

NLPRNUPSNPSNDSNRFPNN CTOT

2222222

FPN2

R2

DSN2

PSN2

PRNU2

NLC


Noise Distribution

[Reibel2003]

ADU = Analog-Digital Unit, e.g. 1 ADU = 0.5 e-


Digital Images

Images are now numbers (corrupted by noise)


Digital

Sensor noise

Dynamic Range

Tone Curve

Recording “Medium”

Monochromatic

Optical

Distortions

Aberrations

Digital Images - Limitations


Dynamic Range


Dynamic Range

max output swing

noise in the dark

Saturation level – dark current

Dark shot noise + readout noise=

“noise in the dark” is random noise sources that

cannot be corrected with circuit tricks

Photon shot noise and read noise

dr =


Dynamic Range of Standard Sensors

ww

w.d

xom

ark

.com

13.5 EVs or f-stops = contrast 11,000:1 = color negative


Dependency of Dynamic Range on ISO

“Unity gain” is where 1 digital unit (ADU) equals 1 electron (e-)

This happens at different ISO settings for different sensors

Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise)

“digital gain” multiplies the digital values

All gain settings beyond unity gain reducedynamic range


What is High Dynamic Range (HDR)?

http://en.wikipedia.org/wiki/High-dynamic-range_imaging


HDR Acquisition – Exposure Brackets

[Debevec & Malik 97]

Radiance Map Tonemapped HDR Image

Exposure Sequence


Shutter Speed

F/stop (aperture)

Neutral Density (ND) Filters

Gain / ISO / Film Speed

(DOF)

(noise)

Ways to vary the exposure


Combining the image

dttxlxI ),()(

• scene constant over exposure time (or ND-filter)

radiance

),()( xltxI

• assumes linear response (radiometric calibration!)

have several measurements with different t


Combining the image

• introduce a weighting function for the pixels:

• centered at the sensor mean value,

e.g. Gaussian (image data in [0,1])

• compute final image as

2

2

2.0

)5.0)((

))((

xI

exIw

i

i

i

iii

finalxIw

txIxIw

xI))((

/)())((

)(


• Important for many vision and graphics algorithms

EIg :1

• Use a color chart with precisely known reflectances.

• Use more camera exposures to fill up the curve.

• Method assumes constant lighting on all patches

• Works best when source is far away (example sunlight).

• Inverse exists (g is monotonic and smooth for all cameras)

Irradiance = const * Reflectance

Pix

el V

alu

es

3.1%9.0%19.8%36.2%59.1%90%

0

255

0 1

g

?

?1g

Radiometric Calibration


Response Curve - Practice

Measurement: ColorCalibrationToolbox http://giana.mmci.uni-saarland.de/website-template/software.php

Example: 29 exposures of Gretag-Macbeth color checker

(uses EXIF info - exiftool)

http://giana.mmci.uni-saarland.de/website-template/software.php


Color Calibration Toolbox

Zoom-in



Mark the patch rectangle



Make sure the patches are properly extracted



Verify response curve – the example is for jpg on the Canon 5D mark II

Make sure the samples are fit well

Response curves

(R,G,B)

Inverse response curves

samples



Check HDR image



How is the curve estimated ?

Variant of Mitsunaga and Nayar, “Radiometric Self Calibration”, CVPR 1999

Polynomial fit to data samples

Variations:

─ enforce monotonicity (derivative > 0)

Prevents “wiggling”

─ enforce passing of curve through (0,0) and (1,1)

map range to range

─ Perform a weighted fit

accounts for sample non-uniformity


Response Curve – Take Home Points

Usually linear for RAW images

Don’t rely on it – verify

Usually non-linear for jpg or other compressed/processed formats

Estimation from random images may be unstable

Use well defined target (color checker)

Prefer continuous-curve algorithms, especially for high bit depths


Applications - HDR Display

• 47” TFT LCD, LED backlight

• aspect ratio 16:9

• resolution 1920 x 1080

• contrast >1,000,000:1

• brightness 4,000 cd/m2

Images courtesy Dolby


Applications - Image Based Lighting

Slid

es b

y P

aul D

ebe

vec


EDR and HDR Cameras

Grass Valley Viper

(10 bits log)

Panavision Genesis

(10 bits log)

Spheron

- Scanning

- 26 f-stops

Extended range High dynamic range


Super CCD (Fuji)

octagonal grid

elements with differentsensitivity

extended DR

better in low light

Used in consumer products (Finepix)

http://www.henner.info/super_ccd.htm


HDRC – Log Encoding

CMOS – pixel amplifier output is logarithmic

U - logarithmic


Per-Pixel Exposure Time Controlwww.pixim.com

no pixim with pixim no pixim with pixim


Adaptive Dynamic Range Imaging

[Nayar and Branzoi 03]


Programmable Imaging

[Nayar

& B

ran

zo

i 0

6]

un

mo

du

late

dm

od

ula

ted

mo

du

latio

n s

ign

al


Textbook

• HDR image / video encoding

• capture, display, tone reproduction

• visible difference predictors

• image based lighting, etc.


Color


Sensing color

Eye has 3 types of color receptors

Therefore we need 3 different spectral sensitivities

sourc

e: K

odak

KA

F-5

101ce

dat

a sh

eet


Ways to sense color

Field-sequential color

simplest to implement

only still scenes

Proudkin-Gorskii, 1911

(Library of Congress exhibition)


Examples - Prokudin-Gorskij

Self-portrait 1915

http://upload.wikimedia.org/wikipedia/commons/b/b2/Sergei-Prokudin-Gorski-Larg.jpg

http://upload.wikimedia.org/wikipedia/commons/b/b2/Sergei-Prokudin-Gorski-Larg.jpg


Photograph 1910, Emir of Bukhara , Prokudin-Gorskii

Examples - Prokudin-Gorskij


Examples - Lew Tolstoy

1910 photograph, Sergey Prokudin-Gorskii1887 painting, Ilya Repin

http://upload.wikimedia.org/wikipedia/commons/c/c6/L.N.Tolstoy_Prokudin-Gorsky.jpg

http://upload.wikimedia.org/wikipedia/commons/c/c6/L.N.Tolstoy_Prokudin-Gorsky.jpg

http://upload.wikimedia.org/wikipedia/commons/b/bb/Ilya_Efimovich_Repin_(1844-1930)_-_Portrait_of_Leo_Tolstoy_(1887).jpg

http://upload.wikimedia.org/wikipedia/commons/b/bb/Ilya_Efimovich_Repin_(1844-1930)_-_Portrait_of_Leo_Tolstoy_(1887).jpg


Color Wheel

one color channel is captured at one shot

3 times the acquisition time

static images only


Liquid Crystal Tunable Filter (LCTF)

Computer controllable spectral filter

VariSpec LCTF


Spectral Response of Lyot Stage

)(cos)( 2

max

t

T

T

Relative

Transmissivity:

Waveplate

parameters

(birefringence,

thickness)

Electrically Tunable Birefringence

is implemented by liquid crystal

in Lyot stage


Spectral Response of 7 Differently tuned Lyot Stages

using several stages in sequence:

product = result


VariSpec spectral curves


Ways to sense color – 3-Chip Camera

dichroic mirrors divide light into wavelength bands

does not remove light: excellent quality but expensive

interacts with lens design

problem with polarization

image: Theuwissen


Foveon Technology

3 layers capture RGB at the same location

takes advantage of silicon’s wavelength selectivity

light decays at different rates for different wavelengths

multilayer CMOS sensor gets3 different spectral sensitivities

don’t get to choose the curves


Ways to sense color

Color filter array

paint each sensor with an individual filter

requires just one chip but loses some spatial resolution

“demosaicing” requires tricky image processing

G R

B G

C M

Y G

primary secondary


Demosaicing

bilinear interpolation

sampling theory

edge-directed/pattern-based interpolation

correlation-based


Demosaicing

Original image Bilinear interpolationRon Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html


Demosaicing

Ron Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html

Bilinear interpolation Edge-weighted interpolation


Bilinear Interpolation

perform interpolation for each color channel separately

G R

B G

= + +



G R

B G

= + +

4

3432141223

RRRRR



G R

B G

= + +

4

3432141223

RRRRR

2

343233

RRR



set all non-measured values to zero then convolve

G R

B G

= + +

4/

121

242

121

,

BRF 4/

010

141

010

GF


Problem: Aliasing

[Alleysson & Suesstrunk05]


Fourier Space

/4


Excessive Blurring


Grid Effect


Bleak colors


Color Aliasing


optimize r1 and r2 to gain best separation

Low-pass filter luminance

High-pass filter chrominance (orthogonal filter)

Demultiplex chrominance

Interpolate opponent colors

Add luminance and interpolated colors

[Alleysson & Suesstrunk05]


Gradient-based (dcraw)

1.Calculate gradients in 5x5 region

2.Select subset of gradients (below threshold)

3.Average color differences in the determined regions

[Chuang et al. 99]


Gradients

“Gradient” S = |1323|3/|818| RRGG

2/|717|2/|919| BBBB

3/|1222|3/|1424| GGGG


Regions

selection: “gradient” < 1.5*Min+0.5*(Max-Min)

e.g. {S,W,NE,SE}

S: R = (R13+R23)/2, G = G18, B = (B17+B19)/2

NE: R = (R13+R5)/2, G = (G4+G8+G10+G14)/4, B = B9

…

S


Average

Rsum = (Rs + Rw + Rne + Rse)/4

Gsum = (Gs + Gw + Gne + Gse)/4

Bsum = (Bs + Bw + Bne + Bse)/4

average of color differences

G13 = R13 + (Gsum-Rsum); B13 = R13 + (Bsum-Rsum)


Demosaicing – Take-home-points

2/3 of your image are just made up!

color resolution is less than image resolution

be careful with spiky BRDFs

combining multiple video frames might help

98% of all demosaicing algorithms are ad-hoc

─ smoothing based on constant hue assumption afterwards


White Balance

daylight

flashflourescent

tungsten

capture the spectral

characteristics of

the light source to

assure correct

color reproduction


White Balance

Human perception adapts to illumination condition

Practice: division of RGB values


White Balance

Camera built-in function

derive scale from white point

sun

incandescent

tungsten

infraredred green blue

ultra violet

wavelength


White Balance

Camera built-in function

derive scale from white point

infraredred green blue

ultra violet

wavelength


White Balance

Human perception adapts to illumination condition

Practice: division of RGB values

Theory: achieve a “neutral” spectrum

(only works for broad band sources and

broad band reflectance)

Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma


Spectrum to Image

do not have spectral display

not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision)

cones 6-7 million

rods ~120 million

rods more sensitive


Color Vision

color vision by cones

significant overlap of the response functions

L = long

M = mid

S = short


Color Vision

L ~63%, M ~31%, S ~6% of cones

eye least sensitive to blue, most sensitive to yellowish-green

spectral region outside of support of the response functions cannot be perceived


Spectral response of human eye

reproducing color is tricky

color matching experiments

─ use light source with known spectral distribution

(i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter

─ additionally, use monochromatic sources

@ 700,546,435 nm

─ let human observers adjust apparent brightness of one of the sources to match the other

Color matching functions


Color Spaces

─ RGB matching functions

negative !


XYZ space

The CIE (1931) standard observer


How to compute a tristimulus image from a spectral representation ?

We have to integrate with the spectrum with the appropriate color matching function

dxlfxIXX ),(ˆ)()(

dxlfxIYY ),(ˆ)()(

dxlfxIZZ ),(ˆ)()(


Now to RGB

convert XYZ to RGB


Horseshoe Diagram


White Point for Different Color Temperatures

Planckian

Locus:

- convert black body

temperature to XYZ and

put intohorseshoe diagram

L_\lambda = spectral radiance [W/m^2/m]

lambda = wavelength [m]

h = Planck’s constant [J.s]

k = Boltzmann constant [J/K]

c = speed of light [m/s]

T = temperature of black body [K]


Display Gamut

white point


Bibliography

Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical

Engineering Press, Bellingham, Washington, 1998.

Theuwissen, A. Solid-State Imaging with Charge-Coupled Devices. Kluwer Academic Publishers, Boston, 1995.

Curless, CSE558 lecture notes (UW, Spring 01).

El Gamal et al., EE392b lecture notes (Spring 01).

Several Kodak Application Notes at http://www.kodak.com/global/en/digital/ccd/publications/a

pplicationNotes.jhtml

Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003

http://www.kodak.com/global/en/digital/ccd/publications/applicationNotes.jhtml


Bibliography

D. Alleysson, S. Suesstrunk: Linear Demosaicinginspired by the Human Visual System, IEEE Trans. on Image Processing, 14(4), 2005.

B. K. Gunturk, Y. Altunbasak, R. M. Mersereau: Color Plane Interpolation Using Alternating Projections, IEEE Trans. on Image Processing, 11(9), 2002.

E. Chang, S. Cheung, D.Y. Pan: Color filter array recovery using a threshold-based variable number of gradients. Proc. SPIE, vol. 3650, pp. 36-43, 1999.

http://www-ise.stanford.edu/~tingchen/algodep/vargra.html

http://www-ise.stanford.edu/~tingchen/algodep/vargra.html


Bibliography

Y. Takahashi, K. Hiraki, H. Kikuchi, S. Muaramtsu: Color Demosaicing Using Asymmetric Directional Interpolation and Hue Vector Smoothing, IEICE 20th Workshop on Circuits and Systems, 2008.

R. Kimmel, Demosaicing: Image Reconstruction from Color CCD Samples, IEEE Trans. on Image Processing. Vol. 8, No. 9, 1999.

Boris Ajdin, Matthias B. Hullin, Christian Fuchs, Hans-Peter Seidel, Hendrik P. A. Lensch: Demosaicing by Smoothing along 1D Features. Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.

http://www.mpi-inf.mpg.de/~lensch/Papers/CVPR08-Ajdin-demosaicing.pdf


(ICC –international color consortium)

color management system

capture the properties of all devices

─ camera and lighting

─ monitor settings

─ output properties

common interchange space

sRGB standard as a definition of RGB

ICC Profiles

input device

(e.g. camera)

input profile

profile

connection

space

output device

(e.g. printer)

output profile

display

device

(e.g.

monitor)

monitor

profile


profile connection spaces

─ CIELAB (perceptual linear)

─ linear CIEXYZ color space

can be used to create an high dynamic range image in the profile connection space

allows for a color calibrated workflow

ICC Profiles and HDR Image Generation

input device

(e.g. camera)

input profile

profile

connection

space

output device

(e.g. printer)

output profile

...

Computational Optical Imaging - Optique Numerique --...

Documents

Transcript of Computational Optical Imaging - Optique Numerique --...