image processing - cs.nyu.edu

3/5/2008

1

Image Processing & High Dynamic Range Displayg y g p y

Admin stuff

• Start homework early!!!!

• Will have homework 1 graded by Wednesday

• Come and see me about projects!!!!

Overview of today• Linear filtering

– Blurring

– Sharpening

– Edge detection

– Wiener denoisingWiener denoising

• Non‐linear filtering– Median filter

– Bilateral filter

– Cross‐bilateral filter

• Application ‐ High dynamic range


– Blurring

– Sharpening

– Edge detection






Linear filtering

• Discrete convolution in spatial domain– Have interpretation in frequency domain

– Might use freq. domain for speed

• Want intensity to stay same, so

• Integer arithmetic can improve speed

Inew(x, y) =Pw/2

i=−w/2Ph/2

j=−h/2 f(i, j)Iold(x− i, y − j)

Pw/2i=−w/2

Ph/2j=−h/2 f(i, j) = 1

Filter Kernel

BlurringBlurring

Used for softening appearance

Convolve with gaussian filterSame as mult. by gaussian in freq. domain, so reduces high-frequency contentGreater the spatial width, smaller the Fourier width, more blurring occurs and vice versa

How to find blurring filter?

Slide credit: Ravi Ramamoorthi

3/5/2008

2

Blurring FilterBlurring Filter

In general, for symmetry f(u,v) = f(u) f(v)You might want to have some fun with asymmetric filters

We will use a Gaussian blur Blur width sigma depends on kernel size n (3,5,7,11,13,19)

Spatial Frequency

2

2

1( ) exp ( / 2) / 222

uf u floor nσσπσ

⎡ ⎤−= =⎢ ⎥

⎣ ⎦Slide credit: Ravi Ramamoorthi

Discrete Filtering, NormalizationDiscrete Filtering, Normalization

Gaussian is infiniteIn practice, finite filter of size n (much less energy beyond 2 sigma or 3 sigma). Must renormalize so entries add up to 1

Si l i l hSimple practical approachTake smallest values as 1 to scale others, round to integersNormalize. E.g. for n = 3, sigma = ½

( )2 2

2 22 2

1 2( , ) exp exp 22 2

u vf u v u vπσ σ π

⎡ ⎤+ ⎡ ⎤= − = − +⎢ ⎥ ⎣ ⎦⎣ ⎦0.012 0.09 0.012 1 7 1

10.09 0.64 0.09 7 54 786

0.012 0.09 0.012 1 7 1

⎛ ⎞ ⎛ ⎞⎜ ⎟ ⎜ ⎟≈ ≈⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠Slide credit: Ravi Ramamoorthi

BlurringBlurring


BlurringBlurring

BlurringBlurring BlurringBlurring

3/5/2008

3

BlurringBlurring


Sharpening Sharpening

2.0

0 33

original

0 0

0.33

Sharpened original

Slide credit: Bill Freeman

Sharpening exampleSharpening example

cien

t

811.2

1.7

8

coef

fi

-0.3original Sharpened

(differences areaccentuated; constant

areas are left untouched).

-0.25


Sharpening FilterSharpening Filter

Unlike blur, want to accentuate high frequencies

Take differences with nearby pixels (rather than avg)

1 2 1⎛ ⎞1 2 11( , ) 2 19 27

1 2 1f x y

− − −⎛ ⎞⎜ ⎟= − −⎜ ⎟⎜ ⎟− − −⎝ ⎠

SharpeningSharpening

before after


Edge DetectionEdge Detection

3/5/2008

4

Edge DetectionEdge Detection Edge DetectionEdge Detection

Complicated topic: subject of many PhD theses

Here, we present one approach (Sobel edge detector)

Step 1: Convolution with gradient (Sobel) filterEd h i di t lEdges occur where image gradients are largeSeparately for horizontal and vertical directions

Step 2: Magnitude of gradientNorm of horizontal and vertical gradients

Step 3: ThresholdingThreshold to detect edges


DetailsDetails

Step 1: Convolution with gradient (Sobel) filterEdges occur where image gradients are largeSeparately for horizontal and vertical directions

1 0 1 1 2 1( ) 2 0 2 ( ) 0 0 0f f

−⎛ ⎞ ⎛ ⎞⎜ ⎟ ⎜ ⎟

Step 2: Magnitude of gradientNorm of horizontal and vertical gradients

Step 3: Thresholding

( , ) 2 0 2 ( , ) 0 0 01 0 1 1 2 1

horiz vertf x y f x y⎜ ⎟ ⎜ ⎟= − =⎜ ⎟ ⎜ ⎟⎜ ⎟ ⎜ ⎟− − − −⎝ ⎠ ⎝ ⎠

22x yG G G= +


Wiener denoising derivation

• See http://www.cs.dartmouth.edu/farid/tutorials/fip.pdf

• Pages 57 59

Wiener denoising

From J.Portilla ICIP’01

White Gaussian noise power(assumed to be known)

3/5/2008

5


– Blurring

– Sharpening

– Edge detection






Median filter

Replace each pixel by the median over N pixels (5 pixels, for these examples). Generalizes to “rank order” filters.

In: Out: Spike noise is removed

Median([1 7 1 5 1]) = 1Mean([1 7 1 5 1]) = 2.8

5‐pixel neighborhood

In: Out:

Monotonic edges remain unchanged

Median filtering resultsBest for salt and pepper noise

http://homepages.inf.ed.ac.uk/rbf/HIPR2/mean.htm#guidelines

A Gentle Introductionto Bilateral Filteringand its Applications

A Gentle Introductionto Bilateral Filteringand its Applications

“Fixing the Gaussian Blur”: the Bilateral Filter

Sylvain Paris – MIT CSAIL

Fredo- Durand – MIT CSAIL

Blur Comes from Averaging across EdgesBlur Comes from Averaging across Edges

*input output

*

*Same Gaussian kernel everywhere.

Bilateral FilterNo Averaging across EdgesBilateral FilterNo Averaging across Edges

*input output

[Aurich 95, Smith 97, Tomasi 98]

*

*The kernel shape depends on the image content.

3/5/2008

6

not new newnew

Bilateral Filter Definition:an Additional Edge TermBilateral Filter Definition:an Additional Edge Term

( ) ( )∑ −−=S

IIIGGW

IBF qqpp qp ||||||1][rs σσ

Same idea: weighted average of pixels.

space weight range weight

I

normalizationfactor

∈SW qp

Illustration a 1D ImageIllustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot• Better visualized as a plot

pixelintensity

pixel position

space

Gaussian Blur and Bilateral FilterGaussian Blur and Bilateral FilterGaussian blur

space

p

q ( )∑∈

−=S

IGIGBq

qp qp ||||][ σ

space rangenormalization

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

Bilateral filter[Aurich 95, Smith 97, Tomasi 98]

p

space

range

p

q

Bilateral Filter on a Height FieldBilateral Filter on a Height Field

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

output input

reproducedfrom [Durand 02]

Space and Range ParametersSpace and Range Parameters

• space σ : spatial extent of the kernel size of

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

space σs : spatial extent of the kernel, size of the considered neighborhood.

• range σr : “minimum” amplitude of an edge

Influence of PixelsInfluence of PixelsOnly pixels close in space and in range are considered.

space

range

pp

range

3/5/2008

7

σs = 2

σr = 0.1 σr = 0.25σr = ∞

(Gaussian blur)

input

Exploring the Parameter SpaceExploring the Parameter Space

σs = 6

σs = 18

σs = 2

σr = 0.1 σr = 0.25σr = ∞

(Gaussian blur)

input

Varying the Range ParameterVarying the Range Parameter

σs = 6

σs = 18

input σr = 0.1

σr = 0.25 σr = ∞(Gaussian blur)

3/5/2008

8

σs = 2

σr = 0.1 σr = 0.25σr = ∞

(Gaussian blur)

input

Varying the Space ParameterVarying the Space Parameter

σs = 6

σs = 18

input

σs = 2 σs = 6

σs = 18How to Set the ParametersHow to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude– e.g., mean or median of image gradients

• independent of resolution and exposure

3/5/2008

9

Bilateral Filter Crosses Thin LinesBilateral Filter Crosses Thin Lines• Bilateral filter averages across

features thinner than ~2σs

• Desirable for smoothing: more pixels = more robust• Different from diffusion that stops at thin lines

close-up kernel

Bilateral Filtering Color ImagesBilateral Filtering Color Images

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

For gray-level images intensity difference

scalar

input

output

( ) ( )∑∈

−−=S

GGW

IBFq

qqpp

p CCCqp ||||||||1][rs σσ

For color images color difference

3D vector (RGB, Lab)

Hard to ComputeHard to Compute

• Nonlinear

• Complex, spatially varying kernels– Cannot be precomputed, no FFT…

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

• Brute-force implementation is slow > 10min

Basic denoisingBasic denoising

Noisy input Bilateral filter 7x7 window

Bilateral filter


Median 3x3 Bilateral filter


Median 5x5

3/5/2008

10


Bilateral filter Bilateral filter – lower sigma Bilateral filter


Bilateral filter – higher sigma

DenoisingDenoising

• Small spatial sigma (e.g. 7x7 window)

• Adapt range sigma to noise level

• Maybe not best denoising method, but best simplicity/quality tradeoff– No need for acceleration (small kernel)

– But the denoising feature in e.g. Photoshop is better

Ordinary Bilateral FilterOrdinary Bilateral Filter

Bilateral two kinds of weights, one image A :

( ) ( )∑∈

−−=S

AAAGGW

ABFq

qqpp

p qp ||||||1][rs σσ

cc

ss

Image A:

DomainDomain

RangeRangef(x)f(x)

xx

‘Joint’ or ‘Cross’ Bilateral Filter‘Joint’ or ‘Cross’ Bilateral Filter

NEW: two kinds of weights, two images

( ) ( )∑∈

−−=S

ABBGGW

ABFq

qqpp

p qp ||||||1][rs σσ

cc

ss

A: Noisy, dim(ambient image)

cc

ss

B: Clean,strong (Flash image)

Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)(too dim for a good quick photo)

Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)(too dim for a good quick photo)

No-flash

3/5/2008

11

Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)(flash: simple light, ALMOST no shadows)

Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)(flash: simple light, ALMOST no shadows)

MERGE BEST OF BOTH: apply‘Cross Bilateral’ or ‘Joint Bilateral’

MERGE BEST OF BOTH: apply‘Cross Bilateral’ or ‘Joint Bilateral’

(it really is much better!)(it really is much better!) Overview of today• Linear filtering

– Blurring

– Sharpening

– Edge detection






Real world dynamic rangeReal world dynamic range

• Eye can adapt from ~ 10-6 to 106 cd/m2

• Often 1 : 10,000 in a scene

10-6 10610 10Real world

High dynamic range

Picture dynamic rangePicture dynamic range

• Typically 1: 20 or 1:50– Black is ~ 50x darker than white

10-6 10610 10

10-6 106

Real world

Picture

Low contrast

3/5/2008

12

Multiple exposure photographyMultiple exposure photography

• Merge multiple exposure to cover full range10-6 106

Real worldHigh dynamic range

• We obtain one single image with floats per pixel– But we still can’t display it

HDRMerge

HDR image using multiple exposureHDR image using multiple exposure

• Given N photos at different exposure

• Recover a HDR color for each pixel

If we know the response curveIf we know the response curve

• Just look up the inverse of the response curve

• But how do we get the curve?

Pixel value

scene value

Calibrating the response curveCalibrating the response curve

• Two basic solutions– Vary scene luminance and see pixel values

• Assumes we control and know scene luminance

– Vary exposure and see pixel value for one scene luminanceluminance

• But note that we can usually not vary exposure more finely than by 1/3 stop

• Best of both: – Vary exposure

– Exploit the large number of pixels

• 3• 3• 3• 3

• 1• 1• 1• 1

• 2• 2• 2• 2

• 3• 3• 3• 3

• 1• 1• 1• 1

• 2• 2• 2• 2

• 3• 3• 3• 3

• 1• 1• 1• 1

• 2• 2• 2• 2

• 3• 3• 3• 3

• 1• 1• 1• 1

• 2• 2• 2• 2

Image seriesImage seriesImage seriesImage series

• 3• 3• 3• 3

• 1• 1• 1• 1

• 2• 2• 2• 2

The AlgorithmThe Algorithm

3 3 3 3

ΔΔt =t =1/100 sec1/100 sec

3 3 3 3

ΔΔt =t =1 sec1 sec

3 3 3 3

ΔΔt =t =1/1000 sec1/1000 sec

3 3 3 3

ΔΔt =t =10 sec10 sec

3 3 3 3

ΔΔt =t =1/10 sec1/10 sec

Exposure = Radiance × Δtlog Exposure = log Radiance + log

Δt

Pixel Value Z = f(Exposure)

Slide adapted from Alyosha Efros who borrowed it from Paul DebeΔ t don't really correspond to pictures. Oh well.

Assuming unit radiancefor each pixel

After adjusting radiances to obtain a smooth response curve

3333

Response curve• Exposure is unknown, fit to find a smooth curve

log Exposure

Pixe

l val

ue

3333

1111

2222

log Exposure

Pixe

l val

ue

Slide stolen from Alyosha Efros who stole it from Paul Debevec

3/5/2008

13

Reconstructed radiance map

Slide stolen from Fredo Durand who stole it from Alyosha Efros who stole it from Paul Debevec

Problem: Contrast reductionProblem: Contrast reduction

• Match limited contrast of the medium

• Preserve details

10-6 106High dynamic range10 10

10-6 106

Real world

Picture

Low contrast

High dynamic range

Tone mappingTone mapping

• Input: high-dynamic-range image– (floating point per pixel)

Naïve techniqueNaïve technique

• Scene has 1:10,000 contrast, display has 1:100

• Simplest contrast reduction?

Naïve: Gamma compressionNaïve: Gamma compression

• X −> Xγ (where γ=0.5 in our case)

• But… colors are washed-out. Why?Input Gamma

Gamma compression on intensityGamma compression on intensity

• Colors are OK, but details (intensity high-frequency) are blurred

Gamma on intensityIntensity

Color

3/5/2008

14

Oppenheim 1968, Chiu et al. 1993Oppenheim 1968, Chiu et al. 1993

• Reduce contrast of low-frequencies (log domain)

• Keep high frequenciesReduce low frequencyLow-freq.

High-freq.

Color

The halo nightmareThe halo nightmare

• For strong edges

• Because they contain high frequencyReduce low frequencyLow-freq.

High-freq.

Color

Bilateral filtering to the rescueBilateral filtering to the rescue

• Large scale = bilateral (log intensity)

• Detail = residualOutputLarge-scale

[Durand & Dorsey 2002]

Detail

Color

Contrast reductionContrast reductionInput HDR image

Contrast too high!

Contrast reductionContrast reduction

Intensity

Input HDR image

Color


Intensity Large scale

Input HDR image

Color

Bilateral Filter (in log domain!)Spatial sigma: 2% image sizeRange sigma: 0.4 (in log 10)

3/5/2008

15


Intensity Large scale

Input HDR image

Detail

Color

Bilateral Filter

Detail = log intensity –large scale(residual)


Intensity Large scaleReduce

Large scale

Input HDR image

Detail

Color

Bilateral Filter

contrast



Large scale

Input HDR image

Detail

Color

Bilateral Filter

contrast

DetailPreserve!



Large scale

OutputInput HDR image

Detail

Color

Bilateral Filter

contrast

Detail

Color

Preserve!

Contrast reduction in log domainContrast reduction in log domain

• Set target large-scale contrast (e.g. log10 10)– In linear output, we want 1:10 contrast for large scale

• Compute range of input large scale layer: – largeRange = max(inLogLarge) – min (inLogLarge)

• Scale factor k= log10 (10) / largeRange

• Normalize so that the biggest value is 0 in log

outLog= inLogDetail + inLogLarge * k – max(inLogLarge)

image processing - cs.nyu.edu

Documents

Transcript of image processing - cs.nyu.edu