Noise Estimation from a Single Image

Ce Liu William T. Freeman

Richard Szeliski Sing Bing Kang

Parameter Tweaking in Computer Vision

Computer vision algorithms suffer from hand tuning parameters for particular images or image sequences

We want vision algorithms that behave properly under varying lighting conditions, blur levels and noise levels

Our work is one step in that direction

Given an image, estimate the noise level

Modify vision algorithms to be independent of noise

Image Noise Is Important in Vision

In image denoising the noise is assumed to be known as Additive Gaussian White Noise (AWGN)

However, in real applications the noise is unknown and non-additive

Many other computer vision algorithms also explicitly or implicitly assume the type and level of image noise

Hard to make vision algorithms fully automatic without knowing noise

Noise Level Function (NLF)

The standard deviation of noise is a function of image brightness I

Measurable by fixing the camera and taking multiple shots of a static scene

For each pixel: Mean: I Standard deviation:

NLF depends on camera, ISO, shutter speed, aperture

Our goal is to estimate NLF from a single image How to estimate noise without separating noise and signal?

I

An Example Image

Piecewise Smooth Image Prior

Patch

= +

Signal Residual

For each RGB channel:

Brightness mean I

Standard deviation

Red Green Blue

0 0.5 1

0.2

0.1

0 I

0 0.5 1

0.2

0.1

0 I

0 0.5 1

0.2

0.1

0 I

Brightness

Sta

nd

ard

d

evi

ati

on

Affine model

Piecewise Smooth Image Prior

Patch

= +

Signal Residual

Red Green Blue

0 0.5 1

0.2

0.1

0 I

0 0.5 1

0.2

0.1

0 I

0 0.5 1

0.2

0.1

0 I

Brightness

Sta

nd

ard

d

evi

ati

on

Piecewise Smooth Image Prior

Patch

= +

Signal Residual

Red Green Blue

0 0.5 1

0.2

0.1

0 I

0 0.5 1

0.2

0.1

0 I

0 0.5 1

0.2

0.1

0 I

Brightness

Sta

nd

ard

d

evi

ati

on

Segmentation-based Approach

Observed image

Segmentation-based Approach

Over-segmentation

Segmentation-based Approach

Signal

Segmentation-based Approach

Residual=

noise + unmodelled

image variation

Estimate NLFs

Assume brightness mean I is accurate estimate

Standard deviation is an over-estimate: (may contain signal)

The lower envelope is the upper bound of NLF

III

Brightness

Resi

du

al st

d.

dev.

Issues

Should the curve be strictly and tightly below the points?

III

Brightness

Resi

du

al st

d.

dev.

Issues

Should the curve be strictly and tightly below the points?

How to handle the missing data?

III

Brightness

Resi

du

al st

d.

dev.

Issues

Should the curve be strictly and tightly below the points?

How to handle the missing data?

Correlation between RGB channels?

III

Brightness

Resi

du

al st

d.

dev.

Solutions

Formulate the inference problem in a probabilistic framework

Learn the prior of noise level functions

III

Brightness

Resi

du

al st

d.

dev.

Outline

Over-segmentation and per-segment variance analysis

Learning the priors of noise level functions (NLF) Synthesize CCD noise Sample noise level functions Learn the prior of noise level functions

Inference: estimate the upper bound of NLF Bayesian MAP to estimate NLFs for RGB channels

Applications Adaptive bilateral filtering Canny edge detection

Noise model

Camera response function (CRF) f: download from Columbia camera response function database (used 196 typical CRFs)

Tsin et. al. Statistical calibration of CCD image process. ICCV, 2001

Camera Noise

Atmospheric Attenuation

Lens/geometricDistortion

CCD Imaging/Bayer Pattern

Fixed PatternNoise

Shot Noise

Thermal Noise

Interpolation/Demosaic

WhiteBalancing

GammaCorrection

A/D Converter

Dark Current Noise

t

QuantizationNoise

SceneRadiance

DigitalImage

CameraIrradiance

)( cs nnLfI

I

L

2)(Var,0)(E sss Lnn Dependent noise:Independent noise: 2)(Var,0)(E ccc nn

Synthesize CCD Noise

Estimate NLF )(),,;( IIEfI Ncs

I NI

Camera response function: f

Dependent noise:

Independent noise:s

c

Sample NLFs by Varying the ParametersCamera response function (CRF) f s

c

Dependent noise:

Independent noise:s

c

0.02

0.18

0.02

0.18

0.02 0.04 0.06

The Prior of NLFs

Likelihood Function

The estimated standard deviation should be probabilistically bigger than and close to the true value

Bayesian MAP inference

nh (n+1)h

)ˆ,ˆ( nnI

I

)|ˆ( p

Validation (1): Synthetic Noise

Add synthetic CCD noise, estimate, compare to the ground truth

— ground truth

estimated———

Validation (2): Measure NLF of a Real Camera

29 images were taken under the same settings (the camera is not in the database for training)

The real NLF is obtained by computing mean and variance per pixel

Validation (3): Robustness Test

Verify that different images from the same camera give the same estimated NLF (camera not in the database for training)

Application (1): Adaptive Bilateral Filtering

Bilateral filter is an edge-preserving low-pass filter Spatial sigma and range sigma

Adaptive bilateral filter Down-weigh RGB values by signal and noise covariance matrices The range sigma is set to be a function of the estimated standard

deviation of the noise

From Durand and Dorsey, SIGGRAPH 02

Input noisy image Smoothing kernel Denoised image

Test on Low and High Noise

low noise high noise

Red Green Blue Red Green Blue

Results—Adaptive Bilateral Filtering

Standard bilateral filtering

Adaptive bilateral filtering

low noise high noise

Results—Adaptive Bilateral Filtering

Zoom in high noise

Standard bilateral filtering

Adaptive bilateral filtering

Application (2): Canny Edge Detection

low noise high noise

Red Green Blue Red Green Blue

Results—Canny Edge Detection

low noise high noise

Parameters adapted in MATLAB

Parameters adapted by estimated noise

Conclusion

Piecewise-smooth image prior model to estimate the upper bound of noise level function (NLF)

Estimate the space of NLF by simulating CCD camera on the existing CRF database

Upper bounds are verified by both synthetic and real experiments

An important step to automate vision algorithms independent of noise

Thank you!

Ce Liu William T. Freeman

CSAIL MIT

Rick Szeliski Sing Bing Kang

Microsoft Research

Noise Estimation from a Single Image