Post on 19-Mar-2016
description
Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois
Frequency-domain Bayer demosaicking
Problem Statement
Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate.Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity.
Frequency-domain Bayer demosaicking
Retinal Cone MosaicThe human visual system must solve a similar problem!
Frequency-domain Bayer demosaicking
Construction of color image from color planes+
Frequency-domain Bayer demosaicking
Lighthouseoriginal
Frequency-domain Bayer demosaicking
Lighthousered original
Frequency-domain Bayer demosaicking
Lighthousegreen original
Frequency-domain Bayer demosaicking
Lighthouseblue original
Frequency-domain Bayer demosaicking
Formation of Color planes
Frequency-domain Bayer demosaicking
Lighthousered subsampled
Frequency-domain Bayer demosaicking
Lighthousegreen subsampled
Frequency-domain Bayer demosaicking
Lighthouseblue subsampled
Frequency-domain Bayer demosaicking
LighthouseBayer CFA image
Frequency-domain Bayer demosaicking
Color plane interpolationGAGBGLGRGIGreen channel: bilinear interpolation
Frequency-domain Bayer demosaicking
Color plane interpolationRCRed channel: bilinear interpolationRNWRNERSWRSERS
Frequency-domain Bayer demosaicking
Lighthousered interpolated
Frequency-domain Bayer demosaicking
Lighthousegreen interpolated
Frequency-domain Bayer demosaicking
Lighthouseblue interpolated
Frequency-domain Bayer demosaicking
LighthouseInterpolated color image
Frequency-domain Bayer demosaicking
Lighthouseoriginal
Frequency-domain Bayer demosaicking
Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
Frequency-domain Bayer demosaicking
Lighthousered interpolatedwith bilinear interpolator
Frequency-domain Bayer demosaicking
Lighthousered interpolatedwith bicubic interpolator
Frequency-domain Bayer demosaicking
Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.We could optically prefilter the image (blur it) so that aliasing is less severe.
Frequency-domain Bayer demosaicking
Lighthousered interpolatedwith bilinear interpolator
Frequency-domain Bayer demosaicking
Lighthouseprefiltered red interpolatedwith bilinear interpolator
Frequency-domain Bayer demosaicking
LighthouseInterpolated color image
Frequency-domain Bayer demosaicking
Lighthouse Prefiltered & Interpolated color image
Frequency-domain Bayer demosaicking
Lighthouse original
Frequency-domain Bayer demosaicking
Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.We could optically prefilter the image (blur it) so that aliasing is less severe.We can process the three color planes together to gather details from all three components.
Frequency-domain Bayer demosaicking
Can we do better?There have been numerous papers and patents describing different algorithms to interpolate the color planes they all work on the three planes together, exploiting the correlation between the three components.Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality). We present here a novel frequency-domain algorithm.
Frequency-domain Bayer demosaicking
Spatial multiplexing modelsubsamplingmultiplexing
Frequency-domain Bayer demosaicking
Spatial multiplexing model
Frequency-domain Bayer demosaicking
Frequency-domain multiplexing modelRe-arranging the spatial multiplexing expression
Frequency-domain Bayer demosaicking
Frequency-domain multiplexing modelDavid Alleysson, EPFL
Frequency-domain Bayer demosaicking
Luma and chrominance components
Frequency-domain Bayer demosaicking
Luma and chrominance componentsLuma fLChroma_1 fC1Chroma_2 fC2
Frequency-domain Bayer demosaicking
Lighthouse BilinearlyInterpolated color image
Frequency-domain Bayer demosaicking
Frequency-domain demosaicking algorithmExtract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to basebandExtract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key)Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L.Matrix the L, C1 and C2 components to get the RGB representation.
Frequency-domain Bayer demosaicking
Spectrum of CFA signalab
Frequency-domain Bayer demosaicking
Using C2a onlyUsing C2b only
Frequency-domain Bayer demosaicking
OriginalFrom C2a onlyFrom C2b onlyDemosaicking using C2a only or C2b only -- details
Frequency-domain Bayer demosaicking
Demosaicking Block Diagram
Frequency-domain Bayer demosaicking
Spectrum of CFA signalab
Frequency-domain Bayer demosaicking
Design IssuesHow to choose the filters h1, h2a and h2bFrequency domain design methodsLeast-squares design methodsSize of the filtersHow to combine the two estimates and Choice of features to guide weighting The two above issues may be inter-related.
Frequency-domain Bayer demosaicking
Filter designGaussian filters (Alleysson)Window design or minimax designDefine ideal response, with pass, stop and transition bandsApproximate using the window design methodRefine using minimax or least pth optimizationCan design low-pass filters and modulate to the center frequency
Frequency-domain Bayer demosaicking
Filter specification u v val0.000 0.00 1.00.110 0.00 1.00.110 0.02 1.00.000 0.10 1.00.030 0.10 1.00.070 0.06 1.00.338 0.00 0.00.338 0.05 0.00.050 0.36 0.00.000 0.36 0.00.184 .205 0.00.500 0.00 0.00.000 0.50 0.00.500 0.50 0.0
Frequency-domain Bayer demosaicking
Ideal response perspective view
Frequency-domain Bayer demosaicking
Ideal response contour plot
Frequency-domain Bayer demosaicking
Window design perspective view
Frequency-domain Bayer demosaicking
Window design contour plot
Frequency-domain Bayer demosaicking
Least pth filter perspective view
Frequency-domain Bayer demosaicking
Least pth filter contour plot
Frequency-domain Bayer demosaicking
21 x 21 filters in SPL published algorithmuvh2ah2bh1
Frequency-domain Bayer demosaicking
Adaptive weighting of C2a and C2bWe want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used
should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice
Frequency-domain Bayer demosaicking
Typical scenarios for local spectrumLC2aC2aC2bC2bC1C1C1C1B: C2b is better estimateLC2aC2aC2bC2bC1C1C1C1A: C2a is better estimateuvvu
Frequency-domain Bayer demosaicking
Scenario AScenario B
Frequency-domain Bayer demosaicking
Typical scenarios for local spectrumLC2aC2aC2bC2bC1C1C1C1B: C2b is better estimateLC2aC2aC2bC2bC1C1C1C1A: C2a is better estimateuvvu
Frequency-domain Bayer demosaicking
Weight selection strategyScenario A: average local energy near (fm, 0) is smaller than near (0, fm ).Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0).Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).
Frequency-domain Bayer demosaicking
Gaussian filters for local energy measurementuvfm = 0.375
Frequency-domain Bayer demosaicking
ResultsResults with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec. 2005. All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods.
Frequency-domain Bayer demosaicking
Mean square error comparison
Frequency-domain Bayer demosaicking