Post on 29-Mar-2015
Transform-based Non-local Methods for Image Restoration
IT530, Lecture Notes
2
Non-local Techniques
Natural images have a great deal of redundancy: patches from different regions can be very similar
NL-Means: a non-local pixel-based method (Buades et al, 2005)
•Awate and Whitaker (PAMI 2007)•Popat and Picard (TIP 1998)•De-Bonet (MIT Tech report 1998)•Wang et al (IEEE SPL 2003) Difference
between patches
3
Transform-based Techniques
DCT coefficientsof a “typical” natural image
Observed property of natural images:Sparse coefficients
in Fourier/DCT/Wavelet domain
4
Transform-based Techniques
Project Patch onto Transform Basis (DCT/Wavelet etc.)
Transform Coefficients
Manipulate coefficients of noisy patch coefficients of
filtered patch
Get filtered patch by inversion of the transform from filtered coefficients
Repeat for all patches in sliding window fashion: final filtered image obtained by averaging overlapping filtered patches.
Noisy Patch
Clean Patch
DCT Coefficients (absolute value)
Transform basis
Patch matrix
Non-local transform based techniques
• Machine learning approach.
• Make use of non-local self-similarity at the patch level to learn good transform bases for denoising.
Non-local PCA
• PCA is the most basic transform to learn!• Assume Gaussian noise (mean zero, known
variance).• Given a patch (called “reference patch”) in a noisy
image, search for similar patches elsewhere in the image.
• Given such a collection of patches, compute their PCA bases.
• Compute the eigen-coefficients of the reference patch.
Non-local PCA
• Manipulate these eigen-coefficients (by hard-thresholding or Wiener filter).
• Reconstruct the reference patch.• Repeat the procedure for every patch in the
noisy image in sliding window fashion.• Average the multiple hypotheses that appear
at a pixel.
Ref: Muresan and Parks, “Adaptive principal components for image denoising”, ICIP 2003.
8
Criterion: Patch Similarity
),0(~
),0(~
2
1
NzPP
NzPP
ref
ref
Two patches: identical
modulo noise )n(~2
||PP|| 222
221
)2
n,2
x(GammaInc)n;x(F
22
Cumulative of chi-squared is incomplete gamma
function.
2
n3)
2
n;99.0(F
221
2
n3
2
||PP|| 2
2
221
A patch P1 is considered similar to patch Pref if their squared
difference is less than or equal to 22n3
Non-local PCA
),0(~, Nnnxy iiii Noisy patch
The l-th coefficient of yi – computed by projection onto the l-th local PCA bases (obtained from similar patches)
We are modelling as zero-mean Gaussian random variable with variance
lix
l
Two-stage non-local PCA
Ref: Zhang et al, Two stage image denoising by principal components analysis with local pixel grouping
BM3D PCA
KSVD
BM3D PCA
KSVD
BM3D PCA
KSVD
Time complexity
• The patches of size p x p are treated as vectors of size p2 x 1.
• Hence the covariance matrices are of size p2 x p2.
• Eigen-analysis will have a time complexity of O(p6) per patch (not counting the time to search for similar patches).
• This is quite expensive.
Non-local collaborative filtering: Block Matching in 3D (BM3D)
• The state of the art method today.• Based on the idea of non-local similarity at the
patch-level.• Given a reference patch in the noisy image,
this method again collects similar patches.• But this time, the similar patches and the
reference patch are arranged in the form of a 3D stack (of say some K patches in all).
Non-local collaborative filtering: Block Matching in 3D (BM3D)
• The stack is projected onto 3D transform bases (typically 3D DCT, or tensor product of 2D DCT and 1D Haar wavelets).
• The 3D transform coefficients are manipulated – usually by hard thresholding using the universal threshold .
• All the patches in the entire stack are reconstructed using an inverse 3D transform.
• This is repeated for every patch in the image. The multiple answers appearing at any pixel are averaged.
)log(2 2Kp
BM3D – second stage
• The preceding steps form the first stage of BM3D.• The output image of the first step is used to
compute patch similarities (this will be more robust than computing the similarities in the noisy image).
• Patches from the first-stage image are then appropriately assembled into a stack.
• Corresponding patches from the noisy image are assembled into a second stack.
BM3D – second stage
• 3D transform coefficients of both the stacks are computed.
• The second stack is denoised using Wiener filtering as follows:
• This is again repeated in sliding-window fashion with averaging.
noisy22onestage
2onestage
stagesecond cc
cc
Note: BM3D does allow individual patches from the patch to be independently filtered. It filters the similar patches from the stack collectively, assuming a dependence between them! This is the reason for its superior performance.
Results
• State of the art denoising method.
• Impressive results even at high noise levels.
• Residual images produced are very noisy.
• Preserves textures and fine details/edges very well.
24
NL-SVD: PSNR 30.88SSIM 0.882
Noisy PSNR: 22.13
BM3D1: PSNR 31.02SSIM 0.884
HOSVD: PSNR 31.53SSIM 0.897
BM3D2: PSNR 31.66SSIM 0.903
NLMeans: PSNR 29.42SSIM 0.821
KSVD: PSNR 30.762SSIM 0.877
25
NL-SVDBM3D1BM3D2HOSVDKSVD
26
NL-SVD: PSNR 30.187SSIM 0.801
Noisy PSNR: 22.13
BM3D1: PSNR 30.395SSIM 0.809
HOSVD: PSNR 30.491SSIM 0.814
BM3D2: PSNR 30.8SSIM 0.824
NLMeans: PSNR 28.91SSIM 0.753
KSVD: PSNR 30.36SSIM 0.803
27
NL-SVDBM3D1BM3D2HOSVDKSVD
28Gaussian Noise sigma = 15