iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation

Accuracy Improvements and Artifacts Removal in Edge Based

Image Interpolation

Andrea [email protected]

UNIVERSITY OF VERONA – ITALYDepartment of Computer Science

Vision, Image Processing & Sound Lab

UNIVERSITY OF CAGLIARI - ITALYFaculty of Mathematical, Physics and Natural Sciences

Department of Computer Science

Nicola [email protected]

N. Asuni, A. Giachetti: Accuracy Improvements and Artifacts Removal in Edge Based Image InterpolationVISAPP 2008 - 22/25 January, 2008 - Funchal, Madeira - Portugal 2

UNIVERSITY OF CAGLIARIFaculty of Mathematical, Physics and Natural SciencesDepartment of Computer Science

Summary

Introductioninterpolation and image upscaling, short literature review

NEDI - New Edge Directed Interpolationhow it works, limits

iNEDI – improved NEDImodified method to reduce NEDI artifacts

Experimental ResultsiNEDI provides the best results in a large set of objective and subjective tests.

Conclusions



InterpolationConstructing new data points from a discrete set of known data points.Construct a function which closely fits a discrete set of known data points (curve fitting or regression analysis).

Generally, it is an ill-conditioned problem.The selection of the adequate interpolation method depends by the nature of know data points.



Digital Image Interpolation (1)

Process that allows to obtain an High Resolution image form its Low Resolution counterpart.

LR

HR

Imag

es

©2

00

5 N

icola

Asu

ni -

Ala

my

upsizing, resampling, zooming, magnification, resolution enhancement



Digital Image Interpolation (2)

LR

HR

The interpolation procedure tries to recover missing information by assuming that there is a known relationship between a low resolution image and the same image acquired with an high resolution sensor.



Why do we need to resample?

Icon

s ©

20

05

ww

w.t

pd

kd

esi

gn

.net

Image upscaling through pixel interpolation is used in different fields and devices to create high resolution images with a "natural" appearance from low resolution acquired data.

Applications of this procedure can be found in image viewing or processing software, photographic printing and Computer Graphics.

Real time algorithms can also be applied to increase the perceived quality of video streaming or textures in virtual navigation tools.



Interpolation Artifacts

aliasing (jaggies) blurring edge halo frequency artifacts

General upsampling algorithms do not add real information. Interpolation cannot reconstruct original high frequencies but edge geometries could be preserved. Sharpness and freedom from artifacts in edges are two critical factors in the perceived quality of images.



Interpolation Methods

Several algorithms have been proposed in literature to obtain better results and several patents have been obtained for "smart" interpolation techniques.

Few systematic comparisons have been, however, presented and it is difficult to determine which method is the best suited for a selected application.

Main categoriesLinear FilteringNon-linear methodsStatical methodsEdge-directed methods



Linear Filtering

OR BCBLNN

The simplest image interpolation algorithms are based on linear filtering.

These methods are computationally efficient and especially the bicubic interpolation provides visually good images, that do not appear, however, "natural" due to blur and jagged contours.



Non-Linear Methods

Lu, X., Hong, P. S., and Smith, M. J. T. (2003). An efficient directional image interpolation method. In Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, volume 3, pages 97–100.

Schultz, R. R. and Stevenson, R. L. (1994). A bayesian approach to image expansion for improved definition. IEEE Trans. Image Processing, 3:233–242.

Takahashi, Y. and Taguchi, A. (2002). An enlargement method of digital images with the prediction of highfrequency components. In Proc. IEEE Int. Conf. Ac. Speech Signal Proc., volume 4, pages 3700–3703.

Morse, B. and Schwartzwald, D. (2001). Image magnification using level-set reconstruction. In Proc. IEEE Conf. Computer Vision Pattern Recognition, volume 3, pages 333–340.

Muresan, D. and Parks, T. (2004). Adaptively quadratic (aqua) image interpolation. IEEE Transactions on Image Processing, 13(5):690–698.

Non linear methods are usually based on an implicit or explicit search of local image features and on a subsequent local adaptation of the interpolation function to the (low resolution) extracted features.



Statistical Methods

Atkins, C. B., Bouman, C. A., and Allebach, J. P. (2001). Optimal image scaling using pixel classification. In Proc. IEEE Int. Conf. Image Processing, volume 3, pages 864–867.

Freeman, W. T., Jones, T. R., and Pasztor, E. C. (2002). Example-based super-resolution. IEEE Computer Graphics and Applications, 22(2):56–65.

Sun, J., Zheng, N., Tao, H., and Shum, H. (2003). Image hallucination with primal sketch priors. In Proceedings IEEE conf. on Computer Vision and Pattern Recognition, volume 2.

Some authors have tried to exploit pixel or texture statistics or databases of example images to obtain good high resolution images.

The huge variety of natural textures and scales makes, however, quite difficult a general purpose use of similar techniques, though they can be efficiently applied to particular tasks such as searching patterns like faces, trees, etc.



Edge-Directed Methods

J. Allebach and P. W. Wong, "Edge-directed interpolation," Proc. IEEE Int. Conf. Image Processing, vol. 3, pp. 707–710, September 1996.

X. Li and M. T. Orchard, "New Edge-Directed Interpolation," IEEE Trans. on Image Processing, Vol. 10, No. 10, pp. 1521-1527, October 2001.

These methods try to improve the accuracy of the interpolation characterizing the edge features in a larger region around the point to interpolate. They aims at interpolating along edges rather than across them to prevent blurring. They usually produce images with sharper edges.

NEDI seems to provide the best results for natural images, even in the case of large scale factors. This is why we start our analysis describing this technique and then proposing several improvements.



NEDI (New Edge-Directed Interpolation)

Edge-directed adaptative method for natural images. Its behavior depends on what it is interpolating. It uses bilinear interpolation for smooth regions.

This method first estimates local covariance coefficients from a low-resolution image and then use these covariance estimates to adapt the interpolation at a higher resolution based on the geometric duality between the low-resolution covariance and the high-resolution covariance.

The image size is doubled in each direction at each cycle.



NEDI – 4th order interpolation (1)

The darker circles represent the pixels from the original Low Resolution grid, the white circles represent the pixels that are to be interpolated and the central circle is the pixel to be interpolated in this first step.



NEDI – 4th order interpolation (2)

The central point is calculated using a fourth-order interpolation algorithm that simply sums the contribute of each diagonal neighbor multiplied by its alpha coefficient.



NEDI – Estimation of Coefficients (1)

NEDI estimates αcoefficients by solving an unconstrained system of linear equations.

The system is obtained by assuming that the coefficients linking each pixel with its four diagonal neighbors do not change with scale and that they are constant in a squared window W centered in the pixel to interpolate.



NEDI – Estimation of Coefficients (2)

The y vector contains the values of the pixels contained on the window W.

The C matrix contains in each row the values of the four diagonal neighbors of each pixel listed on y vector.

NEDI estimates α coefficients by minimizing the Mean Square Error (MSE).



NEDI – Second Step

The second NEDI step fills the remaining gaps in the same way of the first step after a π/4 rotation of the grid.

The darker circles represent the pixels from the original Low Resolution grid, the white circles represent the pixels that are to be interpolated and the gray circles represent the pixels interpolated on the first step.



NEDI – Example 8X

NEAREST NEIGHBOR BICUBIC NEDI

Jagged contours are evident in nearest neighbor and bicubic interpolation.The NEDI interpolation presents sharp edges, even if introduces different artifacts and performs often worser than the bicubic method in quantitative comparisons.



NEDI – Limits

The assumption of local stationarity of the covariance (local regularity in curvature) is violated in several cases.Introduces several artifacts on high-frequency regions.The system is often ill-conditioned. The use of larger windows improves the conditioning of the CTC matrix but produces blurred images.Interpolated pixel values change with the global brightness.Very high computational complexity.



iNEDI (improved NEDI)

Main improvements:Non edge pixels handlingWindows shape and dynamic windowsEdge "segmentation"Value AdjustMatrix conditioning, error propagation and minimum norm solution

iNEDI (improved New Edge-Directed Interpolation) is the proposed technique to fix/reduce the original NEDI problems and increase the interpolation accuracy.



iNEDI – Non edge pixels handling

If the local gray level variation is below a fixed threshold NEDI uses the bilinear interpolation, while iNEDI uses bicubic interpolation.

This choice gives a good trade-off between edge direction preservation, accuracy and speed, using higher values of the threshold (using iNEDI only for strong edges).

When the four pixels used to calculate the interpolated ones have a similar gray level, there is no need to compute the NEDI coefficients. If the covariance is stationary, a small error causes a bad conditioning of the solution.



iNEDI – Windows shape

The squared window Wused by NEDI introduces directional artifacts and makes the algorithm non isotropic. This has been solved by iNEDI using a circular window.

To account for various frequencies regions, iNEDI can dynamically increase the window radius from a minimum to a maximum value while the normalized residual of the least squares fit decreases.



iNEDI – Edge "segmentation"

iNEDI tries to exclude from the circular window all pixels that do not belong to the local edge using a sort of region growing method that excludes all uniform areas and pixels that are not connected to the edge.

Two pixels are connected if they are adjacent and their values are included in a predefined range.



iNEDI – Value Adjust

iNEDI increases the probability of obtaining a good interpolation, but there is still the possibility of having unwanted high frequencies.

For this reason we put a further constraint by replacing any interpolated value outside the intensity range of the four neighbors with the closest of the values delimiting that range.



iNEDI – Matrix conditioning, error propagation and minimum norm solution

The overconstrained system Cα = y is almost always ill-conditioned and a small error in y can cause a large error in the estimated α.

A simple trick is to add a constant value to the gray levels, in order to have all values far from zero. This change is effective in reducing artifacts and wrong estimates.The solution to the least squares problem is not unique. On iNEDI we rewrite the system as α = C+y where C+=(CTC)-1CT is the Moore-Penrose pseudoinverse matrix, and find the minimum norm solution using the Singular Value Decomposition. In this way we avoid the direct inversion of CTCmatrix.



iNEDI – Implementation for Testing

iNEDI prototype has been implemented in matlab language (compatible with MATLAB®

and Octave software applications) and released as a FLOSS by adopting GPL v.2 license. http://inedi.tecnick.com

iNEDI code includes:Simple edge detection on thresholdPadding procedures to preserve image bordersCropping and Rotation proceduresRGB image handling



iNEDI vs NEDI – 16X Example

NEDI iNEDI



Experimental Results

Objective evalutationA simple test often used in literature consists of generating low resolution images by filtering and subsampling high resolution ones and then measure the difference between the differently re-upsized images and the original one.

Subjective evalutationWe have asked to a group of 24 people to sort from the worse to the best a set of 12 RGB images enlarged by a 8x factor with four different algorithms (iNEDI, NEDI, bicubic and bilinear).

iNEDI has been widely tested and compared with other methods found in literature, as well as with the original NEDI. We have used both objective and subjectiveevaluation tests.



Results – Objective MetricsPSNR (Peak-to-Signal Noise Ratio) is used to compare the differences between interpolated and reference images. PSNR measures the accuracy of reconstruction.

The constant MAXPIX is the end scale value of the pixel intensity. MSE is the Mean Square Error between pixel values of the original and interpolated image.

Typical PSNR values are between 20 and 40 dB. Higher values indicates a better image reconstruction. Improvements of 0.1 dB are usually considered significative.



Results – Test images

9 8bit 1025x1025 natural images, subsampled at 256x256 and 128x128 and then upscaled respectively two times and four times using different algorithms.

Different target images (512x512 and 511x511) were used to account the different image shifts introduced by the various techniques.

We also crop a 50 pixel frame before comparison to account for the different behavior of the selected methods near the image border.



Results – 2X PSNR Measures

iNEDI improvements vs NEDI are between 0.54 and 1.63 dB with an average value of 0.85 dB.



Results – 4X PSNR Measures

iNEDI improvements vs NEDI are between 0.52 and 1.41 dB with an average value of 0.85 dB.



Results – PSNR Measures

The iNEDI accuracy of the reconstruction is higher than those obtained with the other techniques.

NOTE: The good results of the bicubic interpolation does not mean that it is surely better than other methods. Original NEDI, as well as the other edge based method tested, are effective in removing the typical artifacts of the bicubic and bilinear interpolation. The lower PSNR is probably due to the other kinds of artifacts affecting NEDI and the excessive smoothing of the other approaches.



Results – 8X Artificial Image

+3.43 dBiNEDI removes the jagged lines effects of the linear methods and the directional artifacts of NEDI.



Results – Extended Comparison

To compare the iNEDI technique with other techniques available on commercial software, we have extended the test provided by the SAR Image Processor Software:http://www.general-cathexis.com/interpolation.html

The iNEDI algorithm provided a PSNR 1 dB higher than the best one of the 42 methods included (DDL with SuperRez postprocessing).



Results – Subjective evaluation

Traditional Signal-to-Noise Ratio measures cannot be compared with the perceived quality by humans.

A group of 24 people have been asked to give a qualitative judgment on 12 color images originally of 80x60 pixels and enlarged of a factor 8 with iNEDI, NEDI, Bicubic and Bilinear interpolation. The qualitative judgment has been performed sorting the images from the worst (1) to the best (4).



Results – Subjective measures

The subjective results confirmed the results of the analysis based on the PSNR.



Conclusions

Images enlarged with the proposed technique appear more natural and less smoothed than those obtained with other approaches presented in literature, and both psychological and quantitative tests confirm this fact.

For selected applications, such as printing or off line extrapolation of high resolution textures from low resolution data, the relevant computational effort is not a problem, while for applications requiring a fast image processing, such as improving quality of video streaming, different methods should be applied, even if the algorithm can be optimized and parallelized.



Thanks for your attention.

Any Questions?

iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation

Technology

Transcript of iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation