QUALITY-ADAPTIVE SHARPNESS ENHANCEMENT BASED ONA NO-REFERENCE BLOCKINESS METRIC

Remco Muijs1 and Jeroen Tegenbosch2

1 Visual Experiences Group, Philips Research Europe, Eindhoven, the Netherlands2 Video Processing Group, Philips Research Europe, Eindhoven, the Netherlands

ABSTRACT

Sharpness enhancement of compressed video often resultsin the undesired amplification of coding artifacts and noise.In this paper, we propose a noise-robust sharpening methodwith which a high picture quality can be achieved over awide range of input quality. The new algorithm involves apreconditioning procedure using adaptive contrast-based bi-lateral filters. Such filters allow the level of digital noise tobe reduced, while preserving the sharpness of object edges.The extent of preconditioning prior to sharpening is reg-ulated on the basis of a no-reference objective blockinessmetric. Tests on JPEG compressed images indicated thatthe proposed system provides superior image quality com-pared to existing sharpness enhancement techniques over awide range of compression ratios.

1. INTRODUCTION

In view of the ongoing convergence between PC’s and tele-vision sets, both the nature and the quality of video signalsinput to state-of-the-art display devices are rapidly becom-ing more heterogeneous. Consumers desire to watch bothhigh-quality digital photography and DVD-content as wellas low-quality personal video and streaming content. Con-sequently, there is a growing need for video processing al-gorithms that consistently deliver optimal image quality tothe end-user over a wide range of compression ratios.

High-performance image processing for variable inputquality can be particularly challenging in the context of sharp-ness enhancement. In literature, there are many algorithmsaimed at fine detail enhancement, which can be divided intwo main categories [1]: (i) methods that increase the am-plitudes of the middle and high frequencies using linear fil-ter techniques (peaking) and (ii) non-linear techniques withwhich the steepness of edges can be increased (transient im-provement). Obviously, linear techniques can modify theamplitudes of existing spatial frequencies, but cannot createnew frequencies. Therefore, non-linear sharpness enhance-ment should complement linear up-scaling techniques, suchas bi-linear or bi-cubic interpolation [2], whenever the dis-

play resolution is higher than the resolution of the incomingvideo. Transient improvement can be implemented in thespatial domain on the basis of a peaking filter followed by aclipping operation in areas where object edges are detected[3], [4]. A frequency domain alternative was proposed in[5]. Optimal sharpness impressions for both object edgesand textured areas can, in turn, be achieved using appro-priate combinations of linear and non-linear sharpness en-hancement modules (e.g. [4], [6]).

In practice, the high-frequency content of video is oftencorrupted by noise and compression artifacts, such as block-iness, ringing and mosquito noise. Without precautionarymeasures, sharpness enhancement may, therefore, signifi-cantly increase the visual strength of such artifacts, partic-ularly when applied to low-quality input video. This draw-back can be partly alleviated when dedicated techniques forcoding artifact reduction are applied to the video prior tosharpening. These methods typically involve adaptive low-pass filter operations in either the spatial (e.g., [7], [8]), thespatio-temporal or frequency (e.g., [9], [10]) domains.

However, as high frequencies are suppressed during cod-ing artifact reduction and enhanced during sharpening, asimple cascade of individually optimized video processingcomponents often represents a sub-optimal system solution.In this paper, we present an integrated approach to noise-robust enhancement with which this shortcoming can be al-leviated. The new system involves a preconditioning proce-dure using adaptive contrast-based low-pass filters that sup-presses coding artifacts prior to sharpening. Consequently,enhancement is applied exclusively on a high-fidelity, noise-free video signal, whereas digital noise and some low-amplitudefine detail are preserved at their original amplitudes by by-passing the enhancement modules. In the following, we firstoutline the main concept of the proposed noise-robust en-hancement system. Then, we describe how this system canbe adapted to the quality of the incoming video on the ba-sis of no-reference objective quality metrics and finally weillustrate its performance on test images.

Fig. 1. Block diagram of (a) the sharpness enhancement system proposed in [4] and (b) the newly proposed quality-adaptive sharpness noise-robust enhancement system.

2. NOISE-ROBUST SHARPNESS ENHANCEMENT

The proposed algorithm is based on a variant of the sharp-ness enhancement system described in [4], which was de-signed for delivering optimal sharpness on high-definitiondisplays for standard-definition input video. Figure 1a showsa block diagram of this system, which consists of a spa-tial up-scaling operation to convert the video to the displayresolution, followed by parallel application of peaking andLuminance Transient Improvement (LTI) components. LTIis implemented by clipping a strongly peaked video signalaround object edges, whereas peaking is performed using asmall set of linear finite impulse response filters. This sys-tem provides high-quality crisp images for input video ofmoderate to high quality (≥3 Mbps for SD resolution), butis also notoriously prone to the amplification of coding arti-facts when the input quality is low.

Analysis of the above system indicated that the unde-sired increase in blockiness and ringing upon enhancementis primarily caused by the peaking filters, whereas LTI op-erates exclusively on high-contrast object edges that are lessaffected by compression. The noise-robustness of the methodcan, therefore, be increased by applying a preconditioningfilter prior to peaking, such that the enhancement is carriedout exclusively on a video signal that is free of coding ar-tifacts. To this end, the incoming video is low-pass filteredusing a non-linear contrast-based bilateral filter [11]. A bi-lateral filter combines a linear spatial filter kernel with aweighting function over the intensity difference between thecurrent pixel and the pixels in the filter support:

Fb (x, xc) = FLP (x, xc) W(I (x)− I (xc)

), (1)

where FLP (x, xc) and Fb (x, xc) denote the filter coeffi-cients of the spatial low-pass filter and the resulting bilateralfilter, respectively, at position x for a central output pixel atxc, and W (I (x)− I (xc)) represents a weighting functionover the intensity difference between the pixels at x and xc.

When the weighting function decreases with increasing in-tensity differences, the blurring operation is performed pre-dominantly over pixels that are similar to the current pixel,such that blurring across edges can be prevented. In the fol-lowing examples, we employ a simple threshold function asweighting:

W(I (x)− I (xc)

)=

{ 0 for(I (x)− I (xc)

)≥ Ith

1 for(I (x)− I (xc)

)< Ith

(2)The above bilateral filter effectively suppresses compres-sion artifacts while preserving the sharpness of object edges.Note that the filter coefficients of the bilateral filter are dif-ferent for each output pixel and the operation as a whole isnon-linear. As opposed to many coding artifact reductionmethods, bilateral filtering does not separate digital noiseand signal on the basis of frequency content alone; it iscomplemented by a local contrast-based neighborhood se-lection, which makes use of the fact that compression ar-tifacts generally have a lower local contrast than the dom-inant image features. The signal removed by the bilateralfilter (quasi-highpass) contains both compression artifactsand low-amplitude fine detail. Because its suppression maycause a loss of detail, the quasi-highpass signal bypasses theenhancement modules and is added to the enhanced videoat its original amplitude. This approach allows fine detail tobe preserved while preventing coding artifact amplification.

3. QUALITY-ADAPTATION

In order to achieve optimal sharpness enhancement over awide range of input quality, the character and strength ofthe preconditioning should be adaptable to the quality ofthe incoming video. This can be readily achieved by appro-priately mixing the quasi-lowpass video VqLP and quasi-highpass video VqHP before they are passed on to the peak-

Fig. 2. Illustration of the proposed noise-robust sharpness enhancement procedure for a low-quality video frame.Shown are (a) the original video frame to be enhanced, (b) the result after application of the conventional enhancementsystem shown in Figure 1a and (c) the result using the noise-robust enhancement system of Figure 1b. Figures (d)-(f) provide detailed views of the top frames. Without precautionary measures, sharpness enhancement results insignificant amplification of ringing around the outline of the face (frames (b) and (e)), whereas the proposed noise-robust enhancement system provides a similar sharpness impression for the dominant object edges while preservingthe noise levels at or below its original level.

ing modules:

Vpeak = VqLP + αVqHP , (3)Vnone = (1− α)VqHP , (4)

where Vpeak denotes the video signal used as input for peak-ing, Vnone is the video signal to which no enhancement isapplied and α is a quality-dependent mixing factor. Forhigh-quality video, α is set to 1, such that no precondition-ing takes place and optimal sharpening for both object edgesand fine detail is achieved. For low-quality video, in con-trast, α is set to 0 to ensure that sharpness enhancement iscarried out exclusively on the preconditioned quasi-lowpasssignal containing predominantly high-contrast object edges.A block diagram of the quality-adaptive noise-robust ap-

proach to sharpness enhancement is shown in Figure 1b.

Adequate regulation of the above system requires robustobjective quality metrics with which the overall subjectivequality can be estimated from the video itself. In manypractical applications, access to the uncompressed video isnot feasible, such that no-reference approaches to qualityestimation (e.g. [12]) should be used. Subjective experi-ments have indicated that blockiness is the most annoyingcoding artifact at low to moderate (<3Mbit/s) bitrates [13]and is highly correlated with the overall perceived qualityof MPEG-2 encoded video. To limit the numerical cost ofthe system, we therefore propose a low-complexity solutionon the basis of the blocking strength [14]. The performanceof this metric was validated using the LIVE image quality

database [15], revealing a high correlation with subjectiveassessments (r=0.94).

The resulting algorithm was applied to a series of videosequences compressed at bitrates varying from 0.5 Mbps to6 Mbps as well as to the LIVE Image quality database [15]and compared against the system of Figure 1a. The per-formance is illustrated in Figure 2. The original frame isdisplayed in Figure 2a, whereas Figures 2b and 2c show theresults generated using the systems of Figures 1a and 1b,respectively. Both sharpness enhancement algorithms gen-erate object edges that are significantly more crisp. How-ever, without precautionary measures, sharpness enhance-ment also results in a substantial amplification of ringingaround the outline of the face and block artifacts in the cheekof the girl (Figures 2b,e). Noise-robust enhancement, incontrast, allows a similar sharpness impression to be achievedfor object edges while preserving coding artifacts at or be-low their original amplitude (Figures 2c,f).

To further quantify the improvement achieved by theproposed noise-robust system, its performance is assessedusing objective metrics for coding artifacts and sharpness.Figure 3 shows the results of this analysis for the imagesailing3, which was JPEG-encoded at three different com-pression ratios and sharpened at ten different gain settings.In this Figure, the performance of the enhancement sys-tems shown in Figures 1a and 1b is expressed in terms ofthe blockiness metric BIM [16] and the sharpness metricSHP , defined as:

SHP =√

eh/e2w, (5)

where eh and ew denote the average height and width ofobject edges detected by a generic Sobel edge detector. Fig-ure 3b confirms the observation that noise-robust enhance-ment provides a similar sharpness impression at consider-ably lower levels of coding artifacts. The new algorithmconsistently outperforms the system of Figure 1a and theimprovement increases with increasing input compressionratio.

4. CONCLUSIONS

We have proposed a new noise-robust approach to sharpnessenhancement with which the image sharpness can be en-hanced over a wide range of input quality. The new systeminvolves a preconditioning procedure based on non-linearcontrast-based bilateral filters to ensure that sharpness en-hancement is applied exclusively on a video signal that isnot corrupted by coding artifacts and noise. A no-referenceobjective metric for blocking artifacts is used as a rough es-timate of the overall input quality with which the strengthand type of the preconditioning can be regulated. Tests on

a)

1 1.5 2 2.5 3 3.52

2.2

2.4

2.6

2.8

3

Objective blockiness (BIM)

Obj

ectiv

e sh

arpn

ess

(SH

P)

b)

Fig. 3. (a) The image sailing3 and (b) evaluation ofthe sharpening performance in terms of objective qual-ity metrics for sharpness and blockiness, respectively.The bottom frame displays the results obtained using thesystems shown in Figure 1a (black) and Figure 1b (red)after JPEG compression at quality levels of 0.9 (circles),0.6 (diamonds) and 0.3 (triangles), respectively. Noise-robust enhancement allows similar sharpness levels tobe attained at considerably lower blockiness levels.

JPEG encoded still images and MPEG-2 encoded video se-quences indicated that the new algorithm provides superiorimage quality compared to existing sharpness enhancementmethods by creating similar sharpness impressions at con-siderably lower levels of compression artifacts.

5. REFERENCES

[1] G. de Haan, ”Video Processing for multimedia sys-tems”, University Press Eindhoven, 2000.

[2] J. R. Parker, ”Algorithms for image processing andcomputer vision”, John Wiley & Sons, 1997.

[3] P. Rieder and G. Scheffer, ”New concepts on denoisingand sharpening of video signal”, Digest of the ICCE,W-PMP2.05, pp 188-189, Los Angeles, June 2001.

[4] E. Bellers, and J. Caussyn, ”A high definition expe-rience from standard definition video,” Proceedingsof the SPIE, Vol. 5022, The International Society forOptical Engineering, Santa Clara, pp.594-603, January2003.

[5] H. Geenspan, C. H. Anderson and S. Akber, ”Imageenhancementby non-linear extrapolation in frequencyspace” IEEE Trans. on Image Processing, Vol. 9, pp1035-1048, June 2000.

[6] J. Tegenbosch, P. Hofman and M. Bosma, ”Improv-ing nonlinear up-scaling by adapting to the local edgeorientation” Proceedings of the SPIE, Vol. 5308, TheInternational Society for Optical Engineering, SantaClara, pp.1181-1190, January 2004.

[7] S. D. Kim, J. Yi, and H. M. Kim, ”A deblocking filterwith two separate modes in block-based video coding”,IEEE Transactions on Circuits and Systems for VideoTechnology, Vol. 9, pp 455-465, February 1999.

[8] A. Kaup, ”Reduction of ringing noise in transform im-age coding using a simple adaptive filter”, ElectronicsLetters, Vol. 34, pp611-615, October 1998.

[9] J. Huang and Y. Q. Shi, ”A space and frequency hybridalgorithm to remove block artifacts”, Proceedings ofFourth International Conference on Signal Processing,Vol. 2, pp. 869 -872, 1998.

[10] A. Nostratinia, ”Denoising of JPEG images by re-application of JPEG”, Journal of VLSI Signal Process-ing, Vol. 27, pp69-79, 2001.

[11] C. Tomasi and R. Manduchi, ”Bilateral filtering forgray and color images,” in Proceedings, IEEE Inter-national Conference on Computer Vision, pp. 839-846,1998.

[12] J. Caviedes and F. Oberti, ”No-reference quality met-ric for degraded and enhanced video”, Proceedings ofthe VCIP, Vol. 19, pp. 147-162, February 2004.

[13] C.C. Koh, S.K. Mitra, J.M. Foley, and I.E.J. Heynd-erickx, ”Annoyance of individual artifacts in MPEG-2compressed video and their relation to overall annoy-ance”, Proceedings of the SPIE, Vol. 5666, The In-ternational Society for Optical Engineering, Santa Jose,pp. 595-606, January 2005.

[14] R. Muijs and I. Kirenko, ”A No-Reference BlockingArtifact Measure for Adaptive Video Processing”, Pro-ceedings of the13th European Signal Processing Con-ference, Antalya, September 2005.

[15] H. R. Sheikh, Z. Wang, L. Cormack and A. C.Bovik, ”LIVE Image Quality Assessment Database”,http://live.ece.utexas.edu/research/quality.

[16] H. R. Wu and M. Yuen, ”A generalized block edgeimpairment metric for video coding”, IEEE SignalProcessing Letters, vol. 4, pp. 317-320, November1997.