1

Blind Image Deblurring using Multi-Resolutionmethods

André Gonçalo Machado AbelhaDepartment of Electrical and Computer Engineering

Instituto Superior TécnicoAv. Rovisco Pais, 1049-001 Lisboa, Portugal

E-mail: andreabelha@ist.utl.pt

Abstract—Blind image deconvolution is a subject of greatinterest in the scientific community, due to the nature of theproblem and also to its many areas of application. Severalarticles have addressed this issue, but due to its complex nature,solutions with promising results are lacking. An effective methodof blind image deblurring has recently been developed. However,in its current state, has a high computational cost. This paperproposes a multi-resolution approach applied to the blind imagedeconvolution method, in order to minimize the computationalcost. The multi-resolution method starts by deblurring imagesat a low resolution, and progressively increases the resolutionas better estimates of image and blurring operator are beingestimated. The objective of this work is to make a comparisonbetween the multi-resolution approach and the current method,based on synthetic degradations and real-life blurred photos.

Index Terms—Blind image deconvolution, image deblurring,image enhancement, blurring operator, multi-resolution.

I. INTRODUCTION

BLIND IMAGE DECONVOLUTION in processing image,is a problem known a long time. But only recently the

methods have evolved in a more meaningful way. Problemcomplexity and practical constrains are the main reasonsfor blind image deblurring having difficult to evolve. First,the problem is blind, requiring to estimate two sets of data(original image and the blurring filter) out of one, which isthe blurred image. There are also practical reasons that haveinfluenced the evolution of the problem. Due to the methodsof dealing with this type of problems usually requires a lotof processing. Fortunately, the methods that deal with thisproblem has evolved lately in a positive way in improvingthe estimates of blurred images. It has been developed newforms of implementation in hardware, [1], [2], [3] and [4].Techniques of multi-resolution applied to blind image deblur-ring, has indicted interesting results, [4] and [5].

The main difficulty of blind image deblurring, it’s due tothe problem being ill-posed since the only data that do existare the data of the blurred image. Therefore, the method thatdeals with the problem, does not have access to most any typeof information. The process of degradation (blurring) into realimages happens when at the time when an image is obtainedthere is a disturbance that causes the blurring. The action ofdegradation is modeled by the convolution of the blurringoperator with the original image. The blurring operator is

April 2011

also known for PSF - Point Spread Function and blind imagedeblurring as BID. The work done in this thesis, it’s based onthe blind method presented in scientific article [6]. This paperis based on a bayesian approach, which allows to estimate theimage degraded and the PSF of the filter. Both, obtained froma blurred image, not having access to any further information.The BID problems starts from being a problem ill-posed, inwhich there are infinite solutions. By being the problem blind,it is estimated the original image and the PSF of the filterfrom one observation (image blurred). Therefore, the problemis indeterminate. Also, the blurring operator is ill-conditioned,in which it is present a great variability. Finally, quality resultsare very sensitive to noise. Another practical problem present,is due to the high computational cost, techniques of multi-resolution could help to minimize the problem.

Multi-resolutions is a method structured by several levelsof resolution used progressively, thus reducing the amount ofdata to be processed. The objectives of this thesis, starts byreducing some of the drawbacks of the method described in thearticle [6], by the application of techniques of multi-resolutionto the BID method. The objectives are:

• Implementation of the multi-resolution method,• Obtain tests that allow the comparison of results between

the two methods,• improve execution time through the use of the multi-

resolution approach,• Improve the quality of the results to the classic method.• Apply both methods (multi-resolution and classic) to

images with real degradations.

II. MULTI-RESOLUTION BLIND IMAGE DECONVOLUTION

A. Deblurring Method

The blurred image is corrupted through the original imageby an unknown filter. This effect is modeled by

y = h ∗ x+ n, (1)

in which, y is the blurred image, x the original image, his the PSF of the blurring operator, n additive noise, and ∗denotes the mathematical convolution operator. Also, a spatial

2

invariant property is present as a resulting consequence. Thedeblurring method is based on two simple facts.• In natural images, leading edges are sparse.• Edges of blurred images are less sparse than those of

sharp images because they occupy a wider area.1) Image Prior: We take advantage of this knowledge in a

partial benefit manner, with the use of an image prior of sparsedistributions, in which we assume the pixels of the edges tobe independent from each other. Hence, a random edge pixelof the image is assume to follow a sparse prior with density,

p[fi(x)] ∝ e−k[fi(x)+ε]q , (2)

where i is an iterative image index, f(.) is an edge detectoroperator, and therefore, f(x) is the image of the intensities ofthe edges of the image x, k is a scaling edge intensity factor,q controls the prior’s sparsity and ε allows the existence offinite lateral derivatives at f=0 (when 0 < q < 1), provides aneasier optimization, and approximates better the prior to theset of data, when used in the deblurring method. It is assumedto be a Gaussian noise in eq. 1, with zero mean and varianceσ2, which allows us to write the likelihood function of theestimated pair (image + filter),

p(x, h|y) ∝ e−1

2σ2‖y−h∗x‖22

∏i

e−k[fi(x)+ε]q , (3)

Since is not simple to compute ML with eq. 2, we have towrite down the Log-Likelihood function, thus (except fromignoring an irrelevant constant),

L(x, h|y) = − 1

2σ2‖y − h ∗ x‖22 − k

∑i

[fi(x) + ε]q, (4)

Maximizing eq.4 is the same as minimizing the cost function,

C(x, h) =1

2‖y − h ∗ x‖22 + λ

∑i

[fi(x) + ε]q, (5)

with λ = kσ2, being a regularization parameter.2) Edge Detector: An edge image, is obtained through the

combination of the results of the convolution of several edgefilters (each with a specific direction) with the natural image.Thus, to perform an all direction edge image, a set of edgefilters is required. All filters can be derived from a base filterd0, through successive rotations. The base filter proposed in[6] is,

dθ =

[1 2 2 1-1 -2 -2 -1

]/12 (6)

Fig. 1. Set of edge detection filters, in the four directions used. The lefmostis the base filter d0

Fig. 2. Edges of ‘Lena’, computed with the edge detector proposed, usingthe set of filters shown in Fig.1

The other filters were generated by multiple rotations of 45o

of the base filter, has it is in Fig.1.Each image edge with a specific direction is given by,

gθ(x) = dθ ∗ x, (7)

where, θ denotes each particularly direction and Θ is the set ofthe filters. Therefore, to combine all images of edges together,it is used an L2 norm.

f(x) =

√∑θ∈Θ

gθ(x)2 (8)

Figure 2, shows the results of the edge detector operatorapplied to the “Lena”.

3) Deblurring: The cost function (eq.5), can be written asfollows,

C(x, h) =1

2‖y − h ∗ x‖22 + λR[f(x)], (9)

R[f(x)] = Rf (x) is a regularizing term, which benefitsthe deblurring process, in which the edges are sparse. Ac-cordingly with eq.5 and 9 it is now simple to identify thefidelity term of the data, 1

2 ‖y − h ∗ x‖22, and the regularizer,

Rf (x) =∑i[fi(x) + ε]q .

The estimation of the image x, is performed with respectto the eq.9, the deblurring process is done by sharpening xprogressively along the time, while denoising and preventingit to become unnatural. As estimation of x gets more details,same happens to the estimation of the blurring filter h.Experience showed, that geometric progression of parameterλ (λn+1 = λn/r), most of times brings good deblurringresults. On the other hand, a more linear behavior for sparsityparameter q is recommended. Due to λ being high, it makesregularizing term too strong, and therefore, only sharp mainedges will survive, (see Fig. 3).

As the iteration goes on, λ decreases, and the regularizationterm becomes weaker, allowing smaller details (such as tex-ture, patterns, etc.) being taken into account, at a rate whichis controlled by the rate of decrease of the value of λ.

Table I, shows the algorithm for BID process.

B. Multi-Resolution Framework

The method using multi-resolution in blind image deblur-ring problems is intended to have several processing levels,

3

Fig. 3. Left: “Lena” image blurred with a 9x9 square blur. Right: Imageestimate obtained for the first value of λ.

Initialization:1 – Set h to the identity operator.2 – Set x equal to y.3 – Set λ and q to λ0 and q0.

Optimization loop:4 – Find a new x estimate: x = argminx C(x, h)5 – Find a new h estimate: h = argminh C(x, h)6 – Set λ and q to the next values in sequence.7 – If λ ≥ λmin go back to 4; otherwise stop.

TABLE IDEBLURRING METHOD

with a starting low resolution level, which takes a fast overallestimation of the original image and blurring filter. The nexthigher resolution levels, provides better detailed estimations,because resolution is being increased. As MRBID evolvethrough higher resolution levels better estimations are derived.In the last resolution level (original image resolution), anoverall image estimate as well filter estimate had already beenmade, and therefore, the deblurring process will only refinethe high frequency details.

The multi-resolution approach, provides an acceleration onthe process comparing to the classic method. Multi-resolutionof blind image deblurring (MRBID), resorts to use less datain processing, hence, better processing times. Furthermore,MRBID can provide better quality, due to the use of previousestimations from lower resolution levels as initial data on thelast level (original resolution). In our MR method, experienceshowed that three different resolution level topology yield thebest results, in compromise with quality and time. Please seefig.4, to better understand the MR method.

Starting from the level N = 4, the initial input data (x0, h0),is the same as in classic BID method, being x0 = y1/4

1

(therefore, the requirement of downscaling, with factor of 14 ,

on the deblurred image) and h0 identity matrix. After finishingthe first level of MRBID, comes the second level N = 2,which takes as initial input data, the N = 4 level estimates(x1/4, h1/4), resized up by a factor of 2. As well like on level2, the last level works the same way, and therefore, the level2 estimates serves the initial input data, resized up by a factorof 2, x0 = upscale2(x1/2) and h0 = upscale2(h1/2).

All MR process, is controlled by hyper-parameters of thecost function. Due to the 3 level framework, three distinguish

1subscript denotes scaling factor

Fig. 4. Flowchart of the MRBID Topology adopted.

hyper-parameters groups should be considered,• λ<N>0 starting value2, determines the starting regularizing

value for each level,• λ<N>f ending value, determines the last regularization to

be made,• λ<N>n+1 =

λ<N>n

r<N>decreasing rate controls the number of

iterations and therefore speed,• q<N>n controls the edge sparsity to be taken at iterationn.

C. Image Resizing

An image resize tool is required, to adapt to the differentlevels of resolution of the MRBID method. Thus, to achieveMRBID framework described on Fig.4 it is required to in-crease and / or decrease images size.

Since, simple upsampling and downsampling are not thebest image resizing operations, it is require to pronounce thefundamental operations of multirate processing:• Decimation - set of operations to decrease resolution of

an image,• Interpolation set - set of operations to increase resolution

of an image.In search for best results, sinc filtering was developed to

approximate better to the ideal case.

D. Tests

1) Quality measurement: Quality measurement is essentialto compare with different parameters among different methods,as well on the process of parameters optimization. The qualitymeasurement called ISNR - increase signal to noise ratio,

2left superscript denotes level downscale factor

4

was used, in order to measure the increase of quality of theestimated image with reference to the blurred one.

The ISNR measurement can be done in decibels, and isperformed by,

ISNR = 10log10

∑i(y

i − xi0)2∑i(x

i − xi0)2(10)

where i indexes image’s pixels. However, this quality compu-tation has some issues. Due to BID problem being ill-posed,solutions have a large variability. So, to the measurementbe fair, an adaptation to the original image should be done.Such adaptation takes into account affine transformations thatchanges pixels intensities and image translation.

Therefore, N(.) denotes the image’s squared error relativeto the original sharped image x0, with a spatial alignment andpixels intensities rescaling. With respect to this new function,the ISNR is now describes as:

ISNR = 10log10

N(y)

N(x)(11)

All tests and quality measurements were performed with aspatial alignment of 1/4 pixel and with a maximum shift of3 pixels. A more detailed description of this quality measure-ment is done on [6].

2) Parameters Optimization: Parameters optimization is notan easy task, due to several reasons.

1) Being problem ill-posed, a wide variability is patented,thus, different set of parameters and conditions leads todifferent optimization paths and therefore different localminimums.

2) Each BID problem has an odd set of optimum parame-ters, no such a unique set of parameters, which wouldprovide the optimum point for all BID’s problems,exists.

3) Extremely high computational cost. Requires a lot ofprocessing, to cover a big slice of all possibilities.

In an attempt to deal with this issues, a test program wasdeveloped to cover the best as possible.

Images Blur FiltersLena Uniform blurBarbara Focus blurCamera Motion blur (linear)Boat Gauss blur

Double delta blurRandom blurMotion blur (non-linear)Circular blur

TABLE IITEST PROGRAM’S CONTENT

Table II describes test program’s content, which contains 4images of 256 x 256 pixels and 8 blur filters of 9 x 9 pixels.The original images can be found in fig. 8. The filters usedon synthetic degradations are plotted on fig. 9.

All test program’s combinations, consists in 32 single BIDproblems.

To evaluate the quality measured of each bank of tests,a mean is perform among all 32 ISNR results, allowing tocompare between different set of parameters. Thereby, threedifferent set of parameters were obtained. One for classic BIDmethod, and two for Multi-Resolution BID method to comparewith classic BID method: one to improve deblurring qualityresults and other for increasing speed of deblurring process.

E. Results

1) Classic Method: In BID method, since, all process isdone on high resolution environment, at the first stages, ahigh regularizing value is required, merely to the survivalof the main edges, as well, higher sparsity parameter isrequired to maintain those edges sharp. Therefore, initial valueof regularizing is λ0 = 2 and for initial value of sparsityq0 = 0.8.

For decreasing ratio the value chosen was r = 1.732.Usually, as estimation goes further to the end, ISNRestimation starts to saturate to a limit value (withoutnoise), that is, usually, the best estimation. The laststep λf , is the next progression value bellow 10−9.For sparsity values, the sequence used was q =[0.8, 0.8, 0.75, 0.7, 0.65, 0.6, 0.55, 0.5, 0.45, 0.4, 0.35, 0.3, 0.25, 0.2],repeating the last value of q when BID process reaches theend of the sequence, until the end of entire process.

Fig. 5. Right: Image Estimation of Lena using classic BID. Left: SquareFilter Estimation

An example of a classic BID estimation, from a syntheticdegradation of a square blur filter, and its correspondent filterestimation is shown in fig.5.

2) Multi-Resolution: Among all different resolution levelsin MRBID, it stills to make sense to decrease the regularizingvalue, but for the sparsity parameter this is not true. The bestsequence of sparsity parameter, is the same for every levelof resolution, therefore, Nqk = qk and is defined by q =[0.4, 0.3, 0.2], once again, repeating the last sequence value tillthe end. The main difference on improving quality or speed,resides on how the regularizing parameters are defined in eachsituation.

To improve quality of MRBID method compared to classicBID, basically, the idea is to use the first two levels, so anoverview of the image estimate and the blur filter are used asinitial data in the last level. The last level starts with a highregularizing value, and runs at low decreasing ratio, to cover

5

a wider range of edge intensities, resulting in better qualityestimations.

Level λ0 r λfN = 4 0.2 5.92 10−3

N = 2 0.1 3.16 10−5

N = 1 0.1 1.73 10−8

TABLE IIIMRBID IMAGE PRIOR REGULARIZATION PARAMETERS FOR QUALITY

IMPROVEMENT.

TableIII, describes the start and stop values as well thedecreasing ratio of regularization.

Fig. 6. Right: Image Estimation of Lena using MRBID with Qualityenhancement. Left: Square Filter Estimation

An example of MRBID estimation with Quality enhance-ment, from a synthetic degradation of a square blur filter,and its correspondent filter estimation is shown in fig.6. Itis possible to perceive the quality improvement between fig.5and fig.6, also, estimation of MRBID is more ringing proofthan in classic BID.

On BID, the average ISNR obtained was 5.95dB, however,MRBID average improvement was 7.25dB. Single tests wereperformed on a 2.4Ghz Core 2 Duo correspondent processorin Matlab, since, quality is the main objective in this test, timeis not critical, therefore, for BID method the average time is7 min 36 seg and for MRBID is 6 min 25 seg.

When improving speed, a progressive approach is adopted,taking all levels as an integral part from the whole process,being the first level responsible for low details estimation,second level medium details estimation and the last level forhigh details estimation.

Level λ0 r λfN = 4 0.15 6.48 3× 10−3

N = 2 0.05 3.46 10−4

N = 1 5× 10−3 5.10 5× 10−5

TABLE IVMRBID IMAGE PRIOR REGULARIZATION PARAMETERS FOR SPEED

IMPROVEMENT.

An example of MRBID estimation with Speed enhance-ment, from a synthetic degradation of a square blur filter, andits correspondent filter estimation is shown in fig.7.

Fig. 7. Right: Image Estimation of Barbara using MRBID with Speedenhancement. Left: Motion non-linear Filter Estimation

Comparing regularizing parameters from table IV with tableIII, stands out, some kind of continuity on parameter’s evolu-tion in an attempt to cover an entire range of values amongdifferent levels, while on quality improvement, parameters ofdifferent levels are much more independent, no such idea ofa global deblurring process exists. Decreasing ratios are muchmore higher, resulting in less iterations and therefore, lesstime. Also, the last regularizing value is bigger, causing theprocess to stop much earlier.

Classic BID ISNR average without Double Delta blur is6.19dB and MRBID for speed improvement is 6.20dB, whichare quite similar. This results in large speed improvement, likealmost 10 times faster! Remind you on classic BID, processingtime is 7 min 36 seg, while this new approach leads to aprocessing time of 48 seg.

3) Deblurring with Noise: To cover as much as possibleof all kind of situations and conditions, a bunch of tests withpresence of noise was also included. All tests with noise wereperformed with addiction of Gaussian i.i.d. noise at BSNR -Blurred Signal to Noise Ratio of 30 dB .

MRBID method with noise was performed with the param-eters obtained for quality enhancement. A non-blind methodwas also performed, to give an idea of the “roof” values.Therefore, classic BID method resulted an average of ISNR= 3.61 dB. MRBID with parameters of quality enhancementresulted an average of ISNR = 4.50 dB and the non-blindmethod resulted an average of ISNR = 7.06 dB.

4) Deblurring Real Images: There were several deblurringtests in images (256×256 pixels) with real degradations. Thoseimages were deblurred using the classic BID method, MRBIDof quality enhancement and MRBID of speed enhancement.The real blurs caused were: Out-of-Focus, motion and handshake.

III. CONCLUSION

The method multi-resolution proposed has been successfullyimplemented, having been achieved the goals set for thisthesis. Tests were carried out with the application of a set ofimages and filters, performed by both methods under variousconditions, covering a wider range of problems of blind imagedeblurring. It is concluded that the method can be used intwo different ways, for the improvement of quality, or theimprovement of speed of execution. We found that with the useof the parameters for the improvement of quality, has provided

6

quality improvements in average of 1.3 dB compared to theclassic BID method. On the other hand, with the parametersfor the improvement of speed, we were able to a reduction of9.5 times the time of execution of the method, keeping on theaverage on the quality of estimates. The method MRBID, whensubmitted to a set of tests under the influence of noise withthe use of the parameters for improving the quality, provided,on average, better results than the classic BID method inapproximately 0.9 dB. Both are still far from the quality ofthe estimates of the non-blind method. Estimates obtained byclassic BID method, MRBID with enhancement on quality andMRBID with enhancement on speed in images submitted toreal blurrings, were successful, with the main advantage of themethod MRBID with enhancement on speed provide resultsthat are as good as the other methods, while processing timeis well below.

APPENDIX

Fig. 8. Originial images used on tests from up-right to bottom-left: “Lena”, “Barbara” , “Camera” , “Boat” .

Fig. 9. Originial filters used on synthetic degradations from up-right tobottom-left: “Uniform blur” , “Focus blur” , “Motion blur (linear)” ,“Gaussian blur” , “Double Delta blur” , “Random blur” , “Motion blur(non-linear)” , “Circular Blur” .

ACKNOWLEDGMENTS

Thanks to Prof. Luís Borges de Almeida and Doctor Mari-ana S. C. Almeida, for their support.

REFERENCES

[1] C. W. Quammen, D. Feng and R. M. Taylor II, “Performance of 3ddeconvolution algorithms on multi-core and many-core architectures,”Chapel Hill Dep. Comp. Sci., Univ. N. Carolina, 2009.

[2] J. Fung and S. Mann, “Using graphics devices in reverse: Gpu-based im-age processing and computer vision,” IEEE ICME, Hannover, Germany,2008.

[3] T. Mazanec, A. Hermanek and J. Kamenicky, “Blind image deconvolutionalgorithm on nvidia cuda platform,” 13th IEEE Symp. Des. Diag. Elec.Cir. Sys., pp. 125–126, 2010.

[4] L. Domanski, P. Vallotton and D. Wang, “Two and three-dimensionalimage deconvolution on graphics hardware,” Cong. IMACS/MODSIM,Cairns, Australia, 2009.

[5] C. Wang, L. Sun, Z.Y. Chen, J.W. Zhang and S.Q. Yang, “Multi-scaleblind motion deblurring using local minimum,” IOP Pub. Inv. Problems,vol. 26, 2010.

[6] M. S. C. Almeida and L. B. Almeida, “Blind and semi-blind deblurringof natural images,” IEEE Trans. Image Process., vol. 19, Jan. 2010, no.1.