348

Blotch Removal for Old Movie Restoration using

Legendre moment and Particle Swarm Optimization

Krishna P J Dept. of Electronics and Communication

College of Engineering Trivandrwn Thiruvanthapuram, India

Email: krishna.pj85@gmail.com

Abstract-A blotch is one among several artifacts that can degrade old movies which had been stored in films. Blotches are seen as randomly occuring black or white spots which are disturbing to the viewer. A novel approach to blotch removal for old movie restoration is presented in this paper. This method combines the merits of image representation using Legendre moments, warped distance interpolation and adaptive particle swarm optimisation. After detecting the blotches using the well known sROD detector, a feature vector comprising of Legendre moments and average gray value is interpolated for windows in the blotch region using the neighbouring pixel values in the known region. A warped distance approach has been used for better interpolation results. This is follwed by a search for the optimum pixel values in the blotch region using adaptive particle swarm optimisation such that the feature vector for the estimated pixels match the interpolated one. Experiments show that this algorithm gives satisfactory results and superior performance.

Index Terms-Blotch removal, Old movie restoration, Adaptive particle swarm optimisation, Warped distance, Legendre mo­ment.

I. INTRODUCTION

A lot of old movies are greatly valued not only for their artistic value or historical content, but also because they are treasure houses of ancient culture and traditions. So preservation and digitisation of these movies has been considered to be given due importance. Many works have been carried out to restore old movies. Degradations occur to these movies due to reasons such as aging, accumulation of dirt, improper handling, bad film quality and wear and tear due to use in equipment. Of these, the commonly seen degradations are blotches, line scratches, intensity flicker and frame misalignment. Blotches, which are seen as randomly occurring disturbing black or white spots, are the subject of concern in this paper. Blotches are caused by dirt and by the loss of the gelatin covering the film due to aging effects and bad film qUality. This paper presents a novel method to remove blotches in old movies.

There are two characteristic features for blotches occurring in degraded movies. One feature is that blotches rarely occur at the same place in successive frames, giving rise to temporal discontinuity. The other one is their spatial coherency. i.e, the pixel values in the blotch region are almost uniform and differ

Santhosh Kumar S Dept. of Electronics and Communication

College of Engineering Trivandrum Thiruvanthapuram, India

Email: santhosh_kumar_s@mailcity.com

considerably from the neighboring pixel values in the non­blotch region. These two properties of blotches can be utilised to detect and remove them.

Assuming the missing areas have been correctly identified, techniques have been proposed to fill them. Storey [5] used a three-tap median operation to interpolate the missing pixels. Kokaram et al. [9] extended this idea by introducing a 3-D median filtering operation on a 3x3x3 motion compensated pixel volume around each missing pixel. Roosmalen et al. [12] combines autoregressive models and Markov random field (MRF) techniques for interpolating missing data. A nonlinear interpolator has also been used to fill in the missing data. Khriji et al.[6] use a rational function filter to fill in missing pixels using neighboring information. Their method assumes a perfect defect localization algorithm.

Nadenau and Mitra [10] use rank ordered differences be­tween the current frame and its previous and next (motion compensated) frames to detect small blotch and scratch arte­facts. Morphological filters have also been used to detect blotches [13].

Most of the current methods for blotch removal rely on pixel based interpolation. These methods do not necessarily preserve the image features inside the blotch region. Our method relies on feature based interpolation. The use of the particular image feature, Legendre moment, is motivated by the orthogonality property of the Legendre polynomial, which guarantees the non redundancy of the description of an image or a shape. By using a windowed approach together with particle swarm optimisation, the computation time can be reduced considerably.

This paper is organised as follows. Section 2.1 describes the detector we have chosen. Section 2.2 is a brief description of Legendre moments and their importance. Section 2.3 is about the warped distance interpolation technique and how we have modified and included it in our algorithm. The adaptive particle swarm optimisation algorithm has been discussed in section 2.4. Then, the proposed method, its mathematical formulation and the flowchart has been presented in section 2.5. Finally the results and conclusion have been given in section 3.

978-1-4244-9799-7/11/$26.00 ©2011 IEEE

II. PROPOSED ALGORITHM FOR BLOTCH REMOVAL

In our method, blotch detection is performed using a simple method called Simplified Rank Order Detector ( sROD) [10). Once the blotch regions have been correctly identified, the next step is the interpolation of the Legendre moment and the average gray value in those regions. This is achieved using a windowed approach. An inverse weighted distance method in which the weight is modified by warping technique [2] is used. The neighboring known windows from the preceding and succeeding frames are used. Then, using the iterative particle swarm optimisation technique, search for the optimum pixel values in the unknown blotch region is carried out until the image features corresponding to the optimum value matches the interpolated one, with minimum error.

A. Simplified Rank Order Detector

In the simplified Rank Order Detector (sROD), in order to determine whether a pixel (x, y) belongs to the blotch region or not, a set of k reference pixels, Pk are chosen from the preceding and succeeding images at location spatially co-sited with pixel (x, y) and its two closest vertical neighbors. This is shown in fig 1. Then these pixels are ordered by their rank to form pixels Rk with R1 ::; R2 ::; R3 ... Rk. The sROD is then calculated as:

min(Rk) - In(x, y) In (x, y) < min(Rk) In(x, y) - max(Rk) In(x, y) > max(Rk)

o

if

if

Otherwise (1)

This sRO D is then compared with a threshold to determine whether the pixel is corrupted. This is a simple and efficient method and can be improved further using motion estimation. Here we have chosen the value of k to be 18.

Frame 0-1

�1 P2 P3

Frame 0

Frame 0+1

�4 PS P6

Fig. I. Selection of neighboring pixels from preceding and next frames

B. Legendre moment

The Legendre moment is a member of moment functions of the two-dimensional image intensity distribution and is used in a variety of applications, as descriptors of shape. Image moments those are invariant with respect to the transforma­tions of scale, translation, and rotation have been used in areas such as pattern recognition, object identification and template matching. Orthogonal moments have additional properties of being more robust in the presence of image noise, and having a near-zero redundancy measure in a feature set. The discrete approximation of Legendre moment of order m + n of a two dimensional intensity distribution Pxy is given by [4]:

(2m + 1)(2n + 1) "" Amn = 4

� � Pm (x)Pn(y)Pxy (2) x y

where x, yare defined over the interval [-1,1]. The nth order Legendre polynomial is defined as:

n Pn (x) = L anjxj

j=O where the Legendre coefficients anj are given by:

. _ -1 (n-j)/2 1 (n + j)! an) - 2n ((n-j)), ((n+j)),.,

2 . 2 .J .

(3)

(4)

The Legendre polynomial Pn(x) can also be calculated using the recursive relation:

2n+ 1 n Pn+1(x) = n + 1 xPn(x) - n + 1 Pn-1 (x) (5)

C. Interpolation using Warped Distance

In order to interpolate the unknown feature fo in a window in the blotch region, the neighbouring features II, 12, . . . 1I4

are considered as shown in fig 2:

,,-

�,--:

Ne ighbour in g window in

adjace nt frame wit.h

known value

Wmdowtobe

--.� mterpoJated

___ p Nearest ne lghbourmg

wlndow wIth kno,,""'D. value

�,--, , ,

Fig. 2. Selection of neighboring values for warped distance interpolation

Then fo is interpolated as:

fi - L:�11i/dr

0-,,14 l/d2 L.n=1 %

(6)

349

where di is the distance from the considered window to the ith window in pixels. In our method we have modified the distance di according to the pixel value. This is called warping [2]. This approach is based on the evaluation of a warped distance between the pixel to be interpolated and its neighbors. Such a distance is then used in place of the Euclidean distance in the interpolation formula. In order to describe the warped distance approach we consider the one dimensional case as shown in fig 3.

a1 a2

t--'!-'-'-' : : d1

p

"

bz * , . ..'l I •

b ./ : 1 ,/

,,;r' , .

Distance-+-

Fig. 3. Warped distance interpolation

Let f (p) be the value to be interpolated and let its neighbors be located as shown in the figure. The normalized distance for d1 is given by

(7)

Now, the warped distance s' is to be computed. This warped distance permits to take into account the values of the available data and accordingly change the interpolation coefficients [2]. The distance is changed in order to "move" the pixel itself towards those neighbors which are able to yield a visually better estimate for it. In order to achieve the desired effect, the asymmetry of the data in the neighborhood of the pixel under consideration is computed.

(8)

A = ° indicates symmetry. Positive values indicate that the edge is more homogeneous towards the right side and the value to be interpolated belongs to the right object. Opposite holds for A < 0. As a consequence, the desired effect shall be obtained by adding to s a quantity which increases with A. The warped distance is then

s'=s + kAs(l-s)

Hence the modified distances are:

d� = s' (d1 + d2)

and

(9)

(10)

(11)

We then extend this interpolation technique to (x, y) and time directions. This technique is able to reduce the error when interpolating edges.

D. Adaptive Particle Swarm Optimisation

Particle swarm optimization (PSO) is a population based stochastic optimization technique developed by Dr. Eberhart and Dr. Kennedy in 1995, inspired by social behaviour of bird flocking or fish schooling. [3]

PSO is initialized with a group of random particles (solu­tions) and then searches for optima by updating generations. In every iteration, each particle is updated by following two "best" values. The first one is the best solution (fitness) it has achieved so far. (The fitness value is also stored.) This value is called pbest. Another "best" value that is tracked by the particle swarm optimizer is the best value, obtained so far by any particle in the population. This best value is a global best and called gbest. When a particle takes part of the population as its topological neighbors, the best value is a local best and is called lbest.

After finding the two best values, the particle updates its velocity and positions with following equations:

v[] = v[] + Cl * randO * (Pbest -present)

+C2 * randO * (gbest -present) (12)

present[] = present[] + v[] (13)

v[] is the particle velocity, present[] is the current particle (solution). randO is a random number between (0,1). Cl,C2 are learning factors. Usually Cl = C2 = 2. The searching is a repeat process, and the stop criteria are that the maximum iteration number is reached or the minimum error condition is satisfied. In our work we use an Adaptive Particle Swarm optimisation technique [3] which can perform a global search over the entire search space with faster convergence speed by automatic control of inertia weight, acceleration coefficients, and other algorithmic parameters at run time.

The APSO consists of two main steps. First, by evaluating the population distribution and particle fitness, a realtime evolutionary state estimation procedure is performed to iden­tify one of the following four defined evolutionary states, including exploration, exploitation, convergence, and jumping out in each generation. It enables the automatic control of the algorithmic parameters at run time. Then, an elitist learning strategy is performed when the evolutionary state is classified as convergence state. The strategy will act on the globally best particle to jump out of the likely local optima.

E. Proposed method

A novel feature preserving blotch removal method for old movie restoration has been proposed in this paper. After generating the blotch detection mask using the sROD detector, for each non overlapping window in the blotch region, the legendre moment together with the average gray value is in­terpolated from neighboring windows in the non-botch region in the same, succeeding and preceding frames respectively. The warped distance interpolation technique is extended to x,

350

y and also to the time domain. Then, an iterative search is done using the adaptive particle swarm optimisation algorithm for the optimum pixel values in each window. The optimisation condition is that the image features for the optimum pixel values must match with the earlier interpolated ones. This is the followed by post-processing using median filtering.

The mathematical formulation of the algorithm is explained with the help of fig 4.

Region D detected as a Blotch

Pxqwindow

Fig. 4. Mathematical Model

Let Zi = f(Xi, Yi) be the value of the pixel located at (Xi, Yi) in the blotch region D. Let gD = [g1, g2,· .. gm] =

f(z1, Z2, ... zn) be the computed m-dimensional feature vector in D and be the feature vector in D interpolated from surrounding spatio - temporal windows. Then the problem is defined as determining the optimum values of Zl, Z2, ... Zn such that the error E = IlgD - gD11 is minimised. The flowchart of our the proposed method for blotch removal is as shown in fig 5.

RegionD

detected as containing a

blotch

Feature

Extraction

Interpolation of

image features at

regionD

Estimation of blotch region D

such that it's features match the

interpolated features subject to

constrain ts

Fig. 5. How chart of proposed method

III. EXPERIMENTAL RESULTS

The proposed method has been applied to several artificially corrupted as well as original degraded movies. The perfor­mance of this method has been compared with that of an already existing method by Kokaram[8], which uses 3D MMF (multilevel median filtering) to remove blotches. The results have been shown in the following figures. The PSNR(Peak signal to noise ratio) and SSIM(Structural similarity index measure) for each frame has also been evaluated for the two methods. Though our method calls for slightly increased computational complexity, it gives higher PSNR as well as SSIM. The PSNR and SSIM values have been plotted in fig.6 to fig 9.

.. 4O.,1t-'---tt-+t-1rt­i 39., iI----lt--tt-1t-�

Fig. 6. PSNR for all frames of the video with blotch

� 4O.,1t-----'i­� 39.,1/-----"­�""I-----

36., \,------

Fig. 7. PSNR for all frames of the corrected video

Fig. 8. SSIM for all frames of the video with blotch

.. � .,..1--------., i."'�-----I 3''''�-----�.922�-----

Fig. 9. SSIM for all frames of the corrected video

IV. CONCLUSION

This paper presented a novel approach to the problem of removing blotches as a part of old movie restoration. The

351

I _

... '

.

Fig. 10. The figure on top shows three consecutive frames in which the middle is corrupted by blotches. The detected mask and the reconstructed frame are given in the bottom.

Fig. II. The figure on top is an artificially corrupted frame and the detected mask and the reconstructed frame are given in the bottom.

advantages of Legendre moment like minimum redundancy, noise immunity and invariance to rotation and scaling makes it suitable as a feature that can be used for feature matching. The concept of warped distance interpolation has been extended to the time domain, yielding to accurate interpolation at less computational compexity. Experiments using this method on several artificially corrupted as well as original degraded movies show superior performance and accuracy, both per­ceptually as well as objectively. The blotch regions have been filled satisfactorily without introducing any artefacts at the boundary. Median filtering also helps in accelerating the convergence of the APSO algorithm. Two such results have been presented here.The PSNR for the reconstructed frame of an artificially corrupted movie shown in fig 7 has been found out to be 44.02 db.

REFERENCES

[I] Zhang, Xiao-Na, Qi, Guo-Qing, Xu, Rong, Zhang, Tao, An Improved Approach of Detection and Restoration Blotches in Archived Films, CISP09(1-5).

[2] Ramponi G, Warped Distance for Space Variant Linear Image Interpo­lation, IEEE Trans Image Process. 1999;8(5):629-39.

[3] Zhi-Hui Zhan, Jun Zhang,Yun Li, Henry Shu-Hung Chung, Adaptive Particle Swarm Optimization, IEEE Transactions on systems, man and cybernetics, vol. 39, No. 6, Dec 2009.

[4] Mukundan, R., Ong, S.H., Lee, P.A Discrete vs. Continuous Orthogo­nal Moments for Image Analysis, International Conference on Imaging Systems, Science and Technology CISST200I, July, 2001. 2329.

[5] R. Storey, Electronic Detection and Concealment of Film Dirt,(SMPTE), pp. 642647, June 1985.

(6) L. Khriji, M. Gabbouj, S. Marsi, G. Ramponi, and E. D. Ferrandiere, Nonlinear interpolators for old movie restoration,in Proc. Int. Conf. Image Processing, Oct. 1999, vol. 3, pp. 169173.

(7) A. Kokaram, Motion Picture Restoration,New York: Springer, 1998. [8] A. C. Kokaram, On missing data treatment for degraded video and film

archives: A survey and a new Bayesian approach, IEEE Trans. Image Process., vol. 13, no. 3, pp. 397415, Mar. 2004.

[9] A. C. Kokaram, R. D. Morris, W. 1. Fitzgerald, and P. 1. W. Rayner, Interpolation of missing data in image sequences,IEEE Trans. Image Process., vol. 4, no. II, pp. 15091519, Nov. 1995.

(10) M. J. Nadenau and S. K. Mitra, Blotch and scratch detection in image sequences based on rank ordered differences,5th Int.Workshop on Time­Varying Image Processing and Moving Object Recognition, Sep. 1996.

[II) 1. Ren and T. Vlachos, Segmentation-assisted dirt detection for the restoration of archived films,presented at the British Machine Vision Conf., 2005.

(12) P. M. B. van Roosmalen, A. C. Kokaram, and J. Biemond, Fast high quality interpolation of missing data in image sequences using a controlled pasting scheme,in Proc. IEEE Conf. Acoustics Speech and Signal Processing, Mar. 1999, vol. 6, pp. 31053108.

[13] S. Marshalll and N.R. Harvey, Film and Video Archive Restoration using Mathematical Morphology,IEE Seminar Digital Restoration of Film and Video Archives (2001/049)

352