A MINIMUM-ENTROPY PROCEDURE FOR ROBUST MOTION ESTIMATION

Sylvain Boltz, Eric Wolsztynski, Eric Debreuve, Eric Thierry, Michel Barlaud, Luc Pronzato

Laboratoire I3SEuclide B, Les Algorithmes

2000 route des Lucioles06903 Sophia Antipolis cedex

France

ABSTRACT

We focus on motion estimation using a block matching approach andsuggest using a minimum-entropy criterion. Many entropy-based es-timation procedures exist, such as plug-in estimators based on Par-zen windowing. We consider here an alternative that is applicable todataof any dimensionand that circumvents the critical issues raisedby kernel-based methods. To the best of our knowledge, this criterionhas not yet been considered for image processing problems. The in-herent robustness property of entropy is expected to provide a robustand efficient estimation of the motion vector of a block of a videosequence. In particular, the minimum-entropy estimator should berobust to occlusions and variations of luminance, for whichstandardapproaches like SSD usually meet their limitations.

Index Terms— image matching, minimum entropy methods,motion compensation, adaptive estimation, image processing, robust-ness

1. INTRODUCTION

In this paper we focus on a block matching (BM) techniquefor motion estimation in video sequences. Matching techniques suitmany purposes, such as segmentation [2, 11, 10], registration [14],or tracking [5]. The BM strategy provides a good trade-off betweencomplexity and efficiency. The focus in a BM procedure can be seton the search strategy or on the criterion used to define the best matchbetween two blocks. Here we are concerned with the latter andinparticular with the performances of the motion estimation procedurein the presence of degradations and/or of variations of luminancethroughout the sequence.

Among the most standard frame-based BM criteria are the sumof squared differences (SSD) and the sum of absolute differences(SAD) between two blocks. These two criteria are praised fortheircomputational simplicity and their performance in terms ofdistor-sion. However, they do not take into account the nature of theimageand the information contained in a frame, and are not statisticallyefficient in a general context (for instance the SSD criterion is notefficient in the presence of non-gaussian errors). Moreover, they arequite sensitive to large (local) deviations from one frame to the next.

In this paper we consider an alternative approach that consists inminimizing (a nonparametric estimate of) the entropy of thedispla-ced frame difference (DFD). The entropy of an image is a measureof the amount of information it contains, and is a function ofits pro-bability distribution. Hence, if one minimizes the entropyof the re-sidual between two images, one maximizes the match between thesetwo images in terms of information. In general terms, the minimum-

entropy criterion is (widely) considered in many contexts of signalor image processing, see e.g. [16, 7, 11, 10]. Here, we consider thenew minimum-entropy scheme presented in [17] for image proces-sing that leads to improved motion estimation, in particular for mul-tivariate data.

In Section 2.2.2 we present the minimum entropy principle ap-plied to motion estimation using BM. The most popular entropy es-timator is a plug-in construction that uses a non-parametric Parzen-type kernel estimator of the density of the image. However, this plug-in approach raises some issues, which are identified in Section 2.2.3.In Section 2.3 we suggest using an alternative entropy estimator pre-sented in [17] and based on thek-th nearest neighbor approach. Thecriterion we obtain can be optimized with fast-search algorithms likethe diamond search. Some simulation results are presented in Sec-tion 3 to illustrate the performance of the proposed motion estimator(in particular its robustness). Some conclusions and perspectives areexposed in Section 4.

2. MINIMUM-ENTROPY MOTION ESTIMATION

2.1. Problem statement

Consider two successive framesIn andIn+1 of a given videosequence. We formulate the BM procedure as follows, leavingasideproblems that might arise at the frame boundaries. For a given pixelp = (x, y) (in terms of spatial coordinates), letv = (v1, v2) be the(unknown) motion vector describing the movement ofp betweenIn

andIn+1. Under the brightness consistency assumption, the residualat pixelp in the DFD is given by

en(p, v) = In+1(p+ v) − In(p) . (1)

Let us now decompose each frame intoM blocks of sizeS × S,containingN = S2 pixels. For them-th blockBm

n in In, let vm bethe motion vector common to all pixels ofBm

n , thus assuming thatall points in the block have the same motion. Given an estimatedvaluev̂m of vm, we denote the value of the estimation error by

emn (p, v̂m) = In+1(p+ v̂m) − In(p), ∀ p ∈ Bm

n . (2)

2.2. Minimum-entropy estimation

2.2.1. Limitations of classical methods

Classical BM procedures for motion estimation use the SSD orSAD criteria. The SSD estimator is fully efficient only in thecaseof normally distributed data, see e.g. [9]. However, the Gaussianassumption is not appropriate in motion estimation problems. Here

we consider instead minimizing the entropy of the estimation errorem

n (. , v) in blockBmn of frameIn. The properties of the measure

of entropy motivate its use as an estimation criterion : entropy is aconvex function of the motion vectorv that coincides locally asymp-totically with the likelihood at its optimum [16]. This suggests thatan estimator minimizing the entropy of the errors should be efficient.Also, the shift-invariance property of the entropy of a density yieldssome robustness to outliers, e.g. unexpected patches of pixels in animage [17], and makes it insensitive to a global variation ofillumi-nance between blocks.

2.2.2. The classical Parzen windowing approach

In the present context we consider the (block of) errorsemn given

by (2), viewed as realizations of a random variable. Knowledge (orestimation) of the probability mass function (in the discrete case) orof the probability density function (in the continuous case) of theseerrors is required in order to compute the entropy of the block. Aminimum-entropy approach for discrete data would require alargenumber of data points (pixels) in order to obtain an accurateenoughestimation of the distribution, which makes it unsuitable for BM pur-poses. Instead we consider turning the DFD into an image of conti-nuously distributed pixel values. In what follows, a uniform noiseU(−.5; .5) is added to the DFD (this choice is arbitrary, other typesof distributions could be used). In this context, a common pdf estima-tion procedure consists in using Parzen windowing technique, whichprovides a smooth estimate that can be plugged into an empirical ex-pression of the Shannon entropy. Minimum-entropy estimation givesa consistent estimator of the parameters in a regression model withunknown distribution of the observations errors, see [16].

Consider them-th block in In. The minimum-entropy estima-tor of the motion vectorvm of Bm

n that minimizes the Ahmad-typeplug-in entropy estimate [1] is defined by

v̂Am = arg min

v∈V−

1

N

Xp∈Bm

n

log f̂mN,σ

N(em

n (p, v)) . (3)

Here, f̂mN,σ

N

denotes the Parzen kernel density estimate based ontheN pixels of them-th block of errorsem

n (. , v), which dependson the parameterv. The kernels have a bandwidthσ

Nset either by a

data-driven procedure [17, 15, 6, 4] or by the user.

2.2.3. Limitations of Parzen windowing

In dimensions larger than 1, e.g. for color images, kernel esti-mation techniques rapidly become inefficient when the dimensionincreases. The main difficulty lies in the choice of the bandwidthσ

N: due to the curse of dimensionality, the bandwidth of a multi-

variate kernel must be large enough to take a sufficient number ofdata points into account, which causes oversmoothing. Thisleadsto a degradation of the performances of the estimator as defined by(3) for data having dimension 2 or 3 and for samples of reasonablesize. For 3D (i.e. color) images, the alternative of using a product ofunivariate kernels still remains relatively computationally costly anddoes not perform well enough in general. Other entropy estimatorsuse possibly negative kernel density estimates, which are inadequatefor entropy estimation, or kernel methods with variable bandwidth[15, 6]. We consider now a special case of the latter.

2.3. Thek-th nearest-neighbor approach

Entropy can be estimated throughk-th nearest neighbors (kNN),as considered in [12] fork = 1. The kNN entropy estimator is ex-

tended tok > 1 in [8] and its consistency is proved in [13] in ge-neral dimensiond under weak conditions on the underlying density.Consider the pixels from them-th block of the DFD,em

n (p, v) ∈ Rd

given by (3). Denote the Euclidean distance from one of thesepointsto its k-th nearest neighbor byρp, k(vm). The kNN minimum en-tropy estimator ofvm then minimizes

Hk, N(v) = d log ρ̄k(v) + T (N, k) , (4)

whereρ̄k(v) =�Πp∈Bm

nρp, k(v)

�1/Nis the geometric mean of the

kNN distances andT (n, k), given by

T (N, k) = log (N − 1) − ψ(k) + log c1(d) ,

with c1(d) = 2πd/2/(dΓ(d/2)) andψ(k) = Γ′(k)/Γ(k). NoticethatT (N, k) does not depend onv. The constantc1(d) gives the vo-lume of thed-dimensional unit ball andψ(k) is the digamma func-tion. One should setk > dim(vm) to avoid singularities. Noticethat in the image processing context the distancesρ may be equalto zero when the pixel values are discrete (even for a large enoughk and relatively large images), which motivates the use of additivecontinuous noise, as discussed in Section 2.2.2. Notice also that (4)is not differentiable inv and is therefore not suitable for gradient-descent approaches. Here we use the (sub-optimal) diamond searchalgorithm, but any other deterministic search method is applicable.This criterion was first applied to an illustrative BM problem in [17],where it was compared in particular to the SSD and SAD criteria.The simulation results that were obtained suggest that the kNN ap-proach is more robust to outliers and to variations of luminance, evenmore so for color images, for a negligible increase in complexity.The results presented hereafter illustrate for the first time its perfor-mance on a real application.

3. EXPERIMENTAL RESULTS

In this section the minimum kNN entropy (kNN-ME) procedureis compared to SSD, SAD and the Ahmad-type minimum entropy(piME) estimate given by (3), for which the kernel bandwidthis gi-ven byσ

N= 2.345σ̃

NN−1/5, whereσ̃

Nis the estimated standard

deviation for theN data points, see e.g. [4]. In the following, for co-lor images (i.e. for 3D data), the piME estimator is constructed usinga product of univariate kernels on each channel component. For co-lor sequences the SSD and SAD criteria are given by the sum of theweighted SSD’s or SAD’s of each component (the Y or luminancechannel being four times heavier than the color channels U and V).We setk = 3 for the kNN-ME estimators.

We consider the mean and standard deviation of the angular errorcompared to the ground truth (i.e. the true value of the motion vec-tors) when it is available, as in [3]. Motion estimation is performedwith a fast sub-optimal BM technique : diamond search, at quarter-pixel precision on16x16 blocks,(−7; 7) search range, on the firsttwo frames of the sequences.

3.1. Application to a synthetic sequence

Let us first compare these different procedures on the synthe-tic video sequenceTranslating Tree. This sequence is standard forevaluation of the performance of optical flows [3].

On this basic sequence, from a visual point of view SAD andSSD fail to correctly estimate the motion vectors of two blocks,piME fails only once and kNN-ME seems to commit no mistake.Indeed, we assume that it is not possible to get better results withBM at quarter-pixel precision as the ground truth is a dense motion

real field. We intentionally omitted to display these results here asthe format does not allow a good reading of them. They are howe-ver available upon request. Table 1 contains the average error and itsstandard deviation for the four 1D criteria. kNN-ME seems toper-form better, while piME and SAD are quite close on average. SSDseems to be less interesting on this sequence.

Criterion SAD SSD piME kNN-MEAverage error 0.75 0.94 0.77 0.56

Standard Deviation 1.62 3.04 1.25 0.79

Table 1. Motion flow error onTranslating Tree: comparison withthe true values of the motion vectors (ground truth)

3.2. Natural difficulties

In this section we compare 3D-SAD and kNN-ME on real stan-dard sequences that present typical difficulties for BM motion es-timation. For all figures given in this section, the image on the lefthandside gives the results obtained with SAD while the imageon theright handside shows the output obtained with kNN-ME.

3.2.1. Occlusions

We first consider theErik sequence, which consists in a headmoving on a static background. We test the methods on a close-upon blocks around the left edge of the face as shown in Figure . Bet-weenIn andIn+1, the face (foreground) occludes the curtain (back-ground). Here SAD does not find the correct match for the blockfrom In (the mid-left block in Figure 1, left) and tries to match itfurther downwards. kNN-ME performs well since it finds no motionin the background (there are no motion vectors for the left blocks inFigure 1, right).

95 100 105 110 115 120 125 130 135 140 145

120

125

130

135

140

145

150

155

160

95 100 105 110 115 120 125 130 135 140 145

120

125

130

135

140

145

150

155

160

Fig. 1. Occlusions : close-up ofErik processed with SAD (left) andkNN-ME (right)

3.2.2. Highly textured images

Here we focus on the highly-textured public in the backgroundof the standardEdbergsequence. Motion estimation on a highly tex-tured image is very sensitive to even slight errors in the motion vector(which happen naturally for instance for non-uniform translations ormotion vectors quantizations) and to suboptimal motion search, asthe common criteria are non-convex. Figure 2 illustrates how SADfails to find a uniform motion in the public (left) whereas kNN-MEgives satisfaction (right).

Other significant improvements can be observed for other kindsof difficulties, such as motion blur, variations of illuminance, or dif-ferent motions in a same block. These results are available upon re-quest.

30 40 50 60 70 80

5

10

15

20

25

30

35

40

45

50

30 40 50 60 70 80

5

10

15

20

25

30

35

40

45

50

Fig. 2. High texture : SAD (left) and kNN-ME (right) applied toEd-berg; close-up of the public

3.3. Some extreme robustness experiments

Let us now consider a synthetic color sequence, for which theground truth is known, in order to perform exact error measure-ments. The YUV color sequence considered here consists in thesuperposition of two images : an horizontal translation of aface(Foreman’s) towards the left handside and a background (a viewof Boston) travelling also horizontally but towards the right hand-side. There is no other movement in the scene ; for instance the facedoes not move. The main difficulty with this sequence residesinfrequent occurrences of occlusions (at the edges of Foreman’s faceor on the borders of the image) and in the texture of the objectandthe background. As the sequence considered here is a pure travelling,there are not enough occlusion problems to illustrate the robustnessproperty of the kNN-ME estimator. We thus consider the same se-quence, that we nameOriginal, but randomly altered as shown inFigure 3 by the addition of

– Flash: variations of brightness between two successive frames(like a camera flash) of40 units of luminance ;

– Noise: "salt and pepper" noise covering10% of a frame ;– Patch: black patches (similarly to scratches on a film) ;– Altered: the 3 previous noises all at the same time, but with5

units of variation of luminance instead of40.

Fig. 3. The "Foreman-Boston" sequence altered with (clockwise fromtop-left) patch, flash, noise and all three together

The results in terms of mean and standard deviation of the angularerror are shown in Table 2. In addition to applying the SSD, piMEand kNN-ME criteria to 3D data, we also consider their 1D alterna-tive applied to the luminance (Y) channel of the image, as well asthe SAD criterion, as these may turn out to be reasonable alterna-tive estimation procedures in practice. Overall, the (3D) kNN-MEmethod outperforms all other procedures and it seems more robustto the perturbations that were introduced. The piME estimator fails

hard, which only highlights that selection of the kernel bandwidthsis a critical problem in the presence of perturbations (it iseven moreimportant with the 3D version). The kNN-ME estimator does notsuffer from these aspects. 1D kNN-ME may provide a reasonablealternative since it is computationally less demanding andprovidesgood results.

Sequence Original Flash Noise Patches Altered

SAD µ 8.9 57.1 14.4 13.7 26.9σ2 28.2 42.9 39.0 33.0 33.7

SSD µ 9.5 45.1 33.3 22.4 44.9σ2 27.7 47.7 36.4 38.7 41.1

piME µ 6.7 7.0 39.8 12.5 44.8σ2 22.1 21.8 40.0 26.8 42.2

kNN-ME µ 4.4 5.0 5.6 4.5 9.7σ2 21.0 21.0 22.2 21.1 28.5

1D-SAD µ 9.1 67.2 15.8 16.91 28.5σ2 27.8 41.6 33.5 35.3 40.5

1D-SSD µ 10.6 48.7 31.3 22.5 43.6σ2 29.4 47.3 34.0 38.5 41.7

1D- µ 9.1 12.8 27.4 17.7 35.1piME σ2 27.1 30.3 34.6 36.5 40.61D- µ 8.8 11.5 9.4 11.0 11.6kNN-ME σ2 27.7 31.3 27.7 30.9 32.5

Table 2. Motion flow error on Foreman-Boston altered with seve-ral methods : comparison with ground truth. Meanµ and standarddeviationσ2 of angular error are given in degrees.

4. CONCLUSION AND PERSPECTIVES

In this paper focus was set on a robust block matching criterionfor motion estimation. Minimizing the entropy of the DFD blockswas suggested ; the natural properties of this measure provide proce-dures that are robust to any additive noise. The kNN minimum en-tropy estimator was presented as an alternative to Parzen windowing.Some experiments using altered video sequences illustratethe signi-ficant improvement provided by the kNN-ME estimator, in terms ofcorrect motion estimation, when compared to classical criteria likeSSD or SAD. Foreseen extensions to this work include applicationsof the suggested procedure to other block or region matchingappli-cations, like image restoration or registration problems,block tra-cking problems, or a full video coding scheme.

5. REFERENCES

[1] I.A. Ahmad and P.E. Lin. A nonparametric estimation of theentropy for absolutely continuous distributions. InIEEE TransInform. Theory, volume 36, pages 688–692, 1989.

[2] G. Aubert, M. Barlaud, O. Faugeras, and S. Jehan-Besson.Image segmentation using active contours : Calculus of varia-tions or shape gradients ?SIAM Applied Mathematics, 1(2),2003.

[3] J.L Barron, D.J Fleet, and S.S Beauchemin. Performance ofoptical flow techniques.International Journal on ComputerVision, pages 43–77, 1994.

[4] A. Berlinet and L. Devroye. A comparison of kernel densityestimates.Publications de l’Institut de Statistique de l’Univer-sité de Paris, 38(3) :3–59, 1994.

[5] D. Comaniciu, V. Ramesh, and P. Meer. Real-time trackingof non-rigid objects using mean shift. InComputer Visionand Pattern Recognition, pages 142–151, Hilton Head Island,South Carolina, 2000.

[6] L.P. Devroye and G. Lugosi. Variable kernel estimates : on theimpossibility of tuning the parameters. In D. Mason E. Ginéand J.A. Wellner, editors,High-Dimensional Probability II,pages 405–424. Springer-Verlag, New York, 2000.

[7] D. Erdogmus and J.C. Principe. An error-entropy minimizationalgorithm for supervised training of nonlinear adaptive sys-tems. InIEEE Transactions on Signal Processing, volume 50,pages 1780–1786, 2002.

[8] M.N. Goria, N.N. Leonenko, V.V. Mergel, and P.L. Novi Inve-rardi. A new class of random vector entropy estimators and itsapplications in testing statistical hypotheses.Journal of Non-parametric Statistics, 17(3) :277–297, 2005.

[9] C. Gourieroux and A. Monfort. Statistics and EconometricModels, volume 1. Cambridge University Press, 1995.

[10] Ariane Herbulot, Stéphanie Jehan-Besson, Stefan Duffner, Mi-chel Barlaud, and Gilles Aubert. Segmentation of vectorialimage features using shape gradients and information mea-sures.Journal of Mathematical Imaging and Vision, 2006, inpress.

[11] Junmo Kim, John W. Fisher III, Anthony J. Yezzi, MüjdatÇetin, and Alan S. Willsky. A nonparametric statistical me-thod for image segmentation using information theory andcurve evolution. IEEE Transactions on Image Processing,14(10) :1486–1502, october 2005.

[12] L. Kozachenko and N. Leonenko. On statistical estimationof entropy of random vector. Problems Infor. Transmiss.,23(2) :95–101, 1987.

[13] N. Leonenko, L. Pronzato, and V. Savani. A class ofRényi information estimators for multidimensional densi-ties. Technical Report I3S/RR-2005-14-FR, 32 pages, 2005.www.i3s.unice.fr/˜mh/RR/rapports.html .

[14] M. Leventon, E. Grimson, and O. Faugeras. Statistical shapeinfluence in geodesic active contour. InComputer Vision andPattern Recognition, pages 1316–1323, Hilton Head Island,South Carolina, 2000.

[15] D.W. Scott. Multivariate Density Estimation : Theory, Prac-tice, and Visualization. Wiley, 1992.

[16] E. Wolsztynski, E. Thierry, and L. Pronzato. Minimum en-tropy estimation in semi parametric models.Signal Processing,85 :937–949, 2005.

[17] E. Wolsztynski, E. Thierry, and L. Pronzato. Minimum entropyestimators in semi parametric regression models. InProc. XIInt. Conf. on Applied Stochastic Models and Data Analysis,pages 882–889, Brest, May 2005.