8/6/2019 beta2.0

1/5

Image difference visualization based on mutual

information

J. Blazek

Charles University, Faculty of Mathematics and Physics, Prague, Czech Republic.

B. Zitova

Academy of Science, Institute of Information Theory and Automation, Prague, Czech Republic.

Abstract. We propose algorithm for local image difference measurement formultimodal image data based on value of mutual information of both images.Algorithm works with registered gray-level images. Similarity measure takes intoaccount entropy of compared images. Results of proposed algorithm can be usedfor image comparison and better difference localization.

Introduction

Multimodal data sets are nowadays very common source for wide range of human activities, for allwe can mention MR, CT, SPECT images in medicine or NIR (near infra red) and UV images in restorersworkouts. Multimodality of these resources is temporal as well as based on technology used for capturingsource images. Processing of multimodal images is composed mainly of image comparison (i.e.: pre andpost images of surgery operative or spectroscopy of materials). In many cases image comparison is nottrivial task and can be made only by specialists.

Our goal is to facilitate the work of experts by highlighting parts of image which differ more thanother (regions which carry more relevant information than other). This intention is poorly specifiedproblem due to various requirements of each comparison. So we resign for any robust method givingcrucial results but we propose method providing better overview on data sets.

Previous work

In spite of wide usability of difference metrics image processing field is very poor in algorithmsoffering difference measures. An approach to this problematics we can see in [Ulstadt, 1973] or in [derWeken et al., 2004] where the difference is computed over normalized images. And in many other paperswe could pass metrics based on MMSE (minimal mean square error) i.e. [Moon and Kim, 1994], but thesemetrics are unusable for multimodal data because we cannot assume Gaussian distribution of coloursneither any distribution equal for both input images. Therefore central moments are for our purposesunusable.

Different modalities are used for extra features which are mostly unpredictable from any othermodality. These features are often two colours separable in only one modality, details in one modalityversus colours in second, changed objects in multi-temporal images. By this reason we can also rejectall approximation approaches in tasks where number of scene classes is unknown in any modality.

Our approach goes out from image registration [Viola, 1995] and image fusion algorithms mainlybecause these algorithms operate with multimodal data more sensitively than common difference metrics.

However algorithms for image registration solve different problems (looking for referring points) andimage fusion algorithms often combines multimodal data by different characteristics (i.e.: details andcolours, segments, etc.) in each image.

Usage of multimodal data sets

In the figure 2 we can see joint histogram of two multimodal images. For a sketchy view to multi-modal data we mention two peaks at point [240, 62] and [240, 162]. These points represent two coloursin NIR spectrum but only one colour in visible spectrum. The first goal is highlighting of regions whichcorrespond with two different colors in NIR spectrum but in visible spectrum are inseparable. Thesedifferent NIR regions we dye by green and yellow colour see figure 2b. As can be seen on the dyed image,this comparison disregards any structure of the image, just separates pixels from different NIR classes.

Instead of separating regions (this is task for image fusion), we should look into structure of the

image (edges, segments, object classes) and highlight these parts of the image, where classification differs.

8/6/2019 beta2.0

2/5

BLAZEK ET ZITOVA: IMAGE DIFFERENCE VISUALIZATION

(a) (b)

Figure 1. Input images in two modalities (a) image in near infra red spectrum (wavelength from 800 nmto 1200 nm), (b) image in visible light spectrum (wavelength from 400nm to 800nm)

(a) (b)

Figure 2. Joint histogram of input images from figure 1. Green color channel (yaxis) in NIR andvalue (HSV colour model) in visible spectrum (These spectra was selected because represents best inter-pretation of input image in one band color space) (xaxis), colour values shows number of pixels withcorresponding value (visible spectrum) and green intensity (NIR spectrum) . At coords [240, 62] and[240, 162] we can see peaks inseparable in visible spectrum. (b) coloured regions separable in NIR butinseparable in visible spectrum intensity (peaks from (a)).

However this formulation converts problem of difference specification to problem of relevant segmentationwhich both are poorly specified.

Method overview

We take entropy (information content) as only comparable measure of two multimodal images[Karmeshu, 2003]. The central moments and approximation schemas are unusable as was mentionedabove and approach through classification of segmented image we consider as very specific for concreteproblem. For now we confine to comparison method.

As entropy we take

H(U) =

xU

p(x)logp(x) (1)

where U is our input image and x substitutes intensity in any pixel in the image. Second quantity,mutual information, of two images we define as in Viola [1995]:

I(U,V) = H(U) + H(V) H(U, V) (2)

where U, V are input images. And joint entropy H(X, Y) is based on joint probability:

H(U, V) = yU

xV

p(x, y)logp(x, y) (3)

8/6/2019 beta2.0

3/5

BLAZEK ET ZITOVA: IMAGE DIFFERENCE VISUALIZATION

For our case would be also useful if our difference meets the criteria of metric. Therefore weextend our formula for mutual information so that it is non-negative and symmetric, keeps identity ofindiscernibles and triangle inequality. These metric is also called Variation of Information and is definedin [Meila, 2005]:

V I(U, V) = H(U, V) I(U,V) (4)

In our case, where we want to show difference value in each pixel, we take pixel gain into V I(U, V).For this difference we obtain formula:

V I(x, y) = p(x|U)logp(x|U) +p(y|V)logp(y|V) 2 p(x, y|x U, y V)logp(x, y|x U, y V) (5)

For inaccurate registration of input images is better to use some smoothing filter before registration(on raw data) or take V I based on weighted mean of V Is computed by equation (5) on pixel x andseveral neighbors of pixel y. We use this modified measure V I in presented results. Experimentalresults on synthetic data shows figure 3

(a) (b) (c)

Figure 3. (a) and (b) represents two synthetic inputs with 6 differences, (c) shows graph of V I ofthese two images, in the graph are very good visible differences - colours are transformed intensities tohue value - zero intensity correspond to red colour, maximal intensity correspond to blue colour

Results

Most important disadvantage of variation of information is global view on input data. Our presented

picture was chosen for its various structure in different parts of image (trees, buildings, sky) where resulton V I on whole image is weak (see figure 4) because values of probabilities p(x) are computed overwhole image and contain sum of statistics over different objects.

(a) (b)

Figure 4. Variation of information computed over whole images 1. Image (a) has enhanced contrast,yellow rectangle shows area of image (b) - this image has original values, enhanced version is shown at 5c

Usefulness ofV I we show in figure 5d where only one object is chosen for difference visualization,for comparison we enhance contrast on 4b as can be seen in 5c. Result of our method shows in dark

8/6/2019 beta2.0

4/5

BLAZEK ET ZITOVA: IMAGE DIFFERENCE VISUALIZATION

gray colours regions where is variation of information minimal. Under deeper look onto capital of pillarswe can verify relevance of this metric. Image in figure 5a is blurred and therefore details on capitals areworse than in NIR spectrum (figure 5b). Dark edges around the window also signalize blurred edges onfigure 5a and also improper registration result.

(a) (b)

(c) (d)

Figure 5. Variation of information computed over: (c) - whole images 1; (d) - over small cut of thebuilding shown at image (a) and (b). Image (c) has enhanced contrast and brightness. Domain range ofraw image (c) is 0, 0.314 and for raw image (d) is 0, 0.835 where values are scaled to interval 0, 1

In figure 5c are differences closer to zero, because distribution of colours in whole image is wider(more colours less probability for each) than distribution of colours on building plaster (more peak valuesin colour histogram). Peaks of V I are therefore also wider on images with more different structuresand so relevance of results is lower. For this reason utilization of this metric needs manual correctionsin terms of relevant region selection. In many cases this task needs insight of an expert, which have to

8/6/2019 beta2.0

5/5

BLAZEK ET ZITOVA: IMAGE DIFFERENCE VISUALIZATION

decide on input region selection.

Conclusion and future work

Ve propose method for measuring differences in multimodal data without taken into account struc-ture or objects from images. This way we offer alternative for MMSE or MMSE based metrics. Our

algorithm works with one band images.Used method has several problems. First kind of problems is error connected with registration

process of input images. Registration is very often approximate and has error greater than one pixel. Thiserror is propagated into difference metric especially on textured regions. Second problem is noise whichvariates due to used technology and which strongly affects entropy values. Third and most importantproblem is structure of input images which is not respected. Values of local V I vary due to global contextof the image.

While first and second kind of problems can be solved by using good preprocessing algorithms(denoising, undersampling, etc.) third kind cannot be solved easily. In spite of this weaknesses ourmethod is useful for all kinds of multimodal images where method robustness should be extended byappropriate choice of input data and expert insight into the image structure.

Our future work will focus on image structure problems which are closely associated with segmen-tation. We want to keep low level of complexity of algorithm by the reason of good suitability forusers.

References

M. S. Ulstadt. An algorithm for estimating small scale differences between two digital images. PatternRecognition, 5:323333, 1973.

Dietrich Van der Weken, Mike Nachtegael, and Etienne E. Kerre. Using similarity measures and homo-geneity for the comparison of images. Image and Vision Computing, 22:695702, August 2004.

Joo-Hee Moon and Jae-Kyoon Kim. On the accuracy and convergence of 2-d motion models usingminimum mse term motion estimation. Signal Processing: Image Communication, 6:319333, August1994.

Paul A. Viola. Alignment by maximization of mutual information. Technical Report 1548, MassachusettsInstitute of Technology, June 1995.

Jawaharlal Karmeshu. Entropy Measures,Maximum Entropy Principle and Emerging Applications.Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2003. ISBN 3540002421.

Marina Meila. Comparing clusterings: an axiomatic view. In ICML 05: Proceedings of the 22ndinternational conference on Machine learning, pages 577584, New York, NY, USA, 2005. ACM.ISBN 1-59593-180-5. doi: http://doi.acm.org/10.1145/1102351.1102424.