Faculty of Computational Mathematics and Cybernetics, Moscow State University

NUS, Singapore, August 14NUS, Singapore, August 14, 2003, 2003

Andrey S. KrylovAndrey S. Krylov

Outline:Outline:• Projection Method (Hermite series approach)Projection Method (Hermite series approach)• ApplicationsApplications

1. Image filtering and deblocking by projection filtering

2. Image matching 3. Texture matching4. Low-level methods for audio signal

processing

The proposed methods is based on the features of Hermite functions. An expansion of signal information into a series of these functions enables one to perform information analysis of the signal and its Fourier transform at the same time.

Hermite transformHermite transform

0

4 29

nn

n i ˆ

),(2 L

n

xn

n

xn

n dx

ed

n

ex

)(

!2

)1()(

22 2/

B) They derivate a full orthonormal in system of functions.

The Hermite functions are defined as:

A)

Original image

2D decoded image by 45 Hermite functions at the first pass and 30 Hermite functions at

the second pass

Difference image(+50% intensity)

Original image

2D decoded image by 90 Hermite functions at the first pass and 60 Hermite functions at

the second pass

Difference image(+50% intensity)

Image filtering and deblocking by projection filtering

Original lossy JPEG image

Enhanced image

Difference image (Subtracted high frequency information)

Original image

Zoomed in:

Enhanced image

Scanned image

Enhanced image

Zoomed in:

Scanned image

Enhanced image

Image matching

Information parameterization for image database retrieval

AlgorithmImage normalizingGraphical information parameterization

Parameterized image retrieval_

Information parameterization for image database retrieval

AlgorithmImage normalizingGraphical information parameterization

Parameterized image retrieval

= +HF component LF component Normalized

image

Recovered image with the recovered image plane

Information parameterization for image database retrieval

AlgorithmImage normalizingGraphical information parameterization

Parameterized image retrieval

Database size Initial images formatNormalized images formatNumber of parameterization coefficients Error rate Search time

– 768 images (4.12Gb)– 1600x1200x24bit (5.5Mb)– 512x512x24bit (0.75Mb)– 32x32x3– <0.14%– 4 sec. (for K7-750)

Image matching results

Image matching results

Texture matching

A method of obtaining the A method of obtaining the texture feature vectorstexture feature vectors

0

)()(i

ii xxf

dxxfxii )()(

),()(),(2121

yxyx nnnn

1)(),( xyx nn

Input function

Fourier coefficients

1-D to 2-Dexpansion

A method of obtaining the A method of obtaining the texture feature vectorstexture feature vectors

1-D Hermite functions:1-D Hermite functions:1-D to 2-D expanded 1-D to 2-D expanded

Hermite functions:Hermite functions:

OrientationsOrientations

Localization problemLocalization problem

Decomposition process is optimal, if localization segments of the input function and filtering functions are equal.

Feature vectorsFeature vectors

In this approach to get the feature vectors we consider the functions n(x,y) where n1=0..64, and 6 energy

coefficients are calculated as:E1 = (0)

2 + (1)2 ,

E2 = (2)2+(3)

2 + (4)2,

E3= (5)2 +(6)

2+(7)2+(8)

2,

…E6 = (33)

2+(34)2 + … + (63)

2 +(64)2 ,

f(x,y) is the source image.

Standard codingStandard coding

Feature vectorsFeature vectors

E1 = (0(1))2 + (1

(1))2 ,

E2 = (0(2))2+(1

(2))2 + (2(2))2 +(3

(2))2 ,

…E6 = (0

(6))2+(1(6))2 + … + (62

(6))2 +(63(6))2 ,

 

dxyxfyxdyA

i

j

i ),(),(1)1(

1,)),(),((),(1

2

)1()(

jdxyxfyxfyxdyA

j

l

li

j

ji

f(x,y) is the source image.

Hierarchical codingHierarchical coding

Feature vectorsFeature vectors

E1 = (0(1))2 + (1

(1))2 ,

E2 = (0(2))2+(1

(2))2 + (2(2))2 +(3

(2))2 ,

…E6 = (0

(6))2+(1(6))2 + … + (62

(6))2 +(63(6))2 , 

 

dxyxfyxdyA

i

j

i ),(),(1)1(

1,),(),(1)(

jdxyxfyxdyA

i

j

ji

f(x,y) is the source image.

Hierarchical coding without subtractionsHierarchical coding without subtractions

Feature vectorsFeature vectors Texture

sample Standard coding

Hierarchical coding

Hierarchical coding without

subtractions A1

A2

A3

A4

B1

B2

B3

B4

C1

C2

C3

C4

Feature vectorsFeature vectors

a b Standard coding

Hierarchical coding

Hierarchical coding without

subtractions

a1 a2 0.004505 0.005349 0.003739

b1 b2 0.009905 0.008160 0.007742

c1 c2 0.017778 0.011848 0.009946

a3 a4 0.008807 0.006047 0.002422

b3 b4 0.020075 0.010667 0.006275

c3 c4 0.128322 0.101591 0.080176

a1 b1 0.488420 0.492238 0.491653

b1 c1 0.315644 0.304597 0.305442

Image segmentation taskImage segmentation task::Brodatz texturesBrodatz textures

Image segmentation taskImage segmentation task

 

Image segmentation taskImage segmentation task

Image segmentation taskImage segmentation task

Texture parameterization Texture parameterization using 2-D Hermite functionsusing 2-D Hermite functions

),(),,(),,(),,(),,(),,(),,( 4,35,27,00,31,22,13,0 yxyxyxyxyxyxyx

Quasiperiod’s waveforms

Signal filteringSpeaker indexingSpeaker recognition using databaseSource separation

Areas of Hermite transform application:Areas of Hermite transform application:

Music Speaker 1 Speaker 2 2 speakers

Audio sample

Quasiperiod waveform Hermite histogram

Speaker indexingSpeaker indexing

Speech/music separationSpeech/music separation

Analysis of signal dynamics (distribution of amplitudes in time)

Analysis of base tone distribution in time

Currently implemented criterions:

Result of speech/music separation:

Mix detectionMix detection

One speaker Mix

ReferencesReferences1.1. Krylov A.S., Liakishev A.N.,Krylov A.S., Liakishev A.N., Numerical Projection Method For Numerical Projection Method For

Inverse Fourier Transform and its Application,Inverse Fourier Transform and its Application, Numerical Numerical Functional Analysis and optimization, vol. 21 (2000) p. 205-216.Functional Analysis and optimization, vol. 21 (2000) p. 205-216.

2.2. Krylov A.S., Kortchagine D.N.,Krylov A.S., Kortchagine D.N., Projection filtering in image Projection filtering in image processingprocessing, , Graphicon’2000 Conf. proceedings, p. 42-45, 2000.Graphicon’2000 Conf. proceedings, p. 42-45, 2000.

3.3. Krylov A.S., Kortchagine D.N., Lukin A.S., Krylov A.S., Kortchagine D.N., Lukin A.S., Streaming Waveform Streaming Waveform Data Processing by Hermite Expansion for Text-Independent Data Processing by Hermite Expansion for Text-Independent Speaker Indexing from Continuous SpeechSpeaker Indexing from Continuous Speech, , Graphicon’2002 Conf. Graphicon’2002 Conf. proceedings, p. 91-98, 2002. proceedings, p. 91-98, 2002.

4.4. Krylov A.S., Kutovoi A.V., Wee Kheng Leow, Krylov A.S., Kutovoi A.V., Wee Kheng Leow, Texture Texture Parameterization With Hermite Functions,Parameterization With Hermite Functions, Graphicon’2002 Conf. Graphicon’2002 Conf. proceedings, p. 190-194, 2002.proceedings, p. 190-194, 2002.

5.5. Mohsen Najafi, Andrey Krylov, Danil Kortchagine Mohsen Najafi, Andrey Krylov, Danil Kortchagine Image Image Deblocking With 2-D Hermite Transform, Deblocking With 2-D Hermite Transform, Graphicon’2003 Conf. Graphicon’2003 Conf. proceedings. proceedings.