E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of...

E.G.M. Petrakis Visual Content 1

Retrieval of Visual Content

Images comprise the vast majority of data in many application domains Remote sensing (NASA, 1 terabyte per

day)AstronomyGeographic Information Systems (GIS)Medicine (CT, MRI, etc.)Criminal investigationTrademark authentication


Images in Multimedia Systems

Images co-exist with other types of data in Multimedia Documentstextattributevideosound


Content-Based Image Retrieval

Descriptions of image content are extracted and stored

Manually: mainly text descriptions DifficultSubjective

Automatically: features from contentComputationally expensiveInexactDomain specific


System Architecture


Design Issues

Feature Extraction (functions)Feature SelectionOrganization of stored information, file

structures, indexingSearch and retrieval strategies

Sequential / Indexed search / Query refinement

Query language: conditional / example queries

User interface design


Image Descriptions

Subjective interpretation of content: means different things to different people

Different features for different applicationsColour is important of out-door image but not

for X-rays, CT, MRI etc.Motion features are sometimes important

(ultrasound)

Different systems for different applications


Levels Representation

Low at pixel level (e.g., intensities, colors)

Intermediate at region level (e.g., region, shape, motion features, motion)

High – Semantic human interpretations (e.g., a class per object or image or domain concepts such as diagnosis …)


Conflicting issues

Dependence on image content, computational overhead and uncertainty increases from low to high level

Selection depends on application, image type, user requirements, query types


Reliability Criteria

Uniqueness Proportionality of variationRobustness against noiseInvariance under translation,

rotation, scalingComputationally efficientContent at various level of detail


Generic Features

Feature vectors of intensity / colortexturespatial relationshipsmotioncombinations of the above

Two kinds of featuresglobal: computed for the entire imagelocal: computed for objects or image

parts


RGB Color Space

Popular hardware oriented scheme

Colors form a unit cube r = R/(R+G+B)g = G/(R+G+B)b = B/(R+G+B)

RGB is good for acquisition and display but not for the perception of colors


Munsell Color Space

Color in cylindrical coordinates Brightness: vertical

axis Hue: angular

displacement Saturation:

cylindrical radius


Color Definitions

Brightness: intensity of color, average intensity over all wavelengths

Hue: proportional to the average wavelength of the color percept

Saturation: amount of white, highly saturated colors have no whitedeep red has S=1pinks have S=0


HSV Color Space

Value

Hue

Saturation

H = undefined for S = 0H = 360 – H if B/V > G/V

)++(31

= BGRV

))(()(2

212

cosBGBRGR

BGRH

),,min(1= ++3 BGRS BGR-


Color in Retrievals

Color Histograms are very commonSimple to compute and compareFor the entire image or for image parts 3D histogram on RGB or HSV space (224 bins!)1D histogram over the 3 primaries (256 bins)

Use HSV histograms: changes in lighting and viewing angles may cause major variations in RGB histograms

Invariant under translation, rotation, viewing angle and scaling


A.Del Bimbo 99

1D histogram


Histogram Comparison

Histogram intersectionQ, I: histograms of a

query and database imageN: histogram bins3D (RGB, HSV)

intersection is defined accordingly

N

i i

N

i ii

Q

QIQIS

1

1),min(

),(

A.Del Bimbo 99

normalized intersection


Reducing Complexity

Reduce number of histogram binsTransform RGB histogram to (rg,by,wb)rg = R – G, by = 2B – R – G, wb = R + G +

BIntensity wb is more coarsely sampled than

rg, by wb (8 sections), rg, by (16 sections)The resulting histogram has 2048 binsReduced sensitivity to variations of

intensity


Reducing Complexity (cont,d)

Clustering detects the K most prominent colors (e.g., K-means)Histogram with K bins (e.g., K=64 or

256)Each bin is the normalized count of

pixels in the cluster


Reducing Complexity (cont,d)

Recognize that only a small number of bins capture the majority of pixelsThreshold to take only the large binsSmall bins are likely to be noisy bins

thus distorting the intersectionDoes not degrade the performance


Distance Function

Certain pairs of bins correspond to perceptually similar colorsIn intersection all bins are compared

independently of each otherDefine new Distance function:

A=(aij) represents bin proximity

aij based on proximity in the L*u*v space

K

i

K

j jijiijt yxyxaQIAQIQID

1 1

2 ))(()()(),(


A.Del Bimbo 99

L*u*v color space


Color Indexing

Color (feature) vector: histogram Problems:

K is large (K=64 or 256)Quadratic complexity of matchingSAMs assume independent attributes

Solution: GEMINIMap to low dimensionality feature space Lower bound distance: Df(I,Q) <= D(I,Q)


Definition of Df(I,Q)

Take some average color value on color space (e.g., R,G,B)average color of image: (Ravg,Gavg,Bavg)=

and

P

i

P

i

P

i

avgavgavg

pBPpGPpRP

BGR

111)(/1),(/1),(/1

),,(

3

1

2)-()-()-(),(i ii

tf yxyxyxQID


GEMINI Approach

Indexing in the 3D color space Df < D(I,Q): see QBIC paper for proof

Map query Q to the same 3D space Search the feature space Clean-up answer set to eliminate

false drops


Texture

Repeative patterns of local variations of intensity

Structural: identify placement rules of structural primitives the less effective approach

Statistical: characterize spatial distribution of intensity in terms of measurementsHaralick, Tamura features etc.


Texture Examples

Ballard and Brown 84


Structural Texture



Statistical Texture



Haralick Features [Haralick 73]

Set of 4 features characterizing the intensity transitions of neighboring pixels in various directions using Gray-Tone Spatial-Dependence (GTSD)

arraysOne GTSD for each pixel neighborhood Neighborhood: pixels in direction θ and

distance d


GTSD Array Pd,θ[i,j]

Counts pixel pairs in distance d having gray levels i, j in direction θOne GTSD for θ=(00, 450,900, 1350) and d=(1,2,..)Intensity in range [0,k-1]: Pd,θ[i,j] is a k x k matrix

2 1 2 0 1

0 2 1 1 2

0 1 2 2 0

1 2 2 0 1

2 0 1 0 1

i

j

0 2 2

2 1 2

2 3 2

1

16

P[i,j]d = 1

0 1 2

0

1

2


Computing Pd,θ[i,j]

Count all pairs of pixels in which the first pixel has value i and its matching pair displaced by d=1 in θ = 450 or 1350 direction has value j

Enter this count in the (i,j) position of Pd,θ[i,j]E.g., there are 3 pairs [2,1], then P[2,1] = 3Pd,θ[i,j] is not symmetric: Pd,θ[i,j] < > Pd,θ[j,i]

Normalize Pd,θ[i,j] by the total number of pairs

Pd,θ[i,j]: probability mass function


Texture Features

1) Angular Second Moment (ASM):

Small values for non homogeneous regions

2) Contrast: Large values for many large transitions

or for many transitions

1

0

1

0

21 ),(

k

i

k

jjipf

1

0

1

0

22 )](,[

k

i

k

jjijipf


Texture Features (cont,d)

3) Correlation:

Frequency of intensity transitions

1

0y

1

0j

2y

1

0

1

1x

1

0

2x

1

0

1

0

1

03

],[ p ,][)( σ ,][

],[ p , ][)( σ ,][

],[

k

i

k

yy

k

j yy

k

j

k

i xx

k

i xx

yx

yx

k

i

k

j

jipjpjjjp

jipipiiip

jiijpf


Texture Features (cont,d)

4) Entropy:

High values for uniform p[i,j] i.e., no preferred gray-level (no texture)

A vector for each Tθ,d=(f1,f2,f3,f4) or A vector for every θ, d taking all Tθ,d in a

sequence Correlated features: apply K-L to de-

correlate and to reduce dimensionality

1

0

1

04 ],[log],[k

i

k

jjipjipf


Shape

Assume that objects are extracted Requires image segmentationDifficult problem

Criteria for reliable shape recognitionUniqueness of representationRobustness against noise and distortionProportionality of variationInvariance under scale, rotation and translationEfficiency of computationOcclusion: handle partially visible shapes


Shape Matching Methods

Two categories of methods based on: Regions: represent and match

properties of regionsContours: represent and match

properties of boundariesTechniques: local/global, model

based, fuzzy, statistical, neural networks


Input/Output

For any two shapes and compute:Their distanceThe correspondences between similar parts

Petrakis 02


Moment Invariants

An object is represented by its binary image

A set of 7 features can be defined based on central moments

00

01

00

10

),( m

my ,

m

mxyxm

Ryx

qppq

R

Ryx

qppq yyxx

),(

0,1,2...qp, ),)((


Central Moments [Hu 62]

Invariant to translation and rotation Use ηpq=μpq/μγ

00 where γ=(p+q)/2 + 1 for p+q=2,3… instead of μ’s in the above formulas to achieve scale invariance


More Shape Methods Moments can also be defined on the closed

bounding contours of objects [Gupta and Shinath 87]

Moments can also be defined for open curves [Koch and Kashyap 89]

Methods based on the Fourier Transform of the bounding contour have also been used [Wallace and Wintz 80, Rauber and Steiger 92]

More efficient methods has also been proposed [Petrakis, Diplaros and Milios 2002]. Examines many of the above methods based on Fourier and Moments and shows many experiments and comparisons


Spatial Relationships

Find images showing similar objects in similar spatial relationships find X-rays similar to Smith’s examinationfind images showing a tree close to a houseone of the two images may contain extra objects

Q I Petrakis02


Methods

Two main categories of methods Spatial projections (2D strings and

variants like 2D C strings, Expanded 2D strings etc).

Attributed Relational Graphs (ARGs)Image distance is defined

accordinglyEditing distance on ARGs2D string matching


Image Segmentation

All methods assume segmented images image are segmented manually or semi-

manuallyimage segmentation is a difficult problem

Petrakis02


Image Features

Individual objects: 5-dimensional vectorsSize: number of pixels in a regionPerimeter: length of bounding contourRoundness: ratio of smallest/largest second

momentorientation: angle with x direction (sin,cos)

Spatial Relationships: 4-dimensional vectorsPosition: inside or outsideDistance: minimum distance of contoursOrientation: angle with x (sin,cos) of c.g.’s


Attributes Relational Graphs (ARGs)

Objects are represented by nodes

Relationships are represented by arcs

Nodes and arcs are labeled by feature vectors

Matching: ARG editing distance, Hungarian [Petrakis 02]

Petrakis02


ARG Editing Distance

Matching: sequence of edit operations that transform a query Q to an image IEdit operation: node or arc insertion,

deletion or substitution

F combines the costs of edit operationsf is the cost of an edit operation defined

as a vector distance

)(),...(),(min

))((min),(

11)(

)'(

fffF

ISFIQD

kkIS

GS


Matching Algorithm [Messmer95]

Find the sequence of edit operations that yield the minimum total cost

Formulated as tree search problemExpand all possible matching sequencesBranch and bound Tree node: matching of ARG nodeTree arc: matching of ARG edgesSubtree: matching of subgraphs of Q, I


Query Q Model I


Hungarian Method [Petrakis 02]

Matching: assignment problem

The relationships are ignored

F: cost of a mappingC(i,F(i)): vector

distance

n

iFF iFiCIQD1

))(,(min),(


2D String [Chang 87]

2D string: projections of c.g.’s along x and y Each object is represented by a name or class

Matching: string matching (type 0,1 and 2)


Discussion [Petrakis 02] The ARG editing distance is the most

accurate method followed by Hungarian and 2D strings

2D strings is the faster method followed by Hungarian and ARG distance

Speed and Accuracy are traded-off: the most accurate a method the slower it is

Indexing: Petrakis 2002, Petrakis & Faloutsos 97 (ARGs), Petrakis 93 (2D strings)

http://www.ced.tuc.gr/~petrakis/publications/publications.htm





Image Segmentation

All methods assumed segmented imagesSegmentation is the process of partitioning

an image into groups of connected pixels (regions) with similar properties Gray levelsColors TexturesMotion characteristics (motion vectors)Edge continuity


Segmentation Methods

Two approaches Region segmentationEdge segmentation

Regions may correspond to objectsNot always perfect (noise, bad

illumination, 3D world etc.)Further reading: "Machine Vision'', R.

Jain, R. Kasturi, B. G. Schunck, Mc Graw-Hill, 1995


Region Segmentation

Converts a gray-level image into a binary one by applying carefully selected thresholds on intensity histograms

The image is partitioned into two setsBlack pixels: objects White pixels: background


Histogram Thresholding

The threshold distinguishes the objects from the background

The objects have similar gray-level values


Thresholding

Find thresholds automatically by analyzing the gray value distribution (histogram) of the image

Objects are dark against a light backgroundTheir gray-value distributions can be separated

putting thresholds between them

Automatic thresholding is based on peackiness and valleyness measurements at each point of the histogram


Further Reading C. Faloutsos et.al. “Efficient and Effective Querying by Image Content

”, Journal of Intelligent Information Systems, Vol. 3, No. ¾, pp. 231-262, 1994

M. Flicknet et.al. “Query by Image and Video Content: the QBIC Systems”, IEEE Computer, Vol. 28, No. 9, pp. 13-32, Sept. 1995

R.C.Veltkamp and M.Tanase “Content-Based Image Retrieval Systems: A Survey”, TR UU-CS-2000-34, Utrecht University, March 2001

A.W.M.Smeulders et.al., “Content Based Image Retrieval at the End of the Early years”, IEEE Transactions on PAMI, 22(12): 1349-1380, 2000

R.Schettini, et al. “A Survey on Methods for Colour Image Indexing and Retrieval in Image Databases” , in: R.Luo and L.MacDonald (Eds.), Color Imaging Science: Exploiting Digital Media, John Wiley, 2001

M. Swain, D.H.Ballard, “Color Indexing”, Intern. Journ. of Comp. Vision, Vol. 7, No. 1, pp. 11-32, 1991

D. Androutsos et.al. “A Novel Vector-Based Approach to Color Image Retrieval Using a Vector Angular-Based Distance Measure”, Comp. Vision and Image Understanding, Vol. 75, No. ½, July/Aug. 1999, pp. 46-58.


References J.R.Smith, S-F.Chang, “Tools and Techniques for Color Image Retrieval”,

IS&T/SPIE Proc., Vol. 2670, Storage and Retrieval for Image and Video Databases IV

R.M. Haralick, K. Shanmungam, I. Dinstein “Textural Features for Image Classification”, IEEE Trans. on Systems Man and Cybernetics, 1973, pp. 610-621.

E.G.M. Petrakis, A. Diplaros and E. Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Trans. on PAMI, Vol. 24, No. 11, Nov. 2002, pp. 1-16.

M.-K. Hu. Visual Pattern Recogn. by Moment Invariants. IRE Trans. on Info. Theory, IT-8:179–187, 1962.

T. P. Wallace and P. A. Wintz. An Efficient Three-Dimensional Aircraft Recognition Algorithm Using Normalized Fourier Descriptors. Computer Graphics and Image Processing, 13:99–126, 1980.

T.W. Rauber and A.S. Steiger-Carcao, “Shape Description by UNL Fourier Features – An Application to Handwritten Character Recognition, 11th IAPR Intern. Conf. on Pattern Recogn., 30.Aug.-3.Sept. 1992, The Hague, The Netherlands (click here for implementation).


References L. Gupta and M.D. Shrinath, “Contour Sequence Moments for the

Classification of Closed Planar Shapes”, Pattern Recognition, Vol. 20, No. 3, pp. 267-272, 1987

M.W.Koch and R.L.Kashyap, “Matching Polygon Fragments”, Pattern Recognition Letters, No. 10, pp. 297-308,1989.

Euripides G.M. Petrakis, Aristeidis Diplaros and Evangelos Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, November 2002, pp. 1-16.

Euripides G.M. Petrakis, Aristeidis Diplaros and Evangelos Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, November 2002, pp. 1-16.

Euripides G.M. Petrakis: "Fast Retrieval by Spatial Structure in Image DataBases", Journal of Visual Languages and Computing, Vol. 13, No. 5, October 2002, pp. 545-569.

Euripides G.M. Petrakis: "Design and Evaluation of Spatial Similarity Approaches for Image Retrieval", Image and Vision Computing, January 2002, Number 1, Volume 20, pp. 59-76.

Euripides G.M. Petrakis and Christos Faloutsos: "Similarity Searching in Medical Image Databases", IEEE Transactions on Knowledge and Data Engineering, Vol. 9, No. 3, pp. 435-447, May/June 1997.

http://www.ced.tuc.gr/~petrakis/publications/pedimi02.pdf




http://www.ced.tuc.gr/~petrakis/publications/petr02.pdf

http://www.ced.tuc.gr/~petrakis/publications/petr02.pdf

http://www.ced.tuc.gr/~petrakis/publications/petr01b.pdf

http://www.ced.tuc.gr/~petrakis/publications/pefa97.pdf

E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of...

Documents

Transcript of E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of...