Survey of Algorithms to Query Image Databases

COMP 290-72:Computational Geometry

Benjamin Lok 11/2/98

Image from Kodak’sPhotoQuilt

Outline of Talk

Overview of the problem Three methods

• Color based• Shape based• Vision based

Conclusions

Image from Microsoft Clip Gallery

Problem

Query an image database

What does a “match” mean?• Application dependent

• Notion of subjectivity

• Sensitivity to noise

Problem

Semantic similarity is still not possible• ex. “All images with cats”

To determine similarity,

we need to a new:• Metric• Space

Images from Microsoft Clip Gallery and website

Ambiguity

Guy

Girl

Guy

Images from “Shrine to Long Haired Men” and “Videos of Women Getting Their Heads Shaved” websites

Yossi Rubner, Leonidas Guibas, Carlos Tomasi (1997)

Stanford Vision Laboratory

The Earth Mover’s Distance, Multi-Dimensional Scaling,

and Color-Based Image Retrieval

Image from Microsoft Clip Gallery

Color Signatures

Utilize the CIE-LAB color space• Based on human perception of color

Map each pixel to a point in color space• Common color values increase weight of point

Group clusters into points (8-12 per image)

Rubner, Guibas, and Tomasi

Earth Mover’s Distance

To compare two images, compute the “work” needed to move the cluster points from one image to the other

Rubner, Guibas, and Tomasi

Earth Mover’s Distance (cont)

Solving a linear programming problem:

Given two signatures:p = {(p1,wp1),…,(pm,wpm)} and

q = {(q1,wq1),…,(qn,wqn)}

Find C where Cij is the amount of weight pi matched qj

),min(1 1

qp

m

i

n

jjiij

ww

qpC

Applications

Visualize Databases (Queries and Results) Scale the multiple dimensions into 2D using

MDS and minimize STRESS

Rubner, Guibas, and Tomasi

Database Visualization

Rubner, Guibas, and Tomasi

Algorithm Recap

Map pixels to 3D color space points Locate and compress “clusters” of points 8 to 12 points determine the color signature Calculate the Earth Mover’s Distance to

determine “distance” between two images

Image from YenPen Stationary Website

Advantages

Based on human perception of color

Some invariance to small change in viewpoint and lighting

Meaningful metric Relatively fast Can embed multiple

metrics

Disadvantages

Depending on application, query format might be not be intuitive

Not much use for non-color images

False positives a real possibility depending on working domain

Shaped-based Image Retrieval Using Geometric Hashing

Scott D. Cohen and Leonidas J. Guibas

1997

Stanford Vision Laboratory

Image from Microsoft Clip Gallery

Overview

Implementation• Search through 500

Chinese characters

Goals• Provide invariance to scale, rotation, and

translation• speed and accuracy

Cohen and Guibas

Generating a Illustration

Illustration - set of curves that summarize an image• Edgel detection• Medial Axis determination

Cohen and Guibas

Approximating with Polylines

Convert medial axis representation to polylines

Tradeoff between speed and accuracy

Cohen and Guibas

Geometric Hashing

Geometric Hashing - method used to compare two point sets under some transformation group

Take each point and use it as the origin of a coordinate system

},1:{)(

},1:{)(

jlnlqqQI

ikmkppPI

jlj

iki

Cohen and Guibas

Geometric Hashing (cont)

If translating P by qj - pi produces a good match Ii(P) and Ij(Q) will match.

This property can be generalized to other transformation groups.

Each line segment is a basis of a coordinate system• Translation, Rotation, and Orientation defined• I(P) = transform all other segments into new CS

Cohen and Guibas

Notes on GH

Each segment will be transformed to 2m Coordinate systems

I(P) stores O(m2) segments Can be done as preprocessing step Expressing the different possible

transformations using each segment as a basis

Cohen and Guibas

Querying the Database

Query image undergoes the

feature extraction process For each query feature, a nearest-neighbor

query is applied and the k closest or within some j

Similarity score increases if database image has a feature that is “close” to the query feature

Cohen and Guibas

“Closeness”

How do you describe the closeness of two lines?

Transform to a 4D space made of (l,,a,b) With two (l,,a,b) descriptions for lines, can

compute distance Divide by standard deviation

over sample of database

features

Details

Closeness is relative to database contents Nearest-neighbor algorithm by Arya,

Mount, et. al (1994). Query time for k nearest features is O(k log n)

Cohen and Guibas

Advantages

Fast• Queries database of

500 characters in 1 second on SGI Indy

Queries based on important features

Disadvantages

Working domain currently limited

Could get too expensive as complexity in images increases

Cohen and Guibas

A Multi-Resolution Technique for Comparing Images Using the Hausdorff Distance

Daniel Huttenlocher and William Rucklidge

1992 Cornell University

Huttenlocher and Rucklidge

Directed Hausdorff Distance

Given A={a1, … , am} and B={b1, … , bm}

Identifies the point in A farthest from any point in B

Measures the degree of mismatch between between two sets.

baBAh

ABhBAhBAH

BbAa

minmax),(

)),(),,(max(),(

Properties of Hausdorff Distances

Not symmetric h(A,B) != h(B,A) Compute kth maximum

• Notion of rank• Reduces sensitivity• Fraction of A within

h(A,B) of B• Obscured portions

h(A,B) = hypothesis

h(B,A) = test

Transformations t( ) =

Given A is an image, B is the model Without Orientation, if A is in B then A

undergoes transformation t.fB(t)=H(t(B),A) t=(tx,ty,sx,sy) forward

fA(t)=H(A,t(B)) reverse

),(min)),(( yyyxxxAaBb

K tbstbsathKABth

Bidirectional Hausdorff Distance

Solve for which values of t,

the following holds:

Results in searching a four

dimensional space

))(,(

))),(()),(,((max))(,(

BtAH

ABthBtAhBtAH

LK

KLLK

Restricting Search Space

Slope of f(t)=HLK(A,t(B))

is linear Divide space into cells Calculate HLK(A,tc(B))

Determine a maximum delta per cell • Based on limit in scale and translation• Allows for quick rejection and acceptance

Label cells as interesting or disregard

Restricting Search Space

Create smaller cells from interesting cells Bounds based on

transformations Quickly narrow

down to areas that

could possibly be

within of A

Subtleties

Discretization useful if working in computer vision domain (integers)

Can compare partially obscured images Optimizations

• Early rejection/acceptance

Pretty slow 200 to 250 seconds

Website on submarines

Huttenlocher and Rucklidge

Advantages

Accurate Geared towards

image processing and vision

Partially obscured images

Searches similar to humans

Disadvantages

Slow No Orientation Database must be

specialized Potential problems in

generating queries

Recap

Three Algorithms• Color Based

• Color Signatures

• Earth Mover’s Distance

• Shaped Based• Polylines

• Transform Invariant Sets

• Vision Based• Hausdorff Distance

• Subdivision of Transformation Space

www.sportsmanscaps.com

Final Thoughts

Algorithms work well in various domains Query construction not formalized Other methods:

• wavelet-based• texture-based• object-based

Took 5 minutes to find “Shrine to Men with Long Hair” and “Videos of Women Getting Their Head Shaved”

www.jerryspringer.com

All other images generated by author using Paint Shop Pro