Computer Vision, Part 1

Topics for Vision Lectures

1. Content-Based Image Retrieval (CBIR)

2. Object recognition and scene “understanding”

Content-Based Image Retrieval

Example: Google “Search by Image”

Query Image

Extract Features(Primitives)

Image Database

Features Database

SimilarityMeasure

MatchedResults

RelevanceFeedbackAlgorithm

From http://www.amrita.edu/cde/downloads/ACBIR.ppt

Basic technique

Each image in database is represented by a feature vector: x1, x2, ...xN, where xi = (xi1, xi2, …, xim)

Query is represented in terms of same features: Q =(Q1, Q2, …, Qm)

Goal: Find stored image with vector xi most similar to query vector Q

• Distance measure:

• Possible distance measures d(Q, xi): – Inner (dot) product– Histogram distance (for histogram features)– Graph matching (for shape features)

.

.

.

€

x i⋅Q = x1iQ1 + x2

iQ2 + ...+ xmiQm

Some issues in designing a CBIR system

• Query format, ease of querying

• Speed

• Crawling, preprocessing

• Interactivity, user relevance feedback

• Visual features — which to use? How to combine?

• Curse of dimensionality

• Indexing

• Evaluation of performance

Types of Features Typically Used

• Intensities

• Color

• Texture

• Shape

• Layout

Intensity histogramshttp://www.clear.rice.edu/elec301/Projects02/artSpy/intensity.html

Color Features

Hue, saturation, value

http://www.owlnet.rice.edu/~elec301/Projects02/artSpy/patmac/mcolhist.gif

Color Histograms (8 colors)

http://www.owlnet.rice.edu/~elec301/Projects02/artSpy/patmac/mcolhist.gif

Color auto-correlogram

• Pick any pixel p1 of color Ci in the image I.

• At distance k away from p1 pick another pixel p2.

• What is the probability that p2 is also of color Ci?

p1

p2k

Red ?

Image: I

From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt

• The auto-correlogram of image I for color Ci , distance k:

• Integrates both color information and space information.

]|,|Pr[|)( 1221)(

iii CCkC IpIpkppI

From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt

Two images with their autocorrelograms. Note that the change in spatial layout would be ignored by color histograms, but causes a significant difference in the autocorrelograms. From http://www.cs.cornell.edu/rdz/Papers/ecdl2/spatial.htm

Color histogram rank: 411; Auto-correlogram rank: 1

Color histogram rank: 310; Auto-correlogram rank: 5

Color histogram rank: 367; Auto-correlogram rank: 1

From: http://www.cs.cornell.edu/rdz/Papers/ecdl2/spatial.htm

Texture representations

• Gray-level co-occurrence

• Entropy

• Contrast

• Fourier and wavelet transforms

• Gabor filters

Texture Representations

Each image has the same intensity distribution, but different textures

Can use auto-correlogram based on intensity (“gray-level co-occurrence”)

Texture from entropy

Images filtered by entropy:

Each output pixel contains entropy value of 9x9 neighborhood around original pixel

From: http://www.siim2011.org/abstracts/advanced_visualization_tools_ss_pao.html

Texture from fractal dimensionFrom: http://www.cs.washington.edu/homes/rahul/data/iccv07.pdf

Texture from Contrast

• Example: http://www.clear.rice.edu/elec301/Projects02/artSpy/graininess.html

Texture from Wavelets

http://www.clear.rice.edu/elec301/Projects02/artSpy/dwt.html

http://www.clear.rice.edu/elec301/Projects02/artSpy/dwt.html

Shape representations

Some of these need segmentation (another whole story!)

• Area, eccentricity, major axis orientation

• Skeletons, shock graphs

• Fourier transformation of boundary

• Histograms of edge orientations

From: http://www.lems.brown.edu/vision/researchAreas/ShockMatching/shock-ed-match-results1.gif

From: http://www.cs.ucl.ac.uk/staff/k.jacobs/teaching/prmv/Edge_histogramming.jpg

Histogram of edge orientations

Visual abilities largely missing from current CBIR systems

• Object recognition

• Perceptual organization

• Similarity between semantic concepts

Image 1 Image 2

Examples of “semantic” similarity

Image 1 Image 2

Examples of “semantic” similarity

Image 1 Image 2

Examples of “semantic” similarity

“In general, current systems have not yet had significant impact on society due to an inability to bridge the semantic gap between computers and humans.”

Image Understanding and Analogy-Making

Bongard problems as an idealized domain for exploring the “semantic gap”