Content-based image retrieval using a mobile

device as a novel interface

Jonathon S. Hare and Paul H. Lewis

Intelligence, Agents, Multimedia GroupDepartment of Electronics and Computer Science

University of Southampton{jsh02r, phl}@ecs.soton.ac.uk

Introduction

• This work investigates the use of a mobile device as a novel interface to a image retrieval system.

• We develop a new two-stage retrieval strategy that exhibits impressive retrieval performance even with the poor imaging quality of the camera on the mobile device.

Content-based Image Retrieval

• A retrieval system that is robust to the poor imaging qualities found in low-end digital cameras.

• Two stage algorithm:

• First stage inspired by Information Retrieval techniques: Vector-Space model.

• Second stage re-ranking of the first stage results based on a geometric constraint.

Salient Regions for CBIR

• Retrieval based on DoG based salient regions.

• DoG SR’s previous shown to be robust to noise, rotation and other degradation's, such as those found in low-end digital cameras.

Local Descriptors

• Lowe’s SIFT Key feature used to describe image in each salient region.

• Robust to small shifts in salient region position.

• Robust to illumination changes.

• Doesn’t use colour.

Retrieval Using Text Retrieval Techniques

• We have tried to apply techniques taken from the information retrieval field:

• Vector-Space Model.

• Documents in the collection are represented by vectors of term-frequency.

Retrieval Using Text Retrieval Techniques

• A vocabulary of ‘visual words’ is built, based on feature vectors from a subset of the images in the database.

• Each feature vector in the database is vector quantised to the closest ‘visual word’.

Retrieval Using Text Retrieval Techniques

• Vector-Space Model.

• Term frequency vectors are weighted:

• TF-IDF.

• Weighted Vector is created for the query image. Documents are ranked by comparing vectors using cosine similarity.

Geometric Consistency

• Second stage of retrieval involves re-ranking based on geometric constancy of matching salient regions.

• Use RANSAC to estimate planar homography between query and target image, then re-rank on percentage of salient region pairs fitting the homography.

Mobile Client-Server Implantation

XML-RPC Web Service

Query Engine

Salient Region & Feature Vector

Generator

Feature Vector and Metadata Database

XML-RPC Encoded query, with embedded

compressed image

XML-RPC Encoded response, with an embedded URL

Example ApplicationViewfinder

Retrieved Metadata

Retrieval Performance

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 20 40 60 80 100

Pe

rce

nta

ge

Rank

CCVColour HistogramMono Histogram

PWTVectorSpace

0

20

40

60

80

100

0 10 20 30 40 50 60 70 80 90 100

Pe

rce

nta

ge

N

% Correctly matched% Incorrectly matched

Performance of retrieval with second-stage, versus number of first-stage results considered

Performance of first-stage retrieval, versus rank of matching image

Future Work• Investigate how to construct an optimal

‘visual’ vocabulary.

• Add a local colour-based descriptor to use in addition to the SIFT descriptors.

• Investigate the use of “stop-words” and their effect on retrieval performance.

• Implement an inverted index to aid efficiency and speed.

Conclusions• We have developed a two-stage image

retrieval algorithm that is able to effectively retrieve correctly matching images with query images from low-quality sources, such as cheap digital cameras.

• The system has been implemented in a client-server fashion, with a mobile device used for generating queries and receiving results.

Any Questions?