Lecture 8
-
Upload
bama-raja-segaran -
Category
Documents
-
view
213 -
download
0
description
Transcript of Lecture 8
Image Indexing and Retrieval
Dr Crawford Revie(with thanks to Prof Fabio Crestani)
Lecture 8Multimedia Information Access
MIA Lecture 8 Crawford Revie (2006) 2
Outline of lecture
Why do we require image retrieval?Overview of techniquesExamples of systems
NB – if you print these slides out then you will probably want to use colour (or at least for some of them, 18-28 in particular don't make a lot of sense in b/w!)
MIA Lecture 8 Crawford Revie (2006) 3
Motivation and Application Areas
Management of Image ArchivesArt Galleries & MuseumsWWW Image Indexing
Science Databases (Medicine, Astronomy, Geography)Industry specific
Trademark DatabasesTextiles & FabricsAdvertisingArchitecture & Design
MIA Lecture 8 Crawford Revie (2006) 4
Approaches
There are 3 main approaches used in practice:1. Keyword based
manual / semi-automatic / automatic2. Based on visual properties
automatic3. Concept based
mostly manual (still in 'research' mode)
MIA Lecture 8 Crawford Revie (2006) 5
Keyword approach: indexing
Images are annotated using keywords
But:manual annotation is very expensive (as it is exceedingly time consuming)low level visual properties are almost impossible to index consistently using manual mark-upeven for 'high level' properties manual annotation is prone to subjectivity
Take a look at the Google Image Labeler 'game'http://images.google.com/imagelabeler
MIA Lecture 8 Crawford Revie (2006) 6
Keyword approach: retrieval
Since image description is textual, we use an almost straight-forward application of standard IR techniques (stop-word removal, stemming, indexing, etc.)
hypertext links have proved to be useful for retrieving images ("retrieval by browsing")thesauri and vocabularies can be more necessary here than in standard IR (e.g. AAT) – see also later discussion of structured keywords and concept-based retrieval
MIA Lecture 8 Crawford Revie (2006) 7
From manual to automatic keyword assignment
Keywords can be assigned to an image by analysing the text associated with the image
this includes the alt and caption attributes of the <img> and <table> HTML tagstext elsewhere on a web page containing the imagetext of the link pointing to the imageeven the name of the file containing the image
MIA Lecture 8 Crawford Revie (2006) 8
Automatic keywords
Why are thesepictures retrieved?
MIA Lecture 8 Crawford Revie (2006) 9
Automatic keywords
Check the text on the web page; the caption; and the filename
MIA Lecture 8 Crawford Revie (2006) 10
Structured keywords: using a database
Some systemsuse a DBMS to handle keywordsand searches
MIA Lecture 8 Crawford Revie (2006) 11
Visually based approaches
Often referred to as Content Based Image Retrieval (CBIR)
Similarity between query and documents is calculated based on visual features:
colourtextureshape
Visual features may be detected automatically or semi-automatically
MIA Lecture 8 Crawford Revie (2006) 12
Feature vectors
Images are represented using a set of feature vectors
),,( 1 inii IfIfIf K=
Queries are represented with the same set of feature vectors
),,( 1 inii QfQfQf K=
MIA Lecture 8 Crawford Revie (2006) 13
Vector features
Each feature has its own representation range of values, variability, etc.
Feature vectors may provide a “synthetic” view of a certain feature
In IR each word is represented by one feature exactly, but here one image characteristic may be represented by many features
similar to audio retrieval
MIA Lecture 8 Crawford Revie (2006) 14
Similarity functions
Need to choose similarity functions carefully:should be a good approximation to human perceptionof similarity between imagesshould have properties that help speed up computations
Different types of similarity evaluations may need to be combined to compare overall similarity (e.g. shape, colour, texture, etc.)
MIA Lecture 8 Crawford Revie (2006) 15
Colour based retrieval
Arguably easiest; earliest to be usedProcess is as follows:
represent image as a rectangular pixel raster (e.g. 1024 columns and 768 rows)represent each pixel as a quantified colour (e.g. 256 colours ranging from red through violet)count the number of pixels in each colour bin (this will produce a vector representation)compute vector similarity (e.g. using the normalised inner product)
MIA Lecture 8 Crawford Revie (2006) 16
Colour based matching
Let's compare some images retrieved using keyword: Godzilla
MIA Lecture 8 Crawford Revie (2006) 17
Colour histograms (for two samples)
MIA Lecture 8 Crawford Revie (2006) 18
Texture matching
Texture characterizes small-scale regularitycolour describes pixels, texture describes regions
Described by several types of featuressmoothness, periodicity, directionality, etc.
Match region size with image characteristicscomputed using filter banks, Gabor wavelets
Perform weighted vector space matchingusually in combination with a colour histogram
MIA Lecture 8 Crawford Revie (2006) 19
Texture matching
MIA Lecture 8 Crawford Revie (2006) 20
Image segmentation (shape)
Global techniques alone yield low precisioncolour & texture are better at characterising objects, not full images
Segment at colour and texture discontinuitieslike “flood fill” in Photoshop
Represent size shape & orientation of objectse.g. in Berkeley’s Blobworld we use ellipses
Represent relative position of objectse.g. angles between lines joining the centers
Segmentation allows us to perform object rotation and scale-invariant matching
MIA Lecture 8 Crawford Revie (2006) 21
"Flood fill" in Photoshop
More sophisticated techniques are needed
MIA Lecture 8 Crawford Revie (2006) 22
CBIR systems: Examples
Commercial systemsVirageQBIC
AcademicBlobworldVisualSeekChabotViper
MIA Lecture 8 Crawford Revie (2006) 23
QBIC
You can sketchan example of
what you are looking for
MIA Lecture 8 Crawford Revie (2006) 24
QBIC
Can you spotthe similarity to your 'query definition'?
MIA Lecture 8 Crawford Revie (2006) 25
Berkeley Blobworld
Can you spot the similarity to your 'query definition'?
MIA Lecture 8 Crawford Revie (2006) 26
Berkeley Blobworld
MIA Lecture 8 Crawford Revie (2006) 27
Blobworld Segmentation (1)
MIA Lecture 8 Crawford Revie (2006) 28
Blobworld Segmentation (2)
MIA Lecture 8 Crawford Revie (2006) 29
Viper: query by example
Provide examples
by indicatingrelevant images
MIA Lecture 8 Crawford Revie (2006) 30
Viper (QBE)
Retrieves similar
images
MIA Lecture 8 Crawford Revie (2006) 31
Concept based approach
Knowledge of the application domain is requirede.g. indexing of medical images requires knowledge of medicine(!) vocabulary + domain specific
System assigns concepts (index terms) to part of the image:
automatic concept assignment: very imprecise and ambiguous processmanual concept assignment: time-consuming and highly subjective process
Few experiments, semi-automatic seems best so far
MIA Lecture 8 Crawford Revie (2006) 32
Simple image retrieval is commercially availablecolour histograms, texture, limited shape information
Segmentation-based retrieval is still in the lab
Some way off:automatic identification and recognition of objects in images and videosconceptual image retrieval
The future?