Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and...
-
date post
20-Dec-2015 -
Category
Documents
-
view
222 -
download
1
Transcript of Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and...
![Page 1: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/1.jpg)
Automatic Image Annotation and Retrieval using Cross-Media Relevance Models
J. Jeon, V. Lavrenko and R. Manmathat
Computer Science DepartmentUniversity of Massachusetts – Amherst
Presenter: Carlos Diuk
![Page 2: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/2.jpg)
Introduction The Problem:
Automatically annotate and retrieve images from large collections.
Retrieval example: answer query “Tigers in grass” with
![Page 3: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/3.jpg)
Introduction Manual annotation being done in
libraries. Different approaches to automatic
image annotation: Co-occurence Model Translation Model Cross-media relevance model
![Page 4: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/4.jpg)
Introduction – related work Co-occurence Model
Looks at co-occurence of words with image regions created using a regular grid.
Translation ModelImage annotation viewed as task of
translating from vocabulary of blobs to vocabulary of words.
![Page 5: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/5.jpg)
Introduction – CMRM Cross-media relevance models
(CMRM) Assume that images may be
described from small vocabulary of blobs.
From a training set of annotated images, learn the joint distribution of blobs and words.
![Page 6: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/6.jpg)
Introduction – CMRM Cross-media relevance models
(CMRM) Allow query expansion:
Standard technique for reducing ambiguity in information retrieval.
Perform initial query and expand by using terms from the top relevant documents.
Example in image context: tigers more often associated with grass, water, trees than with cars or computers.
![Page 7: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/7.jpg)
Introduction – CMRM Variations:
Document based expansion PACMRM (probabilistic annotation CMRM) Blobs corresponding to each test image are used to generate
words and associated probabilities. Each test generates a vector of probabilities for every word in vocabulary.
FACMRM (fixed annotation-based CMRM)Use top N words from PACMRM to annotate images.
Query based expansion DRCMRM (direct-retrieval CMRM) Query words used to generate a set of blob probabilities. Vector
of blob probabilities compared with vector from test image using Kullback-Lieber divergence and resulting KL distance.
![Page 8: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/8.jpg)
Discrete features in images
Segmentation of images into regions yields fragile and erroneous results.
Normalized-cuts are used instead (Duygulu et al): 33 features extracted from images. K (=500) clustering algorithm used to cluster regions based on
features. Vocabulary of 500 blobs.
![Page 9: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/9.jpg)
CMRM Algorithms Image I = {b1 .. bm} set of blobs Training collection of images J =
{b1 .. bm ; w1 .. wn} Two problems:
Given un-annotated image I, assign meaningful keywords.
Given text query, retrieve images that contain objects mentioned.
![Page 10: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/10.jpg)
CMRM Algorithms Calculating probabilities.
![Page 11: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/11.jpg)
CMRM Algorithms Image retrieval
INPUT: query Q = w1 .. wn and collection C of images OUTPUT: images described by query words.
Annotation-based retrieval model (PACMRM-FACMRM)
Annotate images as shown. Perform text retrieval as usual. Fixed-length annotation vs probabilistic annotation:
![Page 12: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/12.jpg)
CMRM Algorithms Image retrieval
INPUT: query Q = w1 .. wn and collection C of images OUTPUT: images described by query words.
Direct retrieval model (DRCMRM) Convert query into language of blobs, instead of
images into words. Estimation:
Ranking:
![Page 13: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/13.jpg)
Results Dataset
Corel Stock Photo CDs (5000 images – 4000 training, 500 evaluation, 500 testing). 371 words and 500 blobs. Manual annotations.
Metrics: Recall: number of correctly retrieved images divided
by number of relevant images. Precision: number of correctly retrieved images
divided by number of retrieved images. Comparisons
Co-occurence vs Translation vs FACMRM
![Page 14: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/14.jpg)
Results Dataset
Corel Stock Photo CDs (5000 images – 4000 training, 500 evaluation, 500 testing). 371 words and 500 blobs. Manual annotations.
Metrics: Recall: number of correctly retrieved images divided
by number of relevant images. Precision: number of correctly retrieved images
divided by number of retrieved images. Comparisons
Co-occurence vs Translation vs FACMRM
![Page 15: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/15.jpg)
Results Precision and recall for 70 one-word queries.
![Page 16: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/16.jpg)
Results PACMRM vs DRCMRM
![Page 17: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/17.jpg)
Some nice examples
Automatically annotated as sunset, but not manually
![Page 18: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/18.jpg)
Some nice examples
Response to query “pillar”
Response to query “tiger”
![Page 19: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/19.jpg)
Some bad examples
![Page 20: Automatic Image Annotation and Retrieval using Cross-Media Relevance Models J. Jeon, V. Lavrenko and R. Manmathat Computer Science Department University.](https://reader035.fdocuments.net/reader035/viewer/2022062320/56649d425503460f94a1d5fa/html5/thumbnails/20.jpg)
Questions - Discussion No semantic representation (just color, texture, shape).
How could we annotate a newspaper’s collection? (“Kennedy”, not just “people”)
Google: cooperative annotation? Google search for “tiger”:
Google search for “Kennedy”: