Image Search Then and Now

Integrated Knowledge Solutionsiksincyahoocom

sikrishangmailcomiksincwordpresscom

Outlinebull Introductionbull Image = Content + Contextbull Content Based Image Retrieval (CBIR)bull Bridging the Semantic Gapbull Using Social Interactions for Retrievalbull Where do we go from here

What is Image Search

bull Image search means retrieving images from an image database that satisfy the userrsquos need

bull The user need may be expressed in the following waysndash Keywords or text describing the image contentndash An exemplar image

bull Other names for image searchndash Image retrievalndash Image similarity searchndash Content based image retrieval (CBIR)

Document Search Not a New Problem

Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang

The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC

However EarlierFew Document Producers

Many Document Consumers

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

What is Image Search

bull Image search means retrieving images from an image database that satisfy the userrsquos need

bull The user need may be expressed in the following waysndash Keywords or text describing the image contentndash An exemplar image

bull Other names for image searchndash Image retrievalndash Image similarity searchndash Content based image retrieval (CBIR)

Document Search Not a New Problem

Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang

The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC

However EarlierFew Document Producers

Many Document Consumers

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

Document Search Not a New Problem

Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang

The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC

However EarlierFew Document Producers

Many Document Consumers

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

Nalanda University was one of the first universities in the world founded in the 5th Century BC and reported to have been visited by the Buddha during his lifetime At its peak in the 7th century AD Nalanda held some 10000 students when it was visited by the Chinese scholar Xuanzang

The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC

However EarlierFew Document Producers

Many Document Consumers

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

The Royal Library of Alexandria in Egypt seems to have been the largest and most significant great library of the ancient world It functioned as a major center of scholarship from its construction in the third century BC until the Roman conquest of Egypt in 48 BC

However EarlierFew Document Producers

Many Document Consumers

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

However EarlierFew Document Producers

Many Document Consumers

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

But Now a Days

No Distinction Between Document Producers and Consumers

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

Some Relevant Numbers

Flickr has over 6 billion pictures as of August 2011 and 35 million images are uploaded daily

Photobucket has more than 10 Billion images and over 4 million images are uploaded everyday

Facebook has over 60 Billion photos and more than 350 million photos are uploaded everyday

Instagram has over 20 billion photos About 60 million photos are uploaded everyday

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

An image now a days is not just a picture but it is a picture with thousand words

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

So image retrieval should benefit from the contextual component if

present

How

But first let us look at image retrieval from the content

perspective only

QBICsignal similarity

Concept semantic similarity

Concept plus context

History of Image Retrieval

1993

2002

1999

A Typical QBIC Type Image Retrieval System

Feature Extraction

FeaturesMedia Collection

Indexing amp Matching

Query Feature Extraction

Retrieved Results

Relevance Feedback

Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)

Semantic Gap

Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept

Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000

httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265

QBICsignal similarity

Concept semantic similarity

Concept plus context

History of Image Retrieval

1993

2002

1999

A Typical QBIC Type Image Retrieval System

Feature Extraction

FeaturesMedia Collection

Indexing amp Matching

Query Feature Extraction

Retrieved Results

Relevance Feedback

Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)

Semantic Gap

Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept

Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000

httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265

A Typical QBIC Type Image Retrieval System

Feature Extraction

FeaturesMedia Collection

Indexing amp Matching

Query Feature Extraction

Retrieved Results

Relevance Feedback

Such systemsapproaches are often referred to as Content Based Image Retrieval (CBIR)

Semantic Gap

Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept

Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000

httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265

Semantic Gap

Early systems produced results wherein the retrieved documents were visually similar (signal level similar) but not necessarily similar in showing the same semantic concept

Content-Based Image Retrieval at the End of the Early YearsIEEE Transactions on Pattern Analysis and Machine Intelligence Arnold Smeulders Marcel Worring Simone Santini Amarnath Gupta Ramesh Jain December 2000

httpwwwsearchenginejournalcom7-similarity-based-image-search-engines8265

Semantic Gap

Users also like to query using descriptive words rather than query images or other multimedia objects This requires retrieval systems to correlate low-level features with high level concepts

Visually dissimilar images representing the same concept

Semantic Gap Challenge

How to Bridge the Semantic GapManual annotation

Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features

Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history

Crowdsourcing for Manual Annotation

Example of Image Search using Keywords

Search result in 2010

Example of Image Search using Keywords

Search result in 2014The results are better organized in sub-categories

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Semantic Gap Challenge

How to Bridge the Semantic GapManual annotation

Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features

Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history

Crowdsourcing for Manual Annotation

Example of Image Search using Keywords

Search result in 2010

Example of Image Search using Keywords

Search result in 2014The results are better organized in sub-categories

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

How to Bridge the Semantic GapManual annotation

Use machine learning tobull Build image category classifiers to perform semantic filtering of the resultsbull Build specific detectors for objects to associate concepts with imagesbullBuild object models using low level features

Exploit contextbull Text surrounding imagesbull Associated sound track and closed captions in videosbull Query history

Crowdsourcing for Manual Annotation

Example of Image Search using Keywords

Search result in 2010

Example of Image Search using Keywords

Search result in 2014The results are better organized in sub-categories

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Crowdsourcing for Manual Annotation

Example of Image Search using Keywords

Search result in 2010

Example of Image Search using Keywords

Search result in 2014The results are better organized in sub-categories

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Example of Image Search using Keywords

Search result in 2010

Example of Image Search using Keywords

Search result in 2014The results are better organized in sub-categories

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Example of Image Search using Keywords

Search result in 2014The results are better organized in sub-categories

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Example of Image Search using Keywords

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Example of Image Search using Keywords

Search result in 2014

Again the results are better organized in sub-categories

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Exploiting Context An Example

Kulesh Petrushin and Sethi ldquoThe PERSEUS Project Creating Personalized Multimedia News Portalrdquo Proceedings Second Intrsquol Workshop on Multimedia Data Mining 2001

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Machine Learning of Image Concepts

bull Challenging problembull Presence of multiple conceptsmultiple instancesbull Disproportionate number of negative examplesbull Manpower need for labeling training examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Feature Extraction Issues

Whole image based features Easy to use but not very effective

Region based features Both regular region structure and segmented regions are popular

Salient objects based features Connected regions corresponding to dominant visual properties of objects in an image

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Scale Invariant Feature Transform (SIFT) Descriptors

SIFT descriptors or its variants are currently the most popular features in use Each image generates thousands of features (key point descriptors) with each feature typically consisting of 128 values

httpwwwvlfeatorg

D G Lowe ldquoDistinctive image features from scale-invariant keypointsrdquo IJCV 2004

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Learning Image Concepts

bull Both supervised and unsupervised learning methods (SVM DT AdaBoost VQ etc) have been used

bull Early work limited to few tens of categories however some of the current systems can work with thousands of categoriesconcepts

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

VQ Based Learning Classifier

TestImage

Best CodebookLabel

Water Codebook

Sky Codebook

Fire Codebook

Mustafa amp Sethi (2004)

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

httpvisionstanfordeduteachingcs223blecturelecture14_intro_objrecog_bow_cs223bpdf

Bag of Words Approach

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Bag of Words Representation of Images

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Co-occurrence of Bag of Words

ImageCollection

Edge AnalysisImages

Collection of Binary Image

Blocks

Clustering

Local Feature

Descriptors(Codewords)

CodewordRepresentation

Of Images

Co-occurrenceMatrices of

Local Features

ComputeDistances

ImageDistanceMatrix

PathfinderNetwork

Mukhopadhyay Ma and Sethi ldquoPathfinder Networks for Content Based Image Retrieval Based on Automated Shape Feature Discoveryrdquo ISMSE 2004

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Co-occurrence of BoW

Original image

Representation byfeature indices (cluster membership)

Co-occurrence matrix

)()(max)( ABhBAhBAH

))max(min()( AaBbbaBAh

Hausdorff metric

Manhattan distance

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Notice how similar images are placed together in the graph

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Object Detectors for Image Concepts

PASCAL Visual Object Classes Challenge

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Project

httplabelmecsailmitedu

Web-based annotation tool to segment and label image regions Labeled objects in images are used as training images to build object detectors

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

IMARS provides a large number of built-in classifiers for visual categories that cover places people objects settings activities and events It is easy to add new ones IMARS can work on PC or laptop (trial version is available at IBM alphaWorks) IMARS can also work at large-scale for high-volume batch processing of millions and images and videos per day Several demos of IMARS are available (see IMARS demos)

Image Category Classifiers Examples

Semantic labeling (a) An MPE semantic retrieval system groups images by semantic concept and learns a probabilistic model for each concept (b) The system represents each image by a vector of posterior concept probabilities

From Pixels to Semantic Spaces Advances in Content-Based Image Retrieval (Nuno Vasconcelos IEEE Computer July 2007)

Image Classification via Probabilistic Modeling

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

Image = Content + Context

TagsCherryblossomJapantownSanFranciscoPeacePagoda

Content Context

Tagging

All time most popular tags at Flickr

About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works

with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification

Why Not Use Social Tags for Retrieval

Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list

Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

Tag Recommendation using TagsCo-occurrences

Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences

Tag Recommendation using TagsCo-occurrences and Visual Similarity

Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)

Given a target image and initial tags use the existing tagged images to suggest tags for the target image

Tag Ranking

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

About Tagsbull User centeredbull Imprecise and often overly personalizedbull Tag distribution follows power lawbull Most users use very few distinct tags while a small group of users works

with extremely large set of tagsbull Also known as Folksonomy social tagging and social classification

Why Not Use Social Tags for Retrieval

Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list

Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

Tag Recommendation using TagsCo-occurrences

Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences

Tag Recommendation using TagsCo-occurrences and Visual Similarity

Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)

Given a target image and initial tags use the existing tagged images to suggest tags for the target image

Tag Ranking

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Why Not Use Social Tags for Retrieval

Problem The relevant tag is often not at the top of the list Only less than 10 of the images have their most relevant tag at the top of the list

Solution Improve tagging by suggesting potential tags to a user tag ranking tag completion etc

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

Tag Recommendation using TagsCo-occurrences

Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences

Tag Recommendation using TagsCo-occurrences and Visual Similarity

Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)

Given a target image and initial tags use the existing tagged images to suggest tags for the target image

Tag Ranking

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag Recommendation using TagsCo-occurrences

Given a target image and initial tags use co-occurrence of tags to recommend tags for the target image This approach doesnrsquot take into account the visual features co-occurrences

Tag Recommendation using TagsCo-occurrences and Visual Similarity

Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)

Given a target image and initial tags use the existing tagged images to suggest tags for the target image

Tag Ranking

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag Recommendation using TagsCo-occurrences and Visual Similarity

Kucuktunc Sevil Tosun Zitouni Duygulu and Can (SAMT 08)

Given a target image and initial tags use the existing tagged images to suggest tags for the target image

Tag Ranking

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag Ranking

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag Ranking Another Approach

Dong Liu Xian-Sheng Hua Linjun Yang Meng Wang Hong-Jiang Zhang Tag Ranking WWW 2009 Madrid Spain

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

How to Compute Tag Similarity

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag Recommendation After Tag Ranking

bull Given an untagged image find its visually similar ldquokrdquo imagesbull Pool the top two ranked tags from k images and select the unique tags as

recommended tags

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag Completion

The complete tag matrix is generated by imposing constraints based on visual similarity tag to tag similarity and similarity with the initial tag matrix The matrix completion is done by an optimization procedure

Wu and Jain IEEE-PAMI JANUARY 2011

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

What about Taggers amp CommentersQuestion How can we incorporate taggerscommenters characteristics for improved tag recommendations

Answer Use three sets of features derived from image to be tagged userrsquos tag history and userrsquos social interactions

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Tag History amp Social Interaction Features

Tag history features are based on the tags the user has used in the past

Social interaction features are derived from tagscomments posted by the userrsquos friendsfavorite posters

X Chen amp H Shin ICDM 2010

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Current Status of Image Searchbull Extensive interest as evident from conferences journals and

special issuesbull Overall solid progress is being madebull Efforts towards performance evaluation with benchmarked

collections are gaining more tractionbull Integration of content and context through tags and

comments is receiving increasing attention to help improve retrieval

bull Killer applications are beginning to emerge as visual search gains prominence

bull Need for more applications outside entertainment

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Performance Evaluation Efforts

ImageCLEF2013 - Annotation Task

- 250000 Training Images- 95 (develop) 116 (test) concepts to be identified

- A lot of label Noise inside the training set due to the automatic label extraction from websites

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Performance Evaluation Efforts

TRECVID workshops an offshoot of TREC are yearly evaluation meetings since 2003 The goal of the workshops is to encourage research in content-based video retrieval and analysis by providing large test collections realistic system tasks uniform scoring procedures and a forum for organizations interested in comparing their results

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Application Examples

Tattoo-ID Automatic Tattoo Image Retrieval for Suspect amp Victim Identification (Anil K Jain Jung-Eun Lee and Rong Jin)

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

CBIR for Whole Slide Imageriesbull The availability of digital whole slide data sets

represent an enormous opportunity to carry out new forms of numerical and data- driven query in modes not based on textual ontological or lexical matchingndash Search image repositories with whole images or

image regions of interestndash Carry our search in real-time via use of scalable

computational architectures

Extraction from Image repositories based upon

spatial information

Analysis of datain the digital domain

hellip001011010111010111

Resultant Surface Map orgallery of matching images

or

Slide courtesy of Ulysses J Balis MDDirector Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Medical Image Retrieval

Text ldquoFind all the cases in which a tumor decrease in size

for less than three month post treatment then resumed a growth pattern after that periodrdquo

QUERY

Text + medical image ldquoFind images with large-sized frontal lobes brain tumors for patients approximately 35 years oldrdquo

+Medical image

QUERY IMAGE-BASED CONCEPTSMedical image ij - Specific Signature

ImageiQuery

VB-Spec CUIp

VB-Gen CUI1

VB-Spec CUIkIMAGE-BASED ONTOLOGY

GENRAL AND SPECIALIZED QUERY MEDICAL IMAGEVISUAL ANALYSIS

Text query

CUIn

CUI1

CUI2

QUERY TEXT-BASED CONCEPTS Textual query i - Indexes

MEDICAL ONTOLOGY

TEXT QUERYCONCEPTS EXTRACTION

Verification

Fusion

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Image Search Products

httpwwwpicalikecomproductssimilarity-searchphp

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Image Search Products

httpwwwpcssocom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Image Search Products

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Image Search Products

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

httpviralimagentuagr

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Take Home Message

bull Imagevideo retrieval is moving in the commercial domain Lot more activity is expected in near future

bull Multimodalcross-modal retrieval is gaining importance

bull Approaches combining social search and visual search techniques are expected to gain prominence

bull Crowdsourcing is a cheap and effective way of tagging media

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Acknowledgement

bull This presentation is based on the work of numerous researchers from the MIRMLCVPR community I have tried to give creditreferences wherever possible Any omission is unintentional and I apologize for that

bull Also want to thank my present and past students and collaborators

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom

Questions

Email iksincyahoocomEmail sikrishangmailcom

• Image Search Then and Now
• Outline
• What is Image Search
• Document Search Not a New Problem
• Slide 5
• Slide 6
• However Earlier
• But Now a Days
• Slide 9
• Some Relevant Numbers
• Slide 11
• Image = Content + Context
• Slide 13
• History of Image Retrieval
• A Typical QBIC Type Image Retrieval System
• Slide 16
• Slide 17
• Slide 18
• Semantic Gap
• Semantic Gap Challenge
• How to Bridge the Semantic Gap
• Crowdsourcing for Manual Annotation
• Slide 23
• Example of Image Search using Keywords
• Example of Image Search using Keywords (2)
• Example of Image Search using Keywords (3)
• Example of Image Search using Keywords (4)
• Exploiting Context An Example
• Machine Learning of Image Concepts
• Feature Extraction Issues
• Scale Invariant Feature Transform (SIFT) Descriptors
• Learning Image Concepts
• VQ Based Learning Classifier
• Bag of Words Approach
• Bag of Words Representation of Images
• Co-occurrence of Bag of Words
• Co-occurrence of BoW
• Slide 38
• Slide 39
• Slide 40
• Object Detectors for Image Concepts
• Project
• Image Category Classifiers Examples
• Image Classification via Probabilistic Modeling
• Image = Content + Context (2)
• Tagging
• About Tags
• Why Not Use Social Tags for Retrieval
• Tag Recommendation using Tags Co-occurrences
• Tag Recommendation using Tags Co-occurrences and Visual Similar
• Tag Ranking
• Tag Ranking Another Approach
• How to Compute Tag Similarity
• Slide 54
• Tag Recommendation After Tag Ranking
• Tag Completion
• What about Taggers amp Commenters
• Tag History amp Social Interaction Features
• Current Status of Image Search
• Performance Evaluation Efforts
• Performance Evaluation Efforts (2)
• Application Examples
• CBIR for Whole Slide Imageries
• Medical Image Retrieval
• Image Search Products
• Image Search Products (2)
• Image Search Products (3)
• Image Search Products (4)
• Slide 69
• Slide 70
• Take Home Message
• Acknowledgement
• Questions Email iksincyahoocom Email sikrishangmailcom