IBM QBIC: Query by Image and Video Content

Jianping Fan

Department of Computer Science

University of North Carolina at CharlotteCharlotte, NC 28223

http://www.cs.uncc.edu/~jfan

How can I access video in database over networks?

1. How to access video ? 2. How to represent video ?3. How to index large-scale videos ? 4. How to access videos in

database ? 5. How to transmit query results over IP ?

Networks

6. How to control user’s access ?

1. System Architecture

How to access video in database?

Images

Videos

Shot extraction1. Representative Frame

2. Motion-based objects3. shotsObject Identification

R-frames

scenesketchPositional

color/texture

objectLocation/

color

User-defined

Color/texture

videoObject motion

Camera motion

Feature extractionscene object

sshotsMotion

objects

1. System Architecture

How to access video in database?

user

Query interfaceColor Texture Shape Multi-object Sketch Location TextPositional color/texture object motion camera

motion

user defined existing image

Match engineDatabase Indexing

Returned via similarity order

2. How to access image or video in QBIC?

a. Video shotsb. Video objects or sketches & drawingsc. Representative frames

Access approaches

a. Example imagesb. user-constructed sketches or drawingsc. Selected color and texture patterns

d. Camera & object motione. Other graphical information

d. Motion types

e. Other information

2. How to obtain accessing units in QBIC?Shot Detection

2. How to obtain accessing units in QBIC?

a. Difference Calculations

b. Automatic Decision MakingVia Pre-Defined Thresholds.

|)()(|)1,( 1

255

0

jHjHiiHD ij

i

sc

sc

TiiHD

TiiHD

)1,(

)1,(

How we do the shot detection?

2. How to obtain accessing units in QBIC?

Object extraction

2. How to obtain accessing units in QBIC?

Object extraction

How we can do the object extraction?

3. How to represent these video units?

Images

Global color

Global texture

Positional color

Positional textureSketch, shape

User-defined color/texture

3. How to represent these video units?

Videos

Global color

Global texture

Positional color

Positional textureSketch, shape

User-defined color/textureCamera motion/object motion

ColorHSV color histogram, dominant color, …

TextureEdge histogram, wavelet coefficients, Tamura features, …

MotionDirectional motion histogram, Camera motion, …

Other features

Video Sequence

Shot 1 Shot i Shot n

How we do the shot representation?

4. How to index images/videos in QBIC?

... . .

.....

.. ...... . .... .....

feature space

Videos in

Database

4. How to index images/videos in QBIC?

High-dimensional visual features

K-L Transform to reduce dimensions

Low-dimensional R*-tree indexing

4. How to index images/videos in QBIC?

Overlap on two Dimensions!

4. How to index images/videos in QBIC?

Karhunen-LoeveTransformation

New Eigenvectors

tUVSMS 1

9

5

4

3

2

1

000000000

......................

000000000

000000000

000000000

000000000

000000000

U

M is the matrix for videos!

S is the KL transform matrix!

tV

5. How to realize query in QBIC?

5. How to realize query in QBIC?

5. How to realize query in QBIC?

How we can do the video query?

5. How to realize query in QBIC?

How we can do the mosaic?

Why we use mosaic for video representation?

6. What lost by QBIC?

a. Mapping from visual features to semantic conceptsIt is hard, but we have to do this. Why?

],.....,[21 niii xxx

.. ..... ...

Visual Features

...

. ..

Semantic Clusters jC

. . . . . . . . . . . . . . . . .Video Contents in

Database

Weighted mapping?

How to do this mapping?

6. What lost by QBIC?

b. High-dimensional visual indexing

It is a basic problem in database area, but only database people cannot solve this challenging problem for visual indexing! Video in Database

Cluster 1 Cluster i Cluster n

Subcluster 11 Subcluster 1j Subcluster n1 Subcluster nl

Subregion 111 Subregion 11k Subregion nl1 Subregion nlm

object1111 object nlm1

Disk for Cluster 1 Disk for Cluster i Disk for Cluster n

ii DN log

6. What lost by QBIC?

c. User input in the query procedure: QBIC can permit user to select something.

6. What lost by QBIC?

d. How to integrate keywords with visual features?

6. What lost by QBIC?

e. How to provide user-intensive browsing?

7. What happen now on QBIC?

You can find the current version of QBIC system on:

http://wwwqbic.almaden.ibm.com/

Homework:

What kind of technique we have discussed used in QBIC?

8. Other Projects

a. Chabot at UC Berkeley

http://www.cs.berkeley.edu/~ginger/chabot.html

b. Viper at Europe

http://viper.unige.ch/

c. Virage

http://www.virage.com/

8. Other Projects

d. PicHunter at NEC

http://www.neci.nj.nec.com/homepages/vision/index.html

e. Ifind at Microsoft

http://wwww.microsoft.com/china/research/group/

f. Photobook at MIT

http://www-white.media.mit.edu/~tpminka/photobook/