CH 11 Multimedia IR: Models and Languages

Presented by: Mohammed Al-Rashidi

Supervised by:Dr. Mourad Ykhlef

Multimedia IR system

The architecture of a Multimedia IR system depends on two main factors The peculiar characteristics of multimedia data The kinds of operations to be performed on such

data

Multimedia IR system

Support variety of data Different kinds of media

Text, images (both still and moving), graphs, and sound Mix of structured and unstructured data

Metadata Semi-structured data

Data whose structure may not match, or only partially match, the structure prescribed by the data schema

The system must typically extract some features from the multimedia objects

Multimedia IR system

Data retrieval Exploiting data attributes and the content of

multimedia objects Basic steps

1. Query specification Fuzzy predicates (Find all images similar to a car) Content-based predicates (Find all objects containing an

apple) Object attributes (Find all red images) Structural predicates (Find all multimedia containing a

video clip).

Multimedia IR system

2. Query processing and optimization Query is parsed and compiled into an internal form

3. Query answer

4. Query iteration

Combine DBMS and IR technology DBMS: data modeling capabilities IR system: similarity-based query capabilities

Data modeling

Main tasks1. A data model should be defined by which the

user can specify the data to be stored into the system

Support conventional and multimedia data types Provide methods to analyze, retrieve, and query such

data

2. Provide a model for the internal representation of multimedia data

Object-oriented DBMS

Provide rich data model More suitable for modeling both multimedia data types and

their semantic relationships Class

Attributes +operations Inheritance

Drawback The performances of storage techniques, query

processing, and transaction management is not comparable to that of relational DBMSs

Highly non-standard

Object-relational DBMS

Extend the relational model Represent complex data types Maintain the performance and the simplicity of

relational DBMSs and related query languages Define abstract data types

Allows one to define ad hoc data types for multimedia data

Internal representation

Using attributes is not sufficient Feature

Information extracted from objects Multimedia object is represented as a set of

features Features can be assigned manually,

automatically, or using a hybrid approach

Internal representation

Values of some specific features are assigned to a object by comparing the object with some previously classified objects

Feature extraction cannot be precise A weight is usually assigned to each feature value

representing the uncertainty of assigning such a value to that feature 80% sure that a shape is a square

SQL3

Support extensible type system Provide constructs to define user-dependent abstr

act data types, in an object-oriented like manner Collection data types

Sets, multisets, and lists The elements of a collection must have compatibl

e types

MULTOS

MULTimedia Office Server Client/server Support filing and retrieval of multimedia objects

Each document is described by a logical structure, a layout structure, and a conceptual structure

Documents having similar conceptual structures are grouped into conceptual types

Document

Place Date Receiver+ Sender

AddressName

CityStreet Country

AddressName

CityStreet Country

Letter_body

spring component type

Conceptual structure of the type Generic_Letter

Complete conceptual structure of the type Business_Product_Letter

Document

Place Date Receiver+ Sender

AddressName

CityStreet Country

AddressName

CityStreet

Company_LogoImage

Country

Product_PresentationText

Product_DescriptionText

Product_CoseText

Signature

Letter_body

Query languages

Relational/object-oriented database system Exact match of the values of attributes

Multimedia IR system Similarity-based approach

Considers the structure and the content of the objects Content-based query

Retrieve multimedia objects depending on their globe content

Query languages

In designing a multimedia query language, three main aspects require attention How the user enters their request to the system Which conditions on multimedia objects can be

specified in the user request How uncertainty, proximity, and weights impact

the design of the query language

Request specification

Interfaces Browsing and navigation Specifying the conditions the objects of interest

must satisfy, by means of queries Queries can be specified in two different

ways Using a specific query language Query by example

Using actual data (object example)

Search Using Sketch in QBIC

Sketch entry

Results of search

Search by Color in QBIC

Color selection15% yellow and 13% blue

Results of search

Search by User Drawn Pattern in QBIC

Pattern

Results of search

Conditions on multimedia data

1. Attribute predicates Concern the attributes for which an exact value is

supplied for each object Exact-match retrieval

2. Structural predicates Concern the structure of multimedia objects Can be answered by metadata and information about

the database schema “Find all multimedia objects containing at least one

image and a video clip”

Conditions on multimedia data

3. Semantic predicates Concern the semantic content of the required data,

depending on the features that have been extracted and stored for each multimedia object

“Find all the red houses” Exact match cannot be applied

Thanks

Any questions