Multimedia IR

Models and Languages

Introduction (1)

• The need for integrated managementfor multimedia data is growing

• “Multimedia Information Systems”becomes one of the most promisingfields in “Information Management”

• The most important characteristic ofMultimedia Information System is anability to support the variety of data

Introduction (2)

• Multimedia systems must have thecapability to store, retrieve, transportand present data with heterogeneouscharacteristic

Multimedia InformationRetrieval is needed!

Comparison of IR System

Traditional IR• does not support

metadatainformation

• does not requiremetadatahandling

Multimedia IR• require some

forms of databaseschema

• require metadatahandling

Multimedia IR: Architecture

The architecture of a Multimedia IRdepends on– characteristic of multimedia data– operations to be performed on such

data

Data Modeling

A Multimedia IR system should be able to– represent and store multimedia objects in a

way that ensures fast retrieval– deal with different kinds of media– deal with semi-structured data (not match

or partially match the structure prescribedby the data schema)

– extract features from multimedia objects

Multimedia IR: Goal

The main goal of a Multimedia IR system:efficiently perform retrieval based onuser requests,exploiting

1. data attributes2. content of multimedia objects

Multimedia IR: Challenges

Challenges in Multimedia IR are1. heterogeneity of data2. fuzziness of information3. loss of information in the creation ofindexes4. need of an interactive refinement ofthe query result

Data Retrieval

Data retrieval relies on the following steps• Query specification: user specifies the

request• Query processing and optimization: the

query is parsed and compiled into aninternal form (then optimized)

• Query answer: returning the retrievedobjects to the user

• Query iteration: repeat until user is satisfied

Query Specification:Fuzzy Predicates

• Proximity predicates: ex. ‘Find all imagessimilar to a car’

• Content-based predicates: ex. ‘Findmultimedia objects containing an apple’

• Conventional predicates on the objectattributes: ex. Conditions on the attribute‘color’ of an images such as ‘Find all redimages’

• Structural predicates: ex. ‘Find allmultimedia objects containing a video clip’

Query Processing andOptimization

• Traditional systems– Query is parsed, compiled into an internal form

(may also be optimized)• Query processing is a very complex activity in

Multimedia IR system– Difficulties arise when deal with the presence of

fuzzy terms, content-based predicates andstructural predicates

– The heterogeneity of data requires different queryprocessing strategies

Query Answer

• The retrieved objects are returned tothe user in decreasing order ofrelevance

• Relevance is measured as a distancefunction from the query object to thestored ones

Query Iteration

• Traditional DBMSs:– Query process ends when the system

returns the answer to the user• Multimedia IR:

– The user supplies additional information tothe system to refine the result

– This is due to the impreciseness of theanswer

Multimedia IR System

• Requires integration of traditional IRtechnology with the technology of multimediadatabase management systems to represent,manage, store multimedia objects

• Object retrieval is based on a similarityapproach

• Should combine DBMS and IR technology– DBMS: data modeling capabilities– IR system: similarity-based query capabilities

Data Modeling

• Traditional DBMSs are targeted tosupport conventional data

• The multimedia data:– are not encoded into attributes provided by

the data schema– require large storage– the content is difficult to analyze and

compare

Data Modeling

Addressing data model in Multimedia IRsystems consisting of 2 main tasks:1. a data model should be defined by which

the user can specify the data to be storedinto the system (integrated support for bothconventional and multimedia data typesand provide methods to analyze, retrieve,and query)

2. the system should provide a model for theinternal representation of multimedia data

Data Modeling Technology

OODBMSs provide:– the richness of the data model: modeling

data types and semantic relationships– the concept of class: defining ad hoc data

types and related set of operations,inheritance hierarchies

– However, performance of the OODBMSsare not as good as relational DBMSs

– Highly non-standard

Internal Representation

• A set of attributes is not sufficient todescribe the data

• Some information should be extractedfrom the objects (called a set offeatures) and used during queryprocessing

• Internally, each multimedia object isrepresented as a list of features

Internal Representation (2)

• Features can be assigned to the objectmanually by the user or automatically by thesystem

• Usually a hybrid approach is used where thesystem selects some values and the usercorrects or augments them

• Feature extraction cannot be precise, aweight is usually assigned to each featurevalue representing the uncertainty ofassigning such a value to that feature

Multimedia Data Support inCommercial DBMSs

• Most current relational DBMSs supportvariable-length data types

• Each DBMS vendor uses differentnames for data types and providessupport for different operations (non-standard)

• Ex. Oracle DBMS provides VARCHAR2

Example: Oracle DBMS

• VARCHAR2: variable length characterstrings

• RAW, LONG RAW: data not to beinterpreted by Oracle

• LOB: Large unstructured data Objectsup to 4GB

• BLOB: Binary LOB data• CLOB: Character LOB data

Example: Sybase SQL server

• IMAGE: images• TEXT: unstructured text• Provides a limited set of functions for

searching and manipulation

SQL3 (1)

• Its predecessor is SQL-92• Provides support for an extensible type

system• Extensibility of the type system is achieved by

providing constructs to define user-dependentabstract data types, in an object-oriented likemanner

• Each type specification consists of bothattribute and function specifications

SQL3 (2)

• A strong form of encapsulation is provided, inthat attribute values can only be accessed byusing some system functions

• User-defined functions can be either visiblefrom any object or only visible in the objectthey refer to

• Both single and multiple inheritance can bedefined among user-defined types anddynamic late binding is provided

SQL3 (3)

• Provides 3 types of collection datatypes: sets, multisets, and lists

Oracle

• Provides data cartridges for text, spatial data,image, audio and video data

• ConText cartridge:– is a text management solution combining data

management capabilities of a traditional DBMSwith advanced text retrieval and natural-languageprocess technology

– supports most popular document formats, ASCII,MS Word, HTML, etc.

– is able to find documents about a specific topic

Illustra

• Provides 3D and 2D spatial data bladesfor modeling spatial data

• Supported data types included boxes,vectors, quadrangles, etc.

• Supported operations are INTERSECT,CONTAINS, OVERLAPS, CENTER,etc.

La Scala Archive Project

• At the Laboratorio di Informatica Musicale ofthe University of Milano

• Development of the multimedia archive ofTeatro alla Scala

• Using the Oracle technology and the relateddata cartridges

• Object-relational technology and extensivetype system

The MULTOS Data Model (1)

• MULTimedia Office Server is amultimedia document server withadvanced document retrievalcapabilities

• Developed in the context of an ESPRITproject in the area of Office Systems

• Based on a client/server architecture

The MULTOS Data Model (2)

• Three different types of document servers aresupported: current servers, dynamic servers,and archive servers (differ in storage capacityand document retrieval speed)

• support filing and retrieval of multimediaobjects based on document collections,document types, document attributes,document text, and images.

The MULTOS Data Model (3)

The MULTOS data model allows• the representation of high level

concepts present in the documentscontained in the database

• the grouping of documents into classesof documents having similar contentand structure

• the expression of conditions on free text

The MULTOS Data Model (4)

Each document is described by• a logical structure: determines arrangements

of logical document components (titles,introduction, chapter, section, etc.)

• a layout structure: deals with the layout of thedocument content and it contains components(pages, frames, etc.)

• a conceptual structure: allows a semantic-oriented description of the document content(as opposed to the syntax-oriented descriptionprovided by the logical and layout structure)

The MULTOS Data Model (5)

• Documents having similar conceptualstructures→ grouped into conceptual types

• Conceptual types are maintained in ahierarchy of generalization

• Types can be strong (completely specifies thestructure of its instance) or weak (partiallyspecifies the structure of its instance)

• Component of unspecified types are calledspring component types

Query Languages

• Relational or OO database systems queriesare based on an exact match mechanism

• Multimedia IR system queries are based onsemi-structured nature of multimedia objects– The user should be able to query by specifying

values of semantic attributes and additionalconditions about the content of multimedia data

• Exact match and similarity-based approach• Consider both structure and content

Main aspects inContent-based Query

• How the user enters request to thesystem

• Which conditions on the multimediaobjects can be specified in the userrequest (depend on the supportprovided by the system)

• How uncertainty, proximity and weightsimpact the design of the query language

Request Specification

• Two different interfaces can bepresented to the user– Browsing and navigation (time consuming)– Querying: traditional approach or by

example approach• Query by example approach requires

the use of a GUI environment

Conditions of Multimedia Data

• Multimedia query languages shouldprovide predicates for expressingconditions on the attributes, the contentand the structure of multimedia objects

• Three groups of query predicates– Attribute predicates– Structural predicates– Semantic predicates

Attribute Predicates

• Predicates against traditional attributes• System applies an exact-match retrieval• Using the same techniques as

traditional DBMSs• Ex. The speaker of an audio object, the size

of an object

Structural Predicates

• Concern the structure of multimediaobjects

• Answer using some form of metadata andinformation about the database schema

• Play a fundamental role in multimediaquery processing, due to the complexstructure of multimedia objects

• Ex. Find all multimedia objects containing at leastone image and a video clip

Semantic Predicates

• Concern the semantic content of thequeried data

• Depending on the features that havebeen extracted and stored for eachmultimedia object

• An exact match cannot be applied• Only degree of relevancy can be used

on a list of retrieved objects• Ex. Find all the red houses

Uncertainty, Proximity, andWeights in Query Expressions

Three ways to specify the degree of relevance ofthe retrieved objects

• Using some imprecise terms and predicates:normal, unacceptable, typical

• Specifying particular proximity predicates: therelationship is based on the computation of asemantic distance between the query objectand the stored ones: The Nearest object search

• Assigning each condition or term a givenweight: HIGH, LOW

Some Proposals

• The SQL3 query language– Extensible type system– Ability to deal with complex objects– Functions and stored procedures: allows

the user to integrate functionalities withdata manipulation

– Active database facilities: support of activerules

Some Proposals

• The MULTOS query language• Version-clause• Scope-clause• Type-clause• Condition-clause

Trends and Research Issues

• Data model• Multimedia query languages• Techniques for ranking the object

returned by a partial-match query