1E.G.M. Petrakis Introduction

Technical University of CreteDepartment of Electronic and Computer

Engineering

Multimedia Data Management

Euripides G.M. Petrakishttp://www.intelligence.tuc.gr/~petrakis

http://courses.ced.tuc.gr

Chania 2010

2E.G.M. Petrakis Introduction

Definition

Multimedia: composite entities combining text, audio, images, video (bit-stream objects), graphics

Multimedia Information Systems: database systems that support all multimedia data types and handle very large volumes of information

3E.G.M. Petrakis Introduction

TEXT

The most common type of information

The least space intensive data typeThe form in which text is stored

varies (plain ascii, word files, spreadsheets, annotations, database fields etc.)

Text fonts are becoming complex allowing special effects (color, shade, fill etc.)

4E.G.M. Petrakis Introduction

AUDIO Space intensive (one minute can take up

Mbytes), presented as analog, digital or MIDI

Analog waveform: electrical signal, amplitude specifies the loudness of the sound in microphones, tapes, records, amplifiers,

speakersDigital waveform audio: digital,

less sensitivity to noise and distortion involves larger processing and storage

capacitiesDigital Audio Tape (DAT), Compact discs (CD)WAV (Microsoft’s wave file format)

5E.G.M. Petrakis Introduction

MIDI (Musical Instrument Digital Interface)

Commands that describe how the music should be played are stored (instead of sound)

A music synthesizer generates sound Provides high data compression, Widely accepted

Furht et.al.96

6E.G.M. Petrakis Introduction

IMAGES

Digital images: sequences of pixelsPixels: numbers interpreted to

display intensity, color, contrast etc Binary (0-1 values), gray-scale (8

bits/pixel), colour (3x8 values for RGB)Space overhead depends on image

type, resolution, compression scheme

Image formats: tiff, bmb, jpeg etc.

7E.G.M. Petrakis Introduction

Image Concepts and Structures

Binary images: 1 bit/pixel black & white photos, facsimile images

Computer Graphics: 4 bits/pixelGrayscale images: 8 bits/sampleColor images: 16, 24 bits/pixel

8E.G.M. Petrakis Introduction

RGB Representation

A color is produced by addingred, green and blue

The straight line R=G=B specifies gray values ranging from black to white

9E.G.M. Petrakis Introduction

YUV Representation

YUV describes the luminance and chrominance components of an image1 luminance: gray scale version of an

imageY = 0.299R + 0.587G + 0.114B

2 chrominance components:U = 0.564(B - Y)V = 0.713(R - Y)

10E.G.M. Petrakis Introduction

Conversions

Conversion between RGB and YUV requires multiplication operationsan approximation: Y = R/4 + G/2 +B/2, U=(B-Y)/2, V=(R-

Y)/2R = Y + 2V, G = Y – (U + V), B = Y + 2U

YCbCr is another color format for compressionCb = U/2 + 0.5, Cr = V/1.6 + 0.5

11E.G.M. Petrakis Introduction

VIDEO

The most space intensive data typeA sequence of framesRealistic video playback, transmission,

compression/decompression require transfer rates about 30frames/sec

Microsoft’s AVI and Apple’s Quicktime file formats integrate video and audio in the same presentation

12E.G.M. Petrakis Introduction

Audio-video Modes of Operation

Can be either stored or used / transmitted live in real-time

Can be used interactively or non-interactively

Furht et.al.96

13E.G.M. Petrakis Introduction

GRAPHICS

Objects described through their basic elements (e.g., 2D, 3D shapes)these elements can have different sizes,

position, orientation, surface, fill etc. compact representationsgenerated and can be manipulated by

design tools (e.g., CAD tools)Their descriptions are stored in files

14E.G.M. Petrakis Introduction

Khoshafian Baker 96

2D and 3D graphics objects

15E.G.M. Petrakis Introduction

MULTIMEDIA objects

Text, audio, images, video, graphics are elements of complex multimedia objects

Various tools or applications integrate, process and combine multimedia

Applications: multimedia authoring applications that output documents and databases and end-user applications (e.g., video on demand)

Tools: for viewing, updating, querying (presentation viewers, browsers etc.)

16E.G.M. Petrakis Introduction

Multimedia Databases (MDB)

Means stored information or database management systems (dbms)

Multimedia dbms (mdbms) integrate conventional database capabilities together with different technologies such as Hierarchical storage management (HSM) and Information retrieval (IR)

17E.G.M. Petrakis Introduction

Multimedia Technologies

Technologies integrated within a mdbms HSM support IR support (exact and approximate)Spatial data types and queriesInteractive querying, relevance feedback,

refiningAutomatic feature extractionAutomatic content retrieval and indexing Query optimization

18E.G.M. Petrakis Introduction

Database capabilities

Persistence: object persist through invocations Transactions: content is inserted, deleted,

updated Concurrency control: transactions run

concurrently Recovery: failed transactions are not propagated

to the db Querying: content can be retrieved Versioning: access previous states of objects Integrity: transactions guarantee consistency of

content Security: constraints for accessing/updating

objects Performance: optimal data structures and

programs

19E.G.M. Petrakis Introduction

Hierarchical Storage Management (HSM)

Support storage of multimedia objects On-line: on RAM, magnetic diskNear-on-line: on optical storageOff-line: on tapes, shelves

Each level has differentPerformance: decreases from top to

bottomCapacity: increases from top to bottomCost: decreases from top to bottom

20E.G.M. Petrakis Introduction

Information Retrieval (IR) Capabilities

Retrieval is the most common operationDeletions and updates are less commonExact match: search based on exact

informationInexact: search based on inexact information

e.g., partial, neighborhood search, can be fuzzy or probabilistic

The results are ranked by order of relevance to the query

Query refinementIterate over query results Adjust weights of query terms or featuresAnd finally resubmit queries

21E.G.M. Petrakis Introduction

Khoshafian Baker 96

MDBMS architecture

22E.G.M. Petrakis Introduction

MDBMS Implementation

Relies on 3rd party vendors for each componentRelational dbms for typical records separate optical storage module for

text/audio/graphics/images/videoText retrieval system (e.g. Lucene)Audio/image/video retrieval systemFeature extraction systemMultimedia object interface system

23E.G.M. Petrakis Introduction

Object-Oriented Multimedia Databases

Better design, better suited for multimedia applications Uniform handling of data and operationsData types are objects with internal structures

and operations that capture the behavior of objects (e.g., audio playback, video browsing)

OO dbms does not satisfy all MM requirementsProvides primitives for object handling Multimedia components need to be

implemented or integrated

24E.G.M. Petrakis Introduction

Multimedia Applications

Multimedia Systems suggest a variety of applicationsMultimedia conferencing Multimedia on demand (interactive TV,

news on demand)See next page for more …

25E.G.M. Petrakis Introduction

Furht et.al.96

Multimedia Applications

26E.G.M. Petrakis Introduction

Multimedia Conferencing (MC)

Multimedia conferencing enable a number of participants to exchange multimedia informationEach participant has a workstation

linked to other workstations over high-speed networks

Each participant can send or receive mm data and perform certain collaborative activities

27E.G.M. Petrakis Introduction

Furht et.al.96

The biggest performance challenge occurs when the participants transmit voice and videoThese are mixed together to form a composite

stream consisting of video and voice streams

A video conference system

28E.G.M. Petrakis Introduction

Furht et.al.96

Software Architecture

29E.G.M. Petrakis Introduction

Architectures

Fully distributed: direct connections between the participants Processing and mixing of media at every

locationShortest delayThe connections increase rapidly

Centralized (star) network: a central is connected to every participantProcessing and mixing at central nodeThe central node waits until all media is

received before mixing and broadcasting

30E.G.M. Petrakis Introduction

Architectures (cont.)

Double star network: a central node from one star network is connected to another central node of another star network

Hierarchical network: intermediate nodes, root and leaves (participants) connected hierarchicallyintermediate nodes perform mixing and

processingthe completely mixed data is sent to root who

broadcasts directly to the leavesreduces network traffic significantly

31E.G.M. Petrakis Introduction

Furht et.al.96

Multimedia conferencing network architectures

32E.G.M. Petrakis Introduction

Video on Demand (VoD)

Fast networks coupled with powerful computers and compression techniques will be capable of delivering stream data in real-time

On-demand multimedia servicesinteractive entertainment video news distributionvideo rental services digital multimedia libraries

33E.G.M. Petrakis Introduction

Interactive Television (ITV)

An ITV system must be capable of providingbasic TVsubscription TVpay per viewvideo on demandshoppingeducationelectronic newspaperfinancial transactionssingle-user and multi-user games

34

This Course

Emphasis on Text, images, video Information retrieval & systems Data organization Web information systems Semantic Web Video & MPEG standards

No emphasis on Architectures Specific applications (VoD, ITV,MC) Services

E.G.M. Petrakis Introduction