E.G.M. Petrakis The MPEG Standard 1

The MPEG StandardMPEG-1 (1992) actually a video player

plays out audio/video streamssame type of access as home VCR

MPEG-2 (1995) introduced for compression and transmission of digital TV signalsstill limited interactivity

MPEG-4 (1999) is completely different high level of interactivity

MPEG-7 (2002) for the description of metadata only

E.G.M. Petrakis The MPEG Standard 2

MPEG-4MPEG-4 addresses the need towards

Mixing of natural and synthetic audiovisual information

High interactivity in the presentation of multimedia content

Deployment of communication systems for real-time or broadcast delivery of coded data streams

A new approach for describing, coding and presenting a scene

MPEG-4 combines different coding tools for Audio/video Synthetic objects and graphics

E.G.M. Petrakis The MPEG Standard 3

MPEG-4 ObjectsThe audio/video components of MPEG-4

Objects are coded, transmitted separately and composed at the decoder site

They can exist independentlyMultiple objects can be grouped together

to form complex objectsVideo and audio can be easily manipulatedPermits choosing appropriate coding tools

for audio, video and graphics objects

E.G.M. Petrakis The MPEG Standard 4

MPEG-4 Object Based Coding

E.G.M. Petrakis The MPEG Standard 5

MPEG-4 CodingThe scene is composed and rendered at the

sender sitevideo frames, audio are coded,

multiplexed and transmitted tools for coding arbitrarily shaped objects

At the receiver the stream is demultiplexedvideo and audio are decoded, composed,

synchronized and presented as defined at the senders site

E.G.M. Petrakis The MPEG Standard 6

Object CodingObjects are described mathematically

(e.g. by their positions)similarly for audio and graphics objectsan object need only be defined oncethe viewer can change their positiontransmit calculations to update the scene

at the receiverthis is a critical feature when the response

has to be fast and bit-rate is limited

E.G.M. Petrakis The MPEG Standard 7

Binary Format for Scenes (BIFS)

MPEG-4’s language for describing and dynamically changing a scene

Borrows concepts from VRMLBoth define representations of the same dataVRML defines objects and actions in textBIFS code is binary (10-15 times shorter)Unlike VRML, MPEG-4 uses BIFS for real-time

streaming: a scene can be built-up and played on the fly

VRML and BIFS evolve consistently

E.G.M. Petrakis The MPEG Standard 8

scene graph

E.G.M. Petrakis The MPEG Standard 9

The Scene GraphRepresents a scene as independent

or compound objects e.g., father and child the audio track of his voicefloor and walls (sprites: for backgrounds)the web site the synthetic image of the furniturea synthetic HDTV set playing a movie

from the families DVD library

E.G.M. Petrakis The MPEG Standard 10

Elementary Streams (ES)The scheme for preparing content for

transmission, storage and decodingObjects are placed in ESsProbably two or more ESs per objectA sound track or a video may have a single

ESScalable objects way have one ES for basic

quality information + one or more enhancement layers for improved quality (e.g., finer detail, faster motion)

ESs are split into packets and sent along with timing information for proper synchronization

E.G.M. Petrakis The MPEG Standard 11

Object Descriptors (OD)MPEG-4s mechanism that informs

the system which ES belongs to a certain objectOD contain Elementary Stream

Descriptors (ESD) which tell the system which decoders to use

ODs are sent in their own stream which allows them to be added or deleted as the scene changes

E.G.M. Petrakis The MPEG Standard 12

Profiles and LevelsMPEG-4 provides a set of tools for coding

multimedia contentsan application may use only subsets of these

toolsProfiles: MPEG-4s definitions of these

subsets for audio, visual, graphics information

Levels: define the computational complexity of the profile’s tool subset

Certain combinations of profiles fit well together

E.G.M. Petrakis The MPEG Standard 13

MPEG-4 Profiles

E.G.M. Petrakis The MPEG Standard 14

MPEG-4 Visual ObjectsArbitrarily shaped objects are coded apart

from their background Binary shape coding: a pixel is or is not

part of an object simple, crude technique, suitable for low-bit

rates, suffers from aliasingAlpha shape (gray scale) coding: each pixel

is assigned a value for its transparencyobjects can be smoothly blended into a

background or with other objects

E.G.M. Petrakis The MPEG Standard 15

Visual ObjectsRectangular natural images and

scenes are coded using MPEG-1, 2Texture is coded separately by a DCT,

block based coding scheme or wavelets

E.g., weather reports: the weatherman’s image seems to be standing in front of a map which is actually generated elsewhere

E.G.M. Petrakis The MPEG Standard 16

Object SegmentationMPEG does not specify how objects

are extracted video object segmentation is difficult e.g., record weatherman’s image in

front of a color background MPEG-4 specifies decoding

implementation of encoding is left to the industry to decide

E.G.M. Petrakis The MPEG Standard 17

MPEG-4 ApplicationsMPEG-4 makes video possible even

at very low bit-rates (e.g., 10 kb/s)mobile devices, internet

Scalable objects for low bit-ratesa base layer conveys all the information

in some basic quality one of more enhancement layers can be

sent to get better qualitysend only the most important objects

E.G.M. Petrakis The MPEG Standard 18

SpritesFor coding unchanged backgrounds The background is defined and coded

only onceMust be updated for each change (e.g.,

when the viewing angles changes) The sprite is sent only onceNew views are created by sending the

new positions

E.G.M. Petrakis The MPEG Standard 19

Advanced FeaturesMap images into computer generated shapes

a 2D or 3D mesh may have an image mapped onto ita few parameters to deform the mesh generate the

impression of a moving picture rather than sending new images for each change,

send commands and parameters to the viewerpre-defined faces are particularly interesting meshesthe appearance of a face may be left to the decoder

(e.g., custom facial models can be downloaded)

E.G.M. Petrakis The MPEG Standard 20

MPEG-4 FacesImages laid over a

wire-frame face Send wire-frame plus

parametersImage reconstruction

at receivers siteSpeech is generated

from text in steps with motions of the mouth, eyes and lips

E.G.M. Petrakis The MPEG Standard 21

MPEG-7MPEG-7 (2002) focuses on description of

multimedia contentmodalities: image, speech, video, graphics and

their combinationsMPEG-7 complements existing MPEG

standards and is applicable even to non-MPEG formats (compressed or uncompressed)

MPEG-7 is driven by trends in technology, market and user needs

Applications: VideoOnDemand, NewsOnDemand, InteractiveTV, multimedia information systems etc.

E.G.M. Petrakis The MPEG Standard 22

Scope of the StandardProvides the means for indexing,

searching, filtering and managing audio-visual contentbroadcast media selection (e.g.,

personalized TV)multimedia editing (e.g., personalized

news service)MPEG-7 interoperable interface

defines syntax and semantics tools may be designed for specific

modalities, aspects or applications

E.G.M. Petrakis The MPEG Standard 23

Interoperable Servicesand Applications

E.G.M. Petrakis The MPEG Standard 24

MPEG-7 Main TasksMultimedia: generate customized

program guides or summaries of broadcast audio-visual content

Archive: generate descriptions of audio-visual content (or elements)

Adaptation: filter and transform multimedia streams in low bit-rate environments (e.g., mobile users)

E.G.M. Petrakis The MPEG Standard 25

MPEG-7 Specific TasksMusic/audio: play a few notes and

return music with similar music/audioImages/graphics: draw a sketch and

return images with similar graphicsMovement: describe movements and

return video clips with the specified temporal and spatial relations

Scenario: describe actions and return scenarios where similar actions take place

E.G.M. Petrakis The MPEG Standard 26

MPEG-7 Elements1. Descriptors (D) : define syntax and

semantics of features of audio-visual content

Application independent Low level: shape, motion, color,

camera motion, harmonicity, timbre for audio ...

Semantic level: events, concepts ...

E.G.M. Petrakis The MPEG Standard 27

MPEG-7 Elements (cont.d)2. Description Schemes (DS): specify the

structure and semantics of the relationships among the constituent Ds or DSs e.g.,

Video DS specify syntax and semantics for segment decomposition, attributes, their relationships

DS related to creation, production, and access of content (e.g., property rights, parental rating, etc.)

E.G.M. Petrakis The MPEG Standard 28

MPEG-7 Elements (cont.d)3. Description Definition Language

(DDL): allows flexible definition of Ds and DSs based on XML schema

Ds and DSs are application independent

DLLs to define specialized tools

E.G.M. Petrakis The MPEG Standard 29

MPEG-7 DescriptionsMPEG-7 allows descriptions at

different levels of abstractionslow level features extracted automaticallysemantic features with human interaction

or textual annotationMPEG-7 does not specify how features

are extracted or used (e.g., filtering, retrieval)their representation must conform to the

MPEG-7 standard

E.G.M. Petrakis The MPEG Standard 30

MPEG-7 PartsSystems: specifies functionality at

system level Preparation of descriptions for efficient

transport and storage synchronization of content and descriptorsdevelopment of decoders

Description Definition Language (DDL): language for specifying new Ds and DSsextension of XML schema

E.G.M. Petrakis The MPEG Standard 31

MPEG-7 VisualSpecifies a set of standardized visual Ds

and DSsColor descriptors: color space, quantizationTexture descriptors: homogeneous texture,

texture browsing, edge histogram ...Shape descriptors: for regions or contoursMotion descriptors: camera motion,

trajectories, motion activity ...Face recognition

E.G.M. Petrakis The MPEG Standard 32

MPEG-7 AudioSpecifies standardized audio

descriptors and descriptor schemes for pure music, pure speech, sound effects, soundtrackssilence descriptor spoken content descriptors sound effects descriptorsmelody contour descriptors

E.G.M. Petrakis The MPEG Standard 33

Multimedia Description Schemes

Specify a framework that allows generic description of all kinds of multimedia data

basic elements: data types, structures, Ds content management: content from

several viewpoints (creation, usage etc.) organization of content by collections,

classification navigation and access user interaction

E.G.M. Petrakis The MPEG Standard 34

Multimedia Description Schemes

E.G.M. Petrakis The MPEG Standard 35

MPEG-7 Reference SoftwareReference implementation of the

relevant parts of the MPEG-7 standardThe focus is on creating bit-streams of

descriptors and description schemes (DDL parser, DDL validation, multimedia description schemes)

Some software for extracting descriptors is also included (visual, audio descriptors)

E.G.M. Petrakis The MPEG Standard 36

References1. “MPEG-4 Multimedia for our Time” R. Koenen,

IEEE Spectrum, Feb. 1999, pp. 26-332. “

Applying and Implementing the MPEG-4 Multimedia Standard”, J. Kneip et.al. IEEE Micro, Nov-Dec 1999, pp. 64-74

3. “Overview of the MPEG-7 Standard”, S.-Fu Chang, T. Sikora and A. Puri, IEEE Transactions on Circuits and Systems for Video Technology, special issue on MPEG-7, June 2001

4. “Everything You Wanted to Know about MPEG-7” F. Nack and A.T. Lindsay, Part I, II, IEEE Multimedia, Aug-Dec1999