8/2/2019 Mpeg Intro

1/35

4/20/2012 1

The MPEG Standard

MPEG-1 (1992) actually a video playerplays out audio/video streams

same type of access as home VCR

MPEG-2 (1995) introduced for compressionand transmission of digital TV signalsstill limited interactivity

MPEG-4 (1999) is completely differenthigh level of interactivity

MPEG-7 (2002) for the description ofmetadata only

8/2/2019 Mpeg Intro

2/35

4/20/2012 2

MPEG-4

MPEG-4 addresses the need towardsMixing of natural and synthetic audiovisual

informationHigh interactivity in the presentation of

multimedia contentDeployment of communication systems for real-

time or broadcast delivery of coded data streams

A new approach for describing, coding and

presenting a sceneMPEG-4 combines different coding tools forAudio/videoSynthetic objects and graphics

8/2/2019 Mpeg Intro

3/35

4/20/2012 3

MPEG-4 Objects

Theaudio/video components of MPEG-4Objects are coded, transmitted separately

and composed at the decoder site

They can exist independently

Multiple objects can be grouped togetherto form complex objects

Video and audio can be easily manipulatedPermits choosing appropriate coding tools

for audio, video and graphics objects

8/2/2019 Mpeg Intro

4/35

4/20/2012 4

MPEG-4 Object Based Coding

8/2/2019 Mpeg Intro

5/35

4/20/2012 5

MPEG-4 Coding

The scene is composed and rendered at thesender site

video frames, audio are coded, multiplexed

and transmittedtools for coding arbitrarily shaped objects

At the receiver the stream is demultiplexed

video and audio are decoded, composed,synchronized and presented as defined atthe senders site

8/2/2019 Mpeg Intro

6/35

4/20/2012 6

Object Coding

Objects are described mathematically(e.g. by their positions)similarly for audio and graphics objects

an object need only be defined once

the viewer can change their position

transmit calculations to update the scene

at the receiverthis is a critical feature when the response

has to be fast and bit-rate is limited

8/2/2019 Mpeg Intro

7/35

4/20/2012 7

Binary Format for Scenes (BIFS)

MPEG-4s language for describing anddynamically changing a scene

Borrows concepts from VRML

Both define representations of the same dataVRML defines objects and actions in text

BIFS code is binary (10-15 times shorter)

Unlike VRML, MPEG-4 uses BIFS for real-time streaming: a scene can be built-up andplayed on the fly

VRML and BIFS evolve consistently

8/2/2019 Mpeg Intro

8/35

4/20/2012 8

scenegraph

8/2/2019 Mpeg Intro

9/35

4/20/2012 9

The Scene Graph

Represents a scene as independent orcompound objects e.g.,father and child

the audio track of his voicefloor and walls (sprites: for backgrounds)the web site

the synthetic image of the furniturea synthetic HDTV set playing a movie from

the families DVD library

8/2/2019 Mpeg Intro

10/35

4/20/2012 10

Elementary Streams (ES)

The scheme for preparing content fortransmission, storage and decoding

Objects are placed in ESsProbably two or more ESs per objectA sound track or a video may have a single ESScalable objects way have one ES for basic

quality information + one or moreenhancement layers for improved quality (e.g.,finer detail, faster motion)

ESs are split into packets and sent along withtiming information for proper synchronization

8/2/2019 Mpeg Intro

11/35

4/20/2012 11

Object Descriptors (OD)

MPEG-4s mechanism that informs thesystem which ES belongs to a certainobjectOD contain Elementary Stream Descriptors

(ESD) which tell the system whichdecoders to use

ODs are sent in their own stream whichallows them to be added or deleted as thescene changes

8/2/2019 Mpeg Intro

12/35

4/20/2012 12

Profiles and Levels

MPEG-4 provides a set of tools forcoding multimedia contentsan application may use only subsets of

these toolsProfiles: MPEG-4s definitions of these

subsets for audio, visual, graphicsinformationLevels: define the computational

complexity of the profiles tool subsetCertain combinations of profiles fit well

together

8/2/2019 Mpeg Intro

13/35

4/20/2012 13

MPEG-4 Profiles

8/2/2019 Mpeg Intro

14/35

4/20/2012 14

MPEG-4 Visual Objects

Arbitrarily shaped objects are coded apartfrom their background

Binary shape coding: a pixel is or is not part

of an objectsimple, crude technique, suitable for low-bit rates,

suffers from aliasing

Alpha shape (gray scale) coding: each pixel isassigned a value for its transparencyobjects can be smoothly blended into a background

or with other objects

8/2/2019 Mpeg Intro

15/35

4/20/2012 15

Visual Objects

Rectangular natural images and scenesare coded using MPEG-1, 2

Texture is coded separately by a DCT,block based coding scheme or wavelets

E.g., weather reports: the weathermansimage seems to be standing in front of amap which is actually generatedelsewhere

8/2/2019 Mpeg Intro

16/35

4/20/2012 16

Object Segmentation

MPEG does not specify how objects areextractedvideo object segmentation is difficult

e.g., record weathermans image in front ofa color background

MPEG-4 specifies decodingimplementation of encoding is left to the

industry to decide

8/2/2019 Mpeg Intro

17/35

4/20/2012 17

MPEG-4 Applications

MPEG-4 makes video possible even atvery low bit-rates (e.g., 10 kb/s)mobile devices, internet

Scalable objects for low bit-ratesa base layer conveys all the information in

some basic quality

one of more enhancement layers can besent to get better qualitysend only the most important objects

8/2/2019 Mpeg Intro

18/35

4/20/2012 18

Sprites

For coding unchanged backgrounds

The background is defined and codedonly once

Must be updated for each change (e.g.,when the viewing angles changes)

The sprite is sent only onceNew views are created by sending the

new positions

8/2/2019 Mpeg Intro

19/35

4/20/2012 19

Advanced Features

Map images into computer generated shapesa 2D or 3D mesh may have an image mapped onto it

a few parameters to deform the mesh generate

the impression of a moving picturerather than sending new images for each change,

send commands and parameters to the viewer

pre-defined faces are particularly interesting

meshesthe appearance of a face may be left to the

decoder (e.g., custom facial models can bedownloaded)

8/2/2019 Mpeg Intro

20/35

4/20/2012 20

MPEG-4 Faces

Images laid over awire-frame face

Send wire-frame plus

parametersImage reconstruction

at receivers site

Speech is generatedfrom text in stepswith motions of themouth, eyes and lips

8/2/2019 Mpeg Intro

21/35

4/20/2012 21

MPEG-7

MPEG-7 (2002) focuses on description ofmultimedia contentmodalities: image, speech, video, graphics and their

combinations

MPEG-7 complements existing MPEGstandards and is applicable even to non-MPEGformats (compressed or uncompressed)

MPEG-7 is driven by trends in technology,

market and user needsApplications: VideoOnDemand,NewsOnDemand, InteractiveTV, multimediainformation systems etc.

8/2/2019 Mpeg Intro

22/35

4/20/2012 22

Scope of the Standard

Provides the means for indexing,searching, filtering and managing audio-visual contentbroadcast media selection (e.g.,

personalized TV)multimedia editing (e.g., personalized news

service)MPEG-7 interoperable interface defines

syntax and semanticstools may be designed for specific

modalities, aspects or applications

8/2/2019 Mpeg Intro

23/35

4/20/2012 23

Interoperable Servicesand Applications

8/2/2019 Mpeg Intro

24/35

4/20/2012 24

MPEG-7 Main Tasks

Multimedia: generate customizedprogram guides or summaries ofbroadcast audio-visual content

Archive: generate descriptions of audio-visual content (or elements)

Adaptation: filter and transformmultimedia streams in low bit-rateenvironments (e.g., mobile users)

8/2/2019 Mpeg Intro

25/35

4/20/2012 25

MPEG-7 Specific Tasks

Music/audio: play a few notes andreturn music with similar music/audioImages/graphics: draw a sketch and

return images with similar graphicsMovement: describe movements and

return video clips with the specified

temporal and spatial relationsScenario: describe actions and return

scenarios where similar actions takeplace

8/2/2019 Mpeg Intro

26/35

4/20/2012 26

MPEG-7 Elements

1. Descriptors (D) : define syntax andsemantics of features of audio-visualcontent Application independent

Low level: shape, motion, color, cameramotion, harmonicity, timbre for audio ...

Semantic level: events, concepts ...

8/2/2019 Mpeg Intro

27/35

4/20/2012 27

MPEG-7 Elements (cont.d)

2. Description Schemes (DS): specify thestructure and semantics of therelationships among the constituent

Ds or DSs e.g., Video DS specify syntax and semantics

for segment decomposition, attributes,

their relationships DS related to creation, production, and

access of content (e.g., property rights,parental rating, etc.)

8/2/2019 Mpeg Intro

28/35

4/20/2012 28

MPEG-7 Elements (cont.d)

3. Description Definition Language (DDL):allows flexible definition of Ds andDSs based on XML schema Ds and DSs are application independent

DLLs to define specialized tools

8/2/2019 Mpeg Intro

29/35

4/20/2012 29

MPEG-7 Descriptions

MPEG-7 allows descriptions at differentlevels of abstractionslow level features extracted automatically

semantic features with human interactionor textual annotation

MPEG-7 does not specify how featuresare extracted or used (e.g., filtering,retrieval)their representation must conform to the

MPEG-7 standard

8/2/2019 Mpeg Intro

30/35

4/20/2012 30

MPEG-7 Parts

Systems: specifies functionality atsystem levelPreparation of descriptions for efficient

transport and storagesynchronization of content and descriptors

development of decoders

Description Definition Language (DDL):language for specifying new Ds and DSsextension of XML schema

8/2/2019 Mpeg Intro

31/35

4/20/2012 31

MPEG-7 Visual

Specifies a set of standardized visualDs and DSsColor descriptors: color space, quantization

Texture descriptors: homogeneous texture,texture browsing, edge histogram ...Shape descriptors: for regions or contoursMotion descriptors: camera motion,

trajectories, motion activity ...Face recognition

8/2/2019 Mpeg Intro

32/35

4/20/2012 32

MPEG-7 Audio

Specifies standardized audiodescriptors and descriptor schemes forpure music, pure speech, sound effects,

soundtrackssilence descriptor

spoken content descriptors

sound effects descriptorsmelody contour descriptors

8/2/2019 Mpeg Intro

33/35

4/20/2012 33

Multimedia Description Schemes

Specify a framework that allowsgeneric description of all kinds ofmultimedia data

basic elements: data types, structures, Ds content management: content from

several viewpoints (creation, usage etc.) organization of content by collections,

classification navigation and access user interaction

8/2/2019 Mpeg Intro

34/35

4/20/2012 34

Multimedia Description Schemes

8/2/2019 Mpeg Intro

35/35

4/20/2012 35

MPEG-7 Reference Software

Reference implementation of therelevant parts of the MPEG-7 standardThe focus is on creating bit-streams of

descriptors and description schemes (DDLparser, DDL validation, multimediadescription schemes)

Some software for extracting descriptorsis also included (visual, audio descriptors)