8/2/2019 Mpeg Intro
1/35
4/20/2012 1
The MPEG Standard
MPEG-1 (1992) actually a video playerplays out audio/video streams
same type of access as home VCR
MPEG-2 (1995) introduced for compressionand transmission of digital TV signalsstill limited interactivity
MPEG-4 (1999) is completely differenthigh level of interactivity
MPEG-7 (2002) for the description ofmetadata only
8/2/2019 Mpeg Intro
2/35
4/20/2012 2
MPEG-4
MPEG-4 addresses the need towardsMixing of natural and synthetic audiovisual
informationHigh interactivity in the presentation of
multimedia contentDeployment of communication systems for real-
time or broadcast delivery of coded data streams
A new approach for describing, coding and
presenting a sceneMPEG-4 combines different coding tools forAudio/videoSynthetic objects and graphics
8/2/2019 Mpeg Intro
3/35
4/20/2012 3
MPEG-4 Objects
Theaudio/video components of MPEG-4Objects are coded, transmitted separately
and composed at the decoder site
They can exist independently
Multiple objects can be grouped togetherto form complex objects
Video and audio can be easily manipulatedPermits choosing appropriate coding tools
for audio, video and graphics objects
8/2/2019 Mpeg Intro
4/35
4/20/2012 4
MPEG-4 Object Based Coding
8/2/2019 Mpeg Intro
5/35
4/20/2012 5
MPEG-4 Coding
The scene is composed and rendered at thesender site
video frames, audio are coded, multiplexed
and transmittedtools for coding arbitrarily shaped objects
At the receiver the stream is demultiplexed
video and audio are decoded, composed,synchronized and presented as defined atthe senders site
8/2/2019 Mpeg Intro
6/35
4/20/2012 6
Object Coding
Objects are described mathematically(e.g. by their positions)similarly for audio and graphics objects
an object need only be defined once
the viewer can change their position
transmit calculations to update the scene
at the receiverthis is a critical feature when the response
has to be fast and bit-rate is limited
8/2/2019 Mpeg Intro
7/35
4/20/2012 7
Binary Format for Scenes (BIFS)
MPEG-4s language for describing anddynamically changing a scene
Borrows concepts from VRML
Both define representations of the same dataVRML defines objects and actions in text
BIFS code is binary (10-15 times shorter)
Unlike VRML, MPEG-4 uses BIFS for real-time streaming: a scene can be built-up andplayed on the fly
VRML and BIFS evolve consistently
8/2/2019 Mpeg Intro
8/35
4/20/2012 8
scenegraph
8/2/2019 Mpeg Intro
9/35
4/20/2012 9
The Scene Graph
Represents a scene as independent orcompound objects e.g.,father and child
the audio track of his voicefloor and walls (sprites: for backgrounds)the web site
the synthetic image of the furniturea synthetic HDTV set playing a movie from
the families DVD library
8/2/2019 Mpeg Intro
10/35
4/20/2012 10
Elementary Streams (ES)
The scheme for preparing content fortransmission, storage and decoding
Objects are placed in ESsProbably two or more ESs per objectA sound track or a video may have a single ESScalable objects way have one ES for basic
quality information + one or moreenhancement layers for improved quality (e.g.,finer detail, faster motion)
ESs are split into packets and sent along withtiming information for proper synchronization
8/2/2019 Mpeg Intro
11/35
4/20/2012 11
Object Descriptors (OD)
MPEG-4s mechanism that informs thesystem which ES belongs to a certainobjectOD contain Elementary Stream Descriptors
(ESD) which tell the system whichdecoders to use
ODs are sent in their own stream whichallows them to be added or deleted as thescene changes
8/2/2019 Mpeg Intro
12/35
4/20/2012 12
Profiles and Levels
MPEG-4 provides a set of tools forcoding multimedia contentsan application may use only subsets of
these toolsProfiles: MPEG-4s definitions of these
subsets for audio, visual, graphicsinformationLevels: define the computational
complexity of the profiles tool subsetCertain combinations of profiles fit well
together
8/2/2019 Mpeg Intro
13/35
4/20/2012 13
MPEG-4 Profiles
8/2/2019 Mpeg Intro
14/35
4/20/2012 14
MPEG-4 Visual Objects
Arbitrarily shaped objects are coded apartfrom their background
Binary shape coding: a pixel is or is not part
of an objectsimple, crude technique, suitable for low-bit rates,
suffers from aliasing
Alpha shape (gray scale) coding: each pixel isassigned a value for its transparencyobjects can be smoothly blended into a background
or with other objects
8/2/2019 Mpeg Intro
15/35
4/20/2012 15
Visual Objects
Rectangular natural images and scenesare coded using MPEG-1, 2
Texture is coded separately by a DCT,block based coding scheme or wavelets
E.g., weather reports: the weathermansimage seems to be standing in front of amap which is actually generatedelsewhere
8/2/2019 Mpeg Intro
16/35
4/20/2012 16
Object Segmentation
MPEG does not specify how objects areextractedvideo object segmentation is difficult
e.g., record weathermans image in front ofa color background
MPEG-4 specifies decodingimplementation of encoding is left to the
industry to decide
8/2/2019 Mpeg Intro
17/35
4/20/2012 17
MPEG-4 Applications
MPEG-4 makes video possible even atvery low bit-rates (e.g., 10 kb/s)mobile devices, internet
Scalable objects for low bit-ratesa base layer conveys all the information in
some basic quality
one of more enhancement layers can besent to get better qualitysend only the most important objects
8/2/2019 Mpeg Intro
18/35
4/20/2012 18
Sprites
For coding unchanged backgrounds
The background is defined and codedonly once
Must be updated for each change (e.g.,when the viewing angles changes)
The sprite is sent only onceNew views are created by sending the
new positions
8/2/2019 Mpeg Intro
19/35
4/20/2012 19
Advanced Features
Map images into computer generated shapesa 2D or 3D mesh may have an image mapped onto it
a few parameters to deform the mesh generate
the impression of a moving picturerather than sending new images for each change,
send commands and parameters to the viewer
pre-defined faces are particularly interesting
meshesthe appearance of a face may be left to the
decoder (e.g., custom facial models can bedownloaded)
8/2/2019 Mpeg Intro
20/35
4/20/2012 20
MPEG-4 Faces
Images laid over awire-frame face
Send wire-frame plus
parametersImage reconstruction
at receivers site
Speech is generatedfrom text in stepswith motions of themouth, eyes and lips
8/2/2019 Mpeg Intro
21/35
4/20/2012 21
MPEG-7
MPEG-7 (2002) focuses on description ofmultimedia contentmodalities: image, speech, video, graphics and their
combinations
MPEG-7 complements existing MPEGstandards and is applicable even to non-MPEGformats (compressed or uncompressed)
MPEG-7 is driven by trends in technology,
market and user needsApplications: VideoOnDemand,NewsOnDemand, InteractiveTV, multimediainformation systems etc.
8/2/2019 Mpeg Intro
22/35
4/20/2012 22
Scope of the Standard
Provides the means for indexing,searching, filtering and managing audio-visual contentbroadcast media selection (e.g.,
personalized TV)multimedia editing (e.g., personalized news
service)MPEG-7 interoperable interface defines
syntax and semanticstools may be designed for specific
modalities, aspects or applications
8/2/2019 Mpeg Intro
23/35
4/20/2012 23
Interoperable Servicesand Applications
8/2/2019 Mpeg Intro
24/35
4/20/2012 24
MPEG-7 Main Tasks
Multimedia: generate customizedprogram guides or summaries ofbroadcast audio-visual content
Archive: generate descriptions of audio-visual content (or elements)
Adaptation: filter and transformmultimedia streams in low bit-rateenvironments (e.g., mobile users)
8/2/2019 Mpeg Intro
25/35
4/20/2012 25
MPEG-7 Specific Tasks
Music/audio: play a few notes andreturn music with similar music/audioImages/graphics: draw a sketch and
return images with similar graphicsMovement: describe movements and
return video clips with the specified
temporal and spatial relationsScenario: describe actions and return
scenarios where similar actions takeplace
8/2/2019 Mpeg Intro
26/35
4/20/2012 26
MPEG-7 Elements
1. Descriptors (D) : define syntax andsemantics of features of audio-visualcontent Application independent
Low level: shape, motion, color, cameramotion, harmonicity, timbre for audio ...
Semantic level: events, concepts ...
8/2/2019 Mpeg Intro
27/35
4/20/2012 27
MPEG-7 Elements (cont.d)
2. Description Schemes (DS): specify thestructure and semantics of therelationships among the constituent
Ds or DSs e.g., Video DS specify syntax and semantics
for segment decomposition, attributes,
their relationships DS related to creation, production, and
access of content (e.g., property rights,parental rating, etc.)
8/2/2019 Mpeg Intro
28/35
4/20/2012 28
MPEG-7 Elements (cont.d)
3. Description Definition Language (DDL):allows flexible definition of Ds andDSs based on XML schema Ds and DSs are application independent
DLLs to define specialized tools
8/2/2019 Mpeg Intro
29/35
4/20/2012 29
MPEG-7 Descriptions
MPEG-7 allows descriptions at differentlevels of abstractionslow level features extracted automatically
semantic features with human interactionor textual annotation
MPEG-7 does not specify how featuresare extracted or used (e.g., filtering,retrieval)their representation must conform to the
MPEG-7 standard
8/2/2019 Mpeg Intro
30/35
4/20/2012 30
MPEG-7 Parts
Systems: specifies functionality atsystem levelPreparation of descriptions for efficient
transport and storagesynchronization of content and descriptors
development of decoders
Description Definition Language (DDL):language for specifying new Ds and DSsextension of XML schema
8/2/2019 Mpeg Intro
31/35
4/20/2012 31
MPEG-7 Visual
Specifies a set of standardized visualDs and DSsColor descriptors: color space, quantization
Texture descriptors: homogeneous texture,texture browsing, edge histogram ...Shape descriptors: for regions or contoursMotion descriptors: camera motion,
trajectories, motion activity ...Face recognition
8/2/2019 Mpeg Intro
32/35
4/20/2012 32
MPEG-7 Audio
Specifies standardized audiodescriptors and descriptor schemes forpure music, pure speech, sound effects,
soundtrackssilence descriptor
spoken content descriptors
sound effects descriptorsmelody contour descriptors
8/2/2019 Mpeg Intro
33/35
4/20/2012 33
Multimedia Description Schemes
Specify a framework that allowsgeneric description of all kinds ofmultimedia data
basic elements: data types, structures, Ds content management: content from
several viewpoints (creation, usage etc.) organization of content by collections,
classification navigation and access user interaction
8/2/2019 Mpeg Intro
34/35
4/20/2012 34
Multimedia Description Schemes
8/2/2019 Mpeg Intro
35/35
4/20/2012 35
MPEG-7 Reference Software
Reference implementation of therelevant parts of the MPEG-7 standardThe focus is on creating bit-streams of
descriptors and description schemes (DDLparser, DDL validation, multimediadescription schemes)
Some software for extracting descriptorsis also included (visual, audio descriptors)
Top Related