NaresuanUniversity

Multimedia

Paisarn Muneesawang

http://www.ecpe.nu.ac.th/paisarn/multimedia

NaresuanUniversity

What is Multimedia?

• Apps that involve more than conventional data types (e.g., text, drawing and images)

• Best examples are continuous media (e.g., animations, audio, and video)– Called continuous media because of time basis

• <text document, graphic, image, audio, video>

NaresuanUniversity

Why is Multimedia?

• Human brain is much more efficient at processing and interpreting visual and audio information– than text, text+graphics

• Human beings and technology developments– image: 2-dimensional data– video: 3-dimensional data+audio– virtual reality

NaresuanUniversity

Challenges

• Multimedia data size

• Real-time nature of multimedia

• Why is the semantic nature of multimedia data a problem?

NaresuanUniversity

Multimedia Data Size

NaresuanUniversity

The First Challenge is Size

• A single good quality colored image could require 6 Mb.

• A video object consist of a sequence of such images (called frames) will be very large.– 30 frames per second, a five minute video clip would require 54 Gb.

• A typical sequence of audio will occupy 8 Kb for each second.

• Data size will effect the storage, retrieval and transmission of multimedia. Thus data compression techniques are crucial.

NaresuanUniversity

Multimedia Data Acquisition

DataCapture LossSampling Predict/

Model Transform Send

DelayReorderReceiveRestoreDisplay

NaresuanUniversity

Digitization of Data

• First Step: capturing analogue signal.

• Second Step: sampling the signal and convert it to digital value,– Theory of sampling by Nyquist: sampling rate >= (2)*(BW ) to accurately

reproduce an analog signal.

• Sampling rate data size

NaresuanUniversity

Multimedia System Design

• Should be designed based on human visual and audio sensory systems:– The human voice is in the frequency range 0-4kHz requiring 8kHz rate

for digitization.

– But, the intelligible part of human speech is carried in the voice band (300-3400 Hz). Thus, we can use smaller bandwidth.

• Should be designed based on applications:– Image in medical application may require higher resolution than images

transmitted on the web application.

NaresuanUniversity

Reducing the Media Object’s Size

• Compression is the technique to remove redundant data.– For example, by abbreviating any repeated information in an image and

eliminating information that is difficult for the human eye to see.

• Compression/decompression process– Lossy compression some information is loss by the process.– Lossless compression non of the original information is lost.

• Many compression methods attempt to address the same objectives:

– Reduced BW and/or storage;– Decode signal should be as close as possible to the original;– Lowest possible implementation strategy;– Application to as many signal types as possible;– Robustness;– Scalability;– extensibility

NaresuanUniversity

Media Data and Format Standards

Media data Special nature Typical digital file formats

Text data ASCII, RTF, HTML, SGML, XML

Audio data Time dependence with digitized data size can affect quality

WAV, AU, MPEG-MP3

Image data Results from pictures, drawings and photographs

JPEG, GIF, TIFF, BMP, PNG

Video data Time-dependent sequence of video frames

AVI, FLI, GIF, JPEG, MPEG

NaresuanUniversity

How to Choose the Formats

• Multimedia data is available in a range of file formats. Although this variety could appear confusing, this gives the designer great flexibility.

• We may select a format that:– Suit the application and the requirements of different users.

This computer-generated graphic image can be stored in several different file formats.

bit map: 1088 KbJPEG: 748 KbGIF: 109 Kb.

Note: JPEG is designed for photographs so it is not necessarily the best format for computer graphic files.

NaresuanUniversity

Digital File Formats, Applications and Features

Digital video format

Application Features

MPEG Multimedia Key frame plus motion encoding

INDEO Web pages Wavelet progression

CENEPAK Computer Video efficient

SORENSON Computer/videophone Video efficient

QUICKTIME Computer Incorporates compression

AVI Computer Incorporates compression

NaresuanUniversity

Important File Formats

• Image: JPEG is a standard of the ISO and can operate in four modes: sequential, progressive, lossless, and hierarchical.

• High quality compression for audio and video is the MPEG standards of the Moving Picture Expert Group.– MPEG-1: Interactive CDROM, video CDROM, Audio-MP3

– MPEG-2: Digital TV, DVD, Audio-Music

– MPEG-4: Multimedia for fixed and mobile web. (for low bandwidth applications)

– MPEG-7: Search video and audio content

– MPEG-21: Multimedia framework for interoperability

NaresuanUniversity

How Large is a Video Object

Time 640x480 320x240 160x120

Compressed

JPEG 25:1

640x480

Compressed MPEG 100:1

640x480

1 s 27 Mb 6.75 Mb 1.68 1.1 Mb 270 Kb

1 min 1.6 Mb 400 Mb 100 65 Mb 16 Mb

1 h 97 Gb 24 Gb 6 Gb 3.9 Gb 970 Mb

1000 h 97 Tb 24 Tb 6 Tb

NaresuanUniversity

Real-Time Nature of Multimedia

NaresuanUniversity

The Real-Time Nature of Multimedia

• Video and audio are continuous periodic media:– The frames of the video must run in the correct sequence and at an

acceptable rate, otherwise it becomes meaningless.

• The effect of time to the media objects is referred to as the real-time nature of multimedia.

• This relationship with time will have significant for the way the media objects are stored, retrieved, transmitted and synchronized.

NaresuanUniversity

Video Data

• Video is a continuous media but for database storage and manipulation such as random access it is important to be able to deal with portions of video object.

• Video Segmentation―cutting long video into portions:shot, scene, and clip

– Shot define a low level syntactic building blocks of video sequence.

– Scene is the logical grouping of shots into semantic unit.

– Clip is not clearly defined so it can last from a few seconds to several hours.

NaresuanUniversity

Video Segmentation

NaresuanUniversity

Story Clip

Scene

Shot a Shot b

NaresuanUniversity

Organization of Video Data

NaresuanUniversity

Why is the Semantic Nature of Multimedia Data a Problem?

NaresuanUniversity

Why is the Semantic Nature of Multimedia Data a Problem?

• “A picture is worth a thousand words”

Subjectivity:Different people have difference opinions.

Multimedia data, instead of having an explicit nature, have an implicit semantic nature.

NaresuanUniversity

Describing Multimedia Content

• Traditional database systems:

Metadata is know as:– Schema definitions, indexes, users, integrity constraints, security constraints etc.

.......Information that is of interest to the system itself.

• But, in multimedia database management systems (MMDBMS) metadata can refer to information about individual objects.

Metadata = Characteristics of the media data:

– Texture for images;– Frequencies for audio;– Speech ― keywords such as identification of the speakers, place, time;– Video clip ― camera motion and lighting

NaresuanUniversity

Techniques for Describing Multimedia

• Image annotation: a simple text description (keywords)– Time consuming for a large image database

– Difficult to describe an image precisely in text

• Content-based technique: An image is described by specific features in the image, such as color or texture.– Description is generated automatically

– Description is the content itself (low-level feature)

NaresuanUniversityImage Content with Low-Level Description

NaresuanUniversity

Metadata for Multimedia

• Text metadata + content descriptors

• A new standard MPEG-7 will have a major impact on the issue of metadata. MPEG-7 multimedia contents include:– Low-level descriptions of each individual object in a scene, such as

shape, color, and movement;

– High-level abstract descriptions of the scene, the objects it contains and the events taking place;

– Audio information such as key, mood and tempo.

• MPEG-7 makes the process of searching and retrieving

an image or a video clip much easier.

NaresuanUniversityKey-Word Search

NaresuanUniversity

Content-Based Retrieval

NaresuanUniversity

Content-Based Retrieval