Multimedia ProjectsOur Experience

Research Projects

NSF Multimedia Laboratory at Florida Atlantic University

(1995-2001)

Director Dr. Borko Furht

ProjectsContent-Based Image Retrieval Using Relevance

Feedback (Oge Marques) IP Simulcast - An Innovative Video and Audio

Broadcasting Technique Over the Internet (Ray Westwater and Jeff Ice)

XYZ Video Encoding Technique (Ray Westwater & Joshua Greenberg)

An Innovative Motion Estimation Algorithm for MPEG Codec (Joshua Greenberg)

A Fast Content-Based Multimedia Retrieval Technique (P. Saksobhavivat)

Interactive Progressive Encoding System (Joe Celi)

Internet Broadcasting or WebcastingBroadcasting multimedia data (audio

and video) over the Internet - from a server (sender) to a large number of clients (receivers)

Applications include: radio and television broadcasting

real-time broadcasting of critical data distance learning videoconferencing database replication electronic software distribution

Broadcast Pyramid Applied in IP Simulcast

SERVER

Client 1

Client 2

Client 3

Client 2

2

Client 4

Client 5

Client 6

Client 9

Client 10

Client 11

Client 12

Client 13

Client 8

Client 14

Client 7

IP Simulcast - An Innovative Technique for Internet Broadcasting

IP Simulcast reduces the server (or sender) overhead by distributing the load to each client (or receiver)

Each receiver becomes a repeater, which rebroadcasts its received content to two child receivers

The needed network bandwidth for the server is significantly reduced

Characteristics of IP Simulcast

It is a radically different model of digital broadcast, referred to as repeater-server model

The server manages and controls the interconnection of repeaters

Each repeater not only plays back the data stream, but also transmits the data to two other repeaters

IP Simulcast provides guaranteed delivery of packets, which is not the case with IP Multicast

Product: AllCastwww.allcast.com

Broadcasting Tree

Once the AllCast Broadcaster is configured, many users can connect to hear/view content. The bandwidth usage is distributed across the participants, as illustrated by the dynamic, self-healing dissemination tree shown in the AllCast main window.

Microsoft Media Player with AllCast Plug-in

•Users can connect to a broadcast using the Microsoft Windows Media Player together with a small, seamlessly integrated AllCast Plug-in. •The plug-in enables the Windows Media Player to participate in peer-to-peer broadcasts.

XYZ - New Video Compression Technique

The XYZ video compression algorithm is based on 3D Discrete Cosine Transform (DCT)

It provides very high compression ratios and excellent video quality

It is very suitable for real-time video compression

Forming Video Cube for XYZ Compression

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

Frames 0 1 2 3 4 5 6 7

8X8X8videocube

Block Diagram of the XYZ Compression

Forward3-DDCT

Quantizer EntropyCoder

QuantizingTables

HuffmanTable

VideoCube

Compressedvideosequence

Key Encoding Equations Both encoder and decoder are symmetrical, which makes

the algorithm suitable for VLSI implementation

Fuv w CuCvCw f x y z

x u y v z w

zyx

( , , ) , ,

cos cos cos

0

7

0

7

0

7

2 1

16

2 1

16

2 1

16

(4.1)

FuvwFuvw

Quvwq ,,,,

,,

(4.3)

XYZ Versus MPEGVideo

CompressionTechnique

CompressionRatio

NormalizedRMSError

ExecutionTime [min]

(8 frames, 320x240)

XYZ(QT1)

34.5 0.079 6.45

XYZ(QT2)

57.7 0.097 6.45

XYZ(QT3)

70.8 0.105 6.45

XYZ(QT4)

101.7 0.120 6.45

XYZ(QT5)

128.1 0.130 6.45

MPEGLogarithmic Search andError Correction

11.0 0.080 21.35

MPEGExhaustive Search andError Correction

15.6 0.080 163.0

MPEGLogarithmic Search andNo Error Correction

27.0 0.140 21.35

MPEGExhaustive Search andNo Error Correction

32.9 0.125 163.0

Complexity of Video Compression Techniques

CompressionAlgorithm

EncoderComplexity

DecoderComplexity

TotalComplexity

H.261/H.263 970 200 1,170MPEGNo B Frames

750 100 850

MPEG70% B Frames

1,120 120 1,240

XYZ 240 240 480

XYZ Versus MPEG

Original

MPEGCr=11,NRMSE=0.08

MPEG, Motion Est. onlyCr=27, NRMSE=0.14

XYZCr=45, NRMSE=0.079

Examples of XYZ Compression

Original

XYZ-compressedCr=51

Examples of XYZ Compression

OriginalXYZ-compressedCr=110

Sensitivity of the XYZ Algorithm to Various Video Effects

Movie ClipVideo Effect

CompressionRatio

Normalized RMSEError

Dick TracyTypical Motion

34.5 0.079

Interview with theVampireCamera Break

32.5 0.087

Interview with theVampireCamera Panning

26.1 0.085

Total RecallCamera Panning

17.5 0.049

Total RecallCamera Zoom

23.9 0.042

InterceptorFast Motion

26.0 0.025

Characteristics of XYZ Video CompressionXYZ gives significantly better

compression ratios than MPEG for the same quality of video

For similar compression ratios, XYZ gives much better quality than MPEG

XYZ is faster than MPEG (lower complexity)

XYZ is simple for implementation

Applications of the XYZ

Interactive TV and TV set-top boxesTV phoneVideo broadcasting on the InternetVideo-on-demand applicationsVideoconferencingWireless video

TV Phone Videophone is a box on the top of TV with a small Videophone is a box on the top of TV with a small

camera, modem, and video/audio codec.camera, modem, and video/audio codec.

Videophone Videophone

Conventionaltelephone network

TV set TV set

Design of the TV Phone

VIDEOPHONE

TV

Video/audio capture and compression

Video-in

Video/audio out

Video/audio out

Video/audio in

Modem Telephone lines

User'scontrol panel

Video/audiodecompressionand conversion to TV format

VLSI chipsimplement XYZ algorithm

Camera

A Fast Content-Based Multimedia Retrieval Technique

Two main approaches in indexing and retrieval of images and videos

Keyword-based indexing and retrieval

Content-based indexing and retrieval

Keyword-Based Retrieval and Indexing

Uses keywords or descriptive text, which is stored together with images and videos in the database

Retrieval is performed by matching the query, given in the form of keywords, with the stored keywords

This approach is not satisfactory - the text-based description is incomplete, imprecise, and inconsistent in specifying visual information

New Algorithm for Similarity-Based Retrieval of Images

Images in the database are stored as JPEG-compressed images

The user submits a request for search-by-similarity by presenting the desired image.

The algorithm calculates the DC coefficients of this image and creates the histogram of DC coefficients.

The algorithm compares the DC histogram of the submitted image with the DC histograms of the stored images.

Histogram of DC Coefficients for the Image “Elephant”

Comparison of Histograms of DC Coefficients

Example of Similarity-Based Retrieval Using the DC Histograms

Similarity-Based Retrieval of Compressed Video

Partitioning video into clips - video segmentation

Key frame extractionIndexing and retrieval of key frames

DC Histogram Technique Applied for Video Partitioning

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Frame number

NSD

x 1

00 [%

]

Example of Similarity-Based Retrieval of Key Frames Using DC Histograms

Interactive Progressive Encoding System

Users submit requests for imagery to the image database via a graphical user interface

Upon an initial request, a DCT image (version of the image based on DC coefficients only) is transmitted and reconstructed at the user site.

The user can then isolate specific regions of interests within the image and request additional levels of details.

Band Transmission in Interactive JPEG System Based on Spectral Selection

All blocks of the image

Amplitude of Coefficients Blocks of the selected region of the image

Band-1 Band-2 Band-3 Band-4

DC AC1 AC2 AC3 …… AC6 AC7 ………. AC63

Transmission

Prototype System - IPES and Experimental Results

Original image “Airport”

Interactive Progressive Transmission in Four Scans

Selection of Two Regions

Cumulative Number of Transmitted Bits

0

200

400

600

800

1000

1200

1400

1600

1800

2000

1 2 3 4

Scans

Cum

ulat

ive

tran

smitt

ed b

its [K

bits

]

Whole Image

1 Region

2 Regions

Extracted Images From a Group of Images

Applications

Retrieval and transmission of complex images over low bandwidth communication channels (image transmission over the Internet, real-time transmission of medical images)

Archiving and browsing visually lossless image databases (medical imaginary, space exploration and military applications)

Content-Based Retrieval

Large, complex, and ever growing, distributed, mostly unstructured, multimedia repositories

Three ways of retrieving multimedia information:– Free browsing (inefficient, time-consuming, doesn’t

scale well)– Text-based retrieval (relies on metadata, time-

consuming, subjective)– Content-based retrieval (requires intelligent

interpretation of the contents)

Design of MUSE System

Image Analysis

Image Feature Extraction-Color-Shape-Texture

Image Representation & Feature Organization

ImageArchive

User

GUI-Image selection-Result viewing

Probability recalculation & candidate ranking

Feature ExtractionSimilarity comparison

Interactive learning& Display update

Off-line Online

Query By Example

Example image

Best result

Similarity Score [0,1]

Relevance Feedback

Good

Bad

Neither

Relevance Feedback - Next

Technology Behind the MUSE System Feature extraction

Extraction of relevant image features impacts the overall performance of the system.

MUSE uses:– color-related features (color histograms, color

space partitioning and/or quantization, color moments, color coherence vectors)

– texture-related features (Multiresolution Simultaneous Autoregressive Model - MSAR)

– frequency-related features (DFT, DCT)

Technology Behind the MUSE System Bayesian formulation

MUSE is based on a Bayesian framework for relevance feedback.

During each iteration of a MUSE session, the system displays a subset of images from its database, and the user takes an action in response, which the system observes.

Based on the user’s actions, the probability distribution over possible targets is refined. (Most systems refine the user’s query)

The best candidates are then displayed back.