02/Mar/2001 MPEG-7 Applications: Multimedia Content Retrieval, IMA Workshop 6B, University of...

02/Mar/2001 MPEG-7 Applications: Multimedia Content Retrieval, IMA Workshop 6B, University of Minnesota

Winter 2001: IMA Workshop 6BDigital Libraries: Classification, Retrieval and Visualization

MPEG-7 Applications: Multimedia Content Retrieval

Neil Day

Digital Garage Inc. Japan

ChairMPEG-7 Applications and Promotions to

Industry (MAPI)

Contactneil@garage.co.jp

MPEG-7 & Industry

MAPI MPEG-7 Applications & Promotion to Industry

MIFG MPEG-7 Industry Focus Group

www.mpeg-7.com

M7AE MPEG-7 Awareness Event

Paris Oct. 2000 Singapore March 2001

MAPI & MIFG & M7AE

Industry/Academia MPEG-7MAPI & IFG & M7AE

MPEG-7 Applications

About MPEG-7

Application Examples

Views on Next Steps

About MPEG•ISO/IEC SC29Wg11 Standards Group

•MPEG-1 began in 1988

•Today, over 300 developers attend working conferences

•Average 4 conferences per year

•Over 35 National Delegations Vote

•MPEG-7 Standard – September 2001

Content is Diversifying

20002000

20032003

TextText

AudioAudio

VideoVideo

GraphicsGraphics

TextText

AudioAudio

VideoVideo

GraphicsGraphics

TextText

GraphicsGraphics19961996

TextText

GraphicsGraphics

19971997

Changing Bandwidth Usage

Data: Inktomi, USA

MPEG-7 Scope

Feature Extraction:Content analysis (D, DS)

Feature extraction (D, DS)

Annotation tools (DS)

Authoring (DS)

MPEG-7 Scope:Description Schemes (DSs)

Descriptors (Ds)

Language (DDL)

Ref: MPEG-7 Concepts

Search Engine:Searching & filtering

Classification

Manipulation

Summarization Indexing

Feature SearchExtraction Engine

MPEG-7Description

standardization

Defining MPEG-7MPEG-7 is a standard

for describing features of multimedia content

• Comprehensive Set of AV descriptions based on:• Catalogue (e.g., title, creator, rights), • Semantic (e.g., the who, what, when, where information about objects and

events)• Structural (e.g., the colour histogram –measurement of the amount of colour

associated with an image or the timbre of an recorded instrument) features of the AV content

• Leverages on AV data representation defined by MPEG-1, 2 and 4

• Interoperability• MPEG-7 uses XML Schema as the language of choice for content description• MPEG-7 will be interoperable with other leading standards such as, SMPTE• Metadata Dictionary, Dublin Core, EBU P/Meta, and TV Anytime.

MPEG Standards

• MPEG-1: For the storage and retrieval of moving pictures and audio on storage media.• MPEG-2: For digital television, it’s the timely response for the satellite broadcasting and cable television industries in their transition from analog to digital formats.• MPEG-4: Codes content as objects and enables those objects to be manipulated individually or collectively on an audiovisual scene.MPEG-1, -2, and -4 make content available. MPEG-7 lets you to find the content you need.

MPEG-7 & XML

Over 100 XML Industry Standard GroupsXML Repository (www.xml.org)

What Makes MPEG-7 Different?

Survey of MPEG-7 Applications & Issues

Some Application Domains

Digital libraries, Education (image catalogue, musical dictionary, Bio-medical imaging catalogues…)l Multimedia editing (personalised electronic news service, media authoring)l Cultural services (history museums, art galleries, etc.),l Multimedia directory services (e.g. yellow pages, Tourist information, Geographical information systems)l Broadcast media selection (radio channel, TV channel,…)l Journalism (e.g. searching speeches of a certain politician using his name, his voice or his face),l E-Commerce (personalised advertising, on-line catalogues, directories of e-shops,…)l Surveillance (traffic control, surface transportation, non-destructive testing in hostile environments, etc.),l Investigation services (human characteristics recognition, forensics),l Home Entertainment (systems for the management of personal multimedia collections, including manipulation of content, e.g. home video editing, searching a game, karaoke,…)l Social (e.g. dating services),

Typical MPEG-7 Queries• Audio:

• Users want to search for songs by humming or whistling a tune.

• Graphics: • Sketch a few lines on a screen and get in return a set of images containing

similar graphics, logos, and ideograms.

• Image: • Define objects, including colour patches or textures and get in return• examples among which users select interesting objects to compose their

image.• Check if a company logo was advertised on a TV Channel as contracted

• Video: • Allow access by mobile phone to video clips of goals scored in a soccer

game. • Automatically search and retrieve any unusual movements from

surveillance videos

MPEG-7 Technical Activities• A set of descriptors:

• A Descriptor (D) is a representation of a Feature. A Descriptor defines the syntax and the semantics of the Feature representation.

• A set of description schemes: • A Description Scheme (DS) specifies the structure and semantics of the relationships

between its components, which may be both Ds and DSs.

• DDL: • A language to specify description schemes, the Description Definition Language

(DDL). It also allows the extension and modification of existing Description Schemes. MPEG-7 decided to adopt XML Schema Language as the MPEG-7 DDL. However the DDL will require some specific extensions to XML Schema Language to satisfy all of the requirements of MPEG-7. These extensions are currently being discussed through liaison activities between MPEG and W3C, the group standardizing XML.

• Binary Representation: • One or more ways (e.g., textual, binary) to encode descriptions. A Coded Description

is a Description that has been encoded to fulfill relevant requirements such as

compression efficiency, error resilience, random access, etc.

MPEG-7 Application Context

ContentManagement

ApplicationsLayer

ContentDescription

ativeN

on-norm

D(s), DS(s), DDL

MM Content & Standards

DDLDescriptions

Internet Streaming Query-by-Sketch

Query-by-Humming

ApplicationDomain

MM Descriptions

MPEG-7Description Tools

Navigation &Access

Summary

Variation

Content management

Content description

Creation &Production

Media Usage

Conceptualaspects

Structuralaspects

Userinteraction

Userpreferences

ModelCollection &Classification

Content organization

Datatype &structures

Link & medialocalization

Basic DSs

Basic elements

SchemaTools

Organisation: MPEG-7 Descriptions

Description Schemes in MPEG-7

Creation and Production: Contains meta information that describes the creation and production of the content; typical features include title, creator, classification, and purpose of the creation.

Usage: Contains meta information that’s related to the usage of the content; typical features involve rights holders, access rights, publication, and financial information.

Media: Contains the description of the storage media; typical features include the storage format, the encoding of the AV content, and elements for the identification of the media.

Structural aspects: Contains the description of the AV content that represent physical, spatial, temporal, or spatio-temporal components of the AV content. (color, texture, shape, motion, audio).

Conceptual Aspects: Contains a description of the AV content from the viewpoint of its conceptual notions.

Basic Elements: deal with basic data types, mathematical structures, schema tools, linking and media localization tools, as well as basic DSs, which are elementary components of more complex DSs.

OtherFace Recognition

LocalizationBounding BoxRegion LocatorSpatio-Temporal Locator

MotionCamera MotionMotion TrajectoryParametric MotionMotion Activity

ShapeRegion ShapeContour Shape3D Shape2D-3D Multiple View

TextureHomogeneous Text.Texture BrowsingEdge histogram

ColorGoF/GoP ColorScalable ColorColor LayoutColor StructureDominant Color

Visual Description Tools

Image Features in MPEG-7

3. Spatial edge distribution

2. Spatial color distribution

4. Object shape

1. Color histogram

Image: Thanks to Ricoh Co. Ltd.

Video Segment Decomposition

Video Segment

Segment Decomposition

Moving Regions

• Time• Mosaic• Text Annotation

• Time• Color• Motion• Texture• Shape• Text Annotation

Segment Relation

Video Segments

Before

Segment Features

Video Segment

Moving Region

Moving Regions

• Time• Mosaic• Annotation

• Time• Color• Motion• Texture• Shape• Annotation

Relation Link

Video Segments

Another Example

MPEG-7 Applications

Application Examples

MPEG-7 Visual Annotation Tool(IBM, USA)

Visual Annotation Example

1. Drag a Still Region element from the Definition Tree to the Description List area.

2. Click Add for Media Information,3. Click Add for Media Profile,4. Click Add for Media Format,5. Enter values for File Format, Medium, Color, …6. Click Add of Media Coding to Media Profile7. Enter values for Frame Width and Frame Height8. Click Add of Media Instance to Media Profile9. Enter value for Identifier10. Click Add of Creation Meta Information, …

Example Package-<Package title="Sports Program Package">

</Package><Package title="Production and Copyright Package">

<scheme name="Creation" type="element" /> <scheme name="Classification" type="element" /> <scheme name="MediaInformation" type="element" /> <scheme name="MediaProfile" type="complexType"

schemeRecurse="false" /> <scheme name="MediaFormat" type="complexType" />

</Package> </Package>

Annotation Process

MPEG-7Schema / Package

0.2 0.2 0.6 0.1 0.8 0.1 0.3 0.3 0.4</Transitions>

<State><Label> Pass </Label></State><State><Label> Shot on goal </Label></State><State><Label> Goal score </Label></State>

</StateTransitionModel>

</sequence></complexType>

Parser

MPEG-7Visual Annotation

Tool Interface

MPEG-7 Schema Definition (DDL) MPEG-7 Package

MPEG-7 Elements

MPEG-7 Description

User Input

MP EG- 7 Sche ma

D ef ini t ion A rea

Descri pt ionEn try Area

M PE G- 7 D escri pti on Area:1. R eusabl e el em ents2. Elem ent s t r ee

3. Vali dat ed de scri pti on tr ee4. MP EG -7 descr ipt io n ( XM L)

Descr ip ti on cop ying

MPEG-7 Camera ApplicationSmart Fire Alert

(Fastcom Technologies, Switzerland)

Alarm Event Management

XML log file (search & retrieve, conversion to HTML, WML)

Alarm transmission via GSM

Remote User Interface

Encrypted transmission

MPEG-7 Camera Features

MVS-135 based platform

Embedded image processing for motion detection and tracking

MPEG-7 encoder: XML formatted description of the motion

Serial transmission

MPEG-7 decoder: XML parser

Application of the MPEG-7 Camera

Broadcast Camera Controler

Initial Coarse Grain Metadata generator for:

• Automatic Document Analyser

• Digital Video from Broadcast camera

Use the technology to create MPEG-7 Broadcast Camera with MPEG-2 (4) bitstream output.

Highlight Generation Tool (ETRI, Korea)

FeaturesFeatures Video content analysis: multi-modal features extractionVideo content analysis: multi-modal features extraction

visual features (camera motion, embedded text, shot boundary, slow motion replay, face)

Textual information (natural language processing using closed-caption text)

speech recognition CC text DB population for news videoCC text DB population for news video Highlights detectionHighlights detection

Knowledge-based detection rules tuned based on genre such as sports, news and drama programs, etc.

MPEG-7 Summary DescriptionMPEG-7 Summary Description Summarization DS (Hierarchical Summary DS)

Highlight Generation ToolHighlight Generation Tool

Media Data Extractor Description EncoderMPEG-7 File Formatter

Summary Server

User Interface

DDLfeature index

Evaluator

Video Sequence

Video Data

MPEG-7 Browser

Feature IndexHS List

HS List

CC Text

Extractor

Key Sound

ExtractorShot BoundaryVisual FeaturesCC Text

SportsNews

Highlight Detection

M7Description Data

Using the Highlight Tool

Web-based MPEG-7 Browser (ETRI, Korea)

Applications: Highlight Tool

Applications: Highlight Generation ToolApplications: Highlight Generation Tool Advanced EPG generationAdvanced EPG generation Video content analysisVideo content analysis Video editing, authoringVideo editing, authoring CC text DB populationCC text DB population

Applications: BrowserApplications: Browser Agent for Agent for Storage-based STBStorage-based STB News video retrievalNews video retrieval Web-based video browserWeb-based video browser

Humming: Elements of Melody

(MIT Media Lab, USA)

Series of related pitches Invariant to transposition Invariant to instrumentation

Rhythm is important

“Bridal Chorus” from Lohengrin(“Here Comes the Bride”)

“O Tannenbaum”

Humming: Example

“Moon River”

Contour: 2 -1 -1 -1 -1 -1 1

Beat Number: 1 4 5 7 8 9 9 10

Humming: System Overview

Server

ScoresMIDIs

Melody

Representation

PlaybackClient

Pitch tracking

Construct query

Streaming over Internet

(Singingfish.com, USA)

Create XSL transformation

From: MSNBC "Partner XML Format"

To:MPEG-7 <Mpeg7Main> root element

Design

Stream Delivery Network

M SN BCPartner X M L Feed

Java FTP Servlet

W ork flowR D BM S

SearchIndex

Prom otionFTP

SchedulerX SL

Engine

Im porter

Prom otionJDBC

Singingfish.comSearch Engine

Prom otion

MSNBC <article>

<filename>tdy_fletcher_mideast_001023</filename>

<headline>Peace hopes slip farther</headline>

<description>The slim hopes for peace in the Mideast are rapidly fading, NBCs Martin Fletcher reports Monday from the outskirts of Jerusalem.</description>

<keywords>Israel, palestinian, Yasser Arafat</keywords>

...</article>

Link to stream

<MediaURI>http://www.msnbc.com/news/asx/video/28/tdy_fletcher_mideast_001023.asx

</MediaURI> </OnlineLocator>

</InstanceLocator></MediaInstance>

<headline>Peace hopes slip farther</headline>

<Title>

</Title>

</Creation>

< description>, <keywords><description>The slim hopes for peace in the Mideast are

rapidly fading, NBCs Martin Fletcher reports Monday from the outskirts of Jerusalem.</description>

<keywords>Israel, palestinian, Yasser Arafat</keywords>

<xsl:value-of select="description"/> </FreeText>

</FreeTerm></WhatObject></Abstract>

MPEG-7Application(DENSO, JAPAN)

Wireless Images Retrieval using Speech Dialogue Agent The agent in the client terminal recognizes user's utterance in

English/Japanese with rather dedicated sentences and send a query profile to the server using wireless transceiver channel (32kbps). The server PC will retrieve the requested images and deliver the compressed video bitstream (H.263) to the client PC. Then the client agent will reply with synthesized voice and display the images.This time we use MPEG-7 CD based format of metadata. Moreover, retrieval speed is several times higher than conventional methods by newly

incorporating dependency information.

How aboutYokohamaInterchange?

Wireless Image Retrieval by Speech Dialogue

32 - 64kbps

Client

PHS-PIAFS Transceiver Mode

This is the current statusof Yokohama Interchange.

PublicNetwork

Base StationServer

：：：

Mobile ApplicationDialogue & Contents Retrieval

System

1. Agent acquires user’s situations and demands.2. Agent automatically sends profiles to centers. 3. Situations are roughly transmitted to centers

before utterance. 4. User can receive safely and conveniently

desirable information.

InteractiveAgent

InformationCenter

Delivery

User’s Preferenceshobby, favorite place,dish, music,display, volume, color, etc.

How about Nagoya Interchange ?

One family restaurant found one kilometer to the southward.

Not crowded right now.

Environments & Situationsseason, current location, destination, crews, purpose, weather, traffic conditions, etc.

Demands & Statuswhat user wants to know, what user wants to do,physical conditions, etc.

ProfilesProfilesDataset

Query &Information

ProfilesProfilesDataset

Are there any restaurants nearby the East Park ?

Mobile: Dialogue & Contents Retrieval System

Scene Retrieval: Sports Events

Ricoh Co. Ltd., Japan

•Aerobatic Movies

•Motegi, Japan 2000

•Search Queries

•Pilot, Figures,

•Aircraft

•URL

•http://ab-movie.ricoh.co.jp

Organisationsnewly interested in MPEG-7

(25th/January – 25th/February 2001)

ARKIVE

www.arkive.org.uk

An initiative of The Wildscreen Trust, ARKive will be the world's electronic archive of photographs, moving images and sounds of endangered species and habitats. It will relate closely to the work of the World Conservation Monitoring Centre (WCMC).

The Digital Library Federation (DLF) is a consortium of research libraries that are transforming themselves and their institutional roles by exploiting network and digital technologies.

http://www.clir.org/diglib/

•Developing and applying appropriate architectures, technologies, systems, and tools;

•Developing sustainable, scaleable, and useful digital collections and services;

•Developing support and other services that enable the digital library to respond to its users' information requirements;

•Gaining experience preserving digital information; and identifying and promoting those standards and practices that enable the digital library cost effectively to develop and maintain its collections and services;

•Gaining a better understanding of the digital library's institutional roles, responsibilities, and potential developmental trajectories

3D-MURALE3D Measurement & Virtual Reconstruction of

Ancient Lost Worlds of Europe

www.brunel.ac.uk/project/murale

EU: IST Project

Create MPEG-7 Database

Sagalassos, Turkey

MPEG-7: (Mitsubishi, Japan)

Visual Access Technologies with MPEG-4 and MPEG-7

"contents search and retrieval" employing MPEG-4 (for contents compression) and MPEG-7 (for contents access).

Kohtaro Asai (asai@isl.melco.co.jp)

MPEG-7 Applications (NEC, Japan)

NEC has developed a real-time video identification system based on MPEG-7 descriptions. This system monitors broadcast TV programs and identifies its contents through more than 50,000 different video segments in real-time. The system requires only one PC and software. This system is especially useful for survey applications of video clips like TV commercial programs, music video clips, etc. Furthermore, it enables home video server applications to access to a desired program and information.

e-kasutani@cq.jp.nec.com

MPEG-7: NECReal-time Video Identification System

MPEG-7: Emblaze

MPEG-4 and MPEG-7

www.emblaze.com

MPEG-7: NTTDocomoCustomized Content Delivery for Mobile Users In the next-generation wireless communication services, multimedia content delivery should be one of the most attractive applications for both mobile users and the content providers. Due to the limited time to play the content at mobile client and huge amount of multimedia content data, it is essential to support the functionality of flexible content summarization depending on the user’s situation. In this application scenario, push technology is used for sending summary content to the mobile users to provide pull access trigger for rich content. Summary content contains an instance of MPEG-7(content structure or pre-defined summaries) and media data(e.g. key frames, annotation) which enables mobile user to look at the overview of the whole content quickly. The information for pull access can be generated or selected using delivered MPEG-7 instance and user preferences input to the client.

Contact: Shun-ichi Sekiguchi,

Email: shun@mml.yrp.nttdocomo.co.jp

MPEG-7 Applications - next

Views on next stepsDigital (Multimedia) LibrariansKey Partners Needed

Search EnginesContent Providers

Mobile Search & Retrieval

InktomiSearch 4.0CCE & Dublin Core

CCE: Content Classification Engine

Inktomi Search CCE (formerly known as Ultraseek Server CCE) helps administrators to establish a system of classification, and to then create a taxonomy.

Example:http://www.inktomi.com

University of Minnesota: Dept. of Natural Resources

Dublin Core & MPEG-71 Title A name given to the

resourceCreationInformation/Creation/Title[@type="original"]

2 Creator An entity primarily responsible for making the content of the resource

CreationInformation/Creation/Creator

3 Subject The topic of the content of the resource

CreationInformation/Classification/Subject

4 Description An account of the content of the resource

Creation/Abstract ContentDescription/Creation/Abstract

5 Publisher An entity responsible for making the resource available

CreationInformation/Creation/Creator [Role/@term="Publisher"]

6 Contributor An entity responsible for making contributions to the content of the resource

CreationInformation/Creation/Creator [Role/@term="Contributor"]

7 Date A date associated with an event in the life cycle of the resource

CreationInformation/Creation/CreationCoordinates/CreationDate

8 Type The nature or genre of the content of the resource

CreationInformation/Classification/Genre

Dublin Core & MPEG-79 Format File format or mime type

[MPEG-1, QuickTime, RealVideo…)

MediaInformation/MediaProfile/MediaFormat/FileFormat

10 Identifier An unambiguous reference to the resource within a given context

MediaInformation/MediaProfile/MediaInstance/InstanceIdentifier

11 Source A Reference to a resource from which the present resource is derived

MediaInformation/MediaIdentification/EntityIdentifier(the “reality”)

12 Language A language of the intellectual content of the resource

CreationInformation/Classifcation/Language

13 Relation A reference to a related resource

CreationInformation/RelatedMaterial/MediaLocator

14 Coverage The extent or scope of the content of the resource

CreationInformation/Creation/Abstract orCreationInformation/Creation/CreationCoordinates/CreationLocation or CreationInformation/Classification/TargetCountry

15 Rights Information about rights held in and over the resource

UsageInformation/Rights/RightsID (Currently, RightsID is simply a UniqueIDType. Further identification of the rights holder awaits MPEG-7 IPMP development.)

Content Provider ExamplesWhole web coverage

Super-fresh mp3 index

Worlds largest full length music video collection

6200+ Videos from leading artists

Broad range of video content

Rich video data

Web's premier source of legitimate MP3s

Only signed artists from leading labels

MPEG-7 Applications - Summary

MPEG-7 StandardMultimedia Content Description

Structural and Semantic

Key Partner Types NeededContent Providers, Search Engines

Key App - Mobile Search & Retrieval

Contacts in MPEG-7

For more information about MPEG-7 http://www.cselt.it/mpeg

MPEG-7 Industry Focus Group http://www.mpeg-7.com

MPEG-7 Awareness EventSingapore, March 10th, 2001

Acknowledgments

The author is extremely grateful to the MPEG-7 developers, teams and companies that provided him with information on their MPEG-7 application development and activities. The author is also thankful to the support of Digital Garage Inc., his employer, for allowing him the time to survey and compile the information about current MPEG-7 applications being created throughout the world.

Thank you!!

Dr. Willard Miller

Dr. B.S. Manjunath

Michelle Glubke

02/Mar/2001 MPEG-7 Applications: Multimedia Content Retrieval, IMA Workshop 6B, University of...

Documents

Transcript of 02/Mar/2001 MPEG-7 Applications: Multimedia Content Retrieval, IMA Workshop 6B, University of...

Capitulo 6b

Death Register 1901 Elphick · 2020. 12. 22. · Victor Henry. O Leed Victoria Maud. O Violet Rosey. O Chelsea rot. 3b. 3b. 6b. 6b. 6b. 11 n. 6b. 9b. 6b. 3b. 3b. 2b. 10 b. 10b. 6b.

Vokiški žodžiai lietuvių kalbos tarmėse¡ki...PARENGĖ : GUSTAS LUKŠYS (6A) TADAS GRIKIS (6B) GABRIELIUS KAZAKEVIČIUS (6B) EDVARDAS MARCINONIS (6B) NOJUS SASNAUSKAS (6B) MOKINIAMS

Lecture 6b

Physics 6B

news88 A TAKEI- KEYS A 1200B 6B F 7 2001 …news88 A TAKEI- KEYS A 1200B 6B F 7 2001" 2001ã8Ê4a 5B 2001" : 2001ã8Ê5a (E) I He€ñ-: TEL03-5742-7542/FAX03-5742-7543 2001 TEL …

Lecture 6b

DoD Financial Management Regulation Volume 6B, Chapter 12 ... · DoD Financial Management Regulation Volume 6B, Chapter 12 + November 2001 120103. Appropriation Groupings for the

of Prize Winners 201… · FONG Fai-wang HONG Nan TSANG Nga-jing WONG Yat-kiu, Kevin Ll Yin-hei NG Cheuk-yin 11 . Secondary 6 TWGSS Academic Scholarships 6B 6B 6B 6B 6B 6B 6B 6B 6B

Romanticismo 6B

IMA > IMA ESC 2003

24 September 2001 - USNA24 September 2001 (3) Approving or disapproving leave donor applications according to the criteria outlined in paragraph 6b and notifying leave donors of the

IMA > IMA 2001

Session 6b

Poesías 6b

ZOMBIE 1st ONEMAN TOUR 6B *ÉThumbUp 6B …v-kei.jp/image/system/pc/flyer/20200408_zombie.pdf2020/04/08 · ZOMBIE 1st ONEMAN TOUR 6B *ÉThumbUp 6B 19B(â) 6B 17 a ZËRe11.FITS ALL

dirigentie 6B

Mamitas 6b

ROUND LAKE VID6002/98 Native Title Application ......2001 2002 2001 2001 2005 9A 2004 2006 3F 3E 6B 3D 12A 3H 1 3 5 3 4 2 2 4 15C 34A 22C 6B 33B 11B 23C 5 3K 3L 2007 ALBACUTYA KURNBRUNIN

IMA Committees as constituted by 215th IMA ... - ima-india.org

ZOMBIE 1st ONEMAN TOUR 6B ÉThumbUp 6B …v-kei.jp/image/system/pc/flyer/20200408_zombie.pdf2020/04/08 · ZOMBIE 1st ONEMAN TOUR 6B ÉThumbUp 6B 19B(â) 6B 17 a ZËRe11.FITS ALL