MUMIS

User Group Workshop

P. Wittenburg

Max-Planck-Institut für PsycholinguistikNijmegen

Agenda

• 11.00 Introduction Peter WittenburgASR Aspects Helmer

StrikNLP/IE Aspects Thierry

Declerck• 11.45 Demo of Prototype Peter Wittenburg• 12.30 Lunchbreak• 13.15 Potential of MUMIS Technology Discussion• 14.00 Typical Questions Discussion• 15.00 Other Points of Interest Discussion• 15.30 End

Objectives

MUMIS is about

• Technology development to automatically index lengthy multimedia recordings

Find and mark interesting scenes

• Technology development to exploit indexed multimedia archives

Search for interesting scenes and play them via Internet

• Test Domain: Soccer Games / UEFA Tournament 2000

Off-line Tasks

Indexing by

• Automatic Speech Recognition (Radio/TV Broadcasts)

Transfer the speech signals into texts automatically

• Natural Language ProcessingTake all texts (newspapers, speech transcript, tickers, ...)and determine interesting scenes

• Create a database with formal annotations

• Use video processing to adjust time marks

Off-line Tasks

indexed video recording

Radio Commenting3 Languages

Radio Commenting3 Languages

Radio Commenting3 Languages

Audio Commenting (TV, Radio)3 Languages

NewspaperText

NewspaperText

NewspaperText

NewspaperTexts

3 Languages

NewspaperText

NewspaperText

NewspaperText

Close Captions3 Languages

Freekick Goal Pass Defense

17 min 18 min 24 min 28min

Foul Freekick Dribbling

Kohler Basler Matthäus Wörns

Basler Bierhoff

25 m 25 m 60 m

Event Table

Event ID Time Player1 Player2 Location Destination Cause Consequence Count RefEnd-Signal ID Time Ref

Offside ID Time Player1 Location RefGoal-Kick ID Time Player1 Player2 Location Destination RefStop-of-Attack

ID Time Player1 Location Cause Consequence Ref

Kick-Off ID Time Player1 Player2 RefChest Ball ID Time Player1 Player2 Location Destination Consequence RefWall Pass ID Time Player1 Player2 Location Destination Consequence RefDribbling ID Time Player1 Location Consequence Ref

Corner ID Time Player1 Player2 Location Destination Cause Consequence RefOwn-Goal ID Time Player1 Location Cause Score RefThrow-In ID Time Player1 Player2 Location Destination Cause Consequence RefNice Ball ID Time Player1 Player2 Location Destination Consequence Ref

Foul ID Time Player1 Player2 Location Cause Consequence RefFreekick ID Time Player1 Player2 Location Destination Cause Consequence Ref

Yellow Card ID Time Player1 Player2 Cause RefHalftime Signal

ID Time Ref

Hands Ball ID Time Player1 Location Cause Consequence RefHead Ball ID Time Player1 Player2 Location Destination Consequence Ref

Top-Post-Ball ID Time Player1 Location Cause RefPass ID Time Player1 Player2 Location Destination Consequence Ref

Post Ball ID Time Player1 Location Cause RefSave ID Time Player1 Player2 Location Cause Ref

Red Card ID Time Player1 Player2 Cause Consequence RefBicycle Kick ID Time Player1 Location Destination Consequence Ref

Sliding ID Time Player1 Player2 Location Consequence RefPenalty ID Time Player1 Player2 Cause Consequence Ref

Goal ID Time Player1 Player2 Cause Score RefShot-on-Goal ID Time Player1 Player2 Location Cause Consequence RefKeeper-Save ID Time Player1 Player2 Location Cause Ref

Injury ID Time Player1 Player2 Location Cause Consequence RefChange ID Time Player1 Player2 Cause Ref

On-line Tasks

Searching and Displaying

• Search for interesting events with formal questions

Give me all goals from Overmars shot with his head in 1. Half.

Event=Goal; Scorer=Overmars; Cause=Head; Time<=45

• Indicate hits by thumbnails & let user select scene

• Play scene via the Internet & allow scrollingOf course: slow motion, fast play, start/stop, etc

On-line Tasks

Knowledge GuidedUser Interface

&Search Engine

München - Ajax1998

München - Porto1996

Deutschland - Brasilien1998

PlayMovie

Fragmentof that Game

Freekick Goal Pass Defense

17 min 18 min 24 min 28min

Foul Freekick Dribbling

Kohler Basler Matthäus Wörns

Basler Bierhoff

25 m 25 m 60 m

On-line SW Architecture

first interaction with Web-server

download of appletcontact between applet

and distributed Java server via RMI

contact via media server via RMI and media streaming

annotations from DB or file server

JMF serves for merging of streams on client

Abstract Corpus Model

this kernel is operational(distributed and local)

On-line HW Architecture

• efficient & reliable storage management (near-line capacity, media change, 2. Location)

• high storage capacity (n TB, 1 h MPEG1 = 1 GB)• powerful media servers / powerful network

2nd locationMirror site

RAID

TapeLibrary3.5 TB

FC Switch

Media Server

Media Server

GB Switch

Internet

MM Stations Notebook

1Gbps

100Mbps

Gb-Switch

TapeLibrary25 TB

Gain

• What gets lost? Is it necessary?• Potential: direct Internet Service, less

dependencies

Current Procedure MUMIS Procedure

Manual Video Annotation Automatic Video Annotation and DB

IntegrationIntegration Central DB

Query via PC Query via PC

Results on PC

Results on PCAnd

Select & Play

Contact Video Archive

Get Video Tapes

Search on Tape on VCR

Segment & Play

Formal Aspects

• Start: 1.7.2000• Duration: 30 Months• EC-Funds: 2.4 Mio Euro

Consortium• CTIT University of Twente, Enschede, NL NLP/IE• TSI University of Nijmegen, Nijmegen, NL ASR• DFKI Saarbrücken, D NLP/IE• MPI Nijmegen, NL Online

Part• DCS University of Sheffield, UK

NLP/IE• ESTEAM Gothenburg, SE (location Athens, GR)

Translation Software

• VDA Hilversum, NLDissemination

Acknowledgements

• UEFA

• DFB, FA, KNVB

• EBU, WDR, NOS