Audiovisual content exploitation in the networked information society

Roeland Ordelman

Research&Development

Netherlands Institute for Sound and Vision

rordelman@beeldengeluid.nl

contents

• NISV context: Images of the Future• “Access”, an important keyword in the

business models ...• ... but what about access in practice• Technology and user interaction: from

a ‘laboratory view’ on users to drawing them into the development chain

NISV context

• +700.000 hours of radio, television, documentaries, films and music, over 2 million photographs, 20.000 objects like cameras, televisions, radios, costumes and pieces of scenery.

• still growing:• digitally born television and radio programs made

by the Dutch public broadcasting companies (video: 15K/hours/year)

• PROARCHIVE: archiving service• selection of (Dutch) user generated content

Images of the Future

• Selection, restoration, digitization, encoding and storage of 137,000 hours of video, 20,000 hours of film, 124,000 hours of audio and more than three million photographs.

• One of the largest digitisation effort in Europe• Three goals:

• Safeguarding heritage for future generations• Creating social- economical value (“unlock the social

and economic potential of the collections”)• Innovation: new infrastructure for strengthening

knowledge economy• To achieve these objectives, the cultural heritage sector is

challenged to re-evaluate its business models

Business model

• The total investment of this initiative sums up to 173 million Euro• A strong business model is necessary to support this kind of

investment and prove that such an investment will result in long-term socio-economic returns

• The outcome of a Cost-Benefit analysis was positive: “The total balance of costs and returns of restoring, preserving and digitising audio-visual material (excluding costs of tax payments) will be between: 20+ and 60+ million.’’

• Economic benefits:• Direct effects of the investment are revenues from sales,

access for specific user groups, the repartition of copyright for the use of the material and so on.

• The indirect effects concern the product markets and labour market.

• Social benefits:• conservation of culture, reinforcement of cultural awareness,

reinforcement of democracy through the accessibility of information, increase in multimedia literacy and contribution to the Lisbon goals set by the EU

http://www.prestoprime.org/project/public.en.html

Content exploitation: from content is king ...

... to metadata rules

MANUAL ANNOTATION

costly & limited

Research on automatic annotation

• automatic information extraction based on:• visual features• information from audio

• crowdsourcing• deploying collateral data sources:

• subtitles, production scripts, meeting minutes, slides

PROGRESS? YES!

Various (laboratory) showcases

Commercial systems (e.g., blinkx, google)

work in progress

• institutional: reorganisation of traditional archival workflows

• national: development of common services • OAI, Persistent Identifiers, ASR service,

Vocabulary Repositories• commercial: uptake by MNCs (Google and

Microsoft) and SMEs • individual: bring about a shift regarding

defensive attitude of content owners towards opening up their funded and protected archives (trust/reliability)

Automatic annotation

• Participation in international research projects• VidioActive, MultiMATCH, VIDI-video, LiWA, P2P-

Fusion, Sterna, EUScreen, PrestoPrime• Collaboration agreement with Dutch research

institutes• Researchers stationed at Sound and Vision• Provide data (TRECVID, VideoCLEF)

• Research environment: exact copy of iMMix production environment for testing new technology• speech recognition• video analysis• fingerprinting• linking of context data (web, program guide,

production data)

Annotation strategies

• crowdsourcing: video labeling game• deploying collateral data sources:

incorporation of subtitles• automatic information extraction: speech

recognition for radio, pilots with visual• technology aided manual annotation:

documentalist support• linking to other information sources

DISPARITY BETWEEN TECHNOLOGY AND USER NEEDS

media professionals

journalists

researchers

educators

general public

Users perspective

• Rapidly evolving networked information society• Opening up• Focus on community specific

requirements• search needs• presentation/interaction needs

• Draw communities into libraries

(import)

metadata

(import)

content

metadata

(conversions)

content

(encoding)

Digital Archive

Digital Born15.000 hours of video40.000 hours of radio

Digitising Legacy MaterialImages for the Future

>250.000 hrs of audio and video

Asset management

ExhibitionsPublic Web Acces

User generated content and metadata

BroadcastProfessional

Education

"if it doesn't spread, it is dead" (Jenkins, 2009)

Open Images

• Open media platform for online access to audiovisual archive material, available for free (creative) reuse

• Built by Sound and Vision & Knowledgeland• Contributers include:

Open, open, open

• Open source media platform (MMBase)• Use of and open video codec (Ogg

Theora)• Use of the HTML5 <video> tag• Use of an open API (OAI-PMH, Atom

feeds)

Licence

• CC-BY-SA as preferred license• 3,000 items from our ‘own’

collection• ‘Internet quality’

COMMUNITY SPECIFIC REQUIREMENTS

From document level search to fragment level search

28

Broadcast professionals

In: Huurnink, Hollink, van Den Heuvel 2009 (submitted)

User survey (broadcast professionals)

Sound and Vision: Education

• Government and ‘Images for the Future’

• Earlier Initiatives

• ED*IT latest development completed with tools for

teacher and student

• ED*IT has been tested and developed in cooperation with many schools

ED*IT: Proposition

• One environment provides access to different controlled content databases (video, audio,

photograps, articles, etc)

• Editorial Staff contextualizes and enriches content for educational use

• Enriched with tools for student and teacher to edit content in an easy way

• For primary-, secondary- and vocational education

ED*IT: Functionalities

Video & Content Editor

Dossier Maker

Cut Videoclips

E- Lesson Maker

Teacher Forum

Upload Files

Presentation Maker

Edit Photographs

Digital Paper Maker

Timeline Maker

ED*IT: Facts & Figures

• Test Accounts: 2500

• Licence: 50 schools

• Licence: 50 educational departments

• Objective: Same market share as Teleblik is 78%

Researchers

• Verteld Verleden aims at establishing a shared information space on distributed Dutch Oral History collections:• distributed collections (harvested via OAI)• search & interlink collections via centralized search

• project goals:

1. provide demonstrator portal to show how technology could help researchers

2. acquire information on specific user requirements • search• collaboration• linking• privacy• dedicated work space

http://www.verteldverleden.org

example VPRO radio interviews

QUOTE

“VISUAL RADIO”

INTERACTION REQUIREMENTS

people expect easy interaction as

in 'every-day tools' they use on the web ...

• The Sound and Vision Experience: a crossover between a museum and amusement park with various archive material, to make audiovisual heritage more acessible to the general public

DRAW COMMUNITIES INTO LIBRARIES

goals

• exploiting community tagging (tagging games, etc)

• exploring the wisdom of crowds by hooking up with user communities (e.g., everyone-as-commentator, unexpected experts)

• capturing relevant information from the internet and aligning this with archived items.

• finding new ways for communities to interact with the data.

Technology perspective

Technology:• provide anchor points for linking up with the

`cloud’ (entity detection, segmentation, cross-collection SID, etc): people, places, events, topics, quotes, etc.• keywords: reliability, speed

• synchronization of UGC with AV documents• users in the loop: UGC for adapting/training

analysis tools• early fushion of multiple modalities (vision,

speech)• technology aided annotation: Documentalist

Support System

Crowdsourcing

14 minutes left for annotation

Play!you score when somebody else uses the same term

fill in words that describe what you see or hear

TAGGING GAME EXAMPLEwww.waisda.nl

Play against ASR

www.beeldengeluidwiki.nlwww.beeldengeluidwiki.nl

Hollands Glorie op Pinkpop

Wrap up

• value of archive is strongly related to access opportunities

• access is to a large extend technology driven• but next to technology development we need

to make a shift:• from a ‘laboratory view’ on users to drawing

users and communities into the loop• NISV is aiming towards this two-way strategy:

• incorporate advanced access technology• discuss access requirements with the

stakeholders