November 15, 2003 CLIS Alumni Chapter

Talking to the Future:The MALACH Project

Douglas W. OardJoanne Archer, Ammie Feijoo, Xiaoli Huang

College of Information Studies

Telling Our Stories

Shoah Foundation’s Collection• Enormous scale

– 116,000 hours; 52,000 interviews; 180 TB

• Grand challenges– 32 languages, accents, elderly, emotional, …

• Accessible– $100 million collection and digitization investment

• Annotated– 10,000 hours (~200,000 segments) fully described

• Users– A department working full time on dissemination

Who Uses the Collection?

• History• Linguistics• Journalism• Material culture• Education• Psychology• Political science• Law enforcement

• Book• Documentary film• Research paper• CDROM• Study guide• Obituary• Evidence• Personal use

Discipline Products

Based on analysis of 280 access requests

Question Types

• Content– Person, organization– Place, type of place (e.g., camp, ghetto)– Time, time period– Event, subject

• Mode of expression– Language– Displayed artifacts (photographs, objects, …) – Affective reaction (e.g., vivid, moving, …)

• Age appropriateness

Full-Description Cataloguing

Subject PersonLocation-Time

Berlin-1939 Employment Josef Stein

Berlin-1939 Family life Gretchen Stein Anna Stein

Dresden-1939 Schooling Gunter Wendt Maria

Dresden-1939 Relocation Transportation-rail inte

rvie

w ti

me

“Real-Time” Cataloguing

Subject PersonLocation-Time

Berlin-1939

Dresden-1939

Employment Josef SteinGretchen SteinAnna Stein

RelocationTransportation-rail

SchoolingGunter Wendt

Family Life

Maria

inte

rvie

w ti

me

Thesaurus-Based Search

The Goal

Dramatically improve access to large multilingual spoken word Collections …

… by capitalizing on the unique characteristics of the Survivors of the Shoah Visual History Foundation's collection of videotaped oral history interviews.

Joanne Archer

Observational Studies

• Four searchers– History/Political Science– Holocaust studies– Holocaust studies– Documentary filmmaker

• Sequential observation• Rich data collection

– Intermediary interaction– Semi-structured interviews– Observational notes– Think-aloud– Screen capture

• Four searchers– Ethnography

– German Studies

– Sociology

– High school teacher

• Simultaneous observation

• Opportunistic data collection– Intermediary interaction

– Semi-structured interviews

– Observational notes

– Focus group discussions

Workshop 1 (June) Workshop 2 (August)

Observed Selection Criteria

• Topicality (57%)Judged based on: Person, place, …

• Accessibility (23%)Judged based on: Time to load video

• Comprehensibility (14%)Judged based on: Language, speaking style

FunctionalityNeeded Function Boolean Search and Ranked Retrieval (13)

Testimony summary (12)

Pre-Interview Questionnaire search/viewer (9)

Rapid access (7)

Related/Alternative search terms (3)

Adding multiple search terms at once (2)

Keywords linked to segment number for easy access(1)

Multi-tasking (1)

Searching testimonies by places under ‘Experience Search’ (1)

Extensive editing within ‘My Project’ (1)

Desired Function Temporary saving of selected testimonies (4)

Remote access (3)

Integrated user tools for note taking (3)

Map presentation (2)

Reference tool (1)

More repositories (1)

Introductory video of system tutorial (1)

Help (1)

Xiaoli Huang

Supporting Information Access

SourceSelection

Search

Query

Selection

Ranked List

Examination

Recording

Delivery

Recording

QueryFormulation

Search System

Query Reformulation and

Relevance Feedback

SourceReselection

AutomaticSearch

BoundaryDetection

InteractiveSelection

ContentTagging

SpeechRecognition

QueryFormulation

ASR SpontaneousAccentedLanguage switching

NLPComponents Multi-scale segmentation

Multilingual classificationEntity normalization Prototype

Evidence integrationMultilingual searchSpatial/temporal

UserNeeds

Observational studiesFormative evaluationSummative evaluation

Description Strategies• Transcription

– Manual transcription (with optional post-editing)

• Annotation– Manually assign descriptors to points in a recording– Recommender systems (ratings, link analysis, …)

• Associated materials– Interviewer’s notes, speech scripts, producer’s logs

• Automatic– Create access points with automatic speech processing

English ASR Error Rate

0

20

40

60

80

100

Wo

rd E

rro

r R

ate

Training: 65 hours (acoustic model)/200 hours (language model)

true

system output

missfalsealarm

Effect of ASR Errors

Building a Test Collection

• Overall relevanceAssessment is informed by the assessments for the individual reasons for relevance (categories of relevance), but the relationship is not straightforward

• Provides direct evidence

• Provides indirect / circumstantial evidence

• Provides context(e.g., causes for the phenomenon of interest)

• Provides comparison (similarity or contrast, same phenomenon in different environment, similar phenomenon)

• Provides pointer to source of information

Ammie Feijoo

Some Statistics

• 2,000 U.S. radio stations Webcasting

• 250,000 hours of oral history in British Library

• 35,000,000 audio streams on the Web

Spoken Word Collections

• Broadcast programming– News, interview, talk radio, sports, entertainment

• Scripted stories– Books on tape, poetry reading, theater

• Spontaneous storytelling– Oral history, folklore

• Incidental recording– Speeches, oral arguments, meetings, phone calls

Building a Web of Spoken Words• Affordable storage

– For $1, you can store 1.5 million spoken words

• Adequate network capacity– Internet capacity: 30 million simultaneous programs

• Works with any modem– You can even read email while playing audio

• Replay capabilities– 38% of US users recently used streaming audio

• Effective search capabilities– Not quite yet …

Looking Forward: 2006

• Working systems in five languages– Real users searching real data

• Rich experience beyond broadcast news– Frameworks, components, systems

• Affordable application-tuned systems– Oral history, lectures, speeches, meetings, …

For More Information

• The MALACH project– http://www.clsp.jhu.edu/research/malach/

• NSF/EU Spoken Word Access Group– http://www.dcs.shef.ac.uk/spandh/projects/swag/

• Speech-based retrieval– http://www.glue.umd.edu/~dlrg/speech/