Experiments with Segmentation Strategies for Passage Retrieval in Audio-Visual Documents
-
Upload
petra-galuscakova -
Category
Data & Analytics
-
view
632 -
download
2
description
Transcript of Experiments with Segmentation Strategies for Passage Retrieval in Audio-Visual Documents
Experiments with Segmentation Strategies for Passage Retrieval in Audio-Visual Documents
Petra Galuščáková and Pavel [email protected]
Institute of Formal and Applied LinguisticsFaculty of Mathematics and Physics
Charles University in Prague
4. 4. 2014
2
Information Retrieval
● Information Retrieval (IR) is a task which involves searching for documents relevant to a given query.
3
Speech Retrieval
● Speech Retrieval focuses on retrieval from audio-visual documents (recordings).
4
Speech Retrieval
● Speech Retrieval is often converted on traditional Information Retrieval
● Automatic Speech Recognition (ASR) system applied to the audio track
5
Speech RetrievalProblems
● Documents are long (e.g. whole TV programmes)● Often unstructured
● Navigation in audio-visual recordings is time consuming● We need to retrieve relevant segments of full documents
● Possibility to browse the recordings using hyperlinks (links between passages)
→ Passage Retrieval
6
Passage Retrieval● Splits texts into smaller units which then function as
documents in the retrieval process
● Makes the retrieval process more precise
● May improve retrieval of full documents● The segmentation is crucial for the quality of the retrieval
→ We focus on segmentation strategies
7
Segmentation Strategies
● Regular (Window-based)● Segments of equal length with regular shift● Claimed to be a very effective approach
● Similarity-based● Measures similarity between neighbouring segments
● Lexical-chain-based● Finds sequences of lexicographically related word occurrences
● Feature-based● Employs machine learning methods to detect segment boundaries
based on various features
8
Feature-based Segmentation in Passage Retrieval
9
ExperimentsTasks Description
10
● MediaEval is a benchmarking initiative dedicated to development, comparison, and improvement of strategies for processing and retrieving multimedia content.
● E.g., speech recognition, multimedia content analysis, music and audio analysis, social networks, geo-coordinates, …
● 2013 Similar Segments in Social Speech Task● 2013 Search and Hyperlinking Task
11
Similar Segments in Social Speech (SSSS) Task
● Scenario:● A new member (e.g., a new student) joins a community or
organization (e.g., a university), which owns an archive of recorded conversations among its members
● A member wants to find information according to his or her interest in the archive
– The student wants to find more segments similar to the ones he or she is interested in and browses the archive using hyperlinks in videos
● The main goal:● To find segments similar to the given ones
12
Similar Segments in Social Speech Task Data
● On purpose recorded interviews (5 hours) of two speakers (university students’ community)
● Divided into training/test data
● Manual and ASR transcripts
● Manually indicated segments (1886 segments), manually grouped into similarity sets
● Query segment - specified by the timestamp of its beginning and end
● Queries - constructed by including all words lying within the boundaries of the query segments
13
Search and Hyperlinking (SH) Task
● Scenario:● A user wants to find a piece of information relevant to a given
query in a collection of TV programmes (Search subtask)● And then navigate through a large archive using hyperlinks to
the retrieved segments (Hyperlinking subtask)● The main goal of the Search Subtask
● Find passages relevant to a user’s interest given by a textual query in a large set of audio-visual recordings
14
Search and Hyperlinking Task Data
● TV programme recordings provided by BBC (1697 hours)
● Subtitles and two ASR transcripts (LIMSI and LIUM)
● 4 training and 50 test queries● Query text: e. g. Boris Johnson● Visual cue: e. g. 2 men sitting opposite each other
● Metadata, synopsis, cast, detected shots, detected faces, visual concepts
15
Passage Retrieval Quality Evaluation
● Full document retrieval →Mean Reciprocal Rank (MRR)– RR = 1 / rank of the first correctly retrieved document
● Retrieval of the exact passages → MRRw and MGAP● MRR-window (MRRw)
– Retrieved starting points are limited to appear less than 60 seconds from the relevant starting points
● Mean Generalized Average Precision (MGAP)– The quality of the retrieved starting point is assessed according to
its distance from the relevant starting point using a penalty function
16
ExperimentsSystem Description
17
Baseline System● We employ the Terrier IR toolkit
● Hiemstra language model● Parameter set to 0.35 (importance of a query term in a
document)● Stopwords removal, stemming
● Post-filtering of the answers● The segments partially overlapping with either the query
segment or a higher ranked segment are removed from the list of results
18
Window-based Segmentation
● Equally-long segments with a regular shift
19
Feature-based Segmentation
● We identify possible segment boundaries (beginnings and ends)
● Model: J48 decision trees
● Training data available for the SSSS task● Manually marked segments
● Binary classification problem● For each word in the transcripts, we predict whether a
segment boundary occurs after this word or not● Classes: segment boundary and segment continuation
20
Features● Cue words and tags (n-grams which frequently occur at the
boundary,most informative n-grams) for segment beginning and end● Segment beginnings: “I’m”, “the”, “are you”, “you have”, ...● Segment ends: “good”, “interesting”, “lot”, ...
● Letter cases● Length of the silence before the word● Division given in transcripts (e.g., speech segments defined in
the LIMSI transcripts)● The output of the TextTiling algorithm
21
Feature-based Segmentation Approaches
22
ExperimentsResults
23
Similar Segments in Social Speech Task - Evaluation
● Best results are obtained by the feature-based segmentation into overlapping segments
● Manual gold-standard segmentation is outperformed by feature-based segmentation (MRRw score on the manual transcripts)
● Manual transcripts are significantly better in all scores
24
Segmentation Model in the SH Task
● Training set used in the SH Search Subtask is very small● We apply the SSSS-trained models in the SH task● Allows us to examine the possibility of creating a universal
model for feature-based segmentation● Potential problems:
● Different vocabulary (student's dialogues vs. TV programmes)● Different ASR systems may prefer different vocabulary● Different distribution of silence, document structure
25
SH Task Evaluation
● Not as consistent as for the SSSS task
● Depending on the type of the transcript
● Feature-based approaches creating overlapping segments - effective when applied on the subtitles
26
Conclusion
27
Conclusion
● Information Retrieval, focus on speech data (Speech Retrieval)
● Focus on retrieval of exact relevant passages
● Importance of segmentation● Experiments in MediaEval benchamark
● Similar Segments in Social Speech Task (university student dialogues) and Search and Hyperlinking Task (BBC programmes)
● We applied window-based segmentation and three types of feature-based segmentations
28
Conclusion cont.
● Feature-based segmentation applied in the two tasksoutperformed regular segmentation● Claimed to be a very effective approach● The improvement in the SSSS Task was statistically
significant on the manual (MRRw and mGAP measures) and ASR (mGAP measure) transcripts
● The results in the SH task were not so conclusive● Some of the results (on the subtitles) are encouraging
29
Thank you
This research has been supported by the project AMALACH (grant n. DF12P01OVV022 of the program NAKI of the Ministry of Culture of the Czech
Republic), the Czech Science Foundation (grant n. P103/12/G084), and the Charles University Grant Agency (grant n. 920913).