Atomatic summarization Atomatic summarization of voicemail messages of voicemail messages
using lexical and prosodic using lexical and prosodic featuresfeatures
Koumpis and RenalsKoumpis and Renals
Presented by Daniel VassilevPresented by Daniel Vassilev
The DomainThe Domain
Voicemail is a special case of Voicemail is a special case of spontaneous speechspontaneous speech
Goal: Enable the voicemail user to Goal: Enable the voicemail user to receive his/her messages anywhere and receive his/her messages anywhere and any time, in particular on mobile devicesany time, in particular on mobile devices
Key components: caller identification, Key components: caller identification, reason for call, information that the caller reason for call, information that the caller requires and a return phone numberrequires and a return phone number
The TaskThe Task
Summarization: obtain the most Summarization: obtain the most important information from the voicemailimportant information from the voicemail
A complete system requires spoken A complete system requires spoken language understanding and language language understanding and language generation generation current technology is not current technology is not adequateadequate
Solution: simplify task by deciding for Solution: simplify task by deciding for each word if it will be in the summaryeach word if it will be in the summary
The TaskThe Task
Voicemail is short, 40s on averageVoicemail is short, 40s on average Summaries must fit into 140 characters Summaries must fit into 140 characters
(mobile devices)(mobile devices) Content more important than coherence Content more important than coherence
and document flowand document flow ASR used on the voicemail, a significant ASR used on the voicemail, a significant
word error rate must be assumed word error rate must be assumed (30%-40% error!)(30%-40% error!)
Voicemail CorpusVoicemail Corpus
IBM Voicemail Corpus-Part IIBM Voicemail Corpus-Part I 1801 messages (1601 for training, 200 1801 messages (1601 for training, 200
for testing)for testing) 14.6 h14.6 h On average 90 words / messageOn average 90 words / message Message topics: Message topics: 27% business related,
25% personal, 17% work-related, 13% technical and 18% in other categories.
The classification The classification problemproblem
Classifier decides if a word is included in Classifier decides if a word is included in the summarythe summary
Using Parcel (feature selection alg.) and Using Parcel (feature selection alg.) and a Receiver Operating Characteristic a Receiver Operating Characteristic (ROC) graphs for feature selection(ROC) graphs for feature selection
Hybrid multi-layer perceptron (MLP) / hidden Markov model (HMM) classifier
Receiver Operating Receiver Operating CharacteristicCharacteristic
Graph plots the true positive rate (sensitivity) Graph plots the true positive rate (sensitivity) vs. 1 - the true negative rate (specificity)vs. 1 - the true negative rate (specificity)
We can shift the positive vs. negative error by We can shift the positive vs. negative error by taking different acceptance thresholds (we taking different acceptance thresholds (we move on the ROC curve)move on the ROC curve)
Different classifiers will have different ROC Different classifiers will have different ROC curvescurves
Sample ROC graphSample ROC graph
System setupSystem setup
The team built a sophisticated, multi-The team built a sophisticated, multi-component system that can capture the component system that can capture the different types of information occurring in different types of information occurring in voicemailvoicemail
Initial trigram language model, Initial trigram language model, augmented with sentences from taugmented with sentences from the Hub-4 Broadcast News and Switchboard language model training corpora
System SetupSystem Setup
Pronunciation dictionary with 10,000 Pronunciation dictionary with 10,000 words from the training datawords from the training data
+ pronunciations obtained from the SPRACH broadcast news system
Annotated summary words in 1,000 messages
System overviewSystem overview
Entities in summariesEntities in summaries
Annotation proceduresAnnotation procedures
1. Pre-annotated NEs were marked as targets, unless unmarked by later rules;
2. The first occurrences of the names of the speaker and recipient were always marked
as targets; later repetitions were unmarked unless they resolved ambiguities;
3. Any words that explicitly determined the reason for calling including important
dates/times and action items were marked; 4. Words in a stopword list with 54 entries were
unmarked;
Annotation proceduresAnnotation procedures
Labeled only on transcribed messages (no Labeled only on transcribed messages (no audio)audio)
Annotators tended to eliminate irrelevant words Annotators tended to eliminate irrelevant words (as opposed to mark content words)(as opposed to mark content words)
Produced summaries about 30% shorter than Produced summaries about 30% shorter than original messageoriginal message
Relatively good level of inter-annotator Relatively good level of inter-annotator agreementagreement
Lexical featuresLexical features
Lexical features from ASR outputLexical features from ASR output
collection frequency (less frequent words more collection frequency (less frequent words more informative)informative)
acoustic confidence (ASR confidence)acoustic confidence (ASR confidence)
All other features considered before and after All other features considered before and after stemming:stemming:
NEs, proper names, tel. numbers, dates and NEs, proper names, tel. numbers, dates and times, word positiontimes, word position
Prosodic featuresProsodic features
Prosodic features from audio using signal Prosodic features from audio using signal processing algorithmsprocessing algorithms
duration normalization over the corpusduration normalization over the corpuspauses (preceding and succeeding)pauses (preceding and succeeding)mean energymean energyF0 range, average, onset and offsetF0 range, average, onset and offset
ResultsResults
Named entities identified very accurately Named entities identified very accurately (without stemming)(without stemming)
Telephone numbers recognized well also Telephone numbers recognized well also by specific named entity lists. Pos also by specific named entity lists. Pos also good as numbers appear towards the good as numbers appear towards the end of the messageend of the messageAll prosodic features but duration had no All prosodic features but duration had no predictive powerpredictive power
ROC curves for different ROC curves for different tasks / featurestasks / features
ResultsResults
Dates / times: best matched by specific Dates / times: best matched by specific named entity list and collection frequencynamed entity list and collection frequency
Prosodic features (duration, energy, F0 Prosodic features (duration, energy, F0 range) more important but still not the range) more important but still not the best predictorsbest predictors
The Parcel bootstrappingThe Parcel bootstrappingalgorithmalgorithm
ConclusionsConclusions
Trade-off between length of summary Trade-off between length of summary and retaining essential content wordsand retaining essential content words
70%-80% agreement with human 70%-80% agreement with human summary for hand-annotated messagessummary for hand-annotated messages
50% agreement when using ASR50% agreement when using ASR
ConclusionsConclusions
Automatic summaries perceived as Automatic summaries perceived as worse than human summaries (duh!)worse than human summaries (duh!)
However, if the summarizer used human However, if the summarizer used human annotated data (as opposed to ASR annotated data (as opposed to ASR output), the perceived quality improved output), the perceived quality improved significantlysignificantly
Top Related