Search and Hyperlinking Task at MediaEval 2012
-
Upload
mediaeval2012 -
Category
Technology
-
view
892 -
download
0
description
Transcript of Search and Hyperlinking Task at MediaEval 2012
![Page 1: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/1.jpg)
Search and Hyperlinking Task at MediaEval 2012
Search and Hyperlinking Taskat MediaEval 2012
Maria Eskevich1, Gareth J.F. Jones1, Shu Chen1
Robin Aly2, Roeland Ordelman2Martha Larson3
1 Dublin City University, Dublin, Ireland2 University of Twente, The Netherlands
3 Delft University of Technology, Delft, The Netherlands
![Page 2: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/2.jpg)
Search and Hyperlinking Task at MediaEval 2012
Background
![Page 3: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/3.jpg)
Search and Hyperlinking Task at MediaEval 2012
Brave New Task
![Page 4: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/4.jpg)
Search and Hyperlinking Task at MediaEval 2012
Brave New Task and Crowdsourcing
![Page 5: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/5.jpg)
Search and Hyperlinking Task at MediaEval 2012
Brave New Task: Search and Hyperlinking
Previous work:I MediaEval 2011: Rich Speech Retrieval TaskI VideoCLEF 2009: Linking Task
What is Brave New about the Search and Hyperlinking?I Unified scenario for 2 tasks:
Search output results as input for LinkingI Use of crowdsourcing for results assessment:
workers define the relevance of the video segments,the HIT can be run for each new submission andenrich overall results
I Search: ME10WWW dataset − > blip10000
![Page 6: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/6.jpg)
Search and Hyperlinking Task at MediaEval 2012
Brave New Task: Search and Hyperlinking
Previous work:I MediaEval 2011: Rich Speech Retrieval TaskI VideoCLEF 2009: Linking Task
What is Brave New about the Search and Hyperlinking?I Unified scenario for 2 tasks:
Search output results as input for LinkingI Use of crowdsourcing for results assessment:
workers define the relevance of the video segments,the HIT can be run for each new submission andenrich overall results
I Search: ME10WWW dataset − > blip10000
![Page 7: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/7.jpg)
Search and Hyperlinking Task at MediaEval 2012
Sub-tasks
I Search sub-task:I Known-item search:
I Textual queries onlyI Multimodal queries (text + video clues)
I Required runs:1 submission for 1-best output of each ASR transcript
I Linking sub-taskI Ad-hoc search for videos that can potentially be linked to
the anchor videos (ground truth of the Search subtask)I Required runs:
use of ASR 1-best transcripts as representationI *Additional runs:
use team output of Search subtask as video anchors forLinking sub-task.
![Page 8: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/8.jpg)
Search and Hyperlinking Task at MediaEval 2012
Data: blip10000
I ASR Transcript:I LIMSI/Vocapia:
I Confusion networksI LIUM:
I 1-bestI LatticesI Confision networks
I Video clues:I Shot boundaries (TU Berlin)
I Concept-based descriptors based on a list of 589 concepts(University of Oxford)
I Face detection results (INRIA)
![Page 9: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/9.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
![Page 10: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/10.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
![Page 11: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/11.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
![Page 12: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/12.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
![Page 13: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/13.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
I Mean Reciprocal Rank (MRR):
RR =1
RANKI Mean Generalized Average Precision (mGAP):
GAP =1
RANK. PENALTY
![Page 14: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/14.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
I Mean Reciprocal Rank (MRR):
RR =1
RANKI Mean Generalized Average Precision (mGAP):
GAP =1
RANK. PENALTY
![Page 15: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/15.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
I Mean Average Segment Precision (MASP):Ranking + Length of (ir)relevant content
Segment Precision (SP[r ]) at rank r :
Average Segment Precision:
ASP =1n.
N∑r=1
SP[r ] · rel(sr )
rel(sr ) = 1, if relevant content is present,otherwise rel(sr ) = 0
![Page 16: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/16.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
I Mean Average Segment Precision (MASP):Ranking + Length of (ir)relevant content
Segment Precision (SP[r ]) at rank r :
Average Segment Precision:
ASP =1n.
N∑r=1
SP[r ] · rel(sr )
rel(sr ) = 1, if relevant content is present,otherwise rel(sr ) = 0
![Page 17: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/17.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
I Mean Average Segment Precision (MASP):Ranking + Length of (ir)relevant content
Segment Precision (SP[r ]) at rank r :
Average Segment Precision:
ASP =1n.
N∑r=1
SP[r ] · rel(sr )
rel(sr ) = 1, if relevant content is present,otherwise rel(sr ) = 0
![Page 18: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/18.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Search sub-task
I Mean Average Segment Precision (MASP):Ranking + Length of (ir)relevant content
Segment Precision (SP[r ]) at rank r :
Average Segment Precision:
ASP =1n.
N∑r=1
SP[r ] · rel(sr )
rel(sr ) = 1, if relevant content is present,otherwise rel(sr ) = 0
![Page 19: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/19.jpg)
Search and Hyperlinking Task at MediaEval 2012
Relevance Evaluation via Crowdsourcing:Amazon MTurk HIT
![Page 20: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/20.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Linking sub-task
* Brave New Task issue: Late evaluation results :-(
![Page 21: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/21.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Linking sub-task
* Brave New Task issue: Late evaluation results :-(
![Page 22: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/22.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Linking sub-task
* Brave New Task issue: Late evaluation results :-(
![Page 23: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/23.jpg)
Search and Hyperlinking Task at MediaEval 2012
Evaluation: Linking sub-task
* Brave New Task issue: Late evaluation results :-(
![Page 24: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/24.jpg)
Search and Hyperlinking Task at MediaEval 2012
Relevance Evaluation via CrowdsourcingDetails
I Assessment level:I VideosI Provided segments:
Length varies across runs and participants
I Workers explanation of relevance judgement:I Relevant:
I ”Same” videosI Videos are on the same topicI Same program different topics
I Irrelevant:I Same program different topicsI Different topicsI Different video styles: interview vs presentation
![Page 25: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/25.jpg)
Search and Hyperlinking Task at MediaEval 2012
Relevance Evaluation via CrowdsourcingDetails
I Assessment level:I VideosI Provided segments:
Length varies across runs and participants
I Workers explanation of relevance judgement:I Relevant:
I ”Same” videosI Videos are on the same topicI Same program different topics
I Irrelevant:I Same program different topicsI Different topicsI Different video styles: interview vs presentation
![Page 26: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/26.jpg)
Search and Hyperlinking Task at MediaEval 2012
Participants
Group Search LinkingCharles University in Prague (CUNI) 5 –
Dublin City University (DCU) 4 6Ghent University (MMLab) 4 4 + 4
INRIA/IRISA – 5University of Twente (UTwente) 3 1
![Page 27: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/27.jpg)
Search and Hyperlinking Task at MediaEval 2012
Participants
Group Search LinkingCharles University in Prague (CUNI) 5 –
Dublin City University (DCU) 4 6Ghent University (MMLab) 4 4 + 4
INRIA/IRISA – 5University of Twente (UTwente) 3 1
![Page 28: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/28.jpg)
Search and Hyperlinking Task at MediaEval 2012
Search sub-task results
![Page 29: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/29.jpg)
Search and Hyperlinking Task at MediaEval 2012
Linking sub-task results
![Page 30: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/30.jpg)
Search and Hyperlinking Task at MediaEval 2012
Thank you for your attention!
Buon compleanno Robin!
Participants presentations:I Search sub-task:
I Maria Eskevich, Dublin City UniversityI Petra Galuscakova, Charles University in Prague
I Search and Linking sub-task:I Danish Nadeem, University of Twente
I Linking sub-task:I Camille Guinaudeau, INRIA/IRISAI Shu Chen, Dublin City University
I Search and Linking sub-task:I Tom de Nies, University of Chent
![Page 31: Search and Hyperlinking Task at MediaEval 2012](https://reader034.fdocuments.net/reader034/viewer/2022051412/5498433db47959654d8b53a6/html5/thumbnails/31.jpg)
Search and Hyperlinking Task at MediaEval 2012
Thank you for your attention!Buon compleanno Robin!
Participants presentations:I Search sub-task:
I Maria Eskevich, Dublin City UniversityI Petra Galuscakova, Charles University in Prague
I Search and Linking sub-task:I Danish Nadeem, University of Twente
I Linking sub-task:I Camille Guinaudeau, INRIA/IRISAI Shu Chen, Dublin City University
I Search and Linking sub-task:I Tom de Nies, University of Chent