Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research...
-
Upload
ashley-mccurdy -
Category
Documents
-
view
216 -
download
1
Transcript of Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research...
Human-Centric Multimedia Research:
Research Opportunities
Nuria Oliver, PhDTelefonica Research
Multimedia and Data Mining & User Modeling Scientific Director
Explosion of Digital Data
6-Fold Growth in Four Years
Information Created, Captured and Replicated
2006161 Exabytes
2010988 Exabytes
Source: IDC, 2007
Exabytes?
Book
Photo
Movie
Print CollectionUS Library of Congress
All printed material (multimedia)
All words ever spoken by human beings (5 exa)
Who will create all this data?
User* Generated
Content692 Exabytes
User* Generated
Content692 Exabytes
Organizational** Touch
Content296 Exabytes
*Consumers and Workerscreating, capturing or replicatingpersonal informaton
**Transported, hostedmanaged or secured
2010 Data988 Exabytes
563 Exabytes
Human-Centric Multimedia
Prosumer
On-the-go
Access
Cont
ext &
Co
nten
t Social
Produce
Search & Discovery Consume
Media, tags, ratings, comments
MediaMedia
A FEW RESEARCH CHALLENGES ANDOPPORTUNITIES
Multi-Modality: Content + contextMulti-modal approaches are needed to
construct novel methodologies to fuse multi-modal content and context information
Multi-modal Multimedia Content Analysis:Feature ExtractionSimilarity Metrics
Ontology definitionIndexing schemes….
Multimedia Context:Higher level knowledge
generated by users (tags, comments…)
User interaction data Wisdom of the crowd
Fusion of content and context-based featuresCreation of large collections of labeled training data
Noise filtering by aggregation of contextual information
Improved search results
Multimedia Tagging
Paris
Eiffel Tower
France Vacation
June 2009
Multimedia Tagging• User generated content is rarely annotated really
difficult, if not impossible to later find it • When annotated, it is typically done in batch, per
session, not per item• Tags significantly improve search results alone or
combined with content-based techniques• Need for novel interfaces to encourage users to
annotate content– Games with a Purpose– Annotations at the time of capture
• More research on tag expansion and automatic tagging
Multimedia Information Overload
Multimedia Information Overload
• Retrieval accuracy is not sufficient due to vast amounts of available information too many relevant results
• New orthogonal dimensions need to be used to extend the notion of relevance and improve retrieval performance e.g., aesthetics
• User generated content is of varying quality• Need for user centric approaches• Multi-disciplinary approaches: – Computer scientists, psychologists, human-computer
interaction researchers– Computer Vision, Pattern Recognition, Machine Learning,
Human Perception, Human Activity Recognition
Video and Audio Feature extractionSignature Creation
Duplicate Detection
Example: Near-Duplicate Videos
How do users perceive near-duplicate videos?
Do they care about them? Which features are important
when defining near-duplicates?
M. Cherubini, R. de Oliveira, and N. Oliver, “Understanding near-duplicate videos,” in Proceedings of ACM MM’09, (Beijing, China), pp. 35–44, ACM Press, October 19-24 2009.
Example: Multimedia Aesthetics
“The interest that a photograph, video or audio piece generates when perceived by human observers, and that incorporates both objective and subjective factors”
The Importance of Aesthetics
“Paris Louvre Night”
The Importance of Aesthetics
• User generated content has a wide range of quality and aesthetic value for the same content
• Aesthetics influence our perception of content• Highly disregarded in state-of-the-art multimedia retrieval
systems• Need for computational models of the aesthetic value of
multimedia content• Need for ground truth databases on image, audio and video
aesthetics• Need for deeper understanding of– The role of aesthetics on user preferences and satisfaction– Universal vs personal aesthetics– Domain-dependent aesthetics– Quality vs aesthetics
Personalization, Recommendation and Exploratory Search
Personalization, Recommendation and Exploratory Search
• Future multimedia search and retrieval systems will need to take into account the user’s preferences, interests and task at hand in order to return relevant content
• Huge amounts of multimedia data – Need for recommendations rather than direct search– Automatic discovery of relevant information to the users
• More research should be devoted to user modeling, personalization and recommendations of multimedia content
• Untapped research challenge: Role that the task at hand plays in determining the optimal multimedia content to retrieve for the user
Multimedia Storytelling
“The conveying of events with words, images and sounds, often with embellishement. “Stories or narratives have been shared in every culture and in every land as a means of entertainment, education, preservation of culture and in order to instill moral values.
Crucial elements of stories and storytelling include plot and characters, as well as the narrative point of view.
Multimedia Storytelling• Despite capturing large amounts of digital multimedia
content, most users rarely access the content again• Sharing the multimedia content is one of the main
reasons why users capture it• Lack of efficient and scalable tools for browsing, finding
and selecting the desired content• Multimedia Storytelling: User-friendly, semi-automatic
and scalable (space and time) multimedia tools that enable users to – Easily retrieve desired multimedia content – Create and share the story they want to create from their
content
Exemplary Workflow for MM Storytelling
StorytellingAlgorithms
Multimedia Analysis Tools
Create Story Slideshow:“Madrid Christmas 2009”
StorytellingUIFlickr e-mail communication
With story slideshow
Face Detection Face RecognitionSmile DetectionAesthetics RerankingClusteringNear-duplicate DetectionTag expansionAutomatic Classification
Identify main ActorsIdentify main ChaptersComplement contentwith external contentSelect images based on *Story Length *Target Audience *Target Device
New Multimedia Experiences
New Multimedia Experiences
• Users are increasingly seeking new ways to experience multimedia content
• Research opportunities combining:– Music + Video: High-quality visual musical experiences– Video + 3D reconstruction: 3D video– Images + Context : Mobile Augmented Reality
• Research challenges in:– Multimedia analysis: Machine learning, pattern
recognition, computer vision– User Modeling– Human-computer interaction
http://research.tid.es/multimedia