Multimodal InteractionMultimodal Interaction and TV

Wei‐Yun YauProgramme Manager, 

Personalised Interactive Tele‐ServicesInstitute for Infocomm Research, A*STARInstitute for Infocomm Research, A STAR

Sheau NgVi P id t & H d f R&DVice President & Head of R&D, 

NBC Universal

W3C Workshop on Rich Multimedia App Development

Singapore

Institute for Infocomm Research (I2R)2

Who We AreA f S i T h l &Agency for Science, Technology & Research (A*STAR) is under the Ministry of Trade & Industry, Singapore.

VisionMission

Fostering world class

g p

Powering a vibrant and strong infocomm

ecosystem for

Fostering world-class infocomm and media

research and developing a deep talent pool of

i f f i ly

Singapore. infocomm professionals to power a vibrant knowledge-

based Singapore.

Singapore’s Largest ICT Research Instituteg p g

600 Staff today 681Industry

Collaborations 1220 Technology Disclosures 5430Research

Publications

Institute for Infocomm Research (I2R)3 Institute for Infocomm Research (I2R)3

INTERNET

T t i lTerrestrialSatellite

CableDVD Player / Blue-Ray Player

Institute for Infocomm Research (I2R)4

Enjoying Related Content Seamlessly

YoutubeNews

Local NewsstationNews

AlJazeera, AFP, Euronews

station (CNA)

Social Media

Institute for Infocomm Research (I2R)5

Challenges• Automated Content Discovery

– Where are the relevant multimedia content?– How relevance is determined? 

• User Preference ModelingWhat the user likes?– What the user likes?

– How to track content consumed from different sources on different devices? 

• Personalised Recommendation– What the user would be interested now?

– What are the external events happening that the user could be interested in?

• Attractive PresentationAttractive Presentation– What to show, how to show, and when?– Easy interaction & navigation to create better engagement?

Institute for Infocomm Research (I2R)6 Institute for Infocomm Research (I2R)6

News Video AnalyticsExternal sources

Story segmentation

Named entities

External sources

Keyword Search /Timeline

Speech‐to‐text

segmentation

Metadata enrichment

Timeline

News Video Tapestry

Video‐OCR

…

Keywords & Summarization

Breaking News Alert

Scene detection

… …Link analytics

…

Linked data

Low-levelAnalytics

detection

High-levelAnalytics

Link analytics

Local news channels

Applications

Linked data

ychannels

Story-K Story-K+1 Story-K+2

Institute for Infocomm Research (I2R)7 Institute for Infocomm Research (I2R)7

DemographicsSocial InteractionsMultiple Modality

Viewing DeviceDemographics

LocationLocation

Mapping Contents to Viewers

Structured Semi‐structured / Unstructured

Institute for Infocomm Research (I2R)8 Institute for Infocomm Research (I2R)8

TV ContentEPG ‐Metadata Web  Content Web Resources

Personalised Recommendation 

Questionnaire

External Events

Remote Usage LogsRemote Usage Logs

Social Groups Feedback to Personalized EPG

Institute for Infocomm Research (I2R)9 Institute for Infocomm Research (I2R)9

Server: Contextual Input On Device: Daily Usage Input

Thank you

Contact: wyyau@i2r a star edu sgContact: wyyau@i2r.a-star.edu.sg

For additional information,please visit our website,

htt // i2 t d /http://www.i2r.a-star.edu.sg/

Institute for Infocomm Research (I2R)10