Personalized multimedia museum experiences

download Personalized multimedia museum experiences

If you can't read please download the document

  • date post

    10-Feb-2017
  • Category

    Documents

  • view

    219
  • download

    2

Embed Size (px)

Transcript of Personalized multimedia museum experiences

  • Passive profiling and natural interactionmetaphors for personalized multimedia

    museum experiences

    Svebor Karaman, Andrew D. Bagdanov, Gianpaolo DAmico, Lea Landucci,Andrea Ferracani, Daniele Pezzatini and Alberto Del Bimbo

    Media Integration and Communication Center (MICC)University of Florence, Florence, Italy

    {svebor.karaman, andrea.ferracani, daniele.pezzatini}@unifi.it,{bagdanov, damico, delbimbo}@dsi.unifi.it, lea.landucci@gmail.com

    http://www.micc.unifi.it/vim/people

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 1 / 18

    http://www.micc.unifi.it/vim/people

  • Introduction

    Museums in the digital era

    Modern museums are awash in physical and digital content that they struggleto catalog, maintain, manage, and most importantly to deliver.

    Trade-off: Provide sufficient information to aid visitors understanding whileavoiding cluttering the environment and reducing the enjoyment of theexhibit.

    The museum experience

    Each visitor wants to access different information: more (or less) informationon different artworks depending on their individual interests...

    ... meanwhile current exhibits target a common denominator visitor.

    Visiting a museum should not be a closed experience but a door opened ontoa broader context of related artworks, authors, artistic trends, etc.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 2 / 18

  • Related work

    Personalized access through handheld devices [BBC+01, BBZB07] andpossibly offering some augmented reality experience [KSZ+11, BFVG06].

    Interest modeling for personalization:I by user input [WSS+09] inside the museum and on its website: a virtuous

    circle of online and offline visits.I based on displacement [HW05] in the museum to personalize audio content

    delivery via an audio guide given to each user.

    Cons

    Intrusive: changes the way the visitor behaves with respect to the museum.

    Requires active participation of the user in front of each artwork of interest.

    Specific hardware for each visitor.

    The MNEMOSYNE project was first presented at MM4CH2011 [BDBLP11].Here we report on our experience implementing MNEMOSYNE.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 3 / 18

  • Outline

    1 Introduction2 Related work3 Overview4 Passive interest profiling5 The augmented museum experience:

    I recommendation systems: knowledge and experience basedI natural interaction tabletop systemI mobile system

    6 Field trial: Le Murate prototype7 Conclusions and future work

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 4 / 18

  • Overview

    persondetectorsperson

    detectors persondescribersperson

    describers identitymodelingidentity

    modeling

    vision databasevision database

    identity modeling and passive profiling

    RDF ontologyRDF ontology

    interactive tableinteractive table

    recommend systemsrecommend systems

    natural interaction system

    visitors

    mobile app

    Figure 1: An overview of the MNEMOSYNE architecture.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 5 / 18

  • Passive interest profiling

    Mapping the physical museum

    MNEMOSYNE can use alreadyinstalled cameras in museums.

    Each camera c calibrated to acommon ground plane.

    Ground plane position of eachartwork of interest input using asingle click.

    Sphere of influence of each artwork:a bi-dimensional Gaussian of meanequal to the ground position of theartwork and variances in x and ydimensions defined by the operator.

    Figure 2: MNEMOSYNE installation

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 6 / 18

  • Passive interest profiling

    Person detection and description

    On the video stream of each camera c C, we run a pedestriandetector [BLMP10]. We obtain a set of N person bounding boxes, describedwith a number of visual, temporal and spatial descriptors:

    di ={dai ,d

    si , d

    ti, d

    ci

    }, for i {1, . . . , N}, (1)

    dai : appearance descriptor consisting of RGB and HS color histogramscomputed on overlapping horizontal stripes and the HoG.

    dsi = (dxi , d

    yi ): absolute position of the person detection on the ground plane.

    dti: an integer timestamp. All video streams are synchronized so that dti and

    dtj are comparable.

    dci : index indicating that the detecion comes from camera c.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 7 / 18

  • Passive interest profiling

    Identity modeling

    Fundamental step in passive profiling: associating detectionsD = {di | i = 1 . . . N} to one another to form groups representing individualvisitors in the museum.

    Distance between a description di and model mj is computed taking intoaccount the appearance and all spatio-temporal information available:

    dist(mj , di) = (1 ) ||maj dai ||2 (appearance contribution)(2)+ distw(msj ,dsi , ws) (spatial contribution) (3)+ distw(mtj , dti, wt) (temporal contribution) (4)

    distw(x, y, w) windowed L2 distance: distw(x, y, w) = min(||xy||2

    w , 1)

    ws and wt: spatial and temporal window around observations.

    Weights and control the contribution of spatial and temporal distances.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 8 / 18

  • Passive interest profiling

    Identity modeling

    Real-time constraint prevents the useof onerous tracking algorithms.

    Detection is associated with a modelif distance is less than . A temporarymodel must accumulate detectionsbefore being promoted to a real one.

    Appearance of a model maj : runningaverage of the detections associatedto it. Position and time: those of thelast matched detection.

    Single association from one camera atthe same time to the same model isallowed.

    Figure 3: Detections with theirestimated ground plane positions.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 9 / 18

  • Passive interest profiling

    Passive interest profiling

    Visitors interest profile isbuilt on-the-fly when thevisitor enters the interactivetable area (see figure 4)and is sent to theinteractive table.

    Every detection associatedwith the visitors modelcontributes to each artworkaccording to its positionand the artwork sphere ofinfluence.

    Figure 4: Detection map with artworks sphereof influence and corresponding profile of interestfor one visitor model.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 10 / 18

  • The augmented museum experience

    Recommendation systems

    Knowledge-based recommendation: subset of 8 monitored artworks from the70 artworks of the National Museum of Bargello.RDF ontology:

    I Models the artworks, but also places, events, historical curiosities and otherartworks from all over the world, and their relations to the 8 chosen artworks.

    I Entities: artist, artwork, museum/place and story.I Relations: author, meanings, materials, techniques, historical context,

    location, multimedia content and thematic links (tags).I SPARQL queries: subgraphs of artwork data, stories related to artworks, and

    resources related by tags or stories.

    Experience-based recommendation: suggesting to the user items that aremore likely to be found interesting using previous visitors profile of interest.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 11 / 18

  • The augmented museum experience

    The natural interaction tabletop system

    Metaphor: a hidden museum waiting to be unveiled, an all-digital environmentgoing beyond the physical museum. Two levels of navigation:

    Artworks level : artworks of the museum with visitor level of interest (seefigure 5a). Touching an artwork reveals the related resources.

    Related resources level : structured environment to navigate the multimediacontent related to this artwork. Four different spaces:

    I stories: stories directly related to the artwork in the ontology;I secrets: resources related to the artwork and its related stories in the ontology

    according to the knowledge-based recommendation system;I recommendations: similar artworks according to the experience

    recommendation system using the visitor profile;I insights: resources related by tags or stories to the artwork according to the

    knowledge-based recommendation system.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 12 / 18

  • The augmented museum experience

    (a) (b)

    Figure 5: (a) Detail of the artwork level: the artworks of the museum are representedwith the original title, a thumbnail and a circular symbol visualizing the amount ofinterest showed by the current user during the visit. (b) Detail of the insights space inthe related resources level: information related thematically to the selected artwork.

    Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 13 / 18

  • The mobile system

    Collects personalized digital content from the interactive tabletop interface.Scanning a QR code displayed on the interface of the interactive table,information on the users favorite artworks and resources are retrieved.Map of points of interest suggested by the recommendation system. Extendthe experience of the visit from an ind