Download - Personalized multimedia museum experiences

Passive profiling and natural interactionmetaphors for personalized multimedia

museum experiences

Svebor Karaman, Andrew D. Bagdanov, Gianpaolo D’Amico, Lea Landucci,Andrea Ferracani, Daniele Pezzatini and Alberto Del Bimbo

Media Integration and Communication Center (MICC)University of Florence, Florence, Italy

{svebor.karaman, andrea.ferracani, daniele.pezzatini}@unifi.it,{bagdanov, damico, delbimbo}@dsi.unifi.it, [email protected]

http://www.micc.unifi.it/vim/people

Svebor Karaman et al. (MICC) Personalized multimedia museum experiences September 9, 2013 1 / 18


Introduction

Museums in the digital era

Modern museums are awash in physical and digital content that they struggleto catalog, maintain, manage, and – most importantly – to deliver.

Trade-off: Provide sufficient information to aid visitors’ understanding whileavoiding cluttering the environment and reducing the enjoyment of theexhibit.

The museum experience

Each visitor wants to access different information: more (or less) informationon different artworks depending on their individual interests...

... meanwhile current exhibits target a “common denominator” visitor.

Visiting a museum should not be a closed experience but a door opened ontoa broader context of related artworks, authors, artistic trends, etc.


Related work

Personalized access through handheld devices [BBC+01, BBZB07] andpossibly offering some augmented reality experience [KSZ+11, BFVG06].

Interest modeling for personalization:I by user input [WSS+09] inside the museum and on its website: a “virtuous

circle” of online and offline visits.I based on displacement [HW05] in the museum to personalize audio content

delivery via an audio guide given to each user.

Cons

Intrusive: changes the way the visitor behaves with respect to the museum.

Requires active participation of the user in front of each artwork of interest.

Specific hardware for each visitor.

The MNEMOSYNE project was first presented at MM4CH’2011 [BDBLP11].Here we report on our experience implementing MNEMOSYNE.


Outline

1 Introduction2 Related work3 Overview4 Passive interest profiling5 The augmented museum experience:

I recommendation systems: knowledge and experience basedI natural interaction tabletop systemI mobile system

6 Field trial: Le Murate prototype7 Conclusions and future work


Overview

persondetectors

persondetectors

persondescribers

persondescribers

identitymodeling

identitymodeling

vision databasevision database

identity modeling and passive profiling

RDF ontologyRDF ontology

interactive tableinteractive table

recommend systemsrecommend systems

natural interaction system

visitors

mobile app

Figure 1: An overview of the MNEMOSYNE architecture.


Passive interest profiling

Mapping the physical museum

MNEMOSYNE can use alreadyinstalled cameras in museums.

Each camera c calibrated to acommon ground plane.

Ground plane position of eachartwork of interest input using asingle click.

Sphere of influence of each artwork:a bi-dimensional Gaussian of meanequal to the ground position of theartwork and variances in x and ydimensions defined by the operator.

Figure 2: MNEMOSYNE installation



Person detection and description

On the video stream of each camera c ∈ C, we run a pedestriandetector [BLMP10]. We obtain a set of N person bounding boxes, describedwith a number of visual, temporal and spatial descriptors:

di ={dai ,d

si , d

ti, d

ci

}, for i ∈ {1, . . . , N}, (1)

dai : appearance descriptor consisting of RGB and HS color histograms

computed on overlapping horizontal stripes and the HoG.

dsi = (dxi , d

yi ): absolute position of the person detection on the ground plane.

dti: an integer timestamp. All video streams are synchronized so that dti anddtj are comparable.

dci : index indicating that the detecion comes from camera c.



Identity modeling

Fundamental step in passive profiling: associating detectionsD = {di | i = 1 . . . N} to one another to form groups representing individualvisitors in the museum.

Distance between a description di and model mj is computed taking intoaccount the appearance and all spatio-temporal information available:

dist(mj , di) = (1− α− β)× ||maj − da

i ||2 (appearance contribution)(2)

+ α× distw(msj ,d

si , ws) (spatial contribution) (3)

+ β × distw(mtj , d

ti, wt) (temporal contribution) (4)

distw(x, y, w) windowed L2 distance: distw(x, y, w) = min( ||x−y||2w , 1)

ws and wt: spatial and temporal window around observations.

Weights α and β control the contribution of spatial and temporal distances.



Identity modeling

Real-time constraint prevents the useof onerous tracking algorithms.

Detection is associated with a modelif distance is less than δ. A temporarymodel must accumulate τ detectionsbefore being promoted to a real one.

Appearance of a model maj : running

average of the detections associatedto it. Position and time: those of thelast matched detection.

Single association from one camera atthe same time to the same model isallowed.

Figure 3: Detections with theirestimated ground plane positions.




Visitor’s interest profile isbuilt on-the-fly when thevisitor enters the interactivetable area (see figure 4)and is sent to theinteractive table.

Every detection associatedwith the visitor’s modelcontributes to each artworkaccording to its positionand the artwork sphere ofinfluence.

Figure 4: Detection map with artworks sphereof influence and corresponding profile of interestfor one visitor model.


The augmented museum experience

Recommendation systems

Knowledge-based recommendation: subset of 8 monitored artworks from the70 artworks of the National Museum of Bargello.RDF ontology:

I Models the artworks, but also places, events, historical curiosities and otherartworks from all over the world, and their relations to the 8 chosen artworks.

I Entities: artist, artwork, museum/place and story.I Relations: author, meanings, materials, techniques, historical context,

location, multimedia content and thematic links (tags).I SPARQL queries: subgraphs of artwork data, stories related to artworks, and

resources related by tags or stories.

Experience-based recommendation: suggesting to the user items that aremore likely to be found interesting using previous visitors profile of interest.



The natural interaction tabletop system

Metaphor: a hidden museum waiting to be unveiled, an all-digital environmentgoing beyond the physical museum. Two levels of navigation:

Artworks level : artworks of the museum with visitor level of interest (seefigure 5a). Touching an artwork reveals the related resources.

Related resources level : structured environment to navigate the multimediacontent related to this artwork. Four different spaces:

I stories: stories directly related to the artwork in the ontology;I secrets: resources related to the artwork and its related stories in the ontology

according to the knowledge-based recommendation system;I recommendations: similar artworks according to the experience

recommendation system using the visitor profile;I insights: resources related by tags or stories to the artwork according to the

knowledge-based recommendation system.



(a) (b)

Figure 5: (a) Detail of the artwork level: the artworks of the museum are representedwith the original title, a thumbnail and a circular symbol visualizing the amount ofinterest showed by the current user during the visit. (b) Detail of the insights space inthe related resources level: information related thematically to the selected artwork.


The mobile system

Collects personalized digital content from the interactive tabletop interface.Scanning a QR code displayed on the interface of the interactive table,information on the user’s favorite artworks and resources are retrieved.Map of points of interest suggested by the recommendation system. Extendthe experience of the visit from an indoor to an outdoor scenario.

Figure 6: The mobile application. Left: user’s favorite artworks; Center: in-depth information

on the artwork; Right: map of suggested points of interest.


Field trial: Le Murate prototype

Prototype of the MNEMOSYNE system at Le Murate

Printed four high resolution images of artworks from the Donatello Room.Single surveillance camera and parameters set to: α = β = 0.2, ws = 5m,wt = 80 frames, δ = 0.75 and τ = 10

Critical issues when running the system continuously for several hours

Ensuring that the system does not lag and that profile messages are thus sentin a timely fashion: lag monitor (max 5 secs).

Limiting confusion between several persons having similar appearance.Notion of “active” model: recent last associated detection time wrt averagevisit time in observed area.

First necessary steps towards bringing MNEMOSYNE to real museums.


Conclusions and Future works

Conclusions

MNEMOSYNE objectives: enhance and personalize museum visits.

Passive observation to estimate visitor’s interest.

Interest profile is used conjointly with an ontology in a recommendationsystem to provide personalized content delivery through a natural interactioninterface.

MNEMOSYNE is operational and has been tested.

Future work

Speed-up of person detection and robustness of identity modeling.

Several aspects must be evaluated with further experiments, user studies, andalso potentially on how the suggested resources impact visitor visits.

Using the mobile application as a bridge between different museums adoptingMNEMOSYNE, by suggesting artworks of interest in other museums.


References I

[BBC+01] Chris Baber, Huw Bristow, Sean-Lee Cheng, Anna Hedley, Yuri Kuriyama, Marc Lien, James Pollard, andPhil Sorrell, Augmenting museums and art galleries, Human-Computer Interaction. In: INTERACT, vol. 1,2001, pp. 439–447.

[BBZB07] Erich Bruns, Benjamin Brombach, Thomas Zeidler, and Oliver Bimber, Enabling mobile phones tosupport large-scale museum guidance, MultiMedia, IEEE 14 (2007), no. 2, 16–25.

[BDBLP11] Andrew D Bagdanov, Alberto Del Bimbo, Lea Landucci, and Federico Pernici, Mnemosyne: Enhancingthe museum experience through interactive media and visual profiling, Multimedia for Cultural Heritage,Springer, 2011, pp. 39–50.

[BFVG06] Herbert Bay, Beat Fasel, and Luc Van Gool, Interactive museum guide: Fast and robust recognition ofmuseum objects, Proceedings of the first international workshop on mobile vision, May 2006.

[BLMP10] Alberto Del Bimbo, Giuseppe Lisanti, Iacopo Masi, and Federico Pernici, Person detection using temporaland geometric context with a pan tilt zoom camera, Pattern Recognition (ICPR), 2010 20th InternationalConference on, IEEE, 2010, pp. 3886–3889.

[HW05] Marek Hatala and Ron Wakkary, User modeling and semantic technologies in support of a tangibleinterface, Journal of User Modeling and User Adapted Interaction 15 (2005), no. 3-4, 339–380.

[KSZ+11] Tsvi Kuflik, Oliviero Stock, Massimo Zancanaro, Ariel Gorfinkel, Sadek Jbara, Shahar Kats, Julia Sheidin,and Nadav Kashtan, A visitor’s guide in an active museum: Presentations, communications, andreflection, Journal on Computing and Cultural Heritage (JOCCH) 3 (2011), no. 3, 11.

[WSS+09] Yiwen Wang, Natalia Stash, Rody Sambeek, Yuri Schuurmans, Lora Aroyo, Guus Schreiber, and PeterGorgels, Cultivating personalized museum tours online and on-site, Interdisciplinary Science Reviews 34(2009), no. 2-3, 2–3.


Passive profiling and natural interactionmetaphors for personalized multimedia

museum experiences

Svebor Karaman, Andrew D. Bagdanov, Gianpaolo D’Amico, Lea Landucci,Andrea Ferracani, Daniele Pezzatini and Alberto Del Bimbo

Media Integration and Communication Center (MICC)University of Florence, Florence, Italy

{svebor.karaman, andrea.ferracani, daniele.pezzatini}@unifi.it,{bagdanov, damico, delbimbo}@dsi.unifi.it, [email protected]


