ExhiSTORY: IoT in the service of Cultural Heritage

Vassilis Poulopoulos1, Costas Vassilakis1, Angela Antoniou2, Manolis Wallace1, George Lepouras2 andMartin Lopez Nores3

Abstract— Creating stories for exhibitions is a fascinatingand in parallel laborious task. As every exhibition is designedto tell a story, museum curators are responsible for analyzingeach exhibit in order to extract messages that form a storyand position accordingly the objects in correct order within themuseum space. In this context, we analyze how the technologicaladvances in the fields of sensors and the Internet of Thingscan be utilized in order to construct a “smart space”, whereexhibits can communicate with the visitors and to each otherand can be organized automatically so that they can generaterich, personalized, coherent and highly stimulating experiences.We present the architecture of the system named “exhiSTORY”,that intends to provide the appropriate infrastructure to be usedin museums and places where exhibitions are held, in order tosupport smart exhibits. We discuss and analyze the architectureof the system and the ways of its application in a cultural space.

I. INTRODUCTIONA museum exhibition consists of a number of objects, that

are carefully selected and positioned together, and its purposeis to have one or more stories to tell. According to Thaggert[1], exhibitions are designed to “narrate a story”, a procedurethat requires curators of high training [2], whose role andwork is directly related to a cultural space’s budget [3].As the aforementioned tasks are usually extremely tediousand time consuming, the exhibits’ storytelling potential aretypically underutilized, and only few of the stories that couldbe possibly narrated are actually “woven” and presented tothe visitors.

In theory, a curator has to set up an exhibition givena set of objects and limited space [4]. In practice, thecurator has to detect interconnections between the objects,and find appropriate “stories” that can be told. In parallel, inorder to maximize the experience of the visit, a number ofsecondary stimuli are used, so that each visitor can recognizea personal “hidden story” behind the exhibits. Despite theaforementioned personalization procedures, the final numberof stories told by the exhibitions is practically limited bythe number of stories “woven” by the curator, which are inturn bound by the resources that are made available to thenarration creation process.

In reality, cultural objects have many different stories totell, especially when they are put together. For example,

1V. Poulopoulos, M. Wallace, Knowledge and Uncertainty Research Lab-oratory, University of the Peloponnese, Greece {vacilos, costas,wallace}@uop.gr

2C. Vassilakis, A. Antoniou and G. Lepouras, Department of In-formatics and Telecommunications, University of Peloponnese, Greece{angelant, gl}@uop.gr

3M. Lopez Nores, University of Vigo, Spainmlnores@det.uvigo.es

Titian’s work Diana and Callisto1 (figure 1a) can tell usabout ancient Greek gods; about deception, wrath and hu-miliation; nudity in art; it exposes the trends of the periodand it reveals the personal style of the artist. But whenthe aforementioned masterpiece is accompanied by Jan vanEyck’s painting The Arnolfini Portrait2 (figure 1b), a newset of stories and connections can emerge, like wantedand unwanted pregnancies (both show pregnant women indifferent situations), maternity practices, and so forth.

Practically, by using the conventional ways of presentationin a museum, it is impossible to cover all aspects of thestories, personalize on users, or utilize all available objectsto the maximum extent. But, with the help of the Internet ofThings, technologies and advanced algorithmic techniques, itis possible to construct stories, and consequently experiencesfor the users, which are larger in numbers, rich in content,diverse regarding their message, and personalized to theliking of individual users.

(a) Titian’s “Diana and Calisto” (b) Jan Van Eyck’s “The ArnolfiniPortrait”

Fig. 1: Exhibits

In this paper, the definition of smart exhibits is analyzed.Smart exhibits are self-aware, meaning that they have knowl-edge of their own data, and they are able to recognize exhibitsin proximity; they can self-organize and set-up more than asimple presentation of themselves in an exhibition, leadingto maximization of experience in cultural spaces. In thisnotion, a framework called exhiSTORY (from the Greekword εξιστoρει: tells a story) is developed [5]. The frame-work allows cultural objects to identify other exhibits thatare in the same area, cooperate and exchange information,

1Diana and Callisto https://www.nationalgallery.org.uk/paintings/titian-diana-and-callisto

2The Arnolfini Portrait http://photodentro.edu.gr/lor/retrieve/44427/15arnol.jpg

75

in order to create several exhibitions that are formulatedautomatically, while each one can narrate a relevant story. Inthis work, we focus on the setup of the special infrastructure,that enables interconnection and interaction between exhibitsand visitors.

The rest of this paper is organized as follows: in sectionII we discuss the related work concentrating on the researchon Internet of Things in relation to cultural heritage, while insection III we analyze the methods that can be used in orderto implement the main object that forms the smart space,which is the smart object. Our proposed system, exhiSTORY,and more specifically its architecture is presented in sectionIV, while the next section explains the usage of the systemin operation. Finally, we conclude in section VI with ourremarks.

II. RELATED WORK

The research area of IoT is evolving expeditiously, and ac-cording to Boring et al. novel functionalities in our everydaylife is realized with the extensive use and combination ofcommunication technologies, sensors, discovery operationsand mechanisms. When examined in relation to culturalheritage, the IoT can shape both smart objects as wellas smart spaces. Chieanese and Piccialli [7] explore theconcept and capabilities of the smart space as a unit. Withan idea similar to what we present, they define a formof “smart exhibit”, where a centralized system can fetchdata from the clients, which in this case are the culturalobjects. Still, these works do not address the issue of self-organizing exhibitions, nor they describe an architecture forthe IoT-based system. Hashemi et al. [8] have done extensiveresearch in the combination of the visit context and IoTleading to identification of several factors that affect themuseum visits, in a study of indoor cultural activities. A moreinteractive system, based on IoT, including “discussions”between exhibits and the visitors is expressed though theiPhone App titled “Take me I’m yours”, which is part of alarger project [9].

According to Wecker et al. [10], any analysis regardingcultural spaces has to take into account the mobility factor,since visitors are “moving” within the cultural space. Andwhile research on IoT examines ways to assist peoples’ everyday activities, the research related to cultural heritage isputting the focus on the interaction of the user’s mobile de-vice with the surrounding environment [11]. Practiced at theUCL Grant museum of Zoology, the award winning projectentitled “QRator”, is an innovative project that introduced theinteractive labels in museum items, that were able to createtheir own narratives in order to expand visitor engagement[12]. Extensive use of IoT technologies was done within thescope of the project.

The presented system, exhiSTORY, is complementary tothe approaches listed above; the goal of the system is toexpand narrative generation capabilities by applying IoTtechnologies within the cultural spaces, enabling communi-cation and collaboration between the cultural objects, whichwill be capable of storing their own semantic information.

Having as our basis, that a cultural related object will be ableto store data, as well as communicate with its environment,which is other objects and visitors, we are proposing anarchitectural schema consisting of ad-hoc networks betweenexhibits and people.

III. THE SMART EXHIBIT

The conventional way of an exhibition setup implies thateach object is followed by a label, presenting in general: atitle, a brief description and the artist. The dynamic creationof exhibitions and the self-organizing of objects impliesthat each object must be followed by a larger number ofcontinuously updated information.

In more detail, the information that needs to be storedin each exhibit (smart object) includes data such as ori-gins, meaning, interpretations and context. In parallel, thisinformation must be dynamically updated, and differentimplementation of IoT technologies may offer differentiatedsmart objects and alterations to the exhiSTORY framework.The different implementation approaches are reviewed belowin an attempt to identify the desired characteristics forexhiSTORYs infrastructure.

Basic implementation. In this implementation an RFIDtag is attached to the exhibits. This implementation caneither store a unique object identifier (can be referred toas Elementary implementation), or it can possibly store anumber of elemental information including title, artist anddescription. The RFID is not interactive, but can be realizedby a grid of readers installed into the different rooms.The readers should be connected with a centralized serverhosted in the cultural space in order to observe each objectexact location. A hosted infrastructure is responsible forthe updates to the objects, ensuring the execution of the“logic” for the visitor communication, dynamic exhibitionformulation and data exchange, as well.

Memory-rich implementation. The memory-rich imple-mentation overcomes problems of connectivity and mem-ory size of the basic solution. FlashAir cards [13] are atechnological solution that can provide both memory andwireless LAN connectivity. They can be positioned ontothe objects and offer the possibility of both storage andconnection. This means that the existing information, andany derived or computed information, can possibly be storedwithin the exhibit. The specific scheme enables mobility andautomation during exhibition organization. As the exhibitslack computational power a local server is responsible forthe exhibition formulation and the stories’ adaptation to thecontext and the visitors.

Agent implementation. In the agent implementation eachexhibit is attached a small device (eg. a low cost embeddeddevice; smartphones could also be used), and thus can actas an intelligent software agent. The system is able to createad-hoc networks formulated by combinations of objects andusers, in order to perform the desired procedures. The levelsof independence of each exhibit is such that the system isnot limited to indoor settings.

76

(a) Memory-rich implementation (b) Agent implementation

Fig. 2: Implementations

Having a significantly lower cost, while in parallel of-fering connectivity and storage, in this work we discuss animplementation based on the memory-rich implementation.

IV. SYSTEM ARCHITECTURE

The system architecture consists of three main subsystems:the “smart space”, the intelligent modules and the knowledgebase (KB).

Fig. 3: The exhiSTORY architecture

Important role in the main idea of exhiSTORY is thatexhibits are able to change position, meaning that the smartspace is a dynamically defined area, that is able to identifychanges in location, and additions or removals of objects.In the case described (i.e. the memory-rich implementation),the location of the objects is established with the help of theWiFi network.

Information about the style of the exhibition and ways ofpresentation form the the museum’s context which is partof the knowledge base. Semantics that is curated informa-tion related to the cultural space’s interest, media, that isany multimedia available that can accompany objects andpresentations and finally, map information that is helpful forpositioning of objects and users in the area, form all the otherinformation included in the knowledge base as well.

The core software of the system is beyond the scopeof this paper, which focuses on the specification of the

sensors’ architecture. In brief, the intelligent modules of thesystem include: the exhibit tracker responsible for trackingthe exact location of each object, the semantics engine, thatis responsible for collecting information about exhibits andtopics related to them, the media engine, responsible foradding extra multimedia to already existing content, the storymaker a module that is responsible for analyzing the contextof each exhibit and locating bonds between the objects withinthe semantics. By utilizing either implied or definitive dataabout each visitor, the user profiler completes the set ofintelligent modules.

V. OPERATION OF THE SYSTEM

exhiSTORY system consists of two main processes, thefirst one being the initialization of the system, while thesecond is the execution during the person’s visit in themuseum.

The configuration process is initiated by establishing infor-mation about the identity and location of each exhibit withinthe smart space. Furthermore, the semantic information andmedia associated with the exhibits are retrieved, to be pro-cessed by the the story maker. The story maker in turn, canuse the data of the smart exhibits in order to develop severalnarrations. As long as the system is aware of the position ofeach object in the smart space, the module identifies groupsof exhibits that could be linked into coherent stories throughsemantic links, and subsequently, creates accordingly thevisitor’s path, i.e. the specification of the exhibits that willbe viewed and in what sequence within the narration ofthe story, complemented by the information and the mediacontent displayed for each exhibit. It is possible that morethan one stories can be produced for a single path, while thestories possibly adapt to the user - a procedure that is partof the visitor’s experience.

The user profiling module plays an important role for thenext step as it is the main input for the story selectionprocedure which finally delivers the produced experienceto the visitor. Naturally, visitors must be equiped with anaccess device (e.g. a smartphone or a tablet), since thememory-rich scenario on which this paper focuses doesnot support a display screen on the exhibits’ side; networkconnectivity is also required. The mobile device needs tohave the exhiSTORY application installed and running. Theapplication is necessary to be installed on the visitor’s end-device, so it must be installed before visiting the culturalspace or it can be downloaded from the museum’s network.

At first the application extracts data about the user profilewith the use of some serious games that are related to culture.Despite the fact that this procedure is not compulsory, itis a simple, engaging and effective means of collectingdata about the user in order to enhance their experience.For example, Bampatzia et al. [14] discuss how the MBTIdimensions related to a user’s profile can be extracted evenfrom simple serious games. The overal user experience canbe furthermore enhanced by data sourced from social mediacontent; these data are not directly related to the visitor butform a part of the current cyberspace information context,

77

e.g. trending topics on Tritter, which reflect the topics of thecyberspace “headlines” [15].

When dealing with smart museum application severalissues arise but we will concentrate on the two most im-portant related to our implementation. The first one dealswith the accuracy of the user’s location within the smartspace, while the second one is the possibility, of an exhibitor an exhibition, having more information to be sent to thevisitor and when this should be done. In relation to the firstissue, as the system already employs a way for preciselyfinding the location of every exhibit, the same method canbe used for the visitors as well. We are referring to WiFipositioning through triangulation. As for the second matter,in order to measure when it is the correct timing to sendupdated information to the user, some simple techniquesthat are related to the positioning of the user, and morespecifically how close she is to an exhibit can be reallyhelpful. In parallel, there should be an option of sendinginformation upon explicit request of the user. The later can beproved to be more accurate, but on the other hand, discussionon the smart space must include fast, efficient and correctinformation updates for the users.

VI. DISCUSSION AND CONCLUSIONS

In this work, we described the architecture and operationof a system that is able to dynamically formulate exhibi-tions, called exhiSTORY. Taking into consideration severalparameters related to each single visitor, and interconnectingthem with the variety of data existing for each object ofthe exhibition the system is able to maximize the experiencewithin the cultural space by offering unique narrations.

The basis for our work is the CrossCult EU Project. TheProject aims to spur a change in the way European citizensappraise History. Within its tasks the exhiSTORY systemhas its own implementation as part of the project’s pilots.The controlled environment will help us observe the pro-cess where the exhibition re-configures itself when specificactions take place like re-arranging the objects, adding newones or even removing some. The day by day evolution ofan exhibition can also be studied by checking the differentcontent generated for each individual.

The concept of the exhiSTORY system is the “smartplace” that includes “smart exhibits”, i.e. self-organizingautonomous objects. In this notion we are creating a newresearch path with a large amount of challenges to be faced.We analyzed a number of alternatives that can be used inthe technical part and described the solutions for each ofthe cases in order to produce a complete system, accordingto each museum’s particularities. In these cases we need toexamine the interconnection of data in different museums aswell as the cost analysis of the seamless and agent drivenimplementations that seem to have large cost especially formuseums with large numbers of objects.

ACKNOWLEDGMENTS

This work has been partially funded by the projectCrossCult: Empowering reuse of digital cultural heritage in

context-aware crosscuts of European history, funded by theEuropean Union’s Horizon 2020 research and innovationprogram, Grant#693150

REFERENCES

[1] Thaggert, M. (2010). Images of Black modernism: Verbal and visualstrategies of the Harlem Renaissance. Amherst: University of Mas-sachusetts Press.

[2] Lord, G. D., & Lord, B. (2009). The manual of museum management.Lanham, MD: AltaMira Press.

[3] Lord, B., Lord, G. D., Nicks, J., & Great Britain. (1989). Thecost of collecting: Collection management in UK Museums. London:H.M.S.O.

[4] Mette Houlberg Rung (2016) Thoughts about an exhibition- using materiality and sensuousness as a narrative device,http://www.smk.dk/en/visit-the-museum/news/artikel/translate-to-english-udstillingsovervejelser- materialitet-og-sanselighed-som-fortaellegreb/, retrieved August 18th, 2018.

[5] Vassilakis, C., Poulopoulos, V., Antoniou, A., Wallace, M., Lepouras,G. and Lopez Nores, M., exhiSTORY: Smart Exhibits That Tell TheirOwn Stories, Future Generation Computer Systems, Volume 81, April2018, pp. 542-556.

[6] Brring, A., Datta, S.K. and Bonnet, C. (2016). A Categorization ofDiscovery Technologies for the Internet of Things. In Proceedings ofthe 6th International Conference on the Internet of Things (IoT’16).ACM, New York, NY, USA, 131-139. DOI: https://doi.org/10.1145/2991561.2991570

[7] Chianese, A. and Piccialli, F. (2014). Designing a smart museum:when Cultural Heritage joins IoT. Proceedings of the Eighth Interna-tional Conference on Next Generation Mobile Applications, Servicesand Technologies, DOI: https://doi.org/10.1109/NGMAST.2014.21

[8] Hashemi, S.H., Hupperetz, W., Kamps, J. and van der Vaart, M.(2016).Effects of Position and Time Bias on Understanding Onsite Users’Behavior. In Proceedings of the 2016 ACM on Conference on Hu-man Information Interaction and Retrieval (CHIIR ’16). ACM, NewYork, NY, USA, 277-280. DOI: http://dx.doi.org/10.1145/2854946.2855004

[9] Speed, C. and Shingleton, D. (2012). Take me I’m yours: mim-icking object agency. In Proceedings of the 2012 ACM Confer-ence on Ubiquitous Computing (UbiComp ’12). ACM, New York,NY, USA, 1167-1170. DOI: http://dx.doi.org/10.1145/2370216.2370465

[10] Wecker, A.J., Lanir, J., Kuflik, T. and Stock, O. (2015). WhereTo Go And How To Get There: Guidelines For Indoor Landmark-Based Navigation In A Museum Context. In Proceedings of the17th International Conference on Human-Computer Interaction withMobile Devices and Services Adjunct (MobileHCI ’15). ACM, NewYork, NY, USA, 789-796. DOI: http://dx.doi.org/10.1145/2786567.2793702

[11] de Freitas, A.A., Nebeling, M., Chen, X.A., Yang, J., Ranithangam,A.S.K.K. and Dey, A.K. (2016). Snap-To-It: A User-Inspired Platformfor Opportunistic Device Interactions. In Proceedings of the 2016CHI Conference on Human Factors in Computing Systems (CHI ’16).ACM, New York, NY, USA, 5909-5920. DOI: https://doi.org/10.1145/2858036.2858177

[12] Hudson-Smith, A., Gray, S., Ross, C., Barthel, R., de Jode, M.,Warwick, C. and Terras, M. (2012). Experiments with the internetof things in museum space: QRator. In Proceedings of the 2012ACM Conference on Ubiquitous Computing (UbiComp ’12). ACM,New York, NY, USA, 1183-1184. DOI: http://dx.doi.org/10.1145/2370216.2370469

[13] Toshiba (2017). FLASHAIRTM. http://www.toshiba-memory.com/cms/en/products/wireless-sd-cards/FlashAir/, retrieved August 18th,2018.

[14] Bampatzia, S., Bourlakos, I., Antoniou, A., Vassilakis, C., Lepouras,G., Wallace, M. (2016)Serious games: Valuable tools for culturalheritage, Games and Learning Alliance Conference (GALA 2016),Dec 5-8, Utrecht, The Netherlands.

[15] Vassilakis, C., Antoniou, A., Lepouras, G., Poulopoulos, V., Wallace,M., Bampatzia, S., Bourlakos, I., Stimulation of Reflection and Dis-cussion in Museum Visits through the Use of Social Media, SocialNetwork Analysis and Mining, 7:40, December 2017.

78