The Ontology for a Multi-Sensory Media ServiceSeungwon Oh and Minsoo Hahn

Digital Media Lab, Information and Communications University, Seoul 135-854 Korea{aegis901, mshahn}@icu.ac.kr

Abstract- Under the development of ubiquitous technologies,people need a novel media system. That means the multi-sensorymedia service which satisfies the five senses and makes peoplefeel realistic. It is necessary to build knowledge about sensoryeffects and devices in order to provide the service. In this paper,we propose an ontology-based knowledge modeling. Theproposed ontology model consists of Effect Ontology and DeviceOntology. The Effect Ontology represents semantic informationabout multimedia contents and sensory effects synchronized withthe contents. The Device Ontology represents semanticinformation about devices generating sensory effects. By usingthe model, we can infer sensory effects and their attributes fromthe predefined semantic information of multimedia contents. Weproved the usefulness of our ontology model by showing resultsaccording to scenario-based experiments.

Keywords - ontology, sensory, media, reality, semantic,representation.

1. Introduction

Ubiquitous computing has become an integral concept in avariety of areas. Through ubiquitous technologies, the devicesfor daily use have had various functions and features overtraditional functions and features. In other words, a traditionalenvironment which had just predefined simple functions hasbeen changed to an environment providing user-focusedcomplex services with ubiquitous technologies. Theremarkable development of ubiquitous technologies indicatesthe necessity of a rather improved novel media system.

Now, people can enjoy an audio-visual multimedia contentat home equipped with advanced audio-visual devices.However, it entertains only our eyes and ears. In the future,people will need the novel media service which satisfies thefive senses and which lets people feel realistic [1]. In thispaper, we will use the 'Multi-Sensory Media Service' as theterminology of this noble service. Figure 1 shows thearchitecture of the multi-sensory media service.

IIII

As shown in figure 1, the service provides users with sensoryeffects generated by sensory devices and audio-visual contents.A media player not only plays contents but also communicateswith other sensory devices. In this case, a sensory effect meansan effect to augment feelings by stimulating human sensoryorgans according to a multimedia content. A sensory devicemeans a device to generate sensory effects such as light, wind,heat, incense, and etc.

In order to provide the multi-sensory media service, manyrelated researches should be studied. First, noble contents haveto be created and devices which can make sensory effects haveto be developed. Next, the service requires information aboutsensory effects for representing multimedia contents andsensory devices for generating sensory effects. Finally, theservice needs a mechanism which manages the informationand controls sensory devices.

In order to meet the requirements for the multi-sensorymedia service, we propose an ontology-based semanticknowledge representation for the service. Ontologies areuseful in building machine-readable semantic knowledge.Ontology-based semantic knowledge has importantadvantages. First, it is easy to share knowledge betweenheterogeneous devices. Next, it has a good structure to expandknowledge with increasing contents and devices. Last, it isuseful for knowledge reuse and logical inference based on asemantic structure.

Therefore, we proposed an ontology-based knowledgemodeling. The proposed ontology model represents semanticinformation about multimedia contents, sensory effectssynchronized with the contents and sensory devices.

2. Related Work

Ubiquitous computing is a computing paradigm that acomputing power becomes invisible in our lives. We caneasily access to computing power anywhere, anytime andanyway through intelligent interfaces. Mark weiser says aboutUbiquitous computing as follows: "Its highest ideal is to makea computer so embedded, so fitting, so natural, that we use itwithout even thinking about it." [2]. One of the main goals ofubiquitous computing is to support relevant information whenit is needed. Therefore, in order to provide a service inubiquitous environment, a knowledge modeling whichrepresents the context about a user or environment is necessary.Ontologies are widely used for modeling context in ubiquitousenvironment because an ontology is useful in knowledgesharing, logic inference and knowledge reuse [4].

There are many researches that utilize ontology to providecontext-aware services, for examples, CoBrA-ONT (2003) [3],

ISBN 978-89-5519-139-4 -1025- Feb. 15-18,2009 ICACT 2009

Authorized licensed use limited to: Korea Advanced Institute of Science and Technology. Downloaded on December 9, 2009 at 21:43 from IEEE Xplore. Restrictions apply.

CONON (2004) [4], SOUPA (2004) [5], GAS (2005) [6], andUbiquitous User Modeling [7].

First, CoBrA-ONT (Context Broker Architecture-Ontology)represents the infonnation about places, agents, events andtheir associated properties in an intelligent meeting roomdomain [3]. In order to support pervasive context-awaresystems, CoBrA-ONT was developed as a component of theContext Broker Architecture.

Second, CONON (Context Ontology) is an ontology whichhas a hierarchical structure. In pervasive computingenvironments, CONON defines context modeling andsupports logic-based context reasoning [4]. CONON consistsof an upper ontology and a domain-specific ontology. Theupper ontology represents general concepts aboutdomain-independent context and the domain-specificontology represents domain-dependent concepts forextensibility.

Third, SOUPA (Standard Ontology for Ubiquitous andPervasive Applications) was developed in order to model andsupport pervasive computing applications [5]. SOUPA wasimplemented with OWL. In order to represent intelligentagents, SOUPA includes various concepts such as userprofiles, actions, events, space, time, beliefs, desires,intentions, and policies for security and privacy. In order torepresent the above concepts, SOUPA references otherontologies, for example, the COBRA-ONT, the RegionalConnection Calculus, the Friend-Of-A-Friend ontology, thespatial ontologies in OpenCyc, the DAML_Time, theMoGATU BDI ontology, and the Rei policy ontology.

Fourth, GAS (Gadgetware Architectural Style) Ontology isan ontology that was developed in order to represent thesemantics of the main concepts of a ubiquitous computingenvironment and define their inter-relations [6]. The GASOntology consists of the GAS Core Ontology and the GASHigher Ontology. The GAS Core Ontology describes thecommon language to communicate and/or collaboratebetween different domains. The GAS Higher Ontologyrepresents both the description ofeach domain and its acquiredknowledge.

Last, Heckmann proposed a complete framework to realizethe new ubiquitous user modeling [7].SITUATIONALSTATEMENTS and USERML weredeveloped on the syntactic level, while UBISWORLDontology and GUMO, the general user model ontology weredeveloped on the semantic level. TheSITUATIONALSTATEMENTS represents descriptions ofsituations like user model entries, context infonnation orlow-level sensor data.

Previous ontology researches focused on the user's state andbehavior for context-aware services. Many useful ontologymodels were implemented for context-aware services.However, they are not sufficient to provide the multi-sensorymedia service. It is necessary to develop the novel ontologymodel that includes not only user context knowledge but alsomultimedia content knowledge, service context knowledgeand sensory device knowledge. Also, the ontology model hasto allow for an expansion possibility and an inferencepossibility. In this paper, we proposed the new ontology modelconsidering contents and devices for the service.

3. Multi-Sensory Media Service

The multi-sensory media service has the overall architectureas shown in figure 1. As shown in figure 1, the serviceprovides users with sensory effects generated by sensorydevices and audio-visual contents. In order to provide theservice, first, a fonnal communication protocol must bedefined in order to communicate between the player andsensory devices. Second, the standardization of stimuluswhich makes people feel realistic must be defined. Based onthe standardization, many sensory devices can stimulatepeople to feel realistic. Third, knowledge which representsinfonnation about multimedia contents and sensory devices isnecessary. According to the given multimedia contentsinfonnation, the player defines sensory effects synchronizedwith the contents and sensory devices provide people with thesensory effects. Although not representing in figure 1, thecontext ofa user or environment may be considered because ofthe nature of the service.

The service is not a predefined simple service but a flexiblecomplex service based on the infonnation of multimediacontents and sensory devices. Also the service consists ofdifferent combinations of sensory devices and providesvarious multimedia contents. Considering the characteristicsof the service, the knowledge which has the following threefeatures is necessary for the service. First, the knowledge hasto be interpreted semantically for the semantic interoperabilityamong different devices. Second, the knowledge has to bestandardized for various devices to share and exchange theknowledge each other. Last, the knowledge has to be possibleto infer new infonnation. Because the knowledge cannotprepare infonnation for all situations, the service must bepossible to infer new infonnation based on the usableknowledge.

:lick wiad. ..... i8ccae. __

Figure 2. Ontology-based MUlti-Sensory Media Service

Therefore, in this paper, we propose an ontology-basedsemantic knowledge representation as a method ofrepresenting infonnation for the multi-sensory media service.Figure 2 shows the ontology-based service architecture. Infigure 2, the Effect Ontology means the semanticrepresentation about sensory effects and multimedia contents.The Device Ontology means the semantic representation aboutsensory devices installed in the service environment. TheInference Engine means the mechanism to infer necessaryinfonnation from given infonnation.

ISBN 978-89-5519-139-4 -1026- Feb. 15-18, 20091CACT 2009

Authorized licensed use limited to: Korea Advanced Institute of Science and Technology. Downloaded on December 9, 2009 at 21:43 from IEEE Xplore. Restrictions apply.

and their relationships. Also we added concepts for thesensory effects to the Effect Ontology.

CIIIS

PropertyI~

The Place Class represents the place information about eachscene ofmultimedia contents. In this case, the role of the PlaceClass is to define influences on Heating and Cooling effects ofeach scene. Although the Place Class meant locationinformation in many different ontologies, we designed fivesubclasses of the Place Class according to climate as shown infigure 6. Therefore, we can define influences on the Heatingand Cooling effects according to the climatic characteristics ofplaces.

The Time Class also represents the time information abouteach scene of multimedia contents. The time information alsohas influence on the sensory effect. The Time Class has theSeason Class and the Day Class as subclasses as shown infigure 7. The Day Class has the following individuals: dawn,morning, forenoon, noon, afternoon, evening and night. TheSeason Class has the following individuals: spring, summer,fall and winter.

c:::> Clus

---+ Property.•••.•• Subclus

The Scene Class represents the description of multimediacontents. The information for the multi-sensory media serviceis organized by the unit of a scene. The Scene Class hasrelations with different classes such as Place, Time andElement as shown in figure 5. In other words, The Scene Classincludes the information of place, time and elementspredefined by a content producer.

~hasElcmcnt

~Macc~hasTirne

~Figure 5. Scene Class

Figure 4. Physical Environment for the Multi-Sensory Media Service

In summary, an ontology is suitable for representingknowledge necessary for the multi-sensory media servicebecause of the following advantages:

- A formal and explicit knowledge representation- A machine-readable semantic knowledge representation- Knowledge sharing between heterogeneous systems- Easy knowledge expansion- Knowledge reuse and logical inference based on a semantic

structureFigure 3 and figure 4 show the physical experiment

environment for the multi-sensory media service. To stimulateuser's five senses, we constitute various devices such as adisplay for video, a 5.1 channel speaker for sound, a light forbrightness, a heater for temperature, a fan for wind and aperfume spray for smell. Practically, we constructed thesystem which satisfies user's senses except for taste.

.,...­Figure 3. Physical Environment for the Multi-Sensory Media Service

Subclus

ClusI::?CT:::?0··C~v

Figure 7. Time Class

The Element Class represents the subject of sensory effects,for example, Sun, Wind, Water, Fog, and so on. In other words,we define sensory effects for representing element information

4. Effect Ontology

The Effect Ontology represents the information of sensoryeffects synchronized with multimedia contents. Appendixesshow the whole structure of the Effect Ontology. Beforerepresenting the sensory effects, the content information mustbe defined ahead. A scene means the minimum unit ofmultimedia contents like movies. A producer puts content intoa movie by the scene. Each scene has basically severalcomponents such as place, time, actor, object, event, and so on.Therefore, in the Effect Ontology, we defined the concept of ascene and then described many concepts included in scenes

ISBN 978-89-5519-139-4 -1027- Feb. 15-18,2009 ICACT 2009

Authorized licensed use limited to: Korea Advanced Institute of Science and Technology. Downloaded on December 9, 2009 at 21:43 from IEEE Xplore. Restrictions apply.

of each scene predefined by a content producer. Figure 8shows the structure of the Element Class. 5. Device Ontology

-+ Property••••••• Subdus

<::> c.0 Value

-+ Property......• SubclaM

~claas

o Value

-+ PropertyATTRIBL'TE

huDircdion ......I1 DIRECTIONhasEfrect

~Figure 10. Device Class

Figure 10 shows the structure of the Device Class. TheDevice Class has the information of installed position in theservice environment by the hasDirection relation. Also, theDevice Class has the information of possible generatingsensory effects and effect attributes.

The Device Ontology represents the information of sensorydevices generating sensory effects in the multi-sensory mediaservice. Appendixes show the whole structure of the DeviceOntology. We defined the Device Ontology in order todescribe each sensory device. In other words, the DeviceOntology defines characteristics of sensory devices installedin the service environment.

The Device Ontology shares the Attribute Class and theEffect Class in the Effect Ontology.

6. Ontology Development and Evaluation

The Web Ontology Language (OWL) is a language fordefining and instantiating Web ontologies. In order to expandthe limited expressiveness of RDF Schema, a more expressiveOWL has been defined by the World Wide Web Consortium(W3C) [8]. So far, OWL is the most widely used andoutstanding ontology language in terms of expressiveness andreasoning capabilities. That's why we selected OWL amongmany ontology languages.

In order to build OWL-based knowledge, we used Protege3.4 [9] as an ontology editing tool. Protege is a free, opensource ontology editor and knowledge-base framework.Protege supports GUI environments for building ontologies.

As ontologies are widely used in a variety of areas, ontologyevaluation has become an important issue. The goal of anontology evaluation is to reuse a useful ontology whichdefines correctly an interesting domain. To evaluate anontology, the following criteria were identified: consistency,completeness and conciseness [10].

First, consistency means whether contradictory knowledgecannot be inferred from other axioms. Second, completenessmeans whether knowledge can represent explicitly a relateddomain. Last, conciseness means whether unnecessaryredundancies or definitions can exist.

In order to prove our ontology, first, we utilize Protege [9]with a consistency evaluation tool. Protege also supports thecorrectness of syntax and logic in building ontologies.

Figure 8. Element Class

The Element Class has the effect information related to eachelement by the hasEffect relation and direction information bythe hasDirection relation. In figure 8, the dotted classesrepresent the classes that are not fully considered in this paper.

The Effect Class represents the sensory effects to stimulatepeople. The information of the Effect Class is inferred by theinformation of a scene. Figure 9 shows the structure of theEffect Class. In figure 9, the dotted classes represent theclasses that are not fully considered in this paper.

We can define each effect according to each element. Eacheffect has attributes defined by information belonging to eachscene. The Attribute Class represents the attributes whateffects can have. We defined the following attributes:BreadthHorizon, BreadthVertical, Color, Frequency,RotateHorizon, RotateVertical, and Temperature. TheBreadthHorizon means that an effect spread horizontally. TheBreadthVertical means that an effect spread vertically. TheColor defines a color of an effect. The Frequency defines thenumber of occurrences of a repeating effect per unit time. TheRotateHorizon means that an effect rotates horizontally. TheRotateVertical means that an effect rotates vertically. TheTemperature means that an effect changes circumferencetemperature.

Also, the Effect Class can have direction and intensityinformation. The direction of an effect is inferred from thedirection of an element defined by a content producer. Theintensity of an effect is defined as the relative value inferredfrom information belonging to each scene.

"..-------.) e'~:-~~mm

: .~. baaDinldion---lcdJ::~~::~:: ~~ D1RECno~

LIGHT \ •••• 1 NTESSITI

eaLj···.~:~CtmA~:>

Figure 9. Effect Class

ISBN 978-89-5519-139-4 -1028- Feb. 15-18, 20091CACT 2009

Authorized licensed use limited to: Korea Advanced Institute of Science and Technology. Downloaded on December 9, 2009 at 21:43 from IEEE Xplore. Restrictions apply.

As shown in above tables, we can confirm that propersensory effects are inferred by the proposed ontology model.Our ontology model can fully represent multimedia contentsand sensory devices for the multi-sensory media service. Also,ifnew contents and devices are added, the ontology model canbe easily expanded by adding just related information withoutits structural change because of the model designconsideration. Therefore, a content provider can determine thesensory effects synchronized with contents by defining simplescene information in order to provide the multi-sensory mediaservice.

Second, we show the result of various scenarios. Scenariosconsist ofdifferent scenes ofmultimedia contents. We explainthe inferred each effect information from each scene.

The experiment scenarios are as follows. The followingfigures are screenshots of scenes. Each table shows theinferred information according to predefined information ineach scene.

Figure 11. Dessert Scene

Table 1. Semantic Representation of the Dessert Scene

Given InformationPlace TimeTemperate Afternoon

ElementTornado

Inferred InformationEffect AttributeFan RotateHorizon

RotateVertical

Given Information Inferred InformationPlace Time Element Effect AttributeDry Noon Sun Heating Temperature

Light Color

Figure 12. Evening Glow Scene

Table 2. Semantic Representation of the Evening Glow SceneGiven Information Inferred InformationPlace Time Element Effect AttributeTemperate Evening Sun Light Color

BreadthHorizon

Figure 13. Snowfield Scene

Table 3. Semantic Representation of the Snowfield SceneGiven Information Inferred InformationPlace Time Element Effect AttributeFrigid Morning Sun Cooling Temperature

Light BreadthHorizonBreadthVerticalColor

Figure 14. Tornado Scene

7. Conclusion

We discussed a framework for the Multi-Sensory MediaService. And then, we introduced our ontology model for theservice. The proposed ontology model consists of EffectOntology and Device Ontology. The Effect Ontologyrepresents semantic information about multimedia contentsand sensory effects synchronized with the contents. TheDevice Ontology represents semantic information aboutsensory devices generating sensory effects. With our ontologymodel, the noble service can make users feel realistic andentertain users.

With the proposed ontology model, we infers sensory effectinformation from the given multimedia contents information.In order to evaluate the proposed ontology model, wedemonstrated various scenarios. Each scenario is a scene of amultimedia content such as a movie. After we defined thesemantic information about the scenes, we confirmed inferredresults. With scenario-based experiments, we proved theeffectiveness of the proposed ontology model.

Therefore, by describing the simple scene information, acontent provider can define the sensory effects for themulti-sensory media service.

8. Future Work

As a future work, we are planning to consider the context ofa user and environment. The satisfaction of sensory effectsmay be under the influence of user preference and physicalinformation of service environment, for example, the size ofspace, the type of space, and so on. Considering these factors,a personalized multi-sensory media service will be possible.

Also, we will research about inferring alternative sensorydevices for generating similar sensory effects when matchedsensory devices do not exist.

ISBN 978-89-5519-139-4 -1029- Feb. 15-18, 20091CACT 2009

Authorized licensed use limited to: Korea Advanced Institute of Science and Technology. Downloaded on December 9, 2009 at 21:43 from IEEE Xplore. Restrictions apply.

ApPENDIXES

•The Effect Ontology

The Device Ontology

REFERENCES

[1] C. Timmerer. A Summary of the Representation of SensoryEffects(RoSE). 2008.

[2] M. Weiser, "The Computer for the 21st Century," Scientific American265, Nr. 3, S. 94-101,1991

[3] H. Chen, T. Finin, and A. Joshi. An ontology for context-aware pervasivecomputing environments. Knowl. Eng. Rev., vol. 18: 197-207,2003.

[4] X. Wang, D. Zhang, T. Gu, and H. Pungo Ontology based contextmodeling and reasoning using OWL. Pervasive Computing andCommunications Workshops, 2004. Proceedings of the Second IEEEAnnual Conference on: 18-22, 2004.

[5] H. Chen, F. Perich, T. Finin, A. Joshi. SOUPA: standard ontology forubiquitous and pervasive applications. Mobile and Ubiquitous Systems:Networking and Services, 2004. MOBIQUITOUS 2004. The FirstAnnual International Conference on: 258-267,2004.

[6] E. Christopoulou and A. Kameas. GAS Ontology: An ontology forcollaboration among ubiquitous computing devices. International Journalof Human-Computer Studies, vol. 62: 664-685,2005.

[7] D. Heckmann. Ubiquitous User Modeling. PhD Thesis. 2005.[8] D. L. McGuinness, F. Harmelen, "OWL Web Ontology Language

Overview," W3C Recommendation,http://www.w3.org/TR/owl-features/, 2004.

[9] Protege, http://protege.stanford.edu/.[10]A. Gomez-Perez, "A Framework to Verify Knowledge Sharing

Technology," Expert Systems with Application. Vol. 11, N. 4. pp.519-529, 1996.

ISBN 978-89-5519-139-4 -1030- Feb. 15-18,2009 ICACT 2009

Authorized licensed use limited to: Korea Advanced Institute of Science and Technology. Downloaded on December 9, 2009 at 21:43 from IEEE Xplore. Restrictions apply.