ProjectProposal V5 final - ABVincitaabvincita.com/docs/Omnisense_Transpose_Proposal.pdf ·...

TRANSPOSE PROPOSAL DOCUMENT

b y o m n i s e n s e

3 APRIL 2014

omnisense Transpose ii

TABLE OF CONTENTS

INTRODUCTION ..................................................................................................................... 1

BACKGROUND RESEARCH .................................................................................................. 2

MULTIMODAL HUMAN COMPUTER INTERACTION .................................................................. 2

CURRENT MMHCI SYSTEMS APPLICATIONS ......................................................................... 2

INTENDED AUDIENCE AND EXPERIENCE ............................................................................ 6

RELEVANCE TO OVERALL STUDIO THEME ........................................................................... 7

IMPLEMENTATION DETAILS ................................................................................................... 8

DEVELOPMENT PLAN ............................................................................................................ 9

ACADEMIC DEADLINES ..................................................................................................... 9

TASKS BREAKDOWN ........................................................................................................ 10

IMPLEMENTATION CONSTRAINTS ...................................................................................... 11

THEORETICAL LIMITATIONS ............................................................................................... 11

TECHNICAL LIMITATIONS ................................................................................................. 11

PRACTICAL CONCERNS .................................................................................................. 13

TEAM ORGANIZATION & INDIVIDUAL CONTRIBUTION .................................................... 14

TEAM MEMBERS AND ROLES ............................................................................................ 14

STRENGTH AND WEAKNESSES .......................................................................................... 14

CONCLUSION ..................................................................................................................... 16

REFERENCES ........................................................................................................................ 17

Project Proposal Document

omnisense Transpose 1

TRANSPOSE b y o m n i s e n s e

Physical Computing & Interaction Design Studio proposal 4 April 2014

Prepared by

Andrea Betarani Vincita 43067327 Andrea Epifani 43298176 Chew Jian Ming 42968490 Jordan Henderson 42909569

Introduction Omnisense “Transpose” is a Multimodal Human Computer Interaction

Device that brings digital feelings to the real world, providing an enhanced view on playful social interaction. Transpose will be demonstrated by making use of a customised video game platform that will allow us to recreate particular social situations and prove the feasibility of the project and its application in real life.

The product is composed by a set of wearable modules, each of which is connected to a central control system installed on a computer. Each module contains a set of sensors and actuators along with a communication device which allows the exchange of information between the central control system and the players. The control system is interfaced to the game platform through an API that connects digital and real world and handles input and output requests through RF communication with the modules. For demonstration only, the control system will also manipulate specific environmental factors within the space in which the user is playing.



Background Research

M U L T I M O D A L H U M A N C O M P U T E R I N T E R A C T I O N

Multimodal Human Computer Interaction (MMHCI) technologies provide multi-sensory interactions between users and computers. The term refers to the "interaction with the virtual and physical environment through natural modes of communication" (Bourguet, 2003, p. 1). As discussed by Jaimes and Sebe (2007), research has been completed within the field but most of the tested experiences have only been focusing on single senses or treated senses singularly, mainly due to scientific and technical issues. However, for the purpose of this research, considering the combination of different sources of inputs to generate various formats of outputs can lead to interesting results.

Considering a Human Cantered Approach while studying MMHCI systems, gives a different perspective to the problem. It is clear, in fact, that inputs include now all of the five human senses: sight, touch, hearing, smell and sound. These can be sensed through computer devices such as cameras, haptic sensors, microphones and other input devices. As in Jaimes and Sebe (2007), any devices that uses a combination of any of the above factors relies on a multimodal input and output system.

C U R R E N T M M H C I S Y S T E M S A P P L I C A T I O N S

Different approaches to multimodal system design have been taken during the years and multiple research process brought a wide variety results. More specifically, related to the scope of this research project, we will consider three particular projects where inputs and outputs were similar to the ones we will be able provide with our product. The proposed examples are applications of multimodal feedback technology systems into everyday life applications. As we are trying to achieve a level of interaction similar to the ones presented, we will be basing our assumptions regarding the users experience on the relevant research within the field.

The first project, a Multimodal ‘Eyes-Free’ Interaction Techniques for Wearable Devices by Brewster, Lumsden, Bell, Hall, and Tasker (2003), used existing mobile technology integrating it with a multimodal interface. The second prototype, MAUI a Multimodal Affective User Interface by Lisetti and Nasoz (2002), tested the introduction of multimodal feedback for supportive car driving. Finally, the third project, an Interactive Racing Game with Graphic



and Haptic Feedback by S.-Y. Kim and Kim (2007), introduced haptic feedback to a racing game and tested various modalities of user-game interaction.

The prototype developed by Brewster et al. (2003) used technology such as 3D Sound and Gesture Recognition to enhance the experience currently provided by mobile devices by interconnecting them to wearable and multimodal technologies. Their research “builds on this to investigate multi-dimensional auditory and gestural techniques that would enable richer and more complex interactions with devices and services when mobile“ (Brewster et al., 2003, p. 1) by creating a system that uses as little visual attention as possible by keeping the user focused on the world surrounding them. In order to achieve that, Brewster et al. (2003), have prototyped multimodal technologies by developing a 3D audio space where sounds related to different activities were played in different spatial positions. They relied on the Cocktail Party Effect (Arons, MIT Media Lab) and utilised user’s head motion feedback to activate one of the active tasks. Feedbacks collected from this research show how interacting with mobile devices by using eyes and hands free systems, especially for easy tasks, reduces the impact that these devices have on our daily life behaviours (Brewster et al., 2003). Running the same experiment with the support dynamic audio feedback to the gestures, they have registered a considerable improvement of users’ understanding of their actions in relation to the system response.

Lisetti and Nasoz (2002), completed a research project which analysed different ways of communication while driving. Exploring different technologies that the previous project, Lisetti and Nasoz focussed their product outputs and feedbacks on the analysis of users’ emotions along with signals generated by the Autonomic Nervous System, “the part of the nervous system responsible for control of the bodily functions not consciously directed, such as breathing, the heartbeat, and digestive processes” (Oxford Online Dictionaries). The MAUI prototype inputs were acquired through various modalities: Visual, from facial images and videos, Kinaesthetic, from the ANS signals, and Auditory, from users’ speech recognition. In addition to these, the system designed by Lisetti and Nasoz, allowed input from “linguistic tools (L) in the form of linguistic terms for emotion concepts, which describe the subjective experience associated with a particular emotion” (Lisetti & Nasoz, 2002, p. 163). The outputs of the system were mostly given through descriptive feedbacks about the users’ current state and suggestions regarding which next action was possible in order to change state (Lisetti & Nasoz, 2002). In



order to obtain valuable results, Lisetti and Nasoz, have deeply analysed the impact of the computer-generated auditory feedbacks to users’ facial expression and emotional reactions.

Even though the entire process adopted by Lisetti and Nasoz is not relevant to the scope of this research, some of its details might be helpful while developing our multi modal feedback system. In particular, the model represented in Figure 1 (Lisetti & Nasoz, 2002, p. 162) shows how human emotions are stimulated by a set of different events, sensory, cognitive and biological. According to Zajonc and Markus (1984) theory, these inputs are transformed into actual affect representations by three phenomena: the activation of the autonomic nervous system, subject experiences and the generation of the actual expressions (Zajonc & Markus, 1984). As proved by Lisetti and Nasoz, this process can now be applied, thanks to multimodal systems and the application of advanced AI techniques, also to system which are not entirely driven by humans. This is made possible by capturing the users biological inputs (see Figure 2, Affect Perception), analysing them by processing them using different techniques (see Figure 2, Affect Recognition) and predicting the users reaction to a specific feedback (see Figure 2, Affect Prediction). Even though this system includes steps that will not be implemented in our product, it is important to understand how emotions are handled by users and which inputs are crucial in order to guarantee the desired interaction and level of engagement with our product.

Finally, it is relevant to consider how outputs can enhance the user experience. Considering the prototype by S.-C. Kim and Kwon (2007), an interactive racing game which experience have been improved with haptic feedback, might provide some more insights about the expected user interaction to our research project. They have tested the game with two different groups of volunteers: the first one experienced the vibrotactile feedback technology incorporated in the game, the second one did not (S.-C. Kim & Kwon, 2007). Before starting the experiment, “through an experiment, we verified that the haptic feedback in the proposed game is shown to be realistic” (S.-C. Kim & Kwon, 2007, p. 76). What they have found is that, during the first experiment, all of the players “answered that they felt the car’s velocity change when they increased or decreased the velocity of the car [...] even though they did not observe the graphical environment [...]” (S.-C. Kim & Kwon, 2007, p. 76). As we are be implementing haptic feedback that will be activated by users’ actions through the game, we will expect users’ to feel completely engaged and involved with the game environment.



* Figure 1: Human’s affect representation

Figure 2: Affect and HCI Interface (Lisetti & Nasoz, 2002, p. 163)



Intended Audience and Experience The project is aimed towards young adults to middle aged participants,

without any limitation on gender or personality. Our concept involves placing two participants into separate, isolated and controlled environments, which allow us to deliver playful simulations that identify social characteristics occurring in everyday interactions.

The scenario will be provided in the form of a multiplayer game possibly situated within a labyrinth or other constrained environment. At this stage, it will be designed to run for a short period of time (no more than 2-4 minutes) and will provide both participants with the opportunity to interact socially, competing against external forces, and potentially each other. The project will also provide multimodal feedback while interacting with the digital world (i.e. walking into a wall, jumping, collisions etc.).

Initially, the first simulation will require both players to navigate the game, looking for clues or hidden objects (or any other objective we specify). The player will make use of multimodal feedback provided by our modules while attempting to outplay the other participant - the losing player will begin to lose the use of visual feedback taken for granted (via the use of in-game and exterior environmental lighting). The losing player will be placed at a further disadvantage through the use of lighting and sound feedback in order to induce fear (while heightening the participants senses and creating a more memorable impact).

The second simulation will occur immediately after the first; placing players in a semi-cooperative environment (allowing the room for potential betrayal). Both participants will share control of a single character - either participants having an inverse role in navigating the character safely through the provided level before the time limit. One player will have full view of the in-game environment, but with no ability to move the character, while the other participant will have limited visibility - enforcing collaboration between the two participants through multimodal communication. The ‘navigating’ player will provide feedback via the products’ sensors in order to safely (or unsafely) direct the player through potentially hazardous environments. This will allow us to observe and record the paradigm shift between a competitive and collaborative social environment as discussed in the relevance to the overall studio theme.



Relevance to Overall Studio Theme The project is highly relevant to the overall studio theme “designing for

playful and open-ended interactions in everyday life”. As mentioned earlier, the project is designed to introduce two players to significant social experiences that may expose the underlying decisions made when dealing with difficult situations. For instance, the two simulations offered (cooperative and competitive) both focus on how a player would react when given the opportunity to either assist or ignore a potential “stranger” or acquaintance both directly and indirectly.

In the competitive simulation, each player's performance throughout the challenge will indirectly affect the environment of another (the results of which will be visualised extensively for viewers to witness). The simulation highlights how people behave in situations when directly faced with a problem introduced by a third party force beyond the player’s control. As both players will indirectly influence the experience introduced to the other player, the aspect of social and moral obligations will become less apparent when placed in an uncomfortable situation, which is the expected outcome of such a simulation.

The second collaborative simulation is designed to inspire and encourage teamwork or betrayal between two participants, when either participant must rely on the directions of (as mentioned earlier) a (potentially) complete stranger or acquaintance in a more direct manner. Players are given the opportunity to either support or hinder one another through means of multimodal feedback communication, bypassing the social constructs present in everyday verbal speech. Players will essentially ‘place their lives in the hands’ of another that they may or may not trust, highlighting how we as a society can enable (or betray) trust between fellow human beings.

Evidently, both scenarios inspire and explore playful interactions within potentially everyday life, as situations where a complete stranger may have control over the safety and well-being of an average commuter is something we, as a society, have become familiar with. The overall outcome (as mentioned earlier) will hopefully allow participants to fully realise the level of trust they assume and take for granted in modern society during everyday interactions.



Implementation details As mentioned above, Transpose will be implemented a set of wearable

bands (modules), as shown by Figure 3. Each component is designed to suit a complete sensing and feedback generation circuit aimed at enhancing the experience of the digital application it is connected to. Additional modules such as the head set and the floor pads represented in Figure 3 are designed to guarantee an even more complete experience, especially in gaming.

Figure 3: Transpose bands and components details

The ideal setup of a wearable band includes the following component, as shown in Figure 4:

• Four or more vibration motors, used to generate feedback.

• One humidity and/or temperature sensor, used to capture the player’s emotional status.



• One heart beat sensor (not present in Figure 4).

• One Gyroscope and Accelerometer module, used to detect the position of the arm or leg wearing the bend.

• One transmitter and receiver module, used to communicate with the central base.

Figure 4: Bend circuit implementation schema

Development Plan

A C A D E M I C D E A D L I N E S

Project start date March 18th, 2014

First semi-functional prototype demonstration May 6th, 2014

Project end date and exhibition June 3rd, 2014

Final documentation submission June 17th, 2014



T A S K S B R E A K D O W N



Implementation Constraints We have considered a variety of implementation constraints we might

incur in while working on the development of the project. They are divided in three broad categories, Theoretical limitations, Technical limitations and Practical Concerns.

T H E O R E T I C A L L I M I T A T I O N S

G A M E C O N C E P T L I M I T A T I O N S

Our decision of implementing the concept of multimodal feedback in the form of a game may give some limitations on the potentials of multimodal feedback in daily life. Users will only receive feedback and give input to the system when they are playing the designated game. As a proposed approach for this limitation, the technology and modules used in this project can be explored further to be utilized in a more general setting of daily life.

A U D I E N C E L I M I T A T I O N S

The game form of this project may not appeal to everyone. People that don’t like to play games may not enjoy this project or may not find it useful for them. The game is also designed so that the user needs to move around with their legs, so it may not be suitable for motion impaired people. Additionally, the initial concept of this project is to encourage user to utilize their other senses such as hearing, touch, temperature, etc. instead of relying on vision alone. Therefore, this project may appeal to vision impaired people but may not be suitable for hearing impaired people. As a proposed solution for this limitation, it is possible to build a customized-reverse mode where the player will utilize their vision instead of hearing to tailor the project for hearing impaired people.

T E C H N I C A L L I M I T A T I O N S

T E M P E R A T U R E C O N T R O L L I N G M O D U L E S

Modules and machine that are able to control the temperature of a room such as air conditioner has the limitation of not being quick enough in adjusting the room’s temperature. Therefore its number of use during the game may be limited to just once or twice.



W I N D / A I R C O N T R O L L I N G M O D U L E S

As described previously, we intend to use fan to give a wind effect to the user while they are playing the game. There are limitations in the use of fan: big fan may take too much time to generate the desired wind, while a small, portable fan may need to be located very close to the user so that the user can feel the wind effect. There is also a placement constraint: a fan can only generate wind from one direction. In order to create wind effect from 4 directions around the user (north, east, south, west), 4 fans need to be located around the user. As of now, the proposed mitigation is to use small, portable fans inside of a closed room so that its wind will be strong enough to be felt by the user.

S O U N D A N D V O I C E O U T P U T M O D U L E S

The use of speakers to give sound and voice output may be impractical during the exhibition as there will be many other exhibits which will make the environment noisy. Additionally, as the intended game is a multiplayer game, the sound intended for a particular player may disturb the experience of the other player. Therefore, the most viable solution is to use headphones instead of speaker. On another note, it may also be difficult to represent a sound that is coming from the front or back of the player (sound from the left and right can be easily produced via the stereo headphone). A sound feedback from the front or back of the player may be supported by other kinds of feedback to ensure that the user receive the intended information from the feedback.

I N P U T M O D U L E S

As of the time this document is written, the kinds of input modules that we plan to use in this project are limited to directional pads for movement, heart rate detection, and an input to communicate with the other player. The player will communicate using one of the feedback modules other than sound or voice modules, but as of this time we haven’t decided on the final form of communication yet.

W E A R A B L E M O D U L E S

Some of the output and input devices are designed to be wearable by the user, such as armbands, leg bands, headbands, etc. The limitations of the proposed wearable modules are they may be quite a hassle for the user to put them all properly before they can play the game. Additionally, the wearable modules may be limited by the sizes of microcontrollers and



other technology available in the market right now, so the wearable modules may not be compact enough.

P R A C T I C A L C O N C E R N S

B U D G E T

This project is partially funded by the university; the members of the team will pay any uncovered expenses. This will definitely limit the variety and quantity of technology that we can use in the project. As a proposed mitigation strategy, we will try to use objects that are commonly available in everyday life as much as possible, such as cloth, garbage bags, etc.

T I M E

The project is limited to the duration of the course (13 week in total) with a deadline of the exhibition day. Given the short time of the project, some of the proposed aspects of the project may not be possible to implement. Our mitigation strategy for the time constraint is to work according to the project plan and make sure that each member is contributing his or her share of work on time.

S A F E T Y

Safety concerns are mainly in the form of the output feedback given to the user. At first we wanted to use electronic pulse as one of the feedback form, but as it can be hazardous to the user, we decided not to implement it. Additionally, as the user will be wearing a lot of wearable modules, there can be several minor risks when the modules or equipment are malfunctioning. We will try to reduce this risk as much as possible by ensuring that there are no exposed electrical components on those modules.

P O R T A B I L I T Y

Portability is considered as one of the constraints as we will need to transport the components of the project from UQ to the exhibition venue at The Edge, Southbank. There will be a lot of modules and components of this project, so there may be some hassle in transporting all the components. But since most of the components will be quite small in size, we assume that our whole project is still feasible to be transported into the exhibition venue.



R E U S A B I L I T Y

As the modules and components will be used extensively during the exhibition, some of the elements may break or get damaged. We will try to mitigate this risk by testing that all the components will be durable enough for extensive use. We will also remind the user to not be too rough with the equipment. We aim to create all the components to still be usable for future uses after the exhibition.

Team Organization & Individual Contribution

T E A M M E M B E R S A N D R O L E S

Name Role Email

Andrea Betarani Vincita Programmer [email protected]

Andrea Epifani Design, Hardware [email protected]

Chew Jian Ming Design, Hardware [email protected]

Jordan Henderson Programmer [email protected]

S T R E N G T H A N D W E A K N E S S E S

A N D R E A B E T A R A N I V I N C I T A

My main strength is programming. I have done a lot of programming in many forms these past 3.5 years, such as Java applications, websites, database, .NET, and mobile applications. I have also done a bit of physical computing with MakeyMakey and camera detection using ReacTIVision, so I hope I can help in building the physical elements of our project as well. My weaknesses are the fact that I haven’t done much of game programming. My only experience in game programming is creating a simple game using ActionScript 3.0 and Flash. Therefore, I still need to learn a lot about game programming, especially understanding how game engines work. Additionally, I’m also not good at graphic design, especially designing 3D models. That would be my other weakness. But despite my weaknesses, I’m very eager to learn new things. Therefore, I’m really looking forward to learn many things during this project.



A N D R E A E P I F A N I

My strengths are interface and products design as well as hardware oriented coding and programming. I have experience in C and C++ programming for hardware and microcontrollers as well as C#, SQL, Cocoa, Java, ActionScript 3.0 and many other programming languages for web oriented coding and mobile development on both iOS and Android. My weaknesses are gaming and knowledge of video games as well as and 3D design and modelling.

C H E W J I A N M I N G

My main strength is design and database. I have done web and graphic design using adobe Photoshop and illustrator for the past 2 years; and I did database for past 3 years. I also have done a bit of physical computing with camera detection using ReacTIVison during DECO2300. My weaknesses mostly related to the fact that I have not done much programming and game programming. I am looking forward that I could pick them up during this course.

J O R D A N H E N D E R S O N

My main strength is programming - I have a strong background in C/C++, C#, SQL, and many other various languages. I have experience working with embedded systems, and an understanding of game engines/various game concepts. I don’t have as much experience working with physical hardware, but have a good understanding of the concepts involved. I have experience working with a dynamic range of applications, languages, and technologies, and believe this knowledge will be useful throughout the semester.



Conclusion While working on the development of Transpose, we hope to achieve

the best possible results by taking into account some of the theoretical findings illustrated in this document as well as hardware and software oriented research that we will complete at different stages of our project plan.

We will cooperate, as a team, and apply our knowledge at best in order to develop and exhibit a solution that can prove how the technology we are studying is important in our everyday lives and how it can be applied to solve simple problems. Demonstrating the potential of such technology though a videogame will give us the opportunity to break the boundaries between real and digital world and illustrate the concept in a more engaging way to the general public.

Where possible, user research will be performed during the testing sessions and during the exhibit, through methodologies that will be described in a later document, and results within the research field this project fits in will be collected and analysed as complimentary part to the formal university assignment.



References Arons, B. A Review of The Cocktail Party Effect. MIT Media Lab.

Bourguet, M.-L. (2003). Designing and Prototyping Multimodal Commands. Paper presented at the INTERACT.

Brewster, S., Lumsden, J., Bell, M., Hall, M., & Tasker, S. (2003). Multimodal 'eyes-free' interaction techniques for wearable devices. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Ft. Lauderdale, Florida, USA.

Dictionaries, O. O. (Ed.).

Jaimes, A., & Sebe, N. (2007). Multimodal human–computer interaction: A survey. Computer vision and image understanding, 108(1), 116-134.

Kim, S.-C., & Kwon, D.-S. (2007). Haptic and Sound Grid for Enhanced Positioning in a 3-D Virtual Environment. In I. Oakley & S. Brewster (Eds.), Haptic and Audio Interaction Design (Vol. 4813, pp. 98-109): Springer Berlin Heidelberg.

Kim, S.-Y., & Kim, K.-Y. (2007). Interactive Racing Game with Graphic and Haptic Feedback Haptic and Audio Interaction Design (Vol. 4813, pp. 69-77): Springer Berlin Heidelberg.

Lisetti, C. L., & Nasoz, F. (2002). MAUI: a multimodal affective user interface. Paper presented at the Proceedings of the tenth ACM international conference on Multimedia, Juan-les-Pins, France.

Zajonc, R., & Markus, H. (1984). 3 Affect and cognition: the hard interface. Emotions, cognition, and behavior, 73.

ProjectProposal V5 final - ABVincitaabvincita.com/docs/Omnisense_Transpose_Proposal.pdf ·...

Documents

Transcript of ProjectProposal V5 final - ABVincitaabvincita.com/docs/Omnisense_Transpose_Proposal.pdf ·...