1

Development of Shooting Game In An Augmented Reality Environment Using Head-Mounted Display

Muhammad Ismail Mat Isham and Farhan Mohamed

1

Faculty of Computing, Universiti Teknologi Malaysia (UTM), 81310 Johor Bharu, Johor, Malaysia

{m.ismail.m.isham@gmail.com, farhan@utm.my}

Abstract. Nowadays there are varieties of new ways for consumers to experience Augmented Reality (AR) such as hologram, glasses and other advanced see-through Head-mounted Display (HMD). AR is a technology that enables the virtual object to be displayed onto the real world. The remaining issues in AR are to ensure this alignment registered accurately and possible to retrieve the depth data that can be recognized by the gyrometer sensor. The display devices are expensive and far reach from general consumers. The current technologies also make human computer interaction difficult and fewer existing AR mobile-based application capable to integrate with HMD. Therefore, this project aims to develop an AR shooting game that significantly integrates with HMD. In order to actualize the aim, this project involves five phases. The first is preliminary investigation and data collection for shooting games in AR using HMD casing. The second phase is to design the well-suite technique in AR shooting game. In the third phase is to develop the AR shooting game suitable for HMD. Then, continues with the phase to integrate the AR shooting game with HMD. Finally the evaluation is carried out to test the usability usage and user acceptance testing. Based on the result, this project achieves its aim to produce a shooting game with AR environment using HMD.

Keywords: Augmented Reality, sensor based tracking technique, Head-mounted Display, 3D augmented environment.

1 Introduction

In 1995, Rekimoto previous research on the Optical See-Through head-mounted display (HMD) had stated the problems with the technology issues that involve the correct relationship between real and virtual positions.[10] Kato and Billinghurst through their research on the marker tracking and calibration had pointed out some issues related to calibration in AR with HMD.[5] Zhou et al. developing an AR-mobile based application is not an easy task because need to consider graphics rendering hardware and software, suitable tracking techniques, tools for registration and display hardware.[12]

There are certain weakness in an AR application such as occlusion handling. HMD and video see-through AR technology is not widely-spread outside the research community today.[8] Olson et al (2011) previous development, they used two mobile phone to integrate with handmade HMD. This cause the HMD heavier for user to use.[9] Furthermore, the current HMD are quite expensive.

UTM Computing ProceedingsInnovations in Computing Technology and Applications

Volume 2 | Year: 2017 | ISBN: 978-967-0194-95-0

2

Objective of this project is to study the implementation of AR game with HMD, to develop the AR game that can be integrate with HMD, and to test and validate the AR game on HMD. The scope of this project is the development of this application is until the testing phase only. The game also for smartphone with android OS and have build-in gyrometer sensor. The game object for this application is free source from Unity asset store. The significance of this project is, first increase the interactivity and communication between human, virtual object and real environment, in the way that the human can view the virtual objects in the real world environment using HMD. Second is to initiate a new revolution in the world of action shooting game, after the using of AR and head-mounted display in this game. Thirdly, the significance of this project is also to open the mind of Malaysian, especially for the new generation, regarding the advancement of technology, so that they will not be left behind the current of modernization as well as to improve our country competitiveness in technology in the international stage.

2 Literature Review

This section will discuss about the analysis and literature review being done related to the project. 2.1 A Study on Augmented Reality

The history of AR began in year 1955, a man named Morton Heilig began to develop a machine named Sensorama combines real environments and Virtual Reality (VR).[3] In 1968, computer scientist, Ivan Sutherland credited for inventing The Sword of Damocles - the first HMD which combines the VR and AR with HMD in Massachusetts Institute of Technology. Sutherland’s HMD displayed output from a computer system in stereoscopic display.[11] In 1975, Myron Krueger develope the first “virtual reality” interface in the form of “Video place” which allowed its users to manipulate and interact with virtual objects and to do so in real-time.[6] P Caudell of Boeing coins AR in 1990 (Burdea, 1994). In 1994, Milgram and Kishino elaborated the virtuality continuum of real environment to virtual reality in which AR is one part of Mixed Reality.[7] In 1997, Azuma defined AR as “an environment that includes both virtual reality and real-world elements. For instance, an AR user might wear translucent goggles; through these, he could see the real world, as well as computer generated images projected on top of that world”.[1] 2.2 Display Technology Head Mounted Displays (HMD) have been used for augmented reality applications since the 1960s and have been the mainstay of many augmented reality systems to date.[11] There are two type of HMD, Optical See-Through HMD and Video See-Through HMD [f]. This device worn on the head or as part of a helmet, which has a small display optic in front of one or each eye. There is HMD with mobile display use external device like smartphone. It possesses all the necessary computer hardware required to function as a VR headset, allowing it to collect and display the input it receives from the mobile.

3

2.3 AR Tracking Technique Sensor-based tracking is one the AR tracking technique. This techniques are based on sensors such as magnetic, acoustic, inertial, optical and/or mechanical sensors. Magnetic sensors have a high update rate and are light, but any nearby metallic substance that disturbs the magnetic field can distort them. More recently, in 2004, Newman et al extend this concept to larger networks of sensors, which support ubiquitous tracking and dynamic data fusion. Depth sensing is one of the input for Sensor-Based Tracking Techniques. Depth sensing is becoming a core component in HMD display technology.[2] 2.4 First-person Shooter genre First-person shooter (FPS) is a video game genre centre on gun and projectile weapon-based combat through a first-person perspective, that is, the player experiences the action through the eyes of the protagonist. The first-person shooter shares common traits with other shooter games, which in turn fall under the heading action game.[4] From the genre's inception, advanced 3D graphics have challenged hardware development, and multiplayer gaming has been integral.

Figure 1. FPS player view concept

3.0 Project Methodolgy

In order to construct AR game with HMD, AR environment with high real-time performance for virtual world need to be mixed with our real world, several phases of project development are planned which are:

Phase 1. Preliminary investigation and data collection for shooting games in AR using HMD. In this phase, development focused on exploring prior approaches of AR for shooting game including their technique to achieve the seamlessness and enhancing reality in AR environment for shooting game using HMD and design process. Moreover, AR is merging real and virtual world can be applied in shooting game with HMD to add precise information and improve the user’s view to a real environment of shooting game.

4

Phase 2. Design the well-suite technique in AR shooting game. Based on the information explored and gathered in the previous phase, a design phase of AR shooting game has been discovered. The main challenge in AR shooting games is to ensure the device location (use sensor-based tracking) is accurate for displaying the game object and the type of display on the phone to integrate with the HMD. The mobile device can integrate with the lower cost HMD, which allow all users to experience this game project. Phase 3. Develop the AR shooting game suitable for HMD. After the design phase is completed, the frameworks of AR shooting game for representing the integration between AR shooting game and HMD is setting up. The prototype applied AR sensor-based tracking and FPS genre game type approach is being develop in this phase. Phase 4. Integrate AR shooting game with HMD. After the development phase is completed, then the game prototype will integrate with HMD. A First-person-shooter (FPS) type shooting game in AR will developed to employ it with HMD in order to test it in testing phase. Phase 5. Testing the prototype. This phase will test the interaction technique between AR shooting game with HMD. This will include comparison with other techniques and packages as well as making additional improvement if needed.

4.0 Implementation

The name of this game is AR shooting game. This game application have three game mode, Meteor Crush, Bug Smash and Balloon Pop. The activities in this chapter are crucial stage to implement and test the game in order to actualize the project aim and the methodology as describe. The activities are the implementation of Augmented Reality (AR) shooting game using Head-mounted Display (HMD). 4.1 Configuration HMD with Mobile Phone Figure 2. demonstrates how to install the mobile phone into HMD device. This HMD called Cardboard. Users need to assemble the HMD by themselves following the instructions given on the manual. Then the mobile device is place at the mobile device port. The picture one the right show different type of HMD. The HMD have the mobile device port where the camera did not covered by any constraint.

5

Figure 2. Configuration Cardboard HMD and Compatible HMD with mobile device port

demonstration

4.2 Sensor-based Tracking

Sensor-based tracking searches the device location as the input. After the inputs are detected, the device location identified based on the depth sensor tracking. Then it tracked the video and 3D or 2D object use mobile device. Next, the system calculated the position and orientation of the mobile device. In Vuforia algorithm, viewpoint calculation process is required to obtain the transformation matrix. The perspective viewpoints for transformation matrix transformed the camera viewpoints as shown in Figure 3.

Figure. 3. Sensor-based framework with Vuforia framework

This perspective transformation matrix can be found from the default initialization of the calibration process. The calibration process is necessary to define the matrix of the world coordinate for camera calibration. Lastly, the virtual object displayed on the screen.

6

Figure. 5. The gameplay view after integrate with HMD

4.3 Concept and Gameplay

This game concept is same like shooting game in outer space where there is various dangerous floating obstacle with direction towards the player. The object does not have a gravity. The player need to survive from the incoming dangerous obstacle object by destroying them with the weapon provided. Player needs to shoot the object before it hit them. However, in this prototype, the player was in the real world. The obstacle came out randomly from the 3D dimension. In other to survive player needs to destroy the incoming obstacle. This game is same as first-person shooter genre, which is centred on gun and projectile weapon-based combat through a first-person perspective, the player experiences the action through the eyes of the user. The background of the game depends on the location where the player played at.

Figure 4. Gameplay view on the mobile phone screen

Figrue. 6. Gyrometer system movement concept.

7

The camera movement use the gyro meter system where user need to move their head to move the pointer cursor to change the target. If player failed to shoot the obstacle and the object hit the player, their health point (HP) will be decreasing. If the HP reached zero, the game will be terminated because the player is considered dead. The camera captures and recognizes the relationships between the coordinates. These relationships used to generate the world coordinates for the shooting mode.

5.0 Result

The result of this project is an AR shooting game using HMD. This game application created for standalone mobile device. This game have two part of scene, main title menu and game scene. Game scene have three kind of game. First is Meteor Crush, which it will bring meteor into the game scene. Second game is Balloon Pop. This will bring various colour of balloon in game scene. Lastly, Bug Smash. This game is not available, only for preview. First game for this AR shooting games is Meteor Crush. In this game part, player need to shoot the incoming meteor toward them. Player need to survive the game in the time given. Figure 7. show the Meteor Crush game scene screenshot. Each meteor have different health point, and different damage. For example, the bigger the meteor, the larger its health point and damage. Player need to be careful and alert with the incoming meteor. Figure 7 shows the Balloon Pop game scene example. Player can shoot the balloon before the balloon touch them. This game created with 2D animation in 3D space. This game is quite easy for player to play because each balloon have same health point and damage. Probability for player to die is low.

Each game scene have one menu button at the top of the game world. This menu can pause the game. When the game is paused, three more menu will popped out. This three menu consist of resume, home and restart the game. Figure 7. show the menu in the game. There is two ending scene for this game, first is when the game timer reached zero. Second is when the player health bar is empty.

Figure 7. Game scene screenshot

8

6 Conclusion

The AR shooting game using HMD is one of the starting point to start revolutionaries in shooting game genre. During the development of this project, the objectives mentioned are achieved in order to deliver a high quality game application. The first objective of this project is study the implementation of AR game on head-mounted display. The objective was achieved where the information regarding AR and HMD were studied from the past research paper, journal, etc. The second objective is to design and develop a prototype that can shows the integration between AR shooting game and HMD was achieved in when the project methodology was designed. The methodology plays an important role as a development guideline in order to sort the process of implementation and development toward the project aim and achieve the requirements. The third objective is to test the AR game on HMD. The implementation stage performed based on the project methodology, where the AR sensor-based tracking technology and HMD technology into the AR shooting game as described, the performance, efficiency as well as interactivity were tested and evaluated. The test results collected and analysed using black box testing as well as user acceptance testing.

References

1. Azuma, R., Behringer, R., Feiner, S., Julier, S., & Macintyre, B. (2001). Recent Advances in

Augmented Reality. IEEE Computer Graphics and Applications, 21(November), 1–15. 2. Fuchs, H., & Ackerman, J. (1999). Displays for augmented reality: Historical remarks and future

prospects. Mixed Reality Merging Real and Virtual Worlds, Ohta Y and Tamura H, Ohmsha Ltd, 31-40.

3. Heilig, M. (1998, January). Beginnings: Sensorama and the telesphere mask. In Digital illusion (pp. 343-351). ACM Press/Addison-Wesley Publishing Co..

4. Jansz, J., & Tanis, M. (2007). Appeal of playing online first person shooter games. CyberPsychology & Behavior, 10(1), 133-136.

5. Kato, H. and Billinghurst, M. (1999). Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In Augmented Reality 1999. (IWAR'99) Proceedings. 2nd IEE and ACM International Workshop on. IEEE, 85-94

6. Krueger, M. W. (1991). Artificial reality II. 7. Milgram, Paul., Takemura, H., Utsumi, A., & Kishino, F. (1994). Augmented Reality: A Class of

Displays on the Reality Virtuality Continuum. SPIE Proceedings volume 2351: Telemanipulator and Teleprescence Technologies, 282-292.

8. Mohring, M., Lessig, C., & Bimber, O. (2004, November). Video see-through AR on consumer cell-phones. In Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality (pp. 252-253). IEEE Computer Society.

9. Olson, J. L., Krum, D. M., Suma, E. A., & Bolas, M. (2011, March). A design for a smartphone-based head mounted display. In Virtual Reality Conference (VR), 2011 IEEE (pp. 233-234). IEEE.

10. Rekimoto, J., & Nagao, K. (1995). The World Through the Computer: Computer Augmented Interaction with Real World Environments. Proceedings of UIST '95 (Pittsburgh, PA, 14-17 November 1995), 29-36

11. Sutherland, I. E. (1965). The ultimate display. Multimedia: From Wagner to virtual reality. 12. Zhou, F., Duh, H. B. L., & Billinghurst, M. (2008, September). Trends in augmented reality

tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (pp. 193-202). IEEE Computer Society.