Hyakunin-Eyesshu: a Tabletop Hyakunin-Isshu Game with Computer Opponent by the Action Prediction

Based on Gaze Detection Michiya Yamamoto Kwansei Gakuin Univ.

2-1 Gakuen, Sanda, Hyogo

669-1337, Japan +81-79-565-7121

michiya.yamamoto@ kwansei.ac.jp

Munehiro Komeda Kwansei Gakuin Univ.

2-1 Gakuen, Sanda, Hyogo

669-1337, Japan +81-79-565-7121

aqg77000@kwansei. ac.jp

Takashi Nagamatsu Kobe Univ.

5-1-1 Fukaeminami, Higashinada, Kobe 658-0022, Japan +81-78-431-6297

nagamatu@kobe-u.ac.jp

Tomio Watanabe Okayama Pref. Univ.

111 Kuboki, Soja, Okayama

719-1197, Japan +81-866-94-2105 watanabe@cse.

oka-pu.ac.jp

ABSTRACT A tabletop interface can enable interactions between images and real objects using various sensors; therefore, it can be used to create many works in the media arts field. By focusing on gaze-and-touch interaction, we proposed the concept of an eye-tracking tabletop interface (ETTI) as a new type of interaction interface for the creation of media artworks. In this study, we developed “Hyakunin-Eyesshu,” a prototype for ETTI content that enables users to play the traditional Japanese card game “Hyakunin-Isshu” with a computer character. In addition, we demonstrated this system at an academic meeting and obtained user feedback. We expect that our work will lead to advancements in interfaces for various interactions and to various new media artworks with precise gaze estimation.

Categories and Subject Descriptors H5.2 [Information interfaces and presentation]: User Interfaces. - Graphical user interfaces.

General Terms Design, Measurement, Human Factors

Keywords Eye tracking, tabletop, media arts, human interaction

1. INTRODUCTION Human beings obtain a considerable amount of information through their eyes. Thus, various artworks have been created to appeal to our sense of sight [1,2]. At the same time, it is necessary to increase the area of interaction of a tabletop interface [3,4]. A game that uses eye tracking has already been developed by Li [5]. Moreover, the concept of a gaze-aware tabletop interface was proposed by Holman [6]. We have already developed an eye-tracking pen display and tabletop interface (ETTI) that enables gaze-and-touch interaction, thus representing a novel interaction

interface for media artworks [7,8]. In this study, we developed Hyakunin-Eyesshu—a tabletop Hyakunin-Isshu game that can control the level through action prediction based on gaze detection using the ETTI. In addition, we evaluated the ETTI by demonstrating Hyakunin-Eyesshu at an academic meeting.

2. ETTI 2.1 Estimation of the optical axis of the eye In our previous study, we developed an eye-tracking pen display and realized eye tracking across a measurement range of approximately 60°. The ETTI was developed by introducing a new aspherical model of the cornea for more precise eye tracking on a tabletop interface [9]. Here, we estimated the optical axis by using two cameras and two IR LEDs. First, we calibrated the external parameters of the cameras to acquire the relative 3D positions of the cameras and displays. Next, based on the image processing results, we detected the position of the pupil and two bright points as a Purkinje image (reflections of the LEDs on the cornea). We assumed that the light source and camera center were at the same position. Then, we obtained a plane that contained vectors A and B , as shown in Figure 1, by using the expression

where X is a point on the plane. One camera and one LED were used for determining a plane that contained the optical axis. Therefore, the optical axis could be obtained as the intersection of two planes by using the two cameras and two LEDs. Then, the user gazed at a point on the tabletop interface for calibration. The difference between the optical axis and the visual

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NGCA '11, May 26-27 2011, Karlskrona, Sweden Copyright 2011 ACM 978-1-4503-0680-5/11/05…$10.00.

Figure 1. Estimation of the Optical Axes.

）（ jjjj CP'B'C )(

0)( jCX ,

axis of the eye was revised by carrying out this calibration. The cross point of the visual axis and tabletop was estimated as the gaze point.

2.2 System configuration Figure 2 shows the system configuration of the ETTI for “Hyakunin-Eyesshu.” The image on the tabletop interface was generated by the PC and projected using an LCD TV (SHARP, LC-52RX5) and short-focus projector (NEC, WT615). Two cameras and two IR LEDs were arranged for the aspherical model. We used two digital monochrome cameras (Point Gray, FFMV-03MTM) equipped with 35-mm lenses and IR filters. An IR LED was attached to each camera. The image was processed using an eye-tracking PC (HP, xw4600 Workstation). The gaze point was estimated and sent to two application PCs (HP, EliteBook 8730w). We used a magnetic sensor (POLHMUS, Fastrak System) to detect the user’s hand motion. The software was developed using OpenCV 1.0 and Microsoft DirectX 9.0. It uses UDP communication for data communication among three PCs.

2.3 Gaze detection accuracy We performed two experiments to analyze the accuracy of the ETTI. First, we selected a task by assuming that a user pushed a button on a table. We prepared a 100 × 100 pixel square on the tabletop and asked a user to gaze at the squares one by one for 5 s. Figure 3 shows an example of the results. With the exception of the screen edge, the measurement results for the square showed an accuracy of almost 100%. The screen edge results had an accuracy of more than 90%.

Next, we measured the traceability by moving the target on the tabletop. Here, four users were asked to follow the cursor, which was moved at a speed of 28 pixels per second along a spiral orbit on the screen. Figure 4 shows an example of the results. The error

between the cursor and the gaze point was 38.7 pixels (16 mm, 1.5°), on an average, in the case of the subject with the best accuracy and 53.8 pixels (22 mm, 2.1°) in the case of a subject with the worst accuracy. There was a large deviation from the cursor position; however, this deviation was attributed to the gap caused by blinking.

3. HYAKUNIN-EYESSHU 3.1 Hyakunin-Isshu “Hyakunin-Isshu” is a traditional anthology style of compiling Japanese waka poetry. It also refers to a very popular card game in Japan (upper part of Figure 5), which uses a deck composed of poems (lower-left part of Figure 5) from one such anthology [10]. In this card game, cards with lines from poems are used (lower right) and two players compete to find cards matching a poem being read aloud. Because the player who gets the most cards wins the game, it is necessary to find and snatch cards very quickly.

3.2 Concept “Hyakunin-Eyesshu” is a tabletop Hyakunin-Isshu game that makes it possible to play with a computer character (Figure 6). The ETTI can determine the moment that the user finds the correct card by analyzing the user’s gaze point, because human beings look proactively during such situations [11]. The quick snatching motion is detected before the card is touched by using the Fastrak. The computer character, i.e., the cat, might snatch away the card at the user’s gaze point just as the user is about to reach for it, thus making the game challenging. Furthermore, the

Figure 4. Results of traceability evaluation.

Figure 3. Results of selection evaluation.

Figure 2. System configuration.

Figure 5. Hyakunin-Isshu.

game’s level of difficulty can be adjusted by analyzing the user’s gaze-and-touch timing. Because the computer controls all of the positions for the cards and the correct answer, the previous computer version of the game was not very exciting or enjoyable. We were able to develop a real-time Hyakunin-Isshu game by simply introducing gaze-and-touch interaction. Because the size of each card was approximately 74 mm × 53 mm, a precise and easily calibrated ETTI was necessary for the game.

3.3 Game outline We developed three opponent characters, as shown in Figure 7. One of the opponent characters is displayed on the screen opposite the user, and the cards appear on the tabletop. The first player to take three cards wins.

The blue ninja cat was designed for beginners (top part of Figure 7). This cat provides hints on the position of a card if a player takes too long to find it. The yellow ninja cat was designed for middle-grade players (middle part of Figure 7). This cat analyzes the gaze and motion of the player and takes the correct card at an appropriate time. The player can thus enjoy a challenging game.

The boss cat is designed for experts (bottom part of Figure 7). This cat snatches the correct card with its tail just as the player finds it and reaches for it. In this manner, using the ETTI, a variety of computer characters was realized for Hyakunin-Eyesshu.

4. DEMONSTRATION EXPERIMENT Hyakunin-Eyesshu was publicly demonstrated on October 23–24, 2010, at Entertainment Computing 2010 (EC2010), held at the Kyoto Institute of Technology (Figure 8). At this demonstration, we first explained the system to visitors and calibrated it. Next, we invited visitors to play the game. Finally, we asked them to fill out a questionnaire form with four items rated on a five-grade bipolar rating system: “I enjoyed the game,” “I want to play again,” “I became interested in Hyakunin-Isshu,” and “The eye tracking was precise.” There was also space for them to provide their comments.

There were approximately 200 paid participants and 150 general participants at this event, from which 73 played our system. Unfortunately, gaze detection failed with 13 participants because of diffused reflections from their glasses or the influence of mascara. However, completed forms were obtained from the remaining 60 participants. Figure 9 shows the results of their five-grade evaluations. In response to “I enjoyed the game” and “I want to play again,” 90% of the participants selected four or five. Almost 90% of the participants selected three or higher in

Figure 8. Hyakunin-Eyesshu in EC2010.

Figure 6. Concept of Hyakunin-Eyesshu.

Figure 7. Computer characters.

Boss cat（Expert）

Blue ninja cat（Beginner）

Yellow ninja cat（Middle-grade player）

Boss cat takes the card just before the user’s hand reaches it

The user looks at the correct card

One of the cards is read aloud

Yellow ninja cat starts taking the card

YESNO

The user looks at the correct card

One of the cards is read aloud

The user moves hand

One of the cards is read aloud

The user can take the card

Blue ninja cat gives hints about the correct card position

YES YES

NO

The user looks at the correct card

The user can’tfind the cardfor a while NO

Figure 9. Results of questionnaire.

(1) I enjoyed the game

５62％(37)

427％(16)

33％(2)

25％(3)

13％(2)

(2) I want to play again

12％(1)

５56％(34)

432％(19)

310％(6)

20％(0)

(3) I became interested in Hyakunin-Isshu

17％(4)

５30％(18)3

32％(19) 425％(15)

26％(4)

518％(11)

4

38％(23)

3

22％(13)

210％(6)

12％(1)

(4) The eye tracking was precise

response to “I became interested in Hyakunin-Isshu.” Moreover, 56% selected four or five in response to “The eye-tracking was precise.”

Finally, we asked the participants who played against the blue ninja cat, yellow ninja cat, and boss cat for their responses to “Hints were comprehensible,” “I played an enthusiastic game,” and “The motion of the boss cat was quick enough,” respectively. Figure 10 shows the results.

We also divided these results according to whether each subject rated the eye-tracking precision as four or five and then applied Friedman’s test. Figure 11 shows the results. We found that the more precisely the gaze was detected, the higher the game was rated.

The users provided many positive comments such as “the boss cat was challenging” and “it was exciting that the cat snatched the card at the last moment.” However, some users commented that “the accuracy was insufficient,” and some players tried to win the game using their peripheral vision.

5. CONCLUSION In this study, we proposed the concept of “Hyakunin-Eyesshu,” the first Hyakunin-Isshu game with computer opponent characters. We developed a prototype of the system and confirmed its effectiveness through a demonstration. Our future objectives are

to realize more precise and robust eye tracking and to improve the opponent characters.

6. ACKNOWLEDGEMENTS This work under our project “Embodied Communication Interface for Mind Connection” has been supported by “New IT Infrastructure for the Information-explosion Era” of MEXT Grant-in-Aid for Scientific Research on Priority Areas. Also, our project "Generation and Control Technology of Human-entrained Embodied Media" has been sup-ported by CREST (Core Research for Evolution Science and Technology) of JST (Japan Science and Technology Agency).

7. REFERENCES [1] Geller, T. 2006. Interactive Tabletop Exhibits in Museums and Galleries. In IEEE Computer Graphics and Applications, 26, 5, 6-11.

[2] Fujiki, J. 2007. OLE coordinate system. In ACM SIGGRAPH 2007 Art Gallery, 197.

[3] Schick, A., van de Camp, F., Ijsselmuiden, J., and Stiefelhagen, R. 2009. Extending touch: towards interaction with large-scale surfaces. In Proceedings of the ACM international Conference on interactive Tabletops and Surfaces, 117-124.

[4] Yoo, B., Han, J., Choi, C., Yi, K., Suh, S., Park, D., and Kim, C. 2010. 3D user interface combining gaze and hand gestures for large-scale display. In Proceedings of the 28th of the international Conference Extended Abstracts on Human Factors in Computing Systems, 3709-3714.

[5] LI J., and JAMES J. C. 2008. Video Game Design Using an Eye-Movement-Dependent Model of Visual Attention. In Proceedings of ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 4, No. 3, Article 22.

[6] Holman, D. 2007. Gazetop: interaction techniques for gaze-aware tabletops. In CHI '07 Extended Abstracts on Human Factors in Computing Systems, 1657-1660.

[7] Yamamoto, M., Nagamatsu, T., and Watanabe, T. 2010. Development of eye-tracking pen display based on stereo bright pupil technique. In Proceedings of the 2010 Symposium on Eye-Tracking Research & Appli-cations, 165-168.

[8] Yamamoto, M., Komeda, M., Nagamatsu, T., and Watanabe, T. Development of Eye-Tracking Tabletop Interface for Media Art Works. Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces 2010, pp.295-296

[9] Nagamatsu, T., Iwamoto, Y., Kamahara, J., Tanaka, N., and Yamamoto, M. 2010. Gaze estimation method based on an aspherical model of the cornea: surface of revolution about the optical axis of the eye. In Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications, 255-258.

[10] http://en.wikipedia.org/wiki/Hyakunin_Isshu

[11] Land, M. F. 2006. Eye movements and the control of actions in everyday life, In Progress in Retinal and Eye Research, 25, 3, 96-324.

Figure 10. Results of cats questionnaire.

19％(1)

５18％(2)

446％(5)

318％(2)

29％(1)

(5) Hints were comprehensible

13％(1)

５19％(6)

434％(11)

328％(9)

216％(5)

(6) I played an enthusiastic game

(7) The motion of the boss cat was quick enough

17％(3)

５15％(6)

444％(18)3

15％(6)

219％(8)

Figure 11. Divided by eye-tracking precision.

***** ** P < 0.01

* P < 0.05

1

2

3

4

5

higher than threethree and under

(1) (2) (6)(5)(3) (7)

(1) I enjoyed the game.(2) I want to play again.

(6) I played an enthusiastic game.(5) Hints were comprehensible.(3) I became interested in Hyakunin-Isshu.

(7) The motion of the boss cat was quick enough.