MultiCross –

A programming Library for multi-party

Collaborative Applications Bachelor Thesis

Silvan Troxler

University of Fribourg 2011

Professor: Prof. Rolf Ingold Head of the DIVA Research Group

Advisors: Dr. Denis Lalanne Dr. Bruno Dumas Matthias Schwaller

1

Table of Contents

1 Introduction .............................................................................................. 4 1.1 Context / Situation ...................................................................................................... 4 1.2 Motivation ................................................................................................................... 5 1.3 Goal ............................................................................................................................. 5 1.4 Approach and Report Organisation ......................................................................... 6

2 Environment ............................................................................................. 7 2.1 Hardware ..................................................................................................................... 7

2.1.1 Wiimote .................................................................................................................. 7 2.1.2 DiamondTouch ...................................................................................................... 7

2.2 Software ...................................................................................................................... 7 2.2.1 WiiRemoteJ ........................................................................................................... 7 2.2.2 BlueCove ............................................................................................................... 7 2.2.3 WIDCOMM Bluetooth Stack .................................................................................. 8 2.2.4 DiamondTouch Developer’s Kit ............................................................................. 8 2.2.5 Choice of programming language – Java .............................................................. 8

3 Development Process .............................................................................. 9 3.1 Client/server architecture .......................................................................................... 9

3.1.1 Design .................................................................................................................... 9 3.1.2 Implementation .................................................................................................... 11 3.1.3 Messages ............................................................................................................ 12

3.2 Developing the Library ............................................................................................ 13 3.2.1 Application Data and Logic .................................................................................. 14 3.2.2 Application specific messages ............................................................................. 14 3.2.3 Adding multiple devices support .......................................................................... 14

3.3 Library Improvements ............................................................................................. 15

4 MultiCross Library ................................................................................. 17 4.1 Package Client .......................................................................................................... 17

4.1.1 Class App ............................................................................................................ 17 4.1.2 Class AppPanel ................................................................................................... 17 4.1.3 Client-Class ......................................................................................................... 18 4.1.4 Thread-Classes ................................................................................................... 18

4.2 Package Server ........................................................................................................ 19 4.2.1 Class Server ........................................................................................................ 19 4.2.2 Class ClientThread .............................................................................................. 19 4.2.3 Class User ........................................................................................................... 20 4.2.4 Class AppInfo ...................................................................................................... 20

4.3 Extending the MultiCross Library ........................................................................... 21

2

4.3.1 Adding a new device type .................................................................................... 21 4.3.2 Adding a new event ............................................................................................. 21

5 Validation ................................................................................................ 22 5.1 Use case 1: „Ligretto“-Game .................................................................................. 22

5.1.1 Game Description ................................................................................................ 22 5.1.2 Creating a Server ................................................................................................ 22 5.1.3 Creating a Client .................................................................................................. 23 5.1.4 Adding behavior ................................................................................................... 25 5.1.5 Adding Images ..................................................................................................... 25

5.2 Use case 2: “DiffusionBarrier”-Game .................................................................... 26 5.2.1 Game description ................................................................................................. 27 5.2.2 Display technique ................................................................................................ 27 5.2.3 Simultaneous actions ........................................................................................... 27

5.3 Performance Test ..................................................................................................... 27 5.4 Validated Features ................................................................................................... 28

6 Discussion .............................................................................................. 30 6.1 Review Development Process ................................................................................ 30 6.2 Review MultiCross Library ...................................................................................... 30

6.2.1 Achievements ...................................................................................................... 30 6.2.2 Improvement capability ........................................................................................ 31

7 Conclusion .............................................................................................. 32

8 Bibliography ........................................................................................... 33

9 Annex ...................................................................................................... 34 9.1 Acknowledgment ...................................................................................................... 34 9.2 How to get the MultiCross library work .................................................................. 34 9.3 Client-Server messages of Ligretto-application ................................................... 35 9.4 Client-Server messages of DiffusionBarrier-application ..................................... 36 9.5 Source Code ............................................................................................................. 37

3

Abstract This report illustrates the development process of a the java library MultiCross, which allows

developers to easily create their own multi-party applications using a stable communication

over the network and using the DiamondTouch Table, Wiimotes as well as the standard

mouse as input devices. Such a library simplifies the developers’ work when creating a col-

laborative application. Beside the native support for the mentioned devices, the MultiCross

Library can be extended in order to work with different devices and to add or modify behav-

iour to the application.

The whole server/client architecture as well as the library itself were firstly planned and then

implemented. Parallel to the library, two applications were designed and implemented as a

help for building and improving the library, but also as use cases to validate the MultiCross

Library at the end.

The development and the result were discussed critically and improvement ideas were given.

The goals of having more than one user per machine and more than one client in the net-

work, but also having support for the DiamondTouch and Wiimotes were fully fulfilled. The

newly formed library can be extended in order to include new devices, new events and there-

fore match the different applications’ needs.

This report documents the whole development process and therefore gives you an idea of

how the goal was achieved and what the major challenges were. Furthermore, with the help

of the two use cases, an example is given of how the library can be used in order to build a

new application and to adjust it for one’s own need.

4

1 Introduction

1.1 Context / Situation Applications are improved all the time and are getting better and better. Developers are al-

ways looking for new kinds of applications to improve them even more. A fairly new approach

of the last several years are collaborative applications, where several people can use the

same application at the same time, working together at one thing and share the application’s

resources.

At the University of Fribourg, different collaborative applications have been developed. For

example there is a mindmap application running on the DiamondTouch Table. This applica-

tion allows up to four people to work on a mindmap, all of them using the DiamondTouch. An-

other application developed in the DIVA group of the University of Fribourg is the so-called

“Communication Board”, which allows two people on different machines to work on the same

application, always seeing the opponent’s actions as well as the person on the other machine

by using videoconference. Additionally, this application can be controlled by a Wiimote.

Both applications are pretty limited in their use. The mindmap application is a single-machine

application and can only be run on the DiamondTouch Table. The “Communication Board”-

application can indeed be used in the network but is limited to two machines and one user per

machine.

This is not only the trend at the University of Fribourg, we can also see this tendency at all the

collaborative applications.

Generally, there are four different kinds of collaborative applications as Figure 1.1 shows.

Figure 1.1: The main types of collaborative applications1

1 Inspired by Robert Johansen (1988): Groupware. Computer Support for Business Teams,

The Free Press, New York and London, 1988.

5

Concentrating on the left part of this graphic, we can see that nowadays, there are principally

two kinds of synchronous collaborative applications: Face to Face and Remote Interaction

Applications.

Face to Face or Single Display Groupware are applications in which multiple users collabo-

rate on a single machine only. For the development of such applications different toolkits

have been programmed in order to support multiple mouse pointers (B.B.Bederson, 1999;

S.Greenberg, 2004).

Different approaches are followed in this field. Hutterer et al. added multiple input device sup-

port to the X.Org server to mention only one example. Therefore, not only multiple mouse

pointers but also multiple keyboard interaction is possible (P.Hutterer, 2007).

Remote Interaction Applications are applications, which allow two or more users to interact

simultaneously over a distance. This is done by using a network or the Internet to pass data

from one computer to another with the aim of supporting efficient collaboration over a dis-

tance. There are already lots of remote interaction software, mainly in the electronic meeting

and videoconference area, network games are also examples of applications of this kind.

What all of these applications has in common is the fact, that they only provide support for

one single user per machine.

1.2 Motivation The two applications described above, developed at the University of Fribourg, as well as the

most today’s collaborative applications are good in their kind but not flexible enough. Espe-

cially the code of the mentioned applications is barely reusable and doesn’t support the de-

veloper by doing his work. For developers, it would be great to have a more general solution

in order to develop multi-party collaborative applications and therefore also close the gap be-

tween collocated and remote applications.

1.3 Goal In this project, the two different approaches - having more than one user per machine on the

one hand, and to have more than one machine in the network on the other hand - are com-

bined. The result will be a collaborative multidevice network application with the following

characteristics:

- At least 10 people can interact at the same time

- Indefinite number of machines in the same network

- DiamondTouch and Communication Board/Wiimote Support

The idea of the project is to program two different multidevice network applications and to

develop a small library to make it easier for other developers to create such collaborative mul-

tidevice network applications.

6

Furthermore the network as well as device performance can be tested with the programmed

applications.

1.4 Approach and Report Organisation The project was be divided into four major parts:

1. The Design & The Implementation of a client/server architecture

2. The Card-Game Application “Ligretto”

3. The Second Application “DiffusionBarrier”

4. Developing the Library in parallel.

After having dealt with the hard- and software used for this project, the client/server architec-

ture is explained. In chapter 4, the library is described. The two different applications ex-

plained in chapter 5 provide good examples of how to use the library.

At the end of this report, you will find a short evaluation as well as a conclusion part.

7

2 Environment

2.1 Hardware

2.1.1 Wiimote1

The Wiimote (short for Wii Remote) is the controller of the game console Wii of the jJpanese

company Nintendo. It is equipped with an infrared camera and an acceleration sensor. Differ-

ent libraries in many languages are available in order to use the Wiimote for computer appli-

cations.

2.1.2 DiamondTouch2 The DiamondTouch Table of Circle Twelve Inc. allows multiple people to interact simultane-

ously without disturbing each other. This feature sets the DiamondTouch Table apart from

other products. In order to distinguish the people the DiamondTouch uses capacitive cou-

pling.

2.2 Software

2.2.1 WiiRemoteJ3

In order to use the Wiimote as a control device for an application, there are different Java li-

braries available. After having looked at some of them, I decided to use WiiRemoteJ since it

has multiple Wiimote support, which is mandatory to achieve the project’s goal. To get

WiiRemoteJ, and therefore the Wiimote as a device, working on windows, a few points should

be taken into account:

- BlueCove is needed

- The WIDCOMM bluethooth stack has to be installed

- A bluetooth dongle based on a broadcom chip must be used

2.2.2 BlueCove4

BlueCove is a Java library for Bluetooth. It is needed to work with the WiiRemoteJ library to

connect the Wiimotes via Bluetooth to the computer. It should be considered that there are

different distributions for different platforms.

1 http://en.wikipedia.org/wiki/Wii_Remote 2 http://www.merl.com/projects/DiamondTouch/ 3 http://www.world-of-cha0s.hostrocket.com/WiiRemoteJ/ 4 http://bluecove.org/

8

2.2.3 WIDCOMM Bluetooth Stack1

To get WiiRemoteJ working properly, it is essential that the WIDCOMM bluetooth stack is in-

stalled on the machine. This software is free and can be downloaded on the website of

Broadcom. Unfortunately, the WIDCOMM bluetooth stack doesn’t work with all bluetooth de-

vices but only with those which have a broadcom chip. As I have seen, it’s not always easy to

find out which chipset was used, since the product descriptions of the bluetooth dongles rare-

ly give information about it. At present, Belkin is using Broadcom chipsets in their dongles.

2.2.4 DiamondTouch Developer’s Kit2

Circle Twelve Inc. provides a Developer’s Kit for their DiamondTouch Table. This Kit simpli-

fies the development of DiamondTouch-applications, since it comes with great features like

getting the device, getting the device’s state or providing the point where a touch occurs.

2.2.5 Choice of programming language – Java Since Java is well known and widely used, it is useful to develop this library in this language.

Most of the modern input devices, such as the DiamondTouch, provide Java support. Also,

with the Sockets (see section 3.1.2), Java is providing a good tool for message passing from

one computer to another.

1 http://www.broadcom.com/support/bluetooth/update.php 2 http://www.merl.com/projects/dtsdk/

9

3 Development Process In order to develop the library, two applications were programmed. Even if this required a lot

of work and most notably a lot of revision during the whole process, it was a good way to see,

which functions the library should provide and how it can be improved in order to be useful for

different applications.

The development of the library and the two applications can be divided in several parts:

1. Designing and implementing of a client/server architecture

2. Designing and implementing application 1 and the library

3. Designing and implementing application 2 and improving the library

In this chapter, the whole development approach with its major problems is going to be shown

and explained. The results of this process, thus the library and the applications themselves

are then presented in the following two chapters.

3.1 Client/server architecture

3.1.1 Design

For a collaborative application that can be executed on several computers in a network at the

same time, it is important to have a good client/server architecture.

There are several possibilities of how the server and the clients can communicate with each

other. All machines could be interconnected and therefore pass messages to any other client

or server.

Figure 3.1: Network with interconnected clients1

1 inspired by http://decode-abap.blogspot.com/

10

Another approach would be to have one central server and have all messages pass through

this server in order to get to another client. In this case, for every change a client wants to

share with another client, it has to send a message to the server and is not able to inform the

client directly. The server handles all incoming messages, executes the desired action and

then informs the other clients.

Figure 3.2: Network with central server1

For this project, the second approach was chosen with the following advantages:

• This implementation provides a better control of what the application is actually doing,

as all the messages have to pass through the server.

• Imagine the clients could communicate directly to each other. It could happen, that

two clients are sending messages at the same time, but the other clients in the net-

work are not going to receive the messages in the same order and therefore, they

would have different program states at the same time.

Additionally, for a multi-party application, we want to have more than one device per client.

Beside the “server” and the “client”, we therefore need to have a third entity “user”. One client

could have multiply users and handles all user interactions and communicates them to the

server if necessary.

To sum up, let’s have a look at the most important entities for the collaborative application:

1 http://decode-abap.blogspot.com/

11

Entity Description

Server The central unit which handles the whole application

Client One single machine

User One single Input Device, such as a hand on the DiamondTouch or a

Wiimote at the communication board. One Client can have more than

one User.

Table 3.1: Most important entities for client/server structure supporting multi-party interaction

3.1.2 Implementation Inspired from an existing open source code1 found on the Internet a solution, which includes

all the needs mentioned in the section before, was implemented.

This solution utilises the well-known observer pattern. The server is the observer, observing

all registered clients. In the java.util package, the two abstract classes Observer and Observ-

able are available and can be subclassed in order to use this design pattern.

The following figure shows the observer pattern and the adjusted implementation for the cli-

ent/server architecture used for this library.

Figure 3.3: Observer Pattern (top) and adjusted implementation (bottom)

Every time a Client connects to the Server, a new instance of the class ClientThread is creat-

ed. This is mandatory, since in Java a ServerSocket object can only receive a connection re-

quest but not handle any messages sent from the client. For each new connection, a sepa-

rate socket is created. Over this socket, messages can be passed from one end to the other.

In this case, the socket is forwarded to the clientThread which can then receive and send 1 http://www.developer.com/print.php/1356891

12

messages from and to one client. Since the clientThread is observed by the server, it can also

notify the server in order to send a message over the different clientThreads to all clients.

3.1.3 Messages

Thus, a clientThread is responsible to receive messages from the client. Also, it parses the

messages and decides how to respond or what to do.

All messages are composed the same way. The first (and sometimes the second) word is the

keyword with the help of which the clientThread decides what to do. All other words or num-

bers are additional information which might be needed. All the words and numbers are sepa-

rated by a semicolon.

The following table lists all possible messages between client and server and explains what

they stand for. A practical example is given afterwards.

Client to Server - Messages

IP;<IP-address>

This is always the first message sent from the client to the server. It is a registration on the

server, telling the server that a new client wants to join the application. The server will then

respond to this request by giving the client a specific client number.

USER;NEW; device_nr>

For each input device at a client’s machine, a user should be created. For this purpose, the

client sends a message to the server, informing the server about the device. The device

number can be chosen by the client itself, but has to be unique. Therefore, it is recommended

to use the client number multiplied by 10 and add 1 for each new device so that we have for

example the device number 21 and 22 on the client number 2.

POSITION;<device_nr>;<x>;<y>

This is the most basic message. It consists of the keyword “POSITION”, the device number

as well as the x and y position of the cursor. This message is used to indicate a change of the

coordinates of the cursor of a device.

Server to Client - Messages

RESP;<client_nr>

Whenever a new client wants and is allowed to join the application, the server respond with

this message. <client_nr> would then be replaced by the actual client number which is used

in messages afterwards. This message is only sent to the client, who wanted to register on

the server.

USER;NEW;<client_nr><device_nr>

Whenever a new user is created, the server responds to all its clients by sending this mes-

sage, in order to ensure that all clients are aware of the fact that a new user has joined the

application.

13

POSITION;<dev_nr>;<x>;<y>

After receiving a cursor position change from a client, the Server automatically sends this

message to all its clients in order to inform them about the change.

ERROR;<message>

If an error occurs while handling the received message, a error message can be passed to

one or all clients by using this message. After the keyword “Error” the error message is

passed as a string.

Here is an example of how a usual communication between the server and its clients could

look like. Suppose we have the server and two clients communicating with each other.

Figure 3.4: client/server communication

In this case, the first client registers itself on the server. Then the second client is doing the

same thing. Client 1 then tells the server, that there is a new user with device id 11. The serv-

er responds to all known clients and informs them that a new user at client 1 with device nr 11

was created. A moment later, the position of the user with device id 11 is changed and client

1 who is aware of that change, sends an adequate message to the server which again in-

forms all its clients about the position change by sending exactly the same message to all

clients.

3.2 Developing the Library The Library and the first application were developed in parallel. Therefore, the first step of

developing the library was actually to design the first application. After having dealt with all

the needed classes and functions, the application specific parts were separated from the

more general parts which could also be used for other applications. Also the client/server-

implementation was included in the library.

14

Since the Library as well as the Applications are explained in detail later, only the structure

and the important points in the development process are treated at this point.

3.2.1 Application Data and Logic

One of the major decisions was to determine whether the application is running on the server

or independently on each client and therefore, where to store the application data. Since for

the first application a shuffle-function is needed, it makes more sense to have run it on the

server side. Suppose each client would shuffle an array itself. Every client would then have a

different order in this array and synchronisation would not be possible. However, if the server

shuffles the array and then passes the order to all its clients, they will all have the same state

at the end. For this reason, the decision was made to run the application on the server, or at

least to let the server keep track of the most important things in order to ensure synchronisa-

tion.

In the shuffle example above, the server would shuffle the array and pass the order to all its

clients. However, whenever the clients need the next element from the array, they don’t have

to ask the server again, since all of them have exactly the same elements in the same order.

Thus, some data is stored both on the server and at each client, but the data on the server is

effective in case of an asynchronous state.

While the server is therefore in charge of the application data, the visualisation of the data is

the clients’ business.

With this approach, another great advantage arises. Since the server knows all about the ap-

plication, it can restore the application state at any time. If, for example, a client is joining the

application later, the server can send all the needed information in order to update this client

about the current application state.

3.2.2 Application specific messages

Of course, in addition to the messages used for the client/server architecture, the developer

should be able to define application specific messages. Since any message can be sent from

and to a client, one only has to be sure the client or the server can decode the message. For

this purpose, the parsing-function of the ClientThread as well of the Client class has to be

adjusted, adding the keywords to recognize the message type and assigning an action to per-

form.

3.2.3 Adding multiple devices support

The library should be able to handle different devices, such as a normal mouse, the Dia-

mondTouch Table as well as Wiimotes. For this purpose the Client class was sub classed for

each type of input device in order to add device specific functions. The client then creates a

separate thread that listens to all inputs on the device. The performed actions on a given in-

put (for example a click or a dragging), have to be specified in the run-method of the thread.

15

Figure 3.5: Simplified class diagram of the client package

3.3 Library Improvements While programming the second application, the strength and the weakness of the existing

library could easily be seen. Since the aim was that the library code has not to be changed to

write an application, but can easily be extended, all the changes, which must have been

made to the library code while implementing the second application, were the library’s weak

points and required a revision.

The following list shows the main problems and their solutions or rather adjustments made to

the library.

The application specific messages had to be added to the ClientThread class. An additional Parser class was created. Any subclass of this Parser class can now be set as

the parser of the application. Every time the server or the client receives a message, the

evaluate-method of this object is called.

To set the application specific parser, an instance of the subclass has to be passed as a con-

structor argument when instantiating a new client or accordingly assigned to the parameter

“parser” of the server object.

Duplicated code in the different client classes

A parent class was created for the different Client classes. This can avoid duplicated code on

the one hand and provide an easy way to integrate new devices on the other hand.

16

In order to have a convenient way to create one of the client’s subclasses, a ClientFactory

class was created.

Inflexible thread classes in order to extend the event based behaviours The thread classes were revised. For each device, the input is analysed by this thread class

and click, press, drag and release events are filtered. Whenever one of this event occurs, a

corresponding method is called. This method can also be changed when subclassing the

thread class. This approach gives more freedom to the developer and is hiding the device-

specific input treatment.

Thus, the programmer is free to add the same behaviour for all devices or to react differently

to the events depending on the device type.

Not enough freedom in using the library

At some points the programmer had to choose between using the library and to disclaim of

his freedom or to program it himself. Since this is not desirable some hooks were included

were the developer can add own behaviour to some functions, but still has all the library’s

advantages.

Extra-Features

To simplify the programmer’s life, a few extras were included in the library, such as an ex-

tended JFrame which provides a good starting point for the applications’ graphical output as

well as some useful functions and a standard interface for choosing the device type.

Also, a logging was included in order to simplify the debugging. All messages received over

the network communication are added in the log-file. For the server as well as for each client,

it can be determined whether the logging is turned on or off.

17

4 MultiCross Library The Library is divided into two packages, the client and the server package. While the classes

of the client package mainly serve the application’s client, the classes of the server class are

important for the server of the application. While only the main classes and functions are pre-

sented in this chapter, examples how to use the different classes are given in chapter 5 of this

report. Furthermore, the complete source code and the javadoc of either the library and the

two use cases can be found on the enclosed CD.

Figure 4.1: Simplified UML diagram of MultiCross Library

4.1 Package Client

4.1.1 Class App

Every application which makes use of the MultiCross library, has to be started by creating an

App object or an object of an App’s subclass. This Class provides some useful functions to

the developer such as opening the interface to choose a device type or to connect to the

server.

The JFrame class is extendend by this class. Therefore, a new instance automatically creates

a JFrame. The content pane of this window has to be of the type AppPanel (or a subclass

thereof), and can either be passed as an argument to the constructor or set afterwards by

calling the method setAppPanel(appPanel).

4.1.2 Class AppPanel

The AppPanel class extends a JLayeredPane of the java.swing package. The only mandatory

thing is to specify the size of the JLayeredPane by setting the variable dimension of type Di-

mension as well as to define a JPanel named cursorPanel as one Panel of the JLayered-

Pane, on which by default, the cursor is displayed afterwards. Apart from that, the developer

is absolutely free in creating his own layout and his own design.

18

4.1.3 Client-Class

The Client class describes a usual client of an application. Such a client is capable of con-

necting to, as well as communicating with, a server.

When instantiating a new Client Object, an application panel of type AppPanel, a parser of

type Parser and a Boolean, to indicate whether logging is on or off, has to be passed as ar-

guments to the constructor.

Here is a list of the most important and most often needed methods of the Client class.

connect(String hostName, int port)

is called to connect to the server. This method needs to be executed before messages can be

sent to the server. After calling this method, the client is listening to all messages comming

from the server.

sendmessage(String msg)

sends a message to the server if it is connected.

afterUserCreation(User u)

hook method to specify an action after a new user has been created.

Table 4.1: The most often needed methods of the class Client

The MultiCross library already includes three subclasses for the three different device types

“Standard Mouse”, “DiamondTouch Table” and “Wiimotes”. Each of this subclass includes

some methods especially needed for the corresponding device. Also, each Client class cre-

ates a new Thread object (see section 4.1.4) and starts it in order to listen to the input on the

specific device.

4.1.4 Thread-Classes

For each device, a separate thread class has to be written in order to listen to the device input

and filter the different events which can occur. The library already includes the thread classes

for the supported device mentioned in the section before. For all of these devices, the follow-

ing events inspired from the standard mouse events are handled:

• Click

• Press

• Release

• Drag

• Moved

If one of this event occurs, the standard actions are performed and the corresponding handle-

function is called. For example if a click event occurs, the position of the device is also updat-

ed automatically by sending an appropriate message to the server. Then, the method

handleMouseClick() is called which, by default, does not perform any action. However, this

19

method can be filled with actions when subclassing the class and thus can be adapted to re-

alise the programmer’s intentions.

4.2 Package Server

4.2.1 Class Server

As explained in section 3.1, the server class is responsible for the clients’ registrations and for

creating clientThreads to listen to this clients’ messages. There are only a few methods which

could be useful to use in your server’s Parser class.

hasUser(int device_id)

returns true if a User with the passed device_id exists. Returns false otherwise.

getUser(int device_id)

Returns an object of type User with the device_id passed as an argument. Before using this

method, the method hasUser(device_id) should have been called in order to get sure that the

user exists.

hasClient(InetAdress s)

returns true if a client with the InetAdress s exists, false otherwise.

getClient(InetAdress s)

Returns an object of type Client with the InetAdress s. Before using this method, the method

hasClient(InetAdress s) should be used in order to get sure that the client exists.

Table 4.2: The Server class' most useful methods

After instantiating the server, the start()-method has to be called. Before calling the start()-

method the server’s appInfo and parser should be set using the two function setAppIn-

fo(AppInfo info) and setParser(Parser parser).

4.2.2 Class ClientThread

For each client a clientThread instance is created on the server. One clientThread is always

listening to one particular client, receives the messages of this client and can send a mes-

sage to this client as well as to all clients registered on the server.

Whenever a message is received, the evaluateString()-method of the specified parser is

called automatically. In this method the field clientThread is a reference of the clientThread

which received the message. Over this variable, messages can be sent back to the client us-

ing the method sendToClient(String msg) or to all clients using the method sentToAll(Srting

msg).

20

4.2.3 Class User

The User class holds information about the users using the application. Usually no change

has to be made to this class. Nevertheless, you can find a brief overview of the most im-

portant methods below.

setVisible(Boolean visible)

This method sets the variable visible to the value passed as an argument. This variable can

then be used in your AppPanel class to specify whether the user’s cursor should be visible or

not.

getClientNr()

Returns the number of the client on which the user is working.

getDeviceNr()

Returns the number of the user’s device.

setColor(Color c)

The first eight users automatically get a color from the MultiCross library. The following user

will all get the color black. Using this method, the color can be set manually.

Table 4.3: the most important methods of the class User

4.2.4 Class AppInfo This class is an empty abstract class. It’s only advantage is to have a new type which can be

subclassed. For a new application, this class has to subclassed and can be filled with both

fields containing all the data needed by the application’s server and functions used to manipu-

late the data.

Figure 4.2: UML of the Server Package

21

4.3 Extending the MultiCross Library

4.3.1 Adding a new device type

To include a new device type, two classes have to be written. First of all a new Thread class

has to be created in order to listen to the new device’s input. This class has to implement the

interface Runnable and therefore provide a run-method. At least the events click, press, re-

lease, drag and depending on the device the event move should be catched and the corre-

sponding handle-functions have to be created.

Additionally a new subclass of the Client class has to be created, which is in charge of start-

ing a new thread with the before created Runnable object.

As well as the ClientFactory class, the chooseDevicePanel class must also be updated.

4.3.2 Adding a new event

Maybe a developer wants to add a new event such as entered or exited. In this case, all

thread classes have to be updated, specifying for each device type when the new event ap-

pears and what should happen by its occurrence. It is also recommended to create a new

handle function for each new event type.

22

5 Validation Even if the two applications and the library were developed in parallel meaning both the li-

brary and the application grew as an entity, the two applications are a good indication for the

library’s gain. The first use case, the Ligretto game, is in need of an efficient system since it

depends on fast actions. The second use case on the other hand validates the collaborative

capabilities of the library.

5.1 Use case 1: „Ligretto“-Game Use case 1 is the famous card game Ligretto, which was implemented using the MultiCross

library.

5.1.1 Game Description

The rules of the game are pretty easy. Every player has 40 cards, numbered from 1 to 10 in 4

different colors. The aim of the game is to get rid of all the cards as fast as possible. All play-

ers are playing simultaneously and therefore attention to the cards being played by others as

well as one's own cards is needed.

When the game starts, players simultaneously discard cards in the middle of the table, build-

ing colored piles in ascending numerical order according to color, and only starting new piles

with a '1' card.

Every player has three “open cards”, a “stack” of ten cards and the rest of the 40 cards re-

main in the player’s hand. While the cards on the stack are covered, the “open cards” are vis-

ible and can be played. If an “open card” is played, a card from the “stack” replaces it. As

soon as the stack of one player is empty, the round is finished. Whenever a player can’t play

any of his “open cards”, he can look at every third card from the card in his hand and always

play the card on the top of this pile. Every played card counts one point while each remaining

card on the “stack” at the end of the round counts minus two points for this player.

5.1.2 Creating a Server

At first, let us have a look in how one can use the MultiCross library to create a new server for

its own application. To do this, a new class has to be created which includes the main func-

tion and therefore is the starting point of the application.

For the Ligretto Application, this class is called LigrettoGameServer and was created as a

singleton class such that the main-function looks like that:

LigrettoGameServer g = LigrettoGameServer.getInstance();

The constructor of this class then does actually do the main task.

server = new Server();

gameInfo = new LigrettoGameInfo();

23

server.setAppInfo(gameInfo);

server.setParser(new LigrettoServerParser(gameInfo));

server.startServer();

On line 1 the new Server is instanciated. Line 2 is responsible for creating a new instance of

the class LigrettoGameInfo which subclasses the AppInfo class of the library. This object is

then set as the server’s appInfo.

Also a parser has to be set in order to parse the application specific message which the cli-

ents will send to the server. For this purpose an instance of the parser class (or a class sub-

classed from this class) has to be set as the server’s parser (line 4).

Once the appInfo and the parser are set, the server can be started (line 5). From now on,

new clients can register at the server.

5.1.3 Creating a Client

The client side of the application is implemented similarly. First of all, a new subclass of the

class App has to be created. This class is the entry point of the application and contains the

main method. As well as the LigrettoGameServer class, the LigrettoGameClient class is de-

signed as a singleton class. Therefore the main method starts with the following statement

LigrettoGameClient game = LigrettoGameClient.getInstance();

Since the super class of this class, the App class of the MultiCross library, expects a argu-

ment of type AppPanel, a corresponding subclass LigrettoGamePanel was created and the

unique instance of this class is passed as an argument to the GameLigrettoClient class’ con-

structor. So this is how the method getInstance() of the GameLigrettoClient class looks like:

public static LigrettoGameClient getInstance() {

if(instance == null) {

instance = new LigrettoGameCl

ent(LigrettoGamePanel.getInstance());

}

return instance;

}

Please note that creating a new object of the type LigrettoGameClient is not creating an actu-

al client, but is a subclass of a JFrame. In order to create a client which can communicate to

24

the server, two more steps are mandatory. Firstly a device has to chosen. The superclass of

the LigrettoGameClient provides a graphical interface to let the user choose his device.

Therefore we call

game.chooseClient();

after having created an object of the LigrettoGameClient class and assigned it to the variable

game. Before calling this method, a ClientFactory class for your application must have been

created, and set as the clientFactory of your own App class.

Then, after having chosen the client type, the application can connect to the server by calling

game.chooseServer(String defaultServerIp);

this method lets the user edit the server ip address on the one hand and connects to the

server on the other.

Once the client is successfully connected to the server, the content pane of the application’s

JFrame is set to the specified appPanel. In the case of the Ligretto game, the content of the

LigrettoGameClient is set to the instance of LigrettoGamePanel. The Application is now fully

started.

For a better understanding of the explained process, the activity diagram of starting the Li-

gretto application is shown in Figure 5.1.

Figure 5.1: Activity Diagram of Application Start

25

5.1.4 Adding behavior After having a server and a client, the behaviour should be implemented. Let’s have a look in

how that was made in the Ligretto application using the drag event in order to move a card.

The thread classes already provide some hook method we can use in order to program be-

haviour (see chapter 4.1.4: Thread-Classes). Therefore, all we have to do is to specify for

each device which action to perform whenever the drag event occurs, So we have to override

the method handleMouseDragged(int x, int y, int device_nr) In the subclass of each thread

class.

Since for the Ligretto game the behaviour should be the same for all the different devices, the

handleMouseDragged()-method looks the same for all thread subclasses. Because we also

don’t want to have duplicate code, the instructions, which have to be performed by a dragging

event, were outsourced in the class LigrettoGameClient. Thus, the handleMouseDragged()-

method of all thread subclasses is looking like this:

@Override

public void handleMouseDragged(int x, int y, int device_nr) {

LigrettoGameClient.getInstance().handleMouseDragged(x,

y, device_nr, bigboss.clientNr);

}

In the handeMouseDragged()-method of the LigrettoGameClient, the actual behaviour was

finally implemented. In this particular case, it is checked if there is a card on the starting point

of the drag event and whether the user, which triggered the event, can move this card. If so, a

new message is generated and sent to the server in order to move the card to the next posi-

tion.

In order to ensure synchronisation, no action is performed without the server’s permission. So

when the client wants to move a card, a message with the keyword “MOVE” is sent to the

server. The server then redirects this message to all its clients, so that the change is made at

all clients at the same time.

5.1.5 Adding Images

The last step of the development of the first application was to include graphics so that the

game actually looks similar to the real Ligretto game. Since the MultiCross library provides a

lot of useful functions, but doesn’t constrain the developer in the graphical part, it was easy

and fast to include the images.

26

Figure 5.2: Screenshot of the Ligretto application

5.2 Use case 2: “DiffusionBarrier”-Game The second application, the DiffusionBarrier-Game is implemented more or less the same

way than the Ligretto application. Therefore, it won’t be explained in detail again at this point.

However, there are some points that extend the previous section and differ from the Ligretto

application. Those points should be pointed out here.

Figure 5.3: Screenshot of the DiffusionBarrier application

27

5.2.1 Game description DiffusionBarrier simulates a very simple cell in which different ions are diffusing. The players

are the membrane of the cell, allowing some ions either to pass or not.

It is a four-player game where always two players form one team. The team on the left side

should try to get all green ions on the left side, while the team on the left should bring all blue

ions on the right side. Both teams should avoid having red ions in their zone on the left or ra-

ther the right margin.

In order to control whether the ions may pass or not, each player can move the hole in his

diffusion barrier by moving the mouse, the Wiimote or the finger up or down. All ions can only

pass the barrier at its hole. Therefore, in order to pass the diffusion barrier, both players of

one team have to place the hole on the same spot. The aim is, to keep the good ions on the

players’ zone while getting rid of the bad red ions. Depending on the color, the ions have

positive or negative scores, which are counted separately for each team. The team with the

higher score after one minute wins the round.

5.2.2 Display technique

While for the Ligretto application a layoutmanager and java.swing objects were used to de-

sign the game, absolute positioning and geometric forms such as rectangles and circles were

used for the diffusionBarrier application. This shows, that the library is flexible and lets the

programmer freely choose between the different design approaches.

5.2.3 Simultaneous actions For the diffusionBarrier game, it must be checked whether two clicks of different users are

made within a small space of time. For this purpose, the ClientThread class of the MultiCross

library provides the method checkSimultaneousTime(). The following parameters have to be

passed as arguments:

• Timestamp t1: Timestamp of first action

• Timestamp t2: Timestamp of second action

• Int msec: the maximal number of milliseconds between the two actions.

The method then returns true, when the difference between the two timestamps t1 and t2 is

smaller than msec and returns false otherwise.

This method was used to implement the functionality of the freeze-button of the diffusionBar-

rier game. Only if the two players of the same team are pressing the freeze button at the

same time (space of time = 1000ms), the balls in the middle area are actually frozen. Other-

wise, nothing happens.

5.3 Performance Test One of the most important questions is whether the application, mainly the network speed, is

fast enough to run the applications. Of course, the speed depends on the network type. But it

28

still is important that the application itself is fast enough. Therefore, some tests where made

in order to evaluate the university’s network capacity on the one hand and the application

speed on the other.

For this purpose, the logging of the application was used. Since the exact time (Timestamp) is

recorded on every message a client sends to the server and also on every message the client

receives, we can see how long it takes to actually perform an action.

Let us observe the following case on the Ligretto game:

One player wants to move a card. In order to do that, the client is sending a message to the

server. The Server than sends this message to all its clients. Nothing happens on the client

until it receives this message. How long does it takes from starting the move-action until it

actually performs?

The following table shows the recorded amount of time passed between the user’s dragging

action and the actual move of the card, once when the server is running on the same ma-

chine as the client, and once when they run on two different machines within the network of

the university.

# Local Network

1 8ms 16ms

2 16ms 135ms

3 1ms 16ms

4 3ms 31ms

5 6ms 31ms

6 7ms 15ms

7 7ms 184ms

8 2ms 8ms

9 16ms 22ms

10 12ms 88ms

Average 7.8ms 54.6ms

As it may have been expected, the communication between the client and server is much

faster when both parties are running on the same machine. But even if they have to com-

municate over the network, it is fast enough to have a fluently running application.

5.4 Validated Features To sum up the validation of the library we have seen that two different applications could be

implemented without making any change on the library itself. The library is therefore inde-

pendent from the application and can be used for any kind of collaborative applications. The

library has the following services:

29

• Multidevice capability: The library can handle more than one client and different input

device types.

• Multipoint capability: The library can handle multiple users per client.

• A stable client/server communication with an unlimited number of clients and users

• Extendable behaviours for all input events

• New input events can be added

• New Device Types can be added

• Logging functionality

Figure 5.4: People playing Ligretto on the DiamondTouch table and the CommunicationBoard.

30

6 Discussion The two use cases indicate the gain of the MultiCross library. Nevertheless, a critical view

should highlight the good as well as the bad points in the development process and judge the

capability of the library, also bring up the library’s weak points and it could be improved.

6.1 Review Development Process Even if there were made some UML class diagrams before the library and the applications

were implemented, this part could have been done better. For such a project, it is important to

have a clear conception of what should be achieved and how it should be implemented, be-

fore you actually start programming. A lot of time can be saved and could also have been

saved in this project if the planning part had been carried out even more thoroughly.

After having planned the things to perfection, it also is necessary to always keep in mind the

overall structure and the overall goals of a project. When the first application was pro-

grammed, it was programmed like this would be the single part of this project. A lot of things

could already have been implemented if the larger idea of having a library at the end had

been taken into account even more.

One of the best parts on the other hand was to program the second application. In order to

have a useful library at the end, it was extremely useful to have not only one, but two use

cases, which were implemented while the library was developed. This gave the possibility to

see what is needed in order to create a library that can be used for different applications with-

out the need of changing the library’s code. A lot of the library improvements could be done

while implementing the second application. Of course, this needed some rework on the first

application. But with this effort only, it was possible to have two applications at the end, based

on the exactly same library without having changed one single line in the library’s source

code.

6.2 Review MultiCross Library As the two use cases illustrate, the library is basically working fine for different applications.

Nevertheless there is always room for improvement. The good points should pointed out first

and the ideas for improvement were mentioned afterwards.

6.2.1 Achievements

A stable implementation of a client/server architecture could be programmed. The chosen

implementation allows having an infinite number of clients and users. This increases the val-

ue of network applications a lot. Also, the server is running independently, not having the

function of both the server and a client on the same time. Therefore, the client and the server

can be developed totally independently.

31

Another outstanding point is the ability to adapt the library for its own use. On several posi-

tions in the library, so-called hook methods are added in order to give the developer the pos-

sibility to add the behaviour of the library without changing the library’s code directly. This al-

lows the developer great latitude and makes the library more useful.

6.2.2 Improvement capability

Even if the library already provides some flexibility and freedom for the developer, the usabil-

ity could still be improved. The developer is indeed free in adding his own behaviour to the

application, but as the case may be, he has to sublass and edit several classes in order to do

so. A better handling of the behaviour would increase the usability of the library a lot. Ideally,

the developer could add the desired behaviour at one single place in order to get it work on

every device. However, it should not prohibit the possibility of having different behaviour for

different device types.

Further, the debugging support of the library could be improved. Errors relying on the library,

such as connection errors, should be captured and included in the log-file. A great thing for

developers would be a level based debugging where one can specify the captured error

types.

At that time, the library automatically takes care of the client and user creation. But by default

nothing is done when a client leaves the application. It would be an improvement to have also

an automatic system taking care of client and user destruction.

32

7 Conclusion The product of this bachelor thesis is the MultiCross library, which allows developers to easily

create their own multi-party applications using a stable communication over the network and

using the DiamondTouch Table, Wiimotes as well as the standard mouse as input devices.

The whole server/client architecture as well as the library itself were firstly planned and then

implemented. Parallel to the library, two applications were designed and implemented as a

help for building and improving the library, but also as use cases to validate the MultiCross

Library at the end. The development and the result were discussed critically and improvement

ideas were given. The goals mentioned at the beginning of the project, namely to have more

than one user per machine and more than one client in the network, but also to have support

for the DiamondTouch and Wiimotes were fully fulfilled. The newly formed library can be ex-

tended in order to include new devices, new events and therefore match the different applica-

tions’ needs.

The whole library is available on sourceforge.net1. Therefore everyone can download and use

the library, but also edit it in order to improve the library and therefore make it even more use-

ful for other developers.

The momentary version of the library provides a good base for further development and more

services. For example, video and audio conferencing could be included. The challenge of this

feature would be how the different users on one machine are handled.

Further, a great improvement would be to integrate keyboard input such that the applications

are not confined to click events.

1 http://sourceforge.net/projects/multicross

33

8 Bibliography • B.B.Bederson, J. J. (1999). Architecture and Implementation of a Java Package for

Multiple Input Devices (MID) (Vols. 99-08). Maryland, USA: University of Maryland.

• P.Hutterer. (2007). Groupware support in the windowing system. In Procedding of the

eight converence on Australasian user interface. Balarat, Australia: Australian

Computer Society.

• S.Greenberg, E. a. (2004). Rapidly prototyping single display groupware through the

SDGToolkit in Proceeding of the fifth converence on Australasian user interface.

Dunedin, New Zealand: Australian Computer Society.

34

9 Annex

9.1 Acknowledgment Firstly I would like to thank Denise Lalanne who supported me during the last two semesters

and provided me assistance for my bachelor project. His kindly manner and effort in helping

me find ideas and solutions for any kind of problem made it pleasurable for me to work with

him on my bachelor thesis.

Further, I also own thanks to Matthias Schwaller for supporting me in the more technical area

and for handing out the lab keys about a 100 times.

Thanks also go to Bruno Dumas who helped me particularly in the first period of the project,

to the Departement of Informatics of the University Fribourg, as well as to all other people that

were in any form part of my project.

9.2 How to get the MultiCross library work In order to use the MultiCross library, follow the instructions below.

1. Copy all needed libraries from the enclosed CD.

Please consider that there are two versions of Bluecove:

BlueCove1.1.0 for Windows

BlueCove1.1.1-SNAPSHOT for Mac OS X

2. Also copy the MultiCross, if you don’t need to change the library. If you want to im-

prove or adapt the library, you can find the source code on the CD as well.

3. Include all the needed libraries in the Java Build Path.

4. For using the Wiimotes on Windows, the WIDCOMM Bluetooth-Stack should be

downloaded and installed. Alternatively you can also find a version on the CD. For

Mac OS X, the AvantanaBluetooth and RTXTxomm-libraries are needed. These li-

braries has to be copied to the /Library/Java/Extensions-folder.

The library and the use case applications were tested on the following platforms:

• MouseClient on Windows XP/Vista/7, Mac OSX 10.6

• WiimoteClient only on Windows Vista, using the WIDOMM Bluetooth Stack and Bel-

kin Bluetooth dongle

• DTouchClient on Windows XP

35

9.3 Client-Server messages of Ligretto-application Server to Client messages

PLAYER;NEW;<device_nr>;<player_nr>;<pos_x>;<pos_y>

PLAYER;UPDATE;<device_nr>;<player_nr>;<pos_x>;<pos_y>

START

MOVE;<player_nr>;<card_nr>;<pos_x>;<pos_y>

DROP;<player_nr>;<card_nr>;<stack_nr>

LAY;<player_nr>;<card_nr>;<card_attributes>

RESET;<player_nr>;<card_nr>

SCORE;<player_nr>;<score>

STOP;<player_nr>

Client to Server messages

PLAYER; NEW;< device_nr>;<player_nr>;

PLAYER;UPDATE

START

GET;<player_nr><card_nr>

MOVE;<player_nr>;<card_nr>;<x>;<y>

DROP;<player_nr>;<card_nr>;<stack_nr>

RESET;<player_nr>;<card_nr>

LAY;<player-nr>;<card_nr>;<card_parameters>

36

SCORE<player_nr>;<score>

9.4 Client-Server messages of DiffusionBarrier-application Server to Client messages

ION;NEW;<id>;<color>;<x>;<y>;<dir_x>;<dir_y>;<frozen>;<visible>

ION;UPDATE;<id>;<color>;<x>;<y>;<dir_x>;<dir_y>;<frozen>;<visible>

PLAYER;NEW;<device_nr>;<player_nr>;<pos_x>;<pos_y>

START

CLOCK;<sec>

FREEZE;<sec>

STOP

Client to Server messages

PLAYER; NEW;< device_nr>;<player_nr>;

ION;NEW

START

FREEZE;<device_nr>

37

9.5 Source Code The complete source code of the library as well as the two use case applications can be

found on this CD as well as at http://sourceforge.net/projects/multicross.